Ponyland
Wiki: https://ponyland.science.ru.nl/
See overview of server and server names.
Most people connect to Applejack first, because it's the only one you can access without VPN. This is a web server so do not do heavy lifting on here.
ewenink@applejack.science.ru.nl
There is a separate page on GPU specs and benchmarks.
The fastest GPUs are also the busiest ones, so it may be that your job is done sooner on the slower GPUs.
Check htop of CPU core activity.
Make sure to check the "NI" column: the "niceness" column.
Everything that is computationally intensive and/or runs for longer than 2 hours has to be run with "nice -n 19". This is as nice as possible.
How do we check GPU usage? A: nvidia-smi
Storing data
There are three places where you can story data:
See wiki -> important locations
- home directory
- The same home directory is shared over machines it is a network disk.
- This can be handy, but this make I/O very slow!
- Limited memory: 10Gb
- Each machine has their own physical disk
scratch - /vol/tensusers
Avoid storing a lot of small items:
- ls becomes very slow
- moving around data becomes extremely slow
- This means administrators also cannot process
Moving a single 1TB file is more efficient than a billion 1kb files, due to overhead.
Python modules
Check python wiki page.
Initially Wessel Stoop managed all Python packages, but this was laborous and led to several issues (e.g. breaking scripts of others due to version differences).
It's now possible to make your own python environment.
The lamachine distribution already contains most machine learning packages as well (e.g. Tensorflow). If something is still not in there, you can still use virtualenv etc.