Clusters¶
The Oden Institute has a number of small clusters that are owned by centers and are used only by those affiliated with the respective center.
CSEM Peano¶
Warning
Peano will be going offline on June 30th. It has outlived its usefullness
Peano is a 40 node compute cluster made up of stampede1 nodes acquired from TACC. Each node contains 2 x 8 core E5-2680 (Sandy Bridge) processors with 32GB RAM, a single 250GB drive, and Infiniband FDR interconnect (Mellanox ConnectX-3).
The login node is a Dell PE R515 with 2 x hex core Opteron processors, 32GB RAM, dual gigE nics, Inifiniband FDR (Mellanox ConnectX-3 Pro), with an attached MD1000 storage array. Storage for home directories is 3.6TB in a RAID 5 configuration. An additional scratch area, ‘/scratch/’, is available and is RAID 5 group with 16TB of available storage.
The array was configured using OpenHPC (http://openhpc.community/). The queuing engine is Slurm (https://slurm.schedmd.com/) and provisioning is handled by Warewulf (http://warewulf.lbl.gov). A brief howto is provided below and is also contained in the MOTD when logging in:
To run an interactive shell, try:
srun -p normal -t 0:30:00 -n 32 --pty /bin/bash -l
For an example slurm job script for MPI, see:
/opt/ohpc/pub/examples/slurm/job.mpi
Further information:
This cluster is only available to students in the CSEM program.
SSH access is only from campus or UT’s VPN.
Home directories are not mounted.
Also note, OpenHPC makes available a pre-packaged builds for a variety of MPI families such as OpenMPI, MPICH, and MVAPICH. Only the GNU family of pre-packaged builds have been installed. Available Intel built MPI families in the module system have not been tested, use at your risk.
To request a user account to access Peano or for any help, please submit a help request to RT.
Note
Home directories and scratch areas on peano are not backed up.
CRIOS sverdrup¶
Note
Sverdrup and its associated storage node (sverdrup-nas) both went under upgrades beginning late March 2024.
System information:
OpenHPC cluster running Rocky Linux 9.3 (https://openhpc.community/)
$HOME is an NFS file system -> /home (2.0 TB)
/scratch is 105TB
/scratch2 is 125TB
/opt/apps/ is an NFS file system -> 100GB
Queuing system is SLURM 22.05. One queue available –> normal (35 Intel Omnipath nodes -> 28 cores/node, 64 GB/node, 980 cores)
Node composition:
Dual-socket 14-core Intel(R) Xeon(R) CPU E5-2695 v3 @ 2.30GHz (Haswell)
64 GB of RAM per node
Omni-Path high performance communications HBA (100Gbs)
10Gbs networking
Wish list¶
Requests by research staff with the CRIOS group.
CRIOS cluster is getting a major reboot from March 25 - April 5 and will be unreachable at that time. Below is a request list of software and features Patrick can pass to Oden RT, in no particular order:
git >= 2.31
Allow all users to ssh into compute nodes used interactively
Update GNU compiler collection >= 12.2, but still need all the current GNU collection
tmux >= 2.7
Keep z-shell as a shell option
Keep the feature to use Jupyter Notebook/lab connection from the
compute node to the local browser
Gbd on all compute nodes
Ability to use Vim and compile codes on compute nodes (I wonder if
that means having the same env as the login node.)
Singularity (open source container platform like Docker, TACC has it)
Okular is no supported, use evince or xpdf.
The following base packages provided with Rocky Linux have been installed:
git-2.39
zsh-5.8
tmux- 3.2a
GNU 12.2 and GNU 13.1, OpenHPC builds as modules
Base OS for gdb installed, 10.2
apptainer-1.3 (formerly singularity)
Module¶
Lmod modules has been installed. Use the module commands to view available modules.
Intel compilers¶
Intel OneAPI compilers have been installed and are available via the module command.
Apptainer¶
Apptainer, formerly Singularity, has been installed on all the nodes using packages provided by Rocky Linux. Version is 1.3. There is no module needed for loading apptainer.
Note
Apptainer is not installed on the login node.
Create a job script, job.apptainer
#!/bin/bash
#SBATCH -J test # Job name
#SBATCH -o job.%j.out # Name of stdout output file (%j expands to jobId)
#SBATCH -N 2 # Total number of nodes requested
#SBATCH -n 16 # Total number of mpi tasks requested
#SBATCH -t 01:30:00 # Run time (hh:mm:ss) - 1.5 hours
# Launch apptainer job
/usr/bin/apptainer run docker://busybox uname -r
Yields
[stew@sverdrup]$ sbatch job.apptainer
Check the status of the log file
INFO: Using cached SIF image
5.14.0-362.24.1.el9_3.x86_64
Compute node access¶
A request was made to allow ssh access to the nodes without having to go through Slurm.
There is an advantage to use srun when reserving nodes. The environment is exported properly to the node(s) for submission. This is not the case for regular ssh sessions to nodes.
Warning
It is not recommended ssh’ing into the nodes unless you have a job running there you own and have queued via Slurm. It’s strongly recommended you use srun to reserve a node over ssh’ing into the node. The underlying queueing engine is not aware of ssh sessions and the node could be allocated by someone else or a job could be queued to a node that someone has ssh’d into. This could cause a conflict resources on the node.
Jupyter Notebook¶
Using a python virtual environment, installed jupyter-notbook into the home directory. Was able to connect to jupyter server over ssh to both the login node and to a compute node using a proxy jump. This should be the same as before.