Cluster Computing

Access to the cluster is supported via SSH and via Open OnDemand. Open OnDemand makes is easy to access (through a web browser) scientific software for data visualization, simulations and modeling.

NCShare OnDemand

Currently, NCShare OnDemand supports running a virtual Linux desktop, a pre-installed Jupyter Lab Apptainer, and an RStudio Apptainer. Experienced users may develop their own containers which also can be accessed through OnDemand (see Cluster Software section for more info).

Current session limits (subject to change as we progress through deployment):

Max wall time: 24 hours
CPUs: 40
RAM: 500 GB

NCShare SSH Access

Users who would like to directly interact with the NCShare Cluster Computing environment through the Slurm Workload Manager may do so by enabling SSH key authentication from their workstation.

Cluster Storage

The NCShare cluster provides several file systems for cluster users. General-purpose shared storage is hosted on Isilon network-attached storage (NAS) arrays connected over 40 Gb/s or 10 Gb/s networks. Slurm jobs may also access node-local temporary storage for the duration of a job.

Warning

Sensitive data is not permitted on cluster storage.

Path	Size	Description
`/hpc/home/<ncshare_uid>`	50 GB	Use for personal scripts and environment setup. When an NCShare account expires, this home directory is automatically removed from the cluster.
`/work/<ncshare_uid>`	100 TB	Unpartitioned, shared network storage for active working data. Files older than 75 days are purged automatically.
`/data/projectname`	100 TB	Available by request only through institution contacts. Projects that need to share data can request a limited-duration allocation of space.
`/scratch`	node-specific	Node-local temporary storage for high-performance I/O available for the duration of a Slurm job.

Warning

NCShare storage is not intended for long-term use, therefore, users should regularly transfer out files they are not using. Access to NCShare must be renewed annually. All files are automatically purged for expired users.

Warning

Archiving a directory (e.g., with tar) does not protect its contents from the purge once extracted. Archive tools preserve each file's original modification time (mtime) by default, so extracted files retain their old mtime rather than getting the extraction time. Since the purge on /work is based on each file's mtime, extracted files older than 75 days are immediately eligible for purging. To reset the mtime on extraction, use tar -xmf archive.tar (the -m/--touch flag sets the mtime to the extraction time instead of restoring the archived one).

Each NCShare compute node has node-local /scratch space backed by NVMe SSDs for high-performance temporary I/O during a Slurm job. Each job receives its own mounted /scratch directory, but the underlying capacity is shared with other jobs on the same node. Data on /scratch should be copied elsewhere before the job ends.

NCShare Cluster Partitions

Slurm partitions are separate queues that divide up a cluster's nodes based on specific attributes. Each partition has its own constraints, which control which jobs can run in it.

Partition	Description
common	Jobs that will run CPU only.
interactive	Short term CPU jobs with a max walltime of 1 hour. Higher priority than `common`.
gpu	Jobs that will run on H200 GPU nodes through general access. Jobs are pre-empted (cancel and requeue) for higher priority jobs. Contact your institutional representative to gain access.
interactive-gpu	Short term GPU jobs with a max walltime of 1 hour. Contact your institutional representative to gain access. Higher priority than `gpu`.
gpu-hp	Jobs that will run on H200 GPU nodes with higher priority. Restricted to partner institutions. Each institution is allocated a monthly pool of GPU-hours. Contact your institutional representative to gain access.
workshop	Temporary partition for workshop and training events. Access is by request only.

Guidance on GPU usage can be found in the GPU documentation.

Note

If a partition is not specified, the default partition is the common partition.

Sample interactive session

The following commands demonstrate requesting an interactive shell on the common partition through SLURM which is then used to execute a command within the Apptainer image Apptainer.def. Using Apptainer in this way lets you interact with the environment like it is a virtual machine.

$ srun -p common --pty bash -i
$ apptainer shell Apptainer.sif

Apptainer> cat /etc/os-release

PRETTY_NAME="Ubuntu 24.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.3 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

Sample batch job script

Users may submit batch jobs through the Slurm scheduler using a batch script such as the one below.

jobscript.sh

#!/bin/bash
#SBATCH -J job_name          # Job name
#SBATCH -p common            # Partition name

cd $SLURM_SUBMIT_DIR
echo "Running on host `hostname`"

If more control is needed over the job parameters, additional SBATCH directives may be added to the script as shown below.

jobscript.sh

#!/bin/bash
#SBATCH -J job_name          # Job name
#SBATCH -p common            # Partition name
#SBATCH -N 2                 # Total no. of nodes
#SBATCH --ntasks-per-node 64 # Tasks per node
#SBATCH --mem=500G           # Memory per node
#SBATCH -t 02:00:00          # Time limit hrs:min:sec

cd $SLURM_SUBMIT_DIR
srun -n $SLURM_NTASKS aims.x > aims.out 2> aims.err

Instead of specifying -N and --ntasks-per-node, users may specify -n to set the total number of tasks for the job. Slurm will then allocate the required number of nodes based on the tasks requested and the default tasks per node for the partition.

For running GPU Slurm batch jobs, please refer to the GPU documentation.

To submit the batch job script, use the sbatch command as shown below.

$ sbatch jobscript.sh

You can monitor the status of your job using the squeue command:

$ squeue -u $USER

The sample job script below (slurm_apptainer.sh) illustrates how to use Apptainer as part of a batch session. All defaults for number of nodes, time etc. are used here.

slurm_apptainer.sh

#!/bin/bash
#SBATCH --job-name=slurm_apptainer
#SBATCH --partition=common

cd $SLURM_SUBMIT_DIR
apptainer exec /opt/apps/containers/users/simple_image.sif cat /etc/os-release

Submitting the batch script:

$ sbatch slurm_apptainer.sh

Submitted batch job 459436

The output of the job can be found in the file slurm-<jobid>.out as shown below:

$ cat slurm-459436.out

PRETTY_NAME="Ubuntu 24.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.3 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

For interactive and batch jobs, the following user limits are in place on the CPU-only partitions (common and interactive),

	common	interactive
Max wall-time	7 days	1 hour
Concurrent jobs per user	16	1

A maximum of 500 GB memory per node can be requested for the CPU-only partitions.