Cluster Architecture

The NCShare Compute Cluster is a small, high-throughput Slurm cluster that supports CPU simulations, data analysis, and GPU-accelerated machine learning and AI workloads. It consists of two homogeneous groups of compute nodes: AMD EPYC CPU nodes and Intel Xeon nodes with NVIDIA H200 GPUs.

This page is the reference for NCShare hardware, CPU and GPU architectures, node configurations, and the network. Hardware values represent a point-in-time inventory and may change as nodes enter or leave service. Use the commands in Querying node information to inspect the current scheduler state.

System overview

Component	Specification
Login nodes	`login-01` (SSH) and `ood-01` (Open OnDemand)
Compute nodes	12 (8 CPU nodes, 4 GPU nodes)
Physical cores	896 (1,792 Slurm CPUs with simultaneous multithreading)
GPUs	32 × NVIDIA H200
Aggregate RAM	~12 TB
Shared storage	NFS v4.2 network-attached storage (see Storage)
CPU architectures	AMD EPYC 7543 (Zen 3, Milan), Intel Xeon Platinum 8568Y+ (Emerald Rapids)
GPU architecture	NVIDIA Hopper
Interconnect	Bonded Ethernet, 2 × 10 Gb/s on CPU nodes and 2 × 25 Gb/s on GPU nodes (see Interconnect)
Operating system	Ubuntu 24.04 LTS
Scheduler	Slurm

GPUs by type

GPU	Architecture	Slurm GRES type	VRAM (GB)	Count	Partitions
H200	Hopper	`h200`	141	32	`gpu`, `interactive-gpu`, `gpu-hp`

GPUs are requested by their GRES type,

#SBATCH --gres=gpu:h200:<num_gpus>

E.g.,

#SBATCH --gres=gpu:h200:2

Access to the GPU partitions must be requested through your institutional representative. See the GPU Guide for access, QoS, and usage details.

CPU architectures

NCShare has exactly two CPU types, and each one maps to a distinct set of partitions, therefore, the partition you submit to determines the CPU architecture your job runs on.

Node type	CPU	Microarchitecture	Sockets × cores × threads	Partitions
CPU node	2 × AMD EPYC 7543	`zen3` (Milan)	2 × 32 × 2 = 128	`common`, `interactive`
GPU node	2 × Intel Xeon Platinum 8568Y+	`emeraldrapids`	2 × 48 × 2 = 192	`gpu`, `interactive-gpu`, `gpu-hp`, `common-lm`

Warning

The common-lm partition runs CPU-only jobs on the GPU nodes, which use a different CPU architecture from common. A job submitted with --partition common,common-lm can start on either type, so build with a portable target (see Compiling for a specific CPU architecture) or restrict the job to a single partition.

Simultaneous multithreading (hyper-threading) is enabled on all compute nodes, so each Slurm CPU is a hardware thread and the CPU counts in the tables below are twice the physical core count. A common node exposes 128 Slurm CPUs backed by 64 physical cores. If your application does not benefit from hyper-threading, disable it for a job with,

#SBATCH --hint=nomultithread

Each node has two sockets, and each socket is a separate NUMA domain. On the AMD nodes, the 32 cores in each socket are further divided into eight core complexes (CCDs) of four cores, each with its own 32 MB L3 cache. Codes that are sensitive to memory bandwidth or cache locality generally run best with MPI ranks or OpenMP threads pinned to cores (e.g., srun --cpu-bind=cores or OMP_PROC_BIND=close).

See Cluster Computing for general job submission documentation and partition limits.

Node configurations

Partition resource totals

Partition	Nodes	Slurm CPUs	RAM (GB)	GPUs	Max walltime
`common`	8	1,024	4,000	-	7 days
`interactive`	8	1,024	4,000	-	1 hour
`common-lm`	4	768	8,059	-	2 days
`gpu`	4	768	8,059	32	2 days
`interactive-gpu`	4	768	8,059	32	1 hour
`gpu-hp`	4	768	8,059	32	7 days

common and interactive share the same eight CPU nodes, and common-lm and the three GPU partitions share the same four GPU nodes. common-lm cannot allocate GPUs; it uses the CPUs and memory left idle by GPU jobs, and its jobs can be preempted by any GPU partition. Two additional partitions, osg and osg-gpu, are restricted to the OSG (Open Science Grid) account and run on the same hardware. Slurm CPUs are schedulable hardware threads, and RAM is the memory Slurm can allocate, which is slightly less than the installed memory. Query current values with scontrol show partition <partition_name>.

Note

Partitions on the GPU nodes enforce a limit of 10 GB of memory per allocated CPU (MaxMemPerCPU=10240). If a job requests more memory than its CPU count allows, Slurm raises the CPU count to match. E.g., --cpus-per-task=4 --mem=200G is allocated 20 CPUs.

`common` and `interactive`

CPU architecture	CPU model	Total CPUs	Physical CPUs	RAM (GB)	Local scratch	nodes	nodelist
zen3	EPYC 7543	128	64	500	~740 GB NVMe	8	`compute-[05-12]`

GPU nodes: `gpu`, `interactive-gpu`, `gpu-hp`, and `common-lm`

CPU architecture	CPU model	Total CPUs	Physical CPUs	RAM (GB)	GPU	Local scratch	nodes	nodelist
emeraldrapids	8568Y+	192	96	2014	8 x H200	~42 TB NVMe	4	`compute-gpu-[01-04]`

H200 nodes

Each GPU node is an 8-GPU NVIDIA HGX H200 system. Each H200 GPU provides,

141 GB of HBM3e memory.
Up to 4.8 TB/s of memory bandwidth.
18 fourth-generation NVLink links connected through NVSwitch, giving 900 GB/s of bidirectional bandwidth between any pair of GPUs in the same node.
PCIe Gen5 connectivity to the host CPUs.
Compute capability 9.0 (sm_90).

Because GPU-to-GPU bandwidth inside a node is far higher than the network between nodes (see Interconnect), multi-GPU training and inference should fit on a single node whenever possible. A single job can use up to 8 GPUs.

See the GPU Guide for access and job submission details.

Querying node information

To list nodes in a partition with their CPU count and memory (in MB), run,

sinfo -p <partition_name> -N -o "%N %c %m %G"

To obtain detailed information about a specific node, run,

scontrol show node <node_name>

To inspect the hardware directly, run a short job on the node type of interest,

srun -p interactive -t 00:05:00 bash -c "lscpu; free -h"
srun -p interactive-gpu --gres=gpu:h200:1 -t 00:05:00 nvidia-smi

Storage

/hpc/home, /work, and /opt/apps are mounted on every compute node over NFS v4.2 from a central TrueNAS server, so their performance is bounded by the node's Ethernet link and shared with all other users. /scratch is node-local NVMe storage: roughly 740 GB per CPU node and 42 TB per GPU node. It is the fastest place for temporary, I/O-heavy files during a job, but its contents are not visible from other nodes. /tmp resides on the same local volume as /scratch.

Quotas, purge policies, and recommended usage are described in Cluster Storage.

Interconnect

All NCShare nodes are connected through Ethernet. Each CPU node has two 10 Gb/s links and each GPU node has two 25 Gb/s links, aggregated with LACP (IEEE 802.3ad) for 20 Gb/s and 50 Gb/s respectively. There is no InfiniBand fabric in service, so MPI, NCCL, and storage traffic all share the same TCP/IP network.

Link aggregation balances traffic per connection, so a single stream between two nodes is limited to the speed of one member link (10 Gb/s on CPU nodes, 25 Gb/s on GPU nodes). In practice this means,

Tightly coupled multi-node MPI jobs are latency-bound and scale poorly beyond a few nodes. A single 64-core common node or 96-core GPU node is often the most efficient choice. See Running a Multi-Node MPI Job on NCShare for a working multi-node example.
Multi-node GPU jobs communicate over TCP sockets rather than GPUDirect RDMA, which is roughly an order of magnitude slower than NVLink within a node.
Jobs that read heavily from /work benefit from copying input data to /scratch at the start of the job.

Compiling for a specific CPU architecture

Applications compiled with architecture-specific optimizations can run significantly faster than generic builds. The system compiler on NCShare is GCC 13.3 (gcc, g++, gfortran), and Open MPI 4.1.6 provides the mpicc, mpicxx, and mpif90 wrappers. As environment modules are not used on NCShare, other compilers and libraries can be installed in your home directory or packaged in a container (see Cluster Software).

Generation	GCC flag	Intel compiler flag	Notable instruction sets	Runs on
`zen3`	`-march=znver3`	`-march=core-avx2`	AVX2	CPU nodes
`emeraldrapids`	`-march=emeraldrapids`	`-xEMERALDRAPIDS`	AVX-512, AVX-512 FP16/BF16, VNNI, AMX	GPU nodes
portable	`-march=x86-64-v3`	`-march=core-avx2`	AVX2	All compute nodes

The AMD EPYC 7543 does not support AVX-512, so a binary built with -march=emeraldrapids, -march=x86-64-v4, or -march=native on a GPU node fails with an illegal-instruction error on common and interactive nodes. Instruction sets are cumulative, so the AVX2 baseline -march=x86-64-v3 runs on every compute node at the cost of not using AVX-512 or AMX on the Intel nodes. Use it for jobs that may land on either node type. Note that Intel -x flags generate code that only runs on genuine Intel CPUs.

For example, a typical optimized build for the common nodes with automake,

CFLAGS="-O3 -march=znver3"
CXXFLAGS="-O3 -march=znver3"
FCFLAGS="-O3 -march=znver3"

and its CMake equivalent,

set(CMAKE_C_FLAGS "-O3 -march=znver3")
set(CMAKE_CXX_FLAGS "-O3 -march=znver3")
set(CMAKE_Fortran_FLAGS "-O3 -march=znver3")

Compile on the same node type you intend to run on. The interactive partition is convenient for building common binaries, and interactive-gpu for GPU-node binaries.

Compiling for a specific GPU architecture

All GPUs on NCShare are H200s, so CUDA code needs a single target.

GPU	Architecture	Compute capability	`nvcc` flag
H200	Hopper	9.0	`-arch=sm_90`

With CMake,

set(CMAKE_CUDA_ARCHITECTURES "90")

The CUDA toolkit (nvcc) is not installed on the host system. Build CUDA code inside a container based on an NVIDIA NGC CUDA image, or install the toolkit into a Conda environment (e.g., conda install -c nvidia cuda-toolkit). The CUDA runtime in a container or environment must not be newer than the version the driver supports, shown in the header of nvidia-smi. Containers must be run with apptainer exec --nv to access the GPUs (see Cluster Software).

Example Slurm job script

To summarize the details in this guide, the following example job submission script requests 2 common nodes with 64 tasks per node (one per physical core, with hyper-threading disabled) and 500 GB of memory per node. The job runs an MPI executable built with Open MPI and -march=znver3 across 128 tasks.

submit.sh

#!/bin/bash
#SBATCH -J jobname            # Job name
#SBATCH -p common             # Partition name
#SBATCH -N 2                  # Total no. of nodes
#SBATCH --ntasks-per-node=64  # Tasks per node
#SBATCH --mem=500G            # Memory per node
#SBATCH -t 00:05:00           # Walltime limit (hh:mm:ss)
#SBATCH --hint=nomultithread  # Disable hyper-threading

# Execution
cd $SLURM_SUBMIT_DIR
mpirun -n $SLURM_NTASKS ./test_mpi.x > test_mpi.out 2> test_mpi.err