Skip to content

Storage and Data Transfer

The NCShare cluster provides several file systems for cluster users. General-purpose shared storage is hosted on a TrueNAS network-attached storage (NAS) server and mounted on all nodes over NFS, through bonded 10 Gb/s (CPU nodes) or 25 Gb/s (GPU nodes) Ethernet links. Slurm jobs may also access node-local temporary storage for the duration of a job.

Warning

Sensitive data is not permitted on cluster storage.

Path Size Description
/hpc/home/<ncshare_uid> 50 GB Use for personal scripts and environment setup. When an NCShare account expires, this home directory is automatically removed from the cluster.
/work/<ncshare_uid> 100 TB Unpartitioned, shared network storage for active working data. Files older than 75 days are purged automatically.
/data/projectname 100 TB Available by request only through institution contacts. Projects that need to share data can request a limited-duration allocation of space.
/scratch node-specific Node-local temporary storage for high-performance, low-latency I/O available for the duration of a Slurm job.

Warning

NCShare storage is not intended for long-term use, therefore, users should regularly transfer out files they are not using. Access to NCShare must be renewed annually. All files are automatically purged for expired users.

Warning

The purge on /work is driven by each file's modification time (mtime), and several common tools preserve an original timestamp rather than setting it to now. A file can therefore arrive already older than 75 days and be eligible for purging immediately.

  • tar restores archived timestamps on extraction, so archiving a directory does not protect its contents from the purge. Extract with tar -xmf archive.tar, where the -m/--touch flag sets the mtime to the extraction time instead of restoring the archived one.
  • wget sets a downloaded file's timestamp from the server. Download with wget --no-use-server-timestamps.
  • rsync -a, cp -p and scp -p carry the source timestamp across. Use rsync -a --no-times, or omit -p, when copying older data onto NCShare storage.

Each NCShare compute node has node-local /scratch space backed by NVMe SSDs for high-performance, low-latency temporary I/O during a Slurm job. Each job receives its own mounted /scratch directory, but the underlying capacity is shared with other jobs on the same node. Data on /scratch should be copied elsewhere before the job ends.

Transferring data

Copy data between your computer and NCShare with scp or rsync. Use scp for individual files and rsync for directories or many files. For occasional single files, you can also upload and download through OnDemand in Jupyter Lab or RStudio.

Bundle your files before transferring

tar your files before transferring. Transferring a large number of small files is much slower than transferring a single large file.

SCP

If you have more than one SSH key, specify which one with -i /path/to/ssh/key, or set it in your SSH config file, usually ~/.ssh/config.

SCP (Secure Copy) a single file

scp source destination

scp <localpath.file> <ncshare_uid>@login.ncshare.org:<ncsharepath>

scp <ncshare_uid>@login.ncshare.org:<ncsharepath> <localpath>

# Note use of ssh keys to simplify login
# Sample command and output pushing a file from my workstation to my home directory:

kk338@CDSS-5630 ~ % scp jobs.txt kkilroy1@login.ncshare.org:/hpc/home/kkilroy1
Enter passphrase for key '/Users/kk338/.ssh/id_rsa':
jobs.txt                                      100% 3847KB   8.4MB/s   00:00

# Sample command and output pulling a file from my home directory to my local workstation (note the use of `.` to denote current working directory):

kk338@CDSS-5630 ~ % scp kkilroy1@login.ncshare.org:/hpc/home/kkilroy1/DailyUsage.xlsx .
Enter passphrase for key '/Users/kk338/.ssh/id_rsa':
DailyUsage.xlsx                                               100%   40KB 361.1KB/s   00:00

rsync

While you can use scp -r to copy a directory recursively, we recommend rsync for a large number of files. If you have more than one SSH key, or haven't configured your SSH config file, specify the key with -e "ssh -i ~/.ssh/nameOfKey".

RSYNC for a large number of files

# -r syncs recursively
# -P shows progress and keeps partial files, so an interrupted transfer can resume
# Leaving out -t (or -a) gives copied files the current modification time,
# so files copied to /work aren't immediately eligible for the purge
rsync -rP source destination

rsync -rP /local/path/dir1/ <ncshare_uid>@login.ncshare.org:<ncsharepath>

rsync -rP <ncshare_uid>@login.ncshare.org:<ncsharepath> /local/path/dir1

(base) rm145@localmac % rsync -e "ssh -i ~/.ssh/ncs_key" -rP ~/testFolder emclaurin1@login.ncshare.org:/hpc/home/emclaurin1
Enter passphrase for key '/Users/rm145/.ssh/ncs_key':
building file list ...
4 files to consider
testFolder/
testFolder/tf1.txt
        0 100%    0.00kB/s    0:00:00 (xfer#1, to-check=2/4)
testFolder/tf2.txt
        0 100%    0.00kB/s    0:00:00 (xfer#2, to-check=1/4)
testFolder/tf3.txt
        0 100%    0.00kB/s    0:00:00 (xfer#3, to-check=0/4)

sent 232 bytes  received 92 bytes  43.20 bytes/sec
total size is 0  speedup is 0.00

Using /scratch in a job

Staging data on /scratch speeds up I/O-heavy jobs and takes pressure off the shared /work file system. CPU nodes have about 740 GB of local scratch and GPU nodes about 42 TB (see Architecture). Because the space is private to the job and temporary, the pattern is copy in, compute, copy out,

scratch-job.sh
#!/bin/bash
#SBATCH -J scratch-example
#SBATCH -p common
#SBATCH -c 8
#SBATCH --mem=32G
#SBATCH -t 04:00:00

IN=/work/$USER/project/input
OUT=/work/$USER/project/results
mkdir -p "$OUT" /scratch/in /scratch/out

# Copy results back even if the analysis fails partway through.
trap 'rsync -a /scratch/out/ "$OUT"/ || true' EXIT

# Stage in.
rsync -a "$IN"/ /scratch/in/

# Compute against node-local NVMe.
./my_analysis --input /scratch/in --output /scratch/out

Submit it with sbatch scratch-job.sh. The trap copies results back even if the analysis fails, but a job killed at its time limit may not get the chance to run it, so request enough walltime or checkpoint straight to $OUT. Staging pays off when a job reads the same data repeatedly or works with many small files. A job that streams a large file once gains little from it.