Skip to content

Data Transfer

The NCShare cluster has several shared file systems available for all users of the cluster. General partitions are on Isilon, 40Gbps or 10Gbps network attached storage arrays. For more detailed information on storage volumes, please see the Cluster Storage.

Transferring data to and from NCShare

You can copy data to and from the NCShare volumes using SCP and rsync. For individual files, we recommend using SCP. For multiple files, we recommend rsync.

SCP

You can use secure copy (SCP) to transfer individual files to and from NCShare.

  • If you have an SSH key setup, you can specify which key using the -i /path/to/ssh/key or by editing your SSH config file, usually located in ~/.ssh See below for the SCP syntax and an example of copying files to and from NCShare.
SCP (Secure Copy) a single file 
scp source destination

$scp <localpath.file> <comanageid>@login.ncshare.org:<ncsharepath>

$scp <comanageid>@login.ncshare.org:<ncsharepath> <localpath>

# Note use of ssh keys to simplfy login
# Sample command and output pushing a file from my workstation to my group directory:

kk338@CDSS-5630 ~ % scp jobs.txt kkilroy1@login.ncshare.org:/hpc/home/kkilroy1
Enter passphrase for key '/Users/kk338/.ssh/id_rsa': 
jobs.txt                                      100% 3847KB   8.4MB/s   00:00   

# Sample command and output pulling a file from my group directory to my local workstation (note the use of `.` to denote current working directory):

kk338@CDSS-5630 ~ % scp kkilroy1@login.ncshare.org:/hpc/home/kkilroy1/DailyUsage.xlsx .
Enter passphrase for key '/Users/kk338/.ssh/id_rsa': 
DailyUsage.xlsx                                               100%   40KB 361.1KB/s   00:00

rsync

While you can use scp -r to recursively copy all of the files in a directory, we recommend the use of rsync for a large number of files.

  • If you have multiple SSH keys, or haven't configured your SSH confi file, you can use -e "ssh -i ~/.ssh/nameOfKey to specify which key to use
  • If you need to copy multiple files, make sure to include the -rP flags after the ssh command to copy all files within the local directory
RSYNC for a larghe number of files 
# -r syncs recursively 
# -P updates the modify time, useful for /work purge script analysis
rsync –rP source destination

$rsync –rP /local/path/dir1/ <comanageid>@login.ncshare.org:<ncsharepath>

$rsync –rP <comanageid>@login.ncshare.org:<ncsharepath> /local/path/dir1

(base) rm145@localmac % rsync -e "ssh -i ~/.ssh/ncs_key" -rP ~/testFolder emclaurin1@login.ncshare.org:/hpc/home/emclaurin1
Enter passphrase for key '/Users/rm145/.ssh/ncs_key': 
building file list ... 
4 files to consider
testFolder/
testFolder/tf1.txt
        0 100%    0.00kB/s    0:00:00 (xfer#1, to-check=2/4)
testFolder/tf2.txt
        0 100%    0.00kB/s    0:00:00 (xfer#2, to-check=1/4)
testFolder/tf3.txt
        0 100%    0.00kB/s    0:00:00 (xfer#3, to-check=0/4)

sent 232 bytes  received 92 bytes  43.20 bytes/sec
total size is 0  speedup is 0.00