The PCFS storage provided by BMC-IT is normally accessed via SMB. However the Uppsala University perimeter firewall in front of UpUnet (Fortigate) is blocking incoming SMB. The Uppsala University HPC center UPPMAX (Uppsala Multidisciplinary Center for Advanced Computational Science) has its own Internet connection via SUNET so SMB access from SUNET is blocked in the firewall.
For those cases Rrsync (restricted Rsync) can be set up as well.
It could for example look like this on the server:
[root@bmc-pcfs4 ~]# tail -5 /etc/ssh/sshd_config Ciphers +arcfour AllowUsers root jny25782 Match User jny25782 ForceCommand /usr/local/bin/rrsync.data.sh Match all [root@bmc-pcfs4 ~]# cat /usr/local/bin/rrsync.data.sh #!/bin/bash exec /usr/local/bin/rrsync /data [root@bmc-pcfs4 ~]#
To access it use Rsync as normally. However, Rsync now is using the /data directory above as base directory. ALl the shares are mounted under the /data directory.
To access the data with Rsync may look like this. Here I tried to download the contents of a share I do not have access to, so just igore the error and use rsync as normal for transfer data.
gforce:~ jerker$ rsync -avx --progress jny25782@IMB-GenomicsKLT2.files.uu.se:IMB-GenomicsKLT2/. tmp/. email@example.com's password: receiving file list ... rsync: opendir "/data/IMB-GenomicsKLT2/GenomicsKLT2" failed: Permission denied (13) 3 files to consider sent 20 bytes received 222 bytes 69.14 bytes/sec total size is 0 speedup is 0.00 rsync error: some files could not be transferred (code 23) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-52/rsync/main.c(1404) [generator=2.6.9] gforce:~ jerker$ ls -la tmp/ total 0 drwxr-xr-x 4 jerker staff 128 Mar 30 2017 . drwxr-xr-x+ 225 jerker staff 7200 Dec 5 11:27 .. drwxr-xr-x 2 jerker staff 64 Dec 5 11:02 .snapshots drwxrwx--- 2 jerker staff 64 Mar 8 2017 GenomicsKLT2 gforce:~ jerker$ ls -la tmp/GenomicsKLT2/ total 0 drwxrwx--- 2 jerker staff 64 Mar 8 2017 . drwxr-xr-x 4 jerker staff 128 Mar 30 2017 .. gforce~ jerker$
If you want to use your public SSH-key instead of your password for authentication then send the SSH key to the administrator. Kerberos single-sign-on is currently not supppoerted.
- Connect to Uppmax with SSH, in this example I am connecting to rackham.uppmax.uu.se. Start a screen so that you can keep
your processes running even when your SSH-client disconnect to UPPMAX when you for example shutdown your computer.
[jerker@rackham3 ~]$ screen
- Then connect to that server with rsync. In the following example we are trying to reach the share that also can be reached as smb://IMB-GenomicsKLT2.files.uu.se/IMB-GenomicsKLT2/
[jerker@rackham3 ~]$ rsync -avx --progress firstname.lastname@example.org:IMB-GenomicsKLT2/. tmp/.
In this example we are using another port because the normal port 22 was at that point still blocked in the firewall between UPPMAX and UpUnet. Also, compression is turned off, faster ciphers are being used and ssh escape characters are turned off.
[jerker@rackham3 ~]$ rsync -e 'ssh -e none -p 2222 -o Compression=no -c arcfour,aes128-ctr' -avx --progress jny25782@IMB-GenomicsKLT2.files.uu.se:IMB-GenomicsKLT2/. tmp/. The authenticity of host '[imb-genomicsklt2.files.uu.se]:2222 ([22.214.171.124]:2222)' can't be established. ECDSA key fingerprint is SHA256:zUs82pMdiZzQoqaR86iGFp2A/6LzHAy6WBbKC+46sSo. ECDSA key fingerprint is MD5:12:c7:98:f6:65:3e:39:0d:df:59:dc:a7:f8:96:2a:f4. Are you sure you want to continue connecting (yes/no)? yes You have to type yes here Warning: Permanently added '[imb-genomicsklt2.files.uu.se]:2222,[126.96.36.199]:2222' (ECDSA) to the list of known hosts. email@example.com's password: Enter your password here receiving incremental file list rsync: opendir "/data/IMB-GenomicsKLT2/GenomicsKLT2" failed: Permission denied (13) .snapshots/ sent 16 bytes received 226 bytes 19.36 bytes/sec total size is 0 speedup is 0.00 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1518) [generator=3.0.9] [jerker@rackham3 ~]$Here is an example when the contents of the directory /proj/mystuff/directory/to/upload/. on UPPMAX will be synced to the directory TLA-ShareName.files.uu.se:TLA-ShareName/ShareName/directory.to.upload/. on PCFS.
This destination can also be reached over SMB as as the smb://TLA-ShareName.files.uu.se/TLA-ShareName/ShareName/directory.to.upload/ or \\TLA-ShareName.files.uu.se\TLA-ShareName\ShareName\directory.to.upload\
The flag --delete will erase all files from the destination that do not exist in the source.x
[jerker@rackham3 ~]$ rsync -e 'ssh -e none -o Compression=no -c arcfour,aes128-ctr' --delete -avx --progress /proj/mystuff/directory/to/upload/. jny25782@TLA-ShareName.files.uu.se:TLA-ShareName/ShareName/directory.to.upload/.
By using the notation above (the /. in the end) the rsync can be run multiple times to update/sync modified files again.
- Have fun! Disconnect from the screen with Ctrl-A Ctrl-D. Attach to the screen again with the command screen -x.