Frequently Asked Questions

« Previous
How do I send a large file to someone outside (or inside) the university?
Next »
How do I use the UUPEL repository?

windows ubuntu macos xibo network zenworks android storage


87. How do I do parallel rsync?

See also: How do I use the UUPEL repository?
See also: What is Rrsync (restricted rsync)? How do I access PCFS storage over rsync?
To do parallel rsync initiating from sender (pushing data), use the command parsync - a parallel rsync wrapper for large data transfers. Parsync has been packaged in the UUPEL repository.

parsync --maxload=16 --NP=12 --startdir=/tank/MOL-EXTBMC EXTBMC root@bmc-pcfs3:/tank/bmc-pcfs2.bmc.uu.se/tank/MOL-EXTBMC

In this example the directory EXTBMC in the directory /tank/MOL-EXTBMC will be synced to the host bmc-pcfs3 into the destination directory /tank/bmc-pcfs2.bmc.uu.se/tank/MOL-EXTBMC. To make parallel rsync initiating from the receiver (pulling data) is harder. We have made a system where we first

  1. Do a initial rsync deleting all files and directories to be deleted.
  2. Sync all directories and no files.
  3. Parallel sync of all directories syncing the files in each directory.
  4. Do a final rsync.

U=username H=host SRC=/source/directory DST=/destination/directory # Delete everything that is supposed to be deleted. rsync -r --delete --existing --ignore-existing $U@$H:$SRC/. $DST/. # Sync directories but no files. rsync -a -f"+ */" -f"- *" $U@$H:$SRC/. $DST/. # For every directory, sync the files in that directory. Run 10 in parallel. find . -type d -print0 | xargs -P 10 -I '{}' -0 rsync -vlptgoxSH $U@$H:$SRC/'{}'/\* $DST/'{}'/. # Run a final sync with delete to make sure everything is ok. rsync -aSH --delete $U@$H:$SRC/. $DST/.



 

This entry parsync was last modified 2018-10-15

   

This documentation is covered by GNU Free Documentation License. 13 ms