87. How do I do parallel rsync?
To do parallel rsync initiating from sender (pushing data), use the command parsync - a parallel rsync wrapper for large data transfers. Parsync has been packaged in the UUPEL repository.
parsync --maxload=16 --NP=12 --startdir=/tank/MOL-EXTBMC EXTBMC root@bmc-pcfs3:/tank/bmc-pcfs2.bmc.uu.se/tank/MOL-EXTBMC
In this example the directory EXTBMC in the directory /tank/MOL-EXTBMC will be synced to the host bmc-pcfs3 into the destination directory /tank/bmc-pcfs2.bmc.uu.se/tank/MOL-EXTBMC. To make parallel rsync initiating from the receiver (pulling data) is harder. We have made a system where we first
- Do a initial rsync deleting all files and directories to be deleted.
- Sync all directories and no files.
- Parallel sync of all directories syncing the files in each directory.
- Do a final rsync.
U=username H=host SRC=/source/directory DST=/destination/directory # Delete everything that is supposed to be deleted. rsync -r --delete --existing --ignore-existing $U@$H:$SRC/. $DST/. # Sync directories but no files. rsync -a -f"+ */" -f"- *" $U@$H:$SRC/. $DST/. # For every directory, sync the files in that directory. Run 10 in parallel. find . -type d -print0 | xargs -P 10 -I '{}' -0 rsync -vlptgoxSH $U@$H:$SRC/'{}'/\* $DST/'{}'/. # Run a final sync with delete to make sure everything is ok. rsync -aSH --delete $U@$H:$SRC/. $DST/.