Moving Many Files Between Systems with rsync

rsync is a program used to synchronize large file sets between servers. It uses an efficient protocol to only send files (or portions of files) that are different, reducing data transfer.

Setting up the Target (Receiver)

On the machine which will receive the files, create the /apps/cms/rsyncd.conf

# /apps/cms/rsyncd.conf

use chroot = no
max connections = 8
pid file = /apps/cms/var/run/rsyncd.pid
lock file = /apps/cms/var/run/rsyncd.lock
log file = /apps/cms/var/run/rsyncd.log

[datastore]
  path = /apps/cms/var/data/aem64-publish/crx-quickstart/repository/datastore
  comment = AEM blob datastore
  read only = false
  auth users = cms
  secrets file = /apps/cms/rsyncd.secrets

Then create the secrets file /apps/cms/rsyncd.secrets with this shell command

echo "cms:$(openssl rand -hex 20)" > /apps/cms/rsyncd.secrets
chmod 0640 /apps/cms/rsyncd.secrets

This will create a file with the password we need later, similar to this one

cms:cd72f9e66ba2ea725245ff677cc5d73af92c45a0

Start the rsync daemon:

rsync --daemon --config ~/rsyncd.conf --port 4500 --no-detach

We run with --no-detach to leave the process in the foreground, otherwise its too easy to leave it running.

On the Source (Sender)

$ cd /apps/cms/var/data/aem64-publish/crx-quickstart/repository
$ rsync -rv ./datastore/ rsync://cms@cmsa-pprd-03.db.vt.edu:4500/datastore/
Password:

rsync will ask for the password we generated earlier, which lives in /apps/cms/rsyncd.secrets on the receiver.