FPSYNC(1) | General Commands Manual | FPSYNC(1) |
fpsync
—
Synchronize directories in parallel using fpart and an
external tool
fpsync |
[-p ] [-n
jobs] [-w
wrks] [-m
tool] [-T
path] [-f
files] [-s
size] [-E ]
[-o toolopts]
[-O fpartopts]
[-S ] [-t
tmpdir] [-d
shdir] [-M
mailaddr] [-v ]
src_dir/ dst_url/ |
fpsync |
-l |
fpsync |
-r runid
[-R ] [OPTIONS...] |
fpsync |
-a runid |
fpsync |
-D runid |
The fpsync
tool synchronizes directories
in parallel using fpart(1) and rsync(1),
cpio(1) or tar(1). It computes subsets
of src_dir/ and spawns jobs to synchronize them to
dst_url/.
Synchronization jobs can be executed either locally or remotely
(using SSH workers, see option -w
) and are executed
on-the-fly while filesystem crawling goes on. This makes
fpsync
a good tool for migrating large
filesystems.
-t
tmpdirfpsync
temporary directory to
tmpdir. This directory remains local and does not
need to be shared amongst SSH workers when using the
-w
option. Default:
/tmp/fpsync-d
shdirfpsync
shared directory to
shdir. This option is mandatory when using SSH
workers and set by default to tmpdir when running
locally. The specified directory must be an absolute path ; it will be
used to handle communications with SSH hosts (sharing partitions and log
files) and, as a consequence, must be made available to all participating
hosts (e.g. through a r/w NFS mount), including the master one running
fpsync
.-M
mailaddrfpsync
.-v
-h
-m
tool-T
path-f
files-s
size-E
fpsync
works with lists of directories instead of
files. That mode may generate coarse-grained lists but enables
rsync(1) 's --delete
option by
default (
WARNING!!!
), making it a good candidate for a final (cleaning) pass after several
incremental passes using standard (file) mode. When option
-E
is specified twice, it enables 'aggressive'
mode which isolates erroneous directories and enables recursive
synchronization for them. This advanced mode can be useful to try to
overcome transcient errors such as Linux SMB client deferring
opendir
()
calls to support compound SMB requests.-o
toolopts--delete
). Default for rsync:
“-lptgoD -v --numeric-ids”. Empty for cpio, tar and
tarify.-O
fpartopts-S
fpsync
).-n
jobs-w
wrks-p
-r
.-l
-r
runid-v
) or prepare
mode (option -p
) and can be retrieved afterwards
using option -l
. Note that filesystem crawling is
skipped when resuming a previous run. As a consequence, options
-m
, -f
,
-s
, -E
,
-o
, -O
,
-S
, src_dir/, and
dst_url/ are ignored.-R
-r
, force
re-synchronizing run's all partitions instead of remaining ones only. That
mode can be useful to skip filesystem crawling when you have to replay a
final pass several times and you know directory structure has not changed
in the meantime (you may miss files if you use replay mode with a
standard, file-based, run).-a
runidfpsync
.-D
runidEach fpsync
run generates a unique
runid, which is displayed in verbose mode (see option
-v
) and within log files. You can use that
runid to resume a previous run (see option
-r
). fpsync
will then
restart synchronizing data from the parts that were being synchronized at
the time it stopped.
This unique feature gives the administrator the ability to stop
fpsync
and restart it later, without having to
restart the whole filesystem crawling and synchronization process. Note that
resuming is only possible when filesystem crawling step has finished.
During synchronization, you can press CTRL-C to interrupt the
process. The first CTRL-C prevents new synchronizations from being submitted
and the process will wait for current synchronizations to be finished before
exiting. If you press CTRL-C again, current synchronizations will be killed
and fpsync
will exit immediately. When using option
-E
to enable directory mode and rsync's
--delete
option, keep in mind that killing rsync
processes may lead to a situation where certain files have been updated and
others not deleted yet (because the deletion process is postponed using
rsync's --delete-after
option).
On certain systems, CTRL-T can be pressed to get the status of
current and remaining parts to be synchronized. This can also be achieved by
sending a SIGINFO to the fpsync
process.
Whether you use verbose mode or not, everything is logged within shdir/log/.
Here are some examples:
fpsync -n
4 /usr/src/ /var/src/
Synchronizes /usr/src/ to /var/src/ using 4 local jobs.
fpsync
-n 2 -w login@machine1 -w login@machine2 -d /mnt/fpsync /mnt/src/
/mnt/dst/
Synchronizes /mnt/src/ to
/mnt/dst/ using 2 concurrent jobs executed
remotely on 2 SSH workers (machine1 and machine2). The shared directory
is set to /mnt/fpsync and mounted on the machine
running fpsync
, as well as on machine1 and
machine2. The source directory (/mnt/src/) is
also available on those 3 machines, while the destination directory
(/mnt/dst/) is mounted on SSH workers only
(machine1 and machine2).
Parallelizing rsync(1) can make several options
not usable, such as --delete
. If your source
directory is live while fpsync
is running, you will
have to delete extra files from destination directory. This is traditionally
done by using a final -offline- rsync(1) pass that will
use this option, but you can also use fpsync
and
option -E
to perform the same task using several
workers.
fpsync
enqueues synchronization jobs on
disk, within the tmpdir/queue directory. Be careful
to host this queue on a filesystem that can handle fine-grained mtime
timestamps (i.e. with a sub-second precision) if you want the queue to be
processed in order when fpart(1) generates several jobs
per second. On FreeBSD, VFS(9)
timestamps' precision can be tuned using the 'vfs.timestamp_precision'
sysctl. See vfs_timestamp(9).
Contrary to rsync(1),
fpsync
enforces the final '/' on the source
directory. It means that directory
contents
are synchronized, not the source directory itself (i.e. you will not get a
subdirectory of the name of the source directory in the target directory
after synchronization).
Before starting filesystem crawling,
fpsync
changes its current working directory to
src_dir/ and generates partitions containing
relative
paths (all starting with './'). This is important to keep in mind when
modifying toolopts or fpartopts
dealing with file or directory paths.
Fpsync has been written by Ganaël LAPLANCHE and is available under the BSD license on http://contribs.martymac.org
No bug known (yet).
January 27, 2015 | Debian |