Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages

  
 
  

home | help
FPSYNC(1)		FreeBSD	General	Commands Manual		     FPSYNC(1)

NAME
     fpsync -- Synchronize directories in parallel using fpart + rsync or cpio

SYNOPSIS
     fpsync [-h] [-v] [-m tool]	[-n jobs] [-f files] [-s size] [-E] [-w	wrks]
	    [-d	shdir] [-t tmpdir] [-r jobname]	[-o toolopts] [-O fpartopts]
	    [-S] src_dir/ dst_url/

DESCRIPTION
     The fpsync	tool synchronizes directories in parallel using	fpart(1) and
     rsync(1) or cpio(1).  It computes subsets of src_dir/ and spawns jobs to
     synchronize them to dst_url/.

     Synchronization jobs can be executed either locally or remotely (using
     SSH workers, see option -w) and are executed on-the-fly while filesystem
     crawling goes on.	This makes fpsync a good tool for migrating large
     filesystems.

OPTIONS
     -h	     Print help

     -v	     Verbose mode.  Can	be be specified	several	times to increase ver-
	     bosity level.

     -m	tool
	     External copy tool	used to	synchronize files.  Currently sup-
	     ported tools are: rsync and cpio.	Default: rsync.	 When using
	     cpio and more than	one worker, directory timestamps may not be
	     replicated.  A second pass	will fix them.	See the	README file
	     for more details.

     -n	jobs
	     Start jobs	concurrent sync	jobs (either locally or	remotely, see
	     below).  Default: 2

     -f	files
	     Transfer at most files files or directories per sync job.	0
	     means unlimited but you must at least specify one file or size
	     limit.  Default: 2000

     -s	size
	     Transfer at most size bytes per sync job.	0 means	unlimited but
	     you must at least specify one file	or size	limit.	You can	use a
	     human-friendly unit suffix	here (k, m, g, t, p).
	     Default: 4294967296 (4 GB)

     -E	     Work on a per-directory basis (rsync tool only).  In that mode,
	     fpsync works with lists of	directories instead of files.  That
	     mode may generate coarse-grained lists but	enables	rsync(1) 's
	     --delete option by	default	( WARNING!!!  ), making	it a good can-
	     didate for	a final	(cleaning) pass	after several incremental
	     passes using standard (file) mode.

     -w	wrks
	     Use remote	SSH wrks to synchronize	files.	Synchronization	jobs
	     are executed locally when this option is not set.	wrks is	a
	     space-separated list of login@machine connection strings and can
	     be	specified several times.  You must be allowed to connect to
	     those machines using a SSH	key to avoid user interaction.

     -d	shdir
	     Set fpsync	shared directory to shdir.  This option	is mandatory
	     when using	SSH workers and	set by default to tmpdir when running
	     locally.  The specified directory must be an absolute path	; it
	     will be used to handle communications with	SSH hosts (sharing
	     partitions	and log	files) and, as a consequence, must be made
	     available to all participating hosts (e.g.	through	a r/w NFS
	     mount), including the master one running fpsync.

     -t	tmpdir
	     Set fpsync	temporary directory to tmpdir.	This directory remains
	     local and does not	need to	be shared amongst SSH workers when us-
	     ing the -w	option.	 Default: /tmp/fpsync

     -r	jobname
	     Resume job	jobname	and restart synchronizing remaining partitions
	     from a previous run.  jobname can be obtained using verbose mode
	     (see option -v).  Note that filesystem crawling is	skipped	when
	     resuming a	previous run.  As a consequence, options -f, -s, -o,
	     -O, -S, src_dir/, and dst_url/ are	ignored.

     -o	toolopts
	     Override default rsync(1) or cpio(1) options with toolopts.  Use
	     this option with care as certain options are incompatible with a
	     parallel usage (e.g. rsync's --delete).  Default for rsync:
	     -lptgoD -v	--numeric-ids.	Empty for cpio.

     -O	fpartopts
	     Override default fpart(1) options with fpartopts.
	     Default: -x .zfs -x .snapshot* -x .ckpt

     -S	     Sudo mode.	 Use sudo(8) for filesystem crawling and synchroniza-
	     tions.

     src_dir/
	     Source directory.	It must	be absolute and	available on all par-
	     ticipating	hosts (including the master one, running fpsync).

     dst_url/
	     Destination directory or URL (rsync tool only).  If a remote URL
	     is	provided, it must be supported by rsync(1).  All participating
	     workers must be able to reach that	target.

RUNNING	FPSYNC
     Each fpsync run generates a unique	jobname, which is displayed in verbose
     mode (see option -v) and within log files.	 You can use that jobname to
     resume a previous run (see	option -r).  fpsync will then restart synchro-
     nizing data from the parts	that were being	synchronized at	the time it
     stopped.

     This unique feature gives the administrator the ability to	stop fpsync
     and restart it later, without having to restart the whole filesystem
     crawling and synchronization process.  Note that resuming is only possi-
     ble when filesystem crawling step has finished.

     During synchronization, you can press CTRL-C to interrupt the process.
     The first CTRL-C prevents new synchronizations from being submitted and
     the process will wait for current synchronizations	to be finished before
     exiting.  If you press CTRL-C again, current synchronizations will	be
     killed and	fpsync will exit immediately.  When using option -E to enable
     directory mode and	rsync's	--delete option, keep in mind that killing
     rsync processes may lead to a situation where certain files have been up-
     dated and others not deleted yet (because the deletion process is post-
     poned using rsync's --delete-after	option).

     On	certain	systems, CTRL-T	can be pressed to get the status of current
     and remaining parts to be synchronized.  This can also be achieved	by
     sending a SIGINFO to the fpsync process.

     Whether you use verbose mode or not, everything is	logged within
     shdir/log/.

EXAMPLES
     Here are some examples:

     fpsync -n 4 /usr/src/ /var/src/

	     Synchronizes /usr/src/ to /var/src/ using 4 local jobs.

     fpsync -n 2 -w login@machine1 -w login@machine2 -d	/mnt/fpsync /mnt/src/
	     /mnt/dst/

	     Synchronizes /mnt/src/ to /mnt/dst/ using 2 concurrent jobs exe-
	     cuted remotely on 2 SSH workers (machine1 and machine2).  The
	     shared directory is set to	/mnt/fpsync and	mounted	on the machine
	     running fpsync, as	well as	on machine1 and	machine2.  The source
	     directory (/mnt/src/) is also available on	those 3	machines,
	     while the destination directory (/mnt/dst/) is mounted on SSH
	     workers only (machine1 and	machine2).

LIMITATIONS
     Parallelizing rsync(1) or cpio(1) makes several options not usable, such
     as	rsync's	--delete.  If your source directory is live while fpsync is
     running, you will have to delete extra files from destination directory.
     This is traditionally done	by using a final -offline- rsync(1) pass that
     will use this option, but you can also use	fpsync and option -E to	per-
     form the same task	using several workers.

     fpsync enqueues synchronization jobs on disk, within the tmpdir/queue di-
     rectory.  Be careful to host this queue on	a filesystem that can handle
     fine-grained mtime	timestamps (i.e. with a	sub-second precision) if you
     want the queue to be processed in order when fpart(1) generates several
     jobs per second.  On FreeBSD, VFS(9) timestamps' precision	can be tuned
     using the 'vfs.timestamp_precision' sysctl.  See vfs_timestamp(9).

     Contrary to rsync(1), fpsync enforces the final '/' on the	source direc-
     tory.  It means that directory contents are synchronized, not the source
     directory itself (i.e. you	will not get a subdirectory of the name	of the
     source directory in the target directory after synchronization).

SEE ALSO
     cpio(1), fpart(1),	rsync(1), sudo(8)

AUTHOR,	AVAILABILITY
     Fpsync has	been written by	Ganael LAPLANCHE and is	available under	the
     BSD license on http://contribs.martymac.org

BUGS
     No	bug known (yet).

FreeBSD	13.0		       January 27, 2015			  FreeBSD 13.0

NAME | SYNOPSIS | DESCRIPTION | OPTIONS | RUNNING FPSYNC | EXAMPLES | LIMITATIONS | SEE ALSO | AUTHOR, AVAILABILITY | BUGS

Want to link to this manual page? Use this URL:
<https://www.freebsd.org/cgi/man.cgi?query=fpsync&manpath=FreeBSD+12.2-RELEASE+and+Ports>

home | help