FreeBSD Manual Pages
BBFTP(1) User Contributed Perl Documentation BBFTP(1) NAME bbftp - transfer files using compression and parallel streams SYNOPSIS bbftp -v bbftp [Options] [-u RemoteUsername] -i ControlFile [RemoteHost] bbftp [Options] [-u RemoteUsername] -e ControlCommands [RemoteHost] DESCRIPTION Use the bbftp command to transfer files between the local host and a remote host. In order to get better performance on a loaded Wide Area Network than transferring with the standard ftp command, use the bbftp command. bbftp has been designed to take advantage of the RFC 1323 and uses multiple streams is order to speed up transfer. It also implements an automatic retry in case of failure of commands contained in the ControlFile or in the ControlCommands. bbftp allows two methods of connection to the remote host, a direct connec- tion on the control port of the remote bbftpd daemon or the ability to re- motely start a bbftpd daemon through an ssh tunnel. For the first method, the bbftpd daemon has to be started (by inetd or as a standalone daemon) on the remote host and all user information (username, password) are transmit- ted encrypted to the daemon; for the second the bbftpd binary has to be ac- cessible somewhere on the remote host and all control data are transmitted through the ssh tunnel. A third additionnal authenticate mode allows to use certificates to log on. This mode is based on the Grid Security Infrastructure and requires Globus software to be installed. The client side needs a certificate to identify itself and the daemon needs a host certificate. The behaviour of bbftp is controled by the ControlFile which contains the commands to be executed (option -i) or by commands separated by semicolons (option -e). The format of these commands are given in CONTROL COMMANDS. bbftp may be used in one of the following ways: % bbftp to print a short help % bbftp -v to write the version of the software and default values to standard output. % bbftp [Options] -i ControlFile [-u RemoteUsername] [RemoteHost] % bbftp [Options] -e ControlCommands [-u RemoteUsername] [RemoteHost] to request the execution of commands contained in the control file Con- trolFile or the ControlCommands using RemoteUsername on RemoteHost. Depending on the Options a password for the RemoteUsername on RemoteHost may be requested interactively (see OPTIONS section, for a full description of the Options). The RemoteUsername value is ignored in the certificate authentication mode. The remote user is automatically detected by the daemon using the Globus gridmap file. OPTIONS Options can be separated into two classes, those describing the way bbftp connects to the bbftpd daemon and those modifying the behaviour of the con- trol commands. CONNECTION OPTIONS -u Use this option to specify the user on the remote host. In SSH mode (see below) this option is optional: if not specified, the user in SSH config file will be used, or the local user if not in the SSH config file. When compiled with the certificates authentication module, the use of this option will determine the way the authentication is done: if -u is used, the classical user/password will be used, otherwise the certificate will be used. -s Use this option to use ssh to remotely start a bbftpd daemon. It usu- ally starts the binary "bbftpd -s", but this can be changed througth the -E option. -S Same as -s but ask ssh to run without asking a question (no password, no host key checking). It implies the usage of the identity file (for those familiar to ssh, this in done in setting the ssh options Batch- Mode to yes and the option StrictHostKeyChecking to no). -E Server command to run This option has to be used if the binary to be started on the remote host is not the default one (usually "bbftpd -s"). This option also im- plies the usage of the ssh mode (no need to add the -s option) -I SSH identity file This option has to be used if the ssh identity file is not the default one (usually $HOME/.ssh/identity). This option also implies the usage of the ssh mode (no need to add the -s option) -L SSH command If your ssh command is not the default one (usually "ssh -q"), use this option to change it. This option also implies the usage of the ssh mode (no need to add the -s option) -P Private authentication string If your client have been compiled with a private authentication schema, this option allow to pass an arbitrary string to the authentication module. You can determine if your client is using a private authentica- tion module with the -v option. -w Control port number Use this option to change the control port for connection to on the Re- moteHost. This option is meaningless in ssh mode. PROTOCOL OPTIONS -D[min:max] Domain for ephemeral ports Use this option to use protocol V2 (non-passive mode) instead of V3 (passive mode). min and max are the range of ports used for data sock- ets. If min and max are not specified, free ports will be used (or built-in ports at compilation time). BEHAVIOUR OPTIONS -b Use this option to run bbftp in the background after all interactive requests. -c Use this option to gzip the data during transmission. Compression and uncompression are done "on the fly" and do not require any additionnal disk space. Do not use it if the files to transmit are not compress- ible, as this will only lead to a waste of CPU and time. This option can be overridden by the control command setoption gzip -f ErrorFile Use this option to redirect the data generated on the standard error to ErrorFile -g Globus service name Use this option in certificate mode if the server does not use a host server. The service name will be the subject of the certificate the server is actually using. -m Use to special output on file transfer. The ouput is in the following format: Direction NumberOfBytes NumberOfSeconds BuffersizePerStream SendWinSize RecvWinSize NumberOfStreams Compression BuffersizePerStream, SendWinSize, RecvWinSize are expressed in Kilo- Bytes For example : get 10000 10 256 256 256 2 gzip means that 10000 Bytes have been transfered in 10 seconds in compressed mode using 2 streams, a buffer size per stream of 256 KBytes, a TCP send window size of 256 KBytes and a TCP receive window size of 256 KBytes. This option is useful when trying to choose the best parameters between two sites. When this option is set -V, -W and -t options are not available. -n Use this option to simulate the transfer. All commands will be executed but no data will be transfered with the get, mget put and mput com- mands. Output is the same as a real transfer. OutputFile -o OutputFile Use this option to redirect the data generated on the standard output to OutputFile -q Use this option to mark packets for QBSS (QBone Scavenger Service) -p NumberOfParallelStreams Use this option to increase the number of streams to use during the file transfer. Default is 1. This option can be overridden by the con- trol command setnbstream -r NumberOfTries Use this option to change the number of tries to use when a transfer fails. The default is usually 5. -R .bbftprc file After a successful connection to the daemon the client is going to exe- cute all control commands located in the "$HOME/.bbftprc" file. The lo- cation of this file can be changed with this option. Take care, not all control commands are allowed in the .bbftprc file (See CONTROL COMMANDS to know the authorized one) If this option is not provided and "$HOME/.bbftprc" does not exists, the client tries to execute all com- mands in "/etc/bbftp.conf". -t Use this option to have a timestamp on all output (overridden by -m op- tion). -V Use this option to set the client in verbose mode (overridden by -m op- tion). -W Use this option to print warnings to stderr (overridden by -m option). CONTROL COMMANDS The control commands are either contained by an ASCII file ( -i option) or written on the bbftp line ( -e option). They can be divided into two classes, the "File related commands" and the "Behaviour commands". All "Behaviour commands" may be put in a .bbftprc file, but all "File re- lated commands" are forbiden in that file. IMPORTANT NOTE Under the RFIO mode (see setoption remoterfio and setoption localrfio ) all file related commands have to be given in absolute mode. FILE RELATED COMMANDS cd "RemoteDirectory" Change the current directory of the daemon on the remote host. If the RemoteDirectory is given in relative mode (not beginning by a /), it is supposed to be relative to the directory where the daemon is currently running. After the first connection, the current directory is the home directory of the RemoteUsername . The client keeps in mind the current remote directory so in case of broken connection during a transfer, it can reset the current directory of the daemon to the correct directory. If the daemon has been set in RFIO mode (see setoption remoterfio ) this option is unavailable. get " RemoteFile LocalFile" Transfer the remote file RemoteFile to the local host with the name Lo- calFile. If the local file already exists it is overwritten (only in case of successful transfer if the setoption tmpfile has been set). Un- der some circumstances (No space left on device, Access denied, File is a directory ...), no retry is done and the next command is processed. If the RemoteFile is given in relative mode (not beginning by a /), it is supposed to be relative to the current directory on the remote host (which is set to the home directory of the RemoteUsername at the begin- ning). If the LocalFile is given in relative mode (not beginning by a /) the file is created relative to the directory where the bbftp com- mand is running (which may have been changed with the lcd command). get " RemoteFile LocalDir/" Transfer the remote file RemoteFile to the local host with the name Re- moteFile (without path) in the LocalDir directory. If the local file already exists it is overwritten (only in case of successful transfer if the setoption tmpfile has been set). Under some circumstances (No space left on device, Access denied, File is a direc- tory ...), no retry is done and the next command is processed. If the RemoteFile is given in relative mode (not beginning by a /), it is supposed to be relative to the current directory on the remote host (which is set to the home directory of the RemoteUsername at the begin- ning). If the LocalDir is given in relative mode (not beginning by a /) the file is created relative to the directory where the bbftp command is running (which may have been changed with the lcd command). get " RemoteFile" Transfer the remote file RemoteFile to the local host with the name Re- moteFile. If the RemoteFile is given in relative mode (not beginning by a /), it is supposed to be relative to the current directory on the remote host (which is set to the home directory of the RemoteUsername at the begin- ning) and created on the local host relative to the directory where the bbftp command is running (which may have been changed with the lcd com- mand). lcd " LocalDirectory" Change the current directory on the local host. If the LocalDirectory is given in relative mode (not beginning by a /), it is supposed to be relative to the directory where the client is currently. If the client has been set into RFIO mode (see setoption localrfio ) this option is unavailable. mget " RemoteFiles LocalDirectory" Expand the RemoteFiles on the remote machine and do a "get" for each file name thus produced. Files are transferred into the LocalDirec- tory. mget " RemoteFiles" Expand the RemoteFiles on the remote machine and do a "get" for each file name thus produced. Files are transferred into the local working directory, which can be changed with the lcd command. mkdir " RemoteDirectory" Create the directory RemoteDirectory on the remote host. If the direc- tory already exist no retry is done and the next command of the file is processed. If the RemoteDirectory is given in relative mode (not begin- ning by a /) the directory is created relative to the current remote directory. If one directory in the given path does not exist the command will fail if the setoption nocreatedir is set. If the setoption createdir has been set all unexisting directories will be created. mput " LocalFiles RemoteDirectory" Expand wild cards in the list of local files given as arguments and do a "put" for each file in the resulting list. Files are transfered into the RemoteDirectory. mput " LocalFiles" Expand wild cards in the list of local files given as arguments and do a "put" for each file in the resulting list. Files are transfered into the current remote directory which can be changed with the cd command. put " LocalFile RemoteFile" Transfer the local file LocalFile to the remote host with the name Re- moteFile. If the remote file already exists it is overwritten. Under some circumstances (No space left on device, Access denied ...), no retry is done and the next command of the file is processed. If the LocalFile is given in relative mode (not beginning by a /) the file is supposed to be relative to the directory where the bbftp com- mand is running (which may have been changed with the lcd command). If the RemoteFile is given in relative mode (not beginning by a /), it is created relative to the current directory on the remote host (which is set to the home directory of the RemoteUsername at the beginning). put " LocalFile RemoteDir/" Transfer the local file LocalFile to the remote host with the name Lo- calFile (without the path) in the RemoteDir directory If the remote file already exists it is overwritten. Under some circum- stances (No space left on device, Access denied ...), no retry is done and the next command of the file is processed. If the LocalFile is given in relative mode (not beginning by a /) the file is supposed to be relative to the directory where the bbftp com- mand is running (which may have been changed with the lcd command). If the RemoteDir is given in relative mode (not beginning by a /), it is created relative to the current directory on the remote host (which is set to the home directory of the RemoteUsername at the beginning). put " LocalFile" Transfer the local file LocalFile to the remote host with the name Lo- calFile. If the LocalFile is given in relative mode (not beginning by a /) the file is supposed to be relative to the directory where the bbftp com- mand is running (which may have been changed with the lcd command) and created relative to the current directory on the remote host (which is set to the home directory of the RemoteUsername at the beginning). dir " RemoteDir" If used with '-m', lists the content of the remote directory. Each line has the following format: LINE: <LINK_CHAR><TYPE_CHAR><SP><FILE_NAME> LINK_CHAR: 'l' if file is a link else blank TYPE_CHAR: 'd' if file is a directory, 'f' if it is a reguler file or 'u' if it is an invalid link SP: ' ' FILE_NAME: STRING If -m is not used, the result will be only 'OK' or 'FAILED'. This can be used to test the existence of a directory without printing its con- tent. rm " RemoteFile" Deletes the remote file RemoteFile stat " RemoteFile" If used with '-m', print statistics about the file or the directory in the following format: device inode mode size block_size blocks uid gid (access time) (modifi- cation time) (change time) df " RemoteDir" If used with '-m', print statistics about the file system that contains the directory RemoteDir in the following format: total_blocks blocks_available block_size total_inode inode_available filename_max_length BEHAVIOUR COMMANDS setoption " Option" To negate an option just add "no" before the option (ie setoption nocreatedir). The options are the following : createdir All file-related commands will create missing directories if needed (default createdir). gzip All file transfers will be compressed using the gzip algorythm (default nogzip). keepaccess The access time and modify time will be kept on each file transferred (default keepaccess). keepmode The file mode will be kept on each file transferred (default keepmode). localrfio All local files will be created with RFIO functions (default nolocalr- fio). remoterfio All remote files will be created with RFIO functions (default nore- moterfio). qbss All the packets will be marked for QBSS (default noqbss). tmpfile All files will be created under a temporary name (File- Name.bbftp.tmp.HostName.Pid) and renamed to the correct file name if transfer is successful (default tmpfile) setbuffersize " Buffersize" Set the size in Kbytes of the buffer used for reading or writing the files. This command set the local and remote buffer size. (Each stream will use the same buffer size) setlocalcos " LocalCos" Set the local COS to the value specified by LocalCos. This COS will be used for further rfio funtions. It is used if the setoption localrfio has been set and if the file is a HPSS file. A value of 0 allows to select the COS according to the file size. A negative value allows to not set the COS. The default value is 0. setlocalumask " LocalUmask" Set the local umask to the value specified by LocalUmask. This umask will be used for further i/o funtions. The LocalUmask has to be given in OCTAL setnbstream " NumberOfParallelStreams" Set the number of parallel streams to NumberOfParallelStreams. This number will be used for further transfer commands. setremotecos " RemoteCos" Set the remote COS to the value specified by RemoteCos. This COS will be used for further rfio funtions. It is used if the setoption re- moterfio has been set and if the file is a HPSS file. A value of 0 al- lows to select the COS according to the file size. A negative value allows to not set the COS. The default value is 0. setremoteumask " RemoteUmask" Set the remote umask to the value specified by RemoteUmask. This re- mote umask will be used for further i/o funtions. The RemoteUmask has to be given in OCTAL setrecvwinsize " WindowSize" Set size in Kbytes of the receive TCP window of each stream of the bbftpd daemon. This also set the send window size of the client to the same value. setsendwinsize " WindowSize" Set size in Kbytes of the send TCP window of each stream of the bbftpd daemon. This also set the receive window size of the client to the same value. setackto " Acknowledge time-out" Set time-out (in seconds) to wait for an acknowledge. Default value is 100 setrecvcontrolto " Input control time-out" Set time-out (in seconds) to wait while reading on the control socket. Default value is 180 setsendcontrolto " Output control time-out" Set time-out (in seconds) to wait while writing on the control socket. Default value is 180 setdatato " Data time-out" Set time-out (in seconds) to wait while reading on the data socket. De- fault value is 300 NOTES If the option tmpfile is used then if the new file ( RemoteFile for a put or LocalFile for a get) did not exist before, bbftp ensures that the file transfer was correct if the file exists. In case of an already existing file, if the size, the last access and modi- fication time are correct (if option keepaccess has been set) bbftp en- sures that the file transfer was correct. TURL NOTATION The TURL notation can be used with commands relatives to files and direc- tory. If you use this notation, you should not specify the server name in com- mand. BBFTP support 3 types of TURL: Remote TURL bbftp://bbftp_servername/path_to_file_or_directory The path to file can be relative or absolute (start with '/') For example: "mkdir bbftp://bbftp.domain/dir" is the command to create a directory dir on the remote bbftp server un- der the user's home directory "cd bbftp://bbftp.domain//tmp" is the command to change into /tmp on the remote bbftp server Local TURL file://path_to_file The path to file can be relative or absolute (start with '/') For example: "put file://file bbftp://bbftp.domain//tmp/file" is the command to transfer the file file located on the local host server under the current directory to the remote host bbftp.domain un- der the /tmp directory Local rfio TURL rfio://path_to_file The path to file can be relative or absolute (start with '/') For example: "put rfio:///path/to/MSS/file bbftp://bbftp.domain//tmp/file" is the command to transfer a file retrived via the rfio path /path/to/MSS/file into the remote host bbftp.domain under the /tmp di- rectory When the 'rfio://' notation is used, the option 'localrfio' is no longer necessary. Remote TURL with rfio It is possible to use rfio on the remote side using mixed TURL like: bbftp://bbftp_servername/rfio://path_to_rfio_file When the 'bbftp://bbftp_servername/rfio://' notation is used, the option 'remoterfio' is no longer necessary. RETURN VALUES The following exit values are returned: "0" if all commands were successfuly executed ">0"if one command failed. It may happend that a non-zero value is returned even if all files were correctly transfered, if during one transfer a retry was needed. This will be corrected in future releases. MESSAGES AND ERRORS All informative messages are written to the standard ouput (or to the Out- putFile ). All error messages are written to the standard error (or to the ErrorFile ). WARNING The bbftp client version 2.0.0 is unable to talk with a daemon in release 1.x.x. The rfioxxx or xxxrfio commands are no longer supported, use instead the options localrfio or remoterfio in conjunction with put and get commands to obtain the same result. RESULT FILE If the -i option was used a result file will be created in the same direc- tory as the ControlFile. Its name is ControlFile with the extension ".res". It contains the same lines as the ControlFile plus the keyword "OK", in case of success, or "FAILED", in case of failure. If the -e option was used and the -V option was not used, the software will print the command executed plus the keyword "OK", in case of success, or "FAILED", in case of failure to standard output. CONNECTION EXAMPLES bbftp -i ctrlfile -u jon -p 5 -c cchost.in2p3.fr means that bbftp is going to connect to remote host cchost.in2p3.fr using username jon. If the connection is successful then the commands in ctrl- file will be executed. All transfer commands will use five streams and gzip compression. bbftp -i ctrlfile -u phg -s cchost.in2p3.fr means that bbftp is going to start a remote bbftp via sshd on host cchost.in2p3.fr using username phg. ssh will first try an RSAAuthentica- tion if it is allowed by cchost.in2p3.fr; otherwise ssh will ask for a password for user phg on cchost.in2p3.fr. Then the sshd on cchost.in2p3.fr will log user phg and try to start the command "bbftpd -s" bbftp -i ctrlfile -u jon -E '/tmp/bbftpd -s' cchost.in2p3.fr Same behaviour as preceding, except that the remote command will be "/tmp/bbftpd -s" bbftp -i ctrlfile -u gilles -S cchost.in2p3.fr means that bbftp is going to start using ssh a remote bbftpd on host cchost.in2p3.fr using username gilles. ssh will try an RSAAuthentication if it is allowed by cchost.in2p3.fr, otherwise the connection will be bro- ken. bbftp -e 'setrecvwinsize 1024 ; put file1 file2' -u jon cchost.in2p3.fr means that bbftp is going to connect to remote host cchost.in2p3.fr using jon username. If the connection is successful then the commands se- trecvwinsize 1024 and put file1 file2 will be executed. All tranfer com- mands will use one stream. bbftp -e 'put file1 file2' cchost.in2p3.fr means (in the certification authentication mode) that bbftp is going to connect to remote host cchost.in2p3.fr using a certificate. The remote user will be detected by the daemon which will check for the certificate provided and will accept or not the connection. Using SSH to start a BBFTPD daemon linked with dynamic libraries If you have linked the daemon with dynamic libraries with -L/path/to/lib option, you need to specify this location in $LD_LIBRARY_PATH. To be taken into account by SSHD, this environment variable must be modified in the $HOME/.ssh/environment file. See your SSH or SSHD manual for more details. EXAMPLES User jon want to transfer files from host localhost to remotehost on the account bbrdist. The bbrdist account has /home/babar/bbrdist as default di- rectory on remotehost but has no subdirectories. We are going to study a control file in order to understand bbftp behaviour (we do not care here about the connection method; see CONNECTION EXAMPLES for that). User jon on the local host is on the /home/babar/jon directory and has the following control file (all lines have a number which must not exists but which are there just for clarity) : 1 setnbstream 20 2 setremoteumask 022 3 setoption nocreatedir 4 put /home/babar/jon/f1 /home/babar/bbrdist/newfiles/f1 5 setoption createdir 6 put /home/babar/jon/f1 /home/babar/bbrdist/newfiles/f1 7 setnbstream 5 8 setrecvwinsize 1024 9 setoption gzip 10 put /home/babar/jon/f2 /home/babar/bbrdist/newfiles/f2 Command 1 just sets the number of parallel streams to 20 for further get or put commands. Command 2 sets the remote umask for further put commands. Command 3 indicates that no directory has to be created on further put or get commands. Command 4 tries to send the local file /home/babar/jon/f1 to /home/babar/bbrdist/newfiles/f1. This command will fail because bbrdist has no subdirectory and directory creation is inhibed. Command 5 resets the createdir option. Command 6 will be successful (if the connection does not break), because the creation of the directory /home/babar/bbrdist/newfiles has been autho- rized by the createdir option. Command 7 reduces the number of streams to 5 Command 8 sets the receive TCP window size to 1024 Kbytes on remotehost and the send TCP window size to 1024 Kbytes on localhost. Command 9 sets the gzip option for further get or put commands. Command 10 will transfer file /home/babar/jon/f2 to /home/babar/bbrdist/newfiles/f2 with 5 streams in compressed mode. AUTHORS bbftp was developed by Gilles Farrache. It is now maintained by Lionel Schwarz and Pierre-Emmanuel Brinette at IN2P3 Computing Center , Villeur- banne (FRANCE). CONTRIBUTORS Tim Adye (Idea and implementation of ssh mode) Gilles Gallot (Mutli-IP addresses support, secondary groups support, port on various systems and bug fixes) Pierre-Emmanuel Brinette (Bug fix) Andrew Goodney (Port to Darwin) Paola Grosso (Idea and implementation of the -q client option) Petr Holub (Port to Windows cygwin) Dan Schrager (Idea and implementation of the -D client option) Rod Walker & Kostas Georgiou (Idea and implementation of the -g client op- tion) Shuwei Ye (Bug fix) BUGS Send bugs / comments to bbftp@in2p3.fr SEE ALSO bbftpd(1). POD ERRORS Hey! The above document had some coding errors, which are explained below: Around line 1: =cut found outside a pod block. Skipping to next block. Around line 565: You can't have =items (as at line 572) unless the first thing after the =over is an =item perl v5.10.0 2013-02-07 BBFTP(1)
NAME | SYNOPSIS | DESCRIPTION | OPTIONS | CONTROL COMMANDS | TURL NOTATION | RETURN VALUES | MESSAGES AND ERRORS | WARNING | RESULT FILE | CONNECTION EXAMPLES | EXAMPLES | AUTHORS | CONTRIBUTORS | BUGS | SEE ALSO | POD ERRORS
Want to link to this manual page? Use this URL:
<https://man.freebsd.org/cgi/man.cgi?query=bbftp&sektion=1&manpath=FreeBSD+Ports+15.1.quarterly>