SHMUX(8) System Manager's Manual SHMUX(8) NAME shmux - Shell Multiplexor SYNOPSIS shmux [ -h | -V ] shmux [ -bBdFmpqQstv ] [ -C timeout ] [ -M max ] [ -r rcmd ] [ -S mode ] [ -e list ] [ -E list ] [ -a analyzer ] [ -A condition ] [ -o dir ] [ -P timeout ] [ -T timeout ] -c command [ - | targets... ] DESCRIPTION shmux is program for executing the same command on many hosts in parallel. For each target, a child process is spawned by shmux, and a shell on the target obtained one of the supported methods: rsh, ssh, or sh. The output produced by the children is received by shmux and either (optionally) out- put in turn to the user, and/or written to files for later processing. The list of targets may either be specified on the command line or be read from the standard input. The default method may be set by using the -r option, and may be overridden for any target by prefixing the target name by the method and a colon. Two special methods, ssh1 and ssh2 may be used to select the SSH protocol ver- sion 1 or 2, respectively. For the sh method, it is up to the specified command to contact the target. In order for the executed command to know which target it is being invoked for, the SHMUX_TARGET environment variable is set by shmux. Before executing the specified command, shmux will optionally ping each target to ensure that it can be reached, and/or run a dummy test command to make sure that the target not only is alive, but that it is possible to cleanly execute a command on it. Both these tests are typically run with a fairly short timeout to quickly dismiss unavailable targets rather than waiting for the standard (longer) network timeout. After executing the specified command, shmux will optionally process the output or run a user specified command to determine whether the command was successful or resulted in an error. Optionally, the command output is sup- pressed for successful targets. Finally shmux will print out a summary of the results and list targets with errors. OPTIONS -h Display a brief help message. -V Display the version information. -c command Specify the command to execute on targets. -C timeout Specify a timeout for the command being executed on targets. This should be a number followed by a time unit. The following are valid time units: s(econds), m(inutes), h(our), d(ays), w(eeks). See the PROCESS MANAGEMENT section for details on how this is handled. -M max Defines the maximum number of spawned processes. While there is no real (or hard coded) limitation for this, the system resources are typically bounded and this affects shmux in turn. The most critical resource is the maximum number of open files a process may have. But one should also consider the load imposed on the system by the creation of a (potentially) large number of processes. -r rcmdDefines the default method used to run a shell on targets. -S modeThis options defines the strategy used by shmux when spawning chil- dren to run the command on targets. If the mode is set to "all", then processes will be spawned as fast as possible. When set to "check", processes will be spawned as long as commands succeed. Fi- nally, the default mode, "one", is to spawn only one process at first, and wait for it to succeed, at which point shmux will switch to the "check" mode. In the "one" or "check" modes, when the com- mand fails on a target, shmux will stop spawning new processes. -F When using either the "one" or "check" spawn modes (see -S), shmux will stop spawning new processes if the command fails on a target. By default, shmux then waits for user input if available, but re- verts to spawning processes if such input cannot be obtained (be- cause the interactive mode is disabled, either via -B or lack of a terminal). Using the -F option changes this behaviour to force shmux to quit once all existing processes have terminated after such a failure. -e listDefines which command exit codes should be considered errors and re- ported as such. The list should be a comma separated list of ranges of the form "x", "x-y", "x-" or "-y" where "x" and "y" are numbers between 0 and 255. -E listDefines which command exit codes should always be output to the screen. Codes that are considered errors (see the -e option) are always shown, but by default, other exit codes are only shown if the -v option is specified. This allows more flexibility. -a analyzer Defines how output should be analyzed by shmux after the command completes on a target. By default, nothing is done. Valid options are: lnregex, lnpcre, regex, pcre and run. This option requires -o to be used as well. -A condition When the -a option is used, it is also necessary to configure the chosen analyzer. For line based regular expressions (lnregex and lnpcre), the -A op- tion should be specified twice: once for the standard output and once for the standard error output. In both cases, the argument should be the name of a file containing a list of regular expres- sions, one per line. Each regular expression must be prefixed by the character '=' or '!'. The output of the command is analyzed line per line by checking whether it matches any of the specified regular expressions. If the first matching regular expression is prefixed by '=', then the command is considered to be successful (so far). Any other result (e.g. no match, or a match prefixed by '!') indicates an error condition. The default is to consider standard output normal, and standard error output indicative of an error. For other regular expressions (regex and pcre), the -A option should be specified twice: once for the standard output and once for the standard error output. This option allows defining the expression to be applied to the output. If the condition starts with the char- acter '=' then a match indicates that the command completed without error. If the condition starts with the character '!' then a fail- ure to match indicates that the command completed without error. The second character of the condition should either be '=' if the regular expression follows or '<' if the regular expression should be read from the file which name follows. (The '=' characters may be omitted.) A condition must be specified for the standard output, but is optional for standard error output. The default for the lat- ter is to consider any output as indicative of an error. For the run analyzer, the -A must be specified at least once with the name of a program to run, and optionally a second time to spec- ify a timeout which defaults to 15 seconds if unset. This timeout is implemented using alarm(3) and will not otherwise be enforced by shmux. The program will be called after a command completes on a target unless the command exit code is considered an error (see -e) with two arguments: the target name and the directory specified with the -o option. The output of the program will always be shown to the user. Exit codes other than 0 indicate that the command failed for the particular target. -o dir If specified, shmux will place the output and (for normal termina- tions) the exit code of the executed commands in files under this directory. For each target, the files will be named after the tar- get name so as to make it easy to identify. The files must not al- ready exist, so it is recommended that the directory be empty. This also means that each target must be unique. The directory will be created if it does not already exist. -p Ping targets to verify they are alive before doing anything. The target names must be unique or bad things will happen. -P timeout Defines the initial target ping timeout in milliseconds, see fping(8). (Implies -p.) -t Before executing the specified command for a target, send a simple "echo" test and verify that the output is correct. This step goes beyond the simple ping test as it verifies that it is possible to get a shell on the target. If this option is specified twice (or once when -T is used), output for failed tests will be displayed to help the user understand what went wrong. -T timeout Defines the test timeout in seconds. (Implies -t.) -m By default the output is displayed as soon as it is received. For multi-line outputs, this will typically result in output from sev- eral targets being mixed. This option may be used to keep each tar- get output separate. -b Show the bare output from the executed commands instead of prefixing each line by the corresponding target name. -B Enables batch mode which forcefully disables the interactive mode, ignoring input from the terminal. See also -s and -F. -s Suppress the progress status line. See also -B. -q Used once, this will suppress the output from successful targets (as defined by the use of options options -e, -a and -A. If using the run analyzer, the output of all targets is suppressed. Note that using this option effectively implies -m in most cases as failure is ultimately determined upon completion of the command. (Only the lnregex and lnpcre analyzers are able to determine errors while the command is running.) When specified twice, this option allows to completely suppress output from the command run on targets. This also changes the default spawn stategy to "all". Using this option (once or twice) requires using -o. -Q Suppress the final summary of results. -v Display internal status messages. -D Display internal debug messages. EXIT CODES shmux will optionally report the exit code of the command it runs for each target. For the sh command, this will be the same as the command itself, but this is the exception. For other methods, the exit code reported by shmux really is the code returned by the program used to spawn command for that method. In particular, rsh typically only returns either 0 or 1 while different im- plementations of ssh have different behaviours which may include returning the actual exit code of the remotely executed command or not. PROCESS MANAGEMENT Children processes spawned by shmux are made process group leaders (using setpgid(2)) in order to group all of their descendants in a single process group which allows to easily send signals to all of these processes. This is especially useful for the sh method which can generate many processes locally, but has no effect on remote processes. shmux uses pipe(2) to capture the output generated by its children (even when this output is redirected to a file with -o) and will read from these until they are closed (by the child as well as its descendants, if any). Once there is nothing left to read, shmux will wait for the child and its descendants to terminate before considering that particular target done with. The timeout is implemented by using alarm(3) which (unless canceled or in- tercepted) will terminate the child process upon receipt but is not inher- ited by descendants of that process. For that reason, shmux will send a SIGALRM signal to the child's process group when the child itself termi- nates upon such signal. After the scheduled SIGALRM time, shmux will wait an additional 5 seconds for all of the processes in the group to terminate to give the child a chance to handle the timeout more gracefully if desired. It will then send a SIGTERM signal to the child's process group. After another 5 seconds, if any process is still alive in the group, a final SIGKILL signal is sent to the process group. Note that only commands which terminate because of an ALRM signal or be- cause of a TERM or KILL signal after such signal has been issued by shmux will be reported as having timed out in the final summary report produced by shmux upon exiting. This is not as accurate as it should be. INTERACTIVE MODE By default, shmux offers a minimal "interactive mode" while running: it reads commands from the terminal and acts upon them accordingly. This can be disabled using the -B option. The following commands are currently rec- ognized: q Prevent shmux from spawning any more children, wen for active chil- dren to complete, and then quit. Q Quit immediately (e.g. abort), not waiting for any active children to terminate. This isn't recommended as children may be left run- ning unattended. A better alternative is to press control-C which will be received by shmux and all running children. These will typ- ically abort, and shmux terminate in a cleaner fashion. space Initiate a pause, this prevents shmux from spawning any more chil- dren to do its work. The following three commands may be used to resume work. 1 This is equivalent to specifying "-S more" on the command line, but may be used at any time. return This is equivalent to specifying "-S check" on the command line, but may be used at any time. + This is equivalent to specifying "-S all" on the command line, but may be used at any time. F This toggles the failure mode for the "one" and "check" spawn modes (see -S) between "pause" and "quit" (see -F). p Display the list of pending targets. r Display the list of targets for which a children process is cur- rently running. f Display the list of targets for which shmux failed to run the com- mand. e Display the list of targets for which the command completed with er- ror(s). s Display the list of targets for which the command completed success- fully. a Display the list of all targets with their status. k Allows sending a signal to a target. v Toggles whether internal messages are displayed or not. D Toggles whether debug messages are displayed or not. ENVIRONMENT VARIABLES shmux will use the following environment variables if set: SHMUX_MAX Specifies the maximum number of processes to spawn simultaneously. If the -M option is specified, it overrides this variable. SHMUX_RCMD Specifies the default command used to run a shell on targets. If the -r option is specified, it overrides this variable. SHMUX_SH Specify the shell to use for targets using the sh method. (Default is "/bin/sh".) SHMUX_RSH Specify an alternative command to "rsh" to use for targets using the rsh method. SHMUX_SSH, SHMUX_SSH1, SHMUX_SSH2 Specify an alternative command to "ssh" to use for targets using the ssh, ssh1 and ssh2 method (respectively). The variable SHMUX_SSH is used for all methods, but superseded by the other specific vari- ables. SHMUX_SSH_OPTS, SHMUX_SSH1_OPTS, SHMUX_SSH2_OPTS The variables may be used to specify replace the default "-xa -oLogLevel=ERROR" options passed to "ssh", "ssh1" and "ssh2" (re- spectively). The variable SHMUX_SSH_OPTS is used for all methods, but superseded by the other specific variables. Multiple arguments may be supplied, using white-space characters to separate them from each other. You can use single- and/or double- quotes to preserve white-spaces in an argument (but not back- slashes). Note that doubling a single-quote inside a single-quoted string gives you a single-quote; likewise for double-quotes (though you need to pay attention to which quotes your shell is parsing and which quotes shmux is parsing). SHMUX_SPAWNMODE Specifies the default spawn mode. If the -S option is specified, it overrides this variable. SHMUX_ERRORCODES Specifies the default list of exit codes to be considered errors. If the -e option is specified, it overrides this variable. SHMUX_SHOWCODES Specifies the default list of exit codes to be always display. If the -E option is specified, it overrides this variable. EXIT STATUS The following exit values are returned: 0 indicates that shmux was able to complete its work. However, this does not mean that all targets successfully ran the command, nor that shmux ran without any error. 1 indicates shmux was cleanly interrupted before it could finish its work. 2 indicates shmux aborted upon user request, possibly leaving some children running. 3 indicates shmux encountered a fatal error which caused it to abort, possibly leaving some children running. 4 indicates a syntax error or a fatal error which caused shmux to abort before it could spawn any children. SEE ALSO fping(8), sh(1), rsh(1), ssh(1), regex(3), pcre(3). AVAILABILITY The latest official release of shmux is available on GitHub at http://github.com/shmux. AUTHOR Christophe Kalt <ck@taranis.org> BUGS All rsh and some ssh implementations effectively achieve user authentica- tion by using privileged ports. Since there are only 1024 such ports, there is an obvious limitation on how many connections may be open using these programs. To make matters worse such ports may be held for four min- utes (by default) as per the TCP specification. When using shmux, one can very quickly run out of privileged ports, at which point failures will start happening. All rsh and ssh implementations fail to accurately return the remote com- mand exit code to the caller. Typically, rsh will simply not bother to re- turn this exit code, and ssh will try its best but is limited by the fact it also tries to return an error code for SSH related errors. In either case, this may pose problems when using -e. Under Cygwin, child processes do not inherit alarm(3) settings from the parent disabling this simple timeout enforcement system used by shmux. Some shells will ignore SIGALRM, others die upon its receipt (regardless of any trap). Again, this will affect the simple timeout enforcement system used by shmux. For bug reports, create an issue at https://github.com/shmux/shmux/issues, or send an email to shmux-bugs@taranis.org. 2017-01-13 SHMUX(8)
NAME | SYNOPSIS | DESCRIPTION | OPTIONS | EXIT CODES | PROCESS MANAGEMENT | INTERACTIVE MODE | ENVIRONMENT VARIABLES | EXIT STATUS | SEE ALSO | AVAILABILITY | AUTHOR | BUGS
Want to link to this manual page? Use this URL:
<https://man.freebsd.org/cgi/man.cgi?query=shmux&sektion=1&manpath=FreeBSD+Ports+15.1.quarterly>