rwflowpack(8) SiLK Tool Suite rwflowpack(8) NAME rwflowpack - Collect flow data and store it in binary SiLK Flow files SYNOPSIS rwflowpack [--input-mode=MODE] INPUT_MODE_SPECIFIC_SWITCHES [--output-mode=MODE] OUTPUT_MODE_SPECIFIC_SWITCHES { --log-destination=DESTINATION | --log-pathname=FILE_PATH | --log-directory=DIR_PATH [--log-basename=LOG_BASENAME] [--log-post-rotate=COMMAND] } [--no-file-locking] [--flush-timeout=VAL] [--file-cache-size=VAL] [--pack-interfaces] [--byte-order=ENDIAN] [--compression-method=COMP_METHOD] [--error-directory=DIR_PATH] [--archive-directory=DIR_PATH] [--flat-archive] [--post-archive-command=COMMAND] [--site-config-file=FILENAME] [--log-level=LEVEL] [--log-sysfacility=NUMBER] [--pidfile=FILE_PATH] [--no-chdir] [--no-daemon] To collect flow data over the network or directory polling (default): rwflowpack [--input-mode=stream] --sensor-configuration=FILE_PATH [--packing-logic=PLUGIN] [--sensor-name=SENSOR] [--polling-interval=NUMBER] ... To collect from local files containing flows created by flowcap(8): rwflowpack --input-mode=fcfiles --incoming-directory=DIR_PATH --sensor-configuration=FILE_PATH [--packing-logic=PLUGIN] [--polling-interval=NUMBER] ... To collect from a single file containing NetFlow v5 PDUs: rwflowpack --input-mode=pdufile --netflow-file=FILE_PATH --sensor-configuration=FILE_PATH [--packing-logic=PLUGIN] [--sensor-name=SENSOR] ... To respool SiLK Flows without modifying the class, type, or sensor: rwflowpack --input-mode=respool --incoming-directory=DIR_PATH [--polling-interval=NUMBER] ... To store the SiLK Flow files on the local machine (default): rwflowpack ... [--output-mode=local-storage] --root-directory=DIR_PATH ... To create incremental files to be processed by rwflowappend(8): rwflowpack ... --output-mode=incremental-files --incremental-directory=DIR_PATH ... To create incremental files to be processed by rwflowappend (deprecated): rwflowpack ... --output-mode=sending --sender-directory=DIR_PATH --incremental-directory=DIR_PATH ... Help options: rwflowpack --sensor-configuration=FILE_PATH [--packing-logic=PLUGIN] { --verify-sensor-config | --verify-sensor-config=VERBOSE } rwflowpack --help rwflowpack --version DESCRIPTION rwflowpack is a daemon that runs as part of the SiLK flow collection and packing tool-chain. The primary job of rwflowpack is to convert each in- coming flow record to the SiLK Flow format, categorize each incoming flow record (e.g., as incoming or outgoing), set the sensor value for the record, and determine which hourly file will ultimately store the record. The settings that rwflowpack uses to categorize each flow record are deter- mined by two textual configuration files and compiled code that is referred to as packing logic. The first of the configuration files is silk.conf(5) which specifies the classes, types, and sensors to use at your site. There are several differ- ent ways to specify the location of this file as detailed in the "FILES" section below. The second configuration file is the sensor.conf(5) file, whose location is specified via the --sensor-configuration switch. This file contains multi- ple sensor blocks, where each block contains information used to categorize flow records captured at that particular sensor. This file also contains probe blocks which specify how to collect NetFlow v5, IPFIX, and/or NetFlow v9 flow records, and a mapping of probes to sensors. See the sen- sor.conf(5) manual page for details. The packing logic uses the information from the silk.conf and sensor.conf files to categorize each flow record. By categorizing a flow record, we mean determine whether the record is incoming or outgoing and assign a class/type pair (also called a flowtype) to the record. The flowtype along with the starting hour of the record and the sensor where the record was collected form a triple which determines into which file a flow record is stored. The files that rwflowpack produces have the form flowType-sensor- Name_YYYYMMDD.HH where flowType encodes the class/type pair, sensorName is the sensor where the flows were collected, and YYYYMMDD.HH is the year, month, day, and hour when the flow started. For more information on how rwflowpack categorizes each flow record and converts data to the SiLK Flow format, see the SiLK Installation Handbook, the sensor.conf(5) manual page, and the manual page for the packing logic plug-in that rwflowpack is using (e.g., packlogic-twoway(3) is the default, packlogic-generic(3)). The compiled code for the packing-logic is normally loaded into rwflowpack as a run-time. (The administrator may choose to compile the packing logic into rwflowpack when building SiLK. See the SiLK Installation Handbook for details.) The name of this plug-in is specified in the packing-logic statement in the silk.conf file. A different location may be specified us- ing the --packing-logic switch. Input Modes There are several ways to input data to rwflowpack, and the method to use is determined by the --input-mode switch, with "stream" being the default when the switch is not provided. "stream" In "stream" input-mode, rwflowpack processes the probe statements in the sensor.conf(5) file which specify how rwflowpack is to capture data from one or more sources. The data is assigned to a sensor based on the probe-sensor mapping in the sensor.conf file. rwflowpack then cat- egorizes the records, converts them to the SiLK Flow format, and writes them to files. The sources of data that rwflowpack supports are: * listening for NetFlow v5 packets on a UDP socket * listening for IPFIX (Internet Protocol Flow Information eXport) packets on a TCP or a UDP socket * listening for NetFlow v9 packets on a UDP socket * listening for sFlow v5 packets on a UDP socket * polling a directory for files containing NetFlow v5 records (see the description of the "pdufile" input-mode for the required format of these files) * polling a directory for files containing IPFIX records (as gener- ated by yaf(1)) * polling a directory for files containing SiLK Flow records (compare to the "respool" input-mode) Multiple sources may be specified. Processing of IPFIX, NetFlow v9, or sFlow is only available when SiLK is compiled with support for libfixbuf-1.7.0 or later. Processing of sFlow records was added in SiLK 3.9.0. libfixbuf is available from <http://tools.netsa.cert.org/fixbuf/>. Configuration of "stream" input-mode is specified in the "Stream Col- lection Switches (--input-mode=stream)" section below. "fcfiles" Instead of having rwflowpack capture data itself, you may configure rwflowpack to work in conjunction with one or more flowcap(8) daemons by specifying "fcfiles" as the input-mode. In this configuration, each flowcap uses the probe statements in the sensor.conf(5) file to determine how to collect the data. flowcap sup- ports the network-based capture methods specified for the "stream" in- put-mode---i.e., flowcap does not support directory polling. flowcap writes the data it captures into small files and includes the probe name in each file's header. Typically, the flowcap processes run on separate machines near the router or flow meter that is generating the records. The rwsender(8) and rwreceiver(8) daemons are often used to move the files from the flowcap machines to the rwflowpack machine. rwflowpack polls a directory for the files created by flowcap. Once it finds a file, it assigns those records a sensor based on the probe-sen- sor mapping in the sensor.conf file, it categorizes the records, and it writes the records to one or more output files. Since rwflowpack does not capture data in "fcfiles" input-mode, rwflow- pack does not use the probe statements in the sensor.conf file, and the statements do not need to be present. The switches that rwflowpack uses in "fcfiles" input-mode are given be- low ("Flowcap Files Collection Switches (--input-mode=fcfiles)"). "pdufile" Setting the input-mode to "pdufile" tells rwflowpack to read a single file containing NetFlow v5 PDU records and then exit. rwflowpack does not become a daemon in this input-mode; instead it remains in the fore- ground, processes the NetFlow file, and exits. The file must be in the format created by NetFlow Collector: The file's size must be an integer multiple of 1464, where each 1464 byte chunk contains a 24-byte NetFlow v5 header and space for thirty 48-byte Net- Flow records. The number of valid records per chunk is specified in the chunk's header. (This is also the format that rwflowpack requires in "stream" input-mode when it is polling a directory for NetFlow v5 files.) To convert single PDU file to a stream of SiLK Flow records, consider using rwpdu2silk(1). In "pdufile" input-mode, the sensor.conf file must define a sensor that maps to a probe that uses the read-from-file statement. However, the argument to that statement is ignored, and the argument is typically set to /dev/null. The NetFlow v5 file's location must be specified with the --netflow-file switch. If sensor.conf contains multiple sen- sor blocks, the --sensor-name switch is required to tell rwflowpack which sensor to use. See the "PDU File Collection Switches (--input-mode=pdufile)" section below for additional configuration information. "respool" Sometimes it is desirable to pull existing SiLK Flow records from one data repository and use them to create a "mini" data repository (for example, for testing). The "respool" input-mode is one way to accom- plish this. In this configuration, rwflowpack polls a directory for SiLK flow files and it uses the existing class/type pair and sensor values to determine where to store the flow record. That is, rwflowpack puts the data into appropriate hourly file, but it does not change any other settings on the flow records. To contrast "respool" input-mode with rwflowpack processing SiLK Flow files in "stream" input-mode: In "respool" input-mode, the existing class, type, and sensor values are used to determine each record's des- tination. In the latter, rwflowpack treats the records as it would any other newly arrived data, assigning the data to a sensor and re-catego- rizing the records to assign a class/type pair to them. Since no categorization occurs in "respool" input-mode, the --sen- sor-configuration and --packing-logic switches are not required and not allowed, and their presence causes rwflowpack to exit with an error code. Output Modes As mentioned above, after rwflowpack categorizes a flow record (that is, determines the sensor, class/type, and starting hour for the record), it uses those values to generate the name of the hourly file that will contain that record, and it writes the record to that file. In order for the record in that file to be retrievable by rwfilter(1), the file must be stored in a SiLK data repository, which is a directory tree of binary SiLK Flow files. The root of this directory tree is called the SILK_DATA_ROOTDIR. The structure of the tree under the root is determined by the path-format setting in the silk.conf(5) file. There are two ways to get the files into the SILK_DATA_ROOTDIR; which method is used is determined by rwflowpack's --output-mode switch. This switch supports the following values: "local-storage" In "local-storage" output-mode, rwflowpack creates the hourly SiLK Flow files directly in the data repository, and it writes the records into these files. rwflowpack uses "local-storage" output-mode when the --output-mode switch is not provided. "incremental-files" When the output-mode is "incremental-files", rwflowpack does not create hourly data files directly. Instead, rwflowpack creates smaller files (called incremental files), and rwflowpack relies on the rwflowap- pend(8) daemon to combine the incremental files into hourly files in the final data repository. In "incremental-files" output-mode, rwflowpack uses a single destina- tion directory whose location is specified by the --incremental-direc- tory switch. In this directory, rwflowpack creates a zero-byte place holder file and a corresponding working file. The name of the place holder file has a unique, random suffix, and the name of the working file is a dot (".") followed by the name of the place holder file. rwflowpack first writes records to the working files. Periodically (as determined by the value specified to --flush-timeout), rwflowpack closes all the working files and moves them over the place holder files. Once rwflowpack has closed and moved a working file, it no longer takes responsibility for it. The rwflowappend process may poll the incremental-directory itself, or another process may poll that directory and pass the files to rwflowap- pend. If rwflowpack and rwflowappend are on different machines, an rwsender(8)/rwreceiver(8) pair may be used to move the files between the machines. Additional reasons one may want to use "incremental-files" output-mode are to support having multiple data repositories or to allow additional processing of the SiLK Flow records (such as by the Analysis Pipeline (<http://tools.netsa.cert.org/analysis-pipeline/>)). Note: This output-mode was introduced in SiLK 3.6.0. This mode is sim- ilar to the "sending" output-mode and is meant to replace it. In "in- cremental-files" output-mode, all writing occurs within the incremen- tal-directory, while the "sending" output-mode uses two directories. "sending" This output-mode works similarly to "incremental-files". The differ- ence is that when rwflowpack flushes the open files, it moves the files from the incremental-directory and to the directory specified by the --sender-directory switch. Once a file is moved to the sender-direc- tory, rwflowpack no longer takes responsibility for it. As of SiLK 3.6.0, the "sending" output-mode is deprecated. OPTIONS Option names may be abbreviated if the abbreviation is unique or is an ex- act match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters. Input and Output Mode As described in the "Input Modes" section above, rwflowpack has multiple ways of getting data, and that data may be stored in one of two methods (c.f., "Output Modes"). Choosing the modes and configuring each mode are described in the sections below. --input-mode=MODE Determine how rwflowpack gathers data. The default input MODE is "stream". The available input-modes are "stream", "fcfiles", "pdu- file", and "respool". --output-mode=MODE Determine what rwflowpack does with the data as it is packed into SiLK binary files. The default output MODE is "local-storage". The avail- able output-modes are "local-storage", "incremental-files" and "send- ing". Stream Collection Switches (--input-mode=stream) In "stream" input-mode, rwflowpack uses the probe statements in the sen- sor.conf file to capture data, and then rwflowpack categorizes the data. The "stream" input-mode is the default when the --input-mode switch is not provided. This input-mode accepts the following switches; the --sen- sor-configuration switch is required, and all other switches are optional. --sensor-configuration=FILE_PATH Give the path to the configuration file that specifies how rwflowpack is to capture data and that tells rwflowpack whether a record repre- sents an incoming or outgoing flow. The complete syntax of the config- uration file is described in the sensor.conf(5) manual page; see also the SiLK Installation Handbook. --packing-logic=PLUGIN Specify the plug-in that rwflowpack should load, where the plug-in pro- vides functions that determine into which class and type each flow record will be categorized and the format of the files that rwflowpack will write. When SiLK has been configured with hard-coded packing logic (i.e., when --enable-packing-logic was specified to the configure script), this switch will not be present on rwflowpack. A default value for this switch may be specified in the silk.conf(5) site config- uration file (see the description of the --site-config-file switch). When PLUGIN does not contain a slash ("/"), rwflowpack attempts to find a file named PLUGIN in the directories listed in the "FILES" section. If rwflowpack finds the file, it uses that path. If PLUGIN contains a slash or if rwflowpack does not find the file, rwflowpack relies on your operating system's dlopen(3) call to find the file. When the SILK_PLUGIN_DEBUG environment variable is non-empty, rwflowpack prints status messages to the standard error as it attempts to find and open each of its plug-ins. rwflowpack exits if it does not have access to the packing logic functions. --sensor-name=SENSOR Cause rwflowpack to ignore all probes in the sensor configuration file except the probes for SENSOR. Only data for SENSOR is collected. This allows a common configuration file to be used by multiple rwflowpack invocations, yet also allow each rwflowpack instance only collect data for a single sensor. There must be a sensor definition for SENSOR in the configuration file. When this switch is not present, rwflowpack collects and packs data for all sensors. --polling-interval=NUMBER Specify the number of seconds rwflowpack waits between scans of the di- rectories specified by the "poll-directory" settings of the probes in the sensor.conf file. The default is 15 seconds. Flowcap Files Collection Switches (--input-mode=fcfiles) As described in the "Input Modes" section above, in "fcfiles" input-mode, rwflowpack processes files created by the flowcap(8) daemon. In "fcfiles" input-mode, the --sensor-configuration and --incoming-directory switches are required. --sensor-configuration=FILE_PATH Give the path to the configuration file that rwflowpack consults to de- termine whether a record represents an incoming or outgoing flow. The complete syntax of the configuration file is described in the sen- sor.conf(5) manual page; see also the SiLK Installation Handbook. --incoming-directory=DIR_PATH Periodically scan the directory DIR_PATH for files that have been cre- ated by flowcap. As rwflowpack scans DIR_PATH, it ignores a file if its name begins with a dot (".") or if its size is 0 bytes. When a file is first detected, rwflowpack records its size, and the file must have the same size for two consecutive scans before rwflowpack processes it. After the file is successfully processed, rwflowpack ei- ther moves it to the archive-directory or deletes it. The interval be- tween scans is set by --polling-interval. DIR_PATH must be a complete directory path. --packing-logic=PLUGIN Specify the plug-in that rwflowpack should load for the packing logic. For more detail, see the description above. --polling-interval=NUMBER Specify the number of seconds rwflowpack waits between polls of the in- coming-directory for new files created by flowcap. If not given, the default value is 15 seconds. PDU File Collection Switches (--input-mode=pdufile) In this input-mode, rwflowpack stays in the foreground, processes a single file of NetFlow v5 data, and exits. The --sensor-configuration and --net- flow-file switches are required. The --sensor-name switch is also required when the sensor.conf file contains more than one sensor. The following switches are available in "pdufile" input-mode: --sensor-configuration=FILE_PATH Give the path to the configuration file that rwflowpack consults to de- termine whether a record represents an incoming or outgoing flow. --netflow-file=FILE_PATH Name the full path of the file from which rwflowpack reads NetFlow v5 PDUs. This switch is required in PDU File input-mode. --sensor-name=SENSOR Cause rwflowpack to ignore all probes in the sensor configuration file except the probes for SENSOR. There must be a sensor definition for SENSOR in the configuration file. This switch is required in this in- put-mode unless the sensor.conf file only defines a single sensor. --packing-logic=PLUGIN Specify the plug-in that rwflowpack should load for the packing logic. For more detail, see the description of this switch in the "stream" in- put-mode. Respooling Switches (--input-mode=respool) When the --input-mode="respool" switch is provided, rwflowpack polls a di- rectory for SiLK Flow files, and writes the records it finds into new hourly files, leaving the sensor and class/type values unchanged in the records. (See "Input Modes" above for additional details.) The first of the following switches is required: --incoming-directory=DIR_PATH Periodically scan the directory DIR_PATH for SiLK Flow files to process. As rwflowpack scans DIR_PATH, it ignores a file if its name begins with a dot (".") or if its size is 0 bytes. When a file is first detected, rwflowpack records its size, and the file must have the same size for two consecutive scans before rwflowpack will process it. After the file is successfully processed, rwflowpack either moves it to the archive-directory or deletes it. The interval between scans is set by --polling-interval. DIR_PATH must be a complete directory path. --polling-interval=NUMBER Specify the number of seconds rwflowpack waits between polls of the in- coming-directory. If not given, the default value is 15 seconds. Local Storage Switches (--output-mode=local-storage) In "local-storage" output-mode, rwflowpack stores SiLK Flow records di- rectly in the data repository. This is the default when the --output-mode switch is not provided. When operating in this output-mode, the following switch is required: --root-directory=DIR_PATH Name the full path of the SILK_DATA_ROOTDIR; that is, the directory un- der which the files containing the packed SiLK Flow records are stored. rwflowpack creates subdirectories below DIR_PATH based on the data re- ceived and the path-format setting in the silk.conf(5) file. Incremental-Files Switches (--output-mode=incremental-files) As described in the "Output Modes" section above, the "incremental-files" output-mode tells rwflowpack to write incremental-files. The rwflowap- pend(8) daemon combines these incremental-files to create a SiLK data repository. When running in "incremental-files" output-mode, the following switch must be provided: --incremental-directory=DIR_PATH Name the full path of the destination directory where incremental-files are both created and stored to await action by another process such as rwflowappend or rwsender. It is recommended that only a single rwflow- pack process write to this directory. Sending Mode Switches (--output-mode=sending) This output-mode is deprecated as of SiLK 3.6.0. This output-mode works similarly to the "incremental-files" output-mode, except rwflowpack moves the files to a second directory periodically. Both the following switches are required in "sending" output-mode: --incremental-directory=DIR_PATH Name the full path of the working directory under which incremen- tal-files are initially created. Periodically (as determined by the --flush-timeout switch), rwflowpack closes the files in this directory and moves them to the sender-directory. An rwflowpack invocation as- sumes it has full control over the files in this directory. When rwflowpack starts, any files in this directory are moved to the sender-directory. --sender-directory=DIR_PATH Name the full path of the destination directory where incremental-files are moved to await action by another process such as rwflowappend or rwsender. Once rwflowpack moves files to this directory, it no longer takes responsibility for them. The other process (e.g., rwsender) is responsible for removing files from this directory. Multiple rwflow- pack invocations may use a single sender-directory. General Configuration The following switches are optional: --no-file-locking Do not use advisory write locks. Normally, rwflowpack obtains a write lock on an data file prior to writing records to it. The write lock prevents two instances of rwflowpack from writing to the same data file simultaneously. However, attempting to use a write lock on some file systems causes rwflowpack to exit with an error, and this switch may be used when writing data to these file systems. --flush-timeout=VAL Set the timeout for flushing any in-memory records to disk to VAL sec- onds. If not specified, the default is 2 minutes (120 seconds). When the output-mode is "local-storage", this value specifies how often the files are flushed to disk to ensure that any records in memory are written to disk. When using the "incremental-files" or "sending" out- put-mode, this value specifies how often to close and move the incre- mental files. See the "Output Modes" section for details. --file-cache-size=VAL Set the maximum number of data files to have open for writing at any one time to VAL. If not specified, the default is 128 files. The min- imum file cache size is 4. This switch also determines how many files rwflowpack reads from simultaneously when using probes that poll direc- tories for files (see sensor.conf(5)). The maximum number of input files open at any one time is limited to one eighth of VAL (with a min- imum of 2), and the number of directory polling operations to perform simultaneously is limited to one sixteenth of VAL (minimum is 1). --pack-interfaces Allow one to override the default file output formats of the packed SiLK Flow files that rwflowpack writes. When this switch is present, rwflowpack writes additional information into the packed files: the router's SNMP input and output interfaces and the next-hop IP address. (When the sensor.conf file contains an "interface-values" attribute whose value is "vlan", the input and output fields contain the vlan IDs instead of SNMP interface values.) The extra data produced by this switch is useful for determining why traffic is being stored in certain files. Note that this switch only affects newly created files. New records are always appended to an existing file in the file's current output format to maintain file integrity. --byte-order=ENDIAN Set the byte order for newly created SiLK Flow files. When appending records to an existing file, the byte order of the file is maintained. The argument is one of the following: "native" Use the byte order of the machine where rwflowpack is running. This is the default. "big" Use network byte order (big endian) for the flow files. "little" Write the flow files in little endian format. --compression-method=COMP_METHOD Specify the compression library to use when creating new files. When this switch is not given, newly created files are compressed using the default chosen when SiLK was compiled. When appending records to an existing file, the compression method of the file is maintained. The valid values for COMP_METHOD are determined by which external libraries were found when SiLK was compiled. To see the available compression methods and the default method, use the --help or --version switch. SiLK can support the following COMP_METHOD values when the required li- braries are available. noneDo not compress the output using an external library. zlibUse the zlib(3) library for compressing the output. Using zlib produces the smallest output files at the cost of speed. lzo1x Use the lzo1x algorithm from the LZO real time compression library for compression. This compression provides good compression with less memory and CPU overhead. snappy Use the snappy library for compression, and always compress the output regardless of the destination. This compression provides good compression with less memory and CPU overhead. Since SiLK 3.13.0. bestUse lzo1x if available, otherwise use snappy if available, other- wise use zlib if available. --site-config-file=FILENAME Read the SiLK site configuration from the named file FILENAME. When this switch is not provided, rwflowpack searches for the site configu- ration file in the locations specified in the "FILES" section. Disposal of Input Flow Files The following switches determine how rwflowpack handles input files once it has processed them. These switches have no effect when rwflowpack reads all of its data directly from the network. Otherwise, the switches affect the named --netflow-file in "pdufile" input-mode, the files read from the --incoming-directory in "fcfiles" and "respool" input-mode, and files read from probes that have a "poll-directory" statement (c.f. sensor.conf(5)) in "stream" input-mode. --error-directory=DIR_PATH Move input files that cannot be opened, have an unexpected format, con- tain an unrecognized probe name in "fcfiles" input-mode, or are not successfully processed into the directory DIR_PATH. DIR_PATH must be a complete directory path. If this switch is not provided, problem files remain in place and cause rwflowpack to exit. --archive-directory=DIR_PATH Move input files that rwflowpack processes successfully into the direc- tory DIR_PATH. DIR_PATH must be a complete directory path. When this switch is not provided and the input-mode is "pdufile", the original NetFlow source file is not modified, moved, or deleted. In all other input-modes, no --archive-directory switch causes rwflowpack to delete each input file after successfully processing it. When the --flat-archive switch is also provided, incoming files are moved into the top of DIR_PATH; when --flat-archive is not given, each file is moved to a subdirectory based on the current UTC time: DIR_PATH/YEAR/MONTH/DAY/HOUR/. Removing files from the archive-direc- tory is not the job of rwflowpack; the system administrator should im- plement a separate process to clean this directory. This switch is re- quired when the --post-archive-command switch is present. --flat-archive When archiving input files via the --archive-directory switch, move the files into the top of the archive-directory, not into subdirectories of the archive-directory. This switch has no effect if --archive-direc- tory is not also specified. This switch may be used to allow another process to watch for new files appearing in the archive-directory. --post-archive-command=COMMAND Run COMMAND on each input file after rwflowpack has successfully processed the file and moved the file into the archive-directory. Each occurrence of the string %s in COMMAND is replaced with the full path to the input file in the archive-directory, and each occurrence of "%%" is replaced with "%". If any other character follows "%", rwflowpack exits with an error. When using this feature, the --archive-directory switch must be specified. See also the rwpollexec(8) daemon. Logging and Daemon Configuration One of the following mutually-exclusive switches is required: --log-destination=DESTINATION Specify the destination where logging messages are written. When DES- TINATION begins with a slash "/", it is treated as a file system path and all log messages are written to that file; there is no log rota- tion. When DESTINATION does not begin with "/", it must be one of the following strings: "none" Messages are not written anywhere. "stdout" Messages are written to the standard output. "stderr" Messages are written to the standard error. "syslog" Messages are written using the syslog(3) facility. "both" Messages are written to the syslog facility and to the standard er- ror (this option is not available on all platforms). --log-directory=DIR_PATH Use DIR_PATH as the directory where the log files are written. DIR_PATH must be a complete directory path. The log files have the form DIR_PATH/LOG_BASENAME-YYYYMMDD.log where YYYYMMDD is the current date and LOG_BASENAME is the application name or the value passed to the --log-basename switch when provided. The log files are rotated: At midnight local time, a new log is opened, the previous file is closed, and the command specified by --log-post-rotate is invoked on the previous day's log file. (Old log files are not removed by rwflowpack; the administrator should use an- other tool to remove them.) When this switch is provided, a process-ID file (PID) is also written in this directory unless the --pidfile switch is provided. --log-pathname=FILE_PATH Use FILE_PATH as the complete path to the log file. The log file is not rotated. The following set of switches is optional: --log-level=LEVEL Set the severity of messages that are logged. The levels from most se- vere to least are: "emerg", "alert", "crit", "err", "warning", "no- tice", "info", "debug". The default is "info". --log-sysfacility=NUMBER Set the facility that syslog(3) uses for logging messages. This switch takes a number as an argument. The default is a value that corresponds to "LOG_USER" on the system where rwflowpack is running. This switch produces an error unless --log-destination=syslog is specified. --log-basename=LOG_BASENAME Use LOG_BASENAME in place of the application name in the name of log files in the log directory. See the description of the --log-directory switch. This switch does not affect the name of the process-ID file. --log-post-rotate=COMMAND Run COMMAND on the previous day's log file after log rotation. When this switch is not specified, the previous day's log file is compressed with gzip(1). When the switch is specified and COMMAND is the empty string, no action is taken on the log file. Each occurrence of the string %s in COMMAND is replaced with the full path to the log file, and each occurrence of "%%" is replaced with "%". If any other charac- ter follows "%", rwflowpack exits with an error. Specifying this switch without also using --log-directory is an error. --pidfile=FILE_PATH Set the complete path to the file in which rwflowpack writes its process ID (PID) when it is running as a daemon. No PID file is writ- ten when --no-daemon is given. When this switch is not present, no PID file is written unless the --log-directory switch is specified, in which case the PID is written to LOGPATH/rwflowpack.pid. --no-chdir Do not change directory to the root directory. When rwflowpack becomes a daemon process, it changes its current directory to the root direc- tory so as to avoid potentially running on a mounted file system. Specifying --no-chdir prevents this behavior, which may be useful dur- ing debugging. The application does not change its directory when --no-daemon is given. --no-daemon Force rwflowpack to run in the foreground---it does not become a daemon process. This may be useful during debugging. Help Options --verify-sensor-config --verify-sensor-config=VERBOSE Verify that the syntax of the sensor configuration file is correct and then exit rwflowpack. If the file is incorrect or if it does not de- fine any sensors, an error message is printed and rwflowpack exits ab- normally. If the file is correct and no argument is provided to the --verify-sensor-config switch, rwflowpack simply exits with status 0. If an argument (other than the empty string and 0) is provided to the switch, the names of the probes and sensors found in the sensor config- uration file are printed to the standard output, and then rwflowpack exits. --help Print the available options and exit. --version Print the version number and information about how SiLK was configured, then exit the application. ENVIRONMENT SILK_IPFIX_PRINT_TEMPLATES When set to 1, rwflowpack writes messages to the log file describing each IPFIX and NetFlow v9 template it receives. This is equivalent to adding "show-templates" to the "log-flags" setting for each probe in the sensor.conf file. See the sensor.conf(5) manual page for the for- mat of these messages. Since SiLK 3.8.2. SILK_LIBFIXBUF_SUPPRESS_WARNINGS When set to 1, rwflowpack disables all warning messages generated by libfixbuf. These warning messages include out-of-sequence packets, data records not having a corresponding template, record count discrep- ancies, and issues decoding list elements. Since SiLK 3.10.0. SILK_CONFIG_FILE This environment variable is used as the value for the --site-con- fig-file when that switch is not provided. SILK_DATA_ROOTDIR This environment variable specifies the root directory of data reposi- tory. When the output-mode is "sending", rwflowpack may use this envi- ronment variable when searching for the SiLK site configuration file. See the "FILES" section for details. SILK_PATH This environment variable gives the root of the install tree. When searching for configuration files, rwflowpack may use this environment variable. See the "FILES" section for details. SILK_PLUGIN_DEBUG When set to 1, rwflowpack print status messages to the standard error as it tries to open the packing logic plug-in. FILES sensor.conf The location of this file must be specified by the --sensor-configura- tion switch. This file specifies probe blocks that tell rwflowpack how to capture data when the --input-mode is "stream". The file also con- tains sensor blocks that map probes to sensors and that the pack- ing-logic uses to determine the category of each flow record. The syn- tax of this file is described in the sensor.conf(5) manual page. ${SILK_CONFIG_FILE} ROOT_DIRECTORY/silk.conf ${SILK_PATH}/share/silk/silk.conf ${SILK_PATH}/share/silk.conf /usr/local/share/silk/silk.conf /usr/local/share/silk.conf Possible locations for the SiLK site configuration file which are checked when the --site-config-file switch is not provided. When rwflowpack is running in "local-storage" output-mode, ROOT_DIRECTORY/ is the directory specified to the --root-directory switch. When the output-mode is "sending", ROOT_DIRECTORY/ is either the value specified in the SILK_DATA_ROOTDIR environment variable or the default data repository directory compiled into rwflowpack (/data). ${SILK_PATH}/lib64/silk/ ${SILK_PATH}/lib64/ ${SILK_PATH}/lib/silk/ ${SILK_PATH}/lib/ /usr/local/lib64/silk/ /usr/local/lib64/ /usr/local/lib/silk/ /usr/local/lib/ Directories that rwflowpack checks when attempting to load the pack- ing-logic plug-in. SEE ALSO SiLK Installation Handbook, sensor.conf(5), silk.conf(5), pack- logic-twoway(3), packlogic-generic(3), flowcap(8), rwfilter(1), rwflowap- pend(8), rwreceiver(8), rwsender(8), rwpollexec(8), rwpdu2silk(1), rw- packchecker(8), silk(7), gzip(1), yaf(1), dlopen(3), zlib(3), syslog(3) NOTES As SiLK 3.6.0, the "incremental-files" output-mode should be used in place of the "sending" output-mode that existed in prior releases of rwflowpack. See "Output Modes" for details. For administrators that use the "sending" output-mode in SiLK 3.5 or older and upgrade to SiLK 3.6 or later: Any incremental files that the older ver- sion of rwflowpack leaves in the incremental-directory will not be moved to the sender-directory by the new version of rwflowpack. Those files will need to be moved by hand. rwflowpack does not check the integrity of the data file before appending records to it. When the disk becomes full or other write errors occur, rwflowpack may leave partially written records or partially written compressed blocks in the files it has open. For each file where a partially written compressed block remains and additional compressed blocks are appended, the newly ap- pended compressed blocks are not readable by other SiLK tools. For each file where a partially written record remains and additional records are appended, other SiLK tools will read the unaligned data as if it were aligned and produce garbage records. Partially writes may also occur if rwflowpack is suddenly killed (e.g., by "kill -9"). When a write error occurs, rwflowpack may leave a zero byte file in the data repository. Such files do affect the exit status of rwfilter(1), though rwfilter warns about being unable to read the header from the file. rwflowpack obtains an advisory write lock on the hourly file it is writing, allowing multiple rwflowpack processes to write to the same hourly file. File locking may be disabled by using the --no-file-locking switch. If this switch is enabled, the administrator must ensure that multiple rwflow- pack processes do not attempt to write to the same hourly file simultane- ously. SiLK 3.22.2 2026-08-27 rwflowpack(8)
NAME | SYNOPSIS | DESCRIPTION | OPTIONS | ENVIRONMENT | FILES | SEE ALSO | NOTES
Want to link to this manual page? Use this URL:
<https://man.freebsd.org/cgi/man.cgi?query=rwflowpack&sektion=8&manpath=FreeBSD+Ports+15.1.quarterly>