rwset(1) SiLK Tool Suite rwset(1) NAME rwset - Generate binary IPset files of unique IP addresses SYNOPSIS rwset {--sip-file=FILE | --dip-file=FILE | --nhip-file=FILE | --any-file=FILE [...]} [--record-version=VERSION] [--invocation-strip] [--note-strip] [--note-add=TEXT] [--note-file-add=FILE] [--print-filenames] [--copy-input=PATH] [--compression-method=COMP_METHOD] [--ipv6-policy={ignore,asv4,mix,force,only}] [--site-config-file=FILENAME] {[--xargs] | [--xargs=FILENAME] | [FILE [FILE ...]]} rwset --help rwset --version DESCRIPTION rwset reads SiLK Flow records and generates one to four binary IPset file(s). In a single pass, rwset can create one of each type of its possi- ble outputs, which are IPset files containing: * the unique source IP addresses * the unique destination IP addresses * the unique next-hop IP addresses * the unique source and destination IP addresses The output files must not exist prior to invoking rwset. To write an IPset file to the standard output, specify "stdout" or "-" as the output file name. rwset will complain if you attempt to write the IPset to the stan- dard output and standard output is connected to the terminal. Only one IPset file may be written to the standard output. rwset reads SiLK Flow records from the files named on the command line or from the standard input when no file names are specified and --xargs is not present. To read the standard input in addition to the named files, use "-" or "stdin" as a file name. If an input file name ends in ".gz", the file is uncompressed as it is read. When the --xargs switch is provided, rwset reads the names of the files to process from the named text file or from the standard input if no file name argument is provided to the switch. The input to --xargs must contain one file name per line. IPset files are in a binary format that efficiently stores a set of IP ad- dresses. The file only stores the presence of an IP address; no volume in- formation (such as a count of the number of times the IP address occurs) is maintained. To store volume information, use rwbag(1). Use rwsetcat(1) to see the IP addresses in a binary IPset file. To create a binary IPset file from a list of IP addresses, use rwsetbuild(1). rwset- tool(1) allows you to perform set operations on binary IPset files. To de- termine if an IP address is a member of a binary IPset, use rwsetmember(1). To list the IPs that appear in the SiLK Flow file flows.rw, the command $ rwset --sip-file=stdout flows.rw | rwsetcat is faster than rwuniq(1), but rwset does not report the number of flow records or compute byte and packets counts. OPTIONS Option names may be abbreviated if the abbreviation is unique or is an ex- act match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters. At least one of the following output switches is required; multiple output switches can be given, but an output switch cannot be repeated. --sip-file=FILE Store the unique source IP addresses in the binary IPset file FILE. rwset will write the IPset file to the standard output when FILE is "stdout" or "-" and the standard output is not a terminal. --dip-file=FILE Store the unique destination IP addresses in the binary IPset file FILE. rwset will write the IPset file to the standard output when FILE is "stdout" or "-" and the standard output is not a terminal. --nhip-file=FILE Store the unique next-hop IP addresses in the binary IPset file FILE. rwset will write the IPset file to the standard output when FILE is "stdout" and the standard output is not a terminal. --any-file=FILE Store the unique source and destination IP addresses in the binary IPset file FILE. rwset will write the IPset file to the standard out- put when FILE is "stdout" or "-" and the standard output is not a ter- minal. Only one of the above switches my use "stdout" as the name of the file. rwset supports these additional switches: --record-version=VERSION Specify the format of the IPset records that are written to the output. VERSION may be 2, 3, 4, 5 or the special value 0. When the switch is not provided, the SILK_IPSET_RECORD_VERSION environment variable is checked for a version. The default version is 0. 0 Use the default version for an IPv4 IPset and an IPv6 IPset. Use the --help switch to see the versions used for your SiLK installa- tion. 2 Create a file that may hold only IPv4 addresses and is readable by all versions of SiLK. 3 Create a file that may hold IPv4 or IPv6 addresses and is readable by SiLK 3.0 and later. 4 Create a file that may hold IPv4 or IPv6 addresses and is readable by SiLK 3.7 and later. These files are more compact that version 3 and often more compact than version 2. 5 Create a file that may hold only IPv6 addresses and is readable by SiLK 3.14 and later. When this version is specified, IPsets con- taining only IPv4 addresses are written in version 4. These files are usually more compact that version 4. --invocation-strip Do not record any command line history: do not copy the invocation his- tory from the input files to the output file, and do not record the current command line invocation in the output. The invocation may be viewed with rwfileinfo(1). --note-strip Do not copy the notes (annotations) from the input files to the output file. Normally notes from the input files are copied to the output. --note-add=TEXT Add the specified TEXT to the header of every output file as an annota- tion. This switch may be repeated to add multiple annotations to a file. To view the annotations, use the rwfileinfo(1) tool. --note-file-add=FILENAME Open FILENAME and add the contents of that file to the header of every output file as an annotation. This switch may be repeated to add mul- tiple annotations. Currently the application makes no effort to ensure that FILENAME contains text; be careful that you do not attempt to add a SiLK data file as an annotation. --print-filenames Print to the standard error the names of input files as they are opened. --copy-input=PATH Copy all binary SiLK Flow records read as input to the specified file or named pipe. PATH may be "stdout" or "-" to write flows to the stan- dard output as long as no IPset file is being written there. --ipv6-policy=POLICY Determine how IPv4 and IPv6 flows are handled when SiLK has been com- piled with IPv6 support. When the switch is not provided, the SILK_IPV6_POLICY environment variable is checked for a policy. If it is also unset or contains an invalid policy, the POLICY is mix. When SiLK has not been compiled with IPv6 support, IPv6 flows are always ig- nored, regardless of the value passed to this switch or in the SILK_IPV6_POLICY variable. The supported values for POLICY are: ignore Ignore any flow record marked as IPv6, regardless of the IP ad- dresses it contains. Only IP addresses contained in IPv4 flow records will be added to the IPset(s). asv4Convert IPv6 flow records that contain addresses in the ::ffff:0:0/96 netblock (that is, IPv4-mapped IPv6 addresses) to IPv4 and ignore all other IPv6 flow records. mix Process the input as a mixture of IPv4 and IPv6 flow records. When the input contains IPv6 addresses outside of the ::ffff:0:0/96 net- block, this policy is equivalent to force; otherwise it is equiva- lent to asv4. force Convert IPv4 flow records to IPv6, mapping the IPv4 addresses into the ::ffff:0:0/96 netblock. onlyProcess only flow records that are marked as IPv6. Only IP ad- dresses contained in IPv6 flow records will be added to the IPset(s). Regardless of the IPv6 policy, when all IPv6 addresses in the IPset are in the ::ffff:0:0/96 netblock, rwset treats them as IPv4 addresses and writes an IPv4 IPset. When any other IPv6 addresses are present in the IPset, the IPv4 addresses in the IPset are mapped into the ::ffff:0:0/96 netblock and rwset writes an IPv6 IPset. --compression-method=COMP_METHOD Specify the compression library to use when writing output files. If this switch is not given, the value in the SILK_COMPRESSION_METHOD en- vironment variable is used if the value names an available compression method. When no compression method is specified, output to the stan- dard output or to named pipes is not compressed, and output to files is compressed using the default chosen when SiLK was compiled. The valid values for COMP_METHOD are determined by which external libraries were found when SiLK was compiled. To see the available compression methods and the default method, use the --help or --version switch. SiLK can support the following COMP_METHOD values when the required libraries are available. noneDo not compress the output using an external library. zlibUse the zlib(3) library for compressing the output, and always com- press the output regardless of the destination. Using zlib pro- duces the smallest output files at the cost of speed. lzo1x Use the lzo1x algorithm from the LZO real time compression library for compression, and always compress the output regardless of the destination. This compression provides good compression with less memory and CPU overhead. snappy Use the snappy library for compression, and always compress the output regardless of the destination. This compression provides good compression with less memory and CPU overhead. Since SiLK 3.13.0. bestUse lzo1x if available, otherwise use snappy if available, other- wise use zlib if available. Only compress the output when writing to a file. --site-config-file=FILENAME Read the SiLK site configuration from the named file FILENAME. When this switch is not provided, rwset searches for the site configuration file in the locations specified in the "FILES" section. --xargs --xargs=FILENAME Read the names of the input files from FILENAME or from the standard input if FILENAME is not provided. The input is expected to have one filename per line. rwset opens each named file in turn and reads records from it as if the filenames had been listed on the command line. --help Print the available options and exit. --version Print the version number and information about how SiLK was configured, then exit the application. EXAMPLES In the following examples, the dollar sign ("$") represents the shell prompt. The text after the dollar sign represents the command line. Lines have been wrapped for improved readability, and the back slash ("\") is used to indicate a wrapped line. rwset is intended to work tightly with rwfilter(1). For example, consider generating two IPsets: the first file, low_packet_tcp.set, contains the source IP addresses for incoming flow records (that is, the external hosts) where the record has no more than three packets in its sessions. The sec- ond IPset file, high_packet_tcp.set, contains the external IPs for records with four or more packets. The first set, for TCP traffic on 03/01/2003 can be generated with: $ rwfilter --start-date=2003/03/01:00 --end-date=2003/03/01:23 \ --proto=6 --packets=1-3 --pass=stdout \ | rwset --sip-file=low_packet_tcp.set The second set with: $ rwfilter --start-date=2003/03/01:00 --end-date=2003/03/01:23 \ --proto=6 --packets=4- --pass=stdout \ | rwset --sip-file=high_packet_tcp.set ENVIRONMENT SILK_IPSET_RECORD_VERSION This environment variable is used as the value for the --record-version when that switch is not provided. Since SiLK 3.7.0. SILK_IPV6_POLICY This environment variable is used as the value for --ipv6-policy when that switch is not provided. SILK_CLOBBER The SiLK tools normally refuse to overwrite existing files. Setting SILK_CLOBBER to a non-empty value removes this restriction. SILK_COMPRESSION_METHOD This environment variable is used as the value for --compression-method when that switch is not provided. Since SiLK 3.13.0. SILK_CONFIG_FILE This environment variable is used as the value for the --site-con- fig-file when that switch is not provided. SILK_DATA_ROOTDIR This environment variable specifies the root directory of data reposi- tory. As described in the "FILES" section, rwset may use this environ- ment variable when searching for the SiLK site configuration file. SILK_PATH This environment variable gives the root of the install tree. When searching for configuration files, rwset may use this environment vari- able. See the "FILES" section for details. FILES ${SILK_CONFIG_FILE} ${SILK_DATA_ROOTDIR}/silk.conf /data/silk.conf ${SILK_PATH}/share/silk/silk.conf ${SILK_PATH}/share/silk.conf /usr/local/share/silk/silk.conf /usr/local/share/silk.conf Possible locations for the SiLK site configuration file which are checked when the --site-config-file switch is not provided. SEE ALSO rwsetbuild(1), rwsetcat(1), rwsettool(1), rwsetmember(1), rwfilter(1), rw- fileinfo(1), rwbag(1), rwuniq(1), silk(7), zlib(3) NOTES Prior to SiLK 3.0, an IPset file could not contain IPv6 addresses and the record version was 2. The --record-version switch was added in SiLK 3.0 and its default was 3. In SiLK 3.6, an argument of 0 was allowed and made the default. Version 4 was added in SiLK 3.7 as was support for the SILK_IPSET_RECORD_VERSION environment variable. Version 5 was added in SiLK 3.14. SiLK 3.22.2 2026-08-27 rwset(1)
NAME | SYNOPSIS | DESCRIPTION | OPTIONS | EXAMPLES | ENVIRONMENT | FILES | SEE ALSO | NOTES
Want to link to this manual page? Use this URL:
<https://man.freebsd.org/cgi/man.cgi?query=rwset&sektion=1&manpath=FreeBSD+Ports+15.1.quarterly>