NMSGTOOL(1) NMSGTOOL(1) NAME nmsgtool - command line interface to libnmsg SYNOPSIS nmsgtool [OPTIONS...] DESCRIPTION nmsgtool is the command line interface to libnmsg, the reference implemen- tation of the NMSG binary structured message interchange format. The NMSG format is an efficient encoding of typed, structured data into payloads which are packed into containers which can be transmitted over the network or stored to disk. Each payload is associated with a specific mes- sage schema. Modules implementing a certain message schema along with func- tionality to convert between binary and presentation formats can be loaded at runtime by libnmsg. nmsgtool provides a command line interface to con- trol the transmission, storage, creation, and conversion of NMSG payloads. nmsgtool is a thin wrapper around libnmsg's I/O engine. In order to run, nmsgtool needs to open at least one input and at least one output. The sup- ported input types are: * Reading binary NMSG data from a file or socket. See the -r, -l options. * Reassembled IP datagrams from a network interface or pcap-savefile(5) file. See the -i, -p options. * Reading ASCII presentation form data from a file. See the -f option. Once input data has been read or converted to NMSG format internally with the help of an external module (in the case of pcap or ASCII data), it is striped or mirror across one or more outputs. The supported output types are: * Writing binary NMSG data to a file or socket. See the -w, -s options. * Writing ASCII presentation form data to a file. See the -o option. Reading or writing data in a non-NMSG format requires the use of an exter- nal module (called an "nmsgpb module") to convert to or from NMSG format. nmsgtool selects an nmsgpb module based on a vendor ID and message type. For input data, these fields must be set with the -V and -T options. If one or more inputs have been specified but no outputs have been speci- fied, an implicit output of presentation form data to stdout will be as- sumed. See the nmsg developer documentation for details on the NMSG wire format, how to interface with the nmsg library, and how to extend nmsg with new message types. OPTIONS -h, --help Display help text and exit. -d, --debug Increment debugging level. -dd is verbose and -dddd is very verbose. If the NMSG_KAFKA_LOG_LEVEL environment variable is set and a Kafka in- put/output is used, Kafka will log non-errors at the specified logging level. -v version Print the version number of nmsgtool. -V vendor, --vendor vendor Set the vendor field of generated NMSG payloads to the vendor identi- fied by vname. This is a symbolic string whose allowed values are de- termined at runtime when nmsgpb modules are loaded. -T msgtype, --msgtype msgtype Set the message type field of generated NMSG payloads to the message type identified by msgtype. This is a symbol string whose allowed val- ues are determined at runtime when nmsgpb modules are loaded. -B byterate, --byterate byterate Set the maximum bytes-per-second that libnmsg will process from file inputs. -D daemon Fork nmsgtool into the background as a daemon process. -P pidfile, --pidfile pidfile Write nmsgtool's process ID into a file identified by pidfile. When nmsgtool exits, this file's contents will be erased. -U username, --username username Attempt to drop root privileges and run nmsgtool as user username. If the current user is username, nmsgtool will exit. -e endline, --endline endline Set the string to be used for the end-of-line continuation separator. By default this value is set to the newline character. This option un- derstands the escape sequences '\n' (newline), '\t' (tab), and '\\' (backslash). Setting this option to ' \\\n\t' will emulate the line continuation behavior of ncaptool. -m mtu, --mtu mtu Set the "maximum transmission unit" for writes to datagram socket out- puts. Note that this option is somewhat misnamed since it does not re- fer to protocol data units of IP but to NMSG containers, so the addi- tional small but bounded overhead of the UDP and NMSG headers need to be taken into account. Since NMSG's UDP transport is tuned for opera- tion on jumbo Ethernet, this value defaults to 8192. To avoid IP frag- mentation over standard Ethernet or Internet transport, this value should be lowered to around 1280 to 1400. This option does not limit the maximum size of an NMSG payload because NMSG containers can be segmented into multiple datagrams. -c count, --count count Stop the process or (with -k specified) reopen outputs after count pay- loads have been processed. Note that each output keeps its own payload output counter and this value refers to that per-output value and not to the sum of all output counters. -t secs, --interval secs Stop the process or (with -k specified) reopen outputs on a time inter- val modulo the secs parameter. For example, -t 3600 means "on the hour". -R, --randomize Randomize the initial offset within the interval that the process is stopped or outputs are reopened. -k cmd, --kicker cmd Make -c and -t continuous. In this mode output file names are suffixed with a timestamp and nmsgtool runs continuously, rotating output files as payload counts or time intervals expire. cmd specifies the command to run on output files after rotation. If cmd is set to the empty string '', then no command is executed and only file rotation is per- formed. --kafkakey fieldname Use the value of the named NMSG message field in its canonical repre- sentation as the key for payloads in the JSON format that are sent to Kafka. Can also be set via NMSG_KAFKA_KEY environment variable. --readkafka kafka Read NMSG payloads in either binary or JSON format from a Kafka end- point. The address kafka has format proto:topic[#parti- tion|%group_id]@broker[:port][,offset]. Either a partition number or a consumer group ID may be optionally supplied. Also optional is an off- set consisting of either a numerical value or the string 'oldest' or 'newest' in order to start retrieval at the oldest/newest messages in the Kafka topic. An example of a possible kafka endpoint is "nmsg:ch202#0@kafka.example.com:9092,3000" to indicate that nmsgtool shall read nmsg containers from topic "ch202" on partition 0 at offset 3000 from the Kafka broker at kafka.example.com, port 9092. Configura- tion for Kafka can be supplied through the NMSG_KAFKA_CONFIG environ- ment variable. This variable accepts either key=value pairs separated by colons, or a path to a configuration file in INI format. -b filter, --bpf filter Filter pcap inputs (-p or -i) with the BPF expression filter. Note that libnmsg's pcap input is designed for IP datagrams and not network frames, so the filter specified by -b will be munged internally into several forms in order to receive IP fragments and filter the reassem- bled IP datagrams. Internally, IPv4 and IPv6 packets are selected for processing, including those received in VLAN tagged frames. -F dso[,param], --filter dso[,param] Filter nmsg payloads with a module loaded from the given DSO file. The dso specified may either be a short, human friendly name (like sample) which will be expanded into an absolute filename in the default sys- tem-wide libnmsg module path, or it may be a name beginning with / or . (like /usr/lib/nmsg/nmsg_flt1_sample.so), in which case the dso value will be treated as an absolute or relative path name. Filter modules may take a module-defined parameter string param. The dso value may be followed by a comma, in which case everything after the comma is treated as the module parameter param and passed to the module's initialization function. Some modules may require a param value or may perform additional validation on the parameter. If the module fails to initialize, nmsgtool will exit with an error message. This option can be specified more than once, in which case the filter modules will be loaded in the order they were specified in on the com- mand-line, and will form a linear filter chain. --policy ACCEPT|DROP If any filter modules have been loaded with -F, --policy may be used to specify the policy action to take if all filters in the filter chain decline to handle a given message. The default policy action is --pol- icy ACCEPT, which causes messages that are declined by the filter chain to be accepted into the output stream. If --policy DROP is specified, any messages which are declined by the filter chain will be silently discarded. -M dso[,param], --statsmod dso[,param] Export nmsg IO statistics through an external statistics module. The dso specified may either be a short, human friendly name (like sample) which will be expanded into an absolute filename in the default sys- tem-wide libnmsg module path, or it may be a name beginning with / or . (like /usr/lib/nmsg/nmsg_stats1_sample.so), in which case the dso value will be treated as an absolute or relative path name. Statistics modules may take a module-defined parameter string param. The dso value may be followed by a comma, in which case everything af- ter the comma is treated as the module parameter param and passed to the module's initialization function. Some modules may require a param value or may perform additional validation on the parameter. If the module fails to initialize, nmsgtool will exit with an error message. This option can be specified more than once, and may also be provided using the NMSG_STATSMOD_CONFIG environment variable. -r file, --readnmsg file Read NMSG payloads from a file. -f file, --readpres file Read presentation format data from a file and convert to NMSG payloads. This option is dependent on the -V and -T options being set in order to select a specific nmsgpb module to perform presentation format to NMSG payload conversion. Not all nmsgpb modules necessarily support this conversion method, in which case nmsgtool will print a "function not implemented" message. -j file, --readjson file Read JSON format data from a file. See documentation for --writejson for format details. -l addr/port, --readsock addr/port Read NMSG payloads from a UDP socket. The addr parameter must be set to a valid system or broadcast IPv4 or IPv6 address and the port parameter may be a single port number or a range of ports in the form port0..portN. -C channel, --readchan channel Read NMSG payloads from one or more UDP sockets specified by an alias channel. nmsgtool will read aliases from the file named nmsg- tool.chalias in the system configuration directory (usually /etc) or from the file defined by the NMSG_CHALIAS_FILE environment variable. The format of this file is one alias per line with each line starting with the alias name channel followed by one or more whitespace delim- ited address/port entries (as would be parsed by the --readsock op- tion). For example, the following alias file would create two channels. Call- ing nmsgtool with -C 123 would be equivalent to calling nmsgtool with -l 192.0.2.1/8430 while -C 234 would be equivalent to -l 192.0.2.255/8430..8437 -l 192.0.2.255/9430. Example 1. nmsgtool.chalias example 123 192.0.2.1/8430 234 192.0.2.255/8430..8437 192.0.2.255/9430 -L zep, --readzsock zep Read NMSG payloads from a ZeroMQ endpoint. The endpoint zep is very similar to the "transport://address" specifiers used by the libzmq li- brary, except that additional configuration may be needed in order to set up the ZMQ connection, which is specified by appending comma-sepa- rated arguments. See the zmq_tcp(7) and zmq_ipc(7) manpages for de- tails. In addition to the base "transport://address" specifier, the user may specifically select between a bound or connected zmq socket by append- ing ",accept" or ",connect" to the zep argument. (If not given, nmsg- tool behaves as if ",connect" was given.) That is, ",accept" uses the zmq_bind(3) function to obtain an zmq endpoint, and ",connect" uses the zmq_connect(3) function. The user may also select between the ZeroMQ PUB/SUB and PUSH/PULL mes- saging patterns by appending either ",pubsub" or ",pushpull". (If not given, nmsgtool behaves as if ",pubsub" was passed.) See the zmq_socket(3) manpage for details. When PUB/SUB is used with -L, nmsg- tool participates in the "SUB" role of the ZeroMQ PUB/SUB messaging pattern, and when PUSH/PULL is used, nmsgtool participates in the "PULL" role. Examples of possible zep arguments to -L include "ipc:///tmp/nmsg.sock,accept,pubsub" to indicate a ZeroMQ endpoint that accepts PUB/SUB connections on the IPC path /tmp/nmsg.sock (in the SUB role), and "tcp://127.0.0.1:5555,accept,pushpull" to indicate a ZeroMQ endpoint that listens for PUSH/PULL connections on the TCP socket 127.0.0.1:5555 (in the PULL role). -S zep, --writezsock zep Write NMSG payloads to a ZeroMQ endpoint. The endpoint zep is very sim- iliar to the "transport://address" specifiers used by the libzmq li- brary, except that additional configuration may be needed in order to set up the zmq connection, which is specified by appending comma-sepa- rated arguments. See the zmq_tcp(7) and zmq_ipc(7) manpages for de- tails. In addition to the base "transport://address" specifier, the user may specifically select between a bound or connected zmq socket by append- ing ",accept" or ",connect" to the zep argument. (If not given, nmsg- tool behaves as if ",connect" was given.) That is, ",accept" uses the zmq_bind(3) function to obtain an zmq endpoint, and ",connect" uses the zmq_connect(3) function. The user may also select between the ZeroMQ PUB/SUB and PUSH/PULL mes- saging patterns by appending either ",pubsub" or ",pushpull". (If not given, nmsgtool behaves as if ",pubsub" was passed.) See the zmq_socket(3) manpage for details. When PUB/SUB is used with -S, nmsg- tool participates in the "PUB" role of the ZeroMQ PUB/SUB messaging pattern, and when PUSH/PULL is used, nmsgtool participates in the "PUSH" role. Examples of possible zep arguments to -S include "ipc:///tmp/nmsg.sock,connect,pubsub" to indicate a ZeroMQ endpoint that connects to a PUB/SUB socket on the IPC path /tmp/nmsg.sock (in the PUB role), and "tcp://127.0.0.1:5555,connect,pushpull" to indicate a ZeroMQ endpoint that connects to a PUSH/PULL socket on the TCP socket 127.0.0.1:5555 (in the PULL role). -p file, --readpcap file Read IP packets from a pcap-savefile(5) file file using the pcap(3) li- brary. These packets are then reassembled into datagrams which are then passed to an nmsgpb module for conversion into NMSG payloads. This op- tion is dependent on the -V and -T options being set in order to select a specific nmsgpb module to perform IP datagram to NMSG payload conver- sion. Not all nmsgpb modules necessarily support this conversion method, in which case nmsgtool will print a "function not implemented" message. -i if[+][,snap] Read IP packets from a network interface if using the pcap(3) library. Reassembly is performed as described for --readpcap. + may be appended to the interface name to capture in promiscuous mode. The capture length snap may be set by appending ,snap. The default capture length is 1522. -V and -T are required. -w file, --writenmsg file Write NMSG payloads to a file. --writekafka kafka Write NMSG payloads in either binary or JSON format to a Kafka end- point. The address kafka has format proto:topic[#parti- tion|%group_id]@broker[:port]. Either a partition number or a consumer group ID may be optionally supplied. An example of a possible kafka endpoint is "nmsg:ch202#0@kafka.example.com:9092" to indicate that nms- gtool shall write nmsg containers to topic "ch202" on partition 0 to Kafka broker kafka.example.com, port 9092. Note that nmsgtool ignores offsets for Kafka producers. Configuration for Kafka can be supplied through the NMSG_KAFKA_CONFIG environment variable. This variable ac- cepts either key=value pairs separated by colons, or a path to a con- figuration file in INI format. -o file, --writepres file Write presentation format payloads to a file. -J file, --writejson file Write JSON payloads to a file. Payloads are rendered as JSON dictionaries with keys: * time: the payload timestamp * vname: the vendor name identifying the message type * mname: the message type name identifying the message type * source: the message source, if present, as l a hexadecimal string * group, operator: the group and operator names or numbers, if present. * message: a dictionary of message fields and values The value of associated with the 'message' key is a JSON dictionary of the message fields. Enumerated types are translated to string represen- tation, if possible. Binary data is base64 encoded and output as a string. Invalid UTF-8 sequences in string fields are replaced with U+FFFD. -s addr/port[,rate[,freq]], --writesock addr/port[,rate[,freq]] Write NMSG payloads to a UDP socket specified by the system or broad- cast IPv4 or IPv6 address addr and the UDP port port. Optionally the output rate may be limited to rate per second by appending ,rate. If an output rate is specified, the scheduling frequency freq may be set by appending ,freq. The default scheduling frequency for rate limits is 100. NMSG payloads are not immediately output to sockets but are instead concatenated into a buffer of a certain size (see the --mtu option) be- fore being sent. To circumvent this behavior see the --unbuffered op- tion. -z, --zlibout Perform transparent zlib compression of written NMSG containers. This applies to both file (-w) and socket (-s) outputs. --mirror Mirror NMSG payloads across data outputs. By default NMSG payloads re- gardless of input source are striped across all available outputs. When --mirror is set, NMSG payloads are duplicated to every output. This op- tion has no effect if there is only a single output. -Z zchannel, --readzchan zchannel Read NMSG payloads from a ZeroMQ endpoint as specified by an alias zchannel. nmsgtool will read aliases from the file named nmsg- tool.chalias in the system configuration directory (usually /usr/lo- cal/etc or /etc) or from the file defined by the NMSG_CHALIAS_FILE en- vironment variable. The format of this file is one alias per line with each line starting with the alias name zchannel followed by one or more whitespace-delimited ZeroMQ endpoints (as described with the --readz- sock option). --setsource sonum Set the "source" field of output NMSG payloads to sonum. NMSG payloads have an optional "source" field which is meant to be used as a unique opaque identifier identifying the immediate source of a re- distributed payload. The sonum value should be specified as an unsigned 32 bit integer in hexadecimal format with a leading "0x". In the nmsg presentation form output header, the source field is the fourth bracketed value. --getsource sonum Filter the "source" field of input NMSG payloads against sonum. --setoperator opname Set the "operator" field of output NMSG payloads to opname. NMSG payloads have an optional "operator" field which is meant to iden- tify the operator generating a payload. The operator field is repre- sented as a 32 bit integer on the wire but is aliased to a symbolic string for presentation purposes by the file nmsg.opalias in the system configuration directory. The alias file contains one number/name pair separated by whitespace per line. In the nmsg presentation form output header, the operator field is the fifth bracketed value. --getoperator opname Filter the "operator" field of input NMSG payloads against opname. --setgroup grname Set the "group" field of output NMSG payloads to grname. NMSG payloads have an optional "group" field which is meant to identify the campaign or group that a payload belongs to. The group field is represented as a 32 bit integer on the wire but is aliased to a sym- bolic string for presentation purposes by the file nmsg.gralias in the system configuration directory. The alias file contains one number/name pair separated by whitespace per line. In the nmsg presentation form output header, the group field is the sixth bracketed value. --getgroup grname Filter the "group" name of input NMSG payloads against grname. EXAMPLES To read NMSG payloads from a socket and write presentation form data to stdout: nmsgtool -l 192.0.2.1/8430 To read NMSG payloads from a file and write presentation form data to std- out: nmsgtool -r /tmp/file.nmsg To read NMSG payloads from a socket and write to a binary NMSG file: nmsgtool -l 192.0.2.1/8430 -w /tmp/file.nmsg To read reassembled IP datagrams from a network interface in promiscuous mode, convert these datagrams to NMSG using the base/ncap nmsgpb module, and write to a file: nmsgtool -i eth0+ -V base -T ncap -w /tmp/ncapfile.nmsg To read NMSG payloads from multiple socket inputs and write to a series of compressed files, rotated every hour: nmsgtool -l 192.0.2.255/8430..8437 -w /tmp/file -t 3600 -k '' -z To read NMSG payloads from a ZeroMQ "PULL" socket over a TCP connection: nmsgtool -L tcp://127.0.0.1:5555,accept,pushpull This waits for TCP connections on 127.0.0.1:5555. To read NMSG payloads from a file and write them to a ZeroMQ "PUSH" socket over a TCP connection: nmsgtool -r /tmp/file.nmsg -S tcp://127.0.0.1:5555,connect,pushpull This attempts to connect to a TCP reader on 127.0.0.1:5555, such as the nmsgtool command in the previous example. To read NMSG payloads from an SIE channel named "ch222" and write them to stdout while writing IO stats every 3 seconds to a file name "ch222.stats": nmsgtool -C ch222 -M sample,out=ch222.stats,secs=3 10/02/2024 NMSGTOOL(1)
NAME | SYNOPSIS | DESCRIPTION | OPTIONS | EXAMPLES
Want to link to this manual page? Use this URL:
<https://man.freebsd.org/cgi/man.cgi?query=nmsgtool&sektion=1&manpath=FreeBSD+Ports+15.1.quarterly>