DNSTABLE-ENCODING(5) DNSTABLE-ENCODING(5) NAME dnstable-encoding - key-value encoding format used by dnstable DESCRIPTION dnstable has been designed to import the data generated by the nmsg-dns-cache program in the NMSG SIE/dnsdedupe format. See the ISC Pas- sive DNS Architecture whitepaper for details about how this data is actu- ally generated. The data model used with the aid of this format preserves a number of important properties like Resource Record type transparency and Resource Record set atomicity. Additionally, dnstable preserves extra meta- data available in the dnsdedupe payloads. dnstable makes use of the MTBL Sorted String Table (SSTable) format, which provides an immutable mapping of keys to optional values. This format al- lows fast lookups of key prefixes, so the precise encoding and layout of dnstable keys are chosen to take advantage of this property. MTBL keys and values are arbitrary byte arrays, and because MTBL keys are stored in sorted order it is possible to look up entire key-value entries by searching for the initial part of a key. The first byte of each dnstable key is a "type" byte, and there are cur- rently six different "types" of dnstable key-value entries which are de- scribed below. These different types are used to encode the full passive DNS payload as well as "indices" into the payload and metadata. Each dnstable key is a compound key that packs individual fields together, and for the key types that include a value, the value is a compound value that also packs individual fields together. In the string literals given in the text below, C-style "\xNN" hex encoding may be used to represent particular byte values. For example, "\x03" is an octet with decimal value 3. Note that this is different than the \DDD en- coding of arbitrary decimal values in DNS names. Key-value entry types All keys and values described below are byte arrays containing packed fields. Each field is concatenated together without delimiters. The follow- ing data types are used: * Byte. * Fixed-width 16 bit integer in little-endian order. * Variable-width integer. * Wire-format DNS record data. * Wire-format DNS domain names. The domain name "www.example.com" would be encoded as "\x03www\x07example\x03com\x00". * Label-reversed wire-format DNS domain names. The domain name "www.exam- ple.com" would be encoded as "\x03com\x07example\x03www\x00". * RRtype union. This can contain one of multiple types, depending upon the length of the value. * If the length is 0, then it is interpreted the same as if all de- fined RRtype bits were set. * If the length is 1, then the value is an RRtype valued from 0 to 255. * If the length is 2, then the value is an RRtype valued from 256 to 65535, in little-endian order. * If the length is 3 or more, then the value is an RRtype bitmap, as defined in RFC4034 section 4.1.2. * Examples of RFC4034 bitmaps: * RRtypes A, NS, and SOA are encoded as bitmap 00 01 62. * RRtypes A, NS, NAPTR, SPF, SOA, MX, TXT, AAAA, and LOC are encoded as bitmap 00 0d 62 01 80 0c 10 00 00 00 00 00 00 00 10. ENTRY_TYPE_RRSET This entry type encodes the complete passive DNS payload. The RRset owner name is encoded at the beginning of the key so that "forward" lookups can be performed on the owner name. The owner name is encoded with labels reversed from the usual DNS order to increase locality and allow for wildcard matches. (For example, "*.example.com".) The key is a packed tuple consisting of the following fields, in order: 1. Type byte. The constant "\x00". 2. RRset owner name. Label-reversed wire-format DNS domain name. 3. RRset type. Variable-width integer. 4. Bailiwick domain name. Label-reversed wire-format DNS domain name. 5. Rdata array. An array of one or more wire-format DNS record data values. Each record data value is preceded by its length, encoded as a variable-width integer. The value consists of the following fields, in order: 1. First timestamp. Variable-width integer. Number of seconds since the epoch. 2. Last timestamp. Variable-width integer. Number of seconds since the epoch. 3. Count. Variable-width integer. Number of observations between the first and last timestamps. ENTRY_TYPE_RRSET_NAME_FWD This entry type encodes only the RRset owner name in the usual "for- ward" direction. This entry type provides an "index" into the EN- TRY_TYPE_RRSET "table" so that RRsets whose owner names start with a particular subsequence of labels can be found. (For example, looking up the label subsequence "www.example.*" using this index would reveal the existence of the domain names "www.example.com", "www.example.net", etc.) The key consists of the following fields, in order: 1. Type byte. The constant "\x01". 2. RRset owner name. Wire-format DNS domain name. In versions before 0.12.0, there was no value associated with this en- try type. As of that version, the value is an RRtype union described above. ENTRY_TYPE_RDATA This entry type encodes individual Resource Record data values; that is, an RRset containing multiple Resource Records will be encoded as multiple entries in this "table". The key consists of the following fields, in order: 1. Type byte. The constant "\x02". 2. Rdata. Wire-format DNS record data. 3. RR type. Variable-width integer. 4. RR owner name. Label-reversed wire-format DNS domain name. 5. Rdata length. Fixed-width 16 bit integer in little-endian order. Number of bytes in the Rdata field. The Rdata length field is encoded at the end of the key so that searches can be performed over the Rdata field without needing to know the length of the record data value in advance (or worse, needing to iterate over all possible length values). An additional ENTRY_TYPE_RDATA entry is produced for DNS records which encode a domain name but do not begin with one (i.e. DNS records which contain a DNS domain name offset into the latter part of the record data): * An MX record contains a domain name offset 2 octets into the record data value. * An SVCB record contains a domain name offset 2 octets into the record data value. * An HTTPS record contains a domain name offset 2 octets into the record data value. * A SRV record contains a domain name offset 6 octets into the record data value. If those record types had been laid out in the applicable DNS RFCs with the domain name at the beginning of the record data the following hack would not be necessary in order to make these record types easily searchable by domain, or for records of different rrtypes that refer- ence the same domains to cluster together. This sub-type "slices" the record data so that the domain name in the latter part of the record appears at the beginning of the key and the initial bytes of the record appear just before the Rdata length field at the end of the key. That is, the key consists of the following fields, in order: 1. Type byte. The constant "\x02". 2. Latter data slice. Wire-format DNS record data. This is the latter part of the record data. 3. RR type. Variable-width integer. 4. RR owner name. Label-reversed wire-format DNS domain name. 5. Initial data slice. Wire-format DNS record data. This is the ini- tial part of the record data. 6. Latter data slice length. Fixed-width 16 bit integer in little-en- dian order. Number of bytes in the latter data slice. The number of bytes in the initial data slice is implicit (the differ- ence between the dnstable entry value length and the sum of the sizes of all previously described mandatory fields, once decoded.) An entry can be determined to be either a sliced or un-sliced record by whether or not this implicit initial data slice length is non-zero or zero, re- spectively. The value consists of the following fields, in order: 1. First timestamp. Variable-width integer. Number of seconds since the epoch. 2. Last timestamp. Variable-width integer. Number of seconds since the epoch. 3. Count. Variable-width integer. Number of observations between the first and last timestamps. This is the same encoding as the values for ENTRY_TYPE_RRSET entries. ENTRY_TYPE_RDATA_NAME_REV This entry type encodes label-reversed domain names from the record data value for certain DNS record types that contain a domain name in the record data, specifically the SOA, NS, CNAME, DNAME, PTR, MX, SRV, SVCB, and HTTPS DNS record types. The "slicing" logic described above in ENTRY_TYPE_RDATA encompasses the extraction of these domain names. The key consists of the following fields, in order: 1. Type byte. The constant "\x03". 2. Rdata name. Label-reversed wire-format DNS domain name. In versions before 0.12.0, there was no value associated with this en- try type. As of that version, the value is an RRtype union described above. ENTRY_TYPE_TIME_RANGE This entry type encodes the overall time range of all ENTRY_TYPE_RRSET and ENTRY_TYPE_RDATA entries in a dnstable data set, i.e. the earliest first timestamp and latest last timestamp of all such entries. The key consists of: 1. Type byte. The constant "\xFE". The value consists of the following fields, in order: 1. First timestamp. Variable-width integer. Number of seconds since the epoch. 2. Last timestamp. Variable-width integer. Number of seconds since the epoch. ENTRY_TYPE_VERSION This entry encodes a version number for the format of another entry type. Each of the above entry types, with the exception of EN- TRY_TYPE_TIME_RANGE may have an associated version number entry. The key consists of the following fields, in order: 1. Type byte. The constant "0xFF". 2. Versioned Type. The Type byte to which this entry applies. The value consists of: 1. Version number. Variable-width integer. EXAMPLES This section gives two example RRsets and shows how they would be encoded using the above entry types. The first example is an NS delegation RRset for the domain name "example.com" and consists of two Resource Records; it will result in the generation of six key-value entries. The second example is an A-record for the domain name "www.isc.org" that consists of a single Resource Record; it will result in the generation of three key-value en- tries. In each example entry, a table of the broken down fields in each entry is given, and then the final, concatenated key and value byte strings. Example 1: example.com./NS Given the following passive DNS payload: +------------+-------------------------+ | Field | Value | +------------+-------------------------+ | | | | rrname | example.com. | +------------+-------------------------+ | | | | rrtype | NS | +------------+-------------------------+ | | | | rdata | ns1.example.com. | +------------+-------------------------+ | | | | rdata | ns2.example.com. | +------------+-------------------------+ | | | | bailiwick | com. | +------------+-------------------------+ | | | | time_first | Mon Apr 2 12:33:20 2012 | +------------+-------------------------+ | | | | time_last | Mon Apr 2 15:20:00 2012 | +------------+-------------------------+ | | | | count | 23 | +------------+-------------------------+ The following entries will be generated: example.com entry #1 (ENTRY_TYPE_RRSET) Key: +-----------+-------------------+------------------+ | Field | Data | Interpretation | +-----------+-------------------+------------------+ | | | | | type | "\x00" | ENTRY_TYPE_RRSET | +-----------+-------------------+------------------+ | | | | | rrname | "\x03com\x07exam- | example.com. | | | ple\x00" | | +-----------+-------------------+------------------+ | | | | | rrtype | "\x02" | NS | +-----------+-------------------+------------------+ | | | | | bailiwick | "\x03com\x00" | com. | +-----------+-------------------+------------------+ | | | | | rdata | "\x11" | ns1.example.com. | | | "\x03ns1\x07exam- | | | | ple\x03com\x00" | | +-----------+-------------------+------------------+ | | | | | rdata | "\x11" | ns2.example.com. | | | "\x03ns2\x07exam- | | | | ple\x03com\x00" | | +-----------+-------------------+------------------+ Value: +------------+------------------------+----------------------+ | Field | Data | Interpretation | +------------+------------------------+----------------------+ | | | | | time_first | "\x90\xb9\xe6\xfb\x04" | 1333370000 (Mon Apr | | | | 2 12:33:20 2012) | +------------+------------------------+----------------------+ | | | | | time_last | "\xa0\x87\xe7\xfb\x04" | 1333380000 (Mon Apr | | | | 2 15:20:00 2012) | +------------+------------------------+----------------------+ | | | | | count | "\x17" | 23 | +------------+------------------------+----------------------+ Final key: "\x00\x03com\x07example\x00\x02\x03com\x00\x11\x03ns1\x07exam- ple\x03com\x00\x11\x03ns2\x07example\x03com\x00" Final value: "\x90\xb9\xe6\xfb\x04\xa0\x87\xe7\xfb\x04\x17" example.com entry #2 (ENTRY_TYPE_RRSET_NAME_FWD) Key: +--------+-----------------+-------------------------+ | Field | Data | Interpretation | +--------+-----------------+-------------------------+ | | | | | type | "\x01" | EN- | | | | TRY_TYPE_RRSET_NAME_FWD | +--------+-----------------+-------------------------+ | | | | | rrname | "\x07exam- | example.com. | | | ple\x03com\x00" | | +--------+-----------------+-------------------------+ Value: +--------------+--------+----------------+ | Field | Data | Interpretation | +--------------+--------+----------------+ | | | | | rrtype union | "\x02" | NS | +--------------+--------+----------------+ Final key: "\x01\x07example\x03com\x00" Final value: "\x02" example.com entry #3 (ENTRY_TYPE_RDATA) Key: +--------------+-------------------+----------------------+ | Field | Data | Interpretation | +--------------+-------------------+----------------------+ | | | | | type | "\x02" | ENTRY_TYPE_RDATA | +--------------+-------------------+----------------------+ | | | | | rdata | "\x03ns1\x07exam- | ns1.example.com. | | | ple\x03com\x00" | | +--------------+-------------------+----------------------+ | | | | | rrtype | "\x02" | NS | +--------------+-------------------+----------------------+ | | | | | rrname | "\x03com\x07exam- | example.com. | | | ple\x00" | | +--------------+-------------------+----------------------+ | | | | | rdata length | "\x00\x11" | rdata field is 17 | | | | octets long | +--------------+-------------------+----------------------+ Value: +------------+------------------------+----------------------+ | Field | Data | Interpretation | +------------+------------------------+----------------------+ | | | | | time_first | "\x90\xb9\xe6\xfb\x04" | 1333370000 (Mon Apr | | | | 2 12:33:20 2012) | +------------+------------------------+----------------------+ | | | | | time_last | "\xa0\x87\xe7\xfb\x04" | 1333380000 (Mon Apr | | | | 2 15:20:00 2012) | +------------+------------------------+----------------------+ | | | | | count | "\x17" | 23 | +------------+------------------------+----------------------+ Final key: "\x02\x03ns1\x07example\x03com\x00\x02\x03com\x07example\x00\x00\x11" Final value: "\x90\xb9\xe6\xfb\x04\xa0\x87\xe7\xfb\x04\x17" example.com entry #4 (ENTRY_TYPE_RDATA) Key: +--------------+-------------------+----------------------+ | Field | Data | Interpretation | +--------------+-------------------+----------------------+ | | | | | type | "\x02" | ENTRY_TYPE_RDATA | +--------------+-------------------+----------------------+ | | | | | rdata | "\x03ns2\x07exam- | ns2.example.com. | | | ple\x03com\x00" | | +--------------+-------------------+----------------------+ | | | | | rrtype | "\x02" | NS | +--------------+-------------------+----------------------+ | | | | | rrname | "\x03com\x07exam- | example.com. | | | ple\x00" | | +--------------+-------------------+----------------------+ | | | | | rdata length | "\x00\x11" | rdata field is 17 | | | | octets long | +--------------+-------------------+----------------------+ Value: +------------+------------------------+----------------------+ | Field | Data | Interpretation | +------------+------------------------+----------------------+ | | | | | time_first | "\x90\xb9\xe6\xfb\x04" | 1333370000 (Mon Apr | | | | 2 12:33:20 2012) | +------------+------------------------+----------------------+ | | | | | time_last | "\xa0\x87\xe7\xfb\x04" | 1333380000 (Mon Apr | | | | 2 15:20:00 2012) | +------------+------------------------+----------------------+ | | | | | count | "\x17" | 23 | +------------+------------------------+----------------------+ Final key: "\x02\x03ns2\x07example\x03com\x00\x02\x03com\x07example\x00\x00\x11" Final value: "\x90\xb9\xe6\xfb\x04\xa0\x87\xe7\xfb\x04\x17" example.com entry #5 (ENTRY_TYPE_RDATA_NAME_REV) Key: +------------+-------------------+-------------------------+ | Field | Data | Interpretation | +------------+-------------------+-------------------------+ | | | | | type | "\x03" | EN- | | | | TRY_TYPE_RDATA_NAME_REV | +------------+-------------------+-------------------------+ | | | | | rdata name | "\x03com\x07exam- | ns1.example.com. | | | ple\x03ns1\x00" | | +------------+-------------------+-------------------------+ Value: +--------------+--------+----------------+ | Field | Data | Interpretation | +--------------+--------+----------------+ | | | | | rrtype union | "\x02" | NS | +--------------+--------+----------------+ Final key: "\x03\x03com\x07example\x03ns1\x00" Final value: "\x02" example.com entry #6 (ENTRY_TYPE_RDATA_NAME_REV) Key: +------------+-------------------+-------------------------+ | Field | Data | Interpretation | +------------+-------------------+-------------------------+ | | | | | type | "\x03" | EN- | | | | TRY_TYPE_RDATA_NAME_REV | +------------+-------------------+-------------------------+ | | | | | rdata name | "\x03com\x07exam- | ns2.example.com. | | | ple\x03ns2\x00" | | +------------+-------------------+-------------------------+ Value: +--------------+--------+----------------+ | Field | Data | Interpretation | +--------------+--------+----------------+ | | | | | rrtype union | "\x02" | NS | +--------------+--------+----------------+ Final key: "\x03\x03com\x07example\x03ns2\x00" Final value: "\x02" Example 2: www.isc.org./A Given the following passive DNS payload: +------------+-------------------------+ | Field | Value | +------------+-------------------------+ | | | | rrname | www.isc.org. | +------------+-------------------------+ | | | | rrtype | A | +------------+-------------------------+ | | | | rdata | 149.20.64.42 | +------------+-------------------------+ | | | | bailiwick | isc.org. | +------------+-------------------------+ | | | | time_first | Mon Apr 2 12:33:20 2012 | +------------+-------------------------+ | | | | time_last | Mon Apr 2 15:20:00 2012 | +------------+-------------------------+ | | | | count | 1 | +------------+-------------------------+ The following entries will be generated: www.isc.org entry #1 (ENTRY_TYPE_RRSET) Key: +-----------+-------------------------+------------------+ | Field | Data | Interpretation | +-----------+-------------------------+------------------+ | | | | | type | "\x00" | ENTRY_TYPE_RRSET | +-----------+-------------------------+------------------+ | | | | | rrname | "\x03org\x03isc\x03www" | www.isc.org. | +-----------+-------------------------+------------------+ | | | | | rrtype | "\x01" | A | +-----------+-------------------------+------------------+ | | | | | bailiwick | "\x03org\x03isc\x00" | isc.org. | +-----------+-------------------------+------------------+ | | | | | rdata | "\x04" | 149.20.64.42 | | | "\x95\x14\x40\x2a" | | +-----------+-------------------------+------------------+ Value: +------------+------------------------+----------------------+ | Field | Data | Interpretation | +------------+------------------------+----------------------+ | | | | | time_first | "\x90\xb9\xe6\xfb\x04" | 1333370000 (Mon Apr | | | | 2 12:33:20 2012) | +------------+------------------------+----------------------+ | | | | | time_last | "\xa0\x87\xe7\xfb\x04" | 1333380000 (Mon Apr | | | | 2 15:20:00 2012) | +------------+------------------------+----------------------+ | | | | | count | "\x01" | 1 | +------------+------------------------+----------------------+ Final key: "\x00\x03org\x03isc\x03www\x01\x03org\x03isc\x00\x04\x95\x14\x40\x2a" Final value: "\x90\xb9\xe6\xfb\x04\xa0\x87\xe7\xfb\x04\x01" www.isc.org entry #2 (ENTRY_TYPE_RRSET_NAME_FWD) Key: +--------+-----------------------------+-------------------------+ | Field | Data | Interpretation | +--------+-----------------------------+-------------------------+ | | | | | type | "\x01" | EN- | | | | TRY_TYPE_RRSET_NAME_FWD | +--------+-----------------------------+-------------------------+ | | | | | rrname | "\x03www\x03isc\x03org\x00" | www.isc.org. | +--------+-----------------------------+-------------------------+ Value: +--------------+--------+----------------+ | Field | Data | Interpretation | +--------------+--------+----------------+ | | | | | rrtype union | "\x01" | A | +--------------+--------+----------------+ Final key: "\x01\x03www\x03isc\x03org\x00" Final value: "\x01" www.isc.org entry #3 (ENTRY_TYPE_RDATA) Key: +--------------+-----------------------------+----------------------+ | Field | Data | Interpretation | +--------------+-----------------------------+----------------------+ | | | | | type | "\x02" | ENTRY_TYPE_RDATA | +--------------+-----------------------------+----------------------+ | | | | | rdata | "\x95\x14\x40\x2a" | 149.20.64.42 | +--------------+-----------------------------+----------------------+ | | | | | rrtype | "\x01" | A | +--------------+-----------------------------+----------------------+ | | | | | rrname | "\x03org\x03isc\x03www\x00" | www.isc.org. | +--------------+-----------------------------+----------------------+ | | | | | rdata length | "\x00\x04" | rdata field is 4 | | | | octets long | +--------------+-----------------------------+----------------------+ Value: +------------+------------------------+----------------------+ | Field | Data | Interpretation | +------------+------------------------+----------------------+ | | | | | time_first | "\x90\xb9\xe6\xfb\x04" | 1333370000 (Mon Apr | | | | 2 12:33:20 2012) | +------------+------------------------+----------------------+ | | | | | time_last | "\xa0\x87\xe7\xfb\x04" | 1333380000 (Mon Apr | | | | 2 15:20:00 2012) | +------------+------------------------+----------------------+ | | | | | count | "\x01" | 1 | +------------+------------------------+----------------------+ Final key: "\x02\x95\x14\x40\x2a\x01\x03osc\x03isc\x03www\x00\x00\x04" Final value: "\x90\xb9\xe6\xfb\x04\xa0\x87\xe7\xfb\x04\x01" SEE ALSO ISC Passive DNS Architecture, https://www.farsightsecurity.com/assets/me- dia/download/passive-dns-architecture.pdf Google Protocol Buffers Encoding, https://developers.google.com/proto- col-buffers/docs/encoding, for the base-128 variable-width integer encoding used in some of the fields in dnstable keys. Also see mtbl_varint(3) for a varint encoding implementation. 08/11/2021 DNSTABLE-ENCODING(5)
NAME | DESCRIPTION | EXAMPLES | SEE ALSO
Want to link to this manual page? Use this URL:
<https://man.freebsd.org/cgi/man.cgi?query=dnstable-encoding&sektion=5&manpath=FreeBSD+Ports+15.1.quarterly>