home | help
resperf(1)		     General Commands Manual		      resperf(1)

NAME
     resperf - test the resolution performance of a caching DNS server

SYNOPSIS
     resperf-report [-a local_addr] [-d datafile] [-R] [-M mode]
     [-s server_addr] [-p port] [-x local_port] [-t timeout] [-b bufsize]
     [-f family] [-e] [-D] [-y [alg:]name:secret] [-h] [-i interval]
     [-m max_qps] [-r rampup_time] [-c constant_traffic_time] [-L max_loss]
     [-C clients] [-q max_outstanding] [-F fall_behind] [-v] [-W]
     [-O option=value]

     resperf [-a local_addr] [-d datafile] [-R] [-M mode] [-s server_addr]
     [-p port] [-x local_port] [-t timeout] [-b bufsize] [-f family] [-e] [-D]
     [-y [alg:]name:secret] [-h] [-i interval] [-m max_qps] [-P plot_data_file]
     [-r rampup_time] [-c constant_traffic_time] [-L max_loss] [-C clients]
     [-q max_outstanding] [-F fall_behind] [-v] [-W] [-O option=value]

DESCRIPTION
     resperf is a companion tool to dnsperf.  dnsperf was primarily designed for
     benchmarking  authoritative servers, and it does not work well with caching
     servers that are talking to the live Internet.  One reason for this is that
     dnsperf uses a "self-pacing" approach, which is  based  on  the  assumption
     that  you	can keep the server 100% busy simply by sending it a small burst
     of back-to-back queries to fill up network buffers, and  then  send  a  new
     query  whenever  you get a response back.	This approach works well for au-
     thoritative servers that process queries in order and one	at  a  time;  it
     also works pretty well for a caching server in a closed laboratory environ-
     ment  talking to a simulated Internet that's all on the same LAN.	Unfortu-
     nately, it does not work well with a caching server talking to  the  actual
     Internet,	which  may  need  to work on thousands of queries in parallel to
     achieve its maximum throughput.  There have been numerous attempts  to  use
     dnsperf  (or  its	predecessor,  queryperf)  for  benchmarking live caching
     servers, usually with poor results.  Therefore, a	separate  tool	designed
     specifically for caching servers is needed.

   How resperf works
     Unlike  the "self-pacing" approach of dnsperf, resperf works by sending DNS
     queries at a controlled, steadily increasing  rate.   By  default,  resperf
     will send traffic for 60 seconds, linearly increasing the amount of traffic
     from zero to 100,000 queries per second (or max_qps).

     During  the  test,  resperf listens for responses from the server and keeps
     track of response rates, failure rates, and latencies.  It will  also  con-
     tinue  listening  for  responses  for an additional 40 seconds after it has
     stopped sending traffic, so that there is time for the server to respond to
     the last queries sent.  This time period was chosen to be longer  than  the
     overall  query  timeout  of both Nominum CacheServe and current versions of
     BIND.

     If the test is successful, the query rate will at some point exceed the ca-
     pacity of the server and queries will be dropped, causing the response rate
     to stop growing or even decrease as the query rate increases.

     The result of the test is a set of measurements of the query rate, response
     rate, failure response rate, and average  query  latency  as  functions  of
     time.

   What you will need
     Benchmarking  a  live  caching  server is serious business.  A fast caching
     server like Nominum CacheServe, resolving	a  mix	of  cacheable  and  non-
     cacheable	queries typical of ISP customer traffic, is capable of resolving
     well over 1,000,000 queries per second.  In the process, it will send  more
     than  40,000  queries  per second to authoritative servers on the Internet,
     and receive responses to most of them.  Assuming an average request size of
     50 bytes and a response size of 150 bytes, this amounts to some  1216  Mbps
     of  outgoing and 448 Mbps of incoming traffic.  If your Internet connection
     can't handle the bandwidth, you will end up measuring the speed of the con-
     nection, not the server, and may saturate the connection causing a degrada-
     tion in service for other users.

     Make sure there is no stateful firewall between the server and  the  Inter-
     net,  because  most of them can't handle the amount of UDP traffic the test
     will generate and will end up dropping packets, skewing the  test	results.
     Some will even lock up or crash.

     You should run resperf on a machine separate from the server under test, on
     the  same LAN.  Preferably, this should be a Gigabit Ethernet network.  The
     machine running resperf should be at least as fast  as  the  machine  being
     tested; otherwise, it may end up being the bottleneck.

     There  should  be no other applications running on the machine running res-
     perf.  Performance testing at the traffic levels involved is essentially  a
     hard  real-time  application  -  consider	the fact that at a query rate of
     100,000 queries per second, if resperf gets delayed by just 1/100 of a sec-
     ond, 1000 incoming UDP packets will arrive in the meantime.  This	is  more
     than  most  operating  systems  will  buffer,  which  means packets will be
     dropped.

     Because the granularity of the timers provided by operating systems is typ-
     ically too coarse to accurately schedule packet transmissions  at	sub-mil-
     lisecond  intervals,  resperf  will busy-wait between packet transmissions,
     constantly polling for responses in the meantime.	Therefore, it is  normal
     for  resperf to consume 100% CPU during the whole test run, even during pe-
     riods where query rates are relatively low.

     You will also need a set of test queries in the dnsperf file  format.   See
     the  dnsperf man page for instructions on how to construct this query file.
     To make the test as realistic as possible, the queries  should  be  derived
     from  recorded  production  client  DNS traffic, without removing duplicate
     queries or other filtering.  With the default settings, resperf will use up
     to 3 million queries in each test run.

     If the caching server to be tested has a configurable limit on  the  number
     of  simultaneous  resolutions,  like the max-recursive-clients statement in
     Nominum CacheServe or the recursive-clients option  in  BIND  9,  you  will
     probably have to increase it.  As a starting point, we recommend a value of
     10000  for  Nominum  CacheServe and 100000 for BIND 9.  Should the limit be
     reached, it will show up in the plots as an increase in the number of fail-
     ure responses.

     The server being tested should be restarted at the beginning of  each  test
     to make sure it is starting with an empty cache.  If the cache already con-
     tains  data from a previous test run that used the same set of queries, al-
     most all queries will be answered from the cache, yielding inflated perfor-
     mance numbers.

     To use the resperf-report script, you need to have gnuplot installed.  Make
     sure your installed version of gnuplot supports the  png  terminal  driver.
     If  your  gnuplot	doesn't support png but does support gif, you can change
     the line saying terminal=png in the resperf-report script to terminal=gif.

   Running the test
     resperf is typically invoked via the resperf-report script, which will  run
     resperf  with its output redirected to a file and then automatically gener-
     ate an illustrated report in HTML format.	Command line arguments given  to
     resperf-report will be passed on unchanged to resperf.

     When  running  resperf-report, you will need to specify at least the server
     IP address and the query data file.  A typical invocation will look like

	    resperf-report -s 10.0.0.2 -d queryfile

     With default settings, the test run will take at most 100 seconds (60  sec-
     onds  of  ramping up traffic and then 40 seconds of waiting for responses),
     but in practice, the 60-second traffic phase will usually be cut short.  To
     be precise, resperf can transition from the traffic-sending  phase  to  the
     waiting-for-responses phase in three different ways:

     * Running	for the full allotted time and successfully reaching the maximum
       query rate (by default, 60 seconds and 100,000 qps, respectively).  Since
       this is a very high query rate, this will  rarely  happen  (with  today's
       hardware);  one of the other two conditions listed below will usually oc-
       cur first.

     * Exceeding 65,536 outstanding queries.  This often happens as a result  of
       (successfully) exceeding the capacity of the server being tested, causing
       the excess queries to be dropped.  The limit of 65,536 queries comes from
       the  number  of possible values for the ID field in the DNS packet.  res-
       perf needs to allocate a unique ID for each  outstanding  query,  and  is
       therefore  unable  to  send further queries if the set of possible IDs is
       exhausted.

     * When resperf finds itself unable to send queries  fast  enough.	 resperf
       will notice if it is falling behind in its scheduled query transmissions,
       and  if	this  backlog reaches 1000 queries, it will print a message like
       "Fell behind by 1000 queries" (or whatever the actual number  is  at  the
       time) and stop sending traffic.

     Regardless  of  which  of	the  above conditions caused the traffic-sending
     phase of the test to end, you should examine the resulting  plots	to  make
     sure  the	server's  response  rate is flattening out toward the end of the
     test.  If it is not, then you are not loading the server  enough.	 If  you
     are  getting  the "Fell behind" message, make sure that the machine running
     resperf is fast enough and has no other applications running.

     You should also monitor the CPU usage of the server under test.  It  should
     reach  close  to  100% CPU at the point of maximum traffic; if it does not,
     you most likely have a bottleneck in some other part of  your  test  setup,
     for example, your external Internet connection.

     The  report  generated  by resperf-report will be stored with a unique file
     name based on the current date and time, e.g., 20060812-1550.html.  The PNG
     images of the plots and other auxiliary files will be  stored  in	separate
     files beginning with the same date-time string.  To view the report, simply
     open the .html file in a web browser.

     If you need to copy the report to a separate machine for viewing, make sure
     to  copy  the  .png files along with the .html file (or simply copy all the
     files, e.g., using scp 20060812-1550.* host:directory/).

   Interpreting the report
     The .html file produced by resperf-report consists of  two  sections.   The
     first  section,  "Resperf output", contains output from the resperf program
     such as progress messages, a summary of the  command  line  arguments,  and
     summary statistics.  The second section, "Plots", contains two plots gener-
     ated by gnuplot: "Query/response/failure rate" and "Latency".

     The "Query/response/failure rate" plot contains three graphs.  The "Queries
     sent  per	second"  graph	shows  the  amount  of traffic being sent to the
     server; this should be very close to a straight diagonal  line,  reflecting
     the linear ramp-up of traffic.

     The  "Total  responses  received  per  second"  graph shows how many of the
     queries received a response from the server.  All	responses  are	counted,
     whether successful (NOERROR or NXDOMAIN) or not (e.g., SERVFAIL).

     The  "Failure  responses  received  per second" graph shows how many of the
     queries received a failure response.  A response  is  considered  to  be  a
     failure if its RCODE is neither NOERROR nor NXDOMAIN.

     By  visually  inspecting  the graphs, you can get an idea of how the server
     behaves under increasing load.  The "Total responses received  per  second"
     graph  will  initially  closely  follow the "Queries sent per second" graph
     (often rendering it invisible in the plot as the two graphs are plotted  on
     top  of  one another), but when the load exceeds the server's capacity, the
     "Total responses received per second" graph may diverge from  the	"Queries
     sent per second" graph and flatten out, indicating that some of the queries
     are being dropped.

     The  "Failure  responses  received  per  second" graph will normally show a
     roughly linear ramp close to the bottom of the plot with some random  fluc-
     tuation,  since typical query traffic will contain some small percentage of
     failing queries randomly interspersed with the successful ones.  As the to-
     tal traffic increases, the number of failures will increase proportionally.

     If the "Failure responses received per second" graph turns sharply upwards,
     this can be another indication that the load has exceeded the server's  ca-
     pacity.  This will happen if the server reacts to overload by sending SERV-
     FAIL  responses  rather than by dropping queries.	Since Nominum CacheServe
     and BIND 9 will both respond with SERVFAIL when they exceed  their  max-re-
     cursive-clients or recursive-clients limit, respectively, a sudden increase
     in the number of failures could mean that the limit needs to be increased.

     The  "Latency" plot contains a single graph marked "Average latency".  This
     shows how the latency varies during the course of the test.  Typically, the
     latency graph will exhibit a downwards trend because the cache hit rate im-
     proves as ever more responses are cached during the test, and  the  latency
     for  a  cache hit is much smaller than for a cache miss.  The latency graph
     is provided as an aid in determining the point where the server gets  over-
     loaded,  which  can  be seen as a sharp upwards turn in the graph.  The la-
     tency graph is not intended for making  absolute  latency	measurements  or
     comparisons  between servers; the latencies shown in the graph are not rep-
     resentative of production latencies due to the initially  empty  cache  and
     the deliberate overloading of the server towards the end of the test.

     Note  that all measurements are displayed on the plot at the horizontal po-
     sition corresponding to the point in time when the query was sent, not when
     the response (if any) was received.  This makes it it easy to  compare  the
     query and response rates; for example, if no queries are dropped, the query
     and  response  graphs  will  be identical.  As another example, if the plot
     shows 10% failure responses at t=5 seconds, this  means  that  10%  of  the
     queries  sent  at	t=5  seconds  eventually failed, not that 10% of the re-
     sponses received at t=5 seconds were failures.

   Determining the server's maximum throughput
     Often, the goal of running resperf is to  determine  the  server's  maximum
     throughput,  in other words, the number of queries per second it is capable
     of handling.  This is not always an easy task, because as a server is  dri-
     ven  into	overload, the service it provides may deteriorate gradually, and
     this deterioration can manifest itself either as queries being dropped,  as
     an increase in the number of SERVFAIL responses, or an increase in latency.
     The  maximum  throughput  may be defined as the highest level of traffic at
     which the server still provides an acceptable level of  service,  but  that
     means you first need to decide what an acceptable level of service means in
     terms of packet drop percentage, SERVFAIL percentage, and latency.

     The  summary  statistics in the "Resperf output" section of the report con-
     tains a "Maximum throughput" value which by default is determined from  the
     maximum  rate at which the server was able to return responses, without re-
     gard to the number of queries being dropped or failing at that point.  This
     method of throughput measurement has the advantage of  simplicity,  but  it
     may or may not be appropriate for your needs; the reported value should al-
     ways  be validated by a visual inspection of the graphs to ensure that ser-
     vice has not already deteriorated unacceptably before the maximum	response
     rate  is  reached.   It  may  also  be helpful to look at the "Lost at that
     point" value in the summary statistics; this indicates  the  percentage  of
     the  queries that was being dropped at the point in the test when the maxi-
     mum throughput was reached.

     Alternatively, you can make resperf report the throughput at the  point  in
     the  test where the percentage of queries dropped exceeds a given limit (or
     the maximum as above if the limit is never exceeded).  This can be  a  more
     realistic	indication of how much the server can be loaded while still pro-
     viding an acceptable level of service.  This is done using the  -L  command
     line option; for example, specifying -L 10 makes resperf report the highest
     throughput  reached  before the server starts dropping more than 10% of the
     queries.

     There is no corresponding way of automatically constraining  results  based
     on the number of failed queries, because unlike dropped queries, resolution
     failures  will  occur  even  when the the server is not overloaded, and the
     number of such failures is heavily dependent on the query data and  network
     conditions.   Therefore,  the  plots should be manually inspected to ensure
     that there is not an abnormal number of failures.

GENERATING CONSTANT TRAFFIC
     In addition to ramping up traffic linearly, resperf also has the capability
     to send a constant stream of traffic.  This can be useful when  using  res-
     perf  for	tasks other than performance measurement; for example, it can be
     used to "soak test" a server by subjecting it to a sustained  load  for  an
     extended period of time.

     To  generate  a  constant traffic load, use the -c command line option, to-
     gether with the -m option which specifies the desired constant query  rate.
     For  example, to send 10000 queries per second for an hour, use -m 10000 -c
     3600.  This will include the usual 30-second gradual ramp-up of traffic  at
     the beginning, which may be useful to avoid initially overwhelming a server
     that  is  starting  with an empty cache.  To start the onslaught of traffic
     instantly, use -m 10000 -c 3600 -r 0.

     To be precise, resperf will do a linear ramp-up of traffic  from  0  to  -m
     queries  per  second  over a period of -r seconds, followed by a plateau of
     steady traffic at -m queries per second lasting for -c seconds, followed by
     waiting for responses for an extra 40 seconds.  Either the ramp-up  or  the
     plateau can be suppressed by supplying a duration of zero seconds with -r 0
     and -c 0, respectively.  The latter is the default.

     Sending  traffic at high rates for hours on end will of course require very
     large amounts of input data.  Also, a long-running  test  will  generate  a
     large  amount of plot data, which is kept in memory for the duration of the
     test.  To reduce the memory usage and the size of the plot  file,	consider
     increasing  the  interval between measurements from the default of 0.5 sec-
     onds using the -i option in long-running tests.

     When using resperf for long-running tests, it is important that the traffic
     rate specified using the -m is one that both resperf itself and the  server
     under test can sustain.  Otherwise, the test is likely to be cut short as a
     result  of  either  running  out  of query IDs (because of large numbers of
     dropped queries) or of resperf falling behind its transmission schedule.

   Using DNS-over-HTTPS
     When using DNS-over-HTTPS you must set the -O doh-uri=... to something that
     works with the server you're sending to.  Also note that the value for max-
     imum outstanding queries will be used to  control	the  maximum  concurrent
     streams within the HTTP/2 connection.

OPTIONS
     Because  the resperf-report script passes its command line options directly
     to the resperf programs, they both accept the same set of options, with one
     exception: resperf-report automatically adds an appropriate -P to the  res-
     perf command line, and therefore does not itself take a -P option.

     -d datafile
	    Specifies  the input data file.  If not specified, resperf will read
	    from standard input.

     -R
	    Reopen the datafile if it runs out of data	before	the  testing  is
	    completed.	 This  allows  for  long running tests on very small and
	    simple query datafile.

     -M mode
	    Specifies the transport mode to use, "udp", "tcp", "dot"  or  "doh".
	    Default is "udp".

     -s server_addr
	    Specifies  the  name or address of the server to which requests will
	    be sent.  The default is the loopback address, 127.0.0.1.

     -p port
	    Sets the port on which the DNS packets are sent.  If not  specified,
	    the standard DNS port (udp/tcp 53, DoT 853, DoH 443) is used.

     -a local_addr
	    Specifies  the  local  address from which to send requests.  The de-
	    fault is the wildcard address.

     -x local_port
	    Specifies the local port from which to send requests.   The  default
	    is the wildcard port (0).

	    If	acting	as  multiple clients and the wildcard port is used, each
	    client will use a different random port.  If a  port  is  specified,
	    the  clients  will	use a range of ports starting with the specified
	    one.

     -t timeout
	    Specifies the request timeout value, in seconds.   resperf	will  no
	    longer  wait  for a response to a particular request after this many
	    seconds have elapsed.  The default is 45 seconds.

	    resperf times out unanswered requests in order to reclaim query  IDs
	    so	that  the query ID space will not be exhausted in a long-running
	    test, such as when "soak testing" a server for an day with -m  10000
	    -c	86400.	 The timeouts and the ability to tune them are of little
	    use in the more typical use case of a performance test lasting  only
	    a minute or two.

	    The  default  timeout of 45 seconds was chosen to be longer than the
	    query timeout of current caching servers.  Note that this is  longer
	    than  the  corresponding default in dnsperf, because caching servers
	    can take many orders of magnitude longer to answer a query than  au-
	    thoritative servers do.

	    If a short timeout is used, there is a possibility that resperf will
	    receive a response after the corresponding request has timed out; in
	    this case, a message like Warning: Received a response with an unex-
	    pected id: 141 will be printed.

     -b bufsize
	    Sets  the  size  of  the socket's send and receive buffers, in kilo-
	    bytes.  If not specified, the operating system's default is used.

     -f family
	    Specifies the address family used for sending DNS packets.	The pos-
	    sible values are "inet", "inet6", or "any".  If "any"  (the  default
	    value)  is	specified,  resperf will use whichever address family is
	    appropriate for the server it is sending packets to.

     -e
	    Enables EDNS0 [RFC2671], by adding an  OPT	record	to  all  packets
	    sent.

     -D
	    Sets  the  DO  (DNSSEC  OK) bit [RFC3225] in all packets sent.  This
	    also enables EDNS0, which is required for DNSSEC.

     -y [alg:]name:secret
	    Add a TSIG record [RFC2845] to all packets sent, using the specified
	    TSIG key algorithm, name and secret, where the algorithm defaults to
	    hmac-md5 and the secret is expressed as a base-64 encoded string.

     -h
	    Print a usage statement and exit.

     -i interval
	    Specifies the time interval between data points in	the  plot  file.
	    The default is 0.5 seconds.

     -m max_qps
	    Specifies  the  target  maximum  query rate (in queries per second).
	    This should be higher than the expected maximum  throughput  of  the
	    server  being  tested.   Traffic will be ramped up at a linearly in-
	    creasing rate until this value is reached, or until one of the other
	    conditions described in the section "Running the test" occurs.   The
	    default is 100000 queries per second.

     -P plot_data_file
	    Specifies  the  name  of  the  plot  data file.  The default is res-
	    perf.gnuplot.

     -r rampup_time
	    Specifies the length of time over which traffic will be  ramped  up.
	    The default is 60 seconds.

     -c constant_traffic_time
	    Specifies  the  length  of	time for which traffic will be sent at a
	    constant rate following the initial ramp-up.  The default is 0  sec-
	    onds, meaning no sending of traffic at a constant rate will be done.

     -L max_loss
	    Specifies  the maximum acceptable query loss percentage for purposes
	    of determining the maximum throughput value.  The default  is  100%,
	    meaning that resperf will measure the maximum throughput without re-
	    gard to query loss.

     -C clients
	    Act  as  multiple clients.	Requests are sent from multiple sockets.
	    The default is to act as 1 client.

     -q max_outstanding
	    Sets the maximum number of outstanding requests.  resperf will  stop
	    ramping  up traffic when this many queries are outstanding.  The de-
	    fault is 64k, and the limit is 64k per client.

     -F fall_behind
	    Sets the maximum number of queries that can fall behind being  sent.
	    resperf  will  stop when this many queries should have been sent and
	    it can be relative easy to hit if max_qps is set too high.	The  de-
	    fault is 1000 and setting it to zero (0) disables the check.

     -v
	    Enables  verbose  mode to report about network readiness and conges-
	    tion.

     -W
	    Log warnings and errors to standard output instead of standard error
	    making it easier for script, test and automation to capture all out-
	    put.

     -O option=value
	    Set an extended long option for various things to control  different
	    aspects  of  testing  or  protocol	modules, see EXTENDED OPTIONS in
	    dnsperf(1) for list of available options.

THE PLOT DATA FILE
     The plot data file is written by the resperf program and contains the  data
     to  be  plotted using gnuplot.  When running resperf via the resperf-report
     script, there is no need for the user to deal with this file directly,  but
     its  format  and  contents are documented here for completeness and in case
     you wish to run resperf directly and use its output for purposes other than
     viewing it with gnuplot.

     The first line of the file is a comment identifying the fields.  It may  be
     recognized as a comment by its leading hash sign (#).

     Subsequent  lines contain the actual plot data.  For purposes of generating
     the plot data file, the test run is divided into time intervals of 0.5 sec-
     onds (or some other length of time specified with the -i command  line  op-
     tion).   Each  line corresponds to one such interval, and contains the fol-
     lowing values as floating-point numbers:

     Time
	    The midpoint of this time interval, in seconds since  the  beginning
	    of the run

     Target queries per second
	    The  number  of queries per second scheduled to be sent in this time
	    interval

     Actual queries per second
	    The number of queries per second actually sent in this time interval

     Responses per second
	    The number of responses received corresponding to  queries	sent  in
	    this time interval, divided by the length of the interval

     Failures per second
	    The  number  of  responses received corresponding to queries sent in
	    this time interval and having an RCODE other than NOERROR  or  NXDO-
	    MAIN, divided by the length of the interval

     Average latency
	    The average time between sending the query and receiving a response,
	    for queries sent in this time interval

     Connections
	    The  number  of  connections  done, including re-connections, during
	    this time interval.  This is only relevant	to  connection	oriented
	    protocols, such as TCP and DoT.

     Average connection latency
	    The  average time between starting to connect and having the connec-
	    tion ready for sending queries to, for this time interval.	This  is
	    only relevant to connection oriented protocols, such as TCP and DoT.

SEE ALSO
     dnsperf(1)

AUTHOR
     Nominum, Inc.

     Maintained by DNS-OARC

	    https://www.dns-oarc.net/

BUGS
     For issues and feature requests please use:

	    https://github.com/DNS-OARC/dnsperf/issues

     For question and help please use:

	    admin@dns-oarc.net

resperf 			     2.14.0			      resperf(1)

home | help