FreeBSD Manual Pages
AMAVIS-LOGWATCH(1) General Commands Manual AMAVIS-LOGWATCH(1) NAME amavis-logwatch - An Amavisd-new log parser and analysis utility SYNOPSIS amavis-logwatch [options] [logfile ...] DESCRIPTION The amavis-logwatch(1) utility is an Amavisd-new log parser that produces summaries, details, and statistics regarding the operation of Amavisd-new (henceforth, simply called Amavis). This utility can be used as a standalone program, or as a Logwatch filter module to produce Amavisd-new summary and detailed reports from within Log- watch. Amavis-logwatch is able to produce a wide range of reports with data grouped and sorted as much as possible to reduce noise and highlight pat- terns. Brief summary reports provide a quick overview of general Amavis operations and message delivery, calling out warnings that may require at- tention. Detailed reports provide easy to scan, hierarchically-arranged and organized information, with as much or little detail as desired. Much of the interesting data is available when Amavis' $log_level is set to at least 2. See Amavis Log Level below. Amavis-logwatch outputs two principal sections: a Summary section and a De- tailed section. For readability and quick scanning, all event or hit counts appear in the left column, followed by brief description of the event type, and finally additional statistics or count representations may appear in the rightmost column. The following segment from a sample Summary report illustrates: ****** Summary ******************************************** 9 Miscellaneous warnings 20313 Total messages scanned ---------------- 100.00% 1008.534M Total bytes scanned 1,057,524,252 ======== ================================================ 1190 Blocked ------------------------------- 5.86% 18 Malware blocked 0.09% 4 Banned name blocked 0.02% 416 Spam blocked 2.05% 752 Spam discarded (no quarantine) 3.70% 19123 Passed -------------------------------- 94.14% 47 Bad header passed 0.23% 19076 Clean passed 93.91% ======== ================================================ 18 Malware ------------------------------- 0.09% 18 Malware blocked 0.09% 4 Banned -------------------------------- 0.02% 4 Banned file blocked 0.02% 1168 Spam ---------------------------------- 5.75% 416 Spam blocked 2.05% 752 Spam discarded (no quarantine) 3.70% 19123 Ham ----------------------------------- 94.14% 47 Bad header passed 0.23% 19076 Clean passed 93.91% ======== ================================================ 1982 SpamAssassin bypassed 32 Released from quarantine 2 DSN notification (debug supplemental) 2 Bounce unverifiable 2369 Whitelisted 2 Blacklisted 12 MIME error 58 Bad header (debug supplemental) 40 Extra code modules loaded at runtime The report indicates there were 9 general warnings, and Amavis scanned a total of 20313 messages for a total of 1008.53 megabytes or 1,057,524,252 bytes. The next summary groups shows the Blocked / Passed overview, with 1190 Blocked messages (broken down as 18 messages blocked as malware, 4 messages with banned names, 416 spam messages, and 752 discarded messages), and 19123 Passed messages (47 messages with bad headers and 19076 clean messages). The next (optional) summary grouping shows message disposition by contents category. There were 18 malware messages and 4 banned file messages (all blocked), 1168 Spam messages, of which 416 were blocked (quarantined) and 752 discarded. Finally, there were 19123 messages consdidered to be Ham (i.e. not spam), 47 of which contained bad headers. Additional count summaries for a variety of events are also listed. There are dozens of sub-sections available in the Detailed report, each of whose output can be controlled in various ways. Each sub-section attempts to group and present the most meaningful data at superior levels, while pushing less useful or noisy data towards inferior levels. The goal is to provide as much benefit as possible from smart grouping of data, to allow faster report scanning, pattern identification, and problem solving. Data is always sorted in descending order by count, and then numerically by IP address or alphabetically as appropriate. The following Spam blocked segment from a sample Detailed report illus- trates the basic hierarchical level structure of amavis-logwatch: ****** Detailed ******************************************* 19346 Spam blocked ----------------------------------- 756 from@example.com 12 10.0.0.2 12 <> 12 192.168.2.2 12 <> 5 192.168.2.1 ... The amavis-logwatch utility reads from STDIN or from the named Amavis log- file. Multiple logfile arguments may be specified, each processed in or- der. The user running amavis-logwatch must have read permission on each named log file. Options The options listed below affect the operation of amavis-logwatch. Options specified later on the command line override earlier ones. Any option may be abbreviated to an unambiguous length. --[no]autolearn --show_autolearn boolean Enables (disables) output of the autolearn report. This report is only available if the default Amavis $log_templ has been modified to provide autolearn results in log entries. This can be done by un- commenting two lines in the Amavis program itself (where the default log templates reside), or by correctly adding the $log_templ vari- able to the amavisd.conf file. See Amavis' README.customize and search near the end of the Amavisd program for "autolearn". --[no]by_ccat_summary --show_by_ccat_summary boolean Enables (disables) the by contents category summary in the Summary section. Default: enabled. -f config_file --config_file config_file Use an alternate configuration file config_file instead of the de- fault. This option may be used more than once. Multiple configura- tion files will be processed in the order presented on the command line. See CONFIGURATION FILE below. --debug keywords Output debug information during the operation of amavis-logwatch. The parameter keywords is one or more comma or space separated key- words. To obtain the list of valid keywords, use --debug xxx where xxx is any invalid keyword. --detail level Sets the maximum detail level for amavis-logwatch to level. This option is global, overriding any other output limiters described be- low. The amavis-logwatch utility produces a Summary section, a Detailed section, and additional report sections. With level less than 5, amavis-logwatch will produce only the Summary section. At level 5 and above, the Detailed section, and any additional report sections are candidates for output. Each incremental increase in level gen- erates one additional hierarchical sub-level of output in the De- tailed section of the report. At level 10, all levels are output. Lines that exceed the maximum report width (specified with max_re- port_width) will be cut. Setting level to 11 will prevent lines in the report from being cut (see also --line_style). --[no]first_recip_only --show_first_recip_only boolean Specifies whether or not to sort by, and show, only the first recip- ient when a scanned messages contains multiple recipients. --help Print usage information and a brief description about command line options. --ipaddr_width width Specifies that IP addresses in address/hostname pairs should be printed with a field width of width characters. Increasing the de- fault may be useful for systems using long IPv6 addresses. -l limiter=levelspec --limit limiter=levelspec Sets the level limiter limiter with the specification levelspec. --line_style style Specifies how to handle long report lines. Three styles are avail- able: full, truncate, and wrap. Setting style to full will prevent cutting lines to max_report_width; this is what occurs when detail is 11 or higher. When style is truncate (the default), long lines will be truncated according to max_report_width. Setting style to wrap will wrap lines longer than max_report_width such that left column hit counts are not obscured. This option takes precedence over the line style implied by the detail level. The options --full, --truncate, and --wrap are synonyms. --nodetail Disables the Detailed section of the report, and all supplemental reports. This option provides a convenient mechanism to quickly disable all sections under the Detailed report, where subsequent command line options may re-enable one or more sections to create specific reports. --sarules `S,H' --sarules default Enables the SpamAssassin Rules Hit report. The comma-separated S and H arguments are top N values for the Spam and Ham reports, re- spectively, and can be any integer greater than or equal to 0, or the keyword all. The keyword default uses the built-in default val- ues. --nosarules Disables the SpamAssassin Rules Hit report. --sa_timings nrows Enables the SpamAssassin Timings percentiles report. The report can be limited to the top N rows with the nrows argument. This report requires Amavis 2.6+ and SpamAssassin 3.3+. --sa_timings_percentiles `P1 [P2 ...]' Specifies the percentiles shown in the SpamAssassin Timings report. The arguments P1 ... are integers from 0 to 100 inclusive. Their order will be preserved in the report. --nosa_timings Disables the SpamAssassin Timings report. --version Print amavis-logwatch version information. --score_frequencies `B1 [B2 ...]' --score_frequencies default Enables the Spam Score Frequency report. The arguments B1 ... are frequency distribution buckets, and can be any real numbers. Their order will be preserved in the report. The keyword default uses the built-in default values. --noscore_frequencies Disables the Spam Score Frequency report. --score_percentiles `P1 [P2 ...]' --score_percentiles default Enables the Spam Score Percentiles report. The arguments P1 ... specify the percentiles shown in the report, and are integers from 0 to 100 inclusive. The keyword default uses the built-in default values. --noscore_percentiles Disables the Spam Score Percentiles report. --[no]sect_vars --show_sect_vars boolean Enables (disables) supplementing each Detailed section title with the name of that section's level limiter. The name displayed is the command line option (or configuration file variable) used to limit that section's output. With the large number of level limiters available in amavis-logwatch, this a convenient mechanism for deter- mining exactly which level limiter affects a section. --[no]startinfo --show_startinfo boolean Enables (disables) the Amavis startup report showing most recent Amavis startup details. --[no]summary --show_summary Enables (disables) displaying of the the Summary section of the re- port. The variable Amavis_Show_Summary in used in a configuration file. --syslog_name namepat Specifies the syslog service name that amavis-logwatch uses to match syslog lines. Only log lines whose service name matches the perl regular expression namepat will be used by amavis-logwatch; all non- matching lines are silently ignored. This is useful when a pre-in- stalled Amavis package uses a name other than the default (amavis). Note: if you use parenthesis in your regular expression, be sure they are cloistering and not capturing: use (?:pattern) instead of (pattern). --timings percent Enables the Amavis Scan Timings percentiles report. The report can be top N-percent limited with the percent argument. --timings_percentiles `P1 [P2 ...]' Specifies the percentiles shown in the Scan Timings report. The ar- guments P1 ... are integers from 0 to 100 inclusive. Their order will be preserved in the report. --notimings Disables the Amavis Scan Timings report. --version Print amavis-logwatch version information. Level Limiters The output of every section in the Detailed report is controlled by a level limiter. The name of the level limiter variable will be output when the sect_vars option is set. Level limiters are set either via command line in standalone mode with --limit limiter=levelspec option, or via configuration file variable $amavis_limiter=levelspec. Each limiter requires a levelspec argument, which is described below in LEVEL CONTROL. The list of level limiters is shown below. Amavis major contents category (ccatmajor) sections, listed in order of priority: VIRUS, BANNED, UNCHECKED, SPAM, SPAMMY, BADH, OVERSIZED, MTA, CLEAN. MalwareBlocked MalwarePassed Blocked or passed messages that contain malware (ccatmajor: VIRUS). BannedNameBlocked BannedNamePassed Blocked or passed messages that contain banned names in MIME parts (ccatmajor: BANNED). UncheckedBlocked UncheckedPassed Blocked or passed messages that were not checked by a virus scanner or SpamAssassin (Amavis ccatmajor: UNCHECKED). SpamBlocked SpamPassed Blocked or passed messages that were considered spam that reached kill level (Amavis ccatmajor: SPAM) SpammyBlocked SpammyPassed Blocked or passed messages that were considered spam, but did not reach kill level (Amavis ccatmajor: SPAMMY) BadHeaderBlocked BadHeaderPassed Blocked or passed messages that contain bad mail headers (ccatmajor: BAD-HEADER). OversizedBlocked OversizedPassed Blocked or passed messages that were considered oversized (Amavis ccatmajor: OVERSIZED). MtaBlocked MtaPassed Blocked or passed messages due to failure to re-inject to MTA (Amavis ccatmajor: MTA-BLOCKED). Occurrences of this event indi- cates a configuration problem. [ note: I don't believe mtapassed occurs, but exists for completeness.] OtherBlocked OtherPassed Blocked or passed messages that are not any of other major contents categories (Amavis ccatmajor: OTHER). TempFailBlocked TempfailPassed Blocked or passed messages that had a temporary failure (Amavis ccatmajor: TEMPFAIL) CleanBlocked CleanPassed Messages blocked or passed which were considered clean (Amavis ccat- major: CLEAN; i.e. non-spam, non-viral). Other sections, arranged alphabetically: AvConnectFailure Problems connecting to Anti-Virus scanner(s). AvTimeout Timeouts awaiting responses from Anti-Virus scanner(s). ArchiveExtract Archive extraction problems. BadHeaderSupp Supplemental debug information regarding messages containing bad mail headers. Bayes Messages frequencies by Bayesian probability buckets. BadAddress Invalid mail address syntax. Blacklisted Messages that were (soft-)blacklisted. See also Whitelisted below. BounceKilled BounceRescued BounceUnverifiable Disposition of incoming bounce messages (DSNs). ContentType MIME attachment breakdown by type/subtype. DccError Errors encountered with or returned by DCC. DefangError Errors encountered during defang process. Defanged Messages defanged (rendered harmless). DsnNotification Errors encountered during attempt to send delivery status notifica- tion. DsnSuppressed Delivery status notification (DSN) intentionally suppressed. ExtraModules Additional code modules Amavis loaded during runtime. FakeSender Forged sender addresses, as determimed by Amavis. Fatal Fatal events. These are presented at the top of the report, as they may require attention. LocalDeliverySkipped Failures delivering to a local address. MalwareByScanner Breakdown of malware by scanner(s) that detected the malware. MimeError Errors encountered during MIME extraction. Panic Panic events. These are presented at the top of the report, as they may require attention. p0f Passive fingerprint (p0f) hits, grouped by mail contents type (virus, unchecked, banned, spam, ham), next by operating system genre, and finally by IP address. Note: Windows systems are refined by Windows OS version, whereas versions of other operating systems are grouped generically. Released Messages that were released from Amavis quarantine. SADiagsDiagnostics as reported from SpamAssassin. SmtpResponse SMTP responses received during dialog with MTA. These log entries are primarly debug. TmpPreserved Temporary directories preserved by Amavis when some component en- counters a problem or failure. Directories listed and their corre- sponding log entries should be evaluated for problems. VirusScanSkipped Messages that could not be scanned by a virus scanner. WarningWarning events not categorized in specific warnings below. These are presented at the top of the report, as they may require atten- tion. WarningAddressModified Incomplete email addresses modified by Amavis for safety. WarningNoQuarantineId Attempts to release a quarantined message that did not contain an X- Quarantine-ID header. WarningSecurity levelspec Insecure configuration or utility used by Amavis. WarningSmtpShutdown Failures during SMTP conversation with MTA. WarningSql Failures to communicate with, or error replies from, SQL service. Whitelisted Messages that were (soft-)whitelisted. See also Blacklisted above. LEVEL CONTROL The Detailed section of the report consists of a number of sub-sections, each of which is controlled both globally and independently. Two settings influence the output provided in the Detailed report: a global detail level (specified with --detail) which has final (big hammer) output-limiting con- trol over the Detailed section, and sub-section specific detail settings (small hammer), which allow further limiting of the output for a sub-sec- tion. Each sub-section may be limited to a specific depth level, and each sub-level may be limited with top N or threshold limits. The levelspec ar- gument to each of the level limiters listed above is used to accomplish this. It is probably best to continue explanation of sub-level limiting with the following well-known outline-style hierarchy, and some basic examples: level 0 level 1 level 2 level 3 level 4 level 4 level 2 level 3 level 4 level 4 level 4 level 3 level 4 level 3 level 1 level 2 level 3 level 4 The simplest form of output limiting suppresses all output below a speci- fied level. For example, a levelspec set to "2" shows only data in levels 0 through 2. Think of this as collapsing each sub-level 2 item, thus hid- ing all inferior levels (3, 4, ...), to yield: level 0 level 1 level 2 level 2 level 1 level 2 Sometimes the volume of output in a section is too great, and it is useful to suppress any data that does not exceed a certain threshold value. Con- sider a dictionary spam attack, which produces very lengthy lists of hit- once recipient email or IP addresses. Each sub-level in the hierarchy can be threshold-limited by setting the levelspec appropriately. Setting lev- elspec to the value "2::5" will suppress any data at level 2 that does not exceed a hit count of 5. Perhaps producing a top N list, such as top 10 senders, is desired. A lev- elspec of "3:10:" limits level 3 data to only the top 10 hits. With those simple examples out of the way, a levelspec is defined as a whitespace- or comma-separated list of one or more of the following: l Specifies the maximum level to be output for this sub-section, with a range from 0 to 10. if l is 0, no levels will be output, effec- tively disabling the sub-section (level 0 data is already provided in the Summary report, so level 1 is considered the first useful level in the Detailed report). Higher values will produce output up to and including the specified level. l.n Same as above, with the addition that n limits this section's level 1 output to the top n items. The value for n can be any integer greater than 1. (This form of limiting has less utility than the syntax shown below. It is provided for backwards compatibility; users are encouraged to use the syntax below). l:n:t This triplet specifies level l, top n, and minimum threshold t. Each of the values are integers, with l being the level limiter as described above, n being a top n limiter for the level l, and t be- ing the threshold limiter for level l. When both n and t are speci- fied, n has priority, allowing top n lists (regardless of threshold value). If the value of l is omitted, the specified values for n and/or t are used for all levels available in the sub-section. This permits a simple form of wildcarding (eg. place minimum threshold limits on all levels). However, specific limiters always override wildcard limiters. The first form of level limiter may be included in levelspec to restrict output, regardless of how many triplets are present. All three forms of limiters are effective only when amavis-logwatch's de- tail level is 5 or greater (the Detailed section is not activated until de- tail is at least 5). See the EXAMPLES section for usage scenarios. CONFIGURATION FILE Amavis-logwatch can read configuration settings from a configuration file. Essentially, any command line option can be placed into a configuration file, and these settings are read upon startup. Because amavis-logwatch can run either standalone or within Logwatch, to minimize confusion, amavis-logwatch inherits Logwatch's configuration file syntax requirements and conventions. These are: * White space lines are ignored. * Lines beginning with # are ignored * Settings are of the form: option = value * Spaces or tabs on either side of the = character are ignored. * Any value protected in double quotes will be case-preserved. * All other content is reduced to lowercase (non-preserving, case insen- sitive). * All amavis-logwatch configuration settings must be prefixed with "$amavis_" or amavis-logwatch will ignore them. * When running under Logwatch, any values not prefixed with "$amavis_" are consumed by Logwatch; it only passes to amavis-logwatch (via envi- ronment variable) settings it considers valid. * The values True and Yes are converted to 1, and False and No are con- verted to 0. * Order of settings is not preserved within a configuration file (since settings are passed by Logwatch via environment variables, which have no defined order). To include a command line option in a configuration file, prefix the com- mand line option name with the word "$amavis_". The following configura- tion file setting and command line option are equivalent: $amavis_Line_Style = Truncate --line_style Truncate Level limiters are also prefixed with $amavis_, but on the command line are specified with the --limit option: $amavis_SpamBlocked = 2 --limit SpamBlocked=2 The order of command line options and configuration file processing occurs as follows: 1) The default configuration file is read if it exists and no --config_file was specified on a command line. 2) Configuration files are read and processed in the order found on the command line. 3) Command line options override any options already set either via command line or from any configuration file. Command line options are interpreted when they are seen on the command line, and later options will override previously set options. EXIT STATUS The amavis-logwatch utility exits with a status code of 0, unless an error occurred, in which case a non-zero exit status is returned. EXAMPLES Running Standalone Note: amavis-logwatch reads its log data from one or more named Amavis log files, or from STDIN. For brevity, where required, the examples below use the word file as the command line argument meaning /path/to/amavis.log. Obviously you will need to substitute file with the appropriate path. To run amavis-logwatch in standalone mode, simply run: amavis-logwatch file A complete list of options and basic usage is available via: amavis-logwatch --help To print a summary only report of Amavis log data: amavis-logwatch --detail 1 file To produce a summary report and a one-level detail report for May 25th: grep 'May 25' file | amavis-logwatch --detail 5 To produce only a top 10 list of Sent email domains, the summary report and detailed reports are first disabled. Since commands line options are read and enabled left-to-right, the Sent section is re-enabled to level 1 with a level 1 top 10 limiter: amavis-logwatch --nosummary --nodetail \ --limit spamblocked '1 1:10:' file The following command and its sample output shows a more complex level lim- iter example. The command gives the top 4 spam blocked recipients (level 1), and under with each recipient the top 2 sending IPs (level 2) and fi- nally below that, only envelope from addresses (level 3) with hit counts greater than 6. Ellipses indicate top N or threshold-limited data: amavis-logwatch --nosummary --nodetail \ --limit spamblocked '1:4: 2:2: 3::6' file 19346 Spam blocked ----------------------------------- 756 joe@example.com 12 10.0.0.1 12 <> 12 10.99.99.99 12 <> ... 640 fred@example.com 8 10.0.0.1 8 <> 8 192.168.3.19 8 <> ... 595 peter@sample.net 8 10.0.0.1 8 <> 7 192.168.3.3 7 <> ... 547 paul@example.us 8 192.168.3.19 8 <> 7 10.0.0.1 7 <> ... ... Running within Logwatch Note: Logwatch versions prior to 7.3.6, unless configured otherwise, re- quired the --print option to print to STDOUT instead of sending reports via email. Since version 7.3.6, STDOUT is the default output destination, and the --print option has been replaced by --output stdout. Check your config- uration to determine where report output will be directed, and add the ap- propriate option to the commands below. To print a summary report for today's Amavis log data: logwatch --service amavis --range today --detail 1 To print a report for today's Amavis log data, with one level of detail in the Detailed section: logwatch --service amavis --range today --detail 5 To print a report for yesterday, with two levels of detail in the Detailed section: logwatch --service amavis --range yesterday --detail 6 To print a report from Dec 12th through Dec 14th, with four levels of de- tail in the Detailed section: logwatch --service amavis --range \ 'between 12/12 and 12/14' --detail 8 To print a report for today, with all levels of detail: logwatch --service amavis --range today --detail 10 Same as above, but leaves long lines uncropped: logwatch --service amavis --range today --detail 11 Amavis Log Level Amavis provides additional log information when the variable $log_level is increased above the default 0 value. This information is used by the amavis-logwatch utility to provide additional reports, not available with the default $log_level=0 value. A $log_level of 2 is suggested. If you prefer not to increase the noise level in your main mail or Amavis logs, you can configure syslog to log Amavis' output to multiple log files, where basic log entries are routed to your main mail log(s) and more de- tailed entries routed to an Amavis-specific log file used to feed the amavis-logwatch utility. A convenient way to accomplish this is to change the Amavis configuration variables in amavisd.conf as shown below: amavisd.conf: $log_level = 2; $syslog_facility = 'local5'; $syslog_priority = 'debug'; This increases $log_level to 2, and sends Amavis' log entries to an alter- nate syslog facility (eg. local5, user), which can then be routed to one or more log files, including your main mail log file: syslog.conf: #mail.info -/var/log/maillog mail.info;local5.notice -/var/log/maillog local5.info -/var/log/amavisd-info.log Amavis' typical $log_level 0 messages will be directed to both your maillog and to the amavisd-info.log file, but higher $log_level messages will only be routed to the amavisd-info.log file. For additional information on Amavis' logging, search the file RELEASE_NOTES in the Amavis distribution for: "syslog priorities are now dynamically derived" ENVIRONMENT The amavis-logwatch program uses the following (automatically set) environ- ment variables when running under Logwatch: LOGWATCH_DETAIL_LEVEL This is the detail level specified with the Logwatch command line argument --detail or the Detail setting in the ...conf/ser- vices/amavis.conf configuration file. LOGWATCH_DEBUG This is the debug level specified with the Logwatch command line ar- gument --debug. amavis_xxx The Logwatch program passes all settings amavis_xxx in the configu- ration file ...conf/services/amavis.conf to the amavis filter (which is actually named .../scripts/services/amavis) via environment vari- able. FILES Standalone mode /usr/local/bin/amavis-logwatch The amavis-logwatch program /usr/local/etc/amavis-logwatch/amavis-logwatch.conf The amavis-logwatch configuration file in standalone mode Logwatch mode /etc/logwatch/scripts/services/amavis The Logwatch amavis filter /etc/logwatch/conf/services/amavis.conf The Logwatch amavis filter configuration file SEE ALSO logwatch(8), system log analyzer and reporter README FILES README, an overview of amavis-logwatch Changes, the version change list history Bugs, a list of the current bugs or other inadequacies Makefile, the rudimentary installer LICENSE, the usage and redistribution licensing terms LICENSE Covered under the included MIT/X-Consortium License: http://www.opensource.org/licenses/mit-license.php AUTHOR(S) Mike Cappella The original amavis Logwatch filter was written by Jim O'Halloran, and has had many contributors over the years. They are entirely not responsible for any errors, problems or failures since the current author's hands have touched the source code. AMAVIS-LOGWATCH(1)
NAME | SYNOPSIS | DESCRIPTION | LEVEL CONTROL | CONFIGURATION FILE | EXIT STATUS | EXAMPLES | ENVIRONMENT | FILES | SEE ALSO | README FILES | LICENSE | AUTHOR(S)
Want to link to this manual page? Use this URL:
<https://man.freebsd.org/cgi/man.cgi?query=amavis-logwatch&sektion=1&manpath=FreeBSD+Ports+15.1.quarterly>