| AMAVIS-LOGWATCH(1) | General Commands Manual | AMAVIS-LOGWATCH(1) |
amavis-logwatch - An Amavisd-new log parser and analysis utility
amavis-logwatch [options] [logfile ...]
The amavis-logwatch(1) utility is an Amavisd-new log parser that produces summaries, details, and statistics regarding the operation of Amavisd-new (henceforth, simply called Amavis).
This utility can be used as a standalone program, or as a Logwatch filter module to produce Amavisd-new summary and detailed reports from within Logwatch.
Amavis-logwatch is able to produce a wide range of reports with data grouped and sorted as much as possible to reduce noise and highlight patterns. Brief summary reports provide a quick overview of general Amavis operations and message delivery, calling out warnings that may require attention. Detailed reports provide easy to scan, hierarchically-arranged and organized information, with as much or little detail as desired.
Much of the interesting data is available when Amavis' $log_level is set to at least 2. See Amavis Log Level below.
Amavis-logwatch outputs two principal sections: a Summary section and a Detailed section. For readability and quick scanning, all event or hit counts appear in the left column, followed by brief description of the event type, and finally additional statistics or count representations may appear in the rightmost column.
The following segment from a sample Summary report illustrates:
****** Summary ********************************************
9 Miscellaneous warnings
20313 Total messages scanned ---------------- 100.00% 1008.534M Total bytes scanned 1,057,524,252 ======== ================================================
1190 Blocked ------------------------------- 5.86%
18 Malware blocked 0.09%
4 Banned name blocked 0.02%
416 Spam blocked 2.05%
752 Spam discarded (no quarantine) 3.70%
19123 Passed -------------------------------- 94.14%
47 Bad header passed 0.23%
19076 Clean passed 93.91% ======== ================================================
18 Malware ------------------------------- 0.09%
18 Malware blocked 0.09%
4 Banned -------------------------------- 0.02%
4 Banned file blocked 0.02%
1168 Spam ---------------------------------- 5.75%
416 Spam blocked 2.05%
752 Spam discarded (no quarantine) 3.70%
19123 Ham ----------------------------------- 94.14%
47 Bad header passed 0.23%
19076 Clean passed 93.91% ======== ================================================
1982 SpamAssassin bypassed
32 Released from quarantine
2 DSN notification (debug supplemental)
2 Bounce unverifiable
2369 Whitelisted
2 Blacklisted
12 MIME error
58 Bad header (debug supplemental)
40 Extra code modules loaded at runtime
The next (optional) summary grouping shows message disposition by contents category. There were 18 malware messages and 4 banned file messages (all blocked), 1168 Spam messages, of which 416 were blocked (quarantined) and 752 discarded. Finally, there were 19123 messages considered to be Ham (i.e. not spam), 47 of which contained bad headers.
Additional count summaries for a variety of events are also listed.
There are dozens of sub-sections available in the Detailed report, each of whose output can be controlled in various ways. Each sub-section attempts to group and present the most meaningful data at superior levels, while pushing less useful or noisy data towards inferior levels. The goal is to provide as much benefit as possible from smart grouping of data, to allow faster report scanning, pattern identification, and problem solving. Data is always sorted in descending order by count, and then numerically by IP address or alphabetically as appropriate.
The following Spam blocked segment from a sample Detailed report illustrates the basic hierarchical level structure of amavis-logwatch:
****** Detailed *******************************************
19346 Spam blocked -----------------------------------
756 from@example.com
12 10.0.0.2
12 <>
12 192.168.2.2
12 <>
5 192.168.2.1
...
The amavis-logwatch utility reads from STDIN or from the named Amavis logfile. Multiple logfile arguments may be specified, each processed in order. The user running amavis-logwatch must have read permission on each named log file.
The options listed below affect the operation of amavis-logwatch. Options specified later on the command line override earlier ones. Any option may be abbreviated to an unambiguous length.
The amavis-logwatch utility produces a Summary section, a Detailed section, and additional report sections. With level less than 5, amavis-logwatch will produce only the Summary section. At level 5 and above, the Detailed section, and any additional report sections are candidates for output. Each incremental increase in level generates one additional hierarchical sub-level of output in the Detailed section of the report. At level 10, all levels are output. Lines that exceed the maximum report width (specified with max_report_width) will be cut. Setting level to 11 will prevent lines in the report from being cut (see also --line_style).
Note: if you use parenthesis in your regular expression, be sure they are cloistering and not capturing: use (?:pattern) instead of (pattern).
The output of every section in the Detailed report is controlled by a level limiter. The name of the level limiter variable will be output when the sect_vars option is set. Level limiters are set either via command line in standalone mode with --limit limiter=levelspec option, or via configuration file variable $amavis_limiter=levelspec. Each limiter requires a levelspec argument, which is described below in LEVEL CONTROL.
The list of level limiters is shown below.
Amavis major contents category (ccatmajor) sections, listed in order of priority: VIRUS, BANNED, UNCHECKED, SPAM, SPAMMY, BADH, OVERSIZED, MTA, CLEAN.
Other sections, arranged alphabetically:
The Detailed section of the report consists of a number of sub-sections, each of which is controlled both globally and independently. Two settings influence the output provided in the Detailed report: a global detail level (specified with --detail) which has final (big hammer) output-limiting control over the Detailed section, and sub-section specific detail settings (small hammer), which allow further limiting of the output for a sub-section. Each sub-section may be limited to a specific depth level, and each sub-level may be limited with top N or threshold limits. The levelspec argument to each of the level limiters listed above is used to accomplish this.
It is probably best to continue explanation of sub-level limiting with the following well-known outline-style hierarchy, and some basic examples:
level 0
level 1
level 2
level 3
level 4
level 4
level 2
level 3
level 4
level 4
level 4
level 3
level 4
level 3
level 1
level 2
level 3
level 4
The simplest form of output limiting suppresses all output below a specified level. For example, a levelspec set to "2" shows only data in levels 0 through 2. Think of this as collapsing each sub-level 2 item, thus hiding all inferior levels (3, 4, ...), to yield:
level 0
level 1
level 2
level 2
level 1
level 2
Sometimes the volume of output in a section is too great, and it is useful to suppress any data that does not exceed a certain threshold value. Consider a dictionary spam attack, which produces very lengthy lists of hit-once recipient email or IP addresses. Each sub-level in the hierarchy can be threshold-limited by setting the levelspec appropriately. Setting levelspec to the value "2::5" will suppress any data at level 2 that does not exceed a hit count of 5.
Perhaps producing a top N list, such as top 10 senders, is desired. A levelspec of "3:10:" limits level 3 data to only the top 10 hits.
With those simple examples out of the way, a levelspec is defined as a whitespace- or comma-separated list of one or more of the following:
All three forms of limiters are effective only when amavis-logwatch's detail level is 5 or greater (the Detailed section is not activated until detail is at least 5).
See the EXAMPLES section for usage scenarios.
Amavis-logwatch can read configuration settings from a configuration file. Essentially, any command line option can be placed into a configuration file, and these settings are read upon startup.
Because amavis-logwatch can run either standalone or within Logwatch, to minimize confusion, amavis-logwatch inherits Logwatch's configuration file syntax requirements and conventions. These are:
option = value
To include a command line option in a configuration file, prefix the command line option name with the word "$amavis_". The following configuration file setting and command line option are equivalent:
Level limiters are also prefixed with $amavis_, but on the command line are specified with the --limit option:
$amavis_Line_Style = Truncate
--line_style Truncate
$amavis_SpamBlocked = 2
--limit SpamBlocked=2
The order of command line options and configuration file processing occurs as follows: 1) The default configuration file is read if it exists and no --config_file was specified on a command line. 2) Configuration files are read and processed in the order found on the command line. 3) Command line options override any options already set either via command line or from any configuration file.
Command line options are interpreted when they are seen on the command line, and later options will override previously set options.
The amavis-logwatch utility exits with a status code of 0, unless an error occurred, in which case a non-zero exit status is returned.
Note: amavis-logwatch reads its log data from one or more named Amavis log files, or from STDIN. For brevity, where required, the examples below use the word file as the command line argument meaning /path/to/amavis.log. Obviously you will need to substitute file with the appropriate path.
To run amavis-logwatch in standalone mode, simply run:
amavis-logwatch file
A complete list of options and basic usage is available via:
amavis-logwatch --help
To print a summary only report of Amavis log data:
amavis-logwatch --detail 1 file
To produce a summary report and a one-level detail report for May 25th:
grep 'May 25' file | amavis-logwatch --detail 5
To produce only a top 10 list of Sent email domains, the summary report and detailed reports are first disabled. Since commands line options are read and enabled left-to-right, the Sent section is re-enabled to level 1 with a level 1 top 10 limiter:
amavis-logwatch --nosummary --nodetail \
--limit spamblocked '1 1:10:' file
The following command and its sample output shows a more complex level limiter example. The command gives the top 4 spam blocked recipients (level 1), and under with each recipient the top 2 sending IPs (level 2) and finally below that, only envelope from addresses (level 3) with hit counts greater than 6. Ellipses indicate top N or threshold-limited data:
amavis-logwatch --nosummary --nodetail \
--limit spamblocked '1:4: 2:2: 3::6' file
19346 Spam blocked -----------------------------------
756 joe@example.com
12 10.0.0.1
12 <>
12 10.99.99.99
12 <> ...
640 fred@example.com
8 10.0.0.1
8 <>
8 192.168.3.19
8 <> ...
595 peter@sample.net
8 10.0.0.1
8 <>
7 192.168.3.3
7 <> ...
547 paul@example.us
8 192.168.3.19
8 <>
7 10.0.0.1
7 <>
... ...
Note: Logwatch versions prior to 7.3.6, unless configured otherwise, required the --print option to print to STDOUT instead of sending reports via email. Since version 7.3.6, STDOUT is the default output destination, and the --print option has been replaced by --output stdout. Check your configuration to determine where report output will be directed, and add the appropriate option to the commands below.
To print a summary report for today's Amavis log data:
logwatch --service amavis --range today --detail 1
To print a report for today's Amavis log data, with one level of detail in the Detailed section:
logwatch --service amavis --range today --detail 5
To print a report for yesterday, with two levels of detail in the Detailed section:
logwatch --service amavis --range yesterday --detail 6
To print a report from Dec 12th through Dec 14th, with four levels of detail in the Detailed section:
logwatch --service amavis --range \
'between 12/12 and 12/14' --detail 8
To print a report for today, with all levels of detail:
logwatch --service amavis --range today --detail 10
Same as above, but leaves long lines uncropped:
logwatch --service amavis --range today --detail 11
Amavis provides additional log information when the variable $log_level is increased above the default 0 value. This information is used by the amavis-logwatch utility to provide additional reports, not available with the default $log_level=0 value. A $log_level of 2 is suggested.
If you prefer not to increase the noise level in your main mail or Amavis logs, you can configure syslog to log Amavis' output to multiple log files, where basic log entries are routed to your main mail log(s) and more detailed entries routed to an Amavis-specific log file used to feed the amavis-logwatch utility.
A convenient way to accomplish this is to change the Amavis configuration variables in amavisd.conf as shown below:
amavisd.conf:
$log_level = 2;
$syslog_facility = 'local5';
$syslog_priority = 'debug';
This increases $log_level to 2, and sends Amavis' log entries to an alternate syslog facility (eg. local5, user), which can then be routed to one or more log files, including your main mail log file:
syslog.conf:
#mail.info -/var/log/maillog
mail.info;local5.notice -/var/log/maillog
local5.info -/var/log/amavisd-info.log
Amavis' typical $log_level 0 messages will be directed to both your maillog and to the amavisd-info.log file, but higher $log_level messages will only be routed to the amavisd-info.log file. For additional information on Amavis' logging, search the file RELEASE_NOTES in the Amavis distribution for:
"syslog priorities are now dynamically derived"
The amavis-logwatch program uses the following (automatically set) environment variables when running under Logwatch:
logwatch(8), system log analyzer and reporter
README, an overview of amavis-logwatch Changes, the version change list history Bugs, a list of the current bugs or other inadequacies Makefile, the rudimentary installer LICENSE, the usage and redistribution licensing terms
Covered under the included MIT/X-Consortium License: http://www.opensource.org/licenses/mit-license.php
Mike Cappella
The original amavis Logwatch filter was written by Jim O'Halloran, and has had many contributors over the years. They are entirely not responsible for any errors, problems or failures since the current author's hands have touched the source code.