calamaris - generate text and graphical statistics out of
log files from Proxy-Cache-Servers
cat log | calamaris [ --config-file .../calamaris.conf ] [
switches ]
Calamaris is used to produce statistical output from
Squid, NetCache, Inktomi Traffic Server, Oops! proxy
server, Compaq Tasksmart, Cisco Content Engines or related
Proxy log files. The resulting output can be ascii or html with or without
graphic and with or without frames. It is possible to cache calculated data
in a file to use them in later runs.
This manual page describes the options of Calamaris and
gives a few examples.
- --config-file
file
-
Not all reports and modification can be made through
command-line-switches. To use all options of Calamaris you´ll
have to use the configuration file. You'll find the
configuration-directives below, always inside of braces. Examples are in
the calamaris.conf which should come with this package.
- --all-useful-reports|-a
- extracts all useful reports available, --all-useful-reports equals
--size-distribution-report 10 --domain-report 20
--performance-report 60 --requester-report 20
--status-report --type-report 20
--response-time-report --errorcode-distribution-report
- --domain-report|-d
n ($domain_report)
- switches the top level and the second level report on. The data is derived
from the URL. The output is limited by n. (-1 is unlimited)
- --domain-report-limit
n ($domain_report_limit)
- limit the domain-report to lines which have n or more requests.
- --domain-report-n-level|-N
n ($domain_report_n_level)
- All URL-Host reports will be switched from 2nd-level to n-level-reports.
(-1 shows a full urlhost-report)
Note: This option is only useful with activated
domain-report.
- --errorcode-distribution-report
($errorcode_distribution_report)
- shows the Response code distribution over all objects
- ($object_freshness_report)
- shows the freshness of objects in your cache. Calamaris looks for
freshness tags like 'TCP_HIT', 'TCP_REFRESH_MISS', ... and make statistics
on it. With this information you can optimize the caching behaviour of
your cache depending on the objects content type. E.g. squid admins could
use this information to configure the refresh_pattern. This option needs
more configuration in the configuration-file.
- --peak-report|-p
type ($peak_report)
- Measures the peaks of the Proxy usage in requests per second, per minute
and per hour. It reports peaks for TCP, UDP and ALL requests. If set to
'old' these values were calculated with the old slow method, if set
to 'new' the new faster (but still slow) method is used.
- --performance-report|-P
n ($performance_report)
- Shows the throughput of TCP requests for every n minutes.
- --performance-report-adjust|-T
n ($performance_report_adjust)
- Time: Adjust the Performancereport in minutes for non GMT-Timezoners.
- --requester-report|-r
n ($requester_report)
- Switches the UDP and TCP requester reports on. The output is limited by n.
(-1 is unlimited)
- --requester-report-no-dns-lookup|-n
($requester_report_no_dns_lookup)
- Switches the IP lookup for the requesters off.
- --requester-report-use-user-info|-u
($requester_report_use_user_info)
- Switches the usage of eventually available ident information for requester
reports on.
Warning: This breaks the privacy of your users! (see
PRIVACY-Section below)
- --requester-report-with-targets|-R
n ($requester_report_with_targets)
- adds to each line of the requester report the requested URLs. The output
is limited by n. (-1 is unlimited, and can result in very very long
reports.)
Warning: Using this option breaks the privacy of your
users! (see PRIVACY-Section below)
- --response-time-report
($response_time_report)
- sums up the time distribution over all objects
- (@response_time_report_interval)
- This array defines the time steps, which should be reported in the
response-time-report.
- --size-distribution-report|-D
n ($size_distribution_report)
- shows size-based distribution of requested objects, smaller numbers result
in more verbose reports. (choose 2, 10 or 100 for useful output.)
- --status-report|-s
($status_report)
- alters the default behaviour of Calamaris and makes the status
reports more verbose.
- --type-report|-t
n ($type_report)
- switches the content type and the file extension report on. The output is
limited by n. (-1 is unlimited)
- --type-report-ignore-case|-c
($type_report_ignore_case)
- Switch to case-insensitive. This is useful for the 'Requested extensions'
report.
- --input-format|-f
type ($input_format)
- sets the type of input logfiles. If set to
'auto' Calamaris tries to guess the input file
format. This is the Default.
Note: If the first line of your input file is
corrupted, Calamaris will stop with an error.
'squid' Calamaris expects native logfile derived
from Squid V1.1.beta26-V2.x or OOPS.
'squid-extended' Calamaris expects native
logfile derived from Squid V1.1.alpha1-V2.x with
log_mime_hdrs enabled or Squid with
Smartfilter-Patch or squid-style logfiles out of Cisco Content
Engines. (This only enables parsing of these kind of logfile, the
additional data will be ignored.)
(Logging of MIME-Headers breaks the privacy of your users!
(see PRIVACY-Section below)
'squid-old' Calamaris expects native logfile
derived from Squid V1.1.alpha1-V1.1.beta25.
'nc' Calamaris expects Logfiles from
NetCache up to V4.x. (Please see the README on this.)
'its' Calamaris expects Logfiles from Inktomi
Traffic Server.
'elff' Calamaris expects Logfiles in Extended
Logfile Format (i.e. from Compaq Tasksmart, Novell Internet
Caching System or NetCache V5.x)
'nse' Calamaris expects Logfiles in Netscape
Extended-1 or Netscape Extended-2 Logfile Format (from
Netscape/iPlanet/SunOne Proxy-Server )
- --ipfilter-exclude
IP/range ($ipfilter_exclude)
- all IPs are analyzed, except IP/range. Format: 1.1.1.1/32:1.1.2.0/24 or
1.1.1.1/255.255.255.255:1.1.2.0/255.255.255.0
IP list separated by ':' This switch needs the perl Module
NetAddr::IP.
Warning: This breaks the privacy of your users! (see
PRIVACY-Section below)
- --ipfilter-include
IP/range ($ipfilter_include)
- no IPs are analyzed, except IP/range. Format: see --ipfilter-exclude
Warning: This breaks the privacy of your users! (see
PRIVACY-Section below)
- --no-input|-z
($no_input)
- Switches reading from standard input off. You can use this to merge many
cache files to one (see --cache-input-file and
--cache-output-file) or to generate a report out of cache files.
- --time-interval|-I
t-t ($time_interval)
- defines which time-interval should be parsed. t has to be the format
yyyymmddhhmmss (localtime)
Note: omitting the beginning or ending date is
allowed.
Standard output format is plain ascii with 80 chars width.
- ($column1_color)
- ($column2_color) defines the colors for the columns in graphics.
(only useful with --output-format graph)
- ($formats[n])
- Through the config-file you are able to modify the width of the report and
alter the culomns that are displayed in the reports. n is the number of
the report, as displayed by --help in the --show-reports-option.
- --hostname|-H
name ($hostname)
- The name for the title or subject of the output. If set to 'lookup'
Calamaris looks up the host name of the system its been run on.
- --image-type
($image_type)
- Sets the image type to gif, png, jpeg, gd or gd2. Only useful when
--output-format graph is set. The available images types are
depending on your GD::Graph installation. Default is 'png'.
- --logo|-l string
($logo)
- add a custom string to a HTML-Report. It'll be added to a table on the top
of the output. -l '<A HREF="https://cord.de/"><IMG
BORDER=0
SRC="https://cord.de/sites/default/files/cordlog2n.gif"
ALT="Cord"></A>' will add my logo with a link to the
Report.
Note: --logo works only in combination with
--output-format html or html-frame
- --meta|-M string
($meta)
- Meta: adds a custom string or the content of a file into the <HEAD>
of a HTML-Report. Useful if you want to add Stylesheets or something to
the Report.
Note: --meta works only in combination with
--output-format html or html-frame
- --output-format|-F
type[,type[,type[,...]]] ($output_format)
- Format: sets the format of the output-report. If set to
'mail' adds a subject header to the beginning of the
report.
'html' all output is given in html with tables. Can be
combined with 'mail' to send html mails.
'html-frame' all output is given in html frames with
tables.
'html-embed' all output is given in html with tables
without HTML-Headers. Useful for Server-Side-Includes.
'graph' enables graphics for html, html-embed or
html-frame.
'unformatted' gives out the raw numbers separated by
spaces. Useful for re-using the output in other scripts. If you use this
along with -U, the byte values are calculated in the given Unit,
and displayed without indication along with the numbers. the indication
moves up to the header of the report.
- --output-path
($output_path)
- output calamaris statistics to /path. In case of graph output, the
graphics destination is /path and the filename is index.html, else it is
calamaris.txt. If --output-path is not given, all graphics are
written to the working directory.
- --output-file
($output_file)
- alter the filename of --output-path.
- --output-file-prefix
($output_file_prefix)
- adds a prefix to --output-file %t is replaced by the
timerange of the report, %h by the hostname (see --hostname
)
- --show-reports|-S
n[,n[,n[,...]]] ($show_reports)
- Show: Shows only the defined reports in the specified order. Default is to
display the reports as they are defined through the report-switches above.
The following numbers are defined:
0 Summary
1 Incoming request peak per protocol
2 Incoming transfer volume peak per protocol
3 Incoming requests by method
4 Incoming UDP-requests by status
5 Incoming TCP-requests by status
6 Outgoing requests by status
7 Outgoing requests by destination
8 Request-destinations by 2ndlevel-domain
9 Request-destinations by toplevel-domain
10 TCP-Request-protocol
11 Requested content-type
12 Requested extensions
13 Incoming UDP-requests by host
14 Incoming TCP-requests by host
15 Size Distribution Diagram
16 Performance in n minute steps
17 UDP-Request duration distribution in msec
18 TCP-Request duration distribution in msec
19 UDP Response code distribution
20 TCP Response code distribution
Note: Using this doesn't make Calamaris any faster,
the internal calculations will be done as the report-switches were set (see
above).
- --sort-order|-O
($sort_order)
- Changes the sort order in the reports to request size, default is sorting
by number of requests.
- ($text_color)
- defines the colors for text/axis/legend/labels in graphics. (only useful
with --output-format graph )
- --unit|-U string
($unit)
- You can define this to K(ilo), M(ega), G(iga) or T(era) for the
Byte-units.
- ($width)
- defines the width of the graphics. height is calculated from this with a
3:2-ratio. (only useful with --output-format graph )
- ($x_scale)
- defines how many datasets should be drawn on the graph. 30 is a good
value, but you can play with this. if $x_scale gets to big, you're on your
own ;-)
- --generate-index
($generate_index)
- generates an index for all reports that match --output-file-prefix.
- --cache-input-file|-i
file ($cache_input_file)
- You can reuse a cache file generated with --cache-output-file file
to add old data to a new report. Several files can be separated with a
':'.
Note: if you use more than one cache file, make sure
they are chronologicaly ordered (oldest first).
Note: if you reuse cache-files, which were not created
with -d -1 -r -1 -t -1 -R -1 the number of 'others' would be
wrong everywhere. In this case the number of 'others' are omitted.
- --cache-output-file|-o
file ($cache_output_file)
- Calamaris stores a summary of the computed information in
file and you can reuse it at a later time with
--cache-input-file
Note: The output file can be the same as the input
file: it is simply overwritten after reading the data. It is not
recommended to change the options between different runs if you include
older data as this can result in strange measurements.
- --dump-loop|-L
- prints the internal loop to STDERR. (for Debugging)
- ($test)
- activates some small tests for the programmer.
- --verbose|-v
($verbose)
- print more information about what is Calamaris is doing and believing.
If you have a problem with Calamaris , please make sure
that you use the recent version of Calamaris (see VERSION below).
Also check that your proxy works correctly and doesn't produce invalid
Logfiles. (see the README for buglist and pointers.)
If you're sure that you've encountered a bug in Calamaris
please report it to Calamaris-bug@cord.de. This also applies if
Calamaris itself says 'please report this'.
Calamaris can be (mis-)used to track what users are
requesting.
So please read the following and think about it, before using
Calamaris to be the Big Brother.
- -
- If you don't trust your users than there is something more wrong than the
loss of productivity.
- -
- Squid has some nice acl-mechanisms. If you think that your users don't use
the net properly, don't let them use it. (You can also open the net at
specific times or to specific sites, if you want.)
- -
- If you still want to use Calamaris that way, let your vict^Wusers
know, that they'll be monitored. (in Germany you have to let them know!)
Cord Beermann <Cord@Wunder-Nett.org>, Michael Pophal
<michael.pophal@nefkom.net>. There are also a lot of people who
contributed code, gave ideas or requested features. Look them up in the
executable.
This man page was written by Philipp Frauenfelder
<pfrauenf@debian.org>, maintainer of the Debian package. Maintenance
is now taken over by Cord Beermann.
Version of this manpage: $Id: calamaris.1,v 3.1 2006-03-19
17:52:48 cord Exp $
It describes the usage of Calamaris V3.0 and later.
Information about new releases, mailing lists, and other related
issues can be found from the Calamaris home page at URL
https://Calamaris.Cord.de/
Calamaris comes with "absolutely no
warranty".
Copyright (C) 1997-2006, 2013, 2015, 2017, 2021 Cord Beermann
This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or (at your
option) any later version.
(If you modify and want to publish it under the name
Calamaris , please ask me. I don't want to confuse the 'audience'
with many different versions of the same name and/or Version number. (This
is not part of the license, it is only a favour i asked of you.))
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software Foundation,
Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.