FreeBSD Manual Pages

home | help

SQUIDANALYZER(1) User Contributed Perl Documentation SQUIDANALYZER(1)

NAME
SquidAnalyzer - Squid access log report generation tool

DESCRIPTION
SquidAnalyzer parse native access log format of the Squid proxy and gener-
ate general statistics about hits, bytes, users, networks, top url, top
second level domain and denied URLs. Common and combined log format are
also supported. SquidGuard logs can also be parsed and ACL's redirection
reported into denied URLs report.

Statistic reports are oriented to user and bandwidth control, this is not a
pure cache statistics generator. SquidAnalyzer use flat files to store data
and don't need any SQL, SQL Lite or Berkeley databases.

This analyzer is incremental so it should be run in a daily cron. Take
care if you have rotate log enable to run it before rotation is done.

REQUIREMENT
Nothing is required than a modern perl version 5.8 or higher. Graphics are
based on the Flotr2 Javascript library so they are drawn at your browser
side without extra installation required.

INSTALLATION
Generic install
If you want the package to be intalled into the Perl distribution just do
the following:

perl Makefile.PL
make
make install

Follow the instruction given at the end of install. With this default in-
stall everything configurable will be installed under /etc/squidanalyzer.
The Perl library SquidAnalyzer.pm will be installed under your site_perl
directory and the squid-analyzer Perl script will be copied under /usr/lo-
cal/bin.

The default output directory for html reports will be /var/www/squidana-
lyzer/.

On FreeBSD, if make install is freezing and you have the following mes-
sages:

FreeBSD: Registering installation in the package database
FreeBSD: Cannot determine short module description
FreeBSD: Cannot determine module description

please proceed as follow:

perl Makefile.PL INSTALLDIRS=site
make
make install

as the issue is related to an install into the default Perl vendor in-
stalldirs it will then use Perl site installdirs.

Custom install
You can create your fully customized SquidAnalyzer installation by using
the Makefile.PL Perl script. Here is a sample:

perl Makefile.PL \
LOGFILE=/var/log/squid3/access.log \
BINDIR=/usr/bin \
CONFDIR=/etc \
HTMLDIR=/var/www/squidreport \
BASEURL=/squidreport \
MANDIR=/usr/man/man3 \
DOCDIR=/usr/share/doc/squidanalyzer

If you want to build a distro package, there are two other options that you
may use. The QUIET option is to tell to Makefile.PL to not show the default
post install README. The DESTDIR is to create and install all files in a
package build base directory. For example for Fedora RPM, thing may look
like that:

# Make Perl and SendmailAnalyzer distrib files
%{__perl} Makefile.PL \
INSTALLDIRS=vendor \
QUIET=1 \
LOGFILE=/var/log/squid/access.log \
BINDIR=%{_bindir} \
CONFDIR=%{_sysconfdir} \
BASEDIR=%{_localstatedir}/lib/%{uname} \
HTMLDIR=%{webdir} \
MANDIR=%{_mandir}/man3 \
DOCDIR=%{_docdir}/%{uname}-%{version} \
DESTDIR=%{buildroot} < /dev/null

See spec file in packaging/RPM for full RPM build script.

Local install
You can also have a custom installation. Just copy the SquidAnalyzer.pm and
the squid-analyzer perl script into a directory, copy and modify the con-
figuration file and run the script from here with the -c option.

Then copy files sorttable.js, squidanalyzer.css and logo-squidanalyzer.png
into the output directory.

Post installation
1. Modify your httpd.conf to allow access to HTML output like follow:

Alias /squidreport /var/www/squidanalyzer
<Directory /var/www/squidanalyzer>
Options -Indexes FollowSymLinks MultiViews
AllowOverride None
Order deny,allow
Deny from all
Allow from 127.0.0.1
</Directory>

2. If necessary, give additional host access to SquidAnalyzer in
httpd.conf. Restart and ensure that httpd is running.

3. Browse to http://my.host.dom/squidreport/ to ensure that things are
working properly.

4. Setup a cronjob to run squid-analyzer daily or more often:

# SquidAnalyzer log reporting daily
0 2 * * * /usr/local/bin/squid-analyzer > /dev/null 2>&1

or run it manually. For more information, see README file.

If your squid logfiles are rotated then cron isn't going to give the ex-
pected result as there exists a time between when the cron is run and the
logfiles are rotated. It would be better to call squid-analyzer from logro-
tate, eg:

/var/log/proxy/squid-access.log {
daily
compress
rotate 730
missingok
nocreate
sharedscripts
postrotate
test ! -e /var/run/squid.pid || /usr/sbin/squid -k rotate
/usr/bin/squid-analyzer -d -l /var/log/proxy/squid-access.log.1
endscript
}

You can also use network name instead of network ip addresses by using the
network-aliases file. Also if you don't have authentication enable and want
to replace client ip addresses by some know user or computer you can use
the user-aliases file to do so.

See the file squidanalyzer.conf to customized your output statistics and
match your network and file system configuration.

USAGE
SquidAnalyzer can be run manually or by cron job using the squid-analyzer
Perl script. Here are authorized usage:

Usage: squid-analyzer [ -c squidanalyzer.conf ] [logfile(s)]

-c | --configfile filename : path to the SquidAnalyzer configuration file.
By default: /etc/squidanalyzer/squidanalyzer.conf
-b | --build_date date : set the date to be rebuilt, format: yyyy-mm-dd
or yyyy-mm or yyyy. Used with -r or --rebuild.
-d | --debug : show debug informations.
-h | --help : show this message and exit.
-j | --jobs number : number of jobs to run at same time. Default is 1,
run as single process.
-p | --preserve number : used to set the statistic obsolescence in
number of month. Older stats will be removed.
-P | --pid_dir directory : set directory where pid file will be stored.
Default /tmp/
-r | --rebuild : use this option to rebuild all html and graphs
output from all data files.
-t, --timezone +/-HH : set number of hours from GMT of the timezone.
Use this to adjust date/time of SquidAnalyzer
output when it is run on a different timezone
than the squid server.
-v | version : show version and exit.
--no-year-stat : disable years statistics, reports will start
from month level only.
--no-week-stat : disable weekly statistics.

Log files to parse can be given as command line arguments or as a comma
separated list of file for the LogFile configuration directive. By default
SquidAnalyer will use file: /var/log/squid/access.log

There is special options like --rebuild that force SquidAnalyzer to rebuild
all HTML reports, useful after an new feature or a bug fix. If you want to
limit the rebuild to a single day, a single month or year, you can use the
--build_date option by specifying the date part to rebuild, format:
yyyy-mm-dd, yyyy-mm or yyyy.

The --preserve option should be used if you want to rotate your statistics
and data. The value is the number of months to keep, older reports and data
will be removed from the filesystem. Useful to preserve space, for example:

squid-analyzer -p 6 -c /etc/squidanalyzer/squidanalyzer.conf

will only preserve six month of statistics from the last run of squidana-
lyzer.

If you have a SquidGuard log you can add it to the list of file to be
parsed, either in the LogFile configuration directive log list, either at
command line:

squid-analyzer /var/log/squid3/access.log /var/log/squid/SquidGuard.log

SquidAnalyzer will automatically detect the log format and report Squid-
Guard ACL's redirection to the Denied Urls report.

MULTIPROCESS
If you have huges squid access.log you will be interested by using multi-
process with SquidAnalyzer. Using the -j or --jobs command line option you
can force SquidAnalyzer to use as many cores/cpus as wanted.

squid-analyzer -j 8 -l /var/log/squid3/huge_access.log

Here SquidAnalyzer will use 8 cpus to parse the file and compute all sta-
tistics reports. It will also use much more memory at the same time.

LOGFORMAT
SquidAnalyzer supports the following predefined log format:

logformat squid %ts.%03tu %6tr %>a %Ss/%03>Hs %<st %rm %ru %un %Sh/%<A %mt
logformat common %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st %Ss:%Sh
logformat combined %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh

The common and combined log format can have one more field to add mime-type
report like with the native squid log format:

logformat common %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st %Ss:%Sh %mt
logformat combined %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh %mt

Those are the default format used by squid, you can switch to any of the
three log format by giving the name at end of the access_log directive:

access_log /var/log/squid3/access.log squid

access_log /var/log/squid3/access.log common

CONFIGURATION
Unless previous version customization of SquidAnalyzer is now done by a
single configuration file squidanalyzer.conf.

Here follow the configuration directives used by Squid Analyzer.

Output output_directory
Where SquidAnalyzer should dump all HTML, data and images files. You
should give a path that can be read by a Web browser.

WebUrl
The URL of the SquidAnalyzer javascript, HTML and images files. De-
fault: /squidreport

CustomHeader
This directive allow you to replace the SquidAnalyze logo by your cus-
tom logo. The default value is defined as follow:

<a href="$self->{WebUrl}">
<img src="$self->{WebUrl}images/logo-squidanalyzer.png" title="SquidAnalyzer $VERSION" border="0">
</a> SquidAnalyzer

Feel free to define your own header but take care to not break current
design. For example:

CustomHeader <a href="http://my.isp.dom/"><img src="http://my.isp.dom/logo.png" title="My ISP link" border="0" width="100" height="110"></a> My ISP Company
126,1 Bas

LogFile squid_access_log_file
Set the path to the Squid log file. This can be a comma separated list
of files to process several files at the same time. If the files comes
from differents Squid servers, they will be merges in a single reports.
You can also add to the list a SquidGuard log file, SquidAnalyzer will
atomatically detect the format.

UseClientDNSName 0
If you want to use DNS name instead of client Ip address as username
enable this directive. When you don't have authentication, the username
is set to the client ip address, this allow you to use the DNS name in-
stead. Note that you must have a working DNS resolution and that it
can really slow down the generation of reports.

DNSLookupTimeout 0.0001
If you have enabled UseClientDNSName and have lot of ip addresses that
do not resolve you may want to increase the DNS lookup timeout. By de-
fault SquidAnalyzer will stop to lookup a DNS name after 0.0001 second
(100 ms).

NetworkAlias network-aliases_file
Set path to the file containing network alias name. Network are show as
Ip addresses so if you want to display name instead create a file with
this format:

LOCATION_NAME IP_NETWORK_ADDRESS

Separator must be a tabulation.

You can use regex to match and group some network addresses. See net-
work-aliases file for examples.

UserAlias user-aliases_file
Set path to the file containing user alias name. If you don't have
auth_proxy enable users are seen as ip addresses. So if you want to
show username or computer name instead, create a file with this format:

FULL_USERNAME IP_ADDRESS

If you have auth_proxy enable but want to replace login name by full
user name for example, create a file with this format:

FULL_USERNAME LOGIN_NAME

Separator for both must be a tabulation.

You can use regex to match and group some user login or ip addresses.
See user-aliases file for examples.

You can also replace default ip address by his DNS name by enabling di-
rective 'UseClientDNSName'.

AnonymizeLogin 0
Set this to 1 if you want to anonymize all user login. The username
will be replaced by an unique id that change at each squid-analyzer
run. Default disable.

OrderNetwork bytes|hits|duration
OrderUser bytes|hits|duration
OrderUrl bytes|hits|duration
Used to set how SquidAnalyzer sort Network, User and User detailed Urls
reports screen. Value can be: bytes, hits or duration. Default is
bytes. Note that OrderUrl is limited to User detailed Urls reports and
does not apply to Top Url and Top domain report where there is three
reports each already ordered.

OrderMime bytes|hits
Used to set how SquidAnalyzer sort Mime types report screen Value can
be: bytes or hits. Default is bytes.

UrlReport 0|1
Should SquidAnalyzer display user url details. This will show all URL
read by user. Take care to have enougth space disk for large user. De-
fault is 0, no url detail report.

UserReport 0|1
Should SquidAnalyzer display user details. This will show statistics
about user. Default is 1, show user detail report. Disable it to be
able to remove any user related reports, statistics about URL and do-
mains will remain.

UrlHitsOnly 0|1
Enable this directive if you don't want the tree Top URL and Domain ta-
bles. You will just have the table of Url/Domain ordered per hits then
you can still sort the URL/Domain order by clicking on each column.
This is useful when you have set a high value to TopNumber.

QuietMode 0|1
Run in quiet mode for batch processing or print debug information. De-
fault is 0, verbose mode.

CostPrice price/Mb
Used to set a cost of the bandwith per Mb. If you want to generate in-
voice per Mb for bandwith traffic this can help you. Value 0 mean no
cost, this is the default value, the "Cost" column is not displayed

Currency currency_abreviation
Used to set the currency of the bandwith cost. Preferably the html spe-
cial character. Default is &euro;

TopNumber number
Used to set the number of top url and second level domain to show. De-
fault is top 100.

TopUrlUser Use this directive to show the top N users that look at an URL
or a domain. Set it to 0 to disable this feature. Default is top 10.
Exclude exclusion_file
Used to set client ip addresses, network addresses, auth login or uri
to exclude from report.

You can define one by line exclusion by specifying first the type of
the exclusion (USER, CLIENT or URI) and a space separated list of valid
regex.

You can also use the NETWORK type to define network address with net-
mask using the CIDR notation: xxx.xxx.xxx.xxx/n

See example bellow:

NETWORK 192.168.1.0/24 10.10.0.0/16
CLIENT 192\.168\.1\.2
CLIENT 10\.169\.1\.\d+ 192\.168\.10\..*
USER myloginstr
USER guestlogin\d+ guestdemo
URI http:\/\/myinternetdomain.dom.*
URI .*\.webmail\.com\/.*\/login\.php.*

you can have multiple line of the same exclusion type.

Include inclusion_file
Used to set client ip addresses, network addresses or auth login to in-
clude into the report. All others will not be included. It works as the
opposite of the Include parameter.

You can define one by line inclusion by specifying first the type of
the inclusion (USER or CLIENT) and a space separated list of valid
regex.

You can also use the NETWORK type to define network address with net-
mask using the CIDR notation: xxx.xxx.xxx.xxx/n

See example bellow:

you can have multiple line of the same inclusion type.

ExcludedMethods
This directive allow exclusion of some unwanted methods in report sta-
tistics like HEAD, POST, CONNECT, etc. Can be a comma separated list of
methods.

ExcludedMimes
This directive allow exclusion of some unwanted mimetypes in report
statistics like text/html, text/plain, or more generally text/*, etc.
Can be a comma separated list of perl regular expression. Ex:

ExcludedMimes text/.*,image/.*

LangUsed to set the translation file to be used. Value must be set to a
file containing all string translated. See the lang directory for
translation files. Default is defined internally in English.

ExcludedCodes
This directive allow exclusion of some unwanted codes in report statis-
tics like TCP_DENIED/403 which are generated when a user accesses a
page the first time without authentication. Can be a comma separated
list of methods. Default is none, all codes will be parsed.

DateFormat
Date format used to display date (year = %y, month = %m and day = %d)
You can also use %M to replace month by its 3 letters abbreviation.
Default: %y-%m-%d

SiblingHit
Adds peer cache hit (CD_SIBLING_HIT) to be taken has local cache hit.
Enabled by default, you must disabled it if you don't want to report
peer cache hit onto your stats.

TransfertUnit
Allow to change the default unit used to display transfert size. De-
fault is BYTES, other possible values are KB, MB and GB.

MinPie
Minimum percentage of data in pie's graphs to not be placed in the oth-
ers item. Lower values will be summarized into the others item.

Locale
Set this to your locale to display generated date in your language. De-
fault is to use the current locale of the system. If you want date in
German for example, set it to de_DE.

Rapport genere le mardi 11 decembre 2012, 15:13:09 (UTC+0100).

with a Locale set to fr_FR.

MaxFormatError
When SquidAnalyzer find a corrupted line in his data file, it exit im-
medialtly. You can force him to wait for a certain amount of errors
before exiting. Of course you might want to remove the corrupted line
before the next run. This can be useful if you have special characters
in some fields like mime type.

TimeZone
Set timezone to use when SquidAnalyzer is used in a different server
than the one running squid and there is a different timezone between
these two machines. The value must follow format: +/-HH. Default is to
use local time. For example:

TimeZone +01

for a log file generated on zone Europe/Paris with UTC+0100 and parsed
on a computer with different timezone.

SUPPORT
Release annoucement
Please follow us on twitter to receive release annoucement and latest news
: https://twitter.com/SquidAnalyzer

Bugs and Feature requests
Please report any bugs, patches, discussion and feature request using tools
on the git repository at https://github.com/darold/squidanalyzer.

How to contribute ?
Any contribution to build a better tool is welcome, you just have to send
me your ideas, features request or patches using the tools on the git
repository at https://github.com/darold/squidanalyzer

You can also support the developper by donate some contribution by clicking
on the "Donate" button on the SquidAnalyzer web site at http://squidana-
lyzer.darold.net/

AUTHOR
Gilles DAROLD <gilles@darold.net>

This package is free software and published under the GPL v3 or above li-
cense.

perl v5.42.3 2016-01-03 SQUIDANALYZER(1)

Want to link to this manual page? Use this URL:
<https://man.freebsd.org/cgi/man.cgi?query=squid-analyzer&sektion=3&manpath=FreeBSD+Ports+15.1.quarterly>

home | help

Header And Logo

Peripheral Links

Site Navigation

FreeBSD Manual Pages

Header And Logo

Peripheral Links

Search

Site Navigation

FreeBSD Manual Pages