Your server can create several types of logs:
For access logs, you can set filters and create reports that help you analyze the information in the access logs. You can also set maintenance options. For error logs, you can set certain maintenance options. For the other types of logs, you can specify the name of the log and where you want it to be filed.
This chapter explains how to tailor the access and error logs to meet your needs, and also how create customized reports from the information in the logs.
The server creates many types of logs. Each day at midnight, the server closes the logs for that day and creates new logs.
The server logs activity in the access log files and stores them on the hard drive each night. At midnight each night, the server closes the current access log and creates a new access log file for the coming day. The access log contains entries for page request mode to the server.
For each access request your server receives, an entry is made in the access log showing:
The server can also create an agent log and a referer log. The agent log indicates which Web browser was used to access a Web page. The referer log identifies the Web page that referred (or linked to) the requested Web page. By default the server writes an entry to the agent and referer logs each time a client sends the server a request. For every entry made in the access log:
The server creates an error log that includes errors encountered by your server's clients, such as timing out or not getting access.
The server also creates a CGI error log that logs standard error output (stderr) from CGI programs.
If your server is running as a proxy, the server can create two different types of logs:
This section describes how to set up the logs to suit your particular needs. If you are satisfied with the default setting for an option, you can skip the step. Look at the sections that apply to you:
Note: You can change the default settings for the logs either by using the online Configuration and Administration forms or by manually editing the directives in the configuration file.
In most cases, you will want to accept the default global settings, which apply to all logs.
If you plan to use the reporting functions described under "Tailoring the reports your server creates", you must accept the default file format, common.
If you want to have log information sent to the Internet Connection Secure Server window in addition to sending it to the log files, you must change the default.
To change the global settings, we recommend that you specify them on the Global Log File Configuration form.
Common file format, which is used by most Web servers and local time format are used. By default, access log information is written only to the access log, (not the syslog) and error log information is written only to the error log, (not the syslog).
This section describes the following tasks:
From the Access Log File Configuration form, you can specify the path and name of the directory where you want to place the access, agent, and referer log files.
We strongly recommend that you accept the default path, which is the value you entered for Logs directory at installation.
With the log maintenance options, you can specify how to handle the accumulation of daily logs for days past.
You can choose whether you want to keep old logs, remove logs after they reach a certain age and/or a collective size, or run your own program at midnight each night to handle old logs. Note that the "collective size" is the collective size of all access logs only (not combined with agent and referer logs), or of all agent logs only (not combined with access and referer logs), or of all referer logs only (not combined with access and agent logs).
To reduce the space the access, agent, and referer logs require, you can specify that the logs be automatically removed, based on the age of the log and/or the collective size of the logs.
If you are interested in running your own backup program to store the logs, you can specify a user exit. In this case, you specify the path to your program and the parameters to pass to your program. The server appends to this information the path to the logs on your hard drive.
We recommend you define these options on the Access Log File Configuration form, but you can edit the configuration file to include the appropriate directives. The settings you specify on the Access Log File Configuration form apply to agent and referer logs, as well.
By default, all access, agent, and referer log files are kept on the hard drive at the path location you specify on the Access Log File Configuration form. (or the AccessLog, AgentLog, and RefererLog directives).
The directives you specify for access logs apply to agent and referer logs, as well.
For details on these directives, refer to "AccessLogArchive - Remove existing access, agent, or referer log files or run a user exit".
For the access log, you can set filters so that the access, agent, and referer logs includes only the information you are interested in.
To improve your ability to use the information included in the access, agent, and referer log files, you can filter out extraneous information so that the log includes only information that is meaningful to you. You filter out information by excluding entries that match a particular pattern. We recommend you define these options on the Access Log File Configuration form, but you can edit the configuration file to include the appropriate directives for the filters you want to set. You can specify filters based on any of the following:
Note: Keep in mind that information filtered out from the access log will not show up in any access report and will not be available for future use.
Here are some reasons for controlling what gets logged.
To reduce the size of the logs: You might be interested in reducing the number of entries in an access log to include only meaningful access requests. Access log files can grow rapidly, since by default they contain entries for all access requests for GIF images, HTML pages, and so on. You might want to configure your access logs so that they include log entries for access requests to HTML pages, but not for the access requests for the GIF images that the HTML contains. For example, an HTML page might include several GIF images, which can cause the size of the access log to grow rapidly.
To collect information about external hits only: You might be interested only in who is accessing your server from outside your company. In this case, you would filter out access requests that originate from internal company IP addresses.
To gather information about who is accessing a particular Web site: To help you determine the size of the audience for a particular Web site, you might want to create an access log that shows only the hits to one URL.
By default, everything is logged to the access log, unless you choose to filter out (exclude) something. From the Access Log File Configuration form, you can specify what you want to filter out from the access log. You do not need to fill in the entire form.
Scroll to the Exclusions from the Access log section of the form. Choose which of the following you want to base filtering on:
If you want to filter based on directories and files or IP addresses and host names, you need to update the index list on the Access Log File Configuration form. You can insert or remove entries in the list to specify what you want filtered out. To exclude entries based on methods, MIME types, or return codes, click the boxes that describe what you want to filter out.
When you have finished specifying what you want to exclude on the Access Log File Configuration form, click Apply to have the filters take effect.
This section describes the following tasks:
From the Error Log File Configuration form, you can specify the path and name of the directory where you want to place the error and CGI error log files. As an alternative, you can specify this information manually by editing the directive listed below.
We strongly recommend that you accept the default path, which is the value you entered for Logs directory at installation.
For path, edit the ErrorLog directive.
You can choose whether you want to keep old logs, remove logs after they reach a certain age and/or a collective size, or run your own program at midnight each night to handle old logs. Note that the "collective size" is the collective size of all error logs only (not combined with CGI error logs) or all CGI error logs only (not combined with error logs).
To reduce the space error and CGI error logs require, you can specify that the logs be automatically removed, based on the age of the log and/or the collective size of the logs.
If you are interested in running your own backup program to store the logs, you can specify a user exit. In this case, you specify the path to your program and the parameters to pass to your program. The server appends to this information the path to the logs on your hard drive.
By default, all error and CGI error log files are kept on the hard drive at the path location you specify on the Error Log File Configuration form (or the ErrorLog directive.)
We recommend you define these options on the Error Log File Configuration form, but you can edit the configuration file to include the appropriate directives. The settings you specify on the Error Log File Configuration form apply to CGI error logs, as well.
The directives you specify for error logs apply to CGI error logs, as well.
For details on these directives, refer to "ErrorLogArchive - Remove existing error or CGI error log files or run a user exit".
In the following example, you have just purchased and installed the Internet Connection Secure Server. You want to set up your server to log access information and error information in the following ways:
You can specify these criteria by using the Configuration and Administration forms, or by updating the configuration file directives.
For the above scenario, update the configuration file as follows:
LogFormat Common LogTime LocalTime AccessLogArchive purge AccessLogExpire 30 AccessLogSizeLimit 25 AccessLogExcludeURL *.gif NoLog 9.67.*.* AccessLogExcludeReturnCode 300 ErrorLogArchive none
From the Access Log File Configuration form, you can specify the path and file name where you want the server to put access requests that are satisfied from the proxy server's cache. As an alternative, you can specify this information manually by editing the directives listed below.
For more information, refer to "CacheAccessLog - Specify the path for the cache access log files".
For the cache access log path, edit the CacheAccessLog directive, in addition to those described under "CacheAccessLog - Specify the path for the cache access log files".
Several types of files are used in report creation. These files are located in the reports root directory
The access log data file (access.mmddyyyy) corresponds to the httpd-log.mmddyyyy file that is in the access log directory. For each entry in the access log file, there is an entry in the access log data file. The format of the data in the access log data file is:
IP_address URL seconds_since_1970 number_of_bytes_transferred method code
The report data file (template_name.txt) includes data from all the access log data files filtered through the template definition. For example, for the template "Top100", there is a data file called Top100.txt, which corresponds to all the access.mmddyyyy files filtered through the Top100 template definition and saved to the Top100.txt data file. The format of the data in the report data file is:
IP_address URL seconds_since_1970 number_of_bytes_transferred method code
The report template file (template_name.log) is the definition of the template. For example, the template "Top100" has a Top100.log file associated with it, which consists of the Top100 template definition in an ASCII file.
The report template list (the file name is Templates) is a list of all the templates that have been defined.
Before you can see a report, you must create a report template that is stored as the report template file. For instructions on how to create a report template, see "Overview of report templates".
Your server creates reports that include some or all of the contents of the access logs. At midnight each night, the server closes the current access log and creates a new access log file for the coming day. Reports are generated at that time using the access log that was just closed. Reports can also be generated for logs that have been archived.
Note: Your server might not be able to create a report for a very large access log file (60 MB, for example). If your system encounters a problem while trying to generate a report, the cause might be an access log that is too large. To generate reports against very large access log files, try increasing the RAM and (or) the swapper file space on your system. A short-term solution to this problem is to turn off report generation by renaming the htlogrep executable file.
Note: If you have specified the CacheAccessLog directive or if you have indicated on the Access Log File Configuration form a path and file name for the proxy server's cache access log, your reports will not contain access requests for cached files. If you do not have a cache access log, access requests for a proxy server are logged in the access log and can be included in an access report.
You control what is included in reports by filtering out entries that match a particular pattern. These options are defined either by using the Configuration and Administration forms or by editing the configuration file. You can use the forms or the configuration file to specify filters based on any of the following:
The contents of the report are governed by the following factors:
At report creation time, you control only the report filters that are currently in effect. You cannot include in the report entries that were filtered out from the log file.
You can specify report filters in two ways; you must decide which is easier in your situation.
In some cases, you will find it simplest to specify both include and exclude filters. In this case, it is important to understand how include and exclude filters work together. The include filters are processed first. The report function searches the access log to find all entries that match any include filter patterns. If several include filters are specified, the filters act as OR Boolean expressions. In other words, entries that match at least one of the include filters are included.
The exclude filters are processed after all include filters have been processed. The exclude filters work only on the set of entries that have been already included by the include filters. For clarification, refer to the examples under "Sample scenarios for configuring reports".
The include and exclude filters are specified on the Access Log Report Template Creation form or can be specified with the AccessReport directives.
Here are some reasons for controlling what gets reported.
To reduce the scope of the report: You might be interested in reducing the scope of the report so that it includes only a portion of what is contained in the log. You can even create several reports, each to gather different information from the same log. You might want to create your report template so that it includes log entries for access requests to HTML pages, but not for the access requests for the GIF images that the HTML contains.
To collect information about external hits only: You might be interested only in who is accessing your server from outside your company. In this case, you would filter out access requests that originate from internal company IP addresses.
To gather information about who is accessing a particular Web site: To help you determine the size of the audience for a particular Web site, you might want to create a report that shows only the hits to one URL.
To discover the top Web pages on your server: To help you determine the popularity of a particular Web site, you filter out everything in the report, except for the most visited Web pages.
Before you create a report, you must modify or create a report template that outlines what you want the report to contain. To start configuring a report template, choose one of the following options from the Access Log Report Templates form:
If you have never created a report, before it might be easier to copy an existing template and edit the copy, rather than creating a new template. To use an existing report template as the basis of a new report, choose Copy existing template.
When you have finished filling in the form, choose Apply.
On this form, you can specify some or all of the following:
Specifying entries to include is a shortcut to specifying many, many excludes. When you want to include only a few types of entries in the report, it is easier to specify what to include rather than excluding nearly everything. For example, if you want to include only access requests for a particular URL, you would include that URL, rather than excluding all the others.
The Access Log Report Template Creation form allows you to specify includes and excludes. It is important to understand how includes and excludes affect each other.
If you are using the Access Log Report Templates form, you see at the bottom of the form the field Report root directory. This field is filled in with a default directory. We recommend that you accept the default, rather than changing it. If you choose to change the default, you will need to create a new directory for the path you specify, give the directory the appropriate permissions and add a PASS statement to enable the server to honor requests to store reports in that directory.
To see a report, from the Configuration and Administration Form page, choose Access reports. From there, select the following options:
The report is created and displayed after you select the options.
You have just purchased and installed the Internet Connection Secure Server and you want to set up your server to automatically generate four different access log reports.
You are interested in knowing which Web pages on your server get the most attention. You decide to create a report that meets the following criteria:
You can specify these criteria by using the Configuration and Administration forms, or by updating specific directives in the configuration file.
AccessReportTemplate Top100 {
AccessReportDescription Top 100 page hits
AccessReportTopList 100
AccessReportExcludeURL *.GIF
}
You are running a site that distributes beta-level software and are interested in knowing what is being written to the beta directory and who is requesting PUT access. You decide to create a report that meets the following criteria:
You can specify these criteria by using the Configuration and Administration forms, or by updating specific directives in the configuration file.
AccessReportTemplate BetaPuts {
AccessReportDescription PUT requests to beta subdirectory
AccessReportIncludeURL /www/beta/*
AccessReportExcludeMethod GET
AccessReportExcludeMethod POST
AccessReportExcludeMethod DELETE
}
You are interested in knowing which files on your server are being accessed. However, you want to exclude beta programs, which have files located in the beta subdirectory. You also do not want to include any information on the "Alpha7" project, which has pages named Alpha7*.* in various subdirectories. You decide to create a report that meets the following criteria:
You can specify these criteria by using the Configuration and Administration forms, or by updating specific directives in the configuration file.
/www/beta/* /www/beta/alpha7*.*
AccessReportTemplate NoBetaAlpha7 {
AccessReportDescription Accesses, excluding beta and alpha7 requests
AccessReportExcludeURL /www/beta/*
AccessReportExcludeURL alpha7*.*
}
Your server is a department server and you want to know the access requests for that server. You also want to know access requests for the beta subdirectory, but you are not interested in knowing access requests for any Alpha7*.* files. You decide to create a report that meets the following criteria:
You can specify these criteria by using the Configuration and Administration forms, or by updating specific directives in the configuration file.
/www/alpha /www/gamma /www/delta Alpha7*.*
AccessReportTemplate {
AccessReportTemplate DeptServer_Beta-NotAlpha7
AccessReportDescription Accesses for Department Server
and accesses for beta subdirectory,
excluding Alpha7 files
AccessReportExcludeURL /www/alpha
AccessReportExcludeURL /www/gamma
AccessReportExcludeURL /www/delta
AccessReportExcludeURL Alpha7*.*
AccessReportIncludeHostName 9.67.*.*
}