Using the LogViewer
The LogViewer provides a convenient way to view the SDVP logs--and much more.
In addition to helping you read and search the logs more easily, the LogViewer also lets you interact with them: organizing the data, analyzing it efficiently, and taking action based on that analysis. While the Settings Manager is essential for getting started with SDVP, you will probably find the LogViewer becoming your main user interface over time.
To begin examining a site's logs, select it in the tree view control and click the LogViewer button, or just choose View Logs from the right-click menu:
This launches the LogViewer, together with the Open Logs dialog (pictured below). Use this dialog to select the log file to open. The current log file for the site you selected will be the default choice. Click the Open button to select it.
To skip the Open Logs dialog and have LogViewer load the current log file immediately, click the site name in the grid portion of the Site Status tab.
Note: SDVP logs are rotated on a twenty-four hour basis, at midnight server-local time.
LogViewer is your main tool for understanding what SDVP is finding in terms of threats and vulnerabilities, as well as for identifying potential false positives that need to be cleared up--ideally before deploying in ON mode. Here is a general view of the LogViewer interface, showing the list view on the top part of the screen, and the details pane below:
If your site is receiving live traffic, or even a substantial amount of simulated load, you may notice that LogViewer is scrolling quite frequently as new data come in. You can at any point pause the addition of new data, refresh the current paused view, or just toggle off the auto-scroll feature while allowing new data to continue coming in. All three of these features are controlled by means of the toolbar buttons pictured below. Note that auto-scrolling also disables automatically when, as pictured here, you highlight a row in the list view pane.
The Pause, Refresh, and Auto-scroll Controls:
Using LogViewer to Assess Vulnerabilities and False Positives
For best results, you should start using LogViewer while SDVP is still in LOG ONLY mode. Consider that each error reported in LogViewer will necessarily represent one of three things:
- A definite security threat, such as attempted SQL Injection, Cross-Site Scripting (XSS), or Remote File Inclusion (RFI)
- Traffic that may not be highly threatening but also not legitimate or desired (e.g., an unfriendly bot trying to harvest emails or spread link spam)
- A false positive, such as an innocent user accidentally triggering an input validation error on a particular form field, and getting blocked because of it
Being able to distinguish between threats, annoyances, and false positives is the key to balancing real Web application security with the functional needs of your Web app and its users. SDVP is designed to make these distinctions automatically as much as possible. But it will do an even better job of that with some guidance from you. LogViewer makes it easy to provide that guidance, by giving you numerous ways to search, understand, and act on data it displays.
Understanding LogViewer Data
The first thing you will notice about LogViewer is that the SDVP logs available in it are far more detailed and specialized than standard Web server transaction logs. The log records (rows) here represent the subset of HTTP requests that SDVP has flagged as actual or potential security threats. In LOG ONLY mode, these errors will not be acted upon by SDVP, but the log record will reflect the action what would have been taken had SDVP been in ON mode, and the reasons for that action.
In addition to the standard fields you might expect to see in any HTTP log (Date, Time, URL, Method, Port, Response Code, Client and Server IPs), as well as those you might find in your IIS logs if, as we recommend, you have extended logging enabled (User-Agent, Referer, Cookie), LogViewer also provides the following specialized log fields to help you assess the nature and severity of each error it records:
||This is a unique identifier for the specific error. This number will be included in any error message that SDVP sends to a browser when a security event occurs.
||This field classifies the error by the type of potential threat it represents. As such, it encapsulates much of SDVP's expert knowledge about Web application vulnerabilities. We will have much more to say about working with the Category field below, when we discuss the use of Log Filters.
||This field classifies the error with respect to its threat potential. Errors can be of high, medium, or low severity, or merely informational.
||This is the action that SDVP would have taken in response to the error, had it been in ON mode. Possible values include BLOCK and ALLOW along with other, more specialized options.
||This simply records the whether SDVP was in LOG ONLY or ON mode when the error was raised.
||This field tells whether or not the Action to be taken is the result of an Exception. (For more information about Exceptions see Creating Exceptions in Log Viewer.)
||This very useful field provides more detailed information about the specific cause of the error, beyond its assignment to an error Category. In fact the Remarks will often indicate the reason why a particular request was assigned to a particular error Category.
||This field is just what you might expect - an indication of the likely country of origin of the request that raised the error.
||This field displays the unique identifier for the SDVP session in which the error occurred. As we will see, it can be very helpful in establishing the context of an error to determine if it was a genuine threat, or not.
Besides the above fields, which are displayed as columns in its main list view, LogViewer also contains a details pane (it can be hidden but is visible by default) where each individual HTTP request that raised an error (including its POST data, if any) can be viewed.
At the top of the details pane is a status bar containing a summary of the most pertinent SDVP log data for the particular request. In most cases, LogViewer will also highlight in red the HTTP header or input parameter containing the data that caused SDVP to flag the request as an error. In the following example, the POST parameter "fname" contained a possible SQL Injection:
Searching The Logs
Since both the number of records in a typical SDVP log file and the amount of data per record can be quite large, LogViewer provides a powerful search feature that finds all occurrences of any string in virtually any LogViewer data field. This makes is very easy to find occurrences of specific URLs, timestamps, IPs, error messages, and so on.
To use the search feature, simply enter the search text in the search box in the upper right corner of LogViewer and press Enter. If there is no match the search text turns red. If there is at least one match LogViewer highlights the first row containing the matching text. In the example below, for instance, the first occurrence of the text string "libwww-perl" is being found in the User-Agent column:
If the log search matches multiple rows, you can jump to the next one by pressing Enter again. To back up through the matching rows, hold down the Shift key and press Tab.
Suppose that the text you are looking for is not in the currently-loaded log file? Perhaps the error you are investigating happened on the previous day, before the logs were recycled. In this case LogViewer makes it easy to explore and open other log files. Simply choose the Open Logs option from the File menu:
This brings up the Open Logs dialog, which lists all available SDVP logs in explorer format, with the Web sites in the tree view on the left, and their associated log files, with date and size information, in the list view on the right. Simply select the log you are interested in and in a moment it will replace the currently-loaded log file in LogViewer:
Using Log Filters
One of the most efficient ways to sort through LogViewer data is to make use of its extensive options for filtering the log data that is displayed. Working with these log filters to examine your log data is also one of the best ways to quickly gain an understanding of the various types of errors that SDVP records (and that, once in ON mode, it will begin acting upon). Below we present a brief tour of your log filtering options.
1. The first stop on our tour is the Filter Options dialog. You can launch this dialog in one of two ways: Either by pressing the toolbar button that bears the filter icon:
Or else by choosing the Set Log Filter option from the View menu:
In either case, you will see the dialog pictured below, which lists the five filters that can be applied to LogViewer data. Each of these filters corresponds to a particular column in the SDVP logs: Category, Severity, Action, Mode, and Exception. (See the LogViewer Data section for definitions of each of these fields.)
2. From here, you can selectively filter out rows from the log data by excluding Categories of errors that you are not interested in seeing at the moment. For instance, you may only be interested in seeing possible SQL Injections and XSS attacks. In that case you would clear all checkboxes in the Categories section except for those two (Hint: Start by clearing the "View All Categories" checkbox):
When you click Save, the Filter Options dialog closes and LogViewer's list view is refreshed, now showing only those errors that are in one of the three selected Categories:
3. Similarly, you can also use the Filter Options dialog to filter out errors containing SDVP "Actions" that don't interest you. You might, for instance, only be interested in errors that resulted in a BLOCK Action. This means SDVP blocked the entire request:
4. The Severity filter lets you look at how SDVP has classified different errors with respect to the threat they pose: high, medium, low, and none (informational). In the below example, for instance, all errors are being filtered out except for those that have been assigned the HIGH severity designation:
5. As you can see, the Filter Options dialog also lets you filter errors based on whether SDVP was in LOG ONLY or ON mode when the error was raised, and also whether or not an Exception was applied to the particular request. LOG ONLY mode and Exceptions both have the effect (for different reasons) of preventing the specified Action from being taken for a given error. These two filters thus let you focus on the Actions that would have been taken had SDVP been allowed to act on all of the errors it recorded:
6. Finally, the Filter Options dialog makes it easy to quickly analyze log data across multiple dimensions simultaneously. To do this, you can simply combine various types of filters to produce composite filters that logically "AND" together different types of criteria.
Suppose for instance that you wished to see all errors in the SQL Injection Category that would have resulted in a BLOCK Action, but for which you have also configured an Exception that would have overridden this Action. You might want to do this in order to check that only genuine false positives as making use of these Exceptions to the default SQL Injection security controls. In this case you would check "SQL Injection" (under Categories), "BLOCK" under Actions, and "TRUE" (under Exceptions) to filter out all errors except these:
7. In addition to working with the Filter Options dialog, you can also apply context-sensitive filters directly from within LogViewer's list view. This can be especially useful when you are working from a single error of interest that you have identified, and about which you wish to know more. For example, you can highlight a specific error in the list view and use the right-click menu option to "View This Category Only":
This will filter out all other Categories of errors so you can concentrate on errors of the selected type only:
8. You can also undo all filters from the right-click menu, by selecting the View All option:
The list view refreshes and all of the errors in the log are once again displayed, regardless of Category:
9. On occasion when assessing the SDVP logs you may wish to eliminate certain categories of errors one-by-one. A typical use for this functionality would be when you are progressively filtering out less-severe and more commonplace errors in order to focus on the more severe, rarer ones. This is easy to with the "Hide This Category" right-click menu option, which filters out all errors of the selected Category type. Here for example we eliminate presumed-innocent 404 errors from our current view of the log data:
When the list view refreshes the presumed-innocent 404 errors are no longer displayed:
10. Besides these context-sensitive Category filters, you can also apply context-sensitive Session or (client) IP filters directly to a particular request. These options are especially valuable when you are investigating a particular error and you wish to see what other errors, if any, may have been raised by the same user or process. Looking at such patterns can quickly make it very clear if you are dealing with a real threat or a false positive.
For example, to see if any other errors were raised within the same SDVP session as the currently-selected error, just right-click and select the "View This Session Only" menu option (or use "View This IP Only" if you suspect the requesting process does not accept cookies and would not have a Session ID associated with it):
After the list view refreshes you will see all of the errors which are associated with the same SDVP Session ID as the previously-selected error:
11. Finally, errors can be filtered based on the checkboxes located in LogViewer's leftmost column. These checkboxes provide a handy mechanism for defining arbitrary groups of errors that are of interest to you.
Since the checkboxes preserve their state when LogViewer is closed, they provide an excellent way to indicate that particular errors have been reviewed by you or otherwise dealt with, and no longer need attention. Alternatively, you can use them to single out a group of errors for later investigation, even if those errors do not share any other common characteristic (such as Error Category, SD Session, or IP address).
To filter out errors that have already been checked, select the View Unchecked Items option from the View menu:
When the view refreshes, the previously-checked errors will be hidden:
To filter out all errors that have not been checked, select the View Checked Items menu option:
When the view refreshes, only the previously-checked errors will be displayed:
To return to viewing both checked and unchecked errors, select View All Items:
When the view refreshes it will once again display all errors, both checked and unchecked:
Lastly, as with all LogViewer columns, the checkbox column is sortable. Thus, you can sort all errors by their checked or unchecked state, simply by clicking on the empty square at the top of the column.
Comparing SDVP and IIS Logs
While LogViewer's log filters provide quite a powerful way to sort through SDVP log data, sometimes, when assessing whether a particular error is a threat or a false positive, you need more context than the SDVP logs alone can provide. After all, those logs only contain requests that SDVP classified as errors -- that is, as violations or at least potential violations of its application security controls. Innocent errors (false positives) will often be easier to spot or confirm if you can see them in context of the non-error-generating traffic that occurred before and after the error.
Fortunately, LogViewer makes it easy to compare its results with those of your standard Web server (IIS) logs. For instance, to see the "normal" (non-error-generating) requests that may have happened during a particular user's SDVP session, select any error in LogViewer and use the "View This Session in IIS Logs" right-click menu option:
You can also choose the "View This IP in IIS Logs" option, if you suspect the error may have been generated by a process that does not accept session cookies:
In either case, the "IIS + SDVP Combined Log View" dialog will be displayed on top of the LogViewer. This dialog shows all of the requests recorded by IIS for the session (or IP) that you selected in LogViewer. The view window will automatically scroll to a point in the IIS log that includes the selected SDVP error. In addition, any requests from the same session (or IP) that raised other SDVP errors will also be highlighted.
Thus, by scrolling through this special view of the IIS logs, you can see the SDVP errors in the context of any "normal traffic" that may have occured in the same session (or originated from the same IP):
Note: Cross-analysis of logs depends on the use of a synchronized time scheme for individual log entries. Since SDVP uses the same W3C Extendend Log File Format that IIS uses by default, the dates and times in the two sets of logs will normally be synchronized to GMT. However if IIS has been configured to use a custom log format, with a non-GMT time scheme, then this feature will not function as expected.
Determining if a particular error in LogViewer is genuine or a false positive can sometimes not be done by relying on log data alone -- no matter how carefully it is analyzed. In these cases it may be useful to supplement log analysis by examining the source of the error directly. To facilitate this, LogViewer includes the following set of built-in forensic tools:
- Open the request (or referring) URL in browser
- Ping the client IP
- Trace the route of the client IP
- Perform a DNS lookup on the client IP
These tools can be accessed using the following toolbar buttons, which become active whenever a particular error is selected in LogViewer's list view:
The same tools can also be launched from the right-click menu:
Exporting Log Data
Sometimes there may be an error that you suspect of being a false positive, but the tools provided within LogViewer are not sufficient to settle the issue. In this case, you can save off all of the data associated with a that error by highlighting the relevant row in the list view and using the Export Incident Report option in the right-click menu. The resulting text file can be saved wherever you wish and used for offline investigation:
Creating Exceptions in LogViewer
Contextual Exceptions |
Global Bypass Exceptions |
Tracking the Use of Exceptions
One of the main activities you will undertake in LogViewer, and especially while SDVP is running in LOG ONLY mode, is that of creating exceptions to the Web application security controls that are otherwise in force.
The purpose of exceptions is typically to prevent false positives. That is why it is generally best, if possible, to configure them while SDVP is still in LOG ONLY mode. This minimizes the chances of legitimate traffic being blocked or otherwise interfered with once SDVP has been moved into ON mode.
When creating exceptions, the goal should always be to craft the narrowest possible exception that still permits legitimate users to do what they need to do. Highly granular exceptions of this kind allow you to maintain a set of strict and generic Web application security controls as the "normal case".
To this end, SDVP provides numerous facilities for creating granular exceptions directly from within LogViewer, so that the loop between assessment of an error (as a potential false positive) and adjustment of the relevant security control (to avoid that false positive) is as tight and specific as possible.
The exceptions you can create from within LogViewer are of two general types: Contextual Exceptions and Global Bypass Exceptions. We will cover each type in turn.
Contextual Exceptions are specific to the Category of error being assessed. (See the Using Log Filters section for a refresher on error Categories.) The four Contextual Exceptions are as follows:
1. Input Exceptions. These are available for all LogViewer errors that fall under the Categories "Input Validation", "SQL Injection" and "XSS". Suppose, for instance, that LogViewer contained a SQL Injection error such as the following, that you had determined was a false positive:
In this case, you could immediately create an Input Exception by selecting the error in the list view and choosing the Add Input Exception option from the right-click menu:
This brings up the Add/Edit Input Exception dialog, which is one of the most powerful exception creation tools in LogViewer:
In creating an Input Exception, there are a number of configuration options designed to help you structure the exception as narrowly as possible, while still eliminating the false positive:
- The Match section of the dialog describes the requests that will have this exception applied to them. When creating an exception, try to keep these Match criteria as restrictive as possible.
- Note that unchecking a Match criterion can have the effect of significantly increasing the scope of an exception, since that criterion will not be used in deciding whether to apply the exception.
- Wildcards (*) are supported for the URL Path and Query String criteria. Use these with caution since they have the effect of widening the scope of the exception.
- If you want the exception to be applied only when the same query string is used, but without requiring the values in the query string parameters to be same, then check both the Query String box and the Ignore Parameter Values box.
- Note that there is no option to disable (uncheck) the Field Name criterion. This is to prevent exceptions from accidentally being configured too broadly and thereby completely undermining SDVP's user input controls.
- The Restrictions section of the dialog specifies which user input restrictions to apply to requests that are subject to the exception. This lets you selectively relax certain user input controls, while leaving in place or even tightening up others.
- The Action pull-down menu under Restrictions specifies the action to be taken if one of the remaining user input restrictions is violated.
- The Threat Checking section allows the major user input categories of SQL injection and/or XSS checking to be selectively disabled.
2. Cookie Exceptions are available when the currently-selected error in LogViewer's list view falls under either the Cookie New or Cookie Tampering error category:
The Add/Edit Cookie Exception dialog is far simpler than the dialog for Input Exceptions:
Despite the relative simplicity of this exception dialog, you will want to pay close attention to the options under the Action pull-down. Cookies that raised a Cookie Tampering false positive may need to be excused from all checking by SDVP if they are intended to be modified on the client side (that is, in the user's browser). On other hand, a cookie that was flagged with a Cookie New error, but that is known to be trustworthy, might only need to be trusted once but then checked upon subsequent requests. This would be appropriate, for example, for a persistent cookie that contains a unique identifier (such as a user ID or user group classification) that is not intended to be changed on the client side.
3. Directory Exceptions are available when the currently-selected error is in the Directory Browsing category:
The Add/Edit Directory Exception dialog is even simpler than the one for Cookie Exceptions. Simply verify that the pre-configured directory path is indeed one for which you wish to permit directory browsing:
4. 404 Exceptions are of course available for all errors categorized as 404 errors in LogViewer. This Contextual Exception is one you may find yourself using during LOG ONLY mode, especially for unrepaired broken links that you want to eliminate as indicating any kind of malicious activity (of course SDVP will probably already have marked these 404s as "presumed innocent"):
In the Add/Edit URL Exceptions dialog that comes up when you select "Add 404 Exception" you can verify that the pre-populated URL path is indeed an "innocent" 404 -- a broken link that SDVP no longer needs to monitor. Do this by setting the Action field to "Allow." Alternatively, you can set up an automatic redirect to temporarily repair the broken link until it can be permanently fixed in your site's source code:
Global Bypass Exceptions
While their very narrow scope makes Contextual Exceptions the preferable way to deal with false positives, more-permissive exceptions are sometimes warranted. There are certain limited scenarios in which it is both safe and convenient to exempt a given resource or a given user from all security controls.
The Global Bypass Exceptions are provided for cases like these. Unlike the Contextual Exceptions, these Global Bypass Exceptions are always available, for every category of error in LogViewer. Use them with caution.
1. The URL Bypass Exception exempts the selected URL(s) from all security checks:
Note that the Bypass URL Completely dialog reproduces many of the same fields found in the "Match" section of the Add/Edit Input Exception dialog. Since however there are no "Restrictions" left to apply once a URL Bypass exception has been configured, you should be extremely cautious in expanding the scope of this type of exception with wildcards (*), or (what amounts to the same thing) by unchecking any of the optional fields. While there are legitimate uses for these options, they should be used with the greatest care so as to avoid opening up more attack surface than you intended:
2. As with the URL Bypass exception, the IP Bypass creates a complete exemption from all security checks, in this case based upon the client IP as recorded in the SDVP logs:
The Bypass IP Completely dialog is straightforward. The client IP address will be pre-populated with the value shown for the selected error in LogViewer. When using this feature on a Web server that is behind an HTTP intermediary such as a load balancer or reverse proxy, be sure that your Web server is receiving unique client IPs, and not the private IP of the intermediary device (Note: for use of the X-Forwarded-For header, see the Operational Settings tab of the Global Settings dialog, which is launched from the Settings Manager's Configure menu):
Tracking the Use of Exceptions
While SDVP exceptions are very powerful and useful mechanisms for avoiding false positives, you should be cognizant of how to monitor and investigate their possible abuse as well. One way to do this is to track the use of exceptions in LogViewer. This is easy to do, assuming that your log settings are set to a sufficiently verbose level (see the Log Management section for details).
When an exception is logged, the Exception column will read TRUE (indicating that an exception was triggered), the Severity column will always show INFO (regardless of the native severity for errors of that type). In addition, the Action will generally be ALLOW or something similar.
Here is an example showing how exceptions are logged. In this case we are showing the use of a URL Exception that was created in response to a 404 error (as you can see, even the error Category itself changes in this 404 exception scenario, from 404_SUSPECT to 404_PRESUMED_INNOCENT):
Blocking Traffic By IP, Country, URL, and Resource Type.
Sometimes it may make sense to block certain types of traffic altogether, without waiting for detailed security checks to determine if that traffic is actually malicious. For instance there might be a particular resource on a site that should never be accessed from untrusted, public IPs. Or there might be particular IPs, or even entire geographic IP ranges, that you wish to prevent from accessing any resources whatsoever.
1. The Block IP Permanently option simply means that all requests from the selected client IP will be categorically rejected. None of the normal security checks are performed since they are not needed to determine that requests from this IP should be blocked:
2. The Block This Country option allows you to direct SDVP to categorically reject all future requests from the selected request's country of origin. Again, requests from the IP addresses in question are blocked preemptively, without the need to perform detailed security checks:
3. The Block URL Completely option also obviates the need for the usual security checks. Instead, the resource corresponding to the URL is made completely unavailable. While this is not a substitute for removing unused resources from one's site, it can be an efficient stopgap measure in such cases:
4. When the requested resource is not found (i.e., a 404), the option to Deny Requests for This Resource appears in the context menu. This allows you to quickly add a given resource type to the list of resources that SDVP will block. Future requests for these non-existent resources will then automatically be treated as potential threat indicators, rather than innocent 404s.
Next: Setting Up Alerting and Reporting »