An In-Depth Explanation

Every Web site, extranet, intranet or Web-based application can benefit from sending data as infrequently as possible. Designing sites with effective cache controls will dramatically reduce page load times for your users -- particularly your most loyal, repeat visitors or customers -- as well as lower your overall bandwidth consumption and free up your server resources.

The Many Types of Caches on the Web

The basic idea behind caching is simple. Instead of wasting efforts by re-downloading a resource every time it is needed, keep a local copy, and reuse it for as long as it is still valid. The most common example of Web caching is the browser cache, which stores copies of images and other page objects on an end user's hard drive for repeated use. However, there are many other caches on the Web – at origin Web servers, along the network path, and even on the end user's local area network (LAN) – but, the purpose of all of these is essentially the same.

Looking out from a local browser cache, you might next encounter a proxy cache on your local network, implemented so that other users on the LAN do not have to access the Web to fetch the same site as you. Next, the Internet Service Provider (ISP) or various transit ISPs further down the line might also employ a proxy cache to serve a site's cacheable objects to any other visitor on their network. Lastly, the actual Web site may utilize a reverse proxy cache to hold copies of dynamically generated pages in finished form. This relieves the server of the burden of repeatedly generating and delivering frequently requested pages.

We can categorize all of these Web caches into two general varieties: private and public. A private cache, most commonly a Web browser, is unique to a single user and is used to store items that should only be available for reuse by an individual end user. On the other hand, proxy and reverse proxy caches are public caches. These are used for shared resources, those items that can safely be reused by more than one end user. Figure 1 shows the common cache types found on the Web:

Figure 1: Cache Usage on the Web

Figure 1: Cache Usage on the Web

Caches are found on the Web in many places and are constantly trying to hold your site content whenever possible. While it is easy to simply allow them to dictate caching behavior, it is vital to purposefully engage them and dictate which objects should or should not be cached, and for how long to maximize site performance

Freshness and Validation

In order to make the best use of any cache we need to indicate caching rules for Web page objects, ranging from setting appropriate expiration times to indicating when a particular object should not be cached at all. Fortunately, we have all of these tools at our disposal in the form of HTTP cache controls rules.

The key to cache awareness lies in understanding the two concepts that govern how caches behave: freshness and validation. Freshness refers to whether or not a cached object is up-to-date, or in more technical terms, whether or not a cached resource is in the same state as that same resource on the origin server. If the browser or other Web cache lacks sufficient information to confirm that a cached object is fresh, it will always err on the side of caution and treat it as possibly out-of-date or stale.

Validation is the process by which a cache checks with the origin server to see whether one of those potentially stale cached objects is fresh or not. If the server confirms that the cached object is still fresh, the browser will use the local resource; if not, a fresh copy must be served.

A Basic Example of Caching

The concepts of freshness and validation are best illustrated with an example (in this case using a browser cache, but the core principles hold true to public caches as well):

Step 1: A remote site contains a page called page1.html. This page references image1.gif, image2.gif, and image3.gif and has a link to page2.html. When we access this page for the first time, the HTML and the associated GIF images are downloaded one-by-one and stored in the local browser cache.

Figure 2: Initial Cache Load

Figure 2: Initial Cache Load

Once the data is downloaded to the cache, it is "stamped," indicating where it came from and at what time it was accessed. It may also be stamped with a third piece of information: when it needs to be reacquired. But, since most sites do not stamp their data with this explicit cache control information, we'll assume that our example lacks this information.

Step 2: The user follows the link to page2.html, which has never been visited before and which references image1.gif, image3.gif, and image4.gif. In this case, the browser downloads the markup for the new page but the question is: should it re-download image1.gif and image3.gif even though it already has them cached? The obvious answer would be no, but, how can we be sure that the images have not changed since we downloaded page1.html? Without cache control information, the truth is that we can't. Therefore, the browser would need to revalidate the image by sending a request to the server in order to check if each image has been modified. If it has not been changed, the server will send a quick 304 Not Modified response that instructs the browser to go ahead and use the cached image. But, if it has been modified, a fresh copy of the image will have to be downloaded. This common request-and-response cycle is shown here:

Figure 3: Cache Check

Figure 3: Cache Check

From this basic example, it is apparent that, even when CSS, images, and JavaScript are fresh, we may not get the caching benefit we expect, since the browser still has to make a round trip to the server before it can reuse the cached copy.

The default "Automatic" caching setting in Internet Explorer partially reduces this continual chatter between browser and server by skipping revalidation of cached objects during a single browser session. You will notice that page load time is generally much quicker when revisiting the same page during the same browser session. To see the performance penalty that would otherwise be incurred by all those 304 Not Modified responses, instead select "Every visit to the page."

Note: While IE's "smart caching" does cut down on unnecessary validation requests, it is also the culprit behind IE's continual reminders to users to clear their caches in order to see new content.

Crafting Cache Control Policies

Minimizing round trips over the Web to revalidate cached items can make a huge difference in browser page load times. Perhaps the most dramatic illustration of this occurs when a user returns to a site for the second time, after an initial browser session. In this case, all page objects will have to be revalidated, each costing valuable fractions of a second (not to mention consuming bandwidth and server cycles).

On the other hand, utilizing proper cache control allows each of these previously viewed objects to be served directly out of the browser's cache without going back to the server. This effect is often visible at page load time, even with a high bandwidth connection. Users may note that your sites appear to paint faster and that "flashing" is reduced between subsequent page loads. Besides improved user perception, the Web server will be freed from responding to cache revalidation requests and can better serve new traffic.

However, in order to enjoy the benefits of caching, a developer needs to take time to write out a set of carefully crafted cache control policies that categorize a site's objects according to their intended lifetimes. Here is an example of a complete set of cache control policies for a simple e-commerce Web site:

Object Type Duration
Navigational, logo images One year
CSS, JavaScript Six months
Main header images Three months
Monthly special offer image One month
Personalized special offer image (private) Two weeks from first access
All other content (HTML, ASP, etc.) Do not cache

From the point of view of cache control, this example site has six different types of objects. Since logos and other corporate branding is unlikely to change, navigational and logo images are treated as virtually permanent. CSS and JavaScript files are given freshness lifetimes to support a regular, semi-annual update schedule.

As fresh site content is important in terms of search engine optimization and user experience, the main header images are set up to be changed a bit more frequently. The monthly "special offer" image is, of course, designed to stay fresh for only one month. There is also a personalized special offer image that remains fresh in a user's cache for two weeks after the initial visit; note that this category is marked "private" to indicate that it is not to be cached by a shared/proxy cache.

Finally, the default policy for everything else on the site states that nothing else should be cached, thus guaranteeing that text and dynamic content is served fresh for each request.

You need to be very careful not to cache HTML pages, whether or not they are statically generated, unless you really know what you are doing; if anything, you should use cache control to make sure these pages are not cached. If a user caches your HTML page, and you set a lengthy expiration time, they will not see any content changes you may make until the cached object expires. On the other hand, if you focus on caching dependent objects, such as images, Flash files, JavaScript, and style sheets, you can replace that cached content simply by renaming objects.

For example, let's say you have a policy to change your site's logo files once a year, but in the middle of that year, your company makes a significant branding change that needs to be reflected on the site. Fortunately, if you have not set your HTML files to be cached, you can still serve the new logo by renaming the file from logo.gif to newlogo.gif and changing the associated HTML <img> references. As the HTML is parsed, the browser will note that it does not have the new image in its cache and will download it. The old image may stay in the user's cache for quite some time, but it will no longer be used.

After carefully considering what site resources should and should not be cached, the next step is to implement those policies.

Taking Charge of Caching

There are three methods of setting cache control rules for the Web:

  1. Specify cache control headers via a tag
  2. Program in Cache Controls
  3. Administer HTTP headers using Web server settings (using CacheRight to simplify configuration)

Each of these approaches has both pros and cons that we briefly summarize:

1. Using Tags for Basic Caching

The simplest way to implement cache control is to use the <meta> tag. For example, we could set the Expires header to sometime in the future:

<meta HTTP-EQUIV="Expires" content="Sun, 31 Oct 2011 23:59:00 GMT" />

In this case, a browser parsing this HTML will assume that this page does not expire until October 2011 and will add it to its cache. Because the page will be stamped with this Expires header, the browser won't re-request the page until after this date or until the user modifies the browser's caching preferences or clears the cache manually.

Of course, while it is often advantageous to cache page data, as we mentioned above there are instances when you would not want to cache data at all. In this case, you might set the Expires value to be sometime in the past:

<meta HTTP-EQUIV="Expires" content="Sat, 13 Dec 2003 12:59:00 GMT" />

(You might be concerned about clock variations on the user's system and therefore set the date far in the past, but in reality, this is rarely an issue since it is the server's Date response header response that matters for cache control.)

The use of Expires with a past date in a <meta> tag should work for both HTTP 1.0- and 1.1-compliant browsers. There are two more <meta> tags that are often used to make sure that a page is not cached. The Pragma tag is used to talk to HTTP 1.0 browser caches, while the Cache-Control tag is used for HTTP 1.1 clients. It never hurts to include both of these if you want to make sure that a page is never cached, regardless of browser type or version:

<meta HTTP-EQUIV="Pragma" content="no-cache" /> <meta HTTP-EQUIV="Cache-Control" content="no-cache" />

As easy as <meta> tags might appear, they suffer from one major problem -- they can’t be read by intermediary proxy caches, which generally do not parse HTML data, but instead rely directly on HTTP headers to control caching policy. Because of this lost potential value, and given the fact that browsers will readily use HTTP headers, <meta>-based cached control really should not be a developer's primary approach to cache control.

2. Programming Cache Control

Most server-side programming environments, such as PHP, ASP, ASP.NET, and ColdFusion, allow you to add or modify the HTTP headers that accompany a particular response. To do this in ASP, for example, you would use properties of the built-in Response object by including code such as this at the top of your page:

Response.Expires = "1440"
Response.CacheControl = "max-age=86400,private"

Here you are asking ASP to create both an Expires header (for HTTP 1.0-compliant caches) and a Cache-Control header (for HTTP 1.1 caches). You are also specifying a freshness lifetime for this cached object of twenty-four hours (note that the Expires property requires a value in minutes while Cache-Control uses seconds). As a result, the following headers would be added to the HTTP response (assuming "now" is 8:46 PM on Friday, February 13th, 2004 Greenwich Mean Time):

Expires: Sat, 14 Feb 2004 20:46:04 GMT
Cache-control: max-age=86400,private

This and similar mechanisms in the other major server-side programming environments is a far more efficient way of communicating with caches than by relying on tags. So, when you have a choice between implementing cache-control policies using the element and doing so using a server-side programming environment like ASP, always choose the latter.

However, there is a different issue about which the server-side programming environment can do nothing. Imagine that the ASP file for which we created the above code links to several images that, according to your cache control policies, have freshness lifetimes of a full year. How would you implement the HTTP headers to tell caches that they can store those images for that long? You could try to use a server-side script to return the images programmatically, but this is both complex and wasteful. A better approach to setting cache control information for static externals like CSS, JavaScript, and binary objects, is by setting cache control information on the server itself.

3. Administering Cache Control through the Web Server

Both Microsoft IIS and Apache Web servers provide a variety of facilities for cache control, and each has a different approach to caching and the delegation of cache control policies.

Microsoft IIS Cache Control

If you are setting cache control rules on Microsoft's Internet Information Services (IIS) Web server, you need access to the IIS Metabase, which is typically accessed via the Internet Service Manager (ISM), the Microsoft management console application that controls IIS administrative settings.

To set an expiration time in IIS, simply open the ISM, bring up the property sheet for the file or directory whose expiration time you want to configure, and click on the "HTTP Headers" tab. Next, put a check in the box labeled "Enable Content Expiration" and, using the radio buttons, choose one of the options provided. You can choose to expire the content immediately, set a relative expiration time (in minutes, hours, or days), or set an absolute expiration time. Note that both Expires and Cache-Control headers will be inserted into the affected responses. The basic idea is shown here.

Figure 5: IIS Cache Control Settings

Figure 5: IIS Cache Control Settings

Despite the user-friendliness of the GUI, it is very time-consuming to set up different cache control policies for different categories of files on IIS. This becomes a greater challenge if the site's files are not neatly segregated into directories based on their desired freshness lifetimes. If you happen to have designed your site with caching in mind, and put different classes of files in different directories, such as /images/dynamic, /images/static, /images/navigation, and so on, it is easy to set up caching policies via the MMC. But, if you don't do this, or if you are working to optimize an existing site, you may literally have to set the policy on each and every file, which would be quite a chore. There is also no easy way in IIS to delegate authority to developers to set cache policy, as modification of the required settings requires MMC access.

Fortunately, you can easily add in these features to IIS with CacheRight from Port80 Software.

CacheRight creates a single, text-based rules file that lives in each Web site's document root that allows both administrators and developers to set expiration directives for an entire site. In addition this ExpiresByPath directive, CacheRight delivers a ExpiresByType directive to control cache control by MIME type. Armed with this functionality, it becomes trivially easy to set both a general cache control policy for files of a given type and to then override that rule with a more specific one for a subset of files of that type, as in this example:

ExpiresByType image/* : 6 months after access public
ExpiresByPath /navimgs/*, /logos/* : 1 year after modification public

Here, all images will have a freshness lifetime of six months, except for those located in the navimgs and logos directories. CacheRight lets you set the expiration times relative to the modification time of the file(s), as well as to the user's first access. This flexibility can be very useful when publication or update schedules are not set in stone, which is, of course, all too common in Web development.

Apache Cache Control

When it comes to implementing cache control policies, users of the Apache Web server can install the mod_expires Apache module.

With mod_expires, a server administrator can set expiration lifetimes for the different objects on a site in the main server configuration file (usually httpd.conf). As is often the case for Apache modules, both the Virtual Host and Directory containers can be used to specify different directives for different sites, or even for different directories within a given site..

Even handier is mod_expires' ExpiresByType directive, which allows you to set the expiration lifetime for all files of a given MIME type with a single line of code. This directive allows you to easily set cache policies for all the scripts, style sheets, or images in your site. Of course, you often need to craft more fine-grained policies based upon object type or directory. In this case, the settings specified in the primary configuration file can be overridden (at the server administrator's discretion) by directives in an .htaccess file for a given directory and its children. In this way, developers can write and maintain their own cache control directives without requiring administrative access to the server, even in a shared hosting environment.

If your Apache server was not built with mod_expires, the best way to enable it is to build it as a shared object (using aspx is generally easiest) and to then include the following line in your httpd.conf file:

LoadModule expires_module modules/

As mentioned above, you can then put your configuration directives right into httpd.conf. However, many administrators will want to locate these in an external configuration file to keep things neat. We will follow this practice in our example by using Apache'sInclude directive in httpd.conf (the IfModule container is an optional, but traditional, safety measure):

<IfModule mod_expires.c>
   Include conf/expires.conf

We can now locate the directives that control how mod_expires behaves in a module-specific configuration file called expires.conf. Here is a sample set of such directives:

ExpiresActive On
ExpiresDefault "access 1 month"
ExpiresByType image/png "access 3 months"

The ExpiresActive directive simply enables mod_expires, while the ExpiresDefault directive sets up a default expiration lifetime that will be used to create Expires and Cache-Control headers for any files that don't have a more specific rule applying to them. Note the syntax used for specifying the expiration lifetime; the time unit can be anything from seconds to years and the base time can be specified as modification, as well as access.

Next is the very useful ExpiresByType directive mentioned earlier, here applied to all .png image files on the server:

<Directory "/usr/local/apache/htdocs/static">
AllowOverride Indexes
ExpiresDefault "access 6 months"

Finally we have a Directory container that overrides all other rules for anything in the directory /static. This directory has its own ExpiresDefault directive, as well as an AllowOverride directive that allows the settings for itself and its children to be overridden by means of .htaccess files. The .htaccess file, in turn, might look like this:

ExpiresByType text/html "access 1 week"

Note that this overrides any directives that would otherwise have applied to files in /static and its children that have the MIME type text/html. By using a combination of configuration file directives, and then overriding those directives by using .htaccess, almost any set of cache control policies, no matter how complex, can be easily implemented by both administrators and, if properly delegated, developers.

Regardless of which Web server you use, it is well worth the time to figure out how to manage cache control here at the server level. As with programmatic cache control, the directives will be respected by well-behaved intermediary caches and not just by browser caches, as in the case of the <meta> tag approach. Furthermore, unlike with programmatic or <meta> tag-based cache control, it is very easy using server-based cache control to set caching policies for all the really heavy objects in a site, such as images or Flash files. It is in the caching of these objects where performance gains are most obvious.

The Benefits of Caching

This article only scratches the surface of the complex topic of caching. Hopefully developers better understand why it is vital to have a set of cache control policies in place. The results of properly applied cache rules will be obvious -- dramatically faster page loads for end users, especially repeat visitors. In addition, effective caching makes more efficient use of bandwidth and server resources to free up availability for new site/app visitors. The fact that all these enhancements can be achieved through little more than a heightened attention to HTTP headers, and perhaps minimal software investments, makes effective, expiration-based cache-control one of the most cost-effective performance optimizations you can make to your Web site or Web-based application.