REST and URI Caching

cachinghateoasresturi

As I understand it, using a hypertext-driven RESTful Web service, a client is not supposed to know anything about server URI layout except for a couple of well-known entry points. This is supposed to enable the server to control its own URI space and reduce coupling with the client.

When a client for the service sends a successful request to create a new resource, the service responds 201 CREATED and gives the URI at which the new resource can be accessed in the Location header field.

Should a client be allowed to store this URI to enable direct access to the resource in the future and if so for how long? If URIs are cached by the client, this seems to be setting up a situation in which every time the server changes its URI layout, it will need to make sure a permanent redirect gets served when old URIs are accessed. Otherwise the client breaks. Over several years, this system of redirects could get out of hand.

This situation would not appear to have given the server much more control over its URI space than a REST-RPC hybrid approach using URI templates.

There's a lot of information available about caching representations, but what about caching URIs in hypertext-driven RESTful systems?

Best Answer

Remember as Tim Berners-Lee said, "cool URLs don't change". Once the server hands out a URI to the client, it is now the server's job to keep the URI working in the future by (for instance) sending a Moved-Permanently response in case the URI changed and someone requests the old one.

This is actually what encourages many to design opaque URIs, such as based on database ids or timestamps, rather than using some human-readable property of the resource in the URI. Anything that people understand, they will want to change.

Introduction

The correct minimum set of headers that works across all mentioned clients (and proxies):

Cache-Control: no-cache, no-store, must-revalidate
Pragma: no-cache
Expires: 0

The Cache-Control is per the HTTP 1.1 spec for clients and proxies (and implicitly required by some clients next to Expires). The Pragma is per the HTTP 1.0 spec for prehistoric clients. The Expires is per the HTTP 1.0 and 1.1 specs for clients and proxies. In HTTP 1.1, the Cache-Control takes precedence over Expires, so it's after all for HTTP 1.0 proxies only.

If you don't care about IE6 and its broken caching when serving pages over HTTPS with only no-store, then you could omit Cache-Control: no-cache.

Cache-Control: no-store, must-revalidate
Pragma: no-cache
Expires: 0

If you don't care about IE6 nor HTTP 1.0 clients (HTTP 1.1 was introduced in 1997), then you could omit Pragma.

Cache-Control: no-store, must-revalidate
Expires: 0

If you don't care about HTTP 1.0 proxies either, then you could omit Expires.

Cache-Control: no-store, must-revalidate

On the other hand, if the server auto-includes a valid Date header, then you could theoretically omit Cache-Control too and rely on Expires only.

Date: Wed, 24 Aug 2016 18:32:02 GMT
Expires: 0

But that may fail if e.g. the end-user manipulates the operating system date and the client software is relying on it.

Other Cache-Control parameters such as max-age are irrelevant if the abovementioned Cache-Control parameters are specified. The Last-Modified header as included in most other answers here is only interesting if you actually want to cache the request, so you don't need to specify it at all.

How to set it?

Using PHP:

header("Cache-Control: no-cache, no-store, must-revalidate"); // HTTP 1.1.
header("Pragma: no-cache"); // HTTP 1.0.
header("Expires: 0"); // Proxies.

Using Java Servlet, or Node.js:

response.setHeader("Cache-Control", "no-cache, no-store, must-revalidate"); // HTTP 1.1.
response.setHeader("Pragma", "no-cache"); // HTTP 1.0.
response.setHeader("Expires", "0"); // Proxies.

Using ASP.NET-MVC

Response.Cache.SetCacheability(HttpCacheability.NoCache);  // HTTP 1.1.
Response.Cache.AppendCacheExtension("no-store, must-revalidate");
Response.AppendHeader("Pragma", "no-cache"); // HTTP 1.0.
Response.AppendHeader("Expires", "0"); // Proxies.

Using ASP.NET Web API:

// `response` is an instance of System.Net.Http.HttpResponseMessage
response.Headers.CacheControl = new CacheControlHeaderValue
{
    NoCache = true,
    NoStore = true,
    MustRevalidate = true
};
response.Headers.Pragma.ParseAdd("no-cache");
// We can't use `response.Content.Headers.Expires` directly
// since it allows only `DateTimeOffset?` values.
response.Content?.Headers.TryAddWithoutValidation("Expires", 0.ToString());

Using ASP.NET:

Response.AppendHeader("Cache-Control", "no-cache, no-store, must-revalidate"); // HTTP 1.1.
Response.AppendHeader("Pragma", "no-cache"); // HTTP 1.0.
Response.AppendHeader("Expires", "0"); // Proxies.

Using ASP.NET Core v3

// using Microsoft.Net.Http.Headers
Response.Headers[HeaderNames.CacheControl] = "no-cache, no-store, must-revalidate";
Response.Headers[HeaderNames.Expires] = "0";
Response.Headers[HeaderNames.Pragma] = "no-cache";

Using ASP:

Response.addHeader "Cache-Control", "no-cache, no-store, must-revalidate" ' HTTP 1.1.
Response.addHeader "Pragma", "no-cache" ' HTTP 1.0.
Response.addHeader "Expires", "0" ' Proxies.

Using Ruby on Rails:

headers["Cache-Control"] = "no-cache, no-store, must-revalidate" # HTTP 1.1.
headers["Pragma"] = "no-cache" # HTTP 1.0.
headers["Expires"] = "0" # Proxies.

Using Python/Flask:

response = make_response(render_template(...))
response.headers["Cache-Control"] = "no-cache, no-store, must-revalidate" # HTTP 1.1.
response.headers["Pragma"] = "no-cache" # HTTP 1.0.
response.headers["Expires"] = "0" # Proxies.

Using Python/Django:

response["Cache-Control"] = "no-cache, no-store, must-revalidate" # HTTP 1.1.
response["Pragma"] = "no-cache" # HTTP 1.0.
response["Expires"] = "0" # Proxies.

Using Python/Pyramid:

request.response.headerlist.extend(
    (
        ('Cache-Control', 'no-cache, no-store, must-revalidate'),
        ('Pragma', 'no-cache'),
        ('Expires', '0')
    )
)

Using Go:

responseWriter.Header().Set("Cache-Control", "no-cache, no-store, must-revalidate") // HTTP 1.1.
responseWriter.Header().Set("Pragma", "no-cache") // HTTP 1.0.
responseWriter.Header().Set("Expires", "0") // Proxies.

Using Clojure (require Ring utils):

(require '[ring.util.response :as r])
(-> response
  (r/header "Cache-Control" "no-cache, no-store, must-revalidate")
  (r/header "Pragma" "no-cache")
  (r/header "Expires" 0))

Using Apache .htaccess file:

<IfModule mod_headers.c>
    Header set Cache-Control "no-cache, no-store, must-revalidate"
    Header set Pragma "no-cache"
    Header set Expires 0
</IfModule>

Using HTML:

<meta http-equiv="Cache-Control" content="no-cache, no-store, must-revalidate">
<meta http-equiv="Pragma" content="no-cache">
<meta http-equiv="Expires" content="0">

HTML meta tags vs HTTP response headers

Important to know is that when an HTML page is served over an HTTP connection, and a header is present in both the HTTP response headers and the HTML <meta http-equiv> tags, then the one specified in the HTTP response header will get precedence over the HTML meta tag. The HTML meta tag will only be used when the page is viewed from a local disk file system via a file:// URL. See also W3 HTML spec chapter 5.2.2. Take care with this when you don't specify them programmatically because the webserver can namely include some default values.

Generally, you'd better just not specify the HTML meta tags to avoid confusion by starters and rely on hard HTTP response headers. Moreover, specifically those <meta http-equiv> tags are invalid in HTML5. Only the http-equiv values listed in HTML5 specification are allowed.

Verifying the actual HTTP response headers

To verify the one and the other, you can see/debug them in the HTTP traffic monitor of the web browser's developer toolset. You can get there by pressing F12 in Chrome/Firefox23+/IE9+, and then opening the "Network" or "Net" tab panel, and then clicking the HTTP request of interest to uncover all detail about the HTTP request and response. The below screenshot is from Chrome:

Chrome developer toolset HTTP traffic monitor showing HTTP response headers on stackoverflow.com

I want to set those headers on file downloads too

First of all, this question and answer are targeted on "web pages" (HTML pages), not "file downloads" (PDF, zip, Excel, etc). You'd better have them cached and make use of some file version identifier somewhere in the URI path or query string to force a redownload on a changed file. When applying those no-cache headers on file downloads anyway, then beware of the IE7/8 bug when serving a file download over HTTPS instead of HTTP. For detail, see IE cannot download foo.jsf. IE was not able to open this internet site. The requested site is either unavailable or cannot be found.

The difference between a URI, a URL and a URN

URIs identify and URLs locate; however, locators are also identifiers, so every URL is also a URI, but there are URIs which are not URLs.

Examples

Roger Pate

This is my name, which is an identifier. It is like a URI, but cannot be a URL, as it tells you nothing about my location or how to contact me. In this case it also happens to identify at least 5 other people in the USA alone.

4914 West Bay Street, Nassau, Bahamas

This is a locator, which is an identifier for that physical location. It is like both a URL and URI (since all URLs are URIs), and also identifies me indirectly as "resident of..". In this case it uniquely identifies me, but that would change if I get a roommate.

I say "like" because these examples do not follow the required syntax.

Popular confusion

From Wikipedia:

In computing, a Uniform Resource Locator (URL) is a subset of the Uniform Resource Identifier (URI) that specifies where an identified resource is available and the mechanism for retrieving it. In popular usage and in many technical documents and verbal discussions it is often incorrectly used as a synonym for URI, ... [emphasis mine]

Because of this common confusion, many products and documentation incorrectly use one term instead of the other, assign their own distinction, or use them synonymously.

URNs

My name, Roger Pate, could be like a URN (Uniform Resource Name), except those are much more regulated and intended to be unique across both space and time.

Because I currently share this name with other people, it's not globally unique and would not be appropriate as a URN. However, even if no other family used this name, I'm named after my paternal grandfather, so it still wouldn't be unique across time. And even if that wasn't the case, the possibility of naming my descendants after me make this unsuitable as a URN.

URNs are different from URLs in this rigid uniqueness constraint, even though they both share the syntax of URIs.