As with most data transfer events, there are two basic ways to send an HTTP message: All at once, or in pieces. In other words, HTTP provides the option of streaming data until there is no more to send, rather than sending all the data in a single logical unit with a known size.
If you are a long-time Web developer, you are probably familiar with how to do flushes in server-side scripts. This technique allows a script to start sending some data down to the user, while it goes on and does some relatively slow processing (say, a costly database query). If you have ever written such a thing, then you have also taken advantage of the underlying HTTP streaming mechanism, whether or not you knew the details of how HTTP was getting the job done.
But this is not just ancient history. If you are a contemporary Web developer or an administrator responsible for taking care of the code written by such developers, then HTTP streaming is probably going to end up being an increasingly important part of your professional life -- whether you realize it or not -- due to the increasing importance of Ajax in contemporary Web applications. The statefulness and responsiveness that developers are turning to Ajax to achieve partly depends on the fact that HTTP (among other neat tricks) can send and receive data in stream-wise fashion.
So, at this point, you might be thinking, "Well it's cool that HTTP does this for me, but I don't really want to know how it all happens. Why should I? I just want to use the darn thing, not become an expert in it."
Here is where we get to trot out one of our favorite Spolskyisms -- Spolsky's Law of Leaky Abstractions -- which says this: All abstractions leak. And there is also an important corollary to the law, which we would phrase like this: When an abstraction starts leaking on you, if you don't understand how that abstraction works, you are going to get very wet.
Now it's true that, most of the time, whether you are a developer or administrator, you don't have to worry about any of this. It is all mercifully abstracted away for you by the HTTP implementation in your development environment's libraries, or in your end-user's browser, or on your Web server. If it weren't for such lovely abstractions that hide all the sausage-making we don't really care about from us, we would never get anything done. Instead, we would be spending all our time googling for things like this article.
But now suppose you are administering a Web application that uses Ajax, and you change something in the Web server's configuration, and a particular widget in the application suddenly starts hanging -- even though the Web server and the server-side of the application appear to be perfectly healthy, and even though the data seems to be getting to client, according to your network traces. Or suppose you are implementing a Web service, and it works fine when the consumer asks the provider for the data using HTTP 1.0, but when it asks using HTTP 1.1, suddenly the message no longer parses correctly. Cases like these, and many others of similar weirdness, could very well be the result of the HTTP streaming abstraction breaking down.
It just might be worth understanding some of the underlying concepts and complexities after all!
For instance, did you know that there are two different ways in HTTP to stream data? All together, that makes three different ways that an HTTP message can get from its provider to its consumer. And did you know that each of these ways that an HTTP message can be sent has different implications, in turn, for what happens on the next lowest layer in the protocol stack, down in TCP? In particular, that each way of sending a message in HTTP affects what happens with the duration of the underlying TCP connection?
Lost yet? If so, don't feel bad. Whenever we at Port80 find ourselves having to bring this kind of thing up around fellow Web professionals, we are always a little surprised by how few of them -- even very seasoned and accomplished ones -- have any clue what we are talking about. Fortunately, HTTP streaming is really not that hard to understand. All you have to do is start at the end.
No, that wasn't a typo. The end really is the place to start. It's like this:
HTTP messages can contain a payload of data: an entity body in HTTP-speak (not that this is always the case: think of HEAD requests). When they do, the HTTP implementation that has to handle that data needs to know when it has finished receiving the last of it. This is a requirement whether we are talking about a client like a browser, a bot, a Web service consumer, or a server handling data from the client, as in the case of a POST. The key to understanding HTTP streaming is to focus on how it is that the process at the receiving end of an HTTP message knows, in the course of handling the data contained in that message, that it has reached the end of the data.
There are only three possible ways the process on the receiving end of the message can know this. As you might have guessed, these three ways happen to correspond to the one non-streaming and the two stream-wise ways of sending data in HTTP:
1) The non-streaming case. The most straightforward way of telling the receiving process where the end of the HTTP message is to use the Content-Length header. When this header is present, an HTTP client or server can count on the message size matching the number of bytes indicated in header value. Content served from static files almost always contains this header, in which case the value is the size (in bytes) of the file itself -- which in the HTTP message becomes the entity body (the data minus the headers). Likewise, POST requests from a client contain a Content-Length header.
Use of Content-Length makes knowing where the end of the data is pretty efficient and foolproof. Unfortunately, it also precludes streaming the data by sending it in an arbitrary number of pieces of arbitrary size. The server streaming data to a client won't know, at the beginning of a stream, how many pieces are to come, or how large each one will be. If it did, it probably wouldn't need to stream the data in the first place. You see the dilemma.
2) Enter HTTP 1.0 style streaming. If an HTTP message has a entity body, but it also lacks a known content length at the point when the message starts to be sent, then the simplest way to tell the HTTP client implementation on the receiving end that it has got all the data is to close the underlying TCP connection. Nothing works like a closed connection to say, "That's it, no more!" If you use an HTTP protocol analyzer (like HttpWatch)to look at the header section of an HTTP response that is streamed in this fashion, you will usually see this header below:
Connection: close
indicating the intention of the server to close the underlying connection, as soon as it is done streaming the data (technically, the header need not be present in an HTTP 1.0 context, since the default connection type for HTTP 1.0 is non-persistent, meaning 1.0 implementations assume that the connection will be closed unless accompanied by a Connection: Keep-Alive header). If you look at this same response at the TCP/IP level (by using Wireshark or an equivalent tool), you will see that immediately after the server sends the last of the data to which the Connection: close header applies, it does in fact initiate a full-duplex close at the TCP layer, by sending a packet with the FIN and ACK bits set. All this works very well as a way of telling the client where the end of the stream is.
There is obviously a rather nasty trade-off here, though. Yes, by this means a server can tell a client that it is done sending data. But the price is that the TCP connection over which the data was sent is torn down, and it will need to be rebuilt if any more messages are going to change hands. This vitiates the whole idea of what HTTP 1.0 used to call keep-alives (what in HTTP 1.1 we call persistent connections).
Is that a big deal? Well, yes, it can be, depending on the circumstances. The reuse of existing TCP connections, when possible, is a major performance optimization. It not only saves the time required to set up a new pair of TCP/IP sockets, it also makes for much more efficient use of networking resources. This is true both for browsers, which can multiplex numerous requests over a few persistent connections when loading a page with many dependencies, as well as servers, which would rapidly accumulate connections in the TIME_WAIT state if clients were not able to keep reusing existing connections. Optional in HTTP 1.0 (where it is indicated by the Connection: Keep-Alive header), persistent connections were made the default behavior in HTTP 1.1 -- precisely to address these types of concerns.
Unfortunately, there was no way in HTTP 1.0 to avoid a sharp trade-off between steaming data and keeping the TCP connection alive. You had to choose, because there was no other way besides cutting the connection to tell the client where the end of the data stream was….
3) You say you would like to be able to avoid this trade-off? So did the folks who wrote the HTTP 1.1 specification. They were keenly aware that the "Keep-Alive" mechanism had been something of an afterthought bolted onto HTTP 1.0, and that it made no provision for streaming. And so we have two big changes in HTTP 1.1: First, persistent connections were made the default connection type (what you get unless you tell the program on the other side that you want to close the connection); second, something called "chunked transfer encoding" was added to the mix.
The presence in the header section of an HTTP 1.1 response of this header below:
Transfer-Encoding: Chunked
means that the message was sent using chunked data. If you examine such a response at the TCP/IP level, you will notice that the connection normally remains open after all the data associated with this header has been sent and received -- there is no FIN/ACK coming immediately from the server after the last chunk. Instead, the socket usually gets reused for the next request in line.
How is this possible? Meaning: How does the browser know where the end of the data stream is? After all, it can't deduce this any more from the fact that the connection is being torn down.
As it happens, chunked encoding provides a way to tell the browser where it is in the stream at all times, as well as when it has gotten to the end. First, each chunk of data is preceded by a field that indicates the length of that particular chunk in hexadecimal bytes (represented using the ASCII characters). The length field and the chunk of data corresponding to it are each terminated with a CRLF sequence. Then comes the next length/data pair, and so on, until all the data is received. Only the last chunk lacks a length field, because it is always a value of zero -- more accurately, it always contains the value 0 before the terminating CRLF. This is how the client is able to determine that the stream has run its course…
Hopefully, this walking tour of streaming and chunked transfer encoding will make it a bit easier to understand these HTTP fundamentals and will help in Web site and application debugging. This topic often comes up at Port80 in relation to HTTP compression with httpZip and ZipEnable, so if we can clarify anything for you, just let us know.
- Port80 Software