[200 OK]: A Port80 Software Blog

HTTP 101: How to Talk to Your Web Server

The world is chock full of wonderful HTTP tools that administrators and developers can use to test, troubleshoot, benchmark, and otherwise fiddle with the Web sites and applications they are responsible for. 

Some of our favorites are Sam Spade and the browser plugins HttpWatch for IE, LiveHeaders for Mozilla, along with a couple of old standbys from Microsoft -- Network Monitor (available as an optional component on Windows Servers) and WFetch (in the IIS 6.0 resource kit tools).  For doing traces on SSL we like Ethereal.

Having said that, it still distresses me when I find out that some of the people who work with Web servers all the time, and who are smart enough to use tools like those mentioned here, have no real understanding of what even the simplest of such tools are actually doing, under the hood. 

Such folks are at a real disadvantage -- they may get some value out of using these tools, but they won't have much of a chance of extrapolating from what the tools tell them, and really understanding what a browser, or a search engine bot, or a cracker's script is doing down on the wire, when it hits their server or application.

You might say that these tools allow admins and developers to eavesdrop on (or emulate) the underlying conversation between a Web server and a Web browser.  But if the eavesdroppers using the tools aren't fluent enough in HTTP, they'll be in the position of eavesdroppers in a foreign land -- they won't understand half of what they are hearing.

Maybe I'm just being cranky, but I strongly feel that there is a big cost associated with such ignorance of the basics.  It's a bit like knowing just enough math to use a calculator -- there's no way you are not going to get in trouble or miss something important when the task turns out to be more than routine. 

Not having a nice, clear picture of the underlying mechanism in your mind, you won't know what to do with unexpected or anomalous results, and you won't have any way of cross-checking what the tool tells you.  Besides, the tools themselves won't always be handy when you need them, to avoid catastrophe and  save the day -- or to look cool at a cocktail party.

 OK, end of sermon. 

Now if I've converted you -- if you too now believe that every Web administrator and developer should know enough HTTP to be able to hold a simple conversation with a Web server -- but if you aren't sure how to go about doing this yourself, keep reading.  What follows is a quick example of the kind of thing I have in mind (the example assumes you're on Windows workstation):

First, some logistics (your still going to use a tool, just a more basic one, and you need to set that tool up).  Open a command window.  Click on the icon in the upper left to a get the menu, and select Properties.  Go to the Layout tab and, in the Screen Buffer Size frame, raise the Height value to something reasonable, like 1000.  You can skip this step if you like but you may regret it later.  Dismiss the Property dialog.

Now in the command window type telnet to get the telnet prompt.  You'll see something like this:

At the telnet prompt, type:

or just localecho (no underscore) if you're on XP (kudos to Jeff for noticing).  This is to make sure you can see your side of the conversation.  [CR/LF] means one carriage return/line feed combination, which you create by pressing the Enter key once, if that's not obvious.  Next type:

where hostname is the name of the Web site you want to check out.  Now you'll see a blank screen.  Don't panic.  You just connected to your Web server.  If you were a browser, making your first request of a session, that's exactly what you would have to do.  How do you know it's a Web server?  Well, strictly speaking you don't, yet.  All you know is that something was listening for requests on port 80 on that host.  Usually, that'll be an HTTP server -- unless somebody is being cute.

Now type the following:

Note that last CR/LF!  If you forget it, the command window will just sit there, blinking at you, and you'll be saying to yourself, "This is silly!  Why won't it work!"  It won't work because your Web server is not a mind reader.  It needs to see two CR/LF combinations to know that you are done sending the request and its associated headers.  Notice the single CR/LFs after each line.  Think of these as punctuation marks that complete your HTTP sentences -- leave out the punctuation and the Web server won't see anything but a big run-on sentence that just goes on forever and ever and never never stops how annoying isn't it? 

The first line is the request line, and in it, you're essentially asking the Web server to give you the default document (or "homepage") and telling it that you're talking HTTP version 1.1, rather than 1.0.  Because you're talking 1.1, you need the second line, which is telling the Web server what hostname you want it to use in resolving the request.  This, by the way, is how HTTP 1.1 supports name-based virtual hosting -- the hosting of multiple Web sites on a single IP/port combination, and differentiating them based on hostname.

If you did it all right, you will immediately see a big mess of ASCII text streaming by in the command window.  That's your Web site's homepage -- or, strictly speaking, the containing file of your Web site's homepage, which is probably some HTML, static or dynamically generated.  It also includes the HTTP response line and headers that the Web server sent back, and which you can look at by scrolling up to the top of the file.  At this point, if your homepage is large and you skipped that step up above (of resetting the command window's screen buffer height) you may be saying, "Hey!  I can't see that header stuff because it scrolled so far up that it disappeared."  Live and learn.

If you can still see the response line and headers from the server (and if your server is an IIS 5.0 box), they will probably look something like this:

followed by a blank line (which is there because the Web server also marks the end of its header section with a double CR/LF combination -- see how HTTP much we're learning here?)  You might also notice at this point that the Web server is still waiting for another request from you (you can test this by trying the request again -- you should get the same result).  This is due to another interesting fact about HTTP 1.1, which is that it supports persistent connections ("keep-alives") by default.  To close the connection, just close your command window.

Congratulations, that completes your first, brief conversation with an HTTP server.  You're on your way to HTTP fluency.

In future posts, we'll talk about how to interpret those results from the server, and about how to vary your requests, so that you can ask the server for different things in different ways (like compressed content, or the answer to the question of whether a particular file has changed since you last requested it).

posted on Monday, February 07, 2005 4:19 PM