Port80 Software has been busy developing new products recently that address various security and access issues for IIS. One thing we have noticed is the pro versus cons of the various approaches we have been implementing. For example, a black list approach of adding in signatures for things you don't want to allow works well in practice, but it is hard to keep up-to-date. In short, a black list of access URLs and techniques is only as good as the latest black list. It's a great business model (think virus definitions), but lousy for real security.
You could reverse the problem and go to a white list (defined, allowed actions are all that can access a site, and everything else should be denied). The challenge there is making sure your white list is 100% inclusive, which may not be easy to do in a Web application that is quite complicated or dynamic – and if your whitelist is not complete, you may accidentally block legitimate traffic. However, compared to the black list approach, white listing is much tighter, security-wise.
Now, what you'd really want is some form of black list “ease” with white list “power” and thus the idea of anomaly detection-style monitoring. Basically, look at Web traffic and things that are out of the ordinary get blocked. Given big-enough traffic, this is pretty solid, but even with the law of big numbers, you can screw-up with and get false positives. For example, today we came across an interesting post about the "Top 5 Google Tricks" (http://www.techeblog.com/index.php/tech-gadget/feature-top-5-google-tricks). We took a look at a tip to find open Webcams and all looked really good -- that is, if you like looking at style shots of various rooms and buildings around the world. In a vain attempt to see some movement we went deeper into the result lists and got the following message we'd never seen from Google:
Now, we assumed that something funny was going on here, because Google was likely getting lots of people digging in later queries like us looking for movement (or maybe in the other people's case adult “entertainment”). However, given the unusualness of these queries, Google.com started blocking them, assuming that we were a bot or some problem, as the message above indicates. We guess the scenario of lots of traffic + unlikely place to look = danger in Google's eyes. If you do the same query in the early parts of the results, you are fine, but if you jump ahead, you get a 403 forbidden type message like this one above.
This issue is likely fixed by now by the Google staff, but it is still an interesting example of the downsides of rejecting the unexpected, in this case the flashmob crowds from the blogs weren't a security problem, rather but legitimate user mass that triggered a believed security problem from Google’s point of view. From this, Google might have lost some cool points, but probably not enough to impact those guys too much!
Cheers,
Port80