Log Management: The Benefits of Cloud Logging Tools

Posted: May 22nd, 2013
Filed under: IIS & HTTP


Logs are good for more than just taking up space on your hard drive. Logs are useful records of an event that took place at a particular time and in a particular manner. Since it’s obviously impossible to see everything is happening in your systems all the time, and certain events (like security incidents or application errors) may require forensic or debug data to assess, logging can be a critical piece of a IT infrastructure. Chances are if you are a system administrator, you’ve come across one or two logs in your time.

Logs are typically generated for things like:

  • Security events
  • Uptime/downtime
  • Errors (5xx, etc.)
  • Traffic/usage

Why Logging is Useful

For some, logs may seem like a nuisance. For others, their purpose is a mystery.

Many organizations use logs to help with things like troubleshooting, monitoring and alerting, analytics, and application debugging. At Port80, we do a considerable amount of logging with our web application firewall, ServerDefender VP. Here we use logs to detail security events such as XSS, SQL injection, input validation, and buffer overflow attacks. They capture detailed information about who, where, and how the event occurred. For us, logs are a way to view events and perform an assessment.

We look to see if legitimate users are being inadvertently blocked by security controls by analyzing the logs of the user moving through the site. If we see lots of bad behavior (attempting to access certain pages, XSS exploits, etc), we’ll know they are likely a malicious user and we might block their IP. If we see some harmless behavior that set off our web application firewall, we might want to adjust our controls for a particular field or page. But without this log data, we would have no way of knowing what type of actions to take because we would have no data to base our decisions on.

For another take on logging, information security video-blogger Javvad Malik produced a quick overview of log management that’s both educational and entertaining: 



 

Typically, logs are stored on a machine’s local hard drive or sent to a database. Until recent years, when the movement of business applications took to the cloud, this data was stored in a central location. But the problem with logging to a single machine or  database  is that you’re putting a lot of importance on the continued viability of your application.

Dangers of Local Logging

While logging data is useful and necessary to certain business operations, there are dangers to storing the data on a local machine. This is especially true when it comes to security logs.

Despite the efforts we all make to secure our sites and web applications, there are times that the most obscure of vulnerabilities get exploited. Typically, these are unknown vulnerabilities that get exploited, and the level of damage caused can range from site defacement to dropping a database. Logs are a valuable forensic tool to determine how a hacker got in and begin to assess prevention tactics. Without logs, all one can really do is guess what the vulnerability might have been, or hire an outside consultant to pen-test the site or app. Really, logs provide the most vital evidence of how someone got in and what they did once inside.

But once a hacker is in, if they have free reign, and the logs are stored locally, can’t they just destroy the evidence? Delete the logs?

Yes, they can. Poof. Gone. No evidence of their entry; only the damage that they’ve done if left to show.

Benefits of Cloud Logging Tools

Cloud-based systems are permeating all aspects of business, so it’s really no surprise that logging has followed suit. Logging has many added benefits associated with moving files, ones that are traditionally stored locally, and moving them to the more readily-accessible cloud.

The primary added benefit of sending logs – in our case security logs – to the cloud is one of insurance. If a hacker completely pwned your server and/or database, they would be able to do whatever they pleased with whatever data they could access.

Let’s say that once inside, this hacker could access your log files. Why wouldn’t they want to delete them to eliminate the evidence of they intrusion? Why would they want you to see how he got in so you could patch up the vulnerability and prevent him from getting back in? They probably wouldn’t. They probably would delete the log files, rendering you helpless: with no forensic evidence and no way to determine the attack vector. Think of it almost as a backup; if you have a third party logging application, you would be able to preserve those logs and have a record of what occurred. We highly recommend sending logs to a cloud-based tool as a security best practice alone.

However, added insurance isn’t the only benefit of third party logging. Cloud logging also can aggregate multiple log feeds into one central location, making it easier for you to monitor different apps, servers, sites, and systems from a single dashboard. Not only are the logs centralized, they may offer additional usability such as string search, tagging, and correlating events that may not be found in the original app’s log viewing tool.

Data visualization through charts and graphs also makes viewing larger data sets more digestible, and allows admins to see trends over time that they might not have otherwise seen. Many cloud tools offer the ability to customize dashboards to the most important data points across any of your systems for big picture view at a quick glance.

All of these features can make for more organized  and streamlined log management, plus many will  find the ability to log into a web interface more convenient than remoting into a machine, which, if compromised, may not even be helpful as the data may be gone.

Notable Cloud Logging Tools

Papertrail

https://papertrailapp.com/

Papertrail boasts that it makes life easier for sysadmins through instant log visibility and the ability to more easily make sense of the logs you’re collecting. The tool is built from Papertrail’s own experience as sysadmins and developers to help “detect, resolve, and avoid infrastructure problems” using log messages. Their laundry list of selling points include time-saving log tools, flexible system groups, team-wide access, long-term archives, charts and analytics exports, monitoring webhooks, and 45-second setup.

Cool features

  • Archived logs on Amazon S3
  • Provides ability to troubleshoot quickly
  • See related events together
  • Share log access with your team
  • Integrate with
  • Custom email alerting
  • Nightly email notifications
  • Supports technologies: Windows, Unix, Linux,  BSD, OSX, Node.js, Python, PHP, and more.

Logentries

The focus for Logentries is simple and comprehensive log management to let you visualize, analyze, search, and understand your logs – all in real time. After you sign up, Logentries has a useful walkthrough process that shows you what everything is and how it works – this definitely earns them some bonus usability points. The dashboard is intuitive and allows users to drill into hosts, events, and view data usage. Logentries prides itself on being very visual, with interactive dashboards for its three primary functions: debugging, monitoring, and analytics.

Cool features

  • Interactive graph to select and drill into time frame
  • Display your most important tags with simple tag filtering
  • Alerts email, mobile notifications, web alerts
  • Simple string search and regular expression search functionality
  • Send logs from Ruby on Rails applications with direct integration
  • Custom event tagging
  • Supported technologies: Windows, .NET, Linux, Node.js, Android, Rails, and more

Try Logentries

Splunk>Storm

Splunk sells itself as an intelligence tool. It performs the standard task of collecting data, but offers additional functionality like search and troubleshooting. It allows you to filter and correlate events, and even connect user transactions across different data sources. It is known for being one of the faster tools to set up and use, with customizable dashboards to visualize data how you want to visualize it. Splunk>Storm also has some great usability points, such as the ability to highlight or clicking on a string in logs to search for similar entries – it’s the little things.

Cool features

  • Visualize data and set up custom dashboards for analytics and monitoring
  • Filtering to correlate events across logs
  • Statistical commands to find trends in data
  • Systems Monitoring and application analytics
  • Free version with access to all features and up to 1GB storage
  • Supported technologies: IIS, Apache, REST API, syslog, Heroku,  and more.

Try Splunk>Storm

Loggly

Loggly says it’s not “your father’s IT company,” and a quick view of their promo video shows their exuberant personality. They focus on developing a product that thrives on big data and makes a difference while doing so. This centralized, cloud-based tool puts all your app, server, and infrastructure logs into their simple UI. Logs can be then viewed for reporting and analytics, while scaling as your logging needs increase.

Cool features

  • Custom alerts through Alert Birds integrated alerting application
  • Custom dashboard setup with quick access to your important saved searches
  • APIs with rich set of calls which allow developers to build custom solutions for their existing offerings and infrastructures.
  • Systems Monitoring and application analytics
  • Powerful and simple shell commands to dive into your data (i.e. graph visual comparisons using comma separated search queries)
  • Add logging library to your app (Node JS, Ruby, Python)
  • Supported technologies: Linux, Unix, Windows, Heroku, OSX, and more

Try Loggly

Cloud-based logging tools are highly useful tools for managing and making sense of log data. When it comes to security, they also valuable insurance   against hackers erasing forensic data that is essential for understanding how an intruder penetrated the system. Many of them offer free trials or even free versions. If you collect log data, we recommend trying out a few and seeing which works best for your needs. If you’re a ServerDefender VP user and want to set up cloud-logging, we’re happy to help with your assessment.

No Comments »

Leave a Reply