Its always great when companies can invest in tools that serve multiple groups and purposes. Ensuring proper setup and use of logs is one toolset that can serve both DevOps and Security.
Why logs are important to DevOps
DevOps use logs primarily to understand what’s going on inside an application server, database system or the operating system. For the most part, logs are a resource that helps DevOps diagnose problems and ensure proper working order of systems. Sometimes logs provide a tracking mechanism that helps DevOps understand what customers are doing in their application. DevOps folks are intimately familiar with newer and popular logging aggregation services like Splunk, SumoLogic and Loggly.
Security Loves Logs Too
The good news is that security teams love logs too. The use, review and management of logs is so important to the security function of an organization, its even part of the SANS top 20 critical security controls. From a security standpoint, logs provide a ton of important data for everything from forensic analysis to audit records.
Recommendations
Since logs are so important to both DevOps and Security teams, it pays to implement your logging tools in a way that will serve the needs of both teams. Here a few quick tips that will help you set up logging tools in a way that will create a better world for both DevOps and Security.
Centralized Logging & Log Off Site
Setup all your systems to log to the same centralized location. If you have all your logs in one place it’s much easier to setup log reviews and alerts. Its also a good idea to log to multiple locations for security reasons. Consider logging to a centralized location within your network and also to a remote location. For example, have all systems log to a central server and also log to a remote service such as SumoLogic, Loggly or Splunk. That way you have multiple copies of your logs available for forensic purposes.
Time Sync
Make sure all your systems are in tight time synchronization. There is nothing worse than reading your centralized logs to find that a few systems are not properly time synchronized. NTP is your friend for setting up time sync. Be sure to use two or more reliable upstream time servers and configure all your servers to peer with each other.
Applications and Operating System Logs
Start with the most common base configuration so that all systems use syslog to capture and distribute logs. In the beginning, the logs in syslog will probably only be from the operating system and few processes that are already configured to use syslog. However, there are a ton of other processes running that should also be logging to your centralized system. The DevOps group will want to ensure that all applications, even custom developed code, logs important information. Make sure that your custom apps log enough useful information that problems can be diagnosed when they occur.
Setup Alerts & Reports
Nobody wants to be trolling through tons and tons of logs. To prevent this, one of tne of the first things you should do is to set up a basic set of alerts and reports. Security teams will want to make sure that logs record and alert when users gain administrative access to a system or an application, but there are lots of other events that should need logs and alerts. Spend some time thinking about events that could mean something is not working correctly and needs attention. Ask the development team to specifically log non-critical events that should be reviewed and incorporate these into a daily report.
Be Verbose, But With Care
Crank up the logging levels on all systems but don’t go too far. You want to be careful you aren’t logging too much. For example, you probably don’t need to run all your systems in debug mode all the time. You also need to make sure you aren’t logging sensitive information like user passwords or authentication tokens. With logs, there is such a thing as too much.
If logs are set up correctly, they will provide nearly endless benefits to both the DevOps and Security teams. Now, go forth and log!