You Don’t Mess With The System Logs

wooden logs

The wooden logs to the left have nothing to do with system logs, other than being a metaphoric reference to the subject matter of this post; ensuring that your system logs are not messed with. In other words, protecting the integrity and confidentiality of system and event log messages, in a distributed logging architecture, and ensuring their central persistence for auditing, debugging, compliance monitoring and myriad other purposes.

The context of this post is a generic problem I had recently been grappling with, concerning multiple users having system administration rights on mission critical servers, with the mission in question involving the management of data of significant value. In my mind all relevant system actions, in particular those carried out by admin users, should be logged for auditing purposes. In fact, on second thought, keeping logs of all pertinent system messages and actions  on mission critical servers seems prudent, as a rule, regardless of who might or might not have administrative rights on a server.

My interest in syslog, and the underlying protocol, as defined in RFC 5424, stems from a Computer Science project I co-supervised involving the large scale monitoring of hundreds to thousands of routers, in what amounts to a distributed system logging architecture, with a central logging server. In this project, we used the open source edition of the syslog-ng logging system, created by BalaBit, with UDP as the transport layer protocol. In the cited project, the transmitted information, such as noise levels and system uptime, where not deemed to be sufficiently sensitive to warrant using Transport Layer Security (TLS) encyption nor mutual host and server authentication using X.509 certificates. When it comes the nature of the systems I am currently dealing with, a different approach is warranted and the technical white paper by BalaBit, entitled “Regulatory compliance and system logging”, appears to answer the questions I have had, and also raised issues I was not even thinking about.

One of the first thoughts I had when reading the white paper is that the cited regulations may be worth considering in terms of compliance regardless of whether one may be required to do so or not. For example U.S. or E.U. regulations do not apply in South Africa, but considering key regulations in those global power houses may be prudent in a general sense, as a best practice, and in my mind the primary motivation should not necessarily be the desire to do business in say the U.S., rather because the regulations may just be prudent to adhere to because they contain plain old good advice gained from hard lessons in the field.

BalaBit, in the cited white paper, lists these regulations and standards for possible compliance:

  1. Sarbanes-Oxley Act (SOX) – US legislation in force since 2002 concerning the regulation of financial practice and corporate governance.
  2. Basel II Accord – international banking standard for regulators to guard against financial and operational risks faced by banks.
  3. Health Insurance Portability and Accountability Act (HIPAA) – US standards for electronic health care transactions and national identifiers for health care providers / institutions.
  4. Payment Card Industry Data Security Standard (PCI-DSS) – Requirements for payment account data security by PCI Security Standards Council (lead by Visa, American Express, MasterCard et. al.).

It is worth noting that since starting to write this post, which I’ve been busy with for a while, I have encountered two of the above in separate business contexts, heightening my interest in syslog-ng and logging best practices in general. The first encounter involves SOX Section 404, with the context being a statement of compliance requested as a part of a mobile telecommunications operator RFP (request for proposal). The second encounter involves  the fourth standard in the above mentioned list, PCI-DSS, with the context being compulsory compliance due later this year relating to an e-commerce application that I have maintained for over three years.

So how and where to get started with protecting the integrity and confidentiality of system log messages? That is how does one best design and implement a system logging architecture? Furthermore, as a case-study, what should be done in terms of logging to ensure compliance with say PCI-DSS?

There is no short answer to the above. With regards to PCI-DSS and sections relating to log management and auditing, in terms of implementation in the first instance one would have the know the e-commerce application and host configuration very well. The cited Balabit white paper seems to offer good guidance when it comes to PCI-DSS, and so I won’t repeat their content here.

So how much does the commercial / premium edition of say syslog-ng, or alternatives, cost? I contacted BalaBit regarding the former, in short it depends on how many hosts you want to monitor and the pricing is between a couple of hundred US dollars and tens of thousands, it is best you contact them directly in this regard. They naturally also offer commercial support. A possible alternative to syslog-ng is Rsyslog, this slightly dated comparison page may be handy.