Background
It's not simple to know what happens in a bigger network. There's a multitude of applications, services and appliances working together. Many of them provide some kind of events or state information. The network administrator needs to get hands on all of them. But they come in many different flavors and multiple canals. Therefore, it's hard to get the big picture. Furthermore, we have learned that it's impossible to protect a system against all malicious attacks and to keep all the possible faulty handling away. A monitoring of the systems to guarantee a pro-active handling is therefore needed..
Therefore, more and more organizations collect and analyze all logfiles in a centralized system, called a SIEM (security information and event management). The technology provides two major functions for security events from networks, systems and applications: log management and compliance reporting (SIM - security information management) and real-time monitoring and incident management (SEM - security event management).
About Apache ALOIS
Apache ALOIS is a log collection and correlation software with reporting and alarming functionalities. ALOIS stands for "Advanced Log Data Insight System" and is meant to be a fully implemented open source SIEM security information and event management system.
While almost all other SIEM software, be it closed or open source, concentrate on the technological part of security monitoring, Apache ALOIS is aimed to monitor the security of the content. It intends to be pro-active in the detection of potential loss, theft, mistaken modification or unauthorized access. Apache ALOIS works on log messages and thus contains all the basic functionality of a conventional SIEM, as centralized collecting, normalizing, aggregation, analyzing and correlating of all log messages, as well as reporting all security related events. Therefore it can be used as any other SIEM.
Why another security information and event management system? It's true, there's already plenty of them. While the proprietary software is way too expensive for smaller to mid-sized companies, we find that the open source solutions are either too simple or not completely open. For example, behind each of the well known systems "OSSIM" and "Prelude", there is a company that either closes central functionality for its own business or has dual licensing and therefore asks the full copyright for all contributed code.
Apache ALOIS is aimed to be totally free and open for all contributions. The openness provided for other programming languages is certainly proof of this. The plug-ability - yet to be further developed - is meant to guarantee that individual needs can be realized without stressing the whole system too much. In our opinion, the Linux kernel is a good example that this can work very well.
Security Information and Event Management (SIEM)
Security Information and Event Management (SIEM) solutions are a combination of the formerly disparate product categories of SIM (security information management) and SEM (security event management). SIEM technology provides real-time analysis of security alerts generated by network hardware and applications. SIEM solutions come as software, appliances or managed services, and are also used to log security data and generate reports for compliance purposes.
The acronyms SEM, SIM and SIEM have been used interchangeably, though there are differences in meaning and product capabilities. The segment of security management that deals with real-time monitoring, correlation of events, notifications and console views is commonly known as Security Event Management (SEM). The second area provides long-term storage, analysis and reporting of log data and is known as Security Information Management (SIM).
The term Security Information Event Management (SIEM), coined by Mark Nicolett and Amrit Williams of Gartner in 2005, describes the product capabilities of gathering, analyzing and presenting information from network and security devices; identity and access management applications; vulnerability management and policy compliance tools; operating system, database and application logs; and external threat data. A key focus is to monitor and help manage user and service privileges, directory services and other system configuration changes; as well as providing log auditing and review and incident response.
[This is an excerpt from Wikipedia. Read full text here: http://en.wikipedia.org/wiki/SIEM.]
Architecture of Apache ALOIS
Apache ALOIS consists of five modules interacting to ensure a scaleable functionality of a SIEM:
- Insink is the message sink, which is the receiving entry point for all the different log messages into Apache ALOIS. It is partly based on the syslog-ng software. Insink listens for messages (UDP), waits for messages (TCP), receives message collections (files, emails) and pre-filters them to prevent from message flow overload.
- Pumpy is the incoming FIFO buffer, implemented as a relational database tables. which contain the incoming original messages (in raw format). In a complex system setup, there may be several insink instances, e.g. for a group of hosts, for specific types of messages, or for high-avaliablity.
- Prisma contains logic to split up the text of log messages into separate fields, based on regular expressions. Actually, "prisma" is a set of "prismi", each one prisma for one type of log message (apache, cisco etc. Several prismi can be applied to the same message. This allows for stacked messages, i.e. forwarded log messages contained in compressed files contained in e-mail messages. The data retrieved form the log messages is stored in a database called Dobby. Due to prisma being written in Ruby, prismi can be applied interactively (when having system access).
- Dobby is the central log database. It should be separated from the Pumpy database for availability and performance reasons. The current implementation is based on MySQL.
- The Analyzer contains the two sub-systems Lizard and Reptor. Lizard is the analysis engine and user interface of Apache ALOIS, implemented in Ruby on Rails using AJAX. It allows for interactive browsing through the collected data, exclusion/inclusion/selection of data, data sorting, data filtering, creation of views, ad-hoc textual and graphical reporting. Reptor allows for automatic activation of views and comparison of these views' results to a predefined result (pattern matching). In case of mismatch, Reptor sends the result to predefined e-mail addresses.
Since an image expalins more than a thousand words, here is an overview of the data flow through the different modules:
Its modular design guarantees Apache ALOIS to scale from little to large organizations. Since there exists a Debian package, it's easy to build a test system or even a productive system for small environments.
Apache ALOIS and data retention
Apache ALOIS is a great tool to collect and analyze logs. As a scalable open source tool it may be used in a personal environment to create a log of personal activities in much the same way as a state data retention would collect logs. It could therefore be used for personal awareness regarding what could and would be retained. However, not all of what state data retention would collect, is currently available to Apache ALOIS. Most prominently, the mobile phone roaming data is only available on the mobile phones themselves, and although these devices are in the user's hand, getting that data to be transferred to Apache ALOIS is as of yet not possible. There is some work to be done on the mobile phone side to allow that.
On the other hand, Apache ALOIS is a dual-use tool and could be used in state data retention environments. As it was not intended to be used this way, as it contains functionality way beyond the collection for retention purposes and as it probably does not (yet) scale well enough for dumping massive data amounts, its use as a state data retention tool is unprobable and no such cases are known. However, it is open source software and may be put to any use.
Outlook
Although the software has been in productive use for a few years, there is still a lot of desired functionality missing. The plugability of new connected systems is given, but needs some revision. It is a given goal of the project to allow modules in other programming language. Furthermore, it has been discussed if parts of the existing implementation may be replaced with other proven open source software, e.g. the correlation engine or the web frontend. The other way round, it has been discussed that the filter creation engine would make a good tool for any kind of structured data, and thus could be separated from ALOIS and standardized as a stand-alone tool.