Customizing AlienVault's OSSIM Plugins

So, I have been playing a lot with the Open Source SIEM product known as OSSIM, from AlienVault.  I have been using it from version 2, mainly with version 3, and now up to the newest version 4.  In that time I have gone through a lot of frustrations, but I have come to start really enjoy the product.  I should state that I just finished my training in San Francisco and will shortly be certified as an AlienVault Certified Security Analyst and Engineer, so this post may be slightly biased as I just learned fully how the product works and interacts and I am still really excited about it all.  To this date I would say one of OSSIM’s greatest strengths is it customization and features.  I really enjoy the fact that at its heart it is a lot of open source tools pieced together interacting in harmony to give you essential visibility in your network.  I will say its weakness, if you consider it that, is the difficulty in the configuration (something I know they are focused on resolving as 4.0 is a big step in the right direction).  However, I feel the difficulty in configuration is somewhat of a strength for implementation.  It forces to the administrator to become intimately knowledgeable with the SIEM and customize it for their environment; something I feel any security professional would admit is essential for a SIEM.  The downside to this is that it takes a lot of time and man hours to implement a system like this on a very large network, which is why I originally state it’s a weakness but it’s all perspective.

Architecture

A bit of a background on how OSSIM is going to work, it essentially consists of four main components:  sensor, database, framework and server.  There is also a logger portion that is provided in the professional version, it’s optimized for long term storage on an indexed database; however, I will not be talking about it (but it’s very cool I suggest you check it out).  The sensors are going to be placed closest to devices you want to monitor; these are going to handle the inputs sent from devices, mostly logs, as well as provide you with tools to use like snort and OpenVAS.  The database is going to be the central storage of these events, which allows for a lot of the cool features you’ll see from the server.  The server is essentially the engine behind a lot of the action going on.  It handles the vulnerability assessments and correlation of the events in the database.  The framework is to tie everything together in your mostly easy to use web-interface. 

So for an example you have a LAN segment you need to monitor.  You would send your firewall/switch logs to OSSIM via syslog most likely and mirror or tap traffic to send to a second network interface to be assessed by snort.  This is handled by the sensor; you would specify the logs to go into a separate log file and have plugin to monitor that log file continuously.  Snort is assessing the interface you selected to be in promiscuous mode and analyzes traffic as, I am assuming, you’re familiar with.  Now the plugin catches events either pre-defined in provided plugins by OSSIM or however you specify in a custom plugin as I will show later.  These events and events generated by snort are then entered into the database.  The server is connected to the database and then will start correlating these events, again on either predefined specifications defined by AlienVault or by you.  So to put it programmatically: 

If Event(A) = Outcome(A)      

Then Search For Event(B) Until Outcome(B) Or Timeout

            If Outcome(A) + Outcome(B) = Risk(High) Then GenerateAlert(e-mail|ticket) Else ExitLoop

Now correlation can go on for several events matching a chain of I think up to 10 events.  However, that complexity is kind of overkill and would be looking for events on par with something like APT most likely.  That is the main point of the server.  The framework is going to tie all these pieces and provide you with Web-Interface to utilize and search all this information.

Now the great thing about this architecture is distribution.  You can place sensors all over your environment and have those reporting to the other pieces as needed.  By default OSSIM will come in an all-in-one fashion which is a lot easier to set up and work with; but will only work for really small environments.  But, the ability for distribution really allows you implement on the larger scale.  One limitation of OSSIM is it cannot handle a lot of events per second, that is one of the perks of their professional version; however if you distribute it over a lot of machines you may be able to work around that limitation (do not recommend it though).

Customizing Plugins

The greatest thing I love about OSSIM is, once you get how it works, it’s pretty easy to customize how it works and customize it to your needs.  I will show the simplicity by providing my own experience and example.  I have been working with Cisco ASA’s for a while now and have been centralizing logs for some time.  I wanted a better way to interpret this data in an easier fashion then pouring through 1000s of logs a day, as I am sure many of you are familiar with.  This is when I found OSSIM, which has its own ASA plugin already available; but it didn’t provide a format I liked personally.  The message they provided for type of event was difficult to differentiate on the fly and did not anticipate for non-IPs in the logs, as in the ASA you can give IPs a hostname to easily sort through all the IPs you’ll see often with connection logs.  However, as I would soon find out, the destination and source IP fields can only populate an IPv4 address at this time; something I greatly hope they can alter and kind of expect to see with IPv6 support coming from what I’ve read on their forums.  So, if you want to see an easier to read name for an IP you have to disable the naming feature in the ASA, and then define that IP in your SIEM in the assets portion of the system.  So, lets dive into building a custom plugin for an ASA, but the methodology will translate to any log format you can populate. 

First you’ll need to configure your source to send logs to an OSSIM server, where they have syslog already listening for events by default.  However when we do that it’s just going to append that message to  /var/log/syslog which can create a lot of messages, so it would be ideal to separate logs from syslog and place them in their own file.  We can do this by creating a configuration file in /etc/rsyslog.d  so we can do this:

• vi /etc/rsyslog.d/asa.con
• Insert Line 1:  if $fromhost-ip == 192.168.9.101 then -/var/log/cisco-asa.log
• Insert Line 2:  if $fromhost-ip == 192.168.9.101 then ~

Now, what this does is create a configuration file for your ASA, which rsyslog is configured in OSSIM to utilize any *.conf file you have created in that folder.  The first line says if the log comes from 192.168.9.101 (our ASA IP) then place that log in cisco-asa.log.  Line two matches the IP as well, but then states place that IP in only cisco-asa.log and not syslog as well.  Now you’re almost set here and have logs coming in, but first you’ll want to configure log rotation at /etc/logrotate.d/  Now for this plugin since it’s a standard syslog, I simply added the log file to the rsyslog rotation already enabled.  This is setup for rotating the log on a daily basis so your storage is maximized.

Now that you a defined log source, you can now build your plugin around that information.  Plugins are all stored in a central location at:  /etc/ossim/agent/plugins/  Here I will create a file called custom-asa.cfg   Take a look at the beginning portion of this configuration file:

;; custom-asa

;; plugin_id: 9010

;;

;; $Id: *-asa.cfg,v 0.2 2012/7/29 Mike Ahrendt

;;

;; The order rules are defined matters.

;;

[DEFAULT]

plugin_id=9010

[config]

type=detector

enable=yes

source=log

location=/var/log/cisco-asa.log

create_file=yes

process=

start=no

stop=no

startup=

shutdown=

This is a pretty straight forward implementation, you have a plugin ID of 9010 (all plugins 9000+ are saved for custom plugins).  Then you simply have that this is a detector plugin that is enabled.  Next we define that our source file is a log that is in the location we created earlier, and if necessary can create the file.  The next portion is not really relevant to our purpose, but is utilized by plugins for tools that are incorporated in OSSIM.  For instance the OSSEC plugin has processes tied to it, so when the plugin is ran it can start the OSSEC process via the plugin automatically.  Again, this is not needed for us.  Also note that ;; is a form of commenting your plugins and not necessary just kind of helpful.

Now, for the really fun part of the plugin creation process, REGULAR EXPRESSION!!!!  Admittedly, I knew nothing about regular expression and after learning to build these plugins I have learned what I am really missing.  If you’re not already familiar with regular expression you should keep a cheat sheet available (I recommend this one: http://www.cheatography.com/davechild/cheat-sheets/regular-expressions/).  The next part of the plugin is reliant upon building regular expression around common format log files.  So looking at this log we would see something like this:

            Jul  2 10:23:00 192.168.9.101 %ASA-0-106100: access-list 129-network_access_in denied tcp 129-network/198.110.72.148(61388) -> Secure-135/ DB-136(1521) hit-cnt 1 first hit [0xbbdcff06, 0x0]

Now we need to build a regular expression to match this input for all logs.  Starting with the date, it is pretty easy.  We have Jul 2 10:23:00 so our regular expression looks like this \D{3,4} (not a number for 3 or 4 characters) \s (space) \d\d (two digits) : \d\d : \d\d put together we have, with the defined variable the AlienVault plugin:

            (?P<date>\D{3,4}\s\d\d:\d\d:\d\d)

Now for the IP Sensor IP of 192.168.9.101, we build \d{1,3} (digit of 1 to 3 digits in length) . \d{1,3} . \d{1,3} . \d{1,3}   Adding to the plugin now:

            (?P<date>\D{3,4}\s\d\d:\d\d:\d\d)\s(?P<sensor>\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})

Adding the ASA Syslog ID we input \s (space) exact %ASA- \d (one digit) - \d+ (digits until there is no longer a number) . (any character) \s  now our expression becomes:

            (?P<date>\D{3,4}\s\d\d:\d\d:\d\d)\s(?P<sensor>\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\s%ASA-\d-\d+.\s

Now for the non-standard stuff with variables for OSSIM, I matched access-list exactly then using \S+ (all non-whitespace until a space) for a variable. Now for these logs you will next see permitted or denied.  So I assign the next portion to a variable and we’ll see why that’s important in a bit.  Next you’ll see the defined protocol in a variable.  Then the network on which the transmission occurred, which I define as the interface in OSSIM.  Next I define the source IP and Port then destination IP and port.  All coming together as: 

            "(?P<date>\D{3,4}\s\d{1,2}\s\d\d:\d\d:\d\d)\s

             (P<sensor>\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\s%\D\D\D-\d+-\d+.\saccess-list\s

             (?P<msg>\S+)\s(?P<sid>\w+)\s(?P<proto>\w+)\s(?P<iface>\S+)\/(?P<src>\S+)\

             ((?P<sport>\d+)\)\s..\s\S+\/(?P<dst>\S+)\((?P<dport>\d+)\)"

In the plugin we will assign this to a rule like so:

[AAAA - Access List]

regexp="(?P<date>\D{3,4}\s\d{1,2}\s\d\d:\d\d:\d\d)\s(?P<sensor>\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\s%\D\D\D-\d+-\d+.\saccess-list\s(?P<msg>\S+)\s(?P<sid>\w+)\s(?P<proto>\w+)\s(?P<iface>\S+)\/(?P<src>\S+)\((?P<sport>\d+)\)\s..\s\S+\/(?P<dst>\S+)\((?P<dport>\d+)\)"

event_type=event

date={normalize_date($date)}

sensor={resolv($sensor)}

plugin_sid={tranlate($sid)}

src_ip={$src}

src_port={$sport}

dst_ip={$dst}

dst_port={$dport}

interface={$iface}

userdata1={$msg}

Now you may notice some interesting things in this plugin.  First note that I start the rule with AAAA, however you’ll need to assign plugin SIDs in numbers, the engine parses these rules alphabetically and using numbers makes for a difficult experience in processing your rules in the order you desire.  Therefore, I suggest you use letters to assign your rule names for the engine.  We also see that there are functions before some variables.  These are built into OSSIM and are a part of /usr/share/ossim-agent/ossim_agent/ParserUtil.py  There are several functions but the major ones are in this plugin.  Normalize_date is self-explanatory I believe.  Resolv will simply parse the IP and normalize it.  Translate is great if you have similar log messages.  If you’ll recall this log varies in the fact it varies at permitted or denied.  So with translate we take the $sid variable and it will read that variable and then translate it to something else via the translation portion of the plugin which looks like this:

[translation]

permitted =600

denied = 601

inbound = 603

outbound = 604

So you’ll see that the plugin will translate the word to either 600 or 601 which we will then use for the rule identifier.    Now this is just one rule of the plugin I wrote.  I have built a plugin to match 19 of the most common logs I have found so far.  Looking through ASA documentation, the ASA-\d-\d\d\d\d\d\d format is unique to each log type.  Specifically the first three digits in the 6 digit number are semi-generic to various types of events.  Example, if we see %ASA-6-212\d\d\d we will know that the plugin has something to do with SNMP.  So it would be possible to build events around those specifications, which is what I did for some rules but they were events that are non-standard for my environment.  So, I am seeing the events I just don’t get the granularity in the OSSIM interface as I do for the events that I really want to see.

Now that there is a plugin configuration created with all our rules, we almost have a working plugin.  Now we need to build a SQL file so that the database knows what the plugins are and will generate the event up a match.  Fairly simple, lets take a look at the SQL:

-- GRCC ASA Plugin

-- plugin_id: 9010

DELETE FROM plugin WHERE id = "9010";

DELETE FROM plugin_sid WHERE plugin_id = "9010";

INSERT IGNORE INTO plugin (id, type, name, description) VALUES (9010, 1, 'custom-asa', 'Custom ASA plugin');

INSERT IGNORE INTO plugin_sid (plugin_id, sid, category_id, class_id, reliability, priority, name) VALUES (9010, 600, NULL, NULL, 2, 2, 'Access List Permitted');

INSERT IGNORE INTO plugin_sid (plugin_id, sid, category_id, class_id, reliability, priority, name) VALUES (9010, 601, NULL, NULL, 2, 4, 'Access List Denied');

The first two lines are simple comments, and not required for the SQL file.  The next two lines are to delete possible redundancies from the database.  Next we insert the plugin ID to the database.  Now come all of our rules, we simply give it the database fields of plugin_id (which we assigned 9010), then the rule ID you want (from the translation table you’ll remember I assign these rules to 600 and 601), then you insert a category_id and class_id (not necessary so I’ve assigned them as NULL).  Now reliability and priority are pretty important.  This is how OSSIM will assess the risk posed by these events.  Something like a permit, not necessarily something to freak out about so I assign its priority to 2.  A deny however is more important to me so I assigned it a 4, as I want those to stand out more.  The name is self-explanatory, just a description of what the rule is.  We then create a custom-asa.sql file at /usr/share/doc/ossim-mysql/contrib/plugins/   Now all that is left is entering the SQL into the DB with a command, which there are a few options to execute from the sql plugin directory mentioned prior:

• cat custom-asa.sql | ossim-db
•  ossim-db < custom-asa.sql

Either command should work; I favor the latter form personally.  I did notice in version 3 sometimes it would take a few times before I would see the plugin come up in the setup interface.  However, in 4 it worked first time around with no problem.  We can verify that the plugin injected into the database properly by logging into the web interface.  Going to Configuration > Collection > Data Sources where we can then search for ID 9010.  It will look like this: 

There we have it!  Our plugin is registered within the database and recognizing the right rules.  We also will see in the console setup, by logging into OSSIM’s terminal and entering “ossim-setup” into our command line, then selecting “Change Sensor Settings” followed by “enable/disable detector plugins” we should see an option for “custom-asa” as well:

Now you simply add the check mark to the plugin and then hit OK and then you'll reconfigure the server through "ossim-setup".  And voila! We have what should be a working plugin.  But, how can we verify this you ask?

We troubleshoot our OSSIM  plugin with some ease.  You will need a few things:  a few consoles open (I suggest using PuTTY), a sample of the logs you are trying to parse, and some time.  First, you’ll want to open up three terminals.  I like to make it look something similar to this:

This will give you two streams of information to follow when you inject the sample log into the monitored log.  The agent.log will display messages regarding the plugin parsing and server.log will notify about added database messages.  Now we need to start injecting the sample into the cisco-asa.log.  Because the plugin monitors in real-time, you can’t simply throw a log in and parse it from the beginning you have to inject it into the process.  This is easily done, I recommend taking a sample from your already collected logs and perform a copy and name it something simple (in the screen above you may see I named my copy INJECTABLE).  Now once we have a copy we simply perform this command:  cat /var/log/INJECTABLE >> /var/log/custom-asa.log   When this is complete you will see that the agent log will start parsing all of the events and then the server log will show all the events that were added to the database.

So now we’ve gone through and turned those horrible logs we’ve seen earlier into this (sanitized) view:

As you'll see we have the a list with the signature we've created, the time, the sensor is the IP of the actual ASA, your source IP and Port with country if an external IP, destination IP and ports as well, your asset values and your associated risk.  The 0.0.0.0 is what occurs because I use non-standard IPs in my ASA configuration.  It converts the host name to 0.0.0.0; however, if you click into the event it will show the whole log with the proper information.  This is the point I wish OSSIM would change, but it's only for my own personal preference.  As I said, you can turn it off in the ASA and then assign the IPs you gave host names to an asset tag that will give you an even easier to read event in the SIEM.

As you can see OSSIM is pretty powerful.  I highly recommend you give a look, as I think it will be improving a lot in a short period.  You can go to the community AlienVault site, where OSSIM is hosted at:  http://communities.alienvault.com/community/.  You can find their documentation on building a plugin here:  http://www.alienvault.com/docs/AlienVault%20Building%20Collector%20Plugins.pdf.   Also, if you check out OSSIM and want to try my plugins you can download my current implemented SQL and configuration files.  I think they're pretty good, I have to update it a little bit to cover all the ASA log types and I also want to build more defined rules (but this is a a start to get you going and customizing for your own needs).  Also, now that I have more defined rules I will be able to build correlation rules for them; but more on that later.