Category: Security

In a previous entry we discussed OSSEC Decoders and how they work. Decoders are only the first step in the log analysis train, though. Once log entries have been decoded, we need to do something with the results. That something is to match them up with rules so that OSSEC sends alerts and/or provides the proper response.

OSSEC rules ultimately determine what log entries will be reported and what entries will trigger active responses. And, as with decoders, rules can build upon one another. You can also chain rules together to alter the ultimate response based on conditions such as frequency and time. So let’s take a look at a simple rule.

<rule noalert=”1″ level=”0″ id=”5700″>
<decoded_as>sshd</decoded_as>
<description>SSHD messages grouped.</description>
</rule>

This is one of the default rules that ships with OSSEC. Each rule is defined with a unique ID. To prevent custom rules from interfering with core rules, the developers have reserved a range of IDs for custom rules. That said, there is nothing preventing you from using whatever ID numbering you desire. Just keep in mind that if you use an ID that is reserved for something else by the developers, new versions of OSSEC may interfere.

Also listed in the rule tag is a level and a noalert attribute. The noalert attribute is pretty straightforward, this rule won’t send an alert when it’s matched. Typically, the noalert tag is used on rules that are designed to be built upon. The level tag determines what level alert this rule will trigger when matched. Because we’re combining it with the noalert tag, the level ends up not meaning a whole lot.

The decoded_as tag identifies the parent decoder for which this rule is written. Only rules with a decoded_as tag matching the decoder used to decode the current log entry will be scanned for a match. This prevents OSSEC from having to scan every rule for a match.

The description tag is a human readable description of what the rule is. When you receive and alert, or when the alert is added to the OSSEC log, this description is added along with it. In the case of this rule, the description identifies its purpose. This rule was defined purely to group together sshd alerts. The intention is that other rules will handle the alerts for any alert-worthy ssh log entries that are detected.

Now let’s look at something that builds on this basic rule. Again, choosing a rule from the default ruleset, we have this:

<rule id=”5702″ level=”5″>
<if_sid>5700</if_sid>
<match>^reverse mapping</match>
<regex>failed – POSSIBLE BREAK</regex>
<description>Reverse lookup error (bad ISP or attack).</description>
</rule>

The first thing to note about this rule is the if_sid tag. The if_sid tag says if the rule number defined in this tag has already matched the incoming log entry, then we want to see if this rule matches. In other words, this rule is a child of the rule identified in the if_sid tag.

The match tag defines a string we’re looking for within the log entry. The regex tag also defines a string we’re trying to match, but regex can use the full set of regular expressions that OSSEC supports. If both the match and the regex are found in the log entry, then this rule matches and will alert at a level of 5.

Finally, let’s look at a more advanced rule. This rule also builds on the previous rules mentioned, but contains a few extras that make it even more powerful.

<rule timeframe=”360″ frequency=”4″ level=”10″ id=”5703″>
<if_matched_sid>5702</if_matched_sid>
<description>Possible breakin attempt </description>
<description>(high number of reverse lookup errors).</description>
</rule>

The intention of this rule is to identify repeated log entries that point at a more severe problem than a simple error. In this case we have multiple incoming ssh connections with bad reverse DNS entries. The frequency and timeframe attributes define how many times within a specific timespan a particular rule has to fire before this rule will kick in.

Notice that instead of the if_sid tag, we’re using the if_matched_sid tag. This is because we’re not necessarily making this rule a child of another, but instead making it a composite rule. In other words, this rule is fired if another rule fires multiple times based on the setting within this rule. As a result, this rule fires with a level of 10.

But now that we have rules and alerts being generated, what else can we do? The answer is that we can trigger active responses based on those rules. Generally, an active response fires when an alert comes in with a level equal to or higher than the alert level of the active response definition. There are ways to alter this, but let’s keep things simple for now.

To configure active response, first you need to define the commands that active-response will use. Note: All command and active-response configuration is placed in the ossec.conf on the server.

<command>
<name>firewall-drop</name>
<executable>firewall-drop.sh</executable>
<expect>srcip</expect>
<timeout_allowed>yes</timeout_allowed>
</command>

This snippet defines a command called firewall-drop. The command depends on an executable called firewall-drop.sh. This file must exist on all agents that will run this command. Unfortunately, there is currently no mechanism to push these files out to agents automatically, but perhaps there will be in the future? (*HINT*)

Th expect tag determines what variables are required from the alert in order to fire this command. Since this particular command adds a block rule to the server firewall, the srcip is required. And finally, timeout_allowed tells OSSEC that this command supports a timeout option. In other words, the firewall block is added and then removed after a given timeout.

Once the command is in place you need to set up the active response itself.

<active-response>
<command>firewall-drop</command>
<location>local</location>
<level>6</level>
<timeout>3600</timeout>
</active-response>

The active-response block identifies the command that will be run as well as the level at which it will fire. For this example, the firewall-drop command is run for any alert with a level of 6 or higher. We have also specified a timeout of 3600 which tells OSSEC that the command needs to be run again after 3600 seconds to remove the firewall-drop.

Also included is a location tag. This tells OSSEC where to run the command. Here we have specified local, which may be slightly confusing. This means that firewall-drop is run on the local agent that triggered the alert. So, if agent 002 triggers an ssh alert with a level of 6, then OSSEC tells agent 002 to run the firewall-drop command with a timeout of 3600 seconds.

You can also specify other options for location that allow you to run commands on the server or on specific agents. For instance, what if you want to block an IP at the edge whenever you get an alert of level 10 or higher. Perhaps you create a command called edge-block and it connects to your edge router to update an ACL located on the router. Running this on every agent is unwieldy at best and probably constitutes a significant security threat. Instead, you can have this script run on the server, or even a specific agent designed to handle this connection.

And that covers the basics for rules. I encourage you to write your own rules and test them out with the ossec-logtest program located in /var/ossec/bin. Learning to write rules is essential to running and tuning an OSSEC installation.

Tune in tomorrow for the final wrap-up of this year’s Week of OSSEC!

One of the strongest features of OSSEC is its log analysis engine. The log collector process monitors all of the specified logs and passes each log entry off to the analysis engine for decoding and rule matching. For agents, the log entries are transmitted to the server for processing.

There are multiple steps to identifying actionable matches starting with the decoding of each log entry. First, pre-decoding breaks apart a single log entry into manageable pieces. Decoders are then used to identify specific programs and log entry types. Decoders can build on one another allowing the operator to target specific patterns within a log entry with ease.

Let’s take a look at a simple example first. The following is a sample log entry from a typical /var/log/secure log. This log uses the syslog format for log entries.

Oct 21 00:01:00 dev sshd[31409]: Failed password for invalid user postfix from 189.126.97.181 port 57608 ssh2

Pre-decoding breaks apart the syslog format into three pieces, the hostname, program_name, and log. Using the ossec-logtest program provided with OSSEC, we can view the process OSSEC goes through for decoding and then rule matching. Pre-decoding this log entry produces the following :

[root@dev bin]# ./ossec-logtest
2010/10/21 00:01:00 ossec-testrule: INFO: Reading local decoder file.
2010/10/21 00:01:00 ossec-testrule: INFO: Started (pid: 1106).
ossec-testrule: Type one log per line.

Oct 21 00:01:00 dev sshd[31409]: Failed password for invalid user postfix from 189.126.97.181 port 57608 ssh2

**Phase 1: Completed pre-decoding.
full event: ‘Oct 21 00:01:00 dev sshd[31409]: Failed password for invalid user postfix from 189.126.97.181 port 57608 ssh2’
hostname: ‘dev’
program_name: ‘sshd’
log: ‘Failed password for invalid user postfix from 189.126.97.181 port 57608 ssh2’

As you can see, the log entry ends up with three parts. Further decoding uses pattern matching on these three parts to further identify and categorize log entries.

<decoder name=”sshd”>
<program_name>^sshd</program_name>
</decoder>

This is about as simple as a decoder can get. The decoder block starts with an attribute identifying the name of the decoder. In this case, that name is sshd. Within the decoder block is a program_name tag that contains a regular expression used to match against the program_name from pre-decoding. If the regex matches, OSSEC will use that decoder to match against any defined rules.

As I mentioned before, however, decoders can build on each other. The first decoder above reduces the number of subsequent decoders that need to be checked before decoding is complete. For example, look at the following decoder definition :

<decoder name=”ssh-invfailed”>
<parent>sshd</parent>
<prematch>^Failed \S+ for invalid user|^Failed \S+ for illegal user</prematch>
<regex offset=”after_prematch”>from (\S+) port \d+ \w+$</regex>
<order>srcip</order>
</decoder>

This decoder builds on the first as can be seen via the parent tag. Decoders work on a first match basis. In other words, the first decoder to match is used to find secondary decoders (children of the first), the first secondary decoder is used to find tertiary (children of the second), etc. If the matching decoder has no children, then that decoder is the final decoder and the decoded information is passed on to rules processing.

There are three other tags within this decoder block worth looking at. First, the prematch tag. Prematch is used as a quick way to determine if the rest of the decoder should be run. Prematches should be written so that the portion of the entry they match can be skipped by the rest of the decoder. For instance, in the decoder example above, the prematch will match the phrase “Failed password for invalid user” in the log entry. This portion of the log contains enough information to identify the type of log entry without requiring us to parse it again to extract information. The remaining part of the log entry has the information we want to capture.

Which brings us to the regex. The regex, or regular expression, is a string used to match and pull apart a log entry. The regex expression in the example is used to extract the source ip address from the log so we can use it in an active response later. The order tag identifies what the extracted information is.

Now, using these two decoders, let’s run ossec-logtest again :

2010/10/21 00:01:00 ossec-testrule: INFO: Reading local decoder file.
2010/10/21 00:01:00 ossec-testrule: INFO: Started (pid: 28358).
ossec-testrule: Type one log per line.

Oct 21 00:01:00 dev sshd[31409]: Failed password for invalid user postfix from 189.126.97.181 port 57608 ssh2

**Phase 1: Completed pre-decoding.
full event: ‘Oct 21 00:01:00 dev sshd[31409]: Failed password for invalid user postfix
from 189.126.97.181 port 57608 ssh2’
hostname: ‘dev’
program_name: ‘sshd’
log: ‘Failed password for invalid user postfix from 189.126.97.181 port 57608 ssh2’

**Phase 2: Completed decoding.
decoder: ‘sshd’
srcip: ‘189.126.97.181’

As you can see, the decoding phase has identified the decoder as sshd. The logtest program reports the name of the parent decoder used, rather than the ultimate matching decoder.

Hopefully this has given you enough information to start writing your own decoders. The decoder.xml file that comes with OSSEC is a great place to look at examples of how to craft decoders. This is a more advanced task, however, and the existing decoders cover most of the standard log entries you’ll see on a Linux or Windows machine.

For more information on decoders, please see the OSSEC manual. You might also check out chapter 4 of the OSSEC book. The book is a little outdated now, but the information within is still quite accurate. Syngress released a few chapters of the book that you can download here.

One of the simplest functions that OSSEC can perform is integrity monitoring. The premise is pretty simple, let the admin know if a file has changed. More often than not, the admin will already know that the file has changed because the admin is the one that changed it. But sometimes files change because of problems in the system that the admin doesn’t know about. Or, the files may change because the server has been compromised by an outside party that has installed rogue software. Either way, the admin needs this information.

OSSEC can be configured to look at a few different aspects of a file in order to determine if it has changed or not, depending on how you configure it. But before we get to that, let’s configure OSSEC to send us alerts to begin with.

There are a number of ways OSSEC can send alerts. Alerts can be sent via syslog, email, stored in a database, or sent to third-party programs such as Prelude and Picviz. To make things a bit simpler, I’m only detailing how to set up email. If you’re interested in other alert setups, please check the OSSEC manual.

Setting up email alerts is as simple as adding two sections of configuration to the ossec.conf file. This configuration is set on the server, or on a standalone installation. In a server/agent setup, the server sends all alerts, including those for agents.

<global>
<email_notification>yes</email_notification>
<email_to>myaddress@example.com</email_to>
<smtp_server>smtp.example.com</smtp_server>
<email_from>ossec@example.com</email_from>
</global>

This first bit of configuration defines the To: and From: addresses as well as the SMTP server address. This configuration goes into the global config section which has a number of other options as well.

<alerts>
<log_alert_level>1</log_alert_level>
<email_alert_level>6</email_alert_level>
</alerts>

This portion of the configuration defines what level alerts should be sent via email and what level alerts should be logged. Don’t worry too much about what a level is, you’ll learn this in a later blog entry when we discuss rules and active response. For now, the config as shown above is enough.

Now that we’ll receive alerts that OSSEC generates, let’s set something up to send an alert!

As I mentioned before, integrity monitoring is pretty straightforward. OSSEC uses a number of different characteristics to identify when a file changes. It’s pretty easy to determine that a file has changed if the owner, permissions, or size changes. OSSEC can also be configured to check the sha1 and/or md5 hash values as well. Hashing is a way of producing a “signature” for a file that is mostly unique. It is possible to create another file with the same hash, but it’s very difficult. Combining all of these checks together makes it very improbable that an intruder can replace a file without you knowing.

Enabling syscheck is done on a per-host basis. What I mean by this is that the config is added to the ossec.conf file for server or standalone systems, and the config is added to the agent.conf for agents. As with other agent configurations, you can specify syscheck blocks for all agents as well as cumulatively for specific agents.

<syscheck>
<frequency>7200</frequency>

<directories check_all=”yes”>/etc</directories>
<ignore>/etc/adjtime</ignore>
</syscheck>

The above config is an example of how to enable syscheck. The frequency block specifies the time, in seconds, between syscheck runs. The example above runs syscheck every 2 hours. Moving further down in the config is the directories tag. Simply put, this tag identifies what directories syscheck should be checking. The directories tag can define multiple directories, separated by a comma. The check_all attribute indicates that all of the previously mentioned checks should be run. The OSSEC manual details what other attributes are available. One other attribute worth mentioning is the realtime attribute. This attribute directs OSSEC to use inotify to alert when files changes, in real time, for quicker notification. Linux is the only OS, that I am aware of, that supports this option. Please check the manual for more information on realtime use.

There are times when you want to ignore files within a directory, or even certain subdirectories. The ignore tag allows you to accomplish this. Without defining any attributes, the ignore tag must define an exact match. For instance, in the example above, the file /etc/adjtime will be ignored. However, /etc/adjtime.new or /etc/adjtime0 will not be. To ignore these files, you will need to either add explicit ignore blocks for them, or you can use a regular expression to grab them all. The type attribute allows you to specify that this tag contains a regular expression. For instance :

<ignore type=”sregex”>^/etc/adjtime</ignore>

This ignore block will drop any file (including path) that starts with /etc/adjtime. Information on the regular expression syntax is in the OSSEC Manual.

There are a number of other tags available for syscheck that you can find in the manual. Among these are some more useful ones such as alert_new_files and, if you’re in a Windows environment, windows_registry. I encourage you to check out these options and identify which, if any, would benefit your environment.

When OSSEC detects that a file has changed, it will send an alert. Alerts will be reported via the alert mechanism you have defined within the configuration. Syscheck alerts are, by default, set to a level of 7, so our alert settings will result in an email being sent. An example alert is below :

OSSEC HIDS Notification.
2010 Oct 13 08:08:05

Received From: (Example) 192.168.0.1->syscheck
Rule: 550 fired (level 7) -> “Integrity checksum changed.”
Portion of the log(s):

Integrity checksum changed for: ‘/etc/adjtime’
Old md5sum was: ‘7234e9b2255e62178c5650982bae9cbc’
New md5sum is : ‘01210c2018146c2a9ca89505118c42f8’
Old sha1sum was: ‘df60021e39119b42a4ae508ad19b65019df089af’
New sha1sum is : ‘694b403b74a2aa339ba323b65a6d724aa8129e3b’

–END OF NOTIFICATION

OSSEC makes some attempts at identifying false positives by automatically ignoring files that change frequently. By default, ossec will begin ignoring any file that has changed three times. You can change this behavior by creating custom rules that override the defaults. I’ll be covering rules in my next blog post, so stay tuned!

Now that you have a functional OSSEC installation it’s time to start configuring it. Proper configuration is really the heart and soul of OSSEC and will be the primary focus for the next few days. The primary OSSEC configuration files use an XML format. If you’re not familiar with XML, don’t worry too much. It’s a pretty easy format to wrap your head around, as you’ll see.

The primary configuration file is the /var/ossec/etc/ossec.conf file. The only exception to this is when you are using centralized agent configuration, which I highly recommend for large deployments. For a centralized setup, the server configuration is located in ossec.conf and the agent configuration is located in /var/ossec/etc/shared/agent.conf. All agents are configured using a single agent.conf file. I’ll explain how this works in a bit.

Before I get to the standard configuration options available to ossec.conf and agent.conf, let’s talk briefly about agents and centralized agent configurations. When using a centralized configuration, I recommend minimizing what you place in the agent’s ossec.conf file. The centralized agent.conf file overrides any conflicting option listed in the agent’s ossec.conf file. Besides, lingering configuration options in the agent’s ossec.conf can result in confusion when you’re not aware of those settings. To make general configuration simple and straightforward, the only option that should be present in the agent’s ossec.conf is the address of the server or servers to use.

<ossec_config>
<client>
<server-ip>192.168.0.1</server-ip>
</client>
</ossec_config>

The agent.conf file contains all of the configuration for every agent in the network. An agent will receive all of the configuration information relevant to that specific agent. The agent.conf file is actually sent to every agent in the network and each agent parses the file looking for configuration settings relevant to that local agent. This means that you can define an initial “generic” configuration block that will apply to all agents, and then use successive configuration blocks to define additional configuration options. Note: Configurations are cumulative.

It’s a bit easier to look at a simple configuration as an example. For now, don’t worry too much about what these options do, instead, concentrate on seeing how the agent builds its configuration.

<agent_config>
<syscheck>
<!– Frequency that syscheck is executed — default every 2 hours –>
<frequency>7200</frequency>

<!– Directories to check (perform all possible verifications) –>
<!– Files/directories to ignore –>
<directories check_all=”yes”>/etc</directories>
<ignore>/etc/adjtime</ignore>
</syscheck>
</agent_config>
<agent_config name=”agent1″>
<syscheck>
<directories check_all=”yes”>/usr/agent1</directories>
</syscheck>
</agent_config>

The above configuration is *very* tiny and very basic. However, it should help to illustrate how the agent config works. The first agent_config block defines a generic configuration that will be used for every agent in the network. The subsequent agent_config block defines an additional directory to check for agent1. So agent1 will run with the primary configuration from the first block, plus the configuration from the second block.

Each agent_config block can be defined specifically for an agent or os using the agent and os attributes. The agent attribute is simple the name (or names separates by a |) of the agent(s) to apply that agent_config block to. The os attribute is based on the uname of the agent operating system. Some examples include Linux, FreeBSD, Windows, and Darwin.

The only other real difference between a standalone or server config and an agent config is that there are some options that belong in the server config as opposed to the agent config. For instance, when we talk about active response later in the week, the active response settings are placed in the server config. Likewise, the rules are detailed in the server config and not in the agent config. This will make more sense when we cover these topics. For now, if you’re unsure as to where a particular configuration option belongs, be sure to check the OSSEC manual.

That’s all for now. Check in tomorrow when we’ll make OSSEC do something more than just sit there!

As it turns out, before you can play with OSSEC and begin learning the intricacies of host-based intrusion detection, you need to install the software. Installation itself is pretty easy and fairly straightforward, provided you know how you want to install and run the system.

What do I mean by this? Well, there are three ways to install the OSSEC software. It can be installed as standalone, as a server, or as an agent. These are pretty much what they sound like. A standalone installation is generally used when you’re administering a single machine, or when circumstances prevent you from running a centralized server. A server/agent installation is for larger installations with centralized management.

For both installation types you need to start with the raw source. From there you can run the install script and choose the type of installation you want. Alternatively, if you’re installing on a linux distribution that uses RPM for package management, you can grab a copy of the source RPM for the latest version here. The RPM version only supports the server/agent type, though you can probably run the server as a standalone install. Note: I make no guarantees about the RPMs I provide, you are expected to know what you are installing and take appropriate precautions such as verifying what you have downloaded.

If you’re going with the raw source, grab a copy here. The current release as of this writing is version 2.5. Download the source, unpack it, and run the installation script. The installation script must be run as root. You probably want to check the md5 and/or sha1 sum prior to installation, just to make sure that the code is original.

# wget http://www.ossec.net/files/ossec-hids-2.5.tar.gz
# sha1sum ossec-hids-2.5.tar.gz
3da46b493f0e50b2453c43990b46ba43e61648bf
# tar zxvf ossec-hids-2.5.tar.gz

# cd ossec-hids-2.5
# ./install.sh

Just follow the prompts an the installer will take care of the rest, including compiling the software.

The RPM install is simplified a bit as you only have to compile the code once per server architecture you want to support. In other words, if you have both 32-bit and 64-bit installations, you’ll likely want to compile the software twice. Once compiled, you have three packages, ossec-hids, ossec-hids-server, and ossec-hids-client (client = agent). The ossec-hids package is installed on every system while the server and client packages go on the appropriate systems.

In order for the agents to talk to the server, you must have port 1514 open. Additionally, you need to register the agents with the server. This is a pretty simple process, though it has to be repeated for every agent. Detailed instructions can be found here. The short and simple version is as follows:

1) Run /var/ossec/bin/manage_agents and choose to add a new agent. Follow the prompts and enter the appropriate data.
2) While still in manage_agents, select the option to extract the agent authentication key. Copy this key as you need to paste it into the agent.
3) On the agent, run /var/ossec/bin/manage_agents. Choose to import the server key. Paste in the key you copied previously.

And that’s all there is to adding a new agent. There are ways to script this, but they are a bit out of scope for an introduction article. If you’re interested in scripting this process, please check the OSSEC mailing list for more details.

Finally, after installation comes configuration! Tune in tomorrow for more details!

Today marks the first day of the Week of OSSEC. What is OSSEC you ask? Well, I’m glad you asked. Allow me to explain.

According to the OSSEC home page, OSSEC is :

OSSEC is an Open Source Host-based Intrusion Detection System. It performs log analysis, file integrity checking, policy monitoring, rootkit detection, real-time alerting and active response.

It runs on most operating systems, including Linux, MacOS, Solaris, HP-UX, AIX and Windows.

But you have an IDS already, right? One of those big fancy all-in-one units that protects your network, has tons of signatures, and spits out endless reams of data. Or maybe you’re on the Open Source track and you’re using Snort combined with some fancy OSS front end for reporting.

Well, OSSEC is a bit different. It’s a HIDS, not an IDS. In short, this means that OSSEC is designed to monitor a single host and react to events occurring on that host. While this doesn’t sound very useful from the outset, I assure you that this is significantly more useful than you think.

For instance, let’s look at a typical brute-force attack. Your average IDS will allow the traffic through the network as perfectly valid traffic. There is nothing unusual happening at a network layer, so the IDS is content with passing the traffic as normal. Once it hits the server, however, the attack makes itself clear, trying to brute force its way into the server. You can ignore this and hope your password policies are strong enough to cope, but if an attacker is persistent enough, they may eventually worm their way in. Another way to deal with this is through session throttling via the server’s firewall. OSSEC deals with this by identifying the attack and blocking the attacker’s IP.

Sure, fail2ban and other packages offer similar capabilities, but can they also be used to monitor file integrity? How about rootkit detection? Can you centralize their capabilities, using a single host to push out configurations to remote systems? Can they be used in an agentless configuration where a host is monitored without having to install software on it?

OSSEC is extremely powerful and is gaining more capabilities over time. It is in active development with no signs of slowing. In fact, version 2.5 was just released on September 27th.

Over the next week I’ll be explaining some of OSSEC’s capabilities. Ultimately, I suggest you install it on a development system and start poking at it yourself. You can also join the OSSEC mailing list and join in the various on-going conversations.