WoO Day 6 : Layin’ Down The Law

In a previous entry we discussed OSSEC Decoders and how they work. Decoders are only the first step in the log analysis train, though. Once log entries have been decoded, we need to do something with the results. That something is to match them up with rules so that OSSEC sends alerts and/or provides the proper response.

OSSEC rules ultimately determine what log entries will be reported and what entries will trigger active responses. And, as with decoders, rules can build upon one another. You can also chain rules together to alter the ultimate response based on conditions such as frequency and time. So let’s take a look at a simple rule.

<rule noalert=”1″ level=”0″ id=”5700″>
<decoded_as>sshd</decoded_as>
<description>SSHD messages grouped.</description>
</rule>

This is one of the default rules that ships with OSSEC. Each rule is defined with a unique ID. To prevent custom rules from interfering with core rules, the developers have reserved a range of IDs for custom rules. That said, there is nothing preventing you from using whatever ID numbering you desire. Just keep in mind that if you use an ID that is reserved for something else by the developers, new versions of OSSEC may interfere.

Also listed in the rule tag is a level and a noalert attribute. The noalert attribute is pretty straightforward, this rule won’t send an alert when it’s matched. Typically, the noalert tag is used on rules that are designed to be built upon. The level tag determines what level alert this rule will trigger when matched. Because we’re combining it with the noalert tag, the level ends up not meaning a whole lot.

The decoded_as tag identifies the parent decoder for which this rule is written. Only rules with a decoded_as tag matching the decoder used to decode the current log entry will be scanned for a match. This prevents OSSEC from having to scan every rule for a match.

The description tag is a human readable description of what the rule is. When you receive and alert, or when the alert is added to the OSSEC log, this description is added along with it. In the case of this rule, the description identifies its purpose. This rule was defined purely to group together sshd alerts. The intention is that other rules will handle the alerts for any alert-worthy ssh log entries that are detected.

Now let’s look at something that builds on this basic rule. Again, choosing a rule from the default ruleset, we have this:

<rule id=”5702″ level=”5″>
<if_sid>5700</if_sid>
<match>^reverse mapping</match>
<regex>failed – POSSIBLE BREAK</regex>
<description>Reverse lookup error (bad ISP or attack).</description>
</rule>

The first thing to note about this rule is the if_sid tag. The if_sid tag says if the rule number defined in this tag has already matched the incoming log entry, then we want to see if this rule matches. In other words, this rule is a child of the rule identified in the if_sid tag.

The match tag defines a string we’re looking for within the log entry. The regex tag also defines a string we’re trying to match, but regex can use the full set of regular expressions that OSSEC supports. If both the match and the regex are found in the log entry, then this rule matches and will alert at a level of 5.

Finally, let’s look at a more advanced rule. This rule also builds on the previous rules mentioned, but contains a few extras that make it even more powerful.

<rule timeframe=”360″ frequency=”4″ level=”10″ id=”5703″>
<if_matched_sid>5702</if_matched_sid>
<description>Possible breakin attempt </description>
<description>(high number of reverse lookup errors).</description>
</rule>

The intention of this rule is to identify repeated log entries that point at a more severe problem than a simple error. In this case we have multiple incoming ssh connections with bad reverse DNS entries. The frequency and timeframe attributes define how many times within a specific timespan a particular rule has to fire before this rule will kick in.

Notice that instead of the if_sid tag, we’re using the if_matched_sid tag. This is because we’re not necessarily making this rule a child of another, but instead making it a composite rule. In other words, this rule is fired if another rule fires multiple times based on the setting within this rule. As a result, this rule fires with a level of 10.

But now that we have rules and alerts being generated, what else can we do? The answer is that we can trigger active responses based on those rules. Generally, an active response fires when an alert comes in with a level equal to or higher than the alert level of the active response definition. There are ways to alter this, but let’s keep things simple for now.

To configure active response, first you need to define the commands that active-response will use. Note: All command and active-response configuration is placed in the ossec.conf on the server.

<command>
<name>firewall-drop</name>
<executable>firewall-drop.sh</executable>
<expect>srcip</expect>
<timeout_allowed>yes</timeout_allowed>
</command>

This snippet defines a command called firewall-drop. The command depends on an executable called firewall-drop.sh. This file must exist on all agents that will run this command. Unfortunately, there is currently no mechanism to push these files out to agents automatically, but perhaps there will be in the future? (*HINT*)

Th expect tag determines what variables are required from the alert in order to fire this command. Since this particular command adds a block rule to the server firewall, the srcip is required. And finally, timeout_allowed tells OSSEC that this command supports a timeout option. In other words, the firewall block is added and then removed after a given timeout.

Once the command is in place you need to set up the active response itself.

<active-response>
<command>firewall-drop</command>
<location>local</location>
<level>6</level>
<timeout>3600</timeout>
</active-response>

The active-response block identifies the command that will be run as well as the level at which it will fire. For this example, the firewall-drop command is run for any alert with a level of 6 or higher. We have also specified a timeout of 3600 which tells OSSEC that the command needs to be run again after 3600 seconds to remove the firewall-drop.

Also included is a location tag. This tells OSSEC where to run the command. Here we have specified local, which may be slightly confusing. This means that firewall-drop is run on the local agent that triggered the alert. So, if agent 002 triggers an ssh alert with a level of 6, then OSSEC tells agent 002 to run the firewall-drop command with a timeout of 3600 seconds.

You can also specify other options for location that allow you to run commands on the server or on specific agents. For instance, what if you want to block an IP at the edge whenever you get an alert of level 10 or higher. Perhaps you create a command called edge-block and it connects to your edge router to update an ACL located on the router. Running this on every agent is unwieldy at best and probably constitutes a significant security threat. Instead, you can have this script run on the server, or even a specific agent designed to handle this connection.

And that covers the basics for rules. I encourage you to write your own rules and test them out with the ossec-logtest program located in /var/ossec/bin. Learning to write rules is essential to running and tuning an OSSEC installation.

Tune in tomorrow for the final wrap-up of this year’s Week of OSSEC!

WoO Day 5 : Decoders Unite!

One of the strongest features of OSSEC is its log analysis engine. The log collector process monitors all of the specified logs and passes each log entry off to the analysis engine for decoding and rule matching. For agents, the log entries are transmitted to the server for processing.

There are multiple steps to identifying actionable matches starting with the decoding of each log entry. First, pre-decoding breaks apart a single log entry into manageable pieces. Decoders are then used to identify specific programs and log entry types. Decoders can build on one another allowing the operator to target specific patterns within a log entry with ease.

Let’s take a look at a simple example first. The following is a sample log entry from a typical /var/log/secure log. This log uses the syslog format for log entries.

Oct 21 00:01:00 dev sshd[31409]: Failed password for invalid user postfix from 189.126.97.181 port 57608 ssh2

Pre-decoding breaks apart the syslog format into three pieces, the hostname, program_name, and log. Using the ossec-logtest program provided with OSSEC, we can view the process OSSEC goes through for decoding and then rule matching. Pre-decoding this log entry produces the following :

[root@dev bin]# ./ossec-logtest
2010/10/21 00:01:00 ossec-testrule: INFO: Reading local decoder file.
2010/10/21 00:01:00 ossec-testrule: INFO: Started (pid: 1106).
ossec-testrule: Type one log per line.

Oct 21 00:01:00 dev sshd[31409]: Failed password for invalid user postfix from 189.126.97.181 port 57608 ssh2

**Phase 1: Completed pre-decoding.
full event: ‘Oct 21 00:01:00 dev sshd[31409]: Failed password for invalid user postfix from 189.126.97.181 port 57608 ssh2’
hostname: ‘dev’
program_name: ‘sshd’
log: ‘Failed password for invalid user postfix from 189.126.97.181 port 57608 ssh2’

As you can see, the log entry ends up with three parts. Further decoding uses pattern matching on these three parts to further identify and categorize log entries.

<decoder name=”sshd”>
<program_name>^sshd</program_name>
</decoder>

This is about as simple as a decoder can get. The decoder block starts with an attribute identifying the name of the decoder. In this case, that name is sshd. Within the decoder block is a program_name tag that contains a regular expression used to match against the program_name from pre-decoding. If the regex matches, OSSEC will use that decoder to match against any defined rules.

As I mentioned before, however, decoders can build on each other. The first decoder above reduces the number of subsequent decoders that need to be checked before decoding is complete. For example, look at the following decoder definition :

<decoder name=”ssh-invfailed”>
<parent>sshd</parent>
<prematch>^Failed \S+ for invalid user|^Failed \S+ for illegal user</prematch>
<regex offset=”after_prematch”>from (\S+) port \d+ \w+$</regex>
<order>srcip</order>
</decoder>

This decoder builds on the first as can be seen via the parent tag. Decoders work on a first match basis. In other words, the first decoder to match is used to find secondary decoders (children of the first), the first secondary decoder is used to find tertiary (children of the second), etc. If the matching decoder has no children, then that decoder is the final decoder and the decoded information is passed on to rules processing.

There are three other tags within this decoder block worth looking at. First, the prematch tag. Prematch is used as a quick way to determine if the rest of the decoder should be run. Prematches should be written so that the portion of the entry they match can be skipped by the rest of the decoder. For instance, in the decoder example above, the prematch will match the phrase “Failed password for invalid user” in the log entry. This portion of the log contains enough information to identify the type of log entry without requiring us to parse it again to extract information. The remaining part of the log entry has the information we want to capture.

Which brings us to the regex. The regex, or regular expression, is a string used to match and pull apart a log entry. The regex expression in the example is used to extract the source ip address from the log so we can use it in an active response later. The order tag identifies what the extracted information is.

Now, using these two decoders, let’s run ossec-logtest again :

2010/10/21 00:01:00 ossec-testrule: INFO: Reading local decoder file.
2010/10/21 00:01:00 ossec-testrule: INFO: Started (pid: 28358).
ossec-testrule: Type one log per line.

Oct 21 00:01:00 dev sshd[31409]: Failed password for invalid user postfix from 189.126.97.181 port 57608 ssh2

**Phase 1: Completed pre-decoding.
full event: ‘Oct 21 00:01:00 dev sshd[31409]: Failed password for invalid user postfix
from 189.126.97.181 port 57608 ssh2’
hostname: ‘dev’
program_name: ‘sshd’
log: ‘Failed password for invalid user postfix from 189.126.97.181 port 57608 ssh2’

**Phase 2: Completed decoding.
decoder: ‘sshd’
srcip: ‘189.126.97.181’

As you can see, the decoding phase has identified the decoder as sshd. The logtest program reports the name of the parent decoder used, rather than the ultimate matching decoder.

Hopefully this has given you enough information to start writing your own decoders. The decoder.xml file that comes with OSSEC is a great place to look at examples of how to craft decoders. This is a more advanced task, however, and the existing decoders cover most of the standard log entries you’ll see on a Linux or Windows machine.

For more information on decoders, please see the OSSEC manual. You might also check out chapter 4 of the OSSEC book. The book is a little outdated now, but the information within is still quite accurate. Syngress released a few chapters of the book that you can download here.

 

WoO Day 4 : Spot the Difference

One of the simplest functions that OSSEC can perform is integrity monitoring. The premise is pretty simple, let the admin know if a file has changed. More often than not, the admin will already know that the file has changed because the admin is the one that changed it. But sometimes files change because of problems in the system that the admin doesn’t know about. Or, the files may change because the server has been compromised by an outside party that has installed rogue software. Either way, the admin needs this information.

OSSEC can be configured to look at a few different aspects of a file in order to determine if it has changed or not, depending on how you configure it. But before we get to that, let’s configure OSSEC to send us alerts to begin with.

There are a number of ways OSSEC can send alerts. Alerts can be sent via syslog, email, stored in a database, or sent to third-party programs such as Prelude and Picviz. To make things a bit simpler, I’m only detailing how to set up email. If you’re interested in other alert setups, please check the OSSEC manual.

Setting up email alerts is as simple as adding two sections of configuration to the ossec.conf file. This configuration is set on the server, or on a standalone installation. In a server/agent setup, the server sends all alerts, including those for agents.

<global>
<email_notification>yes</email_notification>
<email_to>myaddress@example.com</email_to>
<smtp_server>smtp.example.com</smtp_server>
<email_from>ossec@example.com</email_from>
</global>

This first bit of configuration defines the To: and From: addresses as well as the SMTP server address. This configuration goes into the global config section which has a number of other options as well.

<alerts>
<log_alert_level>1</log_alert_level>
<email_alert_level>6</email_alert_level>
</alerts>

This portion of the configuration defines what level alerts should be sent via email and what level alerts should be logged. Don’t worry too much about what a level is, you’ll learn this in a later blog entry when we discuss rules and active response. For now, the config as shown above is enough.

Now that we’ll receive alerts that OSSEC generates, let’s set something up to send an alert!

As I mentioned before, integrity monitoring is pretty straightforward. OSSEC uses a number of different characteristics to identify when a file changes. It’s pretty easy to determine that a file has changed if the owner, permissions, or size changes. OSSEC can also be configured to check the sha1 and/or md5 hash values as well. Hashing is a way of producing a “signature” for a file that is mostly unique. It is possible to create another file with the same hash, but it’s very difficult. Combining all of these checks together makes it very improbable that an intruder can replace a file without you knowing.

Enabling syscheck is done on a per-host basis. What I mean by this is that the config is added to the ossec.conf file for server or standalone systems, and the config is added to the agent.conf for agents. As with other agent configurations, you can specify syscheck blocks for all agents as well as cumulatively for specific agents.

<syscheck>
<frequency>7200</frequency>

<directories check_all=”yes”>/etc</directories>
<ignore>/etc/adjtime</ignore>
</syscheck>

The above config is an example of how to enable syscheck. The frequency block specifies the time, in seconds, between syscheck runs. The example above runs syscheck every 2 hours. Moving further down in the config is the directories tag. Simply put, this tag identifies what directories syscheck should be checking. The directories tag can define multiple directories, separated by a comma. The check_all attribute indicates that all of the previously mentioned checks should be run. The OSSEC manual details what other attributes are available. One other attribute worth mentioning is the realtime attribute. This attribute directs OSSEC to use inotify to alert when files changes, in real time, for quicker notification. Linux is the only OS, that I am aware of, that supports this option. Please check the manual for more information on realtime use.

There are times when you want to ignore files within a directory, or even certain subdirectories. The ignore tag allows you to accomplish this. Without defining any attributes, the ignore tag must define an exact match. For instance, in the example above, the file /etc/adjtime will be ignored. However, /etc/adjtime.new or /etc/adjtime0 will not be. To ignore these files, you will need to either add explicit ignore blocks for them, or you can use a regular expression to grab them all. The type attribute allows you to specify that this tag contains a regular expression. For instance :

<ignore type=”sregex”>^/etc/adjtime</ignore>

This ignore block will drop any file (including path) that starts with /etc/adjtime. Information on the regular expression syntax is in the OSSEC Manual.

There are a number of other tags available for syscheck that you can find in the manual. Among these are some more useful ones such as alert_new_files and, if you’re in a Windows environment, windows_registry. I encourage you to check out these options and identify which, if any, would benefit your environment.

When OSSEC detects that a file has changed, it will send an alert. Alerts will be reported via the alert mechanism you have defined within the configuration. Syscheck alerts are, by default, set to a level of 7, so our alert settings will result in an email being sent. An example alert is below :

OSSEC HIDS Notification.
2010 Oct 13 08:08:05

Received From: (Example) 192.168.0.1->syscheck
Rule: 550 fired (level 7) -> “Integrity checksum changed.”
Portion of the log(s):

Integrity checksum changed for: ‘/etc/adjtime’
Old md5sum was: ‘7234e9b2255e62178c5650982bae9cbc’
New md5sum is : ‘01210c2018146c2a9ca89505118c42f8’
Old sha1sum was: ‘df60021e39119b42a4ae508ad19b65019df089af’
New sha1sum is : ‘694b403b74a2aa339ba323b65a6d724aa8129e3b’

–END OF NOTIFICATION

OSSEC makes some attempts at identifying false positives by automatically ignoring files that change frequently. By default, ossec will begin ignoring any file that has changed three times. You can change this behavior by creating custom rules that override the defaults. I’ll be covering rules in my next blog post, so stay tuned!

 

WoO Day 3 : Meet the agent

Now that you have a functional OSSEC installation it’s time to start configuring it. Proper configuration is really the heart and soul of OSSEC and will be the primary focus for the next few days. The primary OSSEC configuration files use an XML format. If you’re not familiar with XML, don’t worry too much. It’s a pretty easy format to wrap your head around, as you’ll see.

The primary configuration file is the /var/ossec/etc/ossec.conf file. The only exception to this is when you are using centralized agent configuration, which I highly recommend for large deployments. For a centralized setup, the server configuration is located in ossec.conf and the agent configuration is located in /var/ossec/etc/shared/agent.conf. All agents are configured using a single agent.conf file. I’ll explain how this works in a bit.

Before I get to the standard configuration options available to ossec.conf and agent.conf, let’s talk briefly about agents and centralized agent configurations. When using a centralized configuration, I recommend minimizing what you place in the agent’s ossec.conf file. The centralized agent.conf file overrides any conflicting option listed in the agent’s ossec.conf file. Besides, lingering configuration options in the agent’s ossec.conf can result in confusion when you’re not aware of those settings. To make general configuration simple and straightforward, the only option that should be present in the agent’s ossec.conf is the address of the server or servers to use.

<ossec_config>
<client>
<server-ip>192.168.0.1</server-ip>
</client>
</ossec_config>

The agent.conf file contains all of the configuration for every agent in the network. An agent will receive all of the configuration information relevant to that specific agent. The agent.conf file is actually sent to every agent in the network and each agent parses the file looking for configuration settings relevant to that local agent. This means that you can define an initial “generic” configuration block that will apply to all agents, and then use successive configuration blocks to define additional configuration options. Note: Configurations are cumulative.

It’s a bit easier to look at a simple configuration as an example. For now, don’t worry too much about what these options do, instead, concentrate on seeing how the agent builds its configuration.

<agent_config>
<syscheck>
<!– Frequency that syscheck is executed — default every 2 hours –>
<frequency>7200</frequency>

<!– Directories to check (perform all possible verifications) –>
<!– Files/directories to ignore –>
<directories check_all=”yes”>/etc</directories>
<ignore>/etc/adjtime</ignore>
</syscheck>
</agent_config>
<agent_config name=”agent1″>
<syscheck>
<directories check_all=”yes”>/usr/agent1</directories>
</syscheck>
</agent_config>

The above configuration is *very* tiny and very basic. However, it should help to illustrate how the agent config works. The first agent_config block defines a generic configuration that will be used for every agent in the network. The subsequent agent_config block defines an additional directory to check for agent1. So agent1 will run with the primary configuration from the first block, plus the configuration from the second block.

Each agent_config block can be defined specifically for an agent or os using the agent and os attributes. The agent attribute is simple the name (or names separates by a |) of the agent(s) to apply that agent_config block to. The os attribute is based on the uname of the agent operating system. Some examples include Linux, FreeBSD, Windows, and Darwin.

The only other real difference between a standalone or server config and an agent config is that there are some options that belong in the server config as opposed to the agent config. For instance, when we talk about active response later in the week, the active response settings are placed in the server config. Likewise, the rules are detailed in the server config and not in the agent config. This will make more sense when we cover these topics. For now, if you’re unsure as to where a particular configuration option belongs, be sure to check the OSSEC manual.

That’s all for now. Check in tomorrow when we’ll make OSSEC do something more than just sit there!

 

WoO Day 2 : In The Beginning …

As it turns out, before you can play with OSSEC and begin learning the intricacies of host-based intrusion detection, you need to install the software. Installation itself is pretty easy and fairly straightforward, provided you know how you want to install and run the system.

What do I mean by this? Well, there are three ways to install the OSSEC software. It can be installed as standalone, as a server, or as an agent. These are pretty much what they sound like. A standalone installation is generally used when you’re administering a single machine, or when circumstances prevent you from running a centralized server. A server/agent installation is for larger installations with centralized management.

For both installation types you need to start with the raw source. From there you can run the install script and choose the type of installation you want. Alternatively, if you’re installing on a linux distribution that uses RPM for package management, you can grab a copy of the source RPM for the latest version here. The RPM version only supports the server/agent type, though you can probably run the server as a standalone install. Note: I make no guarantees about the RPMs I provide, you are expected to know what you are installing and take appropriate precautions such as verifying what you have downloaded.

If you’re going with the raw source, grab a copy here. The current release as of this writing is version 2.5. Download the source, unpack it, and run the installation script. The installation script must be run as root. You probably want to check the md5 and/or sha1 sum prior to installation, just to make sure that the code is original.

# wget http://www.ossec.net/files/ossec-hids-2.5.tar.gz
# sha1sum ossec-hids-2.5.tar.gz
3da46b493f0e50b2453c43990b46ba43e61648bf
# tar zxvf ossec-hids-2.5.tar.gz

# cd ossec-hids-2.5
# ./install.sh

Just follow the prompts an the installer will take care of the rest, including compiling the software.

The RPM install is simplified a bit as you only have to compile the code once per server architecture you want to support. In other words, if you have both 32-bit and 64-bit installations, you’ll likely want to compile the software twice. Once compiled, you have three packages, ossec-hids, ossec-hids-server, and ossec-hids-client (client = agent). The ossec-hids package is installed on every system while the server and client packages go on the appropriate systems.

In order for the agents to talk to the server, you must have port 1514 open. Additionally, you need to register the agents with the server. This is a pretty simple process, though it has to be repeated for every agent. Detailed instructions can be found here. The short and simple version is as follows:

1) Run /var/ossec/bin/manage_agents and choose to add a new agent. Follow the prompts and enter the appropriate data.
2) While still in manage_agents, select the option to extract the agent authentication key. Copy this key as you need to paste it into the agent.
3) On the agent, run /var/ossec/bin/manage_agents. Choose to import the server key. Paste in the key you copied previously.

And that’s all there is to adding a new agent. There are ways to script this, but they are a bit out of scope for an introduction article. If you’re interested in scripting this process, please check the OSSEC mailing list for more details.

Finally, after installation comes configuration! Tune in tomorrow for more details!

 

WoO Day 1 : Introduction

Today marks the first day of the Week of OSSEC. What is OSSEC you ask? Well, I’m glad you asked. Allow me to explain.

According to the OSSEC home page, OSSEC is :

OSSEC is an Open Source Host-based Intrusion Detection System. It performs log analysis, file integrity checking, policy monitoring, rootkit detection, real-time alerting and active response.

It runs on most operating systems, including Linux, MacOS, Solaris, HP-UX, AIX and Windows.

But you have an IDS already, right? One of those big fancy all-in-one units that protects your network, has tons of signatures, and spits out endless reams of data. Or maybe you’re on the Open Source track and you’re using Snort combined with some fancy OSS front end for reporting.

Well, OSSEC is a bit different. It’s a HIDS, not an IDS. In short, this means that OSSEC is designed to monitor a single host and react to events occurring on that host. While this doesn’t sound very useful from the outset, I assure you that this is significantly more useful than you think.

For instance, let’s look at a typical brute-force attack. Your average IDS will allow the traffic through the network as perfectly valid traffic. There is nothing unusual happening at a network layer, so the IDS is content with passing the traffic as normal. Once it hits the server, however, the attack makes itself clear, trying to brute force its way into the server. You can ignore this and hope your password policies are strong enough to cope, but if an attacker is persistent enough, they may eventually worm their way in. Another way to deal with this is through session throttling via the server’s firewall. OSSEC deals with this by identifying the attack and blocking the attacker’s IP.

Sure, fail2ban and other packages offer similar capabilities, but can they also be used to monitor file integrity? How about rootkit detection? Can you centralize their capabilities, using a single host to push out configurations to remote systems? Can they be used in an agentless configuration where a host is monitored without having to install software on it?

OSSEC is extremely powerful and is gaining more capabilities over time. It is in active development with no signs of slowing. In fact, version 2.5 was just released on September 27th.

Over the next week I’ll be explaining some of OSSEC’s capabilities. Ultimately, I suggest you install it on a development system and start poking at it yourself. You can also join the OSSEC mailing list and join in the various on-going conversations.

 

The Privacy Problem

Live free or die: Death is not the worst of evils.
General John Stark

Is life so dear, or peace so sweet, as to be purchased at the price of chains and slavery? Forbid it, Almighty God! I know not what course others may take; but as for me, give me liberty or give me death!
Patrick Henry

Privacy, n.
1. The state or condition of being alone, undisturbed, or free from public attention, as a matter of choice or right; seclusion; freedom from interference or intrusion.
2. The state of being privy to some act
3. a. Absence or avoidance of publicity or display; secrecy, concealment, discretion; protection from public knowledge or availability.
3. b. The keeping of a secret; reticence.
Oxford English Dictionary

Privacy is often taken for granted. When the US Constitution was drafted, the founding fathers made sure to put in provisions to guarantee the privacy of the citizens they would govern. Most scholars agree that their intention was to prevent government intrusion in private lives and activities. They were very forward thinking, trying to ensure this protection would continue indefinitely into the future. Unfortunately, even the most forward thinking, well intentioned individual won’t be able to cover all of the possible scenarios that will occur in the future.

Since that fateful day in 1787, a war has raged between those advocating absolute privacy and those advocating reasonable intrusion for the sake of security. At the extreme edge of the argument are the non-consequentialists who believe that privacy should be absolute. They believe that privacy is non-negotiable and that the loss of privacy is akin to slavery. A common argument is that giving up privacy merely encourages additional loss. In other words, if you allow your privacy to be compromised once, then those that violate it will expect to be able to violate it again.

At the other edge are those that believe that privacy is irrelevant in the face of potential evil. This is also a non-consequentialist view. Individuals with this view tend to argue that if you have something to hide, then you are obviously guilty of something.

Somewhere in the middle are the consequentialists who believe that privacy is essential to a point. Violation of privacy should be allowed when the benefit of doing so outweighs the benefit of keeping something private. In other words, if disclosing a secret may save a life, or prevent an innocent person from going to jail, then a violation of privacy should be allowed.

The right to privacy has been fought over for years. In more recent years, technological advances have brought to light many of the problems with absolute privacy, and at the same time, have highlighted the need for some transparency. Technology has benefits for both the innocent and the criminal. It makes no delineation between the two, offering the same access to information for both.

New technologies have allowed communication over long distances, allowing criminals to coordinate criminal activities without the need to gather. Technology has brought devastating weaponry to the average citizen. Terrorists can use an Internet search engine to learn how to build bombs, plan attacks, and communicate with relative privacy. Common tools can be used to replicate identification papers, allowing criminals access to secure areas. The Internet can be used to obtain access to remote systems without permission.

Technology can also be used in positive ways. Mapping data can be used to optimize travel, find new places, and get you home when you’re lost. Online stores can be used to conveniently shop from your home, or find products you normally wouldn’t have access to. Social networking can be used to keep in touch with friends and relatives, and to form new friendships with strangers you may never have come in contact with otherwise. Wikipedia can be used for research and updated by complete strangers to spread knowledge. Companies can stay in contact with customers, alerting them of new products, updates to existing ones, or even alert them to potential problems with something they previously purchased.

In the last ten or so years, privacy in the US has been “under attack.” These so-called attacks come from many different sources. Governmental agencies seek access to more and more private information in order to combat terrorism and other criminal activities. Private organizations seek to obtain private information to identify new customers, customize advertisements, prevent fraud, etc. Technology has enabled these organizations to obtain this data in a variety of ways, often unbeknownst to the average user.

When was the last time you went to the airport and waited for someone to arrive at the gate? How about escorting someone to the gate before their flight? As recently as 20 years ago, it was possible to do both. However, since that time, security measures have been put in place to prevent non-ticketed individuals access beyond security checkpoints. Since the 9/11 terrorist attacks, security has been enhanced to include random searches, bomb sniffing, pat downs, full-body scanners, and more. In fact, the Transportation Security Administration (TSA) started random screening at the gate in 2008. Even more recently, the TSA has authorized random swabbing of passenger hands to detect explosive residue. While these measures arguably enhance security, it does so at the expense of the private individual. Many travelers feel violated by the process, even arguing that they are assumed to be guilty, having to prove their innocence every time they fly.

Traditionally, any criminal proceeding is conducted with the assumption of innocence. A criminal is considered innocent of a crime unless and until they are proven guilty. In the airport example above, the passengers are being screened with what can be considered an assumption of guilt. If you refuse to be screened, you are barred from flying, if lucky, or taken in for additional questioning and potentially jailed for the offense. Of course, individuals are not granted the right to fly, but rather offered the opportunity at the expense of giving up some privacy. It’s when these restrictions are applied to daily life, without the consent of the individual, that more serious problems arise.

Each and every day, the government gathers information about its citizens. This information is generally available to the public, although access is not necessarily easy. How this information is used, however, is often a source of criticism by privacy advocates. Massive databases of information have been built with algorithms digging through the data looking for patterns. If these patterns match, the individuals to whom the data belongs can be subject to additional scrutiny. This “fishing” for wrongdoing is often at the crux of the privacy argument. Generally speaking, if you look hard enough, and you gather enough data, you can find wrongdoing. More often, however, false positives pop up and individuals are subjected to additional scrutiny without warrant. In some cases, individuals can be wrongly detained.

Many privacy opposers argue that an innocent person has nothing to hide. However, this argument can be considered a fallacy. Professor Daniel Solove wrote an essay explaining why this argument is faulty. He argues that the “nothing to hide argument” is essentially hollow. Privacy is an inherently individualistic preference. Without knowing the full extent of how information will be used, it is impossible to say that revealing everything will have no ill effects, assuming the individual is innocent of wrongdoing. For instance, data collected by the government may not be used to identify you as a criminal, but it may result in embarrassment or feelings of exposure. What one person may consider a non-issue, others may see as evil or wrong.

These arguments extend beyond government surveillance and into the private sector as well. Companies collect information about consumers at an alarming rate. Information entered into surveys, statistics collected from websites, travel information collected from toll booths, and more can be used to profile individuals. This information is made available, usually at a cost, to other companies or even individuals. This information isn’t always kept secure, either. Criminals often access remote systems, obtaining credit card and social security numbers. Stalkers and pedophiles use social networking sites to follow their victims. Personal information posted on public sites can find its way into credit reports and is even used by some businesses to justify firing employees.

Privacy laws have been put in place to prevent such abuses, but information is already out there. Have you taken the time to put your name into a search engine lately? Give it a try, you may be surprised by the information you can find out about yourself. These are public records that can be accessed by anyone. Financial and real estate information is commonly available to the public, accessible to those knowing how to look for it. Criminal records and court proceedings are published on the web now, allowing anyone a chance to access it.

Whenever you access a website, check out a book from the library, or chat with a friend in email, you run the risk of making that information available to people you don’t want to have it. In recent years, it has been common for potential employers to use the Internet to obtain background information on a potential employee. In some cases, embarrassing information can be uncovered, casting a negative light on an individual. Teachers have been fired because of pictures they posted, innocently, on their profile pages. Are you aware of how the information you publish on the Internet can be used against you?

There is no clear answer on what should and should not be kept private. Likewise, there is no clear answer on what private data the government and private companies should have access to. It is up to you, as an individual, to make a conscious choice as to what you make public. In an ever evolving world, the decisions you make today can and will have an impact on what may happen in the future. What you may think of as an innocent act today can potentially be used against you in the future. It’s up to you to fight for your privacy, both from the government, and from the companies you interact with. Be sure you’re aware of how your data can be used before you provide it. Privacy and private data is being used in new, interesting, and potentially harmful ways every day. Be sure you’re aware of how your data can be used before you provide it.

 

NANOG 46 – Final Thoughts

Nanog 46 is wrapping up today and it has been an incredible experience. This particular Nanog seemed to have an underlying IPv6 current to it, but, if you believe the reports, IPv6 is going to have to become the standard in the next couple of years. We’ll be running dual-stack configurations for some time to come, but IPv6 rollout is necessary.

To date, I haven’t had a lot to do with IPv6. A few years ago I set up one of the many IPv6 shims, just to check out connectivity, but never really went anywhere with it. It was nothing more than a tech demo at the time, with no real content out there to bother with. Content exists today, however, and will continue to grow as time moves on.

IPv6 connectivity is still spotty and problematic for some, though, and there doesn’t seem to be a definitive, workable solution. For instance, if your IPv6 connectivity is not properly configured, you may lose access to some sites as you receive DNS responses pointing you at IPv6 content, but that you cannot reach. This results in either a major delay in falling back to IPv4 connectivity, or complete breakage. So one of the primary problems right now is whether or not to send AAAA record responses to DNS requests when the IPv6 connectivity status of the receiver is unknown. Google, from what I understand, is using a whitelist system. When a provider has sufficient IPv6 connectivity, Google adds them to their whitelist and the provider is then able to receive AAAA records.

Those problems aside, I think rolling out IPv6 will be pretty straightforward. My general take on this is to run dual-stack to start, and probably for the forseeable future, and getting the network to hand out IPv6 addresses. Once that’s in place, then we can start offering AAAA records for services. I’m still unsure at this point how to handle DNS responses to users with possibly poor v6 connectivity.

Another area of great interest this time around is DNSSEC. I’m still quite skeptical about DNSSEC as a technology, partly due to ignorance, partly due to seeing problems with what I do understand. Rest assured, once I have a better handle on this, I’ll finish up my How DNS Works series.

I’m all for securing the DNS infrastructure and doing something to ensure that DNS cannot be poisoned the same way it can today. DNSSEC aims to add security to DNS such that you can trust the responses you receive. However, I have major concerns with what I’ve seen of DNSSEC so far. One of the bigger problems I see is that each and every domain (zone) needs to be signed. Sure, this makes sense, but my concern is the cost involved to do so. SSL Certificates are not cheap and are a recurring cost. Smaller providers may run into major issues with funding such security. As a result, they will be unable to sign their domains and participate in the secure infrastructure.

Another issue I find extremely problematic is the fallback to TCP. Cryptographic signatures are big, and they tend to be bigger, the larger the key you use. As a result, DNS responses are exceeding the size of UDP and falling back to TCP. One reason DNS works so well today is that the DNS server doesn’t have to worry about retransmissions, state of connections, etc. There is no handshake required, and the UDP packets just fly. It’s up to the client to retransmit if necessary. When you move to TCP, the nature of the protocol means that both the client and server need to keep state information and perform any necessary retransmissions. This takes up socket space on the server, takes time, and uses up many more CPU cycles. Based on a lightning talk during today’s session, when the .ORG domain was signed, they saw a 100-fold increase in TCP connections, moving from less than 1 query per second to almost 100. This concerns me greatly as the majority of the Internet has not enabled DNSSEC at this point. I can see this climbing even more, eventually overwhelming the system and bringing DNS to its knees.

I also believe that moving in this direction will allow the “bad guys” to DoS attack servers in much easier ways as they can easily trigger TCP transactions, perform various TCP-based attacks, and generally muck up the system further.

So what’s the alternative? Well, there is DNSCurve, though I know even less about that as it’s very much a fringe technology at this point. In fact, the first workable patch against djbdns was only released in the past few weeks. It’s going to take some time to absorb what’s out there, but based on the current move to DNSSEC, my general feeling is that no matter how much better DNSCurve may or may not be, it doesn’t have much of a chance. Even so, there’s a lot more to learn in this arena.

I also participated in a Security BOF. BOFs are, essentially, less structured talks on a given subject. There is a bit more audience participation and the audience tends to be a bit smaller. The Security BOF was excellent as there were conversations about abuse, spam, and methods of dealing with each. The spam problem is, of course, widespread and it’s comforting to know that you’re not the only one without a definitive answer. Of course, the flip side of that is that it’s somewhat discouraging to know that even the big guys such as Google are still facing major problems with spam. The conversation as a whole, though, was quite enlightening and I learned a lot.

One of the more exciting parts of Nanog for me, though, was to meet some of the Internet greats. I’ve talked to some of these folks via email and on various mailing lists, but to meet them in person is a rare honor. I was able to meet and speak with both Randy Bush and Paul Vixie, both giants in their fields. I was able to rub elbows with folks from Google, Yahoo, and more. I’ve exchanged PGP keys with several people throughout the conference, serving as a geek’s autograph. I have met some incredible people and I look forward to talking with them in the future.

If you’re a network operator, or your interests lie in that direction, I strongly encourage you to make a trip to at least one NANOG in your lifetime. I’m hooked at this point and I’m looking forward to being able to attend more meetings in the future.

 

The Internet Arms Race

I’m here in sunny Philadelphia, attending NANOG46, a conference for network operators. The conference, thus far, has been excellent, with some great information being disseminated. One of the talks was by a long-time Internet pioneer, Paul Vixie. Vixie has had his hands in a lot of different projects ranging from being the primary author of BIND for many years, starting MAPS way back in 1996, and more recently, involvement with the Conficker Working Group.

Vixie’s talk was titled “Internet Superbugs and The Art of War,” and was about the struggle between Internet operators and the “criminal” element that uses the Internet for spam, DDOS attack, etc. The crux of the talk centered around the fact that it costs the bad guys next to nothing to continually evolve their attacks and use the network for their nefarious activities. On the flip side, however, it costs the network operators a good deal of time and money to try and stop these attacks.

Years ago, attacks were generally sourced from a single location and it was relatively easy to mitigate them. In addition, tracking down the source of the attack was simple enough, so legal action could be taken. At the very least, the network provider upstream from the attacker could disable the account and stop the attack.

Fast forward to today and we have botnets that are used for sending spam, performing DDOS attacks, and causing other sorts of havoc. It becomes next to impossible to mitigate a DDOS attack because the attack can be sourced from hundreds and thousands of machines simultaneously. This costs the bad guys nothing to deploy because users are largely ignorant and don’t understand the importance of patching and securing their networks. This results in millions of machines on the Internet that are exploitable. The bad guys write viruses, worms, trojans, etc. that infect these machines and turn them into zombie machines for their botnet.

Fighting these attacks becomes an exercise in futility. We use blacklists to block traffic from places we know are sending spam, we use anti-virus software to prevent infection of our machines, and more. When Conficker was detected and analyzed, researchers realized that this infection was a new evolution of attack. Conficker used cryptographic signatures to verify updates, pseudo-random lists of websites for updates, and more. The website lists are an excellent example of the costs paid by the good guys vs the bad guys.

The first generation of Conficker used a generated list of websites for updates. This list was 250 sites per day, making it difficult, but not impossible to mitigate. So, the people fighting this outbreak started buying up these domains in an attempt to prevent Conficker from updating. The authors of Conficker responded by upping this list to 50,000 per day, making it nearly impossible to buy them up. Fortunately, the people working to prevent the outbreak were able to work with ICANN and the various ccTLD companies to monitor and block purchases of these sites. Sites that already existed were thoroughly checked to ensure they weren’t hosting the new version of Conficker.

Vixie brought up an interesting point about all of this activity, though. The authors of Conficker made a relatively simple change to Conficker to make it use 50,000 domains. The people fighting Conficker spent many hours and days, not to mention a significant amount of money, to mitigate this. Smaller ccTLD companies that don’t have 24×7 abuse staff are unable to cope. They don’t have the budget to be able to do all of this work for free. As the workload climbs, they’re more likely to turn a blind eye.

All of this, in turn, means that our current mode of reacting to these attacks and mitigating them does not scale. It merely results in lost revenue and frustration. Additionally, creating lists of places to avoid, generating lists of bad content, etc. will never be able to scale over time. There is a breaking point, somewhere, and at that point we have no recourse unless we change our way of thinking.

Along the same line of thought, I came across a pretty decent quote today, originally posted by Don Franke from ISC(2):

“PC security is no longer about a virus that trashes your hard drive. It’s about botnets made up of millions of unpatched computers that attack banks, infrastructures, governments. Bandwidth caps will contribute to this unless the thinking of Internet providers and OS vendors change. Because we are all inter-connected now.”

If you read the original post, it explains how moving to bandwidth caps will only exacerbate the security problem because users will no longer be interested in wasting time downloading updates, but rather saving that bandwidth for things they’re interested in.

Overall, it was a very interesting talk and a very different way of thinking. There is no definitive answer as to what direction we need to go in to resolve this, but it’s definitely something that needs to be investigated.

 

Digital Armageddon

April 1, 2009. The major media outlets are all over this one. Digital Armageddon. The end of computing as we know it. Again. But is it? Should we all just “Chill Out?”

So what happens April 1, 2009? Well, Conficker activates. Well, sort of. It activates the latest revision of its auto-update algorithm, switching the number of domains it can find updates on from 250 per day to 50,000 per day. Conficker, in its current form, isn’t really malicious beyond techniques to prevent detection. In order to become malicious, it will need to download an update to the base code.

There are two methods by which Conficker will update its base code. The first method is to download the code via a connection to one of the 50,000 domains it generates. However, it does not scan all 50,000 domains at once. Instead, it creates a random list of 500 of the 50,000 generated domains and scans them for an update. If no update is found, Conficker sleeps for 24 hours and starts over by generating a new list of 50,000 domains, randomly picking 500, and contacting them for an update. The overall result of this is that it becomes nearly impossible to block all of the generated domains, increasing the likelyhood that an update will get through. On the flip side, this process appears that it would result in a very slow spread of updates. It can easily take days, weeks, or months for a single machine to finally stumble upon a live domain.

The second method is to download the code via a peer-to-peer connection between infected hosts. As I understand it, the peer-to-peer mechanism has been active since revision C of Conficker has been in the wild. This mechanism allows an update to spread from system to system in a very rapid manner. Additionally, based on how the peer-to-peer mechanism works, it appears that blocking it is difficult, at best.

So what is the risk here? Seriously, is my computer destined to become a molten heap of slag, a spam factory, or possibly a zombie soldier in a botnet attack against foreign governments? Is all hope lost? Oh my , are we all going to die!

For the love of all things digital, pull it together! It’s not as bad as it looks! First off all, if you consistently update your machines and keep your anti-virus up to date, chances of you being infected are very low. If you don’t keep up to date, then perhaps you should start. At any rate, fire up a web browser and search for a Conficker scanner. Most of the major anti-virus vendors have one. Make sure you’re familiar with the company you’re downloading the scanner from, though, a large number of scam sites have popped up since Conficker hit the mainstream media.

If you’re a network admin, you have a bigger job. First, I’d recommend any windows machines you are responsible for are patched. Yes, that includes those machines on that private network that is oh-so impossible to get to. Conficker can spread via samba shares and USB keys as well. Next, try scanning your network for infections. There are a number of Conficker scanners out there now thanks to the Honeynet Project and Dan Kaminsky. I have personally used both the proof-of-concept python scanner, as well as the latest version of nmap.

If you’re using nmap, the following command line works quite well and is incredibly fast :

nmap -sC –script=smb-check-vulns –script-args=safe=1 -p139,445 \
-d -PN -n -T4 –min-hostgroup 256 –min-parallelism 64 \
-oA conficker_scan

Finally, as a network admin, you should probably have some sort of Intrusion Detection System (IDS) in place. Snort is an open source IDS that works quite well and has a large community following. IDS signatures exist to detect all known variants of Conficker.

So calm down, take a deep breath, and don’t worry. I find it extremely unlikely that April 1 will result in anything more than a blip in network activity. Instead, concentrate on detection and patching. Conficker isn’t Skynet…. Yet.