security – Technological Musings

You got your web in my firewall

This post first appeared on Redhat’s Enable Sysadmin community. You can find the post here.

Firewalls have been around in one form or another since the beginning of networking. The first firewalls weren’t even identified as firewalls. They were nothing more than physical barriers between networks. It wasn’t until the 1980’s that the first device specifically designed to be, and named, a firewall was developed by DEC. Since then, firewalls have evolved into a myriad of forms.

But what is a firewall? At its core, a firewall is a devices designed to allow or deny traffic based on a set of rules. Those rules can be as simple as “allow http and block everything else” or can be infinitely more complex including protocols, ports, addresses, and even application fingerprinting. Some modern firewalls have even incorporated machine learning into the mix.

Like other technologies, as firewalls have evolved, some niche uses have been identified. Web Application Firewalls (WAFs) are one of those niche uses. A WAF is a firewall specifically designed to handle “web” traffic. That is, traffic using the HTTP protocol. Generally speaking, the role of a WAF is to inspect all HTTP traffic destined for a web server, discard “bad” requests, and pass “good” traffic on. The details of how this works are, as you might suspect, a bit more complicated.

Much like “normal” firewalls, a WAF is expected to block certain types of traffic. To do this, you have to provide the WAF with a list of what to block. As a result, early WAF products are very similar to other products such as anti-virus software, IDS/IPS products, and others. This is what is known as signature-based detection. Signatures typically identify a specific characteristic of an HTTP packet that you want to allow or deny.

For instance, WAFs are often used to block SQL Injection attacks. A very simplistic signature may just look for key identifying elements of a typical SQL Injection attack. For instance, it may look for something like ' AND 1=1 included as part of the GET or POST request. If this matches an incoming packet, the WAF marks this as bad and discards it.

Signatures work pretty well but require a lot of maintenance to ensure that false positives are kept to a minimum. Additionally, writing signatures is often more of an art form rather than a straight-forward programming task. And signature writing can be quite complicated as well. You’re often trying to match a general attack pattern without also matching legitimate traffic. To be blunt, this can be pretty nerveracking.

To illustrate this a bit more, let’s look at ModSecurity. The ModSecurity project is an open source WAF project. It started out as a module for the Apache web server but has since evolved into a modular package that works with IIS, nginx, and others. ModSecurity is a signature-based WAF and often ships with a default set of signatures known as the OWASP ModSecurity Core Rule Set.

The Core Rule Set (CRS) is an excellent starting point for deploying a signature-based WAF. It includes signatures for all of the OWASP Top Ten web application security risks as well as a wide variety of other attacks. The developers have done their best to ensure that the CRS has few false alerts but, inevitably, anyone deploying the CRS will need to tweak the rules. This involves learning the rules language and having a deep understanding of the HTTP protocol.

Technology evolves, however, and newer WAF providers are using other approaches to block bad traffic. There has been a pretty widespread move from static configuration approaches such as allow and block lists to more dynamic methods involving APIs and machine learning. This move has been across multiple technologies including traditional firewalls, anti-virus software, and, you guessed it, WAFs.

In the brave new world of dynamic rulesets, WAFs use more intelligent approaches to identifying good and bad traffic. One of the “easier” methods employed is to put the WAF in “learning” mode so it can monitor the traffic flowing to and from the protected web server. The objective here is to “train” the WAF to identify what good traffic looks like. This may include traffic that matches patterns labeled as bad when signatures were used. Once the WAF has been trained, it’s moved to enforcement mode.

Training a WAF like this is similar to what happens when you train an email system to identify spam. Email systems often use a Bayesian filtering algorithm to identify spam. These algorithms work relatively well but can be poisoned to allow spam. Similar issues exist with algorithms used by WAF providers, especially when the WAF is in the learning mode.

More advanced WAF providers are using proprietary techniques to allow and block traffic. These techniques include algorithms that can identify whether certain attacks will work against the target system and only blocking those that would be harmful. Advanced techniques like this, however, are typically only found in WAF SaaS providers and not in self-contained WAF appliances.

WAFs, and firewalls in general, have evolved a lot over the years, moving from static to dynamic methods for identifying and blocking traffic. These techniques will only get better in the future. There are a variety of solutions available from open-source to commercial providers. No matter what your needs, there’s a WAF out there for you.

Journey of a DevOps Engineer

“The advantage of a bad memory is that one enjoys several times the same good things for the first time.”
Friedrich Nietzsche

This post first appeared on Redhat’s Enable Sysadmin community. You can find the post here.

Memory is a funny construct. What we remember is often not what happened, or even the order it happened in. One of the earliest memories I have about technology was a trip my father and I took. We drove to Manhattan, on a mission to buy a Christmas gift for my mother. This crazy new device, The Itty Bitty Book Light, had come out recently and, being an avid reader, my mother had to have one.

We drove to Manhattan, on a mission to buy a Christmas gift for my mother. This crazy new device, The Itty Bitty Book Light, had come out recently and, being an avid reader, my mother had to have one.

After navigating the streets of Manhattan and finding a parking spot, we walked down the block to what turned out to be a large bookstore. You’ve seen bookstores like this on TV and in the movies. It looks small from the outside, but once you walk in, the store is endless. Walls of books, sliding ladders, tables with books piled high. It was pretty incredible, especially for someone like me who loves reading.

But in this particular store, there was something curious going on. One of the tables was surrounded by adults, awed and whispering between one another. Unsure of what was going on, we approached. After pushing through the crowd, what I saw drew me in immediately. On the table, surrounded by books, was a small grey box, the Apple Macintosh. It was on, but no one dared approach it. No one, that is, except me. I was drawn like a magnet, immediately grokking that the small puck like device moved the pointer on the screen. Adults gasped and murmured, but I ignored them all and delved into the unknown. The year was, I believe, 1984.

Somewhere around the same time, though likely a couple years before, my father brought home a TI-99/4A computer. From what I remember, the TI had just been released, so this has to be somewhere around 1982. This machine served as the catalyst for my love of computer technology and was one of the first machines I ever cut code on.

My father tells a story about when I first started programming. He had been working on an inventory database, written from scratch, that he built for his job. I would spend hours looking over his shoulder, absorbing everything I saw. One time, he finished coding, saved the code, and started typing the command to run his code (“RUN”). Accordingly to him, I stopped him with a comment that his code was going to fail. Ignoring me as I was only 5 or 6 at the time (according to his recollection), he ran the code and, as predicted, it failed. He looked at me with awe and I merely looked back and replied “GOSUB but no RETURN”.

At this point I was off and running. Over the years I got my hands on a few other systems including the Timex Sinclair, Commodore 64 and 128, TRS-80, IBM PS/2, and finally, my very own custom built PC from Gateway 2000. Since them I’ve built hundreds of machines and spend most of my time on laptops now.

The Commodore 64 helped introduce me to the online world of Bulletin Board Systems. I spent many hours, much to the chagrin of my father and the phone bill, calling into various BBS systems in many different states. Initially I was just there to play the various door games that were available, but eventually discovered the troves of software available to download. Trading software become a big draw until I eventually stumbled upon the Usenet groups that some boards had available. I spent a lot of time reading, replying, and even ended up in more than one flame war.

My first introduction to Unix based operating systems was in college when I encountered an IBM AIX mainframe as well as a VAX. I also had access to a shell account at the local telephone company turned internet service provider. The ISP I was using helpfully sent out an email to all subscribers about the S.A.T.A.N. toolkit and how accounts found with the software in their home directories would be immediately banned. Being curious, I immediately responded looking for more information. And while my account was never banned, I did learn a lot.

At the same time, my father and I started our own BBS which grew into a local Internet Service Provider offering dial-up services. That company still exists today, though the dial-up service died off several years ago. This introduced me to networks and all the engineering that comes along with it.

From here my journey takes a bit of a turn. Since I was a kid, I’ve wanted to build video games. I’ve read a lot of books on the subject, talked to various developers (including my idol, John Carmack) and written a lot of code. I actually wrote a pacman clone, of sorts, on the Commodore 64 when I was younger. But, alas, I think I was born on the wrong coast. All of the game companies, at the time, were located on the west coast and I couldn’t find a way to get there. So, I redirected my efforts a bit and focused on the technology I could get my hands on.

Running a BBS, and later an ISP, was a great introduction into the world of networking. After working a few standard school-age jobs (fast food, restaurants, etc), I finally found a job doing tech support for an ISP. I paid attention, asked questions, and learned everything I could. I took initiative where I could, even writing a cli-based ticketing system with an mSQL database backing it.

This initiative paid of as I was moved later to the NOC and finally to Network Engineering. I lead the way, learning everything I could about ATM and helping to design and build the standard ATM-based node design used by the company for over a decade. I took over development of the in-house monitoring system, eventually rewriting it in KSH and, later, Perl. I also had a hand in development of a more modern ticketing system written in Perl that is still in use today.

I spent 20+ years as a network engineer, taking time along the way to ensure that we had Linux systems available for the various scripting and monitoring needed to ensure the network performed as it should. I’ve spent many hours writing code in shell, expect, perl, and other languages to automate updates and monitoring. Eventually, I found myself looking for a new role and a host of skills ranging from network and systems administration to coding and security.

About this time, DevOps was quickly becoming the next new thing. Initially I rejected the idea, solely responding to the “Development” and “Operations” tags that make up the name. Over time, however, I came to realize that DevOps is yet another fancy name for what I’ve been doing for decades, albeit with a few twists here and there. And so, I took a role as a DevOps Engineer.

DevOps is a fun discipline, mixing in technologies from across the spectrum in an effort to automate away everything you can. Let the machine do the work for you and spent your time on more interesting projects like building more automation. And with the advent of containers and orchestration, there’s more to do than ever.

These days I find myself leading a team of DevOps engineers, both guiding the path we take as we implement new technology and automate existing processes. I also spend a lot of time learning about new technologies both on my own and for my day job. My trajectory seems to be changing slightly as I head towards the world of the SRE. Sort of like DevOps, but a bit heavier on the development side of things.

Life throws curves and sometimes you’re not able to move in the direction you want. But if you keep at it, there’s something out there for everyone. I still love learning and playing with technology. And who knows, maybe I’ll end up writing games at some point. There’s plenty of time for another new hobby.

Intrusion Prevention Systems

This post first appeared on Redhat’s Enable Sysadmin community. You can find the post here.

“Intruder alert! Strace to PID 45555, log and store.”
“Roger that, overseer. Target acquired.”
“Status report!”
“Suspicious activity verified, permission to terminate?”
“Permission granted, deploy Sigkill.”
“Process terminated at 2146 hours.”

Intrusion Prevention Systems, or IPS, are tools designed to detect and stop intrusions in their tracks. They come two basic flavors, network-based and host-based. As you may suspect, a network-based IPS is meant to be deployed to monitor the network and a host-based IPS is deployed on a host with the intention of monitoring just a single host.

How these tools work varies from vendor to vendor, but the basics are the same. The network-based tool monitors traffic on the network and matches it to a long list of known signatures. These signatures describe a variety of attacks ranging from simple corrupt packets to more specific attacks such as SQL injection.

Host-based tools tend to have more capabilities as they have access to the entire host. A host-based IPS can look at network traffic as well as monitor files and logs. One of the more popular tools, OSSEC-HIDS, monitors traffic, logs, file integrity, and even has signatures for common rootkits.

More advanced tools have additional detection capabilities such as statistical anomaly detection or stateful protocol inspection. Both of these capabilities use algorithms to detect intrusions. This allows detection of intrusions that don’t yet have signatures created for them.

IDS vs IPS

Unlike it’s predecessor, the Intrusion Detection System, or IDS, when an IPS detects an intrusion it moves to block the traffic and prevent it from getting to its target. As you can imagine, ensuring that the system blocks only bad traffic is of utmost importance. Deploying a tool that blocks legitimate traffic is a quick way to frustrate users and get yourself in trouble. An IDS merely detects the traffic and sends an alert. Given the volume of traffic on a typical network or host today, an IDS is of limited use.

The Redhat Enterprise Packages for Enterprise Linux repository includes two great HIDS apps that every admin should look into. HIDS apps are a bit of a hybrid solution in that they’re both an IDS and an IPS. That is, they can be configured to simply alert when they see issues or they can often be configured to run scripts to react to scenarios that trigger alerts.

Tripwire

First up from EPEL is Tripwire, a file integrity monitoring tool, which Seth Kenlon wrote about for Enable Sysadmin back in April. Tripwire’s job in life is to monitor files on the host and send an alert if they change in ways they’re not supposed to. For instance, you can monitor a binary such as /bin/bash and send an alert if the file changes in some way such as permissions or other attributes of the file. Attackers often modify common binaries and add a payload intended to keep their access to the server active. Tripwire can detect that.

fail2ban

The second EPEL package is fail2ban. Fail2ban is more of an IPS style tool in that it monitors and acts when it detects something awry. One common implementation of fail2ban is monitoring the openssh logs. By building a signature that identifies a failed login, fail2ban can detect multiple attempts to login from a single source address and block that source address. Typically, fail2ban does this by adding rules to the host’s firewall, but in reality, it can run any script you can come up with. So, for instance, you can write a script to block the IP on the local firewall and then transmit that IP to some central system that will distribute the firewall block to other systems. Just be careful, however, as globally blocking yourself from every system on the network can be rather embarrassing.

OSSEC

OSSEC-HIDS, mentioned previously, is a personal favorite of mine. It’s much more of a swiss army knife of tools. It combines tools like tripwire and fail2ban together into a single tool. It can be centrally managed and uses encrypted tunnels to communicate with clients. The community is very active and new signatures are created all the time. Definitely worth checking out.

Snort

Snort is a network-based IDS/IPS (NIDS/NIPS). Where HIDS are installed on servers with the intention of monitoring processes on the server itself, NIDS are deployed to monitor network traffic. Snort was first introduced in 1998 and has more recently been acquired by Cisco. Cisco has continued to support Snort as open source while simultaneously incorporating it into their product line.

Snort, like most NIDS/NIPS, works by inspecting each packet as it passes through the system. Snort can be deployed as an IDS by mirroring traffic to the system, or it can be deployed as an IPS by putting it in-line with traffic. In either case, Snort inspects the traffic and compares it to a set of signatures and heuristic patterns, looking for bad traffic. Snort’s signatures are updated on a regular basis and uses libpcap, the same library used by many popular network inspection tools such as tcpdump and wireshark.

Snort can also be configured to capture traffic for later inspection. Be aware, however, that this can eat up disk space pretty rapidly.

Suricata

Suricata is a relatively new IDS/IPS, released in 2009. Suricata is designed to be multi-threaded, making it much faster than competing products. Like Snort, it uses signatures and heuristic detection. In fact, it can use most Snort rules without any changes. It also has it’s own ruleset that allows it to use additional features such as file detection and extraction.

Zeek (Formerly Bro)

Bro was first release in 1994, making it one of the oldest IDS applications mentioned here. Originally named in reference to George Orwell’s book, 1984, more recent times have seen a re-branding to the arguably less offensive name, Zeek.

Zeek is often used as a network analysis tool but can also be deployed as an IDS. Like Snort, Zeek uses libpcap for packet capture. Once packets enter the system, however, any number of frameworks or scripts can be applied. This allows Zeek to be very flexible. Scripts can be written to be very specific, targeting specific types of traffic or scenarios, or Zeek can be deployed as an IDS, using various signatures to identify and report on suspect traffic.

Wrap-Up

One final word of caution. IPS tools are powerful and extremely useful in automating the prevention of intrusions. But like all tools, they need to be maintained properly. Ensure your signatures are up to date and monitor the reports coming from the system. It’s still possible for a skilled attacker to slip by these tools and gain access to your systems. Remember, there is no such thing as a security silver bullet.

How to leverage SNMP and not compromise the security of your server

This post first appeared on Redhat’s Enable Sysadmin community. You can find the post here.

Simple Network Management Protocol, or SNMP, has been around since 1988. While initially intended as an interim protocol as the Internet was first being rolled out, it quickly became a de facto standard for monitoring — and in some cases, managing — network equipment. Today, SNMP is used across most networks, small and large, to monitor the very equipment you likely passed through to get to this blog entry.

There are three primary flavors of SNMP: SNMPv1, SNMPv2c, and SNMPv3. SNMPv1 is, by far, the more popular flavor, despite being considered obsolete due to a complete lack of discernible security. This is likely because of the simplicity of SNMPv1 and that it’s generally used inside of the network and not exposed to the outside world.

The problem, however, is that SNMPv1 and SNMPv2c are unencrypted and even the community string used to “authenticate” is sent in the clear. An attacker can simply listen on the wire and grab the community as it passes by. This gives the attacker access to valuable information on your various devices, and even the ability to make changes if write access is enabled.

But wait, you may be thinking, what about SNMPv3? And you’re right, SNMPv3 *can* be more secure by using authentication and encryption. However, not all devices support SNMPv3 and thus interoperability becomes an issue. At some point, you’ll have to drop down to SNMPv2c or SNMPv1 and you’re back to the “in the clear” issue.

Despite the security shortcoming, SNMP can still be used without compromising the security of your server or network. Much of this security will rely on limiting use of SNMP to read-only and using tools such as iptables to limit where incoming SNMP requests can source from.

To keep things simple, we’ll worry about SNMPv1 and SNMPv2c in this article. SNMPv3 requires some additional setup and, in my opinion, isn’t worth the hassle. So let’s get started with setting up SNMP.

First things first, install the net-snmp package. This can be installed via whatever package manager you use. On the Redhat based systems I use, that tool is yum.

$ yum install net-snmp

Next, we need to configure the snmp daemon, snmpd. The configuration file is located in /etc/snmp/snmpd.conf. Open this file in your favorite editor (vim FTW!) and modify it accordingly. For example, the following configuration enables SNMP, sets up a few specific MIBs, and enables drive monitoring.

################################################################################
# AGENT BEHAVIOUR

agentaddress udp:0.0.0.0:161

################################################################################
# ACCESS CONTROL

# ------------------------------------------------------------------------------
# Traditional Access Control

# ------------------------------------------------------------------------------
# VACM Configuration
#       sec.name       source        community
com2sec notConfigUser default mysecretcommunity


#       groupName      securityModel securityName
group   notConfigGroup v1            notConfigUser
group   notConfigGroup v2c           notConfigUser

#       name          incl/excl  subtree             mask(optional)
view    systemview included .1.3.6.1.2.1.1
view    systemview included .1.3.6.1.2.1.2.2
view    systemview included .1.3.6.1.2.1.25
view    systemview included .1.3.6.1.4.1.2021
view    systemview included .1.3.6.1.4.1.8072.1.3.2.4.1.2

#       group          context sec.model sec.level prefix read       write notif
access  notConfigGroup ""      any       noauth    exact  systemview none  none

# ------------------------------------------------------------------------------
# Typed-View Configuration

################################################################################
# SYSTEM INFORMATION

# ------------------------------------------------------------------------------
# System Group
sysLocation The Internet
sysContact Internet Janitor
sysServices 72
sysName myserver.example.com

################################################################################
# EXTENDING AGENT FUNCTIONALITY


###############################################################################
## Logging
##

## We do not want annoying "Connection from UDP: " messages in syslog.
## If the following option is set to 'no', snmpd will print each incoming
## connection, which can be useful for debugging.

dontLogTCPWrappersConnects no

################################################################################
# OTHER CONFIGURATION

disk /         10%
disk /var      10%
disk /tmp      10%
disk /home     10%

Next, before you start up snmpd, make sure you configure iptables to allow SNMP traffic from trusted sources. SNMP uses UDP port 161, so all you need is a simple rule to allow traffic to pass. Be sure to add an outbound rule as well; UDP traffic is stateless.

iptables -A INPUT -s <ip addr> -p udp -m udp --dport 161 -j ACCEPT

iptables -A OUTPUT -p udp -m udp --sport 161 -j ACCEPT

You can set this up in firewalld as well, just search for SNMP and firewalld on Google.

Now that SNMP is set up, you can point an SNMP client at your server and pull data. You can pull data via the name of the MIB (if you have the MIB definitions installed) or via the OID.

$ snmpget -c mysecretcommunity myserver.example.com hrSystemUptime.0
HOST-RESOURCES-MIB::hrSystemUptime.0 = Timeticks: (6638000) 18:26:20.00

$ snmpget -c mysecretcommunity myserver.example.com .1.3.6.1.2.1.25.1.1.0
HOST-RESOURCES-MIB::hrSystemUptime.0 = Timeticks: (6638000) 18:26:20.00

And that’s about it. It’s called SIMPLE Network Management Protocol for a reason, after all.

One additional side note about SNMP. While SNMP is pretty solid, the security shortcomings are significant. I recommend looking at other solutions such as agent-based systems versus using SNMP. Tools like Nagios and Prometheus have more secure mechanisms for monitoring systems.

A Strange Game.

On March 1st, 2019, an eclectic group of diverse individuals descended upon Bally’s Hotel in Atlantic City, NJ. Their purpose? Attending a new conference melding hardware, software, and security. A conference called WOPR Summit.

I had the good fortune to both attend and volunteer at this fledgling conference. Upon arriving at the registration desk, attendees were greeted by yours truly who provided them with lanyards, stickers, and a badge that doubles as a maker project.

BEHOLD. The WOPR Badge.

The badge is both an attendees pass into the event as well as a PCB. Throughout the conference, attendees could hang out in the maker space and were provided with all of the parts, tools, and directions necessary to build their badge. When completed, a quick visit to Dragorn or Dr Russ was necessary to flash the badge processor with some Arduino code that made the lights blink in an awfully suspicious sequence. Almost as if there were a hidden message. Hrm…

And for those unfamiliar with the black art of soldering, fear not! BiaSciLab to the rescue! Our resident soldering instructor, Bia, held several soldering workshops throughout the conference, providing detailed instruction on how to become a master at fusing small metal bits together with a strong bond of liquid metal. Bia is an amazing teacher and was able to help a lot of people learn this essential life skill. She even brought her own soldering kits that you can read about here.

Not into making things? Not to worry, we had that covered as well. Throughout the conference there were both talks and workshops on a variety of topics. Workshops included topics such as NFC hacking, monitoring and incident response using OSQuery, developing prototypes, and reverse engineering. Talks covered similar topics including presentations on Shodan, biohacking with c00p3r, and a peek behind the scenes of the security industry.

Overall, the conference was an amazing success and quite well run for a first-time con. There are a lot of lessons learned and many suggestions on how to improve it for next year. Planning has already begun and we hope to see you there!

It’s docker, it’s a container, it’s… a process?

In a previous post I discussed Docker from a high level. In this post, we’ll take a closer look at how processes run in a container and how it differs from the common view of the architecture that is used to explain Docker. Remember this?

The problem with this image, however, is that while it helps conceptualize what we’re talking about, it doesn’t reflect reality. If you listed the processes outside of the container, one might think you’d see the docker daemon running and a bunch of additional processes that represent the containers themselves:

[root@dockerhost ~]# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Oct15 ?        00:02:40 /usr/lib/systemd/systemd --switched-root --system --deserialize 21
root         2     0  0 Oct15 ?        00:00:03 [kthreadd]
root         3     2  0 Oct15 ?        00:03:44 [ksoftirqd/0]
...
root      4000     1  0 Oct15 ?        00:03:44 dockerd
root      4353  4000  0 Oct15 ?        00:03:44 myawesomecontainer1
root      4354  4000  0 Oct15 ?        00:03:44 myawesomecontainer2
root      4355  4000  0 Oct15 ?        00:03:44 myawesomecontainer3

And while this might be what you’d expect based on the image above, it does not represent reality. What you’ll actually see is the docker daemon running with a number of additional helper daemons to handle things like networking, and the processes that are running “inside” of the containers like this:

[root@dockerhost ~]# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Oct15 ?        00:02:40 /usr/lib/systemd/systemd --switched-root --system --deserialize 21
root         2     0  0 Oct15 ?        00:00:03 [kthreadd]
root         3     2  0 Oct15 ?        00:03:44 [ksoftirqd/0]
...
root      1514     1  0 Oct15 ?        04:28:40 /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt nat
root      1673  1514  0 Oct15 ?        01:27:08 /usr/bin/docker-containerd-current -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout
root      4035  1673  0 Oct31 ?        00:00:07 /usr/bin/docker-containerd-shim-current d548c5b83fa61d8e3bd86ad42a7ffea9b7c86e3f9d8095c1577d3e1270bb9420 /var/run/docker/libcontainerd/
root      4054  4035  0 Oct31 ?        00:01:24 apache2 -DFOREGROUND
33        6281  4054  0 Nov13 ?        00:00:07 apache2 -DFOREGROUND
33        8526  4054  0 Nov16 ?        00:00:03 apache2 -DFOREGROUND
33       24333  4054  0 04:13 ?        00:00:00 apache2 -DFOREGROUND
root     28489  1514  0 Oct31 ?        00:00:01 /usr/libexec/docker/docker-proxy-current -proto tcp -host-ip 0.0.0.0 -host-port 443 -container-ip 172.22.0.3 -container-port 443
root     28502  1514  0 Oct31 ?        00:00:01 /usr/libexec/docker/docker-proxy-current -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip 172.22.0.3 -container-port 80
33       19216  4054  0 Nov13 ?        00:00:08 apache2 -DFOREGROUND

Without diving too deep into this, the docker processes you see above serve a few processes. There’s the main dockerd process which is responsible for management of docker containers on this host. The containerd processes handle all of the lower level management tasks for the containers themselves. And finally, the docker-proxy processes are responsible for the networking layer between the docker daemon and the host.

You’ll also see a number of apache2 processes mixed in here as well. Those are the processes running within the container, and they look just like regular processes running on a linux system. The key difference is that a number of kernel features are being used to isolate these processes so they are isolated away from the rest of the system. On the docker host you can see them, but when viewing the world from the context of a container, you cannot.

What is this black magic, you ask? Well, it’s primarily two kernel features called Namespaces and cgroups. Let’s take a look at how these work.

Namespaces are essentially internal mapping mechanisms that allow processes to have their own collections of partitioned resources. So, for instance, a process can have a pid namespace allowing that process to start a number of additional processed that can only see each other and not anything outside of the main process that owns the pid namespace. So let’s take a look at our earlier process list example. Inside of a given container you may see this:

[root@dockercontainer ~]# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Nov27 ?        00:00:12 apache2 -DFOREGROUND
www-data    18     1  0 Nov27 ?        00:00:56 apache2 -DFOREGROUND
www-data    20     1  0 Nov27 ?        00:00:24 apache2 -DFOREGROUND
www-data    21     1  0 Nov27 ?        00:00:22 apache2 -DFOREGROUND
root       559     0  0 14:30 ?        00:00:00 ps -ef

While outside of the container, you’ll see this:

[root@dockerhost ~]# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Oct15 ?        00:02:40 /usr/lib/systemd/systemd --switched-root --system --deserialize 21
root         2     0  0 Oct15 ?        00:00:03 [kthreadd]
root         3     2  0 Oct15 ?        00:03:44 [ksoftirqd/0]
...
root      1514     1  0 Oct15 ?        04:28:40 /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt nat
root      1673  1514  0 Oct15 ?        01:27:08 /usr/bin/docker-containerd-current -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout
root      4035  1673  0 Oct31 ?        00:00:07 /usr/bin/docker-containerd-shim-current d548c5b83fa61d8e3bd86ad42a7ffea9b7c86e3f9d8095c1577d3e1270bb9420 /var/run/docker/libcontainerd/
root      4054  4035  0 Oct31 ?        00:01:24 apache2 -DFOREGROUND
33        6281  4054  0 Nov13 ?        00:00:07 apache2 -DFOREGROUND
33        8526  4054  0 Nov16 ?        00:00:03 apache2 -DFOREGROUND
33       24333  4054  0 04:13 ?        00:00:00 apache2 -DFOREGROUND
root     28489  1514  0 Oct31 ?        00:00:01 /usr/libexec/docker/docker-proxy-current -proto tcp -host-ip 0.0.0.0 -host-port 443 -container-ip 172.22.0.3 -container-port 443
root     28502  1514  0 Oct31 ?        00:00:01 /usr/libexec/docker/docker-proxy-current -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip 172.22.0.3 -container-port 8033       19216  4054  0 Nov13 ?        00:00:08 apache2 -DFOREGROUND

There are two things to note here. First, within the container, you’re only seeing the processes that the container runs. No systems, no docker daemons, etc. Only the apache2 and ps processes. From outside of the container, however, you see all of the processes running on the system, including those within the container. And, the PIDs listed inside if the container are different from those outside of the container. In this example, PID 4054 outside of the container would appear to map to PID 1 inside of the container. This provides a layer of security such that running a process inside of a container can only interact with other processes running in the container. And if you kill process 1 inside of a container, the entire container comes to a screeching halt, much as if you kill process 1 on a linux host.

PID namespaces are only one of the namespaces that Docker makes use of. There are also NET, IPC, MNT, UTS, and User namespaces, though User namespaces are disabled by default. Briefly, these namespaces provide the following:

NET
- Isolates a network stack for use within the container. Network stacks can, and typically are, shared between containers.
IPC
- Provides isolated Inter-Process Communications within a container, allowing a container to use features such as shared memory while keeping the communication isolated within the container.
MNT
- Allows mount points to be isolated, preventing new mount points from being added to the host system.
UTS
- Allows different host and domains names to be presented to containers
User
- Allows a mapping of users and groups with container to the host system, thereby preventing a root user within a container from running as pid 1 outside of the container.

The second piece of black magic used is Control Groups or cgroups. Cgroups isolates resource usage for a process. Where Namespaces creates a localized view of resources for a process, cgroups creates a limited pool of resources for a process. For instance, you can assign specific CPU, Memory, and Disk I/O limits to a container. With a cgroup is assigned, the process cannot exceed the limits put on it, thereby preventing processes from “running away” and exhausting system resources. Instead, the process either deals with the lower resource limits, or crashes.

By themselves, these features can be a bit daunting to set up for each process or group of processes. Docker conveniently packages this up, making deployment as simple as a docker run command. Combined with the packaging of a Docker container (which I’ll cover in a future post), Docker becomes a great way to deploy software in a reproducible, secure manner.

Yes, your iToaster needs security

Let’s talk about your house for a moment. For the sake of argument, we’ll assume that you live in a nice house with doors, windows, the works. All of the various entries have the requisite locking devices. As with most homes, these help prevent unwanted entry, though a determined attacker can surely bypass them. For the moment, let’s ignore the determined attacker and just talk about casual attempts.

Throughout your time living in your home, casual attempts at illegal entry have been rebuffed. You may or may not even know about these attempts. They happen pretty randomly, but there’s typically not much in the way of evidence after the attacker gives up and leaves. So you’re pretty happy with how secure things are.

Recently, you’ve heard about this great new garage from a friend who has one. It’s really nice, low cost, and you have room for it on your property, so you decide to purchase one. You place the order and, after a few days, your new garage arrives. It’s everything you could have imagined. Plenty of room to store all the junk you have in the house, plus you can fit the car in there too!

You use the garage every day, moving boxes in and out of the garage as needed until one day you return home and, for some inexplicable reason, your car won’t fit all the way in. Well, that’s pretty weird, you think. You decide that maybe you stored too much in the garage, so you spend the rest of the day cleaning out the garage. You make some tough decisions and eventually you make enough room to put the car back in the garage.

Time passes and this happens a few more times. After a while you start to get a bit frustrated and decide that maybe you need to buy a bigger garage. You pull out your trusty measuring tape to verify the dimensions of the garage and, to your amazement, the garage is smaller than what you remember. You do some more checking and, to your amazement, the garage is bigger on the outside. So you call an expert to figure out what’s going on.

When the expert arrives, she takes one look at the situation and tells you she knows exactly what has happened. You watch with awe as she walks up to the closed garage, places her hand on the door, and the door opens by itself! Curious, you ask how she performed that little magic trick. She explains that this particular model of garage has a little known problem that allows the door to be opened by putting pressure on just the right place. Next, she head into the garage and starts poking around at the walls. After a few moments, one of the walls slides open revealing another room full of stuff you don’t recognize.

Your expert explains that obviously someone else knows about this weakness and has set up a false wall in your garage to hide their own stuff in. This is the source of the shrinking space and your frustration. She helps you clean up the mess and tear down the false wall. After everything is back to normal, she recommends you contact the manufacturer and see if they have a fix for the faulty door.

While this story may sound pretty far fetched when we’re talking about houses and garages, it’s an all too common story for consumer grade appliances. And as we move further into this new age of connected devices, commonly called the Internet of Things (IoT), it’s going to become and even bigger issue.

Network access itself is the first challenge. Many of the major home router vendors have already experienced problems with security. So right out of the gate, home networks are potentially vulnerable. This is a major problem, especially given the potentially sensitive nature of data being transmitted by a variety of new IoT devices.

Today’s devices are incredibly data-centric. From fitness trackers to environmental sensors, our devices are tracking everything. This data is collected and then transmitted to an internet-connected service where it is made available to the user in a variety of ways. Some users may find this data to be sensitive, hoping to keep it relatively private, available only to the user and, anonymously, to the service they subscribe to. Others may make this data public. But in the world of IoT, a security problem with a device compromises that choice.

Or maybe the attacker isn’t after your data at all. Perhaps, like our garage example, they’re looking for resources they can use. Maybe they want to store files, or maybe they’re looking to use your device to process their own data. Years ago, attackers would gain access to a remote system so they could take advantage of the space available on the system, typically storing data and setting up a warez site. That is, illegal copies of software available to those who know where to look. These days, however, storage is everywhere and there are many superior ways to transmit files between users. As a result, the old-school practice of setting up a warez site has mostly fallen by the wayside.

In today’s world, attackers want access to your devices for a variety of reasons. Some attackers use these devices as zombie systems for sending massive amounts of spam. Typically this just results in a slow Internet connection and possibly gets your IP banned from sending mail. Not a big deal for you, but it can be a real headache for those of us dealing with the influx of spam.

More and more, however, attackers are taking over machines to use them for their processing power, or for their connection to the Internet. For instance, some attackers compromise machines just so they can use them to mine bitcoins. It seems harmless enough, but it can be an inconvenience to the owner of the device when it doesn’t respond the way it should because it’s too busy working on something else.

Attackers are also using the Internet connections for nefarious purposes such as setting up denial of service hosts. They use your connection, and the connections of other systems they have compromised, to send massive amounts of data to a remote system. The entire purpose of this activity it to prevent the remote system from being accessible. It was widely reported that this sort of activity is what caused connectivity problems to both Microsoft’s Xbox Live service as well as the Playstation Network during Christmas of 2014.

So what can we do about this? Users clearly want this technology, so we need to do something to make it more secure. And to be clear, this problem goes beyond the vendors, it includes the users as well. Software has and will always have bugs. Some of these bugs can be exploited and result in a security problem. So the first step is ensuring that vendors are patching those bugs when they’re found. And, perhaps, vendors can be convinced to bolster their internal security teams such that secure coding practices are followed.

But vendors patching bugs isn’t the only problem, and in most cases, it’s the easy part of the problem. Once a patch exists, users have to apply that patch to their system. As we’ve seen over the years, patching isn’t something that users are very good at. Thus, automatic update systems such as those used by Microsoft and Apple, are commonplace. But this practice hasn’t carried over to devices yet. Vendors need to work on this and build these features into their hardware. Until they do, these security issues will remain a widespread problem.

So yes, your iToaster needs security. And we need vendors to take the next step and bake in automatic updating so security becomes the default. End users want devices that work without having to worry about how and when to update them. Not all manufacturers have the marketing savvy that Apple uses to make updating sexy. Maybe they can take a page out of the book Microsoft used with the Xbox One. Silent updates, automatically, overnight.

Hacker is not a dirty word

Have you ever had to fix a broken item and you didn’t have the right parts? Instead of just giving up, you looked around and found something that would work for the time being. Occasionally, you come back later and fix it the right way, but more often than not, that fix stays in place indefinitely. Or, perhaps you’ve found a novel new use for a device. It wasn’t built for that purpose, but you figured out that it fit the exact use you had in mind.

Those are the actions of a hacker. No, really. If you look up the definition of a hacker, you get all sort of responses. Wikipedia has three separate entries for the word hacker in relation to technology :

Hacker – someone who seeks and exploits weaknesses in a computer system or computer network
Hacker – (someone) who makes innovative customizations or combinations of retail electronic and computer equipment
Hacker – (someone) who combines excellence, playfulness, cleverness and exploration in performed activities

Google defines it as follows :

1. a person who uses computers to gain unauthorized access to data.
(informal) an enthusiastic and skillful computer programmer or user.
2. a person or thing that hacks or cuts roughly.

And there are more. What’s interesting here is that depending on where you look, the word hacker means different things. It has become a pretty contentious word, mostly because the media has, over time, used it to describe the actions of a particular type of person. Specifically, hacker is often used to describe the criminal actions of a person who gains unauthorized access to computer systems. But make no mistake, the media is completely wrong on this and they’re using the word improperly.

Sure, the person who broke into that computer system and stole all of that data is most likely a hacker. But, first and foremost, that person is a criminal. Being a hacker is a lifestyle and, in many cases, a career choice. Much like being a lawyer or a doctor is a career choice. Why then is hacker used as a negative term to identify criminal activity and not doctor or lawyer? There are plenty of instances where doctors, lawyers, and people from a wide variety of professions have indulged in criminal activity.

Keren Elazari spoke in 2014 at TED about hackers, and their importance in our society. During her talk she discusses the role of hackers in our society, noting that there are hackers who use their skills for criminal activity, but many more who use their skills to better the world. From hacktivist groups like Anonymous to hackers like Barnaby Jack, these people have changed the world in positive ways, helping to identify weaknesses in systems to weaknesses in governments and laws. In her own words :

My years in the hacker world have made me realize both the problem and the beauty about hackers: They just can’t see something broken in the world and leave it be. They are compelled to either exploit it or try and change it, and so they find the vulnerable aspects in our rapidly changing world. They make us, they force us to fix things or demand something better, and I think we need them to do just that, because after all, it is not information that wants to be free, it’s us.

It’s time to stop letting the media use this word improperly. It’s time to take back what is ours. Hacker has long been a term used to describe those we look up to, those we seek to emulate. It is a term we hold dear, a term we seek to defend. When Loyd Blankenship was arrested in 1986, he wrote what has become known as the Hacker’s Manifesto. This document, often misunderstood, describes the struggle many of us went through, and the joy of discovering something we could call our own. Yes, we’re often misunderstood. Yes, we’ve been marginalized for a long time. But times have changed since then and our culture is strong and growing.

Will online retailers be the next major breach target?

In the past year we have seen several high-profile breaches of brick and mortar retailers. Estimates range in the tens of millions of credit cards stolen in each case. For the most part, these retailers have weathered the storm with virtually no ill effects. In fact, it seems the same increase in stock price that TJ Maxx saw after their breach still rings true today. A sad fact indeed.

Regardless, the recent slew of breaches has finally prompted the credit card industry to act. They have declared that 2015 will be the year that chip and pin becomes the standard for all card-present transactions. And while chip and pin isn’t a silver bullet, and attackers will eventually find new and innovative ways to circumvent it, it has proven to be quite effective in Europe where it has been the standard for years.

Chip and pin changes how the credit card information is transmitted to the processor. Instead of the credit card number being read, in plain text, off of the magnetic strip, the card reader initiates an encrypted communication between the chip on the card and the card reader. The card details are encrypted and sent, along with the user’s PIN, to the card processor for verification. It is this encrypted communication between the card and, ultimately, the card processor that results in increased security. In short, the attack vectors used in recent breaches is difficult, if not impossible to pull off with these new readers. Since the information is not decrypted until it hits the card processor, attackers can’t simply skim the information at the card reader. There are, of course, other attacks, though these have not yet proven widespread.

At it’s heart, though, chip and pin only “fixes” one type of credit card transaction, card-present transactions. That is, transactions in which the card holder physically scans their card via a card reader. The other type of transaction, card-not-present transactions, are unaffected by chip and pin. In fact, the move to chip and pin may result in putting online transactions at greater risk. With brick and mortar attacks gone, attackers will move to online retailers. Despite the standard SSL encryption used between shoppers and online retailers, there are plenty of ways to steal credit card data. In fact, one might argue that a single attack could net more card numbers in a shorter time since online retailers often store credit card data as a convenience for the user.

It seems that online fraud, though expected, is being largely ignored for the moment. After all, how are we going to protect that data without supplying card readers to every online shopper? Online solutions such as PayPal, Amazon Payments, and others mitigate this problem slightly, but we still have to rely on the security they’ve put in place to protect cardholder data. Other solutions such as Apple Pay and Google Wallet seemingly combine on and offline protections, but the central data warehouse remains. The problem seems to be the security of the card number itself. And losing this data can be a huge burden for many users as they have to systematically update payment information as the result of a possible breach. This can often lead to late payments, penalties, and more.

One possible alternative is to reduce the impact a single breach can cause. What if the data that retailers stored was of little or no value to an attacker while still allowing the retailer a way to simplify payments for the shopper? What if a breach at a retailer only affected that retailer and resulted in virtually no impact on the user? A solution like this may be just what we need.

Instead of providing a retailer your credit card number and CVV, the retailer is provided a simple token. That token, coupled with a private retailer-specific token should be all that is needed to verify a transaction. Tokens can and should be different for each retailer. If a retailer is compromised, new tokens can be generated, reducing the impact on the user significantly. Attackers who successfully breach a retailer can only submit transactions if they can obtain both the private retailer token as well as the user token. And if processors put simple access-control lists in place, it increases the difficulty an attacker encounters when trying to push through a fraudulent transaction.

Obtaining tokens can be handled by redirecting a user to a payment gateway for their initial transaction. The payment gateway verifies the user and their credit card data, and then passes the generated token back to the retailer. This is similar to how retailers using existing online payment processors such as Paypal and Amazon Payments already handle payments. The credit card data never passes through the retailer network. The number of locations credit card data is stored reduces significantly as well. This, in turn, means that attackers have fewer targets and while this increases the risk a payment processor network incurs, one can argue that these networks should already have significant defenses in place.

This is only one possible solution for online payments. There are many other solutions out there being presented by both security and non-security folks. But there seems to be no significant movement on an online solution. Will it take several high-profile online breaches to convince credit card companies that a solution is needed? Or will credit card companies move to protect retailers and card holders ahead of attackers redirecting their efforts? If history is any indication, get used to having your card re-issued several times a year for the foreseeable future.

Bleeding Heart Security

Unless you’ve been living under a rock the past few days, you’ve probably heard about the Heartbleed vulnerability in OpenSSL that was disclosed on Monday, April 7th. Systems and network administrators across the globe have spent the last few days testing for this vulnerability, patching systems, and probably rocking in the corner while crying. Yes, it’s that bad. What’s more, there are a number of reports that intelligence agencies may have known about this vulnerability for some time now.

The quick and dirty is that a buffer overflow bug in the code allows an attacker to remotely read memory of an affected system in 64k chunks. The only memory accessible to an attacker would be memory used by the process being connected to, but, depending on the process, there may be a LOT of useful data in there. For instance, Yahoo was leaking usernames and passwords until late Tuesday evening.

The fabulous web comic, xkcd, explains how the attack works in layman’s terms. If you’re interested in the real nitty gritty of this vulnerability, though, there’s an excellent write-up on the IOActive Labs blog. If you’re the type that likes to play, you can find proof-of-concept code here. And let’s not forget about the client side, there’s PoC code for that as well.

OpenSSL versions 1.0.1 through 1.0.1f as well as the 1.0.2 beta code are affected. The folks at OpenSSL released version 1.0.1g on Monday which fixed the problem. Or, at least, the current problem. There’s a bit of chatter about other issues that may be lurking in the OpenSSL codebase.

Now that a few days have passed, however, what remains to be done? After all, everyone has patched their servers, right? Merely patching doesn’t make the problem disappear, though. Vulnerable code is out there and mistakes can be made. For the foreseeable future, you should be regularly scanning your network for vulnerable systems with something like Nmap. The Nmap NSE for Heartbleed scanning is already available. Alternatively, you can use something like Nagios to regularly check your existing servers.

Patching immediately may not have prevented a breach, either. Since Heartbleed doesn’t leave much of a trace beyond some oddities that your IDS may have seen, there’s virtually no way to know if anything has been taken. The best way to deal with this is to just go ahead and assume that your private keys are compromised and start replacing them. New keys, new certs. It’s painful, it’s slow, but it’s necessary.

For end users, the best thing you can do is change your passwords. I’m not aware of any “big” websites that have not patched by now, so changing passwords should be relatively safe. However, that said, Wired and Engadget have some of the best advice I’ve seen about this. In short, change your passwords today, then change them again in a few weeks. If you’re really paranoid, change them a third time in about a month. By that time, any site that is going to patch will have already patched.

Unfortunately, I think the fun is just beginning. I expect we’ll start seeing a number of related attacks. Phishing attacks are the most likely in the beginning. If private keys were compromised, then attackers can potentially impersonate websites, including their SSL certificates. This would likely involve a DNS poisoning attack, but could also be accomplished by compromising a user’s local system and setting a hosts file entry. Certificate revocation is a potential defense against this, but since many browsers have CRL checks disabled by default, it probably won’t help. Users will have to watch what they click, where they go, and what software they run. Not much different from the advice given already.

Another possible source of threats are consumer devices. As Bruce Schneier put it, “An upgrade path that involves the trash, a visit to Best Buy, and a credit card isn’t going to be fun for anyone.” What he’s referring to are the many embedded devices we use on a daily basis that may never receive updates to protect the end user. In other words, that router you purchased from the discount store? That may be affected and unless you replace it, you’ll continue to be vulnerable. Fortunately, most of these devices aren’t configured, by default, to face the Internet, so there may yet be hope.

The Heartbleed vulnerability is a serious contender for the worst security vulnerability ever released. I’m not sure of another vulnerability that exposes so many systems to such a degree as this one. Network and systems administrators will be cleaning up after this one for a while.