Category: Security

Journey of a DevOps Engineer

“The advantage of a bad memory is that one enjoys several times the same good things for the first time.”

Friedrich Nietzsche

This post first appeared on Redhat’s Enable Sysadmin community. You can find the post here.

Memory is a funny construct. What we remember is often not what happened, or even the order it happened in. One of the earliest memories I have about technology was a trip my father and I took. We drove to Manhattan, on a mission to buy a Christmas gift for my mother. This crazy new device, The Itty Bitty Book Light, had come out recently and, being an avid reader, my mother had to have one.

We drove to Manhattan, on a mission to buy a Christmas gift for my mother. This crazy new device, The Itty Bitty Book Light, had come out recently and, being an avid reader, my mother had to have one.

After navigating the streets of Manhattan and finding a parking spot, we walked down the block to what turned out to be a large bookstore. You’ve seen bookstores like this on TV and in the movies. It looks small from the outside, but once you walk in, the store is endless. Walls of books, sliding ladders, tables with books piled high. It was pretty incredible, especially for someone like me who loves reading.

But in this particular store, there was something curious going on. One of the tables was surrounded by adults, awed and whispering between one another. Unsure of what was going on, we approached. After pushing through the crowd, what I saw drew me in immediately. On the table, surrounded by books, was a small grey box, the Apple Macintosh. It was on, but no one dared approach it. No one, that is, except me. I was drawn like a magnet, immediately grokking that the small puck like device moved the pointer on the screen. Adults gasped and murmured, but I ignored them all and delved into the unknown. The year was, I believe, 1984.

Somewhere around the same time, though likely a couple years before, my father brought home a TI-99/4A computer. From what I remember, the TI had just been released, so this has to be somewhere around 1982. This machine served as the catalyst for my love of computer technology and was one of the first machines I ever cut code on.

My father tells a story about when I first started programming. He had been working on an inventory database, written from scratch, that he built for his job. I would spend hours looking over his shoulder, absorbing everything I saw. One time, he finished coding, saved the code, and started typing the command to run his code (“RUN”). Accordingly to him, I stopped him with a comment that his code was going to fail. Ignoring me as I was only 5 or 6 at the time (according to his recollection), he ran the code and, as predicted, it failed. He looked at me with awe and I merely looked back and replied “GOSUB but no RETURN”.

At this point I was off and running. Over the years I got my hands on a few other systems including the Timex Sinclair, Commodore 64 and 128, TRS-80, IBM PS/2, and finally, my very own custom built PC from Gateway 2000. Since them I’ve built hundreds of machines and spend most of my time on laptops now.

The Commodore 64 helped introduce me to the online world of Bulletin Board Systems. I spent many hours, much to the chagrin of my father and the phone bill, calling into various BBS systems in many different states. Initially I was just there to play the various door games that were available, but eventually discovered the troves of software available to download. Trading software become a big draw until I eventually stumbled upon the Usenet groups that some boards had available. I spent a lot of time reading, replying, and even ended up in more than one flame war.

My first introduction to Unix based operating systems was in college when I encountered an IBM AIX mainframe as well as a VAX. I also had access to a shell account at the local telephone company turned internet service provider. The ISP I was using helpfully sent out an email to all subscribers about the S.A.T.A.N. toolkit and how accounts found with the software in their home directories would be immediately banned. Being curious, I immediately responded looking for more information. And while my account was never banned, I did learn a lot.

At the same time, my father and I started our own BBS which grew into a local Internet Service Provider offering dial-up services. That company still exists today, though the dial-up service died off several years ago. This introduced me to networks and all the engineering that comes along with it.

From here my journey takes a bit of a turn. Since I was a kid, I’ve wanted to build video games. I’ve read a lot of books on the subject, talked to various developers (including my idol, John Carmack) and written a lot of code. I actually wrote a pacman clone, of sorts, on the Commodore 64 when I was younger. But, alas, I think I was born on the wrong coast. All of the game companies, at the time, were located on the west coast and I couldn’t find a way to get there. So, I redirected my efforts a bit and focused on the technology I could get my hands on.

Running a BBS, and later an ISP, was a great introduction into the world of networking. After working a few standard school-age jobs (fast food, restaurants, etc), I finally found a job doing tech support for an ISP. I paid attention, asked questions, and learned everything I could. I took initiative where I could, even writing a cli-based ticketing system with an mSQL database backing it.

This initiative paid of as I was moved later to the NOC and finally to Network Engineering. I lead the way, learning everything I could about ATM and helping to design and build the standard ATM-based node design used by the company for over a decade. I took over development of the in-house monitoring system, eventually rewriting it in KSH and, later, Perl. I also had a hand in development of a more modern ticketing system written in Perl that is still in use today.

I spent 20+ years as a network engineer, taking time along the way to ensure that we had Linux systems available for the various scripting and monitoring needed to ensure the network performed as it should. I’ve spent many hours writing code in shell, expect, perl, and other languages to automate updates and monitoring. Eventually, I found myself looking for a new role and a host of skills ranging from network and systems administration to coding and security.

About this time, DevOps was quickly becoming the next new thing. Initially I rejected the idea, solely responding to the “Development” and “Operations” tags that make up the name. Over time, however, I came to realize that DevOps is yet another fancy name for what I’ve been doing for decades, albeit with a few twists here and there. And so, I took a role as a DevOps Engineer.

DevOps is a fun discipline, mixing in technologies from across the spectrum in an effort to automate away everything you can. Let the machine do the work for you and spent your time on more interesting projects like building more automation. And with the advent of containers and orchestration, there’s more to do than ever.

These days I find myself leading a team of DevOps engineers, both guiding the path we take as we implement new technology and automate existing processes. I also spend a lot of time learning about new technologies both on my own and for my day job. My trajectory seems to be changing slightly as I head towards the world of the SRE. Sort of like DevOps, but a bit heavier on the development side of things.

Life throws curves and sometimes you’re not able to move in the direction you want. But if you keep at it, there’s something out there for everyone. I still love learning and playing with technology. And who knows, maybe I’ll end up writing games at some point. There’s plenty of time for another new hobby.

Intrusion Prevention Systems

This post first appeared on Redhat’s Enable Sysadmin community. You can find the post here.

“Intruder alert!  Strace to PID 45555, log and store.”
“Roger that, overseer. Target acquired.”
“Status report!”
“Suspicious activity verified, permission to terminate?”
“Permission granted, deploy Sigkill.”
“Process terminated at 2146 hours.”

Intrusion Prevention Systems, or IPS, are tools designed to detect and stop intrusions in their tracks. They come two basic flavors, network-based and host-based. As you may suspect, a network-based IPS is meant to be deployed to monitor the network and a host-based IPS is deployed on a host with the intention of monitoring just a single host.

How these tools work varies from vendor to vendor, but the basics are the same. The network-based tool monitors traffic on the network and matches it to a long list of known signatures. These signatures describe a variety of attacks ranging from simple corrupt packets to more specific attacks such as SQL injection.

Host-based tools tend to have more capabilities as they have access to the entire host. A host-based IPS can look at network traffic as well as monitor files and logs. One of the more popular tools, OSSEC-HIDS, monitors traffic, logs, file integrity, and even has signatures for common rootkits.

More advanced tools have additional detection capabilities such as statistical anomaly detection or stateful protocol inspection. Both of these capabilities use algorithms to detect intrusions. This allows detection of intrusions that don’t yet have signatures created for them.

IDS vs IPS

Unlike it’s predecessor, the Intrusion Detection System, or IDS, when an IPS detects an intrusion it moves to block the traffic and prevent it from getting to its target. As you can imagine, ensuring that the system blocks only bad traffic is of utmost importance. Deploying a tool that blocks legitimate traffic is a quick way to frustrate users and get yourself in trouble. An IDS merely detects the traffic and sends an alert. Given the volume of traffic on a typical network or host today, an IDS is of limited use.

The Redhat Enterprise Packages for Enterprise Linux repository includes two great HIDS apps that every admin should look into. HIDS apps are a bit of a hybrid solution in that they’re both an IDS and an IPS. That is, they can be configured to simply alert when they see issues or they can often be configured to run scripts to react to scenarios that trigger alerts.

Tripwire

First up from EPEL is Tripwire, a file integrity monitoring tool, which Seth Kenlon wrote about for Enable Sysadmin back in April. Tripwire’s job in life is to monitor files on the host and send an alert if they change in ways they’re not supposed to. For instance, you can monitor a binary such as /bin/bash and send an alert if the file changes in some way such as permissions or other attributes of the file. Attackers often modify common binaries and add a payload intended to keep their access to the server active. Tripwire can detect that.

fail2ban

The second EPEL package is fail2ban. Fail2ban is more of an IPS style tool in that it monitors and acts when it detects something awry. One common implementation of fail2ban is monitoring the openssh logs. By building a signature that identifies a failed login, fail2ban can detect multiple attempts to login from a single source address and block that source address. Typically, fail2ban does this by adding rules to the host’s firewall, but in reality, it can run any script you can come up with. So, for instance, you can write a script to block the IP on the local firewall and then transmit that IP to some central system that will distribute the firewall block to other systems. Just be careful, however, as globally blocking yourself from every system on the network can be rather embarrassing.

OSSEC

OSSEC-HIDS, mentioned previously, is a personal favorite of mine. It’s much more of a swiss army knife of tools. It combines tools like tripwire and fail2ban together into a single tool. It can be centrally managed and uses encrypted tunnels to communicate with clients. The community is very active and new signatures are created all the time. Definitely worth checking out.

Snort

Snort is a network-based IDS/IPS (NIDS/NIPS). Where HIDS are installed on servers with the intention of monitoring processes on the server itself, NIDS are deployed to monitor network traffic. Snort was first introduced in 1998 and has more recently been acquired by Cisco. Cisco has continued to support Snort as open source while simultaneously incorporating it into their product line.

Snort, like most NIDS/NIPS, works by inspecting each packet as it passes through the system. Snort can be deployed as an IDS by mirroring traffic to the system, or it can be deployed as an IPS by putting it in-line with traffic. In either case, Snort inspects the traffic and compares it to a set of signatures and heuristic patterns, looking for bad traffic. Snort’s signatures are updated on a regular basis and uses libpcap, the same library used by many popular network inspection tools such as tcpdump and wireshark.

Snort can also be configured to capture traffic for later inspection. Be aware, however, that this can eat up disk space pretty rapidly.

Suricata

Suricata is a relatively new IDS/IPS, released in 2009. Suricata is designed to be multi-threaded, making it much faster than competing products. Like Snort, it uses signatures and heuristic detection. In fact, it can use most Snort rules without any changes. It also has it’s own ruleset that allows it to use additional features such as file detection and extraction.

Zeek (Formerly Bro)

Bro was first release in 1994, making it one of the oldest IDS applications mentioned here. Originally named in reference to George Orwell’s book, 1984, more recent times have seen a re-branding to the arguably less offensive name, Zeek.

Zeek is often used as a network analysis tool but can also be deployed as an IDS. Like Snort, Zeek uses libpcap for packet capture. Once packets enter the system, however, any number of frameworks or scripts can be applied. This allows Zeek to be very flexible. Scripts can be written to be very specific, targeting specific types of traffic or scenarios, or Zeek can be deployed as an IDS, using various signatures to identify and report on suspect traffic.

Wrap-Up

One final word of caution. IPS tools are powerful and extremely useful in automating the prevention of intrusions. But like all tools, they need to be maintained properly. Ensure your signatures are up to date and monitor the reports coming from the system. It’s still possible for a skilled attacker to slip by these tools and gain access to your systems. Remember, there is no such thing as a security silver bullet.

How to leverage SNMP and not compromise the security of your server

This post first appeared on Redhat’s Enable Sysadmin community. You can find the post here.

Simple Network Management Protocol, or SNMP, has been around since 1988. While initially intended as an interim protocol as the Internet was first being rolled out, it quickly became a de facto standard for monitoring — and in some cases, managing — network equipment. Today, SNMP is used across most networks, small and large, to monitor the very equipment you likely passed through to get to this blog entry.

There are three primary flavors of SNMP: SNMPv1, SNMPv2c, and SNMPv3. SNMPv1 is, by far, the more popular flavor, despite being considered obsolete due to a complete lack of discernible security. This is likely because of the simplicity of SNMPv1 and that it’s generally used inside of the network and not exposed to the outside world.

The problem, however, is that SNMPv1 and SNMPv2c are unencrypted and even the community string used to “authenticate” is sent in the clear. An attacker can simply listen on the wire and grab the community as it passes by. This gives the attacker access to valuable information on your various devices, and even the ability to make changes if write access is enabled.

But wait, you may be thinking, what about SNMPv3? And you’re right, SNMPv3 *can* be more secure by using authentication and encryption. However, not all devices support SNMPv3 and thus interoperability becomes an issue. At some point, you’ll have to drop down to SNMPv2c or SNMPv1 and you’re back to the “in the clear” issue.

Despite the security shortcoming, SNMP can still be used without compromising the security of your server or network. Much of this security will rely on limiting use of SNMP to read-only and using tools such as iptables to limit where incoming SNMP requests can source from.

To keep things simple, we’ll worry about SNMPv1 and SNMPv2c in this article. SNMPv3 requires some additional setup and, in my opinion, isn’t worth the hassle. So let’s get started with setting up SNMP.

First things first, install the net-snmp package. This can be installed via whatever package manager you use. On the Redhat based systems I use, that tool is yum.

$ yum install net-snmp

Next, we need to configure the snmp daemon, snmpd. The configuration file is located in /etc/snmp/snmpd.conf. Open this file in your favorite editor (vim FTW!) and modify it accordingly. For example, the following configuration enables SNMP, sets up a few specific MIBs, and enables drive monitoring.

################################################################################
# AGENT BEHAVIOUR

agentaddress udp:0.0.0.0:161

################################################################################
# ACCESS CONTROL

# ------------------------------------------------------------------------------
# Traditional Access Control

# ------------------------------------------------------------------------------
# VACM Configuration
#       sec.name       source        community
com2sec notConfigUser default mysecretcommunity


#       groupName      securityModel securityName
group   notConfigGroup v1            notConfigUser
group   notConfigGroup v2c           notConfigUser

#       name          incl/excl  subtree             mask(optional)
view    systemview included .1.3.6.1.2.1.1
view    systemview included .1.3.6.1.2.1.2.2
view    systemview included .1.3.6.1.2.1.25
view    systemview included .1.3.6.1.4.1.2021
view    systemview included .1.3.6.1.4.1.8072.1.3.2.4.1.2

#       group          context sec.model sec.level prefix read       write notif
access  notConfigGroup ""      any       noauth    exact  systemview none  none

# ------------------------------------------------------------------------------
# Typed-View Configuration

################################################################################
# SYSTEM INFORMATION

# ------------------------------------------------------------------------------
# System Group
sysLocation The Internet
sysContact Internet Janitor
sysServices 72
sysName myserver.example.com

################################################################################
# EXTENDING AGENT FUNCTIONALITY


###############################################################################
## Logging
##

## We do not want annoying "Connection from UDP: " messages in syslog.
## If the following option is set to 'no', snmpd will print each incoming
## connection, which can be useful for debugging.

dontLogTCPWrappersConnects no

################################################################################
# OTHER CONFIGURATION

disk /         10%
disk /var      10%
disk /tmp      10%
disk /home     10%

Next, before you start up snmpd, make sure you configure iptables to allow SNMP traffic from trusted sources. SNMP uses UDP port 161, so all you need is a simple rule to allow traffic to pass. Be sure to add an outbound rule as well; UDP traffic is stateless.

iptables -A INPUT -s <ip addr> -p udp -m udp --dport 161 -j ACCEPT

iptables -A OUTPUT -p udp -m udp --sport 161 -j ACCEPT

You can set this up in firewalld as well, just search for SNMP and firewalld on Google.

Now that SNMP is set up, you can point an SNMP client at your server and pull data. You can pull data via the name of the MIB (if you have the MIB definitions installed) or via the OID.

$ snmpget -c mysecretcommunity myserver.example.com hrSystemUptime.0
HOST-RESOURCES-MIB::hrSystemUptime.0 = Timeticks: (6638000) 18:26:20.00

$ snmpget -c mysecretcommunity myserver.example.com .1.3.6.1.2.1.25.1.1.0
HOST-RESOURCES-MIB::hrSystemUptime.0 = Timeticks: (6638000) 18:26:20.00

And that’s about it. It’s called SIMPLE Network Management Protocol for a reason, after all.

One additional side note about SNMP. While SNMP is pretty solid, the security shortcomings are significant. I recommend looking at other solutions such as agent-based systems versus using SNMP. Tools like Nagios and Prometheus have more secure mechanisms for monitoring systems.

Yes, your iToaster needs security

Let’s talk about your house for a moment. For the sake of argument, we’ll assume that you live in a nice house with doors, windows, the works. All of the various entries have the requisite locking devices. As with most homes, these help prevent unwanted entry, though a determined attacker can surely bypass them. For the moment, let’s ignore the determined attacker and just talk about casual attempts.

Throughout your time living in your home, casual attempts at illegal entry have been rebuffed. You may or may not even know about these attempts. They happen pretty randomly, but there’s typically not much in the way of evidence after the attacker gives up and leaves. So you’re pretty happy with how secure things are.

Recently, you’ve heard about this great new garage from a friend who has one. It’s really nice, low cost, and you have room for it on your property, so you decide to purchase one. You place the order and, after a few days, your new garage arrives. It’s everything you could have imagined. Plenty of room to store all the junk you have in the house, plus you can fit the car in there too!

You use the garage every day, moving boxes in and out of the garage as needed until one day you return home and, for some inexplicable reason, your car won’t fit all the way in. Well, that’s pretty weird, you think. You decide that maybe you stored too much in the garage, so you spend the rest of the day cleaning out the garage. You make some tough decisions and eventually you make enough room to put the car back in the garage.

Time passes and this happens a few more times. After a while you start to get a bit frustrated and decide that maybe you need to buy a bigger garage. You pull out your trusty measuring tape to verify the dimensions of the garage and, to your amazement, the garage is smaller than what you remember. You do some more checking and, to your amazement, the garage is bigger on the outside. So you call an expert to figure out what’s going on.

When the expert arrives, she takes one look at the situation and tells you she knows exactly what has happened. You watch with awe as she walks up to the closed garage, places her hand on the door, and the door opens by itself! Curious, you ask how she performed that little magic trick. She explains that this particular model of garage has a little known problem that allows the door to be opened by putting pressure on just the right place. Next, she head into the garage and starts poking around at the walls. After a few moments, one of the walls slides open revealing another room full of stuff you don’t recognize.

Your expert explains that obviously someone else knows about this weakness and has set up a false wall in your garage to hide their own stuff in. This is the source of the shrinking space and your frustration. She helps you clean up the mess and tear down the false wall. After everything is back to normal, she recommends you contact the manufacturer and see if they have a fix for the faulty door.

While this story may sound pretty far fetched when we’re talking about houses and garages, it’s an all too common story for consumer grade appliances. And as we move further into this new age of connected devices, commonly called the Internet of Things (IoT), it’s going to become and even bigger issue.

Network access itself is the first challenge. Many of the major home router vendors have already experienced problems with security. So right out of the gate, home networks are potentially vulnerable. This is a major problem, especially given the potentially sensitive nature of data being transmitted by a variety of new IoT devices.

Today’s devices are incredibly data-centric. From fitness trackers to environmental sensors, our devices are tracking everything. This data is collected and then transmitted to an internet-connected service where it is made available to the user in a variety of ways. Some users may find this data to be sensitive, hoping to keep it relatively private, available only to the user and, anonymously, to the service they subscribe to. Others may make this data public. But in the world of IoT, a security problem with a device compromises that choice.

Or maybe the attacker isn’t after your data at all. Perhaps, like our garage example, they’re looking for resources they can use. Maybe they want to store files, or maybe they’re looking to use your device to process their own data. Years ago, attackers would gain access to a remote system so they could take advantage of the space available on the system, typically storing data and setting up a warez site. That is, illegal copies of software available to those who know where to look. These days, however, storage is everywhere and there are many superior ways to transmit files between users. As a result, the old-school practice of setting up a warez site has mostly fallen by the wayside.

In today’s world, attackers want access to your devices for a variety of reasons. Some attackers use these devices as zombie systems for sending massive amounts of spam. Typically this just results in a slow Internet connection and possibly gets your IP banned from sending mail. Not a big deal for you, but it can be a real headache for those of us dealing with the influx of spam.

More and more, however, attackers are taking over machines to use them for their processing power, or for their connection to the Internet. For instance, some attackers compromise machines just so they can use them to mine bitcoins. It seems harmless enough, but it can be an inconvenience to the owner of the device when it doesn’t respond the way it should because it’s too busy working on something else.

Attackers are also using the Internet connections for nefarious purposes such as setting up denial of service hosts. They use your connection, and the connections of other systems they have compromised, to send massive amounts of data to a remote system. The entire purpose of this activity it to prevent the remote system from being accessible. It was widely reported that this sort of activity is what caused connectivity problems to both Microsoft’s Xbox Live service as well as the Playstation Network during Christmas of 2014.

So what can we do about this? Users clearly want this technology, so we need to do something to make it more secure. And to be clear, this problem goes beyond the vendors, it includes the users as well. Software has and will always have bugs. Some of these bugs can be exploited and result in a security problem. So the first step is ensuring that vendors are patching those bugs when they’re found. And, perhaps, vendors can be convinced to bolster their internal security teams such that secure coding practices are followed.

But vendors patching bugs isn’t the only problem, and in most cases, it’s the easy part of the problem. Once a patch exists, users have to apply that patch to their system. As we’ve seen over the years, patching isn’t something that users are very good at. Thus, automatic update systems such as those used by Microsoft and Apple, are commonplace. But this practice hasn’t carried over to devices yet. Vendors need to work on this and build these features into their hardware. Until they do, these security issues will remain a widespread problem.

So yes, your iToaster needs security. And we need vendors to take the next step and bake in automatic updating so security becomes the default. End users want devices that work without having to worry about how and when to update them. Not all manufacturers have the marketing savvy that Apple uses to make updating sexy. Maybe they can take a page out of the book Microsoft used with the Xbox One. Silent updates, automatically, overnight.

Hacker is not a dirty word

Have you ever had to fix a broken item and you didn’t have the right parts? Instead of just giving up, you looked around and found something that would work for the time being. Occasionally, you come back later and fix it the right way, but more often than not, that fix stays in place indefinitely. Or, perhaps you’ve found a novel new use for a device. It wasn’t built for that purpose, but you figured out that it fit the exact use you had in mind.

Those are the actions of a hacker. No, really. If you look up the definition of a hacker, you get all sort of responses. Wikipedia has three separate entries for the word hacker in relation to technology :

Hacker – someone who seeks and exploits weaknesses in a computer system or computer network

Hacker – (someone) who makes innovative customizations or combinations of retail electronic and computer equipment

Hacker – (someone) who combines excellence, playfulness, cleverness and exploration in performed activities

Google defines it as follows :

1. a person who uses computers to gain unauthorized access to data.

(informal) an enthusiastic and skillful computer programmer or user.

2. a person or thing that hacks or cuts roughly.

And there are more. What’s interesting here is that depending on where you look, the word hacker means different things. It has become a pretty contentious word, mostly because the media has, over time, used it to describe the actions of a particular type of person. Specifically, hacker is often used to describe the criminal actions of a person who gains unauthorized access to computer systems. But make no mistake, the media is completely wrong on this and they’re using the word improperly.

Sure, the person who broke into that computer system and stole all of that data is most likely a hacker. But, first and foremost, that person is a criminal. Being a hacker is a lifestyle and, in many cases, a career choice. Much like being a lawyer or a doctor is a career choice. Why then is hacker used as a negative term to identify criminal activity and not doctor or lawyer? There are plenty of instances where doctors, lawyers, and people from a wide variety of professions have indulged in criminal activity.

Keren Elazari spoke in 2014 at TED about hackers, and their importance in our society. During her talk she discusses the role of hackers in our society, noting that there are hackers who use their skills for criminal activity, but many more who use their skills to better the world. From hacktivist groups like Anonymous to hackers like Barnaby Jack, these people have changed the world in positive ways, helping to identify weaknesses in systems to weaknesses in governments and laws. In her own words :

My years in the hacker world have made me realize both the problem and the beauty about hackers: They just can’t see something broken in the world and leave it be. They are compelled to either exploit it or try and change it, and so they find the vulnerable aspects in our rapidly changing world. They make us, they force us to fix things or demand something better, and I think we need them to do just that, because after all, it is not information that wants to be free, it’s us.

It’s time to stop letting the media use this word improperly. It’s time to take back what is ours. Hacker has long been a term used to describe those we look up to, those we seek to emulate. It is a term we hold dear, a term we seek to defend. When Loyd Blankenship was arrested in 1986, he wrote what has become known as the Hacker’s Manifesto. This document, often misunderstood, describes the struggle many of us went through, and the joy of discovering something we could call our own. Yes, we’re often misunderstood. Yes, we’ve been marginalized for a long time. But times have changed since then and our culture is strong and growing.

Will online retailers be the next major breach target?

In the past year we have seen several high-profile breaches of brick and mortar retailers. Estimates range in the tens of millions of credit cards stolen in each case. For the most part, these retailers have weathered the storm with virtually no ill effects. In fact, it seems the same increase in stock price that TJ Maxx saw after their breach still rings true today. A sad fact indeed.

Regardless, the recent slew of breaches has finally prompted the credit card industry to act. They have declared that 2015 will be the year that chip and pin becomes the standard for all card-present transactions. And while chip and pin isn’t a silver bullet, and attackers will eventually find new and innovative ways to circumvent it, it has proven to be quite effective in Europe where it has been the standard for years.

Chip and pin changes how the credit card information is transmitted to the processor. Instead of the credit card number being read, in plain text, off of the magnetic strip, the card reader initiates an encrypted communication between the chip on the card and the card reader. The card details are encrypted and sent, along with the user’s PIN, to the card processor for verification. It is this encrypted communication between the card and, ultimately, the card processor that results in increased security. In short, the attack vectors used in recent breaches is difficult, if not impossible to pull off with these new readers. Since the information is not decrypted until it hits the card processor, attackers can’t simply skim the information at the card reader. There are, of course, other attacks, though these have not yet proven widespread.

At it’s heart, though, chip and pin only “fixes” one type of credit card transaction, card-present transactions. That is, transactions in which the card holder physically scans their card via a card reader. The other type of transaction, card-not-present transactions, are unaffected by chip and pin. In fact, the move to chip and pin may result in putting online transactions at greater risk. With brick and mortar attacks gone, attackers will move to online retailers. Despite the standard SSL encryption used between shoppers and online retailers, there are plenty of ways to steal credit card data. In fact, one might argue that a single attack could net more card numbers in a shorter time since online retailers often store credit card data as a convenience for the user.

It seems that online fraud, though expected, is being largely ignored for the moment. After all, how are we going to protect that data without supplying card readers to every online shopper? Online solutions such as PayPal, Amazon Payments, and others mitigate this problem slightly, but we still have to rely on the security they’ve put in place to protect cardholder data. Other solutions such as Apple Pay and Google Wallet seemingly combine on and offline protections, but the central data warehouse remains. The problem seems to be the security of the card number itself. And losing this data can be a huge burden for many users as they have to systematically update payment information as the result of a possible breach. This can often lead to late payments, penalties, and more.

One possible alternative is to reduce the impact a single breach can cause. What if the data that retailers stored was of little or no value to an attacker while still allowing the retailer a way to simplify payments for the shopper? What if a breach at a retailer only affected that retailer and resulted in virtually no impact on the user? A solution like this may be just what we need.

Instead of providing a retailer your credit card number and CVV, the retailer is provided a simple token. That token, coupled with a private retailer-specific token should be all that is needed to verify a transaction. Tokens can and should be different for each retailer. If a retailer is compromised, new tokens can be generated, reducing the impact on the user significantly. Attackers who successfully breach a retailer can only submit transactions if they can obtain both the private retailer token as well as the user token. And if processors put simple access-control lists in place, it increases the difficulty an attacker encounters when trying to push through a fraudulent transaction.

Obtaining tokens can be handled by redirecting a user to a payment gateway for their initial transaction. The payment gateway verifies the user and their credit card data, and then passes the generated token back to the retailer. This is similar to how retailers using existing online payment processors such as Paypal and Amazon Payments already handle payments. The credit card data never passes through the retailer network. The number of locations credit card data is stored reduces significantly as well. This, in turn, means that attackers have fewer targets and while this increases the risk a payment processor network incurs, one can argue that these networks should already have significant defenses in place.

This is only one possible solution for online payments. There are many other solutions out there being presented by both security and non-security folks. But there seems to be no significant movement on an online solution. Will it take several high-profile online breaches to convince credit card companies that a solution is needed? Or will credit card companies move to protect retailers and card holders ahead of attackers redirecting their efforts? If history is any indication, get used to having your card re-issued several times a year for the foreseeable future.

Bleeding Heart Security

Unless you’ve been living under a rock the past few days, you’ve probably heard about the Heartbleed vulnerability in OpenSSL that was disclosed on Monday, April 7th. Systems and network administrators across the globe have spent the last few days testing for this vulnerability, patching systems, and probably rocking in the corner while crying. Yes, it’s that bad. What’s more, there are a number of reports that intelligence agencies may have known about this vulnerability for some time now.

The quick and dirty is that a buffer overflow bug in the code allows an attacker to remotely read memory of an affected system in 64k chunks. The only memory accessible to an attacker would be memory used by the process being connected to, but, depending on the process, there may be a LOT of useful data in there. For instance, Yahoo was leaking usernames and passwords until late Tuesday evening.

The fabulous web comic, xkcd, explains how the attack works in layman’s terms. If you’re interested in the real nitty gritty of this vulnerability, though, there’s an excellent write-up on the IOActive Labs blog. If you’re the type that likes to play, you can find proof-of-concept code here. And let’s not forget about the client side, there’s PoC code for that as well.

OpenSSL versions 1.0.1 through 1.0.1f as well as the 1.0.2 beta code are affected. The folks at OpenSSL released version 1.0.1g on Monday which fixed the problem. Or, at least, the current problem. There’s a bit of chatter about other issues that may be lurking in the OpenSSL codebase.

Now that a few days have passed, however, what remains to be done? After all, everyone has patched their servers, right? Merely patching doesn’t make the problem disappear, though. Vulnerable code is out there and mistakes can be made. For the foreseeable future, you should be regularly scanning your network for vulnerable systems with something like Nmap. The Nmap NSE for Heartbleed scanning is already available. Alternatively, you can use something like Nagios to regularly check your existing servers.

Patching immediately may not have prevented a breach, either. Since Heartbleed doesn’t leave much of a trace beyond some oddities that your IDS may have seen, there’s virtually no way to know if anything has been taken. The best way to deal with this is to just go ahead and assume that your private keys are compromised and start replacing them. New keys, new certs. It’s painful, it’s slow, but it’s necessary.

For end users, the best thing you can do is change your passwords. I’m not aware of any “big” websites that have not patched by now, so changing passwords should be relatively safe. However, that said, Wired and Engadget have some of the best advice I’ve seen about this. In short, change your passwords today, then change them again in a few weeks. If you’re really paranoid, change them a third time in about a month. By that time, any site that is going to patch will have already patched.

Unfortunately, I think the fun is just beginning. I expect we’ll start seeing a number of related attacks. Phishing attacks are the most likely in the beginning. If private keys were compromised, then attackers can potentially impersonate websites, including their SSL certificates. This would likely involve a DNS poisoning attack, but could also be accomplished by compromising a user’s local system and setting a hosts file entry. Certificate revocation is a potential defense against this, but since many browsers have CRL checks disabled by default, it probably won’t help. Users will have to watch what they click, where they go, and what software they run. Not much different from the advice given already.

Another possible source of threats are consumer devices. As Bruce Schneier put it, “An upgrade path that involves the trash, a visit to Best Buy, and a credit card isn’t going to be fun for anyone.” What he’s referring to are the many embedded devices we use on a daily basis that may never receive updates to protect the end user. In other words, that router you purchased from the discount store? That may be affected and unless you replace it, you’ll continue to be vulnerable. Fortunately, most of these devices aren’t configured, by default, to face the Internet, so there may yet be hope.

The Heartbleed vulnerability is a serious contender for the worst security vulnerability ever released. I’m not sure of another vulnerability that exposes so many systems to such a degree as this one. Network and systems administrators will be cleaning up after this one for a while.

Becoming your own CA

SSL, as I mentioned in a previous blog entry, has some issues when it comes to trust. But regardless of the problems with SSL, it is a necessary part of the security toolchain. In certain situations, however, it is possible to overcome these trust issues.

Commercial providers are not the only entities that are capable of being a Certificate Authority. In fact, anyone can become a CA and the tools to do so are available for free. Becoming your own CA is a fairly painless process, though you might want to brush up on your openSSL skills. And lest you think you can just start signing certificates and selling them to third parties, it’s not quite that simple. The well-known certificate authorities have worked with browser vendors to have their root certificates added as part of the browser installation process. You’ll have to convince the browser vendors that they need to add your root certificate as well. Good luck.

Having your own CA provides you the means to import your own root certificate into your browser and use it to validate certificates you use within your network. You can use these SSL certificates for more than just websites as well. LDAP, RADIUS, SMTP, and other common applications use standard SSL certificates for encrypting traffic and validating remote connections. But as mentioned above, be aware that unless a remote user has a copy of your root certificate, they will be unable to validate the authenticity of your signed certificates.

Using certificates signed by your own CA can provide you that extra trust level you may be seeking. Perhaps you configured your mail server to use your certificate for the POP and IMAP protocols. This makes it more difficult for an attacker to masquerade as either of those services without obtaining your signing certificate so they can create their own. This is especially true if you configure your mail client such that your root certificate is the only certificate that can be used for validation.

Using your own signed certificates for internal, non-public facing services provides an even better use-case. Attacks such as DNS cache poisoning make it possible for attackers to trick devices into using the wrong address for an intended destination. If these services are configured to only use your certificates and reject connection attempts from peers with invalid certificates, then attackers will only be able to impersonate the destination if they can somehow obtain a valid certificate signed by your signing certificate.

Sound good? Well, how do we go about creating our own root certificate and all the various machinery necessary to make this work? Fortunately, all of the necessary tools are open-source and part of most Linux distributions. For the purposes of this blog post, I will be explaining how this is accomplished using the CentOS 6.x Linux distribution. I will also endeavor to break down each command and explain what each parameter does. Much of this information can be found in the man pages for the various commands.

OpenSSL is installed as part of a base CentOS install. Included in the install is a directory structure in /etc/pki. All of the necessary tools and configuration files are located in this directory structure, so instead of reinventing the wheel, we’ll use the existing setup.

To get started, edit the default openssl.cnf configuration file. You can find this file in /etc/pki/tls. There are a few options you want to change from their defaults. Search for the following headers and change the options listed within.

[CA_default]
default_md = sha256

[req]
default_bits = 4096
default_md = sha256
  • default_md : This option defined the default message digest to use. Switching this to sha256 result in a stronger message digest being used.
  • default_bits : This option defines the default key size. 2048 is generally considered a minimum these days. I recommend setting this to 4096.

Once the openssl.cnf file is set up, the rest of the process is painless. First, switch into the correct directory.

cd /etc/pki/tls/misc

Next, use the CA command to create a new CA.

[root@localhost misc]# ./CA -newca
CA certificate filename (or enter to create)

Making CA certificate ...
Generating a 4096 bit RSA private key
 ...................................................................................................................................................................................................................................................++
.......................................................................++
writing new private key to '/etc/pki/CA/private/./cakey.pem'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:US
State or Province Name (full name) []:MyState
Locality Name (eg, city) [Default City]:MyCity
Organization Name (eg, company) [Default Company Ltd]:My Company Inc.
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []:cert.example.com
Email Address []:certadmin@example.com

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
Using configuration from /etc/pki/tls/openssl.cnf
Enter pass phrase for /etc/pki/CA/private/./cakey.pem:
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 17886042129551798347 (0xf837fc8d719b304b)
        Validity
            Not Before: Feb 13 18:37:14 2014 GMT
            Not After : Feb 12 18:37:14 2017 GMT
        Subject:
            countryName               = US
            stateOrProvinceName       = MyState
            organizationName          = My Company Inc.
            commonName                = cert.example.com
            emailAddress              = certadmin@example.com
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                14:FC:14:BC:F4:A5:3E:6B:0C:58:3B:DF:3B:26:35:46:A0:BE:EC:F1
            X509v3 Authority Key Identifier:
                keyid:14:FC:14:BC:F4:A5:3E:6B:0C:58:3B:DF:3B:26:35:46:A0:BE:EC:F1

            X509v3 Basic Constraints:
                CA:TRUE
Certificate is to be certified until Feb 12 18:37:14 2017 GMT (1095 days)

Write out database with 1 new entries
Data Base Updated

And that’s about it. The root certificate is located in /etc/pki/CA/cacert.pem. This file can be made public without compromising the security of your system. This is the same certificate you’ll want to import into your browser, email client, etc. in order to validate and certificates you may sign.

Now you can start signing certificates. First you’ll need to create a CSR on the server you want to install it on. The following command creates both the private key and the CSR for you. I recommend using the server name as the name of the CSR and the key.

openssl req -newkey rsa:4096 -keyout www.example.com.key -out www.example.com.csr
  • openssl : The openSSL command itself
  • req : This option tells openSSL that we are performing a certificate signing request (CSR) operation.
  • -newkey : This option creates a new certificate request and a new private key. It will prompt the user for the relevant field values. The rsa:4096 argument indicates that we want to use the RSA algorithm with a key size of 4096 bits.
  • -keyout : This gives the filename to write the newly created private key to.
  • -out : This specifies the output filename to write to.
[root@localhost misc]# openssl req -newkey rsa:4096 -keyout www.example.com.key -out www.example.com.csr Generating a 4096 bit RSA private key
.....................................................................................................................++
..........................................................................................................................................................................................................................................................................................................................................................................................................++
writing new private key to 'www.example.com.key'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:US
State or Province Name (full name) []:MyState
Locality Name (eg, city) [Default City]:MyCity
Organization Name (eg, company) [Default Company Ltd]:My Company Inc.
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []:www.example.com
Email Address []:hostmaster@example.com

Once you have the CSR, copy it over to the server you’re using to sign certificates. Unfortunately, the existing tools don’t make it easy to merely name the CSR you’re trying to sign, so we need to create our own tool. First, create a new directory to put the CSRs in.

mkdir /etc/pki/tls/csr

Next, create the sign_cert.sh script in the directory we just created. This file needs to be executable.

#!/bin/sh

# Revoke last year's certificate first :
# openssl ca -revoke cert.crt

DOMAIN=$1
YEAR=`date +%Y`
rm -f newreq.pem
ln -s $DOMAIN.csr newreq.pem
/etc/pki/tls/misc/CA -sign
mv newcert.pem $DOMAIN.$YEAR.crt

That’s all you need to start signing certificates. Place the CSR you transferred from the other server into the csr directory and use script we just created to sign it.

[root@localhost csr]# ./sign_cert.sh www.example.com
Using configuration from /etc/pki/tls/openssl.cnf
Enter pass phrase for /etc/pki/CA/private/cakey.pem:
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 17886042129551798348 (0xf837fc8d719b304c)
        Validity
            Not Before: Feb 13 18:48:55 2014 GMT
            Not After : Feb 13 18:48:55 2015 GMT
        Subject:
            countryName = US
            stateOrProvinceName = MyState
            localityName = MyCity
            organizationName = My Company Inc.
            commonName = www.example.com
            emailAddress = hostmaster@example.com
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            Netscape Comment:
                OpenSSL Generated Certificate
            X509v3 Subject Key Identifier:
                3A:EE:2B:3A:73:A6:C3:5C:39:90:EA:85:3F:DA:71:33:7B:91:4D:7F
            X509v3 Authority Key Identifier:
                keyid:14:FC:14:BC:F4:A5:3E:6B:0C:58:3B:DF:3B:26:35:46:A0:BE:EC:F1

Certificate is to be certified until Feb 13 18:48:55 2015 GMT (365 days)
Sign the certificate? [y/n]:y

1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 17886042129551798348 (0xf837fc8d719b304c)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, ST=MyState, O=My Company Inc., CN=cert.example.com/emailAddress=certadmin@example.com
        Validity
            Not Before: Feb 13 18:48:55 2014 GMT
            Not After : Feb 13 18:48:55 2015 GMT
    Subject: C=US, ST=MyState, L=MyCity, O=My Company Inc., CN=www.example.com/emailAddress=hostmaster@example.com
    Subject Public Key Info:
        Public Key Algorithm: rsaEncryption
            Public-Key: (4096 bit)
            Modulus:
                00:d9:5a:cc:87:f0:e5:1e:6f:a0:25:cd:fe:36:64:
                6c:68:ae:2f:3e:7e:93:93:a4:69:6f:f1:28:c1:c2:
                4d:5f:3c:3a:61:2e:4e:f0:90:89:54:48:d6:03:83:
                fb:ac:1e:7c:9a:e8:be:cf:c9:8f:93:41:27:3e:1b:
                66:63:db:a1:54:cb:f7:1d:0b:71:bc:5f:80:e1:30:
                e4:28:14:68:1c:09:ba:d0:aa:d3:e6:2b:24:cd:21:
                67:99:dc:8b:7a:2c:94:d0:ed:8e:02:5f:2f:52:06:
                09:0e:8a:b7:bf:64:e8:d7:bf:94:94:ad:80:34:57:
                32:89:51:00:fe:fd:8c:7d:17:35:4c:c7:5f:5b:58:
                f4:97:9b:21:42:9e:a9:6c:86:5f:f4:35:98:a5:81:
                62:9d:fa:15:07:9d:29:25:38:2b:5d:22:74:58:f8:
                58:56:1c:e9:65:a3:62:b5:a7:66:17:95:12:21:ca:
                82:12:90:b6:8a:8d:1f:79:e8:5c:f4:f9:6c:3a:44:
                f9:3a:3f:29:0d:2e:bf:51:98:9f:58:21:e5:d9:ee:
                78:54:ad:5a:a2:6f:d1:85:9a:bc:b9:21:92:e8:76:
                80:b8:0f:96:77:9a:99:5e:3b:06:bb:6f:da:1c:6e:
                f2:10:16:69:ba:2b:57:c8:1a:cc:b6:e4:0c:1d:b2:
                a6:b7:b9:6c:37:2e:80:13:46:a1:46:c3:ca:d6:2b:
                cd:f7:ba:38:98:74:15:7f:f1:67:03:8e:24:89:96:
                55:31:eb:d8:44:54:a5:11:04:59:e6:73:59:42:ed:
                aa:a3:37:13:ab:63:ab:ef:61:65:0a:af:2f:71:91:
                23:40:7d:f8:e8:a1:9d:cf:3f:e5:33:d9:5f:d2:4d:
                06:d0:2c:70:59:63:06:0f:2a:59:ae:ae:12:8d:f4:
                6c:fd:b2:33:76:e8:34:0f:1f:24:91:2a:a8:aa:1b:
                11:8a:0b:86:f3:67:b8:be:b7:a0:06:02:4a:76:ef:
                dd:ed:c4:a9:03:a1:8c:b0:39:9d:35:98:7f:04:1c:
                24:8a:1c:7c:6f:35:56:71:ee:b5:36:b7:3f:14:04:
                eb:48:a1:4f:6f:8e:43:7c:8b:36:4a:bf:ba:e9:8b:
                d9:38:0c:76:24:e9:a3:38:bf:4e:86:fd:31:4d:c3:
                6f:16:07:09:dd:d8:6b:0b:9d:4d:97:eb:1f:92:21:
                b2:a5:f9:d8:55:61:85:d2:99:97:bc:27:12:be:eb:
                55:86:ee:1f:f5:6f:a7:c5:64:2f:4e:c2:67:a3:52:
                97:7a:d9:66:89:05:6a:59:ed:69:7b:22:10:2b:a1:
                14:4e:5d:b8:f0:21:e9:11:d0:25:ae:bc:05:2b:c3:
                db:ad:cf
            Exponent: 65537 (0x10001)
    X509v3 extensions:
        X509v3 Basic Constraints:
            CA:FALSE
        Netscape Comment:
            OpenSSL Generated Certificate
        X509v3 Subject Key Identifier:
            3A:EE:2B:3A:73:A6:C3:5C:39:90:EA:85:3F:DA:71:33:7B:91:4D:7F
        X509v3 Authority Key Identifier:
            keyid:14:FC:14:BC:F4:A5:3E:6B:0C:58:3B:DF:3B:26:35:46:A0:BE:EC:F1

Signature Algorithm: sha256WithRSAEncryption
     ca:66:b2:55:64:e6:40:a5:85:19:11:66:0d:63:89:fb:0d:3a:
     0c:ec:fd:cb:5c:93:44:1e:3f:1b:ca:f5:3d:85:ab:0a:0b:dc:
     f3:18:1d:1f:ec:85:ff:f3:82:52:9e:c7:12:19:07:e9:6a:82:
     bd:32:f6:d1:19:b2:b7:09:1c:34:d7:89:45:7e:51:4d:42:d6:
     4e:78:b6:39:b3:76:58:f8:20:57:b3:d8:7b:e0:b3:2f:ce:9f:
     a2:59:de:f6:31:f2:09:1c:91:3b:7f:97:61:cb:11:a4:b4:73:
     ab:47:64:e8:93:07:98:d5:47:75:8d:9a:8f:a3:8f:e8:f4:42:
     7e:b8:1b:e8:36:72:13:93:f9:a8:cc:6d:b4:85:a7:af:94:fe:
     f3:6e:76:c2:4d:78:c3:c2:0b:a4:48:27:d3:eb:52:c3:46:14:
     c1:26:03:28:a0:53:c7:db:59:c9:95:b8:d9:f0:d9:a8:19:4a:
     a7:0f:81:ad:3c:e1:ec:f2:21:51:0d:bc:f9:f9:f6:b6:75:02:
     9f:43:de:e6:2f:9b:77:d3:c3:72:6f:f6:18:d7:a3:43:91:d2:
     04:2a:c8:bf:67:23:35:b7:41:3f:d1:63:fe:dc:53:a7:26:e9:
     f4:ee:3b:96:d5:2a:9c:6d:05:3d:27:6e:57:2f:c9:dc:12:06:
     2c:cf:0c:1b:09:62:5c:50:82:77:6b:5c:89:32:86:6b:26:30:
     d2:6e:33:20:fc:a6:be:5a:f0:16:1a:9d:b7:e0:d5:d7:bb:d8:
     35:57:d2:be:d5:07:98:b7:3c:18:38:f9:94:4c:26:3a:fe:f2:
     ad:40:e6:95:ef:4b:a9:df:b0:06:87:a2:6c:f2:6a:03:85:3b:
     97:a7:ef:e6:e5:d9:c3:57:87:09:06:ae:8a:5a:63:26:b9:35:
     29:a5:87:4b:7b:08:b9:63:1c:c3:65:7e:97:ae:79:79:ed:c3:
     a3:36:c3:87:1f:54:fe:0a:f1:1a:c1:71:3d:bc:9e:36:fc:da:
     03:2b:61:b5:19:0c:d7:4d:19:37:61:45:91:4c:c9:7a:5b:00:
     cd:c2:2d:36:f9:1f:c2:b1:97:2b:78:86:aa:75:0f:0a:7f:04:
     85:81:c5:8b:be:af:a6:a7:7a:d2:17:26:7a:86:0d:f8:fe:c0:
     27:a8:66:c7:92:cd:c5:34:99:c9:8e:c1:25:f3:98:df:4e:48:
     37:4a:ee:76:4a:fa:e4:66:b4:1f:cd:d8:e0:25:fd:c7:0b:b3:
     12:af:bb:b7:29:98:5e:86:f2:12:8e:20:c6:a9:40:6f:39:14:
     8b:71:9f:98:22:a0:5b:57:d1:f1:88:7d:86:ad:19:04:7b:7d:
     ee:f2:c9:87:f4:ca:06:07
-----BEGIN CERTIFICATE-----
MIIGBDCCA+ygAwIBAgIJAPg3/I1xmzBMMA0GCSqGSIb3DQEBCwUAMHoxCzAJBgNV
BAYTAlVTMRAwDgYDVQQIDAdNeVN0YXRlMRgwFgYDVQQKDA9NeSBDb21wYW55IElu
Yy4xGTAXBgNVBAMMEGNlcnQuZXhhbXBsZS5jb20xJDAiBgkqhkiG9w0BCQEWFWNl
cnRhZG1pbkBleGFtcGxlLmNvbTAeFw0xNDAyMTMxODQ4NTVaFw0xNTAyMTMxODQ4
NTVaMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHTXlTdGF0ZTEPMA0GA1UEBwwG
TXlDaXR5MRgwFgYDVQQKDA9NeSBDb21wYW55IEluYy4xGDAWBgNVBAMMD3d3dy5l
eGFtcGxlLmNvbTElMCMGCSqGSIb3DQEJARYWaG9zdG1hc3RlckBleGFtcGxlLmNv
bTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBANlazIfw5R5voCXN/jZk
bGiuLz5+k5OkaW/xKMHCTV88OmEuTvCQiVRI1gOD+6wefJrovs/Jj5NBJz4bZmPb
oVTL9x0LcbxfgOEw5CgUaBwJutCq0+YrJM0hZ5nci3oslNDtjgJfL1IGCQ6Kt79k
6Ne/lJStgDRXMolRAP79jH0XNUzHX1tY9JebIUKeqWyGX/Q1mKWBYp36FQedKSU4
K10idFj4WFYc6WWjYrWnZheVEiHKghKQtoqNH3noXPT5bDpE+To/KQ0uv1GYn1gh
5dnueFStWqJv0YWavLkhkuh2gLgPlneamV47Brtv2hxu8hAWaborV8gazLbkDB2y
pre5bDcugBNGoUbDytYrzfe6OJh0FX/xZwOOJImWVTHr2ERUpREEWeZzWULtqqM3
E6tjq+9hZQqvL3GRI0B9+Oihnc8/5TPZX9JNBtAscFljBg8qWa6uEo30bP2yM3bo
NA8fJJEqqKobEYoLhvNnuL63oAYCSnbv3e3EqQOhjLA5nTWYfwQcJIocfG81VnHu
tTa3PxQE60ihT2+OQ3yLNkq/uumL2TgMdiTpozi/Tob9MU3DbxYHCd3YawudTZfr
H5IhsqX52FVhhdKZl7wnEr7rVYbuH/Vvp8VkL07CZ6NSl3rZZokFalntaXsiECuh
FE5duPAh6RHQJa68BSvD263PAgMBAAGjezB5MAkGA1UdEwQCMAAwLAYJYIZIAYb4
QgENBB8WHU9wZW5TU0wgR2VuZXJhdGVkIENlcnRpZmljYXRlMB0GA1UdDgQWBBQ6
7is6c6bDXDmQ6oU/2nEze5FNfzAfBgNVHSMEGDAWgBQU/BS89KU+awxYO987JjVG
oL7s8TANBgkqhkiG9w0BAQsFAAOCAgEAymayVWTmQKWFGRFmDWOJ+w06DOz9y1yT
RB4/G8r1PYWrCgvc8xgdH+yF//OCUp7HEhkH6WqCvTL20RmytwkcNNeJRX5RTULW
Tni2ObN2WPggV7PYe+CzL86folne9jHyCRyRO3+XYcsRpLRzq0dk6JMHmNVHdY2a
j6OP6PRCfrgb6DZyE5P5qMxttIWnr5T+8252wk14w8ILpEgn0+tSw0YUwSYDKKBT
x9tZyZW42fDZqBlKpw+BrTzh7PIhUQ28+fn2tnUCn0Pe5i+bd9PDcm/2GNejQ5HS
BCrIv2cjNbdBP9Fj/txTpybp9O47ltUqnG0FPSduVy/J3BIGLM8MGwliXFCCd2tc
iTKGayYw0m4zIPymvlrwFhqdt+DV17vYNVfSvtUHmLc8GDj5lEwmOv7yrUDmle9L
qd+wBoeibPJqA4U7l6fv5uXZw1eHCQauilpjJrk1KaWHS3sIuWMcw2V+l655ee3D
ozbDhx9U/grxGsFxPbyeNvzaAythtRkM100ZN2FFkUzJelsAzcItNvkfwrGXK3iG
qnUPCn8EhYHFi76vpqd60hcmeoYN+P7AJ6hmx5LNxTSZyY7BJfOY305IN0rudkr6
5Ga0H83Y4CX9xwuzEq+7tymYXobyEo4gxqlAbzkUi3GfmCKgW1fR8Yh9hq0ZBHt9
7vLJh/TKBgc=
-----END CERTIFICATE-----
Signed certificate is in newcert.pem

The script automatically renamed the newly signed certificate. In the above example, the signed certificate is in www.example.com.2014.crt. Transfer this file back to the server it belongs on and you’re all set to start using it.

That’s it! You’re now a certificate authority with the power to sign your own certificates. Don’t let all that power go to your head!

SSL “Security”

SSL, a cryptographically secure protocol, was created by Netscape in the mid-1990’s. Today, SSL, and it’s replacement, TLS, are used by web browsers and other programs to create secure connections between devices across the Internet.

SSL provides the means to cryptographically secure a tunnel between endpoints, but there is another aspect of security that is missing. Trust. While a user may be confident that the data received from the other end of the SSL tunnel was sent by the remote system, the user can not be confident that the remote system is the system it claims to be. This problem was partially solved through the use of a Public Key Infrastructure, or PKI.

PKI, in a nutshell, provides the trust structure needed to make SSL secure. Certificates are issued by a certificate authority or CA. The CA cryptographically signs the certificate, enabling anyone to verify that the certificate was issued by the CA. Other PKI constructs offer validation of the registrant, indexing of the public keys, and a key revocation system. It is within these other constructs that the problems begin.

When SSL certificates were first offered for sale, the CAs spent a great deal of time and energy verifying the identity of the registrant. Often, paper copies of the proof had to be sent to the CA before a certificate would be issued. The process could take several days. More recently, the bar for entry has been lowered significantly. Certificates are now issued on an automated process requiring only that the registrant click on a link sent to one of the email addresses listed in the Whois information. This lack of thorough verification has significantly eroded the trust a user can place in the authenticity of a certificate.

CAs have responded to this problem by offering different levels of SSL certificates. Entry level certificates are verified automatically via the click of a link. Higher level SSL certificates have additional identity verification steps. And at the highest level, the Extended Validation, or EV certificate requires a thorough verification of the registrants identity. Often, these different levels of SSL certificates are marketed as stronger levels of encryption. The reality, however, is that the level of encryption for each of these certificates is exactly the same. The only difference is the amount of verification performed by the CA.

Despite the extra level of verification, these certificates are almost indistinguishable from one another. With the exception of EV certificates, the only noticeable difference between differing levels of SSL certificates are the identity details obtained before the certificate is issued. An EV certificate, on the other hand, can only be obtained from certain vendors, and shows up in a web browser with a special green overlay. The intent here seems to be that websites with EV certificates can be trusted more because the identity of the organization running the website was more thoroughly validated.

In the end, though, trust is the ultimate issue. Users have been trained to just trust a website with an SSL certificate. And trust sites with EV certificates even more. In fact, there have been a number of marketing campaigns targeted at convincing users that the “Green Address Bar” means that the website is completely trustworthy. And they’ve been pretty effective. But, as with most marketing, they didn’t quite tell the truth. sure, the EV certificate may mean that the site is more trustworthy, but it’s still possible that the certificate is fake.

There have been a number of well known CAs that have been compromised in recent years. Diginotar and Comodo being two of the more high profile ones. In both cases, it became possible for rogue certificates to be created for any website the attacker wanted to hijack. That certificate plus some creative DNS poisoning and the attacker suddenly looks like your bank, or google, or whatever site the attacker wants to be. And, they’ll have a nice shiny green EV certificate.

So how do we fix this? Well, one way would be to use the certificate revocation system that already exists within the PKI infrastructure. If a certificate is stolen, or a false certificate is created, the CA has the ability to put the signature for that certificate into the revocation system. When a user tries to load a site with a bad certificate, a warning is displayed telling the user that the certificate is not to be trusted.

Checking revocation of a certificate takes time, and what happens if the revocation server is down? Should the browser let the user go to the site anyway? Or should it block by default? The more secure option is to block, of course, but most users won’t understand what’s going on. So most browser manufacturers have either disabled revocation checking completely, or they default to allowing a user to access the site when the revocation site is slow or unavailable.

Without the ability to verify if a certificate is valid or not, there can be no real trust in the security of the connection, and that’s a problem. Perhaps one way to fix this problem is to disconnect the revocation process from the process of loading the webpage. If the revocation check happened in parallel to the page loading, it shouldn’t interfere with the speed of the page load. Additional controls can be put into place to prevent any data from being sent to the remote site without a warning until the revocation check completes. In this manner, the revocation check can take a few seconds to complete without impeding the use of the site. And after the first page load, the revocation information is cached anyway, so subsequent page loads are unaffected.

Another option, floated by the browser builders themselves, is to have the browser vendors host the revocation information. This information is then passed on to the browsers when they’re loaded. This way the revocation process can be handled outside of the CAs, handling situations such as those caused by a CA being compromised. Another idea would be to use short term certificates that expire quickly, dropping the need for revocation checks entirely.

It’s unclear as to what direction the market will move with this issue. It has been over two years since the attacks on Diginotar and Comodo and the immediacy of this problem seems to have passed. At the moment, the only real fix for this is user education. But with the marketing departments for SSL vendors working to convince users of the security of SSL, this seems unlikely.

BSides Delaware 2013

The annual BSides Delaware conference took place this past weekend, November 8th and 9th. BSides Delaware is a free community driven security event that takes place at the Wilmington University New Castle campus. The community is quite open, welcoming seasoned professionals, newcomers, curious individuals, and even children. There were a number of families who attended, bringing their children with them to learn and have fun.

I was fortunate enough to be able to speak at last years BSides and was part of the staff for this years event. There were two tracks for talks, many of which were recorded and are already online thanks to Adrian Crenshaw, the IronGeek. Adrian has honed his video skills and was able to have every recording online by the closing ceremonies on Saturday evening.

In all there were more than 25 talks over the course of two days covering a wide variety of topics, logging, Bitcoins, forensics, and more. While most speakers were established security professionals, there were a few new speakers striving to make a name for themselves.

This year also included a FREE wireless essentials training class. The class was taught by a team of world-class instructors including Mike Kershaw (drag0rn), author of the immensely popular Kismet wireless tool, Russell Handorf from the FBI Cyber Squad, and Rick Farina, lead developer for Pentoo. The class covered everything from wireless basics to software-defined radio hacking. An absolutely amazing class.

In addition to the talks, BSides also features not one, but two lockpick villages. Both Digital Trust as well as Toool were present. The lockpick villages were a big hit with seasoned professionals as well as the very young. It’s amazing to see how adept a young child can be with a lockpick.

Hackers for Charity was present as well with a table of goodies for sale. They also held a silent (and not so silent) auction where all proceeds went to the charity. Hackers for Charity raises money to help with a variety of projects they engage in across the world. From their website :

We employ volunteer hackers and technologists through our Volunteer Network and engage their skills in short projects designed to help charities that can not afford traditional technical resources.

…

We’ve personally witnessed how one person can have a profound impact on the world. By giving of their skills, time and talent our volunteers are profoundly impacting the world, one “hacker” at a time.

BSides 2013 was an amazing experience. This was my second year at the conference and it’s amazing how it has grown. The dates for BSidesDE 2014 have already been announced, November 14th and 15th. Mark your calendars and make an effort to come join in the fun. It’s worth it.