Musings, ramblings, rants…
*cough* *cough*
Awfully dusty in here. Almost as if this place were abandoned. Of course, that was never the case, was it. Just a hiatus of sorts. A reprieve from the noise and the harshness of reality.
But it’s time, now. Time to whip this place back into shape. Time to put the pieces back together. Time to build something new and interesting.
I know it’s been a while, but it’s time to get back in the habit. I’ve learned a lot these past years and I want to start sharing it. Soon.
Let’s talk about your house for a moment. For the sake of argument, we’ll assume that you live in a nice house with doors, windows, the works. All of the various entries have the requisite locking devices. As with most homes, these help prevent unwanted entry, though a determined attacker can surely bypass them. For the moment, let’s ignore the determined attacker and just talk about casual attempts.
Throughout your time living in your home, casual attempts at illegal entry have been rebuffed. You may or may not even know about these attempts. They happen pretty randomly, but there’s typically not much in the way of evidence after the attacker gives up and leaves. So you’re pretty happy with how secure things are.
Recently, you’ve heard about this great new garage from a friend who has one. It’s really nice, low cost, and you have room for it on your property, so you decide to purchase one. You place the order and, after a few days, your new garage arrives. It’s everything you could have imagined. Plenty of room to store all the junk you have in the house, plus you can fit the car in there too!
You use the garage every day, moving boxes in and out of the garage as needed until one day you return home and, for some inexplicable reason, your car won’t fit all the way in. Well, that’s pretty weird, you think. You decide that maybe you stored too much in the garage, so you spend the rest of the day cleaning out the garage. You make some tough decisions and eventually you make enough room to put the car back in the garage.
Time passes and this happens a few more times. After a while you start to get a bit frustrated and decide that maybe you need to buy a bigger garage. You pull out your trusty measuring tape to verify the dimensions of the garage and, to your amazement, the garage is smaller than what you remember. You do some more checking and, to your amazement, the garage is bigger on the outside. So you call an expert to figure out what’s going on.
When the expert arrives, she takes one look at the situation and tells you she knows exactly what has happened. You watch with awe as she walks up to the closed garage, places her hand on the door, and the door opens by itself! Curious, you ask how she performed that little magic trick. She explains that this particular model of garage has a little known problem that allows the door to be opened by putting pressure on just the right place. Next, she head into the garage and starts poking around at the walls. After a few moments, one of the walls slides open revealing another room full of stuff you don’t recognize.
Your expert explains that obviously someone else knows about this weakness and has set up a false wall in your garage to hide their own stuff in. This is the source of the shrinking space and your frustration. She helps you clean up the mess and tear down the false wall. After everything is back to normal, she recommends you contact the manufacturer and see if they have a fix for the faulty door.
While this story may sound pretty far fetched when we’re talking about houses and garages, it’s an all too common story for consumer grade appliances. And as we move further into this new age of connected devices, commonly called the Internet of Things (IoT), it’s going to become and even bigger issue.
Network access itself is the first challenge. Many of the major home router vendors have already experienced problems with security. So right out of the gate, home networks are potentially vulnerable. This is a major problem, especially given the potentially sensitive nature of data being transmitted by a variety of new IoT devices.
Today’s devices are incredibly data-centric. From fitness trackers to environmental sensors, our devices are tracking everything. This data is collected and then transmitted to an internet-connected service where it is made available to the user in a variety of ways. Some users may find this data to be sensitive, hoping to keep it relatively private, available only to the user and, anonymously, to the service they subscribe to. Others may make this data public. But in the world of IoT, a security problem with a device compromises that choice.
Or maybe the attacker isn’t after your data at all. Perhaps, like our garage example, they’re looking for resources they can use. Maybe they want to store files, or maybe they’re looking to use your device to process their own data. Years ago, attackers would gain access to a remote system so they could take advantage of the space available on the system, typically storing data and setting up a warez site. That is, illegal copies of software available to those who know where to look. These days, however, storage is everywhere and there are many superior ways to transmit files between users. As a result, the old-school practice of setting up a warez site has mostly fallen by the wayside.
In today’s world, attackers want access to your devices for a variety of reasons. Some attackers use these devices as zombie systems for sending massive amounts of spam. Typically this just results in a slow Internet connection and possibly gets your IP banned from sending mail. Not a big deal for you, but it can be a real headache for those of us dealing with the influx of spam.
More and more, however, attackers are taking over machines to use them for their processing power, or for their connection to the Internet. For instance, some attackers compromise machines just so they can use them to mine bitcoins. It seems harmless enough, but it can be an inconvenience to the owner of the device when it doesn’t respond the way it should because it’s too busy working on something else.
Attackers are also using the Internet connections for nefarious purposes such as setting up denial of service hosts. They use your connection, and the connections of other systems they have compromised, to send massive amounts of data to a remote system. The entire purpose of this activity it to prevent the remote system from being accessible. It was widely reported that this sort of activity is what caused connectivity problems to both Microsoft’s Xbox Live service as well as the Playstation Network during Christmas of 2014.
So what can we do about this? Users clearly want this technology, so we need to do something to make it more secure. And to be clear, this problem goes beyond the vendors, it includes the users as well. Software has and will always have bugs. Some of these bugs can be exploited and result in a security problem. So the first step is ensuring that vendors are patching those bugs when they’re found. And, perhaps, vendors can be convinced to bolster their internal security teams such that secure coding practices are followed.
But vendors patching bugs isn’t the only problem, and in most cases, it’s the easy part of the problem. Once a patch exists, users have to apply that patch to their system. As we’ve seen over the years, patching isn’t something that users are very good at. Thus, automatic update systems such as those used by Microsoft and Apple, are commonplace. But this practice hasn’t carried over to devices yet. Vendors need to work on this and build these features into their hardware. Until they do, these security issues will remain a widespread problem.
So yes, your iToaster needs security. And we need vendors to take the next step and bake in automatic updating so security becomes the default. End users want devices that work without having to worry about how and when to update them. Not all manufacturers have the marketing savvy that Apple uses to make updating sexy. Maybe they can take a page out of the book Microsoft used with the Xbox One. Silent updates, automatically, overnight.
Have you ever had to fix a broken item and you didn’t have the right parts? Instead of just giving up, you looked around and found something that would work for the time being. Occasionally, you come back later and fix it the right way, but more often than not, that fix stays in place indefinitely. Or, perhaps you’ve found a novel new use for a device. It wasn’t built for that purpose, but you figured out that it fit the exact use you had in mind.
Those are the actions of a hacker. No, really. If you look up the definition of a hacker, you get all sort of responses. Wikipedia has three separate entries for the word hacker in relation to technology :
Hacker – someone who seeks and exploits weaknesses in a computer system or computer network
Hacker – (someone) who makes innovative customizations or combinations of retail electronic and computer equipment
Hacker – (someone) who combines excellence, playfulness, cleverness and exploration in performed activities
Google defines it as follows :
1. a person who uses computers to gain unauthorized access to data.
(informal) an enthusiastic and skillful computer programmer or user.
2. a person or thing that hacks or cuts roughly.
And there are more. What’s interesting here is that depending on where you look, the word hacker means different things. It has become a pretty contentious word, mostly because the media has, over time, used it to describe the actions of a particular type of person. Specifically, hacker is often used to describe the criminal actions of a person who gains unauthorized access to computer systems. But make no mistake, the media is completely wrong on this and they’re using the word improperly.
Sure, the person who broke into that computer system and stole all of that data is most likely a hacker. But, first and foremost, that person is a criminal. Being a hacker is a lifestyle and, in many cases, a career choice. Much like being a lawyer or a doctor is a career choice. Why then is hacker used as a negative term to identify criminal activity and not doctor or lawyer? There are plenty of instances where doctors, lawyers, and people from a wide variety of professions have indulged in criminal activity.
Keren Elazari spoke in 2014 at TED about hackers, and their importance in our society. During her talk she discusses the role of hackers in our society, noting that there are hackers who use their skills for criminal activity, but many more who use their skills to better the world. From hacktivist groups like Anonymous to hackers like Barnaby Jack, these people have changed the world in positive ways, helping to identify weaknesses in systems to weaknesses in governments and laws. In her own words :
My years in the hacker world have made me realize both the problem and the beauty about hackers: They just can’t see something broken in the world and leave it be. They are compelled to either exploit it or try and change it, and so they find the vulnerable aspects in our rapidly changing world. They make us, they force us to fix things or demand something better, and I think we need them to do just that, because after all, it is not information that wants to be free, it’s us.
It’s time to stop letting the media use this word improperly. It’s time to take back what is ours. Hacker has long been a term used to describe those we look up to, those we seek to emulate. It is a term we hold dear, a term we seek to defend. When Loyd Blankenship was arrested in 1986, he wrote what has become known as the Hacker’s Manifesto. This document, often misunderstood, describes the struggle many of us went through, and the joy of discovering something we could call our own. Yes, we’re often misunderstood. Yes, we’ve been marginalized for a long time. But times have changed since then and our culture is strong and growing.
I’ve recently been reading Wired for War by P.W. Singer and one of the concepts he mentions in the book is Network Enhanced Telepathy. This struck me as not only something that sounds incredibly interesting, but something that we’ll probably see hit mainstream in the next 5-10 years.
According to Wikipedia, telepathy is “the purported transmission of information from one person to another without using any of our known sensory channels or physical interaction.“ In other words, you can think *at* someone and communicate. The concept that Singer talks about in the book isn’t quite as “mystical” since it uses technology to perform the heavy lifting. In this case, technology brings fantasy into reality.
Scientists have already developed methods to “read” thoughts from the human mind. These methods are by no means perfect, but they are a start. As we’ve seen with technology across the board from computers to robotics, electric cars to rockets, technological jumps may ramp up slowly, but then they rocket forward at a deafening pace. What seems like a trivial breakthrough at the moment may well lead to the next step in human evolution.
What Singer describes in the book is one step further. If we can read the human mind, and presumably write back to it, then adding a network in-between, allowing communication between minds, is obvious. Thus we have Network Enhanced Telepathy. And, of course, with that comes all of the baggage we associate with networks today. Everything from connectivity issues and lag to security problems.
The security issues associated with something like this range from inconvenient to downright horrifying. If you thought social engineering was bad, wait until we have a direct line straight into someone’s brain. Today, security issues can result in stolen data, denial of service issues, and, in some rare instances, destruction of property. These same issues may exist with this new technology as well.
Stolen data is pretty straightforward. Could an exploit allow an attacker to arbitrarily read data from someone’s mind? How would this work? Could they pinpoint the exact data they want, or would they only have access to the current “thoughts” being transmitted? While access to current thoughts might not be as bad as exact data, it’s still possible this could be used to steal important data such as passwords, secret information, etc. Pinpointing exact data could be absolutely devastating. Imagine, for a moment, what would happen if an attacker was able to pluck your innermost secrets straight out of your mind. Everyone has something to hide, whether that’s a deep dark secret, or maybe just the image of themselves in the bathroom mirror.
I’ve seen social engineering talks wherein the presenter talks about a technique to interrupt a person, mid-thought, and effectively create a buffer overflow of sorts, allowing the social engineer to insert their own directions. Taken to the next level, could an attacker perform a similar attack via a direct link to a person’s mind? If so, what access would the attacker then attain? Could we be looking at the next big thing in brainwashing? Merely insert the new programming, directly into the user.
How about Denial of Service attacks or physical destruction? Could an attacker cause physical damage in their target? Is a connection to the mind enough access to directly modify the cognitive functions of the target? Could an attacker induce something like Locked-In syndrome in a user? What about blocking specific functions, preventing the user from being able to move limbs, or speak? Since the brain performs regulatory control over the body, could an attacker modify the temperature, heart rate, or even induce sensations in their target? These are truly scary scenarios and warrant serious thought and discussion.
Technology is racing ahead at breakneck speeds and the future is an exciting one. These technologies could allow humans to take that next evolutionary step. But as with all technology, we should be looking at it with a critical eye. As technology and biology become more and more intertwined, it is essential that we tread carefully and be sure to address potential problems long before they become a reality.
The Cloud, hailed as a panacea for all your IT related problems. Need storage? Put it in the Cloud. Email? Cloud. Voice? Wireless? Logging? Security? The Cloud is your answer. The Cloud can do it all.
But what does that mean? How is it that all of these problems can be solved by merely signing up for various cloud services? What is the cloud, anyway?
Unfortunately, defining what the cloud actually is remains problematic. It means many things to many people. The cloud can be something “simple” like extra storage space or email. Google, Dropbox, and others offer a service that allows you to store files on their servers, making them available to you from “anywhere” in the world. Anywhere, of course, if the local government and laws allow you to access the services there. These services are often free for a small amount of space.
Google, Microsoft, Yahoo, and many, many others offer email services, many of them “free” for personal use. In this instance, though, free can be tricky. Google, for instance, has algorithms that “read” your email and display advertisements based on the results. So while you may not exchange money for this service, you do exchange a level of privacy.
Cloud can also be pure computing power. Virtual machines running a variety of operating systems, available for the end-user to access and run whatever software they need. Companies like Amazon have turned this into big business, offering a full range of back-end services for cloud-based servers. Databases, storage, raw computing power, it’s all there. In fact, they have developed APIs allowing additional services to be spun up on-demand, augmenting existing services.
As time goes on, more and more services are being added to the cloud model. The temptation to drop self-hosted services and move to the cloud is constantly increasing. The incentives are definitely there. Cloud services are affordable, and there’s no need for additional staff for support. All the benefits with very little of the expense. End-users have access to services they may not have had access to previously, and companies can save money and time by moving services they use to the cloud.
But as with any service, self-hosted or not, there are questions you should be asking. The answers, however, are sometimes a bit hard to get. But even without direct answers, there are some inferences you can make based on what the service is and what data is being transferred.
Data being accessible virtually anywhere, at any time, is one of major draws of cloud services. But there are downsides. What happens when the service is inaccessible? For a self-hosted service, you have control and can spend the necessary time to bring the service back up. In some cases, you may have the ability to access some or all of the data, even without the service being fully restored. When you surrender your data to the cloud, you are at the mercy of the service provider. Not all providers are created equal and you cannot expect uniform performance and availability across all providers. This means that in the event of an outage, you are essentially helpless. Keeping local backups is definitely an option, but oftentimes you’re using the cloud so that you don’t need those local backups.
Speaking of backups, is the cloud service you’re using responsible for backups? Will they guarantee that your data will remain safe? What happens if you accidentally delete a needed file or email? These are important issues that come up quite often for a typical office. What about the other side of the question? If the service is keeping backups, are those backups secure? Is there a way to delete data, permanently, from the service? Accidents happen, so if you’ve uploaded a file containing sensitive information, or sent/received an email with sensitive information, what recourse do you have? Dropbox keeps snapshots of all uploaded data for 30 days, but there doesn’t seem to be an official way to permanently delete a file. There are a number of articles out there claiming that this is possible, just follow the steps they provide, but can you be completely certain that the data is gone?
What about data security? Well, let’s think about the data you’re sending. For an email service, this is a fairly simple answer. Every email goes through that service. In fact, your email is stored on the remote server, and even deleted messages may hang around for a while. So if you’re using email for anything sensitive, the security of that information is mostly out of your control. There’s always the option of using some sort of encryption, but web-based services rarely support that. So data security is definitely an issue, and not necessarily an issue you have any control over. And remember, even the “big guys” make mistakes. Fishnet Security has an excellent list of questions you can ask cloud providers about their security stance.
Liability is an issue as well, though you may not initially realize it. Where, exactly, is your data stored? Do you know? Can you find out? This can be an important issue depending on what your industry is, or what you’re storing. If your data is being stored outside of your home country, it may be subject to the laws and regulations of the country it’s stored in.
There are a lot of aspects to deal with when thinking about cloud services. Before jumping into the fray, do your homework and make sure you’re comfortable with giving up control to a third party. Once you give up control, it may not be that easy to reign it back in.
I have been and always shall be your friend.
It’s a sad day. We’ve lost a dear friend today, someone we grew up with, someone so iconic that he inspired generations. At the age of 83, Leonard Nimoy passed away. He will be missed.
It’s amazing to realize how much someone you’ve never met can mean to you. People larger than life, people who will live on in memory forever. I’ve been continually moved for hours at the outpouring of grief and love online for Leonard. He has meant so much for so many, and his memory will live on forever.
Of all the souls I have encountered in my travels, his was the most… human.
In the past year we have seen several high-profile breaches of brick and mortar retailers. Estimates range in the tens of millions of credit cards stolen in each case. For the most part, these retailers have weathered the storm with virtually no ill effects. In fact, it seems the same increase in stock price that TJ Maxx saw after their breach still rings true today. A sad fact indeed.
Regardless, the recent slew of breaches has finally prompted the credit card industry to act. They have declared that 2015 will be the year that chip and pin becomes the standard for all card-present transactions. And while chip and pin isn’t a silver bullet, and attackers will eventually find new and innovative ways to circumvent it, it has proven to be quite effective in Europe where it has been the standard for years.
Chip and pin changes how the credit card information is transmitted to the processor. Instead of the credit card number being read, in plain text, off of the magnetic strip, the card reader initiates an encrypted communication between the chip on the card and the card reader. The card details are encrypted and sent, along with the user’s PIN, to the card processor for verification. It is this encrypted communication between the card and, ultimately, the card processor that results in increased security. In short, the attack vectors used in recent breaches is difficult, if not impossible to pull off with these new readers. Since the information is not decrypted until it hits the card processor, attackers can’t simply skim the information at the card reader. There are, of course, other attacks, though these have not yet proven widespread.
At it’s heart, though, chip and pin only “fixes” one type of credit card transaction, card-present transactions. That is, transactions in which the card holder physically scans their card via a card reader. The other type of transaction, card-not-present transactions, are unaffected by chip and pin. In fact, the move to chip and pin may result in putting online transactions at greater risk. With brick and mortar attacks gone, attackers will move to online retailers. Despite the standard SSL encryption used between shoppers and online retailers, there are plenty of ways to steal credit card data. In fact, one might argue that a single attack could net more card numbers in a shorter time since online retailers often store credit card data as a convenience for the user.
It seems that online fraud, though expected, is being largely ignored for the moment. After all, how are we going to protect that data without supplying card readers to every online shopper? Online solutions such as PayPal, Amazon Payments, and others mitigate this problem slightly, but we still have to rely on the security they’ve put in place to protect cardholder data. Other solutions such as Apple Pay and Google Wallet seemingly combine on and offline protections, but the central data warehouse remains. The problem seems to be the security of the card number itself. And losing this data can be a huge burden for many users as they have to systematically update payment information as the result of a possible breach. This can often lead to late payments, penalties, and more.
One possible alternative is to reduce the impact a single breach can cause. What if the data that retailers stored was of little or no value to an attacker while still allowing the retailer a way to simplify payments for the shopper? What if a breach at a retailer only affected that retailer and resulted in virtually no impact on the user? A solution like this may be just what we need.
Instead of providing a retailer your credit card number and CVV, the retailer is provided a simple token. That token, coupled with a private retailer-specific token should be all that is needed to verify a transaction. Tokens can and should be different for each retailer. If a retailer is compromised, new tokens can be generated, reducing the impact on the user significantly. Attackers who successfully breach a retailer can only submit transactions if they can obtain both the private retailer token as well as the user token. And if processors put simple access-control lists in place, it increases the difficulty an attacker encounters when trying to push through a fraudulent transaction.
Obtaining tokens can be handled by redirecting a user to a payment gateway for their initial transaction. The payment gateway verifies the user and their credit card data, and then passes the generated token back to the retailer. This is similar to how retailers using existing online payment processors such as Paypal and Amazon Payments already handle payments. The credit card data never passes through the retailer network. The number of locations credit card data is stored reduces significantly as well. This, in turn, means that attackers have fewer targets and while this increases the risk a payment processor network incurs, one can argue that these networks should already have significant defenses in place.
This is only one possible solution for online payments. There are many other solutions out there being presented by both security and non-security folks. But there seems to be no significant movement on an online solution. Will it take several high-profile online breaches to convince credit card companies that a solution is needed? Or will credit card companies move to protect retailers and card holders ahead of attackers redirecting their efforts? If history is any indication, get used to having your card re-issued several times a year for the foreseeable future.
Unless you’ve been living under a rock the past few days, you’ve probably heard about the Heartbleed vulnerability in OpenSSL that was disclosed on Monday, April 7th. Systems and network administrators across the globe have spent the last few days testing for this vulnerability, patching systems, and probably rocking in the corner while crying. Yes, it’s that bad. What’s more, there are a number of reports that intelligence agencies may have known about this vulnerability for some time now.
The quick and dirty is that a buffer overflow bug in the code allows an attacker to remotely read memory of an affected system in 64k chunks. The only memory accessible to an attacker would be memory used by the process being connected to, but, depending on the process, there may be a LOT of useful data in there. For instance, Yahoo was leaking usernames and passwords until late Tuesday evening.
The fabulous web comic, xkcd, explains how the attack works in layman’s terms. If you’re interested in the real nitty gritty of this vulnerability, though, there’s an excellent write-up on the IOActive Labs blog. If you’re the type that likes to play, you can find proof-of-concept code here. And let’s not forget about the client side, there’s PoC code for that as well.
OpenSSL versions 1.0.1 through 1.0.1f as well as the 1.0.2 beta code are affected. The folks at OpenSSL released version 1.0.1g on Monday which fixed the problem. Or, at least, the current problem. There’s a bit of chatter about other issues that may be lurking in the OpenSSL codebase.
Now that a few days have passed, however, what remains to be done? After all, everyone has patched their servers, right? Merely patching doesn’t make the problem disappear, though. Vulnerable code is out there and mistakes can be made. For the foreseeable future, you should be regularly scanning your network for vulnerable systems with something like Nmap. The Nmap NSE for Heartbleed scanning is already available. Alternatively, you can use something like Nagios to regularly check your existing servers.
Patching immediately may not have prevented a breach, either. Since Heartbleed doesn’t leave much of a trace beyond some oddities that your IDS may have seen, there’s virtually no way to know if anything has been taken. The best way to deal with this is to just go ahead and assume that your private keys are compromised and start replacing them. New keys, new certs. It’s painful, it’s slow, but it’s necessary.
For end users, the best thing you can do is change your passwords. I’m not aware of any “big” websites that have not patched by now, so changing passwords should be relatively safe. However, that said, Wired and Engadget have some of the best advice I’ve seen about this. In short, change your passwords today, then change them again in a few weeks. If you’re really paranoid, change them a third time in about a month. By that time, any site that is going to patch will have already patched.
Unfortunately, I think the fun is just beginning. I expect we’ll start seeing a number of related attacks. Phishing attacks are the most likely in the beginning. If private keys were compromised, then attackers can potentially impersonate websites, including their SSL certificates. This would likely involve a DNS poisoning attack, but could also be accomplished by compromising a user’s local system and setting a hosts file entry. Certificate revocation is a potential defense against this, but since many browsers have CRL checks disabled by default, it probably won’t help. Users will have to watch what they click, where they go, and what software they run. Not much different from the advice given already.
Another possible source of threats are consumer devices. As Bruce Schneier put it, “An upgrade path that involves the trash, a visit to Best Buy, and a credit card isn’t going to be fun for anyone.” What he’s referring to are the many embedded devices we use on a daily basis that may never receive updates to protect the end user. In other words, that router you purchased from the discount store? That may be affected and unless you replace it, you’ll continue to be vulnerable. Fortunately, most of these devices aren’t configured, by default, to face the Internet, so there may yet be hope.
The Heartbleed vulnerability is a serious contender for the worst security vulnerability ever released. I’m not sure of another vulnerability that exposes so many systems to such a degree as this one. Network and systems administrators will be cleaning up after this one for a while.
Let’s get this out of the way. One of the primary reasons I’m writing this is in response to a request by John Carmack for coherent commentary about the recent acquisition of Oculus VR by Facebook. My hope is that he does, in fact, read this and maybe drop a comment in response. <fanboy>Hi John!</fanboy> I’ve been a huge Carmack fan since the early ID days, so please excuse the fanboyism.
And I *just* saw the news that Michael Abrash has joined Oculus as well, which is also incredibly exciting. Abrash is an Assembly GOD. <Insert more fanboyism here />
Ok, on to the topic a hand. The Oculus Rift is a VR headset that got its public start with a Kickstarter campaign in September of 2012. It blew away it’s meager goal of $250,000 and raked in almost $2.5 Million. For a mere $275 and some patience, contributors would receive an unassembled prototype of the Oculus Rift. Toss in another $25 and you received an assembled version.
But what is the Oculus Rift? According to the Kickstarter campaign :
Oculus Rift is a new virtual reality (VR) headset designed specifically for video games that will change the way you think about gaming forever. With an incredibly wide field of view, high resolution display, and ultra-low latency head tracking, the Rift provides a truly immersive experience that allows you to step inside your favorite game and explore new worlds like never before.
In short, the Rift is the culmination of every VR lover’s dreams. Put a pair of these puppies on and magic appears before your eyes.
For myself, Rift was interesting, but probably not something I could ever use. Unfortunately, I suffer from Amblyopia, or Lazy Eye as it’s commonly called. I’m told I don’t see 3D. Going to 3D movies pretty much confirms this for me since nothing ever jumps out of the screen. So as cool as VR sounds to me, I would miss out on the 3D aspect. Though it might be possible to “tweak” the headset and adjust the angles a bit to force my eyes to see 3D. I’m not sure if that’s good for my eyes, though.
At any rate, the Rift sounds like an amazing piece of technology. In the past year I’ve watched a number of videos demonstrating the capabilities of the Rift. From the Hak5 crew to Ben Heck, the reviews have all been positive.
And then I learned that John Carmack joined Oculus. I think that was about the time I realized that Oculus was the real deal. John is a visionary in so many different ways. One can argue that modern 3D gaming is largely in part to the work he did in the field. In more recent years, his visions have aimed a bit higher with his rocket company, Armadillo Aerospace. Armadillo started winding down last year, right about the time that John joined Oculus, leaving him plenty of time to deep dive into a new venture.
For anyone paying attention, Oculus was recently acquired by Facebook for a mere $2 Billion. Since the announcement, I’ve seen a lot of hatred being tossed around on Twitter. Some of this hatred seems to be Kickstarter backers who are under some sort of delusion that makes them believe they have a say in anything they back. I see this a lot, especially when a project is taking longer than they believe it should.
I can easily write several blog posts on my personal views about this, but to sum it up quickly, if you back a project, you’re contributing to make something a reality. Sometimes that works, sometimes it doesn’t. But Kickstarter clearly states that you’re merely contributing financial backing, not gaining a stake in a potential product and/or company. Nor are you guaranteed to receive the perks you’ve contributed towards. So suck it up and get over it. You never had control to begin with.
I think Notch, of Minecraft fame, wrote a really good post about his feeling on the subject. I think he has his head right. He contributed, did his part, and though it’s not working out the way he wanted, he’s still willing to wish the venture luck. He may not want to play in that particular sandbox, but that’s his choice.
VR in a social setting is fairly interesting. In his first Oculus blog post, Michael Abrash mentioned reading Neal Stephenson’s incredible novel, Snow Crash. Snow Crash provided me with a view of what virtual reality might bring to daily life. Around the same time, the movie Lawnmower Man was released. Again, VR was brought into the forefront of my mind. But despite the promises of books and movies, VR remained elusive.
More recently, I read a novel by Ernest Cline, Ready Player One. Without giving too much away, the novel centers around a technology called the OASIS. Funnily enough, the OASIS is, effectively, a massive social network that users interact with via VR rigs. OASIS was the first thing I thought about when I heard about the Facebook / Oculus acquisition.
For myself, my concern is Facebook. Despite being a massively popular platform, I think users still distrust Facebook quite a bit. I lasted about 2 weeks on Facebook before having my account deleted. I understand their business model and I have no interest in taking part. Unfortunately, I’m starting to miss out on some aspects of Internet life since some sites are requiring Facebook accounts for access. Ah well, I guess they miss out on me as well.
I have a lot of distrust in Facebook at the moment. They wield an incredible amount of information about users and, to be honest, they’re nowhere near transparent enough for me to believe what they say. Google is slightly better, but there’s some distrust there as well. But more than just the distrust, I’m afraid that Facebook is going to take something amazing and destroy it in a backwards attempt to monetize it. I’m afraid that Facebook is the IOI of this story. (It’s a Ready Player One reference. Go read it, you can thank me later)
Ultimately, I have no stake in this particular game. At least, not yet, anyway. Maybe I’m wrong and Facebook makes all the right moves. Maybe they become a power for good and are able to bring VR to the masses. Maybe people like Carmack and Abrash can protect Oculus and fend off any fumbling attempts Facebook may make at clumsy monetization. I’m not sure how this will play out, only time will tell.
How will we know how things are going? Well, for one, watching his Facebook interacts with this new property will be pretty telling. I think if Facebook is able to sit in the shadows and watch rather than kicking in the front door and taking over, maybe Oculus will have a chance to thrive. Watching what products are ultimately released by Oculus will be another telling aspect. While I fully expect that Oculus will add some sort of Facebook integration into the SDK over time, I’m also hoping that they continue to provide an SDK for standalone applications.
I sincerely wish Carmack, Abrash, and the rest of the Oculus team the best. I think they’re in a position where they can make amazing things happen, and I’m eager to see what comes next.
I was debugging an odd network issue lately that turned out to have a pretty simple explanation. A client on the network was intermittently experiencing significant delays in accessing the network. Upon closer inspection, it turned out that prior to the delay, the client was being left idle for long periods of time. With this additional information it was pretty easy to identify that there was likely a connection between the client and server that was being torn down for being idle.
So in the end, the cause of the problem itself was pretty simple to identify. The fix, however, is more of a conundrum. The obvious answer is to adjust the timers and prevent the connection from being torn down. But what timers should be adjusted? There are the keepalive timers on the client, the keepalive timers on the server, and the idle teardown timers on the firewall in the middle.
TCP keepalive handling varies between operating systems. If we look at the three major operating systems, Linux, Windows, and OS X, then we can make the blanket statement that, by default, keepalives are sent after two hours of idle time. But, most firewalls seem to have a default TCP teardown timer of one hour. These defaults are not conducive to keeping idle connections alive.
The optimal scenario for timeouts is for the clients to have a keepalive timer that fires at an interval lower than that of the idle tcp timeout on the firewall. The actual values to use, as well as which devices should be changed, is up for debate. The firewall is clearly the easier point at which to make such a change. Typically there are very few firewall devices that would need to be updated as compared to the larger number of client devices. Additionally, there will likely be fewer firewalls added to the network over time, so ensuring that timers are properly set is much easier. On the other hand, the defaults that firewalls are generally configured with have been chosen specifically by the vendor for legitimate reasons. So perhaps the clients should conform to the setting on the firewall? What is the optimal solution?
And why would we want to allow idle connections anyway? After all, if a connection is idle, it’s not being used. Clearly, any application that needed a connection to remain open would send some sort of keepalive, right? Is there a valid reason to allow these sorts of connections for an extended period of time?
As it turns out, there are valid reasons for connections to remain active, but idle. For instance, database connections are often kept for longer periods of time for performance purposes. The TCP handshake can take a considerable amount of time to perform as opposed to the simple matter of retrieving data from a database. So if the database connection remains established, additional data can be retrieved without the overhead of TCP setup. But in these instances, shouldn’t the application ensure that keepalives are sent so that the connection is not prematurely terminated by an idle timer somewhere along the data path? Well, yes. Sort of. Allow me to explain.
When I first discovered the source of the network problem we were seeing, I chalked it up to lazy programming. While it shouldn’t take much to add a simple keepalive system to a networked application, it is extra work. As it turns out, however, the answer isn’t quite that simple. All three major operating systems, Windows, Linux, and OS X, all have kernel level mechanisms for TCP keepalives. Each OS has a slightly different take on how keepalive timers should work.
Linux has three parameters related to tcp keepalives :
- tcp_keepalive_time
- The interval between the last data packet sent (simple ACKs are not considered data) and the first keepalive probe; after the connection is marked to need keepalive, this counter is not used any further
- tcp_keepalive_intvl
- The interval between subsequential keepalive probes, regardless of what the connection has exchanged in the meantime
- tcp_keepalive_probes
- The number of unacknowledged probes to send before considering the connection dead and notifying the application layer
OS X works quite similar to Linux, which makes sense since they’re both *nix variants. OS X has four parameters that can be set.
- keepidle
- Amount of time, in milliseconds, that the connection must be idle before keepalive probes (if enabled) are sent. The default is 7200000 msec (2 hours).
- keepintvl
- The interval, in milliseconds, between keepalive probes sent to remote machines, when no response is received on a keepidle probe. The default is 75000 msec.
- keepcnt
- Number of probes sent, with no response, before a connection is dropped. The default is 8 packets.
- always_keepalive
- Assume that SO_KEEPALIVE is set on all TCP connections, the kernel will periodically send a packet to the remote host to verify the connection is still up.
Windows acts very differently from Linux and OS X. Again, there are three parameters, but they perform entirely different tasks. All three parameters are registry entries.
- KeepAliveInterval
- This parameter determines the interval between TCP keep-alive retransmissions until a response is received. Once a response is received, the delay until the next keep-alive transmission is again controlled by the value of KeepAliveTime. The connection is aborted after the number of retransmissions specified by TcpMaxDataRetransmissions have gone unanswered.
- KeepAliveTime
- The parameter controls how often TCP attempts to verify that an idle connection is still intact by sending a keep-alive packet. If the remote system is still reachable and functioning, it acknowledges the keep-alive transmission. Keep-alive packets are not sent by default. This feature may be enabled on a connection by an application.
- TcpMaxDataRetransmissions
- This parameter controls the number of times that TCP retransmits an individual data segment (not connection request segments) before aborting the connection. The retransmission time-out is doubled with each successive retransmission on a connection. It is reset when responses resume. The Retransmission Timeout (RTO) value is dynamically adjusted, using the historical measured round-trip time (Smoothed Round Trip Time) on each connection. The starting RTO on a new connection is controlled by the TcpInitialRtt registry value.
There’s a pretty good reference page with information on how to set these parameters that can be found here.
We still haven’t answered the question of optimal settings. Unfortunately, there doesn’t seem to be a correct answer. The defaults provided by most firewall vendors seem to have been chosen to ensure that the firewall does not run out of resources. Each connection through the firewall must be tracked. As a result, each connection uses up a portion of memory and CPU. Since both memory and CPU are finite resources, administrators must be careful not to exceed the limits of the firewall platform.
There is some good news. Firewalls have had a one hour tcp timeout timer for quite a while. As time has passed and new revisions of firewall hardware are released, the CPU has become more powerful and the amount of memory in each system has grown. The default one hour timer, however, has remained in place. This means that modern firewall platforms are much better prepared to handle an increase in the number of connections tracked. Ultimately, the firewall platform must be monitored and appropriate action taken if resource usage becomes excessive.
My recommendation would be to start by setting the firewall tcp teardown timer to a value slightly higher than that of the clients. For most networks, this would be slightly over two hours. The firewall administrator should monitor the number of connections tracked on the firewall as well as the resources used by the firewall. Adjustments should be made as necessary.
If longer lasting idle connections are unacceptable, then a slightly different tactic can be used. The firewall teardown timer can be set to a level comfortable to the administrator of the network. Problematic clients can be updated to send keepalive packets at a shorter interval. These changes will likely only be necessary on servers. Desktop systems don’t have the same need as servers for long-term establishment of idle connections.