Satellite TV Woes

Back in the day, I had Analog and then Digital cable.  Having been employed by a sister company of the local cable company, I enjoyed free cable.  There were a lot of digital artifacts, and the picture wasn’t always that great, but it was free and I learned to live with it.

After I left that company, I had to pay full price for the digital cable I had installed.  Of course, I was used to a larger package and the price just outright shocked me.  With the cable modem included, it was somewhere in the $150 a month range.  Between the signal issues on the cable TV, and the constant cable modem outages, I happily decided to drop both the cable and the cable modem and move on to DSL and Satellite TV.

My first foray into Satellite TV was with Dish Networks.  The choice to do so was mostly guided by my brother’s employment by Dish.  So, we checked it out and had it installed.  At the time, they were running a free DVR promotion, so we grabbed that as well.

Dish is great.  The DVR was a dual tuner, so we were able to hook two TVs up to it.  We could record two shows at once, and watch two recorded shows at the same time, one on each TV.  It was pure TV bliss, and it got better.  Dish started adding little features here and there that I started noticing more and more.  First, the on-screen guide started showing better summaries of the shows.  Then it would show the year the show was produced in.  And finally, it started showing actual episode number information.  Little things, but it made all the difference.

Dish, however, had it’s problems.  My family and I only watch a few channels.  The kids like the cartoon channels : Cartoon Network, Nickelodeon, Noggin, and Boomerang.  My wife enjoys the local channels for current shows such as CSI and Law and Order, and also the educational channels such as The History Channel, The Science Channel, and Discovery.  And myself, I’m into stuff like Scifi, FX, and occasionally, G4.  CSI and Law and Order are on my menu as well.  The problem is, in order to get all of the channels we wanted, we needed to subscribe to the largest Dish package.  It’s still cheaper than cable, but more money than we wanted to pay to pick up one or two extra channels.

Enter DirecTV.  DirecTV offered all the channels we wanted in their basic package.  So, we ordered it.  As it turns out, they’ve partnered with Verizon, so we can get our phone, DSL, and dish all on the same bill.  Personally, I couldn’t care less about that, but I guess it is convenient.

At any rate, we got DirecTV about a month or so ago.  Again, we got the DVR, but there’s a problem there.  DirecTV doesn’t offer a dual TV DVR.  It’s dual tuner so we can tape two shows simultaneously, but you can only hook a single TV up to it.  Our other TV has a normal DirecTV receiver on it.  Strike one against DirecTV, and we didn’t even have it hooked up yet.

So the guy comes and installs all the new stuff.  They used the same mount that the Dish Networks dish was mounted on, as well as the same cables, so that was convenient.  Dish Networks used some really high quality cables, so I was pleased that we were able to keep them.  Everything was installed, and the installer was pretty cool.  He explained everything and then went on his way.

I started messing around with the DVR and immediately noticed some very annoying problems.  The remote is a universal remote.  Dish Networks used them too.  The problem with the DirecTV remote, however, is that apparently when you interact with the TV, VCR, or DVD player, it needs to send the signal to the DirecTV receiver first before it will send the signal to the other equipment.  This means merely pressing the volume control results in nothing.  You need to hold the volume down for about a second before it will change the volume on the TV.  Very, very annoying.  I also noticed a considerable pause between pressing buttons on the controller and having the DVR respond.  The standalone receiver is much quicker, but there is definitely a noticeable lag there.  Strike two.

Continuing to mess around with the DVR, I started checking out how to set up the record timers and whatnot.  DirecTV has a nice guide feature the automatically breaks down the channels into sub-groups such as movie channels, family channels, etc.  They also have a nicer search feature than Dish does.  As you type in what you’re searching for, it automatically refreshes the list of found items, allowing you a quick shortcut to jump over and choose what you’re looking for.  Dish allows you to put arbitrary words in and record based on title matches, but I’m not sure if DirecTV does.  I never used that feature anyway.  So the subgroups and the search features are a score for DirecTV.

Once in the guide, however, it gets annoying.  Dish will automatically mask out any unsubscribed channels for you, where DirecTV does not.  Or, rather, if they do, it’s buried somewhere in the options and I can’t find it.  Because of this, I find all sorts of movies and programs that look cool, but give me a “you’re not subscribed to this channel” message when I try to watch them.  Quite annoying.

I set up a bunch of timers for shows my family and I like to watch.  It was pretty easy and worked well.  A few days later, I checked the shows that recorded.  DirecTV groups together episodes for shows which is a really nice feature.  However, I noticed that one or two shows never recorded.  Dish had a problem once in a while with recording new shows where the show didn’t have a “new” flag on it and it would skip it.  Thinking this was the problem with DirecTV, I just switched the timer to record all shows.  I’d have to delete a bunch of shows I already saw, but that’s no big deal.

Another week goes by, still no shows.  Apparently DirecTV doesn’t want me to watch my shows.  Now I’m completely frustrated.  Strike three.

Unfortunately, I’m in a two year contract, so I just have to live with this.  I’m definitely looking to get my Dish Networks setup back at the end, though.  That extra few bucks we spent on Dish was well worth it.

 

DirecTV definitely has some features that Dish doesn’t, but the lack of a dual tuner, the lag time between the controller and the receiver, and the refusal to tape some shows is just too much.  The latter two I can live with, but the dual TV DVR was just awesome and I really miss it.  Since I only have the DVR on the main TV in the house, I need to wait until the kids go to bed before I can watch my shows in peace.  Of course, I need to go to bed too since I get up early for work.  This leaves virtually no time for the few shows I watch, and as a result, I have a bunch of stuff recorded that I haven’t been able to watch yet.  And, since it’s that time of the year where most of my shows aren’t being shown, I know that it’s only going to get worse.

I’m just annoyed at this point.  If you have a choice between Dish and DirecTV, I definitely suggest Dish.  It’s much better in the long run and definitely worth the extra few dollars.

Troubleshooting 101

There seems to be a severe lack of understanding and technique when it comes to troubleshooting these days. It seems to me that a large amount of troubleshooting effort is completely wasted on wild ideas and theories while the simplest and most direct solutions are ignored.

Occam’s Razor states: “entities should not be multiplied beyond necessity.” Simply put, the easiest solution is often the best. This is the perfect mindset for anyone who does troubleshooting. There is no need to delve right into the most obscure reasons for a failure, start with the simple stuff.

For instance, questions like “Is the unit plugged in?”, or “Is the power on?” are perfect questions to start with. While it would be wonderful to believe that everyone you encounter has the common sense to check out these simple solutions, you’ll find that, unfortunately, the majority of the population isn’t that bright.

So, how about a real-world example. It’s 2am and you get paged that a router has gone unreachable. After notifying the proper people, you delve into the problem. Using the Occam’s Razor principle, what’s the first thing you should check? Well, for starters, let’s make sure the router really is unreachable. A simple ping should accomplish that. And just for good measure, ping something close to that router just to make sure you’re connected to the network.

Ok, so the router isn’t pingable, now what? Well, let’s look at the next easiest step, power. Since the router is in a remote location, this isn’t easy to check. However, you can check the uplink on the router. You should be able to get to the router just before the one that’s unreachable. Once there, check the interface that feeds your troubled router. Is it up or down? While you’re there, you can check for traffic and errors as well, but don’t focus on these yet, store them for later.

If the interface is down, then it’s quite possibly a physical line issue or, possibly, power. Just for good measure, I would suggest bouncing the interface to see if it’s something temporary. Sometimes, the interface will come back up and start running errors, indicating a physical line issue. What will often happen is that the interface comes back up and starts running errors, but allows limited traffic to get through. Once the error threshold is passed, the line goes back down. At this point, I’d call a technical to look at the physical line itself.

If the interface is up, try pinging the troubled router from the directly connected router. This process can help identify a routing issue in the network. Directly connected interfaces are considered to be the most specific route unless specifically overridden, which isn’t likely. If the ping is successful, take note of the ping time. If it seems overly high, you may be looking at a traffic issue. Depending on the type of router, traffic may be processor switched and cause high CPU usage. This can be identified by a sluggish interface and high ping times. Notes, high ping times don’t always indicate this. Most routers set a very low priority for ICMP traffic destined for the CPU, deeming throughput more important.

Remember the traffic and error counts you looked at previously? Those come into play now. If the traffic on the interface is very high, notably higher than usual, then this is likely the cause of the problem. Or, rather, an effect of the actual cause which may be a DoS attack or Virus outbreak. DoS, or Denial of Service, attacks are targeted attacks against a specific IP or range of IPs. A side effect of these attacks is that interfaces between the attacker and victim are often overloaded.

There are a number of different DoS attacks out there, but often when you see traffic as the cause of the DoS, you’ll notice that small packets are being used. One way to quickly identify this is to take the current bps on the interface, divide it by the packets per second, and then by 8 to get bytes per packet. Generally speaking, a normal interface ranges average packet size between 1000 and 1500 bytes. NOTE : This is referring to traffic received from a remote source such as a web site. Outgoing traffic, to the website, has a much lower average packet size because these packets generally contain control information such as acknowledgements, ICMP, etc.

Once you’ve identified that there is a traffic issue, the next step is to identify where the traffic is sourced from, or destined to. Remember, the end-goal here is to repair the problem so that normal operations can continue. Since you’re already aware of the overloaded interface, it’s easiest to concentrate your efforts there. Identifying the traffic source and destination is usually pretty easy, provided it’s not a distributed attack. On a Cisco router, you can try the “IP accounting” command. This command will show the source and destination for all output packets on an interface. Included is a count of the number of packets and the bits used by those packets. Simply look for rapidly increasing source and destination pairs and you’ll likely find your culprit.

Another option is to use an access list. If the router can handle it, place an access list on the interface that passes all traffic, but logs each packet. Then you can watch the log and try to identify large sources of traffic. Refine the access list to block that traffic until you’ve halted the attack. Be careful, however, as many routers will processor switch the traffic when an access-list is applied. This may cause a spike in CPU usage, sometimes causing a loss of connectivity to the router. If IP accounting is available, use that instead.

Once you identify the source and/or target of the attack, craft an appropriate access list to block the traffic as far upstream as you can. If the DoS attack is distributed, then the most effective means to stop the attack is probably to remove the targeted routes from the routing table and allow it to be blocked at the edges. This will likely result in an outage for that specific customer, but with a distributed attack, that’s often the only solution. From there you can work with your upstream providers to track down the perpetrator of the attack and take it offline permanently.

The preceding seems a bit long when written down, but in reality, this is a 15-30 minute process. Experienced troubleshooters can identify and resolve these problems even quicker. The point, of course, is to identify the most likely causes in the quickest manner possible. Often times, the simplest solution is the correct solution. Take the extra few seconds to check out the obvious before moving on to the more advanced. Often, you’ll resolve the solution quicker and sometimes wind up with a funny story as a bonus!

Please, troubleshoot responsibly.

Backups? Where?

It’s been a bit hectic, sorry for the long time between posting.

 

So, backups.  Backups are important, we all know that.  So how many people actually follow their own advice and back their data up?  Yeah, it’s a sad situation for desktops.  The server world is a little different, though, with literally tens, possibly hundreds of different backup utilities available.

 

My preferred backup tool of choice is the Advanced Maryland Automatic Network Disk Archiver, or AMANDA for short.  AMANDA has been around since before 1997 and has evolved into a pretty decent backup system.  Initially intended for single tape-based backups, options have been added recently to allow for tape spanning and disk-based backups as well.

Getting started with AMANDA can be a bit of a chore.  The hardest part, at least for me, was getting the tape backup machine running.  Once that was out of the way, the rest of it was pretty easy.  The config can be a little overwhelming if you don’t understand the options, but there are a lot of guides on the Internet to explain it.  In fact, the “tutorial” I originally used is located here.

Once it’s up and running, you’ll receive a daily email from Amanda letting you know how the previous nights backup went.  All of the various AMANDA utilities are command-line based.  There is no official GUI at all.  Of course, this causes a lot of people to shy away from the system.  But overall, once you get the hang of it, it’s pretty easy to use.

Recovery from backup is a pretty simple process.  On the machine you’re recovering, run the amrecover program.  You then use regular filesystem commands to locate the files you want to restore and add them to the restore list.  When you’ve added all the files, issue the extract command and it will restore all of the files you’ve chosen.  It’s works quite well, I’ve had to use it once or twice…  Lemme tell ya, the first time I had to restore from backups I was sweatin bullets..  After the first one worked flawlessly, subsequent restores were completed with a much lower stress level.  It’s great to know that there are backups available in the case of an emergency.

AMANDA is a great tool for backing up servers, but what about clients?  There is a Windows client as well that runs using Cygwin, a free open-source Linux-like environment for Windows.  Instructions for setting something like this up are located in the AMANDA documentation.  I haven’t tried this, but it doesn’t look too hard.  Other client backup options include remote NFS and SAMBA shares.

Overall, AMANDA is a great backup tool that has saved me a few times.  I definitely recommend checking it out.

Network Graphing

Visual representations of data can provide additional insight into the inner workings of your network. Merely knowing that one of your main feeds is peaking at 80% utilization isn’t very helpful when you don’t know how long the peak is, at what time, and when it started.

There are a number of graphing solutions available. Some of these are extremely simplistic and don’t do much, while others are overly powerful and provide almost too much. I prefer using Cacti for my graphing needs.

Cacti is a web-based graphing solution built on top of RRDtool. RRDtool is a round-robin data logging and graphing tool developed by Tobias Oetiker of MRTG fame, MRTG being one of the original graphing systems.

Chock full of features, Cacti allows data collection from almost anywhere. It supports SNMP and script-based collection by default, but additional methods can easily be added. Graphs are fully configurable and can display just about any information you want. You can combine multiple sources on a single graph, or create multiple graphs for better resolution. Devices, once added, can be arranged into a variety of hierarchies allowing multiple views for various users. Security features allow the administrator to tailor the data shown to each user.

Cacti is a wonderful tool to have and is invaluable when it comes to tracking down problems with the network. The ability to graph anything that spits out data makes it incredibly useful. For instance, you can create graphs to show you the temperature of equipment, utilization of CPUs, even the number of emails being sent per minute! The possibilities are seemingly endless.

There is a slight learning curve, however. Initial setup is pretty simple, and adding devices is straightforward. The tough part is understanding how Cacti gathers data and relates it all together. There are some really good tutorials on their documentation site that can help you through this part.

Overall, I think Cacti is one of the best graphing tools out there. The graphs come out very professional looking, and the feature set is amazing. Definitely worth looking into.

Host Intrusion Detection

Monitoring your network includes trying to keep the bad guys out. Unfortunately, unless you disconnect your computer and keep it in a locked vault, there’s no real way to ensure that your system is 100% hack proof. So, in addition to securing your network, you need to monitor for intrusions as well. It’s better to be able to catch an intruder early rather than find out after they’ve done a huge amount of damage.

Intrusion detection systems (IDS) are designed to detect possible intrusion attempts. There are a number of different IDS types, but this post concentrates on the Host Intrusion Detection System (HIDS).

My preferred HIDS of choice is Osiris. Osiris uses a client/server architecture, making it one of the more unique HIDS out there. The server stores all of the configurations and databases, and triggers the scanning process. SSL is used between the client and server to ensure communication integrity.

Once a new client is added, the server performs an initial scan. A configuration file is pushed to the client which then scans the computer accordingly, reporting the results back to the server. This first scan is then used as a baseline database for future comparisons.

The host periodically polls the clients and requests scans. The results of those scans are compared to the baseline database and an alert is sent if there are differences. An administrator can then determine if the changes were authorized and take appropriate action. If the changes are ok, Osiris is updated to use the new results as the baseline database. If the changes are suspect, the administrator can look further into them.

Osiris is very configurable. Scanning intervals can be set, allowing you fine-grained control over the time between scans. Multiple administrators can be set up to monitor and accept changes. Emails can be sent for each and every scan, regardless of changes.

The configuration file allows you to pick and choose what files on the client system are to be monitored. Fine-grain control over this allows the administrator to specify whole directories, or individual files. A filtering system can prevent erroneous results to be sent. For instance, some backup systems change the ctime to reflect when the file was last backed up. Without a filter, Osiris would report changes to all of the files each time a backup is run. Setting up a simple filter to ignore ctime on a file allows the administrator to ignore the backup process.

Overall, Osiris is a great tool for monitoring your server. Be prepared, though, monitoring HIDS can get cumbersome, especially with a large number of servers. Every update, change, or new program installed can trigger a HIDS alert.

There are other HIDS packages as well. I have not tested most of these, but they are included for completeness :

  • OSSEC
  • OSSEC is an actively maintained HIDS that supports log analysis, integrity checking, rootkit detection, and more.
  • AFICK
  • AFICK is another actively maintained HIDS that offers both CLI and GUI based operation
  • Samhain
  • Samhain is one of the more popular HIDS that offers a centralized monitoring system similar to that of Osiris.
  • Tripwire
  • Tripwire is a commercial HIDS that allows monitoring of configurations, files, databases and more. Tripwire is quite sophisticated and is mostly intended for large enterprises.
  • Aide
  • Aide is an open-source HIDS that models itself after Tripwire

Network Monitoring

I’ve been working a lot with network monitoring lately.  While mostly dealing with utilization monitoring, I do dabble with general network health systems as well.

There are several ways to monitor a network and determine the “health” of a given element.  The simple, classic example is the ICMP echo request.  Simply ping the device and if it responds, it’s alive and well.

This doesn’t always work out, however.  Take, for instance, a server.  Pinging the server simply indicates that the TCP/IP stack on the server is functioning properly.  But what about the processes running on the server?  How do you make sure those are running properly?

Other “health” related items are utilization, system integrity, and environment.  When designing and/or implementing a network health system, you need to take all of these items into account.

 

I have used several different tools to monitor the health of the networks I’ve dealt with.  These tools range from custom written tools to off-the-shelf products.  Perhaps at some point in the future I can release the custom tools, but for now I’ll focus on the freely available tools.

 

For general network monitoring I use a tool called Argus.  Argus is a pretty robust monitoring system written in Perl.  It’s pretty simple to set up and the config file is pretty self explanatory.  Monitoring capabilities include ping (using fping), SNMP, http, and DNS.  You can monitor specific ports on a device, allowing you to determine the health of a particular service.

Argus also has some unique capabilities that I haven’t seen in many other monitoring systems.  For instance, you can monitor a web page and detect when specific strings within that webpage change.  This is perfect for monitoring software revisions and being alerted to new releases.  Other options include monitoring of databases via the Perl DBI module.

The program can alert you in a number of different manners such as email or paging (using qpage).  Additional notification methods are certainly possible with custom code.

The program provides a web interface similar to that older versions of What’s Up Gold.  There is a fairly robust access control system that allows the administrator to lock users into specific sections of the interface with custom lists of available elements.

Elements can be configured with dependencies, allowing alerts to be suppressed for child elements.  Each element can also be independently configured with a variety of options to allow or suppress alerts, modify monitoring cycle times, send custom alert messages, and more.  Check out the documentation for more information.  There’s also an active mailing list to help you out if you have additional questions.

 

In future posts I’ll touch on some of the other tools I have in my personal toolkit such as host intrusion detection systems, graphing systems, and more.  Stay tuned!

To optimize…

… or not to optimize, that is the programmer’s dilemma.  Optimization is a powerful tool in the programmer’s arsenal, but one that should be used sparingly and with care.  The key to optimization is identifying what exactly to optimize.

 

Shawn Hargreaves, creator of the Allegro game programming library recently posted a link to the blog of Thomas Aylesworth, a.k.a. SwampThingTom.  According to his blog, Thomas is a software engineer for the aerospace and defense industry.  He’s also a hobbyist game developer.

Tom’s first foray into the world of blogging is a series of posts about optimization, specifically targeting XNA based games.  In his first post, he explains the importance of design and what Big O notation is.  His second post delves into prototyping and benchmarking, complete with examples.  In part three, he introduces us to the NProf execution profiler as well as pointing out a few other potential bottlenecks.

Optimization in general is something you shouldn’t really need to worry about until the very end of the development cycle, if ever.  Optimization is a great tool for squeezing just a few more cycles out of your code when you really need it.  What you normally don’t see, however is a significant increase in speed.  If you’re looking for a significant increase in speed, take a look at your underlying algorithm.

Generally speaking, unless you’re writing extremely specialized code, optimization should be used very sparingly.  You’re better off looking for the elegant way to solve your programming dilemma.  Look into alternative algorithms or possibly re-design the data flow.  If you’re not sure where the bottleneck is, look into using a code profiler, or simply add debugging statement that surround suspected “slow” code.  You’ll usually find that a poor design decision is causing the bottleneck and that a simple re-design can result in huge speed increases while keeping your code readable and maintainable.

 

If you’re interested in optimization, or even just curious, take a look at Tom’s articles, they’re a great read.

ISO Recorder Power Toy

I recently had the need to create an .ISO image of a CD. The CD burning software on my computer, however, only created proprietary images. Being my laptop for work, I didn’t want to purchase better software, so I googled around on the net a little bit.

I came across a little utility created by Alex Feinman called ISO Recorder. It runs on Windows XP, Windows Server 2003, and the dreaded Windows Vista. After installation, it adds two options to your right-click menu, “Create ISO Image File” and “Copy CD to CD”.

The Create ISO option appears whenever you right click a folder and allows you to create an image of everything in that folder. This includes folders on your hard drive, so creating an ISO is as simple as moving the relevant files into a single folder. Very convenient.

Alex also has a command-line CD burning utility called CreateCD. I have not had occasion to use this particular piece of software, but it does look interesting. Using this utility, you can automate the creation of ISO images, great for automated backups.

Both of these utilities are free for personal use. Alex does provide a PayPal link for donations, so if you find this software useful, send him a few bucks to show your appreciation!

Common PHP Regular Expression Security Issue

Stefan Esser (PHP Security Blog, Suhosin) recently posted an entry on his blog titled “Holes in most preg_match() filters” about a possible security issue that apparently escapes a lot of notice.

Let me explain the situation.  PHP uses Perl Compatible Regular Expressions, PCRE, for pattern matching.  In PCRE the carat metacharacter (^) is used to match the very beginning of the string, and the dollar-sign metacharacter ($) is used to match the end of the string.  This is extremely useful to ensure that the expression you’ve written has matched the entire string.

However, PCRE_DOLLAR_ENDONLY is not used by default.  This means that the dollar-sign metacharacter still matches to the end of the string, but it also matches is a newline character is at the end of the string.  In other words, a newline character may, or may not be present at the end of the string and you won’t know either way by default.

So, how do we fix this then?  Well, there are two ways.  First, you can add a D modifier to the end of the regular expression like this :

preg_match(‘/^[a-z]+$/D’, $string);

Or, you can use the \z modifier like this :

preg_match(‘/^[a-z]+\z/’, $string);

Either method works, although from the comments at Stefan’s site, it looks like \z is more portable since Perl doesn’t support the D modifier.

Here is short script to “prove” this, as it were :

 

$badstring = urldecode(“test%0a”);

if (preg_match(‘/^[0-9a-zA-Z]+$/’, $badstring)) {

print “Test 1 MATCHES\n”

}

if (preg_match(‘/^[0-9a-zA-Z]+$/D’, $badstring)) {

print “Test 2 MATCHES\n”

}

if (preg_match(‘/^[0-9a-zA-Z]+\z/’, $badstring)) {

print “Test 3 MATCHES\n”

}

 

I’m posting this info for two reasons.  First, it’s something programmers need to know.  It’s important since security holes are a bad thing.  Second, I’m guilty of this myself.  phpTodo used the dollar-sign metacharacter without the D modifier, making my code somewhat insecure.

The good news is that I have corrected the problem and posted a new version.  This is a precautionary measure, I don’t believe this adversely affected the security of the application, but better safe than sorry.  Head over and grab the new version just to be on the safe side.

Windows .ANI Vulnerability – The plot thickens

The Internet Storm Center is reporting that the newly released Microsoft patch is causing some problems. There one known problem and a bunch of reports about other problems.

The first problem is with the Realtek HD Audio Control Panel. Apparently, the control panel won’t start after the patch is installed, complaining about a DLL being illegally relocated. Microsoft has released another patch to resolve this.

The other problems are currently undefined. Microsoft is asking that users experiencing problems contact their support line so they can investigate the issues.

Because of these problems, it may be worth it to take a second look at the ZERT patch. If you’re experiencing problems with the Microsoft patch, try uninstalling it and install the ZERT patch instead. It’s possible that you’ll experience similar problems with the ZERT patch, but it’s worth giving it a shot.

Good luck!