Technological Musings

Configuration management is pretty important, but often overlooked. It's typically easy enough to handle configurations for servers since you have access to standard scripting tools as well as cron. Hardware devices such as switches and routers are a bit more to handle, though, as automating backups of these configs can be daunting, at best.

Several years ago, I took the time to write a fairly comprehensive configuration backup system for the company I was working for. It handled Cisco routers and switches, Fore Systems/Marconi ASX ATM switches, Redback SMS aggregators, and a few other odds and ends. Unfortunately, it was written specifically for that company and not something easily converted for general use.

Fortunately, there's a robust open source alternative called RANCID. The Really Awesome New Cisco confIg Differ, RANCID, is a set of perl scripts designed to automate configuration retrieval from a host of devices including Cisco, Juniper, Redback, ADC, HP, and more. Additionally, since most of the framework is already there, you can extend it as needed to support additional devices.

RANCID has a few interesting features which make life much easier as a network admin. First, when it retrieves the configuration from a device, it checks it in to either a CVS or SVN repository. This gives you the ability to see changes between revisions, as well as the ability to retrieve an old revision of a config from just about any point in time. Additionally, RANCID emails a list of the changes between the current and last revision of a configuration to you. This way you can keep an eye on your equipment, seeing alerts when things change. Very, very useful to detect errors by you and others.

Note: RANCID handles text-based configurations. Binary configurations are a whole different story. While binary configs can be placed in an SVN repository, getting emailed about changes becomes a problem. It's possible to handle binary configs, though I do not believe RANCID has this capability.

Setup of RANCID is pretty straightforward. You can either install straight from source, or use a pre-packaged RPM. For this short tutorial, I'll be using an RPM-based installation. The source RPM I'm using can be found here. It is assumed that you can either rebuild the RPM via the rpmbuild utility, or you can install the software from source.

After the software is installed, there are a few steps required to set up the software. First, I would recommend editing the rancid.conf file. I find making the following modifications to be a good first step:

RCSSYS=svn; export RCSSYS
* Change RCSSYS from cvs to svn. I find SVN to be a superior revisioning system. Your mileage may vary, but I'm going to assume you're using SVN for this tutorial.

FILTER_PWDS=ALL; export FILTER_PWDS
NOCOMMSTR=YES; export NOCOMMSTR
* Uncommenting these and turning them on ensures that passwords are not stored on your server. This is a security consideration as these files are stored in cleartext format.

OLDTIME=4; export OLDTIME
* This setting tells RANCID how long a device can be unreachable before alerting you to the problem. The default is 24 hours. Depending on how often you run RANCID, you may want to change this option.

LIST_OF_GROUPS="routers switches firewalls"
* This is a list of names you'll use to identify devices. These names are arbitrary, so Fred Bob and George are ok. However, I would encourage you to use something meaningful.

The next step is to create the CVS/SVN repositories you'll be using. This can't possibly be easier. Switch to the rancid user, then run rancid-cvs. You'll see output similar to the following:

-bash-3.2$ rancid-cvs
Committed revision 1.
Checked out revision 1.
A configs
Adding configs
Committed revision 2.
A router.db
Adding router.db
Transmitting file data .
Committed revision 3.
Committed revision 4.
Checked out revision 4.
A configs
Adding configs
Committed revision 5.
A router.db
Adding router.db
Transmitting file data .
Committed revision 6.
-bash-3.2$

That's it, your repositories are created. All that's left is to set up the user credentials that rancid will use to access the devices, tell rancid which devices to contact, and finally, where to send email. Again, this is quite straightforward.

User credentials are stores in the .cloginrc file located in the rancid home directory. This file is quite detailed with explanations of the various configuration options. In short, for most Cisco devices, you'll want something like this:

add user * <username>
add password * <login password> <enable password>
add method * ssh

This tells the system to use the given username and passwords for accessing all devices in rancid via ssh. You can specify overrides by adding additional lines above these, replacing the * with the device name.

Next, tell rancid what devices to contact. As the rancid user, switch to the appropriate repository directory. For instance, if we're adding a router, switch to ~rancid/routers and edit the router.db file. Note: This file is always called router.db, regardless of the repository you are in. Each line of this file consists of three fields, separated by colons. Field 1 is the hostname of the device, field 2 is the type of device, and field 3 is either up or down depending on whether the device is up or not. If you remove a device from this file, the configuration is removed from the repository, so be careful.

router.example.com:cisco:up

Finally, set up the mailer addresses for receiving rancid mail. These consist of aliases on the local machine. If you're using sendmail, edit the /etc/aliases file and add the following :

rancid-<group>: <email target>
rancid-admin-<group>: <email target>

There are two different aliases needed for each group. Groups are the names used for the repositories. So, in our previous example, we have three groups, switches, routers, and firewalls. So we set up two aliases for each, sending the results to the appropriate parties. The standard rancid-<group> alias is used for sending config diffs. The rancid-admin-<group> alias is used to send alerts about program problems such as not being able to contact a device.

Make sure you run newaliases when you're done editing the aliases file.

Once these are all set up, we can run a test of rancid. As the rancid user, run rancid-run. This will run through all of the devices you have identified and begin retrieving configurations. Assuming all went well, you should receive notifications via email about the new configurations identified.

If you have successfully run rancid and retrieved configurations, it's time to set up the cron job to have this run automagically. Merely edit the crontab file for rancid and add something similar to the following:

# run config differ 11 minutes after midnight, 2am, 4am, etc.
11 0-23/2 * * * /usr/bin/rancid-run
# clean out config differ logs
50 23 * * * /usr/bin/find /var/rancid/logs -type f -mtime +2 -exec rm {} \;

Offsetting the times a bit is a good practice, just to ensure everything doesn't run at once and bog down the system. The second entry cleans up the rancid log files, removing anything older than 2 days.

And that's it! You're well on your way to being a better admin. Now to finish those other million or so "great ideas" ....