Tag: virtualization

Multiple Personalities With The Linux Kernel

Virtualization is all the rage these days. Taking a single computer system and installing multiple “guest” operating systems on it. The benefits are a reduced footprint and better utilization of existing resources. There is a danger, however, in having many systems dependent on a single piece of hardware. The solution, of course, is to use multiple pieces of hardware and allow your “guests” to be moved between the individual hardware units, thus making the system more resilient to failure.

I’ve started playing a bit with virtualization, specifically, KVM virtualization. For my purposes, I’m using CentOS 6.x on a 64-bit capable system.

The hypervisor itself is a standard CentOS base install with the addition of kvm and various management packages. I installed the hypervisor on a RAID1 LVM, allowing me some room to grow if necessary, and reserving the remainder of the hard drive for virtual hosts. While you can use binary blobs for virtual disk, I prefer using a raided LVM which gives me the ability to grow the disk if necessary as well as minor bumps in speed.

Using yum, adding KVM to an existing installation is a pretty straightforward process :

yum install virt-manager libvirt libvirt-python python-virtinst

That should take care of any dependencies required to get KVM virtualization up and running.

Next up, we need to tackle networking. There are many, many different configurations, far too many to go through here. So, I’m going to keep it simple. For my purposes, I need a single connection to the outside network, all in the same VLAN, as well as a local NAT for some VMs that I need local access to, but that don’t need to be accessed via the Internet.

Setting this up is brilliantly simple. First, copy the /etc/sysconfig/network-scripts/ifcfg-eth0 file to /etc/sysconfig/network-scripts/ifcfg-br0. Next, edit the ifcfg-eth0 file. You’ll need to remove a bunch of lines and add a BRIDGE line as follows :

DEVICE=”eth0″

BRIDGE=”br0″

HWADDR=”00:11:22:33:44:55″

ONBOOT=”yes”

Next, edit the ifcfg-br0 file. All you really need to do here is change the DEVICE= line to reflect br0. I also recommend disabling NM_CONTROLLED … NetworkManager shouldn’t be installed anyway since you used a base install, but better safe than sorry. In the end, the ifcfg-br0 file should look something like this :

DEVICE=”br0″

BOOTPROTO=”static”

BROADCAST=”204.10.167.63″

IPADDR=”204.10.167.50″

NETMASK=”255.255.255.192″

ONBOOT=”yes”

TYPE=”Bridge”

DELAY=”0″

Restart networking and you’ll be all set. The NAT portion of this is handled by KVM itself, so there’s nothing to do there. And networking should be all ready to go.

Without guests, however, all you have is a basic Linux system with a few extra packages taking up space. The real magic starts when you create and install your first VM. My recommendation is to start with creating a template system you can clone later rather than hand-installing every single VM. To install the template, first decide on the base disk size. I’m using 15 GB volumes which is more than enough for the base install and leaves room for most basic server configurations. If you need more space, you can attach additional disks later.

I’m not going to go into how I set up LVM, there are plenty of tutorials out there. For the purposes of this article, I have a volume group names vg_libvirt where I plan to store all of the virtual machines. So first we create the disk necessary for the template :

lvcreate -L15G -n template_base vg_libvirt

Next we install the OS. virt-install is essentially a wrapper script that sets all the necessary values within KVM to get you going. After the settings are configured and the VM is started, girt-installer will automatically attach you to the VM console. The full command I used to install is as follows :

virt-install –accelerate –hvm –connect qemu:///system –network bridge:bra –name template –ram 512 –disk=/dev/mapper/vg_libvirt-template_base –vcpus=1 –check-cpu –nographics –extra-args=”console=ttyS0 text” –location=/tmp/CentOS-6.2-x86_64-bin-DVD1.iso

Since this is effectively a text install, you do run into a bit of a problem. Namely, you can’t configure the drives the way you want. There is a way around this, though it takes a bit of work. Of course, since you’re creating a template, the little bit of work now is easily made up for later. So, here’s how I handled the drive configuration.

First, run through a basic install using the above install method. Once you’re up and running, log into the new VM and head to the root home directory. In that directory you’ll find a kickstart file called anaconda-ks.cfg. Make a local copy of that file and shut down the VM.

The kickstart file gives you the basic parameters that CentOS used to configure the system. You can edit this file and use it yourself to automatically install and configure systems. For our purposes, we’re interested in editing the drive configuration and then using the kickstart file to create the template. So, edit the file and set the parameters as you see fit. An example is as follows :

# Kickstart file automatically generated by anaconda.

#version=DEVEL

install

cdrom

lang en_US.UTF-8

keyboard us

network –onboot no –device eth0 –noipv4 –noipv6

rootpw –iscrypted somerandomstringthatiwontrevealtoyoubutnicetry

firewall –service=ssh

authconfig –enableshadow –passalgo=sha512

selinux –enforcing

timezone –utc America/New_York

bootloader –location=mbr –driveorder=vda –append=” console=ttyS0 crashkernel=auto”

# The following is the partition information you requested

# Note that any partitions you deleted are not expressed

# here so unless you clear all partitions first, this is

# not guaranteed to work

clearpart –all –drives=vda

part /boot –fstype=ext4 –size=500

part swap –size=2048

part pv.253002 –grow –size=1

volgroup VolGroup –pesize=4096 pv.253002

logvol / –fstype=ext4 –name=lv_root –vgname=VolGroup –size=4096

logvol /tmp –fstype=ext4 –name=lv_tmp –vgname=VolGroup –size=2048

logvol /var –fstype=ext4 –name=lv_var –vgname=VolGroup –size=4096

logvol /home –fstype=ext4 –name=lv_home –vgname=VolGroup –size=2048

#repo –name=”CentOS” –baseurl=cdrom:sr0 –cost=100

%packages –nobase

@core

%end

Once you have this, you can re-run the girt-install command from above with a slight tweak to make the install use the kickstart file you created (I named it kick1.ks) :

virt-install –accelerate –hvm –connect qemu:///system –network bridge:bra –name template –ram 512 –disk=/dev/mapper/vg_libvirt-template_base –vcpus=1 –check-cpu –nographics –initrd-inject=/path/to/kick1.ks –extra-args=”ks=file:/kick1.ks console=ttyS0 text” –location=/tmp/CentOS-6.2-x86_64-bin-DVD1.iso

This will nuke the existing VM and replace it with one configured with the drive partitions as set in the kickstart file. And now you almost have a template.

You could use this new VM as a clone, but if you’ve set an IP on it, you’ll run into duplicate IP problems. SSH keys on the machine will be cloned, making all of your systems contain the same keys. And other machine-specific settings will be cloned as well. This can be worked around, though.

I recommend that you first configure this new template with the basic settings you want on all of your VMs. For instance, if you’re using Spacewalk for server management, you can install all of the necessary spacewalk binaries. You can configure a standard iptables template for the system. Maybe you have some standard security software you use such as OSSEC. And, of course, create the standard users on the system so you don’t have to create them each time you clone the VM. Once everything is installed and running how you want it, perform the following actions to make the template :

touch /.unconfigured

rm -rf /etc/ssh/ssh_host_*

poweroff

The VM will power down and you’ll have your template. Cloning this to a new VM is quick and simple. First, create the new logical volume as we did before. Next, clone the VM to the new drive :

virt-clone -o template -n new_vm -f /dev/mapper/vg_libvirt-new_vm_base

Simple enough, right? Run this command and when it completed, you can start the VM and connect to the console. You’ll be greeted with the standard first boot process and then dropped at a login prompt. Congratulations, you now have a VM. Set the IP, configure whatever services you need, and you’re off to the races.

If you need to modify the RAM, number of CPUs, etc., then use the virsh command on the hypervisor. You’ll need to shut down the VM and restart it in order for these changes to take effect.

And that’s really all there is to it. The VMs themselves can be treated as self-contained systems with no special care necessary … One note, however. If you reboot the hypervisor, the VMs are paused before rebooting and resumed after reboot. This leads to an interesting problem in that the uptime on a VM can easily exceed that of the hypervisor. Be aware of this and don’t depend on a VMs uptime to be accurate.

Hey KVM, you’ve got your bridge in my netfilter…

It’s always interesting to see how new technologies alter the way we do things.  Recently, I worked on firewalling for a KVM-based virtualization platform.  From the outset it seems pretty straightforward.  Set up iptables on the host and guest and move on.  But it’s not that simple, and my google-fu initially failed me when searching for an answer.

The primary issue was that when iptables was enabled on the host, the guests became unavailable.  If you enable logging, you can see the traffic being blocked by the host, thus never making it to the guest.  So how do we do this?  Well, if we start with a generic iptables setup, we have something that looks like this:

# Firewall configuration written by system-config-securitylevel
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT – [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp –icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -m state –state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -m state –state NEW -m tcp -p tcp –dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT –reject-with icmp-host-prohibited
COMMIT

Adding logging to identify what’s going on is pretty straightforward.  Add two logging lines, one for the INPUT chain and one for the FORWARD chain.  Make sure these are added as the first rules in the chain, otherwise you’ll jump to the RH-Firewall-1-INPUT chain and never make it to the log.

-A INPUT -j LOG –log-prefix “Firewall INPUT: ”
-A FORWARD -j LOG –log-prefix “Firewall FORWARD: ”

 

Now, with this in place you can try sending traffic to the domU.  If you tail /var/log/messages, you’ll see the blocking done by netfilter.  It should look something like this:

Apr 18 12:00:00 example kernel: Firewall FORWARD: IN=br123 OUT=br123 PHYSIN=vnet0 PHYSOUT=eth1.123 SRC=192.168.1.2 DST=192.168.1.1 LEN=56 TOS=0x00 PREC=0x00 TTL=64 ID=18137 DF PROTO=UDP SPT=56712 DPT=53 LEN=36

There are a few things of note here.  First, this occurs on the FORWARD chain only.  The INPUT chain is bypassed completely.  Second, the system recognizes that this is a bridged connection.  This makes things a bit easier to fix.

My attempt at resolving this was to put in a rule that allowed traffic to pass for the bridged interface.  I added the following:

-A FORWARD -i br123 -o br123 -j ACCEPT

This worked as expected and allowed the traffic through the FORWARD chain, making it to the domU unmolested.  However, this method means I have to add a rule for every bridge interface I create.  While explicitly adding rules for each interface should make this more secure, it means I may need to change iptables while the system is in production and running, not something I want to do.

A bit more googling led me to this post about KVM and iptables.  In short it provides two additional methods for handling this situation.  The first is a more generalized rule for bridged interfaces:

-A FORWARD -m physdev –physdev-is-bridged -j ACCEPT

Essentially, this rule tells netfilter to accept any traffic for bridged interfaces.  This removes the need to add a new rule for each bridged interface you create making management a bit simpler.  The second method is to completely remove bridged interfaces from netfilter.  Set the following sysctl variables:

net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

I’m a little worried about this method as it completely bypasses iptables on dom0.  However, it appears that this is actually a more secure manner of handling bridged interfaces.  According to this bugzilla report and this post, allowing bridged traffic to pass through netfilter on dom0 can result in a possible security vulnerability.  I believe this is somewhat similar to cryptographic hash collision.  Attackers can take advantage of netfilter entries with similar IP/port combinations and possibly modify traffic or access systems.  By using the sysctl method above, the traffic completely bypasses netfilter on dom0 and these attacks are no longer possible.

More testing is required, but I believe the latter method of using sysctl is the way to go.  In addition to the security considerations, bypassing netfilter has a positive impact on throughput.  It seems like a win-win from all angles.

Virtuality, Part Deux

I had the chance to install VirtualBox and I must say, I’m pretty impressed.  To start, VirtualBox is only a 15 meg download.  That’s pretty small when compared to Virtual PC, and downright puny when compared to VMWare Workstation.  There seems to be a lot packed in there, however.  As with VMWare, VirtualBox has special extensions that can be installed into the guest OS for compatibility.

Installation was a snap, similar to that of VMWare, posing no real problem.  The first problem I encountered was after rebooting the guest and logging in.  Apparently, I ran out of memory on the host OS, so VirtualBox gracefully paused the guest OS and alerted me.  After closing some open programs, I was able to resume the guest OS with no problem.  These low memory errors remain the only real problem I have with VirtualBox at this point.

Networking in VirtualBox is a little different from that of VMWare, and took me a few tries before I figured it out.  By default, the system is installed with no virtual adapters, making NAT the only means by which the guest OS can speak to the world.  By installing a virtual interface on the host, through the use of Host Interface Networking (HIF), you can allow the guest OS direct access to the network.  After the interface is created, it is bridged, through the use of a Windows Network Bridge interface, with the interface you want the traffic to flow out of.  Adding and removing an interface in the bridge sometimes takes a minute or two.  I honestly have no idea what Windows is doing during this time, but until the interface is added/removed, networking ceases to function.  I have also noticed that if VirtualBox is running, regardless of the state of the guest OS, modifying the bridge will fail.

Installation of the guest extensions, which required GCC and the kernel headers on the guest OS to be installed, was relatively painless.  After making sure the necessary packages were installed in CentOS, VirtualBox compiled and installed the extensions.  This allowed me to extend my desktop resolution to 1024×768, as well as enabling auto-capture of the mouse pointer when it enters the virtual machine window.  According to the documentation, the extensions also add support for a synchronized clock, shared folders and clipboard, as well as automated Windows logins (assuming you are running a Windows guest OS).

VirtualBox is quite impressive, and I’ve started using it full time.  It’s not quite as polished as VMWare is, but it definitely wins price-wise.  I’m sure VMWare has many more features that I am not using, that may actually justify the price.  For now, I’ll stick with VirtualBox until something forces me to switch.

In related news, I’ve been informed by LonerVamp that VMWare Server, which is free, would also satisfy my needs.  I am a bit confused, though, that a server product would be released for free while a workstation product would not.  I was initially under the impression that the server product merely hosted the OS, allowing another VMWare product to remotely attach to it.  That doesn’t appear to be correct, however.  Can someone explain the major differences to me?  Why would I want to use Workstation as opposed to Server?

Virtuality

I’ve been playing around a bit with two of the major virtualization programs, VMWare Workstation and Microsoft Virtual PC.  My interest at this point has only been to allow me to run an alternative operating systems on my primary PC.  In this particular case, I was looking to run CentOS 5.1 on a Windows XP laptop.

I started out with Virtual PC, primarily because of the price, free.  My goal here is to make my life a little easier when developing web applications.  It would be nice to run everything locally, allowing me freedom to work virtually anywhere, and limiting development code on a publicly accessible machine.  There are, of course, other options such as a private network with VPNs, but I’m really not looking to go that far yet.

Right from the start, Virtual PC is somewhat of a problem.  Microsoft’s website claims that VPC can run nearly any x86-based OS, but they only list Windows operating systems on their website.  This is expected, of course.  So, knowing that this is merely an x86 emulation package, I went ahead and installed it.

The first thing I noticed was the extremely long time it took to install.  Installation of the files themselves seemed to go relatively quickly, but the tail end of the install, where it installs the networking portion of VPC was extremely slow and resulted in a loss of connectivity on my machine for about 5-10 minutes.  I experienced a similar issue uninstalling the software.

Once VPC was installed, I used a DVD copy of CentOS 5.1 to install the guest OS.  I attempted a GUI installation, but VPC wouldn’t support the resolution and complained, resulting in a garbled virtual screen.  I resorted to installation using the TUI.  Installation itself went pretty smoothly, no real problems encountered.  Once completed, I fired up the new OS.

Because I was in text-mode within the virtual OS, the window was pretty small on my screen.  I flipped to full-screen mode to compensate.  I noticed, however, that the fonts were not very sharp, looking very washed out.  I also experienced problems cutting and pasting between the host and guest OS.  Simply put, I could not cut and paste between the two at all.

The guest OS seemed sluggish, but I attributed this to an underpowered laptop running two operating systems simultaneously.  Overall, Virtual PC seemed to work ok, but the lack of graphical support and the inability to cut and paste between the host and guest really made working with it problematic.

My second choice for virtualization was VMWare Workstation.  VMWare has been around for a very long time, and I remember using their product years ago.  VMWare is not free, running a cool $189 for a single license, however there is a 30 day trial key you can get.  I signed up for this and proceeded to install the software.

The first major difference between VPC and Workstation is the size of the program.  VPC clocks in at a measly 30 Megs, while Workstation runs about 330 Megs.  Installation is a snap, however, and proceed quite quickly.  I didn’t experience the same network problems that I did with VPC.

Once installed, I proceeded to load the same guest OS using the same memory and hard drive parameters as I did with VPC.  VMWare correctly configured the graphical display to handle the GUI installer, and I was able to install the Gnome desktop as well.  Installation seemed to go a bit quicker than with VPC, despite the added packaged for X Window and Gnome.

After installation was complete, I booted into the new OS.  VMWare popped up a window notifying me that the Guest OS was not running the VMWare Tools package and that installing it would result in added speed and support.  I clicked OK to bypass the message and allowed the OS to continue loading.

Almost immediately I noticed that VMWare was running quite a bit faster than VPC.  The guest OS was very responsive and I was able to quickly configure MySQL and Apache.  I also noticed that VMWare made the guest OS aware of my sound card, USB controller, and just about every other device I have in the system.  Fortunately, it’s easy enough to remove those from the configuration.  I really have no need for sound effects when I’m merely writing code..  :)

Overall, VMWare has been running very well.  Well enough to make me think about getting a license for it and using it full time.  However, my overall goal is to move over to the OSX platform, so I’m not sure I want to blow $200 on something I’ll only use for a few months (fingers crossed)…  Another alternative may be VirtualBox, an open-source alternative.  I’ll be downloading and checking that out soon enough.

In the end, though, if you’re looking to run high-end applications, or even use a specific OS full-time, there’s nothing better than a full install on a real machine as opposed to a virtual one.