Virtualisation with Xen on CentOS 6.3

It's been a while since I've had to set up a virtualisation server, but today I needed to configure a brand-new install of CentOS 6 to act as a virtual host. The hardware doesn't have virtualisation support (an old G3) so I had to use Xen so that paravirtualisation was available (not currently supported by KVM). Oops, not so easy now that Xen isn't included by default, Red Hat having opted to use KVM instead.

Despite that, getting things set up isn't that hard, although not nearly as easy as it was when you could just

yum install xen

 This documentation details the steps you'll need to follow.

So let's start with a few basic pre-requisites;

yum install sanlock presto libblkid

 

SELinux

As helpful as it is for security, it's going to interfere like nobodies business! Either disable it or make permissive

nano /etc/sysconfig/selinux  
# Change SELINUX=enabled to SELINUX=disabled

This won't be effective until we reboot, but let's get a few bits done first!

 

Bridge the network

The package should already be installed, but just to be safe, run

yum install bridge-utils

Next we need to configure the bridge

cp /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-br0

Then we edit the configuration file we just created, so that we get something like

DEVICE="br0"
NM_CONTROLLED="yes"
ONBOOT=yes
TYPE=Bridge
BOOTPROTO=none
IPADDR=192.168.1.175
PREFIX=24
GATEWAY=192.168.1.254
DNS1=129.168.1.254
DNS2=8.8.4.4
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="System br0"

Next we need to edit the original file and add a line at the end, so

nano /etc/sysconfig/network-scripts/ifcfg-eth0 
#add a line at the bottom

BRIDGE=br0

So now that's set up, we can move on and proceed with the install.

 

Kernel Install

Note: I had a few issues with the kernel we'll be downloading in this step, and eventually resorted to compiling and installing the upstream kernel myself. Steps for that are below.

Use wget to install the aussie repo (I'm sure there must be a closer one!). If you don't have wget start with a yum install wget.

yum install http://au1.mirror.crc.id.au/repo/kernel-xen-release-6-3.noarch.rpm

We now need to go off on a slight tangent from the official walkthroughs now, as attempting to install at this point will tell us that qemu-img isn't available. So, we need to install from repoforge as follows

yum install http://pkgs.repoforge.org/qemu/qemu-img-0.15.0-1.el6.rfx.i686.rpm

Then we can proceed to run

yum install kernel-xen xen

To download and build the bits we actually want! Once that's complete there are a few more steps to go through before we're quite ready to reboot.

 

Bootloader

First we need to reconfigure grub to boot the Xen hypervisor before the kernel

nano /boot/grub/menu.1st

Change

title CentOS (2.6.32.57-2.el6xen.i686)
root (hd0,0)
kernel /vmlinuz-2.6.32.57-2.el6xen.i686 ro root=/dev/mapper/vg_virty-lv_root rd_NO_LUKS KEYBOARDTYPE=pc KEYTABLE=uk LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkerne$
initrd /initramfs-2.6.32.57-2.el6xen.i686.img

So that kernel becomes module. We also need to add a line to load the xen hypervisor so should end up with something like

title CentOS (2.6.32.57-2.el6xen.i686)
root (hd0,0)
kernel /xen.gz dom0_mem=2048M cpufreq=xen dom0_max_vcpus=2 dom0_vcpus_pin
module /vmlinuz-2.6.32.57-2.el6xen.i686 ro root=/dev/mapper/vg_virty-lv_root rd_NO_LUKS KEYBOARDTYPE=pc KEYTABLE=uk LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkerne$
module /initramfs-2.6.32.57-2.el6xen.i686.img

You'll notice dom0_max_vcpus=2. As the G3 I was building on has 2 cores, I set this to 2. Adjust as necessary for your system (the same goes for dom_0_mem=2048M)

 Double check that grub default is set to 0 and then save and close the grub file.

 

LibVirt

We now need to install libvirt, unfortunately there's a few extra steps as the repo builds no longer include support for Xen, so we need to do the following

yum install libvirt python-virtinst

to install the files from the repo.

Now we'll install a few pre-requisites so that we can rebuild the src rpm's with Xen support

yum groupinstall 'Development Tools'
yum install python-devel xen-devel libxml2-devel xhtml1-dtds readline-devel ncurses-devel libtasn1-devel gnutls-devel augeas libudev-devel libpciaccess-devel yajl-devel sanlock-devel libpcap-devel libnl-devel avahi-devel libselinux-devel cyrus-sasl-devel parted-devel device-mapper-devel numactl-devel libcap-ng-devel netcf-devel libcurl-devel audit-libs-devel systemtap-sdt-devel


The next step is to download a source RPM for libvirt, but first we need to know which version we're running

rpm -qa | grep libvirt
libvirt-client-0.9.10-21.el6.i686

We need version 0.9.10. So, browse to http://vault.centos.org/6.3/os/Source/SPackages/ and you'll find that there's a package available there.

mkdir ~/srcbuild
cd ~/srcbuild
wget http://vault.centos.org/6.3/os/Source/SPackages/libvirt-0.9.10-21.el6.src.rpm
rpm -i libvirt-09.10-21.el6.src.rpm

Now that the RPM is installed we need to do some patching (ignore warnings about user mockbuild not existing).

wget http://pasik.reaktio.net/xen/patches/libvirt-spec-rhel6-enable-xen.patch cd /root/rpmbuild/SPECS 
cp -a libvirt.spec libvirt.spec.old
patch -p0 < ~/srcbuild/libvirt-spect-rhel6-enable-xen.patch

We're now ready to build the new RPM,

rpmbuild -bb libvirt.spec

Once this completes you should see a few lines beginning with 'wrote', this will include the path to the new RPM's, probably /root/rpmbuild/RPMS/i686. 

cd /root/rpmbuild/RPMS/i686
rpm -Uvh --force libvirt*

Finally! That's everything installed, we can now reboot

reboot


When the server has restarted, you should find that running uname -r reveals a Xen kernel in use. If not, see Troubleshooting below.

 

Using Xen

Let's start by checking that Xen is actually running

xm list
----------- Name ID Mem VCPUs State Time(s)
Domain-0 0 941 4 r----- 68.1

If instead you receive an error stating that logfiles could not be opened, try and write to any file on the filesystem. If you find that the system is read-only then it means something has gone wrong. There could be a range of causes, but begin by checking for hardware issues (especially disk) as this is the primary cause (dmesg and the syslog are your friends). If that doesn't seem to be the problem, see Read-Only filesystem on Dom0 below

Now we just need to build a VM!

As we configured libvirt we'll switch to using that, first lets check it's working OK

virsh list
------------
Id Name State
----------------------------------------------------
0 Domain-0 running

 

Creating a virtual machine

Libvirt appears to be working, so let's create a LVM based virtual machine using paravirtualisation.

virt-install --prompt --network bridge=br0 --virt-type=xen

 

Follow the prompts, for the install URL I used http://mirror.ellogroup.com/centos/6.3/os/i386/ but lookup a mirror close to you here (browse to the version you want, then find the folder os/ followed by the relevant architecture). If you receive an error about not being able to retrieve a specific file (in my case vmlinux) just use a different mirror.

See Install Troubleshooting if needed

 

Install CentOS 6.3

Once the installer starts, you'll be asked whether you want to use the text installer or start a VNC session so you can use the graphical installer. Make the choice that suits you (I went for text out of convenience).

You're on your own here, follow the CentOS documentation if you struggle with the install!

 

Access the Virtual Host

 

 

 Xen Commands

Useful commands to use with libvirt;

  • virsh start - Start a virtual machine
  • virsh shutdown - Stop a virtual machine
  • virsh destroy - Stop a virtual machine immediately without sending a shut down signal. 
  • virsh list - List all running VMs.
  • virsh console - Log into a virtual machine.
  • virsh help - List of all supported commands.

 

 

 Troubleshooting

 

Troubleshooting - Kernel Installed

 

Kernel doesn't load 

If you find that instead of a rebooted server you get one displaying the message

Error 19: Linux Kernel must be loaded before initrd

It means you forgot to change initrd to module in your Grub config. Reboot and load the second kernel so you can correct the error.

 

Server Continually Reboots

If the server gets stuck in a reboot loop, reboot into the second kernel and edit the grub menu. After the Xen config params add xsave=0

If this doesn't fix the issue, it's possible you've allocated too little RAM in dom0_virt_mem, increase the limit. You'll normally see a warning on the server's console if this is the case

 

Hangs on Boot

Depending on the spec of the server, you may think it's hung towards the end of the boot process. Mine went from a 1 minute boot to a 5 minute boot, so be patient (performance tune once we're done!) as the hypervisor needs to load.

 

Read-Only filesystem on Dom0

Check Dmesg for 'Filesystem with huge files cannot be mounted RDWR without CONFIG_LBDAF'. For whatever reason the Xen kernel comes without this config option set, so you'll need to recompile the kernel from source

mkdir /root/kernel && cd /root/kernel
git clone git://github.com/jsgf/linux-xen.git
cd linux-xen

# You should actually check the files out here really
make oldconfig

# Follow through, accepting defaults where you don't know the answer

nano .config

# Search for CONFIG_LBDAF and ensure it's set to y

make
make modules_install
make install

At this stage we need to configure the bootloader (again if you've already done it once!).

 

TroubleShooting - Creating Image

Vif could not be connected

If you receive something similar to xend.err 'Device 0 (vif) could not be connected. Hotplug scripts not working.' then there's an issue with the bridge. Re-check your configuration file to ensure everything's OK.

That failed, so digging around I found that some people have found that the NetBack (aka netbk xen-netbk) module wasn't loaded. So a quick modprobe netbk later...

 

 

 

 
 Share