It's been a while since I've had to set up a virtualisation server, but today I needed to configure a brand-new install of CentOS 6 to act as a virtual host. The hardware doesn't have virtualisation support (an old G3) so I had to use Xen so that paravirtualisation was available (not currently supported by KVM). Oops, not so easy now that Xen isn't included by default, Red Hat having opted to use KVM instead.

Despite that, getting things set up isn't that hard, although not nearly as easy as it was when you could just

yum install xen

 This documentation details the steps you'll need to follow.

So let's start with a few basic pre-requisites;

yum install sanlock presto libblkid

SELinux

As helpful as it is for security, it's going to interfere like nobodies business! Either disable it or make permissive

nano /etc/sysconfig/selinux  
# Change SELINUX=enabled to SELINUX=disabled

This won't be effective until we reboot, but let's get a few bits done first!

Bridge the network

The package should already be installed, but just to be safe, run

yum install bridge-utils

Next we need to configure the bridge

cp /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-br0

Then we edit the configuration file we just created, so that we get something like

DEVICE="br0"
NM_CONTROLLED="yes"
ONBOOT=yes
TYPE=Bridge
BOOTPROTO=none
IPADDR=192.168.1.175
PREFIX=24
GATEWAY=192.168.1.254
DNS1=129.168.1.254
DNS2=8.8.4.4
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="System br0"

Next we need to edit the original file and add a line at the end, so

nano /etc/sysconfig/network-scripts/ifcfg-eth0 
#add a line at the bottom 

BRIDGE=br0

So now that's set up, we can move on and proceed with the install.

Kernel Install

Note: I had a few issues with the kernel we'll be downloading in this step, and eventually resorted to compiling and installing the upstream kernel myself. Steps for that are below.

Use wget to install the aussie repo (I'm sure there must be a closer one!). If you don't have wget start with a yum install wget.

yum install http://au1.mirror.crc.id.au/repo/kernel-xen-release-6-3.noarch.rpm

We now need to go off on a slight tangent from the official walkthroughs now, as attempting to install at this point will tell us that qemu-img isn't available. So, we need to install from repoforge as follows

yum install http://pkgs.repoforge.org/qemu/qemu-img-0.15.0-1.el6.rfx.i686.rpm

Then we can proceed to run

yum install kernel-xen xen

To download and build the bits we actually want! Once that's complete there are a few more steps to go through before we're quite ready to reboot.

Bootloader

First we need to reconfigure grub to boot the Xen hypervisor before the kernel

nano /boot/grub/menu.1st

Change

title CentOS (2.6.32.57-2.el6xen.i686)
root (hd0,0)
kernel /vmlinuz-2.6.32.57-2.el6xen.i686 ro root=/dev/mapper/vg_virty-lv_root rd_NO_LUKS KEYBOARDTYPE=pc KEYTABLE=uk LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkerne$
initrd /initramfs-2.6.32.57-2.el6xen.i686.img

So that kernel becomes module. We also need to add a line to load the xen hypervisor so should end up with something like

title CentOS (2.6.32.57-2.el6xen.i686)
root (hd0,0)
kernel /xen.gz dom0_mem=2048M cpufreq=xen dom0_max_vcpus=2 dom0_vcpus_pin
module /vmlinuz-2.6.32.57-2.el6xen.i686 ro root=/dev/mapper/vg_virty-lv_root rd_NO_LUKS KEYBOARDTYPE=pc KEYTABLE=uk LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkerne$
module /initramfs-2.6.32.57-2.el6xen.i686.img

You'll notice dom0_max_vcpus=2. As the G3 I was building on has 2 cores, I set this to 2. Adjust as necessary for your system (the same goes for dom_0_mem=2048M)

 Double check that grub default is set to 0 and then save and close the grub file.

LibVirt

We now need to install libvirt, unfortunately there's a few extra steps as the repo builds no longer include support for Xen, so we need to do the following

yum install libvirt python-virtinst

to install the files from the repo.

Now we'll install a few pre-requisites so that we can rebuild the src rpm's with Xen support

yum groupinstall 'Development Tools'

yum install python-devel xen-devel libxml2-devel xhtml1-dtds readline-devel ncurses-devel libtasn1-devel gnutls-devel augeas libudev-devel libpciaccess-devel yajl-devel sanlock-devel libpcap-devel libnl-devel avahi-devel libselinux-devel cyrus-sasl-devel parted-devel device-mapper-devel numactl-devel libcap-ng-devel netcf-devel libcurl-devel audit-libs-devel systemtap-sdt-devel

The next step is to download a source RPM for libvirt, but first we need to know which version we're running

rpm -qa | grep libvirt

libvirt-client-0.9.10-21.el6.i686

We need version 0.9.10. So, browse to http://vault.centos.org/6.3/os/Source/SPackages/ and you'll find that there's a package available there.

mkdir ~/srcbuild

cd ~/srcbuild

wget http://vault.centos.org/6.3/os/Source/SPackages/libvirt-0.9.10-21.el6.src.rpm

rpm -i libvirt-09.10-21.el6.src.rpm

Now that the RPM is installed we need to do some patching (ignore warnings about user mockbuild not existing).

wget http://pasik.reaktio.net/xen/patches/libvirt-spec-rhel6-enable-xen.patch cd /root/rpmbuild/SPECS 
cp -a libvirt.spec libvirt.spec.old 
patch -p0 < ~/srcbuild/libvirt-spect-rhel6-enable-xen.patch

We're now ready to build the new RPM,

rpmbuild -bb libvirt.spec

Once this completes you should see a few lines beginning with 'wrote', this will include the path to the new RPM's, probably /root/rpmbuild/RPMS/i686. 

cd /root/rpmbuild/RPMS/i686

rpm -Uvh --force libvirt*

Finally! That's everything installed, we can now reboot

reboot

When the server has restarted, you should find that running uname -r reveals a Xen kernel in use. If not, see Troubleshooting below.

Using Xen

Let's start by checking that Xen is actually running

xm list
----------- Name ID Mem VCPUs State Time(s)
Domain-0 0 941 4 r----- 68.1

If instead you receive an error stating that logfiles could not be opened, try and write to any file on the filesystem. If you find that the system is read-only then it means something has gone wrong. There could be a range of causes, but begin by checking for hardware issues (especially disk) as this is the primary cause (dmesg and the syslog are your friends). If that doesn't seem to be the problem, see Read-Only filesystem on Dom0 below

Now we just need to build a VM!

As we configured libvirt we'll switch to using that, first lets check it's working OK

virsh list

------------

Id Name State
----------------------------------------------------
 0 Domain-0 running

Creating a virtual machine

Libvirt appears to be working, so let's create a LVM based virtual machine using paravirtualisation.

virt-install --prompt --network bridge=br0 --virt-type=xen

Follow the prompts, for the install URL I used http://mirror.ellogroup.com/centos/6.3/os/i386/ but lookup a mirror close to you here (browse to the version you want, then find the folder os/ followed by the relevant architecture). If you receive an error about not being able to retrieve a specific file (in my case vmlinux) just use a different mirror.

See Install Troubleshooting if needed

Install CentOS 6.3

Once the installer starts, you'll be asked whether you want to use the text installer or start a VNC session so you can use the graphical installer. Make the choice that suits you (I went for text out of convenience).

You're on your own here, follow the CentOS documentation if you struggle with the install!

Access the Virtual Host

 Troubleshooting

Error 19: Linux Kernel must be loaded before initrd

mkdir /root/kernel && cd /root/kernel
git clone git://github.com/jsgf/linux-xen.git 
cd linux-xen 

# You should actually check the files out here really
make oldconfig

# Follow through, accepting defaults where you don't know the answer

nano .config

# Search for CONFIG_LBDAF and ensure it's set to y

make
make modules_install
make install

At this stage we need to configure the bootloader (again if you've already done it once!).