Setting up Xen on Ubuntu 12.04

In order to be able to run some destructive testing on customer's systems, I needed to set up virtual servers. The hardware I have spare doesn't have virtualisation hardware, so KVM is out. Due to time constraints, it means my usual choice of CentOS is out (as RH have dropped support for Xen in RHEL6 and I lack the time to risk delays).

So, I figured I'd use Ubuntu 12.04 (Precise Pangolin) for my Dom0.

The hardware is an old HP G3 with dual Xeon processors and 3GB RAM. It's never going to be much use for testing dedicated servers, but as a lot of VPS configurations are set to 1 core/ 1GB RAM it just about passes the mark.

This documentation details the steps I took to get Xen installed and set up - every step listed can be run via SSH (assuming you do a net install of the base system), but be aware that if something goes wrong you might need physical access to the system to resolve it.

 

Install Ubuntu

The first step is obvious - do a bare metal install of Ubuntu 12.04. When asked about partitioning, select Guided use the entire disk and set up LVM. When asked for the size of the volume group enter 25GB as we want the space free for our hosts later.

When the installer asks what to install, select Basic Ubuntu Server and Virtual Machine Host. The latter will install KVM, but helpfully adds things like libvirt (we're not actually going to use it in this tutorial, but it's handy to have to simplify creation in future!) - we can remove KVM and then replace it with Xen once the install's done.

 

System Installed

 Once the OS is installed, we want to remove KVM and install Xen

sudo -s
apt-get remove kvm
# If you've a 64 bit CPU
apt-get install xen-hypervisor-amd64
sed -i 's/GRUB_DEFAULT=.*\+/GRUB_DEFAULT="Xen 4.1-amd64"/' /etc/default/grub

# If you've a 32 bit CPU
apt-get install xen-hypervisor-i386
sed -i 's/GRUB_DEFAULT=.*\+/GRUB_DEFAULT="Xen 4.1-i386"/' /etc/default/grub

sed -i 's/TOOLSTACK=.*\+/TOOLSTACK="xm"/' /etc/default/xen
mv /etc/grub.d/10_linux /etc/grub.d/50_linux

Note: If you forget to switch the files in /etc/grub.d around, once you've rebooted you may receive the error ERROR: Can't find hypervisor information in sysfs! when you try to run xm list

One thing you may want to consider doing, is to limit the amount of RAM that Dom0 can take, otherwise you may find you can't launch any virtual machines!, to do so (even if you don't want to, the last two steps are necessary) - run all as root or prefix with sudo.

nano /etc/default/grub
# Add the following line but adjust the RAM as desired
GRUB_CMDLINE_XEN="dom0_mem=512M"
# Save and close

update-grub
reboot

When the system comes back up, you should be able to

sudo xm list
Name ID Mem VCPUs State Time(s)
Domain-0 0 512 4 r----- 33.0

Xen's now set up, but we need to create a bridged network next

 

Networking

To set up the networking we simply need to edit /etc/network/interfaces and create the interface, then reload the networking systems

sudo nano /etc/network/interfaces
# Add the following (assuming you want virtual machines to use eth0)
auto xenbr0
iface xenbr0 inet dhcp
bridge_ports eth0

# Save and close

sudo /etc/init.d/networking restart

Now we can move onto creating our first DomU

 

Manually creating a Ubuntu DomU

Let's start off easy, we've already installed one Ubuntu system so lets create one as a virtual host. First we need to get the name of the LVM Virtual Group so we can add a volume to it - in the output generated below it's the VG column

sudo pvs

Now we have that, we can create a Logical Volume - this will be the hard drive that our DomU sees, I'm calling the volume ubuntu (original huh?)

sudo lvcreate -L 4G -n ubuntu /dev/

Next we're going to grab the netboot images so that we can do a network install

sudo -s
mkdir -p /var/lib/xen/images/ubuntu-netboot
cd /var/lib/xen/images/ubuntu-netboot

#64 bit users
wget http://mirror.as29550.net/archive.ubuntu.com/ubuntu/dists/precise/main/installer-amd64/current/images/netboot/xen/initrd.gz
wget http://mirror.as29550.net/archive.ubuntu.com/ubuntu/dists/precise/main/installer-amd64/current/images/netboot/xen/vmlinuz

#32 bit users
wget http://mirror.as29550.net/archive.ubuntu.com/ubuntu/dists/precise/main/installer-i386/current/images/netboot/xen/initrd.gz
wget http://mirror.as29550.net/archive.ubuntu.com/ubuntu/dists/precise/main/installer-i386/current/images/netboot/xen/vmlinuz

Note: if you're on a 32bit machine and download the 64bit images, when you try to start the DomU you'll probably receive " 'Out of memory', "xc_dom_boot_mem_init: can't allocate low memory for domain\n "

Next we create a config file to define the virtual machine, we're going to give it 256MB RAM and call it ubuntu (again, original). Don't forget to swap out the Volume Group name we looked up earlier!

nano /etc/xen/ubuntu.cfg
# Insert the following
name = "ubuntu"
memory = 256
disk = ['phy:/dev/<VG NAME>/ubuntu,xvda,w']
vif = [' ']
kernel = "/var/lib/xen/images/ubuntu-netboot/vmlinuz"
ramdisk = "/var/lib/xen/images/ubuntu-netboot/initrd.gz"
extra = "debian-installer/exit/always_halt=true -- console=hvc0"

We're now ready to launch the DomU and run the installer, so without further ado:

xm create /etc/xen/ubuntu.cfg -c

You should see a Linux system start booting in your console, follow the installer through. Once complete, the DomU will shutdown. Now we're going to want to set-up a bootloader and configuring our DomU to use it

sudo -s
ln -s /usr/lib/xen-4.1/bin/pygrub /usr/bin/pygrub
nano /etc/xen/ubuntu.cfg

#Add this line
bootloader = "pygrub"

#Remove or comment out the kernel, ramdisk and extra lines
#Save and close

We can start the virtual machine with the same command we used before

sudo xm create /etc/xen/ubuntu.cfg -c

See the troubleshooting sectionif you receive the following error

Error: (2, 'Invalid kernel', 'elf_xen_note_check: ERROR: Will only load images built for the generic loader or Linux images')

 

Using Install media

We previously used a netboot image to run the installer, but we're not always going to want to do that - sometimes we want to use an ISO image. The steps are exactly the same, aside from the configuration file for the domU (I've assumed the ISO is stored in /root/). So in /etc/xen/ubuntu.cfg I would have

name = "ubuntu"
memory = 256
disk = ['phy:/dev/{VG NAME}/ubuntu,xvda,w','file:/root/MYINSTALLDISK.iso,hdc:cdrom,r']
vif = [' ']
extra = "debian-installer/exit/always_halt=true -- console=hvc0"
vnc = 1
boot="dc"

What we've changed is the Disk definition (to include the CDROM) and we've added a definition for the boot order. The inclusion of the VNC line allows us to connect to the server via VNC (which is useful if your installer only has a graphical mode).

We've also removed the paths to the kernel and ramdisk because they're on the CD/DVD and it's own bootloader will take care of that.

Once the install is complete, we'll need to either remove the CDROM definition, or at least swap the boot order round (to cd)

 

Starting DomU's automatically

We've gone to the trouble of creating our VM's, but as it stands whenever the host system is restarted, we'll have to manually fire the things up! Setting DomU's to start automatically is simply a case of creating a symbolic link though

sudo ln -s /etc/xen/ubuntu.cfg /etc/xen/auto 

Obviously use the relevant configuration filename and path

 

Creating a Clone

Important: Don't do this when the DomU is running - there'll be things changing on disk all the time!

There may be a myriad of reasons why you want to create a clone of the image we just created - as I'm primarily using the VM's as testing environments I want to be able to restore the system back to it's original state quickly, or to launch several identical systems so that I can play around with other bits.

We'll still need to create the Xen config files, but the first step is to clone the Logical Volume

# This command is much quicker, but can be buggy sometimes
lvcreate -L 25G -s -n ubuntu_clone /dev/<VGNAME>/ubuntu

#These have a better chance of success but can take quite a while
lvcreate -L 25G -n ubuntu_clone <VGNAME>
dd if=/dev/<VGNAME>/ubuntu of=/dev/<VGNAME>/ubuntu_clone

Obviously if your original logical volume wasn't called ubuntu you'll need to change the name

We can now create a new Xen config file referencing our new Logical volume. Hint: You can use more or less any name you want instead of ubuntu_clone, so long as it's unique to the LVM Virtual Group

 

Troubleshooting an installed VM

If when trying to start the Ubuntu VM we created, you receive an error about an invalid kernel, it's because the Xen compatible kernel wasn't installed (if there's an option in the installer, I must have missed it!). There are two packages we need to install.

In theory it's possible to mount the Logical volumes contained within our VM's logical volume, but in all honesty it's just as easy to get Xen to do that for us. Reconfigure the Xen config file to that state it was for install (i.e. remove bootloader and re-insert Kernel, ramdisk and extra).

Fire up the VM, and as soon as you hit the first installation question, press Esc on your keyboard. Select Execute a Shell from the resulting menu and then follow the steps below

mount /dev/xvda1 /mnt
echo "91.189.91.13 security.ubuntu.com" >> /mnt/etc/hosts
#This next line may be incorrect for you.. we'll fix in a minute if so echo "91.189.92.200 gb.archive.ubuntu.com" >> /mnt/etc/hosts
ifconfig eth0 up
dhclient eth0 #Ignore errors about resolv.conf
mount -o bind /proc /mnt/proc
mount -o bind /dev /mnt/dev
chroot /mnt
apt-get install linux-virtual grub-legacy-ec2

If apt fails because it couldn't look up an address, the quickest way is to ping that address from another machine and then add a line to /etc/hosts, then re-run. Once the packages are installed

exit
umount /mnt/dev
umount /mnt/proc
umount /mnt
exit

Finally, choose Abort Installation and wait whilst your VM shuts down. Re-enable the bootloader declaration in the config file and disable kernel, ramdisk and extra. The VM should run successfully when you exectute

xm create /etc/xen/ubuntu.cfg -c

 

Non-Work Uses

We've covered why you might want to set this up for work, but there are just as many non-work reasons, including Creating a DOS Games Server!