Setting up Xen on Ubuntu 12.04
In order to be able to run some destructive testing on customer's systems, I needed to set up virtual servers. The hardware I have spare doesn't have virtualisation hardware, so KVM is out. Due to time constraints, it means my usual choice of CentOS is out (as RH have dropped support for Xen in RHEL6 and I lack the time to risk delays).
So, I figured I'd use Ubuntu 12.04 (Precise Pangolin) for my Dom0.
The hardware is an old HP G3 with dual Xeon processors and 3GB RAM. It's never going to be much use for testing dedicated servers, but as a lot of VPS configurations are set to 1 core/ 1GB RAM it just about passes the mark.
This documentation details the steps I took to get Xen installed and set up - every step listed can be run via SSH (assuming you do a net install of the base system), but be aware that if something goes wrong you might need physical access to the system to resolve it.
Install Ubuntu
The first step is obvious - do a bare metal install of Ubuntu 12.04. When asked about partitioning, select Guided use the entire disk and set up LVM. When asked for the size of the volume group enter 25GB as we want the space free for our hosts later.
When the installer asks what to install, select Basic Ubuntu Server and Virtual Machine Host. The latter will install KVM, but helpfully adds things like libvirt (we're not actually going to use it in this tutorial, but it's handy to have to simplify creation in future!) - we can remove KVM and then replace it with Xen once the install's done.
System Installed
Once the OS is installed, we want to remove KVM and install Xen
sudo -s
apt-get remove kvm
# If you've a 64 bit CPU
apt-get install xen-hypervisor-amd64
sed -i 's/GRUB_DEFAULT=.*\+/GRUB_DEFAULT="Xen 4.1-amd64"/' /etc/default/grub
# If you've a 32 bit CPU
apt-get install xen-hypervisor-i386
sed -i 's/GRUB_DEFAULT=.*\+/GRUB_DEFAULT="Xen 4.1-i386"/' /etc/default/grub
sed -i 's/TOOLSTACK=.*\+/TOOLSTACK="xm"/' /etc/default/xen
mv /etc/grub.d/10_linux /etc/grub.d/50_linux
Note: If you forget to switch the files in /etc/grub.d around, once you've rebooted you may receive the error ERROR: Can't find hypervisor information in sysfs! when you try to run xm list
One thing you may want to consider doing, is to limit the amount of RAM that Dom0 can take, otherwise you may find you can't launch any virtual machines!, to do so (even if you don't want to, the last two steps are necessary) - run all as root or prefix with sudo.
nano /etc/default/grub
# Add the following line but adjust the RAM as desired
GRUB_CMDLINE_XEN="dom0_mem=512M"
# Save and close
update-grub
reboot
When the system comes back up, you should be able to
sudo xm list
Name ID Mem VCPUs State Time(s)
Domain-0 0 512 4 r----- 33.0
Xen's now set up, but we need to create a bridged network next
Networking
To set up the networking we simply need to edit /etc/network/interfaces and create the interface, then reload the networking systems
sudo nano /etc/network/interfaces
# Add the following (assuming you want virtual machines to use eth0)
auto xenbr0
iface xenbr0 inet dhcp
bridge_ports eth0
# Save and close
sudo /etc/init.d/networking restart
Now we can move onto creating our first DomU
Manually creating a Ubuntu DomU
Let's start off easy, we've already installed one Ubuntu system so lets create one as a virtual host. First we need to get the name of the LVM Virtual Group so we can add a volume to it - in the output generated below it's the VG column
sudo pvs
Now we have that, we can create a Logical Volume - this will be the hard drive that our DomU sees, I'm calling the volume ubuntu (original huh?)
sudo lvcreate -L 4G -n ubuntu /dev/
Next we're going to grab the netboot images so that we can do a network install
sudo -s
mkdir -p /var/lib/xen/images/ubuntu-netboot
cd /var/lib/xen/images/ubuntu-netboot
#64 bit users
wget http://mirror.as29550.net/archive.ubuntu.com/ubuntu/dists/precise/main/installer-amd64/current/images/netboot/xen/initrd.gz
wget http://mirror.as29550.net/archive.ubuntu.com/ubuntu/dists/precise/main/installer-amd64/current/images/netboot/xen/vmlinuz
#32 bit users
wget http://mirror.as29550.net/archive.ubuntu.com/ubuntu/dists/precise/main/installer-i386/current/images/netboot/xen/initrd.gz
wget http://mirror.as29550.net/archive.ubuntu.com/ubuntu/dists/precise/main/installer-i386/current/images/netboot/xen/vmlinuz
Note: if you're on a 32bit machine and download the 64bit images, when you try to start the DomU you'll probably receive " 'Out of memory', "xc_dom_boot_mem_init: can't allocate low memory for domain\n "
Next we create a config file to define the virtual machine, we're going to give it 256MB RAM and call it ubuntu (again, original). Don't forget to swap out the Volume Group name we looked up earlier!
nano /etc/xen/ubuntu.cfg
# Insert the following
name = "ubuntu"
memory = 256
disk = ['phy:/dev/<VG NAME>/ubuntu,xvda,w']
vif = [' ']
kernel = "/var/lib/xen/images/ubuntu-netboot/vmlinuz"
ramdisk = "/var/lib/xen/images/ubuntu-netboot/initrd.gz"
extra = "debian-installer/exit/always_halt=true -- console=hvc0"
We're now ready to launch the DomU and run the installer, so without further ado:
xm create /etc/xen/ubuntu.cfg -c
You should see a Linux system start booting in your console, follow the installer through. Once complete, the DomU will shutdown. Now we're going to want to set-up a bootloader and configuring our DomU to use it
sudo -s
ln -s /usr/lib/xen-4.1/bin/pygrub /usr/bin/pygrub
nano /etc/xen/ubuntu.cfg
#Add this line
bootloader = "pygrub"
#Remove or comment out the kernel, ramdisk and extra lines
#Save and close
We can start the virtual machine with the same command we used before
sudo xm create /etc/xen/ubuntu.cfg -c
See the troubleshooting sectionif you receive the following error
Error: (2, 'Invalid kernel', 'elf_xen_note_check: ERROR: Will only load images built for the generic loader or Linux images')
Using Install media
We previously used a netboot image to run the installer, but we're not always going to want to do that - sometimes we want to use an ISO image. The steps are exactly the same, aside from the configuration file for the domU (I've assumed the ISO is stored in /root/). So in /etc/xen/ubuntu.cfg I would have
name = "ubuntu"
memory = 256
disk = ['phy:/dev/{VG NAME}/ubuntu,xvda,w','file:/root/MYINSTALLDISK.iso,hdc:cdrom,r']
vif = [' ']
extra = "debian-installer/exit/always_halt=true -- console=hvc0"
vnc = 1
boot="dc"
What we've changed is the Disk definition (to include the CDROM) and we've added a definition for the boot order. The inclusion of the VNC line allows us to connect to the server via VNC (which is useful if your installer only has a graphical mode).
We've also removed the paths to the kernel and ramdisk because they're on the CD/DVD and it's own bootloader will take care of that.
Once the install is complete, we'll need to either remove the CDROM definition, or at least swap the boot order round (to cd)
Starting DomU's automatically
We've gone to the trouble of creating our VM's, but as it stands whenever the host system is restarted, we'll have to manually fire the things up! Setting DomU's to start automatically is simply a case of creating a symbolic link though
sudo ln -s /etc/xen/ubuntu.cfg /etc/xen/auto
Obviously use the relevant configuration filename and path
Creating a Clone
Important: Don't do this when the DomU is running - there'll be things changing on disk all the time!
There may be a myriad of reasons why you want to create a clone of the image we just created - as I'm primarily using the VM's as testing environments I want to be able to restore the system back to it's original state quickly, or to launch several identical systems so that I can play around with other bits.
We'll still need to create the Xen config files, but the first step is to clone the Logical Volume
# This command is much quicker, but can be buggy sometimes lvcreate -L 25G -s -n ubuntu_clone /dev/<VGNAME>/ubuntu
#These have a better chance of success but can take quite a while
lvcreate -L 25G -n ubuntu_clone <VGNAME>
dd if=/dev/<VGNAME>/ubuntu of=/dev/<VGNAME>/ubuntu_clone
Obviously if your original logical volume wasn't called ubuntu you'll need to change the name
We can now create a new Xen config file referencing our new Logical volume. Hint: You can use more or less any name you want instead of ubuntu_clone, so long as it's unique to the LVM Virtual Group
Troubleshooting an installed VM
If when trying to start the Ubuntu VM we created, you receive an error about an invalid kernel, it's because the Xen compatible kernel wasn't installed (if there's an option in the installer, I must have missed it!). There are two packages we need to install.
In theory it's possible to mount the Logical volumes contained within our VM's logical volume, but in all honesty it's just as easy to get Xen to do that for us. Reconfigure the Xen config file to that state it was for install (i.e. remove bootloader and re-insert Kernel, ramdisk and extra).
Fire up the VM, and as soon as you hit the first installation question, press Esc on your keyboard. Select Execute a Shell from the resulting menu and then follow the steps below
mount /dev/xvda1 /mnt
echo "91.189.91.13 security.ubuntu.com" >> /mnt/etc/hosts
#This next line may be incorrect for you.. we'll fix in a minute if so echo "91.189.92.200 gb.archive.ubuntu.com" >> /mnt/etc/hosts
ifconfig eth0 up
dhclient eth0 #Ignore errors about resolv.conf
mount -o bind /proc /mnt/proc
mount -o bind /dev /mnt/dev
chroot /mnt
apt-get install linux-virtual grub-legacy-ec2
If apt fails because it couldn't look up an address, the quickest way is to ping that address from another machine and then add a line to /etc/hosts, then re-run. Once the packages are installed
exit
umount /mnt/dev
umount /mnt/proc
umount /mnt
exit
Finally, choose Abort Installation and wait whilst your VM shuts down. Re-enable the bootloader declaration in the config file and disable kernel, ramdisk and extra. The VM should run successfully when you exectute
xm create /etc/xen/ubuntu.cfg -c
Non-Work Uses
We've covered why you might want to set this up for work, but there are just as many non-work reasons, including Creating a DOS Games Server!