Skip to content

GizmoLA.com :: xen-how-to

Base install

  • Get Xen pre-compiled Kernel from here:
    • http://www.xensource.com/download/index.html (Click on link "Get the latest open source release of Xen!" link. Get the url for compiled Tarball for "32 bit PAE SMP Release". assumes that this is multiprocessor 1u or 2u server that supports Physical Address Extension. If/when we get to 64 bit machines, then PAE is irrelevant - get 64 Bit SMP release instead.
  • wget the file
    [root@arch1 drolston]# wget http://bits.xensource.com/oss-xen/release/3.0.4-1/bin.tgz/xen-3.0.4_1-install-x86_32p.tgz
                            

     

  • Untar
    tar -xvzf xen-3.0.4_1-install-x86_32p.tgz
                            

     

  • will create a /dist directory. cd to dist, run install.sh
    sh ./install.sh
                            

     

  • Watch the output, if any components are missing, they'll need to be installed. Most likely bridge-utils (check_brctl) will be missing. For RHEL use up2date to get the package, for Centos, Yum (Yum install bridge-utils)
  • Rerun install.sh until all items are "OK"
    [root@arch1 dist]# sh ./install.sh
    Installing Xen from './install' to '/'...
     - installing for hotplug-based system
     - modifying permissions
    All done.
    Checking to see whether prerequisite tools are installed...
    Xen CHECK-INSTALL  Thu Mar 8 19:38:55 PST 2007
    Checking check_brctl: OK
    Checking check_crypto_lib: OK
    Checking check_iproute: OK
    Checking check_libvncserver: unused, OK
    Checking check_python: OK
    Checking check_python_xml: OK
    Checking check_sdl: unused, OK
    Checking check_udev: OK
    Checking check_zlib_lib: OK
    All done.
                            

     

  • Check for the xen kernel in /boot
    [root@arch1 boot]# ls -lath /boot
    total 34M
    drwxr-xr-x   2 root root 1.0K Mar  6 20:20 grub
    drwxr-xr-x  23 root root 4.0K Feb 14 18:09 ..
    drwxr-xr-x   4 root root 3.0K Feb 14 18:06 .
    lrwxrwxrwx   1 root root   21 Feb 14 17:42 vmlinuz-2.6.16-xen -> vmlinuz-2.6.16.33-xen
    lrwxrwxrwx   1 root root   21 Feb 14 17:42 vmlinuz-2.6-xen -> vmlinuz-2.6.16.33-xen
    lrwxrwxrwx   1 root root   14 Feb 14 17:42 xen-3.0.gz -> xen-3.0.4-1.gz
    lrwxrwxrwx   1 root root   14 Feb 14 17:42 xen-3.gz -> xen-3.0.4-1.gz
    lrwxrwxrwx   1 root root   14 Feb 14 17:42 xen.gz -> xen-3.0.4-1.gz
    -rw-rw-r--   1 root root  60K Jan  8 06:39 config-2.6.16.33-xen
    -rw-rw-r--   1 root root 724K Jan  8 06:39 System.map-2.6.16.33-xen
    -rw-r--r--   1 root root 3.9M Jan  8 06:39 vmlinux-syms-2.6.16.33-xen
    -rw-r--r--   1 root root 1.4M Jan  8 06:39 vmlinuz-2.6.16.33-xen
                            

     

    • In this example, the xen kernel is vmlinuz-2.6.16.33-xen.
  • Current versions of RHEL, Centos and Fedora (as well as other 2.6 kernel based distros) employ use of an "Initial Ram Disk" for OS boot loading. The ram disk image is created with the initrd command. However, prior to being able to run mkinitrd, you will probably need to run depmod, to create the symbol/map files for the Xen kernel, which will allow mkinitrd to create the Xen Ram disk image.
    depmod -v 2.6.16.33-xen
                            

     

    • You should see many files created in the directory /lib/modules/2.6.16.33-xen.
  • Now you will be able to make the mkinitrd image. After run, the /boot directory should now contain the ramdisk img file.
    mkinitrd /boot/initrd-2.6.16.33-xen.img 2.6.16.33-xen
                            

     

  • Edit the /etc/boot/grub/grub.conf file, and add a Xen section like the one below. Emulate the boot disk paths of the working install.
    Grub.conf default

    Note, that default=0 will boot the 0th (1st) configuration by default. So placing your new Xen config above the previous config will cause it to boot upon restart.

     

    default=0
    timeout=5
    splashimage=(hd0,0)/grub/splash.xpm.gz
    hiddenmenu
    title XEN (2.6.16-xen)
            root (hd0,0)
            kernel /xen.gz dom0_mem=65536 noreboot
            module /vmlinuz-2.6-xen root=/dev/VolGroup00/LogVol00
            module /initrd-2.6.16.33-xen.img
    title CentOS (2.6.9-34.108.plus.c4smp)
            root (hd0,0)
            kernel /vmlinuz-2.6.9-34.108.plus.c4smp ro root=/dev/VolGroup00/LogVol00
            initrd /initrd-2.6.9-34.108.plus.c4smp.img
                            

     

  • Disable the TLS directory. The Xen manual says this about TLS:
    Xen and TLS

    Users of the XenLinux 2.6 kernel should disable Thread Local Storage (TLS) (e.g. by doing a mv /lib/tls /lib/tls.disabled) before attempting to boot a XenLinux kernel2.4. You can always reenable TLS by restoring the directory to its original location (i.e. mv /lib/tls.disabled /lib/tls).
    The reason for this is that the current TLS implementation uses segmentation in a way that is not permissible under Xen. If TLS is not disabled, an emulation mode is used within Xen which reduces performance substantially. To ensure full performance you should install a `Xen-friendly' (nosegneg) version of the library.

     

mv /lib/tls /lib/tls.disabled
                
64 bit

If 64 bit linux, then check /lib64/

 

  • reboot or shutdown -h now
  • When machine reboots, verify xen kernel is running using uname
    [root@arch1 xen]# uname -a
    Linux arch1 2.6.16.33-xen #1 SMP Mon Jan 8 14:39:02 GMT 2007 i686 i686 i386 GNU/Linux
                            

     

  • Start the Xend service
    service xend start
                            

     

    Libcrypto problems

    I've noticed there seems to be an issue with libcrypto on 4.3 and 4.4, where the version differs from when the xend service requires. If you get an error, go into /lib and create a soft link file pointing to the existing libcrypto using ln -s. You should see something like the directory list below when done. Then restart the service.

     

    lrwxrwxrwx  1 root root   24 Feb 14 18:57 libcrypto.so.0.9.7 -> /lib/libcrypto.so.0.9.7a
    -rwxr-xr-x  1 root root 919K Sep 29 08:09 libcrypto.so.0.9.7a
                            

     

  • check that Xen is running ok:
    [root@arch1 lib]# xm list
    Name                                      ID   Mem VCPUs      State   Time(s)
    Domain-0                                   0    65     2     r-----   7472.6
                            

     

  • Now you're ready to setup a guest!

Guest Host

  • Most of our machines have used the Centos4 image from Jailtime.org.
  • Editing a File system image using loopback device:
    [root@arch1 xen]# mount -o loop /opt/xen/centos.4-3.img /mnt/xenfs
    [root@arch1 xen]# mount
    /dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
    none on /proc type proc (rw)
    none on /sys type sysfs (rw)
    none on /dev/pts type devpts (rw,gid=5,mode=620)
    /dev/cciss/c0d0p1 on /boot type ext3 (rw)
    none on /dev/shm type tmpfs (rw)
    none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
    sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
    /opt/xen/centos.4-3.img on /mnt/xenfs type ext3 (rw,loop=/dev/loop4)
                            

     

    • I recommend to chroot before editing, to make sure you don't accidently edit the host server configs.
      [root@arch1 lib]# cd /mnt/xenfs
      [root@arch1 xenfs]# chroot .
      bash-3.00# ls
      bin  boot  dev  etc  halt  home  initrd  lib  lost+found  media  mnt  opt  proc  root  sbin  selinux  srv  sys  tmp  usr  var
      bash-3.00#
                                      

       

  • At minimum you want to edit the /etc/sysconfig/network
    NETWORKING=yes
    HOSTNAME=arch2
    GATEWAY=10.100.105.1
                            

     

    • - and /etc/sysconfig/network-scripts/ifcfg-eth0
      TYPE=Ethernet
      DEVICE=eth0
      BOOTPROTO=static
      ONBOOT=yes
      IPADDR=10.100.105.41
      NETMASK=255.255.255.0
      NETWORK=10.100.105.0
      BROADCAST=10.100.105.255
                                      
      • – and edit/create /etc/resolv.conf
        domain j2noc.com
        nameserver 10.2.2.111
        nameserver 10.254.254.249
                                                

 

  • Xen config files
    • Put them in /etc/xen
    • Sample file, hopefully this is self explanatory. See Xen site for details.
      kernel = "/boot/vmlinuz-2.6.16.33-xen"
      memory = 128
      name = "centos2.4-3"
      vif = [ '' ]
      ip = "10.100.105.42"
      disk = ['file:/opt/xen/centos/centos2.4-3.img,sda1,w', 'file:/opt/xen/centos/centos2.swap,sda2,w']
      root = "/dev/sda1 ro"
                                      

       

Copying module files to the guest

If the host OS has devices installed that are not part of the guest, you can surface support for those by copying the /lib/modules for the xen kernel into the guest fs.

  • Assumes that you created map files using depmod previously.
  • mount -o loop the guest filesystem.
  • cp -a /lib/modules/2.6.16.33-xen /mnt/xenfs/lib/modules

Starting the Guest

  • Use xm create
    [root@arch1 xen]# xm list
    Name                                      ID   Mem VCPUs      State   Time(s)
    Domain-0                                   0    65     2     r-----   7475.6
    centos.4-3                                 1   128     1     -b----   4230.5
    centos2.4-3                                2   128     1     -b----    314.8
    centos5.4-3                                9   512     1     -b----   5418.3
    [root@arch1 xen]# xm create centos3.4-3.xen.cfg
    Using config file "./centos3.4-3.xen.cfg".
    Started domain centos3.4-3
    [root@arch1 xen]# xm list
    Name                                      ID   Mem VCPUs      State   Time(s)
    Domain-0                                   0    65     2     r-----   7478.1
    centos.4-3                                 1   128     1     -b----   4230.5
    centos2.4-3                                2   128     1     -b----    314.8
    centos3.4-3                               13   256     1     -b----      1.0
    centos5.4-3                                9   512     1     -b----   5418.4
    [root@arch1 xen]#
                            
  • Console into the running guest
    xm console centos2.4-3
                            
    • If you're fast you may catch boot. If you're slow and get a blank screen, hit Return.
      • Exiting back to the Host is a bit like screen
      • Use Ctrl-]
  • If you want to be guaranteed to watch the boot process, you can start the guest in console mode
    xm console centos2.4-3 -c
                            

Expanding the size of a filesystem

  • Bring down the guest instance if it's up!!!
    • Given you want to expand a virtual disk called filesystem.image by 1024MB (1G), do the following:
  • dd if=/dev/zero bs=1M count=1024 >> filesystem.image
  • e2fsck -f filesystem.image
  • resize2fs filesystem.image
  • e2fsck -f filesystem.image

Posted by David Rolston | on