Nested XEN on VMware ESXi

Quick guide how to deploy nested XEN on VMware ESXi

You may want to run XEN s(not a XEN server) for several reasons as virtual machine which run on ESXi.

Gotchas:

Only one vCPU :-/ – otherwise you will get error as show below when boot you CentOS with XEN kernel. There is a bug in VMXNET3 driver module on Linux which has something to do with number of vCPU assigned to VM, which is different then power of 2. I tested several configurations with 2, 3,4, vCPU and non of them was working. Only VM with one vCPU works as expected.

Failed to activate dev eth0:error 1

An error message similar to the following might be written to vmware.log:

VMXNET3 user: failed to activate Ethernet1 with status 0xbad0007

Theoretically should be solved by VMwareKB 2003484 but it is not 🙂

Lab Configuration:

  • Nutanix block NTNX-1050
  • NOS 4.0.1
  • VMware vSphere ESi 5.5 U1

Hypervisor settings

  • enable promiscuous mode on vSwitch which you plan to use for nested XEN

Virtual Machine configuration

  • virtual hardware 8
  • 6GB vRAM
  • 1 vCPU
  • 16 GB hard disk
  •  CentOS 6 64 bit baseline

Nested XEN on VMware ESXi – Installation

 

More about installation you can read from below link

  • http://wiki.centos.org/HowTos/Xen/Xen4QuickStart
  • http://wiki.centos.org/HowTos/Xen/Xen4QuickStart/Xen4Libvirt

Post-installation

Check if SELinux is disabled /etc/sysconfig/selinux

check if XEN server is up and running, type xm info

If you plan to use bridged network model – Disable network manager because it does not support bridged networks anyway

Install libvirt packages

Reboot host

Nested XEN on VMware ESXi – Configuration

Networking

All ifcfg files are case sensitive. In particular Bridge must be written exactly as shown.
Example 1: A physical network device on a bridge: /etc/sysconfig/network-scripts/ifcfg-eth0

Example 3: A single bridged network configured with a static local IP address: /etc/sysconfig/network-scripts/ifcfg-xenbr0

If you need to keep IP address on bridge from eth0 just copy it over to bridge configuration file

Restart networking:

  • services network restart

Check bridge configuration:

Our bridge xenbr0 is visible and it is bridged to eth0 physical interface

 Disable netfilter for bridges. However you can alternatively you can configure iptables to allow all traffic to be forwarded across the bridge as follows:

 

More about networking configuration on XEN you can read on below

  • http://wiki.xen.org/wiki/Network_Configuration_Examples_(Xen_4.1%2B)
  • http://wiki.xenproject.org/wiki/Xen_Networking

 

Nested XEN on VMware ESXi – Spin the first VM up

Another method for a remote install is to connect to the Dom0 machine via ssh and become the root user, and then use virt-install and the console to do an install. This command (as root, from the command line) will allow a text install remotely:

In the above example, the meanings are:

  • -d – Debug mode, lots of text and full config files printed for debuging
  • -n TestVM1 – The name of the VM
  • -r 2048 – Ram size (2048 MB or 2 GB)
  • –vcpus=1 – Number of Virtual CPUS
  • –disk=/var/lib/libvirt/images/TestVM1.img,size=8 – disk image location and size in GB
  • –nographics – since we do not have GUI installed on our Dom0, do not use VNC, etc.
  • -p – use para-virtualization
  • -l “http://mirror.centos.org/centos/6/os/x86_64” – Location of a centos tree
  • –extra-args=”text console=com1 utf8 console=hvc0″ – this tells linux to use com1 and xen to use hvc0 so that you can do a text install via the console

Note: This type of install is only console based, so only a text install can be done this way.

 

Switch over to graphical mode

CentOS 6.5 text mode

CentOS 6.5 text mode

Graphical mode

CentOS 6.5 on XEN installation over VNC

CentOS 6.5 on XEN installation over VNC

After installation you you can connect to VM over SSH.  You can also verify if VM has been registered and visible by XEN hypervisor.

 

 

Artur Krzywdzinski

Artur is Consulting Architect at Nutanix. He has been using, designing and deploying VMware based solutions since 2005 and Microsoft since 2012. He specialize in designing and implementing private and hybrid cloud solution based on VMware and Microsoft software stacks, datacenter migrations and transformation, disaster avoidance. Artur holds VMware Certified Design Expert certification (VCDX #077).