Checking for swappers on XenServer

This article will outline the steps I take to check for swappers on my XenServer and XenClassic setups.

XenServer

• Log into the host the slice/instance is located on
• Run iostat to check for any swappers on this host:

$ iostat -xkd 1

• If swappers are found, cat the device of the slice to get its minor number:

$ cat /sys/block/tdk/dev
432:20 # this is the major:minor number of the device

• Get the UUID for this slice

The `tap-ctl list` command can be used to list all VHDs in use along with their minor version number and the process id of the tapdisk2 process that is responsible for identifying this .vhd file

$ tap-ctl list | grep minor=20
# ~OR~
$ tap-ctl list -m 20
pid=12912 minor=20 state=0 args=vhd:/var/run/sr-mount/eee-eee-eee-eee-eee/fff-fff-fff-fff-fff.vhd

`tap-ctl list` will help you identify the .vhd file or the UUID of the VDI, which is responsible for the slice pushing heavy IO operations. In the above example, the vdi-uuid=fff-fff-fff-fff-fff.

• Check if this slice is a "swap" partition (make sure it is _not_ a root partition!):

$ xe vdi-list uuid=fff-fff-fff-fff-fff
uuid ( RO)                : fff-fff-fff-fff-fff
          name-label ( RW): slice01234_swap
    name-description ( RW): 
             sr-uuid ( RO): 76fa2a40-5729-1297-a4d8-38df5ea4128c
        virtual-size ( RO): 163208757248
            sharable ( RO): false
           read-only ( RO): false

• Check to make sure no other tasks are currently being performed on this host (except for "pending: reboot"-like statuses):

xe task-list

• Now force a reboot:

xe vm-reboot --force name-label=slice01234

• Check that the slice has actually rebooted:

xe vm-list name-label=slice01234 params=start-time

• Verify that the output from the last command is within the current timestamp (UTC):

date -u

• As a final step, you can bring up the slice's console to check for any errors, problems, or wait times (e.g., fscking):

xl console slice01234

That is it! You have successfully killed a swapper and brought the slice back up to a normal load.

• The following does the same as the above (without rebooting) as a single CLI script:

(echo "Slice IO_Read IO_Write Total"; \
(for uuid in $(xe vbd-list params=uuid | awk '$5{print $5}'); do \
  xe vbd-param-list uuid=$uuid | grep -P "^\s*(io_|vm-name-label|device)" | \
  awk '{if($1=="vm-name-label") name=$4; \
  if($1=="device") {\
    if($4=="xvdc" || $4=="xvdd") name=name"-swap"; \
    if($4=="xvda" || $4=="xvdb") name=name"-root";} \
  if($1=="io_read_kbs") ioread=$4; \
  if($1=="io_write_kbs") iowrite=$4}\
  END{if(substr(name,0,9)!="XenServer") print name" "ioread" "iowrite" "ioread+iowrite}';\
done) | sort -k4n) | column -t

XenClassic

This is how you do the above on a XenClassic setup.

• Log into the host the slice/instance is located on
• Run iostat to check for any swappers on this host:

iostat -xkd 1

• If swappers are found, list the device mapper for this slice:

ls -l /dev/mapper/ | grep ' 20 '

• Now "destroy" that slice (we are not permanently destroying this device, that is just the terminology):

xm destroy slice01234

• Finally, re-create this slice (this takes information from a configuration flatfile found under /etc/xen/slice01234):

xm create slice01234

• You can check that the server has fully booted back up with the following command:

xm console slice01234

That is it! You have successfully killed a swapper and brought the slice back up to a normal load.

We can accomplish the above with a single script (well, a series of CLI calls) like so:

(echo "device name tps MB_read/s MB_write/s MB_total/s";\
(ls -l /dev/mapper/; iostat -m 1 2) |\
awk 'BEGIN {section=0} {\
  if($3=="root") devices[$6]=$9;\
  if($1=="Device:") section++;\
  else if(section==2 && $0) {\
    dev=$1; tps=$2; read=$3; write=$4;\
    if(substr(dev,1,2)=="dm") {split(dev,parts,"-"); name=devices[parts[2]]} \
    if(!name) name="unknown";\
    print dev" "name" "tps" "read" "write" "read+write}}' |\
sort -k3n) | column -t

And, if the above command(s) return a slice id, then execute:

xm destroy slice01234 && xm create slice01234 && xm console slice01234

XenClassic slice configuration file

Below is an example of what a XenClassic slice configuration flatfile would look like (which is, by default, located at /etc/xen/slice01234):

name="slice01234"
memory=512
vcpus=4
kernel="/etc/xen/seeds/01/vmlinuz-2.6.35.4-generic"
ramdisk="/etc/xen/seeds/01/initrd.img-2.6.35.4-generic"
vif=['bridge=eth0, ip=xxx.xxx.xxx.xxx, mac=4f:d0:c4:10:ab:0c','bridge=eth1, ip=10.x.x.x, mac=4f:e0:29:9f:c3:fe']
disk=[ 'phy:slices/slice01234_root,sda1,w', 'phy:slices/slice01234_swap,sda2,w' ]
root="/dev/sda1 ro"
cpu_weight=512
extra="xencons=tty console=tty1 clocksource=acpi_pm "

where,

memory = how much RAM should be allocated for the slice

vcpus = number of virtual CPU's that should appear within the customer's environment (default: 4)

kernel = which kernel seed the slice should use

ramdisk = the initrd that corresponds to the kernel being used by the slice

vif = sets IP's and MAC addresses for the various network interfaces on the slice

disk = the storage volumes that the slice will use

root = the storage volume that should be used as the root filesystem

cpu_weight = helps to calculate the slice's share of the CPU (equal to RAM)

extra_ips = if the slice has additional IP's, they will be listed here

extra = additional Xen settings