Checking for swappers on XenServer

This article will outline the steps I take to check for swappers on my XenServer and XenClassic setups.

XenServer

• Log into the huddle the host in question is located in
• Log into the host the slice/instances is located on
• Run iostat to check for any swappers on this host:

iostat -xkd 1

• If swappers are found, cat the device of the slice to get its minor number:

cat /sys/block/tdk/dev  # returns something like: 1234:20

• Get the UUID for this slice

tap-ctl list | grep minor=20
# ~OR~
tap-ctl list -m 20

• Check if this slice is a "swap" partition (make sure it is _not_ a root partition!):

xe vdi-list uuid=xxxx-xxxx-xxxx-xxxx

• Check to make sure no other tasks are currently being performed on this host (except for "pending: reboot"-like statuses):

xe task-list

• Now force a reboot

xe vm-reboot --force name-label=slice01234

• Check that the slice has actually rebooted:

xe vm-list name-label=slice01234 params=start-time

• Verify that the output from the last command is within the current timestamp

date

That's it! You have successfully killed a swapper and brought the slice back up to a normal load.

• The following does the same as the above (without rebooting) as a single CLI script:

(echo "Slice IO_Read IO_Write Total"; \
(for uuid in $(xe vbd-list params=uuid | awk '$5{print $5}'); do \
  xe vbd-param-list uuid=$uuid | grep -P "^\s*(io_|vm-name-label|device)" | \
  awk '{if($1=="vm-name-label") name=$4; \
  if($1=="device") {\
    if($4=="xvdc" || $4=="xvdd") name=name"-swap"; \
    if($4=="xvda" || $4=="xvdb") name=name"-root";} \
  if($1=="io_read_kbs") ioread=$4; \
  if($1=="io_write_kbs") iowrite=$4}\
  END{if(substr(name,0,9)!="XenServer") print name" "ioread" "iowrite" "ioread+iowrite}';\
done) | sort -k4n) | column -t

XenClassic

This is how you do the above on a XenClassic setup.

• Log into the huddle the host in question is located in
• Log into the host the slice/instances is located on
• Run iostat to check for any swappers on this host:

iostat -xkd 1

• If swappers are found, list the device mapper for this slice:

ls -l /dev/mapper/ | grep ' 20 '

• Now "destroy" that slice (we are not permanently destroying this device, that's just the terminology):

xm destroy slice01234

• Finally, re-create this slice (this takes information from a configuration flatfile found under /etc/xen/slice01234):

xm create slice01234

That's it! You have successfully killed a swapper and brought the slice back up to a normal load.

We can accomplish the above with a single script (well, a series of CLI calls) like so:

(echo "device name tps MB_read/s MB_write/s MB_total/s";\
(ls -l /dev/mapper/; iostat -m 1 2) |\
awk 'BEGIN {section=0} {\
  if($3=="root") devices[$6]=$9;\
  if($1=="Device:") section++;\
  else if(section==2 && $0) {\
    dev=$1; tps=$2; read=$3; write=$4;\
    if(substr(dev,1,2)=="dm") {split(dev,parts,"-"); name=devices[parts[2]]} \
    if(!name) name="unknown";\
    print dev" "name" "tps" "read" "write" "read+write}}' |\
sort -k3n) | column -t

And, if the above command(s) return a slice id, then execute:

xm destroy slice01234 && xm create slice01234 && xm console slice01234

XenClassic slice configuration file

Below is an example of what a slice configuration flatfile (which is, by default, located at /etc/xen/slice01234) would look like:

name="slice01234"
memory=512
vcpus=4
kernel="/etc/xen/seeds/01/vmlinuz-2.6.35.4-generic"
ramdisk="/etc/xen/seeds/01/initrd.img-2.6.35.4-generic"
vif=['bridge=eth0, ip=xxx.xxx.xxx.xxx, mac=4f:d0:c4:10:ab:0c','bridge=eth1, ip=10.x.x.x, mac=4f:e0:29:9f:c3:fe']
disk=[ 'phy:slices/slice01234_root,sda1,w', 'phy:slices/slice01234_swap,sda2,w' ]
root="/dev/sda1 ro"
cpu_weight=512
extra="xencons=tty console=tty1 clocksource=acpi_pm "

where,

memory = how much RAM should be allocated for the slice

vcpus = number of virtual CPU's that should appear within the customer's environment (default: 4)

kernel = which kernel seed the slice should use

ramdisk = the initrd that corresponds to the kernel being used by the slice

vif = sets IP's and MAC addresses for the various network interfaces on the slice

disk = the storage volumes that the slice will use

root = the storage volume that should be used as the root filesystem

cpu_weight = helps to calculate the slice's share of the CPU (equal to RAM)

extra_ips = if the slice has additional IP's, they will be listed here

extra = additional Xen settings