Category: XenServer
Adding and Removing Local Storage From XenServer
Adding and Removing Local Storage From XenServer
To add local storage XenServer 6.x
get your device id’s with:
ll /dev/disk/by-id
The host uuid can be copied and pasted from the general tab of your host in XenCenter.
Create your storage:
xe sr-create content-type=user device-config:device=/dev/sdb host-uuid=<Place the host’s UUID here> name-label=”<Name your local storage here>” shared=false type=lvm
NOTE: Make sure that “shared=” is false. If you have shared storage on a hypervisor, you won’t be able to add it to a pool. When a hypervisor is added to a pool, its local storage is automatically shared in that pool.
NOTE: Replace sdb in the above command with the device that you’re adding.
To Remove local storage XenServer 6.x
Go to console in XenCenter or log in to your xenserver host via ssh
List your storage repositories.
xe sr-list
You will see something like this:
uuid ( RO) : <The uuid number you want is here>
name-label ( RW): Local storage
name-description ( RW):
host ( RO): host.example.com
type ( RO): lvm
content-type ( RO): user
uuid string is the Storage Repository uuid (SR-uuid) that you need to be able to do the next step.
Get the Physical Block Device UUID.
xe pbd-list sr-uuid=Your-UUID
uuid ( RO) This is the PBD-uuid
Unplug the local storage.
xe pbd-unplug uuid=Your-PBD-uuid
Delete the PBD:
xe pbd-destroy uuid=your-PBD-uuid
Forget ( remove ) the Local storage from showing up as detached.
xe sr-forget uuid=your-SR-uuid
Now check your XenCenter that it’s removed.
Automating patch installation on XenServer
Automating patch installation on XenServer
I have four instances of freshly installed XenServer 6.2 and there are about a dozen patches for each that need to be applied. Herein, I will attempt to somewhat automated the application of these patches.
What you will need to know:
The list of patches required
The URLs of the download pages for each patch to be applied
The UUID of the patch
The UUID of the Target Host. This can be found by doing, on the console,
a:
xe host-list
The procedure is as follows:
Use the wget command to download the patch. I’ll use Service Pack 1 as an example. We want that patch first, as it is cumulative, and will cut down the number of other patches to be installed.
To find the URL where SP1 resides. go to XenCenter Console, Tools, Check for Updates. This will give you a list of patches available for your server, with links to the download location. Click on the link for XS62ESP1. On the web page that opens, click on “Download”. This will open another page. This is the URL that you want. Copy it to clipboard.
You’ll need the urls and filenames for all the applicable patches when you customize your script.
http://downloadns.citrix.com.edgesuite.net/8707/XS62ESP1.zip
Now initiate the wget command at the console in XenCenter using this URL:
wget http://downloadns.citrix.com.edgesuite.net/8707/XS62ESP1.zip NOTE:Case Sensitive!
Then unzip it:
unzip XS62ESP1.zip
This zip file contains a file ending with the extension “xsupdate” as in “XS62ESP1.xsupdate. That’s our patch
Register the patch on the Target Server:
xe patch-upload file-name=XS62ESP1.xsupdate
This will display the uuid of the patch. We’ll need that for the next command. You can call it up again with:
xe patch-list
Install the patch:
xe patch-apply uuid=<The patch uuid from th xe patch-list command> host-uuid=<The host uuid from the xe host-list command>
Due to limited disk space, we’re going to clean out our working directory:
rm *
Now we’ll write our script: Notice that we’re doing the upload/registration and apply in one command.
Start of Script:
wget http://downloadns.citrix.com.edgesuite.net/8707/XS62ESP1.zip
unzip XS62ESP1.zip
xe patch-apply uuid=`xe patch-upload file-name=XS62ESP1.xsupdate 2>&1|tail -1|awk -F” ” ‘{print $NF}’` host-uuid=`grep -B1 -f /etc/hostname <(xe host-list)|head -n1|awk ‘{print $NF}’`
rm -f *
wget http://downloadns.citrix.com.edgesuite.net/8737/XS62ESP1002.zip
unzip XS62ESP1002.zip
xe patch-apply uuid=`xe patch-upload file-name=XS62ESP1002.xsupdate 2>&1|tail -1|awk -F” ” ‘{print $NF}’` host-uuid=`grep -B1 -f /etc/hostname <(xe host-list)|head -n1|awk ‘{print $NF}’`
rm -f *
wget http://downloadns.citrix.com.edgesuite.net/9058/XS62ESP1005.zip
unzip XS62ESP1005.zip
xe patch-apply uuid=`xe patch-upload file-name=XS62ESP1005.xsupdate 2>&1|tail -1|awk -F” ” ‘{print $NF}’` host-uuid=`grep -B1 -f /etc/hostname <(xe host-list)|head -n1|awk ‘{print $NF}’`
rm -f *
wget http://downloadns.citrix.com.edgesuite.net/9491/XS62ESP1008.zip
unzip XS62ESP1008.zip
xe patch-apply uuid=`xe patch-upload file-name=XS62ESP1008.xsupdate 2>&1|tail -1|awk -F” ” ‘{print $NF}’` host-uuid=`grep -B1 -f /etc/hostname <(xe host-list)|head -n1|awk ‘{print $NF}’`
rm -f *
wget http://downloadns.citrix.com.edgesuite.net/9617/XS62ESP1009.zip
unzip XS62ESP1009.zip
xe patch-apply uuid=`xe patch-upload file-name=XS62ESP1009.xsupdate 2>&1|tail -1|awk -F” ” ‘{print $NF}’` host-uuid=`grep -B1 -f /etc/hostname <(xe host-list)|head -n1|awk ‘{print $NF}’`
rm -f *
wget http://downloadns.citrix.com.edgesuite.net/9698/XS62ESP1011.zip
unzip XS62ESP1011.zip
xe patch-apply uuid=`xe patch-upload file-name=XS62ESP1011.xsupdate 2>&1|tail -1|awk -F” ” ‘{print $NF}’` host-uuid=`grep -B1 -f /etc/hostname <(xe host-list)|head -n1|awk ‘{print $NF}’`
rm -f *
wget http://downloadns.citrix.com.edgesuite.net/9703/XS62ESP1013.zip
unzip XS62ESP1013.zip
xe patch-apply uuid=`xe patch-upload file-name=XS62ESP1013.xsupdate 2>&1|tail -1|awk -F” ” ‘{print $NF}’` host-uuid=`grep -B1 -f /etc/hostname <(xe host-list)|head -n1|awk ‘{print $NF}’`
rm -f *
wget http://downloadns.citrix.com.edgesuite.net/9708/XS62ESP1014.zip
unzip XS62ESP1014.zip
xe patch-apply uuid=`xe patch-upload file-name=XS62ESP1014.xsupdate 2>&1|tail -1|awk -F” ” ‘{print $NF}’` host-uuid=`grep -B1 -f /etc/hostname <(xe host-list)|head -n1|awk ‘{print $NF}’`
rm -f *
wget http://downloadns.citrix.com.edgesuite.net/10128/XS62ESP1015.zip
unzip XS62ESP1015.zip
xe patch-apply uuid=`xe patch-upload file-name=XS62ESP1015.xsupdate 2>&1|tail -1|awk -F” ” ‘{print $NF}’` host-uuid=`grep -B1 -f /etc/hostname <(xe host-list)|head -n1|awk ‘{print $NF}’`
rm -f *
wget http://downloadns.citrix.com.edgesuite.net/10134/XS62ESP1012.zip
unzip XS62ESP1012.zip
xe patch-apply uuid=`xe patch-upload file-name=XS62ESP1012.xsupdate 2>&1|tail -1|awk -F” ” ‘{print $NF}’` host-uuid=`grep -B1 -f /etc/hostname <(xe host-list)|head -n1|awk ‘{print $NF}’`
rm -f *
wget http://downloadns.citrix.com.edgesuite.net/10174/XS62ESP1016.zip
unzip XS62ESP1016.zip
xe patch-apply uuid=`xe patch-upload file-name=XS62ESP1016.xsupdate 2>&1|tail -1|awk -F” ” ‘{print $NF}’` host-uuid=`grep -B1 -f /etc/hostname <(xe host-list)|head -n1|awk ‘{print $NF}’`
rm -f *
Now just copy the text you created with your patches listed in place of mine and paste in to to the console. You’ll be off and running! (Go get Coffee)
Matt Long
02/17/2015
Disk write speed testing different XenServer configurations – single disk vs mdadm vs hardware raid
Disk write speed testing different XenServer configurations – single disk vs mdadm vs hardware raid
In our virtual environment on of the VM Host servers has a hardware raid controller on it . so natuarally we used the hardware raid.
The server is a on a Dell 6100 which uses a low featured LSI SAS RAID controller.
One of the ‘low’ features was that it only allows two RAID volumes at a time. Also it does not do RAID 10
So I decided to create a RAID 1 with two SSD drives for the host, and we would also put the root operating systems for each of the Guest VMs there. It would be fast and redundant. Then we have upto 4 1TB disks for the larger data sets. We have multiple identically configured VM Hosts in our Pool.
For the data drives, with only 1 more RAID volume I could create without a RAID 10, I was limited to either a RAID V, a mirror with 2 spares, a JBOD. In order to get the most space out of the 4 1TB drives, I created the RAIDV. After configuring two identical VM hosts like this, putting a DRBD Primary / Primary connection between the two of them and then OCFS2 filesystem on top of it. I found I got as low as 3MB write speed. I wasnt originally thinking about what speeds I would get, I just kind of expected that the speeds would be somewhere around disk write speed and so I suppose I was expecting to get acceptable speeds beetween 30 and 80 MB/s. When I didn’t, I realized I was going to have to do some simple benchmarking on my 4 1TB drives to see what configuration will work best for me to get the best speed and size configuration out of them.
A couple of environment items
- I will mount the final drive on /data
- I mount temporary drives in /mnt when testing
- We use XenServer for our virtual environment, I will refer to the host as the VM Host or dom0 and to a guest VM as VM Guest or domU.
- The final speed that we are looking to get is on domU, since that is where our application will be, however I will be doing tests in both dom0 and domU environments.
- It is possible that the domU may be the only VM Guest, so we will also test raw disk access from domU for the data (and skip the abstraction level provided by the dom0)
So, as I test the different environments I need to be able to createw and destroy the local storage on the dom0 VM Host. Here are some commands that help me to do it.
I already went through xencenter and removed all connections and virtual disk on the storage I want to remove, I had to click on the device “Local Storage 2” under the host and click the storage tab and make sure each was deleted. {VM Host SR Delete Process}
xe sr-list host=server1 #find and keep the uuid of the sr in my case "c2457be3-be34-f2c1-deac-7d63dcc8a55a"
xe pbd-list sr-uuid=c2457be3-be34-f2c1-deac-7d63dcc8a55a # find and keep the uuid of the pbd connectig sr to dom0 "b8af1711-12d6-5c92-5ab2-c201d25612a9"
xe pbd-unplug uuid=b8af1711-12d6-5c92-5ab2-c201d25612a9 #unplug the device from the sr
xe pbd-destroy uuid=b8af1711-12d6-5c92-5ab2-c201d25612a9 #destroy the devices
xe sr-forget uuid=c2457be3-be34-f2c1-deac-7d63dcc8a55a #destroy the sr
Now that the sr is destroyed, I can work on the raw disks on the dom0 and do some benchmarking on the speeds of differnt soft configurations from their.
Once I have made a change, to the structure of the disks, I can recreate the sr with a new name on top of whatever solution I come up with by :
xe sr-create content-type=user device-config:device=/dev/XXX host-uuid=`grep -B1 -f /etc/hostname <(xe host-list)|head -n1|awk ‘{print $NF}’` name-label=”Local storage XXX on `cat /etc/hostname`” shared=false type=lvm
Replace the red XXX with what works for you
Most of the tests were me just running dd commands and writing the slowest time, and then what seemed to be about the average time in MB/s. It seemed like, the first time a write was done it was a bit slower but each subsequent time it was faster and I am not sure if that means when a disk is idle, it takes a bit longer to speed up and write? if that is the case then there are two scenarios, if the disk is often idle, the it will use the slower number, but if the disk is busy, it will use the higher average number, so I tracked them both. The idle disk issue was not scientific and many of my tests did not wait long enough for the disk to go idle inbetween tests.
The commands I ran for testing were dd commands
dd if=/dev/zero of=data/speetest.`date +%s` bs=1k count=1000 conv=fdatasync #for 1 mb dd if=/dev/zero of=data/speetest.`date +%s` bs=1k count=10000 conv=fdatasync #for 10 mb dd if=/dev/zero of=data/speetest.`date +%s` bs=1k count=100000 conv=fdatasync #for 100 mb dd if=/dev/zero of=data/speetest.`date +%s` bs=1k count=1000000 conv=fdatasync #for 1000 mb
I wont get into the details of every single command I ran as I was creating the different disk configurations and environments but I will document a couple of them
Soft RAID 10 on dom0
dom0>mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sda1 /dev/sdb2 --assume-clean dom0>mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sdc1 /dev/sdd2 --assume-clean dom0>mdadm --create /dev/md10 --level=0 --raid-devices=2 /dev/md0 /dev/md1 --assume-clean dom0>mkfs.ext3 /dev/md10 dom0>xe sr-create content-type=user device-config:device=/dev/md10 host-uuid=`grep -B1 -f /etc/hostname <(xe host-list)|head -n1|awk ‘{print $NF}’` name-label=”Local storage md10 on `cat /etc/hostname`” shared=false type=lvm
Dual Dom0 Mirror – Striped on DomU for an “Extended RAID 10”
dom0> {VM Host SR Delete Process} #to clean out 'Local storage md10' dom0>mdadm --manage /dev/md2 --stop dom0>mkfs.ext3 /dev/md0 && mkfs.ext3 /dev/md1 dom0>xe sr-create content-type=user device-config:device=/dev/md0 host-uuid=`grep -B1 -f /etc/hostname <(xe host-list)|head -n1|awk ‘{print $NF}’` name-label=”Local storage md0 on `cat /etc/hostname`” shared=false type=lvm dom0>xe sr-create content-type=user device-config:device=/dev/md1 host-uuid=`grep -B1 -f /etc/hostname <(xe host-list)|head -n1|awk ‘{print $NF}’` name-label=”Local storage md1 on `cat /etc/hostname`” shared=false type=lvm domU> #at this point use Xen Center to add and attach disks from each of the local md0 and md1 disks to the domU (they were attached on my systems as xvdb and xvdc domU> mdadm --create /dev/md10 --level=0 --raid-devices=2 /dev/xvdb /dev/xvdc domU> mkfs.ext3 /dev/md10 && mount /data /dev/md10
Four disks SR from dom0, soft raid 10 on domU
domU>umount /data
domU> mdadm --manage /dev/md10 --stop
domU> {delete md2 and md1 disks from the storage tab under your VM Host in Xen Center}
dom0> {VM Host SR Delete Process} #to clean out 'Local storage md10'
dom0>mdadm --manage /dev/md2 --stop
dom0>mdadm --manage /dev/md1 --stop
dom0>mdadm --manage /dev/md0 --stop
dom0>fdisk /dev/sda #delete partition and write (d w)
dom0>fdisk /dev/sdb #delete partition and write (d w)
dom0>fdisk /dev/sdc #delete partition and write (d w)
dom0>fdisk /dev/sdd #delete partition and write (d w)
dom0>xe sr-create content-type=user device-config:device=/dev/sda host-uuid=`grep -B1 -f /etc/hostname <(xe host-list)|head -n1|awk '{print $NF}'` name-label="Local storage sda on `cat /etc/hostname`" shared=false type=lvm
dom0>xe sr-create content-type=user device-config:device=/dev/sdb host-uuid=`grep -B1 -f /etc/hostname <(xe host-list)|head -n1|awk '{print $NF}'` name-label="Local storage sdb on `cat /etc/hostname`" shared=false type=lvm
dom0>xe sr-create content-type=user device-config:device=/dev/sdc host-uuid=`grep -B1 -f /etc/hostname <(xe host-list)|head -n1|awk '{print $NF}'` name-label="Local storage sdc on `cat /etc/hostname`" shared=false type=lvm
dom0>xe sr-create content-type=user device-config:device=/dev/sdd host-uuid=`grep -B1 -f /etc/hostname <(xe host-list)|head -n1|awk '{print $NF}'` name-label="Local storage sdd on `cat /etc/hostname`" shared=false type=lvm
domU>mdadm --create /dev/md10 -l10 --raid-devices=4 /dev/xvdb /dev/xvdc /dev/xvde /dev/xvdf
domU>mdadm --detail --scan >> /etc/mdadm/mdadm.conf
domU>echo 100000 > /proc/sys/dev/raid/speed_limit_min #I made the resync go fast, which reduced it from 26 hours to about 3 hours
domU>mdadm --grow /dev/md0 --size=max
Creating a Bootable USB Install Thumb drive for XenServer
Creating a Bootable USB Install Thumb drive for XenServer
We have a couple sites with XenServer VM machines, so part of our redundancy / failure plan is to be able to quickly isntall / reinstall a XenServer hypervisor.
THere are plenty of more involved methods with setting up PXE servers, etc. But the quickest / low tech method is to have a USB thumbdrive on hand.
So we can use one of the plethora of tools to create a USB thumbdrive, (unetbootin, USB to ISO, etc) but they all seem to have problems with the ISO, (OS not found, error with install, etc)
So I found one that works well
http://rufus.akeo.ie/
He keeps his software upto date it appears. Download it and run it, select your USB drive then check the box to ‘create a bootable disk using ISO Image, select the image to use from your hard drive. I downloaded the iso image from
http://xenserver.org/overview-xenserver-open-source-virtualization/download.html
– XenServer Installation ISO
Then just boot from the USB drive and the install should start.
Xen Center – import from exported XVA file for restoring – does not create a new VM
Xen Center – import from exported XVA file for restoring – does not create a new VM
Building a backup procedure for Xen Center
This started when defining the procedure for how to backup VMs ‘off site’ in a way that would later allow us to restore them, should some sort of unrecoverable error occur.
The concept is, we want to be able to take a VM, which is currently running and get a backup of some kind which can then be restored at a later point.
First I will explain what I have found to be the way that Xen Center currenlty allows, I dont know what the ‘Best Practice’ is suggested for this procedure, I couldnt find it from searching and so I explored the options available within Xen Center and at the command line
Exporting (stopping a VM first)
This method requires you to stop a VM first, which means that it does not work well for VMs which are already in production, and the method is not a viable “backup” solution for us, but I explain it here to make the distinction between the different types of exporting. This method is something that would work well for backing up defined templates to an offsite location, but would not work well for saving a full running VM in a way that could be restored from an offsite location.
Xen Center allows you to export a VM, which you later import: First, shut down your VM, right click on it and go to Export and follow the steps in the wizard to put the export on your local drive. you can conveniently do an export of multiple stopped VMs. the progress is displayed in the Logs tab for the pool. I told the export process to ‘verify’ the exported files, which added A LONG time to the process, be prepared for this
Once the export is complete, you can move these files where ever you want, to restore, you simply right click on your pool and go to Import select the file from disk, follow the wizard and the VMS will be started (I am not sure what happens with any kind of MAC address collisions here if the same VM you exported is currently running)
Exporting a live VM
It seems reasonable that a currently running VM could not be exported directly to a file, because the VM is running and the changes that occur in a running VM would be inconsitant during the process of the export. here is how we work around this.
In short, here are the steps
- create a snapshot of a vm
- export the snapshot to a file offsite using Xen Center (we are backed up)
- start the restore by creating a vm martyr by installing a new vm or template (hopefully a small )
- destroy the martyrs existing vdi
- import the snapshot from a file (could be on the same server or pool, or a completely new build)
- attach the imported vdi as a bootable disk
- rename from martry to your correct name and start the new VM
WIth more detail:
First take snapshot of the running VM, then go to the Snapshots tab for th VM and export that Snapshot. Follow the wizard and save it to a file.
When it comes time to re import, we have a little preparation we should do and keep track of these numbers
#Find the UUID for the SR that you will be importing the file back to
xe sr-list host=HOSTNAME
>079eb2c5-091d-5c2f-7e84-17705a8158cf
#get a list of all of the CURRENT uuids for the virtual disks on the sr
xe vdi-list sr-uuid=079eb2c5-091d-5c2f-7e84-17705a8158cf|grep uuid
>uuid ( RO) : bb8015b0-0672-45af-aed5-c5308f60b914
>uuid ( RO) : f0b67634-25bc-486d-b38e-0e8294de7df6
>uuid ( RO) : cdc13e40-9ffe-497c-91ff-d426a52aaf2a
Now we import the file. Right click on the host you would like to restore it to and click ‘Import’ THe import process asks you a couple of pieces of information about the restore, host name, network, etc. go through the steps an click finish. The vm will be imported again, the progress will be shows in the Logs tab of the host and pool, when complete, we now have a virtual disk unattached to a VM, which we need to attach to a VM,
Here things are a bit more complex. First we create a VM ‘martyr’, this is what I call a VM, that we create through some other method, soley for the purpose of attaching our reimported snapshot to it. we will take the guts out of whatever VM we create and put the guts from our import into it. on the technical side, we take a VM, disconnect the existing bootable vdi and reconnect the vdi we just imported. I create the VM using a template or install I dont cover that here, but I name it martyr_for_import
#get a list of the latest uuids for the virtual disks on the sr
xe vdi-list sr-uuid=079eb2c5-091d-5c2f-7e84-17705a8158cf|grep uuid
>uuid ( RO) : bb8015b0-0672-45af-aed5-c5308f60b914
>uuid ( RO) : f0b67634-25bc-486d-b38e-0e8294de7df6
>uuid ( RO) : cdc13e40-9ffe-497c-91ff-d426a52aaf2a
>uuid ( RO) : 04a7f80e-e108-4468-9bd3-fada613e9a42
#each time I have done this, the imported uuid is listed last, but I run the list, before and after, just to make sure, in this case my vdi is: 04a7f80e-e108-4468-9bd3-fada613e9a42
#find the current vbds attached to this vm
xe vbd-list vm-label-name-label=martyr_for_import
>uuid ( RO) : b0f4cb5e-5285-bbec-13a3-f581c6e6d287 vm-uuid ( RO): 708b633a-683d-859f-1f1f-bf8495d17fe8 vm-name-label ( RO): martyr_for_import vdi-uuid ( RO): a36d6025-039b-4f6e-9d19-f7eb7d1d4c46 empty ( RO): false device ( RO): xvdd
uuid ( RO) : eb12fdac-c36c-78fa-8eb6-67fa3a5a1d85 vm-uuid ( RO): 708b633a-683d-859f-1f1f-bf8495d17fe8 vm-name-label ( RO): martyr_for_import vdi-uuid ( RO): cdc13e40-9ffe-497c-91ff-d426a52aaf2a empty ( RO): false device ( RO): xvda #shut down the vm xe vm-shutdown uuid=708b633a-683d-859f-1f1f-bf8495d17fe8 #destroy the vdi virtual disk that is attached to our marty as the current xvda vbd xe vdi-destroy uuid=cdc13e40-9ffe-497c-91ff-d426a52aaf2a #you can verify that it has been destroyed and detached by running xe vbd-list vm-label-name-label=martyr_for_import again #now attach our snapshot vdi as a new vbd bottable device as xvda again. (note the bootable=true and type=Disk) xe vbd-create vm-uuid=708b633a-683d-859f-1f1f-bf8495d17fe8 vdi-uuid=04a7f80e-e108-4468-9bd3-fada613e9a42 bootable=true device=xvda type=Disk #okay we are attached (you can verify by running xe vbd-list vm-label-name-label=martyr_for_import again #go ahead and start the vm through your Xen scenter or run this command xe vm-start uuid=708b633a-683d-859f-1f1f-bf8495d17fe8
Deleting Orphaned Disks in Citrix XenServer
Deleting Orphaned Disks in Citrix XenServer
I found that while building my virtual environment with templates ready to deploy I created quite a few templates and snapshots.
I did a pretty good job of deleting the extras when I didn’t need them any more, but in some cases when deleting a VM I no longer needed, I forgot to check the box to delete the snapshots that went WITH that VM.
I could see under the dom0 host -> Storage tab that space was still allocated to the snapshots, (Usage was higher than the combined visible suage of servers and templates, and Virtual allocation was way higher than it should be)
But without a place that listed the snapshots that were taking up space. When looking into the way to delete these orphaned snapshots (and the disk snapshots that went with them) I found some cumbersome command line methods.
Like this old method that someone used - http://blog.appsense.com/2009/11/deleting-orphaned-disks-in-citrix-xenserver-5-5/
After a big more digging, i found that by just clicking on the Local Storage under the domU then clicking on the ‘Storage’ tab under there, I would see a list of all of the storage elements that are allocated. I would see some that were for snapshots without a name. Turns out those were the ones that were orphaned, If they were allocated to a live server the delete button would not be highlighted so I just deleted those old ones.
Resizing a VDI on XenServer using XenCenter and Commandline
Resizing a VDI on XenServer using XenCenter and Commandline
Occassionally I have a need to change the size of a disk, perhaps to allocate more data to the os.
To do this, on the host I unmount the disk
umount /data
Click on the domU server in XenCenter and click on the Storage tab, select the storage item I want to resize and click ‘Detach’
at the command line on one of the dom0 hosts
xe sr-list host=dom0hostname
write down the uuid of the SR which the Virtual Disk was in. (we will use XXXXX-XXXXX-XXXX)
xe vdi-list sr-uuid=XXXXX-XXXXX-XXXX
write down the uuid of the disk that you wanted to resize(we will use YYYY-YYYY-YYYYY)
Also, note that the the virtual-size parameter that shows. VDIs can not be shrunk so you will need a disk size LARGER than the size displayed here.
xe vdi-resize sr-uuid=YYYY-YYYY-YYYYY disk-size=9887654
XenCenter – live migrating a vm in a pool to another host
XenCenter – live migrating a vm in a pool to another host
When migrating a vm server from one host to another host in the pool I found it to be very easy at first.
In fact, it was one of the first things test I did after setting up my first vm on a host in a pool. 4 steps
- Simply right click on the vm in XenCenter ->
- Migrate to Server ->
- Select from your available servers.
- Follow the wizzard
In building some servers, I wanted to get some base templates which are ‘aware’ of the network I am putting together. This would involve adding some packages and configuration, taking a snapshot and then turning that snapshot into a template that I could easily restart next time I wanted a similar server. Then when I went to migrate one of the servers into its final resting place. I found an interesting error.
- Right click on the vm in XenCenter ->
- Migrate to Server ->
- All servers listed – Cannot see required storage
I found this odd since I was sure that the pool could see all of the required storage (In fact I was able to start a new VM on the storage available, so I new the storage was there)
I soon found out though that the issue is that the live migrate feature, just doesnt work when there is more than one snapshot. I will have to look into my snapshot management on how I want to do this now, but basically I found that by removing old snapshots does to where the VM only had one snapshot (I left one that was a couple of days old) I was able to follow the original 4 steps
Note: the way I found out about the limitation of the number of snapshots was by
- Eight click on the vm in XenCenter ->
- Migrate to Server ->
- The available servers are all grayed out, so Select “Migrate VM Wizard”
- In the wizard that comes up select the current pool for “Destination”
- This populates a list of VMs with Home Server in the destination pool want to migrate the VM (My understanding, is that this will move the VM to that server AND make that new server the “Home Server” for that VM)
- When you attempt to select from the drop down list under Home Server, you see a message “You attempted to migrate a VM with more than one snaphot”
Using that information I removed all but one snapshot and was able to migrate. I am sure there is some logical reason behind snapshot / migration limitation but for now I will work around it and come up with some other way to handle my snapshots than just leaving them under the snapshot tab of the server.
Notes on Recovering from a XenServer Pool failure
Notes on Recovering from a XenServer Pool failure
For my pool I have 8 XenServers (plot1, plot2, plot3, plot4, plot5, plot6, plot7 and plot8)
At the start of my tests, plot1 is the pool master.
If the pool master goes down, another server must take over as master.
To simulate this, I just ran ‘shut down’ on the master host
A large issue here is that all of the slaves in the pool, just disabled their management interfaces so they can not be connected to using XenCenter (something I did not expect), so I connected to plot2 via SSH
THen I connected to another server in the pool, and verified its state
xe host-is-in-emergency-mode
The server said FALSE!?! the server didn’t even know that the pool was in trouble? so I ran pool-list
xe pool-list
The command took a long time so I figured I would stop it and put a time command in front of it to find out how long it really tool
time xe pool-list
Turns out, when I shut down the pool master, I am shutting down the pool! , I am not simulating an error at all. Somehow the pool master notified the slaves that it was gracefully shutting down, telling the slaves dont worry, I will be all right., the commands above never returned. so I just told plot2 to take over as master to see how we could recover from this situation.
xe pool-emergency-transition-to-master
At this point on plot 2, the pool was restored but we could still not connect to the management interfaces of any of the other plots in the pool. But XenCenter WAS able to connect to plot2, and it synchronized the entire pool, showing all of the other hosts (including plot1 which was the master previously) as down.
The other hosts in the pool are still running all of their services (SSH, apache or whathever) they just can not communicate about the pool so I have to ‘recover’ them back into the pool.
On the new master I run
xe pool-recover-slaves
This brings the slaves back into the pool so they are visible within XenCenter again. plot1, the original master is still turned off, but visible as turned off in XenCenter, so I right click on it. in XenCenter and Power On. It begins booting and I hold my breath to see if there are any master master conflicts, since the shut down host thought it was the all powerful one when it shut down.
Once it comes up (3 minutes later) I find that plot1 gracefully fell into place as a slave. So the moral of this story,
!Dont shut down the pool master, if you do you will lose XenCenter access to all of the hosts in the pools so you MUST either 1) bring it backup immediately or 2) SSH to the console of another host run #xe pool-emergency-transition-to-master and then #xe pool-recover-slaves – this will restore your pool minus the host that was originally the master. reconnect with XenCenter to the new poolmaster, using the XenCenter then power on the host that was the pool master
!Best Practice: before stopping a host that is currently the poolmaster, connect to another host in the pool and run #xe pool-emergency-transition-to-master and then #xe pool-recover-slaves prior too shutting down the host.
Well, so now that we know shutting down the master does not simulate a failure, we will have to use another ‘onsite’ method.
!Simulation2:
On plot2 (current pool master) I disconnected the ethernet cables.
The XenCenter console can no longer connect to the pool again, so I have to use SSH, This time I will connect to plot3 and find out what it thinks of the pool issue.
xe host-is-in-emergency-mode
This command returns false, somehow the host thinks every thing is okay, I run xe pool-list and xe host-list, both of which never return, come one host shouldn’t you recognize a failure here?
I ping the same IP as the pool master and the ping fails, but the xe host-is-in-emergency-mode still returns false, for some reason, this host just does not think it has a problem
so, I guess I just can’t trust xe host-is-in-emergency-mode,
Even after 2 hours, the xe host-is-in-emergency-mode still returns false.
So for monitoring, I will have to come up with some other method. but the rules for how to recover are the same
xe pool-emergency-transition-to-master
xe pool-recover-slaves
This brought the pool up again on plot3 with plot3 as the new master.
Now the trick is to bring plot2 back on, in this case, plot 2 never ended up going offline, so it is still running without the ethernet cable plugged in, so when I plug it back in, I may end up with some master – master conflicts ….. here goes!.
After reconnecting the ethernet cable to plot 2 (the old master):
– plot3 did not recognize automatically that the host is backup, infact in XenCenter, it still shows red as though it is shut down, I right clicked on it and told it to power on, but it didn’t do anything but wait.
– plot2 did not make any changes, it appears they both, happily think they are the masters.
To test how the pool reacted, I attempted to disable one of the slaves from plot2 xe host-disable uuid=xxxxxxxx (my thought is that plot 2 is incorrectly considered down and not connected so the disable should not be let through.)
It turns out that plot2 could not disable the host, because the host ‘could not be contacted’ , this is good because it makes sure that none of the slaves are confused, in fact, plot3 is not confused either, it is only plot2, the master that went missing that is confused (I have seen in xen docs that they call this a barrier of some sort)
I tried to connect to plot2 with XenCenter, but XenCenter smartly told me that I can not connect because it appears that the server was created as a backup from my pool and that the dangerous operation is not allowed. (I will try to trick XenCenter into connecting by removing references to my pool from it and then trying again)
AH! it let me! that means that XenCenter is smart enough to recognize when you are attempting to make two connection separately to the split brain masters of a pool, but prevents it.
To dig further into this issue. I decided to further ‘break’ the pool by splitting the two masters further with different definitions of the pool. On the plot2 master I used XenCenter to destroy the disconnected host plot7. XenCenter let me do this. Now when I go to reconnect, I will be attempting to pull the orphaned master with a different definition of the pool, back into the pool.
Now the trick is to determine what the best way to bring the plot2 old master back into the current pool as a slave. We need to tell the new master to recover slaves.
xe pool-recover-slaves
That pulls plot2 back in as a slave, and GREAT it did not use any of the pool definition from plot2. plot3 property asserted its role as the true pool master
I can imagine a bad scenario happening if I told the “OLD” master to recover slave, I imagine that either the split would have gotten much worse, Or (if the barrier was really working, the the pool would have told the old master that it was not possible).
Other methods that I did not use which may have worked but were nto tried (they dont feel right):
– from the orphaned master: xe pool-join force=1 ….. server username password (i doubt this would work since it is already the member of a pool)
– from the orphaned master xe pool-reset-master master-server= ip of new master (this one I am not sure of, would be worth a shot if for some reason pool is not working)
THe thing that you NEVER want to do while a master or any other server is orphaned or down, is remove the server from the pool. What can happen in this sitation is that the server that is down, still thinks it is in the pool when it comes back up but the pool does not know about it. We get into a race condition that I have only ever found one way out of. The orphaned server thinks it is in a pool, but can not get out of the pool without connecting to the master. The master will not recognize the orphaned server so the server cant do anything. (the way out of this was to promote the orphaned server to master, the remove all of the hosts in the pool, then delete all of the stored resources and pbd and then join the pool anew. This sucked because everything on the server was destroyed so I could have just r reinstalled xenserver.
I have heard but not attempted to reinstall xenserver without selecting the disks
http://support.citrix.com/article/CTX120962
Promoting a XenServer host to pool master
Promoting a XenServer host to pool master
There are a couple of problems that can be found when attempting to promote a XenServer host to pool master.
- If you are on host1 attempting to designate host2 through the xsconsole, you are prompted to select from all of the other hostnames in the pool, however xsconsole appears to use the hostname only and if your server is not configured to be able to refer to the other host by the hostname xsconsole will show an error, can not connect to host.
This could be resolved by making /etc/hosts entries for each host, but that is overkill when you need to make a quick poolmaster change - The next idea is to by convention, only designate a host to pool master FROM that host, and make sure that each host can refer to itself using it hostname.
Not too much problem here, that seems pretty reasonable to have the current hostname in the /etc/hosts file, however the default XenServer install does not do this.
So the best way i find is to have a convention to only promot a pool master FROM that host, using the command line method, using the UUID.
# xe pool-designate-new-master host-uuid=`grep -B1 -f /etc/hostname <(xe host-list)|head -n1|awk ‘{print $NF}’`
A couple of problems can still prevent new pool master from being designated.
- All hosts in the pool are not available. For example, One of the hosts in the pool was down, and I received an error because host5 was not available. I solved this using XenCenter to destroy the other hosts, that seemed like it was not a good idea since I wanted them to come up at sometime in the future and rejoin the pool, but I guess that has to be done manually.
- You attempted an operation which involves a host which could not be contacted.
host: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx (HOST5)
- You attempted an operation which involves a host which could not be contacted.
- Another problem that I have not encountered yet but is still possible, is an issue where you are unable to designate a new master while you have servers down which the pool thinks may still be running. (sounds like some sort of split brain ‘protection’ or something)
- http://discussions.citrix.com/topic/250603-how-to-forcefully-remove-a-dead-host-from-a-pool/