|
|
| Line 1: |
Line 1: |
| | '''<code>vhd-util</code>''' is a utility for working with Virtual HDD (VHDs) in a [[XenServer]] environment. | | '''<code>vhd-util</code>''' is a utility for working with Virtual HDD (VHDs) in a [[XenServer]] environment. |
| − |
| |
| − | ==Fixing a corrupt VHD footer==
| |
| − | ''Note: The following section is based on [https://one.rackspace.com/pages/viewpage.action?title=Fixing+Corrupt+VHD+Footer&spaceKey=SH Mark Silence's] original article.''
| |
| − |
| |
| − | ;Issue: Sometimes, the footer on the customer's VHD will become corrupted.
| |
| − | ;Symptoms: There are several possible symptoms that are manifested by this issue.
| |
| − |
| |
| − | If the customer takes an image of this server, and then attempts to build a new server from it, they may encounter an error like this:
| |
| − |
| |
| − | <pre>
| |
| − | 2012-10-31 18:06:12 TRACE nova.utils [instance: 4dc16179-9e98-416e-ada0-4a0f1bdbb690]
| |
| − | Failure: ['XENAPI_PLUGIN_FAILURE', 'download_vhd', 'Exception',
| |
| − | "Unexpected output '/var/run/sr-mount/9bfaf37b-720b-7cf1-13a9-8c4ef3237322/tmpFco2q9/c8f6bdd0-8e73-407c-9449-546b7dc958aa.vhd is valid\n
| |
| − | /var/run/sr-mount/9bfaf37b-720b-7cf1-13a9-8c4ef3237322/tmpFco2q9/4f7306a6-818a-4a39-aa25-2768f53ed594.vhd is valid\n
| |
| − | primary footer invalid: invalid cookie\n
| |
| − | /var/run/sr-mount/9bfaf37b-720b-7cf1-13a9-8c4ef3237322/tmpFco2q9/24e9f9a6-c4a6-4fcf-a40b-b57b7c90d2f2.vhd appears invalid; dumping metadata\n
| |
| − | VHD Footer Summary:\n-------------------\nCookie : conectix\n
| |
| − | Features : (0x00000002) <RESV>\nFile format version : Major: 1, Minor: 0\nData offset : 512\n
| |
| − | Timestamp : Tue May 1 20:54:45 2012\nCreator Application : 'tap'\n
| |
| − | Creator version : Major: 1, Minor: 3\nCreator OS : Unknown!\nOriginal disk size : 9728 MB (10200547328 Bytes)\n
| |
| − | Current disk size : 163840 MB (171798691840 Bytes)\nGeometry : Cyl: 65535, Hds: 16, Sctrs: 255\n
| |
| − | : = 130558 MB (136899993600 Bytes)\nDisk type : Dynamic hard disk\nChecksum : 0xffffef6f|0xffffef6f (Good!)\nUUID : ae71e079-63e8-455d-8682-7c44d04b3168\nSaved state : No\nHidden : 1\n\nVHD Header Summary:
| |
| − | \n-------------------\nCookie : cxsparse\nData offset (unusd) : 18446744073709\nTable offset : 1536\n
| |
| − | Header version : 0x00010000\nMax BAT size : 81920\nBlock size : 2097152 (2 MB)\nParent name : \n
| |
| − | Parent UUID : 00000000-0000-0000-0000-000000000000\nParent timestamp : Sat Jan 1 00:00:00 2000\n
| |
| − | Checksum : 0xfffff436|0xfffff436 (Good!)\n\nVHD Batmap Summary:\n-------------------\n
| |
| − | Batmap offset : 329728\nBatmap size (secs) : 20\nBatmap version : 0x00010002\n
| |
| − | Checksum : 0xfff6398e|0xfff6398e (Good!)\n\nerror checking parents: -22\n' from vhd-util"]
| |
| − | 2012-10-31 18:06:12 TRACE nova.utils [instance: 4dc16179-9e98-416e-ada0-4a0f1bdbb690]
| |
| − | </pre>
| |
| − | Notice the message "vhd is valid \ primary footer invalid".
| |
| − |
| |
| − | The VHD contains a header and a footer, which hold duplicate information. This is used to help verify the integrity of the VHD file. Sometimes, for currently unknown reasons, the footer becomes corrupted or goes missing completely. As long as the customer's data does not extend to the very end of their VHD, we can duplicate the information from the header to create a new footer.
| |
| − |
| |
| − | ;Solution
| |
| − |
| |
| − | If this problem is observer with a saved image, there is no fixing the image. You will have to located the original server the image was created from, and fix that server, and then take a new image.
| |
| − |
| |
| − | You can use this to look at the footers for all VHDs to see which ones have problems. You can also run it specifically on one if you know it has issues.
| |
| − |
| |
| − | for item in /var/run/sr-mount/*/*; do echo $item && tail -c 512 $item | hexdump -C; done;
| |
| − |
| |
| − | Note: Sometimes, if the customer is not using all of their VHD yet, and the footer is missing, you will see nothing when running a <code>tail -c 512</code>. I have had to bump it up to <code>tail -c 5120000</code> before seeing any data before, it just depends on how much empty space is at the end of their VHD.
| |
| − |
| |
| − | You can also use this utility to confirm the integrity of the VHD, also great for checking that a repair worked afterwards.
| |
| − |
| |
| − | vhd-util read -p -n /var/run/sr-mount/<SR-UUID>/<VHD-UUID>.vhd
| |
| − |
| |
| − | As long as their is not data at the end, you can usually use this to place the footer back on the file, but sometimes you will miss the 512-byte boundary and it will not work.
| |
| − |
| |
| − | head -c 512 /var/run/sr-mount/$SR-UUID/XXXX..vhd >> /var/run/sr-mount/$SR-UUID/XXXXXXX.vhd && vhd-util repair -n /var/run/sr-mount/$SR-UUID/$UUID_of_VHD
| |
| − |
| |
| − | ==Corrupt Truncated VHD Repair (Conectix Cookie)==
| |
| − | ''Note: The following section is based on [https://one.rackspace.com/pages/viewpage.action?pageId=29786734 Chase Naquin's] original article.''
| |
| − |
| |
| − | '''WARNING: DO NOT RUN ANY OF THE FOLLOWING COMMANDS UNLESS YOU KNOW WHAT YOU ARE DOING!'''
| |
| − |
| |
| − | * Find Corrupt VHDs:
| |
| − |
| |
| − | for item in /var/run/sr-mount/*/*; do echo $item && tail -c 512 $item | hexdump -C; done;
| |
| − |
| |
| − | * Repair a single corrupt VHD:
| |
| − |
| |
| − | CVHD=c5c0cdc3-c2c4-4c43-a5a2-845f36b866a4;
| |
| − | fn=$(ls /var/run/sr-mount/*/${CVHD}*.vhd);
| |
| − | DATA=$(wc -c $fn | awk '{print (512-($1 % 512))%512}');
| |
| − | if [ "$DATA" -ne "0" ]; then for i in $(eval echo {1..$DATA}); do echo -n '0' >> $fn; done; fi;
| |
| − | head -c 512 $fn >> $fn;
| |
| − | vhd-util repair -n $fn;
| |
| − |
| |
| − | * Repair a corrupt VHD in SMlog:
| |
| − |
| |
| − | error_log_fn='/var/log/SMlog';
| |
| − | CVHD="$(awk '/EXCEPTION/ && /corrupt/ {print substr($8,2,8)}' $error_log_fn | tail -1)";
| |
| − | fn=$(ls /var/run/sr-mount/*/${CVHD}*.vhd);
| |
| − | DATA=$(wc -c $fn | awk '{print (512-($1 % 512))%512}');
| |
| − | if [ "$DATA" -ne "0" ]; then for i in $(eval echo {1..$DATA}); do echo -n '0' >> $fn; done; fi;
| |
| − | head -c 512 $fn >> $fn;
| |
| − | vhd-util repair -n $fn;
| |
| − |
| |
| − | * Teran's fix all the things at once. Make sure the box has dc before running.
| |
| − |
| |
| − | for vhd in /var/run/sr-mount/*/*.vhd; do
| |
| − | vhd-util check -n $vhd > /dev/null || (DATA=$((stat -c%s $vhd; echo -n "512 %p") | dc);
| |
| − | if [ "$DATA" -ne "0" ]; then
| |
| − | for i in $(eval echo {1..$DATA}); do
| |
| − | echo -n '0' >> $vhd;
| |
| − | done;
| |
| − | fi;
| |
| − | head -c 512 $vhd >> $vhd;vhd-util repair -n $vhd);
| |
| − | done
| |
| − |
| |
| − | * Teran's check for byte misaligned if curious:
| |
| − |
| |
| − | for vhd in /var/run/sr-mount/*/*.vhd; do (stat -c%s $vhd; echo 512%p) | dc; done | uniq
| |
| − |
| |
| − | * Example error/exception:
| |
| − | <pre>
| |
| − | [26919] 2012-05-15 00:18:35.127755 FAILED: (errno 22) stdout: 'primary footer invalid: invalid cookie
| |
| − | /var/run/sr-mount/7a142a29-9556-ee47-1e04-1a8517813369/1d994938-058e-102f-80dc-001ec95cb0b6.vhd appears invalid;
| |
| − | dumping metadata
| |
| − | VHD Footer Summary:
| |
| − | -------------------
| |
| − | Cookie : conectix
| |
| − | Features : (0x00000002) <RESV>
| |
| − | File format version : Major: 1, Minor: 0
| |
| − | Data offset : 512
| |
| − | Timestamp : Wed Oct 27 09:45:01 2010
| |
| − | Creator Application : 'tap'
| |
| − | Creator version : Major: 1, Minor: 3
| |
| − | Creator OS : Unknown!
| |
| − | Original disk size : 20480 MB (21474836480 Bytes)
| |
| − | Current disk size : 40960 MB (42949672960 Bytes)
| |
| − | Geometry : Cyl: 20560, Hds: 16, Sctrs: 255
| |
| − | : = 40959 MB (42949017600 Bytes)
| |
| − | Disk type : Dynamic hard disk
| |
| − | Checksum : 0xfffff246|0xfffff246 (Good!)
| |
| − | UUID : 4223b679-0941-49af-b46a-2077b947176e
| |
| − | Saved state : No
| |
| − | Hidden : 1
| |
| − | VHD Header Summary:
| |
| − | -------------------
| |
| − | Cookie : cxsparse
| |
| − | Data offset (unusd) : 18446744073709
| |
| − | Table offset : 1536
| |
| − | Header version : 0x00010000
| |
| − | Max BAT size : 20480
| |
| − | Block size : 2097152 (2 MB)
| |
| − | Parent name :
| |
| − | Parent UUID : 00000000-0000-0000-0000-000000000000
| |
| − | Parent timestamp : Sat Jan 1 00:00:00 2000
| |
| − | Checksum : 0xfffff427|0xfffff427 (Good!)
| |
| − | VHD Batmap Summary:
| |
| − | -------------------
| |
| − | Batmap offset : 83968
| |
| − | Batmap size (secs) : 5
| |
| − | Batmap version : 0x00010002
| |
| − | Checksum : 0xfffccebd|0xfffccebd (Good!)
| |
| − | ', stderr: ''
| |
| − | <26919> 2012-05-15 00:18:35.128104 In cleanup
| |
| − | <26919> 2012-05-15 00:18:35.129656 ***
| |
| − | <26919> 2012-05-15 00:18:35.129711 ***********************
| |
| − | <26919> 2012-05-15 00:18:35.129754 * E X C E P T I O N *
| |
| − | <26919> 2012-05-15 00:18:35.129797 ***********************
| |
| − | <26919> 2012-05-15 00:18:35.129850 coalesce: EXCEPTION util.SMException, VHD *1d994938(40.00G/14.27G) corrupted
| |
| − | <26919> 2012-05-15 00:18:35.129894 File "/opt/xensource/sm/cleanup.py", line 1548, in _gcLoop
| |
| − | candidate.coalesce(dryRun)
| |
| − | File "/opt/xensource/sm/cleanup.py", line 477, in coalesce
| |
| − | self._coalesceBegin()
| |
| − | File "/opt/xensource/sm/cleanup.py", line 619, in _coalesceBegin
| |
| − | self.parent.validate()
| |
| − | File "/opt/xensource/sm/cleanup.py", line 605, in validate
| |
| − | raise util.SMException("VHD %s corrupted" % self.toString())
| |
| − | <14322> 2012-08-21 12:22:23.300356 *
| |
| − | </pre>
| |
| − |
| |
| − | * Example check
| |
| − | <pre>
| |
| − | [root@9-48-89-300825 ~]# vhd-util read -p -n /var/run/sr-mount/7a142a29-9556-ee47-1e04-1a8517813369/1d994938-058e-102f-80dc-001ec95cb0b6.vhd
| |
| − | VHD Footer Summary:
| |
| − | -------------------
| |
| − | Cookie : conectix
| |
| − | Features : (0x00000002) <RESV>
| |
| − | File format version : Major: 1, Minor: 0
| |
| − | Data offset : 512
| |
| − | Timestamp : Wed Oct 27 09:45:01 2010
| |
| − | Creator Application : 'tap'
| |
| − | Creator version : Major: 1, Minor: 3
| |
| − | Creator OS : Unknown!
| |
| − | Original disk size : 20480 MB (21474836480 Bytes)
| |
| − | Current disk size : 40960 MB (42949672960 Bytes)
| |
| − | Geometry : Cyl: 20560, Hds: 16, Sctrs: 255
| |
| − | : = 40959 MB (42949017600 Bytes)
| |
| − | Disk type : Dynamic hard disk
| |
| − | Checksum : 0xfffff246|0xfffff246 (Good!)
| |
| − | UUID : 4223b679-0941-49af-b46a-2077b947176e
| |
| − | Saved state : No
| |
| − | Hidden : 1
| |
| − | VHD Header Summary:
| |
| − | -------------------
| |
| − | Cookie : cxsparse
| |
| − | Data offset (unusd) : 18446744073709
| |
| − | Table offset : 1536
| |
| − | Header version : 0x00010000
| |
| − | Max BAT size : 20480
| |
| − | Block size : 2097152 (2 MB)
| |
| − | Parent name :
| |
| − | Parent UUID : 00000000-0000-0000-0000-000000000000
| |
| − | Parent timestamp : Sat Jan 1 00:00:00 2000
| |
| − | Checksum : 0xfffff427|0xfffff427 (Good!)
| |
| − | VHD Batmap Summary:
| |
| − | -------------------
| |
| − | Batmap offset : 83968
| |
| − | Batmap size (secs) : 5
| |
| − | Batmap version : 0x00010002
| |
| − | Checksum : 0xfffccebd|0xfffccebd (Good!)
| |
| − | </pre>
| |
| − |
| |
| − | * The header contains the initial cookie:
| |
| − | <pre>
| |
| − | [root@9-48-89-300825 ~]# head -c 512 /var/run/sr-mount/7a142a29-9556-ee47-1e04-1a8517813369/1d994938-058e-102f-80dc-001ec95cb0b6.vhd | hexdump -C
| |
| − | 00000000 63 6f 6e 65 63 74 69 78 00 00 00 02 00 01 00 00 |conectix........|
| |
| − | 00000010 00 00 00 00 00 00 02 00 14 5a b1 1d 74 61 70 00 |.........Z..tap.|
| |
| − | 00000020 00 01 00 03 00 00 00 00 00 00 00 05 00 00 00 00 |................|
| |
| − | 00000030 00 00 00 0a 00 00 00 00 50 50 10 ff 00 00 00 03 |........PP......|
| |
| − | 00000040 ff ff f2 46 42 23 b6 79 09 41 49 af b4 6a 20 77 |...FB#.y.AI..j w|
| |
| − | 00000050 b9 47 17 6e 00 01 00 00 00 00 00 00 00 00 00 00 |.G.n............|
| |
| − | 00000060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
| |
| − | *
| |
| − | 00000200
| |
| − | </pre>
| |
| − |
| |
| − | * Footer output:
| |
| − | <pre>
| |
| − | [root@9-48-89-300825 ~]# tail -c 512 /var/run/sr-mount/7a142a29-9556-ee47-1e04-1a8517813369/1d994938-058e-102f-80dc-001ec95cb0b6.vhd | hexdump -C
| |
| − | 00000000 00 49 00 6e 00 74 00 65 00 72 00 66 00 61 00 63 |.I.n.t.e.r.f.a.c|
| |
| − | 00000010 00 65 00 73 00 2c 00 20 00 56 00 65 00 72 00 73 |.e.s.,. .V.e.r.s|
| |
| − | 00000020 00 69 00 6f 00 6e 00 3d 00 31 00 2e 00 30 00 2e |.i.o.n.=.1...0..|
| |
| − | 00000030 00 35 00 30 00 30 00 30 00 2e 00 30 00 2c 00 20 |.5.0.0.0...0.,. |
| |
| − | 00000040 00 43 00 75 00 6c 00 74 00 75 00 72 00 65 00 3d |.C.u.l.t.u.r.e.=|
| |
| − | 00000050 00 6e 00 65 00 75 00 74 00 72 00 61 00 6c 00 2c |.n.e.u.t.r.a.l.,|
| |
| − | 00000060 00 20 00 50 00 75 00 62 00 6c 00 69 00 63 00 4b |. .P.u.b.l.i.c.K|
| |
| − | 00000070 00 65 00 79 00 54 00 6f 00 6b 00 65 00 6e 00 3d |.e.y.T.o.k.e.n.=|
| |
| − | 00000080 00 62 00 30 00 33 00 66 00 35 00 66 00 37 00 66 |.b.0.3.f.5.f.7.f|
| |
| − | 00000090 00 31 00 31 00 64 00 35 00 30 00 61 00 33 00 61 |.1.1.d.5.0.a.3.a|
| |
| − | 000000a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
| |
| − | 000000b0 90 00 00 00 6e 6b 20 00 c0 3b 24 09 58 06 cc 01 |....nk ..;$.X...|
| |
| − | 000000c0 00 00 00 00 98 af c5 02 00 00 00 00 00 00 00 00 |................|
| |
| − | 000000d0 ff ff ff ff ff ff ff ff 00 00 00 00 ff ff ff ff |................|
| |
| − | 000000e0 ff ff ff ff ff ff ff ff 00 00 00 00 00 00 00 00 |................|
| |
| − | 000000f0 00 00 00 00 00 00 00 00 48 00 00 00 08 00 00 00 |........H.......|
| |
| − | 00000100 33 30 33 34 37 32 38 33 38 00 00 00 6c 68 00 00 |303472838...lh..|
| |
| − | 00000110 70 62 c6 02 95 12 01 00 28 00 00 00 76 6b 0b 00 |pb......(...vk..|
| |
| − | 00000120 ac 00 00 00 c8 62 c6 02 01 00 00 00 01 00 ff ff |.....b..........|
| |
| − | 00000130 44 69 73 70 6c 61 79 4e 61 6d 65 ff ff 81 00 00 |DisplayName.....|
| |
| − | 00000140 c0 00 00 00 6e 6b 20 00 e0 61 60 0e 58 06 cc 01 |....nk ..a`.X...|
| |
| − | 00000150 00 00 00 00 60 6e c3 02 00 00 00 00 00 00 00 00 |....`n..........|
| |
| − | 00000160 ff ff ff ff ff ff ff ff 04 00 00 00 00 6f c3 02 |.............o..|
| |
| − | 00000170 ff ff ff ff ff ff ff ff 00 00 00 00 00 00 00 00 |................|
| |
| − | 00000180 16 00 00 00 8e 00 00 00 00 00 00 00 03 00 00 00 |................|
| |
| − | 00000190 32 30 66 00 00 00 00 00 68 00 00 00 76 6b 03 00 |20f.....h...vk..|
| |
| − | 000001a0 24 00 00 00 b8 ef c5 02 03 00 00 00 01 00 00 00 |$...............|
| |
| − | 000001b0 53 49 47 00 00 00 00 00 28 00 00 00 48 13 73 8f |SIG.....(...H.s.|
| |
| − | 000001c0 96 ab ce 4e b1 f7 5a 3f 42 ec 22 e0 15 8a 2d 0e |...N..Z?B."...-.|
| |
| − | 000001d0 51 c6 42 b4 67 dd 93 ba e6 c7 25 f8 b0 71 74 c9 |Q.B.g.....%..qt.|
| |
| − | 000001e0 20 00 00 00 76 6b 06 00 04 00 00 80 02 10 00 00 | ...vk..........|
| |
| − | 000001f0 04 00 00 00 01 00 00 00 53 74 61 74 75 73 00 00 |........Status..|
| |
| − | 00000200
| |
| − | 00000200
| |
| − | </pre>
| |
| − |
| |
| − | * Append the "header" to the "footer":
| |
| − | head -c 512 /var/run/sr-mount/7a142a29-9556-ee47-1e04-1a8517813369/1d994938-058e-102f-80dc-001ec95cb0b6.vhd >> /var/run/sr-mount/7a142a29-9556-ee47-1e04-1a8517813369/1d994938-058e-102f-80dc-001ec95cb0b6.vhd
| |
| − |
| |
| − | * Now, check that the "footer" is the same as the header:
| |
| − | <pre>
| |
| − | tail -c 512 /var/run/sr-mount/7a142a29-9556-ee47-1e04-1a8517813369/1d994938-058e-102f-80dc-001ec95cb0b6.vhd | hexdump -C
| |
| − | 00000000 63 6f 6e 65 63 74 69 78 00 00 00 02 00 01 00 00 |conectix........|
| |
| − | 00000010 00 00 00 00 00 00 02 00 14 5a b1 1d 74 61 70 00 |.........Z..tap.|
| |
| − | 00000020 00 01 00 03 00 00 00 00 00 00 00 05 00 00 00 00 |................|
| |
| − | 00000030 00 00 00 0a 00 00 00 00 50 50 10 ff 00 00 00 03 |........PP......|
| |
| − | 00000040 ff ff f2 46 42 23 b6 79 09 41 49 af b4 6a 20 77 |...FB#.y.AI..j w|
| |
| − | 00000050 b9 47 17 6e 00 01 00 00 00 00 00 00 00 00 00 00 |.G.n............|
| |
| − | 00000060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
| |
| − | *
| |
| − | 00000200
| |
| − | </pre>
| |
| − |
| |
| − | There are times in which the cookie does not land on a 512-byte boundary, so try the <code>vhd-util</code> repair, and if unsuccessful you will have to append the arbitrary amount of data to the end of the file until that 512 barrier is reached. Then you can append the cookie and re-run the <code>vhd-util</code> repair.
| |
| | | | |
| | [[Category:XenServer]] | | [[Category:XenServer]] |
