Vhd-util

From Christoph's Personal Wiki
Revision as of 10:23, 21 August 2013 by Christoph (Talk | contribs) (New page: '''<code>vhd-util</code>''' is a utility for working with Virtual HDD (VHDs) in a XenServer environment. ==Fixing a corrupt VHD footer== ''Note: The following section is based on [htt...)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

vhd-util is a utility for working with Virtual HDD (VHDs) in a XenServer environment.

Fixing a corrupt VHD footer

Note: The following section is based on Mark Silence's original article.

Issue
Sometimes, the footer on the customer's VHD will become corrupted.
Symptoms
There are several possible symptoms that are manifested by this issue.

If the customer takes an image of this server, and then attempts to build a new server from it, they may encounter an error like this: 

2012-10-31 18:06:12 TRACE nova.utils [instance: 4dc16179-9e98-416e-ada0-4a0f1bdbb690]
Failure: ['XENAPI_PLUGIN_FAILURE', 'download_vhd', 'Exception', 
"Unexpected output '/var/run/sr-mount/9bfaf37b-720b-7cf1-13a9-8c4ef3237322/tmpFco2q9/c8f6bdd0-8e73-407c-9449-546b7dc958aa.vhd is valid\n
/var/run/sr-mount/9bfaf37b-720b-7cf1-13a9-8c4ef3237322/tmpFco2q9/4f7306a6-818a-4a39-aa25-2768f53ed594.vhd is valid\n
primary footer invalid: invalid cookie\n
/var/run/sr-mount/9bfaf37b-720b-7cf1-13a9-8c4ef3237322/tmpFco2q9/24e9f9a6-c4a6-4fcf-a40b-b57b7c90d2f2.vhd appears invalid; dumping metadata\n
VHD Footer Summary:\n-------------------\nCookie : conectix\n
Features : (0x00000002) <RESV>\nFile format version : Major: 1, Minor: 0\nData offset : 512\n
Timestamp : Tue May 1 20:54:45 2012\nCreator Application : 'tap'\n
Creator version : Major: 1, Minor: 3\nCreator OS : Unknown!\nOriginal disk size : 9728 MB (10200547328 Bytes)\n
Current disk size : 163840 MB (171798691840 Bytes)\nGeometry : Cyl: 65535, Hds: 16, Sctrs: 255\n
: = 130558 MB (136899993600 Bytes)\nDisk type : Dynamic hard disk\nChecksum : 0xffffef6f|0xffffef6f (Good!)\nUUID : ae71e079-63e8-455d-8682-7c44d04b3168\nSaved state : No\nHidden : 1\n\nVHD Header Summary:
\n-------------------\nCookie : cxsparse\nData offset (unusd) : 18446744073709\nTable offset : 1536\n
Header version : 0x00010000\nMax BAT size : 81920\nBlock size : 2097152 (2 MB)\nParent name : \n
Parent UUID : 00000000-0000-0000-0000-000000000000\nParent timestamp : Sat Jan 1 00:00:00 2000\n
Checksum : 0xfffff436|0xfffff436 (Good!)\n\nVHD Batmap Summary:\n-------------------\n
Batmap offset : 329728\nBatmap size (secs) : 20\nBatmap version : 0x00010002\n
Checksum : 0xfff6398e|0xfff6398e (Good!)\n\nerror checking parents: -22\n' from vhd-util"] 
2012-10-31 18:06:12 TRACE nova.utils [instance: 4dc16179-9e98-416e-ada0-4a0f1bdbb690]

Notice the message "vhd is valid \ primary footer invalid".

The VHD contains a header and a footer, which hold duplicate information. This is used to help verify the integrity of the VHD file. Sometimes, for currently unknown reasons, the footer becomes corrupted or goes missing completely. As long as the customer's data does not extend to the very end of their VHD, we can duplicate the information from the header to create a new footer.

Solution

If this problem is observer with a saved image, there is no fixing the image. You will have to located the original server the image was created from, and fix that server, and then take a new image.

You can use this to look at the footers for all VHDs to see which ones have problems. You can also run it specifically on one if you know it has issues. 

for item in /var/run/sr-mount/*/*; do echo $item && tail -c 512 $item | hexdump -C; done;

Note: Sometimes, if the customer is not using all of their VHD yet, and the footer is missing, you will see nothing when running a tail -c 512. I have had to bump it up to tail -c 5120000 before seeing any data before, it just depends on how much empty space is at the end of their VHD.

You can also use this utility to confirm the integrity of the VHD, also great for checking that a repair worked afterwards. 

vhd-util read -p -n /var/run/sr-mount/<SR-UUID>/<VHD-UUID>.vhd

As long as their is not data at the end, you can usually use this to place the footer back on the file, but sometimes you will miss the 512-byte boundary and it will not work. 

head -c 512 /var/run/sr-mount/$SR-UUID/XXXX..vhd >> /var/run/sr-mount/$SR-UUID/XXXXXXX.vhd && vhd-util repair -n /var/run/sr-mount/$SR-UUID/$UUID_of_VHD

Corrupt Truncated VHD Repair (Conectix Cookie)

Note: The following section is based on Chase Naquin's original article.

WARNING: DO NOT RUN ANY OF THE FOLLOWING COMMANDS UNLESS YOU KNOW WHAT YOU ARE DOING!

  • Find Corrupt VHDs:
for item in /var/run/sr-mount/*/*; do echo $item && tail -c 512 $item | hexdump -C; done;
  • Repair a single corrupt VHD:
CVHD=c5c0cdc3-c2c4-4c43-a5a2-845f36b866a4;
fn=$(ls /var/run/sr-mount/*/${CVHD}*.vhd);
DATA=$(wc -c $fn | awk '{print (512-($1 % 512))%512}');
if [ "$DATA" -ne "0" ]; then for i in $(eval echo {1..$DATA}); do echo -n '0' >> $fn; done; fi;
head -c 512 $fn >> $fn;
vhd-util repair -n $fn;
  • Repair a corrupt VHD in SMlog:
error_log_fn='/var/log/SMlog';
CVHD="$(awk '/EXCEPTION/ && /corrupt/ {print substr($8,2,8)}' $error_log_fn | tail -1)";
fn=$(ls /var/run/sr-mount/*/${CVHD}*.vhd);
DATA=$(wc -c $fn | awk '{print (512-($1 % 512))%512}');
if [ "$DATA" -ne "0" ]; then for i in $(eval echo {1..$DATA}); do echo -n '0' >> $fn; done; fi;
head -c 512 $fn >> $fn;
vhd-util repair -n $fn;
  • Teran's fix all the things at once. Make sure the box has dc before running.
for vhd in /var/run/sr-mount/*/*.vhd; do
  vhd-util check -n $vhd > /dev/null || (DATA=$((stat -c%s $vhd; echo -n "512 %p") | dc);
    if [ "$DATA" -ne "0" ]; then
      for i in $(eval echo {1..$DATA}); do
        echo -n '0' >> $vhd;
      done;
    fi;
  head -c 512 $vhd >> $vhd;vhd-util repair -n $vhd);
done
  • Teran's check for byte misaligned if curious:
for vhd in /var/run/sr-mount/*/*.vhd; do (stat -c%s $vhd; echo 512%p) | dc; done | uniq
  • Example error/exception:
[26919] 2012-05-15 00:18:35.127755 FAILED: (errno 22) stdout: 'primary footer invalid: invalid cookie
/var/run/sr-mount/7a142a29-9556-ee47-1e04-1a8517813369/1d994938-058e-102f-80dc-001ec95cb0b6.vhd appears invalid;
dumping metadata
VHD Footer Summary:
-------------------
Cookie              : conectix
Features            : (0x00000002) <RESV>
File format version : Major: 1, Minor: 0
Data offset         : 512
Timestamp           : Wed Oct 27 09:45:01 2010
Creator Application : 'tap'
Creator version     : Major: 1, Minor: 3
Creator OS          : Unknown!
Original disk size  : 20480 MB (21474836480 Bytes)
Current disk size   : 40960 MB (42949672960 Bytes)
Geometry            : Cyl: 20560, Hds: 16, Sctrs: 255
                    : = 40959 MB (42949017600 Bytes)
Disk type           : Dynamic hard disk
Checksum            : 0xfffff246|0xfffff246 (Good!)
UUID                : 4223b679-0941-49af-b46a-2077b947176e
Saved state         : No
Hidden              : 1
VHD Header Summary:
-------------------
Cookie              : cxsparse
Data offset (unusd) : 18446744073709
Table offset        : 1536
Header version      : 0x00010000
Max BAT size        : 20480
Block size          : 2097152 (2 MB)
Parent name         :
Parent UUID         : 00000000-0000-0000-0000-000000000000
Parent timestamp    : Sat Jan  1 00:00:00 2000
Checksum            : 0xfffff427|0xfffff427 (Good!)
VHD Batmap Summary:
-------------------
Batmap offset       : 83968
Batmap size (secs)  : 5
Batmap version      : 0x00010002
Checksum            : 0xfffccebd|0xfffccebd (Good!)
', stderr: ''
<26919> 2012-05-15 00:18:35.128104      In cleanup
<26919> 2012-05-15 00:18:35.129656      ***
<26919> 2012-05-15 00:18:35.129711               ***********************
<26919> 2012-05-15 00:18:35.129754               *  E X C E P T I O N  *
<26919> 2012-05-15 00:18:35.129797               ***********************
<26919> 2012-05-15 00:18:35.129850      coalesce: EXCEPTION util.SMException, VHD *1d994938(40.00G/14.27G) corrupted
<26919> 2012-05-15 00:18:35.129894        File "/opt/xensource/sm/cleanup.py", line 1548, in _gcLoop
    candidate.coalesce(dryRun)
  File "/opt/xensource/sm/cleanup.py", line 477, in coalesce
    self._coalesceBegin()
  File "/opt/xensource/sm/cleanup.py", line 619, in _coalesceBegin
    self.parent.validate()
  File "/opt/xensource/sm/cleanup.py", line 605, in validate
    raise util.SMException("VHD %s corrupted" % self.toString())
<14322> 2012-08-21 12:22:23.300356    *
  • Example check
[root@9-48-89-300825 ~]# vhd-util read -p -n /var/run/sr-mount/7a142a29-9556-ee47-1e04-1a8517813369/1d994938-058e-102f-80dc-001ec95cb0b6.vhd
VHD Footer Summary:
-------------------
Cookie              : conectix
Features            : (0x00000002) <RESV>
File format version : Major: 1, Minor: 0
Data offset         : 512
Timestamp           : Wed Oct 27 09:45:01 2010
Creator Application : 'tap'
Creator version     : Major: 1, Minor: 3
Creator OS          : Unknown!
Original disk size  : 20480 MB (21474836480 Bytes)
Current disk size   : 40960 MB (42949672960 Bytes)
Geometry            : Cyl: 20560, Hds: 16, Sctrs: 255
                    : = 40959 MB (42949017600 Bytes)
Disk type           : Dynamic hard disk
Checksum            : 0xfffff246|0xfffff246 (Good!)
UUID                : 4223b679-0941-49af-b46a-2077b947176e
Saved state         : No
Hidden              : 1
VHD Header Summary:
-------------------
Cookie              : cxsparse
Data offset (unusd) : 18446744073709
Table offset        : 1536
Header version      : 0x00010000
Max BAT size        : 20480
Block size          : 2097152 (2 MB)
Parent name         :
Parent UUID         : 00000000-0000-0000-0000-000000000000
Parent timestamp    : Sat Jan  1 00:00:00 2000
Checksum            : 0xfffff427|0xfffff427 (Good!)
VHD Batmap Summary:
-------------------
Batmap offset       : 83968
Batmap size (secs)  : 5
Batmap version      : 0x00010002
Checksum            : 0xfffccebd|0xfffccebd (Good!)
  • The header contains the initial cookie:
[root@9-48-89-300825 ~]# head -c 512 /var/run/sr-mount/7a142a29-9556-ee47-1e04-1a8517813369/1d994938-058e-102f-80dc-001ec95cb0b6.vhd | hexdump -C
00000000  63 6f 6e 65 63 74 69 78  00 00 00 02 00 01 00 00  |conectix........|
00000010  00 00 00 00 00 00 02 00  14 5a b1 1d 74 61 70 00  |.........Z..tap.|
00000020  00 01 00 03 00 00 00 00  00 00 00 05 00 00 00 00  |................|
00000030  00 00 00 0a 00 00 00 00  50 50 10 ff 00 00 00 03  |........PP......|
00000040  ff ff f2 46 42 23 b6 79  09 41 49 af b4 6a 20 77  |...FB#.y.AI..j w|
00000050  b9 47 17 6e 00 01 00 00  00 00 00 00 00 00 00 00  |.G.n............|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200
  • Footer output:
[root@9-48-89-300825 ~]# tail -c 512 /var/run/sr-mount/7a142a29-9556-ee47-1e04-1a8517813369/1d994938-058e-102f-80dc-001ec95cb0b6.vhd | hexdump -C
00000000  00 49 00 6e 00 74 00 65  00 72 00 66 00 61 00 63  |.I.n.t.e.r.f.a.c|
00000010  00 65 00 73 00 2c 00 20  00 56 00 65 00 72 00 73  |.e.s.,. .V.e.r.s|
00000020  00 69 00 6f 00 6e 00 3d  00 31 00 2e 00 30 00 2e  |.i.o.n.=.1...0..|
00000030  00 35 00 30 00 30 00 30  00 2e 00 30 00 2c 00 20  |.5.0.0.0...0.,. |
00000040  00 43 00 75 00 6c 00 74  00 75 00 72 00 65 00 3d  |.C.u.l.t.u.r.e.=|
00000050  00 6e 00 65 00 75 00 74  00 72 00 61 00 6c 00 2c  |.n.e.u.t.r.a.l.,|
00000060  00 20 00 50 00 75 00 62  00 6c 00 69 00 63 00 4b  |. .P.u.b.l.i.c.K|
00000070  00 65 00 79 00 54 00 6f  00 6b 00 65 00 6e 00 3d  |.e.y.T.o.k.e.n.=|
00000080  00 62 00 30 00 33 00 66  00 35 00 66 00 37 00 66  |.b.0.3.f.5.f.7.f|
00000090  00 31 00 31 00 64 00 35  00 30 00 61 00 33 00 61  |.1.1.d.5.0.a.3.a|
000000a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000b0  90 00 00 00 6e 6b 20 00  c0 3b 24 09 58 06 cc 01  |....nk ..;$.X...|
000000c0  00 00 00 00 98 af c5 02  00 00 00 00 00 00 00 00  |................|
000000d0  ff ff ff ff ff ff ff ff  00 00 00 00 ff ff ff ff  |................|
000000e0  ff ff ff ff ff ff ff ff  00 00 00 00 00 00 00 00  |................|
000000f0  00 00 00 00 00 00 00 00  48 00 00 00 08 00 00 00  |........H.......|
00000100  33 30 33 34 37 32 38 33  38 00 00 00 6c 68 00 00  |303472838...lh..|
00000110  70 62 c6 02 95 12 01 00  28 00 00 00 76 6b 0b 00  |pb......(...vk..|
00000120  ac 00 00 00 c8 62 c6 02  01 00 00 00 01 00 ff ff  |.....b..........|
00000130  44 69 73 70 6c 61 79 4e  61 6d 65 ff ff 81 00 00  |DisplayName.....|
00000140  c0 00 00 00 6e 6b 20 00  e0 61 60 0e 58 06 cc 01  |....nk ..a`.X...|
00000150  00 00 00 00 60 6e c3 02  00 00 00 00 00 00 00 00  |....`n..........|
00000160  ff ff ff ff ff ff ff ff  04 00 00 00 00 6f c3 02  |.............o..|
00000170  ff ff ff ff ff ff ff ff  00 00 00 00 00 00 00 00  |................|
00000180  16 00 00 00 8e 00 00 00  00 00 00 00 03 00 00 00  |................|
00000190  32 30 66 00 00 00 00 00  68 00 00 00 76 6b 03 00  |20f.....h...vk..|
000001a0  24 00 00 00 b8 ef c5 02  03 00 00 00 01 00 00 00  |$...............|
000001b0  53 49 47 00 00 00 00 00  28 00 00 00 48 13 73 8f  |SIG.....(...H.s.|
000001c0  96 ab ce 4e b1 f7 5a 3f  42 ec 22 e0 15 8a 2d 0e  |...N..Z?B."...-.|
000001d0  51 c6 42 b4 67 dd 93 ba  e6 c7 25 f8 b0 71 74 c9  |Q.B.g.....%..qt.|
000001e0  20 00 00 00 76 6b 06 00  04 00 00 80 02 10 00 00  | ...vk..........|
000001f0  04 00 00 00 01 00 00 00  53 74 61 74 75 73 00 00  |........Status..|
00000200
00000200
  • Append the "header" to the "footer":
head -c 512 /var/run/sr-mount/7a142a29-9556-ee47-1e04-1a8517813369/1d994938-058e-102f-80dc-001ec95cb0b6.vhd >> /var/run/sr-mount/7a142a29-9556-ee47-1e04-1a8517813369/1d994938-058e-102f-80dc-001ec95cb0b6.vhd
  • Now, check that the "footer" is the same as the header:
tail -c 512 /var/run/sr-mount/7a142a29-9556-ee47-1e04-1a8517813369/1d994938-058e-102f-80dc-001ec95cb0b6.vhd | hexdump -C
00000000  63 6f 6e 65 63 74 69 78  00 00 00 02 00 01 00 00  |conectix........|
00000010  00 00 00 00 00 00 02 00  14 5a b1 1d 74 61 70 00  |.........Z..tap.|
00000020  00 01 00 03 00 00 00 00  00 00 00 05 00 00 00 00  |................|
00000030  00 00 00 0a 00 00 00 00  50 50 10 ff 00 00 00 03  |........PP......|
00000040  ff ff f2 46 42 23 b6 79  09 41 49 af b4 6a 20 77  |...FB#.y.AI..j w|
00000050  b9 47 17 6e 00 01 00 00  00 00 00 00 00 00 00 00  |.G.n............|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200

There are times in which the cookie does not land on a 512-byte boundary, so try the vhd-util repair, and if unsuccessful you will have to append the arbitrary amount of data to the end of the file until that 512 barrier is reached. Then you can append the cookie and re-run the vhd-util repair.

Retrieved from "http://wiki.christophchamp.com/index.php?title=Vhd-util&oldid=5889"