[PATCH 0/4] (Refactored) VHD Support Addition

[PATCH 0/4] (Refactored) VHD Support Addition

[Ben Guthro]

[PATCH 0/4] (Refactored) VHD Support addition
The patches that follow implement Microsoft's VHD format support, and
apply to xen-unstable.hg CS 15414:3827b12d24dd
Per Mark McLoughlin's suggestions - these have been reworked to use the 
tapaio wrapper.

This is based on the VHD format spec covered by Microsoft's "Open
Specification Promise":
http://www.microsoft.com/interop/osp/default.mspx
MS VHD Spec can be found at the following link:
http://download.microsoft.com/download/f/f/e/ffef50a5-07dd-4cf8-aaa3-442c0673a029/Virtual%20Hard%20Disk%20Format%20Spec_10_18_06.doc

It is divided up into the following patches:
qemu-vhd-support.patch        [1/4]
tapdisk-vhd-support.patch     [2/4]
blktap-fixes.patch            [3/4]
vdisk-support.patch           [4/4]

### libvdisk ####
These patches implement a shared library (libvdisk) between qemu, and
blktap which can be expanded upon for additional formats in the future.
This was done in order to avoid code duplcation between the two areas.
At this time, it only supports the vhd format.

### vdisk_tool ###
The vdisk_tool provides a means of creating, and converting vhd files.
Like libvdisk, it is designed with future support of additional formats
in mind, but currently only supports vhd.

Usage: vdisk_tool OPTIONS -# <format-specific options> <filename>
  OPTIONS:
    [-f <format>] [-C] [-H] [-M] [-D <block> [-b <num_blocks>] -o outfile]]
    -C        Create a vdisk
    -H        Read vdisk headers from file
    -M        Modify a vdisk
    -D        Dump a vhd
      block       first block to read (required)
      num_blocks       number of blocks to read. If not
                specified, whole file will be read
      outfile       output file. If not specified,
                stdout is used
    

With this, you can convert foo.vhd to raw volume /dev/foo/bar:
      vdisk_tool -D 0 -o /dev/foo/bar foo.vhd

To convert foo.vhd to fixed bar.vhd
      vdisk_tool -D 0 -o bar.vhd foo.vhd
      vdisk_tool -C -# -S <sizeB> -f bar.vhd

To read blocks 2 through 6 from foo.vhd to stdout:
      vdisk_tool -D 2 -b 5 foo.vhd

Re: [PATCH 0/4] (Refactored) VHD Support Addition

[Ben Guthro]

My mistake - I had my patches pushed when I did the hg tip.

This applies against CS 15410:a83632dfbb28

Ben Guthro wrote:
> [PATCH 0/4] (Refactored) VHD Support addition
> The patches that follow implement Microsoft's VHD format support, and
> apply to xen-unstable.hg CS 15414:3827b12d24dd
> Per Mark McLoughlin's suggestions - these have been reworked to use the 
> tapaio wrapper.
>
> This is based on the VHD format spec covered by Microsoft's "Open
> Specification Promise":
> http://www.microsoft.com/interop/osp/default.mspx
> MS VHD Spec can be found at the following link:
> http://download.microsoft.com/download/f/f/e/ffef50a5-07dd-4cf8-aaa3-442c0673a029/Virtual%20Hard%20Disk%20Format%20Spec_10_18_06.doc
>
> It is divided up into the following patches:
> qemu-vhd-support.patch        [1/4]
> tapdisk-vhd-support.patch     [2/4]
> blktap-fixes.patch            [3/4]
> vdisk-support.patch           [4/4]
>
> ### libvdisk ####
> These patches implement a shared library (libvdisk) between qemu, and
> blktap which can be expanded upon for additional formats in the future.
> This was done in order to avoid code duplcation between the two areas.
> At this time, it only supports the vhd format.
>
> ### vdisk_tool ###
> The vdisk_tool provides a means of creating, and converting vhd files.
> Like libvdisk, it is designed with future support of additional formats
> in mind, but currently only supports vhd.
>
> Usage: vdisk_tool OPTIONS -# <format-specific options> <filename>
>   OPTIONS:
>     [-f <format>] [-C] [-H] [-M] [-D <block> [-b <num_blocks>] -o outfile]]
>     -C        Create a vdisk
>     -H        Read vdisk headers from file
>     -M        Modify a vdisk
>     -D        Dump a vhd
>       block       first block to read (required)
>       num_blocks       number of blocks to read. If not
>                 specified, whole file will be read
>       outfile       output file. If not specified,
>                 stdout is used
>     
>
> With this, you can convert foo.vhd to raw volume /dev/foo/bar:
>       vdisk_tool -D 0 -o /dev/foo/bar foo.vhd
>
> To convert foo.vhd to fixed bar.vhd
>       vdisk_tool -D 0 -o bar.vhd foo.vhd
>       vdisk_tool -C -# -S <sizeB> -f bar.vhd
>
> To read blocks 2 through 6 from foo.vhd to stdout:
>       vdisk_tool -D 2 -b 5 foo.vhd
>
>
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>   

Re: [PATCH 0/4] (Refactored) VHD Support Addition

[Daniel P. Berrange]

On Thu, Jun 21, 2007 at 01:27:32PM -0400, Ben Guthro wrote:
> [PATCH 0/4] (Refactored) VHD Support addition
> The patches that follow implement Microsoft's VHD format support, and
> apply to xen-unstable.hg CS 15414:3827b12d24dd
> Per Mark McLoughlin's suggestions - these have been reworked to use the 
> tapaio wrapper.
> 
> This is based on the VHD format spec covered by Microsoft's "Open
> Specification Promise":
> http://www.microsoft.com/interop/osp/default.mspx
> MS VHD Spec can be found at the following link:
> http://download.microsoft.com/download/f/f/e/ffef50a5-07dd-4cf8-aaa3-442c0673a029/Virtual%20Hard%20Disk%20Format%20Spec_10_18_06.doc
> 
> It is divided up into the following patches:
> qemu-vhd-support.patch        [1/4]
> tapdisk-vhd-support.patch     [2/4]
> blktap-fixes.patch            [3/4]
> vdisk-support.patch           [4/4]
> 
> ### libvdisk ####
> These patches implement a shared library (libvdisk) between qemu, and
> blktap which can be expanded upon for additional formats in the future.
> This was done in order to avoid code duplcation between the two areas.
> At this time, it only supports the vhd format.
> 
> ### vdisk_tool ###
> The vdisk_tool provides a means of creating, and converting vhd files.
> Like libvdisk, it is designed with future support of additional formats
> in mind, but currently only supports vhd.

Why not just add vhd support to qemu's existing qemu-img tool, then it will
support all image formats, right from the start, rather than inventing yet
another tool :-(

Any changes to QEMU should be sent to the QEMU upstream mailing lists for
review/inclusion too, otherwise the Xen fork gets even worse than it 
already is

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: [PATCH 0/4] (Refactored) VHD Support Addition

[Steve Ofsthun]

Daniel P. Berrange wrote:
> On Thu, Jun 21, 2007 at 01:27:32PM -0400, Ben Guthro wrote:

>> ### vdisk_tool ###
>> The vdisk_tool provides a means of creating, and converting vhd files.
>> Like libvdisk, it is designed with future support of additional formats
>> in mind, but currently only supports vhd.
> 
> Why not just add vhd support to qemu's existing qemu-img tool, then it will
> support all image formats, right from the start, rather than inventing yet
> another tool :-(

Well, when we started this, it wasn't clear we were going to be able to
publicly release any code in support of the VHD format.  Microsoft later
relaxed the restrictions, and by that time the separate tool already
existed.  We can investigate integrating the work into qemu-img.

> Any changes to QEMU should be sent to the QEMU upstream mailing lists for
> review/inclusion too, otherwise the Xen fork gets even worse than it 
> already is

Agreed.

Steve
-- 
Steve Ofsthun - Virtual Iron Software, Inc.

Re: [PATCH 0/4] (Refactored) VHD Support Addition

[Anthony Liguori]

Steve Ofsthun wrote:
> Daniel P. Berrange wrote:
>> On Thu, Jun 21, 2007 at 01:27:32PM -0400, Ben Guthro wrote:
> 
>>> ### vdisk_tool ###
>>> The vdisk_tool provides a means of creating, and converting vhd files.
>>> Like libvdisk, it is designed with future support of additional formats
>>> in mind, but currently only supports vhd.
>> Why not just add vhd support to qemu's existing qemu-img tool, then it will
>> support all image formats, right from the start, rather than inventing yet
>> another tool :-(
> 
> Well, when we started this, it wasn't clear we were going to be able to
> publicly release any code in support of the VHD format.  Microsoft later
> relaxed the restrictions, and by that time the separate tool already
> existed.  We can investigate integrating the work into qemu-img.

QEMU already supports VHD (it calls it VPC) and supported it long before 
Microsoft released any specs.  QEMU supported VMDK before VMware release 
the specs for that too.

Is there a particular feature of VHD/VPC that your code adds that QEMU 
didn't already support?

Regards,

Anthony Liguori

>> Any changes to QEMU should be sent to the QEMU upstream mailing lists for
>> review/inclusion too, otherwise the Xen fork gets even worse than it 
>> already is
> 
> Agreed.
> 
> Steve

Re: [PATCH 0/4] (Refactored) VHD Support Addition

[Steve Ofsthun]

Anthony Liguori wrote:
> Steve Ofsthun wrote:
>> Daniel P. Berrange wrote:
>>> On Thu, Jun 21, 2007 at 01:27:32PM -0400, Ben Guthro wrote:
>>
>>>> ### vdisk_tool ###
>>>> The vdisk_tool provides a means of creating, and converting vhd files.
>>>> Like libvdisk, it is designed with future support of additional formats
>>>> in mind, but currently only supports vhd.
>>> Why not just add vhd support to qemu's existing qemu-img tool, then
>>> it will
>>> support all image formats, right from the start, rather than
>>> inventing yet
>>> another tool :-(
>>
>> Well, when we started this, it wasn't clear we were going to be able to
>> publicly release any code in support of the VHD format.  Microsoft later
>> relaxed the restrictions, and by that time the separate tool already
>> existed.  We can investigate integrating the work into qemu-img.
> 
> QEMU already supports VHD (it calls it VPC) and supported it long before
> Microsoft released any specs.  QEMU supported VMDK before VMware release
> the specs for that too.

This code was developed in the Xen 3.0.2 days when the integrated qemu had
no vpc support.  Our original code was targeted at PV on HVM drivers using
blocktap.  We needed to support the same format unaccelerated and so the
qemu integration was also done.

> Is there a particular feature of VHD/VPC that your code adds that QEMU
> didn't already support?

Not having any experience with VPC, I couldn't tell you.  I can say that we
tried to adhere to the Microsoft spec rather than reverse engineering the
format from examples.  Perusing the VPC code, there are many references to
unknown fields within the format.

Is there a complete and concise summary of the VPC feature coverage?

Thanks,
Steve
-- 
Steve Ofsthun - Virtual Iron Software, Inc.

Re: [PATCH 0/4] (Refactored) VHD Support Addition

[Anthony Liguori]

Steve Ofsthun wrote:
>> QEMU already supports VHD (it calls it VPC) and supported it long before
>> Microsoft released any specs.  QEMU supported VMDK before VMware release
>> the specs for that too.
>>     
>
> This code was developed in the Xen 3.0.2 days when the integrated qemu had
> no vpc support.  Our original code was targeted at PV on HVM drivers using
> blocktap.  We needed to support the same format unaccelerated and so the
> qemu integration was also done.
>   

I suspect there are really two possible ways to go here:

1) Add generic support to mount blktap disks to QEMU via block-add in 
dom0 and then apply your stuff to blktap.

2) Just enhance the VPC in QEMU to cover anything it doesn't now.  That 
doesn't really address blktap but that's the right thing to do from the 
QEMU perspective.

Adding a second block-vhd to QEMU is really a bad idea from a 
maintainability perspective.  #1 is much better since it avoids that 
problem.

Regards,

Anthony Liguori

>> Is there a particular feature of VHD/VPC that your code adds that QEMU
>> didn't already support?
>>     
>
> Not having any experience with VPC, I couldn't tell you.  I can say that we
> tried to adhere to the Microsoft spec rather than reverse engineering the
> format from examples.  Perusing the VPC code, there are many references to
> unknown fields within the format.
>
> Is there a complete and concise summary of the VPC feature coverage?
>
> Thanks,
> Steve
>