[Qemu-devel] vmdk stream-optimised format

[Qemu-devel] vmdk stream-optimised format

[Alex Bligh]

I note some recent work on vmdk file support. Has anyone looked at
supporting vmdk streaming format as an /output/ file format (I
think we currently support it as an input format). This is what
you need to use to upload a VM to vCenter. Currently the route
is convert to raw then use:

  https://github.com/imcleod/VMDK-stream-converter

which is some Python from Ian McLeod <imcleod@redhat.com>
Life would be easier if qemu-img supported this directly.
It does not look fantastically difficult.

-- 
Alex Bligh

Re: [Qemu-devel] vmdk stream-optimised format

[Fam Zheng]

On Mon, 08/19 12:09, Alex Bligh wrote:
> I note some recent work on vmdk file support. Has anyone looked at
> supporting vmdk streaming format as an /output/ file format (I
> think we currently support it as an input format). This is what
> you need to use to upload a VM to vCenter. Currently the route
> is convert to raw then use:
> 
>  https://github.com/imcleod/VMDK-stream-converter
> 
> which is some Python from Ian McLeod <imcleod@redhat.com>
> Life would be easier if qemu-img supported this directly.
> It does not look fantastically difficult.
> 
> -- 
> Alex Bligh
> 

Yes, that would be something possible to do. Is it the only format to
upload a VM to vCenter now, shouldn't normal VMDK be supported as well?

Thanks,

Fam

Re: [Qemu-devel] vmdk stream-optimised format

[Alex Bligh]

On 19 Aug 2013, at 12:23, Fam Zheng wrote:

> Yes, that would be something possible to do. Is it the only format to
> upload a VM to vCenter now, shouldn't normal VMDK be supported as well?

Ian puts it better than I can:

> This format is only useful if you are attempting to import virtual machines
> into ESX using the vSphere SOAP API and HTTP POST uploads of image files.
> (In which case, it is required.)

IE it is the ONLY format you can use using the vSphere SOAP API, which is
the only way to automate image uploading. vCenter converts other formats
to this format.

Extensive testing with qemu in Q1 this year indicates that it is not
possible to use qemu-img convert to produce anything that will upload.
I went as far as hexdump at the time. I believe I tried compat6
and the various subformat options.

I believe we /read/ this format fine.

-- 
Alex Bligh

Re: [Qemu-devel] vmdk stream-optimised format

[Fam Zheng]

On Mon, 08/19 12:50, Alex Bligh wrote:
> 
> On 19 Aug 2013, at 12:23, Fam Zheng wrote:
> 
> > Yes, that would be something possible to do. Is it the only format to
> > upload a VM to vCenter now, shouldn't normal VMDK be supported as well?
> 
> Ian puts it better than I can:
> 
> > This format is only useful if you are attempting to import virtual machines
> > into ESX using the vSphere SOAP API and HTTP POST uploads of image files.
> > (In which case, it is required.)
> 
> IE it is the ONLY format you can use using the vSphere SOAP API, which is
> the only way to automate image uploading. vCenter converts other formats
> to this format.
> 
> Extensive testing with qemu in Q1 this year indicates that it is not
> possible to use qemu-img convert to produce anything that will upload.
> I went as far as hexdump at the time. I believe I tried compat6
> and the various subformat options.
> 
> I believe we /read/ this format fine.
> 
OK, thanks for explaination. That sounds a valid use case for
streamOptimized. However I am afraid QEMU and its users benefit not much
from this feature anyway, because it's moving a VM away to VMware, :)
that might be the reason it's not there yet, and I don't know about any
plan to do it in the near future.

But if someone sends patches for this, I think it is possible to get in.

Thanks,
Fam

Re: [Qemu-devel] vmdk stream-optimised format

[Alex Bligh]

On 20 Aug 2013, at 02:42, Fam Zheng wrote:

> OK, thanks for explaination. That sounds a valid use case for
> streamOptimized. However I am afraid QEMU and its users benefit not much
> from this feature anyway, because it's moving a VM away to VMware, :)
> that might be the reason it's not there yet, and I don't know about any
> plan to do it in the near future.

Well, given it is an open source project, the more interoperability
the better. Even if it just means users need not worry about lock
in to faster hypervisors ... Being more serious, qemu-img is
part of the project too.

> But if someone sends patches for this, I think it is possible to get in.

I guessed "send code" might be the answer :-)

What I'm not sure of is whether the streaming format has to be written
sequentially from as opposed to random writes. I believe the way
qemu-img convert works, one can't guarantee the writes are
sequential.

-- 
Alex Bligh

Re: [Qemu-devel] vmdk stream-optimised format

[Fam Zheng]

On Tue, 08/20 07:51, Alex Bligh wrote:
> 
> On 20 Aug 2013, at 02:42, Fam Zheng wrote:
> 
> > OK, thanks for explaination. That sounds a valid use case for
> > streamOptimized. However I am afraid QEMU and its users benefit not much
> > from this feature anyway, because it's moving a VM away to VMware, :)
> > that might be the reason it's not there yet, and I don't know about any
> > plan to do it in the near future.
> 
> Well, given it is an open source project, the more interoperability
> the better. Even if it just means users need not worry about lock
> in to faster hypervisors ... Being more serious, qemu-img is
> part of the project too.
> 
> > But if someone sends patches for this, I think it is possible to get in.
> 
> I guessed "send code" might be the answer :-)
> 
> What I'm not sure of is whether the streaming format has to be written
> sequentially from as opposed to random writes. I believe the way
> qemu-img convert works, one can't guarantee the writes are
> sequential.
> 
The order of sectors doesn't matter, but granularity should be aligned
to, as the data is compressed cluster by cluster. And no overwrite, of
course. The challenge may be that header comes at the end of file
(well, called footer), which is not decided at create time.

Thanks,
Fam

Re: [Qemu-devel] vmdk stream-optimised format

[Paolo Bonzini]

Il 20/08/2013 09:08, Fam Zheng ha scritto:
> On Tue, 08/20 07:51, Alex Bligh wrote:
>>
>> On 20 Aug 2013, at 02:42, Fam Zheng wrote:
>>
>>> OK, thanks for explaination. That sounds a valid use case for
>>> streamOptimized. However I am afraid QEMU and its users benefit not much
>>> from this feature anyway, because it's moving a VM away to VMware, :)
>>> that might be the reason it's not there yet, and I don't know about any
>>> plan to do it in the near future.
>>
>> Well, given it is an open source project, the more interoperability
>> the better. Even if it just means users need not worry about lock
>> in to faster hypervisors ... Being more serious, qemu-img is
>> part of the project too.
>>
>>> But if someone sends patches for this, I think it is possible to get in.
>>
>> I guessed "send code" might be the answer :-)
>>
>> What I'm not sure of is whether the streaming format has to be written
>> sequentially from as opposed to random writes. I believe the way
>> qemu-img convert works, one can't guarantee the writes are
>> sequential.
>>
> The order of sectors doesn't matter, but granularity should be aligned
> to, as the data is compressed cluster by cluster. And no overwrite, of
> course. The challenge may be that header comes at the end of file
> (well, called footer), which is not decided at create time.

It doesn't look too different from what "qemu-img convert -c" does,
except that you need to use the right "-o" incantation to specify the
format type.

Paolo

Re: [Qemu-devel] vmdk stream-optimised format

[Alex Bligh]

On 20 Aug 2013, at 11:53, Paolo Bonzini wrote:

> It doesn't look too different from what "qemu-img convert -c" does,
> except that you need to use the right "-o" incantation to specify the
> format type.

Extensive incanting did not produce a result that works. Possibly
inadequate haruspication on my part. I suspect it's only a minor
modification to support it.

-- 
Alex Bligh

Re: [Qemu-devel] vmdk stream-optimised format

[Paolo Bonzini]

Il 20/08/2013 15:37, Alex Bligh ha scritto:
> 
> On 20 Aug 2013, at 11:53, Paolo Bonzini wrote:
> 
>> It doesn't look too different from what "qemu-img convert -c" does,
>> except that you need to use the right "-o" incantation to specify the
>> format type.
> 
> Extensive incanting did not produce a result that works. Possibly
> inadequate haruspication on my part. I suspect it's only a minor
> modification to support it.

Yes, because vmdk does not implement bdrv_write_compressed.  Sorry for
leaving out the small "detail". :(

Paolo

Re: [Qemu-devel] vmdk stream-optimised format

[Alex Bligh]

On 20 Aug 2013, at 14:37, Paolo Bonzini wrote:

> Yes, because vmdk does not implement bdrv_write_compressed.  Sorry for
> leaving out the small "detail". :(

Would that be easy for me to fix?

-- 
Alex Bligh

Re: [Qemu-devel] vmdk stream-optimised format

[Stefan Hajnoczi]

On Tue, Aug 20, 2013 at 07:51:40AM +0100, Alex Bligh wrote:
> What I'm not sure of is whether the streaming format has to be written
> sequentially from as opposed to random writes. I believe the way
> qemu-img convert works, one can't guarantee the writes are
> sequential.

It should be possible to support stream-optimized VMDK writing in
qemu-img convert.

The file format supports non-sequential layout of data - each piece of
guest data also has a logical block address with it.  You can write out
the grain table/directory at the end of the image file once you know how
data is layed out.

The VMDK spec is not great so I guess you'll also need a VMware setup to
verify your code changes.

Stefan