[Qemu-devel] KVM call agenda for Jan 25

[Qemu-devel] KVM call agenda for Jan 25

[Chris Wright]

Please send in any agenda items you are interested in covering.

thanks,
-chris

[Qemu-devel] Re: KVM call agenda for Jan 25

[Anthony Liguori]

On 01/24/2011 07:25 AM, Chris Wright wrote:
> Please send in any agenda items you are interested in covering.
>    

- coroutines for the block layer
- glib everywhere

Regards,

Anthony Liguori

> thanks,
> -chris
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>    

Re: [Qemu-devel] Re: KVM call agenda for Jan 25

[Luiz Capitulino]

On Mon, 24 Jan 2011 16:06:34 -0600
Anthony Liguori <anthony@codemonkey.ws> wrote:

> On 01/24/2011 07:25 AM, Chris Wright wrote:
> > Please send in any agenda items you are interested in covering.
> >    
> 
> - coroutines for the block layer
> - glib everywhere

- Let's start planning our next release in advance, here's a simple example:

  http://wiki.qemu.org/Planning/0.15-example

> 
> Regards,
> 
> Anthony Liguori
> 
> > thanks,
> > -chris
> > --
> > To unsubscribe from this list: send the line "unsubscribe kvm" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >    
> 
> 

Re: [Qemu-devel] Re: KVM call agenda for Jan 25

[Luiz Capitulino]

On Tue, 25 Jan 2011 11:57:27 -0200
Luiz Capitulino <lcapitulino@redhat.com> wrote:

> On Mon, 24 Jan 2011 16:06:34 -0600
> Anthony Liguori <anthony@codemonkey.ws> wrote:
> 
> > On 01/24/2011 07:25 AM, Chris Wright wrote:
> > > Please send in any agenda items you are interested in covering.
> > >    
> > 
> > - coroutines for the block layer
> > - glib everywhere
> 
> - Let's start planning our next release in advance, here's a simple example:
> 
>   http://wiki.qemu.org/Planning/0.15-example

Forgot:

 - Google summer of code 2011 is on, are we interested? (note: I just saw the
   news, I don't have any information yet)
> 
> > 
> > Regards,
> > 
> > Anthony Liguori
> > 
> > > thanks,
> > > -chris
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe kvm" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > >    
> > 
> > 
> 
> 

Re: [Qemu-devel] Re: KVM call agenda for Jan 25

[Stefan Hajnoczi]

On Tue, Jan 25, 2011 at 2:02 PM, Luiz Capitulino <lcapitulino@redhat.com> wrote:
>  - Google summer of code 2011 is on, are we interested? (note: I just saw the
>   news, I don't have any information yet)

http://www.google-melange.com/document/show/gsoc_program/google/gsoc2011/timeline

I'd like to see an in-place QCOW2 <-> QED image converter with tests.
I'm interested in mentoring this year.

Stefan

Re: [Qemu-devel] Re: KVM call agenda for Jan 25

[Dushyant Bansal]

> http://www.google-melange.com/document/show/gsoc_program/google/gsoc2011/timeline
>
> I'd like to see an in-place QCOW2<->  QED image converter with tests.
> I'm interested in mentoring this year.
>
> Stefan
>
>    
Hi,
I am interested in working on this project as part of gsoc. I have prior 
experience working with qemu.
  After reading some basic information on qcow2 and qed from:
1. http://lists.nongnu.org/archive/html/qemu-devel/2010-09/msg00310.html
2. http://people.gnome.org/~markmc/qcow-image-format.html

I am now planning to start reading their source code. What do you 
suggest? Are there any other specific things that I should read about?

P.S. I just found out another new qemu image format 
FVD(http://wiki.qemu.org/Features/FVD/Compare). I will try to read it too.

Regards,
Dushyant

Re: [Qemu-devel] Re: KVM call agenda for Jan 25

[Stefan Hajnoczi]

On Sat, Jan 29, 2011 at 10:50 AM, Dushyant Bansal
<cs5070214@cse.iitd.ac.in> wrote:
> I am interested in working on this project as part of gsoc. I have prior
> experience working with qemu.

Great.  I have put up a more detailed description on the QEMU GSoC 2011 page:
http://wiki.qemu.org/Google_Summer_of_Code_2011#QCOW2_.3C-.3E_QED_image_converter

Please be aware that organizations have not yet been selected by
Google and that GSoC planning is still in the early stages.  I want to
make sure that your expectations and the amount of upfront effort you
invest at this early stage reflect that so you're not disappointed if
things don't come together.

> I am now planning to start reading their source code. What do you suggest?
> Are there any other specific things that I should read about?

btrfs has an ext3 conversion mode that could be inspiring:
https://btrfs.wiki.kernel.org/index.php/Conversion_from_Ext3

You could tackle open bugs to familiarize yourself with the codebase:
http://goo.gl/DMxOD

Or this: which is faster, qemu-img convert -f <format> -O <format>
<src-image> <dst-image> or cp <src-image> <dst-image>?  What about for
raw images, shouldn't that be the same speed as cp(1)?  Poke around
the source code, profile it, understand what it's doing, think about
ways to improve it.  No need to do everything, just doing part of this
will give you background on QEMU's block layer.

Contributing patches is a good way get up to speed and show your
skills.  If time doesn't permit that, just think about the problem and
how you intend to solve it, and feel free to bounce ideas off me.

Stefan

Re: [Qemu-devel] Re: KVM call agenda for Jan 25

[Dushyant Bansal]

On Saturday 29 January 2011 04:20 PM, Dushyant Bansal wrote:
> Or this: which is faster, qemu-img convert -f<format>  -O<format>
> <src-image>  <dst-image>  or cp<src-image>  <dst-image>?  What about for
> raw images, shouldn't that be the same speed as cp(1)?  Poke around
> the source code, profile it, understand what it's doing, think about
> ways to improve it.  No need to do everything, just doing part of this
> will give you background on QEMU's block layer.
>
> Contributing patches is a good way get up to speed and show your
> skills.  If time doesn't permit that, just think about the problem and
> how you intend to solve it, and feel free to bounce ideas off me.
>    
I explored 'qemu-img create and convert' and got a basic understanding 
of how they work.

cp faster than qemu-img convert

For raw->raw
In cp, it just copies all the disk blocks actually occupied by the file.
And, with qemu-img convert, it checks all the sectors and copy those, 
which contains atleast one non-NUL byte.
The better performance of cp over qemu-img convert is the result of 
overhead of this checking.

I tried a few variations:
1. just copy all the sectors without checking
So, actual size becomes equal to virtual size.
2. In is_allocated_sectors,out of n sectors, if any sector has a non-NUL 
byte then break and copy all n sectors.
As expected, resultant raw image was quite large in size.

Looking forward to your comments.

Thanks,
Dushyant

Re: [Qemu-devel] Re: KVM call agenda for Jan 25

[Stefan Hajnoczi]

On Fri, Feb 25, 2011 at 5:42 PM, Dushyant Bansal
<cs5070214@cse.iitd.ac.in> wrote:
> On Saturday 29 January 2011 04:20 PM, Dushyant Bansal wrote:
>>
>> Or this: which is faster, qemu-img convert -f<format>  -O<format>
>> <src-image>  <dst-image>  or cp<src-image>  <dst-image>?  What about for
>> raw images, shouldn't that be the same speed as cp(1)?  Poke around
>> the source code, profile it, understand what it's doing, think about
>> ways to improve it.  No need to do everything, just doing part of this
>> will give you background on QEMU's block layer.
>>
>> Contributing patches is a good way get up to speed and show your
>> skills.  If time doesn't permit that, just think about the problem and
>> how you intend to solve it, and feel free to bounce ideas off me.
>>
>
> I explored 'qemu-img create and convert' and got a basic understanding of
> how they work.

Great, it's good to hear from you.

> cp faster than qemu-img convert

Yes, I've experienced that too.

> For raw->raw
> In cp, it just copies all the disk blocks actually occupied by the file.
> And, with qemu-img convert, it checks all the sectors and copy those, which
> contains atleast one non-NUL byte.
> The better performance of cp over qemu-img convert is the result of overhead
> of this checking.

How did you find out what cp(1) and qemu-img do?

How does cp(1) know which disk blocks are actually occupied?

> I tried a few variations:
> 1. just copy all the sectors without checking
> So, actual size becomes equal to virtual size.

Did that make qemu-img faster for the image file you tested?

> 2. In is_allocated_sectors,out of n sectors, if any sector has a non-NUL
> byte then break and copy all n sectors.
> As expected, resultant raw image was quite large in size.

This is kind of like what cp(1) does, except it limits n to 32 KB
maximum at a time.  Maybe if you add this tweak they will show similar
performance.  The drawback is that the output image is larger than
with the current approach.

Stefan

Re: [Qemu-devel] Re: KVM call agenda for Jan 25

[Dushyant Bansal]

>> In cp, it just copies all the disk blocks actually occupied by the file.
>> And, with qemu-img convert, it checks all the sectors and copy those, which
>> contains atleast one non-NUL byte.
>> The better performance of cp over qemu-img convert is the result of overhead
>> of this checking.
>>      
> How did you find out what cp(1) and qemu-img do?
> How does cp(1) know which disk blocks are actually occupied?
>    
I have looked into their source code.
  Yes, that was not correct. cp also checks each block and copy those 
which contains non-null byte.
qemu-img does the same thing with sectors.
>> I tried a few variations:
>> 1. just copy all the sectors without checking
>> So, actual size becomes equal to virtual size.
>>      
> Did that make qemu-img faster for the image file you tested?
>    
No, in fact it becomes slower. I guess it is due to the increase in disk 
write.
>> 2. In is_allocated_sectors,out of n sectors, if any sector has a non-NUL
>> byte then break and copy all n sectors.
>> As expected, resultant raw image was quite large in size.
>>      
> This is kind of like what cp(1) does, except it limits n to 32 KB
> maximum at a time.  Maybe if you add this tweak they will show similar
> performance.  The drawback is that the output image is larger than
> with the current approach.
>
> Stefan
>
>    
Disk block size is usually 512 bytes and in qemu-img, sector size is 
also 512B. And, this change would  copy n sectors even if only one of 
them actually contains data (while cp checks and copies one 
block(=sector) at a time). Therefore, it will end up writing more data 
than cp.

virtual size: 10G (10737418240 bytes)
disk size: 569M

convert-> original
time    0m52.522s

convert-> modified (resultant disk size: 5.3G)
time    2m12.744s

cp
time    0m51.724s (same disk size)
---------------------------------------------------------------------------
virtual size: 10G (10737418240 bytes)
disk size: 3.6G

convert-> original
time    1m52.249s

convert-> modified (resultant disk size: 7.1G)
time    3m2.891s

cp
time    1m55.320s (same disk size)
---------------------------------------------------------------------------
In these results, we can see that resultant disk size has increased.

Thanks,
Dushyant

Re: [Qemu-devel] Re: KVM call agenda for Jan 25

[Stefan Hajnoczi]

On Sat, Feb 26, 2011 at 9:50 PM, Dushyant Bansal
<cs5070214@cse.iitd.ac.in> wrote:
> Disk block size is usually 512 bytes and in qemu-img, sector size is also
> 512B. And, this change would  copy n sectors even if only one of them
> actually contains data (while cp checks and copies one block(=sector) at a
> time). Therefore, it will end up writing more data than cp.

cp(1) from GNU coreutils 8.5 works in units of 32 KB on my system.  It
reads 32 KB and checks for all zeroes.  If there are all zeroes, it
seeks ahead 32 KB in the output file.  Otherwise it writes the entire
32 KB.

You can check what cp(1) is doing using strace(1).

> virtual size: 10G (10737418240 bytes)
> disk size: 569M
>
> convert-> original
> time    0m52.522s
>
> convert-> modified (resultant disk size: 5.3G)
> time    2m12.744s
>
> cp
> time    0m51.724s (same disk size)
> ---------------------------------------------------------------------------
> virtual size: 10G (10737418240 bytes)
> disk size: 3.6G
>
> convert-> original
> time    1m52.249s
>
> convert-> modified (resultant disk size: 7.1G)
> time    3m2.891s
>
> cp
> time    1m55.320s (same disk size)
> ---------------------------------------------------------------------------
> In these results, we can see that resultant disk size has increased.

If I'm reading this correctly then qemu-img convert is within a few
seconds of cp(1) for you?

Stefan

Re: [Qemu-devel] Re: KVM call agenda for Jan 25

[Markus Armbruster]

Stefan Hajnoczi <stefanha@gmail.com> writes:

> On Sat, Feb 26, 2011 at 9:50 PM, Dushyant Bansal
> <cs5070214@cse.iitd.ac.in> wrote:
>> Disk block size is usually 512 bytes and in qemu-img, sector size is also
>> 512B. And, this change would  copy n sectors even if only one of them
>> actually contains data (while cp checks and copies one block(=sector) at a
>> time). Therefore, it will end up writing more data than cp.
>
> cp(1) from GNU coreutils 8.5 works in units of 32 KB on my system.  It
> reads 32 KB and checks for all zeroes.  If there are all zeroes, it
> seeks ahead 32 KB in the output file.  Otherwise it writes the entire
> 32 KB.

The latest version[*] of cp does better:

    cp now copies sparse files efficiently on file systems with FIEMAP
    support (ext4, btrfs, xfs, ocfs2).  Before, it had to read 2^20 bytes
    when copying a 1MiB sparse file.  Now, it copies bytes only for the
    non-sparse sections of a file.  Similarly, to induce a hole in the
    output file, it had to detect a long sequence of zero bytes.  Now,
    it knows precisely where each hole in an input file is, and can
    reproduce them efficiently in the output file.  mv also benefits
    when it resorts to copying, e.g., between file systems.

But beware of kernel bugs[**].

> You can check what cp(1) is doing using strace(1).

Often enlightening :)

[...]

[*] http://lists.gnu.org/archive/html/coreutils-announce/2011-02/msg00000.html

[**] http://lwn.net/Articles/429345/
Subscribera-only, should become public soon.  In the meantime, try
http://lwn.net/Articles/429349/

Re: [Qemu-devel] Re: KVM call agenda for Jan 25

[Dushyant Bansal]

[-- Attachment #1: Type: text/plain, Size: 1845 bytes --]

On Sunday 27 February 2011 04:19 PM, Stefan Hajnoczi wrote:
> On Sat, Feb 26, 2011 at 9:50 PM, Dushyant Bansal
> <cs5070214@cse.iitd.ac.in>  wrote:
>    
>> Disk block size is usually 512 bytes and in qemu-img, sector size is also
>> 512B. And, this change would  copy n sectors even if only one of them
>> actually contains data (while cp checks and copies one block(=sector) at a
>> time). Therefore, it will end up writing more data than cp.
>>      
> cp(1) from GNU coreutils 8.5 works in units of 32 KB on my system.  It
> reads 32 KB and checks for all zeroes.  If there are all zeroes, it
> seeks ahead 32 KB in the output file.  Otherwise it writes the entire
> 32 KB.
>
> You can check what cp(1) is doing using strace(1).
>    
yes, you are right. I was reading from a much older coreutils source.
>    
>> virtual size: 10G (10737418240 bytes)
>> disk size: 569M
>>
>> convert->  original
>> time    0m52.522s
>>
>> convert->  modified (resultant disk size: 5.3G)
>> time    2m12.744s
>>
>> cp
>> time    0m51.724s (same disk size)
>> ---------------------------------------------------------------------------
>> virtual size: 10G (10737418240 bytes)
>> disk size: 3.6G
>>
>> convert->  original
>> time    1m52.249s
>>
>> convert->  modified (resultant disk size: 7.1G)
>> time    3m2.891s
>>
>> cp
>> time    1m55.320s (same disk size)
>> ---------------------------------------------------------------------------
>> In these results, we can see that resultant disk size has increased.
>>      
> If I'm reading this correctly then qemu-img convert is within a few
> seconds of cp(1) for you?
>    
I have collected and attached some more time results for different 
operations on a 2.2G image.
qemu-img convert is close to cp.

qemu(modified):
IO_BUF_SIZE = (2 * 1024 )
if any sector is non-null, write all sectors

--
Dushyant

[-- Attachment #2: results.xls --]
[-- Type: application/vnd.ms-excel, Size: 9728 bytes --]

Re: [Qemu-devel] Re: KVM call agenda for Jan 25

[Stefan Hajnoczi]

On Mon, Feb 28, 2011 at 8:41 PM, Dushyant Bansal
<cs5070214@cse.iitd.ac.in> wrote:
> On Sunday 27 February 2011 04:19 PM, Stefan Hajnoczi wrote:
>>
>> On Sat, Feb 26, 2011 at 9:50 PM, Dushyant Bansal
>> <cs5070214@cse.iitd.ac.in>  wrote:
>>>
>>> virtual size: 10G (10737418240 bytes)
>>> disk size: 569M
>>>
>>> convert->  original
>>> time    0m52.522s
>>>
>>> convert->  modified (resultant disk size: 5.3G)
>>> time    2m12.744s
>>>
>>> cp
>>> time    0m51.724s (same disk size)
>>>
>>> ---------------------------------------------------------------------------
>>> virtual size: 10G (10737418240 bytes)
>>> disk size: 3.6G
>>>
>>> convert->  original
>>> time    1m52.249s
>>>
>>> convert->  modified (resultant disk size: 7.1G)
>>> time    3m2.891s
>>>
>>> cp
>>> time    1m55.320s (same disk size)
>>>
>>> ---------------------------------------------------------------------------
>>> In these results, we can see that resultant disk size has increased.
>>>
>>
>> If I'm reading this correctly then qemu-img convert is within a few
>> seconds of cp(1) for you?
>>
>
> I have collected and attached some more time results for different
> operations on a 2.2G image.
> qemu-img convert is close to cp.
>
> qemu(modified):
> IO_BUF_SIZE = (2 * 1024 )
> if any sector is non-null, write all sectors

Nice that qemu-img convert isn't that far out by default on raw :).

About Google Summer of Code, I have posted my take on applying and
want to share that with you and qemu-devel:

http://blog.vmsplice.net/2011/03/advice-for-students-applying-to-google.html

Stefan

Re: [Qemu-devel] Re: KVM call agenda for Jan 25

[Dushyant Bansal]

>
> Nice that qemu-img convert isn't that far out by default on raw :).
>
> About Google Summer of Code, I have posted my take on applying and
> want to share that with you and qemu-devel:
>
> http://blog.vmsplice.net/2011/03/advice-for-students-applying-to-google.html
>    
Thanks for sharing your experiences.

After reading about qcow2 and qed and how they organize data (thanks to 
the newly added qcow2 doc and discussions on the mailing list), this is 
what I understand.

So, the main difference between qed and qcow2 is the absence of 
reference count structure in qed(means less meta data).
It improves performance due to:
1. For write operations, less or no metadata to update.
2. Data write and metadata write can be in any order

This also means these features are no longer supported:
1. Internal snapshots,
2. CPU/device state snapshots,
3. Compression,
4. Encryption

Now, coming to qed<-->qcow2 conversion, I want to clarify some things.

1. header_size: variable in qed, equals to cluster size in qcow2:
When will it be larger than 1 cluster in qed? So, what will happen to 
that extra data on qed->qcow2 conversion.

2. L2 table size: equals to L1 table size in qed, equals to cluster size 
in qcow2:
we need to take it into account during conversion.

3. refcount table:
qcow2->qed:we do not keep refcount structure
qed->qcow2: initialize refcount structure

So, a qcow2->qed->qcow2 conversion means if earlier, qcow2 image was 
using additional features{1-4}, all information related to that will be 
lost.

What do you think? Please correct me if I am making some incorrect 
statement.

Thanks,
Dushyant

Re: [Qemu-devel] Re: KVM call agenda for Jan 25

[Kevin Wolf]

Am 14.03.2011 16:13, schrieb Dushyant Bansal:
>>
>> Nice that qemu-img convert isn't that far out by default on raw :).
>>
>> About Google Summer of Code, I have posted my take on applying and
>> want to share that with you and qemu-devel:
>>
>> http://blog.vmsplice.net/2011/03/advice-for-students-applying-to-google.html
>>    
> Thanks for sharing your experiences.
> 
> After reading about qcow2 and qed and how they organize data (thanks to 
> the newly added qcow2 doc and discussions on the mailing list), this is 
> what I understand.
> 
> So, the main difference between qed and qcow2 is the absence of 
> reference count structure in qed(means less meta data).
> It improves performance due to:
> 1. For write operations, less or no metadata to update.
> 2. Data write and metadata write can be in any order
> 
> This also means these features are no longer supported:
> 1. Internal snapshots,
> 2. CPU/device state snapshots,
> 3. Compression,
> 4. Encryption
> 
> Now, coming to qed<-->qcow2 conversion, I want to clarify some things.
> 
> 1. header_size: variable in qed, equals to cluster size in qcow2:
> When will it be larger than 1 cluster in qed? So, what will happen to 
> that extra data on qed->qcow2 conversion.

If you have an feature that is used in the original image, but cannot be
represented in the new format, I think you should just get an error.

> 2. L2 table size: equals to L1 table size in qed, equals to cluster size 
> in qcow2:
> we need to take it into account during conversion.

Right. I think we'll have to rewrite all of the metadata.

I wonder if we can manage to have a nice block driver interface for
in-place image conversions so that we don't only get a qed<->qcow2
converter, but also can implement the interface in e.g. VMDK and get
VMDK<->qcow2 and VMDK<->qed as well.

> 3. refcount table:
> qcow2->qed:we do not keep refcount structure
> qed->qcow2: initialize refcount structure

Yes, refcounts can be rebuilt after the mapping has been converted.

> So, a qcow2->qed->qcow2 conversion means if earlier, qcow2 image was 
> using additional features{1-4}, all information related to that will be 
> lost.

We shouldn't lose anything but just abort if a conversion isn't
possible. The user can still use qemu-img convert for the more
complicated cases, for example for removing encryption or compression.

Kevin

Re: [Qemu-devel] Re: KVM call agenda for Jan 25

[Dushyant Bansal]

>> 1. header_size: variable in qed, equals to cluster size in qcow2:
>> When will it be larger than 1 cluster in qed? So, what will happen to
>> that extra data on qed->qcow2 conversion.
>>      
> If you have an feature that is used in the original image, but cannot be
> represented in the new format, I think you should just get an error.
>
>    
>> 2. L2 table size: equals to L1 table size in qed, equals to cluster size
>> in qcow2:
>> we need to take it into account during conversion.
>>      
> Right. I think we'll have to rewrite all of the metadata.
>
> I wonder if we can manage to have a nice block driver interface for
> in-place image conversions so that we don't only get a qed<->qcow2
> converter, but also can implement the interface in e.g. VMDK and get
> VMDK<->qcow2 and VMDK<->qed as well.
>    
>> 3. refcount table:
>> qcow2->qed:we do not keep refcount structure
>> qed->qcow2: initialize refcount structure
>>      
> Yes, refcounts can be rebuilt after the mapping has been converted.
>
>    
>> So, a qcow2->qed->qcow2 conversion means if earlier, qcow2 image was
>> using additional features{1-4}, all information related to that will be
>> lost.
>>      
> We shouldn't lose anything but just abort if a conversion isn't
> possible. The user can still use qemu-img convert for the more
> complicated cases, for example for removing encryption or compression.
>
>    
Thanks for clearing up those issues.

--
Dushyant

Re: [Qemu-devel] Re: KVM call agenda for Jan 25

[Stefan Hajnoczi]

On Tue, Mar 15, 2011 at 10:27 AM, Kevin Wolf <kwolf@redhat.com> wrote:
> Am 14.03.2011 16:13, schrieb Dushyant Bansal:
>>>
>>> Nice that qemu-img convert isn't that far out by default on raw :).
>>>
>>> About Google Summer of Code, I have posted my take on applying and
>>> want to share that with you and qemu-devel:
>>>
>>> http://blog.vmsplice.net/2011/03/advice-for-students-applying-to-google.html
>>>
>> Thanks for sharing your experiences.
>>
>> After reading about qcow2 and qed and how they organize data (thanks to
>> the newly added qcow2 doc and discussions on the mailing list), this is
>> what I understand.
>>
>> So, the main difference between qed and qcow2 is the absence of
>> reference count structure in qed(means less meta data).
>> It improves performance due to:
>> 1. For write operations, less or no metadata to update.
>> 2. Data write and metadata write can be in any order
>>
>> This also means these features are no longer supported:
>> 1. Internal snapshots,
>> 2. CPU/device state snapshots,
>> 3. Compression,
>> 4. Encryption
>>
>> Now, coming to qed<-->qcow2 conversion, I want to clarify some things.
>>
>> 1. header_size: variable in qed, equals to cluster size in qcow2:
>> When will it be larger than 1 cluster in qed? So, what will happen to
>> that extra data on qed->qcow2 conversion.
>
> If you have an feature that is used in the original image, but cannot be
> represented in the new format, I think you should just get an error.
>
>> 2. L2 table size: equals to L1 table size in qed, equals to cluster size
>> in qcow2:
>> we need to take it into account during conversion.
>
> Right. I think we'll have to rewrite all of the metadata.
>
> I wonder if we can manage to have a nice block driver interface for
> in-place image conversions so that we don't only get a qed<->qcow2
> converter, but also can implement the interface in e.g. VMDK and get
> VMDK<->qcow2 and VMDK<->qed as well.

I think this will be tricky but would be very interested if someone
has ideas.  Code-wise an in-place converter probably needs access to
both format's on-disk structures or internal functions.  I don't think
abstracting this is easy because the more you abstract the less
control you have over keeping things in-place and cleanly putting the
new structures in place.

On the other hand, I think the starting point for a generic in-place
converter would be a loop that does something like bdrv_is_allocated()
but translates the guest position in the block device into an offset
into the image file.  That, together with some sort of free map or
space allocation bitmap would allow a generic approach to figuring out
the data mapping and which parts of the file can be safely used.

The big benefit doing an interface for in-place conversion is that we
can support n-to-n conversions with at most n converter code rather
than having to code n * n - n different in-place converters.

>> 3. refcount table:
>> qcow2->qed:we do not keep refcount structure
>> qed->qcow2: initialize refcount structure
>
> Yes, refcounts can be rebuilt after the mapping has been converted.
>
>> So, a qcow2->qed->qcow2 conversion means if earlier, qcow2 image was
>> using additional features{1-4}, all information related to that will be
>> lost.
>
> We shouldn't lose anything but just abort if a conversion isn't
> possible. The user can still use qemu-img convert for the more
> complicated cases, for example for removing encryption or compression.

I agree with Kevin on the points made.

Stefan

Re: [Qemu-devel] Re: KVM call agenda for Jan 25

[Kevin Wolf]

Am 16.03.2011 18:47, schrieb Stefan Hajnoczi:
> On Tue, Mar 15, 2011 at 10:27 AM, Kevin Wolf <kwolf@redhat.com> wrote:
>> Am 14.03.2011 16:13, schrieb Dushyant Bansal:
>>>>
>>>> Nice that qemu-img convert isn't that far out by default on raw :).
>>>>
>>>> About Google Summer of Code, I have posted my take on applying and
>>>> want to share that with you and qemu-devel:
>>>>
>>>> http://blog.vmsplice.net/2011/03/advice-for-students-applying-to-google.html
>>>>
>>> Thanks for sharing your experiences.
>>>
>>> After reading about qcow2 and qed and how they organize data (thanks to
>>> the newly added qcow2 doc and discussions on the mailing list), this is
>>> what I understand.
>>>
>>> So, the main difference between qed and qcow2 is the absence of
>>> reference count structure in qed(means less meta data).
>>> It improves performance due to:
>>> 1. For write operations, less or no metadata to update.
>>> 2. Data write and metadata write can be in any order
>>>
>>> This also means these features are no longer supported:
>>> 1. Internal snapshots,
>>> 2. CPU/device state snapshots,
>>> 3. Compression,
>>> 4. Encryption
>>>
>>> Now, coming to qed<-->qcow2 conversion, I want to clarify some things.
>>>
>>> 1. header_size: variable in qed, equals to cluster size in qcow2:
>>> When will it be larger than 1 cluster in qed? So, what will happen to
>>> that extra data on qed->qcow2 conversion.
>>
>> If you have an feature that is used in the original image, but cannot be
>> represented in the new format, I think you should just get an error.
>>
>>> 2. L2 table size: equals to L1 table size in qed, equals to cluster size
>>> in qcow2:
>>> we need to take it into account during conversion.
>>
>> Right. I think we'll have to rewrite all of the metadata.
>>
>> I wonder if we can manage to have a nice block driver interface for
>> in-place image conversions so that we don't only get a qed<->qcow2
>> converter, but also can implement the interface in e.g. VMDK and get
>> VMDK<->qcow2 and VMDK<->qed as well.
> 
> I think this will be tricky but would be very interested if someone
> has ideas.  Code-wise an in-place converter probably needs access to
> both format's on-disk structures or internal functions.  I don't think
> abstracting this is easy because the more you abstract the less
> control you have over keeping things in-place and cleanly putting the
> new structures in place.

Well, if it was easy, I would have suggested a specific way of doing it. ;-)

But it would be a really cool thing to have, and I think it's more fun
for a GSoC participant to actually think about a hard problem than just
doing mostly mechanical work.

> On the other hand, I think the starting point for a generic in-place
> converter would be a loop that does something like bdrv_is_allocated()
> but translates the guest position in the block device into an offset
> into the image file.  That, together with some sort of free map or
> space allocation bitmap would allow a generic approach to figuring out
> the data mapping and which parts of the file can be safely used.

We can discuss the detailed API later, but I agree that the critical
thing to convert is the mapping.

You would probably open the file with the source format driver read-only
and with the destination driver read-write. For qcow2 you would start
with writing a refcount table that marks the whole file as used, other
formats use the file size anyway. Then you can start creating L1 and L2
tables and copy the mapping over. Once this is done, you do an fsck to
free the metadata of the old format.

One thing that may become tricky is the image header which both drivers
may want to use and which is fixed at offset 0. And of course, you must
make sure that the image is safe at any point if the converter crashes.

> The big benefit doing an interface for in-place conversion is that we
> can support n-to-n conversions with at most n converter code rather
> than having to code n * n - n different in-place converters.

Yes, this was the idea.

Kevin

Re: [Qemu-devel] Re: KVM call agenda for Jan 25

[Dushyant Bansal]

>> On the other hand, I think the starting point for a generic in-place
>> converter would be a loop that does something like bdrv_is_allocated()
>> but translates the guest position in the block device into an offset
>> into the image file.  That, together with some sort of free map or
>> space allocation bitmap would allow a generic approach to figuring out
>> the data mapping and which parts of the file can be safely used.
>>      
> We can discuss the detailed API later, but I agree that the critical
> thing to convert is the mapping.
>
> You would probably open the file with the source format driver read-only
> and with the destination driver read-write. For qcow2 you would start
> with writing a refcount table that marks the whole file as used, other
> formats use the file size anyway. Then you can start creating L1 and L2
> tables and copy the mapping over. Once this is done, you do an fsck to
> free the metadata of the old format.
>
> One thing that may become tricky is the image header which both drivers
> may want to use and which is fixed at offset 0. And of course, you must
> make sure that the image is safe at any point if the converter crashes.
>    
For image header issue, this is the approach that comes to mind.
Lets say, destination format is qcow2.
BDRVQcowState is responsible for header fields inside BlockDriverState. 
We need qcow2 image header to initiliaze all the fields of 
BDRVQcowState, which is done by bdrv_open(qcow2_open()).

So initially, for the qcow2 driver, we do not copy the qcow2 image 
header (we keep the source header). We can then manually set fields of 
BDRVQcowState with the desired header fields.
And after all other metadata has been copied for the qcow2 format, we 
can replace the source image header with the qcow2 header.


Thanks,
Dushyant

Re: [Qemu-devel] Re: KVM call agenda for Jan 25

[Kevin Wolf]

Am 26.03.2011 22:56, schrieb Dushyant Bansal:
> 
>>> On the other hand, I think the starting point for a generic in-place
>>> converter would be a loop that does something like bdrv_is_allocated()
>>> but translates the guest position in the block device into an offset
>>> into the image file.  That, together with some sort of free map or
>>> space allocation bitmap would allow a generic approach to figuring out
>>> the data mapping and which parts of the file can be safely used.
>>>      
>> We can discuss the detailed API later, but I agree that the critical
>> thing to convert is the mapping.
>>
>> You would probably open the file with the source format driver read-only
>> and with the destination driver read-write. For qcow2 you would start
>> with writing a refcount table that marks the whole file as used, other
>> formats use the file size anyway. Then you can start creating L1 and L2
>> tables and copy the mapping over. Once this is done, you do an fsck to
>> free the metadata of the old format.
>>
>> One thing that may become tricky is the image header which both drivers
>> may want to use and which is fixed at offset 0. And of course, you must
>> make sure that the image is safe at any point if the converter crashes.
>>    
> For image header issue, this is the approach that comes to mind.
> Lets say, destination format is qcow2.
> BDRVQcowState is responsible for header fields inside BlockDriverState. 
> We need qcow2 image header to initiliaze all the fields of 
> BDRVQcowState, which is done by bdrv_open(qcow2_open()).
> 
> So initially, for the qcow2 driver, we do not copy the qcow2 image 
> header (we keep the source header). We can then manually set fields of 
> BDRVQcowState with the desired header fields.
> And after all other metadata has been copied for the qcow2 format, we 
> can replace the source image header with the qcow2 header.

The question is if you can make sure that while you convert the image,
qcow2 will never try to modify the header. It will definitely do so if
you have to increase the L1 or refcount table, for example.

In the end, we may have to use tricks like using a protocol which keeps
the first cluster in a ramdisk until the conversion is complete.

Kevin

Re: [Qemu-devel] Re: KVM call agenda for Jan 25

[Aurelien Jarno]

Luiz Capitulino a écrit :
> On Mon, 24 Jan 2011 16:06:34 -0600
> Anthony Liguori <anthony@codemonkey.ws> wrote:
> 
>> On 01/24/2011 07:25 AM, Chris Wright wrote:
>>> Please send in any agenda items you are interested in covering.
>>>    
>> - coroutines for the block layer
>> - glib everywhere
> 
> - Let's start planning our next release in advance, here's a simple example:
> 
>   http://wiki.qemu.org/Planning/0.15-example
> 

What about planning the 0.14 release first? From what I see on the
mailing list, we are more in a development phase than in a bug fix phase
before a release.

-- 
Aurelien Jarno                          GPG: 1024D/F1BCDB73
aurelien@aurel32.net                 http://www.aurel32.net

Re: [Qemu-devel] Re: KVM call agenda for Jan 25

[Anthony Liguori]

On 01/25/2011 08:11 AM, Aurelien Jarno wrote:
> Luiz Capitulino a écrit :
>    
>> On Mon, 24 Jan 2011 16:06:34 -0600
>> Anthony Liguori<anthony@codemonkey.ws>  wrote:
>>
>>      
>>> On 01/24/2011 07:25 AM, Chris Wright wrote:
>>>        
>>>> Please send in any agenda items you are interested in covering.
>>>>
>>>>          
>>> - coroutines for the block layer
>>> - glib everywhere
>>>        
>> - Let's start planning our next release in advance, here's a simple example:
>>
>>    http://wiki.qemu.org/Planning/0.15-example
>>
>>      
> What about planning the 0.14 release first? From what I see on the
> mailing list, we are more in a development phase than in a bug fix phase
> before a release.
>    

Yeah, lesson learned is releasing before a holiday is probably a bad idea.

What would be a reasonable window to get us in bug fixing mode to do a 
release?  How would a 2/1 stable-0.14 branch sound?

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: KVM call agenda for Jan 25

[Kevin Wolf]

Am 25.01.2011 15:11, schrieb Aurelien Jarno:
> Luiz Capitulino a écrit :
>> On Mon, 24 Jan 2011 16:06:34 -0600
>> Anthony Liguori <anthony@codemonkey.ws> wrote:
>>
>>> On 01/24/2011 07:25 AM, Chris Wright wrote:
>>>> Please send in any agenda items you are interested in covering.
>>>>    
>>> - coroutines for the block layer
>>> - glib everywhere
>>
>> - Let's start planning our next release in advance, here's a simple example:
>>
>>   http://wiki.qemu.org/Planning/0.15-example
>>
> 
> What about planning the 0.14 release first? From what I see on the
> mailing list, we are more in a development phase than in a bug fix phase
> before a release.

If we want to get into a bug fix phase, all we need to have is a
stable-0.14 branch. It's not like creating a branch is a lot of work, it
just didn't happen on the announced date.

And I'm almost sure that when it happens, it's going to come as a
surprise once again...

Kevin

Re: [Qemu-devel] Re: KVM call agenda for Jan 25

[Aurelien Jarno]

On Tue, Jan 25, 2011 at 03:42:04PM +0100, Kevin Wolf wrote:
> Am 25.01.2011 15:11, schrieb Aurelien Jarno:
> > Luiz Capitulino a écrit :
> >> On Mon, 24 Jan 2011 16:06:34 -0600
> >> Anthony Liguori <anthony@codemonkey.ws> wrote:
> >>
> >>> On 01/24/2011 07:25 AM, Chris Wright wrote:
> >>>> Please send in any agenda items you are interested in covering.
> >>>>    
> >>> - coroutines for the block layer
> >>> - glib everywhere
> >>
> >> - Let's start planning our next release in advance, here's a simple example:
> >>
> >>   http://wiki.qemu.org/Planning/0.15-example
> >>
> > 
> > What about planning the 0.14 release first? From what I see on the
> > mailing list, we are more in a development phase than in a bug fix phase
> > before a release.
> 
> If we want to get into a bug fix phase, all we need to have is a
> stable-0.14 branch. It's not like creating a branch is a lot of work, it
> just didn't happen on the announced date.

Agreed. That said when you see the branch date approaching you already 
focus more on identifying and fixing bugs than on new features, that's 
human. For example after seeing the mail from Anthony I started to test
the main TCG targets (actually 8 of them) on the main supported hosts 
(7 of them if you count endianness). That's quite a lot of work that has
been quite useful, that said given the number of patches that have been
committed in the meanwhile, it's probably something to redo.

> And I'm almost sure that when it happens, it's going to come as a
> surprise once again...

If we decide on a date in advance, it's something I can easily do if
Anthony or someone else is busy. We just have to agree on that before.

-- 
Aurelien Jarno	                        GPG: 1024D/F1BCDB73
aurelien@aurel32.net                 http://www.aurel32.net

[Qemu-devel] Re: KVM call agenda for Jan 25

[Avi Kivity]

On 01/25/2011 12:06 AM, Anthony Liguori wrote:
> On 01/24/2011 07:25 AM, Chris Wright wrote:
>> Please send in any agenda items you are interested in covering.
>
> - coroutines for the block layer

I have a perpetually in progress branch for this, and would very much 
like to see this done.

-- 
error compiling committee.c: too many arguments to function

[Qemu-devel] Re: KVM call agenda for Jan 25

[Stefan Hajnoczi]

On Tue, Jan 25, 2011 at 2:26 PM, Avi Kivity <avi@redhat.com> wrote:
> On 01/25/2011 12:06 AM, Anthony Liguori wrote:
>>
>> On 01/24/2011 07:25 AM, Chris Wright wrote:
>>>
>>> Please send in any agenda items you are interested in covering.
>>
>> - coroutines for the block layer
>
> I have a perpetually in progress branch for this, and would very much like
> to see this done.

Seen this?
http://repo.or.cz/w/qemu/stefanha.git/commit/8179e8ff20bb3f14f361109afe5b3bf2bac24f0d
http://repo.or.cz/w/qemu/stefanha.git/shortlog/8179e8ff20bb3f14f361109afe5b3bf2bac24f0d

And the qemu-devel thread:
http://www.mail-archive.com/qemu-devel@nongnu.org/msg52522.html

Stefan

[Qemu-devel] Re: KVM call agenda for Jan 25

[Avi Kivity]

On 01/25/2011 04:35 PM, Stefan Hajnoczi wrote:
> On Tue, Jan 25, 2011 at 2:26 PM, Avi Kivity<avi@redhat.com>  wrote:
> >  On 01/25/2011 12:06 AM, Anthony Liguori wrote:
> >>
> >>  On 01/24/2011 07:25 AM, Chris Wright wrote:
> >>>
> >>>  Please send in any agenda items you are interested in covering.
> >>
> >>  - coroutines for the block layer
> >
> >  I have a perpetually in progress branch for this, and would very much like
> >  to see this done.
>
> Seen this?
> http://repo.or.cz/w/qemu/stefanha.git/commit/8179e8ff20bb3f14f361109afe5b3bf2bac24f0d
> http://repo.or.cz/w/qemu/stefanha.git/shortlog/8179e8ff20bb3f14f361109afe5b3bf2bac24f0d
>
> And the qemu-devel thread:
> http://www.mail-archive.com/qemu-devel@nongnu.org/msg52522.html

Thanks.  Will follow up on the thread.

-- 
error compiling committee.c: too many arguments to function