[Qemu-devel] [RFC] lively write vmstate with predictable size

[Qemu-devel] [RFC] lively write vmstate with predictable size

[Wenchao Xia]

resent the mail to mail-list.
-------------------

Hi, Paolo and Juan
    Currently savevm needs pause vm, and I am working on that make it
 lively. Considering the flexibility I'd like to split out the
 functions apart as following:
 1) snapshot lively internal/external
 2) save vmstate lively internal/external
 3) assemble them as will

 1) was sent at
 http://lists.nongnu.org/archive/html/qemu-devel/2012-12/msg02393.html

 but for 2), I think it have problem because file size may grow to a size
 out of control. Considering the migration code, I'd like to propose a
 way to fix it as following:

 Migration logic:
 Src send->dest recv->data analysis->copy data
 Savevm logic:
 Src send->write data to qcow2.

My suggestion:
 Savevm logic:
 Src send->dest recv->data analysis->write data to qcow2/external with
 addr.

   The idea is do the write operation after data analysis, and overwrite
 old data if address overlaps. So this will need qcow2 support
 write snapshot data at "address", and also change some savevm logic.

   Could u give some some comments on this to see if it is workable?


-- 
Best Regards

Wenchao Xia

Re: [Qemu-devel] [RFC] lively write vmstate with predictable size

[Juan Quintela]

Wenchao Xia <xiawenc@linux.vnet.ibm.com> wrote:
> resent the mail to mail-list.
> -------------------
>
> Hi, Paolo and Juan
>     Currently savevm needs pause vm, and I am working on that make it
>  lively. Considering the flexibility I'd like to split out the
>  functions apart as following:
>  1) snapshot lively internal/external
>  2) save vmstate lively internal/external
>  3) assemble them as will
>
>  1) was sent at
>  http://lists.nongnu.org/archive/html/qemu-devel/2012-12/msg02393.html
>
>  but for 2), I think it have problem because file size may grow to a size
>  out of control. Considering the migration code, I'd like to propose a
>  way to fix it as following:
>
>  Migration logic:
>  Src send->dest recv->data analysis->copy data
>  Savevm logic:
>  Src send->write data to qcow2.
>
> My suggestion:
>  Savevm logic:
>  Src send->dest recv->data analysis->write data to qcow2/external with
>  addr.
>
>    The idea is do the write operation after data analysis, and overwrite
>  old data if address overlaps. So this will need qcow2 support
>  write snapshot data at "address", and also change some savevm logic.

We could change the code to do it, but it is not going to be
trivial. Furthermore we should disable xbzrle (in this case, it just
makes no sense)

The easiest way that I can think of is changing the
arch_init.c:

We can:
a- reserve a big enough area in the qcow2 image/external to store the
whole ram
b- change ram_save_block() on that file to directly write to the "right"
   position inside the image.
c- change ram_load() to understand the new format.

Notice that this would have to be a different implementation that
sending data over tcp, as there we want (something) similar to the
current code.


>    Could u give some some comments on this to see if it is workable?

It needs lots of code rearragement as far as I can think, it is doable
but it is quite a bit of work.

Later, Juan.

Re: [Qemu-devel] [RFC] lively write vmstate with predictable size

[Wenchao Xia]

Hi, Juan
  Thank u for reviewing on this, have some questions below.

> Wenchao Xia <xiawenc@linux.vnet.ibm.com> wrote:
>> resent the mail to mail-list.
>> -------------------
>>
>> Hi, Paolo and Juan
>>      Currently savevm needs pause vm, and I am working on that make it
>>   lively. Considering the flexibility I'd like to split out the
>>   functions apart as following:
>>   1) snapshot lively internal/external
>>   2) save vmstate lively internal/external
>>   3) assemble them as will
>>
>>   1) was sent at
>>   http://lists.nongnu.org/archive/html/qemu-devel/2012-12/msg02393.html
>>
>>   but for 2), I think it have problem because file size may grow to a size
>>   out of control. Considering the migration code, I'd like to propose a
>>   way to fix it as following:
>>
>>   Migration logic:
>>   Src send->dest recv->data analysis->copy data
>>   Savevm logic:
>>   Src send->write data to qcow2.
>>
>> My suggestion:
>>   Savevm logic:
>>   Src send->dest recv->data analysis->write data to qcow2/external with
>>   addr.
>>
>>     The idea is do the write operation after data analysis, and overwrite
>>   old data if address overlaps. So this will need qcow2 support
>>   write snapshot data at "address", and also change some savevm logic.
> 
> We could change the code to do it, but it is not going to be
> trivial. Furthermore we should disable xbzrle (in this case, it just
> makes no sense)
> 
> The easiest way that I can think of is changing the
> arch_init.c:
> 
> We can:
> a- reserve a big enough area in the qcow2 image/external to store the
> whole ram
> b- change ram_save_block() on that file to directly write to the "right"
>     position inside the image.
> c- change ram_load() to understand the new format.
> 
  I agree with your way for that it saves computation for
data analysis. But why a big area should be reserved and new format
is needed? I thought every thing would be same as old vmstate save
case with vm stopped, if qemu can overwrite the contents in the right
place and leaves remaining tag unchanged.

  how about write a new function replace save_block_hdr() and following
buffer writing, which would be an abstract function used to write data
with addr? It still encapsulate it as save_block_hdr() logic, but in
savevm case it will directly write data to a right place. In this way
at vmstate contents level, nothing changes, it still stores the same
stream as if vm is stopped.

> Notice that this would have to be a different implementation that
> sending data over tcp, as there we want (something) similar to the
> current code.
> 
  I think it would be a choice between performance and code unifying,
personally I guess performance is the first to consider for qemu.

> 
>>     Could u give some some comments on this to see if it is workable?
> 
> It needs lots of code rearragement as far as I can think, it is doable
> but it is quite a bit of work.
> 
> Later, Juan.
> 


-- 
Best Regards

Wenchao Xia

Re: [Qemu-devel] [RFC] lively write vmstate with predictable size

[Wenchao Xia]

> Hi, Juan
>    Thank u for reviewing on this, have some questions below.
> 
>> Wenchao Xia <xiawenc@linux.vnet.ibm.com> wrote:
>>> resent the mail to mail-list.
>>> -------------------
>>>
>>> Hi, Paolo and Juan
>>>       Currently savevm needs pause vm, and I am working on that make it
>>>    lively. Considering the flexibility I'd like to split out the
>>>    functions apart as following:
>>>    1) snapshot lively internal/external
>>>    2) save vmstate lively internal/external
>>>    3) assemble them as will
>>>
>>>    1) was sent at
>>>    http://lists.nongnu.org/archive/html/qemu-devel/2012-12/msg02393.html
>>>
>>>    but for 2), I think it have problem because file size may grow to a size
>>>    out of control. Considering the migration code, I'd like to propose a
>>>    way to fix it as following:
>>>
>>>    Migration logic:
>>>    Src send->dest recv->data analysis->copy data
>>>    Savevm logic:
>>>    Src send->write data to qcow2.
>>>
>>> My suggestion:
>>>    Savevm logic:
>>>    Src send->dest recv->data analysis->write data to qcow2/external with
>>>    addr.
>>>
>>>      The idea is do the write operation after data analysis, and overwrite
>>>    old data if address overlaps. So this will need qcow2 support
>>>    write snapshot data at "address", and also change some savevm logic.
>>
>> We could change the code to do it, but it is not going to be
>> trivial. Furthermore we should disable xbzrle (in this case, it just
>> makes no sense)
>>
>> The easiest way that I can think of is changing the
>> arch_init.c:
>>
>> We can:
>> a- reserve a big enough area in the qcow2 image/external to store the
>> whole ram
>> b- change ram_save_block() on that file to directly write to the "right"
>>      position inside the image.
>> c- change ram_load() to understand the new format.
>>
>    I agree with your way for that it saves computation for
> data analysis. But why a big area should be reserved and new format
> is needed? I thought every thing would be same as old vmstate save
> case with vm stopped, if qemu can overwrite the contents in the right
> place and leaves remaining tag unchanged.
> 
Hi, Juan
  After reconsideration, I think preallocation is indeed needed, or a
file format is needed which make write to a position larger than the
file should extend the file to the same size of postion(maybe some
filesystem support hole can make it)?
  I suppose a solution to make things relative easy:
1) at start qemu make sure space is preallocated or a file support
 holes.
2) calculate the vmstate size, and arrange the contents lineiar with
communication headers, that is what is in save_block_hdr() and other key
stamps.
3) write those data lively with headers to the arranged position.

  In this way the qcow2 internal snapshot format need not be changed,
but sacrifice some space used to store communication stamps, which I
think would not be too many. And if the coding in this approach is
carefully and predict the remove the write stamps, it would be easy
to remove the stamps later as a new snapshot format, if a flag is
added in the code. But before all a good arrangement of vmstate
is needed.
  what do you think about it? :)

>    how about write a new function replace save_block_hdr() and following
> buffer writing, which would be an abstract function used to write data
> with addr? It still encapsulate it as save_block_hdr() logic, but in
> savevm case it will directly write data to a right place. In this way
> at vmstate contents level, nothing changes, it still stores the same
> stream as if vm is stopped.
> 
>> Notice that this would have to be a different implementation that
>> sending data over tcp, as there we want (something) similar to the
>> current code.
>>
>    I think it would be a choice between performance and code unifying,
> personally I guess performance is the first to consider for qemu.
> 
>>
>>>      Could u give some some comments on this to see if it is workable?
>>
>> It needs lots of code rearragement as far as I can think, it is doable
>> but it is quite a bit of work.
>>
>> Later, Juan.
>>
> 
> 


-- 
Best Regards

Wenchao Xia

Re: [Qemu-devel] [RFC] lively write vmstate with predictable size

[Stefan Hajnoczi]

On Fri, Dec 21, 2012 at 10:25:51AM +0800, Wenchao Xia wrote:
> resent the mail to mail-list.
> -------------------
> 
> Hi, Paolo and Juan
>     Currently savevm needs pause vm, and I am working on that make it
>  lively.

What is the benefit over migrating to file?  I thought we can already do
that today?

Stefan

Re: [Qemu-devel] [RFC] lively write vmstate with predictable size

[Juan Quintela]

Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Fri, Dec 21, 2012 at 10:25:51AM +0800, Wenchao Xia wrote:
>> resent the mail to mail-list.
>> -------------------
>> 
>> Hi, Paolo and Juan
>>     Currently savevm needs pause vm, and I am working on that make it
>>  lively.
>
> What is the benefit over migrating to file?  I thought we can already do
> that today?

Load is faster.

He wants to "overwrite" the memory, so, if we dirty one page during
migration, it overwrite it, instead of having two copies that only make
the image bigger and slower to load.  Problem is that we need a new
"format" to be able to do that.

Later, Juan.

Re: [Qemu-devel] [RFC] lively write vmstate with predictable size

[Wenchao Xia]

于 2013-1-5 6:42, Juan Quintela 写道:
> Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> On Fri, Dec 21, 2012 at 10:25:51AM +0800, Wenchao Xia wrote:
>>> resent the mail to mail-list.
>>> -------------------
>>>
>>> Hi, Paolo and Juan
>>>      Currently savevm needs pause vm, and I am working on that make it
>>>   lively.
>>
>> What is the benefit over migrating to file?  I thought we can already do
>> that today?
> 
> Load is faster.
> 
> He wants to "overwrite" the memory, so, if we dirty one page during
> migration, it overwrite it, instead of having two copies that only make
> the image bigger and slower to load.  Problem is that we need a new
> "format" to be able to do that.
> 
> Later, Juan.
> 
  I guess migration to file makes vm stopped, and lack a step
to take snapshot in the end of migration, also the usage seems
strange makes me feel something is lacking. I'd like to
have:
1 take blk snapshot internal/external lively.
2 take vmstate snapshot internal/external lively.
3 combination.

-- 
Best Regards

Wenchao Xia

Re: [Qemu-devel] [RFC] lively write vmstate with predictable size

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1344 bytes --]

On 01/05/2013 01:21 AM, Wenchao Xia wrote:
>   I guess migration to file makes vm stopped, and lack a step
> to take snapshot in the end of migration, also the usage seems
> strange makes me feel something is lacking. I'd like to
> have:
> 1 take blk snapshot internal/external lively.
> 2 take vmstate snapshot internal/external lively.
> 3 combination.

Libvirt 1.0.1 can already do external live snapshots - it does a live
migration to file, and when that finally converges enough to pause,
there is less than a second of downtime when libvirt then follows up
with an external disk snapshot and then resumes the guest.  Of course,
this approach requires multiple files, and there is no way to take an
internal snapshot with current QMP support.  And the point remains in
this thread that a live migration to file takes more space than a paused
migration, in part because earlier portions of the migration stream are
revisited later in the stream as memory pages continue to be touched
during the live migration.  So a new format for seekable migration to
file, different than current migration being usable over a fifo, may
indeed allow for less disk space and faster booting when reverting to
that vmstate.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: [Qemu-devel] [RFC] lively write vmstate with predictable size

[Stefan Hajnoczi]

On Sat, Jan 05, 2013 at 08:28:18PM -0700, Eric Blake wrote:
> On 01/05/2013 01:21 AM, Wenchao Xia wrote:
> >   I guess migration to file makes vm stopped, and lack a step
> > to take snapshot in the end of migration, also the usage seems
> > strange makes me feel something is lacking. I'd like to
> > have:
> > 1 take blk snapshot internal/external lively.
> > 2 take vmstate snapshot internal/external lively.
> > 3 combination.
> 
> Libvirt 1.0.1 can already do external live snapshots - it does a live
> migration to file, and when that finally converges enough to pause,
> there is less than a second of downtime when libvirt then follows up
> with an external disk snapshot and then resumes the guest.  Of course,
> this approach requires multiple files, and there is no way to take an
> internal snapshot with current QMP support.  And the point remains in
> this thread that a live migration to file takes more space than a paused
> migration, in part because earlier portions of the migration stream are
> revisited later in the stream as memory pages continue to be touched
> during the live migration.  So a new format for seekable migration to
> file, different than current migration being usable over a fifo, may
> indeed allow for less disk space and faster booting when reverting to
> that vmstate.

Thanks for the explanations Wenchao, Juan, and Eric.

Stefan