[Qemu-devel] [PULL 00/34] migration thread and queue

[Qemu-devel] [PULL 00/34] migration thread and queue

[Juan Quintela]

Hi

Changes for yesterday:
- Paolo Acked the series
- Rebaso on top of today git (only conflicts were due to header re-shuffle)

Please pull.

[20121219]

This is my queue for migration-thread and patches associated.  This
integrates review comments & code for Paolo.  This is the subset from
both approachs that we agreed with. rest of patches need more review
and are not here.

Migrating and idle guest with upstwream:

(qemu) info migrate
capabilities: xbzrle: off
Migration status: completed
total time: 34251 milliseconds
downtime: 492 milliseconds
transferred ram: 762458 kbytes
remaining ram: 0 kbytes
total ram: 14688768 kbytes
duplicate: 3492606 pages
normal: 189762 pages
normal bytes: 759048 kbytes

with this series of patches.

(qemu) info migrate
capabilities: xbzrle: off
Migration status: completed
total time: 30712 milliseconds
downtime: 29 milliseconds
transferred ram: 738857 kbytes
remaining ram: 0 kbytes
total ram: 14688768 kbytes
duplicate: 3503423 pages
normal: 176671 pages
normal bytes: 706684 kbytes

Notice the big difference in downtime.  And that is also seen inside
the guest a program that just do an idle loop seeing how "long" it
takes to wait for 10ms.

with upstream:

[root@d1 ~]# ./timer
delay of 452 ms
delay of 114 ms
delay of 136 ms
delay of 135 ms
delay of 136 ms
delay of 131 ms
delay of 134 ms

with this series of patches, wait never takes 100ms, nothing is printed.

Please pull.

Thanks, Juan.


The following changes since commit 27dd7730582be85c7d4f680f5f71146629809c86:

  Merge remote-tracking branch 'bonzini/header-dirs' into staging (2012-12-19 17:15:39 -0600)

are available in the git repository at:


  git://repo.or.cz/qemu/quintela.git thread-20121220.next

for you to fetch changes up to 9848a40427cd76628d04d918fa4751c542527915:

  migration: merge QEMUFileBuffered into MigrationState (2012-12-20 23:09:40 +0100)

----------------------------------------------------------------
Juan Quintela (25):
      migration: include qemu-file.h
      migration-fd: remove duplicate include
      buffered_file: Move from using a timer to use a thread
      migration: make qemu_fopen_ops_buffered() return void
      migration: move migration thread init code to migrate_fd_put_ready
      migration: make writes blocking
      migration: remove unfreeze logic
      migration: just lock migrate_fd_put_ready
      buffered_file: Unfold the trick to restart generating migration data
      buffered_file: don't flush on put buffer
      buffered_file: unfold buffered_append in buffered_put_buffer
      savevm: New save live migration method: pending
      migration: move buffered_file.c code into migration.c
      migration: add XFER_LIMIT_RATIO
      migration: move migration_fd_put_ready()
      migration: Inline qemu_fopen_ops_buffered into migrate_fd_connect
      migration: move migration notifier
      ram: rename last_block to last_seen_block
      ram: Add last_sent_block
      memory: introduce memory_region_test_and_clear_dirty
      ram: Use memory_region_test_and_clear_dirty
      ram: optimize migration bitmap walking
      ram: account the amount of transferred ram better
      ram: refactor ram_save_block() return value
      migration: merge QEMUFileBuffered into MigrationState

Paolo Bonzini (7):
      migration: fix migration_bitmap leak
      buffered_file: do not send more than s->bytes_xfer bytes per tick
      migration: remove double call to migrate_fd_close
      exec: change ramlist from MRU order to a 1-item cache
      exec: change RAM list to a TAILQ
      exec: sort the memory from biggest to smallest
      migration: fix qemu_get_fd for BufferedFile

Umesh Deshpande (2):
      add a version number to ram_list
      protect the ramlist with a separate mutex

 Makefile.objs                 |   3 +-
 arch_init.c                   | 245 +++++++++++++-------------
 block-migration.c             |  49 ++----
 buffered_file.c               | 268 -----------------------------
 buffered_file.h               |  22 ---
 dump.c                        |   8 +-
 exec.c                        | 128 +++++++++-----
 include/exec/cpu-all.h        |  15 +-
 include/exec/memory.h         |  16 ++
 include/migration/migration.h |  13 +-
 include/migration/qemu-file.h |   5 -
 include/migration/vmstate.h   |   1 +
 include/sysemu/sysemu.h       |   1 +
 memory.c                      |  16 ++
 memory_mapping.c              |   4 +-
 migration-exec.c              |   3 +-
 migration-fd.c                |   4 +-
 migration-tcp.c               |   3 +-
 migration-unix.c              |   3 +-
 migration.c                   | 390 +++++++++++++++++++++++++++++++-----------
 savevm.c                      |  24 ++-
 target-i386/arch_dump.c       |   2 +-
 22 files changed, 600 insertions(+), 623 deletions(-)
 delete mode 100644 buffered_file.c
 delete mode 100644 buffered_file.h

Re: [Qemu-devel] [PULL 00/34] migration thread and queue

[Anthony Liguori]

Juan Quintela <quintela@redhat.com> writes:

> Hi
>
> Changes for yesterday:
> - Paolo Acked the series
> - Rebaso on top of today git (only conflicts were due to header re-shuffle)
>
> Please pull.
>
> [20121219]
>
> This is my queue for migration-thread and patches associated.  This
> integrates review comments & code for Paolo.  This is the subset from
> both approachs that we agreed with. rest of patches need more review
> and are not here.

This breaks migration.  Here's my test case:

#!/bin/sh

gzip -d -c lidb.ssmall.img.gz | \
    ~/build/qemu/x86_64-softmmu/qemu-system-x86_64 -cdrom \
    ~/isos/OCDC-lucid-Test-Drive-20110523_140333.iso -enable-kvm -m 2G \
    -incoming exec:dd

With lidb.ssmall.img.gz being generated from a 'migrate exec:dd of=...'
from qemu.git just a week or two ago.

I can't bisect tonight but can attempt to tomorrow.  How has this been
tested?

I'm a little concerned here about the timing.  With the Christmas
and New Years holiday we're pretty darn close to soft freeze for 1.4.

Has this series gone through a full autotest run with multiple guests?

Regards,

Anthony Liguori

>
> Migrating and idle guest with upstwream:
>
> (qemu) info migrate
> capabilities: xbzrle: off
> Migration status: completed
> total time: 34251 milliseconds
> downtime: 492 milliseconds
> transferred ram: 762458 kbytes
> remaining ram: 0 kbytes
> total ram: 14688768 kbytes
> duplicate: 3492606 pages
> normal: 189762 pages
> normal bytes: 759048 kbytes
>
> with this series of patches.
>
> (qemu) info migrate
> capabilities: xbzrle: off
> Migration status: completed
> total time: 30712 milliseconds
> downtime: 29 milliseconds
> transferred ram: 738857 kbytes
> remaining ram: 0 kbytes
> total ram: 14688768 kbytes
> duplicate: 3503423 pages
> normal: 176671 pages
> normal bytes: 706684 kbytes
>
> Notice the big difference in downtime.  And that is also seen inside
> the guest a program that just do an idle loop seeing how "long" it
> takes to wait for 10ms.
>
> with upstream:
>
> [root@d1 ~]# ./timer
> delay of 452 ms
> delay of 114 ms
> delay of 136 ms
> delay of 135 ms
> delay of 136 ms
> delay of 131 ms
> delay of 134 ms
>
> with this series of patches, wait never takes 100ms, nothing is printed.
>
> Please pull.
>
> Thanks, Juan.
>
>
> The following changes since commit 27dd7730582be85c7d4f680f5f71146629809c86:
>
>   Merge remote-tracking branch 'bonzini/header-dirs' into staging (2012-12-19 17:15:39 -0600)
>
> are available in the git repository at:
>
>
>   git://repo.or.cz/qemu/quintela.git thread-20121220.next
>
> for you to fetch changes up to 9848a40427cd76628d04d918fa4751c542527915:
>
>   migration: merge QEMUFileBuffered into MigrationState (2012-12-20 23:09:40 +0100)
>
> ----------------------------------------------------------------
> Juan Quintela (25):
>       migration: include qemu-file.h
>       migration-fd: remove duplicate include
>       buffered_file: Move from using a timer to use a thread
>       migration: make qemu_fopen_ops_buffered() return void
>       migration: move migration thread init code to migrate_fd_put_ready
>       migration: make writes blocking
>       migration: remove unfreeze logic
>       migration: just lock migrate_fd_put_ready
>       buffered_file: Unfold the trick to restart generating migration data
>       buffered_file: don't flush on put buffer
>       buffered_file: unfold buffered_append in buffered_put_buffer
>       savevm: New save live migration method: pending
>       migration: move buffered_file.c code into migration.c
>       migration: add XFER_LIMIT_RATIO
>       migration: move migration_fd_put_ready()
>       migration: Inline qemu_fopen_ops_buffered into migrate_fd_connect
>       migration: move migration notifier
>       ram: rename last_block to last_seen_block
>       ram: Add last_sent_block
>       memory: introduce memory_region_test_and_clear_dirty
>       ram: Use memory_region_test_and_clear_dirty
>       ram: optimize migration bitmap walking
>       ram: account the amount of transferred ram better
>       ram: refactor ram_save_block() return value
>       migration: merge QEMUFileBuffered into MigrationState
>
> Paolo Bonzini (7):
>       migration: fix migration_bitmap leak
>       buffered_file: do not send more than s->bytes_xfer bytes per tick
>       migration: remove double call to migrate_fd_close
>       exec: change ramlist from MRU order to a 1-item cache
>       exec: change RAM list to a TAILQ
>       exec: sort the memory from biggest to smallest
>       migration: fix qemu_get_fd for BufferedFile
>
> Umesh Deshpande (2):
>       add a version number to ram_list
>       protect the ramlist with a separate mutex
>
>  Makefile.objs                 |   3 +-
>  arch_init.c                   | 245 +++++++++++++-------------
>  block-migration.c             |  49 ++----
>  buffered_file.c               | 268 -----------------------------
>  buffered_file.h               |  22 ---
>  dump.c                        |   8 +-
>  exec.c                        | 128 +++++++++-----
>  include/exec/cpu-all.h        |  15 +-
>  include/exec/memory.h         |  16 ++
>  include/migration/migration.h |  13 +-
>  include/migration/qemu-file.h |   5 -
>  include/migration/vmstate.h   |   1 +
>  include/sysemu/sysemu.h       |   1 +
>  memory.c                      |  16 ++
>  memory_mapping.c              |   4 +-
>  migration-exec.c              |   3 +-
>  migration-fd.c                |   4 +-
>  migration-tcp.c               |   3 +-
>  migration-unix.c              |   3 +-
>  migration.c                   | 390 +++++++++++++++++++++++++++++++-----------
>  savevm.c                      |  24 ++-
>  target-i386/arch_dump.c       |   2 +-
>  22 files changed, 600 insertions(+), 623 deletions(-)
>  delete mode 100644 buffered_file.c
>  delete mode 100644 buffered_file.h

Re: [Qemu-devel] [PULL 00/34] migration thread and queue

[Juan Quintela]

Anthony Liguori <anthony@codemonkey.ws> wrote:
> Juan Quintela <quintela@redhat.com> writes:
>
>> Hi
>>
>> Changes for yesterday:
>> - Paolo Acked the series
>> - Rebaso on top of today git (only conflicts were due to header re-shuffle)
>>
>> Please pull.
>>
>> [20121219]
>>
>> This is my queue for migration-thread and patches associated.  This
>> integrates review comments & code for Paolo.  This is the subset from
>> both approachs that we agreed with. rest of patches need more review
>> and are not here.
>
> This breaks migration.  Here's my test case:
>
> #!/bin/sh
>
> gzip -d -c lidb.ssmall.img.gz | \
>     ~/build/qemu/x86_64-softmmu/qemu-system-x86_64 -cdrom \
>     ~/isos/OCDC-lucid-Test-Drive-20110523_140333.iso -enable-kvm -m 2G \
>     -incoming exec:dd
>
> With lidb.ssmall.img.gz being generated from a 'migrate exec:dd of=...'
> from qemu.git just a week or two ago.
>
> I can't bisect tonight but can attempt to tomorrow.  How has this been
> tested?

I have tested with tcp, load/not-load with guests form 4GB to 16GB RAM.
Will test tomorrow with your test case.  I didn't tested exec:, though.

>
> I'm a little concerned here about the timing.  With the Christmas
> and New Years holiday we're pretty darn close to soft freeze for 1.4.
>
> Has this series gone through a full autotest run with multiple guests?

Will re-test tomorrow with autotest.

Sorry, Juan.

Re: [Qemu-devel] [PULL 00/34] migration thread and queue

[Juan Quintela]

Juan Quintela <quintela@redhat.com> wrote:
> Anthony Liguori <anthony@codemonkey.ws> wrote:
>> Juan Quintela <quintela@redhat.com> writes:
>>
>>> Hi
>>>
>>> Changes for yesterday:
>>> - Paolo Acked the series
>>> - Rebaso on top of today git (only conflicts were due to header re-shuffle)
>>>
>>> Please pull.
>>>
>>> [20121219]
>>>
>>> This is my queue for migration-thread and patches associated.  This
>>> integrates review comments & code for Paolo.  This is the subset from
>>> both approachs that we agreed with. rest of patches need more review
>>> and are not here.
>>
>> This breaks migration.  Here's my test case:
>>
>> #!/bin/sh
>>
>> gzip -d -c lidb.ssmall.img.gz | \
>>     ~/build/qemu/x86_64-softmmu/qemu-system-x86_64 -cdrom \
>>     ~/isos/OCDC-lucid-Test-Drive-20110523_140333.iso -enable-kvm -m 2G \
>>     -incoming exec:dd
>>
>> With lidb.ssmall.img.gz being generated from a 'migrate exec:dd of=...'
>> from qemu.git just a week or two ago.
>>
>> I can't bisect tonight but can attempt to tomorrow.  How has this been
>> tested?
>
> I have tested with tcp, load/not-load with guests form 4GB to 16GB RAM.
> Will test tomorrow with your test case.  I didn't tested exec:, though.

This is failing to me with master, nothing to do with the changes on
this thread.  Investigating what is going on.

Later, Juan.

Re: [Qemu-devel] [PULL 00/34] migration thread and queue

[Anthony Liguori]

Juan Quintela <quintela@redhat.com> writes:

> Juan Quintela <quintela@redhat.com> wrote:
>> Anthony Liguori <anthony@codemonkey.ws> wrote:
>>> Juan Quintela <quintela@redhat.com> writes:
>>>
>>>> Hi
>>>>
>>>> Changes for yesterday:
>>>> - Paolo Acked the series
>>>> - Rebaso on top of today git (only conflicts were due to header re-shuffle)
>>>>
>>>> Please pull.
>>>>
>>>> [20121219]
>>>>
>>>> This is my queue for migration-thread and patches associated.  This
>>>> integrates review comments & code for Paolo.  This is the subset from
>>>> both approachs that we agreed with. rest of patches need more review
>>>> and are not here.
>>>
>>> This breaks migration.  Here's my test case:
>>>
>>> #!/bin/sh
>>>
>>> gzip -d -c lidb.ssmall.img.gz | \
>>>     ~/build/qemu/x86_64-softmmu/qemu-system-x86_64 -cdrom \
>>>     ~/isos/OCDC-lucid-Test-Drive-20110523_140333.iso -enable-kvm -m 2G \
>>>     -incoming exec:dd
>>>
>>> With lidb.ssmall.img.gz being generated from a 'migrate exec:dd of=...'
>>> from qemu.git just a week or two ago.
>>>
>>> I can't bisect tonight but can attempt to tomorrow.  How has this been
>>> tested?
>>
>> I have tested with tcp, load/not-load with guests form 4GB to 16GB RAM.
>> Will test tomorrow with your test case.  I didn't tested exec:, though.
>
> This is failing to me with master, nothing to do with the changes on
> this thread.  Investigating what is going on.

Master works perfectly fine for me.  It only fails with your series
merged in.

I can bisect if you like.

Regards,

Anthony Liguori

>
> Later, Juan.

Re: [Qemu-devel] [PULL 00/34] migration thread and queue

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 792 bytes --]

Il 21/12/2012 02:39, Juan Quintela ha scritto:
>> I can't bisect tonight but can attempt to tomorrow.  How has this been
>> tested?
> 
> I have tested with tcp, load/not-load with guests form 4GB to 16GB RAM.
> Will test tomorrow with your test case.  I didn't tested exec:, though.
> 
>>
>> I'm a little concerned here about the timing.  With the Christmas
>> and New Years holiday we're pretty darn close to soft freeze for 1.4.

(To expand on my previous message, might as well declare soft freeze
today if that is the case).

>> Has this series gone through a full autotest run with multiple guests?
> 
> Will re-test tomorrow with autotest.

You haven't integrated the fixes I sent you yesterday (attached and
placed at migration-thread-20121220-lite).  I'm running autotest now.

Paolo

[-- Attachment #2: migr.patch --]
[-- Type: text/x-patch, Size: 1798 bytes --]

diff --git a/Makefile.objs b/Makefile.objs
index 4ef0a71..971b2f5 100644
--- a/Makefile.objs
+++ b/Makefile.objs
@@ -78,7 +78,6 @@ extra-obj-$(CONFIG_LINUX) += fsdev/
 
 common-obj-y += tcg-runtime.o host-utils.o main-loop.o
 common-obj-y += migration.o migration-tcp.o
-common-obj-y += migration.o migration-tcp.o
 common-obj-y += qemu-char.o #aio.o
 common-obj-y += block-migration.o iohandler.o
 common-obj-y += bitmap.o bitops.o
diff --git a/arch_init.c b/arch_init.c
index 86f8544..dada6de 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -642,12 +642,13 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
         i++;
     }
 
+    qemu_mutex_unlock_ramlist();
+
     if (ret < 0) {
         bytes_transferred += total_sent;
         return ret;
     }
 
-    qemu_mutex_unlock_ramlist();
     qemu_put_be64(f, RAM_SAVE_FLAG_EOS);
     total_sent += 8;
     bytes_transferred += total_sent;
@@ -657,9 +658,8 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
 
 static int ram_save_complete(QEMUFile *f, void *opaque)
 {
-    migration_bitmap_sync();
-
     qemu_mutex_lock_ramlist();
+    migration_bitmap_sync();
 
     /* try transferring iterative blocks of memory */
 
diff --git a/migration.c b/migration.c
index c69e864..ecda263 100644
--- a/migration.c
+++ b/migration.c
@@ -20,8 +20,8 @@
 #include "sysemu/sysemu.h"
 #include "block/block.h"
 #include "qemu/sockets.h"
-#include "migration/block.h"
 #include "qemu/thread.h"
+#include "migration/block.h"
 #include "qmp-commands.h"
 
 //#define DEBUG_MIGRATION
@@ -650,8 +650,8 @@ static int64_t buffered_set_rate_limit(void *opaque, int64_t new_rate)
         new_rate = SIZE_MAX;
     }
 
-    s->xfer_limit = new_rate / 10;
-
+    s->xfer_limit = new_rate / XFER_LIMIT_RATIO;
+    
 out:
     return s->xfer_limit;
 }

Re: [Qemu-devel] [PULL 00/34] migration thread and queue

[Juan Quintela]

Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 21/12/2012 02:39, Juan Quintela ha scritto:
>>> I can't bisect tonight but can attempt to tomorrow.  How has this been
>>> tested?
>> 
>> I have tested with tcp, load/not-load with guests form 4GB to 16GB RAM.
>> Will test tomorrow with your test case.  I didn't tested exec:, though.
>> 
>>>
>>> I'm a little concerned here about the timing.  With the Christmas
>>> and New Years holiday we're pretty darn close to soft freeze for 1.4.
>
> (To expand on my previous message, might as well declare soft freeze
> today if that is the case).
>
>>> Has this series gone through a full autotest run with multiple guests?
>> 
>> Will re-test tomorrow with autotest.
>
> You haven't integrated the fixes I sent you yesterday (attached and
> placed at migration-thread-20121220-lite).  I'm running autotest now.

I didn't included them jsut to do the pull request with the patches
already reviewed, was going to add them to next pull request (patches
are "trivial" right).  famous last words.

Just including patches in a pull request not before is something Anthony
don't like.

Later, Juan.

Re: [Qemu-devel] [PULL 00/34] migration thread and queue

[Anthony Liguori]

Paolo Bonzini <pbonzini@redhat.com> writes:

> Il 21/12/2012 02:39, Juan Quintela ha scritto:
>>> I can't bisect tonight but can attempt to tomorrow.  How has this been
>>> tested?
>> 
>> I have tested with tcp, load/not-load with guests form 4GB to 16GB RAM.
>> Will test tomorrow with your test case.  I didn't tested exec:, though.
>> 
>>>
>>> I'm a little concerned here about the timing.  With the Christmas
>>> and New Years holiday we're pretty darn close to soft freeze for 1.4.
>
> (To expand on my previous message, might as well declare soft freeze
> today if that is the case).

I think you misread my comment.  I'm not saying that I don't want to
take this series before soft freeze.  I'm reminding everyone that we're
getting close to soft freeze so we need to get this merged ASAP.

After the Christmas break, we'll have less than two weeks before soft
freeze.  That's not a tremendous amount of time to work out issues...

>
>>> Has this series gone through a full autotest run with multiple guests?
>> 
>> Will re-test tomorrow with autotest.
>
> You haven't integrated the fixes I sent you yesterday (attached and
> placed at migration-thread-20121220-lite).  I'm running autotest now.
>
> Paolo
> diff --git a/Makefile.objs b/Makefile.objs
> index 4ef0a71..971b2f5 100644
> --- a/Makefile.objs
> +++ b/Makefile.objs
> @@ -78,7 +78,6 @@ extra-obj-$(CONFIG_LINUX) += fsdev/
>  
>  common-obj-y += tcg-runtime.o host-utils.o main-loop.o
>  common-obj-y += migration.o migration-tcp.o
> -common-obj-y += migration.o migration-tcp.o
>  common-obj-y += qemu-char.o #aio.o
>  common-obj-y += block-migration.o iohandler.o
>  common-obj-y += bitmap.o bitops.o
> diff --git a/arch_init.c b/arch_init.c
> index 86f8544..dada6de 100644
> --- a/arch_init.c
> +++ b/arch_init.c
> @@ -642,12 +642,13 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
>          i++;
>      }
>  
> +    qemu_mutex_unlock_ramlist();
> +
>      if (ret < 0) {
>          bytes_transferred += total_sent;
>          return ret;
>      }
>  
> -    qemu_mutex_unlock_ramlist();
>      qemu_put_be64(f, RAM_SAVE_FLAG_EOS);
>      total_sent += 8;
>      bytes_transferred += total_sent;
> @@ -657,9 +658,8 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
>  
>  static int ram_save_complete(QEMUFile *f, void *opaque)
>  {
> -    migration_bitmap_sync();
> -
>      qemu_mutex_lock_ramlist();
> +    migration_bitmap_sync();
>  
>      /* try transferring iterative blocks of memory */
>  
> diff --git a/migration.c b/migration.c
> index c69e864..ecda263 100644
> --- a/migration.c
> +++ b/migration.c
> @@ -20,8 +20,8 @@
>  #include "sysemu/sysemu.h"
>  #include "block/block.h"
>  #include "qemu/sockets.h"
> -#include "migration/block.h"
>  #include "qemu/thread.h"
> +#include "migration/block.h"
>  #include "qmp-commands.h"
>  
>  //#define DEBUG_MIGRATION
> @@ -650,8 +650,8 @@ static int64_t buffered_set_rate_limit(void *opaque, int64_t new_rate)
>          new_rate = SIZE_MAX;
>      }
>  
> -    s->xfer_limit = new_rate / 10;
> -
> +    s->xfer_limit = new_rate / XFER_LIMIT_RATIO;
> +    
>  out:
>      return s->xfer_limit;
>  }

Do you think this would cause the behavior I've seen?  Not clear to me.

Regards,

Anthony Liguori

Re: [Qemu-devel] [PULL 00/34] migration thread and queue

[Paolo Bonzini]

Il 21/12/2012 14:39, Anthony Liguori ha scritto:
> Do you think this would cause the behavior I've seen?  Not clear to me.

No idea, the first two hunks however are locking fixes, who knows what
they might cause.

Paolo

Re: [Qemu-devel] [PULL 00/34] migration thread and queue

[Juan Quintela]

Juan Quintela <quintela@redhat.com> wrote:
> Anthony Liguori <anthony@codemonkey.ws> wrote:
>> Juan Quintela <quintela@redhat.com> writes:
>>
>>> Hi
>>>
>>> Changes for yesterday:
>>> - Paolo Acked the series
>>> - Rebaso on top of today git (only conflicts were due to header re-shuffle)
>>>
>>> Please pull.
>>>
>>> [20121219]
>>>
>>> This is my queue for migration-thread and patches associated.  This
>>> integrates review comments & code for Paolo.  This is the subset from
>>> both approachs that we agreed with. rest of patches need more review
>>> and are not here.
>>
>> This breaks migration.  Here's my test case:
>>
>> #!/bin/sh
>>
>> gzip -d -c lidb.ssmall.img.gz | \
>>     ~/build/qemu/x86_64-softmmu/qemu-system-x86_64 -cdrom \
>>     ~/isos/OCDC-lucid-Test-Drive-20110523_140333.iso -enable-kvm -m 2G \
>>     -incoming exec:dd
>>
>> With lidb.ssmall.img.gz being generated from a 'migrate exec:dd of=...'
>> from qemu.git just a week or two ago.
>>
>> I can't bisect tonight but can attempt to tomorrow.  How has this been
>> tested?
>
> I have tested with tcp, load/not-load with guests form 4GB to 16GB RAM.
> Will test tomorrow with your test case.  I didn't tested exec:, though.
>
>>
>> I'm a little concerned here about the timing.  With the Christmas
>> and New Years holiday we're pretty darn close to soft freeze for 1.4.
>>
>> Has this series gone through a full autotest run with multiple guests?
>
> Will re-test tomorrow with autotest.

Tested with autotest, it worked both upstream and the pull request.

Anthony, any news about your problems?

Can you pull the series?

Or do you preffer that I integrate Paolo fixes in the middle and sent
another pull request?

Thanks, Juan.

Re: [Qemu-devel] [PULL 00/34] migration thread and queue

[Paolo Bonzini]

Il 21/12/2012 01:27, Anthony Liguori ha scritto:
> Juan Quintela <quintela@redhat.com> writes:
> 
>> > Hi
>> >
>> > Changes for yesterday:
>> > - Paolo Acked the series
>> > - Rebaso on top of today git (only conflicts were due to header re-shuffle)
>> >
>> > Please pull.
>> >
>> > [20121219]
>> >
>> > This is my queue for migration-thread and patches associated.  This
>> > integrates review comments & code for Paolo.  This is the subset from
>> > both approachs that we agreed with. rest of patches need more review
>> > and are not here.
> This breaks migration.  Here's my test case:
> 
> #!/bin/sh
> 
> gzip -d -c lidb.ssmall.img.gz | \
>     ~/build/qemu/x86_64-softmmu/qemu-system-x86_64 -cdrom \
>     ~/isos/OCDC-lucid-Test-Drive-20110523_140333.iso -enable-kvm -m 2G \
>     -incoming exec:dd
> 
> With lidb.ssmall.img.gz being generated from a 'migrate exec:dd of=...'
> from qemu.git just a week or two ago.
> 
> I can't bisect tonight but can attempt to tomorrow.  How has this been
> tested?
> 
> I'm a little concerned here about the timing.  With the Christmas
> and New Years holiday we're pretty darn close to soft freeze for 1.4.

Come on, we're _a month_ from soft freeze.

Paolo

Re: [Qemu-devel] [PULL 00/34] migration thread and queue

[Anthony Liguori]

Anthony Liguori <anthony@codemonkey.ws> writes:

> Juan Quintela <quintela@redhat.com> writes:
>
>> Hi
>>
>> Changes for yesterday:
>> - Paolo Acked the series
>> - Rebaso on top of today git (only conflicts were due to header re-shuffle)
>>
>> Please pull.
>>
>> [20121219]
>>
>> This is my queue for migration-thread and patches associated.  This
>> integrates review comments & code for Paolo.  This is the subset from
>> both approachs that we agreed with. rest of patches need more review
>> and are not here.
>
> This breaks migration.  Here's my test case:
>
> #!/bin/sh
>
> gzip -d -c lidb.ssmall.img.gz | \
>     ~/build/qemu/x86_64-softmmu/qemu-system-x86_64 -cdrom \
>     ~/isos/OCDC-lucid-Test-Drive-20110523_140333.iso -enable-kvm -m 2G \
>     -incoming exec:dd
>
> With lidb.ssmall.img.gz being generated from a 'migrate exec:dd of=...'
> from qemu.git just a week or two ago.
>
> I can't bisect tonight but can attempt to tomorrow.  How has this been
> tested?
>
> I'm a little concerned here about the timing.  With the Christmas
> and New Years holiday we're pretty darn close to soft freeze for 1.4.
>
> Has this series gone through a full autotest run with multiple guests?

FYI, I'm having a heck of a time reproducing now attempting to do a
bisect.

This could indicate either a lingering build dependency problem or even
that I merged the wrong branch previously.

I'll keep testing and trying to reproduce.  I still would like to see a
full autotest run here though.

Regards,

Anthony Liguori

>
> Regards,
>
> Anthony Liguori
>
>>
>> Migrating and idle guest with upstwream:
>>
>> (qemu) info migrate
>> capabilities: xbzrle: off
>> Migration status: completed
>> total time: 34251 milliseconds
>> downtime: 492 milliseconds
>> transferred ram: 762458 kbytes
>> remaining ram: 0 kbytes
>> total ram: 14688768 kbytes
>> duplicate: 3492606 pages
>> normal: 189762 pages
>> normal bytes: 759048 kbytes
>>
>> with this series of patches.
>>
>> (qemu) info migrate
>> capabilities: xbzrle: off
>> Migration status: completed
>> total time: 30712 milliseconds
>> downtime: 29 milliseconds
>> transferred ram: 738857 kbytes
>> remaining ram: 0 kbytes
>> total ram: 14688768 kbytes
>> duplicate: 3503423 pages
>> normal: 176671 pages
>> normal bytes: 706684 kbytes
>>
>> Notice the big difference in downtime.  And that is also seen inside
>> the guest a program that just do an idle loop seeing how "long" it
>> takes to wait for 10ms.
>>
>> with upstream:
>>
>> [root@d1 ~]# ./timer
>> delay of 452 ms
>> delay of 114 ms
>> delay of 136 ms
>> delay of 135 ms
>> delay of 136 ms
>> delay of 131 ms
>> delay of 134 ms
>>
>> with this series of patches, wait never takes 100ms, nothing is printed.
>>
>> Please pull.
>>
>> Thanks, Juan.
>>
>>
>> The following changes since commit 27dd7730582be85c7d4f680f5f71146629809c86:
>>
>>   Merge remote-tracking branch 'bonzini/header-dirs' into staging (2012-12-19 17:15:39 -0600)
>>
>> are available in the git repository at:
>>
>>
>>   git://repo.or.cz/qemu/quintela.git thread-20121220.next
>>
>> for you to fetch changes up to 9848a40427cd76628d04d918fa4751c542527915:
>>
>>   migration: merge QEMUFileBuffered into MigrationState (2012-12-20 23:09:40 +0100)
>>
>> ----------------------------------------------------------------
>> Juan Quintela (25):
>>       migration: include qemu-file.h
>>       migration-fd: remove duplicate include
>>       buffered_file: Move from using a timer to use a thread
>>       migration: make qemu_fopen_ops_buffered() return void
>>       migration: move migration thread init code to migrate_fd_put_ready
>>       migration: make writes blocking
>>       migration: remove unfreeze logic
>>       migration: just lock migrate_fd_put_ready
>>       buffered_file: Unfold the trick to restart generating migration data
>>       buffered_file: don't flush on put buffer
>>       buffered_file: unfold buffered_append in buffered_put_buffer
>>       savevm: New save live migration method: pending
>>       migration: move buffered_file.c code into migration.c
>>       migration: add XFER_LIMIT_RATIO
>>       migration: move migration_fd_put_ready()
>>       migration: Inline qemu_fopen_ops_buffered into migrate_fd_connect
>>       migration: move migration notifier
>>       ram: rename last_block to last_seen_block
>>       ram: Add last_sent_block
>>       memory: introduce memory_region_test_and_clear_dirty
>>       ram: Use memory_region_test_and_clear_dirty
>>       ram: optimize migration bitmap walking
>>       ram: account the amount of transferred ram better
>>       ram: refactor ram_save_block() return value
>>       migration: merge QEMUFileBuffered into MigrationState
>>
>> Paolo Bonzini (7):
>>       migration: fix migration_bitmap leak
>>       buffered_file: do not send more than s->bytes_xfer bytes per tick
>>       migration: remove double call to migrate_fd_close
>>       exec: change ramlist from MRU order to a 1-item cache
>>       exec: change RAM list to a TAILQ
>>       exec: sort the memory from biggest to smallest
>>       migration: fix qemu_get_fd for BufferedFile
>>
>> Umesh Deshpande (2):
>>       add a version number to ram_list
>>       protect the ramlist with a separate mutex
>>
>>  Makefile.objs                 |   3 +-
>>  arch_init.c                   | 245 +++++++++++++-------------
>>  block-migration.c             |  49 ++----
>>  buffered_file.c               | 268 -----------------------------
>>  buffered_file.h               |  22 ---
>>  dump.c                        |   8 +-
>>  exec.c                        | 128 +++++++++-----
>>  include/exec/cpu-all.h        |  15 +-
>>  include/exec/memory.h         |  16 ++
>>  include/migration/migration.h |  13 +-
>>  include/migration/qemu-file.h |   5 -
>>  include/migration/vmstate.h   |   1 +
>>  include/sysemu/sysemu.h       |   1 +
>>  memory.c                      |  16 ++
>>  memory_mapping.c              |   4 +-
>>  migration-exec.c              |   3 +-
>>  migration-fd.c                |   4 +-
>>  migration-tcp.c               |   3 +-
>>  migration-unix.c              |   3 +-
>>  migration.c                   | 390 +++++++++++++++++++++++++++++++-----------
>>  savevm.c                      |  24 ++-
>>  target-i386/arch_dump.c       |   2 +-
>>  22 files changed, 600 insertions(+), 623 deletions(-)
>>  delete mode 100644 buffered_file.c
>>  delete mode 100644 buffered_file.h

Re: [Qemu-devel] [PULL 00/34] migration thread and queue

[Paolo Bonzini]

Il 21/12/2012 01:27, Anthony Liguori ha scritto:
> #!/bin/sh
> 
> gzip -d -c lidb.ssmall.img.gz | \
>     ~/build/qemu/x86_64-softmmu/qemu-system-x86_64 -cdrom \
>     ~/isos/OCDC-lucid-Test-Drive-20110523_140333.iso -enable-kvm -m 2G \
>     -incoming exec:dd
> 
> With lidb.ssmall.img.gz being generated from a 'migrate exec:dd of=...'
> from qemu.git just a week or two ago.

So it breaks *incoming* migration, which would restrict the problem to
the beginning of the patch (reorganization of the RAM block list).
Everything else touches only outgoing migration.

However, I tested bidirectional migration between origin/master and
Juan's branch and it passed

  x86_64-softmmu/qemu-system-x86_64 \
    -drive if=virtio,cache=none,file=/../images/jeos-17-64.qcow2 \
    -vnc :1 -incoming "exec:cat foo" --enable-kvm

I also tested both my and Juan's branches and they both pass all
migration tests.

> I can't bisect tonight but can attempt to tomorrow.  How has this been
> tested?

Indeed old->new and new->old migration wasn't tested AFAIK.  Not by me
at least.

Paolo

> I'm a little concerned here about the timing.  With the Christmas
> and New Years holiday we're pretty darn close to soft freeze for 1.4.
> 
> Has this series gone through a full autotest run with multiple guests?

Re: [Qemu-devel] [PULL 00/34] migration thread and queue

[Anthony Liguori]

Juan Quintela <quintela@redhat.com> writes:

> Hi
>
> Changes for yesterday:
> - Paolo Acked the series
> - Rebaso on top of today git (only conflicts were due to header re-shuffle)
>
> Please pull.

Pulled. Thanks.

Regards,

Anthony Liguori

>
> [20121219]
>
> This is my queue for migration-thread and patches associated.  This
> integrates review comments & code for Paolo.  This is the subset from
> both approachs that we agreed with. rest of patches need more review
> and are not here.
>
> Migrating and idle guest with upstwream:
>
> (qemu) info migrate
> capabilities: xbzrle: off
> Migration status: completed
> total time: 34251 milliseconds
> downtime: 492 milliseconds
> transferred ram: 762458 kbytes
> remaining ram: 0 kbytes
> total ram: 14688768 kbytes
> duplicate: 3492606 pages
> normal: 189762 pages
> normal bytes: 759048 kbytes
>
> with this series of patches.
>
> (qemu) info migrate
> capabilities: xbzrle: off
> Migration status: completed
> total time: 30712 milliseconds
> downtime: 29 milliseconds
> transferred ram: 738857 kbytes
> remaining ram: 0 kbytes
> total ram: 14688768 kbytes
> duplicate: 3503423 pages
> normal: 176671 pages
> normal bytes: 706684 kbytes
>
> Notice the big difference in downtime.  And that is also seen inside
> the guest a program that just do an idle loop seeing how "long" it
> takes to wait for 10ms.
>
> with upstream:
>
> [root@d1 ~]# ./timer
> delay of 452 ms
> delay of 114 ms
> delay of 136 ms
> delay of 135 ms
> delay of 136 ms
> delay of 131 ms
> delay of 134 ms
>
> with this series of patches, wait never takes 100ms, nothing is printed.
>
> Please pull.
>
> Thanks, Juan.
>
>
> The following changes since commit 27dd7730582be85c7d4f680f5f71146629809c86:
>
>   Merge remote-tracking branch 'bonzini/header-dirs' into staging (2012-12-19 17:15:39 -0600)
>
> are available in the git repository at:
>
>
>   git://repo.or.cz/qemu/quintela.git thread-20121220.next
>
> for you to fetch changes up to 9848a40427cd76628d04d918fa4751c542527915:
>
>   migration: merge QEMUFileBuffered into MigrationState (2012-12-20
>   23:09:40 +0100)
>
> ----------------------------------------------------------------
> Juan Quintela (25):
>       migration: include qemu-file.h
>       migration-fd: remove duplicate include
>       buffered_file: Move from using a timer to use a thread
>       migration: make qemu_fopen_ops_buffered() return void
>       migration: move migration thread init code to migrate_fd_put_ready
>       migration: make writes blocking
>       migration: remove unfreeze logic
>       migration: just lock migrate_fd_put_ready
>       buffered_file: Unfold the trick to restart generating migration data
>       buffered_file: don't flush on put buffer
>       buffered_file: unfold buffered_append in buffered_put_buffer
>       savevm: New save live migration method: pending
>       migration: move buffered_file.c code into migration.c
>       migration: add XFER_LIMIT_RATIO
>       migration: move migration_fd_put_ready()
>       migration: Inline qemu_fopen_ops_buffered into migrate_fd_connect
>       migration: move migration notifier
>       ram: rename last_block to last_seen_block
>       ram: Add last_sent_block
>       memory: introduce memory_region_test_and_clear_dirty
>       ram: Use memory_region_test_and_clear_dirty
>       ram: optimize migration bitmap walking
>       ram: account the amount of transferred ram better
>       ram: refactor ram_save_block() return value
>       migration: merge QEMUFileBuffered into MigrationState
>
> Paolo Bonzini (7):
>       migration: fix migration_bitmap leak
>       buffered_file: do not send more than s->bytes_xfer bytes per tick
>       migration: remove double call to migrate_fd_close
>       exec: change ramlist from MRU order to a 1-item cache
>       exec: change RAM list to a TAILQ
>       exec: sort the memory from biggest to smallest
>       migration: fix qemu_get_fd for BufferedFile
>
> Umesh Deshpande (2):
>       add a version number to ram_list
>       protect the ramlist with a separate mutex
>
>  Makefile.objs                 |   3 +-
>  arch_init.c                   | 245 +++++++++++++-------------
>  block-migration.c             |  49 ++----
>  buffered_file.c               | 268 -----------------------------
>  buffered_file.h               |  22 ---
>  dump.c                        |   8 +-
>  exec.c                        | 128 +++++++++-----
>  include/exec/cpu-all.h        |  15 +-
>  include/exec/memory.h         |  16 ++
>  include/migration/migration.h |  13 +-
>  include/migration/qemu-file.h |   5 -
>  include/migration/vmstate.h   |   1 +
>  include/sysemu/sysemu.h       |   1 +
>  memory.c                      |  16 ++
>  memory_mapping.c              |   4 +-
>  migration-exec.c              |   3 +-
>  migration-fd.c                |   4 +-
>  migration-tcp.c               |   3 +-
>  migration-unix.c              |   3 +-
>  migration.c                   | 390 +++++++++++++++++++++++++++++++-----------
>  savevm.c                      |  24 ++-
>  target-i386/arch_dump.c       |   2 +-
>  22 files changed, 600 insertions(+), 623 deletions(-)
>  delete mode 100644 buffered_file.c
>  delete mode 100644 buffered_file.h

[Qemu-devel] [PULL 00/34] migration thread and queue

[Juan Quintela]

Hi

Changes for last version:
- inlined paolo fixes, attached here (they were very small, and only change error cases locking)
- I tested with autotest and see no problems

Please pull.

Thanks, Juan.


diff --git a/arch_init.c b/arch_init.c
index 86f8544..ea75441 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -641,13 +641,13 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
         }
         i++;
     }
+    qemu_mutex_unlock_ramlist();

     if (ret < 0) {
         bytes_transferred += total_sent;
         return ret;
     }

-    qemu_mutex_unlock_ramlist();
     qemu_put_be64(f, RAM_SAVE_FLAG_EOS);
     total_sent += 8;
     bytes_transferred += total_sent;
@@ -657,9 +657,8 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)

 static int ram_save_complete(QEMUFile *f, void *opaque)
 {
-    migration_bitmap_sync();
-
     qemu_mutex_lock_ramlist();
+    migration_bitmap_sync();

     /* try transferring iterative blocks of memory */

diff --git a/migration.c b/migration.c
index c69e864..d6ec3e8 100644
--- a/migration.c
+++ b/migration.c
@@ -650,7 +650,7 @@ static int64_t buffered_set_rate_limit(void *opaque, int64_t new_rate)
         new_rate = SIZE_MAX;
     }

-    s->xfer_limit = new_rate / 10;
+    s->xfer_limit = new_rate / XFER_LIMIT_RATIO;

 out:
     return s->xfer_limit;


[20121220]
Changes for yesterday:
- Paolo Acked the series
- Rebase on top of today git (only conflicts were due to header re-shuffle)

Please pull.

[20121219]

This is my queue for migration-thread and patches associated.  This
integrates review comments & code for Paolo.  This is the subset from
both approachs that we agreed with. rest of patches need more review
and are not here.

Migrating and idle guest with upstwream:

(qemu) info migrate
capabilities: xbzrle: off
Migration status: completed
total time: 34251 milliseconds
downtime: 492 milliseconds
transferred ram: 762458 kbytes
remaining ram: 0 kbytes
total ram: 14688768 kbytes
duplicate: 3492606 pages
normal: 189762 pages
normal bytes: 759048 kbytes

with this series of patches.

(qemu) info migrate
capabilities: xbzrle: off
Migration status: completed
total time: 30712 milliseconds
downtime: 29 milliseconds
transferred ram: 738857 kbytes
remaining ram: 0 kbytes
total ram: 14688768 kbytes
duplicate: 3503423 pages
normal: 176671 pages
normal bytes: 706684 kbytes

Notice the big difference in downtime.  And that is also seen inside
the guest a program that just do an idle loop seeing how "long" it
takes to wait for 10ms.

with upstream:

[root@d1 ~]# ./timer
delay of 452 ms
delay of 114 ms
delay of 136 ms
delay of 135 ms
delay of 136 ms
delay of 131 ms
delay of 134 ms

with this series of patches, wait never takes 100ms, nothing is printed.

Please pull.

Thanks, Juan.

The following changes since commit 27dd7730582be85c7d4f680f5f71146629809c86:

  Merge remote-tracking branch 'bonzini/header-dirs' into staging (2012-12-19 17:15:39 -0600)

are available in the git repository at:


  git://repo.or.cz/qemu/quintela.git thread.next

for you to fetch changes up to 381c08083929f50f4780ea272ea36f7e5899b3b6:

  migration: merge QEMUFileBuffered into MigrationState (2012-12-21 20:01:24 +0100)

----------------------------------------------------------------
Juan Quintela (25):
      migration: include qemu-file.h
      migration-fd: remove duplicate include
      buffered_file: Move from using a timer to use a thread
      migration: make qemu_fopen_ops_buffered() return void
      migration: move migration thread init code to migrate_fd_put_ready
      migration: make writes blocking
      migration: remove unfreeze logic
      migration: just lock migrate_fd_put_ready
      buffered_file: Unfold the trick to restart generating migration data
      buffered_file: don't flush on put buffer
      buffered_file: unfold buffered_append in buffered_put_buffer
      savevm: New save live migration method: pending
      migration: move buffered_file.c code into migration.c
      migration: add XFER_LIMIT_RATIO
      migration: move migration_fd_put_ready()
      migration: Inline qemu_fopen_ops_buffered into migrate_fd_connect
      migration: move migration notifier
      ram: rename last_block to last_seen_block
      ram: Add last_sent_block
      memory: introduce memory_region_test_and_clear_dirty
      ram: Use memory_region_test_and_clear_dirty
      ram: optimize migration bitmap walking
      ram: account the amount of transferred ram better
      ram: refactor ram_save_block() return value
      migration: merge QEMUFileBuffered into MigrationState

Paolo Bonzini (7):
      migration: fix migration_bitmap leak
      buffered_file: do not send more than s->bytes_xfer bytes per tick
      migration: remove double call to migrate_fd_close
      exec: change ramlist from MRU order to a 1-item cache
      exec: change RAM list to a TAILQ
      exec: sort the memory from biggest to smallest
      migration: fix qemu_get_fd for BufferedFile

Umesh Deshpande (2):
      add a version number to ram_list
      protect the ramlist with a separate mutex

 Makefile.objs                 |   3 +-
 arch_init.c                   | 244 +++++++++++++-------------
 block-migration.c             |  49 ++----
 buffered_file.c               | 268 -----------------------------
 buffered_file.h               |  22 ---
 dump.c                        |   8 +-
 exec.c                        | 128 +++++++++-----
 include/exec/cpu-all.h        |  15 +-
 include/exec/memory.h         |  16 ++
 include/migration/migration.h |  13 +-
 include/migration/qemu-file.h |   5 -
 include/migration/vmstate.h   |   1 +
 include/sysemu/sysemu.h       |   1 +
 memory.c                      |  16 ++
 memory_mapping.c              |   4 +-
 migration-exec.c              |   3 +-
 migration-fd.c                |   4 +-
 migration-tcp.c               |   3 +-
 migration-unix.c              |   3 +-
 migration.c                   | 390 +++++++++++++++++++++++++++++++-----------
 savevm.c                      |  24 ++-
 target-i386/arch_dump.c       |   2 +-
 22 files changed, 599 insertions(+), 623 deletions(-)
 delete mode 100644 buffered_file.c
 delete mode 100644 buffered_file.h

Re: [Qemu-devel] [PULL 00/34] migration thread and queue

[Alexandre DERUMIER]

Hi,
I'm currently testing new migration code with last qemu.git,

it's working pretty fine (around 30ms downtime with standard workload).


But I have add some problem, with high memory workload vm. (playing an HD video for example).

Target vm is pause after migration,
# info status
VM status: paused (internal-error)

(downtime is around 700ms)


I can reproduce it 100%

Regards,

Alexandre Derumier


----- Mail original -----

De: "Juan Quintela" <quintela@redhat.com>
À: qemu-devel@nongnu.org
Cc: anthony@codemonkey.ws
Envoyé: Vendredi 21 Décembre 2012 20:41:03
Objet: [Qemu-devel] [PULL 00/34] migration thread and queue

Hi

Changes for last version:
- inlined paolo fixes, attached here (they were very small, and only change error cases locking)
- I tested with autotest and see no problems

Please pull.

Thanks, Juan.


diff --git a/arch_init.c b/arch_init.c
index 86f8544..ea75441 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -641,13 +641,13 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
}
i++;
}
+ qemu_mutex_unlock_ramlist();

if (ret < 0) {
bytes_transferred += total_sent;
return ret;
}

- qemu_mutex_unlock_ramlist();
qemu_put_be64(f, RAM_SAVE_FLAG_EOS);
total_sent += 8;
bytes_transferred += total_sent;
@@ -657,9 +657,8 @@ static int ram_save_iterate(QEMUFile *f, void *opaque) 

static int ram_save_complete(QEMUFile *f, void *opaque)
{
- migration_bitmap_sync();
-
qemu_mutex_lock_ramlist();
+ migration_bitmap_sync();

/* try transferring iterative blocks of memory */

diff --git a/migration.c b/migration.c
index c69e864..d6ec3e8 100644
--- a/migration.c
+++ b/migration.c
@@ -650,7 +650,7 @@ static int64_t buffered_set_rate_limit(void *opaque, int64_t new_rate)
new_rate = SIZE_MAX;
}

- s->xfer_limit = new_rate / 10;
+ s->xfer_limit = new_rate / XFER_LIMIT_RATIO;

out:
return s->xfer_limit;


[20121220]
Changes for yesterday:
- Paolo Acked the series
- Rebase on top of today git (only conflicts were due to header re-shuffle)

Please pull.

[20121219]

This is my queue for migration-thread and patches associated. This
integrates review comments & code for Paolo. This is the subset from
both approachs that we agreed with. rest of patches need more review
and are not here.

Migrating and idle guest with upstwream:

(qemu) info migrate
capabilities: xbzrle: off
Migration status: completed
total time: 34251 milliseconds
downtime: 492 milliseconds
transferred ram: 762458 kbytes
remaining ram: 0 kbytes
total ram: 14688768 kbytes
duplicate: 3492606 pages
normal: 189762 pages
normal bytes: 759048 kbytes

with this series of patches.

(qemu) info migrate
capabilities: xbzrle: off
Migration status: completed
total time: 30712 milliseconds
downtime: 29 milliseconds
transferred ram: 738857 kbytes
remaining ram: 0 kbytes
total ram: 14688768 kbytes
duplicate: 3503423 pages
normal: 176671 pages
normal bytes: 706684 kbytes

Notice the big difference in downtime. And that is also seen inside
the guest a program that just do an idle loop seeing how "long" it
takes to wait for 10ms.

with upstream:

[root@d1 ~]# ./timer
delay of 452 ms
delay of 114 ms
delay of 136 ms
delay of 135 ms
delay of 136 ms
delay of 131 ms
delay of 134 ms

with this series of patches, wait never takes 100ms, nothing is printed.

Please pull.

Thanks, Juan.

The following changes since commit 27dd7730582be85c7d4f680f5f71146629809c86:

Merge remote-tracking branch 'bonzini/header-dirs' into staging (2012-12-19 17:15:39 -0600)

are available in the git repository at:


git://repo.or.cz/qemu/quintela.git thread.next

for you to fetch changes up to 381c08083929f50f4780ea272ea36f7e5899b3b6:

migration: merge QEMUFileBuffered into MigrationState (2012-12-21 20:01:24 +0100)

----------------------------------------------------------------
Juan Quintela (25):
migration: include qemu-file.h
migration-fd: remove duplicate include
buffered_file: Move from using a timer to use a thread
migration: make qemu_fopen_ops_buffered() return void
migration: move migration thread init code to migrate_fd_put_ready
migration: make writes blocking
migration: remove unfreeze logic
migration: just lock migrate_fd_put_ready
buffered_file: Unfold the trick to restart generating migration data
buffered_file: don't flush on put buffer
buffered_file: unfold buffered_append in buffered_put_buffer
savevm: New save live migration method: pending
migration: move buffered_file.c code into migration.c
migration: add XFER_LIMIT_RATIO
migration: move migration_fd_put_ready()
migration: Inline qemu_fopen_ops_buffered into migrate_fd_connect
migration: move migration notifier
ram: rename last_block to last_seen_block
ram: Add last_sent_block
memory: introduce memory_region_test_and_clear_dirty
ram: Use memory_region_test_and_clear_dirty
ram: optimize migration bitmap walking
ram: account the amount of transferred ram better
ram: refactor ram_save_block() return value
migration: merge QEMUFileBuffered into MigrationState

Paolo Bonzini (7):
migration: fix migration_bitmap leak
buffered_file: do not send more than s->bytes_xfer bytes per tick
migration: remove double call to migrate_fd_close
exec: change ramlist from MRU order to a 1-item cache
exec: change RAM list to a TAILQ
exec: sort the memory from biggest to smallest
migration: fix qemu_get_fd for BufferedFile

Umesh Deshpande (2):
add a version number to ram_list
protect the ramlist with a separate mutex

Makefile.objs | 3 +-
arch_init.c | 244 +++++++++++++-------------
block-migration.c | 49 ++----
buffered_file.c | 268 -----------------------------
buffered_file.h | 22 ---
dump.c | 8 +-
exec.c | 128 +++++++++-----
include/exec/cpu-all.h | 15 +-
include/exec/memory.h | 16 ++
include/migration/migration.h | 13 +-
include/migration/qemu-file.h | 5 -
include/migration/vmstate.h | 1 +
include/sysemu/sysemu.h | 1 +
memory.c | 16 ++
memory_mapping.c | 4 +-
migration-exec.c | 3 +-
migration-fd.c | 4 +-
migration-tcp.c | 3 +-
migration-unix.c | 3 +-
migration.c | 390 +++++++++++++++++++++++++++++++-----------
savevm.c | 24 ++-
target-i386/arch_dump.c | 2 +-
22 files changed, 599 insertions(+), 623 deletions(-)
delete mode 100644 buffered_file.c
delete mode 100644 buffered_file.h

Re: [Qemu-devel] [PULL 00/34] migration thread and queue

[Paolo Bonzini]

Il 27/12/2012 16:30, Alexandre DERUMIER ha scritto:
> Hi,
> I'm currently testing new migration code with last qemu.git,
> 
> it's working pretty fine (around 30ms downtime with standard workload).
> 
> 
> But I have add some problem, with high memory workload vm. (playing an HD video for example).
> 
> Target vm is pause after migration, 
> # info status
> VM status: paused (internal-error)
> 
> (downtime is around 700ms)

1) What happened before these patches?  If something different, can you
bisect it?

2) Do you get anything on the console (stdout)?  See
kvm_handle_internal_error in kvm-all.c for what to expect.

Paolo

> I can reproduce it 100%
> 
> Regards,
> 
> Alexandre Derumier
> 
> 
> ----- Mail original ----- 
> 
> De: "Juan Quintela" <quintela@redhat.com> 
> À: qemu-devel@nongnu.org 
> Cc: anthony@codemonkey.ws 
> Envoyé: Vendredi 21 Décembre 2012 20:41:03 
> Objet: [Qemu-devel] [PULL 00/34] migration thread and queue 
> 
> Hi 
> 
> Changes for last version: 
> - inlined paolo fixes, attached here (they were very small, and only change error cases locking) 
> - I tested with autotest and see no problems 
> 
> Please pull. 
> 
> Thanks, Juan. 
> 
> 
> diff --git a/arch_init.c b/arch_init.c 
> index 86f8544..ea75441 100644 
> --- a/arch_init.c 
> +++ b/arch_init.c 
> @@ -641,13 +641,13 @@ static int ram_save_iterate(QEMUFile *f, void *opaque) 
> } 
> i++; 
> } 
> + qemu_mutex_unlock_ramlist(); 
> 
> if (ret < 0) { 
> bytes_transferred += total_sent; 
> return ret; 
> } 
> 
> - qemu_mutex_unlock_ramlist(); 
> qemu_put_be64(f, RAM_SAVE_FLAG_EOS); 
> total_sent += 8; 
> bytes_transferred += total_sent; 
> @@ -657,9 +657,8 @@ static int ram_save_iterate(QEMUFile *f, void *opaque) 
> 
> static int ram_save_complete(QEMUFile *f, void *opaque) 
> { 
> - migration_bitmap_sync(); 
> - 
> qemu_mutex_lock_ramlist(); 
> + migration_bitmap_sync(); 
> 
> /* try transferring iterative blocks of memory */ 
> 
> diff --git a/migration.c b/migration.c 
> index c69e864..d6ec3e8 100644 
> --- a/migration.c 
> +++ b/migration.c 
> @@ -650,7 +650,7 @@ static int64_t buffered_set_rate_limit(void *opaque, int64_t new_rate) 
> new_rate = SIZE_MAX; 
> } 
> 
> - s->xfer_limit = new_rate / 10; 
> + s->xfer_limit = new_rate / XFER_LIMIT_RATIO; 
> 
> out: 
> return s->xfer_limit; 
> 
> 
> [20121220] 
> Changes for yesterday: 
> - Paolo Acked the series 
> - Rebase on top of today git (only conflicts were due to header re-shuffle) 
> 
> Please pull. 
> 
> [20121219] 
> 
> This is my queue for migration-thread and patches associated. This 
> integrates review comments & code for Paolo. This is the subset from 
> both approachs that we agreed with. rest of patches need more review 
> and are not here. 
> 
> Migrating and idle guest with upstwream: 
> 
> (qemu) info migrate 
> capabilities: xbzrle: off 
> Migration status: completed 
> total time: 34251 milliseconds 
> downtime: 492 milliseconds 
> transferred ram: 762458 kbytes 
> remaining ram: 0 kbytes 
> total ram: 14688768 kbytes 
> duplicate: 3492606 pages 
> normal: 189762 pages 
> normal bytes: 759048 kbytes 
> 
> with this series of patches. 
> 
> (qemu) info migrate 
> capabilities: xbzrle: off 
> Migration status: completed 
> total time: 30712 milliseconds 
> downtime: 29 milliseconds 
> transferred ram: 738857 kbytes 
> remaining ram: 0 kbytes 
> total ram: 14688768 kbytes 
> duplicate: 3503423 pages 
> normal: 176671 pages 
> normal bytes: 706684 kbytes 
> 
> Notice the big difference in downtime. And that is also seen inside 
> the guest a program that just do an idle loop seeing how "long" it 
> takes to wait for 10ms. 
> 
> with upstream: 
> 
> [root@d1 ~]# ./timer 
> delay of 452 ms 
> delay of 114 ms 
> delay of 136 ms 
> delay of 135 ms 
> delay of 136 ms 
> delay of 131 ms 
> delay of 134 ms 
> 
> with this series of patches, wait never takes 100ms, nothing is printed. 
> 
> Please pull. 
> 
> Thanks, Juan. 
> 
> The following changes since commit 27dd7730582be85c7d4f680f5f71146629809c86: 
> 
> Merge remote-tracking branch 'bonzini/header-dirs' into staging (2012-12-19 17:15:39 -0600) 
> 
> are available in the git repository at: 
> 
> 
> git://repo.or.cz/qemu/quintela.git thread.next 
> 
> for you to fetch changes up to 381c08083929f50f4780ea272ea36f7e5899b3b6: 
> 
> migration: merge QEMUFileBuffered into MigrationState (2012-12-21 20:01:24 +0100) 
> 
> ---------------------------------------------------------------- 
> Juan Quintela (25): 
> migration: include qemu-file.h 
> migration-fd: remove duplicate include 
> buffered_file: Move from using a timer to use a thread 
> migration: make qemu_fopen_ops_buffered() return void 
> migration: move migration thread init code to migrate_fd_put_ready 
> migration: make writes blocking 
> migration: remove unfreeze logic 
> migration: just lock migrate_fd_put_ready 
> buffered_file: Unfold the trick to restart generating migration data 
> buffered_file: don't flush on put buffer 
> buffered_file: unfold buffered_append in buffered_put_buffer 
> savevm: New save live migration method: pending 
> migration: move buffered_file.c code into migration.c 
> migration: add XFER_LIMIT_RATIO 
> migration: move migration_fd_put_ready() 
> migration: Inline qemu_fopen_ops_buffered into migrate_fd_connect 
> migration: move migration notifier 
> ram: rename last_block to last_seen_block 
> ram: Add last_sent_block 
> memory: introduce memory_region_test_and_clear_dirty 
> ram: Use memory_region_test_and_clear_dirty 
> ram: optimize migration bitmap walking 
> ram: account the amount of transferred ram better 
> ram: refactor ram_save_block() return value 
> migration: merge QEMUFileBuffered into MigrationState 
> 
> Paolo Bonzini (7): 
> migration: fix migration_bitmap leak 
> buffered_file: do not send more than s->bytes_xfer bytes per tick 
> migration: remove double call to migrate_fd_close 
> exec: change ramlist from MRU order to a 1-item cache 
> exec: change RAM list to a TAILQ 
> exec: sort the memory from biggest to smallest 
> migration: fix qemu_get_fd for BufferedFile 
> 
> Umesh Deshpande (2): 
> add a version number to ram_list 
> protect the ramlist with a separate mutex 
> 
> Makefile.objs | 3 +- 
> arch_init.c | 244 +++++++++++++------------- 
> block-migration.c | 49 ++---- 
> buffered_file.c | 268 ----------------------------- 
> buffered_file.h | 22 --- 
> dump.c | 8 +- 
> exec.c | 128 +++++++++----- 
> include/exec/cpu-all.h | 15 +- 
> include/exec/memory.h | 16 ++ 
> include/migration/migration.h | 13 +- 
> include/migration/qemu-file.h | 5 - 
> include/migration/vmstate.h | 1 + 
> include/sysemu/sysemu.h | 1 + 
> memory.c | 16 ++ 
> memory_mapping.c | 4 +- 
> migration-exec.c | 3 +- 
> migration-fd.c | 4 +- 
> migration-tcp.c | 3 +- 
> migration-unix.c | 3 +- 
> migration.c | 390 +++++++++++++++++++++++++++++++----------- 
> savevm.c | 24 ++- 
> target-i386/arch_dump.c | 2 +- 
> 22 files changed, 599 insertions(+), 623 deletions(-) 
> delete mode 100644 buffered_file.c 
> delete mode 100644 buffered_file.h 
> 
> 

Re: [Qemu-devel] [PULL 00/34] migration thread and queue

[Alexandre DERUMIER]

>>1) What happened before these patches? If something different, can you
>>bisect it?
Well, qemu 1.3 stable release, was working slowly (bigger downtime), but no crash.
I don't see any changes since 1.3 and theses patches.


>>2) Do you get anything on the console (stdout)? See
>>kvm_handle_internal_error in kvm-all.c for what to expect.
I'll have a look at this.
Currently I start the target vm with --daemonize, does I need to remove this option to see stdout ?


----- Mail original -----

De: "Paolo Bonzini" <pbonzini@redhat.com>
À: "Alexandre DERUMIER" <aderumier@odiso.com>
Cc: "Juan Quintela" <quintela@redhat.com>, qemu-devel@nongnu.org, anthony@codemonkey.ws
Envoyé: Jeudi 27 Décembre 2012 16:44:40
Objet: Re: [PULL 00/34] migration thread and queue

Il 27/12/2012 16:30, Alexandre DERUMIER ha scritto:
> Hi,
> I'm currently testing new migration code with last qemu.git,
>
> it's working pretty fine (around 30ms downtime with standard workload).
>
>
> But I have add some problem, with high memory workload vm. (playing an HD video for example).
>
> Target vm is pause after migration,
> # info status
> VM status: paused (internal-error)
>
> (downtime is around 700ms)

1) What happened before these patches? If something different, can you
bisect it?

2) Do you get anything on the console (stdout)? See
kvm_handle_internal_error in kvm-all.c for what to expect.

Paolo

> I can reproduce it 100%
>
> Regards,
>
> Alexandre Derumier
>
>
> ----- Mail original -----
>
> De: "Juan Quintela" <quintela@redhat.com>
> À: qemu-devel@nongnu.org
> Cc: anthony@codemonkey.ws
> Envoyé: Vendredi 21 Décembre 2012 20:41:03
> Objet: [Qemu-devel] [PULL 00/34] migration thread and queue
>
> Hi
>
> Changes for last version:
> - inlined paolo fixes, attached here (they were very small, and only change error cases locking)
> - I tested with autotest and see no problems
>
> Please pull.
>
> Thanks, Juan.
>
>
> diff --git a/arch_init.c b/arch_init.c
> index 86f8544..ea75441 100644
> --- a/arch_init.c
> +++ b/arch_init.c
> @@ -641,13 +641,13 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
> }
> i++;
> }
> + qemu_mutex_unlock_ramlist();
>
> if (ret < 0) {
> bytes_transferred += total_sent;
> return ret;
> }
>
> - qemu_mutex_unlock_ramlist();
> qemu_put_be64(f, RAM_SAVE_FLAG_EOS);
> total_sent += 8;
> bytes_transferred += total_sent;
> @@ -657,9 +657,8 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
>
> static int ram_save_complete(QEMUFile *f, void *opaque)
> {
> - migration_bitmap_sync();
> -
> qemu_mutex_lock_ramlist();
> + migration_bitmap_sync();
>
> /* try transferring iterative blocks of memory */
>
> diff --git a/migration.c b/migration.c
> index c69e864..d6ec3e8 100644
> --- a/migration.c
> +++ b/migration.c
> @@ -650,7 +650,7 @@ static int64_t buffered_set_rate_limit(void *opaque, int64_t new_rate)
> new_rate = SIZE_MAX;
> }
>
> - s->xfer_limit = new_rate / 10;
> + s->xfer_limit = new_rate / XFER_LIMIT_RATIO;
>
> out:
> return s->xfer_limit;
>
>
> [20121220]
> Changes for yesterday:
> - Paolo Acked the series
> - Rebase on top of today git (only conflicts were due to header re-shuffle)
>
> Please pull.
>
> [20121219]
>
> This is my queue for migration-thread and patches associated. This
> integrates review comments & code for Paolo. This is the subset from
> both approachs that we agreed with. rest of patches need more review
> and are not here.
>
> Migrating and idle guest with upstwream:
>
> (qemu) info migrate
> capabilities: xbzrle: off
> Migration status: completed
> total time: 34251 milliseconds
> downtime: 492 milliseconds
> transferred ram: 762458 kbytes
> remaining ram: 0 kbytes
> total ram: 14688768 kbytes
> duplicate: 3492606 pages
> normal: 189762 pages
> normal bytes: 759048 kbytes
>
> with this series of patches.
>
> (qemu) info migrate
> capabilities: xbzrle: off
> Migration status: completed
> total time: 30712 milliseconds
> downtime: 29 milliseconds
> transferred ram: 738857 kbytes
> remaining ram: 0 kbytes
> total ram: 14688768 kbytes
> duplicate: 3503423 pages
> normal: 176671 pages
> normal bytes: 706684 kbytes
>
> Notice the big difference in downtime. And that is also seen inside
> the guest a program that just do an idle loop seeing how "long" it
> takes to wait for 10ms.
>
> with upstream:
>
> [root@d1 ~]# ./timer
> delay of 452 ms
> delay of 114 ms
> delay of 136 ms
> delay of 135 ms
> delay of 136 ms
> delay of 131 ms
> delay of 134 ms
>
> with this series of patches, wait never takes 100ms, nothing is printed. 
>
> Please pull.
>
> Thanks, Juan.
>
> The following changes since commit 27dd7730582be85c7d4f680f5f71146629809c86:
>
> Merge remote-tracking branch 'bonzini/header-dirs' into staging (2012-12-19 17:15:39 -0600)
>
> are available in the git repository at:
>
>
> git://repo.or.cz/qemu/quintela.git thread.next
>
> for you to fetch changes up to 381c08083929f50f4780ea272ea36f7e5899b3b6: 
>
> migration: merge QEMUFileBuffered into MigrationState (2012-12-21 20:01:24 +0100)
>
> ----------------------------------------------------------------
> Juan Quintela (25):
> migration: include qemu-file.h
> migration-fd: remove duplicate include
> buffered_file: Move from using a timer to use a thread
> migration: make qemu_fopen_ops_buffered() return void
> migration: move migration thread init code to migrate_fd_put_ready
> migration: make writes blocking
> migration: remove unfreeze logic
> migration: just lock migrate_fd_put_ready
> buffered_file: Unfold the trick to restart generating migration data
> buffered_file: don't flush on put buffer
> buffered_file: unfold buffered_append in buffered_put_buffer
> savevm: New save live migration method: pending
> migration: move buffered_file.c code into migration.c
> migration: add XFER_LIMIT_RATIO
> migration: move migration_fd_put_ready()
> migration: Inline qemu_fopen_ops_buffered into migrate_fd_connect
> migration: move migration notifier
> ram: rename last_block to last_seen_block
> ram: Add last_sent_block
> memory: introduce memory_region_test_and_clear_dirty
> ram: Use memory_region_test_and_clear_dirty
> ram: optimize migration bitmap walking
> ram: account the amount of transferred ram better
> ram: refactor ram_save_block() return value
> migration: merge QEMUFileBuffered into MigrationState
>
> Paolo Bonzini (7):
> migration: fix migration_bitmap leak
> buffered_file: do not send more than s->bytes_xfer bytes per tick
> migration: remove double call to migrate_fd_close
> exec: change ramlist from MRU order to a 1-item cache
> exec: change RAM list to a TAILQ
> exec: sort the memory from biggest to smallest
> migration: fix qemu_get_fd for BufferedFile
>
> Umesh Deshpande (2):
> add a version number to ram_list
> protect the ramlist with a separate mutex
>
> Makefile.objs | 3 +-
> arch_init.c | 244 +++++++++++++-------------
> block-migration.c | 49 ++----
> buffered_file.c | 268 -----------------------------
> buffered_file.h | 22 ---
> dump.c | 8 +-
> exec.c | 128 +++++++++-----
> include/exec/cpu-all.h | 15 +-
> include/exec/memory.h | 16 ++
> include/migration/migration.h | 13 +-
> include/migration/qemu-file.h | 5 -
> include/migration/vmstate.h | 1 +
> include/sysemu/sysemu.h | 1 +
> memory.c | 16 ++
> memory_mapping.c | 4 +-
> migration-exec.c | 3 +-
> migration-fd.c | 4 +-
> migration-tcp.c | 3 +-
> migration-unix.c | 3 +-
> migration.c | 390 +++++++++++++++++++++++++++++++-----------
> savevm.c | 24 ++-
> target-i386/arch_dump.c | 2 +-
> 22 files changed, 599 insertions(+), 623 deletions(-)
> delete mode 100644 buffered_file.c
> delete mode 100644 buffered_file.h
>
>

Re: [Qemu-devel] [PULL 00/34] migration thread and queue

[Paolo Bonzini]

Il 27/12/2012 16:51, Alexandre DERUMIER ha scritto:
>>> 1) What happened before these patches? If something different, can you 
>>> bisect it? 
> Well, qemu 1.3 stable release, was working slowly (bigger downtime), but no crash.
> I don't see any changes since 1.3 and theses patches.

These patches are 34. :)  Bisecting among them will already be of huge help.

>>> 2) Do you get anything on the console (stdout)? See 
>>> kvm_handle_internal_error in kvm-all.c for what to expect. 
> I'll have a look at this. 
> Currently I start the target vm with --daemonize, does I need to remove this option to see stdout ?

Yes, please do that so that you can also bisect the problem.

BTW, what kernel version?  Does it reproduce with

    migrate exec:cat>foo.save

and then, when migration finishes, loading it with "-incoming 'exec:cat
foo.save'"?

Paolo

Re: [Qemu-devel] [PULL 00/34] migration thread and queue

[Alexandre DERUMIER]

Ok,I'll try to bisect it tomorrow and will do more tests.

I'll keep you in touch!

Alexandre


----- Mail original -----

De: "Paolo Bonzini" <pbonzini@redhat.com>
À: "Alexandre DERUMIER" <aderumier@odiso.com>
Cc: qemu-devel@nongnu.org, anthony@codemonkey.ws, "Juan Quintela" <quintela@redhat.com>
Envoyé: Jeudi 27 Décembre 2012 17:08:23
Objet: Re: [Qemu-devel] [PULL 00/34] migration thread and queue

Il 27/12/2012 16:51, Alexandre DERUMIER ha scritto:
>>> 1) What happened before these patches? If something different, can you 
>>> bisect it?
> Well, qemu 1.3 stable release, was working slowly (bigger downtime), but no crash.
> I don't see any changes since 1.3 and theses patches.

These patches are 34. :) Bisecting among them will already be of huge help.

>>> 2) Do you get anything on the console (stdout)? See
>>> kvm_handle_internal_error in kvm-all.c for what to expect.
> I'll have a look at this.
> Currently I start the target vm with --daemonize, does I need to remove this option to see stdout ?

Yes, please do that so that you can also bisect the problem.

BTW, what kernel version? Does it reproduce with

migrate exec:cat>foo.save

and then, when migration finishes, loading it with "-incoming 'exec:cat
foo.save'"?

Paolo