[Qemu-devel] [PULL 00/22] migration queue

[Qemu-devel] [PULL 00/22] migration queue

[Juan Quintela]

Hi Anthony

This pull request include:

- Peter Lieven zero page optimizations
- David Gibson new VMSTATE macros
- Orit asynchronous writes.
- migration documentation (me)

All of them have had several rounds through qemu-devel@

Please, pull.

Later, Juan.


The following changes since commit dcadaa9b40d6019ac18d6fd7763d43048ef79218:

  Merge remote-tracking branch 'stefanha/net' into staging (2013-03-25 13:14:26 -0500)

are available in the git repository at:


  git://github.com/juanquintela/qemu.git migration.next

for you to fetch changes up to 500f0061d628b52220038939728f0d7aee634468:

  Use qemu_put_buffer_async for guest memory pages (2013-03-26 13:32:33 +0100)

----------------------------------------------------------------
David Gibson (5):
      savevm: Add VMSTATE_UINT64_EQUAL helpers
      savevm: Add VMSTATE_UINTTL_EQUAL helper
      savevm: Add VMSTATE_FLOAT64 helpers
      savevm: Add VMSTATE_STRUCT_VARRAY_POINTER_UINT32
      savevm: Fix bugs in the VMSTATE_VBUFFER_MULTIPLY definition

Juan Quintela (1):
      migration: Improve QMP documentation

Orit Wasserman (7):
      Add QemuFileWritevBuffer QemuFileOps
      Add socket_writev_buffer function
      Update bytes_xfer in qemu_put_byte
      Store the data to send also in iovec
      Use writev ops if available
      Add qemu_put_buffer_async
      Use qemu_put_buffer_async for guest memory pages

Peter Lieven (9):
      move vector definitions to qemu-common.h
      cutils: add a function to find non-zero content in a buffer
      buffer_is_zero: use vector optimizations if possible
      bitops: unroll while loop in find_next_bit()
      migration: search for zero instead of dup pages
      migration: add an indicator for bulk state of ram migration
      migration: do not sent zero pages in bulk stage
      migration: do not search dirty pages in bulk stage
      migration: use XBZRLE only after bulk stage

 arch_init.c                   |  76 +++++++++++-----------
 hmp.c                         |   2 +
 hw/hw.h                       |   6 ++
 include/migration/migration.h |   2 +
 include/migration/qemu-file.h |  12 ++++
 include/migration/vmstate.h   |  43 +++++++++++--
 include/qemu-common.h         |  31 +++++++++
 migration.c                   |   3 +-
 qapi-schema.json              |   8 ++-
 qmp-commands.hx               |  55 +++++++++-------
 savevm.c                      | 146 +++++++++++++++++++++++++++++++++++++-----
 util/bitops.c                 |  18 +++++-
 util/cutils.c                 |  60 +++++++++++++++++
 13 files changed, 375 insertions(+), 87 deletions(-)

[Qemu-devel] [PULL 00/22] Migration queue

[Juan Quintela]

Hi

Peter, please pull:


- RDMA bugfixes (Michael)
- Static checker from Amit (v5)
- acked patches from vmstate (thanks Dave)

Later, Juan.


The following changes since commit d9c1647d896d3192cba9dbf98fb7efab876edde5:

  Merge remote-tracking branch 'remotes/stefanha/tags/block-pull-request' into staging (2014-06-23 12:55:22 +0100)

are available in the git repository at:


  git://github.com/juanquintela/qemu.git tags/migration/20140623

for you to fetch changes up to 4ea7df4e5c6b9d8b8a474680d6b1687fc8eacbac:

  vmstate: Refactor & increase tests for primitive types (2014-06-23 19:14:52 +0200)

----------------------------------------------------------------
migration/next for 20140623

----------------------------------------------------------------
Amit Shah (18):
      migration: dump vmstate info as a json file for static analysis
      vmstate-static-checker: script to validate vmstate changes
      tests: vmstate static checker: add dump1 and dump2 files
      tests: vmstate static checker: incompat machine types
      tests: vmstate static checker: add version error in main section
      tests: vmstate static checker: version mismatch inside a Description
      tests: vmstate static checker: minimum_version_id check
      tests: vmstate static checker: remove a section
      tests: vmstate static checker: remove a field
      tests: vmstate static checker: remove last field in a struct
      tests: vmstate static checker: change description name
      tests: vmstate static checker: remove Fields
      tests: vmstate static checker: remove Description
      tests: vmstate static checker: remove Description inside Fields
      tests: vmstate static checker: remove a subsection
      tests: vmstate static checker: remove Subsections
      tests: vmstate static checker: add substructure for usb-kbd for hid section
      tests: vmstate static checker: add size mismatch inside substructure

Juan Quintela (3):
      migration: Remove unneeded minimum_version_id_old
      vmstate: Return error in case of error
      vmstate: Refactor & increase tests for primitive types

Michael R. Hines (1):
      rdma: bug fixes

 hw/usb/hcd-ohci.c                            |    7 +-
 include/migration/vmstate.h                  |    2 +
 migration-rdma.c                             |   20 +-
 qemu-options.hx                              |   14 +
 savevm.c                                     |  139 +++
 scripts/vmstate-static-checker.py            |  345 ++++++++
 tests/test-vmstate.c                         |  273 ++++--
 tests/vmstate-static-checker-data/dump1.json | 1163 ++++++++++++++++++++++++++
 tests/vmstate-static-checker-data/dump2.json |  968 +++++++++++++++++++++
 vl.c                                         |   13 +
 vmstate.c                                    |    4 +
 11 files changed, 2872 insertions(+), 76 deletions(-)
 create mode 100755 scripts/vmstate-static-checker.py
 create mode 100644 tests/vmstate-static-checker-data/dump1.json
 create mode 100644 tests/vmstate-static-checker-data/dump2.json

Re: [Qemu-devel] [PULL 00/22] Migration queue

[Peter Maydell]

On 24 June 2014 14:09, Juan Quintela <quintela@redhat.com> wrote:
> Hi
>
> Peter, please pull:
>
>
> - RDMA bugfixes (Michael)
> - Static checker from Amit (v5)
> - acked patches from vmstate (thanks Dave)
>
> Later, Juan.
>
>
> The following changes since commit d9c1647d896d3192cba9dbf98fb7efab876edde5:
>
>   Merge remote-tracking branch 'remotes/stefanha/tags/block-pull-request' into staging (2014-06-23 12:55:22 +0100)
>
> are available in the git repository at:
>
>
>   git://github.com/juanquintela/qemu.git tags/migration/20140623
>
> for you to fetch changes up to 4ea7df4e5c6b9d8b8a474680d6b1687fc8eacbac:
>
>   vmstate: Refactor & increase tests for primitive types (2014-06-23 19:14:52 +0200)

> ----------------------------------------------------------------
> migration/next for 20140623

Applied, thanks.

-- PMM

[Qemu-devel] [PULL 00/22] migration queue

[Dr. David Alan Gilbert (git)]

From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>

The following changes since commit b5b6b2b912bbcd3953407da938a8f969577ad3a1:

  Merge remote-tracking branch 'remotes/amarkovic/tags/mips-queue-mar-05-2019' into staging (2019-03-05 21:07:29 +0000)

are available in the Git repository at:

  git://github.com/dagrh/qemu.git tags/pull-migration-20190306a

for you to fetch changes up to b5922fc5891261153f1a0f20e814c620aabeb6ac:

  qapi/migration.json: Remove a variable that doesn't exist in example (2019-03-06 10:49:18 +0000)

----------------------------------------------------------------
Migation pull 2019-03-06

(This replaces the pull sent yesterday)

   a) 4 small fixes including the cancel problem
     that caused the ahci migration test to fail
     intermittently
   b) Yury's ignore-shared feature
   c) Juan's extra tests
   d) Wei Wang's free page hinting
   e) Some Colo fixes from Zhang Chen

Diff from yesterdays pull:
  1) A missing fix of mine (cleanup during exit)
  2) Changes from Eric/Markus on 'Create socket-address parameter'

----------------------------------------------------------------
Dr. David Alan Gilbert (3):
      migration: Fix cancel state
      migration/rdma: Fix qemu_rdma_cleanup null check
      migration: Cleanup during exit

Juan Quintela (3):
      tests: Add migration xbzrle test
      migration: Create socket-address parameter
      tests: Add basic migration precopy tcp test

Marcel Apfelbaum (1):
      migration/rdma: clang compilation fix

Wei Wang (7):
      bitmap: fix bitmap_count_one
      bitmap: bitmap_count_one_with_offset
      migration: use bitmap_mutex in migration_bitmap_clear_dirty
      migration: API to clear bits of guest free pages from the dirty bitmap
      migration/ram.c: add a notifier chain for precopy
      migration/ram.c: add the free page optimization enable flag
      virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_HINT

Yury Kotov (5):
      exec: Change RAMBlockIterFunc definition
      migration: Introduce ignore-shared capability
      migration: Add an ability to ignore shared RAM blocks
      tests/migration-test: Add a test for ignore-shared capability
      migration: Add capabilities validation

Zhang Chen (3):
      Migration/colo.c: Fix double close bug when occur COLO failover
      Migration/colo.c: Make COLO node running after failover
      qapi/migration.json: Remove a variable that doesn't exist in example

 exec.c                             |  38 ++---
 hmp.c                              |  33 ++++
 hw/virtio/virtio-balloon.c         | 263 +++++++++++++++++++++++++++++++
 include/exec/cpu-common.h          |   7 +-
 include/hw/virtio/virtio-balloon.h |  28 +++-
 include/migration/misc.h           |  24 ++-
 include/qemu/bitmap.h              |  17 ++
 migration/colo.c                   |   2 +-
 migration/migration.c              |  62 +++++++-
 migration/migration.h              |   9 +-
 migration/postcopy-ram.c           |  48 +++---
 migration/ram.c                    | 231 ++++++++++++++++++++++++----
 migration/rdma.c                   |  18 ++-
 migration/savevm.c                 | 152 ++++++++++++++++++
 migration/socket.c                 |  11 ++
 qapi/migration.json                |  13 +-
 stubs/ram-block.c                  |  15 ++
 tests/migration-test.c             | 308 +++++++++++++++++++++++++++++++++----
 util/vfio-helpers.c                |   6 +-
 vl.c                               |   8 +-
 20 files changed, 1167 insertions(+), 126 deletions(-)

Re: [Qemu-devel] [PULL 00/22] migration queue

[no-reply]

Patchew URL: https://patchew.org/QEMU/20190306114227.9125-1-dgilbert@redhat.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: 20190306114227.9125-1-dgilbert@redhat.com
Subject: [Qemu-devel] [PULL 00/22] migration queue

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 * [new tag]               patchew/20190306114227.9125-1-dgilbert@redhat.com -> patchew/20190306114227.9125-1-dgilbert@redhat.com
Switched to a new branch 'test'
060f501bdc qapi/migration.json: Remove a variable that doesn't exist in example
79985c89a1 Migration/colo.c: Make COLO node running after failover
04fa41f3fe Migration/colo.c: Fix double close bug when occur COLO failover
fad6a7a2fe virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_HINT
ea055843a6 migration/ram.c: add the free page optimization enable flag
04f12f4493 migration/ram.c: add a notifier chain for precopy
acffe8fac2 migration: API to clear bits of guest free pages from the dirty bitmap
28ba7920b5 migration: use bitmap_mutex in migration_bitmap_clear_dirty
50d141e21b bitmap: bitmap_count_one_with_offset
d09d8c34fe bitmap: fix bitmap_count_one
0b3f876f73 tests: Add basic migration precopy tcp test
b9b8858224 migration: Create socket-address parameter
c3e10e38d0 tests: Add migration xbzrle test
9891a178e1 migration: Add capabilities validation
a3ba078ea3 tests/migration-test: Add a test for ignore-shared capability
40a11f6d72 migration: Add an ability to ignore shared RAM blocks
b08a8bcb8f migration: Introduce ignore-shared capability
502f3e7690 exec: Change RAMBlockIterFunc definition
1419fc4f68 migration/rdma: clang compilation fix
af4a8e34f7 migration: Cleanup during exit
6078675984 migration/rdma: Fix qemu_rdma_cleanup null check
f414a0d58c migration: Fix cancel state

=== OUTPUT BEGIN ===
1/22 Checking commit f414a0d58c61 (migration: Fix cancel state)
2/22 Checking commit 60786759846a (migration/rdma: Fix qemu_rdma_cleanup null check)
3/22 Checking commit af4a8e34f76a (migration: Cleanup during exit)
4/22 Checking commit 1419fc4f68a6 (migration/rdma: clang compilation fix)
5/22 Checking commit 502f3e769033 (exec: Change RAMBlockIterFunc definition)
6/22 Checking commit b08a8bcb8f55 (migration: Introduce ignore-shared capability)
7/22 Checking commit 40a11f6d720b (migration: Add an ability to ignore shared RAM blocks)
ERROR: Macros with multiple statements should be enclosed in a do - while loop
#149: FILE: migration/ram.c:169:
+#define RAMBLOCK_FOREACH_NOT_IGNORED(block)            \
+    INTERNAL_RAMBLOCK_FOREACH(block)                   \
+        if (ramblock_is_ignored(block)) {} else

ERROR: trailing statements should be on next line
#151: FILE: migration/ram.c:171:
+        if (ramblock_is_ignored(block)) {} else

total: 2 errors, 0 warnings, 386 lines checked

Patch 7/22 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

8/22 Checking commit a3ba078ea311 (tests/migration-test: Add a test for ignore-shared capability)
ERROR: if this code is redundant consider removing it
#235: FILE: tests/migration-test.c:829:
+#if 0

total: 1 errors, 0 warnings, 255 lines checked

Patch 8/22 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

9/22 Checking commit 9891a178e1cb (migration: Add capabilities validation)
10/22 Checking commit c3e10e38d0a5 (tests: Add migration xbzrle test)
11/22 Checking commit b9b885822408 (migration: Create socket-address parameter)
12/22 Checking commit 0b3f876f7352 (tests: Add basic migration precopy tcp test)
13/22 Checking commit d09d8c34fe5f (bitmap: fix bitmap_count_one)
14/22 Checking commit 50d141e21bda (bitmap: bitmap_count_one_with_offset)
15/22 Checking commit 28ba7920b5e4 (migration: use bitmap_mutex in migration_bitmap_clear_dirty)
16/22 Checking commit acffe8fac2f4 (migration: API to clear bits of guest free pages from the dirty bitmap)
17/22 Checking commit 04f12f44931e (migration/ram.c: add a notifier chain for precopy)
18/22 Checking commit ea055843a6f6 (migration/ram.c: add the free page optimization enable flag)
19/22 Checking commit fad6a7a2fe5e (virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_HINT)
20/22 Checking commit 04fa41f3fed6 (Migration/colo.c: Fix double close bug when occur COLO failover)
21/22 Checking commit 79985c89a1be (Migration/colo.c: Make COLO node running after failover)
22/22 Checking commit 060f501bdcb2 (qapi/migration.json: Remove a variable that doesn't exist in example)
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20190306114227.9125-1-dgilbert@redhat.com/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [Qemu-devel] [PULL 00/22] migration queue

[Peter Maydell]

On Wed, 6 Mar 2019 at 11:44, Dr. David Alan Gilbert (git)
<dgilbert@redhat.com> wrote:
>
> From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
>
> The following changes since commit b5b6b2b912bbcd3953407da938a8f969577ad3a1:
>
>   Merge remote-tracking branch 'remotes/amarkovic/tags/mips-queue-mar-05-2019' into staging (2019-03-05 21:07:29 +0000)
>
> are available in the Git repository at:
>
>   git://github.com/dagrh/qemu.git tags/pull-migration-20190306a
>
> for you to fetch changes up to b5922fc5891261153f1a0f20e814c620aabeb6ac:
>
>   qapi/migration.json: Remove a variable that doesn't exist in example (2019-03-06 10:49:18 +0000)
>
> ----------------------------------------------------------------
> Migation pull 2019-03-06
>
> (This replaces the pull sent yesterday)
>
>    a) 4 small fixes including the cancel problem
>      that caused the ahci migration test to fail
>      intermittently
>    b) Yury's ignore-shared feature
>    c) Juan's extra tests
>    d) Wei Wang's free page hinting
>    e) Some Colo fixes from Zhang Chen
>
> Diff from yesterdays pull:
>   1) A missing fix of mine (cleanup during exit)
>   2) Changes from Eric/Markus on 'Create socket-address parameter'
>
> ----------------------------------------------------------------

Applied, thanks.

Please update the changelog at https://wiki.qemu.org/ChangeLog/4.0
for any user-visible changes.

-- PMM