All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v20 00/15] virtio-net: live-TAP local migration
@ 2026-07-29  9:12 Vladimir Sementsov-Ogievskiy
  2026-07-29  9:12 ` [PATCH v20 01/15] net/tap: rework tap_parse_script Vladimir Sementsov-Ogievskiy
                   ` (14 more replies)
  0 siblings, 15 replies; 22+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2026-07-29  9:12 UTC (permalink / raw)
  To: jasowang, mst
  Cc: armbru, peterx, farosas, raphael.s.norwitz, bchaney, vsementsov,
	qemu-devel, berrange, pbonzini, yc-core, mark.caveayland

Hi all!

Here is a migration for TAP net backend, including its properties and
open fds.

With this new feature, management software doesn't need to initialize
new TAP and do a switch to it. Nothing should be done around
virtio-net in local migration: it just migrates and continues to use
same TAP device. So we avoid extra logic in management software, extra
allocations in kernel (for new TAP), and corresponding extra delay in
migration downtime.

v20:

Main changes:

 - mark the new API unstable for now
 - support CPR -- actually only drop UNIX-socket requirement
   when mode is cpr-exec both in documentation (09) and check (10)

Details:

01: - update commit message
    - rename script_arg -> script_arg_value, to simplify 03
    - keep r-bs by Markus and Ben
02: add r-b by Ben
03: - improve warn_report() wording
    - synchronize QAPI and cmdline documentation
    - keep r-bs by Ben and Markus
06-08: add r-b by Ben
09: - drop "on the same OS" from @local documentation [Markus]
    - mention cpr-exec mode
    - mark @local as unstable (to be consistent
      with x-local-migration-supported, which was discussed to be
      unstable, seems better to keep the complete new API as unstable
      for a while)
    - mention why not simply detect "is UNIX socket" in commit message
    - drop r-b/a-b
10: - skip UNIX-socket requirement for cpr-exec migration mode
    - drop r-b
11: add r-b by Ben
12: - rework the patch after discussion with Ben
    - drop r-b
13: - rename local-migration-supported -> x-local-migration-supported and
      mark it unstable in QAPI (we want "x-" because it's not only QAPI
      field but also QOM property)
    - improve documentation and error reporting
    - keep r-b by Ben
15: - improve cleanup (more guarantee to cleanup /dev/shm)
    - some improvements to make further introduction of CPR-tests simpler
       - move some initial things to setUp()
       - drop default nic
       - rework prepare_and_launch_vm() to be only prepare_vm()
       - add migrate() helper
    - drop r-b

v20 is pushed to

    https://gitlab.com/vsementsov/qemu.git
      tag: up-tap-fd-migration-with-bk-opt-v20

To run the test, use sudo, as test needs to configure TAP device:

    sudo PYTHONPATH=python:tests/functional \
    QEMU_TEST_QEMU_BINARY=$PWD/build/qemu-system-x86_64 \
    MESON_BUILD_ROOT=$PWD/build \
    ./build/pyvenv/bin/python3 tests/functional/x86_64/test_tap_migration.py

Or, to test the feature by hand, you may follow the instruction at
the and of this document (search for "How to run local fd migration by hand"
caption).


Why this series, and why do it this way
=======================================

This came up in the v19 discussion, so it's summarized here to avoid
repeating the same questions.
There was a large discussion at v19, covered many questions. I'm
summarizing them below.

v19: https://lore.kernel.org/qemu-devel/20260714154246.1242856-1-vsementsov@yandex-team.ru/

Why local migration matters
---------------------------

Local migration (migrating to a new QEMU process on the same host) is not
a rare operation for in the cloud.  When upgrading QEMU, _all_ VMs on
_all_ hosts are locally migrated.  The freeze time of such update affects
_all_ customers, and some are sensitive to even small freezes.  Remote
migrations, by contrast, are done mostly to release a physical server for
service/redeploy, which is relatively infrequent.

The aggregate impact is large, and reducing the per-VM downtime is
significant.

Why not just create a new TAP and switch to it
----------------------------------------------

It is possible to create a new TAP device on the target and
migrate the usual way.  But the overhead is in switching to the new TAP
in our cloud virtual networking component.  Could the switching itself
be optimized to always be very fast?  Probably yes, but we chose a
different approach: drop extra components from the process and avoid
any switching at all.  Both the cloud networking component and the
storage backend (with the vhost-user-blk live-update series) don't even
know that QEMU is migrating.  This gives a simpler design (fewer
components involved) and shows good results -- several times reduction
in freeze time.

Also, on hosts with many CPUs, TAP queues allocate a noticeable amount of
RAM, so having 2x queues during migration is an overhead.

Why not use a persistent TAP (open the same TAP on both source and target)
-------------------------------------------------------------------------

First, same memory overhead like with "new TAP" case.

Second, MAX_TAP_QUEUES is 256 in the kernel; if the source already has more
  than 128 queues open, the target cannot open the same device.

[Note also, that such persistent TAP migration will not "just work",
it also needs QEMU changes like patch 12/15 to avoid packet lost]

Why not pass FDs externally (e.g. from libvirt / management tool)
-----------------------------------------------------------------

- The management tool would need to store (or retrieve from source QEMU)
  and pass these FDs, requiring it to know about FDs that QEMU already
  owns.  Libvirt may restart, and that shouldn't break migration.
- Adding an API to retrieve FDs from QEMU just to pass them to target QEMU
  is not a clearer solution.
- QEMU owns the TAP fd and has full knowledge of its state.  Pushing
  this responsibility to an external tool means the tool needs to
  understand QEMU internals -- that's a worse separation of concerns.
- QEMU already passes FDs through migration for VFIO devices (CPR), we
  do continue this practice

Looking at wider picture, where we want to migrate not only TAP, but also
vfio devices (already implemented as CPR migration, but may be updated to
use similar approach as in this series, to use one migration channel),
vhost-user-blk (my another series in flight) and vhost-user-fs (not yet
published), vhost-vsock migration ("[PATCH v3 0/7] migration/cpr: support
vhost-vsock devices" in flight-series from Virtuozzo), it seems to be a
good generic approach: simply pass backends (including open FDs) to the
target, not involving mgmt. Qemu has full knowledge about these FDs and
owns the whole state. Migrating them in QEMU looks correct.

Why not use the cpr-transfer approach (pass FDs before device creation)
-----------------------------------------------------------------------

The CPR approach passes FDs _before_ any devices are created on the
target.  While this avoids touching frontend code (patch 11/15), it
means the target starts using (calling ioctls on) FDs that are still
actively used by the running source QEMU.  That works for TAP, but it
does _not_ work with vhost-user-blk, where sharing the fd between
source and running target can break things.  My series uses the same
approach for both net and storage, and it also works with CPR (see
below).

That's not saying about "local" approach provides simpler user
experience than cpr-transfer:

- no need in secondary migration channel
- no need to work with unavailable QMP for target process until
  "migrate" on source
- more like usual migration

Does this work with CPR?
------------------------

Yes.  For cpr-transfer, TAP migrates through the main migration channel
like other (non-CPR) devices.  For cpr-exec, TAP FDs don't have
FD_CLOEXEC set, so they are inherited by the new process as-is; the
same core mechanism (load_fd / save_fd in migration/vmstate-types.c) is
used.  The 'local' migration parameter documentation now also covers
cpr-exec mode (skip the UNIX-socket requirement in that case).
New test cases exercising both cpr-transfer and cpr-exec with TAP
local migration will be sent as separate series.


How to run local fd migration by hand
=====================================

The walkthrough uses four terminals:

  source-cmd  -- source VM console (serial output, guest login)
  source-qmp  -- QMP connection to the source QEMU
  target-cmd  -- target VM console (serial output, guest login after migration)
  target-qmp  -- QMP connection to the target QEMU

1. Prerequisites
----------------

    QEMU=/path/to/your/build/qemu-system-x86_64

# download same image as in test

    wget -O /tmp/alpine.iso "https://dl-cdn.alpinelinux.org/alpine/v3.22/releases/x86_64/alpine-standard-3.22.1-x86_64.iso"


# prepare tap device (be careful to not break your own networks)

    sudo ip tuntap add dev tap0 mode tap multi_queue
    sudo ip addr add 192.168.100.1/24 dev tap0
    sudo ip link set tap0 up


2. Start source VM
------------------

In source-cmd, run:

    $QEMU \
        -name source \
        -machine q35 \
        -accel kvm \
        -m 1G \
        -object memory-backend-file,id=ram0,size=1G,mem-path=/dev/shm/qemu_migration_test,share=on \
        -machine memory-backend=ram0 \
        -drive file=/tmp/alpine.iso,media=cdrom,format=raw \
        -device pcie-pci-bridge,id=pci.1,bus=pcie.0 \
        -serial stdio \
        -nographic \
        -S \
        -qmp unix:/tmp/qmp-source.sock,server=on,wait=off

Note: the netdev and virtio-net device are added via QMP below for symmetry
with the target.  On the source you could also pass them on the command line
(with x-local-migration-supported=on).

In source-qmp, connect and add the TAP netdev and virtio-net device:

    socat - UNIX-CONNECT:/tmp/qmp-source.sock

    {"execute": "qmp_capabilities"}

    {"execute": "netdev_add", "arguments": {
        "id": "netdev.1",
        "type": "tap",
        "ifname": "tap0",
        "queues": 4,
        "vnet_hdr": true,
        "script": "no",
        "downscript": "no",
        "x-local-migration-supported": true
    }}

    {"execute": "device_add", "arguments": {
        "driver": "virtio-net-pci",
        "id": "vnet.1",
        "netdev": "netdev.1",
        "bus": "pci.1",
        "mq": true,
        "vectors": 18,
        "romfile": "",
        "disable-legacy": "off"
    }}

    {"execute": "cont"}

Wait for Alpine to boot in source-cmd.  When you see the login prompt, log
in as root (no password):

    localhost login: root

Configure the guest network:

    ip addr add 192.168.100.2/24 dev eth0
    ip link set eth0 up

Verify connectivity from the guest:

    ping -c 3 192.168.100.1

And from the host (in any spare terminal):

    ping -c 3 192.168.100.2


3. Start target VM
------------------

The TAP netdev must be created via QMP after enabling the "local" migration
parameter -- the target will not open tap0 itself; instead it will receive the
TAP file descriptors from the source over the migration channel.

In target-cmd, run:

    $QEMU \
        -name target \
        -machine q35 \
        -accel kvm \
        -m 1G \
        -object memory-backend-file,id=ram0,size=1G,mem-path=/dev/shm/qemu_migration_test,share=on \
        -machine memory-backend=ram0 \
        -drive file=/tmp/alpine.iso,media=cdrom,format=raw \
        -device pcie-pci-bridge,id=pci.1,bus=pcie.0 \
        -serial stdio \
        -nographic \
        -qmp unix:/tmp/qmp-target.sock,server=on,wait=off \
        -incoming defer

In target-qmp, connect, enable local migration, and add the TAP netdev and
virtio-net device.  The "local" parameter must be set before creating the TAP.
Do not pass ifname/fd -- the fd will arrive via the migration channel:

    socat - UNIX-CONNECT:/tmp/qmp-target.sock

    {"execute": "qmp_capabilities"}

    {"execute": "migrate-set-capabilities", "arguments": {
        "capabilities": [
            {"capability": "events",          "state": true},
            {"capability": "x-ignore-shared", "state": true}
        ]
    }}

    {"execute": "migrate-set-parameters", "arguments": {"local": true}}

    {"execute": "netdev_add", "arguments": {
        "id": "netdev.1",
        "type": "tap",
        "queues": 4,
        "script": "",
        "downscript": "",
        "x-local-migration-supported": true
    }}

    {"execute": "device_add", "arguments": {
        "driver": "virtio-net-pci",
        "id": "vnet.1",
        "netdev": "netdev.1",
        "bus": "pci.1",
        "mq": true,
        "vectors": 18,
        "romfile": "",
        "disable-legacy": "off"
    }}

4. Start migration
------------------

In target-qmp, tell the target to listen for the incoming migration:

    {"execute": "migrate-incoming",
     "arguments": {"uri": "unix:/tmp/migration.sock"}}

In source-qmp, configure migration capabilities and parameters:

    {"execute": "migrate-set-capabilities", "arguments": {
        "capabilities": [
            {"capability": "events",          "state": true},
            {"capability": "x-ignore-shared", "state": true}
        ]
    }}

    {"execute": "migrate-set-parameters", "arguments": {"local": true}}

In source-qmp, trigger the migration:

    {"execute": "migrate",
     "arguments": {"uri": "unix:/tmp/migration.sock"}}

Poll migration status until it completes (source-qmp):

    {"execute": "query-migrate"}
    # repeat until "status" == "completed"

Or just wait for the MIGRATION event that QEMU emits automatically:

    # {"event": "MIGRATION", "data": {"status": "completed"}, ...}

Once the source reports "completed", resume the target VM (target-qmp):

    {"execute": "cont"}

The target VM is now running with the migrated state and the TAP file
descriptors that were passed from the source. Still, target console
(in target-cmd) may still be empty, until you at least press Enter in
it.

Verify that the guest is still reachable from the host:

    ping -c 3 192.168.100.2

And from inside the guest in target-cmd:

    ping -c 3 192.168.100.1


5. Cleanup
----------

Shut down the target VM (target-qmp):

    {"execute": "quit"}

Shut down the source VM (source-qmp):

    {"execute": "quit"}

Remove the TAP device:

    sudo ip tuntap del tap0 mode tap multi_queue

Remove the shared memory file:

    rm /dev/shm/qemu_migration_test

Remove leftover sockets if they still exist:

    rm -f /tmp/migration.sock /tmp/qmp-source.sock /tmp/qmp-target.sock

Vladimir Sementsov-Ogievskiy (15):
  net/tap: rework tap_parse_script
  net/tap: improve script/downscript options documentation
  net/tap: deprecate "no" as special value for script/downscript
  net/tap: move vhost-net open() calls to tap_parse_vhost_fds()
  net/tap: move vhost initialization to tap_setup_vhost()
  net/tap: use container_of instead of DO_UPCAST
  net/tap: QOMify tap backend
  net/tap: add TYPE_VMSTATE_IF interface
  qapi: add local migration parameter
  migration/channel: check that transfer is UNIX socket when "local" set
  virtio-net: support local migration of backend
  net/tap: disable read polling for stopped VM
  net/tap: support local migration with virtio-net
  tests/functional: add skipWithoutSudo() decorator
  tests/functional: add test_tap_migration

 docs/about/deprecated.rst                     |  18 +
 docs/system/i386/microvm.rst                  |   4 +-
 docs/system/i386/xenpvh.rst                   |   2 +-
 docs/system/ppc/ppce500.rst                   |   4 +-
 docs/system/riscv/microchip-icicle-kit.rst    |   2 +-
 docs/system/riscv/sifive_u.rst                |   2 +-
 hw/net/virtio-net.c                           |  89 +++-
 include/hw/virtio/virtio-net.h                |   1 +
 include/migration/misc.h                      |   2 +
 include/migration/vmstate.h                   |   2 +
 include/net/net.h                             |   9 +
 include/net/tap.h                             |   2 +
 migration/channel.c                           |  17 +
 migration/options.c                           |  18 +-
 net/net.c                                     |  14 +-
 net/tap.c                                     | 446 ++++++++++++++---
 qapi/migration.json                           |  22 +-
 qapi/net.json                                 |  45 +-
 qemu-options.hx                               |  13 +-
 tests/functional/qemu_test/decorators.py      |  16 +
 tests/functional/x86_64/meson.build           |   1 +
 tests/functional/x86_64/test_tap_migration.py | 461 ++++++++++++++++++
 22 files changed, 1081 insertions(+), 109 deletions(-)
 create mode 100755 tests/functional/x86_64/test_tap_migration.py

-- 
2.43.0



^ permalink raw reply	[flat|nested] 22+ messages in thread

end of thread, other threads:[~2026-07-29 14:10 UTC | newest]

Thread overview: 22+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-07-29  9:12 [PATCH v20 00/15] virtio-net: live-TAP local migration Vladimir Sementsov-Ogievskiy
2026-07-29  9:12 ` [PATCH v20 01/15] net/tap: rework tap_parse_script Vladimir Sementsov-Ogievskiy
2026-07-29  9:12 ` [PATCH v20 02/15] net/tap: improve script/downscript options documentation Vladimir Sementsov-Ogievskiy
2026-07-29  9:12 ` [PATCH v20 03/15] net/tap: deprecate "no" as special value for script/downscript Vladimir Sementsov-Ogievskiy
2026-07-29  9:12 ` [PATCH v20 04/15] net/tap: move vhost-net open() calls to tap_parse_vhost_fds() Vladimir Sementsov-Ogievskiy
2026-07-29  9:12 ` [PATCH v20 05/15] net/tap: move vhost initialization to tap_setup_vhost() Vladimir Sementsov-Ogievskiy
2026-07-29  9:12 ` [PATCH v20 06/15] net/tap: use container_of instead of DO_UPCAST Vladimir Sementsov-Ogievskiy
2026-07-29  9:12 ` [PATCH v20 07/15] net/tap: QOMify tap backend Vladimir Sementsov-Ogievskiy
2026-07-29  9:12 ` [PATCH v20 08/15] net/tap: add TYPE_VMSTATE_IF interface Vladimir Sementsov-Ogievskiy
2026-07-29  9:12 ` [PATCH v20 09/15] qapi: add local migration parameter Vladimir Sementsov-Ogievskiy
2026-07-29 11:10   ` Markus Armbruster
2026-07-29 14:05   ` Peter Xu
2026-07-29  9:12 ` [PATCH v20 10/15] migration/channel: check that transfer is UNIX socket when "local" set Vladimir Sementsov-Ogievskiy
2026-07-29 14:09   ` Peter Xu
2026-07-29  9:12 ` [PATCH v20 11/15] virtio-net: support local migration of backend Vladimir Sementsov-Ogievskiy
2026-07-29  9:12 ` [PATCH v20 12/15] net/tap: disable read polling for stopped VM Vladimir Sementsov-Ogievskiy
2026-07-29  9:12 ` [PATCH v20 13/15] net/tap: support local migration with virtio-net Vladimir Sementsov-Ogievskiy
2026-07-29 11:30   ` Markus Armbruster
2026-07-29 12:25     ` Vladimir Sementsov-Ogievskiy
2026-07-29 12:45       ` Markus Armbruster
2026-07-29  9:12 ` [PATCH v20 14/15] tests/functional: add skipWithoutSudo() decorator Vladimir Sementsov-Ogievskiy
2026-07-29  9:12 ` [PATCH v20 15/15] tests/functional: add test_tap_migration Vladimir Sementsov-Ogievskiy

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.