[Qemu-devel] [PATCH v3 00/16] Make migration work with hotplug

[Alex Williamson <alex.williamson@redhat.com>]

v3:

Still looking for comments...

changes:
  - Rebase to latest upstream to pickup a few more callers
  - Fix a bug in patch introducing qemu_ram_free().  Since we now
    try to place blocks into existing holes in the address space,
    the end of the current block isn't necessarily the top
    ram_addr_t for realloc'ing the dirty bitmap.

v2:

Not too many comments, hope that's because everyone agrees ;)
A couple minor changes.  The 2nd patch is new and provides a
bit of an optimization for large memory pc guets.  The first
two patches stand on their own even if we're undecided about
the rest.  Thanks,

Alex

changes:
  - Use pci_find_domain() for PCI domain, thanks Isaku
  - Convert pc to allocate all ram in one chunk, which avoids
    penalizing large memory VMs bouncing between ramblocks
    during migration.

v1:

Ok, new approach.  I'm going to attempt to extract myself for the
canonical device path approach, because we're missing too many pieces
to make that work.  Instead, I'll take Anthony's advice and try to
simplify.  We still want a unique name for ramblocks and savevm, but
the hotplug problem today is only for PCI devices.  PCI conveniently
has globally unique, dare I say canonical, addressing in the form of
<domain>:<bus>:<device>.<func>.  To get to this, let's add a new
function on the BusInfo structure called get_dev_path().  For a PCI
device, we can simply traverse up the qdev tree to the BusInfo
structure, look for the function, and call it to return a global PCI
address.

For some buses, these functions could chain up to their parent bus
appending strings together to get a unique path.  An example would be
USB, where the USB port number may not be unique.  If we traverse up
to the PCI device providing USB, and then to the PCI bus, we get a
globally unique PCI path, appended with a USB port number.

To make this work for ramblocks and savevm, we need a DeviceState
pointer when the they are create/registered, and we need a caller
provided context in case there are multiple ramblocks/savevm
associated with a device.  Savevm already provies the context,
and I've attempted to make reasonable guesses at these for the
ramblocks.  Note that most of the ramblocks aren't associated with
a device, so I don't think it makes sense to link savevm and
ramblocks together with the same absolute id string.

Once we have savevm with unique id strings, rather than hotplug
unfriendly instance numbers, we can be sure that the right driver
instance is loading the correct vmstate.  I've also implemented
a compat field for this, so we can still accept incoming migrations
from previous versions.

Once we have ramblocks with a unique id string, we can switch to
using id + offset for migration, which enables a ram_addr_t space
that supports gaps, which enables us to implement qemu_ram_free().  
With that, I think we can finally do migrations reliable after
hotplug!  Note that the target VM still needs to be created to
match the current devices and bus addresses of the source VM.  We
can also still maintain compatibility for migrations here by bumping
the ram migration version and supporting both new and old (just
hope the source hasn't done any hotplugs).

Sound reasonable?  Is get_dev_path the right name?  In the right
place?  The PCI return is currently "dddd:bb:dd.f", should this be
"PCI:dddd:bb:dd.f"?  Something else?  Thanks,

Alex

---

Alex Williamson (16):
      ramblocks: No more being lazy about duplicate names
      pci: Free the space allocated for the option rom on removal
      qemu_ram_free: Implement it
      savevm: Create a new continue flag to avoid resending block name
      savevm: Use RAM blocks for basis of migration
      savevm: Migrate RAM based on name/offset
      ramblocks: Make use of DeviceState pointer and BusInfo.get_dev_path
      qemu_ram_alloc: Add DeviceState and name parameters
      virtio-net: Incorporate a DeviceState pointer and let savevm track instances
      eepro100: Add a dev field to eeprom new/free functions
      savevm: Make use of DeviceState
      savevm: Add DeviceState param
      pci: Implement BusInfo.get_dev_path()
      qdev: Add a get_dev_path() function to BusInfo
      pc: Allocate all ram in a single qemu_ram_alloc()
      Remove uses of ram.last_offset (aka last_ram_offset)


 arch_init.c                   |  183 +++++++++++++++++++++++++++++++++++------
 audio/audio.c                 |    2 
 block-migration.c             |    4 -
 cpu-all.h                     |    5 +
 cpu-common.h                  |    2 
 exec.c                        |  107 +++++++++++++++++++++---
 hw/adb.c                      |    4 -
 hw/ads7846.c                  |    2 
 hw/an5206.c                   |    4 -
 hw/arm_gic.c                  |    2 
 hw/arm_timer.c                |    4 -
 hw/armv7m.c                   |    9 +-
 hw/armv7m_nvic.c              |    2 
 hw/axis_dev88.c               |    4 -
 hw/cirrus_vga.c               |    2 
 hw/cuda.c                     |    2 
 hw/dma.c                      |    4 -
 hw/dummy_m68k.c               |    2 
 hw/eepro100.c                 |    8 +-
 hw/eeprom93xx.c               |    8 +-
 hw/eeprom93xx.h               |    4 -
 hw/etraxfs.c                  |    6 +
 hw/g364fb.c                   |    4 -
 hw/grackle_pci.c              |    4 -
 hw/gt64xxx.c                  |    3 -
 hw/gumstix.c                  |    6 +
 hw/heathrow_pic.c             |    2 
 hw/hw.h                       |   18 ++--
 hw/i2c.c                      |    2 
 hw/i8254.c                    |    2 
 hw/i8259.c                    |    2 
 hw/ide/cmd646.c               |    2 
 hw/ide/isa.c                  |    2 
 hw/ide/macio.c                |    2 
 hw/ide/microdrive.c           |    2 
 hw/ide/mmio.c                 |    2 
 hw/ide/piix.c                 |    2 
 hw/ide/via.c                  |    2 
 hw/integratorcp.c             |    4 -
 hw/m48t59.c                   |    2 
 hw/mac_dbdma.c                |    2 
 hw/mac_nvram.c                |    4 -
 hw/mainstone.c                |    6 +
 hw/max111x.c                  |    3 -
 hw/mcf5208.c                  |    4 -
 hw/mips_fulong2e.c            |    6 +
 hw/mips_jazz.c                |    4 -
 hw/mips_malta.c               |    4 -
 hw/mips_mipssim.c             |    4 -
 hw/mips_r4k.c                 |    6 +
 hw/mipsnet.c                  |    4 -
 hw/mst_fpga.c                 |    3 -
 hw/musicpal.c                 |   11 ++
 hw/nand.c                     |    2 
 hw/omap1.c                    |    6 +
 hw/omap2.c                    |    6 +
 hw/omap_sx1.c                 |   12 ++-
 hw/onenand.c                  |    2 
 hw/openpic.c                  |    5 +
 hw/palm.c                     |    3 -
 hw/pc.c                       |   27 +++---
 hw/pci.c                      |   34 +++++++-
 hw/pckbd.c                    |    2 
 hw/petalogix_s3adsp1800_mmu.c |    7 +-
 hw/piix4.c                    |    2 
 hw/pl011.c                    |    2 
 hw/pl022.c                    |    2 
 hw/pl061.c                    |    2 
 hw/ppc405_boards.c            |   18 ++--
 hw/ppc405_uc.c                |    2 
 hw/ppc4xx_devs.c              |    4 +
 hw/ppc4xx_pci.c               |    4 -
 hw/ppc_newworld.c             |    6 +
 hw/ppc_oldworld.c             |    6 +
 hw/ppc_prep.c                 |    4 -
 hw/ppce500_mpc8544ds.c        |    3 -
 hw/ppce500_pci.c              |    4 -
 hw/ps2.c                      |    4 -
 hw/pxa2xx.c                   |   39 +++++----
 hw/pxa2xx_dma.c               |    2 
 hw/pxa2xx_gpio.c              |    2 
 hw/pxa2xx_keypad.c            |    2 
 hw/pxa2xx_lcd.c               |    2 
 hw/pxa2xx_mmci.c              |    2 
 hw/pxa2xx_pic.c               |    3 -
 hw/pxa2xx_timer.c             |    2 
 hw/qdev.c                     |    4 -
 hw/qdev.h                     |    3 +
 hw/r2d.c                      |    4 -
 hw/rc4030.c                   |    2 
 hw/realview.c                 |    6 +
 hw/s390-virtio.c              |    2 
 hw/serial.c                   |    4 -
 hw/sm501.c                    |    2 
 hw/spitz.c                    |   11 +-
 hw/ssd0323.c                  |    3 -
 hw/ssi-sd.c                   |    2 
 hw/stellaris.c                |   11 +-
 hw/stellaris_enet.c           |    4 -
 hw/stellaris_input.c          |    2 
 hw/sun4m.c                    |    8 +-
 hw/sun4u.c                    |    4 -
 hw/syborg.c                   |    2 
 hw/syborg_fb.c                |    2 
 hw/syborg_interrupt.c         |    3 -
 hw/syborg_keyboard.c          |    2 
 hw/syborg_pointer.c           |    2 
 hw/syborg_rtc.c               |    3 -
 hw/syborg_serial.c            |    2 
 hw/syborg_timer.c             |    2 
 hw/tc6393xb.c                 |    2 
 hw/tcx.c                      |    2 
 hw/tosa.c                     |    2 
 hw/tsc2005.c                  |    2 
 hw/tsc210x.c                  |    4 -
 hw/unin_pci.c                 |    6 +
 hw/versatilepb.c              |    2 
 hw/vga-isa-mm.c               |    2 
 hw/vga-isa.c                  |    2 
 hw/vga.c                      |    2 
 hw/virtio-balloon.c           |    3 -
 hw/virtio-blk.c               |    2 
 hw/virtio-net.c               |    7 +-
 hw/virtio-serial-bus.c        |    2 
 hw/vmmouse.c                  |    2 
 hw/vmware_vga.c               |    4 -
 hw/zaurus.c                   |    2 
 qemu-timer.c                  |    2 
 savevm.c                      |  106 +++++++++++++++++++++---
 slirp/slirp.c                 |    5 +
 vl.c                          |    2 
 131 files changed, 670 insertions(+), 300 deletions(-)