* Re: [PATCH v4] failover: unregister ROM on unplug
2021-07-21 16:09 [PATCH v4] failover: unregister ROM on unplug Laurent Vivier
@ 2021-07-21 16:19 ` Philippe Mathieu-Daudé
2021-07-21 16:19 ` Michael S. Tsirkin
` (2 subsequent siblings)
3 siblings, 0 replies; 9+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-07-21 16:19 UTC (permalink / raw)
To: Laurent Vivier, qemu-devel
Cc: Juan Quintela, Jason Wang, Michael S. Tsirkin,
Dr . David Alan Gilbert, Igor Mammedov, Jens Freimann
On 7/21/21 6:09 PM, Laurent Vivier wrote:
> The intend of failover is to allow to migrate a VM with a VFIO
> networking card without disrupting the network operation by switching
> to a virtio-net device during the migration.
>
> This simple change allows to test failover with a simulated device
> like e1000e rather than a vfio device, even if it's useless in real
> life it can help to debug failover.
>
> This is interesting to developers that want to test failover on
> a system with no vfio device. Moreover it simplifies host networking
> configuration as we can use the same bridge for virtio-net and
> the other failover networking device.
>
> Without this change the migration of a system configured with failover
> fails with:
>
> ...
> -device virtio-net-pci,id=virtionet0,failover=on,... \
> -device e1000,failover_pair_id=virtionet0,... \
> ...
>
> (qemu) migrate ...
>
> Unknown ramblock "0000:00:01.1:00.0/e1000e.rom", cannot accept migration
> error while loading state for instance 0x0 of device 'ram'
> load of migration failed: Invalid argument
>
> This happens because QEMU correctly unregisters the interface vmstate but
> not the ROM one. This patch fixes that.
>
> Signed-off-by: Laurent Vivier <lvivier@redhat.com>
> ---
>
> Notes:
> v4:
> export and use pci_del_option_rom()
>
> v3:
> remove useless space before comma
>
> v2:
> reset has_rom to false
> update commit log message
>
> include/hw/pci/pci.h | 2 ++
> hw/net/virtio-net.c | 1 +
> hw/pci/pci.c | 3 +--
> 3 files changed, 4 insertions(+), 2 deletions(-)
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [PATCH v4] failover: unregister ROM on unplug
2021-07-21 16:09 [PATCH v4] failover: unregister ROM on unplug Laurent Vivier
2021-07-21 16:19 ` Philippe Mathieu-Daudé
@ 2021-07-21 16:19 ` Michael S. Tsirkin
2021-07-21 17:01 ` Laurent Vivier
2021-07-23 8:55 ` Laurent Vivier
2021-08-03 14:04 ` Michael S. Tsirkin
2021-08-11 6:50 ` Michael S. Tsirkin
3 siblings, 2 replies; 9+ messages in thread
From: Michael S. Tsirkin @ 2021-07-21 16:19 UTC (permalink / raw)
To: Laurent Vivier
Cc: Juan Quintela, Jason Wang, qemu-devel, Dr . David Alan Gilbert,
Igor Mammedov, Jens Freimann, Philippe Mathieu-Daudé
On Wed, Jul 21, 2021 at 06:09:05PM +0200, Laurent Vivier wrote:
> The intend of failover is to allow to migrate a VM with a VFIO
> networking card without disrupting the network operation by switching
> to a virtio-net device during the migration.
>
> This simple change allows to test failover with a simulated device
> like e1000e rather than a vfio device, even if it's useless in real
> life it can help to debug failover.
>
> This is interesting to developers that want to test failover on
> a system with no vfio device. Moreover it simplifies host networking
> configuration as we can use the same bridge for virtio-net and
> the other failover networking device.
>
> Without this change the migration of a system configured with failover
> fails with:
>
> ...
> -device virtio-net-pci,id=virtionet0,failover=on,... \
> -device e1000,failover_pair_id=virtionet0,... \
> ...
>
> (qemu) migrate ...
>
> Unknown ramblock "0000:00:01.1:00.0/e1000e.rom", cannot accept migration
> error while loading state for instance 0x0 of device 'ram'
> load of migration failed: Invalid argument
>
> This happens because QEMU correctly unregisters the interface vmstate but
> not the ROM one. This patch fixes that.
>
> Signed-off-by: Laurent Vivier <lvivier@redhat.com>
And does failover_replug_primary roll it all back then?
> ---
>
> Notes:
> v4:
> export and use pci_del_option_rom()
>
> v3:
> remove useless space before comma
>
> v2:
> reset has_rom to false
> update commit log message
>
> include/hw/pci/pci.h | 2 ++
> hw/net/virtio-net.c | 1 +
> hw/pci/pci.c | 3 +--
> 3 files changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h
> index d0f4266e3725..84707034cbf8 100644
> --- a/include/hw/pci/pci.h
> +++ b/include/hw/pci/pci.h
> @@ -369,6 +369,8 @@ void pci_register_vga(PCIDevice *pci_dev, MemoryRegion *mem,
> void pci_unregister_vga(PCIDevice *pci_dev);
> pcibus_t pci_get_bar_addr(PCIDevice *pci_dev, int region_num);
>
> +void pci_del_option_rom(PCIDevice *pdev);
> +
> int pci_add_capability(PCIDevice *pdev, uint8_t cap_id,
> uint8_t offset, uint8_t size,
> Error **errp);
> diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
> index 16d20cdee52a..d6f03633f1b3 100644
> --- a/hw/net/virtio-net.c
> +++ b/hw/net/virtio-net.c
> @@ -3256,6 +3256,7 @@ static void virtio_net_handle_migration_primary(VirtIONet *n, MigrationState *s)
> if (migration_in_setup(s) && !should_be_hidden) {
> if (failover_unplug_primary(n, dev)) {
> vmstate_unregister(VMSTATE_IF(dev), qdev_get_vmsd(dev), dev);
> + pci_del_option_rom(PCI_DEVICE(dev));
> qapi_event_send_unplug_primary(dev->id);
> qatomic_set(&n->failover_primary_hidden, true);
> } else {
> diff --git a/hw/pci/pci.c b/hw/pci/pci.c
> index 23d2ae2ab232..c210d92b5ba7 100644
> --- a/hw/pci/pci.c
> +++ b/hw/pci/pci.c
> @@ -228,7 +228,6 @@ static PCIBus *pci_find_bus_nr(PCIBus *bus, int bus_num);
> static void pci_update_mappings(PCIDevice *d);
> static void pci_irq_handler(void *opaque, int irq_num, int level);
> static void pci_add_option_rom(PCIDevice *pdev, bool is_default_rom, Error **);
> -static void pci_del_option_rom(PCIDevice *pdev);
>
> static uint16_t pci_default_sub_vendor_id = PCI_SUBVENDOR_ID_REDHAT_QUMRANET;
> static uint16_t pci_default_sub_device_id = PCI_SUBDEVICE_ID_QEMU;
> @@ -2429,7 +2428,7 @@ static void pci_add_option_rom(PCIDevice *pdev, bool is_default_rom,
> pci_register_bar(pdev, PCI_ROM_SLOT, 0, &pdev->rom);
> }
>
> -static void pci_del_option_rom(PCIDevice *pdev)
> +void pci_del_option_rom(PCIDevice *pdev)
> {
> if (!pdev->has_rom)
> return;
> --
> 2.31.1
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [PATCH v4] failover: unregister ROM on unplug
2021-07-21 16:19 ` Michael S. Tsirkin
@ 2021-07-21 17:01 ` Laurent Vivier
2021-07-23 8:55 ` Laurent Vivier
1 sibling, 0 replies; 9+ messages in thread
From: Laurent Vivier @ 2021-07-21 17:01 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Juan Quintela, Jason Wang, qemu-devel, Dr . David Alan Gilbert,
Igor Mammedov, Jens Freimann, Philippe Mathieu-Daudé
On 21/07/2021 18:19, Michael S. Tsirkin wrote:
> On Wed, Jul 21, 2021 at 06:09:05PM +0200, Laurent Vivier wrote:
>> The intend of failover is to allow to migrate a VM with a VFIO
>> networking card without disrupting the network operation by switching
>> to a virtio-net device during the migration.
>>
>> This simple change allows to test failover with a simulated device
>> like e1000e rather than a vfio device, even if it's useless in real
>> life it can help to debug failover.
>>
>> This is interesting to developers that want to test failover on
>> a system with no vfio device. Moreover it simplifies host networking
>> configuration as we can use the same bridge for virtio-net and
>> the other failover networking device.
>>
>> Without this change the migration of a system configured with failover
>> fails with:
>>
>> ...
>> -device virtio-net-pci,id=virtionet0,failover=on,... \
>> -device e1000,failover_pair_id=virtionet0,... \
>> ...
>>
>> (qemu) migrate ...
>>
>> Unknown ramblock "0000:00:01.1:00.0/e1000e.rom", cannot accept migration
>> error while loading state for instance 0x0 of device 'ram'
>> load of migration failed: Invalid argument
>>
>> This happens because QEMU correctly unregisters the interface vmstate but
>> not the ROM one. This patch fixes that.
>>
>> Signed-off-by: Laurent Vivier <lvivier@redhat.com>
>
> And does failover_replug_primary roll it all back then?
It seems not.
To check I have started a migration, then I have canceled it, and I have started a
migration to a file (migrate "exec:cat > mig") and I have analyzed the result with the
script scripts/analyze-migration.py
But:
* with VFIO it's not a problem as we never migrate VFIO device itself,
* with an emulated PCI device, it's only a problem if we disable failover after having
canceled a first try (because if we keep failover we acts like for VFIO and we unplug the
card and don't migrate vmstates). This means we need a scenario like that to hit the bug:
- enable failover with an emulated PCI device
- migrate to another machine
- cancel/abort the migration before the end of the migration
- unplug the virtio-net device to disable the failover behavior,
- migrate the machine again with only the emulated PCI device
but as I said previously failover with emulated PCI device is only for developers and
test purpose and not to use in production...
Thanks,
Laurent
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [PATCH v4] failover: unregister ROM on unplug
2021-07-21 16:19 ` Michael S. Tsirkin
2021-07-21 17:01 ` Laurent Vivier
@ 2021-07-23 8:55 ` Laurent Vivier
1 sibling, 0 replies; 9+ messages in thread
From: Laurent Vivier @ 2021-07-23 8:55 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Juan Quintela, Jason Wang, qemu-devel, Dr . David Alan Gilbert,
Igor Mammedov, Jens Freimann, Philippe Mathieu-Daudé
On 21/07/2021 18:19, Michael S. Tsirkin wrote:
> On Wed, Jul 21, 2021 at 06:09:05PM +0200, Laurent Vivier wrote:
>> The intend of failover is to allow to migrate a VM with a VFIO
>> networking card without disrupting the network operation by switching
>> to a virtio-net device during the migration.
>>
>> This simple change allows to test failover with a simulated device
>> like e1000e rather than a vfio device, even if it's useless in real
>> life it can help to debug failover.
>>
>> This is interesting to developers that want to test failover on
>> a system with no vfio device. Moreover it simplifies host networking
>> configuration as we can use the same bridge for virtio-net and
>> the other failover networking device.
>>
>> Without this change the migration of a system configured with failover
>> fails with:
>>
>> ...
>> -device virtio-net-pci,id=virtionet0,failover=on,... \
>> -device e1000,failover_pair_id=virtionet0,... \
>> ...
>>
>> (qemu) migrate ...
>>
>> Unknown ramblock "0000:00:01.1:00.0/e1000e.rom", cannot accept migration
>> error while loading state for instance 0x0 of device 'ram'
>> load of migration failed: Invalid argument
>>
>> This happens because QEMU correctly unregisters the interface vmstate but
>> not the ROM one. This patch fixes that.
>>
>> Signed-off-by: Laurent Vivier <lvivier@redhat.com>
>
> And does failover_replug_primary roll it all back then?
In fact, I think we cannot roll it back because we don't have all the information to do
the vmstate_register() again. Perhaps we can implement a
vmstate_disable()/vmstate_enable() to disable the state migration without removing the
information from the list?
Thanks,
Laurent
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v4] failover: unregister ROM on unplug
2021-07-21 16:09 [PATCH v4] failover: unregister ROM on unplug Laurent Vivier
2021-07-21 16:19 ` Philippe Mathieu-Daudé
2021-07-21 16:19 ` Michael S. Tsirkin
@ 2021-08-03 14:04 ` Michael S. Tsirkin
2021-08-03 14:49 ` Laurent Vivier
2021-08-11 6:50 ` Michael S. Tsirkin
3 siblings, 1 reply; 9+ messages in thread
From: Michael S. Tsirkin @ 2021-08-03 14:04 UTC (permalink / raw)
To: Laurent Vivier
Cc: Juan Quintela, Jason Wang, qemu-devel, Dr . David Alan Gilbert,
Igor Mammedov, Jens Freimann, Philippe Mathieu-Daudé
On Wed, Jul 21, 2021 at 06:09:05PM +0200, Laurent Vivier wrote:
> The intend of failover is to allow to migrate a VM with a VFIO
> networking card without disrupting the network operation by switching
> to a virtio-net device during the migration.
>
> This simple change allows to test failover with a simulated device
> like e1000e rather than a vfio device, even if it's useless in real
> life it can help to debug failover.
>
> This is interesting to developers that want to test failover on
> a system with no vfio device. Moreover it simplifies host networking
> configuration as we can use the same bridge for virtio-net and
> the other failover networking device.
>
> Without this change the migration of a system configured with failover
> fails with:
>
> ...
> -device virtio-net-pci,id=virtionet0,failover=on,... \
> -device e1000,failover_pair_id=virtionet0,... \
> ...
>
> (qemu) migrate ...
>
> Unknown ramblock "0000:00:01.1:00.0/e1000e.rom", cannot accept migration
> error while loading state for instance 0x0 of device 'ram'
> load of migration failed: Invalid argument
>
> This happens because QEMU correctly unregisters the interface vmstate but
> not the ROM one. This patch fixes that.
>
> Signed-off-by: Laurent Vivier <lvivier@redhat.com>
Build fails on qemu-system-m68k:
/usr/bin/ld: libqemu-m68k-softmmu.fa.p/hw_net_virtio-net.c.o: in function `virtio_net_handle_migration_primary':
/scm/qemu/build/../hw/net/virtio-net.c:3259: undefined reference to `pci_del_option_rom'
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
make[1]: *** [Makefile:154: run-ninja] Error 1
It's not pretty to poke at pci from generic virtio.
Should we maybe wrap vmstate_unregister and pci_del_option_rom
to allow removing all migrateable things related to the device
in one go somehow?
> ---
>
> Notes:
> v4:
> export and use pci_del_option_rom()
>
> v3:
> remove useless space before comma
>
> v2:
> reset has_rom to false
> update commit log message
>
> include/hw/pci/pci.h | 2 ++
> hw/net/virtio-net.c | 1 +
> hw/pci/pci.c | 3 +--
> 3 files changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h
> index d0f4266e3725..84707034cbf8 100644
> --- a/include/hw/pci/pci.h
> +++ b/include/hw/pci/pci.h
> @@ -369,6 +369,8 @@ void pci_register_vga(PCIDevice *pci_dev, MemoryRegion *mem,
> void pci_unregister_vga(PCIDevice *pci_dev);
> pcibus_t pci_get_bar_addr(PCIDevice *pci_dev, int region_num);
>
> +void pci_del_option_rom(PCIDevice *pdev);
> +
> int pci_add_capability(PCIDevice *pdev, uint8_t cap_id,
> uint8_t offset, uint8_t size,
> Error **errp);
> diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
> index 16d20cdee52a..d6f03633f1b3 100644
> --- a/hw/net/virtio-net.c
> +++ b/hw/net/virtio-net.c
> @@ -3256,6 +3256,7 @@ static void virtio_net_handle_migration_primary(VirtIONet *n, MigrationState *s)
> if (migration_in_setup(s) && !should_be_hidden) {
> if (failover_unplug_primary(n, dev)) {
> vmstate_unregister(VMSTATE_IF(dev), qdev_get_vmsd(dev), dev);
> + pci_del_option_rom(PCI_DEVICE(dev));
> qapi_event_send_unplug_primary(dev->id);
> qatomic_set(&n->failover_primary_hidden, true);
> } else {
> diff --git a/hw/pci/pci.c b/hw/pci/pci.c
> index 23d2ae2ab232..c210d92b5ba7 100644
> --- a/hw/pci/pci.c
> +++ b/hw/pci/pci.c
> @@ -228,7 +228,6 @@ static PCIBus *pci_find_bus_nr(PCIBus *bus, int bus_num);
> static void pci_update_mappings(PCIDevice *d);
> static void pci_irq_handler(void *opaque, int irq_num, int level);
> static void pci_add_option_rom(PCIDevice *pdev, bool is_default_rom, Error **);
> -static void pci_del_option_rom(PCIDevice *pdev);
>
> static uint16_t pci_default_sub_vendor_id = PCI_SUBVENDOR_ID_REDHAT_QUMRANET;
> static uint16_t pci_default_sub_device_id = PCI_SUBDEVICE_ID_QEMU;
> @@ -2429,7 +2428,7 @@ static void pci_add_option_rom(PCIDevice *pdev, bool is_default_rom,
> pci_register_bar(pdev, PCI_ROM_SLOT, 0, &pdev->rom);
> }
>
> -static void pci_del_option_rom(PCIDevice *pdev)
> +void pci_del_option_rom(PCIDevice *pdev)
> {
> if (!pdev->has_rom)
> return;
> --
> 2.31.1
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [PATCH v4] failover: unregister ROM on unplug
2021-08-03 14:04 ` Michael S. Tsirkin
@ 2021-08-03 14:49 ` Laurent Vivier
0 siblings, 0 replies; 9+ messages in thread
From: Laurent Vivier @ 2021-08-03 14:49 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Juan Quintela, Jason Wang, qemu-devel, Dr . David Alan Gilbert,
Igor Mammedov, Jens Freimann, Philippe Mathieu-Daudé
On 03/08/2021 16:04, Michael S. Tsirkin wrote:
> On Wed, Jul 21, 2021 at 06:09:05PM +0200, Laurent Vivier wrote:
>> The intend of failover is to allow to migrate a VM with a VFIO
>> networking card without disrupting the network operation by switching
>> to a virtio-net device during the migration.
>>
>> This simple change allows to test failover with a simulated device
>> like e1000e rather than a vfio device, even if it's useless in real
>> life it can help to debug failover.
>>
>> This is interesting to developers that want to test failover on
>> a system with no vfio device. Moreover it simplifies host networking
>> configuration as we can use the same bridge for virtio-net and
>> the other failover networking device.
>>
>> Without this change the migration of a system configured with failover
>> fails with:
>>
>> ...
>> -device virtio-net-pci,id=virtionet0,failover=on,... \
>> -device e1000,failover_pair_id=virtionet0,... \
>> ...
>>
>> (qemu) migrate ...
>>
>> Unknown ramblock "0000:00:01.1:00.0/e1000e.rom", cannot accept migration
>> error while loading state for instance 0x0 of device 'ram'
>> load of migration failed: Invalid argument
>>
>> This happens because QEMU correctly unregisters the interface vmstate but
>> not the ROM one. This patch fixes that.
>>
>> Signed-off-by: Laurent Vivier <lvivier@redhat.com>
>
>
> Build fails on qemu-system-m68k:
>
> /usr/bin/ld: libqemu-m68k-softmmu.fa.p/hw_net_virtio-net.c.o: in function `virtio_net_handle_migration_primary':
> /scm/qemu/build/../hw/net/virtio-net.c:3259: undefined reference to `pci_del_option_rom'
> collect2: error: ld returned 1 exit status
> ninja: build stopped: subcommand failed.
> make[1]: *** [Makefile:154: run-ninja] Error 1
>
> It's not pretty to poke at pci from generic virtio.
I agree with you. The problem is failover is implemented in virtio-net while it relies on
virtio-pci hotplug capability.
> Should we maybe wrap vmstate_unregister and pci_del_option_rom
> to allow removing all migrateable things related to the device
> in one go somehow?
I'm going to have a look to see how to do.
I have already a patch that moves all the failover PCI unplug/hotplug functions into the
PCI device implementation (and removes all this stuff from virtio-net). Perhaps we can
rely on that. The bonus with this moves is it allows to automatically unplug/hotplug any
PCI device during a migration without the failover environment (but failover uses it).
Thanks,
Laurent
>
>> ---
>>
>> Notes:
>> v4:
>> export and use pci_del_option_rom()
>>
>> v3:
>> remove useless space before comma
>>
>> v2:
>> reset has_rom to false
>> update commit log message
>>
>> include/hw/pci/pci.h | 2 ++
>> hw/net/virtio-net.c | 1 +
>> hw/pci/pci.c | 3 +--
>> 3 files changed, 4 insertions(+), 2 deletions(-)
>>
>> diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h
>> index d0f4266e3725..84707034cbf8 100644
>> --- a/include/hw/pci/pci.h
>> +++ b/include/hw/pci/pci.h
>> @@ -369,6 +369,8 @@ void pci_register_vga(PCIDevice *pci_dev, MemoryRegion *mem,
>> void pci_unregister_vga(PCIDevice *pci_dev);
>> pcibus_t pci_get_bar_addr(PCIDevice *pci_dev, int region_num);
>>
>> +void pci_del_option_rom(PCIDevice *pdev);
>> +
>> int pci_add_capability(PCIDevice *pdev, uint8_t cap_id,
>> uint8_t offset, uint8_t size,
>> Error **errp);
>> diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
>> index 16d20cdee52a..d6f03633f1b3 100644
>> --- a/hw/net/virtio-net.c
>> +++ b/hw/net/virtio-net.c
>> @@ -3256,6 +3256,7 @@ static void virtio_net_handle_migration_primary(VirtIONet *n, MigrationState *s)
>> if (migration_in_setup(s) && !should_be_hidden) {
>> if (failover_unplug_primary(n, dev)) {
>> vmstate_unregister(VMSTATE_IF(dev), qdev_get_vmsd(dev), dev);
>> + pci_del_option_rom(PCI_DEVICE(dev));
>> qapi_event_send_unplug_primary(dev->id);
>> qatomic_set(&n->failover_primary_hidden, true);
>> } else {
>> diff --git a/hw/pci/pci.c b/hw/pci/pci.c
>> index 23d2ae2ab232..c210d92b5ba7 100644
>> --- a/hw/pci/pci.c
>> +++ b/hw/pci/pci.c
>> @@ -228,7 +228,6 @@ static PCIBus *pci_find_bus_nr(PCIBus *bus, int bus_num);
>> static void pci_update_mappings(PCIDevice *d);
>> static void pci_irq_handler(void *opaque, int irq_num, int level);
>> static void pci_add_option_rom(PCIDevice *pdev, bool is_default_rom, Error **);
>> -static void pci_del_option_rom(PCIDevice *pdev);
>>
>> static uint16_t pci_default_sub_vendor_id = PCI_SUBVENDOR_ID_REDHAT_QUMRANET;
>> static uint16_t pci_default_sub_device_id = PCI_SUBDEVICE_ID_QEMU;
>> @@ -2429,7 +2428,7 @@ static void pci_add_option_rom(PCIDevice *pdev, bool is_default_rom,
>> pci_register_bar(pdev, PCI_ROM_SLOT, 0, &pdev->rom);
>> }
>>
>> -static void pci_del_option_rom(PCIDevice *pdev)
>> +void pci_del_option_rom(PCIDevice *pdev)
>> {
>> if (!pdev->has_rom)
>> return;
>> --
>> 2.31.1
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v4] failover: unregister ROM on unplug
2021-07-21 16:09 [PATCH v4] failover: unregister ROM on unplug Laurent Vivier
` (2 preceding siblings ...)
2021-08-03 14:04 ` Michael S. Tsirkin
@ 2021-08-11 6:50 ` Michael S. Tsirkin
2021-08-11 9:51 ` Laurent Vivier
3 siblings, 1 reply; 9+ messages in thread
From: Michael S. Tsirkin @ 2021-08-11 6:50 UTC (permalink / raw)
To: Laurent Vivier
Cc: Juan Quintela, Jason Wang, qemu-devel, Dr . David Alan Gilbert,
Igor Mammedov, Jens Freimann, Philippe Mathieu-Daudé
On Wed, Jul 21, 2021 at 06:09:05PM +0200, Laurent Vivier wrote:
> The intend of failover is to allow to migrate a VM with a VFIO
> networking card without disrupting the network operation by switching
> to a virtio-net device during the migration.
>
> This simple change allows to test failover with a simulated device
> like e1000e rather than a vfio device, even if it's useless in real
> life it can help to debug failover.
>
> This is interesting to developers that want to test failover on
> a system with no vfio device. Moreover it simplifies host networking
> configuration as we can use the same bridge for virtio-net and
> the other failover networking device.
>
> Without this change the migration of a system configured with failover
> fails with:
>
> ...
> -device virtio-net-pci,id=virtionet0,failover=on,... \
> -device e1000,failover_pair_id=virtionet0,... \
> ...
>
> (qemu) migrate ...
>
> Unknown ramblock "0000:00:01.1:00.0/e1000e.rom", cannot accept migration
> error while loading state for instance 0x0 of device 'ram'
> load of migration failed: Invalid argument
>
> This happens because QEMU correctly unregisters the interface vmstate but
> not the ROM one. This patch fixes that.
>
> Signed-off-by: Laurent Vivier <lvivier@redhat.com>
I began to wonder about this. Why are we sending the option ROM at all?
I think there's no need to do it for the primary ...
> ---
>
> Notes:
> v4:
> export and use pci_del_option_rom()
>
> v3:
> remove useless space before comma
>
> v2:
> reset has_rom to false
> update commit log message
>
> include/hw/pci/pci.h | 2 ++
> hw/net/virtio-net.c | 1 +
> hw/pci/pci.c | 3 +--
> 3 files changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h
> index d0f4266e3725..84707034cbf8 100644
> --- a/include/hw/pci/pci.h
> +++ b/include/hw/pci/pci.h
> @@ -369,6 +369,8 @@ void pci_register_vga(PCIDevice *pci_dev, MemoryRegion *mem,
> void pci_unregister_vga(PCIDevice *pci_dev);
> pcibus_t pci_get_bar_addr(PCIDevice *pci_dev, int region_num);
>
> +void pci_del_option_rom(PCIDevice *pdev);
> +
> int pci_add_capability(PCIDevice *pdev, uint8_t cap_id,
> uint8_t offset, uint8_t size,
> Error **errp);
> diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
> index 16d20cdee52a..d6f03633f1b3 100644
> --- a/hw/net/virtio-net.c
> +++ b/hw/net/virtio-net.c
> @@ -3256,6 +3256,7 @@ static void virtio_net_handle_migration_primary(VirtIONet *n, MigrationState *s)
> if (migration_in_setup(s) && !should_be_hidden) {
> if (failover_unplug_primary(n, dev)) {
> vmstate_unregister(VMSTATE_IF(dev), qdev_get_vmsd(dev), dev);
> + pci_del_option_rom(PCI_DEVICE(dev));
> qapi_event_send_unplug_primary(dev->id);
> qatomic_set(&n->failover_primary_hidden, true);
> } else {
> diff --git a/hw/pci/pci.c b/hw/pci/pci.c
> index 23d2ae2ab232..c210d92b5ba7 100644
> --- a/hw/pci/pci.c
> +++ b/hw/pci/pci.c
> @@ -228,7 +228,6 @@ static PCIBus *pci_find_bus_nr(PCIBus *bus, int bus_num);
> static void pci_update_mappings(PCIDevice *d);
> static void pci_irq_handler(void *opaque, int irq_num, int level);
> static void pci_add_option_rom(PCIDevice *pdev, bool is_default_rom, Error **);
> -static void pci_del_option_rom(PCIDevice *pdev);
>
> static uint16_t pci_default_sub_vendor_id = PCI_SUBVENDOR_ID_REDHAT_QUMRANET;
> static uint16_t pci_default_sub_device_id = PCI_SUBDEVICE_ID_QEMU;
> @@ -2429,7 +2428,7 @@ static void pci_add_option_rom(PCIDevice *pdev, bool is_default_rom,
> pci_register_bar(pdev, PCI_ROM_SLOT, 0, &pdev->rom);
> }
>
> -static void pci_del_option_rom(PCIDevice *pdev)
> +void pci_del_option_rom(PCIDevice *pdev)
> {
> if (!pdev->has_rom)
> return;
> --
> 2.31.1
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [PATCH v4] failover: unregister ROM on unplug
2021-08-11 6:50 ` Michael S. Tsirkin
@ 2021-08-11 9:51 ` Laurent Vivier
0 siblings, 0 replies; 9+ messages in thread
From: Laurent Vivier @ 2021-08-11 9:51 UTC (permalink / raw)
To: Michael S. Tsirkin, Dr . David Alan Gilbert
Cc: Juan Quintela, Jason Wang, qemu-devel, Igor Mammedov,
Jens Freimann, Philippe Mathieu-Daudé
On 11/08/2021 08:50, Michael S. Tsirkin wrote:
> On Wed, Jul 21, 2021 at 06:09:05PM +0200, Laurent Vivier wrote:
>> The intend of failover is to allow to migrate a VM with a VFIO
>> networking card without disrupting the network operation by switching
>> to a virtio-net device during the migration.
>>
>> This simple change allows to test failover with a simulated device
>> like e1000e rather than a vfio device, even if it's useless in real
>> life it can help to debug failover.
>>
>> This is interesting to developers that want to test failover on
>> a system with no vfio device. Moreover it simplifies host networking
>> configuration as we can use the same bridge for virtio-net and
>> the other failover networking device.
>>
>> Without this change the migration of a system configured with failover
>> fails with:
>>
>> ...
>> -device virtio-net-pci,id=virtionet0,failover=on,... \
>> -device e1000,failover_pair_id=virtionet0,... \
>> ...
>>
>> (qemu) migrate ...
>>
>> Unknown ramblock "0000:00:01.1:00.0/e1000e.rom", cannot accept migration
>> error while loading state for instance 0x0 of device 'ram'
>> load of migration failed: Invalid argument
>>
>> This happens because QEMU correctly unregisters the interface vmstate but
>> not the ROM one. This patch fixes that.
>>
>> Signed-off-by: Laurent Vivier <lvivier@redhat.com>
>
> I began to wonder about this. Why are we sending the option ROM at all?
> I think there's no need to do it for the primary ...
I think there is no way to check the ROM is the same on source and destination, and it has
to be.
By sending the ROM:
- we can check the size is the same, otherwise the migration fails,
- after the migration, even if a different ROM was provided on the destination side, the
guest executes the ROM provided by the source.
Thanks,
Laurent
^ permalink raw reply [flat|nested] 9+ messages in thread