* Re: [Qemu-devel] [PATCH V2] vhost: fix a migration failed becauseof vhost region merge
@ 2017-07-24 10:32 peng.hao2
2017-07-24 11:53 ` Igor Mammedov
0 siblings, 1 reply; 4+ messages in thread
From: peng.hao2 @ 2017-07-24 10:32 UTC (permalink / raw)
To: imammedo; +Cc: mst, wang.yechao255, qemu-devel
> On Sun, 23 Jul 2017 20:46:11 +0800
> Peng Hao <peng.hao2@zte.com.cn> wrote:
> > When a guest that has several hotplugged dimms is migrated, on
> > destination it will fail to resume. Because regions on source
> > are merged and on destination the order of realizing devices
> > is different from on source with dimms, so when part of devices
> > are realizd some region can not be merged.That may be more than
> > vhost slot limit.
> >
> > Signed-off-by: Peng Hao <peng.hao2@zte.com.cn>
> > Signed-off-by: Wang Yechao <wang.yechao255@zte.com.cn>
> > ---
> > hw/mem/pc-dimm.c | 2 +-
> > include/sysemu/sysemu.h | 1 +
> > vl.c | 5 +++++
> > 3 files changed, 7 insertions(+), 1 deletion(-)
> >
> > diff --git a/hw/mem/pc-dimm.c b/hw/mem/pc-dimm.c
> > index ea67b46..13f3db5 100644
> > --- a/hw/mem/pc-dimm.c
> > +++ b/hw/mem/pc-dimm.c
> > @@ -101,7 +101,7 @@ void pc_dimm_memory_plug(DeviceState *dev, MemoryHotplugState *hpms,
> > goto out
> > }
> >
> > - if (!vhost_has_free_slot()) {
> > + if (!vhost_has_free_slot() && qemu_is_machine_init_done()) {
> > error_setg(&local_err, "a used vhost backend has no free"
> > " memory slots left")
> that doesn't fix issue,
> 1st: number of used entries is changing after machine_init_done() is called
> as regions continue to mapped/unmapped during runtime
> 2nd: it brings regression and allows to start QEMU with number memory
> regions more than supported by backend, which combined with missing
> error handling in vhost will lead to qemu crashes or obscure bugs in
> guest breaking vhost enabled drivers.
> i.e. patch undoes what were fixed by
> https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg00789.html
I don't think I undo the previous patch. There are tow scenarios :
hotplugging memory happens after machine_init_done(). so the modified code is no
influence.
hotplugged memory's VM is just migrated . on source its regions is less than
supported by backend ,so on destination it should satisfy. During restoring VM's regions
may be more than supported by backend but after machine_init_done VM's regions
can be less than supported by backend.
> goto out
> diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
> index b213696..48228ad 100644
> --- a/include/sysemu/sysemu.h
> +++ b/include/sysemu/sysemu.h
> @@ -88,6 +88,7 @@ void qemu_system_guest_panicked(GuestPanicInformation *info)
> void qemu_add_exit_notifier(Notifier *notify)
> void qemu_remove_exit_notifier(Notifier *notify)
>
> +bool qemu_is_machine_init_done(void)
> void qemu_add_machine_init_done_notifier(Notifier *notify)
> void qemu_remove_machine_init_done_notifier(Notifier *notify)
>
> diff --git a/vl.c b/vl.c
> index fb6b2ef..43aee22 100644
> --- a/vl.c
> +++ b/vl.c
> @@ -2681,6 +2681,11 @@ static void qemu_run_exit_notifiers(void)
>
> static bool machine_init_done
>
> +bool qemu_is_machine_init_done(void)
> +{
> + return machine_init_done
> +}
> +
> void qemu_add_machine_init_done_notifier(Notifier *notify)
> {
> notifier_list_add(&machine_init_done_notifiers, notify)
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Qemu-devel] [PATCH V2] vhost: fix a migration failed becauseof vhost region merge
2017-07-24 10:32 [Qemu-devel] [PATCH V2] vhost: fix a migration failed becauseof vhost region merge peng.hao2
@ 2017-07-24 11:53 ` Igor Mammedov
2017-07-24 20:55 ` Michael S. Tsirkin
0 siblings, 1 reply; 4+ messages in thread
From: Igor Mammedov @ 2017-07-24 11:53 UTC (permalink / raw)
To: peng.hao2; +Cc: wang.yechao255, qemu-devel, mst
On Mon, 24 Jul 2017 18:32:35 +0800 (CST)
<peng.hao2@zte.com.cn> wrote:
> > On Sun, 23 Jul 2017 20:46:11 +0800
>
>
>
>
>
> > Peng Hao <peng.hao2@zte.com.cn> wrote:
>
> > > When a guest that has several hotplugged dimms is migrated, on
> > > destination it will fail to resume. Because regions on source
> > > are merged and on destination the order of realizing devices
> > > is different from on source with dimms, so when part of devices
> > > are realizd some region can not be merged.That may be more than
> > > vhost slot limit.
> > >
> > > Signed-off-by: Peng Hao <peng.hao2@zte.com.cn>
> > > Signed-off-by: Wang Yechao <wang.yechao255@zte.com.cn>
> > > ---
> > > hw/mem/pc-dimm.c | 2 +-
> > > include/sysemu/sysemu.h | 1 +
> > > vl.c | 5 +++++
> > > 3 files changed, 7 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/hw/mem/pc-dimm.c b/hw/mem/pc-dimm.c
> > > index ea67b46..13f3db5 100644
> > > --- a/hw/mem/pc-dimm.c
> > > +++ b/hw/mem/pc-dimm.c
> > > @@ -101,7 +101,7 @@ void pc_dimm_memory_plug(DeviceState *dev, MemoryHotplugState *hpms,
> > > goto out
> > > }
> > >
> > > - if (!vhost_has_free_slot()) {
> > > + if (!vhost_has_free_slot() && qemu_is_machine_init_done()) {
> > > error_setg(&local_err, "a used vhost backend has no free"
> > > " memory slots left")
> > that doesn't fix issue,
> > 1st: number of used entries is changing after machine_init_done() is called
> > as regions continue to mapped/unmapped during runtime
> > 2nd: it brings regression and allows to start QEMU with number memory
> > regions more than supported by backend, which combined with missing
> > error handling in vhost will lead to qemu crashes or obscure bugs in
> > guest breaking vhost enabled drivers.
> > i.e. patch undoes what were fixed by
> > https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg00789.html
> I don't think I undo the previous patch. There are tow scenarios :
>
> hotplugging memory happens after machine_init_done(). so the modified code is no
>
> influence.
>
> hotplugged memory's VM is just migrated . on source its regions is less than
>
> supported by backend ,so on destination it should satisfy. During restoring VM's regions
>
> may be more than supported by backend but after machine_init_done VM's regions
>
>
>
> can be less than supported by backend.
here is simulation with vhost-kernel where /sys/module/vhost/parameters/max_mem_regions set to 8
for limit to look like vhost user.
qemu-system-x86_64 --enable-kvm -m 128,slots=256,maxmem=1T \
-netdev type=tap,id=guest0,vhost=on,script=/bin/true,vhostforce \
-device virtio-net-pci,netdev=guest0 \
`i=0; while [ $i -lt 10 ]; do echo "-object memory-backend-ram,id=m$i,size=128M -device pc-dimm,id=d$i,memdev=m$i"; i=$(($i + 1)); done`
it end ups with 12 used_memslots, and prints following error messages:
qemu-system-x86_64: vhost_set_mem_table failed: Argument list too long (7)
qemu-system-x86_64: unable to start vhost net: 7: falling back on userspace virtio
above CLI should fail to startup as it's above supported limit even with merging
(with merging available slots is 'random' number and merging could happen regardless
of the order devices are created).
vhost_dev_init() also has checks vhost_backend_memslots_limit(),
and skipping check in pc_dimm_memory_plug() might lead to failure
later in vhost_dev_init() - I'm not sure when it's called and to what
consequences it would lead.
> > goto out
> > diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
> > index b213696..48228ad 100644
> > --- a/include/sysemu/sysemu.h
> > +++ b/include/sysemu/sysemu.h
> > @@ -88,6 +88,7 @@ void qemu_system_guest_panicked(GuestPanicInformation *info)
> > void qemu_add_exit_notifier(Notifier *notify)
> > void qemu_remove_exit_notifier(Notifier *notify)
> >
> > +bool qemu_is_machine_init_done(void)
> > void qemu_add_machine_init_done_notifier(Notifier *notify)
> > void qemu_remove_machine_init_done_notifier(Notifier *notify)
> >
> > diff --git a/vl.c b/vl.c
> > index fb6b2ef..43aee22 100644
> > --- a/vl.c
> > +++ b/vl.c
> > @@ -2681,6 +2681,11 @@ static void qemu_run_exit_notifiers(void)
> >
> > static bool machine_init_done
> >
> > +bool qemu_is_machine_init_done(void)
> > +{
> > + return machine_init_done
> > +}
> > +
> > void qemu_add_machine_init_done_notifier(Notifier *notify)
> > {
> > notifier_list_add(&machine_init_done_notifiers, notify
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Qemu-devel] [PATCH V2] vhost: fix a migration failed becauseof vhost region merge
2017-07-24 11:53 ` Igor Mammedov
@ 2017-07-24 20:55 ` Michael S. Tsirkin
0 siblings, 0 replies; 4+ messages in thread
From: Michael S. Tsirkin @ 2017-07-24 20:55 UTC (permalink / raw)
To: Igor Mammedov; +Cc: peng.hao2, wang.yechao255, qemu-devel
On Mon, Jul 24, 2017 at 01:53:33PM +0200, Igor Mammedov wrote:
> On Mon, 24 Jul 2017 18:32:35 +0800 (CST)
> <peng.hao2@zte.com.cn> wrote:
>
> > > On Sun, 23 Jul 2017 20:46:11 +0800
> >
> >
> >
> >
> >
> > > Peng Hao <peng.hao2@zte.com.cn> wrote:
> >
> > > > When a guest that has several hotplugged dimms is migrated, on
> > > > destination it will fail to resume. Because regions on source
> > > > are merged and on destination the order of realizing devices
> > > > is different from on source with dimms, so when part of devices
> > > > are realizd some region can not be merged.That may be more than
> > > > vhost slot limit.
> > > >
> > > > Signed-off-by: Peng Hao <peng.hao2@zte.com.cn>
> > > > Signed-off-by: Wang Yechao <wang.yechao255@zte.com.cn>
> > > > ---
> > > > hw/mem/pc-dimm.c | 2 +-
> > > > include/sysemu/sysemu.h | 1 +
> > > > vl.c | 5 +++++
> > > > 3 files changed, 7 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git a/hw/mem/pc-dimm.c b/hw/mem/pc-dimm.c
> > > > index ea67b46..13f3db5 100644
> > > > --- a/hw/mem/pc-dimm.c
> > > > +++ b/hw/mem/pc-dimm.c
> > > > @@ -101,7 +101,7 @@ void pc_dimm_memory_plug(DeviceState *dev, MemoryHotplugState *hpms,
> > > > goto out
> > > > }
> > > >
> > > > - if (!vhost_has_free_slot()) {
> > > > + if (!vhost_has_free_slot() && qemu_is_machine_init_done()) {
> > > > error_setg(&local_err, "a used vhost backend has no free"
> > > > " memory slots left")
> > > that doesn't fix issue,
> > > 1st: number of used entries is changing after machine_init_done() is called
> > > as regions continue to mapped/unmapped during runtime
> > > 2nd: it brings regression and allows to start QEMU with number memory
> > > regions more than supported by backend, which combined with missing
> > > error handling in vhost will lead to qemu crashes or obscure bugs in
> > > guest breaking vhost enabled drivers.
> > > i.e. patch undoes what were fixed by
> > > https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg00789.html
> > I don't think I undo the previous patch. There are tow scenarios :
> >
> > hotplugging memory happens after machine_init_done(). so the modified code is no
> >
> > influence.
> >
> > hotplugged memory's VM is just migrated . on source its regions is less than
> >
> > supported by backend ,so on destination it should satisfy. During restoring VM's regions
> >
> > may be more than supported by backend but after machine_init_done VM's regions
> >
> >
> >
> > can be less than supported by backend.
>
> here is simulation with vhost-kernel where /sys/module/vhost/parameters/max_mem_regions set to 8
> for limit to look like vhost user.
>
> qemu-system-x86_64 --enable-kvm -m 128,slots=256,maxmem=1T \
> -netdev type=tap,id=guest0,vhost=on,script=/bin/true,vhostforce \
> -device virtio-net-pci,netdev=guest0 \
> `i=0; while [ $i -lt 10 ]; do echo "-object memory-backend-ram,id=m$i,size=128M -device pc-dimm,id=d$i,memdev=m$i"; i=$(($i + 1)); done`
>
> it end ups with 12 used_memslots, and prints following error messages:
>
> qemu-system-x86_64: vhost_set_mem_table failed: Argument list too long (7)
> qemu-system-x86_64: unable to start vhost net: 7: falling back on userspace virtio
>
> above CLI should fail to startup as it's above supported limit even with merging
> (with merging available slots is 'random' number and merging could happen regardless
> of the order devices are created).
Without hotplug there is not need to fail it early at all.
It should fail in vhost, need to debug it - falling back should
only happen when vhost is not forced.
>
> vhost_dev_init() also has checks vhost_backend_memslots_limit(),
> and skipping check in pc_dimm_memory_plug() might lead to failure
> later in vhost_dev_init() - I'm not sure when it's called and to what
> consequences it would lead.
It should fail cleanly. Only reason for your patch is for memory hotplug
where it's too late to stop vhost.
> > > goto out
> > > diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
> > > index b213696..48228ad 100644
> > > --- a/include/sysemu/sysemu.h
> > > +++ b/include/sysemu/sysemu.h
> > > @@ -88,6 +88,7 @@ void qemu_system_guest_panicked(GuestPanicInformation *info)
> > > void qemu_add_exit_notifier(Notifier *notify)
> > > void qemu_remove_exit_notifier(Notifier *notify)
> > >
> > > +bool qemu_is_machine_init_done(void)
> > > void qemu_add_machine_init_done_notifier(Notifier *notify)
> > > void qemu_remove_machine_init_done_notifier(Notifier *notify)
> > >
> > > diff --git a/vl.c b/vl.c
> > > index fb6b2ef..43aee22 100644
> > > --- a/vl.c
> > > +++ b/vl.c
> > > @@ -2681,6 +2681,11 @@ static void qemu_run_exit_notifiers(void)
> > >
> > > static bool machine_init_done
> > >
> > > +bool qemu_is_machine_init_done(void)
> > > +{
> > > + return machine_init_done
> > > +}
> > > +
> > > void qemu_add_machine_init_done_notifier(Notifier *notify)
> > > {
> > > notifier_list_add(&machine_init_done_notifiers, notify
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Qemu-devel] [PATCH V2] vhost: fix a migration failed becauseof vhost region merge
@ 2017-07-28 14:21 peng.hao2
0 siblings, 0 replies; 4+ messages in thread
From: peng.hao2 @ 2017-07-28 14:21 UTC (permalink / raw)
To: imammedo; +Cc: mst, wang.yechao255, qemu-devel, pbonzini
>On Wed, 26 Jul 2017 19:01:39 +0300
>"Michael S. Tsirkin" <mst@redhat.com> wrote:
>> On Wed, Jul 26, 2017 at 04:05:43PM +0200, Igor Mammedov wrote:
>> > On Tue, 25 Jul 2017 22:47:18 +0300
>> > "Michael S. Tsirkin" <mst@redhat.com> wrote:
>> >
>> > > On Tue, Jul 25, 2017 at 10:44:38AM +0200, Igor Mammedov wrote:
>> > > > On Mon, 24 Jul 2017 23:50:00 +0300
>> > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
>> > > >
>> > > > > On Mon, Jul 24, 2017 at 11:14:19AM +0200, Igor Mammedov wrote:
>> > > > > > On Sun, 23 Jul 2017 20:46:11 +0800
>> > > > > > Peng Hao <peng.hao2@zte.com.cn> wrote:
>> > > > > >
>> > > > > > > When a guest that has several hotplugged dimms is migrated, on
>> > > > > > > destination it will fail to resume. Because regions on source
>> > > > > > > are merged and on destination the order of realizing devices
>> > > > > > > is different from on source with dimms, so when part of devices
>> > > > > > > are realizd some region can not be merged.That may be more than
>> > > > > > > vhost slot limit.
>> > > > > > >
>> > > > > > > Signed-off-by: Peng Hao <peng.hao2@zte.com.cn>
>> > > > > > > Signed-off-by: Wang Yechao <wang.yechao255@zte.com.cn>
>> > > > > > > ---
>> > > > > > > hw/mem/pc-dimm.c | 2 +-
>> > > > > > > include/sysemu/sysemu.h | 1 +
>> > > > > > > vl.c | 5 +++++
>> > > > > > > 3 files changed, 7 insertions(+), 1 deletion(-)
>> > > > > > >
>> > > > > > > diff --git a/hw/mem/pc-dimm.c b/hw/mem/pc-dimm.c
>> > > > > > > index ea67b46..13f3db5 100644
>> > > > > > > --- a/hw/mem/pc-dimm.c
>> > > > > > > +++ b/hw/mem/pc-dimm.c
>> > > > > > > @@ -101,7 +101,7 @@ void pc_dimm_memory_plug(DeviceState *dev, MemoryHotplugState *hpms,
>> > > > > > > goto out
>> > > > > > > }
>> > > > > > >
>> > > > > > > - if (!vhost_has_free_slot()) {
>> > > > > > > + if (!vhost_has_free_slot() && qemu_is_machine_init_done()) {
>> > > > > > > error_setg(&local_err, "a used vhost backend has no free"
>> > > > > > > " memory slots left")
>> > > > > > that doesn't fix issue,
>> > > > > > 1st: number of used entries is changing after machine_init_done() is called
>> > > > > > as regions continue to mapped/unmapped during runtime
>> > > > >
>> > > > > But that's fine, we want hotplug to fail if we can not guarantee vhost
>> > > > > will work.
>> > > > don't we want guarantee that vhost will work with dimm devices at startup
>> > > > if it were requested on CLI or fail startup cleanly if it can't?
>> > >
>> > > Yes. And failure to start vhost will achieve this without need to much with
>> > > DIMMs. The issue is only with DIMM hotplug when vhost is already running,
>> > > specifically because notifiers have no way to report or handle errors.
>> > >
>> > > > > > 2nd: it brings regression and allows to start QEMU with number memory
>> > > > > > regions more than supported by backend, which combined with missing
>> > > > > > error handling in vhost will lead to qemu crashes or obscure bugs in
>> > > > > > guest breaking vhost enabled drivers.
>> > > > > > i.e. patch undoes what were fixed by
>> > > > > > https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg00789.html
>> > > > >
>> > > > > Why does it? The issue you fixed there is hotplug, and that means
>> > > > > pc_dimm_memory_plug called after machine done.
>> > > > I wasn't able to crash fc24 guest with current qemu/rhen7 kernel,
>> > > > it fallbacks back to virtio and switches off vhost.
>> > >
>> > > I think vhostforce should make vhost fail and not fall back,
>> > > but that is another bug.
>> > currently vhostforce is broken, qemu continues to happily run with this patch
>> > and without patch it fails to start up so I'd just NACK this patch
>> > on this behavioral change and ask to fix both issues in the same series.
>>
>> Please do not send nacks. They are not really helpful.
>>
>> Ack is like +1. You save some space since all you are saying is "all's
>> well". But if there's an issue you want to explain what it is 99% of the
>> time. So nack does not save any space and just pushes contributors away.
>> Especially if it's in all caps, that's just against netiquette.
>I'm sorry to the author if it were taken as offense,
>an intent was to say that by itself patch allows to start QEMU in invalid
>configuration and that it should be fixed as well.
>Anyway I've just posted an alternative patch that should workaround issue
>at the hand while not removing check:
>[PATCH for 2.10] pc: make 'pc.rom' readonly when machine has PCI enabled
>Peng Hao,
>could you check if it solves the problem for you
yes,it works. I never think of it before.
Thanks. Mst,too.
.
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2017-07-28 14:22 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-07-24 10:32 [Qemu-devel] [PATCH V2] vhost: fix a migration failed becauseof vhost region merge peng.hao2
2017-07-24 11:53 ` Igor Mammedov
2017-07-24 20:55 ` Michael S. Tsirkin
-- strict thread matches above, loose matches on Subject: below --
2017-07-28 14:21 peng.hao2
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).