[Qemu-devel] 答复: Re: [PATCH] vhost: fix a migration failed because ofvhost region merge

[Qemu-devel] 答复: Re: [PATCH] vhost: fix a migration failed because ofvhost region merge

[peng.hao2]

原始邮件



发件人： <mst@redhat.com>
收件人： <imammedo@redhat.com>
抄送人： <dgilbert@redhat.com> <marcandre.lureau@redhat.com> <maxime.coquelin@redhat.com>彭浩10096742王业超10154425 <qemu-devel@nongnu.org>
日 期 ：2017年07月19日 23:53
主 题 ：Re: [Qemu-devel] [PATCH] vhost: fix a migration failed because ofvhost region merge





On Wed, Jul 19, 2017 at 03:24:27PM +0200, Igor Mammedov wrote:
> On Wed, 19 Jul 2017 12:46:13 +0100
> "Dr. David Alan Gilbert" <dgilbert@redhat.com> wrote:
> 
> > * Igor Mammedov (imammedo@redhat.com) wrote:
> > > On Wed, 19 Jul 2017 23:17:32 +0800
> > > Peng Hao <peng.hao2@zte.com.cn> wrote:
> > >   
> > > > When a guest that has several hotplugged dimms is migrated, in
> > > > destination host it will fail to resume. Because vhost regions of
> > > > several dimms in source host are merged and in the restore stage
> > > > in destination host it computes whether more than vhost slot limit
> > > > before merging vhost regions of several dimms.  
> > > could you provide a bit more detailed description of the problem
> > > including command line+used device_add commands on source and
> > > command line on destination?  
> > 
> > (ccing in Marc Andre and Maxime)
> > 
> > Hmm, I'd like to understade the situation where you get merging between
> > RAMBlocks that complicates some stuff for postcopy.
> and probably inconsistent merging breaks vhost as well
> 
> merging might happen if regions are adjacent or overlap
> but for that to happen merged regions must have equal
> distance between their GPA:HVA pairs, so that following
> translation would work:
> 
> if gva in regionX[gva_start, len, hva_start]
>    hva = hva_start + gva - gva_start
> 
> while GVA of regions is under QEMU control and deterministic
> HVA is not, so in migration case merging might happen on source
> side but not on destination, resulting in different memory maps.
> 
> Maybe Michael might know details why migration works in vhost usecase,
> but I don't see vhost sending any vmstate data.

We aren't merging ramblocks at all.
When we are passing blocks A and B to vhost, if we see that

hvaB=hvaA + lenA
gpaB=gpaA + lenA

then we can improve performance a bit by passing a single
chunk to vhost: hvaA,gpaA,lena+lenB

so it does not affect migration normally.

----- I think it is like this:

in source                                       in destination:(restore)

realize device 1                              realize device 1

realize device 2                              realize dimm 0

     ...                                               realize dimm1

                                                       ....

realize device n                              realize dimmx

                                                       realize  device m

realize dimm0                                  .....

realize dimm1                                  .....

......                                                  .....

realize dimmx                                realize device n




In restore stage ,the sort of realizing device  is different from starting vm because of adding dimms.

So it may in some stage during restoring can't merge vhost regions.














> 
> > 
> > > > 
> > > > Signed-off-by: Peng Hao <peng.hao2@zte.com.cn>
> > > > Signed-off-by: Wang Yechao <wang.yechao255@zte.com.cn>
> > > > ---
> > > >  hw/mem/pc-dimm.c | 2 +-
> > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > > 
> > > > diff --git a/hw/mem/pc-dimm.c b/hw/mem/pc-dimm.c
> > > > index ea67b46..bb0fa08 100644
> > > > --- a/hw/mem/pc-dimm.c
> > > > +++ b/hw/mem/pc-dimm.c
> > > > @@ -101,7 +101,7 @@ void pc_dimm_memory_plug(DeviceState *dev, MemoryHotplugState *hpms,
> > > >          goto out
> > > >      }
> > > >  
> > > > -    if (!vhost_has_free_slot()) {
> > > > +    if (!vhost_has_free_slot() && runstate_is_running()) {
> > > >          error_setg(&local_err, "a used vhost backend has no free"
> > > >                                 " memory slots left")
> > > >          goto out  
> > 
> > Even this produces the wrong error message in this case,
> > it also makes me think if the existing code should undo a lot of
> > the object_property_set's that happen.
> > 
> > Dave
> > > 
> > >   
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] 答复: Re: [PATCH] vhost: fix a migration failed because ofvhost region merge

[Michael S. Tsirkin]

On Thu, Jul 20, 2017 at 10:57:57AM +0800, peng.hao2@zte.com.cn wrote:
> 原始邮件
> 发件人： <mst@redhat.com>;
> 收件人： <imammedo@redhat.com>;
> 抄送人： <dgilbert@redhat.com>; <marcandre.lureau@redhat.com>;
> <maxime.coquelin@redhat.com>;彭浩10096742;王业超10154425;
> <qemu-devel@nongnu.org>;
> 日期：2017年07月19日 23:53
> 主题：Re: [Qemu-devel] [PATCH] vhost: fix a migration failed because ofvhost
> region merge
> 
> 
> On Wed, Jul 19, 2017 at 03:24:27PM +0200, Igor Mammedov wrote:
> > On Wed, 19 Jul 2017 12:46:13 +0100
> > "Dr. David Alan Gilbert" <dgilbert@redhat.com> wrote:
> > 
> > > * Igor Mammedov (imammedo@redhat.com) wrote:
> > > > On Wed, 19 Jul 2017 23:17:32 +0800
> > > > Peng Hao <peng.hao2@zte.com.cn> wrote:
> > > >   
> > > > > When a guest that has several hotplugged dimms is migrated, in
> > > > > destination host it will fail to resume. Because vhost regions of
> > > > > several dimms in source host are merged and in the restore stage
> > > > > in destination host it computes whether more than vhost slot limit
> > > > > before merging vhost regions of several dimms.  
> > > > could you provide a bit more detailed description of the problem
> > > > including command line+used device_add commands on source and
> > > > command line on destination?  
> > > 
> > > (ccing in Marc Andre and Maxime)
> > > 
> > > Hmm, I'd like to understade the situation where you get merging between
> > > RAMBlocks; that complicates some stuff for postcopy.
> > and probably inconsistent merging breaks vhost as well
> > 
> > merging might happen if regions are adjacent or overlap
> > but for that to happen merged regions must have equal
> > distance between their GPA:HVA pairs, so that following
> > translation would work:
> > 
> > if gva in regionX[gva_start, len, hva_start]
> >    hva = hva_start + gva - gva_start
> > 
> > while GVA of regions is under QEMU control and deterministic
> > HVA is not, so in migration case merging might happen on source
> > side but not on destination, resulting in different memory maps.
> > 
> > Maybe Michael might know details why migration works in vhost usecase,
> > but I don't see vhost sending any vmstate data.
> 
> We aren't merging ramblocks at all.
> When we are passing blocks A and B to vhost, if we see that
> 
> hvaB=hvaA + lenA
> gpaB=gpaA + lenA
> 
> then we can improve performance a bit by passing a single
> chunk to vhost: hvaA,gpaA,lena+lenB
> 
> 
> so it does not affect migration normally.
> 
> ----- I think it is like this:
> 
> in source;                                       in destination:(restore)
> 
> realize device 1                              realize device 1
> 
> realize device 2                              realize dimm 0
> 
>      ...                                               realize dimm1
> 
>                                                        ....
> 
> realize device n                              realize dimmx
> 
>                                                        realize  device m
> 
> realize dimm0                                  .....
> 
> realize dimm1                                  .....
> 
> ......                                                  .....
> 
> realize dimmx                                realize device n
> 
> 
> In restore stage ,the sort of realizing device  is different from starting vm
> because of adding dimms.
> 
> So it may in some stage during restoring can't merge vhost regions.

If you run over the number of regions supported by vhost on destination
then you won't be able to start a VM there until you disable vhost.



> 
> 
> 
> 
> 
> > 
> > > 
> > > > > 
> > > > > Signed-off-by: Peng Hao <peng.hao2@zte.com.cn>
> > > > > Signed-off-by: Wang Yechao <wang.yechao255@zte.com.cn>
> > > > > ---
> > > > >  hw/mem/pc-dimm.c | 2 +-
> > > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > > > 
> > > > > diff --git a/hw/mem/pc-dimm.c b/hw/mem/pc-dimm.c
> > > > > index ea67b46..bb0fa08 100644
> > > > > --- a/hw/mem/pc-dimm.c
> > > > > +++ b/hw/mem/pc-dimm.c
> > > > > @@ -101,7 +101,7 @@ void pc_dimm_memory_plug
> (DeviceState *dev, MemoryHotplugState *hpms,
> > > > >          goto out;
> > > > >      }
> > > > >  
> > > > > -    if (!vhost_has_free_slot()) {
> > > > > +    if (!vhost_has_free_slot() && runstate_is_running()) {
> > > > >          error_setg(&local_err, "a used vhost backend has no free"
> > > > >                                 " memory slots left");
> > > > >          goto out;  
> > > 
> > > Even this produces the wrong error message in this case,
> > > it also makes me think if the existing code should undo a lot of
> > > the object_property_set's that happen.
> > > 
> > > Dave
> > > > 
> > > >   
> > > --
> > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> 
> 
> 

[Qemu-devel] 答复: Re: [PATCH] vhost: fix a migration failed because ofvhost region merge

[peng.hao2]

* Michael S. Tsirkin (mst@redhat.com) wrote:





>> On Wed, Jul 19, 2017 at 03:24:27PM +0200, Igor Mammedov wrote:
> > > On Wed, 19 Jul 2017 12:46:13 +0100
> > > "Dr. David Alan Gilbert" <dgilbert@redhat.com> wrote:
> > > 
> > > > * Igor Mammedov (imammedo@redhat.com) wrote:
> > > > > On Wed, 19 Jul 2017 23:17:32 +0800
> > > > > Peng Hao <peng.hao2@zte.com.cn> wrote:
> > > > >   
> > > > > > When a guest that has several hotplugged dimms is migrated, in
> > > > > > destination host it will fail to resume. Because vhost regions of
> > > > > > several dimms in source host are merged and in the restore stage
> > > > > > in destination host it computes whether more than vhost slot limit
> > > > > > before merging vhost regions of several dimms.  
> > > > > could you provide a bit more detailed description of the problem
> > > > > including command line+used device_add commands on source and
> > > > > command line on destination?  
> > > > 
> > > > (ccing in Marc Andre and Maxime)
> > > > 
> > > > Hmm, I'd like to understade the situation where you get merging between
> > > > RAMBlocks that complicates some stuff for postcopy.
> > > and probably inconsistent merging breaks vhost as well
> > > 
> > > merging might happen if regions are adjacent or overlap
> > > but for that to happen merged regions must have equal
> > > distance between their GPA:HVA pairs, so that following
> > > translation would work:
> > > 
> > > if gva in regionX[gva_start, len, hva_start]
> > >    hva = hva_start + gva - gva_start
> > > 
> > > while GVA of regions is under QEMU control and deterministic
> > > HVA is not, so in migration case merging might happen on source
> > > side but not on destination, resulting in different memory maps.
> > > 
> > > Maybe Michael might know details why migration works in vhost usecase,
> > > but I don't see vhost sending any vmstate data.
> > 
>> We aren't merging ramblocks at all.
>> When we are passing blocks A and B to vhost, if we see that
>> 
>> hvaB=hvaA + lenA
>> gpaB=gpaA + lenA
>> 
>> then we can improve performance a bit by passing a single
>> chunk to vhost: hvaA,gpaA,lena+lenB

>OK, but that means that a region can incorporate multiple
>RAMBlocks though? Hmm that's not fun on postcopy.

>> so it does not affect migration normally.

>Well, why? What's required - if the region sizes/lengths/orders
>are different on the source and destination does it matter - if
>it does then that means we have a problem, since that heuristic
>is non-deterministic.
Yes,I encounter the different  region creation orders  on the source and destination

if you add dimms on the source before migration. so I just distinguish according to 

vm state.

>Dave


> 
> > 
> > > 
> > > > > 
> > > > > Signed-off-by: Peng Hao <peng.hao2@zte.com.cn>
> > > > > Signed-off-by: Wang Yechao <wang.yechao255@zte.com.cn>
> > > > > ---
> > > > >  hw/mem/pc-dimm.c | 2 +-
> > > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > > > 
> > > > > diff --git a/hw/mem/pc-dimm.c b/hw/mem/pc-dimm.c
> > > > > index ea67b46..bb0fa08 100644
> > > > > --- a/hw/mem/pc-dimm.c
> > > > > +++ b/hw/mem/pc-dimm.c
> > > > > @@ -101,7 +101,7 @@ void pc_dimm_memory_plug(DeviceState *dev, MemoryHotplugState *hpms,
> > > > >          goto out
> > > > >      }
> > > > >  
> > > > > -    if (!vhost_has_free_slot()) {
> > > > > +    if (!vhost_has_free_slot() && runstate_is_running()) {
> > > > >          error_setg(&local_err, "a used vhost backend has no free"
> > > > >                                 " memory slots left")
> > > > >          goto out  
> > > 
> > > Even this produces the wrong error message in this case,
> > > it also makes me think if the existing code should undo a lot of
> > > the object_property_set's that happen.
> > > 
> > > Dave
> > > > 
> > > >   
> > > --
> > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

[Qemu-devel] 答复: Re: [PATCH] vhost: fix a migration failed because ofvhost region merge

[peng.hao2]

发件人： <imammedo@redhat.com>
收件人：彭浩10096742
抄送人： <mst@redhat.com>王业超10154425 <qemu-devel@nongnu.org>
日 期 ：2017年07月19日 15:51
主 题 ：Re: [Qemu-devel] [PATCH] vhost: fix a migration failed because ofvhost region merge





On Wed, 19 Jul 2017 23:17:32 +0800
Peng Hao <peng.hao2@zte.com.cn> wrote:

> When a guest that has several hotplugged dimms is migrated, in
> destination host it will fail to resume. Because vhost regions of
> several dimms in source host are merged and in the restore stage
> in destination host it computes whether more than vhost slot limit
> before merging vhost regions of several dimms.
could you provide a bit more detailed description of the problem
including command line+used device_add commands on source and
command line on destination?

-------I just migrate guest vm. The guest has vhost-user net. Vhost-user has a limit of slots

 (VHOST_MEMORY_MAX_NREGIONS=8),hotplug memory (dimm) can increase vhost used slots

,but vhost can merge region to decrease vhost used slots.

Before migration guset has hotplugged  several dimms one by one (more than vhost slot limit,

but after merging it less than  vhost slot limit). In migrating to destination host ,qemu begins to 

execute restore guest and it realize  foreach  dimms device at a time and 


compute whether vhost used slots more than vhost slot limit before merging.


So report "a used vhost backend has no free  memory slots left".


I just make a distinction using a vm state.

> 
> Signed-off-by: Peng Hao <peng.hao2@zte.com.cn>
> Signed-off-by: Wang Yechao <wang.yechao255@zte.com.cn>
> ---
>  hw/mem/pc-dimm.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/hw/mem/pc-dimm.c b/hw/mem/pc-dimm.c
> index ea67b46..bb0fa08 100644
> --- a/hw/mem/pc-dimm.c
> +++ b/hw/mem/pc-dimm.c
> @@ -101,7 +101,7 @@ void pc_dimm_memory_plug(DeviceState *dev, MemoryHotplugState *hpms,
>          goto out
>      }
>  
> -    if (!vhost_has_free_slot()) {
> +    if (!vhost_has_free_slot() && runstate_is_running()) {
>          error_setg(&local_err, "a used vhost backend has no free"
>                                 " memory slots left")
>          goto out