Re: [Qemu-devel] QEMU terminates during reboot after memory unplug with vhost=on

[Igor Mammedov <imammedo@redhat.com>]

On Thu, 14 Sep 2017 13:48:26 +0530
Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:

> On Thu, Sep 14, 2017 at 10:00:11AM +0200, Igor Mammedov wrote:
> > On Thu, 14 Sep 2017 12:31:18 +0530
> > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> >   
> > > Hi,
> > > 
> > > QEMU hits the below assert
> > > 
> > > qemu-system-ppc64: used ring relocated for ring 2
> > > qemu-system-ppc64: qemu/hw/virtio/vhost.c:649: vhost_commit: Assertion `r >= 0' failed.
> > > 
> > > in the following scenario:
> > > 
> > > 1. Boot guest with vhost=on
> > >   -netdev tap,id=mynet0,script=qemu-ifup,downscript=qemu-ifdown,vhost=on -device virtio-net-pci,netdev=mynet0
> > > 2. Hot add a DIMM device 
> > > 3. Reboot
> > >    When the guest reboots, we can see
> > >    vhost_virtqueue_start:vq->used_phys getting assigned an address that
> > >    falls in the hotplugged memory range.
> > > 4. Remove the DIMM device
> > >    Guest refuses the removal as the hotplugged memory is under use.
> > > 5. Reboot  
> >   
> > >    QEMU forces the removal of the DIMM device during reset and that's
> > >    when we hit the above assert.  
> > I don't recall implementing forced removal om DIMM,
> > could you point out to the related code, pls?  
> 
> This is ppc specific. We have DR Connector objects for each LMB (multiple
> LMBs make up one DIMM device) and during reset we invoke the
> release routine for these LMBs which will further invoke
> pc_dimm_memory_unplug().
> 
> See hw/ppc/spapr_drc.c: spapr_drc_reset()
> hw/ppc/spapr.c: spapr_lmb_release()
> 
> >   
> > > Any pointers on why we are hitting this assert ? Shouldn't vhost be
> > > done with using the hotplugged memory when we hit reset ?  
> >   
> > >From another point of view,  
> > DIMM shouldn't be removed unless guest explicitly ejects it
> > (at least that should be so in x86 case).  
> 
> While that is true for ppc also, shouldn't we start fresh from reset ?
we should.

when it aborts vhost should print out error from vhost_verify_ring_mappings()

   if (r == -ENOMEM) {
       error_report("Unable to map %s for ring %d", part_name[j], i);           
   } else if (r == -EBUSY) {                                                    
       error_report("%s relocated for ring %d", part_name[j], i);    

that might give a clue where that memory stuck in.

Michael might point out where to start look at, but he's on vacation
so ...

> Related comment from hw/ppc/spapr_drc.c: spapr_drc_reset()
> 
>     /* immediately upon reset we can safely assume DRCs whose devices
>      * are pending removal can be safely removed.
>      */
>     if (drc->unplug_requested) {
>         spapr_drc_release(drc);
>     }
> 
> So essentially in the scenario I listed, the unplug request is rejected
> by the guest, but during next reboot, QEMU excersies the above code
> and removes any devices (memory, CPU etc) that are marked for removal.
I'd remove pending removal on reset for DIMM/CPU if it's acceptable from
PPC HW pov.

> 
> Regards,
> Bharata.
>