RCU stall with high number of KVM vcpus

[jan.glauber@caviumnetworks.com (Jan Glauber)]

On Mon, Nov 13, 2017 at 06:13:08PM +0000, Shameerali Kolothum Thodi wrote:

[...]

> > > > numbers don't look good, see waittime-max:
> > > >
> > > > ---------------------------------------------------------------------------------------------------
> > -------------------------------------------------------------------------------------------------------
> > -------------------
> > > >                               class name    con-bounces    contentions   waittime-min
> > waittime-max waittime-total   waittime-avg    acq-bounces   acquisitions
> > holdtime-min   holdtime-max holdtime-total   holdtime-avg
> > > > ---------------------------------------------------------------------------------------------------
> > -------------------------------------------------------------------------------------------------------
> > -------------------
> > > >
> > > >                 &(&kvm->mmu_lock)->rlock:      99346764       99406604
> > 0.14  1321260806.59 710654434972.0        7148.97      154228320
> > 225122857           0.13   917688890.60  3705916481.39          16.46
> > > >                 ------------------------
> > > >                 &(&kvm->mmu_lock)->rlock       99365598
> > [<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950
> > > >                 &(&kvm->mmu_lock)->rlock          25164
> > [<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8
> > > >                 &(&kvm->mmu_lock)->rlock          14934
> > [<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68
> > > >                 &(&kvm->mmu_lock)->rlock            908
> > [<ffff00000810a1f0>] __cond_resched_lock+0x68/0xb8
> > > >                 ------------------------
> > > >                 &(&kvm->mmu_lock)->rlock              3          [<ffff0000080b34c8>]
> > stage2_flush_vm+0x60/0xd8
> > > >                 &(&kvm->mmu_lock)->rlock       99186296
> > [<ffff0000080b43b8>] kvm_handle_guest_abort+0x4c0/0x950
> > > >                 &(&kvm->mmu_lock)->rlock         179238
> > [<ffff0000080a4e30>] kvm_mmu_notifier_invalidate_range_start+0x70/0xe8
> > > >                 &(&kvm->mmu_lock)->rlock          19181
> > [<ffff0000080a7eec>] kvm_mmu_notifier_invalidate_range_end+0x24/0x68
> 
> That looks like something similar we had on our hip07 platform when multiple VMs
> were launched.  The issue was tracked down to CONFIG_NUMA set with memory_less
> nodes. This results in lot of individual 4K pages and unmap_stage2_ptes() takes a good
> amount of time coupled with some HW cache flush latencies. I am not sure you are
> seeing the same thing, but may be worth checking.

Hi Shameer,

thanks for the tip. We don't have memory-less nodes but it might me
related to NUMA. I've tried putting the guest onto one node but that did
not help.

PID                               Node 0          Node 1           Total
-----------------------  --------------- --------------- ---------------
56753 (qemu-nbd)                    4.48           11.16           15.64
56813 (qemu-system-aar)             2.02         1685.72         1687.75
-----------------------  --------------- --------------- ---------------
Total                               6.51         1696.88         1703.39

I'll try switching to 64K pages in the host next.

thanks,
Jan