Re: [PATCH RFC V9 0/19] Paravirtualized ticket spinlocks

[Gleb Natapov <gleb@redhat.com>]

On Tue, Jul 09, 2013 at 02:41:30PM +0530, Raghavendra K T wrote:
> On 06/26/2013 11:24 PM, Raghavendra K T wrote:
> >On 06/26/2013 09:41 PM, Gleb Natapov wrote:
> >>On Wed, Jun 26, 2013 at 07:10:21PM +0530, Raghavendra K T wrote:
> >>>On 06/26/2013 06:22 PM, Gleb Natapov wrote:
> >>>>On Wed, Jun 26, 2013 at 01:37:45PM +0200, Andrew Jones wrote:
> >>>>>On Wed, Jun 26, 2013 at 02:15:26PM +0530, Raghavendra K T wrote:
> >>>>>>On 06/25/2013 08:20 PM, Andrew Theurer wrote:
> >>>>>>>On Sun, 2013-06-02 at 00:51 +0530, Raghavendra K T wrote:
> >>>>>>>>This series replaces the existing paravirtualized spinlock
> >>>>>>>>mechanism
> >>>>>>>>with a paravirtualized ticketlock mechanism. The series provides
> >>>>>>>>implementation for both Xen and KVM.
> >>>>>>>>
> >>>>>>>>Changes in V9:
> >>>>>>>>- Changed spin_threshold to 32k to avoid excess halt exits that are
> >>>>>>>>    causing undercommit degradation (after PLE handler
> >>>>>>>>improvement).
> >>>>>>>>- Added  kvm_irq_delivery_to_apic (suggested by Gleb)
> >>>>>>>>- Optimized halt exit path to use PLE handler
> >>>>>>>>
> >>>>>>>>V8 of PVspinlock was posted last year. After Avi's suggestions
> >>>>>>>>to look
> >>>>>>>>at PLE handler's improvements, various optimizations in PLE
> >>>>>>>>handling
> >>>>>>>>have been tried.
> >>>>>>>
> >>>>>>>Sorry for not posting this sooner.  I have tested the v9
> >>>>>>>pv-ticketlock
> >>>>>>>patches in 1x and 2x over-commit with 10-vcpu and 20-vcpu VMs.  I
> >>>>>>>have
> >>>>>>>tested these patches with and without PLE, as PLE is still not
> >>>>>>>scalable
> >>>>>>>with large VMs.
> >>>>>>>
> >>>>>>
> >>>>>>Hi Andrew,
> >>>>>>
> >>>>>>Thanks for testing.
> >>>>>>
> >>>>>>>System: x3850X5, 40 cores, 80 threads
> >>>>>>>
> >>>>>>>
> >>>>>>>1x over-commit with 10-vCPU VMs (8 VMs) all running dbench:
> >>>>>>>----------------------------------------------------------
> >>>>>>>                        Total
> >>>>>>>Configuration                Throughput(MB/s)    Notes
> >>>>>>>
> >>>>>>>3.10-default-ple_on            22945            5% CPU in host
> >>>>>>>kernel, 2% spin_lock in guests
> >>>>>>>3.10-default-ple_off            23184            5% CPU in host
> >>>>>>>kernel, 2% spin_lock in guests
> >>>>>>>3.10-pvticket-ple_on            22895            5% CPU in host
> >>>>>>>kernel, 2% spin_lock in guests
> >>>>>>>3.10-pvticket-ple_off            23051            5% CPU in host
> >>>>>>>kernel, 2% spin_lock in guests
> >>>>>>>[all 1x results look good here]
> >>>>>>
> >>>>>>Yes. The 1x results look too close
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>2x over-commit with 10-vCPU VMs (16 VMs) all running dbench:
> >>>>>>>-----------------------------------------------------------
> >>>>>>>                        Total
> >>>>>>>Configuration                Throughput        Notes
> >>>>>>>
> >>>>>>>3.10-default-ple_on             6287            55% CPU  host
> >>>>>>>kernel, 17% spin_lock in guests
> >>>>>>>3.10-default-ple_off             1849            2% CPU in host
> >>>>>>>kernel, 95% spin_lock in guests
> >>>>>>>3.10-pvticket-ple_on             6691            50% CPU in host
> >>>>>>>kernel, 15% spin_lock in guests
> >>>>>>>3.10-pvticket-ple_off            16464            8% CPU in host
> >>>>>>>kernel, 33% spin_lock in guests
> >>>>>>
> >>>>>>I see 6.426% improvement with ple_on
> >>>>>>and 161.87% improvement with ple_off. I think this is a very good
> >>>>>>sign
> >>>>>>  for the patches
> >>>>>>
> >>>>>>>[PLE hinders pv-ticket improvements, but even with PLE off,
> >>>>>>>  we still off from ideal throughput (somewhere >20000)]
> >>>>>>>
> >>>>>>
> >>>>>>Okay, The ideal throughput you are referring is getting around
> >>>>>>atleast
> >>>>>>80% of 1x throughput for over-commit. Yes we are still far away from
> >>>>>>there.
> >>>>>>
> >>>>>>>
> >>>>>>>1x over-commit with 20-vCPU VMs (4 VMs) all running dbench:
> >>>>>>>----------------------------------------------------------
> >>>>>>>                        Total
> >>>>>>>Configuration                Throughput        Notes
> >>>>>>>
> >>>>>>>3.10-default-ple_on            22736            6% CPU in host
> >>>>>>>kernel, 3% spin_lock in guests
> >>>>>>>3.10-default-ple_off            23377            5% CPU in host
> >>>>>>>kernel, 3% spin_lock in guests
> >>>>>>>3.10-pvticket-ple_on            22471            6% CPU in host
> >>>>>>>kernel, 3% spin_lock in guests
> >>>>>>>3.10-pvticket-ple_off            23445            5% CPU in host
> >>>>>>>kernel, 3% spin_lock in guests
> >>>>>>>[1x looking fine here]
> >>>>>>>
> >>>>>>
> >>>>>>I see ple_off is little better here.
> >>>>>>
> >>>>>>>
> >>>>>>>2x over-commit with 20-vCPU VMs (8 VMs) all running dbench:
> >>>>>>>----------------------------------------------------------
> >>>>>>>                        Total
> >>>>>>>Configuration                Throughput        Notes
> >>>>>>>
> >>>>>>>3.10-default-ple_on             1965            70% CPU in host
> >>>>>>>kernel, 34% spin_lock in guests
> >>>>>>>3.10-default-ple_off              226            2% CPU in host
> >>>>>>>kernel, 94% spin_lock in guests
> >>>>>>>3.10-pvticket-ple_on             1942            70% CPU in host
> >>>>>>>kernel, 35% spin_lock in guests
> >>>>>>>3.10-pvticket-ple_off             8003            11% CPU in host
> >>>>>>>kernel, 70% spin_lock in guests
> >>>>>>>[quite bad all around, but pv-tickets with PLE off the best so far.
> >>>>>>>  Still quite a bit off from ideal throughput]
> >>>>>>
> >>>>>>This is again a remarkable improvement (307%).
> >>>>>>This motivates me to add a patch to disable ple when pvspinlock is
> >>>>>>on.
> >>>>>>probably we can add a hypercall that disables ple in kvm init patch.
> >>>>>>but only problem I see is what if the guests are mixed.
> >>>>>>
> >>>>>>  (i.e one guest has pvspinlock support but other does not. Host
> >>>>>>supports pv)
> >>>>>
> >>>>>How about reintroducing the idea to create per-kvm ple_gap,ple_window
> >>>>>state. We were headed down that road when considering a dynamic
> >>>>>window at
> >>>>>one point. Then you can just set a single guest's ple_gap to zero,
> >>>>>which
> >>>>>would lead to PLE being disabled for that guest. We could also revisit
> >>>>>the dynamic window then.
> >>>>>
> >>>>Can be done, but lets understand why ple on is such a big problem.
> >>>>Is it
> >>>>possible that ple gap and SPIN_THRESHOLD are not tuned properly?
> >>>>
> >>>
> >>>The one obvious reason I see is commit awareness inside the guest. for
> >>>under-commit there is no necessity to do PLE, but unfortunately we do.
> >>>
> >>>atleast we return back immediately in case of potential undercommits,
> >>>but we still incur vmexit delay.
> >>But why do we? If SPIN_THRESHOLD will be short enough (or ple windows
> >>long enough) to not generate PLE exit we will not go into PLE handler
> >>at all, no?
> >>
> >
> >Yes. you are right. dynamic ple window was an attempt to solve it.
> >
> >Probelm is, reducing the SPIN_THRESHOLD is resulting in excess halt
> >exits in under-commits and increasing ple_window may be sometimes
> >counter productive as it affects other busy-wait constructs such as
> >flush_tlb AFAIK.
> >So if we could have had a dynamically changing SPIN_THRESHOLD too, that
> >would be nice.
> >
> 
> Gleb, Andrew,
> I tested with the global ple window change (similar to what I posted
> here https://lkml.org/lkml/2012/11/11/14 ),
This does not look global. It changes PLE per vcpu.

> But did not see good result. May be it is good to go with per VM
> ple_window.
> 
> Gleb,
> Can you elaborate little more on what you have in mind regarding per
> VM ple_window. (maintaining part of it as a per vm variable is clear
> to
>  me), but is it that we have to load that every time of guest entry?
> 
Only when it changes, shouldn't be to often no?

> I 'll try that idea next.
> 
> Ingo, Gleb,
> 
> From the results perspective, Andrew Theurer, Vinod's test results are
> pro-pvspinlock.
> Could you please help me to know what will make it a mergeable
> candidate?.
> 
I need to spend more time reviewing it :) The problem with PV interfaces
is that they are easy to add but hard to get rid of if better solution
(HW or otherwise) appears.

> I agree that Jiannan's Preemptable Lock idea is promising and we could
> evaluate that  approach, and make the best one get into kernel and also
> will carry on discussion with Jiannan to improve that patch.
That would be great. The work is stalled from what I can tell.

> Experiments so far have been good for smaller machine but it is not
> scaling for bigger machines.

--
			Gleb.