Re: Fw: Benchmarking for vhost polling patch

["Michael S. Tsirkin" <mst@redhat.com>]

Hi Razya,
Thanks for the update.
So that's reasonable I think, and I think it makes sense
to keep working on this in isolation - it's more
manageable at this size.

The big questions in my mind:
- What happens if system is lightly loaded?
  E.g. a ping/pong benchmark. How much extra CPU are
  we wasting?
- We see the best performance on your system is with 10usec worth of polling.
  It's OK to be able to tune it for best performance, but
  most people don't have the time or the inclination.
  So what would be the best value for other CPUs?
- Should this be tunable from usespace per vhost instance?
  Why is it only tunable globally?
- How bad is it if you don't pin vhost and vcpu threads?
  Is the scheduler smart enough to pull them apart?
- What happens in overcommit scenarios? Does polling make things
  much worse?
  Clearly polling will work worse if e.g. vhost and vcpu
  share the host cpu. How can we avoid conflicts?

  For two last questions, better cooperation with host scheduler will
  likely help here.
  See e.g.  http://thread.gmane.org/gmane.linux.kernel/1771791/focus=1772505
  I'm currently looking at pushing something similar upstream,
  if it goes in vhost polling can do something similar.

Any data points to shed light on these questions?

On Thu, Jan 01, 2015 at 02:59:21PM +0200, Razya Ladelsky wrote:
> Hi Michael,
> Just a follow up on the polling patch numbers,..
> Please let me know if you find these numbers satisfying enough to continue 
> with submitting this patch.
> Otherwise - we'll have this patch submitted as part of the larger Elvis 
> patch set rather than independently.
> Thank you,
> Razya 
> 
> ----- Forwarded by Razya Ladelsky/Haifa/IBM on 01/01/2015 09:37 AM -----
> 
> From:   Razya Ladelsky/Haifa/IBM@IBMIL
> To:     mst@redhat.com
> Cc: 
> Date:   25/11/2014 02:43 PM
> Subject:        Re: Benchmarking for vhost polling patch
> Sent by:        kvm-owner@vger.kernel.org
> 
> 
> 
> Hi Michael,
> 
> > Hi Razya,
> > On the netperf benchmark, it looks like polling=10 gives a modest but
> > measureable gain.  So from that perspective it might be worth it if it's
> > not too much code, though we'll need to spend more time checking the
> > macro effect - we barely moved the needle on the macro benchmark and
> > that is suspicious.
> 
> I ran memcached with various values for the key & value arguments, and 
> managed to see a bigger impact of polling than when I used the default 
> values,
> Here are the numbers:
> 
> key=250     TPS      net    vhost vm   TPS/cpu  TPS/CPU
> value=2048           rate   util  util          change
> 
> polling=0   101540   103.0  46   100   695.47
> polling=5   136747   123.0  83   100   747.25   0.074440609
> polling=7   140722   125.7  84   100   764.79   0.099663658
> polling=10  141719   126.3  87   100   757.85   0.089688003
> polling=15  142430   127.1  90   100   749.63   0.077863015
> polling=25  146347   128.7  95   100   750.49   0.079107993
> polling=50  150882   131.1  100  100   754.41   0.084733701
> 
> Macro benchmarks are less I/O intensive than the micro benchmark, which is 
> why 
> we can expect less impact for polling as compared to netperf. 
> However, as shown above, we managed to get 10% TPS/CPU improvement with 
> the 
> polling patch.
> 
> > Is there a chance you are actually trading latency for throughput?
> > do you observe any effect on latency?
> 
> No.
> 
> > How about trying some other benchmark, e.g. NFS?
> > 
> 
> Tried, but didn't have enough I/O produced (vhost was at most at 15% util)

OK but was there a regression in this case?


> > 
> > Also, I am wondering:
> > 
> > since vhost thread is polling in kernel anyway, shouldn't
> > we try and poll the host NIC?
> > that would likely reduce at least the latency significantly,
> > won't it?
> > 
> 
> Yes, it could be a great addition at some point, but needs a thorough 
> investigation. In any case, not a part of this patch...
> 
> Thanks,
> Razya
> 
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>