Re: [patch 2/2] KVM: x86: add option to advance tscdeadline hrtimer expiration

Andy Lutomirski
2014-12-11T21:08:16Z

On Thu, Dec 11, 2014 at 12:58 PM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
> On Thu, Dec 11, 2014 at 12:48:36PM -0800, Andy Lutomirski wrote:
>> On 12/10/2014 07:07 PM, Marcelo Tosatti wrote:
>> > On Thu, Dec 11, 2014 at 12:37:57AM +0100, Paolo Bonzini wrote:
>> >>
>> >>
>> >> On 10/12/2014 21:57, Marcelo Tosatti wrote:
>> >>> For the hrtimer which emulates the tscdeadline timer in the guest,
>> >>> add an option to advance expiration, and busy spin on VM-entry waiting
>> >>> for the actual expiration time to elapse.
>> >>>
>> >>> This allows achieving low latencies in cyclictest (or any scenario
>> >>> which requires strict timing regarding timer expiration).
>> >>>
>> >>> Reduces cyclictest avg latency by 50%.
>> >>>
>> >>> Note: this option requires tuning to find the appropriate value
>> >>> for a particular hardware/guest combination. One method is to measure the
>> >>> average delay between apic_timer_fn and VM-entry.
>> >>> Another method is to start with 1000ns, and increase the value
>> >>> in say 500ns increments until avg cyclictest numbers stop decreasing.
>> >>
>> >> What values are you using in practice for the parameter?
>> >
>> > 7us.
>>
>>
>> It takes 7us to get from TSC deadline expiration to the *start* of
>> vmresume? That seems rather extreme.
>>
>> Is it possible that almost all of that latency is from deadline
>> expiration to C-state exit? If so, can we teach the timer code to wake
>> up early to account for that? We're supposed to know our idle exit
>> latency these days.
>
> 7us includes:
>
> idle thread wakeup
> idle schedout
> ksoftirqd schedin
> ksoftirqd schedout
> qemu schedin
> vm-entry

Is there some secret way to get perf to profile this? Like some way
to tell perf to only record samples after the IRQ fires and before
vmresume? Because 7 us seems waaaaay slower than it should be for
this.

Yes, Rik, I know that we're wasting all kinds of time doing dumb
things with xstate, but that shouldn't be anywhere near 7us on modern
hardware, unless we're actually taking a device-not-available
exception in there. :) There might be a whopping size xstate
operations, but those should be 60ns each, perhaps, if the state is
dirty?

Maybe everything misses cache?

>
> C-states are disabled of course.
>

:)

--Andy