Re: [patch 2/2] KVM: x86: add option to advance tscdeadline hrtimer expiration

[Rik van Riel <riel@redhat.com>]

On 12/11/2014 04:07 PM, Andy Lutomirski wrote:
> On Thu, Dec 11, 2014 at 12:58 PM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
>> On Thu, Dec 11, 2014 at 12:48:36PM -0800, Andy Lutomirski wrote:
>>> On 12/10/2014 07:07 PM, Marcelo Tosatti wrote:
>>>> On Thu, Dec 11, 2014 at 12:37:57AM +0100, Paolo Bonzini wrote:
>>>>>
>>>>>
>>>>> On 10/12/2014 21:57, Marcelo Tosatti wrote:
>>>>>> For the hrtimer which emulates the tscdeadline timer in the guest,
>>>>>> add an option to advance expiration, and busy spin on VM-entry waiting
>>>>>> for the actual expiration time to elapse.
>>>>>>
>>>>>> This allows achieving low latencies in cyclictest (or any scenario
>>>>>> which requires strict timing regarding timer expiration).
>>>>>>
>>>>>> Reduces cyclictest avg latency by 50%.
>>>>>>
>>>>>> Note: this option requires tuning to find the appropriate value
>>>>>> for a particular hardware/guest combination. One method is to measure the
>>>>>> average delay between apic_timer_fn and VM-entry.
>>>>>> Another method is to start with 1000ns, and increase the value
>>>>>> in say 500ns increments until avg cyclictest numbers stop decreasing.
>>>>>
>>>>> What values are you using in practice for the parameter?
>>>>
>>>> 7us.
>>>
>>>
>>> It takes 7us to get from TSC deadline expiration to the *start* of
>>> vmresume?  That seems rather extreme.
>>>
>>> Is it possible that almost all of that latency is from deadline
>>> expiration to C-state exit?  If so, can we teach the timer code to wake
>>> up early to account for that?  We're supposed to know our idle exit
>>> latency these days.
>>
>> 7us includes:
>>
>> idle thread wakeup
>> idle schedout
>> ksoftirqd schedin
>> ksoftirqd schedout
>> qemu schedin
>> vm-entry
> 
> Is there some secret way to get perf to profile this?  Like some way
> to tell perf to only record samples after the IRQ fires and before
> vmresume?  Because 7 us seems waaaaay slower than it should be for
> this.
> 
> Yes, Rik, I know that we're wasting all kinds of time doing dumb
> things with xstate, but that shouldn't be anywhere near 7us on modern
> hardware, unless we're actually taking a device-not-available
> exception in there. :)  There might be a whopping size xstate
> operations, but those should be 60ns each, perhaps, if the state is
> dirty?
> 
> Maybe everything misses cache?

I don't expect the xstate stuff to be more than about half
a microsecond, with cache misses and failure to optimize
XSAVEOPT / XRSTOR across vmenter/vmexit.

We get another microsecond or so from not trapping from the
guest to the host every time the guest accesses the FPU for
the first time. Marcelo already has a hack for that in his
tree, and my series merely has something that achieves the
same in an automated (and hopefully upstreamable) way.