Re: vtime accounting

Radim Krčmář
2017-03-14T16:59:03Z

2017-03-14 09:26+0100, Christoffer Dall:
> On Mon, Mar 13, 2017 at 06:28:16PM +0100, Radim Krčmář wrote:
>> 2017-03-08 02:57-0800, Christoffer Dall:
>> > Hi Paolo,
>> >
>> > I'm looking at improving KVM/ARM a bit by calling guest_exit_irqoff
>> > before enabling interrupts when coming back from the guest.
>> >
>> > Unfortunately, this appears to mess up my view of CPU usage using
>> > something like htop on the host, because it appears all time is spent
>> > inside the kernel.
>> >
>> > From my analysis, I think this is because we never handle any interrupts
>> > before enabling interrupts, where the x86 code does its
>> > handle_external_intr, and the result on ARM is that we never increment
>> > jiffies before doing the vtime accounting.
>>
>> (Hm, the counting might be broken on nohz_full then.)
>>
>
> Don't you still have a scheduler tick even with nohz_full and something
> that will eventually update jiffies then?

Probably, I don't understand jiffies accounting too well and didn't see
anything that would bump the jiffies in or before guest_exit_irqoff().

>> > So my current idea is to increment jiffies according to the clocksource
>> > before calling guest_exit_irqoff, but this would require some main
>> > clocksource infrastructure changes.
>>
>> This seems similar to calling the function from the timer interrupt.
>> The timer interrupt would be delivered after that and only wasted time,
>> so it might actually be slower than just delivering it before ...
>
> That's assuming that the timer interrupt hits at every exit. I don't
> think that's the case, but I should measure it.

There cannot be less vm exits and I think there are far more vm exits,
but if there was no interrupt, then jiffies shouldn't raise and we would
get the same result as with plain guest_exit_irqoff().

>> How expensive is the interrupt enable/disable cycle that this
>> optimization saves?
>
> I'll have to go back and measure this bit specifically again, but I
> recall it being a couple of hundred cycles. Not alariming, but
> worthwhile looking into.

Yeah, sounds good.

>> > My question is: how important is the vtime accounting on the host from
>> > your point of view?
>>
>> No idea. I'd keep the same behavior on all architectures, though.
>>
>> The precision of accounting is in jiffies (millions of cycles), so we
>> could maybe move it from the hot path to vcpu_load/put(?) without
>> affecting the count in usual cases ...
>>
>
> So since sending my original e-mail I found out that the vtime
> accounting logic was changed from ktime to jiffies, which is partly why
> we're having problems on arm. See:
>
> ff9a9b4c4334b53b52ee9279f30bd5dd92ea9bdd
> sched, time: Switch VIRT_CPU_ACCOUNTING_GEN to jiffy granularity
>
> Moving to load/put depends on the semantics of this vtime thing. Is
> this counting cycles spent in the VM as opposed to in the host kernel
> and IRQ handling, and is that useful for system profiling or scheduling
> decisions, in which case moving to vcpu_load/put doesn't work...

Right.

> I assume there's a good reason why we call guest_enter() and
> guest_exit() in the hot path on every KVM architecture?

I consider myself biased when it comes to jiffies, so no judgement. :)

>From what I see, the mode switch is used only for statistics.
The original series is

5e84cfde51cf303d368fcb48f22059f37b3872de~1..d172fcd3ae1ca7ac27ec8904242fd61e0e11d332

It didn't introduce the overhead with interrupt window and it didn't
count host kernel irq time as user time, so it was better at that time.