Re: QEMU PIC indirection patch for in-kernel APIC work

Anthony Liguori
2007-04-05T15:22:45Z

Avi Kivity wrote:
> Dong, Eddie wrote:
>> Avi Kivity wrote:
>>
>>>> With PIC in Xen, CPU2K gets 6.5% performance gain in old 1000HZ linux
>>>> kernel, KB gets 14% gain. We also did a shared PIC model which share
>>>> PIC state among Qemu & VMM with less LOC in VMM, it can get
>>>> similar performance gain (5.8% in my test).
>>>> BTW, at that time, PIT is in VMM already.
>>>>
>>>>
>>> I expect that the gain in kvm will be smaller. Xen has to schedule
>>> dom0 to process the event channel (possibly on another cpu), dom0 has
>>> to schedule qemu-dm (again, possibly on another cpu), qemu does its
>>> thing, and then Xen has to schedule domU again. With kvm, we are
>>> always on the same cpu, and the only overhead is the system call,
>>> which is a few hundred nanoseconds. I expect with current hardware
>>> that it will be negligible (as a vmexit is measured in microseconds),
>>> but to become measurable as hardware improves.
>>>
>> Yes very possible.
>> We can take a quick mesurement to see how many cycles are spent in a
>> dummy
>> I/O emulation in KVM/Qemu. In Xen, one of my old P4 3.8GHZ platform
>> takes about 50-60K cycles. We can see how many is it in KVM.
>> BTW, today Linux kernel is no longer 1000HZ :-)
>> thx,eddie
>>
>
> There's some (old) data here:
>
> http://virt.kernelnewbies.org/KVM/Performance
>
> showing pio latency of ~5600 cycles. Note that this is on AMD, which
> takes less cycles to switch than the P4, but on the other hand, we
> still do a save/restore of the fpu state on every exit, so we can
> speed it up even more.

It varies quite a lot. On more modern Intel processors, the PIO latency
is pretty good. Also, the number on the wiki is worse than what there
is today as back then there was an extra syscall pair in the PIO path.

I posted a timeline a while ago on kvm-devel. IIRC, going to userspace
added < 1k cycles. However, this will get much more pronounced as we
improve the VMEXIT latency in KVM.

Right now we save a lot of state that we strictly don't have to. Avi
mentioned the FPU state but we also save many MSRs on every exit that we
strictly only have to save/restore when we lose the CPU (such as the
sysenter MSRs). The saving and restoring of these MSRs make up the bulk
of the (avoidable) VMEXIT overhead. Eliminating these will actually
make the difference between handling in-kernel and userspace much more
pronounced.

Xen PIO latency is definitely much worse (at least a factor of 3).
However, what really kills Xen is that occasionally an undesirable
scheduling decision is made and the PIO latency jumps to an enormous
value. This ends making the mean PIO latency considerably worse than
the median latency.

Regards,

Anthony Liguori