[RFC] Handling CP15 timer without in-kernel irqchip

[RFC] Handling CP15 timer without in-kernel irqchip

[Pavel Fedin]

 Hello!

 Since c2f58514cfb374d5368c9da945f1765cd48eb0da ("arm/arm64: KVM: vgic: Check for
!irqchip_in_kernel() when mapping resources") we can run qemu with kernel_irqchip=off option. The
only remaining problem is how to handle CP15 timer in this case.
 I have applied my old experimental patches to kernel 4.2.2. Using them, i can run 'virt' machine
with software-emulated GICv2 on GICv3 hardware without v2 backwards compatibility. Also, i studied
4.3 code, and it seems to be feasible to add this support to 4.3 and on.
 The only main question now is how would we do it. We can actually use two possible approaches:

 1. If there's no in-kernel irqchip, we revert to older timer behavior (ARCH_TIMER_CTRL_IT_MASK
hack), and forward the timer IRQ to userspace using new KVM_EXIT_IRQ return code. Timer IRQ has to
be treated as edge-triggered in this case. This is the approach which my current implementation
uses.

 2. Another possible approach, based on how device tree binding is handled by Linux. It is possible
to remove virtual timer IRQ from the device tree, in this case the kernel reverts to using physical
timer. When running under hypervisor, accesses to physical CP15 timer are trapped into HYP,
therefore we can forward them to userspace using new exit code, something like KVM_EXIT_REG_ACCESS.
In this case the timer would be also emulated by the userspace, which is slower, but allows better
emulation. Also, this could be used in order to run some other guests which just expect physical
timer to be there.

 Both approaches have their own limitations, but anyway this is much better than nothing. What do
you think, and which approach do you like more?

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia

Re: [RFC] Handling CP15 timer without in-kernel irqchip

[Paolo Bonzini]

On 02/10/2015 09:28, Pavel Fedin wrote:
>  2. Another possible approach, based on how device tree binding is handled by Linux. It is possible
> to remove virtual timer IRQ from the device tree, in this case the kernel reverts to using physical
> timer. When running under hypervisor, accesses to physical CP15 timer are trapped into HYP,
> therefore we can forward them to userspace using new exit code, something like KVM_EXIT_REG_ACCESS.
> In this case the timer would be also emulated by the userspace, which is slower, but allows better
> emulation. Also, this could be used in order to run some other guests which just expect physical
> timer to be there.
> 
>  Both approaches have their own limitations, but anyway this is much better than nothing. What do
> you think, and which approach do you like more?

I like the latter.  But I guess one could even do both?

Paolo

RE: [RFC] Handling CP15 timer without in-kernel irqchip

[Pavel Fedin]

 Hello!

> I like the latter.  But I guess one could even do both?

 You know, you are right, they absolutely do not contradict each other.
 Patch set for #1 is quite not big, if you are interested, and nobody votes strongly against, i can
rebase it to 4.3 and post.

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia

Re: [RFC] Handling CP15 timer without in-kernel irqchip

[Peter Maydell]

On 2 October 2015 at 10:30, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
>
> On 02/10/2015 09:28, Pavel Fedin wrote:
>>  2. Another possible approach, based on how device tree binding is handled by Linux. It is possible
>> to remove virtual timer IRQ from the device tree, in this case the kernel reverts to using physical
>> timer. When running under hypervisor, accesses to physical CP15 timer are trapped into HYP,
>> therefore we can forward them to userspace using new exit code, something like KVM_EXIT_REG_ACCESS.
>> In this case the timer would be also emulated by the userspace, which is slower, but allows better
>> emulation. Also, this could be used in order to run some other guests which just expect physical
>> timer to be there.
>>
>>  Both approaches have their own limitations, but anyway this is much better than nothing. What do
>> you think, and which approach do you like more?
>
> I like the latter.  But I guess one could even do both?

I definitely dislike the latter -- userspace ends up having to
emulate part of the CPU even though that CPU support is really
there in hardware. Also it requires us to edit the device tree,
which means it won't work at all on boards other than 'virt'
where we use the kernel's device tree rather than creating our
own. Better for the kernel to forward the timer
interrupts back out to userspace's irq controller.

-- PMM

Re: [RFC] Handling CP15 timer without in-kernel irqchip

[Paolo Bonzini]

On 02/10/2015 11:58, Peter Maydell wrote:
> On 2 October 2015 at 10:30, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>
>>
>> On 02/10/2015 09:28, Pavel Fedin wrote:
>>>  2. Another possible approach, based on how device tree binding is handled by Linux. It is possible
>>> to remove virtual timer IRQ from the device tree, in this case the kernel reverts to using physical
>>> timer. When running under hypervisor, accesses to physical CP15 timer are trapped into HYP,
>>> therefore we can forward them to userspace using new exit code, something like KVM_EXIT_REG_ACCESS.
>>> In this case the timer would be also emulated by the userspace, which is slower, but allows better
>>> emulation. Also, this could be used in order to run some other guests which just expect physical
>>> timer to be there.
>>>
>>>  Both approaches have their own limitations, but anyway this is much better than nothing. What do
>>> you think, and which approach do you like more?
>>
>> I like the latter.  But I guess one could even do both?
> 
> I definitely dislike the latter -- userspace ends up having to
> emulate part of the CPU even though that CPU support is really
> there in hardware. Also it requires us to edit the device tree,
> which means it won't work at all on boards other than 'virt'
> where we use the kernel's device tree rather than creating our
> own. Better for the kernel to forward the timer
> interrupts back out to userspace's irq controller.

How do boards other than 'virt' work when emulated without KVM?  It must
be possible to emulate the physical timer in QEMU.

Paolo

Re: [RFC] Handling CP15 timer without in-kernel irqchip

[Peter Maydell]

On 2 October 2015 at 11:05, Paolo Bonzini <pbonzini@redhat.com> wrote:
> On 02/10/2015 11:58, Peter Maydell wrote:
>> I definitely dislike the latter -- userspace ends up having to
>> emulate part of the CPU even though that CPU support is really
>> there in hardware. Also it requires us to edit the device tree,
>> which means it won't work at all on boards other than 'virt'
>> where we use the kernel's device tree rather than creating our
>> own. Better for the kernel to forward the timer
>> interrupts back out to userspace's irq controller.
>
> How do boards other than 'virt' work when emulated without KVM?  It must
> be possible to emulate the physical timer in QEMU.

Without KVM is easy -- we emulate the physical timer as just
one of the parts of the emulated CPU. With KVM, we don't emulate
the CPU at all. We don't try to handle a "half TCG half KVM" setup.

thanks
-- PMM

Re: [RFC] Handling CP15 timer without in-kernel irqchip

[Paolo Bonzini]

On 02/10/2015 12:16, Peter Maydell wrote:
> On 2 October 2015 at 11:05, Paolo Bonzini <pbonzini@redhat.com> wrote:
>> On 02/10/2015 11:58, Peter Maydell wrote:
>>> I definitely dislike the latter -- userspace ends up having to
>>> emulate part of the CPU even though that CPU support is really
>>> there in hardware. Also it requires us to edit the device tree,
>>> which means it won't work at all on boards other than 'virt'
>>> where we use the kernel's device tree rather than creating our
>>> own. Better for the kernel to forward the timer
>>> interrupts back out to userspace's irq controller.
>>
>> How do boards other than 'virt' work when emulated without KVM?  It must
>> be possible to emulate the physical timer in QEMU.
> 
> Without KVM is easy -- we emulate the physical timer as just
> one of the parts of the emulated CPU. With KVM, we don't emulate
> the CPU at all. We don't try to handle a "half TCG half KVM" setup.

I mean in the device tree.  Does the boot loader realize it's under a
hypervisor, and provide different device trees to the kernel?

Paolo

RE: [RFC] Handling CP15 timer without in-kernel irqchip

[Pavel Fedin]

 Hello!

> How do boards other than 'virt' work when emulated without KVM?

 Under "emulated without KVM" we normally suppose full software emulation (known in qemu as TCG). In this case CPU code is interpreted, and KVM is not used at all.
 Some boards (like vexpress) have own platform-specific timer, so they can work under KVM without CP15 timer. In order to make it working one should either disable CP15 timer in guest .config or remove the node from device tree.
 CP15 timer never worked in this configuration (KVM without irqchip).

> It must be possible to emulate the physical timer in QEMU.

 Of course it is, and with TCG it works. But we currently cannot use it with KVM because KVM assumes that it emulates all CP15 (or system) registers by itself, and never hands any of them outside.

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia

Re: [RFC] Handling CP15 timer without in-kernel irqchip

[Marc Zyngier]

On Fri, 2 Oct 2015 10:28:42 +0300
Pavel Fedin <p.fedin@samsung.com> wrote:

[...]

>  1. If there's no in-kernel irqchip, we revert to older timer
> behavior (ARCH_TIMER_CTRL_IT_MASK hack), and forward the timer IRQ to
> userspace using new KVM_EXIT_IRQ return code. Timer IRQ has to be
> treated as edge-triggered in this case. This is the approach which my
> current implementation uses.

No way I'm putting this hack back in. We're working hard enough to make
KVM architecture compliant (making the timer LEVEL triggered), and this
hack prevents legitimate guests from running properly. So it is gone,
never to return.

>  2. Another possible approach, based on how device tree binding is
> handled by Linux. It is possible to remove virtual timer IRQ from the
> device tree, in this case the kernel reverts to using physical timer.
> When running under hypervisor, accesses to physical CP15 timer are
> trapped into HYP, therefore we can forward them to userspace using
> new exit code, something like KVM_EXIT_REG_ACCESS. In this case the
> timer would be also emulated by the userspace, which is slower, but
> allows better emulation. Also, this could be used in order to run
> some other guests which just expect physical timer to be there.

A few things:

- How do you find out that your guest is Linux?
- How do you ensure that the guest will not require the virtual timer?
- Who is going to maintain this new ABI that would exist for the sole
  benefit of broken hardware?
- What is wrong with exposing another, memory mapped timer and remove
  the architected timer entirely?

My guess is that this particular ball of hacks will just bitrot as soon
as your product has shipped, because the world will have moved on and
ditched/reworked the broken HW.

So let me put it another way. The only way I look into this is when we
have this particular platform fully supported in mainline.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny.

RE: [RFC] Handling CP15 timer without in-kernel irqchip

[Pavel Fedin]

 Hello!

> I mean in the device tree.  Does the boot loader realize it's under a
> hypervisor, and provide different device trees to the kernel?

 This has nothing to do with the device tree. Device tree is static. The logic sits inside Linux kernel itself:
 a) If we are running in HYP mode (we are host) - use physical timer
 b) If there's no virtual timer IRQ specified - use physical timer
 c) Otherwise - use virtual vimer.

 On systems without virtualization support virtual timer is simply aliased to physical timer, this is architectural requirement. Therefore this logic just works everywhere.

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia

Re: [RFC] Handling CP15 timer without in-kernel irqchip

[Peter Maydell]

On 2 October 2015 at 11:18, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
>
> On 02/10/2015 12:16, Peter Maydell wrote:
>> On 2 October 2015 at 11:05, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>> On 02/10/2015 11:58, Peter Maydell wrote:
>>>> I definitely dislike the latter -- userspace ends up having to
>>>> emulate part of the CPU even though that CPU support is really
>>>> there in hardware. Also it requires us to edit the device tree,
>>>> which means it won't work at all on boards other than 'virt'
>>>> where we use the kernel's device tree rather than creating our
>>>> own. Better for the kernel to forward the timer
>>>> interrupts back out to userspace's irq controller.
>>>
>>> How do boards other than 'virt' work when emulated without KVM?  It must
>>> be possible to emulate the physical timer in QEMU.
>>
>> Without KVM is easy -- we emulate the physical timer as just
>> one of the parts of the emulated CPU. With KVM, we don't emulate
>> the CPU at all. We don't try to handle a "half TCG half KVM" setup.
>
> I mean in the device tree.  Does the boot loader realize it's under a
> hypervisor, and provide different device trees to the kernel?

No. The device tree says "you have both physical and virtual
timers", the guest kernel always prefers the virtual timer, and
this works for both KVM and TCG.

-- PMM

RE: [RFC] Handling CP15 timer without in-kernel irqchip

[Pavel Fedin]

 Hello!

> - How do you find out that your guest is Linux?

 Yes, i agree. No way. And it's not host kernel's job anyway.

> - How do you ensure that the guest will not require the virtual timer?

 Agree again. Except device tree, no way. If the guest has its own hardcoded assumptions, it's
doomed. (*)

> - Who is going to maintain this new ABI that would exist for the sole
>   benefit of broken hardware?

 The ABI is simple, just one more exit code. And it's not only for broken hardware. It is quite
legitimate thing to have GICv3 without v2 backwards compatibility.

> - What is wrong with exposing another, memory mapped timer and remove
>   the architected timer entirely?

 Well... qemu guys don't want to remove CP15, i already proposed it, and it was very simple thing to
do. :) But, again, this is also device-tree-based, the same concern applies as to (*).

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia

RE: [RFC] Handling CP15 timer without in-kernel irqchip

[Pavel Fedin]

 Hello!

> So let me put it another way. The only way I look into this is when we
> have this particular platform fully supported in mainline.

 I am sorry for possible misunderstanding. Please give me one more minute to defend myself...
 So far, we are not putting back timer disable hack. So, i'd like to clarify some things about
variant 2. From kernel's point of view, this is not a hack, but pure feature enhancement. The idea
is to add KVM API which would allow to emulate system registers, unhandled by KVM, in userspace.
Currently KVM just prints error message about unhandled system register access and feeds guest with
"illegal instruction" exception. What i actually propose is to add an API which would allow to
handle these things in userspace. This will even be architecture-agnostic, and it can be useful for
emulating absolutely any future peripherials which could use system register (or coprocessor, on
ARM32) interface.
 The rest of timer-related stuff would be needed to be implemented in userspace, and this would have
absolutely nothing to do with kernel. By the way, this would also allow to run under KVM legitimate
guests which for some reason expect both timers (are there any? RTOS?)
 So, what is your final word? Would you consider this improvement?

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia

Re: [RFC] Handling CP15 timer without in-kernel irqchip

[Christoffer Dall]

On Fri, Oct 02, 2015 at 05:54:53PM +0300, Pavel Fedin wrote:
>  Hello!
> 
> > So let me put it another way. The only way I look into this is when we
> > have this particular platform fully supported in mainline.
> 
>  I am sorry for possible misunderstanding. Please give me one more minute to defend myself...
>  So far, we are not putting back timer disable hack. So, i'd like to clarify some things about
> variant 2. From kernel's point of view, this is not a hack, but pure feature enhancement. The idea
> is to add KVM API which would allow to emulate system registers, unhandled by KVM, in userspace.
> Currently KVM just prints error message about unhandled system register access and feeds guest with
> "illegal instruction" exception. What i actually propose is to add an API which would allow to
> handle these things in userspace. This will even be architecture-agnostic, and it can be useful for
> emulating absolutely any future peripherials which could use system register (or coprocessor, on
> ARM32) interface.
>  The rest of timer-related stuff would be needed to be implemented in userspace, and this would have
> absolutely nothing to do with kernel.

Except that we now have an ABI that we need to maintain, userspace
functionality we need to maintain, we need to ensure that people running
VMs don't accidentally use userspace timers with poor performance
without noticigin, and there's a high chance that this whole thing will
bit-rot; we've seen it happen before.

I appreciate your enthusiasm on this topic, but we are already swamped
with a lot of other important things to do, and this is just not very
high on the radar at the moment.

> By the way, this would also allow to run under KVM legitimate
> guests which for some reason expect both timers (are there any? RTOS?)
>  So, what is your final word? Would you consider this improvement?
> 
I don't want to deny it flat out, and it's hard to say not having seen
the code, the ABI definition, and hear what the userspace maintainers
think about this.

-Christoffer