* KVM ARM interrupt virtualization
@ 2011-09-02 3:01 carson bill
2011-09-07 15:35 ` Catalin Marinas
0 siblings, 1 reply; 9+ messages in thread
From: carson bill @ 2011-09-02 3:01 UTC (permalink / raw)
To: linux-arm-kernel
Hi, all
I am trying to put interrupt virtualization support into KVM ARM
implementation by Christoffer.
The approach is like this:
Distributor emulation stays in Qemu, whenever Guest OS touch
Distributor iomem, this action is handled by Qemu as it is.
Each time a device in Qemu call 'gic_update' to inject an
interrupt, the emulated Distributor figure out the 'best_irq'.
With the help of GIC virtulization, this "best_irq" can be routed
to virtual cpu interface which Guest OS has directly access into.
Of course, when Guest OS query which interrupt fired, it need not
trapped out into Qemu anymore.
How to do this:
For now, I didn't take SMP into consideration.
a: create identical mapping from GPA->HPA to allow Guest OS access
its own cpu interface register.
b: tell hypovisor the 'best_irq'
struct kvm_irq_level->level upper bits could be used to represent
this 'best_irq', and goes into hypovisor through KVM_IRQ_LINE IOCTL
once inside kernel, 'kvm_arch_vm_ioctl_irq_line' fetch the 'best_irq'
and by updating list register, route this interrupt into Guest OS
virtual cpu interface.
Problem:
I did the experiment, and found the biggest obstoble on the way
is creating the mapping.
arch/arm/mach-vexpress/include/mach/ct-ca9x4.h
#define CT_CA9X4_MPIC (0x1e000000)
#define A9_MPCORE_SCU (CT_CA9X4_MPIC + 0x0000)
#define A9_MPCORE_GIC_CPU (CT_CA9X4_MPIC + 0x0100)
#define A9_MPCORE_GIT (CT_CA9X4_MPIC + 0x0200)
#define A9_MPCORE_TWD (CT_CA9X4_MPIC + 0x0600)
#define A9_MPCORE_GIC_DIST (CT_CA9X4_MPIC + 0x1000)
If I create an identical mapping at A9_MPCORE_GIC_CPU, all the iomem
from CT_CA9X4_MPIC ~ CT_CA9X4_MPIC + 0x1000 could be access for
Guest OS, which means access to SCU/GIT/TWD will not been trapped
any more.
Any comments and suggestions would be truly welcome.
bill
^ permalink raw reply [flat|nested] 9+ messages in thread* KVM ARM interrupt virtualization
2011-09-02 3:01 KVM ARM interrupt virtualization carson bill
@ 2011-09-07 15:35 ` Catalin Marinas
2011-09-08 1:44 ` carson bill
0 siblings, 1 reply; 9+ messages in thread
From: Catalin Marinas @ 2011-09-07 15:35 UTC (permalink / raw)
To: linux-arm-kernel
On 2 September 2011 04:01, carson bill <bill4carson@gmail.com> wrote:
> I am trying to put interrupt virtualization support into KVM ARM
> implementation by Christoffer.
>
> The approach is like this:
>
> ? Distributor emulation stays in Qemu, whenever Guest OS touch
> ? Distributor iomem, this action is handled by Qemu as it is.
>
> ? Each time a device in Qemu call 'gic_update' to inject an
> ? interrupt, the emulated Distributor figure out the 'best_irq'.
> ? With the help of GIC virtulization, this "best_irq" can be routed
> ? to virtual cpu interface which Guest OS has directly access into.
>
> ? Of course, when Guest OS query which interrupt fired, it need not
> ? trapped out into Qemu anymore.
...
> Problem:
> ? ?I did the experiment, and found the biggest obstoble on the way
> ? ?is creating the mapping.
>
> ? ?arch/arm/mach-vexpress/include/mach/ct-ca9x4.h
> ? ?#define CT_CA9X4_MPIC ? ? ? (0x1e000000)
> ? ?#define A9_MPCORE_SCU ? ? ? (CT_CA9X4_MPIC + 0x0000)
> ? ?#define A9_MPCORE_GIC_CPU ? (CT_CA9X4_MPIC + 0x0100)
> ? ?#define A9_MPCORE_GIT ? ? ? (CT_CA9X4_MPIC + 0x0200)
> ? ?#define A9_MPCORE_TWD ? ? ? (CT_CA9X4_MPIC + 0x0600)
> ? ?#define A9_MPCORE_GIC_DIST ?(CT_CA9X4_MPIC + 0x1000)
>
> ? ?If I create an identical mapping at A9_MPCORE_GIC_CPU, all the iomem
> ? ?from CT_CA9X4_MPIC ~ CT_CA9X4_MPIC + 0x1000 could be access for
> ? ?Guest OS, which means access to SCU/GIT/TWD will not been trapped
> ? ?any more.
Shouldn't you use the CT_CA15X4_MPIC base address? As for TWD, with
A15 we use the generic timers (which also have support for
virtualisation). The SCU doesn't need to be touched on A15, AFAIK.
With the VE/A15 memory map, the GIC Dist is placed at offset 0x1000
and the GIC CPU interface a 0x2000. I think the VGIC interfaces are
placed as follows:
base + 0x4000 - GICV (Virtual Machine CPU interface - same registers
as the physical GIC CPU interface)
base + 0x6000 - GICH (GIC Hypervisor control registers)
So you would have to map GICH inside the host OS to control the VGIC.
The guest OS would most likely want to use the same address as the
physical GIC CPU interface (at offset 0x2000). In this case, the host
OS needs to set up a stage 2 translation from the GIC CPU interface
physical page at offset 0x2000 to offset 0x4000 where the VGIC is
found.
--
Catalin
^ permalink raw reply [flat|nested] 9+ messages in thread
* KVM ARM interrupt virtualization
2011-09-07 15:35 ` Catalin Marinas
@ 2011-09-08 1:44 ` carson bill
2011-09-08 8:54 ` Catalin Marinas
0 siblings, 1 reply; 9+ messages in thread
From: carson bill @ 2011-09-08 1:44 UTC (permalink / raw)
To: linux-arm-kernel
2011/9/7 Catalin Marinas <catalin.marinas@arm.com>:
> On 2 September 2011 04:01, carson bill <bill4carson@gmail.com> wrote:
>> I am trying to put interrupt virtualization support into KVM ARM
>> implementation by Christoffer.
>>
>> The approach is like this:
>>
>> ? Distributor emulation stays in Qemu, whenever Guest OS touch
>> ? Distributor iomem, this action is handled by Qemu as it is.
>>
>> ? Each time a device in Qemu call 'gic_update' to inject an
>> ? interrupt, the emulated Distributor figure out the 'best_irq'.
>> ? With the help of GIC virtulization, this "best_irq" can be routed
>> ? to virtual cpu interface which Guest OS has directly access into.
>>
>> ? Of course, when Guest OS query which interrupt fired, it need not
>> ? trapped out into Qemu anymore.
> ...
>> Problem:
>> ? ?I did the experiment, and found the biggest obstoble on the way
>> ? ?is creating the mapping.
>>
>> ? ?arch/arm/mach-vexpress/include/mach/ct-ca9x4.h
>> ? ?#define CT_CA9X4_MPIC ? ? ? (0x1e000000)
>> ? ?#define A9_MPCORE_SCU ? ? ? (CT_CA9X4_MPIC + 0x0000)
>> ? ?#define A9_MPCORE_GIC_CPU ? (CT_CA9X4_MPIC + 0x0100)
>> ? ?#define A9_MPCORE_GIT ? ? ? (CT_CA9X4_MPIC + 0x0200)
>> ? ?#define A9_MPCORE_TWD ? ? ? (CT_CA9X4_MPIC + 0x0600)
>> ? ?#define A9_MPCORE_GIC_DIST ?(CT_CA9X4_MPIC + 0x1000)
>>
>> ? ?If I create an identical mapping at A9_MPCORE_GIC_CPU, all the iomem
>> ? ?from CT_CA9X4_MPIC ~ CT_CA9X4_MPIC + 0x1000 could be access for
>> ? ?Guest OS, which means access to SCU/GIT/TWD will not been trapped
>> ? ?any more.
>
> Shouldn't you use the CT_CA15X4_MPIC base address? As for TWD, with
> A15 we use the generic timers (which also have support for
> virtualisation). The SCU doesn't need to be touched on A15, AFAIK.
>
> With the VE/A15 memory map, the GIC Dist is placed at offset 0x1000
> and the GIC CPU interface a 0x2000. I think the VGIC interfaces are
> placed as follows:
>
> base + 0x4000 - GICV (Virtual Machine CPU interface - same registers
> as the physical GIC CPU interface)
> base + 0x6000 - GICH (GIC Hypervisor control registers)
>
> So you would have to map GICH inside the host OS to control the VGIC.
> The guest OS would most likely want to use the same address as the
> physical GIC CPU interface (at offset 0x2000). In this case, the host
> OS needs to set up a stage 2 translation from the GIC CPU interface
> physical page at offset 0x2000 to offset 0x4000 where the VGIC is
> found.
>
Thanks Catalin for your reply :)
Do you mean Guest OS have to be built with A15 too ?
Currently Versatile Express Cortex-A9x4 tile is used as Guest OS.
That's why I have to map A9_MPCORE_GIC_CPU from Guest OS into
A15_MPCORE_GIC_CPU at host side.
> --
> Catalin
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* KVM ARM interrupt virtualization
2011-09-08 1:44 ` carson bill
@ 2011-09-08 8:54 ` Catalin Marinas
2011-09-08 10:32 ` bill4carson
0 siblings, 1 reply; 9+ messages in thread
From: Catalin Marinas @ 2011-09-08 8:54 UTC (permalink / raw)
To: linux-arm-kernel
On Thu, Sep 08, 2011 at 02:44:45AM +0100, carson bill wrote:
> 2011/9/7 Catalin Marinas <catalin.marinas@arm.com>:
> > On 2 September 2011 04:01, carson bill <bill4carson@gmail.com> wrote:
> >> I am trying to put interrupt virtualization support into KVM ARM
> >> implementation by Christoffer.
...
> >> Problem:
> >> ? ?I did the experiment, and found the biggest obstoble on the way
> >> ? ?is creating the mapping.
> >>
> >> ? ?arch/arm/mach-vexpress/include/mach/ct-ca9x4.h
> >> ? ?#define CT_CA9X4_MPIC ? ? ? (0x1e000000)
> >> ? ?#define A9_MPCORE_SCU ? ? ? (CT_CA9X4_MPIC + 0x0000)
> >> ? ?#define A9_MPCORE_GIC_CPU ? (CT_CA9X4_MPIC + 0x0100)
> >> ? ?#define A9_MPCORE_GIT ? ? ? (CT_CA9X4_MPIC + 0x0200)
> >> ? ?#define A9_MPCORE_TWD ? ? ? (CT_CA9X4_MPIC + 0x0600)
> >> ? ?#define A9_MPCORE_GIC_DIST ?(CT_CA9X4_MPIC + 0x1000)
> >>
> >> ? ?If I create an identical mapping at A9_MPCORE_GIC_CPU, all the iomem
> >> ? ?from CT_CA9X4_MPIC ~ CT_CA9X4_MPIC + 0x1000 could be access for
> >> ? ?Guest OS, which means access to SCU/GIT/TWD will not been trapped
> >> ? ?any more.
> >
> > Shouldn't you use the CT_CA15X4_MPIC base address? As for TWD, with
> > A15 we use the generic timers (which also have support for
> > virtualisation). The SCU doesn't need to be touched on A15, AFAIK.
...
> Thanks Catalin for your reply :)
>
> Do you mean Guest OS have to be built with A15 too ?
Only if you need to benefit from the virtual GIC interface. The GIC CPU
interface on A9 starts at offset 0x100, which isn't multiple of
PAGE_SIZE, so there is no way to map the GICV page (PAGE_SIZE aligned)
there. As you noted, there are other private peripherals in the same
page like TWD. This was cleaned up with A15 making the GIC interfaces
page-aligned.
> Currently Versatile Express Cortex-A9x4 tile is used as Guest OS.
> That's why I have to map A9_MPCORE_GIC_CPU from Guest OS into
> A15_MPCORE_GIC_CPU at host side.
Can the guest OS not run with the VE/A15 memory map? You don't even need
to have LPAE enabled, just the A15 core tile support for VE
(unfortunately kernel.org is down now and cannot point you to the
relevant patches).
--
Catalin
^ permalink raw reply [flat|nested] 9+ messages in thread
* KVM ARM interrupt virtualization
2011-09-08 8:54 ` Catalin Marinas
@ 2011-09-08 10:32 ` bill4carson
2011-09-08 12:01 ` [Android-virt] " Peter Maydell
0 siblings, 1 reply; 9+ messages in thread
From: bill4carson @ 2011-09-08 10:32 UTC (permalink / raw)
To: linux-arm-kernel
On 2011?09?08? 16:54, Catalin Marinas wrote:
> On Thu, Sep 08, 2011 at 02:44:45AM +0100, carson bill wrote:
>> 2011/9/7 Catalin Marinas<catalin.marinas@arm.com>:
>>> On 2 September 2011 04:01, carson bill<bill4carson@gmail.com> wrote:
>>>> I am trying to put interrupt virtualization support into KVM ARM
>>>> implementation by Christoffer.
> ...
>>>> Problem:
>>>> I did the experiment, and found the biggest obstoble on the way
>>>> is creating the mapping.
>>>>
>>>> arch/arm/mach-vexpress/include/mach/ct-ca9x4.h
>>>> #define CT_CA9X4_MPIC (0x1e000000)
>>>> #define A9_MPCORE_SCU (CT_CA9X4_MPIC + 0x0000)
>>>> #define A9_MPCORE_GIC_CPU (CT_CA9X4_MPIC + 0x0100)
>>>> #define A9_MPCORE_GIT (CT_CA9X4_MPIC + 0x0200)
>>>> #define A9_MPCORE_TWD (CT_CA9X4_MPIC + 0x0600)
>>>> #define A9_MPCORE_GIC_DIST (CT_CA9X4_MPIC + 0x1000)
>>>>
>>>> If I create an identical mapping at A9_MPCORE_GIC_CPU, all the iomem
>>>> from CT_CA9X4_MPIC ~ CT_CA9X4_MPIC + 0x1000 could be access for
>>>> Guest OS, which means access to SCU/GIT/TWD will not been trapped
>>>> any more.
>>> Shouldn't you use the CT_CA15X4_MPIC base address? As for TWD, with
>>> A15 we use the generic timers (which also have support for
>>> virtualisation). The SCU doesn't need to be touched on A15, AFAIK.
> ...
>> Thanks Catalin for your reply :)
>>
>> Do you mean Guest OS have to be built with A15 too ?
> Only if you need to benefit from the virtual GIC interface. The GIC CPU
> interface on A9 starts at offset 0x100, which isn't multiple of
> PAGE_SIZE, so there is no way to map the GICV page (PAGE_SIZE aligned)
> there. As you noted, there are other private peripherals in the same
> page like TWD. This was cleaned up with A15 making the GIC interfaces
> page-aligned.
>
>> Currently Versatile Express Cortex-A9x4 tile is used as Guest OS.
>> That's why I have to map A9_MPCORE_GIC_CPU from Guest OS into
>> A15_MPCORE_GIC_CPU at host side.
> Can the guest OS not run with the VE/A15 memory map? You don't even need
> to have LPAE enabled, just the A15 core tile support for VE
> (unfortunately kernel.org is down now and cannot point you to the
> relevant patches).
>
Hi, Catalin
Thanks for your kindness.
I will wait for that.
There is much more involved if I want to use A15 as Guest OS,
cause Qemu haven't got any A15 support yet.
I'm trying to work on this first, apparently I need more detailed
information
about FastModelPortfolio_6.1/examples/RTSM_VE/Build_Cortex-A15x1.
I didn't find any relevant A15 board doc/spec at arm site, especially
memory map.
Could you please point it out for me?
tons of thanks
bill
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Android-virt] KVM ARM interrupt virtualization
2011-09-08 10:32 ` bill4carson
@ 2011-09-08 12:01 ` Peter Maydell
2011-09-13 5:24 ` bill4carson
0 siblings, 1 reply; 9+ messages in thread
From: Peter Maydell @ 2011-09-08 12:01 UTC (permalink / raw)
To: linux-arm-kernel
On 8 September 2011 11:32, bill4carson <bill4carson@gmail.com> wrote:
> There is much more involved if I want to use A15 as Guest OS,
> cause Qemu haven't got any A15 support yet.
In the interests of not accidentally duplicating work, I should
mention that QEMU A15 work is on my (Linaro) todo list; basically
I'm thinking in terms of adding enough support to boot an A15
Linux guest, but not actually implementing emulation of the
virtualization/LPAE extensions. If you'd like to do some of the
work here we should probably coordinate on that...
The Fast Model Versatile Express platform (including its memory
map) is documented here:
http://infocenter.arm.com/help/topic/com.arm.doc.dui0423j/BEIEAGAA.html
The memory maps (legacy A9 and "Cortex A-Series") for VE
hardware are here:
http://infocenter.arm.com/help/topic/com.arm.doc.dui0447e/CACIHGFE.html
-- PMM
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Android-virt] KVM ARM interrupt virtualization
2011-09-08 12:01 ` [Android-virt] " Peter Maydell
@ 2011-09-13 5:24 ` bill4carson
2011-09-13 7:43 ` Peter Maydell
0 siblings, 1 reply; 9+ messages in thread
From: bill4carson @ 2011-09-13 5:24 UTC (permalink / raw)
To: linux-arm-kernel
On 2011?09?08? 20:01, Peter Maydell wrote:
> On 8 September 2011 11:32, bill4carson<bill4carson@gmail.com> wrote:
>> There is much more involved if I want to use A15 as Guest OS,
>> cause Qemu haven't got any A15 support yet.
> In the interests of not accidentally duplicating work, I should
> mention that QEMU A15 work is on my (Linaro) todo list; basically
> I'm thinking in terms of adding enough support to boot an A15
> Linux guest, but not actually implementing emulation of the
> virtualization/LPAE extensions. If you'd like to do some of the
> work here we should probably coordinate on that...
>
> The Fast Model Versatile Express platform (including its memory
> map) is documented here:
> http://infocenter.arm.com/help/topic/com.arm.doc.dui0423j/BEIEAGAA.html
>
> The memory maps (legacy A9 and "Cortex A-Series") for VE
> hardware are here:
> http://infocenter.arm.com/help/topic/com.arm.doc.dui0447e/CACIHGFE.html
>
> -- PMM
>
Hi Peter
Thanks for your informations, any contribution and help would be welcome.
The initial Qemu A15 support has been added, and could boot with KVM
support.
The generic timer has been removed as I cannot find its base address.
bill
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Android-virt] KVM ARM interrupt virtualization
2011-09-13 5:24 ` bill4carson
@ 2011-09-13 7:43 ` Peter Maydell
2011-09-13 9:50 ` bill4carson
0 siblings, 1 reply; 9+ messages in thread
From: Peter Maydell @ 2011-09-13 7:43 UTC (permalink / raw)
To: linux-arm-kernel
On 13 September 2011 06:24, bill4carson <bill4carson@gmail.com> wrote:
> Thanks for your informations, any contribution and help would be welcome.
> The initial Qemu A15 support has been added, and could boot with KVM
> support.
...added where? If you have qemu patches can you point me at them,
or submit them to qemu-devel, please?
thanks
-- PMM
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Android-virt] KVM ARM interrupt virtualization
2011-09-13 7:43 ` Peter Maydell
@ 2011-09-13 9:50 ` bill4carson
0 siblings, 0 replies; 9+ messages in thread
From: bill4carson @ 2011-09-13 9:50 UTC (permalink / raw)
To: linux-arm-kernel
On 2011?09?13? 15:43, Peter Maydell wrote:
> On 13 September 2011 06:24, bill4carson<bill4carson@gmail.com> wrote:
>> Thanks for your informations, any contribution and help would be welcome.
>> The initial Qemu A15 support has been added, and could boot with KVM
>> support.
> ...added where? If you have qemu patches can you point me at them,
> or submit them to qemu-devel, please?
>
It's on my local private tree.
I'll send the review request to qemu-devel ASAP.
bill
> thanks
> -- PMM
>
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2011-09-13 9:50 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-09-02 3:01 KVM ARM interrupt virtualization carson bill
2011-09-07 15:35 ` Catalin Marinas
2011-09-08 1:44 ` carson bill
2011-09-08 8:54 ` Catalin Marinas
2011-09-08 10:32 ` bill4carson
2011-09-08 12:01 ` [Android-virt] " Peter Maydell
2011-09-13 5:24 ` bill4carson
2011-09-13 7:43 ` Peter Maydell
2011-09-13 9:50 ` bill4carson
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).