[PATCH 0/2] ARM: OMAP5: Couple of patches for KVM

[PATCH 0/2] ARM: OMAP5: Couple of patches for KVM

[Santosh Shilimkar]

Couple of patches for OMAP5 machines towards KVM support.

Cc: Marc Zyngier <marc.zyngier@arm.com>
Cc: Christoffer Dall <christoffer.dall@linaro.org>
Cc: Tony Lindgren <tony@atomide.com>

Santosh Shilimkar (2):
  ARM: dts: OMAP5: Add maintenance interrupt for virtualisation
  ARM: OMAP5: Add HYP mode entry support for secondary CPUs

 arch/arm/boot/dts/omap5.dtsi       |    1 +
 arch/arm/mach-omap2/omap-headsmp.S |    7 +++++++
 2 files changed, 8 insertions(+)

-- 
1.7.9.5

[PATCH 1/2] ARM: dts: OMAP5: Add maintenance interrupt for virtualisation

[Santosh Shilimkar]

Add a maintenance IRQ using PPI 9 to OMAP5 device tree
needed for virtualisation.

Cc: Marc Zyngier <marc.zyngier@arm.com>
Cc: Christoffer Dall <christoffer.dall@linaro.org>
Cc: Beno?t Cousson <bcousson@baylibre.com>
Cc: Tony Lindgren <tony@atomide.com>
Signed-off-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
---
 arch/arm/boot/dts/omap5.dtsi |    1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/arm/boot/dts/omap5.dtsi b/arch/arm/boot/dts/omap5.dtsi
index fc3fad5..c9f1ae4 100644
--- a/arch/arm/boot/dts/omap5.dtsi
+++ b/arch/arm/boot/dts/omap5.dtsi
@@ -74,6 +74,7 @@
 		      <0x48212000 0x1000>,
 		      <0x48214000 0x2000>,
 		      <0x48216000 0x2000>;
+		interrupts = <1 9 0x304>;
 	};
 
 	/*
-- 
1.7.9.5

[PATCH 1/2] ARM: dts: OMAP5: Add maintenance interrupt for virtualisation

[Santosh Shilimkar]

On Saturday 23 November 2013 07:07 PM, Santosh Shilimkar wrote:
> Add a maintenance IRQ using PPI 9 to OMAP5 device tree
> needed for virtualisation.
> 
> Cc: Marc Zyngier <marc.zyngier@arm.com>
> Cc: Christoffer Dall <christoffer.dall@linaro.org>
> Cc: Beno?t Cousson <bcousson@baylibre.com>
> Cc: Tony Lindgren <tony@atomide.com>
> Signed-off-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
> ---
>  arch/arm/boot/dts/omap5.dtsi |    1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/arch/arm/boot/dts/omap5.dtsi b/arch/arm/boot/dts/omap5.dtsi
> index fc3fad5..c9f1ae4 100644
> --- a/arch/arm/boot/dts/omap5.dtsi
> +++ b/arch/arm/boot/dts/omap5.dtsi
> @@ -74,6 +74,7 @@
>  		      <0x48212000 0x1000>,
>  		      <0x48214000 0x2000>,
>  		      <0x48216000 0x2000>;
> +		interrupts = <1 9 0x304>;
I should have used the GIC flags. Updated version end of the
email.

Regards,
Santosh


>From 91cbd5f65ccd9a0780614fa7ab5505922bdce577 Mon Sep 17 00:00:00 2001
From: Santosh Shilimkar <santosh.shilimkar@ti.com>
Date: Thu, 12 Sep 2013 15:53:13 -0400
Subject: [PATCH 1/2] ARM: dts: OMAP5: Add maintenance interrupt for
 virtualisation

Add a maintenance IRQ using PPI 9 to OMAP5 device tree
needed for virtualisation.

Cc: Marc Zyngier <marc.zyngier@arm.com>
Cc: Christoffer Dall <christoffer.dall@linaro.org>
Cc: Beno?t Cousson <bcousson@baylibre.com>
Cc: Tony Lindgren <tony@atomide.com>
Signed-off-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
---
 arch/arm/boot/dts/omap5.dtsi |    1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/arm/boot/dts/omap5.dtsi b/arch/arm/boot/dts/omap5.dtsi
index fc3fad5..907ab7b 100644
--- a/arch/arm/boot/dts/omap5.dtsi
+++ b/arch/arm/boot/dts/omap5.dtsi
@@ -74,6 +74,7 @@
 		      <0x48212000 0x1000>,
 		      <0x48214000 0x2000>,
 		      <0x48216000 0x2000>;
+		interrupts = <GIC_PPI 9 (GIC_CPU_MASK_SIMPLE(2) | IRQ_TYPE_LEVEL_HIGH)>;
 	};
 
 	/*
-- 
1.7.9.5

[PATCH 2/2] ARM: OMAP5: Add HYP mode entry support for secondary CPUs

[Santosh Shilimkar]

Boot-CPU entry into the HYP mode is managed in boot-loader but
the secondary CPUs directly jumps to kernel during boot. Same
path is also used for CPU hotplug as well during suspend for
secondary CPU.

Hence patch the secondary CPU boot path for hyp mode etry.

Cc: Marc Zyngier <marc.zyngier@arm.com>
Cc: Christoffer Dall <christoffer.dall@linaro.org>
Cc: Tony Lindgren <tony@atomide.com>
Signed-off-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
---
 arch/arm/mach-omap2/omap-headsmp.S |    7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/arch/arm/mach-omap2/omap-headsmp.S b/arch/arm/mach-omap2/omap-headsmp.S
index 75e9295..4844dd8 100644
--- a/arch/arm/mach-omap2/omap-headsmp.S
+++ b/arch/arm/mach-omap2/omap-headsmp.S
@@ -22,6 +22,7 @@
 
 /* Physical address needed since MMU not enabled yet on secondary core */
 #define AUX_CORE_BOOT0_PA			0x48281800
+#define API_HYP_ENTRY				0x102
 
 /*
  * OMAP5 specific entry point for secondary CPU to jump from ROM
@@ -38,6 +39,12 @@ wait:	ldr	r2, =AUX_CORE_BOOT0_PA	@ read from AuxCoreBoot0
 	and	r4, r4, #0x0f
 	cmp	r0, r4
 	bne	wait
+#ifdef CONFIG_KVM_ARM_HOST
+	ldr	r12, =API_HYP_ENTRY
+	adr	r0, hyp_boot
+	smc	#0
+hyp_boot:
+#endif
 	b	secondary_startup
 END(omap5_secondary_startup)
 /*
-- 
1.7.9.5

[PATCH 2/2] ARM: OMAP5: Add HYP mode entry support for secondary CPUs

[Christoffer Dall]

On 23 November 2013 16:07, Santosh Shilimkar <santosh.shilimkar@ti.com> wrote:
> Boot-CPU entry into the HYP mode is managed in boot-loader but
> the secondary CPUs directly jumps to kernel during boot. Same
> path is also used for CPU hotplug as well during suspend for
> secondary CPU.
>
> Hence patch the secondary CPU boot path for hyp mode etry.
>
> Cc: Marc Zyngier <marc.zyngier@arm.com>
> Cc: Christoffer Dall <christoffer.dall@linaro.org>
> Cc: Tony Lindgren <tony@atomide.com>
> Signed-off-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
> ---
>  arch/arm/mach-omap2/omap-headsmp.S |    7 +++++++
>  1 file changed, 7 insertions(+)
>
> diff --git a/arch/arm/mach-omap2/omap-headsmp.S b/arch/arm/mach-omap2/omap-headsmp.S
> index 75e9295..4844dd8 100644
> --- a/arch/arm/mach-omap2/omap-headsmp.S
> +++ b/arch/arm/mach-omap2/omap-headsmp.S
> @@ -22,6 +22,7 @@
>
>  /* Physical address needed since MMU not enabled yet on secondary core */
>  #define AUX_CORE_BOOT0_PA                      0x48281800
> +#define API_HYP_ENTRY                          0x102
>
>  /*
>   * OMAP5 specific entry point for secondary CPU to jump from ROM
> @@ -38,6 +39,12 @@ wait:        ldr     r2, =AUX_CORE_BOOT0_PA  @ read from AuxCoreBoot0
>         and     r4, r4, #0x0f
>         cmp     r0, r4
>         bne     wait
> +#ifdef CONFIG_KVM_ARM_HOST
> +       ldr     r12, =API_HYP_ENTRY
> +       adr     r0, hyp_boot
> +       smc     #0
> +hyp_boot:
> +#endif
>         b       secondary_startup
>  END(omap5_secondary_startup)
>  /*

hmm, this means that currently running this in a guest will fail to
bring-up SMP, right?

Couldn't you create a little wrapper-pen in U-Boot instead, which
replicates the omap boot protocol and takes care of the hyp-mode
startup there instead, keeping this completely out of the kernel?

-Christoffer

[PATCH 2/2] ARM: OMAP5: Add HYP mode entry support for secondary CPUs

[Santosh Shilimkar]

On Monday 25 November 2013 10:09 AM, Christoffer Dall wrote:
> On 23 November 2013 16:07, Santosh Shilimkar <santosh.shilimkar@ti.com> wrote:
>> Boot-CPU entry into the HYP mode is managed in boot-loader but
>> the secondary CPUs directly jumps to kernel during boot. Same
>> path is also used for CPU hotplug as well during suspend for
>> secondary CPU.
>>
>> Hence patch the secondary CPU boot path for hyp mode etry.
>>
>> Cc: Marc Zyngier <marc.zyngier@arm.com>
>> Cc: Christoffer Dall <christoffer.dall@linaro.org>
>> Cc: Tony Lindgren <tony@atomide.com>
>> Signed-off-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
>> ---
>>  arch/arm/mach-omap2/omap-headsmp.S |    7 +++++++
>>  1 file changed, 7 insertions(+)
>>
>> diff --git a/arch/arm/mach-omap2/omap-headsmp.S b/arch/arm/mach-omap2/omap-headsmp.S
>> index 75e9295..4844dd8 100644
>> --- a/arch/arm/mach-omap2/omap-headsmp.S
>> +++ b/arch/arm/mach-omap2/omap-headsmp.S
>> @@ -22,6 +22,7 @@
>>
>>  /* Physical address needed since MMU not enabled yet on secondary core */
>>  #define AUX_CORE_BOOT0_PA                      0x48281800
>> +#define API_HYP_ENTRY                          0x102
>>
>>  /*
>>   * OMAP5 specific entry point for secondary CPU to jump from ROM
>> @@ -38,6 +39,12 @@ wait:        ldr     r2, =AUX_CORE_BOOT0_PA  @ read from AuxCoreBoot0
>>         and     r4, r4, #0x0f
>>         cmp     r0, r4
>>         bne     wait
>> +#ifdef CONFIG_KVM_ARM_HOST
>> +       ldr     r12, =API_HYP_ENTRY
>> +       adr     r0, hyp_boot
>> +       smc     #0
>> +hyp_boot:
>> +#endif
>>         b       secondary_startup
>>  END(omap5_secondary_startup)
>>  /*
> 
> hmm, this means that currently running this in a guest will fail to
> bring-up SMP, right?
> 
Nope. Because the code under 'KVM_ARM_HOST' macro. Guest build 
will not enable CONFIG_KVM_ARM_HOST and things should be fine then.
Right ? 

> Couldn't you create a little wrapper-pen in U-Boot instead, which
> replicates the omap boot protocol and takes care of the hyp-mode
> startup there instead, keeping this completely out of the kernel?
> 
Its not just booting but CPU hotplug also follows the same path
so we need the mechanism in kernel to switch mode.

In general, I think its important to consider the aspect with
CPU PM. CPUs are not going to go through the boot-loaders in
those paths and hence need of HYP entry in the kernel will
be must.

Regards,
Santosh

Regards,
Santosh

[PATCH 2/2] ARM: OMAP5: Add HYP mode entry support for secondary CPUs

[Christoffer Dall]

On 25 November 2013 08:28, Santosh Shilimkar <santosh.shilimkar@ti.com> wrote:
> On Monday 25 November 2013 10:09 AM, Christoffer Dall wrote:
>> On 23 November 2013 16:07, Santosh Shilimkar <santosh.shilimkar@ti.com> wrote:
>>> Boot-CPU entry into the HYP mode is managed in boot-loader but
>>> the secondary CPUs directly jumps to kernel during boot. Same
>>> path is also used for CPU hotplug as well during suspend for
>>> secondary CPU.
>>>
>>> Hence patch the secondary CPU boot path for hyp mode etry.
>>>
>>> Cc: Marc Zyngier <marc.zyngier@arm.com>
>>> Cc: Christoffer Dall <christoffer.dall@linaro.org>
>>> Cc: Tony Lindgren <tony@atomide.com>
>>> Signed-off-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
>>> ---
>>>  arch/arm/mach-omap2/omap-headsmp.S |    7 +++++++
>>>  1 file changed, 7 insertions(+)
>>>
>>> diff --git a/arch/arm/mach-omap2/omap-headsmp.S b/arch/arm/mach-omap2/omap-headsmp.S
>>> index 75e9295..4844dd8 100644
>>> --- a/arch/arm/mach-omap2/omap-headsmp.S
>>> +++ b/arch/arm/mach-omap2/omap-headsmp.S
>>> @@ -22,6 +22,7 @@
>>>
>>>  /* Physical address needed since MMU not enabled yet on secondary core */
>>>  #define AUX_CORE_BOOT0_PA                      0x48281800
>>> +#define API_HYP_ENTRY                          0x102
>>>
>>>  /*
>>>   * OMAP5 specific entry point for secondary CPU to jump from ROM
>>> @@ -38,6 +39,12 @@ wait:        ldr     r2, =AUX_CORE_BOOT0_PA  @ read from AuxCoreBoot0
>>>         and     r4, r4, #0x0f
>>>         cmp     r0, r4
>>>         bne     wait
>>> +#ifdef CONFIG_KVM_ARM_HOST
>>> +       ldr     r12, =API_HYP_ENTRY
>>> +       adr     r0, hyp_boot
>>> +       smc     #0
>>> +hyp_boot:
>>> +#endif
>>>         b       secondary_startup
>>>  END(omap5_secondary_startup)
>>>  /*
>>
>> hmm, this means that currently running this in a guest will fail to
>> bring-up SMP, right?
>>
> Nope. Because the code under 'KVM_ARM_HOST' macro. Guest build
> will not enable CONFIG_KVM_ARM_HOST and things should be fine then.
> Right ?
>

That really goes against the whole single binary on all platforms
thing. With multi-platform support you really shouldn't have to
compile your kernel any differently for running as a guest as when
you're running on a host.  Someone may even emulate an OMAP5 in QEMU
and you'd certainly want your kvm-enabled kernel to run as both guest
and host.  After all, this is not a paravirtualization solution.

>> Couldn't you create a little wrapper-pen in U-Boot instead, which
>> replicates the omap boot protocol and takes care of the hyp-mode
>> startup there instead, keeping this completely out of the kernel?
>>
> Its not just booting but CPU hotplug also follows the same path
> so we need the mechanism in kernel to switch mode.
>
> In general, I think its important to consider the aspect with
> CPU PM. CPUs are not going to go through the boot-loaders in
> those paths and hence need of HYP entry in the kernel will
> be must.
>
I agree, and PSCI is the obvious only correct answer to this.

We have discussed this a bit earlier (I think Will Deacon brought this
up - cc'ed), but I don't think anyone had any bright ideas.

However, we broadly agreed on the fact that for KVM/hyp support, you
need to boot your kernel in that mode, and this is definitely pulling
in the wrong direction.

-Christoffer

HYP Kernel boot requirements [Was ...Re: [PATCH 2/2] ARM: OMAP5: Add HYP mode entry support for secondary CPUs]

[Santosh Shilimkar]

On Monday 25 November 2013 11:33 AM, Christoffer Dall wrote:
> On 25 November 2013 08:28, Santosh Shilimkar <santosh.shilimkar@ti.com> wrote:
>> On Monday 25 November 2013 10:09 AM, Christoffer Dall wrote:
>>> On 23 November 2013 16:07, Santosh Shilimkar <santosh.shilimkar@ti.com> wrote:
>>>> Boot-CPU entry into the HYP mode is managed in boot-loader but
>>>> the secondary CPUs directly jumps to kernel during boot. Same
>>>> path is also used for CPU hotplug as well during suspend for
>>>> secondary CPU.
>>>>
>>>> Hence patch the secondary CPU boot path for hyp mode etry.
>>>>
>>>> Cc: Marc Zyngier <marc.zyngier@arm.com>
>>>> Cc: Christoffer Dall <christoffer.dall@linaro.org>
>>>> Cc: Tony Lindgren <tony@atomide.com>
>>>> Signed-off-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
>>>> ---
>>>>  arch/arm/mach-omap2/omap-headsmp.S |    7 +++++++
>>>>  1 file changed, 7 insertions(+)
>>>>
>>>> diff --git a/arch/arm/mach-omap2/omap-headsmp.S b/arch/arm/mach-omap2/omap-headsmp.S
>>>> index 75e9295..4844dd8 100644
>>>> --- a/arch/arm/mach-omap2/omap-headsmp.S
>>>> +++ b/arch/arm/mach-omap2/omap-headsmp.S
>>>> @@ -22,6 +22,7 @@
>>>>
>>>>  /* Physical address needed since MMU not enabled yet on secondary core */
>>>>  #define AUX_CORE_BOOT0_PA                      0x48281800
>>>> +#define API_HYP_ENTRY                          0x102
>>>>
>>>>  /*
>>>>   * OMAP5 specific entry point for secondary CPU to jump from ROM
>>>> @@ -38,6 +39,12 @@ wait:        ldr     r2, =AUX_CORE_BOOT0_PA  @ read from AuxCoreBoot0
>>>>         and     r4, r4, #0x0f
>>>>         cmp     r0, r4
>>>>         bne     wait
>>>> +#ifdef CONFIG_KVM_ARM_HOST
>>>> +       ldr     r12, =API_HYP_ENTRY
>>>> +       adr     r0, hyp_boot
>>>> +       smc     #0
>>>> +hyp_boot:
>>>> +#endif
>>>>         b       secondary_startup
>>>>  END(omap5_secondary_startup)
>>>>  /*
>>>
>>> hmm, this means that currently running this in a guest will fail to
>>> bring-up SMP, right?
>>>
>> Nope. Because the code under 'KVM_ARM_HOST' macro. Guest build
>> will not enable CONFIG_KVM_ARM_HOST and things should be fine then.
>> Right ?
>>
> 
> That really goes against the whole single binary on all platforms
> thing. With multi-platform support you really shouldn't have to
> compile your kernel any differently for running as a guest as when
> you're running on a host.  Someone may even emulate an OMAP5 in QEMU
> and you'd certainly want your kvm-enabled kernel to run as both guest
> and host.  After all, this is not a paravirtualization solution.
> 
Fair enough.

>>> Couldn't you create a little wrapper-pen in U-Boot instead, which
>>> replicates the omap boot protocol and takes care of the hyp-mode
>>> startup there instead, keeping this completely out of the kernel?
>>>
>> Its not just booting but CPU hotplug also follows the same path
>> so we need the mechanism in kernel to switch mode.
>>
>> In general, I think its important to consider the aspect with
>> CPU PM. CPUs are not going to go through the boot-loaders in
>> those paths and hence need of HYP entry in the kernel will
>> be must.
>>
> I agree, and PSCI is the obvious only correct answer to this.
> 
> We have discussed this a bit earlier (I think Will Deacon brought this
> up - cc'ed), but I don't think anyone had any bright ideas.
> 
> However, we broadly agreed on the fact that for KVM/hyp support, you
> need to boot your kernel in that mode, and this is definitely pulling
> in the wrong direction.
> 
What I am saying is the platforms like OMAP5 already support PM in
mainline kernel and we can't break that for KVM. Boot-loaders
would be thrashed after boot so you need something which runs
in Kernel or along with Kernel to have equivalent of hyp
switching.

Am not challenging the agreed direction but we need to solve the
PM problem as well before making "all CPU runs boot-loader for
HYP kernels" as a must have. At least its is a change in boot
strategy from existing kernels.

CC'ing few more folks and changing the subject line

Regards,
Santosh

 

HYP Kernel boot requirements [Was ...Re: [PATCH 2/2] ARM: OMAP5: Add HYP mode entry support for secondary CPUs]

[Catalin Marinas]

On Mon, Nov 25, 2013 at 04:59:16PM +0000, Santosh Shilimkar wrote:
> On Monday 25 November 2013 11:33 AM, Christoffer Dall wrote:
> > On 25 November 2013 08:28, Santosh Shilimkar <santosh.shilimkar@ti.com> wrote:
> >> On Monday 25 November 2013 10:09 AM, Christoffer Dall wrote:
> >>> On 23 November 2013 16:07, Santosh Shilimkar <santosh.shilimkar@ti.com> wrote:
> >>>> Boot-CPU entry into the HYP mode is managed in boot-loader but
> >>>> the secondary CPUs directly jumps to kernel during boot. Same
> >>>> path is also used for CPU hotplug as well during suspend for
> >>>> secondary CPU.
> >>>>
> >>>> Hence patch the secondary CPU boot path for hyp mode etry.
> >>>>
> >>>> Cc: Marc Zyngier <marc.zyngier@arm.com>
> >>>> Cc: Christoffer Dall <christoffer.dall@linaro.org>
> >>>> Cc: Tony Lindgren <tony@atomide.com>
> >>>> Signed-off-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
> >>>> ---
> >>>>  arch/arm/mach-omap2/omap-headsmp.S |    7 +++++++
> >>>>  1 file changed, 7 insertions(+)
> >>>>
> >>>> diff --git a/arch/arm/mach-omap2/omap-headsmp.S b/arch/arm/mach-omap2/omap-headsmp.S
> >>>> index 75e9295..4844dd8 100644
> >>>> --- a/arch/arm/mach-omap2/omap-headsmp.S
> >>>> +++ b/arch/arm/mach-omap2/omap-headsmp.S
> >>>> @@ -22,6 +22,7 @@
> >>>>
> >>>>  /* Physical address needed since MMU not enabled yet on secondary core */
> >>>>  #define AUX_CORE_BOOT0_PA                      0x48281800
> >>>> +#define API_HYP_ENTRY                          0x102
> >>>>
> >>>>  /*
> >>>>   * OMAP5 specific entry point for secondary CPU to jump from ROM
> >>>> @@ -38,6 +39,12 @@ wait:        ldr     r2, =AUX_CORE_BOOT0_PA  @ read from AuxCoreBoot0
> >>>>         and     r4, r4, #0x0f
> >>>>         cmp     r0, r4
> >>>>         bne     wait
> >>>> +#ifdef CONFIG_KVM_ARM_HOST
> >>>> +       ldr     r12, =API_HYP_ENTRY
> >>>> +       adr     r0, hyp_boot
> >>>> +       smc     #0
> >>>> +hyp_boot:
> >>>> +#endif
> >>>>         b       secondary_startup
> >>>>  END(omap5_secondary_startup)
> >>>>  /*
> >>>
> >>> hmm, this means that currently running this in a guest will fail to
> >>> bring-up SMP, right?
> >>>
> >> Nope. Because the code under 'KVM_ARM_HOST' macro. Guest build
> >> will not enable CONFIG_KVM_ARM_HOST and things should be fine then.
> >> Right ?
> >>
> > 
> > That really goes against the whole single binary on all platforms
> > thing. With multi-platform support you really shouldn't have to
> > compile your kernel any differently for running as a guest as when
> > you're running on a host.  Someone may even emulate an OMAP5 in QEMU
> > and you'd certainly want your kvm-enabled kernel to run as both guest
> > and host.  After all, this is not a paravirtualization solution.
> 
> Fair enough.
> 
> >>> Couldn't you create a little wrapper-pen in U-Boot instead, which
> >>> replicates the omap boot protocol and takes care of the hyp-mode
> >>> startup there instead, keeping this completely out of the kernel?
> >>>
> >> Its not just booting but CPU hotplug also follows the same path
> >> so we need the mechanism in kernel to switch mode.
> >>
> >> In general, I think its important to consider the aspect with
> >> CPU PM. CPUs are not going to go through the boot-loaders in
> >> those paths and hence need of HYP entry in the kernel will
> >> be must.
> >>
> > I agree, and PSCI is the obvious only correct answer to this.
> > 
> > We have discussed this a bit earlier (I think Will Deacon brought this
> > up - cc'ed), but I don't think anyone had any bright ideas.
> > 
> > However, we broadly agreed on the fact that for KVM/hyp support, you
> > need to boot your kernel in that mode, and this is definitely pulling
> > in the wrong direction.
> 
> What I am saying is the platforms like OMAP5 already support PM in
> mainline kernel and we can't break that for KVM. Boot-loaders
> would be thrashed after boot so you need something which runs
> in Kernel or along with Kernel to have equivalent of hyp
> switching.
> 
> Am not challenging the agreed direction but we need to solve the
> PM problem as well before making "all CPU runs boot-loader for
> HYP kernels" as a must have. At least its is a change in boot
> strategy from existing kernels.

Of course I recommend PSCI which covers both hotplug and suspend ;), but
I guess it's not the case for OMAP5. Since OMAP has its own secondary
booting protocol and CPU hotplug re-entry, can you not just use
different entry point when the primary CPU was initially started in Hyp
mode (e.g. omap5_hyp_secondary_startup)?

-- 
Catalin

HYP Kernel boot requirements

[Santosh Shilimkar]

On Monday 25 November 2013 12:28 PM, Catalin Marinas wrote:
> On Mon, Nov 25, 2013 at 04:59:16PM +0000, Santosh Shilimkar wrote:
>> On Monday 25 November 2013 11:33 AM, Christoffer Dall wrote:
>>> On 25 November 2013 08:28, Santosh Shilimkar <santosh.shilimkar@ti.com> wrote:
>>>> On Monday 25 November 2013 10:09 AM, Christoffer Dall wrote:
>>>>> On 23 November 2013 16:07, Santosh Shilimkar <santosh.shilimkar@ti.com> wrote:
>>>>>> Boot-CPU entry into the HYP mode is managed in boot-loader but
>>>>>> the secondary CPUs directly jumps to kernel during boot. Same
>>>>>> path is also used for CPU hotplug as well during suspend for
>>>>>> secondary CPU.
>>>>>>
>>>>>> Hence patch the secondary CPU boot path for hyp mode etry.
>>>>>>
>>>>>> Cc: Marc Zyngier <marc.zyngier@arm.com>
>>>>>> Cc: Christoffer Dall <christoffer.dall@linaro.org>
>>>>>> Cc: Tony Lindgren <tony@atomide.com>
>>>>>> Signed-off-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
>>>>>> ---
>>>>>>  arch/arm/mach-omap2/omap-headsmp.S |    7 +++++++
>>>>>>  1 file changed, 7 insertions(+)
>>>>>>
>>>>>> diff --git a/arch/arm/mach-omap2/omap-headsmp.S b/arch/arm/mach-omap2/omap-headsmp.S
>>>>>> index 75e9295..4844dd8 100644
>>>>>> --- a/arch/arm/mach-omap2/omap-headsmp.S
>>>>>> +++ b/arch/arm/mach-omap2/omap-headsmp.S
>>>>>> @@ -22,6 +22,7 @@
>>>>>>
>>>>>>  /* Physical address needed since MMU not enabled yet on secondary core */
>>>>>>  #define AUX_CORE_BOOT0_PA                      0x48281800
>>>>>> +#define API_HYP_ENTRY                          0x102
>>>>>>
>>>>>>  /*
>>>>>>   * OMAP5 specific entry point for secondary CPU to jump from ROM
>>>>>> @@ -38,6 +39,12 @@ wait:        ldr     r2, =AUX_CORE_BOOT0_PA  @ read from AuxCoreBoot0
>>>>>>         and     r4, r4, #0x0f
>>>>>>         cmp     r0, r4
>>>>>>         bne     wait
>>>>>> +#ifdef CONFIG_KVM_ARM_HOST
>>>>>> +       ldr     r12, =API_HYP_ENTRY
>>>>>> +       adr     r0, hyp_boot
>>>>>> +       smc     #0
>>>>>> +hyp_boot:
>>>>>> +#endif
>>>>>>         b       secondary_startup
>>>>>>  END(omap5_secondary_startup)
>>>>>>  /*
>>>>>
>>>>> hmm, this means that currently running this in a guest will fail to
>>>>> bring-up SMP, right?
>>>>>
>>>> Nope. Because the code under 'KVM_ARM_HOST' macro. Guest build
>>>> will not enable CONFIG_KVM_ARM_HOST and things should be fine then.
>>>> Right ?
>>>>
>>>
>>> That really goes against the whole single binary on all platforms
>>> thing. With multi-platform support you really shouldn't have to
>>> compile your kernel any differently for running as a guest as when
>>> you're running on a host.  Someone may even emulate an OMAP5 in QEMU
>>> and you'd certainly want your kvm-enabled kernel to run as both guest
>>> and host.  After all, this is not a paravirtualization solution.
>>
>> Fair enough.
>>
>>>>> Couldn't you create a little wrapper-pen in U-Boot instead, which
>>>>> replicates the omap boot protocol and takes care of the hyp-mode
>>>>> startup there instead, keeping this completely out of the kernel?
>>>>>
>>>> Its not just booting but CPU hotplug also follows the same path
>>>> so we need the mechanism in kernel to switch mode.
>>>>
>>>> In general, I think its important to consider the aspect with
>>>> CPU PM. CPUs are not going to go through the boot-loaders in
>>>> those paths and hence need of HYP entry in the kernel will
>>>> be must.
>>>>
>>> I agree, and PSCI is the obvious only correct answer to this.
>>>
>>> We have discussed this a bit earlier (I think Will Deacon brought this
>>> up - cc'ed), but I don't think anyone had any bright ideas.
>>>
>>> However, we broadly agreed on the fact that for KVM/hyp support, you
>>> need to boot your kernel in that mode, and this is definitely pulling
>>> in the wrong direction.
>>
>> What I am saying is the platforms like OMAP5 already support PM in
>> mainline kernel and we can't break that for KVM. Boot-loaders
>> would be thrashed after boot so you need something which runs
>> in Kernel or along with Kernel to have equivalent of hyp
>> switching.
>>
>> Am not challenging the agreed direction but we need to solve the
>> PM problem as well before making "all CPU runs boot-loader for
>> HYP kernels" as a must have. At least its is a change in boot
>> strategy from existing kernels.
> 
> Of course I recommend PSCI which covers both hotplug and suspend ;), but
> I guess it's not the case for OMAP5. Since OMAP has its own secondary
> booting protocol and CPU hotplug re-entry, can you not just use
> different entry point when the primary CPU was initially started in Hyp
> mode (e.g. omap5_hyp_secondary_startup)?
> 
How will that solve the guest secondary boot failure case when using
the same kernel binary for guest-boot ? Even for primary CPU which
will be suspended it needs to resume already in HYP mode and its not
going to go through boot-loader. So the low power code needs to have
HYP switch code so that CPU resumes in HYP mode.

I will look at PSCI more closely and see what can be done here.

Regards,
Santosh

HYP Kernel boot requirements

[Catalin Marinas]

On Mon, Nov 25, 2013 at 07:44:08PM +0000, Santosh Shilimkar wrote:
> On Monday 25 November 2013 12:28 PM, Catalin Marinas wrote:
> > On Mon, Nov 25, 2013 at 04:59:16PM +0000, Santosh Shilimkar wrote:
> >> What I am saying is the platforms like OMAP5 already support PM in
> >> mainline kernel and we can't break that for KVM. Boot-loaders
> >> would be thrashed after boot so you need something which runs
> >> in Kernel or along with Kernel to have equivalent of hyp
> >> switching.
> >>
> >> Am not challenging the agreed direction but we need to solve the
> >> PM problem as well before making "all CPU runs boot-loader for
> >> HYP kernels" as a must have. At least its is a change in boot
> >> strategy from existing kernels.
> > 
> > Of course I recommend PSCI which covers both hotplug and suspend ;), but
> > I guess it's not the case for OMAP5. Since OMAP has its own secondary
> > booting protocol and CPU hotplug re-entry, can you not just use
> > different entry point when the primary CPU was initially started in Hyp
> > mode (e.g. omap5_hyp_secondary_startup)?
> > 
> How will that solve the guest secondary boot failure case when using
> the same kernel binary for guest-boot ? Even for primary CPU which
> will be suspended it needs to resume already in HYP mode and its not
> going to go through boot-loader. So the low power code needs to have
> HYP switch code so that CPU resumes in HYP mode.

Is it late to rewrite the OMAP5 firmware?

What I meant is that you have the same kernel binary but with two sets
of entry points, or maybe a single set of entry points and some global
variable that you set during cold boot if the primary CPU is entered in
Hyp mode. So you change the boot loader to switch to Hyp before it
starts the kernel and leave the additional switching to be done by the
secondary entry points (or PM code) based on the variable you set during
cold boot. But I would recommend rewriting the firmware.

> I will look at PSCI more closely and see what can be done here.

PSCI has a different secondary start-up and hotplug/suspend protocol and
CPUs wake up in Hyp if available. But this requires changing your
firmware, PSCI is not a kernel-only thing. Do you need to support both
older and newer kernels with the OMAP5 firmware (and possibly same DT)?

-- 
Catalin

HYP Kernel boot requirements

[Santosh Shilimkar]

On Tuesday 26 November 2013 09:13 AM, Catalin Marinas wrote:
> On Mon, Nov 25, 2013 at 07:44:08PM +0000, Santosh Shilimkar wrote:
>> On Monday 25 November 2013 12:28 PM, Catalin Marinas wrote:
>>> On Mon, Nov 25, 2013 at 04:59:16PM +0000, Santosh Shilimkar wrote:
>>>> What I am saying is the platforms like OMAP5 already support PM in
>>>> mainline kernel and we can't break that for KVM. Boot-loaders
>>>> would be thrashed after boot so you need something which runs
>>>> in Kernel or along with Kernel to have equivalent of hyp
>>>> switching.
>>>>
>>>> Am not challenging the agreed direction but we need to solve the
>>>> PM problem as well before making "all CPU runs boot-loader for
>>>> HYP kernels" as a must have. At least its is a change in boot
>>>> strategy from existing kernels.
>>>
>>> Of course I recommend PSCI which covers both hotplug and suspend ;), but
>>> I guess it's not the case for OMAP5. Since OMAP has its own secondary
>>> booting protocol and CPU hotplug re-entry, can you not just use
>>> different entry point when the primary CPU was initially started in Hyp
>>> mode (e.g. omap5_hyp_secondary_startup)?
>>>
>> How will that solve the guest secondary boot failure case when using
>> the same kernel binary for guest-boot ? Even for primary CPU which
>> will be suspended it needs to resume already in HYP mode and its not
>> going to go through boot-loader. So the low power code needs to have
>> HYP switch code so that CPU resumes in HYP mode.
> 
> Is it late to rewrite the OMAP5 firmware?
> 
Well its ROM'ed unfortunately so no choice. OMAP5 ROM did implement
a secure API which lets you enter into HYP mode and thats the only
thing can be used.

> What I meant is that you have the same kernel binary but with two sets
> of entry points, or maybe a single set of entry points and some global
> variable that you set during cold boot if the primary CPU is entered in
> Hyp mode. So you change the boot loader to switch to Hyp before it
> starts the kernel and leave the additional switching to be done by the
> secondary entry points (or PM code) based on the variable you set during
> cold boot. But I would recommend rewriting the firmware.
> 
>> I will look at PSCI more closely and see what can be done here.
> 
> PSCI has a different secondary start-up and hotplug/suspend protocol and
> CPUs wake up in Hyp if available. But this requires changing your
> firmware, PSCI is not a kernel-only thing. Do you need to support both
> older and newer kernels with the OMAP5 firmware (and possibly same DT)?
>
Hmmm.. So thats rules out PSCI for OMAP5 then.

Keystone actually has nice way to deal with the problem since the
boot-monitor code can be patched up and hence I had no trouble
implementing the "CPU starts in HYP mode" requirement on it.

Regards,
Santosh


 

HYP Kernel boot requirements

[Dave Martin]

On Tue, Nov 26, 2013 at 09:47:13AM -0500, Santosh Shilimkar wrote:
> On Tuesday 26 November 2013 09:13 AM, Catalin Marinas wrote:
> > On Mon, Nov 25, 2013 at 07:44:08PM +0000, Santosh Shilimkar wrote:
> >> On Monday 25 November 2013 12:28 PM, Catalin Marinas wrote:
> >>> On Mon, Nov 25, 2013 at 04:59:16PM +0000, Santosh Shilimkar wrote:
> >>>> What I am saying is the platforms like OMAP5 already support PM in
> >>>> mainline kernel and we can't break that for KVM. Boot-loaders
> >>>> would be thrashed after boot so you need something which runs
> >>>> in Kernel or along with Kernel to have equivalent of hyp
> >>>> switching.
> >>>>
> >>>> Am not challenging the agreed direction but we need to solve the
> >>>> PM problem as well before making "all CPU runs boot-loader for
> >>>> HYP kernels" as a must have. At least its is a change in boot
> >>>> strategy from existing kernels.
> >>>
> >>> Of course I recommend PSCI which covers both hotplug and suspend ;), but
> >>> I guess it's not the case for OMAP5. Since OMAP has its own secondary
> >>> booting protocol and CPU hotplug re-entry, can you not just use
> >>> different entry point when the primary CPU was initially started in Hyp
> >>> mode (e.g. omap5_hyp_secondary_startup)?
> >>>
> >> How will that solve the guest secondary boot failure case when using
> >> the same kernel binary for guest-boot ? Even for primary CPU which
> >> will be suspended it needs to resume already in HYP mode and its not
> >> going to go through boot-loader. So the low power code needs to have
> >> HYP switch code so that CPU resumes in HYP mode.
> > 
> > Is it late to rewrite the OMAP5 firmware?
> > 
> Well its ROM'ed unfortunately so no choice. OMAP5 ROM did implement
> a secure API which lets you enter into HYP mode and thats the only
> thing can be used.

If the ROM is capable of loading some additional signed Secure World
firmware after the ROM itself has booted, PSCI could be implemented
in the second, resident firmware payload.

Some SoCs ship with boot ROMs that can do that -- is this not the case
for OMAP5?

Cheers
---Dave

HYP Kernel boot requirements

[Santosh Shilimkar]

On Tuesday 26 November 2013 12:37 PM, Dave Martin wrote:
> On Tue, Nov 26, 2013 at 09:47:13AM -0500, Santosh Shilimkar wrote:
>> On Tuesday 26 November 2013 09:13 AM, Catalin Marinas wrote:
>>> On Mon, Nov 25, 2013 at 07:44:08PM +0000, Santosh Shilimkar wrote:
>>>> On Monday 25 November 2013 12:28 PM, Catalin Marinas wrote:
>>>>> On Mon, Nov 25, 2013 at 04:59:16PM +0000, Santosh Shilimkar wrote:
>>>>>> What I am saying is the platforms like OMAP5 already support PM in
>>>>>> mainline kernel and we can't break that for KVM. Boot-loaders
>>>>>> would be thrashed after boot so you need something which runs
>>>>>> in Kernel or along with Kernel to have equivalent of hyp
>>>>>> switching.
>>>>>>
>>>>>> Am not challenging the agreed direction but we need to solve the
>>>>>> PM problem as well before making "all CPU runs boot-loader for
>>>>>> HYP kernels" as a must have. At least its is a change in boot
>>>>>> strategy from existing kernels.
>>>>>
>>>>> Of course I recommend PSCI which covers both hotplug and suspend ;), but
>>>>> I guess it's not the case for OMAP5. Since OMAP has its own secondary
>>>>> booting protocol and CPU hotplug re-entry, can you not just use
>>>>> different entry point when the primary CPU was initially started in Hyp
>>>>> mode (e.g. omap5_hyp_secondary_startup)?
>>>>>
>>>> How will that solve the guest secondary boot failure case when using
>>>> the same kernel binary for guest-boot ? Even for primary CPU which
>>>> will be suspended it needs to resume already in HYP mode and its not
>>>> going to go through boot-loader. So the low power code needs to have
>>>> HYP switch code so that CPU resumes in HYP mode.
>>>
>>> Is it late to rewrite the OMAP5 firmware?
>>>
>> Well its ROM'ed unfortunately so no choice. OMAP5 ROM did implement
>> a secure API which lets you enter into HYP mode and thats the only
>> thing can be used.
> 
> If the ROM is capable of loading some additional signed Secure World
> firmware after the ROM itself has booted, PSCI could be implemented
> in the second, resident firmware payload.
> 
> Some SoCs ship with boot ROMs that can do that -- is this not the case
> for OMAP5?
> 
On OMAP, the secure devices there is a way to do this but not for 
general purpose devices. General purpose devices once you
exit the ROM code, we exit out of security and only way to
re-enter secure word is via ROM implemented monitor/secure
APIs.

regards,
Santosh

HYP Kernel boot requirements

[Lorenzo Pieralisi]

On Tue, Nov 26, 2013 at 09:49:10PM +0000, Santosh Shilimkar wrote:
> On Tuesday 26 November 2013 12:37 PM, Dave Martin wrote:
> > On Tue, Nov 26, 2013 at 09:47:13AM -0500, Santosh Shilimkar wrote:
> >> On Tuesday 26 November 2013 09:13 AM, Catalin Marinas wrote:
> >>> On Mon, Nov 25, 2013 at 07:44:08PM +0000, Santosh Shilimkar wrote:
> >>>> On Monday 25 November 2013 12:28 PM, Catalin Marinas wrote:
> >>>>> On Mon, Nov 25, 2013 at 04:59:16PM +0000, Santosh Shilimkar wrote:
> >>>>>> What I am saying is the platforms like OMAP5 already support PM in
> >>>>>> mainline kernel and we can't break that for KVM. Boot-loaders
> >>>>>> would be thrashed after boot so you need something which runs
> >>>>>> in Kernel or along with Kernel to have equivalent of hyp
> >>>>>> switching.
> >>>>>>
> >>>>>> Am not challenging the agreed direction but we need to solve the
> >>>>>> PM problem as well before making "all CPU runs boot-loader for
> >>>>>> HYP kernels" as a must have. At least its is a change in boot
> >>>>>> strategy from existing kernels.
> >>>>>
> >>>>> Of course I recommend PSCI which covers both hotplug and suspend ;), but
> >>>>> I guess it's not the case for OMAP5. Since OMAP has its own secondary
> >>>>> booting protocol and CPU hotplug re-entry, can you not just use
> >>>>> different entry point when the primary CPU was initially started in Hyp
> >>>>> mode (e.g. omap5_hyp_secondary_startup)?
> >>>>>
> >>>> How will that solve the guest secondary boot failure case when using
> >>>> the same kernel binary for guest-boot ? Even for primary CPU which
> >>>> will be suspended it needs to resume already in HYP mode and its not
> >>>> going to go through boot-loader. So the low power code needs to have
> >>>> HYP switch code so that CPU resumes in HYP mode.
> >>>
> >>> Is it late to rewrite the OMAP5 firmware?
> >>>
> >> Well its ROM'ed unfortunately so no choice. OMAP5 ROM did implement
> >> a secure API which lets you enter into HYP mode and thats the only
> >> thing can be used.
> > 
> > If the ROM is capable of loading some additional signed Secure World
> > firmware after the ROM itself has booted, PSCI could be implemented
> > in the second, resident firmware payload.
> > 
> > Some SoCs ship with boot ROMs that can do that -- is this not the case
> > for OMAP5?
> > 
> On OMAP, the secure devices there is a way to do this but not for 
> general purpose devices. General purpose devices once you
> exit the ROM code, we exit out of security and only way to
> re-enter secure word is via ROM implemented monitor/secure
> APIs.

As last resort, you can still write a bunch of instructions acting as
warm-bootloader, reserve a page and add them there. Code required is just
a shim layer and should not require more than few bytes, after all
it has just to check some variables and jump accordingly (+ calling
smc if required).

Not pretty, agreed, but still better than patching the kernel IMHO
(basically you will end up patching the bootloader - and possibly the
dtb - instead, to install the shim mentioned above).

Lorenzo

[PATCH 2/2] ARM: OMAP5: Add HYP mode entry support for secondary CPUs

[Marc Zyngier]

On 25/11/13 16:28, Santosh Shilimkar wrote:
> On Monday 25 November 2013 10:09 AM, Christoffer Dall wrote:
>> On 23 November 2013 16:07, Santosh Shilimkar <santosh.shilimkar@ti.com> wrote:
>>> Boot-CPU entry into the HYP mode is managed in boot-loader but
>>> the secondary CPUs directly jumps to kernel during boot. Same
>>> path is also used for CPU hotplug as well during suspend for
>>> secondary CPU.
>>>
>>> Hence patch the secondary CPU boot path for hyp mode etry.
>>>
>>> Cc: Marc Zyngier <marc.zyngier@arm.com>
>>> Cc: Christoffer Dall <christoffer.dall@linaro.org>
>>> Cc: Tony Lindgren <tony@atomide.com>
>>> Signed-off-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
>>> ---
>>>  arch/arm/mach-omap2/omap-headsmp.S |    7 +++++++
>>>  1 file changed, 7 insertions(+)
>>>
>>> diff --git a/arch/arm/mach-omap2/omap-headsmp.S b/arch/arm/mach-omap2/omap-headsmp.S
>>> index 75e9295..4844dd8 100644
>>> --- a/arch/arm/mach-omap2/omap-headsmp.S
>>> +++ b/arch/arm/mach-omap2/omap-headsmp.S
>>> @@ -22,6 +22,7 @@
>>>
>>>  /* Physical address needed since MMU not enabled yet on secondary core */
>>>  #define AUX_CORE_BOOT0_PA                      0x48281800
>>> +#define API_HYP_ENTRY                          0x102
>>>
>>>  /*
>>>   * OMAP5 specific entry point for secondary CPU to jump from ROM
>>> @@ -38,6 +39,12 @@ wait:        ldr     r2, =AUX_CORE_BOOT0_PA  @ read from AuxCoreBoot0
>>>         and     r4, r4, #0x0f
>>>         cmp     r0, r4
>>>         bne     wait
>>> +#ifdef CONFIG_KVM_ARM_HOST
>>> +       ldr     r12, =API_HYP_ENTRY
>>> +       adr     r0, hyp_boot
>>> +       smc     #0
>>> +hyp_boot:
>>> +#endif
>>>         b       secondary_startup
>>>  END(omap5_secondary_startup)
>>>  /*
>>
>> hmm, this means that currently running this in a guest will fail to
>> bring-up SMP, right?
>>
> Nope. Because the code under 'KVM_ARM_HOST' macro. Guest build 
> will not enable CONFIG_KVM_ARM_HOST and things should be fine then.
> Right ? 

Absolutely not. We're way past the "one image per usage" model, and I
fully expect the same kernel to boot both as a guest and a host.
Otherwise, multiplatform is just a joke.

>> Couldn't you create a little wrapper-pen in U-Boot instead, which
>> replicates the omap boot protocol and takes care of the hyp-mode
>> startup there instead, keeping this completely out of the kernel?
>>
> Its not just booting but CPU hotplug also follows the same path
> so we need the mechanism in kernel to switch mode.
> 
> In general, I think its important to consider the aspect with
> CPU PM. CPUs are not going to go through the boot-loaders in
> those paths and hence need of HYP entry in the kernel will
> be must.

I definitely agree that you need to consider PM as well. But we've
decided a long time ago that we wouldn't take that kind of code in the
kernel. Your CPU has to reach the kernel in HYP, and that is down to
your boot protocol to take care off that.

There are a few options. Christoffer mentioned having a pen, and you
also could implement PSCI fairly easily (I've recently posted a u-boot
patch series to take care of this, though there is still a fair bit of
work required).

Whatever the approach is, the bottom line remains: your CPU enters the
kernel in HYP.

Cheers,

	M.
-- 
Jazz is not dead. It just smells funny...

[PATCH 2/2] ARM: OMAP5: Add HYP mode entry support for secondary CPUs

[Santosh Shilimkar]

On Monday 25 November 2013 11:42 AM, Marc Zyngier wrote:
> On 25/11/13 16:28, Santosh Shilimkar wrote:
>> On Monday 25 November 2013 10:09 AM, Christoffer Dall wrote:
>>> On 23 November 2013 16:07, Santosh Shilimkar <santosh.shilimkar@ti.com> wrote:
>>>> Boot-CPU entry into the HYP mode is managed in boot-loader but
>>>> the secondary CPUs directly jumps to kernel during boot. Same
>>>> path is also used for CPU hotplug as well during suspend for
>>>> secondary CPU.
>>>>
>>>> Hence patch the secondary CPU boot path for hyp mode etry.
>>>>
>>>> Cc: Marc Zyngier <marc.zyngier@arm.com>
>>>> Cc: Christoffer Dall <christoffer.dall@linaro.org>
>>>> Cc: Tony Lindgren <tony@atomide.com>
>>>> Signed-off-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
>>>> ---
>>>>  arch/arm/mach-omap2/omap-headsmp.S |    7 +++++++
>>>>  1 file changed, 7 insertions(+)
>>>>
>>>> diff --git a/arch/arm/mach-omap2/omap-headsmp.S b/arch/arm/mach-omap2/omap-headsmp.S
>>>> index 75e9295..4844dd8 100644
>>>> --- a/arch/arm/mach-omap2/omap-headsmp.S
>>>> +++ b/arch/arm/mach-omap2/omap-headsmp.S
>>>> @@ -22,6 +22,7 @@
>>>>
>>>>  /* Physical address needed since MMU not enabled yet on secondary core */
>>>>  #define AUX_CORE_BOOT0_PA                      0x48281800
>>>> +#define API_HYP_ENTRY                          0x102
>>>>
>>>>  /*
>>>>   * OMAP5 specific entry point for secondary CPU to jump from ROM
>>>> @@ -38,6 +39,12 @@ wait:        ldr     r2, =AUX_CORE_BOOT0_PA  @ read from AuxCoreBoot0
>>>>         and     r4, r4, #0x0f
>>>>         cmp     r0, r4
>>>>         bne     wait
>>>> +#ifdef CONFIG_KVM_ARM_HOST
>>>> +       ldr     r12, =API_HYP_ENTRY
>>>> +       adr     r0, hyp_boot
>>>> +       smc     #0
>>>> +hyp_boot:
>>>> +#endif
>>>>         b       secondary_startup
>>>>  END(omap5_secondary_startup)
>>>>  /*
>>>
>>> hmm, this means that currently running this in a guest will fail to
>>> bring-up SMP, right?
>>>
>> Nope. Because the code under 'KVM_ARM_HOST' macro. Guest build 
>> will not enable CONFIG_KVM_ARM_HOST and things should be fine then.
>> Right ? 
> 
> Absolutely not. We're way past the "one image per usage" model, and I
> fully expect the same kernel to boot both as a guest and a host.
> Otherwise, multiplatform is just a joke.
> 
>>> Couldn't you create a little wrapper-pen in U-Boot instead, which
>>> replicates the omap boot protocol and takes care of the hyp-mode
>>> startup there instead, keeping this completely out of the kernel?
>>>
>> Its not just booting but CPU hotplug also follows the same path
>> so we need the mechanism in kernel to switch mode.
>>
>> In general, I think its important to consider the aspect with
>> CPU PM. CPUs are not going to go through the boot-loaders in
>> those paths and hence need of HYP entry in the kernel will
>> be must.
> 
> I definitely agree that you need to consider PM as well. But we've
> decided a long time ago that we wouldn't take that kind of code in the
> kernel. Your CPU has to reach the kernel in HYP, and that is down to
> your boot protocol to take care off that.
> 
> There are a few options. Christoffer mentioned having a pen, and you
> also could implement PSCI fairly easily (I've recently posted a u-boot
> patch series to take care of this, though there is still a fair bit of
> work required).
> 
> Whatever the approach is, the bottom line remains: your CPU enters the
> kernel in HYP.
> 
Our emails crossed. I understand the stand from you guys. Lets
continue discussion on other thread.