[U-Boot] Question re HYP mode and IRQ/FIQ stack setting

[U-Boot] Question re HYP mode and IRQ/FIQ stack setting

[Albert ARIBAUD]

(Cc:ing Andre and Christoffer as they have discussed HYP on the ML.)

Hello,

I am working on changing the way IRQ/FIQ stacks are set up, from
"on-the-fly in a hurry while in the handler" to "during init, so that
when entering the handler, the stack is already correct".

Setting the stack then requires switching from the current mode (in
most cases, SVC32, 0x13) to IRQ (0x11) or FIQ (0x12) mode, in order to
set the right banked SP, then back into the original mode.

However, in the first lines of reset in arch/arm/cpu/armv7/start.S, the
possibility of U-Boot being started in HYP mode (0x1A) is considered
and, if in HYP mode, no switch to SVC32 is performed.

I understand that the problem here is, if we drop from HYP to SVC32,
then we cannot go back to HYP, and we want to be able to remain in HYP.

Does this also apply to dropping from HYP to IRQ or FIQ mode, i.e., if
we do such a drop, are we prevented from rising back from IRQ or FIQ
mode to HYP? I seem to remember such an issue, but I am no specialist
in HYP, so any help is welcome.

Amicalement,
-- 
Albert.

[U-Boot] Question re HYP mode and IRQ/FIQ stack setting

[Christoffer Dall]

On 12 November 2013 03:41, Albert ARIBAUD <albert.u.boot@aribaud.net> wrote:
> (Cc:ing Andre and Christoffer as they have discussed HYP on the ML.)
>
> Hello,
>
> I am working on changing the way IRQ/FIQ stacks are set up, from
> "on-the-fly in a hurry while in the handler" to "during init, so that
> when entering the handler, the stack is already correct".
>
> Setting the stack then requires switching from the current mode (in
> most cases, SVC32, 0x13) to IRQ (0x11) or FIQ (0x12) mode, in order to
> set the right banked SP, then back into the original mode.
>
> However, in the first lines of reset in arch/arm/cpu/armv7/start.S, the
> possibility of U-Boot being started in HYP mode (0x1A) is considered
> and, if in HYP mode, no switch to SVC32 is performed.
>
> I understand that the problem here is, if we drop from HYP to SVC32,
> then we cannot go back to HYP, and we want to be able to remain in HYP.

correct (not without setting up a trap handler in Hyp mode and
trapping to Hyp mode)

>
> Does this also apply to dropping from HYP to IRQ or FIQ mode, i.e., if
> we do such a drop, are we prevented from rising back from IRQ or FIQ
> mode to HYP? I seem to remember such an issue, but I am no specialist
> in HYP, so any help is welcome.

Yes, it also applies.  Hyp is strictly more privileged (PL2) than all
the PL1 modes (SVC, SYS, IRQ, FIQ, ABT, UND) and therefore requires a
trap to go from PL1 to PL2 (basically this is how hardware protection
works - just like with syscalls from user mode to PL1).

You can use MSR and MRS instructions to access the IRQ and FIQ
registers directly from Hyp mode though.

-Christoffer

[U-Boot] Question re HYP mode and IRQ/FIQ stack setting

[Andre Przywara]

On 11/12/2013 05:28 PM, Christoffer Dall wrote:
> On 12 November 2013 03:41, Albert ARIBAUD <albert.u.boot@aribaud.net> wrote:
>> (Cc:ing Andre and Christoffer as they have discussed HYP on the ML.)
>>
>> Hello,
>>
>> I am working on changing the way IRQ/FIQ stacks are set up, from
>> "on-the-fly in a hurry while in the handler" to "during init, so that
>> when entering the handler, the stack is already correct".
>>
>> Setting the stack then requires switching from the current mode (in
>> most cases, SVC32, 0x13) to IRQ (0x11) or FIQ (0x12) mode, in order to
>> set the right banked SP, then back into the original mode.
>>
>> However, in the first lines of reset in arch/arm/cpu/armv7/start.S, the
>> possibility of U-Boot being started in HYP mode (0x1A) is considered
>> and, if in HYP mode, no switch to SVC32 is performed.
>>
>> I understand that the problem here is, if we drop from HYP to SVC32,
>> then we cannot go back to HYP, and we want to be able to remain in HYP.

Right, that is to keep the HYP mode in case the firmware already enabled 
it. This is for instance the case on the new Calxeda Midway. Actually 
this approach will become more widespread, since it is required to 
provide proper PSCI support (which needs to run in secure state, so 
requires an even higher privilege level than HYP: EL3 in the new ARM speak).

> correct (not without setting up a trap handler in Hyp mode and
> trapping to Hyp mode)
>
>>
>> Does this also apply to dropping from HYP to IRQ or FIQ mode, i.e., if
>> we do such a drop, are we prevented from rising back from IRQ or FIQ
>> mode to HYP? I seem to remember such an issue, but I am no specialist
>> in HYP, so any help is welcome.
>
> Yes, it also applies.  Hyp is strictly more privileged (PL2) than all
> the PL1 modes (SVC, SYS, IRQ, FIQ, ABT, UND) and therefore requires a
> trap to go from PL1 to PL2 (basically this is how hardware protection
> works - just like with syscalls from user mode to PL1).

Thanks Christoffer for clarifying this, I wasn't sure about FIQ, but of 
course your explanation (EL1 vs. EL2) makes totally sense.

But I wonder what happens when we enter FIQ or IRQ due to an actual 
interrupt. Will the CPU return into HYP mode when the handler returns?
That is subject to some HYP mode register configuration, right?

> You can use MSR and MRS instructions to access the IRQ and FIQ
> registers directly from Hyp mode though.

Albert,
so does "msr sp_{fiq,irq}, r<n>" fix your problem? Or do you still need 
to actually go into one of these modes for further setup?

Regards,
Andre.

[U-Boot] Question re HYP mode and IRQ/FIQ stack setting

[Christoffer Dall]

On 12 November 2013 08:53, Andre Przywara <andre.przywara@linaro.org> wrote:
> On 11/12/2013 05:28 PM, Christoffer Dall wrote:
>>
>> On 12 November 2013 03:41, Albert ARIBAUD <albert.u.boot@aribaud.net>
>> wrote:
>>>
>>> (Cc:ing Andre and Christoffer as they have discussed HYP on the ML.)
>>>
>>> Hello,
>>>
>>> I am working on changing the way IRQ/FIQ stacks are set up, from
>>> "on-the-fly in a hurry while in the handler" to "during init, so that
>>> when entering the handler, the stack is already correct".
>>>
>>> Setting the stack then requires switching from the current mode (in
>>> most cases, SVC32, 0x13) to IRQ (0x11) or FIQ (0x12) mode, in order to
>>> set the right banked SP, then back into the original mode.
>>>
>>> However, in the first lines of reset in arch/arm/cpu/armv7/start.S, the
>>> possibility of U-Boot being started in HYP mode (0x1A) is considered
>>> and, if in HYP mode, no switch to SVC32 is performed.
>>>
>>> I understand that the problem here is, if we drop from HYP to SVC32,
>>> then we cannot go back to HYP, and we want to be able to remain in HYP.
>
>
> Right, that is to keep the HYP mode in case the firmware already enabled it.
> This is for instance the case on the new Calxeda Midway. Actually this
> approach will become more widespread, since it is required to provide proper
> PSCI support (which needs to run in secure state, so requires an even higher
> privilege level than HYP: EL3 in the new ARM speak).
>

What Andre is referring to here is that the PSCI specs mandates the
CPUs that a are turned on using PSCI go through the highest privilege
level implemented in the non-secure state.  I couldn't actually find
the place in the PSCI specs where this is also true for the boot CPU,
but granted, anything else will be quite strange.

So to stay in 32-bit ARMv7 terminology here, a likely case for systems
with PSCI support in the firmware is that all cores will enter U-Boot
in non-secure PL2 (hyp) mode and power management is done by calling
SMC calls to the firmware that sits in the secure monitor.

>
>> correct (not without setting up a trap handler in Hyp mode and
>> trapping to Hyp mode)
>>
>>>
>>> Does this also apply to dropping from HYP to IRQ or FIQ mode, i.e., if
>>> we do such a drop, are we prevented from rising back from IRQ or FIQ
>>> mode to HYP? I seem to remember such an issue, but I am no specialist
>>> in HYP, so any help is welcome.
>>
>>
>> Yes, it also applies.  Hyp is strictly more privileged (PL2) than all
>> the PL1 modes (SVC, SYS, IRQ, FIQ, ABT, UND) and therefore requires a
>> trap to go from PL1 to PL2 (basically this is how hardware protection
>> works - just like with syscalls from user mode to PL1).
>
>
> Thanks Christoffer for clarifying this, I wasn't sure about FIQ, but of
> course your explanation (EL1 vs. EL2) makes totally sense.
>
> But I wonder what happens when we enter FIQ or IRQ due to an actual
> interrupt. Will the CPU return into HYP mode when the handler returns?
> That is subject to some HYP mode register configuration, right?
>

Not quite, interrupts in Hyp mode will always be taken to Hyp mode
(unless it's secure interrupts, which are taken to monitor mode).  In
fact, when you have the virtualization extensions, you also have the
security extensions, and I think all FIQs go to the secure mode here,
so you can't ever see an FIQ in Hyp mode.  I may be mistaken here,
someone should look it up and verify.

Now, you can configure Hyp mode to trap interrupts raised while in any
PL1 or PL0 mode to Hyp mode.

I suspect that if you are in Hyp mode, you should not worry about
FIQ/IRQ mode, but just make sure to configure Hyp mode properly to
handle interrupts.  (it's a separate entry in the exception vector and
you probably need to look at the HSR register whn you've taken an
interrupt).  So, as Andre suggests below, it depends on your use case.

>
>> You can use MSR and MRS instructions to access the IRQ and FIQ
>> registers directly from Hyp mode though.
>
>
> Albert,
> so does "msr sp_{fiq,irq}, r<n>" fix your problem? Or do you still need to
> actually go into one of these modes for further setup?
>
> Regards,
> Andre.
>

[U-Boot] Question re HYP mode and IRQ/FIQ stack setting

[Albert ARIBAUD]

Hi Christoffer,

On Tue, 12 Nov 2013 09:09:23 -0800, Christoffer Dall
<christoffer.dall@linaro.org> wrote:

> I suspect that if you are in Hyp mode, you should not worry about
> FIQ/IRQ mode, but just make sure to configure Hyp mode properly to
> handle interrupts.  (it's a separate entry in the exception vector and
> you probably need to look at the HSR register whn you've taken an
> interrupt).  So, as Andre suggests below, it depends on your use case.

Ok, so les me try to sum that up from my perspective (IRQ and FIQ
stacks in U-Boot):

Some ARM U-Boot targets run in SVC32 mode, and some in HYP mode.

For targets which run in SVC32 mode: aborts execute in abort mode, IRQs
in IRQ mode, FIQs in FIQ mode, etc., each mode having its own stack.

For targets which run in HYP mode: aborts, FIQ and IRQs all run in HYP
mode, using always the same stack.

In both types of targers: prefetch and data aborts, FIQs and IRQs
execute through the usual vectors; in SVC32 mode, because there's no
other way; in HYP mode, because exceptions occurring while in HYP
mode use the normal vector, not the HYP vector.

Correct?

If so, then U-Boot stack requirements for SVC32 and HYP mode are
as follows: in SVC32, we need separate IRQ and FIQ stacks, and the main
(SVC32) stack does not have to accommodate for interrupt handler
context storage. In HYP mode, the only stack is the HYP one and
exception handlers will use it too, so it has to accommodate for
their context storage.  

Still correct?

Amicalement,
-- 
Albert.

[U-Boot] Question re HYP mode and IRQ/FIQ stack setting

[Christoffer Dall]

On 12 November 2013 13:29, Albert ARIBAUD <albert.u.boot@aribaud.net> wrote:
> Hi Christoffer,
>
> On Tue, 12 Nov 2013 09:09:23 -0800, Christoffer Dall
> <christoffer.dall@linaro.org> wrote:
>
>> I suspect that if you are in Hyp mode, you should not worry about
>> FIQ/IRQ mode, but just make sure to configure Hyp mode properly to
>> handle interrupts.  (it's a separate entry in the exception vector and
>> you probably need to look at the HSR register whn you've taken an
>> interrupt).  So, as Andre suggests below, it depends on your use case.
>
> Ok, so les me try to sum that up from my perspective (IRQ and FIQ
> stacks in U-Boot):
>
> Some ARM U-Boot targets run in SVC32 mode, and some in HYP mode.
>
> For targets which run in SVC32 mode: aborts execute in abort mode, IRQs
> in IRQ mode, FIQs in FIQ mode, etc., each mode having its own stack.

correct

>
> For targets which run in HYP mode: aborts, FIQ and IRQs all run in HYP
> mode, using always the same stack.

correct.

>
> In both types of targers: prefetch and data aborts, FIQs and IRQs
> execute through the usual vectors; in SVC32 mode, because there's no
> other way; in HYP mode, because exceptions occurring while in HYP
> mode use the normal vector, not the HYP vector.
>
> Correct?
>

no, this sounds fishy. Look at the ARM ARMv7 (DDI 0406C.b) page
B1-1166.  Table B1-3 will tell you what you need to know.

Hyp mode has its own vectors, pointed to by the HVBAR control register.

I strongly suggest you familiarize yourself with these parts of the
ARM ARM before writing code to that effect.

> If so, then U-Boot stack requirements for SVC32 and HYP mode are
> as follows: in SVC32, we need separate IRQ and FIQ stacks, and the main
> (SVC32) stack does not have to accommodate for interrupt handler
> context storage. In HYP mode, the only stack is the HYP one and
> exception handlers will use it too, so it has to accommodate for
> their context storage.
>
That is correct, your Hyp mode stack should always be valid (have a
page or so, like the kernel) and then you can always push things on
there, even when taking an exception.

-Christoffer

[U-Boot] Question re HYP mode and IRQ/FIQ stack setting

[Albert ARIBAUD]

Hi Christoffer,

On Tue, 12 Nov 2013 14:34:00 -0800, Christoffer Dall
<christoffer.dall@linaro.org> wrote:

> On 12 November 2013 13:29, Albert ARIBAUD <albert.u.boot@aribaud.net> wrote:
> > Hi Christoffer,
> >
> > On Tue, 12 Nov 2013 09:09:23 -0800, Christoffer Dall
> > <christoffer.dall@linaro.org> wrote:
> >
> >> I suspect that if you are in Hyp mode, you should not worry about
> >> FIQ/IRQ mode, but just make sure to configure Hyp mode properly to
> >> handle interrupts.  (it's a separate entry in the exception vector and
> >> you probably need to look at the HSR register whn you've taken an
> >> interrupt).  So, as Andre suggests below, it depends on your use case.
> >
> > Ok, so les me try to sum that up from my perspective (IRQ and FIQ
> > stacks in U-Boot):
> >
> > Some ARM U-Boot targets run in SVC32 mode, and some in HYP mode.
> >
> > For targets which run in SVC32 mode: aborts execute in abort mode, IRQs
> > in IRQ mode, FIQs in FIQ mode, etc., each mode having its own stack.
> 
> correct
> 
> >
> > For targets which run in HYP mode: aborts, FIQ and IRQs all run in HYP
> > mode, using always the same stack.
> 
> correct.
> 
> >
> > In both types of targers: prefetch and data aborts, FIQs and IRQs
> > execute through the usual vectors; in SVC32 mode, because there's no
> > other way; in HYP mode, because exceptions occurring while in HYP
> > mode use the normal vector, not the HYP vector.
> >
> > Correct?
> >
> 
> no, this sounds fishy. Look at the ARM ARMv7 (DDI 0406C.b) page
> B1-1166.  Table B1-3 will tell you what you need to know.
> 
> Hyp mode has its own vectors, pointed to by the HVBAR control register.
> 
> I strongly suggest you familiarize yourself with these parts of the
> ARM ARM before writing code to that effect.

To clarify my meaning: I wasn't speaking about the exceptions base
address, but about the exception vector offset. IOW, I did not mean to
say that in HYP mode, aborts, FIQs, IRQs etc use the SVC32 table of
vectors rather than the HYP table. I meant to say that e.g. an IRQ will
fire at offset 0x18, whether it fires while from SVC32 into IRQ mode,
or from HYP mode into HYP mode. This seemed consistent with figure B1-8
on page B1-1179 of DDI 0406C.b, respectively first and sixth "exit box"
on the right.

Is this clearer and more correct?

> > If so, then U-Boot stack requirements for SVC32 and HYP mode are
> > as follows: in SVC32, we need separate IRQ and FIQ stacks, and the main
> > (SVC32) stack does not have to accommodate for interrupt handler
> > context storage. In HYP mode, the only stack is the HYP one and
> > exception handlers will use it too, so it has to accommodate for
> > their context storage.
> >
> That is correct, your Hyp mode stack should always be valid (have a
> page or so, like the kernel) and then you can always push things on
> there, even when taking an exception.

Thanks Christoffer! This should allow me to refactor interrupt stack
settings as I intended.

> -Christoffer

Amicalement,
-- 
Albert.

[U-Boot] Question re HYP mode and IRQ/FIQ stack setting

[Christoffer Dall]

On 12 November 2013 22:22, Albert ARIBAUD <albert.u.boot@aribaud.net> wrote:
> Hi Christoffer,
>
> On Tue, 12 Nov 2013 14:34:00 -0800, Christoffer Dall
> <christoffer.dall@linaro.org> wrote:
>
>> On 12 November 2013 13:29, Albert ARIBAUD <albert.u.boot@aribaud.net> wrote:
>> > Hi Christoffer,
>> >
>> > On Tue, 12 Nov 2013 09:09:23 -0800, Christoffer Dall
>> > <christoffer.dall@linaro.org> wrote:
>> >
>> >> I suspect that if you are in Hyp mode, you should not worry about
>> >> FIQ/IRQ mode, but just make sure to configure Hyp mode properly to
>> >> handle interrupts.  (it's a separate entry in the exception vector and
>> >> you probably need to look at the HSR register whn you've taken an
>> >> interrupt).  So, as Andre suggests below, it depends on your use case.
>> >
>> > Ok, so les me try to sum that up from my perspective (IRQ and FIQ
>> > stacks in U-Boot):
>> >
>> > Some ARM U-Boot targets run in SVC32 mode, and some in HYP mode.
>> >
>> > For targets which run in SVC32 mode: aborts execute in abort mode, IRQs
>> > in IRQ mode, FIQs in FIQ mode, etc., each mode having its own stack.
>>
>> correct
>>
>> >
>> > For targets which run in HYP mode: aborts, FIQ and IRQs all run in HYP
>> > mode, using always the same stack.
>>
>> correct.
>>
>> >
>> > In both types of targers: prefetch and data aborts, FIQs and IRQs
>> > execute through the usual vectors; in SVC32 mode, because there's no
>> > other way; in HYP mode, because exceptions occurring while in HYP
>> > mode use the normal vector, not the HYP vector.
>> >
>> > Correct?
>> >
>>
>> no, this sounds fishy. Look at the ARM ARMv7 (DDI 0406C.b) page
>> B1-1166.  Table B1-3 will tell you what you need to know.
>>
>> Hyp mode has its own vectors, pointed to by the HVBAR control register.
>>
>> I strongly suggest you familiarize yourself with these parts of the
>> ARM ARM before writing code to that effect.
>
> To clarify my meaning: I wasn't speaking about the exceptions base
> address, but about the exception vector offset. IOW, I did not mean to
> say that in HYP mode, aborts, FIQs, IRQs etc use the SVC32 table of
> vectors rather than the HYP table. I meant to say that e.g. an IRQ will
> fire at offset 0x18, whether it fires while from SVC32 into IRQ mode,
> or from HYP mode into HYP mode. This seemed consistent with figure B1-8
> on page B1-1179 of DDI 0406C.b, respectively first and sixth "exit box"
> on the right.
>
> Is this clearer and more correct?
>

yes, now I understand. It's correct.

>> > If so, then U-Boot stack requirements for SVC32 and HYP mode are
>> > as follows: in SVC32, we need separate IRQ and FIQ stacks, and the main
>> > (SVC32) stack does not have to accommodate for interrupt handler
>> > context storage. In HYP mode, the only stack is the HYP one and
>> > exception handlers will use it too, so it has to accommodate for
>> > their context storage.
>> >
>> That is correct, your Hyp mode stack should always be valid (have a
>> page or so, like the kernel) and then you can always push things on
>> there, even when taking an exception.
>
> Thanks Christoffer! This should allow me to refactor interrupt stack
> settings as I intended.
>
>> -Christoffer
>
> Amicalement,
> --
> Albert.