MCA/INIT: need to wire CURRENT_STACK?

MCA/INIT: need to wire CURRENT_STACK?

[Keith Owens]

I put this comment in arch/ia64/kernel/mca_asm.S, where we switch to
the MCA/INIT stack.  The code runs fine without wiring
IA64_KR_CURRENT_STACK and IA64_TR_CURRENT_STACK.  Is that coincidence
or do the extra registers really need to be updated?  The MCA/INIT
stacks are initialized early in boot, they are in region 7.

Remember that the MCA/INIT handlers run with interrupts disabled.

ia64_set_kernel_registers:
        add temp3=MCA_SP_OFFSET, r3
        add temp4=MCA_SOS_OFFSET+IA64_SAL_OS_STATE_OS_GP_OFFSET, r3
        mov b0=r2               // save return address
        GET_IA64_MCA_DATA(temp1)
        ;;
        add temp4=temp4, temp1  // &struct ia64_sal_os_state.os_gp
        add r12=temp1, temp3    // kernel stack pointer on MCA/INIT stack
        add r13=temp1, r3       // set current to start of MCA/INIT stack
        ;;
        ld8 r1=[temp4]          // OS GP from SAL OS state
        ;;
        DATA_PA_TO_VA(r1,temp1)
        DATA_PA_TO_VA(r12,temp2)
        DATA_PA_TO_VA(r13,temp3)
        ;;
        mov IA64_KR(CURRENT)=r13

        // FIXME: do I need to wire IA64_KR_CURRENT_STACK and IA64_TR_CURRENT_STACK?

        br.sptk b0

RE: MCA/INIT: need to wire CURRENT_STACK?

[Luck, Tony]

>I put this comment in arch/ia64/kernel/mca_asm.S, where we switch to
>the MCA/INIT stack.  The code runs fine without wiring
>IA64_KR_CURRENT_STACK and IA64_TR_CURRENT_STACK.  Is that coincidence
>or do the extra registers really need to be updated?  The MCA/INIT
>stacks are initialized early in boot, they are in region 7.

Those registers keep track of which address we have pinned with
DTR[2] so that context switch code (ia64_switch_to) can figure
out whether it needs to reload DTR[2] to map the stack of the
new process.  Since you aren't going to context switch from your
MCA code (I hope!) this should't be an issue.

BUT ... it raises the question of whether you will do anything
in your MCA handler that will run into problems because the stack
is perhaps[1] not pinned ... just mapped by a DTC entry.  I forget
what the exact issue is that led to pinning the stack with DTR[2],
but there is a correctness issue involved.  David: memory jog time??

-Tony

[1] It might be pinned if it is allocated from the same granule
as the stack of the most recently running process on this cpu,
since DTR[2] just maps a whole granule at a time.  You might also
have a DTC mapping active, so skip lightly past the problem in
simple tests.

Re: MCA/INIT: need to wire CURRENT_STACK?

[David Mosberger-Tang]

> BUT ... it raises the question of whether you will do anything
> in your MCA handler that will run into problems because the stack
> is perhaps[1] not pinned ... just mapped by a DTC entry.  I forget
> what the exact issue is that led to pinning the stack with DTR[2],
> but there is a correctness issue involved.  David: memory jog time??

Any virtually mapped memory that needs to be accessed with PSR.IC=0
needs to be pinned because otherwise you'd get a nested TLB fault and
things would go south from there real fast.

  --david
--
Mosberger Consulting LLC, voice/fax: 510-744-9372,
http://www.mosberger-consulting.com/
35706 Runckel Lane, Fremont, CA 94536

Re: MCA/INIT: need to wire CURRENT_STACK?

[Keith Owens]

On Wed, 21 Sep 2005 11:12:58 -0700, 
David Mosberger-Tang <David.Mosberger@acm.org> wrote:
>> BUT ... it raises the question of whether you will do anything
>> in your MCA handler that will run into problems because the stack
>> is perhaps[1] not pinned ... just mapped by a DTC entry.  I forget
>> what the exact issue is that led to pinning the stack with DTR[2],
>> but there is a correctness issue involved.  David: memory jog time??
>
>Any virtually mapped memory that needs to be accessed with PSR.IC=0
>needs to be pinned because otherwise you'd get a nested TLB fault and
>things would go south from there real fast.

The virtual mode handlers are entered with psr.ic=1 and psr.i=0.
External interrupts are blocked until the handlers resume to the
original context.  Internal interrupts like KDB_ENTER() use the
handler's stack in virtual mode with psr.ic=0, but by the time it gets
to ivt.S::non_syscall, the C code has already accessed the handler
stack with psr.ic=1.  Worst case, the first access to the handler stack
in mca.c (which is with psr.ic) will take an alt DTLB fault and drop in
a DTC for the handler stack.

There is a theoretical problem case.  After the DTC for the handler
stack has been loaded, the handler does enough data accesses that its
own DTC is flushed.  Then the handler, without accessing its own stack,
causes an internal interrupt, e.g. KDB_ENTER().  That could cause a
nested DTLB fault.

I can change the mca_asm code to save, modify and restore
IA64_KR_CURRENT_STACK and IA64_TR_CURRENT_STACK around the C handlers.
But that gets awkward if the C code wants to resume to a new context,
the C code would have to set new values for IA64_KR_CURRENT_STACK and
IA64_TR_CURRENT_STACK while it was still using its own stack, which
might open the same problem.

Re: MCA/INIT: need to wire CURRENT_STACK?

[David Mosberger-Tang]

On 9/21/05, Keith Owens <kaos@sgi.com> wrote:
> On Wed, 21 Sep 2005 11:12:58 -0700,

> The virtual mode handlers are entered with psr.ic=1 and psr.i=0.

I'm not sure what you mean by virtual mode handlers, but execution in
the IVT starts with PSR.IC=0.

  --david

Re: MCA/INIT: need to wire CURRENT_STACK?

[Keith Owens]

On Wed, 21 Sep 2005 16:09:49 -0700, 
David Mosberger-Tang <David.Mosberger@acm.org> wrote:
>On 9/21/05, Keith Owens <kaos@sgi.com> wrote:
>> On Wed, 21 Sep 2005 11:12:58 -0700,
>
>> The virtual mode handlers are entered with psr.ic=1 and psr.i=0.
>
>I'm not sure what you mean by virtual mode handlers,

The C code that starts with ia64_mca_handler and ia64_init_handler.
mca_asm.S is entered in physical mode with psr.ic=0, it switches to
virtual with psr.ic=1 and psr.i=0 before calling the handlers.

>but execution in
>the IVT starts with PSR.IC=0.

Right, so the real question is - is there any path that causes an
interrupt before the virtual mode handlers have accessed their stacks
and the required DTC has been created?  Alternatively can the DTC for
the MCA/INIT stacks be lost once the handlers have started running?

Given the uncertainty and the fact the the TLB error recovery in
mca_asm.S assumes that CURRENT_STACK is defined, I need to wire
CURRENT_STACK around the C code handlers.  Patch coming soon.

BTW David, are you aware that your mailer is using quoted printable,
even for plain text messages?  All the '=' signs come out as '='.