[Qemu-devel] Wrong reset of MIPS hflags EXL after interrupt?

[Qemu-devel] Wrong reset of MIPS hflags EXL after interrupt?

[Dirk Behme]

Hi,

I'm not sure, but while playing with MIPS interrupts, it 
seems to me that something with reset of interrupt flag 
MIPS_HFLAG_EXL (0x04) at exception exit (eret) is wrong. It 
seems to me that only one interrupt is executed because 
after eret, MIPS_HFLAG_EXL stays set in env->hflags. Then, 
at next interrupt, system correctly checks for 
MIPS_HFLAG_EXL, but this is still set and no further 
interrupt happens.

Debugging shows that op_eret() in MIPS op.c correctly reset 
this bit: env->hflags &= ~MIPS_HFLAG_EXL; But debug output 
at end of e.g. save_cpu_state() (debug output of ctx->hflags 
and ctx->saved_hflags ) or in function which tries to issue 
(next) timer interrupt (debug output of env->hflags) 
MIPS_HFLAG_EXL is still (again?) set everywhere. Looks like 
the correct env->hflags from op_eret() is overwritten 
somewhere later with wrong value.

These three ctx->hflags, ctx->saved_hflags and env->hflags 
are confusing me ;) Where are they synchronized after eret? 
Or who overwrites the env->hflags correctly set by eret 
again? Any ideas, why eret sets env->hflags correctly and 
later global env->hflags has still/again wrong value? Any 
other hints?

Many thanks

Dirk

Debug output shows something like:

save_cpu_state(): ctx->hflags 00000000 ctx->saved_hflags 
00000000
MIPS Timer #1: Status: 0x80408401, Cause: 0x00008000, 
env->hflags: 0x00000000
save_cpu_state():ctx->hflags 00000004 ctx->saved_hflags 00000004
** Interrupt handler called...
** Start of op_eret(): env->hflags 0x00000204 Status 0x80408403
** End of op_eret(): env->hflags 0x00000200 Status 0x80408401
save_cpu_state():ctx->hflags 00000004 ctx->saved_hflags 00000004
save_cpu_state():ctx->hflags 00000004 ctx->saved_hflags 00000004
save_cpu_state():ctx->hflags 00000004 ctx->saved_hflags 00000004
...
MIPS Timer #2: Status: 0x80408401, Cause: 0x00008000, 
env->hflags: 0x00000004
MIPS Timer #3: Status: 0x80408401, Cause: 0x00008000, 
env->hflags: 0x00000004
MIPS Timer #4: Status: 0x80408401, Cause: 0x00008000, 
env->hflags: 0x00000004
MIPS Timer #5: Status: 0x80408401, Cause: 0x00008000, 
env->hflags: 0x00000004
....

Re: [Qemu-devel] Wrong reset of MIPS hflags EXL after interrupt?

[Thiemo Seufer]

Dirk Behme wrote:
> Hi,
> 
> I'm not sure, but while playing with MIPS interrupts, it 
> seems to me that something with reset of interrupt flag 
> MIPS_HFLAG_EXL (0x04) at exception exit (eret) is wrong. It 
> seems to me that only one interrupt is executed because 
> after eret, MIPS_HFLAG_EXL stays set in env->hflags. Then, 
> at next interrupt, system correctly checks for 
> MIPS_HFLAG_EXL, but this is still set and no further 
> interrupt happens.

This explains some weirdness I saw on my hacked up qemu
when running a mips32r2-compiled Linux kernel.

> Debugging shows that op_eret() in MIPS op.c correctly reset 
> this bit: env->hflags &= ~MIPS_HFLAG_EXL; But debug output 
> at end of e.g. save_cpu_state() (debug output of ctx->hflags 
> and ctx->saved_hflags ) or in function which tries to issue 
> (next) timer interrupt (debug output of env->hflags) 
> MIPS_HFLAG_EXL is still (again?) set everywhere. Looks like 
> the correct env->hflags from op_eret() is overwritten 
> somewhere later with wrong value.
> 
> These three ctx->hflags, ctx->saved_hflags and env->hflags 
> are confusing me ;) Where are they synchronized after eret? 
> Or who overwrites the env->hflags correctly set by eret 
> again? Any ideas, why eret sets env->hflags correctly and 
> later global env->hflags has still/again wrong value? Any 
> other hints?

AFAIU qemu maintains an environment stack, I guess popping the
environment restores the old flag contents.


Thiemo

Re: [Qemu-devel] Wrong reset of MIPS hflags EXL after interrupt?

[Dirk Behme]

Thiemo Seufer wrote:
> Dirk Behme wrote:
>>I'm not sure, but while playing with MIPS interrupts, it 
>>seems to me that something with reset of interrupt flag 
>>MIPS_HFLAG_EXL (0x04) at exception exit (eret) is wrong. It 
>>seems to me that only one interrupt is executed because 
>>after eret, MIPS_HFLAG_EXL stays set in env->hflags. Then, 
>>at next interrupt, system correctly checks for 
>>MIPS_HFLAG_EXL, but this is still set and no further 
>>interrupt happens.
> 
> This explains some weirdness I saw on my hacked up qemu
> when running a mips32r2-compiled Linux kernel.

This was the reason why I first tried it with a small test 
program before using Linux ;)

>>Debugging shows that op_eret() in MIPS op.c correctly reset 
>>this bit: env->hflags &= ~MIPS_HFLAG_EXL; But debug output 
>>at end of e.g. save_cpu_state() (debug output of ctx->hflags 
>>and ctx->saved_hflags ) or in function which tries to issue 
>>(next) timer interrupt (debug output of env->hflags) 
>>MIPS_HFLAG_EXL is still (again?) set everywhere. Looks like 
>>the correct env->hflags from op_eret() is overwritten 
>>somewhere later with wrong value.
>>
>>These three ctx->hflags, ctx->saved_hflags and env->hflags 
>>are confusing me ;) Where are they synchronized after eret? 
>>Or who overwrites the env->hflags correctly set by eret 
>>again? Any ideas, why eret sets env->hflags correctly and 
>>later global env->hflags has still/again wrong value? Any 
>>other hints?
> 
> AFAIU qemu maintains an environment stack, I guess popping the
> environment restores the old flag contents.

Anybody with a short explanation of the basics of this? I 
think this would really help debugging this issue.

Many thanks

Dirk

Re: [Qemu-devel] Wrong reset of MIPS hflags EXL after interrupt?

[Marius Groeger]

On Wed, 16 Aug 2006, Dirk Behme wrote:

>> AFAIU qemu maintains an environment stack, I guess popping the
>> environment restores the old flag contents.
>
> Anybody with a short explanation of the basics of this? I think this would 
> really help debugging this issue.

I don't think it's really a stack (see translate.c:save_cpu_state()), 
but anyway as far as I got it it is used to save the context where the 
emulation has to prepare to deliver that context either to an 
exception or to the code managing branches (delay slots, likely's 
etc.) In some instances you'll see that the "pc" counter of the saved 
context is manually incremented by one insn; one obvious example is 
the "wait" instruction where you really want to continue with the 
following insn once an exception kicks the emulation out of its nap.

I agree, though, that in your debugging, you probably examined a wrong
context. The actual, current context's EXL should be correct, otherwise
things wouldn't be working at all.

Having said that, I'm currently playing with nested interrupts - let's see
how that checks out... :-)

Glad for any correction of my half-understanding of qemu,
Marius

-- 
Marius Groeger <mgroeger@sysgo.com>
SYSGO AG                      Embedded and Real-Time Software
Voice: +49 6136 9948 0                  FAX: +49 6136 9948 10
www.sysgo.com | www.elinos.com | www.osek.de | www.pikeos.com

Re: [Qemu-devel] Wrong reset of MIPS hflags EXL after interrupt?

[Dirk Behme]

Marius Groeger wrote:
> I agree, though, that in your debugging, you probably examined a wrong
> context. The actual, current context's EXL should be correct, otherwise
> things wouldn't be working at all.

Argh, debugging different context sounds nice ;) It's my 
feeling that while switching between these different context 
anything with hflags goes wrong. But I'm only guessing here. 
Let's see...

> Having said that, I'm currently playing with nested interrupts - let's see
> how that checks out... :-)

Do several consecutively, non nested, interrupts (e.g. 
priodical timer) work for you?

Cheers

Dirk

Re: [Qemu-devel] Wrong reset of MIPS hflags EXL after interrupt?

[Marius Groeger]

On Thu, 17 Aug 2006, Dirk Behme wrote:

>> Having said that, I'm currently playing with nested interrupts - let's see
>> how that checks out... :-)
>
> Do several consecutively, non nested, interrupts (e.g. priodical timer) work 
> for you?

Yes those are working. I once saw issues with nested(!) Timer/NIC 
interrupts, but I rather suspece issues with the tricky EXL/IE/IMASK 
maniplation code which is involved when allowing for nested 
interrupts. I don't know whether Linux does that, actually.

Marius

-- 
Marius Groeger <mgroeger@sysgo.com>
SYSGO AG                      Embedded and Real-Time Software
Voice: +49 6136 9948 0                  FAX: +49 6136 9948 10
www.sysgo.com | www.elinos.com | www.osek.de | www.pikeos.com

Re: [Qemu-devel] Wrong reset of MIPS hflags EXL after interrupt?

[Marius Groeger]

On Wed, 16 Aug 2006, Thiemo Seufer wrote:

> Dirk Behme wrote:
>> Hi,
>>
>> I'm not sure, but while playing with MIPS interrupts, it
>> seems to me that something with reset of interrupt flag
>> MIPS_HFLAG_EXL (0x04) at exception exit (eret) is wrong. It
>> seems to me that only one interrupt is executed because
>> after eret, MIPS_HFLAG_EXL stays set in env->hflags. Then,
>> at next interrupt, system correctly checks for
>> MIPS_HFLAG_EXL, but this is still set and no further
>> interrupt happens.

Dirk and I have been following up on this privately and could verify that it
was indeed an issue with the testcase. QEMU is not causing any problem 
here.

> This explains some weirdness I saw on my hacked up qemu
> when running a mips32r2-compiled Linux kernel.

What exactly included that hack? Some new mips32r2 insns like rdhw?

FWIW, since there seems to be some FUD in the air, occasionally: I 
would like to state that Linux 2.6.15 and QEMU CVS HEAD works for me 
out-of-the-box. I'm using a MIPS r4k big endian configuration.

On Thu, 17 Aug 2006, Marius Groeger wrote:

> Having said that, I'm currently playing with nested interrupts - let's see
> how that checks out... :-)

I would like to take the opportunity to state that I verified nested
interrupts on QEMU and they seem to be working. The tested sequence was:

userspace
  -> NIC interrupt exception
  -> interrupt service for NIC
  -> Enable interrupts
     -> Timer interrupt exception
 	-> interrupt service for Timer
 	-> Enable Interrupts
 	-> ERET
 	ERET
back in userspace

Regards,
Marius

-- 
Marius Groeger <mgroeger@sysgo.com>
SYSGO AG                      Embedded and Real-Time Software
Voice: +49 6136 9948 0                  FAX: +49 6136 9948 10
www.sysgo.com | www.elinos.com | www.osek.de | www.pikeos.com

Re: [Qemu-devel] Wrong reset of MIPS hflags EXL after interrupt?

[Thiemo Seufer]

Marius Groeger wrote:
> On Wed, 16 Aug 2006, Thiemo Seufer wrote:
> 
> >Dirk Behme wrote:
> >>Hi,
> >>
> >>I'm not sure, but while playing with MIPS interrupts, it
> >>seems to me that something with reset of interrupt flag
> >>MIPS_HFLAG_EXL (0x04) at exception exit (eret) is wrong. It
> >>seems to me that only one interrupt is executed because
> >>after eret, MIPS_HFLAG_EXL stays set in env->hflags. Then,
> >>at next interrupt, system correctly checks for
> >>MIPS_HFLAG_EXL, but this is still set and no further
> >>interrupt happens.
> 
> Dirk and I have been following up on this privately and could verify that it
> was indeed an issue with the testcase. QEMU is not causing any problem 
> here.
> 
> >This explains some weirdness I saw on my hacked up qemu
> >when running a mips32r2-compiled Linux kernel.
> 
> What exactly included that hack? Some new mips32r2 insns like rdhw?

All new r2 instructions. rdhwr wasn't used by the userland.
A recent r2 compiled Kernel will use the di/ei instructions, a
recursive exception will IIRC see a EXL flag set when there shouldn't
be one. The r1 kernel code happens to mask the EXL flag.


Thiemo