tlbia and PPC603

tlbia and PPC603

[Etsushi Kato]

Hi all,

Does _tlbia in arch/ppc/kernel/misc.S work fine on PPC603 Machintosh??


I've experienced instability (random crash) on UMAX C500 with benH
and bk kernel since last November.  Several users of PCI Performa also
reported this symptom as well, like
http://lists.linuxppc.org/linuxppc-user/200205/msg00061.html and
http://lists.linuxppc.org/linuxppc-user/200203/msg00027.html.


My Macintosh are very old, two Nubus Mac (PM6100AV, Performa6200), and
one oldworld PCI Mac (UMAX C500).  And I would like to use linux on
these machines.  Recently, I've updated Nubus-Pmac patch to current
PowerPC Linux tree (2.4.19-pre10).  In this process, I found this
instability also occur on Performa6200 (PPC603) but not in PM6100AV
(PPC601 and also with G3 upgrade card).  Performa6200 sometimes went
to xmon randomly and crashed at local_flush_tlb_mm().  And sometimes
just showing segmentation fault or illegal instruction error.  Then I
suspect _tlbia() is causing the crash, so I experimentally added
"isync" just after "sync" in _GLOBAL(_tlbia) in
arch/ppc/kernel/misc.S.

It appears this change made Performa6200 stable.  Also I tested this
change on UMAX C500 with linux-2.4.19-pre10, and UMAX C500 also became
stable.

I'm not sure adding "isync" is good choice, but it seems some problem
can be solved this.

Cheers,
--
Etsushi Kato
ekato@ees.hokudai.ac.jp

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: tlbia and PPC603

[Benjamin Herrenschmidt]

>"isync" just after "sync" in _GLOBAL(_tlbia) in
>arch/ppc/kernel/misc.S.
>
>It appears this change made Performa6200 stable.  Also I tested this
>change on UMAX C500 with linux-2.4.19-pre10, and UMAX C500 also became
>stable.
>
>I'm not sure adding "isync" is good choice, but it seems some problem
>can be solved this.

There are 2 "sync's" in there, which one did you add the isync after ?
The one before tlbia or the one after tlbia ?

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: tlbia and PPC603

[Benjamin Herrenschmidt]

Ok, after more investigations it appears that:

 - There is no tlbia on 603. Actually, tlbia is #defined in the
kernel to be a loop of tlbie's

 - According to the PEM, tlbie requires a context synchronizing
instruction _before_, and a context sync. or a plain "sync"
instruction _after_. We did have the later but lacked the former
in our _tlbia function (though we did have it in our _tlbie
implementation).

So the proper fix, according to the spec, would be to just replace
the sync we currently do before with an isync. I've updated my rsync
with such a fix, can you confirm it works ?

If it's stable this way, i'll push the fix to the bk trees and
to Marcelo.

I'm not completely sure if we should or shouldn't keep the sync
around. We might have a pending store, I'm not too sure how it may
be affected by the tlbia and if the isync is enough to sync it.
For now, I removed it, but I would like someone from IBM or moto
to speak up here.

That's all for the 603. However, for the hashtable based CPUs, I
noticed we don't have strict context sync. neither before doing
tlbie's in the hash table management code (flush_hash_page,
add_hash_page, and hash_page). The hash miss code path might not
be a problem as the exception itself should be context synchronizing,
but I'm worried about add_hash_page() and flush_hash_page(). The
former is supposed to only add a valid HPTE where none existed, and
whatever may get evicted doesn't really become invalid, so I don't
think there is much risk here. flush_hash_page() should be used only
on a mapping that is beeing teared down; so I don't expect much
pending stores to it neither. But still, I would like advice from
some expert here...

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: tlbia and PPC603

[Etsushi Kato]

Thank you very much Ben,

On Wed, Jun 19, 2002 at 05:10:37AM +0200,
Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:

> Ok, after more investigations it appears that:
>
>  - There is no tlbia on 603. Actually, tlbia is #defined in the
> kernel to be a loop of tlbie's
>
>  - According to the PEM, tlbie requires a context synchronizing
> instruction _before_, and a context sync. or a plain "sync"
> instruction _after_. We did have the later but lacked the former
> in our _tlbia function (though we did have it in our _tlbie
> implementation).
>
> So the proper fix, according to the spec, would be to just replace
> the sync we currently do before with an isync. I've updated my rsync
> with such a fix, can you confirm it works ?

Before your advice, I did add isync just after both of two sync.
Without isync, almost always kernel crashes at the point of mounting
root file system or doing ifconfig at booting time.  But even with
this modification, the random crash still sometimes occurs, but the
probability seems to be decreased.  I tested this kernel with doing
boot, login, and connect by ssh from other machine 10 times, it
dropped to xmon 8 times, and worked fine 2 times.  Even dropped to
xmon, the system boot fine and works after typing 'x' at the >mon.

Oops messages are like this.

Oops: kernel access of bad area, sig: 11
NIP: C01B0CC0 XER: 00000000 LR: C001D0AC SP: C85D9EC0 REGS: c85d9e10 TRAP: 0300    Not tainted
MSR: 00009032 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 11
DAR: 30017980, DSISR: 20000000
TASK = c85d8000[188] 'sysctl' Last syscall: 4
last math c85d8000 last altivec 00000000
GPR00: C01B0CBC C85D9EC0 C85D8000 C0239B6C 00000001 C8C979C0 30016000 C85D9EE8
GPR08: 00000000 00000000 C003B18C C85A3E00 4800288C 10019A5C 00000000 00000000
GPR16: 00000000 00000000 00000000 00000000 00009032 085D9F40 00000000 C0005ED4
GPR24: C0005C40 7FFFFBCC 00000000 30016000 00000002 C8C979C0 00000001 C0239B6C
Call backtrace:
C00287FC C001D078 C001D134 C003B254 C0005C9C 10000000 0FF1F6EC
0FF205C8 0FF1F044 0FF13C90 10000FC4 10000D48 10000978 0FEC8B90
00000000
Oops: Exception in kernel mode, sig: 4
NIP: C01B0CC4 XER: 00000000 LR: C001D0AC SP: C7FA9EC0 REGS: c7fa9e10 TRAP: 0700    Not tainted
MSR: 00089032 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 11
TASK = c7fa8000[509] 'S90firewall' Last syscall: 4
last math 00000000 last altivec 00000000
GPR00: C01B0CBC C7FA9EC0 C7FA8000 C0239B6C 00000001 C80495C0 100B1190 C7FA9EE8
GPR08: 00000000 00000000 C003B18C C85A3E00 48022880 100AD23C 00000000 00000000
GPR16: 100B5BF0 00000000 00000000 1005372C 00009032 07FA9F40 00000000 C0005ED4
GPR24: C0005C40 100A0000 00000001 100AF810 00000002 C80495C0 00000000 C0239B6C
Call backtrace:
0000000A C001D078 C001D134 C003B254 C0005C9C FFFFFFFF 0FED86EC
0FED8B74 0FEDAB70 0FED545C 100539BC 10022CD4 1002345C 10022994
1001F114 100206A4 1001F448 1001E998 10020628 1001F448 1001E998
10020628 1001F448 1001E998 10020628 1001F448 1001E998 10020628
1001F448 1001E998 10021984 1001F308 1001E998


I also tested your latest rsync tree.  With the same test, the kernel
also dropped to xmon 8 times, and worked stable 2 times.


Even the situation have improved with isync, I'm becoming not sure
about _tlbia is the problem.  Does anyone know other things causing
problem on PPC603 Macintosh??

I tested this on UMAX C500 (144M RAM, 240MHz 603ev, DEC21140 ether card).

Thank you,
--
Etsushi Kato
ekato@ees.hokudai.ac.jp

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: tlbia and PPC603

[Benjamin Herrenschmidt]

>I also tested your latest rsync tree.  With the same test, the kernel
>also dropped to xmon 8 times, and worked stable 2 times.
>
>
>Even the situation have improved with isync, I'm becoming not sure
>about _tlbia is the problem.  Does anyone know other things causing
>problem on PPC603 Macintosh??
>
>I tested this on UMAX C500 (144M RAM, 240MHz 603ev, DEC21140 ether card).

Try without the ethernet card and switch IDE DMA off and tell me if it is
stable, there could be some cache coherency issues with some of the chipsets
used on these machines.

Another thing you can try is in set_context(), add an isync on the line
that contain the label "3:" as the doc states a context synchronizing
instruction should also be used for data sync. before mtsrin.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: tlbia and PPC603

[Etsushi Kato]

Hi Ben,
Thank you for your reply.

On Wed, Jun 19, 2002 at 05:51:37AM +0200,
Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:

> Try without the ethernet card and switch IDE DMA off and tell me if it is
> stable, there could be some cache coherency issues with some of the chipsets
> used on these machines.

I was able to use both IDE DMA and ethernet card on
linux-2.4.13-pre3-ben0 without problems.
Anyway I tried rsync kernel without CONFIG_BLK_DEV_IDEDMA_PMAC_AUTO
and compiled tulip as modules (and not loaded), but it still dropped
to xmon randomly.

> Another thing you can try is in set_context(), add an isync on the line
> that contain the label "3:" as the doc states a context synchronizing
> instruction should also be used for data sync. before mtsrin.

I changed set_context() and  it might improve stableness somewhat, but
still dropped to xmon at some occasion.

> Hrm... just in case, try also changing the definition of tlbia macro
> in ppc_asm.h to put isync's inside the loop, both before and after the
> tlbie's (add too much is no problem for such a test) and tell me if it
> has any effect.

With this change combined with set_context() change, crashes
disappeared during boot time (tested 10 times and all trial
succeeded).  It is really improvement.  But still sometimes (5 times
per 10) dropped to xmon while sshd is working by ssh connection from
other machine.


Thanks,
--
Etsushi Kato
ekato@ees.hokudai.ac.jp

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: tlbia and PPC603

[Benjamin Herrenschmidt]

>Hi Ben,
>Thank you for your reply.
>
>On Wed, Jun 19, 2002 at 05:51:37AM +0200,
>Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:
>
>> Try without the ethernet card and switch IDE DMA off and tell me if it is
>> stable, there could be some cache coherency issues with some of the
chipsets
>> used on these machines.
>
>I was able to use both IDE DMA and ethernet card on
>linux-2.4.13-pre3-ben0 without problems.

Do you still have this kernel source ? My local repository got
corrupted at some point so I can't go that far in the history.

I would appreciate if you could send me a tarball with at least
the content of arch/ppc/kernel, arch/ppc/mm and include/asm-ppc
so I can do some comparisons.

>Anyway I tried rsync kernel without CONFIG_BLK_DEV_IDEDMA_PMAC_AUTO
>and compiled tulip as modules (and not loaded), but it still dropped
>to xmon randomly.

Ok.

>> Another thing you can try is in set_context(), add an isync on the line
>> that contain the label "3:" as the doc states a context synchronizing
>> instruction should also be used for data sync. before mtsrin.
>
>I changed set_context() and  it might improve stableness somewhat, but
>still dropped to xmon at some occasion.

Hrm...

>> Hrm... just in case, try also changing the definition of tlbia macro
>> in ppc_asm.h to put isync's inside the loop, both before and after the
>> tlbie's (add too much is no problem for such a test) and tell me if it
>> has any effect.
>
>With this change combined with set_context() change, crashes
>disappeared during boot time (tested 10 times and all trial
>succeeded).  It is really improvement.  But still sometimes (5 times
>per 10) dropped to xmon while sshd is working by ssh connection from
>other machine.

Can you send me a couple of xmon backtraces with lookup in System.map ?

Also, please try editing arch/ppc/kernel/ppc_asm.h and change the definition
of SYNC, SYNC_601 and ISYNC_601 so they unconditionally do the various sync &
isync's (that is remove the BEGIN_FTR_SECTION and END_FTR_SECTION(...) lines
from them.

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: tlbia and PPC603

[Gabriel Paubert]

On Wed, 19 Jun 2002, Benjamin Herrenschmidt wrote:

>
> Ok, after more investigations it appears that:
>
>  - There is no tlbia on 603. Actually, tlbia is #defined in the
> kernel to be a loop of tlbie's
>
>  - According to the PEM, tlbie requires a context synchronizing
> instruction _before_, and a context sync. or a plain "sync"
> instruction _after_. We did have the later but lacked the former
> in our _tlbia function (though we did have it in our _tlbie
> implementation).
>
> So the proper fix, according to the spec, would be to just replace
> the sync we currently do before with an isync. I've updated my rsync
> with such a fix, can you confirm it works ?

IMHO, this can't be the problem. A sync synchronizes the memory queues so
it should work with either.

>
> If it's stable this way, i'll push the fix to the bk trees and
> to Marcelo.
>
> I'm not completely sure if we should or shouldn't keep the sync
> around. We might have a pending store, I'm not too sure how it may
> be affected by the tlbia and if the isync is enough to sync it.
> For now, I removed it, but I would like someone from IBM or moto
> to speak up here.

If the processor is not buggy (sadly a big if on PPC it seems) an isync
will not allow the following instructions to be executed until after all
the previous instructions have got past the point where they can produce
exceptions, of course this includes TLB faults. After this point the store
is considered committed, the data may still be in a store queue along with
its physical address for cache reloads, etc..., but the only possible
remaining exception is a machine check.

> That's all for the 603. However, for the hashtable based CPUs, I
> noticed we don't have strict context sync. neither before doing
> tlbie's in the hash table management code (flush_hash_page,
> add_hash_page, and hash_page). The hash miss code path might not
> be a problem as the exception itself should be context synchronizing,
> but I'm worried about add_hash_page() and flush_hash_page(). The
> former is supposed to only add a valid HPTE where none existed, and
> whatever may get evicted doesn't really become invalid, so I don't
> think there is much risk here.

Do we actually do the tlbie if somebody munmaps the evicted page and we
find that the entry is not present in the hash table ?

If not, a tlbie on the preceding valid virtual address is necessary.
Rebuilding the virtual address from the previous first word of the hash
table is trivial.

And I'd certainly prefer to have the guarantee that the TLB are always
a strict subset of the hash table. I don't believe that trying to allow
such dangling TLB entries has any advantage.

	Gabriel.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: tlbia and PPC603

[Etsushi Kato]

Thank you for information, Gabriel and Ben.

I learned ppc603 software hash table some extent.  And I finally
noticed the bug in current arch/ppc/mm/ppc_mmc.c.  In old code, there
was blr at hash_page[0] and did flush_icache_range() from hash_page[0]
to hash_page[1].  But in current code, this code is removed when
comparing CPU_FT_HPTE_TABLE.

Adding this to current ppc_mmu.c have solved random crash both in
Nubus Performa and UMAX C500.  And isync before tlbie and mtsrin seems
to be not necessary.


--
Etsushi Kato
ekato@ees.hokudai.ac.jp

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: tlbia and PPC603

[Benjamin Herrenschmidt]

>I learned ppc603 software hash table some extent.  And I finally
>noticed the bug in current arch/ppc/mm/ppc_mmc.c.  In old code, there
>was blr at hash_page[0] and did flush_icache_range() from hash_page[0]
>to hash_page[1].  But in current code, this code is removed when
>comparing CPU_FT_HPTE_TABLE.
>
>Adding this to current ppc_mmu.c have solved random crash both in
>Nubus Performa and UMAX C500.  And isync before tlbie and mtsrin seems
>to be not necessary.

Good catch !

We'll fix it asap.

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: tlbia and PPC603

[Benjamin Herrenschmidt]

Your xmon trace show something interesting: xmon itself crashes with a
trap 300 while trying to access the hash table. The hash beeing mapped
with a BAT, that shouldn't be an issue with the TLB handlers nor with
a stale TLB entry. I currently fail to see what could be going on there,
could you try to reproduce that crash ? In this case, once "h" fails
within xmon, type

 di %pc
 r

and send me what it days.

Ben.

---

vector: 300 at pc = c0190888, lr = c01907fc
msr = 9032, sp = c7e49e10 [c7e49d60]
dar = 1180, dsisr = 20000000
current = c7e48000, pid = 579, comm = sshd
mon> t
backtrace:
c01907fc
c01aef10
c016c6b4
c016c7d8
c003b128
c0005cbc
exception:c00 [c7e49f50] fd98038
100040e0
10005cbc
0fce5b90
00000000
mon> S
msr = 1032, pvr = 71201
sprg0-3 = c0210000 1 0 7e48270
srr0 = c0010340, srr1 = 80011032
sr0-15 = 607bbb50 607bbc61 607bbd72 6
07bbe83 607bbf94 607bc0a5 607bc1b6 607bc2c7 20000888 20000999 20000aaa
20000bbb 20000ccc 20000ddd 20000eee 20000fff
sp = c7e49b80 toc = c7e480000
mon> h
Mappings for context 0
vector: 300 at pc = c00c2ef4, lr = c00c3114
msr = 1032, sp = c7e49b50 [c7e49aa0]
dar = 0, dsisr = 20000000
current = c7e48000, pid = 579, comm = sshd


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: tlbia and PPC603

[Etsushi Kato]

On Thu, Jun 20, 2002 at 09:53:55AM +0200,
Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:

> Your xmon trace show something interesting: xmon itself crashes with a
> trap 300 while trying to access the hash table. The hash beeing mapped
> with a BAT, that shouldn't be an issue with the TLB handlers nor with
> a stale TLB entry. I currently fail to see what could be going on there,
> could you try to reproduce that crash ? In this case, once "h" fails
> within xmon, type
>
>  di %pc

>mon di %pc
c00c2ef4   7d7c002e	lwzx	r11,r28,r0
c00c2ef8   7e6a4b78	or	r10,r19,r9
c00c2efc   7c0b5000	cmpw	r11,r10
c00c2f00   3b280001	addi	r25,r8,1
c00c2f04   3bc00000	li	r30,0
c00c2f08   7ffc0214	add	r31,r28,r0
c00c2f0c   39200000	li	r9,0
c00c2f10   3b5d1000	addi	r26,r29,4096
c00c2f14   4800001c	b	0xc00c2f30
c00c2f18   39290001	addi	r9,r9,1
c00c2f1c   2c090007	cmpwi	r9,7
c00c2f20   3bff0008	addi	r31,r31,8
c00c2f24   41810014	bgt	0xc00c2f38
c00c2f28   801f0000	lwz	r0,0(r31)
c00c2f2c   7c005000	cmpw	r0,r10
c00c2f30   4082ffe8	bne	0xc00c2f18

>  r

>mon r
R00 = 00000000	R01 = c796db50	R02 = c796c000	R03 = 00000000
R04 = 00000000	R05 = 0ffff000	R06 = 00000000	R07 = 00000000
R08 = 00000000	R09 = 80000000	R10 = 0f0f0f0f	R11 = 00000000
R12 = 84282424	R13 = 1005c3bc	R14 = c796dee8	R15 = 240004e3
R16 = c02d62c0	R17 = c796c000	R18 = c02b0000	R19 = 00000000
R20 = 00009032	R21 = 0000ffff	R22 = 00000000	R23 = 00000000
R24 = ffffffff	R25 = c0512bc0	R26 = c02a0000	R27 = 00000000
R28 = 00000000	R29 = 00000000	R30 = c02a0000	R31 = 00000000
pc  = c00c2ef4	msr = 00001032	lr  = c00c3114	cr  = 84282424
ctr = 00000000	xer = 20000000	trap =  300


--
Etsushi Kato
ekato@ees.hokudai.ac.jp

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: tlbia and PPC603

[Benjamin Herrenschmidt]

>> Your xmon trace show something interesting: xmon itself crashes with a
>> trap 300 while trying to access the hash table. The hash beeing mapped
>> with a BAT, that shouldn't be an issue with the TLB handlers nor with
>> a stale TLB entry. I currently fail to see what could be going on there,
>> could you try to reproduce that crash ? In this case, once "h" fails
>> within xmon, type
>>
>>  di %pc
>
>>mon di %pc

Hrm... I should think before replying !

It's completely normal that "h" crashes on a 603 as there is simply no
hash table on this CPU.

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/