2.6 on IP22 (Indy)

2.6 on IP22 (Indy)

[Markus Dahms]

Hi there,

I'm trying to run a current 2.6 kernel (LinuxMIPS CVS) on my
Indy(s). It should be a 64-bit kernel (just for fun, but I tried
32-bit, too). From the mailing list archives some time ago (about
half a year) I learned that there are known problems.

My experiments: Indy with R4600PC (133MHz) boots to userspace

| ...
| EXT3-fs: mounted filesystem with ordered data mode.
| atkbd.c: keyboard reset failed on hpc3ps2/serio0
| VFS: Mounted root (ext3 filesystem) readonly.
| Freeing unused kernel memory: 204k freed
| INIT: version 2.86 booting

and dies then :(. The same machine, but with a R4000PC (100MHz)
processor module doesn't even come so far:

| arcsboot: ARCS Linux ext2fs loader 0.3.8.6
|
| Loading 2.6.12-64 from scsi(0)disk(2)rdisk(0)partition(0)
| Allocated 0x38 bytes for segments
| Loading 64-bit executable
| Loading program segment 1 at 0x88004000, offset=0x0 4000, size = \
| 0x0 3c4086
| 3c0000      (cache: 95.3%)Zeroing memory at 0x883c8086, size = \
| 0x42f9a
| Starting ELF64 kernel

no more action at this point....

Are there chances to get the machine working with a current kernel
(2.4.x works fine :), is there work going on?

Thanks anyway,

        Markus

--
Yip yip yip yip yap yap yip *BANG* --- NO TERRIER

Re: 2.6 on IP22 (Indy)

[Maciej W. Rozycki]

On Mon, 27 Jun 2005, Markus Dahms wrote:

> I'm trying to run a current 2.6 kernel (LinuxMIPS CVS) on my
> Indy(s). It should be a 64-bit kernel (just for fun, but I tried
> 32-bit, too). From the mailing list archives some time ago (about
> half a year) I learned that there are known problems.
> 
> My experiments: Indy with R4600PC (133MHz) boots to userspace
> 
> | ...
> | EXT3-fs: mounted filesystem with ordered data mode.
> | atkbd.c: keyboard reset failed on hpc3ps2/serio0
> | VFS: Mounted root (ext3 filesystem) readonly.
> | Freeing unused kernel memory: 204k freed
> | INIT: version 2.86 booting
> 
> and dies then :(. The same machine, but with a R4000PC (100MHz)

 Hmm, it might be a problem with TLB handlers that have been changed to be 
built at the run time.  Perhaps the R4600 isn't handled right as a result.  
What's the CPU revision ID? -- it's printed right at the beginning.

> processor module doesn't even come so far:
> 
> | arcsboot: ARCS Linux ext2fs loader 0.3.8.6
> |
> | Loading 2.6.12-64 from scsi(0)disk(2)rdisk(0)partition(0)
> | Allocated 0x38 bytes for segments
> | Loading 64-bit executable
> | Loading program segment 1 at 0x88004000, offset=0x0 4000, size = \
> | 0x0 3c4086
> | 3c0000      (cache: 95.3%)Zeroing memory at 0x883c8086, size = \
> | 0x42f9a
> | Starting ELF64 kernel
> 
> no more action at this point....

 Strange.  The system should be capable enough for early printk() to be 
trivially implemented using firmware call-backs.  It would be more useful 
to get some feedback from it this way, otherwise it's asking for a crystal 
ball (mine is currently being serviced, if you recall)...  It's probably a 
BUG() or an Oops somewhere.  It might be related to the lack of an S-cache 
on this system -- there's been another report recently about it being a 
problem on a different machine -- try the patch sent there as a test.

  Maciej

Re: 2.6 on IP22 (Indy)

[Markus Dahms]

>> My experiments: Indy with R4600PC (133MHz) boots to userspace
>> | INIT: version 2.86 booting
>> and dies then :(. The same machine, but with a R4000PC (100MHz)
>  Hmm, it might be a problem with TLB handlers that have been changed to be 
> built at the run time.  Perhaps the R4600 isn't handled right as a result.  
> What's the CPU revision ID? -- it's printed right at the beginning.

| CPU revision is: 00002020
| FPU revision is: 00002020
| ...
| Synthesized TLB refill handler (30 instructions).
| Synthesized TLB load handler fastpath (43 instructions).
| Synthesized TLB store handler fastpath (43 instructions).
| Synthesized TLB modify handler fastpath (42 instructions).

the TLB stuff, if it's of interest...

| ...
| Calibrating system timer... warning: timer counts differ, retrying...\
| disagreement, using average... 44500 [89.0000 MHz CPU]
| Using 44.500 MHz high precision timer.

this is strange, too. It's a 133MHz CPU as kernel 2.4.x correctly
recognizes.

For the R4000 there are two other things I could try: console on newport
instead of serial port and a 32-bit kernel, which I only tried on the
R4600.
I'll also try the said patch (you're referring to "blast_scache nop ...", do
you?).

Markus

-- 
No RISC - No fun!

Re: 2.6 on IP22 (Indy)

[Maciej W. Rozycki]

On Mon, 27 Jun 2005, Markus Dahms wrote:

> >  Hmm, it might be a problem with TLB handlers that have been changed to be 
> > built at the run time.  Perhaps the R4600 isn't handled right as a result.  
> > What's the CPU revision ID? -- it's printed right at the beginning.
> 
> | CPU revision is: 00002020
> | FPU revision is: 00002020

 That's what I'm interested in -- the R4600 is fancy enough they've 
implemented different quirks with different revisions. ;-)

> | Synthesized TLB refill handler (30 instructions).
> | Synthesized TLB load handler fastpath (43 instructions).
> | Synthesized TLB store handler fastpath (43 instructions).
> | Synthesized TLB modify handler fastpath (42 instructions).
> 
> the TLB stuff, if it's of interest...

 I might ask about more detailed dumps of these, but for now I'll just 
check the sources for obvious problems.

> | ...
> | Calibrating system timer... warning: timer counts differ, retrying...\
> | disagreement, using average... 44500 [89.0000 MHz CPU]
> | Using 44.500 MHz high precision timer.
> 
> this is strange, too. It's a 133MHz CPU as kernel 2.4.x correctly
> recognizes.
> 
> For the R4000 there are two other things I could try: console on newport
> instead of serial port and a 32-bit kernel, which I only tried on the
> R4600.

 Well, I don't know what newport is, but if it's capable of providing 
output that early, it'll do.

> I'll also try the said patch (you're referring to "blast_scache nop ...", do
> you?).

 Precisely.

  Maciej

Re: 2.6 on IP22 (Indy)

[Markus Dahms]

Hello again,

>> For the R4000 there are two other things I could try: console on newport
>> instead of serial port and a 32-bit kernel, which I only tried on the
>> R4600.
>  Well, I don't know what newport is, but if it's capable of providing 
> output that early, it'll do.

Newport is the most common graphics option for the Indy and there exists
a console driver for it. Luckily it outputs some more stuff than the
serial console:

| CPU revision is: 00000430
| FPU revision is: 00000500
| ...
| Checking for the multiply/shift bug... yes, workaround... no.
| kernel panic - not syncing: Reliable operation impossible!
| Configure for R4000 to enable the workaround.

I configured the kernel for R4X00. There are a few references to
CONFIG_CPU_R4000 in the source which doesn't seem to be a config
option anymore, but I couldn't find a workaround somewhere...

>> I'll also try the said patch (you're referring to "blast_scache nop ...",
>> do you?).
> Precisely.

doesn't change anything, neither for R4000PC nor for R4600PC.

Markus

-- 
Reactor error - core dumped!

Re: 2.6 on IP22 (Indy)

[Maciej W. Rozycki]

On Tue, 28 Jun 2005, Markus Dahms wrote:

> | CPU revision is: 00000430
> | FPU revision is: 00000500
> | ...
> | Checking for the multiply/shift bug... yes, workaround... no.
> | kernel panic - not syncing: Reliable operation impossible!
> | Configure for R4000 to enable the workaround.
> 
> I configured the kernel for R4X00. There are a few references to
> CONFIG_CPU_R4000 in the source which doesn't seem to be a config
> option anymore, but I couldn't find a workaround somewhere...

 Well, yesterday evening I realized you mentioned the R4000 which has 
currently no way to run in the 64-bit mode with upstream Linux as there 
are grave errata in the CPU (in a couple of 64-bit instructions) that 
prevent reliable operation.  And I mean it -- with no workarounds enabled 
all I observed with my R4000 in my DECstation were random lock-ups, 
sometimes even before reaching userland (depending on stuff like cache 
alignment of some code with a given compilation).

 I implemented all the necessary workarounds and this includes Linux, GCC 
and binutils (further bits are needed for handcoded assembly programs if 
you want to run n32 or (n)64 userland; so far I've identified and fixed 
glibc and gmp).  If you'd like to use this system in the 64-bit mode you 
are most welcomed.  For this you can get toolchain bits from my site and I 
can supply you with a patch for Linux 2.4; I'll work on porting it to 2.6 
and actually applying upstream soon.

 As you may not be interested in binary RPM packages, you may just pull 
the necessary patches, which are all called "*-mips-nodaddi-*" and you 
have to make sure to rebuild all 64-bit binaries to be run on the R4000 
with either "-march=r4000" or "-mfix-r4000".  Unfortunately at the current 
state of affairs the GCC patch is not going to be accepted for inclusion 
upstream which means all the others have to live outside their respective 
official sources as well.

 I have successfully run 64-bit Linux on my R4000SC and early R4400SC 
which are both affected by these errata (but not exactly the same for both 
;-) ) for quite some time now.

 I have actually forgotten to mention you might indeed want to try a 
32-bit kernel as some low level bits differ and this is especially true 
given the above -- sorry about that.

> >> I'll also try the said patch (you're referring to "blast_scache nop ...",
> >> do you?).
> > Precisely.
> 
> doesn't change anything, neither for R4000PC nor for R4600PC.

 Well, the R4600 case was easy -- proofreading revealed the bug.  There is 
a cp0 hazard between updating a TLB entry and using it for an instruction 
fetch and for the R4600 it requires two instructions to clear.  
Unfortunately our handlers currently only execute a lone "eret" after TLB 
update instructions, which for TLB faults on instruction fetches triggers 
this hazard.  For data transfers there is no hazard in the R4600 and this 
is why your system has been able to progress through `init'; otherwise you 
would not see any output from it.

 Here's a patch.  I'm inclined to apply it as obviously correct but I'll 
resist for a while to let you provide some feedback.  The same hazard 
conditions are present for the R4700 and the R5000.  I've included the 
R5000A as well which, given the name, I've assumed was just a minor update 
to the R5000; anyone please correct me if I am wrong (but we don't ever 
use this CPU ID in Linux, so that would be for documentation only).

  Maciej

patch-mips-2.6.12-rc4-20050526-tlbex-r4600-0
diff -up --recursive --new-file linux-mips-2.6.12-rc4-20050526.macro/arch/mips/mm/tlbex.c linux-mips-2.6.12-rc4-20050526/arch/mips/mm/tlbex.c
--- linux-mips-2.6.12-rc4-20050526.macro/arch/mips/mm/tlbex.c	2005-04-29 04:56:05.000000000 +0000
+++ linux-mips-2.6.12-rc4-20050526/arch/mips/mm/tlbex.c	2005-06-28 01:14:39.000000000 +0000
@@ -828,11 +828,16 @@ static __init void build_tlb_write_entry
 		i_nop(p);
 		break;
 
-	case CPU_R4300:
 	case CPU_R4600:
 	case CPU_R4700:
 	case CPU_R5000:
 	case CPU_R5000A:
+		i_nop(p);
+		tlbw(p);
+		i_nop(p);
+		break;
+
+	case CPU_R4300:
 	case CPU_5KC:
 	case CPU_TX49XX:
 	case CPU_AU1000:

Re: 2.6 on IP22 (Indy)

[Markus Dahms]

Hello Maciej,

[R4000 in 64-bit mode]

I propably won't find the time to build a patched toolchain for R4000,
so my 64-bit experiments will concentrate on the R4600.

The R4000 now successfully boots to prompt using a 32-bit kernel. I'll
try to enable "Support for 64-bit physical address space" in the next
kernel build ;).

[R4600 tlbex.c patch]

This doesn't seem to be enough. The patch applies almost cleanly on
current CVS (offset -1 line), but the resulting kernel (I tried 64
and 32-bit) still stops after "INIT: ...".

Markus

Re: 2.6 on IP22 (Indy)

[Maciej W. Rozycki]

Hello Markus,

> [R4000 in 64-bit mode]
> 
> I propably won't find the time to build a patched toolchain for R4000,
> so my 64-bit experiments will concentrate on the R4600.

 Well, you can still use my readily available binaries. :-)  This should 
be quite straightforward for the Linux kernel itself and you can keep 
running o32 userland for the time being.  You may need to find a way to 
pass "-meb" to GCC (and perhaps "-EB" to ld and gas as they may be invoked 
explicitly for some reason) to ask it for big-endian code though.  Or you 
may ask someone to build a patched toolchain. ;-)  There is no change in 
code generation for non-affected configurations as a result of these 
patches, so the resulting toolchain is not crippled in any way.

 This won't solve the lack of necessary support in Linux 2.6, though... 

> The R4000 now successfully boots to prompt using a 32-bit kernel. I'll
> try to enable "Support for 64-bit physical address space" in the next
> kernel build ;).

 I guess nobody will bother if it turns out non-working. ;-)

> [R4600 tlbex.c patch]
> 
> This doesn't seem to be enough. The patch applies almost cleanly on
> current CVS (offset -1 line), but the resulting kernel (I tried 64
> and 32-bit) still stops after "INIT: ...".

 Well, there can be something else.  But to be sure I haven't missed 
anything in these TLB handlers, could you please generate the dumps I 
mentioned yesterday and send them to me?  You need to uncomment the 
definition of DEBUG_TLB at the top of arch/mips/mm/tlbex.c for that.

  Maciej

Re: 2.6 on IP22 (Indy)

[Stephen P. Becker]

> [R4600 tlbex.c patch]
> 
> This doesn't seem to be enough. The patch applies almost cleanly on
> current CVS (offset -1 line), but the resulting kernel (I tried 64
> and 32-bit) still stops after "INIT: ...".
> 

This might be a good time to mention that cvs HEAD seems to have serious
problems with 64-bit ip22 kernels (at least on r5000 systems...I don't
have a working r4x00 box to test with), and it may be contributing to
your problems.  Basically, numerous messages about vmalloc dump to the
kernel log while doing a number of things disk related:

allocation failed: out of vmalloc space - use vmalloc=<size> to increase
size.

Symptons include ricerfs partitions not being able to mount (ext3 seems
to work, but still triggers the vmalloc error message), swapon failing
to turn the swap partition on, dd having major problems when dumping
large blocks, etc.  The initial probing of the scsi controller at boot
time also triggers the message.

I haven't tried a 32-bit kernel since 2.6.12_rc2 or so, however when I
did, everything seemed to work just fine, while a 64-bit kernel compiled
from the same checkout had the same problems I mention above.  At one
point I was trying to work out when this was introduced, and I *think*
it was the initial 2.6.11 merge, but I would have to double check that.
 I've been running a 64-bit 2.6.10 checkout since the beginning of the
year on that box, and the machine has been very stable.

-Steve

Re: 2.6 on IP22 (Indy)

[Maciej W. Rozycki]

On Tue, 28 Jun 2005, Stephen P. Becker wrote:

> I haven't tried a 32-bit kernel since 2.6.12_rc2 or so, however when I
> did, everything seemed to work just fine, while a 64-bit kernel compiled
> from the same checkout had the same problems I mention above.  At one
> point I was trying to work out when this was introduced, and I *think*
> it was the initial 2.6.11 merge, but I would have to double check that.
>  I've been running a 64-bit 2.6.10 checkout since the beginning of the
> year on that box, and the machine has been very stable.

 It's either not 2.6.11 or it must be something platform-specific (here 
meaning the specific CPU model and/or system type) as 2.6.12-rc1 as of the 
end of March seems to be rock-solid running 64-bit on the BCM1250.  Or 
perhaps SMP works and only UP does not. ;-)

  Maciej

Re: 2.6 on IP22 (Indy)

[Stephen P. Becker]

> It's either not 2.6.11 or it must be something platform-specific (here 
> meaning the specific CPU model and/or system type) as 2.6.12-rc1 as of the 
> end of March seems to be rock-solid running 64-bit on the BCM1250.  Or 
> perhaps SMP works and only UP does not. ;-)
> 

Yeah, it is ip22 specific.  I have been running 64-bit ip32 kernels
(R5000) from the same checkouts with no problems.

-Steve

Re: 2.6 on IP22 (Indy)

[Markus Dahms]

Hello Maciej,

>> [R4600 tlbex.c patch]
>> This doesn't seem to be enough.
> Well, there can be something else.  But to be sure I haven't missed 
> anything in these TLB handlers, could you please generate the dumps I 
> mentioned yesterday and send them to me?  You need to uncomment the 
> definition of DEBUG_TLB at the top of arch/mips/mm/tlbex.c for that.

I think I found the trick. Just following the hint that the R4600
is similar to the R5000 I added CPU_R4600 to build_tlb_probe_entry().

After that I got a prompt :-). Everything worked fine so far...

The next problem appeared on shutdown, where I got a kernel panic
(as the R4000 does with the 32-bit kernel, too) - something related
too the UART/serial console I suspect. As it's not on serial output
I need to find a way to get it without pencil and paper ;).

If you have further things to try, just send patches :)

Markus

Re: 2.6 on IP22 (Indy)

[Maciej W. Rozycki]

Hello Markus,

> I think I found the trick. Just following the hint that the R4600
> is similar to the R5000 I added CPU_R4600 to build_tlb_probe_entry().

 Well, they are not meant to be errata-compatible. ;-)  I haven't been 
able to locate any reference for tlbp being problematic on the R4600, in 
particular not in the chip errata document, and the old handlers used to 
have a nop before that instruction unconditionally (perhaps just in case 
;-) ), so the problem was covered.  If it fixes the problem for you, then 
it should probably be applied, too.  Not sure about the R4700 -- it fixed 
a few problems that were there in the R4600 as a side effect, so perhaps 
we should wait for some report first.

  Maciej

Re: 2.6 on IP22 (Indy)

[Markus Dahms]

Hello Maciej,

[R4600PC problems]
> Well, they are not meant to be errata-compatible. ;-)  I haven't been 
> able to locate any reference for tlbp being problematic on the R4600, in 
> particular not in the chip errata document, and the old handlers used to 
> have a nop before that instruction unconditionally (perhaps just in case 
> ;-) ), so the problem was covered.  If it fixes the problem for you, then 
> it should probably be applied, too.

I just built a 64-bit kernel from clean CVS and had to apply the patch
below again to get it to userspace. Maybe it just hides the real error
but at least "it works for me" [tm].
Please apply to CVS if there are no objections.

Markus

--- a/arch/mips/mm/tlbex.c    2005-07-19 20:12:32.000000000 +0200
+++ b/arch/mips/mm/tlbex.c    2005-07-19 20:10:29.000000000 +0200
@@ -779,6 +779,7 @@
 static __init void __attribute__((unused)) build_tlb_probe_entry(u32 **p)
 {
    switch (current_cpu_data.cputype) {
+   case CPU_R4600:
    case CPU_R5000:
    case CPU_R5000A:
    case CPU_NEVADA:

Re: 2.6 on IP22 (Indy)

[Thiemo Seufer]

Markus Dahms wrote:
> Hello Maciej,
> 
> [R4600PC problems]
> > Well, they are not meant to be errata-compatible. ;-)  I haven't been 
> > able to locate any reference for tlbp being problematic on the R4600, in 
> > particular not in the chip errata document, and the old handlers used to 
> > have a nop before that instruction unconditionally (perhaps just in case 
> > ;-) ), so the problem was covered.  If it fixes the problem for you, then 
> > it should probably be applied, too.
> 
> I just built a 64-bit kernel from clean CVS and had to apply the patch
> below again to get it to userspace. Maybe it just hides the real error
> but at least "it works for me" [tm].
> Please apply to CVS if there are no objections.
> 
> Markus
> 
> --- a/arch/mips/mm/tlbex.c    2005-07-19 20:12:32.000000000 +0200
> +++ b/arch/mips/mm/tlbex.c    2005-07-19 20:10:29.000000000 +0200
> @@ -779,6 +779,7 @@
>  static __init void __attribute__((unused)) build_tlb_probe_entry(u32 **p)
>  {
>     switch (current_cpu_data.cputype) {
> +   case CPU_R4600:
>     case CPU_R5000:
>     case CPU_R5000A:
>     case CPU_NEVADA:

FWIW, this patch makes no difference for my Indy with R4600 v2.0, it still
hangs, usually while or shortly after mounting filesystems.


Thiemo