MIPS - 64bit woes

MIPS - 64bit woes

[Jim Gifford]

I've been working on getting the RaQ2 to work with the current 2.6.14 
kernel, but no luck at all. The last success I had was with 2.6.12.x 
series.

I'm looking for ideas, patches or whatever to get this working again. I 
know it has to do with the kernel using 32bit addressing instead of 64 
bit, but have no clue where to start.

Here is what I get when I attempt to boot it.

 > nfs 172.16.0.1 /nfsroot boot/vmlinux-2.6.14.gz
nfs: mounted "/nfsroot"
nfs: lookup "boot"
nfs: lookup "vmlinux-2.6.14.gz"
nfs: mode <0100755>
1349KB loaded (899KB/sec)
001512e8 1381096t
nfs: unmounted "/nfsroot"
 > execute root=/dev/nfs nfsroot=172.16.0.1:/nfsroot 
console=ttyS0,115200 ip=dhcp
elf64: 00080000 - 003b901f (ffffffff.80357000) (ffffffff.80000000)
elf64: ffffffff.80080000 (80080000) 3170438t + 208794t
net: interface down

Thanx for all your assistance.

-- 
----
Jim Gifford
maillist@jg555.com

Re: MIPS - 64bit woes

[Kumba]

Jim Gifford wrote:
> I've been working on getting the RaQ2 to work with the current 2.6.14 
> kernel, but no luck at all. The last success I had was with 2.6.12.x 
> series.
> 
> I'm looking for ideas, patches or whatever to get this working again. I 
> know it has to do with the kernel using 32bit addressing instead of 64 
> bit, but have no clue where to start.
> 
> Here is what I get when I attempt to boot it.
> 
>  > nfs 172.16.0.1 /nfsroot boot/vmlinux-2.6.14.gz
> nfs: mounted "/nfsroot"
> nfs: lookup "boot"
> nfs: lookup "vmlinux-2.6.14.gz"
> nfs: mode <0100755>
> 1349KB loaded (899KB/sec)
> 001512e8 1381096t
> nfs: unmounted "/nfsroot"
>  > execute root=/dev/nfs nfsroot=172.16.0.1:/nfsroot 
> console=ttyS0,115200 ip=dhcp
> elf64: 00080000 - 003b901f (ffffffff.80357000) (ffffffff.80000000)
> elf64: ffffffff.80080000 (80080000) 3170438t + 208794t
> net: interface down
> 
> Thanx for all your assistance.

Hmm, I'm unable to boot a 64bit kernel on either IP27 or IP30 here.  Appears to 
die very early in the kernel code.  It looks like your kernel also dies before 
doing anything meaningful, so I wonder if this is some common thing.


--Kumba

-- 
Gentoo/MIPS Team Lead
Gentoo Foundation Board of Trustees

"Such is oft the course of deeds that move the wheels of the world: small hands 
do them because they must, while the eyes of the great are elsewhere."  --Elrond

Re: MIPS - 64bit woes

[Jim Gifford]

I've talked to a few others, who are having similar issues also Kumba, I 
made a diff of 2.6.12 and 2.6.14, trying to figure out what's causing 
this. Looks like some major rewrites have occured in some areas.

If anyone would like to view the diff I have it on my website, 
http://ftp.jg555.com/mips_diff.txt

-- 
----
Jim Gifford
maillist@jg555.com

Re: MIPS - 64bit woes

[Stuart Anderson]

On Mon, 7 Nov 2005, Jim Gifford wrote:

> I've talked to a few others, who are having similar issues also Kumba, I made 
> a diff of 2.6.12 and 2.6.14, trying to figure out what's causing this. Looks 
> like some major rewrites have occured in some areas.

I have a 2.6.14-rc1 kernel that works great, and when I tried
2.6.14-rc2, it would have up on boot. In my case, I'm using NFS root,
and the networking would just go away when the system got busy. This
would cause the nfsroot to hang, and then any process that touched a file
would hang. It sort of felt like interrupts stopped working, but I was
not able to verify that. It seems that a serial console continued to
work, so it doesn't seem like all interrupts stopped working.

I have backported nearly all of the relevent arch/mips files to my
2.6.14-rc1 tree, and everything is still happy, so I'm inclined to think
it might not be something inside arch/mips.

I've stared at the diffs between rc1 and rc2 until I'm corss-eyed, but nothing
has jumped out at me as being obviously broken.

                                 Stuart

Stuart R. Anderson                               anderson@netsweng.com
Network & Software Engineering                   http://www.netsweng.com/
1024D/37A79149:                                  0791 D3B8 9A4C 2CDC A31F
                                                  BD03 0A62 E534 37A7 9149

Re: MIPS - 64bit woes

[Maciej W. Rozycki]

On Sat, 5 Nov 2005, Jim Gifford wrote:

> I've been working on getting the RaQ2 to work with the current 2.6.14 
> kernel, but no luck at all. The last success I had was with 2.6.12.x 
> series.
> 
> I'm looking for ideas, patches or whatever to get this working again. I 
> know it has to do with the kernel using 32bit addressing instead of 64 
> bit, but have no clue where to start.

 It must be platform-specific -- I haven't checked 2.6.14, but 64-bit
2.6.13 is good enough to boot into multi-user with the SWARM.

  Maciej

Re: MIPS - 64bit woes

[Kumba]

Maciej W. Rozycki wrote:
> 
>  It must be platform-specific -- I haven't checked 2.6.14, but 64-bit
> 2.6.13 is good enough to boot into multi-user with the SWARM.

2.6.13.4 works on all systems I can test (O2, Octane, Origin*, Indy, I2 Impact), 
but for me, it appears 2.6.14 dies very early in kernel initialization on both 
Octane and Origin.  I have yet to try another, like O2, but I'm going to do that 
shortly in the hopes O2 may initialize an output device and give me an idea of 
what's going on.

* 2.6.13.4 Works on Origin, but can't allocate PCI resources for the scsi 
properly.  2.6.12.5 is what works best for this system.


--Kumba

-- 
Gentoo/MIPS Team Lead
Gentoo Foundation Board of Trustees

"Such is oft the course of deeds that move the wheels of the world: small hands 
do them because they must, while the eyes of the great are elsewhere."  --Elrond

Re: MIPS - 64bit woes

[Jim Gifford]

Here is what I did to track down the errors.

I created a diff between the last working kernel 2.6.12 and the current 
kernel. Started with cpu-probe.c, that to me seemed the logical choice.

After patching the rest of files needed to support the patch in 
cpu-probe.c, I was finally able to produce a kernel under 2.6.12 with 
the same problem.

The files that I ported from 2.6.14 to 2.6.12 are the following
cpu-probe.c
cpu.h
mipsregs.h
cache.c
cpu-features.h

Here is a link to the patches I used http://ftp.jg555.com/errors

Looking at mipsregs.h, something doesn't look right for a 64 bit system. 
But I'm not the expert.

Here are my findings, I hope someone out there has an idea.

-- 
----
Jim Gifford
maillist@jg555.com

Re: MIPS - 64bit woes

[Kumba]

Jim Gifford wrote:
> Here is what I did to track down the errors.
> 
> I created a diff between the last working kernel 2.6.12 and the current 
> kernel. Started with cpu-probe.c, that to me seemed the logical choice.
> 
> After patching the rest of files needed to support the patch in 
> cpu-probe.c, I was finally able to produce a kernel under 2.6.12 with 
> the same problem.
> 
> The files that I ported from 2.6.14 to 2.6.12 are the following
> cpu-probe.c
> cpu.h
> mipsregs.h
> cache.c
> cpu-features.h
> 
> Here is a link to the patches I used http://ftp.jg555.com/errors
> 
> Looking at mipsregs.h, something doesn't look right for a 64 bit system. 
> But I'm not the expert.
> 
> Here are my findings, I hope someone out there has an idea.

Stanislaw pretty much figured it out last night.

See the if statement in the 'void __init cpu_cache_init(void)' function in 
arch/mips/mm/cache.c.  IP30, IP27, and cobalt didn't define the appropriate 
cache type in cpu-features.h, thus failed that check and panicked.  I believe 
it's fixed in the newest IP30 patches, of which I have yet to take a look at.


--Kumba

-- 
Gentoo/MIPS Team Lead
Gentoo Foundation Board of Trustees

"Such is oft the course of deeds that move the wheels of the world: small hands 
do them because they must, while the eyes of the great are elsewhere."  --Elrond

Re: MIPS - 64bit woes

[Jim Gifford]

Does look like it. I just checked the latest

-- 
----
Jim Gifford
maillist@jg555.com

Re: MIPS - 64bit woes

[Jim Gifford]

I've isolated the problem to 2.6.13-rc3. What I've done is built every 
kernel from 2.6.13-rc1 till it failed.

Also, Ralf, when I tried using git to pull those out of the git repo, 
they were missing files, had to use the cvs, not sure if you can verify 
it. I tried 2.6.13-rc1 and 2.6.13-rc2.

-- 
----
Jim Gifford
maillist@jg555.com

Re: MIPS - 64bit woes

[Jim Gifford]

Ok,
    Isolated the problem down to one file, will see if this patch will 
fix the issue with 2.6.14, the patch is the a diff of a file from 
2.6.13-rc2 and 2.6.13-rc3 which introduced the issue. Here is a link to 
the patch http://ftp.jg555.com/cobalt/culprit

Will test 2.6.14 with this patch a report back.

diff -Naur linux-mips-2.6.13-rc3/arch/mips/kernel/cpu-probe.c 
testbed/arch/mips/kernel/cpu-probe.c
--- linux-mips-2.6.13-rc3/arch/mips/kernel/cpu-probe.c    2005-07-27 
14:48:12.000000000 -0700
+++ testbed/arch/mips/kernel/cpu-probe.c    2005-11-17 
14:18:56.000000000 -0800
@@ -71,27 +71,11 @@
         : : "r" (au1k_wait));
 }
 
-static int __initdata nowait = 0;
-
-int __init wait_disable(char *s)
-{
-    nowait = 1;
-
-    return 1;
-}
-
-__setup("nowait", wait_disable);
-
 static inline void check_wait(void)
 {
     struct cpuinfo_mips *c = &current_cpu_data;
 
     printk("Checking for 'wait' instruction... ");
-    if (nowait) {
-        printk (" disabled.\n");
-        return;
-    }
-
     switch (c->cputype) {
     case CPU_R3081:
     case CPU_R3081E:
@@ -121,7 +105,6 @@
     case CPU_24K:
     case CPU_25KF:
     case CPU_34K:
-     case CPU_PR4450:
         cpu_wait = r4k_wait;
         printk(" available.\n");
         break;
@@ -147,6 +130,58 @@
     check_wait();
 }
 
+#ifdef CONFIG_64BIT
+
+/*
+ * On RM5230/5231 all accesses to XKPHYS by LL(D) are forced
+ * to be uncached, bits 61-59 of the address are ignored.
+ *
+ * Apparently fixed on RM5230A/5231A.
+ */
+static inline int check_lld(void)
+{
+    unsigned long flags, value, match, phys, *addr;
+
+    printk("Checking for lld bug... ");
+
+    /* hope the stack is in the low 512MB */
+    phys = CPHYSADDR((unsigned long) &value);
+
+    /* write value to memory */
+    value = 0xfedcba9876543210;
+    addr = (unsigned long *) PHYS_TO_XKPHYS(K_CALG_UNCACHED, phys);
+    *addr = value;
+
+    /* stop spurious flushes */
+    local_irq_save(flags);
+
+    /* flip cached value */
+    value = ~value;
+
+    /* read value, supposedly from cache */
+    addr = (unsigned long *) PHYS_TO_XKPHYS(K_CALG_NONCOHERENT, phys);
+    asm volatile("lld %0, %1" : "=r" (match) : "m" (*addr));
+
+    local_irq_restore(flags);
+
+    match ^= value;
+
+    switch ((long) match) {
+    case 0:
+        printk("no.\n");
+        break;
+    case -1:
+        printk("yes.\n");
+        break;
+    default:
+        printk("yikes yes! (%lx/%lx@%p)\nPlease report to 
<linux-mips@linux-mips.org>.", value, match, &value);
+    }
+
+    return !match;
+}
+
+#endif
+
 /*
  * Probe whether cpu has config register by trying to play with
  * alternate cache bit and see whether it matters.
@@ -283,8 +318,7 @@
     case PRID_IMP_R4600:
         c->cputype = CPU_R4600;
         c->isa_level = MIPS_CPU_ISA_III;
-        c->options = R4K_OPTS | MIPS_CPU_FPU | MIPS_CPU_32FPR |
-                 MIPS_CPU_LLSC;
+        c->options = R4K_OPTS | MIPS_CPU_FPU | MIPS_CPU_LLSC;
         c->tlbsize = 48;
         break;
     #if 0
@@ -364,7 +398,11 @@
         c->cputype = CPU_NEVADA;
         c->isa_level = MIPS_CPU_ISA_IV;
         c->options = R4K_OPTS | MIPS_CPU_FPU | MIPS_CPU_32FPR |
-                     MIPS_CPU_DIVEC | MIPS_CPU_LLSC;
+                     MIPS_CPU_DIVEC;
+#ifdef CONFIG_64BIT
+        if (check_lld())
+#endif
+            c->options |= MIPS_CPU_LLSC;
         c->tlbsize = 48;
         break;
     case PRID_IMP_R6000:
@@ -509,12 +547,6 @@
         c->ases |= MIPS_ASE_SMARTMIPS;
     if (config3 & MIPS_CONF3_DSP)
         c->ases |= MIPS_ASE_DSP;
-    if (config3 & MIPS_CONF3_VINT)
-        c->ases |= MIPS_CPU_VINT;
-    if (config3 & MIPS_CONF3_VEIC)
-        c->ases |= MIPS_CPU_VEIC;
-    if (config3 & MIPS_CONF3_MT)
-                c->ases |= MIPS_CPU_MIPSMT;
 
     return config3 & MIPS_CONF_M;
 }
@@ -632,21 +664,6 @@
     }
 }
 
-static inline void cpu_probe_philips(struct cpuinfo_mips *c)
-{
-    decode_configs(c);
-    switch (c->processor_id & 0xff00) {
-    case PRID_IMP_PR4450:
-        c->cputype = CPU_PR4450;
-        c->isa_level = MIPS_CPU_ISA_M32;
-        break;
-    default:
-        panic("Unknown Philips Core!"); /* REVISIT: die? */
-        break;
-    }
-}
-
-
 __init void cpu_probe(void)
 {
     struct cpuinfo_mips *c = &current_cpu_data;
@@ -672,9 +689,6 @@
     case PRID_COMP_SANDCRAFT:
         cpu_probe_sandcraft(c);
         break;
-     case PRID_COMP_PHILIPS:
-        cpu_probe_philips(c);
-         break;
     default:
         c->cputype = CPU_UNKNOWN;
     }

-- 
----
Jim Gifford
maillist@jg555.com

Re: MIPS - 64bit woes

[Jim Gifford]

Got a fix for 2.6.14, http://ftp.jg555.com/cobalt/fix-2.6.14.

Ralf, I know this is probably not the fix you would like to see, any 
suggestions.

diff -Naur linux-mips-2.6.14.orig/arch/mips/kernel/cpu-probe.c 
linux-mips-2.6.14/arch/mips/kernel/cpu-probe.c
--- linux-mips-2.6.14.orig/arch/mips/kernel/cpu-probe.c    2005-11-17 
11:42:19.000000000 -0800
+++ linux-mips-2.6.14/arch/mips/kernel/cpu-probe.c    2005-11-17 
15:00:11.000000000 -0800
@@ -121,7 +105,6 @@
     case CPU_24K:
     case CPU_25KF:
     case CPU_34K:
-     case CPU_PR4450:
         cpu_wait = r4k_wait;
         printk(" available.\n");
         break;
@@ -147,6 +130,58 @@
     check_wait();
 }
 
+#ifdef CONFIG_64BIT
+
+/*
+ * On RM5230/5231 all accesses to XKPHYS by LL(D) are forced
+ * to be uncached, bits 61-59 of the address are ignored.
+ *
+ * Apparently fixed on RM5230A/5231A.
+ */
+static inline int check_lld(void)
+{
+    unsigned long flags, value, match, phys, *addr;
+
+    printk("Checking for lld bug... ");
+
+    /* hope the stack is in the low 512MB */
+    phys = CPHYSADDR((unsigned long) &value);
+
+    /* write value to memory */
+    value = 0xfedcba9876543210;
+    addr = (unsigned long *) PHYS_TO_XKPHYS(K_CALG_UNCACHED, phys);
+    *addr = value;
+
+    /* stop spurious flushes */
+    local_irq_save(flags);
+
+    /* flip cached value */
+    value = ~value;
+
+    /* read value, supposedly from cache */
+    addr = (unsigned long *) PHYS_TO_XKPHYS(K_CALG_NONCOHERENT, phys);
+    asm volatile("lld %0, %1" : "=r" (match) : "m" (*addr));
+
+    local_irq_restore(flags);
+
+    match ^= value;
+
+    switch ((long) match) {
+    case 0:
+        printk("no.\n");
+        break;
+    case -1:
+        printk("yes.\n");
+        break;
+    default:
+        printk("yikes yes! (%lx/%lx@%p)\nPlease report to 
<linux-mips@linux-mips.org>.", value, match, &value);
+    }
+
+    return !match;
+}
+
+#endif
+
 /*
  * Probe whether cpu has config register by trying to play with
  * alternate cache bit and see whether it matters.
@@ -285,8 +320,7 @@
     case PRID_IMP_R4600:
         c->cputype = CPU_R4600;
         c->isa_level = MIPS_CPU_ISA_III;
-        c->options = R4K_OPTS | MIPS_CPU_FPU | MIPS_CPU_32FPR |
-                 MIPS_CPU_LLSC;
+        c->options = R4K_OPTS | MIPS_CPU_FPU | MIPS_CPU_LLSC;
         c->tlbsize = 48;
         break;
     #if 0
@@ -366,7 +400,11 @@
         c->cputype = CPU_NEVADA;
         c->isa_level = MIPS_CPU_ISA_IV;
         c->options = R4K_OPTS | MIPS_CPU_FPU | MIPS_CPU_32FPR |
-                     MIPS_CPU_DIVEC | MIPS_CPU_LLSC;
+                     MIPS_CPU_DIVEC;
+#ifdef CONFIG_64BIT
+        if (check_lld())
+#endif
+            c->options |= MIPS_CPU_LLSC;
         c->tlbsize = 48;
         break;
     case PRID_IMP_R6000:

-- 



-- 
----
Jim Gifford
maillist@jg555.com

Re: MIPS - 64bit woes

[Ralf Baechle]

On Thu, Nov 17, 2005 at 05:21:27PM -0800, Jim Gifford wrote:

> Got a fix for 2.6.14, http://ftp.jg555.com/cobalt/fix-2.6.14.
> 
> Ralf, I know this is probably not the fix you would like to see, any 
> suggestions.
> 
> diff -Naur linux-mips-2.6.14.orig/arch/mips/kernel/cpu-probe.c 
> linux-mips-2.6.14/arch/mips/kernel/cpu-probe.c
> --- linux-mips-2.6.14.orig/arch/mips/kernel/cpu-probe.c    2005-11-17 
> 11:42:19.000000000 -0800
> +++ linux-mips-2.6.14/arch/mips/kernel/cpu-probe.c    2005-11-17 
> 15:00:11.000000000 -0800
> @@ -121,7 +105,6 @@
>     case CPU_24K:
>     case CPU_25KF:
>     case CPU_34K:
> -     case CPU_PR4450:
>         cpu_wait = r4k_wait;
>         printk(" available.\n");
>         break;

Ehhh?

As for the remainder of your patch - the fact that this actually works
made me notice that there is no cpu-features-override.h for Cobalt which
means that Cobalt kernels carry a rather heavyweight generic cachecode
including spinlocks and all sorts of atomic operations that will at
runtime select between ll/sc and non-ll/sc variants.  That's slow and
I would rather suggest you get rid of that overhead by simply
pretending not to have ll/sc at all on Cobalt but putting something like

#ifdef CONFIG_64BIT
#define cpu_has_llsc            0
#else
#define cpu_has_llsc            1
#endif

into the Cobalt cpu-features-override.h.

  Ralf

Re: MIPS - 64bit woes

[Jim Gifford]

I'll give it a shot and send back results and a updated patch.

One more question, what is the future off the iomap.c file, I didn't 
include it in my patch. Could it be simply be added to arch/mips/cobalt? 
I can only speak for the RaQ2, but is it required for any of the other 
MIPS based machines?

-- 
----
Jim Gifford
maillist@jg555.com

Re: MIPS - 64bit woes

[Ralf Baechle]

On Fri, Nov 18, 2005 at 09:24:56AM -0800, Jim Gifford wrote:

> I'll give it a shot and send back results and a updated patch.
> 
> One more question, what is the future off the iomap.c file, I didn't 
> include it in my patch. Could it be simply be added to arch/mips/cobalt? 
> I can only speak for the RaQ2, but is it required for any of the other 
> MIPS based machines?

No.  It's broken for machines with multiple PCI busses and I've explicitly
rejected the patch which is in kernel.org before.

  Ralf

mips iomap.c (Was: Re: MIPS - 64bit woes)

[Atsushi Nemoto]

>>>>> On Fri, 18 Nov 2005 17:29:48 +0000, Ralf Baechle <ralf@linux-mips.org> said:

ralf> No.  It's broken for machines with multiple PCI busses and I've
ralf> explicitly rejected the patch which is in kernel.org before.

Could you explain a bit _how_ broken current kernel.org's
arch/mips/lib/iomap.c ?  Is it a single mips_io_port_base issue?

I suppose it works as well as traditional way (request_region +
in[bwl] for IO resource, request_mem_region + iomap + read[bwl] for
MEM resource).

I think it is better than generic iomap.c (except that
ioremap_cacheable_cow which is not available for R3000 is used).

---
Atsushi Nemoto