On 11/03/2014 20:43, David Daney wrote: > On 11/03/2014 05:34 PM, Joshua Kinard wrote: >> On 11/03/2014 20:23, David Daney wrote: >>> On 11/03/2014 05:08 PM, Joshua Kinard wrote: >>>> On 11/03/2014 13:52, David Daney wrote: >>>>> On 11/02/2014 02:53 AM, Joshua Kinard wrote: >>>>>> >>>>>> So I have been testing the Onyx2 I have out the last few days with the IOC3 >>>>>> metadriver used on Octane, and I can get it to boot, but if >>>>>> CONFIG_TRANSPARENT_HUGEPAGE is enabled in the kernel, bus errors can happen. >>>>>> >>>>>> If I use CONFIG_PAGE_SIZE_4KB, I get bus errors rather frequently -- running >>>>>> Gentoo's 'emerge' command can produce one. Switch to >>>>>> CONFIG_PAGE_SIZE_16KB, >>>>>> and the bus errors are far less frequent. I suspect CONFIG_PAGE_SIZE_64KB >>>>>> will >>>>>> be even less. >>>>>> >>>>>> Disable CONFIG_TRANSPARENT_HUGEPAGE, and the machine works pretty good. >>>>>> It's >>>>>> been up for almost 8 hours compiling, and not a single bus error yet. It's >>>>>> got >>>>>> 2x node board with dual R12K/400MHz CPUs per node. >>>>>> >>>>>> I'm not really sure what CONFIG_TRANSPARENT_HUGEPAGE is enabling that's >>>>>> causing >>>>>> R12K CPUs on the IP27 such a headache (and on Octane, really screws up R14K >>>>>> CPUs). I tried getting a core dump on one of the bus errors, but that >>>>>> produces a >>>>>> truncated or corrupted core file that actually crashed GDB, plus I get a >>>>>> nice >>>>>> oops message in dmesg: >>>>> >>>>> Well, as its name implies, if you enable CONFIG_TRANSPARENT_HUGEPAGE, huge >>>>> pages will be created and used in the background transparently to the >>>>> userspace >>>>> application. >>>>> >>>>> With 4KB base page size, the huge pages will be 2MB in size.. I don't know >>>>> much about the R10K/R12K/R14K CPUs, but it is possible that either their TLBs >>>>> cannot handle such pages, or that the TLB Exception handlers don't contain >>>>> proper code for these CPUs. >>>>> >>>>> For each doubling of the base PAGE_SIZE, the huge page size will increase >>>>> by a >>>>> factor of 4. So with 16KB base pages the huge page size would be 32MB, since >>>>> there are many fewer opportunities to transparently use a 32MB page, I would >>>>> expect any errors related to huge pages to be correspondingly less frequent. >>>>> >>>>> With 64KB PAGE_SIZE the huge page size is 512MB, and It is likely that that >>>>> could never be used by normal userspace programs. >>>> >>>> I checked the R10K/R12K manual, and the PageMask register there has bits 24:13 >>>> open for setting a mask value. It looks like these CPUs only support a page >>>> size from 4KB to 16MB (so a 2MB page size should work w/ transparent >>>> hugepages). I assume that the R14K on the Octane might be the same (but I >>>> don't have a manual specific to the R14k, so I don't know). All of the >>>> remaining bits in that register read 0 and must have 0's written back. >>>> >>>> I guess I could find a way to have the kernel trigger a non-fatal oops/dump >>>> the >>>> registers on a bus error and get a look at the cause register to see if that >>>> sheds any light on things. Doesn't a SIGBUS on MIPS typically mean that an >>>> address wasn't aligned on a 32-bit boundary? Or could it also mean other >>>> things? >>>> >>>> I believe that the R10K is largely compatible with the R4K-style TLB setup, >>>> but >>>> Ralf or someone else more knowledge in that area will have to verify. Maybe >>>> the R10k-family CPUs need their own TLB routines, or what currently exists >>>> needs modifications? I have not tried to understand the whole TLB thing in >>>> MIPS yet, so that's a bit of voodoo to me. >>> >>> I haven't checked, but there may be workarounds required in the TLB management >>> code that are not in place for the huge page case. When the huge TLB code was >>> developed, we didn't do any testing on R10K. Somebody should dump the >>> exception handlers and carefully look at the rest of the huge TLB management >>> code, and check to see that any required workarounds are in place. >> >> How does one dump the exception handlers? Is it a debug switch somewhere? >> > > Add as the very first line of tlbex.c "#define DEBUG 1" > > Then rebuild, and pass "debug" on the kernel command line. > > The output can be fed though gas, and then disassembled with objdump -d Had to fiddle with gas a little bit, but that was because I was using cross-compiler. Got it to work, though. tlb1-no-transparent-hugepage.dis is without CONFIG_TRANSPARENT_HUGEPAGE, while tlb2-transparent-hugepage.dis is with that option enabled. I don't think any of the other patches/hacks I've added to my IP27 build have affected the output. I saw that CPU1 through CPU3 also dumped r4000_tlb_refill only, but that looks to be the same across all of the CPUs, so I only compiled/disassembled the output from CPU0. --J >>>>>> [ 1302.260000] CPU: 0 PID: 1179 Comm: emerge Not tainted >>>>>> 3.17.1-mipsgit-20141006 #57 >>>>>> [ 1302.260000] task: a8000000ffbbf288 ti: a8000000fa6f0000 task.ti: >>>>>> a8000000fa6f0000 >>>>>> [ 1302.260000] $ 0 : 0000000000000000 0000000000000001 0000000000000000 >>>>>> a8000000ff5ad800 >>>>>> [ 1302.260000] $ 4 : a8000000006d5480 00000000000f9c00 00000001f380173f >>>>>> a800000001000000 >>>>>> [ 1302.260000] $ 8 : 00000001f380173f 0000000000100077 a8000000fe77a000 >>>>>> 0000000000000000 >>>>>> [ 1302.260000] $12 : 0000000000660000 0000000000000000 0000000000000000 >>>>>> 776bc40c00000004 >>>>>> [ 1302.260000] $16 : 0000000000e00000 0000000000000000 00000000018ee000 >>>>>> 6db6db6db6db6db7 >>>>>> [ 1302.260000] $20 : 00000000000000ca a8000000006d5480 a8000000ff65fa68 >>>>>> 0000000000001000 >>>>>> [ 1302.260000] $24 : 0000000000000000 a8000000000469c0 >>>>>> [ 1302.260000] $28 : a8000000fa6f0000 a8000000fa6f3a00 0000000000e00000 >>>>>> a800000000046720 >>>>>> [ 1302.260000] Hi : 00000000002ed400 >>>>>> [ 1302.260000] Lo : 00000000000f9c00 >>>>>> [ 1302.260000] epc : a8000000000467e4 r4k_flush_cache_page+0x104/0x2e0 >>>>>> [ 1302.260000] Not tainted >>>>>> [ 1302.260000] ra : a800000000046720 r4k_flush_cache_page+0x40/0x2e0 >>>>>> [ 1302.260000] Status: 90001ce3 KX SX UX KERNEL EXL IE >>>>>> [ 1302.260000] Cause : 0000c010 >>>>>> [ 1302.260000] BadVA : 00000001f380173f >>>>>> [ 1302.260000] PrId : 00000e35 (R12000) >>>>>> [ 1302.260000] Process emerge (pid: 1179, threadinfo=a8000000fa6f0000, >>>>>> task=a8000000ffbbf288, tls=00000000778d2490) >>>>>> [ 1302.260000] Stack : a8000000ff65fa68 0000000000e00000 00000000000f9c00 >>>>>> a8000000006d5480 >>>>>> a8000000ff65fa68 0000000000001000 0000000000e00000 >>>>>> a80000000010cb00 >>>>>> a8000000046a2000 a8000000ff65fa68 00000000018ee000 >>>>>> 6db6db6db6db6db7 >>>>>> a8000000fe7fdce0 a8000000000375ec a8000000ff4e5800 >>>>>> a8000000005fbd90 >>>>>> 0000000300000080 a8000000ff668580 a8000000005fbd90 >>>>>> 5349474900000080 >>>>>> a8000000fa6f3ad8 a8000000005fbd90 0000000600000088 >>>>>> a8000000ff5ad928 >>>>>> a8000000005fbd90 46494c4500002bf9 c000000000101000 >>>>>> 0000000a00000080 >>>>>> 0000000000000000 0000000000000000 0000000000000000 >>>>>> 0000000000000000 >>>>>> 0000000000000000 0000000000000000 0000000000000000 >>>>>> 0000000000000000 >>>>>> 0000000000000000 0000000000000000 0000000000000000 >>>>>> 0000000000000000 >>>>>> ... >>>>>> [ 1302.260000] Call Trace: >>>>>> [ 1302.260000] [] r4k_flush_cache_page+0x104/0x2e0 >>>>>> [ 1302.260000] [] get_dump_page+0xc8/0xe8 >>>>>> [ 1302.260000] [] elf_core_dump+0x1294/0x14d8 >>>>>> [ 1302.260000] [] do_coredump+0x5e4/0x1048 >>>>>> [ 1302.260000] [] get_signal+0x1b8/0x710 >>>>>> [ 1302.260000] [] do_signal+0x18/0x240 >>>>>> [ 1302.260000] [] do_notify_resume+0x70/0x88 >>>>>> [ 1302.260000] [] work_notifysig+0x10/0x18 >>>>>> [ 1302.260000] >>>>>> [ 1302.260000] >>>>>> Code: 0010327a 30c60ff8 00c8302d 30c80001 1100003e 00000000 >>>>>> bfb40000 df880000 >>>>>> [ 1305.340000] ---[ end trace c7649a6433db8d18 ]--- >>>>>> >>>>>> Thoughts? -- Joshua Kinard Gentoo/MIPS kumba@gentoo.org 4096R/D25D95E3 2011-03-28 "The past tempts us, the present confuses us, the future frightens us. And our lives slip away, moment by moment, lost in that vast, terrible in-between." --Emperor Turhan, Centauri Republic