ibook2r2 & strange freeze.

ibook2r2 & strange freeze.

[Brice Figureau]

Hi all,

I apologize if this is the wrong list to ask my questions...
I don't know if anyone here could have an answer to my problem.

I have an Apple iBook 2.rev2 from last mid year.
I was running almost perfectly one of the Benjamin Herrenschmidt's
2.4.19 kernel.
A few days ago I completely re-installed the beast with the latest
Mandrake PPC 9.1 which is shipping with a non working kernel for those
ibooks.

I then compiled & installed the latest benh stable kernel (2.4.20-ben10)
which is really unstable on my computer when booted with the default
parameters.

When I boot with video=ofonly there are no problem at all (except that
the backlight controller does not work anymore...).
Several other mandrake user owning ibook2r2 reported that their computer
where not stable without video=ofonly.

By unstable I mean that the computer completely freeze during boot time,
around the time rc.sysinit is launched. Unfortunately I'm rather new to
linux kernel so I don't know how to provide more information on the
crash (maybe should I configure the kernel with xmon ?).
In a divide&conquer approach I found that the part of rc.sysinit which
'uploads' the console font to the font layer was triggering the crash.

I then re-installed from scratch everything (because I thought there
could have been a fs corruption somewhere which had corrupted the
fonts), but no chance it crashed again.

I then commented-out this part of rc.sysinit and then I could boot up to
the login prompt.

But the kernel was still unstable, and still froze at completely random
time (mounting a cdrom, vi'ing some text files, rsyncing a kernel,
compiling the kernel).

I decided to have a look to the latest devel kernel (namely
2.4.21-pre7-ben0 and 2.4.21-rc1-ben0) but no chance again, when booted
with video=radeonfb (or with nothing) they're oopsing during the boot (I
can provide more information if needed). With video=ofonly, everything
went smooth.

As I wasn't sure the crash came from my compiled kernel, I also tried
pre-built kernel from penguinppc.org (the following dmesg output has
been taken from this kernel). This kernel was relatively stable with my
modified rc.sysinit, but failed with the original rc.sysinit.

So, what should I do to have a stable kernel with radeonfb ?

Many thanks,
Regards,

Brice


------------------------------- cpuinfo output ----------------------

# cat /proc/cpuinfo
cpu             : 750FX
temperature     : 72 C (uncalibrated)
clock           : 700MHz
revision        : 1.2 (pvr 7000 0102)
bogomips        : 1389.36
machine         : PowerBook4,3
motherboard     : PowerBook4,3 MacRISC2 MacRISC Power Macintosh
detected as     : 257 (iBook 2 rev. 2)
pmac flags      : 0000000b
L2 cache        : 512K unified
memory          : 384MB
pmac-generation : NewWorld


--------------------------- lspci output ---------------------------

# lspci -v
00:0b.0 Host bridge: Apple Computer Inc. UniNorth/Pangea AGP
        Flags: bus master, 66Mhz, medium devsel, latency 16
        Capabilities: [80] AGP version 1.0

00:10.0 VGA compatible controller: ATI Technologies Inc Radeon Mobility
M6 LY (prog-if 00 [VGA])
        Subsystem: ATI Technologies Inc Radeon Mobility M6 LY
        Flags: bus master, stepping, 66Mhz, medium devsel, latency 255,
IRQ 48
        Memory at 98000000 (32-bit, prefetchable) [size=128M]
        I/O ports at 802400 [disabled] [size=256]
        Memory at 90000000 (32-bit, non-prefetchable) [size=64K]
        Expansion ROM at 90020000 [disabled] [size=128K]
        Capabilities: [58] AGP version 2.0
        Capabilities: [50] Power Management version 2

10:0b.0 Host bridge: Apple Computer Inc. UniNorth/Pangea PCI
        Flags: bus master, 66Mhz, medium devsel, latency 16

10:17.0 Class ff00: Apple Computer Inc. KeyLargo/Pangea Mac I/O
        Flags: bus master, medium devsel, latency 16
        Memory at 80000000 (32-bit, non-prefetchable) [size=512K]

10:18.0 USB Controller: Apple Computer Inc. KeyLargo/Pangea USB (prog-if
10 [OHCI])
        Flags: bus master, medium devsel, latency 16, IRQ 27
        Memory at 80081000 (32-bit, non-prefetchable) [size=4K]
        Capabilities: [50] Power Management version 2

10:19.0 USB Controller: Apple Computer Inc. KeyLargo/Pangea USB (prog-if
10 [OHCI])
        Flags: bus master, medium devsel, latency 16, IRQ 28
        Memory at 80080000 (32-bit, non-prefetchable) [size=4K]
        Capabilities: [50] Power Management version 2

20:0b.0 Host bridge: Apple Computer Inc. UniNorth/Pangea Internal PCI
        Flags: bus master, 66Mhz, medium devsel, latency 16

20:0e.0 FireWire (IEEE 1394): Apple Computer Inc. UniNorth/Pangea
FireWire (prog-if 10 [OHCI])
        Subsystem: Apple Computer Inc. UniNorth/Pangea FireWire
        Flags: bus master, 66Mhz, medium devsel, latency 16, IRQ 40
        Memory at f5000000 (32-bit, non-prefetchable) [size=4K]
        Capabilities: [44] Power Management version 2

20:0f.0 Ethernet controller: Apple Computer Inc. UniNorth/Pangea GMAC
(Sun GEM)
        Flags: bus master, 66Mhz, slow devsel, latency 6, IRQ 41
        Memory at f5200000 (32-bit, non-prefetchable) [size=2M]
        Expansion ROM at f5100000 [disabled] [size=1M]

-------------------------- 2.4.20-ben10 dmesg --------------------------

Memory BAT mapping: BAT2=256Mb, BAT3=128Mb, residual: 0Mb
Total memory = 384MB; using 1024kB for hash table (at c0400000)
Linux version 2.4.20-ben10 (tmcnulty@dual) (gcc version 3.2.1) #1 Wed
Apr 2 21:07:23 EST 2003
Found Uninorth memory controller & host bridge, revision: 192
Mapped at 0xfdf3c000
Found a Pangea mac-io controller, rev: 0, mapped at 0xfdebc000
Processor NAP mode on idle enabled.
PowerMac motherboard: iBook 2 rev. 2
PMU driver 2 initialized for Core99, firmware: 0c
CPU HID1 : 0x70000044
Found UniNorth PCI host bridge at 0xf0000000. Firmware bus number: 0->0
hose pci, pci_mem_offset: 00000000, start0: f1000000
  io_base_virt: fd6b5000, io_base_phys: f0000000, isa_mem_base: 00000000
Found UniNorth PCI host bridge at 0xf2000000. Firmware bus number: 0->0
hose pci, pci_mem_offset: 00000000, start0: f3000000
  io_base_virt: fceb3000, io_base_phys: f2000000, isa_mem_base: 00000000
Found UniNorth PCI host bridge at 0xf4000000. Firmware bus number: 0->0
hose pci, pci_mem_offset: 00000000, start0: f5000000
  io_base_virt: fc6b1000, io_base_phys: f4000000, isa_mem_base: 00000000
On node 0 totalpages: 98304
zone(0): 98304 pages.
zone(1): 0 pages.
zone(2): 0 pages.
Kernel command line: root=/dev/hda13 ro  devfs=mount hdb=ide-scsi 3
ide_setup: hdb=ide-scsi
PowerMac using OpenPIC irq controller
OpenPIC Version 1.2 (4 CPUs and 64 IRQ sources) at fc66d000
OpenPIC timer frequency is 4.160000 MHz
GMT Delta read from XPRAM: 120 minutes, DST: on
time_init: decrementer frequency = 24.835245 MHz
Console: colour dummy device 80x25
Calibrating delay loop... 1389.36 BogoMIPS
Memory: 383744k available (1992k kernel code, 892k data, 288k init, 0k
highmem)
AGP special page: 0xd7fff000
Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
Inode cache hash table entries: 32768 (order: 6, 262144 bytes)
Mount-cache hash table entries: 8192 (order: 4, 65536 bytes)
Buffer-cache hash table entries: 32768 (order: 5, 131072 bytes)
Page-cache hash table entries: 131072 (order: 7, 524288 bytes)
POSIX conformance testing by UNIFIX
PCI: Probing PCI hardware
Scanning bus 00
Found 00:58 [106b/0027] 000600 00
Found 00:80 [1002/4c59] 000300 00
Fixup res 1 (101) of dev 00:10.0: 400 -> 802400
Fixups for bus 00
Bus scan for 00 returning with max=00
Scanning bus 10
Found 10:58 [106b/0028] 000600 00
Found 10:b8 [106b/0025] 00ff00 00
Found 10:c0 [106b/0026] 000c03 00
Found 10:c8 [106b/0026] 000c03 00
Fixups for bus 10
Bus scan for 10 returning with max=10
Scanning bus 20
Found 20:58 [106b/0029] 000600 00
Found 20:70 [106b/0030] 000c00 00
Found 20:78 [106b/0024] 000200 00
Fixups for bus 20
Bus scan for 20 returning with max=20
PCI->OF bus map:
0 -> 0
16 -> 0
32 -> 0
PCI:00:10.0: Resource 0: 98000000-9fffffff (f=1208), vd: 1002, dev: 4c59
PCI:00:10.0: Resource 2: 90000000-9000ffff (f=200), vd: 1002, dev: 4c59
PCI:10:17.0: Resource 0: 80000000-8007ffff (f=200), vd: 106b, dev: 0025
PCI:20:0f.0: Resource 0: f5200000-f53fffff (f=200), vd: 106b, dev: 0024
PCI:00:10.0: Resource 1: 00802400-008024ff (f=101), vd: 1002, dev: 4c59
PCI:10:18.0: Resource 0: 80081000-80081fff (f=200), vd: 106b, dev: 0026
PCI:10:19.0: Resource 0: 80080000-80080fff (f=200), vd: 106b, dev: 0026
PCI:20:0e.0: Resource 0: f5000000-f5000fff (f=200), vd: 106b, dev: 0030
Linux NET4.0 for Linux 2.4
Based upon Swansea University Computer Society NET3.039
Initializing RT netlink socket
Thermal assist unit using timers, shrink_timer: 200 jiffies
Starting kswapd
Journalled Block Device driver loaded
devfs: v1.12c (20020818) Richard Gooch (rgooch@atnf.csiro.au)
devfs: boot_options: 0x1
Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
PCI: Enabling device 00:10.0 (0086 -> 0087)
radeonfb: ref_clk=2700, ref_div=12, xclk=16615 from OF
radeonfb: detected DFP panel size from EDID: 1024x768
Console: switching to colour frame buffer device 128x48
Registered "ati" backlight controller, level: 15/15
radeonfb: ATI Radeon M6 LY DDR SGRAM 16 MB
radeonfb: DVI port LCD monitor connected
radeonfb: CRT port no monitor connected
radeonfb: Power Management enabled for Mobility chipsets
input0: Macintosh mouse button emulation
pty: 256 Unix98 ptys configured
Uniform Multi-Platform E-IDE driver Revision: 6.31
ide: Assuming 33MHz system bus speed for PIO modes; override with
idebus=xx
ide0: Found Apple KeyLargo ATA-4 controller, bus ID 2
Probing IDE interface ide0...
hda: IBM-IC25N040ATCS04-0, ATA DISK drive
hdb: TOSHIBA DVD-ROM SD-R2102, ATAPI CD/DVD-ROM drive
ide0 at 0xdd153000-0xdd153007,0xdd153160 on irq 19
hda: Enabling Ultra DMA 2
blk: queue c02fe780, I/O limit 4095Mb (mask 0xffffffff)
hda: 78140160 sectors (40008 MB) w/1768KiB Cache, CHS=77520/16/63,
UDMA(33)
ide-floppy driver 0.99.newide
hdb: Enabling Ultra DMA 2
Partition check:
 /dev/ide/host0/bus0/target0/lun0: [mac] p1 p2 p3 p4 p5 p6 p7 p8 p9 p10
p11 p12 p13 p14 p15 p16 p17
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
loop: loaded (max 8 devices)
PPP generic driver version 2.4.2
PPP Deflate Compression module registered
ide-floppy driver 0.99.newide
SCSI subsystem driver Revision: 1.00
scsi0 : SCSI host adapter emulation for IDE ATAPI devices
  Vendor: TOSHIBA   Model: DVD-ROM SD-R2102  Rev: 1K27
  Type:   CD-ROM                             ANSI SCSI revision: 02
mesh: configured for synchronous 5 MB/s
Attached scsi CD-ROM sr0 at scsi0, channel 0, id 0, lun 0
sr0: scsi3-mmc drive: 24x/24x writer cd/rw xa/form2 cdda tray
Uniform CD-ROM driver Revision: 3.12
orinoco.c 0.11b (David Gibson <hermes@gibson.dropbear.id.au> and others)
hermes.c: 5 Apr 2002 David Gibson <hermes@gibson.dropbear.id.au>
PowerMac Z8530 serial driver version 2.0
tty00 at 0xda00e020 (irq = 22) is a Z8530 ESCC
tty01 at 0xda015000 (irq = 23) is a Z8530 ESCC
Macintosh non-volatile memory driver v1.0
apm_emu: APM Emulation 0.5 initialized.
usb.c: registered new driver usbdevfs
usb.c: registered new driver hub
PCI: Enabling device 10:18.0 (0000 -> 0002)
PCI: Enabling bus mastering for device 10:18.0
usb-ohci.c: USB OHCI at membase 0xda01b000, IRQ 27
usb-ohci.c: usb-10:18.0, Apple Computer Inc. KeyLargo/Pangea USB
adb: starting probe task...
usb.c: new USB bus registered, assigned bus number 1
hub.c: USB hub found
hub.c: 2 ports detected
PCI: Enabling device 10:19.0 (0000 -> 0002)
PCI: Enabling bus mastering for device 10:19.0
usb-ohci.c: USB OHCI at membase 0xda01d000, IRQ 28
usb-ohci.c: usb-10:19.0, Apple Computer Inc. KeyLargo/Pangea USB (#2)
usb.c: new USB bus registered, assigned bus number 2
hub.c: USB hub found
hub.c: 2 ports detected
usb.c: registered new driver hid
hid-core.c: v1.8.1 Andreas Gal, Vojtech Pavlik <vojtech@suse.cz>
hid-core.c: USB HID support drivers
mice: PS/2 mouse device common for all mice
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP, IGMP
IP: routing cache hash table of 4096 buckets, 32Kbytes
TCP: Hash tables configured (established 32768 bind 65536)
NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.
RAMDISK: Compressed image found at block 0
Freeing initrd memory: 592k freed
VFS: Mounted root (ext2 filesystem).
Mounted devfs on /dev
adb devices: [2]: 2 c4 [3]: 3 1 [7]: 7 1f
ADB keyboard at 2, handler 1
Detected ADB keyboard, type ISO, swapping keys.
input1: ADB HID on ID 2:2.c4
input2: ADB HID on ID 7:7.1f
ADB mouse at 3, handler set to 4 (trackpad)<6>kjournald starting.
Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.

input3: ADB HID on ID 3:3.01
adb: finished probe task...
Mounted devfs on /dev
Freeing unused kernel memory: 288k init 8k chrp 8k prep
hub.c: new USB device 10:19.0-1, assigned address 2
input4: USB HID v10.01 Mouse [Logitech USB-PS/2 Optical Mouse] on
usb2:2.0
EXT3 FS 2.4-0.9.19, 19 August 2002 on ide0(3,13), internal journal
Adding Swap: 516088k swap-space (priority -1)
kjournald starting.  Commit interval 5 seconds
EXT3 FS 2.4-0.9.19, 19 August 2002 on ide0(3,17), internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS 2.4-0.9.19, 19 August 2002 on ide0(3,15), internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS 2.4-0.9.19, 19 August 2002 on ide0(3,16), internal journal
EXT3-fs: mounted filesystem with ordered data mode.
eth0: GMAC at 00:03:93:9d:5f:18, driver v1.5k4
i2c-core.o: i2c core module
i2c-core.o: adapter uni-n 0 registered as adapter 0.
i2c-core.o: adapter uni-n 1 registered as adapter 1.
Found KeyWest i2c on "uni-n", 2 channels, stepping: 4 bits
i2c-core.o: adapter mac-io 0 registered as adapter 2.
Found KeyWest i2c on "mac-io", 1 channel, stepping: 4 bits
tas driver [TAS3004 driver V 0.3])
using i2c address: 0x35 from device-tree
i2c-core.o: driver TAS3004 driver V 0.3 registered.
i2c-core.o: client [tas Digital Equalizer] registered to adapter [mac-io
0](pos. 0).
Audio jack unplugged, enabling speakers.
AE-Init snapper mixer
PowerMac Snapper  DMA sound driver rev 016 installed
Core driver edition 01.06 : PowerMac Built-in Sound driver edition 00.07
Write will use    4 fragments of   32768 bytes as default
eth0: PHY ID: 0x004061e4
eth0: Found Broadcom BCM5221 PHY
eth0: Link state change, phy_status: 0x782d
eth0:    Link up ! BCM5201/5221 aux_stat: 0x003f
eth0:    Full Duplex: 1, Speed: 100
svc: unknown version (3)


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: ibook2r2 & strange freeze.

[Benjamin Herrenschmidt]

> By unstable I mean that the computer completely freeze during boot time,
> around the time rc.sysinit is launched. Unfortunately I'm rather new to
> linux kernel so I don't know how to provide more information on the
> crash (maybe should I configure the kernel with xmon ?).
> In a divide&conquer approach I found that the part of rc.sysinit which
> 'uploads' the console font to the font layer was triggering the crash.

Can you try 2 things and let me know if one of them helps ?

 - Compile without CPUFREQ support
 - Use video=radeon:noaccel on the command line to disable acceleration
   on console

> I then re-installed from scratch everything (because I thought there
> could have been a fs corruption somewhere which had corrupted the
> fonts), but no chance it crashed again.
>
> I then commented-out this part of rc.sysinit and then I could boot up to
> the login prompt.
>
> But the kernel was still unstable, and still froze at completely random
> time (mounting a cdrom, vi'ing some text files, rsyncing a kernel,
> compiling the kernel).
>
> I decided to have a look to the latest devel kernel (namely
> 2.4.21-pre7-ben0 and 2.4.21-rc1-ben0) but no chance again, when booted
> with video=radeonfb (or with nothing) they're oopsing during the boot (I
> can provide more information if needed). With video=ofonly, everything
> went smooth.
>
> As I wasn't sure the crash came from my compiled kernel, I also tried
> pre-built kernel from penguinppc.org (the following dmesg output has
> been taken from this kernel). This kernel was relatively stable with my
> modified rc.sysinit, but failed with the original rc.sysinit.
>
> So, what should I do to have a stable kernel with radeonfb ?

That's interesting. I need to investigate...

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: ibook2r2 & strange freeze.

[Benjamin Herrenschmidt]

On Fri, 2003-04-25 at 19:43, Benjamin Herrenschmidt wrote:
> > By unstable I mean that the computer completely freeze during boot time,
> > around the time rc.sysinit is launched. Unfortunately I'm rather new to
> > linux kernel so I don't know how to provide more information on the
> > crash (maybe should I configure the kernel with xmon ?).
> > In a divide&conquer approach I found that the part of rc.sysinit which
> > 'uploads' the console font to the font layer was triggering the crash.
>
> Can you try 2 things and let me know if one of them helps ?
>
>  - Compile without CPUFREQ support
>  - Use video=radeon:noaccel on the command line to disable acceleration
>    on console

Also, it would be helpful if you could go backward the kernel
versions to point when the problem started. You could try
2.4.20-ben1. If it doesn't have the problem, then -ben5, etc...
(do a dichotomy).

If -ben1 is affected, then we'll need to do something different

(Patches are available on kernel.org)

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: ibook2r2 & strange freeze.

[Brice Figureau]

Hi Benjamin,

On Friday, April 25, 2003, at 07:50 PM, Benjamin Herrenschmidt wrote:
> On Fri, 2003-04-25 at 19:43, Benjamin Herrenschmidt wrote:
>>> By unstable I mean that the computer completely freeze during boot
>>> time,
>>> around the time rc.sysinit is launched. Unfortunately I'm rather new
>>> to
>>> linux kernel so I don't know how to provide more information on the
>>> crash (maybe should I configure the kernel with xmon ?).
>>> In a divide&conquer approach I found that the part of rc.sysinit
>>> which
>>> 'uploads' the console font to the font layer was triggering the
>>> crash.
>>
>> Can you try 2 things and let me know if one of them helps ?
>>
>>  - Compile without CPUFREQ support
>>  - Use video=radeon:noaccel on the command line to disable
>> acceleration
>>    on console
This produced the same hard freeze (I tried every combination of the
two things).

> Also, it would be helpful if you could go backward the kernel
> versions to point when the problem started. You could try
> 2.4.20-ben1. If it doesn't have the problem, then -ben5, etc...
> (do a dichotomy).
Unfortunately there seems to be only from ben3 to ben10 available on
your site.

And fortunately ben3 is *not* affected !
So the problem lies between 4 and 10. I guess my next kernel compile
will be 6.

I'll compile your other patches tomorrow and let you know at which
point it fails.

Many thanks,
Regards,

Brice


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: ibook2r2 & strange freeze.

[Remco Treffkorn]

On Friday 25 April 2003 08:27, Brice Figureau wrote:
...
> I then commented-out this part of rc.sysinit and then I could boot up to
> the login prompt.

Seems to me you have more than one problem. I have trouble with CONFIG_RTC
set. Hwclock will hang in rc.sysinit. Try configuring without RTC.

Cheers,
Remco

--
Remco Treffkorn (RT445)
HAM DC2XT
remco@rvt.com   (831) 685-1201

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: ibook2r2 & strange freeze.

[Brice Figureau]

On Friday, April 25, 2003, at 10:12 PM, Remco Treffkorn wrote:
> On Friday 25 April 2003 08:27, Brice Figureau wrote:
> ...
>> I then commented-out this part of rc.sysinit and then I could boot up
>> to
>> the login prompt.
>
> Seems to me you have more than one problem. I have trouble with
> CONFIG_RTC
> set. Hwclock will hang in rc.sysinit. Try configuring without RTC.
No, CONFIG_RTC is disabled. The crash usually occurs usually before
hwclock.

Anyway, thanks for the idea.

Brice


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: ibook2r2 & strange freeze.

[Brice Figureau]

On Friday, April 25, 2003, at 07:50 PM, Benjamin Herrenschmidt wrote:
> On Fri, 2003-04-25 at 19:43, Benjamin Herrenschmidt wrote:
>>> By unstable I mean that the computer completely freeze during boot
>>> time,
>>> around the time rc.sysinit is launched. Unfortunately I'm rather new
>>> to
>>> linux kernel so I don't know how to provide more information on the
>>> crash (maybe should I configure the kernel with xmon ?).
>>> In a divide&conquer approach I found that the part of rc.sysinit
>>> which
>>> 'uploads' the console font to the font layer was triggering the
>>> crash.
>>
>> Can you try 2 things and let me know if one of them helps ?
>>
>>  - Compile without CPUFREQ support
>>  - Use video=radeon:noaccel on the command line to disable
>> acceleration
>>    on console
>
> Also, it would be helpful if you could go backward the kernel
> versions to point when the problem started. You could try
> 2.4.20-ben1. If it doesn't have the problem, then -ben5, etc...
> (do a dichotomy).
After several hours of kernel compiling (my ibook is not that fast
;-)), I found that the latest working patch is -ben8.
The first non working is then -ben9.
I can crash ben9 almost reproductibly and xmon is triggered if that can
help you (I didn't understand how to use the System.map with xmon to
decipher the stack trace, but I might send xmon output and my
System.map).
Note this might not be the same crash as ben10, because xmon is not
fired with this kernel, although it seems to crash at the early
beginning of rc.sysinit (I'm almost sure this is something about the
fonts).

I hope this will help you to find the problem.
I can do more test if you wants me to.
Just drop me a mail.

Many thanks,

Brice


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: ibook2r2 & strange freeze.

[Mich Lanners]

On  25 Apr, this message from Remco Treffkorn echoed through cyberspace:
> Seems to me you have more than one problem. I have trouble with
> CONFIG_RTC set. Hwclock will hang in rc.sysinit.

CONFIG_RTC is known to cause havoc on Macs. This has been mentioned at
least once per week here :-).

> Try configuring
> without RTC.

Rather, set like this:

CONFIG_PPC_RTC=y
# CONFIG_RTC is not set

CONFIG_RTC is generic i386 RTC. The driver tries to access hardcoded IO ports.

Cheers

Michel

-------------------------------------------------------------------------
Michel Lanners                 |  " Read Philosophy.  Study Art.
23, Rue Paul Henkes            |    Ask Questions.  Make Mistakes.
L-1710 Luxembourg              |
email   mlan@cpu.lu            |
http://www.cpu.lu/~mlan        |                     Learn Always. "

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: ibook2r2 & strange freeze.

[Benjamin Herrenschmidt]

> After several hours of kernel compiling (my ibook is not that fast
> ;-)), I found that the latest working patch is -ben8.
> The first non working is then -ben9.

Great, many thanks. That will help.

Can you try now to put drivers/video/radeonfb.c from -ben8 into
-ben9 (and then -ben10 if it works) and tell me if that helps ?

> I can crash ben9 almost reproductibly and xmon is triggered if that can
> help you (I didn't understand how to use the System.map with xmon to
> decipher the stack trace, but I might send xmon output and my
> System.map).
> Note this might not be the same crash as ben10, because xmon is not
> fired with this kernel, although it seems to crash at the early
> beginning of rc.sysinit (I'm almost sure this is something about the
> fonts).

The system.map thing might not help (if the crash is random), but
basically, you just need to copy System.map next to vmlinux (in
/boot typically) and edit yaboot.conf to add a line

  sysmap=/boot/System.map

after the image=... line (and re-run ybin to install the new yaboot.conf)

With this, xmon will know about symbols. Then, you can use the "t"
command in xmon to get a backtrace.

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: ibook2r2 & strange freeze.

[Brice Figureau]

On Saturday, April 26, 2003, at 12:33 PM, Benjamin Herrenschmidt wrote:
>
>> After several hours of kernel compiling (my ibook is not that fast
>> ;-)), I found that the latest working patch is -ben8.
>> The first non working is then -ben9.
>
> Great, many thanks. That will help.
>
> Can you try now to put drivers/video/radeonfb.c from -ben8 into
> -ben9 (and then -ben10 if it works) and tell me if that helps ?
Here is the results:
  -ben9 with -ben8 radeon crashes exactly at the same place
  -ben10 with -ben8 radeon crashes exacctly like -ben9

Just to remind you:
  -ben9 crashes always at the same place (and triggers xmon, have a look
at the backtrace)
  -ben10 freezes randomly, usually in the boot process. It freezes often
right after the font uploading to the console layer, but sometimes
farther (I once could go until the network started). When it freezes
the last line at the bottom of the screen is printed *twice* (which
might confirm the -ben9 backtrace).

All the tested kernels were configured with CONFIG_PPC_RTC enabled (and
CONFIG_RTC disabled), and CONFIG_CPU_FREQ disabled too.

noaccel parameters didn't seem to change anything. I might have to
check again.

>> I can crash ben9 almost reproductibly and xmon is triggered if that
>> can
>> help you (I didn't understand how to use the System.map with xmon to
>> decipher the stack trace, but I might send xmon output and my
>> System.map).
>> Note this might not be the same crash as ben10, because xmon is not
>> fired with this kernel, although it seems to crash at the early
>> beginning of rc.sysinit (I'm almost sure this is something about the
>> fonts).
>
> With this, xmon will know about symbols. Then, you can use the "t"
> command in xmon to get a backtrace.
Here is the -ben9 crash stack trace (hand copied from the screen)
Vector:200 at pc=.... (t fbcon_radeon_bmove+0x1c8) lr=..... (t
fbcon_scroll+0x7b4)
...
backtrace:

T do_page_faults+0x298
t fbcon_scroll+0xac
t scrup+0x11c
t lf+0x98
t do_con_trol+0xda0
t do_con_write_0x480
t con_put_char+0x50
t opost+0xbc
t write_char+0x1c4
t do_tty_write+0x90
t tty_write+0x12c
t sys_write+0xbc
t ret_from_syscall_1+0x0

As far as I can see, the crash occurs while scrolling the screen.
Since the line is printed twice in the ben10 freeze, it might be the
same problem as ben9.
If only I could crash ben10 and end up into xmon...

I hope this will help you. If you need more information, let me know.


Thanks,
Regards,
Brice


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: ibook2r2 & strange freeze.

[Benjamin Herrenschmidt]

On Sat, 2003-04-26 at 22:22, Brice Figureau wrote:
> On Saturday, April 26, 2003, at 12:33 PM, Benjamin Herrenschmidt wrote:
> >
> >> After several hours of kernel compiling (my ibook is not that fast
> >> ;-)), I found that the latest working patch is -ben8.
> >> The first non working is then -ben9.
> >
> > Great, many thanks. That will help.
> >
> > Can you try now to put drivers/video/radeonfb.c from -ben8 into
> > -ben9 (and then -ben10 if it works) and tell me if that helps ?
> Here is the results:
>   -ben9 with -ben8 radeon crashes exactly at the same place
>   -ben10 with -ben8 radeon crashes exacctly like -ben9

That's weird since you say using no video driver (that is offb) seems
to cure the problem and the backtrace tend to show an fbcon problem,
but there is nothing different in the fbcon layer afaik.

So there is probably some memory corruption going on... I'm comparing
other bits of ben8 and ben9 now and see nothing relevants. The changes
where in the cache flush affecting cpufreq and sleep, and radeonfb...

> Just to remind you:
>   -ben9 crashes always at the same place (and triggers xmon, have a look
> at the backtrace)
>   -ben10 freezes randomly, usually in the boot process. It freezes often
> right after the font uploading to the console layer, but sometimes
> farther (I once could go until the network started). When it freezes
> the last line at the bottom of the screen is printed *twice* (which
> might confirm the -ben9 backtrace).
>
> All the tested kernels were configured with CONFIG_PPC_RTC enabled (and
> CONFIG_RTC disabled), and CONFIG_CPU_FREQ disabled too.
>
> noaccel parameters didn't seem to change anything. I might have to
> check again.

Just in case, the proper syntax in yaboot.conf is:

append="video=radeon:noaccel"

The backtrace seem to indicate something wrong with fbcon_scroll, though
I fail to see what/why, I suspect something else is causing it to die.

With xmon, can you do:

di %pc

and then

r

So I can have a better idea of what it's doing. It's probably using
a pointer that got overwritten, but the actual cause of the corruption
can be completely elsewhere in the kernel. It may also be present in
earlier kernels and happen to corrupt some other harmless bit of memory
in them... Unfortunately, this one may be difficult to track down.

Ben.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: ibook2r2 & strange freeze.

[Brice Figureau]

On Saturday, April 26, 2003, at 10:35 PM, Benjamin Herrenschmidt wrote:
> On Sat, 2003-04-26 at 22:22, Brice Figureau wrote:
>> On Saturday, April 26, 2003, at 12:33 PM, Benjamin Herrenschmidt
>> wrote:
>>>
>>>> After several hours of kernel compiling (my ibook is not that fast
>>>> ;-)), I found that the latest working patch is -ben8.
>>>> The first non working is then -ben9.
>>>
>>> Great, many thanks. That will help.
>>>
>>> Can you try now to put drivers/video/radeonfb.c from -ben8 into
>>> -ben9 (and then -ben10 if it works) and tell me if that helps ?
>> Here is the results:
>>   -ben9 with -ben8 radeon crashes exactly at the same place
>>   -ben10 with -ben8 radeon crashes exacctly like -ben9
>
> That's weird since you say using no video driver (that is offb) seems
> to cure the problem and the backtrace tend to show an fbcon problem,
> but there is nothing different in the fbcon layer afaik.
The pc was in fbcon_radeon_bmove, so even if it's a fbcon problem, the
crash is happening into the radeonfb module...
video=ofonly works fine because it does not use fbcon_radeon_bmove I
guess...

> So there is probably some memory corruption going on... I'm comparing
> other bits of ben8 and ben9 now and see nothing relevants. The changes
> where in the cache flush affecting cpufreq and sleep, and radeonfb...
Since the crash occurs in the scoll, if there is a problem with the
font (corrupted font or something) the scroll computation might be
wrong, and thus fbcon_radeon_bmove might copy some memory from/to a bad
area.

>> Just to remind you:
>>   -ben9 crashes always at the same place (and triggers xmon, have a
>> look
>> at the backtrace)
>>   -ben10 freezes randomly, usually in the boot process. It freezes
>> often
>> right after the font uploading to the console layer, but sometimes
>> farther (I once could go until the network started). When it freezes
>> the last line at the bottom of the screen is printed *twice* (which
>> might confirm the -ben9 backtrace).
>>
>> All the tested kernels were configured with CONFIG_PPC_RTC enabled
>> (and
>> CONFIG_RTC disabled), and CONFIG_CPU_FREQ disabled too.
>>
>> noaccel parameters didn't seem to change anything. I might have to
>> check again.
>
> Just in case, the proper syntax in yaboot.conf is:
>
> append="video=radeon:noaccel"
Yes I know, it was just shorter to write noaccel in my mail ;-)

> The backtrace seem to indicate something wrong with fbcon_scroll,
> though
> I fail to see what/why, I suspect something else is causing it to die.

> With xmon, can you do:
>
> di %pc
>
> and then
>
> r
>
> So I can have a better idea of what it's doing. It's probably using
> a pointer that got overwritten, but the actual cause of the corruption
> can be completely elsewhere in the kernel. It may also be present in
> earlier kernels and happen to corrupt some other harmless bit of memory
> in them... Unfortunately, this one may be difficult to track down.
Unfortunately those bugs are the most difficult to find and fix.
What I could do is disable some part of the -ben9 patch in favor of the
-ben8 version, until I find in which part it happens.

I'll let you know what I found.
Thanks,

Brice


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: ibook2r2 & strange freeze.

[Benjamin Herrenschmidt]

> The pc was in fbcon_radeon_bmove, so even if it's a fbcon problem, the
> crash is happening into the radeonfb module...
> video=ofonly works fine because it does not use fbcon_radeon_bmove I
> guess...

Well... The only way I see fbcon_radeon_bmove crash this way
would be because of a corrupt struct display.

> > So there is probably some memory corruption going on... I'm comparing
> > other bits of ben8 and ben9 now and see nothing relevants. The changes
> > where in the cache flush affecting cpufreq and sleep, and radeonfb...
> Since the crash occurs in the scoll, if there is a problem with the
> font (corrupted font or something) the scroll computation might be
> wrong, and thus fbcon_radeon_bmove might copy some memory from/to a bad
> area.

I don't think so. But try commenting out setting of the font in the
init scripts and let me know if that helps.

> Unfortunately those bugs are the most difficult to find and fix.
> What I could do is disable some part of the -ben9 patch in favor of the
> -ben8 version, until I find in which part it happens.

Can you try the xmon tricks I told you so I get a disassembly
(only copy the first 2 or 3 lines) and the register dump ?

I'm not sure anything in -ben9 compared to -ben8 explains the
problem if using -ben8's radeonfb in -ben9 still dies. I think
something get corrupt in struct display, it would be useful to
figure out what.

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: ibook2r2 & strange freeze.

[Brice Figureau]

On Saturday, April 26, 2003, at 11:01 PM, Benjamin Herrenschmidt wrote:
>> The pc was in fbcon_radeon_bmove, so even if it's a fbcon problem, the
>> crash is happening into the radeonfb module...
>> video=ofonly works fine because it does not use fbcon_radeon_bmove I
>> guess...
>
> Well... The only way I see fbcon_radeon_bmove crash this way
> would be because of a corrupt struct display.
>
>>> So there is probably some memory corruption going on... I'm comparing
>>> other bits of ben8 and ben9 now and see nothing relevants. The
>>> changes
>>> where in the cache flush affecting cpufreq and sleep, and radeonfb...
>> Since the crash occurs in the scoll, if there is a problem with the
>> font (corrupted font or something) the scroll computation might be
>> wrong, and thus fbcon_radeon_bmove might copy some memory from/to a
>> bad
>> area.
>
> I don't think so. But try commenting out setting of the font in the
> init scripts and let me know if that helps.
I already did that the first time with ben10, before writing to this
list, and usually it deferred the problem. It then froze randomly while
using the computer (usually a few minutes after using it). I'll try
again soon.

>> Unfortunately those bugs are the most difficult to find and fix.
>> What I could do is disable some part of the -ben9 patch in favor of
>> the
>> -ben8 version, until I find in which part it happens.
>
> Can you try the xmon tricks I told you so I get a disassembly
> (only copy the first 2 or 3 lines) and the register dump
I copied 2 lines before pc and several after so that you can find the
exact part in the code.
disassembly:
c0184554 380b0e40 addi	r0,r11,3640
c0184558 7c00042c lwbrx	r0,r0,r0
c018455c 0c000000 twi	0,r0,0   <---- pc
c0184560 4c00012c isync
c0184564 2c000000 cmpwi r0,0
c0184568 39290001 addi	r9,r9,1
c018456c 7ce00026 mfcr	r7
c0184570 7f895000 cmpw	cr7,r9,r10
c0184574 90c10014 stw	r7,20(r1)
c0184578 40800014 bge	0xc018458c
c018457c 409dffd8 ble	0xc0184554
...

registers:
r00=ffffffff r01=d7d85d00 r02=d7d84000 r03=001e847f
r04=00000010 r05=00000000 r06=00000000 r07=84222888
r08=001e847f r09=000005ac r10=001e847f r11=d9000000
r12=c0930000 r13=100c50bc r14=c0345f04 r15=d7d85e58
r16=00000001 r17=00000000 r18=c0345f04 r19=000000ff
r20=00000001 r21=ffffffff r22=00000000 r23=c19bc100
r24=00000001 r25=00000000 r26=00000030 r27=00000000
r28=44222888 r29=00000003 r30=00000023 r31=000002f0
pc =c018455c msr=00049030 lr =c0178f3e crc=84222888
ctr=c0184394 xcr=00000000 trap=200

This was copied by hand, so I hope there is no errors in the
transcription.
Do you know if it's possible to obtain this trace on another computer
(like with a serial console) on computers without serial port ?

> I'm not sure anything in -ben9 compared to -ben8 explains the
> problem if using -ben8's radeonfb in -ben9 still dies. I think
> something get corrupt in struct display, it would be useful to
> figure out what.
I can provide you more information if you want. -ben9 crash is fully
reproducible.

Thanks,
Brice


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: ibook2r2 & strange freeze.

[Benjamin Herrenschmidt]

> disassembly:
> c0184554 380b0e40 addi	r0,r11,3640
> c0184558 7c00042c lwbrx	r0,r0,r0
> c018455c 0c000000 twi	0,r0,0   <---- pc
> c0184560 4c00012c isync
> c0184564 2c000000 cmpwi r0,0
> c0184568 39290001 addi	r9,r9,1
> c018456c 7ce00026 mfcr	r7
> c0184570 7f895000 cmpw	cr7,r9,r10
> c0184574 90c10014 stw	r7,20(r1)
> c0184578 40800014 bge	0xc018458c
> c018457c 409dffd8 ble	0xc0184554
> ...
>
> registers:
> r00=ffffffff r01=d7d85d00 r02=d7d84000 r03=001e847f
> r04=00000010 r05=00000000 r06=00000000 r07=84222888
> r08=001e847f r09=000005ac r10=001e847f r11=d9000000
> r12=c0930000 r13=100c50bc r14=c0345f04 r15=d7d85e58
> r16=00000001 r17=00000000 r18=c0345f04 r19=000000ff
> r20=00000001 r21=ffffffff r22=00000000 r23=c19bc100
> r24=00000001 r25=00000000 r26=00000030 r27=00000000
> r28=44222888 r29=00000003 r30=00000023 r31=000002f0
> pc =c018455c msr=00049030 lr =c0178f3e crc=84222888
> ctr=c0184394 xcr=00000000 trap=200
>
> This was copied by hand, so I hope there is no errors in the
> transcription.
> Do you know if it's possible to obtain this trace on another computer
> (like with a serial console) on computers without serial port ?

Hrm... Interesting. It dies reading from the chip (MMIO). Can you
check with xmon "h" command if there's a translation for 0xd9000000 ?

(Or hack radeonfb init to printk the result of ioremap of the registers
to see if it's really 0xd9000000 or if the structure was corrupted some
way)

What is the exact error in xmon ? (vector number)


Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: ibook2r2 & strange freeze.

[Brice Figureau]

On Sunday, April 27, 2003, at 12:13 AM, Benjamin Herrenschmidt wrote:
>> disassembly:
>> c0184554 380b0e40 addi	r0,r11,3640
>> c0184558 7c00042c lwbrx	r0,r0,r0
>> c018455c 0c000000 twi	0,r0,0   <---- pc
>> c0184560 4c00012c isync
> [snip]

> Hrm... Interesting. It dies reading from the chip (MMIO). Can you
> check with xmon "h" command if there's a translation for 0xd9000000 ?
I'll do that at my next reboot.

> (Or hack radeonfb init to printk the result of ioremap of the registers
> to see if it's really 0xd9000000 or if the structure was corrupted some
> way)
I'll try to find where is the init.

>
> What is the exact error in xmon ? (vector number)
Vector 200

Brice


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: ibook2r2 & strange freeze.

[Benjamin Herrenschmidt]

On Sun, 2003-04-27 at 00:32, Brice Figureau wrote:
> On Sunday, April 27, 2003, at 12:13 AM, Benjamin Herrenschmidt wrote:
> >> disassembly:
> >> c0184554 380b0e40 addi	r0,r11,3640
> >> c0184558 7c00042c lwbrx	r0,r0,r0
> >> c018455c 0c000000 twi	0,r0,0   <---- pc
> >> c0184560 4c00012c isync
> > [snip]
>
> > Hrm... Interesting. It dies reading from the chip (MMIO). Can you
> > check with xmon "h" command if there's a translation for 0xd9000000 ?
> I'll do that at my next reboot.
>
> > (Or hack radeonfb init to printk the result of ioremap of the registers
> > to see if it's really 0xd9000000 or if the structure was corrupted some
> > way)
> I'll try to find where is the init.
>
> >
> > What is the exact error in xmon ? (vector number)
> Vector 200

Ah, that's a machine check... interesting. So either the address above
is bogus, or the card is crashed. In this later case, noaccel should
have helped, except if, for some reason, the argument was ignored
when setting the initial mode...


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: ibook2r2 & strange freeze.

[Brice Figureau]

On Sunday, April 27, 2003, at 12:40 AM, Benjamin Herrenschmidt wrote:
> On Sun, 2003-04-27 at 00:32, Brice Figureau wrote:
>> On Sunday, April 27, 2003, at 12:13 AM, Benjamin Herrenschmidt wrote:
>>>> disassembly:
>>>> c0184554 380b0e40 addi	r0,r11,3640
>>>> c0184558 7c00042c lwbrx	r0,r0,r0
>>>> c018455c 0c000000 twi	0,r0,0   <---- pc
>>>> c0184560 4c00012c isync
>>> [snip]
>>
>>> Hrm... Interesting. It dies reading from the chip (MMIO). Can you
>>> check with xmon "h" command if there's a translation for 0xd9000000 ?
>> I'll do that at my next reboot.
There was a translation for d9000000 (sorry, I don't remember wich
address it was).

>>> (Or hack radeonfb init to printk the result of ioremap of the
>>> registers
>>> to see if it's really 0xd9000000 or if the structure was corrupted
>>> some
>>> way)
>> I'll try to find where is the init.
The MMIO ioremap gave 0xd9000000.
There is also another ioremap (I can't remember for which member of the
struct), it gave 0xd9005000.

>>>
>>> What is the exact error in xmon ? (vector number)
>> Vector 200
>
> Ah, that's a machine check... interesting. So either the address above
> is bogus, or the card is crashed. In this later case, noaccel should
> have helped, except if, for some reason, the argument was ignored
> when setting the initial mode...
Since the address does not seem to be bogus, could it be possible that
my computer is broken in a certain way ? (bad memory...).
I'll do another noaccel check.

Brice


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: ibook2r2 & strange freeze.

[Benjamin Herrenschmidt]

> > Ah, that's a machine check... interesting. So either the address above
> > is bogus, or the card is crashed. In this later case, noaccel should
> > have helped, except if, for some reason, the argument was ignored
> > when setting the initial mode...
> Since the address does not seem to be bogus, could it be possible that
> my computer is broken in a certain way ? (bad memory...).
> I'll do another noaccel check.

Well, it's weird... It seems the ATI chip isn't anwsering on the bus on
a specific access.

Are you sure of that specific bit of disassembly ?

c0184554 380b0e40 addi  r0,r11,3640

Is't it 3648 instead ? (the opcode seem to indicate that). In this case,
the problem is that we are getting a machine check in radeon_fifo_wait()
while reading RBBM_STATUS register.

I have absolutely no idea what can cause this (but bad HW ?) in xmon,
if you first check out r0 (it may not be d9000000 again ...) and then
do "m d9000000.l" (or replace d9000000 by whatever it has become) and
finally tell me if you atually read a value or not.

I completely fail to understand why a problem like this would happen
with -ben9 and not -ben8. Are you absolutely sure you properly tried
-ben9 with the radeonfb from -ben8 ?

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: ibook2r2 & strange freeze.

[Brice Figureau]

On Sunday, April 27, 2003, at 01:14 AM, Benjamin Herrenschmidt wrote:
>>> Ah, that's a machine check... interesting. So either the address
>>> above
>>> is bogus, or the card is crashed. In this later case, noaccel should
>>> have helped, except if, for some reason, the argument was ignored
>>> when setting the initial mode...
>> Since the address does not seem to be bogus, could it be possible that
>> my computer is broken in a certain way ? (bad memory...).
>> I'll do another noaccel check.
>
> Well, it's weird... It seems the ATI chip isn't anwsering on the bus on
> a specific access.
>
> Are you sure of that specific bit of disassembly ?
>
> c0184554 380b0e40 addi  r0,r11,3640
>
> Is't it 3648 instead ? (the opcode seem to indicate that). In this
> case,
> the problem is that we are getting a machine check in
> radeon_fifo_wait()
> while reading RBBM_STATUS register.
You were right, that was 3648, it was late yesterday when I copied
that...

> I have absolutely no idea what can cause this (but bad HW ?) in xmon,
> if you first check out r0 (it may not be d9000000 again ...) and then
> do "m d9000000.l" (or replace d9000000 by whatever it has become) and
> finally tell me if you atually read a value or not.
The address seems to exists (and it didn't change):
m d9000000.l gave:
	00030000
	6b000a00
	88030000
...
I don't think my hardware is bad since I run macosx and ben8 without
any problem (XFree86 is working fine).

> I completely fail to understand why a problem like this would happen
> with -ben9 and not -ben8. Are you absolutely sure you properly tried
> -ben9 with the radeonfb from -ben8 ?
Yes.

I'm now sure that the problem does not come from radeonfb !
To proove it, I took -ben8 and backported -ben9 changes one after one
into it, until I got the same crash as before.
It began to crash when I backported:
	 cputable.c
	 head.S
	 cputable.h

As I'm not really good at powerpc assembly, I'll let you find exactly
where the problem is ;-)

Brice


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/