2.6.12.2 dies after 24 hours

["Rob Mueller" <robm@fastmail.fm>]

As background, we've been using a relatively old kernel (2.6.4-mm2) on some 
IBM x235 machines with 6G of RAM, umem cards, and serveraid storage. These 
machines are under continuous heavy-ish load, load avg between about 1 and 
5, with between 2500-3500 procs at all times, with several largish ReiserFS 
partitions and have been running *really* well with >250 days uptime on one 
machine.

We recently tried upgrading one of the machines to the latest kernel 
(2.6.12.2) and it's died after about 24 hours. It seemed to end up in some 
weird state where we could ssh into it, and some commands worked (eg uptime) 
but process list related commands (ps) would just freeze up into an 
unkillable state and we'd have to close the seesion and ssh in again.

I did manage to get a full sysrq-t dump. I've placed it, a kernel config 
dump, and a dmesg boot dump here:

http://robm.fastmail.fm/kernel/t7/

Hope this provides some useful data to track down the problem.

Rob

PS. Yes, I know this is a non-PAE kernel on a 6G machine so 2G was unused. 
This was a mistake in this case, but it still doesn't explain the crash...