Re: Oops after 30 days of uptime

Willy Tarreau
2006-09-10T08:31:22Z

Hi Ondrej,

OK, I've analysed your oops with your kernel. My conclusions are that you
have a hardware problem (most probably the CPU), because you've hit an
impossible case :

ip_nat_cheat_check() pushed the size of the data (8) on the stack, followed
by the pointer to the data, then called csum_partial() :

c01e657f: 6a 08 push $0x8
c01e6581: 52 push %edx
c01e6582: e8 a5 85 00 00 call c01eeb2c <csum_partial>

In csum_partial(), ECX is filled with the size (8) and ESI with the data
pointer (0xc0227ce8) :

c01eeb32: 8b 4c 24 10 mov 0x10(%esp),%ecx
c01eeb36: 8b 74 24 0c mov 0xc(%esp),%esi

Then, the size is divided by 32 to count how many 32 bytes blocks can be read
at a time. If the size is lower than 32, the code branches to a special
location which reads 1 word at a time :

c01eeb78: 89 ca mov %ecx,%edx
c01eeb7a: c1 e9 05 shr $0x5,%ecx
c01eeb7d: 74 32 je c01eebb1 <csum_partial+0x85>

Your oops comes from a few instructions below. The branch has not been taken
while it should have because (8 >> 5) == 0. You can also see from EDX in the
oops that it really was 0x8 when copied from ECX. The rest is pretty obvious.
The data are read 32 bytes at a time after ESI, and ECX is decreased by 1
every 32 bytes. When ESI+0x18 reaches an unmapped area (0xc2000000), you get
the oops, and ECX = 0xfff113e8 as in your oops.

Given that the failing instruction is the most common conditionnal jump, it
is very fortunate that your system can work 30 days before crashing. I think
that your CPU might be running too hot and might get wrong results during
branch prediction. It's also possible that you have a poor power supply.
However, I'm pretty sure that this is not a RAM problem.

Best regards,
Willy