Re: 2.6.1 IO lockup on SMP systems

[Mike Fedyk <mfedyk@matchmail.com>]

Sergey S. Kostyliov wrote:
> Hello Andrew,
> 
> Now this happens for the third time.
> 
> 
>>>>I've just reproduced this lockup with 2.6.3.
>>>>
>>>>
>>>>>You may need a serial console to be able to capture all the output.
>>>>>
>>>>>Also, it would be useful to know what sort of load the machines are
>>>>>under, and what filesystems are in use.
>>>>
>>>>The machine is a http server. The main applications are:
>>>>1) apache 1.3 which serves php pages (mod_php):
>>>>	 15.3 requests/sec - 111.9 kB/second - 7.3 kB/request
>>>>	 54 requests currently being processed, 19 idle servers
>>>>2) mysql:
>>>>	Threads: 19  Questions: 26922012  Slow queries: 9799  Opens: 64980
>>>>	Flush tables: 1  Open tables: 630  Queries per second avg: 143.547
>>>>
>>>>This is an IO bound machine in general. All filesystems are reiserfs.
>>>>
>>>>Here is a sysrq-T output obtained from a locked box via serail console:
>>>
>>>OK, so everything is stuck trying to allocate memory.  Perhaps you ran out
>>>of swapspace, or some process has gone berzerk allocating memory.
> 
> 
> The memory exhaustion is indeed possible for this box. I'll double check
> ulimit and /etc/security/limits.conf stuff. The only thing which worries
> me that this box had been running for months without any problems with
> 2.4.23aa1.
> 
> I have added another 2Gb to swap space (hope this give enough time
> to find the memory hungry process(es)).

Also check how much memory is being used for slab in /proc/meminfo