long stalls

long stalls

[Brian Tinsley]

We have been having terrible problems with long stalls, meaning from a 
couple of minutes to an hour, happening when filesystem I/O load gets 
high. The system time as reported by vmstat or sar will increase up to 
99% and as it spreads to each procesor, the system becomes completely 
unresponsive (except that it responds to pings just fine - 
interesting!). When the system finally returns to the world of the 
living, the only evidence that something bad has happened is the runtime 
for kswapd is abnormally high. I have seen this happen with the stock 
2.4.17, 2.4.19, and 2.4.20 kernels on SMP PIII and PIV machines (either 
4GB or 8GB RAM, all SCSI disks, dual GigE NICs). I've searched the lkml 
archives and google and have found several similar postings, but there 
is never an explanation or resolution. Any help would be *very* much 
appreciated! If any info from the system in question is desired, I will 
be glad to provide it.



-- 

-[========================]-
-[      Brian Tinsley     ]-
-[ Chief Systems Engineer ]-
-[        Emageon         ]-
-[========================]-

Re: long stalls

[Russell Leighton]

I can't help, but I can echo a "me too".

We only see it when I have 2 file I/O intensive processes...they both 
will just stop for some few seconds, system seems idle...then
they just start again. RH7.3 SMP, Dual PIII, 4GB RAM, 3com RAID Controller .

Brian Tinsley wrote:

> We have been having terrible problems with long stalls, meaning from a 
> couple of minutes to an hour, happening when filesystem I/O load gets 
> high. The system time as reported by vmstat or sar will increase up to 
> 99% and as it spreads to each procesor, the system becomes completely 
> unresponsive (except that it responds to pings just fine - 
> interesting!). When the system finally returns to the world of the 
> living, the only evidence that something bad has happened is the 
> runtime for kswapd is abnormally high. I have seen this happen with 
> the stock 2.4.17, 2.4.19, and 2.4.20 kernels on SMP PIII and PIV 
> machines (either 4GB or 8GB RAM, all SCSI disks, dual GigE NICs). I've 
> searched the lkml archives and google and have found several similar 
> postings, but there is never an explanation or resolution. Any help 
> would be *very* much appreciated! If any info from the system in 
> question is desired, I will be glad to provide it.
>
>
>

Re: long stalls

[Brian Tinsley]

Out of curiosity, which RH kernel are you using? I moved on to 2.4.19 
and 2.4.20 primarily because the RH 2.4.18 series of kernels apparently 
has a scheduler bug (at least one) that causes the heartbeat software 
from Linux-HA to loose heartbeat signals and failover. Not a good 
scenario when you are trying to provide HA systems to hospitals!


Russell Leighton wrote:

>
> I can't help, but I can echo a "me too".
>
> We only see it when I have 2 file I/O intensive processes...they both 
> will just stop for some few seconds, system seems idle...then
> they just start again. RH7.3 SMP, Dual PIII, 4GB RAM, 3com RAID 
> Controller .
>
> Brian Tinsley wrote:
>
>> We have been having terrible problems with long stalls, meaning from 
>> a couple of minutes to an hour, happening when filesystem I/O load 
>> gets high. The system time as reported by vmstat or sar will increase 
>> up to 99% and as it spreads to each procesor, the system becomes 
>> completely unresponsive (except that it responds to pings just fine - 
>> interesting!). When the system finally returns to the world of the 
>> living, the only evidence that something bad has happened is the 
>> runtime for kswapd is abnormally high. I have seen this happen with 
>> the stock 2.4.17, 2.4.19, and 2.4.20 kernels on SMP PIII and PIV 
>> machines (either 4GB or 8GB RAM, all SCSI disks, dual GigE NICs). 
>> I've searched the lkml archives and google and have found several 
>> similar postings, but there is never an explanation or resolution. 
>> Any help would be *very* much appreciated! If any info from the 
>> system in question is desired, I will be glad to provide it.
>>
>>
>>
>

-- 

-[========================]-
-[      Brian Tinsley     ]-
-[ Chief Systems Engineer ]-
-[        Emageon         ]-
-[========================]-

Re: long stalls

[Brian Gerst]

Brian Tinsley wrote:

> We have been having terrible problems with long stalls, meaning from a
> couple of minutes to an hour, happening when filesystem I/O load gets
> high. The system time as reported by vmstat or sar will increase up to
> 99% and as it spreads to each procesor, the system becomes completely
> unresponsive (except that it responds to pings just fine -
> interesting!). When the system finally returns to the world of the
> living, the only evidence that something bad has happened is the runtime
> for kswapd is abnormally high. I have seen this happen with the stock
> 2.4.17, 2.4.19, and 2.4.20 kernels on SMP PIII and PIV machines (either
> 4GB or 8GB RAM, all SCSI disks, dual GigE NICs). I've searched the lkml
> archives and google and have found several similar postings, but there
> is never an explanation or resolution. Any help would be *very* much
> appreciated! If any info from the system in question is desired, I will
> be glad to provide it.
>
>
>
With 4GB of memory you are likely boucing I/O requests to low memory. 
This has been fixed in 2.5.  I do not know if a backport exists for 2.4.

--
				Brian Gerst

Re: long stalls

[Brian Tinsley]

Thanks for the reply!

I thought highmem I/O was addressed in 2.4.20? Am I off-base here?

I actually just built a 2.4.20 kernel with highmem debugging turned on. 
We'll see if anything pops up.


Brian Gerst wrote:

> Brian Tinsley wrote:
>
>> We have been having terrible problems with long stalls, meaning from a
>> couple of minutes to an hour, happening when filesystem I/O load gets
>> high. The system time as reported by vmstat or sar will increase up to
>> 99% and as it spreads to each procesor, the system becomes completely
>> unresponsive (except that it responds to pings just fine -
>> interesting!). When the system finally returns to the world of the
>> living, the only evidence that something bad has happened is the runtime
>> for kswapd is abnormally high. I have seen this happen with the stock
>> 2.4.17, 2.4.19, and 2.4.20 kernels on SMP PIII and PIV machines (either
>> 4GB or 8GB RAM, all SCSI disks, dual GigE NICs). I've searched the lkml
>> archives and google and have found several similar postings, but there
>> is never an explanation or resolution. Any help would be *very* much
>> appreciated! If any info from the system in question is desired, I will
>> be glad to provide it.
>>
>>
>>
> With 4GB of memory you are likely boucing I/O requests to low memory. 
> This has been fixed in 2.5.  I do not know if a backport exists for 2.4.
>
> -- 
>                 Brian Gerst


-- 

-[========================]-
-[      Brian Tinsley     ]-
-[ Chief Systems Engineer ]-
-[        Emageon         ]-
-[========================]-

Re: long stalls

[Russell Leighton]

Minor correction: 3ware RAID controller.

Russell Leighton wrote:

>
> I can't help, but I can echo a "me too".
>
> We only see it when I have 2 file I/O intensive processes...they both 
> will just stop for some few seconds, system seems idle...then
> they just start again. RH7.3 SMP, Dual PIII, 4GB RAM, 3com RAID 
> Controller .
>
> Brian Tinsley wrote:
>
>> We have been having terrible problems with long stalls, meaning from 
>> a couple of minutes to an hour, happening when filesystem I/O load 
>> gets high. The system time as reported by vmstat or sar will increase 
>> up to 99% and as it spreads to each procesor, the system becomes 
>> completely unresponsive (except that it responds to pings just fine - 
>> interesting!). When the system finally returns to the world of the 
>> living, the only evidence that something bad has happened is the 
>> runtime for kswapd is abnormally high. I have seen this happen with 
>> the stock 2.4.17, 2.4.19, and 2.4.20 kernels on SMP PIII and PIV 
>> machines (either 4GB or 8GB RAM, all SCSI disks, dual GigE NICs). 
>> I've searched the lkml archives and google and have found several 
>> similar postings, but there is never an explanation or resolution. 
>> Any help would be *very* much appreciated! If any info from the 
>> system in question is desired, I will be glad to provide it.
>>
>>
>>
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe 
> linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>
>

Re: long stalls

[Russell Leighton]

I am pretty sure we are at 2.4.19.

Brian Tinsley wrote:

> Out of curiosity, which RH kernel are you using? I moved on to 2.4.19 
> and 2.4.20 primarily because the RH 2.4.18 series of kernels 
> apparently has a scheduler bug (at least one) that causes the 
> heartbeat software from Linux-HA to loose heartbeat signals and 
> failover. Not a good scenario when you are trying to provide HA 
> systems to hospitals!
>
>
> Russell Leighton wrote:
>
>>
>> I can't help, but I can echo a "me too".
>>
>> We only see it when I have 2 file I/O intensive processes...they both 
>> will just stop for some few seconds, system seems idle...then
>> they just start again. RH7.3 SMP, Dual PIII, 4GB RAM, 3com RAID 
>> Controller .
>>
>> Brian Tinsley wrote:
>>
>>> We have been having terrible problems with long stalls, meaning from 
>>> a couple of minutes to an hour, happening when filesystem I/O load 
>>> gets high. The system time as reported by vmstat or sar will 
>>> increase up to 99% and as it spreads to each procesor, the system 
>>> becomes completely unresponsive (except that it responds to pings 
>>> just fine - interesting!). When the system finally returns to the 
>>> world of the living, the only evidence that something bad has 
>>> happened is the runtime for kswapd is abnormally high. I have seen 
>>> this happen with the stock 2.4.17, 2.4.19, and 2.4.20 kernels on SMP 
>>> PIII and PIV machines (either 4GB or 8GB RAM, all SCSI disks, dual 
>>> GigE NICs). I've searched the lkml archives and google and have 
>>> found several similar postings, but there is never an explanation or 
>>> resolution. Any help would be *very* much appreciated! If any info 
>>> from the system in question is desired, I will be glad to provide it.
>>>
>>>
>>>
>>
>

Re: long stalls

[Juergen Sawinski]

On Wed, 2003-01-08 at 02:51, Russell Leighton wrote:
> 
> I can't help, but I can echo a "me too".
> 
> We only see it when I have 2 file I/O intensive processes...they both 
> will just stop for some few seconds, system seems idle...then
> they just start again. RH7.3 SMP, Dual PIII, 4GB RAM, 3com RAID Controller .

Same thing here with a Promise SX6000 RAID controller (P4, 1GB RAM,
system is completely on RAID, 2.4.20-pre10-ac1). But, this seems not to
be related. At least in my case, it's the controller that causes the
stalls, 'cause only processes depending on file IO (including swap) get
into D state. Everything else just runs fine.

George

-- 
Juergen "George" Sawinski                  |  Phone:  +49-6221-486-308
Max-Planck Institute for Medical Research  |  Fax:    +49-6221-486-325
Dept. of Biomedical Optics                 |  Mobile: +49-171-532 5302
Jahnstr. 29                                |  
D-69120 Heidelberg                         |  
Germany                                    |  

GPG Key/Fingerprint: 9A5F7A31/86F2E5D5EDF4D9983BDD3F23986F154F9A5F7A31