2.4.4 kernel freeze for unknown reason

2.4.4 kernel freeze for unknown reason

[Jacky Liu]

Dear All,
 
I would like to post a question regarding to a problem of unknown freeze of my linux firewall/gateway.
 
Here is the hardware configuration of my machine:
 
AMD K-6 233 MHz
2theMax P-55 VP3 mobo
64Mb RAM in a single module (PC-100)
Maxtor 6G UDMA-33 harddisk
Matrox MG-II display card w/8Mb RAM
3Com 3C905B-TX NIC
RealTek 8129 10/100 NIC
 
It's running 2.4.4 kernel (RedHat 7.1) and acting as a firewall using Netfilter (gShield and Snort), DNS (Cache-Only DNS) and NAT gateway (ip-masq.) for my home network. I used 3C905B-TX NIC as my internal NIC and RealTek 8129 as my external NIC. Here is the problem:
 
The machine has been randomly lockup (totally freeze) for number of times without any traceable clue or error message. Usually the time frame between each lockup is between 24 to 72 hours. The screen just freeze when it's lockup (either in Console or X) and no "kernel panic" type or any error message prompt up. All services (SSH, DNS, etc..) are dead when it's lockup and I cannot find any useful information in /var/log/messages. I cannot reproduce the lockup since it's totally randomly. The lockup happened either when I was playing online game (A LOT, like getting thousands of server status in counter-strike in a very short time frame, of NAT traffic), surfing the web (normal traffic) or the machine was totally in idle (lockup when I was sleeping). It was lockup this morning when I was playing online game (when my game machine was trying to establish connection to a game server).
 
If there is any information you would like to obtain, please let me know. I would like to receive a copy of your reply, thank you very much for your kindly attention.
 
Best Regards,
Jacky Liu
 
"Genius or Wacko? Majority or Minority… "

------------------------------------------------------------
--== Sent via Deja.com ==--
http://www.deja.com/

Re: 2.4.4 kernel freeze for unknown reason

[Jacky Liu]

Hi Mark,

I think you pin-pointed one of the possible reason of the unknown freeze.. which is FB mode. Yes, I am using FB mode.. hm.. I will go back to recompile my kernel without FB mode and see whether this method can fix my problem or not.

You are right for the other assumption, I am running the harddisk in UDMA w/32bits mode. Are you suggesting me to turn off both functions? But if I turn them off, the performance will decrease alot.

Is there any way I can get any crash information? e.g. any function I can turn on for logging or something?

Thanks for your reply..

Best Regards,
Jacky Liu

"Genius or Wacko? Majority or Minority?"


>Date: Thu, 10 May 2001 10:24:30 -0400 (EDT)
>From: Mark Hahn <hahn@coffee.psychology.mcmaster.ca>
>To: Jacky Liu <jq419@my-deja.com>
>SUBJECT
>> I would like to post a question regarding to a problem of unknown freeze of my linux firewall/gateway.
>
>since it's a gateway/firewall, is it safe to assume you're not running
>the video in graphics or framebuffer mode?  there were plenty of problems
>on machines like this the chipset never releasing the PCI bus from
>ownership by the video card.  the result was a hang, of course.
>
>is it also safe to assume that you have the disk running in dma/udma mode?
>(not just that it's a udma33-capable!)
>
>
>>  
>> Here is the hardware configuration of my machine:
>>  
>> AMD K-6 233 MHz
>> 2theMax P-55 VP3 mobo
>> 64Mb RAM in a single module (PC-100)
>> Maxtor 6G UDMA-33 harddisk
>> Matrox MG-II display card w/8Mb RAM
>> 3Com 3C905B-TX NIC
>> RealTek 8129 10/100 NIC
>>  
>> It's running 2.4.4 kernel (RedHat 7.1) and acting as a firewall using Netfilter (gShield and Snort), DNS (Cache-Only DNS) and NAT gateway (ip-masq.) for my home network. I used 3C905B-TX NIC as my internal NIC and RealTek 8129 as my external NIC. Here is the problem:
>>  
>> The machine has been randomly lockup (totally freeze) for number of times without any traceable clue or error message. Usually the time frame between each lockup is between 24 to 72 hours. The screen just freeze when it's lockup (either in Console or X) and no "kernel panic" type or any error message prompt up. All services (SSH, DNS, etc..) are dead when it's lockup and I cannot find any useful information in /var/log/messages. I cannot reproduce the lockup since it's totally randomly. The lockup happened either when I was playing online game (A LOT, like getting thousands of server status in counter-strike in a very short time frame, of NAT traffic), surfing the web (normal traffic) or the machine was totally in idle (lockup when I was sleeping). It was lockup this morning when I was playing online game (when my game machine was trying to establish connection to a game server).
>>  
>> If there is any information you would like to obtain, please let me know. I would like to receive a copy of your reply, thank you very much for your kindly attention.
>>  
>> Best Regards,
>> Jacky Liu
>>  
>> "Genius or Wacko? Majority or Minority?"
>> 
>> ------------------------------------------------------------
>> --== Sent via Deja.com ==--
>> http://www.deja.com/
>> -
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at  http://www.tux.org/lkml/
>> 
>
>-- 
>operator may differ from spokesperson.                 hahn@coffee.mcmaster.ca
>                                              http://java.mcmaster.ca/~hahn


------------------------------------------------------------
--== Sent via Deja.com ==--
http://www.deja.com/

Re: 2.4.4 kernel freeze for unknown reason

[Vincent Stemen]

On Wednesday 09 May 2001 22:57, Jacky Liu wrote:
> Dear All,
>
> I would like to post a question regarding to a problem of unknown freeze of
> my linux firewall/gateway.
>
> Here is the hardware configuration of my machine:
>
> AMD K-6 233 MHz
> 2theMax P-55 VP3 mobo
> 64Mb RAM in a single module (PC-100)
> Maxtor 6G UDMA-33 harddisk
> Matrox MG-II display card w/8Mb RAM
> 3Com 3C905B-TX NIC
> RealTek 8129 10/100 NIC
>
> It's running 2.4.4 kernel (RedHat 7.1) and acting as a firewall using
> Netfilter (gShield and Snort), DNS (Cache-Only DNS) and NAT gateway
> (ip-masq.) for my home network. I used 3C905B-TX NIC as my internal NIC and
> RealTek 8129 as my external NIC. Here is the problem:
>
> The machine has been randomly lockup (totally freeze) for number of times
> without any traceable clue or error message. Usually the time frame between
> each lockup is between 24 to 72 hours. The screen just freeze when it's
> lockup (either in Console or X) and no "kernel panic" type or any error
> message prompt up. All services (SSH, DNS, etc..) are dead when it's lockup
> and I cannot find any useful information in /var/log/messages. I cannot
> reproduce the lockup since it's totally randomly. The lockup happened
> either when I was playing online game (A LOT, like getting thousands of
> server status in counter-strike in a very short time frame, of NAT
> traffic), surfing the web (normal traffic) or the machine was totally in
> idle (lockup when I was sleeping). It was lockup this morning when I was
> playing online game (when my game machine was trying to establish
> connection to a game server).
>
> If there is any information you would like to obtain, please let me know. I
> would like to receive a copy of your reply, thank you very much for your
> kindly attention.
>

I have been experiencing these same problems since version 2.4.0.
Although, I think it has improved a little in 2.4.4, it still locks
up.  The problem seems to be related to memory management and/or swap,
and is seems to do it primarily on machines with over 128Mb of RAM.
Although, I have not tested systematically enough to confirm this.

I have been monitoring the memory usage constantly with the gnome
memory usage meter and noticed that as swap grows it is never freed
back up.  I can kill off most of the large applications, such as
netscape, xemacs, etc, and little or no memory and swap will be freed.
Once swap is full after a few days, my machine will lock up.

If I turn swap off all together or turn it off and back on
periodically to clear the swap before it gets full, I do not seem to
experience the lockups.

I am running on an AMD K6-400 with 256 Mb RAM but we have experienced
these problems with various other machines as well.

I hope this information is helpfull in tracking down the problem.


The features of the 2.4.x kernels are great and I believe all the
kernel developers have done a fantastic job!  However, I am
disappointed that we are now on the forth 2.4.x kernel version and
such as serious problem that has been there since 2.4.0 still exists.
This is pretty much a show stopper for having a production machine.
In the past when I was running on 2.0.3x kernels, I boasted to
everybody about how rock solid Linux was and I was able to run for 5
months without a single reboot or problem, but for the last year or
more I have not had much better uptime than certain other commercial
operating systems.  As important as the new features are, stability
should be top priority for production systems.  I apologize if this
sounds a bit like a rant but I feel that ever since the 2.2.x kernel
series, the Linux kernel community has veered away from it's original
philosophy of only releasing stable kernels with even numbered
versions.  In fact, I have seen several even numbered kernels released
with far less stability than most of the odd numbered development
kernels.

- Vincent 

Re: 2.4.4 kernel freeze for unknown reason

[Alan Cox]

> I have been monitoring the memory usage constantly with the gnome
> memory usage meter and noticed that as swap grows it is never freed
> back up.  I can kill off most of the large applications, such as

The swap handling in 2.4 is somewhat hosed at the moment.

> If I turn swap off all together or turn it off and back on
> periodically to clear the swap before it gets full, I do not seem to
> experience the lockups.

That sounds right. I can give you a tiny patch that should fix the lockups
and instead it will kill processes out of memory but thats obviously not
the actual fix 8)

Re: 2.4.4 kernel freeze for unknown reason

[Mike Galbraith]

On Fri, 11 May 2001, Alan Cox wrote:

> > I have been monitoring the memory usage constantly with the gnome
> > memory usage meter and noticed that as swap grows it is never freed
> > back up.  I can kill off most of the large applications, such as

I've seen this mentioned a few times now and am curious enough to ask.

> The swap handling in 2.4 is somewhat hosed at the moment.
>
> > If I turn swap off all together or turn it off and back on
> > periodically to clear the swap before it gets full, I do not seem to
> > experience the lockups.

Why do I not see this behavior with a heavy swap throughput test load?
It seems decidedly odd to me that swapspace should remain allocated on
other folks lightly loaded boxen given that my heavily loaded box does
release swapspace quite regularly.  What am I missing?

	-Mike

Re: 2.4.4 kernel freeze for unknown reason

[Alan Cox]

> > > If I turn swap off all together or turn it off and back on
> > > periodically to clear the swap before it gets full, I do not seem to
> > > experience the lockups.
> 
> Why do I not see this behavior with a heavy swap throughput test load?
> It seems decidedly odd to me that swapspace should remain allocated on
> other folks lightly loaded boxen given that my heavily loaded box does
> release swapspace quite regularly.  What am I missing?

If you swap really hard it seems much happier. If you vaguely swap stuff out 
over time then I too see the description above only I have Rik's dont deadlock
on oom tweak so I see apps die.

Alan

Re: 2.4.4 kernel freeze for unknown reason

[Mike Galbraith]

On Sat, 12 May 2001, Alan Cox wrote:

> > > > If I turn swap off all together or turn it off and back on
> > > > periodically to clear the swap before it gets full, I do not seem to
> > > > experience the lockups.
> >
> > Why do I not see this behavior with a heavy swap throughput test load?
> > It seems decidedly odd to me that swapspace should remain allocated on
> > other folks lightly loaded boxen given that my heavily loaded box does
> > release swapspace quite regularly.  What am I missing?
>
> If you swap really hard it seems much happier. If you vaguely swap stuff out
> over time then I too see the description above only I have Rik's dont deadlock
> on oom tweak so I see apps die.

Does any swap write/release if you hit such a box with heavy duty IO?
(pages on dirty list, swapspace allocated but writeout defered?)

If not, I'd be interested in seeing sysrq-m of a box in such a state..
particularly so if the total pages on active, dirty and clean lists is
only a small fraction of total pages.  (information leak?)

	-Mike

Re: 2.4.4 kernel freeze for unknown reason

[Alan Cox]

> Does any swap write/release if you hit such a box with heavy duty IO?
> (pages on dirty list, swapspace allocated but writeout defered?)

Hard to tell. I switched my desktop box back to 2.2 a while back
until the VM works. 

Alan

Re: 2.4.4 kernel freeze for unknown reason

[Mike Galbraith]

On Sat, 12 May 2001, Alan Cox wrote:

> > Does any swap write/release if you hit such a box with heavy duty IO?
> > (pages on dirty list, swapspace allocated but writeout defered?)
>
> Hard to tell. I switched my desktop box back to 2.2 a while back
> until the VM works.

I should have reversed to/cc.. my question was intended for the list.

	-Mike

Re: 2.4.4 kernel freeze for unknown reason

[Vincent Stemen]

On Friday 11 May 2001 13:46, Alan Cox wrote:
> > I have been monitoring the memory usage constantly with the gnome
> > memory usage meter and noticed that as swap grows it is never freed
> > back up.  I can kill off most of the large applications, such as
>
> The swap handling in 2.4 is somewhat hosed at the moment.
>
> > If I turn swap off all together or turn it off and back on
> > periodically to clear the swap before it gets full, I do not seem to
> > experience the lockups.
>
> That sounds right. I can give you a tiny patch that should fix the
> lockups and instead it will kill processes out of memory but thats
> obviously not the actual fix 8)

I am not sure if you got my reply to this Alan, but I would like to
have you send me the patch.  Thanks.

Also, I got to thinking back and it dawned on me that there was
another significant kernel change when we started experiencing the
freezes from the vm problems.  We changed the compiler from
gcc-2.7.2.3 to egcs-1.1.2 starting with kernel 2.4.0-test10 (That is
the version in which gcc-2.7.2.3 stopped being supported for the
kernel) and it seems like that was just about the same time frame that
we started experiencing the vm/swap problems.  I would have to go back
and run on the test10 kernel for a while to confirm if it started with
it or 2.4.0.

I am assuming the source of the vm problem is still unknown or it
would have been fixed by now.  Is it possible that it could be related
to the compiler change?  If nobody has considered that, I just thought
I would bring the possibility to your attention.  

- Vincent Stemen

Re: 2.4.4 kernel freeze for unknown reason

[Christoph Rohland]

Hi Mike,

On Sat, 12 May 2001, Mike Galbraith wrote:
> Why do I not see this behavior with a heavy swap throughput test
> load?  It seems decidedly odd to me that swapspace should remain
> allocated on other folks lightly loaded boxen given that my heavily
> loaded box does release swapspace quite regularly.  What am I
> missing?

Are you using a database or something other which mostly uses shared
mem/tmpfs? This does reclaim swap space on swap in.

Greetings
		Christoph

Re: 2.4.4 kernel freeze for unknown reason

[Mike Galbraith]

On 13 May 2001, Christoph Rohland wrote:

> Hi Mike,
>
> On Sat, 12 May 2001, Mike Galbraith wrote:
> > Why do I not see this behavior with a heavy swap throughput test
> > load?  It seems decidedly odd to me that swapspace should remain
> > allocated on other folks lightly loaded boxen given that my heavily
> > loaded box does release swapspace quite regularly.  What am I
> > missing?
>
> Are you using a database or something other which mostly uses shared
> mem/tmpfs? This does reclaim swap space on swap in.

No, just doing parallel kernel builds.

	-Mike

Re: 2.4.4 kernel freeze for unknown reason

[Jacky Liu]

Hi everyone,

Mark, I got your point about the dma/udma stuffs. My hdparm setting is UDMA w/ MultiSector 16..

I had recompiled my kernel and disabled the FB option but my linux box still hanged (another completely freeze) yesterday... Oh well..

I have been tracking this thread for a few days and it seem the source of this problem is related to swap space. Vincent, would you mind to send me the patch for swap space problem if Alan had sent it to you? So I can test it on my machine and report the result later.

Mark, please suggest a setting for the hdparm so I can test it on my machine. Thanks alot for your time.

Best Regards,
Jacky Liu


>Date: Thu, 10 May 2001 23:20:17 -0400 (EDT)
>From: Mark Hahn <hahn@coffee.psychology.mcmaster.ca>
>To: Jacky Liu <jq419@my-deja.com>
>SUBJECT
>> You are right for the other assumption, I am running the harddisk in UDMA w/32bits mode. Are you suggesting me to turn off both functions? But if I turn them off, the performance will decrease alot.
>
>the only thing that matters is dma/udma or not.  32b mode is irrelevant
>to the actual dma/udma transfer, as is -m settings.  even -u has no
>measurable effect.
>
>> Is there any way I can get any crash information? e.g. any function I can turn on for logging or something?
>
>if your hang is as I'm imagining, there's nothing you can do, since it's
>purely hardware.  you could try compiling in magic sysrq, but I suspect 
>the hardware is hung.




------------------------------------------------------------
--== Sent via Deja.com ==--
http://www.deja.com/

Re: 2.4.4 kernel freeze for unknown reason

[Vincent Stemen]

On Tuesday 15 May 2001 02:13, Jacky Liu wrote:
> Hi everyone,
>
> Mark, I got your point about the dma/udma stuffs. My hdparm setting is
> UDMA w/ MultiSector 16..
>
> I had recompiled my kernel and disabled the FB option but my linux box
> still hanged (another completely freeze) yesterday... Oh well..
>
> I have been tracking this thread for a few days and it seem the source of
> this problem is related to swap space. Vincent, would you mind to send me
> the patch for swap space problem if Alan had sent it to you? So I can
> test it on my machine and report the result later.
>

I havn't received the patch but, on Byron Stanoszek's suggestion, I
have compiled 2.4.4 with gcc-2.95.3 to run on it a few days to see if
it is any better.  So far, it appears at least as stable as with
egcs-1.1.2 and most of the swap was freed up this morning for the
first time since 2.4.0.  However, it is pretty full tonight.  I should
know tomorrow if it really made a difference.  This is usually the
state in which I find it locked up the next morning.  It has been up a
little over 2 days now.  Byron actually suggested I use the ac7 patch
but I first wanted to see if the compiler had anything to do with it
without changing anything else.

> Mark, please suggest a setting for the hdparm so I can test it on my
> machine. Thanks alot for your time.
>
> Best Regards,
> Jacky Liu
>

Re: 2.4.4 kernel freeze for unknown reason

[Vasco Figueira]

Hi all,

On Fri May 11 2001 - 13:45:24 EST, Vincent Stemen 
(linuxkernel@AdvancedResearch.org) wrote:

 >On Wednesday 09 May 2001 22:57, Jacky Liu wrote:

 >> The machine has been randomly lockup (totally freeze) for number of
 >> times without any traceable clue or error message. Usually the time
 >> frame between each lockup is between 24 to 72 hours. The screen just
 >> freeze when it's lockup (either in Console or X) and no "kernel >panic"
 >> type or any error message prompt up. All services (SSH, DNS, etc..)
 >> are dead when it's lockup
(...)

 >I have been experiencing these same problems since version 2.4.0.
 >Although, I think it has improved a little in 2.4.4, it still locks
 >up. The problem seems to be related to memory management and/or swap,
 >and is seems to do it primarily on machines with over 128Mb of RAM.
 >Although, I have not tested systematically enough to confirm this.

I have the same problem on a Toshiba satellite 4070, 366 celeron, 64M 
ram, redhat 7.1 and vanilla 2.4.5. Exactly the same bug description. 
Totally reproducible.

 >I have been monitoring the memory usage constantly with the gnome
 >memory usage meter and noticed that as swap grows it is never freed
 >back up. I can kill off most of the large applications, such as
 >netscape, xemacs, etc, and little or no memory and swap will be freed.
 >Once swap is full after a few days, my machine will lock up.

After a few *hours*.

Then I have (as you said) to do swapoff /dev/hda4 ; swapon -a in order 
to free the swap. If I do this, everything is fine... till it fills up 
again.

 >(...)I am
 >disappointed that we are now on the forth 2.4.x kernel version and
 >such as serious problem that has been there since 2.4.0 still exists.
 >This is pretty much a show stopper for having a production machine.

Totally agree. This is quite a showstopper. Do_try_to_free_pages, err... 
sorry, fix_bug.

Regards,

Vasco Figueira