Still bug-hunting

Still bug-hunting

[Christian Schmid]

Hello.

I have found out that the slow-down issue with many sockets is a memory problem now. Not that I do 
not have enough but the vm isnt able to reclaim fast-enough. Problem temporarily solved by setting 
lower_zone_protection to 1024. Still I wonder why /proc/net/sockstat shows this:

sockets: used 1637
TCP: inuse 1761 orphan 179 tw 3743 alloc 1761 mem 54224

The send-buffer is set to 128 and the rcv-buffer to 16 so theoretically it should be 1637*144=235728 
so I assume the buffers are allocated dynamically when they are really needed... I suppose the 
slow-down wouldnt appear if the buffers are allocated statically. Is there any way to check this?

Chris

Re: Still bug-hunting

[David S. Miller]

On Sat, 05 Mar 2005 18:57:00 +0100
Christian Schmid <webmaster@rapidforum.com> wrote:

> I have found out that the slow-down issue with many sockets is a memory problem now.

Thanks for the test case we asked for.

Re: Still bug-hunting

[Christian Schmid]

You as an expert should know that this is no one-liner. It is a very complex server with 
multiplexing and forking like Apache. I dont feel like writing an apache-like program just for 
testing. I hope you understand that I prefered investing my time in acutally searching the bug and 
now its found. Now I hope that you can fix it or at least tell me your ideas.

David S. Miller wrote:
> On Sat, 05 Mar 2005 19:36:05 +0100
> Christian Schmid <webmaster@rapidforum.com> wrote:
> 
> 
>>Sorry? What do you mean?
> 
> 
> I'm being sarcastic in that you didn't provide the test
> case we asked you to provide so we could reproduce the
> many-socket slowdown too and help you figure out what
> was going wrong.
> 
> 

Re: Still bug-hunting

[David S. Miller]

On Sat, 05 Mar 2005 19:54:49 +0100
Christian Schmid <webmaster@rapidforum.com> wrote:

> You as an expert should know that this is no one-liner. It is a very complex server with 
> multiplexing and forking like Apache. I dont feel like writing an apache-like program just for 
> testing. I hope you understand that I prefered investing my time in acutally searching the bug and 
> now its found. Now I hope that you can fix it or at least tell me your ideas.

You could give us the program you actually used, no writing
necessary.  Obviously you had this program, else you wouldn't
have anything to report at all.

There's nothing "to write", you have it already by implication.

Instead, you told us "do something like this, or that" and lots
of vague statements and brief incomplete code snippets, then
expect us to figure out how the magic beans work and fix your
problem.

And you want respect and help from us? :-)

Re: Still bug-hunting

[Christian Schmid]

Actually we are a company and the big ones decided to buy new servers to work-around this bug. This 
is a common technique in companies. I say this sucks and thats why I experienced in our production 
system in order to give linux something back by reporting and helping to fix a bug which only 
appears on big systems. I do not understand why you start flaming at me.

David S. Miller wrote:
> On Sat, 05 Mar 2005 19:54:49 +0100
> Christian Schmid <webmaster@rapidforum.com> wrote:
> 
> 
>>You as an expert should know that this is no one-liner. It is a very complex server with 
>>multiplexing and forking like Apache. I dont feel like writing an apache-like program just for 
>>testing. I hope you understand that I prefered investing my time in acutally searching the bug and 
>>now its found. Now I hope that you can fix it or at least tell me your ideas.
> 
> 
> You could give us the program you actually used, no writing
> necessary.  Obviously you had this program, else you wouldn't
> have anything to report at all.
> 
> There's nothing "to write", you have it already by implication.
> 
> Instead, you told us "do something like this, or that" and lots
> of vague statements and brief incomplete code snippets, then
> expect us to figure out how the magic beans work and fix your
> problem.
> 
> And you want respect and help from us? :-)
> 
> 

Re: Still bug-hunting

[Thomas Graf]

* Christian Schmid <422A0354.3080800@rapidforum.com> 2005-03-05 20:07
> Actually we are a company and the big ones decided to buy new servers to 
> work-around this bug. This is a common technique in companies. I say this 
> sucks and thats why I experienced in our production system in order to give 
> linux something back by reporting and helping to fix a bug which only 
> appears on big systems. I do not understand why you start flaming at me.

It's quite simple, provide a test case and someone will start looking
into the problem (given there is one). The information you provided so far
is quite vague, it would be pure luck to spot the bug. You have do
understand that looking into such a bug may cost dozens of hours or days if
it can't be reproduced by the person. Nobody does this without at least very
strong evidences that the bug actually exists and your appereance so
far didn't help too much I guess.

Re: Still bug-hunting

[Christian Schmid]

Thomas Graf wrote:
> * Christian Schmid <422A0354.3080800@rapidforum.com> 2005-03-05 20:07
> 
>>Actually we are a company and the big ones decided to buy new servers to 
>>work-around this bug. This is a common technique in companies. I say this 
>>sucks and thats why I experienced in our production system in order to give 
>>linux something back by reporting and helping to fix a bug which only 
>>appears on big systems. I do not understand why you start flaming at me.
> 
> 
> It's quite simple, provide a test case and someone will start looking
> into the problem (given there is one). The information you provided so far
> is quite vague, it would be pure luck to spot the bug. You have do
> understand that looking into such a bug may cost dozens of hours or days if
> it can't be reproduced by the person. Nobody does this without at least very
> strong evidences that the bug actually exists and your appereance so
> far didn't help too much I guess.

I just asked for a way how to change the dynamic memory-allocation for buffers to a static one. I 
dont think that this is much work and is a quite direct task.

Re: Still bug-hunting

[Francois Romieu]

Christian Schmid <webmaster@rapidforum.com> :
[...]
> I just asked for a way how to change the dynamic memory-allocation for 
> buffers to a static one. I dont think that this is much work and is a quite 
> direct task.

Simple answer: rewrite alloc_skb and friends.

--
Ueimor

Re: Still bug-hunting

[Christian Schmid]

In less-rude words: "No, this is not easy. No way, sorry."

Thank you for this information. I have sent very detailed bug-reports to LKML. Maybe you have read 
already.

Francois Romieu wrote:
> Christian Schmid <webmaster@rapidforum.com> :
> [...]
> 
>>I just asked for a way how to change the dynamic memory-allocation for 
>>buffers to a static one. I dont think that this is much work and is a quite 
>>direct task.
> 
> 
> Simple answer: rewrite alloc_skb and friends.
> 
> --
> Ueimor
> 
> 

Re: Still bug-hunting

[Francois Romieu]

Christian Schmid <webmaster@rapidforum.com> :
> In less-rude words: "No, this is not easy. No way, sorry."

If you can not distinguish a factual answer from a flame, it is time
to take a break.

[...]
> Thank you for this information. I have sent very detailed bug-reports to 
> LKML. Maybe you have read already.

Just done so but I fail to see if you provided N. Piggin with the data
he requested. Anyway, I can return to more basic issues since he and
the others will do a better job than me.

--
Ueimor