Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

Harry Edmon
2006-06-17T23:23:46Z

I assume you are talking about using TCP_NODELAY as a socket option within the
LDM software. I could give that a try.

There is a lot of traffic on this node, on the order of 2000 packets in and out
per second, so the tcpdump output will grow pretty fast. How long a tcpdump
would be useful, and what options would you suggest?

I should also note that my network interfaces are Intel, using the latest e1000
driver.

Andrew Morton wrote:
> On Fri, 16 Jun 2006 09:01:23 -0700
> Harry Edmon <harry@atmos.washington.edu> wrote:
>
>> I have a system with a strange network performance degradation from
>> 2.6.11.12 to most recent kernels including 2.6.16.20 and 2.6.17-rc6.
>> The system is has Dual single core Xeons with hyperthreading on. The
>> application is the LDM system from UCAR/Unidata
>> (http://www.unidata.ucar.edu/software/ldm). This system requests
>> weather data from a variety of systems using RPC calls over a reserved
>> TCP port (388), puts them into a memory mapped queue file, and then
>> sends the data out to a variety of downstream requesting systems, again
>> using RPC calls. When the load is heavy, the 2.6.16.20 kernel falls way
>> behind with the data ingestion. The 2.6.11.12 kernel does not. I have
>> tried an experiment with a 2.6.17-rc6 system where it just does the
>> ingestion, and not the downstream distribution, and it is able to keep
>> up. I would really appreciate any pointers as to where the problem may
>> be and how to diagnose it. I have attached the config files from both
>> kernels and the sysctl.conf file I am using. I have also included the
>> output from "netstat -s" on the 2.6.16.20 system during a time when it
>> was having problems.
>>
>
> (added netdev)
>
> A quick grep indicates that it isn't using TCP_NODELAY - we've had problems
> with that in the past.
>
> Perhaps a tcpdump of the net traffic will help to determine what's going on.

--
Dr. Harry Edmon E-MAIL: harry@atmos.washington.edu
206-543-0547 harry@u.washington.edu
Dept of Atmospheric Sciences FAX: 206-543-0308
University of Washington, Box 351640, Seattle, WA 98195-1640