Network Performance?

Network Performance?

[Tim Sailer]

This may not be the right forum to ask this. If not, please let me know
where to ask.

I have a Debian box with 2 NICs. Both 100/full duplex. This machine is 
running as a ftp proxy (T.Rex suite). As part of the traffic going through the
box, some streams have 1000k window size for a certain reason. How do
I tune the NICs to handle the streams better? There are ways of doing this
on other OSs. Right now, the box only does about 1.8Mb when it should be doing
80+Mb.

Thanks,
Tim

PS: This is really something to do with the window size and WAN latency.
The ultimate source and destination points are either Solaris or AIX
boxes. The files being sent are > 1GB in size.
The box does well when traffic goes in one NIC and out the other, as long
as the end point is local When it hits the WAN, it all dies. Traffic not
going through the box just flies right along, as long as both the end points
have the large tcp window size. Putting the Linux box in the middle is a 
severe choke point. :(


-- 
Tim Sailer <sailer@bnl.gov> Cyber Security Operations
Brookhaven National Laboratory  (631) 344-3001
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Network Performance?

[Tim Sailer]

On Thu, Jan 04, 2001 at 01:33:40AM -0500, Tim Sailer wrote:
> This may not be the right forum to ask this. If not, please let me know
> where to ask.
> 
> I have a Debian box with 2 NICs. Both 100/full duplex. This machine is 
> running as a ftp proxy (T.Rex suite). As part of the traffic going through the
> box, some streams have 1000k window size for a certain reason. How do
> I tune the NICs to handle the streams better? There are ways of doing this
> on other OSs. Right now, the box only does about 1.8Mb when it should be doing
> 80+Mb.
> 
> Thanks,
> Tim
> 
> PS: This is really something to do with the window size and WAN latency.
> The ultimate source and destination points are either Solaris or AIX
> boxes. The files being sent are > 1GB in size.
> The box does well when traffic goes in one NIC and out the other, as long
> as the end point is local When it hits the WAN, it all dies. Traffic not
> going through the box just flies right along, as long as both the end points
> have the large tcp window size. Putting the Linux box in the middle is a 
> severe choke point. :(

I have followed the suggestions in http://www.psc.edu/networking/perf_tune.html
but I still can not get any kind of real throughput. 250kB is all I can
get from the Linux box. Setting [r|w]mem_[default|max] larger than 16k
makes no difference, smaller slows things down even more. Has anyone else
ran across this and fixed it? I can't be the only one with a Linux box
on a fat pipe looking for maximum throughput...

Tim

-- 
Tim Sailer <sailer@bnl.gov> Cyber Security Operations
Brookhaven National Laboratory  (631) 344-3001
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Network Performance?

[Andrew Morton]

Tim Sailer wrote:
> 
> On Thu, Jan 04, 2001 at 01:33:40AM -0500, Tim Sailer wrote:
> > This may not be the right forum to ask this. If not, please let me know
> > where to ask.
> >
> > I have a Debian box with 2 NICs. Both 100/full duplex. This machine is
> > running as a ftp proxy (T.Rex suite). As part of the traffic going through the
> > box, some streams have 1000k window size for a certain reason. How do
> > I tune the NICs to handle the streams better? There are ways of doing this
> > on other OSs. Right now, the box only does about 1.8Mb when it should be doing
> > 80+Mb.
> >
> > Thanks,
> > Tim
> >
> > PS: This is really something to do with the window size and WAN latency.
> > The ultimate source and destination points are either Solaris or AIX
> > boxes. The files being sent are > 1GB in size.
> > The box does well when traffic goes in one NIC and out the other, as long
> > as the end point is local When it hits the WAN, it all dies. Traffic not
> > going through the box just flies right along, as long as both the end points
> > have the large tcp window size. Putting the Linux box in the middle is a
> > severe choke point. :(
> 
> I have followed the suggestions in http://www.psc.edu/networking/perf_tune.html
> but I still can not get any kind of real throughput. 250kB is all I can
> get from the Linux box. Setting [r|w]mem_[default|max] larger than 16k
> makes no difference, smaller slows things down even more. Has anyone else
> ran across this and fixed it? I can't be the only one with a Linux box
> on a fat pipe looking for maximum throughput...

Tim,

this issue was discussed on the netdev mailing list a few weeks
back.

It's very unfortunate that the web archives of netdev
stopped working several months ago and there now appears
to be no web archive of netdev@oss.sgi.com.

Go to http://oss.sgi.com/projects/netdev/archive/ and
pull down the November and December archives.

The subject was "linux to solaris tcp issues on WAN".

The conclusion was "The problem is also fixed with
2.4.0-test12pre3". Dunno about kernel 2.2 though.


-
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Network Performance?

[Alan Cox]

> The conclusion was "The problem is also fixed with
> 2.4.0-test12pre3". Dunno about kernel 2.2 though.

DaveM sent me a patch to address the problem its in 2.2.19pre3
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Network Performance?

[Tim Sailer]

On Sat, Jan 06, 2001 at 10:11:40PM +1100, Andrew Morton wrote:
> this issue was discussed on the netdev mailing list a few weeks
> back.
> 
> It's very unfortunate that the web archives of netdev
> stopped working several months ago and there now appears
> to be no web archive of netdev@oss.sgi.com.
> 
> Go to http://oss.sgi.com/projects/netdev/archive/ and
> pull down the November and December archives.
> 
> The subject was "linux to solaris tcp issues on WAN".
> 
> The conclusion was "The problem is also fixed with
> 2.4.0-test12pre3". Dunno about kernel 2.2 though.

Well, on Friday, we pulled down the 'official' 2.4.0, and had the
same experience... nothing better. Should I get the -test12-pre3 kernel
and try that one specifically?

Tim

-- 
Tim Sailer <sailer@bnl.gov> Cyber Security Operations
Brookhaven National Laboratory  (631) 344-3001
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Network Performance?

[Andrew Morton]

Tim Sailer wrote:
> 
> On Sat, Jan 06, 2001 at 10:11:40PM +1100, Andrew Morton wrote:
> > this issue was discussed on the netdev mailing list a few weeks
> > back.
> >
> > It's very unfortunate that the web archives of netdev
> > stopped working several months ago and there now appears
> > to be no web archive of netdev@oss.sgi.com.
> >
> > Go to http://oss.sgi.com/projects/netdev/archive/ and
> > pull down the November and December archives.
> >
> > The subject was "linux to solaris tcp issues on WAN".
> >
> > The conclusion was "The problem is also fixed with
> > 2.4.0-test12pre3". Dunno about kernel 2.2 though.
> 
> Well, on Friday, we pulled down the 'official' 2.4.0, and had the
> same experience... nothing better. Should I get the -test12-pre3 kernel
> and try that one specifically?

I doubt if that would help.

I claim no expertise in this area, but perhaps we can
get some protocol gurus interested.

To recap:

You're sending and receiving FTP/TCP/IP4 to Solaris and AIX hosts
You have a 1000kbyte window size
You have an 80 megabit/sec pipe.
You're getting 1.8 megabits/sec.

What is the round-trip time on the WAN?

Packet loss?

Does the problem occur in both directions?

Are you _sure_ the window size is being set correctly? How
is it being set?

Are you able to generate TCP dumps when the problem is happening?

-
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Network Performance?

[Tim Sailer]

On Mon, Jan 08, 2001 at 09:26:23PM +1100, Andrew Morton wrote:
> Tim Sailer wrote:
> > 
> > On Sat, Jan 06, 2001 at 10:11:40PM +1100, Andrew Morton wrote:
> > > this issue was discussed on the netdev mailing list a few weeks
> > > back.
> > >
> > > It's very unfortunate that the web archives of netdev
> > > stopped working several months ago and there now appears
> > > to be no web archive of netdev@oss.sgi.com.
> > >
> > > Go to http://oss.sgi.com/projects/netdev/archive/ and
> > > pull down the November and December archives.
> > >
> > > The subject was "linux to solaris tcp issues on WAN".
> > >
> > > The conclusion was "The problem is also fixed with
> > > 2.4.0-test12pre3". Dunno about kernel 2.2 though.
> > 
> > Well, on Friday, we pulled down the 'official' 2.4.0, and had the
> > same experience... nothing better. Should I get the -test12-pre3 kernel
> > and try that one specifically?
> 
> I doubt if that would help.
> 
> I claim no expertise in this area, but perhaps we can
> get some protocol gurus interested.
> 
> To recap:
> 
> You're sending and receiving FTP/TCP/IP4 to Solaris and AIX hosts

Yup

> You have a 1000kbyte window size

Yup

> You have an 80 megabit/sec pipe.

Actually, 100 to the router, and then the WAN connection to the remote
system is OC-3

> You're getting 1.8 megabits/sec.

Yup

> What is the round-trip time on the WAN?
> 
> Packet loss?

101 packets transmitted, 101 packets received, 0% packet loss
round-trip min/avg/max = 109.6/110.3/112.2 ms

> Does the problem occur in both directions?

Good question. I'll find out.

> Are you _sure_ the window size is being set correctly? How
> is it being set?

I'm fairly sure. We echo the value to the file. catting it back
shows the correct value. If we go lower than default, it slows
down even more.

> Are you able to generate TCP dumps when the problem is happening?

We can, if it will help.

Tim

-- 
Tim Sailer <sailer@bnl.gov> Cyber Security Operations
Brookhaven National Laboratory  (631) 344-3001
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Network Performance?

[Erik Mouw]

On Mon, Jan 08, 2001 at 09:06:45AM -0500, Tim Sailer wrote:
> On Mon, Jan 08, 2001 at 09:26:23PM +1100, Andrew Morton wrote:
> > You're sending and receiving FTP/TCP/IP4 to Solaris and AIX hosts
> 
> Yup
> 
> > You have a 1000kbyte window size
> 
> Yup
> 
> > You have an 80 megabit/sec pipe.
> 
> Actually, 100 to the router, and then the WAN connection to the remote
> system is OC-3
> 
> > You're getting 1.8 megabits/sec.
> 
> Yup
> 
> > What is the round-trip time on the WAN?
> > 
> > Packet loss?
> 
> 101 packets transmitted, 101 packets received, 0% packet loss
> round-trip min/avg/max = 109.6/110.3/112.2 ms

I had similar problems two weeks ago. Turned out the connection between
two switches: one of them was hard wired to 100Mbit/s full duplex, the
other one to 100Mbit/s half duplex. Just to rule out the obvious...


Erik

-- 
J.A.K. (Erik) Mouw, Information and Communication Theory Group, Department
of Electrical Engineering, Faculty of Information Technology and Systems,
Delft University of Technology, PO BOX 5031,  2600 GA Delft, The Netherlands
Phone: +31-15-2783635  Fax: +31-15-2781843  Email: J.A.K.Mouw@its.tudelft.nl
WWW: http://www-ict.its.tudelft.nl/~erik/
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Network Performance?

[Tim Sailer]

On Mon, Jan 08, 2001 at 07:07:18PM +0100, Erik Mouw wrote:
> I had similar problems two weeks ago. Turned out the connection between
> two switches: one of them was hard wired to 100Mbit/s full duplex, the
> other one to 100Mbit/s half duplex. Just to rule out the obvious...

We check that as the first thing. Both are set the same. No collisions
out of the ordinary.

Tim

-- 
Tim Sailer <sailer@bnl.gov> Cyber Security Operations
Brookhaven National Laboratory  (631) 344-3001
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Network Performance?

[Martin Josefsson]

On Tue, 9 Jan 2001, Tim Sailer wrote:

> On Mon, Jan 08, 2001 at 07:07:18PM +0100, Erik Mouw wrote:
> > I had similar problems two weeks ago. Turned out the connection between
> > two switches: one of them was hard wired to 100Mbit/s full duplex, the
> > other one to 100Mbit/s half duplex. Just to rule out the obvious...
> 
> We check that as the first thing. Both are set the same. No collisions
> out of the ordinary.

Are you using netfilter? And if so, does netfilter support window-scaling
without the tcp-window-tracking patch?

/Martin

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Network Performance?

[Tim Sailer]

On Tue, Jan 09, 2001 at 05:52:34PM +0100, Martin Josefsson wrote:
> On Tue, 9 Jan 2001, Tim Sailer wrote:
> 
> > On Mon, Jan 08, 2001 at 07:07:18PM +0100, Erik Mouw wrote:
> > > I had similar problems two weeks ago. Turned out the connection between
> > > two switches: one of them was hard wired to 100Mbit/s full duplex, the
> > > other one to 100Mbit/s half duplex. Just to rule out the obvious...
> > 
> > We check that as the first thing. Both are set the same. No collisions
> > out of the ordinary.
> 
> Are you using netfilter? And if so, does netfilter support window-scaling

Nope. We have it stripped down bare at this point to try to pinpoint
the problem.

Tim

> without the tcp-window-tracking patch?
> 
> /Martin

-- 
Tim Sailer <sailer@bnl.gov> Cyber Security Operations
Brookhaven National Laboratory  (631) 344-3001
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Network Performance?

[Craig I. Hagan]

> 101 packets transmitted, 101 packets received, 0% packet loss
> round-trip min/avg/max = 109.6/110.3/112.2 ms
> 
> > Does the problem occur in both directions?
> 
> Good question. I'll find out.
> 
> > Are you _sure_ the window size is being set correctly? How
> > is it being set?
> 
> I'm fairly sure. We echo the value to the file. catting it back
> shows the correct value. If we go lower than default, it slows
> down even more.

what are you setting it to on the solaris machine? what window
sizes have you tried?

Your pipe looks like it will have quite a few bits in flight due to its
latency. From my quick guess math, which sucks, it appears that you can fit 1.2
to 1.5 megabytes on the wire (100mbit machine<-> machine) times 100-120ms wire
time. This is a rather large number, so you may want to see what hosts really
support, perhaps starting with 64k or 128k and work up. Make sure that you have
window scaling turned on if you go with very large windows.

Also, have you upped your socket buffers to match your window sizes?

Last, solaris tends to have poorly tuned tcp values out of the box, look at
this link and tune the solaris stack to better reflect reality.
http://www.google.com/search?q=cache:www.rvs.uni-hannover.de/people/voeckler/tune/EN/tune.html+%2Bwan+%2Bwindow+%2Bscale+%2Bsize+%2Bnetwork&hl=en

linux tuning has a decent amount of data in the docs section of the kernel
sources.

-- craig

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Network Performance?

[Tim Sailer]

On Mon, Jan 08, 2001 at 01:40:57PM -0500, Craig I. Hagan wrote:
> > 101 packets transmitted, 101 packets received, 0% packet loss
> > round-trip min/avg/max = 109.6/110.3/112.2 ms
> > 
> > > Does the problem occur in both directions?
> > 
> > Good question. I'll find out.
> > 
> > > Are you _sure_ the window size is being set correctly? How
> > > is it being set?
> > 
> > I'm fairly sure. We echo the value to the file. catting it back
> > shows the correct value. If we go lower than default, it slows
> > down even more.
> 
> what are you setting it to on the solaris machine? what window
> sizes have you tried?
> 
> Your pipe looks like it will have quite a few bits in flight due to its

Yup. That's why the tuning. WAN performance here is very important.

> latency. From my quick guess math, which sucks, it appears that you can fit 1.2
> to 1.5 megabytes on the wire (100mbit machine<-> machine) times 100-120ms wire

Hmm. 100/8 is about 12, no? 

> time. This is a rather large number, so you may want to see what hosts really
> support, perhaps starting with 64k or 128k and work up. Make sure that you have
> window scaling turned on if you go with very large windows.

Yes, we have that enabled too.

> Also, have you upped your socket buffers to match your window sizes?

We are using straight ftp for the testing.

> Last, solaris tends to have poorly tuned tcp values out of the box, look at
> this link and tune the solaris stack to better reflect reality.
> http://www.google.com/search?q=cache:www.rvs.uni-hannover.de/people/voeckler/tune/EN/tune.html+%2Bwan+%2Bwindow+%2Bscale+%2Bsize+%2Bnetwork&hl=en
> 
> linux tuning has a decent amount of data in the docs section of the kernel
> sources.

I'll take a look. THanks.

Tim

-- 
Tim Sailer <sailer@bnl.gov> Cyber Security Operations
Brookhaven National Laboratory  (631) 344-3001
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Network Performance?

[John Heffner]

On Mon, 8 Jan 2001, Tim Sailer wrote:

> > What is the round-trip time on the WAN?
> > 
> > Packet loss?
> 
> 101 packets transmitted, 101 packets received, 0% packet loss
> round-trip min/avg/max = 109.6/110.3/112.2 ms

Packet loss and RTT can be greatly affected by how much data you're
sending through a path, so a simple ping is probably not adequate.

Also, even a much smaller packet loss rate than 1/100 could really kill
your throughput.

> > Does the problem occur in both directions?
> 
> Good question. I'll find out.
> 
> > Are you _sure_ the window size is being set correctly? How
> > is it being set?
> 
> I'm fairly sure. We echo the value to the file. catting it back
> shows the correct value. If we go lower than default, it slows
> down even more.

You have to be certain you restart any applications currently running, but
it seems like you're probably already doing that, since you observe a
slow-down with smaller windows.

Is window scaling enabled in the kernel?
(/proc/sys/net/ipv4/tcp_window_scaling)

You said you're using this as a proxy.  To rule out the real ftp server
machine as a problem, are you doing an FTP directly from the Linux machine
to your destination?

Also, your destination machine must have an appropriate receive
window/buffer.

> > Are you able to generate TCP dumps when the problem is happening?
> 
> We can, if it will help.

Viewing a trace of the connection is often the most informative and
effective means of debugging a connection.  One note, traces should
probably be collected from the sender.  Receiver traces rarely contain
useful information.

Other questions: What kind of router(s) are you using?  How big are its
buffers?  What drop algorithm does it use?  Have you successfully had any
bulk TCP flows (from other OS's) go through this particular path at near
the full 80Mb?

There are a number of difficulties that may occur when trying to get full
bandwidth out of a long fat pipe (LFP).  You certainly aren't the only one
having problems.  Unfortunately it often takes a TCP expert to diagnose
the problem (or problems), since there are so many possible problems.

  -John

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Network Performance?

[Tim Sailer]

Here is some more data:

        Inbound = 99.66 kB/s
        Outbound= 151 kB/s


--------------------------------------------
ports:/home/ftp# sysctl -a | fgrep net/core
net/core/optmem_max = 10240
net/core/message_burst = 50
net/core/message_cost = 5
net/core/netdev_max_backlog = 300
net/core/rmem_default = 32767
net/core/wmem_default = 32767
net/core/rmem_max = 2097152
net/core/wmem_max = 2097152
--------------------------------------------


Tim

-- 
Tim Sailer <sailer@bnl.gov> Cyber Security Operations
Brookhaven National Laboratory  (631) 344-3001
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Network Performance?

[John Heffner]

On Tue, 9 Jan 2001, Tim Sailer wrote:

> Here is some more data:
> 
>         Inbound = 99.66 kB/s
>         Outbound= 151 kB/s
> 
> 
> --------------------------------------------
> ports:/home/ftp# sysctl -a | fgrep net/core
> net/core/optmem_max = 10240
> net/core/message_burst = 50
> net/core/message_cost = 5
> net/core/netdev_max_backlog = 300
> net/core/rmem_default = 32767			<<<<<<<<<
> net/core/wmem_default = 32767			<<<<<<<<<
> net/core/rmem_max = 2097152
> net/core/wmem_max = 2097152
> --------------------------------------------

The defaults must be large unless your application calls setsockopt() to
set the buffers itself.  (Some FTP clients and servers can do this, but
for testing, your're still probably better always having the _max's and
_default's the same.)

  -John

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Network Performance?

[Tim Sailer]

On Tue, Jan 09, 2001 at 02:29:36PM -0500, John Heffner wrote:
> > --------------------------------------------
> > ports:/home/ftp# sysctl -a | fgrep net/core
> > net/core/optmem_max = 10240
> > net/core/message_burst = 50
> > net/core/message_cost = 5
> > net/core/netdev_max_backlog = 300
> > net/core/rmem_default = 32767			<<<<<<<<<
> > net/core/wmem_default = 32767			<<<<<<<<<
> > net/core/rmem_max = 2097152
> > net/core/wmem_max = 2097152
> > --------------------------------------------
> 
> The defaults must be large unless your application calls setsockopt() to
> set the buffers itself.  (Some FTP clients and servers can do this, but
> for testing, your're still probably better always having the _max's and
> _default's the same.)

Hm.. OK. I think we tried that, but I'll check again.

Thanks,
Tim

-- 
Tim Sailer <sailer@bnl.gov> Cyber Security Operations
Brookhaven National Laboratory  (631) 344-3001
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Network Performance?

[Pekka Pietikainen]

On Tue, Jan 09, 2001 at 03:56:11PM -0500, Tim Sailer wrote:
> > The defaults must be large unless your application calls setsockopt() to
> > set the buffers itself.  (Some FTP clients and servers can do this, but
> > for testing, your're still probably better always having the _max's and
> > _default's the same.)
> 
> Hm.. OK. I think we tried that, but I'll check again.
And make sure your ftp client/server isn't resetting it to something small
afterwards. For testing this, I'd use a real IP benchmarking program
like iperf/netperf/ttcp, as they'll let you test different buffer sizes
easily (and in the case of iperf tell you what you're actually using
if you hit the limit) For a fast WAN you want something like 
512k-1M buffers easily.

-- 
Pekka Pietikainen



-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/