Inbound TCP Circuits over PPP Stall; MTUs and Kppp

Inbound TCP Circuits over PPP Stall; MTUs and Kppp

[Randall R Schulz]

Hello,

In my lonely quest to solve the problem I'm having with the stalling 
of inbound (only, it seems) TCP streams over PPP (but not local 
loopback, it seems), I'm reaching the grasping at straws stage.

By using tcpdump, I have discovered what seems to be a relevant fact: 
When the stream stalls, the sending side (always the remote) is 
resending the same range of bytes (sequence numbers) over and over 
again with an inter-packet arrival time that increases gradually 
until it reaches about two minutes where it holds until one side or 
the other gives up and closes the stream. Each such packet has the 
PUSH flag set and elicits an ACK from my system, but for whatever 
reason, the remote just keeps sending repetitions of a packet 
containing the same range of sequence numbers (with a total of 1448 
data bytes each).

I have captured the tcpdump output from one of these "sessions" 
(beginning when I clicked a FTP URL link in Navigator through until I 
cancelled the stalled transfer after several repetitions at the 
two-minute inter-packet arrival time. There are 140 lines of tcpdump 
output. If anybody can make use of this in diagnosing the problem, 
I'd be happy to send it...

This problem is easily repeated, so if there's a better experiment to 
run, I'd likewise be happy to give that a try. If there are specific 
software version numbers to check for known bugs, please inform me. 
I'm comfortable with just about any kind of software reconfiguration, 
but have never built a Linux kernel.


In looking over various messages to LinuxPPC-Dev and -User, I've 
noticed that there's a variety of MTU values in use for PPP links. 
Values I've seen are:  296, 542, 1500, 1514. Every time I've run 
ifconfig while I have a PPP link active, I've seen 1500 on my system.

I'm wondering what is the significance of the particular value used, 
especially given the wide range of values that appear to be used. Of 
course, mostly I'm wondering if there might be some subtle effect 
(read "bug") related to the 1500 value in effect on my system and my 
problem with stalling TCP streams.

I have always used kppp to establish my PPP connections and it has 
always worked very well. I am curious if there's some issue there 
that I'm not aware of. I used to run MkLinux on an 8500 where I put 
together PPP configuration by hand and never had these problems. By 
scouring the LinuxPPP- mailing list archives and 
comp.os.linux.networking I see that I'm not entirely alone in 
experiencing symptoms that seem similar to mine, but obviously it's 
not a common problem.

I'd like to try changing my PPP MTU from 1500 to 1514, but I don't 
know where to control it under kppp and the documentation doesn't 
mention it.


Lastly, a little configuration information: LinuxPPC recently 
installed from scratch using LinuxPPC Q3 CD-ROM (all packages except 
non-Macintosh kernels installed using the X-based installer). Apple 
Macintosh B&W G3 rev. 1. Built-in Apple-supplied modem. Modem init 
string copied from MacOS OT/PPP Remote Access modem script for this 
modem. All (active) hard drives are on the Apple-supplied Adaptec 
2940U2B Ultra 2 LVD SCSI bus. 384 Mbyte memory and 25 Gbyte disk (not 
all ext2 volumes). Initio 9100UW (Ignored). DVD-ROM and Zip drives on 
IDE/ATA bus.


Any and all help will be greatly appreciated!

Randy Schulz
Mountain View, CA USA

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Inbound TCP Circuits over PPP Stall; MTUs and Kppp

[Michel Lanners]

Hi all,

On  25 Sep, this message from Randall R Schulz echoed through cyberspace:
> In my lonely quest to solve the problem I'm having with the stalling 
> of inbound (only, it seems) TCP streams over PPP (but not local 
> loopback, it seems), I'm reaching the grasping at straws stage.

I've had that same problem repeatedly in the past; I don't see it now
because I don't use PPP anymore.

What was weired for me is that the problem seemed to depend on file
content, so the actual data transfered did make a difference.

For instance, I always had problems with the glibc RPM files.
Downloading them to a different machine at my provider, gzip'ing them
there, I was able to download them to my box without a problem.

> By using tcpdump, I have discovered what seems to be a relevant fact: 
> When the stream stalls, the sending side (always the remote) is 
> resending the same range of bytes (sequence numbers) over and over 
> again with an inter-packet arrival time that increases gradually 
> until it reaches about two minutes where it holds until one side or 
> the other gives up and closes the stream. Each such packet has the 
> PUSH flag set and elicits an ACK from my system, but for whatever 
> reason, the remote just keeps sending repetitions of a packet 
> containing the same range of sequence numbers (with a total of 1448 
> data bytes each).

Does your box send out the ACK's for the incoming data? In that case,
either these never reach the remote end, or the ACK packets contain
some form of error...


Michel

-------------------------------------------------------------------------
Michel Lanners                 |  " Read Philosophy.  Study Art.
23, Rue Paul Henkes            |    Ask Questions.  Make Mistakes.
L-1710 Luxembourg              |
email   mlan@cpu.lu            |
http://www.cpu.lu/~mlan        |                     Learn Always. "


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Inbound TCP Circuits over PPP Stall; MTUs and Kppp

[Randall R Schulz]

Michel,

Thanks for the reply. So far, it's the first one with any meaningful 
content I've gotten on this topic.

Yes, I, too, have the hunch that the problem is data sensitive and 
may relate to checksum generation. There were 17 messages in 
LinuxPPC-Dev (topic: "TCPv4 checksum errors") late last December 
about checksum generation problems, but the thread ended and it 
appeared that either the problem was fixed or was deemed spurious to 
begin with.

I should note that my kernel is generating no complaints about 
checksum errors and ifconfig reports zero errors of any sort 
throughout many megabytes of transfers. I also tried sending a few 
large files out and those transfers never hung. Bringing them same 
files back in order to verify their integrity required repeated use 
of the FTP "reget" command.

I tried turning off the G3 cache just in case it was introducing a 
subtle error (i.e., I was "grasping at straws"), but that seemed to 
make no difference.

Yes, my system sends ACK packets. Here's an excerpt of the exchange 
once the circuit has stalled from my tcpdump log:

18:51:27.860000 207.96.122.8.1268 > 206.173.238.104.1039: P 3523084713:3523086161(1448) ack 728409552 win 32120 <nop,nop,timestamp 314786368 515764> (DF)
18:51:27.860000 206.173.238.104.1039 > 207.96.122.8.1268: . ack 3523086161 win 31856 <nop,nop,timestamp 516296 314786368,nop,nop, sack 1 {3523087605:3523090501} > (DF)
18:51:39.090000 207.96.122.8.1268 > 206.173.238.104.1039: P 3523084713:3523086161(1448) ack 728409552 win 32120 <nop,nop,timestamp 314787492 515764> (DF)
18:51:39.090000 206.173.238.104.1039 > 207.96.122.8.1268: . ack 3523086161 win 31856 <nop,nop,timestamp 517419 314787492,nop,nop, sack 1 {3523087605:3523090501} > (DF)
18:52:01.500000 207.96.122.8.1268 > 206.173.238.104.1039: P 3523084713:3523086161(1448) ack 728409552 win 32120 <nop,nop,timestamp 314789740 515764> (DF)
18:52:01.500000 206.173.238.104.1039 > 207.96.122.8.1268: . ack 3523086161 win 31856 <nop,nop,timestamp 519660 314789740,nop,nop, sack 1 {3523087605:3523090501} > (DF)

My host is 206.173.238.104, the remote is 207.96.122.8 (gwyn.tux.org, 
a.k.a. ftp.xemacs.org). There are several repetitions of this 
exchange and only my impatience ended it.

Thanks for the feedback.

Randy Schulz
Mountain View, CA USA


>Hi all,
>
>On  25 Sep, this message from Randall R Schulz echoed through cyberspace:
> > In my lonely quest to solve the problem I'm having with the stalling 
> > of inbound (only, it seems) TCP streams over PPP (but not local 
> > loopback, it seems), I'm reaching the grasping at straws stage.
>
>I've had that same problem repeatedly in the past; I don't see it now
>because I don't use PPP anymore.
>
>What was weired for me is that the problem seemed to depend on file
>content, so the actual data transfered did make a difference.
>
>For instance, I always had problems with the glibc RPM files.
>Downloading them to a different machine at my provider, gzip'ing them
>there, I was able to download them to my box without a problem.
>
>Does your box send out the ACK's for the incoming data? In that case,
>either these never reach the remote end, or the ACK packets contain
>some form of error...
>
>
>Michel


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Inbound TCP Circuits over PPP Stall; MTUs and Kppp

[Paul Mackerras]

Randall R Schulz <rrschulz@cris.com> wrote:

> By using tcpdump, I have discovered what seems to be a relevant fact: 
> When the stream stalls, the sending side (always the remote) is 
> resending the same range of bytes (sequence numbers) over and over 
> again with an inter-packet arrival time that increases gradually 
> until it reaches about two minutes where it holds until one side or 
> the other gives up and closes the stream. Each such packet has the 
> PUSH flag set and elicits an ACK from my system, but for whatever 
> reason, the remote just keeps sending repetitions of a packet 
> containing the same range of sequence numbers (with a total of 1448 
> data bytes each).

This sounds like the remote system isn't getting the ACKs (or they are
getting corrupted).  Try putting the `novj' option in your ~/.ppprc
file and see if that makes any difference.

> I have captured the tcpdump output from one of these "sessions" 
> (beginning when I clicked a FTP URL link in Navigator through until I 
> cancelled the stalled transfer after several repetitions at the 
> two-minute inter-packet arrival time. There are 140 lines of tcpdump 
> output. If anybody can make use of this in diagnosing the problem, 
> I'd be happy to send it...

I would be interested to see it.

> This problem is easily repeated,

Hopefully we can track it down then.

> I'd like to try changing my PPP MTU from 1500 to 1514, but I don't 
> know where to control it under kppp and the documentation doesn't 
> mention it.

Try making yourself a ~/.ppprc file containing the line `mru 1514'.
In fact the MTU is determined by the MRU (max receive unit) that the
peer asks for so you can't directly control it except to give a
maximum value with the mtu option.

Paul.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Inbound TCP Circuits over PPP Stall; MTUs and Kppp

[Geert Uytterhoeven]

On Sun, 26 Sep 1999, Randall R Schulz wrote:
> Yes, I, too, have the hunch that the problem is data sensitive and 
> may relate to checksum generation. There were 17 messages in 
> LinuxPPC-Dev (topic: "TCPv4 checksum errors") late last December 
> about checksum generation problems, but the thread ended and it 
> appeared that either the problem was fixed or was deemed spurious to 
> begin with.

The bug in the checksum routines was fixed (by Petr Vandrovec, IIRC).

Greetings,

						Geert

--
Geert Uytterhoeven ----------------- Sony Suprastructure Center Europe (SUPC-E)
Geert.Uytterhoeven@sonycom.com ------------------- Sint Stevens Woluwestraat 55
Phone +32-2-7248648 Fax +32-2-7262686 ---------------- B-1130 Brussels, Belgium


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Inbound TCP Circuits over PPP Stall; MTUs and Kppp

[Randall R Schulz]

Hi, Everybody,

Thanks to Paul and all the others who responded with information and 
suggestions.

At Paul M.'s suggestion, I tried using the "novj" option (only) in my 
~/.ppprc file. I made several tests against a few files on a couple 
of different servers.

The good news (at least as far as it offers a work-around): It does 
appear that disabling Van Jacobsen compression alleviates the symptom.

However, I did also discover that even with VJ compression left 
enabled, not all remote hosts experience this problem. As one of my 
tests, I transferred a 2.2 Mbyte file from my private ISP's personal 
web page site (via the FTP interface to that file area). Two attempts 
to transfer that file both succeeded flawlessly even with VJ 
compression turned on. That server is "ftp.cris.com."  I don't know 
for a fact, but I'm guessing that host does *not* run Linux.

On the other hand, transferring files from "ftp.xemacs.org" (a.k.a. 
"gwyn.tux.org") is virtually impossible while VJ compression is 
enabled but works just fine with VJ disabled. In this case, my guess 
is that this host *does* run Linux.

It does also appear to be data related (not surprising if it's a 
problem with the implementation of the compression algorithm, whether 
in LinuxPPC 2.2.6 or whatever's running on ftp.xemacs.org). In 
particular this file: 
<ftp://ftp.xemacs.org/pub/xemacs/docs/letter/internals-letter.pdf.gz> 
will stall the TCP circuit after just a few K have been transferred. 
I repeated this twice and (like every file I tried) and it does 
transfer fine with VJ off.

So, it may be the case that the LinuxPPC code in my kernel is OK (as 
evidenced by the fact that connecting to ftp.cris.com works OK) but 
that the implementation on ftp.xemacs.org is not correct or fully 
compliant with the VJ specification. It's also possible that my 
kernel's VJ code has a bug that is somehow tolerated by 
ftp.cris.com's VJ code.

Right now, I'm content that I can successfully access the net. 
Naturally, the maximum throughput I can get is reduced without VJ 
compression, but that's far better than dealing with constantly 
stalling connections.

One last question: Can I turn VJ compression on and off while a PPP 
link is active, or must I choose before connecting?

Thanks again. If there's anything else I can do for people wishing to 
investigate further, please just let me know.

Randy Schulz
Mountain View, CA USA


At 10:14 +1000 9/27/99, Paul Mackerras wrote:
>Randall R Schulz <rrschulz@cris.com> wrote:
>
> >...
>
>This sounds like the remote system isn't getting the ACKs (or they are
>getting corrupted).  Try putting the `novj' option in your ~/.ppprc
>file and see if that makes any difference.
>
>Paul.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Inbound TCP Circuits over PPP Stall; MTUs and Kppp

[Michel Lanners]

Hi all,

On  27 Sep, this message from Randall R Schulz echoed through cyberspace:
> At Paul M.'s suggestion, I tried using the "novj" option (only) in my 
> ~/.ppprc file. I made several tests against a few files on a couple 
> of different servers.
> 
> The good news (at least as far as it offers a work-around): It does 
> appear that disabling Van Jacobsen compression alleviates the symptom.

Aha...

> However, I did also discover that even with VJ compression left 
> enabled, not all remote hosts experience this problem.
[snip]
> On the other hand, transferring files from "ftp.xemacs.org" (a.k.a. 
> "gwyn.tux.org") is virtually impossible while VJ compression is 
> enabled but works just fine with VJ disabled. In this case, my guess 
> is that this host *does* run Linux.
> 
> It does also appear to be data related (not surprising if it's a 
> problem with the implementation of the compression algorithm, whether 
> in LinuxPPC 2.2.6 or whatever's running on ftp.xemacs.org). In 
> particular this file: 
> <ftp://ftp.xemacs.org/pub/xemacs/docs/letter/internals-letter.pdf.gz> 
> will stall the TCP circuit after just a few K have been transferred. 
> I repeated this twice and (like every file I tried) and it does 
> transfer fine with VJ off.

OK, I saw similar things as well. I had more trouble with the
ftp.linuxppc.org server at that time (running some 2.1.x kernel), than
with my provider's HP server...

> So, it may be the case that the LinuxPPC code in my kernel is OK (as 
> evidenced by the fact that connecting to ftp.cris.com works OK) but 
> that the implementation on ftp.xemacs.org is not correct or fully 
> compliant with the VJ specification. It's also possible that my 
> kernel's VJ code has a bug that is somehow tolerated by 
> ftp.cris.com's VJ code.

EEeeeepppp.... wrong conclusion here ;-). VJ compression is only active
on your PPP link, not end-to-end. So the remote server knows nothing of
the VJ copmpression your machine negotiates with the access server at
your provider.

Which brings us to another variable in the connection chain: the access
server at your provider. In my case, it was a Cisco 2509 with
USRobotics modems. You might want to check with your provider, just in
case... If it really is a VJ problem, then the provider's box could
have an influence, too.

> One last question: Can I turn VJ compression on and off while a PPP 
> link is active, or must I choose before connecting?

I don't think Linux's PPP can renegotiate link parameters. In any case,
the remote end must then be prepared to renegotiate as well...

Michel

-------------------------------------------------------------------------
Michel Lanners                 |  " Read Philosophy.  Study Art.
23, Rue Paul Henkes            |    Ask Questions.  Make Mistakes.
L-1710 Luxembourg              |
email   mlan@cpu.lu            |
http://www.cpu.lu/~mlan        |                     Learn Always. "


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Inbound TCP Circuits over PPP Stall; MTUs and Kppp

[Lou Langholtz]

Another Data point...

I can't help but suspect that this is also related to the "FB.
overflow" messages and the complete-freezing-up of systems that people have
been experiencing and reporting as well. These problems all seems to occur
with use of the PPP system and get worse over time. Sometimes I've even
noticed that even after rebooting my PPC Linux box making a
TCP/IP connection to the same hosts I just hung on (via netscape most
often), I suddenly hang again. It's like the connection stayed open and when
my box tries to reconnect, at some point it starts getting packets again
from its old stream and the kernel locks up on that.

I've been trying to hack on the macserial.c code to fix the "FB.
overflow" problem to no avail so its making more and more sense that the
problem is actually elsewhere in the flow. This latest thread brings me
seriously back to suspecting the PPP and TCP/IP network layer code. Why it
tweaks only the PPC port though has me scratching.

I have a friend who's had the same setup as me but been lucky enough to use
an ISDN router instead of PPP from his box. He's never seen these problems.
Unfortuantely with my modem and my PPP connection I see these just all to
frequently. It seems to ussually go like this...

  1. First my IP connections over PPP get slower and slower. I have a
     28.8Kbaud modem and ussually connect at 21Kbaud and get about that in
     transfer rate. After a couple days of packets my transfer rates get
     worse and worse. I start seeing 38 byte per second transfers and then a
     few hours later the connection just seizes up all-together. I have a
     Cisco 2511 at the other end and suspect it has something to do with the
     problem. At least in encouraging it.
  2. Then I reconnect my modem via netcfg de-activating PPP and
     power-cycling my modem. For a while again I get reasonable transfer
     rates. But then the same thing happens again where the rate drops
     abysmally.
  3. Finally, after a few connection lock / PPP reconnect cycles, the next
     lock-up is my whole machine. Sometimes, after the reboot, connecting to
     the same places I hang my whole machine again right away. The longer
     I wait before connecting to those places however the less I've noticed
     that it locks right back up again. That leads me to believe that the
     lock up problem is going on at the TCP layer; not at the PPP layer. Of
     course if it didn't have anything to do with the PPP layer then people
     not using PPP would see this problem as well.

Well anyhows... I really hope someone more kernel capable than me can find
this helpful since it'll probably be more time effective for me just
throwing money at the problem with a router of some kind to replace the
PPP link.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Inbound TCP Circuits over PPP Stall; MTUs and Kppp

[Geert Uytterhoeven]

On Tue, 28 Sep 1999, Lou Langholtz wrote:
>   1. First my IP connections over PPP get slower and slower. I have a
>      28.8Kbaud modem and ussually connect at 21Kbaud and get about that in
>      transfer rate. After a couple days of packets my transfer rates get
>      worse and worse. I start seeing 38 byte per second transfers and then a
>      few hours later the connection just seizes up all-together. I have a
>      Cisco 2511 at the other end and suspect it has something to do with the
>      problem. At least in encouraging it.
>   2. Then I reconnect my modem via netcfg de-activating PPP and
>      power-cycling my modem. For a while again I get reasonable transfer
>      rates. But then the same thing happens again where the rate drops
>      abysmally.
>   3. Finally, after a few connection lock / PPP reconnect cycles, the next
>      lock-up is my whole machine. Sometimes, after the reboot, connecting to
>      the same places I hang my whole machine again right away. The longer
>      I wait before connecting to those places however the less I've noticed
>      that it locks right back up again. That leads me to believe that the
>      lock up problem is going on at the TCP layer; not at the PPP layer. Of
>      course if it didn't have anything to do with the PPP layer then people
>      not using PPP would see this problem as well.

I see similar things with Ethernet (de4x5), including the lock up when trying
to ifconfig the interface down.

A similar bug was just fixed in the 2.2.x branch of vger. I asked Dave Miller
whether the same bug is also present in 2.3.x, and he said no.

Greetings,

						Geert

--
Geert Uytterhoeven ----------------- Sony Suprastructure Center Europe (SUPC-E)
Geert.Uytterhoeven@sonycom.com ------------------- Sint Stevens Woluwestraat 55
Phone +32-2-7248648 Fax +32-2-7262686 ---------------- B-1130 Brussels, Belgium


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/