After many hours all outbound connections get stuck in SYN_SENT

After many hours all outbound connections get stuck in SYN_SENT

[James Nichols]

Hello,

I have a Java application that makes a large number of outbound
webservice calls over HTTP/TCP.  The hosts contacted are a fixed set
of about 2000 hosts and a web service call is made to each of them
approximately every 5 mintues by a pool of 200 Java threads.  Over
time, on average a percentage of these hosts are unreachable for one
reason or another, usually because they are on wireless cell phone
NICs, so there is a persistent count of sockets in the SYN_SENT state
in the range of about 60-80.  This is fine, as these failed connection
attempts eventually time out.

However, after approximately 38 hours of operation, all outbound
connection attempts get stuck in the SYN_SENT state.  It happens
instantaneously, where I go from the baseline of about 60-80 sockets
in SYN_SENT to a count of 200 (corresponding to the # of java threads
that make these calls).

When I stop and start the Java application, all the new outbound
connections still get stuck in SYN_SENT state.  During this time, I am
still able to SSH to the box and run wget to Google, cnn, etc, so the
problem appears to be specific to the hosts that I'm accessing via the
webservices.

For a long time, the only thing that would resolve this was rebooting
the entire machine.  Once I did this, the outbound connections could
be made succesfully.  However, very recently when I had once of these
incidents I disabled tcp_sack via:

echo "0" > /proc/sys/net/ipv4/tcp_sack

And the problem almost instanteaously resolved itself and outbound
connection attempts were succesful.  I hadn't attempted this before
because I assumed that if any of my network
equipment or remote hosts had a problem with SACK, that it would never
work.  In my case, it worked fine for about 38 hours before hitting a
wall where no outbound connections could be made.

I'm running kernel 2.6.18 on RedHat, but have had this problem occur
on earlier kernel versions (all 2.4 and 2.6).  I know a lot of people
will say it must be the firewall, but I've seen had this issue on
different router vendors, firewall vendors, different co-location
facilities, NICs, and several other variables.  I've totaly rebuilt
every piece of the archtiecture at one time or another and still see
this issue.  I've had this problem to varying degrees of severity for
the past 4 years or so.  Up until this point, the only thing other
than a complete machine restart that fixes the problem is disabling
tcp_sack.  When I disable it, the problem goes away almost
instantaneously.

Is there a kernel buffer or some data structure that tcp_sack uses
that gets filled up after an extended period of operation?
How can I debug this problem in the kernel to find out what the root cause is?

I emailed linux-kernel and they asked for output of netstat -s, I can
get this the next
time it occurs- any other usefull data to collect?

I've temporarily signed up on this list, but may cancel signup if I can't
handle the traffic, so please CC me directly on any replies.

Thanks,

James Nichols

Re: After many hours all outbound connections get stuck in SYN_SENT

[James Nichols]

Here is some additional information about this problem as requested.
I ran ss -m, but no data was returned, what options should I use with
ss to gather relevant information?

The output of netstat -s:

Ip:
    1346453452 total packets received
    0 forwarded
    0 incoming packets discarded
    1345744076 incoming packets delivered
    1338284375 requests sent out
    50 reassemblies required
    15 packets reassembled ok
    15 fragments received ok
    50 fragments created
Icmp:
    431 ICMP messages received
    0 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 42
        echo requests: 6
        echo replies: 377
        timestamp request: 2
        address mask request: 2
    747 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 739
        echo replies: 6
        timestamp replies: 2
Tcp:
    13115640 active connections openings
    1291131 passive connection openings
    381803 failed connection attempts
    6445 connection resets received
    148 connections established
    1339571927 segments received
    1330375560 segments send out
    2443951 segments retransmited
    345 bad segments received.
    61292 resets sent
Udp:
    5608790 packets received
    725 packets to unknown port received.
    0 packet receive errors
    5609766 packets sent
TcpExt:
    1916 resets received for embryonic SYN_RECV sockets
    1290 packets pruned from receive queue because of socket buffer overrun
    1250631 TCP sockets finished time wait in fast timer
    43568 time wait sockets recycled by time stamp
    16323 active connections rejected because of time stamp
    262 packets rejects in established connections because of timestamp
    18505058 delayed acks sent
    3931 delayed acks further delayed because of locked socket
    Quick ack mode was activated 434830 times
    1902 times the listen queue of a socket overflowed
    1902 SYNs to LISTEN sockets ignored
    1068352581 packets directly queued to recvmsg prequeue.
    92424765 packets directly received from backlog
    800659035 packets directly received from prequeue
    1158417138 packets header predicted
    2223869 packets header predicted and directly queued to user
    22256941 acknowledgments not containing data received
    1109445014 predicted acknowledgments
    96 times recovered from packet loss due to fast retransmit
    325 times recovered from packet loss due to SACK data
    1 bad SACKs received
    Detected reordering 8 times using FACK
    Detected reordering 7 times using time stamp
    21 congestion windows fully recovered
    29 congestion windows partially recovered using Hoe heuristic
    452978 congestion windows recovered after partial ack
    97 TCP data loss events
    2269 timeouts after reno fast retransmit
    144 timeouts after SACK recovery
    12690 timeouts in loss state
    731 fast retransmits
    70 forward retransmits
    38188 retransmits in slow start
    959183 other TCP timeouts
    TCPRenoRecoveryFail: 67
    38 sack retransmits failed
    42 times receiver scheduled too late for direct processing
    75627 packets collapsed in receive queue due to low socket buffer
    6003 DSACKs sent for old packets
    13 DSACKs sent for out of order packets
    136 DSACKs received
    4038 connections reset due to unexpected data
    557 connections reset due to early user close
    319219 connections aborted due to timeout






On 12/16/07, James Nichols <jamesnichols3@gmail.com> wrote:
> Hello,
>
> I have a Java application that makes a large number of outbound
> webservice calls over HTTP/TCP.  The hosts contacted are a fixed set
> of about 2000 hosts and a web service call is made to each of them
> approximately every 5 mintues by a pool of 200 Java threads.  Over
> time, on average a percentage of these hosts are unreachable for one
> reason or another, usually because they are on wireless cell phone
> NICs, so there is a persistent count of sockets in the SYN_SENT state
> in the range of about 60-80.  This is fine, as these failed connection
> attempts eventually time out.
>
> However, after approximately 38 hours of operation, all outbound
> connection attempts get stuck in the SYN_SENT state.  It happens
> instantaneously, where I go from the baseline of about 60-80 sockets
> in SYN_SENT to a count of 200 (corresponding to the # of java threads
> that make these calls).
>
> When I stop and start the Java application, all the new outbound
> connections still get stuck in SYN_SENT state.  During this time, I am
> still able to SSH to the box and run wget to Google, cnn, etc, so the
> problem appears to be specific to the hosts that I'm accessing via the
> webservices.
>
> For a long time, the only thing that would resolve this was rebooting
> the entire machine.  Once I did this, the outbound connections could
> be made succesfully.  However, very recently when I had once of these
> incidents I disabled tcp_sack via:
>
> echo "0" > /proc/sys/net/ipv4/tcp_sack
>
> And the problem almost instanteaously resolved itself and outbound
> connection attempts were succesful.  I hadn't attempted this before
> because I assumed that if any of my network
> equipment or remote hosts had a problem with SACK, that it would never
> work.  In my case, it worked fine for about 38 hours before hitting a
> wall where no outbound connections could be made.
>
> I'm running kernel 2.6.18 on RedHat, but have had this problem occur
> on earlier kernel versions (all 2.4 and 2.6).  I know a lot of people
> will say it must be the firewall, but I've seen had this issue on
> different router vendors, firewall vendors, different co-location
> facilities, NICs, and several other variables.  I've totaly rebuilt
> every piece of the archtiecture at one time or another and still see
> this issue.  I've had this problem to varying degrees of severity for
> the past 4 years or so.  Up until this point, the only thing other
> than a complete machine restart that fixes the problem is disabling
> tcp_sack.  When I disable it, the problem goes away almost
> instantaneously.
>
> Is there a kernel buffer or some data structure that tcp_sack uses
> that gets filled up after an extended period of operation?
> How can I debug this problem in the kernel to find out what the root cause is?
>
> I emailed linux-kernel and they asked for output of netstat -s, I can
> get this the next
> time it occurs- any other usefull data to collect?
>
> I've temporarily signed up on this list, but may cancel signup if I can't
> handle the traffic, so please CC me directly on any replies.
>
> Thanks,
>
> James Nichols
>

Re: After many hours all outbound connections get stuck in SYN_SENT

[Eric Dumazet]

James Nichols a écrit :
>> Well... please dont start a flame war :(
>>
>> Back to your SYN_SENT problem, I suppose the remote IP is known, so you
>> probably could post here the result of a tcdpump ?
>>
>> tcpdump -p -n -s 1600 host IP_of_problematic_peer -c 500
>>
>> Most probably remote peer received too many attempts from you, and a
>> anti DOS mechanism is droping all SYN packets.
>>
>> Ah well... I remember now that you mentioned tcp_sack setting had an
>> effect, so forget the "Most probably..." and give some tcpdump traces :)
> 
> 
> I've run tcpdump for all IPs during this problem.  I haven't tried
> doing it for a single explicit IP address- due to the nature of the
> workload it's very difficult to know which IPs will be hit at any
> given moment.   What I did see in the full IP captures is that the
> returning ACKs don't show up in the packet capture.  Unfortunately,
> tcpdump reported that some packets were dropped during the capture.
> Is it possible that the kernel was dropping the packets before they
> could be captured by tcpdump?

Yes it can happens, because an active sniffer makes the stack using more
  cpu cycles (timestamping for example).

So you see outgoing SYN packets, but no SYN replies coming from the remote 
peer ?  (you mention ACKS, but the first packet received from the remote
  peer should be a SYN+ACK),

client->server   SYN
server->client   SYN+ACK
client->server   ACK


> 
> Also, I have some doubts about it being the end points or an
> intermediate router, please let me know if these are unreasonable:
> 1)  We've completely replaced our routing equipment several times in
> the past 4 years... totally different colos, router vendors, firewall
> vendors, firewall rules, etc.
> 2)  It occurs across all remote end points at the exact same time.
> The endpoints are hetrogenous, run brain-dead OS's that don't do any
> DOS detection, reboot at random times of the day, are geographically
> distributed, are on different ISPs, etc. etc.
> 3)  Turning of tcp_sack instantaneously makes the problem go away.  If
> it were endpoints or a router, it seems like a stretch that removing a
> single TCP option would make the problem instantly resolve itself in
> so many places other than the originating host.

CC to netdev where linux network guys might have an idea.

When the problem comes, instead of restarting the application, please take a 
tcpdump of say 10.000 packets.
Then turn off tcp_sack and take a 2nd tcpdump sample, and make both samples 
available to us.

If turning off tcp_sack makes the problem go away, why dont you turn it off 
all the time ?

Re: After many hours all outbound connections get stuck in SYN_SENT

[Jan Engelhardt]

On Dec 18 2007 21:37, Eric Dumazet wrote:
>
> If turning off tcp_sack makes the problem go away, why dont you
> turn it off all the time ?
>
That would just be workaround. I welcome the efforts to track this;
not all users have the time to do so.
Disabling tcp_sack also disabled it kernel-wide, which, well... for
2.6.25 there is a TCPOPTSTRIP netfilter target slated with which
SACK could be stripped only for a given host list or processes from
a UID.

Re: After many hours all outbound connections get stuck in SYN_SENT

[Ilpo Järvinen]

On Sun, 16 Dec 2007, James Nichols wrote:

> I have a Java application that makes a large number of outbound
> webservice calls over HTTP/TCP.  The hosts contacted are a fixed set
> of about 2000 hosts and a web service call is made to each of them
> approximately every 5 mintues by a pool of 200 Java threads.  Over
> time, on average a percentage of these hosts are unreachable for one
> reason or another, usually because they are on wireless cell phone
> NICs, so there is a persistent count of sockets in the SYN_SENT state
> in the range of about 60-80.  This is fine, as these failed connection
> attempts eventually time out.
> 
> However, after approximately 38 hours of operation, all outbound
> connection attempts get stuck in the SYN_SENT state.  It happens
> instantaneously, where I go from the baseline of about 60-80 sockets
> in SYN_SENT to a count of 200 (corresponding to the # of java threads
> that make these calls).
> 
> When I stop and start the Java application, all the new outbound
> connections still get stuck in SYN_SENT state.

Is it so that they don't timeout at all? You can collect some of their 
state from /proc/net/tcp (shows at least timers and attempt counters)....

> During this time, I am
> still able to SSH to the box and run wget to Google, cnn, etc, so the
> problem appears to be specific to the hosts that I'm accessing via the
> webservices.

Are you sure that you just don't get unlucky at some point of time and 
all 200 available threads are just temporarily stuck and your application 
is just very slowly progressing then?

> For a long time, the only thing that would resolve this was rebooting
> the entire machine.  Once I did this, the outbound connections could
> be made succesfully.

To the very same hosts? Or to another set of hosts?

> However, very recently when I had once of these incidents I disabled 
> tcp_sack via:
> 
> echo "0" > /proc/sys/net/ipv4/tcp_sack
> 
> And the problem almost instanteaously resolved itself and outbound
> connection attempts were succesful.

New or the pending ones?

> I hadn't attempted this before because I assumed that if any of my 
> network
> equipment or remote hosts had a problem with SACK, that it would never
> work.

Many bugs just are not like that at all... Usually people who coded things 
had at least some clue :-), so things work "almost correctly"...

>  In my case, it worked fine for about 38 hours before hitting a
> wall where no outbound connections could be made.

How accurate number? Is the lockup somehow related to daytime cycle?

> Is there a kernel buffer or some data structure that tcp_sack uses
> that gets filled up after an extended period of operation?

SACK has pretty little meaning in context of SYNs, there's only the 
sackperm(itted) TCP option which is sent along with the SYN/SYN-ACK.

The SACK scoreboard is currently included to the skbs (has been like 
this for very long time), so no additional data structures should be
there because of SACK...

> How can I debug this problem in the kernel to find out what the root cause is?
> 
> I emailed linux-kernel and they asked for output of netstat -s, I can
> get this the next
> time it occurs- any other usefull data to collect?

/proc/net/tcp couple of times in a row, try something something like
this:

for i in (seq 1 40); do cat /proc/net/tcp; echo "-----"; sleep 10; done


> I'm running kernel 2.6.18 on RedHat, but have had this problem occur
> on earlier kernel versions (all 2.4 and 2.6).

I've done some fixes to SACK processing since 2.6.18 (not sure if RedHat 
has backported them). Though they're not that critical nor anything in 
them should affect in SYN_SENT state.


-- 
 i.

Re: After many hours all outbound connections get stuck in SYN_SENT

[James Nichols]

> So you see outgoing SYN packets, but no SYN replies coming from the remote
> peer ?  (you mention ACKS, but the first packet received from the remote
>  peer should be a SYN+ACK),

Right, I meant to say SYN+ACK.  I don't see them coming back.


> When the problem comes, instead of restarting the application, please take a
> tcpdump of say 10.000 packets.
> Then turn off tcp_sack and take a 2nd tcpdump sample, and make both samples
> available to us.

I can take these captures and take a look at the results.
Unfortunately, I don't think I'll be able to make the captures
available to the general public.



> If turning off tcp_sack makes the problem go away, why dont you turn it off
> all the time ?

Unfortunately, I think that will be the answer if I can't get any help
fixing this problem in the kernel.  It's a bummer, because many of the
remote hosts my application communicates with are on wireless links,
so there may be performance implications to turning SACK off.

Re: After many hours all outbound connections get stuck in SYN_SENT

[Eric Dumazet]

James Nichols a écrit :
>> So you see outgoing SYN packets, but no SYN replies coming from the remote
>> peer ?  (you mention ACKS, but the first packet received from the remote
>>  peer should be a SYN+ACK),
> 
> Right, I meant to say SYN+ACK.  I don't see them coming back.

So... Really unlikely a linux problem, but ...

> 
> 
>> When the problem comes, instead of restarting the application, please take a
>> tcpdump of say 10.000 packets.
>> Then turn off tcp_sack and take a 2nd tcpdump sample, and make both samples
>> available to us.
> 
> I can take these captures and take a look at the results.
> Unfortunately, I don't think I'll be able to make the captures
> available to the general public.

I dont understand, why dont you change IPs to mask them with 192.168.X.Y, or 
just ME, and peer1, peer2, peer...

> 
> 
> 
>> If turning off tcp_sack makes the problem go away, why dont you turn it off
>> all the time ?
> 
> Unfortunately, I think that will be the answer if I can't get any help
> fixing this problem in the kernel.  It's a bummer, because many of the
> remote hosts my application communicates with are on wireless links,
> so there may be performance implications to turning SACK off.
> 

Random ideas :

1) Is your server behind a NET router or something ?
2) Are you sure you are not using connection tracking, and hit a limit on it ?

Re: After many hours all outbound connections get stuck in SYN_SENT

[James Nichols]

> > When I stop and start the Java application, all the new outbound
> > connections still get stuck in SYN_SENT state.
>
> Is it so that they don't timeout at all? You can collect some of their
> state from /proc/net/tcp (shows at least timers and attempt counters)....

The outbound connections to timeout.  I've watched that they send
tcp_syn_retries SYN packets before eventually timing out.



> Are you sure that you just don't get unlucky at some point of time and
> all 200 available threads are just temporarily stuck and your application
> is just very slowly progressing then?

Yeah, I'm sure that it isn't an unlucky point of time.  If I restart
the application when this problem occurs, all the outbound connections
still fail.


> > For a long time, the only thing that would resolve this was rebooting
> > the entire machine.  Once I did this, the outbound connections could
> > be made succesfully.
>
> To the very same hosts? Or to another set of hosts?

Yes, to the same exact set of hosts.



> > And the problem almost instanteaously resolved itself and outbound
> > connection attempts were succesful.
>
> New or the pending ones?

I'm fairly sure that sockets that were already open in SYN_SENT state
when I turned tcp_sack off started to work as the count of sockets in
SYN_SENT state drops very rapidly.


> >  In my case, it worked fine for about 38 hours before hitting a
> > wall where no outbound connections could be made.
>
> How accurate number? Is the lockup somehow related to daytime cycle?

It is 38 hours +/- a half hour or so.  It isn't related to the time of
day, as it happens through out day/night depending on when the server
was restarted.  A new developement in this area is that after the
first 38 hours of system time the problem would occur, so I disable
tcp_sack and the problem clears itself up and outbound connections are
succesful.  After a couple of hours I re-enable tcp_sack and the next
SYN_SENT issue doesn't occur until more than 50 hours later (so like
90 hours after system start).  It's as if the first time it occurs and
I turn tcp_sack off, it doesn't just reset the clock another 38 hours,
but gives even more time until the problem occurs again.




> > Is there a kernel buffer or some data structure that tcp_sack uses
> > that gets filled up after an extended period of operation?
>
> SACK has pretty little meaning in context of SYNs, there's only the
> sackperm(itted) TCP option which is sent along with the SYN/SYN-ACK.
>
> The SACK scoreboard is currently included to the skbs (has been like
> this for very long time), so no additional data structures should be
> there because of SACK...

I've been seeing this problem for about 4 years, so could it be
related to the scoreboard implementation somehow?



> /proc/net/tcp couple of times in a row, try something something like
> this:
>
> for i in (seq 1 40); do cat /proc/net/tcp; echo "-----"; sleep 10; done

I can set up to do this the next time the problem occurs.


> > I'm running kernel 2.6.18 on RedHat, but have had this problem occur
> > on earlier kernel versions (all 2.4 and 2.6).
>
> I've done some fixes to SACK processing since 2.6.18 (not sure if RedHat
> has backported them). Though they're not that critical nor anything in
> them should affect in SYN_SENT state.

Ok, unless there is direct evidence that there is a fix to this
problem in a later kernel I won't be able to upgrade.  If there is a
redhat provided patch I can probably do that.

Re: After many hours all outbound connections get stuck in SYN_SENT

[James Nichols]

On 12/19/07, Eric Dumazet <dada1@cosmosbay.com> wrote:
> James Nichols a écrit :
> >> So you see outgoing SYN packets, but no SYN replies coming from the remote
> >> peer ?  (you mention ACKS, but the first packet received from the remote
> >>  peer should be a SYN+ACK),
> >
> > Right, I meant to say SYN+ACK.  I don't see them coming back.
>
> So... Really unlikely a linux problem, but ...
>


I don't know how you can be so sure.  Turning tcp_sack off instantly
resovles the problem and all connections are succesful.  I can't
imagine even the most far-fetched scenario where a router or every
single remote endpoints would suddenly stop causing the problem just
by removing a single TCP option.


> > I can take these captures and take a look at the results.
> > Unfortunately, I don't think I'll be able to make the captures
> > available to the general public.
>
> I dont understand, why dont you change IPs to mask them with 192.168.X.Y, or
> just ME, and peer1, peer2, peer...

I will see if I can do that, but it's major pain with 2000 hosts.
Plus, there is application data in the packets that I can't allow into
the public domain.  I really don't think I can pull it off... I
literally would have to go through our legal department.

>
> Random ideas :
>
> 1) Is your server behind a NET router or something ?

What's a NET router?  I am behind a Cisco router and a firewall, but
these network components have completely been replaced/rebuilt several
times in the 4+ years that we've had this problem.  I've looked at the
logs there and neither are doing anything other than passing the
traffic along.

> 2) Are you sure you are not using connection tracking, and hit a limit on it ?

I'm using ip_conntrack, but the limit I have for max entries is 65K.
The most I've seen in there are a couple thousand- that was one of the
first things I monitored very closely.

Re: After many hours all outbound connections get stuck in SYN_SENT

[Jan Engelhardt]

On Dec 19 2007 12:43, James Nichols wrote:
>On 12/19/07, Eric Dumazet <dada1@cosmosbay.com> wrote:
>> James Nichols a écrit :
>> >> So you see outgoing SYN packets, but no SYN replies coming from the remote
>> >> peer ?  (you mention ACKS, but the first packet received from the remote
>> >>  peer should be a SYN+ACK),
>> >
>> > Right, I meant to say SYN+ACK.  I don't see them coming back.
>>
>> So... Really unlikely a linux problem, but ...
>
>I don't know how you can be so sure.  Turning tcp_sack off instantly
>resovles the problem and all connections are succesful.  I can't
>imagine even the most far-fetched scenario where a router or every
>single remote endpoints would suddenly stop causing the problem just
>by removing a single TCP option.

The router could be sooo crappy that it drops all packets from
TCP streams that have SACK enabled and the client has opened
200+ SACK connections previously... something like that?

Re: After many hours all outbound connections get stuck in SYN_SENT

[Eric Dumazet]

James Nichols a écrit :
> On 12/19/07, Eric Dumazet <dada1@cosmosbay.com> wrote:
>> James Nichols a écrit :
>>>> So you see outgoing SYN packets, but no SYN replies coming from the remote
>>>> peer ?  (you mention ACKS, but the first packet received from the remote
>>>>  peer should be a SYN+ACK),
>>> Right, I meant to say SYN+ACK.  I don't see them coming back.
>> So... Really unlikely a linux problem, but ...
>>
> 
> 
> I don't know how you can be so sure.  Turning tcp_sack off instantly
> resovles the problem and all connections are succesful.  I can't
> imagine even the most far-fetched scenario where a router or every
> single remote endpoints would suddenly stop causing the problem just
> by removing a single TCP option.
> 
> 
>>> I can take these captures and take a look at the results.
>>> Unfortunately, I don't think I'll be able to make the captures
>>> available to the general public.
>> I dont understand, why dont you change IPs to mask them with 192.168.X.Y, or
>> just ME, and peer1, peer2, peer...
> 
> I will see if I can do that, but it's major pain with 2000 hosts.
> Plus, there is application data in the packets that I can't allow into
> the public domain.  I really don't think I can pull it off... I
> literally would have to go through our legal department.

I still dont understand.

"tcpdump -p -n -s 1600 -c 10000" doesnt reveal User data at all.

Without any exact data from you, I am afraid nobody can help.

> 
>> Random ideas :
>>
>> 1) Is your server behind a NET router or something ?
> 
> What's a NET router?  I am behind a Cisco router and a firewall, but
> these network components have completely been replaced/rebuilt several
> times in the 4+ years that we've had this problem.  I've looked at the
> logs there and neither are doing anything other than passing the
> traffic along.

Typo error, I meant NAT. Most routers doing NAT have some limits, timers, hacks...

> 
>> 2) Are you sure you are not using connection tracking, and hit a limit on it ?
> 
> I'm using ip_conntrack, but the limit I have for max entries is 65K.
> The most I've seen in there are a couple thousand- that was one of the
> first things I monitored very closely.

Now please try without conn tracking module. I saw many failures in the past
that were trigered by conntrack.

Do you have some firewall rules, using some netfilter modules like hashlimit ?


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: After many hours all outbound connections get stuck in SYN_SENT

[James Nichols]

> The router could be sooo crappy that it drops all packets from
> TCP streams that have SACK enabled and the client has opened
> 200+ SACK connections previously... something like that?

I don't know, maybe.  My router is a fairly new model Cisco and is
pretty major (i.e. pretty expensive), so it's not just a total piece
of crap.  Plus, I never restart it when I see these issues.  I just
turn tcp_sack off, the problem goes away, and I'm able to renable
tcp_sack a few hours later and it works fine until many hours later
when I see the SYN_SENT problem again.

Re: After many hours all outbound connections get stuck in SYN_SENT

[Ilpo Järvinen]

On Wed, 19 Dec 2007, James Nichols wrote:

> > > And the problem almost instanteaously resolved itself and outbound
> > > connection attempts were succesful.
> >
> > New or the pending ones?
> 
> I'm fairly sure that sockets that were already open in SYN_SENT state
> when I turned tcp_sack off started to work as the count of sockets in
> SYN_SENT state drops very rapidly.

Heh, "very rapidly" :-/, considering that you have 200 x SYN_SENT flows 
and if tcp_syn_retries is set to default, you will see something happening 
quite quickly for sure and the whole situation is over in ~3 mins if I 
counted correctly.

> > > Is there a kernel buffer or some data structure that tcp_sack uses
> > > that gets filled up after an extended period of operation?
> >
> > SACK has pretty little meaning in context of SYNs, there's only the
> > sackperm(itted) TCP option which is sent along with the SYN/SYN-ACK.
> >
> > The SACK scoreboard is currently included to the skbs (has been like
> > this for very long time), so no additional data structures should be
> > there because of SACK...
> 
> I've been seeing this problem for about 4 years, so could it be
> related to the scoreboard implementation somehow?

Scoreboard has no meaning in this context, it is used while _input_ 
packets are processed. If you don't get SYN-ACKs at all, it doesn't
have any meaning.

> > /proc/net/tcp couple of times in a row, try something something like
> > this:
> >
> > for i in (seq 1 40); do cat /proc/net/tcp; echo "-----"; sleep 10; done
> 
> I can set up to do this the next time the problem occurs.

for i in $(seq 1 40); ... is the right way to do the loop. :-)

> > > I'm running kernel 2.6.18 on RedHat, but have had this problem occur
> > > on earlier kernel versions (all 2.4 and 2.6).
> >
> > I've done some fixes to SACK processing since 2.6.18 (not sure if RedHat
> > has backported them). Though they're not that critical nor anything in
> > them should affect in SYN_SENT state.
> 
> Ok, unless there is direct evidence that there is a fix to this
> problem in a later kernel I won't be able to upgrade.  If there is a
> redhat provided patch I can probably do that.

...They won't affect in SYN_SENT.

-- 
 i.

Re: After many hours all outbound connections get stuck in SYN_SENT

[Ilpo Järvinen]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 972 bytes --]

On Wed, 19 Dec 2007, Eric Dumazet wrote:

> James Nichols a écrit :
> > On 12/19/07, Eric Dumazet <dada1@cosmosbay.com> wrote:
> > > James Nichols a écrit :
> > > > > So you see outgoing SYN packets, but no SYN replies coming from the
> > > > > remote
> > > > > peer ?  (you mention ACKS, but the first packet received from the
> > > > > remote
> > > > >  peer should be a SYN+ACK),
> > > > Right, I meant to say SYN+ACK.  I don't see them coming back.
> > > So... Really unlikely a linux problem, but ...
> > >
> > 
> > 
> > I don't know how you can be so sure.  Turning tcp_sack off instantly
> > resovles the problem and all connections are succesful.  I can't
> > imagine even the most far-fetched scenario where a router or every
> > single remote endpoints would suddenly stop causing the problem just
> > by removing a single TCP option.

You could also check if you can pull same trick off by touching 
tcp_timestamps. It affects the TCP header as well.



-- 
 i.

Re: After many hours all outbound connections get stuck in SYN_SENT

[Glen Turner]

[speculation by network engineer -- not kernel hacker -- follows]

> The router could be sooo crappy that it drops all packets from
> TCP streams that have SACK enabled and the client has opened
> 200+ SACK connections previously... something like that?

As far as any third party is concerned the existing TCP connections
continue to have negotiated "SACK Permitted". Only new connections
will not negotiate this.  So "router crappiness" promptly disappearing
doesn't seem too likely (a way I could see this happening is if the
Linux box sends a Ack for each connection and this clears out Sack
datastructures on the third party).

But I'd be very surprised if the router is acting as anything more
that a network-layer device. It might perhaps have some soft connection
state being used for generating accounting records.  Being Cisco
it's probably a switch-router, so it might carry some per-port hard
state for validating source IP addresses and ARPs on each port.

The firewall is much more likely to be carrying per-flow Sack
state. The Cisco PIX had a bug with SACK handling (CSCse14419,
fixed in 7.0(7), 7.1(2.34), 7.2(2.2), 8.0(0.141) but perhaps it
has regressed). A simple trace either side of the firewall will
show the inconsistency between the TCP sequence number (which
gets randomised) and the Sack sequence number (which didn't).
You could disable the TCP Sequence Number Randomisation feature
and see if the fault reoccurs.

You'd probably should also investigate the Linux kernel,
especially the size and locks of the components of the Sack data
structures and what happens to those data structures after Sack is
disabled (presumably the Sack data structure is in some unhappy
circumstance, and disabling Sack allows the data to be discarded,
magically unclaging the box).

In the absence of the reporter wanting to dump the kernel's
core, how about a patch to print the Sack datastructure when
the command to disable Sack is received by the kernel?
Maybe just print the last 16b of the IP address?

Best wishes, Glen

Re: After many hours all outbound connections get stuck in SYN_SENT

[James Nichols]

> I still dont understand.
>
> "tcpdump -p -n -s 1600 -c 10000" doesnt reveal User data at all.
>
> Without any exact data from you, I am afraid nobody can help.

Oh, I didn't see that you specified specific options.  I'll still have
to anonymize 2000+ IP addresses, but I think there is an open source
tool that will do this for you.



> >> 2) Are you sure you are not using connection tracking, and hit a limit on it ?
> >
> > I'm using ip_conntrack, but the limit I have for max entries is 65K.
> > The most I've seen in there are a couple thousand- that was one of the
> > first things I monitored very closely.
>
> Now please try without conn tracking module. I saw many failures in the past
> that were trigered by conntrack.
>
> Do you have some firewall rules, using some netfilter modules like hashlimit ?

I will have to look into this.

Re: After many hours all outbound connections get stuck in SYN_SENT

[James Nichols]

> But I'd be very surprised if the router is acting as anything more
> that a network-layer device. It might perhaps have some soft connection
> state being used for generating accounting records.  Being Cisco
> it's probably a switch-router, so it might carry some per-port hard
> state for validating source IP addresses and ARPs on each port.
>
> The firewall is much more likely to be carrying per-flow Sack
> state. The Cisco PIX had a bug with SACK handling (CSCse14419,
> fixed in 7.0(7), 7.1(2.34), 7.2(2.2), 8.0(0.141) but perhaps it
> has regressed). A simple trace either side of the firewall will
> show the inconsistency between the TCP sequence number (which
> gets randomised) and the Sack sequence number (which didn't).
> You could disable the TCP Sequence Number Randomisation feature
> and see if the fault reoccurs.

I do have TCP Sequence # Randomization enabled on my router.  However,
if this was causing an issue, wouldn't it always occur and cause
connection issues, not just after 38 hours of correct operation?  I
can look into turning this off, but I'll likely have to jump through
several hoops which will be challenging if I don't have a very clear
definitive reason why this is causing this issue.  Plus, I've had this
problem with at least 2 other sets of network switches over the past 4
years.  I'm actually running 7.0(6), which doesn't have the fix you
mentioned.  If it really is possible that this issue wouldn't always
cause problems, but only after hours of succesful operation, then I
could probably motivate the upgrade.  I can try to setup a trace, but
this is a lot of work for other people in my organization, so it will
take quite some time.


> You'd probably should also investigate the Linux kernel,
> especially the size and locks of the components of the Sack data
> structures and what happens to those data structures after Sack is
> disabled (presumably the Sack data structure is in some unhappy
> circumstance, and disabling Sack allows the data to be discarded,
> magically unclaging the box).
>
> In the absence of the reporter wanting to dump the kernel's
> core, how about a patch to print the Sack datastructure when
> the command to disable Sack is received by the kernel?
> Maybe just print the last 16b of the IP address?

Given the fact that I've had this problem for so long, over a variety
of networking hardware vendors and colo-facilities, this really sounds
good to me.  It will be challenging for me to justify a kernel core
dump, but a simple patch to dump the Sack data would be do-able.

Re: After many hours all outbound connections get stuck in SYN_SENT

[Ilpo Järvinen]

On Thu, 20 Dec 2007, James Nichols wrote:

> > I still dont understand.
> >
> > "tcpdump -p -n -s 1600 -c 10000" doesnt reveal User data at all.
> >
> > Without any exact data from you, I am afraid nobody can help.
> 
> Oh, I didn't see that you specified specific options.  I'll still have
> to anonymize 2000+ IP addresses, but I think there is an open source
> tool that will do this for you.

Even a simple for loop in shell can do that. It's not that hard and 
there's very little need for manual work! Ingrediments: for, cut, grep
and sed.


-- 
 i.

Re: After many hours all outbound connections get stuck in SYN_SENT

[Justin Banks]

James Nichols wrote
> > I still dont understand.
> >
> > "tcpdump -p -n -s 1600 -c 10000" doesnt reveal User data at all.
> >
> > Without any exact data from you, I am afraid nobody can help.
> 
> Oh, I didn't see that you specified specific options.  I'll still have
> to anonymize 2000+ IP addresses, but I think there is an open source
> tool that will do this for you.


tcpdump -p -n -s 1600 -c 10000 | perl -pe 's/(\d+\.\d+\.\d+\.\d+)/HIDE.THIS.IP.ADDR/g'

-justinb

-- 
Justin Banks
BakBone Software
justinb@bakbone.com

Re: After many hours all outbound connections get stuck in SYN_SENT

[Ilpo Järvinen]

On Thu, 20 Dec 2007, James Nichols wrote:

> > You'd probably should also investigate the Linux kernel,
> > especially the size and locks of the components of the Sack data
> > structures and what happens to those data structures after Sack is
> > disabled (presumably the Sack data structure is in some unhappy
> > circumstance, and disabling Sack allows the data to be discarded,
> > magically unclaging the box).

...Not sure if you want now to invent such structure. Yes, we have per skb 
->sacked but again in SYN_SENT there are very few things who touch it at 
all, and they just set it to zero (though it would not even be mandatory 
for tcp_transmit_skb, IIRC, checked that just couple of days ago due to 
other things).

Another thing is the rx_opt.sack_ok which is just couple flag bits that 
tell the TCP variant in use (and it's mostly used only after SYN handshake 
completes). The rest (the actual SACK blocks) is in the ack_skb but again 
it has very little meaning in SYN_SENT state unless somebody is crazy 
enough to add SACK blocks to SYN-ACKs :-).

> > In the absence of the reporter wanting to dump the kernel's
> > core, how about a patch to print the Sack datastructure when
> > the command to disable Sack is received by the kernel?
> > Maybe just print the last 16b of the IP address?
> 
> Given the fact that I've had this problem for so long, over a variety
> of networking hardware vendors and colo-facilities, this really sounds
> good to me.  It will be challenging for me to justify a kernel core
> dump, but a simple patch to dump the Sack data would be do-able.

If your symptoms really are: SYNs leaving (if they show up in tcpdump, for 
sure they've left TCP code already) and SYN-ACK not showing up even in 
something as early as in tcpdump (for sure TCP side code didn't execute at 
that point yet), there's very little change that Linux' TCP code has some 
bug in it, only things that do something in such scenario are the SYN 
generation and retransmitting SYNs (and those are trivially verifiable 
from tcpdump).


-- 
 i.

Re: After many hours all outbound connections get stuck in SYN_SENT

[Glen Turner]

> I do have TCP Sequence # Randomization enabled on my router.

Huh?  Do you mean a PIX blade in a Cisco switch-router chassis? It
would be very useful if you could be less vague about the
equipment in use.

>  However,
> if this was causing an issue, wouldn't it always occur and cause
> connection issues, not just after 38 hours of correct operation?

That depends more on your customers' networking attributes
then you are sharing or perhaps even know.  Perhaps your customer
base is very Window-skewed and you simply aren't seeing any Sack
Permitted negotiations for the first 37.999 hours. Or
perhaps you've had a network glitch, and all of your
connections have done a Selective Ack, which the firewall
has trashed, leaving all the connections in a wacko state,
not just a few which you haven't noticed.

The actual failure mode needs a packet trace to determine,
but you should be able to do this yourself (or ask your
local network engineering staff).

If your firewall is trashing the Sack field, then it needs
to be fixed.  Time to raise a case with the Cisco TAC and
ask them directly if your PIX version has bug CSCse14419.
You can't expect Sack to work when it's being fed trash,
so it is important to make sure that is not happening.

Cheers, Glen
#include <network_engineer.h>
#undef KERNEL_HACKER

Re: After many hours all outbound connections get stuck in SYN_SENT

[Jan Engelhardt]

On Dec 20 2007 23:05, Ilpo Järvinen wrote:
>> 
>> Given the fact that I've had this problem for so long, over a variety
>> of networking hardware vendors and colo-facilities, this really sounds
>> good to me.  It will be challenging for me to justify a kernel core
>> dump, but a simple patch to dump the Sack data would be do-able.
>
>If your symptoms really are: SYNs leaving (if they show up in tcpdump, for 
>sure they've left TCP code already) and SYN-ACK not showing up even in 
>something as early as in tcpdump (for sure TCP side code didn't execute at 
>that point yet), there's very little change that Linux' TCP code has some 
>bug in it, only things that do something in such scenario are the SYN 
>generation and retransmitting SYNs (and those are trivially verifiable 
>from tcpdump).
>
Take a machine, put two interfaces in it, configure as bridge (br0
over eth0 and eth1 without any assigned ip addresses), put it between
end node and the cisco. tcpdump there, which should give an unbiased
view wrt. endnode/cisco. Then perhaps, also configure such a network
listening bridge on the other side of the cisco, e.g. on the link to
the internet and watch that. Compare the two tcpdumpds and see if
sack got trashed.

Re: After many hours all outbound connections get stuck in SYN_SENT

[James Nichols]

> Huh?  Do you mean a PIX blade in a Cisco switch-router chassis? It
> would be very useful if you could be less vague about the
> equipment in use.

Right it's a PIX blade in Cisco chassis.  The PIX is running ASA version 7.0(6)



> That depends more on your customers' networking attributes
> then you are sharing or perhaps even know.  Perhaps your customer
> base is very Window-skewed and you simply aren't seeing any Sack
> Permitted negotiations for the first 37.999 hours. Or
> perhaps you've had a network glitch, and all of your
> connections have done a Selective Ack, which the firewall
> has trashed, leaving all the connections in a wacko state,
> not just a few which you haven't noticed.

I definitely see SACKs over the course of the 38 hours.  I don't have
any Windows hosts, they are all running Linux except for a very small
number that run a proprietary OS and webserver.  If the firewall were
to get trashed and hose the currently active connections, I would
expect that newly initiated connections would work.


> The actual failure mode needs a packet trace to determine,
> but you should be able to do this yourself (or ask your
> local network engineering staff).
>
> If your firewall is trashing the Sack field, then it needs
> to be fixed.  Time to raise a case with the Cisco TAC and
> ask them directly if your PIX version has bug CSCse14419.
> You can't expect Sack to work when it's being fed trash,
> so it is important to make sure that is not happening.

I've pinged our dude that handles the PIX stuff to see about getting
an upgrade to 7.0(7).  I should be able to get a packet trace, but it
may take some time.  At this point I'm getting a lot of resistance and
people here telling me to just turn SACK off and not worry about what
is causing this issue, but I'd really like to get to the bottom of it.