Bug with Fedora's 2.6.23.9-85 kernel (at least) and ESTABLISHED and SACK

Bug with Fedora's 2.6.23.9-85 kernel (at least) and ESTABLISHED and SACK

[Zan Lynx]

Please CC me on any replies as I am not subscribed.

I was downloading a new Google Earth when I noticed a LOT of max-size 
dropped packets in my firewall log.  I only allow RELATED,ESTABLISHED 
sessions into my firewall.

tcpdump showed that every time Google sent a packet to satisfy the 
missing data identified by SACK, that packet was rejected.  So it must 
have been missing the ESTABLISHED rule.

I fixed the problem by adding an ALLOW source port 80 rule for the 
Google download site IP.

This makes me wonder how often this has happened and I haven't noticed 
it.  Is this a known bug or something new?

BTW, your netfilter Bugzilla is dead or at least 404 missing.

Re: Bug with Fedora's 2.6.23.9-85 kernel (at least) and ESTABLISHED and SACK

[Jan Engelhardt]

On Jan 27 2008 22:14, Zan Lynx wrote:
>
> Please CC me on any replies as I am not subscribed.
>
> I was downloading a new Google Earth when I noticed a LOT of
> max-size dropped packets in my firewall log.  I only allow
> RELATED,ESTABLISHED sessions into my firewall.
>
> tcpdump showed that every time Google sent a packet to satisfy the
> missing data identified by SACK, that packet was rejected.  So it
> must have been missing the ESTABLISHED rule.

You can check the connection state with `conntrack -L`, or in
/proc/net/{ip,nf}_conntrack. What does it read?

Re: Bug with Fedora's 2.6.23.9-85 kernel (at least) and ESTABLISHED and SACK

[Zan Lynx]

[-- Attachment #1: Type: text/plain, Size: 822 bytes --]


On Mon, 2008-01-28 at 10:53 +0100, Jan Engelhardt wrote:
> On Jan 27 2008 22:14, Zan Lynx wrote:
> >
> > Please CC me on any replies as I am not subscribed.
> >
> > I was downloading a new Google Earth when I noticed a LOT of
> > max-size dropped packets in my firewall log.  I only allow
> > RELATED,ESTABLISHED sessions into my firewall.
> >
> > tcpdump showed that every time Google sent a packet to satisfy the
> > missing data identified by SACK, that packet was rejected.  So it
> > must have been missing the ESTABLISHED rule.
> 
> You can check the connection state with `conntrack -L`, or in
> /proc/net/{ip,nf}_conntrack. What does it read?

I am sorry.  The download completed and I have not yet had time to
reproduce it.  Perhaps I can get something tonight.
-- 
Zan Lynx <zlynx@acm.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Bug with Fedora's 2.6.23.9-85 kernel (at least) and ESTABLISHED and SACK

[Krzysztof Oledzki]

On Sun, 27 Jan 2008, Zan Lynx wrote:

> Please CC me on any replies as I am not subscribed.
>
> I was downloading a new Google Earth when I noticed a LOT of max-size dropped 
> packets in my firewall log.  I only allow RELATED,ESTABLISHED sessions into 
> my firewall.
>
> tcpdump showed that every time Google sent a packet to satisfy the missing 
> data identified by SACK, that packet was rejected.  So it must have been 
> missing the ESTABLISHED rule.
>
> I fixed the problem by adding an ALLOW source port 80 rule for the Google 
> download site IP.
>
> This makes me wonder how often this has happened and I haven't noticed it. 
> Is this a known bug or something new?

Which kernel version?

Most likely there is a broken box along the path that mangles seq numbers 
but forgets about sacks. Could you please provide a dump:
"tcpdump -v -S host (...)"?

As a short-term workaround you may disable net.ipv4.tcp_sack or use 
TCPOPTSTRIP to strip SackOK advertisement to the host.

Best regards,

 				Krzysztof Oledzki

Re: Bug with Fedora's 2.6.23.9-85 kernel (at least) and ESTABLISHED and SACK

[Zan Lynx]

[-- Attachment #1: Type: text/plain, Size: 1759 bytes --]


On Mon, 2008-01-28 at 14:38 +0100, Krzysztof Oledzki wrote:
> 
> On Sun, 27 Jan 2008, Zan Lynx wrote:
> 
> > Please CC me on any replies as I am not subscribed.
> >
> > I was downloading a new Google Earth when I noticed a LOT of max-size dropped 
> > packets in my firewall log.  I only allow RELATED,ESTABLISHED sessions into 
> > my firewall.
> >
> > tcpdump showed that every time Google sent a packet to satisfy the missing 
> > data identified by SACK, that packet was rejected.  So it must have been 
> > missing the ESTABLISHED rule.
> >
> > I fixed the problem by adding an ALLOW source port 80 rule for the Google 
> > download site IP.
> >
> > This makes me wonder how often this has happened and I haven't noticed it. 
> > Is this a known bug or something new?
> 
> Which kernel version?

As in the Subject, Fedora's 2.6.23.9-85.

> 
> Most likely there is a broken box along the path that mangles seq numbers 
> but forgets about sacks. Could you please provide a dump:
> "tcpdump -v -S host (...)"?

I have not yet been able to reproduce it so I cannot provide a dump.

I doubt that the sequence numbers were being mangled.  I doubt this
because as soon as I added the rule to accept all port 80 packets from
the server IP, the TCP session accepted the packet with the missing data
and the hole described by the SACK options disappeared.  The session was
sourced from the machine itself via a Squid proxy.

> 
> As a short-term workaround you may disable net.ipv4.tcp_sack or use 
> TCPOPTSTRIP to strip SackOK advertisement to the host.

Yes, I could do this if it seems to happen again.  I will keep an eye on
it.

> 
> Best regards,
> 
>  				Krzysztof Oledzki
-- 
Zan Lynx <zlynx@acm.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Bug with Fedora's 2.6.23.9-85 kernel (at least) and ESTABLISHED and SACK

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1802 bytes --]



On Mon, 28 Jan 2008, Zan Lynx wrote:

>
> On Mon, 2008-01-28 at 14:38 +0100, Krzysztof Oledzki wrote:
>>
>> On Sun, 27 Jan 2008, Zan Lynx wrote:
>>
>>> Please CC me on any replies as I am not subscribed.
>>>
>>> I was downloading a new Google Earth when I noticed a LOT of max-size dropped
>>> packets in my firewall log.  I only allow RELATED,ESTABLISHED sessions into
>>> my firewall.
>>>
>>> tcpdump showed that every time Google sent a packet to satisfy the missing
>>> data identified by SACK, that packet was rejected.  So it must have been
>>> missing the ESTABLISHED rule.
>>>
>>> I fixed the problem by adding an ALLOW source port 80 rule for the Google
>>> download site IP.
>>>
>>> This makes me wonder how often this has happened and I haven't noticed it.
>>> Is this a known bug or something new?
>>
>> Which kernel version?
>
> As in the Subject, Fedora's 2.6.23.9-85.

Right, sorry. I must have overlooked this.


>> Most likely there is a broken box along the path that mangles seq numbers
>> but forgets about sacks. Could you please provide a dump:
>> "tcpdump -v -S host (...)"?
>
> I have not yet been able to reproduce it so I cannot provide a dump.
>
> I doubt that the sequence numbers were being mangled.  I doubt this
> because as soon as I added the rule to accept all port 80 packets from
> the server IP, the TCP session accepted the packet with the missing data
> and the hole described by the SACK options disappeared.  The session was
> sourced from the machine itself via a Squid proxy.

It is because most tcp stacks (including the one available in Linux) are 
less restrictive than netfilter code and allow tcp packets with invalid 
sacks but valid acks by ignoring sack data.


Best regards,

 			Krzysztof Olędzki