[Fwd: [Bug 3397] New: Network connections hang going through an OpenBSD firewall]

[Fwd: [Bug 3397] New: Network connections hang going through an OpenBSD firewall]

[Nivedita Singhvi]

Can you reproduce on the latest kernel, please?
Is the OpenBSD mangling the packet in any way?

Can anyone tell me if this smells like something
recently fixed (MTU issues)? Doesn't sound like
the windowscaling problem but could be related.


thanks,
Nivedita


-------- Original Message --------
Subject: [Bug 3397] New: Network connections hang going through an
OpenBSD firewall
Date: Tue, 14 Sep 2004 09:38:24 -0700
From: bugme-daemon@osdl.org
To: niv@us.ibm.com

http://bugme.osdl.org/show_bug.cgi?id=3397

            Summary: Network connections hang going through an OpenBSD
                     firewall
     Kernel Version: 2.6.6+
             Status: NEW
           Severity: blocking
              Owner: niv@us.ibm.com
          Submitter: vuksan-hoforums@veus.hr


Distribution: Fedora Core 2, Gentoo
Hardware Environment: All
Software Environment: All
Problem Description:
We have seen a number of issues with people accessing our website
http://www.cs.unm.edu/ using kernels 2.6.6+. Doing some network sniffing
we can
see that data request is received by the web server and the web server
responds
however after certain amount of bytes it simply stops.

I have played with MTU sizes and if Ethereal is to be believed the transfer
stops after MTU + 77 bytes.

This has been reported to us by a number of different people running
different
distributions ie. Fedora Core 2, Gentoo. For example 2.6.5 kernel that comes
with FC2 works. Secure IMAP and Secure POP don't seem to work either
using 2.6.6+.

What is even stranger is that SSH connections don't exhibit this kind of a
problem ie. you can SSH withouth a hitch.

A data point is that we are using transparent (in-line) OpenBSD firewall
with
Packetfilter.


Steps to reproduce:

Boot into 2.6.6+ kernel try pulling up http://www.cs.unm.edu/ through a web
browser. It won't show up. Boot into a 2.6.5 and below and the page will
show up.

Any clues will be appreciated.

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

Re: [Fwd: [Bug 3397] New: Network connections hang going through an OpenBSD firewall]

[Stephen Hemminger]

On Tue, 14 Sep 2004 14:00:26 -0700
Nivedita Singhvi <niv@us.ibm.com> wrote:

> Can you reproduce on the latest kernel, please?
> Is the OpenBSD mangling the packet in any way?
> 
> Can anyone tell me if this smells like something
> recently fixed (MTU issues)? Doesn't sound like
> the windowscaling problem but could be related.

OpenBSD pf is easy to configure to break window scaling.
The developer claims its not a bug. Basically stateless
TCP connection tracking will never work, it's a bad idea.

Daniel Hartmeier <daniel@benzedrine.cx> wrote:

> The problem arises when the user creates a complicated ruleset that
> passes the first SYN of a connection without creating a state entry. You
> might argue that this shouldn't be possible, but some forms of stateless
> filtering are being used. The man page warns against this, too.
> 
> In this particular case, the ruleset tells pf to pass the initial SYN
> without creating state (and therefore without any place to note the
> window option and first scale factor). When, later, the SYN+ACK creates
> state, the state just doesn't contain the information to follow the
> scaled windows.

Re: [Fwd: [Bug 3397] New: Network connections hang going through an OpenBSD firewall]

[Vladimir]

Nivedita Singhvi wrote:

> Can you reproduce on the latest kernel, please?

I will try it with the latest kernel. I have tried this with kernel 
2.6.8-1.521 from

http://www.atrpms.net/dist/

It doesn't work with it. Should I download 2.6.8-1 from kernel.org and 
compile it by hand ?

> Is the OpenBSD mangling the packet in any way?


It is certainly possible but everything works fine with all other OSes 
and kernels 2.6.5 and below which leads me to believe something was 
changed in 2.6.6 that broke it.

Are you able to reproduce it ?

Thanks,

Vladimir Vuksan

> Can anyone tell me if this smells like something
> recently fixed (MTU issues)? Doesn't sound like
> the windowscaling problem but could be related.



> http://bugme.osdl.org/show_bug.cgi?id=3397

Re: [Fwd: [Bug 3397] New: Network connections hang going through an OpenBSD firewall]

[David S. Miller]

On Tue, 14 Sep 2004 15:09:50 -0600
Vladimir <vlists@veus.hr> wrote:

> > Is the OpenBSD mangling the packet in any way?
> 
> It is certainly possible but everything works fine with all other OSes 
> and kernels 2.6.5 and below which leads me to believe something was 
> changed in 2.6.6 that broke it.
> 
> Are you able to reproduce it ?

OpenBSD packet filter is busted, and the maintainer of
it claims this is not a bug.

That changes in 2.6.6 didn't "break" things, it enabled a
feature in TCP that OpenBSD stateless TCP connection tracking
cannot handle, and old TCP feature in fact, window scaling.

See here for more info:

http://lwn.net/Articles/92727/

Re: [Fwd: [Bug 3397] New: Network connections hang going through an OpenBSD firewall]

[Nivedita Singhvi]

Vladimir wrote:

> It doesn't work with it. Should I download 2.6.8-1 from kernel.org and 
> compile it by hand ?

If you can, download the latest (linux-2.6.9-rc2). You can
get the patch against linux-2.6.8.1 vanilla from
kernel.org.

> It is certainly possible but everything works fine with all other OSes 
> and kernels 2.6.5 and below which leads me to believe something was 
> changed in 2.6.6 that broke it.
> 
> Are you able to reproduce it ?

Unfortunately, I have zero cycles right now (and in the near
future) to address any issues, I'm afraid. Someone on netdev
will likely be able to help you or have run into it already.
See Stephen's post from a few minutes ago..

thanks,
Nivedita

Re: [Fwd: [Bug 3397] New: Network connections hang going through an OpenBSD firewall]

[Vladimir]

David S. Miller wrote:

>OpenBSD packet filter is busted, and the maintainer of
>it claims this is not a bug.
>
>That changes in 2.6.6 didn't "break" things, it enabled a
>feature in TCP that OpenBSD stateless TCP connection tracking
>cannot handle, and old TCP feature in fact, window scaling.
>
>See here for more info:
>
>http://lwn.net/Articles/92727/
>
Thanks. We were able to fix our firewall so things work properly now. 
The problem is that this is "insidious" since it is not immediately 
apparent what the problem is especially since it tends to work with all 
other OSes except Linux with 2.6.6+.

I will note this on the bug I submitted and close it.

Thanks a lot,

Vladimir Vuksan

Re: [Fwd: [Bug 3397] New: Network connections hang going through an OpenBSD firewall]

[Tomasz Torcz]

On Tue, Sep 14, 2004 at 02:21:41PM -0700, Nivedita Singhvi wrote:
> If you can, download the latest (linux-2.6.9-rc2). You can
> get the patch against linux-2.6.8.1 vanilla from
> kernel.org.

 Patch against 2.6.8. It won't apply clean on 2.6.8.1.

-- 
Tomasz Torcz                 Morality must always be based on practicality.
zdzichu@irc.-nie.spam-.pl                -- Baron Vladimir Harkonnen

Re: [Fwd: [Bug 3397] New: Network connections hang going through an OpenBSD firewall]

[Stephen Hemminger]

On Tue, 14 Sep 2004 14:21:41 -0700
Nivedita Singhvi <niv@us.ibm.com> wrote:

> Vladimir wrote:
> 
> > It doesn't work with it. Should I download 2.6.8-1 from kernel.org and 
> > compile it by hand ?
> 
> If you can, download the latest (linux-2.6.9-rc2). You can
> get the patch against linux-2.6.8.1 vanilla from
> kernel.org.
> 
> > It is certainly possible but everything works fine with all other OSes 
> > and kernels 2.6.5 and below which leads me to believe something was 
> > changed in 2.6.6 that broke it.
> > 
> > Are you able to reproduce it ?
> 
> Unfortunately, I have zero cycles right now (and in the near
> future) to address any issues, I'm afraid. Someone on netdev
> will likely be able to help you or have run into it already.
> See Stephen's post from a few minutes ago..

Since the latest kernel chooses window scaling factor automatically, the 
value chosen by default is small enough that OpenBSD
busted pf will just make performance suck. If you want test it, you have to
increase tcp_rmem[2] or tcp_rmem_max to force a window scale > 2.