Why does this connection stop being tracked?

Why does this connection stop being tracked?

[Andy Smith]

[-- Attachment #1: Type: text/plain, Size: 7828 bytes --]

Hi,

This is rather a long email and so I hope that someone who knows
about netfilter, bridging and possibly Xen will have patience to
read it all the way through.

I have a server that I run Xen
(http://www.cl.cam.ac.uk/Research/SRG/netos/xen/) on, with 6 xen
user domains (virtual machines).

For those unfamiliar with Xen, the dom0 (host machine) has a virtual
network interface for each user domain and each of those virtual
interfaces are bridged onto xen-br0, along with the machine's real
eth0.  In each user domain, the virtual interface appears as eth0.
        
In dom0 I have iptables running, with the eb-nf support of linux
2.6.11 and the physdev module loaded so that I can match traffic
coming in to each of my user domains.

Part of my ruleset looks like this:

        $IPT -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT

        $IPT -A FORWARD -m physdev --physdev-out vif+ -j domu_forward_in
        $IPT -A FORWARD -m physdev --physdev-in vif+ -j domu_forward_out

        ######################################################################
        # strugglers.net
        ######################################################################

        $IPT -A domu_forward_in -m physdev --physdev-out vif-struggler.0 -j domu_forward_in_strugglers
        $IPT -A domu_forward_in_strugglers -p tcp --syn -j domu_forward_in_strugglers_tcp
        $IPT -A domu_forward_in_strugglers_tcp -p tcp --dport 22 -j ACCEPT
        $IPT -A domu_forward_in_strugglers -m limit --limit 1/s -j LOG --log-prefix "FWD DROP: "
        $IPT -A domu_forward_in_strugglers -j DROP

Now, I have noticed that while this works most of the time, for
reasons unknown to me, some TCP connections just seem to stop being
tracked and hit the DROP rule.  Even though they have been tracked
fine for several hours.  This happens on every user domain to all
kinds of TCP connections, but I have pared the ruleset down to just
the one domain (strugglers.net) and SSH to demonstrate.

If I add a rule in domu_forward_in_strugglers to allow all TCP to
port 22 regardless of state the I have no problems.

This does not seem to affect the INPUT table where I have a similar
set of rules.

Today I decided to take a tcpdump while I was ssh'd in up until when
it kicked me out.  I ssh'd in at approx 13:07 GMT and got kicked out
at approx 15:32:49 GMT.  Here is a selection of what got logged on
the console of dom0:

Jun 14 15:32:34 curacao kernel: FWD DROP: IN=xen-br0 OUT=xen-br0 PHYSIN=eth0 PHYSOUT=vif-struggler.0 SRC=82.44.131.131 DST=212.13.198.70 LEN=92 TOS=0x00 PREC=0x00 TTL=119 ID=23572 DF PROTO=TCP SPT=31994 DPT=22 WINDOW=63492 RES=0x00 ACK PSH URGP=0
Jun 14 15:32:35 curacao kernel: FWD DROP: IN=xen-br0 OUT=xen-br0 PHYSIN=eth0 PHYSOUT=vif-struggler.0 SRC=82.44.131.131 DST=212.13.198.70 LEN=820 TOS=0x00 PREC=0x00 TTL=118 ID=23647 DF PROTO=TCP SPT=31994 DPT=22 WINDOW=64292 RES=0x00 ACK PSH URGP=0
Jun 14 15:32:37 curacao kernel: FWD DROP: IN=xen-br0 OUT=xen-br0 PHYSIN=eth0 PHYSOUT=vif-struggler.0 SRC=82.44.131.131 DST=212.13.198.70 LEN=820 TOS=0x00 PREC=0x00 TTL=118 ID=23675 DF PROTO=TCP SPT=31994 DPT=22 WINDOW=64292 RES=0x00 ACK PSH URGP=0
Jun 14 15:32:38 curacao kernel: FWD DROP: IN=xen-br0 OUT=xen-br0 PHYSIN=eth0 PHYSOUT=vif-struggler.0 SRC=82.44.131.131 DST=212.13.198.70 LEN=40 TOS=0x00 PREC=0x00 TTL=118 ID=23685 DF PROTO=TCP SPT=31994 DPT=22 WINDOW=64292 RES=0x00 ACK URGP=0
Jun 14 15:32:41 curacao kernel: FWD DROP: IN=xen-br0 OUT=xen-br0 PHYSIN=eth0 PHYSOUT=vif-struggler.0 SRC=82.44.131.131 DST=212.13.198.70 LEN=92 TOS=0x00 PREC=0x00 TTL=118 ID=23708 DF PROTO=TCP SPT=31994 DPT=22 WINDOW=64292 RES=0x00 ACK PSH URGP=0
Jun 14 15:32:41 curacao kernel: FWD DROP: IN=xen-br0 OUT=xen-br0 PHYSIN=eth0 PHYSOUT=vif-struggler.0 SRC=82.44.131.131 DST=212.13.198.70 LEN=40 TOS=0x00 PREC=0x00 TTL=118 ID=23719 DF PROTO=TCP SPT=31994 DPT=22 WINDOW=64292 RES=0x00 ACK URGP=0
Jun 14 15:32:41 curacao kernel: FWD DROP: IN=xen-br0 OUT=xen-br0 PHYSIN=eth0 PHYSOUT=vif-struggler.0 SRC=82.44.131.131 DST=212.13.198.70 LEN=92 TOS=0x00 PREC=0x00 TTL=118 ID=23729 DF PROTO=TCP SPT=31994 DPT=22 WINDOW=64292 RES=0x00 ACK PSH URGP=0
Jun 14 15:32:42 curacao kernel: FWD DROP: IN=xen-br0 OUT=xen-br0 PHYSIN=eth0 PHYSOUT=vif-struggler.0 SRC=82.44.131.131 DST=212.13.198.70 LEN=924 TOS=0x00 PREC=0x00 TTL=118 ID=23743 DF PROTO=TCP SPT=31994 DPT=22 WINDOW=64292 RES=0x00 ACK PSH URGP=0
Jun 14 15:32:45 curacao kernel: FWD DROP: IN=xen-br0 OUT=xen-br0 PHYSIN=eth0 PHYSOUT=vif-struggler.0 SRC=82.44.131.131 DST=212.13.198.70 LEN=76 TOS=0x00 PREC=0x00 TTL=118 ID=23757 DF PROTO=TCP SPT=31994 DPT=22 WINDOW=64292 RES=0x00 ACK PSH URGP=0
Jun 14 15:32:49 curacao kernel: FWD DROP: IN=xen-br0 OUT=xen-br0 PHYSIN=eth0 PHYSOUT=vif-struggler.0 SRC=82.44.131.131 DST=212.13.198.70 LEN=40 TOS=0x00 PREC=0x00 TTL=118 ID=23788 DF PROTO=TCP SPT=31994 DPT=22 WINDOW=64292 RES=0x00 ACK URGP=0

At the same time I see a lot of TCP connections suddenly being
denied to a number of other user domains, so I suspect that all TCP
connect tracking was purged then for some reason.

Although I was kicked out, I was able to reconnect straight away (as you would
expect from the above ruleset, it allows the SYN to port 22 and away we go) and
in fact that is how I am typing this email to you now.

Here is the bridge setup:

[andy@curacao src]$ brctl show
bridge name     bridge id               STP enabled     interfaces
xen-br0         8000.00e081641d07       no              eth0
                                                        vif-admin.0
                                                        vif-cholet.0
                                                        vif-outpostlo.0
                                                        vif-ruminant.0
                                                        vif-seinfeld.0
                                                        vif-struggler.0
[andy@curacao src]$ ip link
1: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:e0:81:64:1d:07 brd ff:ff:ff:ff:ff:ff
2: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
    link/ether 00:e0:81:64:1d:08 brd ff:ff:ff:ff:ff:ff
3: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
4: sit0: <NOARP> mtu 1480 qdisc noop
    link/sit 0.0.0.0 brd 0.0.0.0
5: xen-br0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
    link/ether 00:e0:81:64:1d:07 brd ff:ff:ff:ff:ff:ff
6: vif-admin.0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
7: vif-cholet.0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
8: vif-outpostlo.0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
9: vif-ruminant.0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
10: vif-seinfeld.0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
11: vif-struggler.0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff

As I said above, I did a:

[andy@curacao src]$ sudo tcpdump -w /tmp/xen-br0.dump -i xen-br0 'host 212.13.198.70 and host 82.44.131.131'

just after ssh'ing in and left it running until just after my ssh client
gave up.  That file (1.5MB) can be found here:

http://strugglers.net/~andy/tmp/xen-br0.dump

But I cannot see anything obviously wrong with it.

Anyone have any ideas?  I can give up on connection tracking for my
user domains but it's troubling that it doesn't work.  Is it an
issue with using a bridge?

Thanks,
Andy

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Why does this connection stop being tracked?

[Jozsef Kadlecsik]

On Tue, 14 Jun 2005, Andy Smith wrote:

> In dom0 I have iptables running, with the eb-nf support of linux
> 2.6.11 and the physdev module loaded so that I can match traffic
> coming in to each of my user domains.
[...]
> Now, I have noticed that while this works most of the time, for
> reasons unknown to me, some TCP connections just seem to stop being
> tracked and hit the DROP rule.  Even though they have been tracked
> fine for several hours.  This happens on every user domain to all
> kinds of TCP connections, but I have pared the ruleset down to just
> the one domain (strugglers.net) and SSH to demonstrate.

You have two choices: either disable TCP SACK support on all your
real/virtual machines behind your firewall, or upgrade the kernel on the
firewall. There is a SACK related bug in netfilter connection tracking in
2.6.11 (and below).  According to the dumped traffic your connections
suffer from packet losses, SACK kicks in and conntrack screws up tracking
the given TCP connections. (Sorry, I can't recall at which rc release was
the fix submitted in.)

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: Why does this connection stop being tracked?

[Andy Smith]

[-- Attachment #1: Type: text/plain, Size: 1686 bytes --]

Thanks Jozsef for looking at this.

On Wed, Jun 15, 2005 at 01:18:38PM +0200, Jozsef Kadlecsik wrote:
> On Tue, 14 Jun 2005, Andy Smith wrote:
> 
> > In dom0 I have iptables running, with the eb-nf support of linux
> > 2.6.11 and the physdev module loaded so that I can match traffic
> > coming in to each of my user domains.
> [...]
> > Now, I have noticed that while this works most of the time, for
> > reasons unknown to me, some TCP connections just seem to stop being
> > tracked and hit the DROP rule.  Even though they have been tracked
> > fine for several hours.  This happens on every user domain to all
> > kinds of TCP connections, but I have pared the ruleset down to just
> > the one domain (strugglers.net) and SSH to demonstrate.
> 
> You have two choices: either disable TCP SACK support on all your
> real/virtual machines behind your firewall, or upgrade the kernel on the
> firewall.

Do you have any instructions or a pointer to documentation onhow to
temporarily disable SACK?  If it was a /proc setting that would be
ideal; I don't really want to have to recompile kernels though.

> There is a SACK related bug in netfilter connection tracking in
> 2.6.11 (and below).  According to the dumped traffic your connections
> suffer from packet losses,
        
Interesting; this may explain why I only notice this when I'm coming
from 82.44.131.131 - its network is kind of sucky. :)

>        SACK kicks in and conntrack screws up tracking
> the given TCP connections. (Sorry, I can't recall at which rc release was
> the fix submitted in.)

How sure are you that this is the problem I am seeing?

Thanks again for your help.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Why does this connection stop being tracked?

[Jozsef Kadlecsik]

On Wed, 15 Jun 2005, Andy Smith wrote:

> > You have two choices: either disable TCP SACK support on all your
> > real/virtual machines behind your firewall, or upgrade the kernel on the
> > firewall.
>
> Do you have any instructions or a pointer to documentation onhow to
> temporarily disable SACK?  If it was a /proc setting that would be
> ideal; I don't really want to have to recompile kernels though.

echo 0 > /proc/sys/net/ipv4/tcp_sack

> > There is a SACK related bug in netfilter connection tracking in
> > 2.6.11 (and below).  According to the dumped traffic your connections
> > suffer from packet losses,
>
> Interesting; this may explain why I only notice this when I'm coming
> from 82.44.131.131 - its network is kind of sucky. :)
>
> >        SACK kicks in and conntrack screws up tracking
> > the given TCP connections. (Sorry, I can't recall at which rc release was
> > the fix submitted in.)
>
> How sure are you that this is the problem I am seeing?

The dump file shows that the communicating parties advertise sack support
and later on in the traffic they do use sack options. And because living
connections hangs up, that indicates the sack bug. You can simply check it
by disabling sack.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: Why does this connection stop being tracked?

[R. DuFresne]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


 	[SNIP]

>>
>> You have two choices: either disable TCP SACK support on all your
>> real/virtual machines behind your firewall, or upgrade the kernel on the
>> firewall.
>
> Do you have any instructions or a pointer to documentation onhow to
> temporarily disable SACK?  If it was a /proc setting that would be
> ideal; I don't really want to have to recompile kernels though.
>

why?  you are certainly missing out on how to fix and patch a systems when 
bugs in the kernel affect it, to the ability to add features that your 
dist maintainer has not enabled by default, or to change params in the 
kernel such as moving away or to kernel modules as opposed to stack 
functionality mapping.  Not to mention the abilities to streamline the 
kernel to fit your requirements and remove all the xtra trash that gets 
loaded in to make a kernel fit all purposes/needs/enduser-requirements.

basically, you are defeating one of the finer points in the linux realm <as 
well as the BSD's net, open, free>  you are avoiding taking actually 
control of what you are playing with <smile>.  Granted one does not do 
this sort of thing in a prod env on the fly, one tests such things on a 
dev server or desktop emulating what might be in prod. but, it's not all 
that tough to master, and certainly will likely be required at one time or 
another to get things working that were not originally provided, move to a 
newer cleaner kernel, or even to fix problems encountered over the 
stresses of time and all that.  The recipe for doing such is not all that 
complex, and if one backsup the old kernel and properly runs lilo to 
include it in the potential boot process, not all that damaging should on 
finger-fart and make a bed new kernel on first draft. but all admins in 
the free *nix-like realm should learn the particulars of rebuilding 
kernels, it will  at one time or another save their asses.

No salt for the avoiders.

Thanks,

Ron DuFresne
- -- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
         admin & senior security consultant:  sysinfo.com
                         http://sysinfo.com
Key fingerprint = 9401 4B13 B918 164C 647A  E838 B2DF AFCC 94B0 6629

...We waste time looking for the perfect lover
instead of creating the perfect love.

                 -Tom Robbins <Still Life With Woodpecker>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFCsFJdst+vzJSwZikRAiaQAKCWHlgggJUxBXu9/CeR//pLYbzHGACfRVev
kG/17gNRcUin+Dk63ai8gCA=
=2VQV
-----END PGP SIGNATURE-----

Re: Why does this connection stop being tracked?

[Andy Smith]

[-- Attachment #1: Type: text/plain, Size: 1314 bytes --]

On Wed, Jun 15, 2005 at 12:07:52PM -0400, R. DuFresne wrote:
> >> You have two choices: either disable TCP SACK support on all your
> >> real/virtual machines behind your firewall, or upgrade the kernel on the
> >> firewall.
> >
> > Do you have any instructions or a pointer to documentation onhow to
> > temporarily disable SACK?  If it was a /proc setting that would be
> > ideal; I don't really want to have to recompile kernels though.
> >
> 
> why?  you are certainly missing out on how to fix and patch a systems when 
> bugs in the kernel affect it, to the ability to add features that your 
> dist maintainer has not enabled by default, or to change params in the 
> kernel such as moving away or to kernel modules as opposed to stack 
> functionality mapping.

I'm sorry, I didn't phrase that very well.  I'm perfectly happy to
compile new kernels and indeed I am required to run a patched 2.6.11
plus some other patches that I have to apply manually in order to
use Xen.

$ uname -a
Linux curacao.strugglers.net 2.6.11curacaoxen0-steven-hand1 #1 Sat Jun 4 18:49:26 UTC 2005 i686 GNU/Linux

I just didn't want to make a new kernel and reboot in order to test
the suggestion of disabling SACK as the downtime of a reboot on a
machine with multiple virtual machines is unpopular.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Why does this connection stop being tracked?

[R. DuFresne]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, 15 Jun 2005, Andy Smith wrote:

> On Wed, Jun 15, 2005 at 12:07:52PM -0400, R. DuFresne wrote:
>>>> You have two choices: either disable TCP SACK support on all your
>>>> real/virtual machines behind your firewall, or upgrade the kernel on the
>>>> firewall.
>>>
>>> Do you have any instructions or a pointer to documentation onhow to
>>> temporarily disable SACK?  If it was a /proc setting that would be
>>> ideal; I don't really want to have to recompile kernels though.
>>>
>>
>> why?  you are certainly missing out on how to fix and patch a systems when
>> bugs in the kernel affect it, to the ability to add features that your
>> dist maintainer has not enabled by default, or to change params in the
>> kernel such as moving away or to kernel modules as opposed to stack
>> functionality mapping.
>
> I'm sorry, I didn't phrase that very well.  I'm perfectly happy to
> compile new kernels and indeed I am required to run a patched 2.6.11
> plus some other patches that I have to apply manually in order to
> use Xen.
>
> $ uname -a
> Linux curacao.strugglers.net 2.6.11curacaoxen0-steven-hand1 #1 Sat Jun 4 18:49:26 UTC 2005 i686 GNU/Linux
>
> I just didn't want to make a new kernel and reboot in order to test
> the suggestion of disabling SACK as the downtime of a reboot on a
> machine with multiple virtual machines is unpopular.
>


Oh, well, then that is a different situation <smile>, nevermind...

Thanks,


Ron DuFresne
- -- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
         admin & senior security consultant:  sysinfo.com
                         http://sysinfo.com
Key fingerprint = 9401 4B13 B918 164C 647A  E838 B2DF AFCC 94B0 6629

...We waste time looking for the perfect lover
instead of creating the perfect love.

                 -Tom Robbins <Still Life With Woodpecker>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFCsFY/st+vzJSwZikRAmQoAJ9ZztFRW/7lDOvnX/cFIfJ0AttoFQCfeZ5S
DpMMHOGiMSpHBcFH3koU4Yg=
=eJeJ
-----END PGP SIGNATURE-----