It seems I've found why conntrack blocks some packets

It seems I've found why conntrack blocks some packets

[Carlos Pastorino]

Hi everyone,

I always wondered why conntrack blocked some packets that were
supposed to pass through my ESTABLISHED,RELATED rule. Now it seems
that I've found the answer.

Bear with me, because there will be questions in the end.

So, what happens is: from time to time, I see my firewall blocking a
packet like this:

Mar 28 14:48:21 SRVA kernel: FORWARD blocked: IN=eth1 OUT=eth0
SRC=webserverip DST=customerip LEN=48 TOS=0x00 PREC=0x00 TTL=63 ID=0
DF PROTO=TCP SPT=80 DPT=10458 WINDOW=5840 RES=0x00 ACK SYN URGP=0

Well, it does call my attention because it's a blocked packet FROM my
webserver. In any case, it shouldn't be blocked, because the
connection is not even 2 minutes old.

But, on the webserver, I was running tcpdump this time, so I could see
what really happened:

14:46:47.573738 customerip.10458 > webserverip.80: S [tcp sum ok]
23512000:23512000(0) win 65535 <mss 1460,nop,nop,sackOK> (DF) (ttl
120, id 41065, len 48)
14:46:47.573747 webserverip.80 > customerip.10458: S [tcp sum ok]
4131634297:4131634297(0) ack 23512001 win 5840 <mss
1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
14:46:51.327629 webserverip.80 > customerip.10458: S [tcp sum ok]
4131634297:4131634297(0) ack 23512001 win 5840 <mss
1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
14:46:57.327623 webserverip.80 > customerip.10458: S [tcp sum ok]
4131634297:4131634297(0) ack 23512001 win 5840 <mss
1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
14:47:09.327609 webserverip.80 > customerip.10458: S [tcp sum ok]
4131634297:4131634297(0) ack 23512001 win 5840 <mss
1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
14:47:33.527575 webserverip.80 > customerip.10458: S [tcp sum ok]
4131634297:4131634297(0) ack 23512001 win 5840 <mss
1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
14:47:34.216642 customerip.10458 > webserverip.80: R [tcp sum ok]
0:0(0) ack 0 win 0 (ttl 26, id 1, len 40)
14:48:21.727515 webserverip.80 > customerip.10458: S [tcp sum ok]
4131634297:4131634297(0) ack 23512001 win 5840 <mss
1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)

As you can see, the customer connected with a SYN packet, and my
webserver responded with 6 ACK/SYN packets. BUT, before the 6th
ACK/SYN response, the customer sent an ACK/RST packet, resetting the
connection, and thus the 6th ACK/SYN packet was blocked by the
firewall because the connection was no longer in the conntrack. Yes,
clocks in the firewall and webserver are synchronized.

Questions are:

1) Does anyone have seen this happening too?

2) How can I silently drop that package, without compromising the on
going connections? I would like to get rid of those "false positives".

Thanks,

Carlos

Re: It seems I've found why conntrack blocks some packets

[Steven M Campbell]

Carlos Pastorino wrote:
> Hi everyone,
> 
> I always wondered why conntrack blocked some packets that were
> supposed to pass through my ESTABLISHED,RELATED rule. Now it seems
> that I've found the answer.
> 
> Bear with me, because there will be questions in the end.
> 
> So, what happens is: from time to time, I see my firewall blocking a
> packet like this:
> 
> Mar 28 14:48:21 SRVA kernel: FORWARD blocked: IN=eth1 OUT=eth0
> SRC=webserverip DST=customerip LEN=48 TOS=0x00 PREC=0x00 TTL=63 ID=0
> DF PROTO=TCP SPT=80 DPT=10458 WINDOW=5840 RES=0x00 ACK SYN URGP=0
> 
> Well, it does call my attention because it's a blocked packet FROM my
> webserver. In any case, it shouldn't be blocked, because the
> connection is not even 2 minutes old.
> 
> But, on the webserver, I was running tcpdump this time, so I could see
> what really happened:
> 
> 14:46:47.573738 customerip.10458 > webserverip.80: S [tcp sum ok]
> 23512000:23512000(0) win 65535 <mss 1460,nop,nop,sackOK> (DF) (ttl
> 120, id 41065, len 48)
> 14:46:47.573747 webserverip.80 > customerip.10458: S [tcp sum ok]
> 4131634297:4131634297(0) ack 23512001 win 5840 <mss
> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
> 14:46:51.327629 webserverip.80 > customerip.10458: S [tcp sum ok]
> 4131634297:4131634297(0) ack 23512001 win 5840 <mss
> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
> 14:46:57.327623 webserverip.80 > customerip.10458: S [tcp sum ok]
> 4131634297:4131634297(0) ack 23512001 win 5840 <mss
> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
> 14:47:09.327609 webserverip.80 > customerip.10458: S [tcp sum ok]
> 4131634297:4131634297(0) ack 23512001 win 5840 <mss
> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
> 14:47:33.527575 webserverip.80 > customerip.10458: S [tcp sum ok]
> 4131634297:4131634297(0) ack 23512001 win 5840 <mss
> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
> 14:47:34.216642 customerip.10458 > webserverip.80: R [tcp sum ok]
> 0:0(0) ack 0 win 0 (ttl 26, id 1, len 40)
> 14:48:21.727515 webserverip.80 > customerip.10458: S [tcp sum ok]
> 4131634297:4131634297(0) ack 23512001 win 5840 <mss
> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
> 
> As you can see, the customer connected with a SYN packet, and my
> webserver responded with 6 ACK/SYN packets. BUT, before the 6th
> ACK/SYN response, the customer sent an ACK/RST packet, resetting the
> connection, and thus the 6th ACK/SYN packet was blocked by the
> firewall because the connection was no longer in the conntrack. Yes,
> clocks in the firewall and webserver are synchronized.
> 
> Questions are:
> 
> 1) Does anyone have seen this happening too?
> 
> 2) How can I silently drop that package, without compromising the on
> going connections? I would like to get rid of those "false positives".
> 
> Thanks,
> 
> Carlos
> 

The real question here is what bad thing happened in the the data stream that the customer sent the reset packet?  The answer is not to ignore the reset but to find out why it is being sent, the client believes this connection should be aborted for some reason.

Re: It seems I've found why conntrack blocks some packets

[Roger Hamilton]

On 29/03/06, Steven M Campbell <Netfilter@scampbell.net> wrote:
> Carlos Pastorino wrote:
> > Hi everyone,
> >
> > I always wondered why conntrack blocked some packets that were
> > supposed to pass through my ESTABLISHED,RELATED rule. Now it seems
> > that I've found the answer.
> >
> > Bear with me, because there will be questions in the end.
> >
> > So, what happens is: from time to time, I see my firewall blocking a
> > packet like this:
> >
> > Mar 28 14:48:21 SRVA kernel: FORWARD blocked: IN=eth1 OUT=eth0
> > SRC=webserverip DST=customerip LEN=48 TOS=0x00 PREC=0x00 TTL=63 ID=0
> > DF PROTO=TCP SPT=80 DPT=10458 WINDOW=5840 RES=0x00 ACK SYN URGP=0
> >
> > Well, it does call my attention because it's a blocked packet FROM my
> > webserver. In any case, it shouldn't be blocked, because the
> > connection is not even 2 minutes old.
> >
> > But, on the webserver, I was running tcpdump this time, so I could see
> > what really happened:
> >
> > 14:46:47.573738 customerip.10458 > webserverip.80: S [tcp sum ok]
> > 23512000:23512000(0) win 65535 <mss 1460,nop,nop,sackOK> (DF) (ttl
> > 120, id 41065, len 48)
> > 14:46:47.573747 webserverip.80 > customerip.10458: S [tcp sum ok]
> > 4131634297:4131634297(0) ack 23512001 win 5840 <mss
> > 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
> > 14:46:51.327629 webserverip.80 > customerip.10458: S [tcp sum ok]
> > 4131634297:4131634297(0) ack 23512001 win 5840 <mss
> > 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
> > 14:46:57.327623 webserverip.80 > customerip.10458: S [tcp sum ok]
> > 4131634297:4131634297(0) ack 23512001 win 5840 <mss
> > 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
> > 14:47:09.327609 webserverip.80 > customerip.10458: S [tcp sum ok]
> > 4131634297:4131634297(0) ack 23512001 win 5840 <mss
> > 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
> > 14:47:33.527575 webserverip.80 > customerip.10458: S [tcp sum ok]
> > 4131634297:4131634297(0) ack 23512001 win 5840 <mss
> > 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
> > 14:47:34.216642 customerip.10458 > webserverip.80: R [tcp sum ok]
> > 0:0(0) ack 0 win 0 (ttl 26, id 1, len 40)
> > 14:48:21.727515 webserverip.80 > customerip.10458: S [tcp sum ok]
> > 4131634297:4131634297(0) ack 23512001 win 5840 <mss
> > 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
> >
> > As you can see, the customer connected with a SYN packet, and my
> > webserver responded with 6 ACK/SYN packets. BUT, before the 6th
> > ACK/SYN response, the customer sent an ACK/RST packet, resetting the
> > connection, and thus the 6th ACK/SYN packet was blocked by the
> > firewall because the connection was no longer in the conntrack. Yes,
> > clocks in the firewall and webserver are synchronized.
> >
> > Questions are:
> >
> > 1) Does anyone have seen this happening too?
> >
> > 2) How can I silently drop that package, without compromising the on
> > going connections? I would like to get rid of those "false positives".
> >
> > Thanks,
> >
> > Carlos
> >
>
> The real question here is what bad thing happened in the the data stream that the customer sent the reset packet?  The answer is not to ignore the reset but to find out why it is being sent, the client believes this connection should be aborted for some reason.
>
>
>
Doesn't this happen if the customer closes the browser/stops the web
page loading before the page has completely loaded? If so then one way
to ignore these packets is to silently drop ACK/SYN packets
originating from port 80.

Re: It seems I've found why conntrack blocks some packets

[Steven M Campbell]

Roger Hamilton wrote:
> On 29/03/06, Steven M Campbell <Netfilter@scampbell.net> wrote:
>> Carlos Pastorino wrote:
>>> Hi everyone,
>>>
>>> I always wondered why conntrack blocked some packets that were
>>> supposed to pass through my ESTABLISHED,RELATED rule. Now it seems
>>> that I've found the answer.
>>>
>>> Bear with me, because there will be questions in the end.
>>>
>>> So, what happens is: from time to time, I see my firewall blocking a
>>> packet like this:
>>>
>>> Mar 28 14:48:21 SRVA kernel: FORWARD blocked: IN=eth1 OUT=eth0
>>> SRC=webserverip DST=customerip LEN=48 TOS=0x00 PREC=0x00 TTL=63 ID=0
>>> DF PROTO=TCP SPT=80 DPT=10458 WINDOW=5840 RES=0x00 ACK SYN URGP=0
>>>
>>> Well, it does call my attention because it's a blocked packet FROM my
>>> webserver. In any case, it shouldn't be blocked, because the
>>> connection is not even 2 minutes old.
>>>
>>> But, on the webserver, I was running tcpdump this time, so I could see
>>> what really happened:
>>>
>>> 14:46:47.573738 customerip.10458 > webserverip.80: S [tcp sum ok]
>>> 23512000:23512000(0) win 65535 <mss 1460,nop,nop,sackOK> (DF) (ttl
>>> 120, id 41065, len 48)
>>> 14:46:47.573747 webserverip.80 > customerip.10458: S [tcp sum ok]
>>> 4131634297:4131634297(0) ack 23512001 win 5840 <mss
>>> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
>>> 14:46:51.327629 webserverip.80 > customerip.10458: S [tcp sum ok]
>>> 4131634297:4131634297(0) ack 23512001 win 5840 <mss
>>> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
>>> 14:46:57.327623 webserverip.80 > customerip.10458: S [tcp sum ok]
>>> 4131634297:4131634297(0) ack 23512001 win 5840 <mss
>>> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
>>> 14:47:09.327609 webserverip.80 > customerip.10458: S [tcp sum ok]
>>> 4131634297:4131634297(0) ack 23512001 win 5840 <mss
>>> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
>>> 14:47:33.527575 webserverip.80 > customerip.10458: S [tcp sum ok]
>>> 4131634297:4131634297(0) ack 23512001 win 5840 <mss
>>> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
>>> 14:47:34.216642 customerip.10458 > webserverip.80: R [tcp sum ok]
>>> 0:0(0) ack 0 win 0 (ttl 26, id 1, len 40)
>>> 14:48:21.727515 webserverip.80 > customerip.10458: S [tcp sum ok]
>>> 4131634297:4131634297(0) ack 23512001 win 5840 <mss
>>> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
>>>
>>> As you can see, the customer connected with a SYN packet, and my
>>> webserver responded with 6 ACK/SYN packets. BUT, before the 6th
>>> ACK/SYN response, the customer sent an ACK/RST packet, resetting the
>>> connection, and thus the 6th ACK/SYN packet was blocked by the
>>> firewall because the connection was no longer in the conntrack. Yes,
>>> clocks in the firewall and webserver are synchronized.
>>>
>>> Questions are:
>>>
>>> 1) Does anyone have seen this happening too?
>>>
>>> 2) How can I silently drop that package, without compromising the on
>>> going connections? I would like to get rid of those "false positives".
>>>
>>> Thanks,
>>>
>>> Carlos
>>>
>> The real question here is what bad thing happened in the the data stream that the customer sent the reset packet?  The answer is not to ignore the reset but to find out why it is being sent, the client believes this connection should be aborted for some reason.
>>
>>
>>
> Doesn't this happen if the customer closes the browser/stops the web
> page loading before the page has completely loaded? If so then one way
> to ignore these packets is to silently drop ACK/SYN packets
> originating from port 80.

The original arguement is that conntrack was blocking packets that it's not supposed to block.  It certainly should block packets that are arriving on a terminated connection.  I don't believe the issue here has anything to do with connection tracking being faulty and workarounds to that effect are only hidng the real issue.  Why did the customer not see the ack packets and finally send a rst?   

Re: It seems I've found why conntrack blocks some packets

[Carlos Pastorino]

On 3/29/06, Steven M Campbell <Netfilter@scampbell.net> wrote:
> Roger Hamilton wrote:
> > On 29/03/06, Steven M Campbell <Netfilter@scampbell.net> wrote:
> >> Carlos Pastorino wrote:
> >>> Hi everyone,
> >>>
> >>> I always wondered why conntrack blocked some packets that were
> >>> supposed to pass through my ESTABLISHED,RELATED rule. Now it seems
> >>> that I've found the answer.
> >>>
> >>> Bear with me, because there will be questions in the end.
> >>>
> >>> So, what happens is: from time to time, I see my firewall blocking a
> >>> packet like this:
> >>>
> >>> Mar 28 14:48:21 SRVA kernel: FORWARD blocked: IN=eth1 OUT=eth0
> >>> SRC=webserverip DST=customerip LEN=48 TOS=0x00 PREC=0x00 TTL=63 ID=0
> >>> DF PROTO=TCP SPT=80 DPT=10458 WINDOW=5840 RES=0x00 ACK SYN URGP=0
> >>>
> >>> Well, it does call my attention because it's a blocked packet FROM my
> >>> webserver. In any case, it shouldn't be blocked, because the
> >>> connection is not even 2 minutes old.
> >>>
> >>> But, on the webserver, I was running tcpdump this time, so I could see
> >>> what really happened:
> >>>
> >>> 14:46:47.573738 customerip.10458 > webserverip.80: S [tcp sum ok]
> >>> 23512000:23512000(0) win 65535 <mss 1460,nop,nop,sackOK> (DF) (ttl
> >>> 120, id 41065, len 48)
> >>> 14:46:47.573747 webserverip.80 > customerip.10458: S [tcp sum ok]
> >>> 4131634297:4131634297(0) ack 23512001 win 5840 <mss
> >>> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
> >>> 14:46:51.327629 webserverip.80 > customerip.10458: S [tcp sum ok]
> >>> 4131634297:4131634297(0) ack 23512001 win 5840 <mss
> >>> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
> >>> 14:46:57.327623 webserverip.80 > customerip.10458: S [tcp sum ok]
> >>> 4131634297:4131634297(0) ack 23512001 win 5840 <mss
> >>> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
> >>> 14:47:09.327609 webserverip.80 > customerip.10458: S [tcp sum ok]
> >>> 4131634297:4131634297(0) ack 23512001 win 5840 <mss
> >>> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
> >>> 14:47:33.527575 webserverip.80 > customerip.10458: S [tcp sum ok]
> >>> 4131634297:4131634297(0) ack 23512001 win 5840 <mss
> >>> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
> >>> 14:47:34.216642 customerip.10458 > webserverip.80: R [tcp sum ok]
> >>> 0:0(0) ack 0 win 0 (ttl 26, id 1, len 40)
> >>> 14:48:21.727515 webserverip.80 > customerip.10458: S [tcp sum ok]
> >>> 4131634297:4131634297(0) ack 23512001 win 5840 <mss
> >>> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
> >>>
> >>> As you can see, the customer connected with a SYN packet, and my
> >>> webserver responded with 6 ACK/SYN packets. BUT, before the 6th
> >>> ACK/SYN response, the customer sent an ACK/RST packet, resetting the
> >>> connection, and thus the 6th ACK/SYN packet was blocked by the
> >>> firewall because the connection was no longer in the conntrack. Yes,
> >>> clocks in the firewall and webserver are synchronized.
> >>>
> >>> Questions are:
> >>>
> >>> 1) Does anyone have seen this happening too?
> >>>
> >>> 2) How can I silently drop that package, without compromising the on
> >>> going connections? I would like to get rid of those "false positives".
> >>>
> >>> Thanks,
> >>>
> >>> Carlos
> >>>
> >> The real question here is what bad thing happened in the the data stream that the customer sent the reset packet?  The answer is not to ignore the reset but to find out why it is being sent, the client believes this connection should be aborted for some reason.
> >>
> >>
> >>
> > Doesn't this happen if the customer closes the browser/stops the web
> > page loading before the page has completely loaded? If so then one way
> > to ignore these packets is to silently drop ACK/SYN packets
> > originating from port 80.
>
> The original arguement is that conntrack was blocking packets that it's not supposed to block.  It certainly should block packets that are arriving on a terminated connection.  I don't believe the issue here has anything to do with connection tracking being faulty and workarounds to that effect are only hidng the real issue.  Why did the customer not see the ack packets and finally send a rst?
>
>

Well, actually, before I thought that iptables was blocking packets
incorrectly that I believed were supposed to be allowed because of
conntrack.

Now I am sure that iptables is blocking them correctly, because the
connection has been terminated.

Steven, I have the same doubts as you do: why, in this case, the 3-way
handshake did not complete?

I mean "in this case" because I have more than 3,000 customers a day,
and this happens for about 200 times every day.

But Roger may be on to something here: my website has a pop-up. What
if the customer's browser is configured to block pop-ups? I will
configure my browser to block pop-ups and test it, and then I will let
you know.

Regards,

Carlos

Re: It seems I've found why conntrack blocks some packets

[Carlos Pastorino]

On 3/29/06, Carlos Pastorino <carlos.pastorino@gmail.com> wrote:
> On 3/29/06, Steven M Campbell <Netfilter@scampbell.net> wrote:
> > Roger Hamilton wrote:
> > > On 29/03/06, Steven M Campbell <Netfilter@scampbell.net> wrote:
> > >> Carlos Pastorino wrote:
> > >>> Hi everyone,
> > >>>
> > >>> I always wondered why conntrack blocked some packets that were
> > >>> supposed to pass through my ESTABLISHED,RELATED rule. Now it seems
> > >>> that I've found the answer.
> > >>>
> > >>> Bear with me, because there will be questions in the end.
> > >>>
> > >>> So, what happens is: from time to time, I see my firewall blocking a
> > >>> packet like this:
> > >>>
> > >>> Mar 28 14:48:21 SRVA kernel: FORWARD blocked: IN=eth1 OUT=eth0
> > >>> SRC=webserverip DST=customerip LEN=48 TOS=0x00 PREC=0x00 TTL=63 ID=0
> > >>> DF PROTO=TCP SPT=80 DPT=10458 WINDOW=5840 RES=0x00 ACK SYN URGP=0
> > >>>
> > >>> Well, it does call my attention because it's a blocked packet FROM my
> > >>> webserver. In any case, it shouldn't be blocked, because the
> > >>> connection is not even 2 minutes old.
> > >>>
> > >>> But, on the webserver, I was running tcpdump this time, so I could see
> > >>> what really happened:
> > >>>
> > >>> 14:46:47.573738 customerip.10458 > webserverip.80: S [tcp sum ok]
> > >>> 23512000:23512000(0) win 65535 <mss 1460,nop,nop,sackOK> (DF) (ttl
> > >>> 120, id 41065, len 48)
> > >>> 14:46:47.573747 webserverip.80 > customerip.10458: S [tcp sum ok]
> > >>> 4131634297:4131634297(0) ack 23512001 win 5840 <mss
> > >>> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
> > >>> 14:46:51.327629 webserverip.80 > customerip.10458: S [tcp sum ok]
> > >>> 4131634297:4131634297(0) ack 23512001 win 5840 <mss
> > >>> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
> > >>> 14:46:57.327623 webserverip.80 > customerip.10458: S [tcp sum ok]
> > >>> 4131634297:4131634297(0) ack 23512001 win 5840 <mss
> > >>> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
> > >>> 14:47:09.327609 webserverip.80 > customerip.10458: S [tcp sum ok]
> > >>> 4131634297:4131634297(0) ack 23512001 win 5840 <mss
> > >>> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
> > >>> 14:47:33.527575 webserverip.80 > customerip.10458: S [tcp sum ok]
> > >>> 4131634297:4131634297(0) ack 23512001 win 5840 <mss
> > >>> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
> > >>> 14:47:34.216642 customerip.10458 > webserverip.80: R [tcp sum ok]
> > >>> 0:0(0) ack 0 win 0 (ttl 26, id 1, len 40)
> > >>> 14:48:21.727515 webserverip.80 > customerip.10458: S [tcp sum ok]
> > >>> 4131634297:4131634297(0) ack 23512001 win 5840 <mss
> > >>> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
> > >>>
> > >>> As you can see, the customer connected with a SYN packet, and my
> > >>> webserver responded with 6 ACK/SYN packets. BUT, before the 6th
> > >>> ACK/SYN response, the customer sent an ACK/RST packet, resetting the
> > >>> connection, and thus the 6th ACK/SYN packet was blocked by the
> > >>> firewall because the connection was no longer in the conntrack. Yes,
> > >>> clocks in the firewall and webserver are synchronized.
> > >>>
> > >>> Questions are:
> > >>>
> > >>> 1) Does anyone have seen this happening too?
> > >>>
> > >>> 2) How can I silently drop that package, without compromising the on
> > >>> going connections? I would like to get rid of those "false positives".
> > >>>
> > >>> Thanks,
> > >>>
> > >>> Carlos
> > >>>
> > >> The real question here is what bad thing happened in the the data stream that the customer sent the reset packet?  The answer is not to ignore the reset but to find out why it is being sent, the client believes this connection should be aborted for some reason.
> > >>
> > >>
> > >>
> > > Doesn't this happen if the customer closes the browser/stops the web
> > > page loading before the page has completely loaded? If so then one way
> > > to ignore these packets is to silently drop ACK/SYN packets
> > > originating from port 80.
> >
> > The original arguement is that conntrack was blocking packets that it's not supposed to block.  It certainly should block packets that are arriving on a terminated connection.  I don't believe the issue here has anything to do with connection tracking being faulty and workarounds to that effect are only hidng the real issue.  Why did the customer not see the ack packets and finally send a rst?
> >
> >
>
> Well, actually, before I thought that iptables was blocking packets
> incorrectly that I believed were supposed to be allowed because of
> conntrack.
>
> Now I am sure that iptables is blocking them correctly, because the
> connection has been terminated.
>
> Steven, I have the same doubts as you do: why, in this case, the 3-way
> handshake did not complete?
>
> I mean "in this case" because I have more than 3,000 customers a day,
> and this happens for about 200 times every day.
>
> But Roger may be on to something here: my website has a pop-up. What
> if the customer's browser is configured to block pop-ups? I will
> configure my browser to block pop-ups and test it, and then I will let
> you know.
>
> Regards,
>
> Carlos
>

The test showed that blocking pop-ups does not generate the desired
behavior, because the 3-way handshake completes normally, even for the
blocked pop-up.

So I dug a little deeper. I checked other connections from the same
customer, that were also around the same time. They didn't caught my
attention before because there were no packets blocked FROM the
webserver. The problem is, this shows that the problem can be in my
firewall after all. Here are the connections.

First, the blocked packets on the firewall:

Mar 28 14:46:50 SRVA kernel: FORWARD blocked: IN=eth0 OUT=eth1
SRC=customerip DST=webserverip LEN=40 TOS=0x00 PREC=0x00 TTL=26 ID=1
PROTO=TCP SPT=2184 DPT=80 WINDOW=0 RES=0x00 ACK URGP=0
Mar 28 14:46:56 SRVA kernel: FORWARD blocked: IN=eth0 OUT=eth1
SRC=customerip DST=webserverip LEN=40 TOS=0x00 PREC=0x00 TTL=26 ID=1
PROTO=TCP SPT=2184 DPT=80 WINDOW=0 RES=0x00 ACK URGP=0
Mar 28 14:47:08 SRVA kernel: FORWARD blocked: IN=eth0 OUT=eth1
SRC=customerip DST=webserverip LEN=40 TOS=0x00 PREC=0x00 TTL=26 ID=1
PROTO=TCP SPT=2184 DPT=80 WINDOW=0 RES=0x00 ACK URGP=0
Mar 28 14:47:32 SRVA kernel: FORWARD blocked: IN=eth0 OUT=eth1
SRC=customerip DST=webserverip LEN=40 TOS=0x00 PREC=0x00 TTL=26 ID=1
PROTO=TCP SPT=2184 DPT=80 WINDOW=0 RES=0x00 ACK URGP=0
Mar 28 14:48:20 SRVA kernel: FORWARD blocked: IN=eth0 OUT=eth1
SRC=customerip DST=webserverip LEN=40 TOS=0x00 PREC=0x00 TTL=26 ID=1
PROTO=TCP SPT=2184 DPT=80 WINDOW=0 RES=0x00 ACK URGP=0
Mar 28 14:57:34 SRVA kernel: FORWARD blocked: IN=eth0 OUT=eth1
SRC=customerip DST=webserverip LEN=40 TOS=0x00 PREC=0x00 TTL=26 ID=1
PROTO=TCP SPT=2184 DPT=80 WINDOW=0 RES=0x00 ACK RST URGP=0

and now the tcpdump on the webserver:

14:46:46.153552 customerip.2184 > webserverip.80: S [tcp sum ok]
290916053:290916053(0) win 65535 <mss 1460,nop,nop,sackOK> (DF) (ttl
120, id 41051, len 48)
14:46:46.153561 webserverip.80 > customerip.2184: S [tcp sum ok]
4139392508:4139392508(0) ack 290916054 win 5840 <mss
1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
14:46:50.527630 webserverip.80 > customerip.2184: S [tcp sum ok]
4139392508:4139392508(0) ack 290916054 win 5840 <mss
1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
14:46:56.527621 webserverip.80 > customerip.2184: S [tcp sum ok]
4139392508:4139392508(0) ack 290916054 win 5840 <mss
1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
14:47:08.527607 webserverip.80 > customerip.2184: S [tcp sum ok]
4139392508:4139392508(0) ack 290916054 win 5840 <mss
1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
14:47:32.727574 webserverip.80 > customerip.2184: S [tcp sum ok]
4139392508:4139392508(0) ack 290916054 win 5840 <mss
1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
14:48:20.927521 webserverip.80 > customerip.2184: S [tcp sum ok]
4139392508:4139392508(0) ack 290916054 win 5840 <mss
1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)

As you can see, here even the ACKs that were supposed to complete the
3-way handshake are getting blocked.

What could possibly be the reason for this?

Regards,

Carlos

RE: It seems I've found why conntrack blocks some packets

[Gary W. Smith]

Are you by chance doing any rate limiting actions in your firewall or on
your box?  Is there any type of balanced connection in place?  Port
rewrites, etc?


-----Original Message-----
From: netfilter-bounces@lists.netfilter.org
[mailto:netfilter-bounces@lists.netfilter.org] On Behalf Of Carlos
Pastorino
Sent: Wednesday, March 29, 2006 9:06 PM
To: Steven M Campbell
Cc: netfilter@lists.netfilter.org
Subject: Re: It seems I've found why conntrack blocks some packets


The test showed that blocking pop-ups does not generate the desired
behavior, because the 3-way handshake completes normally, even for the
blocked pop-up.

So I dug a little deeper. I checked other connections from the same
customer, that were also around the same time. They didn't caught my
attention before because there were no packets blocked FROM the
webserver. The problem is, this shows that the problem can be in my
firewall after all. Here are the connections.

First, the blocked packets on the firewall:

Mar 28 14:46:50 SRVA kernel: FORWARD blocked: IN=eth0 OUT=eth1
SRC=customerip DST=webserverip LEN=40 TOS=0x00 PREC=0x00 TTL=26 ID=1
PROTO=TCP SPT=2184 DPT=80 WINDOW=0 RES=0x00 ACK URGP=0
Mar 28 14:46:56 SRVA kernel: FORWARD blocked: IN=eth0 OUT=eth1
SRC=customerip DST=webserverip LEN=40 TOS=0x00 PREC=0x00 TTL=26 ID=1
PROTO=TCP SPT=2184 DPT=80 WINDOW=0 RES=0x00 ACK URGP=0
Mar 28 14:47:08 SRVA kernel: FORWARD blocked: IN=eth0 OUT=eth1
SRC=customerip DST=webserverip LEN=40 TOS=0x00 PREC=0x00 TTL=26 ID=1
PROTO=TCP SPT=2184 DPT=80 WINDOW=0 RES=0x00 ACK URGP=0
Mar 28 14:47:32 SRVA kernel: FORWARD blocked: IN=eth0 OUT=eth1
SRC=customerip DST=webserverip LEN=40 TOS=0x00 PREC=0x00 TTL=26 ID=1
PROTO=TCP SPT=2184 DPT=80 WINDOW=0 RES=0x00 ACK URGP=0
Mar 28 14:48:20 SRVA kernel: FORWARD blocked: IN=eth0 OUT=eth1
SRC=customerip DST=webserverip LEN=40 TOS=0x00 PREC=0x00 TTL=26 ID=1
PROTO=TCP SPT=2184 DPT=80 WINDOW=0 RES=0x00 ACK URGP=0
Mar 28 14:57:34 SRVA kernel: FORWARD blocked: IN=eth0 OUT=eth1
SRC=customerip DST=webserverip LEN=40 TOS=0x00 PREC=0x00 TTL=26 ID=1
PROTO=TCP SPT=2184 DPT=80 WINDOW=0 RES=0x00 ACK RST URGP=0

and now the tcpdump on the webserver:

14:46:46.153552 customerip.2184 > webserverip.80: S [tcp sum ok]
290916053:290916053(0) win 65535 <mss 1460,nop,nop,sackOK> (DF) (ttl
120, id 41051, len 48)
14:46:46.153561 webserverip.80 > customerip.2184: S [tcp sum ok]
4139392508:4139392508(0) ack 290916054 win 5840 <mss
1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
14:46:50.527630 webserverip.80 > customerip.2184: S [tcp sum ok]
4139392508:4139392508(0) ack 290916054 win 5840 <mss
1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
14:46:56.527621 webserverip.80 > customerip.2184: S [tcp sum ok]
4139392508:4139392508(0) ack 290916054 win 5840 <mss
1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
14:47:08.527607 webserverip.80 > customerip.2184: S [tcp sum ok]
4139392508:4139392508(0) ack 290916054 win 5840 <mss
1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
14:47:32.727574 webserverip.80 > customerip.2184: S [tcp sum ok]
4139392508:4139392508(0) ack 290916054 win 5840 <mss
1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
14:48:20.927521 webserverip.80 > customerip.2184: S [tcp sum ok]
4139392508:4139392508(0) ack 290916054 win 5840 <mss
1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)

As you can see, here even the ACKs that were supposed to complete the
3-way handshake are getting blocked.

What could possibly be the reason for this?

Regards,

Carlos

Re: It seems I've found why conntrack blocks some packets

[Carlos Pastorino]

On 3/30/06, Gary W. Smith <gary@primeexalia.com> wrote:
> Are you by chance doing any rate limiting actions in your firewall or on
> your box?  Is there any type of balanced connection in place?  Port
> rewrites, etc?
>
>
> -----Original Message-----
> From: netfilter-bounces@lists.netfilter.org
> [mailto:netfilter-bounces@lists.netfilter.org] On Behalf Of Carlos
> Pastorino
> Sent: Wednesday, March 29, 2006 9:06 PM
> To: Steven M Campbell
> Cc: netfilter@lists.netfilter.org
> Subject: Re: It seems I've found why conntrack blocks some packets
>
>
> The test showed that blocking pop-ups does not generate the desired
> behavior, because the 3-way handshake completes normally, even for the
> blocked pop-up.
>
> So I dug a little deeper. I checked other connections from the same
> customer, that were also around the same time. They didn't caught my
> attention before because there were no packets blocked FROM the
> webserver. The problem is, this shows that the problem can be in my
> firewall after all. Here are the connections.
>
> First, the blocked packets on the firewall:
>
> Mar 28 14:46:50 SRVA kernel: FORWARD blocked: IN=eth0 OUT=eth1
> SRC=customerip DST=webserverip LEN=40 TOS=0x00 PREC=0x00 TTL=26 ID=1
> PROTO=TCP SPT=2184 DPT=80 WINDOW=0 RES=0x00 ACK URGP=0
> Mar 28 14:46:56 SRVA kernel: FORWARD blocked: IN=eth0 OUT=eth1
> SRC=customerip DST=webserverip LEN=40 TOS=0x00 PREC=0x00 TTL=26 ID=1
> PROTO=TCP SPT=2184 DPT=80 WINDOW=0 RES=0x00 ACK URGP=0
> Mar 28 14:47:08 SRVA kernel: FORWARD blocked: IN=eth0 OUT=eth1
> SRC=customerip DST=webserverip LEN=40 TOS=0x00 PREC=0x00 TTL=26 ID=1
> PROTO=TCP SPT=2184 DPT=80 WINDOW=0 RES=0x00 ACK URGP=0
> Mar 28 14:47:32 SRVA kernel: FORWARD blocked: IN=eth0 OUT=eth1
> SRC=customerip DST=webserverip LEN=40 TOS=0x00 PREC=0x00 TTL=26 ID=1
> PROTO=TCP SPT=2184 DPT=80 WINDOW=0 RES=0x00 ACK URGP=0
> Mar 28 14:48:20 SRVA kernel: FORWARD blocked: IN=eth0 OUT=eth1
> SRC=customerip DST=webserverip LEN=40 TOS=0x00 PREC=0x00 TTL=26 ID=1
> PROTO=TCP SPT=2184 DPT=80 WINDOW=0 RES=0x00 ACK URGP=0
> Mar 28 14:57:34 SRVA kernel: FORWARD blocked: IN=eth0 OUT=eth1
> SRC=customerip DST=webserverip LEN=40 TOS=0x00 PREC=0x00 TTL=26 ID=1
> PROTO=TCP SPT=2184 DPT=80 WINDOW=0 RES=0x00 ACK RST URGP=0
>
> and now the tcpdump on the webserver:
>
> 14:46:46.153552 customerip.2184 > webserverip.80: S [tcp sum ok]
> 290916053:290916053(0) win 65535 <mss 1460,nop,nop,sackOK> (DF) (ttl
> 120, id 41051, len 48)
> 14:46:46.153561 webserverip.80 > customerip.2184: S [tcp sum ok]
> 4139392508:4139392508(0) ack 290916054 win 5840 <mss
> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
> 14:46:50.527630 webserverip.80 > customerip.2184: S [tcp sum ok]
> 4139392508:4139392508(0) ack 290916054 win 5840 <mss
> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
> 14:46:56.527621 webserverip.80 > customerip.2184: S [tcp sum ok]
> 4139392508:4139392508(0) ack 290916054 win 5840 <mss
> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
> 14:47:08.527607 webserverip.80 > customerip.2184: S [tcp sum ok]
> 4139392508:4139392508(0) ack 290916054 win 5840 <mss
> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
> 14:47:32.727574 webserverip.80 > customerip.2184: S [tcp sum ok]
> 4139392508:4139392508(0) ack 290916054 win 5840 <mss
> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
> 14:48:20.927521 webserverip.80 > customerip.2184: S [tcp sum ok]
> 4139392508:4139392508(0) ack 290916054 win 5840 <mss
> 1460,nop,nop,sackOK> (DF) (ttl 64, id 0, len 48)
>
> As you can see, here even the ACKs that were supposed to complete the
> 3-way handshake are getting blocked.
>
> What could possibly be the reason for this?
>
> Regards,
>
> Carlos
>
>


Hi Gary,

Nope. No rate limiting, no balance and no port rewrites. Here's my
firewall script (showing just the FORWARD table and a few other
things):

=============================================

#!/bin/sh

[VARIABLE DEFINITIONS SUPRESSED]

/sbin/depmod -a

/sbin/modprobe ip_conntrack
/sbin/modprobe ip_conntrack_ftp
/sbin/modprobe ip_nat_ftp
/sbin/modprobe ip_tables
/sbin/modprobe ipt_limit
/sbin/modprobe ipt_LOG
/sbin/modprobe ipt_REJECT
/sbin/modprobe ipt_state
/sbin/modprobe iptable_filter
/sbin/modprobe iptable_mangle
/sbin/modprobe iptable_nat
#/sbin/modprobe ip_conntrack_irc
#/sbin/modprobe ip_nat_irc
#/sbin/modprobe ipt_MASQUERADE
#/sbin/modprobe ipt_owner

echo "1" > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts
echo "0" > /proc/sys/net/ipv4/icmp_echo_ignore_all
echo "1" > /proc/sys/net/ipv4/icmp_ignore_bogus_error_responses
echo "1" > /proc/sys/net/ipv4/ip_forward
echo "1" > /proc/sys/net/ipv4/tcp_syncookies
echo "0" > /proc/sys/net/ipv4/conf/all/accept_redirects
echo "0" > /proc/sys/net/ipv4/conf/all/accept_source_route
echo "0" > /proc/sys/net/ipv4/conf/all/log_martians
echo "1" > /proc/sys/net/ipv4/conf/all/rp_filter

$IPTABLES -F
$IPTABLES -X
$IPTABLES -t nat -F
$IPTABLES -t nat -X
$IPTABLES -t mangle -F
$IPTABLES -t mangle -X

$IPTABLES -P INPUT DROP
$IPTABLES -P OUTPUT DROP
$IPTABLES -P FORWARD DROP

$IPTABLES -N pre_analysis

$IPTABLES -A pre_analysis -p TCP --tcp-flags SYN,ACK SYN,ACK -m state
--state NEW -j LOG --log-prefix "syn-ack-new: "
$IPTABLES -A pre_analysis -p TCP --tcp-flags SYN,ACK SYN,ACK -m state
--state NEW -j REJECT --reject-with tcp-reset

$IPTABLES -A pre_analysis -p TCP ! --syn -m state --state NEW -j LOG
--log-prefix "new-not-syn: "
$IPTABLES -A pre_analysis -p TCP ! --syn -m state --state NEW -j DROP

[INPUT TABLE SUPRESSED]

[OUTPUT TABLE SUPRESSED]

# FORWARD TABLE

$IPTABLES -A FORWARD -p ALL -m state --state ESTABLISHED,RELATED -j ACCEPT

$IPTABLES -A FORWARD -p ICMP -i $DMZ_IFACE -s $DMZ_RANGE -j ACCEPT

$IPTABLES -A FORWARD -p TCP -i $DMZ_IFACE -o $INET_IFACE -s $DMZ_RANGE
-d $0/0 --syn --dport domain -j ACCEPT
$IPTABLES -A FORWARD -p TCP -i $DMZ_IFACE -o $INET_IFACE -s $DMZ_RANGE
-d 0/0 --syn --dport ftp -j ACCEPT
$IPTABLES -A FORWARD -p TCP -i $DMZ_IFACE -o $INET_IFACE -s $DMZ_RANGE
-d 0/0 --syn --dport http -j ACCEPT
$IPTABLES -A FORWARD -p TCP -i $DMZ_IFACE -o $INET_IFACE -s $DMZ_RANGE
-d 0/0 --syn --dport smtp -j ACCEPT
$IPTABLES -A FORWARD -p UDP -i $DMZ_IFACE -o $INET_IFACE -s $DMZ_RANGE
-d 0/0 --dport domain -j ACCEPT
$IPTABLES -A FORWARD -p UDP -i $DMZ_IFACE -o $INET_IFACE -s $DMZ_RANGE
-d 0/0 --dport ntp -j ACCEPT

$IPTABLES -A FORWARD -p ALL -i $INET_IFACE -s 10.0.0.0/8 -j LOG
--log-prefix "spoofed: "
$IPTABLES -A FORWARD -p ALL -i $INET_IFACE -s 10.0.0.0/8 -j DROP
$IPTABLES -A FORWARD -p ALL -i $INET_IFACE -s 172.16.0.0/12 -j LOG
--log-prefix "spoofed: "
$IPTABLES -A FORWARD -p ALL -i $INET_IFACE -s 172.16.0.0/12 -j DROP
$IPTABLES -A FORWARD -p ALL -i $INET_IFACE -s 192.168.0.0/16 -j LOG
--log-prefix "spoofed: "
$IPTABLES -A FORWARD -p ALL -i $INET_IFACE -s 192.168.0.0/16 -j DROP
$IPTABLES -A FORWARD -p ALL -i $INET_IFACE -s 127.0.0.0/8 -j LOG
--log-prefix "spoofed: "
$IPTABLES -A FORWARD -p ALL -i $INET_IFACE -s 127.0.0.0/8 -j DROP
$IPTABLES -A FORWARD -p ALL -i $INET_IFACE -s 169.254.0.0/16 -j LOG
--log-prefix "spoofed: "
$IPTABLES -A FORWARD -p ALL -i $INET_IFACE -s 169.254.0.0/16 -j DROP

$IPTABLES -A FORWARD -p TCP -i $INET_IFACE -s 0/0 -j pre_analysis

$IPTABLES -A FORWARD -p TCP -i $INET_IFACE -o $DMZ_IFACE -s 0/0 -d
$DMZ_RANGE --syn --dport ssh -j ACCEPT
$IPTABLES -A FORWARD -p UDP -i $INET_IFACE -o $DMZ_IFACE -s
$MONITORING -d $DMZ_RANGE --dport snmp -j ACCEPT
$IPTABLES -A FORWARD -p ICMP -i $INET_IFACE -o $DMZ_IFACE -s
$MONITORING -d $DMZ_RANGE -j ACCEPT
$IPTABLES -A FORWARD -p ICMP -i $INET_IFACE -o $DMZ_IFACE -s 0/0 -d
$DMZ_RANGE -j DROP

$IPTABLES -A FORWARD -p TCP -i $INET_IFACE -o $DMZ_IFACE -s
$MONITORING -d $DMZ_WEBSERVER --syn --dport ftp -j ACCEPT
$IPTABLES -A FORWARD -p TCP -i $INET_IFACE -o $DMZ_IFACE -s 0/0 -d
$DMZ_WEBSERVER --syn --dport http -j ACCEPT
$IPTABLES -A FORWARD -p TCP -i $INET_IFACE -o $DMZ_IFACE -s 0/0 -d
$DMZ_WEBSERVER --syn --dport https -j ACCEPT

$IPTABLES -A FORWARD -p TCP -i $INET_IFACE -o $DMZ_IFACE -s 0/0 -d
$DMZ_MAILSERVER --syn --dport auth -j REJECT --reject-with tcp-reset
$IPTABLES -A FORWARD -p TCP -i $INET_IFACE -o $DMZ_IFACE -s 0/0 -d
$DMZ_MAILSERVER --syn --dport pop3 -j ACCEPT
$IPTABLES -A FORWARD -p TCP -i $INET_IFACE -o $DMZ_IFACE -s 0/0 -d
$DMZ_MAILSERVER --syn --dport smtp -j ACCEPT

[SOME FORWARD LINES SUPPRESSED]

$IPTABLES -A FORWARD -j LOG --log-prefix "FORWARD blocked: "

service iptables save

=============================================

Also, a few diagnostics about the firewall server, which is:
Pentium 4
512 MB RAM
Fedora Core 3
iptables-1.2.11-3.1.FC3

top output:
top - 10:57:51 up 6 days,  3:04,  3 users,  load average: 0.00, 0.00, 0.00
Tasks:  65 total,   1 running,  64 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.2% us,  0.0% sy,  0.0% ni, 99.7% id,  0.0% wa,  0.0% hi,  0.2% si
Mem:    513644k total,   446352k used,    67292k free,   206144k buffers
Swap:  1052152k total,      192k used,  1051960k free,   118336k cached

ifconfig output on INET_IFACE:
          RX packets:5399773 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6578080 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:820222396 (782.2 MiB)  TX bytes:2120245609 (1.9 GiB)

ifconfig output on DMZ_IFACE:
          RX packets:7193253 errors:0 dropped:0 overruns:8 frame:0
          TX packets:6176877 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2338744002 (2.1 GiB)  TX bytes:2625446017 (2.4 GiB)

=========================================

So, could there be anything wrong in the script above?

Re: It seems I've found why conntrack blocks some packets

[Steven M Campbell]

We know from the message that we fell off of the end of the FORWARD chain (because the --log-prefix "FORWARD blocked: " is the only one to match the message....

Carlos Pastorino wrote:
	
> 
> $IPTABLES -A FORWARD -p ALL -m state --state ESTABLISHED,RELATED -j ACCEPT
> 
> $IPTABLES -A FORWARD -p ICMP -i $DMZ_IFACE -s $DMZ_RANGE -j ACCEPT
> 
> $IPTABLES -A FORWARD -p TCP -i $DMZ_IFACE -o $INET_IFACE -s $DMZ_RANGE
> -d $0/0 --syn --dport domain -j ACCEPT
> $IPTABLES -A FORWARD -p TCP -i $DMZ_IFACE -o $INET_IFACE -s $DMZ_RANGE
> -d 0/0 --syn --dport ftp -j ACCEPT
> $IPTABLES -A FORWARD -p TCP -i $DMZ_IFACE -o $INET_IFACE -s $DMZ_RANGE
> -d 0/0 --syn --dport http -j ACCEPT
> $IPTABLES -A FORWARD -p TCP -i $DMZ_IFACE -o $INET_IFACE -s $DMZ_RANGE
> -d 0/0 --syn --dport smtp -j ACCEPT
> $IPTABLES -A FORWARD -p UDP -i $DMZ_IFACE -o $INET_IFACE -s $DMZ_RANGE
> -d 0/0 --dport domain -j ACCEPT
> $IPTABLES -A FORWARD -p UDP -i $DMZ_IFACE -o $INET_IFACE -s $DMZ_RANGE
> -d 0/0 --dport ntp -j ACCEPT
> 

deleted a bunch of drop and logs, these aren't the problem

> 
> $IPTABLES -A FORWARD -p TCP -i $INET_IFACE -s 0/0 -j pre_analysis
> 

I'm removing lines that deal with ports other than http

> $IPTABLES -A FORWARD -p ICMP -i $INET_IFACE -o $DMZ_IFACE -s
> $MONITORING -d $DMZ_RANGE -j ACCEPT
> $IPTABLES -A FORWARD -p TCP -i $INET_IFACE -o $DMZ_IFACE -s 0/0 -d
> $DMZ_WEBSERVER --syn --dport http -j ACCEPT
> $IPTABLES -A FORWARD -p TCP -i $INET_IFACE -o $DMZ_IFACE -s 0/0 -d
> $DMZ_WEBSERVER --syn --dport https -j ACCEPT
> 
> 
> [SOME FORWARD LINES SUPPRESSED]
> 
> $IPTABLES -A FORWARD -j LOG --log-prefix "FORWARD blocked: "
> 
> 

Unfortunately, you've needed to obscure the actual ip address (I understand) but I can't match the 'customerip' and 'webserverip' to the ${variables} above because I don't know the actual values for any of them.

Try to walk through the rules in your forward chain using the ip addresses you've captured and identify the rule you believe should allow these ack packets to go out.

Re: It seems I've found why conntrack blocks some packets

[Steven M Campbell]

Steven M Campbell wrote:
> We know from the message that we fell off of the end of the FORWARD 
> chain (because the --log-prefix "FORWARD blocked: " is the only one to 
> match the message....
> 
>

One other thought to this, if I were to presume the ${variables} and ...ip's then I would presume that the RELATED rules should allow these ack's to go through.  The only reason I know of for them not do (again, assuming all the addressing is really ok) would be that the conntrack table has filled up.

To see the maximum connnections that can be tracked:

# cat /proc/sys/net/ipv4/ip_conntrack_max
32760

To see how many you are using at a given moment

# wc  -l /proc/net/ip_conntrack
16 /proc/net/ip_conntrack


This from my house and there really isn't all that much going on, I would expect far larger counts, you may need to up ip_conntrack_max.  This really out in the SWAG arena because I can't see the details of your installation.

Re: It seems I've found why conntrack blocks some packets

[Carlos Pastorino]

Now, commenting on this message: I actually didn't know that the
conntrack table had a limit. Learning something every day. I will
check its value on Monday, during peak time.

Another thought: if the ACKs that are being blocked are for some
reason malformed, wouldn't they be blocked as well by the last rule?

> One other thought to this, if I were to presume the ${variables} and ...ip's then I would presume that the RELATED rules should allow these ack's to go through.  The only reason I know of for them not do (again, assuming all the addressing is really ok) would be that the conntrack table has filled up.
>
> To see the maximum connnections that can be tracked:
>
> # cat /proc/sys/net/ipv4/ip_conntrack_max
> 32760
>
> To see how many you are using at a given moment
>
> # wc  -l /proc/net/ip_conntrack
> 16 /proc/net/ip_conntrack
>
>
> This from my house and there really isn't all that much going on, I would expect far larger counts, you may need to up ip_conntrack_max.  This really out in the SWAG arena because I can't see the details of your installation.
>
>

Re: It seems I've found why conntrack blocks some packets

[Steven M Campbell]

Carlos Pastorino wrote:
> Now, commenting on this message: I actually didn't know that the
> conntrack table had a limit. Learning something every day. I will
> check its value on Monday, during peak time.

A related thought to this, I wonder how many connections are not being closed nicely and then have to hang around in the conntrack table. If you find that you are approaching the limits then you might want to look into the various connection tracking timings.


> Another thought: if the ACKs that are being blocked are for some
> reason malformed, wouldn't they be blocked as well by the last rule?

The last rule is a log-only rule:

	$IPTABLES -A FORWARD -j LOG --log-prefix "FORWARD blocked: "

It's the one generating the log messages we see, therefore we are actually falling off the table and taking the default policy which is 'DROP'  ($IPTABLES -P FORWARD DROP)

Also, there really isn't that much to the syn-ack packets, kinda hard to malform them too much. 


> 
>> One other thought to this, if I were to presume the ${variables} and ...ip's then I would presume that the RELATED rules should allow these ack's to go through.  The only reason I know of for them not do (again, assuming all the addressing is really ok) would be that the conntrack table has filled up.
>>
>> To see the maximum connnections that can be tracked:
>>
>> # cat /proc/sys/net/ipv4/ip_conntrack_max
>> 32760
>>
>> To see how many you are using at a given moment
>>
>> # wc  -l /proc/net/ip_conntrack
>> 16 /proc/net/ip_conntrack
>>
>>
>> This from my house and there really isn't all that much going on, I would expect far larger counts, you may need to up ip_conntrack_max.  This really out in the SWAG arena because I can't see the details of your installation.
>>
>>
> 

Re: It seems I've found why conntrack blocks some packets

[Carlos Pastorino]

Hi Steven,

> To see how many you are using at a given moment
>
> # wc  -l /proc/net/ip_conntrack
> 16 /proc/net/ip_conntrack

I checked the conntrack table. I setup a cron job to look at it every
minute, for 24 hours. You'll be surprised: the top number, around peak
time, is:

255 /proc/net/ip_conntrack

So, it can't be the limit on the conntrack table.

I've found another clue though. When I first configured this firewall,
I enabled rp_filter, with the command below:

echo "1" > /proc/sys/net/ipv4/conf/all/rp_filter

And I've found this text on the Internet about it:

"If instead you decide to enable forwarding, you will also be able to
modify the rp_filter setting; something which is often misunderstood
by network administrators. The rp_filter can reject incoming packets
if their source address doesn't match the network interface that
they're arriving on, which helps to prevent IP spoofing. Turning this
on, however, has its consequences: If your host has several IP
addresses on different interfaces, or if your single interface has
multiple IP addresses on it, you'll find that your kernel may end up
rejecting valid traffic. It's also important to note that even if you
do not enable the rp_filter, protection against broadcast spoofing is
always on. Also, the protection it provides is only against spoofed
internal addresses; external addresses can still be spoofed.. By
default, it is disabled."

And that's what's been happening: The firewall has been rejecting a
few valid packets.

I'll disable it and see what happens, and then I'll let you know.

By the way, do you keep rp_filter enabled or disabled?

Re: It seems I've found why conntrack blocks some packets

[Steven M Campbell]

Carlos Pastorino wrote:

> 
> By the way, do you keep rp_filter enabled or disabled?
> 

enabled but I'm originally a network geek by trade so my network is very clean with regards to where subnets are so reverse path filters work for me.

Re: It seems I've found why conntrack blocks some packets

[Carlos Pastorino]

Well, disabling it didn't work. I'll enable it back.

I'm sorta giving up on this. Thanks a lot for your attention and help.

Regards.

On 4/5/06, Steven M Campbell <Netfilter@scampbell.net> wrote:
> Carlos Pastorino wrote:
>
> >
> > By the way, do you keep rp_filter enabled or disabled?
> >
>
> enabled but I'm originally a network geek by trade so my network is very clean with regards to where subnets are so reverse path filters work for me.
>

Re: It seems I've found why conntrack blocks some packets

[Carlos Pastorino]

Hi Steven,

> Unfortunately, you've needed to obscure the actual ip address (I understand) but I can't match the 'customerip' and 'webserverip' to the ${variables} above because I don't know the actual values for any of them.

Well, the customerip is from some unknown Internet user. So it's an
external IP, and its connection comes in via $INET_IFACE.

The webserverip is in my DMZ, so it matches $DMZ_RANGE or $DMZ_WEBSERVER.

> Try to walk through the rules in your forward chain using the ip addresses you've captured and identify the rule you believe should allow these ack packets to go out.

Well, it's actually the ACK packets that should come in, and the rule
that must match them is the ESTABLISHED,RELATED rule. And it actually
does match for more than 3,000 connections a day. But, for 200 or so
of them, this odd behavior occurs.

Your next e-mail may have shed some light. I'll comment on it.

Regards.