Some packets flagged INVALID

Some packets flagged INVALID

[Bob Sauvage]

Hi *, 

I'm running a high volume web application that uses Apache 2.2.15 mod_proxy to reverse proxy content from apache to JBoss 6. 

I found 503 errors which happen sporadically throughout the day on random requests (perhaps 1/1000 of daily requests).

After investigations, I noticed that every error coincides with an invalid tcp packet: 

kernel: invalid:IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=33082 DF PROTO=TCP SPT=48340 DPT=8080 WINDOW=32792 RES=0x00 SYN URGP=0

After some investigations, this SYN packet is not acknowledged by JBoss in order to perform the TCP 3-Way Handshake. Mhmm, strange, I decide to investigate in firewall rules, build by another sysadmin:

In the INPUT chain, I found a rule that logs and REJECTS all INVALID packets: 

iptables -A INPUT -m state --state INVALID -j LOG --log-prefix "invalid:" 
iptables -A INPUT -m state --state INVALID -j REJECT

Then logs and REJECTS not SYN but new: 

iptables -A INPUT -p tcp ! --syn -m state --state NEW -j LOG --log-prefix "new-not-syn:"
iptables -A INPUT -p tcp ! --syn -m state --state NEW -j REJECT --reject-with tcp-reset

I decided to add a rule in order to ACCEPT all packets from 127.0.0.1 to or from port 8080.

Since this update, I don't see this kind of errors anymore.

Why does iptables tag this packet as invalid ?

Thanks,

Bob

Re: Some packets flagged INVALID

[Neal Murphy]

On Tuesday, February 18, 2014 04:52:18 PM Bob Sauvage wrote:
> Hi *,
> 
> I'm running a high volume web application that uses Apache 2.2.15 mod_proxy
> to reverse proxy content from apache to JBoss 6.
> 
> I found 503 errors which happen sporadically throughout the day on random
> requests (perhaps 1/1000 of daily requests).
> 
> After investigations, I noticed that every error coincides with an invalid
> tcp packet:
> 
> kernel: invalid:IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00
> SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=33082 DF
> PROTO=TCP SPT=48340 DPT=8080 WINDOW=32792 RES=0x00 SYN URGP=0
> 
> After some investigations, this SYN packet is not acknowledged by JBoss in
> order to perform the TCP 3-Way Handshake. Mhmm, strange, I decide to
> investigate in firewall rules, build by another sysadmin:
> 
> In the INPUT chain, I found a rule that logs and REJECTS all INVALID
> packets:
> 
> iptables -A INPUT -m state --state INVALID -j LOG --log-prefix "invalid:"
> iptables -A INPUT -m state --state INVALID -j REJECT
> 
> Then logs and REJECTS not SYN but new:
> 
> iptables -A INPUT -p tcp ! --syn -m state --state NEW -j LOG --log-prefix
> "new-not-syn:"
> iptables -A INPUT -p tcp ! --syn -m state --state NEW -j
> REJECT --reject-with tcp-reset
> 
> I decided to add a rule in order to ACCEPT all packets from 127.0.0.1 to or
> from port 8080.
> 
> Since this update, I don't see this kind of errors anymore.
> 
> Why does iptables tag this packet as invalid ?

I believe INVALID applies to all packets that netfilter has no clue why it has 
receieved them. Often, it's a RST or FIN for a non-existent conn or one 
recently torn-down. DDoS packets are often INVALID.

I think those 'syn' rules above are 'defective' (or extraneous). For TCP, only 
a SYN packet should be tagged NEW because there is no other way to initiate a 
TCP conn.

In this particular case, it may be that the SYN arrived during the grace 
period after a previous matching conn was torn down. (Related to 
SO_REUSEADDR?)

Re: Some packets flagged INVALID

[Mart Frauenlob]

On 18.02.2014 23:26, Neal Murphy wrote:
> On Tuesday, February 18, 2014 04:52:18 PM Bob Sauvage wrote:
>> Hi *,
>>
>> I'm running a high volume web application that uses Apache 2.2.15 mod_proxy
>> to reverse proxy content from apache to JBoss 6.
>>
>> I found 503 errors which happen sporadically throughout the day on random
>> requests (perhaps 1/1000 of daily requests).
>>
>> After investigations, I noticed that every error coincides with an invalid
>> tcp packet:
>>
>> kernel: invalid:IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00
>> SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=33082 DF
>> PROTO=TCP SPT=48340 DPT=8080 WINDOW=32792 RES=0x00 SYN URGP=0
>>
>> After some investigations, this SYN packet is not acknowledged by JBoss in
>> order to perform the TCP 3-Way Handshake. Mhmm, strange, I decide to
>> investigate in firewall rules, build by another sysadmin:
>>
>> In the INPUT chain, I found a rule that logs and REJECTS all INVALID
>> packets:
>>
>> iptables -A INPUT -m state --state INVALID -j LOG --log-prefix "invalid:"
>> iptables -A INPUT -m state --state INVALID -j REJECT
>>
>> Then logs and REJECTS not SYN but new:
>>
>> iptables -A INPUT -p tcp ! --syn -m state --state NEW -j LOG --log-prefix
>> "new-not-syn:"
>> iptables -A INPUT -p tcp ! --syn -m state --state NEW -j
>> REJECT --reject-with tcp-reset
>>
>> I decided to add a rule in order to ACCEPT all packets from 127.0.0.1 to or
>> from port 8080.
>>
>> Since this update, I don't see this kind of errors anymore.
>>
>> Why does iptables tag this packet as invalid ?
>
> I believe INVALID applies to all packets that netfilter has no clue why it has
> receieved them. Often, it's a RST or FIN for a non-existent conn or one
> recently torn-down. DDoS packets are often INVALID.
>
> I think those 'syn' rules above are 'defective' (or extraneous). For TCP, only
> a SYN packet should be tagged NEW because there is no other way to initiate a
> TCP conn.

right, but in a fail-over scenario with conntrackd you need to set a 
sysctl to allow:

nf_conntrack_tcp_loose - BOOLEAN
         0 - disabled
         not 0 - enabled (default)

         If it is set to zero, we disable picking up already established
         connections.


or you might tune the behaviour with:

nf_conntrack_tcp_be_liberal - BOOLEAN
         0 - disabled (default)
         not 0 - enabled

         Be conservative in what you do, be liberal in what you accept 
from others.
         If it's non-zero, we mark only out of window RST segments as 
INVALID.

see:
Documentation/networking/nf_conntrack-sysctl.txt

in your kernel source dir.

>
> In this particular case, it may be that the SYN arrived during the grace
> period after a previous matching conn was torn down. (Related to
> SO_REUSEADDR?)