inadequate NEW, INVALID state definitions

inadequate NEW, INVALID state definitions

[Jeremiah Crockett]

I've searched the list archives and googled available iptables documentation,
but haven't yet found satisfactory explanations for iptables states, in
particular for "NEW" and "INVALID".  Does there exist a thorough reference,
short of trying to read the code, about states and transitions? 

An iptables tutorial,
<http://www.frozentux.net/iptables-tutorial/iptablestutorial.html#STATEMACHINE>
says: "The INVALID state means that the packet can't be identified or that it
does not have any state" but "does not have any state" isn't very helpful, since
it applies to initial NEW packets as well.  And "can't be identified", which is
almost identical to the manpage definition, is on a par with "UFO" for
non-descriptiveness.  Because of this, I used to think INVALID effectively meant
NOT(NEW|ESTABLISHED|RELATED), a state for packets that didn't match other
states, but I gather it's not so passive a determination as this.  Posts on this
list indicate INVALID can replace flag combination checks for nonsense
combinations, and the existence of booleans like
netfilter.ip_conntrack_tcp_be_liberal, that control whether out-of-window
packets are INVALID or not, together these hint at considerable complexity in
the actual logic.  If this is so, these packets aren't really "unidentified",
they are in fact identified and marked INVALID for a variety of reasons.  WHAT
are those reasons, is it not possible to list or describe them somewhere?
 
The same tutorial says "The NEW state tells us that the packet is the first
packet that we see."  This of course is also totally inadequate to distinguish
from INVALID.  And it goes on to say the syn flag doesn't have to be set, which
"may lead to certain problems in some instances, but it may also be extremely
helpful when we need to pick up lost connections from other firewalls, or when a
connection has already timed out, but in reality is not closed."  Well,
"pick[ing] up lost connections from other firewalls" sounds like such an edge
case.  And what are the "certain problems" that might arise, what exactly are we
letting through by allowing all NEW traffic?   Conversely, what do we break by
allowing only NEW+syn?  

It seems like connection tracking states and transitions are such a fundamental
concept to netfilter, it's hard to believe there is no clear documentation on
these topics.  Thanks for any references or clarification.

Jeremiah

Re: inadequate NEW, INVALID state definitions

[Curby]

On Tue, Jun 1, 2010 at 5:31 PM, Jeremiah Crockett <jrmhcrcktt@gmail.com> wrote:
> I've searched the list archives and googled available iptables documentation,
> but haven't yet found satisfactory explanations for iptables states, in

After talking to a colleague about a recent ruleset of mine, we came
to the same conclusion, i.e. the existing docs aren't very good at
explaining specifically what causes a packet to be marked one way or
another.  As one example of many, this doesn't seem to be complete, or
even necessarily correct:

http://www.faqs.org/docs/iptables/userlandstates.html

The problem is that a packet that is NEW could be marked INVALID due
to a problem within the packet, e.g. an impossible or
standards-noncompliant combination of flags, so the determination of
NEW is only made after INVALID, RELATED, and ESTABLISHED are shown to
not apply.  This is counter to a lot of documentation which starts by
mentioning NEW first, from which readers might infer that NEW packets
will never match INVALID because they were already marked NEW.

In short, I agree with Jeremiah in that a more complete description of
these states would be very useful.  "Go read the code" does not
suffice, but it's currently the only option.

--Mike

Re: inadequate NEW, INVALID state definitions

[Jan Engelhardt]

On Wednesday 2010-06-02 18:58, Curby wrote:
>On Tue, Jun 1, 2010 at 5:31 PM, Jeremiah Crockett <jrmhcrcktt@gmail.com> wrote:
>> I've searched the list archives and googled available iptables documentation,
>> but haven't yet found satisfactory explanations for iptables states, in
>
>After talking to a colleague about a recent ruleset of mine, we came
>to the same conclusion, i.e. the existing docs aren't very good at
>explaining specifically what causes a packet to be marked one way or
>another.

Yeah unfortunately most documents are not written by developers and
thus accumulate a lot of half-knowledge or outright guessing.

>As one example of many, this doesn't seem to be complete, or
>even necessarily correct:
>
>http://www.faqs.org/docs/iptables/userlandstates.html

The document is:

 1. indeed outdated
 2. indeed incomplete (owing to point 1.)
 3. does have bad description though
 4. claims that are, technically, incorrect
   * like "to be considered as RELATED, we must first have a connection
     that is considered ESTABLISHED. "

>The problem is that a packet that is NEW could be marked INVALID due

Interesting you are copying the document's problems in wording :)

	(=>) Packets cannot be NEW. Connections are.

>to a problem within the packet, e.g. an impossible or
>standards-noncompliant combination of flags, so the determination of
>NEW is only made after INVALID, RELATED, and ESTABLISHED are shown to
>not apply.  This is counter to a lot of documentation which starts by
>mentioning NEW first, from which readers might infer that NEW packets
>will never match INVALID because they were already marked NEW.

The documentation is basically correct despite imprecise. The five
--ctstates {NEW, ESTABLISHED, RELATED, INVALID, UNTRACKED} _are
mutually exclusive_ to another.


An skb will start out zeroed, i.e.

	skb->nfct = NULL
	skb->nfctinfo = 0; /* IP_CT_NEW == 0 */

Since the logic for xt_conntrack ctstate testing is:

	if (skb->nfct == NULL)
		"INVALID";
	else if (skb->nfct == &nf_conntrack_untracked)
		"UNTRACKED";
	else if ((skb->nfctinfo % IP_CT_IS_REPLY) == IP_CT_NEW)
		"NEW";
	else if ((skb->nfctinfo % IP_CT_IS_REPLY) == IP_CT_ESTABLISHED)
		"ESTABLISHED";
	else if ((skb->nfctinfo % IP_CT_IS_REPLY) == IP_CT_RELATED)
		"RELATED";

Packets basically start without any associated connection, that is,
will be matched by -m conntrack --ctstate INVALID.

When it's nf_conntrack's turn, it may assign something to skb->nfct,
which makes --ctstate NEW match. Or it may not (faulty packet, no
association for ICMP, etc.), skb->nfct* is left as is, in other
words, it will continue to match --ctstate INVALID.

Re: inadequate NEW, INVALID state definitions

[Curby]

On Wed, Jun 2, 2010 at 2:35 PM, Jeremiah Crockett <jrmhcrcktt@gmail.com> wrote:
> On Wed, Jun 2, 2010 at 1:27 PM, Jan Engelhardt <jengelh@medozas.de> wrote:
>> . The five
>> --ctstates {NEW, ESTABLISHED, RELATED, INVALID, UNTRACKED} _are
>> mutually exclusive_ to another.
>
> Are they all inclusive, i.e., is it possible for a packet not to match any
> of these states?  Or do packets "fall-through" into UNTRACKED if nothing
> else matches?

I believe that the default is INVALID, not UNTRACKED.  From my limited
understanding, UNTRACKED is used for traffic that was sent to the
NOTRACK target.

On Wed, Jun 2, 2010 at 1:27 PM, Jan Engelhardt <jengelh@medozas.de> wrote:
> Packets basically start without any associated connection, that is,
> will be matched by -m conntrack --ctstate INVALID.

Thanks Jan.  So two important points that I gleaned from your post are
(1) the states are mutually exclusive and (2) all packets start with a
default of INVALID.

This seems to get a bit murkier as some conditions that match INVALID,
including invalid combinations of TCP flags, seem to be an "active
match" instead of a "passive default."  In other words, if the state
starts as INVALID, and conntrack determines that it should be state
NEW, something somehow must be applied afterwards to force it back to
INVALID due to it being a malformed packet.  How does that "something"
fit in with this chain of events, and what does it check for?
Alternately, do the checks for placing a packet into the other states
check for malformed packets, in which case no second check for
malformed packets needs to be done because anything that moves from
INVALID is already known to be well-formed?

I think part of the confusion stems from this: Based on my reading of
the admittedly suboptimal documentation, there are two ways that
packets can be marked INVALID:

The packet is not part of any connection marked NEW, RELATED,
ESTABLISHED, or UNTRACKED.
The packet is malformed or otherwise problematic.

The first is determined in the context of the conntrack system, and
takes into consideration conntrack's knowledge of the state of the
network and the connections therein.  The second treats packets in a
vacuum and evaluates parts of the packet against other parts to
determine its consistency.

It's likely that my confusion stems from a still-incomplete knowledge
of how packets traverse the different parts of netfilter and xtables.

Thanks again, everyone!

--Mike

Re: inadequate NEW, INVALID state definitions

[Jan Engelhardt]

On Wednesday 2010-06-02 22:35, Jeremiah Crockett wrote:
>On Wed, Jun 2, 2010 at 1:27 PM, Jan Engelhardt <jengelh@medozas.de> wrote:
>
>>The five --ctstates {NEW, ESTABLISHED, RELATED, INVALID, UNTRACKED} 
>>_are mutually exclusive_ to another.
>
>[...] is it possible for a packet not to 
>match any of these states?

No.

>      An skb will start out zeroed, i.e.
>      ...
>      Since the logic for xt_conntrack ctstate testing is:
>      ...
>      Packets basically start without any associated connection, that
>      is,
>      will be matched by -m conntrack --ctstate INVALID.
>
>      When it's nf_conntrack's turn, it may assign something to
>      skb->nfct,
>      which makes --ctstate NEW match. Or it may not (faulty packet,
>      no
>      association for ICMP, etc.), skb->nfct* is left as is, in other
>      words, it will continue to match --ctstate INVALID.
>
>
>This is perfect, as far as it goes.  It seems to me there is a great need
>for someone to continue this,

Well ask more. With a bit of luck enough text will accumulate and 
I'll get fired up to put it into book form, just as I did with 
the "Writing Netfilter Modules" PDF.
But it needs assistance from novices , because experienced people see 
less of a problem that they would consider worth documenting.

>an explanation of the logic, perhaps
>illustrated by code snippets, but not requiring every netfilter user to be
>able to derive it for themself. 

Re: inadequate NEW, INVALID state definitions

[Jan Engelhardt]

On Wednesday 2010-06-02 22:48, Curby wrote:
>
>This seems to get a bit murkier as some conditions that match INVALID,
>including invalid combinations of TCP flags, seem to be an "active
>match" instead of a "passive default."  In other words, if the state
>starts as INVALID, and conntrack determines that it should be state
>NEW, something somehow must be applied afterwards to force it back to
>INVALID due to it being a malformed packet.  How does that "something"
>fit in with this chain of events, and what does it check for?

The 'error' check function (bad length, flags, zz.)
runs before creating a ct entry.

>The first is determined in the context of the conntrack system, and
>takes into consideration conntrack's knowledge of the state of the
>network and the connections therein.  The second treats packets in a
>vacuum and evaluates parts of the packet against other parts to
>determine its consistency.

conntrack has no knowledge of the network layout, topology,
speed,  or any other condition, other than perhaps the recently-added
user-settable zone identifier.

>It's likely that my confusion stems from a still-incomplete knowledge
>of how packets traverse the different parts of netfilter and xtables.

Start by reading http://en.wikipedia.org/wiki/Netfilter and /iptables,
perhaps.
Their current version is clean and approved by me :p

Re: inadequate NEW, INVALID state definitions

[Jozsef Kadlecsik]

On Wed, 2 Jun 2010, Curby wrote:

> This seems to get a bit murkier as some conditions that match INVALID,
> including invalid combinations of TCP flags, seem to be an "active
> match" instead of a "passive default."  In other words, if the state
> starts as INVALID, and conntrack determines that it should be state
> NEW, something somehow must be applied afterwards to force it back to
> INVALID due to it being a malformed packet. 

It depends on how one looks at conntrack and it's relation with the other 
parts of netfilter. I'd not say that "every packet starts with the state 
INVALID". Of course the state of the packet *before* conntrack invoked is 
INVALID, but what else could it be, if conntrack hadn't got a chance yet 
to associate a state to the packet?

The current documentations try to describe the states so that it does not 
need a deep TCP/IP knowledge from the readers. A rough description on how 
a packet traverses conntrack and what state are associated to it could be 
the following:

- loopback or untracked packets are ignored
- if the IP(v4/6) header cannot be extracted then the packet is dropped
- if the packet, according to the Layer 4 protocol, is broken (like
  for TCP: the packet does not carry a complete TCP header or it's 
  malformed or the TCP checksum/TCP flag combination is invalid; for UDP:
  the header is too small, truncated or malformed, etc.) then the INVALID 
  state is associated to the packet
- if the IP addresses or the protocol parameters (ports) for a unique
  tuple (src ip, dst ip, proto, src "port", dst "port") cannot be 
  extracted from the packet, it's state is set to INVALID
- then, using the tuple, we try to find an already existing connection
  to which the packet belongs. If we do find such a connection, then
  a lot of checking happens according to the protocol. Just for TCP, those 
  can result in dropping simply the packet (like when we cannot handle 
  reopened connections or when conntrack lost sync and that way we
  can force the client to resend an initial SYN); or set the state to 
  INVALID (when it's s SYN/ACK packet and we haven't seen a SYN 
  previously, or it's a FIN "answering" a SYN packet, or invalid RST
  segment, or out of window packet, etc). If the packet is OK and
  associated to a connection, then it's state will be ESTABLISHED.
- if we could not find an already existing connection for the packet
  then it's state will be NEW - or RELATED, if we expected the packet.
  Or INVALID, if the given packet may not initiate a new connection, 
  according to the protocol (and tcp_loose is disabled for TCP).

I left out smaller parts and referred TCP mostly. However there are ICMP 
subleties, and other handled protocols as well. So the full picture would 
need a complete article.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: inadequate NEW, INVALID state definitions

[Jan Engelhardt]

On Thursday 2010-06-03 00:13, Jozsef Kadlecsik wrote:
>On Wed, 2 Jun 2010, Curby wrote:
>
>> This seems to get a bit murkier as some conditions that match INVALID,
>> including invalid combinations of TCP flags, seem to be an "active
>> match" instead of a "passive default."  In other words, if the state
>> starts as INVALID, and conntrack determines that it should be state
>> NEW, something somehow must be applied afterwards to force it back to
>> INVALID due to it being a malformed packet. 
>
>It depends on how one looks at conntrack and it's relation with the other 
>parts of netfilter. I'd not say that "every packet starts with the state 
>INVALID".

Well yes, I had already considered skb_copy in my back-head, but I
put it off as a developer thing, because if you have not looked into
code, you don't really know whether ipt_REJECT copies the packet and
resets/truncates it or creates a new one from scratch, then copies.
(Answer: in the timeline of Linux, both ways have been tried
already.)

>Of course the state of the packet *before* conntrack invoked is 
>INVALID, but what else could it be, if conntrack hadn't got a chance yet 
>to associate a state to the packet?

In case of skb_copy, that would be oldskb->nfct* :-)

>The current documentations try to describe the states so that it does not 
>need a deep TCP/IP knowledge from the readers. A rough description on how 
>a packet traverses conntrack and what state are associated to it could be 
>the following:
>
>- loopback or untracked packets are ignored

And by loopback, Jozsef does not mean "lo" device transmissions,
but skb->nfct != NULL (happens when passing twice through
nf_conntrack, such as with xfrm).

Re: inadequate NEW, INVALID state definitions

[Curby]

On Wed, Jun 2, 2010 at 4:13 PM, Jozsef Kadlecsik
<kadlec@blackhole.kfki.hu> wrote:
> need a deep TCP/IP knowledge from the readers. A rough description on how
> a packet traverses conntrack and what state are associated to it could be
> the following:

Jozsef's reply goes far in giving us what we're looking for.  To
answer my question as to when INVALID can get applied or re-applied,
the answer is at almost any time.  In other words, while there is a
default of INVALID, conntrack will check for invalid conditions as it
applies its normal logic in seeing if other states should be used
instead.

For a ruleset writer, this applies to the question of how to order the
following two rules:

-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

With my new understanding, it seems that putting INVALID first is not
useful: there is no need to "protect" EST/REL connections from INVALID
packets by placing the INVALID rule first because anything INVALID
would not match EST/REL anyway; they are mutually exclusive as Jan
mentioned.  With the goal of creating a short path for a large
majority of packets, EST/REL should be listed as close to the top as
possible and therefore before the INVALID rule.  In most cases, there
is negligible gain in removing a single rule from the path, but
there's nothing gained in terms of security from putting INVALID
first.

Thanks again for the insights!

--Mike

Re: inadequate NEW, INVALID state definitions

[Jan Engelhardt]

On Thursday 2010-06-03 01:03, Curby wrote:
>On Wed, Jun 2, 2010 at 4:13 PM, Jozsef Kadlecsik
><kadlec@blackhole.kfki.hu> wrote:
>> need a deep TCP/IP knowledge from the readers. A rough description on how
>> a packet traverses conntrack and what state are associated to it could be
>> the following:
>
>Jozsef's reply goes far in giving us what we're looking for.  To
>answer my question as to when INVALID can get applied or re-applied,
>the answer is at almost any time.

Not at any time. In nf_conntrack_core.c, you have

	* l4proto->error(...)
	* resolve_normal_ct
	* l4proto->packet(...)

resolve_normal_ct would make nfct!=NULL, and a failure in l4packet
would make it NULL again. All of this happens within this one hook
(nf_conntrack_in), so Xtables does not see this intermediate result.
And after nf_conntrack_in is finished, skb->nfct should stay as-is
over the rest of the course through the tables traversal.

>For a ruleset writer, this applies to the question of how to order the
>following two rules:
>
>-A INPUT -m conntrack --ctstate INVALID -j DROP
>-A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
>
>With my new understanding, it seems that putting INVALID first is not
>useful: there is no need to "protect" EST/REL connections from INVALID
>packets by placing the INVALID rule first because anything INVALID
>would not match EST/REL anyway; they are mutually exclusive as Jan
>mentioned.  With the goal of creating a short path for a large
>majority of packets, EST/REL should be listed as close to the top as
>possible and therefore before the INVALID rule.

..As explained in The Perfect Ruleset ;-)