The latest nf_conntrack

The latest nf_conntrack

[Yasuyuki Kozakai]

Hi, all,

I finished synchronizing nf_conntrack with ip_conntrack in linux 2.6.10-rc1.
I need to cleanup and do more tests, anyway, I committed it to pom-ng.

>From the discussion at Netfilter developer workshop, I deleted the very complex
code reallocating conntrack from the basic slab cache when destroying the slab
cache for conntrack which has "help" part. Instead, all slab caches are
destroyed when unloading nf_conntrack.ko .

And I re-implemented handling fragmented IPv6 packets. This code is what
I imagined after workshop, but I recently found that my idea is different
with the results of discussion at workshop. Then I may change some parts.

The summary of handling IPv6 packets is as follows.

- at the hook NF_IP6_PRE_ROUTING, priority NF_IP6_CONNTRACK_DEFRAG,
	- The fragments are cloned and queued. clone skb has the pointer
	  to original skb.
	- When all fragments are gathered, cloned skb are reassembled.
	- All original skbs keep the pointer to reassembled skb in new member
	  of sk_buff "nfct_reasm".
	- The following hooks of NF_IP6_PRE_ROUTING are called. Please note
	  that reassembled skb isn't passed to hooks but original fragments are
	  passed.
	- NF_STOLEN is returned.

- at the hook NF_IP6_PRE_ROUTING, priority NF_IP6_CONNTRACK,
	- If skb has reassembled skb, that is, packet is fragmented,
		- If reassembled skb is already parsed, fragmented skb
		  reserves the pointer to the conntrack bound with reassembled
		  skb. Otherwise, reassembled skb is parsed.
	  Otherwise, skb is ordinarily parsed.
	- The original fragments are passed to next hooks and IPv6 core
	  processing.

By this processing,
	1. We can prevent disturbing IPv6 stack. If we introduce
	   ip_conntrack style, and when large fragmented packets came in,
	   "Too big" isn't sent. Instead, resized fragmented packets are
	   forwarded. That isn't natural.

	2. Packet filter can re-use reassembled packet. For example,
	   udp_match() can evaluate ports even though packet isn't 1st fragment
	   and doesn't include UDP header.

comments ?

-----------------------------------------------------------------
Yasuyuki KOZAKAI @ USAGI Project <yasuyuki.kozakai@toshiba.co.jp>

Re: The latest nf_conntrack

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 4004 bytes --]

Hi Yasuyuki!

On Mon, Nov 01, 2004 at 05:27:18PM +0900, Yasuyuki Kozakai wrote:

> I finished synchronizing nf_conntrack with ip_conntrack in linux 2.6.10-rc1.
> I need to cleanup and do more tests, anyway, I committed it to pom-ng.

Thanks again, Yasuyuki.  I didn't have time to review your latest
version until today.  I am happy that the issues we discussed at the
workshop have been resolved.

I allowed myself to commit a minor modification:  The include path for
new header files is <linux/netfilter/ipv4> instead of
<linux/netfilter_ipv4>.  Old headers stay at their current position, but
I think in the end (after nf_conntrack is merged) they should be moved
too (and we just provide dummy include files that include the new files
for backwards compatibility).  

> From the discussion at Netfilter developer workshop, I deleted the
> very complex code reallocating conntrack from the basic slab cache
> when destroying the slab cache for conntrack which has "help" part.
> Instead, all slab caches are destroyed when unloading nf_conntrack.ko

Great.

> And I re-implemented handling fragmented IPv6 packets. This code is what
> I imagined after workshop, but I recently found that my idea is different
> with the results of discussion at workshop. Then I may change some parts.

Which differences are you talking about?  Is it that you pass the
individual fragments instead of the refreagmented packet?  I think we
were undecided at the workshop about what to do, so it's not really a
difference from the proposed solution ;)

> The summary of handling IPv6 packets is as follows.
> 
> - at the hook NF_IP6_PRE_ROUTING, priority NF_IP6_CONNTRACK_DEFRAG,
> 	- The fragments are cloned and queued. clone skb has the pointer
> 	  to original skb.
> 	- When all fragments are gathered, cloned skb are reassembled.
> 	- All original skbs keep the pointer to reassembled skb in new member
> 	  of sk_buff "nfct_reasm".
> 	- The following hooks of NF_IP6_PRE_ROUTING are called. Please note
> 	  that reassembled skb isn't passed to hooks but original fragments are
> 	  passed.
> 	- NF_STOLEN is returned.

this sounds exactly lik our plan.

> - at the hook NF_IP6_PRE_ROUTING, priority NF_IP6_CONNTRACK,
> 	- If skb has reassembled skb, that is, packet is fragmented,
> 		- If reassembled skb is already parsed, fragmented skb
> 		  reserves the pointer to the conntrack bound with reassembled
> 		  skb. Otherwise, reassembled skb is parsed.
> 	  Otherwise, skb is ordinarily parsed.
> 	- The original fragments are passed to next hooks and IPv6 core
> 	  processing.

again, great.

> By this processing,
> 	1. We can prevent disturbing IPv6 stack. If we introduce
> 	   ip_conntrack style, and when large fragmented packets came in,
> 	   "Too big" isn't sent. Instead, resized fragmented packets are
> 	   forwarded. That isn't natural.

yes.  I think it is even worth to change nf_conntrack_ipv4 to do it the
same way as ipv6 now does it.  But let's not change too many things at
the same time,

> comments ?

I again have to congratulate you on your work.  I think nf_conntrack was
not an easy task, and you managed very well, especially without any strong
previous netfilter development background.  Thanks also to USAGI and
Toshiba for enabling you to contribute this work.

Let me give it some testing.  Maybe I even get around doing some
benchmarking of ip_conntrack versus nf_conntrack, just to get an idea
how big the performance impact of the generalization is.

If everything goes well, I think we can merge in one to two weeks!

-- 
- Harald Welte <laforge@netfilter.org>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: The latest nf_conntrack

[Yasuyuki Kozakai]

Hi, Harald,

From: Harald Welte <laforge@netfilter.org>
Date: Mon, 8 Nov 2004 14:47:09 +0100

> I allowed myself to commit a minor modification:  The include path for
> new header files is <linux/netfilter/ipv4> instead of
> <linux/netfilter_ipv4>.  Old headers stay at their current position, but
> I think in the end (after nf_conntrack is merged) they should be moved
> too (and we just provide dummy include files that include the new files
> for backwards compatibility).  

OK.

> > And I re-implemented handling fragmented IPv6 packets. This code is what
> > I imagined after workshop, but I recently found that my idea is different
> > with the results of discussion at workshop. Then I may change some parts.
> 
> Which differences are you talking about?  Is it that you pass the
> individual fragments instead of the refreagmented packet?

Yes,

>                                                           I think we
> were undecided at the workshop about what to do, so it's not really a
> difference from the proposed solution ;)

OK, I look like misunderstanding.

> > By this processing,
> > 	1. We can prevent disturbing IPv6 stack. If we introduce
> > 	   ip_conntrack style, and when large fragmented packets came in,
> > 	   "Too big" isn't sent. Instead, resized fragmented packets are
> > 	   forwarded. That isn't natural.
>
> yes.  I think it is even worth to change nf_conntrack_ipv4 to do it the
> same way as ipv6 now does it.  But let's not change too many things at
> the same time,

Sounds good and it's challenging because we have to consider about IPv4 NAT.

> > comments ?
> 
> I again have to congratulate you on your work.  I think nf_conntrack was
> not an easy task, and you managed very well, especially without any strong
> previous netfilter development background.  Thanks also to USAGI and
> Toshiba for enabling you to contribute this work.
> 
> Let me give it some testing.  Maybe I even get around doing some
> benchmarking of ip_conntrack versus nf_conntrack, just to get an idea
> how big the performance impact of the generalization is.
> 
> If everything goes well, I think we can merge in one to two weeks!

Great! I think it's better to test more before merging.
I may forget some changes of ip_conntrack. Actually, I found a missing
code for marking and fixed today.

By the way, could you check copyright and so on ? I have no idea about
the correct way. At least, nf_conntrack_l3proto_ipv6.c is written from scratch
and has no problem, I think.

And please note that I applied some patches from Michal to add some changes
for ip_conntrack in linux 2.6.9. His Signed-off-by is following.

	Signed-off-by: Michal Rokos <michal@rokos.info>

But as you know, these changes are mixed in nf_conntrack of pom-ng
and I don't have trees just before/after applying them.

And a lot codes are copied from ip_conntrack, I suspects that we can
enumerate all Signed-off-by. How we should manage this ?

> -- 
> - Harald Welte <laforge@netfilter.org>             http://www.netfilter.org/
> ============================================================================
>   "Fragmentation is like classful addressing -- an interesting early
>    architectural error that shows how much experimentation was going
>    on while IP was being designed."                    -- Paul Vixie

Re: The latest nf_conntrack

[Henrik Nordstrom]

On Thu, 11 Nov 2004, Yasuyuki Kozakai wrote:

>> I think we were undecided at the workshop about what to do, so it's not 
>> really a difference from the proposed solution ;)
>
> OK, I look like misunderstanding.

There was a lot of minor confusion at the workshop when this specific 
topic (handling of fragments within IPv6 connection tracking) was 
discussed. I remember that it was quite apparent when taking the notes 
that there appeared to be conflicts in several of the ideas, and me 
loosing the relevant notes of the discussion did not make this any 
clearer..

>> yes.  I think it is even worth to change nf_conntrack_ipv4 to do it the
>> same way as ipv6 now does it.  But let's not change too many things at
>> the same time,
>
> Sounds good and it's challenging because we have to consider about IPv4 NAT.

What about keeping the fragments around, and reuse them later unless the 
payload needed to be touched by NAT helpers?

Plain NAT is not really a problem here even if IP theoretically allows 
fragments covering the protocol header, only the use of NAT helpers is of 
major concern to how to handle fragments in IPv4 NAT. Fragments covering 
the protocol headers can be considered invalid as these would require the 
initial fragment to go below the minimum allowed fragment size.

Regards
Henrik

Re: The latest nf_conntrack

[David S. Miller]

Hello Yasuyuki-san,

On Thu, 11 Nov 2004 06:51:23 +0900 (JST)
Yasuyuki Kozakai <yasuyuki.kozakai@toshiba.co.jp> wrote:

> And a lot codes are copied from ip_conntrack, I suspects that we can
> enumerate all Signed-off-by. How we should manage this ?

The best thing to do in this case is something like this:

/*
 ...
 * Based largely upon the original ip_conntrack code which
 * had the following copyright information:
 *
 *   (C) 1999-2001 Paul `Rusty' Russell  
 *   (C) 2002-2004 Netfilter Core Team <coreteam@netfilter.org>
 ...
 */

Re: The latest nf_conntrack

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 1614 bytes --]

On Thu, Nov 11, 2004 at 06:51:23AM +0900, Yasuyuki Kozakai wrote:
> > yes.  I think it is even worth to change nf_conntrack_ipv4 to do it the
> > same way as ipv6 now does it.  But let's not change too many things at
> > the same time,
> 
> Sounds good and it's challenging because we have to consider about IPv4 NAT.

Well, in the case of NAT having to enlarge the packet, we could fall
back to modifying the defragmented skb, and fragmenting it at
POST_ROUTING.

But as stated before, that's future extensions/enhancements.

> By the way, could you check copyright and so on ? I have no idea about
> the correct way. At least, nf_conntrack_l3proto_ipv6.c is written from scratch
> and has no problem, I think.

I've reviewed the copyright headers of each file, and I think they are 
all fine the way you did them.

> And a lot codes are copied from ip_conntrack, I suspects that we can
> enumerate all Signed-off-by. How we should manage this ?

This is not neccessarry.  If somebody has already submitted a patch with
'signed-off-by' to the mainline kernel (and this patch is even included
in the tree), then we don't need to document this again.  At least from
a copyright point of view it doesn't make sense.

-- 
- Harald Welte <laforge@netfilter.org>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: The latest nf_conntrack

[Yasuyuki Kozakai]

Hi,

From: Harald Welte <laforge@netfilter.org>
Date: Thu, 11 Nov 2004 08:41:18 +0100

> > By the way, could you check copyright and so on ? I have no idea about
> > the correct way. At least, nf_conntrack_l3proto_ipv6.c is written from scratch
> > and has no problem, I think.
> 
> I've reviewed the copyright headers of each file, and I think they are 
> all fine the way you did them.

From: "David S. Miller" <davem@davemloft.net>
Date: Wed, 10 Nov 2004 15:20:04 -0800

> > And a lot codes are copied from ip_conntrack, I suspects that we can
> > enumerate all Signed-off-by. How we should manage this ?
> 
> The best thing to do in this case is something like this:
> 
> /*
>  ...
>  * Based largely upon the original ip_conntrack code which
>  * had the following copyright information:
>  *
>  *   (C) 1999-2001 Paul `Rusty' Russell  
>  *   (C) 2002-2004 Netfilter Core Team <coreteam@netfilter.org>
>  ...
>  */

I discussed about these parts in all files with Yoshifuji-san and
we changed them.

Harald, if you find undesirable parts, please let me know them.

Basically, I deleted Copyrights of USAGI/WIDE Project from the files
which I didn't add change almost part to. And I added comments as David said
to IPv6/ICMPv6 modules.

Regards,
-----------------------------------------------------------------
Yasuyuki KOZAKAI @ USAGI Project <yasuyuki.kozakai@toshiba.co.jp>