been a while...what happened to NFC_ALTERED?

been a while...what happened to NFC_ALTERED?

[Dave Horton]

I've hunted around a bit but can't find the answer to my question,  
even though it seems like it must be covered somewhere obvious -  
apologies in advance if I've missed it.

Here's my situation: I wrote a simple netfilter target a couple of  
years back, been running it on Redhat 3, 2.4 kernel successfully for  
quite a while.  I install it on the PREROUTING chain, and I mangle UDP  
packets; modifiying the source and destination addresses.  My code was  
then setting the NFC_ALTERED bit in the nf_cache of the sk_buff so  
that it would get re-processed by netfilter.  Everything fine.  Now I  
simply want to update my code to run on Centos 5, kernel 2.6.18 and  
iptables 1.4.  Whoa, now I find my kernel module doesn't compile  
because that data structure has changed.  If I remove that line of  
code, it compiles, but the packet forwarding does not work (and it  
seems like I stop getting packets routed to my target at all once I  
mangle the first one).

Can someone help me out?  What is the new improved way of indicating  
that I've changed a packet's header or body so that netfilter re- 
examines it?

Re: been a while...what happened to NFC_ALTERED?

[Jan Engelhardt]

On Thursday 2008-09-25 18:38, Dave Horton wrote:

> Here's my situation: I wrote a simple netfilter target a couple of
> years back, been running it on Redhat 3, 2.4 kernel successfully
> for quite a while.  I install it on the PREROUTING chain, and I
> mangle UDP packets; modifiying the source and destination
> addresses.

Modifying it in which way? Maybe what you are trying to achieve is
already solved by other modules.

> My code was then setting the NFC_ALTERED bit in the
> nf_cache of the sk_buff so that it would get re-processed by
> netfilter. Everything fine.  Now I simply want to update my code to
> run on Centos 5, kernel 2.6.18 and iptables 1.4.  Whoa, now I find
> my kernel module doesn't compile because that data structure has
> changed.  If I remove that line of code, it compiles, but the
> packet forwarding does not work (and it seems like I stop getting
> packets routed to my target at all once I mangle the first one).
>
> Can someone help me out?  What is the new improved way of
> indicating that I've changed a packet's header or body so that
> netfilter re-examines it?

The funny thing is, even in Linux 2.5.0, only the IPv6 code ever
examines for NFC_ALTERED. It's like the thing was never used.
See http://jengelh.medozas.de/documents/Netfilter_Modules.pdf
if you need any assistance for moving your extension forward
in time (though 2.6.18 is now so old again...)

RE: been a while...what happened to NFC_ALTERED?

[Horton, Dave]

> > Here's my situation: I wrote a simple netfilter target a couple of
> > years back, been running it on Redhat 3, 2.4 kernel successfully
> > for quite a while.  I install it on the PREROUTING chain, and I
> > mangle UDP packets; modifiying the source and destination
> > addresses.
> 
> Modifying it in which way? Maybe what you are trying to achieve is
> already solved by other modules.

Actually, my code is rather simple.  I've posted it at the bottom of
this response.

> > My code was then setting the NFC_ALTERED bit in the
> > nf_cache of the sk_buff so that it would get re-processed by
> > netfilter. Everything fine.  Now I simply want to update my code to
> > run on Centos 5, kernel 2.6.18 and iptables 1.4.  Whoa, now I find
> > my kernel module doesn't compile because that data structure has
> > changed.  If I remove that line of code, it compiles, but the
> > packet forwarding does not work (and it seems like I stop getting
> > packets routed to my target at all once I mangle the first one).
> >
> > Can someone help me out?  What is the new improved way of
> > indicating that I've changed a packet's header or body so that
> > netfilter re-examines it?
> 
> The funny thing is, even in Linux 2.5.0, only the IPv6 code ever
> examines for NFC_ALTERED. It's like the thing was never used.
> See http://jengelh.medozas.de/documents/Netfilter_Modules.pdf
> if you need any assistance for moving your extension forward
> in time (though 2.6.18 is now so old again...)

Interesting....I was on linux 2.4 kernel though, if that makes any
difference.  
Many thanks for the pointer to the doc, I will read it now.


Here is the main piece of my code:

static unsigned int
target (struct sk_buff **pskb,
		unsigned int hooknum, const struct net_device *in, 
		const struct net_device *out, const void *targetinfo,
		void *userinfo)
{

	struct iphdr *iph = (*pskb)->nh.iph;
	struct udphdr *udph = (void *)iph + (iph->ihl<<2);
	struct ipt_rtp_tuple *ptr, hold;
	int disposition = IPT_CONTINUE ;

	u_int16_t orig_dest = ntohs(udph->dest) ;

	/* Check if we received the packet on a port that is in the 
	 * range we care about.
	 */
	if (orig_dest >= start_port && orig_dest <= end_port) {
		int entry = orig_dest - start_port ;
		
		//spin_lock(&lock);
		ptr = sip_ua_agents + entry;
		hold = *ptr;
		//spin_unlock(&lock);
		
		ptr = &hold;

		switch( ptr->state ) {
		
		case IPT_PACTOLUS_STATE_DISCARD:
			disposition = NF_DROP ;
			break ;

		case IPT_PACTOLUS_STATE_ACCEPT:
			break ;

		case IPT_PACTOLUS_STATE_FORWARD:


			/* modify the source ip:port and the dest
ip:port, 
			 * recalc checksums 
			 */
			udph->source = ptr->new_src_port ;
			udph->dest = ptr->new_dst_port ;
			if (udph->check) {
				u_int32_t newudplen = (*pskb)->len - 
				  				iph->ihl
* 4;

				/* calculate checksum of the data
portion */
				(*pskb)->csum = csum_partial((char
*)udph +
					sizeof(struct udphdr),
					newudplen - sizeof(struct
udphdr), 0);
				udph->check = 0;
				udph->check =
csum_tcpudp_magic(iph->daddr, 
				    	ptr->new_dst_ip,
					newudplen, IPPROTO_UDP, 
					csum_partial((char *)udph,
					sizeof(struct udphdr),
(*pskb)->csum));
			}

			iph->saddr = iph->daddr;
			iph->daddr = ptr->new_dst_ip;

			iph->check = 0 ;
			iph->check = pcs_cheat_check ((unsigned char
*)iph, 
					sizeof (struct iphdr));

			(*pskb)->nfcache |= NFC_ALTERED;
			break ;

		case IPT_PACTOLUS_STATE_REJECT:
			//TODO: implement
			break ;

		default:
			printk(KERN_ALERT "Unknown state '%d' for port
%d\n", ptr->state, orig_dest ) ;
			break ;
		}
	}
	return disposition;
}

RE: been a while...what happened to NFC_ALTERED?

[Jan Engelhardt]

On Thursday 2008-09-25 20:47, Horton, Dave wrote:
>
>Actually, my code is rather simple.  I've posted it at the bottom of
>this response.
>
>Here is the main piece of my code:
>
>		case IPT_PACTOLUS_STATE_FORWARD:
>
>
>			/* modify the source ip:port and the dest
>ip:port, 
>			 * recalc checksums 
>			 */
>			udph->source = ptr->new_src_port ;
>			udph->dest = ptr->new_dst_port ;
>			if (udph->check) {
>				u_int32_t newudplen = (*pskb)->len - 
>				  				iph->ihl
>* 4;
>
>				/* calculate checksum of the data
>portion */
>				(*pskb)->csum = csum_partial((char
>*)udph +
>					sizeof(struct udphdr),
>					newudplen - sizeof(struct
>udphdr), 0);
>				udph->check = 0;
>				udph->check =
>csum_tcpudp_magic(iph->daddr, 
>				    	ptr->new_dst_ip,
>					newudplen, IPPROTO_UDP, 
>					csum_partial((char *)udph,
>					sizeof(struct udphdr),
>(*pskb)->csum));
>			}
>
>			iph->saddr = iph->daddr;
>			iph->daddr = ptr->new_dst_ip;

You cannot change the IP address or L4 port without making conntrack
very unhappy. In the simplest case, you're "just" voiding any state
matching and flow accounting. In the worst case, iph->daddr (and
other fields) does not match skb->nfct->....->daddr (respectively)
anymore and cosmic rays could catalyze up, er, undefined behavior
could arise.

RE: been a while...what happened to NFC_ALTERED?

[Horton, Dave]

> You cannot change the IP address or L4 port without making conntrack
> very unhappy. In the simplest case, you're "just" voiding any state
> matching and flow accounting. In the worst case, iph->daddr (and
> other fields) does not match skb->nfct->....->daddr (respectively)
> anymore and cosmic rays could catalyze up, er, undefined behavior
> could arise.

Hmm.  Well, maybe that's my problem?  Something that I could get away
with in the 2.4 kernel, which I can't any longer?  It is true that in
order for this to work for me on RH3 I had to build a custom kernel that
had a small change to net/ipv4/fib_frontend.c:

--- net/ipv4/fib_frontend.c	Sun Mar 21 07:22:00 1999
+++ net/ipv4/fib_frontend.c	Wed Dec 29 20:29:38 1999
@@ -233,8 +233,13 @@
 
 	if (fib_lookup(&key, &res))
 		goto last_resort;
+#ifdef CONFIG_ALLOW_SPOOFING
+	if (res.type != RTN_UNICAST && res.type != RTN_LOCAL)
+		goto e_inval_res;
+#else
 	if (res.type != RTN_UNICAST)
 		goto e_inval_res;
+#endif
	*spec_dst = FIB_RES_PREFSRC(res);
	fib_combine_itag(itag, &res);
 #ifdef CONFIG_IP_ROUTE_MULTIPATH

I've made the same change on the 2.6 kernel I am trying to get running
with.

This is all in the context of a single-purpose, specialized server that
I build which does a very specific packet forwarding task.  I control
the whole server (no other software needs to run on it) so hopefully I
can somewhat control interactions. Regardless of the fact that it is
non-standard, it does what I need and I need to somehow create that same
functionality under 2.6, if possible.  Do you think I should be looking
at some sort of alternative approach?  The simplest way to describe what
I need to do is that I need to receive UDP packets on a specified set of
ports and send them to a specified remote address:port after changing
the source ip address:port to my local address and a specified port.

Re: been a while...what happened to NFC_ALTERED?

[Philip Craig]

Horton, Dave wrote:
> This is all in the context of a single-purpose, specialized server that
> I build which does a very specific packet forwarding task.  I control
> the whole server (no other software needs to run on it) so hopefully I
> can somewhat control interactions. Regardless of the fact that it is
> non-standard, it does what I need and I need to somehow create that same
> functionality under 2.6, if possible.  Do you think I should be looking
> at some sort of alternative approach?  The simplest way to describe what
> I need to do is that I need to receive UDP packets on a specified set of
> ports and send them to a specified remote address:port after changing
> the source ip address:port to my local address and a specified port.

Standard nat rules can do that:
iptables -t nat -A PREROUTING -p udp --dport start:end -j DNAT --to-destination newdest:newdestport
iptables -t nat -A POSTROUTING -p udp -d newdest --dport newdestport -j SNAT --to-source newsrc:newsrcport

RE: been a while...what happened to NFC_ALTERED?

[Horton, Dave]

> Standard nat rules can do that:
> iptables -t nat -A PREROUTING -p udp --dport start:end -j DNAT --to-
> destination newdest:newdestport
> iptables -t nat -A POSTROUTING -p udp -d newdest --dport newdestport
-j
> SNAT --to-source newsrc:newsrcport

Well, maybe I could use that, or more likely base my kernel module off
that code.  The reason that I probably can't use the DNAT target as is,
is because I also have to inspect the packets, and for some (very small)
number of them, send the packet up to userspace for further processing.
Also, my userspace process needs to frequently (hundreds of times per
second) change the forwarding rules, so for performance reasons I want
to specify those changes via commands over a netlink socket rather than
adding and removing iptables rules.

Does that make sense?  Should I be looking at creating a target similar
to DNAT, and using that code as reference?

Re: been a while...what happened to NFC_ALTERED?

[Philip Craig]

Horton, Dave wrote:
> Well, maybe I could use that, or more likely base my kernel module off
> that code.  The reason that I probably can't use the DNAT target as is,
> is because I also have to inspect the packets, and for some (very small)
> number of them, send the packet up to userspace for further processing.
> Also, my userspace process needs to frequently (hundreds of times per
> second) change the forwarding rules, so for performance reasons I want
> to specify those changes via commands over a netlink socket rather than
> adding and removing iptables rules.
> 
> Does that make sense?  Should I be looking at creating a target similar
> to DNAT, and using that code as reference?

Try to use the existing DNAT target if possible.  You can make the DNAT
conditional by putting all your logic in a custom match.  But also
consider using the ipset patch rather than a custom match.  You may
be able to structure the rules so you only need to update the ipset when
your forwarding rules change.  ipset is currently sockopt only, but
netlink version is in progress.

Also do your forwarding rules change per packet or per connection?
Note that iptables nat is based on connection tracking.  If you want per
packet nat decisions (stateless nat) then use iproute2.

RE: been a while...what happened to NFC_ALTERED?

[Horton, Dave]

I hope you guys can stick with me and give me a bit more guidance.  I've
made a few changes, but haven't yet got it working.  I don't think I can
take the suggestion of using the dnat target, because I need to see and
process every packet which I understand I can't do with nat.

Not sure if I said this before, but I am installing my target on the
PREROUTING chain and mangle table with a rule like this:

iptables -I PREROUTING 1 -t mangle --proto udp --dport 20000:21000 -j
<my-target> --ports 20000:21000

Secondly, since I last wrote I followed the instructions in the doc Jan
passed me, so that I have now updated my code to xtables. Besides my
target function, I also have a checkentry function to process the
parameters passed from iptables.  I've put in printk debug statements to
verify that the iptables integration and is working fine, and that I am
indeed getting the packets from netfilter which I am.

That brings me to the first of my questions.  Phillip Craig mentioned to
me on a earlier thread that my target function is called from
ipt_local_hook ("In net/ipv4/netfilter/iptable_mangle.c,
ipt_local_hook() saves the the values which affect routing, then
compares them after processing the rules.").  However, I went ahead and
put some printk statements into that function and it doesn't seem like
it is being called for each packet that comes to me.  In other words, I
see my target function getting called lots of times without being called
from ipt_local_hook; I do seem ipt_local_hook being called once or twice
but much less and generally not for packets that are coming to my
target.  Is this because I am hanging my hook off the PREROUTING chain?
Is there different code that calls my target in this case?  The reason I
would like to know is so that I can look at the code and see if it is
examining the changes that I have made to the headers.

So, the current state is that I am getting the packets, changing the ip
destination address and udp port (and ip source address and udp port)
but the packets do not appear to get re-routed and dispatched back to
the network. I even tried explicitly calling ip_route_me_harder from
within my target but that did not change anything.

--Dave
> -----Original Message-----
> From: jengelh@sovereign.computergmbh.de
> [mailto:jengelh@sovereign.computergmbh.de] On Behalf Of Jan Engelhardt
> Sent: Thursday, September 25, 2008 9:13 PM
> To: Horton, Dave
> Cc: netfilter-devel@vger.kernel.org
> Subject: RE: been a while...what happened to NFC_ALTERED?
> 
> 
> On Thursday 2008-09-25 20:47, Horton, Dave wrote:
> >
> >Actually, my code is rather simple.  I've posted it at the bottom of
> >this response.
> >
> >Here is the main piece of my code:
> >
> >		case IPT_PACTOLUS_STATE_FORWARD:
> >
> >
> >			/* modify the source ip:port and the dest
> >ip:port,
> >			 * recalc checksums
> >			 */
> >			udph->source = ptr->new_src_port ;
> >			udph->dest = ptr->new_dst_port ;
> >			if (udph->check) {
> >				u_int32_t newudplen = (*pskb)->len -
> >				  				iph->ihl
> >* 4;
> >
> >				/* calculate checksum of the data
> >portion */
> >				(*pskb)->csum = csum_partial((char
> >*)udph +
> >					sizeof(struct udphdr),
> >					newudplen - sizeof(struct
> >udphdr), 0);
> >				udph->check = 0;
> >				udph->check =
> >csum_tcpudp_magic(iph->daddr,
> >				    	ptr->new_dst_ip,
> >					newudplen, IPPROTO_UDP,
> >					csum_partial((char *)udph,
> >					sizeof(struct udphdr),
> >(*pskb)->csum));
> >			}
> >
> >			iph->saddr = iph->daddr;
> >			iph->daddr = ptr->new_dst_ip;
> 
> You cannot change the IP address or L4 port without making conntrack
> very unhappy. In the simplest case, you're "just" voiding any state
> matching and flow accounting. In the worst case, iph->daddr (and
> other fields) does not match skb->nfct->....->daddr (respectively)
> anymore and cosmic rays could catalyze up, er, undefined behavior
> could arise.

Re: been a while...what happened to NFC_ALTERED?

[Philip Craig]

Dave Horton wrote:
> Can someone help me out?  What is the new improved way of indicating  
> that I've changed a packet's header or body so that netfilter re- 
> examines it?

In net/ipv4/netfilter/iptable_mangle.c, ipt_local_hook() saves the
the values which affect routing, then compares them after processing
the rules.  What are you changing that isn't covered by that already?