SO_ORIGINAL_DST does not work in nat/OUTPUT anymore

SO_ORIGINAL_DST does not work in nat/OUTPUT anymore

[Jens Hoelldampf]

Hi,

the upstream patch from

https://bugzilla.netfilter.org/bugzilla/show_bug.cgi?id=91

destroys the functionality of SO_ORIGINAL_DST for local generated
packets. It returns the local address the connection is redirected to
instead of the original destination address. As a result you cannot use
local transparent proxies anymore.

The responsible change is

diff -ru linux-orig/net/ipv4/ip_output.c linux-new/net/ipv4/ip_output.c
--- linux-orig/net/ipv4/ip_output.c	2005-03-25 14:48:13.543911760 -0500
+++ linux-new/net/ipv4/ip_output.c	2005-03-26 01:01:13.064616240 -0500
@@ -195,6 +195,8 @@
 	nf_debug_ip_finish_output2(skb);
 #endif /*CONFIG_NETFILTER_DEBUG*/

+	nf_reset(skb);
+
 	if (hh) {
 		int hh_alen;

After removing the added line SO_ORIGINAL_DST returns the expected address.

Should I open a new bug or reopen the old one?

Regards,
 Jens

Re: SO_ORIGINAL_DST does not work in nat/OUTPUT anymore

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 1106 bytes --]

On Sun, Jul 10, 2005 at 04:18:16PM +0200, Jens Hoelldampf wrote:
> Hi,
> 
> the upstream patch from
> 
> https://bugzilla.netfilter.org/bugzilla/show_bug.cgi?id=91
> 
> destroys the functionality of SO_ORIGINAL_DST for local generated
> packets. It returns the local address the connection is redirected to
> instead of the original destination address. As a result you cannot use
> local transparent proxies anymore.

ouch.  It seems like we really need to revert that change and find a
different solution for conntrack unloading :(

Thanks for reporting this.

Patrick, are you already working on this?

> Should I open a new bug or reopen the old one?

please open a new one, since it's a different problem.

-- 
- Harald Welte <laforge@netfilter.org>                 http://netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: SO_ORIGINAL_DST does not work in nat/OUTPUT anymore

[Patrick McHardy]

Harald Welte wrote:
> On Sun, Jul 10, 2005 at 04:18:16PM +0200, Jens Hoelldampf wrote:
> 
>>the upstream patch from
>>
>>https://bugzilla.netfilter.org/bugzilla/show_bug.cgi?id=91
>>
>>destroys the functionality of SO_ORIGINAL_DST for local generated
>>packets. It returns the local address the connection is redirected to
>>instead of the original destination address. As a result you cannot use
>>local transparent proxies anymore.
> 
> ouch.  It seems like we really need to revert that change and find a
> different solution for conntrack unloading :(
> 
> Thanks for reporting this.
> 
> Patrick, are you already working on this?

I was thinking about reverting it because of the RST over loopback
problem reported in another thread, but wanted to discuss it with
you first. I don't see how this change can affect SO_ORIGINAL_DST
though, all it does is look up the conntrack entry by a tuple
generated from data stored in the socket. It never even gets to
see a packet.

Regards
Patrick

Re: SO_ORIGINAL_DST does not work in nat/OUTPUT anymore

[Henrik Nordstrom]

On Sun, 10 Jul 2005, Patrick McHardy wrote:

> you first. I don't see how this change can affect SO_ORIGINAL_DST
> though, all it does is look up the conntrack entry by a tuple
> generated from data stored in the socket. It never even gets to
> see a packet.

I would guess the problem is that there may be two conntracks for the 
session, one before nat and another after, or that the tuple lookup does 
not look at the expected tuple variant needed to find the conntrack.. If I 
understood the Jens problem correctly this traffic goes over loopback via 
DNAT..  (output -> DNAT -> loopback -> input -> socket)

Regards
Henrik

Re: SO_ORIGINAL_DST does not work in nat/OUTPUT anymore

[Patrick McHardy]

Henrik Nordstrom wrote:
> On Sun, 10 Jul 2005, Patrick McHardy wrote:
> 
>> you first. I don't see how this change can affect SO_ORIGINAL_DST
>> though, all it does is look up the conntrack entry by a tuple
>> generated from data stored in the socket. It never even gets to
>> see a packet.
> 
> I would guess the problem is that there may be two conntracks for the
> session, one before nat and another after, or that the tuple lookup does
> not look at the expected tuple variant needed to find the conntrack.. If
> I understood the Jens problem correctly this traffic goes over loopback
> via DNAT..  (output -> DNAT -> loopback -> input -> socket)

You're absolutely right. After performing DNAT the packet doesn't match
either side of the conntrack in PRE_ROUTING and a new one is created.

iptables -t nat -A OUTPUT -d 1.2.3.4 -j DNAT --to-dest 127.0.0.1

telnet 1.2.3.4 22 results in this conntrack:

tcp      6 431997 ESTABLISHED src=172.16.1.123 dst=1.2.3.4 sport=37395
dport=22 packets=3 bytes=164 src=127.0.0.1 dst=172.16.1.123 sport=22
dport=37395 packets=2 bytes=153 [ASSURED] mark=0 use=1

In PRE_ROUTING the packet will be 172.16.1.123:37395->127.0.0.1:22,
which doesn't match and a new conntrack is created:

tcp      6 431997 ESTABLISHED src=172.16.1.123 dst=127.0.0.1 sport=37395
dport=22 packets=3 bytes=164 src=127.0.0.1 dst=172.16.1.123 sport=22
dport=1024 packets=2 bytes=153 [ASSURED] mark=0 use=1

This is the one where SO_ORIGINAL_DST gets its destination from.

Thanks for the hint. Harald, if I can't think of anything else (seems
likely) I'll revert the patch.

Regards
Patrick

Re: SO_ORIGINAL_DST does not work in nat/OUTPUT anymore

[Harald Welte]

On Sun, Jul 10, 2005 at 07:53:16PM +0200, Patrick McHardy wrote:

> Thanks for the hint. Harald, if I can't think of anything else (seems
> likely) I'll revert the patch.

Yes, I'm also in favour of reverting the patch.  The number of functional bugs
it has caused by far outweigh the gain of save 'rmmod' without loops.

-- 
- Harald Welte <laforge@netfilter.org>                 http://netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

Re: SO_ORIGINAL_DST does not work in nat/OUTPUT anymore

[Patrick McHardy]

Harald Welte wrote:
> Yes, I'm also in favour of reverting the patch.  The number of functional bugs
> it has caused by far outweigh the gain of save 'rmmod' without loops.

Agreed, unfortunately. If we can't get module unload to work reliably we
should introduce another way to flush the conntrack table however. I'm
thinking of something easy like a /proc file.

Regards
Patrick

Re: SO_ORIGINAL_DST does not work in nat/OUTPUT anymore

[Jan Engelhardt]

>> Yes, I'm also in favour of reverting the patch.  The number of functional bugs
>> it has caused by far outweigh the gain of save 'rmmod' without loops.
>
>Agreed, unfortunately. If we can't get module unload to work reliably we
>should introduce another way to flush the conntrack table however. I'm
>thinking of something easy like a /proc file.

But in the time that passes between echo 1 >/proc/flushme and rmmod new 
entries could show up, if it's "just" a flush and not a flush-and-disable 
operation.


Jan Engelhardt                                                               
--                                                                            
| Alphagate Systems, http://alphagate.hopto.org/

Re: SO_ORIGINAL_DST does not work in nat/OUTPUT anymore

[Patrick McHardy]

Jan Engelhardt wrote:
>>>Yes, I'm also in favour of reverting the patch.  The number of functional bugs
>>>it has caused by far outweigh the gain of save 'rmmod' without loops.
>>
>>Agreed, unfortunately. If we can't get module unload to work reliably we
>>should introduce another way to flush the conntrack table however. I'm
>>thinking of something easy like a /proc file.
> 
> But in the time that passes between echo 1 >/proc/flushme and rmmod new 
> entries could show up, if it's "just" a flush and not a flush-and-disable 
> operation.

"flush-and-disable" (should actually be disable-and-flush) can be
achieved by adding a rule to drop everything before flushing.
Its not meant to clean up before rmmod (conntrack tries to do this
itself) but to kick out old conntrack entries after ruleset changes.

Regards
Patrick

Re: SO_ORIGINAL_DST does not work in nat/OUTPUT anymore

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 1300 bytes --]

On Mon, Jul 11, 2005 at 01:06:19PM +0200, Jan Engelhardt wrote:
> >> Yes, I'm also in favour of reverting the patch.  The number of functional bugs
> >> it has caused by far outweigh the gain of save 'rmmod' without loops.
> >
> >Agreed, unfortunately. If we can't get module unload to work reliably we
> >should introduce another way to flush the conntrack table however. I'm
> >thinking of something easy like a /proc file.
> 
> But in the time that passes between echo 1 >/proc/flushme and rmmod new 
> entries could show up, if it's "just" a flush and not a flush-and-disable 
> operation.

I think you misunderstood Patrkick.  The /proc file would be to flush
the table and therefore making the rmmod-insmod cycle if ip_conntrack no
longer necessarry.

If we could easily get rid of all conntracks [and their references], we
wouldn't need the /proc file, but could do it in the modules fini()
function.

-- 
- Harald Welte <laforge@netfilter.org>                 http://netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: SO_ORIGINAL_DST does not work in nat/OUTPUT anymore

[Phil Oester]

[-- Attachment #1: Type: text/plain, Size: 837 bytes --]

On Sun, Jul 10, 2005 at 10:24:00PM +0200, Harald Welte wrote:
> On Sun, Jul 10, 2005 at 07:53:16PM +0200, Patrick McHardy wrote:
> 
> > Thanks for the hint. Harald, if I can't think of anything else (seems
> > likely) I'll revert the patch.
> 
> Yes, I'm also in favour of reverting the patch.  The number of functional bugs
> it has caused by far outweigh the gain of save 'rmmod' without loops.

I humbly re-resubmit my original patch.  The problem occurs (most frequently)
in the case of raw sockets, which my patch addresses.  Patrick will point out
that stopped qdiscs are not handled by this patch, and I concur.  However, it
would seem that fixing the most common cause of module unload problems is
better than the plethora of problems the alternative patch has caused...

Phil

Signed-off-by: Phil Oester <kernel@linuxace.com>



[-- Attachment #2: patch-unload --]
[-- Type: text/plain, Size: 1938 bytes --]

diff -purN linux-orig/net/ipv4/ip_output.c linux-new/net/ipv4/ip_output.c
--- linux-orig/net/ipv4/ip_output.c	2005-06-17 15:48:29.000000000 -0400
+++ linux-new/net/ipv4/ip_output.c	2005-06-20 14:47:58.000000000 -0400
@@ -196,8 +196,6 @@ static inline int ip_finish_output2(stru
 	nf_debug_ip_finish_output2(skb);
 #endif /*CONFIG_NETFILTER_DEBUG*/
 
-	nf_reset(skb);
-
 	if (hh) {
 		int hh_alen;
 
diff -purN linux-orig/net/ipv4/netfilter/ip_conntrack_standalone.c linux-new/net/ipv4/netfilter/ip_conntrack_standalone.c
--- linux-orig/net/ipv4/netfilter/ip_conntrack_standalone.c	2005-06-17 15:48:29.000000000 -0400
+++ linux-new/net/ipv4/netfilter/ip_conntrack_standalone.c	2005-06-20 14:47:58.000000000 -0400
@@ -432,6 +432,13 @@ static unsigned int ip_conntrack_defrag(
 				        const struct net_device *out,
 				        int (*okfn)(struct sk_buff *))
 {
+#if !defined(CONFIG_IP_NF_NAT) && !defined(CONFIG_IP_NF_NAT_MODULE)
+	/* Previously seen (loopback)?  Ignore.  Do this before
+           fragment check. */
+	if ((*pskb)->nfct)
+		return NF_ACCEPT;
+#endif
+
 	/* Gather fragments. */
 	if ((*pskb)->nh.iph->frag_off & htons(IP_MF|IP_OFFSET)) {
 		*pskb = ip_ct_gather_frags(*pskb,
diff -purN linux-orig/net/packet/af_packet.c linux-new/net/packet/af_packet.c
--- linux-orig/net/packet/af_packet.c	2005-06-17 15:48:29.000000000 -0400
+++ linux-new/net/packet/af_packet.c	2005-06-20 14:48:38.000000000 -0400
@@ -274,6 +274,9 @@ static int packet_rcv_spkt(struct sk_buf
 	dst_release(skb->dst);
 	skb->dst = NULL;
 
+	/* drop conntrack reference */
+	nf_reset(skb);
+
 	spkt = (struct sockaddr_pkt*)skb->cb;
 
 	skb_push(skb, skb->data-skb->mac.raw);
@@ -517,6 +520,9 @@ static int packet_rcv(struct sk_buff *sk
 	dst_release(skb->dst);
 	skb->dst = NULL;
 
+	/* drop conntrack reference */
+	nf_reset(skb);
+
 	spin_lock(&sk->sk_receive_queue.lock);
 	po->stats.tp_packets++;
 	__skb_queue_tail(&sk->sk_receive_queue, skb);

Re: SO_ORIGINAL_DST does not work in nat/OUTPUT anymore

[Patrick McHardy]

Phil Oester wrote:
> I humbly re-resubmit my original patch.

It doesn't apply to the current tree. I'm going to fix it up manually,
but please base future patches on linus's latest tree.

> The problem occurs (most frequently)
> in the case of raw sockets, which my patch addresses.

Unfortunately this is only true in your case. There was a report without
any packet sockets involved just two weeks ago and I have multiple
reports at work without any packet sockets. In these cases qdiscs aren't
the problem either, so it seems there are still spots that need fixing
after reverting the patch.

Anyway, your patch is the best we can do right now, thanks.

Re: SO_ORIGINAL_DST does not work in nat/OUTPUT anymore

[Roberto Nibali]

Patrick McHardy wrote:
> Phil Oester wrote:
> 
>>I humbly re-resubmit my original patch.
> 
> 
> It doesn't apply to the current tree. I'm going to fix it up manually,
> but please base future patches on linus's latest tree.

... and for the 2.4.x case? I most certainly haven't had the time to test this
use case yet but the problems should be the same.

Cheers,
Roberto Nibali, ratz
-- 
-------------------------------------------------------------
addr://Rathausgasse 31, CH-5001 Aarau  tel://++41 62 823 9355
http://www.terreactive.com             fax://++41 62 823 9356
-------------------------------------------------------------
terreActive AG                       Wir sichern Ihren Erfolg
-------------------------------------------------------------

Re: SO_ORIGINAL_DST does not work in nat/OUTPUT anymore

[Patrick McHardy]

Roberto Nibali wrote:
> ... and for the 2.4.x case? I most certainly haven't had the time to test this
> use case yet but the problems should be the same.

Yes, we can't submit the patch for 2.4 either. It has no bridging,
so this problem is 2.6 only, but the RST over loopback and the
SO_ORIGINAL_DST problem remain. This means we still need to figure
out where your hangs are coming from.

Regards
Patrick

Re: SO_ORIGINAL_DST does not work in nat/OUTPUT anymore

[Roberto Nibali]

Patrick McHardy wrote:
> Roberto Nibali wrote:
> 
>> ... and for the 2.4.x case? I most certainly haven't had the time to
>> test this
>> use case yet but the problems should be the same.
> 
> Yes, we can't submit the patch for 2.4 either.

Darn. But glad we didn't.

> It has no bridging,

Well, I do have the bridging code patch in my 2.4.x patchset because we need the
bridging functionality for a couple of setups. But haven't come around to
testing it yet. I think a considerable amount of people running 2.4.x still also
have the bridging code patched.

> so this problem is 2.6 only, but the RST over loopback and the
> SO_ORIGINAL_DST problem remain. This means we still need to figure
> out where your hangs are coming from.

How may I assist in resolving this issue? Can't we track those references in a
better way?

Regards,
Roberto Nibali, ratz
-- 
-------------------------------------------------------------
addr://Rathausgasse 31, CH-5001 Aarau  tel://++41 62 823 9355
http://www.terreactive.com             fax://++41 62 823 9356
-------------------------------------------------------------
terreActive AG                       Wir sichern Ihren Erfolg
-------------------------------------------------------------

Re: SO_ORIGINAL_DST does not work in nat/OUTPUT anymore

[Patrick McHardy]

Roberto Nibali wrote:
> Patrick McHardy wrote:
>
>>so this problem is 2.6 only, but the RST over loopback and the
>>SO_ORIGINAL_DST problem remain. This means we still need to figure
>>out where your hangs are coming from.
> 
> How may I assist in resolving this issue?

I'm looking for spots where we hold the reference but don't need it
anymore. If I find something interesting I'll let you know.

> Can't we track those references in a better way?

None I can think of.

Regards
Patrick