Re: [netfilter-core] Mangle table rules are not taken into account in preliminary routing decision

Re: [netfilter-core] Mangle table rules are not taken into account in preliminary routing decision

[Patrick McHardy]

Konstantin Ushakov wrote:
> Netfilter team,
> 
> we use netfilter under linux kernel 2.4.31 and have the problem
> described below.
> !Note, that it can be easily reproduced for latest kernels.
> 
>  - we want to connect to some on tcp port #80
>  - in the kernel and in some time we get to ip_route_output_slow function
>  - in this function kernel searches for route for the packet:
>          if (fib_lookup(&key, &res)) {
>             res.fi = NULL;
>             if (oldkey->oif) {               <- oif is zero at this
> point, so we miss that "if"
>                 ......
>             }
>             ...
>             err = -ENETUNREACH;
>             goto out;
>          }
>     It fails to find one as we don't have fwmark set for the packet and
> there is not route for packets without fwmark (see configuration
> attached). So, ENETUNREACH is returned and the packet fails to be sent.
> In fact the packet could be routed
> correctly, but this would happen in ip_build_xmit function in netfilter
> hook for LOCAL_OUT packets.
> 
> Questions:
>  - is it a bug or it's a deliberate decision to have such behaviour?
>  - is there any known add-hock solution for the problem?


Its a consequence of how routing by fwmark works. Its not perfect,
but I don't see a better solution since the initial routing takes
place before we even have a packet.

Just add a route to the dummy device or something like that, that
should make sure you don't get ENETUNREACH.

Re: [netfilter-core] Mangle table rules are not taken into account in preliminary routing decision

[Konstantin Ushakov]

Patrick McHardy wrote:
> Konstantin Ushakov wrote:
>> Netfilter team,
>>
>> we use netfilter under linux kernel 2.4.31 and have the problem
>> described below.
>> !Note, that it can be easily reproduced for latest kernels.
>>
>>  - we want to connect to some on tcp port #80
>>  - in the kernel and in some time we get to ip_route_output_slow
>> function
>>  - in this function kernel searches for route for the packet:
>>          if (fib_lookup(&key, &res)) {
>>             res.fi = NULL;
>>             if (oldkey->oif) {               <- oif is zero at this
>> point, so we miss that "if"
>>                 ......
>>             }
>>             ...
>>             err = -ENETUNREACH;
>>             goto out;
>>          }
>>     It fails to find one as we don't have fwmark set for the packet and
>> there is not route for packets without fwmark (see configuration
>> attached). So, ENETUNREACH is returned and the packet fails to be sent.
>> In fact the packet could be routed
>> correctly, but this would happen in ip_build_xmit function in netfilter
>> hook for LOCAL_OUT packets.
>>
>> Questions:
>>  - is it a bug or it's a deliberate decision to have such behaviour?
>>  - is there any known add-hock solution for the problem?
>
>
> Its a consequence of how routing by fwmark works. Its not perfect,
> but I don't see a better solution since the initial routing takes
> place before we even have a packet.
>
> Just add a route to the dummy device or something like that, that
> should make sure you don't get ENETUNREACH.

I'm afraid that dummy route does not solve the problem. I mean
  - we should not pass out the packets, so where should the route lead?
To loopback?
  - another thing is that on 'send' (for, say, some external address,
port 239)
    with dummy route we hang, but if in fact the packet can't be routed,
    we should get ENETUNREACH.

Idea that we had is the following:

we mark all packets that have passed netfilter (mangle table) with a
specific mark (see configuration below).
We add 2 rules:
 - unreachable, for packets that have passed mange table but should not
be routed
 - rule that lookup table #100 for all packets, in table #100 we have
route like
   ip route add default via 127.0.0.2 table 100

Local traffic that goes to tcp port 80 is routed correctly. Forwarded
traffic is not routed,
ENETUNREACH is received on the lan side. BUT for local traffic that
should not be forwarded,
we don't receive UNREACH, 'send' just hangs.

Example:

on host on LAN side of the router:
bash$ nc 192.168.1.5 81
(UNKNOWN) [192.168.1.5] 80 (www) : No route to host

BUT if we issue that same command on the router itself, it handgs.


That's the situation, so two questions:
   - is there any other common way, except for dummy route... or what
should the route look, so
     it does not change behaviour of applications (see comments above).
   - what is wrong with our new idea? I mean it's looks like a bug in
the kernel, but I don't
     understand exactly where it is.

=================================================
Configuration:

# iptables -L -nv -t mangle
Chain PREROUTING (policy ACCEPT 715 packets, 80324 bytes)
 pkts bytes target     prot opt in     out     source              
destination        
  707 78919 MARK       all  --  *      *       0.0.0.0/0           
0.0.0.0/0           MARK set 0xb

Chain INPUT (policy ACCEPT 700 packets, 78677 bytes)
 pkts bytes target     prot opt in     out     source              
destination        

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source              
destination        

Chain OUTPUT (policy ACCEPT 118 packets, 64650 bytes)
 pkts bytes target     prot opt in     out     source              
destination        
    0     0 MARK       all  --  *      *       0.0.0.0/0           
0.0.0.0/0           MARK set 0xb
    0     0 MARK       tcp  --  *      *       0.0.0.0/0           
0.0.0.0/0           tcp dpt:80 MARK set 0xa

Chain POSTROUTING (policy ACCEPT 52 packets, 4796 bytes)
 pkts bytes target     prot opt in     out     source              
destination 


Rules:
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default
32768:  from all fwmark 0xa lookup 10
40000:  from all fwmark 0xb lookup 99 unreachable
50000:  from all lookup 100

Thanks once again,
Konstantin.

Re: [netfilter-core] Mangle table rules are not taken into account in preliminary routing decision

[Patrick McHardy]

Konstantin Ushakov wrote:
> Patrick McHardy wrote:
>>> Questions:
>>>  - is it a bug or it's a deliberate decision to have such behaviour?
>>>  - is there any known add-hock solution for the problem?
>>
>> Its a consequence of how routing by fwmark works. Its not perfect,
>> but I don't see a better solution since the initial routing takes
>> place before we even have a packet.
>>
>> Just add a route to the dummy device or something like that, that
>> should make sure you don't get ENETUNREACH.
> 
> I'm afraid that dummy route does not solve the problem. I mean
>   - we should not pass out the packets, so where should the route lead?
> To loopback?


As I said, to the dummy device.

>   - another thing is that on 'send' (for, say, some external address,
> port 239)
>     with dummy route we hang, but if in fact the packet can't be routed,
>     we should get ENETUNREACH.
> [...]
> Idea that we had is the following:
> 
> we mark all packets that have passed netfilter (mangle table) with a
> specific mark (see configuration below).
> We add 2 rules:
>  - unreachable, for packets that have passed mange table but should not
> be routed
>  - rule that lookup table #100 for all packets, in table #100 we have
> route like
>    ip route add default via 127.0.0.2 table 100
> 
> Local traffic that goes to tcp port 80 is routed correctly. Forwarded
> traffic is not routed,
> ENETUNREACH is received on the lan side. BUT for local traffic that
> should not be forwarded,
> we don't receive UNREACH, 'send' just hangs.
> 
> Example:
> 
> on host on LAN side of the router:
> bash$ nc 192.168.1.5 81
> (UNKNOWN) [192.168.1.5] 80 (www) : No route to host
> 
> BUT if we issue that same command on the router itself, it handgs.


Ah, I see the problem. The route returns unreachable, which
iptable_mangle translates to NF_DROP. The problem is that
netfilter itself can't return ENETUNREACH and there is no
valid output function attached to the dst_entry that would
send an icmp unreachable. I think the only thing you could
do is manually call icmp_send(ICMP_DEST_UNREACH) in
ip_route_me_harder for this case.

Re: [netfilter-core] Mangle table rules are not taken into account in preliminary routing decision

[Pascal Hambourg]

Hello,

Patrick McHardy a écrit :
> 
> Ah, I see the problem. The route returns unreachable, which
> iptable_mangle translates to NF_DROP. The problem is that
> netfilter itself can't return ENETUNREACH and there is no
> valid output function attached to the dst_entry that would
> send an icmp unreachable. I think the only thing you could
> do is manually call icmp_send(ICMP_DEST_UNREACH) in
> ip_route_me_harder for this case.

What about the REJECT target ?
-
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [netfilter-core] Mangle table rules are not taken into account in preliminary routing decision

[Konstantin Ushakov]

Pascal Hambourg wrote:
> Hello,
>
> Patrick McHardy a écrit :
>>
>> Ah, I see the problem. The route returns unreachable, which
>> iptable_mangle translates to NF_DROP. The problem is that
>> netfilter itself can't return ENETUNREACH and there is no
>> valid output function attached to the dst_entry that would
>> send an icmp unreachable. I think the only thing you could
>> do is manually call icmp_send(ICMP_DEST_UNREACH) in
>> ip_route_me_harder for this case.
>
> What about the REJECT target ?
Correct me if I'm mistaken, but REJECT target is only valid in filter
table. But the
packet does not reach filter table because of reasons described by
Patric (as we DROP
it after mangle). It is clearly observed by me when I insert LOG into
filter table.
-
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [netfilter-core] Mangle table rules are not taken into account in preliminary routing decision

[Pascal Hambourg]

Konstantin Ushakov a écrit :
>>
>>What about the REJECT target ?
> 
> Correct me if I'm mistaken, but REJECT target is only valid in filter
> table.

Correct.

> But the packet does not reach filter table because of reasons
> described by Patric (as we DROP it after mangle).

Im meant to use the REJECT target /instead of/ an "unreachable" routing 
rule.

Remove
ip rule add from all fwmark 0xb lookup 99 unreachable prio 40000

And add
iptables -t filter -A OUTPUT -m mark --mark 0xb \
   -j REJECT --rejected-with icmp-net-unreachable
-
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html