Policy routing problem

Policy routing problem

[Christoph Pleger]

Hello,

I am running a DHCP-Server that serves multiple subnets. The server has
an IP address in all of these subnets, and its primary IP address in a
subnet that is not served by DHCP. Every IP address has its own VLAN
Ethernet interface, eth0.102, eth0.104, etc. In this setup, the DHCP
server often does not send its unicast replies on the interface where it
received the corresponding request, but on the interface of its primary
IP address, and with that IP. My first thought how to change this was by
setting routes depending on destination addresses, but this would cause
big problems with other services running on the same machine, so I tried
to combine iproute2 and iptables, like this:

iptables -t mangle -A OUTPUT -p udp --source-port bootps \
   -d xxx.xxx.22.0/24 -j MARK --set-mark 122

ip route add to xxx.xxx.22.0/24 dev eth0.122 table 122

ip rule add fwmark 122 table 122

Though I can see in my logs that iptables really sets that mark, routing
does not work as expected and the server still uses its primary IP address
for sending unicast DHCP replies.

What am I doing wrong and what must be done to achieve the desired
behaviour?

Regards
  Christoph

Re: Policy routing problem

[Oleg]

On Tue, Oct 23, 2012 at 01:05:04PM +0200, Christoph Pleger wrote:
> Hello,

  Hi.

> 
> I am running a DHCP-Server that serves multiple subnets. The server has
> an IP address in all of these subnets, and its primary IP address in a
> subnet that is not served by DHCP. Every IP address has its own VLAN
> Ethernet interface, eth0.102, eth0.104, etc. In this setup, the DHCP
> server often does not send its unicast replies on the interface where it
> received the corresponding request, but on the interface of its primary
> IP address, and with that IP. My first thought how to change this was by
> setting routes depending on destination addresses, but this would cause
> big problems with other services running on the same machine, so I tried
> to combine iproute2 and iptables, like this:

  If you have a separate IP for an each interface, everything should be
ok, because of device routes.
  Show ip rou output for your vlans.

Re: Policy routing problem

[Jimmy Thrasibule]

On Tue, 2012-10-23 at 13:05 +0200, Christoph Pleger wrote:
> Hello,

Hello,

> I am running a DHCP-Server that serves multiple subnets. The server has
> an IP address in all of these subnets, and its primary IP address in a
> subnet that is not served by DHCP. Every IP address has its own VLAN
> Ethernet interface, eth0.102, eth0.104, etc. In this setup, the DHCP
> server often does not send its unicast replies on the interface where it
> received the corresponding request, but on the interface of its primary
> IP address, and with that IP. My first thought how to change this was by
> setting routes depending on destination addresses, but this would cause
> big problems with other services running on the same machine, so I tried
> to combine iproute2 and iptables, like this:

Unless you implicitly defined an interface to bind on, your DHCP server
should be listening on all broadcast interfaces. Alternatively, you can
force your DHCP server to bind on the interfaces you want.

If everything is OK on this point, some DHCP requests might come from
the "primary IP" address making the server to respond on that interface.

> iptables -t mangle -A OUTPUT -p udp --source-port bootps \
>    -d xxx.xxx.22.0/24 -j MARK --set-mark 122
> 
> ip route add to xxx.xxx.22.0/24 dev eth0.122 table 122
> 
> ip rule add fwmark 122 table 122
> 
> Though I can see in my logs that iptables really sets that mark, routing
> does not work as expected and the server still uses its primary IP address
> for sending unicast DHCP replies.
> 
> What am I doing wrong and what must be done to achieve the desired
> behaviour?

In OUTPUT chain, the outbound interface is already defined. You should
be marking in PREROUTING. But check your network configuration first as
the problem might come from there.


Jimmy

Re: Policy routing problem

[Christoph Pleger]

Hello,

>   If you have a separate IP for an each interface, everything should be
> ok, because of device routes.
>   Show ip rou output for your vlans.

I explicitly deleted from the main routing table all of what I guess you
mean with device routes, except the one for the primary network interface,
and created a new routing table for every device. To be more accurate than
before, my primary IP address is on eth0 and the VLANs are on eth1.
Routing tables are currently like follows:

root@machine:~# ip rule show
0:      from all lookup local
32749:  from 129.217.63.0/26 lookup 211
32750:  from 129.217.62.0/24 lookup 76
32751:  from 129.217.47.192/26 lookup 214
32752:  from 129.217.47.128/26 lookup 215
32753:  from 129.217.45.0/25 lookup 145
32754:  from 129.217.38.0/24 lookup 138
32755:  from 129.217.33.0/24 lookup 133
32756:  from 129.217.28.0/24 lookup 128
32757:  from 129.217.26.0/24 lookup 126
32758:  from 129.217.22.0/24 lookup 122
32759:  from 129.217.16.0/24 lookup 116
32760:  from 129.217.14.0/24 lookup 114
32761:  from 129.217.12.0/24 lookup 112
32762:  from 129.217.11.64/26 lookup 226
32763:  from 129.217.10.0/24 lookup 110
32764:  from 129.217.2.0/24 lookup 102
32765:  from 129.217.4.0/24 lookup 104
32766:  from all lookup main
32767:  from all lookup default

root@machine:~# ip route show
129.217.5.0/24 dev eth0  proto kernel  scope link  src 129.217.5.61
default via 129.217.5.20 dev eth0

One of the newly created tables looks like this, others are similar:

root@machine:~# ip route show table 122
129.217.22.0/24 dev eth1.122 scope link

Some tables additionally define a default gateway:

root@machine:~# ip route show table 112
129.217.12.0/24 dev eth1.112  scope link
default via 129.217.12.20 dev eth1.112

I need this because I am using several virtual servers
(http://linux-vserver.org) on that machine, all of them sharing the same
kernel (and with  that, the same routing tables), but having different IP
addresses. So, I am using multiple routing tables for setting up a
different routing for every virtual machine.

The DHCP server is running in one of the virtual machines, having its
primary IP on eth1.104, and though I explicitly defined a list of
interfaces DHCP should listen on and eth1.104 is not included in this
list, DHCP sends out unicast replies on eth1.104.

Regards
  Christoph

Re: Policy routing problem

[Christoph Pleger]

Hello,

> Alternatively, you can force your DHCP server to bind on the
> interfaces you want.

I already did that, and even though the primary interface is not in that
list, the DHCP sends out replies on that interface.

>> What am I doing wrong and what must be done to achieve the desired
>> behaviour?
>
> In OUTPUT chain, the outbound interface is already defined. You should
> be marking in PREROUTING.

Going on from my routing configuration I described in my answer to Oleg, I
typed the following:

root@machine:~# iptables -t mangle -A PREROUTING -p udp --source-port
bootps -d 129.217.22.0/24 -j MARK --set-mark 122

root@machine:~# ip rule add fwmark 122 table 122

root@machine:~# ip rule show
0:      from all lookup local
19999:  from all fwmark 0x7a lookup 122
20000:  from 129.217.4.0/24 lookup 104
20000:  from 129.217.2.0/24 lookup 102
20000:  from 129.217.10.0/24 lookup 110
20000:  from 129.217.11.64/26 lookup 226
20000:  from 129.217.12.0/24 lookup 112
20000:  from 129.217.14.0/24 lookup 114
20000:  from 129.217.16.0/24 lookup 116
20000:  from 129.217.22.0/24 lookup 122
20000:  from 129.217.26.0/24 lookup 126
20000:  from 129.217.28.0/24 lookup 128
20000:  from 129.217.33.0/24 lookup 133
20000:  from 129.217.38.0/24 lookup 138
20000:  from 129.217.45.0/25 lookup 145
20000:  from 129.217.47.128/26 lookup 215
20000:  from 129.217.47.192/26 lookup 214
20000:  from 129.217.62.0/24 lookup 76
20000:  from 129.217.63.0/26 lookup 211
32766:  from all lookup main
32767:  from all lookup default

This does not work, DHCP unicast replies to 129.217.22.0/24 are still
being send over eth1.104.

Regards
  Christoph

Re: Policy routing problem

[Oleg]

On Tue, Oct 23, 2012 at 03:23:25PM +0200, Christoph Pleger wrote:
> Hello,
> 
> >   If you have a separate IP for an each interface, everything should be
> > ok, because of device routes.
> >   Show ip rou output for your vlans.
> 
> I explicitly deleted from the main routing table all of what I guess you
> mean with device routes, except the one for the primary network interface,
> and created a new routing table for every device. To be more accurate than
> before, my primary IP address is on eth0 and the VLANs are on eth1.
> Routing tables are currently like follows:

  Are you have real tagged vlan interfaces or by vlan you mean an interface
aliases (where multiple IP exists on one interface)?

> root@machine:~# ip rule show
> 0:      from all lookup local
> 32749:  from 129.217.63.0/26 lookup 211
> 32750:  from 129.217.62.0/24 lookup 76
> 32751:  from 129.217.47.192/26 lookup 214
> 32752:  from 129.217.47.128/26 lookup 215
> 32753:  from 129.217.45.0/25 lookup 145
> 32754:  from 129.217.38.0/24 lookup 138
> 32755:  from 129.217.33.0/24 lookup 133
> 32756:  from 129.217.28.0/24 lookup 128
> 32757:  from 129.217.26.0/24 lookup 126
> 32758:  from 129.217.22.0/24 lookup 122
> 32759:  from 129.217.16.0/24 lookup 116
> 32760:  from 129.217.14.0/24 lookup 114
> 32761:  from 129.217.12.0/24 lookup 112
> 32762:  from 129.217.11.64/26 lookup 226
> 32763:  from 129.217.10.0/24 lookup 110
> 32764:  from 129.217.2.0/24 lookup 102
> 32765:  from 129.217.4.0/24 lookup 104
> 32766:  from all lookup main
> 32767:  from all lookup default
> 
> root@machine:~# ip route show
> 129.217.5.0/24 dev eth0  proto kernel  scope link  src 129.217.5.61
> default via 129.217.5.20 dev eth0
> 
> One of the newly created tables looks like this, others are similar:
> 
> root@machine:~# ip route show table 122
> 129.217.22.0/24 dev eth1.122 scope link
> 
> Some tables additionally define a default gateway:
> 
> root@machine:~# ip route show table 112
> 129.217.12.0/24 dev eth1.112  scope link
> default via 129.217.12.20 dev eth1.112
> 
> I need this because I am using several virtual servers
> (http://linux-vserver.org) on that machine, all of them sharing the same
> kernel (and with  that, the same routing tables), but having different IP
> addresses. So, I am using multiple routing tables for setting up a
> different routing for every virtual machine.
> 
> The DHCP server is running in one of the virtual machines, having its
> primary IP on eth1.104, and though I explicitly defined a list of
> interfaces DHCP should listen on and eth1.104 is not included in this
> list, DHCP sends out unicast replies on eth1.104.
> 
> Regards
>   Christoph
> --
> To unsubscribe from this list: send the line "unsubscribe linux-admin" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Policy routing problem

[Christoph Pleger]

> Are you have real tagged vlan interfaces or by vlan you mean an interface
> aliases (where multiple IP exists on one interface)?

I have real VLAN interfaces, but one of these interfaces has two IPs
(there are two Vservers in that subnet).

Re: Policy routing problem

[Oleg]

On Tue, Oct 23, 2012 at 05:14:15PM +0200, Christoph Pleger wrote:
> > Are you have real tagged vlan interfaces or by vlan you mean an interface
> > aliases (where multiple IP exists on one interface)?
> 
> I have real VLAN interfaces, but one of these interfaces has two IPs
> (there are two Vservers in that subnet).

  For real vlans all should be fine without a special configuration. For
example, i have a dhcp server with many tagged vlans:

~# ip a
....
4: vlan23@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 18:03:73:73:03:14 brd ff:ff:ff:ff:ff:ff
    inet 10.16.23.2/24 scope global vlan23
5: vlan24@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 18:03:73:73:03:14 brd ff:ff:ff:ff:ff:ff
    inet 10.16.24.2/24 scope global vlan24
6: vlan25@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 18:03:73:73:03:14 brd ff:ff:ff:ff:ff:ff
    inet 10.16.25.2/24 scope global vlan25
....
~# ip -d link
....
4: vlan23@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT 
    link/ether 18:03:73:73:03:14 brd ff:ff:ff:ff:ff:ff
    vlan id 102 <REORDER_HDR> 
4: vlan24@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT 
    link/ether 18:03:73:73:03:14 brd ff:ff:ff:ff:ff:ff
    vlan id 103 <REORDER_HDR> 
4: vlan25@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT 
    link/ether 18:03:73:73:03:14 brd ff:ff:ff:ff:ff:ff
    vlan id 104 <REORDER_HDR> 
....
~# ip rou
....
10.16.23.0/24 dev vlan23  proto kernel  scope link  src 10.16.23.2
10.16.24.0/24 dev vlan24  proto kernel  scope link  src 10.16.24.2
10.16.25.0/24 dev vlan25  proto kernel  scope link  src 10.16.25.2
....

Thus, all routing related to these interfaces should work fine. If
you have a multiple IP on a single interface, all routing must be
fine in this case too, because you have device routes like above ones(for 
vlan23, vlan24, vlan25). If you create separate routing tables you must
duplicate to it the same device routes as you have in main for these
interfaces or/and play with src option.

Re: Policy routing problem

[Giles Coochey]

[-- Attachment #1: Type: text/plain, Size: 1168 bytes --]

On 23/10/2012 12:05, Christoph Pleger wrote:
> Hello,
>
> I am running a DHCP-Server that serves multiple subnets. The server has
> an IP address in all of these subnets, and its primary IP address in a
> subnet that is not served by DHCP. Every IP address has its own VLAN
> Ethernet interface, eth0.102, eth0.104, etc. In this setup, the DHCP
> server often does not send its unicast replies on the interface where it
> received the corresponding request, but on the interface of its primary
> IP address, and with that IP. My first thought how to change this was by
> setting routes depending on destination addresses, but this would cause
> big problems with other services running on the same machine, so I tried
> to combine iproute2 and iptables, like this:
>
If there any particular reason why you have the DHCP server on the same 
subnet as your DHCP clients?

I know I'm not answering your questions, but I wouldn't set it up that 
way in the first place on my own environments.

-- 
Regards,

Giles Coochey, CCNA, CCNAS
NetSecSpec Ltd
+44 (0) 7983 877438
http://www.coochey.net
http://www.netsecspec.co.uk
giles@coochey.net



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4968 bytes --]

Re: Policy routing problem

[Christoph Pleger]

Hello,

>If you create separate routing tables you must
>duplicate to it the same device routes as you have in main for these
>interfaces or/and play with src option.

Ok, somehow that works without making too many problems, but I would
really like to find a solution that simulates these device routes only if
the source port is 67.

In the meanwhile, I found out that my solution with for example "ip rule
add fwmark 114 table 114", "ip route add to 129.217.14.0/24 dev eth1.114
table 114" and "iptables -t mangle -A OUTPUT -p udp --source-port bootps
-d 129.217.14.0/24 -j MARK --set-mark 114" works in so far that unicast
DHCP replies are sent from the same device where the request came in, but
still the wrong IP is used. This does not change when I use a specific
source address in "ip route". I tried to set the correct IP address with
SNAT, but it seems that SNAT refuses to use the same port where some
application is listening on. So, if I use "iptables -t nat -A POSTROUTING
... --to-source 129.217.14.81:67", nothing is sent, in contrast to other
ports.

Regards
  Christoph

Re: Policy routing problem

[Christoph Pleger]

Hello,

> If there any particular reason why you have the DHCP server on the same
> subnet as your DHCP clients?

Routers do not forward broadcast messages to other broadcast domains.

Regards
  Christoph

Re: Policy routing problem

[Giles Coochey]

[-- Attachment #1: Type: text/plain, Size: 606 bytes --]

On 25/10/2012 10:25, Christoph Pleger wrote:
> Hello,
>
>> If there any particular reason why you have the DHCP server on the same
>> subnet as your DHCP clients?
> Routers do not forward broadcast messages to other broadcast domains.
>
>
Routers usually have a DHCP helper feature that allows a DHCP server to 
remotely serve IP addresses to subnets they are not actually members of.

http://en.wikipedia.org/wiki/UDP_Helper_Address

-- 
Regards,

Giles Coochey, CCNA, CCNAS
NetSecSpec Ltd
+44 (0) 7983 877438
http://www.coochey.net
http://www.netsecspec.co.uk
giles@coochey.net



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4968 bytes --]

Re: Policy routing problem

[Christoph Pleger]

Hello,

> I tried to set the correct IP address with
> SNAT, but it seems that SNAT refuses to use the same port where some
> application is listening on. So, if I use "iptables -t nat -A POSTROUTING
> ... --to-source 129.217.14.81:67", nothing is sent, in contrast to other
> ports.

For some experiments, I added an iptables-rule, so that the mark is not
only set if the source port is bootps, but also if the destination port is
bootpc. And now, I am completely confused, because I found out that though
there is a relation between the port I use in "iptables -t nat ..." and if
a message is sent, there must also be another reason why messages are not
sent with this kind of NAT. When a DHCP client sends a unicast request to
port bootps, the DHCP unicast reply is never sent when I use port 67 in
SNAT. But when I use netcat sessions in a command line on the DHCP server
to send messages to port bootpc of a client, messages are sent in some
netcat sessions, but in other sessions, they do not, no matter which port
I use in SNAT. Maybe the OS sometimes decides that the port is already in
use and refuses to send messages from that port?

Regards
  Christoph