Re: How to override the default source ipv4 address on packets originating from a router

[Brian Burch <brian@pingtoo.com>]

On 08/01/14 17:19, Joel Gerber wrote:
> You're going to have to re-acquaint me with your network layout. I'm having a hard time remembering what all of your interfaces are being used for and how they fit into your network.

Oh! Sorry, Joel. I forgot this is a simple mailing list, without a cute 
threaded web interface. I "helpfully" snipped out our initial discussion 
to lighten the payload...

I will resend my last post within the entire reply, then you will have 
everything in one place. Discard this one and the "full" version will 
follow soon.

Thanks,

Brian

> Joel Gerber
> Network Specialist
> Network Operations
> Eastlink
> E: Joel.Gerber@corp.eastlink.ca T: 519.786.1241
>
> -----Original Message-----
> From: lartc-owner@vger.kernel.org [mailto:lartc-owner@vger.kernel.org] On Behalf Of Brian Burch
> Sent: January-08-14 11:18 AM
> To: Linux Advanced Routing List
> Subject: Re: How to override the default source ipv4 address on packets originating from a router
>
> On 21/12/13 12:53, Brian Burch wrote:
>> On 20/12/13 18:46, Brian Burch wrote:
>>> On 20/12/13 13:05, Joel Gerber wrote:
>>>> Your problem is one I've come up against a number of times in the
>>>> past. You are definitely on the right route using "policy routing"
>>>> (otherwise known as utilizing IP Rules), but the snag you're hitting
>>>> is that the source address of a locally generated packet isn't
>>>> chosen until a destination route has already been selected. Using
>>>> the from statement in an ip rules table is only useful when you're
>>>> routing packets from other subnets.
>>>
>>> What a quick and helpful reply! Not only did my problem motivate you
>>> to respond, but I am very relieved to find the answer isn't just RTFM.
>>>
>>> Your explanation about the timing for kernel selection of the source
>>> address is crucial and now my failed experiments all make sense. Your
>>> suggested solution also makes sense, although it is counter-intuitive...
>>> at least to me!
>>>
>>>> Now, just so you know, the local table should normally be left alone.
>>>> It's used for locally connected subnets only. It is given the
>>>> highest priority on purpose... don't mess with it! :)
>>>
>>> I read somewhere (possibly the howto guide or the ip-cref) a
>>> statement that the kernel treats this table as read-only. My system
>>> is running ubuntu 13.04 with a custom non-pae 3.8.0 kernel, so it is
>>> reasonably modern. When I thought more about the purpose of this
>>> table, I decided there would be no point in trying to modify it, even
>>> if that was possible.
>>>
>>>> There are a few methods you could try here, but I'm only going to
>>>> recommend one of them. Have your main routing table hold the routes
>>>> that your local machine will use, and then have a separate table for
>>>> all routed hosts. Then, use ip rule statements to force any traffic
>>>> coming in from a routed host into that routing table.
>>>>
>>>> IE: on firewall-router (assuming eth1 is facing your static subnet
>>>> hosts):
>>>>
>>>> # ip rule list
>>>> 0:  from all lookup local
>>>> 32765:  iif eth1 lookup static-hosts
>>>> 32766:  from all lookup main
>>>> 32767:  from all lookup default
>>>>
>>>> Have table main include all of the routes your local firewall-router
>>>> should use, and have static-hosts contain all of the routes that
>>>> your static-subnet-hosts should use.
>>>
>>> Great! I understand the principle you propose and will try to
>>> implement it over the weekend. I will update this thread when I have
>>> something to report.
>>
>> Excellent news!! My router/firewall is currently collecting 200+
>> updated packages from the repository server!
>>
>> The solution was achieved using ONLY policy routing, based on Joel's
>> suggested approach.
>>
>> However, I hit some fairly nasty snags along the way and had to do a
>> lot of trial and error fiddling to make it work. I am certain my
>> working solution can be made more elegant, but that will take a couple
>> of hours pen-and-pencil time to map out the current routing table
>> entries in selection order and then perform some experiments.
>>
>> I don't have time to do it at the moment, but will get started over
>> the next couple of days. If I get stuck, I will post the "current best"
>> configuration and ask for help to improve it.
>
> <snip>
>
> I tried to improve my "current best" definitions, but my efforts to remove apparent redundancy have all failed. Here is the setup:
>
> brian@chenin:~$ ip address show
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
>       link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>       inet 127.0.0.1/8 scope host lo
>       inet6 ::1/128 scope host
>          valid_lft forever preferred_lft forever
> 2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
>       link/ether 00:15:8a:01:91:e3 brd ff:ff:ff:ff:ff:ff
>       inet 217.154.193.209/28 brd 217.154.193.223 scope global eth1
>       inet6 fe80::215:8aff:fe01:91e3/64 scope link
>          valid_lft forever preferred_lft forever
> 3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
>       link/ether 00:40:63:f5:a5:72 brd ff:ff:ff:ff:ff:ff
>       inet 172.16.101.2/24 brd 172.16.101.255 scope global eth0
>       inet6 fe80::240:63ff:fef5:a572/64 scope link
>          valid_lft forever preferred_lft forever
>
>
> brian@chenin:~$ ip rule list
> 0:	from all lookup local
> 32764:	from 217.154.193.209 lookup CHENIN_TO_INTERNET
> 32765:	from all iif eth1 lookup SUBNET_217
> 32766:	from all lookup main
> 32767:	from all lookup default
>
>
> brian@chenin:~$ ip route list table CHENIN_TO_INTERNET
> 163.1.221.67 via 172.16.101.1 dev eth0
>
>
> brian@chenin:~$ ip route list table SUBNET_217 default via 172.16.101.1 dev eth0
>
>
> brian@chenin:~$ ip route list table main
> 163.1.221.67 dev eth0  scope link  src 217.154.193.209
> 169.254.0.0/16 dev eth0  scope link  metric 1000
> 172.16.101.0/24 dev eth0  proto kernel  scope link  src 172.16.101.2
> 217.154.193.208/28 dev eth1  proto kernel  scope link  src 217.154.193.209
>
>
> I know Joel's suggestion did not include my CHENIN_TO_INTERNET rule, but without it chenin simply times out on its http gets.
>
> I also tried keeping CHENIN_TO_INTERNET, and removing the explicit route in the main table, but that fails too, although I wasn't very surprised.
>
> I have tried trimming these rules and routes to discover the minimum requirement, but this seems to be the smallest working subset.
>
>
> Here is my theory to account for what is happening:
>
> 1. The SUBNET_217 default route MUST be able to handle ANY traffic
> arriving on eth1. When I changed the rule to match on just
> 217.154.193.208/28, the forwarded traffic became very glitchy, with poor
> response times and some dropped tcp sessions.
>
> 2. The route to 163.1.221.67 in the main table is required to assign the
> eth1 source address 217.154.193.209 to its outbound packets. This works
> because the kernel cannot find any other matching routes from chenin to
> 163.1.221.67 (which might encourage it to assign eth0's 172.16.101.2 as
> the source address).
>
> 4. Having assigned the 217.154.193.209 source address, the packet must
> be routed to the next hop. The CHENIN_TO_INTERNET rule matches the
> explicit source host address at a higher priority than the SUBNET_217
> rule, so 172.16.101 is selected.
>
> 5. The main table has a route to 172.16.101.1 via eth0, so the packet is
> sent on its way.
>
>
> That seems to be three passes through the policy routing tables. Am I
> correct?
>
> I still feel it should be possible to improve the solution, but I don't
> understand what is going on well enough to see how. I would be very
> grateful if anyone can enlighten me.
>
> Thanks,
>
> Brian
> --
> To unsubscribe from this list: send the line "unsubscribe lartc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe lartc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>