Routing over multiple interfaces

Routing over multiple interfaces

[David Woodhouse]

I have two ADSL lines, and the ISP routes the same set of addresses over
both of them -- so I can saturate my incoming bandwidth over both lines
with a single TCP connection. This is not MLPPP; just two separate PPP
connections.

TCP can cope with a little bit of packet re-ordering, which obviously
happens when packets come over both lines.

Everything works really well for downstream traffic -- but not so well
for upstream, because Linux doesn't use the full available upstream
bandwidth. I have the default route set up thus:

default  src 90.155.92.214 
	nexthop dev ppp0 weight 1
	nexthop dev ppp1 weight 1

But when I do a large upload, I find that the kernel is only ever using
a *single* link at a time, rather than both. How can I make it use
*both* links? It's fine to confine each flow to a single link if it
doesn't saturate that link... but once the queue is full, it should
overflow onto the other device.

I worked around this earlier today by splitting the large file I needed
to upload into two parts, shifting one of them to a *different* machine,
connecting that to the VPN too and then uploading the two parts
separately. But that isn't really a viable option in the general case.

-- 
dwmw2

Re: Routing over multiple interfaces

[David Miller]

From: David Woodhouse <dwmw2@infradead.org>
Date: Mon, 01 Nov 2010 17:12:02 -0400

> But when I do a large upload, I find that the kernel is only ever using
> a *single* link at a time, rather than both. How can I make it use
> *both* links? It's fine to confine each flow to a single link if it
> doesn't saturate that link... but once the queue is full, it should
> overflow onto the other device.

Once a TCP socket gets a routing cache entry, that's what it uses
for the rest of the life of the connection.

The multi-pathing decision happens at the time the routing
cache entry is created.

What you want is multi-path routing support in the routing cache.

We used to have that, but the guy who implemented it (after bugging
me to integrate it for 4 months straight, non-stop) just did a code
dump and then disappeared and fixed none of the serious fundamental
problems which existed in his code.

After a year of no action, I simply tore out all of his code.

More recently Ulrich Weber gave a presentation at netfilter workshop
on some uplink load balancing work he is doing, you might have
a look at his talk and get in contact with him:

http://people.astaro.com/uweber/uplink_balancing.pdf

Re: Routing over multiple interfaces

[Eric Dumazet]

Le lundi 01 novembre 2010 à 14:16 -0700, David Miller a écrit :
> From: David Woodhouse <dwmw2@infradead.org>
> Date: Mon, 01 Nov 2010 17:12:02 -0400
> 
> > But when I do a large upload, I find that the kernel is only ever using
> > a *single* link at a time, rather than both. How can I make it use
> > *both* links? It's fine to confine each flow to a single link if it
> > doesn't saturate that link... but once the queue is full, it should
> > overflow onto the other device.
> 
> Once a TCP socket gets a routing cache entry, that's what it uses
> for the rest of the life of the connection.
> 
> The multi-pathing decision happens at the time the routing
> cache entry is created.
> 
> What you want is multi-path routing support in the routing cache.
> 
> We used to have that, but the guy who implemented it (after bugging
> me to integrate it for 4 months straight, non-stop) just did a code
> dump and then disappeared and fixed none of the serious fundamental
> problems which existed in his code.
> 
> After a year of no action, I simply tore out all of his code.
> 
> More recently Ulrich Weber gave a presentation at netfilter workshop
> on some uplink load balancing work he is doing, you might have
> a look at his talk and get in contact with him:
> 
> http://people.astaro.com/uweber/uplink_balancing.pdf
> --

Astaro case is a bit different, since links have different IP addresses,
and a single upload uses a single link anyway because of hashing or
policy that selects one source IP address.

David W. probably wants to use teql or some bonding ?

# tc qdisc add dev ppp0 root teql0
# tc qdisc add dev ppp1 root teql0
# ip link set dev teql0 up
# ip route add default src 90.155.92.214 dev teql0

Re: Routing over multiple interfaces

[David Woodhouse]

On Mon, 2010-11-01 at 22:35 +0100, Eric Dumazet wrote:
> David W. probably wants to use teql or some bonding ?
> 
> # tc qdisc add dev ppp0 root teql0
> # tc qdisc add dev ppp1 root teql0
> # ip link set dev teql0 up
> # ip route add default src 90.155.92.214 dev teql0 

That looks very likely; thanks. I'll try it... but when I get home next
week, not from the wrong side of the Atlantic ;)

-- 
dwmw2

Re: Routing over multiple interfaces

[Pascal Hambourg]

Hello,

Eric Dumazet a écrit :
> 
> David W. probably wants to use teql or some bonding ?

I doubt that (ethernet) bonding works with PPP links.

> # tc qdisc add dev ppp0 root teql0
> # tc qdisc add dev ppp1 root teql0
> # ip link set dev teql0 up
> # ip route add default src 90.155.92.214 dev teql0

What about using iptables + routing rules ?
Mark every other packet going through the default PPP link with
iptables, and reroute marked packets through the other PPP link.

Re: Routing over multiple interfaces

[Eric Dumazet]

Le mardi 02 novembre 2010 à 20:46 +0100, Pascal Hambourg a écrit :
> Hello,
> 
> Eric Dumazet a écrit :
> > 
> > David W. probably wants to use teql or some bonding ?
> 
> I doubt that (ethernet) bonding works with PPP links.
> 

Well, I was asking if some ppp bonding was possible. I am not a ppp
user.

> > # tc qdisc add dev ppp0 root teql0
> > # tc qdisc add dev ppp1 root teql0
> > # ip link set dev teql0 up
> > # ip route add default src 90.155.92.214 dev teql0
> 
> What about using iptables + routing rules ?
> Mark every other packet going through the default PPP link with
> iptables, and reroute marked packets through the other PPP link.

OK. I provided a working setup, maybe you also could provide one based
on iptables as well ?

Re: Routing over multiple interfaces

[Pascal Hambourg]

Eric Dumazet a écrit :
> Le mardi 02 novembre 2010 à 20:46 +0100, Pascal Hambourg a écrit :
> 
>> What about using iptables + routing rules ?
>> Mark every other packet going through the default PPP link with
>> iptables, and reroute marked packets through the other PPP link.
> 
> OK. I provided a working setup, maybe you also could provide one based
> on iptables as well ?

Arnd Hannemann provided something quite close to what I was thinking
about. I would just make a few adjustments. I added a rule for locally
generated traffic if needed. Also, using the PPP peer as gateway could
be troublesome if both links have the same peer address, so I used the
device instead.

iptables -t mangle -N mark6
iptables -t mangle -A mark6 -m statistic --mode nth --every 2 -j MARK
--set-mark 6

# forwarded traffic, $LANDEV is the interface connected to the LAN
iptables -t mangle -A PREROUTING -i $LANDEV -j mark6
# locally generated traffic to the PPP link
iptables -t mangle -A OUTPUT -o ppp0 -j mark6

ip rule add fwmark 6 table ppp1
ip route replace default dev ppp0
ip route replace default dev ppp1 table ppp1

It still needs some refinements such as excluding non-external
destinations from the PREROUTING rule. Your setup seems much simpler and
efficient.

Re: Routing over multiple interfaces

[David Woodhouse]

On Mon, 2010-11-01 at 22:35 +0100, Eric Dumazet wrote:
> 
> David W. probably wants to use teql or some bonding ?
> 
> # tc qdisc add dev ppp0 root teql0
> # tc qdisc add dev ppp1 root teql0
> # ip link set dev teql0 up
> # ip route add default src 90.155.92.214 dev teql0

That works; thanks. Or it should do... there's a slight complication
right now in that one of my lines is actually routed through a separate
ADSL<->Ethernet box, because one of the ports on my Solos card is a bit
shagged and syncs a lot slower than it should. So it's actually eth1 and
ppp1 that I'm sharing. That should be fixed relatively soon as they're
sending me a new box.

There's a minor issue with this setup though — ideally, any given TCP
connection would tend to use the *same* interface as much as possible,
and only 'overflow' onto the other interface if it's actually saturating
the uplink on the first. That would tend to reduce the amount of packet
re-ordering. At the moment it will split outbound packets over both
lines even when they're relatively idle, introducing packet re-ordering
where it wasn't really necessary.

That said, the world is supposed to cope with packet re-ordering, so I'm
not going to lose a lot of sleep over it. And I have a feeling the
downstream link from the ISP does it exactly the same.

-- 
dwmw2

Re: Routing over multiple interfaces

[Eric Dumazet]

Le mercredi 10 novembre 2010 à 14:50 +0000, David Woodhouse a écrit :
> On Mon, 2010-11-01 at 22:35 +0100, Eric Dumazet wrote:
> > 
> > David W. probably wants to use teql or some bonding ?
> > 
> > # tc qdisc add dev ppp0 root teql0
> > # tc qdisc add dev ppp1 root teql0
> > # ip link set dev teql0 up
> > # ip route add default src 90.155.92.214 dev teql0
> 
> That works; thanks. Or it should do... there's a slight complication
> right now in that one of my lines is actually routed through a separate
> ADSL<->Ethernet box, because one of the ports on my Solos card is a bit
> shagged and syncs a lot slower than it should. So it's actually eth1 and
> ppp1 that I'm sharing. That should be fixed relatively soon as they're
> sending me a new box.
> 
> There's a minor issue with this setup though — ideally, any given TCP
> connection would tend to use the *same* interface as much as possible,
> and only 'overflow' onto the other interface if it's actually saturating
> the uplink on the first. That would tend to reduce the amount of packet
> re-ordering. At the moment it will split outbound packets over both
> lines even when they're relatively idle, introducing packet re-ordering
> where it wasn't really necessary.
> 
> That said, the world is supposed to cope with packet re-ordering, so I'm
> not going to lose a lot of sleep over it. And I have a feeling the
> downstream link from the ISP does it exactly the same.
> 

Its a bit problematic, as you said you want to use both links to do one
upload ;)

Maybe it is possible to instruct teql to try to not change links unless
one link is full...

Re: Routing over multiple interfaces

[David Woodhouse]

On Wed, 2010-11-10 at 16:08 +0100, Eric Dumazet wrote:
> Its a bit problematic, as you said you want to use both links to do
> one upload ;)

When it's *necessary*, yes :)

> Maybe it is possible to instruct teql to try to not change links
> unless one link is full... 

Hm... teql doesn't have any visibility into which packets belong to
which flow, does it?

Perhaps the answer is based on my previous setup ('ip route add default
src 90.155.92.214 nexthop dev ppp0 nexthop dev ppp1'). That would
naturally distribute connections between the interfaces, so all we need
on top is a qdisc which will move packets from one interface to the
other when one queue becomes full (or at a given rate limit).

-- 
dwmw2

Re: Routing over multiple interfaces

[Bandan Das]

On  0, David Miller <davem@davemloft.net> wrote:
> From: David Woodhouse <dwmw2@infradead.org>
> Date: Mon, 01 Nov 2010 17:12:02 -0400
> 
> > But when I do a large upload, I find that the kernel is only ever using
> > a *single* link at a time, rather than both. How can I make it use
> > *both* links? It's fine to confine each flow to a single link if it
> > doesn't saturate that link... but once the queue is full, it should
> > overflow onto the other device.
> 
> Once a TCP socket gets a routing cache entry, that's what it uses
> for the rest of the life of the connection.
> 
> The multi-pathing decision happens at the time the routing
> cache entry is created.
> 
> What you want is multi-path routing support in the routing cache.
> 
> We used to have that, but the guy who implemented it (after bugging
> me to integrate it for 4 months straight, non-stop) just did a code
> dump and then disappeared and fixed none of the serious fundamental
> problems which existed in his code.

You are talking about the equalize patch, right ? 

I had a similar requirement once a long time back and I didn't 
know of any other way to do it, so, I just "forced" this patch 
on a 2.6.23 kernel.
David W, may be you can take a look just to get an idea..

http://th.oughts.org/equalize_2.6_0.3.patch

Bandan

Re: Routing over multiple interfaces

[Arnd Hannemann]

Am 01.11.2010 22:16, schrieb David Miller:
> From: David Woodhouse <dwmw2@infradead.org>
> Date: Mon, 01 Nov 2010 17:12:02 -0400
> 
>> But when I do a large upload, I find that the kernel is only ever using
>> a *single* link at a time, rather than both. How can I make it use
>> *both* links? It's fine to confine each flow to a single link if it
>> doesn't saturate that link... but once the queue is full, it should
>> overflow onto the other device.
> 
> Once a TCP socket gets a routing cache entry, that's what it uses
> for the rest of the life of the connection.
> 
> The multi-pathing decision happens at the time the routing
> cache entry is created.

You should be able to come a solution with netfilter (probably not so efficient):

iptables -t mangle -A PREROUTING -d $EXTERNAL -m statistic --mode nth --every 2 -j MARK --set-mark 6
ip rule add fwmark 6 table ppp1
ip route replace $EXTERNAL via $PPP0GW
ip route replace $EXTERNAL via $PPP1GW table ppp1

Regards,
Arnd

Re: Routing over multiple interfaces

[Patrick Schaaf]

> iptables -t mangle -A PREROUTING -d $EXTERNAL -m statistic --mode nth --every 2 -j MARK --set-mark 6

If statistics match is missing, a pretty good alternative I recently
"found" is using u32 to match for a bit from the IP ID. That is a
stateless decision, and here it probably has the theoretical advantage
of putting all fragments of a given packet onto the same link.

iptables -t mangle -A PREROUTING ... -m u32 --u32 0x2&0x1=0x0 -j MARK
--set-mark 6

best regards
  Patrick

Re: Routing over multiple interfaces

[Eric Dumazet]

Le jeudi 04 novembre 2010 à 13:51 +0100, Patrick Schaaf a écrit :
> > iptables -t mangle -A PREROUTING -d $EXTERNAL -m statistic --mode nth --every 2 -j MARK --set-mark 6
> 
> If statistics match is missing, a pretty good alternative I recently
> "found" is using u32 to match for a bit from the IP ID. That is a
> stateless decision, and here it probably has the theoretical advantage
> of putting all fragments of a given packet onto the same link.
> 
> iptables -t mangle -A PREROUTING ... -m u32 --u32 0x2&0x1=0x0 -j MARK
> --set-mark 6

Sure, thats a good tip/idea, but note many UDP frames have IP.id = 0

Re: Routing over multiple interfaces

[Benjamin LaHaise]

On Mon, Nov 01, 2010 at 05:12:02PM -0400, David Woodhouse wrote:
> I worked around this earlier today by splitting the large file I needed
> to upload into two parts, shifting one of them to a *different* machine,
> connecting that to the VPN too and then uploading the two parts
> separately. But that isn't really a viable option in the general case.

Have a look at eql.

		-ben