UDP stream load balancing across multiple uplinks

UDP stream load balancing across multiple uplinks

[Barry A Rich]

I've configured a kubuntu 8.04 PC as router with one LAN interface and three
Internet uplinks. A UDP stream is being produced on the LAN. I'd like the
UPD packets from the stream to be load balanced across the uplinks.
Currently, the UDP stream is tracked as a single connection and all the UPD
packets are sent to the same uplink. What's the best way to accomplish this?

RE: UDP stream load balancing across multiple uplinks

[Barry A Rich]

Bonding is not required in this case. The system that receives the UDP stream does not care if the UDP packets have different source IP addresses. NAT would not be required in this case, but the ISP might drop packets if the source address does not match the uplink address.

Given those assumptions, what's the best way to load balance across the uplinks? If netfilter cannot be configured to do this, can I change ip_conntrack to ignore UDP packets?

> 
> Why would it need to be tracked?  Are you using NAT?  NAT necessarily
> must assign one public address to the outgoing connection as it
> traverses the router.  Therefore, unless all three uplinks can be
> bonded together to have the same IP address, you can not do that.
> 
> On Thu, Jul 24, 2008 at 20:23, Barry A Rich <barich@trisectrix.com> wrote:
> > I've configured a kubuntu 8.04 PC as router with one LAN interface and three
> > Internet uplinks. A UDP stream is being produced on the LAN. I'd like the
> > UPD packets from the stream to be load balanced across the uplinks.
> > Currently, the UDP stream is tracked as a single connection and all the UPD
> > packets are sent to the same uplink. What's the best way to accomplish this?
> >
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe netfilter" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >

Re: UDP stream load balancing across multiple uplinks

[Grant Taylor]

On 07/25/08 09:19, Barry A Rich wrote:
> Bonding is not required in this case. The system that receives the 
> UDP stream does not care if the UDP packets have different source IP 
> addresses. NAT would not be required in this case, but the ISP might 
> drop packets if the source address does not match the uplink address.
> 
> Given those assumptions, what's the best way to load balance across 
> the uplinks? If netfilter cannot be configured to do this, can I 
> change ip_conntrack to ignore UDP packets?

Given that the source IP of your packets can change mid stream, you 
could use the "nth" match extension.  Using "nth" you would match 
packets to decide how to mark them and then use the mark to determine 
what routing table to use which would ultimately decide which outbound 
path to use.



Grant. . . .

RE: UDP stream load balancing across multiple uplinks

[Barry A Rich]

> -----Original Message-----
> From: netfilter-owner@vger.kernel.org
[mailto:netfilter-owner@vger.kernel.org]
> On Behalf Of Grant Taylor
> Sent: Friday, July 25, 2008 11:10 AM
> 
> Given that the source IP of your packets can change mid stream, you
> could use the "nth" match extension.  Using "nth" you would match
> packets to decide how to mark them and then use the mark to determine
> what routing table to use which would ultimately decide which outbound
> path to use.

The nth match looks promising. Given that I want to avoid connection
tracking, I tried the following commands:

iptables -t raw -A PREROUTING -i eth0 -p udp --sport $SPORT -m statistic
--mode nth --every 3 --packet 0 -j SNAT --to-source $UPLINK1IP
iptables -t raw -A PREROUTING -i eth0 -p udp --sport $SPORT -m statistic
--mode nth --every 3 --packet 1 -j SNAT --to-source $UPLINK2IP
iptables -t raw -A PREROUTING -i eth0 -p udp --sport $SPORT -m statistic
--mode nth --every 3 --packet 2 -j SNAT --to-source $UPLINK3IP

Each of these produce the following error:

	iptables: Invalid argument

Any suggestions on what is wrong? Is this the correct way to bypass
connection tracking?

Thanks.

RE: UDP stream load balancing across multiple uplinks

[Barry A Rich]

> -----Original Message-----
> From: netfilter-owner@vger.kernel.org 
> [mailto:netfilter-owner@vger.kernel.org]
> On Behalf Of Grant Taylor
> Sent: Friday, July 25, 2008 11:10 AM
> 
> Given that the source IP of your packets can change mid stream, you 
> could use the "nth" match extension.  Using "nth" you would match 
> packets to decide how to mark them and then use the mark to determine 
> what routing table to use which would ultimately decide which outbound 
> path to use.

The following setup does load balancing across the uplinks. However, the
source address in the outgoing packets is not being SNAT'ed. Any
suggestions?

iptables -t raw -A PREROUTING -i eth0 -p udp --sport $SPORT -j NOTRACK

iptables -t mangle -A PREROUTING -p udp --sport $SPORT -m statistic --mode
nth --every 3 --packet 1 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING -p udp --sport $SPORT -m statistic --mode
nth --every 3 --packet 2 -j MARK --set-mark 2
iptables -t mangle -A PREROUTING -p udp --sport $SPORT -m statistic --mode
nth --every 3 --packet 3 -j MARK --set-mark 3

iptables -t nat -A POSTROUTING -o $UPLINK1IF -j SNAT --to $UPLINK1IP
iptables -t nat -A POSTROUTING -o $UPLINK2IF -j SNAT --to $UPLINK2IP
iptables -t nat -A POSTROUTING -o $UPLINK3IF -j SNAT --to $UPLINK3IP

ip route add $UPLINK1NET dev $UPLINK1IF src $UPLINK1IP table uplink1
ip route add default via $UPLINK1GW table uplink1
ip rule add fwmark 1 table uplink1
ip route add $UPLINK2NET dev $UPLINK2IF src $UPLINK2IP table uplink2
ip route add default via $UPLINK2GW table uplink2
ip rule add fwmark 2 table uplink2
ip route add $UPLINK3NET dev $UPLINK3IF src $UPLINK23P table uplink3
ip route add default via $UPLINK3GW table uplink3
ip rule add fwmark 3 table uplink3

RE: UDP stream load balancing across multiple uplinks

[Покотиленко Костик]

В Пнд, 28/07/2008 в 09:33 -0400, Barry A Rich пишет:
> > -----Original Message-----
> > From: netfilter-owner@vger.kernel.org 
> > [mailto:netfilter-owner@vger.kernel.org]
> > On Behalf Of Grant Taylor
> > Sent: Friday, July 25, 2008 11:10 AM
> > 
> > Given that the source IP of your packets can change mid stream, you 
> > could use the "nth" match extension.  Using "nth" you would match 
> > packets to decide how to mark them and then use the mark to determine 
> > what routing table to use which would ultimately decide which outbound 
> > path to use.
> 
> The following setup does load balancing across the uplinks. However, the
> source address in the outgoing packets is not being SNAT'ed. Any
> suggestions?

Only first packet of udp "connection" is reaching the rule in the nat
table and does the job, since nat occurs for all packets in this
connection rest of rules would not match.

-- 
Покотиленко Костик <casper@meteor.dp.ua>

RE: UDP stream load balancing across multiple uplinks

[Barry A Rich]

> -----Original Message-----
> From: netfilter-owner@vger.kernel.org
[mailto:netfilter-owner@vger.kernel.org]
> On Behalf Of ??????????? ??????
> Sent: Monday, August 04, 2008 3:26 AM
> To: barich@trisectrix.com
> Cc: 'Mail List - Netfilter'
> Subject: RE: UDP stream load balancing across multiple uplinks
> 
> В Пнд, 28/07/2008 в 09:33 -0400, Barry A Rich пишет:
> > > -----Original Message-----
> > > From: netfilter-owner@vger.kernel.org
> > > [mailto:netfilter-owner@vger.kernel.org]
> > > On Behalf Of Grant Taylor
> > > Sent: Friday, July 25, 2008 11:10 AM
> > >
> > > Given that the source IP of your packets can change mid stream, you
> > > could use the "nth" match extension.  Using "nth" you would match
> > > packets to decide how to mark them and then use the mark to determine
> > > what routing table to use which would ultimately decide which outbound
> > > path to use.
> >
> > The following setup does load balancing across the uplinks. However, the
> > source address in the outgoing packets is not being SNAT'ed. Any
> > suggestions?
> 
> Only first packet of udp "connection" is reaching the rule in the nat
> table and does the job, since nat occurs for all packets in this
> connection rest of rules would not match.

I understand better. I removed the following rule:

#iptables -t raw -A PREROUTING -i eth0 -p udp --sport $SPORT -j NOTRACK

The connection is tracked and the outgoing UDP packets are SNAT-ed. However,
the source address for all UDP packets is the IP address of the uplink
assigned to the connection (regardless of the uplink they actually are
sent).

Is there a way to SNAT without connection tracking?

RE: UDP stream load balancing across multiple uplinks

[Покотиленко Костик]

В Пнд, 04/08/2008 в 09:57 -0400, Barry A Rich пишет:
> > -----Original Message-----
> > From: netfilter-owner@vger.kernel.org
> [mailto:netfilter-owner@vger.kernel.org]
> > On Behalf Of ??????????? ??????
> > Sent: Monday, August 04, 2008 3:26 AM
> > To: barich@trisectrix.com
> > Cc: 'Mail List - Netfilter'
> > Subject: RE: UDP stream load balancing across multiple uplinks
> > 
> > В Пнд, 28/07/2008 в 09:33 -0400, Barry A Rich пишет:
> > > > -----Original Message-----
> > > > From: netfilter-owner@vger.kernel.org
> > > > [mailto:netfilter-owner@vger.kernel.org]
> > > > On Behalf Of Grant Taylor
> > > > Sent: Friday, July 25, 2008 11:10 AM
> > > >
> > > > Given that the source IP of your packets can change mid stream, you
> > > > could use the "nth" match extension.  Using "nth" you would match
> > > > packets to decide how to mark them and then use the mark to determine
> > > > what routing table to use which would ultimately decide which outbound
> > > > path to use.
> > >
> > > The following setup does load balancing across the uplinks. However, the
> > > source address in the outgoing packets is not being SNAT'ed. Any
> > > suggestions?
> > 
> > Only first packet of udp "connection" is reaching the rule in the nat
> > table and does the job, since nat occurs for all packets in this
> > connection rest of rules would not match.
> 
> I understand better. I removed the following rule:
> 
> #iptables -t raw -A PREROUTING -i eth0 -p udp --sport $SPORT -j NOTRACK
> 
> The connection is tracked and the outgoing UDP packets are SNAT-ed. However,
> the source address for all UDP packets is the IP address of the uplink
> assigned to the connection (regardless of the uplink they actually are
> sent).
> 
> Is there a way to SNAT without connection tracking?

Sure:

http://linux-ip.net/html/nat-stateless.html

-- 
Покотиленко Костик <casper@meteor.dp.ua>

RE: UDP stream load balancing across multiple uplinks

[Barry A Rich]

> -----Original Message-----
> From: Покотиленко Костик [mailto:casper@meteor.dp.ua]
> Sent: Monday, August 04, 2008 10:41 AM
> To: barich@trisectrix.com
> Cc: 'Mail List - Netfilter'
> Subject: RE: UDP stream load balancing across multiple uplinks
> 
> В Пнд, 04/08/2008 в 09:57 -0400, Barry A Rich пишет:
> > > -----Original Message-----
> > > From: netfilter-owner@vger.kernel.org
> > [mailto:netfilter-owner@vger.kernel.org]
> > > On Behalf Of ??????????? ??????
> > > Sent: Monday, August 04, 2008 3:26 AM
> > > To: barich@trisectrix.com
> > > Cc: 'Mail List - Netfilter'
> > > Subject: RE: UDP stream load balancing across multiple uplinks
> > >
> > > В Пнд, 28/07/2008 в 09:33 -0400, Barry A Rich пишет:
> > > > > -----Original Message-----
> > > > > From: netfilter-owner@vger.kernel.org
> > > > > [mailto:netfilter-owner@vger.kernel.org]
> > > > > On Behalf Of Grant Taylor
> > > > > Sent: Friday, July 25, 2008 11:10 AM
> > > > >
> > > > > Given that the source IP of your packets can change mid stream,
you
> > > > > could use the "nth" match extension.  Using "nth" you would match
> > > > > packets to decide how to mark them and then use the mark to
determine
> > > > > what routing table to use which would ultimately decide which
outbound
> > > > > path to use.
> > > >
> > > > The following setup does load balancing across the uplinks. However,
the
> > > > source address in the outgoing packets is not being SNAT'ed. Any
> > > > suggestions?
> > >
> > > Only first packet of udp "connection" is reaching the rule in the nat
> > > table and does the job, since nat occurs for all packets in this
> > > connection rest of rules would not match.
> >
> > I understand better. I removed the following rule:
> >
> > #iptables -t raw -A PREROUTING -i eth0 -p udp --sport $SPORT -j NOTRACK
> >
> > The connection is tracked and the outgoing UDP packets are SNAT-ed.
However,
> > the source address for all UDP packets is the IP address of the uplink
> > assigned to the connection (regardless of the uplink they actually are
> > sent).
> >
> > Is there a way to SNAT without connection tracking?
> 
> Sure:
> 
> http://linux-ip.net/html/nat-stateless.html

Stateless NAT with IPROUTE2 does not appear to be selective. I only want to
SNAT UDP packets with a specific source port.

RE: UDP stream load balancing across multiple uplinks

[Покотиленко Костик]

В Пнд, 04/08/2008 в 11:48 -0400, Barry A Rich пишет:
> > -----Original Message-----
> > From: Покотиленко Костик [mailto:casper@meteor.dp.ua]
> > Sent: Monday, August 04, 2008 10:41 AM
> > To: barich@trisectrix.com
> > Cc: 'Mail List - Netfilter'
> > Subject: RE: UDP stream load balancing across multiple uplinks
> > 
> > В Пнд, 04/08/2008 в 09:57 -0400, Barry A Rich пишет:
> > > > -----Original Message-----
> > > > From: netfilter-owner@vger.kernel.org
> > > [mailto:netfilter-owner@vger.kernel.org]
> > > > On Behalf Of ??????????? ??????
> > > > Sent: Monday, August 04, 2008 3:26 AM
> > > > To: barich@trisectrix.com
> > > > Cc: 'Mail List - Netfilter'
> > > > Subject: RE: UDP stream load balancing across multiple uplinks
> > > >
> > > > В Пнд, 28/07/2008 в 09:33 -0400, Barry A Rich пишет:
> > > > > > -----Original Message-----
> > > > > > From: netfilter-owner@vger.kernel.org
> > > > > > [mailto:netfilter-owner@vger.kernel.org]
> > > > > > On Behalf Of Grant Taylor
> > > > > > Sent: Friday, July 25, 2008 11:10 AM
> > > > > >
> > > > > > Given that the source IP of your packets can change mid stream,
> you
> > > > > > could use the "nth" match extension.  Using "nth" you would match
> > > > > > packets to decide how to mark them and then use the mark to
> determine
> > > > > > what routing table to use which would ultimately decide which
> outbound
> > > > > > path to use.
> > > > >
> > > > > The following setup does load balancing across the uplinks. However,
> the
> > > > > source address in the outgoing packets is not being SNAT'ed. Any
> > > > > suggestions?
> > > >
> > > > Only first packet of udp "connection" is reaching the rule in the nat
> > > > table and does the job, since nat occurs for all packets in this
> > > > connection rest of rules would not match.
> > >
> > > I understand better. I removed the following rule:
> > >
> > > #iptables -t raw -A PREROUTING -i eth0 -p udp --sport $SPORT -j NOTRACK
> > >
> > > The connection is tracked and the outgoing UDP packets are SNAT-ed.
> However,
> > > the source address for all UDP packets is the IP address of the uplink
> > > assigned to the connection (regardless of the uplink they actually are
> > > sent).
> > >
> > > Is there a way to SNAT without connection tracking?
> > 
> > Sure:
> > 
> > http://linux-ip.net/html/nat-stateless.html
> 
> Stateless NAT with IPROUTE2 does not appear to be selective. I only want to
> SNAT UDP packets with a specific source port.

Maybe it is possible to use FWMARK, not sure...

-- 
Покотиленко Костик <casper@meteor.dp.ua>

RE: UDP stream load balancing across multiple uplinks

[Barry A Rich]

My research came to the same conclusion. Conntrack tracks the UDP stream and puts all related packets to the same interface. The UDP stream shows up in /proc/net/ip_conntrack.

Unfortunately, I cannot test without loading conntrack. The UDP stream producer uses TCP connections to coordinate with the other end. The TCP connections require conntrack.

It appears my only option is to modify conntrack. Any other opinions?



> I'm not an expert in this area, but i believe the per-stream affinity
> to a given interface is because of conntrack.  You could test this
> before expending a great deal of energy by not loading conntrack, and
> seeing it it still sticks to one nic.
> 
> Sofar as dynamically bypassing a loaded module for some traffic (UDP)
> but not all, good luck.  I've never even thought of that before just
> now.
> 
> On Fri, Jul 25, 2008 at 08:22, Barry A Rich <barich@trisectrix.com> wrote:
> > Bonding is not required in this case. The system that receives the UDP
> stream does not care if the UDP packets have different source IP addresses.
> NAT would not be required in this case, but the ISP might drop packets if the
> source address does not match the uplink address.
> >
> > Given those assumptions, what's the best way to load balance across the
> uplinks? If netfilter cannot be configured to do this, can I change
> ip_conntrack to ignore UDP packets?
> >
> >>
> >> Why would it need to be tracked?  Are you using NAT?  NAT necessarily
> >> must assign one public address to the outgoing connection as it
> >> traverses the router.  Therefore, unless all three uplinks can be
> >> bonded together to have the same IP address, you can not do that.
> >>
> >> On Thu, Jul 24, 2008 at 20:23, Barry A Rich <barich@trisectrix.com> wrote:
> >> > I've configured a kubuntu 8.04 PC as router with one LAN interface and
> three
> >> > Internet uplinks. A UDP stream is being produced on the LAN. I'd like the
> >> > UPD packets from the stream to be load balanced across the uplinks.
> >> > Currently, the UDP stream is tracked as a single connection and all the
> UPD
> >> > packets are sent to the same uplink. What's the best way to accomplish
> this?