How to reset everything

How to reset everything

[Barry A Rich]

We use Netfilter to load balance UDP packets across multiple uplinks (ppp0,
ppp1, ppp2, ppp3). Uplinks can be added or removed on the fly. When this
happens, we reset everything and run the firewall/routing script that
matches the new uplink configuration. The reset looks like this:

######################### Begin reset #########################

iptables -F INPUT
iptables -P INPUT DROP
iptables -F OUTPUT
iptables -P OUTPUT DROP
iptables -F FORWARD
iptables -P FORWARD DROP
iptables -F -t raw
iptables -F -t nat
iptables -F -t mangle

ip route del default
ip route flush table uplink1
ip route flush table uplink2
ip route flush table uplink3
ip route flush table uplink4
ip route flush dev ppp0
ip route flush dev ppp1
ip route flush dev ppp2
ip route flush dev ppp3

tc qdisc del dev ppp0 root
tc qdisc del dev ppp1 root
tc qdisc del dev ppp2 root
tc qdisc del dev ppp3 root

ip route flush cache

######################### End reset #########################

For two uplinks, the setup looks like this:

######################### Begin setup #########################

iptables -t raw -A PREROUTING -i eth0 -p udp --sport 6970 -j NOTRACK

iptables -t mangle -A PREROUTING -p udp --sport 6970 -m statistic --mode nth
--every 2 --packet 0 -j MARK --set-mark 1

iptables -t mangle -A PREROUTING -p udp --sport 6970 -m statistic --mode nth
--every 2 --packet 1 -j MARK --set-mark 2

tc qdisc add dev ppp0 root handle 1: prio

tc qdisc add dev ppp1 root handle 1: prio

tc filter add dev ppp0 parent 1:0 protocol ip prio 1 \
    handle 1 fw flowid 1:1 action nat egress x.x.x.x/32 y.y.y.y

tc filter add dev ppp1 parent 1:0 protocol ip prio 1 \
    handle 2 fw flowid 1:1 action nat egress x.x.x.x/32 z.z.z.z

######################### End setup #########################

The UDP stream continues to be received on the LAN interface during the
reset/setup. The reset/setup works most of the time, but occasionally the
packets going out ppp0 do not get NAT'd after a reset/setup. Repeating the
setup/reset sequence a second time seems to make it work, but I'd rather
understand what's wrong and fix it.

All help is appreciated.

Thanks.

Re: How to reset everything

[Покотиленко Костик]

В Срд, 06/05/2009 в 18:03 -0400, Barry A Rich пишет:
> We use Netfilter to load balance UDP packets across multiple uplinks (ppp0,
> ppp1, ppp2, ppp3). Uplinks can be added or removed on the fly. When this
> happens, we reset everything and run the firewall/routing script that
> matches the new uplink configuration. The reset looks like this:
> 
> ######################### Begin reset #########################
> 
> iptables -F INPUT
> iptables -P INPUT DROP
> iptables -F OUTPUT
> iptables -P OUTPUT DROP
> iptables -F FORWARD
> iptables -P FORWARD DROP
> iptables -F -t raw
> iptables -F -t nat
> iptables -F -t mangle
> 
> ip route del default
> ip route flush table uplink1
> ip route flush table uplink2
> ip route flush table uplink3
> ip route flush table uplink4
> ip route flush dev ppp0
> ip route flush dev ppp1
> ip route flush dev ppp2
> ip route flush dev ppp3
> 
> tc qdisc del dev ppp0 root
> tc qdisc del dev ppp1 root
> tc qdisc del dev ppp2 root
> tc qdisc del dev ppp3 root
> 
> ip route flush cache
> 
> ######################### End reset #########################
> 
> For two uplinks, the setup looks like this:
> 
> ######################### Begin setup #########################
> 
> iptables -t raw -A PREROUTING -i eth0 -p udp --sport 6970 -j NOTRACK
> 
> iptables -t mangle -A PREROUTING -p udp --sport 6970 -m statistic --mode nth
> --every 2 --packet 0 -j MARK --set-mark 1
> 
> iptables -t mangle -A PREROUTING -p udp --sport 6970 -m statistic --mode nth
> --every 2 --packet 1 -j MARK --set-mark 2
> 
> tc qdisc add dev ppp0 root handle 1: prio
> 
> tc qdisc add dev ppp1 root handle 1: prio
> 
> tc filter add dev ppp0 parent 1:0 protocol ip prio 1 \
>     handle 1 fw flowid 1:1 action nat egress x.x.x.x/32 y.y.y.y
> 
> tc filter add dev ppp1 parent 1:0 protocol ip prio 1 \
>     handle 2 fw flowid 1:1 action nat egress x.x.x.x/32 z.z.z.z
> 
> ######################### End setup #########################
> 
> The UDP stream continues to be received on the LAN interface during the
> reset/setup. The reset/setup works most of the time, but occasionally the
> packets going out ppp0 do not get NAT'd after a reset/setup. Repeating the
> setup/reset sequence a second time seems to make it work, but I'd rather
> understand what's wrong and fix it.
> 
> All help is appreciated.
> 
> Thanks.

Try adding to your reset script:

conntrack -F conntrack
conntrack -F expect

-- 
Покотиленко Костик <casper@meteor.dp.ua>

RE: How to reset everything

[Barry A Rich]

> -----Original Message-----
> From: Покотиленко Костик [mailto:casper@meteor.dp.ua]
> Sent: Thursday, May 07, 2009 6:04 AM
> To: barich@trisectrix.com
> Cc: netfilter@vger.kernel.org
> Subject: Re: How to reset everything
> 
> В Срд, 06/05/2009 в 18:03 -0400, Barry A Rich пишет:
> > We use Netfilter to load balance UDP packets across multiple uplinks
(ppp0,
> > ppp1, ppp2, ppp3). Uplinks can be added or removed on the fly. When this
> > happens, we reset everything and run the firewall/routing script that
> > matches the new uplink configuration. The reset looks like this:
> >
> > ######################### Begin reset #########################
> >
> > iptables -F INPUT
> > iptables -P INPUT DROP
> > iptables -F OUTPUT
> > iptables -P OUTPUT DROP
> > iptables -F FORWARD
> > iptables -P FORWARD DROP
> > iptables -F -t raw
> > iptables -F -t nat
> > iptables -F -t mangle
> >
> > ip route del default
> > ip route flush table uplink1
> > ip route flush table uplink2
> > ip route flush table uplink3
> > ip route flush table uplink4
> > ip route flush dev ppp0
> > ip route flush dev ppp1
> > ip route flush dev ppp2
> > ip route flush dev ppp3
> >
> > tc qdisc del dev ppp0 root
> > tc qdisc del dev ppp1 root
> > tc qdisc del dev ppp2 root
> > tc qdisc del dev ppp3 root
> >
> > ip route flush cache
> >
> > ######################### End reset #########################
> >
> > For two uplinks, the setup looks like this:
> >
> > ######################### Begin setup #########################
> >
> > iptables -t raw -A PREROUTING -i eth0 -p udp --sport 6970 -j NOTRACK
> >
> > iptables -t mangle -A PREROUTING -p udp --sport 6970 -m statistic --mode
nth
> > --every 2 --packet 0 -j MARK --set-mark 1
> >
> > iptables -t mangle -A PREROUTING -p udp --sport 6970 -m statistic --mode
nth
> > --every 2 --packet 1 -j MARK --set-mark 2
> >
> > tc qdisc add dev ppp0 root handle 1: prio
> >
> > tc qdisc add dev ppp1 root handle 1: prio
> >
> > tc filter add dev ppp0 parent 1:0 protocol ip prio 1 \
> >     handle 1 fw flowid 1:1 action nat egress x.x.x.x/32 y.y.y.y
> >
> > tc filter add dev ppp1 parent 1:0 protocol ip prio 1 \
> >     handle 2 fw flowid 1:1 action nat egress x.x.x.x/32 z.z.z.z
> >
> > ######################### End setup #########################
> >
> > The UDP stream continues to be received on the LAN interface during the
> > reset/setup. The reset/setup works most of the time, but occasionally
the
> > packets going out ppp0 do not get NAT'd after a reset/setup. Repeating
the
> > setup/reset sequence a second time seems to make it work, but I'd rather
> > understand what's wrong and fix it.
> >
> > All help is appreciated.
> >
> > Thanks.
> 
> Try adding to your reset script:
> 
> conntrack -F conntrack
> conntrack -F expect
> 
> --
> Покотиленко Костик <casper@meteor.dp.ua>

That did not solve the problem. Could it be packets that get queued up while
the reset is in progress? Should the uplink queues get flushed? If so, how
is that done?

RE: How to reset everything

[Покотиленко Костик]

В Чтв, 07/05/2009 в 09:49 -0400, Barry A Rich пишет:
> > -----Original Message-----
> > From: Покотиленко Костик [mailto:casper@meteor.dp.ua]
> > Sent: Thursday, May 07, 2009 6:04 AM
> > To: barich@trisectrix.com
> > Cc: netfilter@vger.kernel.org
> > Subject: Re: How to reset everything
> > 
> > В Срд, 06/05/2009 в 18:03 -0400, Barry A Rich пишет:
> > > We use Netfilter to load balance UDP packets across multiple uplinks
> (ppp0,
> > > ppp1, ppp2, ppp3). Uplinks can be added or removed on the fly. When this
> > > happens, we reset everything and run the firewall/routing script that
> > > matches the new uplink configuration. The reset looks like this:
> > >
> > > ######################### Begin reset #########################
> > >
> > > iptables -F INPUT
> > > iptables -P INPUT DROP
> > > iptables -F OUTPUT
> > > iptables -P OUTPUT DROP
> > > iptables -F FORWARD
> > > iptables -P FORWARD DROP
> > > iptables -F -t raw
> > > iptables -F -t nat
> > > iptables -F -t mangle
> > >
> > > ip route del default
> > > ip route flush table uplink1
> > > ip route flush table uplink2
> > > ip route flush table uplink3
> > > ip route flush table uplink4
> > > ip route flush dev ppp0
> > > ip route flush dev ppp1
> > > ip route flush dev ppp2
> > > ip route flush dev ppp3
> > >
> > > tc qdisc del dev ppp0 root
> > > tc qdisc del dev ppp1 root
> > > tc qdisc del dev ppp2 root
> > > tc qdisc del dev ppp3 root
> > >
> > > ip route flush cache
> > >
> > > ######################### End reset #########################
> > >
> > > For two uplinks, the setup looks like this:
> > >
> > > ######################### Begin setup #########################
> > >
> > > iptables -t raw -A PREROUTING -i eth0 -p udp --sport 6970 -j NOTRACK
> > >
> > > iptables -t mangle -A PREROUTING -p udp --sport 6970 -m statistic --mode
> nth
> > > --every 2 --packet 0 -j MARK --set-mark 1
> > >
> > > iptables -t mangle -A PREROUTING -p udp --sport 6970 -m statistic --mode
> nth
> > > --every 2 --packet 1 -j MARK --set-mark 2
> > >
> > > tc qdisc add dev ppp0 root handle 1: prio
> > >
> > > tc qdisc add dev ppp1 root handle 1: prio
> > >
> > > tc filter add dev ppp0 parent 1:0 protocol ip prio 1 \
> > >     handle 1 fw flowid 1:1 action nat egress x.x.x.x/32 y.y.y.y
> > >
> > > tc filter add dev ppp1 parent 1:0 protocol ip prio 1 \
> > >     handle 2 fw flowid 1:1 action nat egress x.x.x.x/32 z.z.z.z
> > >
> > > ######################### End setup #########################
> > >
> > > The UDP stream continues to be received on the LAN interface during the
> > > reset/setup. The reset/setup works most of the time, but occasionally
> the
> > > packets going out ppp0 do not get NAT'd after a reset/setup. Repeating
> the
> > > setup/reset sequence a second time seems to make it work, but I'd rather
> > > understand what's wrong and fix it.
> > >
> > > All help is appreciated.
> > >
> > > Thanks.
> > 
> > Try adding to your reset script:
> > 
> > conntrack -F conntrack
> > conntrack -F expect
> > 
> > --
> > Покотиленко Костик <casper@meteor.dp.ua>
> 
> That did not solve the problem. Could it be packets that get queued up while
> the reset is in progress?

If this is the case the delay shouldn't be long. Also, if this is the
case next packets should behave as you expected and you wouldn't be
reseting second time.

>  Should the uplink queues get flushed? If so, how
> is that done?

tc qdics del && tc qdisc add(???)

-- 
Покотиленко Костик <casper@meteor.dp.ua>

RE: How to reset everything

[Barry A Rich]

> -----Original Message-----
> From: netfilter-owner@vger.kernel.org
[mailto:netfilter-owner@vger.kernel.org]
> On Behalf Of ??????????? ??????
> Sent: Thursday, May 07, 2009 11:52 AM
> To: barich@trisectrix.com
> Cc: netfilter@vger.kernel.org
> Subject: RE: How to reset everything
> 
> В Чтв, 07/05/2009 в 09:49 -0400, Barry A Rich пишет:
> > > -----Original Message-----
> > > From: Покотиленко Костик [mailto:casper@meteor.dp.ua]
> > > Sent: Thursday, May 07, 2009 6:04 AM
> > > To: barich@trisectrix.com
> > > Cc: netfilter@vger.kernel.org
> > > Subject: Re: How to reset everything
> > >
> > > В Срд, 06/05/2009 в 18:03 -0400, Barry A Rich пишет:
> > > > We use Netfilter to load balance UDP packets across multiple uplinks
> > (ppp0,
> > > > ppp1, ppp2, ppp3). Uplinks can be added or removed on the fly. When
this
> > > > happens, we reset everything and run the firewall/routing script
that
> > > > matches the new uplink configuration. The reset looks like this:
> > > >
> > > > ######################### Begin reset #########################
> > > >
> > > > iptables -F INPUT
> > > > iptables -P INPUT DROP
> > > > iptables -F OUTPUT
> > > > iptables -P OUTPUT DROP
> > > > iptables -F FORWARD
> > > > iptables -P FORWARD DROP
> > > > iptables -F -t raw
> > > > iptables -F -t nat
> > > > iptables -F -t mangle
> > > >
> > > > ip route del default
> > > > ip route flush table uplink1
> > > > ip route flush table uplink2
> > > > ip route flush table uplink3
> > > > ip route flush table uplink4
> > > > ip route flush dev ppp0
> > > > ip route flush dev ppp1
> > > > ip route flush dev ppp2
> > > > ip route flush dev ppp3
> > > >
> > > > tc qdisc del dev ppp0 root
> > > > tc qdisc del dev ppp1 root
> > > > tc qdisc del dev ppp2 root
> > > > tc qdisc del dev ppp3 root
> > > >
> > > > ip route flush cache
> > > >
> > > > ######################### End reset #########################
> > > >
> > > > For two uplinks, the setup looks like this:
> > > >
> > > > ######################### Begin setup #########################
> > > >
> > > > iptables -t raw -A PREROUTING -i eth0 -p udp --sport 6970 -j NOTRACK
> > > >
> > > > iptables -t mangle -A PREROUTING -p udp --sport 6970 -m statistic
--mode
> > nth
> > > > --every 2 --packet 0 -j MARK --set-mark 1
> > > >
> > > > iptables -t mangle -A PREROUTING -p udp --sport 6970 -m statistic
--mode
> > nth
> > > > --every 2 --packet 1 -j MARK --set-mark 2
> > > >
> > > > tc qdisc add dev ppp0 root handle 1: prio
> > > >
> > > > tc qdisc add dev ppp1 root handle 1: prio
> > > >
> > > > tc filter add dev ppp0 parent 1:0 protocol ip prio 1 \
> > > >     handle 1 fw flowid 1:1 action nat egress x.x.x.x/32 y.y.y.y
> > > >
> > > > tc filter add dev ppp1 parent 1:0 protocol ip prio 1 \
> > > >     handle 2 fw flowid 1:1 action nat egress x.x.x.x/32 z.z.z.z
> > > >
> > > > ######################### End setup #########################
> > > >
> > > > The UDP stream continues to be received on the LAN interface during
the
> > > > reset/setup. The reset/setup works most of the time, but
occasionally
> > the
> > > > packets going out ppp0 do not get NAT'd after a reset/setup.
Repeating
> > the
> > > > setup/reset sequence a second time seems to make it work, but I'd
rather
> > > > understand what's wrong and fix it.
> > > >
> > > > All help is appreciated.
> > > >
> > > > Thanks.
> > >
> > > Try adding to your reset script:
> > >
> > > conntrack -F conntrack
> > > conntrack -F expect
> > >
> > > --
> > > Покотиленко Костик <casper@meteor.dp.ua>
> >
> > That did not solve the problem. Could it be packets that get queued up
while
> > the reset is in progress?
> 
> If this is the case the delay shouldn't be long. Also, if this is the
> case next packets should behave as you expected and you wouldn't be
> reseting second time.
> 
> >  Should the uplink queues get flushed? If so, how
> > is that done?
> 
> tc qdics del && tc qdisc add(???)
> 
> --
> Покотиленко Костик <casper@meteor.dp.ua>

When this problem occurs after a reset/setup, the UDP packets are still
routed to the both ppp0 and ppp1. They just are not NAT'd on ppp0.

This seems to indicate:

- The statistic module is correctly marking the packets, and
- The packets are being routed correctly, and
- The traffic control filter is not NAT'ing the packets.

Is there a way to log the packets (with the marks) as they exit traffic
control to verify this?