two gateways with one NIC

two gateways with one NIC

[W Agtail]

Hope you can help.

I have the following setup using LVS (Linux Virtual Servers):

LAN--------------------192.168.0.0/24-----------------  <= CLIENTS
        |                               |
        |                               |
        LVS1                            LVS2
         vip1: 192.168.0.111             vip2: 192.168.0.121
         eth0: 192.168.0.110             eth0: 192.168.0.120
         eth1: 10.18.35.10               eth1: 10.18.35.20
         gw1:  10.18.35.11               gw2:  10.18.35.21
                |                               |
                |                               |
LAN--------------------10.18.35.0/24-----------------
                |                               |
                |                               |
Apache>         WEB1 10.18.35.51:8088           WEB2 10.18.35.52:8088
Apache>         WEB1 10.18.35.51:8089           WEB2 10.18.35.52:8088


### LVS ###
The two LVS servers have a VIP and a GW.
LVS1 & LVS2 have ip_forward set to 1.

LVS1 has the following iptables:
iptables -t nat -A PREROUTING  -i eth0 -j DNAT --to 192.168.0.111
iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to 192.168.0.111
with ipvsadm forwarding vip1:8088 to web1:8088 & web2:8088

LVS2 has the following iptables:
iptables -t nat -A PREROUTING  -i eth0 -j DNAT --to 192.168.0.121
iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to 192.168.0.121
with ipvsadm forwarding vip1:8089 to web1:8089 & web2:8089

### WEB ###
The two Web servers have 2 virtual web servers listening on ports 8088 &
8089 and have the following iptables & iproute2 config:
iptables -t mangle -A PREROUTING -p tcp --dport 8088 -i eth0 -j MARK
--set-mark 1
iptables -t mangle -A PREROUTING -p tcp --dport 8089 -i eth0 -j MARK
--set-mark 2

ip route add table 1 default via 10.18.35.11 dev eth0
ip route add table 2 default via 10.18.35.21 dev eth0

ip rule add fwmark 1 table 1
ip rule add fwmark 2 table 2

WEB1's default GW is set to gw1.
WEB2's default GW is set to gw2.

CLIENTS should be able to connect to vip1:8088 and vip2:8089

### MY PROBLEM ###

If i set WEB2's default GW to gw1, everything works as expected (as I
now only have one GW).
But when trying to set WEB2's default GW to gw2, things don't work.
For example, if i was to run: curl vip1:8088 from a CLIENT, I would be
able to connect to web1:8088 via LVS OK, but unable to connect to
web2:8088 should LVS take me to web2.

Its as though the iptables/ip route settings are not working as they
should.

Any ideas what I'm doing wrong?
Many thanks, W Agtail.

Re: two gateways with one NIC

[Lennart Sorensen]

On Sun, Apr 08, 2007 at 04:35:53AM +0100, W Agtail wrote:
> Hope you can help.
> 
> I have the following setup using LVS (Linux Virtual Servers):
> 
> LAN--------------------192.168.0.0/24-----------------  <= CLIENTS
>         |                               |
>         |                               |
>         LVS1                            LVS2
>          vip1: 192.168.0.111             vip2: 192.168.0.121
>          eth0: 192.168.0.110             eth0: 192.168.0.120
>          eth1: 10.18.35.10               eth1: 10.18.35.20
>          gw1:  10.18.35.11               gw2:  10.18.35.21
>                 |                               |
>                 |                               |
> LAN--------------------10.18.35.0/24-----------------
>                 |                               |
>                 |                               |
> Apache>         WEB1 10.18.35.51:8088           WEB2 10.18.35.52:8088
> Apache>         WEB1 10.18.35.51:8089           WEB2 10.18.35.52:8088
> 
> 
> ### LVS ###
> The two LVS servers have a VIP and a GW.
> LVS1 & LVS2 have ip_forward set to 1.
> 
> LVS1 has the following iptables:
> iptables -t nat -A PREROUTING  -i eth0 -j DNAT --to 192.168.0.111
> iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to 192.168.0.111
> with ipvsadm forwarding vip1:8088 to web1:8088 & web2:8088
> 
> LVS2 has the following iptables:
> iptables -t nat -A PREROUTING  -i eth0 -j DNAT --to 192.168.0.121
> iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to 192.168.0.121
> with ipvsadm forwarding vip1:8089 to web1:8089 & web2:8089
> 
> ### WEB ###
> The two Web servers have 2 virtual web servers listening on ports 8088 &
> 8089 and have the following iptables & iproute2 config:
> iptables -t mangle -A PREROUTING -p tcp --dport 8088 -i eth0 -j MARK
> --set-mark 1
> iptables -t mangle -A PREROUTING -p tcp --dport 8089 -i eth0 -j MARK
> --set-mark 2
> 
> ip route add table 1 default via 10.18.35.11 dev eth0
> ip route add table 2 default via 10.18.35.21 dev eth0
> 
> ip rule add fwmark 1 table 1
> ip rule add fwmark 2 table 2
> 
> WEB1's default GW is set to gw1.
> WEB2's default GW is set to gw2.
> 
> CLIENTS should be able to connect to vip1:8088 and vip2:8089
> 
> ### MY PROBLEM ###
> 
> If i set WEB2's default GW to gw1, everything works as expected (as I
> now only have one GW).
> But when trying to set WEB2's default GW to gw2, things don't work.
> For example, if i was to run: curl vip1:8088 from a CLIENT, I would be
> able to connect to web1:8088 via LVS OK, but unable to connect to
> web2:8088 should LVS take me to web2.
> 
> Its as though the iptables/ip route settings are not working as they
> should.
> 
> Any ideas what I'm doing wrong?
> Many thanks, W Agtail.

Well give I am not sure what you are trying to do, I will take a guess.
I think you are trying to have redundant load balancers and multiple web
servers behind those two load balancers.  Here is how I would do it:

LAN--------------------192.168.0.0/24-----------------  <= CLIENTS
        |                               |
        |                               |
        LVS1                            LVS2
         vrrp: 192.168.0.110 (linked)    vrrp: 192.168.0.110 (linked)
         eth0: 192.168.0.111             eth0: 192.168.0.112

         eth1: 10.18.35.11               eth1: 10.18.35.12
         vrrp: 10.18.35.10 (master)      vrrp: 10.18.35.10 (slave)
                |                               |
                |                               |
LAN--------------------10.18.35.0/24-----------------
                |                               |
                |                               |
Apache>         WEB1 10.18.35.51:8088           WEB2 10.18.35.52:8088
Apache>         WEB1 10.18.35.51:8089           WEB2 10.18.35.52:8088

So using VRRP to have a shared virtual IP between the two load
balancers, any client can connect to 192.168.0.110 and be sent through
to one of the web servers.  The server side interface also has a VRRP
virtual IP shared between the two load balancers, which is linked to the
other virtual IP, so that if the link goes down on one side of the load
balancer, it will automatically drop the virtual IP on both sides to let
the slave machine take over control of the IP.  To the clients this
should be pretty transparent since they don't need to know the IP
changed, other than the momentary change in mac address (letting vrrp
play with the mac address just causes a terrible mess in my experience,
and I have had much better luck by simply changing IPs and letting the
clients relear the new mac).

keepalived's vrrp works very well (Hmm, actually I think I made some
fixes to it, which I don't remember if I sent back upstream yet.  I
should check that tomorrow).

You could run multiple vrrps per interface if you want to somehow have
one be the master of one IP and the other the master of another to allow
different traffic to use each load balancer by default, but everything
going through one in case of a failure.

--
Len Sorensen

Re: two gateways with one NIC

[W Agtail]

Hi, and thanks very much for your response. Your guess sounds spot on. 

As you've mentioned, using one sync group works quite well and gives you
an active/passive LVS cluster (not sure of correct terminology here -
sorry), thus all traffic goes via LVS1, leaving LVS2 not doing much
unless LVS1 fails.

I thought it would be a cool idea to setup two sync groups to ultimately
handle several Apache instances on the two Apache servers. This way,
both LVS servers would be used in a kind of active/active fashion and
would be a master/slave to each other. For example, vip1 & gw1 could
possibly end up on LVS2 with vip2 & gw2.

The challenge though in having two sync groups, with two GWs. I would
like all traffic coming through vip1 to be returned via gw1 and all
traffic coming through vip2 to be returned via gw2.

I am using keepalived (v1.1.13) with two sync groups. One with vip1 &
gw1 and another with vip2 & gw2. Port 8088 will always comes through
vip1/gw1, load balancing to web1:8088 and web2:8088. Port 8089 will
always come through vip2/gw2, load balancing to web1:8089 and web2:8089.

Web1's default gw is set to gw1 and web2's default gw is set to gw2. But
this causing issues when say, vip1:8088 gets forwarded through gw1 to
web2:8088 and doesn't get back back via gw2. To get round this, I need
something like iproute2 on web2 to send all 8088 traffic back through
gw1.

Hope this makes a little more sense to what I'm trying to achieve?
Thanks again.

On Sun, 2007-04-08 at 11:01 -0400, Lennart Sorensen wrote:
> On Sun, Apr 08, 2007 at 04:35:53AM +0100, W Agtail wrote:
> > Hope you can help.
> > 
> > I have the following setup using LVS (Linux Virtual Servers):
> > 
> > LAN--------------------192.168.0.0/24-----------------  <= CLIENTS
> >         |                               |
> >         |                               |
> >         LVS1                            LVS2
> >          vip1: 192.168.0.111             vip2: 192.168.0.121
> >          eth0: 192.168.0.110             eth0: 192.168.0.120
> >          eth1: 10.18.35.10               eth1: 10.18.35.20
> >          gw1:  10.18.35.11               gw2:  10.18.35.21
> >                 |                               |
> >                 |                               |
> > LAN--------------------10.18.35.0/24-----------------
> >                 |                               |
> >                 |                               |
> > Apache>         WEB1 10.18.35.51:8088           WEB2 10.18.35.52:8088
> > Apache>         WEB1 10.18.35.51:8089           WEB2 10.18.35.52:8088
> > 
> > 
> > ### LVS ###
> > The two LVS servers have a VIP and a GW.
> > LVS1 & LVS2 have ip_forward set to 1.
> > 
> > LVS1 has the following iptables:
> > iptables -t nat -A PREROUTING  -i eth0 -j DNAT --to 192.168.0.111
> > iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to 192.168.0.111
> > with ipvsadm forwarding vip1:8088 to web1:8088 & web2:8088
> > 
> > LVS2 has the following iptables:
> > iptables -t nat -A PREROUTING  -i eth0 -j DNAT --to 192.168.0.121
> > iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to 192.168.0.121
> > with ipvsadm forwarding vip1:8089 to web1:8089 & web2:8089
> > 
> > ### WEB ###
> > The two Web servers have 2 virtual web servers listening on ports 8088 &
> > 8089 and have the following iptables & iproute2 config:
> > iptables -t mangle -A PREROUTING -p tcp --dport 8088 -i eth0 -j MARK
> > --set-mark 1
> > iptables -t mangle -A PREROUTING -p tcp --dport 8089 -i eth0 -j MARK
> > --set-mark 2
> > 
> > ip route add table 1 default via 10.18.35.11 dev eth0
> > ip route add table 2 default via 10.18.35.21 dev eth0
> > 
> > ip rule add fwmark 1 table 1
> > ip rule add fwmark 2 table 2
> > 
> > WEB1's default GW is set to gw1.
> > WEB2's default GW is set to gw2.
> > 
> > CLIENTS should be able to connect to vip1:8088 and vip2:8089
> > 
> > ### MY PROBLEM ###
> > 
> > If i set WEB2's default GW to gw1, everything works as expected (as I
> > now only have one GW).
> > But when trying to set WEB2's default GW to gw2, things don't work.
> > For example, if i was to run: curl vip1:8088 from a CLIENT, I would be
> > able to connect to web1:8088 via LVS OK, but unable to connect to
> > web2:8088 should LVS take me to web2.
> > 
> > Its as though the iptables/ip route settings are not working as they
> > should.
> > 
> > Any ideas what I'm doing wrong?
> > Many thanks, W Agtail.
> 
> Well give I am not sure what you are trying to do, I will take a guess.
> I think you are trying to have redundant load balancers and multiple web
> servers behind those two load balancers.  Here is how I would do it:
> 
> LAN--------------------192.168.0.0/24-----------------  <= CLIENTS
>         |                               |
>         |                               |
>         LVS1                            LVS2
>          vrrp: 192.168.0.110 (linked)    vrrp: 192.168.0.110 (linked)
>          eth0: 192.168.0.111             eth0: 192.168.0.112
> 
>          eth1: 10.18.35.11               eth1: 10.18.35.12
>          vrrp: 10.18.35.10 (master)      vrrp: 10.18.35.10 (slave)
>                 |                               |
>                 |                               |
> LAN--------------------10.18.35.0/24-----------------
>                 |                               |
>                 |                               |
> Apache>         WEB1 10.18.35.51:8088           WEB2 10.18.35.52:8088
> Apache>         WEB1 10.18.35.51:8089           WEB2 10.18.35.52:8088
> 
> So using VRRP to have a shared virtual IP between the two load
> balancers, any client can connect to 192.168.0.110 and be sent through
> to one of the web servers.  The server side interface also has a VRRP
> virtual IP shared between the two load balancers, which is linked to the
> other virtual IP, so that if the link goes down on one side of the load
> balancer, it will automatically drop the virtual IP on both sides to let
> the slave machine take over control of the IP.  To the clients this
> should be pretty transparent since they don't need to know the IP
> changed, other than the momentary change in mac address (letting vrrp
> play with the mac address just causes a terrible mess in my experience,
> and I have had much better luck by simply changing IPs and letting the
> clients relear the new mac).
> 
> keepalived's vrrp works very well (Hmm, actually I think I made some
> fixes to it, which I don't remember if I sent back upstream yet.  I
> should check that tomorrow).
> 
> You could run multiple vrrps per interface if you want to somehow have
> one be the master of one IP and the other the master of another to allow
> different traffic to use each load balancer by default, but everything
> going through one in case of a failure.
> 
> --
> Len Sorensen

Re: two gateways with one NIC

[Lennart Sorensen]

On Sun, Apr 08, 2007 at 05:10:15PM +0100, W Agtail wrote:
> Hi, and thanks very much for your response. Your guess sounds spot on. 
> 
> As you've mentioned, using one sync group works quite well and gives you
> an active/passive LVS cluster (not sure of correct terminology here -
> sorry), thus all traffic goes via LVS1, leaving LVS2 not doing much
> unless LVS1 fails.
> 
> I thought it would be a cool idea to setup two sync groups to ultimately
> handle several Apache instances on the two Apache servers. This way,
> both LVS servers would be used in a kind of active/active fashion and
> would be a master/slave to each other. For example, vip1 & gw1 could
> possibly end up on LVS2 with vip2 & gw2.
> 
> The challenge though in having two sync groups, with two GWs. I would
> like all traffic coming through vip1 to be returned via gw1 and all
> traffic coming through vip2 to be returned via gw2.
> 
> I am using keepalived (v1.1.13) with two sync groups. One with vip1 &
> gw1 and another with vip2 & gw2. Port 8088 will always comes through
> vip1/gw1, load balancing to web1:8088 and web2:8088. Port 8089 will
> always come through vip2/gw2, load balancing to web1:8089 and web2:8089.
> 
> Web1's default gw is set to gw1 and web2's default gw is set to gw2. But
> this causing issues when say, vip1:8088 gets forwarded through gw1 to
> web2:8088 and doesn't get back back via gw2. To get round this, I need
> something like iproute2 on web2 to send all 8088 traffic back through
> gw1.

You have to set up both web servers to use the same gateway.  You can
setup an alternate routing table and tag packets from the apache on port
8089 to use the other gateway IP instead, but any traffic handled by
LVS1 _must_ be returned through LVS1.  So both web servers have to have
identical configuration (which is also much simpler to maintain).

You can use iptables to tag packets matching the source port of 8089 and
have ip route route all packets with that specific tag using an
alternate routing table, which will then use the other LVS.

So if you have two VRRP groups, you have port 8088 return by the regular
default gateway going to the first group IP, and you have tagging flag
all port 8089 packets to go through the second vrrp IP.  If an LVS
fails, both vrrp groups end up on the working LVS and everything still
works, but while both works, one LVS handles one port, and the other the
other port.  Of course routing packets is hardly a lot of work, so it
may not really be worth the bother to do anything extra with two groups.
You really have to configure both web servers identically though in
terms of routes.

> Hope this makes a little more sense to what I'm trying to achieve?
> Thanks again.

--
Len Sorensen

Re: two gateways with one NIC

[W Agtail]

Hi, please refer to comments below.

On Sun, 2007-04-08 at 14:22 -0400, Lennart Sorensen wrote:
> On Sun, Apr 08, 2007 at 05:10:15PM +0100, W Agtail wrote:
> > Hi, and thanks very much for your response. Your guess sounds spot on. 
> > 
> > As you've mentioned, using one sync group works quite well and gives you
> > an active/passive LVS cluster (not sure of correct terminology here -
> > sorry), thus all traffic goes via LVS1, leaving LVS2 not doing much
> > unless LVS1 fails.
> > 
> > I thought it would be a cool idea to setup two sync groups to ultimately
> > handle several Apache instances on the two Apache servers. This way,
> > both LVS servers would be used in a kind of active/active fashion and
> > would be a master/slave to each other. For example, vip1 & gw1 could
> > possibly end up on LVS2 with vip2 & gw2.
> > 
> > The challenge though in having two sync groups, with two GWs. I would
> > like all traffic coming through vip1 to be returned via gw1 and all
> > traffic coming through vip2 to be returned via gw2.
> > 
> > I am using keepalived (v1.1.13) with two sync groups. One with vip1 &
> > gw1 and another with vip2 & gw2. Port 8088 will always comes through
> > vip1/gw1, load balancing to web1:8088 and web2:8088. Port 8089 will
> > always come through vip2/gw2, load balancing to web1:8089 and web2:8089.
> > 
> > Web1's default gw is set to gw1 and web2's default gw is set to gw2. But
> > this causing issues when say, vip1:8088 gets forwarded through gw1 to
> > web2:8088 and doesn't get back back via gw2. To get round this, I need
> > something like iproute2 on web2 to send all 8088 traffic back through
> > gw1.
> 
> You have to set up both web servers to use the same gateway.  You can
> setup an alternate routing table and tag packets from the apache on port
> 8089 to use the other gateway IP instead, but any traffic handled by
> LVS1 _must_ be returned through LVS1.  So both web servers have to have
> identical configuration (which is also much simpler to maintain).
> 
> You can use iptables to tag packets matching the source port of 8089 and
> have ip route route all packets with that specific tag using an
> alternate routing table, which will then use the other LVS.
> 
> So if you have two VRRP groups, you have port 8088 return by the regular
> default gateway going to the first group IP, and you have tagging flag
> all port 8089 packets to go through the second vrrp IP.  If an LVS
> fails, both vrrp groups end up on the working LVS and everything still
> works, but while both works, one LVS handles one port, and the other the
> other port.  Of course routing packets is hardly a lot of work, so it
> may not really be worth the bother to do anything extra with two groups.
> You really have to configure both web servers identically though in
> terms of routes.

This is what I'm trying to achieve with the following iptables/iproute2
configuration on both web servers:

iptables -t mangle -A PREROUTING -p tcp --dport 8088 -i eth0 -j LOG
--log-prefix "fwmark 1: "
iptables -t mangle -A PREROUTING -p tcp --dport 8089 -i eth0 -j LOG
--log-prefix "fwmark 2: "

iptables -t mangle -A PREROUTING -p tcp --dport 8088 -i eth0 -j MARK
--set-mark 1
iptables -t mangle -A PREROUTING -p tcp --dport 8089 -i eth0 -j MARK
--set-mark 2

iptables -t mangle -A PREROUTING -m mark --mark 1 -j LOG --log-prefix
"marked 1: "
iptables -t mangle -A PREROUTING -m mark --mark 2 -j LOG --log-prefix
"marked 2: "

ip route add table 1 default via 10.18.35.11 dev eth0 # GW1
ip route add table 2 default via 10.18.35.21 dev eth0 # GW2

ip rule add fwmark 1 table 1
ip rule add fwmark 2 table 2

On web2, the default gw is set to gw2 and in /var/log/messages, I can
see packets appear to be marked. However, for some reason, 8088 is still
routing back via gw2 (default gw) rather than being routed via gw1,
which I'm trying to do with the above ip rules etc.

Is the above the correct syntax? or I guess I could totally be missing
the plot?

Many thanks for your time on this one.

> > Hope this makes a little more sense to what I'm trying to achieve?
> > Thanks again.
> 
> --
> Len Sorensen

Re: two gateways with one NIC

[Lennart Sorensen]

On Sun, Apr 08, 2007 at 08:29:07PM +0100, W Agtail wrote:
> This is what I'm trying to achieve with the following iptables/iproute2
> configuration on both web servers:
> 
> iptables -t mangle -A PREROUTING -p tcp --dport 8088 -i eth0 -j LOG
> --log-prefix "fwmark 1: "
> iptables -t mangle -A PREROUTING -p tcp --dport 8089 -i eth0 -j LOG
> --log-prefix "fwmark 2: "
> 
> iptables -t mangle -A PREROUTING -p tcp --dport 8088 -i eth0 -j MARK
> --set-mark 1
> iptables -t mangle -A PREROUTING -p tcp --dport 8089 -i eth0 -j MARK
> --set-mark 2

You are supposed to mangle things _coming_ from port 8088 and 8089.
After all it is the replies you are trying to affect, not the requests.
So it should be the --sport not --dport.  And of course outbound not
incoming on eth0.

> iptables -t mangle -A PREROUTING -m mark --mark 1 -j LOG --log-prefix
> "marked 1: "
> iptables -t mangle -A PREROUTING -m mark --mark 2 -j LOG --log-prefix
> "marked 2: "
> 
> ip route add table 1 default via 10.18.35.11 dev eth0 # GW1
> ip route add table 2 default via 10.18.35.21 dev eth0 # GW2
> 
> ip rule add fwmark 1 table 1
> ip rule add fwmark 2 table 2
> 
> On web2, the default gw is set to gw2 and in /var/log/messages, I can
> see packets appear to be marked. However, for some reason, 8088 is still
> routing back via gw2 (default gw) rather than being routed via gw1,
> which I'm trying to do with the above ip rules etc.
> 
> Is the above the correct syntax? or I guess I could totally be missing
> the plot?
> 
> Many thanks for your time on this one.

Hope that helps.

--
Len Sorensen

Re: two gateways with one NIC

[W Agtail]

On Mon, 2007-04-09 at 10:54 -0400, Lennart Sorensen wrote:
> On Sun, Apr 08, 2007 at 08:29:07PM +0100, W Agtail wrote:
> > This is what I'm trying to achieve with the following iptables/iproute2
> > configuration on both web servers:
> > 
> > iptables -t mangle -A PREROUTING -p tcp --dport 8088 -i eth0 -j LOG
> > --log-prefix "fwmark 1: "
> > iptables -t mangle -A PREROUTING -p tcp --dport 8089 -i eth0 -j LOG
> > --log-prefix "fwmark 2: "
> > 
> > iptables -t mangle -A PREROUTING -p tcp --dport 8088 -i eth0 -j MARK
> > --set-mark 1
> > iptables -t mangle -A PREROUTING -p tcp --dport 8089 -i eth0 -j MARK
> > --set-mark 2
> 
> You are supposed to mangle things _coming_ from port 8088 and 8089.
> After all it is the replies you are trying to affect, not the requests.
> So it should be the --sport not --dport.  And of course outbound not
> incoming on eth0.
> 
> > iptables -t mangle -A PREROUTING -m mark --mark 1 -j LOG --log-prefix
> > "marked 1: "
> > iptables -t mangle -A PREROUTING -m mark --mark 2 -j LOG --log-prefix
> > "marked 2: "
> > 
> > ip route add table 1 default via 10.18.35.11 dev eth0 # GW1
> > ip route add table 2 default via 10.18.35.21 dev eth0 # GW2
> > 
> > ip rule add fwmark 1 table 1
> > ip rule add fwmark 2 table 2
> > 
> > On web2, the default gw is set to gw2 and in /var/log/messages, I can
> > see packets appear to be marked. However, for some reason, 8088 is still
> > routing back via gw2 (default gw) rather than being routed via gw1,
> > which I'm trying to do with the above ip rules etc.
> > 
> > Is the above the correct syntax? or I guess I could totally be missing
> > the plot?
> > 
> > Many thanks for your time on this one.
> 
> Hope that helps.
> 
> --
> Len Sorensen

Hi there, and thanks v. much for getting back to me on this one.
I now have changed iptables on the web servers to the following:

iptables -t mangle -A POSTROUTING -p tcp --sport 8088 -o eth0 -j MARK
--set-mark 1
iptables -t mangle -A POSTROUTING -p tcp --sport 8089 -o eth0 -j MARK
--set-mark 2

But I'm still seeing traffic being returned via gw2 for port 8088 :(
Any ideas? Thanks.

Re: two gateways with one NIC

[Lennart Sorensen]

On Mon, Apr 09, 2007 at 04:56:20PM +0100, W Agtail wrote:
> Hi there, and thanks v. much for getting back to me on this one.
> I now have changed iptables on the web servers to the following:
> 
> iptables -t mangle -A POSTROUTING -p tcp --sport 8088 -o eth0 -j MARK
> --set-mark 1
> iptables -t mangle -A POSTROUTING -p tcp --sport 8089 -o eth0 -j MARK
> --set-mark 2
> 
> But I'm still seeing traffic being returned via gw2 for port 8088 :(
> Any ideas? Thanks.

Hmm, I know I have done something like this before (I made all port 80
traffic return through another host rather than the default gateway, but
it was about 3 years ago and I don't remember the exact syntax).

Does your logs show both mark 1 and 2 being used?

Are you sure that the syntax for the ip route rules is right?  I don't
remember if it was fwmark or something else.  I also remember I was
using a mark of 30000ish not, 1 or 2.  No idea if low numbers are
reserved or any such thing.

--
Len Sorensen

Re: two gateways with one NIC

[Patrick McHardy]

Lennart Sorensen wrote:
> On Mon, Apr 09, 2007 at 04:56:20PM +0100, W Agtail wrote:
> 
>>Hi there, and thanks v. much for getting back to me on this one.
>>I now have changed iptables on the web servers to the following:
>>
>>iptables -t mangle -A POSTROUTING -p tcp --sport 8088 -o eth0 -j MARK
>>--set-mark 1
>>iptables -t mangle -A POSTROUTING -p tcp --sport 8089 -o eth0 -j MARK
>>--set-mark 2
>>
>>But I'm still seeing traffic being returned via gw2 for port 8088 :(
>>Any ideas? Thanks.
> 
> 
> Hmm, I know I have done something like this before (I made all port 80
> traffic return through another host rather than the default gateway, but
> it was about 3 years ago and I don't remember the exact syntax).


As the name suggests, POSTROUTING comes after routing, so marking
packets there doesn't affect routing. Use PREROUTING for forwarded
traffic and OUTPUT for locally generated traffic.

Re: two gateways with one NIC

[Lennart Sorensen]

On Mon, Apr 09, 2007 at 06:13:50PM +0200, Patrick McHardy wrote:
> As the name suggests, POSTROUTING comes after routing, so marking
> packets there doesn't affect routing. Use PREROUTING for forwarded
> traffic and OUTPUT for locally generated traffic.

I didn't even notice that had been changed.  It used to say PREROUTING
when it was for --dport, and all I suggested changing was --dport to
--sport and change the -o part (probably to nothing at all really since
routing hasn't been decided yet).

Yes it absolutely has to be done PREROUTING.

--
Len Sorensen

Re: two gateways with one NIC

[W Agtail]

On Mon, 2007-04-09 at 12:19 -0400, Lennart Sorensen wrote:
> On Mon, Apr 09, 2007 at 06:13:50PM +0200, Patrick McHardy wrote:
> > As the name suggests, POSTROUTING comes after routing, so marking
> > packets there doesn't affect routing. Use PREROUTING for forwarded
> > traffic and OUTPUT for locally generated traffic.
> 
> I didn't even notice that had been changed.  It used to say PREROUTING
> when it was for --dport, and all I suggested changing was --dport to
> --sport and change the -o part (probably to nothing at all really since
> routing hasn't been decided yet).
> 
> Yes it absolutely has to be done PREROUTING.
> 
> --
> Len Sorensen

Thanks Patrick for your comments too.
It seems that you can't mix PREROUTING with --sport or -o.
I've also changed the ip rule tables to higher numbers, so I now have:

iptables -t mangle -A PREROUTING -p tcp --dport 8088 -i eth0 -j LOG
--log-prefix "fwmark 1: "
iptables -t mangle -A PREROUTING -p tcp --dport 8089 -i eth0 -j LOG
--log-prefix "fwmark 2: "

iptables -t mangle -A PREROUTING -p tcp --dport 8088 -i eth0 -j MARK
--set-mark 1
iptables -t mangle -A PREROUTING -p tcp --dport 8089 -i eth0 -j MARK
--set-mark 2
iptables -t mangle -A PREROUTING -m mark --mark 1 -j LOG --log-prefix
"marked 1: "
iptables -t mangle -A PREROUTING -m mark --mark 2 -j LOG --log-prefix
"marked 2: "

ip route add table 8088 default via 10.18.35.11 dev eth0
ip route add table 8089 default via 10.18.35.21 dev eth0

ip rule add fwmark 1 table 8088
ip rule add fwmark 2 table 8089


# Confirmation of syntax:
iptables -t mangle --list -v -n
Chain PREROUTING (policy ACCEPT 5921 packets, 403K bytes)
 pkts bytes target     prot opt in     out     source
destination
   18   984 LOG        tcp  --  eth0   *       0.0.0.0/0
0.0.0.0/0           tcp dpt:8088 LOG flags 0 level 4 prefix `fwmark 1: '
    0     0 LOG        tcp  --  eth0   *       0.0.0.0/0
0.0.0.0/0           tcp dpt:8089 LOG flags 0 level 4 prefix `fwmark 2: '
   18   984 MARK       tcp  --  eth0   *       0.0.0.0/0
0.0.0.0/0           tcp dpt:8088 MARK set 0x1
    0     0 MARK       tcp  --  eth0   *       0.0.0.0/0
0.0.0.0/0           tcp dpt:8089 MARK set 0x2
   18   984 LOG        all  --  *      *       0.0.0.0/0
0.0.0.0/0           MARK match 0x1 LOG flags 0 level 4 prefix `marked 1:
'
    0     0 LOG        all  --  *      *       0.0.0.0/0
0.0.0.0/0           MARK match 0x2 LOG flags 0 level 4 prefix `marked 2:
'


ip rule list
0:      from all lookup local
32764:  from all fwmark 0x2 lookup 8089
32765:  from all fwmark 0x1 lookup 8088
32766:  from all lookup main
32767:  from all lookup default

ip route list table 8088; ip route list table 8089
default via 10.18.35.11 dev eth0
default via 10.18.35.21 dev eth0

This is what I see in web2's /var/log/messages:
Apr  9 06:46:58 web2-fc6 kernel: fwmark 1: IN=eth0 OUT=
MAC=00:0c:29:d1:08:48:00:0c:29:49:04:9f:08:00 SRC=192.168.0.241
DST=10.18.35.52 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=42359 DF PROTO=TCP
SPT=33321 DPT=8088 WINDOW=5840 RES=0x00 SYN URGP=0

Apr  9 06:46:58 web2-fc6 kernel: marked 1: IN=eth0 OUT=
MAC=00:0c:29:d1:08:48:00:0c:29:49:04:9f:08:00 SRC=192.168.0.241
DST=10.18.35.52 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=42359 DF PROTO=TCP
SPT=33321 DPT=8088 WINDOW=5840 RES=0x00 SYN URGP=0

As you can see, packets appear to be marked.
But here's a tcpdump on gw2's eth1:

07:20:35.004205 192.168.0.241.59438 > 10.18.35.52.8088: S
221760494:221760494(0) win 5840 <mss 1460,sackOK,timestamp 1320423
0,nop,wscale 6> (DF)
07:20:35.013144 10.18.35.52.8088 > 192.168.0.241.59438: S
2705868365:2705868365(0) ack 221760495 win 5792 <mss
1460,sackOK,timestamp 2191014 1320423,nop,wscale 1> (DF)
07:20:35.021857 192.168.0.241.59438 > 10.18.35.52.8088: R
221760495:221760495(0) win 0 (DF)
07:20:38.069688 192.168.0.241.59438 > 10.18.35.52.8088: S
221760494:221760494(0) win 5840 <mss 1460,sackOK,timestamp 1321173
0,nop,wscale 6> (DF)
07:20:38.069695 10.18.35.52.8088 > 192.168.0.241.59438: S
2706988830:2706988830(0) ack 221760495 win 5792 <mss
1460,sackOK,timestamp 2192135 1321173,nop,wscale 1> (DF)
07:20:38.071232 192.168.0.241.59438 > 10.18.35.52.8088: R
221760495:221760495(0) win 0 (DF)

So, traffic is being returned via gw2, rather than gw1 :(

Thanks again for your time.

Re: two gateways with one NIC

[Patrick McHardy]

W Agtail wrote:
> Thanks Patrick for your comments too.
> It seems that you can't mix PREROUTING with --sport or -o.

-o only works after routing.

> I've also changed the ip rule tables to higher numbers, so I now have:
> iptables -t mangle -A PREROUTING -p tcp --dport 8088 -i eth0 -j MARK
> --set-mark 1
> iptables -t mangle -A PREROUTING -p tcp --dport 8089 -i eth0 -j MARK
> --set-mark 2


You should really read some iptables tutorial or at least the manpage.
Its quite easy to set up what you want, but you're again not properly
marking the return traffic.

Re: two gateways with one NIC

[Lennart Sorensen]

On Mon, Apr 09, 2007 at 06:02:23PM +0100, W Agtail wrote:
> Thanks Patrick for your comments too.
> It seems that you can't mix PREROUTING with --sport or -o.
> I've also changed the ip rule tables to higher numbers, so I now have:

I thought you could have --sport, but NOT -o.  No need for -o of course.

> iptables -t mangle -A PREROUTING -p tcp --dport 8088 -i eth0 -j LOG
> --log-prefix "fwmark 1: "
> iptables -t mangle -A PREROUTING -p tcp --dport 8089 -i eth0 -j LOG
> --log-prefix "fwmark 2: "
> 
> iptables -t mangle -A PREROUTING -p tcp --dport 8088 -i eth0 -j MARK
> --set-mark 1
> iptables -t mangle -A PREROUTING -p tcp --dport 8089 -i eth0 -j MARK
> --set-mark 2
> iptables -t mangle -A PREROUTING -m mark --mark 1 -j LOG --log-prefix
> "marked 1: "
> iptables -t mangle -A PREROUTING -m mark --mark 2 -j LOG --log-prefix
> "marked 2: "

The thing is that the destination port will NEVER be 8088 for the
outgoing packets from apache.  The source port will be.

Try this:

iptables -t mangle -A PREROUTING -p tcp --sport 8088 -j LOG --log-prefix "fwmark 1: "
iptables -t mangle -A PREROUTING -p tcp --sport 8089 -j LOG --log-prefix "fwmark 2: "
iptables -t mangle -A PREROUTING -p tcp --sport 8088 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING -p tcp --sport 8089 -j MARK --set-mark 2
iptables -t mangle -A PREROUTING -m mark --mark 1 -j LOG --log-prefix "marked 1: "
iptables -t mangle -A PREROUTING -m mark --mark 2 -j LOG --log-prefix "marked 2: "

> ip route add table 8088 default via 10.18.35.11 dev eth0
> ip route add table 8089 default via 10.18.35.21 dev eth0
> 
> ip rule add fwmark 1 table 8088
> ip rule add fwmark 2 table 8089
> 
> 
> # Confirmation of syntax:
> iptables -t mangle --list -v -n
> Chain PREROUTING (policy ACCEPT 5921 packets, 403K bytes)
>  pkts bytes target     prot opt in     out     source
> destination
>    18   984 LOG        tcp  --  eth0   *       0.0.0.0/0
> 0.0.0.0/0           tcp dpt:8088 LOG flags 0 level 4 prefix `fwmark 1: '
>     0     0 LOG        tcp  --  eth0   *       0.0.0.0/0
> 0.0.0.0/0           tcp dpt:8089 LOG flags 0 level 4 prefix `fwmark 2: '
>    18   984 MARK       tcp  --  eth0   *       0.0.0.0/0
> 0.0.0.0/0           tcp dpt:8088 MARK set 0x1
>     0     0 MARK       tcp  --  eth0   *       0.0.0.0/0
> 0.0.0.0/0           tcp dpt:8089 MARK set 0x2
>    18   984 LOG        all  --  *      *       0.0.0.0/0
> 0.0.0.0/0           MARK match 0x1 LOG flags 0 level 4 prefix `marked 1:
> '
>     0     0 LOG        all  --  *      *       0.0.0.0/0
> 0.0.0.0/0           MARK match 0x2 LOG flags 0 level 4 prefix `marked 2:
> '
> 
> 
> ip rule list
> 0:      from all lookup local
> 32764:  from all fwmark 0x2 lookup 8089
> 32765:  from all fwmark 0x1 lookup 8088
> 32766:  from all lookup main
> 32767:  from all lookup default
> 
> ip route list table 8088; ip route list table 8089
> default via 10.18.35.11 dev eth0
> default via 10.18.35.21 dev eth0
> 
> This is what I see in web2's /var/log/messages:
> Apr  9 06:46:58 web2-fc6 kernel: fwmark 1: IN=eth0 OUT=
> MAC=00:0c:29:d1:08:48:00:0c:29:49:04:9f:08:00 SRC=192.168.0.241
> DST=10.18.35.52 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=42359 DF PROTO=TCP
> SPT=33321 DPT=8088 WINDOW=5840 RES=0x00 SYN URGP=0
> 
> Apr  9 06:46:58 web2-fc6 kernel: marked 1: IN=eth0 OUT=
> MAC=00:0c:29:d1:08:48:00:0c:29:49:04:9f:08:00 SRC=192.168.0.241
> DST=10.18.35.52 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=42359 DF PROTO=TCP
> SPT=33321 DPT=8088 WINDOW=5840 RES=0x00 SYN URGP=0
> 
> As you can see, packets appear to be marked.
> But here's a tcpdump on gw2's eth1:
> 
> 07:20:35.004205 192.168.0.241.59438 > 10.18.35.52.8088: S
> 221760494:221760494(0) win 5840 <mss 1460,sackOK,timestamp 1320423
> 0,nop,wscale 6> (DF)
> 07:20:35.013144 10.18.35.52.8088 > 192.168.0.241.59438: S
> 2705868365:2705868365(0) ack 221760495 win 5792 <mss
> 1460,sackOK,timestamp 2191014 1320423,nop,wscale 1> (DF)
> 07:20:35.021857 192.168.0.241.59438 > 10.18.35.52.8088: R
> 221760495:221760495(0) win 0 (DF)
> 07:20:38.069688 192.168.0.241.59438 > 10.18.35.52.8088: S
> 221760494:221760494(0) win 5840 <mss 1460,sackOK,timestamp 1321173
> 0,nop,wscale 6> (DF)
> 07:20:38.069695 10.18.35.52.8088 > 192.168.0.241.59438: S
> 2706988830:2706988830(0) ack 221760495 win 5792 <mss
> 1460,sackOK,timestamp 2192135 1321173,nop,wscale 1> (DF)
> 07:20:38.071232 192.168.0.241.59438 > 10.18.35.52.8088: R
> 221760495:221760495(0) win 0 (DF)
> 
> So, traffic is being returned via gw2, rather than gw1 :(

They are marked I guess, but much too late.

--
Len Sorensen

Re: two gateways with one NIC

[W Agtail]

Nice one, but unfortunately still doesn't work.
I'm now not seeing any marked messages in /var/log/messages and traffic
still going via gw2 for port 8088.

Thanks again. 

On Mon, 2007-04-09 at 13:23 -0400, Lennart Sorensen wrote:
> On Mon, Apr 09, 2007 at 06:02:23PM +0100, W Agtail wrote:
> > Thanks Patrick for your comments too.
> > It seems that you can't mix PREROUTING with --sport or -o.
> > I've also changed the ip rule tables to higher numbers, so I now have:
> 
> I thought you could have --sport, but NOT -o.  No need for -o of course.
> 
> > iptables -t mangle -A PREROUTING -p tcp --dport 8088 -i eth0 -j LOG
> > --log-prefix "fwmark 1: "
> > iptables -t mangle -A PREROUTING -p tcp --dport 8089 -i eth0 -j LOG
> > --log-prefix "fwmark 2: "
> > 
> > iptables -t mangle -A PREROUTING -p tcp --dport 8088 -i eth0 -j MARK
> > --set-mark 1
> > iptables -t mangle -A PREROUTING -p tcp --dport 8089 -i eth0 -j MARK
> > --set-mark 2
> > iptables -t mangle -A PREROUTING -m mark --mark 1 -j LOG --log-prefix
> > "marked 1: "
> > iptables -t mangle -A PREROUTING -m mark --mark 2 -j LOG --log-prefix
> > "marked 2: "
> 
> The thing is that the destination port will NEVER be 8088 for the
> outgoing packets from apache.  The source port will be.
> 
> Try this:
> 
> iptables -t mangle -A PREROUTING -p tcp --sport 8088 -j LOG --log-prefix "fwmark 1: "
> iptables -t mangle -A PREROUTING -p tcp --sport 8089 -j LOG --log-prefix "fwmark 2: "
> iptables -t mangle -A PREROUTING -p tcp --sport 8088 -j MARK --set-mark 1
> iptables -t mangle -A PREROUTING -p tcp --sport 8089 -j MARK --set-mark 2
> iptables -t mangle -A PREROUTING -m mark --mark 1 -j LOG --log-prefix "marked 1: "
> iptables -t mangle -A PREROUTING -m mark --mark 2 -j LOG --log-prefix "marked 2: "
> 
> > ip route add table 8088 default via 10.18.35.11 dev eth0
> > ip route add table 8089 default via 10.18.35.21 dev eth0
> > 
> > ip rule add fwmark 1 table 8088
> > ip rule add fwmark 2 table 8089
> > 
> > 
> > # Confirmation of syntax:
> > iptables -t mangle --list -v -n
> > Chain PREROUTING (policy ACCEPT 5921 packets, 403K bytes)
> >  pkts bytes target     prot opt in     out     source
> > destination
> >    18   984 LOG        tcp  --  eth0   *       0.0.0.0/0
> > 0.0.0.0/0           tcp dpt:8088 LOG flags 0 level 4 prefix `fwmark 1: '
> >     0     0 LOG        tcp  --  eth0   *       0.0.0.0/0
> > 0.0.0.0/0           tcp dpt:8089 LOG flags 0 level 4 prefix `fwmark 2: '
> >    18   984 MARK       tcp  --  eth0   *       0.0.0.0/0
> > 0.0.0.0/0           tcp dpt:8088 MARK set 0x1
> >     0     0 MARK       tcp  --  eth0   *       0.0.0.0/0
> > 0.0.0.0/0           tcp dpt:8089 MARK set 0x2
> >    18   984 LOG        all  --  *      *       0.0.0.0/0
> > 0.0.0.0/0           MARK match 0x1 LOG flags 0 level 4 prefix `marked 1:
> > '
> >     0     0 LOG        all  --  *      *       0.0.0.0/0
> > 0.0.0.0/0           MARK match 0x2 LOG flags 0 level 4 prefix `marked 2:
> > '
> > 
> > 
> > ip rule list
> > 0:      from all lookup local
> > 32764:  from all fwmark 0x2 lookup 8089
> > 32765:  from all fwmark 0x1 lookup 8088
> > 32766:  from all lookup main
> > 32767:  from all lookup default
> > 
> > ip route list table 8088; ip route list table 8089
> > default via 10.18.35.11 dev eth0
> > default via 10.18.35.21 dev eth0
> > 
> > This is what I see in web2's /var/log/messages:
> > Apr  9 06:46:58 web2-fc6 kernel: fwmark 1: IN=eth0 OUT=
> > MAC=00:0c:29:d1:08:48:00:0c:29:49:04:9f:08:00 SRC=192.168.0.241
> > DST=10.18.35.52 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=42359 DF PROTO=TCP
> > SPT=33321 DPT=8088 WINDOW=5840 RES=0x00 SYN URGP=0
> > 
> > Apr  9 06:46:58 web2-fc6 kernel: marked 1: IN=eth0 OUT=
> > MAC=00:0c:29:d1:08:48:00:0c:29:49:04:9f:08:00 SRC=192.168.0.241
> > DST=10.18.35.52 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=42359 DF PROTO=TCP
> > SPT=33321 DPT=8088 WINDOW=5840 RES=0x00 SYN URGP=0
> > 
> > As you can see, packets appear to be marked.
> > But here's a tcpdump on gw2's eth1:
> > 
> > 07:20:35.004205 192.168.0.241.59438 > 10.18.35.52.8088: S
> > 221760494:221760494(0) win 5840 <mss 1460,sackOK,timestamp 1320423
> > 0,nop,wscale 6> (DF)
> > 07:20:35.013144 10.18.35.52.8088 > 192.168.0.241.59438: S
> > 2705868365:2705868365(0) ack 221760495 win 5792 <mss
> > 1460,sackOK,timestamp 2191014 1320423,nop,wscale 1> (DF)
> > 07:20:35.021857 192.168.0.241.59438 > 10.18.35.52.8088: R
> > 221760495:221760495(0) win 0 (DF)
> > 07:20:38.069688 192.168.0.241.59438 > 10.18.35.52.8088: S
> > 221760494:221760494(0) win 5840 <mss 1460,sackOK,timestamp 1321173
> > 0,nop,wscale 6> (DF)
> > 07:20:38.069695 10.18.35.52.8088 > 192.168.0.241.59438: S
> > 2706988830:2706988830(0) ack 221760495 win 5792 <mss
> > 1460,sackOK,timestamp 2192135 1321173,nop,wscale 1> (DF)
> > 07:20:38.071232 192.168.0.241.59438 > 10.18.35.52.8088: R
> > 221760495:221760495(0) win 0 (DF)
> > 
> > So, traffic is being returned via gw2, rather than gw1 :(
> 
> They are marked I guess, but much too late.
> 
> --
> Len Sorensen

Re: two gateways with one NIC

[Ben Greear]

W Agtail wrote:
> Nice one, but unfortunately still doesn't work.
> I'm now not seeing any marked messages in /var/log/messages and traffic
> still going via gw2 for port 8088.

Maybe you could use something like my mac-vlan virtual device to make
your single NIC look like two NICs?  You can find links to the patch and
the macvlan-config tool on this page:

http://www.candelatech.com/~greear/vlan.html

Ben


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: two gateways with one NIC

[Lennart Sorensen]

On Mon, Apr 09, 2007 at 07:05:31PM +0100, W Agtail wrote:
> Nice one, but unfortunately still doesn't work.
> I'm now not seeing any marked messages in /var/log/messages and traffic
> still going via gw2 for port 8088.

What does 'iptables -v -t mangle -L' show at the moment?  Have you been
flushing it between attemps to make sure you don't have conflicting
rules?

--
Len Sorensen

Re: two gateways with one NIC

[W Agtail]

On Mon, 2007-04-09 at 14:11 -0400, Lennart Sorensen wrote:
> On Mon, Apr 09, 2007 at 07:05:31PM +0100, W Agtail wrote:
> > Nice one, but unfortunately still doesn't work.
> > I'm now not seeing any marked messages in /var/log/messages and traffic
> > still going via gw2 for port 8088.
> 
> What does 'iptables -v -t mangle -L' show at the moment?  Have you been
> flushing it between attemps to make sure you don't have conflicting
> rules?
> 
> --
> Len Sorensen

Yup, I've been flushing iptables each time. This is what we have atm:
iptables -n -v -t mangle -L

Chain PREROUTING (policy ACCEPT 12656 packets, 2518K bytes)
 pkts bytes target     prot opt in     out     source
destination
    0     0 LOG        tcp  --  *      *       0.0.0.0/0
0.0.0.0/0           tcp spt:8088 LOG flags 0 level 4 prefix `fwmark 1: '
    0     0 LOG        tcp  --  *      *       0.0.0.0/0
0.0.0.0/0           tcp spt:8089 LOG flags 0 level 4 prefix `fwmark 2: '
    0     0 MARK       tcp  --  *      *       0.0.0.0/0
0.0.0.0/0           tcp spt:8088 MARK set 0x1
    0     0 MARK       tcp  --  *      *       0.0.0.0/0
0.0.0.0/0           tcp spt:8089 MARK set 0x2
    0     0 LOG        all  --  *      *       0.0.0.0/0
0.0.0.0/0           MARK match 0x1 LOG flags 0 level 4 prefix `marked 1:
'
    0     0 LOG        all  --  *      *       0.0.0.0/0
0.0.0.0/0           MARK match 0x2 LOG flags 0 level 4 prefix `marked 2:
'

Chain INPUT (policy ACCEPT 10664 packets, 2438K bytes)
 pkts bytes target     prot opt in     out     source
destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source
destination

Chain OUTPUT (policy ACCEPT 6311 packets, 896K bytes)
 pkts bytes target     prot opt in     out     source
destination

Chain POSTROUTING (policy ACCEPT 6311 packets, 896K bytes)
 pkts bytes target     prot opt in     out     source
destination

Re: two gateways with one NIC

[Lennart Sorensen]

On Mon, Apr 09, 2007 at 07:24:07PM +0100, W Agtail wrote:
> Yup, I've been flushing iptables each time. This is what we have atm:
> iptables -n -v -t mangle -L
> 
> Chain PREROUTING (policy ACCEPT 12656 packets, 2518K bytes)
>  pkts bytes target     prot opt in     out     source
> destination
>     0     0 LOG        tcp  --  *      *       0.0.0.0/0
> 0.0.0.0/0           tcp spt:8088 LOG flags 0 level 4 prefix `fwmark 1: '
>     0     0 LOG        tcp  --  *      *       0.0.0.0/0
> 0.0.0.0/0           tcp spt:8089 LOG flags 0 level 4 prefix `fwmark 2: '
>     0     0 MARK       tcp  --  *      *       0.0.0.0/0
> 0.0.0.0/0           tcp spt:8088 MARK set 0x1
>     0     0 MARK       tcp  --  *      *       0.0.0.0/0
> 0.0.0.0/0           tcp spt:8089 MARK set 0x2
>     0     0 LOG        all  --  *      *       0.0.0.0/0
> 0.0.0.0/0           MARK match 0x1 LOG flags 0 level 4 prefix `marked 1:
> '
>     0     0 LOG        all  --  *      *       0.0.0.0/0
> 0.0.0.0/0           MARK match 0x2 LOG flags 0 level 4 prefix `marked 2:
> '
> 
> Chain INPUT (policy ACCEPT 10664 packets, 2438K bytes)
>  pkts bytes target     prot opt in     out     source
> destination
> 
> Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
>  pkts bytes target     prot opt in     out     source
> destination
> 
> Chain OUTPUT (policy ACCEPT 6311 packets, 896K bytes)
>  pkts bytes target     prot opt in     out     source
> destination
> 
> Chain POSTROUTING (policy ACCEPT 6311 packets, 896K bytes)
>  pkts bytes target     prot opt in     out     source
> destination

Odd how the packet count on those mangle table entries is 0.  It seems
like it is never even getting to there.  Do you need a rule in the
output chain telling it to send some packets to the mangle table?  That
doesn't make sense either though.

--
Len Sorensen

Re: two gateways with one NIC

[W Agtail]

On Mon, 2007-04-09 at 11:11 -0700, Ben Greear wrote:
> W Agtail wrote:
> > Nice one, but unfortunately still doesn't work.
> > I'm now not seeing any marked messages in /var/log/messages and traffic
> > still going via gw2 for port 8088.
> 
> Maybe you could use something like my mac-vlan virtual device to make
> your single NIC look like two NICs?  You can find links to the patch and
> the macvlan-config tool on this page:
> 
> http://www.candelatech.com/~greear/vlan.html
> 
> Ben
> 
> 

Thanks Ben, this looks quite an interesting idea.
Is it possible to create /etc/sysconfig/network-scripts/* in the same
way as ethN:N scripts I wonder?

Re: two gateways with one NIC

[Ben Greear]

W Agtail wrote:
> On Mon, 2007-04-09 at 11:11 -0700, Ben Greear wrote:
>   
>> W Agtail wrote:
>>     
>>> Nice one, but unfortunately still doesn't work.
>>> I'm now not seeing any marked messages in /var/log/messages and traffic
>>> still going via gw2 for port 8088.
>>>       
>> Maybe you could use something like my mac-vlan virtual device to make
>> your single NIC look like two NICs?  You can find links to the patch and
>> the macvlan-config tool on this page:
>>
>> http://www.candelatech.com/~greear/vlan.html
>>
>> Ben
>>
>>
>>     
>
> Thanks Ben, this looks quite an interesting idea.
> Is it possible to create /etc/sysconfig/network-scripts/* in the same
> way as ethN:N scripts I wonder?
>   
No idea...I create these things using mvl_config tool.  At the least, 
you could
edit rc.local or similar.

Thanks,
Ben

> -
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>   


-- 
Ben Greear <greearb@candelatech.com> 
Candela Technologies Inc  http://www.candelatech.com

Re: two gateways with one NIC

[W Agtail]

Well, I found that with CentOS/Fedora/RHEL, I could use their standard
network-scripts to create VLAN devices.

I got VLAN devices running OK, but then ended up in the same boat as
before.

Also, it might be nice if keepalived/LVS had the option of entering a
VLAN device in keepalived.conf? (might come handy for someone one day?)
I just did a bit of a fudge while testing.

My only option, seems only to create a single sided 'active/passive'
KeepAliveD/LVS cluster.

Thanks all for your help :)

On Tue, 2007-04-10 at 08:32 -0700, Ben Greear wrote:
> W Agtail wrote:
> > On Mon, 2007-04-09 at 11:11 -0700, Ben Greear wrote:
> >   
> >> W Agtail wrote:
> >>     
> >>> Nice one, but unfortunately still doesn't work.
> >>> I'm now not seeing any marked messages in /var/log/messages and traffic
> >>> still going via gw2 for port 8088.
> >>>       
> >> Maybe you could use something like my mac-vlan virtual device to make
> >> your single NIC look like two NICs?  You can find links to the patch and
> >> the macvlan-config tool on this page:
> >>
> >> http://www.candelatech.com/~greear/vlan.html
> >>
> >> Ben
> >>
> >>
> >>     
> >
> > Thanks Ben, this looks quite an interesting idea.
> > Is it possible to create /etc/sysconfig/network-scripts/* in the same
> > way as ethN:N scripts I wonder?
> >   
> No idea...I create these things using mvl_config tool.  At the least, 
> you could
> edit rc.local or similar.
> 
> Thanks,
> Ben
> 
> > -
> > To unsubscribe from this list: send the line "unsubscribe netdev" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >   
> 
>