moving ipvs() to POST/PREROUTING

moving ipvs() to POST/PREROUTING

[Jason Stubbs]

[-- Attachment #1: Type: text/plain, Size: 3453 bytes --]

Greetings,

Ok, things are mostly working now. The patch is a little messy as in there's 
old comments remaining and function names are left as is, but hopefully 
reviewable. If it's not, I'll split it up and add appropriate comments...

Functions I've tested with the patch:
* LVS-NAT
* LVS-NAT + SNAT
* LVS-DR
* local node

With local node, 127.0.0.1 doesn't work but an IP address on a local interface 
does. When the address is 127.0.0.1, the SYN makes it all the way through 
INPUT, but the SYN/ACK doesn't come into OUTPUT. Something to investigate 
further... Also, null_xmit doesn't work as ipvs_in is being done in 
POSTROUTING, so I've simple aliased LOCAL to MASQ for the time being.

What I haven't tested:
* LVS-TUN
* ICMP for LVS-NAT
* IP_VS_CONN_F_BYPASS - what is this?
* likely others ;)

LVS-TUN should work as LVS-DR didn't require any direct modification, but it's 
a little bit of a pain to set up for testing at this stage. I'm not sure how 
I should go about testing the others.

I realized I haven't explained at all why I chose POST/PRE as the hook points. 
Firstly the cropped output from a LOG target in every mangle table for the 
SYN SYN/ACK of a LVS-NAT connection:

PREROUTING IN=eth0 OUT= SRC=192.168.0.104 DST=192.168.0.SYN
FORWARD IN=eth0 OUT=eth0 SRC=192.168.0.104 DST=192.168.0.7 SYN
POSTROUTING IN= OUT=eth0 SRC=192.168.0.104 DST=192.168.0.7 SYN
POSTROUTING IN= OUT=eth1 SRC=192.168.0.104 DST=192.168.1.3 SYN

PREROUTING IN=eth1 OUT= SRC=192.168.0.7 DST=192.168.0.104 ACK SYN
FORWARD IN=eth1 OUT=eth0 SRC=192.168.0.7 DST=192.168.0.104 ACK SYN
POSTROUTING IN= OUT=eth0 SRC=192.168.0.7 DST=192.168.0.104 ACK SYN

192.168.0.104 is the client, 192.168.0.7 is the VIP and 192.168.1.3 is the 
real server. Other than the second POSTROUTING entry on the SYN side, 
netfilter isn't dealing with the real server's IP at all. This will 
theoretically make writing firewall rules much easier and also limits what 
netfilter's conntracking has to deal with.

Actually, I don't know why the second POSTROUTING entry is there at all. It 
seems that after the packet is injected into the end of POSTROUTING, a 
routing decision is being made again and POSTROUTING is rerun. Preferable the 
packet would go straight out the appropriate interface after ipvs_in is run.

Similar behaviour happens with a local node:

PREROUTING IN=eth0 OUT= SRC=192.168.0.104 DST=192.168.0.7 SYN
FORWARD IN=eth0 OUT=eth0 SRC=192.168.0.104 DST=192.168.0.7 SYN
POSTROUTING IN= OUT=eth0 SRC=192.168.0.104 DST=192.168.0.7 SYN
POSTROUTING IN= OUT=lo SRC=192.168.0.104 DST=192.168.0.5 SYN
PREROUTING IN=lo OUT= SRC=192.168.0.104 DST=192.168.0.5 SYN
INPUT IN=lo OUT= SRC=192.168.0.104 DST=192.168.0.5 SYN

OUTPUT IN= OUT=eth0 SRC=192.168.0.7 DST=192.168.0.104 ACK SYN
POSTROUTING IN= OUT=eth0 SRC=192.168.0.7 DST=192.168.0.104 ACK SYN

192.168.0.5 is an IP local to the director. I had to add the ipvs_out hooks to 
the beginning of OUTPUT as the local reply never hits PREROUTING. Again with 
the above, I'd prefer the POST/PRE/INPUT disappear.

Anyway, that's pretty much my intention. Is there any problem with essentially 
hiding the real servers from netfilter? Is there a way to get the packet out 
of the netfilter loop earlier?

-- 
Jason Stubbs <j.stubbs@linkthink.co.jp>
LINKTHINK INC.
東京都渋谷区桜ヶ丘町22-14 N.E.S S棟 3F
TEL 03-5728-4772  FAX 03-5728-4773

[-- Attachment #2: ip_vs_prepost.patch --]
[-- Type: text/x-diff, Size: 6080 bytes --]

--- linux-2.6.24-gentoo-r4/net/ipv4/ipvs/ip_vs_core.c	2008-01-25 07:58:37.000000000 +0900
+++ linux-2.6.24-gentoo-r4-jason/net/ipv4/ipvs/ip_vs_core.c	2008-04-11 12:56:38.000000000 +0900
@@ -480,25 +480,6 @@
 }
 
 
-/*
- *      It is hooked before NF_IP_PRI_NAT_SRC at the NF_IP_POST_ROUTING
- *      chain, and is used for VS/NAT.
- *      It detects packets for VS/NAT connections and sends the packets
- *      immediately. This can avoid that iptable_nat mangles the packets
- *      for VS/NAT.
- */
-static unsigned int ip_vs_post_routing(unsigned int hooknum,
-				       struct sk_buff *skb,
-				       const struct net_device *in,
-				       const struct net_device *out,
-				       int (*okfn)(struct sk_buff *))
-{
-	if (!skb->ipvs_property)
-		return NF_ACCEPT;
-	/* The packet was sent from IPVS, exit this chain */
-	return NF_STOP;
-}
-
 __sum16 ip_vs_checksum_complete(struct sk_buff *skb, int offset)
 {
 	return csum_fold(skb_checksum(skb, offset, skb->len - offset, 0));
@@ -767,15 +748,6 @@
 	ip_hdr(skb)->saddr = cp->vaddr;
 	ip_send_check(ip_hdr(skb));
 
-	/* For policy routing, packets originating from this
-	 * machine itself may be routed differently to packets
-	 * passing through.  We want this packet to be routed as
-	 * if it came from this machine itself.  So re-compute
-	 * the routing information.
-	 */
-	if (ip_route_me_harder(skb, RTN_LOCAL) != 0)
-		goto drop;
-
 	IP_VS_DBG_PKT(10, pp, skb, 0, "After SNAT");
 
 	ip_vs_out_stats(cp, skb);
@@ -905,18 +877,8 @@
 	int ret, restart;
 	int ihl;
 
-	/*
-	 *	Big tappo: only PACKET_HOST (neither loopback nor mcasts)
-	 *	... don't know why 1st test DOES NOT include 2nd (?)
-	 */
-	if (unlikely(skb->pkt_type != PACKET_HOST
-		     || skb->dev->flags & IFF_LOOPBACK || skb->sk)) {
-		IP_VS_DBG(12, "packet type=%d proto=%d daddr=%d.%d.%d.%d ignored\n",
-			  skb->pkt_type,
-			  ip_hdr(skb)->protocol,
-			  NIPQUAD(ip_hdr(skb)->daddr));
+	if (skb->ipvs_property)
 		return NF_ACCEPT;
-	}
 
 	iph = ip_hdr(skb);
 	if (unlikely(iph->protocol == IPPROTO_ICMP)) {
@@ -1032,8 +994,8 @@
 	.hook		= ip_vs_in,
 	.owner		= THIS_MODULE,
 	.pf		= PF_INET,
-	.hooknum        = NF_IP_LOCAL_IN,
-	.priority       = 100,
+	.hooknum        = NF_IP_POST_ROUTING,
+	.priority       = NF_IP_PRI_LAST,
 };
 
 /* After packet filtering, change source only for VS/NAT */
@@ -1041,8 +1003,17 @@
 	.hook		= ip_vs_out,
 	.owner		= THIS_MODULE,
 	.pf		= PF_INET,
-	.hooknum        = NF_IP_FORWARD,
-	.priority       = 100,
+	.hooknum        = NF_IP_PRE_ROUTING,
+	.priority       = NF_IP_PRI_FIRST+1,
+};
+
+/* After packet filtering, change source only for VS/NAT */
+static struct nf_hook_ops ip_vs_local_out_ops = {
+	.hook		= ip_vs_out,
+	.owner		= THIS_MODULE,
+	.pf		= PF_INET,
+	.hooknum        = NF_IP_LOCAL_OUT,
+	.priority       = NF_IP_PRI_FIRST+1,
 };
 
 /* After packet filtering (but before ip_vs_out_icmp), catch icmp
@@ -1051,17 +1022,18 @@
 	.hook		= ip_vs_forward_icmp,
 	.owner		= THIS_MODULE,
 	.pf		= PF_INET,
-	.hooknum        = NF_IP_FORWARD,
-	.priority       = 99,
+	.hooknum        = NF_IP_PRE_ROUTING,
+	.priority       = NF_IP_PRI_FIRST,
 };
 
-/* Before the netfilter connection tracking, exit from POST_ROUTING */
-static struct nf_hook_ops ip_vs_post_routing_ops = {
-	.hook		= ip_vs_post_routing,
+/* After packet filtering (but before ip_vs_out_icmp), catch icmp
+   destined for 0.0.0.0/0, which is for incoming IPVS connections */
+static struct nf_hook_ops ip_vs_local_icmp_ops = {
+	.hook		= ip_vs_forward_icmp,
 	.owner		= THIS_MODULE,
 	.pf		= PF_INET,
-	.hooknum        = NF_IP_POST_ROUTING,
-	.priority       = NF_IP_PRI_NAT_SRC-1,
+	.hooknum        = NF_IP_LOCAL_OUT,
+	.priority       = NF_IP_PRI_FIRST,
 };
 
 
@@ -1103,22 +1075,32 @@
 		IP_VS_ERR("can't register out hook.\n");
 		goto cleanup_inops;
 	}
-	ret = nf_register_hook(&ip_vs_post_routing_ops);
+
+	ret = nf_register_hook(&ip_vs_forward_icmp_ops);
 	if (ret < 0) {
-		IP_VS_ERR("can't register post_routing hook.\n");
+		IP_VS_ERR("can't register forward_icmp hook.\n");
 		goto cleanup_outops;
 	}
-	ret = nf_register_hook(&ip_vs_forward_icmp_ops);
+
+	ret = nf_register_hook(&ip_vs_local_out_ops);
 	if (ret < 0) {
-		IP_VS_ERR("can't register forward_icmp hook.\n");
-		goto cleanup_postroutingops;
+		IP_VS_ERR("can't register local out hook.\n");
+		goto cleanup_icmpops;
+	}
+
+	ret = nf_register_hook(&ip_vs_local_icmp_ops);
+	if (ret < 0) {
+		IP_VS_ERR("can't register local icmp hook.\n");
+		goto cleanup_localout;
 	}
 
 	IP_VS_INFO("ipvs loaded.\n");
 	return ret;
 
-  cleanup_postroutingops:
-	nf_unregister_hook(&ip_vs_post_routing_ops);
+  cleanup_localout:
+    nf_unregister_hook(&ip_vs_local_out_ops);
+  cleanup_icmpops:
+    nf_unregister_hook(&ip_vs_forward_icmp_ops);
   cleanup_outops:
 	nf_unregister_hook(&ip_vs_out_ops);
   cleanup_inops:
@@ -1137,7 +1119,6 @@
 static void __exit ip_vs_cleanup(void)
 {
 	nf_unregister_hook(&ip_vs_forward_icmp_ops);
-	nf_unregister_hook(&ip_vs_post_routing_ops);
 	nf_unregister_hook(&ip_vs_out_ops);
 	nf_unregister_hook(&ip_vs_in_ops);
 	ip_vs_conn_cleanup();
--- linux-2.6.24-gentoo-r4/net/ipv4/ipvs/ip_vs_ctl.c	2008-01-25 07:58:37.000000000 +0900
+++ linux-2.6.24-gentoo-r4-jason/net/ipv4/ipvs/ip_vs_ctl.c	2008-04-11 12:23:59.000000000 +0900
@@ -706,7 +706,7 @@
 	/* check if local node and update the flags */
 	if (inet_addr_type(udest->addr) == RTN_LOCAL) {
 		conn_flags = (conn_flags & ~IP_VS_CONN_F_FWD_MASK)
-			| IP_VS_CONN_F_LOCALNODE;
+			| IP_VS_CONN_F_MASQ;
 	}
 
 	/* set the IP_VS_CONN_F_NOOUTPUT flag if not masquerading/NAT */
--- linux-2.6.24-gentoo-r4/net/ipv4/ipvs/ip_vs_xmit.c	2008-04-10 18:20:03.000000000 +0900
+++ linux-2.6.24-gentoo-r4-jason/net/ipv4/ipvs/ip_vs_xmit.c	2008-04-11 12:15:14.000000000 +0900
@@ -129,8 +129,8 @@
 do {							\
 	(skb)->ipvs_property = 1;			\
 	skb_forward_csum(skb);				\
-	NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, (skb), NULL,	\
-		(rt)->u.dst.dev, dst_output);		\
+	NF_HOOK_THRESH(PF_INET, NF_IP_POST_ROUTING, (skb), NULL,	\
+		(rt)->u.dst.dev, dst_output, NF_IP_PRI_LAST);		\
 } while (0)
 
 

Re: moving ipvs() to POST/PREROUTING

[Joseph Mack NA3T]

On Fri, 11 Apr 2008, Jason Stubbs wrote:

> With local node, 127.0.0.1 doesn't work but an IP address on a local interface
> does.

that will do.

Local node isn't real important. It was there because it 
could be done, rather than because it was needed. If you can 
do it, we'll take it, but otherwise don't worry a whole lot 
about it.

> LVS-TUN should work as LVS-DR didn't require any direct 
> modification, but it's a little bit of a pain to set up 
> for testing at this stage.

have the same physical setup as LVS-DR and just change the 
if on the realservers to tunl0 and change the 
appropriated ipvsadm lines.

> Is there any problem with essentially hiding the real 
> servers from netfilter?

I don't know what this means (I didn't know that netfilter 
knew about the realservers).

Will your setup handle the F5-SNAT situation?

http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.non-modified_realservers.html#F5_snat

Joe

-- 
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

Re: moving ipvs() to POST/PREROUTING

[Jason Stubbs]

On Friday 11 April 2008 21:37:03 JST, Joseph Mack NA3T wrote:
> On Fri, 11 Apr 2008, Jason Stubbs wrote:
> > With local node, 127.0.0.1 doesn't work but an IP address on a local
> > interface does.
>
> that will do.
>
> Local node isn't real important. It was there because it
> could be done, rather than because it was needed. If you can
> do it, we'll take it, but otherwise don't worry a whole lot
> about it.

I did look a little bit further into it. The iptables REDIRECT module maps to 
127.0.0.1 for locally generated traffic and the first IP on the first 
interface otherwise. I haven't tried yet, but the same thing could probably 
done here.

> > LVS-TUN should work as LVS-DR didn't require any direct
> > modification, but it's a little bit of a pain to set up
> > for testing at this stage.
>
> have the same physical setup as LVS-DR and just change the
> if on the realservers to tunl0 and change the
> appropriated ipvsadm lines.

Hmm.. Well seeing I'm trying getting my hands dirty every else, I may as well 
do so here too. Will give it a try on Monday.

> > Is there any problem with essentially hiding the real
> > servers from netfilter?
>
> I don't know what this means (I didn't know that netfilter
> knew about the realservers).

I mean that it'd be nice for rules to go something like:
* Allow from external to VIP
* Allow anything established
* Drop everything else

Depending on where LVS translations are placed in the netfilter path, rules 
allowing traffic from external to RIPs may also be needed. That can get 
pretty complicated, but there might be some need for it that I can't see...

> Will your setup handle the F5-SNAT situation?
>
> http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.non-modified_realser
>vers.html#F5_snat

Yep, this is just SNAT as far as I can tell. I tested SNAT on both sides of 
the director and there weren't any problems.

--
Jason Stubbs

Re: moving ipvs() to POST/PREROUTING

[Joseph Mack NA3T]

On Fri, 11 Apr 2008, Jason Stubbs wrote:

>>> Is there any problem with essentially hiding the real
>>> servers from netfilter?
>>
>> I don't know what this means (I didn't know that netfilter
>> knew about the realservers).
>
> I mean that it'd be nice for rules to go something like:
> * Allow from external to VIP
> * Allow anything established
> * Drop everything else
>
> Depending on where LVS translations are placed in the netfilter path, rules
> allowing traffic from external to RIPs may also be needed.

I would hope people don't do this. RIPs should be private, 
for security reasons and to preserve the fiction that the 
LVS setup is one machine. The LVS'ed application running on 
the realserver might start a client process that needs to 
contact 0/0, but that can be nat'ed out, possibly through 
the VIP on the director, or maybe some other public IP 
available to the realserver. Is this what you want to do?

see

http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.LVS-DR.html#Pearthree

>> Will your setup handle the F5-SNAT situation?
>>
>> http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.non-modified_realser
>> vers.html#F5_snat
>
> Yep, this is just SNAT as far as I can tell. I tested SNAT on both sides of
> the director and there weren't any problems.

neato. Some people will be very happy about this.

I take it that you're working late at night on this :-)

Joe

-- 
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

Re: moving ipvs() to POST/PREROUTING

[Jason Stubbs]

On Friday 11 April 2008 23:38:30 JST, Joseph Mack NA3T wrote:
> On Fri, 11 Apr 2008, Jason Stubbs wrote:
> >>> Is there any problem with essentially hiding the real
> >>> servers from netfilter?
> >>
> >> I don't know what this means (I didn't know that netfilter
> >> knew about the realservers).
> >
> > I mean that it'd be nice for rules to go something like:
> > * Allow from external to VIP
> > * Allow anything established
> > * Drop everything else
> >
> > Depending on where LVS translations are placed in the netfilter path,
> > rules allowing traffic from external to RIPs may also be needed.
>
> I would hope people don't do this. RIPs should be private,
> for security reasons and to preserve the fiction that the
> LVS setup is one machine.

This is precisely why I chose the hooks that I did. My intention was for the 
netfilter chains to only ever see the VIP, but packets with the RIP are going 
through too after IP_VS_XMIT is called.

> The LVS'ed application running on the realserver might start a client 
> process that needs to contact 0/0, but that can be nat'ed out, possibly 
> through the VIP on the director, or maybe some other public IP available to 
> the realserver. Is this what you want to do?   
>
> see
> http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.LVS-DR.html#Pearthree

I didn't quite follow this. Are you referring to services such as FTP? Nothing 
should have changed in this regard with my patch. The link did remind me that 
I need to test the sync daemon with my patch though. :)

> I take it that you're working late at night on this :-)

Nope, I'm not that crazy. Just reading and responding to work emails from home 
as per usual. ;)

-- 
Jason Stubbs

Re: moving ipvs() to POST/PREROUTING

[Joseph Mack NA3T]

On Sat, 12 Apr 2008, Jason Stubbs wrote:

>> I would hope people don't do this. RIPs should be private,
>> for security reasons and to preserve the fiction that the
>> LVS setup is one machine.
>
> This is precisely why I chose the hooks that I did. My intention was for the
> netfilter chains to only ever see the VIP, but packets with the RIP are going
> through too after IP_VS_XMIT is called.

hmm. still don't know what you're referring to then. Is this 
LVS-NAT, LVS-DR...?

netfilter sees the source and dest on the packets. How can 
netfilter only see the VIP?

>> see
>> http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.LVS-DR.html#Pearthree

sounds like you have the same problem with what I'm saying.

> I didn't quite follow this. Are you referring to services such as FTP?

no. this webpage shows why clients shouldn't know about the 
RIP and what you can do to make sure they don't find out 
about it.

Joe
-- 
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

Re: moving ipvs() to POST/PREROUTING

[Jason Stubbs]

On Saturday 12 April 2008 01:14:08 JST, Joseph Mack NA3T wrote:
> On Sat, 12 Apr 2008, Jason Stubbs wrote:
> >> I would hope people don't do this. RIPs should be private,
> >> for security reasons and to preserve the fiction that the
> >> LVS setup is one machine.
> >
> > This is precisely why I chose the hooks that I did. My intention was for
> > the netfilter chains to only ever see the VIP, but packets with the RIP
> > are going through too after IP_VS_XMIT is called.
>
> hmm. still don't know what you're referring to then. Is this
> LVS-NAT, LVS-DR...?
>
> netfilter sees the source and dest on the packets. How can
> netfilter only see the VIP?

Current LVS-NAT goes something like:
PREROUTING  client => VIP
LOCAL_IN    client => VIP
    LVS in hook
LOCAL_OUT   client => RIP
POSTROUTING client => RIP

With my patch it goes like:
PREROUTING  client => VIP
FORWARD     client => VIP
POSTROUTING client => VIP
    LVS hook
POSTROUTING client => RIP

LVS-DR is pretty much the same flow except that the packet's MAC destination 
is changed instead. What I mean with netfilter only seeing the VIP is that 
I'd like the second POSTROUTING run that happens with my patch to not happen. 
Doing so makes rule writing on the director a little easier as the rules 
don't need to know about the RIPs at all.

The configuration I'm planning to use is a LVS-NAT director/firewall that 
allows external traffic to VIPs, SNATs traffic from RIPs to external 
(including to VIPs) and drops everything else.

With LVS-DR, I am pretty sure that all the issues mentioned in the page you 
linked to still apply as my patch only changes the flow through the director. 
The rules mentioned in the following link would be needed whether the 
SERVER_GW is separate device to the director or not.

http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.non-lvs_clients_on_realservers.html#3-tier_routes

-- 
Jason Stubbs

Re: moving ipvs() to POST/PREROUTING

[Joseph Mack NA3T]

On Sat, 12 Apr 2008, Jason Stubbs wrote:

I'm not cc:ing you any more. There's no need to cc me, I 
read everything from lvs. I cc people when I think they may 
not be paying attention, Horms because he's busy, Julian 
because most traffic doesn't concern his patches.

>> netfilter sees the source and dest on the packets. How can
>> netfilter only see the VIP?
>
> Current LVS-NAT goes something like:
> PREROUTING  client => VIP
> LOCAL_IN    client => VIP
>    LVS in hook
> LOCAL_OUT   client => RIP
> POSTROUTING client => RIP

yup

> With my patch it goes like:
> PREROUTING  client => VIP
> FORWARD     client => VIP
> POSTROUTING client => VIP
>    LVS hook
> POSTROUTING client => RIP
>
> LVS-DR is pretty much the same flow except that the 
> packet's MAC destination is changed instead. What I mean 
> with netfilter only seeing the VIP is that I'd like the 
> second POSTROUTING run that happens with my patch to not 
> happen. Doing so makes rule writing on the director a 
> little easier as the rules don't need to know about the 
> RIPs at all.

hmm, maybe I see.

Let me see if I've got it.

1.
Because the packet is controlled by LVS-NAT, it should be 
accepted as OK and will never have to be inspected for 
bogosity (you will have done that already, if you'd needed 
it). ie you aren't going to be filtering on RIPs.

> The configuration I'm planning to use is a LVS-NAT 
> director/firewall that allows external traffic to VIPs, 
> SNATs traffic from RIPs to external (including to VIPs) 
> and drops everything else.

2.
your particular setup does this, and reasonably other 
LVS-NAT setups will be doing much the same.

> With LVS-DR, I am pretty sure that all the issues 
> mentioned in the page you linked to still apply as my 
> patch only changes the flow through the director. The 
> rules mentioned in the following link would be needed 
> whether the SERVER_GW is separate device to the director 
> or not.
>
> http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.non-lvs_clients_on_realservers.html#3-tier_routes

OK

You'd better run it by Horms. I'm happy to talk about this 
while he's busy, but he knows the code better than I do and 
will be able to cast a more critical eye

Hey Horms,
 	Can you give Jason's scheme the once over?

Joe

-- 
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

Re: moving ipvs() to POST/PREROUTING

[Julian Anastasov]

	Hello,

On Fri, 11 Apr 2008, Jason Stubbs wrote:

> Greetings,
> 
> Ok, things are mostly working now. The patch is a little messy as in there's 
> old comments remaining and function names are left as is, but hopefully 
> reviewable. If it's not, I'll split it up and add appropriate comments...

	Your changes will break existing setups. I'll recommend
you to start by reading http://www.ssi.bg/~ja/LVS.txt. I just updated
it with some 2.6 info as it was too old document. There you can
see some requirements and motivation why IPVS uses specific hooks and
priority.

	I think, for such changes there are many things to be considered
and carefully tested:

- all forwarding methods can be tested on LAN, even LVS-TUN
- forwarding of related ICMP traffic (ICMP errors) in both directions,
for all methods
- ICMP generation to both sides (client and real server): when there is no 
real server, when skb is longer than PMTU.
- scheduling by nfmark
- firewall: at least basic packet fields matching
- ip_vs_ftp testing (LVS-NAT) when netfilter ftp module is in
effect: test if double NAT happens resulting in broken packets (TCP 
sequence numbers or payload) when payload is changed if IP:PORT
strings in FTP commands have different length (VIP and RIP).

	Note that there are many new changes in Netfilter and
Networking after IPVS was included in early 2.6. Even I already don't know 
what happens in latest kernels for POST_ROUTING, with fragmentation, etc. 
May be some things work by luck because IPVS tries to work closely with 
Netfilter without breaking things. That is why a careful testing is needed
for any new changes if such changes are planned for inclusion in kernel.

> With local node, 127.0.0.1 doesn't work but an IP address on a local interface 
> does. When the address is 127.0.0.1, the SYN makes it all the way through 
> INPUT, but the SYN/ACK doesn't come into OUTPUT. Something to investigate 
> further... Also, null_xmit doesn't work as ipvs_in is being done in 
> POSTROUTING, so I've simple aliased LOCAL to MASQ for the time being.

	LOCAL replaced with MASQ? Such changes can not be accepted for 
inclusion, they break existing setups just because something does not 
work in your new way to handle things. You should always remember that 
there must be a reason some code to exist. If you really want to modify 
IPVS I'll recommend you to create some short document that explains:

- how do you plan out->in (ip_vs_in) and in->out (ip_vs_out) packets
to traverse netfilter hooks, when addresses, ports and payload are
modified (ip_vs_ftp)

- what setups you are going to break because you consider them as
not used anymore

- use defines/configuration options to preserve old handling for
existing setups.

	If your changes are not planned for inclusion you can do
whatever you want, of course.

> What I haven't tested:
> * LVS-TUN
> * ICMP for LVS-NAT

	You can test if related ICMP errors are forwarded by
adding REJECT-with-ICMP rules in client and real server.

> * IP_VS_CONN_F_BYPASS - what is this?

	IP_VS_CONN_F_BYPASS is used for transparent proxy setups when
real server (cache server) is not present and we should forward the 
traffic to original destination. The idea is request still to be served.
In such case IPVS traffic uses the original destination instead of real 
server.

> I realized I haven't explained at all why I chose POST/PRE as the hook points. 
> Firstly the cropped output from a LOG target in every mangle table for the 
> SYN SYN/ACK of a LVS-NAT connection:
> 
> PREROUTING IN=eth0 OUT= SRC=192.168.0.104 DST=192.168.0.SYN
> FORWARD IN=eth0 OUT=eth0 SRC=192.168.0.104 DST=192.168.0.7 SYN
> POSTROUTING IN= OUT=eth0 SRC=192.168.0.104 DST=192.168.0.7 SYN
> POSTROUTING IN= OUT=eth1 SRC=192.168.0.104 DST=192.168.1.3 SYN
> 
> PREROUTING IN=eth1 OUT= SRC=192.168.0.7 DST=192.168.0.104 ACK SYN
> FORWARD IN=eth1 OUT=eth0 SRC=192.168.0.7 DST=192.168.0.104 ACK SYN
> POSTROUTING IN= OUT=eth0 SRC=192.168.0.7 DST=192.168.0.104 ACK SYN
> 
> 192.168.0.104 is the client, 192.168.0.7 is the VIP and 192.168.1.3 is the 
> real server. Other than the second POSTROUTING entry on the SYN side, 
> netfilter isn't dealing with the real server's IP at all. This will 
> theoretically make writing firewall rules much easier and also limits what 
> netfilter's conntracking has to deal with.
> 
> Actually, I don't know why the second POSTROUTING entry is there at all. It 
> seems that after the packet is injected into the end of POSTROUTING, a 
> routing decision is being made again and POSTROUTING is rerun. Preferable the 
> packet would go straight out the appropriate interface after ipvs_in is run.

	Not sure what happens, it is a good idea to put some printk()s
in netfilter (eg. hooks) when testing IPVS changes.

> Similar behaviour happens with a local node:
> 
> PREROUTING IN=eth0 OUT= SRC=192.168.0.104 DST=192.168.0.7 SYN
> FORWARD IN=eth0 OUT=eth0 SRC=192.168.0.104 DST=192.168.0.7 SYN
> POSTROUTING IN= OUT=eth0 SRC=192.168.0.104 DST=192.168.0.7 SYN
> POSTROUTING IN= OUT=lo SRC=192.168.0.104 DST=192.168.0.5 SYN
> PREROUTING IN=lo OUT= SRC=192.168.0.104 DST=192.168.0.5 SYN
> INPUT IN=lo OUT= SRC=192.168.0.104 DST=192.168.0.5 SYN
> 
> OUTPUT IN= OUT=eth0 SRC=192.168.0.7 DST=192.168.0.104 ACK SYN
> POSTROUTING IN= OUT=eth0 SRC=192.168.0.7 DST=192.168.0.104 ACK SYN
> 
> 192.168.0.5 is an IP local to the director. I had to add the ipvs_out hooks to 
> the beginning of OUTPUT as the local reply never hits PREROUTING. Again with 
> the above, I'd prefer the POST/PRE/INPUT disappear.

	Why? I don't think it is possible without changes in Netfilter.
There are some issues that prevent IPVS to benefit from Netfilter
connection tracking:

- Netfilter's NAT and routing are not in single place (hook), difficult to
handle LVS-DR
- Netfilter can re-route sometimes (eg. after mangle), it can cause
properly routed LVS-DR traffic to fail.
- Double NAT for ip_vs_ftp

> Anyway, that's pretty much my intention. Is there any problem with essentially 
> hiding the real servers from netfilter? Is there a way to get the packet out 
> of the netfilter loop earlier?

	IPVS traffic should not be NAT-ed by Netfilter. This double-NAT
leads to broken packets as I already mentioned above.

	What I do not understand is what is the end goal for your
changes? Speed or IPVS traffic to fully benefit from Netfilter features?
Or some setup does not work?

Regards

--
Julian Anastasov <ja@ssi.bg>

Re: moving ipvs() to POST/PREROUTING

[Jason Stubbs]

On Saturday 12 April 2008 18:07:59 JST, Julian Anastasov wrote:
> 	Hello,
>
> On Fri, 11 Apr 2008, Jason Stubbs wrote:
> > Greetings,
> >
> > Ok, things are mostly working now. The patch is a little messy as in
> > there's old comments remaining and function names are left as is, but
> > hopefully reviewable. If it's not, I'll split it up and add appropriate
> > comments...
>
> 	Your changes will break existing setups. I'll recommend
> you to start by reading http://www.ssi.bg/~ja/LVS.txt. I just updated
> it with some 2.6 info as it was too old document. There you can
> see some requirements and motivation why IPVS uses specific hooks and
> priority.

My initial expectation was that this sort of patch wouldn't be integrated 
because it does break existing setups, but that these or similar changes 
would need to occur regardless. Thanks for link. I'd already learnt about 
half of what's written from reading the netfilter sources, but it answers a 
lot of the more difficult questions.

> 	I think, for such changes there are many things to be considered
> and carefully tested:
>
> - all forwarding methods can be tested on LAN, even LVS-TUN
> - forwarding of related ICMP traffic (ICMP errors) in both directions,
> for all methods
> - ICMP generation to both sides (client and real server): when there is no
> real server, when skb is longer than PMTU.
> - scheduling by nfmark
> - firewall: at least basic packet fields matching
> - ip_vs_ftp testing (LVS-NAT) when netfilter ftp module is in
> effect: test if double NAT happens resulting in broken packets (TCP
> sequence numbers or payload) when payload is changed if IP:PORT
> strings in FTP commands have different length (VIP and RIP).

I've tested LVS-NAT and LVS-DR on a LAN, but haven't tested any of the others 
yet. I'm confident that they all still work or can be made to work with very 
little tweaking though. Another thing listed in your link above that I 
probably need to test and address is fragmented packet handling.

> 	Note that there are many new changes in Netfilter and
> Networking after IPVS was included in early 2.6. Even I already don't know
> what happens in latest kernels for POST_ROUTING, with fragmentation, etc.
> May be some things work by luck because IPVS tries to work closely with
> Netfilter without breaking things. That is why a careful testing is needed
> for any new changes if such changes are planned for inclusion in kernel.

Whether included or not, I need to test the changes before putting it in 
production. ;)

> > With local node, 127.0.0.1 doesn't work but an IP address on a local
> > interface does. When the address is 127.0.0.1, the SYN makes it all the
> > way through INPUT, but the SYN/ACK doesn't come into OUTPUT. Something to
> > investigate further... Also, null_xmit doesn't work as ipvs_in is being
> > done in POSTROUTING, so I've simple aliased LOCAL to MASQ for the time
> > being.
>
> 	LOCAL replaced with MASQ? Such changes can not be accepted for
> inclusion, they break existing setups just because something does not
> work in your new way to handle things.

Local node via a local interface address works fine - only 127.0.0.1 doesn't. 
I have a plan on how to fix this though, so existing setups won't be broken 
for this point.

> You should always remember that there must be a reason some code to exist. 
> If you really want to modify IPVS I'll recommend you to create some short 
> document that explains:  

<snip>

I'll put those things together. And thanks for the other info on how to test 
things.

> 	IPVS traffic should not be NAT-ed by Netfilter. This double-NAT
> leads to broken packets as I already mentioned above.
>
> 	What I do not understand is what is the end goal for your
> changes? Speed or IPVS traffic to fully benefit from Netfilter features?
> Or some setup does not work?

I can understand that trying to use DNAT with LVS is not a good idea, but SNAT 
is needed to allow access to VIPs from real servers when using LVS-NAT. That 
and making firewall rules a little easier to write are my goals. 
Unfortunately speed will actually decrease a little unless connection 
tracking is disabled.

--
Jason Stubbs

Re: moving ipvs() to POST/PREROUTING

[Jason Stubbs]

Replying again now that I've got a lot of testing done. I'm heading home soon 
so just posting an update and will test the last few things tomorrow. I'll 
also try to submit a broken up patch tomorrow so that specific changes and 
reasoning can be discussed more easily.

On Saturday 12 April 2008 18:07:59 Julian Anastasov wrote:
> - all forwarding methods can be tested on LAN, even LVS-TUN

Haven't been able to test LVS-TUN yet, but I can't see any reason why it 
shouldn't work. Will test tomorrow morning.

> - forwarding of related ICMP traffic (ICMP errors) in both directions,
> for all methods

Tested.

> - ICMP generation to both sides (client and real server): when there is no
> real server, when skb is longer than PMTU.

I tested the no real server case. How can i test the skb-too-large case? I 
tried changing the MTU on the director's interface on the real-server side, 
but traffic continued to pass fine...

> - scheduling by nfmark

Tested.

> - firewall: at least basic packet fields matching

iptables -t filter -A FORWARD -d 192.168.0.7 -i eth0 -j ACCEPT
iptables -t filter -A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -t filter -P FORWARD REJECT

where 192.168.0.7 is the VIP I'm testing with worked fine. Pretty much any 
firewall rules should work fine as I've moved LVS hooks to the points of 
entry/exit of the netfilter flow.

> - ip_vs_ftp testing (LVS-NAT) when netfilter ftp module is in
> effect: test if double NAT happens resulting in broken packets (TCP
> sequence numbers or payload) when payload is changed if IP:PORT
> strings in FTP commands have different length (VIP and RIP).

Tested plain LVS-NAT as well as with SNAT and DNAT. Surprisingly, it works 
even with the above conntrack-based filter in place. I think that indicates 
that double NAT is happening (else RELATED wouldn't match) but with the order 
that hooks are called things don't get screwed up - I think. I'll investigate 
this a little further.

> > * IP_VS_CONN_F_BYPASS - what is this?
>
> 	IP_VS_CONN_F_BYPASS is used for transparent proxy setups when
> real server (cache server) is not present and we should forward the
> traffic to original destination. The idea is request still to be served.
> In such case IPVS traffic uses the original destination instead of real
> server.

Not tested yet. I assume I just need to add a real server with the same 
IP/port as the virtual server?

> There are some issues that prevent IPVS to benefit from Netfilter
> connection tracking:
>
> - Netfilter's NAT and routing are not in single place (hook), difficult to
> handle LVS-DR

LVS-DR with real server as a client using SNAT... Will test this tomorrow too.

> - Netfilter can re-route sometimes (eg. after mangle), it can cause
> properly routed LVS-DR traffic to fail.

I don't understand exactly what you mean by this. It could only happen if the 
user sets rules that causes it to happen right?

> - Double NAT for ip_vs_ftp

Addressed above.

-- 
Jason Stubbs <j.stubbs@linkthink.co.jp>
LINKTHINK INC.
東京都渋谷区桜ヶ丘町22-14 N.E.S S棟 3F
TEL 03-5728-4772  FAX 03-5728-4773

Re: moving ipvs() to POST/PREROUTING

[Julian Anastasov]

	Hello,

On Mon, 14 Apr 2008, Jason Stubbs wrote:

> > > * IP_VS_CONN_F_BYPASS - what is this?
> >
> > 	IP_VS_CONN_F_BYPASS is used for transparent proxy setups when
> > real server (cache server) is not present and we should forward the
> > traffic to original destination. The idea is request still to be served.
> > In such case IPVS traffic uses the original destination instead of real
> > server.
> 
> Not tested yet. I assume I just need to add a real server with the same 
> IP/port as the virtual server?

	Not sure, may be IP:PORT of LVS's uplink gateway. In such
setups clients are usually internal hosts using LVS box as gateway.

> > - Netfilter can re-route sometimes (eg. after mangle), it can cause
> > properly routed LVS-DR traffic to fail.
> 
> I don't understand exactly what you mean by this. It could only happen if the 
> user sets rules that causes it to happen right?

	May be the things have changed, not sure. The problem is when
functions like ip_route_me_harder() are called for packets that
are already forwarded by IPVS (skb has attached route for the
real server). In such case skb still shows VIP as iph->daddr and
a rerouting can result in local route. But latest kernels may be
reroute only in LOCAL_OUT, so this is not a problem.

	As for the double POST_ROUTING log entries ... I'm checking
this NF_HOOK_THRESH call with NF_IP_PRI_LAST. For me, it looks like
net/netfilter/core.c:nf_iterate() calls only handlers when
elem->priority >= hook_thresh. But you put ip_vs_in() at the same
priority NF_IP_PRI_LAST. May be ip_vs_in() is called twice?

	Also, if ip_vs_in() is called  after SNAT I'm curious, isn't 
ip_vs_ftp working with DNAT-ed skb (if you have netfilter ftp nat
module) ? May be things don't break because IPVS is careful not to
damage packets it can not recognize. But do we work properly with
the FTP commands in ip_vs_ftp_in(), do we create properly FTP data 
connections in IPVS? I assume you test both passive and active FTP.
If the goal is -m state to work correctly, are you sure the IPVS FTP
data connections work correctly (as RELATED traffic)?

Regards

--
Julian Anastasov <ja@ssi.bg>

Re: moving ipvs() to POST/PREROUTING

[Jason Stubbs]

On Tuesday 15 April 2008 08:16:58 Julian Anastasov wrote:
> On Mon, 14 Apr 2008, Jason Stubbs wrote:
> > > > * IP_VS_CONN_F_BYPASS - what is this?
> > >
> > > 	IP_VS_CONN_F_BYPASS is used for transparent proxy setups when
> > > real server (cache server) is not present and we should forward the
> > > traffic to original destination. The idea is request still to be
> > > served. In such case IPVS traffic uses the original destination instead
> > > of real server.
> >
> > Not tested yet. I assume I just need to add a real server with the same
> > IP/port as the virtual server?
>
> 	Not sure, may be IP:PORT of LVS's uplink gateway. In such
> setups clients are usually internal hosts using LVS box as gateway.

I figured this one out by looking at the source and it tested fine too. The 
virtual server must be fwmark and the sysctl ip.vs.cache_bypass must be set 
to 1. With those settings, the packet is passed when there are no real 
servers available rather than being rejected with ICMP_PORT_UNREACH.

> > > - Netfilter can re-route sometimes (eg. after mangle), it can cause
> > > properly routed LVS-DR traffic to fail.
> >
> > I don't understand exactly what you mean by this. It could only happen if
> > the user sets rules that causes it to happen right?
>
> 	May be the things have changed, not sure. The problem is when
> functions like ip_route_me_harder() are called for packets that
> are already forwarded by IPVS (skb has attached route for the
> real server). In such case skb still shows VIP as iph->daddr and
> a rerouting can result in local route. But latest kernels may be
> reroute only in LOCAL_OUT, so this is not a problem.


> 	As for the double POST_ROUTING log entries ... I'm checking
> this NF_HOOK_THRESH call with NF_IP_PRI_LAST. For me, it looks like
> net/netfilter/core.c:nf_iterate() calls only handlers when
> elem->priority >= hook_thresh. But you put ip_vs_in() at the same
> priority NF_IP_PRI_LAST. May be ip_vs_in() is called twice?

The dst_output specified to NF_HOOK_THRESH is called after nf_iterate() is 
finished. dst_output then calls NF_HOOK_THRESH with POST_ROUTING again 
passing the result to an internal function to actually send the packet out.
There's no way around it except perhaps with an NF_STOP hook. I'd like to 
figure out how traffic (congestion) control fits in first though.

> 	Also, if ip_vs_in() is called  after SNAT I'm curious, isn't
> ip_vs_ftp working with DNAT-ed skb (if you have netfilter ftp nat
> module) ? May be things don't break because IPVS is careful not to
> damage packets it can not recognize. But do we work properly with
> the FTP commands in ip_vs_ftp_in(), do we create properly FTP data
> connections in IPVS? I assume you test both passive and active FTP.
> If the goal is -m state to work correctly, are you sure the IPVS FTP
> data connections work correctly (as RELATED traffic)?

I tried again with the accept rule as:

iptables -t filter -A FORWARD -p tcp -d 192.168.0.7 --dport 21 -i eth0 -j 
ACCEPT

However, it still worked fine in both passive and active modes. The patch 
series I posted may make it clear, but the way I visualize what's happening 
is that there are two proxies between the client and the server. Something 
like: client => DNAT => LVS => server. DNAT and LVS are unaware of each 
other's existence, but there is no problems because the order of processing 
make it transparent.

-- 
Jason Stubbs <j.stubbs@linkthink.co.jp>
LINKTHINK INC.
東京都渋谷区桜ヶ丘町22-14 N.E.S S棟 3F
TEL 03-5728-4772  FAX 03-5728-4773