Re: Fwd: Issue migrating "iptables -m socket --transparent" into nftables

Re: Fwd: Issue migrating "iptables -m socket --transparent" into nftables

[Nirgal Vourgère]

[-- Attachment #1: Type: text/plain, Size: 2260 bytes --]

On Tuesday, 18 August 2020 11:18:50 CEST Balazs Scheidler wrote:
>> Does any one know the proper equivalent to
>>     iptables -t mangle -A PREROUTING -m socket --transparent -j MARK --set-mark 1
>> using nft?
>
> The original iptables "socket" match had an extra check so that it wouldn't
> match listener sockets, at least by default (that is if --nowildcard is not
> specified).
> 
> I don't see however how "outbound masqueraded connection" could be
> impacted. The "socket transparent 1" expression should require that the
> socket being matched has IP_TRANSPARENT setsockopt set. Are those
> connections also initiated by haproxy?
> 
> In any case, I think the check to ignore wildcard bound listener sockets is
> definitely missing, however I am not sure how to properly add it to
> nftables. If I added it to the socket match implementation that might break
> a few currently well behaving use-cases. @pablo@netfilter.org
> <pablo@netfilter.org> can you please advise? This is the check that is in
> iptables -m socket:
> 
>                 wildcard = (!(info->flags & XT_SOCKET_NOWILDCARD) &&
>                             sk_fullsock(sk) &&
>                             inet_sk(sk)->inet_rcv_saddr == 0);
> 
> And then if --transparent is used, these sockets are not accepted / the
> rule does not match.

That's it I guess:

I tried adding --nowildcard to my working iptables rules and I got the same error, https connections from the lan side are not masqueraded toward the wan, but routed locally to the socket listening to *:443.
(thanks tcptraceroute for the info)

So basically
    nft > socket transparent 1 meta mark set 1
may be the equivalent of
    iptables > -m socket --transparent --nowildcard -j MARK --set-mark 1
while I'm looking for *not* having "--nowildcard".

Any idea about how work around this? I was thinking of using the "fib" rules to match the wan side packets since they have a destination ip address that match one of the local address, while the wan bound packets don't.


Regarding your question about IP_TRANSPARENT setsockopt, I don't quite know how to look at that easily. Attached is a fragment from my haproxy.cfg file, the key points being "source 0.0.0.0 usesrc clientip" and "bind :443 strict-sni transparent".

[-- Attachment #2: haproxy.cfg --]
[-- Type: text/plain, Size: 1996 bytes --]

############################################################################
# DO NOT EDIT THAT FILE
# Notice: That file was generated using:
#    /root/bin/update-virtualhosts /etc/haproxy/virtualhosts.haproxy
# See /etc/haproxy/virtualhosts.haproxy/config
############################################################################
global
	log /dev/log	local0
	log /dev/log	local1 notice
	chroot /var/lib/haproxy
	stats socket /run/haproxy/admin.sock mode 660 level admin
	stats timeout 30s
	# user haproxy  # transparent proxying requires root privileges
	group haproxy
	daemon

	# Default SSL material locations
	ca-base /etc/ssl/certs
	crt-base /etc/ssl/private

	# Default ciphers to use on SSL-enabled listening sockets.
	# For more information, see ciphers(1SSL). This list is from:
	#  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
	# An alternative list with additional directives can be obtained from
	#  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
	ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
	ssl-default-bind-options no-sslv3

defaults
	source 0.0.0.0 usesrc clientip
	log	global
	#mode	http
	#option	httplog
	option	dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
	errorfile 400 /etc/haproxy/errors/400.http
	errorfile 403 /etc/haproxy/errors/403.http
	errorfile 408 /etc/haproxy/errors/408.http
	errorfile 500 /etc/haproxy/errors/500.http
	errorfile 502 /etc/haproxy/errors/502.http
	errorfile 503 /etc/haproxy/errors/503.http
	errorfile 504 /etc/haproxy/errors/504.http

frontend https4-in
        bind :443 strict-sni transparent
        mode tcp
        tcp-request inspect-delay 5s
        tcp-request content accept if { req_ssl_hello_type 1 }
        default_backend https4-www2.in.nirgal.com

backend https4-www2.in.nirgal.com
	server https4-www2.in.nirgal.com ipv4@192.168.1.99:443

Re: Fwd: Issue migrating "iptables -m socket --transparent" into nftables

[Pablo Neira Ayuso]

[-- Attachment #1: Type: text/plain, Size: 2276 bytes --]

On Tue, Aug 18, 2020 at 01:08:43PM +0200, Nirgal Vourgère wrote:
> On Tuesday, 18 August 2020 11:18:50 CEST Balazs Scheidler wrote:
> >> Does any one know the proper equivalent to
> >>     iptables -t mangle -A PREROUTING -m socket --transparent -j MARK --set-mark 1
> >> using nft?
> >
> > The original iptables "socket" match had an extra check so that it wouldn't
> > match listener sockets, at least by default (that is if --nowildcard is not
> > specified).
> > 
> > I don't see however how "outbound masqueraded connection" could be
> > impacted. The "socket transparent 1" expression should require that the
> > socket being matched has IP_TRANSPARENT setsockopt set. Are those
> > connections also initiated by haproxy?
> > 
> > In any case, I think the check to ignore wildcard bound listener sockets is
> > definitely missing, however I am not sure how to properly add it to
> > nftables. If I added it to the socket match implementation that might break
> > a few currently well behaving use-cases. @pablo@netfilter.org
> > <pablo@netfilter.org> can you please advise? This is the check that is in
> > iptables -m socket:
> > 
> >                 wildcard = (!(info->flags & XT_SOCKET_NOWILDCARD) &&
> >                             sk_fullsock(sk) &&
> >                             inet_sk(sk)->inet_rcv_saddr == 0);
> > 
> > And then if --transparent is used, these sockets are not accepted / the
> > rule does not match.
> 
> That's it I guess:
> 
> I tried adding --nowildcard to my working iptables rules and I got the same error, https connections from the lan side are not masqueraded toward the wan, but routed locally to the socket listening to *:443.
> (thanks tcptraceroute for the info)
> 
> So basically
>     nft > socket transparent 1 meta mark set 1
> may be the equivalent of
>     iptables > -m socket --transparent --nowildcard -j MARK --set-mark 1
> while I'm looking for *not* having "--nowildcard".
> 
> Any idea about how work around this? I was thinking of using the
> "fib" rules to match the wan side packets since they have a
> destination ip address that match one of the local address, while
> the wan bound packets don't.

I'll post this patch to netfilter-devel to add the missing logic.

[-- Attachment #2: x.patch --]
[-- Type: text/x-diff, Size: 1737 bytes --]

diff --git a/include/uapi/linux/netfilter/nf_tables.h b/include/uapi/linux/netfilter/nf_tables.h
index 42f351c1f5c5..fed3514395a5 100644
--- a/include/uapi/linux/netfilter/nf_tables.h
+++ b/include/uapi/linux/netfilter/nf_tables.h
@@ -1008,10 +1008,12 @@ enum nft_socket_attributes {
  *
  * @NFT_SOCKET_TRANSPARENT: Value of the IP(V6)_TRANSPARENT socket option
  * @NFT_SOCKET_MARK: Value of the socket mark
+ * @NFT_SOCKET_WILDCARD: Socket listener is bound to any address
  */
 enum nft_socket_keys {
 	NFT_SOCKET_TRANSPARENT,
 	NFT_SOCKET_MARK,
+	NFT_SOCKET_WILDCARD,
 	__NFT_SOCKET_MAX
 };
 #define NFT_SOCKET_MAX	(__NFT_SOCKET_MAX - 1)
diff --git a/net/netfilter/nft_socket.c b/net/netfilter/nft_socket.c
index 637ce3e8c575..7511640e513a 100644
--- a/net/netfilter/nft_socket.c
+++ b/net/netfilter/nft_socket.c
@@ -14,6 +14,23 @@ struct nft_socket {
 	};
 };
 
+static void nft_socket_wildcard(const struct nft_pktinfo *pkt,
+				struct nft_regs *regs, struct sock *sk,
+				u32 *dest)
+{
+	switch (nft_pf(pkt)) {
+	case NFPROTO_IPV4:
+		nft_reg_store8(dest, inet_sk(sk)->inet_rcv_saddr == 0);
+		break;
+	case NFPROTO_IPV6:
+		nft_reg_store8(dest, ipv6_addr_any(&sk->sk_v6_rcv_saddr));
+		break;
+	default:
+		regs->verdict.code = NFT_BREAK;
+		return;
+	}
+}
+
 static void nft_socket_eval(const struct nft_expr *expr,
 			    struct nft_regs *regs,
 			    const struct nft_pktinfo *pkt)
@@ -59,6 +76,13 @@ static void nft_socket_eval(const struct nft_expr *expr,
 			return;
 		}
 		break;
+	case NFT_SOCKET_WILDCARD:
+		if (!sk_fullsock(sk)) {
+			regs->verdict.code = NFT_BREAK;
+			return;
+		}
+		nft_socket_wildcard(pkt, regs, sk, dest);
+		break;
 	default:
 		WARN_ON(1);
 		regs->verdict.code = NFT_BREAK;
-- 
2.20.1

Re: Fwd: Issue migrating "iptables -m socket --transparent" into nftables

[Pablo Neira Ayuso]

[-- Attachment #1: Type: text/plain, Size: 768 bytes --]

On Fri, Aug 21, 2020 at 05:15:21PM +0200, Balazs Scheidler wrote:
> Hi,
> 
> Here's the accompanying nftables patch, just in case Pablo didn't do it.

Thanks Balazs, this looks good to me!

> Pablo do you want me to submit these as a pull request?

You can just send them via git format-patch to
netfilter-devel@vger.kernel.org.

> All I did for testing was that it did compile this ruleset and attempted to
> submit it via netlink to the kernel, which it refused, as I didn't patch my
> kernel.

I'm attaching the kernel patch, compiled-tested only by now.

> ```
> table inet haproxy {
>   chain prerouting {
>      type filter hook prerouting priority -150; policy accept;
>      socket transparent 1 socket wildcard 0 mark set 0x00000001
>    }
> }
> ```

Thanks.

[-- Attachment #2: 0001-netfilter-nft_socket-add-wildcard-support.patch --]
[-- Type: text/x-diff, Size: 2498 bytes --]

From 6c7ffee435cead6d6b97eef62455e77a35537fd8 Mon Sep 17 00:00:00 2001
From: Pablo Neira Ayuso <pablo@netfilter.org>
Date: Wed, 19 Aug 2020 09:47:40 +0200
Subject: [PATCH] netfilter: nft_socket: add wildcard support

Add NFT_SOCKET_WILDCARD to match to wildcard socket listener.

Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
---
 include/uapi/linux/netfilter/nf_tables.h |  2 ++
 net/netfilter/nft_socket.c               | 25 ++++++++++++++++++++++++
 2 files changed, 27 insertions(+)

diff --git a/include/uapi/linux/netfilter/nf_tables.h b/include/uapi/linux/netfilter/nf_tables.h
index 42f351c1f5c5..fed3514395a5 100644
--- a/include/uapi/linux/netfilter/nf_tables.h
+++ b/include/uapi/linux/netfilter/nf_tables.h
@@ -1008,10 +1008,12 @@ enum nft_socket_attributes {
  *
  * @NFT_SOCKET_TRANSPARENT: Value of the IP(V6)_TRANSPARENT socket option
  * @NFT_SOCKET_MARK: Value of the socket mark
+ * @NFT_SOCKET_WILDCARD: Socket listener is bound to any address
  */
 enum nft_socket_keys {
 	NFT_SOCKET_TRANSPARENT,
 	NFT_SOCKET_MARK,
+	NFT_SOCKET_WILDCARD,
 	__NFT_SOCKET_MAX
 };
 #define NFT_SOCKET_MAX	(__NFT_SOCKET_MAX - 1)
diff --git a/net/netfilter/nft_socket.c b/net/netfilter/nft_socket.c
index 637ce3e8c575..684a7e493f45 100644
--- a/net/netfilter/nft_socket.c
+++ b/net/netfilter/nft_socket.c
@@ -14,6 +14,23 @@ struct nft_socket {
 	};
 };
 
+static void nft_socket_wildcard(const struct nft_pktinfo *pkt,
+				struct nft_regs *regs, struct sock *sk,
+				u32 *dest)
+{
+	switch (nft_pf(pkt)) {
+	case NFPROTO_IPV4:
+		nft_reg_store8(dest, inet_sk(sk)->inet_rcv_saddr == 0);
+		break;
+	case NFPROTO_IPV6:
+		nft_reg_store8(dest, ipv6_addr_any(&sk->sk_v6_rcv_saddr));
+		break;
+	default:
+		regs->verdict.code = NFT_BREAK;
+		return;
+	}
+}
+
 static void nft_socket_eval(const struct nft_expr *expr,
 			    struct nft_regs *regs,
 			    const struct nft_pktinfo *pkt)
@@ -59,6 +76,13 @@ static void nft_socket_eval(const struct nft_expr *expr,
 			return;
 		}
 		break;
+	case NFT_SOCKET_WILDCARD:
+		if (!sk_fullsock(sk)) {
+			regs->verdict.code = NFT_BREAK;
+			return;
+		}
+		nft_socket_wildcard(pkt, regs, sk, dest);
+		break;
 	default:
 		WARN_ON(1);
 		regs->verdict.code = NFT_BREAK;
@@ -97,6 +121,7 @@ static int nft_socket_init(const struct nft_ctx *ctx,
 	priv->key = ntohl(nla_get_u32(tb[NFTA_SOCKET_KEY]));
 	switch(priv->key) {
 	case NFT_SOCKET_TRANSPARENT:
+	case NFT_SOCKET_WILDCARD:
 		len = sizeof(u8);
 		break;
 	case NFT_SOCKET_MARK:
-- 
2.20.1

Re: Fwd: Issue migrating "iptables -m socket --transparent" into nftables

[Nirgal Vourgère]

I should be able to test the whole thing by tomorrow.
You rock guys! :)

On Friday, 21 August 2020 17:23:33 CEST Pablo Neira Ayuso wrote:
> On Fri, Aug 21, 2020 at 05:15:21PM +0200, Balazs Scheidler wrote:
> > Hi,
> > 
> > Here's the accompanying nftables patch, just in case Pablo didn't do it.
> 
> Thanks Balazs, this looks good to me!
> 
> > Pablo do you want me to submit these as a pull request?
> 
> You can just send them via git format-patch to
> netfilter-devel@vger.kernel.org.
> 
> > All I did for testing was that it did compile this ruleset and attempted to
> > submit it via netlink to the kernel, which it refused, as I didn't patch my
> > kernel.
> 
> I'm attaching the kernel patch, compiled-tested only by now.
> 
> > ```
> > table inet haproxy {
> >   chain prerouting {
> >      type filter hook prerouting priority -150; policy accept;
> >      socket transparent 1 socket wildcard 0 mark set 0x00000001
> >    }
> > }
> > ```
> 
> Thanks.
> 

Re: Fwd: Issue migrating "iptables -m socket --transparent" into nftables

[Nirgal Vourgère]

[-- Attachment #1: Type: text/plain, Size: 542 bytes --]

I applied the patches to my kernel and to nftables.

>>> table inet haproxy {
>>>   chain prerouting {
>>>      type filter hook prerouting priority -150; policy accept;
>>>      socket transparent 1 socket wildcard 0 mark set 0x00000001

This works like a charm for ipv4. :)

But ipv6 outbound connections still are grabbed by the socket rather than be routed to the wan and masqueraded.
This works with 
> ip46tables -m socket --transparent -j MARK --set-mark 1

Attached is a more complete extract from my haproxy.cfg, with both v4 and v6.

[-- Attachment #2: haproxy.cfg --]
[-- Type: text/plain, Size: 4198 bytes --]

############################################################################
# DO NOT EDIT THAT FILE
# Notice: That file was generated using:
#    /root/bin/update-virtualhosts /etc/haproxy/virtualhosts.haproxy
# See /etc/haproxy/virtualhosts.haproxy/config
############################################################################
global
	log /dev/log	local0
	log /dev/log	local1 notice
	chroot /var/lib/haproxy
	stats socket /run/haproxy/admin.sock mode 660 level admin
	stats timeout 30s
	# user haproxy  # transparent proxying requires root privileges
	group haproxy
	daemon

	# Default SSL material locations
	ca-base /etc/ssl/certs
	crt-base /etc/ssl/private

	# Default ciphers to use on SSL-enabled listening sockets.
	# For more information, see ciphers(1SSL). This list is from:
	#  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
	# An alternative list with additional directives can be obtained from
	#  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
	ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
	ssl-default-bind-options no-sslv3

defaults
	source 0.0.0.0 usesrc clientip
	log	global
	#mode	http
	#option	httplog
	option	dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
	errorfile 400 /etc/haproxy/errors/400.http
	errorfile 403 /etc/haproxy/errors/403.http
	errorfile 408 /etc/haproxy/errors/408.http
	errorfile 500 /etc/haproxy/errors/500.http
	errorfile 502 /etc/haproxy/errors/502.http
	errorfile 503 /etc/haproxy/errors/503.http
	errorfile 504 /etc/haproxy/errors/504.http

frontend http4-in
        bind :80 transparent
        mode http
        option  httplog
        default_backend http4-www2.in.nirgal.com
        use_backend http4-vcs.in.nirgal.com if { hdr(host) -i vcs.nirgal.com }
        use_backend http4-vcs.in.nirgal.com if { hdr(host) -i svn.nirgal.com }
        use_backend http4-vcs.in.nirgal.com if { hdr(host) -i git.nirgal.com }

frontend https4-in
        bind :443 strict-sni transparent
        mode tcp
        tcp-request inspect-delay 5s
        tcp-request content accept if { req_ssl_hello_type 1 }
        default_backend https4-www2.in.nirgal.com
        use_backend https4-vcs.in.nirgal.com if { req_ssl_sni -i vcs.nirgal.com }
        use_backend https4-vcs.in.nirgal.com if { req_ssl_sni -i svn.nirgal.com }
        use_backend https4-vcs.in.nirgal.com if { req_ssl_sni -i git.nirgal.com }

backend http4-www2.in.nirgal.com
        mode http
        server http4-www2.in.nirgal.com ipv4@192.168.1.99:80

backend http4-vcs.in.nirgal.com
        mode http
        server http4-vcs.in.nirgal.com ipv4@192.168.1.98:80

backend https4-www2.in.nirgal.com
	server https4-www2.in.nirgal.com ipv4@192.168.1.99:443

backend https4-vcs.in.nirgal.com
        server https4-vcs.in.nirgal.com ipv4@192.168.1.98:443

frontend http6-in
        bind :::80 v6only transparent
        mode http
        option  httplog
        default_backend http6-www2.in.nirgal.com
        use_backend http6-vcs.in.nirgal.com if { hdr(host) -i vcs.nirgal.com }
        use_backend http6-vcs.in.nirgal.com if { hdr(host) -i svn.nirgal.com }
        use_backend http6-vcs.in.nirgal.com if { hdr(host) -i git.nirgal.com }

frontend https6-in
        bind :::443 v6only strict-sni transparent
        mode tcp
        tcp-request inspect-delay 5s
        tcp-request content accept if { req_ssl_hello_type 1 }
        use_backend http6-vcs.in.nirgal.com if { hdr(host) -i vcs.nirgal.com }
        use_backend http6-vcs.in.nirgal.com if { hdr(host) -i svn.nirgal.com }
        use_backend http6-vcs.in.nirgal.com if { hdr(host) -i git.nirgal.com }

backend http6-www2.in.nirgal.com
        mode http
        server http6-www2.in.nirgal.com ipv6@[fd77:7777:7777::99]:80

backend http6-vcs.in.nirgal.com
        mode http
        server http6-vcs.in.nirgal.com ipv6@[fd77:7777:7777::98]:80

backend https6-www2.in.nirgal.com
        server https6-www2.in.nirgal.com ipv6@[fd77:7777:7777::99]:443

backend https6-vcs.in.nirgal.com
        server https6-vcs.in.nirgal.com ipv6@[fd77:7777:7777::98]:443

Re: Fwd: Issue migrating "iptables -m socket --transparent" into nftables

[Balazs Scheidler]

On Sat, Aug 22, 2020 at 08:45:22AM +0200, Balazs Scheidler wrote:
> hmm.. judging the code alone, I can't see the difference between xt_socket
> and nft_socket, they are checking the same things for ipv6.
> 
> I am not sure when sk->sk_v6_rcv_saddr is properly set, if that's non-zero
> we would yield "socket wildcard 0" and cause a match.
> 
> I can see that __inet6_bind() sets this properly, so as long as haproxy
> binds it to port 80/443 we should be fine.
> 
> I still cannot get what you mean under "But ipv6 outbound connections still
> are grabbed by the socket rather than be routed to the wan and
> masqueraded." exactly.
> 
> 1) haproxy binds to [::0]:80 and I assume it does that to receive external
> traffic. Do you use tproxy rules to redirect traffic here?
> 2) then haproxy would establish a connection from [clientip]:randomport ->
> internal server:80. The return traffic should be matched by "socket
> transparent 1 socket wildcard 0" and redirected as such.
> 
> So which of the two connections above is what you mean?

The question still stands, which of the two connection is getting the wrong
treatment?

@Pablo: do you want me to test/push the kernel piece to netfilter-devel?
Also, I have a usability question in the email I sent there.

Hopefully, by now I can send email to vger :)

Cheers,
-- 
Bazsi

Re: Fwd: Issue migrating "iptables -m socket --transparent" into nftables

[Nirgal Vourgère]

My apologies for the delay.
I ran into an *unrelated* issue, that troubled ipv6 outbound (masqueraded) connections.

I did some more tests with your patches for linux & nftables.

All look good, both ipv4 & v6, for both inbound (redirected to the socket) and outbound (masqueraded) connections.

Thank you again! :)

On Tuesday, 25 August 2020 11:45:30 CEST Balazs Scheidler wrote:
> On Sat, Aug 22, 2020 at 08:45:22AM +0200, Balazs Scheidler wrote:
> > hmm.. judging the code alone, I can't see the difference between xt_socket
> > and nft_socket, they are checking the same things for ipv6.
> > 
> > I am not sure when sk->sk_v6_rcv_saddr is properly set, if that's non-zero
> > we would yield "socket wildcard 0" and cause a match.
> > 
> > I can see that __inet6_bind() sets this properly, so as long as haproxy
> > binds it to port 80/443 we should be fine.
> > 
> > I still cannot get what you mean under "But ipv6 outbound connections still
> > are grabbed by the socket rather than be routed to the wan and
> > masqueraded." exactly.
> > 
> > 1) haproxy binds to [::0]:80 and I assume it does that to receive external
> > traffic. Do you use tproxy rules to redirect traffic here?
> > 2) then haproxy would establish a connection from [clientip]:randomport ->
> > internal server:80. The return traffic should be matched by "socket
> > transparent 1 socket wildcard 0" and redirected as such.
> > 
> > So which of the two connections above is what you mean?
> 
> The question still stands, which of the two connection is getting the wrong
> treatment?
> 
> @Pablo: do you want me to test/push the kernel piece to netfilter-devel?
> Also, I have a usability question in the email I sent there.
> 
> Hopefully, by now I can send email to vger :)
> 
> Cheers,
>