What wrong with snat in nftables?

What wrong with snat in nftables?

[sorcus]

I have two virtual machines (server, client) with wireguard vpn.
When i try ping any IPv6 resource from client packets doesn't return to 
client.
Tcpdump show me ICMP Reply packets in enp0s3 interface (server), not in 
wg0 (vpn interface on server).
But if disable nftables and start ip6tables, all works.
After this step i disable ip6tables and enable nftables... All continue 
works...

Software versions:
NixOS: 17.09.git.ebaff59 (Hummingbird)
WireGuard: 0.0.20170706
Nftables: 0.7

Build ISO images with next commands:
Server: nix-build -A config.system.build.isoImage -I 
nixos-config=./wireguard_server_10.nix ./nixpkgs/nixos/default.nix
Client: nix-build -A config.system.build.isoImage -I 
nixos-config=./wireguard_client_20.nix ./nixpkgs/nixos/default.nix

Here nix files - 
https://gist.github.com/MrSorcus/d6d8b8b6acff715368844a643775c980

Create virtual machines with next commands:

Server: virt-install \
--name NixOSVS10 \
--ram 1024 \
--vcpus 1 \
--cdrom /tmp/nixos_10.iso \
--os-type linux \
--nodisk \
--network bridge=br0 \
--graphics vnc,password="ABCDEF",port=5910,listen=2a01:4f8:xx:xx::13 \
--autostart \
--noautoconsole

Client: virt-install \
--name NixOSVS20 \
--ram 1024 \
--vcpus 1 \
--cdrom /tmp/nixos_20.iso \
--os-type linux \
--nodisk \
--network bridge=br0 \
--graphics vnc,password="ABCDEF",port=5920,listen=2a01:4f8:xx:xx::13 \
--autostart \
--noautoconsole

Output for ip a, ip -6 route, route -6, wg, sysctl -a, dmesg, lsmod.

Server: 
https://gist.github.com/MrSorcus/1a8c9f5aacf8957502299d707a38c5fc
Client: 
https://gist.github.com/MrSorcus/b7dc077249ca513ca8f307a68c62d1ce

Tcpdump logs from client. Ping IPv6 address 2001:19f0:7400:87a2::64 
(https://ipv6.net/)
https://gist.github.com/MrSorcus/03e716fba67c4119772012777847c569

Output from /proc/net/nf_conntrack:
With nftables - 
https://gist.github.com/MrSorcus/601170680ff644c52a11e5352997879a
With ip6tables - 
https://gist.github.com/MrSorcus/e043101f98e787c8cbf6d0605fd9de7e

Snat doesn't work correctly in nftables. But work after next steps:
[root@nixos:~]# systemctl stop nftables
[root@nixos:~]# ip6tables -t nat -A POSTROUTING -o enp0s3 -j SNAT 
--to-source 2a01:4f8:xx:xx::10

Re: What wrong with snat in nftables?

[Anton Danilov]

Hello.
I haven't seen the nftables rules.

Seems like you have misconfiguration in the nftables rules.

NAT action are being done in the conntrack subsystem, and *tables just
apply the rule for first packets of NEW state connection and create
special conntrack entries. Other packets of this known connection
bypass the rules in the nat table. So it caused working after moving
back from ip6tables to nftables, because it hasn't affected the
conntrack entries.

So, these are two recommends:

1. flush conntrack entries after changing of rules in the nat table.
2. add counters to nftables nat rule to check that rule really hit the packets.



2017-07-10 16:36 GMT+03:00  <sorcus@inwebse.com>:
> I have two virtual machines (server, client) with wireguard vpn.
> When i try ping any IPv6 resource from client packets doesn't return to
> client.
> Tcpdump show me ICMP Reply packets in enp0s3 interface (server), not in wg0
> (vpn interface on server).
> But if disable nftables and start ip6tables, all works.
> After this step i disable ip6tables and enable nftables... All continue
> works...
>
> Software versions:
> NixOS: 17.09.git.ebaff59 (Hummingbird)
> WireGuard: 0.0.20170706
> Nftables: 0.7
>
> Build ISO images with next commands:
> Server: nix-build -A config.system.build.isoImage -I
> nixos-config=./wireguard_server_10.nix ./nixpkgs/nixos/default.nix
> Client: nix-build -A config.system.build.isoImage -I
> nixos-config=./wireguard_client_20.nix ./nixpkgs/nixos/default.nix
>
> Here nix files -
> https://gist.github.com/MrSorcus/d6d8b8b6acff715368844a643775c980
>
> Create virtual machines with next commands:
>
> Server: virt-install \
> --name NixOSVS10 \
> --ram 1024 \
> --vcpus 1 \
> --cdrom /tmp/nixos_10.iso \
> --os-type linux \
> --nodisk \
> --network bridge=br0 \
> --graphics vnc,password="ABCDEF",port=5910,listen=2a01:4f8:xx:xx::13 \
> --autostart \
> --noautoconsole
>
> Client: virt-install \
> --name NixOSVS20 \
> --ram 1024 \
> --vcpus 1 \
> --cdrom /tmp/nixos_20.iso \
> --os-type linux \
> --nodisk \
> --network bridge=br0 \
> --graphics vnc,password="ABCDEF",port=5920,listen=2a01:4f8:xx:xx::13 \
> --autostart \
> --noautoconsole
>
> Output for ip a, ip -6 route, route -6, wg, sysctl -a, dmesg, lsmod.
>
> Server: https://gist.github.com/MrSorcus/1a8c9f5aacf8957502299d707a38c5fc
> Client: https://gist.github.com/MrSorcus/b7dc077249ca513ca8f307a68c62d1ce
>
> Tcpdump logs from client. Ping IPv6 address 2001:19f0:7400:87a2::64
> (https://ipv6.net/)
> https://gist.github.com/MrSorcus/03e716fba67c4119772012777847c569
>
> Output from /proc/net/nf_conntrack:
> With nftables -
> https://gist.github.com/MrSorcus/601170680ff644c52a11e5352997879a
> With ip6tables -
> https://gist.github.com/MrSorcus/e043101f98e787c8cbf6d0605fd9de7e
>
> Snat doesn't work correctly in nftables. But work after next steps:
> [root@nixos:~]# systemctl stop nftables
> [root@nixos:~]# ip6tables -t nat -A POSTROUTING -o enp0s3 -j SNAT
> --to-source 2a01:4f8:xx:xx::10
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Anton.

Re: What wrong with snat in nftables?

[sorcus]

Nftables rules for server -
https://gist.github.com/MrSorcus/17827298ad81970cc272b9e7f779923f

Nftables rules for client -
https://gist.github.com/MrSorcus/ad05f19759e02f92aac5d9c246f56cf6

I used information from here -
https://wiki.nftables.org/wiki-nftables/index.php/Performing_Network_Address_Translation_(NAT)

> 1. flush conntrack entries after changing of rules in the nat table.

"conntrack -D" - show me "0 flow entries have been deleted.". But
"/proc/net/nf_conntrack" has 3 entries...

> 2. add counters to nftables nat rule to check that rule really hit the 
> packets.

chain postrouting {
   type nat hook postrouting priority 0; policy drop;
   oif "enp0s3" nftrace set 1 counter packets 3 bytes 312 snat to 
2a01:4f8:xx:xx::10
   nftrace set 1 counter packets 0 bytes 0 accept comment "IPv6 < Nat < 
Postrouting"
}

With git version of nftables, libmnl & libnftnl and kernels 4.9, 4.11 & 
4.12 doesn't work...
With archlinux server doesn't work too...


> On 2017-07-11 10:10, Anton Danilov wrote:
>> Hello.
>> I haven't seen the nftables rules.
>> 
>> Seems like you have misconfiguration in the nftables rules.
>> 
>> NAT action are being done in the conntrack subsystem, and *tables just
>> apply the rule for first packets of NEW state connection and create
>> special conntrack entries. Other packets of this known connection
>> bypass the rules in the nat table. So it caused working after moving
>> back from ip6tables to nftables, because it hasn't affected the
>> conntrack entries.
>> 
>> So, these are two recommends:
>> 
>> 1. flush conntrack entries after changing of rules in the nat table.
>> 2. add counters to nftables nat rule to check that rule really hit the 
>> packets.

Re: What wrong with snat in nftables?

[sorcus]

Output for command nft --debug all -f ruleset

https://gist.github.com/MrSorcus/2c8c65461e3c65fb70364f3a70d95439


On 2017-07-14 01:06, sorcus@inwebse.com wrote:
> Nftables rules for server -
> https://gist.github.com/MrSorcus/17827298ad81970cc272b9e7f779923f
> 
> Nftables rules for client -
> https://gist.github.com/MrSorcus/ad05f19759e02f92aac5d9c246f56cf6
> 
> I used information from here -
> https://wiki.nftables.org/wiki-nftables/index.php/Performing_Network_Address_Translation_(NAT)
> 
>> 1. flush conntrack entries after changing of rules in the nat table.
> 
> "conntrack -D" - show me "0 flow entries have been deleted.". But
> "/proc/net/nf_conntrack" has 3 entries...
> 
>> 2. add counters to nftables nat rule to check that rule really hit the 
>> packets.
> 
> chain postrouting {
>   type nat hook postrouting priority 0; policy drop;
>   oif "enp0s3" nftrace set 1 counter packets 3 bytes 312 snat to
> 2a01:4f8:xx:xx::10
>   nftrace set 1 counter packets 0 bytes 0 accept comment "IPv6 < Nat <
> Postrouting"
> }
> 
> With git version of nftables, libmnl & libnftnl and kernels 4.9, 4.11
> & 4.12 doesn't work...
> With archlinux server doesn't work too...

Re: What wrong with snat in nftables?

[sorcus]

Solved. http://marc.info/?l=netfilter-devel&m=150027256708621&w=2

You are probably lacking the reply NAT chain, which needs to be 
registered.
https://wiki.nftables.org/wiki-nftables/index.php/Performing_Network_Address_Translation_(NAT)
I'm updating right now the wiki to put this in bold.


On 2017-07-15 22:47, sorcus@inwebse.com wrote:
> Output for command nft --debug all -f ruleset
> 
> https://gist.github.com/MrSorcus/2c8c65461e3c65fb70364f3a70d95439