nftables iifname and currently unknown interfaces

nftables iifname and currently unknown interfaces

[Robert Sander]

[-- Attachment #1.1: Type: text/plain, Size: 738 bytes --]

Hi.

with iptables it was possible to specify "-i ifacename" even when the
interface was currently not available.

nft bails out with an error:

./nft:225:1-75: Error: Could not process rule: No such file or directory
add rule ip filter FORWARD iifname bond0.16 oifname bond0.42 accept

We are generating a single firewall configuration for a number of
firewalls with different interfaces. How do we migrate to nftables?

Regards
-- 
Robert Sander
Heinlein Support GmbH
Schwedter Str. 8/9b, 10119 Berlin

http://www.heinlein-support.de

Tel: 030 / 405051-43
Fax: 030 / 405051-19

Zwangsangaben lt. §35a GmbHG:
HRB 93818 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: nftables iifname and currently unknown interfaces

[Pablo Neira Ayuso]

On Fri, Oct 16, 2020 at 12:37:58PM +0200, Robert Sander wrote:
> Hi.
> 
> with iptables it was possible to specify "-i ifacename" even when the
> interface was currently not available.
> 
> nft bails out with an error:
> 
> ./nft:225:1-75: Error: Could not process rule: No such file or directory
> add rule ip filter FORWARD iifname bond0.16 oifname bond0.42 accept
>
> We are generating a single firewall configuration for a number of
> firewalls with different interfaces. How do we migrate to nftables?

Strange, that rule works fine here and I don't have such device.

iifname allows you match on the device name, so such interface does
not need to be available.

What nft version and kernel are you using there?

Error: Could not process rule: No such file or directory
add rule ip filter FORWARD iifname bond0.16 oifname bond0.42 accept
            ^^^^^^

With relatively recent nft userspace and kernel, you should get
context on why the ENOENT error is displayed.

Either by missing table like above, or missing chain:

Error: Could not process rule: No such file or directory
add rule ip filter FORWARD iifname bond0.16 oifname bond0.42 accept
                   ^^^^^^^

What nft version are you using?

Then, moving forward, a general error means that some of your kernel
components in missing. Did you compile kernel, if so, could you also
post your .config file for your kernel?

Re: nftables iifname and currently unknown interfaces

[Florian Westphal]

Robert Sander <r.sander@heinlein-support.de> wrote:
> Hi.
> 
> with iptables it was possible to specify "-i ifacename" even when the
> interface was currently not available.
> 
> nft bails out with an error:
> 
> ./nft:225:1-75: Error: Could not process rule: No such file or directory
> add rule ip filter FORWARD iifname bond0.16 oifname bond0.42 accept

This error is not related to the interface names.

nft should annotate this error with ^^^^ markers with more recent
version, e.g.:

nft add rule inet filter forwad iifname bond0.16 oifname bond0.42 accept
Error: Could not process rule: No such file or directory
add rule inet filter forwad iifname bond0.16 oifname bond0.42 accept
                     ^^^^^^
(typo in chain name).

Re: nftables iifname and currently unknown interfaces

[Robert Sander]

[-- Attachment #1.1: Type: text/plain, Size: 773 bytes --]

Am 16.10.20 um 12:56 schrieb Florian Westphal:

> nft add rule inet filter forwad iifname bond0.16 oifname bond0.42 accept
> Error: Could not process rule: No such file or directory
> add rule inet filter forwad iifname bond0.16 oifname bond0.42 accept
>                      ^^^^^^
> (typo in chain name).

Oh, how embarrassing. :)

The rule originally contained a jump to a chain that wa slater declared
in the nft commands. Bummer. Sorry for the noise.

Regards
-- 
Robert Sander
Heinlein Support GmbH
Schwedter Str. 8/9b, 10119 Berlin

http://www.heinlein-support.de

Tel: 030 / 405051-43
Fax: 030 / 405051-19

Zwangsangaben lt. §35a GmbHG:
HRB 93818 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: nftables iifname and currently unknown interfaces

[Pablo Neira Ayuso]

On Fri, Oct 16, 2020 at 01:10:21PM +0200, Robert Sander wrote:
> Am 16.10.20 um 12:56 schrieb Florian Westphal:
> 
> > nft add rule inet filter forwad iifname bond0.16 oifname bond0.42 accept
> > Error: Could not process rule: No such file or directory
> > add rule inet filter forwad iifname bond0.16 oifname bond0.42 accept
> >                      ^^^^^^
> > (typo in chain name).
> 
> Oh, how embarrassing. :)
> 
> The rule originally contained a jump to a chain that wa slater declared
> in the nft commands. Bummer. Sorry for the noise.

For the record, the most recent nftables (0.9.7) with recent kernel
versions provide a hint for this error:

# nft add rule x y jump test
Error: Could not process rule: No such file or directory
add rule x y jump test
                  ^^^^

Trying to provision flowtable returns error

[Martin Gignac]

Hi,

I want to try out flowtables and I am trying to create a new flow
table (based on
https://www.kernel.org/doc/Documentation/networking/nf_flowtable.txt)
by inserting the below statement in my 'firewall.nft' file:

----------
table inet filter {

  flowtable f {
    hook ingress priority 0; devices = { br0, br1 };
  }

  chain input {
    type filter hook input priority 0; policy drop;

[remainder removed]
----------

When I run 'nft -c -f firewall.nft' it returns the following error:

----------
/etc/firewall.nft:10:13-13: Error: Could not process rule: No such
file or directory
  flowtable f {
----------

I haven't been able to figure out what's wrong about my syntax so far.

Any hints?

Thanks,
-Martin

Re: Trying to provision flowtable returns error

[Duncan Roe]

Hi Martin,

On Wed, Nov 04, 2020 at 12:30:31AM -0500, Martin Gignac wrote:
> Hi,
>
> I want to try out flowtables and I am trying to create a new flow
> table (based on
> https://www.kernel.org/doc/Documentation/networking/nf_flowtable.txt)
> by inserting the below statement in my 'firewall.nft' file:
>
> ----------
> table inet filter {
>
>   flowtable f {
>     hook ingress priority 0; devices = { br0, br1 };
>   }
>
>   chain input {
>     type filter hook input priority 0; policy drop;
>
> [remainder removed]
> ----------
>
> When I run 'nft -c -f firewall.nft' it returns the following error:
>
> ----------
> /etc/firewall.nft:10:13-13: Error: Could not process rule: No such
> file or directory
>   flowtable f {
> ----------
>
> I haven't been able to figure out what's wrong about my syntax so far.
>
> Any hints?
>
> Thanks,
> -Martin

"No such file or directory" refers to something that is missing, rather than to
command syntax.

What may be missing is a required kernel module. Does 'lsmod' show
'nf_flow_table' and 'nf_flow_table_inet'?

Also, what is you kernel version?

Cheers ... Duncan.

Re: Trying to provision flowtable returns error

[Martin Gignac]

Hi Duncan,


> What may be missing is a required kernel module. Does 'lsmod' show
> 'nf_flow_table' and 'nf_flow_table_inet'?
>
> Also, what is you kernel version?

I actually observed the issue on both a Fedora 30 Server node (kernel
5.6.13-100.fc30.x86_64) and a RHEL8 node (kernel
4.18.0-147.8.1.el8_1.x86_64), but just to be sure I just installed a
brand new Fedora 33 Server VM with the latest DNF updates and tried
again using the example at
https://www.kernel.org/doc/Documentation/networking/nf_flowtable.txt
verbatim:

    [root@localhost ~]# cat toto.nft
    table inet x {
    flowtable f {
        hook ingress priority 0; devices = { eth0, eth1 };
    }
            chain y {
                    type filter hook forward priority 0; policy accept;
                    ip protocol tcp flow offload @f
                    counter packets 0 bytes 0
            }
    }

Still the same error:

    [root@localhost ~]# nft -c -f toto.nft
    toto.nft:2:11-11: Error: Could not process rule: No such file or directory
    flowtable f {
            ^
    toto.nft:7:17-47: Error: Could not process rule: No such file or directory
                    ip protocol tcp flow offload @f
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Loaded modules:

    [root@localhost ~]# lsmod | grep nf
    nft_ct                 20480  0
    nft_counter            16384  0
    nft_flow_offload       16384  0
    nf_flow_table_inet     16384  0
    nf_flow_table          40960  2 nft_flow_offload,nf_flow_table_inet
    nf_conntrack          163840  3 nft_flow_offload,nft_ct,nf_flow_table
    nf_defrag_ipv6         24576  1 nf_conntrack
    nf_defrag_ipv4         16384  1 nf_conntrack
    nf_tables             237568  4
nft_flow_offload,nft_ct,nft_counter,nf_flow_table_inet
    nfnetlink              16384  1 nf_tables

Kernel version:

    [root@localhost ~]# uname -a
    Linux localhost.localdomain 5.8.17-300.fc33.x86_64 #1 SMP Thu Oct
29 15:55:40 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

NFT version:

    [root@localhost ~]# nft -v
    nftables v0.9.3 (Topsy)

I'm sure I'm missing something, but I can't tell what...

Thanks,
-Martin

Re: Trying to provision flowtable returns error

[Florian Westphal]

Martin Gignac <martin.gignac@gmail.com> wrote:
> Hi Duncan,
> 
> 
> > What may be missing is a required kernel module. Does 'lsmod' show
> > 'nf_flow_table' and 'nf_flow_table_inet'?
> >
> > Also, what is you kernel version?
> 
> I actually observed the issue on both a Fedora 30 Server node (kernel
> 5.6.13-100.fc30.x86_64) and a RHEL8 node (kernel
> 4.18.0-147.8.1.el8_1.x86_64), but just to be sure I just installed a
> brand new Fedora 33 Server VM with the latest DNF updates and tried
> again using the example at
> https://www.kernel.org/doc/Documentation/networking/nf_flowtable.txt
> verbatim:
> 
>     [root@localhost ~]# cat toto.nft
>     table inet x {
>     flowtable f {
>         hook ingress priority 0; devices = { eth0, eth1 };

eth0 and eth1 must exist.

>     Linux localhost.localdomain 5.8.17-300.fc33.x86_64 #1 SMP Thu Oct
> 29 15:55:40 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Fedora 33 works fine, i tested it a minute ago.

Re: Trying to provision flowtable returns error

[Martin Gignac]

Hi Florian,

OK, I understand now.

- on RHEL/CentOS 8 I *was* using existing network interfaces, but
there are *no* nf_flow kernel modules available, and
- on Fedora 30/33 there *are* nf_flow kernel modules available, but I
was *not* using existing interface names

Both cases give off the same error message, which confused me.

I guess then this makes it problematic to use logical interfaces such
as bonds or bridges with flow tables, correct? In Fedora the systemd
unit file for the nftables.services states that nftables should start
before the network.pre-target:

    [root@localhost system]# cat nftables.service
    [Unit]
    Description=Netfilter Tables
    Documentation=man:nft(8)
    Wants=network-pre.target
    Before=network-pre.target

    [Service]
    Type=oneshot
    ProtectSystem=full
    ProtectHome=true
    ExecStart=/sbin/nft -f /etc/sysconfig/nftables.conf
    ExecReload=/sbin/nft 'flush ruleset; include
"/etc/sysconfig/nftables.conf";'
    ExecStop=/sbin/nft flush ruleset
    RemainAfterExit=yes

    [Install]
    WantedBy=multi-user.target

At this stage Network Manager hasn't even begun to initialize, much
less create any logical interfaces. Therefore any bond or bridge
logical interface do not yet exist, and if they are referred to in the
*.nft files the nftables.service will simply fail with the "Error:
Could not process rule: No such file or directory" message.

Ideally I'd want to start nftables before the network comes online to
make sure everything is secure, but if I use logical interfaces is my
only recourse to change the nftables.service file so that nftables
only starts up after the network-online.target, which would open a
very short window where the network is up but not yet protected by
nftables yet? Or is there another way that I am not seeing?

Thanks,
-Martin

Re: Trying to provision flowtable returns error

[Florian Westphal]

Martin Gignac <martin.gignac@gmail.com> wrote:
> Hi Florian,
> 
> OK, I understand now.
> 
> - on RHEL/CentOS 8 I *was* using existing network interfaces, but
> there are *no* nf_flow kernel modules available, and
> - on Fedora 30/33 there *are* nf_flow kernel modules available, but I
> was *not* using existing interface names
> 
> Both cases give off the same error message, which confused me.

Yes, I agree its confusing. In first case its because module isn't found and
in second case its the interface names that could not be found.

> In Fedora the systemd
> unit file for the nftables.services states that nftables should start
> before the network.pre-target:
[..]

> At this stage Network Manager hasn't even begun to initialize, much
> less create any logical interfaces.

Sure.  I think the systemd default is sane though.

> Or is there another way that I am not seeing?

Can't nm execute some action when an interface comes up?
That way you could just add the flowtable later.

Re: Trying to provision flowtable returns error

[Martin Gignac]

> Can't nm execute some action when an interface comes up?

Looks like there might be:
(https://askubuntu.com/questions/1111652/network-manager-script-when-interface-up).

> That way you could just add the flowtable later.

However, this would only insert the flow table statements on server
bootup. Since '/etc/nftables/firewall.nft' ttself *wouldn't* contain
the flow tables statements, any 'systemctl reload nftables' or 'nft -f
/etc/nftables/firewall.nft' action (to apply a rule change, for
example) would essentially get rid of the flow tables mechanism from
the running system, wouldn't it?

-Martin

Re: Trying to provision flowtable returns error

[Martin Gignac]

> However, this would only insert the flow table statements on server
> bootup. Since '/etc/nftables/firewall.nft' ttself *wouldn't* contain
> the flow tables statements, any 'systemctl reload nftables' or 'nft -f
> /etc/nftables/firewall.nft' action (to apply a rule change, for
> example) would essentially get rid of the flow tables mechanism from
> the running system, wouldn't it?

I guess there's no "equivalent" of iifname/oifname for flow table
devices where you could refer to a device that does not (yet) exist?

-Martin

Re: Trying to provision flowtable returns error

[Pablo Neira Ayuso]

On Thu, Nov 05, 2020 at 01:41:53PM -0500, Martin Gignac wrote:
> > However, this would only insert the flow table statements on server
> > bootup. Since '/etc/nftables/firewall.nft' ttself *wouldn't* contain
> > the flow tables statements, any 'systemctl reload nftables' or 'nft -f
> > /etc/nftables/firewall.nft' action (to apply a rule change, for
> > example) would essentially get rid of the flow tables mechanism from
> > the running system, wouldn't it?
> 
> I guess there's no "equivalent" of iifname/oifname for flow table
> devices where you could refer to a device that does not (yet) exist?

You can dynamically add/delete devices to/from flowtables since Linux
kernel 5.8

Re: Trying to provision flowtable returns error

[Martin Gignac]

Hi Pablo,

> You can dynamically add/delete devices to/from flowtables since Linux
> kernel 5.8

Are you referring to this patch ?:
https://www.spinics.net/lists/netfilter-devel/msg67310.html

I tried with Fedora 33 (5.8.17-300.fc33.x86_64) and this file:

    [root@localhost ~]# cat /etc/nftables/firewall.nft
    flush ruleset

    table inet x {
        flowtable f {
                hook ingress priority 0;
    }
            chain y {
                    type filter hook forward priority 0; policy accept;
                    ip protocol tcp flow offload @f
                    counter packets 0 bytes 0
            }
    }

and indeed it does load without error, although I had to compile the
latest version of nft (v0.9.7) as v0.9.3 (which comes with Fedora 33)
was giving me this error:

    [root@localhost ~]# nft -f /etc/nftables/firewall.nft
    /etc/nftables/firewall.nft:4:12-12: Error: Unbound flowtable not
allowed (must specify devices)
        flowtable f {
                ^
Once I added my br0 interface wih 'ip link add br0 type bridge' I was
able to run 'nft add flowtable inet x f { devices = { br0 } \; }'
without error.

However, if I run 'nft -f /etc/nftables/firewall.nft' again and then
'nft list ruleset', br0 is gone. Does this mean that it is no longer
bound to a flow table?

The way I have been handling rule changes so far is to modify a single
'/etc/nftables/firewall.nft' file every time I need to modify rules
and then run 'nft -f /etc/nftables/firewall.nft' to reload and apply
those changes (I don't tend to run single nft commands to update
things here and there -- I prefer to modify a single file as the
source of truth and then reload the ruleset completely). Running
something like 'nft add flowtable inet x f { devices = { br0 } \; }'
once upon boot up when a logical interface comes up is fine, but does
my workflow require that I then run 'nft add flowtable inet x f {
devices = { br0 } \; }' after every time I run 'nft -f
/etc/nftables/firewall.nft'?

Is my workflow wrong? It is based on my experience with pf/pf.conf
from OpenBSD/FreeBSD where all changes to rules are made in one file
which is then reloaded atomically. Has my familiarity with OpenBSD
steered me wrong with regards to nftables?

Thanks,
-Martin

Re: Trying to provision flowtable returns error

[Pablo Neira Ayuso]

Hi,

This works fine here, see below.

On Thu, Nov 05, 2020 at 04:45:31PM -0500, Martin Gignac wrote:
> Hi Pablo,
> 
> > You can dynamically add/delete devices to/from flowtables since Linux
> > kernel 5.8
> 
> Are you referring to this patch ?:
> https://www.spinics.net/lists/netfilter-devel/msg67310.html
> 
> I tried with Fedora 33 (5.8.17-300.fc33.x86_64) and this file:
> 
>     [root@localhost ~]# cat /etc/nftables/firewall.nft
>     flush ruleset
> 
>     table inet x {
>         flowtable f {
>                 hook ingress priority 0;
>     }
>             chain y {
>                     type filter hook forward priority 0; policy accept;
>                     ip protocol tcp flow offload @f
>                     counter packets 0 bytes 0
>             }
>     }
> 
> and indeed it does load without error, although I had to compile the
> latest version of nft (v0.9.7) as v0.9.3 (which comes with Fedora 33)
> was giving me this error:
> 
>     [root@localhost ~]# nft -f /etc/nftables/firewall.nft
>     /etc/nftables/firewall.nft:4:12-12: Error: Unbound flowtable not
> allowed (must specify devices)
>         flowtable f {
>                 ^
> Once I added my br0 interface wih 'ip link add br0 type bridge' I was
> able to run 'nft add flowtable inet x f { devices = { br0 } \; }'
> without error.
>
> However, if I run 'nft -f /etc/nftables/firewall.nft' again and then
> 'nft list ruleset', br0 is gone. Does this mean that it is no longer
> bound to a flow table?
> 
> The way I have been handling rule changes so far is to modify a single
> '/etc/nftables/firewall.nft' file every time I need to modify rules
> and then run 'nft -f /etc/nftables/firewall.nft' to reload and apply
> those changes (I don't tend to run single nft commands to update
> things here and there -- I prefer to modify a single file as the
> source of truth and then reload the ruleset completely). Running
> something like 'nft add flowtable inet x f { devices = { br0 } \; }'
> once upon boot up when a logical interface comes up is fine, but does
> my workflow require that I then run 'nft add flowtable inet x f {
> devices = { br0 } \; }' after every time I run 'nft -f
> /etc/nftables/firewall.nft'?

# cat firewall.nft
table ip x {
        flowtable y {
                hook ingress priority filter
        }

        chain y {
                flow add @y
        }
}
# nft -f firewall.nft
# nft list ruleset
table ip x {
        flowtable y {
                hook ingress priority filter
        }

        chain y {
                flow add @y
        }
}

This is your base ruleset.

Now you add devices to the flowtable (requirements: kernel >= 5.8
and nftables >= 0.9.7):

# nft add flowtable x y { devices = { eth0, eth1 } \; }

Listing shows:

# nft list ruleset
table ip x {
        flowtable y {
                hook ingress priority filter
                devices = { eth0, eth1 }
        }

        chain y {
                flow add @y
        }
}

Note: If eth0 is gone, then this is automatically removed from the
flowtable.

Is "flush ruleset" at the very beginning of your firewall.nft file?

If so, that is tearing down everything and creating it from scratch,
so the devices you have dynamically added are gone since they are not
in the original firewall.nft file.

I would expect you load firewall.nft at boot time, then dynamically
add devices as needed in run time.

Re: Trying to provision flowtable returns error

[Martin Gignac]

Hi,

> Is "flush ruleset" at the very beginning of your firewall.nft file?

Yes it is, since when I run 'nft -f /etc/nftables/firewall.nft' I
don't want to end up with duplicate rules.

Even if I didn't explicitly put the 'flush statement' at the top of my
/etc/nftables/firewall.nft file, the systemd nftables.service file in
Fedora/RHEL/CentOS also flushes the ruleset before re-applying the
rules when running 'systemctl reload nftables', which I have always
perceived as a proper way to pull in firewall rule changes:

    [root@localhost system]# grep Reload
/usr/lib/systemd/system/nftables.service
    ExecReload=/sbin/nft 'flush ruleset; include
"/etc/sysconfig/nftables.conf";'

My whole way of thinking about atomic rule replacement is based on
https://wiki.nftables.org/wiki-nftables/index.php/Atomic_rule_replacement,
where it states that: "If you prepend the "flush table filter" line at
the very beginning of the filter-table file, you achieve atomic
rule-set replacement equivalent to what iptables-restore provides. The
kernel handles the rule commands in the file in one single
transaction, so basically the flushing and the load of the new rules
happens in one single shot. If you choose not to flush your tables
then you will see duplicate rules for each time you reloaded the
config."

Granted, I am have been using "flush ruleset" at the beginning of the
file, instead of the stated "flush table filter". However, since I may
have multiple tables in the file (filter, nat, etc.) with
indeterminate names, I went and assumed that "flush ruleset" was an
all-encompassing alternative to individual "flush table <tablename>"
statements.

That being said, I ran a test by replacing "flush ruleset" at the top
of my firewall.nft file with "flush table inet x" and I was surprised
to discover that, while "flush table inet x" *does* remove all
configured rules, thereby preventing duplicate rules, it does _not_
remove the devices from the flow table.

So:

    [root@localhost system]# cat /etc/nftables/firewall.nft
    flush table inet x

    table inet x {
        flowtable f {
                hook ingress priority 0;
    }
            chain y {
                    type filter hook forward priority 0; policy accept;
                    ip protocol tcp flow offload @f
                    counter packets 0 bytes 0
            tcp dport 80
            }
    }
    [root@localhost system]# nft -f /etc/nftables/firewall.nft
    [root@localhost system]# nft list ruleset
    table inet x {
        flowtable f {
            hook ingress priority filter
        }

        chain y {
            type filter hook forward priority filter; policy accept;
            ip protocol tcp flow add @f
            counter packets 0 bytes 0
            tcp dport 80
        }
    }
    [root@localhost system]# nft add flowtable inet x f { hook ingress
priority filter \; devices = { br0 } \; }
    [root@localhost system]# nft list ruleset
    table inet x {
        flowtable f {
            hook ingress priority filter
            devices = { br0 }
        }

        chain y {
            type filter hook forward priority filter; policy accept;
            ip protocol tcp flow add @f
            counter packets 0 bytes 0
            tcp dport 80
        }
    }
    [root@localhost system]# nft -f /etc/nftables/firewall.nft
    [root@localhost system]# nft list ruleset
    table inet x {
        flowtable f {
            hook ingress priority filter
            devices = { br0 }
        }

        chain y {
            type filter hook forward priority filter; policy accept;
            ip protocol tcp flow add @f
            counter packets 0 bytes 0
            tcp dport 80
        }
    }

Whereas:

    [root@localhost system]# cat /etc/nftables/firewall.nft
    flush ruleset

    table inet x {
        flowtable f {
                hook ingress priority 0;
    }
            chain y {
                    type filter hook forward priority 0; policy accept;
                    ip protocol tcp flow offload @f
                    counter packets 0 bytes 0
            tcp dport 80
            }
    }
    [root@localhost system]# nft -f /etc/nftables/firewall.nft
    [root@localhost system]# nft list ruleset
    table inet x {
        flowtable f {
            hook ingress priority filter
        }

        chain y {
            type filter hook forward priority filter; policy accept;
            ip protocol tcp flow add @f
            counter packets 0 bytes 0
            tcp dport 80
        }
    }
    [root@localhost system]# nft add flowtable inet x f { hook ingress
priority filter \; devices = { br0 } \; }
    [root@localhost system]# nft list ruleset
    table inet x {
        flowtable f {
            hook ingress priority filter
            devices = { br0 }
        }

        chain y {
            type filter hook forward priority filter; policy accept;
            ip protocol tcp flow add @f
            counter packets 0 bytes 0
            tcp dport 80
        }
    }
    [root@localhost system]# nft -f /etc/nftables/firewall.nft
    [root@localhost system]# nft list ruleset
    table inet x {
        flowtable f {
            hook ingress priority filter
        }

        chain y {
            type filter hook forward priority filter; policy accept;
            ip protocol tcp flow add @f
            counter packets 0 bytes 0
            tcp dport 80
        }
    }

This difference in behavior then begets a new question:

Does prepending the "more destructive" 'flush ruleset' statement at
the very beginning of the 'firewall.nft' file still honor the
"atomicity" guarantee of running 'nft -f' again this file, or is this
guarantee only honored when prepending 'flush table' statements? In
other words, is there a minute period after running 'flush ruleset' in
my file where the node is unprotected?

Thanks for your time,
-Martin

Re: Trying to provision flowtable returns error

[Martin Gignac]

I think I just answered my previous question:

> Does prepending the "more destructive" 'flush ruleset' statement at
> the very beginning of the 'firewall.nft' file still honor the
> "atomicity" guarantee of running 'nft -f' again this file, or is this
> guarantee only honored when prepending 'flush table' statements? In
> other words, is there a minute period after running 'flush ruleset' in
> my file where the node is unprotected?

According to https://wiki.nftables.org/wiki-nftables/index.php/Operations_at_ruleset_level:

    BACKUP/RESTORE

    You can combine these two commands above to backup your ruleset:

    % echo "nft flush ruleset" > backup.nft
    % nft list ruleset >> backup.nft

    And load it atomically:

    % nft -f backup.nft

I interpret this to mean that my original method of doing things is as
atomic as using 'flush table <tablename>', even if it is more
destructive. I guess going forward I will have to make sure to prepend
'flush table' statements for every individual table I refer to in my
'firewall.nft' file.

Thanks,
-Martin

Re: Trying to provision flowtable returns error

[Pablo Neira Ayuso]

On Fri, Nov 06, 2020 at 10:24:35AM -0500, Martin Gignac wrote:
> I think I just answered my previous question:
> 
> > Does prepending the "more destructive" 'flush ruleset' statement at
> > the very beginning of the 'firewall.nft' file still honor the
> > "atomicity" guarantee of running 'nft -f' again this file, or is this
> > guarantee only honored when prepending 'flush table' statements? In
> > other words, is there a minute period after running 'flush ruleset' in
> > my file where the node is unprotected?
> 
> According to https://wiki.nftables.org/wiki-nftables/index.php/Operations_at_ruleset_level:
> 
>     BACKUP/RESTORE
> 
>     You can combine these two commands above to backup your ruleset:
> 
>     % echo "nft flush ruleset" > backup.nft
>     % nft list ruleset >> backup.nft
> 
>     And load it atomically:
> 
>     % nft -f backup.nft
> 
> I interpret this to mean that my original method of doing things is as
> atomic as using 'flush table <tablename>', even if it is more
> destructive. I guess going forward I will have to make sure to prepend
> 'flush table' statements for every individual table I refer to in my
> 'firewall.nft' file.

It is not "flush ruleset" that makes things atomic, this is just an
operation to tear down everything.

You achieve atomic ruleset updates by using `nft -f'.

Sorry, I'm getting a bit lost regarding what the problem is at this
stage regarding the flowtable infrastructure.

Re: Trying to provision flowtable returns error

[Martin Gignac]

> Sorry, I'm getting a bit lost regarding what the problem is at this
> stage regarding the flowtable infrastructure.

I think the confusion stems from the fact that I have set up an
nftables workflow for myself that is "opinionated" and that doesn't
currently fit "out-of-the-box" with the way nft and flowtables
interact. I wrote a helper/wrapper program for myself that simplifies
nftables rules management by offering show, edit, check, commit, diff
and versioning functions to mimic the Juniper firewall behaviors that
I am accustomed to and find very convenient. Some of these functions
are plainly just front-ends to 'nft' commands, while some others (such
as diff and versioning) require more work in order to allow diffs and
versioning between both 'raw' /etc/nftables/firewall.nft files as well
as the 'compiled' (pretty-printed, comments removed) result of running
'nft' on these files. This workflow relied on assumptions that I had
made that my desire to integrate flow tables suddenly blew out of the
water. The "destructive" nature of 'flush ruleset' (which I was using
so far) was not an issue before, but with flow tables it suddenly
became one simply because I had not understood its reach. Had I been
using dynamic sets (which I hadn't so far) this likely would have
presented some issues too. But the fact that 'flush table' statements
_do_ exhibit the behavior that I had wrongly attributed to 'flush
ruleset', I now know that I have a way to use my program with flow
tables; I just need to code some logic to find out the existing table
names and generate the required 'flush table' statements as part of my
commit function.

What also doesn't help is that I'm probably not explaining myself
properly, and as a developer of nftables your intimate knowledge of it
is at odds with my amateurish understanding of it. :o)

Thanks to both of you for taking the time to answer, and for your work
on nftables!

Regards,
-Martin

Re: Trying to provision flowtable returns error

[Gordon Fisher]

On 11/6/2020 7:24 AM, Martin Gignac wrote:
> I think I just answered my previous question:
>
>> Does prepending the "more destructive" 'flush ruleset' statement at
>> the very beginning of the 'firewall.nft' file still honor the
>> "atomicity" guarantee of running 'nft -f' again this file, or is this
>> guarantee only honored when prepending 'flush table' statements? In
>> other words, is there a minute period after running 'flush ruleset' in
>> my file where the node is unprotected?
> According to https://wiki.nftables.org/wiki-nftables/index.php/Operations_at_ruleset_level:
>
>      BACKUP/RESTORE
>
>      You can combine these two commands above to backup your ruleset:
>
>      % echo "nft flush ruleset" > backup.nft
>      % nft list ruleset >> backup.nft
The above could also be condensed into a single line (assuming a 
Bourne-based shell):

$ { echo 'nft flush ruleset'; nft list ruleset; } > backup.nft

Which can be useful for writing backup.nft in one go.

>      And load it atomically:
>
>      % nft -f backup.nft
>
> I interpret this to mean that my original method of doing things is as
> atomic as using 'flush table <tablename>', even if it is more
> destructive. I guess going forward I will have to make sure to prepend
> 'flush table' statements for every individual table I refer to in my
> 'firewall.nft' file.
-- 
gfish

Re: Trying to provision flowtable returns error

[Pablo Neira Ayuso]

On Fri, Nov 06, 2020 at 11:58:33AM +0100, Pablo Neira Ayuso wrote:
> Hi,
> 
> This works fine here, see below.
> 
> On Thu, Nov 05, 2020 at 04:45:31PM -0500, Martin Gignac wrote:
> > Hi Pablo,
> > 
> > > You can dynamically add/delete devices to/from flowtables since Linux
> > > kernel 5.8
> > 
> > Are you referring to this patch ?:
> > https://www.spinics.net/lists/netfilter-devel/msg67310.html
> > 
> > I tried with Fedora 33 (5.8.17-300.fc33.x86_64) and this file:
> > 
> >     [root@localhost ~]# cat /etc/nftables/firewall.nft
> >     flush ruleset
> > 
> >     table inet x {
> >         flowtable f {
> >                 hook ingress priority 0;
> >     }
> >             chain y {
> >                     type filter hook forward priority 0; policy accept;
> >                     ip protocol tcp flow offload @f
> >                     counter packets 0 bytes 0
> >             }
> >     }
> > 
> > and indeed it does load without error, although I had to compile the
> > latest version of nft (v0.9.7) as v0.9.3 (which comes with Fedora 33)
> > was giving me this error:
> > 
> >     [root@localhost ~]# nft -f /etc/nftables/firewall.nft
> >     /etc/nftables/firewall.nft:4:12-12: Error: Unbound flowtable not
> > allowed (must specify devices)
> >         flowtable f {
> >                 ^
> > Once I added my br0 interface wih 'ip link add br0 type bridge' I was
> > able to run 'nft add flowtable inet x f { devices = { br0 } \; }'
> > without error.
> >
> > However, if I run 'nft -f /etc/nftables/firewall.nft' again and then
> > 'nft list ruleset', br0 is gone. Does this mean that it is no longer
> > bound to a flow table?
> > 
> > The way I have been handling rule changes so far is to modify a single
> > '/etc/nftables/firewall.nft' file every time I need to modify rules
> > and then run 'nft -f /etc/nftables/firewall.nft' to reload and apply
> > those changes (I don't tend to run single nft commands to update
> > things here and there -- I prefer to modify a single file as the
> > source of truth and then reload the ruleset completely). Running
> > something like 'nft add flowtable inet x f { devices = { br0 } \; }'
> > once upon boot up when a logical interface comes up is fine, but does
> > my workflow require that I then run 'nft add flowtable inet x f {
> > devices = { br0 } \; }' after every time I run 'nft -f
> > /etc/nftables/firewall.nft'?
> 
> # cat firewall.nft
> table ip x {
>         flowtable y {
>                 hook ingress priority filter
>         }
> 
>         chain y {
>                 flow add @y
>         }
> }
> # nft -f firewall.nft
> # nft list ruleset
> table ip x {
>         flowtable y {
>                 hook ingress priority filter
>         }
> 
>         chain y {
>                 flow add @y
>         }
> }
> 
> This is your base ruleset.
> 
> Now you add devices to the flowtable (requirements: kernel >= 5.8
> and nftables >= 0.9.7):
> 
> # nft add flowtable x y { devices = { eth0, eth1 } \; }
> 
> Listing shows:
> 
> # nft list ruleset
> table ip x {
>         flowtable y {
>                 hook ingress priority filter
>                 devices = { eth0, eth1 }
>         }
> 
>         chain y {
>                 flow add @y
>         }
> }
> 
> Note: If eth0 is gone, then this is automatically removed from the
> flowtable.
> 
> Is "flush ruleset" at the very beginning of your firewall.nft file?
> 
> If so, that is tearing down everything and creating it from scratch,
> so the devices you have dynamically added are gone since they are not
> in the original firewall.nft file.
> 
> I would expect you load firewall.nft at boot time, then dynamically
> add devices as needed in run time.

Just to clarify.

You can still load device at boot time, ie.

# cat ruleset.nft
table ip x {
        flowtable y {
                hook ingress priority filter
                devices = { eth0, eth1 }
        }

        chain y {
                flow add @y
        }
}
# nft -f ruleset.nft

You can just update the flowtable later on with more (new) devices in
case you require this. Kernel >= 5.8 and nft >= 0.9.7 are just
providing a bit more flexibility in case you dynamically load an
interface and you would like to incrementally update a flowtable
without reloading the whole ruleset.