ct state module issue

ct state module issue

[Matt Zagrabelny]

Greetings netfilter,

I'm running kernel: 6.1.0-10-amd64
and
nftables v1.0.6 (Lester Gooch #5)

I have a set of nftables rules that have served me well for Debian 11
- thanks in large part to the netfilter mailing list, so...thank you!
nftables on Debian 11 is: 0.9.8-3.1+deb11u1

I have recently installed Debian 12 and tried my nftables rules and
have hit a snag with the connection tracking and a verdict map.
nftables on Debian 12 is: 1.0.6-2+deb12u1

When I run the offending snippet:

# nft -f /etc/nftables.conf.d/300-common.d/200-connection-tracking.nft
/etc/nftables.conf.d/300-common.d/200-connection-tracking.nft:4:9-16:
Error: Could not process rule: No such file or directory
        ct state vmap {
        ^^^^^^^^
# cat /etc/nftables.conf.d/300-common.d/200-connection-tracking.nft
table inet filter {
    chain input {
        # accept traffic originated from us
        ct state vmap {
            established: accept,
            related:     accept,
            invalid:     drop,
        }
    }
}

When I watch the kernel logs (journalctl), I see:

Jul 25 13:44:04 localhost kernel: BPF: [99725] STRUCT
Jul 25 13:44:04 localhost kernel: BPF: size=104 vlen=12
Jul 25 13:44:04 localhost kernel: BPF:
Jul 25 13:44:04 localhost kernel: BPF: Invalid name
Jul 25 13:44:04 localhost kernel: BPF:
Jul 25 13:44:04 localhost kernel: failed to validate module
[nf_conntrack] BTF: -22
Jul 25 13:44:04 localhost kernel: missing module BTF, cannot register kfuncs

I've tried to load the module manually:

# lsmod | rg nf
nf_defrag_ipv6         24576  0
nf_defrag_ipv4         16384  0
nf_tables             290816  0
libcrc32c              16384  1 nf_tables
nfnetlink              20480  1 nf_tables
binfmt_misc            24576  1
configfs               57344  1

# modprobe nft_ct
modprobe: ERROR: could not insert 'nft_ct': Unknown symbol in module,
or unknown parameter (see dmesg)

dmesg shows the same as the kernel message as above.

I'm starting to struggle with where to look for debugging clues. Any
help would be very appreciated.

Thank you!

-m

Re: ct state module issue

[Florian Westphal]

Matt Zagrabelny <mzagrabe@d.umn.edu> wrote:

[ CCing bpf/btf experts ]

> I'm running kernel: 6.1.0-10-amd64
> and
> nftables v1.0.6 (Lester Gooch #5)
> 
> I have a set of nftables rules that have served me well for Debian 11
> - thanks in large part to the netfilter mailing list, so...thank you!
> nftables on Debian 11 is: 0.9.8-3.1+deb11u1
> 
> I have recently installed Debian 12 and tried my nftables rules and
> have hit a snag with the connection tracking and a verdict map.
> nftables on Debian 12 is: 1.0.6-2+deb12u1
> 
> When I run the offending snippet:
> 
> # nft -f /etc/nftables.conf.d/300-common.d/200-connection-tracking.nft
> /etc/nftables.conf.d/300-common.d/200-connection-tracking.nft:4:9-16:
> Error: Could not process rule: No such file or directory
>         ct state vmap {

[..]
  	^^^^^^^^
> When I watch the kernel logs (journalctl), I see:
> 
> Jul 25 13:44:04 localhost kernel: BPF: [99725] STRUCT
> Jul 25 13:44:04 localhost kernel: BPF: size=104 vlen=12
> Jul 25 13:44:04 localhost kernel: BPF:
> Jul 25 13:44:04 localhost kernel: BPF: Invalid name
> Jul 25 13:44:04 localhost kernel: BPF:
> Jul 25 13:44:04 localhost kernel: failed to validate module
> [nf_conntrack] BTF: -22
> Jul 25 13:44:04 localhost kernel: missing module BTF, cannot register kfuncs

So nf_conntrack.ko fails to load because of a btf issue.

My question to bpf folks is:

Should we make register_nf_conntrack_bpf() return 'void'?

This way normal conntrack would still work.  bpf programs using
conntrack kfuncs would fail, but above dmesg splat already gives
a clue as to why conntrack kfuncs aren't there.

No idea about the actual problem or how to debug that, but bpf
people should know.

Re: ct state module issue

[Alexei Starovoitov]

On Tue, Jul 25, 2023 at 12:33 PM Florian Westphal <fw@strlen.de> wrote:
>
> Matt Zagrabelny <mzagrabe@d.umn.edu> wrote:
>
> [ CCing bpf/btf experts ]
>
> > I'm running kernel: 6.1.0-10-amd64
> > and
> > nftables v1.0.6 (Lester Gooch #5)
> >
> > I have a set of nftables rules that have served me well for Debian 11
> > - thanks in large part to the netfilter mailing list, so...thank you!
> > nftables on Debian 11 is: 0.9.8-3.1+deb11u1
> >
> > I have recently installed Debian 12 and tried my nftables rules and
> > have hit a snag with the connection tracking and a verdict map.
> > nftables on Debian 12 is: 1.0.6-2+deb12u1
> >
> > When I run the offending snippet:
> >
> > # nft -f /etc/nftables.conf.d/300-common.d/200-connection-tracking.nft
> > /etc/nftables.conf.d/300-common.d/200-connection-tracking.nft:4:9-16:
> > Error: Could not process rule: No such file or directory
> >         ct state vmap {
>
> [..]
>         ^^^^^^^^
> > When I watch the kernel logs (journalctl), I see:
> >
> > Jul 25 13:44:04 localhost kernel: BPF: [99725] STRUCT
> > Jul 25 13:44:04 localhost kernel: BPF: size=104 vlen=12
> > Jul 25 13:44:04 localhost kernel: BPF:
> > Jul 25 13:44:04 localhost kernel: BPF: Invalid name
> > Jul 25 13:44:04 localhost kernel: BPF:
> > Jul 25 13:44:04 localhost kernel: failed to validate module
> > [nf_conntrack] BTF: -22
> > Jul 25 13:44:04 localhost kernel: missing module BTF, cannot register kfuncs
>
> So nf_conntrack.ko fails to load because of a btf issue.
>
> My question to bpf folks is:
>
> Should we make register_nf_conntrack_bpf() return 'void'?
>
> This way normal conntrack would still work.  bpf programs using
> conntrack kfuncs would fail, but above dmesg splat already gives
> a clue as to why conntrack kfuncs aren't there.
>
> No idea about the actual problem or how to debug that, but bpf
> people should know.

The pr_err() was changed to pr_warn() in
commit 3de4d22cc9ac ("bpf, btf: Warn but return no error for NULL btf
from __register_btf_kfunc_id_set()").


Please upgrade the kernel and ignore the warn if you don't need bpf/btf/kfuncs.

Three links in that commit provide more details.

Re: ct state module issue

[Alexei Starovoitov]

On Tue, Jul 25, 2023 at 12:33 PM Florian Westphal <fw@strlen.de> wrote:
>
> Matt Zagrabelny <mzagrabe@d.umn.edu> wrote:
>
> [ CCing bpf/btf experts ]
>
> > I'm running kernel: 6.1.0-10-amd64
> > and
> > nftables v1.0.6 (Lester Gooch #5)
> >
> > I have a set of nftables rules that have served me well for Debian 11
> > - thanks in large part to the netfilter mailing list, so...thank you!
> > nftables on Debian 11 is: 0.9.8-3.1+deb11u1
> >
> > I have recently installed Debian 12 and tried my nftables rules and
> > have hit a snag with the connection tracking and a verdict map.
> > nftables on Debian 12 is: 1.0.6-2+deb12u1
> >
> > When I run the offending snippet:
> >
> > # nft -f /etc/nftables.conf.d/300-common.d/200-connection-tracking.nft
> > /etc/nftables.conf.d/300-common.d/200-connection-tracking.nft:4:9-16:
> > Error: Could not process rule: No such file or directory
> >         ct state vmap {
>
> [..]
>         ^^^^^^^^
> > When I watch the kernel logs (journalctl), I see:
> >
> > Jul 25 13:44:04 localhost kernel: BPF: [99725] STRUCT
> > Jul 25 13:44:04 localhost kernel: BPF: size=104 vlen=12
> > Jul 25 13:44:04 localhost kernel: BPF:
> > Jul 25 13:44:04 localhost kernel: BPF: Invalid name
> > Jul 25 13:44:04 localhost kernel: BPF:
> > Jul 25 13:44:04 localhost kernel: failed to validate module
> > [nf_conntrack] BTF: -22
> > Jul 25 13:44:04 localhost kernel: missing module BTF, cannot register kfuncs
>
> So nf_conntrack.ko fails to load because of a btf issue.
>
> My question to bpf folks is:
>
> Should we make register_nf_conntrack_bpf() return 'void'?
>
> This way normal conntrack would still work.  bpf programs using
> conntrack kfuncs would fail, but above dmesg splat already gives
> a clue as to why conntrack kfuncs aren't there.
>
> No idea about the actual problem or how to debug that, but bpf
> people should know.

The pr_err() was changed to pr_warn() in
commit 3de4d22cc9ac ("bpf, btf: Warn but return no error for NULL btf
from __register_btf_kfunc_id_set()").


Please upgrade the kernel and ignore the warn if you don't need bpf/btf/kfuncs.

Three links in that commit provide more details.

Re: ct state module issue

[Pablo Neira Ayuso]

Hi Alexei,

On Tue, Jul 25, 2023 at 12:57:13PM -0700, Alexei Starovoitov wrote:
> On Tue, Jul 25, 2023 at 12:33 PM Florian Westphal <fw@strlen.de> wrote:
> >
> > Matt Zagrabelny <mzagrabe@d.umn.edu> wrote:
> >
> > [ CCing bpf/btf experts ]
> >
> > > I'm running kernel: 6.1.0-10-amd64
> > > and
> > > nftables v1.0.6 (Lester Gooch #5)
> > >
> > > I have a set of nftables rules that have served me well for Debian 11
> > > - thanks in large part to the netfilter mailing list, so...thank you!
> > > nftables on Debian 11 is: 0.9.8-3.1+deb11u1
> > >
> > > I have recently installed Debian 12 and tried my nftables rules and
> > > have hit a snag with the connection tracking and a verdict map.
> > > nftables on Debian 12 is: 1.0.6-2+deb12u1
> > >
> > > When I run the offending snippet:
> > >
> > > # nft -f /etc/nftables.conf.d/300-common.d/200-connection-tracking.nft
> > > /etc/nftables.conf.d/300-common.d/200-connection-tracking.nft:4:9-16:
> > > Error: Could not process rule: No such file or directory
> > >         ct state vmap {
> >
> > [..]
> >         ^^^^^^^^
> > > When I watch the kernel logs (journalctl), I see:
> > >
> > > Jul 25 13:44:04 localhost kernel: BPF: [99725] STRUCT
> > > Jul 25 13:44:04 localhost kernel: BPF: size=104 vlen=12
> > > Jul 25 13:44:04 localhost kernel: BPF:
> > > Jul 25 13:44:04 localhost kernel: BPF: Invalid name
> > > Jul 25 13:44:04 localhost kernel: BPF:
> > > Jul 25 13:44:04 localhost kernel: failed to validate module
> > > [nf_conntrack] BTF: -22
> > > Jul 25 13:44:04 localhost kernel: missing module BTF, cannot register kfuncs
> >
> > So nf_conntrack.ko fails to load because of a btf issue.
> >
> > My question to bpf folks is:
> >
> > Should we make register_nf_conntrack_bpf() return 'void'?
> >
> > This way normal conntrack would still work.  bpf programs using
> > conntrack kfuncs would fail, but above dmesg splat already gives
> > a clue as to why conntrack kfuncs aren't there.
> >
> > No idea about the actual problem or how to debug that, but bpf
> > people should know.
> 
> The pr_err() was changed to pr_warn() in
> commit 3de4d22cc9ac ("bpf, btf: Warn but return no error for NULL btf
> from __register_btf_kfunc_id_set()").

OK, no ENOENT anymore, hence no bail out.

> Please upgrade the kernel and ignore the warn if you don't need bpf/btf/kfuncs.
> 
> Three links in that commit provide more details.

Jul 25 13:44:04 localhost kernel: BPF: [99725] STRUCT
Jul 25 13:44:04 localhost kernel: BPF: size=104 vlen=12
Jul 25 13:44:04 localhost kernel: BPF:
Jul 25 13:44:04 localhost kernel: BPF: Invalid name
Jul 25 13:44:04 localhost kernel: BPF:

Are these debugging logs above still displayed? Maybe remove them too
or only display them when all required things are in place and users
opt-in to use this new infrastructure?

Thanks.

Re: ct state module issue

[Pablo Neira Ayuso]

Hi Alexei,

On Tue, Jul 25, 2023 at 12:57:13PM -0700, Alexei Starovoitov wrote:
> On Tue, Jul 25, 2023 at 12:33 PM Florian Westphal <fw@strlen.de> wrote:
> >
> > Matt Zagrabelny <mzagrabe@d.umn.edu> wrote:
> >
> > [ CCing bpf/btf experts ]
> >
> > > I'm running kernel: 6.1.0-10-amd64
> > > and
> > > nftables v1.0.6 (Lester Gooch #5)
> > >
> > > I have a set of nftables rules that have served me well for Debian 11
> > > - thanks in large part to the netfilter mailing list, so...thank you!
> > > nftables on Debian 11 is: 0.9.8-3.1+deb11u1
> > >
> > > I have recently installed Debian 12 and tried my nftables rules and
> > > have hit a snag with the connection tracking and a verdict map.
> > > nftables on Debian 12 is: 1.0.6-2+deb12u1
> > >
> > > When I run the offending snippet:
> > >
> > > # nft -f /etc/nftables.conf.d/300-common.d/200-connection-tracking.nft
> > > /etc/nftables.conf.d/300-common.d/200-connection-tracking.nft:4:9-16:
> > > Error: Could not process rule: No such file or directory
> > >         ct state vmap {
> >
> > [..]
> >         ^^^^^^^^
> > > When I watch the kernel logs (journalctl), I see:
> > >
> > > Jul 25 13:44:04 localhost kernel: BPF: [99725] STRUCT
> > > Jul 25 13:44:04 localhost kernel: BPF: size=104 vlen=12
> > > Jul 25 13:44:04 localhost kernel: BPF:
> > > Jul 25 13:44:04 localhost kernel: BPF: Invalid name
> > > Jul 25 13:44:04 localhost kernel: BPF:
> > > Jul 25 13:44:04 localhost kernel: failed to validate module
> > > [nf_conntrack] BTF: -22
> > > Jul 25 13:44:04 localhost kernel: missing module BTF, cannot register kfuncs
> >
> > So nf_conntrack.ko fails to load because of a btf issue.
> >
> > My question to bpf folks is:
> >
> > Should we make register_nf_conntrack_bpf() return 'void'?
> >
> > This way normal conntrack would still work.  bpf programs using
> > conntrack kfuncs would fail, but above dmesg splat already gives
> > a clue as to why conntrack kfuncs aren't there.
> >
> > No idea about the actual problem or how to debug that, but bpf
> > people should know.
> 
> The pr_err() was changed to pr_warn() in
> commit 3de4d22cc9ac ("bpf, btf: Warn but return no error for NULL btf
> from __register_btf_kfunc_id_set()").

OK, no ENOENT anymore, hence no bail out.

> Please upgrade the kernel and ignore the warn if you don't need bpf/btf/kfuncs.
> 
> Three links in that commit provide more details.

Jul 25 13:44:04 localhost kernel: BPF: [99725] STRUCT
Jul 25 13:44:04 localhost kernel: BPF: size=104 vlen=12
Jul 25 13:44:04 localhost kernel: BPF:
Jul 25 13:44:04 localhost kernel: BPF: Invalid name
Jul 25 13:44:04 localhost kernel: BPF:

Are these debugging logs above still displayed? Maybe remove them too
or only display them when all required things are in place and users
opt-in to use this new infrastructure?

Thanks.

Re: ct state module issue

[Alexei Starovoitov]

On Wed, Jul 26, 2023 at 12:40 AM Pablo Neira Ayuso <pablo@netfilter.org> wrote:
>
> Hi Alexei,
>
> On Tue, Jul 25, 2023 at 12:57:13PM -0700, Alexei Starovoitov wrote:
> > On Tue, Jul 25, 2023 at 12:33 PM Florian Westphal <fw@strlen.de> wrote:
> > >
> > > Matt Zagrabelny <mzagrabe@d.umn.edu> wrote:
> > >
> > > [ CCing bpf/btf experts ]
> > >
> > > > I'm running kernel: 6.1.0-10-amd64
> > > > and
> > > > nftables v1.0.6 (Lester Gooch #5)
> > > >
> > > > I have a set of nftables rules that have served me well for Debian 11
> > > > - thanks in large part to the netfilter mailing list, so...thank you!
> > > > nftables on Debian 11 is: 0.9.8-3.1+deb11u1
> > > >
> > > > I have recently installed Debian 12 and tried my nftables rules and
> > > > have hit a snag with the connection tracking and a verdict map.
> > > > nftables on Debian 12 is: 1.0.6-2+deb12u1
> > > >
> > > > When I run the offending snippet:
> > > >
> > > > # nft -f /etc/nftables.conf.d/300-common.d/200-connection-tracking.nft
> > > > /etc/nftables.conf.d/300-common.d/200-connection-tracking.nft:4:9-16:
> > > > Error: Could not process rule: No such file or directory
> > > >         ct state vmap {
> > >
> > > [..]
> > >         ^^^^^^^^
> > > > When I watch the kernel logs (journalctl), I see:
> > > >
> > > > Jul 25 13:44:04 localhost kernel: BPF: [99725] STRUCT
> > > > Jul 25 13:44:04 localhost kernel: BPF: size=104 vlen=12
> > > > Jul 25 13:44:04 localhost kernel: BPF:
> > > > Jul 25 13:44:04 localhost kernel: BPF: Invalid name
> > > > Jul 25 13:44:04 localhost kernel: BPF:
> > > > Jul 25 13:44:04 localhost kernel: failed to validate module
> > > > [nf_conntrack] BTF: -22
> > > > Jul 25 13:44:04 localhost kernel: missing module BTF, cannot register kfuncs
> > >
> > > So nf_conntrack.ko fails to load because of a btf issue.
> > >
> > > My question to bpf folks is:
> > >
> > > Should we make register_nf_conntrack_bpf() return 'void'?
> > >
> > > This way normal conntrack would still work.  bpf programs using
> > > conntrack kfuncs would fail, but above dmesg splat already gives
> > > a clue as to why conntrack kfuncs aren't there.
> > >
> > > No idea about the actual problem or how to debug that, but bpf
> > > people should know.
> >
> > The pr_err() was changed to pr_warn() in
> > commit 3de4d22cc9ac ("bpf, btf: Warn but return no error for NULL btf
> > from __register_btf_kfunc_id_set()").
>
> OK, no ENOENT anymore, hence no bail out.
>
> > Please upgrade the kernel and ignore the warn if you don't need bpf/btf/kfuncs.
> >
> > Three links in that commit provide more details.
>
> Jul 25 13:44:04 localhost kernel: BPF: [99725] STRUCT
> Jul 25 13:44:04 localhost kernel: BPF: size=104 vlen=12
> Jul 25 13:44:04 localhost kernel: BPF:
> Jul 25 13:44:04 localhost kernel: BPF: Invalid name
> Jul 25 13:44:04 localhost kernel: BPF:
>
> Are these debugging logs above still displayed? Maybe remove them too
> or only display them when all required things are in place and users
> opt-in to use this new infrastructure?

Kernel doesn't print them to console. These messages go to BTF verifier log
supplied by user space. It's not clear what process sends them to journalctl.