Re: ct state module issue

Re: ct state module issue

[Florian Westphal]

Matt Zagrabelny <mzagrabe@d.umn.edu> wrote:

[ CCing bpf/btf experts ]

> I'm running kernel: 6.1.0-10-amd64
> and
> nftables v1.0.6 (Lester Gooch #5)
> 
> I have a set of nftables rules that have served me well for Debian 11
> - thanks in large part to the netfilter mailing list, so...thank you!
> nftables on Debian 11 is: 0.9.8-3.1+deb11u1
> 
> I have recently installed Debian 12 and tried my nftables rules and
> have hit a snag with the connection tracking and a verdict map.
> nftables on Debian 12 is: 1.0.6-2+deb12u1
> 
> When I run the offending snippet:
> 
> # nft -f /etc/nftables.conf.d/300-common.d/200-connection-tracking.nft
> /etc/nftables.conf.d/300-common.d/200-connection-tracking.nft:4:9-16:
> Error: Could not process rule: No such file or directory
>         ct state vmap {

[..]
  	^^^^^^^^
> When I watch the kernel logs (journalctl), I see:
> 
> Jul 25 13:44:04 localhost kernel: BPF: [99725] STRUCT
> Jul 25 13:44:04 localhost kernel: BPF: size=104 vlen=12
> Jul 25 13:44:04 localhost kernel: BPF:
> Jul 25 13:44:04 localhost kernel: BPF: Invalid name
> Jul 25 13:44:04 localhost kernel: BPF:
> Jul 25 13:44:04 localhost kernel: failed to validate module
> [nf_conntrack] BTF: -22
> Jul 25 13:44:04 localhost kernel: missing module BTF, cannot register kfuncs

So nf_conntrack.ko fails to load because of a btf issue.

My question to bpf folks is:

Should we make register_nf_conntrack_bpf() return 'void'?

This way normal conntrack would still work.  bpf programs using
conntrack kfuncs would fail, but above dmesg splat already gives
a clue as to why conntrack kfuncs aren't there.

No idea about the actual problem or how to debug that, but bpf
people should know.

Re: ct state module issue

[Alexei Starovoitov]

On Tue, Jul 25, 2023 at 12:33 PM Florian Westphal <fw@strlen.de> wrote:
>
> Matt Zagrabelny <mzagrabe@d.umn.edu> wrote:
>
> [ CCing bpf/btf experts ]
>
> > I'm running kernel: 6.1.0-10-amd64
> > and
> > nftables v1.0.6 (Lester Gooch #5)
> >
> > I have a set of nftables rules that have served me well for Debian 11
> > - thanks in large part to the netfilter mailing list, so...thank you!
> > nftables on Debian 11 is: 0.9.8-3.1+deb11u1
> >
> > I have recently installed Debian 12 and tried my nftables rules and
> > have hit a snag with the connection tracking and a verdict map.
> > nftables on Debian 12 is: 1.0.6-2+deb12u1
> >
> > When I run the offending snippet:
> >
> > # nft -f /etc/nftables.conf.d/300-common.d/200-connection-tracking.nft
> > /etc/nftables.conf.d/300-common.d/200-connection-tracking.nft:4:9-16:
> > Error: Could not process rule: No such file or directory
> >         ct state vmap {
>
> [..]
>         ^^^^^^^^
> > When I watch the kernel logs (journalctl), I see:
> >
> > Jul 25 13:44:04 localhost kernel: BPF: [99725] STRUCT
> > Jul 25 13:44:04 localhost kernel: BPF: size=104 vlen=12
> > Jul 25 13:44:04 localhost kernel: BPF:
> > Jul 25 13:44:04 localhost kernel: BPF: Invalid name
> > Jul 25 13:44:04 localhost kernel: BPF:
> > Jul 25 13:44:04 localhost kernel: failed to validate module
> > [nf_conntrack] BTF: -22
> > Jul 25 13:44:04 localhost kernel: missing module BTF, cannot register kfuncs
>
> So nf_conntrack.ko fails to load because of a btf issue.
>
> My question to bpf folks is:
>
> Should we make register_nf_conntrack_bpf() return 'void'?
>
> This way normal conntrack would still work.  bpf programs using
> conntrack kfuncs would fail, but above dmesg splat already gives
> a clue as to why conntrack kfuncs aren't there.
>
> No idea about the actual problem or how to debug that, but bpf
> people should know.

The pr_err() was changed to pr_warn() in
commit 3de4d22cc9ac ("bpf, btf: Warn but return no error for NULL btf
from __register_btf_kfunc_id_set()").


Please upgrade the kernel and ignore the warn if you don't need bpf/btf/kfuncs.

Three links in that commit provide more details.

Re: ct state module issue

[Pablo Neira Ayuso]

Hi Alexei,

On Tue, Jul 25, 2023 at 12:57:13PM -0700, Alexei Starovoitov wrote:
> On Tue, Jul 25, 2023 at 12:33 PM Florian Westphal <fw@strlen.de> wrote:
> >
> > Matt Zagrabelny <mzagrabe@d.umn.edu> wrote:
> >
> > [ CCing bpf/btf experts ]
> >
> > > I'm running kernel: 6.1.0-10-amd64
> > > and
> > > nftables v1.0.6 (Lester Gooch #5)
> > >
> > > I have a set of nftables rules that have served me well for Debian 11
> > > - thanks in large part to the netfilter mailing list, so...thank you!
> > > nftables on Debian 11 is: 0.9.8-3.1+deb11u1
> > >
> > > I have recently installed Debian 12 and tried my nftables rules and
> > > have hit a snag with the connection tracking and a verdict map.
> > > nftables on Debian 12 is: 1.0.6-2+deb12u1
> > >
> > > When I run the offending snippet:
> > >
> > > # nft -f /etc/nftables.conf.d/300-common.d/200-connection-tracking.nft
> > > /etc/nftables.conf.d/300-common.d/200-connection-tracking.nft:4:9-16:
> > > Error: Could not process rule: No such file or directory
> > >         ct state vmap {
> >
> > [..]
> >         ^^^^^^^^
> > > When I watch the kernel logs (journalctl), I see:
> > >
> > > Jul 25 13:44:04 localhost kernel: BPF: [99725] STRUCT
> > > Jul 25 13:44:04 localhost kernel: BPF: size=104 vlen=12
> > > Jul 25 13:44:04 localhost kernel: BPF:
> > > Jul 25 13:44:04 localhost kernel: BPF: Invalid name
> > > Jul 25 13:44:04 localhost kernel: BPF:
> > > Jul 25 13:44:04 localhost kernel: failed to validate module
> > > [nf_conntrack] BTF: -22
> > > Jul 25 13:44:04 localhost kernel: missing module BTF, cannot register kfuncs
> >
> > So nf_conntrack.ko fails to load because of a btf issue.
> >
> > My question to bpf folks is:
> >
> > Should we make register_nf_conntrack_bpf() return 'void'?
> >
> > This way normal conntrack would still work.  bpf programs using
> > conntrack kfuncs would fail, but above dmesg splat already gives
> > a clue as to why conntrack kfuncs aren't there.
> >
> > No idea about the actual problem or how to debug that, but bpf
> > people should know.
> 
> The pr_err() was changed to pr_warn() in
> commit 3de4d22cc9ac ("bpf, btf: Warn but return no error for NULL btf
> from __register_btf_kfunc_id_set()").

OK, no ENOENT anymore, hence no bail out.

> Please upgrade the kernel and ignore the warn if you don't need bpf/btf/kfuncs.
> 
> Three links in that commit provide more details.

Jul 25 13:44:04 localhost kernel: BPF: [99725] STRUCT
Jul 25 13:44:04 localhost kernel: BPF: size=104 vlen=12
Jul 25 13:44:04 localhost kernel: BPF:
Jul 25 13:44:04 localhost kernel: BPF: Invalid name
Jul 25 13:44:04 localhost kernel: BPF:

Are these debugging logs above still displayed? Maybe remove them too
or only display them when all required things are in place and users
opt-in to use this new infrastructure?

Thanks.

Re: ct state module issue

[Alexei Starovoitov]

On Wed, Jul 26, 2023 at 12:40 AM Pablo Neira Ayuso <pablo@netfilter.org> wrote:
>
> Hi Alexei,
>
> On Tue, Jul 25, 2023 at 12:57:13PM -0700, Alexei Starovoitov wrote:
> > On Tue, Jul 25, 2023 at 12:33 PM Florian Westphal <fw@strlen.de> wrote:
> > >
> > > Matt Zagrabelny <mzagrabe@d.umn.edu> wrote:
> > >
> > > [ CCing bpf/btf experts ]
> > >
> > > > I'm running kernel: 6.1.0-10-amd64
> > > > and
> > > > nftables v1.0.6 (Lester Gooch #5)
> > > >
> > > > I have a set of nftables rules that have served me well for Debian 11
> > > > - thanks in large part to the netfilter mailing list, so...thank you!
> > > > nftables on Debian 11 is: 0.9.8-3.1+deb11u1
> > > >
> > > > I have recently installed Debian 12 and tried my nftables rules and
> > > > have hit a snag with the connection tracking and a verdict map.
> > > > nftables on Debian 12 is: 1.0.6-2+deb12u1
> > > >
> > > > When I run the offending snippet:
> > > >
> > > > # nft -f /etc/nftables.conf.d/300-common.d/200-connection-tracking.nft
> > > > /etc/nftables.conf.d/300-common.d/200-connection-tracking.nft:4:9-16:
> > > > Error: Could not process rule: No such file or directory
> > > >         ct state vmap {
> > >
> > > [..]
> > >         ^^^^^^^^
> > > > When I watch the kernel logs (journalctl), I see:
> > > >
> > > > Jul 25 13:44:04 localhost kernel: BPF: [99725] STRUCT
> > > > Jul 25 13:44:04 localhost kernel: BPF: size=104 vlen=12
> > > > Jul 25 13:44:04 localhost kernel: BPF:
> > > > Jul 25 13:44:04 localhost kernel: BPF: Invalid name
> > > > Jul 25 13:44:04 localhost kernel: BPF:
> > > > Jul 25 13:44:04 localhost kernel: failed to validate module
> > > > [nf_conntrack] BTF: -22
> > > > Jul 25 13:44:04 localhost kernel: missing module BTF, cannot register kfuncs
> > >
> > > So nf_conntrack.ko fails to load because of a btf issue.
> > >
> > > My question to bpf folks is:
> > >
> > > Should we make register_nf_conntrack_bpf() return 'void'?
> > >
> > > This way normal conntrack would still work.  bpf programs using
> > > conntrack kfuncs would fail, but above dmesg splat already gives
> > > a clue as to why conntrack kfuncs aren't there.
> > >
> > > No idea about the actual problem or how to debug that, but bpf
> > > people should know.
> >
> > The pr_err() was changed to pr_warn() in
> > commit 3de4d22cc9ac ("bpf, btf: Warn but return no error for NULL btf
> > from __register_btf_kfunc_id_set()").
>
> OK, no ENOENT anymore, hence no bail out.
>
> > Please upgrade the kernel and ignore the warn if you don't need bpf/btf/kfuncs.
> >
> > Three links in that commit provide more details.
>
> Jul 25 13:44:04 localhost kernel: BPF: [99725] STRUCT
> Jul 25 13:44:04 localhost kernel: BPF: size=104 vlen=12
> Jul 25 13:44:04 localhost kernel: BPF:
> Jul 25 13:44:04 localhost kernel: BPF: Invalid name
> Jul 25 13:44:04 localhost kernel: BPF:
>
> Are these debugging logs above still displayed? Maybe remove them too
> or only display them when all required things are in place and users
> opt-in to use this new infrastructure?

Kernel doesn't print them to console. These messages go to BTF verifier log
supplied by user space. It's not clear what process sends them to journalctl.