[PATCH bpf 1/2] bpf: Add dummy type reference to nf_conn___init to fix type deduplication

[PATCH bpf 1/2] bpf: Add dummy type reference to nf_conn___init to fix type deduplication

[Toke Høiland-Jørgensen]

The bpf_ct_set_nat_info() kfunc is defined in the nf_nat.ko module, and
takes as a parameter the nf_conn___init struct, which is allocated through
the bpf_xdp_ct_alloc() helper defined in the nf_conntrack.ko module.
However, because kernel modules can't deduplicate BTF types between each
other, and the nf_conn___init struct is not referenced anywhere in vmlinux
BTF, this leads to two distinct BTF IDs for the same type (one in each
module). This confuses the verifier, as described here:

https://lore.kernel.org/all/87leoh372s.fsf@toke.dk/

As a workaround, add a dummy pointer to the type in net/filter.c, so the
type definition gets included in vmlinux BTF. This way, both modules can
refer to the same type ID (as they both build on top of vmlinux BTF), and
the verifier is no longer confused.

Fixes: 820dc0523e05 ("net: netfilter: move bpf_ct_set_nat_info kfunc in nf_nat_bpf.c")
Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com>
---
 net/core/filter.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/net/core/filter.c b/net/core/filter.c
index bb0136e7a8e4..1bdf9efe8593 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -80,6 +80,7 @@
 #include <net/tls.h>
 #include <net/xdp.h>
 #include <net/mptcp.h>
+#include <net/netfilter/nf_conntrack_bpf.h>
 
 static const struct bpf_func_proto *
 bpf_sk_base_func_proto(enum bpf_func_id func_id);
@@ -11531,3 +11532,17 @@ bpf_sk_base_func_proto(enum bpf_func_id func_id)
 
 	return func;
 }
+
+#if IS_MODULE(CONFIG_NF_CONNTRACK) && IS_ENABLED(CONFIG_DEBUG_INFO_BTF_MODULES)
+/* The nf_conn___init type is used in the NF_CONNTRACK kfuncs. The kfuncs are
+ * defined in two different modules, and we want to be able to use them
+ * interchangably with the same BTF type ID. Because modules can't de-duplicate
+ * BTF IDs between each other, we need the type to be referenced in the vmlinux
+ * BTF or the verifier will get confused about the different types. So we add
+ * this dummy pointer to serve as a type reference which will be included in
+ * vmlinux BTF, allowing both modules to refer to the same type ID.
+ *
+ * We use a pointer as that is smaller than an instance of the struct.
+ */
+const struct nf_conn___init *ctinit;
+#endif
-- 
2.38.1

[PATCH bpf 2/2] selftests/bpf: Add local definition of enum nf_nat_manip_type to bpf_nf test

[Toke Høiland-Jørgensen]

The bpf_nf selftest calls the bpf_ct_set_nat_info() kfunc, which takes a
parameter of type enum nf_nat_manip_type. However, if the nf_nat code is
compiled as a module, that enum is not defined in vmlinux BTF, and
compilation of the selftest fails.

A previous patch suggested just hard-coding the enum values:

https://lore.kernel.org/r/tencent_4C0B445E0305A18FACA04B4A959B57835107@qq.com

However, this doesn't work as the compiler then complains about an
incomplete type definition in the function prototype. Instead, just add a
local definition of the enum to the selftest code.

Fixes: b06b45e82b59 ("selftests/bpf: add tests for bpf_ct_set_nat_info kfunc")
Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com>
---
 tools/testing/selftests/bpf/progs/test_bpf_nf.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/tools/testing/selftests/bpf/progs/test_bpf_nf.c b/tools/testing/selftests/bpf/progs/test_bpf_nf.c
index 227e85e85dda..6350d11ec6f6 100644
--- a/tools/testing/selftests/bpf/progs/test_bpf_nf.c
+++ b/tools/testing/selftests/bpf/progs/test_bpf_nf.c
@@ -43,6 +43,11 @@ struct bpf_ct_opts___local {
 	u8 reserved[3];
 } __attribute__((preserve_access_index));
 
+enum nf_nat_manip_type {
+	NF_NAT_MANIP_SRC,
+	NF_NAT_MANIP_DST
+};
+
 struct nf_conn *bpf_xdp_ct_alloc(struct xdp_md *, struct bpf_sock_tuple *, u32,
 				 struct bpf_ct_opts___local *, u32) __ksym;
 struct nf_conn *bpf_xdp_ct_lookup(struct xdp_md *, struct bpf_sock_tuple *, u32,
-- 
2.38.1

Re: [PATCH bpf 1/2] bpf: Add dummy type reference to nf_conn___init to fix type deduplication

[Yonghong Song]

On 11/30/22 6:42 AM, Toke Høiland-Jørgensen wrote:
> The bpf_ct_set_nat_info() kfunc is defined in the nf_nat.ko module, and
> takes as a parameter the nf_conn___init struct, which is allocated through
> the bpf_xdp_ct_alloc() helper defined in the nf_conntrack.ko module.
> However, because kernel modules can't deduplicate BTF types between each
> other, and the nf_conn___init struct is not referenced anywhere in vmlinux
> BTF, this leads to two distinct BTF IDs for the same type (one in each
> module). This confuses the verifier, as described here:
> 
> https://lore.kernel.org/all/87leoh372s.fsf@toke.dk/

We might have similar issues later for other types.
Not sure whether the root cause is in libbpf or verifier. But we know
the kfunc from (module, btf_id), so for arguments, we could first
search the corresponding module and then vmlinux for btf_id matching?
This way we might fix potential other cases?

> 
> As a workaround, add a dummy pointer to the type in net/filter.c, so the
> type definition gets included in vmlinux BTF. This way, both modules can
> refer to the same type ID (as they both build on top of vmlinux BTF), and
> the verifier is no longer confused.
> 
> Fixes: 820dc0523e05 ("net: netfilter: move bpf_ct_set_nat_info kfunc in nf_nat_bpf.c")
> Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com>
> ---
>   net/core/filter.c | 15 +++++++++++++++
>   1 file changed, 15 insertions(+)
> 
> diff --git a/net/core/filter.c b/net/core/filter.c
> index bb0136e7a8e4..1bdf9efe8593 100644
> --- a/net/core/filter.c
> +++ b/net/core/filter.c
> @@ -80,6 +80,7 @@
>   #include <net/tls.h>
>   #include <net/xdp.h>
>   #include <net/mptcp.h>
> +#include <net/netfilter/nf_conntrack_bpf.h>
>   
>   static const struct bpf_func_proto *
>   bpf_sk_base_func_proto(enum bpf_func_id func_id);
> @@ -11531,3 +11532,17 @@ bpf_sk_base_func_proto(enum bpf_func_id func_id)
>   
>   	return func;
>   }
> +
> +#if IS_MODULE(CONFIG_NF_CONNTRACK) && IS_ENABLED(CONFIG_DEBUG_INFO_BTF_MODULES)
> +/* The nf_conn___init type is used in the NF_CONNTRACK kfuncs. The kfuncs are
> + * defined in two different modules, and we want to be able to use them
> + * interchangably with the same BTF type ID. Because modules can't de-duplicate
> + * BTF IDs between each other, we need the type to be referenced in the vmlinux
> + * BTF or the verifier will get confused about the different types. So we add
> + * this dummy pointer to serve as a type reference which will be included in
> + * vmlinux BTF, allowing both modules to refer to the same type ID.
> + *
> + * We use a pointer as that is smaller than an instance of the struct.
> + */
> +const struct nf_conn___init *ctinit;
> +#endif

Re: [PATCH bpf 1/2] bpf: Add dummy type reference to nf_conn___init to fix type deduplication

[Andrii Nakryiko]

On Wed, Nov 30, 2022 at 6:42 AM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>
> The bpf_ct_set_nat_info() kfunc is defined in the nf_nat.ko module, and
> takes as a parameter the nf_conn___init struct, which is allocated through
> the bpf_xdp_ct_alloc() helper defined in the nf_conntrack.ko module.
> However, because kernel modules can't deduplicate BTF types between each
> other, and the nf_conn___init struct is not referenced anywhere in vmlinux
> BTF, this leads to two distinct BTF IDs for the same type (one in each
> module). This confuses the verifier, as described here:
>

Argh, shouldn't have wasted writing [1], but oh well.

  [1] https://lore.kernel.org/bpf/CAEf4Bza2xDZ45kxxa3dg1C_RWE=UB5UFYEuFp6rbXgX=LRHv-A@mail.gmail.com/

> https://lore.kernel.org/all/87leoh372s.fsf@toke.dk/
>
> As a workaround, add a dummy pointer to the type in net/filter.c, so the
> type definition gets included in vmlinux BTF. This way, both modules can
> refer to the same type ID (as they both build on top of vmlinux BTF), and
> the verifier is no longer confused.
>
> Fixes: 820dc0523e05 ("net: netfilter: move bpf_ct_set_nat_info kfunc in nf_nat_bpf.c")
> Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com>
> ---
>  net/core/filter.c | 15 +++++++++++++++
>  1 file changed, 15 insertions(+)
>
> diff --git a/net/core/filter.c b/net/core/filter.c
> index bb0136e7a8e4..1bdf9efe8593 100644
> --- a/net/core/filter.c
> +++ b/net/core/filter.c
> @@ -80,6 +80,7 @@
>  #include <net/tls.h>
>  #include <net/xdp.h>
>  #include <net/mptcp.h>
> +#include <net/netfilter/nf_conntrack_bpf.h>
>
>  static const struct bpf_func_proto *
>  bpf_sk_base_func_proto(enum bpf_func_id func_id);
> @@ -11531,3 +11532,17 @@ bpf_sk_base_func_proto(enum bpf_func_id func_id)
>
>         return func;
>  }
> +
> +#if IS_MODULE(CONFIG_NF_CONNTRACK) && IS_ENABLED(CONFIG_DEBUG_INFO_BTF_MODULES)
> +/* The nf_conn___init type is used in the NF_CONNTRACK kfuncs. The kfuncs are
> + * defined in two different modules, and we want to be able to use them
> + * interchangably with the same BTF type ID. Because modules can't de-duplicate
> + * BTF IDs between each other, we need the type to be referenced in the vmlinux
> + * BTF or the verifier will get confused about the different types. So we add
> + * this dummy pointer to serve as a type reference which will be included in
> + * vmlinux BTF, allowing both modules to refer to the same type ID.
> + *
> + * We use a pointer as that is smaller than an instance of the struct.
> + */
> +const struct nf_conn___init *ctinit;
> +#endif

Use BTF_TYPE_EMIT() instead maybe?

> --
> 2.38.1
>

Re: [PATCH bpf 2/2] selftests/bpf: Add local definition of enum nf_nat_manip_type to bpf_nf test

[Andrii Nakryiko]

On Wed, Nov 30, 2022 at 6:42 AM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>
> The bpf_nf selftest calls the bpf_ct_set_nat_info() kfunc, which takes a
> parameter of type enum nf_nat_manip_type. However, if the nf_nat code is
> compiled as a module, that enum is not defined in vmlinux BTF, and
> compilation of the selftest fails.
>
> A previous patch suggested just hard-coding the enum values:
>
> https://lore.kernel.org/r/tencent_4C0B445E0305A18FACA04B4A959B57835107@qq.com
>
> However, this doesn't work as the compiler then complains about an
> incomplete type definition in the function prototype. Instead, just add a
> local definition of the enum to the selftest code.
>
> Fixes: b06b45e82b59 ("selftests/bpf: add tests for bpf_ct_set_nat_info kfunc")
> Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com>
> ---
>  tools/testing/selftests/bpf/progs/test_bpf_nf.c | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/tools/testing/selftests/bpf/progs/test_bpf_nf.c b/tools/testing/selftests/bpf/progs/test_bpf_nf.c
> index 227e85e85dda..6350d11ec6f6 100644
> --- a/tools/testing/selftests/bpf/progs/test_bpf_nf.c
> +++ b/tools/testing/selftests/bpf/progs/test_bpf_nf.c
> @@ -43,6 +43,11 @@ struct bpf_ct_opts___local {
>         u8 reserved[3];
>  } __attribute__((preserve_access_index));
>
> +enum nf_nat_manip_type {
> +       NF_NAT_MANIP_SRC,
> +       NF_NAT_MANIP_DST
> +};
> +

and enum redefinition error if vmlinux.h already defines it?...

>  struct nf_conn *bpf_xdp_ct_alloc(struct xdp_md *, struct bpf_sock_tuple *, u32,
>                                  struct bpf_ct_opts___local *, u32) __ksym;
>  struct nf_conn *bpf_xdp_ct_lookup(struct xdp_md *, struct bpf_sock_tuple *, u32,
> --
> 2.38.1
>

Re: [PATCH bpf 2/2] selftests/bpf: Add local definition of enum nf_nat_manip_type to bpf_nf test

[Andrii Nakryiko]

On Wed, Nov 30, 2022 at 5:18 PM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Wed, Nov 30, 2022 at 6:42 AM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
> >
> > The bpf_nf selftest calls the bpf_ct_set_nat_info() kfunc, which takes a
> > parameter of type enum nf_nat_manip_type. However, if the nf_nat code is
> > compiled as a module, that enum is not defined in vmlinux BTF, and
> > compilation of the selftest fails.
> >
> > A previous patch suggested just hard-coding the enum values:
> >
> > https://lore.kernel.org/r/tencent_4C0B445E0305A18FACA04B4A959B57835107@qq.com
> >
> > However, this doesn't work as the compiler then complains about an
> > incomplete type definition in the function prototype. Instead, just add a
> > local definition of the enum to the selftest code.
> >
> > Fixes: b06b45e82b59 ("selftests/bpf: add tests for bpf_ct_set_nat_info kfunc")
> > Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com>
> > ---
> >  tools/testing/selftests/bpf/progs/test_bpf_nf.c | 5 +++++
> >  1 file changed, 5 insertions(+)
> >
> > diff --git a/tools/testing/selftests/bpf/progs/test_bpf_nf.c b/tools/testing/selftests/bpf/progs/test_bpf_nf.c
> > index 227e85e85dda..6350d11ec6f6 100644
> > --- a/tools/testing/selftests/bpf/progs/test_bpf_nf.c
> > +++ b/tools/testing/selftests/bpf/progs/test_bpf_nf.c
> > @@ -43,6 +43,11 @@ struct bpf_ct_opts___local {
> >         u8 reserved[3];
> >  } __attribute__((preserve_access_index));
> >
> > +enum nf_nat_manip_type {
> > +       NF_NAT_MANIP_SRC,
> > +       NF_NAT_MANIP_DST
> > +};
> > +
>
> and enum redefinition error if vmlinux.h already defines it?...


... which is apparently proven by our CI already:

  [0] https://github.com/kernel-patches/bpf/actions/runs/3584446939/jobs/6031141757

>
> >  struct nf_conn *bpf_xdp_ct_alloc(struct xdp_md *, struct bpf_sock_tuple *, u32,
> >                                  struct bpf_ct_opts___local *, u32) __ksym;
> >  struct nf_conn *bpf_xdp_ct_lookup(struct xdp_md *, struct bpf_sock_tuple *, u32,
> > --
> > 2.38.1
> >

Re: [PATCH bpf 1/2] bpf: Add dummy type reference to nf_conn___init to fix type deduplication

[Toke Høiland-Jørgensen]

Andrii Nakryiko <andrii.nakryiko@gmail.com> writes:

> On Wed, Nov 30, 2022 at 6:42 AM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>>
>> The bpf_ct_set_nat_info() kfunc is defined in the nf_nat.ko module, and
>> takes as a parameter the nf_conn___init struct, which is allocated through
>> the bpf_xdp_ct_alloc() helper defined in the nf_conntrack.ko module.
>> However, because kernel modules can't deduplicate BTF types between each
>> other, and the nf_conn___init struct is not referenced anywhere in vmlinux
>> BTF, this leads to two distinct BTF IDs for the same type (one in each
>> module). This confuses the verifier, as described here:
>>
>
> Argh, shouldn't have wasted writing [1], but oh well.
>
>   [1] https://lore.kernel.org/bpf/CAEf4Bza2xDZ45kxxa3dg1C_RWE=UB5UFYEuFp6rbXgX=LRHv-A@mail.gmail.com/

Ah, yeah, crossed streams; as you can see I came to the same conclusion
wrt types being conceptually independent.

>> https://lore.kernel.org/all/87leoh372s.fsf@toke.dk/
>>
>> As a workaround, add a dummy pointer to the type in net/filter.c, so the
>> type definition gets included in vmlinux BTF. This way, both modules can
>> refer to the same type ID (as they both build on top of vmlinux BTF), and
>> the verifier is no longer confused.
>>
>> Fixes: 820dc0523e05 ("net: netfilter: move bpf_ct_set_nat_info kfunc in nf_nat_bpf.c")
>> Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com>
>> ---
>>  net/core/filter.c | 15 +++++++++++++++
>>  1 file changed, 15 insertions(+)
>>
>> diff --git a/net/core/filter.c b/net/core/filter.c
>> index bb0136e7a8e4..1bdf9efe8593 100644
>> --- a/net/core/filter.c
>> +++ b/net/core/filter.c
>> @@ -80,6 +80,7 @@
>>  #include <net/tls.h>
>>  #include <net/xdp.h>
>>  #include <net/mptcp.h>
>> +#include <net/netfilter/nf_conntrack_bpf.h>
>>
>>  static const struct bpf_func_proto *
>>  bpf_sk_base_func_proto(enum bpf_func_id func_id);
>> @@ -11531,3 +11532,17 @@ bpf_sk_base_func_proto(enum bpf_func_id func_id)
>>
>>         return func;
>>  }
>> +
>> +#if IS_MODULE(CONFIG_NF_CONNTRACK) && IS_ENABLED(CONFIG_DEBUG_INFO_BTF_MODULES)
>> +/* The nf_conn___init type is used in the NF_CONNTRACK kfuncs. The kfuncs are
>> + * defined in two different modules, and we want to be able to use them
>> + * interchangably with the same BTF type ID. Because modules can't de-duplicate
>> + * BTF IDs between each other, we need the type to be referenced in the vmlinux
>> + * BTF or the verifier will get confused about the different types. So we add
>> + * this dummy pointer to serve as a type reference which will be included in
>> + * vmlinux BTF, allowing both modules to refer to the same type ID.
>> + *
>> + * We use a pointer as that is smaller than an instance of the struct.
>> + */
>> +const struct nf_conn___init *ctinit;
>> +#endif
>
> Use BTF_TYPE_EMIT() instead maybe?

Ah, TIL about that macro; thanks, will fix!

-Toke

Re: [PATCH bpf 2/2] selftests/bpf: Add local definition of enum nf_nat_manip_type to bpf_nf test

[Toke Høiland-Jørgensen]

Andrii Nakryiko <andrii.nakryiko@gmail.com> writes:

> On Wed, Nov 30, 2022 at 5:18 PM Andrii Nakryiko
> <andrii.nakryiko@gmail.com> wrote:
>>
>> On Wed, Nov 30, 2022 at 6:42 AM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>> >
>> > The bpf_nf selftest calls the bpf_ct_set_nat_info() kfunc, which takes a
>> > parameter of type enum nf_nat_manip_type. However, if the nf_nat code is
>> > compiled as a module, that enum is not defined in vmlinux BTF, and
>> > compilation of the selftest fails.
>> >
>> > A previous patch suggested just hard-coding the enum values:
>> >
>> > https://lore.kernel.org/r/tencent_4C0B445E0305A18FACA04B4A959B57835107@qq.com
>> >
>> > However, this doesn't work as the compiler then complains about an
>> > incomplete type definition in the function prototype. Instead, just add a
>> > local definition of the enum to the selftest code.
>> >
>> > Fixes: b06b45e82b59 ("selftests/bpf: add tests for bpf_ct_set_nat_info kfunc")
>> > Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com>
>> > ---
>> >  tools/testing/selftests/bpf/progs/test_bpf_nf.c | 5 +++++
>> >  1 file changed, 5 insertions(+)
>> >
>> > diff --git a/tools/testing/selftests/bpf/progs/test_bpf_nf.c b/tools/testing/selftests/bpf/progs/test_bpf_nf.c
>> > index 227e85e85dda..6350d11ec6f6 100644
>> > --- a/tools/testing/selftests/bpf/progs/test_bpf_nf.c
>> > +++ b/tools/testing/selftests/bpf/progs/test_bpf_nf.c
>> > @@ -43,6 +43,11 @@ struct bpf_ct_opts___local {
>> >         u8 reserved[3];
>> >  } __attribute__((preserve_access_index));
>> >
>> > +enum nf_nat_manip_type {
>> > +       NF_NAT_MANIP_SRC,
>> > +       NF_NAT_MANIP_DST
>> > +};
>> > +
>>
>> and enum redefinition error if vmlinux.h already defines it?...
>
>
> ... which is apparently proven by our CI already:
>
>   [0] https://github.com/kernel-patches/bpf/actions/runs/3584446939/jobs/6031141757

Doh *facepalm*! Will fix...

-Toke