[PATCH bpf-next] bpf: pack struct bpf_fib_lookup

[PATCH bpf-next] bpf: pack struct bpf_fib_lookup

[Anton Protopopov]

The struct bpf_fib_lookup is supposed to be of size 64. A recent commit
59b418c7063d ("bpf: Add a check for struct bpf_fib_lookup size") added
a static assertion to check this property so that future changes to the
structure will not accidentally break this assumption.

As it immediately turned out, on some 32-bit arm systems, when AEABI=n,
the total size of the structure was equal to 68, see [1]. This happened
because the bpf_fib_lookup structure contains a union of two 16-bit
fields:

    union {
            __u16 tot_len;
            __u16 mtu_result;
    };

which was supposed to compile to a 16-bit-aligned 16-bit field. On the
aforementioned setups it was instead both aligned and padded to 32-bits.

Declare this inner union as __attribute__((packed, aligned(2))) such
that it always is of size 2 and is aligned to 16 bits.

  [1] https://lore.kernel.org/all/CA+G9fYtsoP51f-oP_Sp5MOq-Ffv8La2RztNpwvE6+R1VtFiLrw@mail.gmail.com/#t

Signed-off-by: Anton Protopopov <aspsk@isovalent.com>
---
 include/uapi/linux/bpf.h       | 2 +-
 tools/include/uapi/linux/bpf.h | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index 79c548276b6b..6fe9f11c8abe 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -7157,7 +7157,7 @@ struct bpf_fib_lookup {
 
 		/* output: MTU value */
 		__u16	mtu_result;
-	};
+	} __attribute__((packed, aligned(2)));
 	/* input: L3 device index for lookup
 	 * output: device index from FIB lookup
 	 */
diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index 79c548276b6b..6fe9f11c8abe 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -7157,7 +7157,7 @@ struct bpf_fib_lookup {
 
 		/* output: MTU value */
 		__u16	mtu_result;
-	};
+	} __attribute__((packed, aligned(2)));
 	/* input: L3 device index for lookup
 	 * output: device index from FIB lookup
 	 */
-- 
2.34.1

Re: [PATCH bpf-next] bpf: pack struct bpf_fib_lookup

[Alexander Lobakin]

From: Anton Protopopov <aspsk@isovalent.com>
Date: Wed,  3 Apr 2024 14:33:03 +0200

> The struct bpf_fib_lookup is supposed to be of size 64. A recent commit
> 59b418c7063d ("bpf: Add a check for struct bpf_fib_lookup size") added
> a static assertion to check this property so that future changes to the
> structure will not accidentally break this assumption.
> 
> As it immediately turned out, on some 32-bit arm systems, when AEABI=n,
> the total size of the structure was equal to 68, see [1]. This happened
> because the bpf_fib_lookup structure contains a union of two 16-bit
> fields:
> 
>     union {
>             __u16 tot_len;
>             __u16 mtu_result;
>     };
> 
> which was supposed to compile to a 16-bit-aligned 16-bit field. On the
> aforementioned setups it was instead both aligned and padded to 32-bits.
> 
> Declare this inner union as __attribute__((packed, aligned(2))) such
> that it always is of size 2 and is aligned to 16 bits.
> 
>   [1] https://lore.kernel.org/all/CA+G9fYtsoP51f-oP_Sp5MOq-Ffv8La2RztNpwvE6+R1VtFiLrw@mail.gmail.com/#t
> 
> Signed-off-by: Anton Protopopov <aspsk@isovalent.com>

Reviewed-by: Alexander Lobakin <aleksander.lobakin@intel.com>

> ---
>  include/uapi/linux/bpf.h       | 2 +-
>  tools/include/uapi/linux/bpf.h | 2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
> index 79c548276b6b..6fe9f11c8abe 100644
> --- a/include/uapi/linux/bpf.h
> +++ b/include/uapi/linux/bpf.h
> @@ -7157,7 +7157,7 @@ struct bpf_fib_lookup {
>  
>  		/* output: MTU value */
>  		__u16	mtu_result;
> -	};
> +	} __attribute__((packed, aligned(2)));
>  	/* input: L3 device index for lookup
>  	 * output: device index from FIB lookup
>  	 */
> diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
> index 79c548276b6b..6fe9f11c8abe 100644
> --- a/tools/include/uapi/linux/bpf.h
> +++ b/tools/include/uapi/linux/bpf.h
> @@ -7157,7 +7157,7 @@ struct bpf_fib_lookup {
>  
>  		/* output: MTU value */
>  		__u16	mtu_result;
> -	};
> +	} __attribute__((packed, aligned(2)));
>  	/* input: L3 device index for lookup
>  	 * output: device index from FIB lookup
>  	 */

Thanks,
Olek

Re: [PATCH bpf-next] bpf: pack struct bpf_fib_lookup

[Daniel Borkmann]

On 4/3/24 2:33 PM, Anton Protopopov wrote:
> The struct bpf_fib_lookup is supposed to be of size 64. A recent commit
> 59b418c7063d ("bpf: Add a check for struct bpf_fib_lookup size") added
> a static assertion to check this property so that future changes to the
> structure will not accidentally break this assumption.
> 
> As it immediately turned out, on some 32-bit arm systems, when AEABI=n,
> the total size of the structure was equal to 68, see [1]. This happened
> because the bpf_fib_lookup structure contains a union of two 16-bit
> fields:
> 
>      union {
>              __u16 tot_len;
>              __u16 mtu_result;
>      };
> 
> which was supposed to compile to a 16-bit-aligned 16-bit field. On the
> aforementioned setups it was instead both aligned and padded to 32-bits.
> 
> Declare this inner union as __attribute__((packed, aligned(2))) such
> that it always is of size 2 and is aligned to 16 bits.
> 
>    [1] https://lore.kernel.org/all/CA+G9fYtsoP51f-oP_Sp5MOq-Ffv8La2RztNpwvE6+R1VtFiLrw@mail.gmail.com/#t
> 
> Signed-off-by: Anton Protopopov <aspsk@isovalent.com>

Reported-by: Naresh Kamboju <naresh.kamboju@linaro.org>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>

Re: [PATCH bpf-next] bpf: pack struct bpf_fib_lookup

[Andrii Nakryiko]

On Wed, Apr 3, 2024 at 6:14 AM Daniel Borkmann <daniel@iogearbox.net> wrote:
>
> On 4/3/24 2:33 PM, Anton Protopopov wrote:
> > The struct bpf_fib_lookup is supposed to be of size 64. A recent commit
> > 59b418c7063d ("bpf: Add a check for struct bpf_fib_lookup size") added
> > a static assertion to check this property so that future changes to the
> > structure will not accidentally break this assumption.
> >
> > As it immediately turned out, on some 32-bit arm systems, when AEABI=n,
> > the total size of the structure was equal to 68, see [1]. This happened
> > because the bpf_fib_lookup structure contains a union of two 16-bit
> > fields:
> >
> >      union {
> >              __u16 tot_len;
> >              __u16 mtu_result;
> >      };
> >
> > which was supposed to compile to a 16-bit-aligned 16-bit field. On the
> > aforementioned setups it was instead both aligned and padded to 32-bits.
> >
> > Declare this inner union as __attribute__((packed, aligned(2))) such
> > that it always is of size 2 and is aligned to 16 bits.
> >
> >    [1] https://lore.kernel.org/all/CA+G9fYtsoP51f-oP_Sp5MOq-Ffv8La2RztNpwvE6+R1VtFiLrw@mail.gmail.com/#t
> >
> > Signed-off-by: Anton Protopopov <aspsk@isovalent.com>
>
> Reported-by: Naresh Kamboju <naresh.kamboju@linaro.org>
> Acked-by: Daniel Borkmann <daniel@iogearbox.net>

Let's continue the tag fest :) Also fixes tag:

Fixes: e1850ea9bd9e ("bpf: bpf_fib_lookup return MTU value as output
when looked up")

Should this also go through the bpf tree instead of bpf-next?

Re: [PATCH bpf-next] bpf: pack struct bpf_fib_lookup

[Arnd Bergmann]

On Wed, Apr 3, 2024, at 14:33, Anton Protopopov wrote:
> The struct bpf_fib_lookup is supposed to be of size 64. A recent commit
> 59b418c7063d ("bpf: Add a check for struct bpf_fib_lookup size") added
> a static assertion to check this property so that future changes to the
> structure will not accidentally break this assumption.
>
> As it immediately turned out, on some 32-bit arm systems, when AEABI=n,
> the total size of the structure was equal to 68, see [1]. This happened
> because the bpf_fib_lookup structure contains a union of two 16-bit
> fields:
>
>     union {
>             __u16 tot_len;
>             __u16 mtu_result;
>     };

This union was introduced three years ago, so fixing it now means
another incompatible ABI change. While you clearly change it back
to what it should have been the entire time, it's not obvious that
this is the correct thing to do after three years.

I do wonder to what degree we still want to care about OABI
at all. As I mentioned, I stopped testing it myself because it
is no longer relevant to most people, and there are probably
a number of other ABI issues.

> which was supposed to compile to a 16-bit-aligned 16-bit field. On the
> aforementioned setups it was instead both aligned and padded to 32-bits.
>
> Declare this inner union as __attribute__((packed, aligned(2))) such
> that it always is of size 2 and is aligned to 16 bits.

I think you probably want 32-bit alignment for the structure,
to keep the ABI unchanged on all other architectures.

     Arnd

Re: [PATCH bpf-next] bpf: pack struct bpf_fib_lookup

[Daniel Borkmann]

On 4/3/24 10:09 PM, Arnd Bergmann wrote:
> On Wed, Apr 3, 2024, at 14:33, Anton Protopopov wrote:
>> The struct bpf_fib_lookup is supposed to be of size 64. A recent commit
>> 59b418c7063d ("bpf: Add a check for struct bpf_fib_lookup size") added
>> a static assertion to check this property so that future changes to the
>> structure will not accidentally break this assumption.
>>
>> As it immediately turned out, on some 32-bit arm systems, when AEABI=n,
>> the total size of the structure was equal to 68, see [1]. This happened
>> because the bpf_fib_lookup structure contains a union of two 16-bit
>> fields:
>>
>>      union {
>>              __u16 tot_len;
>>              __u16 mtu_result;
>>      };
> 
> This union was introduced three years ago, so fixing it now means
> another incompatible ABI change. While you clearly change it back
> to what it should have been the entire time, it's not obvious that
> this is the correct thing to do after three years.
> 
> I do wonder to what degree we still want to care about OABI
> at all. As I mentioned, I stopped testing it myself because it
> is no longer relevant to most people, and there are probably
> a number of other ABI issues.
> 
>> which was supposed to compile to a 16-bit-aligned 16-bit field. On the
>> aforementioned setups it was instead both aligned and padded to 32-bits.
>>
>> Declare this inner union as __attribute__((packed, aligned(2))) such
>> that it always is of size 2 and is aligned to 16 bits.
> 
> I think you probably want 32-bit alignment for the structure,
> to keep the ABI unchanged on all other architectures.

Fwiw, on x86 nothing should change on this regard, see below pahole dump
before/after. I think similar might be true for other archs as otherwise
we should have seen a kbuild bot complaint on hitting the size assert.

So it seems at this point only for the reported case of AEABI=n ... if it
is no longer relevant to most people, I'd think the likelihood of such
users and users who utilize this specific helper is even smaller. But we
could queue it to bpf-next to give it some time.

Before patch:

struct bpf_fib_lookup {
         __u8                       family;               /*     0     1 */
         __u8                       l4_protocol;          /*     1     1 */
         __be16                     sport;                /*     2     2 */
         __be16                     dport;                /*     4     2 */
         union {
                 __u16              tot_len;              /*     6     2 */
                 __u16              mtu_result;           /*     6     2 */
         };                                               /*     6     2 */
         __u32                      ifindex;              /*     8     4 */
         union {
                 __u8               tos;                  /*    12     1 */
                 __be32             flowinfo;             /*    12     4 */
                 __u32              rt_metric;            /*    12     4 */
         };                                               /*    12     4 */
         union {
                 __be32             ipv4_src;             /*    16     4 */
                 __u32              ipv6_src[4];          /*    16    16 */
         };                                               /*    16    16 */
         union {
                 __be32             ipv4_dst;             /*    32     4 */
                 __u32              ipv6_dst[4];          /*    32    16 */
         };                                               /*    32    16 */
         union {
                 struct {
                         __be16     h_vlan_proto;         /*    48     2 */
                         __be16     h_vlan_TCI;           /*    50     2 */
                 };                                       /*    48     4 */
                 __u32              tbid;                 /*    48     4 */
         };                                               /*    48     4 */
         __u8                       smac[6];              /*    52     6 */
         __u8                       dmac[6];              /*    58     6 */

         /* size: 64, cachelines: 1, members: 12 */
};

After patch:

struct bpf_fib_lookup {
         __u8                       family;               /*     0     1 */
         __u8                       l4_protocol;          /*     1     1 */
         __be16                     sport;                /*     2     2 */
         __be16                     dport;                /*     4     2 */
         union {
                 __u16              tot_len;              /*     6     2 */
                 __u16              mtu_result;           /*     6     2 */
         } __attribute__((__aligned__(2)));               /*     6     2 */
         __u32                      ifindex;              /*     8     4 */
         union {
                 __u8               tos;                  /*    12     1 */
                 __be32             flowinfo;             /*    12     4 */
                 __u32              rt_metric;            /*    12     4 */
         };                                               /*    12     4 */
         union {
                 __be32             ipv4_src;             /*    16     4 */
                 __u32              ipv6_src[4];          /*    16    16 */
         };                                               /*    16    16 */
         union {
                 __be32             ipv4_dst;             /*    32     4 */
                 __u32              ipv6_dst[4];          /*    32    16 */
         };                                               /*    32    16 */
         union {
                 struct {
                         __be16     h_vlan_proto;         /*    48     2 */
                         __be16     h_vlan_TCI;           /*    50     2 */
                 };                                       /*    48     4 */
                 __u32              tbid;                 /*    48     4 */
         };                                               /*    48     4 */
         __u8                       smac[6];              /*    52     6 */
         __u8                       dmac[6];              /*    58     6 */

         /* size: 64, cachelines: 1, members: 12 */
         /* forced alignments: 1 */
} __attribute__((__aligned__(4)));

Thanks,
Daniel

Re: [PATCH bpf-next] bpf: pack struct bpf_fib_lookup

[Arnd Bergmann]

On Wed, Apr 3, 2024, at 23:00, Daniel Borkmann wrote:
> On 4/3/24 10:09 PM, Arnd Bergmann wrote:
>> On Wed, Apr 3, 2024, at 14:33, Anton Protopopov wrote:
>>>
>>> Declare this inner union as __attribute__((packed, aligned(2))) such
>>> that it always is of size 2 and is aligned to 16 bits.
>> 
>> I think you probably want 32-bit alignment for the structure,
>> to keep the ABI unchanged on all other architectures.
>
> Fwiw, on x86 nothing should change on this regard, see below pahole dump
> before/after. I think similar might be true for other archs as otherwise
> we should have seen a kbuild bot complaint on hitting the size assert.

It's not the structure layout that changes, just its alignment.
Of course this is unlikely to cause actual bugs, but if there there
is no real need to change it, I would leave the alignment the same
as before.

        Arnd

Re: [PATCH bpf-next] bpf: pack struct bpf_fib_lookup

[Anton Protopopov]

On 24/04/04 12:31, Arnd Bergmann wrote:
> On Wed, Apr 3, 2024, at 23:00, Daniel Borkmann wrote:
> > On 4/3/24 10:09 PM, Arnd Bergmann wrote:
> >> On Wed, Apr 3, 2024, at 14:33, Anton Protopopov wrote:
> >>>
> >>> Declare this inner union as __attribute__((packed, aligned(2))) such
> >>> that it always is of size 2 and is aligned to 16 bits.
> >> 
> >> I think you probably want 32-bit alignment for the structure,
> >> to keep the ABI unchanged on all other architectures.
> >
> > Fwiw, on x86 nothing should change on this regard, see below pahole dump
> > before/after. I think similar might be true for other archs as otherwise
> > we should have seen a kbuild bot complaint on hitting the size assert.
> 
> It's not the structure layout that changes, just its alignment.
> Of course this is unlikely to cause actual bugs, but if there there
> is no real need to change it, I would leave the alignment the same
> as before.

I think the struct will now be automatically 4-byte aligned, as it
has the following layout:

    struct {
            u8 a;
            u8 b;
            u16 c;
            u16 d;
            union { u16 e; u16 f; } __aligned__(2);
            ...
    };

So if the union is 2-byte aligned, then the struct is automatically
4-byte aligned, because its address is 6 bytes less than the address
of the union. In fact, as Daniel posted above, pahole shows that the
struct actually has __aligned__(4) attribute in the patched version.
I can add explicit __aligned__(4) to make this clear.

>         Arnd

Re: [PATCH bpf-next] bpf: pack struct bpf_fib_lookup

[Alexander Lobakin]

From: Anton Protopopov <aspsk@isovalent.com>
Date: Thu, 4 Apr 2024 09:56:51 +0200

> On 24/04/04 12:31, Arnd Bergmann wrote:
>> On Wed, Apr 3, 2024, at 23:00, Daniel Borkmann wrote:
>>> On 4/3/24 10:09 PM, Arnd Bergmann wrote:
>>>> On Wed, Apr 3, 2024, at 14:33, Anton Protopopov wrote:
>>>>>
>>>>> Declare this inner union as __attribute__((packed, aligned(2))) such
>>>>> that it always is of size 2 and is aligned to 16 bits.
>>>>
>>>> I think you probably want 32-bit alignment for the structure,
>>>> to keep the ABI unchanged on all other architectures.
>>>
>>> Fwiw, on x86 nothing should change on this regard, see below pahole dump
>>> before/after. I think similar might be true for other archs as otherwise
>>> we should have seen a kbuild bot complaint on hitting the size assert.
>>
>> It's not the structure layout that changes, just its alignment.
>> Of course this is unlikely to cause actual bugs, but if there there
>> is no real need to change it, I would leave the alignment the same
>> as before.
> 
> I think the struct will now be automatically 4-byte aligned, as it
> has the following layout:
> 
>     struct {
>             u8 a;
>             u8 b;
>             u16 c;
>             u16 d;
>             union { u16 e; u16 f; } __aligned__(2);
>             ...
>     };
> 
> So if the union is 2-byte aligned, then the struct is automatically
> 4-byte aligned, because its address is 6 bytes less than the address
> of the union. In fact, as Daniel posted above, pahole shows that the
> struct actually has __aligned__(4) attribute in the patched version.
> I can add explicit __aligned__(4) to make this clear.

I think it would be fine to not introduce an explicit attribute as long
as pahole shows the structure is already 4-byte aligned.

Regarding aligning it to 32 or 64 bytes -- the actual benefit can be
checked via scripts/bloat-o-meter. Usually alignment bigger than 8 bytes
don't change anything in the object code, at least on x86_64.

> 
>>         Arnd

Thanks,
Olek

Re: [PATCH bpf-next] bpf: pack struct bpf_fib_lookup

[Arnd Bergmann]

On Thu, Apr 4, 2024, at 09:56, Anton Protopopov wrote:
> On 24/04/04 12:31, Arnd Bergmann wrote:
>> On Wed, Apr 3, 2024, at 23:00, Daniel Borkmann wrote:
>> > On 4/3/24 10:09 PM, Arnd Bergmann wrote:
>> >> On Wed, Apr 3, 2024, at 14:33, Anton Protopopov wrote:
>> >>>
>> >>> Declare this inner union as __attribute__((packed, aligned(2))) such
>> >>> that it always is of size 2 and is aligned to 16 bits.
>> >> 
>> >> I think you probably want 32-bit alignment for the structure,
>> >> to keep the ABI unchanged on all other architectures.
>> >
>> > Fwiw, on x86 nothing should change on this regard, see below pahole dump
>> > before/after. I think similar might be true for other archs as otherwise
>> > we should have seen a kbuild bot complaint on hitting the size assert.
>> 
>> It's not the structure layout that changes, just its alignment.
>> Of course this is unlikely to cause actual bugs, but if there there
>> is no real need to change it, I would leave the alignment the same
>> as before.
>
> I think the struct will now be automatically 4-byte aligned, as it
> has the following layout:
>
>     struct {
>             u8 a;
>             u8 b;
>             u16 c;
>             u16 d;
>             union { u16 e; u16 f; } __aligned__(2);
>             ...
>     };
>
> So if the union is 2-byte aligned, then the struct is automatically
> 4-byte aligned, because its address is 6 bytes less than the address
> of the union. In fact, as Daniel posted above, pahole shows that the
> struct actually has __aligned__(4) attribute in the patched version.
> I can add explicit __aligned__(4) to make this clear.

Ah right. I misread your patch and assumed that you
gave the outer structure 2-byte alignment as well, but
it's only the inner union that needs it, which you did
correctly.

     Arnd

Re: [PATCH bpf-next] bpf: pack struct bpf_fib_lookup

[patchwork-bot+netdevbpf]

Hello:

This patch was applied to bpf/bpf-next.git (master)
by Andrii Nakryiko <andrii@kernel.org>:

On Wed,  3 Apr 2024 14:33:03 +0200 you wrote:
> The struct bpf_fib_lookup is supposed to be of size 64. A recent commit
> 59b418c7063d ("bpf: Add a check for struct bpf_fib_lookup size") added
> a static assertion to check this property so that future changes to the
> structure will not accidentally break this assumption.
> 
> As it immediately turned out, on some 32-bit arm systems, when AEABI=n,
> the total size of the structure was equal to 68, see [1]. This happened
> because the bpf_fib_lookup structure contains a union of two 16-bit
> fields:
> 
> [...]

Here is the summary with links:
  - [bpf-next] bpf: pack struct bpf_fib_lookup
    https://git.kernel.org/bpf/bpf-next/c/f91717007217

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html