Possible build time regression affecting stable kernels

Possible build time regression affecting stable kernels

[Luiz Capitulino]

Hi Paul,

A number of stable kernels recently backported this upstream commit:

"""
commit 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5
Author: Paul Moore <paul@paul-moore.com>
Date:   Wed Apr 12 13:29:11 2023 -0400

     selinux: ensure av_permissions.h is built when needed
"""

We're seeing a build issue with this commit where the "crash" tool will fail
to start, it complains that the vmlinux image and /proc/version don't match.

A minimum reproducer would be having "make" version before 4.3 and building
the kernel with:

$ make bzImages
$ make modules

Then compare the version strings in the bzImage and vmlinux images,
we can use "strings" for this. For example, in the 5.10.181 kernel I get:

$ strings vmlinux | egrep '^Linux version'
Linux version 5.10.181 (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #2 SMP Thu Jun 1 01:26:38 UTC 2023

$ strings ./arch/x86_64/boot/bzImage | egrep 'ld version'
5.10.181 (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:23:59 UTC 2023

The version string in the bzImage doesn't have the "Linux version" part, but
I think this is added by the kernel when printing. If you compare the strings,
you'll see that they have a different build date and the "#1" and "#2" are
different.

This only happens with commit 4ce1f694eb5 applied and older "make", in my case I
have "make" version 3.82.

If I revert 4ce1f694eb5 or use "make" version 4.3 I get identical strings (except
for the "Linux version" part):

$ strings vmlinux | egrep '^Linux version'
Linux version 5.10.181+ (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:29:11 UTC 2023

$ strings ./arch/x86_64/boot/bzImage | egrep 'ld version'
5.10.181+ (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:29:11 UTC 2023

Maybe the grouped target usage in 4ce1f694eb5 with older "make" is causing a
rebuild of the vmlinux image in "make modules"? If yes, is this expected?

I'm afraid this issue could be high impact for distros with older user-space.

- Luiz

Re: Possible build time regression affecting stable kernels

[Greg KH]

On Wed, May 31, 2023 at 10:12:40PM -0400, Luiz Capitulino wrote:
> Hi Paul,
> 
> A number of stable kernels recently backported this upstream commit:
> 
> """
> commit 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5
> Author: Paul Moore <paul@paul-moore.com>
> Date:   Wed Apr 12 13:29:11 2023 -0400
> 
>     selinux: ensure av_permissions.h is built when needed
> """
> 
> We're seeing a build issue with this commit where the "crash" tool will fail
> to start, it complains that the vmlinux image and /proc/version don't match.
> 
> A minimum reproducer would be having "make" version before 4.3 and building
> the kernel with:
> 
> $ make bzImages
> $ make modules
> 
> Then compare the version strings in the bzImage and vmlinux images,
> we can use "strings" for this. For example, in the 5.10.181 kernel I get:
> 
> $ strings vmlinux | egrep '^Linux version'
> Linux version 5.10.181 (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #2 SMP Thu Jun 1 01:26:38 UTC 2023
> 
> $ strings ./arch/x86_64/boot/bzImage | egrep 'ld version'
> 5.10.181 (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:23:59 UTC 2023
> 
> The version string in the bzImage doesn't have the "Linux version" part, but
> I think this is added by the kernel when printing. If you compare the strings,
> you'll see that they have a different build date and the "#1" and "#2" are
> different.
> 
> This only happens with commit 4ce1f694eb5 applied and older "make", in my case I
> have "make" version 3.82.
> 
> If I revert 4ce1f694eb5 or use "make" version 4.3 I get identical strings (except
> for the "Linux version" part):
> 
> $ strings vmlinux | egrep '^Linux version'
> Linux version 5.10.181+ (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:29:11 UTC 2023
> 
> $ strings ./arch/x86_64/boot/bzImage | egrep 'ld version'
> 5.10.181+ (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:29:11 UTC 2023
> 
> Maybe the grouped target usage in 4ce1f694eb5 with older "make" is causing a
> rebuild of the vmlinux image in "make modules"? If yes, is this expected?
> 
> I'm afraid this issue could be high impact for distros with older user-space.

Is this issue also in 6.4-rc1 where this change came from?  What about
the other stable releases?

thanks,

greg k-h

Re: Possible build time regression affecting stable kernels

[Luiz Capitulino]

On 2023-06-01 02:06, Greg KH wrote:

> 
> 
> 
> On Wed, May 31, 2023 at 10:12:40PM -0400, Luiz Capitulino wrote:
>> Hi Paul,
>>
>> A number of stable kernels recently backported this upstream commit:
>>
>> """
>> commit 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5
>> Author: Paul Moore <paul@paul-moore.com>
>> Date:   Wed Apr 12 13:29:11 2023 -0400
>>
>>      selinux: ensure av_permissions.h is built when needed
>> """
>>
>> We're seeing a build issue with this commit where the "crash" tool will fail
>> to start, it complains that the vmlinux image and /proc/version don't match.
>>
>> A minimum reproducer would be having "make" version before 4.3 and building
>> the kernel with:
>>
>> $ make bzImages
>> $ make modules
>>
>> Then compare the version strings in the bzImage and vmlinux images,
>> we can use "strings" for this. For example, in the 5.10.181 kernel I get:
>>
>> $ strings vmlinux | egrep '^Linux version'
>> Linux version 5.10.181 (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #2 SMP Thu Jun 1 01:26:38 UTC 2023
>>
>> $ strings ./arch/x86_64/boot/bzImage | egrep 'ld version'
>> 5.10.181 (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:23:59 UTC 2023
>>
>> The version string in the bzImage doesn't have the "Linux version" part, but
>> I think this is added by the kernel when printing. If you compare the strings,
>> you'll see that they have a different build date and the "#1" and "#2" are
>> different.
>>
>> This only happens with commit 4ce1f694eb5 applied and older "make", in my case I
>> have "make" version 3.82.
>>
>> If I revert 4ce1f694eb5 or use "make" version 4.3 I get identical strings (except
>> for the "Linux version" part):
>>
>> $ strings vmlinux | egrep '^Linux version'
>> Linux version 5.10.181+ (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:29:11 UTC 2023
>>
>> $ strings ./arch/x86_64/boot/bzImage | egrep 'ld version'
>> 5.10.181+ (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:29:11 UTC 2023
>>
>> Maybe the grouped target usage in 4ce1f694eb5 with older "make" is causing a
>> rebuild of the vmlinux image in "make modules"? If yes, is this expected?
>>
>> I'm afraid this issue could be high impact for distros with older user-space.
> 
> Is this issue also in 6.4-rc1 where this change came from?  

Yes. I'm reporting this here because I'm more concerned with -stable kernels since
they're more likely to be running on older user-space.

> What about
> the other stable releases?

I tested against: 6.1.31, 5.15.114, 5.10.181 they all reproduce it. I'd guess any
kernel having a backport of 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5 will have this.

One detail though, I tested once with "make binrpm-pkg" and didn't reproduce it. I
think this could mean that it depends on the make targets you use to build the
kernel. But "make bzImage", "make modules", "make modules_install" is pretty
standard (or used to be).

- Luiz

Re: Possible build time regression affecting stable kernels

[Greg KH]

On Thu, Jun 01, 2023 at 09:13:21AM -0400, Luiz Capitulino wrote:
> 
> 
> On 2023-06-01 02:06, Greg KH wrote:
> 
> > 
> > 
> > 
> > On Wed, May 31, 2023 at 10:12:40PM -0400, Luiz Capitulino wrote:
> > > Hi Paul,
> > > 
> > > A number of stable kernels recently backported this upstream commit:
> > > 
> > > """
> > > commit 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5
> > > Author: Paul Moore <paul@paul-moore.com>
> > > Date:   Wed Apr 12 13:29:11 2023 -0400
> > > 
> > >      selinux: ensure av_permissions.h is built when needed
> > > """
> > > 
> > > We're seeing a build issue with this commit where the "crash" tool will fail
> > > to start, it complains that the vmlinux image and /proc/version don't match.
> > > 
> > > A minimum reproducer would be having "make" version before 4.3 and building
> > > the kernel with:
> > > 
> > > $ make bzImages
> > > $ make modules
> > > 
> > > Then compare the version strings in the bzImage and vmlinux images,
> > > we can use "strings" for this. For example, in the 5.10.181 kernel I get:
> > > 
> > > $ strings vmlinux | egrep '^Linux version'
> > > Linux version 5.10.181 (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #2 SMP Thu Jun 1 01:26:38 UTC 2023
> > > 
> > > $ strings ./arch/x86_64/boot/bzImage | egrep 'ld version'
> > > 5.10.181 (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:23:59 UTC 2023
> > > 
> > > The version string in the bzImage doesn't have the "Linux version" part, but
> > > I think this is added by the kernel when printing. If you compare the strings,
> > > you'll see that they have a different build date and the "#1" and "#2" are
> > > different.
> > > 
> > > This only happens with commit 4ce1f694eb5 applied and older "make", in my case I
> > > have "make" version 3.82.
> > > 
> > > If I revert 4ce1f694eb5 or use "make" version 4.3 I get identical strings (except
> > > for the "Linux version" part):
> > > 
> > > $ strings vmlinux | egrep '^Linux version'
> > > Linux version 5.10.181+ (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:29:11 UTC 2023
> > > 
> > > $ strings ./arch/x86_64/boot/bzImage | egrep 'ld version'
> > > 5.10.181+ (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:29:11 UTC 2023
> > > 
> > > Maybe the grouped target usage in 4ce1f694eb5 with older "make" is causing a
> > > rebuild of the vmlinux image in "make modules"? If yes, is this expected?
> > > 
> > > I'm afraid this issue could be high impact for distros with older user-space.
> > 
> > Is this issue also in 6.4-rc1 where this change came from?
> 
> Yes. I'm reporting this here because I'm more concerned with -stable kernels since
> they're more likely to be running on older user-space.

Yeah, we are bug-compatible!  :)

When this gets fixed in Linus's tree, I'll be glad to backport the
changes to the other kernels.  Please work with the developers to get
that fixed there.

thanks,

greg k-h

Re: Possible build time regression affecting stable kernels

[Luiz Capitulino]

On 2023-06-01 09:20, Greg KH wrote:

> 
> 
> 
> On Thu, Jun 01, 2023 at 09:13:21AM -0400, Luiz Capitulino wrote:
>>
>>
>> On 2023-06-01 02:06, Greg KH wrote:
>>
>>>
>>>
>>>
>>> On Wed, May 31, 2023 at 10:12:40PM -0400, Luiz Capitulino wrote:
>>>> Hi Paul,
>>>>
>>>> A number of stable kernels recently backported this upstream commit:
>>>>
>>>> """
>>>> commit 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5
>>>> Author: Paul Moore <paul@paul-moore.com>
>>>> Date:   Wed Apr 12 13:29:11 2023 -0400
>>>>
>>>>       selinux: ensure av_permissions.h is built when needed
>>>> """
>>>>
>>>> We're seeing a build issue with this commit where the "crash" tool will fail
>>>> to start, it complains that the vmlinux image and /proc/version don't match.
>>>>
>>>> A minimum reproducer would be having "make" version before 4.3 and building
>>>> the kernel with:
>>>>
>>>> $ make bzImages
>>>> $ make modules
>>>>
>>>> Then compare the version strings in the bzImage and vmlinux images,
>>>> we can use "strings" for this. For example, in the 5.10.181 kernel I get:
>>>>
>>>> $ strings vmlinux | egrep '^Linux version'
>>>> Linux version 5.10.181 (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #2 SMP Thu Jun 1 01:26:38 UTC 2023
>>>>
>>>> $ strings ./arch/x86_64/boot/bzImage | egrep 'ld version'
>>>> 5.10.181 (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:23:59 UTC 2023
>>>>
>>>> The version string in the bzImage doesn't have the "Linux version" part, but
>>>> I think this is added by the kernel when printing. If you compare the strings,
>>>> you'll see that they have a different build date and the "#1" and "#2" are
>>>> different.
>>>>
>>>> This only happens with commit 4ce1f694eb5 applied and older "make", in my case I
>>>> have "make" version 3.82.
>>>>
>>>> If I revert 4ce1f694eb5 or use "make" version 4.3 I get identical strings (except
>>>> for the "Linux version" part):
>>>>
>>>> $ strings vmlinux | egrep '^Linux version'
>>>> Linux version 5.10.181+ (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:29:11 UTC 2023
>>>>
>>>> $ strings ./arch/x86_64/boot/bzImage | egrep 'ld version'
>>>> 5.10.181+ (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:29:11 UTC 2023
>>>>
>>>> Maybe the grouped target usage in 4ce1f694eb5 with older "make" is causing a
>>>> rebuild of the vmlinux image in "make modules"? If yes, is this expected?
>>>>
>>>> I'm afraid this issue could be high impact for distros with older user-space.
>>>
>>> Is this issue also in 6.4-rc1 where this change came from?
>>
>> Yes. I'm reporting this here because I'm more concerned with -stable kernels since
>> they're more likely to be running on older user-space.
> 
> Yeah, we are bug-compatible!  :)
> 
> When this gets fixed in Linus's tree, I'll be glad to backport the
> changes to the other kernels.  Please work with the developers to get
> that fixed there.

OK, I'll report it there, but shouldn't we avoid regressing -stable kernels?

- Luiz

Re: Possible build time regression affecting stable kernels

[Greg KH]

On Thu, Jun 01, 2023 at 09:26:30AM -0400, Luiz Capitulino wrote:
> 
> 
> On 2023-06-01 09:20, Greg KH wrote:
> 
> > 
> > 
> > 
> > On Thu, Jun 01, 2023 at 09:13:21AM -0400, Luiz Capitulino wrote:
> > > 
> > > 
> > > On 2023-06-01 02:06, Greg KH wrote:
> > > 
> > > > 
> > > > 
> > > > 
> > > > On Wed, May 31, 2023 at 10:12:40PM -0400, Luiz Capitulino wrote:
> > > > > Hi Paul,
> > > > > 
> > > > > A number of stable kernels recently backported this upstream commit:
> > > > > 
> > > > > """
> > > > > commit 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5
> > > > > Author: Paul Moore <paul@paul-moore.com>
> > > > > Date:   Wed Apr 12 13:29:11 2023 -0400
> > > > > 
> > > > >       selinux: ensure av_permissions.h is built when needed
> > > > > """
> > > > > 
> > > > > We're seeing a build issue with this commit where the "crash" tool will fail
> > > > > to start, it complains that the vmlinux image and /proc/version don't match.
> > > > > 
> > > > > A minimum reproducer would be having "make" version before 4.3 and building
> > > > > the kernel with:
> > > > > 
> > > > > $ make bzImages
> > > > > $ make modules
> > > > > 
> > > > > Then compare the version strings in the bzImage and vmlinux images,
> > > > > we can use "strings" for this. For example, in the 5.10.181 kernel I get:
> > > > > 
> > > > > $ strings vmlinux | egrep '^Linux version'
> > > > > Linux version 5.10.181 (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #2 SMP Thu Jun 1 01:26:38 UTC 2023
> > > > > 
> > > > > $ strings ./arch/x86_64/boot/bzImage | egrep 'ld version'
> > > > > 5.10.181 (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:23:59 UTC 2023
> > > > > 
> > > > > The version string in the bzImage doesn't have the "Linux version" part, but
> > > > > I think this is added by the kernel when printing. If you compare the strings,
> > > > > you'll see that they have a different build date and the "#1" and "#2" are
> > > > > different.
> > > > > 
> > > > > This only happens with commit 4ce1f694eb5 applied and older "make", in my case I
> > > > > have "make" version 3.82.
> > > > > 
> > > > > If I revert 4ce1f694eb5 or use "make" version 4.3 I get identical strings (except
> > > > > for the "Linux version" part):
> > > > > 
> > > > > $ strings vmlinux | egrep '^Linux version'
> > > > > Linux version 5.10.181+ (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:29:11 UTC 2023
> > > > > 
> > > > > $ strings ./arch/x86_64/boot/bzImage | egrep 'ld version'
> > > > > 5.10.181+ (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:29:11 UTC 2023
> > > > > 
> > > > > Maybe the grouped target usage in 4ce1f694eb5 with older "make" is causing a
> > > > > rebuild of the vmlinux image in "make modules"? If yes, is this expected?
> > > > > 
> > > > > I'm afraid this issue could be high impact for distros with older user-space.
> > > > 
> > > > Is this issue also in 6.4-rc1 where this change came from?
> > > 
> > > Yes. I'm reporting this here because I'm more concerned with -stable kernels since
> > > they're more likely to be running on older user-space.
> > 
> > Yeah, we are bug-compatible!  :)
> > 
> > When this gets fixed in Linus's tree, I'll be glad to backport the
> > changes to the other kernels.  Please work with the developers to get
> > that fixed there.
> 
> OK, I'll report it there, but shouldn't we avoid regressing -stable kernels?

We should avoid regressing Linus's kernel tree just as much.  It's
always been this way, I don't want to revert patches unless they are
really bad if the fix is not in Linus's tree already.  Gives people
"encouragement" to resolve the issue quicker.

thanks,

greg k-h

Re: Possible build time regression affecting stable kernels

[Luiz Capitulino]

On 2023-06-01 10:13, Greg KH wrote:

> 
> 
> 
> On Thu, Jun 01, 2023 at 09:26:30AM -0400, Luiz Capitulino wrote:
>>
>>
>> On 2023-06-01 09:20, Greg KH wrote:
>>
>>>
>>>
>>>
>>> On Thu, Jun 01, 2023 at 09:13:21AM -0400, Luiz Capitulino wrote:
>>>>
>>>>
>>>> On 2023-06-01 02:06, Greg KH wrote:
>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, May 31, 2023 at 10:12:40PM -0400, Luiz Capitulino wrote:
>>>>>> Hi Paul,
>>>>>>
>>>>>> A number of stable kernels recently backported this upstream commit:
>>>>>>
>>>>>> """
>>>>>> commit 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5
>>>>>> Author: Paul Moore <paul@paul-moore.com>
>>>>>> Date:   Wed Apr 12 13:29:11 2023 -0400
>>>>>>
>>>>>>        selinux: ensure av_permissions.h is built when needed
>>>>>> """
>>>>>>
>>>>>> We're seeing a build issue with this commit where the "crash" tool will fail
>>>>>> to start, it complains that the vmlinux image and /proc/version don't match.
>>>>>>
>>>>>> A minimum reproducer would be having "make" version before 4.3 and building
>>>>>> the kernel with:
>>>>>>
>>>>>> $ make bzImages
>>>>>> $ make modules
>>>>>>
>>>>>> Then compare the version strings in the bzImage and vmlinux images,
>>>>>> we can use "strings" for this. For example, in the 5.10.181 kernel I get:
>>>>>>
>>>>>> $ strings vmlinux | egrep '^Linux version'
>>>>>> Linux version 5.10.181 (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #2 SMP Thu Jun 1 01:26:38 UTC 2023
>>>>>>
>>>>>> $ strings ./arch/x86_64/boot/bzImage | egrep 'ld version'
>>>>>> 5.10.181 (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:23:59 UTC 2023
>>>>>>
>>>>>> The version string in the bzImage doesn't have the "Linux version" part, but
>>>>>> I think this is added by the kernel when printing. If you compare the strings,
>>>>>> you'll see that they have a different build date and the "#1" and "#2" are
>>>>>> different.
>>>>>>
>>>>>> This only happens with commit 4ce1f694eb5 applied and older "make", in my case I
>>>>>> have "make" version 3.82.
>>>>>>
>>>>>> If I revert 4ce1f694eb5 or use "make" version 4.3 I get identical strings (except
>>>>>> for the "Linux version" part):
>>>>>>
>>>>>> $ strings vmlinux | egrep '^Linux version'
>>>>>> Linux version 5.10.181+ (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:29:11 UTC 2023
>>>>>>
>>>>>> $ strings ./arch/x86_64/boot/bzImage | egrep 'ld version'
>>>>>> 5.10.181+ (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:29:11 UTC 2023
>>>>>>
>>>>>> Maybe the grouped target usage in 4ce1f694eb5 with older "make" is causing a
>>>>>> rebuild of the vmlinux image in "make modules"? If yes, is this expected?
>>>>>>
>>>>>> I'm afraid this issue could be high impact for distros with older user-space.
>>>>>
>>>>> Is this issue also in 6.4-rc1 where this change came from?
>>>>
>>>> Yes. I'm reporting this here because I'm more concerned with -stable kernels since
>>>> they're more likely to be running on older user-space.
>>>
>>> Yeah, we are bug-compatible!  :)
>>>
>>> When this gets fixed in Linus's tree, I'll be glad to backport the
>>> changes to the other kernels.  Please work with the developers to get
>>> that fixed there.
>>
>> OK, I'll report it there, but shouldn't we avoid regressing -stable kernels?
> 
> We should avoid regressing Linus's kernel tree just as much.  It's
> always been this way, I don't want to revert patches unless they are
> really bad if the fix is not in Linus's tree already.  Gives people
> "encouragement" to resolve the issue quicker.

Fair enough. Paul is in this thread, let's see if he picks it up from here. If not, I'll
report it on lkml.

- Luiz

Re: Possible build time regression affecting stable kernels

[Paul Moore]

On Wed, May 31, 2023 at 10:13 PM Luiz Capitulino <luizcap@amazon.com> wrote:
>
> Hi Paul,
>
> A number of stable kernels recently backported this upstream commit:
>
> """
> commit 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5
> Author: Paul Moore <paul@paul-moore.com>
> Date:   Wed Apr 12 13:29:11 2023 -0400
>
>      selinux: ensure av_permissions.h is built when needed
> """
>
> We're seeing a build issue with this commit where the "crash" tool will fail
> to start, it complains that the vmlinux image and /proc/version don't match.
>
> A minimum reproducer would be having "make" version before 4.3 and building
> the kernel with:
>
> $ make bzImages
> $ make modules

...

> This only happens with commit 4ce1f694eb5 applied and older "make", in my case I
> have "make" version 3.82.
>
> If I revert 4ce1f694eb5 or use "make" version 4.3 I get identical strings (except
> for the "Linux version" part):

Thanks Luiz, this is a fun one :/

Based on a quick search, it looks like the grouped target may be the
cause, especially for older (pre-4.3) versions of make.  Looking
through the rest of the kernel I don't see any other grouped targets,
and in fact the top level Makefile even mentions holding off on using
grouped targets until make v4.3 is common/required.

I don't have an older userspace immediately available, would you mind
trying the fix/patch below to see if it resolves the problem on your
system?  It's a cut-n-paste so the patch may not apply directly, but
it basically just removes the '&' from the make rule, turning it into
an old-fashioned non-grouped target.

diff --git a/security/selinux/Makefile b/security/selinux/Makefile
index 0aecf9334ec3..df35d4ec46f0 100644
--- a/security/selinux/Makefile
+++ b/security/selinux/Makefile
@@ -26,5 +26,5 @@ quiet_cmd_flask = GEN     $(obj)/flask.h $(obj)/av_permissions
.h
      cmd_flask = $< $(obj)/flask.h $(obj)/av_permissions.h

targets += flask.h av_permissions.h
-$(obj)/flask.h $(obj)/av_permissions.h &: scripts/selinux/genheaders/genheaders
FORCE
+$(obj)/flask.h $(obj)/av_permissions.h: scripts/selinux/genheaders/genheaders F
ORCE
       $(call if_changed,flask)

-- 
paul-moore.com

Re: Possible build time regression affecting stable kernels

[Paul Moore]

On Thu, Jun 1, 2023 at 9:20 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> On Thu, Jun 01, 2023 at 09:13:21AM -0400, Luiz Capitulino wrote:

...

> > Yes. I'm reporting this here because I'm more concerned with -stable kernels since
> > they're more likely to be running on older user-space.
>
> Yeah, we are bug-compatible!  :)

While I really don't want to go back into the old arguments about what
does, and does not, get backported to -stable, I do want to ask if
there is some way to signal to the -stable maintainers that a patch
should not be backported?  Anything coming from the LSM, SELinux, or
audit trees that I believe should be backported is explicitly marked
with a stable@vger CC, as documented in stable-kernel-rules.rst,
however it is generally my experience that patches with a 'Fixes:' tag
are generally pulled into the -stable releases as well.

I could start dropping the 'Fixes:' tag from non-stable tagged
commits, but that's a step backwards in my opinion.

I could start replying to every -stable backport email notice, but
that seems like a lot of unnecessary work for something that was never
marked for -stable in the first place.  I'm guessing it would also add
some additional management/testing burden to the -stable folks as
well.

--
paul-moore.com

Re: Possible build time regression affecting stable kernels

[Luiz Capitulino]

[-- Attachment #1: Type: text/plain, Size: 2141 bytes --]



On 2023-06-01 10:27, Paul Moore wrote:

> 
> 
> 
> On Wed, May 31, 2023 at 10:13 PM Luiz Capitulino <luizcap@amazon.com> wrote:
>>
>> Hi Paul,
>>
>> A number of stable kernels recently backported this upstream commit:
>>
>> """
>> commit 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5
>> Author: Paul Moore <paul@paul-moore.com>
>> Date:   Wed Apr 12 13:29:11 2023 -0400
>>
>>       selinux: ensure av_permissions.h is built when needed
>> """
>>
>> We're seeing a build issue with this commit where the "crash" tool will fail
>> to start, it complains that the vmlinux image and /proc/version don't match.
>>
>> A minimum reproducer would be having "make" version before 4.3 and building
>> the kernel with:
>>
>> $ make bzImages
>> $ make modules
> 
> ...
> 
>> This only happens with commit 4ce1f694eb5 applied and older "make", in my case I
>> have "make" version 3.82.
>>
>> If I revert 4ce1f694eb5 or use "make" version 4.3 I get identical strings (except
>> for the "Linux version" part):
> 
> Thanks Luiz, this is a fun one :/

It was a fun to debug TBH :-)

> Based on a quick search, it looks like the grouped target may be the
> cause, especially for older (pre-4.3) versions of make.  Looking
> through the rest of the kernel I don't see any other grouped targets,
> and in fact the top level Makefile even mentions holding off on using
> grouped targets until make v4.3 is common/required.

Exactly.
  
> I don't have an older userspace immediately available, would you mind
> trying the fix/patch below to see if it resolves the problem on your
> system?  It's a cut-n-paste so the patch may not apply directly, but
> it basically just removes the '&' from the make rule, turning it into
> an old-fashioned non-grouped target.

I tried the attached patch on top of latest Linus tree (ac2263b588dffd),
but unfortunately I got the same issue which is puzzling. Reverting
4ce1f694eb5d8ca607fed8542d32a33b4f1217a5 does solve the issue though.

I have no problem trying patches or helping debug, but if you want to
give it a try on reproducing you could try with make-3.82 from:

https://ftp.gnu.org/gnu/make/make-3.82.tar.bz2

- Luiz

[-- Attachment #2: paul.patch --]
[-- Type: text/x-patch, Size: 553 bytes --]

diff --git a/security/selinux/Makefile b/security/selinux/Makefile
index 0aecf9334ec3..df35d4ec46f0 100644
--- a/security/selinux/Makefile
+++ b/security/selinux/Makefile
@@ -26,5 +26,5 @@ quiet_cmd_flask = GEN     $(obj)/flask.h $(obj)/av_permissions.h
       cmd_flask = $< $(obj)/flask.h $(obj)/av_permissions.h
 
 targets += flask.h av_permissions.h
-$(obj)/flask.h $(obj)/av_permissions.h &: scripts/selinux/genheaders/genheaders FORCE
+$(obj)/flask.h $(obj)/av_permissions.h: scripts/selinux/genheaders/genheaders FORCE
 	$(call if_changed,flask)

Re: Possible build time regression affecting stable kernels

[Paul Moore]

On Thu, Jun 1, 2023 at 11:03 AM Luiz Capitulino <luizcap@amazon.com> wrote:
> On 2023-06-01 10:27, Paul Moore wrote:
> > On Wed, May 31, 2023 at 10:13 PM Luiz Capitulino <luizcap@amazon.com> wrote:
> >>
> >> Hi Paul,
> >>
> >> A number of stable kernels recently backported this upstream commit:
> >>
> >> """
> >> commit 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5
> >> Author: Paul Moore <paul@paul-moore.com>
> >> Date:   Wed Apr 12 13:29:11 2023 -0400
> >>
> >>       selinux: ensure av_permissions.h is built when needed
> >> """
> >>
> >> We're seeing a build issue with this commit where the "crash" tool will fail
> >> to start, it complains that the vmlinux image and /proc/version don't match.
> >>
> >> A minimum reproducer would be having "make" version before 4.3 and building
> >> the kernel with:
> >>
> >> $ make bzImages
> >> $ make modules
> >
> > ...
> >
> >> This only happens with commit 4ce1f694eb5 applied and older "make", in my case I
> >> have "make" version 3.82.
> >>
> >> If I revert 4ce1f694eb5 or use "make" version 4.3 I get identical strings (except
> >> for the "Linux version" part):
> >
> > Thanks Luiz, this is a fun one :/
>
> It was a fun to debug TBH :-)
>
> > Based on a quick search, it looks like the grouped target may be the
> > cause, especially for older (pre-4.3) versions of make.  Looking
> > through the rest of the kernel I don't see any other grouped targets,
> > and in fact the top level Makefile even mentions holding off on using
> > grouped targets until make v4.3 is common/required.
>
> Exactly.
>
> > I don't have an older userspace immediately available, would you mind
> > trying the fix/patch below to see if it resolves the problem on your
> > system?  It's a cut-n-paste so the patch may not apply directly, but
> > it basically just removes the '&' from the make rule, turning it into
> > an old-fashioned non-grouped target.
>
> I tried the attached patch on top of latest Linus tree (ac2263b588dffd),
> but unfortunately I got the same issue which is puzzling. Reverting
> 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5 does solve the issue though.

I'm at a bit of a loss here ... the only thing that seems to jump out
is that the genheaders tool is run twice without the grouped target
approach, but with both runs happening at the same point in the build
and the second run updating both header files, I'm a bit at a loss as
to why this would be problematic.

I don't want to block on fixing the kernel build while I keep chasing
some esoteric build behavior so I'm just going to revert the patch
with a note to revisit this when we require make >= 4.3.

Regardless, thanks for the report and the help testing, expect a
patch/revert shortly ...

-- 
paul-moore.com

Re: Possible build time regression affecting stable kernels

[Luiz Capitulino]

On 2023-06-01 11:45, Paul Moore wrote:

> 
> 
> 
> On Thu, Jun 1, 2023 at 11:03 AM Luiz Capitulino <luizcap@amazon.com> wrote:
>> On 2023-06-01 10:27, Paul Moore wrote:
>>> On Wed, May 31, 2023 at 10:13 PM Luiz Capitulino <luizcap@amazon.com> wrote:
>>>>
>>>> Hi Paul,
>>>>
>>>> A number of stable kernels recently backported this upstream commit:
>>>>
>>>> """
>>>> commit 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5
>>>> Author: Paul Moore <paul@paul-moore.com>
>>>> Date:   Wed Apr 12 13:29:11 2023 -0400
>>>>
>>>>        selinux: ensure av_permissions.h is built when needed
>>>> """
>>>>
>>>> We're seeing a build issue with this commit where the "crash" tool will fail
>>>> to start, it complains that the vmlinux image and /proc/version don't match.
>>>>
>>>> A minimum reproducer would be having "make" version before 4.3 and building
>>>> the kernel with:
>>>>
>>>> $ make bzImages
>>>> $ make modules
>>>
>>> ...
>>>
>>>> This only happens with commit 4ce1f694eb5 applied and older "make", in my case I
>>>> have "make" version 3.82.
>>>>
>>>> If I revert 4ce1f694eb5 or use "make" version 4.3 I get identical strings (except
>>>> for the "Linux version" part):
>>>
>>> Thanks Luiz, this is a fun one :/
>>
>> It was a fun to debug TBH :-)
>>
>>> Based on a quick search, it looks like the grouped target may be the
>>> cause, especially for older (pre-4.3) versions of make.  Looking
>>> through the rest of the kernel I don't see any other grouped targets,
>>> and in fact the top level Makefile even mentions holding off on using
>>> grouped targets until make v4.3 is common/required.
>>
>> Exactly.
>>
>>> I don't have an older userspace immediately available, would you mind
>>> trying the fix/patch below to see if it resolves the problem on your
>>> system?  It's a cut-n-paste so the patch may not apply directly, but
>>> it basically just removes the '&' from the make rule, turning it into
>>> an old-fashioned non-grouped target.
>>
>> I tried the attached patch on top of latest Linus tree (ac2263b588dffd),
>> but unfortunately I got the same issue which is puzzling. Reverting
>> 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5 does solve the issue though.
> 
> I'm at a bit of a loss here ... the only thing that seems to jump out
> is that the genheaders tool is run twice without the grouped target
> approach, but with both runs happening at the same point in the build
> and the second run updating both header files, I'm a bit at a loss as
> to why this would be problematic.
> 
> I don't want to block on fixing the kernel build while I keep chasing
> some esoteric build behavior so I'm just going to revert the patch
> with a note to revisit this when we require make >= 4.3.
> 
> Regardless, thanks for the report and the help testing, expect a
> patch/revert shortly ...

Thank you Paul, I really appreciate your fast response. I'd also
appreciate if you CC me in the revert patch so that I don't loose
track of it.

- Luiz

Re: Possible build time regression affecting stable kernels

[Greg KH]

On Thu, Jun 01, 2023 at 10:56:24AM -0400, Paul Moore wrote:
> On Thu, Jun 1, 2023 at 9:20 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> > On Thu, Jun 01, 2023 at 09:13:21AM -0400, Luiz Capitulino wrote:
> 
> ...
> 
> > > Yes. I'm reporting this here because I'm more concerned with -stable kernels since
> > > they're more likely to be running on older user-space.
> >
> > Yeah, we are bug-compatible!  :)
> 
> While I really don't want to go back into the old arguments about what
> does, and does not, get backported to -stable, I do want to ask if
> there is some way to signal to the -stable maintainers that a patch
> should not be backported?  Anything coming from the LSM, SELinux, or
> audit trees that I believe should be backported is explicitly marked
> with a stable@vger CC, as documented in stable-kernel-rules.rst,
> however it is generally my experience that patches with a 'Fixes:' tag
> are generally pulled into the -stable releases as well.

Really?  Right now we HAVE to pick up the Fixes: tagged commits in those
subsystems as you are missing lots of real fixes.  I just quick looked
and noticed 8cf0a1bc1287 ("capabilities: fix potential memleak on error
path from vfs_getxattr_alloc()") which you should have tagged, right?

In fact, I've considered most of the LSM code as a "we never tag
anything for stable so we rely on the Fixes: pickup to clean up after
us" subsystem.  We have many of those in the kernel, so you are in good
company :)

> I could start dropping the 'Fixes:' tag from non-stable tagged
> commits, but that's a step backwards in my opinion.

If a commit has fixes: why wouldn't it be ok for stable trees?  That
feels very odd to me.

Anyway, if you really want, yes, we can add you to the "list of
subsystems we do not pick anything for except by explicit cc: stable
marking" that we have, but note, that feels wrong to me based on the
very low number of patches being tagged for these directories over time.

> I could start replying to every -stable backport email notice, but
> that seems like a lot of unnecessary work for something that was never
> marked for -stable in the first place.  I'm guessing it would also add
> some additional management/testing burden to the -stable folks as
> well.

We have a list, so if you really want it, we can add you to it.  But can
you point out any Fixes: commits that we backported that we shouldn't
have?  If so, why was a Fixes: tag on it?

thanks,

greg k-h

Re: Possible build time regression affecting stable kernels

[Greg KH]

On Thu, Jun 01, 2023 at 11:50:16AM -0400, Luiz Capitulino wrote:
> 
> 
> On 2023-06-01 11:45, Paul Moore wrote:
> 
> > 
> > 
> > 
> > On Thu, Jun 1, 2023 at 11:03 AM Luiz Capitulino <luizcap@amazon.com> wrote:
> > > On 2023-06-01 10:27, Paul Moore wrote:
> > > > On Wed, May 31, 2023 at 10:13 PM Luiz Capitulino <luizcap@amazon.com> wrote:
> > > > > 
> > > > > Hi Paul,
> > > > > 
> > > > > A number of stable kernels recently backported this upstream commit:
> > > > > 
> > > > > """
> > > > > commit 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5
> > > > > Author: Paul Moore <paul@paul-moore.com>
> > > > > Date:   Wed Apr 12 13:29:11 2023 -0400
> > > > > 
> > > > >        selinux: ensure av_permissions.h is built when needed
> > > > > """
> > > > > 
> > > > > We're seeing a build issue with this commit where the "crash" tool will fail
> > > > > to start, it complains that the vmlinux image and /proc/version don't match.
> > > > > 
> > > > > A minimum reproducer would be having "make" version before 4.3 and building
> > > > > the kernel with:
> > > > > 
> > > > > $ make bzImages
> > > > > $ make modules
> > > > 
> > > > ...
> > > > 
> > > > > This only happens with commit 4ce1f694eb5 applied and older "make", in my case I
> > > > > have "make" version 3.82.
> > > > > 
> > > > > If I revert 4ce1f694eb5 or use "make" version 4.3 I get identical strings (except
> > > > > for the "Linux version" part):
> > > > 
> > > > Thanks Luiz, this is a fun one :/
> > > 
> > > It was a fun to debug TBH :-)
> > > 
> > > > Based on a quick search, it looks like the grouped target may be the
> > > > cause, especially for older (pre-4.3) versions of make.  Looking
> > > > through the rest of the kernel I don't see any other grouped targets,
> > > > and in fact the top level Makefile even mentions holding off on using
> > > > grouped targets until make v4.3 is common/required.
> > > 
> > > Exactly.
> > > 
> > > > I don't have an older userspace immediately available, would you mind
> > > > trying the fix/patch below to see if it resolves the problem on your
> > > > system?  It's a cut-n-paste so the patch may not apply directly, but
> > > > it basically just removes the '&' from the make rule, turning it into
> > > > an old-fashioned non-grouped target.
> > > 
> > > I tried the attached patch on top of latest Linus tree (ac2263b588dffd),
> > > but unfortunately I got the same issue which is puzzling. Reverting
> > > 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5 does solve the issue though.
> > 
> > I'm at a bit of a loss here ... the only thing that seems to jump out
> > is that the genheaders tool is run twice without the grouped target
> > approach, but with both runs happening at the same point in the build
> > and the second run updating both header files, I'm a bit at a loss as
> > to why this would be problematic.
> > 
> > I don't want to block on fixing the kernel build while I keep chasing
> > some esoteric build behavior so I'm just going to revert the patch
> > with a note to revisit this when we require make >= 4.3.
> > 
> > Regardless, thanks for the report and the help testing, expect a
> > patch/revert shortly ...
> 
> Thank you Paul, I really appreciate your fast response. I'd also
> appreciate if you CC me in the revert patch so that I don't loose
> track of it.

And please add a cc: stable@ to it too :)

Re: Possible build time regression affecting stable kernels

[Paul Moore]

On Thu, Jun 1, 2023 at 1:05 PM Greg KH <gregkh@linuxfoundation.org> wrote:
> On Thu, Jun 01, 2023 at 11:50:16AM -0400, Luiz Capitulino wrote:
> > On 2023-06-01 11:45, Paul Moore wrote:

...

> > > I don't want to block on fixing the kernel build while I keep chasing
> > > some esoteric build behavior so I'm just going to revert the patch
> > > with a note to revisit this when we require make >= 4.3.
> > >
> > > Regardless, thanks for the report and the help testing, expect a
> > > patch/revert shortly ...
> >
> > Thank you Paul, I really appreciate your fast response. I'd also
> > appreciate if you CC me in the revert patch so that I don't loose
> > track of it.
>
> And please add a cc: stable@ to it too :)

Done, and done.  I just sent the patch so give it a few minutes if you
haven't seen it yet.  I'll leave it for a few hours in case anyone can
give it a test build before I send it up to Linus.

-- 
paul-moore.com

Re: Possible build time regression affecting stable kernels

[Luiz Capitulino]

On 2023-06-01 14:10, Paul Moore wrote:

> 
> 
> 
> On Thu, Jun 1, 2023 at 1:05 PM Greg KH <gregkh@linuxfoundation.org> wrote:
>> On Thu, Jun 01, 2023 at 11:50:16AM -0400, Luiz Capitulino wrote:
>>> On 2023-06-01 11:45, Paul Moore wrote:
> 
> ...
> 
>>>> I don't want to block on fixing the kernel build while I keep chasing
>>>> some esoteric build behavior so I'm just going to revert the patch
>>>> with a note to revisit this when we require make >= 4.3.
>>>>
>>>> Regardless, thanks for the report and the help testing, expect a
>>>> patch/revert shortly ...
>>>
>>> Thank you Paul, I really appreciate your fast response. I'd also
>>> appreciate if you CC me in the revert patch so that I don't loose
>>> track of it.
>>
>> And please add a cc: stable@ to it too :)
> 
> Done, and done.  I just sent the patch so give it a few minutes if you
> haven't seen it yet.  I'll leave it for a few hours in case anyone can
> give it a test build before I send it up to Linus.

Thanks Paul, I can test it but it may take more than a few hours for me.

- Luiz

Re: Possible build time regression affecting stable kernels

[Paul Moore]

On Thu, Jun 1, 2023 at 11:51 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> On Thu, Jun 01, 2023 at 10:56:24AM -0400, Paul Moore wrote:
> > On Thu, Jun 1, 2023 at 9:20 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> > > On Thu, Jun 01, 2023 at 09:13:21AM -0400, Luiz Capitulino wrote:
> >
> > ...
> >
> > > > Yes. I'm reporting this here because I'm more concerned with -stable kernels since
> > > > they're more likely to be running on older user-space.
> > >
> > > Yeah, we are bug-compatible!  :)
> >
> > While I really don't want to go back into the old arguments about what
> > does, and does not, get backported to -stable, I do want to ask if
> > there is some way to signal to the -stable maintainers that a patch
> > should not be backported?  Anything coming from the LSM, SELinux, or
> > audit trees that I believe should be backported is explicitly marked
> > with a stable@vger CC, as documented in stable-kernel-rules.rst,
> > however it is generally my experience that patches with a 'Fixes:' tag
> > are generally pulled into the -stable releases as well.
>
> Really?

Yes, really.

> Right now we HAVE to pick up the Fixes: tagged commits in those
> subsystems as you are missing lots of real fixes.

This starts to bring us back to the old argument about what is
appropriate for -stable, but I've been sticking as close as possible
to what is documented in stable-kernel-rules.rst which (ignoring
things like HW enablement) advises that only patches which fix build
issues or "serious issues" should be considered for -stable.  I
consider every bug fix that goes into the LSM, SELinux, and audit
trees to see if it meets those criteria, if it does I mark it with a
-stable tag, if not I leave the -stable tag and ensure it carries a
'Fixes:' tag if it makes sense and an appropriate root-cause commit is
identified.

We definitely have different opinions on where the -stable bug fix
threshold lies.  I am of the opinion that every -stable backport
carries risk, and I consider that when deciding if a commit should be
marked for -stable.  I do not believe that every bug fix, or every
commit with a 'Fixes:' tag, should be backported to -stable.

> I just quick looked
> and noticed 8cf0a1bc1287 ("capabilities: fix potential memleak on error
> path from vfs_getxattr_alloc()") which you should have tagged, right?

Nope.  That's a memleak for a weird corner case that isn't really
going to be triggered in practice; it was found only by code
inspection and requires code modification to exercise.  See the
discussion around the original revision for more information:

https://lore.kernel.org/linux-security-module/20221025104544.2298829-1-cuigaosheng1@huawei.com/

Can you explain why you HAD to pick up that commit?

> In fact, I've considered most of the LSM code as a "we never tag
> anything for stable so we rely on the Fixes: pickup to clean up after
> us" subsystem.  We have many of those in the kernel, so you are in good
> company :)

That would be amusing if it were correct, but your use of absolutes
betrays you.  Looking at code under security/ from v5.10 to current on
I see 71 -stable tags as of today.  It looks like the last one I
marked was from October of 2022.  We could deep dive more on this, but
I think that's a waste of everyone's time.

> > I could start dropping the 'Fixes:' tag from non-stable tagged
> > commits, but that's a step backwards in my opinion.
>
> If a commit has fixes: why wouldn't it be ok for stable trees?  That
> feels very odd to me.

Backports have risk, and in my opinion not every bug fix rises to the
level of severity to sufficiently counter that risk.  The -stable
kernel documentation seems to support that; if you believe every bug
fix should be backported to the -stable kernels I would suggest you
update the documentation.

> Anyway, if you really want, yes, we can add you to the "list of
> subsystems we do not pick anything for except by explicit cc: stable
> marking" that we have, but note, that feels wrong to me based on the
> very low number of patches being tagged for these directories over time.

That's great, I didn't know you maintained such a list.  Please add
the LSM, SELinux, and audit trees to the subsystems which are only
backported when there is an explicit -stable tag.  It is worth noting
that this should not affect the -stable backport policy for other
LSMs, e.g. AppArmor, although I can mention this list to the other LSM
maintainers as they may want to be added.

-- 
paul-moore.com

Re: Possible build time regression affecting stable kernels

[Paul Moore]

On Thu, Jun 1, 2023 at 2:15 PM Luiz Capitulino <luizcap@amazon.com> wrote:
> On 2023-06-01 14:10, Paul Moore wrote:
> > On Thu, Jun 1, 2023 at 1:05 PM Greg KH <gregkh@linuxfoundation.org> wrote:
> >> On Thu, Jun 01, 2023 at 11:50:16AM -0400, Luiz Capitulino wrote:
> >>> On 2023-06-01 11:45, Paul Moore wrote:
> >
> > ...
> >
> >>>> I don't want to block on fixing the kernel build while I keep chasing
> >>>> some esoteric build behavior so I'm just going to revert the patch
> >>>> with a note to revisit this when we require make >= 4.3.
> >>>>
> >>>> Regardless, thanks for the report and the help testing, expect a
> >>>> patch/revert shortly ...
> >>>
> >>> Thank you Paul, I really appreciate your fast response. I'd also
> >>> appreciate if you CC me in the revert patch so that I don't loose
> >>> track of it.
> >>
> >> And please add a cc: stable@ to it too :)
> >
> > Done, and done.  I just sent the patch so give it a few minutes if you
> > haven't seen it yet.  I'll leave it for a few hours in case anyone can
> > give it a test build before I send it up to Linus.
>
> Thanks Paul, I can test it but it may take more than a few hours for me.

Okay, no worries, I'll hold off until I hear back from you.  I'd
rather it take a little longer in order to get the warm fuzzies you
get from a 'Tested-by:' by one of the 'Reported-by:' people :)

Thanks for all your help with this Luiz.

-- 
paul-moore.com

Re: Possible build time regression affecting stable kernels

[Greg KH]

On Thu, Jun 01, 2023 at 02:39:00PM -0400, Paul Moore wrote:
> On Thu, Jun 1, 2023 at 11:51 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> > On Thu, Jun 01, 2023 at 10:56:24AM -0400, Paul Moore wrote:
> > > On Thu, Jun 1, 2023 at 9:20 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> > > > On Thu, Jun 01, 2023 at 09:13:21AM -0400, Luiz Capitulino wrote:
> > >
> > > ...
> > >
> > > > > Yes. I'm reporting this here because I'm more concerned with -stable kernels since
> > > > > they're more likely to be running on older user-space.
> > > >
> > > > Yeah, we are bug-compatible!  :)
> > >
> > > While I really don't want to go back into the old arguments about what
> > > does, and does not, get backported to -stable, I do want to ask if
> > > there is some way to signal to the -stable maintainers that a patch
> > > should not be backported?  Anything coming from the LSM, SELinux, or
> > > audit trees that I believe should be backported is explicitly marked
> > > with a stable@vger CC, as documented in stable-kernel-rules.rst,
> > > however it is generally my experience that patches with a 'Fixes:' tag
> > > are generally pulled into the -stable releases as well.
> >
> > Really?
> 
> Yes, really.
> 
> > Right now we HAVE to pick up the Fixes: tagged commits in those
> > subsystems as you are missing lots of real fixes.
> 
> This starts to bring us back to the old argument about what is
> appropriate for -stable, but I've been sticking as close as possible
> to what is documented in stable-kernel-rules.rst which (ignoring
> things like HW enablement) advises that only patches which fix build
> issues or "serious issues" should be considered for -stable.  I
> consider every bug fix that goes into the LSM, SELinux, and audit
> trees to see if it meets those criteria, if it does I mark it with a
> -stable tag, if not I leave the -stable tag and ensure it carries a
> 'Fixes:' tag if it makes sense and an appropriate root-cause commit is
> identified.
> 
> We definitely have different opinions on where the -stable bug fix
> threshold lies.  I am of the opinion that every -stable backport
> carries risk, and I consider that when deciding if a commit should be
> marked for -stable.  I do not believe that every bug fix, or every
> commit with a 'Fixes:' tag, should be backported to -stable.

Ok, I'll not argue here, but it feels like there is a lack of changes
for some of these portions of the kernel that end up in stable kernels.
I'll trust you on this.

So, can I get a directory list or file list of what we should be
ignoring for the AUTOSEL and "Fixes: only" tools to be ignoring?

thanks,

greg k-h

Re: Possible build time regression affecting stable kernels

[Paul Moore]

On Wed, Jun 28, 2023 at 2:33 PM Greg KH <gregkh@linuxfoundation.org> wrote:
> On Thu, Jun 01, 2023 at 02:39:00PM -0400, Paul Moore wrote:

...

> > We definitely have different opinions on where the -stable bug fix
> > threshold lies.  I am of the opinion that every -stable backport
> > carries risk, and I consider that when deciding if a commit should be
> > marked for -stable.  I do not believe that every bug fix, or every
> > commit with a 'Fixes:' tag, should be backported to -stable.
>
> Ok, I'll not argue here, but it feels like there is a lack of changes
> for some of these portions of the kernel that end up in stable kernels.
> I'll trust you on this.

I don't know what to say here ... aside from the previously discussed
difference of opinion regarding stable tags, we just haven't had many
bad bugs in the LSM layer, SELinux, or audit the past few years.  I'd
like to claim better code, better review, yadda yadda but in reality
I'm sure it's just dumb luck.

> So, can I get a directory list or file list of what we should be
> ignoring for the AUTOSEL and "Fixes: only" tools to be ignoring?

I've been trying to ensure that the files/directories entries in
MAINTAINERS are current, so that is probably as good a place as any to
pull that info.  Do the stable tools use that info already?  In other
words, if we update the entries in MAINTAINERS should we also notify
you guys, or will you get it automatically?

Regardless, here is a list:

* Audit
include/asm-generic/audit_*.h
include/linux/audit.h
include/linux/audit_arch.h
include/uapi/linux/audit.h
kernel/audit*
lib/*audit.c

* LSM layer
security/
(NOTE: the individual sub-dirs under security/ belong to the
individual LSMs, not the LSM layer)

* SELinux
include/trace/events/avc.h
include/uapi/linux/selinux_netlink.h
scripts/selinux/
security/selinux/

-- 
paul-moore.com

Re: Possible build time regression affecting stable kernels

[Greg KH]

On Wed, Jun 28, 2023 at 07:33:27PM -0400, Paul Moore wrote:
> > So, can I get a directory list or file list of what we should be
> > ignoring for the AUTOSEL and "Fixes: only" tools to be ignoring?
> 
> I've been trying to ensure that the files/directories entries in
> MAINTAINERS are current, so that is probably as good a place as any to
> pull that info.  Do the stable tools use that info already?  In other
> words, if we update the entries in MAINTAINERS should we also notify
> you guys, or will you get it automatically?

We do not use (or at least I don't, I can't speak for Sasha here, but
odds are we should unify this now), the MAINTAINERS file for this, but
rather a list like you provided below, thanks.

> Regardless, here is a list:
> 
> * Audit
> include/asm-generic/audit_*.h
> include/linux/audit.h
> include/linux/audit_arch.h
> include/uapi/linux/audit.h
> kernel/audit*
> lib/*audit.c
> 
> * LSM layer
> security/
> (NOTE: the individual sub-dirs under security/ belong to the
> individual LSMs, not the LSM layer)

So security/*.c would cover this, not below that, right?

> * SELinux
> include/trace/events/avc.h
> include/uapi/linux/selinux_netlink.h
> scripts/selinux/
> security/selinux/

Looks good, thanks for this.

greg k-h

Re: Possible build time regression affecting stable kernels

[Paul Moore]

On Thu, Jun 29, 2023 at 4:43 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> On Wed, Jun 28, 2023 at 07:33:27PM -0400, Paul Moore wrote:
> > > So, can I get a directory list or file list of what we should be
> > > ignoring for the AUTOSEL and "Fixes: only" tools to be ignoring?
> >
> > I've been trying to ensure that the files/directories entries in
> > MAINTAINERS are current, so that is probably as good a place as any to
> > pull that info.  Do the stable tools use that info already?  In other
> > words, if we update the entries in MAINTAINERS should we also notify
> > you guys, or will you get it automatically?
>
> We do not use (or at least I don't, I can't speak for Sasha here, but
> odds are we should unify this now), the MAINTAINERS file for this, but
> rather a list like you provided below, thanks.

Fair enough, if we ever have any significant restructuring I'll try to
remember to update the stable folks.  Although I'm guessing such a
change would likely end up being self-reporting anyway.

> > Regardless, here is a list:
> >
> > * Audit
> > include/asm-generic/audit_*.h
> > include/linux/audit.h
> > include/linux/audit_arch.h
> > include/uapi/linux/audit.h
> > kernel/audit*
> > lib/*audit.c
> >
> > * LSM layer
> > security/
> > (NOTE: the individual sub-dirs under security/ belong to the
> > individual LSMs, not the LSM layer)
>
> So security/*.c would cover this, not below that, right?

Yes, that should work.

> > * SELinux
> > include/trace/events/avc.h
> > include/uapi/linux/selinux_netlink.h
> > scripts/selinux/
> > security/selinux/
>
> Looks good, thanks for this.

Thanks for maintaining the exception list.

-- 
paul-moore.com

Re: Possible build time regression affecting stable kernels

[Greg KH]

On Thu, Jun 29, 2023 at 11:55:12AM -0400, Paul Moore wrote:
> On Thu, Jun 29, 2023 at 4:43 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> > On Wed, Jun 28, 2023 at 07:33:27PM -0400, Paul Moore wrote:
> > > > So, can I get a directory list or file list of what we should be
> > > > ignoring for the AUTOSEL and "Fixes: only" tools to be ignoring?
> > >
> > > I've been trying to ensure that the files/directories entries in
> > > MAINTAINERS are current, so that is probably as good a place as any to
> > > pull that info.  Do the stable tools use that info already?  In other
> > > words, if we update the entries in MAINTAINERS should we also notify
> > > you guys, or will you get it automatically?
> >
> > We do not use (or at least I don't, I can't speak for Sasha here, but
> > odds are we should unify this now), the MAINTAINERS file for this, but
> > rather a list like you provided below, thanks.
> 
> Fair enough, if we ever have any significant restructuring I'll try to
> remember to update the stable folks.  Although I'm guessing such a
> change would likely end up being self-reporting anyway.
> 
> > > Regardless, here is a list:
> > >
> > > * Audit
> > > include/asm-generic/audit_*.h
> > > include/linux/audit.h
> > > include/linux/audit_arch.h
> > > include/uapi/linux/audit.h
> > > kernel/audit*
> > > lib/*audit.c
> > >
> > > * LSM layer
> > > security/
> > > (NOTE: the individual sub-dirs under security/ belong to the
> > > individual LSMs, not the LSM layer)
> >
> > So security/*.c would cover this, not below that, right?
> 
> Yes, that should work.
> 
> > > * SELinux
> > > include/trace/events/avc.h
> > > include/uapi/linux/selinux_netlink.h
> > > scripts/selinux/
> > > security/selinux/
> >
> > Looks good, thanks for this.
> 
> Thanks for maintaining the exception list.

Cool, it's maintained here:
	https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git/tree/ignore_list

if it's ever needed to be updated in the future.

thanks,

greg k-h

Re: Possible build time regression affecting stable kernels

[Paul Moore]

On Thu, Jun 29, 2023 at 12:07 PM Greg KH <gregkh@linuxfoundation.org> wrote:
> On Thu, Jun 29, 2023 at 11:55:12AM -0400, Paul Moore wrote:
> > On Thu, Jun 29, 2023 at 4:43 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> > > On Wed, Jun 28, 2023 at 07:33:27PM -0400, Paul Moore wrote:
> > > > > So, can I get a directory list or file list of what we should be
> > > > > ignoring for the AUTOSEL and "Fixes: only" tools to be ignoring?
> > > >
> > > > I've been trying to ensure that the files/directories entries in
> > > > MAINTAINERS are current, so that is probably as good a place as any to
> > > > pull that info.  Do the stable tools use that info already?  In other
> > > > words, if we update the entries in MAINTAINERS should we also notify
> > > > you guys, or will you get it automatically?
> > >
> > > We do not use (or at least I don't, I can't speak for Sasha here, but
> > > odds are we should unify this now), the MAINTAINERS file for this, but
> > > rather a list like you provided below, thanks.
> >
> > Fair enough, if we ever have any significant restructuring I'll try to
> > remember to update the stable folks.  Although I'm guessing such a
> > change would likely end up being self-reporting anyway.
> >
> > > > Regardless, here is a list:
> > > >
> > > > * Audit
> > > > include/asm-generic/audit_*.h
> > > > include/linux/audit.h
> > > > include/linux/audit_arch.h
> > > > include/uapi/linux/audit.h
> > > > kernel/audit*
> > > > lib/*audit.c
> > > >
> > > > * LSM layer
> > > > security/
> > > > (NOTE: the individual sub-dirs under security/ belong to the
> > > > individual LSMs, not the LSM layer)
> > >
> > > So security/*.c would cover this, not below that, right?
> >
> > Yes, that should work.
> >
> > > > * SELinux
> > > > include/trace/events/avc.h
> > > > include/uapi/linux/selinux_netlink.h
> > > > scripts/selinux/
> > > > security/selinux/
> > >
> > > Looks good, thanks for this.
> >
> > Thanks for maintaining the exception list.
>
> Cool, it's maintained here:
>         https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git/tree/ignore_list
>
> if it's ever needed to be updated in the future.

Noted, thanks.

-- 
paul-moore.com