Re: [PATCH 5/6] selftests: Preserve subtarget failures in all/install

Re: [PATCH 5/6] selftests: Preserve subtarget failures in all/install

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 2321 bytes --]

On Fri, Mar 20, 2026 at 03:29:20PM -0300, Ricardo B. Marlière wrote:
> Track failures explicitly in the top-level selftests all/install loops.
> 
> The current code multiplies `ret` by each sub-make exit status. For
> example, with `TARGETS=net`, the implicit `net/lib` dependency runs after
> `net`, so a failed `net` build can be followed by a successful `net/lib`
> build and reset the final result to success.
> 
> Set `ret` to 1 on any non-zero sub-make exit code and keep it sticky, so
> the top-level make returns failure when any selected selftest target
> fails.

This patch, which is now in mainline as 7e47389142b8, is breaking a
bunch of CI systems - at least KernelCI, our Arm internal CI and my
personal stuff.  It causes the equivalent of FORCE_TARGETS behaviour in
the top level Makefile, the prior behaviour where the exit status of the
top level Makefile ignores failures from individual directories is
desirable since by default we try to build almost all the selftests but
between quality issues and build time dependencies it's very common for
at least one of them to fail.  With this commit unless the user has
configured a more restricted set of selftests it would be surprising if
we manage to get a successful build and install.

As well as being a poor default due to the very high likelyhood of build
failures this also has the undesirable effect of causing a build failure
in one selftest to cause the whole install target to fail, meaning that
the build failure is escallated to a complete lost of coverge for all
selftests in common CI usage.

This wasn't showing up in my -next build tests since I set FORCE_TARGETS
and explicitly choose a restricted set of kselftests which actually
build with my system and configuration.  It was less obvious than it
should have been with the other systems since they did not expect there
to be a complete failure to generate a kselftest tarball and variously
masked the error or reported it in a manner that looked like an
infrastructure issue.

It would be really nice to get to the point where we can reasonably do
this but we're simply not there at the current time.  At the moment if
people want to see build failures reported at the top level that really
needs to be opt in, we have FORCE_TARGETS for that.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH 5/6] selftests: Preserve subtarget failures in all/install

[Shuah Khan]

On 4/15/26 07:58, Mark Brown wrote:
> On Fri, Mar 20, 2026 at 03:29:20PM -0300, Ricardo B. Marlière wrote:
>> Track failures explicitly in the top-level selftests all/install loops.
>>
>> The current code multiplies `ret` by each sub-make exit status. For
>> example, with `TARGETS=net`, the implicit `net/lib` dependency runs after
>> `net`, so a failed `net` build can be followed by a successful `net/lib`
>> build and reset the final result to success.
>>
>> Set `ret` to 1 on any non-zero sub-make exit code and keep it sticky, so
>> the top-level make returns failure when any selected selftest target
>> fails.
> 
> This patch, which is now in mainline as 7e47389142b8, is breaking a
> bunch of CI systems - at least KernelCI, our Arm internal CI and my
> personal stuff.  It causes the equivalent of FORCE_TARGETS behaviour in
> the top level Makefile, the prior behaviour where the exit status of the
> top level Makefile ignores failures from individual directories is
> desirable since by default we try to build almost all the selftests but
> between quality issues and build time dependencies it's very common for
> at least one of them to fail.  With this commit unless the user has
> configured a more restricted set of selftests it would be surprising if
> we manage to get a successful build and install.
> 
> As well as being a poor default due to the very high likelyhood of build
> failures this also has the undesirable effect of causing a build failure
> in one selftest to cause the whole install target to fail, meaning that
> the build failure is escallated to a complete lost of coverge for all
> selftests in common CI usage.
> 
> This wasn't showing up in my -next build tests since I set FORCE_TARGETS
> and explicitly choose a restricted set of kselftests which actually
> build with my system and configuration.  It was less obvious than it
> should have been with the other systems since they did not expect there
> to be a complete failure to generate a kselftest tarball and variously
> masked the error or reported it in a manner that looked like an
> infrastructure issue.

I didn't see it when I did test on linux-next and my repo. I did install
to catch problems.

Sorry for not catching this. We can drop this patch.

> 
> It would be really nice to get to the point where we can reasonably do
> this but we're simply not there at the current time.  At the moment if
> people want to see build failures reported at the top level that really
> needs to be opt in, we have FORCE_TARGETS for that.

Good point - I will go look and see if we document this in kselftest doc
and add it.

Mark, would you like to a revert for this?

thanks,
-- Shuah

Re: [PATCH 5/6] selftests: Preserve subtarget failures in all/install

[Ricardo B. Marlière]

On Wed Apr 15, 2026 at 12:40 PM -03, Shuah Khan wrote:
> On 4/15/26 07:58, Mark Brown wrote:
>> On Fri, Mar 20, 2026 at 03:29:20PM -0300, Ricardo B. Marlière wrote:
>>> Track failures explicitly in the top-level selftests all/install loops.
>>>
>>> The current code multiplies `ret` by each sub-make exit status. For
>>> example, with `TARGETS=net`, the implicit `net/lib` dependency runs after
>>> `net`, so a failed `net` build can be followed by a successful `net/lib`
>>> build and reset the final result to success.
>>>
>>> Set `ret` to 1 on any non-zero sub-make exit code and keep it sticky, so
>>> the top-level make returns failure when any selected selftest target
>>> fails.
>> 
>> This patch, which is now in mainline as 7e47389142b8, is breaking a
>> bunch of CI systems - at least KernelCI, our Arm internal CI and my
>> personal stuff.  It causes the equivalent of FORCE_TARGETS behaviour in
>> the top level Makefile, the prior behaviour where the exit status of the
>> top level Makefile ignores failures from individual directories is
>> desirable since by default we try to build almost all the selftests but
>> between quality issues and build time dependencies it's very common for
>> at least one of them to fail.  With this commit unless the user has
>> configured a more restricted set of selftests it would be surprising if
>> we manage to get a successful build and install.
>> 
>> As well as being a poor default due to the very high likelyhood of build
>> failures this also has the undesirable effect of causing a build failure
>> in one selftest to cause the whole install target to fail, meaning that
>> the build failure is escallated to a complete lost of coverge for all
>> selftests in common CI usage.
>> 
>> This wasn't showing up in my -next build tests since I set FORCE_TARGETS
>> and explicitly choose a restricted set of kselftests which actually
>> build with my system and configuration.  It was less obvious than it
>> should have been with the other systems since they did not expect there
>> to be a complete failure to generate a kselftest tarball and variously
>> masked the error or reported it in a manner that looked like an
>> infrastructure issue.
>
> I didn't see it when I did test on linux-next and my repo. I did install
> to catch problems.
>
> Sorry for not catching this. We can drop this patch.
>
>> 
>> It would be really nice to get to the point where we can reasonably do
>> this but we're simply not there at the current time.  At the moment if
>> people want to see build failures reported at the top level that really
>> needs to be opt in, we have FORCE_TARGETS for that.
>
> Good point - I will go look and see if we document this in kselftest doc
> and add it.

It's not documented. It would have solved my issue, sorry for
overlooking this!

>
> Mark, would you like to a revert for this?
>
> thanks,
> -- Shuah

Re: [PATCH 5/6] selftests: Preserve subtarget failures in all/install

[Shuah Khan]

On 4/15/26 09:42, Ricardo B. Marlière wrote:
> On Wed Apr 15, 2026 at 12:40 PM -03, Shuah Khan wrote:
>> On 4/15/26 07:58, Mark Brown wrote:
>>> On Fri, Mar 20, 2026 at 03:29:20PM -0300, Ricardo B. Marlière wrote:
>>>> Track failures explicitly in the top-level selftests all/install loops.
>>>>
>>>> The current code multiplies `ret` by each sub-make exit status. For
>>>> example, with `TARGETS=net`, the implicit `net/lib` dependency runs after
>>>> `net`, so a failed `net` build can be followed by a successful `net/lib`
>>>> build and reset the final result to success.
>>>>
>>>> Set `ret` to 1 on any non-zero sub-make exit code and keep it sticky, so
>>>> the top-level make returns failure when any selected selftest target
>>>> fails.
>>>
>>> This patch, which is now in mainline as 7e47389142b8, is breaking a
>>> bunch of CI systems - at least KernelCI, our Arm internal CI and my
>>> personal stuff.  It causes the equivalent of FORCE_TARGETS behaviour in
>>> the top level Makefile, the prior behaviour where the exit status of the
>>> top level Makefile ignores failures from individual directories is
>>> desirable since by default we try to build almost all the selftests but
>>> between quality issues and build time dependencies it's very common for
>>> at least one of them to fail.  With this commit unless the user has
>>> configured a more restricted set of selftests it would be surprising if
>>> we manage to get a successful build and install.
>>>
>>> As well as being a poor default due to the very high likelyhood of build
>>> failures this also has the undesirable effect of causing a build failure
>>> in one selftest to cause the whole install target to fail, meaning that
>>> the build failure is escallated to a complete lost of coverge for all
>>> selftests in common CI usage.
>>>
>>> This wasn't showing up in my -next build tests since I set FORCE_TARGETS
>>> and explicitly choose a restricted set of kselftests which actually
>>> build with my system and configuration.  It was less obvious than it
>>> should have been with the other systems since they did not expect there
>>> to be a complete failure to generate a kselftest tarball and variously
>>> masked the error or reported it in a manner that looked like an
>>> infrastructure issue.
>>
>> I didn't see it when I did test on linux-next and my repo. I did install
>> to catch problems.
>>
>> Sorry for not catching this. We can drop this patch.
>>
>>>
>>> It would be really nice to get to the point where we can reasonably do
>>> this but we're simply not there at the current time.  At the moment if
>>> people want to see build failures reported at the top level that really
>>> needs to be opt in, we have FORCE_TARGETS for that.
>>
>> Good point - I will go look and see if we document this in kselftest doc
>> and add it.
> 
> It's not documented. It would have solved my issue, sorry for
> overlooking this!

Ricardo, Would you like to send me doc patch for this?

thanks,
-- Shuah

Re: [PATCH 5/6] selftests: Preserve subtarget failures in all/install

[Ricardo B. Marlière]

On Wed Apr 15, 2026 at 12:52 PM -03, Shuah Khan wrote:
> On 4/15/26 09:42, Ricardo B. Marlière wrote:
>> On Wed Apr 15, 2026 at 12:40 PM -03, Shuah Khan wrote:
>>> On 4/15/26 07:58, Mark Brown wrote:
>>>> On Fri, Mar 20, 2026 at 03:29:20PM -0300, Ricardo B. Marlière wrote:
>>>>> Track failures explicitly in the top-level selftests all/install loops.
>>>>>
>>>>> The current code multiplies `ret` by each sub-make exit status. For
>>>>> example, with `TARGETS=net`, the implicit `net/lib` dependency runs after
>>>>> `net`, so a failed `net` build can be followed by a successful `net/lib`
>>>>> build and reset the final result to success.
>>>>>
>>>>> Set `ret` to 1 on any non-zero sub-make exit code and keep it sticky, so
>>>>> the top-level make returns failure when any selected selftest target
>>>>> fails.
>>>>
>>>> This patch, which is now in mainline as 7e47389142b8, is breaking a
>>>> bunch of CI systems - at least KernelCI, our Arm internal CI and my
>>>> personal stuff.  It causes the equivalent of FORCE_TARGETS behaviour in
>>>> the top level Makefile, the prior behaviour where the exit status of the
>>>> top level Makefile ignores failures from individual directories is
>>>> desirable since by default we try to build almost all the selftests but
>>>> between quality issues and build time dependencies it's very common for
>>>> at least one of them to fail.  With this commit unless the user has
>>>> configured a more restricted set of selftests it would be surprising if
>>>> we manage to get a successful build and install.
>>>>
>>>> As well as being a poor default due to the very high likelyhood of build
>>>> failures this also has the undesirable effect of causing a build failure
>>>> in one selftest to cause the whole install target to fail, meaning that
>>>> the build failure is escallated to a complete lost of coverge for all
>>>> selftests in common CI usage.
>>>>
>>>> This wasn't showing up in my -next build tests since I set FORCE_TARGETS
>>>> and explicitly choose a restricted set of kselftests which actually
>>>> build with my system and configuration.  It was less obvious than it
>>>> should have been with the other systems since they did not expect there
>>>> to be a complete failure to generate a kselftest tarball and variously
>>>> masked the error or reported it in a manner that looked like an
>>>> infrastructure issue.
>>>
>>> I didn't see it when I did test on linux-next and my repo. I did install
>>> to catch problems.
>>>
>>> Sorry for not catching this. We can drop this patch.
>>>
>>>>
>>>> It would be really nice to get to the point where we can reasonably do
>>>> this but we're simply not there at the current time.  At the moment if
>>>> people want to see build failures reported at the top level that really
>>>> needs to be opt in, we have FORCE_TARGETS for that.
>>>
>>> Good point - I will go look and see if we document this in kselftest doc
>>> and add it.
>> 
>> It's not documented. It would have solved my issue, sorry for
>> overlooking this!
>
> Ricardo, Would you like to send me doc patch for this?

Sure, thanks!

>
> thanks,
> -- Shuah

Re: [PATCH 5/6] selftests: Preserve subtarget failures in all/install

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

On Wed, Apr 15, 2026 at 09:40:37AM -0600, Shuah Khan wrote:
> On 4/15/26 07:58, Mark Brown wrote:

> > This wasn't showing up in my -next build tests since I set FORCE_TARGETS
> > and explicitly choose a restricted set of kselftests which actually
> > build with my system and configuration.  It was less obvious than it
> > should have been with the other systems since they did not expect there
> > to be a complete failure to generate a kselftest tarball and variously
> > masked the error or reported it in a manner that looked like an
> > infrastructure issue.

> I didn't see it when I did test on linux-next and my repo. I did install
> to catch problems.

> Sorry for not catching this. We can drop this patch.

Yeah, it's easy to miss if you're set up to build everything.

> Mark, would you like to a revert for this?

Yes, I'll massage the text from my report into a changelog send
something - it'll probably be tomorrow now.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH 5/6] selftests: Preserve subtarget failures in all/install

[Shuah Khan]

On 4/15/26 10:25, Mark Brown wrote:
> On Wed, Apr 15, 2026 at 09:40:37AM -0600, Shuah Khan wrote:
>> On 4/15/26 07:58, Mark Brown wrote:
> 
>>> This wasn't showing up in my -next build tests since I set FORCE_TARGETS
>>> and explicitly choose a restricted set of kselftests which actually
>>> build with my system and configuration.  It was less obvious than it
>>> should have been with the other systems since they did not expect there
>>> to be a complete failure to generate a kselftest tarball and variously
>>> masked the error or reported it in a manner that looked like an
>>> infrastructure issue.
> 
>> I didn't see it when I did test on linux-next and my repo. I did install
>> to catch problems.
> 
>> Sorry for not catching this. We can drop this patch.
> 
> Yeah, it's easy to miss if you're set up to build everything.
> 
>> Mark, would you like to a revert for this?
> 
> Yes, I'll massage the text from my report into a changelog send
> something - it'll probably be tomorrow now.

Thank you Mark

-- Shuah

Re: [PATCH 5/6] selftests: Preserve subtarget failures in all/install

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 479 bytes --]

On Wed, Apr 15, 2026 at 05:25:40PM +0100, Mark Brown wrote:
> On Wed, Apr 15, 2026 at 09:40:37AM -0600, Shuah Khan wrote:

> > Mark, would you like to a revert for this?

> Yes, I'll massage the text from my report into a changelog send
> something - it'll probably be tomorrow now.

Sending shortly, FWIW I'm also seeing a buch of issues with:

./run_kselftest.sh: 5: ./kselftest/runner.sh: Bad substitution
./run_kselftest.sh: 5: .: cannot open ./ktap_helpers.sh: No such file

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH 5/6] selftests: Preserve subtarget failures in all/install

[Shuah Khan]

On 4/16/26 07:16, Mark Brown wrote:
> On Wed, Apr 15, 2026 at 05:25:40PM +0100, Mark Brown wrote:
>> On Wed, Apr 15, 2026 at 09:40:37AM -0600, Shuah Khan wrote:
> 
>>> Mark, would you like to a revert for this?
> 
>> Yes, I'll massage the text from my report into a changelog send
>> something - it'll probably be tomorrow now.
> 
> Sending shortly, FWIW I'm also seeing a buch of issues with:
> 
> ./run_kselftest.sh: 5: ./kselftest/runner.sh: Bad substitution
> ./run_kselftest.sh: 5: .: cannot open ./ktap_helpers.sh: No such file

I will take a look and isolate the two commits that changes
run_kselftest.sh - sorry for the trouble with CI this time
around.

thanks,
-- Shuah

Re: [PATCH 5/6] selftests: Preserve subtarget failures in all/install

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 685 bytes --]

On Thu, Apr 16, 2026 at 09:08:17AM -0600, Shuah Khan wrote:
> On 4/16/26 07:16, Mark Brown wrote:

> > Sending shortly, FWIW I'm also seeing a buch of issues with:

> > ./run_kselftest.sh: 5: ./kselftest/runner.sh: Bad substitution
> > ./run_kselftest.sh: 5: .: cannot open ./ktap_helpers.sh: No such file

> I will take a look and isolate the two commits that changes
> run_kselftest.sh - sorry for the trouble with CI this time
> around.

Pretty sure it's 2964f6b816c2 (selftests: Use ktap helpers for
runner.sh) - I've just put together a fix for that one, will throw it at
my testing and send it later today all being well.  It uses readlink -e
which isn't implemented by busybox.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH 5/6] selftests: Preserve subtarget failures in all/install

[Shuah Khan]

On 4/16/26 09:15, Mark Brown wrote:
> On Thu, Apr 16, 2026 at 09:08:17AM -0600, Shuah Khan wrote:
>> On 4/16/26 07:16, Mark Brown wrote:
> 
>>> Sending shortly, FWIW I'm also seeing a buch of issues with:
> 
>>> ./run_kselftest.sh: 5: ./kselftest/runner.sh: Bad substitution
>>> ./run_kselftest.sh: 5: .: cannot open ./ktap_helpers.sh: No such file
> 
>> I will take a look and isolate the two commits that changes
>> run_kselftest.sh - sorry for the trouble with CI this time
>> around.
> 
> Pretty sure it's 2964f6b816c2 (selftests: Use ktap helpers for
> runner.sh) - I've just put together a fix for that one, will throw it at
> my testing and send it later today all being well.  It uses readlink -e
> which isn't implemented by busybox.

Yes 2964f6b816c2 is my first suspect with ktap_helpers.sh no such file.
Thank you for fixing this. That way we can patch it and not revert it.

thanks,
-- SHuah

Re: [PATCH 5/6] selftests: Preserve subtarget failures in all/install

[Shuah Khan]

On 4/16/26 09:21, Shuah Khan wrote:
> On 4/16/26 09:15, Mark Brown wrote:
>> On Thu, Apr 16, 2026 at 09:08:17AM -0600, Shuah Khan wrote:
>>> On 4/16/26 07:16, Mark Brown wrote:
>>
>>>> Sending shortly, FWIW I'm also seeing a buch of issues with:
>>
>>>> ./run_kselftest.sh: 5: ./kselftest/runner.sh: Bad substitution
>>>> ./run_kselftest.sh: 5: .: cannot open ./ktap_helpers.sh: No such file
>>
>>> I will take a look and isolate the two commits that changes
>>> run_kselftest.sh - sorry for the trouble with CI this time
>>> around.
>>
>> Pretty sure it's 2964f6b816c2 (selftests: Use ktap helpers for
>> runner.sh) - I've just put together a fix for that one, will throw it at
>> my testing and send it later today all being well.  It uses readlink -e
>> which isn't implemented by busybox.
> 
> Yes 2964f6b816c2 is my first suspect with ktap_helpers.sh no such file.
> Thank you for fixing this. That way we can patch it and not revert it.
> 

Just a quick note to say I applied all 3 patches and will send pr tomorrow.

thanks,
-- Shuah

Re: [PATCH 5/6] selftests: Preserve subtarget failures in all/install

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 441 bytes --]

On Thu, Apr 16, 2026 at 03:15:30PM -0600, Shuah Khan wrote:
> On 4/16/26 09:21, Shuah Khan wrote:

> > Yes 2964f6b816c2 is my first suspect with ktap_helpers.sh no such file.
> > Thank you for fixing this. That way we can patch it and not revert it.

> Just a quick note to say I applied all 3 patches and will send pr tomorrow.

Thanks Shuah!  I'll try to confirm everything looks good when they're
all merged together after today's -next.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH 5/6] selftests: Preserve subtarget failures in all/install

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 414 bytes --]

On Thu, Apr 16, 2026 at 03:15:30PM -0600, Shuah Khan wrote:

> Just a quick note to say I applied all 3 patches and will send pr tomorrow.

Confirmed that everything seems to be running fine in -next for me.  I
am seeing the test number duplicated in the log messages from the
wrapper:

  ok 59 59 selftests: arm64: syscall-abi

which is confusing my parser a little but is less of an emergency, I'll
take a look.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH 5/6] selftests: Preserve subtarget failures in all/install

[Shuah Khan]

On 4/17/26 07:30, Mark Brown wrote:
> On Thu, Apr 16, 2026 at 03:15:30PM -0600, Shuah Khan wrote:
> 
>> Just a quick note to say I applied all 3 patches and will send pr tomorrow.
> 
> Confirmed that everything seems to be running fine in -next for me.  I
> am seeing the test number duplicated in the log messages from the
> wrapper:
> 
>    ok 59 59 selftests: arm64: syscall-abi
> 
> which is confusing my parser a little but is less of an emergency, I'll
> take a look.

Just about to send pr when I noticed your patch to fix the above.
I applied it to linux-kselftest next and will send a combined pr
with all 4 patches tomorrow.

Thanks for fixing these regressions.

thanks,
-- Shuah