Missing fix backports detected by syzbot

Missing fix backports detected by syzbot

[Aleksandr Nogikh]

Hi Greg, Sasha,

A number of commits were identified[1] by syzbot as non-backported
fixes for the fuzzer-detected findings in various Linux LTS trees.

[1] https://syzkaller.appspot.com/upstream/backports

Please consider backporting the following commits to LTS v6.1:
9a8ec9e8ebb5a7c0cfbce2d6b4a6b67b2b78e8f3 "Bluetooth: SCO: Fix possible circular locking dependency on sco_connect_cfm"
(fixes 9a8ec9e) 3dcaa192ac2159193bc6ab57bc5369dcb84edd8e "Bluetooth: SCO: fix sco_conn related locking and validity issues"
3f5424790d4377839093b68c12b130077a4e4510 "ext4: fix inode tree inconsistency caused by ENOMEM"
7b0151caf73a656b75b550e361648430233455a0 "KVM: x86: Remove WARN sanity check on hypervisor timer vs. UNINITIALIZED vCPU"
c2efd13a2ed4f29bf9ef14ac2fbb7474084655f8 "udf: Limit file size to 4TB"
4b827b3f305d1fcf837265f1e12acc22ee84327c "xfs: remove WARN when dquot cache insertion fails"

These were verified to apply cleanly on top of v6.1.107 and to
build/boot.

The following commits to LTS v5.15:
8216776ccff6fcd40e3fdaa109aa4150ebe760b3 "ext4: reject casefold inode flag without casefold feature"
c2efd13a2ed4f29bf9ef14ac2fbb7474084655f8 "udf: Limit file size to 4TB"

These were verified to apply cleanly on top of v5.15.165 and to
build/boot.

The following commits to LTS v5.10:
04e568a3b31cfbd545c04c8bfc35c20e5ccfce0f "ext4: handle redirtying in ext4_bio_write_page()"
2a1fc7dc36260fbe74b6ca29dc6d9088194a2115 "KVM: x86: Suppress MMIO that is triggered during task switch emulation"
2454ad83b90afbc6ed2c22ec1310b624c40bf0d3 "fs: Restrict lock_two_nondirectories() to non-directory inodes"
(fixes 2454ad) 33ab231f83cc12d0157711bbf84e180c3be7d7bc "fs: don't assume arguments are non-NULL"

These were verified to apply cleanly on top of v5.10.224 and to
build/boot.

There are also a lot of syzbot-detected fix commits that did not apply
cleanly, but the conflicts seem to be quite straightforward to resolve
manually. Could you please share what the current process is with
respect to such fix patches? For example, are you sending emails
asking developers to adjust the non-applied patch (if they want), or
is it the other way around -- you expect the authors to be proactive
and send the adjusted patch versions themselves?

Some sample commits, which failed to apply to v6.1.107:
ff91059932401894e6c86341915615c5eb0eca48 "bpf, sockmap: Prevent lock inversion deadlock in map delete elem"
f8f210dc84709804c9f952297f2bfafa6ea6b4bd "btrfs: calculate the right space for delayed refs when updating global reserve"

-- 
Aleksandr

Re: Missing fix backports detected by syzbot

[Greg KH]

On Wed, Sep 04, 2024 at 12:24:55PM +0200, Aleksandr Nogikh wrote:
> There are also a lot of syzbot-detected fix commits that did not apply
> cleanly, but the conflicts seem to be quite straightforward to resolve
> manually. Could you please share what the current process is with
> respect to such fix patches? For example, are you sending emails
> asking developers to adjust the non-applied patch (if they want), or
> is it the other way around -- you expect the authors to be proactive
> and send the adjusted patch versions themselves?

We expect someone to send us patches, we aren't going to be able to do
it ourselves :)

thanks,

greg k-h

Re: Missing fix backports detected by syzbot

[Greg KH]

On Wed, Sep 04, 2024 at 12:24:55PM +0200, Aleksandr Nogikh wrote:
> Hi Greg, Sasha,
> 
> A number of commits were identified[1] by syzbot as non-backported
> fixes for the fuzzer-detected findings in various Linux LTS trees.
> 
> [1] https://syzkaller.appspot.com/upstream/backports
> 
> Please consider backporting the following commits to LTS v6.1:
> 9a8ec9e8ebb5a7c0cfbce2d6b4a6b67b2b78e8f3 "Bluetooth: SCO: Fix possible circular locking dependency on sco_connect_cfm"
> (fixes 9a8ec9e) 3dcaa192ac2159193bc6ab57bc5369dcb84edd8e "Bluetooth: SCO: fix sco_conn related locking and validity issues"
> 3f5424790d4377839093b68c12b130077a4e4510 "ext4: fix inode tree inconsistency caused by ENOMEM"
> 7b0151caf73a656b75b550e361648430233455a0 "KVM: x86: Remove WARN sanity check on hypervisor timer vs. UNINITIALIZED vCPU"

Note, for kvm, and for:

> 4b827b3f305d1fcf837265f1e12acc22ee84327c "xfs: remove WARN when dquot cache insertion fails"

xfs patches, we need explicit approval from the subsystem maintainers to
take them into stable as they were not marked for such.

Please work with them to get those patches merged if you wish to see
them in stable trees.

thanks,

greg k-h

Re: Missing fix backports detected by syzbot

[Greg KH]

On Wed, Sep 04, 2024 at 12:24:55PM +0200, Aleksandr Nogikh wrote:
> Hi Greg, Sasha,
> 
> A number of commits were identified[1] by syzbot as non-backported
> fixes for the fuzzer-detected findings in various Linux LTS trees.
> 
> [1] https://syzkaller.appspot.com/upstream/backports
> 
> Please consider backporting the following commits to LTS v6.1:
> 9a8ec9e8ebb5a7c0cfbce2d6b4a6b67b2b78e8f3 "Bluetooth: SCO: Fix possible circular locking dependency on sco_connect_cfm"
> (fixes 9a8ec9e) 3dcaa192ac2159193bc6ab57bc5369dcb84edd8e "Bluetooth: SCO: fix sco_conn related locking and validity issues"
> 3f5424790d4377839093b68c12b130077a4e4510 "ext4: fix inode tree inconsistency caused by ENOMEM"
> 7b0151caf73a656b75b550e361648430233455a0 "KVM: x86: Remove WARN sanity check on hypervisor timer vs. UNINITIALIZED vCPU"
> c2efd13a2ed4f29bf9ef14ac2fbb7474084655f8 "udf: Limit file size to 4TB"
> 4b827b3f305d1fcf837265f1e12acc22ee84327c "xfs: remove WARN when dquot cache insertion fails"
> 
> These were verified to apply cleanly on top of v6.1.107 and to
> build/boot.
> 
> The following commits to LTS v5.15:
> 8216776ccff6fcd40e3fdaa109aa4150ebe760b3 "ext4: reject casefold inode flag without casefold feature"

Wait, what about 6.1 for this?  We can't move to a new kernel and have a
regression.

> c2efd13a2ed4f29bf9ef14ac2fbb7474084655f8 "udf: Limit file size to 4TB"
> 
> These were verified to apply cleanly on top of v5.15.165 and to
> build/boot.
> 
> The following commits to LTS v5.10:
> 04e568a3b31cfbd545c04c8bfc35c20e5ccfce0f "ext4: handle redirtying in ext4_bio_write_page()"

Same here, what about 5.15.y?

> 2a1fc7dc36260fbe74b6ca29dc6d9088194a2115 "KVM: x86: Suppress MMIO that is triggered during task switch emulation"
> 2454ad83b90afbc6ed2c22ec1310b624c40bf0d3 "fs: Restrict lock_two_nondirectories() to non-directory inodes"
> (fixes 2454ad) 33ab231f83cc12d0157711bbf84e180c3be7d7bc "fs: don't assume arguments are non-NULL"

Why are these last two needed?

Can you provide full lists of what needs to go to what tree, and better
yet, tested patch series for this type of thing in the future?

thanks,

greg k-h

Re: Missing fix backports detected by syzbot

[Aleksandr Nogikh]

Hi Greg,

Thank you very much for your comments!

On Thu, Sep 5, 2024 at 10:04 AM Greg KH <gregkh@linuxfoundation.org> wrote:
>
> Note, for kvm, and for:
>
> > 4b827b3f305d1fcf837265f1e12acc22ee84327c "xfs: remove WARN when dquot cache insertion fails"
>
> xfs patches, we need explicit approval from the subsystem maintainers to
> take them into stable as they were not marked for such.

Is it specific only to some subsystems (like kvm and xfs), or is it
due to some general rule like "if the commit was not initially marked
as a stable backport candidate, you need an approval from the
subsystem maintainer"?

On Thu, Sep 5, 2024 at 10:12 AM Greg KH <gregkh@linuxfoundation.org> wrote:
>
> On Wed, Sep 04, 2024 at 12:24:55PM +0200, Aleksandr Nogikh wrote:
> > Hi Greg, Sasha,
> >
> > A number of commits were identified[1] by syzbot as non-backported
> > fixes for the fuzzer-detected findings in various Linux LTS trees.
> >
> > [1] https://syzkaller.appspot.com/upstream/backports
> >
> > Please consider backporting the following commits to LTS v6.1:
> > 9a8ec9e8ebb5a7c0cfbce2d6b4a6b67b2b78e8f3 "Bluetooth: SCO: Fix possible circular locking dependency on sco_connect_cfm"
> > (fixes 9a8ec9e) 3dcaa192ac2159193bc6ab57bc5369dcb84edd8e "Bluetooth: SCO: fix sco_conn related locking and validity issues"
> > 3f5424790d4377839093b68c12b130077a4e4510 "ext4: fix inode tree inconsistency caused by ENOMEM"
> > 7b0151caf73a656b75b550e361648430233455a0 "KVM: x86: Remove WARN sanity check on hypervisor timer vs. UNINITIALIZED vCPU"
> > c2efd13a2ed4f29bf9ef14ac2fbb7474084655f8 "udf: Limit file size to 4TB"
> > 4b827b3f305d1fcf837265f1e12acc22ee84327c "xfs: remove WARN when dquot cache insertion fails"
> >
> > These were verified to apply cleanly on top of v6.1.107 and to
> > build/boot.
> >
> > The following commits to LTS v5.15:
> > 8216776ccff6fcd40e3fdaa109aa4150ebe760b3 "ext4: reject casefold inode flag without casefold feature"
>
> Wait, what about 6.1 for this?  We can't move to a new kernel and have a
> regression.

Indeed!
Syzbot currently constructs missing backports lists independently for
each fuzzed LTS, so the page [1] does have some holes. But in any
case, it's totally reasonable that if we backport a commit to an older
kernel, newer ones should also get it.

>
> > c2efd13a2ed4f29bf9ef14ac2fbb7474084655f8 "udf: Limit file size to 4TB"
> >
> > These were verified to apply cleanly on top of v5.15.165 and to
> > build/boot.
> >
> > The following commits to LTS v5.10:
> > 04e568a3b31cfbd545c04c8bfc35c20e5ccfce0f "ext4: handle redirtying in ext4_bio_write_page()"
>
> Same here, what about 5.15.y?
>
> > 2a1fc7dc36260fbe74b6ca29dc6d9088194a2115 "KVM: x86: Suppress MMIO that is triggered during task switch emulation"
> > 2454ad83b90afbc6ed2c22ec1310b624c40bf0d3 "fs: Restrict lock_two_nondirectories() to non-directory inodes"
> > (fixes 2454ad) 33ab231f83cc12d0157711bbf84e180c3be7d7bc "fs: don't assume arguments are non-NULL"
>
> Why are these last two needed?

The last two address the "WARNING: bad unlock balance in
unlock_two_nondirectories" bug on a Linux 5.10-based kernel.
The crash report is very similar to
https://syzkaller.appspot.com/bug?id=32c54626e170a6b327ca2c8ae4c1aea666a8c20b

>
> Can you provide full lists of what needs to go to what tree, and better
> yet, tested patch series for this type of thing in the future?

Sure!
Could you please clarify a bit what should be the criteria for the
full lists and what exact kinds of tests you mean?

If the list only contains the patches that apply cleanly, it is
unfortunately not very big anyway since the vast majority of the
detected missing backports just cannot be cherry-picked automatically.
I guess, otherwise most of them would have already been backported :)
I plan to try to manually adjust and resend some of them to better
understand how much manual effort it actually requires.

For the commits I listed so far, I checked that they apply without
problems, whether there are fix commits for them and whether the
kernel builds and boots fine. It should be easy to also verify whether
syzbot is able to reproduce the bug after the fix is cherry-picked.
What are some other tests that would be great to have run?

-- 
Aleksandr

>
> thanks,
>
> greg k-h

Re: Missing fix backports detected by syzbot

[Greg KH]

On Fri, Sep 06, 2024 at 04:55:07PM +0200, Aleksandr Nogikh wrote:
> Hi Greg,
> 
> Thank you very much for your comments!
> 
> On Thu, Sep 5, 2024 at 10:04 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> >
> > Note, for kvm, and for:
> >
> > > 4b827b3f305d1fcf837265f1e12acc22ee84327c "xfs: remove WARN when dquot cache insertion fails"
> >
> > xfs patches, we need explicit approval from the subsystem maintainers to
> > take them into stable as they were not marked for such.
> 
> Is it specific only to some subsystems (like kvm and xfs), or is it
> due to some general rule like "if the commit was not initially marked
> as a stable backport candidate, you need an approval from the
> subsystem maintainer"?

We have a list, see this file:
	https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git/tree/ignore_list

> > > A number of commits were identified[1] by syzbot as non-backported
> > > fixes for the fuzzer-detected findings in various Linux LTS trees.
> > >
> > > [1] https://syzkaller.appspot.com/upstream/backports
> > >
> > > Please consider backporting the following commits to LTS v6.1:
> > > 9a8ec9e8ebb5a7c0cfbce2d6b4a6b67b2b78e8f3 "Bluetooth: SCO: Fix possible circular locking dependency on sco_connect_cfm"
> > > (fixes 9a8ec9e) 3dcaa192ac2159193bc6ab57bc5369dcb84edd8e "Bluetooth: SCO: fix sco_conn related locking and validity issues"
> > > 3f5424790d4377839093b68c12b130077a4e4510 "ext4: fix inode tree inconsistency caused by ENOMEM"
> > > 7b0151caf73a656b75b550e361648430233455a0 "KVM: x86: Remove WARN sanity check on hypervisor timer vs. UNINITIALIZED vCPU"
> > > c2efd13a2ed4f29bf9ef14ac2fbb7474084655f8 "udf: Limit file size to 4TB"
> > > 4b827b3f305d1fcf837265f1e12acc22ee84327c "xfs: remove WARN when dquot cache insertion fails"
> > >
> > > These were verified to apply cleanly on top of v6.1.107 and to
> > > build/boot.
> > >
> > > The following commits to LTS v5.15:
> > > 8216776ccff6fcd40e3fdaa109aa4150ebe760b3 "ext4: reject casefold inode flag without casefold feature"
> >
> > Wait, what about 6.1 for this?  We can't move to a new kernel and have a
> > regression.
> 
> Indeed!
> Syzbot currently constructs missing backports lists independently for
> each fuzzed LTS, so the page [1] does have some holes. But in any
> case, it's totally reasonable that if we backport a commit to an older
> kernel, newer ones should also get it.

It's actually a requirement, and I am pretty sure, is documented
somewhere...

> > > c2efd13a2ed4f29bf9ef14ac2fbb7474084655f8 "udf: Limit file size to 4TB"
> > >
> > > These were verified to apply cleanly on top of v5.15.165 and to
> > > build/boot.
> > >
> > > The following commits to LTS v5.10:
> > > 04e568a3b31cfbd545c04c8bfc35c20e5ccfce0f "ext4: handle redirtying in ext4_bio_write_page()"
> >
> > Same here, what about 5.15.y?
> >
> > > 2a1fc7dc36260fbe74b6ca29dc6d9088194a2115 "KVM: x86: Suppress MMIO that is triggered during task switch emulation"
> > > 2454ad83b90afbc6ed2c22ec1310b624c40bf0d3 "fs: Restrict lock_two_nondirectories() to non-directory inodes"
> > > (fixes 2454ad) 33ab231f83cc12d0157711bbf84e180c3be7d7bc "fs: don't assume arguments are non-NULL"
> >
> > Why are these last two needed?
> 
> The last two address the "WARNING: bad unlock balance in
> unlock_two_nondirectories" bug on a Linux 5.10-based kernel.
> The crash report is very similar to
> https://syzkaller.appspot.com/bug?id=32c54626e170a6b327ca2c8ae4c1aea666a8c20b

As the changelog does not show that, I'd prefer to get an ack from the
developers involved before accepting core vfs changes that are not
obviously a bugfix and no one has run into before in the wild.

Also, just lockdep fixes to add markings aren't always needed to be
backported, UNLESS someone is reporting that this is annoying their
system tests, as sometimes happens.

> > Can you provide full lists of what needs to go to what tree, and better
> > yet, tested patch series for this type of thing in the future?
> 
> Sure!
> Could you please clarify a bit what should be the criteria for the
> full lists and what exact kinds of tests you mean?

A full list of the git ids, for each branch, you didn't do that here as
a bunch were missed for newer kernels.

> If the list only contains the patches that apply cleanly, it is
> unfortunately not very big anyway since the vast majority of the
> detected missing backports just cannot be cherry-picked automatically.
> I guess, otherwise most of them would have already been backported :)

Sometimes, but if they are not explicitly marked to be backported with a
 cc: stable line, there is no guarantee at all that they will be.

> I plan to try to manually adjust and resend some of them to better
> understand how much manual effort it actually requires.
> 
> For the commits I listed so far, I checked that they apply without
> problems, whether there are fix commits for them and whether the
> kernel builds and boots fine. It should be easy to also verify whether
> syzbot is able to reproduce the bug after the fix is cherry-picked.
> What are some other tests that would be great to have run?

That's all a great start, try doing that first and we can go from there.

thanks,

greg k-h