Re: [syzbot] [ext4?] possible deadlock in ext4_setattr

Re: [syzbot] [ext4?] possible deadlock in ext4_setattr

[syzbot]

syzbot has bisected this issue to:

commit a3c06ae158dd6fa8336157c31d9234689d068d02
Author: Parav Pandit <parav@nvidia.com>
Date:   Tue Jan 5 10:32:03 2021 +0000

    vdpa_sim_net: Add support for user supported devices

bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=16e372c6280000
start commit:   e922ba281a8d Add linux-next specific files for 20230512
git tree:       linux-next
final oops:     https://syzkaller.appspot.com/x/report.txt?x=15e372c6280000
console output: https://syzkaller.appspot.com/x/log.txt?x=11e372c6280000
kernel config:  https://syzkaller.appspot.com/x/.config?x=17a4c2d44484b62f
dashboard link: https://syzkaller.appspot.com/bug?extid=cbb68193bdb95af4340a
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=172a21c6280000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=16b67fc6280000

Reported-by: syzbot+cbb68193bdb95af4340a@syzkaller.appspotmail.com
Fixes: a3c06ae158dd ("vdpa_sim_net: Add support for user supported devices")

For information about bisection process see: https://goo.gl/tpsmEJ#bisection

Re: [syzbot] [ext4?] possible deadlock in ext4_setattr

[Michael S. Tsirkin]

On Sun, May 14, 2023 at 12:24:32PM -0700, syzbot wrote:
> syzbot has bisected this issue to:
> 
> commit a3c06ae158dd6fa8336157c31d9234689d068d02
> Author: Parav Pandit <parav@nvidia.com>
> Date:   Tue Jan 5 10:32:03 2021 +0000
> 
>     vdpa_sim_net: Add support for user supported devices
> 
> bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=16e372c6280000
> start commit:   e922ba281a8d Add linux-next specific files for 20230512
> git tree:       linux-next
> final oops:     https://syzkaller.appspot.com/x/report.txt?x=15e372c6280000
> console output: https://syzkaller.appspot.com/x/log.txt?x=11e372c6280000
> kernel config:  https://syzkaller.appspot.com/x/.config?x=17a4c2d44484b62f
> dashboard link: https://syzkaller.appspot.com/bug?extid=cbb68193bdb95af4340a
> syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=172a21c6280000
> C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=16b67fc6280000
> 
> Reported-by: syzbot+cbb68193bdb95af4340a@syzkaller.appspotmail.com
> Fixes: a3c06ae158dd ("vdpa_sim_net: Add support for user supported devices")
> 
> For information about bisection process see: https://goo.gl/tpsmEJ#bisection

I don't see how this can be related, I don't think the test setup uses
vdpa sim at all.

-- 
MST

Re: [syzbot] [ext4?] possible deadlock in ext4_setattr

[Theodore Ts'o]

On Sun, May 14, 2023 at 04:39:36PM -0400, Michael S. Tsirkin wrote:
> On Sun, May 14, 2023 at 12:24:32PM -0700, syzbot wrote:
> > syzbot has bisected this issue to:
> > 
> > commit a3c06ae158dd6fa8336157c31d9234689d068d02
> > Author: Parav Pandit <parav@nvidia.com>
> > Date:   Tue Jan 5 10:32:03 2021 +0000
> > 
> >     vdpa_sim_net: Add support for user supported devices
> > 
> > For information about bisection process see: https://goo.gl/tpsmEJ#bisection
> 
> I don't see how this can be related, I don't think the test setup uses
> vdpa sim at all.

Yeah, it's totally bogus.  You can see it by looking at the bisection
log[1].

[1] https://syzkaller.appspot.com/text?tag=Log&x=16e372c6280000

The initial bisection logs make it clear that it is *trivially* easy
to reproduce, and that the failure signature is:

crashed: possible deadlock in {ext4_xattr_set_handle,ext4_setattr,ext4_xattr_set_handle}

However, somewhere in the bisection, we start seeing this:

run #0: boot failed: WARNING in kvm_wait
run #1: boot failed: WARNING in kvm_wait
run #2: OK
run #3: OK
run #4: OK
run #5: OK
run #6: OK
run #7: OK
run #8: OK
run #9: OK

This is a completely different failure signature, and the "WARNING in
kvm_wait" should be ignored, and this should be considered a "git
bisect good".  However, the syzkaller bisection doesn't understand
this, and so it gets sent down the primrose path.  :-(

Unfortunately, there isn't a good way to tell the syzkaller bisection,
"go home, your drunk", and to tell it to try again, perhaps with a
human-supplied discriminator of what should be considered a valid
failure signature.

In the ideal world, a human should be able to give it a setup set of
bisection "good" and "bad" commits, along with a regexp for what a
failure should look like, and perhaps with a hint of how many tries it
should use before assuming that a bisection result is "good", and to
teach it to assume that the bisection has "bad" after seeing a single
failure signature.

If the above is too much to ask, in the slightly-less-ideal world,
there would be a way to give "#syz test" a count argument, so we could
have syzkaller try N times, for when we need to do a manual bisection
using "#syz test"....

						- Ted

Re: [syzbot] [ext4?] possible deadlock in ext4_setattr

[Theodore Ts'o]

On Sun, May 14, 2023 at 08:53:21PM -0400, Theodore Ts'o wrote:
> However, somewhere in the bisection, we start seeing this:
> 
> run #0: boot failed: WARNING in kvm_wait
   ...
> 
> This is a completely different failure signature, and the "WARNING in
> kvm_wait" should be ignored, and this should be considered a "git
> bisect good".  However, the syzkaller bisection doesn't understand
> this, and so it gets sent down the primrose path.  :-(

Sorry, I spoke too soon.  It did get this right; it looks like it
ignores "boot failed: " messages, although it treats is as one of the
10 tries for a particular commit.

I don't see anything obviously wrong with the bisect log, although
obviously I'd want to retry some of the "All OK" results to see if
somehow things got confused.  In any case, there have been a large
number of timees where the bisection results have been less than
correct, and unfortunately, there's not much that can be done other
than just to ignore them.  It would be nice to have a human being be
able to mark the bisection as Obviously Wrong(tm), and maybe ask it do
do a slightly different bisection.

Also unfortunate is that I've had more than one case where the problem
doesn't reproduce at all using KVM, but only reproduces using #syz
test.  Which means that manual bisection may be the only way for now
to work some of these.

					- Ted