* Re: BUG: corrupted list in __dentry_kill (2)
2019-12-12 5:59 syzbot
@ 2019-12-12 6:12 ` Al Viro
2019-12-12 6:48 ` Dmitry Vyukov
2019-12-12 11:35 ` syzbot
2019-12-12 11:35 ` syzbot
2 siblings, 1 reply; 11+ messages in thread
From: Al Viro @ 2019-12-12 6:12 UTC (permalink / raw)
To: syzbot; +Cc: linux-fsdevel, linux-kernel, syzkaller-bugs
On Wed, Dec 11, 2019 at 09:59:11PM -0800, syzbot wrote:
> Hello,
>
> syzbot found the following crash on:
>
> HEAD commit: 938f49c8 Add linux-next specific files for 20191211
> git tree: linux-next
> console output: https://syzkaller.appspot.com/x/log.txt?x=150eba1ee00000
> kernel config: https://syzkaller.appspot.com/x/.config?x=96834c884ba7bb81
> dashboard link: https://syzkaller.appspot.com/bug?extid=31043da7725b6ec210f1
> compiler: gcc (GCC) 9.0.0 20181231 (experimental)
> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=12dc83dae00000
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=16ac8396e00000
>
> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> Reported-by: syzbot+31043da7725b6ec210f1@syzkaller.appspotmail.com
Already fixed in a3d1e7eb5abe3aa1095bc75d1a6760d3809bd672
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: BUG: corrupted list in __dentry_kill (2)
2019-12-12 6:12 ` Al Viro
@ 2019-12-12 6:48 ` Dmitry Vyukov
2019-12-12 13:38 ` Al Viro
0 siblings, 1 reply; 11+ messages in thread
From: Dmitry Vyukov @ 2019-12-12 6:48 UTC (permalink / raw)
To: Al Viro; +Cc: syzbot, linux-fsdevel, LKML, syzkaller-bugs
On Thu, Dec 12, 2019 at 7:12 AM Al Viro <viro@zeniv.linux.org.uk> wrote:
>
> On Wed, Dec 11, 2019 at 09:59:11PM -0800, syzbot wrote:
> > Hello,
> >
> > syzbot found the following crash on:
> >
> > HEAD commit: 938f49c8 Add linux-next specific files for 20191211
> > git tree: linux-next
> > console output: https://syzkaller.appspot.com/x/log.txt?x=150eba1ee00000
> > kernel config: https://syzkaller.appspot.com/x/.config?x=96834c884ba7bb81
> > dashboard link: https://syzkaller.appspot.com/bug?extid=31043da7725b6ec210f1
> > compiler: gcc (GCC) 9.0.0 20181231 (experimental)
> > syz repro: https://syzkaller.appspot.com/x/repro.syz?x=12dc83dae00000
> > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=16ac8396e00000
> >
> > IMPORTANT: if you fix the bug, please add the following tag to the commit:
> > Reported-by: syzbot+31043da7725b6ec210f1@syzkaller.appspotmail.com
>
> Already fixed in a3d1e7eb5abe3aa1095bc75d1a6760d3809bd672
This commit was in the tested tree already as far as I can see.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: BUG: corrupted list in __dentry_kill (2)
2019-12-12 6:48 ` Dmitry Vyukov
@ 2019-12-12 13:38 ` Al Viro
2019-12-12 15:57 ` Dmitry Vyukov
0 siblings, 1 reply; 11+ messages in thread
From: Al Viro @ 2019-12-12 13:38 UTC (permalink / raw)
To: Dmitry Vyukov; +Cc: syzbot, linux-fsdevel, LKML, syzkaller-bugs
On Thu, Dec 12, 2019 at 07:48:14AM +0100, Dmitry Vyukov wrote:
> On Thu, Dec 12, 2019 at 7:12 AM Al Viro <viro@zeniv.linux.org.uk> wrote:
> >
> > On Wed, Dec 11, 2019 at 09:59:11PM -0800, syzbot wrote:
> > > Hello,
> > >
> > > syzbot found the following crash on:
> > >
> > > HEAD commit: 938f49c8 Add linux-next specific files for 20191211
> > > git tree: linux-next
> > > console output: https://syzkaller.appspot.com/x/log.txt?x=150eba1ee00000
> > > kernel config: https://syzkaller.appspot.com/x/.config?x=96834c884ba7bb81
> > > dashboard link: https://syzkaller.appspot.com/bug?extid=31043da7725b6ec210f1
> > > compiler: gcc (GCC) 9.0.0 20181231 (experimental)
> > > syz repro: https://syzkaller.appspot.com/x/repro.syz?x=12dc83dae00000
> > > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=16ac8396e00000
> > >
> > > IMPORTANT: if you fix the bug, please add the following tag to the commit:
> > > Reported-by: syzbot+31043da7725b6ec210f1@syzkaller.appspotmail.com
> >
> > Already fixed in a3d1e7eb5abe3aa1095bc75d1a6760d3809bd672
>
> This commit was in the tested tree already as far as I can see.
Broken version (653f0d05be0948e7610bb786e6570bb6c48a4e75) is there, its
fixed replacement (a3d1e7eb5abe3aa1095bc75d1a6760d3809bd672) is not.
Look, I realize that your setup is oriented to "followup commit Y fixes
a bug in earlier commit X", and sometimes it's the only possibility
(when X has already been in mainline), but in general it's spelled
"bisection hazard for no damn reason". Fixes are folded in.
Routinely. What's more, in this case the fixed version had been done
(and pushed out) before syzbot has seen the original, so putting
any metadata into commit message hadn't been an option.
If there is some format understandable for syzbot for such cases
("bug is caused by commit X; Y is a replacement that should not
exhibit the same bug, so if you see that behaviour on a tree
that doesn't contain X, report it. X-containing trees ought
to go extinct reasonably soon"), please tell what it is.
Otherwise this situation will keep repeating - I am not going
to stop folding fixes into developing patches.
Speaking of bisect hazards, I'd recommend to check how your bisect
went - the bug is definitely local to this commit and I really
wonder what had caused the bisect to go wrong in this particular
case.
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: BUG: corrupted list in __dentry_kill (2)
2019-12-12 13:38 ` Al Viro
@ 2019-12-12 15:57 ` Dmitry Vyukov
2019-12-12 18:34 ` Al Viro
0 siblings, 1 reply; 11+ messages in thread
From: Dmitry Vyukov @ 2019-12-12 15:57 UTC (permalink / raw)
To: Al Viro; +Cc: syzbot, linux-fsdevel, LKML, syzkaller-bugs
On Thu, Dec 12, 2019 at 2:38 PM Al Viro <viro@zeniv.linux.org.uk> wrote:
>
> On Thu, Dec 12, 2019 at 07:48:14AM +0100, Dmitry Vyukov wrote:
> > On Thu, Dec 12, 2019 at 7:12 AM Al Viro <viro@zeniv.linux.org.uk> wrote:
> > >
> > > On Wed, Dec 11, 2019 at 09:59:11PM -0800, syzbot wrote:
> > > > Hello,
> > > >
> > > > syzbot found the following crash on:
> > > >
> > > > HEAD commit: 938f49c8 Add linux-next specific files for 20191211
> > > > git tree: linux-next
> > > > console output: https://syzkaller.appspot.com/x/log.txt?x=150eba1ee00000
> > > > kernel config: https://syzkaller.appspot.com/x/.config?x=96834c884ba7bb81
> > > > dashboard link: https://syzkaller.appspot.com/bug?extid=31043da7725b6ec210f1
> > > > compiler: gcc (GCC) 9.0.0 20181231 (experimental)
> > > > syz repro: https://syzkaller.appspot.com/x/repro.syz?x=12dc83dae00000
> > > > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=16ac8396e00000
> > > >
> > > > IMPORTANT: if you fix the bug, please add the following tag to the commit:
> > > > Reported-by: syzbot+31043da7725b6ec210f1@syzkaller.appspotmail.com
> > >
> > > Already fixed in a3d1e7eb5abe3aa1095bc75d1a6760d3809bd672
> >
> > This commit was in the tested tree already as far as I can see.
>
> Broken version (653f0d05be0948e7610bb786e6570bb6c48a4e75) is there, its
> fixed replacement (a3d1e7eb5abe3aa1095bc75d1a6760d3809bd672) is not.
Right, sorry, I looked only at the title when checked if it's in the
tree or not.
> Look, I realize that your setup is oriented to "followup commit Y fixes
> a bug in earlier commit X", and sometimes it's the only possibility
> (when X has already been in mainline), but in general it's spelled
> "bisection hazard for no damn reason". Fixes are folded in.
> Routinely. What's more, in this case the fixed version had been done
> (and pushed out) before syzbot has seen the original, so putting
> any metadata into commit message hadn't been an option.
>
> If there is some format understandable for syzbot for such cases
> ("bug is caused by commit X; Y is a replacement that should not
> exhibit the same bug, so if you see that behaviour on a tree
> that doesn't contain X, report it. X-containing trees ought
> to go extinct reasonably soon"), please tell what it is.
> Otherwise this situation will keep repeating - I am not going
> to stop folding fixes into developing patches.
I did not find anything that could serve as these X/Y here. Commit
hashes in linux-next are pointless and won't match against other
trees. Commit titles seem to be the best bet for kernel, but this is
exactly the case where they fall short.
But in this case I think we can safely mark this as:
#syz fix:
simple_recursive_removal(): kernel-side rm -rf for ramfs-style filesystems
This should work because syzbot does coarser-grained check than "if
you see that behaviour on a tree that doesn't contain X". Instead it
will wait until _all_ tested trees will have X and only then close the
bug and report it again if it sees it again. Since the commit is
already fixed in linux-next, when all trees including mainline and
bpf/bpf-next will get it, it should be the fixed version.
> Speaking of bisect hazards, I'd recommend to check how your bisect
> went - the bug is definitely local to this commit and I really
> wonder what had caused the bisect to go wrong in this particular
> case.
I did not get the relation of folding to bisection. Or you mean these
are just separate things?
In this particular case the reason seems to be the same as for most
kernel bisections -- too many unrelated bugs layering one onto
another.
If you are interested in more details about why kernel bisection go
wrong in general, I've published a bunch of info here (including
detailed stats on reason):
https://groups.google.com/g/syzkaller/c/sR8aAXaWEF4/m/tTWYRgvmAwAJ
https://github.com/google/syzkaller/issues/1051
https://docs.google.com/spreadsheets/d/1WdBAN54-csaZpD3LgmTcIMR7NDFuQoOZZqPZ-CUqQgA/edit#gid=0
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: BUG: corrupted list in __dentry_kill (2)
2019-12-12 15:57 ` Dmitry Vyukov
@ 2019-12-12 18:34 ` Al Viro
2019-12-13 9:10 ` Dmitry Vyukov
0 siblings, 1 reply; 11+ messages in thread
From: Al Viro @ 2019-12-12 18:34 UTC (permalink / raw)
To: Dmitry Vyukov; +Cc: syzbot, linux-fsdevel, LKML, syzkaller-bugs
On Thu, Dec 12, 2019 at 04:57:14PM +0100, Dmitry Vyukov wrote:
> > Speaking of bisect hazards, I'd recommend to check how your bisect
> > went - the bug is definitely local to this commit and I really
> > wonder what had caused the bisect to go wrong in this particular
> > case.
>
> I did not get the relation of folding to bisection. Or you mean these
> are just separate things?
Suppose instead of folding the fix in I would've done a followup commit
just with the fix. And left the branch in that form, eventually getting
it pulled into mainline. From that point on, *ANY* bisect stepping into
the first commit would've been thrown off. For ever and ever, since
once it's in mainline, it really won't go away.
That's what folding avoids - accumulation of scar tissue, if you will.
Sure, there's enough cases when bug is found too late - it's already
in mainline or pulled into net-next or some other branch with similar
"no rebase, no reorder" policy. But if you look at the patchsets posted
on the lists and watch them from iteration to iteration, you'll see
a _lot_ of fix-folding. IME (both by my own practice and by watching
the patchsets posted by others) it outnumbers the cases when fix can't
be folded by quite a factor. I wouldn't be surprised if it was an
order of magnitude...
Strict "never fold fixes" policy would've accelerated the accumulation
of bisect hazards in the mainline. And while useful bisect may be a lost
cause for CI bots, it isn't that for intelligent developers. Anything
that makes it more painful is not going to be welcome.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: BUG: corrupted list in __dentry_kill (2)
2019-12-12 18:34 ` Al Viro
@ 2019-12-13 9:10 ` Dmitry Vyukov
0 siblings, 0 replies; 11+ messages in thread
From: Dmitry Vyukov @ 2019-12-13 9:10 UTC (permalink / raw)
To: Al Viro; +Cc: syzbot, linux-fsdevel, LKML, syzkaller-bugs
On Thu, Dec 12, 2019 at 7:34 PM Al Viro <viro@zeniv.linux.org.uk> wrote:
>
> On Thu, Dec 12, 2019 at 04:57:14PM +0100, Dmitry Vyukov wrote:
>
> > > Speaking of bisect hazards, I'd recommend to check how your bisect
> > > went - the bug is definitely local to this commit and I really
> > > wonder what had caused the bisect to go wrong in this particular
> > > case.
> >
> > I did not get the relation of folding to bisection. Or you mean these
> > are just separate things?
>
> Suppose instead of folding the fix in I would've done a followup commit
> just with the fix. And left the branch in that form, eventually getting
> it pulled into mainline. From that point on, *ANY* bisect stepping into
> the first commit would've been thrown off. For ever and ever, since
> once it's in mainline, it really won't go away.
>
> That's what folding avoids - accumulation of scar tissue, if you will.
> Sure, there's enough cases when bug is found too late - it's already
> in mainline or pulled into net-next or some other branch with similar
> "no rebase, no reorder" policy. But if you look at the patchsets posted
> on the lists and watch them from iteration to iteration, you'll see
> a _lot_ of fix-folding. IME (both by my own practice and by watching
> the patchsets posted by others) it outnumbers the cases when fix can't
> be folded by quite a factor. I wouldn't be surprised if it was an
> order of magnitude...
>
> Strict "never fold fixes" policy would've accelerated the accumulation
> of bisect hazards in the mainline. And while useful bisect may be a lost
> cause for CI bots, it isn't that for intelligent developers. Anything
> that makes it more painful is not going to be welcome.
Ah, I see. Yes, folding will help future bisections. In fact, an
unfolded bug somewhere in kernel history is exactly what caused wrong
result on bisection for this bug.
Just in case, I did not propose to not do folding here (or anywhere
else as far as I remember). Handling of folded fixes is documented in
syzbot docs:
https://goo.gl/tpsmEJ#rebuilt-treesamended-patches
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: BUG: corrupted list in __dentry_kill (2)
2019-12-12 5:59 syzbot
2019-12-12 6:12 ` Al Viro
@ 2019-12-12 11:35 ` syzbot
2019-12-12 11:35 ` syzbot
2 siblings, 0 replies; 11+ messages in thread
From: syzbot @ 2019-12-12 11:35 UTC (permalink / raw)
To: a, alex.aring, allison, andrew, andy, ap420073, ast, ast,
b.a.t.m.a.n, bpf, bridge, cleech, daniel, davem, dsa, dsahern,
dvyukov, f.fainelli, fw, gregkh, haiyangz, hawk, hdanton, idosch,
info, j.vosburgh, j, jakub.kicinski, jhs, jiri, jiri,
johan.hedberg, johannes.berg, john.fastabend, john.hurley, jwi,
kafai, kstewart, kvalo, kys, linux-bluetooth
syzbot has bisected this bug to:
commit ab92d68fc22f9afab480153bd82a20f6e2533769
Author: Taehee Yoo <ap420073@gmail.com>
Date: Mon Oct 21 18:47:51 2019 +0000
net: core: add generic lockdep keys
bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=12d37cb6e00000
start commit: 938f49c8 Add linux-next specific files for 20191211
git tree: linux-next
final crash: https://syzkaller.appspot.com/x/report.txt?x=11d37cb6e00000
console output: https://syzkaller.appspot.com/x/log.txt?x=16d37cb6e00000
kernel config: https://syzkaller.appspot.com/x/.config?x=96834c884ba7bb81
dashboard link: https://syzkaller.appspot.com/bug?extid=31043da7725b6ec210f1
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=12dc83dae00000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=16ac8396e00000
Reported-by: syzbot+31043da7725b6ec210f1@syzkaller.appspotmail.com
Fixes: ab92d68fc22f ("net: core: add generic lockdep keys")
For information about bisection process see: https://goo.gl/tpsmEJ#bisection
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: BUG: corrupted list in __dentry_kill (2)
2019-12-12 5:59 syzbot
@ 2019-12-12 11:35 ` syzbot
2019-12-12 11:35 ` syzbot
2019-12-12 11:35 ` syzbot
2 siblings, 0 replies; 11+ messages in thread
From: syzbot @ 2019-12-12 11:35 UTC (permalink / raw)
To: a, alex.aring, allison, andrew, andy, ap420073, ast, ast,
b.a.t.m.a.n, bpf, bridge, cleech, daniel, davem, dsa, dsahern,
dvyukov, f.fainelli, fw, gregkh, haiyangz, hawk, hdanton, idosch,
info, j.vosburgh, j, jakub.kicinski, jhs, jiri, jiri,
johan.hedberg, johannes.berg, john.fastabend, john.hurley, jwi,
kafai, kstewart, kvalo, kys, linux-bluetooth, linux-fsdevel,
linux-hams, linux-hyperv, linux-kernel, linux-ppp, linux-wireless,
linux-wpan, liuhangbin, marcel
syzbot has bisected this bug to:
commit ab92d68fc22f9afab480153bd82a20f6e2533769
Author: Taehee Yoo <ap420073@gmail.com>
Date: Mon Oct 21 18:47:51 2019 +0000
net: core: add generic lockdep keys
bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=12d37cb6e00000
start commit: 938f49c8 Add linux-next specific files for 20191211
git tree: linux-next
final crash: https://syzkaller.appspot.com/x/report.txt?x=11d37cb6e00000
console output: https://syzkaller.appspot.com/x/log.txt?x=16d37cb6e00000
kernel config: https://syzkaller.appspot.com/x/.config?x=96834c884ba7bb81
dashboard link: https://syzkaller.appspot.com/bug?extid=31043da7725b6ec210f1
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=12dc83dae00000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=16ac8396e00000
Reported-by: syzbot+31043da7725b6ec210f1@syzkaller.appspotmail.com
Fixes: ab92d68fc22f ("net: core: add generic lockdep keys")
For information about bisection process see: https://goo.gl/tpsmEJ#bisection
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: BUG: corrupted list in __dentry_kill (2)
@ 2019-12-12 11:35 ` syzbot
0 siblings, 0 replies; 11+ messages in thread
From: syzbot @ 2019-12-12 11:35 UTC (permalink / raw)
To: a, alex.aring, allison, andrew, andy, ap420073, ast, ast,
b.a.t.m.a.n, bpf, bridge, cleech, daniel, davem, dsa, dsahern,
dvyukov, f.fainelli, fw, gregkh, haiyangz, hawk, hdanton, idosch,
info, j.vosburgh, j, jakub.kicinski, jhs, jiri, jiri,
johan.hedberg, johannes.berg, john.fastabend, john.hurley, jwi,
kafai, kstewart, kvalo, kys, linux-bluetooth, linux-fsdevel,
linux-hams, linux-hyperv, linux-kernel, linux-ppp, linux-wireless,
linux-wpan, liuhangbin, marcel
syzbot has bisected this bug to:
commit ab92d68fc22f9afab480153bd82a20f6e2533769
Author: Taehee Yoo <ap420073@gmail.com>
Date: Mon Oct 21 18:47:51 2019 +0000
net: core: add generic lockdep keys
bisection log: https://syzkaller.appspot.com/x/bisect.txt?x\x12d37cb6e00000
start commit: 938f49c8 Add linux-next specific files for 20191211
git tree: linux-next
final crash: https://syzkaller.appspot.com/x/report.txt?x\x11d37cb6e00000
console output: https://syzkaller.appspot.com/x/log.txt?x\x16d37cb6e00000
kernel config: https://syzkaller.appspot.com/x/.config?x–834c884ba7bb81
dashboard link: https://syzkaller.appspot.com/bug?extid1043da7725b6ec210f1
syz repro: https://syzkaller.appspot.com/x/repro.syz?x\x12dc83dae00000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x\x16ac8396e00000
Reported-by: syzbot+31043da7725b6ec210f1@syzkaller.appspotmail.com
Fixes: ab92d68fc22f ("net: core: add generic lockdep keys")
For information about bisection process see: https://goo.gl/tpsmEJ#bisection
^ permalink raw reply [flat|nested] 11+ messages in thread