[syzbot] [fuse?] WARNING in fuse_writepages

[syzbot] [fuse?] WARNING in fuse_writepages

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit:    e32cde8d2bd7 Merge tag 'sched_ext-for-6.12-rc1-fixes-1' of..
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=12e8bdd0580000
kernel config:  https://syzkaller.appspot.com/x/.config?x=1b5201b91035a876
dashboard link: https://syzkaller.appspot.com/bug?extid=217a976dc26ef2fa8711
compiler:       gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/a585cdb91cda/disk-e32cde8d.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/dbeec5d7b296/vmlinux-e32cde8d.xz
kernel image: https://storage.googleapis.com/syzbot-assets/000fd790e08a/bzImage-e32cde8d.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+217a976dc26ef2fa8711@syzkaller.appspotmail.com

------------[ cut here ]------------
WARNING: CPU: 0 PID: 5296 at fs/fuse/file.c:1989 fuse_write_file_get fs/fuse/file.c:1989 [inline]
WARNING: CPU: 0 PID: 5296 at fs/fuse/file.c:1989 fuse_write_file_get fs/fuse/file.c:1986 [inline]
WARNING: CPU: 0 PID: 5296 at fs/fuse/file.c:1989 fuse_writepages+0x497/0x5a0 fs/fuse/file.c:2368
Modules linked in:
CPU: 0 UID: 0 PID: 5296 Comm: kworker/u8:8 Not tainted 6.12.0-rc1-syzkaller-00031-ge32cde8d2bd7 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
Workqueue: writeback wb_workfn (flush-0:52)
RIP: 0010:fuse_write_file_get fs/fuse/file.c:1989 [inline]
RIP: 0010:fuse_write_file_get fs/fuse/file.c:1986 [inline]
RIP: 0010:fuse_writepages+0x497/0x5a0 fs/fuse/file.c:2368
Code: 00 00 00 44 89 f8 5b 5d 41 5c 41 5d 41 5e 41 5f c3 cc cc cc cc e8 79 b6 90 fe 48 8b 7c 24 08 e8 af 6f 27 08 e8 6a b6 90 fe 90 <0f> 0b 90 41 bf fb ff ff ff eb 8b e8 59 b6 90 fe 48 8b 7c 24 18 be
RSP: 0018:ffffc900044ff4a8 EFLAGS: 00010293
RAX: 0000000000000000 RBX: ffffc900044ff4f8 RCX: 0000000000000000
RDX: ffff88802d42da00 RSI: ffffffff82fcd286 RDI: 0000000000000001
RBP: ffff88805c994aa0 R08: 0000000000000000 R09: ffffed100b9329d7
R10: ffff88805c994ebb R11: 0000000000000003 R12: ffffc900044ff840
R13: ffff88805c994880 R14: ffff88805f330000 R15: ffff88805c994d50
FS:  0000000000000000(0000) GS:ffff8880b8600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000020055000 CR3: 000000005df4a000 CR4: 00000000003526f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <TASK>
 do_writepages+0x1a3/0x7f0 mm/page-writeback.c:2683
 __writeback_single_inode+0x166/0xfa0 fs/fs-writeback.c:1658
 writeback_sb_inodes+0x603/0xfa0 fs/fs-writeback.c:1954
 wb_writeback+0x199/0xb50 fs/fs-writeback.c:2134
 wb_do_writeback fs/fs-writeback.c:2281 [inline]
 wb_workfn+0x294/0xbc0 fs/fs-writeback.c:2321
 process_one_work+0x9c5/0x1ba0 kernel/workqueue.c:3229
 process_scheduled_works kernel/workqueue.c:3310 [inline]
 worker_thread+0x6c8/0xf00 kernel/workqueue.c:3391
 kthread+0x2c1/0x3a0 kernel/kthread.c:389
 ret_from_fork+0x45/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

Re: [syzbot] [fuse?] WARNING in fuse_writepages

[Joanne Koong]

On Tue, Oct 1, 2024 at 12:24 PM syzbot
<syzbot+217a976dc26ef2fa8711@syzkaller.appspotmail.com> wrote:
>
> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit:    e32cde8d2bd7 Merge tag 'sched_ext-for-6.12-rc1-fixes-1' of..
> git tree:       upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=12e8bdd0580000
> kernel config:  https://syzkaller.appspot.com/x/.config?x=1b5201b91035a876
> dashboard link: https://syzkaller.appspot.com/bug?extid=217a976dc26ef2fa8711
> compiler:       gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40
>
> Unfortunately, I don't have any reproducer for this issue yet.
>
> Downloadable assets:
> disk image: https://storage.googleapis.com/syzbot-assets/a585cdb91cda/disk-e32cde8d.raw.xz
> vmlinux: https://storage.googleapis.com/syzbot-assets/dbeec5d7b296/vmlinux-e32cde8d.xz
> kernel image: https://storage.googleapis.com/syzbot-assets/000fd790e08a/bzImage-e32cde8d.xz
>
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+217a976dc26ef2fa8711@syzkaller.appspotmail.com
>
> ------------[ cut here ]------------
> WARNING: CPU: 0 PID: 5296 at fs/fuse/file.c:1989 fuse_write_file_get fs/fuse/file.c:1989 [inline]
> WARNING: CPU: 0 PID: 5296 at fs/fuse/file.c:1989 fuse_write_file_get fs/fuse/file.c:1986 [inline]
> WARNING: CPU: 0 PID: 5296 at fs/fuse/file.c:1989 fuse_writepages+0x497/0x5a0 fs/fuse/file.c:2368
> Modules linked in:
> CPU: 0 UID: 0 PID: 5296 Comm: kworker/u8:8 Not tainted 6.12.0-rc1-syzkaller-00031-ge32cde8d2bd7 #0
> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
> Workqueue: writeback wb_workfn (flush-0:52)
> RIP: 0010:fuse_write_file_get fs/fuse/file.c:1989 [inline]
> RIP: 0010:fuse_write_file_get fs/fuse/file.c:1986 [inline]
> RIP: 0010:fuse_writepages+0x497/0x5a0 fs/fuse/file.c:2368
> Code: 00 00 00 44 89 f8 5b 5d 41 5c 41 5d 41 5e 41 5f c3 cc cc cc cc e8 79 b6 90 fe 48 8b 7c 24 08 e8 af 6f 27 08 e8 6a b6 90 fe 90 <0f> 0b 90 41 bf fb ff ff ff eb 8b e8 59 b6 90 fe 48 8b 7c 24 18 be
> RSP: 0018:ffffc900044ff4a8 EFLAGS: 00010293
> RAX: 0000000000000000 RBX: ffffc900044ff4f8 RCX: 0000000000000000
> RDX: ffff88802d42da00 RSI: ffffffff82fcd286 RDI: 0000000000000001
> RBP: ffff88805c994aa0 R08: 0000000000000000 R09: ffffed100b9329d7
> R10: ffff88805c994ebb R11: 0000000000000003 R12: ffffc900044ff840
> R13: ffff88805c994880 R14: ffff88805f330000 R15: ffff88805c994d50
> FS:  0000000000000000(0000) GS:ffff8880b8600000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000000020055000 CR3: 000000005df4a000 CR4: 00000000003526f0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Call Trace:
>  <TASK>
>  do_writepages+0x1a3/0x7f0 mm/page-writeback.c:2683
>  __writeback_single_inode+0x166/0xfa0 fs/fs-writeback.c:1658
>  writeback_sb_inodes+0x603/0xfa0 fs/fs-writeback.c:1954
>  wb_writeback+0x199/0xb50 fs/fs-writeback.c:2134
>  wb_do_writeback fs/fs-writeback.c:2281 [inline]
>  wb_workfn+0x294/0xbc0 fs/fs-writeback.c:2321
>  process_one_work+0x9c5/0x1ba0 kernel/workqueue.c:3229
>  process_scheduled_works kernel/workqueue.c:3310 [inline]
>  worker_thread+0x6c8/0xf00 kernel/workqueue.c:3391
>  kthread+0x2c1/0x3a0 kernel/kthread.c:389
>  ret_from_fork+0x45/0x80 arch/x86/kernel/process.c:147
>  ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
>  </TASK>
>

#syz dup: [syzbot] [fuse?] WARNING in fuse_write_file_get (2)

This is the same warning reported in
https://lore.kernel.org/linux-fsdevel/66fbae38.050a0220.6bad9.0051.GAE@google.com/T/#u

The warning is complaining about this WARN_ON here
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/fuse/file.c#n1989.
I think this warning can get triggered if there's a race between a
write() and a close() where the page is dirty in the cache after the
release has happened. Then when writeback (eg fuse_writepages()) is
triggered, we hit this warning. (this possibility has always existed,
it was surfaced after this refactoring commit 4046d3adcca4: "move fuse
file initialization to wpa allocation time" but the actual logic
hasn't been changed).

I think we can address this by instead calling "data.ff =
__fuse_write_file_get(fi);" in fuse_writepages(). I'll submit a fix
for this to Miklos's tree.


Thanks,
Joanne

>
> ---
> This report is generated by a bot. It may contain errors.
> See https://goo.gl/tpsmEJ for more information about syzbot.
> syzbot engineers can be reached at syzkaller@googlegroups.com.
>
> syzbot will keep track of this issue. See:
> https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
>
> If the report is already addressed, let syzbot know by replying with:
> #syz fix: exact-commit-title
>
> If you want to overwrite report's subsystems, reply with:
> #syz set subsystems: new-subsystem
> (See the list of subsystem names on the web dashboard)
>
> If the report is a duplicate of another one, reply with:
> #syz dup: exact-subject-of-another-report
>
> If you want to undo deduplication, reply with:
> #syz undup
>

Re: [syzbot] [fuse?] WARNING in fuse_writepages

[Joanne Koong]

On Tue, Oct 1, 2024 at 5:02 PM Joanne Koong <joannelkoong@gmail.com> wrote:
>
> On Tue, Oct 1, 2024 at 12:24 PM syzbot
> <syzbot+217a976dc26ef2fa8711@syzkaller.appspotmail.com> wrote:
> >
> > Hello,
> >
> > syzbot found the following issue on:
> >
> > HEAD commit:    e32cde8d2bd7 Merge tag 'sched_ext-for-6.12-rc1-fixes-1' of..
> > git tree:       upstream
> > console output: https://syzkaller.appspot.com/x/log.txt?x=12e8bdd0580000
> > kernel config:  https://syzkaller.appspot.com/x/.config?x=1b5201b91035a876
> > dashboard link: https://syzkaller.appspot.com/bug?extid=217a976dc26ef2fa8711
> > compiler:       gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40
> >
> > Unfortunately, I don't have any reproducer for this issue yet.
> >
> > Downloadable assets:
> > disk image: https://storage.googleapis.com/syzbot-assets/a585cdb91cda/disk-e32cde8d.raw.xz
> > vmlinux: https://storage.googleapis.com/syzbot-assets/dbeec5d7b296/vmlinux-e32cde8d.xz
> > kernel image: https://storage.googleapis.com/syzbot-assets/000fd790e08a/bzImage-e32cde8d.xz
> >
> > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > Reported-by: syzbot+217a976dc26ef2fa8711@syzkaller.appspotmail.com
> >
> > ------------[ cut here ]------------
> > WARNING: CPU: 0 PID: 5296 at fs/fuse/file.c:1989 fuse_write_file_get fs/fuse/file.c:1989 [inline]
> > WARNING: CPU: 0 PID: 5296 at fs/fuse/file.c:1989 fuse_write_file_get fs/fuse/file.c:1986 [inline]
> > WARNING: CPU: 0 PID: 5296 at fs/fuse/file.c:1989 fuse_writepages+0x497/0x5a0 fs/fuse/file.c:2368
> > Modules linked in:
> > CPU: 0 UID: 0 PID: 5296 Comm: kworker/u8:8 Not tainted 6.12.0-rc1-syzkaller-00031-ge32cde8d2bd7 #0
> > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
> > Workqueue: writeback wb_workfn (flush-0:52)
> > RIP: 0010:fuse_write_file_get fs/fuse/file.c:1989 [inline]
> > RIP: 0010:fuse_write_file_get fs/fuse/file.c:1986 [inline]
> > RIP: 0010:fuse_writepages+0x497/0x5a0 fs/fuse/file.c:2368
> > Code: 00 00 00 44 89 f8 5b 5d 41 5c 41 5d 41 5e 41 5f c3 cc cc cc cc e8 79 b6 90 fe 48 8b 7c 24 08 e8 af 6f 27 08 e8 6a b6 90 fe 90 <0f> 0b 90 41 bf fb ff ff ff eb 8b e8 59 b6 90 fe 48 8b 7c 24 18 be
> > RSP: 0018:ffffc900044ff4a8 EFLAGS: 00010293
> > RAX: 0000000000000000 RBX: ffffc900044ff4f8 RCX: 0000000000000000
> > RDX: ffff88802d42da00 RSI: ffffffff82fcd286 RDI: 0000000000000001
> > RBP: ffff88805c994aa0 R08: 0000000000000000 R09: ffffed100b9329d7
> > R10: ffff88805c994ebb R11: 0000000000000003 R12: ffffc900044ff840
> > R13: ffff88805c994880 R14: ffff88805f330000 R15: ffff88805c994d50
> > FS:  0000000000000000(0000) GS:ffff8880b8600000(0000) knlGS:0000000000000000
> > CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > CR2: 0000000020055000 CR3: 000000005df4a000 CR4: 00000000003526f0
> > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> > Call Trace:
> >  <TASK>
> >  do_writepages+0x1a3/0x7f0 mm/page-writeback.c:2683
> >  __writeback_single_inode+0x166/0xfa0 fs/fs-writeback.c:1658
> >  writeback_sb_inodes+0x603/0xfa0 fs/fs-writeback.c:1954
> >  wb_writeback+0x199/0xb50 fs/fs-writeback.c:2134
> >  wb_do_writeback fs/fs-writeback.c:2281 [inline]
> >  wb_workfn+0x294/0xbc0 fs/fs-writeback.c:2321
> >  process_one_work+0x9c5/0x1ba0 kernel/workqueue.c:3229
> >  process_scheduled_works kernel/workqueue.c:3310 [inline]
> >  worker_thread+0x6c8/0xf00 kernel/workqueue.c:3391
> >  kthread+0x2c1/0x3a0 kernel/kthread.c:389
> >  ret_from_fork+0x45/0x80 arch/x86/kernel/process.c:147
> >  ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
> >  </TASK>
> >
>
> #syz dup: [syzbot] [fuse?] WARNING in fuse_write_file_get (2)
>
> This is the same warning reported in
> https://lore.kernel.org/linux-fsdevel/66fbae38.050a0220.6bad9.0051.GAE@google.com/T/#u
>
> The warning is complaining about this WARN_ON here
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/fuse/file.c#n1989.
> I think this warning can get triggered if there's a race between a
> write() and a close() where the page is dirty in the cache after the
> release has happened. Then when writeback (eg fuse_writepages()) is
> triggered, we hit this warning. (this possibility has always existed,
> it was surfaced after this refactoring commit 4046d3adcca4: "move fuse
> file initialization to wpa allocation time" but the actual logic
> hasn't been changed).

Actually, it's not clear how this WARN_ON is getting triggered.

I will wait for syzbot to surface a repro first before taking further action.

>
> I think we can address this by instead calling "data.ff =
> __fuse_write_file_get(fi);" in fuse_writepages(). I'll submit a fix
> for this to Miklos's tree.
>
>
> Thanks,
> Joanne
>
> >
> > ---
> > This report is generated by a bot. It may contain errors.
> > See https://goo.gl/tpsmEJ for more information about syzbot.
> > syzbot engineers can be reached at syzkaller@googlegroups.com.
> >
> > syzbot will keep track of this issue. See:
> > https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
> >
> > If the report is already addressed, let syzbot know by replying with:
> > #syz fix: exact-commit-title
> >
> > If you want to overwrite report's subsystems, reply with:
> > #syz set subsystems: new-subsystem
> > (See the list of subsystem names on the web dashboard)
> >
> > If the report is a duplicate of another one, reply with:
> > #syz dup: exact-subject-of-another-report
> >
> > If you want to undo deduplication, reply with:
> > #syz undup
> >

Re: [syzbot] [fuse?] WARNING in fuse_writepages

[Miklos Szeredi]

On Fri, 4 Oct 2024 at 21:04, Joanne Koong <joannelkoong@gmail.com> wrote:

> > The warning is complaining about this WARN_ON here
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/fuse/file.c#n1989.
> > I think this warning can get triggered if there's a race between a
> > write() and a close() where the page is dirty in the cache after the
> > release has happened. Then when writeback (eg fuse_writepages()) is
> > triggered, we hit this warning. (this possibility has always existed,
> > it was surfaced after this refactoring commit 4046d3adcca4: "move fuse
> > file initialization to wpa allocation time" but the actual logic
> > hasn't been changed).
>
> Actually, it's not clear how this WARN_ON is getting triggered.
>
> I will wait for syzbot to surface a repro first before taking further action.

I think the issue is that fuse_writepages() might be called with no
dirty pages after all writable opens were closed.  The exact mechanism
is unclear, but it's pretty likely that this is the case.

Commit 672c3b7457fc ("fuse: move initialization of fuse_file to
fuse_writepages() instead of in callback") broke this case.

Maybe reverting this is the simplest fix?

Thanks,
Miklos

Re: [syzbot] [fuse?] WARNING in fuse_writepages

[Joanne Koong]

On Fri, Oct 18, 2024 at 3:54 AM Miklos Szeredi <miklos@szeredi.hu> wrote:
>
> On Fri, 4 Oct 2024 at 21:04, Joanne Koong <joannelkoong@gmail.com> wrote:
>
> > > The warning is complaining about this WARN_ON here
> > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/fuse/file.c#n1989.
> > > I think this warning can get triggered if there's a race between a
> > > write() and a close() where the page is dirty in the cache after the
> > > release has happened. Then when writeback (eg fuse_writepages()) is
> > > triggered, we hit this warning. (this possibility has always existed,
> > > it was surfaced after this refactoring commit 4046d3adcca4: "move fuse
> > > file initialization to wpa allocation time" but the actual logic
> > > hasn't been changed).
> >
> > Actually, it's not clear how this WARN_ON is getting triggered.
> >
> > I will wait for syzbot to surface a repro first before taking further action.
>
> I think the issue is that fuse_writepages() might be called with no
> dirty pages after all writable opens were closed.  The exact mechanism
> is unclear, but it's pretty likely that this is the case.
>
> Commit 672c3b7457fc ("fuse: move initialization of fuse_file to
> fuse_writepages() instead of in callback") broke this case.
>
> Maybe reverting this is the simplest fix?

Reverting this sounds good to me.

I guess we don't run into this warning in the original code because if
there are no dirty pages, write_cache_pages() calls into
folio_prepare_writeback() which skips the folio if it's not dirty.

Thanks,
Joanne
>
> Thanks,
> Miklos

Re: [syzbot] [fuse?] WARNING in fuse_writepages

[Miklos Szeredi]

On Fri, 18 Oct 2024 at 20:13, Joanne Koong <joannelkoong@gmail.com> wrote:

> I guess we don't run into this warning in the original code because if
> there are no dirty pages, write_cache_pages() calls into
> folio_prepare_writeback() which skips the folio if it's not dirty.

Added the revert to fuse.git#for-next.  Hopefully the syzbot reports
will go away.

Thanks,
Miklos