* [syzbot] [ext4?] KASAN: slab-use-after-free Read in fsnotify
@ 2024-04-11 8:11 syzbot
2024-04-11 12:13 ` Jan Kara
2024-04-13 1:40 ` Hillf Danton
0 siblings, 2 replies; 26+ messages in thread
From: syzbot @ 2024-04-11 8:11 UTC (permalink / raw)
To: amir73il, jack, linux-ext4, linux-fsdevel, linux-kernel, repnop,
syzkaller-bugs
Hello,
syzbot found the following issue on:
HEAD commit: 6ebf211bb11d Add linux-next specific files for 20240410
git tree: linux-next
console+strace: https://syzkaller.appspot.com/x/log.txt?x=12be955d180000
kernel config: https://syzkaller.appspot.com/x/.config?x=16ca158ef7e08662
dashboard link: https://syzkaller.appspot.com/bug?extid=5e3f9b2a67b45f16d4e6
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=13c91175180000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=1621af9d180000
Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/b050f81f73ed/disk-6ebf211b.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/412c9b9a536e/vmlinux-6ebf211b.xz
kernel image: https://storage.googleapis.com/syzbot-assets/016527216c47/bzImage-6ebf211b.xz
mounted in repro: https://storage.googleapis.com/syzbot-assets/75ad050c9945/mount_0.gz
IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+5e3f9b2a67b45f16d4e6@syzkaller.appspotmail.com
Quota error (device loop0): do_check_range: Getting block 0 out of range 1-5
EXT4-fs error (device loop0): ext4_release_dquot:6905: comm kworker/u8:4: Failed to release dquot type 1
==================================================================
BUG: KASAN: slab-use-after-free in fsnotify+0x2a4/0x1f70 fs/notify/fsnotify.c:539
Read of size 8 at addr ffff88802f1dce80 by task kworker/u8:4/62
CPU: 0 PID: 62 Comm: kworker/u8:4 Not tainted 6.9.0-rc3-next-20240410-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
Workqueue: events_unbound quota_release_workfn
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x241/0x360 lib/dump_stack.c:114
print_address_description mm/kasan/report.c:377 [inline]
print_report+0x169/0x550 mm/kasan/report.c:488
kasan_report+0x143/0x180 mm/kasan/report.c:601
fsnotify+0x2a4/0x1f70 fs/notify/fsnotify.c:539
fsnotify_sb_error include/linux/fsnotify.h:456 [inline]
__ext4_error+0x255/0x3b0 fs/ext4/super.c:843
ext4_release_dquot+0x326/0x450 fs/ext4/super.c:6903
quota_release_workfn+0x39f/0x650 fs/quota/dquot.c:840
process_one_work kernel/workqueue.c:3218 [inline]
process_scheduled_works+0xa2c/0x1830 kernel/workqueue.c:3299
worker_thread+0x86d/0xd70 kernel/workqueue.c:3380
kthread+0x2f0/0x390 kernel/kthread.c:389
ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
</TASK>
Allocated by task 5085:
kasan_save_stack mm/kasan/common.c:47 [inline]
kasan_save_track+0x3f/0x80 mm/kasan/common.c:68
poison_kmalloc_redzone mm/kasan/common.c:370 [inline]
__kasan_kmalloc+0x98/0xb0 mm/kasan/common.c:387
kasan_kmalloc include/linux/kasan.h:211 [inline]
kmalloc_trace_noprof+0x19c/0x2b0 mm/slub.c:4109
kmalloc_noprof include/linux/slab.h:660 [inline]
kzalloc_noprof include/linux/slab.h:775 [inline]
fsnotify_attach_info_to_sb fs/notify/mark.c:600 [inline]
fsnotify_add_mark_list fs/notify/mark.c:692 [inline]
fsnotify_add_mark_locked+0x3b2/0xe60 fs/notify/mark.c:777
fanotify_add_new_mark fs/notify/fanotify/fanotify_user.c:1267 [inline]
fanotify_add_mark+0xbbd/0x1330 fs/notify/fanotify/fanotify_user.c:1334
do_fanotify_mark+0xbcc/0xd90 fs/notify/fanotify/fanotify_user.c:1896
__do_sys_fanotify_mark fs/notify/fanotify/fanotify_user.c:1919 [inline]
__se_sys_fanotify_mark fs/notify/fanotify/fanotify_user.c:1915 [inline]
__x64_sys_fanotify_mark+0xb5/0xd0 fs/notify/fanotify/fanotify_user.c:1915
do_syscall_x64 arch/x86/entry/common.c:52 [inline]
do_syscall_64+0xf5/0x240 arch/x86/entry/common.c:83
entry_SYSCALL_64_after_hwframe+0x77/0x7f
Freed by task 5085:
kasan_save_stack mm/kasan/common.c:47 [inline]
kasan_save_track+0x3f/0x80 mm/kasan/common.c:68
kasan_save_free_info+0x40/0x50 mm/kasan/generic.c:579
poison_slab_object+0xe0/0x150 mm/kasan/common.c:240
__kasan_slab_free+0x37/0x60 mm/kasan/common.c:256
kasan_slab_free include/linux/kasan.h:184 [inline]
slab_free_hook mm/slub.c:2190 [inline]
slab_free mm/slub.c:4393 [inline]
kfree+0x149/0x350 mm/slub.c:4514
fsnotify_sb_delete+0x686/0x6f0 fs/notify/fsnotify.c:106
generic_shutdown_super+0xa5/0x2d0 fs/super.c:632
kill_block_super+0x44/0x90 fs/super.c:1675
ext4_kill_sb+0x68/0xa0 fs/ext4/super.c:7323
deactivate_locked_super+0xc4/0x130 fs/super.c:472
cleanup_mnt+0x426/0x4c0 fs/namespace.c:1267
task_work_run+0x24f/0x310 kernel/task_work.c:180
exit_task_work include/linux/task_work.h:38 [inline]
do_exit+0xa1b/0x27e0 kernel/exit.c:878
do_group_exit+0x207/0x2c0 kernel/exit.c:1027
__do_sys_exit_group kernel/exit.c:1038 [inline]
__se_sys_exit_group kernel/exit.c:1036 [inline]
__x64_sys_exit_group+0x3f/0x40 kernel/exit.c:1036
do_syscall_x64 arch/x86/entry/common.c:52 [inline]
do_syscall_64+0xf5/0x240 arch/x86/entry/common.c:83
entry_SYSCALL_64_after_hwframe+0x77/0x7f
The buggy address belongs to the object at ffff88802f1dce80
which belongs to the cache kmalloc-32 of size 32
The buggy address is located 0 bytes inside of
freed 32-byte region [ffff88802f1dce80, ffff88802f1dcea0)
The buggy address belongs to the physical page:
page: refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x2f1dc
flags: 0xfff80000000000(node=0|zone=1|lastcpupid=0xfff)
page_type: 0xffffefff(slab)
raw: 00fff80000000000 ffff888015041500 ffffea000096b540 dead000000000004
raw: 0000000000000000 0000000080400040 00000001ffffefff 0000000000000000
page dumped because: kasan: bad access detected
page_owner tracks the page as allocated
page last allocated via order 0, migratetype Unmovable, gfp_mask 0x12cc0(GFP_KERNEL|__GFP_NOWARN|__GFP_NORETRY), pid 1, tgid -625457465 (swapper/0), ts 1, free_ts 0
set_page_owner include/linux/page_owner.h:32 [inline]
post_alloc_hook+0x1f3/0x230 mm/page_alloc.c:1468
prep_new_page mm/page_alloc.c:1476 [inline]
get_page_from_freelist+0x2ce2/0x2d90 mm/page_alloc.c:3438
__alloc_pages_noprof+0x256/0x6c0 mm/page_alloc.c:4696
__alloc_pages_node_noprof include/linux/gfp.h:244 [inline]
alloc_pages_node_noprof include/linux/gfp.h:271 [inline]
alloc_slab_page+0x5f/0x120 mm/slub.c:2259
allocate_slab+0x5a/0x2e0 mm/slub.c:2422
new_slab mm/slub.c:2475 [inline]
___slab_alloc+0xcd1/0x14b0 mm/slub.c:3624
__slab_alloc+0x58/0xa0 mm/slub.c:3714
__slab_alloc_node mm/slub.c:3767 [inline]
slab_alloc_node mm/slub.c:3945 [inline]
__do_kmalloc_node mm/slub.c:4077 [inline]
kmalloc_node_track_caller_noprof+0x286/0x440 mm/slub.c:4098
kstrdup+0x3a/0x80 mm/util.c:62
kobject_set_name_vargs+0x61/0x120 lib/kobject.c:274
kobject_add_varg lib/kobject.c:368 [inline]
kobject_init_and_add+0xde/0x190 lib/kobject.c:457
sysfs_slab_add+0x7a/0x290 mm/slub.c:6877
slab_sysfs_init+0x66/0x170 mm/slub.c:6961
do_one_initcall+0x248/0x880 init/main.c:1263
do_initcall_level+0x157/0x210 init/main.c:1325
do_initcalls+0x3f/0x80 init/main.c:1341
page_owner free stack trace missing
Memory state around the buggy address:
ffff88802f1dcd80: 00 00 05 fc fc fc fc fc 00 00 00 00 fc fc fc fc
ffff88802f1dce00: 00 00 00 06 fc fc fc fc fa fb fb fb fc fc fc fc
>ffff88802f1dce80: fa fb fb fb fc fc fc fc fa fb fb fb fc fc fc fc
^
ffff88802f1dcf00: fa fb fb fb fc fc fc fc fa fb fb fb fc fc fc fc
ffff88802f1dcf80: fa fb fb fb fc fc fc fc 00 00 00 fc fc fc fc fc
==================================================================
---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.
syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title
If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.
If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)
If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report
If you want to undo deduplication, reply with:
#syz undup
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [syzbot] [ext4?] KASAN: slab-use-after-free Read in fsnotify
2024-04-11 8:11 syzbot
@ 2024-04-11 12:13 ` Jan Kara
2024-04-11 16:07 ` Amir Goldstein
2024-04-13 1:40 ` Hillf Danton
1 sibling, 1 reply; 26+ messages in thread
From: Jan Kara @ 2024-04-11 12:13 UTC (permalink / raw)
To: syzbot
Cc: amir73il, jack, linux-ext4, linux-fsdevel, linux-kernel, repnop,
syzkaller-bugs
On Thu 11-04-24 01:11:20, syzbot wrote:
> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit: 6ebf211bb11d Add linux-next specific files for 20240410
> git tree: linux-next
> console+strace: https://syzkaller.appspot.com/x/log.txt?x=12be955d180000
> kernel config: https://syzkaller.appspot.com/x/.config?x=16ca158ef7e08662
> dashboard link: https://syzkaller.appspot.com/bug?extid=5e3f9b2a67b45f16d4e6
> compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=13c91175180000
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=1621af9d180000
>
> Downloadable assets:
> disk image: https://storage.googleapis.com/syzbot-assets/b050f81f73ed/disk-6ebf211b.raw.xz
> vmlinux: https://storage.googleapis.com/syzbot-assets/412c9b9a536e/vmlinux-6ebf211b.xz
> kernel image: https://storage.googleapis.com/syzbot-assets/016527216c47/bzImage-6ebf211b.xz
> mounted in repro: https://storage.googleapis.com/syzbot-assets/75ad050c9945/mount_0.gz
>
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+5e3f9b2a67b45f16d4e6@syzkaller.appspotmail.com
>
> Quota error (device loop0): do_check_range: Getting block 0 out of range 1-5
> EXT4-fs error (device loop0): ext4_release_dquot:6905: comm kworker/u8:4: Failed to release dquot type 1
> ==================================================================
> BUG: KASAN: slab-use-after-free in fsnotify+0x2a4/0x1f70 fs/notify/fsnotify.c:539
> Read of size 8 at addr ffff88802f1dce80 by task kworker/u8:4/62
>
> CPU: 0 PID: 62 Comm: kworker/u8:4 Not tainted 6.9.0-rc3-next-20240410-syzkaller #0
> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
> Workqueue: events_unbound quota_release_workfn
> Call Trace:
> <TASK>
> __dump_stack lib/dump_stack.c:88 [inline]
> dump_stack_lvl+0x241/0x360 lib/dump_stack.c:114
> print_address_description mm/kasan/report.c:377 [inline]
> print_report+0x169/0x550 mm/kasan/report.c:488
> kasan_report+0x143/0x180 mm/kasan/report.c:601
> fsnotify+0x2a4/0x1f70 fs/notify/fsnotify.c:539
> fsnotify_sb_error include/linux/fsnotify.h:456 [inline]
> __ext4_error+0x255/0x3b0 fs/ext4/super.c:843
> ext4_release_dquot+0x326/0x450 fs/ext4/super.c:6903
> quota_release_workfn+0x39f/0x650 fs/quota/dquot.c:840
> process_one_work kernel/workqueue.c:3218 [inline]
> process_scheduled_works+0xa2c/0x1830 kernel/workqueue.c:3299
> worker_thread+0x86d/0xd70 kernel/workqueue.c:3380
> kthread+0x2f0/0x390 kernel/kthread.c:389
> ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
> ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
> </TASK>
Amir, I believe this happens on umount when the filesystem calls
fsnotify_sb_error() after calling fsnotify_sb_delete(). In theory these two
calls can even run in parallel and fsnotify() can be holding
fsnotify_sb_info pointer while fsnotify_sb_delete() is freeing it so we
need to figure out some proper synchronization for that...
Honza
> Allocated by task 5085:
> kasan_save_stack mm/kasan/common.c:47 [inline]
> kasan_save_track+0x3f/0x80 mm/kasan/common.c:68
> poison_kmalloc_redzone mm/kasan/common.c:370 [inline]
> __kasan_kmalloc+0x98/0xb0 mm/kasan/common.c:387
> kasan_kmalloc include/linux/kasan.h:211 [inline]
> kmalloc_trace_noprof+0x19c/0x2b0 mm/slub.c:4109
> kmalloc_noprof include/linux/slab.h:660 [inline]
> kzalloc_noprof include/linux/slab.h:775 [inline]
> fsnotify_attach_info_to_sb fs/notify/mark.c:600 [inline]
> fsnotify_add_mark_list fs/notify/mark.c:692 [inline]
> fsnotify_add_mark_locked+0x3b2/0xe60 fs/notify/mark.c:777
> fanotify_add_new_mark fs/notify/fanotify/fanotify_user.c:1267 [inline]
> fanotify_add_mark+0xbbd/0x1330 fs/notify/fanotify/fanotify_user.c:1334
> do_fanotify_mark+0xbcc/0xd90 fs/notify/fanotify/fanotify_user.c:1896
> __do_sys_fanotify_mark fs/notify/fanotify/fanotify_user.c:1919 [inline]
> __se_sys_fanotify_mark fs/notify/fanotify/fanotify_user.c:1915 [inline]
> __x64_sys_fanotify_mark+0xb5/0xd0 fs/notify/fanotify/fanotify_user.c:1915
> do_syscall_x64 arch/x86/entry/common.c:52 [inline]
> do_syscall_64+0xf5/0x240 arch/x86/entry/common.c:83
> entry_SYSCALL_64_after_hwframe+0x77/0x7f
>
> Freed by task 5085:
> kasan_save_stack mm/kasan/common.c:47 [inline]
> kasan_save_track+0x3f/0x80 mm/kasan/common.c:68
> kasan_save_free_info+0x40/0x50 mm/kasan/generic.c:579
> poison_slab_object+0xe0/0x150 mm/kasan/common.c:240
> __kasan_slab_free+0x37/0x60 mm/kasan/common.c:256
> kasan_slab_free include/linux/kasan.h:184 [inline]
> slab_free_hook mm/slub.c:2190 [inline]
> slab_free mm/slub.c:4393 [inline]
> kfree+0x149/0x350 mm/slub.c:4514
> fsnotify_sb_delete+0x686/0x6f0 fs/notify/fsnotify.c:106
> generic_shutdown_super+0xa5/0x2d0 fs/super.c:632
> kill_block_super+0x44/0x90 fs/super.c:1675
> ext4_kill_sb+0x68/0xa0 fs/ext4/super.c:7323
> deactivate_locked_super+0xc4/0x130 fs/super.c:472
> cleanup_mnt+0x426/0x4c0 fs/namespace.c:1267
> task_work_run+0x24f/0x310 kernel/task_work.c:180
> exit_task_work include/linux/task_work.h:38 [inline]
> do_exit+0xa1b/0x27e0 kernel/exit.c:878
> do_group_exit+0x207/0x2c0 kernel/exit.c:1027
> __do_sys_exit_group kernel/exit.c:1038 [inline]
> __se_sys_exit_group kernel/exit.c:1036 [inline]
> __x64_sys_exit_group+0x3f/0x40 kernel/exit.c:1036
> do_syscall_x64 arch/x86/entry/common.c:52 [inline]
> do_syscall_64+0xf5/0x240 arch/x86/entry/common.c:83
> entry_SYSCALL_64_after_hwframe+0x77/0x7f
>
> The buggy address belongs to the object at ffff88802f1dce80
> which belongs to the cache kmalloc-32 of size 32
> The buggy address is located 0 bytes inside of
> freed 32-byte region [ffff88802f1dce80, ffff88802f1dcea0)
>
> The buggy address belongs to the physical page:
> page: refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x2f1dc
> flags: 0xfff80000000000(node=0|zone=1|lastcpupid=0xfff)
> page_type: 0xffffefff(slab)
> raw: 00fff80000000000 ffff888015041500 ffffea000096b540 dead000000000004
> raw: 0000000000000000 0000000080400040 00000001ffffefff 0000000000000000
> page dumped because: kasan: bad access detected
> page_owner tracks the page as allocated
> page last allocated via order 0, migratetype Unmovable, gfp_mask 0x12cc0(GFP_KERNEL|__GFP_NOWARN|__GFP_NORETRY), pid 1, tgid -625457465 (swapper/0), ts 1, free_ts 0
> set_page_owner include/linux/page_owner.h:32 [inline]
> post_alloc_hook+0x1f3/0x230 mm/page_alloc.c:1468
> prep_new_page mm/page_alloc.c:1476 [inline]
> get_page_from_freelist+0x2ce2/0x2d90 mm/page_alloc.c:3438
> __alloc_pages_noprof+0x256/0x6c0 mm/page_alloc.c:4696
> __alloc_pages_node_noprof include/linux/gfp.h:244 [inline]
> alloc_pages_node_noprof include/linux/gfp.h:271 [inline]
> alloc_slab_page+0x5f/0x120 mm/slub.c:2259
> allocate_slab+0x5a/0x2e0 mm/slub.c:2422
> new_slab mm/slub.c:2475 [inline]
> ___slab_alloc+0xcd1/0x14b0 mm/slub.c:3624
> __slab_alloc+0x58/0xa0 mm/slub.c:3714
> __slab_alloc_node mm/slub.c:3767 [inline]
> slab_alloc_node mm/slub.c:3945 [inline]
> __do_kmalloc_node mm/slub.c:4077 [inline]
> kmalloc_node_track_caller_noprof+0x286/0x440 mm/slub.c:4098
> kstrdup+0x3a/0x80 mm/util.c:62
> kobject_set_name_vargs+0x61/0x120 lib/kobject.c:274
> kobject_add_varg lib/kobject.c:368 [inline]
> kobject_init_and_add+0xde/0x190 lib/kobject.c:457
> sysfs_slab_add+0x7a/0x290 mm/slub.c:6877
> slab_sysfs_init+0x66/0x170 mm/slub.c:6961
> do_one_initcall+0x248/0x880 init/main.c:1263
> do_initcall_level+0x157/0x210 init/main.c:1325
> do_initcalls+0x3f/0x80 init/main.c:1341
> page_owner free stack trace missing
>
> Memory state around the buggy address:
> ffff88802f1dcd80: 00 00 05 fc fc fc fc fc 00 00 00 00 fc fc fc fc
> ffff88802f1dce00: 00 00 00 06 fc fc fc fc fa fb fb fb fc fc fc fc
> >ffff88802f1dce80: fa fb fb fb fc fc fc fc fa fb fb fb fc fc fc fc
> ^
> ffff88802f1dcf00: fa fb fb fb fc fc fc fc fa fb fb fb fc fc fc fc
> ffff88802f1dcf80: fa fb fb fb fc fc fc fc 00 00 00 fc fc fc fc fc
> ==================================================================
>
>
> ---
> This report is generated by a bot. It may contain errors.
> See https://goo.gl/tpsmEJ for more information about syzbot.
> syzbot engineers can be reached at syzkaller@googlegroups.com.
>
> syzbot will keep track of this issue. See:
> https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
>
> If the report is already addressed, let syzbot know by replying with:
> #syz fix: exact-commit-title
>
> If you want syzbot to run the reproducer, reply with:
> #syz test: git://repo/address.git branch-or-commit-hash
> If you attach or paste a git patch, syzbot will apply it before testing.
>
> If you want to overwrite report's subsystems, reply with:
> #syz set subsystems: new-subsystem
> (See the list of subsystem names on the web dashboard)
>
> If the report is a duplicate of another one, reply with:
> #syz dup: exact-subject-of-another-report
>
> If you want to undo deduplication, reply with:
> #syz undup
>
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [syzbot] [ext4?] KASAN: slab-use-after-free Read in fsnotify
2024-04-11 12:13 ` Jan Kara
@ 2024-04-11 16:07 ` Amir Goldstein
2024-04-11 19:25 ` Gabriel Krisman Bertazi
2024-04-11 20:06 ` syzbot
0 siblings, 2 replies; 26+ messages in thread
From: Amir Goldstein @ 2024-04-11 16:07 UTC (permalink / raw)
To: Jan Kara
Cc: syzbot, linux-ext4, linux-fsdevel, linux-kernel, repnop,
syzkaller-bugs, Gabriel Krisman Bertazi
On Thu, Apr 11, 2024 at 3:13 PM Jan Kara <jack@suse.cz> wrote:
>
> On Thu 11-04-24 01:11:20, syzbot wrote:
> > Hello,
> >
> > syzbot found the following issue on:
> >
> > HEAD commit: 6ebf211bb11d Add linux-next specific files for 20240410
> > git tree: linux-next
> > console+strace: https://syzkaller.appspot.com/x/log.txt?x=12be955d180000
> > kernel config: https://syzkaller.appspot.com/x/.config?x=16ca158ef7e08662
> > dashboard link: https://syzkaller.appspot.com/bug?extid=5e3f9b2a67b45f16d4e6
> > compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
> > syz repro: https://syzkaller.appspot.com/x/repro.syz?x=13c91175180000
> > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=1621af9d180000
> >
> > Downloadable assets:
> > disk image: https://storage.googleapis.com/syzbot-assets/b050f81f73ed/disk-6ebf211b.raw.xz
> > vmlinux: https://storage.googleapis.com/syzbot-assets/412c9b9a536e/vmlinux-6ebf211b.xz
> > kernel image: https://storage.googleapis.com/syzbot-assets/016527216c47/bzImage-6ebf211b.xz
> > mounted in repro: https://storage.googleapis.com/syzbot-assets/75ad050c9945/mount_0.gz
> >
> > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > Reported-by: syzbot+5e3f9b2a67b45f16d4e6@syzkaller.appspotmail.com
> >
> > Quota error (device loop0): do_check_range: Getting block 0 out of range 1-5
> > EXT4-fs error (device loop0): ext4_release_dquot:6905: comm kworker/u8:4: Failed to release dquot type 1
> > ==================================================================
> > BUG: KASAN: slab-use-after-free in fsnotify+0x2a4/0x1f70 fs/notify/fsnotify.c:539
> > Read of size 8 at addr ffff88802f1dce80 by task kworker/u8:4/62
> >
> > CPU: 0 PID: 62 Comm: kworker/u8:4 Not tainted 6.9.0-rc3-next-20240410-syzkaller #0
> > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
> > Workqueue: events_unbound quota_release_workfn
> > Call Trace:
> > <TASK>
> > __dump_stack lib/dump_stack.c:88 [inline]
> > dump_stack_lvl+0x241/0x360 lib/dump_stack.c:114
> > print_address_description mm/kasan/report.c:377 [inline]
> > print_report+0x169/0x550 mm/kasan/report.c:488
> > kasan_report+0x143/0x180 mm/kasan/report.c:601
> > fsnotify+0x2a4/0x1f70 fs/notify/fsnotify.c:539
> > fsnotify_sb_error include/linux/fsnotify.h:456 [inline]
> > __ext4_error+0x255/0x3b0 fs/ext4/super.c:843
> > ext4_release_dquot+0x326/0x450 fs/ext4/super.c:6903
> > quota_release_workfn+0x39f/0x650 fs/quota/dquot.c:840
> > process_one_work kernel/workqueue.c:3218 [inline]
> > process_scheduled_works+0xa2c/0x1830 kernel/workqueue.c:3299
> > worker_thread+0x86d/0xd70 kernel/workqueue.c:3380
> > kthread+0x2f0/0x390 kernel/kthread.c:389
> > ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
> > ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
> > </TASK>
>
> Amir, I believe this happens on umount when the filesystem calls
> fsnotify_sb_error() after calling fsnotify_sb_delete(). In theory these two
> calls can even run in parallel and fsnotify() can be holding
> fsnotify_sb_info pointer while fsnotify_sb_delete() is freeing it so we
> need to figure out some proper synchronization for that...
Is it really needed to handle any for non SB_ACTIVE sb?
How about something like this?
Is that enough? or more synchronization is needed?
#syz test: https://github.com/amir73il/linux fsnotify-fixes
Thanks,
Amir.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [syzbot] [ext4?] KASAN: slab-use-after-free Read in fsnotify
2024-04-11 16:07 ` Amir Goldstein
@ 2024-04-11 19:25 ` Gabriel Krisman Bertazi
2024-04-11 20:27 ` Khazhy Kumykov
2024-04-11 20:06 ` syzbot
1 sibling, 1 reply; 26+ messages in thread
From: Gabriel Krisman Bertazi @ 2024-04-11 19:25 UTC (permalink / raw)
To: Amir Goldstein
Cc: Jan Kara, syzbot, linux-ext4, linux-fsdevel, linux-kernel, repnop,
syzkaller-bugs, khazhy
Amir Goldstein <amir73il@gmail.com> writes:
> On Thu, Apr 11, 2024 at 3:13 PM Jan Kara <jack@suse.cz> wrote:
>>
>> On Thu 11-04-24 01:11:20, syzbot wrote:
>> > Hello,
>> >
>> > syzbot found the following issue on:
>> >
>> > HEAD commit: 6ebf211bb11d Add linux-next specific files for 20240410
>> > git tree: linux-next
>> > console+strace: https://syzkaller.appspot.com/x/log.txt?x=12be955d180000
>> > kernel config: https://syzkaller.appspot.com/x/.config?x=16ca158ef7e08662
>> > dashboard link: https://syzkaller.appspot.com/bug?extid=5e3f9b2a67b45f16d4e6
>> > compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
>> > syz repro: https://syzkaller.appspot.com/x/repro.syz?x=13c91175180000
>> > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=1621af9d180000
>> >
>> > Downloadable assets:
>> > disk image: https://storage.googleapis.com/syzbot-assets/b050f81f73ed/disk-6ebf211b.raw.xz
>> > vmlinux: https://storage.googleapis.com/syzbot-assets/412c9b9a536e/vmlinux-6ebf211b.xz
>> > kernel image: https://storage.googleapis.com/syzbot-assets/016527216c47/bzImage-6ebf211b.xz
>> > mounted in repro: https://storage.googleapis.com/syzbot-assets/75ad050c9945/mount_0.gz
>> >
>> > IMPORTANT: if you fix the issue, please add the following tag to the commit:
>> > Reported-by: syzbot+5e3f9b2a67b45f16d4e6@syzkaller.appspotmail.com
>> >
>> > Quota error (device loop0): do_check_range: Getting block 0 out of range 1-5
>> > EXT4-fs error (device loop0): ext4_release_dquot:6905: comm kworker/u8:4: Failed to release dquot type 1
>> > ==================================================================
>> > BUG: KASAN: slab-use-after-free in fsnotify+0x2a4/0x1f70 fs/notify/fsnotify.c:539
>> > Read of size 8 at addr ffff88802f1dce80 by task kworker/u8:4/62
>> >
>> > CPU: 0 PID: 62 Comm: kworker/u8:4 Not tainted 6.9.0-rc3-next-20240410-syzkaller #0
>> > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
>> > Workqueue: events_unbound quota_release_workfn
>> > Call Trace:
>> > <TASK>
>> > __dump_stack lib/dump_stack.c:88 [inline]
>> > dump_stack_lvl+0x241/0x360 lib/dump_stack.c:114
>> > print_address_description mm/kasan/report.c:377 [inline]
>> > print_report+0x169/0x550 mm/kasan/report.c:488
>> > kasan_report+0x143/0x180 mm/kasan/report.c:601
>> > fsnotify+0x2a4/0x1f70 fs/notify/fsnotify.c:539
>> > fsnotify_sb_error include/linux/fsnotify.h:456 [inline]
>> > __ext4_error+0x255/0x3b0 fs/ext4/super.c:843
>> > ext4_release_dquot+0x326/0x450 fs/ext4/super.c:6903
>> > quota_release_workfn+0x39f/0x650 fs/quota/dquot.c:840
>> > process_one_work kernel/workqueue.c:3218 [inline]
>> > process_scheduled_works+0xa2c/0x1830 kernel/workqueue.c:3299
>> > worker_thread+0x86d/0xd70 kernel/workqueue.c:3380
>> > kthread+0x2f0/0x390 kernel/kthread.c:389
>> > ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
>> > ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
>> > </TASK>
>>
>> Amir, I believe this happens on umount when the filesystem calls
>> fsnotify_sb_error() after calling fsnotify_sb_delete(). In theory these two
>> calls can even run in parallel and fsnotify() can be holding
>> fsnotify_sb_info pointer while fsnotify_sb_delete() is freeing it so we
>> need to figure out some proper synchronization for that...
>
> Is it really needed to handle any for non SB_ACTIVE sb?
I think it should be fine to exclude volumes being teared down. Cc'ing
Khazhy, who sponsored this work at the time and owned the use-case.
--
Gabriel Krisman Bertazi
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [syzbot] [ext4?] KASAN: slab-use-after-free Read in fsnotify
2024-04-11 16:07 ` Amir Goldstein
2024-04-11 19:25 ` Gabriel Krisman Bertazi
@ 2024-04-11 20:06 ` syzbot
2024-04-12 7:52 ` Amir Goldstein
1 sibling, 1 reply; 26+ messages in thread
From: syzbot @ 2024-04-11 20:06 UTC (permalink / raw)
To: amir73il, jack, krisman, linux-ext4, linux-fsdevel, linux-kernel,
repnop, syzkaller-bugs
Hello,
syzbot tried to test the proposed patch but the build/boot failed:
viperboard
[ 7.499011][ T1] usbcore: registered new interface driver dln2
[ 7.500804][ T1] usbcore: registered new interface driver pn533_usb
[ 7.507181][ T1] nfcsim 0.2 initialized
[ 7.508964][ T1] usbcore: registered new interface driver port100
[ 7.511844][ T1] usbcore: registered new interface driver nfcmrvl
[ 7.519814][ T1] Loading iSCSI transport class v2.0-870.
[ 7.539126][ T1] virtio_scsi virtio0: 1/0/0 default/read/poll queues
[ 7.550224][ T1] ------------[ cut here ]------------
[ 7.551264][ T1] refcount_t: decrement hit 0; leaking memory.
[ 7.552627][ T1] WARNING: CPU: 0 PID: 1 at lib/refcount.c:31 refcount_warn_saturate+0xfa/0x1d0
[ 7.554218][ T1] Modules linked in:
[ 7.554791][ T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.9.0-rc3-syzkaller-00014-geb06a4b6cca5 #0
[ 7.556609][ T1] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
[ 7.558128][ T1] RIP: 0010:refcount_warn_saturate+0xfa/0x1d0
[ 7.559937][ T1] Code: b2 00 00 00 e8 87 70 e7 fc 5b 5d c3 cc cc cc cc e8 7b 70 e7 fc c6 05 0c 5d e5 0a 01 90 48 c7 c7 40 4b 1f 8c e8 17 ee a9 fc 90 <0f> 0b 90 90 eb d9 e8 5b 70 e7 fc c6 05 e9 5c e5 0a 01 90 48 c7 c7
[ 7.563097][ T1] RSP: 0000:ffffc90000066e18 EFLAGS: 00010246
[ 7.564240][ T1] RAX: 6f40bd285f2a6000 RBX: ffff888147ed00fc RCX: ffff8880166d0000
[ 7.565743][ T1] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 7.567236][ T1] RBP: 0000000000000004 R08: ffffffff815800a2 R09: fffffbfff1c39af8
[ 7.568531][ T1] R10: dffffc0000000000 R11: fffffbfff1c39af8 R12: ffffea000503edc0
[ 7.570021][ T1] R13: ffffea000503edc8 R14: 1ffffd4000a07db9 R15: 0000000000000000
[ 7.571764][ T1] FS: 0000000000000000(0000) GS:ffff8880b9400000(0000) knlGS:0000000000000000
[ 7.573270][ T1] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 7.574232][ T1] CR2: ffff88823ffff000 CR3: 000000000e134000 CR4: 00000000003506f0
[ 7.575566][ T1] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 7.576737][ T1] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 7.578004][ T1] Call Trace:
[ 7.578712][ T1] <TASK>
[ 7.579189][ T1] ? __warn+0x163/0x4e0
[ 7.580052][ T1] ? refcount_warn_saturate+0xfa/0x1d0
[ 7.580896][ T1] ? report_bug+0x2b3/0x500
[ 7.581593][ T1] ? refcount_warn_saturate+0xfa/0x1d0
[ 7.582383][ T1] ? handle_bug+0x3e/0x70
[ 7.583169][ T1] ? exc_invalid_op+0x1a/0x50
[ 7.584335][ T1] ? asm_exc_invalid_op+0x1a/0x20
[ 7.585285][ T1] ? __warn_printk+0x292/0x360
[ 7.586058][ T1] ? refcount_warn_saturate+0xfa/0x1d0
[ 7.586882][ T1] ? refcount_warn_saturate+0xf9/0x1d0
[ 7.587666][ T1] __free_pages_ok+0xc60/0xd90
[ 7.588339][ T1] make_alloc_exact+0xa3/0xf0
[ 7.589220][ T1] vring_alloc_queue_split+0x20a/0x600
[ 7.590378][ T1] ? __pfx_vring_alloc_queue_split+0x10/0x10
[ 7.591429][ T1] ? vp_find_vqs+0x4c/0x4e0
[ 7.592174][ T1] ? virtscsi_probe+0x3ea/0xf60
[ 7.592853][ T1] ? virtio_dev_probe+0x991/0xaf0
[ 7.593892][ T1] ? really_probe+0x2b8/0xad0
[ 7.594581][ T1] ? driver_probe_device+0x50/0x430
[ 7.595325][ T1] vring_create_virtqueue_split+0xc6/0x310
[ 7.596200][ T1] ? ret_from_fork+0x4b/0x80
[ 7.597705][ T1] ? __pfx_vring_create_virtqueue_split+0x10/0x10
[ 7.599051][ T1] vring_create_virtqueue+0xca/0x110
[ 7.599905][ T1] ? __pfx_vp_notify+0x10/0x10
[ 7.600626][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 7.601770][ T1] setup_vq+0xe9/0x2d0
[ 7.602429][ T1] ? __pfx_vp_notify+0x10/0x10
[ 7.603153][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 7.604003][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 7.604758][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 7.605623][ T1] vp_setup_vq+0xbf/0x330
[ 7.606388][ T1] ? __pfx_vp_config_changed+0x10/0x10
[ 7.607239][ T1] ? ioread16+0x2f/0x90
[ 7.608129][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 7.609047][ T1] vp_find_vqs_msix+0x8b2/0xc80
[ 7.609880][ T1] vp_find_vqs+0x4c/0x4e0
[ 7.610578][ T1] virtscsi_init+0x8db/0xd00
[ 7.611247][ T1] ? __pfx_virtscsi_init+0x10/0x10
[ 7.612058][ T1] ? __pfx_default_calc_sets+0x10/0x10
[ 7.612860][ T1] ? scsi_host_alloc+0xa57/0xea0
[ 7.613671][ T1] ? vp_get+0xfd/0x140
[ 7.614243][ T1] virtscsi_probe+0x3ea/0xf60
[ 7.614969][ T1] ? __pfx_virtscsi_probe+0x10/0x10
[ 7.615847][ T1] ? kernfs_add_one+0x156/0x8b0
[ 7.616678][ T1] ? virtio_no_restricted_mem_acc+0x9/0x10
[ 7.617627][ T1] ? virtio_features_ok+0x10c/0x270
[ 7.618392][ T1] virtio_dev_probe+0x991/0xaf0
[ 7.619262][ T1] ? __pfx_virtio_dev_probe+0x10/0x10
[ 7.620216][ T1] really_probe+0x2b8/0xad0
[ 7.621124][ T1] __driver_probe_device+0x1a2/0x390
[ 7.621980][ T1] driver_probe_device+0x50/0x430
[ 7.622710][ T1] __driver_attach+0x45f/0x710
[ 7.623413][ T1] ? __pfx___driver_attach+0x10/0x10
[ 7.624299][ T1] bus_for_each_dev+0x239/0x2b0
[ 7.625118][ T1] ? __pfx___driver_attach+0x10/0x10
[ 7.625993][ T1] ? __pfx_bus_for_each_dev+0x10/0x10
[ 7.626858][ T1] ? do_raw_spin_unlock+0x13c/0x8b0
[ 7.627837][ T1] bus_add_driver+0x347/0x620
[ 7.628735][ T1] driver_register+0x23a/0x320
[ 7.629982][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
[ 7.631006][ T1] virtio_scsi_init+0x65/0xe0
[ 7.631802][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
[ 7.632612][ T1] do_one_initcall+0x248/0x880
[ 7.633404][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
[ 7.634540][ T1] ? __pfx_do_one_initcall+0x10/0x10
[ 7.635786][ T1] ? lockdep_hardirqs_on_prepare+0x43d/0x780
[ 7.636889][ T1] ? __pfx_parse_args+0x10/0x10
[ 7.637652][ T1] ? do_initcalls+0x1c/0x80
[ 7.638456][ T1] ? rcu_is_watching+0x15/0xb0
[ 7.639227][ T1] do_initcall_level+0x157/0x210
[ 7.640192][ T1] do_initcalls+0x3f/0x80
[ 7.640818][ T1] kernel_init_freeable+0x435/0x5d0
[ 7.641593][ T1] ? __pfx_kernel_init_freeable+0x10/0x10
[ 7.642546][ T1] ? __pfx_lockdep_hardirqs_on_prepare+0x10/0x10
[ 7.643506][ T1] ? __pfx_kernel_init+0x10/0x10
[ 7.644295][ T1] ? __pfx_kernel_init+0x10/0x10
[ 7.645313][ T1] ? __pfx_kernel_init+0x10/0x10
[ 7.646036][ T1] kernel_init+0x1d/0x2b0
[ 7.646660][ T1] ret_from_fork+0x4b/0x80
[ 7.647368][ T1] ? __pfx_kernel_init+0x10/0x10
[ 7.648542][ T1] ret_from_fork_asm+0x1a/0x30
[ 7.649812][ T1] </TASK>
[ 7.650346][ T1] Kernel panic - not syncing: kernel: panic_on_warn set ...
[ 7.651620][ T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.9.0-rc3-syzkaller-00014-geb06a4b6cca5 #0
[ 7.653389][ T1] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
[ 7.655321][ T1] Call Trace:
[ 7.655801][ T1] <TASK>
[ 7.656252][ T1] dump_stack_lvl+0x241/0x360
[ 7.657006][ T1] ? __pfx_dump_stack_lvl+0x10/0x10
[ 7.657825][ T1] ? __pfx__printk+0x10/0x10
[ 7.658542][ T1] ? _printk+0xd5/0x120
[ 7.659343][ T1] ? vscnprintf+0x5d/0x90
[ 7.659705][ T1] panic+0x349/0x860
[ 7.659705][ T1] ? __warn+0x172/0x4e0
[ 7.659705][ T1] ? __pfx_panic+0x10/0x10
[ 7.659705][ T1] ? show_trace_log_lvl+0x4e6/0x520
[ 7.659705][ T1] ? ret_from_fork_asm+0x1a/0x30
[ 7.659705][ T1] __warn+0x346/0x4e0
[ 7.659705][ T1] ? refcount_warn_saturate+0xfa/0x1d0
[ 7.659705][ T1] report_bug+0x2b3/0x500
[ 7.659705][ T1] ? refcount_warn_saturate+0xfa/0x1d0
[ 7.659705][ T1] handle_bug+0x3e/0x70
[ 7.659705][ T1] exc_invalid_op+0x1a/0x50
[ 7.659705][ T1] asm_exc_invalid_op+0x1a/0x20
[ 7.659705][ T1] RIP: 0010:refcount_warn_saturate+0xfa/0x1d0
[ 7.659705][ T1] Code: b2 00 00 00 e8 87 70 e7 fc 5b 5d c3 cc cc cc cc e8 7b 70 e7 fc c6 05 0c 5d e5 0a 01 90 48 c7 c7 40 4b 1f 8c e8 17 ee a9 fc 90 <0f> 0b 90 90 eb d9 e8 5b 70 e7 fc c6 05 e9 5c e5 0a 01 90 48 c7 c7
[ 7.659705][ T1] RSP: 0000:ffffc90000066e18 EFLAGS: 00010246
[ 7.659705][ T1] RAX: 6f40bd285f2a6000 RBX: ffff888147ed00fc RCX: ffff8880166d0000
[ 7.659705][ T1] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 7.659705][ T1] RBP: 0000000000000004 R08: ffffffff815800a2 R09: fffffbfff1c39af8
[ 7.659705][ T1] R10: dffffc0000000000 R11: fffffbfff1c39af8 R12: ffffea000503edc0
[ 7.659705][ T1] R13: ffffea000503edc8 R14: 1ffffd4000a07db9 R15: 0000000000000000
[ 7.659705][ T1] ? __warn_printk+0x292/0x360
[ 7.659705][ T1] ? refcount_warn_saturate+0xf9/0x1d0
[ 7.659705][ T1] __free_pages_ok+0xc60/0xd90
[ 7.659705][ T1] make_alloc_exact+0xa3/0xf0
[ 7.659705][ T1] vring_alloc_queue_split+0x20a/0x600
[ 7.659705][ T1] ? __pfx_vring_alloc_queue_split+0x10/0x10
[ 7.659705][ T1] ? vp_find_vqs+0x4c/0x4e0
[ 7.659705][ T1] ? virtscsi_probe+0x3ea/0xf60
[ 7.659705][ T1] ? virtio_dev_probe+0x991/0xaf0
[ 7.659705][ T1] ? really_probe+0x2b8/0xad0
[ 7.659705][ T1] ? driver_probe_device+0x50/0x430
[ 7.659705][ T1] vring_create_virtqueue_split+0xc6/0x310
[ 7.659705][ T1] ? ret_from_fork+0x4b/0x80
[ 7.659705][ T1] ? __pfx_vring_create_virtqueue_split+0x10/0x10
[ 7.659705][ T1] vring_create_virtqueue+0xca/0x110
[ 7.659705][ T1] ? __pfx_vp_notify+0x10/0x10
[ 7.659705][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 7.659705][ T1] setup_vq+0xe9/0x2d0
[ 7.659705][ T1] ? __pfx_vp_notify+0x10/0x10
[ 7.659705][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 7.659705][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 7.659705][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 7.659705][ T1] vp_setup_vq+0xbf/0x330
[ 7.659705][ T1] ? __pfx_vp_config_changed+0x10/0x10
[ 7.659705][ T1] ? ioread16+0x2f/0x90
[ 7.659705][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 7.709861][ T1] vp_find_vqs_msix+0x8b2/0xc80
[ 7.709861][ T1] vp_find_vqs+0x4c/0x4e0
[ 7.709861][ T1] virtscsi_init+0x8db/0xd00
[ 7.709861][ T1] ? __pfx_virtscsi_init+0x10/0x10
[ 7.709861][ T1] ? __pfx_default_calc_sets+0x10/0x10
[ 7.709861][ T1] ? scsi_host_alloc+0xa57/0xea0
[ 7.709861][ T1] ? vp_get+0xfd/0x140
[ 7.709861][ T1] virtscsi_probe+0x3ea/0xf60
[ 7.709861][ T1] ? __pfx_virtscsi_probe+0x10/0x10
[ 7.709861][ T1] ? kernfs_add_one+0x156/0x8b0
[ 7.709861][ T1] ? virtio_no_restricted_mem_acc+0x9/0x10
[ 7.709861][ T1] ? virtio_features_ok+0x10c/0x270
[ 7.709861][ T1] virtio_dev_probe+0x991/0xaf0
[ 7.709861][ T1] ? __pfx_virtio_dev_probe+0x10/0x10
[ 7.709861][ T1] really_probe+0x2b8/0xad0
[ 7.709861][ T1] __driver_probe_device+0x1a2/0x390
[ 7.709861][ T1] driver_probe_device+0x50/0x430
[ 7.709861][ T1] __driver_attach+0x45f/0x710
[ 7.709861][ T1] ? __pfx___driver_attach+0x10/0x10
[ 7.709861][ T1] bus_for_each_dev+0x239/0x2b0
[ 7.709861][ T1] ? __pfx___driver_attach+0x10/0x10
[ 7.709861][ T1] ? __pfx_bus_for_each_dev+0x10/0x10
[ 7.709861][ T1] ? do_raw_spin_unlock+0x13c/0x8b0
[ 7.709861][ T1] bus_add_driver+0x347/0x620
[ 7.709861][ T1] driver_register+0x23a/0x320
[ 7.709861][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
[ 7.709861][ T1] virtio_scsi_init+0x65/0xe0
[ 7.709861][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
[ 7.709861][ T1] do_one_initcall+0x248/0x880
[ 7.709861][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
[ 7.709861][ T1] ? __pfx_do_one_initcall+0x10/0x10
[ 7.709861][ T1] ? lockdep_hardirqs_on_prepare+0x43d/0x780
[ 7.709861][ T1] ? __pfx_parse_args+0x10/0x10
[ 7.709861][ T1] ? do_initcalls+0x1c/0x80
[ 7.709861][ T1] ? rcu_is_watching+0x15/0xb0
[ 7.709861][ T1] do_initcall_level+0x157/0x210
[ 7.709861][ T1] do_initcalls+0x3f/0x80
[ 7.709861][ T1] kernel_init_freeable+0x435/0x5d0
[ 7.709861][ T1] ? __pfx_kernel_init_freeable+0x10/0x10
[ 7.709861][ T1] ? __pfx_lockdep_hardirqs_on_prepare+0x10/0x10
[ 7.709861][ T1] ? __pfx_kernel_init+0x10/0x10
[ 7.709861][ T1] ? __pfx_kernel_init+0x10/0x10
[ 7.709861][ T1] ? __pfx_kernel_init+0x10/0x10
[ 7.709861][ T1] kernel_init+0x1d/0x2b0
[ 7.709861][ T1] ret_from_fork+0x4b/0x80
[ 7.709861][ T1] ? __pfx_kernel_init+0x10/0x10
[ 7.709861][ T1] ret_from_fork_asm+0x1a/0x30
[ 7.709861][ T1] </TASK>
[ 7.709861][ T1] Kernel Offset: disabled
[ 7.709861][ T1] Rebooting in 86400 seconds..
syzkaller build log:
go env (err=<nil>)
GO111MODULE='auto'
GOARCH='amd64'
GOBIN=''
GOCACHE='/syzkaller/.cache/go-build'
GOENV='/syzkaller/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/syzkaller/jobs-2/linux/gopath/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/syzkaller/jobs-2/linux/gopath'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.4'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/syzkaller/jobs-2/linux/gopath/src/github.com/google/syzkaller/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1437292946=/tmp/go-build -gno-record-gcc-switches'
git status (err=<nil>)
HEAD detached at 56086b24b
nothing to commit, working tree clean
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
Makefile:31: run command via tools/syz-env for best compatibility, see:
Makefile:32: https://github.com/google/syzkaller/blob/master/docs/contributing.md#using-syz-env
go list -f '{{.Stale}}' ./sys/syz-sysgen | grep -q false || go install ./sys/syz-sysgen
make .descriptions
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
Makefile:31: run command via tools/syz-env for best compatibility, see:
Makefile:32: https://github.com/google/syzkaller/blob/master/docs/contributing.md#using-syz-env
bin/syz-sysgen
touch .descriptions
GOOS=linux GOARCH=amd64 go build "-ldflags=-s -w -X github.com/google/syzkaller/prog.GitRevision=56086b24bdfd822d3b227edb3064db443cd8c971 -X 'github.com/google/syzkaller/prog.gitRevisionDate=20240409-083312'" "-tags=syz_target syz_os_linux syz_arch_amd64 " -o ./bin/linux_amd64/syz-fuzzer github.com/google/syzkaller/syz-fuzzer
GOOS=linux GOARCH=amd64 go build "-ldflags=-s -w -X github.com/google/syzkaller/prog.GitRevision=56086b24bdfd822d3b227edb3064db443cd8c971 -X 'github.com/google/syzkaller/prog.gitRevisionDate=20240409-083312'" "-tags=syz_target syz_os_linux syz_arch_amd64 " -o ./bin/linux_amd64/syz-execprog github.com/google/syzkaller/tools/syz-execprog
GOOS=linux GOARCH=amd64 go build "-ldflags=-s -w -X github.com/google/syzkaller/prog.GitRevision=56086b24bdfd822d3b227edb3064db443cd8c971 -X 'github.com/google/syzkaller/prog.gitRevisionDate=20240409-083312'" "-tags=syz_target syz_os_linux syz_arch_amd64 " -o ./bin/linux_amd64/syz-stress github.com/google/syzkaller/tools/syz-stress
mkdir -p ./bin/linux_amd64
gcc -o ./bin/linux_amd64/syz-executor executor/executor.cc \
-m64 -O2 -pthread -Wall -Werror -Wparentheses -Wunused-const-variable -Wframe-larger-than=16384 -Wno-stringop-overflow -Wno-array-bounds -Wno-format-overflow -Wno-unused-but-set-variable -Wno-unused-command-line-argument -static-pie -fpermissive -w -DGOOS_linux=1 -DGOARCH_amd64=1 \
-DHOSTGOOS_linux=1 -DGIT_REVISION=\"56086b24bdfd822d3b227edb3064db443cd8c971\"
Error text is too large and was truncated, full error text is at:
https://syzkaller.appspot.com/x/error.txt?x=1523635d180000
Tested on:
commit: eb06a4b6 fsnotify: do not handle events on a shutting ..
git tree: https://github.com/amir73il/linux fsnotify-fixes
kernel config: https://syzkaller.appspot.com/x/.config?x=3335b6cc973feecf
dashboard link: https://syzkaller.appspot.com/bug?extid=5e3f9b2a67b45f16d4e6
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
Note: no patches were applied.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [syzbot] [ext4?] KASAN: slab-use-after-free Read in fsnotify
2024-04-11 19:25 ` Gabriel Krisman Bertazi
@ 2024-04-11 20:27 ` Khazhy Kumykov
2024-04-12 10:24 ` Amir Goldstein
0 siblings, 1 reply; 26+ messages in thread
From: Khazhy Kumykov @ 2024-04-11 20:27 UTC (permalink / raw)
To: Gabriel Krisman Bertazi
Cc: Amir Goldstein, Jan Kara, syzbot, linux-ext4, linux-fsdevel,
linux-kernel, repnop, syzkaller-bugs
On Thu, Apr 11, 2024 at 12:25 PM Gabriel Krisman Bertazi
<krisman@suse.de> wrote:
>
> Amir Goldstein <amir73il@gmail.com> writes:
>
> > On Thu, Apr 11, 2024 at 3:13 PM Jan Kara <jack@suse.cz> wrote:
> >>
> >> On Thu 11-04-24 01:11:20, syzbot wrote:
> >> > Hello,
> >> >
> >> > syzbot found the following issue on:
> >> >
> >> > HEAD commit: 6ebf211bb11d Add linux-next specific files for 20240410
> >> > git tree: linux-next
> >> > console+strace: https://syzkaller.appspot.com/x/log.txt?x=12be955d180000
> >> > kernel config: https://syzkaller.appspot.com/x/.config?x=16ca158ef7e08662
> >> > dashboard link: https://syzkaller.appspot.com/bug?extid=5e3f9b2a67b45f16d4e6
> >> > compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
> >> > syz repro: https://syzkaller.appspot.com/x/repro.syz?x=13c91175180000
> >> > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=1621af9d180000
> >> >
> >> > Downloadable assets:
> >> > disk image: https://storage.googleapis.com/syzbot-assets/b050f81f73ed/disk-6ebf211b.raw.xz
> >> > vmlinux: https://storage.googleapis.com/syzbot-assets/412c9b9a536e/vmlinux-6ebf211b.xz
> >> > kernel image: https://storage.googleapis.com/syzbot-assets/016527216c47/bzImage-6ebf211b.xz
> >> > mounted in repro: https://storage.googleapis.com/syzbot-assets/75ad050c9945/mount_0.gz
> >> >
> >> > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> >> > Reported-by: syzbot+5e3f9b2a67b45f16d4e6@syzkaller.appspotmail.com
> >> >
> >> > Quota error (device loop0): do_check_range: Getting block 0 out of range 1-5
> >> > EXT4-fs error (device loop0): ext4_release_dquot:6905: comm kworker/u8:4: Failed to release dquot type 1
> >> > ==================================================================
> >> > BUG: KASAN: slab-use-after-free in fsnotify+0x2a4/0x1f70 fs/notify/fsnotify.c:539
> >> > Read of size 8 at addr ffff88802f1dce80 by task kworker/u8:4/62
> >> >
> >> > CPU: 0 PID: 62 Comm: kworker/u8:4 Not tainted 6.9.0-rc3-next-20240410-syzkaller #0
> >> > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
> >> > Workqueue: events_unbound quota_release_workfn
> >> > Call Trace:
> >> > <TASK>
> >> > __dump_stack lib/dump_stack.c:88 [inline]
> >> > dump_stack_lvl+0x241/0x360 lib/dump_stack.c:114
> >> > print_address_description mm/kasan/report.c:377 [inline]
> >> > print_report+0x169/0x550 mm/kasan/report.c:488
> >> > kasan_report+0x143/0x180 mm/kasan/report.c:601
> >> > fsnotify+0x2a4/0x1f70 fs/notify/fsnotify.c:539
> >> > fsnotify_sb_error include/linux/fsnotify.h:456 [inline]
> >> > __ext4_error+0x255/0x3b0 fs/ext4/super.c:843
> >> > ext4_release_dquot+0x326/0x450 fs/ext4/super.c:6903
> >> > quota_release_workfn+0x39f/0x650 fs/quota/dquot.c:840
> >> > process_one_work kernel/workqueue.c:3218 [inline]
> >> > process_scheduled_works+0xa2c/0x1830 kernel/workqueue.c:3299
> >> > worker_thread+0x86d/0xd70 kernel/workqueue.c:3380
> >> > kthread+0x2f0/0x390 kernel/kthread.c:389
> >> > ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
> >> > ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
> >> > </TASK>
> >>
> >> Amir, I believe this happens on umount when the filesystem calls
> >> fsnotify_sb_error() after calling fsnotify_sb_delete().
Hmm, so we're releasing dquots after already shutting down the
filesystem? Is that expected? This "Failed to release dquot type"
error message only appears if we have an open handle from
ext4_journal_start (although this filesystem was mounted without a
journal, so we hit ext4_get_nojournal()...)
> In theory these two
> >> calls can even run in parallel and fsnotify() can be holding
> >> fsnotify_sb_info pointer while fsnotify_sb_delete() is freeing it so we
> >> need to figure out some proper synchronization for that...
> >
> > Is it really needed to handle any for non SB_ACTIVE sb?
>
> I think it should be fine to exclude volumes being teared down. Cc'ing
> Khazhy, who sponsored this work at the time and owned the use-case.
In terms of real-world use case... not sure there is one - you'll
notice the errors when you fsck/mount again later on, and any users
who care about notifs should be gone by the time we're unmounting. But
it seems weird to me that we can get write errors shutting everything
down.
>
> --
> Gabriel Krisman Bertazi
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [syzbot] [ext4?] KASAN: slab-use-after-free Read in fsnotify
2024-04-11 20:06 ` syzbot
@ 2024-04-12 7:52 ` Amir Goldstein
2024-04-12 14:30 ` syzbot
0 siblings, 1 reply; 26+ messages in thread
From: Amir Goldstein @ 2024-04-12 7:52 UTC (permalink / raw)
To: syzbot
Cc: jack, krisman, linux-ext4, linux-fsdevel, linux-kernel, repnop,
syzkaller-bugs
On Thu, Apr 11, 2024 at 11:06 PM syzbot
<syzbot+5e3f9b2a67b45f16d4e6@syzkaller.appspotmail.com> wrote:
>
> Hello,
>
> syzbot tried to test the proposed patch but the build/boot failed:
>
> viperboard
> [ 7.499011][ T1] usbcore: registered new interface driver dln2
> [ 7.500804][ T1] usbcore: registered new interface driver pn533_usb
> [ 7.507181][ T1] nfcsim 0.2 initialized
> [ 7.508964][ T1] usbcore: registered new interface driver port100
> [ 7.511844][ T1] usbcore: registered new interface driver nfcmrvl
> [ 7.519814][ T1] Loading iSCSI transport class v2.0-870.
> [ 7.539126][ T1] virtio_scsi virtio0: 1/0/0 default/read/poll queues
> [ 7.550224][ T1] ------------[ cut here ]------------
> [ 7.551264][ T1] refcount_t: decrement hit 0; leaking memory.
> [ 7.552627][ T1] WARNING: CPU: 0 PID: 1 at lib/refcount.c:31 refcount_warn_saturate+0xfa/0x1d0
> [ 7.554218][ T1] Modules linked in:
> [ 7.554791][ T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.9.0-rc3-syzkaller-00014-geb06a4b6cca5 #0
> [ 7.556609][ T1] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
> [ 7.558128][ T1] RIP: 0010:refcount_warn_saturate+0xfa/0x1d0
> [ 7.559937][ T1] Code: b2 00 00 00 e8 87 70 e7 fc 5b 5d c3 cc cc cc cc e8 7b 70 e7 fc c6 05 0c 5d e5 0a 01 90 48 c7 c7 40 4b 1f 8c e8 17 ee a9 fc 90 <0f> 0b 90 90 eb d9 e8 5b 70 e7 fc c6 05 e9 5c e5 0a 01 90 48 c7 c7
> [ 7.563097][ T1] RSP: 0000:ffffc90000066e18 EFLAGS: 00010246
> [ 7.564240][ T1] RAX: 6f40bd285f2a6000 RBX: ffff888147ed00fc RCX: ffff8880166d0000
> [ 7.565743][ T1] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
> [ 7.567236][ T1] RBP: 0000000000000004 R08: ffffffff815800a2 R09: fffffbfff1c39af8
> [ 7.568531][ T1] R10: dffffc0000000000 R11: fffffbfff1c39af8 R12: ffffea000503edc0
> [ 7.570021][ T1] R13: ffffea000503edc8 R14: 1ffffd4000a07db9 R15: 0000000000000000
> [ 7.571764][ T1] FS: 0000000000000000(0000) GS:ffff8880b9400000(0000) knlGS:0000000000000000
> [ 7.573270][ T1] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 7.574232][ T1] CR2: ffff88823ffff000 CR3: 000000000e134000 CR4: 00000000003506f0
> [ 7.575566][ T1] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 7.576737][ T1] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> [ 7.578004][ T1] Call Trace:
> [ 7.578712][ T1] <TASK>
> [ 7.579189][ T1] ? __warn+0x163/0x4e0
> [ 7.580052][ T1] ? refcount_warn_saturate+0xfa/0x1d0
> [ 7.580896][ T1] ? report_bug+0x2b3/0x500
> [ 7.581593][ T1] ? refcount_warn_saturate+0xfa/0x1d0
> [ 7.582383][ T1] ? handle_bug+0x3e/0x70
> [ 7.583169][ T1] ? exc_invalid_op+0x1a/0x50
> [ 7.584335][ T1] ? asm_exc_invalid_op+0x1a/0x20
> [ 7.585285][ T1] ? __warn_printk+0x292/0x360
> [ 7.586058][ T1] ? refcount_warn_saturate+0xfa/0x1d0
> [ 7.586882][ T1] ? refcount_warn_saturate+0xf9/0x1d0
> [ 7.587666][ T1] __free_pages_ok+0xc60/0xd90
> [ 7.588339][ T1] make_alloc_exact+0xa3/0xf0
> [ 7.589220][ T1] vring_alloc_queue_split+0x20a/0x600
> [ 7.590378][ T1] ? __pfx_vring_alloc_queue_split+0x10/0x10
> [ 7.591429][ T1] ? vp_find_vqs+0x4c/0x4e0
> [ 7.592174][ T1] ? virtscsi_probe+0x3ea/0xf60
> [ 7.592853][ T1] ? virtio_dev_probe+0x991/0xaf0
> [ 7.593892][ T1] ? really_probe+0x2b8/0xad0
> [ 7.594581][ T1] ? driver_probe_device+0x50/0x430
> [ 7.595325][ T1] vring_create_virtqueue_split+0xc6/0x310
> [ 7.596200][ T1] ? ret_from_fork+0x4b/0x80
> [ 7.597705][ T1] ? __pfx_vring_create_virtqueue_split+0x10/0x10
> [ 7.599051][ T1] vring_create_virtqueue+0xca/0x110
> [ 7.599905][ T1] ? __pfx_vp_notify+0x10/0x10
> [ 7.600626][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
> [ 7.601770][ T1] setup_vq+0xe9/0x2d0
> [ 7.602429][ T1] ? __pfx_vp_notify+0x10/0x10
> [ 7.603153][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
> [ 7.604003][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
> [ 7.604758][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
> [ 7.605623][ T1] vp_setup_vq+0xbf/0x330
> [ 7.606388][ T1] ? __pfx_vp_config_changed+0x10/0x10
> [ 7.607239][ T1] ? ioread16+0x2f/0x90
> [ 7.608129][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
> [ 7.609047][ T1] vp_find_vqs_msix+0x8b2/0xc80
> [ 7.609880][ T1] vp_find_vqs+0x4c/0x4e0
> [ 7.610578][ T1] virtscsi_init+0x8db/0xd00
> [ 7.611247][ T1] ? __pfx_virtscsi_init+0x10/0x10
> [ 7.612058][ T1] ? __pfx_default_calc_sets+0x10/0x10
> [ 7.612860][ T1] ? scsi_host_alloc+0xa57/0xea0
> [ 7.613671][ T1] ? vp_get+0xfd/0x140
> [ 7.614243][ T1] virtscsi_probe+0x3ea/0xf60
> [ 7.614969][ T1] ? __pfx_virtscsi_probe+0x10/0x10
> [ 7.615847][ T1] ? kernfs_add_one+0x156/0x8b0
> [ 7.616678][ T1] ? virtio_no_restricted_mem_acc+0x9/0x10
> [ 7.617627][ T1] ? virtio_features_ok+0x10c/0x270
> [ 7.618392][ T1] virtio_dev_probe+0x991/0xaf0
> [ 7.619262][ T1] ? __pfx_virtio_dev_probe+0x10/0x10
> [ 7.620216][ T1] really_probe+0x2b8/0xad0
> [ 7.621124][ T1] __driver_probe_device+0x1a2/0x390
> [ 7.621980][ T1] driver_probe_device+0x50/0x430
> [ 7.622710][ T1] __driver_attach+0x45f/0x710
> [ 7.623413][ T1] ? __pfx___driver_attach+0x10/0x10
> [ 7.624299][ T1] bus_for_each_dev+0x239/0x2b0
> [ 7.625118][ T1] ? __pfx___driver_attach+0x10/0x10
> [ 7.625993][ T1] ? __pfx_bus_for_each_dev+0x10/0x10
> [ 7.626858][ T1] ? do_raw_spin_unlock+0x13c/0x8b0
> [ 7.627837][ T1] bus_add_driver+0x347/0x620
> [ 7.628735][ T1] driver_register+0x23a/0x320
> [ 7.629982][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
> [ 7.631006][ T1] virtio_scsi_init+0x65/0xe0
> [ 7.631802][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
> [ 7.632612][ T1] do_one_initcall+0x248/0x880
> [ 7.633404][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
> [ 7.634540][ T1] ? __pfx_do_one_initcall+0x10/0x10
> [ 7.635786][ T1] ? lockdep_hardirqs_on_prepare+0x43d/0x780
> [ 7.636889][ T1] ? __pfx_parse_args+0x10/0x10
> [ 7.637652][ T1] ? do_initcalls+0x1c/0x80
> [ 7.638456][ T1] ? rcu_is_watching+0x15/0xb0
> [ 7.639227][ T1] do_initcall_level+0x157/0x210
> [ 7.640192][ T1] do_initcalls+0x3f/0x80
> [ 7.640818][ T1] kernel_init_freeable+0x435/0x5d0
> [ 7.641593][ T1] ? __pfx_kernel_init_freeable+0x10/0x10
> [ 7.642546][ T1] ? __pfx_lockdep_hardirqs_on_prepare+0x10/0x10
> [ 7.643506][ T1] ? __pfx_kernel_init+0x10/0x10
> [ 7.644295][ T1] ? __pfx_kernel_init+0x10/0x10
> [ 7.645313][ T1] ? __pfx_kernel_init+0x10/0x10
> [ 7.646036][ T1] kernel_init+0x1d/0x2b0
> [ 7.646660][ T1] ret_from_fork+0x4b/0x80
> [ 7.647368][ T1] ? __pfx_kernel_init+0x10/0x10
> [ 7.648542][ T1] ret_from_fork_asm+0x1a/0x30
> [ 7.649812][ T1] </TASK>
> [ 7.650346][ T1] Kernel panic - not syncing: kernel: panic_on_warn set ...
> [ 7.651620][ T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.9.0-rc3-syzkaller-00014-geb06a4b6cca5 #0
> [ 7.653389][ T1] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
> [ 7.655321][ T1] Call Trace:
> [ 7.655801][ T1] <TASK>
> [ 7.656252][ T1] dump_stack_lvl+0x241/0x360
> [ 7.657006][ T1] ? __pfx_dump_stack_lvl+0x10/0x10
> [ 7.657825][ T1] ? __pfx__printk+0x10/0x10
> [ 7.658542][ T1] ? _printk+0xd5/0x120
> [ 7.659343][ T1] ? vscnprintf+0x5d/0x90
> [ 7.659705][ T1] panic+0x349/0x860
> [ 7.659705][ T1] ? __warn+0x172/0x4e0
> [ 7.659705][ T1] ? __pfx_panic+0x10/0x10
> [ 7.659705][ T1] ? show_trace_log_lvl+0x4e6/0x520
> [ 7.659705][ T1] ? ret_from_fork_asm+0x1a/0x30
> [ 7.659705][ T1] __warn+0x346/0x4e0
> [ 7.659705][ T1] ? refcount_warn_saturate+0xfa/0x1d0
> [ 7.659705][ T1] report_bug+0x2b3/0x500
> [ 7.659705][ T1] ? refcount_warn_saturate+0xfa/0x1d0
> [ 7.659705][ T1] handle_bug+0x3e/0x70
> [ 7.659705][ T1] exc_invalid_op+0x1a/0x50
> [ 7.659705][ T1] asm_exc_invalid_op+0x1a/0x20
> [ 7.659705][ T1] RIP: 0010:refcount_warn_saturate+0xfa/0x1d0
> [ 7.659705][ T1] Code: b2 00 00 00 e8 87 70 e7 fc 5b 5d c3 cc cc cc cc e8 7b 70 e7 fc c6 05 0c 5d e5 0a 01 90 48 c7 c7 40 4b 1f 8c e8 17 ee a9 fc 90 <0f> 0b 90 90 eb d9 e8 5b 70 e7 fc c6 05 e9 5c e5 0a 01 90 48 c7 c7
> [ 7.659705][ T1] RSP: 0000:ffffc90000066e18 EFLAGS: 00010246
> [ 7.659705][ T1] RAX: 6f40bd285f2a6000 RBX: ffff888147ed00fc RCX: ffff8880166d0000
> [ 7.659705][ T1] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
> [ 7.659705][ T1] RBP: 0000000000000004 R08: ffffffff815800a2 R09: fffffbfff1c39af8
> [ 7.659705][ T1] R10: dffffc0000000000 R11: fffffbfff1c39af8 R12: ffffea000503edc0
> [ 7.659705][ T1] R13: ffffea000503edc8 R14: 1ffffd4000a07db9 R15: 0000000000000000
> [ 7.659705][ T1] ? __warn_printk+0x292/0x360
> [ 7.659705][ T1] ? refcount_warn_saturate+0xf9/0x1d0
> [ 7.659705][ T1] __free_pages_ok+0xc60/0xd90
> [ 7.659705][ T1] make_alloc_exact+0xa3/0xf0
> [ 7.659705][ T1] vring_alloc_queue_split+0x20a/0x600
> [ 7.659705][ T1] ? __pfx_vring_alloc_queue_split+0x10/0x10
> [ 7.659705][ T1] ? vp_find_vqs+0x4c/0x4e0
> [ 7.659705][ T1] ? virtscsi_probe+0x3ea/0xf60
> [ 7.659705][ T1] ? virtio_dev_probe+0x991/0xaf0
> [ 7.659705][ T1] ? really_probe+0x2b8/0xad0
> [ 7.659705][ T1] ? driver_probe_device+0x50/0x430
> [ 7.659705][ T1] vring_create_virtqueue_split+0xc6/0x310
> [ 7.659705][ T1] ? ret_from_fork+0x4b/0x80
> [ 7.659705][ T1] ? __pfx_vring_create_virtqueue_split+0x10/0x10
> [ 7.659705][ T1] vring_create_virtqueue+0xca/0x110
> [ 7.659705][ T1] ? __pfx_vp_notify+0x10/0x10
> [ 7.659705][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
> [ 7.659705][ T1] setup_vq+0xe9/0x2d0
> [ 7.659705][ T1] ? __pfx_vp_notify+0x10/0x10
> [ 7.659705][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
> [ 7.659705][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
> [ 7.659705][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
> [ 7.659705][ T1] vp_setup_vq+0xbf/0x330
> [ 7.659705][ T1] ? __pfx_vp_config_changed+0x10/0x10
> [ 7.659705][ T1] ? ioread16+0x2f/0x90
> [ 7.659705][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
> [ 7.709861][ T1] vp_find_vqs_msix+0x8b2/0xc80
> [ 7.709861][ T1] vp_find_vqs+0x4c/0x4e0
> [ 7.709861][ T1] virtscsi_init+0x8db/0xd00
> [ 7.709861][ T1] ? __pfx_virtscsi_init+0x10/0x10
> [ 7.709861][ T1] ? __pfx_default_calc_sets+0x10/0x10
> [ 7.709861][ T1] ? scsi_host_alloc+0xa57/0xea0
> [ 7.709861][ T1] ? vp_get+0xfd/0x140
> [ 7.709861][ T1] virtscsi_probe+0x3ea/0xf60
> [ 7.709861][ T1] ? __pfx_virtscsi_probe+0x10/0x10
> [ 7.709861][ T1] ? kernfs_add_one+0x156/0x8b0
> [ 7.709861][ T1] ? virtio_no_restricted_mem_acc+0x9/0x10
> [ 7.709861][ T1] ? virtio_features_ok+0x10c/0x270
> [ 7.709861][ T1] virtio_dev_probe+0x991/0xaf0
> [ 7.709861][ T1] ? __pfx_virtio_dev_probe+0x10/0x10
> [ 7.709861][ T1] really_probe+0x2b8/0xad0
> [ 7.709861][ T1] __driver_probe_device+0x1a2/0x390
> [ 7.709861][ T1] driver_probe_device+0x50/0x430
> [ 7.709861][ T1] __driver_attach+0x45f/0x710
> [ 7.709861][ T1] ? __pfx___driver_attach+0x10/0x10
> [ 7.709861][ T1] bus_for_each_dev+0x239/0x2b0
> [ 7.709861][ T1] ? __pfx___driver_attach+0x10/0x10
> [ 7.709861][ T1] ? __pfx_bus_for_each_dev+0x10/0x10
> [ 7.709861][ T1] ? do_raw_spin_unlock+0x13c/0x8b0
> [ 7.709861][ T1] bus_add_driver+0x347/0x620
> [ 7.709861][ T1] driver_register+0x23a/0x320
> [ 7.709861][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
> [ 7.709861][ T1] virtio_scsi_init+0x65/0xe0
> [ 7.709861][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
> [ 7.709861][ T1] do_one_initcall+0x248/0x880
> [ 7.709861][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
> [ 7.709861][ T1] ? __pfx_do_one_initcall+0x10/0x10
> [ 7.709861][ T1] ? lockdep_hardirqs_on_prepare+0x43d/0x780
> [ 7.709861][ T1] ? __pfx_parse_args+0x10/0x10
> [ 7.709861][ T1] ? do_initcalls+0x1c/0x80
> [ 7.709861][ T1] ? rcu_is_watching+0x15/0xb0
> [ 7.709861][ T1] do_initcall_level+0x157/0x210
> [ 7.709861][ T1] do_initcalls+0x3f/0x80
> [ 7.709861][ T1] kernel_init_freeable+0x435/0x5d0
> [ 7.709861][ T1] ? __pfx_kernel_init_freeable+0x10/0x10
> [ 7.709861][ T1] ? __pfx_lockdep_hardirqs_on_prepare+0x10/0x10
> [ 7.709861][ T1] ? __pfx_kernel_init+0x10/0x10
> [ 7.709861][ T1] ? __pfx_kernel_init+0x10/0x10
> [ 7.709861][ T1] ? __pfx_kernel_init+0x10/0x10
> [ 7.709861][ T1] kernel_init+0x1d/0x2b0
> [ 7.709861][ T1] ret_from_fork+0x4b/0x80
> [ 7.709861][ T1] ? __pfx_kernel_init+0x10/0x10
> [ 7.709861][ T1] ret_from_fork_asm+0x1a/0x30
> [ 7.709861][ T1] </TASK>
> [ 7.709861][ T1] Kernel Offset: disabled
> [ 7.709861][ T1] Rebooting in 86400 seconds..
>
>
Not sure what this is about.
Let's try again after rebase to current master:
#syz test: https://github.com/amir73il/linux fsnotify-fixes
Thanks,
Amir.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [syzbot] [ext4?] KASAN: slab-use-after-free Read in fsnotify
2024-04-11 20:27 ` Khazhy Kumykov
@ 2024-04-12 10:24 ` Amir Goldstein
0 siblings, 0 replies; 26+ messages in thread
From: Amir Goldstein @ 2024-04-12 10:24 UTC (permalink / raw)
To: Khazhy Kumykov
Cc: Gabriel Krisman Bertazi, Jan Kara, syzbot, linux-ext4,
linux-fsdevel, linux-kernel, repnop, syzkaller-bugs
On Thu, Apr 11, 2024 at 11:27 PM Khazhy Kumykov <khazhy@chromium.org> wrote:
>
> On Thu, Apr 11, 2024 at 12:25 PM Gabriel Krisman Bertazi
> <krisman@suse.de> wrote:
> >
> > Amir Goldstein <amir73il@gmail.com> writes:
> >
> > > On Thu, Apr 11, 2024 at 3:13 PM Jan Kara <jack@suse.cz> wrote:
> > >>
> > >> On Thu 11-04-24 01:11:20, syzbot wrote:
> > >> > Hello,
> > >> >
> > >> > syzbot found the following issue on:
> > >> >
> > >> > HEAD commit: 6ebf211bb11d Add linux-next specific files for 20240410
> > >> > git tree: linux-next
> > >> > console+strace: https://syzkaller.appspot.com/x/log.txt?x=12be955d180000
> > >> > kernel config: https://syzkaller.appspot.com/x/.config?x=16ca158ef7e08662
> > >> > dashboard link: https://syzkaller.appspot.com/bug?extid=5e3f9b2a67b45f16d4e6
> > >> > compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
> > >> > syz repro: https://syzkaller.appspot.com/x/repro.syz?x=13c91175180000
> > >> > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=1621af9d180000
> > >> >
> > >> > Downloadable assets:
> > >> > disk image: https://storage.googleapis.com/syzbot-assets/b050f81f73ed/disk-6ebf211b.raw.xz
> > >> > vmlinux: https://storage.googleapis.com/syzbot-assets/412c9b9a536e/vmlinux-6ebf211b.xz
> > >> > kernel image: https://storage.googleapis.com/syzbot-assets/016527216c47/bzImage-6ebf211b.xz
> > >> > mounted in repro: https://storage.googleapis.com/syzbot-assets/75ad050c9945/mount_0.gz
> > >> >
> > >> > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > >> > Reported-by: syzbot+5e3f9b2a67b45f16d4e6@syzkaller.appspotmail.com
> > >> >
> > >> > Quota error (device loop0): do_check_range: Getting block 0 out of range 1-5
> > >> > EXT4-fs error (device loop0): ext4_release_dquot:6905: comm kworker/u8:4: Failed to release dquot type 1
> > >> > ==================================================================
> > >> > BUG: KASAN: slab-use-after-free in fsnotify+0x2a4/0x1f70 fs/notify/fsnotify.c:539
> > >> > Read of size 8 at addr ffff88802f1dce80 by task kworker/u8:4/62
> > >> >
> > >> > CPU: 0 PID: 62 Comm: kworker/u8:4 Not tainted 6.9.0-rc3-next-20240410-syzkaller #0
> > >> > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
> > >> > Workqueue: events_unbound quota_release_workfn
> > >> > Call Trace:
> > >> > <TASK>
> > >> > __dump_stack lib/dump_stack.c:88 [inline]
> > >> > dump_stack_lvl+0x241/0x360 lib/dump_stack.c:114
> > >> > print_address_description mm/kasan/report.c:377 [inline]
> > >> > print_report+0x169/0x550 mm/kasan/report.c:488
> > >> > kasan_report+0x143/0x180 mm/kasan/report.c:601
> > >> > fsnotify+0x2a4/0x1f70 fs/notify/fsnotify.c:539
> > >> > fsnotify_sb_error include/linux/fsnotify.h:456 [inline]
> > >> > __ext4_error+0x255/0x3b0 fs/ext4/super.c:843
> > >> > ext4_release_dquot+0x326/0x450 fs/ext4/super.c:6903
> > >> > quota_release_workfn+0x39f/0x650 fs/quota/dquot.c:840
> > >> > process_one_work kernel/workqueue.c:3218 [inline]
> > >> > process_scheduled_works+0xa2c/0x1830 kernel/workqueue.c:3299
> > >> > worker_thread+0x86d/0xd70 kernel/workqueue.c:3380
> > >> > kthread+0x2f0/0x390 kernel/kthread.c:389
> > >> > ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
> > >> > ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
> > >> > </TASK>
> > >>
> > >> Amir, I believe this happens on umount when the filesystem calls
> > >> fsnotify_sb_error() after calling fsnotify_sb_delete().
> Hmm, so we're releasing dquots after already shutting down the
> filesystem? Is that expected? This "Failed to release dquot type"
> error message only appears if we have an open handle from
> ext4_journal_start (although this filesystem was mounted without a
> journal, so we hit ext4_get_nojournal()...)
> > In theory these two
> > >> calls can even run in parallel and fsnotify() can be holding
> > >> fsnotify_sb_info pointer while fsnotify_sb_delete() is freeing it so we
> > >> need to figure out some proper synchronization for that...
> > >
> > > Is it really needed to handle any for non SB_ACTIVE sb?
> >
> > I think it should be fine to exclude volumes being teared down. Cc'ing
> > Khazhy, who sponsored this work at the time and owned the use-case.
> In terms of real-world use case... not sure there is one - you'll
> notice the errors when you fsck/mount again later on, and any users
> who care about notifs should be gone by the time we're unmounting. But
> it seems weird to me that we can get write errors shutting everything
> down.
That's what I thought.
Anyway, my naive fix breaks IN_UNMOUNT event as well as some other
fanotify tests, so I will need to work on another fix.
Thanks,
Amir.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [syzbot] [ext4?] KASAN: slab-use-after-free Read in fsnotify
2024-04-12 7:52 ` Amir Goldstein
@ 2024-04-12 14:30 ` syzbot
2024-04-12 14:32 ` Aleksandr Nogikh
0 siblings, 1 reply; 26+ messages in thread
From: syzbot @ 2024-04-12 14:30 UTC (permalink / raw)
To: amir73il, jack, krisman, linux-ext4, linux-fsdevel, linux-kernel,
repnop, syzkaller-bugs
Hello,
syzbot tried to test the proposed patch but the build/boot failed:
viperboard
[ 7.815991][ T1] usbcore: registered new interface driver dln2
[ 7.818221][ T1] usbcore: registered new interface driver pn533_usb
[ 7.823865][ T1] nfcsim 0.2 initialized
[ 7.825314][ T1] usbcore: registered new interface driver port100
[ 7.828562][ T1] usbcore: registered new interface driver nfcmrvl
[ 7.837679][ T1] Loading iSCSI transport class v2.0-870.
[ 7.856316][ T1] virtio_scsi virtio0: 1/0/0 default/read/poll queues
[ 7.867881][ T1] ------------[ cut here ]------------
[ 7.869240][ T1] refcount_t: decrement hit 0; leaking memory.
[ 7.870647][ T1] WARNING: CPU: 0 PID: 1 at lib/refcount.c:31 refcount_warn_saturate+0xfa/0x1d0
[ 7.872673][ T1] Modules linked in:
[ 7.873450][ T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.9.0-rc3-syzkaller-00222-ga4170c0055a4 #0
[ 7.875073][ T1] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
[ 7.876781][ T1] RIP: 0010:refcount_warn_saturate+0xfa/0x1d0
[ 7.877963][ T1] Code: b2 00 00 00 e8 e7 63 e7 fc 5b 5d c3 cc cc cc cc e8 db 63 e7 fc c6 05 6c d2 e4 0a 01 90 48 c7 c7 c0 30 1f 8c e8 77 e1 a9 fc 90 <0f> 0b 90 90 eb d9 e8 bb 63 e7 fc c6 05 49 d2 e4 0a 01 90 48 c7 c7
[ 7.881853][ T1] RSP: 0000:ffffc90000066e18 EFLAGS: 00010246
[ 7.883056][ T1] RAX: 7664671302135400 RBX: ffff8880202afc6c RCX: ffff8880166d0000
[ 7.885276][ T1] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 7.887369][ T1] RBP: 0000000000000004 R08: ffffffff815880a2 R09: fffffbfff1c39b48
[ 7.888750][ T1] R10: dffffc0000000000 R11: fffffbfff1c39b48 R12: ffffea000502ddc0
[ 7.890131][ T1] R13: ffffea000502ddc8 R14: 1ffffd4000a05bb9 R15: 0000000000000000
[ 7.891813][ T1] FS: 0000000000000000(0000) GS:ffff8880b9400000(0000) knlGS:0000000000000000
[ 7.893900][ T1] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 7.895917][ T1] CR2: ffff88823ffff000 CR3: 000000000e134000 CR4: 00000000003506f0
[ 7.897448][ T1] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 7.898766][ T1] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 7.900462][ T1] Call Trace:
[ 7.901245][ T1] <TASK>
[ 7.901998][ T1] ? __warn+0x163/0x4e0
[ 7.902783][ T1] ? refcount_warn_saturate+0xfa/0x1d0
[ 7.904160][ T1] ? report_bug+0x2b3/0x500
[ 7.905119][ T1] ? refcount_warn_saturate+0xfa/0x1d0
[ 7.906474][ T1] ? handle_bug+0x3e/0x70
[ 7.907513][ T1] ? exc_invalid_op+0x1a/0x50
[ 7.908463][ T1] ? asm_exc_invalid_op+0x1a/0x20
[ 7.909468][ T1] ? __warn_printk+0x292/0x360
[ 7.910957][ T1] ? refcount_warn_saturate+0xfa/0x1d0
[ 7.912558][ T1] ? refcount_warn_saturate+0xf9/0x1d0
[ 7.913803][ T1] __free_pages_ok+0xc60/0xd90
[ 7.914583][ T1] make_alloc_exact+0xa3/0xf0
[ 7.915425][ T1] vring_alloc_queue_split+0x20a/0x600
[ 7.916903][ T1] ? __pfx_vring_alloc_queue_split+0x10/0x10
[ 7.917937][ T1] ? vp_find_vqs+0x4c/0x4e0
[ 7.918930][ T1] ? virtscsi_probe+0x3ea/0xf60
[ 7.919998][ T1] ? virtio_dev_probe+0x991/0xaf0
[ 7.921234][ T1] ? really_probe+0x2b8/0xad0
[ 7.921893][ T1] ? driver_probe_device+0x50/0x430
[ 7.922882][ T1] vring_create_virtqueue_split+0xc6/0x310
[ 7.924087][ T1] ? ret_from_fork+0x4b/0x80
[ 7.924906][ T1] ? __pfx_vring_create_virtqueue_split+0x10/0x10
[ 7.926406][ T1] vring_create_virtqueue+0xca/0x110
[ 7.927871][ T1] ? __pfx_vp_notify+0x10/0x10
[ 7.929037][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 7.929986][ T1] setup_vq+0xe9/0x2d0
[ 7.930782][ T1] ? __pfx_vp_notify+0x10/0x10
[ 7.931882][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 7.932643][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 7.933619][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 7.934781][ T1] vp_setup_vq+0xbf/0x330
[ 7.935911][ T1] ? __pfx_vp_config_changed+0x10/0x10
[ 7.937334][ T1] ? ioread16+0x2f/0x90
[ 7.938489][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 7.939978][ T1] vp_find_vqs_msix+0x8b2/0xc80
[ 7.940989][ T1] vp_find_vqs+0x4c/0x4e0
[ 7.941655][ T1] virtscsi_init+0x8db/0xd00
[ 7.942427][ T1] ? __pfx_virtscsi_init+0x10/0x10
[ 7.943724][ T1] ? __pfx_default_calc_sets+0x10/0x10
[ 7.944616][ T1] ? scsi_host_alloc+0xa57/0xea0
[ 7.946041][ T1] ? vp_get+0xfd/0x140
[ 7.946895][ T1] virtscsi_probe+0x3ea/0xf60
[ 7.947740][ T1] ? __pfx_virtscsi_probe+0x10/0x10
[ 7.948718][ T1] ? kernfs_add_one+0x156/0x8b0
[ 7.949578][ T1] ? virtio_no_restricted_mem_acc+0x9/0x10
[ 7.950761][ T1] ? virtio_features_ok+0x10c/0x270
[ 7.951946][ T1] virtio_dev_probe+0x991/0xaf0
[ 7.953286][ T1] ? __pfx_virtio_dev_probe+0x10/0x10
[ 7.954435][ T1] really_probe+0x2b8/0xad0
[ 7.955311][ T1] __driver_probe_device+0x1a2/0x390
[ 7.956686][ T1] driver_probe_device+0x50/0x430
[ 7.957771][ T1] __driver_attach+0x45f/0x710
[ 7.958769][ T1] ? __pfx___driver_attach+0x10/0x10
[ 7.959612][ T1] bus_for_each_dev+0x239/0x2b0
[ 7.960946][ T1] ? __pfx___driver_attach+0x10/0x10
[ 7.962251][ T1] ? __pfx_bus_for_each_dev+0x10/0x10
[ 7.964315][ T1] ? do_raw_spin_unlock+0x13c/0x8b0
[ 7.966148][ T1] bus_add_driver+0x347/0x620
[ 7.967112][ T1] driver_register+0x23a/0x320
[ 7.968628][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
[ 7.969941][ T1] virtio_scsi_init+0x65/0xe0
[ 7.970833][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
[ 7.972203][ T1] do_one_initcall+0x248/0x880
[ 7.973355][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
[ 7.974515][ T1] ? __pfx_do_one_initcall+0x10/0x10
[ 7.975602][ T1] ? lockdep_hardirqs_on_prepare+0x43d/0x780
[ 7.977269][ T1] ? __pfx_parse_args+0x10/0x10
[ 7.978148][ T1] ? do_initcalls+0x1c/0x80
[ 7.979254][ T1] ? rcu_is_watching+0x15/0xb0
[ 7.980350][ T1] do_initcall_level+0x157/0x210
[ 7.981349][ T1] do_initcalls+0x3f/0x80
[ 7.982742][ T1] kernel_init_freeable+0x435/0x5d0
[ 7.984461][ T1] ? __pfx_kernel_init_freeable+0x10/0x10
[ 7.986325][ T1] ? __pfx_lockdep_hardirqs_on_prepare+0x10/0x10
[ 7.987513][ T1] ? __pfx_kernel_init+0x10/0x10
[ 7.988803][ T1] ? __pfx_kernel_init+0x10/0x10
[ 7.990065][ T1] ? __pfx_kernel_init+0x10/0x10
[ 7.991335][ T1] kernel_init+0x1d/0x2b0
[ 7.992855][ T1] ret_from_fork+0x4b/0x80
[ 7.994528][ T1] ? __pfx_kernel_init+0x10/0x10
[ 7.996074][ T1] ret_from_fork_asm+0x1a/0x30
[ 7.997423][ T1] </TASK>
[ 7.998234][ T1] Kernel panic - not syncing: kernel: panic_on_warn set ...
[ 7.999604][ T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.9.0-rc3-syzkaller-00222-ga4170c0055a4 #0
[ 8.001827][ T1] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
[ 8.004098][ T1] Call Trace:
[ 8.005064][ T1] <TASK>
[ 8.005501][ T1] dump_stack_lvl+0x241/0x360
[ 8.006050][ T1] ? __pfx_dump_stack_lvl+0x10/0x10
[ 8.006050][ T1] ? __pfx__printk+0x10/0x10
[ 8.006050][ T1] ? _printk+0xd5/0x120
[ 8.006050][ T1] ? vscnprintf+0x5d/0x90
[ 8.006050][ T1] panic+0x349/0x860
[ 8.006050][ T1] ? __warn+0x172/0x4e0
[ 8.006050][ T1] ? __pfx_panic+0x10/0x10
[ 8.006050][ T1] ? show_trace_log_lvl+0x4e6/0x520
[ 8.006050][ T1] ? ret_from_fork_asm+0x1a/0x30
[ 8.006050][ T1] __warn+0x346/0x4e0
[ 8.006050][ T1] ? refcount_warn_saturate+0xfa/0x1d0
[ 8.006050][ T1] report_bug+0x2b3/0x500
[ 8.006050][ T1] ? refcount_warn_saturate+0xfa/0x1d0
[ 8.006050][ T1] handle_bug+0x3e/0x70
[ 8.006050][ T1] exc_invalid_op+0x1a/0x50
[ 8.006050][ T1] asm_exc_invalid_op+0x1a/0x20
[ 8.006050][ T1] RIP: 0010:refcount_warn_saturate+0xfa/0x1d0
[ 8.006050][ T1] Code: b2 00 00 00 e8 e7 63 e7 fc 5b 5d c3 cc cc cc cc e8 db 63 e7 fc c6 05 6c d2 e4 0a 01 90 48 c7 c7 c0 30 1f 8c e8 77 e1 a9 fc 90 <0f> 0b 90 90 eb d9 e8 bb 63 e7 fc c6 05 49 d2 e4 0a 01 90 48 c7 c7
[ 8.006050][ T1] RSP: 0000:ffffc90000066e18 EFLAGS: 00010246
[ 8.006050][ T1] RAX: 7664671302135400 RBX: ffff8880202afc6c RCX: ffff8880166d0000
[ 8.006050][ T1] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 8.006050][ T1] RBP: 0000000000000004 R08: ffffffff815880a2 R09: fffffbfff1c39b48
[ 8.006050][ T1] R10: dffffc0000000000 R11: fffffbfff1c39b48 R12: ffffea000502ddc0
[ 8.006050][ T1] R13: ffffea000502ddc8 R14: 1ffffd4000a05bb9 R15: 0000000000000000
[ 8.006050][ T1] ? __warn_printk+0x292/0x360
[ 8.006050][ T1] ? refcount_warn_saturate+0xf9/0x1d0
[ 8.006050][ T1] __free_pages_ok+0xc60/0xd90
[ 8.006050][ T1] make_alloc_exact+0xa3/0xf0
[ 8.006050][ T1] vring_alloc_queue_split+0x20a/0x600
[ 8.006050][ T1] ? __pfx_vring_alloc_queue_split+0x10/0x10
[ 8.006050][ T1] ? vp_find_vqs+0x4c/0x4e0
[ 8.006050][ T1] ? virtscsi_probe+0x3ea/0xf60
[ 8.006050][ T1] ? virtio_dev_probe+0x991/0xaf0
[ 8.006050][ T1] ? really_probe+0x2b8/0xad0
[ 8.006050][ T1] ? driver_probe_device+0x50/0x430
[ 8.006050][ T1] vring_create_virtqueue_split+0xc6/0x310
[ 8.006050][ T1] ? ret_from_fork+0x4b/0x80
[ 8.055954][ T1] ? __pfx_vring_create_virtqueue_split+0x10/0x10
[ 8.055954][ T1] vring_create_virtqueue+0xca/0x110
[ 8.055954][ T1] ? __pfx_vp_notify+0x10/0x10
[ 8.055954][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 8.055954][ T1] setup_vq+0xe9/0x2d0
[ 8.055954][ T1] ? __pfx_vp_notify+0x10/0x10
[ 8.055954][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 8.055954][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 8.055954][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 8.055954][ T1] vp_setup_vq+0xbf/0x330
[ 8.055954][ T1] ? __pfx_vp_config_changed+0x10/0x10
[ 8.055954][ T1] ? ioread16+0x2f/0x90
[ 8.055954][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 8.055954][ T1] vp_find_vqs_msix+0x8b2/0xc80
[ 8.055954][ T1] vp_find_vqs+0x4c/0x4e0
[ 8.055954][ T1] virtscsi_init+0x8db/0xd00
[ 8.055954][ T1] ? __pfx_virtscsi_init+0x10/0x10
[ 8.055954][ T1] ? __pfx_default_calc_sets+0x10/0x10
[ 8.055954][ T1] ? scsi_host_alloc+0xa57/0xea0
[ 8.055954][ T1] ? vp_get+0xfd/0x140
[ 8.055954][ T1] virtscsi_probe+0x3ea/0xf60
[ 8.055954][ T1] ? __pfx_virtscsi_probe+0x10/0x10
[ 8.055954][ T1] ? kernfs_add_one+0x156/0x8b0
[ 8.055954][ T1] ? virtio_no_restricted_mem_acc+0x9/0x10
[ 8.055954][ T1] ? virtio_features_ok+0x10c/0x270
[ 8.055954][ T1] virtio_dev_probe+0x991/0xaf0
[ 8.055954][ T1] ? __pfx_virtio_dev_probe+0x10/0x10
[ 8.055954][ T1] really_probe+0x2b8/0xad0
[ 8.055954][ T1] __driver_probe_device+0x1a2/0x390
[ 8.055954][ T1] driver_probe_device+0x50/0x430
[ 8.055954][ T1] __driver_attach+0x45f/0x710
[ 8.055954][ T1] ? __pfx___driver_attach+0x10/0x10
[ 8.055954][ T1] bus_for_each_dev+0x239/0x2b0
[ 8.055954][ T1] ? __pfx___driver_attach+0x10/0x10
[ 8.055954][ T1] ? __pfx_bus_for_each_dev+0x10/0x10
[ 8.055954][ T1] ? do_raw_spin_unlock+0x13c/0x8b0
[ 8.055954][ T1] bus_add_driver+0x347/0x620
[ 8.055954][ T1] driver_register+0x23a/0x320
[ 8.055954][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
[ 8.055954][ T1] virtio_scsi_init+0x65/0xe0
[ 8.055954][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
[ 8.055954][ T1] do_one_initcall+0x248/0x880
[ 8.055954][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
[ 8.055954][ T1] ? __pfx_do_one_initcall+0x10/0x10
[ 8.105903][ T1] ? lockdep_hardirqs_on_prepare+0x43d/0x780
[ 8.105903][ T1] ? __pfx_parse_args+0x10/0x10
[ 8.105903][ T1] ? do_initcalls+0x1c/0x80
[ 8.105903][ T1] ? rcu_is_watching+0x15/0xb0
[ 8.105903][ T1] do_initcall_level+0x157/0x210
[ 8.105903][ T1] do_initcalls+0x3f/0x80
[ 8.105903][ T1] kernel_init_freeable+0x435/0x5d0
[ 8.105903][ T1] ? __pfx_kernel_init_freeable+0x10/0x10
[ 8.105903][ T1] ? __pfx_lockdep_hardirqs_on_prepare+0x10/0x10
[ 8.105903][ T1] ? __pfx_kernel_init+0x10/0x10
[ 8.105903][ T1] ? __pfx_kernel_init+0x10/0x10
[ 8.105903][ T1] ? __pfx_kernel_init+0x10/0x10
[ 8.105903][ T1] kernel_init+0x1d/0x2b0
[ 8.105903][ T1] ret_from_fork+0x4b/0x80
[ 8.105903][ T1] ? __pfx_kernel_init+0x10/0x10
[ 8.105903][ T1] ret_from_fork_asm+0x1a/0x30
[ 8.105903][ T1] </TASK>
[ 8.105903][ T1] Kernel Offset: disabled
[ 8.105903][ T1] Rebooting in 86400 seconds..
syzkaller build log:
go env (err=<nil>)
GO111MODULE='auto'
GOARCH='amd64'
GOBIN=''
GOCACHE='/syzkaller/.cache/go-build'
GOENV='/syzkaller/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/syzkaller/jobs-2/linux/gopath/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/syzkaller/jobs-2/linux/gopath'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.4'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/syzkaller/jobs-2/linux/gopath/src/github.com/google/syzkaller/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1668308411=/tmp/go-build -gno-record-gcc-switches'
git status (err=<nil>)
HEAD detached at 56086b24b
nothing to commit, working tree clean
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
Makefile:31: run command via tools/syz-env for best compatibility, see:
Makefile:32: https://github.com/google/syzkaller/blob/master/docs/contributing.md#using-syz-env
go list -f '{{.Stale}}' ./sys/syz-sysgen | grep -q false || go install ./sys/syz-sysgen
make .descriptions
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
Makefile:31: run command via tools/syz-env for best compatibility, see:
Makefile:32: https://github.com/google/syzkaller/blob/master/docs/contributing.md#using-syz-env
bin/syz-sysgen
touch .descriptions
GOOS=linux GOARCH=amd64 go build "-ldflags=-s -w -X github.com/google/syzkaller/prog.GitRevision=56086b24bdfd822d3b227edb3064db443cd8c971 -X 'github.com/google/syzkaller/prog.gitRevisionDate=20240409-083312'" "-tags=syz_target syz_os_linux syz_arch_amd64 " -o ./bin/linux_amd64/syz-fuzzer github.com/google/syzkaller/syz-fuzzer
GOOS=linux GOARCH=amd64 go build "-ldflags=-s -w -X github.com/google/syzkaller/prog.GitRevision=56086b24bdfd822d3b227edb3064db443cd8c971 -X 'github.com/google/syzkaller/prog.gitRevisionDate=20240409-083312'" "-tags=syz_target syz_os_linux syz_arch_amd64 " -o ./bin/linux_amd64/syz-execprog github.com/google/syzkaller/tools/syz-execprog
GOOS=linux GOARCH=amd64 go build "-ldflags=-s -w -X github.com/google/syzkaller/prog.GitRevision=56086b24bdfd822d3b227edb3064db443cd8c971 -X 'github.com/google/syzkaller/prog.gitRevisionDate=20240409-083312'" "-tags=syz_target syz_os_linux syz_arch_amd64 " -o ./bin/linux_amd64/syz-stress github.com/google/syzkaller/tools/syz-stress
mkdir -p ./bin/linux_amd64
gcc -o ./bin/linux_amd64/syz-executor executor/executor.cc \
-m64 -O2 -pthread -Wall -Werror -Wparentheses -Wunused-const-variable -Wframe-larger-than=16384 -Wno-stringop-overflow -Wno-array-bounds -Wno-format-overflow -Wno-unused-but-set-variable -Wno-unused-command-line-argument -static-pie -fpermissive -w -DGOOS_linux=1 -DGOARCH_amd64=1 \
-DHOSTGOOS_linux=1 -DGIT_REVISION=\"56086b24bdfd822d3b227edb3064db443cd8c971\"
Error text is too large and was truncated, full error text is at:
https://syzkaller.appspot.com/x/error.txt?x=122f8f97180000
Tested on:
commit: a4170c00 fsnotify: do not handle events on a shutting ..
git tree: https://github.com/amir73il/linux fsnotify-fixes
kernel config: https://syzkaller.appspot.com/x/.config?x=9995779c8305f57e
dashboard link: https://syzkaller.appspot.com/bug?extid=5e3f9b2a67b45f16d4e6
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
Note: no patches were applied.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [syzbot] [ext4?] KASAN: slab-use-after-free Read in fsnotify
2024-04-12 14:30 ` syzbot
@ 2024-04-12 14:32 ` Aleksandr Nogikh
0 siblings, 0 replies; 26+ messages in thread
From: Aleksandr Nogikh @ 2024-04-12 14:32 UTC (permalink / raw)
To: syzbot
Cc: amir73il, jack, krisman, linux-ext4, linux-fsdevel, linux-kernel,
repnop, syzkaller-bugs
FWIW all latest v6.9 release candidates just do not boot with the syzbot config.
There's a "mm,page_owner: Fix refcount imbalance" series that
addresses it, but it has not apparently reached torvalds yet.
On Fri, Apr 12, 2024 at 4:30 PM syzbot
<syzbot+5e3f9b2a67b45f16d4e6@syzkaller.appspotmail.com> wrote:
>
> Hello,
>
> syzbot tried to test the proposed patch but the build/boot failed:
>
> viperboard
> [ 7.815991][ T1] usbcore: registered new interface driver dln2
> [ 7.818221][ T1] usbcore: registered new interface driver pn533_usb
> [ 7.823865][ T1] nfcsim 0.2 initialized
> [ 7.825314][ T1] usbcore: registered new interface driver port100
> [ 7.828562][ T1] usbcore: registered new interface driver nfcmrvl
> [ 7.837679][ T1] Loading iSCSI transport class v2.0-870.
> [ 7.856316][ T1] virtio_scsi virtio0: 1/0/0 default/read/poll queues
> [ 7.867881][ T1] ------------[ cut here ]------------
> [ 7.869240][ T1] refcount_t: decrement hit 0; leaking memory.
> [ 7.870647][ T1] WARNING: CPU: 0 PID: 1 at lib/refcount.c:31 refcount_warn_saturate+0xfa/0x1d0
> [ 7.872673][ T1] Modules linked in:
> [ 7.873450][ T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.9.0-rc3-syzkaller-00222-ga4170c0055a4 #0
> [ 7.875073][ T1] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
> [ 7.876781][ T1] RIP: 0010:refcount_warn_saturate+0xfa/0x1d0
> [ 7.877963][ T1] Code: b2 00 00 00 e8 e7 63 e7 fc 5b 5d c3 cc cc cc cc e8 db 63 e7 fc c6 05 6c d2 e4 0a 01 90 48 c7 c7 c0 30 1f 8c e8 77 e1 a9 fc 90 <0f> 0b 90 90 eb d9 e8 bb 63 e7 fc c6 05 49 d2 e4 0a 01 90 48 c7 c7
> [ 7.881853][ T1] RSP: 0000:ffffc90000066e18 EFLAGS: 00010246
> [ 7.883056][ T1] RAX: 7664671302135400 RBX: ffff8880202afc6c RCX: ffff8880166d0000
> [ 7.885276][ T1] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
> [ 7.887369][ T1] RBP: 0000000000000004 R08: ffffffff815880a2 R09: fffffbfff1c39b48
> [ 7.888750][ T1] R10: dffffc0000000000 R11: fffffbfff1c39b48 R12: ffffea000502ddc0
> [ 7.890131][ T1] R13: ffffea000502ddc8 R14: 1ffffd4000a05bb9 R15: 0000000000000000
> [ 7.891813][ T1] FS: 0000000000000000(0000) GS:ffff8880b9400000(0000) knlGS:0000000000000000
> [ 7.893900][ T1] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 7.895917][ T1] CR2: ffff88823ffff000 CR3: 000000000e134000 CR4: 00000000003506f0
> [ 7.897448][ T1] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 7.898766][ T1] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> [ 7.900462][ T1] Call Trace:
> [ 7.901245][ T1] <TASK>
> [ 7.901998][ T1] ? __warn+0x163/0x4e0
> [ 7.902783][ T1] ? refcount_warn_saturate+0xfa/0x1d0
> [ 7.904160][ T1] ? report_bug+0x2b3/0x500
> [ 7.905119][ T1] ? refcount_warn_saturate+0xfa/0x1d0
> [ 7.906474][ T1] ? handle_bug+0x3e/0x70
> [ 7.907513][ T1] ? exc_invalid_op+0x1a/0x50
> [ 7.908463][ T1] ? asm_exc_invalid_op+0x1a/0x20
> [ 7.909468][ T1] ? __warn_printk+0x292/0x360
> [ 7.910957][ T1] ? refcount_warn_saturate+0xfa/0x1d0
> [ 7.912558][ T1] ? refcount_warn_saturate+0xf9/0x1d0
> [ 7.913803][ T1] __free_pages_ok+0xc60/0xd90
> [ 7.914583][ T1] make_alloc_exact+0xa3/0xf0
> [ 7.915425][ T1] vring_alloc_queue_split+0x20a/0x600
> [ 7.916903][ T1] ? __pfx_vring_alloc_queue_split+0x10/0x10
> [ 7.917937][ T1] ? vp_find_vqs+0x4c/0x4e0
> [ 7.918930][ T1] ? virtscsi_probe+0x3ea/0xf60
> [ 7.919998][ T1] ? virtio_dev_probe+0x991/0xaf0
> [ 7.921234][ T1] ? really_probe+0x2b8/0xad0
> [ 7.921893][ T1] ? driver_probe_device+0x50/0x430
> [ 7.922882][ T1] vring_create_virtqueue_split+0xc6/0x310
> [ 7.924087][ T1] ? ret_from_fork+0x4b/0x80
> [ 7.924906][ T1] ? __pfx_vring_create_virtqueue_split+0x10/0x10
> [ 7.926406][ T1] vring_create_virtqueue+0xca/0x110
> [ 7.927871][ T1] ? __pfx_vp_notify+0x10/0x10
> [ 7.929037][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
> [ 7.929986][ T1] setup_vq+0xe9/0x2d0
> [ 7.930782][ T1] ? __pfx_vp_notify+0x10/0x10
> [ 7.931882][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
> [ 7.932643][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
> [ 7.933619][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
> [ 7.934781][ T1] vp_setup_vq+0xbf/0x330
> [ 7.935911][ T1] ? __pfx_vp_config_changed+0x10/0x10
> [ 7.937334][ T1] ? ioread16+0x2f/0x90
> [ 7.938489][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
> [ 7.939978][ T1] vp_find_vqs_msix+0x8b2/0xc80
> [ 7.940989][ T1] vp_find_vqs+0x4c/0x4e0
> [ 7.941655][ T1] virtscsi_init+0x8db/0xd00
> [ 7.942427][ T1] ? __pfx_virtscsi_init+0x10/0x10
> [ 7.943724][ T1] ? __pfx_default_calc_sets+0x10/0x10
> [ 7.944616][ T1] ? scsi_host_alloc+0xa57/0xea0
> [ 7.946041][ T1] ? vp_get+0xfd/0x140
> [ 7.946895][ T1] virtscsi_probe+0x3ea/0xf60
> [ 7.947740][ T1] ? __pfx_virtscsi_probe+0x10/0x10
> [ 7.948718][ T1] ? kernfs_add_one+0x156/0x8b0
> [ 7.949578][ T1] ? virtio_no_restricted_mem_acc+0x9/0x10
> [ 7.950761][ T1] ? virtio_features_ok+0x10c/0x270
> [ 7.951946][ T1] virtio_dev_probe+0x991/0xaf0
> [ 7.953286][ T1] ? __pfx_virtio_dev_probe+0x10/0x10
> [ 7.954435][ T1] really_probe+0x2b8/0xad0
> [ 7.955311][ T1] __driver_probe_device+0x1a2/0x390
> [ 7.956686][ T1] driver_probe_device+0x50/0x430
> [ 7.957771][ T1] __driver_attach+0x45f/0x710
> [ 7.958769][ T1] ? __pfx___driver_attach+0x10/0x10
> [ 7.959612][ T1] bus_for_each_dev+0x239/0x2b0
> [ 7.960946][ T1] ? __pfx___driver_attach+0x10/0x10
> [ 7.962251][ T1] ? __pfx_bus_for_each_dev+0x10/0x10
> [ 7.964315][ T1] ? do_raw_spin_unlock+0x13c/0x8b0
> [ 7.966148][ T1] bus_add_driver+0x347/0x620
> [ 7.967112][ T1] driver_register+0x23a/0x320
> [ 7.968628][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
> [ 7.969941][ T1] virtio_scsi_init+0x65/0xe0
> [ 7.970833][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
> [ 7.972203][ T1] do_one_initcall+0x248/0x880
> [ 7.973355][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
> [ 7.974515][ T1] ? __pfx_do_one_initcall+0x10/0x10
> [ 7.975602][ T1] ? lockdep_hardirqs_on_prepare+0x43d/0x780
> [ 7.977269][ T1] ? __pfx_parse_args+0x10/0x10
> [ 7.978148][ T1] ? do_initcalls+0x1c/0x80
> [ 7.979254][ T1] ? rcu_is_watching+0x15/0xb0
> [ 7.980350][ T1] do_initcall_level+0x157/0x210
> [ 7.981349][ T1] do_initcalls+0x3f/0x80
> [ 7.982742][ T1] kernel_init_freeable+0x435/0x5d0
> [ 7.984461][ T1] ? __pfx_kernel_init_freeable+0x10/0x10
> [ 7.986325][ T1] ? __pfx_lockdep_hardirqs_on_prepare+0x10/0x10
> [ 7.987513][ T1] ? __pfx_kernel_init+0x10/0x10
> [ 7.988803][ T1] ? __pfx_kernel_init+0x10/0x10
> [ 7.990065][ T1] ? __pfx_kernel_init+0x10/0x10
> [ 7.991335][ T1] kernel_init+0x1d/0x2b0
> [ 7.992855][ T1] ret_from_fork+0x4b/0x80
> [ 7.994528][ T1] ? __pfx_kernel_init+0x10/0x10
> [ 7.996074][ T1] ret_from_fork_asm+0x1a/0x30
> [ 7.997423][ T1] </TASK>
> [ 7.998234][ T1] Kernel panic - not syncing: kernel: panic_on_warn set ...
> [ 7.999604][ T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.9.0-rc3-syzkaller-00222-ga4170c0055a4 #0
> [ 8.001827][ T1] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
> [ 8.004098][ T1] Call Trace:
> [ 8.005064][ T1] <TASK>
> [ 8.005501][ T1] dump_stack_lvl+0x241/0x360
> [ 8.006050][ T1] ? __pfx_dump_stack_lvl+0x10/0x10
> [ 8.006050][ T1] ? __pfx__printk+0x10/0x10
> [ 8.006050][ T1] ? _printk+0xd5/0x120
> [ 8.006050][ T1] ? vscnprintf+0x5d/0x90
> [ 8.006050][ T1] panic+0x349/0x860
> [ 8.006050][ T1] ? __warn+0x172/0x4e0
> [ 8.006050][ T1] ? __pfx_panic+0x10/0x10
> [ 8.006050][ T1] ? show_trace_log_lvl+0x4e6/0x520
> [ 8.006050][ T1] ? ret_from_fork_asm+0x1a/0x30
> [ 8.006050][ T1] __warn+0x346/0x4e0
> [ 8.006050][ T1] ? refcount_warn_saturate+0xfa/0x1d0
> [ 8.006050][ T1] report_bug+0x2b3/0x500
> [ 8.006050][ T1] ? refcount_warn_saturate+0xfa/0x1d0
> [ 8.006050][ T1] handle_bug+0x3e/0x70
> [ 8.006050][ T1] exc_invalid_op+0x1a/0x50
> [ 8.006050][ T1] asm_exc_invalid_op+0x1a/0x20
> [ 8.006050][ T1] RIP: 0010:refcount_warn_saturate+0xfa/0x1d0
> [ 8.006050][ T1] Code: b2 00 00 00 e8 e7 63 e7 fc 5b 5d c3 cc cc cc cc e8 db 63 e7 fc c6 05 6c d2 e4 0a 01 90 48 c7 c7 c0 30 1f 8c e8 77 e1 a9 fc 90 <0f> 0b 90 90 eb d9 e8 bb 63 e7 fc c6 05 49 d2 e4 0a 01 90 48 c7 c7
> [ 8.006050][ T1] RSP: 0000:ffffc90000066e18 EFLAGS: 00010246
> [ 8.006050][ T1] RAX: 7664671302135400 RBX: ffff8880202afc6c RCX: ffff8880166d0000
> [ 8.006050][ T1] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
> [ 8.006050][ T1] RBP: 0000000000000004 R08: ffffffff815880a2 R09: fffffbfff1c39b48
> [ 8.006050][ T1] R10: dffffc0000000000 R11: fffffbfff1c39b48 R12: ffffea000502ddc0
> [ 8.006050][ T1] R13: ffffea000502ddc8 R14: 1ffffd4000a05bb9 R15: 0000000000000000
> [ 8.006050][ T1] ? __warn_printk+0x292/0x360
> [ 8.006050][ T1] ? refcount_warn_saturate+0xf9/0x1d0
> [ 8.006050][ T1] __free_pages_ok+0xc60/0xd90
> [ 8.006050][ T1] make_alloc_exact+0xa3/0xf0
> [ 8.006050][ T1] vring_alloc_queue_split+0x20a/0x600
> [ 8.006050][ T1] ? __pfx_vring_alloc_queue_split+0x10/0x10
> [ 8.006050][ T1] ? vp_find_vqs+0x4c/0x4e0
> [ 8.006050][ T1] ? virtscsi_probe+0x3ea/0xf60
> [ 8.006050][ T1] ? virtio_dev_probe+0x991/0xaf0
> [ 8.006050][ T1] ? really_probe+0x2b8/0xad0
> [ 8.006050][ T1] ? driver_probe_device+0x50/0x430
> [ 8.006050][ T1] vring_create_virtqueue_split+0xc6/0x310
> [ 8.006050][ T1] ? ret_from_fork+0x4b/0x80
> [ 8.055954][ T1] ? __pfx_vring_create_virtqueue_split+0x10/0x10
> [ 8.055954][ T1] vring_create_virtqueue+0xca/0x110
> [ 8.055954][ T1] ? __pfx_vp_notify+0x10/0x10
> [ 8.055954][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
> [ 8.055954][ T1] setup_vq+0xe9/0x2d0
> [ 8.055954][ T1] ? __pfx_vp_notify+0x10/0x10
> [ 8.055954][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
> [ 8.055954][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
> [ 8.055954][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
> [ 8.055954][ T1] vp_setup_vq+0xbf/0x330
> [ 8.055954][ T1] ? __pfx_vp_config_changed+0x10/0x10
> [ 8.055954][ T1] ? ioread16+0x2f/0x90
> [ 8.055954][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
> [ 8.055954][ T1] vp_find_vqs_msix+0x8b2/0xc80
> [ 8.055954][ T1] vp_find_vqs+0x4c/0x4e0
> [ 8.055954][ T1] virtscsi_init+0x8db/0xd00
> [ 8.055954][ T1] ? __pfx_virtscsi_init+0x10/0x10
> [ 8.055954][ T1] ? __pfx_default_calc_sets+0x10/0x10
> [ 8.055954][ T1] ? scsi_host_alloc+0xa57/0xea0
> [ 8.055954][ T1] ? vp_get+0xfd/0x140
> [ 8.055954][ T1] virtscsi_probe+0x3ea/0xf60
> [ 8.055954][ T1] ? __pfx_virtscsi_probe+0x10/0x10
> [ 8.055954][ T1] ? kernfs_add_one+0x156/0x8b0
> [ 8.055954][ T1] ? virtio_no_restricted_mem_acc+0x9/0x10
> [ 8.055954][ T1] ? virtio_features_ok+0x10c/0x270
> [ 8.055954][ T1] virtio_dev_probe+0x991/0xaf0
> [ 8.055954][ T1] ? __pfx_virtio_dev_probe+0x10/0x10
> [ 8.055954][ T1] really_probe+0x2b8/0xad0
> [ 8.055954][ T1] __driver_probe_device+0x1a2/0x390
> [ 8.055954][ T1] driver_probe_device+0x50/0x430
> [ 8.055954][ T1] __driver_attach+0x45f/0x710
> [ 8.055954][ T1] ? __pfx___driver_attach+0x10/0x10
> [ 8.055954][ T1] bus_for_each_dev+0x239/0x2b0
> [ 8.055954][ T1] ? __pfx___driver_attach+0x10/0x10
> [ 8.055954][ T1] ? __pfx_bus_for_each_dev+0x10/0x10
> [ 8.055954][ T1] ? do_raw_spin_unlock+0x13c/0x8b0
> [ 8.055954][ T1] bus_add_driver+0x347/0x620
> [ 8.055954][ T1] driver_register+0x23a/0x320
> [ 8.055954][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
> [ 8.055954][ T1] virtio_scsi_init+0x65/0xe0
> [ 8.055954][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
> [ 8.055954][ T1] do_one_initcall+0x248/0x880
> [ 8.055954][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
> [ 8.055954][ T1] ? __pfx_do_one_initcall+0x10/0x10
> [ 8.105903][ T1] ? lockdep_hardirqs_on_prepare+0x43d/0x780
> [ 8.105903][ T1] ? __pfx_parse_args+0x10/0x10
> [ 8.105903][ T1] ? do_initcalls+0x1c/0x80
> [ 8.105903][ T1] ? rcu_is_watching+0x15/0xb0
> [ 8.105903][ T1] do_initcall_level+0x157/0x210
> [ 8.105903][ T1] do_initcalls+0x3f/0x80
> [ 8.105903][ T1] kernel_init_freeable+0x435/0x5d0
> [ 8.105903][ T1] ? __pfx_kernel_init_freeable+0x10/0x10
> [ 8.105903][ T1] ? __pfx_lockdep_hardirqs_on_prepare+0x10/0x10
> [ 8.105903][ T1] ? __pfx_kernel_init+0x10/0x10
> [ 8.105903][ T1] ? __pfx_kernel_init+0x10/0x10
> [ 8.105903][ T1] ? __pfx_kernel_init+0x10/0x10
> [ 8.105903][ T1] kernel_init+0x1d/0x2b0
> [ 8.105903][ T1] ret_from_fork+0x4b/0x80
> [ 8.105903][ T1] ? __pfx_kernel_init+0x10/0x10
> [ 8.105903][ T1] ret_from_fork_asm+0x1a/0x30
> [ 8.105903][ T1] </TASK>
> [ 8.105903][ T1] Kernel Offset: disabled
> [ 8.105903][ T1] Rebooting in 86400 seconds..
>
>
> syzkaller build log:
> go env (err=<nil>)
> GO111MODULE='auto'
> GOARCH='amd64'
> GOBIN=''
> GOCACHE='/syzkaller/.cache/go-build'
> GOENV='/syzkaller/.config/go/env'
> GOEXE=''
> GOEXPERIMENT=''
> GOFLAGS=''
> GOHOSTARCH='amd64'
> GOHOSTOS='linux'
> GOINSECURE=''
> GOMODCACHE='/syzkaller/jobs-2/linux/gopath/pkg/mod'
> GONOPROXY=''
> GONOSUMDB=''
> GOOS='linux'
> GOPATH='/syzkaller/jobs-2/linux/gopath'
> GOPRIVATE=''
> GOPROXY='https://proxy.golang.org,direct'
> GOROOT='/usr/local/go'
> GOSUMDB='sum.golang.org'
> GOTMPDIR=''
> GOTOOLCHAIN='auto'
> GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
> GOVCS=''
> GOVERSION='go1.21.4'
> GCCGO='gccgo'
> GOAMD64='v1'
> AR='ar'
> CC='gcc'
> CXX='g++'
> CGO_ENABLED='1'
> GOMOD='/syzkaller/jobs-2/linux/gopath/src/github.com/google/syzkaller/go.mod'
> GOWORK=''
> CGO_CFLAGS='-O2 -g'
> CGO_CPPFLAGS=''
> CGO_CXXFLAGS='-O2 -g'
> CGO_FFLAGS='-O2 -g'
> CGO_LDFLAGS='-O2 -g'
> PKG_CONFIG='pkg-config'
> GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1668308411=/tmp/go-build -gno-record-gcc-switches'
>
> git status (err=<nil>)
> HEAD detached at 56086b24b
> nothing to commit, working tree clean
>
>
> tput: No value for $TERM and no -T specified
> tput: No value for $TERM and no -T specified
> Makefile:31: run command via tools/syz-env for best compatibility, see:
> Makefile:32: https://github.com/google/syzkaller/blob/master/docs/contributing.md#using-syz-env
> go list -f '{{.Stale}}' ./sys/syz-sysgen | grep -q false || go install ./sys/syz-sysgen
> make .descriptions
> tput: No value for $TERM and no -T specified
> tput: No value for $TERM and no -T specified
> Makefile:31: run command via tools/syz-env for best compatibility, see:
> Makefile:32: https://github.com/google/syzkaller/blob/master/docs/contributing.md#using-syz-env
> bin/syz-sysgen
> touch .descriptions
> GOOS=linux GOARCH=amd64 go build "-ldflags=-s -w -X github.com/google/syzkaller/prog.GitRevision=56086b24bdfd822d3b227edb3064db443cd8c971 -X 'github.com/google/syzkaller/prog.gitRevisionDate=20240409-083312'" "-tags=syz_target syz_os_linux syz_arch_amd64 " -o ./bin/linux_amd64/syz-fuzzer github.com/google/syzkaller/syz-fuzzer
> GOOS=linux GOARCH=amd64 go build "-ldflags=-s -w -X github.com/google/syzkaller/prog.GitRevision=56086b24bdfd822d3b227edb3064db443cd8c971 -X 'github.com/google/syzkaller/prog.gitRevisionDate=20240409-083312'" "-tags=syz_target syz_os_linux syz_arch_amd64 " -o ./bin/linux_amd64/syz-execprog github.com/google/syzkaller/tools/syz-execprog
> GOOS=linux GOARCH=amd64 go build "-ldflags=-s -w -X github.com/google/syzkaller/prog.GitRevision=56086b24bdfd822d3b227edb3064db443cd8c971 -X 'github.com/google/syzkaller/prog.gitRevisionDate=20240409-083312'" "-tags=syz_target syz_os_linux syz_arch_amd64 " -o ./bin/linux_amd64/syz-stress github.com/google/syzkaller/tools/syz-stress
> mkdir -p ./bin/linux_amd64
> gcc -o ./bin/linux_amd64/syz-executor executor/executor.cc \
> -m64 -O2 -pthread -Wall -Werror -Wparentheses -Wunused-const-variable -Wframe-larger-than=16384 -Wno-stringop-overflow -Wno-array-bounds -Wno-format-overflow -Wno-unused-but-set-variable -Wno-unused-command-line-argument -static-pie -fpermissive -w -DGOOS_linux=1 -DGOARCH_amd64=1 \
> -DHOSTGOOS_linux=1 -DGIT_REVISION=\"56086b24bdfd822d3b227edb3064db443cd8c971\"
>
>
> Error text is too large and was truncated, full error text is at:
> https://syzkaller.appspot.com/x/error.txt?x=122f8f97180000
>
>
> Tested on:
>
> commit: a4170c00 fsnotify: do not handle events on a shutting ..
> git tree: https://github.com/amir73il/linux fsnotify-fixes
> kernel config: https://syzkaller.appspot.com/x/.config?x=9995779c8305f57e
> dashboard link: https://syzkaller.appspot.com/bug?extid=5e3f9b2a67b45f16d4e6
> compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
>
> Note: no patches were applied.
>
> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/000000000000774dfe0615e71b29%40google.com.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [syzbot] [ext4?] KASAN: slab-use-after-free Read in fsnotify
2024-04-11 8:11 syzbot
2024-04-11 12:13 ` Jan Kara
@ 2024-04-13 1:40 ` Hillf Danton
2024-04-13 6:42 ` Amir Goldstein
2024-04-13 8:40 ` syzbot
1 sibling, 2 replies; 26+ messages in thread
From: Hillf Danton @ 2024-04-13 1:40 UTC (permalink / raw)
To: syzbot; +Cc: linux-fsdevel, syzkaller-bugs
On Thu, 11 Apr 2024 01:11:20 -0700
> syzbot found the following issue on:
>
> HEAD commit: 6ebf211bb11d Add linux-next specific files for 20240410
> git tree: linux-next
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=1621af9d180000
#syz test https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git 6ebf211bb11d
--- x/fs/notify/fsnotify.c
+++ y/fs/notify/fsnotify.c
@@ -101,8 +101,8 @@ void fsnotify_sb_delete(struct super_blo
wait_var_event(fsnotify_sb_watched_objects(sb),
!atomic_long_read(fsnotify_sb_watched_objects(sb)));
WARN_ON(fsnotify_sb_has_priority_watchers(sb, FSNOTIFY_PRIO_CONTENT));
- WARN_ON(fsnotify_sb_has_priority_watchers(sb,
- FSNOTIFY_PRIO_PRE_CONTENT));
+ WARN_ON(fsnotify_sb_has_priority_watchers(sb, FSNOTIFY_PRIO_PRE_CONTENT));
+ synchronize_srcu(&fsnotify_mark_srcu);
kfree(sbinfo);
}
@@ -499,7 +499,7 @@ int fsnotify(__u32 mask, const void *dat
{
const struct path *path = fsnotify_data_path(data, data_type);
struct super_block *sb = fsnotify_data_sb(data, data_type);
- struct fsnotify_sb_info *sbinfo = fsnotify_sb_info(sb);
+ struct fsnotify_sb_info *sbinfo;
struct fsnotify_iter_info iter_info = {};
struct mount *mnt = NULL;
struct inode *inode2 = NULL;
@@ -529,6 +529,8 @@ int fsnotify(__u32 mask, const void *dat
inode2_type = FSNOTIFY_ITER_TYPE_PARENT;
}
+ iter_info.srcu_idx = srcu_read_lock(&fsnotify_mark_srcu);
+ sbinfo = fsnotify_sb_info(sb);
/*
* Optimization: srcu_read_lock() has a memory barrier which can
* be expensive. It protects walking the *_fsnotify_marks lists.
@@ -539,8 +541,10 @@ int fsnotify(__u32 mask, const void *dat
if ((!sbinfo || !sbinfo->sb_marks) &&
(!mnt || !mnt->mnt_fsnotify_marks) &&
(!inode || !inode->i_fsnotify_marks) &&
- (!inode2 || !inode2->i_fsnotify_marks))
- return 0;
+ (!inode2 || !inode2->i_fsnotify_marks)) {
+ ret = 0;
+ goto out;
+ }
marks_mask = sb->s_fsnotify_mask;
if (mnt)
@@ -558,10 +562,10 @@ int fsnotify(__u32 mask, const void *dat
* Otherwise, return if none of the marks care about this type of event.
*/
test_mask = (mask & ALL_FSNOTIFY_EVENTS);
- if (!(test_mask & marks_mask))
- return 0;
-
- iter_info.srcu_idx = srcu_read_lock(&fsnotify_mark_srcu);
+ if (!(test_mask & marks_mask)) {
+ ret = 0;
+ goto out;
+ }
if (sbinfo) {
iter_info.marks[FSNOTIFY_ITER_TYPE_SB] =
--
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [syzbot] [ext4?] KASAN: slab-use-after-free Read in fsnotify
2024-04-13 1:40 ` Hillf Danton
@ 2024-04-13 6:42 ` Amir Goldstein
2024-04-13 8:58 ` syzbot
2024-04-13 8:40 ` syzbot
1 sibling, 1 reply; 26+ messages in thread
From: Amir Goldstein @ 2024-04-13 6:42 UTC (permalink / raw)
To: Hillf Danton; +Cc: syzbot, linux-fsdevel, syzkaller-bugs
[-- Attachment #1: Type: text/plain, Size: 2675 bytes --]
On Sat, Apr 13, 2024 at 4:41 AM Hillf Danton <hdanton@sina.com> wrote:
>
> On Thu, 11 Apr 2024 01:11:20 -0700
> > syzbot found the following issue on:
> >
> > HEAD commit: 6ebf211bb11d Add linux-next specific files for 20240410
> > git tree: linux-next
> > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=1621af9d180000
>
> #syz test https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git 6ebf211bb11d
>
> --- x/fs/notify/fsnotify.c
> +++ y/fs/notify/fsnotify.c
> @@ -101,8 +101,8 @@ void fsnotify_sb_delete(struct super_blo
> wait_var_event(fsnotify_sb_watched_objects(sb),
> !atomic_long_read(fsnotify_sb_watched_objects(sb)));
> WARN_ON(fsnotify_sb_has_priority_watchers(sb, FSNOTIFY_PRIO_CONTENT));
> - WARN_ON(fsnotify_sb_has_priority_watchers(sb,
> - FSNOTIFY_PRIO_PRE_CONTENT));
> + WARN_ON(fsnotify_sb_has_priority_watchers(sb, FSNOTIFY_PRIO_PRE_CONTENT));
> + synchronize_srcu(&fsnotify_mark_srcu);
> kfree(sbinfo);
> }
>
> @@ -499,7 +499,7 @@ int fsnotify(__u32 mask, const void *dat
> {
> const struct path *path = fsnotify_data_path(data, data_type);
> struct super_block *sb = fsnotify_data_sb(data, data_type);
> - struct fsnotify_sb_info *sbinfo = fsnotify_sb_info(sb);
> + struct fsnotify_sb_info *sbinfo;
> struct fsnotify_iter_info iter_info = {};
> struct mount *mnt = NULL;
> struct inode *inode2 = NULL;
> @@ -529,6 +529,8 @@ int fsnotify(__u32 mask, const void *dat
> inode2_type = FSNOTIFY_ITER_TYPE_PARENT;
> }
>
> + iter_info.srcu_idx = srcu_read_lock(&fsnotify_mark_srcu);
> + sbinfo = fsnotify_sb_info(sb);
> /*
> * Optimization: srcu_read_lock() has a memory barrier which can
> * be expensive. It protects walking the *_fsnotify_marks lists.
See comment above. This kills the optimization.
It is not worth letting all the fsnotify hooks suffer the consequence
for the edge case of calling fsnotify hook during fs shutdown.
Also, fsnotify_sb_info(sb) in fsnotify_sb_has_priority_watchers()
is also not protected and using srcu_read_lock() there completely
nullifies the purpose of fsnotify_sb_info.
Here is a simplified fix for fsnotify_sb_error() rebased on the
pending mm fixes for this syzbot boot failure:
#syz test: https://github.com/amir73il/linux fsnotify-fixes
Jan,
I think that all the functions called from fs shutdown context
should observe that SB_ACTIVE is cleared but wasn't sure?
Thanks,
Amir.
[-- Attachment #2: 0001-fsnotify-do-not-handle-events-on-a-shutting-down-fil.patch --]
[-- Type: text/x-patch, Size: 1096 bytes --]
From 9e5897865c4ba8296a81f451d2463dbd6b49fb3c Mon Sep 17 00:00:00 2001
From: Amir Goldstein <amir73il@gmail.com>
Date: Thu, 11 Apr 2024 18:59:08 +0300
Subject: [PATCH] fsnotify: do not handle events on a shutting down filesystem
Protect against use after free when filesystem calls fsnotify_sb_error()
during fs shutdown.
Reported-by: syzbot+5e3f9b2a67b45f16d4e6@syzkaller.appspotmail.com
Fixes: 07a3b8d0bf72 ("fsnotify: lazy attach fsnotify_sb_info state to sb")
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
include/linux/fsnotify.h | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/include/linux/fsnotify.h b/include/linux/fsnotify.h
index 4da80e92f804..66512a965824 100644
--- a/include/linux/fsnotify.h
+++ b/include/linux/fsnotify.h
@@ -453,6 +453,10 @@ static inline int fsnotify_sb_error(struct super_block *sb, struct inode *inode,
.sb = sb,
};
+ /* fs may be calling this hook from without shutdown */
+ if (unlikely(!(sb->s_flags & SB_ACTIVE)))
+ return 0;
+
return fsnotify(FS_ERROR, &report, FSNOTIFY_EVENT_ERROR,
NULL, NULL, NULL, 0);
}
--
2.34.1
^ permalink raw reply related [flat|nested] 26+ messages in thread
* Re: [syzbot] [ext4?] KASAN: slab-use-after-free Read in fsnotify
2024-04-13 1:40 ` Hillf Danton
2024-04-13 6:42 ` Amir Goldstein
@ 2024-04-13 8:40 ` syzbot
1 sibling, 0 replies; 26+ messages in thread
From: syzbot @ 2024-04-13 8:40 UTC (permalink / raw)
To: hdanton, linux-fsdevel, linux-kernel, syzkaller-bugs
Hello,
syzbot has tested the proposed patch but the reproducer is still triggering an issue:
KASAN: slab-use-after-free Read in fsnotify
Quota error (device loop0): do_check_range: Getting block 0 out of range 1-5
EXT4-fs error (device loop0): ext4_release_dquot:6905: comm kworker/u8:4: Failed to release dquot type 1
==================================================================
BUG: KASAN: slab-use-after-free in fsnotify+0x3e5/0x1f00 fs/notify/fsnotify.c:541
Read of size 8 at addr ffff88802d133300 by task kworker/u8:4/62
CPU: 0 PID: 62 Comm: kworker/u8:4 Not tainted 6.9.0-rc3-next-20240410-syzkaller-dirty #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
Workqueue: events_unbound quota_release_workfn
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x241/0x360 lib/dump_stack.c:114
print_address_description mm/kasan/report.c:377 [inline]
print_report+0x169/0x550 mm/kasan/report.c:488
kasan_report+0x143/0x180 mm/kasan/report.c:601
fsnotify+0x3e5/0x1f00 fs/notify/fsnotify.c:541
fsnotify_sb_error include/linux/fsnotify.h:456 [inline]
__ext4_error+0x255/0x3b0 fs/ext4/super.c:843
ext4_release_dquot+0x326/0x450 fs/ext4/super.c:6903
quota_release_workfn+0x39f/0x650 fs/quota/dquot.c:840
process_one_work kernel/workqueue.c:3218 [inline]
process_scheduled_works+0xa2c/0x1830 kernel/workqueue.c:3299
worker_thread+0x86d/0xd70 kernel/workqueue.c:3380
kthread+0x2f0/0x390 kernel/kthread.c:389
ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
</TASK>
Allocated by task 5511:
kasan_save_stack mm/kasan/common.c:47 [inline]
kasan_save_track+0x3f/0x80 mm/kasan/common.c:68
poison_kmalloc_redzone mm/kasan/common.c:370 [inline]
__kasan_kmalloc+0x98/0xb0 mm/kasan/common.c:387
kasan_kmalloc include/linux/kasan.h:211 [inline]
kmalloc_trace_noprof+0x19c/0x2b0 mm/slub.c:4109
kmalloc_noprof include/linux/slab.h:660 [inline]
kzalloc_noprof include/linux/slab.h:775 [inline]
fsnotify_attach_info_to_sb fs/notify/mark.c:600 [inline]
fsnotify_add_mark_list fs/notify/mark.c:692 [inline]
fsnotify_add_mark_locked+0x3b2/0xe60 fs/notify/mark.c:777
fanotify_add_new_mark fs/notify/fanotify/fanotify_user.c:1267 [inline]
fanotify_add_mark+0xbbd/0x1330 fs/notify/fanotify/fanotify_user.c:1334
do_fanotify_mark+0xbcc/0xd90 fs/notify/fanotify/fanotify_user.c:1896
__do_sys_fanotify_mark fs/notify/fanotify/fanotify_user.c:1919 [inline]
__se_sys_fanotify_mark fs/notify/fanotify/fanotify_user.c:1915 [inline]
__x64_sys_fanotify_mark+0xb5/0xd0 fs/notify/fanotify/fanotify_user.c:1915
do_syscall_x64 arch/x86/entry/common.c:52 [inline]
do_syscall_64+0xf5/0x240 arch/x86/entry/common.c:83
entry_SYSCALL_64_after_hwframe+0x77/0x7f
Freed by task 5442:
kasan_save_stack mm/kasan/common.c:47 [inline]
kasan_save_track+0x3f/0x80 mm/kasan/common.c:68
kasan_save_free_info+0x40/0x50 mm/kasan/generic.c:579
poison_slab_object+0xe0/0x150 mm/kasan/common.c:240
__kasan_slab_free+0x37/0x60 mm/kasan/common.c:256
kasan_slab_free include/linux/kasan.h:184 [inline]
slab_free_hook mm/slub.c:2190 [inline]
slab_free mm/slub.c:4393 [inline]
kfree+0x149/0x350 mm/slub.c:4514
fsnotify_sb_delete+0x692/0x6f0 fs/notify/fsnotify.c:106
generic_shutdown_super+0xa5/0x2d0 fs/super.c:632
kill_block_super+0x44/0x90 fs/super.c:1675
ext4_kill_sb+0x68/0xa0 fs/ext4/super.c:7323
deactivate_locked_super+0xc4/0x130 fs/super.c:472
cleanup_mnt+0x426/0x4c0 fs/namespace.c:1267
task_work_run+0x24f/0x310 kernel/task_work.c:180
resume_user_mode_work include/linux/resume_user_mode.h:50 [inline]
exit_to_user_mode_loop kernel/entry/common.c:114 [inline]
exit_to_user_mode_prepare include/linux/entry-common.h:328 [inline]
__syscall_exit_to_user_mode_work kernel/entry/common.c:207 [inline]
syscall_exit_to_user_mode+0x168/0x370 kernel/entry/common.c:218
do_syscall_64+0x102/0x240 arch/x86/entry/common.c:89
entry_SYSCALL_64_after_hwframe+0x77/0x7f
The buggy address belongs to the object at ffff88802d133300
which belongs to the cache kmalloc-32 of size 32
The buggy address is located 0 bytes inside of
freed 32-byte region [ffff88802d133300, ffff88802d133320)
The buggy address belongs to the physical page:
page: refcount:1 mapcount:0 mapping:0000000000000000 index:0xffff88802d133180 pfn:0x2d133
flags: 0xfff80000000200(workingset|node=0|zone=1|lastcpupid=0xfff)
page_type: 0xffffefff(slab)
raw: 00fff80000000200 ffff888015041500 ffffea00007c7590 ffffea0001aa96d0
raw: ffff88802d133180 000000000040003d 00000001ffffefff 0000000000000000
page dumped because: kasan: bad access detected
page_owner tracks the page as allocated
page last allocated via order 0, migratetype Unmovable, gfp_mask 0x12cc0(GFP_KERNEL|__GFP_NOWARN|__GFP_NORETRY), pid 1, tgid 177503641 (swapper/0), ts 1, free_ts 13042593494
set_page_owner include/linux/page_owner.h:32 [inline]
post_alloc_hook+0x1f3/0x230 mm/page_alloc.c:1468
prep_new_page mm/page_alloc.c:1476 [inline]
get_page_from_freelist+0x2ce2/0x2d90 mm/page_alloc.c:3438
__alloc_pages_noprof+0x256/0x6c0 mm/page_alloc.c:4696
__alloc_pages_node_noprof include/linux/gfp.h:244 [inline]
alloc_pages_node_noprof include/linux/gfp.h:271 [inline]
alloc_slab_page+0x5f/0x120 mm/slub.c:2259
allocate_slab+0x5a/0x2e0 mm/slub.c:2422
new_slab mm/slub.c:2475 [inline]
___slab_alloc+0xcd1/0x14b0 mm/slub.c:3624
__slab_alloc+0x58/0xa0 mm/slub.c:3714
__slab_alloc_node mm/slub.c:3767 [inline]
slab_alloc_node mm/slub.c:3945 [inline]
__do_kmalloc_node mm/slub.c:4077 [inline]
kmalloc_node_track_caller_noprof+0x286/0x440 mm/slub.c:4098
__do_krealloc mm/slab_common.c:1190 [inline]
krealloc_noprof+0x7d/0x120 mm/slab_common.c:1223
add_sysfs_param+0x137/0x7f0 kernel/params.c:663
kernel_add_sysfs_param+0xb4/0x130 kernel/params.c:817
param_sysfs_builtin+0x16e/0x1f0 kernel/params.c:856
param_sysfs_builtin_init+0x31/0x40 kernel/params.c:990
do_one_initcall+0x248/0x880 init/main.c:1263
do_initcall_level+0x157/0x210 init/main.c:1325
do_initcalls+0x3f/0x80 init/main.c:1341
page last free pid 782 tgid 782 stack trace:
reset_page_owner include/linux/page_owner.h:25 [inline]
free_pages_prepare mm/page_alloc.c:1088 [inline]
free_unref_page+0xd22/0xea0 mm/page_alloc.c:2601
vfree+0x186/0x2e0 mm/vmalloc.c:3338
delayed_vfree_work+0x56/0x80 mm/vmalloc.c:3259
process_one_work kernel/workqueue.c:3218 [inline]
process_scheduled_works+0xa2c/0x1830 kernel/workqueue.c:3299
worker_thread+0x86d/0xd70 kernel/workqueue.c:3380
kthread+0x2f0/0x390 kernel/kthread.c:389
ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
Memory state around the buggy address:
ffff88802d133200: 00 00 00 00 fc fc fc fc 00 00 00 00 fc fc fc fc
ffff88802d133280: 00 00 00 00 fc fc fc fc 00 00 00 00 fc fc fc fc
>ffff88802d133300: fa fb fb fb fc fc fc fc 00 00 00 00 fc fc fc fc
^
ffff88802d133380: 00 00 00 fc fc fc fc fc 00 00 00 04 fc fc fc fc
ffff88802d133400: 00 00 00 fc fc fc fc fc 00 00 00 00 fc fc fc fc
==================================================================
Tested on:
commit: 6ebf211b Add linux-next specific files for 20240410
git tree: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
console output: https://syzkaller.appspot.com/x/log.txt?x=10c3ac7d180000
kernel config: https://syzkaller.appspot.com/x/.config?x=16ca158ef7e08662
dashboard link: https://syzkaller.appspot.com/bug?extid=5e3f9b2a67b45f16d4e6
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
patch: https://syzkaller.appspot.com/x/patch.diff?x=165b0243180000
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [syzbot] Re: [syzbot] [ext4?] KASAN: slab-use-after-free Read in fsnotify
[not found] <00000000000095bb400615f4b0ed@google.com>
@ 2024-04-13 8:45 ` Hillf Danton
2024-04-13 9:32 ` Amir Goldstein
0 siblings, 1 reply; 26+ messages in thread
From: Hillf Danton @ 2024-04-13 8:45 UTC (permalink / raw)
To: amir73il; +Cc: syzbot, linux-fsdevel, syzkaller-bugs, linux-kernel
On Fri, 12 Apr 2024 23:42:19 -0700 Amir Goldstein
> On Sat, Apr 13, 2024 at 4:41=E2=80=AFAM Hillf Danton <hdanton@sina.com> wrote:
> >
> > On Thu, 11 Apr 2024 01:11:20 -0700
> > > syzbot found the following issue on:
> > >
> > > HEAD commit: 6ebf211bb11d Add linux-next specific files for 20240410
> > > git tree: linux-next
> > > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=3D1621af9d180000
> >
> > #syz test https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git 6ebf211bb11d
> >
> > --- x/fs/notify/fsnotify.c
> > +++ y/fs/notify/fsnotify.c
> > @@ -101,8 +101,8 @@ void fsnotify_sb_delete(struct super_blo
> > wait_var_event(fsnotify_sb_watched_objects(sb),
> > !atomic_long_read(fsnotify_sb_watched_objects(sb)));
> > WARN_ON(fsnotify_sb_has_priority_watchers(sb, FSNOTIFY_PRIO_CONTENT));
> > - WARN_ON(fsnotify_sb_has_priority_watchers(sb,
> > - FSNOTIFY_PRIO_PRE_CONTENT));
> > + WARN_ON(fsnotify_sb_has_priority_watchers(sb, FSNOTIFY_PRIO_PRE_CONTENT));
> > + synchronize_srcu(&fsnotify_mark_srcu);
> > kfree(sbinfo);
> > }
> >
> > @@ -499,7 +499,7 @@ int fsnotify(__u32 mask, const void *dat
> > {
> > const struct path *path =3D fsnotify_data_path(data, data_type);
> > struct super_block *sb =3D fsnotify_data_sb(data, data_type);
> > - struct fsnotify_sb_info *sbinfo =3D fsnotify_sb_info(sb);
> > + struct fsnotify_sb_info *sbinfo;
> > struct fsnotify_iter_info iter_info = {};
> > struct mount *mnt =3D NULL;
> > struct inode *inode2 =3D NULL;
> > @@ -529,6 +529,8 @@ int fsnotify(__u32 mask, const void *dat
> > inode2_type =3D FSNOTIFY_ITER_TYPE_PARENT;
> > }
> >
> > + iter_info.srcu_idx =3D srcu_read_lock(&fsnotify_mark_srcu);
> > + sbinfo =3D fsnotify_sb_info(sb);
> > /*
> > * Optimization: srcu_read_lock() has a memory barrier which can
> > * be expensive. It protects walking the *_fsnotify_marks lists.
>
>
> See comment above. This kills the optimization.
> It is not worth letting all the fsnotify hooks suffer the consequence
> for the edge case of calling fsnotify hook during fs shutdown.
Say nothing before reading your fix.
>
> Also, fsnotify_sb_info(sb) in fsnotify_sb_has_priority_watchers()
> is also not protected and using srcu_read_lock() there completely
> nullifies the purpose of fsnotify_sb_info.
>
> Here is a simplified fix for fsnotify_sb_error() rebased on the
> pending mm fixes for this syzbot boot failure:
>
> #syz test: https://github.com/amir73il/linux fsnotify-fixes
Feel free to post your patch at lore because not everyone has
access to sites like github.
>
> Jan,
>
> I think that all the functions called from fs shutdown context
> should observe that SB_ACTIVE is cleared but wasn't sure?
If you composed fix based on SB_ACTIVE that is cleared in
generic_shutdown_super() with &sb->s_umount held for write,
I wonder what simpler serialization than srcu you could
find/create in fsnotify.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [syzbot] [ext4?] KASAN: slab-use-after-free Read in fsnotify
2024-04-13 6:42 ` Amir Goldstein
@ 2024-04-13 8:58 ` syzbot
0 siblings, 0 replies; 26+ messages in thread
From: syzbot @ 2024-04-13 8:58 UTC (permalink / raw)
To: amir73il, hdanton, linux-fsdevel, linux-kernel, syzkaller-bugs
Hello,
syzbot has tested the proposed patch and the reproducer did not trigger any issue:
Reported-and-tested-by: syzbot+5e3f9b2a67b45f16d4e6@syzkaller.appspotmail.com
Tested on:
commit: 9e589786 fsnotify: do not handle events on a shutting ..
git tree: https://github.com/amir73il/linux fsnotify-fixes
console output: https://syzkaller.appspot.com/x/log.txt?x=11cdb161180000
kernel config: https://syzkaller.appspot.com/x/.config?x=9995779c8305f57e
dashboard link: https://syzkaller.appspot.com/bug?extid=5e3f9b2a67b45f16d4e6
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
patch: https://syzkaller.appspot.com/x/patch.diff?x=14f52d33180000
Note: testing is done by a robot and is best-effort only.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [syzbot] Re: [syzbot] [ext4?] KASAN: slab-use-after-free Read in fsnotify
2024-04-13 8:45 ` [syzbot] Re: [syzbot] [ext4?] KASAN: slab-use-after-free Read in fsnotify Hillf Danton
@ 2024-04-13 9:32 ` Amir Goldstein
2024-04-13 12:04 ` Hillf Danton
2024-04-15 14:03 ` [syzbot] " Jan Kara
0 siblings, 2 replies; 26+ messages in thread
From: Amir Goldstein @ 2024-04-13 9:32 UTC (permalink / raw)
To: Hillf Danton, Jan Kara
Cc: syzbot, linux-fsdevel, syzkaller-bugs, linux-kernel
On Sat, Apr 13, 2024 at 11:45 AM Hillf Danton <hdanton@sina.com> wrote:
>
> On Fri, 12 Apr 2024 23:42:19 -0700 Amir Goldstein
> > On Sat, Apr 13, 2024 at 4:41=E2=80=AFAM Hillf Danton <hdanton@sina.com> wrote:
> > >
> > > On Thu, 11 Apr 2024 01:11:20 -0700
> > > > syzbot found the following issue on:
> > > >
> > > > HEAD commit: 6ebf211bb11d Add linux-next specific files for 20240410
> > > > git tree: linux-next
> > > > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=3D1621af9d180000
> > >
> > > #syz test https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git 6ebf211bb11d
> > >
> > > --- x/fs/notify/fsnotify.c
> > > +++ y/fs/notify/fsnotify.c
> > > @@ -101,8 +101,8 @@ void fsnotify_sb_delete(struct super_blo
> > > wait_var_event(fsnotify_sb_watched_objects(sb),
> > > !atomic_long_read(fsnotify_sb_watched_objects(sb)));
> > > WARN_ON(fsnotify_sb_has_priority_watchers(sb, FSNOTIFY_PRIO_CONTENT));
> > > - WARN_ON(fsnotify_sb_has_priority_watchers(sb,
> > > - FSNOTIFY_PRIO_PRE_CONTENT));
> > > + WARN_ON(fsnotify_sb_has_priority_watchers(sb, FSNOTIFY_PRIO_PRE_CONTENT));
> > > + synchronize_srcu(&fsnotify_mark_srcu);
> > > kfree(sbinfo);
> > > }
> > >
> > > @@ -499,7 +499,7 @@ int fsnotify(__u32 mask, const void *dat
> > > {
> > > const struct path *path =3D fsnotify_data_path(data, data_type);
> > > struct super_block *sb =3D fsnotify_data_sb(data, data_type);
> > > - struct fsnotify_sb_info *sbinfo =3D fsnotify_sb_info(sb);
> > > + struct fsnotify_sb_info *sbinfo;
> > > struct fsnotify_iter_info iter_info = {};
> > > struct mount *mnt =3D NULL;
> > > struct inode *inode2 =3D NULL;
> > > @@ -529,6 +529,8 @@ int fsnotify(__u32 mask, const void *dat
> > > inode2_type =3D FSNOTIFY_ITER_TYPE_PARENT;
> > > }
> > >
> > > + iter_info.srcu_idx =3D srcu_read_lock(&fsnotify_mark_srcu);
> > > + sbinfo =3D fsnotify_sb_info(sb);
> > > /*
> > > * Optimization: srcu_read_lock() has a memory barrier which can
> > > * be expensive. It protects walking the *_fsnotify_marks lists.
> >
> >
> > See comment above. This kills the optimization.
> > It is not worth letting all the fsnotify hooks suffer the consequence
> > for the edge case of calling fsnotify hook during fs shutdown.
>
> Say nothing before reading your fix.
> >
> > Also, fsnotify_sb_info(sb) in fsnotify_sb_has_priority_watchers()
> > is also not protected and using srcu_read_lock() there completely
> > nullifies the purpose of fsnotify_sb_info.
> >
> > Here is a simplified fix for fsnotify_sb_error() rebased on the
> > pending mm fixes for this syzbot boot failure:
> >
> > #syz test: https://github.com/amir73il/linux fsnotify-fixes
>
> Feel free to post your patch at lore because not everyone has
> access to sites like github.
> >
> > Jan,
> >
> > I think that all the functions called from fs shutdown context
> > should observe that SB_ACTIVE is cleared but wasn't sure?
>
> If you composed fix based on SB_ACTIVE that is cleared in
> generic_shutdown_super() with &sb->s_umount held for write,
> I wonder what simpler serialization than srcu you could
> find/create in fsnotify.
As far as I can tell there is no need for serialisation.
The problem is that fsnotify_sb_error() can be called from the
context of ->put_super() call from generic_shutdown_super().
It's true that in the repro the thread calling fsnotify_sb_error()
in the worker thread running quota deferred work from put_super()
but I think there are sufficient barriers for this worker thread to
observer the cleared SB_ACTIVE flag.
Anyway, according to syzbot, repro does not trigger the UAF
with my last fix.
To be clear, any fsnotify_sb_error() that is a result of a user operation
would be holding an active reference to sb so cannot race with
fsnotify_sb_delete(), but I am not sure that same is true for ext4
worker threads.
Jan,
You wrote that "In theory these two calls can even run in parallel
and fsnotify() can be holding fsnotify_sb_info pointer while
fsnotify_sb_delete() is freeing".
Can you give an example of this case?
Thanks,
Amir.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [syzbot] [ext4?] KASAN: slab-use-after-free Read in fsnotify
2024-04-13 9:32 ` Amir Goldstein
@ 2024-04-13 12:04 ` Hillf Danton
2024-04-15 14:03 ` [syzbot] " Jan Kara
1 sibling, 0 replies; 26+ messages in thread
From: Hillf Danton @ 2024-04-13 12:04 UTC (permalink / raw)
To: Amir Goldstein
Cc: Jan Kara, syzbot, syzkaller-bugs, linux-fsdevel, linux-kernel
On Sat, 13 Apr 2024 12:32:32 +0300 Amir Goldstein
> On Sat, Apr 13, 2024 at 11:45=E2=80=AFAM Hillf Danton <hdanton@sina.com> wrote:
> >
> > If you composed fix based on SB_ACTIVE that is cleared in
> > generic_shutdown_super() with &sb->s_umount held for write,
> > I wonder what simpler serialization than srcu you could
> > find/create in fsnotify.
>
> As far as I can tell there is no need for serialisation.
>
> The problem is that fsnotify_sb_error() can be called from the
> context of ->put_super() call from generic_shutdown_super().
>
> It's true that in the repro the thread calling fsnotify_sb_error()
> in the worker thread running quota deferred work from put_super()
> but I think there are sufficient barriers for this worker thread to
> observer the cleared SB_ACTIVE flag.
>
do_exit quota_release_workfn
--- ---
cleanup_mnt() ext4_release_dquot()
__super_lock_excl(s); __ext4_error()
deactivate_locked_super(s); fsnotify_sb_error()
ext4_kill_sb()
kill_block_super()
generic_shutdown_super()
if (!(sb->s_flags & SB_ACTIVE))
return;
sb->s_flags &= ~SB_ACTIVE;
fsnotify_sb_delete()
fsnotify()
Because of no sync like taking &sb->s_umount in the worker context,
checking SB_ACTIVE added in your fix is unable to close the race.
> Anyway, according to syzbot, repro does not trigger the UAF
> with my last fix.
>
Note: testing is done by a robot and is best-effort only.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [syzbot] Re: [syzbot] [ext4?] KASAN: slab-use-after-free Read in fsnotify
2024-04-13 9:32 ` Amir Goldstein
2024-04-13 12:04 ` Hillf Danton
@ 2024-04-15 14:03 ` Jan Kara
2024-04-15 14:47 ` Amir Goldstein
1 sibling, 1 reply; 26+ messages in thread
From: Jan Kara @ 2024-04-15 14:03 UTC (permalink / raw)
To: Amir Goldstein
Cc: Hillf Danton, Jan Kara, syzbot, linux-fsdevel, syzkaller-bugs,
linux-kernel
On Sat 13-04-24 12:32:32, Amir Goldstein wrote:
> On Sat, Apr 13, 2024 at 11:45 AM Hillf Danton <hdanton@sina.com> wrote:
> > On Fri, 12 Apr 2024 23:42:19 -0700 Amir Goldstein
> > > On Sat, Apr 13, 2024 at 4:41=E2=80=AFAM Hillf Danton <hdanton@sina.com> wrote:
> > > > On Thu, 11 Apr 2024 01:11:20 -0700
> > > > > syzbot found the following issue on:
> > > > >
> > > > > HEAD commit: 6ebf211bb11d Add linux-next specific files for 20240410
> > > > > git tree: linux-next
> > > > > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=3D1621af9d180000
> > > >
> > > > #syz test https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git 6ebf211bb11d
> > > >
> > > > --- x/fs/notify/fsnotify.c
> > > > +++ y/fs/notify/fsnotify.c
> > > > @@ -101,8 +101,8 @@ void fsnotify_sb_delete(struct super_blo
> > > > wait_var_event(fsnotify_sb_watched_objects(sb),
> > > > !atomic_long_read(fsnotify_sb_watched_objects(sb)));
> > > > WARN_ON(fsnotify_sb_has_priority_watchers(sb, FSNOTIFY_PRIO_CONTENT));
> > > > - WARN_ON(fsnotify_sb_has_priority_watchers(sb,
> > > > - FSNOTIFY_PRIO_PRE_CONTENT));
> > > > + WARN_ON(fsnotify_sb_has_priority_watchers(sb, FSNOTIFY_PRIO_PRE_CONTENT));
> > > > + synchronize_srcu(&fsnotify_mark_srcu);
> > > > kfree(sbinfo);
> > > > }
> > > >
> > > > @@ -499,7 +499,7 @@ int fsnotify(__u32 mask, const void *dat
> > > > {
> > > > const struct path *path =3D fsnotify_data_path(data, data_type);
> > > > struct super_block *sb =3D fsnotify_data_sb(data, data_type);
> > > > - struct fsnotify_sb_info *sbinfo =3D fsnotify_sb_info(sb);
> > > > + struct fsnotify_sb_info *sbinfo;
> > > > struct fsnotify_iter_info iter_info = {};
> > > > struct mount *mnt =3D NULL;
> > > > struct inode *inode2 =3D NULL;
> > > > @@ -529,6 +529,8 @@ int fsnotify(__u32 mask, const void *dat
> > > > inode2_type =3D FSNOTIFY_ITER_TYPE_PARENT;
> > > > }
> > > >
> > > > + iter_info.srcu_idx =3D srcu_read_lock(&fsnotify_mark_srcu);
> > > > + sbinfo =3D fsnotify_sb_info(sb);
> > > > /*
> > > > * Optimization: srcu_read_lock() has a memory barrier which can
> > > > * be expensive. It protects walking the *_fsnotify_marks lists.
> > >
> > >
> > > See comment above. This kills the optimization.
> > > It is not worth letting all the fsnotify hooks suffer the consequence
> > > for the edge case of calling fsnotify hook during fs shutdown.
> >
> > Say nothing before reading your fix.
> > >
> > > Also, fsnotify_sb_info(sb) in fsnotify_sb_has_priority_watchers()
> > > is also not protected and using srcu_read_lock() there completely
> > > nullifies the purpose of fsnotify_sb_info.
> > >
> > > Here is a simplified fix for fsnotify_sb_error() rebased on the
> > > pending mm fixes for this syzbot boot failure:
> > >
> > > #syz test: https://github.com/amir73il/linux fsnotify-fixes
> >
> > Feel free to post your patch at lore because not everyone has
> > access to sites like github.
> > >
> > > Jan,
> > >
> > > I think that all the functions called from fs shutdown context
> > > should observe that SB_ACTIVE is cleared but wasn't sure?
> >
> > If you composed fix based on SB_ACTIVE that is cleared in
> > generic_shutdown_super() with &sb->s_umount held for write,
> > I wonder what simpler serialization than srcu you could
> > find/create in fsnotify.
>
> As far as I can tell there is no need for serialisation.
>
> The problem is that fsnotify_sb_error() can be called from the
> context of ->put_super() call from generic_shutdown_super().
>
> It's true that in the repro the thread calling fsnotify_sb_error()
> in the worker thread running quota deferred work from put_super()
> but I think there are sufficient barriers for this worker thread to
> observer the cleared SB_ACTIVE flag.
>
> Anyway, according to syzbot, repro does not trigger the UAF
> with my last fix.
>
> To be clear, any fsnotify_sb_error() that is a result of a user operation
> would be holding an active reference to sb so cannot race with
> fsnotify_sb_delete(), but I am not sure that same is true for ext4
> worker threads.
>
> Jan,
>
> You wrote that "In theory these two calls can even run in parallel
> and fsnotify() can be holding fsnotify_sb_info pointer while
> fsnotify_sb_delete() is freeing".
>
> Can you give an example of this case?
Yeah, basically what Hilf writes:
Task 1 Task 2
umount() some delayed work, transaction
commit, whatever is still running
before ext4_put_super() completes
... ext4_error()
fsnotify_sb_error()
fsnotify()
fetches fsnotify_sb_info
generic_shutdown_super()
fsnotify_sb_delete()
frees fsnotify_sb_info
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [syzbot] Re: [syzbot] [ext4?] KASAN: slab-use-after-free Read in fsnotify
2024-04-15 14:03 ` [syzbot] " Jan Kara
@ 2024-04-15 14:47 ` Amir Goldstein
2024-04-16 10:47 ` Amir Goldstein
2024-04-16 13:22 ` [syzbot] " Jan Kara
0 siblings, 2 replies; 26+ messages in thread
From: Amir Goldstein @ 2024-04-15 14:47 UTC (permalink / raw)
To: Jan Kara
Cc: Hillf Danton, syzbot, linux-fsdevel, syzkaller-bugs, linux-kernel
On Mon, Apr 15, 2024 at 5:03 PM Jan Kara <jack@suse.cz> wrote:
>
> On Sat 13-04-24 12:32:32, Amir Goldstein wrote:
> > On Sat, Apr 13, 2024 at 11:45 AM Hillf Danton <hdanton@sina.com> wrote:
> > > On Fri, 12 Apr 2024 23:42:19 -0700 Amir Goldstein
> > > > On Sat, Apr 13, 2024 at 4:41=E2=80=AFAM Hillf Danton <hdanton@sina.com> wrote:
> > > > > On Thu, 11 Apr 2024 01:11:20 -0700
> > > > > > syzbot found the following issue on:
> > > > > >
> > > > > > HEAD commit: 6ebf211bb11d Add linux-next specific files for 20240410
> > > > > > git tree: linux-next
> > > > > > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=3D1621af9d180000
> > > > >
> > > > > #syz test https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git 6ebf211bb11d
> > > > >
> > > > > --- x/fs/notify/fsnotify.c
> > > > > +++ y/fs/notify/fsnotify.c
> > > > > @@ -101,8 +101,8 @@ void fsnotify_sb_delete(struct super_blo
> > > > > wait_var_event(fsnotify_sb_watched_objects(sb),
> > > > > !atomic_long_read(fsnotify_sb_watched_objects(sb)));
> > > > > WARN_ON(fsnotify_sb_has_priority_watchers(sb, FSNOTIFY_PRIO_CONTENT));
> > > > > - WARN_ON(fsnotify_sb_has_priority_watchers(sb,
> > > > > - FSNOTIFY_PRIO_PRE_CONTENT));
> > > > > + WARN_ON(fsnotify_sb_has_priority_watchers(sb, FSNOTIFY_PRIO_PRE_CONTENT));
> > > > > + synchronize_srcu(&fsnotify_mark_srcu);
> > > > > kfree(sbinfo);
> > > > > }
> > > > >
> > > > > @@ -499,7 +499,7 @@ int fsnotify(__u32 mask, const void *dat
> > > > > {
> > > > > const struct path *path =3D fsnotify_data_path(data, data_type);
> > > > > struct super_block *sb =3D fsnotify_data_sb(data, data_type);
> > > > > - struct fsnotify_sb_info *sbinfo =3D fsnotify_sb_info(sb);
> > > > > + struct fsnotify_sb_info *sbinfo;
> > > > > struct fsnotify_iter_info iter_info = {};
> > > > > struct mount *mnt =3D NULL;
> > > > > struct inode *inode2 =3D NULL;
> > > > > @@ -529,6 +529,8 @@ int fsnotify(__u32 mask, const void *dat
> > > > > inode2_type =3D FSNOTIFY_ITER_TYPE_PARENT;
> > > > > }
> > > > >
> > > > > + iter_info.srcu_idx =3D srcu_read_lock(&fsnotify_mark_srcu);
> > > > > + sbinfo =3D fsnotify_sb_info(sb);
> > > > > /*
> > > > > * Optimization: srcu_read_lock() has a memory barrier which can
> > > > > * be expensive. It protects walking the *_fsnotify_marks lists.
> > > >
> > > >
> > > > See comment above. This kills the optimization.
> > > > It is not worth letting all the fsnotify hooks suffer the consequence
> > > > for the edge case of calling fsnotify hook during fs shutdown.
> > >
> > > Say nothing before reading your fix.
> > > >
> > > > Also, fsnotify_sb_info(sb) in fsnotify_sb_has_priority_watchers()
> > > > is also not protected and using srcu_read_lock() there completely
> > > > nullifies the purpose of fsnotify_sb_info.
> > > >
> > > > Here is a simplified fix for fsnotify_sb_error() rebased on the
> > > > pending mm fixes for this syzbot boot failure:
> > > >
> > > > #syz test: https://github.com/amir73il/linux fsnotify-fixes
> > >
> > > Feel free to post your patch at lore because not everyone has
> > > access to sites like github.
> > > >
> > > > Jan,
> > > >
> > > > I think that all the functions called from fs shutdown context
> > > > should observe that SB_ACTIVE is cleared but wasn't sure?
> > >
> > > If you composed fix based on SB_ACTIVE that is cleared in
> > > generic_shutdown_super() with &sb->s_umount held for write,
> > > I wonder what simpler serialization than srcu you could
> > > find/create in fsnotify.
> >
> > As far as I can tell there is no need for serialisation.
> >
> > The problem is that fsnotify_sb_error() can be called from the
> > context of ->put_super() call from generic_shutdown_super().
> >
> > It's true that in the repro the thread calling fsnotify_sb_error()
> > in the worker thread running quota deferred work from put_super()
> > but I think there are sufficient barriers for this worker thread to
> > observer the cleared SB_ACTIVE flag.
> >
> > Anyway, according to syzbot, repro does not trigger the UAF
> > with my last fix.
> >
> > To be clear, any fsnotify_sb_error() that is a result of a user operation
> > would be holding an active reference to sb so cannot race with
> > fsnotify_sb_delete(), but I am not sure that same is true for ext4
> > worker threads.
> >
> > Jan,
> >
> > You wrote that "In theory these two calls can even run in parallel
> > and fsnotify() can be holding fsnotify_sb_info pointer while
> > fsnotify_sb_delete() is freeing".
> >
> > Can you give an example of this case?
>
> Yeah, basically what Hilf writes:
>
> Task 1 Task 2
> umount() some delayed work, transaction
> commit, whatever is still running
> before ext4_put_super() completes
> ... ext4_error()
> fsnotify_sb_error()
> fsnotify()
> fetches fsnotify_sb_info
> generic_shutdown_super()
> fsnotify_sb_delete()
> frees fsnotify_sb_info
OK, so what do you say about Hillf's fix patch?
Maybe it is ok to let go of the optimization in fsnotify(), considering
that we now have stronger optimizations in the inline hooks and
in __fsnotify_parent()?
I think that Hillf's patch is missing setting s_fsnotify_info to NULL?
@@ -101,8 +101,8 @@ void fsnotify_sb_delete(struct super_blo
wait_var_event(fsnotify_sb_watched_objects(sb),
!atomic_long_read(fsnotify_sb_watched_objects(sb)));
WARN_ON(fsnotify_sb_has_priority_watchers(sb, FSNOTIFY_PRIO_CONTENT));
+ WRITE_ONCE(sb->s_fsnotify_info, NULL);
+ synchronize_srcu(&fsnotify_mark_srcu);
kfree(sbinfo);
}
Thanks,
Amir.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [syzbot] Re: [syzbot] [ext4?] KASAN: slab-use-after-free Read in fsnotify
2024-04-15 14:47 ` Amir Goldstein
@ 2024-04-16 10:47 ` Amir Goldstein
2024-04-17 2:03 ` syzbot
2024-04-16 13:22 ` [syzbot] " Jan Kara
1 sibling, 1 reply; 26+ messages in thread
From: Amir Goldstein @ 2024-04-16 10:47 UTC (permalink / raw)
To: Jan Kara
Cc: Hillf Danton, syzbot, linux-fsdevel, syzkaller-bugs, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 6844 bytes --]
On Mon, Apr 15, 2024 at 5:47 PM Amir Goldstein <amir73il@gmail.com> wrote:
>
> On Mon, Apr 15, 2024 at 5:03 PM Jan Kara <jack@suse.cz> wrote:
> >
> > On Sat 13-04-24 12:32:32, Amir Goldstein wrote:
> > > On Sat, Apr 13, 2024 at 11:45 AM Hillf Danton <hdanton@sina.com> wrote:
> > > > On Fri, 12 Apr 2024 23:42:19 -0700 Amir Goldstein
> > > > > On Sat, Apr 13, 2024 at 4:41=E2=80=AFAM Hillf Danton <hdanton@sina.com> wrote:
> > > > > > On Thu, 11 Apr 2024 01:11:20 -0700
> > > > > > > syzbot found the following issue on:
> > > > > > >
> > > > > > > HEAD commit: 6ebf211bb11d Add linux-next specific files for 20240410
> > > > > > > git tree: linux-next
> > > > > > > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=3D1621af9d180000
> > > > > >
> > > > > > #syz test https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git 6ebf211bb11d
> > > > > >
> > > > > > --- x/fs/notify/fsnotify.c
> > > > > > +++ y/fs/notify/fsnotify.c
> > > > > > @@ -101,8 +101,8 @@ void fsnotify_sb_delete(struct super_blo
> > > > > > wait_var_event(fsnotify_sb_watched_objects(sb),
> > > > > > !atomic_long_read(fsnotify_sb_watched_objects(sb)));
> > > > > > WARN_ON(fsnotify_sb_has_priority_watchers(sb, FSNOTIFY_PRIO_CONTENT));
> > > > > > - WARN_ON(fsnotify_sb_has_priority_watchers(sb,
> > > > > > - FSNOTIFY_PRIO_PRE_CONTENT));
> > > > > > + WARN_ON(fsnotify_sb_has_priority_watchers(sb, FSNOTIFY_PRIO_PRE_CONTENT));
> > > > > > + synchronize_srcu(&fsnotify_mark_srcu);
> > > > > > kfree(sbinfo);
> > > > > > }
> > > > > >
> > > > > > @@ -499,7 +499,7 @@ int fsnotify(__u32 mask, const void *dat
> > > > > > {
> > > > > > const struct path *path =3D fsnotify_data_path(data, data_type);
> > > > > > struct super_block *sb =3D fsnotify_data_sb(data, data_type);
> > > > > > - struct fsnotify_sb_info *sbinfo =3D fsnotify_sb_info(sb);
> > > > > > + struct fsnotify_sb_info *sbinfo;
> > > > > > struct fsnotify_iter_info iter_info = {};
> > > > > > struct mount *mnt =3D NULL;
> > > > > > struct inode *inode2 =3D NULL;
> > > > > > @@ -529,6 +529,8 @@ int fsnotify(__u32 mask, const void *dat
> > > > > > inode2_type =3D FSNOTIFY_ITER_TYPE_PARENT;
> > > > > > }
> > > > > >
> > > > > > + iter_info.srcu_idx =3D srcu_read_lock(&fsnotify_mark_srcu);
> > > > > > + sbinfo =3D fsnotify_sb_info(sb);
> > > > > > /*
> > > > > > * Optimization: srcu_read_lock() has a memory barrier which can
> > > > > > * be expensive. It protects walking the *_fsnotify_marks lists.
> > > > >
> > > > >
> > > > > See comment above. This kills the optimization.
> > > > > It is not worth letting all the fsnotify hooks suffer the consequence
> > > > > for the edge case of calling fsnotify hook during fs shutdown.
> > > >
> > > > Say nothing before reading your fix.
> > > > >
> > > > > Also, fsnotify_sb_info(sb) in fsnotify_sb_has_priority_watchers()
> > > > > is also not protected and using srcu_read_lock() there completely
> > > > > nullifies the purpose of fsnotify_sb_info.
> > > > >
> > > > > Here is a simplified fix for fsnotify_sb_error() rebased on the
> > > > > pending mm fixes for this syzbot boot failure:
> > > > >
> > > > > #syz test: https://github.com/amir73il/linux fsnotify-fixes
> > > >
> > > > Feel free to post your patch at lore because not everyone has
> > > > access to sites like github.
> > > > >
> > > > > Jan,
> > > > >
> > > > > I think that all the functions called from fs shutdown context
> > > > > should observe that SB_ACTIVE is cleared but wasn't sure?
> > > >
> > > > If you composed fix based on SB_ACTIVE that is cleared in
> > > > generic_shutdown_super() with &sb->s_umount held for write,
> > > > I wonder what simpler serialization than srcu you could
> > > > find/create in fsnotify.
> > >
> > > As far as I can tell there is no need for serialisation.
> > >
> > > The problem is that fsnotify_sb_error() can be called from the
> > > context of ->put_super() call from generic_shutdown_super().
> > >
> > > It's true that in the repro the thread calling fsnotify_sb_error()
> > > in the worker thread running quota deferred work from put_super()
> > > but I think there are sufficient barriers for this worker thread to
> > > observer the cleared SB_ACTIVE flag.
> > >
> > > Anyway, according to syzbot, repro does not trigger the UAF
> > > with my last fix.
> > >
> > > To be clear, any fsnotify_sb_error() that is a result of a user operation
> > > would be holding an active reference to sb so cannot race with
> > > fsnotify_sb_delete(), but I am not sure that same is true for ext4
> > > worker threads.
> > >
> > > Jan,
> > >
> > > You wrote that "In theory these two calls can even run in parallel
> > > and fsnotify() can be holding fsnotify_sb_info pointer while
> > > fsnotify_sb_delete() is freeing".
> > >
> > > Can you give an example of this case?
> >
> > Yeah, basically what Hilf writes:
> >
> > Task 1 Task 2
> > umount() some delayed work, transaction
> > commit, whatever is still running
> > before ext4_put_super() completes
> > ... ext4_error()
> > fsnotify_sb_error()
> > fsnotify()
> > fetches fsnotify_sb_info
> > generic_shutdown_super()
> > fsnotify_sb_delete()
> > frees fsnotify_sb_info
>
> OK, so what do you say about Hillf's fix patch?
>
> Maybe it is ok to let go of the optimization in fsnotify(), considering
> that we now have stronger optimizations in the inline hooks and
> in __fsnotify_parent()?
>
> I think that Hillf's patch is missing setting s_fsnotify_info to NULL?
>
> @@ -101,8 +101,8 @@ void fsnotify_sb_delete(struct super_blo
> wait_var_event(fsnotify_sb_watched_objects(sb),
> !atomic_long_read(fsnotify_sb_watched_objects(sb)));
> WARN_ON(fsnotify_sb_has_priority_watchers(sb, FSNOTIFY_PRIO_CONTENT));
> + WRITE_ONCE(sb->s_fsnotify_info, NULL);
> + synchronize_srcu(&fsnotify_mark_srcu);
> kfree(sbinfo);
> }
>
I reworked Hillf's patch so it won't break the optimizations for
the common case and added setting s_fsnotify_info to NULL (attached).
Let's see what syzbot has to say:
#syz test: https://github.com/amir73il/linux fsnotify-fixes
Thanks,
Amir.
[-- Attachment #2: 0001-fsnotify-fix-UAF-from-FS_ERROR-event-on-a-shutting-d.patch --]
[-- Type: text/x-patch, Size: 3344 bytes --]
From 4e42e6b66c24e42b4089b1212f0eccb01b7b7eda Mon Sep 17 00:00:00 2001
From: Amir Goldstein <amir73il@gmail.com>
Date: Thu, 11 Apr 2024 18:59:08 +0300
Subject: [PATCH] fsnotify: fix UAF from FS_ERROR event on a shutting down
filesystem
Protect against use after free when filesystem calls fsnotify_sb_error()
during fs shutdown.
fsnotify_sb_error() may be called without an s_active reference, so use
SRCU to synchronize access to fsnotify_sb_info() in this case.
Reported-by: syzbot+5e3f9b2a67b45f16d4e6@syzkaller.appspotmail.com
Suggested-by: Hillf Danton <hdanton@sina.com>
Link: https://lore.kernel.org/linux-fsdevel/20240413014033.1722-1-hdanton@sina.com/
Fixes: 07a3b8d0bf72 ("fsnotify: lazy attach fsnotify_sb_info state to sb")
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
fs/notify/fsnotify.c | 14 ++++++++++++--
include/linux/fsnotify_backend.h | 3 +++
2 files changed, 15 insertions(+), 2 deletions(-)
diff --git a/fs/notify/fsnotify.c b/fs/notify/fsnotify.c
index 2ae965ef37e8..ec9b535d333a 100644
--- a/fs/notify/fsnotify.c
+++ b/fs/notify/fsnotify.c
@@ -103,6 +103,8 @@ void fsnotify_sb_delete(struct super_block *sb)
WARN_ON(fsnotify_sb_has_priority_watchers(sb, FSNOTIFY_PRIO_CONTENT));
WARN_ON(fsnotify_sb_has_priority_watchers(sb,
FSNOTIFY_PRIO_PRE_CONTENT));
+ WRITE_ONCE(sb->s_fsnotify_info, NULL);
+ synchronize_srcu(&fsnotify_mark_srcu);
kfree(sbinfo);
}
@@ -506,6 +508,7 @@ int fsnotify(__u32 mask, const void *data, int data_type, struct inode *dir,
struct dentry *moved;
int inode2_type;
int ret = 0;
+ bool sb_active_ref = !(mask & FS_EVENTS_POSS_ON_SHUTDOWN);
__u32 test_mask, marks_mask;
if (path)
@@ -535,8 +538,10 @@ int fsnotify(__u32 mask, const void *data, int data_type, struct inode *dir,
* However, if we do not walk the lists, we do not have to do
* SRCU because we have no references to any objects and do not
* need SRCU to keep them "alive".
+ * We only need SRCU to keep sbinfo "alive" for events possible
+ * during shutdown (e.g. FS_ERROR).
*/
- if ((!sbinfo || !sbinfo->sb_marks) &&
+ if ((!sbinfo || (sb_active_ref && !sbinfo->sb_marks)) &&
(!mnt || !mnt->mnt_fsnotify_marks) &&
(!inode || !inode->i_fsnotify_marks) &&
(!inode2 || !inode2->i_fsnotify_marks))
@@ -562,7 +567,12 @@ int fsnotify(__u32 mask, const void *data, int data_type, struct inode *dir,
return 0;
iter_info.srcu_idx = srcu_read_lock(&fsnotify_mark_srcu);
-
+ /*
+ * For events possible during shutdown (e.g. FS_ERROR) we may not hold
+ * an s_active reference on sb, so we need to re-fetch sbinfo with
+ * srcu_read_lock() held.
+ */
+ sbinfo = fsnotify_sb_info(sb);
if (sbinfo) {
iter_info.marks[FSNOTIFY_ITER_TYPE_SB] =
fsnotify_first_mark(&sbinfo->sb_marks);
diff --git a/include/linux/fsnotify_backend.h b/include/linux/fsnotify_backend.h
index 7f1ab8264e41..f10987662d1f 100644
--- a/include/linux/fsnotify_backend.h
+++ b/include/linux/fsnotify_backend.h
@@ -97,6 +97,9 @@
*/
#define FS_EVENTS_POSS_TO_PARENT (FS_EVENTS_POSS_ON_CHILD)
+/* Events that could be generated while fs is shutting down */
+#define FS_EVENTS_POSS_ON_SHUTDOWN (FS_ERROR)
+
/* Events that can be reported to backends */
#define ALL_FSNOTIFY_EVENTS (ALL_FSNOTIFY_DIRENT_EVENTS | \
FS_EVENTS_POSS_ON_CHILD | \
--
2.34.1
^ permalink raw reply related [flat|nested] 26+ messages in thread
* Re: [syzbot] Re: [syzbot] [ext4?] KASAN: slab-use-after-free Read in fsnotify
2024-04-15 14:47 ` Amir Goldstein
2024-04-16 10:47 ` Amir Goldstein
@ 2024-04-16 13:22 ` Jan Kara
2024-04-16 16:27 ` Amir Goldstein
1 sibling, 1 reply; 26+ messages in thread
From: Jan Kara @ 2024-04-16 13:22 UTC (permalink / raw)
To: Amir Goldstein
Cc: Jan Kara, Hillf Danton, syzbot, linux-fsdevel, syzkaller-bugs,
linux-kernel
On Mon 15-04-24 17:47:45, Amir Goldstein wrote:
> On Mon, Apr 15, 2024 at 5:03 PM Jan Kara <jack@suse.cz> wrote:
> >
> > On Sat 13-04-24 12:32:32, Amir Goldstein wrote:
> > > On Sat, Apr 13, 2024 at 11:45 AM Hillf Danton <hdanton@sina.com> wrote:
> > > > On Fri, 12 Apr 2024 23:42:19 -0700 Amir Goldstein
> > > > > On Sat, Apr 13, 2024 at 4:41=E2=80=AFAM Hillf Danton <hdanton@sina.com> wrote:
> > > > > > On Thu, 11 Apr 2024 01:11:20 -0700
> > > > > > > syzbot found the following issue on:
> > > > > > >
> > > > > > > HEAD commit: 6ebf211bb11d Add linux-next specific files for 20240410
> > > > > > > git tree: linux-next
> > > > > > > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=3D1621af9d180000
> > > > > >
> > > > > > #syz test https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git 6ebf211bb11d
> > > > > >
> > > > > > --- x/fs/notify/fsnotify.c
> > > > > > +++ y/fs/notify/fsnotify.c
> > > > > > @@ -101,8 +101,8 @@ void fsnotify_sb_delete(struct super_blo
> > > > > > wait_var_event(fsnotify_sb_watched_objects(sb),
> > > > > > !atomic_long_read(fsnotify_sb_watched_objects(sb)));
> > > > > > WARN_ON(fsnotify_sb_has_priority_watchers(sb, FSNOTIFY_PRIO_CONTENT));
> > > > > > - WARN_ON(fsnotify_sb_has_priority_watchers(sb,
> > > > > > - FSNOTIFY_PRIO_PRE_CONTENT));
> > > > > > + WARN_ON(fsnotify_sb_has_priority_watchers(sb, FSNOTIFY_PRIO_PRE_CONTENT));
> > > > > > + synchronize_srcu(&fsnotify_mark_srcu);
> > > > > > kfree(sbinfo);
> > > > > > }
> > > > > >
> > > > > > @@ -499,7 +499,7 @@ int fsnotify(__u32 mask, const void *dat
> > > > > > {
> > > > > > const struct path *path =3D fsnotify_data_path(data, data_type);
> > > > > > struct super_block *sb =3D fsnotify_data_sb(data, data_type);
> > > > > > - struct fsnotify_sb_info *sbinfo =3D fsnotify_sb_info(sb);
> > > > > > + struct fsnotify_sb_info *sbinfo;
> > > > > > struct fsnotify_iter_info iter_info = {};
> > > > > > struct mount *mnt =3D NULL;
> > > > > > struct inode *inode2 =3D NULL;
> > > > > > @@ -529,6 +529,8 @@ int fsnotify(__u32 mask, const void *dat
> > > > > > inode2_type =3D FSNOTIFY_ITER_TYPE_PARENT;
> > > > > > }
> > > > > >
> > > > > > + iter_info.srcu_idx =3D srcu_read_lock(&fsnotify_mark_srcu);
> > > > > > + sbinfo =3D fsnotify_sb_info(sb);
> > > > > > /*
> > > > > > * Optimization: srcu_read_lock() has a memory barrier which can
> > > > > > * be expensive. It protects walking the *_fsnotify_marks lists.
> > > > >
> > > > >
> > > > > See comment above. This kills the optimization.
> > > > > It is not worth letting all the fsnotify hooks suffer the consequence
> > > > > for the edge case of calling fsnotify hook during fs shutdown.
> > > >
> > > > Say nothing before reading your fix.
> > > > >
> > > > > Also, fsnotify_sb_info(sb) in fsnotify_sb_has_priority_watchers()
> > > > > is also not protected and using srcu_read_lock() there completely
> > > > > nullifies the purpose of fsnotify_sb_info.
> > > > >
> > > > > Here is a simplified fix for fsnotify_sb_error() rebased on the
> > > > > pending mm fixes for this syzbot boot failure:
> > > > >
> > > > > #syz test: https://github.com/amir73il/linux fsnotify-fixes
> > > >
> > > > Feel free to post your patch at lore because not everyone has
> > > > access to sites like github.
> > > > >
> > > > > Jan,
> > > > >
> > > > > I think that all the functions called from fs shutdown context
> > > > > should observe that SB_ACTIVE is cleared but wasn't sure?
> > > >
> > > > If you composed fix based on SB_ACTIVE that is cleared in
> > > > generic_shutdown_super() with &sb->s_umount held for write,
> > > > I wonder what simpler serialization than srcu you could
> > > > find/create in fsnotify.
> > >
> > > As far as I can tell there is no need for serialisation.
> > >
> > > The problem is that fsnotify_sb_error() can be called from the
> > > context of ->put_super() call from generic_shutdown_super().
> > >
> > > It's true that in the repro the thread calling fsnotify_sb_error()
> > > in the worker thread running quota deferred work from put_super()
> > > but I think there are sufficient barriers for this worker thread to
> > > observer the cleared SB_ACTIVE flag.
> > >
> > > Anyway, according to syzbot, repro does not trigger the UAF
> > > with my last fix.
> > >
> > > To be clear, any fsnotify_sb_error() that is a result of a user operation
> > > would be holding an active reference to sb so cannot race with
> > > fsnotify_sb_delete(), but I am not sure that same is true for ext4
> > > worker threads.
> > >
> > > Jan,
> > >
> > > You wrote that "In theory these two calls can even run in parallel
> > > and fsnotify() can be holding fsnotify_sb_info pointer while
> > > fsnotify_sb_delete() is freeing".
> > >
> > > Can you give an example of this case?
> >
> > Yeah, basically what Hilf writes:
> >
> > Task 1 Task 2
> > umount() some delayed work, transaction
> > commit, whatever is still running
> > before ext4_put_super() completes
> > ... ext4_error()
> > fsnotify_sb_error()
> > fsnotify()
> > fetches fsnotify_sb_info
> > generic_shutdown_super()
> > fsnotify_sb_delete()
> > frees fsnotify_sb_info
>
> OK, so what do you say about Hillf's fix patch?
>
> Maybe it is ok to let go of the optimization in fsnotify(), considering
> that we now have stronger optimizations in the inline hooks and
> in __fsnotify_parent()?
>
> I think that Hillf's patch is missing setting s_fsnotify_info to NULL?
>
> @@ -101,8 +101,8 @@ void fsnotify_sb_delete(struct super_blo
> wait_var_event(fsnotify_sb_watched_objects(sb),
> !atomic_long_read(fsnotify_sb_watched_objects(sb)));
> WARN_ON(fsnotify_sb_has_priority_watchers(sb, FSNOTIFY_PRIO_CONTENT));
> + WRITE_ONCE(sb->s_fsnotify_info, NULL);
> + synchronize_srcu(&fsnotify_mark_srcu);
> kfree(sbinfo);
> }
So I had a look into this. Yes, something like this should work. We'll see
whether synchronize_srcu() won't slow down umount too much. If someone will
complain, we'll have to find a better solution.
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [syzbot] Re: [syzbot] [ext4?] KASAN: slab-use-after-free Read in fsnotify
2024-04-16 13:22 ` [syzbot] " Jan Kara
@ 2024-04-16 16:27 ` Amir Goldstein
2024-04-16 17:32 ` Jan Kara
0 siblings, 1 reply; 26+ messages in thread
From: Amir Goldstein @ 2024-04-16 16:27 UTC (permalink / raw)
To: Jan Kara
Cc: Hillf Danton, syzbot, linux-fsdevel, syzkaller-bugs, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 7326 bytes --]
On Tue, Apr 16, 2024 at 4:22 PM Jan Kara <jack@suse.cz> wrote:
>
> On Mon 15-04-24 17:47:45, Amir Goldstein wrote:
> > On Mon, Apr 15, 2024 at 5:03 PM Jan Kara <jack@suse.cz> wrote:
> > >
> > > On Sat 13-04-24 12:32:32, Amir Goldstein wrote:
> > > > On Sat, Apr 13, 2024 at 11:45 AM Hillf Danton <hdanton@sina.com> wrote:
> > > > > On Fri, 12 Apr 2024 23:42:19 -0700 Amir Goldstein
> > > > > > On Sat, Apr 13, 2024 at 4:41=E2=80=AFAM Hillf Danton <hdanton@sina.com> wrote:
> > > > > > > On Thu, 11 Apr 2024 01:11:20 -0700
> > > > > > > > syzbot found the following issue on:
> > > > > > > >
> > > > > > > > HEAD commit: 6ebf211bb11d Add linux-next specific files for 20240410
> > > > > > > > git tree: linux-next
> > > > > > > > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=3D1621af9d180000
> > > > > > >
> > > > > > > #syz test https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git 6ebf211bb11d
> > > > > > >
> > > > > > > --- x/fs/notify/fsnotify.c
> > > > > > > +++ y/fs/notify/fsnotify.c
> > > > > > > @@ -101,8 +101,8 @@ void fsnotify_sb_delete(struct super_blo
> > > > > > > wait_var_event(fsnotify_sb_watched_objects(sb),
> > > > > > > !atomic_long_read(fsnotify_sb_watched_objects(sb)));
> > > > > > > WARN_ON(fsnotify_sb_has_priority_watchers(sb, FSNOTIFY_PRIO_CONTENT));
> > > > > > > - WARN_ON(fsnotify_sb_has_priority_watchers(sb,
> > > > > > > - FSNOTIFY_PRIO_PRE_CONTENT));
> > > > > > > + WARN_ON(fsnotify_sb_has_priority_watchers(sb, FSNOTIFY_PRIO_PRE_CONTENT));
> > > > > > > + synchronize_srcu(&fsnotify_mark_srcu);
> > > > > > > kfree(sbinfo);
> > > > > > > }
> > > > > > >
> > > > > > > @@ -499,7 +499,7 @@ int fsnotify(__u32 mask, const void *dat
> > > > > > > {
> > > > > > > const struct path *path =3D fsnotify_data_path(data, data_type);
> > > > > > > struct super_block *sb =3D fsnotify_data_sb(data, data_type);
> > > > > > > - struct fsnotify_sb_info *sbinfo =3D fsnotify_sb_info(sb);
> > > > > > > + struct fsnotify_sb_info *sbinfo;
> > > > > > > struct fsnotify_iter_info iter_info = {};
> > > > > > > struct mount *mnt =3D NULL;
> > > > > > > struct inode *inode2 =3D NULL;
> > > > > > > @@ -529,6 +529,8 @@ int fsnotify(__u32 mask, const void *dat
> > > > > > > inode2_type =3D FSNOTIFY_ITER_TYPE_PARENT;
> > > > > > > }
> > > > > > >
> > > > > > > + iter_info.srcu_idx =3D srcu_read_lock(&fsnotify_mark_srcu);
> > > > > > > + sbinfo =3D fsnotify_sb_info(sb);
> > > > > > > /*
> > > > > > > * Optimization: srcu_read_lock() has a memory barrier which can
> > > > > > > * be expensive. It protects walking the *_fsnotify_marks lists.
> > > > > >
> > > > > >
> > > > > > See comment above. This kills the optimization.
> > > > > > It is not worth letting all the fsnotify hooks suffer the consequence
> > > > > > for the edge case of calling fsnotify hook during fs shutdown.
> > > > >
> > > > > Say nothing before reading your fix.
> > > > > >
> > > > > > Also, fsnotify_sb_info(sb) in fsnotify_sb_has_priority_watchers()
> > > > > > is also not protected and using srcu_read_lock() there completely
> > > > > > nullifies the purpose of fsnotify_sb_info.
> > > > > >
> > > > > > Here is a simplified fix for fsnotify_sb_error() rebased on the
> > > > > > pending mm fixes for this syzbot boot failure:
> > > > > >
> > > > > > #syz test: https://github.com/amir73il/linux fsnotify-fixes
> > > > >
> > > > > Feel free to post your patch at lore because not everyone has
> > > > > access to sites like github.
> > > > > >
> > > > > > Jan,
> > > > > >
> > > > > > I think that all the functions called from fs shutdown context
> > > > > > should observe that SB_ACTIVE is cleared but wasn't sure?
> > > > >
> > > > > If you composed fix based on SB_ACTIVE that is cleared in
> > > > > generic_shutdown_super() with &sb->s_umount held for write,
> > > > > I wonder what simpler serialization than srcu you could
> > > > > find/create in fsnotify.
> > > >
> > > > As far as I can tell there is no need for serialisation.
> > > >
> > > > The problem is that fsnotify_sb_error() can be called from the
> > > > context of ->put_super() call from generic_shutdown_super().
> > > >
> > > > It's true that in the repro the thread calling fsnotify_sb_error()
> > > > in the worker thread running quota deferred work from put_super()
> > > > but I think there are sufficient barriers for this worker thread to
> > > > observer the cleared SB_ACTIVE flag.
> > > >
> > > > Anyway, according to syzbot, repro does not trigger the UAF
> > > > with my last fix.
> > > >
> > > > To be clear, any fsnotify_sb_error() that is a result of a user operation
> > > > would be holding an active reference to sb so cannot race with
> > > > fsnotify_sb_delete(), but I am not sure that same is true for ext4
> > > > worker threads.
> > > >
> > > > Jan,
> > > >
> > > > You wrote that "In theory these two calls can even run in parallel
> > > > and fsnotify() can be holding fsnotify_sb_info pointer while
> > > > fsnotify_sb_delete() is freeing".
> > > >
> > > > Can you give an example of this case?
> > >
> > > Yeah, basically what Hilf writes:
> > >
> > > Task 1 Task 2
> > > umount() some delayed work, transaction
> > > commit, whatever is still running
> > > before ext4_put_super() completes
> > > ... ext4_error()
> > > fsnotify_sb_error()
> > > fsnotify()
> > > fetches fsnotify_sb_info
> > > generic_shutdown_super()
> > > fsnotify_sb_delete()
> > > frees fsnotify_sb_info
> >
> > OK, so what do you say about Hillf's fix patch?
> >
> > Maybe it is ok to let go of the optimization in fsnotify(), considering
> > that we now have stronger optimizations in the inline hooks and
> > in __fsnotify_parent()?
> >
> > I think that Hillf's patch is missing setting s_fsnotify_info to NULL?
> >
> > @@ -101,8 +101,8 @@ void fsnotify_sb_delete(struct super_blo
> > wait_var_event(fsnotify_sb_watched_objects(sb),
> > !atomic_long_read(fsnotify_sb_watched_objects(sb)));
> > WARN_ON(fsnotify_sb_has_priority_watchers(sb, FSNOTIFY_PRIO_CONTENT));
> > + WRITE_ONCE(sb->s_fsnotify_info, NULL);
> > + synchronize_srcu(&fsnotify_mark_srcu);
> > kfree(sbinfo);
> > }
>
> So I had a look into this. Yes, something like this should work. We'll see
> whether synchronize_srcu() won't slow down umount too much. If someone will
> complain, we'll have to find a better solution.
>
Actually, kfree_rcu(sbinfo) may be enough.
We do not actually access sbinfo during mark iteration and
event handling, we only access it to get to the sb connector.
Something like the attached patch?
Thanks,
Amir.
[-- Attachment #2: v2-0001-fsnotify-fix-UAF-from-FS_ERROR-event-on-a-shuttin.patch --]
[-- Type: text/x-patch, Size: 3802 bytes --]
From 8907b0e96a1d7d3b090982abc6d9eb396eb467f0 Mon Sep 17 00:00:00 2001
From: Amir Goldstein <amir73il@gmail.com>
Date: Thu, 11 Apr 2024 18:59:08 +0300
Subject: [PATCH v2] fsnotify: fix UAF from FS_ERROR event on a shutting down
filesystem
Protect against use after free when filesystem calls fsnotify_sb_error()
during fs shutdown.
fsnotify_sb_error() may be called without an s_active reference, so use
RCU to synchronize access to fsnotify_sb_info() in this case.
Reported-by: syzbot+5e3f9b2a67b45f16d4e6@syzkaller.appspotmail.com
Suggested-by: Hillf Danton <hdanton@sina.com>
Link: https://lore.kernel.org/linux-fsdevel/20240413014033.1722-1-hdanton@sina.com/
Fixes: 07a3b8d0bf72 ("fsnotify: lazy attach fsnotify_sb_info state to sb")
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
fs/notify/fsnotify.c | 17 ++++++++++++++---
include/linux/fsnotify_backend.h | 4 ++++
2 files changed, 18 insertions(+), 3 deletions(-)
diff --git a/fs/notify/fsnotify.c b/fs/notify/fsnotify.c
index 2ae965ef37e8..310c8e845482 100644
--- a/fs/notify/fsnotify.c
+++ b/fs/notify/fsnotify.c
@@ -103,7 +103,8 @@ void fsnotify_sb_delete(struct super_block *sb)
WARN_ON(fsnotify_sb_has_priority_watchers(sb, FSNOTIFY_PRIO_CONTENT));
WARN_ON(fsnotify_sb_has_priority_watchers(sb,
FSNOTIFY_PRIO_PRE_CONTENT));
- kfree(sbinfo);
+ WRITE_ONCE(sb->s_fsnotify_info, NULL);
+ kfree_rcu(sbinfo, rcu);
}
/*
@@ -506,6 +507,7 @@ int fsnotify(__u32 mask, const void *data, int data_type, struct inode *dir,
struct dentry *moved;
int inode2_type;
int ret = 0;
+ bool sb_active_ref = !(mask & FS_EVENTS_POSS_ON_SHUTDOWN);
__u32 test_mask, marks_mask;
if (path)
@@ -535,8 +537,10 @@ int fsnotify(__u32 mask, const void *data, int data_type, struct inode *dir,
* However, if we do not walk the lists, we do not have to do
* SRCU because we have no references to any objects and do not
* need SRCU to keep them "alive".
+ * We need RCU read side to keep sbinfo "alive" for events possible
+ * during shutdown (e.g. FS_ERROR).
*/
- if ((!sbinfo || !sbinfo->sb_marks) &&
+ if ((!sbinfo || (sb_active_ref && !sbinfo->sb_marks)) &&
(!mnt || !mnt->mnt_fsnotify_marks) &&
(!inode || !inode->i_fsnotify_marks) &&
(!inode2 || !inode2->i_fsnotify_marks))
@@ -562,11 +566,18 @@ int fsnotify(__u32 mask, const void *data, int data_type, struct inode *dir,
return 0;
iter_info.srcu_idx = srcu_read_lock(&fsnotify_mark_srcu);
-
+ /*
+ * For events possible during shutdown (e.g. FS_ERROR) we may not hold
+ * an s_active reference on sb, so we need to dereference sbinfo with
+ * rcu_read_lock() held.
+ */
+ rcu_read_lock();
+ sbinfo = fsnotify_sb_info(sb);
if (sbinfo) {
iter_info.marks[FSNOTIFY_ITER_TYPE_SB] =
fsnotify_first_mark(&sbinfo->sb_marks);
}
+ rcu_read_unlock();
if (mnt) {
iter_info.marks[FSNOTIFY_ITER_TYPE_VFSMOUNT] =
fsnotify_first_mark(&mnt->mnt_fsnotify_marks);
diff --git a/include/linux/fsnotify_backend.h b/include/linux/fsnotify_backend.h
index 7f1ab8264e41..f11047125522 100644
--- a/include/linux/fsnotify_backend.h
+++ b/include/linux/fsnotify_backend.h
@@ -97,6 +97,9 @@
*/
#define FS_EVENTS_POSS_TO_PARENT (FS_EVENTS_POSS_ON_CHILD)
+/* Events that could be generated while fs is shutting down */
+#define FS_EVENTS_POSS_ON_SHUTDOWN (FS_ERROR)
+
/* Events that can be reported to backends */
#define ALL_FSNOTIFY_EVENTS (ALL_FSNOTIFY_DIRENT_EVENTS | \
FS_EVENTS_POSS_ON_CHILD | \
@@ -488,6 +491,7 @@ struct fsnotify_mark_connector {
*/
struct fsnotify_sb_info {
struct fsnotify_mark_connector __rcu *sb_marks;
+ struct rcu_head rcu;
/*
* Number of inode/mount/sb objects that are being watched in this sb.
* Note that inodes objects are currently double-accounted.
--
2.34.1
^ permalink raw reply related [flat|nested] 26+ messages in thread
* Re: [syzbot] Re: [syzbot] [ext4?] KASAN: slab-use-after-free Read in fsnotify
2024-04-16 16:27 ` Amir Goldstein
@ 2024-04-16 17:32 ` Jan Kara
2024-04-16 17:55 ` Amir Goldstein
2024-04-16 22:50 ` Hillf Danton
0 siblings, 2 replies; 26+ messages in thread
From: Jan Kara @ 2024-04-16 17:32 UTC (permalink / raw)
To: Amir Goldstein
Cc: Jan Kara, Hillf Danton, syzbot, linux-fsdevel, syzkaller-bugs,
linux-kernel
On Tue 16-04-24 19:27:05, Amir Goldstein wrote:
> On Tue, Apr 16, 2024 at 4:22 PM Jan Kara <jack@suse.cz> wrote:
> >
> > On Mon 15-04-24 17:47:45, Amir Goldstein wrote:
> > > On Mon, Apr 15, 2024 at 5:03 PM Jan Kara <jack@suse.cz> wrote:
> > > >
> > > > On Sat 13-04-24 12:32:32, Amir Goldstein wrote:
> > > > > On Sat, Apr 13, 2024 at 11:45 AM Hillf Danton <hdanton@sina.com> wrote:
> > > > > > On Fri, 12 Apr 2024 23:42:19 -0700 Amir Goldstein
> > > > > > > On Sat, Apr 13, 2024 at 4:41=E2=80=AFAM Hillf Danton <hdanton@sina.com> wrote:
> > > > > > > > On Thu, 11 Apr 2024 01:11:20 -0700
> > > > > > > > > syzbot found the following issue on:
> > > > > > > > >
> > > > > > > > > HEAD commit: 6ebf211bb11d Add linux-next specific files for 20240410
> > > > > > > > > git tree: linux-next
> > > > > > > > > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=3D1621af9d180000
> > > > > > > >
> > > > > > > > #syz test https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git 6ebf211bb11d
> > > > > > > >
> > > > > > > > --- x/fs/notify/fsnotify.c
> > > > > > > > +++ y/fs/notify/fsnotify.c
> > > > > > > > @@ -101,8 +101,8 @@ void fsnotify_sb_delete(struct super_blo
> > > > > > > > wait_var_event(fsnotify_sb_watched_objects(sb),
> > > > > > > > !atomic_long_read(fsnotify_sb_watched_objects(sb)));
> > > > > > > > WARN_ON(fsnotify_sb_has_priority_watchers(sb, FSNOTIFY_PRIO_CONTENT));
> > > > > > > > - WARN_ON(fsnotify_sb_has_priority_watchers(sb,
> > > > > > > > - FSNOTIFY_PRIO_PRE_CONTENT));
> > > > > > > > + WARN_ON(fsnotify_sb_has_priority_watchers(sb, FSNOTIFY_PRIO_PRE_CONTENT));
> > > > > > > > + synchronize_srcu(&fsnotify_mark_srcu);
> > > > > > > > kfree(sbinfo);
> > > > > > > > }
> > > > > > > >
> > > > > > > > @@ -499,7 +499,7 @@ int fsnotify(__u32 mask, const void *dat
> > > > > > > > {
> > > > > > > > const struct path *path =3D fsnotify_data_path(data, data_type);
> > > > > > > > struct super_block *sb =3D fsnotify_data_sb(data, data_type);
> > > > > > > > - struct fsnotify_sb_info *sbinfo =3D fsnotify_sb_info(sb);
> > > > > > > > + struct fsnotify_sb_info *sbinfo;
> > > > > > > > struct fsnotify_iter_info iter_info = {};
> > > > > > > > struct mount *mnt =3D NULL;
> > > > > > > > struct inode *inode2 =3D NULL;
> > > > > > > > @@ -529,6 +529,8 @@ int fsnotify(__u32 mask, const void *dat
> > > > > > > > inode2_type =3D FSNOTIFY_ITER_TYPE_PARENT;
> > > > > > > > }
> > > > > > > >
> > > > > > > > + iter_info.srcu_idx =3D srcu_read_lock(&fsnotify_mark_srcu);
> > > > > > > > + sbinfo =3D fsnotify_sb_info(sb);
> > > > > > > > /*
> > > > > > > > * Optimization: srcu_read_lock() has a memory barrier which can
> > > > > > > > * be expensive. It protects walking the *_fsnotify_marks lists.
> > > > > > >
> > > > > > >
> > > > > > > See comment above. This kills the optimization.
> > > > > > > It is not worth letting all the fsnotify hooks suffer the consequence
> > > > > > > for the edge case of calling fsnotify hook during fs shutdown.
> > > > > >
> > > > > > Say nothing before reading your fix.
> > > > > > >
> > > > > > > Also, fsnotify_sb_info(sb) in fsnotify_sb_has_priority_watchers()
> > > > > > > is also not protected and using srcu_read_lock() there completely
> > > > > > > nullifies the purpose of fsnotify_sb_info.
> > > > > > >
> > > > > > > Here is a simplified fix for fsnotify_sb_error() rebased on the
> > > > > > > pending mm fixes for this syzbot boot failure:
> > > > > > >
> > > > > > > #syz test: https://github.com/amir73il/linux fsnotify-fixes
> > > > > >
> > > > > > Feel free to post your patch at lore because not everyone has
> > > > > > access to sites like github.
> > > > > > >
> > > > > > > Jan,
> > > > > > >
> > > > > > > I think that all the functions called from fs shutdown context
> > > > > > > should observe that SB_ACTIVE is cleared but wasn't sure?
> > > > > >
> > > > > > If you composed fix based on SB_ACTIVE that is cleared in
> > > > > > generic_shutdown_super() with &sb->s_umount held for write,
> > > > > > I wonder what simpler serialization than srcu you could
> > > > > > find/create in fsnotify.
> > > > >
> > > > > As far as I can tell there is no need for serialisation.
> > > > >
> > > > > The problem is that fsnotify_sb_error() can be called from the
> > > > > context of ->put_super() call from generic_shutdown_super().
> > > > >
> > > > > It's true that in the repro the thread calling fsnotify_sb_error()
> > > > > in the worker thread running quota deferred work from put_super()
> > > > > but I think there are sufficient barriers for this worker thread to
> > > > > observer the cleared SB_ACTIVE flag.
> > > > >
> > > > > Anyway, according to syzbot, repro does not trigger the UAF
> > > > > with my last fix.
> > > > >
> > > > > To be clear, any fsnotify_sb_error() that is a result of a user operation
> > > > > would be holding an active reference to sb so cannot race with
> > > > > fsnotify_sb_delete(), but I am not sure that same is true for ext4
> > > > > worker threads.
> > > > >
> > > > > Jan,
> > > > >
> > > > > You wrote that "In theory these two calls can even run in parallel
> > > > > and fsnotify() can be holding fsnotify_sb_info pointer while
> > > > > fsnotify_sb_delete() is freeing".
> > > > >
> > > > > Can you give an example of this case?
> > > >
> > > > Yeah, basically what Hilf writes:
> > > >
> > > > Task 1 Task 2
> > > > umount() some delayed work, transaction
> > > > commit, whatever is still running
> > > > before ext4_put_super() completes
> > > > ... ext4_error()
> > > > fsnotify_sb_error()
> > > > fsnotify()
> > > > fetches fsnotify_sb_info
> > > > generic_shutdown_super()
> > > > fsnotify_sb_delete()
> > > > frees fsnotify_sb_info
> > >
> > > OK, so what do you say about Hillf's fix patch?
> > >
> > > Maybe it is ok to let go of the optimization in fsnotify(), considering
> > > that we now have stronger optimizations in the inline hooks and
> > > in __fsnotify_parent()?
> > >
> > > I think that Hillf's patch is missing setting s_fsnotify_info to NULL?
> > >
> > > @@ -101,8 +101,8 @@ void fsnotify_sb_delete(struct super_blo
> > > wait_var_event(fsnotify_sb_watched_objects(sb),
> > > !atomic_long_read(fsnotify_sb_watched_objects(sb)));
> > > WARN_ON(fsnotify_sb_has_priority_watchers(sb, FSNOTIFY_PRIO_CONTENT));
> > > + WRITE_ONCE(sb->s_fsnotify_info, NULL);
> > > + synchronize_srcu(&fsnotify_mark_srcu);
> > > kfree(sbinfo);
> > > }
> >
> > So I had a look into this. Yes, something like this should work. We'll see
> > whether synchronize_srcu() won't slow down umount too much. If someone will
> > complain, we'll have to find a better solution.
> >
>
> Actually, kfree_rcu(sbinfo) may be enough.
> We do not actually access sbinfo during mark iteration and
> event handling, we only access it to get to the sb connector.
>
> Something like the attached patch?
Hum, thinking about this some more - what if we just freed sb_info from
destroy_super_work()? By then we definitely are not getting fsnotify()
calls for the superblock so all the problems are solved.
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [syzbot] Re: [syzbot] [ext4?] KASAN: slab-use-after-free Read in fsnotify
2024-04-16 17:32 ` Jan Kara
@ 2024-04-16 17:55 ` Amir Goldstein
2024-04-16 22:50 ` Hillf Danton
1 sibling, 0 replies; 26+ messages in thread
From: Amir Goldstein @ 2024-04-16 17:55 UTC (permalink / raw)
To: Jan Kara
Cc: Hillf Danton, syzbot, linux-fsdevel, syzkaller-bugs, linux-kernel,
Christian Brauner
> > > > Maybe it is ok to let go of the optimization in fsnotify(), considering
> > > > that we now have stronger optimizations in the inline hooks and
> > > > in __fsnotify_parent()?
> > > >
> > > > I think that Hillf's patch is missing setting s_fsnotify_info to NULL?
> > > >
> > > > @@ -101,8 +101,8 @@ void fsnotify_sb_delete(struct super_blo
> > > > wait_var_event(fsnotify_sb_watched_objects(sb),
> > > > !atomic_long_read(fsnotify_sb_watched_objects(sb)));
> > > > WARN_ON(fsnotify_sb_has_priority_watchers(sb, FSNOTIFY_PRIO_CONTENT));
> > > > + WRITE_ONCE(sb->s_fsnotify_info, NULL);
> > > > + synchronize_srcu(&fsnotify_mark_srcu);
> > > > kfree(sbinfo);
> > > > }
> > >
> > > So I had a look into this. Yes, something like this should work. We'll see
> > > whether synchronize_srcu() won't slow down umount too much. If someone will
> > > complain, we'll have to find a better solution.
> > >
> >
> > Actually, kfree_rcu(sbinfo) may be enough.
> > We do not actually access sbinfo during mark iteration and
> > event handling, we only access it to get to the sb connector.
> >
> > Something like the attached patch?
>
> Hum, thinking about this some more - what if we just freed sb_info from
> destroy_super_work()? By then we definitely are not getting fsnotify()
> calls for the superblock so all the problems are solved.
>
Considering that this is the solution for security_sb_free()
I don't see why not have fsnotify_sb_free().
I'll prepare a patch.
Thanks!
Amir.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [syzbot] Re: [syzbot] [ext4?] KASAN: slab-use-after-free Read in fsnotify
2024-04-16 17:32 ` Jan Kara
2024-04-16 17:55 ` Amir Goldstein
@ 2024-04-16 22:50 ` Hillf Danton
1 sibling, 0 replies; 26+ messages in thread
From: Hillf Danton @ 2024-04-16 22:50 UTC (permalink / raw)
To: Jan Kara
Cc: Amir Goldstein, syzbot, linux-fsdevel, syzkaller-bugs,
linux-kernel, Christian Brauner
On Tue, 16 Apr 2024 19:32:11 +0200 Jan Kara <jack@suse.cz>
>
> Hum, thinking about this some more - what if we just freed sb_info from
> destroy_super_work()? By then we definitely are not getting fsnotify()
> calls for the superblock so all the problems are solved.
>
Sounds better :)
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [syzbot] [ext4?] KASAN: slab-use-after-free Read in fsnotify
2024-04-16 10:47 ` Amir Goldstein
@ 2024-04-17 2:03 ` syzbot
0 siblings, 0 replies; 26+ messages in thread
From: syzbot @ 2024-04-17 2:03 UTC (permalink / raw)
To: amir73il, hdanton, jack, linux-fsdevel, linux-kernel,
syzkaller-bugs
Hello,
syzbot tried to test the proposed patch but the build/boot failed:
failed to apply patch:
checking file fs/notify/fsnotify.c
Hunk #1 FAILED at 103.
Hunk #2 succeeded at 510 (offset 4 lines).
Hunk #3 succeeded at 540 (offset 4 lines).
Hunk #4 succeeded at 569 (offset 4 lines).
1 out of 4 hunks FAILED
checking file include/linux/fsnotify_backend.h
Tested on:
commit: 2f012b22 fsnotify: fix UAF from FS_ERROR event on a sh..
git tree: https://github.com/amir73il/linux fsnotify-fixes
kernel config: https://syzkaller.appspot.com/x/.config?x=16ca158ef7e08662
dashboard link: https://syzkaller.appspot.com/bug?extid=5e3f9b2a67b45f16d4e6
compiler:
patch: https://syzkaller.appspot.com/x/patch.diff?x=10103657180000
^ permalink raw reply [flat|nested] 26+ messages in thread
end of thread, other threads:[~2024-04-17 2:03 UTC | newest]
Thread overview: 26+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <00000000000095bb400615f4b0ed@google.com>
2024-04-13 8:45 ` [syzbot] Re: [syzbot] [ext4?] KASAN: slab-use-after-free Read in fsnotify Hillf Danton
2024-04-13 9:32 ` Amir Goldstein
2024-04-13 12:04 ` Hillf Danton
2024-04-15 14:03 ` [syzbot] " Jan Kara
2024-04-15 14:47 ` Amir Goldstein
2024-04-16 10:47 ` Amir Goldstein
2024-04-17 2:03 ` syzbot
2024-04-16 13:22 ` [syzbot] " Jan Kara
2024-04-16 16:27 ` Amir Goldstein
2024-04-16 17:32 ` Jan Kara
2024-04-16 17:55 ` Amir Goldstein
2024-04-16 22:50 ` Hillf Danton
2024-04-11 8:11 syzbot
2024-04-11 12:13 ` Jan Kara
2024-04-11 16:07 ` Amir Goldstein
2024-04-11 19:25 ` Gabriel Krisman Bertazi
2024-04-11 20:27 ` Khazhy Kumykov
2024-04-12 10:24 ` Amir Goldstein
2024-04-11 20:06 ` syzbot
2024-04-12 7:52 ` Amir Goldstein
2024-04-12 14:30 ` syzbot
2024-04-12 14:32 ` Aleksandr Nogikh
2024-04-13 1:40 ` Hillf Danton
2024-04-13 6:42 ` Amir Goldstein
2024-04-13 8:58 ` syzbot
2024-04-13 8:40 ` syzbot
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).