* fanotify use after free. @ 2014-01-22 6:27 Dave Jones 2014-01-22 16:43 ` Dave Jones 2014-01-22 18:20 ` Linus Torvalds 0 siblings, 2 replies; 19+ messages in thread From: Dave Jones @ 2014-01-22 6:27 UTC (permalink / raw) To: jack; +Cc: Linux Kernel Jan, since yesterdays changes, on boot I see a flood of messages from slub debug during boot.. ============================================================================= BUG fanotify_event_info (Not tainted): Poison overwritten ----------------------------------------------------------------------------- Disabling lock debugging due to kernel taint INFO: 0xffff880247e45bc8-0xffff880247e45bcb. First byte 0x0 instead of 0x6b INFO: Allocated in fanotify_handle_event+0x136/0x390 age=0 cpu=0 pid=293 __slab_alloc+0x456/0x565 kmem_cache_alloc+0x1fe/0x260 fanotify_handle_event+0x136/0x390 send_to_group+0xd3/0x1c0 fsnotify+0x1c8/0x340 open_exec+0xe2/0x120 load_elf_binary+0x7b7/0x18e0 search_binary_handler+0x94/0x1b0 do_execve_common.isra.26+0x5d7/0x7d0 SyS_execve+0x36/0x50 stub_execve+0x69/0xa0 INFO: Freed in fanotify_free_event+0x2e/0x40 age=0 cpu=3 pid=290 __slab_free+0x4a/0x382 kmem_cache_free+0x1c9/0x210 fanotify_free_event+0x2e/0x40 fsnotify_destroy_event+0x21/0x30 fanotify_read+0x39e/0x5e0 vfs_read+0x9b/0x160 SyS_read+0x58/0xb0 tracesys+0xdd/0xe2 INFO: Slab 0xffffea00091f9100 objects=20 used=20 fp=0x (null) flags=0x20000000004080 INFO: Object 0xffff880247e45b90 @offset=7056 fp=0xffff880247e44000 Bytes b4 ffff880247e45b80: 00 00 00 00 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ........ZZZZZZZZ Object ffff880247e45b90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Object ffff880247e45ba0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Object ffff880247e45bb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Object ffff880247e45bc0: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 6b 6b 6b a5 kkkkkkkk....kkk. Redzone ffff880247e45bd0: bb bb bb bb bb bb bb bb ........ Padding ffff880247e45d10: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ CPU: 0 PID: 293 Comm: mount Tainted: G B 3.13.0+ #28 ffff880247e45b90 000000008c7fe87c ffff8800874cbb28 ffffffff9c710632 ffff88024a776ac0 ffff8800874cbb68 ffffffff9c194dad 0000000000000008 ffff880200000001 ffff880247e45bcc ffff88024a776ac0 000000000000006b Call Trace: [<ffffffff9c710632>] dump_stack+0x4e/0x7a [<ffffffff9c194dad>] print_trailer+0x14d/0x200 [<ffffffff9c19505f>] check_bytes_and_report+0xcf/0x110 [<ffffffff9c196037>] check_object+0x1d7/0x250 [<ffffffff9c1f4ae6>] ? fanotify_handle_event+0x136/0x390 [<ffffffff9c70ead7>] alloc_debug_processing+0x76/0x118 [<ffffffff9c70f77d>] __slab_alloc+0x456/0x565 [<ffffffff9c1f4ae6>] ? fanotify_handle_event+0x136/0x390 [<ffffffff9c1ccea4>] ? mntput+0x24/0x40 [<ffffffff9c1b5dc9>] ? terminate_walk+0x69/0x70 [<ffffffff9c1ba6fe>] ? do_last+0x25e/0x1390 [<ffffffff9c1b6cf8>] ? inode_permission+0x18/0x50 [<ffffffff9c1f4ae6>] ? fanotify_handle_event+0x136/0x390 [<ffffffff9c1980fe>] kmem_cache_alloc+0x1fe/0x260 [<ffffffff9c1f4ae6>] fanotify_handle_event+0x136/0x390 [<ffffffff9c1bb8fd>] ? path_openat+0xcd/0x6a0 [<ffffffff9c1f0e63>] send_to_group+0xd3/0x1c0 [<ffffffff9c1f0fdf>] ? fsnotify+0x8f/0x340 [<ffffffff9c1f1118>] fsnotify+0x1c8/0x340 [<ffffffff9c1a9b4f>] do_sys_open+0x19f/0x230 [<ffffffff9c1a9bfe>] SyS_open+0x1e/0x20 [<ffffffff9c723764>] tracesys+0xdd/0xe2 FIX fanotify_event_info: Restoring 0xffff880247e45bc8-0xffff880247e45bcb=0x6b ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free. 2014-01-22 6:27 fanotify use after free Dave Jones @ 2014-01-22 16:43 ` Dave Jones 2014-01-22 18:20 ` Linus Torvalds 1 sibling, 0 replies; 19+ messages in thread From: Dave Jones @ 2014-01-22 16:43 UTC (permalink / raw) To: jack, Linux Kernel; +Cc: Linus Torvalds On Wed, Jan 22, 2014 at 01:27:30AM -0500, Dave Jones wrote: > Jan, > > since yesterdays changes, on boot I see a flood of messages from slub debug during boot.. > > ============================================================================= > BUG fanotify_event_info (Not tainted): Poison overwritten > ----------------------------------------------------------------------------- > > Disabling lock debugging due to kernel taint > INFO: 0xffff880247e45bc8-0xffff880247e45bcb. First byte 0x0 instead of 0x6b > INFO: Allocated in fanotify_handle_event+0x136/0x390 age=0 cpu=0 pid=293 > __slab_alloc+0x456/0x565 > kmem_cache_alloc+0x1fe/0x260 > fanotify_handle_event+0x136/0x390 > send_to_group+0xd3/0x1c0 > fsnotify+0x1c8/0x340 > open_exec+0xe2/0x120 > load_elf_binary+0x7b7/0x18e0 > search_binary_handler+0x94/0x1b0 > do_execve_common.isra.26+0x5d7/0x7d0 > SyS_execve+0x36/0x50 > stub_execve+0x69/0xa0 > INFO: Freed in fanotify_free_event+0x2e/0x40 age=0 cpu=3 pid=290 > __slab_free+0x4a/0x382 > kmem_cache_free+0x1c9/0x210 > fanotify_free_event+0x2e/0x40 > fsnotify_destroy_event+0x21/0x30 > fanotify_read+0x39e/0x5e0 > vfs_read+0x9b/0x160 > SyS_read+0x58/0xb0 > tracesys+0xdd/0xe2 > INFO: Slab 0xffffea00091f9100 objects=20 used=20 fp=0x (null) flags=0x20000000004080 Reverting 7053aee26a3548ebaba046ae2e52396ccf56ac6c makes this go away. Dave ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free. 2014-01-22 6:27 fanotify use after free Dave Jones 2014-01-22 16:43 ` Dave Jones @ 2014-01-22 18:20 ` Linus Torvalds 2014-01-22 23:36 ` Jan Kara 1 sibling, 1 reply; 19+ messages in thread From: Linus Torvalds @ 2014-01-22 18:20 UTC (permalink / raw) To: Dave Jones, Jan Kara, Linux Kernel On Tue, Jan 21, 2014 at 10:27 PM, Dave Jones <davej@redhat.com> wrote: > > BUG fanotify_event_info (Not tainted): Poison overwritten Looking at the poison data, it seems that is is the u32 response; field that has been overwritten (with all zero). That doesn't really help me guess where the bug is, though. That code is crazy. For example, looking at one place where it is set, we have: - process_access_response(): re->event->response = response; wake_up(&group->fanotify_data.access_waitq); kmem_cache_free(fanotify_response_event_cache, re); which looks buggy in *so* many ways. In particular, we're doing a kmem_cache_free() on "re", but what happened to "re->event" that we just used? There was no release of that anywhere. Wut? So it seems that the lifetime of these "fanotify_event_info" structures is completely buggered. I don't even see any *attempt* to maintain reference counts or other lifetime info. People free the containers that point to them without doing anything at all about the fsnotify_event that contains the fanotify_event_info that they point to. Jan - how is the lifetime of the fanotify_event_info tied to the lifetime of the fanotify_response_event structure that contains pointers into it? Because I don't see it. And considering that it's the response field that gets overwritten, it really sounds like *that* is the exact issue at play here - there is some fanotify_response_event structure holding a pointer to the fanotify_event that is embedded into a fanotify_event_info that has been freed. Jan? Linus ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free. 2014-01-22 18:20 ` Linus Torvalds @ 2014-01-22 23:36 ` Jan Kara 2014-01-23 0:08 ` Linus Torvalds 0 siblings, 1 reply; 19+ messages in thread From: Jan Kara @ 2014-01-22 23:36 UTC (permalink / raw) To: Linus Torvalds; +Cc: Dave Jones, Jan Kara, Linux Kernel, jkosina [-- Attachment #1: Type: text/plain, Size: 2479 bytes --] On Wed 22-01-14 10:20:01, Linus Torvalds wrote: > On Tue, Jan 21, 2014 at 10:27 PM, Dave Jones <davej@redhat.com> wrote: > > > > BUG fanotify_event_info (Not tainted): Poison overwritten > > Looking at the poison data, it seems that is is the > > u32 response; > > field that has been overwritten (with all zero). > > That doesn't really help me guess where the bug is, though. That code > is crazy. For example, looking at one place where it is set, we have: > > - process_access_response(): > > re->event->response = response; > > wake_up(&group->fanotify_data.access_waitq); > > kmem_cache_free(fanotify_response_event_cache, re); > > which looks buggy in *so* many ways. In particular, we're doing a > kmem_cache_free() on "re", but what happened to "re->event" that we > just used? There was no release of that anywhere. Wut? > > So it seems that the lifetime of these "fanotify_event_info" > structures is completely buggered. I don't even see any *attempt* to > maintain reference counts or other lifetime info. People free the > containers that point to them without doing anything at all about the > fsnotify_event that contains the fanotify_event_info that they point > to. > > Jan - how is the lifetime of the fanotify_event_info tied to the > lifetime of the fanotify_response_event structure that contains > pointers into it? Because I don't see it. Yeah, I messed that up. They aren't tied in any way - well, in fact they end up being tied but in a wrong way. fanotify_event_info lives from the moment event happens to the moment user reads the event. At that moment the fanotify_response_event gets created (for those special permission events), pointing to fanotify_event_info which will get freed just several lines further :-| But refcounting seems like an overkill for this - there is exactly one fanotify_response_event structure iff it is a permission event. So something like the (completely untested) attached patch should fix the problem. But I agree it's a bit ugly so we might want something different. I'll try to think about something better tomorrow. > And considering that it's the response field that gets overwritten, it > really sounds like *that* is the exact issue at play here - there is > some fanotify_response_event structure holding a pointer to the > fanotify_event that is embedded into a fanotify_event_info that has > been freed. Honza -- Jan Kara <jack@suse.cz> SUSE Labs, CR [-- Attachment #2: fanotify_corruption.diff --] [-- Type: text/x-patch, Size: 1098 bytes --] diff --git a/fs/notify/fanotify/fanotify.c b/fs/notify/fanotify/fanotify.c index 58772623f02a..756e9b047e27 100644 --- a/fs/notify/fanotify/fanotify.c +++ b/fs/notify/fanotify/fanotify.c @@ -201,8 +201,10 @@ static int fanotify_handle_event(struct fsnotify_group *group, } #ifdef CONFIG_FANOTIFY_ACCESS_PERMISSIONS - if (fsn_event->mask & FAN_ALL_PERM_EVENTS) + if (fsn_event->mask & FAN_ALL_PERM_EVENTS) { ret = fanotify_get_response_from_access(group, event); + fsnotify_destroy_event(group, event); + } #endif return ret; } diff --git a/fs/notify/fanotify/fanotify_user.c b/fs/notify/fanotify/fanotify_user.c index 57d7c083cb4b..d493c72c71fd 100644 --- a/fs/notify/fanotify/fanotify_user.c +++ b/fs/notify/fanotify/fanotify_user.c @@ -319,7 +319,8 @@ static ssize_t fanotify_read(struct file *file, char __user *buf, if (IS_ERR(kevent)) break; ret = copy_event_to_user(group, kevent, buf); - fsnotify_destroy_event(group, kevent); + if (!(kevent->mask & FAN_ALL_PERM_EVENTS)) + fsnotify_destroy_event(group, kevent); if (ret < 0) break; buf += ret; ^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: fanotify use after free. 2014-01-22 23:36 ` Jan Kara @ 2014-01-23 0:08 ` Linus Torvalds 2014-01-23 0:32 ` Dave Jones 2014-01-23 10:23 ` Jiri Kosina 0 siblings, 2 replies; 19+ messages in thread From: Linus Torvalds @ 2014-01-23 0:08 UTC (permalink / raw) To: Jan Kara; +Cc: Dave Jones, Linux Kernel, Jiri Kosina On Wed, Jan 22, 2014 at 3:36 PM, Jan Kara <jack@suse.cz> wrote: > > But refcounting seems like an overkill for this - there is exactly one > fanotify_response_event structure iff it is a permission event. So > something like the (completely untested) attached patch should fix the > problem. But I agree it's a bit ugly so we might want something different. > I'll try to think about something better tomorrow. Ok, In the meantime, Dave, can you verify whether this hacky patch fixes your problem? Linus ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free. 2014-01-23 0:08 ` Linus Torvalds @ 2014-01-23 0:32 ` Dave Jones 2014-01-23 15:05 ` Jan Kara 2014-01-23 10:23 ` Jiri Kosina 1 sibling, 1 reply; 19+ messages in thread From: Dave Jones @ 2014-01-23 0:32 UTC (permalink / raw) To: Linus Torvalds; +Cc: Jan Kara, Linux Kernel, Jiri Kosina On Wed, Jan 22, 2014 at 04:08:52PM -0800, Linus Torvalds wrote: > On Wed, Jan 22, 2014 at 3:36 PM, Jan Kara <jack@suse.cz> wrote: > > > > But refcounting seems like an overkill for this - there is exactly one > > fanotify_response_event structure iff it is a permission event. So > > something like the (completely untested) attached patch should fix the > > problem. But I agree it's a bit ugly so we might want something different. > > I'll try to think about something better tomorrow. > > Ok, In the meantime, Dave, can you verify whether this hacky patch > fixes your problem? It actually seems worse. I see the tail end of what looks like a slab corruption trace, and then a total lockup. And of course none of this makes it over ttyUSB0 because it happens so early. Grr. Dave ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free. 2014-01-23 0:32 ` Dave Jones @ 2014-01-23 15:05 ` Jan Kara 0 siblings, 0 replies; 19+ messages in thread From: Jan Kara @ 2014-01-23 15:05 UTC (permalink / raw) To: Dave Jones; +Cc: Linus Torvalds, Jan Kara, Linux Kernel, Jiri Kosina On Wed 22-01-14 19:32:40, Dave Jones wrote: > On Wed, Jan 22, 2014 at 04:08:52PM -0800, Linus Torvalds wrote: > > On Wed, Jan 22, 2014 at 3:36 PM, Jan Kara <jack@suse.cz> wrote: > > > > > > But refcounting seems like an overkill for this - there is exactly one > > > fanotify_response_event structure iff it is a permission event. So > > > something like the (completely untested) attached patch should fix the > > > problem. But I agree it's a bit ugly so we might want something different. > > > I'll try to think about something better tomorrow. > > > > Ok, In the meantime, Dave, can you verify whether this hacky patch > > fixes your problem? > > It actually seems worse. I see the tail end of what looks like a slab corruption > trace, and then a total lockup. And of course none of this makes it over ttyUSB0 > because it happens so early. Grr. Drat. Since this seems reasonably reproducible, I'll try to reproduce and debug this (it seems systemd is using fanotify in a way that triggers this). Thanks for testing. Honza -- Jan Kara <jack@suse.cz> SUSE Labs, CR ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free. 2014-01-23 0:08 ` Linus Torvalds 2014-01-23 0:32 ` Dave Jones @ 2014-01-23 10:23 ` Jiri Kosina 2014-01-23 15:05 ` Jan Kara 1 sibling, 1 reply; 19+ messages in thread From: Jiri Kosina @ 2014-01-23 10:23 UTC (permalink / raw) To: Linus Torvalds, Jan Kara; +Cc: Dave Jones, Linux Kernel On Wed, 22 Jan 2014, Linus Torvalds wrote: > > But refcounting seems like an overkill for this - there is exactly one > > fanotify_response_event structure iff it is a permission event. So > > something like the (completely untested) attached patch should fix the > > problem. But I agree it's a bit ugly so we might want something different. > > I'll try to think about something better tomorrow. > > Ok, In the meantime, Dave, can you verify whether this hacky patch > fixes your problem? I reported the same slab corruption yesterday as well here: https://lkml.org/lkml/2014/1/22/173 With the patch applied, I am still seeing the slab corruption, preceeded by GPF (which is not there without the patch) in lockref_put_or_lock(&dentry->d_lockref) in dput(): general protection fault: 0000 [#1] SMP Modules linked in: tpm_tis(+) tpm wmi acpi_cpufreq autofs4 uhci_hcd ehci_hcd i915 drm_kms_helper drm i2c_algo_bit button usbcore video usb_common edd fan processor ata_generic thermal thermal_sys CPU: 1 PID: 275 Comm: systemd-readahe Not tainted 3.13.0-03478-g670d0ac #1 Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008 task: ffff880037c09150 ti: ffff88007359e000 task.ti: ffff88007359e000 RIP: 0010:[<ffffffff810a51e7>] [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160 RSP: 0018:ffff88007359fc78 EFLAGS: 00010286 RAX: ffff880037c09150 RBX: 6b6b6b6b6b6b6beb RCX: 0000000000000000 tpm_tis 00:09: 1.2 TPM (device-id 0x1020, rev-id 6) tpm_tis 00:09: Intel iTPM workaround enabled RDX: 0000000000000000 RSI: 0000000000000000 RDI: 6b6b6b6b6b6b6beb RBP: ffff88007359fc98 R08: 0000000000000002 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000 R12: 6b6b6b6b6b6b6beb R13: ffff880037310690 R14: 0000000000000020 R15: ffff880036dbfc10 FS: 00007fa953e4a700(0000) GS:ffff88007c280000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fff76a10258 CR3: 000000003659c000 CR4: 00000000000007e0 Stack: 6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb ffff880037310690 ffff88007359fcb8 ffffffff8159c59c ffffffff812fe101 6b6b6b6b6b6b6beb ffff88007359fcd8 ffffffff812fe101 ffff88007359fd08 6b6b6b6b6b6b6b6b Call Trace: [<ffffffff8159c59c>] _raw_spin_lock+0x3c/0x50 [<ffffffff812fe101>] ? lockref_put_or_lock+0x11/0x40 [<ffffffff812fe101>] lockref_put_or_lock+0x11/0x40 [<ffffffff811b1442>] dput+0x22/0x130 [<ffffffff811a3d45>] path_put+0x15/0x30 [<ffffffff811e0bc5>] fanotify_free_event+0x15/0x40 [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30 [<ffffffff811e1041>] fanotify_handle_event+0x341/0x390 [<ffffffff811dd18b>] send_to_group+0xfb/0x180 [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0 [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0 [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0 [<ffffffff811987ad>] do_sys_open+0x1ad/0x220 [<ffffffff812fdd6e>] ? trace_hardirqs_on_thunk+0x3a/0x3f [<ffffffff81198859>] SyS_open+0x19/0x20 [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b Code: 0d 7e 81 48 89 df e8 29 ff ff ff eb 94 0f 1f 80 00 00 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 48 89 fb 4c 89 6d f8 <81> 7f 04 ad 4e ad de 74 0c 48 c7 c6 b5 0d 7e 81 e8 f4 fe ff ff RIP [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160 RSP <ffff88007359fc78> ---[ end trace 7a918209ee213d28 ]--- BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:20 in_atomic(): 1, irqs_disabled(): 0, pid: 275, name: systemd-readahe INFO: lockdep is turned off. CPU: 1 PID: 275 Comm: systemd-readahe Tainted: G D 3.13.0-03478-g670d0ac #1 Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008 ffff880037c09150 ffff88007359fa78 ffffffff8159702b ffff88007359fa98 ffffffff8107f621 ffffffff81a3d000 ffff880037b98d90 ffff88007359fab8 ffffffff8159b4ff ffff880037c09150 ffff880037c09150 ffff88007359fae8 Call Trace: [<ffffffff8159702b>] dump_stack+0x72/0x87 [<ffffffff8107f621>] __might_sleep+0xe1/0x100 [<ffffffff8159b4ff>] down_read+0x1f/0x60 [<ffffffff810627ff>] exit_signals+0x1f/0x140 [<ffffffff81079491>] ? blocking_notifier_call_chain+0x11/0x20 [<ffffffff81052844>] do_exit+0xb4/0x4b0 [<ffffffff8159e23c>] oops_end+0xdc/0xe0 [<ffffffff81005f86>] die+0x56/0x90 [<ffffffff8159dea2>] do_general_protection+0x162/0x170 [<ffffffff8159d40c>] ? restore_args+0x30/0x30 [<ffffffff8159d592>] general_protection+0x22/0x30 [<ffffffff810a51e7>] ? do_raw_spin_lock+0x17/0x160 [<ffffffff8159c59c>] _raw_spin_lock+0x3c/0x50 [<ffffffff812fe101>] ? lockref_put_or_lock+0x11/0x40 [<ffffffff812fe101>] lockref_put_or_lock+0x11/0x40 [<ffffffff811b1442>] dput+0x22/0x130 [<ffffffff811a3d45>] path_put+0x15/0x30 [<ffffffff811e0bc5>] fanotify_free_event+0x15/0x40 [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30 [<ffffffff811e1041>] fanotify_handle_event+0x341/0x390 [<ffffffff811dd18b>] send_to_group+0xfb/0x180 [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0 [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0 [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0 [<ffffffff811987ad>] do_sys_open+0x1ad/0x220 [<ffffffff812fdd6e>] ? trace_hardirqs_on_thunk+0x3a/0x3f [<ffffffff81198859>] SyS_open+0x19/0x20 [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b note: systemd-readahe[275] exited with preempt_count 1 BUG: scheduling while atomic: systemd-readahe/275/0x00000002 INFO: lockdep is turned off. Modules linked in: tpm_tis(+) tpm wmi acpi_cpufreq autofs4 uhci_hcd ehci_hcd i915 drm_kms_helper drm i2c_algo_bit button usbcore video usb_common edd fan processor ata_generic thermal thermal_sys CPU: 1 PID: 275 Comm: systemd-readahe Tainted: G D 3.13.0-03478-g670d0ac #1 Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008 ffff88007c293a00 ffff88007359f6f8 ffffffff8159702b ffff88007359f718 ffffffff810810f1 ffff88007c293a00 0000000000000001 ffff88007359f848 ffffffff8159789c ffff88007359f758 ffff88007359f768 ffff88007359e010 Call Trace: [<ffffffff8159702b>] dump_stack+0x72/0x87 [<ffffffff810810f1>] __schedule_bug+0x61/0x80 [<ffffffff8159789c>] __schedule+0xbc/0x7c0 [<ffffffff8105defc>] ? mod_timer+0x14c/0x1f0 [<ffffffff815980e4>] schedule+0x24/0x70 [<ffffffff81597205>] schedule_timeout+0x1c5/0x210 [<ffffffff8159914f>] ? wait_for_completion+0xcf/0x120 [<ffffffff810a0d8d>] ? trace_hardirqs_on+0xd/0x10 [<ffffffff81599157>] wait_for_completion+0xd7/0x120 [<ffffffff81083330>] ? try_to_wake_up+0x250/0x250 [<ffffffff810b93bf>] ? srcu_reschedule+0x4f/0xf0 [<ffffffff810b965c>] __synchronize_srcu+0xec/0x130 [<ffffffff810b96e0>] ? srcu_barrier+0x10/0x10 [<ffffffff810b96c8>] synchronize_srcu+0x18/0x20 [<ffffffff811ddbdd>] fsnotify_destroy_group+0x1d/0x40 [<ffffffff811dfdf1>] inotify_release+0x21/0x50 [<ffffffff8119b2dd>] __fput+0xbd/0x2b0 [<ffffffff8119b569>] ____fput+0x9/0x10 [<ffffffff81070f41>] task_work_run+0xb1/0xe0 [<ffffffff81052979>] do_exit+0x1e9/0x4b0 [<ffffffff8159e23c>] oops_end+0xdc/0xe0 [<ffffffff81005f86>] die+0x56/0x90 [<ffffffff8159dea2>] do_general_protection+0x162/0x170 [<ffffffff8159d40c>] ? restore_args+0x30/0x30 [<ffffffff8159d592>] general_protection+0x22/0x30 [<ffffffff810a51e7>] ? do_raw_spin_lock+0x17/0x160 [<ffffffff8159c59c>] _raw_spin_lock+0x3c/0x50 [<ffffffff812fe101>] ? lockref_put_or_lock+0x11/0x40 [<ffffffff812fe101>] lockref_put_or_lock+0x11/0x40 [<ffffffff811b1442>] dput+0x22/0x130 [<ffffffff811a3d45>] path_put+0x15/0x30 [<ffffffff811e0bc5>] fanotify_free_event+0x15/0x40 [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30 [<ffffffff811e1041>] fanotify_handle_event+0x341/0x390 [<ffffffff811dd18b>] send_to_group+0xfb/0x180 [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0 [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0 [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0 [<ffffffff811987ad>] do_sys_open+0x1ad/0x220 [<ffffffff812fdd6e>] ? trace_hardirqs_on_thunk+0x3a/0x3f [<ffffffff81198859>] SyS_open+0x19/0x20 [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b ACPI: AC Adapter [AC] (on-line) Slab corruption (Tainted: G D W ): fanotify_event_info start=ffff880037310690, len=64 Redzone: 0x9f911029d74e35b/0x9f911029d74e35b. Last user: [<ffffffff811e0bdd>](fanotify_free_event+0x2d/0x40) 030: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 6b 6b 6b a5 kkkkkkkk....kkk. Prev obj: start=ffff880037310638, len=64 Redzone: 0x9f911029d74e35b/0x9f911029d74e35b. Last user: [<ffffffff811e0bdd>](fanotify_free_event+0x2d/0x40) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Next obj: start=ffff8800373106e8, len=64 Redzone: 0x9f911029d74e35b/0x9f911029d74e35b. Last user: [<ffffffff811e0bdd>](fanotify_free_event+0x2d/0x40) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk -- Jiri Kosina SUSE Labs ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free. 2014-01-23 10:23 ` Jiri Kosina @ 2014-01-23 15:05 ` Jan Kara 2014-01-23 15:07 ` Jiri Kosina 0 siblings, 1 reply; 19+ messages in thread From: Jan Kara @ 2014-01-23 15:05 UTC (permalink / raw) To: Jiri Kosina; +Cc: Linus Torvalds, Jan Kara, Dave Jones, Linux Kernel On Thu 23-01-14 11:23:53, Jiri Kosina wrote: > On Wed, 22 Jan 2014, Linus Torvalds wrote: > > > > But refcounting seems like an overkill for this - there is exactly one > > > fanotify_response_event structure iff it is a permission event. So > > > something like the (completely untested) attached patch should fix the > > > problem. But I agree it's a bit ugly so we might want something different. > > > I'll try to think about something better tomorrow. > > > > Ok, In the meantime, Dave, can you verify whether this hacky patch > > fixes your problem? > > I reported the same slab corruption yesterday as well here: > > https://lkml.org/lkml/2014/1/22/173 > > With the patch applied, I am still seeing the slab corruption, preceeded > by GPF (which is not there without the patch) in > lockref_put_or_lock(&dentry->d_lockref) in dput(): Hmm, OK. Can you please send me your .config? I'll try to reproduce this myself. Honza > > general protection fault: 0000 [#1] SMP > Modules linked in: tpm_tis(+) tpm wmi acpi_cpufreq autofs4 uhci_hcd ehci_hcd i915 drm_kms_helper drm i2c_algo_bit button usbcore video usb_common edd fan processor ata_generic thermal thermal_sys > CPU: 1 PID: 275 Comm: systemd-readahe Not tainted 3.13.0-03478-g670d0ac #1 > Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008 > task: ffff880037c09150 ti: ffff88007359e000 task.ti: ffff88007359e000 > RIP: 0010:[<ffffffff810a51e7>] [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160 > RSP: 0018:ffff88007359fc78 EFLAGS: 00010286 > RAX: ffff880037c09150 RBX: 6b6b6b6b6b6b6beb RCX: 0000000000000000 > tpm_tis 00:09: 1.2 TPM (device-id 0x1020, rev-id 6) > tpm_tis 00:09: Intel iTPM workaround enabled > RDX: 0000000000000000 RSI: 0000000000000000 RDI: 6b6b6b6b6b6b6beb > RBP: ffff88007359fc98 R08: 0000000000000002 R09: 0000000000000000 > R10: 0000000000000000 R11: 0000000000000000 R12: 6b6b6b6b6b6b6beb > R13: ffff880037310690 R14: 0000000000000020 R15: ffff880036dbfc10 > FS: 00007fa953e4a700(0000) GS:ffff88007c280000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > CR2: 00007fff76a10258 CR3: 000000003659c000 CR4: 00000000000007e0 > Stack: > 6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb ffff880037310690 > ffff88007359fcb8 ffffffff8159c59c ffffffff812fe101 6b6b6b6b6b6b6beb > ffff88007359fcd8 ffffffff812fe101 ffff88007359fd08 6b6b6b6b6b6b6b6b > Call Trace: > [<ffffffff8159c59c>] _raw_spin_lock+0x3c/0x50 > [<ffffffff812fe101>] ? lockref_put_or_lock+0x11/0x40 > [<ffffffff812fe101>] lockref_put_or_lock+0x11/0x40 > [<ffffffff811b1442>] dput+0x22/0x130 > [<ffffffff811a3d45>] path_put+0x15/0x30 > [<ffffffff811e0bc5>] fanotify_free_event+0x15/0x40 > [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30 > [<ffffffff811e1041>] fanotify_handle_event+0x341/0x390 > [<ffffffff811dd18b>] send_to_group+0xfb/0x180 > [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0 > [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0 > [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0 > [<ffffffff811987ad>] do_sys_open+0x1ad/0x220 > [<ffffffff812fdd6e>] ? trace_hardirqs_on_thunk+0x3a/0x3f > [<ffffffff81198859>] SyS_open+0x19/0x20 > [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b > Code: 0d 7e 81 48 89 df e8 29 ff ff ff eb 94 0f 1f 80 00 00 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 48 89 fb 4c 89 6d f8 <81> 7f 04 ad 4e ad de 74 0c 48 c7 c6 b5 0d 7e 81 e8 f4 fe ff ff > RIP [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160 > RSP <ffff88007359fc78> > ---[ end trace 7a918209ee213d28 ]--- > BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:20 > in_atomic(): 1, irqs_disabled(): 0, pid: 275, name: systemd-readahe > INFO: lockdep is turned off. > CPU: 1 PID: 275 Comm: systemd-readahe Tainted: G D 3.13.0-03478-g670d0ac #1 > Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008 > ffff880037c09150 ffff88007359fa78 ffffffff8159702b ffff88007359fa98 > ffffffff8107f621 ffffffff81a3d000 ffff880037b98d90 ffff88007359fab8 > ffffffff8159b4ff ffff880037c09150 ffff880037c09150 ffff88007359fae8 > Call Trace: > [<ffffffff8159702b>] dump_stack+0x72/0x87 > [<ffffffff8107f621>] __might_sleep+0xe1/0x100 > [<ffffffff8159b4ff>] down_read+0x1f/0x60 > [<ffffffff810627ff>] exit_signals+0x1f/0x140 > [<ffffffff81079491>] ? blocking_notifier_call_chain+0x11/0x20 > [<ffffffff81052844>] do_exit+0xb4/0x4b0 > [<ffffffff8159e23c>] oops_end+0xdc/0xe0 > [<ffffffff81005f86>] die+0x56/0x90 > [<ffffffff8159dea2>] do_general_protection+0x162/0x170 > [<ffffffff8159d40c>] ? restore_args+0x30/0x30 > [<ffffffff8159d592>] general_protection+0x22/0x30 > [<ffffffff810a51e7>] ? do_raw_spin_lock+0x17/0x160 > [<ffffffff8159c59c>] _raw_spin_lock+0x3c/0x50 > [<ffffffff812fe101>] ? lockref_put_or_lock+0x11/0x40 > [<ffffffff812fe101>] lockref_put_or_lock+0x11/0x40 > [<ffffffff811b1442>] dput+0x22/0x130 > [<ffffffff811a3d45>] path_put+0x15/0x30 > [<ffffffff811e0bc5>] fanotify_free_event+0x15/0x40 > [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30 > [<ffffffff811e1041>] fanotify_handle_event+0x341/0x390 > [<ffffffff811dd18b>] send_to_group+0xfb/0x180 > [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0 > [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0 > [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0 > [<ffffffff811987ad>] do_sys_open+0x1ad/0x220 > [<ffffffff812fdd6e>] ? trace_hardirqs_on_thunk+0x3a/0x3f > [<ffffffff81198859>] SyS_open+0x19/0x20 > [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b > note: systemd-readahe[275] exited with preempt_count 1 > BUG: scheduling while atomic: systemd-readahe/275/0x00000002 > INFO: lockdep is turned off. > Modules linked in: tpm_tis(+) tpm wmi acpi_cpufreq autofs4 uhci_hcd ehci_hcd i915 drm_kms_helper drm i2c_algo_bit button usbcore video usb_common edd fan processor ata_generic thermal thermal_sys > CPU: 1 PID: 275 Comm: systemd-readahe Tainted: G D 3.13.0-03478-g670d0ac #1 > Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008 > ffff88007c293a00 ffff88007359f6f8 ffffffff8159702b ffff88007359f718 > ffffffff810810f1 ffff88007c293a00 0000000000000001 ffff88007359f848 > ffffffff8159789c ffff88007359f758 ffff88007359f768 ffff88007359e010 > Call Trace: > [<ffffffff8159702b>] dump_stack+0x72/0x87 > [<ffffffff810810f1>] __schedule_bug+0x61/0x80 > [<ffffffff8159789c>] __schedule+0xbc/0x7c0 > [<ffffffff8105defc>] ? mod_timer+0x14c/0x1f0 > [<ffffffff815980e4>] schedule+0x24/0x70 > [<ffffffff81597205>] schedule_timeout+0x1c5/0x210 > [<ffffffff8159914f>] ? wait_for_completion+0xcf/0x120 > [<ffffffff810a0d8d>] ? trace_hardirqs_on+0xd/0x10 > [<ffffffff81599157>] wait_for_completion+0xd7/0x120 > [<ffffffff81083330>] ? try_to_wake_up+0x250/0x250 > [<ffffffff810b93bf>] ? srcu_reschedule+0x4f/0xf0 > [<ffffffff810b965c>] __synchronize_srcu+0xec/0x130 > [<ffffffff810b96e0>] ? srcu_barrier+0x10/0x10 > [<ffffffff810b96c8>] synchronize_srcu+0x18/0x20 > [<ffffffff811ddbdd>] fsnotify_destroy_group+0x1d/0x40 > [<ffffffff811dfdf1>] inotify_release+0x21/0x50 > [<ffffffff8119b2dd>] __fput+0xbd/0x2b0 > [<ffffffff8119b569>] ____fput+0x9/0x10 > [<ffffffff81070f41>] task_work_run+0xb1/0xe0 > [<ffffffff81052979>] do_exit+0x1e9/0x4b0 > [<ffffffff8159e23c>] oops_end+0xdc/0xe0 > [<ffffffff81005f86>] die+0x56/0x90 > [<ffffffff8159dea2>] do_general_protection+0x162/0x170 > [<ffffffff8159d40c>] ? restore_args+0x30/0x30 > [<ffffffff8159d592>] general_protection+0x22/0x30 > [<ffffffff810a51e7>] ? do_raw_spin_lock+0x17/0x160 > [<ffffffff8159c59c>] _raw_spin_lock+0x3c/0x50 > [<ffffffff812fe101>] ? lockref_put_or_lock+0x11/0x40 > [<ffffffff812fe101>] lockref_put_or_lock+0x11/0x40 > [<ffffffff811b1442>] dput+0x22/0x130 > [<ffffffff811a3d45>] path_put+0x15/0x30 > [<ffffffff811e0bc5>] fanotify_free_event+0x15/0x40 > [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30 > [<ffffffff811e1041>] fanotify_handle_event+0x341/0x390 > [<ffffffff811dd18b>] send_to_group+0xfb/0x180 > [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0 > [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0 > [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0 > [<ffffffff811987ad>] do_sys_open+0x1ad/0x220 > [<ffffffff812fdd6e>] ? trace_hardirqs_on_thunk+0x3a/0x3f > [<ffffffff81198859>] SyS_open+0x19/0x20 > [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b > ACPI: AC Adapter [AC] (on-line) > Slab corruption (Tainted: G D W ): fanotify_event_info start=ffff880037310690, len=64 > Redzone: 0x9f911029d74e35b/0x9f911029d74e35b. > Last user: [<ffffffff811e0bdd>](fanotify_free_event+0x2d/0x40) > 030: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 6b 6b 6b a5 kkkkkkkk....kkk. > Prev obj: start=ffff880037310638, len=64 > Redzone: 0x9f911029d74e35b/0x9f911029d74e35b. > Last user: [<ffffffff811e0bdd>](fanotify_free_event+0x2d/0x40) > 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk > 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk > Next obj: start=ffff8800373106e8, len=64 > Redzone: 0x9f911029d74e35b/0x9f911029d74e35b. > Last user: [<ffffffff811e0bdd>](fanotify_free_event+0x2d/0x40) > 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk > 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk > > -- > Jiri Kosina > SUSE Labs -- Jan Kara <jack@suse.cz> SUSE Labs, CR ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free. 2014-01-23 15:05 ` Jan Kara @ 2014-01-23 15:07 ` Jiri Kosina 2014-01-23 23:55 ` Jan Kara 0 siblings, 1 reply; 19+ messages in thread From: Jiri Kosina @ 2014-01-23 15:07 UTC (permalink / raw) To: Jan Kara; +Cc: Linus Torvalds, Dave Jones, Linux Kernel [-- Attachment #1: Type: TEXT/PLAIN, Size: 983 bytes --] On Thu, 23 Jan 2014, Jan Kara wrote: > > > > But refcounting seems like an overkill for this - there is exactly one > > > > fanotify_response_event structure iff it is a permission event. So > > > > something like the (completely untested) attached patch should fix the > > > > problem. But I agree it's a bit ugly so we might want something different. > > > > I'll try to think about something better tomorrow. > > > > > > Ok, In the meantime, Dave, can you verify whether this hacky patch > > > fixes your problem? > > > > I reported the same slab corruption yesterday as well here: > > > > https://lkml.org/lkml/2014/1/22/173 > > > > With the patch applied, I am still seeing the slab corruption, preceeded > > by GPF (which is not there without the patch) in > > lockref_put_or_lock(&dentry->d_lockref) in dput(): > Hmm, OK. Can you please send me your .config? I'll try to reproduce this > myself. Attached. The userspace is systemd-based. -- Jiri Kosina SUSE Labs [-- Attachment #2: Type: APPLICATION/x-gzip, Size: 22901 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free. 2014-01-23 15:07 ` Jiri Kosina @ 2014-01-23 23:55 ` Jan Kara 2014-01-24 7:26 ` Jiri Kosina 0 siblings, 1 reply; 19+ messages in thread From: Jan Kara @ 2014-01-23 23:55 UTC (permalink / raw) To: Jiri Kosina; +Cc: Jan Kara, Linus Torvalds, Dave Jones, Linux Kernel On Thu 23-01-14 16:07:45, Jiri Kosina wrote: > On Thu, 23 Jan 2014, Jan Kara wrote: > > > > > > But refcounting seems like an overkill for this - there is exactly one > > > > > fanotify_response_event structure iff it is a permission event. So > > > > > something like the (completely untested) attached patch should fix the > > > > > problem. But I agree it's a bit ugly so we might want something different. > > > > > I'll try to think about something better tomorrow. > > > > > > > > Ok, In the meantime, Dave, can you verify whether this hacky patch > > > > fixes your problem? > > > > > > I reported the same slab corruption yesterday as well here: > > > > > > https://lkml.org/lkml/2014/1/22/173 > > > > > > With the patch applied, I am still seeing the slab corruption, preceeded > > > by GPF (which is not there without the patch) in > > > lockref_put_or_lock(&dentry->d_lockref) in dput(): > > Hmm, OK. Can you please send me your .config? I'll try to reproduce this > > myself. > > Attached. > > The userspace is systemd-based. Strange. I've installed systemd system (openSUSE 13.1) and it boots with the latest Linus' kernel just fine (and I have at least FANOTIFY and SLAB debugging set the same way as you). But it was only a KVM guest. I'll try tomorrow with a physical machine I guess. Honza -- Jan Kara <jack@suse.cz> SUSE Labs, CR ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free. 2014-01-23 23:55 ` Jan Kara @ 2014-01-24 7:26 ` Jiri Kosina 2014-01-27 23:40 ` Jan Kara 0 siblings, 1 reply; 19+ messages in thread From: Jiri Kosina @ 2014-01-24 7:26 UTC (permalink / raw) To: Jan Kara; +Cc: Linus Torvalds, Dave Jones, Linux Kernel On Fri, 24 Jan 2014, Jan Kara wrote: > Strange. I've installed systemd system (openSUSE 13.1) and it boots > with the latest Linus' kernel just fine (and I have at least FANOTIFY > and SLAB debugging set the same way as you). But it was only a KVM > guest. I'll try tomorrow with a physical machine I guess. FWIW the system I am reliably able to reproduce this on is opensuse 12.3 with this systemd version: Version : 195 Release : 13.18.1 -- Jiri Kosina SUSE Labs ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free. 2014-01-24 7:26 ` Jiri Kosina @ 2014-01-27 23:40 ` Jan Kara 2014-01-28 6:10 ` Dave Jones 2014-01-28 10:53 ` Jiri Kosina 0 siblings, 2 replies; 19+ messages in thread From: Jan Kara @ 2014-01-27 23:40 UTC (permalink / raw) To: Jiri Kosina; +Cc: Jan Kara, Linus Torvalds, Dave Jones, Linux Kernel [-- Attachment #1: Type: text/plain, Size: 1000 bytes --] On Fri 24-01-14 08:26:45, Jiri Kosina wrote: > On Fri, 24 Jan 2014, Jan Kara wrote: > > > Strange. I've installed systemd system (openSUSE 13.1) and it boots > > with the latest Linus' kernel just fine (and I have at least FANOTIFY > > and SLAB debugging set the same way as you). But it was only a KVM > > guest. I'll try tomorrow with a physical machine I guess. > > FWIW the system I am reliably able to reproduce this on is opensuse 12.3 > with this systemd version: > > Version : 195 > Release : 13.18.1 Hum, still no luck with reproduction (either on physical machine or with KVM). Anyway, I've looked at the code again and the previous patch had a stupid bug (passing different pointer to fsnotify_destroy_event() than we should have), plus also the merging function in fanotify was too aggressive. Can you try the attached patch? It boots for me but that means nothing since I cannot reproduce the issue... Thanks! Honza -- Jan Kara <jack@suse.cz> SUSE Labs, CR [-- Attachment #2: fanotify.diff --] [-- Type: text/x-patch, Size: 1737 bytes --] diff --git a/fs/notify/fanotify/fanotify.c b/fs/notify/fanotify/fanotify.c index 58772623f02a..1b3dd9de8518 100644 --- a/fs/notify/fanotify/fanotify.c +++ b/fs/notify/fanotify/fanotify.c @@ -18,7 +18,7 @@ static bool should_merge(struct fsnotify_event *old_fsn, #ifdef CONFIG_FANOTIFY_ACCESS_PERMISSIONS /* dont merge two permission events */ - if ((old_fsn->mask & FAN_ALL_PERM_EVENTS) && + if ((old_fsn->mask & FAN_ALL_PERM_EVENTS) || (new_fsn->mask & FAN_ALL_PERM_EVENTS)) return false; #endif @@ -201,8 +201,10 @@ static int fanotify_handle_event(struct fsnotify_group *group, } #ifdef CONFIG_FANOTIFY_ACCESS_PERMISSIONS - if (fsn_event->mask & FAN_ALL_PERM_EVENTS) + if (fsn_event->mask & FAN_ALL_PERM_EVENTS) { ret = fanotify_get_response_from_access(group, event); + fsnotify_destroy_event(group, fsn_event); + } #endif return ret; } @@ -221,7 +223,8 @@ static void fanotify_free_event(struct fsnotify_event *fsn_event) struct fanotify_event_info *event; event = FANOTIFY_E(fsn_event); - path_put(&event->path); + if (event->path.mnt) + path_put(&event->path); put_pid(event->tgid); kmem_cache_free(fanotify_event_cachep, event); } diff --git a/fs/notify/fanotify/fanotify_user.c b/fs/notify/fanotify/fanotify_user.c index 57d7c083cb4b..d493c72c71fd 100644 --- a/fs/notify/fanotify/fanotify_user.c +++ b/fs/notify/fanotify/fanotify_user.c @@ -319,7 +319,8 @@ static ssize_t fanotify_read(struct file *file, char __user *buf, if (IS_ERR(kevent)) break; ret = copy_event_to_user(group, kevent, buf); - fsnotify_destroy_event(group, kevent); + if (!(kevent->mask & FAN_ALL_PERM_EVENTS)) + fsnotify_destroy_event(group, kevent); if (ret < 0) break; buf += ret; ^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: fanotify use after free. 2014-01-27 23:40 ` Jan Kara @ 2014-01-28 6:10 ` Dave Jones 2014-01-28 8:02 ` Jan Kara 2014-01-28 10:53 ` Jiri Kosina 1 sibling, 1 reply; 19+ messages in thread From: Dave Jones @ 2014-01-28 6:10 UTC (permalink / raw) To: Jan Kara; +Cc: Jiri Kosina, Linus Torvalds, Linux Kernel On Tue, Jan 28, 2014 at 12:40:17AM +0100, Jan Kara wrote: > On Fri 24-01-14 08:26:45, Jiri Kosina wrote: > > On Fri, 24 Jan 2014, Jan Kara wrote: > > > > > Strange. I've installed systemd system (openSUSE 13.1) and it boots > > > with the latest Linus' kernel just fine (and I have at least FANOTIFY > > > and SLAB debugging set the same way as you). But it was only a KVM > > > guest. I'll try tomorrow with a physical machine I guess. > > > > FWIW the system I am reliably able to reproduce this on is opensuse 12.3 > > with this systemd version: > > > > Version : 195 > > Release : 13.18.1 > Hum, still no luck with reproduction (either on physical machine or with > KVM). Anyway, I've looked at the code again and the previous patch had a > stupid bug (passing different pointer to fsnotify_destroy_event() than we > should have), plus also the merging function in fanotify was too > aggressive. Can you try the attached patch? It boots for me but that means > nothing since I cannot reproduce the issue... Thanks! still not good I'm afraid. I still see corruption very early on in boot and now it panics and locks up too. Again, this happens so early that I can't grab it over usb-serial. I stuck an mdelay(10000) in the slub corruption detector, and managed to grab a photo of the first trace. Trace: ? preempt_schedule lock_acquire ? lockref_put_or_lock _raw_spin_lock ? lockref_put_or_lock dput path_put fanotify_free_event fsnotify_destroy_event fanotify_handle_event ? mntput ? path_openat ? handle_mm_fault send_to_group ? fsnotify fsnotify do_sys_open sys_open RIP: lock_acquire 2b:* 4d 8b 64 c6 08 mov 0x8(%r14,%rax,8),%r12 <-- trapping instruction R14 is 0x6b6b6b6b6b6b6c03, which looks like a use-after-free. I also notice you mention SLAB above, but I've been using SLUB. I don't know if the choice of allocator makes a difference in reproducability. It's also worth noting that I have lockdep enabled, which may be perturbing things to some degree. Dave ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free. 2014-01-28 6:10 ` Dave Jones @ 2014-01-28 8:02 ` Jan Kara 2014-01-28 11:07 ` Jiri Kosina 0 siblings, 1 reply; 19+ messages in thread From: Jan Kara @ 2014-01-28 8:02 UTC (permalink / raw) To: Dave Jones; +Cc: Jan Kara, Jiri Kosina, Linus Torvalds, Linux Kernel [-- Attachment #1: Type: text/plain, Size: 3153 bytes --] On Tue 28-01-14 01:10:37, Dave Jones wrote: > On Tue, Jan 28, 2014 at 12:40:17AM +0100, Jan Kara wrote: > > On Fri 24-01-14 08:26:45, Jiri Kosina wrote: > > > On Fri, 24 Jan 2014, Jan Kara wrote: > > > > > > > Strange. I've installed systemd system (openSUSE 13.1) and it boots > > > > with the latest Linus' kernel just fine (and I have at least FANOTIFY > > > > and SLAB debugging set the same way as you). But it was only a KVM > > > > guest. I'll try tomorrow with a physical machine I guess. > > > > > > FWIW the system I am reliably able to reproduce this on is opensuse 12.3 > > > with this systemd version: > > > > > > Version : 195 > > > Release : 13.18.1 > > Hum, still no luck with reproduction (either on physical machine or with > > KVM). Anyway, I've looked at the code again and the previous patch had a > > stupid bug (passing different pointer to fsnotify_destroy_event() than we > > should have), plus also the merging function in fanotify was too > > aggressive. Can you try the attached patch? It boots for me but that means > > nothing since I cannot reproduce the issue... Thanks! > > still not good I'm afraid. I still see corruption very early on in boot > and now it panics and locks up too. Ew, thanks for testing. > Again, this happens so early that I can't grab it over usb-serial. > I stuck an mdelay(10000) in the slub corruption detector, and managed > to grab a photo of the first trace. > > Trace: > ? preempt_schedule > lock_acquire > ? lockref_put_or_lock > _raw_spin_lock > ? lockref_put_or_lock > dput > path_put > fanotify_free_event > fsnotify_destroy_event > fanotify_handle_event > ? mntput > ? path_openat > ? handle_mm_fault > send_to_group > ? fsnotify > fsnotify > do_sys_open > sys_open > RIP: lock_acquire > > 2b:* 4d 8b 64 c6 08 mov 0x8(%r14,%rax,8),%r12 <-- trapping instruction > > R14 is 0x6b6b6b6b6b6b6c03, which looks like a use-after-free. Yup. But I'm somewhat puzzled by the trace. We crash when calling fsnotify_destroy_event() from fanotify_handle_event(). The fsnotify code has been called from do_sys_open() so the event was a 'FS_OPEN' which fails the fsn_event->mask & FAN_ALL_PERM_EVENTS test. Slapping my forehead, that's a really stupid bug. The event fsnotify_add_notify_event() returns may be freed by the time we return because we already dropped the notification mutex. And then fsn_event->mask & FAN_ALL_PERM_EVENTS test will pass because FAN_ALL_PERM_EVENTS matches with the poison pattern 0x6b6b6b6b. So yet another hacked up version of fanotify fix is attached. And I have to seriously think about use counts for fanotify version of that struct. > I also notice you mention SLAB above, but I've been using SLUB. I don't > know if the choice of allocator makes a difference in reproducability. Jiri Kosina has SLAB so SLAB/SLUB apparently doesn't matter. > It's also worth noting that I have lockdep enabled, which may be perturbing things > to some degree. And I've compiled my kernel with lockdep as well since Jiri has it in his config. But no luck. Honza -- Jan Kara <jack@suse.cz> SUSE Labs, CR [-- Attachment #2: fanotify.diff --] [-- Type: text/x-patch, Size: 1726 bytes --] diff --git a/fs/notify/fanotify/fanotify.c b/fs/notify/fanotify/fanotify.c index 58772623f02a..f80895fb5ca1 100644 --- a/fs/notify/fanotify/fanotify.c +++ b/fs/notify/fanotify/fanotify.c @@ -18,7 +18,7 @@ static bool should_merge(struct fsnotify_event *old_fsn, #ifdef CONFIG_FANOTIFY_ACCESS_PERMISSIONS /* dont merge two permission events */ - if ((old_fsn->mask & FAN_ALL_PERM_EVENTS) && + if ((old_fsn->mask & FAN_ALL_PERM_EVENTS) || (new_fsn->mask & FAN_ALL_PERM_EVENTS)) return false; #endif @@ -201,8 +201,10 @@ static int fanotify_handle_event(struct fsnotify_group *group, } #ifdef CONFIG_FANOTIFY_ACCESS_PERMISSIONS - if (fsn_event->mask & FAN_ALL_PERM_EVENTS) + if (mask & FAN_ALL_PERM_EVENTS) { ret = fanotify_get_response_from_access(group, event); + fsnotify_destroy_event(group, fsn_event); + } #endif return ret; } @@ -221,7 +223,8 @@ static void fanotify_free_event(struct fsnotify_event *fsn_event) struct fanotify_event_info *event; event = FANOTIFY_E(fsn_event); - path_put(&event->path); + if (event->path.mnt) + path_put(&event->path); put_pid(event->tgid); kmem_cache_free(fanotify_event_cachep, event); } diff --git a/fs/notify/fanotify/fanotify_user.c b/fs/notify/fanotify/fanotify_user.c index 57d7c083cb4b..d493c72c71fd 100644 --- a/fs/notify/fanotify/fanotify_user.c +++ b/fs/notify/fanotify/fanotify_user.c @@ -319,7 +319,8 @@ static ssize_t fanotify_read(struct file *file, char __user *buf, if (IS_ERR(kevent)) break; ret = copy_event_to_user(group, kevent, buf); - fsnotify_destroy_event(group, kevent); + if (!(kevent->mask & FAN_ALL_PERM_EVENTS)) + fsnotify_destroy_event(group, kevent); if (ret < 0) break; buf += ret; ^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: fanotify use after free. 2014-01-28 8:02 ` Jan Kara @ 2014-01-28 11:07 ` Jiri Kosina 2014-01-28 14:53 ` Jan Kara 0 siblings, 1 reply; 19+ messages in thread From: Jiri Kosina @ 2014-01-28 11:07 UTC (permalink / raw) To: Jan Kara; +Cc: Dave Jones, Linus Torvalds, Linux Kernel On Tue, 28 Jan 2014, Jan Kara wrote: > > 2b:* 4d 8b 64 c6 08 mov 0x8(%r14,%rax,8),%r12 <-- trapping instruction > > > > R14 is 0x6b6b6b6b6b6b6c03, which looks like a use-after-free. > Yup. But I'm somewhat puzzled by the trace. We crash when calling > fsnotify_destroy_event() from fanotify_handle_event(). The fsnotify code > has been called from do_sys_open() so the event was a 'FS_OPEN' which fails > the fsn_event->mask & FAN_ALL_PERM_EVENTS test. > > Slapping my forehead, that's a really stupid bug. The event > fsnotify_add_notify_event() returns may be freed by the time we return > because we already dropped the notification mutex. And then fsn_event->mask > & FAN_ALL_PERM_EVENTS test will pass because FAN_ALL_PERM_EVENTS matches > with the poison pattern 0x6b6b6b6b. So yet another hacked up version of > fanotify fix is attached. And I have to seriously think about use counts > for fanotify version of that struct. With the fixed version of the patch, all the fanotify-related oopses are gone on my system. -- Jiri Kosina SUSE Labs ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free. 2014-01-28 11:07 ` Jiri Kosina @ 2014-01-28 14:53 ` Jan Kara 2014-01-28 15:24 ` Dave Jones 0 siblings, 1 reply; 19+ messages in thread From: Jan Kara @ 2014-01-28 14:53 UTC (permalink / raw) To: Jiri Kosina; +Cc: Jan Kara, Dave Jones, Linus Torvalds, Linux Kernel On Tue 28-01-14 12:07:51, Jiri Kosina wrote: > On Tue, 28 Jan 2014, Jan Kara wrote: > > > > 2b:* 4d 8b 64 c6 08 mov 0x8(%r14,%rax,8),%r12 <-- trapping instruction > > > > > > R14 is 0x6b6b6b6b6b6b6c03, which looks like a use-after-free. > > Yup. But I'm somewhat puzzled by the trace. We crash when calling > > fsnotify_destroy_event() from fanotify_handle_event(). The fsnotify code > > has been called from do_sys_open() so the event was a 'FS_OPEN' which fails > > the fsn_event->mask & FAN_ALL_PERM_EVENTS test. > > > > Slapping my forehead, that's a really stupid bug. The event > > fsnotify_add_notify_event() returns may be freed by the time we return > > because we already dropped the notification mutex. And then fsn_event->mask > > & FAN_ALL_PERM_EVENTS test will pass because FAN_ALL_PERM_EVENTS matches > > with the poison pattern 0x6b6b6b6b. So yet another hacked up version of > > fanotify fix is attached. And I have to seriously think about use counts > > for fanotify version of that struct. > > With the fixed version of the patch, all the fanotify-related oopses are > gone on my system. Thanks for testing. So now I have to come up with something mergeable :) Honza -- Jan Kara <jack@suse.cz> SUSE Labs, CR ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free. 2014-01-28 14:53 ` Jan Kara @ 2014-01-28 15:24 ` Dave Jones 0 siblings, 0 replies; 19+ messages in thread From: Dave Jones @ 2014-01-28 15:24 UTC (permalink / raw) To: Jan Kara; +Cc: Jiri Kosina, Linus Torvalds, Linux Kernel On Tue, Jan 28, 2014 at 03:53:27PM +0100, Jan Kara wrote: > On Tue 28-01-14 12:07:51, Jiri Kosina wrote: > > On Tue, 28 Jan 2014, Jan Kara wrote: > > > > > > 2b:* 4d 8b 64 c6 08 mov 0x8(%r14,%rax,8),%r12 <-- trapping instruction > > > > > > > > R14 is 0x6b6b6b6b6b6b6c03, which looks like a use-after-free. > > > Yup. But I'm somewhat puzzled by the trace. We crash when calling > > > fsnotify_destroy_event() from fanotify_handle_event(). The fsnotify code > > > has been called from do_sys_open() so the event was a 'FS_OPEN' which fails > > > the fsn_event->mask & FAN_ALL_PERM_EVENTS test. > > > > > > Slapping my forehead, that's a really stupid bug. The event > > > fsnotify_add_notify_event() returns may be freed by the time we return > > > because we already dropped the notification mutex. And then fsn_event->mask > > > & FAN_ALL_PERM_EVENTS test will pass because FAN_ALL_PERM_EVENTS matches > > > with the poison pattern 0x6b6b6b6b. So yet another hacked up version of > > > fanotify fix is attached. And I have to seriously think about use counts > > > for fanotify version of that struct. > > > > With the fixed version of the patch, all the fanotify-related oopses are > > gone on my system. > Thanks for testing. So now I have to come up with something mergeable :) Yep, looks good to me too. Thanks. Dave ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free. 2014-01-27 23:40 ` Jan Kara 2014-01-28 6:10 ` Dave Jones @ 2014-01-28 10:53 ` Jiri Kosina 1 sibling, 0 replies; 19+ messages in thread From: Jiri Kosina @ 2014-01-28 10:53 UTC (permalink / raw) To: Jan Kara; +Cc: Linus Torvalds, Dave Jones, Linux Kernel On Tue, 28 Jan 2014, Jan Kara wrote: > Hum, still no luck with reproduction (either on physical machine or with > KVM). Anyway, I've looked at the code again and the previous patch had a > stupid bug (passing different pointer to fsnotify_destroy_event() than we > should have), plus also the merging function in fanotify was too > aggressive. Can you try the attached patch? It boots for me but that means > nothing since I cannot reproduce the issue... Thanks! I am attaching dmesg with the patch applied; I've removed irrelevant parts. There is a GPF, followed by scheduling in atomic context, followed by slab corruption, followed by another scheduling while atomic and leak of preempt_count. [ 0.000000] Initializing cgroup subsys cpuset [ 5.081301] systemd-udevd[332]: starting version 195 [ 5.083694] random: nonblocking pool is initialized [ 5.299400] systemd-journald[307]: Received SIGUSR1 [ 5.625120] general protection fault: 0000 [#1] SMP [ 5.626464] Modules linked in: acpi_cpufreq autofs4 uhci_hcd ehci_hcd i915 drm_kms_helper drm usbcore i2c_algo_bit usb_common button video edd fan processor ata_generic thermal thermal_sys [ 5.628008] CPU: 0 PID: 302 Comm: systemd-readahe Not tainted 3.13.0-03478-gae75a37 #1 [ 5.628008] Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008 [ 5.628008] task: ffff8800364b04d0 ti: ffff8800734b8000 task.ti: ffff8800734b8000 [ 5.628008] RIP: 0010:[<ffffffff810a51e7>] [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160 [ 5.628008] RSP: 0018:ffff8800734b9c68 EFLAGS: 00010282 [ 5.628008] RAX: ffff8800364b04d0 RBX: 6b6b6b6b6b6b6beb RCX: 0000000000000000 [ 5.628008] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 6b6b6b6b6b6b6beb [ 5.628008] RBP: ffff8800734b9c88 R08: 0000000000000002 R09: 0000000000000000 [ 5.628008] R10: 0000000000000000 R11: 0000000000000000 R12: 6b6b6b6b6b6b6beb [ 5.628008] R13: ffff880035d0db28 R14: 0000000000000020 R15: ffff880037fffd50 [ 5.628008] FS: 00007fd2f4728700(0000) GS:ffff88007c200000(0000) knlGS:0000000000000000 [ 5.628008] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5.628008] CR2: 00007fa7dc98c000 CR3: 0000000037f3b000 CR4: 00000000000007f0 [ 5.628008] Stack: [ 5.628008] 6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb ffff880035d0db28 [ 5.628008] ffff8800734b9ca8 ffffffff8159c5ac ffffffff812fe111 6b6b6b6b6b6b6beb [ 5.628008] ffff8800734b9cc8 ffffffff812fe111 ffff8800734b9cf8 6b6b6b6b6b6b6b6b [ 5.628008] Call Trace: [ 5.628008] [<ffffffff8159c5ac>] _raw_spin_lock+0x3c/0x50 [ 5.628008] [<ffffffff812fe111>] ? lockref_put_or_lock+0x11/0x40 [ 5.628008] [<ffffffff812fe111>] lockref_put_or_lock+0x11/0x40 [ 5.628008] [<ffffffff811b1442>] dput+0x22/0x130 [ 5.628008] [<ffffffff811a3d45>] path_put+0x15/0x30 [ 5.628008] [<ffffffff811e0bcc>] fanotify_free_event+0x1c/0x40 [ 5.628008] [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30 [ 5.628008] [<ffffffff811e1052>] fanotify_handle_event+0x342/0x390 [ 5.628008] [<ffffffff811a3d4d>] ? path_put+0x1d/0x30 [ 5.628008] [<ffffffff811dd18b>] send_to_group+0xfb/0x180 [ 5.628008] [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0 [ 5.628008] [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0 [ 5.628008] [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0 [ 5.628008] [<ffffffff811987ad>] do_sys_open+0x1ad/0x220 [ 5.628008] [<ffffffff812fdd7e>] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 5.628008] [<ffffffff81198859>] SyS_open+0x19/0x20 [ 5.628008] [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b [ 5.628008] Code: 0d 7e 81 48 89 df e8 29 ff ff ff eb 94 0f 1f 80 00 00 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 48 89 fb 4c 89 6d f8 <81> 7f 04 ad 4e ad de 74 0c 48 c7 c6 b5 0d 7e 81 e8 f4 fe ff ff [ 5.628008] RIP [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160 [ 5.628008] RSP <ffff8800734b9c68> [ 5.683491] ---[ end trace 5b4e9ae52ab9b0f6 ]--- [ 5.685076] BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:20 [ 5.686578] in_atomic(): 1, irqs_disabled(): 0, pid: 302, name: systemd-readahe [ 5.688058] INFO: lockdep is turned off. [ 5.689503] CPU: 0 PID: 302 Comm: systemd-readahe Tainted: G D 3.13.0-03478-gae75a37 #1 [ 5.690966] Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008 [ 5.692464] ffff8800364b04d0 ffff8800734b9a68 ffffffff8159703b ffff8800734b9a88 [ 5.694027] ffffffff8107f621 ffffffff81a3d000 ffff88003641c750 ffff8800734b9aa8 [ 5.695555] ffffffff8159b50f ffff8800364b04d0 ffff8800364b04d0 ffff8800734b9ad8 [ 5.697111] Call Trace: [ 5.698595] [<ffffffff8159703b>] dump_stack+0x72/0x87 [ 5.700108] [<ffffffff8107f621>] __might_sleep+0xe1/0x100 [ 5.701623] [<ffffffff8159b50f>] down_read+0x1f/0x60 [ 5.703133] [<ffffffff810627ff>] exit_signals+0x1f/0x140 [ 5.704661] [<ffffffff81079491>] ? blocking_notifier_call_chain+0x11/0x20 [ 5.706105] [<ffffffff81052844>] do_exit+0xb4/0x4b0 [ 5.707470] [<ffffffff8159e23c>] oops_end+0xdc/0xe0 [ 5.708845] [<ffffffff81005f86>] die+0x56/0x90 [ 5.710229] [<ffffffff8159dea2>] do_general_protection+0x162/0x170 [ 5.711545] [<ffffffff8159d40c>] ? restore_args+0x30/0x30 [ 5.712883] [<ffffffff8159d592>] general_protection+0x22/0x30 [ 5.714213] [<ffffffff810a51e7>] ? do_raw_spin_lock+0x17/0x160 [ 5.715491] [<ffffffff8159c5ac>] _raw_spin_lock+0x3c/0x50 [ 5.716781] [<ffffffff812fe111>] ? lockref_put_or_lock+0x11/0x40 [ 5.718113] [<ffffffff812fe111>] lockref_put_or_lock+0x11/0x40 [ 5.719390] [<ffffffff811b1442>] dput+0x22/0x130 [ 5.720681] [<ffffffff811a3d45>] path_put+0x15/0x30 [ 5.721984] [<ffffffff811e0bcc>] fanotify_free_event+0x1c/0x40 [ 5.723243] [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30 [ 5.724507] [<ffffffff811e1052>] fanotify_handle_event+0x342/0x390 [ 5.725779] [<ffffffff811a3d4d>] ? path_put+0x1d/0x30 [ 5.727014] [<ffffffff811dd18b>] send_to_group+0xfb/0x180 [ 5.728258] [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0 [ 5.729515] [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0 [ 5.730734] [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0 [ 5.731945] [<ffffffff811987ad>] do_sys_open+0x1ad/0x220 [ 5.733169] [<ffffffff812fdd7e>] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 5.734456] [<ffffffff81198859>] SyS_open+0x19/0x20 [ 5.735681] [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b [ 5.736933] note: systemd-readahe[302] exited with preempt_count 1 [ 5.738378] BUG: scheduling while atomic: systemd-readahe/302/0x00000002 [ 5.739652] INFO: lockdep is turned off. [ 5.740925] Modules linked in: acpi_cpufreq autofs4 uhci_hcd ehci_hcd i915 drm_kms_helper drm usbcore i2c_algo_bit usb_common button video edd fan processor ata_generic thermal thermal_sys [ 5.743781] CPU: 0 PID: 302 Comm: systemd-readahe Tainted: G D 3.13.0-03478-gae75a37 #1 [ 5.745231] Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008 [ 5.746708] ffff88007c213a00 ffff8800734b96e8 ffffffff8159703b ffff8800734b9708 [ 5.748190] ffffffff810810f1 ffff88007c213a00 0000000000000000 ffff8800734b9838 [ 5.749690] ffffffff815978ac ffff8800734b9748 ffff8800734b9758 ffff8800734b8010 [ 5.751161] Call Trace: [ 5.752635] [<ffffffff8159703b>] dump_stack+0x72/0x87 [ 5.754126] [<ffffffff810810f1>] __schedule_bug+0x61/0x80 [ 5.755551] [<ffffffff815978ac>] __schedule+0xbc/0x7c0 [ 5.756925] [<ffffffff8105defc>] ? mod_timer+0x14c/0x1f0 [ 5.758302] [<ffffffff815980f4>] schedule+0x24/0x70 [ 5.759632] [<ffffffff81597215>] schedule_timeout+0x1c5/0x210 [ 5.760982] [<ffffffff8159915f>] ? wait_for_completion+0xcf/0x120 [ 5.762327] [<ffffffff810a0d8d>] ? trace_hardirqs_on+0xd/0x10 [ 5.763630] [<ffffffff81599167>] wait_for_completion+0xd7/0x120 [ 5.764935] [<ffffffff81083330>] ? try_to_wake_up+0x250/0x250 [ 5.766261] [<ffffffff810b93bf>] ? srcu_reschedule+0x4f/0xf0 [ 5.767521] [<ffffffff810b965c>] __synchronize_srcu+0xec/0x130 [ 5.768775] [<ffffffff810b96e0>] ? srcu_barrier+0x10/0x10 [ 5.770059] [<ffffffff810b96c8>] synchronize_srcu+0x18/0x20 [ 5.771302] [<ffffffff811ddbdd>] fsnotify_destroy_group+0x1d/0x40 [ 5.772550] [<ffffffff811dfdf1>] inotify_release+0x21/0x50 [ 5.773814] [<ffffffff8119b2dd>] __fput+0xbd/0x2b0 [ 5.775055] [<ffffffff8119b569>] ____fput+0x9/0x10 [ 5.776311] [<ffffffff81070f41>] task_work_run+0xb1/0xe0 [ 5.777577] [<ffffffff81052979>] do_exit+0x1e9/0x4b0 [ 5.778803] [<ffffffff8159e23c>] oops_end+0xdc/0xe0 [ 5.780027] [<ffffffff81005f86>] die+0x56/0x90 [ 5.781264] [<ffffffff8159dea2>] do_general_protection+0x162/0x170 [ 5.782472] [<ffffffff8159d40c>] ? restore_args+0x30/0x30 [ 5.783687] [<ffffffff8159d592>] general_protection+0x22/0x30 [ 5.784917] [<ffffffff810a51e7>] ? do_raw_spin_lock+0x17/0x160 [ 5.786160] [<ffffffff8159c5ac>] _raw_spin_lock+0x3c/0x50 [ 5.787367] [<ffffffff812fe111>] ? lockref_put_or_lock+0x11/0x40 [ 5.788606] [<ffffffff812fe111>] lockref_put_or_lock+0x11/0x40 [ 5.789848] [<ffffffff811b1442>] dput+0x22/0x130 [ 5.791073] [<ffffffff811a3d45>] path_put+0x15/0x30 [ 5.792324] [<ffffffff811e0bcc>] fanotify_free_event+0x1c/0x40 [ 5.793589] [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30 [ 5.794809] [<ffffffff811e1052>] fanotify_handle_event+0x342/0x390 [ 5.796040] [<ffffffff811a3d4d>] ? path_put+0x1d/0x30 [ 5.797283] [<ffffffff811dd18b>] send_to_group+0xfb/0x180 [ 5.798505] [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0 [ 5.799732] [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0 [ 5.800970] [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0 [ 5.802219] [<ffffffff811987ad>] do_sys_open+0x1ad/0x220 [ 5.803461] [<ffffffff812fdd7e>] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 5.804729] [<ffffffff81198859>] SyS_open+0x19/0x20 [ 5.805970] [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b [ ... snip ... ] [ 5.968718] Slab corruption (Tainted: G D W ): fanotify_event_info start=ffff880035d2a798, len=64 [ 5.968923] hub 7-0:1.0: USB hub found [ 5.971406] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b. [ 5.972756] Last user: [<ffffffff811e0be4>](fanotify_free_event+0x34/0x40) [ 5.974098] 030: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 6b 6b 6b a5 kkkkkkkk....kkk. [ 5.974837] hub 7-0:1.0: 6 ports detected [ 5.976799] Prev obj: start=ffff880035d2a740, len=64 [ 5.978189] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b. [ 5.979526] Last user: [<ffffffff811e0be4>](fanotify_free_event+0x34/0x40) [ 5.980910] 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 5.982307] 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 5.983691] Next obj: start=ffff880035d2a7f0, len=64 [ 5.985070] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b. [ 5.986455] Last user: [<ffffffff811e0be4>](fanotify_free_event+0x34/0x40) [ 5.987835] 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 5.989248] 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ ... snip ... ] [ 7.044083] usb 2-1: new full-speed USB device number 4 using uhci_hcd [ 7.131735] general protection fault: 0000 [#2] SMP [ 7.131842] Slab corruption (Tainted: G D W ): fanotify_event_info start=ffff880035da5320, len=64 [ 7.131844] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b. [ 7.131850] Last user: [<ffffffff811e0be4>](fanotify_free_event+0x34/0x40) [ 7.131853] 030: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 6b 6b 6b a5 kkkkkkkk....kkk. [ 7.131854] Prev obj: start=ffff880035da52c8, len=64 [ 7.131855] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b. [ 7.131857] Last user: [<ffffffff811e0be4>](fanotify_free_event+0x34/0x40) [ 7.131859] 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 7.131861] 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 7.131862] Next obj: start=ffff880035da5378, len=64 [ 7.131863] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b. [ 7.131864] Last user: [<ffffffff811e0be4>](fanotify_free_event+0x34/0x40) [ 7.131866] 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 7.131868] 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 7.135045] Modules linked in: cpufreq_conservative cpufreq_userspace snd_hda_codec_conexant cpufreq_powersave snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hwdep snd_pcm thinkpad_acpi kvm_intel snd_seq iTCO_wdt iTCO_vendor_support kvm iwldvm mac80211 snd_timer snd_seq_device btusb bluetooth iwlwifi sg cfg80211 e1000e snd ptp pcspkr lpc_ich i2c_i801 mfd_core rfkill pps_core ehci_pci wmi soundcore battery ac tpm_tis tpm acpi_cpufreq autofs4 uhci_hcd ehci_hcd i915 drm_kms_helper drm usbcore i2c_algo_bit usb_common button video edd fan processor ata_generic thermal thermal_sys [ 7.135045] CPU: 1 PID: 757 Comm: grep Tainted: G D W 3.13.0-03478-gae75a37 #1 [ 7.135045] Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008 [ 7.135045] task: ffff8800362fddd0 ti: ffff880036d42000 task.ti: ffff880036d42000 [ 7.135045] RIP: 0010:[<ffffffff810a51e7>] [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160 [ 7.135045] RSP: 0018:ffff880036d43c68 EFLAGS: 00010282 [ 7.135045] RAX: ffff8800362fddd0 RBX: 6b6b6b6b6b6b6beb RCX: 0000000000000000 [ 7.135045] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 6b6b6b6b6b6b6beb [ 7.135045] RBP: ffff880036d43c88 R08: 0000000000000002 R09: 0000000000000000 [ 7.135045] R10: 0000000000000000 R11: 0000000000000000 R12: 6b6b6b6b6b6b6beb [ 7.135045] R13: ffff880035d0db28 R14: 0000000000000020 R15: ffff880037900310 [ 7.135045] FS: 0000000000000000(0000) GS:ffff88007c280000(0000) knlGS:0000000000000000 [ 7.135045] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7.135045] CR2: 00007fff296c9f54 CR3: 0000000079067000 CR4: 00000000000007e0 [ 7.135045] Stack: [ 7.135045] 6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb ffff880035d0db28 [ 7.135045] ffff880036d43ca8 ffffffff8159c5ac ffffffff812fe111 6b6b6b6b6b6b6beb [ 7.135045] ffff880036d43cc8 ffffffff812fe111 ffff880036d43cf8 6b6b6b6b6b6b6b6b [ 7.135045] Call Trace: [ 7.135045] [<ffffffff8159c5ac>] _raw_spin_lock+0x3c/0x50 [ 7.135045] [<ffffffff812fe111>] ? lockref_put_or_lock+0x11/0x40 [ 7.135045] [<ffffffff812fe111>] lockref_put_or_lock+0x11/0x40 [ 7.135045] [<ffffffff811b1442>] dput+0x22/0x130 [ 7.135045] [<ffffffff811a3d45>] path_put+0x15/0x30 [ 7.135045] [<ffffffff811e0bcc>] fanotify_free_event+0x1c/0x40 [ 7.135045] [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30 [ 7.135045] [<ffffffff811e1052>] fanotify_handle_event+0x342/0x390 [ 7.135045] [<ffffffff815a0c84>] ? __do_page_fault+0x2c4/0x480 [ 7.135045] [<ffffffff811dd18b>] send_to_group+0xfb/0x180 [ 7.135045] [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0 [ 7.135045] [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0 [ 7.135045] [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0 [ 7.135045] [<ffffffff811987ad>] do_sys_open+0x1ad/0x220 [ 7.135045] [<ffffffff812fdd7e>] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 7.135045] [<ffffffff81198859>] SyS_open+0x19/0x20 [ 7.135045] [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b [ 7.135045] Code: 0d 7e 81 48 89 df e8 29 ff ff ff eb 94 0f 1f 80 00 00 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 48 89 fb 4c 89 6d f8 <81> 7f 04 ad 4e ad de 74 0c 48 c7 c6 b5 0d 7e 81 e8 f4 fe ff ff [ 7.135045] RIP [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160 [ 7.135045] RSP <ffff880036d43c68> [ 7.212496] ---[ end trace 5b4e9ae52ab9b0f7 ]--- [ 7.214048] BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:20 [ 7.214049] in_atomic(): 1, irqs_disabled(): 0, pid: 757, name: grep [ 7.214050] INFO: lockdep is turned off. [ 7.214051] CPU: 1 PID: 757 Comm: grep Tainted: G D W 3.13.0-03478-gae75a37 #1 [ 7.214052] Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008 [ 7.214055] ffff8800362fddd0 ffff880036d43a68 ffffffff8159703b ffff880036d43a88 [ 7.214057] ffffffff8107f621 ffffffff81a3d000 ffff880037946cd0 ffff880036d43aa8 [ 7.214059] ffffffff8159b50f ffff8800362fddd0 ffff8800362fddd0 ffff880036d43ad8 [ 7.214060] Call Trace: [ 7.214071] [<ffffffff8159703b>] dump_stack+0x72/0x87 [ 7.214074] [<ffffffff8107f621>] __might_sleep+0xe1/0x100 [ 7.214076] [<ffffffff8159b50f>] down_read+0x1f/0x60 [ 7.214079] [<ffffffff810627ff>] exit_signals+0x1f/0x140 [ 7.214083] [<ffffffff81079491>] ? blocking_notifier_call_chain+0x11/0x20 [ 7.214086] [<ffffffff81052844>] do_exit+0xb4/0x4b0 [ 7.214089] [<ffffffff8159e23c>] oops_end+0xdc/0xe0 [ 7.214092] [<ffffffff81005f86>] die+0x56/0x90 [ 7.214095] [<ffffffff8159dea2>] do_general_protection+0x162/0x170 [ 7.214097] [<ffffffff8159d40c>] ? restore_args+0x30/0x30 [ 7.214099] [<ffffffff8159d592>] general_protection+0x22/0x30 [ 7.214102] [<ffffffff810a51e7>] ? do_raw_spin_lock+0x17/0x160 [ 7.214104] [<ffffffff8159c5ac>] _raw_spin_lock+0x3c/0x50 [ 7.214107] [<ffffffff812fe111>] ? lockref_put_or_lock+0x11/0x40 [ 7.214109] [<ffffffff812fe111>] lockref_put_or_lock+0x11/0x40 [ 7.214113] [<ffffffff811b1442>] dput+0x22/0x130 [ 7.214115] [<ffffffff811a3d45>] path_put+0x15/0x30 [ 7.214117] [<ffffffff811e0bcc>] fanotify_free_event+0x1c/0x40 [ 7.214119] [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30 [ 7.214121] [<ffffffff811e1052>] fanotify_handle_event+0x342/0x390 [ 7.214124] [<ffffffff815a0c84>] ? __do_page_fault+0x2c4/0x480 [ 7.214127] [<ffffffff811dd18b>] send_to_group+0xfb/0x180 [ 7.214129] [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0 [ 7.214131] [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0 [ 7.214134] [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0 [ 7.214136] [<ffffffff811987ad>] do_sys_open+0x1ad/0x220 [ 7.214139] [<ffffffff812fdd7e>] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 7.214141] [<ffffffff81198859>] SyS_open+0x19/0x20 [ 7.214143] [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b [ 7.214145] note: grep[757] exited with preempt_count 1 -- Jiri Kosina SUSE Labs ^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2014-01-28 15:24 UTC | newest] Thread overview: 19+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-01-22 6:27 fanotify use after free Dave Jones 2014-01-22 16:43 ` Dave Jones 2014-01-22 18:20 ` Linus Torvalds 2014-01-22 23:36 ` Jan Kara 2014-01-23 0:08 ` Linus Torvalds 2014-01-23 0:32 ` Dave Jones 2014-01-23 15:05 ` Jan Kara 2014-01-23 10:23 ` Jiri Kosina 2014-01-23 15:05 ` Jan Kara 2014-01-23 15:07 ` Jiri Kosina 2014-01-23 23:55 ` Jan Kara 2014-01-24 7:26 ` Jiri Kosina 2014-01-27 23:40 ` Jan Kara 2014-01-28 6:10 ` Dave Jones 2014-01-28 8:02 ` Jan Kara 2014-01-28 11:07 ` Jiri Kosina 2014-01-28 14:53 ` Jan Kara 2014-01-28 15:24 ` Dave Jones 2014-01-28 10:53 ` Jiri Kosina
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).