From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752006AbaC3QEo (ORCPT ); Sun, 30 Mar 2014 12:04:44 -0400 Received: from userp1040.oracle.com ([156.151.31.81]:24747 "EHLO userp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751653AbaC3QEm (ORCPT ); Sun, 30 Mar 2014 12:04:42 -0400 Message-ID: <53384080.9020502@oracle.com> Date: Sun, 30 Mar 2014 12:04:16 -0400 From: Sasha Levin User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.4.0 MIME-Version: 1.0 To: Alessandro Zummo CC: Tejun Heo , Greg KH , rtc-linux@googlegroups.com, LKML Subject: Re: kernfs/rtc: circular dependency between kernfs and ops_lock References: <5308E3AE.3040405@oracle.com> <20140222205217.GC16272@mtj.dyndns.org> <20140224120427.6394d5dd@linux.lan.towertech.it> <5331FA9E.20506@oracle.com> <20140325233936.655a798e@linux.lan.towertech.it> <53321D04.2040102@oracle.com> <5337653D.6020002@oracle.com> In-Reply-To: <5337653D.6020002@oracle.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Source-IP: ucsinet22.oracle.com [156.151.31.94] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 03/29/2014 08:28 PM, Sasha Levin wrote: > On 03/25/2014 08:19 PM, Sasha Levin wrote: >> On 03/25/2014 06:39 PM, Alessandro Zummo wrote: >>> On Tue, 25 Mar 2014 17:52:30 -0400 >>> Sasha Levin wrote: >>> >>>>> >>>>> Pretty interesting indeed. One option would be to remove >>>>> the sys files before acquiring the lock. But I wonder >>>>> if this could lead to other issues. >>>>> >>>> >>>> Ping? Seems this thing got lost. >>> >>> What if we move >>> >>> rtc_sysfs_del_device(rtc); >>> >>> before acquiring the lock? >> >> Trying it now, I'll leave it going for a week and see if it reproduces. > > Look good, thanks! Or not... Hit it again during overnight fuzzing: [ 1288.605629] ====================================================== [ 1288.606622] [ INFO: possible circular locking dependency detected ] [ 1288.607544] 3.14.0-rc8-next-20140328-sasha-00016-geb1c595-dirty #297 Tainted: G W [ 1288.608861] ------------------------------------------------------- [ 1288.609719] trinity-c79/32073 is trying to acquire lock: [ 1288.610463] (&rtc->ops_lock){+.+.+.}, at: [] rtc_read_time+0x30/0x 70 [ 1288.610490] [ 1288.610490] but task is already holding lock: [ 1288.610490] (s_active#19){++++.+}, at: [] kernfs_seq_start+0x44/0x a0 [ 1288.610490] [ 1288.610490] which lock already depends on the new lock. [ 1288.610490] [ 1288.610490] [ 1288.610490] the existing dependency chain (in reverse order) is: [ 1288.610490] -> #1 (s_active#19){++++.+}: [ 1288.610490] [] lock_acquire+0x19a/0x240 [ 1288.610490] [] __kernfs_remove+0x180/0x320 [ 1288.610490] [] kernfs_remove_by_name_ns+0x7c/0xb0 [ 1288.610490] [] remove_files.isra.1+0x51/0x80 [ 1288.610490] [] sysfs_remove_group+0x83/0xb0 [ 1288.610490] [] sysfs_remove_groups+0x3b/0x60 [ 1288.610490] [] device_remove_groups+0x12/0x20 [ 1288.610490] [] device_remove_attrs+0x68/0x80 [ 1288.610490] [] device_del+0x157/0x1c0 [ 1288.610490] [] device_unregister+0x58/0x70 [ 1288.610490] [] rtc_device_unregister+0x53/0x80 [ 1288.610490] [] cmos_do_probe+0x3b3/0x400 [ 1288.610490] [] cmos_platform_probe+0x44/0x4d [ 1288.610490] [] platform_drv_probe+0x32/0x90 [ 1288.610490] [] driver_probe_device+0x175/0x370 [ 1288.610490] [] __driver_attach+0x84/0xc0 [ 1288.610490] [] bus_for_each_dev+0x69/0xb0 [ 1288.610490] [] driver_attach+0x1e/0x20 [ 1288.610490] [] bus_add_driver+0x138/0x260 [ 1288.610490] [] driver_register+0x98/0xe0 [ 1288.610490] [] __platform_driver_register+0x4a/0x50 [ 1288.610490] [] platform_driver_probe+0x24/0xc0 [ 1288.610490] [] cmos_init+0x3c/0x74 [ 1288.610490] [] do_one_initcall+0xca/0x1d0 [ 1288.610490] [] kernel_init_freeable+0x1d5/0x26c [ 1288.610490] [] kernel_init+0xe/0x100 [ 1288.610490] [] ret_from_fork+0x7c/0xb0 [ 1288.610490] -> #0 (&rtc->ops_lock){+.+.+.}: [ 1288.610490] [] __lock_acquire+0x1804/0x2120 [ 1288.610490] [] lock_acquire+0x19a/0x240 [ 1288.610490] [] mutex_lock_interruptible_nested+0x6f/0x680 [ 1288.610490] [] rtc_read_time+0x30/0x70 [ 1288.610490] [] date_show+0x1a/0x50 [ 1288.610490] [] dev_attr_show+0x20/0x60 [ 1288.610490] [] sysfs_kf_seq_show+0x98/0x110 [ 1288.610490] [] kernfs_seq_show+0x26/0x30 [ 1288.610490] [] seq_read+0x19c/0x3a0 [ 1288.610490] [] kernfs_fop_read+0x47/0x160 [ 1288.610490] [] vfs_read+0xb4/0x190 [ 1288.610490] [] SyS_read+0x52/0xc0 [ 1288.610490] [] tracesys+0xe1/0xe6 [ 1288.610490] [ 1288.610490] other info that might help us debug this: [ 1288.610490] [ 1288.610490] Possible unsafe locking scenario: [ 1288.610490] [ 1288.610490] CPU0 CPU1 [ 1288.610490] ---- ---- [ 1288.610490] lock(s_active#19); [ 1288.610490] lock(&rtc->ops_lock); [ 1288.610490] lock(s_active#19); [ 1288.610490] lock(&rtc->ops_lock); [ 1288.610490] [ 1288.610490] *** DEADLOCK *** [ 1288.610490] [ 1288.610490] 4 locks held by trinity-c79/32073: [ 1288.610490] #0: (&f->f_pos_lock){+.+.+.}, at: [] __fdget_pos+0x44/0x50 [ 1288.610490] #1: (&p->lock){+.+.+.}, at: [] seq_read+0x3d/0x3a0 [ 1288.610490] #2: (&of->mutex){+.+.+.}, at: [] kernfs_seq_start+0x3c/0xa0 [ 1288.610490] #3: (s_active#19){++++.+}, at: [] kernfs_seq_start+0x44/0xa0 [ 1288.610490] [ 1288.610490] stack backtrace: [ 1288.610490] CPU: 1 PID: 32073 Comm: trinity-c79 Tainted: G W 3.14.0-rc8-next-20140328-sasha-00016-geb1c595-dirty #297 [ 1288.610490] ffffffff9f9efb40 ffff88062c785ad8 ffffffff9c4b8507 0000000000000000 [ 1288.610490] ffffffff9f9efb40 ffff88062c785b28 ffffffff9c4ab007 0000000000000004 [ 1288.610490] ffff88062c785bb8 ffff88062c785b28 ffff88062c630d98 ffff88062c630dd0 [ 1288.610490] Call Trace: [ 1288.610490] [] dump_stack+0x4f/0x7c [ 1288.610490] [] print_circular_bug+0x1fb/0x20c [ 1288.610490] [] __lock_acquire+0x1804/0x2120 [ 1288.610490] [] ? kvm_clock_read+0x24/0x40 [ 1288.610490] [] ? sched_clock+0x15/0x20 [ 1288.610490] [] lock_acquire+0x19a/0x240 [ 1288.610490] [] ? rtc_read_time+0x30/0x70 [ 1288.610490] [] mutex_lock_interruptible_nested+0x6f/0x680 [ 1288.610490] [] ? rtc_read_time+0x30/0x70 [ 1288.610490] [] ? __slab_alloc+0x5c8/0x637 [ 1288.610490] [] ? rtc_read_time+0x30/0x70 [ 1288.610490] [] rtc_read_time+0x30/0x70 [ 1288.610490] [] date_show+0x1a/0x50 [ 1288.610490] [] dev_attr_show+0x20/0x60 [ 1288.610490] [] ? sysfs_file_ops+0x35/0x80 [ 1288.610490] [] sysfs_kf_seq_show+0x98/0x110 [ 1288.610490] [] kernfs_seq_show+0x26/0x30 [ 1288.610490] [] seq_read+0x19c/0x3a0 [ 1288.610490] [] kernfs_fop_read+0x47/0x160 [ 1288.610490] [] vfs_read+0xb4/0x190 [ 1288.610490] [] SyS_read+0x52/0xc0 [ 1288.610490] [] tracesys+0xe1/0xe6 Thanks, Sasha