From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751759AbbD3Gen (ORCPT ); Thu, 30 Apr 2015 02:34:43 -0400 Received: from mail-pa0-f46.google.com ([209.85.220.46]:33702 "EHLO mail-pa0-f46.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751240AbbD3Gem (ORCPT ); Thu, 30 Apr 2015 02:34:42 -0400 Date: Thu, 30 Apr 2015 15:34:57 +0900 From: Sergey Senozhatsky To: Minchan Kim Cc: Sergey Senozhatsky , Andrew Morton , Nitin Gupta , linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: Re: [PATCHv3 9/9] zram: add dynamic device add/remove functionality Message-ID: <20150430063457.GA950@swordfish> References: <1430140911-7818-1-git-send-email-sergey.senozhatsky@gmail.com> <1430140911-7818-10-git-send-email-sergey.senozhatsky@gmail.com> <20150429001624.GA3917@swordfish> <20150429064858.GA5125@blaptop> <20150429070218.GA616@swordfish> <20150429072328.GA2987@swordfish> <20150430054702.GA21771@blaptop> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150430054702.GA21771@blaptop> User-Agent: Mutt/1.5.23 (2014-03-12) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hello Minchan, On (04/30/15 14:47), Minchan Kim wrote: [..] > > Isn't it related to bd_mutex? I think it is: [ 216.713922] Possible unsafe locking scenario: [ 216.713923] CPU0 CPU1 [ 216.713924] ---- ---- [ 216.713925] lock(&bdev->bd_mutex); [ 216.713927] lock(s_active#162); [ 216.713929] lock(&bdev->bd_mutex); [ 216.713930] lock(s_active#162); [ 216.713932] *** DEADLOCK *** > I think the problem of deadlock is that you are trying to remove sysfs file > in sysfs handler. > > #> echo 1 > /sys/xxx/zram_remove > > kernfs_fop_write - hold s_active > -> zram_remove_store > -> zram_remove > -> sysfs_remove_group - hold s_active *again* > > Right? > are those same s_active locks? we hold (s_active#163) and (&bdev->bd_mutex) and want to acquire (s_active#162) [ 216.713934] 5 locks held by bash/342: [ 216.713935] #0: (sb_writers#5){.+.+.+}, at: [] vfs_write+0xaf/0x145 [ 216.713938] #1: (&of->mutex){+.+.+.}, at: [] kernfs_fop_write+0x9c/0x14c [ 216.713942] #2: (s_active#163){.+.+.+}, at: [] kernfs_fop_write+0xa5/0x14c [ 216.713946] #3: (zram_index_mutex){+.+.+.}, at: [] zram_remove_store+0x45/0xba [zram] [ 216.713950] #4: (&bdev->bd_mutex){+.+.+.}, at: [] zram_remove+0x41/0xf0 [zram] full log: [ 216.713826] ====================================================== [ 216.713827] [ INFO: possible circular locking dependency detected ] [ 216.713829] 4.1.0-rc1-next-20150430-dbg-00010-ga86accf-dirty #121 Tainted: G O [ 216.713831] ------------------------------------------------------- [ 216.713832] bash/342 is trying to acquire lock: [ 216.713833] (s_active#162){++++.+}, at: [] kernfs_remove_by_name_ns+0x70/0x8c [ 216.713840] but task is already holding lock: [ 216.713842] (&bdev->bd_mutex){+.+.+.}, at: [] zram_remove+0x41/0xf0 [zram] [ 216.713846] which lock already depends on the new lock. [ 216.713848] the existing dependency chain (in reverse order) is: [ 216.713849] -> #1 (&bdev->bd_mutex){+.+.+.}: [ 216.713852] [] __lock_acquire+0x10c2/0x11cb [ 216.713856] [] lock_acquire+0x13d/0x250 [ 216.713858] [] mutex_lock_nested+0x5e/0x35f [ 216.713860] [] revalidate_disk+0x4b/0x7c [ 216.713863] [] disksize_store+0x1b1/0x1f4 [zram] [ 216.713866] [] dev_attr_store+0x19/0x23 [ 216.713870] [] sysfs_kf_write+0x48/0x54 [ 216.713872] [] kernfs_fop_write+0x101/0x14c [ 216.713874] [] __vfs_write+0x26/0xbe [ 216.713877] [] vfs_write+0xc0/0x145 [ 216.713879] [] SyS_write+0x51/0x8f [ 216.713881] [] system_call_fastpath+0x12/0x6f [ 216.713884] -> #0 (s_active#162){++++.+}: [ 216.713886] [] check_prevs_add+0x19e/0x747 [ 216.713889] [] __lock_acquire+0x10c2/0x11cb [ 216.713891] [] lock_acquire+0x13d/0x250 [ 216.713892] [] __kernfs_remove+0x1b6/0x2cd [ 216.713895] [] kernfs_remove_by_name_ns+0x70/0x8c [ 216.713897] [] remove_files+0x42/0x67 [ 216.713899] [] sysfs_remove_group+0x69/0x88 [ 216.713901] [] zram_remove+0x66/0xf0 [zram] [ 216.713904] [] zram_remove_store+0x95/0xba [zram] [ 216.713906] [] class_attr_store+0x1c/0x26 [ 216.713909] [] sysfs_kf_write+0x48/0x54 [ 216.713911] [] kernfs_fop_write+0x101/0x14c [ 216.713913] [] __vfs_write+0x26/0xbe [ 216.713915] [] vfs_write+0xc0/0x145 [ 216.713917] [] SyS_write+0x51/0x8f [ 216.713918] [] system_call_fastpath+0x12/0x6f [ 216.713920] other info that might help us debug this: [ 216.713922] Possible unsafe locking scenario: [ 216.713923] CPU0 CPU1 [ 216.713924] ---- ---- [ 216.713925] lock(&bdev->bd_mutex); [ 216.713927] lock(s_active#162); [ 216.713929] lock(&bdev->bd_mutex); [ 216.713930] lock(s_active#162); [ 216.713932] *** DEADLOCK *** [ 216.713934] 5 locks held by bash/342: [ 216.713935] #0: (sb_writers#5){.+.+.+}, at: [] vfs_write+0xaf/0x145 [ 216.713938] #1: (&of->mutex){+.+.+.}, at: [] kernfs_fop_write+0x9c/0x14c [ 216.713942] #2: (s_active#163){.+.+.+}, at: [] kernfs_fop_write+0xa5/0x14c [ 216.713946] #3: (zram_index_mutex){+.+.+.}, at: [] zram_remove_store+0x45/0xba [zram] [ 216.713950] #4: (&bdev->bd_mutex){+.+.+.}, at: [] zram_remove+0x41/0xf0 [zram] [ 216.713954] stack backtrace: [ 216.713957] CPU: 1 PID: 342 Comm: bash Tainted: G O 4.1.0-rc1-next-20150430-dbg-00010-ga86accf-dirty #121 [ 216.713958] Hardware name: SAMSUNG ELECTRONICS CO.,LTD Samsung DeskTop System/Samsung DeskTop System, BIOS 05CC 04/09/2010 [ 216.713960] ffffffff82400210 ffff8800ba367a28 ffffffff815265b1 ffffffff810785f2 [ 216.713962] ffffffff8242f970 ffff8800ba367a78 ffffffff8107aac7 ffffffff817bd85e [ 216.713965] ffff8800bdeca1a0 ffff8800bdeca9c0 ffff8800bdeca998 ffff8800bdeca9c0 [ 216.713967] Call Trace: [ 216.713971] [] dump_stack+0x4c/0x6e [ 216.713973] [] ? up+0x39/0x3e [ 216.713975] [] print_circular_bug+0x2b1/0x2c2 [ 216.713976] [] check_prevs_add+0x19e/0x747 [ 216.713979] [] __lock_acquire+0x10c2/0x11cb [ 216.713981] [] lock_acquire+0x13d/0x250 [ 216.713983] [] ? kernfs_remove_by_name_ns+0x70/0x8c [ 216.713985] [] __kernfs_remove+0x1b6/0x2cd [ 216.713987] [] ? kernfs_remove_by_name_ns+0x70/0x8c [ 216.713989] [] ? kernfs_find_ns+0xcd/0x10e [ 216.713990] [] ? mutex_lock_nested+0x32c/0x35f [ 216.713992] [] kernfs_remove_by_name_ns+0x70/0x8c [ 216.713994] [] remove_files+0x42/0x67 [ 216.713996] [] sysfs_remove_group+0x69/0x88 [ 216.713999] [] zram_remove+0x66/0xf0 [zram] [ 216.714001] [] zram_remove_store+0x95/0xba [zram] [ 216.714003] [] class_attr_store+0x1c/0x26 [ 216.714005] [] sysfs_kf_write+0x48/0x54 [ 216.714007] [] kernfs_fop_write+0x101/0x14c [ 216.714009] [] __vfs_write+0x26/0xbe [ 216.714011] [] ? __close_fd+0x25/0xdd [ 216.714013] [] ? __lock_is_held+0x3c/0x57 [ 216.714015] [] vfs_write+0xc0/0x145 [ 216.714017] [] SyS_write+0x51/0x8f [ 216.714019] [] system_call_fastpath+0x12/0x6f [ 216.714063] zram: Removed device: zram0 -ss