From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by oss.sgi.com (8.14.3/8.14.3/SuSE Linux 0.8) with ESMTP id o0L1LHGI042833 for ; Wed, 20 Jan 2010 19:21:18 -0600 Message-ID: <4B57ADB4.70002@kernel.org> Date: Thu, 21 Jan 2010 10:28:20 +0900 From: Tejun Heo MIME-Version: 1.0 Subject: Circular locking dependency warning involing xfs on 2.6.27 List-Id: XFS Filesystem from SGI List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: xfs-bounces@oss.sgi.com Errors-To: xfs-bounces@oss.sgi.com To: lkml , Alex Elder , xfs@oss.sgi.com Cc: Carsten Aulbert Hello, This is on a pretty old kernel but Carsten reports that it happens occassionally in a pretty large setup involving 1500+ machines (please feel free to fill in). This happens pretty sporadically so reporting it just in case this is a yet unknown problem. As I understand it, testing with newer kernel isn't out of question but it being a production environment, not a very easy thing. Thanks. [12207.016809] ======================================================= [12207.021881] [ INFO: possible circular locking dependency detected ] [12207.021881] 2.6.27.43-atlas-generic #1 [12207.021881] ------------------------------------------------------- [12207.231253] inspiralSensMon/8833 is trying to acquire lock: [12207.231253] (iprune_mutex){--..}, at: [] shrink_icache_memory+0x4b/0x23b [12207.231253] [12207.231253] but task is already holding lock: [12207.231253] (&(&ip->i_iolock)->mr_lock){----}, at: [] xfs_ilock+0x44/0x79 [12207.231253] [12207.231253] which lock already depends on the new lock. [12207.231253] [12207.231253] [12207.231253] the existing dependency chain (in reverse order) is: [12207.231253] [12207.231253] -> #1 (&(&ip->i_iolock)->mr_lock){----}: [12207.231253] [] __lock_acquire+0x12a7/0x15bc [12207.231253] [] lock_acquire+0x8d/0xba [12207.231253] [] down_write_nested+0x2a/0x36 [12207.231253] [] xfs_ilock+0x27/0x79 [12207.231253] [] xfs_ireclaim+0x41/0x87 [12207.231253] [] xfs_finish_reclaim+0x142/0x150 [12207.231253] [] xfs_reclaim+0x77/0xfc [12207.231253] [] xfs_fs_clear_inode+0xe2/0x106 [12207.231253] [] clear_inode+0x71/0xca [12207.231253] [] dispose_list+0x5b/0x10e [12207.231253] [] shrink_icache_memory+0x205/0x23b [12207.231253] [] shrink_slab+0xe3/0x158 [12207.231253] [] kswapd+0x40e/0x572 [12207.231253] [] kthread+0x49/0x76 [12207.231253] [] child_rip+0xa/0x11 [12207.231253] [] 0xffffffffffffffff [12207.231253] [12207.231253] -> #0 (iprune_mutex){--..}: [12207.231253] [] __lock_acquire+0xfbc/0x15bc [12207.231253] [] lock_acquire+0x8d/0xba [12207.231253] [] mutex_lock_nested+0xe3/0x28a [12207.231253] [] shrink_icache_memory+0x4b/0x23b [12207.231253] [] shrink_slab+0xe3/0x158 [12207.231253] [] try_to_free_pages+0x227/0x313 [12207.231253] [] __alloc_pages_internal+0x261/0x40c [12207.231253] [] __do_page_cache_readahead+0xf0/0x1f7 [12207.231253] [] ondemand_readahead+0x1cf/0x1e4 [12207.231253] [] page_cache_async_readahead+0x78/0x84 [12207.231253] [] generic_file_aio_read+0x27d/0x5d6 [12207.231253] [] xfs_read+0x18e/0x205 [12207.231253] [] xfs_file_aio_read+0x51/0x53 [12207.231253] [] do_sync_read+0xe7/0x12d [12207.231253] [] vfs_read+0xab/0x134 [12207.231253] [] sys_read+0x47/0x6f [12207.231253] [] system_call_fastpath+0x16/0x1b [12207.231253] [] 0xffffffffffffffff [12207.231253] [12207.231253] other info that might help us debug this: [12207.231253] [12207.231253] 2 locks held by inspiralSensMon/8833: [12207.231253] #0: (&(&ip->i_iolock)->mr_lock){----}, at: [] xfs_ilock+0x44/0x79 [12207.231253] #1: (shrinker_rwsem){----}, at: [] shrink_slab+0x38/0x158 [12207.231253] [12207.231253] stack backtrace: [12207.231253] Pid: 8833, comm: inspiralSensMon Not tainted 2.6.27.43-atlas-generic #1 [12207.231253] [12207.231253] Call Trace: [12207.231253] [] print_circular_bug_tail+0xb8/0xc3 [12207.231253] [] __lock_acquire+0xfbc/0x15bc [12207.231253] [] lock_acquire+0x8d/0xba [12207.231253] [] ? shrink_icache_memory+0x4b/0x23b [12207.231253] [] mutex_lock_nested+0xe3/0x28a [12207.231253] [] ? shrink_icache_memory+0x4b/0x23b [12207.231253] [] ? shrink_icache_memory+0x4b/0x23b [12207.231253] [] shrink_icache_memory+0x4b/0x23b [12207.231253] [] shrink_slab+0xe3/0x158 [12207.231253] [] try_to_free_pages+0x227/0x313 [12207.231253] [] ? isolate_pages_global+0x0/0x34 [12207.231253] [] __alloc_pages_internal+0x261/0x40c [12207.231253] [] __do_page_cache_readahead+0xf0/0x1f7 [12207.231253] [] ? __do_page_cache_readahead+0x7b/0x1f7 [12207.231253] [] ondemand_readahead+0x1cf/0x1e4 [12207.231253] [] page_cache_async_readahead+0x78/0x84 [12207.231253] [] ? find_get_page+0x0/0xc9 [12207.231253] [] generic_file_aio_read+0x27d/0x5d6 [12207.231253] [] ? xfs_ilock+0x44/0x79 [12207.231253] [] xfs_read+0x18e/0x205 [12207.231253] [] xfs_file_aio_read+0x51/0x53 [12207.231253] [] do_sync_read+0xe7/0x12d [12207.231253] [] ? __up_write+0x29/0x11e [12207.231253] [] ? autoremove_wake_function+0x0/0x38 [12207.231253] [] ? trace_hardirqs_on_caller+0xf7/0x11b [12207.231253] [] ? __up_write+0x10f/0x11e [12207.231253] [] ? security_file_permission+0x11/0x13 [12207.231253] [] vfs_read+0xab/0x134 [12207.231253] [] sys_read+0x47/0x6f [12207.231253] [] system_call_fastpath+0x16/0x1b -- tejun _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs