From mboxrd@z Thu Jan 1 00:00:00 1970 From: Wu Fengguang Subject: [BUG 2.6.39-rc3] NFS spinlock recursion Date: Fri, 15 Apr 2011 10:47:58 +0800 Message-ID: <20110415024758.GA20420@localhost> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: linux-fsdevel@vger.kernel.org, LKML To: Trond Myklebust Return-path: Content-Disposition: inline Sender: linux-kernel-owner@vger.kernel.org List-Id: linux-fsdevel.vger.kernel.org Hi Trond, I got these errors when testing writeback, did you see it before? (I've removed all local changes to NFS code..) Thanks, Fengguang --- [ 15.463942] XFS (sda5): Mounting Filesystem [ 15.468446] XFS: Mounting Filesystem [ 15.548984] XFS (sda5): Ending clean mount [ 15.553347] XFS: Ending clean mount [ 89.917428] BUG: spinlock recursion on CPU#3, flush-0:24/2548 [ 89.923647] lock: ffff8801223c9240, .magic: dead4ead, .owner: flush-0:24/2548, .owner_cpu: 3 [ 89.932677] Pid: 2548, comm: flush-0:24 Not tainted 2.6.39-rc3-dt7+ #175 [ 89.939649] Call Trace: [ 89.942356] [] spin_bug+0x9c/0xa3 [ 89.947584] [] do_raw_spin_lock+0x47/0x137 [ 89.953600] [] _raw_spin_lock+0x56/0x69 [ 89.959345] [] ? __mark_inode_dirty+0x66/0x1d0 [ 89.965734] [] __mark_inode_dirty+0x66/0x1d0 [ 89.971952] [] nfs_commit_inode+0xf1/0x1c1 [ 89.978022] [] nfs_write_inode+0x3e/0x93 [ 89.983913] [] ? _raw_spin_unlock+0x2b/0x2f [ 89.990068] [] writeback_single_inode+0x17a/0x267 [ 89.996739] [] writeback_sb_inodes+0xcf/0x157 [ 90.003032] [] writeback_inodes_wb+0x131/0x143 [ 90.009415] [] wb_writeback+0x27a/0x3c3 [ 90.015166] [] wb_do_writeback+0x1bb/0x1d6 [ 90.021206] [] bdi_writeback_thread+0x8b/0x212 [ 90.027572] [] ? wb_do_writeback+0x1d6/0x1d6 [ 90.033758] [] kthread+0x8e/0x96 [ 90.038906] [] kernel_thread_helper+0x4/0x10 [ 90.045115] [] ? retint_restore_args+0x13/0x13 [ 90.051494] [] ? __init_kthread_worker+0x5b/0x5b [ 90.058047] [] ? gs_change+0x13/0x13 [ 214.206745] BUG: spinlock lockup on CPU#4, cat/2538, ffff8801223c9240 [ 214.213459] Pid: 2538, comm: cat Not tainted 2.6.39-rc3-dt7+ #175 [ 214.219823] Call Trace: [ 214.222563] [] do_raw_spin_lock+0x10c/0x137 [ 214.228658] [] _raw_spin_lock+0x56/0x69 [ 214.234406] [] ? nfs_page_find_request+0x26/0x45 [ 214.240928] [] nfs_page_find_request+0x26/0x45 [ 214.242705] BUG: spinlock lockup on CPU#2, kworker/2:0/12, ffff8801223c9240 [ 214.242708] Pid: 12, comm: kworker/2:0 Not tainted 2.6.39-rc3-dt7+ #175 [ 214.242710] Call Trace: [ 214.242717] [] do_raw_spin_lock+0x10c/0x137 [ 214.242721] [] _raw_spin_lock+0x56/0x69 [ 214.242727] [] ? __mark_inode_dirty+0x66/0x1d0 [ 214.242729] [] __mark_inode_dirty+0x66/0x1d0 [ 214.242733] [] nfs_mark_request_commit+0xe4/0xed [ 214.242736] [] nfs_writeback_release_full+0x115/0x18f [ 214.242739] [] ? rpc_async_schedule+0x17/0x17 [ 214.242742] [] rpc_release_calldata+0x17/0x19 [ 214.242745] [] rpc_free_task+0x5e/0x67 [ 214.242747] [] rpc_async_release+0x15/0x17 [ 214.242751] [] process_one_work+0x236/0x3ea [ 214.242753] [] ? process_one_work+0x1a8/0x3ea [ 214.242756] [] worker_thread+0x17d/0x242 [ 214.242758] [] ? manage_workers+0x173/0x173 [ 214.242760] [] ? manage_workers+0x173/0x173 [ 214.242763] [] kthread+0x8e/0x96 [ 214.242767] [] kernel_thread_helper+0x4/0x10 [ 214.242770] [] ? retint_restore_args+0x13/0x13 [ 214.242772] [] ? __init_kthread_worker+0x5b/0x5b [ 214.242774] [] ? gs_change+0x13/0x13 [ 214.385858] [] ? grab_cache_page_write_begin+0x74/0xa4 [ 214.392930] [] nfs_flush_incompatible+0x2b/0x93 [ 214.399404] [] nfs_write_begin+0xeb/0x1c4 [ 214.405328] [] generic_file_buffered_write+0x119/0x240 [ 214.412371] [] ? nfs3_getxattr+0x52/0xc0 [ 214.418195] [] ? generic_file_aio_write+0x47/0xbe [ 214.424804] [] __generic_file_aio_write+0x334/0x364 [ 214.431626] [] generic_file_aio_write+0x63/0xbe [ 214.438075] [] nfs_file_write+0xe4/0x167 [ 214.443914] [] do_sync_write+0xcb/0x108 [ 214.449684] [] ? mutex_unlock+0xe/0x10 [ 214.455338] [] vfs_write+0xb8/0x157 [ 214.460751] [] sys_write+0x4d/0x77 [ 214.466069] [] system_call_fastpath+0x16/0x1b [ 236.471550] BUG: spinlock lockup on CPU#3, flush-0:24/2548, ffff8801223c9240 [ 236.478866] Pid: 2548, comm: flush-0:24 Not tainted 2.6.39-rc3-dt7+ #175 [ 236.485850] Call Trace: [ 236.488584] [] do_raw_spin_lock+0x10c/0x137 [ 236.494668] [] _raw_spin_lock+0x56/0x69 [ 236.500439] [] ? __mark_inode_dirty+0x66/0x1d0 [ 236.506795] [] __mark_inode_dirty+0x66/0x1d0 [ 236.512992] [] nfs_commit_inode+0xf1/0x1c1 [ 236.519017] [] nfs_write_inode+0x3e/0x93 [ 236.524849] [] ? _raw_spin_unlock+0x2b/0x2f [ 236.530956] [] writeback_single_inode+0x17a/0x267 [ 236.537584] [] writeback_sb_inodes+0xcf/0x157 [ 236.543855] [] writeback_inodes_wb+0x131/0x143 [ 236.550210] [] wb_writeback+0x27a/0x3c3 [ 236.562384] [] wb_do_writeback+0x1bb/0x1d6 [ 236.568385] [] bdi_writeback_thread+0x8b/0x212 [ 236.574746] [] ? wb_do_writeback+0x1d6/0x1d6 [ 236.580930] [] kthread+0x8e/0x96 [ 236.586092] [] kernel_thread_helper+0x4/0x10 [ 236.592268] [] ? retint_restore_args+0x13/0x13 [ 236.598617] [] ? __init_kthread_worker+0x5b/0x5b [ 236.605141] [] ? gs_change+0x13/0x13 [ 355.807574] BUG: spinlock lockup on CPU#7, test-dd.sh/3661, ffff8801223c9240 [ 355.814886] Pid: 3661, comm: test-dd.sh Not tainted 2.6.39-rc3-dt7+ #175 [ 355.821869] Call Trace: [ 355.824604] [] do_raw_spin_lock+0x10c/0x137 [ 355.830696] [] ? __d_lookup_rcu+0x14a/0x14a [ 355.836794] [] _raw_spin_lock+0x56/0x69 [ 355.842543] [] ? nfs_refresh_inode+0x2d/0x50 [ 355.848729] [] nfs_refresh_inode+0x2d/0x50 [ 355.854733] [] nfs_readdir_page_filler+0x25a/0x46f [ 355.861431] [] nfs_readdir_xdr_to_array+0x1ee/0x25a [ 355.868217] [] nfs_readdir_filler+0x26/0x7b [ 355.874309] [] ? add_to_page_cache_lru+0x53/0x5b [ 355.880830] [] do_read_cache_page+0x85/0x136 [ 355.887038] [] ? nfs_readdir_xdr_to_array+0x25a/0x25a [ 355.894018] [] read_cache_page_async+0x1c/0x1e [ 355.900380] [] read_cache_page+0xe/0x18 [ 355.906123] [] nfs_readdir+0x139/0x42b [ 355.911770] [] ? filldir64+0xd5/0xd5 [ 355.917272] [] ? nfs3_xdr_dec_readlink3res+0x11d/0x11d [ 355.924313] [] ? filldir64+0xd5/0xd5 [ 355.929792] [] vfs_readdir+0x6a/0xa3 [ 355.935284] [] sys_getdents+0x85/0xd8 [ 355.940862] [] system_call_fastpath+0x16/0x1b [ 617.266081] flush-8:0 used greatest stack depth: 2600 bytes left