From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Chinner Subject: Re: [PATCH 00/46] rcu-walk and dcache scaling Date: Wed, 8 Dec 2010 14:32:12 +1100 Message-ID: <20101208033212.GF29333@dastard> References: <20101207215653.GA25864@dastard> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: Nick Piggin , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org To: Nick Piggin Return-path: Content-Disposition: inline In-Reply-To: Sender: linux-kernel-owner@vger.kernel.org List-Id: linux-fsdevel.vger.kernel.org On Wed, Dec 08, 2010 at 12:47:42PM +1100, Nick Piggin wrote: > On Wed, Dec 8, 2010 at 8:56 AM, Dave Chinner wr= ote: > > On Sat, Nov 27, 2010 at 09:15:58PM +1100, Nick Piggin wrote: > >> > >> git://git.kernel.org/pub/scm/linux/kernel/git/npiggin/linux-npiggi= n.git vfs-scale-working > >> > >> Here is an new set of vfs patches for review, not that there was m= uch interest > >> last time they were posted. It is structured like: > >> > >> * preparation patches > >> * introduce new locks to take over dcache_lock, then remove it > >> * cleaning up and reworking things for new locks > >> * rcu-walk path walking > >> * start on some fine grained locking steps > > > > Stress test doing: > > > > =A0 =A0 =A0 =A0single thread 50M inode create > > =A0 =A0 =A0 =A0single thread rm -rf > > =A0 =A0 =A0 =A02-way 50M inode create > > =A0 =A0 =A0 =A02-way rm -rf > > =A0 =A0 =A0 =A04-way 50M inode create > > =A0 =A0 =A0 =A04-way rm -rf > > =A0 =A0 =A0 =A08-way 50M inode create > > =A0 =A0 =A0 =A08-way rm -rf > > =A0 =A0 =A0 =A08-way 250M inode create > > =A0 =A0 =A0 =A08-way rm -rf > > > > Failed about 5 minutes into the "4-way rm -rf" (~3 hours into the t= est) > > with a CPU stuck spinning on here: > > > > [37372.084012] NMI backtrace for cpu 5 > > [37372.084012] CPU 5 > > [37372.084012] Modules linked in: > > [37372.084012] > > [37372.084012] Pid: 15214, comm: rm Not tainted 2.6.37-rc4-dgc+ #79= 7 /Bochs > > [37372.084012] RIP: 0010:[] =A0[] __ticket_spin_lock+0x14/0x20 > > [37372.084012] RSP: 0018:ffff880114643c98 =A0EFLAGS: 00000213 > > [37372.084012] RAX: 0000000000008801 RBX: ffff8800687be6c0 RCX: fff= f8800c4eb2688 > > [37372.084012] RDX: ffff880114643d38 RSI: ffff8800dfd4ea80 RDI: fff= f880114643d14 > > [37372.084012] RBP: ffff880114643c98 R08: 0000000000000003 R09: 000= 0000000000000 > > [37372.084012] R10: 0000000000000000 R11: dead000000200200 R12: fff= f880114643d14 > > [37372.084012] R13: ffff880114643cb8 R14: ffff880114643d38 R15: fff= f8800687be71c > > [37372.084012] FS: =A000007fd6d7c93700(0000) GS:ffff8800dfd40000(00= 00) knlGS:0000000000000000 > > [37372.084012] CS: =A00010 DS: 0000 ES: 0000 CR0: 000000008005003b > > [37372.084012] CR2: 0000000000bbd108 CR3: 0000000107146000 CR4: 000= 00000000006e0 > > [37372.084012] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 000= 0000000000000 > > [37372.084012] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 000= 0000000000400 > > [37372.084012] Process rm (pid: 15214, threadinfo ffff880114642000,= task ffff88011b16f890) > > [37372.084012] Stack: > > [37372.084012] =A0ffff880114643ca8 ffffffff81ad044e ffff880114643cf= 8 ffffffff81167ae7 > > [37372.084012] =A00000000000000000 ffff880114643d38 000000000000000= e ffff88011901d800 > > [37372.084012] =A0ffff8800cdb7cf5c ffff88011901d8e0 000000000000000= 0 0000000000000000 > > [37372.084012] Call Trace: > > [37372.084012] =A0[] _raw_spin_lock+0xe/0x20 > > [37372.084012] =A0[] shrink_dentry_list+0x47/0x37= 0 > > [37372.084012] =A0[] __shrink_dcache_sb+0x14e/0x1= e0 > > [37372.084012] =A0[] shrink_dcache_parent+0x276/0= x2d0 > > [37372.084012] =A0[] ? _raw_spin_lock+0xe/0x20 > > [37372.084012] =A0[] dentry_unhash+0x42/0x80 > > [37372.084012] =A0[] vfs_rmdir+0x68/0x100 > > [37372.084012] =A0[] do_rmdir+0x113/0x130 > > [37372.084012] =A0[] ? filp_close+0x5d/0x90 > > [37372.084012] =A0[] sys_unlinkat+0x35/0x40 > > [37372.084012] =A0[] system_call_fastpath+0x16/0x= 1b >=20 > OK good, with any luck, that's the same bug. >=20 > Is this XFS? Yes. > Is there any concurrent activity happening on the same dentries? Not from an application perspective. > Ie. are the rm -rf threads running on the same directories, No, each thread operating on a different directory. > or is there any reclaim happening in the background? IIRC, kswapd was consuming about 5-10% of a CPU during parallel unlink tests. Mainly reclaiming XFS inodes, I think, but there may be dentry cache reclaim going as well. Cheers, Dave. --=20 Dave Chinner david@fromorbit.com