From mboxrd@z Thu Jan 1 00:00:00 1970 From: Nick Piggin Subject: Re: more dput lock contentions in 2.6.38-rc? Date: Tue, 25 Jan 2011 13:09:48 +1100 Message-ID: References: <1295915725.1949.967.camel@sli10-conroe> <1295917864.1949.970.camel@sli10-conroe> <1295919258.1949.973.camel@sli10-conroe> <20110125020123.GA30965@sli10-conroe.sh.intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: "linux-fsdevel@vger.kernel.org" , lkml , Andrew Morton , Nick Piggin , "Chen, Tim C" To: Shaohua Li Return-path: In-Reply-To: <20110125020123.GA30965@sli10-conroe.sh.intel.com> Sender: linux-kernel-owner@vger.kernel.org List-Id: linux-fsdevel.vger.kernel.org On Tue, Jan 25, 2011 at 1:01 PM, Shaohua Li wrot= e: > On Tue, Jan 25, 2011 at 09:44:45AM +0800, Nick Piggin wrote: >> On Tue, Jan 25, 2011 at 12:34 PM, Shaohua Li = wrote: >> > On Tue, 2011-01-25 at 09:26 +0800, Nick Piggin wrote: >> >> On Tue, Jan 25, 2011 at 12:11 PM, Shaohua Li wrote: >> >> > On Tue, 2011-01-25 at 09:04 +0800, Nick Piggin wrote: >> >> >> On Tue, Jan 25, 2011 at 11:35 AM, Shaohua Li wrote: >> >> >> > Hi, >> >> >> > we are testing dbench benchmark and see big drop of 2.6.38-r= c compared >> >> >> > to 2.6.37 in several machines with 2 sockets or 4 sockets. W= e have 12 >> >> >> > disks mount to /mnt/stp/dbenchdata/sd*/ and dbench runs agai= nst data of >> >> >> > the disks. According to perf, we saw more lock contentions: >> >> >> > In 2.6.37: 13.00% =A0 =A0 =A0 =A0dbench =A0[kernel.kallsyms]= =A0 [k] _raw_spin_lock >> >> >> > In 2.6.38-rc: 69.45% =A0 =A0 =A0 =A0dbench =A0[kernel.kallsy= ms] =A0 [k]_raw_spin_lock >> >> >> > - =A0 =A0 69.45% =A0 =A0 =A0 =A0dbench =A0[kernel.kallsyms] = =A0 [k] _raw_spin_lock >> >> >> > =A0 - _raw_spin_lock >> >> >> > =A0 =A0 =A0- 48.41% dput >> >> >> > =A0 =A0 =A0 =A0 - 61.17% path_put >> >> >> > =A0 =A0 =A0 =A0 =A0 =A0- 60.47% do_path_lookup >> >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 + 53.18% user_path_at >> >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 + 42.13% do_filp_open >> >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 + 4.69% user_path_parent >> >> >> >> >> >> What filesystems are mounted on the path? >> >> > ext3 or ext4 >> >> >> >> ext3 or 4 along every step of the path? Are there >> >> any acls loaded, or security policy running? >> > all disks are formated with the same fs, just some machines use ex= t3 and >> > others ext4. no we don't have acl or security policy. >> >> It may be possible that they're all coming from >> >> /proc/ access. >> > I added trace in dput just after the lock taken. and most files ar= e '/', >> > 'mnt', 'stp'. the percentage of 'proc' is small actually. >> >> Hm, OK well I could send you a patch to gather some statistics for >> why rcu-walk gets dropped. It'll have to wait until I get home, thou= gh. > Sure, I'm still at office tomorrow. I can test before that day, other= wise > maybe Tim can help. There _are_ a lot of dput contentions coming from d_path. Some of the dentries you're seeing in dput could be coming from path_put in d_path (mountpoints, eg. 'mnt'). What does an actual snippet from perf with callgraphs look like?