From mboxrd@z Thu Jan  1 00:00:00 1970
From: Nick Piggin <npiggin@gmail.com>
Subject: Re: more dput lock contentions in 2.6.38-rc?
Date: Tue, 25 Jan 2011 12:26:38 +1100
Message-ID: <AANLkTimwSNsjR5PnVP8JvzwEBwy3VtDp4DvNecKsTnxJ@mail.gmail.com>
References: <1295915725.1949.967.camel@sli10-conroe>
	<AANLkTikOdiQR+rWXzoU9z5sJv9Q2=ehdhi1NCHXT0a-w@mail.gmail.com>
	<1295917864.1949.970.camel@sli10-conroe>
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: QUOTED-PRINTABLE
Cc: "linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>,
	lkml <linux-kernel@vger.kernel.org>,
	Andrew Morton <akpm@linux-foundation.org>,
	Nick Piggin <npiggin@kernel.dk>,
	"Chen, Tim C" <tim.c.chen@intel.com>
To: Shaohua Li <shaohua.li@intel.com>
Return-path: <linux-kernel-owner@vger.kernel.org>
In-Reply-To: <1295917864.1949.970.camel@sli10-conroe>
Sender: linux-kernel-owner@vger.kernel.org
List-Id: linux-fsdevel.vger.kernel.org

On Tue, Jan 25, 2011 at 12:11 PM, Shaohua Li <shaohua.li@intel.com> wro=
te:
> On Tue, 2011-01-25 at 09:04 +0800, Nick Piggin wrote:
>> On Tue, Jan 25, 2011 at 11:35 AM, Shaohua Li <shaohua.li@intel.com> =
wrote:
>> > Hi,
>> > we are testing dbench benchmark and see big drop of 2.6.38-rc comp=
ared
>> > to 2.6.37 in several machines with 2 sockets or 4 sockets. We have=
 12
>> > disks mount to /mnt/stp/dbenchdata/sd*/ and dbench runs against da=
ta of
>> > the disks. According to perf, we saw more lock contentions:
>> > In 2.6.37: 13.00% =A0 =A0 =A0 =A0dbench =A0[kernel.kallsyms] =A0 [=
k] _raw_spin_lock
>> > In 2.6.38-rc: 69.45% =A0 =A0 =A0 =A0dbench =A0[kernel.kallsyms] =A0=
 [k]_raw_spin_lock
>> > - =A0 =A0 69.45% =A0 =A0 =A0 =A0dbench =A0[kernel.kallsyms] =A0 [k=
] _raw_spin_lock
>> > =A0 - _raw_spin_lock
>> > =A0 =A0 =A0- 48.41% dput
>> > =A0 =A0 =A0 =A0 - 61.17% path_put
>> > =A0 =A0 =A0 =A0 =A0 =A0- 60.47% do_path_lookup
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 + 53.18% user_path_at
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 + 42.13% do_filp_open
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 + 4.69% user_path_parent
>>
>> What filesystems are mounted on the path?
> ext3 or ext4

ext3 or 4 along every step of the path? Are there
any acls loaded, or security policy running?

It may be possible that they're all coming from
/proc/ access.

>
>> > =A0 =A0 =A0 =A0 =A0 =A0- 35.56% d_path
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 seq_path
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 show_vfsmnt
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 seq_read
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 vfs_read
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 sys_read
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 system_call_fastpath
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 __GI___libc_read
>>
>> This guy is from glibc's statvfs call that dbench uses. It
>> parses /proc/mounts for mount flags which is racy (and
>> not a good idea to do with any frequency).
>>
>> A patch went into the kernel that allows glibc to get the
>> flags directly. Not sure about glibc status, I imagine it
>> will get there in another decade or two... Can you try
>> commenting it out of dbench source code?
> Sure, maybe after Chinese new year holiday, sorry.

No problem.

Thanks,
Nick