From: Eric Dumazet <eric.dumazet@gmail.com>
To: "Zhang, Yanmin" <yanmin_zhang@linux.intel.com>
Cc: Christoph Lameter <cl@linux-foundation.org>,
netdev <netdev@vger.kernel.org>, Tejun Heo <tj@kernel.org>,
Pekka Enberg <penberg@cs.helsinki.fi>,
alex.shi@intel.com,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"Ma, Ling" <ling.ma@intel.com>,
"Chen, Tim C" <tim.c.chen@intel.com>,
Andrew Morton <akpm@linux-foundation.org>
Subject: Re: hackbench regression due to commit 9dfc6e68bfe6e
Date: Wed, 07 Apr 2010 11:20:33 +0200 [thread overview]
Message-ID: <1270632033.2091.875.camel@edumazet-laptop> (raw)
In-Reply-To: <1270631267.2078.380.camel@ymzhang.sh.intel.com>
Le mercredi 07 avril 2010 à 17:07 +0800, Zhang, Yanmin a écrit :
> >
> > One experiment on your Nehalem machine would be to change hackbench so
> > that each group (20 senders/ 20 receivers) run on a particular NUMA
> > node.
> I expect process scheduler to work well in scheduling different groups
> to different nodes.
>
> I suspected dynamic percpu data didn't take care of NUMA, but kernel dump shows
> it does take care of NUMA.
>
hackbench allocates all unix sockets on one single node, then
forks/spans its children.
Thats huge node imbalance.
You can see this with lsof on a running hackbench :
# lsof -p 14802
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
hackbench 14802 root cwd DIR 104,7 4096 12927240 /data/src/linux-2.6
hackbench 14802 root rtd DIR 104,2 4096 2 /
hackbench 14802 root txt REG 104,2 17524 697317 /usr/bin/hackbench
hackbench 14802 root mem REG 104,2 112212 558042 /lib/ld-2.3.4.so
hackbench 14802 root mem REG 104,2 1547588 558043 /lib/tls/libc-2.3.4.so
hackbench 14802 root mem REG 104,2 107928 557058 /lib/tls/libpthread-2.3.4.so
hackbench 14802 root mem REG 0,0 0 [heap] (stat: No such file or directory)
hackbench 14802 root 0u CHR 136,0 3 /dev/pts/0
hackbench 14802 root 1u CHR 136,0 3 /dev/pts/0
hackbench 14802 root 2u CHR 136,0 3 /dev/pts/0
hackbench 14802 root 3u unix 0xffff8800ac0da100 28939 socket
hackbench 14802 root 4u unix 0xffff8800ac0da400 28940 socket
hackbench 14802 root 5u unix 0xffff8800ac0da700 28941 socket
hackbench 14802 root 6u unix 0xffff8800ac0daa00 28942 socket
hackbench 14802 root 8u unix 0xffff8800aeac1800 28984 socket
hackbench 14802 root 9u unix 0xffff8800aeac1e00 28986 socket
hackbench 14802 root 10u unix 0xffff8800aeac2400 28988 socket
hackbench 14802 root 11u unix 0xffff8800aeac2a00 28990 socket
hackbench 14802 root 12u unix 0xffff8800aeac3000 28992 socket
hackbench 14802 root 13u unix 0xffff8800aeac3600 28994 socket
hackbench 14802 root 14u unix 0xffff8800aeac3c00 28996 socket
hackbench 14802 root 15u unix 0xffff8800aeac4200 28998 socket
hackbench 14802 root 16u unix 0xffff8800aeac4800 29000 socket
hackbench 14802 root 17u unix 0xffff8800aeac4e00 29002 socket
hackbench 14802 root 18u unix 0xffff8800aeac5400 29004 socket
hackbench 14802 root 19u unix 0xffff8800aeac5a00 29006 socket
hackbench 14802 root 20u unix 0xffff8800aeac6000 29008 socket
hackbench 14802 root 21u unix 0xffff8800aeac6600 29010 socket
hackbench 14802 root 22u unix 0xffff8800aeac6c00 29012 socket
hackbench 14802 root 23u unix 0xffff8800aeac7200 29014 socket
hackbench 14802 root 24u unix 0xffff8800aeac0f00 29016 socket
hackbench 14802 root 25u unix 0xffff8800aeac0900 29018 socket
hackbench 14802 root 26u unix 0xffff8800aeac7b00 29020 socket
hackbench 14802 root 27u unix 0xffff8800aeac7500 29022 socket
All sockets structures (where all _hot_ locks reside) are on a single node.
next prev parent reply other threads:[~2010-04-07 9:20 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <1269506457.4513.141.camel@alexs-hp.sh.intel.com>
[not found] ` <alpine.DEB.2.00.1003250942080.2670@router.home>
[not found] ` <1269570902.9614.92.camel@alexs-hp.sh.intel.com>
[not found] ` <1270114166.2078.107.camel@ymzhang.sh.intel.com>
[not found] ` <alpine.DEB.2.00.1004011050340.16531@router.home>
[not found] ` <1270195589.2078.116.camel@ymzhang.sh.intel.com>
[not found] ` <alpine.DEB.2.00.1004050853300.23149@router.home>
[not found] ` <i2z84144f021004051030k7ff5190cyc083aa12c552dfac@mail.gmail.com>
[not found] ` <4BBA8DF9.8010409@kernel.org>
[not found] ` <1270542497.2078.123.camel@ymzhang.sh.intel.com>
[not found] ` <alpine.DEB.2.00.1004061033330.18750@router.home>
[not found] ` <alpine.DEB.2.00.1004061552500.19151@router.home>
2010-04-06 22:10 ` hackbench regression due to commit 9dfc6e68bfe6e Eric Dumazet
2010-04-07 2:34 ` Zhang, Yanmin
2010-04-07 6:39 ` Eric Dumazet
2010-04-07 9:07 ` Zhang, Yanmin
2010-04-07 9:20 ` Eric Dumazet [this message]
2010-04-07 10:47 ` Pekka Enberg
2010-04-07 16:30 ` Christoph Lameter
2010-04-07 16:43 ` Christoph Lameter
2010-04-07 16:49 ` Pekka Enberg
2010-04-07 16:52 ` Pekka Enberg
2010-04-07 18:20 ` Christoph Lameter
2010-04-07 18:25 ` Pekka Enberg
2010-04-07 19:30 ` Christoph Lameter
2010-04-07 18:38 ` Eric Dumazet
2010-04-08 1:05 ` Zhang, Yanmin
2010-04-08 4:59 ` Eric Dumazet
2010-04-08 5:39 ` Eric Dumazet
2010-04-08 7:00 ` Eric Dumazet
2010-04-08 7:05 ` David Miller
2010-04-08 7:20 ` David Miller
2010-04-08 7:25 ` Eric Dumazet
2010-04-08 7:54 ` Zhang, Yanmin
2010-04-08 7:54 ` Eric Dumazet
2010-04-08 8:09 ` Eric Dumazet
2010-04-08 15:34 ` Christoph Lameter
2010-04-08 15:52 ` Eric Dumazet
2010-04-07 18:18 ` Christoph Lameter
2010-04-08 7:18 ` Zhang, Yanmin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1270632033.2091.875.camel@edumazet-laptop \
--to=eric.dumazet@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=alex.shi@intel.com \
--cc=cl@linux-foundation.org \
--cc=ling.ma@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=penberg@cs.helsinki.fi \
--cc=tim.c.chen@intel.com \
--cc=tj@kernel.org \
--cc=yanmin_zhang@linux.intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox