From: Greg Lindahl <greg@blekko.com>
To: Stephen Hemminger <shemminger@vyatta.com>
Cc: David Miller <davem@davemloft.net>,
therbert@google.com, eric.dumazet@gmail.com,
netdev@vger.kernel.org
Subject: Re: [PATCH] xps-mq: Transmit Packet Steering for multiqueue
Date: Wed, 1 Sep 2010 23:41:36 -0700 [thread overview]
Message-ID: <20100902064136.GA8633@bx9.net> (raw)
In-Reply-To: <20100901185627.239ad165@nehalam>
On Wed, Sep 01, 2010 at 06:56:27PM -0700, Stephen Hemminger wrote:
> Just to be contrarian :-) This same idea had started before when IBM
> proposed a user-space NUMA API. It never got any traction, the concept
> of "lets make the applications NUMA aware" never got accepted because
> it is so hard to do right and fragile that it was the wrong idea
> to start with. The only people that can manage it are the engineers
> tweeking a one off database benchmark.
As an non-database user-space example, there are many applications
which know about the typical 'first touch' locality policy for pages
and use that to be NUMA-aware. Just about every OpenMP program ever
written does that; it's even fairly portable among OSes.
A second user-level example is MPI implementations such as OpenMPI.
Those guys run 1 process per core and they don't need to move around,
so getting process locked to a core and all the pages in the right
place is a nice win without the MPI programmer doing anything.
For kernel (but non-Ethernet) networking examples, HPC interconnects
typically go out of their way to ensure locality of kernel pages
related to a given core's workload. Examples include Myrinet's
OpenMX+MPI and the InfiniPath InfiniBand adapater, whatever QLogic
renamed it to this week (TrueScale, I suppose.) How can you get ~ 1
microsecond messages if you've got a buffer in the wrong place? Or
achieve extremely high messaging rates when you're waiting for remote
memory all the time?
> I would rather see a "good enough" policy in the kernel that works
> for everything from a single-core embedded system to a 100 core
> server environment.
I'd like a pony. Yes, it's challenging to directly aapply the above
networking example to Ethernet networking, but there's a pony in there
somewhere.
-- greg
next prev parent reply other threads:[~2010-09-02 7:14 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-08-23 5:39 [PATCH] xps-mq: Transmit Packet Steering for multiqueue Tom Herbert
2010-08-23 17:09 ` Ben Hutchings
[not found] ` <AANLkTinST5zaS0NtBjrzyLbsg=w_EVsHE3DCDcrmQNc6@mail.gmail.com>
2010-08-23 17:50 ` Ben Hutchings
2010-08-23 17:59 ` Stephen Hemminger
2010-09-01 15:41 ` Tom Herbert
2010-09-01 15:54 ` Eric Dumazet
2010-09-01 16:24 ` Tom Herbert
2010-09-02 1:32 ` David Miller
2010-09-02 1:48 ` Stephen Hemminger
2010-09-02 16:00 ` Loke, Chetan
2010-09-02 19:52 ` Tom Herbert
2010-09-02 23:17 ` Loke, Chetan
2010-09-02 1:56 ` Stephen Hemminger
2010-09-02 6:41 ` Greg Lindahl [this message]
2010-09-02 16:18 ` Loke, Chetan
2010-09-02 15:55 ` Loke, Chetan
2010-09-16 21:52 ` Ben Hutchings
2010-09-19 17:24 ` Michael S. Tsirkin
2010-09-20 12:44 ` [RFC][PATCH 1/3] IRQ: Add irq_get_numa_node() Ben Hutchings
2010-09-20 13:04 ` Eric Dumazet
2010-09-20 12:45 ` [RFC][PATCH 2/3] ethtool: NUMA affinity control Ben Hutchings
2010-09-20 12:48 ` [RFC][PATCH 3/3] sfc: Add support for " Ben Hutchings
2011-02-21 18:19 ` [PATCH] xps-mq: Transmit Packet Steering for multiqueue Ben Hutchings
2011-02-21 19:31 ` Jeremy Eder
2011-02-26 7:09 ` David Miller
2010-09-01 16:09 ` David Miller
2010-08-24 4:31 ` Bill Fink
2010-08-24 4:37 ` Tom Herbert
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20100902064136.GA8633@bx9.net \
--to=greg@blekko.com \
--cc=davem@davemloft.net \
--cc=eric.dumazet@gmail.com \
--cc=netdev@vger.kernel.org \
--cc=shemminger@vyatta.com \
--cc=therbert@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).