Re: [PATCH net-next-2.6] sched: use xps information for qdisc NUMA affinity

[Eric Dumazet <eric.dumazet@gmail.com>]

Le mardi 30 novembre 2010 à 10:48 -0800, David Miller a écrit :

> Most drivers do, and all drivers ought to, allocate DMA queues and
> whatnot when the interface is brought up.
> 
> That solves this particular issue.
> 
> For example, drivers/net/niu.c does this by calling
> niu_alloc_channels() via niu_open().
> 
> The only thing we really can't handle currently is the netdev
> itself (and the associated driver private).  Jesse Brandeburg
> has been reminding me about this over and over :-)
> 
> There might be some things we can even do about that part.  For
> example, we can put all of the things the driver touches in the
> RX and TX fast paths via indirect pointers and therefore be able
> to allocate and reallocate those portions as we want long after
> device registry.
> 
> Doing the core netdev struct itself is too hard because it sits
> in so many tables.

netdev struct itself is shared by all cpus, so there is no real choice,
unless you know one netdev will be used by a restricted set of
cpus/nodes... Probably very unlikely in practice.

This can probably be done right now with 
numactl .... modprobe ...


We could change (only on NUMA setups maybe)

struct netdev_queue *_tx;

to a

struct netdev_queue **_tx;

and allocate each "struct netdev_queue" on appropriate node, but adding
one indirection level might be overkill...

For very hot small structures, (one or two cache lines), I am not sure
its worth the pain.


Ben, could you remind us what was your ethtool interface ?

Something to setup a NUMA map for RX queues, TX queues ?

I can probably play with bnx2x and custom module param to test how it
can help raw performance...