Re: [RFC] Idea about increasing efficency of skb allocation in network devices

[Brice Goglin <Brice.Goglin@inria.fr>]

David Miller wrote:
> From: Neil Horman <nhorman@tuxdriver.com>
> Date: Sun, 26 Jul 2009 20:36:09 -0400
>
>   
>> 	Since Network devices dma their memory into a provided DMA
>> buffer (which can usually be at an arbitrary location, as they must
>> cross potentially several pci busses to reach any memory location),
>> I'm postulating that it would increase our receive path efficiency
>> to provide a hint to the driver layer as to which node to allocate
>> an skb data buffer on.  This hint would be determined by a feedback
>> mechanism.  I was thinking that we could provide a callback function
>> via the skb, that accepted the skb and the originating net_device.
>> This callback can track statistics on which numa nodes consume
>> (read: copy data from) skbs that were produced by specific net
>> devices.  Then, when in the future that netdevice allocates a new
>> skb (perhaps via netdev_alloc_skb), we can use that statistical
>> profile to determine if the data buffer should be allocated on the
>> local node, or on a remote node instead.
>>     
>
> No matter what, you will do an inter-node memory operation.
>
> Unless, the consumer NUMA node is the same as the one the
> device is on.
>
> Because since the device is on a NUMA node, if you DMA remotely
> you've eaten the NUMA cost already.
>
> If you always DMA to the device's NUMA node (what we try to do now) at
> least the is the possibility of eliminating cross-NUMA traffic.
>
> Better to move the application or stack processing towards the NUMA
> node the network device is on, I think.
>   

Is there an easy way to get this NUMA node from the application socket
descriptor?
Also, one question that was raised at the Linux Symposium is: how do you
know which processors run the receive queue for a specific connection ?
It would be nice to have a way to retrieve such information in the
application to avoid inter-node and inter-core/cache traffic.

Brice