how to get a DMA channel near a process?

how to get a DMA channel near a process?

[Brice Goglin]

Hello,

I am playing with DMA engine on a Nehalem box with two X58 chipsets
(Supermicro X8DAH). My understanding is that there are 8 DMA channels on
each chipset, so 8 channels near each processor. Unfortunately, my BIOS
and 2.6.31-rc still wrongly reports the physical location of my devices
(it claims all PCI devices are near the first processor) but I worked
around the problem manually.

The offloaded copy performance changes a lot depending on whether the
process memory is allocated near the DMA device. So first I would like
to know if DMA channels are allocated near the requesting
processor/process. Then I guess it's possible to read the cpu mask near
a given chan by following chan->dev.device up to the pci device, right?
But is there any way to request a DMA channel near a specific socket or
NUMA node or cpu mask?

Thanks,
Brice

Re: how to get a DMA channel near a process?

[Robert Hancock]

On 08/25/2009 07:05 AM, Brice Goglin wrote:
> Hello,
>
> I am playing with DMA engine on a Nehalem box with two X58 chipsets
> (Supermicro X8DAH). My understanding is that there are 8 DMA channels on
> each chipset, so 8 channels near each processor. Unfortunately, my BIOS

Don't think that DMA channel is the term you are looking for, maybe PCI 
Express lanes? DMA channels are an ancient ISA thing.

> and 2.6.31-rc still wrongly reports the physical location of my devices
> (it claims all PCI devices are near the first processor) but I worked
> around the problem manually.

Presumably incorrect information in the BIOS tables (SLIT or MADT, maybe?)

>
> The offloaded copy performance changes a lot depending on whether the
> process memory is allocated near the DMA device. So first I would like
> to know if DMA channels are allocated near the requesting
> processor/process. Then I guess it's possible to read the cpu mask near

Not sure quite what you are trying to say here. Normally when memory is 
allocated for DMA use for a specific device I believe the kernel tries 
to allocate it on the same node that the device is attached to.

> a given chan by following chan->dev.device up to the pci device, right?
> But is there any way to request a DMA channel near a specific socket or
> NUMA node or cpu mask?
>
> Thanks,
> Brice
>
>

Re: how to get a DMA channel near a process?

[Brice Goglin]

Robert Hancock wrote:
> On 08/25/2009 07:05 AM, Brice Goglin wrote:
>> Hello,
>>
>> I am playing with DMA engine on a Nehalem box with two X58 chipsets
>> (Supermicro X8DAH). My understanding is that there are 8 DMA channels on
>> each chipset, so 8 channels near each processor. Unfortunately, my BIOS
>
> Don't think that DMA channel is the term you are looking for, maybe
> PCI Express lanes? DMA channels are an ancient ISA thing.

Talking about DMA Engine devices/channels such as Intel I/OAT. See
include/linux/dmaengine.h

Brice

Re: how to get a DMA channel near a process?

[Dan Williams]

On Tue, Aug 25, 2009 at 6:05 AM, Brice Goglin<Brice.Goglin@inria.fr> wrote:
> Hello,
>
> I am playing with DMA engine on a Nehalem box with two X58 chipsets
> (Supermicro X8DAH). My understanding is that there are 8 DMA channels on
> each chipset, so 8 channels near each processor. Unfortunately, my BIOS
> and 2.6.31-rc still wrongly reports the physical location of my devices
> (it claims all PCI devices are near the first processor) but I worked
> around the problem manually.
>
> The offloaded copy performance changes a lot depending on whether the
> process memory is allocated near the DMA device. So first I would like
> to know if DMA channels are allocated near the requesting
> processor/process. Then I guess it's possible to read the cpu mask near
> a given chan by following chan->dev.device up to the pci device, right?
> But is there any way to request a DMA channel near a specific socket or
> NUMA node or cpu mask?

If we can determine the topology of which chipset is closer to which
cpu package then we can modify the dmaengine channel_table to return
"local" channels via dma_find_channel().  The current allocation
policy is blind per-cpu round robin such that cpu-0 gets channel-0,
cpu-1 gets channel-1...  As you pointed out the challenge will be
finding a reliable source for the topology information.

--
Dan