Re: problems doing direct dma from a pci device to pci-e device

[Mark Hounschell <markh@compro.net>]

Richard B. Johnson wrote:
> ----- Original Message ----- From: "Mark Hounschell" <markh@compro.net>
> To: <linux-pci@vger.kernel.org>
> Cc: "Mark Hounschell" <dmarkh@cfl.rr.com>; <linux-kernel@vger.kernel.org>
> Sent: Friday, September 11, 2009 10:13 AM
> Subject: problems doing direct dma from a pci device to pci-e device
> 
> 
>> I know this is really just a pci issue but have CC'd LKML just in
>> case. Sorry LKML for the noise.
>>
>> I have a situation where a card on a regular PCI bus (Compro gpiohsd)
>> does dma's
>> directly into another PCI cards memory (VMIC-5565 reflective memory)
>> living on
>> another PCI bus. These 2 cards are sometimes seperated by many bridges.
>> Expansion racks etc. We've been doing this forever. No problem (mostly).
>>
>> I now have an AM3 based DFI DK 790FXB-M3H5 motherboard. This board has
>> 3 regular
>> PCI slots and 3 PCI-E (16x) slots. I also have a PCI-E (x4) version of
>> the VMIC-5565
>> reflective memory  card in one of the PCI-E slots and our gpiohsd card
>> in one of the regular
>> PCI slots. All on the motherboard. No expansion slots being used.
>> However I cannot get
>> data from our gpiohsd into the PCI-E VMIC-5565 cards memory. I can
>> certainly get the data there
>> from a userland buffer, no problem. Just not from one card to the
>> other directly. Oh and when
>> I put the regular PCI version of the VMIC into one of the regular PCI
>> slots everything works
>> as expected. They are then both on the same PCI bus and no bridges are
>> involved though.
>>
> 
> [SNIPPED...]
> 
> Please check your software to make sure that after the DMA has presumed
> to complete, you READ something from the destination PCI bus address!
> With so many PCI writes queued, they may all be stuck in the many
> hardware FIFOs and the sure way to complete the writes is to perform a
> read.
> Cheers,
> Richard B. Johnson
> http://Route495Software.com/
> 
> 
> 
> 
> 

Thanks, that's actually how I know the data never gets to it's destination. I read it. 
The data gets lost somewhere.

Regards
Mark