Re: PCIe device driver question

[Robert Hancock <hancockr@shaw.ca>]

Sanka Piyaratna wrote:
> I allocate memory in the user land using memalign function (typically I allocate about 500 MB) and pass this to the kernel space. In my device driver, I call get_user_pages() to lock down the memory and extract the relevant pages. A scatter-gather list is generated using these page addresses and hence derive the dma_addresses using page_to_phys() function. These addresses are programmed into a FIFO in the hardware device using a memory mapped register interface (PCI BAR based). Subsequently the hardware start filling up the pages and interrupt when a block of pages are complete. I notice the hardware hang (PCIe packets don't seem to get the acknowledgements from the root complex) when the DMA address is < 0x0000_0001_0000_0000. I have verified in the hardware that the PCIe packet is created with the correct address as programed by the device driver dma_address. If i can guard some how that the memory allocation is with in a certain area, I can stop the
>  problem from occuring. Are there any bridge functionality in the Intel architecture that may mask a certain region of memory?

How are you formatting the addresses in the TLP? The PCI Express spec 
says that for addresses below 4GB the 32-bit addressing format must be 
used (you can't use the 64-bit format unconditionally). It could be that 
is what is making the chipset unhappy.