Re: PCIe device driver question

Re: PCIe device driver question

[Robert Hancock]

V.Radhakrishnan wrote:
>>> am testing this in an X86_64 architecture machine with 4 GB of RAM. I
>>> am able to successfully dma data into any memory (dma) address >
>>> 0x0000_0001_0000_0000.
> 
> How can you DMA "successfully" into this address which is > 4 GB when
> you have only 4 GB RAM ? Or am I missing something ?

The MMIO and other reserved memory space at the top of the 32-bit memory 
space will cause the top part of memory to be relocated above 4GB.

Re: PCIe device driver question

[V.Radhakrishnan]

Hi Robert,

Thanks for the reply. I was thinking that the MMIO and reserve memory
will be below 4 GB was only applicable for 32-bit environments, since I
don't have much experience in 64-bit.

However, I had an IDENTICAL problem over 2 years ago. I had used
posix_memalign() in user space to allocate pages aligned to 4096 byte
pages, allocated several additional memaligned pages in user space, used
mlock() to lock all these pages, gathered the user space addresses into
the original pages as arrays of structures, passed this array into the
kernel using an ioctl() call, used get_user_pages() to extract the
struct page pointers, performed a kmap() to get the kernel virtual
addresses and then extracted the physical addresses and 'sent' this to
the chip to perform DMA.

This situation is almost identical to what has been reported and hence
my interest.

However, I had a PCI access problem. The DMA was just NOT happening on
any machine which had highmem, i.e over 896 MB.

I "solved" the problem since I didn't have much time to do R&D, by
booting with kernel command line option of mem=512M and the DMA went
thru successfully.

This was the linux-2.6.15 kernel then. Since the project was basically
to test the DMA capability of the device, the actual address to where it
was DMA-ed didn't matter, and I got paid for my work. However, this
matter was always at the back of my head.

What could have been the problem with the x86 32-bit PCI ? 

Thanks and regards

V. Radhakrishnan
www.atr-labs.com

On Wed, 2008-07-30 at 13:21 -0600, Robert Hancock wrote:
> V.Radhakrishnan wrote:
> >>> am testing this in an X86_64 architecture machine with 4 GB of RAM. I
> >>> am able to successfully dma data into any memory (dma) address >
> >>> 0x0000_0001_0000_0000.
> > 
> > How can you DMA "successfully" into this address which is > 4 GB when
> > you have only 4 GB RAM ? Or am I missing something ?
> 
> The MMIO and other reserved memory space at the top of the 32-bit memory 
> space will cause the top part of memory to be relocated above 4GB.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: PCIe device driver question

[Robert Hancock]

V.Radhakrishnan wrote:
> Hi Robert,
> 
> Thanks for the reply. I was thinking that the MMIO and reserve memory
> will be below 4 GB was only applicable for 32-bit environments, since I
> don't have much experience in 64-bit.
> 
> However, I had an IDENTICAL problem over 2 years ago. I had used
> posix_memalign() in user space to allocate pages aligned to 4096 byte
> pages, allocated several additional memaligned pages in user space, used
> mlock() to lock all these pages, gathered the user space addresses into
> the original pages as arrays of structures, passed this array into the
> kernel using an ioctl() call, used get_user_pages() to extract the
> struct page pointers, performed a kmap() to get the kernel virtual
> addresses and then extracted the physical addresses and 'sent' this to
> the chip to perform DMA.
> 
> This situation is almost identical to what has been reported and hence
> my interest.
> 
> However, I had a PCI access problem. The DMA was just NOT happening on
> any machine which had highmem, i.e over 896 MB.

My guess there was a bug in your DMA mapping code. I don't think kmap is 
what is normally used for this. I think with get_user_pages one usually 
takes the returned page pointers to create an SG list and uses 
dma_map_sg to create a DMA mapping for them.

> 
> I "solved" the problem since I didn't have much time to do R&D, by
> booting with kernel command line option of mem=512M and the DMA went
> thru successfully.
> 
> This was the linux-2.6.15 kernel then. Since the project was basically
> to test the DMA capability of the device, the actual address to where it
> was DMA-ed didn't matter, and I got paid for my work. However, this
> matter was always at the back of my head.
> 
> What could have been the problem with the x86 32-bit PCI ? 
> 
> Thanks and regards
> 
> V. Radhakrishnan
> www.atr-labs.com
> 
> On Wed, 2008-07-30 at 13:21 -0600, Robert Hancock wrote:
>> V.Radhakrishnan wrote:
>>>>> am testing this in an X86_64 architecture machine with 4 GB of RAM. I
>>>>> am able to successfully dma data into any memory (dma) address >
>>>>> 0x0000_0001_0000_0000.
>>> How can you DMA "successfully" into this address which is > 4 GB when
>>> you have only 4 GB RAM ? Or am I missing something ?
>> The MMIO and other reserved memory space at the top of the 32-bit memory 
>> space will cause the top part of memory to be relocated above 4GB.
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at  http://www.tux.org/lkml/
> 
> 

Re: PCIe device driver question

[V.Radhakrishnan]

> My guess there was a bug in your DMA mapping code. I don't think kmap is 
> what is normally used for this. I think with get_user_pages one usually 
> takes the returned page pointers to create an SG list and uses 
> dma_map_sg to create a DMA mapping for them.

Looking at the actual code, I see that I had used kmap() only to obtain
kernel virtual addresses for the array of struct pages obtained from
user space by using get_user_pages.

Subsequently, I had used dma_map_single() and dma_unmap_single() calls
for single buffer calls.

The code didn't have bugs IMHO since it was used for extensive stress
testing the initial FPGA prototype as well as the final ASIC chip ,
sometimes running for over 4 days non-stop without breaking.

However, using Test Access Points on the board and using a Logic
Analyzer showed that DMA was NOT taking place when RAM > 896 MB was
used. The hardware gurus said that PCI bus cycles just didn't seem to be
taking place when RAM > 896 MB was used as the source OR destination
address.

Perhaps this was a problem in the earlier kernel(s) and has since been
rectified ? ( I was using 2.6.15 then ... )

I am just curious since Sanka Piyaratna reported a 'similar' kind of
situation.

Regards

V. Radhakrishnan



On Thu, 2008-07-31 at 11:37 -0600, Robert Hancock wrote:
> V.Radhakrishnan wrote:
> > Hi Robert,
> > 
> > Thanks for the reply. I was thinking that the MMIO and reserve memory
> > will be below 4 GB was only applicable for 32-bit environments, since I
> > don't have much experience in 64-bit.
> > 
> > However, I had an IDENTICAL problem over 2 years ago. I had used
> > posix_memalign() in user space to allocate pages aligned to 4096 byte
> > pages, allocated several additional memaligned pages in user space, used
> > mlock() to lock all these pages, gathered the user space addresses into
> > the original pages as arrays of structures, passed this array into the
> > kernel using an ioctl() call, used get_user_pages() to extract the
> > struct page pointers, performed a kmap() to get the kernel virtual
> > addresses and then extracted the physical addresses and 'sent' this to
> > the chip to perform DMA.
> > 
> > This situation is almost identical to what has been reported and hence
> > my interest.
> > 
> > However, I had a PCI access problem. The DMA was just NOT happening on
> > any machine which had highmem, i.e over 896 MB.
> 
> My guess there was a bug in your DMA mapping code. I don't think kmap is 
> what is normally used for this. I think with get_user_pages one usually 
> takes the returned page pointers to create an SG list and uses 
> dma_map_sg to create a DMA mapping for them.
> 
> > 
> > I "solved" the problem since I didn't have much time to do R&D, by
> > booting with kernel command line option of mem=512M and the DMA went
> > thru successfully.
> > 
> > This was the linux-2.6.15 kernel then. Since the project was basically
> > to test the DMA capability of the device, the actual address to where it
> > was DMA-ed didn't matter, and I got paid for my work. However, this
> > matter was always at the back of my head.
> > 
> > What could have been the problem with the x86 32-bit PCI ? 
> > 
> > Thanks and regards
> > 
> > V. Radhakrishnan
> > www.atr-labs.com
> > 
> > On Wed, 2008-07-30 at 13:21 -0600, Robert Hancock wrote:
> >> V.Radhakrishnan wrote:
> >>>>> am testing this in an X86_64 architecture machine with 4 GB of RAM. I
> >>>>> am able to successfully dma data into any memory (dma) address >
> >>>>> 0x0000_0001_0000_0000.
> >>> How can you DMA "successfully" into this address which is > 4 GB when
> >>> you have only 4 GB RAM ? Or am I missing something ?
> >> The MMIO and other reserved memory space at the top of the 32-bit memory 
> >> space will cause the top part of memory to be relocated above 4GB.
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >> Please read the FAQ at  http://www.tux.org/lkml/
> > 
> > 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: PCIe device driver question

[Robert Hancock]

V.Radhakrishnan wrote:
>> My guess there was a bug in your DMA mapping code. I don't think kmap is 
>> what is normally used for this. I think with get_user_pages one usually 
>> takes the returned page pointers to create an SG list and uses 
>> dma_map_sg to create a DMA mapping for them.
> 
> Looking at the actual code, I see that I had used kmap() only to obtain
> kernel virtual addresses for the array of struct pages obtained from
> user space by using get_user_pages.
> 
> Subsequently, I had used dma_map_single() and dma_unmap_single() calls
> for single buffer calls.

I'm suspicious about this usage, I don't know if that will actually 
work. There is a dma_map_page call which is meant for doing a DMA 
mapping on a struct page which should likely be used instead.

> 
> The code didn't have bugs IMHO since it was used for extensive stress
> testing the initial FPGA prototype as well as the final ASIC chip ,
> sometimes running for over 4 days non-stop without breaking.
> 
> However, using Test Access Points on the board and using a Logic
> Analyzer showed that DMA was NOT taking place when RAM > 896 MB was
> used. The hardware gurus said that PCI bus cycles just didn't seem to be
> taking place when RAM > 896 MB was used as the source OR destination
> address.

Are you sure the address being passed to the device was correct in this 
case? There should be nothing magical about 896MB from a hardware point 
of view, and the kernel in general cannot stop you from DMAing anywhere 
you like.

> 
> Perhaps this was a problem in the earlier kernel(s) and has since been
> rectified ? ( I was using 2.6.15 then ... )
> 
> I am just curious since Sanka Piyaratna reported a 'similar' kind of
> situation.
> 
> Regards
> 
> V. Radhakrishnan

Re: PCIe device driver question

[V.Radhakrishnan]

Thanks Robert !

Unfortunately, I cannot test this with recent kernels, since this was a
WiMAX card based on a CardBus inteface  and the Toshiba laptop I have
has only 512 MB RAM ( though I have PCs with more RAM, the PCs don't
have a CardBus interface ).

BRgds

RK


On Thu, 2008-07-31 at 12:52 -0600, Robert Hancock wrote:
> V.Radhakrishnan wrote:
> >> My guess there was a bug in your DMA mapping code. I don't think kmap is 
> >> what is normally used for this. I think with get_user_pages one usually 
> >> takes the returned page pointers to create an SG list and uses 
> >> dma_map_sg to create a DMA mapping for them.
> > 
> > Looking at the actual code, I see that I had used kmap() only to obtain
> > kernel virtual addresses for the array of struct pages obtained from
> > user space by using get_user_pages.
> > 
> > Subsequently, I had used dma_map_single() and dma_unmap_single() calls
> > for single buffer calls.
> 
> I'm suspicious about this usage, I don't know if that will actually 
> work. There is a dma_map_page call which is meant for doing a DMA 
> mapping on a struct page which should likely be used instead.
> 
> > 
> > The code didn't have bugs IMHO since it was used for extensive stress
> > testing the initial FPGA prototype as well as the final ASIC chip ,
> > sometimes running for over 4 days non-stop without breaking.
> > 
> > However, using Test Access Points on the board and using a Logic
> > Analyzer showed that DMA was NOT taking place when RAM > 896 MB was
> > used. The hardware gurus said that PCI bus cycles just didn't seem to be
> > taking place when RAM > 896 MB was used as the source OR destination
> > address.
> 
> Are you sure the address being passed to the device was correct in this 
> case? There should be nothing magical about 896MB from a hardware point 
> of view, and the kernel in general cannot stop you from DMAing anywhere 
> you like.
> 
> > 
> > Perhaps this was a problem in the earlier kernel(s) and has since been
> > rectified ? ( I was using 2.6.15 then ... )
> > 
> > I am just curious since Sanka Piyaratna reported a 'similar' kind of
> > situation.
> > 
> > Regards
> > 
> > V. Radhakrishnan
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: PCIe device driver question

[Sanka Piyaratna]

Hi Robert,

Thanks for that, I did not notice that in the spec. I am currently using 64 bit format unconditionally and this must be what is going wrong. 

Thank you so much for that.


Sanka


----- Original Message ----
From: Robert Hancock <hancockr@shaw.ca>
To: Sanka Piyaratna <cesanka@yahoo.com>
Cc: linux-kernel@vger.kernel.org
Sent: Thursday, 31 July, 2008 6:14:54 PM
Subject: Re: PCIe device driver question

Sanka Piyaratna wrote:
> I allocate memory in the user land using memalign function (typically I allocate about 500 MB) and pass this to the kernel space. In my device driver, I call get_user_pages() to lock down the memory and extract the relevant pages. A scatter-gather list is generated using these page addresses and hence derive the dma_addresses using page_to_phys() function. These addresses are programmed into a FIFO in the hardware device using a memory mapped register interface (PCI BAR based). Subsequently the hardware start filling up the pages and interrupt when a block of pages are complete. I notice the hardware hang (PCIe packets don't seem to get the acknowledgements from the root complex) when the DMA address is < 0x0000_0001_0000_0000. I have verified in the hardware that the PCIe packet is created with the correct address as programed by the device driver dma_address. If i can guard some how that the memory allocation is with in a certain area, I can stop the
>  problem from occuring. Are there any bridge functionality in the Intel architecture that may mask a certain region of memory?

How are you formatting the addresses in the TLP? The PCI Express spec 
says that for addresses below 4GB the 32-bit addressing format must be 
used (you can't use the 64-bit format unconditionally). It could be that 
is what is making the chipset unhappy.



      Find a better answer, faster with the new Yahoo!7 Search. www.yahoo7.com.au/search

Re: PCIe device driver question

[Sanka Piyaratna]

I allocate memory in the user land using memalign function (typically I allocate about 500 MB) and pass this to the kernel space. In my device driver, I call get_user_pages() to lock down the memory and extract the relevant pages. A scatter-gather list is generated using these page addresses and hence derive the dma_addresses using page_to_phys() function. These addresses are programmed into a FIFO in the hardware device using a memory mapped register interface (PCI BAR based). Subsequently the hardware start filling up the pages and interrupt when a block of pages are complete. I notice the hardware hang (PCIe packets don't seem to get the acknowledgements from the root complex) when the DMA address is < 0x0000_0001_0000_0000. I have verified in the hardware that the PCIe packet is created with the correct address as programed by the device driver dma_address. If i can guard some how that the memory allocation is with in a certain area, I can stop the
 problem from occuring. Are there any bridge functionality in the Intel architecture that may mask a certain region of memory?

Thanks and regards,

Sanka



----- Original Message ----
From: Robert Hancock <hancockr@shaw.ca>
To: Sanka Piyaratna <cesanka@yahoo.com>
Cc: linux-kernel@vger.kernel.org
Sent: Thursday, 31 July, 2008 4:54:48 AM
Subject: Re: PCIe device driver question

Sanka Piyaratna wrote:
> Hi,
> 
> I am currently developing a PCIe data capture card hardware and the
> device drivers to drive this. I have implemented DMA on the data
> capture and the scatter-gather DMA is implemented in the hardware.  I
> am testing this in an X86_64 architecture machine with 4 GB of RAM. I
> am able to successfully dma data into any memory (dma) address >
> 0x0000_0001_0000_0000. However, my problem is to dma data to any
> address less than this. When I try to DMA data to an address less than
> 0x0000_0001_0000_0000, the hardware device hangs indicating that the
> address does not exist.
> 
> I have implemented the DMA mask to be full 64 bit and my hardware is
> capable of transfering data to any address < 8TB. I am using kernel
> version 2.6.23.11. 
> 
> Could you please let me know what I might be doing wrong?

The kernel can't do anything to stop you from DMAing anywhere you want 
(barring the system having special IOMMU hardware). If you overwrite 
something you shouldn't have you'll cause a crash, but the kernel has no 
influence on it really.

Unless you're messing up the DMA addresses somehow and writing into a 
space that's not actually RAM (like the MMIO memory hole or something), 
my guess is it's likely a hardware problem..



      Find a better answer, faster with the new Yahoo!7 Search. www.yahoo7.com.au/search

Re: PCIe device driver question

[Robert Hancock]

Sanka Piyaratna wrote:
> I allocate memory in the user land using memalign function (typically I allocate about 500 MB) and pass this to the kernel space. In my device driver, I call get_user_pages() to lock down the memory and extract the relevant pages. A scatter-gather list is generated using these page addresses and hence derive the dma_addresses using page_to_phys() function. These addresses are programmed into a FIFO in the hardware device using a memory mapped register interface (PCI BAR based). Subsequently the hardware start filling up the pages and interrupt when a block of pages are complete. I notice the hardware hang (PCIe packets don't seem to get the acknowledgements from the root complex) when the DMA address is < 0x0000_0001_0000_0000. I have verified in the hardware that the PCIe packet is created with the correct address as programed by the device driver dma_address. If i can guard some how that the memory allocation is with in a certain area, I can stop the
>  problem from occuring. Are there any bridge functionality in the Intel architecture that may mask a certain region of memory?

How are you formatting the addresses in the TLP? The PCI Express spec 
says that for addresses below 4GB the 32-bit addressing format must be 
used (you can't use the 64-bit format unconditionally). It could be that 
is what is making the chipset unhappy.

Re: PCIe device driver question

[Robert Hancock]

Sanka Piyaratna wrote:
> Hi,
> 
> I am currently developing a PCIe data capture card hardware and the
> device drivers to drive this. I have implemented DMA on the data
> capture and the scatter-gather DMA is implemented in the hardware.  I
> am testing this in an X86_64 architecture machine with 4 GB of RAM. I
> am able to successfully dma data into any memory (dma) address >
> 0x0000_0001_0000_0000. However, my problem is to dma data to any
> address less than this. When I try to DMA data to an address less than
> 0x0000_0001_0000_0000, the hardware device hangs indicating that the
> address does not exist.
> 
> I have implemented the DMA mask to be full 64 bit and my hardware is
> capable of transfering data to any address < 8TB. I am using kernel
> version 2.6.23.11. 
> 
> Could you please let me know what I might be doing wrong?

The kernel can't do anything to stop you from DMAing anywhere you want 
(barring the system having special IOMMU hardware). If you overwrite 
something you shouldn't have you'll cause a crash, but the kernel has no 
influence on it really.

Unless you're messing up the DMA addresses somehow and writing into a 
space that's not actually RAM (like the MMIO memory hole or something), 
my guess is it's likely a hardware problem..

Re: PCIe device driver question

[Sanka Piyaratna]

Hi Alan,

Actually the failures occur at addresses lot higher than this range, I am noticing the failure at dma address 0x0000000037845000. This would be at around 900MB ?

Thanks

Sanka



----- Original Message ----
From: Alan Cox <alan@lxorguk.ukuu.org.uk>
To: Sanka Piyaratna <cesanka@yahoo.com>
Cc: linux-kernel@vger.kernel.org
Sent: Thursday, 31 July, 2008 1:17:18 AM
Subject: Re: PCIe device driver question

On Wed, 30 Jul 2008 09:00:58 -0700 (PDT)
Sanka Piyaratna <cesanka@yahoo.com> wrote:

> Hi,
> 
> I am currently developing a PCIe data capture card hardware and the
> device drivers to drive this. I have implemented DMA on the data
> capture and the scatter-gather DMA is implemented in the hardware.  I
> am testing this in an X86_64 architecture machine with 4 GB of RAM. I
> am able to successfully dma data into any memory (dma) address >
> 0x0000_0001_0000_0000. However, my problem is to dma data to any
> address less than this. When I try to DMA data to an address less than
> 0x0000_0001_0000_0000, the hardware device hangs indicating that the
> address does not exist.

Assuming the failures are in the range 640K-1MB then I would imagine your
bridge doesn't permit transfers via DMA to the ISA hole.



      Find a better answer, faster with the new Yahoo!7 Search. www.yahoo7.com.au/search

Re: PCIe device driver question

[V.Radhakrishnan]

> > am testing this in an X86_64 architecture machine with 4 GB of RAM. I
> > am able to successfully dma data into any memory (dma) address >
> > 0x0000_0001_0000_0000.

How can you DMA "successfully" into this address which is > 4 GB when
you have only 4 GB RAM ? Or am I missing something ?

V. Radhakrishnan
http://www.atr-labs.com

On Wed, 2008-07-30 at 09:09 -0700, Sanka Piyaratna wrote:
> Hi Alan,
> 
> Actually the failures occur at addresses lot higher than this range, I am noticing the failure at dma address 0x0000000037845000. This would be at around 900MB ?
> 
> Thanks
> 
> Sanka
> 
> 
> 
> ----- Original Message ----
> From: Alan Cox <alan@lxorguk.ukuu.org.uk>
> To: Sanka Piyaratna <cesanka@yahoo.com>
> Cc: linux-kernel@vger.kernel.org
> Sent: Thursday, 31 July, 2008 1:17:18 AM
> Subject: Re: PCIe device driver question
> 
> On Wed, 30 Jul 2008 09:00:58 -0700 (PDT)
> Sanka Piyaratna <cesanka@yahoo.com> wrote:
> 
> > Hi,
> > 
> > I am currently developing a PCIe data capture card hardware and the
> > device drivers to drive this. I have implemented DMA on the data
> > capture and the scatter-gather DMA is implemented in the hardware.  I
> > am testing this in an X86_64 architecture machine with 4 GB of RAM. I
> > am able to successfully dma data into any memory (dma) address >
> > 0x0000_0001_0000_0000. However, my problem is to dma data to any
> > address less than this. When I try to DMA data to an address less than
> > 0x0000_0001_0000_0000, the hardware device hangs indicating that the
> > address does not exist.
> 
> Assuming the failures are in the range 640K-1MB then I would imagine your
> bridge doesn't permit transfers via DMA to the ISA hole.
> 
> 
> 
>       Find a better answer, faster with the new Yahoo!7 Search. www.yahoo7.com.au/search
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: PCIe device driver question

[Alan Cox]

On Wed, 30 Jul 2008 09:09:36 -0700 (PDT)
Sanka Piyaratna <cesanka@yahoo.com> wrote:

> Hi Alan,
> 
> Actually the failures occur at addresses lot higher than this range, I am noticing the failure at dma address 0x0000000037845000. This would be at around 900MB ?

For x86 we don't do anything special and if your hardware is flagging the
memory as absent perhaps you should consult the hardware/bridge
documentation firt ?

In some cases (over 4GB and 32bit DMA masks set) we will use the GART on
x86-64 but that will still see addresses below 4GB being sent to the
device.

Alan

PCIe device driver question

[Sanka Piyaratna]

Hi,

I am currently developing a PCIe data capture card hardware and the
device drivers to drive this. I have implemented DMA on the data
capture and the scatter-gather DMA is implemented in the hardware.  I
am testing this in an X86_64 architecture machine with 4 GB of RAM. I
am able to successfully dma data into any memory (dma) address >
0x0000_0001_0000_0000. However, my problem is to dma data to any
address less than this. When I try to DMA data to an address less than
0x0000_0001_0000_0000, the hardware device hangs indicating that the
address does not exist.

I have implemented the DMA mask to be full 64 bit and my hardware is
capable of transfering data to any address < 8TB. I am using kernel
version 2.6.23.11. 

Could you please let me know what I might be doing wrong?

Thanks and regards,

Sanka


      Find a better answer, faster with the new Yahoo!7 Search. www.yahoo7.com.au/search

Re: PCIe device driver question

[Alan Cox]

On Wed, 30 Jul 2008 09:00:58 -0700 (PDT)
Sanka Piyaratna <cesanka@yahoo.com> wrote:

> Hi,
> 
> I am currently developing a PCIe data capture card hardware and the
> device drivers to drive this. I have implemented DMA on the data
> capture and the scatter-gather DMA is implemented in the hardware.  I
> am testing this in an X86_64 architecture machine with 4 GB of RAM. I
> am able to successfully dma data into any memory (dma) address >
> 0x0000_0001_0000_0000. However, my problem is to dma data to any
> address less than this. When I try to DMA data to an address less than
> 0x0000_0001_0000_0000, the hardware device hangs indicating that the
> address does not exist.

Assuming the failures are in the range 640K-1MB then I would imagine your
bridge doesn't permit transfers via DMA to the ISA hole.