MAX_DMA_ADDRESS in include/asm/asm-i386/dma.h (2.6.x and 2.4.x)

MAX_DMA_ADDRESS in include/asm/asm-i386/dma.h (2.6.x and 2.4.x)

[Amit D. Chaudhary]

While writing some DMA page gathering code, I realize that 
__get_free_page or kmalloc does not return memory more than 16 MB 
(typically around 11-12 MB even if it done right after a reboot.)

Since this is for a PCI device (A Fibre channel HBA), I remembered that 
the book Linux Device Driver, edition 2 mentions that the 16 MB limit is 
for DMA with ISA devices, while PCI DMA can access upto 950 MB or so, 
using 32 bit addresses.

Digging further into the code, I found that this being a Intel PC (a P4 
2.8 GHz with 1GB RAM) with High Mem for 4 GB enabled kernel, the define 
in linux/include/asm/asm-i386/dma.h for MAX_DMA_ADDRESS is probably what 
is limiting this. The #define is the same in 2.6.8-rc1 as well as 2.4.x 
(2.4.20 and 2.4.25 is what I checked.)

/* The maximum address that we can perform a DMA transfer to on this 
platform */
#define MAX_DMA_ADDRESS      (PAGE_OFFSET+0x1000000)

 From what I can understand so far, this implies 0xC000 0000 + 
+0x1000000 = A total of 16 MB

I changed this to 256 MB which was less than the limit in the book and 
enough for the test code purpose
#define MAX_DMA_ADDRESS      (PAGE_OFFSET+0x10000000)

After running some FC traffic and delibrately not free'ing memory, the 
memory above 16 MB was DMA'abled and the data was read and written 
correctly.

So, the question is, can someone DMA and x86 architecture expert comment 
on this? Am I missing something?
And if the above code is ok, is there a way to have the above change in 
the main kernel so that it works with ISA and PCI devices?

Thanks
Amit

Re: MAX_DMA_ADDRESS in include/asm/asm-i386/dma.h (2.6.x and 2.4.x)

[Deepak Saxena]

On Jul 16 2004, at 14:37, Amit D. Chaudhary was caught saying:
> While writing some DMA page gathering code, I realize that 
> __get_free_page or kmalloc does not return memory more than 16 MB 
> (typically around 11-12 MB even if it done right after a reboot.)
> 
> Since this is for a PCI device (A Fibre channel HBA), I remembered that 
> the book Linux Device Driver, edition 2 mentions that the 16 MB limit is 
> for DMA with ISA devices, while PCI DMA can access upto 950 MB or so, 
> using 32 bit addresses.

Using __get_free_page() or kmalloc() for device DMA'ble descriptors (I am
guessing that's what you are doing) is wrong. See Documentation/DMA-API.txt
and Documentation/DMA-mapping.txt for the proper way to do this. 

~Deepak

-- 
Deepak Saxena - dsaxena at plexity dot net - http://www.plexity.net/

"Unlike me, many of you have accepted the situation of your imprisonment and
 will die here like rotten cabbages." - Number 6

Re: MAX_DMA_ADDRESS in include/asm/asm-i386/dma.h (2.6.x and 2.4.x)

[Amit D. Chaudhary]

Deepak,

I am missing what you are directing me to.

If it is,
pci_alloc_consistent(), linux-2.4.25/arch/i386/kernel/pci-dma.c
dma_alloc_coherent(), linux-2.6.8-rc1/arch/i386/kernel/pci-dma.c

They internally seem to __get_free_pages()

If you meant,
pci_pool_create()\pci_pool_alloc
the code in question does not need fixed size buffers like kmem_cache_alloc.

Here is the actual usage planned,
The DMA buffer is a variable size buffer (typically 1k to 128K) 
represent a read or write from an FC initiator. This memory is passed as 
data segments (sg list) with various CTIOs (Type of IOCB on Qlogic FC 
HBA chips). For a write command, the memory is used to read the data 
into the memory and then pass it to a user application without any copy 
(think mmap.)
This is part of implementing a virtual targets in a Target mode device 
driver.

The memory need not be page size, as a matter of fact, using a large 
consecutive block, for example using alloc_bootmem_low() during kernel 
bootup, will simplify the data transfer and result in no internal 
fragmentation, it does introduce inflexibility in changing the size and 
other issues.

Hopefully that sheds more light on the topic.

Amit


Deepak Saxena wrote:
> On Jul 16 2004, at 14:37, Amit D. Chaudhary was caught saying:
> 
>>While writing some DMA page gathering code, I realize that 
>>__get_free_page or kmalloc does not return memory more than 16 MB 
>>(typically around 11-12 MB even if it done right after a reboot.)
>>
>>Since this is for a PCI device (A Fibre channel HBA), I remembered that 
>>the book Linux Device Driver, edition 2 mentions that the 16 MB limit is 
>>for DMA with ISA devices, while PCI DMA can access upto 950 MB or so, 
>>using 32 bit addresses.
> 
> 
> Using __get_free_page() or kmalloc() for device DMA'ble descriptors (I am
> guessing that's what you are doing) is wrong. See Documentation/DMA-API.txt
> and Documentation/DMA-mapping.txt for the proper way to do this. 
> 
> ~Deepak
> 

Re: MAX_DMA_ADDRESS in include/asm/asm-i386/dma.h (2.6.x and 2.4.x)

[Deepak Saxena]

On Jul 16 2004, at 15:11, Amit D. Chaudhary was caught saying:
> Deepak,
> 
> I am missing what you are directing me to.
> 
> If it is,
> pci_alloc_consistent(), linux-2.4.25/arch/i386/kernel/pci-dma.c
> dma_alloc_coherent(), linux-2.6.8-rc1/arch/i386/kernel/pci-dma.c
> 
> They internally seem to __get_free_pages()

Correct, but take a second look at the code (2.6):

        void *ret;
        /* ignore region specifiers */
        gfp &= ~(__GFP_DMA | __GFP_HIGHMEM);

        if (dev == NULL || (dev->coherent_dma_mask < 0xffffffff))
                gfp |= GFP_DMA;

        ret = (void *)__get_free_pages(gfp, get_order(size));

It uses GFP_DMA iff your coherent_dma_mask is != 0xffffffff.  Assuming
your device can address a the full 32-bit PCI address space, you
need to set the coherent_dma_mask appropriately and you will get
buffers from all addressable lowmem. I don't do much x86, so not
sure how you go about allocating highmem DMA buffers.

> The memory need not be page size, as a matter of fact, using a large 
> consecutive block, for example using alloc_bootmem_low() during kernel 
> bootup, will simplify the data transfer and result in no internal 
> fragmentation, it does introduce inflexibility in changing the size and 
> other issues.

If you are using alloc_bootmem_low(), all you should have to do after
allocating the memory is call pci_dma_map_single()/map_sg() to get PCI-DMA 
addresses. You still should have no reason to touch MAX_DMA_ADDRESS.

~Deepak

-- 
Deepak Saxena - dsaxena at plexity dot net - http://www.plexity.net/

"Unlike me, many of you have accepted the situation of your imprisonment and
 will die here like rotten cabbages." - Number 6

Re: MAX_DMA_ADDRESS in include/asm/asm-i386/dma.h (2.6.x and 2.4.x)

[Amit D. Chaudhary]

Deepak Saxena wrote:
> On Jul 16 2004, at 15:11, Amit D. Chaudhary was caught saying:
> 
>>Deepak,
>>
>>I am missing what you are directing me to.
>>
>>If it is,
>>pci_alloc_consistent(), linux-2.4.25/arch/i386/kernel/pci-dma.c
>>dma_alloc_coherent(), linux-2.6.8-rc1/arch/i386/kernel/pci-dma.c
>>
>>They internally seem to __get_free_pages()
> 
> 
> Correct, but take a second look at the code (2.6):
> 
>         void *ret;
>         /* ignore region specifiers */
>         gfp &= ~(__GFP_DMA | __GFP_HIGHMEM);
> 
>         if (dev == NULL || (dev->coherent_dma_mask < 0xffffffff))
>                 gfp |= GFP_DMA;
> 
>         ret = (void *)__get_free_pages(gfp, get_order(size));
> 
> It uses GFP_DMA iff your coherent_dma_mask is != 0xffffffff.  Assuming
> your device can address a the full 32-bit PCI address space, you
> need to set the coherent_dma_mask appropriately and you will get
> buffers from all addressable lowmem. I don't do much x86, so not
> sure how you go about allocating highmem DMA buffers.
Thanks, noted and verified.

This chip cannot DMA with a memory buffer returned by kmalloc without a 
GFP_DMA flag, that is memory addresses like 0xf67e0000, it works with 
0xcxxx xxxx.

I verified it by modifying the code and trying it out.

>>The memory need not be page size, as a matter of fact, using a large 
>>consecutive block, for example using alloc_bootmem_low() during kernel 
>>bootup, will simplify the data transfer and result in no internal 
>>fragmentation, it does introduce inflexibility in changing the size and 
>>other issues.
> 
> 
> If you are using alloc_bootmem_low(), all you should have to do after
> allocating the memory is call pci_dma_map_single()/map_sg() to get PCI-DMA 
> addresses. You still should have no reason to touch MAX_DMA_ADDRESS.
This was a backup approach, I mentioned to provide details about the 
memory being allocated. I would like to avoid this approach. See reasons 
above.

Amit

Re: MAX_DMA_ADDRESS in include/asm/asm-i386/dma.h (2.6.x and 2.4.x)

[Amit D. Chaudhary]

Deepak,

Thanks a lot for the inputs. kmalloc with GFP_DMA is not to be used for 
pci DMA.
I continued trying it and it does work, there was some alignment for the 
header which was incorrect and that I limited the size to 128k.

Amit

Amit D. Chaudhary wrote:
> 
> 
> Deepak Saxena wrote:
> 
>> On Jul 16 2004, at 15:11, Amit D. Chaudhary was caught saying:
>>
>>> Deepak,
>>>
>>> I am missing what you are directing me to.
>>>
>>> If it is,
>>> pci_alloc_consistent(), linux-2.4.25/arch/i386/kernel/pci-dma.c
>>> dma_alloc_coherent(), linux-2.6.8-rc1/arch/i386/kernel/pci-dma.c
>>>
>>> They internally seem to __get_free_pages()
>>
>>
>>
>> Correct, but take a second look at the code (2.6):
>>
>>         void *ret;
>>         /* ignore region specifiers */
>>         gfp &= ~(__GFP_DMA | __GFP_HIGHMEM);
>>
>>         if (dev == NULL || (dev->coherent_dma_mask < 0xffffffff))
>>                 gfp |= GFP_DMA;
>>
>>         ret = (void *)__get_free_pages(gfp, get_order(size));
>>
>> It uses GFP_DMA iff your coherent_dma_mask is != 0xffffffff.  Assuming
>> your device can address a the full 32-bit PCI address space, you
>> need to set the coherent_dma_mask appropriately and you will get
>> buffers from all addressable lowmem. I don't do much x86, so not
>> sure how you go about allocating highmem DMA buffers.
> 
> Thanks, noted and verified.
> 
> This chip cannot DMA with a memory buffer returned by kmalloc without a 
> GFP_DMA flag, that is memory addresses like 0xf67e0000, it works with 
> 0xcxxx xxxx.
> 
> I verified it by modifying the code and trying it out.
> 
>>> The memory need not be page size, as a matter of fact, using a large 
>>> consecutive block, for example using alloc_bootmem_low() during 
>>> kernel bootup, will simplify the data transfer and result in no 
>>> internal fragmentation, it does introduce inflexibility in changing 
>>> the size and other issues.
>>
>>
>>
>> If you are using alloc_bootmem_low(), all you should have to do after
>> allocating the memory is call pci_dma_map_single()/map_sg() to get 
>> PCI-DMA addresses. You still should have no reason to touch 
>> MAX_DMA_ADDRESS.
> 
> This was a backup approach, I mentioned to provide details about the 
> memory being allocated. I would like to avoid this approach. See reasons 
> above.
> 
> Amit
>