Re: [PATCH]: PCI: GART iommu alignment fixes [v2]

[Prarit Bhargava <prarit@redhat.com>]

Joerg Roedel wrote:
> On Wed, Jul 23, 2008 at 07:47:03PM -0400, Prarit Bhargava wrote:
>   
>>>> Interesting. Have you experienced any problems because of that
>>>> misbehavior in the GART code? AMD IOMMU currently also violates this
>>>> requirement. I will send a patch to fix that there too.
>>>>    
>>>>         
>>>  
>>>       
>> Joerg,  yes I can see misbehavior caused by this code.  O/w I wouldn't 
>> be spending my time fixing it :) :)
>>
>> See below ....
>>
>>     
>>> IIRC, only PARISC and POWER IOMMUs follow the above rule. So I also
>>> wondered what problem he hit.
>>>  
>>>       
>> I wonder if IBM's Calgary IOMMU needs this fix? ... I've added Ed 
>> Pollard to find out.
>>
>> On big memory footprint (16G or above) systems it is possible that the 
>> e820 map reserves most of the lower 4G of memory for system use*.  So 
>> it's possible that the 4G region is almost completely reserved at boot 
>> time and so the kernel starts using the IOMMU for DMA (see 
>> dma_alloc_coherent()).  The addresses returned are not properly aligned, 
>> and this causes serious problems for some drivers that require a 
>> physical aligned address for the device.
>>     
>
> Do you have a list of driver which require this? 

No, I don't have a list. :(

But it seems that the skge driver suffers from this because this code 
exists in the driver:

        skge->mem = pci_alloc_consistent(hw->pdev, skge->mem_size, 
&skge->dma);
        if (!skge->mem)
                return -ENOMEM;

        BUG_ON(skge->dma & 7);

        if ((u64)skge->dma >> 32 != ((u64) skge->dma + skge->mem_size) 
 >> 32) {
                printk(KERN_ERR PFX "pci_alloc_consistent region crosses 
4G boundary\n");
                err = -EINVAL;
                goto free_pci_mem;
        }


If pci_alloc_consistent did the "right" thing, we should *never* see 
that warning message.

In theory, any 32-bit device attempting to request larger than PAGE_SIZE 
DMA memory on a system where no memory is available below 4G should show 
this problem.

> I would like to
> reproduce this issue. Does it also happen when you start the kernel with
> iommu=force (GART should then be used for all DMA remapping) too?
>   

Yes, this happens if you specify iommu=force on the command line.

P.