Workaround hardware bug addressing physical address

Workaround hardware bug addressing physical address

[Unai Uribarri]

Hello, 

I'm writing a device driver for a Cavium Octeon 56xx network adapter, which has a silicon bug: it sends invalid PCI Express TLP headers when writing to physical addresses between 2GB and 4GB. It sends a 64 bit memory write when the PCI specs mandates a 32 bit memory write, and some PCIe bridges refuses such transactions (at least Intel 5520/X58 refuses them). 

Now, when allocating memory using alloc_page, I check the allocated memory and leak the unusable pages. This "solution" works but I'm leaking huge amounts of memory, so I'm planning to adopt a more elegant solution. I'm going to link together all the unusable pages and, after all the memory is allocated, free the unusable pages.

Do someone knows if more hardware shares this bug? Will a generic solution be useful for other hardware that doesn't accept certain zones of memory? Has anyone already developed such solution?

Thanks.

Re: Workaround hardware bug addressing physical address

[Unai Uribarri]

----- "Marin Mitov" <mitov@issp.bas.bg> wrote:

| Hi,
| 
| This is pci driver. You can set dma mask:
| 
| dma_set_coheren_mask(pdev, DMA_BIT_MASK(31)) 
| 
| All further alloc_coherent() should be from the region 0-2GB.
| 

But I'm using a 64 bit operating system with 32GB of RAM. It's a pity to be unable to use 4GB-32GB range because the 2-4GB range is unusable. So I've written this code to skip invalid areas. Do you think this code could be useful for other drivers?


/** Structure used to store all the information about a DMA allocation which is
 * needed to free it. It also has a next field to chain the allocations.
 */
typedef struct dma_allocation
{
    struct device *dev;
    size_t size;
    dma_addr_t dma_handle;
    struct list_head list;
} dma_allocation_t;

/** Validates a dma allocation
 * @param{in} ptr is the pointer to the allocated dma memory
 * @param{in} dma_handle is the handle to the allocated dma memory
 * @return 0 if the allocation isn't valid.
 */
typedef int (*dma_validator_t)(void *ptr, struct device *dev, size_t size, dma_addr_t handle);

/** Allocates validated coherent DMA memory.
 *
 * Same paremeters and return value as dma_alloc_coherent, and:
 *  @param{in} invalid_allocs is a pointer to a list where invalid allocations are enqueued
 *  @param{in} validator is the validation function
 *
 * While the chunk of memory allocated by dma_alloc_coherent doesn't pass the validator filter,
 * it's queued in invalid_allocs and another chunk of memory is allocated.
 */
static void *dma_validated_alloc_coherent(struct list_head *invalid_allocs, dma_validator_t validator,
     struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t flag)
{
    void *res = dma_alloc_coherent(dev, size, dma_handle, flag);

    while (!IS_ERR(res) && !validator(res, dev, size, *dma_handle)) {
        dma_allocation_t *entry = res;
        entry->dev = dev;
        entry->size = size;
        entry->dma_handle = *dma_handle;
        list_add_tail(&entry->list, invalid_allocs);

        res = dma_alloc_coherent(dev, size, dma_handle, flag);
    }
    return res;
}

/** Frees all the coherent dma allocations contained in list */
static void dma_list_free_coherent(struct list_head *list)
{
    dma_allocation_t *cur, *next;
    list_for_each_entry_safe(cur, next, list, list) {
        dma_free_coherent(cur->dev, cur->size, cur, cur->dma_handle);
    }
}

Re: Workaround hardware bug addressing physical address

[Marin Mitov]

On Wednesday, July 14, 2010 08:06:49 pm you wrote:
> ----- "Marin Mitov" <mitov@issp.bas.bg> wrote:
> 
> | Hi,
> | 
> | This is pci driver. You can set dma mask:
> | 
> | dma_set_coheren_mask(pdev, DMA_BIT_MASK(31)) 
> | 
> | All further alloc_coherent() should be from the region 0-2GB.
> | 
> 
> But I'm using a 64 bit operating system with 32GB of RAM. It's a pity to be unable to use 4GB-32GB range because the 2-4GB range is unusable. So I've written this code to skip invalid areas. Do you think this code could be useful for other drivers?

Let me summarize if I have correctly understood what you do.

First, your hardware has problems when the physical (bus) address
is out of the 0-2GB region, so you cannot use buffers that are out 
of this range in any case. And the defect is in the peripheral, not in 
the bridge between it and the memory.

Second, you do allocations and check if dma_addr_t is in this range.
If it is, you keep it for use. If not, you put it in a list and allocate a new one.
When you allocate sufficiently memory in 0-2GB region, you deallocate 
the memory (kept in the list) that is out of the region. Right?

The same effect you could have if you first call 

dma_set_coheren_mask(pdev, DMA_BIT_MASK(31))

after that call every alloc_coherent() for this device only 
should return memory from the region 0-2GB

Setting the coherent mask is per device (the first argument of the function)
All the other devices will use 0-32GB memory according their own dma masks.

Hope that helps.

Marin Mitov

> 
> 
> /** Structure used to store all the information about a DMA allocation which is
>  * needed to free it. It also has a next field to chain the allocations.
>  */
> typedef struct dma_allocation
> {
>     struct device *dev;
>     size_t size;
>     dma_addr_t dma_handle;
>     struct list_head list;
> } dma_allocation_t;
> 
> /** Validates a dma allocation
>  * @param{in} ptr is the pointer to the allocated dma memory
>  * @param{in} dma_handle is the handle to the allocated dma memory
>  * @return 0 if the allocation isn't valid.
>  */
> typedef int (*dma_validator_t)(void *ptr, struct device *dev, size_t size, dma_addr_t handle);
> 
> /** Allocates validated coherent DMA memory.
>  *
>  * Same paremeters and return value as dma_alloc_coherent, and:
>  *  @param{in} invalid_allocs is a pointer to a list where invalid allocations are enqueued
>  *  @param{in} validator is the validation function
>  *
>  * While the chunk of memory allocated by dma_alloc_coherent doesn't pass the validator filter,
>  * it's queued in invalid_allocs and another chunk of memory is allocated.
>  */
> static void *dma_validated_alloc_coherent(struct list_head *invalid_allocs, dma_validator_t validator,
>      struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t flag)
> {
>     void *res = dma_alloc_coherent(dev, size, dma_handle, flag);
> 
>     while (!IS_ERR(res) && !validator(res, dev, size, *dma_handle)) {
>         dma_allocation_t *entry = res;
>         entry->dev = dev;
>         entry->size = size;
>         entry->dma_handle = *dma_handle;
>         list_add_tail(&entry->list, invalid_allocs);
> 
>         res = dma_alloc_coherent(dev, size, dma_handle, flag);
>     }
>     return res;
> }
> 
> /** Frees all the coherent dma allocations contained in list */
> static void dma_list_free_coherent(struct list_head *list)
> {
>     dma_allocation_t *cur, *next;
>     list_for_each_entry_safe(cur, next, list, list) {
>         dma_free_coherent(cur->dev, cur->size, cur, cur->dma_handle);
>     }
> }
>