MPC7455 DMA buffer strangeness

MPC7455 DMA buffer strangeness

[Oliver Korpilla]

Hello!

I'm mapping a DMA buffer allocated with pci_alloc_consistent() into user
space with a mmap hook like this:

int vme_mmap(struct file *file_ptr, struct vm_area_struct *vma)
{
        DPRINTF("Attempting to map %#lx bytes of memory at "
                "physical address %#lx\n", vma->vm_end - vma->vm_start,
                vma->vm_pgoff << PAGE_SHIFT);

        vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);

        vma->vm_flags |= VM_RESERVED;

#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,5,3) || defined RH9BRAINDAMAGE
        return remap_page_range(vma, vma->vm_start, vma->vm_pgoff <<
PAGE_SHIFT,
                                vma->vm_end - vma->vm_start,
vma->vm_page_prot);
#else
        return remap_page_range(vma->vm_start, vma->vm_pgoff <<
PAGE_SHIFT,
                                vma->vm_end - vma->vm_start,
vma->vm_page_prot);
#endif
}

Actually this produces the desired effect on a MPC8240, where the buffer
is writeable and readable from user space and kernel space (with its
original virtual address on allocation).

On the MPC7455, though, write accesses seem to be applied or not applied
in a somewhat random fashion. Sometimes an offsetted write into the buffer
is there, and sometimes not. Writing at the begin of the buffer seems to
disapper always.

What's going on here?

Thanks in advance,
Oliver Korpilla

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: MPC7455 DMA buffer strangeness

[Adrian Cox]

On Mon, 2004-07-05 at 09:31, Oliver Korpilla wrote:
> I'm mapping a DMA buffer allocated with pci_alloc_consistent() into user
> space with a mmap hook like this:
>
> int vme_mmap(struct file *file_ptr, struct vm_area_struct *vma)
> {
>         DPRINTF("Attempting to map %#lx bytes of memory at "
>                 "physical address %#lx\n", vma->vm_end - vma->vm_start,
>                 vma->vm_pgoff << PAGE_SHIFT);
>
>         vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);

The memory allocated by pci_alloc_consistent will be cacheable memory,
as PowerPC 60x is a cache coherent architecture. You've set up a
non-cacheable userspace mapping to the same address by using
pgprot_noncached(). Two mappings of the same physical memory must have
the same cache settings.

> On the MPC7455, though, write accesses seem to be applied or not applied
> in a somewhat random fashion. Sometimes an offsetted write into the buffer
> is there, and sometimes not. Writing at the begin of the buffer seems to
> disapper always.

This probably only shows on the 7455 because the caches of the 7455 are
much larger.

- Adrian Cox
Humboldt Solutions Ltd.


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: MPC7455 DMA buffer strangeness

[Oliver Korpilla]

Hello, Adrian!

I thought of the same today, but didn't submit it because I'm not sure about
some parts of that logic.

Adrian Cox wrote:
>
> The memory allocated by pci_alloc_consistent will be cacheable
> memory, as PowerPC 60x is a cache coherent architecture. You've set
> up a non-cacheable userspace mapping to the same address by using
> pgprot_noncached(). Two mappings of the same physical memory must have
> the same cache settings.
>
>>On the MPC7455, though, write accesses seem to be applied or not
>>applied in a somewhat random fashion. Sometimes an offsetted write
>>into the buffer is there, and sometimes not. Writing at the begin of
>>the buffer seems to disapper always.
>
> This probably only shows on the 7455 because the caches of the 7455
> are much larger.

Of course the 3-layer caching of the 7455 is much more likely to interfere than
that of the 603e, and may be a reason for the flakiness in reproducing the
errors I observed. I though of that, but the following reasoning led me to
dismiss it at first:

How can a cacheable kernel mapping influence my DMA transfer?

The buffer is not written to in kernel space, but in user space. An ioctl() is
made for triggering the DMA. The user space mapping will lead to direct writes
to that physical buffer address, so the DMA engine of my PCI-VME-bridge chip
should see only what I wrote in user space, regardless of the kernel mapping.
(But it doesn't in reality!)

Or doesn't it work like that? (Maybe a flush of the kernel cache of my pages
does corrupt my writes to the same physical addresses?)

Sadly enough I found no module-compatible way yet to map my pages with the
correct G- and I-bits yet in the kernel itself. Allocation functions don't ask
for the pgprot_t, change_protection() is not exported.

Trying to rewrite pci_alloc_consistent like this produced no change in
protection bits of the PTEs (just for testing):

void *pci_alloc_consistent(struct pci_dev *hwdev, size_t size,
			   dma_addr_t *dma_handle)
{
	void *ret;
	int gfp = GFP_ATOMIC;
	/* Obtain flags for guarded/non-cached */
	pgprot_t prot;
	pgprot_val(prot) = 0;
	prot = pgprot_noncached(prot);

	if (hwdev == NULL || hwdev->dma_mask != 0xffffffff)
		gfp |= GFP_DMA;

#ifdef CONFIG_NOT_COHERENT_CACHE
	ret = consistent_alloc(gfp, size, dma_handle);
#else
	ret = (void *)__get_free_pages(gfp, get_order(size));

	if (ret != NULL) {
		memset(ret, 0, size);
		*dma_handle = virt_to_bus(ret);
	}
#endif /* CONFIG_NOT_COHERENT_CACHE */

	if (ret != NULL)
	  {
	    change_protection ((unsigned long int) ret,
		(unsigned long int) (ret+size), prot);
	  }

	return ret;
}

Do you have any idea about this?

Thanks,
Oliver Korpilla

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: MPC7455 DMA buffer strangeness

[Adrian Cox]

On Tue, 2004-07-06 at 09:47, Oliver Korpilla wrote:
> The buffer is not written to in kernel space, but in user space. An ioctl() is
> made for triggering the DMA. The user space mapping will lead to direct writes
> to that physical buffer address, so the DMA engine of my PCI-VME-bridge chip
> should see only what I wrote in user space, regardless of the kernel mapping.
> (But it doesn't in reality!)

It's certainly possible that part of the page has ended up in cache.

Why are you trying to turn off cache coherency? This should all work if
your userspace mapping is cache coherent.

- Adrian Cox
Humboldt Solutions Ltd.


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: MPC7455 DMA buffer strangeness

[Oliver Korpilla]

Hello, Adrian!

Perhaps it is easier, if I explain what I want to do, why and perhaps to find  a
solution to that:

I have two kinds to access data on the VME bus via the Tundra Universe 2 PCI-VME
bridge:
1) Master Windows:
I want to access VME bus locations via memory pointer. The memory pointer is
dereferenced, translated by the MMU to a physical address, a cycle is generated
on the PCI (the physical address is from a I/O resource), the cycle is answered
by the Universe device (which is programmed to answer for the range of PCI
addresses defining the Master Window with a range of VME bus addresses). This
does only work since the pgprot_noncached() stuff is in my mmap() hook - and it
does work correctly.
2.) DMA:
A DMA buffer gets allocated with pci_alloc_consistent(), mapped to user space
via the mmap() hook, written to (only the write goes wrong on the 7455) and the
actual transfer (involving writing to the bridge registers) is triggered by an
ioctl().

 > It's certainly possible that part of the page has ended up in cache.
 >
 > Why are you trying to turn off cache coherency? This should all work if
 > your userspace mapping is cache coherent.

So - after your explanations - it seems like I do have a conflict of interest
there: using pgprot_noncached() kills coherency in my mmap() hook, so this maybe
a reason for DMA failing. pgprot_noncached is needed for Master Windows, so I
need it there.

First I will try out removing the pgprot_noncached() and try to test DMA again.
I'll let you know the results. If DMA works fine then, I need to devise a way to
differentiate between Master Window and DMA mmap() calls and treat them different.

If not, I'm really stumped... :(

Thanks,
Oliver Korpilla

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: MPC7455 DMA buffer strangeness

[Oliver Korpilla]

Hello, Adrian!

When removing the line

vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);

from my mmap() hook, DMA works as expected.

So my problem just transformed to "Differentiate mapping for Master Windows
(uncached) from mapping for DMA (cache coherent)".

Many a lot of thanks,
Oliver Korpilla

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/