PREP memory layout and PCI transactions

PREP memory layout and PCI transactions

[Christophe Lizzi]

Hi,

I'm currently porting a third-party RAID driver from ix86 to LinuxPPC.
The target is an MTX board with 32 Mbytes of RAM, running a 2.2.10 kernel.

The driver requests the device to fill some memory locations in kernel
space via DMA. This works just fine when such memory locations come
from pages allocated by get_free_page() or kmalloc().

However, this fails when these locations come from the *static* data
of the module, or from pages allocated by vmalloc().

Specifically, the pages returned by get_free_page() or kmalloc() are located
around kernel virtual addresses C1xxxxxx (bus addresses 81xxxxxx), while the
static data in question and the pages returned by vmalloc() are located
behind virtual addresses C4xxxxxx (bus addresses 84xxxxxx).

The PCI analyser sees on the PCI bus a memory write transaction initiated
by the device to such an 84xxxxxx address in memory. We can observe that this
transaction is never completed because the DEVSEL signal on the bus is not
asserted during the correct 5 clock time interval. The transaction is then
terminated as with a master-abort termination, meaning that this address
is not known by the memory controler as an address that can be accessed
from the PCI bus.

Is this a feature, a known restriction, or a bug? Maybe a register
describing the accessible address range from the bus is not correctly
initialised into the Falcon chip?

Thanks for your attention and time,

--Christophe

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: PREP memory layout and PCI transactions

[Gabriel Paubert]

On Fri, 28 Jan 2000, Christophe Lizzi wrote:

> Hi,
>
> I'm currently porting a third-party RAID driver from ix86 to LinuxPPC.
> The target is an MTX board with 32 Mbytes of RAM, running a 2.2.10 kernel.
>
> The driver requests the device to fill some memory locations in kernel
> space via DMA. This works just fine when such memory locations come
> from pages allocated by get_free_page() or kmalloc().
>
> However, this fails when these locations come from the *static* data
> of the module, or from pages allocated by vmalloc().

This is normal, all physical memory is mapped starting from 0xc0000000
in the kernel virtual address space. [I think that copying blindly the
x86 setup is a mistake but it's not the time to discuss this ;-)]

But module code and data are allocated with vmalloc, and te fixed offset
between physical and kernel virtual addresses no more hold (obviously).
So you can only do DMA to memory acquired by kmalloc (and incidentally to
local buffers on the stack but don't forget that the size is very limited
so I recommend against unless you have a very strong argument in favor).

It is possible the the raw io patches which are in recent kernels allow
for DMA to vmalloced memory but I did not study them and I'm not sure.


> Specifically, the pages returned by get_free_page() or kmalloc() are located
> around kernel virtual addresses C1xxxxxx (bus addresses 81xxxxxx), while the
> static data in question and the pages returned by vmalloc() are located
> behind virtual addresses C4xxxxxx (bus addresses 84xxxxxx).

Bus address 84xxxxxx will not point to your physical memory and you'll end
up with some weird error on the PCI bus.

> The PCI analyser sees on the PCI bus a memory write transaction initiated
> by the device to such an 84xxxxxx address in memory. We can observe that this
> transaction is never completed because the DEVSEL signal on the bus is not
> asserted during the correct 5 clock time interval. The transaction is then
> terminated as with a master-abort termination, meaning that this address
> is not known by the memory controler as an address that can be accessed
> from the PCI bus.

Normal....

> Is this a feature, a known restriction, or a bug? Maybe a register

It is a very well known and documented restriction, which is a problem if
your device can't do scatter gather since the size of buffers you can
allocate with kmalloc is limited.

> describing the accessible address range from the bus is not correctly
> initialised into the Falcon chip?

No, oh and by the way if you want to use the Falcon/Raven/Hawk in CHRP
mode, grab my patches at:

ftp://vlab1.iram.es/linux-2.[23]

they work on my MVME26xx and 24xx, but the MTX is quite similar AFAICT.

	Gabriel.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: PREP memory layout and PCI transactions

[Michel Lanners]

Hello there,

> The driver requests the device to fill some memory locations in kernel
> space via DMA. This works just fine when such memory locations come
> from pages allocated by get_free_page() or kmalloc().
>
> However, this fails when these locations come from the *static* data
> of the module, or from pages allocated by vmalloc().

This is expected behaviour, since the way the MM code allocates memory is
different between kmalloc()'ed and vmalloc()'ed memory.

kmalloc() returns a memory object consiting of a number of consecutive pages
of RAM, adressed through the existing memory translation from (all RAM) to
(kernel virtual memory). The full physical RAM is mapped starting at 0xc0000000,
and kmalloc will return an address within that range.

However, vmalloc() returns a memory object consisting of pages that are only
consecutive in kernel virtual address space, but which is composed of individual
pages retreived through get_free_pages(). Therefore, these pages are not
necessarily consecutive in physical RAM, and the full object is mapped by
a new translation to virtual addresses _above_ physical RAM. As a result,
the simple offset calculation between kernel virtual address and physical
address doesn't work here.

> Specifically, the pages returned by get_free_page() or kmalloc() are located
> around kernel virtual addresses C1xxxxxx (bus addresses 81xxxxxx), while the
> static data in question and the pages returned by vmalloc() are located
> behind virtual addresses C4xxxxxx (bus addresses 84xxxxxx).

But bus address 0x84xxxxxx doesn't correspond to any existing RAM.

For DMA, you need to use kmalloc(), or assemble your buffer the dirty way
through successive calls to get_free_pages() and other wizardry.

Michel

____________________________________________________________________
.sig home alone

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/