Accessing memory remote across the bus in the same way as local memory

Accessing memory remote across the bus in the same way as local memory

[James Bottomley]

The question has come up [in the DMA API issues thread on lkml] of
whether we can access on chip memory areas as though they were normal
memory (i.e. use normal reads and writes to access it).

The memory regions in question would have to be accessible via MMIO
cycles on the bus and be aligned and sized as multiples of PAGE_SIZE.

The question is given the constraints, can we pull page table tricks on
every platform to make this bus remote memory look like real memory?

I know ioremap() read[bwl]/write[bwl] etc. is what we currently tell
people to do, but I'm curious to know if there's actually a platform
that has some type of limitation that requires bus remote memory to be
treated differently.  Rather than simply being a convenience, like we
use it as on parisc today.

James

Re: Accessing memory remote across the bus in the same way as local memory

[David S. Miller]

On 18 Jun 2004 18:26:37 -0500
James Bottomley <James.Bottomley@SteelEye.com> wrote:

> The question is given the constraints, can we pull page table tricks on
> every platform to make this bus remote memory look like real memory?

There are three issues:

1) How to form the physical address (for a PCI device we have PCI device
   pointer and resource, stuff like that, we just need a translator
   interface or similar to pass that info in)

2) Whether such accesses are legal.  On Sparc64 one device cannot DMA
   to/from the MMIO of a device behind a _DIFFERENT_ PCI controller.

3) Some logic to get page table non-cacheable bits set etc. since this
   is to non-memory space.

Re: Accessing memory remote across the bus in the same way as local memory

[James Bottomley]

On Fri, 2004-06-18 at 18:32, David S. Miller wrote:
> There are three issues:

Oh, yes, I know this isn't easy, it would be an absolute pig on parisc
too because we use absloute instruction accesses currently to MMIO
space...if we start remapping pieces we end up with cache aliasing
issues.

But what I'm more concerned about is is there a basic show stopper that
would prevent a platform from doing this?

> 1) How to form the physical address (for a PCI device we have PCI device
>    pointer and resource, stuff like that, we just need a translator
>    interface or similar to pass that info in)

That's really something the DMA API does well today.  I think as part of
the implementation, we'd just have the device register the area and its
own internal address for it.

> 2) Whether such accesses are legal.  On Sparc64 one device cannot DMA
>    to/from the MMIO of a device behind a _DIFFERENT_ PCI controller.

Yes, this one could be tricky: no using the memory you obtained for DMA
to or from anything other than this one device.

> 3) Some logic to get page table non-cacheable bits set etc. since this
>    is to non-memory space.

This would be up to the platform piece of the API to sort out, I think.

I'm not anywhere close to proposing this, I'm just exploring the
possibility at the moment.  I'm still arguing "no" in the DMA API
thread...

James

Re: Accessing memory remote across the bus in the same way as local memory

[Paul Mackerras]

James Bottomley writes:

> The question has come up [in the DMA API issues thread on lkml] of
> whether we can access on chip memory areas as though they were normal
> memory (i.e. use normal reads and writes to access it).
> 
> The memory regions in question would have to be accessible via MMIO
> cycles on the bus and be aligned and sized as multiples of PAGE_SIZE.
> 
> The question is given the constraints, can we pull page table tricks on
> every platform to make this bus remote memory look like real memory?

Not on iSeries machines (ppc64 machines running OS/400).  There we
have to do a hypervisor call for every access to PCI memory or I/O
space, unfortunately.  Fortunately people don't tend to put arbitrary
PCI adaptors in iSeries boxes, and they don't run X servers.  Linux
partitions on iSeries machines are mostly used for things like
firewalls and web servers.

What did you want to be able to do this for?  It probably isn't the
end of the world if whatever you want to do with this doesn't work on
iSeries.

Paul.

Re: Accessing memory remote across the bus in the same way as local memory

[James Bottomley]

On Fri, 2004-06-18 at 23:05, Paul Mackerras wrote:
> Not on iSeries machines (ppc64 machines running OS/400).  There we
> have to do a hypervisor call for every access to PCI memory or I/O
> space, unfortunately.  Fortunately people don't tend to put arbitrary
> PCI adaptors in iSeries boxes, and they don't run X servers.  Linux
> partitions on iSeries machines are mostly used for things like
> firewalls and web servers.

Heh, I guessed there'd be one architecture that couldn't do this.

> What did you want to be able to do this for?  It probably isn't the
> end of the world if whatever you want to do with this doesn't work on
> iSeries.

The basic issue is can we declare memory that resides on a device chip
(or on a memory controller behind a PCI bridge) to the system so that it
can be used almost identically to real memory (just with a bunch of
restrictions).

The request is coming from an ARM developer who has an OHCI chip with 32
on chip sram.  He wants dma_alloc_coherent() to hand out the on-chip
memory so that he doesn't have to modify the generic OHCI driver to be
able to use a separate memory area.

However, most modern devices seem to come with on-board memory for
descriptors, ring buffers and the like (mainly for efficiency since the
device can access them without having to do a bus to main memory
transaction), so I was wondering if allocating from this on-board pool
could be folded into the DMA API so that dma_alloc_coherent() could
manage it for every such device.

James