Re: DMA API issues

Re: DMA API issues

[James Bottomley]

    
    
    My colleagues and I are encountering a number of difficulties with the
    DMA API, to which a generic solution is required (or risk multiple
    architectures, busses, and devices going their own way...)
    
    Here is an example system that illustrates these problems:
    
    I have a System On Chip device which, among other functions, contains an
    OHCI controller and 32K of SRAM.
    
    heres the catch:- The OHCI controller has a different address space than
    the host bus, and worse, can *only* DMA data from its internal SRAM.
    
    The architecture is not broken, merely unusual.
    
    This causes the following problems:
    
    1) The DMA API provides no methods to set up a mapping between the host
       memory map and the devices view of the space
            example:
               the OHCI controller above would see its 32K of SRAM as
               mapped from 0x10000 - 0x1ffff and not 0xXXX10000 - 0xXXX1ffff
               which is the address the CPU sees.

Erm, well this isn't unusual.  A lot of devices have on board memory to
offload accesses to.  All the later Symbios SCSI chips for instance.  If
you look at the drivers, you'll see they ioremap the region and then use
it via the normal memory accessors.

    2) The DMA API assumes the device can access SDRAM
            example:
               the OHCI controller base is mapped at 0x10000000 on my platform.
               this is NOT is SDRAM, its in IO space.

The usual thing for devices to do is maintain their own mapping of the
the regions, since the device physical locations has no meaning at all
to the platform systems (other than it better not clash with real
memory).
    
    If these points are possible to be addressed, it would allow at LEAST three chips *in \
    use* in linux devices able to use mainline OHCI code directly - TC6393XB (in toshiba \
    PDAs), SAMCOP (Ipaqs), and mediaQ (dell axims).
    
    I am told HPPA has some similar problems also.
    
I don't understand what you're asking for beyond what we currently do. 
These devices should be able to operate using ioremap and memcpy_toio,
what more do you want?

There was one extension to the DMA API that I considered for a Q720 SCSI
card which has 2MB of onboard memory.  That was to allow the device to
declare the memory region to the platform so the platform could hand it
out as coherent memory (this is platform dependent, some platforms
simply cannot allow direct memory access to bus space like this). 
However, something with a memory space as tiny as 32kB is unlikely to
benefit from this.

James

Re: DMA API issues

[Ian Molton]

On 18 Jun 2004 13:20:43 -0500
James Bottomley <James.Bottomley@SteelEye.com> wrote:

> 
> Erm, well this isn't unusual.  A lot of devices have on board memory
> to offload accesses to.  All the later Symbios SCSI chips for
> instance.  If you look at the drivers, you'll see they ioremap the
> region and then use it via the normal memory accessors.

Thats all well and good for devices which have their own drivers, but
thats not the case always.

the device I described is an OHCI controller, and in theory, it should
be able to use the OHCI driver in the kernel without any modification,
*as long as* the DMA API returns valid device and virtual addresses,
which, at present, it does not.

Re: DMA API issues

[James Bottomley]

On Fri, 2004-06-18 at 13:35, Ian Molton wrote:
> Thats all well and good for devices which have their own drivers, but
> thats not the case always.
> 
> the device I described is an OHCI controller, and in theory, it should
> be able to use the OHCI driver in the kernel without any modification,
> *as long as* the DMA API returns valid device and virtual addresses,
> which, at present, it does not.

Yes, this sounds similar to the Q720 problem.  I wanted to use the
generic ncr53c8xx driver (being lazy) but I wanted to persuade the
driver to use my onboard memory.  This sounds like your issue because
the ncr driver has been sliced apart to become simply a chip driver and
I supply a small skeleton NCR_Q720.c to glue it on to the bus.

You still haven't explained what you want to do though.  Apart from the
occasional brush with usbstorage, I don't have a good knowledge of the
layout of the USB drivers.  I assume you simply want to persuade the
ohci driver to use your memory area somehow, but what do you actually
want the ohci driver to do with it?  And how much leeway do you get to
customise the driver.

The reason I'm asking is beause it's still unclear whether this is a DMA
API issue or an ohci one.  I could solve my Q720 issue simply by
exporting an interface from the ncr driver to supply alternative memory
allocation use and descriptors.

James

Re: DMA API issues

[Ian Molton]

On 18 Jun 2004 13:52:46 -0500
James Bottomley <James.Bottomley@SteelEye.com> wrote:

> 
> You still haven't explained what you want to do though.  Apart from the
> occasional brush with usbstorage, I don't have a good knowledge of the
> layout of the USB drivers.  I assume you simply want to persuade the
> ohci driver to use your memory area somehow, but what do you actually
> want the ohci driver to do with it?  And how much leeway do you get to
> customise the driver.


In *theory* the OHCI driver is doing everything right - its asking for DMAable memory and using it. if the DMA api simply understood the device in question, and alocated accordingly, it would just work.

there are two solutions:

1) Break up the OHCI driver and make it into a chip driver as you describe
2) Make the DMA API do the right thing with these devices

1) means everyone gets to write their own allocator - not pretty
2) means we get to share code and it all just works.

Re: DMA API issues

[David Brownell]

Ian Molton wrote:
> On 18 Jun 2004 13:52:46 -0500
> James Bottomley <James.Bottomley@SteelEye.com> wrote:
> 
> 
>>You still haven't explained what you want to do though.  Apart from the
>>occasional brush with usbstorage, I don't have a good knowledge of the
>>layout of the USB drivers.  I assume you simply want to persuade the
>>ohci driver to use your memory area somehow, but what do you actually
>>want the ohci driver to do with it?  And how much leeway do you get to
>>customise the driver.

A fair amount, so long as drivers above it don't need to
change much at all (this is "stable").  That includes
usbcore and all the usb drivers.

> 
> In *theory* the OHCI driver is doing everything right - its asking for DMAable memory and using it. if the DMA api simply understood the device in question, and alocated accordingly, it would just work.
> 
> there are two solutions:
> 
> 1) Break up the OHCI driver and make it into a chip driver as you describe

It's heading that way already ... breaking along the lines
where standard APIs solve problems, such as this.

> 2) Make the DMA API do the right thing with these devices
> 
> 1) means everyone gets to write their own allocator - not pretty
> 2) means we get to share code and it all just works.
> 

For example, if usbaudio uses usb_buffer_alloc to stream data,
that eliminates dma bouncing.  That's dma_alloc_coherent at
its core ... it should allocate from that 32K region.

- Dave

Re: DMA API issues

[Ian Molton]

On Fri, 18 Jun 2004 12:20:24 -0700
David Brownell <david-b@pacbell.net> wrote:

> For example, if usbaudio uses usb_buffer_alloc to stream data,
> that eliminates dma bouncing.  That's dma_alloc_coherent at
> its core ... it should allocate from that 32K region.

Agreed.

Re: DMA API issues

[James Bottomley]

On Fri, 2004-06-18 at 14:44, Ian Molton wrote:
> > For example, if usbaudio uses usb_buffer_alloc to stream data,
> > that eliminates dma bouncing.  That's dma_alloc_coherent at
> > its core ... it should allocate from that 32K region.
> 
> Agreed.

There are complications to this: not all platforms can access PCI memory
directly.  That's why ioremap and memcpy_toio and friends exist.  What
should happen on these platforms?

James

Re: DMA API issues

[David Brownell]

James Bottomley wrote:
> On Fri, 2004-06-18 at 14:44, Ian Molton wrote:
> 
>>>For example, if usbaudio uses usb_buffer_alloc to stream data,
>>>that eliminates dma bouncing.  That's dma_alloc_coherent at
>>>its core ... it should allocate from that 32K region.
>>
>>Agreed.
> 
> 
> There are complications to this: not all platforms can access PCI memory
> directly.  That's why ioremap and memcpy_toio and friends exist.  What
> should happen on these platforms?

I'm not following you.  This isn't using the PCI DMA calls.
These dots don't connect:  different hardware needs different
solutions.  How would those calls make dma_alloc_coherent work?

- Dave

Re: DMA API issues

[James Bottomley]

On Fri, 2004-06-18 at 16:08, David Brownell wrote:
> I'm not following you.  This isn't using the PCI DMA calls.
> These dots don't connect:  different hardware needs different
> solutions.  How would those calls make dma_alloc_coherent work?


The statement was "That's dma_alloc_coherent at its core ... it should
allocate from that 32K region." and what I was pointing out is that not
all platforms can treat an on-chip memory region as a real memory area. 
That's why we have the iomem accessor functions.

James

Re: DMA API issues

[David Brownell]

James Bottomley wrote:
> On Fri, 2004-06-18 at 16:08, David Brownell wrote:
> 
>>I'm not following you.  This isn't using the PCI DMA calls.
>>These dots don't connect:  different hardware needs different
>>solutions.  How would those calls make dma_alloc_coherent work?
> 
> 
> 
> The statement was "That's dma_alloc_coherent at its core ... it should
> allocate from that 32K region." and what I was pointing out is that not
> all platforms can treat an on-chip memory region as a real memory area. 

But this one can, and it sure seems like the appropriate
solution.  For reasons like the one not quoted above:  it's
a good way to eliminate what would otherwise be a case
where a dmabounce is needed.  And hey wow, it even uses
the API designed to reduce such DMA "mapping" costs, and
there are drivers already using it for such purposes.


> That's why we have the iomem accessor functions.

You mentioned ioremap(), which doesn't help here since
the need is for a block of memory, not just address space,
and also memcpy_toio(), which just another tool to implement
the dma bouncing (which is on the "strongly avoid!" list).

As I said, those still don't make dma_alloc_coherent() work.

- Dave


> James
> 
> 
> 

Re: DMA API issues

[James Bottomley]

On Fri, 2004-06-18 at 17:38, David Brownell wrote:
> James Bottomley wrote:
> > The statement was "That's dma_alloc_coherent at its core ... it should
> > allocate from that 32K region." and what I was pointing out is that not
> > all platforms can treat an on-chip memory region as a real memory area. 
> 
> But this one can, and it sure seems like the appropriate
> solution.  For reasons like the one not quoted above:  it's
> a good way to eliminate what would otherwise be a case
> where a dmabounce is needed.  And hey wow, it even uses
> the API designed to reduce such DMA "mapping" costs, and
> there are drivers already using it for such purposes.

Well, yes, but the problem: chips have onboard memory is generic.  The
proposed solution in the DMA API can't only work on certain platforms.

> 
> > That's why we have the iomem accessor functions.
> 
> You mentioned ioremap(), which doesn't help here since
> the need is for a block of memory, not just address space,
> and also memcpy_toio(), which just another tool to implement
> the dma bouncing (which is on the "strongly avoid!" list).
> 
> As I said, those still don't make dma_alloc_coherent() work.

Right, that's rather the point.  The memory you get by doing an ioremap
on this chip area may have to be treated differently from real memory on
some platforms.

That's the fundamental problem of trying to treat it as memory obtained
from dma_alloc_coherent().

James

Re: DMA API issues

[Ian Molton]

On 18 Jun 2004 18:07:30 -0500
James Bottomley <James.Bottomley@SteelEye.com> wrote:

> 
> Well, yes, but the problem: chips have onboard memory is generic.  The
> proposed solution in the DMA API can't only work on certain platforms.

It will work on any platform where the memory is directly visible to the
CPU...

if this sint the case its not *D* MA (ie. *direct* memory access is it?

Re: DMA API issues

[David Brownell]

James Bottomley wrote:
> On Fri, 2004-06-18 at 17:38, David Brownell wrote:

>>You mentioned ioremap(), which doesn't help here since
>>the need is for a block of memory, not just address space,
>>and also memcpy_toio(), which just another tool to implement
>>the dma bouncing (which is on the "strongly avoid!" list).
>>
>>As I said, those still don't make dma_alloc_coherent() work.
> 
> 
> Right, that's rather the point.  The memory you get by doing an ioremap
> on this chip area may have to be treated differently from real memory on
> some platforms.

And that point/difference would be ... what?  It IS real memory.
Not "main memory", so the device's DMA access never consumes
bandwidth on the "main" memory bus, but real nonetheless.

I'm having to guess at your point here, even from other emails.
You've asserted a difference, but not what it is.  Maybe it's
something to do with the problem's NUMA nature?  Are you for
some reason applying DMA _mapping_ requirements (main-memory
only) to the DMA memory _allocation_ problem?


> That's the fundamental problem of trying to treat it as memory obtained
> from dma_alloc_coherent().

Well, no other memory in the entire system meets the requirements
for the dma_alloc_coherent() API, since _only that_ chunk of memory
is works with that device's DMA hardware.  Which is the fundamental
problem that needs to be solved.  It can clearly done at the platform
level, using device- or bus-specific implementations.

- Dave

Re: DMA API issues

[Russell King]

On Sat, Jun 19, 2004 at 11:23:23AM -0700, David Brownell wrote:
> I'm having to guess at your point here, even from other emails.
> You've asserted a difference, but not what it is.  Maybe it's
> something to do with the problem's NUMA nature?  Are you for
> some reason applying DMA _mapping_ requirements (main-memory
> only) to the DMA memory _allocation_ problem?

I suspect the problem is concerning how Linux backs the memory and
what is assumed to be backing memory returned from the DMA coherent
API.

Currently, there are drivers which assume that it's possible that
dma_alloc_coherent memory is backed by system memory, which has
page structures associated with each page.  For this "new" memory,
there are no such page structures, so things like bus_to_virt()
don't work on them (not that they were guaranteed to work on a
dma_addr_t _anyway_ but that doesn't stop driver authors thinking
that they do work - and as code presently stands, they do indeed
work.)

In addition, the ARM implementation of dma_alloc_coherent()
implicitly believes in struct page pointers - they're a fundamental
properly of the way that has been implemented, so any deviation from
"memory with struct page" means more or less a rewrite this.

I would say that, yes, from a perfectly objective view, if you are
unable to do coherent DMA from system memory, but your system provides
you with a totally separate memory system which does indeed provide
coherent DMA, it seems logical to allow dma_alloc_coherent() to use
it - at risk of breaking some drivers making incorrect assumptions.

And I don't see _that_ case as being vastly different from Ian's
case.

So, I think as long as we can ensure that drivers do not make bad
assumptions about dma_alloc_coherent() _and_ we have a suitable DMA
MMAP API to solve the cases where device drivers want to mmap DMA
buffers (and they _do_ want to do this) it should be possible.

Depending on how I look at the problem, I'm oscillating between "yes
it should be done" (if its an overall system thing like DMA memory
on a PCI north bridge separate from your normal system non-DMA
memory) and "no it's out of the question."

> Well, no other memory in the entire system meets the requirements
> for the dma_alloc_coherent() API, since _only that_ chunk of memory
> is works with that device's DMA hardware.  Which is the fundamental
> problem that needs to be solved.  It can clearly done at the platform
> level, using device- or bus-specific implementations.

The counter-argument here is to consider the case of video cards.
They do almost continuous DMA from on-board RAM, yet the DMA API
doesn't get involved...

So I'm afraid I'm sitting on the fence between the two sides not
really knowing which side to fall off to.  It sounds to me like
other people here are in a similar situation.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core

Re: DMA API issues

[James Bottomley]

On Sat, 2004-06-19 at 15:41, Russell King wrote:
> I suspect the problem is concerning how Linux backs the memory and
> what is assumed to be backing memory returned from the DMA coherent
> API.

More or less, yes.  The basic problem is platforms that simply cannot
make this type of bus remote memory visible in the CPU page tables at
all (the IBM AS/400 apparently falls into that).  Then there are the
ones that could be persuaded to do this with great difficulty and a lot
of restrictions (sparc and parisc).

> Currently, there are drivers which assume that it's possible that
> dma_alloc_coherent memory is backed by system memory, which has
> page structures associated with each page.  For this "new" memory,
> there are no such page structures, so things like bus_to_virt()
> don't work on them (not that they were guaranteed to work on a
> dma_addr_t _anyway_ but that doesn't stop driver authors thinking
> that they do work - and as code presently stands, they do indeed
> work.)
> 
> In addition, the ARM implementation of dma_alloc_coherent()
> implicitly believes in struct page pointers - they're a fundamental
> properly of the way that has been implemented, so any deviation from
> "memory with struct page" means more or less a rewrite this.

Yes, I think everyone has more or less a struct page backed allocator,
although we could first consult an exception table hanging off the
device somehow to get around this.

> I would say that, yes, from a perfectly objective view, if you are
> unable to do coherent DMA from system memory, but your system provides
> you with a totally separate memory system which does indeed provide
> coherent DMA, it seems logical to allow dma_alloc_coherent() to use
> it - at risk of breaking some drivers making incorrect assumptions.
> 
> And I don't see _that_ case as being vastly different from Ian's
> case.
> 
> So, I think as long as we can ensure that drivers do not make bad
> assumptions about dma_alloc_coherent() _and_ we have a suitable DMA
> MMAP API to solve the cases where device drivers want to mmap DMA
> buffers (and they _do_ want to do this) it should be possible.

But we still need some sort of fallback where the platform really cannot
do this.  And that fallback is going to be ioremap and all the other
paraphenalia.  So, the thing that bothers me is that if we have to have
the fallback which is identical to what every other driver that uses
on-chip memory does anyway, is there any point to placing this in the
DMA API?

James

Re: DMA API issues

[Ian Molton]

On 19 Jun 2004 16:46:42 -0500
James Bottomley <James.Bottomley@SteelEye.com> wrote:

> 
> But we still need some sort of fallback where the platform really
> cannot do this.  And that fallback is going to be ioremap and all the
> other paraphenalia.  So, the thing that bothers me is that if we have
> to have the fallback which is identical to what every other driver
> that uses on-chip memory does anyway, is there any point to placing
> this in the DMA API?

Can you describe a system where its impossible to use the DMA API or one
of the modifications proposed here? what sort of hardware does this and
why?

Re: DMA API issues

[James Bottomley]

On Sat, 2004-06-19 at 17:49, Ian Molton wrote:
> James Bottomley <James.Bottomley@SteelEye.com> wrote:
> > But we still need some sort of fallback where the platform really
> > cannot do this.  And that fallback is going to be ioremap and all the
> > other paraphenalia.  So, the thing that bothers me is that if we have
> > to have the fallback which is identical to what every other driver
> > that uses on-chip memory does anyway, is there any point to placing
> > this in the DMA API?
> 
> Can you describe a system where its impossible to use the DMA API or one
> of the modifications proposed here? what sort of hardware does this and
> why?

There's no architecture currently that can't use the DMA API.

The modification you propose, to make on chip memory visible as normal
memory can't be done on the IBM iserie, AS/400 as I said in the the
email you quote:

On Sat, 2004-06-19 at 16:46, James Bottomley wrote:
> More or less, yes.  The basic problem is platforms that simply cannot
> make this type of bus remote memory visible in the CPU page tables at
> all (the IBM AS/400 apparently falls into that).  Then there are the
> ones that could be persuaded to do this with great difficulty and a lot
> of restrictions (sparc and parisc).

The iseries can't because the PCI bus sits behind the hypervisor and has
to use accessors to get at the on chip memory, it can't simply be mapped
into the address space like it can on ARM.

James

Re: DMA API issues

[Ian Molton]

On 20 Jun 2004 08:37:58 -0500
James Bottomley <James.Bottomley@SteelEye.com> wrote:

> 
> There's no architecture currently that can't use the DMA API.
> 
> The modification you propose, to make on chip memory visible as normal
> memory can't be done on the IBM iserie, AS/400 as I said in the the
> email you quote:

Those two statements are contradictory. clearly the iseries cant use the
DMA API *now* so I dont see how that makes any difference. We're talking
about adding propper support for *addresssable* memory mapped devices
with limited size DMA-able windows to the DMA API, not adding support
for a whole new weird way of talking to devices. These devices work the
same way as all the other devices that use the DMA API but are simply
restricted in the range of addresses they can DMA from. they require no
special 'accessors'.

iseries cant work the usual way now and wont with these modifications -
so nothing is made worse.

Re: DMA API issues

[Jeff Garzik]

On Sun, Jun 20, 2004 at 04:50:42PM +0100, Ian Molton wrote:
> Those two statements are contradictory. clearly the iseries cant use the
> DMA API *now* so I dont see how that makes any difference. We're talking
> about adding propper support for *addresssable* memory mapped devices
> with limited size DMA-able windows to the DMA API, not adding support
> for a whole new weird way of talking to devices. These devices work the
> same way as all the other devices that use the DMA API but are simply
> restricted in the range of addresses they can DMA from. they require no
> special 'accessors'.
> 
> iseries cant work the usual way now and wont with these modifications -
> so nothing is made worse.

Are you purposefully ignoring James?

He is saying the DMA API must be uniform across all platforms.  Your
proposal 

1) breaks this

2) is unneeded, as many other drivers in this same situation simply use
ioremap

Re: DMA API issues

[Ian Molton]

On Sun, 20 Jun 2004 12:26:11 -0400
Jeff Garzik <jgarzik@pobox.com> wrote:

> 
> Are you purposefully ignoring James?

No, Im failing to see his point which appears to be that if we modify
the DMA API it has to work on all platforms, even if it doesnt work on
them currently.
 
> He is saying the DMA API must be uniform across all platforms.  Your
> proposal 
> 
> 1) breaks this

How? I havent proposed any alteration to the API, just the allocation
system inside it. No drivers would need modification and any platform
not using these allocators would continue to not use them just as it
currently does(nt).

> 2) is unneeded, as many other drivers in this same situation simply
> use ioremap

Except that the OHCI driver (for example) doesnt. it uses the DMA API
(quite rightly) and to use ioremap would mean creating two ohci drivers,
one using the DMA API and one not, which is wasteful and asking for
trouble.

Re: DMA API issues

[David Brownell]

Jeff Garzik wrote:

> 2) is unneeded, as many other drivers in this same situation simply use
> ioremap

But ioremap() only solves the problem of allocating virtual address
space to match this memory.  It doesn't solve the problem of then
allocating/freeing such memory through dma_alloc_coherent(), so that
the drivers that need to share access to that memory can do so.

Re: DMA API issues

[James Bottomley]

On Sun, 2004-06-20 at 10:50, Ian Molton wrote:
> Those two statements are contradictory. clearly the iseries cant use the
> DMA API *now* so I dont see how that makes any difference. We're talking
> about adding propper support for *addresssable* memory mapped devices
> with limited size DMA-able windows to the DMA API, not adding support
> for a whole new weird way of talking to devices. These devices work the
> same way as all the other devices that use the DMA API but are simply
> restricted in the range of addresses they can DMA from. they require no
> special 'accessors'.
> 
> iseries cant work the usual way now and wont with these modifications -
> so nothing is made worse.

OK, let's try and make this as simple as I know how.  The system looks
like this


       +-----+
       | CPU |
       +--+--+
          |
    <-----+-----+----------------------+-----> Central Bus
                |                      |
          +-----+------+        +------+-----+
          |   Memory   |        |    I/O     |
          | Controller |        | Controller |
          +-----+------+        +------+-----+
                |                      |
            +---+-----+             +--+---+    +--------+
            | Memory  |             | OHCI |----| Memory |
            +---------+             +------+    +--------+

In order to access this OHCI memory, both the I/O controller and the
OHCI have to respond to the memory access cycles, rather than the memory
controller.  This is why such memory is termed "bus remote".

Even though ARM can programm the I/O controller and the OHCI device to
access this memory as though it were behind the memory controller (i.e.
using normal CPU memory cycles), you'll find that even on ARM there's
probably special page table trickery involved (probably to do with cache
coherency issues).  Next, you'll find that no other device can see this
memory without some type of i2o support, so it can't be the target of a
DMA transaction. So even on ARM, you can't treat it as "normal" memory.

On iSeries, the I/O controller sits behind the hypervisor and can't be
the target of normal memory cycles, that's why the CPU can't address
this bus remote memory normally. As I've explained.

The DMA API is about allowing devices to transact directly with memory
behind the memory controller, it's an API that essentially allows the
I/O controller and memory controller to communicate without CPU
intervention.  This is still possible through the hypervisor, so the
iSeries currently fully implements the DMA API.

James

Re: DMA API issues

[Oliver Neukum]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


>        +-----+
>        | CPU |
>        +--+--+
>           |
>     <-----+-----+----------------------+-----> Central Bus
>                 |                      |
>           +-----+------+        +------+-----+
>           |   Memory   |        |    I/O     |
>           | Controller |        | Controller |
>           +-----+------+        +------+-----+
>                 |                      |
>             +---+-----+             +--+---+    +--------+
>             | Memory  |             | OHCI |----| Memory |
>             +---------+             +------+    +--------+
> 
> In order to access this OHCI memory, both the I/O controller and the
> OHCI have to respond to the memory access cycles, rather than the memory
> controller.  This is why such memory is termed "bus remote".

Why in an abstract API would we care how the memory is connected to
the system? Isn't the only relevant issue whether the memory is in the
CPU's normal address space?
 
> Even though ARM can programm the I/O controller and the OHCI device to
> access this memory as though it were behind the memory controller (i.e.
> using normal CPU memory cycles), you'll find that even on ARM there's
> probably special page table trickery involved (probably to do with cache
> coherency issues).  Next, you'll find that no other device can see this
> memory without some type of i2o support, so it can't be the target of a
> DMA transaction. So even on ARM, you can't treat it as "normal" memory.

How is it relevant whether memory can be DMAed to by devices other
than the device in question? Even on regular x86 there's the low 16MB,
isn't there?
 
> The DMA API is about allowing devices to transact directly with memory
> behind the memory controller, it's an API that essentially allows the
> I/O controller and memory controller to communicate without CPU
> intervention.  This is still possible through the hypervisor, so the
> iSeries currently fully implements the DMA API.

Then what's the problem?

	Regards
		Oliver
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFA1dFEbuJ1a+1Sn8oRAiFKAKCJZ9mDNzeDRie7xyLAFD+nv1gpSwCfWQke
TISWtAZS6nw8J4dabkk+8lo=
=yTfH
-----END PGP SIGNATURE-----

Re: DMA API issues

[James Bottomley]

On Sun, 2004-06-20 at 13:02, Oliver Neukum wrote:
> > The DMA API is about allowing devices to transact directly with memory
> > behind the memory controller, it's an API that essentially allows the
> > I/O controller and memory controller to communicate without CPU
> > intervention.  This is still possible through the hypervisor, so the
> > iSeries currently fully implements the DMA API.
> 
> Then what's the problem?

If you look at the diagram, you'll see that the OHCI memory isn't behind
the memory controller...

James

Re: DMA API issues

[Oliver Neukum]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Am Sonntag, 20. Juni 2004 21:27 schrieb James Bottomley:
> On Sun, 2004-06-20 at 13:02, Oliver Neukum wrote:
> > > The DMA API is about allowing devices to transact directly with memory
> > > behind the memory controller, it's an API that essentially allows the
> > > I/O controller and memory controller to communicate without CPU
> > > intervention.  This is still possible through the hypervisor, so the
> > > iSeries currently fully implements the DMA API.
> > 
> > Then what's the problem?
> 
> If you look at the diagram, you'll see that the OHCI memory isn't behind
> the memory controller...

And that means what? We don't care about how memory is implemented
in the machines we run on. All we care about is the consequences of the
implementation.
We have an area of memory here that the CPU and a subset of devices
can directly access. What to do about it?

	Regards
		Oliver
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFA1ebgbuJ1a+1Sn8oRAnFdAJ45v2h3fUe7NIPVxyS9CmH69vEHLgCghgQe
qXQVLQOlkEabmPVqmCOaSq8=
=35je
-----END PGP SIGNATURE-----

Re: DMA API issues

[David Brownell]

James Bottomley wrote:

> The DMA API is about allowing devices to transact directly with memory
> behind the memory controller, ...

Nope, that's certainly not in the API spec.  Never has been,
and I'd have objected if it had been ... because I knew this
was one of the types of hardware the API needed to support as
Linux ran on more systems.  The call syntax just returns two
addresses, usable by host and by device:

     void *
     dma_alloc_coherent(struct device *dev, size_t size,
                 dma_addr_t *dma_handle, int flag);

That doesn't say ANYTHING about where that memory lives, or
who may or may not have set up a "struct page" (or what the
page size is for that component).  It certainly doesn't make
assumptions about which busses are used by CPU or device for
memory or control access, or how they're linked.

- Dave

Re: DMA API issues

[David Brownell]

James Bottomley wrote:

> The iseries can't because the PCI bus sits behind the hypervisor and has
> to use accessors to get at the on chip memory, it can't simply be mapped
> into the address space like it can on ARM.

So you're saying that the iSeries can't implement dma_alloc_coherent()
for such hardware, yes?  In that case, it should return a polite failure
status.  Meanwhile, hardware that _can_ implement it should do so.

- Dave

Re: DMA API issues

[David Brownell]

Russell King wrote:
> On Sat, Jun 19, 2004 at 11:23:23AM -0700, David Brownell wrote:
> 
>>I'm having to guess at your point here, even from other emails.
>>You've asserted a difference, but not what it is.  Maybe it's
>>something to do with the problem's NUMA nature?  Are you for
>>some reason applying DMA _mapping_ requirements (main-memory
>>only) to the DMA memory _allocation_ problem?
> 
> ...
> 
> Currently, there are drivers which assume that it's possible that
> dma_alloc_coherent memory is backed by system memory, which has
> page structures associated with each page.  For this "new" memory,
> there are no such page structures, so things like bus_to_virt()
> don't work on them (not that they were guaranteed to work on a

This shouldn't include the USB stack, FWIW; nothing in drivers/usb
makes such calls (on 2.6).  The device drivers don't have any
reason to use such calls, either.


> In addition, the ARM implementation of dma_alloc_coherent()
> implicitly believes in struct page pointers - they're a fundamental
> properly of the way that has been implemented, so any deviation from
> "memory with struct page" means more or less a rewrite this.

Or just bypassing it before it gets to __dma_alloc(), which would
be ugly but functional.  Or flagging the devices in question as
needing to use that particular memory zone ... so for example a
dma_zone(dev) macro, used with alloc_pages_node(), might be a
cleaner approach.


> I would say that, yes, from a perfectly objective view, if you are
> unable to do coherent DMA from system memory, but your system provides
> you with a totally separate memory system which does indeed provide
> coherent DMA, it seems logical to allow dma_alloc_coherent() to use
> it - at risk of breaking some drivers making incorrect assumptions.
> 
> And I don't see _that_ case as being vastly different from Ian's
> case.

That's hardly a vast difference, no!


> So, I think as long as we can ensure that drivers do not make bad
> assumptions about dma_alloc_coherent() _and_ we have a suitable DMA
> MMAP API to solve the cases where device drivers want to mmap DMA
> buffers (and they _do_ want to do this) it should be possible.

Right, and the whole USB stack should "just work" once that's done.
Subject to memory cramping for some uses!  :)


> Depending on how I look at the problem, I'm oscillating between "yes
> it should be done" (if its an overall system thing like DMA memory
> on a PCI north bridge separate from your normal system non-DMA
> memory) and "no it's out of the question."

I'm clearly on the "yes" side, though I suspect Deepak is right
that doing it very cleanly as an all-arch extension may not be
a near-term 2.6 option.

- Dave

Re: DMA API issues

[Ian Molton]

On 18 Jun 2004 14:57:01 -0500
James Bottomley <James.Bottomley@SteelEye.com> wrote:

> 
> There are complications to this: not all platforms can access PCI memory
> directly.  That's why ioremap and memcpy_toio and friends exist.  What
> should happen on these platforms?

I wasnt talking about a PCI system here.

Re: DMA API issues

[James Bottomley]

On Fri, 2004-06-18 at 18:25, Ian Molton wrote:
> On 18 Jun 2004 14:57:01 -0500
> James Bottomley <James.Bottomley@SteelEye.com> wrote:
> > There are complications to this: not all platforms can access PCI memory
> > directly.  That's why ioremap and memcpy_toio and friends exist.  What
> > should happen on these platforms?
> 
> I wasnt talking about a PCI system here.

ioremap is used for all bus remote MMIO regions, not just PCI.

James

Re: DMA API issues

[Ian Molton]

On 18 Jun 2004 18:29:22 -0500
James Bottomley <James.Bottomley@SteelEye.com> wrote:

> > 
> > I wasnt talking about a PCI system here.
> 
> ioremap is used for all bus remote MMIO regions, not just PCI.

Im aware of that but the OHCI core doesnt do that, it uses the DMA API,
which is entirely reasonable, given its tring to access a DMA-able chunk
of memory.

I *could* write a new driver ohci-ioremapped for all these chips but its
needless duplication which is going to result in bugs bveing fixed in
one ohci driver and not the other.

why not simply expand the DMA API to allow DMA to these easily DMA-able
chips ?

Re: DMA API issues

[James Bottomley]

On Fri, 2004-06-18 at 18:51, Ian Molton wrote:
> Im aware of that but the OHCI core doesnt do that, it uses the DMA API,
> which is entirely reasonable, given its tring to access a DMA-able chunk
> of memory.
> 
> I *could* write a new driver ohci-ioremapped for all these chips but its
> needless duplication which is going to result in bugs bveing fixed in
> one ohci driver and not the other.
> 
> why not simply expand the DMA API to allow DMA to these easily DMA-able
> chips ?

Because the piece of memory you wish to access is bus remote.  Our
current API for bus remote memory requires the use of ioremap and the
accessor functions.  Just because it could be accessed directly on ARM
doesn't mean it can be on all platforms.  The DMA API is a platform
generic API.  To propose an addition to it to make use of this memory,
it would have to work on *all* platforms.

James

Re: DMA API issues

[Ian Molton]

On 18 Jun 2004 19:04:11 -0500
James Bottomley <James.Bottomley@SteelEye.com> wrote:

> Because the piece of memory you wish to access is bus remote. 

No, its *not*

my CPU can write there directly.

no strings attached.

the DMA API just only understands how to map from RAM, not anything
else.

Re: DMA API issues

[James Bottomley]

On Fri, 2004-06-18 at 19:14, Ian Molton wrote:
> On 18 Jun 2004 19:04:11 -0500
> James Bottomley <James.Bottomley@SteelEye.com> wrote:
> 
> > Because the piece of memory you wish to access is bus remote. 
> 
> No, its *not*
> 
> my CPU can write there directly.
> 
> no strings attached.
> 
> the DMA API just only understands how to map from RAM, not anything
> else.

I think you'll actually find that it is.  OHCI is a device (representing
a USB hub), it's attached to the system by some interface that
constitutes a bus (the bus interface transforming the CPU access cycles
to device access cycles, translating interrupts etc.).

But even if you've somehow managed to glue an OHCI directly on to the
system memory controller, from the point of view of the DMA API, the
memory the device contains is still bus remote.  To be useful, the API
has to deal with bus remote memory in all its forms.

James

Re: DMA API issues

[Jamey Hicks]

James Bottomley wrote:

>On Fri, 2004-06-18 at 19:14, Ian Molton wrote:
>  
>
>>On 18 Jun 2004 19:04:11 -0500
>>James Bottomley <James.Bottomley@SteelEye.com> wrote:
>>
>>    
>>
>>>Because the piece of memory you wish to access is bus remote. 
>>>      
>>>
>>No, its *not*
>>
>>my CPU can write there directly.
>>
>>no strings attached.
>>
>>the DMA API just only understands how to map from RAM, not anything
>>else.
>>    
>>
>
>I think you'll actually find that it is.  OHCI is a device (representing
>a USB hub), it's attached to the system by some interface that
>constitutes a bus (the bus interface transforming the CPU access cycles
>to device access cycles, translating interrupts etc.).
>
>  
>
Bus remote is a red herring in this case.  The only difference between 
this case and the ones supported by the coherent_dma_mask is that the 
constraint on placement of the allocated memory cannot be encoded as a 
bitmask.

Jamey

DMA API issues... summary

[Ian Molton]

Ok, heres a summary of the problems we have (feel free to add any more problems).

We have two types of "device": single function devices and 'system on chip' devices which have multiple functions.

Single chip devices may be able to either access system memory directly, or may only be able to access their internal SRAM pool. in the case of the latter the system can either directly access the SRAM or not, depending on the device/bus setup. Its possible the devices may have more than one non-continuous SRAM mapping.

The same goes for SOC devices, however they could come in two 'classes'. In one type, we would essentially have multiple independant devices in a single chip. In another case (which appears to be fairly common) we can have multiple devices sharing a common SRAM pool. its also possible to have some devices sharing the pool and some having their own in the same chip.

Can anyone describe another type of chip we need to accomodate?

Re: DMA API issues... summary

[Joshua Wise]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jumping into the discussion from the middle of nowhere ......

> Single chip devices may be able to either access system memory directly, or
> may only be able to access their internal SRAM pool. in the case of the
> latter the system can either directly access the SRAM or not, depending on
> the device/bus setup. Its possible the devices may have more than one
> non-continuous SRAM mapping.
>
> The same goes for SOC devices, however they could come in two 'classes'. In
> one type, we would essentially have multiple independant devices in a
> single chip. In another case (which appears to be fairly common) we can
> have multiple devices sharing a common SRAM pool. its also possible to have
> some devices sharing the pool and some having their own in the same chip.

First off ... Why just say the internal SRAM pool? I think it woudl be better 
to say that the devices can only access their own address space, and can only 
DMA from a subset of that (which may be the full original set, QED)

Second... Remember that the SOC can also be the CPU! At least on XScale, there 
are some portions that we can DMA from main memory (I think USB is one).

Of course this might not be at all relevant to the discussion, and of course I 
could be using my rectally-based knowledge, but I _THINK_ that this is 
correct.

joshua
- -- 
Joshua Wise | www.joshuawise.com
GPG Key     | 0xEA80E0B3
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFA1fhdPn9tWOqA4LMRAuwVAKCq2kcu0V1nnBtZZgwSkTAM2a/izACbBAKu
0ef8cBr9EfxqeYILtdzrgvw=
=B2Yf
-----END PGP SIGNATURE-----

Re: DMA API issues

[James Bottomley]

On Fri, 2004-06-18 at 13:57, Ian Molton wrote:
> In *theory* the OHCI driver is doing everything right - its asking for DMAable memory and using it. if the DMA api simply understood the device in question, and alocated accordingly, it would just work.
> 
> there are two solutions:
> 
> 1) Break up the OHCI driver and make it into a chip driver as you describe
> 2) Make the DMA API do the right thing with these devices

Could you please just describe what the problem actually is.

The ohci driver looks to be reasonably modular already, with a chip
piece and a bus attachment piece.

Is your problem that you'd like the dma pools it uses to come out of the
on chip buffer?

James

Re: DMA API issues

[Ian Molton]

On 18 Jun 2004 14:30:01 -0500
James Bottomley <James.Bottomley@SteelEye.com> wrote:

> Is your problem that you'd like the dma pools it uses to come out of the
> on chip buffer?

Correct. its the right way to do this.

Re: DMA API issues

[Jamey Hicks]

James Bottomley wrote:

>You still haven't explained what you want to do though.  Apart from the
>occasional brush with usbstorage, I don't have a good knowledge of the
>layout of the USB drivers.  I assume you simply want to persuade the
>ohci driver to use your memory area somehow, but what do you actually
>want the ohci driver to do with it?  And how much leeway do you get to
>customise the driver.
>
>  
>
It's really not a question of laziness.  The ASICs we are interested in 
implement OHCI, so I think the core OHCI driver should work unmodified.  
OHCI driver allocates dma_pools for managing endpoint descriptors (ED) 
and transaction descriptors (TD).  I expect that the driver wrapper that 
initializes the OHCI controller driver will create dma_pools drawing 
from the ASIC's private SRAM.  The OHCI driver uses 
dma_{alloc,free}_coherent to manage the space used for the top level 
control structure shared between the driver and the controller 
hardware.  This also needs to be allocated in the SRAM.  Finally, in 
drivers/usb/core/usb.c, the USB drivers call dma_map_single and 
dma_unmap_single given pointers to transfer buffers allocated by the USB 
device drivers.  If the USB device is a network device (as it is on the 
iPAQ), the transfer buffers are allocated via dev_alloc_skb. 

>The reason I'm asking is beause it's still unclear whether this is a DMA
>API issue or an ohci one.  I could solve my Q720 issue simply by
>exporting an interface from the ncr driver to supply alternative memory
>allocation use and descriptors.
>
>  
>
I really think this is a DMA API implementation issue.  The problem 
touches more than the USB drivers.  I say implementation because the DMA 
API already takes struct device, so the public interface would not have 
to change or would not have to change much.  However, we would like to 
be able to provide device-specific implementations of the dma 
operations.  One way to implement this would be a pointer to 
dma_operations from struct device.

Jamey

Re: DMA API issues

[James Bottomley]

On Fri, 2004-06-18 at 14:22, Jamey Hicks wrote:
> It's really not a question of laziness.  The ASICs we are interested in 
> implement OHCI, so I think the core OHCI driver should work unmodified.  
> OHCI driver allocates dma_pools for managing endpoint descriptors (ED) 
> and transaction descriptors (TD).  I expect that the driver wrapper that 
> initializes the OHCI controller driver will create dma_pools drawing 
> from the ASIC's private SRAM.  The OHCI driver uses 
> dma_{alloc,free}_coherent to manage the space used for the top level 
> control structure shared between the driver and the controller 
> hardware.  This also needs to be allocated in the SRAM.  Finally, in 
> drivers/usb/core/usb.c, the USB drivers call dma_map_single and 
> dma_unmap_single given pointers to transfer buffers allocated by the USB 
> device drivers.  If the USB device is a network device (as it is on the 
> iPAQ), the transfer buffers are allocated via dev_alloc_skb. 

Well, I thought it was something like that.  So the problem could be
solved simply by rejigging ohci to export td_alloc and td_free as
overrideable methods?

Your map and unmap single could also be handled this way: with usb
specific overrides that default to dma_map_single.  None of this would
cause much perturbation in usb, and it would give you everthing you
need.

I assume your implementation of dma_map_single is simply to copy the
memory into the on chip area?

> I really think this is a DMA API implementation issue.  The problem 
> touches more than the USB drivers.  I say implementation because the DMA 
> API already takes struct device, so the public interface would not have 
> to change or would not have to change much.  However, we would like to 
> be able to provide device-specific implementations of the dma 
> operations.  One way to implement this would be a pointer to 
> dma_operations from struct device.

The DMA API is highly platform specific.  It basically embodies a
contract between the platform and its attached busses.  It wasn't
designed to embody a contract between two busses (or in this case, a bus
and its implementing driver).

James

Re: DMA API issues

[Oliver Neukum]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Am Freitag, 18. Juni 2004 21:41 schrieb James Bottomley:
> Well, I thought it was something like that.  So the problem could be
> solved simply by rejigging ohci to export td_alloc and td_free as
> overrideable methods?

Unfortunately no. Usb_buffer_alloc() needs to know about the restriction,
too.

	Regards
		Oliver

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFA00pDbuJ1a+1Sn8oRAoqjAKDVMBJCgjrysIZlQYdLDFCTEic6JgCfQ6t/
g4B4/fqQwvFNelxVo4sQO3o=
=q4nt
-----END PGP SIGNATURE-----

Re: DMA API issues

[James Bottomley]

On Fri, 2004-06-18 at 15:02, Oliver Neukum wrote:
> Am Freitag, 18. Juni 2004 21:41 schrieb James Bottomley:
> > Well, I thought it was something like that.  So the problem could be
> > solved simply by rejigging ohci to export td_alloc and td_free as
> > overrideable methods?
> 
> Unfortunately no. Usb_buffer_alloc() needs to know about the restriction,
> too.

But usb_buffer_alloc is already an indirected operation, it looks like
it can be easily overridden to do precisely what you want.

James

Re: DMA API issues

[Benjamin Herrenschmidt]

> I really think this is a DMA API implementation issue.  The problem 
> touches more than the USB drivers.  I say implementation because the DMA 
> API already takes struct device, so the public interface would not have 
> to change or would not have to change much.  However, we would like to 
> be able to provide device-specific implementations of the dma 
> operations.  One way to implement this would be a pointer to 
> dma_operations from struct device.

I wanted to do just that a while ago, and ended up doing things a bit
differently, but still, I agree that would help. The thing is, you
can do that in your platform code. just use the platform data pointer
in struct device to stuff a ptr to the structure with your "ops"

Ben.

Re: DMA API issues

[James Bottomley]

On Fri, 2004-06-18 at 15:14, Benjamin Herrenschmidt wrote:
> I wanted to do just that a while ago, and ended up doing things a bit
> differently, but still, I agree that would help. The thing is, you
> can do that in your platform code. just use the platform data pointer
> in struct device to stuff a ptr to the structure with your "ops"

Yes, we do this on parisc too.  We actually have a hidden method pointer
(per platform) and cache the iommu (we have more than one) accessors in
platform_data.

The problem is, though, that I really don't think this interface would
benefit from being made public (or part of the API).  There are too many
nasty quirks in the platform for this (at least in our case).

James

Re: DMA API issues

[Russell King]

On Fri, Jun 18, 2004 at 03:24:45PM -0500, James Bottomley wrote:
> On Fri, 2004-06-18 at 15:14, Benjamin Herrenschmidt wrote:
> > I wanted to do just that a while ago, and ended up doing things a bit
> > differently, but still, I agree that would help. The thing is, you
> > can do that in your platform code. just use the platform data pointer
> > in struct device to stuff a ptr to the structure with your "ops"
> 
> Yes, we do this on parisc too.  We actually have a hidden method pointer
> (per platform) and cache the iommu (we have more than one) accessors in
> platform_data.

Except that platform_data already has multiple other uses, especially for
platform devices.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core

Re: DMA API issues

[Ian Molton]

On Fri, 18 Jun 2004 22:20:14 +0100
Russell King <rmk+lkml@arm.linux.org.uk> wrote:

> > Yes, we do this on parisc too.  We actually have a hidden method
> > pointer(per platform) and cache the iommu (we have more than one)
> > accessors in platform_data.
> 
> Except that platform_data already has multiple other uses, especially
> for platform devices.

In the case of the SOC devices I described, its actually appropriate to
make the allocator system tied to the bus - as several devices end up
sharing the same 32K pool in the device. at the device level the
allocator would be useless in these cases.

Re: DMA API issues

[Deepak Saxena]

On Jun 19 2004, at 00:20, Ian Molton was caught saying:
> On Fri, 18 Jun 2004 22:20:14 +0100
> Russell King <rmk+lkml@arm.linux.org.uk> wrote:
> 
> > > Yes, we do this on parisc too.  We actually have a hidden method
> > > pointer(per platform) and cache the iommu (we have more than one)
> > > accessors in platform_data.
> > 
> > Except that platform_data already has multiple other uses, especially
> > for platform devices.
> 
> In the case of the SOC devices I described, its actually appropriate to
> make the allocator system tied to the bus - as several devices end up
> sharing the same 32K pool in the device. at the device level the
> allocator would be useless in these cases.

I've followed the whole thread and I still think what you are trying
to do can be currently accomplished with the existing API by simply 
overriding the generic ARM implementation and providing one specific
to your platform. All the dma_* APIs take a dev structure as a
parameter, so you simply have to look at that and if it's your OHCI
device, call your specific allocator/deallocator/mapping function/etc.
Your dma_alloc_coherent() would simply ioremap() and return the
ioremap'd address as the virtual address and the OHCI-bus address 
as the DMA_ADDR. Your dma_map_single() simply bounces data between
the SRAM and system memory.  Having per-bus or per-device allocators 
as you are proposing is very possibly the correct way to go, but I don't 
think it's a simple enough change that we want to do it on 2.6.

~Deepak

-- 
Deepak Saxena - dsaxena at plexity dot net - http://www.plexity.net/

"Unlike me, many of you have accepted the situation of your imprisonment and
 will die here like rotten cabbages." - Number 6

DMA API issues

[Ian Molton]

Hi.

This may have come up previously but I havent seen it, so...

My colleagues and I are encountering a number of difficulties with the
DMA API, to which a generic solution is required (or risk multiple
architectures, busses, and devices going their own way...)

Here is an example system that illustrates these problems:

I have a System On Chip device which, among other functions, contains an
OHCI controller and 32K of SRAM.

heres the catch:- The OHCI controller has a different address space than
the host bus, and worse, can *only* DMA data from its internal SRAM.

The architecture is not broken, merely unusual.

This causes the following problems:

1) The DMA API provides no methods to set up a mapping between the host
   memory map and the devices view of the space
        example:
           the OHCI controller above would see its 32K of SRAM as
           mapped from 0x10000 - 0x1ffff and not 0xXXX10000 - 0xXXX1ffff
           which is the address the CPU sees.
2) The DMA API assumes the device can access SDRAM
        example:
           the OHCI controller base is mapped at 0x10000000 on my platform.
           this is NOT is SDRAM, its in IO space.

If these points are possible to be addressed, it would allow at LEAST three chips *in use* in linux devices able to use mainline OHCI code directly - TC6393XB (in toshiba PDAs), SAMCOP (Ipaqs), and mediaQ (dell axims).

I am told HPPA has some similar problems also.

PS. please make use of CC: when replying

Re: DMA API issues

[Matt Porter]

On Fri, Jun 18, 2004 at 05:59:02PM +0100, Ian Molton wrote:
> I have a System On Chip device which, among other functions, contains an
> OHCI controller and 32K of SRAM.
> 
> heres the catch:- The OHCI controller has a different address space than
> the host bus, and worse, can *only* DMA data from its internal SRAM.
> 
> The architecture is not broken, merely unusual.
> 
> This causes the following problems:
> 
> 1) The DMA API provides no methods to set up a mapping between the host
>    memory map and the devices view of the space
>         example:
>            the OHCI controller above would see its 32K of SRAM as
>            mapped from 0x10000 - 0x1ffff and not 0xXXX10000 - 0xXXX1ffff
>            which is the address the CPU sees.
> 2) The DMA API assumes the device can access SDRAM
>         example:
>            the OHCI controller base is mapped at 0x10000000 on my platform.
>            this is NOT is SDRAM, its in IO space.

Can't you just implement an arch-specific allocator for your 32KB
SRAM, then implement the DMA API streaming and dma_alloc/free APIs
on top of that?  Since this architecture is obviously not designed
for performance, it doesn't seem to be a big deal to have the streaming
APIs copy to/from the kmalloced (or whatever) buffer to/from the SRAM
allocated memory and then have those APIs return the proper dma_addr_t
for the embedded OHCI's address space view of the SRAM. Same thing
goes for implementing dma_alloc/free (used by dma_pool*). I don't
have the knowledge of USB subsystem buffer usage to know how quickly
that little 32KB of SRAM is going to run out. But at a DMA API level,
this seems doable, albeit with the greater possibility of negative
retvals from these calls.

-Matt

Re: DMA API issues

[Ian Molton]

On Fri, 18 Jun 2004 11:07:21 -0700
Matt Porter <mporter@kernel.crashing.org> wrote:

> Can't you just implement an arch-specific allocator for your 32KB
> SRAM, then implement the DMA API streaming and dma_alloc/free APIs
> on top of that?

Yes but thats not very generic is it? Im not the only one with this
problem.

>  Since this architecture is obviously not designed
> for performance

What makes you think writes to the 32K SRAM are any slower than to the
SDRAM? the device is completely memory mapped.

>, it doesn't seem to be a big deal to have the streaming
> APIs copy to/from the kmalloced (or whatever) buffer to/from the SRAM
> allocated memory and then have those APIs return the proper dma_addr_t
> for the embedded OHCI's address space view of the SRAM.

Again its a suboptimal solution, and on an architecture where the CPU
isnt *that* fast in the first place it seems wrong to deliberately
choose the slowest possible route...

Re: DMA API issues

[Matt Porter]

On Fri, Jun 18, 2004 at 07:19:58PM +0100, Ian Molton wrote:
> On Fri, 18 Jun 2004 11:07:21 -0700
> Matt Porter <mporter@kernel.crashing.org> wrote:
> 
> > Can't you just implement an arch-specific allocator for your 32KB
> > SRAM, then implement the DMA API streaming and dma_alloc/free APIs
> > on top of that?
> 
> Yes but thats not very generic is it? Im not the only one with this
> problem.

Yes, it's suboptimal, but an option.
 
> >  Since this architecture is obviously not designed
> > for performance
> 
> What makes you think writes to the 32K SRAM are any slower than to the
> SDRAM? the device is completely memory mapped.

I was referring to the small amount of space allowed for DMA
operations, obviously not the speed of accessing SRAM.
 
> >, it doesn't seem to be a big deal to have the streaming
> > APIs copy to/from the kmalloced (or whatever) buffer to/from the SRAM
> > allocated memory and then have those APIs return the proper dma_addr_t
> > for the embedded OHCI's address space view of the SRAM.
> 
> Again its a suboptimal solution, and on an architecture where the CPU
> isnt *that* fast in the first place it seems wrong to deliberately
> choose the slowest possible route...

Ok, so you're looking for a complete change to the streaming DMA APIs,
I guess.  Possibly requiring another call to allocate streaming-capable
memory since kmalloced buffers can't be used directly on your arch (or
all arches).  I agree it's suboptimal, it's one option to make it work
in the current API.

-Matt

Re: DMA API issues

[Jamey Hicks]

Matt Porter wrote:

>On Fri, Jun 18, 2004 at 05:59:02PM +0100, Ian Molton wrote:
>  
>
>>I have a System On Chip device which, among other functions, contains an
>>OHCI controller and 32K of SRAM.
>>
>>heres the catch:- The OHCI controller has a different address space than
>>the host bus, and worse, can *only* DMA data from its internal SRAM.
>>
>>The architecture is not broken, merely unusual.
>>
>>This causes the following problems:
>>
>>1) The DMA API provides no methods to set up a mapping between the host
>>   memory map and the devices view of the space
>>        example:
>>           the OHCI controller above would see its 32K of SRAM as
>>           mapped from 0x10000 - 0x1ffff and not 0xXXX10000 - 0xXXX1ffff
>>           which is the address the CPU sees.
>>2) The DMA API assumes the device can access SDRAM
>>        example:
>>           the OHCI controller base is mapped at 0x10000000 on my platform.
>>           this is NOT is SDRAM, its in IO space.
>>    
>>
>
>Can't you just implement an arch-specific allocator for your 32KB
>SRAM, then implement the DMA API streaming and dma_alloc/free APIs
>on top of that?  Since this architecture is obviously not designed
>for performance, it doesn't seem to be a big deal to have the streaming
>APIs copy to/from the kmalloced (or whatever) buffer to/from the SRAM
>allocated memory and then have those APIs return the proper dma_addr_t
>for the embedded OHCI's address space view of the SRAM. Same thing
>goes for implementing dma_alloc/free (used by dma_pool*). I don't
>have the knowledge of USB subsystem buffer usage to know how quickly
>that little 32KB of SRAM is going to run out. But at a DMA API level,
>this seems doable, albeit with the greater possibility of negative
>retvals from these calls.
>
>  
>
Your analysis of what is needed is correct.  However, we would prefer to 
fit this into the DMA API in a generic way, rather than having a 
specialized API that is not acceptable upstream.  I'm more concerned 
with finding the best way to address this problem than with whether this 
is a 2.6 or 2.7 issue.

We do need a memory allocator for the pool of SRAM.  If we're not going 
to have to build customized versions of the OHCI driver, we need that 
pool hooked into the dma_pool and dma_alloc_coherent interfaces.  We 
could do that with a platform specific implementation of 
dma_alloc_coherent, but a pointer to dma_{alloc,free} from struct device 
seems like a cleaner solution.  We also will need bounce buffers, as you 
described, but there is already a good start towards that support in 
2.6.  The other thing that might be needed is passing device to 
page_to_dma so that device specific dma addresses can be constructed.

Deepak Saxena wrote a pretty good summary as part if the discussion 
about this issue on the linux-arm-kernel mailing list:

  
http://lists.arm.linux.org.uk/pipermail/linux-arm-kernel/2004-June/022796.html

I think I'm looking for something like the PARISC hppa_dma_ops but more 
generic:
  
http://lists.arm.linux.org.uk/pipermail/linux-arm-kernel/2004-June/022813.html


Jamey Hicks

Re: DMA API issues

[Matt Porter]

On Fri, Jun 18, 2004 at 02:33:18PM -0400, Jamey Hicks wrote:
> Matt Porter wrote:
> >On Fri, Jun 18, 2004 at 05:59:02PM +0100, Ian Molton wrote:
> >>1) The DMA API provides no methods to set up a mapping between the host
> >>   memory map and the devices view of the space
> >>        example:
> >>           the OHCI controller above would see its 32K of SRAM as
> >>           mapped from 0x10000 - 0x1ffff and not 0xXXX10000 - 0xXXX1ffff
> >>           which is the address the CPU sees.
> >>2) The DMA API assumes the device can access SDRAM
> >>        example:
> >>           the OHCI controller base is mapped at 0x10000000 on my platform.
> >>           this is NOT is SDRAM, its in IO space.
> >
> >Can't you just implement an arch-specific allocator for your 32KB
> >SRAM, then implement the DMA API streaming and dma_alloc/free APIs
> >on top of that?  Since this architecture is obviously not designed
> >for performance, it doesn't seem to be a big deal to have the streaming
> >APIs copy to/from the kmalloced (or whatever) buffer to/from the SRAM
> >allocated memory and then have those APIs return the proper dma_addr_t
> >for the embedded OHCI's address space view of the SRAM. Same thing
> >goes for implementing dma_alloc/free (used by dma_pool*). I don't
> >have the knowledge of USB subsystem buffer usage to know how quickly
> >that little 32KB of SRAM is going to run out. But at a DMA API level,
> >this seems doable, albeit with the greater possibility of negative
> >retvals from these calls.
> >
> Your analysis of what is needed is correct.  However, we would prefer to 
> fit this into the DMA API in a generic way, rather than having a 
> specialized API that is not acceptable upstream.  I'm more concerned 
> with finding the best way to address this problem than with whether this 
> is a 2.6 or 2.7 issue.
> 
> We do need a memory allocator for the pool of SRAM.  If we're not going 
> to have to build customized versions of the OHCI driver, we need that 
> pool hooked into the dma_pool and dma_alloc_coherent interfaces.  We 
> could do that with a platform specific implementation of 
> dma_alloc_coherent, but a pointer to dma_{alloc,free} from struct device 
> seems like a cleaner solution.  We also will need bounce buffers, as you 

Yes, it can be cleaner, but it's not something I would say is
completely broken with the DMA API.  It does provide a way to
make your device specific implementation, regardless of whether
it's managed ideally via a struct device pointer.  Migrating to
dev->dma* sounds suspiciously like a 2.7ism.

> described, but there is already a good start towards that support in 
> 2.6.  The other thing that might be needed is passing device to 

Where's that code?

> page_to_dma so that device specific dma addresses can be constructed.

A struct device argument to page_to_dma seems like a no brainer to be
included.
 
> Deepak Saxena wrote a pretty good summary as part if the discussion 
> about this issue on the linux-arm-kernel mailing list:
> 
> http://lists.arm.linux.org.uk/pipermail/linux-arm-kernel/2004-June/022796.html

Ahh, ok.  Deepak and I have discussed this idea F2F on a several
occassions, I recall he needed it for the small floating PCI window
he has to manage on the IXP* ports.  It may help in some embedded
PPC areas as well.

> I think I'm looking for something like the PARISC hppa_dma_ops but more 
> generic:
>   
> http://lists.arm.linux.org.uk/pipermail/linux-arm-kernel/2004-June/022813.html

I see that's somewhat like what David Brownell suggested before...a single
pointer to a set of dma ops from struct device.  hppa_dma_ops translated
into a generic dma_ops entity with fields corresponding to existing
DMA API calls would be a good starting point. We can get rid of some
address translation hacks in a lot of custom embedded PPC drivers
with something like this.

-Matt

Re: DMA API issues

[Russell King]

On Fri, Jun 18, 2004 at 12:21:12PM -0700, Matt Porter wrote:
> > page_to_dma so that device specific dma addresses can be constructed.
> 
> A struct device argument to page_to_dma seems like a no brainer to be
> included.

Tony Lindgren recently submitted a patch for this:

http://www.arm.linux.org.uk/developer/patches/viewpatch.php?id=1931/1

which now pending for Linus.  ARM platforms now have three macros to
define if they want to override the default struct page to DMA address
translation.

> I see that's somewhat like what David Brownell suggested before...a single
> pointer to a set of dma ops from struct device.  hppa_dma_ops translated
> into a generic dma_ops entity with fields corresponding to existing
> DMA API calls would be a good starting point. We can get rid of some
> address translation hacks in a lot of custom embedded PPC drivers
> with something like this.

I really don't think we need to go this far.

As I understand it, the issue seems to surround DMA coherent memory
for USB descriptors, and what happens when we call the streaming DMA
API.

We have the latter solved with Deepak's DMA bounce code (already merged)
provided it's given the right information to determine when bounce
buffers are needed.

The bounce code only needs a way to get at the "safe" DMA memory, and
it uses DMA pools for that.  DMA pools in turn take their memory from
dma_alloc_coherent.

So, we just need a way to make dma_alloc_coherent do the right thing.
One suggestion from James yesterday was to have a way to register
device-private "coherent DMA" backing memory with dma_alloc_coherent,
which it would use in preference to calling alloc_pages().  However,
this memory would have no struct page pointer associated with it, and
our dma_alloc_coherent implementation currently relies completely on
that condition existing - so it would mean a complete rewrite of that.

Note also that some drivers (notably ALSA) assume that memory returned
from dma_alloc_coherent() does have struct page pointers, so this would
also break ALSA (which in turn provides much more of a justification
for the DMA MMAP API I was trying (and failed) to propose a few months
back.)

Also, we need to consider what the DMA mask means in this case?  Should
it be zero (to mean "can't DMA from system memory"?)

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core

Re: DMA API issues

[Takashi Iwai]

At Fri, 18 Jun 2004 20:43:22 +0100,
Russell King wrote:
> 
> On Fri, Jun 18, 2004 at 12:21:12PM -0700, Matt Porter wrote:
> > > page_to_dma so that device specific dma addresses can be constructed.
> > 
> > A struct device argument to page_to_dma seems like a no brainer to be
> > included.
> 
> Tony Lindgren recently submitted a patch for this:
> 
> http://www.arm.linux.org.uk/developer/patches/viewpatch.php?id=1931/1
> 
> which now pending for Linus.  ARM platforms now have three macros to
> define if they want to override the default struct page to DMA address
> translation.

Wouldn't it be nicer to define a more generic style like

struct page *dma_to_page(struct device *, void *, dma_addr_t)

??


> 
> > I see that's somewhat like what David Brownell suggested before...a single
> > pointer to a set of dma ops from struct device.  hppa_dma_ops translated
> > into a generic dma_ops entity with fields corresponding to existing
> > DMA API calls would be a good starting point. We can get rid of some
> > address translation hacks in a lot of custom embedded PPC drivers
> > with something like this.
> 
> I really don't think we need to go this far.
> 
> As I understand it, the issue seems to surround DMA coherent memory
> for USB descriptors, and what happens when we call the streaming DMA
> API.
> 
> We have the latter solved with Deepak's DMA bounce code (already merged)
> provided it's given the right information to determine when bounce
> buffers are needed.
> 
> The bounce code only needs a way to get at the "safe" DMA memory, and
> it uses DMA pools for that.  DMA pools in turn take their memory from
> dma_alloc_coherent.
> 
> So, we just need a way to make dma_alloc_coherent do the right thing.
> One suggestion from James yesterday was to have a way to register
> device-private "coherent DMA" backing memory with dma_alloc_coherent,
> which it would use in preference to calling alloc_pages().  However,
> this memory would have no struct page pointer associated with it, and
> our dma_alloc_coherent implementation currently relies completely on
> that condition existing - so it would mean a complete rewrite of that.
> 
> Note also that some drivers (notably ALSA) assume that memory returned
> from dma_alloc_coherent() does have struct page pointers, so this would
> also break ALSA (which in turn provides much more of a justification
> for the DMA MMAP API I was trying (and failed) to propose a few months
> back.)

Yes, the struct page pointer is needed for vma_ops.nopage in mmap on
ALSA.  So far, this is broken on some architectures like ARM.  We need
a proper conversion from virtual/bus pointer to a page struct.

Of course, mmap is not mandatory, and we have a fallback via
ioctl/read/write.  But it's important to _know_ whether remapping is
available...


--
Takashi Iwai <tiwai@suse.de>		ALSA Developer - www.alsa-project.org

Re: DMA API issues

[Russell King]

On Mon, Jun 21, 2004 at 03:35:42PM +0200, Takashi Iwai wrote:
> At Fri, 18 Jun 2004 20:43:22 +0100,
> Russell King wrote:
> > 
> > On Fri, Jun 18, 2004 at 12:21:12PM -0700, Matt Porter wrote:
> > > > page_to_dma so that device specific dma addresses can be constructed.
> > > 
> > > A struct device argument to page_to_dma seems like a no brainer to be
> > > included.
> > 
> > Tony Lindgren recently submitted a patch for this:
> > 
> > http://www.arm.linux.org.uk/developer/patches/viewpatch.php?id=1931/1
> > 
> > which now pending for Linus.  ARM platforms now have three macros to
> > define if they want to override the default struct page to DMA address
> > translation.
> 
> Wouldn't it be nicer to define a more generic style like
> 
> struct page *dma_to_page(struct device *, void *, dma_addr_t)

What's the 'void *' for?  Hint: this has nothing to do with the virtual
address and DMA address returned from dma_alloc_coherent().

page_to_dma - converts a struct page to a DMA address for a given device
dma_to_virt - converts a DMA address to a CPU direct-mapped virtual address
virt_to_dma - converts a CPU direct-mapped virtual address to a DMA address

Each one well defined for our _current_ interfaces.

> Yes, the struct page pointer is needed for vma_ops.nopage in mmap on
> ALSA.  So far, this is broken on some architectures like ARM.  We need
> a proper conversion from virtual/bus pointer to a page struct.

Please get away from your nopage implementation.  We've been around this
before with Linus, and the conclusion was that it's up to architectures
to provide a MMAP method for DMA memory, which _may_ use the nopage
method if and only if it is appropriate for their implementation.

There's just no way I'm going to implement a half-baked "nopage" function
for ALSA.

This is actually the same issue for some framebuffer devices as well -
which also need DMA MMAP to be correct.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core

Re: DMA API issues

[Jeff Garzik]

Russell King wrote:
> On Mon, Jun 21, 2004 at 03:35:42PM +0200, Takashi Iwai wrote:
>>Yes, the struct page pointer is needed for vma_ops.nopage in mmap on
>>ALSA.  So far, this is broken on some architectures like ARM.  We need
>>a proper conversion from virtual/bus pointer to a page struct.
> 
> 
> Please get away from your nopage implementation.  We've been around this
> before with Linus, and the conclusion was that it's up to architectures
> to provide a MMAP method for DMA memory, which _may_ use the nopage
> method if and only if it is appropriate for their implementation.


Can you elaborate?

When I was writing sound/oss/via82cxxx_audio.c, Linus specifically 
recommended using ->nopage, which worked out quite well (modulo cleaning 
up VM_xxx flags).

	Jeff

Re: DMA API issues

[Linus Torvalds]

On Mon, 21 Jun 2004, Jeff Garzik wrote:
> 
> When I was writing sound/oss/via82cxxx_audio.c, Linus specifically 
> recommended using ->nopage, which worked out quite well (modulo cleaning 
> up VM_xxx flags).

It does need some architecture-specific support for getting the caching
behaviour right, but I do believe that as long as there is some "struct
page" that can be returned, together with a way of setting the
vm_page_prot bits right, it should be ok.

The argument at some point was that some architectures may not even _have_
a "struct page" for DMA memory, since it's not "normal" memory (ie "slow
memory" on m68k). However, I thought we all agreed that such a "struct
page" could be furnished if that architecture wants so support mmap'ing.

		Linus

Re: DMA API issues

[Linus Torvalds]

On Mon, 21 Jun 2004, Linus Torvalds wrote:
> 
> The argument at some point was that some architectures may not even _have_
> a "struct page" for DMA memory, since it's not "normal" memory (ie "slow
> memory" on m68k). However, I thought we all agreed that such a "struct
> page" could be furnished if that architecture wants so support mmap'ing.

.. which is not to say that we shouldn't have a "pci_mmap_pages()" thing
_too_. Pretty clearly the easiest interface often is to just map the pages
at mmap() time, and then we should just have a helper function to do that. 

I thought we did one already, but hey, maybe not.

		Linus

Re: DMA API issues

[Takashi Iwai]

At Mon, 21 Jun 2004 20:26:39 -0700 (PDT),
Linus Torvalds wrote:
> 
> On Mon, 21 Jun 2004, Linus Torvalds wrote:
> > 
> > The argument at some point was that some architectures may not even _have_
> > a "struct page" for DMA memory, since it's not "normal" memory (ie "slow
> > memory" on m68k). However, I thought we all agreed that such a "struct
> > page" could be furnished if that architecture wants so support mmap'ing.
> 
> .. which is not to say that we shouldn't have a "pci_mmap_pages()" thing
> _too_. Pretty clearly the easiest interface often is to just map the pages
> at mmap() time, and then we should just have a helper function to do that. 
> 
> I thought we did one already, but hey, maybe not.

I don't think we have such.

Russell has once proposed a similar one (but not pci-specific), and I
believe it makes sense for many drivers.  We can hide the
architecture-specific cache handling inside the helper function, too.


Takashi

Re: DMA API issues

[Russell King]

On Tue, Jun 22, 2004 at 12:40:45PM +0200, Takashi Iwai wrote:
> At Mon, 21 Jun 2004 20:26:39 -0700 (PDT),
> Linus Torvalds wrote:
> > 
> > On Mon, 21 Jun 2004, Linus Torvalds wrote:
> > > 
> > > The argument at some point was that some architectures may not even _have_
> > > a "struct page" for DMA memory, since it's not "normal" memory (ie "slow
> > > memory" on m68k). However, I thought we all agreed that such a "struct
> > > page" could be furnished if that architecture wants so support mmap'ing.
> > 
> > .. which is not to say that we shouldn't have a "pci_mmap_pages()" thing
> > _too_. Pretty clearly the easiest interface often is to just map the pages
> > at mmap() time, and then we should just have a helper function to do that. 
> > 
> > I thought we did one already, but hey, maybe not.
> 
> I don't think we have such.
> 
> Russell has once proposed a similar one (but not pci-specific), and I
> believe it makes sense for many drivers.  We can hide the
> architecture-specific cache handling inside the helper function, too.

Ok.  So does anyone have any objections if I push the ARM DMA mmap
interface upstream, which consists of:

/**
 * dma_mmap_coherent - map a coherent DMA allocation into user space
 * @dev: valid struct device pointer, or NULL for ISA and EISA-like devices
 * @vma: vm_area_struct describing requested user mapping
 * @cpu_addr: kernel CPU-view address returned from dma_alloc_coherent
 * @handle: device-view address returned from dma_alloc_coherent
 * @size: size of memory originally requested in dma_alloc_coherent
 *
 * Map a coherent DMA buffer previously allocated by dma_alloc_coherent
 * into user space.  The coherent DMA buffer must not be freed by the
 * driver until the user space mapping has been released.
 */
int dma_mmap_coherent(struct device *dev, struct vm_area_struct *vma,
                      void *cpu_addr, dma_addr_t handle, size_t size);

and a similar one for the ARM-specific "write combining" case (for
framebuffers utilising the DMA API)?

This was discussed on linux-arch, and the discussion died while trying
to settle on a suitable interface through apparant lack of interest.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core

Re: DMA API issues

[Takashi Iwai]

At Wed, 23 Jun 2004 13:34:23 +0100,
Russell King wrote:
> 
> On Tue, Jun 22, 2004 at 12:40:45PM +0200, Takashi Iwai wrote:
> > At Mon, 21 Jun 2004 20:26:39 -0700 (PDT),
> > Linus Torvalds wrote:
> > > 
> > > On Mon, 21 Jun 2004, Linus Torvalds wrote:
> > > > 
> > > > The argument at some point was that some architectures may not even _have_
> > > > a "struct page" for DMA memory, since it's not "normal" memory (ie "slow
> > > > memory" on m68k). However, I thought we all agreed that such a "struct
> > > > page" could be furnished if that architecture wants so support mmap'ing.
> > > 
> > > .. which is not to say that we shouldn't have a "pci_mmap_pages()" thing
> > > _too_. Pretty clearly the easiest interface often is to just map the pages
> > > at mmap() time, and then we should just have a helper function to do that. 
> > > 
> > > I thought we did one already, but hey, maybe not.
> > 
> > I don't think we have such.
> > 
> > Russell has once proposed a similar one (but not pci-specific), and I
> > believe it makes sense for many drivers.  We can hide the
> > architecture-specific cache handling inside the helper function, too.
> 
> Ok.  So does anyone have any objections if I push the ARM DMA mmap
> interface upstream, which consists of:
> 
> /**
>  * dma_mmap_coherent - map a coherent DMA allocation into user space
>  * @dev: valid struct device pointer, or NULL for ISA and EISA-like devices
>  * @vma: vm_area_struct describing requested user mapping
>  * @cpu_addr: kernel CPU-view address returned from dma_alloc_coherent
>  * @handle: device-view address returned from dma_alloc_coherent
>  * @size: size of memory originally requested in dma_alloc_coherent
>  *
>  * Map a coherent DMA buffer previously allocated by dma_alloc_coherent
>  * into user space.  The coherent DMA buffer must not be freed by the
>  * driver until the user space mapping has been released.
>  */
> int dma_mmap_coherent(struct device *dev, struct vm_area_struct *vma,
>                       void *cpu_addr, dma_addr_t handle, size_t size);

This looks easy to use for me.


> and a similar one for the ARM-specific "write combining" case (for
> framebuffers utilising the DMA API)?

pgprot_noncached() is used on many other architectures in fbmem.c
(well, not really, but the result is identical).
Should it be provided as another one, or is it used as default in
dma_mmap_coherent()?

Also, it would be nice to have a version for sg-buffer, too.


Takashi

Re: DMA API issues

[Russell King]

On Wed, Jun 23, 2004 at 05:36:57PM +0200, Takashi Iwai wrote:
> > and a similar one for the ARM-specific "write combining" case (for
> > framebuffers utilising the DMA API)?
> 
> pgprot_noncached() is used on many other architectures in fbmem.c
> (well, not really, but the result is identical).
> Should it be provided as another one, or is it used as default in
> dma_mmap_coherent()?

The whole point is to kill the idea that drivers should have to know
about page protection crap.  That should be wholely contained within
the architecture implementation.

> Also, it would be nice to have a version for sg-buffer, too.

Well, we don't have a sg-buffer version of dma_alloc_coherent(),
so we don't have a sg-buffer version of dma_mmap_coherent().

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core

Re: DMA API issues

[Takashi Iwai]

At Wed, 23 Jun 2004 16:44:19 +0100,
Russell King wrote:
> 
> On Wed, Jun 23, 2004 at 05:36:57PM +0200, Takashi Iwai wrote:
> > > and a similar one for the ARM-specific "write combining" case (for
> > > framebuffers utilising the DMA API)?
> > 
> > pgprot_noncached() is used on many other architectures in fbmem.c
> > (well, not really, but the result is identical).
> > Should it be provided as another one, or is it used as default in
> > dma_mmap_coherent()?
> 
> The whole point is to kill the idea that drivers should have to know
> about page protection crap.  That should be wholely contained within
> the architecture implementation.

I agree.  My question is whether we need to handle different cases
with cached, non-cached and writecombine, according to the demand of
the driver.  In other words, do we always handle these mmap pages to
be non-cached (except for writecombine version)?


> > Also, it would be nice to have a version for sg-buffer, too.
> 
> Well, we don't have a sg-buffer version of dma_alloc_coherent(),
> so we don't have a sg-buffer version of dma_mmap_coherent().

That sound reasonable :)


Takashi

Re: DMA API issues

[Russell King]

On Wed, Jun 23, 2004 at 06:01:04PM +0200, Takashi Iwai wrote:
> At Wed, 23 Jun 2004 16:44:19 +0100,
> Russell King wrote:
> > 
> > On Wed, Jun 23, 2004 at 05:36:57PM +0200, Takashi Iwai wrote:
> > > > and a similar one for the ARM-specific "write combining" case (for
> > > > framebuffers utilising the DMA API)?
> > > 
> > > pgprot_noncached() is used on many other architectures in fbmem.c
> > > (well, not really, but the result is identical).
> > > Should it be provided as another one, or is it used as default in
> > > dma_mmap_coherent()?
> > 
> > The whole point is to kill the idea that drivers should have to know
> > about page protection crap.  That should be wholely contained within
> > the architecture implementation.
> 
> I agree.  My question is whether we need to handle different cases
> with cached, non-cached and writecombine, according to the demand of
> the driver.  In other words, do we always handle these mmap pages to
> be non-cached (except for writecombine version)?

No - the driver should have _zero_ visibility of this.  The only time
the cache property comes in is where the architecture needs to alter
these properties to achieve the requirements of the interface - IOW
to provide user space with a DMA-coherent mapping.

Remember that some architectures are unable to disable caching on a
page by page basis, yet may be able to offer fully-cached DMA coherent
mmaps, and therefore such driver interference is wrong.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core

Re: DMA API issues

[Takashi Iwai]

At Tue, 22 Jun 2004 00:08:38 +0100,
Russell King wrote:
> 
> On Mon, Jun 21, 2004 at 03:35:42PM +0200, Takashi Iwai wrote:
> > At Fri, 18 Jun 2004 20:43:22 +0100,
> > Russell King wrote:
> > > 
> > > On Fri, Jun 18, 2004 at 12:21:12PM -0700, Matt Porter wrote:
> > > > > page_to_dma so that device specific dma addresses can be constructed.
> > > > 
> > > > A struct device argument to page_to_dma seems like a no brainer to be
> > > > included.
> > > 
> > > Tony Lindgren recently submitted a patch for this:
> > > 
> > > http://www.arm.linux.org.uk/developer/patches/viewpatch.php?id=1931/1
> > > 
> > > which now pending for Linus.  ARM platforms now have three macros to
> > > define if they want to override the default struct page to DMA address
> > > translation.
> > 
> > Wouldn't it be nicer to define a more generic style like
> > 
> > struct page *dma_to_page(struct device *, void *, dma_addr_t)
> 
> What's the 'void *' for?  Hint: this has nothing to do with the virtual
> address and DMA address returned from dma_alloc_coherent().
> 
> page_to_dma - converts a struct page to a DMA address for a given device
> dma_to_virt - converts a DMA address to a CPU direct-mapped virtual address
> virt_to_dma - converts a CPU direct-mapped virtual address to a DMA address

Well, I thought the address handling can be dependent on the device
type, thus passing the both virtual and BUS addresses _might_ help to
calculate the DMA address in some cases.  But forget it, seems
exaggerated.


Takashi

Re: DMA API issues

[Jamey Hicks]

Matt Porter wrote:

>On Fri, Jun 18, 2004 at 02:33:18PM -0400, Jamey Hicks wrote:
>  
>
>Yes, it can be cleaner, but it's not something I would say is
>completely broken with the DMA API.  It does provide a way to
>make your device specific implementation, regardless of whether
>it's managed ideally via a struct device pointer.  Migrating to
>dev->dma* sounds suspiciously like a 2.7ism.
>  
>
The DMA API is certainly not completely broken.  Except for page_to_dma 
needing struct device the interface could be implemented with per-device 
dma operation implementations.

>>described, but there is already a good start towards that support in 
>>2.6.  The other thing that might be needed is passing device to 
>>    
>>
>
>Where's that code?
>
>  
>
arch/arm/common/dmabounce.c

>>page_to_dma so that device specific dma addresses can be constructed.
>>    
>>
>
>A struct device argument to page_to_dma seems like a no brainer to be
>included.
> 
>  
>
>>Deepak Saxena wrote a pretty good summary as part if the discussion 
>>about this issue on the linux-arm-kernel mailing list:
>>
>>http://lists.arm.linux.org.uk/pipermail/linux-arm-kernel/2004-June/022796.html
>>    
>>
>
>Ahh, ok.  Deepak and I have discussed this idea F2F on a several
>occassions, I recall he needed it for the small floating PCI window
>he has to manage on the IXP* ports.  It may help in some embedded
>PPC areas as well.
>
>  
>
>>I think I'm looking for something like the PARISC hppa_dma_ops but more 
>>generic:
>>  
>>http://lists.arm.linux.org.uk/pipermail/linux-arm-kernel/2004-June/022813.html
>>    
>>
>
>I see that's somewhat like what David Brownell suggested before...a single
>pointer to a set of dma ops from struct device.  hppa_dma_ops translated
>into a generic dma_ops entity with fields corresponding to existing
>DMA API calls would be a good starting point. We can get rid of some
>address translation hacks in a lot of custom embedded PPC drivers
>with something like this.
>  
>
Yes, I think this would be generally useful.

Jamey

Re: DMA API issues

[Jeff Garzik]

Ian Molton wrote:
> Hi.
> 
> This may have come up previously but I havent seen it, so...
> 
> My colleagues and I are encountering a number of difficulties with the
> DMA API, to which a generic solution is required (or risk multiple
> architectures, busses, and devices going their own way...)
> 
> Here is an example system that illustrates these problems:
> 
> I have a System On Chip device which, among other functions, contains an
> OHCI controller and 32K of SRAM.
> 
> heres the catch:- The OHCI controller has a different address space than
> the host bus, and worse, can *only* DMA data from its internal SRAM.

Unfortunately, I tend to think that you should not be using the DMA API 
for this, since it's not host RAM.

There are loads of devices with on-board SRAM/DRAM/... you really have 
to manage your own resources at that point.

I know that sucks, WRT driver re-use.  Maybe you can wrap it inside the 
OHCI driver, ohci_dma_alloc_xxx() etc.  For the normal case, the wrapper 
resolves directly to the DMA API.  For the embedded case, the wrapper 
does SRAM-specific stuff.

You _might_ convince the kernel DMA gurus that this could be done by 
creating a driver-specific bus, and pointing struct device to that 
internal bus, but that seems like an awful lot of work as opposed to the 
wrappers.

	Jeff

Re: DMA API issues

[Richard B. Johnson]

On Fri, 18 Jun 2004, Jeff Garzik wrote:

> Ian Molton wrote:
> > Hi.
> >
> > This may have come up previously but I havent seen it, so...
> >
> > My colleagues and I are encountering a number of difficulties with the
> > DMA API, to which a generic solution is required (or risk multiple
> > architectures, busses, and devices going their own way...)
> >
> > Here is an example system that illustrates these problems:
> >
> > I have a System On Chip device which, among other functions, contains an
> > OHCI controller and 32K of SRAM.
> >
> > heres the catch:- The OHCI controller has a different address space than
> > the host bus, and worse, can *only* DMA data from its internal SRAM.
>
> Unfortunately, I tend to think that you should not be using the DMA API
> for this, since it's not host RAM.
>
> There are loads of devices with on-board SRAM/DRAM/... you really have
> to manage your own resources at that point.
>
> I know that sucks, WRT driver re-use.  Maybe you can wrap it inside the
> OHCI driver, ohci_dma_alloc_xxx() etc.  For the normal case, the wrapper
> resolves directly to the DMA API.  For the embedded case, the wrapper
> does SRAM-specific stuff.
>
> You _might_ convince the kernel DMA gurus that this could be done by
> creating a driver-specific bus, and pointing struct device to that
> internal bus, but that seems like an awful lot of work as opposed to the
> wrappers.
>
> 	Jeff
>

I don't think that any DMA lends itself very well to a generic
solution and I see that some have dedicated a lot of effort trying
to make such an API.

Take the simple PLX chip that is pretty common, at least as a clone
or macro-cell in many PCI/Bus devices. For its bus-master DMA, you
need a place to build a scatter-list in physical RAM with
a physical address that's known, plus you need kernel-mode virtual
address pointers to be able to build the list.  Then you need to
get the physical address of some user buffer and lock it down during
the transfer, or you need to have kernel buffers that you know the
physical address of, plus have kernel mode pointers to that area.
That area may not even be RAM! And, certainly the place that you
get the data from inside the device may be a single longword with
auto-incrementing hardware.

The fact that you write a 3 to some register somewhere to start
a DMA for a lot of chips, doesn't mean that there will
ever be a generic solution to DMA. Every driver is different.

Even the busmaster Ethernet boards fail to fully utilize the
capability of DMA. The idea of allocating a DMA-able block,
DMAing to it (waiting for the DMA to complete), then de-allocating
that block is extremely wasteful.

I think the whole thing should be re-thought. Something like:
On startup, some list of every page of RAM should be built
(once). A simple algorithm should be made that uses that
list. With a few machine instructions one should be able to
find:

(1)  The physical page of the starting location.
(2)  The amount of space available in that page.
(3)  The physical page of the next location.
(4)  Etc.

Using that information a scatter-list can be built for
the DMA transfer(s) and those pages need to be brought into
memory and locked down only for the duration of the DMA.

This would allow any resident page anywhere to be the source
or target of any DMA operation.


Cheers,
Dick Johnson
Penguin : Linux version 2.4.26 on an i686 machine (5570.56 BogoMips).
            Note 96.31% of all statistics are fiction.

Re: DMA API issues

[Ian Molton]

On Fri, 18 Jun 2004 18:12:48 -0400 (EDT)
"Richard B. Johnson" <root@chaos.analogic.com> wrote:

> 
> Take the simple PLX chip that is pretty common, at least as a clone
> or macro-cell in many PCI/Bus devices.

<cut>

But this *isnt* one of those. its a simple, straightforward, and fairly
common architecture thats so close to the existing model support is near
trivial.

Re: DMA API issues

[Ian Molton]

On Fri, 18 Jun 2004 17:08:03 -0400
Jeff Garzik <jgarzik@pobox.com> wrote:

> You _might_ convince the kernel DMA gurus that this could be done by 
> creating a driver-specific bus, and pointing struct device to that 
> internal bus, but that seems like an awful lot of work as opposed to the 
> wrappers.

Its an awful lot less work than re-writing all those drivers!

Re: DMA API issues

[James Bottomley]

On Fri, 2004-06-18 at 18:26, Ian Molton wrote:
> On Fri, 18 Jun 2004 17:08:03 -0400
> Jeff Garzik <jgarzik@pobox.com> wrote:
> > You _might_ convince the kernel DMA gurus that this could be done by 
> > creating a driver-specific bus, and pointing struct device to that 
> > internal bus, but that seems like an awful lot of work as opposed to the 
> > wrappers.
> 
> Its an awful lot less work than re-writing all those drivers!

Every other driver bar this one already copes correctly with on chip
memory using the ioremap methods.  That's why we're all wondering if it
isn't simpler to fix this driver.

James

Re: DMA API issues

[Jeff Garzik]

James Bottomley wrote:
> On Fri, 2004-06-18 at 18:26, Ian Molton wrote:
> 
>>On Fri, 18 Jun 2004 17:08:03 -0400
>>Jeff Garzik <jgarzik@pobox.com> wrote:
>>
>>>You _might_ convince the kernel DMA gurus that this could be done by 
>>>creating a driver-specific bus, and pointing struct device to that 
>>>internal bus, but that seems like an awful lot of work as opposed to the 
>>>wrappers.
>>
>>Its an awful lot less work than re-writing all those drivers!
> 
> 
> Every other driver bar this one already copes correctly with on chip
> memory using the ioremap methods.  That's why we're all wondering if it
> isn't simpler to fix this driver.


Indeed...  if the SRAM is accessible via ioremapped memory, DMA API 
shouldn't be needed.

	Jeff

Re: DMA API issues

[Ian Molton]

On Fri, 18 Jun 2004 20:22:07 -0400
Jeff Garzik <jgarzik@pobox.com> wrote:

> > my CPU can write directly to this 32K of SRAM. the chip can DMA from
> > it.
> 
> 
> Yes, write via MMIO.  Non-local RAM is not DMA.

I fail to see your point. this is NOT MMIO. in MMIO you write and the
device reads as you write. in DMA, you write and then tell the chip to
read from the memory you wrote to.

this is exactly what Im talking about.

Heres what the DMA mapping code deals with:

CPU-----host bus------>RAM-----io bus---->device


and heres what I have:

CPU-----host bus----->RAM-----io bus----->device

the *only* difference is that the RAM in the first case is SDRAM and in
the latter is SRAM. the type of RAM is irrelevant to the DMA system.

I *could* (at great expense) replace the SDRAM in a PC with SRAM. DMA
would still work the same way.

sure, the SRAM is *on* the die of my OCHI/multi-io controller. whats
that got to do with it?

and theres the other issue - if I made an ohci allocator for the SRAM
I'd have to partition off a small ammount ONLY for OHCI. with only 32K
to begin with thats a huge penalty for all the other devices in the
multi IO chip. a bus level DMA allocator solves this AND makes drivers
cleaner/more maintainable.

What more do you want?

Re: DMA API issues

[Tony Lindgren]

* Ian Molton <spyro@f2s.com> [040618 17:34]:
> On Fri, 18 Jun 2004 20:22:07 -0400
> Jeff Garzik <jgarzik@pobox.com> wrote:
> 
> > > my CPU can write directly to this 32K of SRAM. the chip can DMA from
> > > it.
> > 
> > 
> > Yes, write via MMIO.  Non-local RAM is not DMA.
> 
> I fail to see your point. this is NOT MMIO. in MMIO you write and the
> device reads as you write. in DMA, you write and then tell the chip to
> read from the memory you wrote to.

<snip>

I could possibly see some use for allowing multiple DMA domains in
addition to the system memory. Maybe something to allow overriding
the dma allocator per device? We could have optional dev->map() and
unmap(), and then use the dma-mapping API only to keep track of what's
allocated.

But I don't know how much sense that makes because there would be
only one driver using that DMA buffer. The DMA area would not be 
shared with other drivers like the system memory is.

So maybe the only advantage would be to allow more portable generic
device drivers.

Luckily I don't need this right now :)

Tony