Re: linux DMA capabilities in MV64460

Re: linux DMA capabilities in MV64460

[Mark A. Greer]

Hi Phil,

[Note: I'm cc'ing linuxppc-embedded for others to reference and to
add their thoughts.]
---

On Tue, Dec 20, 2005 at 10:49:58AM +1030, Phil Nitschke wrote:
> Hi Mark,
> 
> I'm developing a device driver to run in the 2.6.10 kernel.  I want to

That's a pretty old kernel.  Do you have the option of using a more
recent one like 2.6.14?

> get large amounts of data from a custom peripheral on the PCI bus.  The
> software is running on an Artesyn PmPPC7448, which includes a Discovery
> III bridge.

Can you share exact platform you're using?

> The custom device is a digitizer, which runs at over 200 million samples
> per second, so we're looking to use a fair amount of PCI bandwidth to
> get the data into main memory.  Burst reads/writes seem like the only
> way.
> 
> I have seen that you wrote some code for the mainstream kernel that sets
> up the mv64460.  One source told me:
> 
>   > In order to do PCI bursts, you'll need to use a DMA engine.  The
>   > MV64460 does contain a DMA engine, but you'd need to write a driver
>   > to access it.

That is not correct (assuming the quote is not out of context).
The bridge supports bursting on the PCI bus as long as the bridge is
configured correctly and the PCI device is making an appropriate request.
Note, however, that there are many errata for the Marvell parts including
some with cache coherency.  If your system is running with coherency on,
you may have to limit your bursts to 32 bytes (i.e., the size of one
cache line).

You can see how the bursting is set up on the bridge by looking at the
platform file for your board (e.g., <file:arch/ppc/platforms/katana.c> in
the latest linux kernel)--search for 'BURST'.

> Is there a summary of what is possible and/or not possible with the 4
> IDMA channels on the mv64460?

The only real documentation is the bridge's user manual from Marvell.
Unfortunately, you must sign an NDA to get access to it so I can't share
mine with you.  You will need access to that info to get very far so I
recommend you contact the people in your company that can make that
happen, ASAP.

> For example, if the device that I'm trying to get data from supported a
> DMA engine capable of initiating bursts on the PCI bus (it currently
> can't do this), does the current kernel code support that?

That's a hardware feature so its not really an issue of kernel support
other than ensuring that the firmware and/or kernel configures the bridge
correctly.  IOW, it can be supported by software but its an issue of
whether your hardware supports it (and it actually works).

> Or if I wanted to suck the data into main memory using the mv64460 IDMA
> controller (assuming the device couldn't initiate its own burst writes),
> is there a standard kernel interface to allow me to do this?

Yes.  You would make a "dma ctlr driver" for the dma ctlr(s).
I don't know what the best example would be but hopefully someone else
has a suggestion.

> Sorry if the questions are silly or inappropriately directed.

Not at all.  Those were good questions.

> At the
> moment I'm still trying to understand some really basic things, like
> what determines the difference between burst reads and ordinary reads,
> and what is the difference between SDMA and IDMA, etc ;-)

You may want to pick up "PCI System Architecture" from Mindshare, Inc.
There are ones for PCI-X and PCI-Express too, I think.  Well worth the money.

> Many thanks for any help/suggestions/information you can offer.
> 
> --
> Phil

I hope this helps,

Mark

Re: linux DMA capabilities in MV64460

[Brian Waite]

On Monday 19 December 2005 8:01 pm, Mark A. Greer wrote:
> >
> > up the mv64460.  One source told me:
> >   > In order to do PCI bursts, you'll need to use a DMA engine.  The
> >   > MV64460 does contain a DMA engine, but you'd need to write a driver
> >   > to access it.
>
> That is not correct (assuming the quote is not out of context).
> The bridge supports bursting on the PCI bus as long as the bridge is
> configured correctly and the PCI device is making an appropriate request.
> Note, however, that there are many errata for the Marvell parts including
> some with cache coherency.
There are many many errata regarding cache coherency. Also, the PCI bandwidth 
the bridge is capable of with coherency enabled is very small. You will most 
likely need to use sw coherency for any real speed.
> If your system is running with coherency on, 
> you may have to limit your bursts to 32 bytes (i.e., the size of one
> cache line).
You will have to limit your self to 32 byte bursts with coherency. This is a 
requirement not an errata. 

> > Is there a summary of what is possible and/or not possible with the 4
> > IDMA channels on the mv64460?
>
> The only real documentation is the bridge's user manual from Marvell.
> Unfortunately, you must sign an NDA to get access to it so I can't share
> mine with you.  You will need access to that info to get very far so I
> recommend you contact the people in your company that can make that
> happen, ASAP.
Ask for the errata sheets while your at it. You WILL need them :)
You really do need the chipset docs to do anything intelligent with the IDMA 
controllers.

Thanks
Brian

Re: linux DMA capabilities in MV64460

[Mark A. Greer]

On Tue, Dec 20, 2005 at 09:27:35AM -0500, Brian Waite wrote:
> On Monday 19 December 2005 8:01 pm, Mark A. Greer wrote:
> > >
> > > up the mv64460.  One source told me:
> > >   > In order to do PCI bursts, you'll need to use a DMA engine.  The
> > >   > MV64460 does contain a DMA engine, but you'd need to write a driver
> > >   > to access it.
> >
> > That is not correct (assuming the quote is not out of context).
> > The bridge supports bursting on the PCI bus as long as the bridge is
> > configured correctly and the PCI device is making an appropriate request.
> > Note, however, that there are many errata for the Marvell parts including
> > some with cache coherency.
> There are many many errata regarding cache coherency. Also, the PCI bandwidth 
> the bridge is capable of with coherency enabled is very small. You will most 
> likely need to use sw coherency for any real speed.

I've managed to get reasonable speed with coherency on but you have to
jack the burst sizes to the max (see my comment below).  The speed was
hugely affected by the burst size; coherency enabled had only a minor impact.
The problem for me, however, was the board(s) I have do not have the
necessary hw coherency errata workarounds implemented so the bridge
eventually hangs with coherency enabled.

Unfortunately, I think I deleted the file with my performance
results in one of my cleanup binges.  IIRC, though, I got ~750 Mbps
with coherency off and ~725 Mbps with it on using an e1000 & a 750fx
or gx clocked around 800 MHz.

> > If your system is running with coherency on, 
> > you may have to limit your bursts to 32 bytes (i.e., the size of one
> > cache line).
> You will have to limit your self to 32 byte bursts with coherency. This is a 
> requirement not an errata. 

That is only true for the 64360.  The 64460 does not have that
restriction AFAICT.

<snip>

Mark

Re: linux DMA capabilities in MV64460

[Phil Nitschke]

>>>>> "MAG" == Mark A Greer <mgreer@mvista.com> writes:

  MAG> Hi Phil,
  MAG> [Note: I'm cc'ing linuxppc-embedded for others to reference and to
  MAG> add their thoughts.]

OK, I've just subscribed...

  MAG> On Tue, Dec 20, 2005 at 10:49:58AM +1030, Phil Nitschke wrote:
  >> Hi Mark,
  >>
  >> I'm developing a device driver to run in the 2.6.10 kernel.  I want to

  MAG> That's a pretty old kernel.  Do you have the option of using a more
  MAG> recent one like 2.6.14?

That might be possible if I reverse-engineer a patch file by comparing
the Artesyn reference kernel (2.6.10) with the kernel.org version, then
trying to apply that patch to the latest kernel.  I'll try this later...

  >> get large amounts of data from a custom peripheral on the PCI bus.  The
  >> software is running on an Artesyn PmPPC7448, which includes a Discovery
  >> III bridge.

  MAG> Can you share exact platform you're using?

I'm using a PMC processor on a custom carrier card (not made by Avalon).
Here are the respective links:

  Carrier:   http://www.tenix.com.au/Main.asp?ID=938
  Processor: http://www.artesyncp.com/products/PmPPC7448.html

  MAG> The bridge supports bursting on the PCI bus as long as the bridge
  MAG> is configured correctly and the PCI device is making an
  MAG> appropriate request.  Note, however, that there are many errata
  MAG> for the Marvell parts including some with cache coherency.  If
  MAG> your system is running with coherency on, you may have to limit
  MAG> your bursts to 32 bytes (i.e., the size of one cache line).

  MAG> You can see how the bursting is set up on the bridge by looking
  MAG> at the platform file for your board (e.g.,
  MAG> <file:arch/ppc/platforms/katana.c> in the latest linux
  MAG> kernel)--search for 'BURST'.

As far as I can tell, there is no platform file for this board in the
mainstream kernel.

In the reference kernel provided by Artesyn, there is a file named
arch/ppc/configs/pmppc7447_defconfig, where CONFIG_NOT_COHERENT_CACHE=y

Therefore in arch/ppc/platforms/pmppc7447.c, there is some code which
does this:

#if defined(CONFIG_NOT_COHERENT_CACHE)
        mv64x60_write(&bh, MV64360_SRAM_CONFIG, 0x00160000);
#else
        mv64x60_write(&bh, MV64360_SRAM_CONFIG, 0x001600b2);
#endif

... and later ...

        for (i = 0; i < MV64x60_CPU2MEM_WINDOWS; i++) {
#if defined(CONFIG_NOT_COHERENT_CACHE)
                si.cpu_prot_options[i] = 0;
                si.enet_options[i] = MV64360_ENET2MEM_SNOOP_NONE;
                si.mpsc_options[i] = MV64360_MPSC2MEM_SNOOP_NONE;
                si.idma_options[i] = MV64360_IDMA2MEM_SNOOP_NONE;
                si.pci_0.acc_cntl_options[i] =
                    MV64360_PCI_ACC_CNTL_SNOOP_NONE |
                    MV64360_PCI_ACC_CNTL_SWAP_NONE |
                    MV64360_PCI_ACC_CNTL_MBURST_128_BYTES |
                    MV64360_PCI_ACC_CNTL_RDSIZE_256_BYTES;
#else
                si.cpu_prot_options[i] = 0;
                si.enet_options[i] = MV64360_ENET2MEM_SNOOP_NONE;       /* errata */
                si.mpsc_options[i] = MV64360_MPSC2MEM_SNOOP_NONE;       /* errata */
                si.idma_options[i] = MV64360_IDMA2MEM_SNOOP_NONE;       /* errata */
                si.pci_0.acc_cntl_options[i] =
                    MV64360_PCI_ACC_CNTL_SNOOP_WB |
                    MV64360_PCI_ACC_CNTL_SWAP_NONE |
                    MV64360_PCI_ACC_CNTL_MBURST_32_BYTES |
                    MV64360_PCI_ACC_CNTL_RDSIZE_32_BYTES;
#endif
        }

But I'm yet to learn what all this means...

  >> Is there a summary of what is possible and/or not possible with the 4
  >> IDMA channels on the mv64460?

  MAG> The only real documentation is the bridge's user manual from Marvell.
  MAG> Unfortunately, you must sign an NDA to get access to it so I can't share
  MAG> mine with you.  You will need access to that info to get very far so I
  MAG> recommend you contact the people in your company that can make that
  MAG> happen, ASAP.

I talked with a person from Marvell's only Australian distributor, who
told me that they'd not be too keen to give us an NDA, since we're not
developing a project specifically for the Marvell, rather we're using a
Marvell which has already been integrated in the Artesyn card.
Therefore, he argued, Marvell would tell me to go to Artesyn for the
info, as they already have the NDA.  So for now, assume no NDA, no errata.

  >> For example, if the device that I'm trying to get data from supported a
  >> DMA engine capable of initiating bursts on the PCI bus (it currently
  >> can't do this), does the current kernel code support that?

  MAG> That's a hardware feature so its not really an issue of kernel support
  MAG> other than ensuring that the firmware and/or kernel configures the bridge
  MAG> correctly.  IOW, it can be supported by software but its an issue of
  MAG> whether your hardware supports it (and it actually works).

I'm not sure here whether you're talking about the hardware in the
CPU/bridge, or the hardware in the device.  Since the device interfaces
to the PCI bus using firmware inside an FPGA, this is configurable (to a
certain extent).

Currently there is a 2M aperture on the device, but it is not being seen
as "prefetchable", so when I try to get data from the device using
repetitive reads, they are very slow.  Hence my efforts to get DMA
happening.

Presumably the CPU/bridge discovers PCI device memory regions during bus
enumeration.  What characteristic of a device determines whether the
memory region is going to be marked as "prefetchable"?

Does this attribute also affect whether DMA will work?

  >> Or if I wanted to suck the data into main memory using the mv64460 IDMA
  >> controller (assuming the device couldn't initiate its own burst writes),
  >> is there a standard kernel interface to allow me to do this?

  MAG> Yes.  You would make a "dma ctlr driver" for the dma ctlr(s).  I
  MAG> don't know what the best example would be but hopefully someone
  MAG> else has a suggestion.

OK, I'll look into this.  I've been using the O'Reilly book "Linux
Device Drivers, Third Edition" by Jonathan Corbet, Alessandro Rubini,
and Greg Kroah-Hartman.  They say "The kernel developers recommend the
use of streaming mappings over coherent mappings whenever possible."

I'm not sure how the H/W vs S/W coherency discussion has anything to do
with their assertion.  I had previously thought that allocating a huge
buffer (for example at boot time) would be the way to go, but perhaps
getting the CPU to collect the data in smaller amounts into cache
coherent memory will give me the best performance?

  MAG> You may want to pick up "PCI System Architecture" from Mindshare,
  MAG> Inc.  There are ones for PCI-X and PCI-Express too, I think.
  MAG> Well worth the money.

Sounds like a good idea.  I'd hoped not to have to become a PCI expert,
but it seems that there is a lot for me to learn just to determine how
best to design my driver.

Thanks for your input.

--
Phil

Re: linux DMA capabilities in MV64460

[KokHow Teh]

>Currently there is a 2M aperture on the device, but it is not being seen
>as "prefetchable", so when I try to get data from the device using
>repetitive reads, they are very slow.  Hence my efforts to get DMA
>happening.

>Presumably the CPU/bridge discovers PCI device memory regions during bus
>enumeration.  What characteristic of a device determines whether the
>memory region is going to be marked as "prefetchable"?

Being "prefetchable" or not is determined by bit-3 of PCI Memory BAR.

>Does this attribute also affect whether DMA will work?


  MAG> You may want to pick up "PCI System Architecture" from Mindshare,
  MAG> Inc.  There are ones for PCI-X and PCI-Express too, I think.
  MAG> Well worth the money.

>Sounds like a good idea.  I'd hoped not to have to become a PCI expert,
>but it seems that there is a lot for me to learn just to determine how
>best to design my driver.

Here is a good online reference but it does not cover dma and
cache-coherency in great details.
http://www.tldp.org/LDP/tlk/dd/pci.html

Re: linux DMA capabilities in MV64460

[Phil Nitschke]

>>>>> "MAG" == Mark A Greer <mgreer@mvista.com> writes:
  MAG> On Tue, Dec 20, 2005 at 10:49:58AM +1030, Phil Nitschke wrote:

  >> ... if I wanted to suck the data into main memory using the mv64460
  >> IDMA controller (assuming the device couldn't initiate its own
  >> burst writes), is there a standard kernel interface to allow me to
  >> do this?

  MAG> Yes.  You would make a "dma ctlr driver" for the dma ctlr(s).  I
  MAG> don't know what the best example would be but hopefully someone
  MAG> else has a suggestion.

I'm following up to a thread started in December last year.  The thread
was talking about this device:
  http://www.marvell.com/products/communication/discoveryIII/index.jsp
  http://www.marvell.com/products/communication/Discovery%20MV64460%20FINAL.pdf

Currently I've found the linux kernel has the following files relevant
to this hardware (2.6.15 kernel):

    ./arch/ppc/syslib/mv64360_pic.c
    ./arch/ppc/syslib/mv64x60.c
    ./arch/ppc/syslib/mv64x60_dbg.c
    ./arch/ppc/syslib/mv64x60_win.c
    ./drivers/char/watchdog/mv64x60_wdt.c
    ./drivers/i2c/busses/i2c-mv64xxx.c
    ./drivers/net/mv643xx_eth.c

I'm now looking seriously (and reluctantly!) at writing a DMA Controller
Driver to supplement these functions, and I've started the process of
getting an NDA from Marvell, so I can get their User Manual (and errata!).

Can someone point me in the right direction to get me started?  I need
to come up the learning curve to find out where to start.  

How is a DMA controlled (from a device driver writer's perspective) when
a third-party (i.e. in the bridge) DMA controller needs to do the work
to get the data from a PCI Target into main memory?

What kernel API should be provided by the DMA Controller Driver?  

Any documentation, examples, similar device drivers, etc, would be
appreciated.  TIA,

--
Phil

Re: linux DMA capabilities in MV64460

[Brian Waite]

> I'm now looking seriously (and reluctantly!) at writing a DMA Controller
> Driver to supplement these functions, and I've started the process of
> getting an NDA from Marvell, so I can get their User Manual (and errata!).
>
You won't get very far without the user manual, then I think you will find it 
pretty easy to write the driver with the frameworks already in the kernel.
> Can someone point me in the right direction to get me started?  I need
> to come up the learning curve to find out where to start.
>
> How is a DMA controlled (from a device driver writer's perspective) when
> a third-party (i.e. in the bridge) DMA controller needs to do the work
> to get the data from a PCI Target into main memory?
>
> What kernel API should be provided by the DMA Controller Driver?
>
My first guess would be to follow something like Documentation/DMA-API.txt and 
Documentation/DMA-mapping.txt

> Any documentation, examples, similar device drivers, etc, would be
> appreciated.  TIA,
You could look to followup that by groking arch/ppc/kernel/dma-mapping.c
Finally, arch/ppc/syslib ppc4xx_dma.c seems to show an example of a low level 
driver. 
I didn't notice any platform dma controllers like MAG reccommended, but that 
should be a better way to go. 

Thanks
Brian

Re: linux DMA capabilities in MV64460

[Phil Nitschke]

>>>>> "BW" == Brian Waite <brian@waitefamily.us> writes:

  >> I'm now looking seriously (and reluctantly!) at writing a DMA
  >> Controller Driver to supplement these functions, and I've started
  >> the process of getting an NDA from Marvell, so I can get their
  >> User Manual (and errata!).

  BW> You won't get very far without the user manual, then I think you
  BW> will find it pretty easy to write the driver with the frameworks
  BW> already in the kernel.

  >> Can someone point me in the right direction to get me started?  I
  >> need to come up the learning curve to find out where to start.
  >> 
  >> How is a DMA controlled (from a device driver writer's
  >> perspective) when a third-party (i.e. in the bridge) DMA
  >> controller needs to do the work to get the data from a PCI Target
  >> into main memory?
  >> 
  >> What kernel API should be provided by the DMA Controller Driver?

  BW> My first guess would be to follow something like
  BW> Documentation/DMA-API.txt and Documentation/DMA-mapping.txt

I can't see how trying to match those DMA functions would work.  

The dma_map_xxx() functions and friends appear to be invoked from
driver interrupt handlers, which are either responding to an interrupt
from a PCI Target which wants to commence a bus-mastered DMA to/from
main memory, or responding to a second interrupt to say the interrupt
is complete.

In my case, my driver receives one interrupt from a PCI Target which
cannot perform a DMA, and I presumably have to use an alternative API
(i.e. my "new" set of ("DMA Controller") driver functions) to get the
platform DMA controller to fetch the data and wake (interrupt) me once
complete.  Right?

  >> Any documentation, examples, similar device drivers, etc, would
  >> be appreciated.  TIA,

  BW> You could look to followup that by groking
  BW> arch/ppc/kernel/dma-mapping.c

OK.

  BW> Finally, arch/ppc/syslib ppc4xx_dma.c seems to show an example
  BW> of a low level driver.  

OK, so suppose I write a bunch of functions, like this:

    mv64x60_disable_dma();
    mv64x60_disable_dma_interrupt();
    mv64x60_enable_dma();
    mv64x60_enable_dma_interrupt();
    mv64x60_get_dma_status();
    mv64x60_init_dma_channel();
    ... etc,

then how do these get called; directly from my other driver?  

(I could not see any place that the ppc4xx_ dma functions were being
called, i.e. they don't seem to dovetail into some higher level kernel
API...?)

  BW> I didn't notice any platform dma controllers like MAG
  BW> reccommended, but that should be a better way to go.

Perhaps Mark was talking about ppc4xx_dma.c and ppc4xx_sgdma.c ?

Sorry if I'm too slow on the uptake.  Thanks for your input,

-- 
Phil

Re: linux DMA capabilities in MV64460

[Adrian Cox]

On Mon, 2006-03-06 at 14:39 +1030, Phil Nitschke wrote:
> How is a DMA controlled (from a device driver writer's perspective) when
> a third-party (i.e. in the bridge) DMA controller needs to do the work
> to get the data from a PCI Target into main memory?
> 
> What kernel API should be provided by the DMA Controller Driver?  

There is no current API for this in the kernel, but there are some
proposals. From Intel, we have I/OAT, which targets network operations:
http://lkml.org/lkml/2006/3/3/219

There's some overlap with the ADMA feature set, which is intended to
accelerate RAID operations:
http://lkml.org/lkml/2006/2/2/442


-- 
Adrian Cox <adrian@humboldt.co.uk>

Re: linux DMA capabilities in MV64460

[Phil Nitschke]

On Wed, 2006-03-08 at 22:36 +0000, Adrian Cox wrote:
> On Mon, 2006-03-06 at 14:39 +1030, Phil Nitschke wrote:
> > How is a DMA controlled (from a device driver writer's perspective) when
> > a third-party (i.e. in the bridge) DMA controller needs to do the work
> > to get the data from a PCI Target into main memory?
> > 
> > What kernel API should be provided by the DMA Controller Driver?  
> 
> There is no current API for this in the kernel, but there are some
> proposals. From Intel, we have I/OAT, which targets network operations:
> http://lkml.org/lkml/2006/3/3/219
> 
> There's some overlap with the ADMA feature set, which is intended to
> accelerate RAID operations:
> http://lkml.org/lkml/2006/2/2/442
> 

Thanks, Adrian, these are the sorts of APIs I was asking about.  At a
quick glance, they look a little "bleeding edge" for my comfort zone.
(And not an exact match for what I'm trying to do...)  Nonetheless, I'll
take a closer look at them.

Thanks,

-- 
Phil Nitschke <Phil.Nitschke@avalon.com.au>
Avalon Systems Pty Ltd