pmppc7448/mv64x60 DMA from PCI to memory

pmppc7448/mv64x60 DMA from PCI to memory

[Phil Nitschke]

Hi,

I'm working on a project using this processor:
http://www.artesyncp.com/products/PmPPC7448.html
on a custom VME carrier, as shown below.  We're wanting to suck
large amounts of data from a PCI device which _cannot_ perform
bus-mastered DMA (it is a PCI Target only).

The Marvell Chip used by the PmPPC7448 is this one:
  http://www.marvell.com/products/communication/Discovery%20MV64460%
20FINAL.pdf

I've written a collection of simple routines to program the Marvell IDMA
controller, for example:
    mv64x6x_init_dma_channel();
    mv64x6x_set_dma_mode();
    mv64x6x_set_src_addr();
    mv64x6x_set_dst_addr();
    mv64x6x_set_dma_count();
    mv64x6x_enable_dma();
or rather more simply:
    mv64x6x_memcpy_dma(dst_handle, src_handle, size);

This works OK for copying from a memory to memory, where the buffers are
allocated using:
    src = dma_alloc_noncoherent(NULL, BUF_SZ, &src_handle, GFP_KERNEL);
The src_handle is passed directly to mv64x6x_set_src_addr();

But when the src address is the FIFO on the PCI bus, I don't know how to
get the IDMA controller to play nicely.  The FIFO sits in the middle of
the PCI device's I/O mem range 0x9fe00000 - 0x9fffffff.  I've programmed
and enabled a 5th address window in the IDMA controller which
encompasses the 0x200000 bytes of the PCI memory range, and I'm not
seeing any address violation or address miss errors.  The PCI->memory
DMA "completes" without any traffic every touching the PCI bus, so
obviously I need to do something else/differently.

For this scenario, can anyone tell me:
        * Should I be using the same src address as that reported via
        the 'lspci' command - this _is_ the PCI bus address, isn't it?
        
        * Do I have to do anything special to tell the IDMA controller
        to source data from the PCI bus and shift it into memory?
        
        * Looking through mv64x60.c in the 2.6.16 kernel, I note that 4
        of the 8 possible IDMA address windows are configured (for each
        of the 512 MB DRAM on our processor card).  Do I need to add
        tests to my source and destination regions, to determine if they
        cross one of the 512 MB regions, and hence will require a
        different CSx line (and thus the DMA will need to be broken into
        two transactions), or does kernel already take care to ensure
        allocated regions will not cross these boundaries?
        
TIA for any help that anyone can offer.

-- 
Phil

       +--------------------------------------------------+
       |            Custom VME64x Carrier Card            |
       |  +-------------------+    +-------------------+  |
       |  | Artesyn PmPPC7448 |    | Altera FPGA       |  |
       |  |                   |    |                   |  |
       |  |+-----------------+|    |+-----------------+|  |
       |  ||Marvell MV64460  ||    || Altera PCI I/F  ||  |
       |  ||(NOT coherent    ||    ||(non-prefetchable)|  |
       |  || cache)  (+IDMA) ||    ||       FIFO      ||  |
       |  |+--------#--------+|    |+---------#-------+|  |
       |  +---------#---------+    +----------#--------+  |
       |            #         PCI Bus         #           |
       |   ###########################################    |
       +--------------------------------------------------+

p.s. Since my driver was developed using documentation obtained from
Marvell under a very restrictive NDA, I cannot release it as open source
just yet.  Marvell's Vice President and General Counsel is currently
reviewing the matter, and I expect to hear from them in the next couple
weeks.

Re: pmppc7448/mv64x60 DMA from PCI to memory

[Mark A. Greer]

Hi Phil,

On Wed, May 24, 2006 at 12:20:04AM +0930, Phil Nitschke wrote:
> I've written a collection of simple routines to program the Marvell IDMA
> controller, for example:
>     mv64x6x_init_dma_channel();
>     mv64x6x_set_dma_mode();
>     mv64x6x_set_src_addr();
>     mv64x6x_set_dst_addr();
>     mv64x6x_set_dma_count();
>     mv64x6x_enable_dma();
> or rather more simply:
>     mv64x6x_memcpy_dma(dst_handle, src_handle, size);

Listing routine names but no code doesn't really help.

> This works OK for copying from a memory to memory, where the buffers are
> allocated using:
>     src = dma_alloc_noncoherent(NULL, BUF_SZ, &src_handle, GFP_KERNEL);
> The src_handle is passed directly to mv64x6x_set_src_addr();
> 
> But when the src address is the FIFO on the PCI bus, I don't know how to

Do you really mean a FIFO?  If so, that would explain a lot...

> get the IDMA controller to play nicely.  The FIFO sits in the middle of
> the PCI device's I/O mem range 0x9fe00000 - 0x9fffffff.  I've programmed
> and enabled a 5th address window in the IDMA controller which
> encompasses the 0x200000 bytes of the PCI memory range, and I'm not
> seeing any address violation or address miss errors.  The PCI->memory
> DMA "completes" without any traffic every touching the PCI bus, so
> obviously I need to do something else/differently.

You say that you don't see any PCI traffic.  Does that mean you
have a PCI analyzer and that you are sure that its set up correctly?
If so, then you have something botched in your IDMA->PCI window setup
or in the pgming of the DMA itself (e.g., in your descriptor(s)).

Also, set the SrcHold bit [3] of the channel control reg (low).
If its really a FIFO, you are--or will be once you get your windows
and descriptors set up correctly--reading the FIFO once then
incrementing past it.

> For this scenario, can anyone tell me:
>         * Should I be using the same src address as that reported via
>         the 'lspci' command - this _is_ the PCI bus address, isn't it?

"man lspci" (read up on the '-b' option)

>         * Do I have to do anything special to tell the IDMA controller
>         to source data from the PCI bus and shift it into memory?

You're talking to a *FIFO* which means you read all the data from one
address and write all the data to one, probably different, address.
Its not normal PCI memory, you don't increment through it.  Its a
register of a FIFO that's in PCI memory space.  That's why you need
to tell the IDMA ctlr to "hold" and read/write to the same address
each time.

>         * Looking through mv64x60.c in the 2.6.16 kernel, I note that 4
>         of the 8 possible IDMA address windows are configured (for each
>         of the 512 MB DRAM on our processor card).

No they aren't.  They're configured (or not) according to the setup info
that you pass in.

> Do I need to add
>         tests to my source and destination regions, to determine if they
>         cross one of the 512 MB regions, and hence will require a
>         different CSx line (and thus the DMA will need to be broken into
>         two transactions), or does kernel already take care to ensure
>         allocated regions will not cross these boundaries?

No.  You need to do what's appropriate for the hardware that you are
essentially writing a driver for.  YOU are supposed to know what the
limitations of your hardware are.  Even if you don't have a manual... ;)

Mark

Re: pmppc7448/mv64x60 DMA from PCI to memory

[Phil Nitschke]

On Tue, 2006-05-23 at 16:54 -0700, Mark A. Greer wrote:

> You say that you don't see any PCI traffic.  Does that mean you
> have a PCI analyzer and that you are sure that its set up correctly?

I don't have a PCI analyzer, however the JTAG used to program the PCI
device has been configured to display 4 K samples of PCI bus signals
(about 20 microsecs?) around the time of an interrupt which results in
the DMA being requested.  Since my last post, I have managed to see some
traffic, but the PCI STOP# line is asserted, so I'm not seeing any data
being read.  I'll investigate further...

> If so, then you have something botched in your IDMA->PCI window setup

Quite possibly.  The patch below shows how I've set this window up.  It
is not intended as a 'final' piece of code, so please forgive the magic
numbers.  Could you review this for me?

> or in the pgming of the DMA itself (e.g., in your descriptor(s)).

Well the memory to memory DMA is working OK.  I just didn't know what
the correct procedure was for determining the bus address of the FIFO.
For example, this mapping returns a dma handle which does not work:

    fifo_dma_handle = pci_map_single(dev, my_card.bar1+fifo_address[0],
                                     FIFO_SIZE, PCI_DMA_FROMDEVICE);

Whereas without DMA I would just use this:
    ioread32_rep(my_card.bar1 + fifo_address[0], buf, 6);

Was I misguided in trying to use pci_map_single in this way?

> Also, set the SrcHold bit [3] of the channel control reg (low).
> If its really a FIFO, you are--or will be once you get your windows
> and descriptors set up correctly--reading the FIFO once then
> incrementing past it.

I can either address it as a FIFO, or as a memory range.  I can read
from any address in the range and it returns the "next FIFO value".
Anyway, I've tried both src address hold settings...

> > For this scenario, can anyone tell me:
> >         * Should I be using the same src address as that reported via
> >         the 'lspci' command - this _is_ the PCI bus address, isn't it?
> 
> "man lspci" (read up on the '-b' option)

I cannot see any difference with the '-b' flag.  Maybe that is the way
of things on my architecture?

> >         * Looking through mv64x60.c in the 2.6.16 kernel, I note that 4
> >         of the 8 possible IDMA address windows are configured (for each
> >         of the 512 MB DRAM on our processor card).
> 
> No they aren't.  They're configured (or not) according to the setup info
> that you pass in.

OK.  I also note there are several cases where this is used in
mv64x60.c:

        for (i=0; i<3; i++)

Why is 3 used in these loops, and not some other constant like
MV64360_xxxxx_WINDOWS (which are usually 4, not 3)?

> > Do I need to add
> >         tests to my source and destination regions, to determine if they
> >         cross one of the 512 MB regions, and hence will require a
> >         different CSx line (and thus the DMA will need to be broken into
> >         two transactions), or does kernel already take care to ensure
> >         allocated regions will not cross these boundaries?
> 
> No.  You need to do what's appropriate for the hardware that you are
> essentially writing a driver for.  YOU are supposed to know what the
> limitations of your hardware are.  

OK, I know how my hardware is configured, but when trying to write a
generic driver, perhaps I need to have the mv64x60.c code remember the
CSx barriers, e.g. in the mv64x60_chip_info, so the IDMA engine can
access it.  Do you think this would be possible/beneficial?

Thanks again,

-- 
Phil

Here is the patch to configure IDMA to PCI window(s):

--- linux-2.6.16/arch/ppc/syslib/mv64x60.c      2006-03-20
16:23:29.000000000 +1030
+++ linux-2.6.16-patched/arch/ppc/syslib/mv64x60.c      2006-05-23
16:33:52.000000000 +0930
@@ -535,6 +581,7 @@
                mv64x60_config_pci_params(bh->hose_a, &si->pci_0);

                mv64x60_config_cpu2pci_windows(bh, &si->pci_0, 0);
+                mv64x60_config_idma2pci_windows(bh, &si->pci_0, 0);
                mv64x60_config_pci2mem_windows(bh, bh->hose_a,
&si->pci_0, 0,
                        mem_windows);
                bh->ci->set_pci2regs_window(bh, bh->hose_a, 0,
@@ -548,6 +595,7 @@
                mv64x60_config_pci_params(bh->hose_b, &si->pci_1);

                mv64x60_config_cpu2pci_windows(bh, &si->pci_1, 1);
+                mv64x60_config_idma2pci_windows(bh, &si->pci_1, 1);
                mv64x60_config_pci2mem_windows(bh, bh->hose_b,
&si->pci_1, 1,
                        mem_windows);
                bh->ci->set_pci2regs_window(bh, bh->hose_b, 1,
@@ -1136,6 +1188,42 @@
                        bh->ci->disable_window_32bit(bh, win_tab[bus][i
+1]);
 }

+static u32 idma_tab_xtra[MV64x60_CPU2MEM_WINDOWS] __initdata = {
+        MV64x60_IDMA2MEM_0_WIN, MV64x60_IDMA2MEM_1_WIN,
+        MV64x60_IDMA2MEM_2_WIN, MV64x60_IDMA2MEM_3_WIN,
+};
+
+void __init
+mv64x60_config_idma2pci_windows(struct mv64x60_handle *bh,
+        struct mv64x60_pci_info *pi, u32 bus)
+{
+        u32     attributes, unit_id;
+        int     i;
+
+       /* Target Unit IDs: PCI0 = 3, PCI1 = 4. */
+       unit_id = bus ? 0x4 : 0x3;
+       /* 0x1d == No swap data, PCI-X NS attribute asserted, PCI memory
+        * space, PCIx_REQ64n asserted according to requested data size.
*/
+       attributes = (0x1d << 8) | unit_id;
+
+        for (i=0; i<3; i++)
+                if (pi->pci_mem[i].size > 0) {
+                        mv64x60_set_32bit_window(bh, idma_tab_xtra[i],
+                                pi->pci_mem[i].cpu_base,
pi->pci_mem[i].size,
+                                attributes);
+                        bh->ci->enable_window_32bit(bh,
idma_tab_xtra[i]);
+                        /* Give idma r/w access to PCI memory region */
+                        mv64x60_set_bits(bh,
MV64360_IDMA2MEM_ACC_PROT_0,
+                                (0x3 << (i << 1)));
+                        mv64x60_set_bits(bh,
MV64360_IDMA2MEM_ACC_PROT_1,
+                                (0x3 << (i << 1)));
+                        mv64x60_set_bits(bh,
MV64360_IDMA2MEM_ACC_PROT_2,
+                                (0x3 << (i << 1)));
+                        mv64x60_set_bits(bh,
MV64360_IDMA2MEM_ACC_PROT_3,
+                                (0x3 << (i << 1)));
+                }
+}
+
 /*

*****************************************************************************
  *
@@ -2220,8 +2311,8 @@
 };

 static u32 idma_tab[MV64x60_CPU2MEM_WINDOWS] __initdata = {
-       MV64x60_IDMA2MEM_0_WIN, MV64x60_IDMA2MEM_1_WIN,
-       MV64x60_IDMA2MEM_2_WIN, MV64x60_IDMA2MEM_3_WIN,
+        MV64x60_IDMA2MEM_4_WIN, MV64x60_IDMA2MEM_5_WIN,
+        MV64x60_IDMA2MEM_6_WIN, MV64x60_IDMA2MEM_7_WIN,
 };

 static u32 dram_selects[MV64x60_CPU2MEM_WINDOWS] __initdata =
@@ -2285,13 +2376,13 @@

                        /* Give idma r/w access to memory region */
                        mv64x60_set_bits(bh,
MV64360_IDMA2MEM_ACC_PROT_0,
-                               (0x3 << (i << 1)));
+                                (0x3 << ((i+4) << 1)));
                        mv64x60_set_bits(bh,
MV64360_IDMA2MEM_ACC_PROT_1,
-                               (0x3 << (i << 1)));
+                                (0x3 << ((i+4) << 1)));
                        mv64x60_set_bits(bh,
MV64360_IDMA2MEM_ACC_PROT_2,
-                               (0x3 << (i << 1)));
+                                (0x3 << ((i+4) << 1)));
                        mv64x60_set_bits(bh,
MV64360_IDMA2MEM_ACC_PROT_3,
-                               (0x3 << (i << 1)));
+                                (0x3 << ((i+4) << 1)));
                }
 }

Re: pmppc7448/mv64x60 DMA from PCI to memory

[Mark A. Greer]

On Wed, May 24, 2006 at 11:53:54AM +0930, Phil Nitschke wrote:
> On Tue, 2006-05-23 at 16:54 -0700, Mark A. Greer wrote:
> 
> > You say that you don't see any PCI traffic.  Does that mean you
> > have a PCI analyzer and that you are sure that its set up correctly?
> 
> I don't have a PCI analyzer, however the JTAG used to program the PCI
> device has been configured to display 4 K samples of PCI bus signals
> (about 20 microsecs?) around the time of an interrupt which results in
> the DMA being requested.  Since my last post, I have managed to see some
> traffic, but the PCI STOP# line is asserted, so I'm not seeing any data
> being read.  I'll investigate further...

OK

> > If so, then you have something botched in your IDMA->PCI window setup
> 
> Quite possibly.  The patch below shows how I've set this window up.  It
> is not intended as a 'final' piece of code, so please forgive the magic
> numbers.  Could you review this for me?

Sure.

> > or in the pgming of the DMA itself (e.g., in your descriptor(s)).
> 
> Well the memory to memory DMA is working OK.  I just didn't know what
> the correct procedure was for determining the bus address of the FIFO.
> For example, this mapping returns a dma handle which does not work:
> 
>     fifo_dma_handle = pci_map_single(dev, my_card.bar1+fifo_address[0],
>                                      FIFO_SIZE, PCI_DMA_FROMDEVICE);
>
> Whereas without DMA I would just use this:
>     ioread32_rep(my_card.bar1 + fifo_address[0], buf, 6);
> 
> Was I misguided in trying to use pci_map_single in this way?

Yes.  Use pci_map_single() to map an already allocated, physically
contiguous memory buffer not pci memory.  Try pci_iomap() instead.

You should read Documentation/DMA-API.txt & DMA-mapping.txt thoroughly
then look at the many pci drivers that already exist as examples.

> > Also, set the SrcHold bit [3] of the channel control reg (low).
> > If its really a FIFO, you are--or will be once you get your windows
> > and descriptors set up correctly--reading the FIFO once then
> > incrementing past it.
> 
> I can either address it as a FIFO, or as a memory range.  I can read
> from any address in the range and it returns the "next FIFO value".
> Anyway, I've tried both src address hold settings...

OK

> > > For this scenario, can anyone tell me:
> > >         * Should I be using the same src address as that reported via
> > >         the 'lspci' command - this _is_ the PCI bus address, isn't it?
> > 
> > "man lspci" (read up on the '-b' option)
> 
> I cannot see any difference with the '-b' flag.  Maybe that is the way
> of things on my architecture?

That probably means that your pci io & mem space are mapped at the same
addrs in pci io & mem space and phys cpu space.  If that's not the case,
something may be wrong.

> > >         * Looking through mv64x60.c in the 2.6.16 kernel, I note that 4
> > >         of the 8 possible IDMA address windows are configured (for each
> > >         of the 512 MB DRAM on our processor card).
> > 
> > No they aren't.  They're configured (or not) according to the setup info
> > that you pass in.
> 
> OK.  I also note there are several cases where this is used in
> mv64x60.c:
> 
>         for (i=0; i<3; i++)
> 
> Why is 3 used in these loops, and not some other constant like
> MV64360_xxxxx_WINDOWS (which are usually 4, not 3)?

Different things.  The "i<3;" are when looping through windows that are 
related to a struct pci_controller's mem_resource.  From the definition
of pci_controller:

	struct resource mem_resources[3];

> > > Do I need to add
> > >         tests to my source and destination regions, to determine if they
> > >         cross one of the 512 MB regions, and hence will require a
> > >         different CSx line (and thus the DMA will need to be broken into
> > >         two transactions), or does kernel already take care to ensure
> > >         allocated regions will not cross these boundaries?
> > 
> > No.  You need to do what's appropriate for the hardware that you are
> > essentially writing a driver for.  YOU are supposed to know what the
> > limitations of your hardware are.  
> 
> OK, I know how my hardware is configured, but when trying to write a
> generic driver, perhaps I need to have the mv64x60.c code remember the
> CSx barriers, e.g. in the mv64x60_chip_info, so the IDMA engine can
> access it.  Do you think this would be possible/beneficial?

No.  Just set up and enable an IDMA window to access all of pci mem space
and be done with it.

> Thanks again,
> 
> -- 
> Phil
> 
> Here is the patch to configure IDMA to PCI window(s):
> 
> --- linux-2.6.16/arch/ppc/syslib/mv64x60.c      2006-03-20
> 16:23:29.000000000 +1030
> +++ linux-2.6.16-patched/arch/ppc/syslib/mv64x60.c      2006-05-23
> 16:33:52.000000000 +0930
> @@ -535,6 +581,7 @@
>                 mv64x60_config_pci_params(bh->hose_a, &si->pci_0);
> 
>                 mv64x60_config_cpu2pci_windows(bh, &si->pci_0, 0);
> +                mv64x60_config_idma2pci_windows(bh, &si->pci_0, 0);
>                 mv64x60_config_pci2mem_windows(bh, bh->hose_a,
> &si->pci_0, 0,
>                         mem_windows);
>                 bh->ci->set_pci2regs_window(bh, bh->hose_a, 0,
> @@ -548,6 +595,7 @@
>                 mv64x60_config_pci_params(bh->hose_b, &si->pci_1);
> 
>                 mv64x60_config_cpu2pci_windows(bh, &si->pci_1, 1);
> +                mv64x60_config_idma2pci_windows(bh, &si->pci_1, 1);
>                 mv64x60_config_pci2mem_windows(bh, bh->hose_b,
> &si->pci_1, 1,
>                         mem_windows);
>                 bh->ci->set_pci2regs_window(bh, bh->hose_b, 1,
> @@ -1136,6 +1188,42 @@
>                         bh->ci->disable_window_32bit(bh, win_tab[bus][i
> +1]);
>  }
> 
> +static u32 idma_tab_xtra[MV64x60_CPU2MEM_WINDOWS] __initdata = {
> +        MV64x60_IDMA2MEM_0_WIN, MV64x60_IDMA2MEM_1_WIN,
> +        MV64x60_IDMA2MEM_2_WIN, MV64x60_IDMA2MEM_3_WIN,
> +};
> +
> +void __init
> +mv64x60_config_idma2pci_windows(struct mv64x60_handle *bh,
> +        struct mv64x60_pci_info *pi, u32 bus)
> +{
> +        u32     attributes, unit_id;
> +        int     i;
> +
> +       /* Target Unit IDs: PCI0 = 3, PCI1 = 4. */
> +       unit_id = bus ? 0x4 : 0x3;
> +       /* 0x1d == No swap data, PCI-X NS attribute asserted, PCI memory
> +        * space, PCIx_REQ64n asserted according to requested data size.
> */
> +       attributes = (0x1d << 8) | unit_id;
> +
> +        for (i=0; i<3; i++)
> +                if (pi->pci_mem[i].size > 0) {
> +                        mv64x60_set_32bit_window(bh, idma_tab_xtra[i],
> +                                pi->pci_mem[i].cpu_base,
> pi->pci_mem[i].size,
> +                                attributes);
> +                        bh->ci->enable_window_32bit(bh,
> idma_tab_xtra[i]);
> +                        /* Give idma r/w access to PCI memory region */
> +                        mv64x60_set_bits(bh,
> MV64360_IDMA2MEM_ACC_PROT_0,
> +                                (0x3 << (i << 1)));
> +                        mv64x60_set_bits(bh,
> MV64360_IDMA2MEM_ACC_PROT_1,
> +                                (0x3 << (i << 1)));
> +                        mv64x60_set_bits(bh,
> MV64360_IDMA2MEM_ACC_PROT_2,
> +                                (0x3 << (i << 1)));
> +                        mv64x60_set_bits(bh,
> MV64360_IDMA2MEM_ACC_PROT_3,
> +                                (0x3 << (i << 1)));
> +                }
> +}
> +
>  /*
> 
> *****************************************************************************
>   *
> @@ -2220,8 +2311,8 @@
>  };
> 
>  static u32 idma_tab[MV64x60_CPU2MEM_WINDOWS] __initdata = {
> -       MV64x60_IDMA2MEM_0_WIN, MV64x60_IDMA2MEM_1_WIN,
> -       MV64x60_IDMA2MEM_2_WIN, MV64x60_IDMA2MEM_3_WIN,
> +        MV64x60_IDMA2MEM_4_WIN, MV64x60_IDMA2MEM_5_WIN,
> +        MV64x60_IDMA2MEM_6_WIN, MV64x60_IDMA2MEM_7_WIN,
>  };
> 
>  static u32 dram_selects[MV64x60_CPU2MEM_WINDOWS] __initdata =
> @@ -2285,13 +2376,13 @@
> 
>                         /* Give idma r/w access to memory region */
>                         mv64x60_set_bits(bh,
> MV64360_IDMA2MEM_ACC_PROT_0,
> -                               (0x3 << (i << 1)));
> +                                (0x3 << ((i+4) << 1)));
>                         mv64x60_set_bits(bh,
> MV64360_IDMA2MEM_ACC_PROT_1,
> -                               (0x3 << (i << 1)));
> +                                (0x3 << ((i+4) << 1)));
>                         mv64x60_set_bits(bh,
> MV64360_IDMA2MEM_ACC_PROT_2,
> -                               (0x3 << (i << 1)));
> +                                (0x3 << ((i+4) << 1)));
>                         mv64x60_set_bits(bh,
> MV64360_IDMA2MEM_ACC_PROT_3,
> -                               (0x3 << (i << 1)));
> +                                (0x3 << ((i+4) << 1)));
>                 }
>  }

I didn't go through this in great detail but it looks like you have
the right idea (IMHO).  Although, I don't know why you didn't just
use windows 4-7 for the idma->pci mappings and leave the idma->mem code
alone.

Mark

Re: pmppc7448/mv64x60 DMA from PCI to memory

[Phil Nitschke]

On Wed, 2006-05-24 at 13:52 -0700, Mark A. Greer wrote:
> On Wed, May 24, 2006 at 11:53:54AM +0930, Phil Nitschke wrote:
> > On Tue, 2006-05-23 at 16:54 -0700, Mark A. Greer wrote:
> > 
> > > You say that you don't see any PCI traffic.  Does that mean you
> > > have a PCI analyzer and that you are sure that its set up correctly?
> > 
> > I don't have a PCI analyzer, however the JTAG used to program the PCI
> > device has been configured to display 4 K samples of PCI bus signals
> > (about 20 microsecs?) around the time of an interrupt which results in
> > the DMA being requested.  Since my last post, I have managed to see some
> > traffic, but the PCI STOP# line is asserted, so I'm not seeing any data
> > being read.  I'll investigate further...

It turns out that the PCI device firmware was not responding correctly
to MRL and MRM (memory read multiple) PCI commands, but was working for
MR... Fixed now, and DMA from PCI is working.  Just looking at byte
order today.

> > OK.  I also note there are several cases where this is used in
> > mv64x60.c:
> > 
> >         for (i=0; i<3; i++)
> > 
> > Why is 3 used in these loops, and not some other constant like
> > MV64360_xxxxx_WINDOWS (which are usually 4, not 3)?
> 
> Different things.  The "i<3;" are when looping through windows that are 
> related to a struct pci_controller's mem_resource.

OK.

> > > > Do I need to add
> > > >         tests to my source and destination regions, to determine if they
> > > >         cross one of the 512 MB regions, and hence will require a
> > > >         different CSx line (and thus the DMA will need to be broken into
> > > >         two transactions), or does kernel already take care to ensure
> > > >         allocated regions will not cross these boundaries?
> > > 
> > > No.  You need to do what's appropriate for the hardware that you are
> > > essentially writing a driver for.  YOU are supposed to know what the
> > > limitations of your hardware are.  
> > 
> > OK, I know how my hardware is configured, but when trying to write a
> > generic driver, perhaps I need to have the mv64x60.c code remember the
> > CSx barriers, e.g. in the mv64x60_chip_info, so the IDMA engine can
> > access it.  Do you think this would be possible/beneficial?
> 
> No.  Just set up and enable an IDMA window to access all of pci mem space
> and be done with it.

No, this is different.  The patch I posted does map all the PCI mem
space as you've suggested.  The problem I'm trying to avoid is if the
IDMA engine tries to transfer data from this PCI mem region into a
buffer that crosses one of the DRAM address windows (and hence uses
different CSn lines).  Then the transfer needs to be broken into two
separate DMAs.  But if this information is not stored in the chip info,
how is the DMA driver to know where the memory boundaries are (except by
reading the already programmed windows and deducing these boundaries)?

> I didn't go through this in great detail but it looks like you have
> the right idea (IMHO).  Although, I don't know why you didn't just
> use windows 4-7 for the idma->pci mappings and leave the idma->mem code
> alone.

Two reasons (however flaky).  The lower 4 windows have an upper 32-bit
address register, so it is better to leave these for users (lucky
bastards!) that have more than 4 GB address space.  Secondly, the IDMA
supports the address override feature (which I was trying to use in
desperation when nothing was working for me), wherein the transaction
target interface, attributes and upper 32-bit address are taken from
BAR1, BAR2 or BAR3.  So I thought it would be better to leave these
alone.

-- 
Phil