Problems with Freescale DMA driver

Problems with Freescale DMA driver

[Ira Snyder]

Hello LKML,

I'm developing on a Freescale MPC8349EA MDS evaluation board. I'm
currently testing the DMA controller. It uses the CONFIG_FSL_DMA driver.

All tests were done with a recent git pull.

I built the CONFIG_DMATEST DMA test client, and it runs perfectly in the
absence of other DMA activity. Enabling CONFIG_NET_DMA and running the
DMA test client while network traffic is running causes failures such as
the following:

of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout!
dma0chan0-test0: dstbuf[0x9c0] mismatch! Expected c2, got df
dma0chan0-test0: dstbuf[0x9c1] mismatch! Expected c1, got df
dma0chan0-test0: dstbuf[0x9c2] mismatch! Expected c0, got df
dma0chan0-test0: dstbuf[0x9c3] not copied! Expected df, got 3c
dma0chan0-test0: dstbuf[0x9c4] not copied! Expected de, got 3b
dma0chan0-test0: dstbuf[0x9c5] not copied! Expected dd, got 3a
dma0chan0-test0: dstbuf[0x9c6] not copied! Expected dc, got 39
dma0chan0-test0: dstbuf[0x9c7] not copied! Expected db, got 38
dma0chan0-test0: dstbuf[0x9c8] not copied! Expected da, got 37
dma0chan0-test0: dstbuf[0x9c9] not copied! Expected d9, got 36
dma0chan0-test0: dstbuf[0x9ca] not copied! Expected d8, got 35
dma0chan0-test0: dstbuf[0x9d8] mismatch! Expected ca, got c2
dma0chan0-test0: dstbuf[0x9d9] mismatch! Expected c9, got c1
dma0chan0-test0: dstbuf[0x9da] mismatch! Expected c8, got c0
dma0chan0-test0: dstbuf[0x9db] mismatch! Expected c7, got df
dma0chan0-test0: dstbuf[0x9dc] mismatch! Expected c6, got de
dma0chan0-test0: dstbuf[0x9dd] mismatch! Expected c5, got dd
dma0chan0-test0: dstbuf[0x9de] mismatch! Expected c4, got dc
dma0chan0-test0: dstbuf[0x9df] mismatch! Expected c3, got db
dma0chan0-test0: dstbuf[0x9e0] mismatch! Expected c2, got da
dma0chan0-test0: dstbuf[0x9e1] mismatch! Expected c1, got d9
dma0chan0-test0: dstbuf[0x9e2] mismatch! Expected c0, got d8
dma0chan0-test0: dstbuf[0x9e3] mismatch! Expected df, got d7
dma0chan0-test0: dstbuf[0x9e4] mismatch! Expected de, got d6
dma0chan0-test0: dstbuf[0x9e5] mismatch! Expected dd, got d5
dma0chan0-test0: dstbuf[0x9e6] mismatch! Expected dc, got d4
dma0chan0-test0: dstbuf[0x9e7] mismatch! Expected db, got d3
dma0chan0-test0: dstbuf[0x9e8] mismatch! Expected da, got d2
dma0chan0-test0: dstbuf[0x9e9] mismatch! Expected d9, got d1
dma0chan0-test0: dstbuf[0x9ea] mismatch! Expected d8, got d0
dma0chan0-test0: dstbuf[0x9eb] mismatch! Expected d7, got cf
dma0chan0-test0: dstbuf[0x9ec] mismatch! Expected d6, got ce
dma0chan0-test0: 7011 errors suppressed
dma0chan0-test0: #12268: 7043 errors with src_off=0x1b6b dst_off=0x96e len=0x1be2

Is this to be expected? I thought the whole point of the DMAEngine API
was to enable sharing of DMA channels.

Another issue I noticed was running a network transfer from another
computer to the board. I get the following in dmesg, varying numbers of
times. This is with no other DMA activity running, dmatest is not
loaded. NET_DMA should be the only thing using the controller:

of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout!
of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout!
of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout!
of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout!
of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout!
of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout!
of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout!
of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout!
of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout!

On the run where the above errors were printed,
/sys/class/dma/dma0chan0/memcpy_count increased by ~36000. Certainly the
DMA halt error doesn't happen for every single dma transaction.

Is this normal behavior, or is something broken in the driver?

I've CC'd the maintainers responsible for the FSL DMA driver, as well as
the generic DMA offload system.

Please CC me on any responses, I'm not subscribed to LKML.

I'm happy to help test any changes, or add information if it is needed,
please just ask.

Thanks,
Ira

Re: Problems with Freescale DMA driver

[Dan Williams]

[adding Timur to the cc]

On Mon, Aug 25, 2008 at 9:51 AM, Ira Snyder <iws@ovro.caltech.edu> wrote:
> Hello LKML,
>
> I'm developing on a Freescale MPC8349EA MDS evaluation board. I'm
> currently testing the DMA controller. It uses the CONFIG_FSL_DMA driver.
>
> All tests were done with a recent git pull.
>
> I built the CONFIG_DMATEST DMA test client, and it runs perfectly in the
> absence of other DMA activity. Enabling CONFIG_NET_DMA and running the
> DMA test client while network traffic is running causes failures such as
> the following:
>
> of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout!
> dma0chan0-test0: dstbuf[0x9c0] mismatch! Expected c2, got df
> dma0chan0-test0: dstbuf[0x9c1] mismatch! Expected c1, got df
> dma0chan0-test0: dstbuf[0x9c2] mismatch! Expected c0, got df
> dma0chan0-test0: dstbuf[0x9c3] not copied! Expected df, got 3c
> dma0chan0-test0: dstbuf[0x9c4] not copied! Expected de, got 3b
> dma0chan0-test0: dstbuf[0x9c5] not copied! Expected dd, got 3a
> dma0chan0-test0: dstbuf[0x9c6] not copied! Expected dc, got 39
> dma0chan0-test0: dstbuf[0x9c7] not copied! Expected db, got 38
> dma0chan0-test0: dstbuf[0x9c8] not copied! Expected da, got 37
> dma0chan0-test0: dstbuf[0x9c9] not copied! Expected d9, got 36
> dma0chan0-test0: dstbuf[0x9ca] not copied! Expected d8, got 35
> dma0chan0-test0: dstbuf[0x9d8] mismatch! Expected ca, got c2
> dma0chan0-test0: dstbuf[0x9d9] mismatch! Expected c9, got c1
> dma0chan0-test0: dstbuf[0x9da] mismatch! Expected c8, got c0
> dma0chan0-test0: dstbuf[0x9db] mismatch! Expected c7, got df
> dma0chan0-test0: dstbuf[0x9dc] mismatch! Expected c6, got de
> dma0chan0-test0: dstbuf[0x9dd] mismatch! Expected c5, got dd
> dma0chan0-test0: dstbuf[0x9de] mismatch! Expected c4, got dc
> dma0chan0-test0: dstbuf[0x9df] mismatch! Expected c3, got db
> dma0chan0-test0: dstbuf[0x9e0] mismatch! Expected c2, got da
> dma0chan0-test0: dstbuf[0x9e1] mismatch! Expected c1, got d9
> dma0chan0-test0: dstbuf[0x9e2] mismatch! Expected c0, got d8
> dma0chan0-test0: dstbuf[0x9e3] mismatch! Expected df, got d7
> dma0chan0-test0: dstbuf[0x9e4] mismatch! Expected de, got d6
> dma0chan0-test0: dstbuf[0x9e5] mismatch! Expected dd, got d5
> dma0chan0-test0: dstbuf[0x9e6] mismatch! Expected dc, got d4
> dma0chan0-test0: dstbuf[0x9e7] mismatch! Expected db, got d3
> dma0chan0-test0: dstbuf[0x9e8] mismatch! Expected da, got d2
> dma0chan0-test0: dstbuf[0x9e9] mismatch! Expected d9, got d1
> dma0chan0-test0: dstbuf[0x9ea] mismatch! Expected d8, got d0
> dma0chan0-test0: dstbuf[0x9eb] mismatch! Expected d7, got cf
> dma0chan0-test0: dstbuf[0x9ec] mismatch! Expected d6, got ce
> dma0chan0-test0: 7011 errors suppressed
> dma0chan0-test0: #12268: 7043 errors with src_off=0x1b6b dst_off=0x96e len=0x1be2
>
> Is this to be expected? I thought the whole point of the DMAEngine API
> was to enable sharing of DMA channels.
>
> Another issue I noticed was running a network transfer from another
> computer to the board. I get the following in dmesg, varying numbers of
> times. This is with no other DMA activity running, dmatest is not
> loaded. NET_DMA should be the only thing using the controller:
>
> of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout!
> of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout!
> of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout!
> of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout!
> of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout!
> of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout!
> of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout!
> of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout!
> of-fsl-dma-channel e0008100.dma-channe: DMA halt timeout!
>
> On the run where the above errors were printed,
> /sys/class/dma/dma0chan0/memcpy_count increased by ~36000. Certainly the
> DMA halt error doesn't happen for every single dma transaction.
>
> Is this normal behavior, or is something broken in the driver?
>
> I've CC'd the maintainers responsible for the FSL DMA driver, as well as
> the generic DMA offload system.
>
> Please CC me on any responses, I'm not subscribed to LKML.
>
> I'm happy to help test any changes, or add information if it is needed,
> please just ask.
>
> Thanks,
> Ira

Re: Problems with Freescale DMA driver

[Timur Tabi]

Dan Williams wrote:
> [adding Timur to the cc]
> 
> On Mon, Aug 25, 2008 at 9:51 AM, Ira Snyder <iws@ovro.caltech.edu> wrote:
>> Hello LKML,
>>
>> I'm developing on a Freescale MPC8349EA MDS evaluation board. I'm
>> currently testing the DMA controller. It uses the CONFIG_FSL_DMA driver.
>>
>> All tests were done with a recent git pull.

I just posted two patches lkml that affect the FSL DMA driver:

[PATCH v2] dmatest: properly handle duplicate DMA channels
[PATCH] fsl-dma: allow Freescale Elo DMA driver to be compiled as a module

Could you apply them both and re-run the tests?

I will also try to duplicate this here on an 8360.

-- 
Timur Tabi
Linux kernel developer at Freescale

Re: Problems with Freescale DMA driver

[Ira Snyder]

On Thu, Sep 18, 2008 at 05:02:51PM -0500, Timur Tabi wrote:
> Dan Williams wrote:
> > [adding Timur to the cc]
> > 
> > On Mon, Aug 25, 2008 at 9:51 AM, Ira Snyder <iws@ovro.caltech.edu> wrote:
> >> Hello LKML,
> >>
> >> I'm developing on a Freescale MPC8349EA MDS evaluation board. I'm
> >> currently testing the DMA controller. It uses the CONFIG_FSL_DMA driver.
> >>
> >> All tests were done with a recent git pull.
> 
> I just posted two patches lkml that affect the FSL DMA driver:
> 
> [PATCH v2] dmatest: properly handle duplicate DMA channels
> [PATCH] fsl-dma: allow Freescale Elo DMA driver to be compiled as a module
> 
> Could you apply them both and re-run the tests?
> 

I have tested today's git pull with your patches. I had to add the
cell-index property to each DMA channel in the dts for my board. I just
added them as 0, 1, 2, and 3 respectively.

When your patch makes it in, you should probably add the cell-index to
all of the dts files that are missing them.

I was still able to cause the "DMA Halt Timeout" error message (on
channel 0, just like in the previous report).

To cause this, I built a kernel with CONFIG_DMATEST=y and
CONFIG_NET_DMA=y. Then, I ran the following concurrently:

netperf -H localhost -l 600
netperf -H 192.168.17.65 -l 600

While they were running, I got the "DMA Halt Timeout" message 6 times.
Here is the output (all messages were the same):
of_platform e0008100.dma-channe: DMA halt timeout!

The following output leads me to believe that NET_DMA is using channel 0
only:

cat /sys/class/dma/*/memcpy_count
1278897
198585
198681
198591

I was not able to cause the DMA controller to copy bad data. Perhaps the
kernel just had a bug at the time. It was still relatively early on in
the development cycle.

> I will also try to duplicate this here on an 8360.
> 

Thanks.

In a related note, I've been somewhat following the discussion on LKML
about creating an API for requesting a single DMA channel. This would be
great in a driver I've written for PCI communication over a backplane.
(I have a test board running in PCI Agent mode. It creates a virtual
ethernet interface that passes data over the PCI bus.)

Also, are there any plans to support the external start feature on the
83XX parts? (According to my datasheet, it is supported.) I will be
using this feature in a driver I will be starting fairly soon.

Thanks,
Ira

Re: Problems with Freescale DMA driver

[Timur Tabi]

Ira Snyder wrote:

> When your patch makes it in, you should probably add the cell-index to
> all of the dts files that are missing them.

Ah, I see that Kumar added a bunch of DMA nodes back in June and forgot the
cell-index properties.

> I was not able to cause the DMA controller to copy bad data. Perhaps the
> kernel just had a bug at the time. It was still relatively early on in
> the development cycle.

dmatest runs all the time, so it's possible that it's eating up DMA resources.
Try running your tests without CONFIG_DMATEST.

> In a related note, I've been somewhat following the discussion on LKML
> about creating an API for requesting a single DMA channel. This would be
> great in a driver I've written for PCI communication over a backplane.
> (I have a test board running in PCI Agent mode. It creates a virtual
> ethernet interface that passes data over the PCI bus.)

PCI devices should not be using generic DMA resources, so I don't see how this
would apply.

> Also, are there any plans to support the external start feature on the
> 83XX parts? (According to my datasheet, it is supported.) I will be
> using this feature in a driver I will be starting fairly soon.

External start is very device-specific, so I don't see how a generic DMA driver,
which is intended only for memory-to-memory DMA operations, can coordinate with
an external master.

-- 
Timur Tabi
Linux kernel developer at Freescale

Re: Problems with Freescale DMA driver

[Ira Snyder]

On Tue, Sep 23, 2008 at 09:03:38AM -0500, Timur Tabi wrote:
> Ira Snyder wrote:
> 
> > When your patch makes it in, you should probably add the cell-index to
> > all of the dts files that are missing them.
> 
> Ah, I see that Kumar added a bunch of DMA nodes back in June and forgot the
> cell-index properties.
> 
> > I was not able to cause the DMA controller to copy bad data. Perhaps the
> > kernel just had a bug at the time. It was still relatively early on in
> > the development cycle.
> 
> dmatest runs all the time, so it's possible that it's eating up DMA resources.
> Try running your tests without CONFIG_DMATEST.
> 

I am unable to cause any problems without dmatest running. You are
probably correct, it is just using lots of resources.

> > In a related note, I've been somewhat following the discussion on LKML
> > about creating an API for requesting a single DMA channel. This would be
> > great in a driver I've written for PCI communication over a backplane.
> > (I have a test board running in PCI Agent mode. It creates a virtual
> > ethernet interface that passes data over the PCI bus.)
> 
> PCI devices should not be using generic DMA resources, so I don't see how this
> would apply.
> 

Let me try to explain what I'm doing. If you have any ideas of a better
way to do this, I'm completely open to suggestions.

I have the following:
1) A normal x86 desktop computer, running Linux
2) An MPC8349EMDS eval board, running in PCI agent/slave mode. It is
   plugged into a PCI slot in the desktop. It also runs Linux.

Naturally, I want to transfer lots of data between them. They are
already connected by the PCI bus, so transferring data across that seems
logical. Using the network infrastructure in the kernel seemed like a
good solution, since it handles multiplexing/demultiplexing of traffic
nicely. I chose to use an ethernet device because there is some
documentation for them in LDD3.

So I wrote a device driver for both computers which creates an ethernet
interface. The actual data transfer must happen over the PCI bus, which
just shows up as memory in the MPC8349's address space.

Since using the CPU to do data transfer is very slow, I need to use the
DMA controller to transfer the data. I used the DMAEngine API because it
existed, and so I didn't have to program the DMA controller registers
manually. The MPC8349 handles all of the data transfer using its DMA
controller.

I hope that makes sense. I can post the code if anyone would like to see
it. It is not ready for mainline inclusion yet, but it is in a fairly
good state.

> > Also, are there any plans to support the external start feature on the
> > 83XX parts? (According to my datasheet, it is supported.) I will be
> > using this feature in a driver I will be starting fairly soon.
> 
> External start is very device-specific, so I don't see how a generic DMA driver,
> which is intended only for memory-to-memory DMA operations, can coordinate with
> an external master.
> 

Ok. I was just wondering, since drivers/dma/fsl_dma.c has some code that
appears to handle external master. It is 85XX specific though. There are
no users in the kernel tree showing example uses.

I will be using it in the same setup as above. The board I am working on
is based on the MPC8349EMDS board, but with 5 FPGA's on it for data
processing. The FPGA's will do some calculations, then use the external
start to trigger a DMA to copy the data from the FPGA memory to the
MPC8349's DDR, where it is buffered up for later processing.

Sorry if this has gone off topic.

Thanks,
Ira

Re: Problems with Freescale DMA driver

[Timur Tabi]

(Removing various CC:'s since this thread is changing topic)

Ira Snyder wrote:

> I hope that makes sense. I can post the code if anyone would like to see
> it. It is not ready for mainline inclusion yet, but it is in a fairly
> good state.

Well, it's a bit goofy, but it does make sense.  Since you don't have an actual
single-function PCI device, you need to use the MPC8349's DMA controller.

I don't see where you need the ability to reserve a DMA channel, though.  You're
using fsldma.c as its intended.

> Ok. I was just wondering, since drivers/dma/fsl_dma.c has some code that
> appears to handle external master. It is 85XX specific though. There are
> no users in the kernel tree showing example uses.

Indeed it does.  It looks like a feature that was thrown in but never given much
thought or testing.  It appears that the current code just lets you turn on the
external master pause bit in the MR register, but there's nothing to make sure
that it actually works in a meaningful way.  If there's a queue of transfers for
a given channel, I don't see where the fsldma avoids stalled queues.

-- 
Timur Tabi
Linux kernel developer at Freescale

Re: Problems with Freescale DMA driver

[Ira Snyder]

On Tue, Sep 23, 2008 at 11:10:05AM -0500, Timur Tabi wrote:
> (Removing various CC:'s since this thread is changing topic)
> 
> Ira Snyder wrote:
> 
> > I hope that makes sense. I can post the code if anyone would like to see
> > it. It is not ready for mainline inclusion yet, but it is in a fairly
> > good state.
> 
> Well, it's a bit goofy, but it does make sense.  Since you don't have an actual
> single-function PCI device, you need to use the MPC8349's DMA controller.
> 
> I don't see where you need the ability to reserve a DMA channel, though.  You're
> using fsldma.c as its intended.
> 

Ok. I don't really care which channel I get, just that I get one. I
won't be adding or removing DMA devices from the system, so it shouldn't
be a problem.

> > Ok. I was just wondering, since drivers/dma/fsl_dma.c has some code that
> > appears to handle external master. It is 85XX specific though. There are
> > no users in the kernel tree showing example uses.
> 
> Indeed it does.  It looks like a feature that was thrown in but never given much
> thought or testing.  It appears that the current code just lets you turn on the
> external master pause bit in the MR register, but there's nothing to make sure
> that it actually works in a meaningful way.  If there's a queue of transfers for
> a given channel, I don't see where the fsldma avoids stalled queues.
> 

That was also my impression. I'll probably start by programming the
registers directly, and hopefully come up with a nice interface. When I
figure it out, I'll send patches.

In this case, I probably would like to reserve a DMA channel for my
exclusive use. That way I won't run into problems by programming the DMA
by hand.

Thanks for all the input.
Ira

Re: Problems with Freescale DMA driver

[Timur Tabi]

Ira Snyder wrote:

> In this case, I probably would like to reserve a DMA channel for my
> exclusive use. That way I won't run into problems by programming the DMA
> by hand.

If you write your own DMA driver, the current method for reserving a DMA channel
is to change the compatible property for that channel in the device tree to
something other than "fsl,elo-dma-channel" (or "fsl,eloplus-dma-channel").

-- 
Timur Tabi
Linux kernel developer at Freescale