Channel swapping issue on TI OMAP3/TWL4030

Channel swapping issue on TI OMAP3/TWL4030

[ylin]

Hi,

We are using a customized TI OMAP3 with TWL4030, similar to OMAP3EVM 
board.  We noticed occasionally (after bring up few hours) the mic 
channel  audio is swapped.  Has anyone seen similar problem? Any 
suggestion where the problem might be?

Thanks,
Ying

Re: Channel swapping issue on TI OMAP3/TWL4030

[Jarkko Nikula]

On Sat, 12 Feb 2011 11:16:42 -0500
ylin@mail.com wrote:

> We are using a customized TI OMAP3 with TWL4030, similar to OMAP3EVM 
> board.  We noticed occasionally (after bring up few hours) the mic 
> channel  audio is swapped.  Has anyone seen similar problem? Any 
> suggestion where the problem might be?
> 
Is this with recent kernels? There were this channel swapping problem
before but it got fixed by Eero Nurkkala for the 2.6.32 kernel by
following commits:

commit 9abea08e43c6cfc18399e42cbc6028d111532f61
"OMAP: McBSP: Provide functions for ASoC frame syncronization"
and
commit ca6e2ce08679c094878d7f39a0349a7db1d13675
"ASoC: Always syncronize audio transfers on frames"

-- 
Jarkko

Re: Channel swapping issue on TI OMAP3/TWL4030

[peter.ujfalusi]

Hi,
> ________________________________________
> From: alsa-devel-bounces@alsa-project.org [alsa-devel-bounces@alsa-project.org] on behalf of ext Jarkko Nikula [jhnikula@gmail.com]
> Sent: Sunday, February 13, 2011 11:25
> To: ylin@mail.com
> Cc: alsa-devel@alsa-project.org
> Subject: Re: [alsa-devel] Channel swapping issue on TI OMAP3/TWL4030
> 
> On Sat, 12 Feb 2011 11:16:42 -0500
> ylin@mail.com wrote:
> 
> > We are using a customized TI OMAP3 with TWL4030, similar to OMAP3EVM
> > board.  We noticed occasionally (after bring up few hours) the mic
> > channel  audio is swapped.  Has anyone seen similar problem? Any
> > suggestion where the problem might be?
> >
> Is this with recent kernels? There were this channel swapping problem
> before but it got fixed by Eero Nurkkala for the 2.6.32 kernel by
> following commits:

This is something different.
The channel swapping/shifting issue was affecting the playback
stream only.

Peter 

Re: Channel swapping issue on TI OMAP3/TWL4030

[Jarkko Nikula]

On Sun, 13 Feb 2011 09:48:00 +0000
<peter.ujfalusi@nokia.com> wrote:

> > > We are using a customized TI OMAP3 with TWL4030, similar to OMAP3EVM
> > > board.  We noticed occasionally (after bring up few hours) the mic
> > > channel  audio is swapped.  Has anyone seen similar problem? Any
> > > suggestion where the problem might be?
> > >
> > Is this with recent kernels? There were this channel swapping problem
> > before but it got fixed by Eero Nurkkala for the 2.6.32 kernel by
> > following commits:
> 
> This is something different.
> The channel swapping/shifting issue was affecting the playback
> stream only.
> 
Hmm.. could be. Also IRCC before the swapping was quite frequent.
Especially during the stream startup time but I think also sometimes
while playback was running? This is infrequent but happens anyway.

-- 
Jarkko

Re: Channel swapping issue on TI OMAP3/TWL4030

[peter.ujfalusi]

Hi,
> ________________________________________
> From: alsa-devel-bounces@alsa-project.org [alsa-devel-bounces@alsa-project.org] on behalf of ext ylin@mail.com [ylin@mail.com]
> Sent: Saturday, February 12, 2011 18:16
> To: alsa-devel@alsa-project.org
> Subject: [alsa-devel] Channel swapping issue on TI OMAP3/TWL4030
> 
> Hi,
> 
> We are using a customized TI OMAP3 with TWL4030, similar to OMAP3EVM
> board.  We noticed occasionally (after bring up few hours) the mic
> channel  audio is swapped.  Has anyone seen similar problem? Any
> suggestion where the problem might be?

I have not experienced with channel swapping during capture..
It would help, if I know:
what is your kernel version
Source of the recording (digimic, HS, main mic, etc)

To clarify:
you experience sudden channel switch during active recording, when
the recording was running for few hours?
Or is it that you start/stop the recording, and sometimes the channels
are swapped?

Peter

Re: Channel swapping issue on TI OMAP3/TWL4030

[ylin]

> I have not experienced with channel swapping during capture..
> It would help, if I know:
> what is your kernel version
> Source of the recording (digimic, HS, main mic, etc)

We are using 2.6.32.  The source of the recording is using Analog Mic.  
The Left Main mic is always on, and Right Sub mic can be either on or 
off when this occurs.  We are not using AUX or digimic.

> To clarify:
> you experience sudden channel switch during active recording, when
> the recording was running for few hours?
> Or is it that you start/stop the recording, and sometimes the channels
> are swapped?

It is sudden channel switch during active recording.  There is no XRUN 
or any noticeable error when this happens.

This is a VoIP app, we have duplex audio when this happens.  Both 
capture and playback are setup to run at 48kHz, 2 channel, S16_LE, with 
period size 10 mSec.

For playing, the left and right are the same data, we could not tell if 
the channel switch happens to playback as well.

Thanks,
Ying

Re: Channel swapping issue on TI OMAP3/TWL4030

[ylin]

Some update:
We peeked into the substream->dma_buffer.area, and found out the 
channels are already switched in DMA buffer.  We are looking into 
McBSP2 and DMA to find some clues, but not familiar in these areas.  
Any suggestion how to further narrow down the problem?

Thanks,
Ying

Re: Channel swapping issue on TI OMAP3/TWL4030

[Peter Ujfalusi]

Hi,

On 02/15/11 01:33, ext ylin@mail.com wrote:
> Some update:
> We peeked into the substream->dma_buffer.area, and found out the 
> channels are already switched in DMA buffer.  We are looking into 
> McBSP2 and DMA to find some clues, but not familiar in these areas.  
> Any suggestion how to further narrow down the problem?

Few more things:
I suppose the twl4030 is the master on the bus (McBSP slave)
Since you are using stereo capture, I assume the digital interface is in
I2S mode.

I could think of three source for channel switch:
1. The switch happens within the twl4030 codec itself
2. Switch happens because of underflow situation in McBSP
3. Switch happens because of overflow situation in McBSP


To narrow down the problem:
1. You need tracepoints on you board for McBSP2 pins, and a scope
You need to monitor the DX, and FS lines (bitclock might be useful as well).
Keep the right channel disabled, and run your application, observe the
DX/FS on the scope. When you notice the channel swap, check back on the
scope, if the data moved from left to right channel.

2/3. This is a bit more trickier, but I have done this several times:
You need to use the PRCM compliant interrupt configuration.
Do not request the TX/RX (62, 63 for McBSP2) interrupts in
omap_mcbsp_request, but request only the common IRQ (17 for McBSP2).
Enable the ROVFLEN, RUNDFEN, and RSYNCERREN bits in IRQENABLE_REG.
In the ir handler check for these enabled events (in IRQSTATUS_REG), and
print out the reason for the event (Do not forget to write back the
register content to IRQSTATUS_REG to ack the event).

I hope this helps narrowing down the problem.

-- 
Péter

Re: Channel swapping issue on TI OMAP3/TWL4030

[Peter Ujfalusi]

On 02/14/11 19:58, ext ylin@mail.com wrote:
> 
>> I have not experienced with channel swapping during capture..
>> It would help, if I know:
>> what is your kernel version
>> Source of the recording (digimic, HS, main mic, etc)
> 
> We are using 2.6.32.  The source of the recording is using Analog Mic.  
> The Left Main mic is always on, and Right Sub mic can be either on or 
> off when this occurs.  We are not using AUX or digimic.
> 
>> To clarify:
>> you experience sudden channel switch during active recording, when
>> the recording was running for few hours?
>> Or is it that you start/stop the recording, and sometimes the channels
>> are swapped?
> 
> It is sudden channel switch during active recording.  There is no XRUN 
> or any noticeable error when this happens.
> 
> This is a VoIP app, we have duplex audio when this happens.  Both 
> capture and playback are setup to run at 48kHz, 2 channel, S16_LE, with 
> period size 10 mSec.

I forgot to ask:
what mode you are using the McBSP?
Is it in element, or threshold mode?
You can check/change the McBSP mode in:
/sys/devices/platform/omap-mcbsp.2/dma_op_mode

> For playing, the left and right are the same data, we could not tell if 
> the channel switch happens to playback as well.

I see. Is there a way for you to run your application to capture only,
and from a separate app (aplay?) play a sample, which has audio only on
one channel?

-- 
Péter

Re: Channel swapping issue on TI OMAP3/TWL4030

[ylin]

> I forgot to ask:
> what mode you are using the McBSP?
> Is it in element, or threshold mode?
> You can check/change the McBSP mode in:
> /sys/devices/platform/omap-mcbsp.2/dma_op_mode

It is in threshold mode, with max threshold 1023.  I will try other 
modes.

> > For playing, the left and right are the same data, we could not 
tell if
> > the channel switch happens to playback as well.
>
> I see. Is there a way for you to run your application to capture only,
> and from a separate app (aplay?) play a sample, which has audio only 
on
> one channel?

I can fill zero to one of the play channel in our app to test it.

Thanks,
Ying

Re: Channel swapping issue on TI OMAP3/TWL4030

[Peter Ujfalusi]

On 02/15/11 18:23, ext ylin@mail.com wrote:
> 
>> I forgot to ask:
>> what mode you are using the McBSP?
>> Is it in element, or threshold mode?
>> You can check/change the McBSP mode in:
>> /sys/devices/platform/omap-mcbsp.2/dma_op_mode
> 
> It is in threshold mode, with max threshold 1023.  I will try other 
> modes.

This sounds safe, give the fact that the McBSP2 FIFO is 1280 word long.
The McBSP FIFO configuration has been corrected in newer kernels.

One thing that I would try is to synchronize the McBSP2 FIFO
configuration with the period size you use:
48KHz/stereo/16bit 10ms = 480 samples,
so configure the McBSP2 FIFO to 960 (for both tx, and rx threshold).
Might not help, but it worth a try...

>>> For playing, the left and right are the same data, we could not 
> tell if
>>> the channel switch happens to playback as well.
>>
>> I see. Is there a way for you to run your application to capture only,
>> and from a separate app (aplay?) play a sample, which has audio only 
> on
>> one channel?
> 
> I can fill zero to one of the play channel in our app to test it.

Great.
Let's see, if the swap also happens in the playback path as well.

-- 
Péter

Re: Channel swapping issue on TI OMAP3/TWL4030

[ylin]

> >
> >> I forgot to ask:
> >> what mode you are using the McBSP?
> >> Is it in element, or threshold mode?
> >> You can check/change the McBSP mode in:
> >> /sys/devices/platform/omap-mcbsp.2/dma_op_mode
> >
> > It is in threshold mode, with max threshold 1023.  I will try other
> > modes.
>
> This sounds safe, give the fact that the McBSP2 FIFO is 1280 word 
long.
> The McBSP FIFO configuration has been corrected in newer kernels.
>
> One thing that I would try is to synchronize the McBSP2 FIFO
> configuration with the period size you use:
> 48KHz/stereo/16bit 10ms = 480 samples,
> so configure the McBSP2 FIFO to 960 (for both tx, and rx threshold).
> Might not help, but it worth a try...

We tried element and frame mode, playback doesn't sound correct with 
these mode, but the capture is OK.  We will stick with threshold mode, 
and try the new threshold.

> >>> For playing, the left and right are the same data, we could not
> > tell if
> >>> the channel switch happens to playback as well.
> >>
> >> I see. Is there a way for you to run your application to capture 
only,
> >> and from a separate app (aplay?) play a sample, which has audio 
only
> > on
> >> one channel?
> >
> > I can fill zero to one of the play channel in our app to test it.
>
> Great.
> Let's see, if the swap also happens in the playback path as well.

None of our test shows the playback problem.  I think the playback path 
is fine, but will keep testing.

Also, we have verified the audio data from I2S interface looks fine 
using a scope even when the channels are switched.  We can eliminate 
twl4030 codec from the problem, and focus on McBSP and DMA.  We don't 
have chance to check McBSP overflow/underflow as you suggested yet.

Thanks,
Ying

Re: Channel swapping issue on TI OMAP3/TWL4030

[Peter Ujfalusi]

On 02/17/11 05:40, ext ylin@mail.com wrote:
> We tried element and frame mode, playback doesn't sound correct with 
> these mode, but the capture is OK.

Hrm, this does not sound right... Historically the element mode was the
first and only mode of using McBSP. The threshold mode has been added later.
Could you explain what do you mean, when you said that the playback was
not correct?
I have never experienced problems with the element mode.
In what mode you are using the TWL4030 codec? Is it master?
If it is master, can you compare your machine driver with for example
the Beagle board's machine driver?
Or can you share the relevant configuration (from your machine driver)
with us?

> None of our test shows the playback problem.  I think the playback path 
> is fine, but will keep testing.
> 
> Also, we have verified the audio data from I2S interface looks fine 
> using a scope even when the channels are switched.  We can eliminate 
> twl4030 codec from the problem,

That's good.

> and focus on McBSP and DMA.  We don't 
> have chance to check McBSP overflow/underflow as you suggested yet.

If we don't figure out what is the problem, you might need to do this...

-- 
Péter

Re: Channel swapping issue on TI OMAP3/TWL4030

[ylin]

> > We tried element and frame mode, playback doesn't sound correct 
with
> > these mode, but the capture is OK.
>
> Hrm, this does not sound right... Historically the element mode was 
the
> first and only mode of using McBSP. The threshold mode has been added 
later.
> Could you explain what do you mean, when you said that the playback 
was
> not correct?
> I have never experienced problems with the element mode.

The element mode works fine with playback after I fill more data to the 
buffer.  We try to optimize the delay and keep the minimum data in the 
buffer.  However, the element mode timing is slightly different from 
threshold, and we have under-run condition.  With filling extra data to 
the buffer, the element mode works fine.  We are testing if we see the 
channel switch with this mode.

> In what mode you are using the TWL4030 codec? Is it master?
> If it is master, can you compare your machine driver with for example
> the Beagle board's machine driver?

It is in master mode.  We are using OMAP3EVM board as reference.  Our 
kernel is based on the TI P3P release.
# /proc/ti-psp-version
AM37x/DM37x  Linux PSP version 03.00.01.06 (OMAP3PHX)
We are using the same audio drivers for OMAP3EVM.  There is only some 
unrelated patches to the driver, such as ramp delay and Bluetooth PCM 
support.

> Or can you share the relevant configuration (from your machine driver)
> with us?

I am not sure exactly the relevant configuration to share.  Here is  
the TWL4030 codec registers with both Tx and Rx audio running.  Please 
let me know if you like to see other configurations and I can pull them 
out.
# cat /sys/devices/platform/soc-audio/codec_reg
twl4030 registers
 0:    0
 1:   a3
 2:   c3
 3:    0
 4:    7
 5:   31
 6:    0
 7:    8
 8:    0
 9:    0
 a:    9
 b:    9
 c:    9
 d:    9
 e:    1
 f:    0
10:    0
11:    0
12:   3f
13:   3f
14:    0
15:    0
16:    0
17:    c
18:    0
19:    0
1a:    0
1b:   2b
1c:   2b
1d:    0
1e:    0
1f:    0
20:    0
21:   10
22:    0
23:    5
24:    4
25:    0
26:    0
27:    0
28:    0
29:    0
2a:   3f
2b:    0
2c:    0
2d:    0
2e:    0
2f:    0
30:    0
31:    0
32:    0
33:    0
34:    0
35:    0
36:    0
37:    0
38:    0
39:    0
3a:   16
3b:    0
3c:    0
3d:    0
3e:    2
3f:    0
40:    0
41:    0
42:    0
43:    0
44:    0
45:    0
46:    0
47:    0
48:   22
49:    0
4a:    2
#

> > and focus on McBSP and DMA.  We don't
> > have chance to check McBSP overflow/underflow as you suggested yet.
>
> If we don't figure out what is the problem, you might need to do 
this...

I am working on this.

Thanks,
Ying

Re: Channel swapping issue on TI OMAP3/TWL4030

[ylin]

> > Or can you share the relevant configuration (from your machine 
driver)
> > with us?

And, these are McBSP2 registers with both Tx and Rx audio running.
**** McBSP2 regs ****
DRR2:  0xffee
DRR1:  0x0000
DXR2:  0x0000
DXR1:  0x0000
SPCR2: 0x02f5
SPCR1: 0x0031
RCR2:  0x8041
RCR1:  0x0040
XCR2:  0x8041
XCR1:  0x0040
SRGR2: 0x001f
SRGR1: 0x0f00
PCR0:  0x000f
***********************

Thanks,
Ying

Re: Channel swapping issue on TI OMAP3/TWL4030

[Peter Ujfalusi]

Hi,

On 02/20/11 06:01, ext ylin@mail.com wrote:
>>> Or can you share the relevant configuration (from your machine 
> driver)
>>> with us?
> 
> And, these are McBSP2 registers with both Tx and Rx audio running.
> **** McBSP2 regs ****
> DRR2:  0xffee
> DRR1:  0x0000
> DXR2:  0x0000
> DXR1:  0x0000
> SPCR2: 0x02f5
> SPCR1: 0x0031
> RCR2:  0x8041
> RCR1:  0x0040
> XCR2:  0x8041
> XCR1:  0x0040
> SRGR2: 0x001f
> SRGR1: 0x0f00
> PCR0:  0x000f
> ***********************

Both the twl and McBSP config seams to be in sync.
I do not see anything, which stands out...

I think you need to implement the PRCM compliant interrupt config to
check for McBSP underflow/overflow.

I'll try to resurrect my Beagle board, since that as well have the twl
codec on McBSP2...

-- 
Péter

Re: Channel swapping issue on TI OMAP3/TWL4030

[ylin]

> I think you need to implement the PRCM compliant interrupt config to
> check for McBSP underflow/overflow.

I have implemented the interrupt handle for IRQ 17 for receive error 
and underflow/overflow.  However, the devices are running for 2 days 
without channel swapped.  I will keep them running.

> I'll try to resurrect my Beagle board, since that as well have the twl
> codec on McBSP2...

I will test it on EVM board as well.

We have tested the element mode for dma_op_mode.  It works fine, we 
don't see any channel swapping in this mode.

Thanks,
Ying

Re: Channel swapping issue on TI OMAP3/TWL4030

[ylin]

> > I think you need to implement the PRCM compliant interrupt config to
> > check for McBSP underflow/overflow.
>
> I have implemented the interrupt handle for IRQ 17 for receive error
> and underflow/overflow.  However, the devices are running for 2 days
> without channel swapped.  I will keep them running.

Finally, after 4 days, we caught the problem again.  When channel is 
swapped, the IRQ is trigger with receive underrun (IRQ_STATUS_REG = 
0x171e).  And, from the status register, the transmit is overflow as 
well.  However, I didn't fill zero to one of the transmit channel, and 
couldn't tell if the playback is swap as well.  Any idea why underun 
happens, or any suggestion to fix it?

Thanks,
Ying

Re: Channel swapping issue on TI OMAP3/TWL4030

[Peter Ujfalusi]

Hi,

Sorry, I got lost of this mail thread...

On 02/25/11 17:32, ext ylin@mail.com wrote:
> Finally, after 4 days, we caught the problem again.  When channel is 
> swapped, the IRQ is trigger with receive underrun (IRQ_STATUS_REG = 
> 0x171e).  And, from the status register, the transmit is overflow as 
> well.  However, I didn't fill zero to one of the transmit channel, and 
> couldn't tell if the playback is swap as well.  Any idea why underun 
> happens, or any suggestion to fix it?

Hrm, underflow on receive, and overflow on transmit at the same time?
Do you had the transmit overflow irq enabled as well?

Do you have access to the OMAP Errata documents? I have seen an Errata
for McBSP2, which was about corruption on transmit operation. It might
be, that the same thing causes nasty effects on the receive side as well.

Because of the nature of this happening in your case, I suspect that we
are facing with some HW race/bug problem.
Receive underflow happens, if DMA tries to read data from McBSP receive
data register when it is empty.
This should not happen, since the McBSP threshold and the DMA size is in
sync, so we should read an amount of data, which is for sure was in the
buffer, when McBSP signaled DMA.

Do you have support contact with TI?
I'll try to reach ours meanwhile, if they have any idea.

Regards,
Péter

Re: Channel swapping issue on TI OMAP3/TWL4030

[ylin]

> > It is in threshold mode, with max threshold 1023.  I will try other
> > modes.
>
> This sounds safe, give the fact that the McBSP2 FIFO is 1280 word 
long.
> The McBSP FIFO configuration has been corrected in newer kernels.
>
> One thing that I would try is to synchronize the McBSP2 FIFO
> configuration with the period size you use:
> 48KHz/stereo/16bit 10ms = 480 samples,
> so configure the McBSP2 FIFO to 960 (for both tx, and rx threshold).
> Might not help, but it worth a try...

I checked the threshold registers, both are 960 (959 to be precious).  
It is a safe value, but the channel is switched with 960.  In addition, 
we have tested with sample rate at 16kHz and still have the problem.

Thanks,
Ying

Re: Channel swapping issue on TI OMAP3/TWL4030

[Jarkko Nikula]

On Mon, 14 Feb 2011 12:58:35 -0500
ylin@mail.com wrote:

> 
> > I have not experienced with channel swapping during capture..
> > It would help, if I know:
> > what is your kernel version
> > Source of the recording (digimic, HS, main mic, etc)
> 
> We are using 2.6.32.  The source of the recording is using Analog Mic.  
> The Left Main mic is always on, and Right Sub mic can be either on or 
> off when this occurs.  We are not using AUX or digimic.
> 
I was thinking is this 2.6.32 derived completely from Vanilla version or
only partially? I think I've seen some case where the kernel was forked
earlier and got upgraded to the upstream version only partially.

Even Peter told that those channel switching patches in 2.6.32 should
fix the playback case only, worth to check just in case are those fixes
in your kernel tree.

-- 
Jarkko

Re: Channel swapping issue on TI OMAP3/TWL4030

[ylin]

> >
> > We are using 2.6.32.  The source of the recording is using Analog 
Mic.
> > The Left Main mic is always on, and Right Sub mic can be either on 
or
> > off when this occurs.  We are not using AUX or digimic.
> >
> I was thinking is this 2.6.32 derived completely from Vanilla version 
or
> only partially? I think I've seen some case where the kernel was 
forked
> earlier and got upgraded to the upstream version only partially.
>
> Even Peter told that those channel switching patches in 2.6.32 should
> fix the playback case only, worth to check just in case are those 
fixes
> in your kernel tree.

Those fixes are not in our kernel, I will apply them and re-test.

Thanks,
Ying

Re: Channel swapping issue on TI OMAP3/TWL4030

[ylin]

> >
> > Even Peter told that those channel switching patches in 2.6.32 
should
> > fix the playback case only, worth to check just in case are those
> fixes
> > in your kernel tree.
>
> Those fixes are not in our kernel, I will apply them and re-test.

We applied those 2 patches, but still have channel swapping issue from 
capture audio.

Thanks,
Ying

Re: Channel swapping issue on TI OMAP3/TWL4030

[Jarkko Nikula]

On Wed, 16 Feb 2011 22:53:25 -0500
ylin@mail.com wrote:

> > > Even Peter told that those channel switching patches in 2.6.32 
> should
> > > fix the playback case only, worth to check just in case are those
> > fixes
> > > in your kernel tree.
> >
> > Those fixes are not in our kernel, I will apply them and re-test.
> 
> We applied those 2 patches, but still have channel swapping issue from 
> capture audio.
> 
Thanks, this is very good information and seems to indicate the same
problem must be in upstream also as I don't recall that there were any
other swapping related fixes than those that fix the playback case.

-- 
Jarkko

Re: Channel swapping issue on TI OMAP3/TWL4030

[Peter Ujfalusi]

On 02/17/11 10:17, ext Jarkko Nikula wrote:
> Thanks, this is very good information and seems to indicate the same
> problem must be in upstream also as I don't recall that there were any
> other swapping related fixes than those that fix the playback case.

This is the first time I have heard of channel swapping related to
capture on OMAP platforms.
In playback case we can still have corner cases, when channels would
swap/shift, but in capture?
The only thing that I can think of is FIFO overflow, when odd number of
word got discarded, so from application point the channels would swap.

-- 
Péter