RT tests on OMAP3

RT tests on OMAP3

[Cliff Brake]

Hello,

I'm trying to run 2.6.31-rc8-rt9-omap1 on a OMAP3 cpu.  So far, it
seems fairly unstable:

http://www.bec-systems.com/omap-rt-tests/

2.6.31-rc8-rt9 boots and runs well on a PXA270.

The following mail also references udev failures:
http://lkml.org/lkml/2009/8/11/250

Any suggestions on how to debug this?  I'd be glad to do testing, etc.

The OMAP3 is superscaler where the PXA270 is not.

Thanks,
Cliff

Re: RT tests on OMAP3

[Cliff Brake]

On Mon, Aug 31, 2009 at 4:50 PM, Cliff Brake<cliff.brake@gmail.com> wrote:
> Hello,
>
> I'm trying to run 2.6.31-rc8-rt9-omap1 on a OMAP3 cpu.  So far, it
> seems fairly unstable:
>
> http://www.bec-systems.com/omap-rt-tests/
>
> 2.6.31-rc8-rt9 boots and runs well on a PXA270.
>
> The following mail also references udev failures:
> http://lkml.org/lkml/2009/8/11/250

A few more observations:

Applying

http://cgit.openembedded.org/cgit.cgi/openembedded/diff/recipes/linux/linux-omap-2.6.29/0001-implement-TIF_RESTORE_SIGMASK-support-and-enable-the.patch?id=1f0d91e152f16fbd40bb2fb3c44a30d774d4dede

Seems to fix or mask the udev problems.  But, with preempt-rt, and
EHCI enabled, I still get messages continually scrolling:

<7>ehci-omap ehci-omap.0: detected XactErr len 0/2048 retry 27
ehci-omap ehci-omap.0: detected XactErr len 0/2048 retry 28
ehci-omap ehci-omap.0: detected XactErr len 0/2048 retry 29
ehci-omap ehci-omap.0: detected XactErr len 0/2048 retry 30
ehci-omap ehci-omap.0: detected XactErr len 0/2048 retry 31
ehci-omap ehci-omap.0: devpath 2.1 ep2in 3strikes
ehci-omap ehci-omap.0: detected XactErr len 0/2048 retry 1
ehci-oeteehci-omap ehci-omap.0: detected XactErr len 0/2048 retry 27
ehci-omap ehci-omap.0: detected XactErr len 0/2048 retry 28
ehci-omap ehci-omap.0: detected XactErr len 0/2048 retry 29
ehci-omap ehci-omap.0: detected XactErr len 0/2048 retry 30
ehci-omap ehci-omap.0: detected XactErr len 0/2048 retry 31
ehci-omap ehci-omap.0: devpath 2.1 ep2in 3strikes
ehci-omap ehci-omap.0: detected XactErr -omap.0: detected XactErr len
0/2048 retry 8
ehci-omap ehci-omap.0: detected XactErr len 0/2048 retry 9
ehci-omap ehci-omap.0: detected XactErr len 0/2048 retry 10
ehci-omap ehci-omap.0: detected XactErr len 0/2048 retry 11

The same hardware works fine without the RT patch.

I got the following message early in the boot sequence:

Starting udevusb%d: Error reading RX_CTL register: ffffffb9
usb%d: Failed to write RX_CTL mode to 0x0000: ffffffb9
asix: probe of 2-2.1:1.0 failed with error -71
hub 2-2:1.0: hub_port_status failed (err = -71)
hub 2-0:1.0: port 2 disabled by hub (EMI?), re-enabling...
usb 2-2: USB disconnect, address 2
usb 2-2.1: USB disconnect, address 3
Unable to handle kernel NULL pointer dereference at virtual address
00000014
usb 2-2: new high speed USB device using ehci-omap and address 4
pgd = cf050000
[00000014] *pgd=8f040031, *pte=00000000, *ppte=00000000
Internal error: Oops: 0 [#1] PREEMPT
Modules linked in:
CPU: 0    Not tainted  (2.6.31-rc8-rt9-omap1 #12)
PC is at 0x14
LR is at block_nodename+0x24/0x2c
pc : [<00000014>]    lr : [<c01d3c4c>]    psr: 20000033
sp : cfaafe98  ip : 00000007  fp : 00000000
r10: c054dd28  r9 : cf024620  r8 : cf80cb60
r7 : cf983658  r6 : cf983650  r5 : cfaafebc  r4 : cf983650
r3 : 00000015  r2 : cf983600  r1 : cfaafebc  r0 : cf983600
Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA Thumb  Segment user
Control: 10c5387d  Table: 8f050019  DAC: 00000015
Process udevadm (pid: 606, stack limit = 0xcfaae2f0)
Stack: (0xcfaafe98 to 0xcfab0000)
fe80:                                                       00000000
c0234ed0
fea0: cf983658 cf983650 cf8c6000 cf8c6000 cf983658 c0234fc4 cf983650
00000000
fec0: c01dfc10 00000000 00000003 c01dfc78 00000000 cfaafee0 c03d6aac
00000000
fee0: c048d034 c0497048 00000000 cfaafef8 cf8bbec4 00000000 00000000
cf983650
ff00: 00000003 cf95b160 cf983658 c054dd68 cf95b178 cfaaff80 0001eb30
c0235128
ff20: cf08c6e0 00000000 cf95b178 00000003 cfb64758 c02346ac cfb64758
c0113908
ff40: cf08c6e0 0001df0c cfaaff80 0001df0c 00000003 cfaae000 00026df0
c00c9070
ff60: cf564d00 00000020 00000000 00000000 cf08c6e0 0001df0c 00000003
c00c91d4
ff80: 00000000 00000000 00000000 00000000 0002cf58 00000003 bef8b178
00000004
ffa0: c0033084 c0032f00 0002cf58 00000003 00000003 0001df0c 00000003
00000000
ffc0: 0002cf58 00000003 bef8b178 00000004 00026df4 00027008 00026df0
0001eb30
ffe0: 0001df0c bef8b168 0001a888 400d8bfc 40000010 00000003 41ffcf00
00fbcf00
Code: bad PC value.
usb 2-2: udev 4, busnum 2, minor = 131
usb 2-2: New USB device found, idVendor=2001, idProduct=f103
usb 2-2: New USB device strings: Mfr=0, Product=0, SerialNumber=0
usb 2-2: uevent

It makes sense that enabling EHCI may trigger some more udev activity.

Cliff
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RT tests on OMAP3

[Cliff Brake]

On Mon, Aug 31, 2009 at 11:16 PM, Cliff Brake<cliff.brake@gmail.com> wrote:

> Applying
>
> http://cgit.openembedded.org/cgit.cgi/openembedded/diff/recipes/linux/linux-omap-2.6.29/0001-implement-TIF_RESTORE_SIGMASK-support-and-enable-the.patch?id=1f0d91e152f16fbd40bb2fb3c44a30d774d4dede
>
> Seems to fix or mask the udev problems.  But, with preempt-rt, and
> EHCI enabled, I still get messages continually scrolling:

A few more notes, the above patch is not required to make the PXA270
work.  Without EHCI enabled, the system boots and performs fairly
normal (I'm able to log in, etc).

Cliff
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RT tests on OMAP3

[Kevin Hilman]

Cliff Brake <cliff.brake@gmail.com> writes:

> On Mon, Aug 31, 2009 at 11:16 PM, Cliff Brake<cliff.brake@gmail.com> wrote:
>
>> Applying
>>
>> http://cgit.openembedded.org/cgit.cgi/openembedded/diff/recipes/linux/linux-omap-2.6.29/0001-implement-TIF_RESTORE_SIGMASK-support-and-enable-the.patch?id=1f0d91e152f16fbd40bb2fb3c44a30d774d4dede
>>
>> Seems to fix or mask the udev problems.  But, with preempt-rt, and
>> EHCI enabled, I still get messages continually scrolling:
>
> A few more notes, the above patch is not required to make the PXA270
> work.  Without EHCI enabled, the system boots and performs fairly
> normal (I'm able to log in, etc).

This looks to me like a case of poor (or non-existent) locking in the
USB/EHCI driver, resulting in some corruption since the PC is at a
bogus address (0x14).

Does your PXA270 use the same EHCI driver?  I assume not.

I've been using -rt on a basic OMAP3 kernel, but have not enabled
EHCI so haven't seen this.

Might be useful to enable some of the in-kernel lock debugging to
see if it turns up anything.

Kevin

--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RT tests on OMAP3

[Cliff Brake]

On Thu, Sep 3, 2009 at 10:28 AM, Kevin
Hilman<khilman@deeprootsystems.com> wrote:

> I've been using -rt on a basic OMAP3 kernel, but have not enabled
> EHCI so haven't seen this.

What kernel version are you running, and what are you using for your
timer config?

# CONFIG_OMAP_MBOX_FWK is not set
# CONFIG_OMAP_MPU_TIMER is not set
CONFIG_OMAP_32K_TIMER=y
CONFIG_OMAP_32K_TIMER_HZ=128
CONFIG_OMAP_DM_TIMER=y

Thanks,
Cliff

Re: RT tests on OMAP3

[Kevin Hilman]

Cliff Brake <cliff.brake@gmail.com> writes:

> On Thu, Sep 3, 2009 at 10:28 AM, Kevin
> Hilman<khilman@deeprootsystems.com> wrote:
>
>> I've been using -rt on a basic OMAP3 kernel, but have not enabled
>> EHCI so haven't seen this.
>
> What kernel version are you running, and what are you using for your
> timer config?
>
> # CONFIG_OMAP_MBOX_FWK is not set
> # CONFIG_OMAP_MPU_TIMER is not set
> CONFIG_OMAP_32K_TIMER=y
> CONFIG_OMAP_32K_TIMER_HZ=128
> CONFIG_OMAP_DM_TIMER=y

I'm using the 32K timer as you are for PM reasons.  The 32k timer is
in the wakeup powerdomain so is not disabled during low power
idle/suspend.

Kevin

Re: RT tests on OMAP3

[Koen Kooi]

Op 1 sep 2009, om 05:16 heeft Cliff Brake het volgende geschreven:

> On Mon, Aug 31, 2009 at 4:50 PM, Cliff Brake<cliff.brake@gmail.com>  
> wrote:
>> Hello,
>>
>> I'm trying to run 2.6.31-rc8-rt9-omap1 on a OMAP3 cpu.  So far, it
>> seems fairly unstable:
>>
>> http://www.bec-systems.com/omap-rt-tests/
>>
>> 2.6.31-rc8-rt9 boots and runs well on a PXA270.
>>
>> The following mail also references udev failures:
>> http://lkml.org/lkml/2009/8/11/250
>
> A few more observations:
>
> Applying
>
> http://cgit.openembedded.org/cgit.cgi/openembedded/diff/recipes/linux/linux-omap-2.6.29/0001-implement-TIF_RESTORE_SIGMASK-support-and-enable-the.patch?id=1f0d91e152f16fbd40bb2fb3c44a30d774d4dede

If people want to push that upstream, please do so, I only updated the  
patch to a more recent kernel but don't have the time nor skills to  
make it suitable for upstream.

regards,

KOen

Re: RT tests on OMAP3

[Cliff Brake]

On Thu, Sep 3, 2009 at 10:44 AM, Koen Kooi<k.kooi@student.utwente.nl> wrote:
>>
>> http://cgit.openembedded.org/cgit.cgi/openembedded/diff/recipes/linux/linux-omap-2.6.29/0001-implement-TIF_RESTORE_SIGMASK-support-and-enable-the.patch?id=1f0d91e152f16fbd40bb2fb3c44a30d774d4dede
>
> If people want to push that upstream, please do so, I only updated the patch
> to a more recent kernel but don't have the time nor skills to make it
> suitable for upstream.

I think this is already queued:

http://www.spinics.net/lists/arm-kernel/msg72676.html

Cliff