PXA270 Random Hangs at Low Core Freq

PXA270 Random Hangs at Low Core Freq

[Michael Cashwell]

Greetings,

I've been fighting inexplicable hangs on two different PXA270 designs running various kernels since early 2.6.28.x. The first board was custom (and of dubious integrity) but I'm currently seeing it on Gumstix Verdex PROs running 2.6.35.7.

At first I thought it was a problem with CPU-freq code itself hitting some unaddressed errata but I've eliminated that possibility. The hangs occur even if CPU-freq is disabled in the kernel config. All that's required in that case is for u-boot to run the kernel at a core frequency of 312MHz or lower. If it runs at 416MHz or higher no hangs occur. (With CPU-freq enabled the hangs only occur if speeds at or below 312MHz are allowed. With those commented out, no hangs.)

At low core frequencies I can trigger the hang every time by copying a several-MB file to a uSD card via the MMC interface. The copy command completes but a few seconds later (while the background processes write to the card) the CPU hangs.** However, the issue is not specific to uSD/MMC. Another developer told me he has seen hangs while receiving data through a UART connected via I2C.

The hangs are hard (rendering JTAG inoperable) and since I have no hardware with the Embedded Trace Module exposed I have no way to gather a trace of execution up to the freeze.

I'd like to solve the problem since the lower speeds are more power efficient but I'm somewhat stuck regarding how to debug it. I'd like to know if it's something I'm doing wrong so my first step is to ask the ARM-Linux community if anyone else has run recent kernels on a PXA270 system at core clock rates of 312MHz or less. (That's also a not so veiled plea for someone to perhaps do so if they have the hardware and a few spare cycles.)

Thanks for any information.
-Mike

**: Here's a hang where I'd just started top while background threads were writing to the uSD card. top did its first pass and then everything froze. (Is the high rate of context switches and interrupts normal?)

last pid:   389;  load avg:  0.13,  0.05,  0.01;  up 0+00:28:15        12:40:22
27 processes: 1 running, 24 sleeping, 2 uninterruptable
CPU states:  0.0% user,  0.0% nice, 31.1% system, 13.6% idle, 55.3% iowait
Kernel: 30832 ctxsw, 16262 intr
Memory: 32M used, 92M free, 244K buffers, 27M cached
Swap: 
 This terminal can only display 16 processes
  PID USERNAME  THR PRI NICE  SIZE   RES   SHR STATE   TIME    CPU COMMAND
  236 root        1  20    0    0K    0K    0K disk    0:00 28.43% mmcqd
  389 root        1  20    0 2696K  804K  648K run     0:00  3.92% top
  378 root        1  20    0    0K    0K    0K disk    0:00  1.96% flush-179:0
    1 root        1  20    0 2784K  672K  588K sleep   0:00  0.00% init
  119 root        1  20    0    0K    0K    0K sleep   0:00  0.00% rpciod/0
  371 root        1  20    0 4208K 1592K 1340K sleep   0:00  0.00% sshd
  369 root        1  20    0 2788K  716K  632K sleep   0:00  0.00% ash
    4 root        1  20    0    0K    0K    0K sleep   0:00  0.00% events/0
  128 root        1  20    0    0K    0K    0K sleep   0:00  0.00% nfsiod
  251 root        1  20    0 2784K  460K  380K sleep   0:00  0.00% klogd
  249 root        1  20    0 2784K  464K  396K sleep   0:00  0.00% syslogd
    5 root        1  20    0    0K    0K    0K sleep   0:00  0.00% khelper
  370 root        1  20    0 2784K  440K  376K sleep   0:00  0.00% busybox
  103 root        1  20    0    0K    0K    0K sleep   0:00  0.00% kmmcd
    2 root        1  20    0    0K    0K    0K sleep   0:00  0.00% kthreadd
    3 root        1  20    0    0K    0K    0K sleep   0:00  0.00% ksoftirqd/0

PXA270 Random Hangs at Low Core Freq

[Marek Vasut]

Dne Po 18. ??jna 2010 19:38:49 Michael Cashwell napsal(a):
> Greetings,
> 
> I've been fighting inexplicable hangs on two different PXA270 designs
> running various kernels since early 2.6.28.x. The first board was custom
> (and of dubious integrity) but I'm currently seeing it on Gumstix Verdex
> PROs running 2.6.35.7.
> 
> At first I thought it was a problem with CPU-freq code itself hitting some
> unaddressed errata but I've eliminated that possibility. The hangs occur
> even if CPU-freq is disabled in the kernel config. All that's required in
> that case is for u-boot to run the kernel at a core frequency of 312MHz or
> lower. If it runs at 416MHz or higher no hangs occur. (With CPU-freq
> enabled the hangs only occur if speeds at or below 312MHz are allowed.
> With those commented out, no hangs.)
> 
> At low core frequencies I can trigger the hang every time by copying a
> several-MB file to a uSD card via the MMC interface. The copy command
> completes but a few seconds later (while the background processes write to
> the card) the CPU hangs.** However, the issue is not specific to uSD/MMC.
> Another developer told me he has seen hangs while receiving data through a
> UART connected via I2C.
> 
> The hangs are hard (rendering JTAG inoperable) and since I have no hardware
> with the Embedded Trace Module exposed I have no way to gather a trace of
> execution up to the freeze.
> 
> I'd like to solve the problem since the lower speeds are more power
> efficient but I'm somewhat stuck regarding how to debug it. I'd like to
> know if it's something I'm doing wrong so my first step is to ask the
> ARM-Linux community if anyone else has run recent kernels on a PXA270
> system at core clock rates of 312MHz or less. (That's also a not so veiled
> plea for someone to perhaps do so if they have the hardware and a few
> spare cycles.)

Could it be memory-related ? Like, RAM crashes because of refresh speed or 
something?

Cheers
> 
> Thanks for any information.
> -Mike
> 
> **: Here's a hang where I'd just started top while background threads were
> writing to the uSD card. top did its first pass and then everything froze.
> (Is the high rate of context switches and interrupts normal?)
> 
> last pid:   389;  load avg:  0.13,  0.05,  0.01;  up 0+00:28:15       
> 12:40:22 27 processes: 1 running, 24 sleeping, 2 uninterruptable
> CPU states:  0.0% user,  0.0% nice, 31.1% system, 13.6% idle, 55.3% iowait
> Kernel: 30832 ctxsw, 16262 intr
> Memory: 32M used, 92M free, 244K buffers, 27M cached
> Swap:
>  This terminal can only display 16 processes
>   PID USERNAME  THR PRI NICE  SIZE   RES   SHR STATE   TIME    CPU COMMAND
>   236 root        1  20    0    0K    0K    0K disk    0:00 28.43% mmcqd
>   389 root        1  20    0 2696K  804K  648K run     0:00  3.92% top
>   378 root        1  20    0    0K    0K    0K disk    0:00  1.96%
> flush-179:0 1 root        1  20    0 2784K  672K  588K sleep   0:00  0.00%
> init 119 root        1  20    0    0K    0K    0K sleep   0:00  0.00%
> rpciod/0 371 root        1  20    0 4208K 1592K 1340K sleep   0:00  0.00%
> sshd 369 root        1  20    0 2788K  716K  632K sleep   0:00  0.00% ash
> 4 root        1  20    0    0K    0K    0K sleep   0:00  0.00% events/0
> 128 root        1  20    0    0K    0K    0K sleep   0:00  0.00% nfsiod
> 251 root        1  20    0 2784K  460K  380K sleep   0:00  0.00% klogd 249
> root        1  20    0 2784K  464K  396K sleep   0:00  0.00% syslogd 5
> root        1  20    0    0K    0K    0K sleep   0:00  0.00% khelper 370
> root        1  20    0 2784K  440K  376K sleep   0:00  0.00% busybox 103
> root        1  20    0    0K    0K    0K sleep   0:00  0.00% kmmcd 2 root 
>       1  20    0    0K    0K    0K sleep   0:00  0.00% kthreadd 3 root    
>    1  20    0    0K    0K    0K sleep   0:00  0.00% ksoftirqd/0
> 
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

PXA270 Random Hangs at Low Core Freq

[Haojian Zhuang]

On Fri, Oct 22, 2010 at 8:22 AM, Marek Vasut <marek.vasut@gmail.com> wrote:
> Dne Po 18. ??jna 2010 19:38:49 Michael Cashwell napsal(a):
>> Greetings,
>>
>> I've been fighting inexplicable hangs on two different PXA270 designs
>> running various kernels since early 2.6.28.x. The first board was custom
>> (and of dubious integrity) but I'm currently seeing it on Gumstix Verdex
>> PROs running 2.6.35.7.
>>
>> At first I thought it was a problem with CPU-freq code itself hitting some
>> unaddressed errata but I've eliminated that possibility. The hangs occur
>> even if CPU-freq is disabled in the kernel config. All that's required in
>> that case is for u-boot to run the kernel at a core frequency of 312MHz or
>> lower. If it runs at 416MHz or higher no hangs occur. (With CPU-freq
>> enabled the hangs only occur if speeds at or below 312MHz are allowed.
>> With those commented out, no hangs.)
>>
>> At low core frequencies I can trigger the hang every time by copying a
>> several-MB file to a uSD card via the MMC interface. The copy command
>> completes but a few seconds later (while the background processes write to
>> the card) the CPU hangs.** However, the issue is not specific to uSD/MMC.
>> Another developer told me he has seen hangs while receiving data through a
>> UART connected via I2C.
>>
>> The hangs are hard (rendering JTAG inoperable) and since I have no hardware
>> with the Embedded Trace Module exposed I have no way to gather a trace of
>> execution up to the freeze.
>>
>> I'd like to solve the problem since the lower speeds are more power
>> efficient but I'm somewhat stuck regarding how to debug it. I'd like to
>> know if it's something I'm doing wrong so my first step is to ask the
>> ARM-Linux community if anyone else has run recent kernels on a PXA270
>> system at core clock rates of 312MHz or less. (That's also a not so veiled
>> plea for someone to perhaps do so if they have the hardware and a few
>> spare cycles.)
>
> Could it be memory-related ? Like, RAM crashes because of refresh speed or
> something?
>
> Cheers
>>
>> Thanks for any information.
>> -Mike
>>
>> **: Here's a hang where I'd just started top while background threads were
>> writing to the uSD card. top did its first pass and then everything froze.
>> (Is the high rate of context switches and interrupts normal?)
>>
>> last pid: ? 389; ?load avg: ?0.13, ?0.05, ?0.01; ?up 0+00:28:15
>> 12:40:22 27 processes: 1 running, 24 sleeping, 2 uninterruptable
>> CPU states: ?0.0% user, ?0.0% nice, 31.1% system, 13.6% idle, 55.3% iowait
>> Kernel: 30832 ctxsw, 16262 intr
>> Memory: 32M used, 92M free, 244K buffers, 27M cached
>> Swap:
>> ?This terminal can only display 16 processes
>> ? PID USERNAME ?THR PRI NICE ?SIZE ? RES ? SHR STATE ? TIME ? ?CPU COMMAND
>> ? 236 root ? ? ? ?1 ?20 ? ?0 ? ?0K ? ?0K ? ?0K disk ? ?0:00 28.43% mmcqd
>> ? 389 root ? ? ? ?1 ?20 ? ?0 2696K ?804K ?648K run ? ? 0:00 ?3.92% top
>> ? 378 root ? ? ? ?1 ?20 ? ?0 ? ?0K ? ?0K ? ?0K disk ? ?0:00 ?1.96%
>> flush-179:0 1 root ? ? ? ?1 ?20 ? ?0 2784K ?672K ?588K sleep ? 0:00 ?0.00%
>> init 119 root ? ? ? ?1 ?20 ? ?0 ? ?0K ? ?0K ? ?0K sleep ? 0:00 ?0.00%
>> rpciod/0 371 root ? ? ? ?1 ?20 ? ?0 4208K 1592K 1340K sleep ? 0:00 ?0.00%
>> sshd 369 root ? ? ? ?1 ?20 ? ?0 2788K ?716K ?632K sleep ? 0:00 ?0.00% ash
>> 4 root ? ? ? ?1 ?20 ? ?0 ? ?0K ? ?0K ? ?0K sleep ? 0:00 ?0.00% events/0
>> 128 root ? ? ? ?1 ?20 ? ?0 ? ?0K ? ?0K ? ?0K sleep ? 0:00 ?0.00% nfsiod
>> 251 root ? ? ? ?1 ?20 ? ?0 2784K ?460K ?380K sleep ? 0:00 ?0.00% klogd 249
>> root ? ? ? ?1 ?20 ? ?0 2784K ?464K ?396K sleep ? 0:00 ?0.00% syslogd 5
>> root ? ? ? ?1 ?20 ? ?0 ? ?0K ? ?0K ? ?0K sleep ? 0:00 ?0.00% khelper 370
>> root ? ? ? ?1 ?20 ? ?0 2784K ?440K ?376K sleep ? 0:00 ?0.00% busybox 103
>> root ? ? ? ?1 ?20 ? ?0 ? ?0K ? ?0K ? ?0K sleep ? 0:00 ?0.00% kmmcd 2 root
>> ? ? ? 1 ?20 ? ?0 ? ?0K ? ?0K ? ?0K sleep ? 0:00 ?0.00% kthreadd 3 root
>> ? ?1 ?20 ? ?0 ? ?0K ? ?0K ? ?0K sleep ? 0:00 ?0.00% ksoftirqd/0
>>
>>

Suggest to check errata first. Maybe we need some special code sequence.

PXA270 Random Hangs at Low Core Freq

[Michael Cashwell]

On Oct 22, 2010, at 4:41 AM, Haojian Zhuang wrote:

> On Fri, Oct 22, 2010 at 8:22 AM, Marek Vasut <marek.vasut@gmail.com> wrote:
>> Dne Po 18. ??jna 2010 19:38:49 Michael Cashwell napsal(a):
>> 
>> 
>>> I've been fighting inexplicable hangs on two different PXA270 designs running various kernels since early 2.6.28.x. The first board was custom (and of dubious integrity) but I'm currently seeing it on Gumstix Verdex PROs running 2.6.35.7.
>> 
>> Could it be memory-related ? Like, RAM crashes because of refresh speed or something?

I wondered about this myself, especially on the custom board and u-boot. That's why I hadn't turned to the community until I was on unmodified commercial hardware.

What I'd like to test would be a Gumstix-supplied uboot and current Linux with CPUfreq disabled. That would eliminate all the CPUfreq code and its plethora of errata from consideration. The problem is that such a setup runs at a fixed *high* CPU core frequency. I've never seen that hang.

The best I've been able to do is two alternate cases (stock high-freq u-boot + a current Linux with CPUfreq enabled so that it lowers the speed; and my custom u-boot that runs only at low speed + a current Linux with CPUfreq disabled so linux just uses what u-boot sets). Both of these hang as described.

Meaningfully rolling back to the Gumstix's vendor-supplied Linux distro is more difficult. It's running a 3+ year old 2.6.21 kernel variant that lacks support for recent MTD FLASH parts and features like UBIFS. I used that version on an OMAP and recall its MMC support was rather sketchy and since writing to a uSD card is my test case I'm somewhat doubtful that the results from such testing would mean much.

Lastly, I've run Charles Cazabon's memtester (c. 2006 v 4.0.6) over 99% of the free SDRAM for days at a time and have seen no problems at all. If it were an outright SDRAM init/timing/refresh issue I'd expect something that strenuous to report an error.

> Suggest to check errata first. Maybe we need some special code sequence.

I agree that it has the feel of a CPU errata. (What else would hang the core so badly that JTAG would disconnect?) So I've re-read the Rev E (April 3rd 2009) errata docs from Marvell from start to finish but nothing jumps out at me. The problem is not knowing where to focus. Having eliminated the CPU-freq code itself (which relies on CPU features with many errata) I'm left rather empty handed.

My only idea is that some kernel code path (likely interacting with an integrated peripheral) is too slow at low core frequencies and is either violating the hardware spec outright or is hitting an errata.

My testing to date (and that I'm on commercial hardware) makes me think that others should be seeing the same problem. It won't occur out-of-the-box because normally people want highest performance (high core CPU frequencies). With the default u-boot and without CPU-freq to lower it the hang doesn't happen.

But perhaps someone has prioritized power savings as I'm trying to do and then they should see it.

-Mike

PXA270 Random Hangs at Low Core Freq

[Eric Miao]

On Tue, Oct 26, 2010 at 3:02 AM, Michael Cashwell <mboards@prograde.net> wrote:
> On Oct 22, 2010, at 4:41 AM, Haojian Zhuang wrote:
>
>> On Fri, Oct 22, 2010 at 8:22 AM, Marek Vasut <marek.vasut@gmail.com> wrote:
>>> Dne Po 18. ??jna 2010 19:38:49 Michael Cashwell napsal(a):
>>>
>>>
>>>> I've been fighting inexplicable hangs on two different PXA270 designs running various kernels since early 2.6.28.x. The first board was custom (and of dubious integrity) but I'm currently seeing it on Gumstix Verdex PROs running 2.6.35.7.
>>>
>>> Could it be memory-related ? Like, RAM crashes because of refresh speed or something?
>
> I wondered about this myself, especially on the custom board and u-boot. That's why I hadn't turned to the community until I was on unmodified commercial hardware.
>
> What I'd like to test would be a Gumstix-supplied uboot and current Linux with CPUfreq disabled. That would eliminate all the CPUfreq code and its plethora of errata from consideration. The problem is that such a setup runs at a fixed *high* CPU core frequency. I've never seen that hang.
>
> The best I've been able to do is two alternate cases (stock high-freq u-boot + a current Linux with CPUfreq enabled so that it lowers the speed; and my custom u-boot that runs only at low speed + a current Linux with CPUfreq disabled so linux just uses what u-boot sets). Both of these hang as described.
>
> Meaningfully rolling back to the Gumstix's vendor-supplied Linux distro is more difficult. It's running a 3+ year old 2.6.21 kernel variant that lacks support for recent MTD FLASH parts and features like UBIFS. I used that version on an OMAP and recall its MMC support was rather sketchy and since writing to a uSD card is my test case I'm somewhat doubtful that the results from such testing would mean much.
>
> Lastly, I've run Charles Cazabon's memtester (c. 2006 v 4.0.6) over 99% of the free SDRAM for days at a time and have seen no problems at all. If it were an outright SDRAM init/timing/refresh issue I'd expect something that strenuous to report an error.
>
>> Suggest to check errata first. Maybe we need some special code sequence.
>
> I agree that it has the feel of a CPU errata. (What else would hang the core so badly that JTAG would disconnect?) So I've re-read the Rev E (April 3rd 2009) errata docs from Marvell from start to finish but nothing jumps out at me. The problem is not knowing where to focus. Having eliminated the CPU-freq code itself (which relies on CPU features with many errata) I'm left rather empty handed.
>
> My only idea is that some kernel code path (likely interacting with an integrated peripheral) is too slow at low core frequencies and is either violating the hardware spec outright or is hitting an errata.
>
> My testing to date (and that I'm on commercial hardware) makes me think that others should be seeing the same problem. It won't occur out-of-the-box because normally people want highest performance (high core CPU frequencies). With the default u-boot and without CPU-freq to lower it the hang doesn't happen.
>
> But perhaps someone has prioritized power savings as I'm trying to do and then they should see it.
>

Several suggestions:

1) Check the status of the CPU frequency related registers, e.g. CCCR, CCLKCFG
and so on

2) Check the HW - like external power/voltage changes happen when switching
frequencies?

3) If by any chance it's a fake hang, try SysRq or JTAG hot debug to see where
it is stuck exactly.

> -Mike
>
>

PXA270 Random Hangs at Low Core Freq

[Marek Vasut]

On Monday 25 October 2010 21:02:42 Michael Cashwell wrote:
> On Oct 22, 2010, at 4:41 AM, Haojian Zhuang wrote:
> > On Fri, Oct 22, 2010 at 8:22 AM, Marek Vasut <marek.vasut@gmail.com> wrote:
> >> Dne Po 18. ??jna 2010 19:38:49 Michael Cashwell napsal(a):
> >>> I've been fighting inexplicable hangs on two different PXA270 designs
> >>> running various kernels since early 2.6.28.x. The first board was
> >>> custom (and of dubious integrity) but I'm currently seeing it on
> >>> Gumstix Verdex PROs running 2.6.35.7.
> >> 
> >> Could it be memory-related ? Like, RAM crashes because of refresh speed
> >> or something?
> 
> I wondered about this myself, especially on the custom board and u-boot.
> That's why I hadn't turned to the community until I was on unmodified
> commercial hardware.
> 
> What I'd like to test would be a Gumstix-supplied uboot and current Linux
> with CPUfreq disabled. That would eliminate all the CPUfreq code and its
> plethora of errata from consideration. The problem is that such a setup
> runs at a fixed *high* CPU core frequency. I've never seen that hang.
> 
> The best I've been able to do is two alternate cases (stock high-freq
> u-boot + a current Linux with CPUfreq enabled so that it lowers the speed;
> and my custom u-boot that runs only at low speed + a current Linux with
> CPUfreq disabled so linux just uses what u-boot sets). Both of these hang
> as described.
> 
> Meaningfully rolling back to the Gumstix's vendor-supplied Linux distro is
> more difficult. It's running a 3+ year old 2.6.21 kernel variant that
> lacks support for recent MTD FLASH parts and features like UBIFS. I used
> that version on an OMAP and recall its MMC support was rather sketchy and
> since writing to a uSD card is my test case I'm somewhat doubtful that the
> results from such testing would mean much.
> 
> Lastly, I've run Charles Cazabon's memtester (c. 2006 v 4.0.6) over 99% of
> the free SDRAM for days at a time and have seen no problems at all. If it
> were an outright SDRAM init/timing/refresh issue I'd expect something that
> strenuous to report an error.
> 
> > Suggest to check errata first. Maybe we need some special code sequence.
> 
> I agree that it has the feel of a CPU errata. (What else would hang the
> core so badly that JTAG would disconnect?) So I've re-read the Rev E
> (April 3rd 2009) errata docs from Marvell from start to finish but nothing
> jumps out at me. The problem is not knowing where to focus. Having
> eliminated the CPU-freq code itself (which relies on CPU features with
> many errata) I'm left rather empty handed.
> 
I think I noticed something similar running ZipitZ2 on 312MHz ... yet I was 
unable to figure out the problem as well ...

> My only idea is that some kernel code path (likely interacting with an
> integrated peripheral) is too slow at low core frequencies and is either
> violating the hardware spec outright or is hitting an errata.
> 
> My testing to date (and that I'm on commercial hardware) makes me think
> that others should be seeing the same problem. It won't occur
> out-of-the-box because normally people want highest performance (high core
> CPU frequencies). With the default u-boot and without CPU-freq to lower it
> the hang doesn't happen.
> 
> But perhaps someone has prioritized power savings as I'm trying to do and
> then they should see it.
> 
> -Mike

PXA270 Random Hangs at Low Core Freq

[Michael Cashwell]

On Oct 25, 2010, at 3:46 PM, Marek Vasut wrote:

> I think I noticed something similar running ZipitZ2 on 312MHz ... yet I was unable to figure out the problem as well ...

Out of curiosity, what was your trigger event? MMC activity or something else?

-Mike

PXA270 Random Hangs at Low Core Freq

[Haojian Zhuang]

On Tue, Oct 26, 2010 at 3:22 AM, Eric Miao <eric.y.miao@gmail.com> wrote:
> On Tue, Oct 26, 2010 at 3:02 AM, Michael Cashwell <mboards@prograde.net> wrote:
>> On Oct 22, 2010, at 4:41 AM, Haojian Zhuang wrote:
>>
>>> On Fri, Oct 22, 2010 at 8:22 AM, Marek Vasut <marek.vasut@gmail.com> wrote:
>>>> Dne Po 18. ??jna 2010 19:38:49 Michael Cashwell napsal(a):
>>>>
>>>>
>>>>> I've been fighting inexplicable hangs on two different PXA270 designs running various kernels since early 2.6.28.x. The first board was custom (and of dubious integrity) but I'm currently seeing it on Gumstix Verdex PROs running 2.6.35.7.
>>>>
>>>> Could it be memory-related ? Like, RAM crashes because of refresh speed or something?
>>
>> I wondered about this myself, especially on the custom board and u-boot. That's why I hadn't turned to the community until I was on unmodified commercial hardware.
>>
>> What I'd like to test would be a Gumstix-supplied uboot and current Linux with CPUfreq disabled. That would eliminate all the CPUfreq code and its plethora of errata from consideration. The problem is that such a setup runs at a fixed *high* CPU core frequency. I've never seen that hang.
>>
>> The best I've been able to do is two alternate cases (stock high-freq u-boot + a current Linux with CPUfreq enabled so that it lowers the speed; and my custom u-boot that runs only at low speed + a current Linux with CPUfreq disabled so linux just uses what u-boot sets). Both of these hang as described.
>>
>> Meaningfully rolling back to the Gumstix's vendor-supplied Linux distro is more difficult. It's running a 3+ year old 2.6.21 kernel variant that lacks support for recent MTD FLASH parts and features like UBIFS. I used that version on an OMAP and recall its MMC support was rather sketchy and since writing to a uSD card is my test case I'm somewhat doubtful that the results from such testing would mean much.
>>
>> Lastly, I've run Charles Cazabon's memtester (c. 2006 v 4.0.6) over 99% of the free SDRAM for days at a time and have seen no problems at all. If it were an outright SDRAM init/timing/refresh issue I'd expect something that strenuous to report an error.
>>
>>> Suggest to check errata first. Maybe we need some special code sequence.
>>
>> I agree that it has the feel of a CPU errata. (What else would hang the core so badly that JTAG would disconnect?) So I've re-read the Rev E (April 3rd 2009) errata docs from Marvell from start to finish but nothing jumps out at me. The problem is not knowing where to focus. Having eliminated the CPU-freq code itself (which relies on CPU features with many errata) I'm left rather empty handed.
>>
>> My only idea is that some kernel code path (likely interacting with an integrated peripheral) is too slow at low core frequencies and is either violating the hardware spec outright or is hitting an errata.
>>
>> My testing to date (and that I'm on commercial hardware) makes me think that others should be seeing the same problem. It won't occur out-of-the-box because normally people want highest performance (high core CPU frequencies). With the default u-boot and without CPU-freq to lower it the hang doesn't happen.
>>
>> But perhaps someone has prioritized power savings as I'm trying to do and then they should see it.
>>
>
> Several suggestions:
>
> 1) Check the status of the CPU frequency related registers, e.g. CCCR, CCLKCFG
> and so on
>
> 2) Check the HW - like external power/voltage changes happen when switching
> frequencies?
>
> 3) If by any chance it's a fake hang, try SysRq or JTAG hot debug to see where
> it is stuck exactly.
>
While you're debugging, try to disable idle mode. Use fake idle since
it may block JTAG using.

Thanks
Haojian

PXA270 Random Hangs at Low Core Freq

[Marek Vasut]

On Monday 25 October 2010 22:53:32 Michael Cashwell wrote:
> On Oct 25, 2010, at 3:46 PM, Marek Vasut wrote:
> > I think I noticed something similar running ZipitZ2 on 312MHz ... yet I
> > was unable to figure out the problem as well ...
> 
> Out of curiosity, what was your trigger event? MMC activity or something
> else?

I believe it might have been MMC activity, as in initramfs it wasn't triggered
> 
> -Mike

PXA270 Random Hangs at Low Core Freq

[Michael Cashwell]

On Oct 25, 2010, at 3:22 PM, Eric Miao wrote:

> Several suggestions:
> 
> 1) Check the status of the CPU frequency related registers, e.g. CCCR, CCLKCFG and so on

OK. I've decomposed those in the past but here's what I presently am using with CPUfreq disabled:

CCSR: 30000190, MDREFR: 200fc031, CLKCFG: 0000000b
Run Mode clock: 208.00MHz (*16)
Turbo Mode clock: 312.00MHz (*1.5, active)
Half-Turbo Mode clock: 156.00MHz (*3, inactive)
Memory clock: 208.00MHz
System bus clock: 208.00MHz 
SD clock 0: disabled
SD clock 1: 104.00MHz 
SD clock 2: 104.00MHz 
SD clock 3: disabled

It's worth noting that all of these are set by the bootloader and if CPUfreq is disabled I think the kernel leaves them alone. (This is ignoring the code paths for suspend/resume. My kernel does not use these as the concept of suspend does not apply to our application.)

> 2) Check the HW - like external power/voltage changes happen when switching frequencies?

Trying to investigate the problem along these lines is precisely why I have disabled CPUfreq and the voltage regulators yet the hangs still happen. (All that's needed is to be running at 312MHz or less.) So whatever is happening is not due to *changing* the core frequency or voltage.

Voltage slewing is an area I had been working on. The TPS65023 driver has some problems (like failing to actually command a voltage change!). I'm going to submit patches for this but it's not involved in the problem it's disabled in my kernel config. The Verdex Pro PMIC is strapped to come out of reset at a core voltage of 1.55VDC (verified on a scope). That's high enough to support all core speeds.

> 3) If by any chance it's a fake hang, try SysRq ...

That's on and when running normally a "break-t" shows me all thread states as expected. When hung it does nothing.

> 3) ... or JTAG hot debug to see where it is stuck exactly.

I'm using a Lauterbach PowerTrace. I can pause/run and while paused I can see memory, stacks, registers and the PXA peripherals. When it hangs the JTAG software complains that the debug port has "timed out" when I attempt to pause. No JTAG operation other than sys.down works at that point.

I'm unfamiliar with "hot debug" but if that means to logically attach to a running processor then (like SysRq) it works in general but is non-responsive when attempted on a hung CPU.

-Mike

PXA270 Random Hangs at Low Core Freq

[Pavel Machek]

Hi!

> > The best I've been able to do is two alternate cases (stock high-freq u-boot + a current Linux with CPUfreq enabled so that it lowers the speed; and my custom u-boot that runs only at low speed + a current Linux with CPUfreq disabled so linux just uses what u-boot sets). Both of these hang as described.
> >
> > Meaningfully rolling back to the Gumstix's vendor-supplied Linux distro is more difficult. It's running a 3+ year old 2.6.21 kernel variant that lacks support for recent MTD FLASH parts and features like UBIFS. I used that version on an OMAP and recall its MMC support was rather sketchy and since writing to a uSD card is my test case I'm somewhat doubtful that the results from such testing would mean much.
> >
> > Lastly, I've run Charles Cazabon's memtester (c. 2006 v 4.0.6) over 99% of the free SDRAM for days at a time and have seen no problems at all. If it were an outright SDRAM init/timing/refresh issue I'd expect something that strenuous to report an error.
> >
> >> Suggest to check errata first. Maybe we need some special code sequence.
> >
> > I agree that it has the feel of a CPU errata. (What else would hang the core so badly that JTAG would disconnect?) So I've re-read the Rev E (April 3rd 2009) errata docs from Marvell from start to finish but nothing jumps out at me. The problem is not knowing where to focus. Having eliminated the CPU-freq code itself (which relies on CPU features with many errata) I'm left rather empty handed.
> >
> > My only idea is that some kernel code path (likely interacting with an integrated peripheral) is too slow at low core frequencies and is either violating the hardware spec outright or is hitting an errata.
> >
> > My testing to date (and that I'm on commercial hardware) makes me think that others should be seeing the same problem. It won't occur out-of-the-box because normally people want highest performance (high core CPU frequencies). With the default u-boot and without CPU-freq to lower it the hang doesn't happen.
> >
> > But perhaps someone has prioritized power savings as I'm trying to do and then they should see it.
> >

ZAurus Spitz (pxa270) has always had problems -- inside suspend memory
is corrupted fairly often, and MMC produces high number of bit errors
-- like one in 20MB.

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

PXA270 Random Hangs at Low Core Freq

[Robert Jarzmik]

Marek Vasut <marek.vasut@gmail.com> writes:

> On Monday 25 October 2010 22:53:32 Michael Cashwell wrote:
>> On Oct 25, 2010, at 3:46 PM, Marek Vasut wrote:
>> > I think I noticed something similar running ZipitZ2 on 312MHz ... yet I
>> > was unable to figure out the problem as well ...
>> 
>> Out of curiosity, what was your trigger event? MMC activity or something
>> else?
>
> I believe it might have been MMC activity, as in initramfs it wasn't triggered

On the mioa701 with a PXA272, the problem occurs as well in the lowest frequency
(104 MHz). If I block the frequency at 208MHz, no hang occurs.

The hang occured several times in the boot sequence, before rootfs was even
loaded.

This happens even with no SD card inserted, with USB gadget driver loading the
CPU, and then idling.

--
Robert

PXA270 Random Hangs at Low Core Freq

[Vasily Khoruzhick]

On Tuesday 26 October 2010 20:06:36 Michael Cashwell wrote:

> Voltage slewing is an area I had been working on. The TPS65023 driver has
> some problems (like failing to actually command a voltage change!). I'm
> going to submit patches for this but it's not involved in the problem it's
> disabled in my kernel config. The Verdex Pro PMIC is strapped to come out
> of reset at a core voltage of 1.55VDC (verified on a scope). That's high
> enough to support all core speeds.

Hi, Michael, can you share your patches for TPS65023?

Regards
Vasily

PXA270 Random Hangs at Low Core Freq

[Michael Cashwell]

On Feb 3, 2011, at 7:04 AM, Vasily Khoruzhick wrote:

> On Tuesday 26 October 2010 20:06:36 Michael Cashwell wrote:
> 
>> Voltage slewing is an area I had been working on. The TPS65023 driver has some problems (like failing to actually command a voltage change!). I'm going to submit patches for this but it's not involved in the problem as it's disabled in my kernel config. The Verdex Pro PMIC is strapped to come out of reset at a core voltage of 1.55VDC (verified on a scope). That's high enough to support all core speeds.
> 
> Hi, Michael, can you share your patches for TPS65023?

Sure! Happy to. It applies cleanly at least up to 2.6.35.7 which is what I'm actually currently using.

There's certainly room for improvement given the ugly udelay() but at least it works. (My main idea for improvement would be to actually do the math to calculate the voltage delta and wait only the needed time rather than the full-swing worst case.)

-Mike

diff -Nurp linux-2.6.33.2/drivers/regulator/tps65023-regulator.c linux-2.6.33.2-arm-gum-MOD/drivers/regulator/tps65023-regulator.c
--- linux-2.6.33.2/drivers/regulator/tps65023-regulator.c	2010-04-01 19:02:33.000000000 -0400
+++ linux-2.6.33.2-arm-gum-MOD/drivers/regulator/tps65023-regulator.c	2010-05-12 13:57:27.000000000 -0400
@@ -126,6 +126,7 @@ struct tps_pmic {
 	struct regulator_dev *rdev[TPS65023_NUM_REGULATOR];
 	const struct tps_info *info[TPS65023_NUM_REGULATOR];
 	struct mutex io_lock;
+	int last_core_index;
 };
 
 static inline int tps_65023_read(struct tps_pmic *tps, u8 reg)
@@ -325,7 +326,7 @@ static int tps65023_dcdc_set_voltage(str
 {
 	struct tps_pmic *tps = rdev_get_drvdata(dev);
 	int dcdc = rdev_get_id(dev);
-	int vsel;
+	int i, data, vsel;
 
 	if (dcdc != TPS65023_DCDC_1)
 		return -EINVAL;
@@ -346,11 +347,63 @@ static int tps65023_dcdc_set_voltage(str
 			break;
 	}
 
-	/* write to the register in case we found a match */
+	/* it's an error if no acceptable entry was found */
 	if (vsel == tps->info[dcdc]->table_len)
 		return -EINVAL;
-	else
-		return tps_65023_reg_write(tps, TPS65023_REG_DEF_CORE, vsel);
+
+	/* set new target voltage index */
+	data = tps_65023_reg_write(tps, TPS65023_REG_DEF_CORE, vsel);
+	if (data < 0)
+		return data;
+
+	/* read CTRL2 */
+	data = tps_65023_reg_read(tps, TPS65023_REG_CON_CTRL2);
+	if (data < 0)
+		return data;
+
+	/* error if GO bit is still set from a previous op */
+	if (data & BIT(7))
+		return -EIO;
+
+	/* set the GO bit while preserving only the discharge enables */
+	data = BIT(7) | (data & 7);
+
+	data = tps_65023_reg_write(tps, TPS65023_REG_CON_CTRL2, data);
+	if (data < 0)
+		return data;
+
+	/* and if we're raising the voltage, wait for the PMIC to process it.
+	otherwise we return too soon and the CPU frequency increase that's
+	likely being paired with this step up in voltage could happen before
+	the voltage actually increases. this can cause core failures. */
+
+	if (vsel > tps->last_core_index) {
+		/* first, wait for the PMIC to clear the GO bit.
+		that indicates the voltage slew has started. */
+		i = 4;	/* in practice, one read seems to be enough */
+		do {
+			i--;
+			data = tps_65023_reg_read(tps, TPS65023_REG_CON_CTRL2);
+			if (data < 0)
+				return data;
+		} while (i && (data & BIT(7)));
+
+		if (i == 0)
+			return -EIO;
+
+		/* it would be nice if the PMIC provided an indication for
+		when slewing is complete. but alas, no. so we wait long
+		enough to cover the worst-case.
+		slewing 550mV@the default rate of 14.4mV/uS is about
+		40uS. but that's just the setpoint slew. full output
+		regulation takes even longer due to filter caps. */
+		udelay(100);
+	}
+	
+	/* note the new index for next time. */
+	tps->last_core_index = vsel;
+
+	return 0;
 }
 
 static int tps65023_ldo_get_voltage(struct regulator_dev *dev)

PXA270 Random Hangs at Low Core Freq

[Vasily Khoruzhick]

Hi, Michael, recently I discovered that PXA270 (at least in Zipit Z2) works 
unstable (just hangs randomly) with fastbus bit set in CCLKCFG, check if 
bootloader sets it on your board, and if it does try to clear it.

Regards
Vasily

PXA270 Random Hangs at Low Core Freq

[Michael Cashwell]

On Mar 4, 2011, at 11:16 AM, Vasily Khoruzhick wrote:

> Hi, Michael, recently I discovered that PXA270 (at least in Zipit Z2) works unstable (just hangs randomly) with fastbus bit set in CCLKCFG, check if bootloader sets it on your board, and if it does try to clear it.

Hi Vasily,

Thanks for the info. The curious thing is that arch/arm/mach-pxa/cpufreq-pxa2xx.c has a data table called pxa27x_freqs[] that seems to have FASTBUS set to true for all frequencies. So if CPUFREQ is enabled (which I do to save power) the kernel will enable FASTBUS even if the boot loader does not.

The boot loader would control that feature only if CPUFREQ were turned off (either in the config or dynamically). Is that the case you mean? If you are not using CPUFREQ in Linux what fixed frequency are you running?

What I've ultimately done is comment out all the lower frequencies. If it only switches between 416, 520 and 624 MHz it works fine. It only hangs for me if I also allow the lower frequencies, and interestingly, the lower I allow the sooner the hang happens. It's unfortunate since the lower frequencies would save more power but I simply cannot make them work reliably.

I've still not been able to identify why this happens (largely since JTAG becomes unresponsive) or find any published errata that I'm not accounting for.

-Mike

PXA270 Random Hangs at Low Core Freq

[Vasily Khoruzhick]

On Friday 04 March 2011 20:42:17 Michael Cashwell wrote:
 
> Hi Vasily,
> 
> Thanks for the info. The curious thing is that
> arch/arm/mach-pxa/cpufreq-pxa2xx.c has a data table called pxa27x_freqs[]
> that seems to have FASTBUS set to true for all frequencies. So if CPUFREQ
> is enabled (which I do to save power) the kernel will enable FASTBUS even
> if the boot loader does not.

Yep, cpufreq was unstable on Zipit Z2, but after fixing pxa27x_freqs table to 
disable fastbus bit it works OK.
 
> The boot loader would control that feature only if CPUFREQ were turned off
> (either in the config or dynamically). Is that the case you mean? If you
> are not using CPUFREQ in Linux what fixed frequency are you running?
> 
> What I've ultimately done is comment out all the lower frequencies. If it
> only switches between 416, 520 and 624 MHz it works fine. It only hangs
> for me if I also allow the lower frequencies, and interestingly, the lower
> I allow the sooner the hang happens. It's unfortunate since the lower
> frequencies would save more power but I simply cannot make them work
> reliably.

Can you check if lower freqs work for you with fastbit not set?

> I've still not been able to identify why this happens (largely since JTAG
> becomes unresponsive) or find any published errata that I'm not accounting
> for.

Maybe this errata is not documented yet? :)

> -Mike

Regards
Vasily

PXA270 Random Hangs at Low Core Freq

[Michael Cashwell]

On Mar 4, 2011, at 1:51 PM, Vasily Khoruzhick wrote:

> Can you check if lower freqs work for you with fastbit not set?
> 
>> I've still not been able to identify why this happens (largely since JTAG becomes unresponsive) or find any published errata that I'm not accounting for.
> 
> Maybe this errata is not documented yet? :)

Hi Vasily,

Sorry this took so long. Swamped on another project...

With my local fix for an errata regarding changing memclk during a frequency change (I actually have a lot of local patches to this file) I've found the following table works correctly:

static pxa_freqs_t pxa27x_freqs[] = {
       /* CPU   MEMCLK  CCCR        A   L N2  DIV2         B HT  T     uVmin    uVmax */
       {104000,  52000, PXA27x_CCCR(1,  8, 2), 0, CCLKCFG2(0, 0, 1), 1000000, 1705000 },
       {156000,  52000, PXA27x_CCCR(1,  8, 3), 0, CCLKCFG2(0, 0, 1), 1000000, 1705000 },
       {208000, 104000, PXA27x_CCCR(1, 16, 2), 0, CCLKCFG2(0, 0, 1), 1180000, 1705000 },
       {312000, 104000, PXA27x_CCCR(1, 16, 3), 0, CCLKCFG2(0, 0, 1), 1250000, 1705000 },
       {416000, 208000, PXA27x_CCCR(1, 16, 4), 1, CCLKCFG2(1, 0, 1), 1350000, 1705000 },
       {520000, 208000, PXA27x_CCCR(1, 16, 5), 1, CCLKCFG2(1, 0, 1), 1450000, 1705000 },
       {624000, 208000, PXA27x_CCCR(1, 16, 6), 1, CCLKCFG2(1, 0, 1), 1550000, 1705000 }
};

The 52MHz memclk points aren't listed in the PXA dev manual but I've never been clear if that table is a complete list of all valid points or just a partial listing. At any rate, lowering the SDRAM clock in the low-speed case is certainly within the SDRAM specs, so I went with it.

I've run days of tests and found no more hangs. And when idle at 104MHz instead of the previous stable minimum of 416MHz I'm seeing about a 7% power savings. Everyone was very happy about this.

You could try my data table above or perhaps I need to clean up and actually submit all of my changes in this file as a proper patch.

To that end, I think I've read in Marvel docs that HT should never be used while performing a freq. change. That agrees with the table above since HT is never set. I also note that the Turbo mode (T) is always set as is A (which makes the memclk the same as sysbus).

Is there any point in cluttering the code with options that are not used? If I did submit a real patch wouldn't removing all the T, HT, and A stuff make sense? Just an idea.

-Mike