Re: [RESEND PATCH 0/1] Fix the "hard LOCKUP" when running a heavy loading

Will Deacon
2015-11-06T12:17:44Z

On Tue, Nov 03, 2015 at 11:00:20AM -0800, Doug Anderson wrote:
> Hi,

Hey Doug,

> When CPUs are hard locked up, they are often found at:
>
> <c0117c8c> v7_coherent_kern_range+0x58/0x74
> or
> <c0118278> v7wbi_flush_user_tlb_range+0x30/0x38
>
> That made me think that an errata might be the root cause of our hard
> lockups, since ARM errata often trigger in cache/tlb functions. I
> think Caesar dug up this old errata fix in response to my suggestion.

I still don't see how 818325 is related, since there aren't any conditional
stores in the sequences below.

> If you know of any ARM errata that might trigger hard lockups like
> this, I'd certainly be all ears. It's also possible that we've got
> something running at too low of a voltage or we've got clock dividers
> or cache timings programmed incorrectly somewhere. To give a more
> full disassembly of one of the crashes:
>
> <4>[ 1623.480846] SMP: failed to stop secondary CPUs
> <3>[ 1623.480862] CPU1 PC: <c01827e8> __unqueue_futex+0x68/0x88
> <3>[ 1623.480879] CPU2 PC: <c0117c8c> v7_coherent_kern_range+0x58/0x74
> <3>[ 1623.480895] CPU3 PC: <c0118268> v7wbi_flush_user_tlb_range+0x20/0x38
>
> ---

Do you have any register values for these CPUs?

> c01827dc: e2841010 add r1, r4, #16
> c01827e0: e2445004 sub r5, r4, #4
> c01827e4: eb068d33 bl c0325cb8 <plist_del> (File
> Offset: 0x235cb8)
> => c01827e8: f595f000 pldw [r5]
> c01827ec: e1953f9f ldrex r3, [r5]
> c01827f0: e2433001 sub r3, r3, #1
> c01827f4: e1852f93 strex r2, r3, [r5]
> c01827f8: e3320000 teq r2, #0
> c01827fc: 1afffffa bne c01827ec
> <__unqueue_futex+0x6c> (File Offset: 0x927ec)
> c0182800: e89da830 ldm sp, {r4, r5, fp, sp, pc}

For example, the futex address in r5 ...

> c0117c80: e08cc002 add ip, ip, r2
> c0117c84: e15c0001 cmp ip, r1
> c0117c88: 3afffffb bcc c0117c7c
> <v7_coherent_kern_range+0x48> (File Offset: 0x27c7c)
> => c0117c8c: e3a00000 mov r0, #0
> c0117c90: ee070fd1 mcr 15, 0, r0, cr7, cr1, {6}
> c0117c94: f57ff04a dsb ishst
> c0117c98: f57ff06f isb sy
> c0117c9c: e1a0f00e mov pc, lr

... the address in r0 for the cache maintenance ...

> c0118260: e1830600 orr r0, r3, r0, lsl #12
> c0118264: e1a01601 lsl r1, r1, #12
> => c0118268: ee080f33 mcr 15, 0, r0, cr8, cr3, {1}
> c011826c: e2800a01 add r0, r0, #4096 ; 0x1000
> c0118270: e1500001 cmp r0, r1
> c0118274: 3afffffb bcc c0118268
> <v7wbi_flush_user_tlb_range+0x20> (File Offset: 0x28268)
> c0118278: f57ff04b dsb ish
> c011827c: e1a0f00e mov pc, lr

... and the address in r0 for the TLBI.

Are the cores executing instructions at this point, or by "hard LOCKUP"
do you mean that they're deadlocked in hardware?

Will