* Linux fails to start secondary cores when system resumes from Suspend-to-RAM
@ 2016-12-15 15:18 Mason
2016-12-16 5:14 ` Yu Chen
0 siblings, 1 reply; 6+ messages in thread
From: Mason @ 2016-12-15 15:18 UTC (permalink / raw)
To: linux-arm-kernel
Hello,
I'm playing with suspend-to-RAM on the tango platform:
http://lxr.free-electrons.com/source/arch/arm/mach-tango/platsmp.c
When the system is suspended, the CPU is completely powered down
(receives no power whatsoever). When the system receives a wake-up
event, the CPU is powered up, and starts up exactly the same way
as for a cold boot (I think).
However, while Linux successfully starts the secondary cores when
the system first boots, it fails when the system resumes from "S3".
I added printascii() calls inside secondary_start_kernel() and I can
see that the following instruction are "properly" run:
cpu_switch_mm(mm->pgd, mm);
local_flush_bp_all();
enter_lazy_tlb(mm, current);
but it seems local_flush_tlb_all(); never returns... :-(
http://lxr.free-electrons.com/source/arch/arm/include/asm/tlbflush.h#L332
Looking more closely at that function, it seems to be failing in:
tlb_op(TLB_V7_UIS_FULL, "c8, c7, 0", zero);
(meaning: I get a log before, but not after)
On my system, tlb_op(TLB_V7_UIS_FULL, "c8, c7, 0", zero);
resolves to:
c010ce18: e3170602 tst r7, #2097152 ; 0x200000
c010ce1c: 1e086f17 mcrne 15, 0, r6, cr8, cr7, {0}
What could be happening?
Can a core "hang" on this instruction?
Can a core "crash" on this instruction (meaning, an exception
is raised, and the core loops inside the exception code without
Linux noticing... that seems unlikely)
I'm stumped. Could someone throw me a clue?
Mark Rutland offered a guess about IPIs not working correctly.
Could this explain the behavior I'm seeing?
Regards.
^ permalink raw reply [flat|nested] 6+ messages in thread* Linux fails to start secondary cores when system resumes from Suspend-to-RAM 2016-12-15 15:18 Linux fails to start secondary cores when system resumes from Suspend-to-RAM Mason @ 2016-12-16 5:14 ` Yu Chen 2016-12-16 7:25 ` Mason 0 siblings, 1 reply; 6+ messages in thread From: Yu Chen @ 2016-12-16 5:14 UTC (permalink / raw) To: linux-arm-kernel On Thu, Dec 15, 2016 at 11:18 PM, Mason <slash.tmp@free.fr> wrote: > Hello, > > I'm playing with suspend-to-RAM on the tango platform: > > http://lxr.free-electrons.com/source/arch/arm/mach-tango/platsmp.c > > When the system is suspended, the CPU is completely powered down > (receives no power whatsoever). When the system receives a wake-up > event, the CPU is powered up, and starts up exactly the same way > as for a cold boot (I think). > > However, while Linux successfully starts the secondary cores when > the system first boots, it fails when the system resumes from "S3". > > I added printascii() calls inside secondary_start_kernel() and I can > see that the following instruction are "properly" run: > > cpu_switch_mm(mm->pgd, mm); > local_flush_bp_all(); > enter_lazy_tlb(mm, current); > > but it seems local_flush_tlb_all(); never returns... :-( > > http://lxr.free-electrons.com/source/arch/arm/include/asm/tlbflush.h#L332 > > > Looking more closely at that function, it seems to be failing in: > > tlb_op(TLB_V7_UIS_FULL, "c8, c7, 0", zero); > > (meaning: I get a log before, but not after) > > On my system, tlb_op(TLB_V7_UIS_FULL, "c8, c7, 0", zero); > resolves to: > > c010ce18: e3170602 tst r7, #2097152 ; 0x200000 > c010ce1c: 1e086f17 mcrne 15, 0, r6, cr8, cr7, {0} > > What could be happening? > Can a core "hang" on this instruction? > Can a core "crash" on this instruction (meaning, an exception > is raised, and the core loops inside the exception code without > Linux noticing... that seems unlikely) > > I'm stumped. Could someone throw me a clue? try online/offline the nonboot CPUs via /sys/devices/system/cpu/cpuX/online ^ permalink raw reply [flat|nested] 6+ messages in thread
* Linux fails to start secondary cores when system resumes from Suspend-to-RAM 2016-12-16 5:14 ` Yu Chen @ 2016-12-16 7:25 ` Mason 2016-12-29 12:10 ` Mason 0 siblings, 1 reply; 6+ messages in thread From: Mason @ 2016-12-16 7:25 UTC (permalink / raw) To: linux-arm-kernel On 16/12/2016 06:14, Yu Chen wrote: > On Thu, Dec 15, 2016 at 11:18 PM, Mason wrote: > >> I'm playing with suspend-to-RAM on the tango platform: >> >> http://lxr.free-electrons.com/source/arch/arm/mach-tango/platsmp.c >> >> When the system is suspended, the CPU is completely powered down >> (receives no power whatsoever). When the system receives a wake-up >> event, the CPU is powered up, and starts up exactly the same way >> as for a cold boot (I think). >> >> However, while Linux successfully starts the secondary cores when >> the system first boots, it fails when the system resumes from "S3". >> >> I added printascii() calls inside secondary_start_kernel() and I can >> see that the following instruction are "properly" run: >> >> cpu_switch_mm(mm->pgd, mm); >> local_flush_bp_all(); >> enter_lazy_tlb(mm, current); >> >> but it seems local_flush_tlb_all(); never returns... :-( >> >> http://lxr.free-electrons.com/source/arch/arm/include/asm/tlbflush.h#L332 >> >> >> Looking more closely at that function, it seems to be failing in: >> >> tlb_op(TLB_V7_UIS_FULL, "c8, c7, 0", zero); >> >> (meaning: I get a log before, but not after) >> >> On my system, tlb_op(TLB_V7_UIS_FULL, "c8, c7, 0", zero); >> resolves to: >> >> c010ce18: e3170602 tst r7, #2097152 ; 0x200000 >> c010ce1c: 1e086f17 mcrne 15, 0, r6, cr8, cr7, {0} >> >> What could be happening? >> Can a core "hang" on this instruction? >> Can a core "crash" on this instruction (meaning, an exception >> is raised, and the core loops inside the exception code without >> Linux noticing... that seems unlikely) >> >> I'm stumped. Could someone throw me a clue? > > try online/offline the nonboot CPUs via > /sys/devices/system/cpu/cpuX/online offline + online secondary core works. Note: all cores are in the same power domain, so even if all secondary cores are offline, the CPU block remains powered up (secondary cores are just held in reset, or spinning in WFI, depending on the firmware version). When the system is suspended, the CPU block (as well as 99% of the system) is powered down. Thus, upon resume, all cores will run the boot sequence (again). I'm guessing that something goes wrong during this second boot sequence. Could there be a race condition between the primary core and one of the secondary cores? What is different in the Linux boot sequence between cold boot and resume? I'm thinking that the state stored in RAM is in fact incompatible with what Linux expects when it resumes... Regards. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Linux fails to start secondary cores when system resumes from Suspend-to-RAM 2016-12-16 7:25 ` Mason @ 2016-12-29 12:10 ` Mason 2016-12-29 14:27 ` Mason 0 siblings, 1 reply; 6+ messages in thread From: Mason @ 2016-12-29 12:10 UTC (permalink / raw) To: linux-arm-kernel On 16/12/2016 08:25, Mason wrote: > On 16/12/2016 06:14, Yu Chen wrote: > >> On Thu, Dec 15, 2016 at 11:18 PM, Mason wrote: >> >>> I'm playing with suspend-to-RAM on the tango platform: >>> >>> http://lxr.free-electrons.com/source/arch/arm/mach-tango/platsmp.c >>> >>> When the system is suspended, the CPU is completely powered down >>> (receives no power whatsoever). When the system receives a wake-up >>> event, the CPU is powered up, and starts up exactly the same way >>> as for a cold boot (I think). >>> >>> However, while Linux successfully starts the secondary cores when >>> the system first boots, it fails when the system resumes from "S3". >>> >>> I added printascii() calls inside secondary_start_kernel() and I can >>> see that the following instruction are "properly" run: >>> >>> cpu_switch_mm(mm->pgd, mm); >>> local_flush_bp_all(); >>> enter_lazy_tlb(mm, current); >>> >>> but it seems local_flush_tlb_all(); never returns... :-( >>> >>> http://lxr.free-electrons.com/source/arch/arm/include/asm/tlbflush.h#L332 >>> >>> >>> Looking more closely at that function, it seems to be failing in: >>> >>> tlb_op(TLB_V7_UIS_FULL, "c8, c7, 0", zero); >>> >>> (meaning: I get a log before, but not after) >>> >>> On my system, tlb_op(TLB_V7_UIS_FULL, "c8, c7, 0", zero); >>> resolves to: >>> >>> c010ce18: e3170602 tst r7, #2097152 ; 0x200000 >>> c010ce1c: 1e086f17 mcrne 15, 0, r6, cr8, cr7, {0} >>> >>> What could be happening? >>> Can a core "hang" on this instruction? >>> Can a core "crash" on this instruction (meaning, an exception >>> is raised, and the core loops inside the exception code without >>> Linux noticing... that seems unlikely) >> >> try online/offline the nonboot CPUs via >> /sys/devices/system/cpu/cpuX/online > > offline + online secondary core works. > > Note: all cores are in the same power domain, so even if all > secondary cores are offline, the CPU block remains powered up > (secondary cores are just held in reset, or spinning in WFI, > depending on the firmware version). > > When the system is suspended, the CPU block (as well as 99% > of the system) is powered down. Thus, upon resume, all cores > will run the boot sequence (again). > > I'm guessing that something goes wrong during this second > boot sequence. Could there be a race condition between the > primary core and one of the secondary cores? > > What is different in the Linux boot sequence between cold > boot and resume? I'm thinking that the state stored in RAM > is in fact incompatible with what Linux expects when it resumes... I've taken a closer look at the MCR instruction. tlb_op(TLB_V7_UIS_FULL, "c8, c7, 0", zero); #define tlb_op(f, regs, arg) __tlb_op(f, "p15, 0, %0, " regs, arg) #define __tlb_op(f, insnarg, arg) \ do { \ if (always_tlb_flags & (f)) \ asm("mcr " insnarg \ : : "r" (arg) : "cc"); \ else if (possible_tlb_flags & (f)) \ asm("tst %1, %2\n\t" \ "mcrne " insnarg \ : : "r" (arg), "r" (__tlb_flag), "Ir" (f) \ : "cc"); \ } while (0) c010dd64: e3130c12 tst r3, #4608 ; 0x1200 c010dd68: 1e081f17 mcrne 15, 0, r1, cr8, cr7, {0} c010dd6c: e3120602 tst r2, #2097152 ; 0x200000 c010dd70: 1e081f17 mcrne 15, 0, r1, cr8, cr7, {0} c010dd74: f57ff047 dsb un c010dd78: f57ff06f isb sy A8.6.92 MCR, MCR2 Move to Coprocessor from ARM core register passes the value of an ARM core register to a coprocessor. If no coprocessor can execute the instruction, an Undefined Instruction exception is generated. This is a generic coprocessor instruction. Some of the fields have no functionality defined by the architecture and are free for use by the coprocessor instruction set designer. These fields are the opc1, opc2, CRn, and CRm fields. Operation if ConditionPassed() then EncodingSpecificOperations(); if !Coproc_Accepted(cp, ThisInstr()) then GenerateCoprocessorException(); else Coproc_SendOneWord(R[t], cp, ThisInstr()); I.7.5 Coproc_Accepted() This function determines, for a coprocessor and one of its coprocessor instructions: - Whether access to the coprocessor is permitted by the CPACR and, if the Security Extensions are implemented, the NSACR. - If access is permitted, whether the instruction is accepted by the coprocessor. The coprocessor architecture definition specifies which instructions it accepts and in what circumstances. It returns TRUE if access is permitted and the coprocessor accepts the instruction, and FALSE otherwise. boolean Coproc_Accepted(integer cp_num, bits(32) instr) I.7.17 GenerateCoprocessorException() This procedure generates the appropriate exception for a rejected coprocessor instruction. In all architecture variants and profiles described in this manual, GenerateCoprocessorException() generates an Undefined Instruction exception. Assuming I'm hitting this, I would expect the kernel to print something if the secondary cores trigger an exception. Am I mistaken? mov r1, #0 mcrne 15, 0, r1, cr8, cr7, {0} B3.12.34 CP15 c8, TLB maintenance operations On ARMv7-A implementations, CP15 c8 operations are used for TLB maintenance functions. Figure B3-20 shows the CP15 c8 encodings. TLBIALL*, invalidate unified TLB * See text for more information about these mnemonics Invalidate entire unified TLB (When separate instruction and data TLBs are implemented, these operations are performed on both TLBs.) Rt data = Ignore Invalidate entire TLB The Invalidate entire TLB operations invalidate all unlocked entries in the TLB. The value in the register Rt specified by the MCR instruction used to perform the operation is ignored. You do not have to write a value to the register before issuing the MCR instruction. Maybe this is a red herring... I don't see why the TLBIALL would raise an exception when the system resumes... I first suspected TrustZone shenanigans, but it looks like TLBIALL just flushes what it can, so TrustZone seems to be a non-issue. I'm stumped... Still looking for a clue :-( Maybe I should instrument the exception handler... if I only knew where it was. Regards. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Linux fails to start secondary cores when system resumes from Suspend-to-RAM 2016-12-29 12:10 ` Mason @ 2016-12-29 14:27 ` Mason 2016-12-30 20:10 ` Pavel Machek 0 siblings, 1 reply; 6+ messages in thread From: Mason @ 2016-12-29 14:27 UTC (permalink / raw) To: linux-arm-kernel On Thu, Dec 15, 2016 at 11:18 PM, Mason wrote: > However, while Linux successfully starts the secondary cores when > the system first boots, it fails when the system resumes from "S3". Oh boy... Turns out the firmware was, in fact, (upon resume) stomping over parts of the Linux memory image in RAM, triggering all kinds of "interesting" nasal demons when Linux ran (or, more accurately, limped). The guilty party will be properly dealt with. (Where's my foam bat?) Regards. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Linux fails to start secondary cores when system resumes from Suspend-to-RAM 2016-12-29 14:27 ` Mason @ 2016-12-30 20:10 ` Pavel Machek 0 siblings, 0 replies; 6+ messages in thread From: Pavel Machek @ 2016-12-30 20:10 UTC (permalink / raw) To: linux-arm-kernel Hi! On Thu 2016-12-29 15:27:12, Mason wrote: > On Thu, Dec 15, 2016 at 11:18 PM, Mason wrote: > > > However, while Linux successfully starts the secondary cores when > > the system first boots, it fails when the system resumes from "S3". > > Oh boy... > > Turns out the firmware was, in fact, (upon resume) stomping over parts > of the Linux memory image in RAM, triggering all kinds of "interesting" > nasal demons when Linux ran (or, more accurately, limped). Well... firmware is always fun :-). Good that it got solved... Pavel (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: Digital signature URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20161230/7f8c5b18/attachment.sig> ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2016-12-30 20:10 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2016-12-15 15:18 Linux fails to start secondary cores when system resumes from Suspend-to-RAM Mason 2016-12-16 5:14 ` Yu Chen 2016-12-16 7:25 ` Mason 2016-12-29 12:10 ` Mason 2016-12-29 14:27 ` Mason 2016-12-30 20:10 ` Pavel Machek
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).