RFC: Proper suspend-to-ram implementation of Ingenic SoCs

RFC: Proper suspend-to-ram implementation of Ingenic SoCs

[Mike Yang]

The suspend-to-ram implementation of Ingenic SoCs in the current kernel is nowhere near usable, especially for the X series SoCs. Since it involves turning off CPU core power and putting DRAM into self-refresh mode, things are a bit complicated. Turning off CPU core power means all register files and cache contents are lost. Putting DRAM into self-refresh mode means it will no longer respond to bus transactions.

I ported the implementation from Ingenic's 3.10 kernel to 5.18, and it worked. But it involves a separate piece of executable code, and apparently there's no way to eliminate it. During pm_enter(), various CPM registers are configured to turn off CPU core and put DRAM into self-refresh upon issuing the "wait" instruction, this piece of executable code will be copied to the on-chip SRAM, and its entry address will be written into the CPM.SLPC register. Then, cache will be flushed and CPU register files (incl. CP0, CP1 stuff) will also be saved in the SRAM. Finally, the "wait" instruction will be issued, and the suspend procedure completed. When any external events trigger a resume, the CPU is powered on, and immediately jumps to the PC stored in CPM.SLPC, and starts executing the piece of code. The code will perform the usual crt0 stuff on MIPS machines, reconfigure the DRAM into normal mode, and finally restore the register files. Then the control flow goes back to pm_enter(), and the resume procedure is completed.

The suspend-to-ram really saves a lot of power. For my particular board, the idle power consumption is about 0.24W (1.25V Vcore, 1.2GHz, 1000Hz, preempt). After suspend-to-ram, it drops to only 0.045W.

So here are my questions:
1. I don't see a way to eliminate the piece of executable code in SRAM. Is there any other ways?
2. If we can't eliminate the code in SRAM, what's the accepted way of integrating it into the kernel tree?
3. If the hardware doesn't have 32k crystal connected, or the RTC is stripped off (e.g. X1501), some CPM registers need to be configured differently. How could we provide this configuration?


Regards,
Mike

Re: RFC: Proper suspend-to-ram implementation of Ingenic SoCs

[Paul Cercueil]

Hi Mike,

Le mer., juil. 13 2022 at 03:19:32 +0800, Mike Yang 
<reimu@sudomaker.com> a écrit :
> The suspend-to-ram implementation of Ingenic SoCs in the current 
> kernel is nowhere near usable, especially for the X series SoCs. 
> Since it involves turning off CPU core power and putting DRAM into 
> self-refresh mode, things are a bit complicated. Turning off CPU core 
> power means all register files and cache contents are lost. Putting 
> DRAM into self-refresh mode means it will no longer respond to bus 
> transactions.

Suspend-to-RAM is well-tested and has been working fine for ages on all 
JZ SoCs, so I wouldn't call it "nowhere near usable". Zhou also 
implemented it on X-series SoCs.

> I ported the implementation from Ingenic's 3.10 kernel to 5.18, and 
> it worked. But it involves a separate piece of executable code, and 
> apparently there's no way to eliminate it. During pm_enter(), various 
> CPM registers are configured to turn off CPU core and put DRAM into 
> self-refresh upon issuing the "wait" instruction, this piece of 
> executable code will be copied to the on-chip SRAM, and its entry 
> address will be written into the CPM.SLPC register. Then, cache will 
> be flushed and CPU register files (incl. CP0, CP1 stuff) will also be 
> saved in the SRAM. Finally, the "wait" instruction will be issued, 
> and the suspend procedure completed. When any external events trigger 
> a resume, the CPU is powered on, and immediately jumps to the PC 
> stored in CPM.SLPC, and starts executing the piece of code. The code 
> will perform the usual crt0 stuff on MIPS machines, reconfigure the 
> DRAM into normal mode, and finally restore the register files. Then 
> the control flow goes back to pm_enter(), and the resume procedure is 
> completed.

This sounds extremely complex and way overkill. But you don't need any 
of this.

> The suspend-to-ram really saves a lot of power. For my particular 
> board, the idle power consumption is about 0.24W (1.25V Vcore, 
> 1.2GHz, 1000Hz, preempt). After suspend-to-ram, it drops to only 
> 0.045W.

Yes, doesn't surprise me. The RG-350 (JZ4770 based) can last about ~6 
hours of up-time, and when put  to sleep it will survive a few weeks.

> So here are my questions:
> 1. I don't see a way to eliminate the piece of executable code in 
> SRAM. Is there any other ways?

There is what's already implemented, yes. When triggering a suspend, 
the CPM.LCR.LPM setting is set to SLEEP mode 
(drivers/clk/ingenic/pm.c), then the ingenic_pm_enter() function 
(arch/mips/generic/board-ingenic.c) just executes the "wait" CPU 
instruction to put the CPU to sleep. All clocks but the RTC one are 
disabled until an interrupt is raised.

> 2. If we can't eliminate the code in SRAM, what's the accepted way of 
> integrating it into the kernel tree?

Already upstream :)

> 3. If the hardware doesn't have 32k crystal connected, or the RTC is 
> stripped off (e.g. X1501), some CPM registers need to be configured 
> differently. How could we provide this configuration?

It's already supported. The RTC clock can be re-parented (in device 
tree) to the EXT/512 clock, which is (as its name suggests) derived 
from the external EXT oscillator.

Hopefully I answered all your questions.

Cheers,
-Paul

Re: RFC: Proper suspend-to-ram implementation of Ingenic SoCs

[Mike Yang]

Hi Paul,

Thanks for the information.


On 7/13/22 04:28, Paul Cercueil wrote:
> Hi Mike,
> 
> Le mer., juil. 13 2022 at 03:19:32 +0800, Mike Yang <reimu@sudomaker.com> a écrit :
>> The suspend-to-ram implementation of Ingenic SoCs in the current kernel is nowhere near usable, especially for the X series SoCs. Since it involves turning off CPU core power and putting DRAM into self-refresh mode, things are a bit complicated. Turning off CPU core power means all register files and cache contents are lost. Putting DRAM into self-refresh mode means it will no longer respond to bus transactions.
> 
> Suspend-to-RAM is well-tested and has been working fine for ages on all JZ SoCs, so I wouldn't call it "nowhere near usable". Zhou also implemented it on X-series SoCs.

With the vanilla 5.18 kernel, the system will simply become unresponsive after typing "echo mem > /sys/power/state". It won't respond to WKUP and other interrupt-enabled GPIO pins. The power consumption is a bit lower, but nowhere near 0.045W. The behavior is the same for X1000(E) and X1501.

I asked Dr. Zhou about this in person and he said he never tested the suspend-to-ram, nor he confirmed it working.

> 
>> I ported the implementation from Ingenic's 3.10 kernel to 5.18, and it worked. But it involves a separate piece of executable code, and apparently there's no way to eliminate it. During pm_enter(), various CPM registers are configured to turn off CPU core and put DRAM into self-refresh upon issuing the "wait" instruction, this piece of executable code will be copied to the on-chip SRAM, and its entry address will be written into the CPM.SLPC register. Then, cache will be flushed and CPU register files (incl. CP0, CP1 stuff) will also be saved in the SRAM. Finally, the "wait" instruction will be issued, and the suspend procedure completed. When any external events trigger a resume, the CPU is powered on, and immediately jumps to the PC stored in CPM.SLPC, and starts executing the piece of code. The code will perform the usual crt0 stuff on MIPS machines, reconfigure the DRAM into normal mode, and finally restore the register files. Then the control flow goes back to
>> pm_enter(), and the resume procedure is completed.
> 
> This sounds extremely complex and way overkill. But you don't need any of this.
> 
>> The suspend-to-ram really saves a lot of power. For my particular board, the idle power consumption is about 0.24W (1.25V Vcore, 1.2GHz, 1000Hz, preempt). After suspend-to-ram, it drops to only 0.045W.
> 
> Yes, doesn't surprise me. The RG-350 (JZ4770 based) can last about ~6 hours of up-time, and when put  to sleep it will survive a few weeks.
> 
>> So here are my questions:
>> 1. I don't see a way to eliminate the piece of executable code in SRAM. Is there any other ways?
> 
> There is what's already implemented, yes. When triggering a suspend, the CPM.LCR.LPM setting is set to SLEEP mode (drivers/clk/ingenic/pm.c), then the ingenic_pm_enter() function (arch/mips/generic/board-ingenic.c) just executes the "wait" CPU instruction to put the CPU to sleep. All clocks but the RTC one are disabled until an interrupt is raised.
> 
>> 2. If we can't eliminate the code in SRAM, what's the accepted way of integrating it into the kernel tree?
> 
> Already upstream :)
> 
>> 3. If the hardware doesn't have 32k crystal connected, or the RTC is stripped off (e.g. X1501), some CPM registers need to be configured differently. How could we provide this configuration?
> 
> It's already supported. The RTC clock can be re-parented (in device tree) to the EXT/512 clock, which is (as its name suggests) derived from the external EXT oscillator.
> 
> Hopefully I answered all your questions.
> 
> Cheers,
> -Paul
> 
> 

I'm afraid the above didn't work for me. Have you tested suspend-to-ram in person on a X series SoC?


Regards,
Mike

Re: RFC: Proper suspend-to-ram implementation of Ingenic SoCs

[Paul Cercueil]

Le mer., juil. 13 2022 at 04:51:05 +0800, Mike Yang 
<reimu@sudomaker.com> a écrit :
> Hi Paul,
> 
> Thanks for the information.
> 
> 
> On 7/13/22 04:28, Paul Cercueil wrote:
>>  Hi Mike,
>> 
>>  Le mer., juil. 13 2022 at 03:19:32 +0800, Mike Yang 
>> <reimu@sudomaker.com> a écrit :
>>>  The suspend-to-ram implementation of Ingenic SoCs in the current 
>>> kernel is nowhere near usable, especially for the X series SoCs. 
>>> Since it involves turning off CPU core power and putting DRAM into 
>>> self-refresh mode, things are a bit complicated. Turning off CPU 
>>> core power means all register files and cache contents are lost. 
>>> Putting DRAM into self-refresh mode means it will no longer respond 
>>> to bus transactions.
>> 
>>  Suspend-to-RAM is well-tested and has been working fine for ages on 
>> all JZ SoCs, so I wouldn't call it "nowhere near usable". Zhou also 
>> implemented it on X-series SoCs.
> 
> With the vanilla 5.18 kernel, the system will simply become 
> unresponsive after typing "echo mem > /sys/power/state". It won't 
> respond to WKUP and other interrupt-enabled GPIO pins. The power 
> consumption is a bit lower, but nowhere near 0.045W. The behavior is 
> the same for X1000(E) and X1501.

Well, do you know why it fails? Did you try to debug it? Does it still 
become unresponsive if you comment the "wait" instruction?

It's a bit early to talk about power consumption if it doesn't suspend 
properly yet.

> I asked Dr. Zhou about this in person and he said he never tested the 
> suspend-to-ram, nor he confirmed it working.
> 
>> 
>>>  I ported the implementation from Ingenic's 3.10 kernel to 5.18, 
>>> and it worked. But it involves a separate piece of executable code, 
>>> and apparently there's no way to eliminate it. During pm_enter(), 
>>> various CPM registers are configured to turn off CPU core and put 
>>> DRAM into self-refresh upon issuing the "wait" instruction, this 
>>> piece of executable code will be copied to the on-chip SRAM, and 
>>> its entry address will be written into the CPM.SLPC register. Then, 
>>> cache will be flushed and CPU register files (incl. CP0, CP1 stuff) 
>>> will also be saved in the SRAM. Finally, the "wait" instruction 
>>> will be issued, and the suspend procedure completed. When any 
>>> external events trigger a resume, the CPU is powered on, and 
>>> immediately jumps to the PC stored in CPM.SLPC, and starts 
>>> executing the piece of code. The code will perform the usual crt0 
>>> stuff on MIPS machines, reconfigure the DRAM into normal mode, and 
>>> finally restore the register files. Then the control flow goes back 
>>> to
>>>  pm_enter(), and the resume procedure is completed.
>> 
>>  This sounds extremely complex and way overkill. But you don't need 
>> any of this.
>> 
>>>  The suspend-to-ram really saves a lot of power. For my particular 
>>> board, the idle power consumption is about 0.24W (1.25V Vcore, 
>>> 1.2GHz, 1000Hz, preempt). After suspend-to-ram, it drops to only 
>>> 0.045W.
>> 
>>  Yes, doesn't surprise me. The RG-350 (JZ4770 based) can last about 
>> ~6 hours of up-time, and when put  to sleep it will survive a few 
>> weeks.
>> 
>>>  So here are my questions:
>>>  1. I don't see a way to eliminate the piece of executable code in 
>>> SRAM. Is there any other ways?
>> 
>>  There is what's already implemented, yes. When triggering a 
>> suspend, the CPM.LCR.LPM setting is set to SLEEP mode 
>> (drivers/clk/ingenic/pm.c), then the ingenic_pm_enter() function 
>> (arch/mips/generic/board-ingenic.c) just executes the "wait" CPU 
>> instruction to put the CPU to sleep. All clocks but the RTC one are 
>> disabled until an interrupt is raised.
>> 
>>>  2. If we can't eliminate the code in SRAM, what's the accepted way 
>>> of integrating it into the kernel tree?
>> 
>>  Already upstream :)
>> 
>>>  3. If the hardware doesn't have 32k crystal connected, or the RTC 
>>> is stripped off (e.g. X1501), some CPM registers need to be 
>>> configured differently. How could we provide this configuration?
>> 
>>  It's already supported. The RTC clock can be re-parented (in device 
>> tree) to the EXT/512 clock, which is (as its name suggests) derived 
>> from the external EXT oscillator.
>> 
>>  Hopefully I answered all your questions.
>> 
>>  Cheers,
>>  -Paul
>> 
>> 
> 
> I'm afraid the above didn't work for me. Have you tested 
> suspend-to-ram in person on a X series SoC?

I didn't test on X-series, I mostly work with JZ. But that part of the 
design didn't change since the JZ4740.

Cheers,
-Paul

Re: RFC: Proper suspend-to-ram implementation of Ingenic SoCs

[Mike Yang]

Hi Paul,


On 7/13/22 06:20, Paul Cercueil wrote:
> 
> 
> Le mer., juil. 13 2022 at 04:51:05 +0800, Mike Yang <reimu@sudomaker.com> a écrit :
>> Hi Paul,
>>
>> Thanks for the information.
>>
>>
>> On 7/13/22 04:28, Paul Cercueil wrote:
>>>  Hi Mike,
>>>
>>>  Le mer., juil. 13 2022 at 03:19:32 +0800, Mike Yang <reimu@sudomaker.com> a écrit :
>>>>  The suspend-to-ram implementation of Ingenic SoCs in the current kernel is nowhere near usable, especially for the X series SoCs. Since it involves turning off CPU core power and putting DRAM into self-refresh mode, things are a bit complicated. Turning off CPU core power means all register files and cache contents are lost. Putting DRAM into self-refresh mode means it will no longer respond to bus transactions.
>>>
>>>  Suspend-to-RAM is well-tested and has been working fine for ages on all JZ SoCs, so I wouldn't call it "nowhere near usable". Zhou also implemented it on X-series SoCs.
>>
>> With the vanilla 5.18 kernel, the system will simply become unresponsive after typing "echo mem > /sys/power/state". It won't respond to WKUP and other interrupt-enabled GPIO pins. The power consumption is a bit lower, but nowhere near 0.045W. The behavior is the same for X1000(E) and X1501.
> 
> Well, do you know why it fails? Did you try to debug it? Does it still become unresponsive if you comment the "wait" instruction?
> 
> It's a bit early to talk about power consumption if it doesn't suspend properly yet.

If I comment the "wait" instruction, it will exit the suspend process immediately. And yes, I don't think it suspended properly.

> 
>> I asked Dr. Zhou about this in person and he said he never tested the suspend-to-ram, nor he confirmed it working.
>>
>>>
>>>>  I ported the implementation from Ingenic's 3.10 kernel to 5.18, and it worked. But it involves a separate piece of executable code, and apparently there's no way to eliminate it. During pm_enter(), various CPM registers are configured to turn off CPU core and put DRAM into self-refresh upon issuing the "wait" instruction, this piece of executable code will be copied to the on-chip SRAM, and its entry address will be written into the CPM.SLPC register. Then, cache will be flushed and CPU register files (incl. CP0, CP1 stuff) will also be saved in the SRAM. Finally, the "wait" instruction will be issued, and the suspend procedure completed. When any external events trigger a resume, the CPU is powered on, and immediately jumps to the PC stored in CPM.SLPC, and starts executing the piece of code. The code will perform the usual crt0 stuff on MIPS machines, reconfigure the DRAM into normal mode, and finally restore the register files. Then the control flow goes back to
>>>>  pm_enter(), and the resume procedure is completed.
>>>
>>>  This sounds extremely complex and way overkill. But you don't need any of this.
>>>
>>>>  The suspend-to-ram really saves a lot of power. For my particular board, the idle power consumption is about 0.24W (1.25V Vcore, 1.2GHz, 1000Hz, preempt). After suspend-to-ram, it drops to only 0.045W.
>>>
>>>  Yes, doesn't surprise me. The RG-350 (JZ4770 based) can last about ~6 hours of up-time, and when put  to sleep it will survive a few weeks.
>>>
>>>>  So here are my questions:
>>>>  1. I don't see a way to eliminate the piece of executable code in SRAM. Is there any other ways?
>>>
>>>  There is what's already implemented, yes. When triggering a suspend, the CPM.LCR.LPM setting is set to SLEEP mode (drivers/clk/ingenic/pm.c), then the ingenic_pm_enter() function (arch/mips/generic/board-ingenic.c) just executes the "wait" CPU instruction to put the CPU to sleep. All clocks but the RTC one are disabled until an interrupt is raised.
>>>
>>>>  2. If we can't eliminate the code in SRAM, what's the accepted way of integrating it into the kernel tree?
>>>
>>>  Already upstream :)
>>>
>>>>  3. If the hardware doesn't have 32k crystal connected, or the RTC is stripped off (e.g. X1501), some CPM registers need to be configured differently. How could we provide this configuration?
>>>
>>>  It's already supported. The RTC clock can be re-parented (in device tree) to the EXT/512 clock, which is (as its name suggests) derived from the external EXT oscillator.
>>>
>>>  Hopefully I answered all your questions.
>>>
>>>  Cheers,
>>>  -Paul
>>>
>>>
>>
>> I'm afraid the above didn't work for me. Have you tested suspend-to-ram in person on a X series SoC?
> 
> I didn't test on X-series, I mostly work with JZ. But that part of the design didn't change since the JZ4740.
> 
> Cheers,
> -Paul
> 
> 


To be honest, I never owned a board with a JZ series SoC. And sorry for assuming the suspend-to-ram is unusable on all Ingenic SoCs. IIRC, all the JZ series SoCs have external DRAM, while the X series SoCs have internal DRAM. Also Ingenic advertised the power saving features of the X series SoCs heavily. Things might be different since it may involve additional power management.



At the time of writing the last sentence of the email, Dr. Zhou just pointed out that it may has something to do with the secure boot feature introduced in the X series SoC, although the feature is not enabled. I already mailed my X1000E & X1501 boards to Dr. Zhou for further tests. You may want to get a X1000(E) board (e.g. halley2) and test this by yourself.


Regards,
Mike

Re: RFC: Proper suspend-to-ram implementation of Ingenic SoCs

[Paul Cercueil]

Hi Mike,
[...]

> If I comment the "wait" instruction, it will exit the suspend process 
> immediately. And yes, I don't think it suspended properly.

Ok. I was suggesting to try that since it would show if the crash 
happens when a particular device gets suspended.

Are you certain that your wakeup IRQ is unmasked?

[...]

>>>>  I'm afraid the above didn't work for me. Have you tested 
>>>> suspend-to-ram in person on a X series SoC?
>> 
>>  I didn't test on X-series, I mostly work with JZ. But that part of 
>> the design didn't change since the JZ4740.
>> 
>>  Cheers,
>>  -Paul
>> 
>> 
> 
> 
> To be honest, I never owned a board with a JZ series SoC. And sorry 
> for assuming the suspend-to-ram is unusable on all Ingenic SoCs. 
> IIRC, all the JZ series SoCs have external DRAM, while the X series 
> SoCs have internal DRAM. Also Ingenic advertised the power saving 
> features of the X series SoCs heavily. Things might be different 
> since it may involve additional power management.

Even if the 3.x method you were describing works, the currently 
upstream method should work as well, and if it doesn't, we probably 
should try to figure why.

I remember doing some tests on the JZ4770 some years ago, and I would 
get a power consumption of 7mA when suspended - that's for the whole 
board, measured at the 3.7V battery, so about 0.026 W. The only things 
powered ON then are the RAM chips and the SoC's RTC module.

> At the time of writing the last sentence of the email, Dr. Zhou just 
> pointed out that it may has something to do with the secure boot 
> feature introduced in the X series SoC, although the feature is not 
> enabled. I already mailed my X1000E & X1501 boards to Dr. Zhou for 
> further tests. You may want to get a X1000(E) board (e.g. halley2) 
> and test this by yourself.

I do have a Cu1000-Neo board, but I have never used it, I wouldn't know 
how to test this.

Cheers,
-Paul

Re: RFC: Proper suspend-to-ram implementation of Ingenic SoCs

[Zhou Yanjie]

Hi Paul,

On 2022/7/14 上午12:08, Paul Cercueil wrote:
> Hi Mike,
> [...]
>
>> If I comment the "wait" instruction, it will exit the suspend process 
>> immediately. And yes, I don't think it suspended properly.
>
> Ok. I was suggesting to try that since it would show if the crash 
> happens when a particular device gets suspended.
>
> Are you certain that your wakeup IRQ is unmasked?
>
> [...]
>
>>>>>  I'm afraid the above didn't work for me. Have you tested 
>>>>> suspend-to-ram in person on a X series SoC?
>>>
>>>  I didn't test on X-series, I mostly work with JZ. But that part of 
>>> the design didn't change since the JZ4740.
>>>
>>>  Cheers,
>>>  -Paul
>>>
>>>
>>
>>
>> To be honest, I never owned a board with a JZ series SoC. And sorry 
>> for assuming the suspend-to-ram is unusable on all Ingenic SoCs. 
>> IIRC, all the JZ series SoCs have external DRAM, while the X series 
>> SoCs have internal DRAM. Also Ingenic advertised the power saving 
>> features of the X series SoCs heavily. Things might be different 
>> since it may involve additional power management.
>
> Even if the 3.x method you were describing works, the currently 
> upstream method should work as well, and if it doesn't, we probably 
> should try to figure why.
>
> I remember doing some tests on the JZ4770 some years ago, and I would 
> get a power consumption of 7mA when suspended - that's for the whole 
> board, measured at the 3.7V battery, so about 0.026 W. The only things 
> powered ON then are the RAM chips and the SoC's RTC module.
>
>> At the time of writing the last sentence of the email, Dr. Zhou just 
>> pointed out that it may has something to do with the secure boot 
>> feature introduced in the X series SoC, although the feature is not 
>> enabled. I already mailed my X1000E & X1501 boards to Dr. Zhou for 
>> further tests. You may want to get a X1000(E) board (e.g. halley2) 
>> and test this by yourself.
>
> I do have a Cu1000-Neo board, but I have never used it, I wouldn't 
> know how to test this.
>

The CU1000-Neo board does not lead out the wakeup pins, maybe it can be 
tested with GKD350 (X1830)?


Thanks and best regards!


> Cheers,
> -Paul
>

Re: RFC: Proper suspend-to-ram implementation of Ingenic SoCs

[Paul Cercueil]

Hi Zhou,

Le jeu., juil. 14 2022 at 00:13:22 +0800, Zhou Yanjie 
<zhouyanjie@wanyeetech.com> a écrit :
> Hi Paul,
> 
> On 2022/7/14 上午12:08, Paul Cercueil wrote:
>> Hi Mike,
>> [...]
>> 
>>> If I comment the "wait" instruction, it will exit the suspend 
>>> process \x7f\x7fimmediately. And yes, I don't think it suspended properly.
>> 
>> Ok. I was suggesting to try that since it would show if the crash 
>> \x7fhappens when a particular device gets suspended.
>> 
>> Are you certain that your wakeup IRQ is unmasked?
>> 
>> [...]
>> 
>>>>>>  I'm afraid the above didn't work for me. Have you tested 
>>>>>> \x7f\x7f\x7f\x7f\x7fsuspend-to-ram in person on a X series SoC?
>>>> 
>>>>  I didn't test on X-series, I mostly work with JZ. But that part 
>>>> of \x7f\x7f\x7fthe design didn't change since the JZ4740.
>>>> 
>>>>  Cheers,
>>>>  -Paul
>>>> 
>>>> 
>>> 
>>> 
>>> To be honest, I never owned a board with a JZ series SoC. And sorry 
>>> \x7f\x7ffor assuming the suspend-to-ram is unusable on all Ingenic SoCs. 
>>> \x7f\x7fIIRC, all the JZ series SoCs have external DRAM, while the X 
>>> series \x7f\x7fSoCs have internal DRAM. Also Ingenic advertised the power 
>>> saving \x7f\x7ffeatures of the X series SoCs heavily. Things might be 
>>> different \x7f\x7fsince it may involve additional power management.
>> 
>> Even if the 3.x method you were describing works, the currently 
>> \x7fupstream method should work as well, and if it doesn't, we probably 
>> \x7fshould try to figure why.
>> 
>> I remember doing some tests on the JZ4770 some years ago, and I 
>> would \x7fget a power consumption of 7mA when suspended - that's for 
>> the whole \x7fboard, measured at the 3.7V battery, so about 0.026 W. 
>> The only things \x7fpowered ON then are the RAM chips and the SoC's RTC 
>> module.
>> 
>>> At the time of writing the last sentence of the email, Dr. Zhou 
>>> just \x7f\x7fpointed out that it may has something to do with the secure 
>>> boot \x7f\x7ffeature introduced in the X series SoC, although the feature 
>>> is not \x7f\x7fenabled. I already mailed my X1000E & X1501 boards to Dr. 
>>> Zhou for \x7f\x7ffurther tests. You may want to get a X1000(E) board 
>>> (e.g. halley2) \x7f\x7fand test this by yourself.
>> 
>> I do have a Cu1000-Neo board, but I have never used it, I wouldn't 
>> \x7fknow how to test this.
>> 
> 
> The CU1000-Neo board does not lead out the wakeup pins, maybe it can 
> be tested with GKD350 (X1830)?

Well any GPIO can act as a wakeup pin, really. "The" wakeup pin is 
actually masked during suspend unless you specify it as wakeup-enable.
See: 
https://github.com/torvalds/linux/blob/master/arch/mips/boot/dts/ingenic/gcw0.dts#L231

Cheers,
-Paul

Re: RFC: Proper suspend-to-ram implementation of Ingenic SoCs

[Zhou Yanjie]

Hi Paul,

On 2022/7/14 上午12:16, Paul Cercueil wrote:
> Hi Zhou,
>
> Le jeu., juil. 14 2022 at 00:13:22 +0800, Zhou Yanjie 
> <zhouyanjie@wanyeetech.com> a écrit :
>> Hi Paul,
>>
>> On 2022/7/14 上午12:08, Paul Cercueil wrote:
>>> Hi Mike,
>>> [...]
>>>
>>>> If I comment the "wait" instruction, it will exit the suspend 
>>>> process \x7f\x7fimmediately. And yes, I don't think it suspended properly.
>>>
>>> Ok. I was suggesting to try that since it would show if the crash 
>>> \x7fhappens when a particular device gets suspended.
>>>
>>> Are you certain that your wakeup IRQ is unmasked?
>>>
>>> [...]
>>>
>>>>>>>  I'm afraid the above didn't work for me. Have you tested 
>>>>>>> \x7f\x7f\x7f\x7f\x7fsuspend-to-ram in person on a X series SoC?
>>>>>
>>>>>  I didn't test on X-series, I mostly work with JZ. But that part 
>>>>> of \x7f\x7f\x7fthe design didn't change since the JZ4740.
>>>>>
>>>>>  Cheers,
>>>>>  -Paul
>>>>>
>>>>>
>>>>
>>>>
>>>> To be honest, I never owned a board with a JZ series SoC. And sorry 
>>>> \x7f\x7ffor assuming the suspend-to-ram is unusable on all Ingenic SoCs. 
>>>> \x7f\x7fIIRC, all the JZ series SoCs have external DRAM, while the X 
>>>> series \x7f\x7fSoCs have internal DRAM. Also Ingenic advertised the power 
>>>> saving \x7f\x7ffeatures of the X series SoCs heavily. Things might be 
>>>> different \x7f\x7fsince it may involve additional power management.
>>>
>>> Even if the 3.x method you were describing works, the currently 
>>> \x7fupstream method should work as well, and if it doesn't, we probably 
>>> \x7fshould try to figure why.
>>>
>>> I remember doing some tests on the JZ4770 some years ago, and I 
>>> would \x7fget a power consumption of 7mA when suspended - that's for 
>>> the whole \x7fboard, measured at the 3.7V battery, so about 0.026 W. 
>>> The only things \x7fpowered ON then are the RAM chips and the SoC's RTC 
>>> module.
>>>
>>>> At the time of writing the last sentence of the email, Dr. Zhou 
>>>> just \x7f\x7fpointed out that it may has something to do with the secure 
>>>> boot \x7f\x7ffeature introduced in the X series SoC, although the feature 
>>>> is not \x7f\x7fenabled. I already mailed my X1000E & X1501 boards to Dr. 
>>>> Zhou for \x7f\x7ffurther tests. You may want to get a X1000(E) board 
>>>> (e.g. halley2) \x7f\x7fand test this by yourself.
>>>
>>> I do have a Cu1000-Neo board, but I have never used it, I wouldn't 
>>> \x7fknow how to test this.
>>>
>>
>> The CU1000-Neo board does not lead out the wakeup pins, maybe it can 
>> be tested with GKD350 (X1830)?
>
> Well any GPIO can act as a wakeup pin, really. "The" wakeup pin is 
> actually masked during suspend unless you specify it as wakeup-enable.
> See: 
> https://github.com/torvalds/linux/blob/master/arch/mips/boot/dts/ingenic/gcw0.dts#L231
>

Got it, thanks!


> Cheers,
> -Paul
>

Re: RFC: Proper suspend-to-ram implementation of Ingenic SoCs

[Mike Yang]

Hi Paul,

On 7/14/22 00:08, Paul Cercueil wrote:
> Hi Mike,
> [...]
> 
>> If I comment the "wait" instruction, it will exit the suspend process immediately. And yes, I don't think it suspended properly.
> 
> Ok. I was suggesting to try that since it would show if the crash happens when a particular device gets suspended.
> 
> Are you certain that your wakeup IRQ is unmasked?

I'm not sure. Which register should I check?

> 
> [...]
> 
>>>>>  I'm afraid the above didn't work for me. Have you tested suspend-to-ram in person on a X series SoC?
>>>
>>>  I didn't test on X-series, I mostly work with JZ. But that part of the design didn't change since the JZ4740.
>>>
>>>  Cheers,
>>>  -Paul
>>>
>>>
>>
>>
>> To be honest, I never owned a board with a JZ series SoC. And sorry for assuming the suspend-to-ram is unusable on all Ingenic SoCs. IIRC, all the JZ series SoCs have external DRAM, while the X series SoCs have internal DRAM. Also Ingenic advertised the power saving features of the X series SoCs heavily. Things might be different since it may involve additional power management.
> 
> Even if the 3.x method you were describing works, the currently upstream method should work as well, and if it doesn't, we probably should try to figure why.
> 
> I remember doing some tests on the JZ4770 some years ago, and I would get a power consumption of 7mA when suspended - that's for the whole board, measured at the 3.7V battery, so about 0.026 W. The only things powered ON then are the RAM chips and the SoC's RTC module.
> 
>> At the time of writing the last sentence of the email, Dr. Zhou just pointed out that it may has something to do with the secure boot feature introduced in the X series SoC, although the feature is not enabled. I already mailed my X1000E & X1501 boards to Dr. Zhou for further tests. You may want to get a X1000(E) board (e.g. halley2) and test this by yourself.
> 
> I do have a Cu1000-Neo board, but I have never used it, I wouldn't know how to test this.
> 
> Cheers,
> -Paul
> 
> 

Earlier today, Dr. Zhou and I talked to a senior engineer from Ingenic. He said an extra piece of code is necessary for the X series, and more CPM registers (other than LPM) are needed to be configured. The X series can't reconfigure the DRAM to exit self-refresh mode by themselves. He also said, if we really don't want to put the code inside the kernel, it's possible to store the $pc somewhere in the RAM and modify UBoot SPL to do additional checks (e.g. P0 powerup flag) and jump back to the $pc after reconfiguring DRAM. I'm not sure if this will work, since the core will boot straight from the BROM, and the SFC and/or MSC peripherals will be reconfigured before it can load SPL again into the SRAM. It may cause confusion to the kernel SFC/MSC drivers. From his words, we can have another method: incorporate the code inside UBoot and write it to the SRAM prior to booting the kernel. What's your opinion?


Regards,
Mike

Re: RFC: Proper suspend-to-ram implementation of Ingenic SoCs

[Paul Cercueil]

Le jeu., juil. 14 2022 at 03:44:34 +0800, Mike Yang 
<reimu@sudomaker.com> a écrit :
> Hi Paul,
> 
> On 7/14/22 00:08, Paul Cercueil wrote:
>>  Hi Mike,
>>  [...]
>> 
>>>  If I comment the "wait" instruction, it will exit the suspend 
>>> process immediately. And yes, I don't think it suspended properly.
>> 
>>  Ok. I was suggesting to try that since it would show if the crash 
>> happens when a particular device gets suspended.
>> 
>>  Are you certain that your wakeup IRQ is unmasked?
> 
> I'm not sure. Which register should I check?

Check the IMCR0 / IMCR1 registers. Everything should be masked except 
your wakeup source. If your wakeup source is a GPIO, also check that 
the mask register that corresponds to your GPIO.

>> 
>>  [...]
>> 
>>>>>>   I'm afraid the above didn't work for me. Have you tested 
>>>>>> suspend-to-ram in person on a X series SoC?
>>>> 
>>>>   I didn't test on X-series, I mostly work with JZ. But that part 
>>>> of the design didn't change since the JZ4740.
>>>> 
>>>>   Cheers,
>>>>   -Paul
>>>> 
>>>> 
>>> 
>>> 
>>>  To be honest, I never owned a board with a JZ series SoC. And 
>>> sorry for assuming the suspend-to-ram is unusable on all Ingenic 
>>> SoCs. IIRC, all the JZ series SoCs have external DRAM, while the X 
>>> series SoCs have internal DRAM. Also Ingenic advertised the power 
>>> saving features of the X series SoCs heavily. Things might be 
>>> different since it may involve additional power management.
>> 
>>  Even if the 3.x method you were describing works, the currently 
>> upstream method should work as well, and if it doesn't, we probably 
>> should try to figure why.
>> 
>>  I remember doing some tests on the JZ4770 some years ago, and I 
>> would get a power consumption of 7mA when suspended - that's for the 
>> whole board, measured at the 3.7V battery, so about 0.026 W. The 
>> only things powered ON then are the RAM chips and the SoC's RTC 
>> module.
>> 
>>>  At the time of writing the last sentence of the email, Dr. Zhou 
>>> just pointed out that it may has something to do with the secure 
>>> boot feature introduced in the X series SoC, although the feature 
>>> is not enabled. I already mailed my X1000E & X1501 boards to Dr. 
>>> Zhou for further tests. You may want to get a X1000(E) board (e.g. 
>>> halley2) and test this by yourself.
>> 
>>  I do have a Cu1000-Neo board, but I have never used it, I wouldn't 
>> know how to test this.
>> 
>>  Cheers,
>>  -Paul
>> 
>> 
> 
> Earlier today, Dr. Zhou and I talked to a senior engineer from 
> Ingenic. He said an extra piece of code is necessary for the X 
> series, and more CPM registers (other than LPM) are needed to be 
> configured. The X series can't reconfigure the DRAM to exit 
> self-refresh mode by themselves. He also said, if we really don't 
> want to put the code inside the kernel, it's possible to store the 
> $pc somewhere in the RAM and modify UBoot SPL to do additional checks 
> (e.g. P0 powerup flag) and jump back to the $pc after reconfiguring 
> DRAM. I'm not sure if this will work, since the core will boot 
> straight from the BROM, and the SFC and/or MSC peripherals will be 
> reconfigured before it can load SPL again into the SRAM. It may cause 
> confusion to the kernel SFC/MSC drivers. From his words, we can have 
> another method: incorporate the code inside UBoot and write it to the 
> SRAM prior to booting the kernel. What's your opinion?

The X1000 has more CPM registers and do support turning the CPU 
completely off, which is new compared to the JZ4780, so that part is 
true. However, the regular method to enter SLEEP mode is still 
described in the X1000, X1830 and X2000 programming manuals, and it's 
the exact same method described in the JZ4780 and even in the JZ4740 
programming manuals. So allow me to doubt.

Knowing that Ingenic's 3.x kernel implements the "complete shutdown" 
sleep mode, I would think that this is why your senior engineer said 
that an extra piece of code is necessary - because that's how they 
implemented it. But that does not mean that it is required, and nothing 
in any of the X-series programming manuals suggests that it is required.

Cheers,
-Paul

Re: RFC: Proper suspend-to-ram implementation of Ingenic SoCs

[Zhou Yanjie]

Hi Paul,

On 2022/7/14 上午4:57, Paul Cercueil wrote:
>
>
> Le jeu., juil. 14 2022 at 03:44:34 +0800, Mike Yang 
> <reimu@sudomaker.com> a écrit :
>> Hi Paul,
>>
>> On 7/14/22 00:08, Paul Cercueil wrote:
>>>  Hi Mike,
>>>  [...]
>>>
>>>>  If I comment the "wait" instruction, it will exit the suspend 
>>>> process immediately. And yes, I don't think it suspended properly.
>>>
>>>  Ok. I was suggesting to try that since it would show if the crash 
>>> happens when a particular device gets suspended.
>>>
>>>  Are you certain that your wakeup IRQ is unmasked?
>>
>> I'm not sure. Which register should I check?
>
> Check the IMCR0 / IMCR1 registers. Everything should be masked except 
> your wakeup source. If your wakeup source is a GPIO, also check that 
> the mask register that corresponds to your GPIO.
>

Do you mean ICMR0 and ICMR1 in  the Interrupt Controller?


>>>
>>>  [...]
>>>
>>>>>>>   I'm afraid the above didn't work for me. Have you tested 
>>>>>>> suspend-to-ram in person on a X series SoC?
>>>>>
>>>>>   I didn't test on X-series, I mostly work with JZ. But that part 
>>>>> of the design didn't change since the JZ4740.
>>>>>
>>>>>   Cheers,
>>>>>   -Paul
>>>>>
>>>>>
>>>>
>>>>
>>>>  To be honest, I never owned a board with a JZ series SoC. And 
>>>> sorry for assuming the suspend-to-ram is unusable on all Ingenic 
>>>> SoCs. IIRC, all the JZ series SoCs have external DRAM, while the X 
>>>> series SoCs have internal DRAM. Also Ingenic advertised the power 
>>>> saving features of the X series SoCs heavily. Things might be 
>>>> different since it may involve additional power management.
>>>
>>>  Even if the 3.x method you were describing works, the currently 
>>> upstream method should work as well, and if it doesn't, we probably 
>>> should try to figure why.
>>>
>>>  I remember doing some tests on the JZ4770 some years ago, and I 
>>> would get a power consumption of 7mA when suspended - that's for the 
>>> whole board, measured at the 3.7V battery, so about 0.026 W. The 
>>> only things powered ON then are the RAM chips and the SoC's RTC module.
>>>
>>>>  At the time of writing the last sentence of the email, Dr. Zhou 
>>>> just pointed out that it may has something to do with the secure 
>>>> boot feature introduced in the X series SoC, although the feature 
>>>> is not enabled. I already mailed my X1000E & X1501 boards to Dr. 
>>>> Zhou for further tests. You may want to get a X1000(E) board (e.g. 
>>>> halley2) and test this by yourself.
>>>
>>>  I do have a Cu1000-Neo board, but I have never used it, I wouldn't 
>>> know how to test this.
>>>
>>>  Cheers,
>>>  -Paul
>>>
>>>
>>
>> Earlier today, Dr. Zhou and I talked to a senior engineer from 
>> Ingenic. He said an extra piece of code is necessary for the X 
>> series, and more CPM registers (other than LPM) are needed to be 
>> configured. The X series can't reconfigure the DRAM to exit 
>> self-refresh mode by themselves. He also said, if we really don't 
>> want to put the code inside the kernel, it's possible to store the 
>> $pc somewhere in the RAM and modify UBoot SPL to do additional checks 
>> (e.g. P0 powerup flag) and jump back to the $pc after reconfiguring 
>> DRAM. I'm not sure if this will work, since the core will boot 
>> straight from the BROM, and the SFC and/or MSC peripherals will be 
>> reconfigured before it can load SPL again into the SRAM. It may cause 
>> confusion to the kernel SFC/MSC drivers. From his words, we can have 
>> another method: incorporate the code inside UBoot and write it to the 
>> SRAM prior to booting the kernel. What's your opinion?
>
> The X1000 has more CPM registers and do support turning the CPU 
> completely off, which is new compared to the JZ4780, so that part is 
> true. However, the regular method to enter SLEEP mode is still 
> described in the X1000, X1830 and X2000 programming manuals, and it's 
> the exact same method described in the JZ4780 and even in the JZ4740 
> programming manuals. So allow me to doubt.
>
> Knowing that Ingenic's 3.x kernel implements the "complete shutdown" 
> sleep mode, I would think that this is why your senior engineer said 
> that an extra piece of code is necessary - because that's how they 
> implemented it. But that does not mean that it is required, and 
> nothing in any of the X-series programming manuals suggests that it is 
> required.
>
> Cheers,
> -Paul
>

Re: RFC: Proper suspend-to-ram implementation of Ingenic SoCs

[Paul Cercueil]

Hi Zhou,

Le jeu., juil. 14 2022 at 12:14:07 +0800, Zhou Yanjie 
<zhouyanjie@wanyeetech.com> a écrit :
> Hi Paul,
> 
> On 2022/7/14 上午4:57, Paul Cercueil wrote:
>> 
>> 
>> Le jeu., juil. 14 2022 at 03:44:34 +0800, Mike Yang 
>> \x7f<reimu@sudomaker.com> a écrit :
>>> Hi Paul,
>>> 
>>> On 7/14/22 00:08, Paul Cercueil wrote:
>>>>  Hi Mike,
>>>>  [...]
>>>> 
>>>>>  If I comment the "wait" instruction, it will exit the suspend 
>>>>> \x7f\x7f\x7f\x7fprocess immediately. And yes, I don't think it suspended 
>>>>> properly.
>>>> 
>>>>  Ok. I was suggesting to try that since it would show if the crash 
>>>> \x7f\x7f\x7fhappens when a particular device gets suspended.
>>>> 
>>>>  Are you certain that your wakeup IRQ is unmasked?
>>> 
>>> I'm not sure. Which register should I check?
>> 
>> Check the IMCR0 / IMCR1 registers. Everything should be masked 
>> except \x7fyour wakeup source. If your wakeup source is a GPIO, also 
>> check that \x7fthe mask register that corresponds to your GPIO.
>> 
> 
> Do you mean ICMR0 and ICMR1 in  the Interrupt Controller?

Yes, sorry.

Cheers,
-Paul

>>>> 
>>>>  [...]
>>>> 
>>>>>>>>   I'm afraid the above didn't work for me. Have you tested 
>>>>>>>> \x7f\x7f\x7f\x7f\x7f\x7f\x7fsuspend-to-ram in person on a X series SoC?
>>>>>> 
>>>>>>   I didn't test on X-series, I mostly work with JZ. But that 
>>>>>> part \x7f\x7f\x7f\x7f\x7fof the design didn't change since the JZ4740.
>>>>>> 
>>>>>>   Cheers,
>>>>>>   -Paul
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>>  To be honest, I never owned a board with a JZ series SoC. And 
>>>>> \x7f\x7f\x7f\x7fsorry for assuming the suspend-to-ram is unusable on all 
>>>>> Ingenic \x7f\x7f\x7f\x7fSoCs. IIRC, all the JZ series SoCs have external 
>>>>> DRAM, while the X \x7f\x7f\x7f\x7fseries SoCs have internal DRAM. Also 
>>>>> Ingenic advertised the power \x7f\x7f\x7f\x7fsaving features of the X series 
>>>>> SoCs heavily. Things might be \x7f\x7f\x7f\x7fdifferent since it may involve 
>>>>> additional power management.
>>>> 
>>>>  Even if the 3.x method you were describing works, the currently 
>>>> \x7f\x7f\x7fupstream method should work as well, and if it doesn't, we 
>>>> probably \x7f\x7f\x7fshould try to figure why.
>>>> 
>>>>  I remember doing some tests on the JZ4770 some years ago, and I 
>>>> \x7f\x7f\x7fwould get a power consumption of 7mA when suspended - that's 
>>>> for the \x7f\x7f\x7fwhole board, measured at the 3.7V battery, so about 
>>>> 0.026 W. The \x7f\x7f\x7fonly things powered ON then are the RAM chips and 
>>>> the SoC's RTC module.
>>>> 
>>>>>  At the time of writing the last sentence of the email, Dr. Zhou 
>>>>> \x7f\x7f\x7f\x7fjust pointed out that it may has something to do with the 
>>>>> secure \x7f\x7f\x7f\x7fboot feature introduced in the X series SoC, although 
>>>>> the feature \x7f\x7f\x7f\x7fis not enabled. I already mailed my X1000E & 
>>>>> X1501 boards to Dr. \x7f\x7f\x7f\x7fZhou for further tests. You may want to 
>>>>> get a X1000(E) board (e.g. \x7f\x7f\x7f\x7fhalley2) and test this by yourself.
>>>> 
>>>>  I do have a Cu1000-Neo board, but I have never used it, I 
>>>> wouldn't \x7f\x7f\x7fknow how to test this.
>>>> 
>>>>  Cheers,
>>>>  -Paul
>>>> 
>>>> 
>>> 
>>> Earlier today, Dr. Zhou and I talked to a senior engineer from 
>>> \x7f\x7fIngenic. He said an extra piece of code is necessary for the X 
>>> \x7f\x7fseries, and more CPM registers (other than LPM) are needed to be 
>>> \x7f\x7fconfigured. The X series can't reconfigure the DRAM to exit 
>>> \x7f\x7fself-refresh mode by themselves. He also said, if we really don't 
>>> \x7f\x7fwant to put the code inside the kernel, it's possible to store 
>>> the \x7f\x7f$pc somewhere in the RAM and modify UBoot SPL to do 
>>> additional checks \x7f\x7f(e.g. P0 powerup flag) and jump back to the $pc 
>>> after reconfiguring \x7f\x7fDRAM. I'm not sure if this will work, since 
>>> the core will boot \x7f\x7fstraight from the BROM, and the SFC and/or MSC 
>>> peripherals will be \x7f\x7freconfigured before it can load SPL again 
>>> into the SRAM. It may cause \x7f\x7fconfusion to the kernel SFC/MSC 
>>> drivers. From his words, we can have \x7f\x7fanother method: incorporate 
>>> the code inside UBoot and write it to the \x7f\x7fSRAM prior to booting 
>>> the kernel. What's your opinion?
>> 
>> The X1000 has more CPM registers and do support turning the CPU 
>> \x7fcompletely off, which is new compared to the JZ4780, so that part 
>> is \x7ftrue. However, the regular method to enter SLEEP mode is still 
>> \x7fdescribed in the X1000, X1830 and X2000 programming manuals, and 
>> it's \x7fthe exact same method described in the JZ4780 and even in the 
>> JZ4740 \x7fprogramming manuals. So allow me to doubt.
>> 
>> Knowing that Ingenic's 3.x kernel implements the "complete shutdown" 
>> \x7fsleep mode, I would think that this is why your senior engineer 
>> said \x7fthat an extra piece of code is necessary - because that's how 
>> they \x7fimplemented it. But that does not mean that it is required, 
>> and \x7fnothing in any of the X-series programming manuals suggests 
>> that it is \x7frequired.
>> 
>> Cheers,
>> -Paul
>>