timers & suspend

timers & suspend

[Sören Brinkmann]

Hi,

I'm currently working on suspend for Zynq and try to track down some
spurious wakes. It looks like the spurious wakes are caused by timers,
hence I was wondering whether there are any special requirements for
timer drivers when it comes to suspend support or if I just missed
something.

Zynq sets the 'IRQCHIP_MASK_ON_SUSPEND' flag, which should mask all
interrupts but the wake source. Reading through kernel/irq/pm.c
indicates, that timer interrupts get some special treatment though.
Therefore I implemented some suspend/resume callbacks for the
cadence_ttc which disable and clear the timer's interrupts when going
into suspend. That seems to mitigate the issue quite a bit, but I still
saw spurious wakes - just a lot less often.
Digging a little deeper revealed, the spurious wakes are caused by the
ARM's smp_twd timer now. Given that that driver is probably used by a few
more ARM platforms, I get the feeling that I'm missing something.

It's probably worth mentioning that the suspend state in Zynq does not
power off the CPU cores. It just asserts the resets on secondary cores
and the primary one waits in wfi.

	Thanks,
	S?ren

timers & suspend

[Daniel Lezcano]

On 06/30/2014 08:39 PM, S?ren Brinkmann wrote:
> Hi,
>
> I'm currently working on suspend for Zynq and try to track down some
> spurious wakes. It looks like the spurious wakes are caused by timers,
> hence I was wondering whether there are any special requirements for
> timer drivers when it comes to suspend support or if I just missed
> something.
>
> Zynq sets the 'IRQCHIP_MASK_ON_SUSPEND' flag, which should mask all
> interrupts but the wake source. Reading through kernel/irq/pm.c
> indicates, that timer interrupts get some special treatment though.
> Therefore I implemented some suspend/resume callbacks for the
> cadence_ttc which disable and clear the timer's interrupts when going
> into suspend. That seems to mitigate the issue quite a bit, but I still
> saw spurious wakes - just a lot less often.
> Digging a little deeper revealed, the spurious wakes are caused by the
> ARM's smp_twd timer now. Given that that driver is probably used by a few
> more ARM platforms, I get the feeling that I'm missing something.

Do you receive any interrupt from the cadence_ttc ? (/proc/interrupts)

That's funny because I realize that you cadence ttc timer is never used 
as there are the architected timers. The cadence ttc would be only 
useful if there were an idle state powering down the smp_twd timers but 
it is not the case on this board, IIUC.

> It's probably worth mentioning that the suspend state in Zynq does not
> power off the CPU cores. It just asserts the resets on secondary cores
> and the primary one waits in wfi.

Why do you need to reset them ?

-- 
  <http://www.linaro.org/> Linaro.org ? Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog

timers & suspend

[Sören Brinkmann]

On Thu, 2014-07-03 at 02:21PM +0200, Daniel Lezcano wrote:
> On 06/30/2014 08:39 PM, S?ren Brinkmann wrote:
> >Hi,
> >
> >I'm currently working on suspend for Zynq and try to track down some
> >spurious wakes. It looks like the spurious wakes are caused by timers,
> >hence I was wondering whether there are any special requirements for
> >timer drivers when it comes to suspend support or if I just missed
> >something.
> >
> >Zynq sets the 'IRQCHIP_MASK_ON_SUSPEND' flag, which should mask all
> >interrupts but the wake source. Reading through kernel/irq/pm.c
> >indicates, that timer interrupts get some special treatment though.
> >Therefore I implemented some suspend/resume callbacks for the
> >cadence_ttc which disable and clear the timer's interrupts when going
> >into suspend. That seems to mitigate the issue quite a bit, but I still
> >saw spurious wakes - just a lot less often.
> >Digging a little deeper revealed, the spurious wakes are caused by the
> >ARM's smp_twd timer now. Given that that driver is probably used by a few
> >more ARM platforms, I get the feeling that I'm missing something.
> 
> Do you receive any interrupt from the cadence_ttc ? (/proc/interrupts)
> 
> That's funny because I realize that you cadence ttc timer is never
> used as there are the architected timers. The cadence ttc would be
> only useful if there were an idle state powering down the smp_twd
> timers but it is not the case on this board, IIUC.
Yes they are used. They TTC is the only broadcast capable timer for
Zynq. In my experience, I can not even boot without it (may have
dependencies on CPUidle or something).

> 
> >It's probably worth mentioning that the suspend state in Zynq does not
> >power off the CPU cores. It just asserts the resets on secondary cores
> >and the primary one waits in wfi.
> 
> Why do you need to reset them ?
CPU0 is not reset, but all secondary cores are. That is as close to
power off as we get and works well with hotplug.

	S?ren

timers & suspend

[Daniel Lezcano]

On 07/03/2014 06:09 PM, S?ren Brinkmann wrote:
> On Thu, 2014-07-03 at 02:21PM +0200, Daniel Lezcano wrote:
>> On 06/30/2014 08:39 PM, S?ren Brinkmann wrote:
>>> Hi,
>>>
>>> I'm currently working on suspend for Zynq and try to track down some
>>> spurious wakes. It looks like the spurious wakes are caused by timers,
>>> hence I was wondering whether there are any special requirements for
>>> timer drivers when it comes to suspend support or if I just missed
>>> something.
>>>
>>> Zynq sets the 'IRQCHIP_MASK_ON_SUSPEND' flag, which should mask all
>>> interrupts but the wake source. Reading through kernel/irq/pm.c
>>> indicates, that timer interrupts get some special treatment though.
>>> Therefore I implemented some suspend/resume callbacks for the
>>> cadence_ttc which disable and clear the timer's interrupts when going
>>> into suspend. That seems to mitigate the issue quite a bit, but I still
>>> saw spurious wakes - just a lot less often.
>>> Digging a little deeper revealed, the spurious wakes are caused by the
>>> ARM's smp_twd timer now. Given that that driver is probably used by a few
>>> more ARM platforms, I get the feeling that I'm missing something.
>>
>> Do you receive any interrupt from the cadence_ttc ? (/proc/interrupts)
>>
>> That's funny because I realize that you cadence ttc timer is never
>> used as there are the architected timers. The cadence ttc would be
>> only useful if there were an idle state powering down the smp_twd
>> timers but it is not the case on this board, IIUC.
> Yes they are used. They TTC is the only broadcast capable timer for
> Zynq. In my experience, I can not even boot without it (may have
> dependencies on CPUidle or something).

Actually the cpuidle driver is wrong. It assumes the state #1 will power 
off the different cores with their architected timers and then switch to 
the broadcast timer. But this one is not needed as we don't power down 
the core with the twd timers, so no need to switch to a backup timer device.

The implementation of the DDR self refresh idle state (incoming 
patchset) removes the cpu_pm notifiers + the flag TIMER_STOP. The result 
is 0 interrupts for ttc cadence timer. I removed in the dts the cadence 
ttc and my board booted without problem (it is a zynq 702).

Except I missed something, the cadence ttc is actually not used at all.

>>> It's probably worth mentioning that the suspend state in Zynq does not
>>> power off the CPU cores. It just asserts the resets on secondary cores
>>> and the primary one waits in wfi.
>>
>> Why do you need to reset them ?
> CPU0 is not reset, but all secondary cores are. That is as close to
> power off as we get and works well with hotplug.

Ok, I was not aware of this approach. Is there any patchset available in 
the net ?

Thanks

   -- Daniel

-- 
  <http://www.linaro.org/> Linaro.org ? Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog

timers & suspend

[Sören Brinkmann]

On Thu, 2014-07-03 at 07:26PM +0200, Daniel Lezcano wrote:
> On 07/03/2014 06:09 PM, S?ren Brinkmann wrote:
> >On Thu, 2014-07-03 at 02:21PM +0200, Daniel Lezcano wrote:
> >>On 06/30/2014 08:39 PM, S?ren Brinkmann wrote:
> >>>Hi,
> >>>
> >>>I'm currently working on suspend for Zynq and try to track down some
> >>>spurious wakes. It looks like the spurious wakes are caused by timers,
> >>>hence I was wondering whether there are any special requirements for
> >>>timer drivers when it comes to suspend support or if I just missed
> >>>something.
> >>>
> >>>Zynq sets the 'IRQCHIP_MASK_ON_SUSPEND' flag, which should mask all
> >>>interrupts but the wake source. Reading through kernel/irq/pm.c
> >>>indicates, that timer interrupts get some special treatment though.
> >>>Therefore I implemented some suspend/resume callbacks for the
> >>>cadence_ttc which disable and clear the timer's interrupts when going
> >>>into suspend. That seems to mitigate the issue quite a bit, but I still
> >>>saw spurious wakes - just a lot less often.
> >>>Digging a little deeper revealed, the spurious wakes are caused by the
> >>>ARM's smp_twd timer now. Given that that driver is probably used by a few
> >>>more ARM platforms, I get the feeling that I'm missing something.
> >>
> >>Do you receive any interrupt from the cadence_ttc ? (/proc/interrupts)
> >>
> >>That's funny because I realize that you cadence ttc timer is never
> >>used as there are the architected timers. The cadence ttc would be
> >>only useful if there were an idle state powering down the smp_twd
> >>timers but it is not the case on this board, IIUC.
> >Yes they are used. They TTC is the only broadcast capable timer for
> >Zynq. In my experience, I can not even boot without it (may have
> >dependencies on CPUidle or something).
> 
> Actually the cpuidle driver is wrong. It assumes the state #1 will
> power off the different cores with their architected timers and then
> switch to the broadcast timer. But this one is not needed as we
> don't power down the core with the twd timers, so no need to switch
> to a backup timer device.
> 
> The implementation of the DDR self refresh idle state (incoming
> patchset) removes the cpu_pm notifiers + the flag TIMER_STOP. The
> result is 0 interrupts for ttc cadence timer. I removed in the dts
> the cadence ttc and my board booted without problem (it is a zynq
> 702).
> 
> Except I missed something, the cadence ttc is actually not used at all.
You mean with your patches, or without? I'd have to check that.

> 
> >>>It's probably worth mentioning that the suspend state in Zynq does not
> >>>power off the CPU cores. It just asserts the resets on secondary cores
> >>>and the primary one waits in wfi.
> >>
> >>Why do you need to reset them ?
> >CPU0 is not reset, but all secondary cores are. That is as close to
> >power off as we get and works well with hotplug.
> 
> Ok, I was not aware of this approach. Is there any patchset
> available in the net ?
It's in mainline for a while. arch/arm/mach-zynq/platsmp.c defines the
SMP ops. Zynq's cpu_die/kill result in the CPU reset being asserted. And
when the core is plugged in again, it gets deasserted and goes through
the secondary boot path.
That code is a little cleaned up in our vendor tree, but overall it
should be the same functionality wise.

	S?ren

timers & suspend

[Daniel Lezcano]

On 07/03/2014 07:30 PM, S?ren Brinkmann wrote:
> On Thu, 2014-07-03 at 07:26PM +0200, Daniel Lezcano wrote:
>> On 07/03/2014 06:09 PM, S?ren Brinkmann wrote:
>>> On Thu, 2014-07-03 at 02:21PM +0200, Daniel Lezcano wrote:
>>>> On 06/30/2014 08:39 PM, S?ren Brinkmann wrote:
>>>>> Hi,
>>>>>
>>>>> I'm currently working on suspend for Zynq and try to track down some
>>>>> spurious wakes. It looks like the spurious wakes are caused by timers,
>>>>> hence I was wondering whether there are any special requirements for
>>>>> timer drivers when it comes to suspend support or if I just missed
>>>>> something.
>>>>>
>>>>> Zynq sets the 'IRQCHIP_MASK_ON_SUSPEND' flag, which should mask all
>>>>> interrupts but the wake source. Reading through kernel/irq/pm.c
>>>>> indicates, that timer interrupts get some special treatment though.
>>>>> Therefore I implemented some suspend/resume callbacks for the
>>>>> cadence_ttc which disable and clear the timer's interrupts when going
>>>>> into suspend. That seems to mitigate the issue quite a bit, but I still
>>>>> saw spurious wakes - just a lot less often.
>>>>> Digging a little deeper revealed, the spurious wakes are caused by the
>>>>> ARM's smp_twd timer now. Given that that driver is probably used by a few
>>>>> more ARM platforms, I get the feeling that I'm missing something.
>>>>
>>>> Do you receive any interrupt from the cadence_ttc ? (/proc/interrupts)
>>>>
>>>> That's funny because I realize that you cadence ttc timer is never
>>>> used as there are the architected timers. The cadence ttc would be
>>>> only useful if there were an idle state powering down the smp_twd
>>>> timers but it is not the case on this board, IIUC.
>>> Yes they are used. They TTC is the only broadcast capable timer for
>>> Zynq. In my experience, I can not even boot without it (may have
>>> dependencies on CPUidle or something).
>>
>> Actually the cpuidle driver is wrong. It assumes the state #1 will
>> power off the different cores with their architected timers and then
>> switch to the broadcast timer. But this one is not needed as we
>> don't power down the core with the twd timers, so no need to switch
>> to a backup timer device.
>>
>> The implementation of the DDR self refresh idle state (incoming
>> patchset) removes the cpu_pm notifiers + the flag TIMER_STOP. The
>> result is 0 interrupts for ttc cadence timer. I removed in the dts
>> the cadence ttc and my board booted without problem (it is a zynq
>> 702).
>>
>> Except I missed something, the cadence ttc is actually not used at all.
> You mean with your patches, or without? I'd have to check that.

I meant with my patches.

>>>>> It's probably worth mentioning that the suspend state in Zynq does not
>>>>> power off the CPU cores. It just asserts the resets on secondary cores
>>>>> and the primary one waits in wfi.
>>>>
>>>> Why do you need to reset them ?
>>> CPU0 is not reset, but all secondary cores are. That is as close to
>>> power off as we get and works well with hotplug.
>>
>> Ok, I was not aware of this approach. Is there any patchset
>> available in the net ?
> It's in mainline for a while. arch/arm/mach-zynq/platsmp.c defines the
> SMP ops. Zynq's cpu_die/kill result in the CPU reset being asserted. And
> when the core is plugged in again, it gets deasserted and goes through
> the secondary boot path.
> That code is a little cleaned up in our vendor tree, but overall it
> should be the same functionality wise.

Ah, interesting. Thanks for the precision. I was pointing to v3.16-rc3 
and I didn't see the upstream xilinx code. So my patchset is now 
out-dated :(

I will look at the code closely.

Thanks
   -- Daniel






-- 
  <http://www.linaro.org/> Linaro.org ? Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog

timers & suspend

[Sören Brinkmann]

On Thu, 2014-07-03 at 07:26PM +0200, Daniel Lezcano wrote:
> On 07/03/2014 06:09 PM, S?ren Brinkmann wrote:
> >On Thu, 2014-07-03 at 02:21PM +0200, Daniel Lezcano wrote:
> >>On 06/30/2014 08:39 PM, S?ren Brinkmann wrote:
> >>>Hi,
> >>>
> >>>I'm currently working on suspend for Zynq and try to track down some
> >>>spurious wakes. It looks like the spurious wakes are caused by timers,
> >>>hence I was wondering whether there are any special requirements for
> >>>timer drivers when it comes to suspend support or if I just missed
> >>>something.
> >>>
> >>>Zynq sets the 'IRQCHIP_MASK_ON_SUSPEND' flag, which should mask all
> >>>interrupts but the wake source. Reading through kernel/irq/pm.c
> >>>indicates, that timer interrupts get some special treatment though.
> >>>Therefore I implemented some suspend/resume callbacks for the
> >>>cadence_ttc which disable and clear the timer's interrupts when going
> >>>into suspend. That seems to mitigate the issue quite a bit, but I still
> >>>saw spurious wakes - just a lot less often.
> >>>Digging a little deeper revealed, the spurious wakes are caused by the
> >>>ARM's smp_twd timer now. Given that that driver is probably used by a few
> >>>more ARM platforms, I get the feeling that I'm missing something.
> >>
> >>Do you receive any interrupt from the cadence_ttc ? (/proc/interrupts)
> >>
> >>That's funny because I realize that you cadence ttc timer is never
> >>used as there are the architected timers. The cadence ttc would be
> >>only useful if there were an idle state powering down the smp_twd
> >>timers but it is not the case on this board, IIUC.
> >Yes they are used. They TTC is the only broadcast capable timer for
> >Zynq. In my experience, I can not even boot without it (may have
> >dependencies on CPUidle or something).
> 
> Actually the cpuidle driver is wrong. It assumes the state #1 will
> power off the different cores with their architected timers and then
> switch to the broadcast timer. But this one is not needed as we
> don't power down the core with the twd timers, so no need to switch
> to a backup timer device.
> 
> The implementation of the DDR self refresh idle state (incoming
> patchset) removes the cpu_pm notifiers + the flag TIMER_STOP. The
> result is 0 interrupts for ttc cadence timer. I removed in the dts
> the cadence ttc and my board booted without problem (it is a zynq
> 702).
> 
> Except I missed something, the cadence ttc is actually not used at all.
I tested on the current master branch. I see TTC interrupts on CPU0 and
timer_list shows the TTC to be the broadcast device. Removing the TTC
nodes from my DT results in boot hanging - shortly after cpuidle is
started.

Removing cpuidle from my kernel makes the system boot. And it has no
broadcast device anymore.

	S?ren

timers & suspend

[Daniel Lezcano]

On 07/03/2014 07:40 PM, S?ren Brinkmann wrote:
> On Thu, 2014-07-03 at 07:26PM +0200, Daniel Lezcano wrote:
>> On 07/03/2014 06:09 PM, S?ren Brinkmann wrote:
>>> On Thu, 2014-07-03 at 02:21PM +0200, Daniel Lezcano wrote:
>>>> On 06/30/2014 08:39 PM, S?ren Brinkmann wrote:
>>>>> Hi,
>>>>>
>>>>> I'm currently working on suspend for Zynq and try to track down some
>>>>> spurious wakes. It looks like the spurious wakes are caused by timers,
>>>>> hence I was wondering whether there are any special requirements for
>>>>> timer drivers when it comes to suspend support or if I just missed
>>>>> something.
>>>>>
>>>>> Zynq sets the 'IRQCHIP_MASK_ON_SUSPEND' flag, which should mask all
>>>>> interrupts but the wake source. Reading through kernel/irq/pm.c
>>>>> indicates, that timer interrupts get some special treatment though.
>>>>> Therefore I implemented some suspend/resume callbacks for the
>>>>> cadence_ttc which disable and clear the timer's interrupts when going
>>>>> into suspend. That seems to mitigate the issue quite a bit, but I still
>>>>> saw spurious wakes - just a lot less often.
>>>>> Digging a little deeper revealed, the spurious wakes are caused by the
>>>>> ARM's smp_twd timer now. Given that that driver is probably used by a few
>>>>> more ARM platforms, I get the feeling that I'm missing something.
>>>>
>>>> Do you receive any interrupt from the cadence_ttc ? (/proc/interrupts)
>>>>
>>>> That's funny because I realize that you cadence ttc timer is never
>>>> used as there are the architected timers. The cadence ttc would be
>>>> only useful if there were an idle state powering down the smp_twd
>>>> timers but it is not the case on this board, IIUC.
>>> Yes they are used. They TTC is the only broadcast capable timer for
>>> Zynq. In my experience, I can not even boot without it (may have
>>> dependencies on CPUidle or something).
>>
>> Actually the cpuidle driver is wrong. It assumes the state #1 will
>> power off the different cores with their architected timers and then
>> switch to the broadcast timer. But this one is not needed as we
>> don't power down the core with the twd timers, so no need to switch
>> to a backup timer device.
>>
>> The implementation of the DDR self refresh idle state (incoming
>> patchset) removes the cpu_pm notifiers + the flag TIMER_STOP. The
>> result is 0 interrupts for ttc cadence timer. I removed in the dts
>> the cadence ttc and my board booted without problem (it is a zynq
>> 702).
>>
>> Except I missed something, the cadence ttc is actually not used at all.
> I tested on the current master branch. I see TTC interrupts on CPU0 and
> timer_list shows the TTC to be the broadcast device. Removing the TTC
> nodes from my DT results in boot hanging - shortly after cpuidle is
> started.
>
> Removing cpuidle from my kernel makes the system boot. And it has no
> broadcast device anymore.

You can safely remove the CPUIDLE_FLAG_TIMER_STOP in the cpuidle drivers 
and I am pretty sure it will boot without problem.

-- 
  <http://www.linaro.org/> Linaro.org ? Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog

timers & suspend

[Sören Brinkmann]

On Thu, 2014-07-03 at 07:46PM +0200, Daniel Lezcano wrote:
> On 07/03/2014 07:40 PM, S?ren Brinkmann wrote:
> >On Thu, 2014-07-03 at 07:26PM +0200, Daniel Lezcano wrote:
> >>On 07/03/2014 06:09 PM, S?ren Brinkmann wrote:
> >>>On Thu, 2014-07-03 at 02:21PM +0200, Daniel Lezcano wrote:
> >>>>On 06/30/2014 08:39 PM, S?ren Brinkmann wrote:
> >>>>>Hi,
> >>>>>
> >>>>>I'm currently working on suspend for Zynq and try to track down some
> >>>>>spurious wakes. It looks like the spurious wakes are caused by timers,
> >>>>>hence I was wondering whether there are any special requirements for
> >>>>>timer drivers when it comes to suspend support or if I just missed
> >>>>>something.
> >>>>>
> >>>>>Zynq sets the 'IRQCHIP_MASK_ON_SUSPEND' flag, which should mask all
> >>>>>interrupts but the wake source. Reading through kernel/irq/pm.c
> >>>>>indicates, that timer interrupts get some special treatment though.
> >>>>>Therefore I implemented some suspend/resume callbacks for the
> >>>>>cadence_ttc which disable and clear the timer's interrupts when going
> >>>>>into suspend. That seems to mitigate the issue quite a bit, but I still
> >>>>>saw spurious wakes - just a lot less often.
> >>>>>Digging a little deeper revealed, the spurious wakes are caused by the
> >>>>>ARM's smp_twd timer now. Given that that driver is probably used by a few
> >>>>>more ARM platforms, I get the feeling that I'm missing something.
> >>>>
> >>>>Do you receive any interrupt from the cadence_ttc ? (/proc/interrupts)
> >>>>
> >>>>That's funny because I realize that you cadence ttc timer is never
> >>>>used as there are the architected timers. The cadence ttc would be
> >>>>only useful if there were an idle state powering down the smp_twd
> >>>>timers but it is not the case on this board, IIUC.
> >>>Yes they are used. They TTC is the only broadcast capable timer for
> >>>Zynq. In my experience, I can not even boot without it (may have
> >>>dependencies on CPUidle or something).
> >>
> >>Actually the cpuidle driver is wrong. It assumes the state #1 will
> >>power off the different cores with their architected timers and then
> >>switch to the broadcast timer. But this one is not needed as we
> >>don't power down the core with the twd timers, so no need to switch
> >>to a backup timer device.
> >>
> >>The implementation of the DDR self refresh idle state (incoming
> >>patchset) removes the cpu_pm notifiers + the flag TIMER_STOP. The
> >>result is 0 interrupts for ttc cadence timer. I removed in the dts
> >>the cadence ttc and my board booted without problem (it is a zynq
> >>702).
> >>
> >>Except I missed something, the cadence ttc is actually not used at all.
> >I tested on the current master branch. I see TTC interrupts on CPU0 and
> >timer_list shows the TTC to be the broadcast device. Removing the TTC
> >nodes from my DT results in boot hanging - shortly after cpuidle is
> >started.
> >
> >Removing cpuidle from my kernel makes the system boot. And it has no
> >broadcast device anymore.
> 
> You can safely remove the CPUIDLE_FLAG_TIMER_STOP in the cpuidle
> drivers and I am pretty sure it will boot without problem.
Yes, that works.

	Thanks,
	S?ren

timers & suspend

[Sören Brinkmann]

Let me extend the audience a bit.

On Mon, 2014-06-30 at 11:39AM -0700, S?ren Brinkmann wrote:
> Hi,
> 
> I'm currently working on suspend for Zynq and try to track down some
> spurious wakes. It looks like the spurious wakes are caused by timers,
> hence I was wondering whether there are any special requirements for
> timer drivers when it comes to suspend support or if I just missed
> something.
> 
> Zynq sets the 'IRQCHIP_MASK_ON_SUSPEND' flag, which should mask all
> interrupts but the wake source. Reading through kernel/irq/pm.c
> indicates, that timer interrupts get some special treatment though.
> Therefore I implemented some suspend/resume callbacks for the
> cadence_ttc which disable and clear the timer's interrupts when going
> into suspend. That seems to mitigate the issue quite a bit, but I still
> saw spurious wakes - just a lot less often.
> Digging a little deeper revealed, the spurious wakes are caused by the
> ARM's smp_twd timer now. Given that that driver is probably used by a few
> more ARM platforms, I get the feeling that I'm missing something.
> 
> It's probably worth mentioning that the suspend state in Zynq does not
> power off the CPU cores. It just asserts the resets on secondary cores
> and the primary one waits in wfi.

I think I found the issue. When going into suspend, all device
interrupts get disabled, but timers are kept running until very late.
Then in kernel/power/suspend.c:
 - arch_suspend_disable_irqs() disables interrupts (locally)
 - syscore_suspend is called, which disables timers through
   tick_suspend()

I think what happens is: The interrupts get disabled locally, but the
timers are still running and generating interrupts.
Such an interrupt happens and stays pending since interrupts are already
disabled and no longer handled.
Then, since Zynq does not power off but only goes into wfi, it
immediately resumes due to a pending timer IRQ.

Especially with the TTC this can happen quite often since it is only
16 bit wide. But I also see spurious wakes caused by the twd.

Does that sound like a possible scenario?

	S?ren

timers & suspend

[Sören Brinkmann]

On Tue, 2014-07-08 at 04:50PM -0700, S?ren Brinkmann wrote:
> Let me extend the audience a bit.
> 
> On Mon, 2014-06-30 at 11:39AM -0700, S?ren Brinkmann wrote:
> > Hi,
> > 
> > I'm currently working on suspend for Zynq and try to track down some
> > spurious wakes. It looks like the spurious wakes are caused by timers,
> > hence I was wondering whether there are any special requirements for
> > timer drivers when it comes to suspend support or if I just missed
> > something.
> > 
> > Zynq sets the 'IRQCHIP_MASK_ON_SUSPEND' flag, which should mask all
> > interrupts but the wake source. Reading through kernel/irq/pm.c
> > indicates, that timer interrupts get some special treatment though.
> > Therefore I implemented some suspend/resume callbacks for the
> > cadence_ttc which disable and clear the timer's interrupts when going
> > into suspend. That seems to mitigate the issue quite a bit, but I still
> > saw spurious wakes - just a lot less often.
> > Digging a little deeper revealed, the spurious wakes are caused by the
> > ARM's smp_twd timer now. Given that that driver is probably used by a few
> > more ARM platforms, I get the feeling that I'm missing something.
> > 
> > It's probably worth mentioning that the suspend state in Zynq does not
> > power off the CPU cores. It just asserts the resets on secondary cores
> > and the primary one waits in wfi.
> 
> I think I found the issue. When going into suspend, all device
> interrupts get disabled, but timers are kept running until very late.
> Then in kernel/power/suspend.c:
>  - arch_suspend_disable_irqs() disables interrupts (locally)
>  - syscore_suspend is called, which disables timers through
>    tick_suspend()
> 
> I think what happens is: The interrupts get disabled locally, but the
> timers are still running and generating interrupts.
> Such an interrupt happens and stays pending since interrupts are already
> disabled and no longer handled.
> Then, since Zynq does not power off but only goes into wfi, it
> immediately resumes due to a pending timer IRQ.
> 
> Especially with the TTC this can happen quite often since it is only
> 16 bit wide. But I also see spurious wakes caused by the twd.
> 
> Does that sound like a possible scenario?

As another data point: I don't see any spurious wakes with the changes
below.

	S?ren

---------------8<------------------8<-----------------8<----------------8<------
diff --git a/drivers/clocksource/cadence_ttc_timer.c b/drivers/clocksource/cadence_ttc_timer.c
index e0a81327e10c..3abe2d7031ed 100644
--- a/drivers/clocksource/cadence_ttc_timer.c
+++ b/drivers/clocksource/cadence_ttc_timer.c
@@ -321,6 +321,19 @@ static int ttc_rate_change_clocksource_cb(struct notifier_block *nb,
 	return NOTIFY_DONE;
 }
 
+static void ttc_ce_suspend(struct clock_event_device *ce)
+{
+	struct ttc_timer_clockevent *ttcce = to_ttc_timer_clkevent(ce);
+
+	readl_relaxed(ttcce->ttc.base_addr + TTC_ISR_OFFSET);
+	disable_irq(ce->irq);
+}
+
+static void ttc_ce_resume(struct clock_event_device *ce)
+{
+	enable_irq(ce->irq);
+}
+
 static void __init ttc_setup_clocksource(struct clk *clk, void __iomem *base)
 {
 	struct ttc_timer_clocksource *ttccs;
@@ -428,6 +441,8 @@ static void __init ttc_setup_clockevent(struct clk *clk,
 	ttcce->ce.features = CLOCK_EVT_FEAT_PERIODIC | CLOCK_EVT_FEAT_ONESHOT;
 	ttcce->ce.set_next_event = ttc_set_next_event;
 	ttcce->ce.set_mode = ttc_set_mode;
+	ttcce->ce.suspend = ttc_ce_suspend;
+	ttcce->ce.resume = ttc_ce_resume;
 	ttcce->ce.rating = 200;
 	ttcce->ce.irq = irq;
 	ttcce->ce.cpumask = cpu_possible_mask;
diff --git a/arch/arm/kernel/smp_twd.c b/arch/arm/kernel/smp_twd.c
index 6591e26fc13f..956d40d9281f 100644
--- a/arch/arm/kernel/smp_twd.c
+++ b/arch/arm/kernel/smp_twd.c
@@ -264,6 +264,18 @@ static void twd_get_clock(struct device_node *np)
 	twd_timer_rate = clk_get_rate(twd_clk);
 }
 
+static void twd_suspend(struct clock_event_device *ce)
+{
+	struct clock_event_device *clk = __this_cpu_ptr(twd_evt);
+	disable_percpu_irq(clk->irq);
+}
+
+static void twd_resume(struct clock_event_device *ce)
+{
+	struct clock_event_device *clk = __this_cpu_ptr(twd_evt);
+	enable_percpu_irq(clk->irq, 0);
+}
+
 /*
  * Setup the local clock events for a CPU.
  */
@@ -300,6 +312,8 @@ static void twd_timer_setup(void)
 	clk->set_next_event = twd_set_next_event;
 	clk->irq = twd_ppi;
 	clk->cpumask = cpumask_of(cpu);
+	clk->suspend = twd_suspend;
+	clk->resume = twd_resume;
 
 	clockevents_config_and_register(clk, twd_timer_rate,
 					0xf, 0xffffffff);