Re: [PATCH v1] pmdomain: ti_sci: re-sync TIFS with genpd on resume

Re: [PATCH v1] pmdomain: ti_sci: re-sync TIFS with genpd on resume

[Sebin Francis]

Hi Vitor

On 04/05/26 11:56, Vitor Soares wrote:
> Hello Sebin
> 
> On Thu, 2026-04-30 at 16:17 +0530, Sebin Francis wrote:
>> Hi Vitor,
>>
>> On 29/04/26 21:56, Vitor Soares wrote:
>>> Hi Vignesh
>>>
>>> Thank you for the review.
>>>
>>> On Wed, 2026-04-29 at 10:03 +0530, Vignesh Raghavendra wrote:
>>>> Hi Vitor
>>>>
>>>> On 27/04/26 13:18, Vitor Soares wrote:
>>>>> From: Vitor Soares <vitor.soares@toradex.com>
>>>>>
>>>>> When a device in a TI SCI power domain is on the wakeup path of a
>>>>> wakeup-capable child, the suspend path skips genpd_sync_power_off().
>>>>> No put_device is sent to TIFS and the domain's genpd status remains
>>>>> ON.
>>>>
>>>> Correction of terminologies: TIFS is Root of trust component and is not
>>>> usually involved in power management, that would be DM (Device Manager)
>>>>
>>>
>>> Thank you for the clarification. I will address this on v2. Also, I was
>>> thinking
>>> to replace put_device/get_device with ti_sci_pd_power_off/ti_sci_pd_power_on
>>> if
>>> that makes more clear the content.
>>>
>>>> But to be really sure who is doing what, Could you provide an example
>>>> and the platform on which you see the issue / external abort?
>>>>
>>>
>>> This was reproduced on our Toradex Verdin AM62P WB and the driver for our
>>> Wi-Fi
>>> module on the SDIO bus calls device_init_wakeup() during the initialization.
>>>
>>> After enter in suspend, it show the following error resume path:
>>>
>>>
>>> [   41.759341] Internal error: synchronous external abort: 0000000096000010
>>> [#1]
>>> SMP
>>> [   41.843286] CPU: 0 UID: 0 PID: 933 Comm: rtcwake Tainted: G   M       O
>>> 6.18.21-dirty #3 PREEMPT
>>> [   41.852762] Tainted: [M]=MACHINE_CHECK, [O]=OOT_MODULE
>>> [   41.857891] Hardware name: Toradex Verdin AM62P WB on Verdin Development
>>> Board (DT)
>>> [   41.865537] pstate: 200000c5 (nzCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=-
>>> -)
>>> [   41.872492] pc : regmap_mmio_read32le+0x8/0x20
>>> [   41.876941] lr : regmap_mmio_read+0x44/0x70
>>> [   41.881120] sp : ffff800081fdb8e0
>>> [   41.884428] x29: ffff800081fdb8e0 x28: 0000000000000000 x27:
>>> ffffa95bb64aa9c8
>>> [   41.891563] x26: 0000000000000000 x25: 0000000000000000 x24:
>>> 0000000000000000
>>> [   41.898697] x23: 0000000080000000 x22: ffff000002df5c00 x21:
>>> ffff800081fdb9b4
>>> [   41.905831] x20: 0000000000000100 x19: ffff000001286400 x18:
>>> 0000000000000000
>>> [   41.912965] x17: 2d69696d67722f79 x16: 687020726f662067 x15:
>>> ffff00007fb74f40
>>> [   41.920100] x14: 00000000000002ea x13: 000000000000031f x12:
>>> 0000000000000000
>>> [   41.927234] x11: 00000000000000c0 x10: 00000000000009e0 x9 :
>>> ffff800081fdb7a0
>>> [   41.934368] x8 : ffff00007fb6ce00 x7 : 0000000000000000 x6 :
>>> 0000000000000000
>>> [   41.941502] x5 : ffffa95bb57948d8 x4 : 0000000000000100 x3 :
>>> 0000000000000100
>>> [   41.948636] x2 : ffffa95bb5795034 x1 : 0000000000000100 x0 :
>>> ffff80008025d100
>>> [   41.955770] Call trace:
>>> [   41.958211]  regmap_mmio_read32le+0x8/0x20 (P)
>>> [   41.962655]  _regmap_bus_reg_read+0x70/0xb0
>>> [   41.966839]  _regmap_read+0x64/0xdc
>>> [   41.970327]  _regmap_update_bits+0xf4/0x140
>>> [   41.974509]  regmap_update_bits_base+0x64/0x98
>>> [   41.978952]  sdhci_am654_runtime_resume+0x138/0x208
>>> [   41.983830]  pm_generic_runtime_resume+0x2c/0x44
>>> [   41.988445]  __genpd_runtime_resume+0x30/0x7c
>>> [   41.992804]  genpd_runtime_resume+0xdc/0x2e8
>>> [   41.997073]  pm_runtime_force_resume+0x68/0xf4
>>> [   42.001517]  dpm_run_callback+0x8c/0x14c
>>> [   42.005439]  device_resume+0x11c/0x34c
>>> [   42.009188]  dpm_resume+0x178/0x1f0
>>> [   42.012673]  dpm_resume_end+0x18/0x34
>>> [   42.016332]  suspend_devices_and_enter+0x4a4/0x668
>>> [   42.021123]  pm_suspend+0x170/0x2dc
>>> [   42.024610]  state_store+0x80/0x104
>>> [   42.028096]  kobj_attr_store+0x18/0x2c
>>> [   42.031845]  sysfs_kf_write+0x7c/0x94
>>> [   42.035508]  kernfs_fop_write_iter+0x130/0x1fc
>>> [   42.039949]  vfs_write+0x200/0x370
>>> [   42.043351]  ksys_write+0x6c/0x100
>>> [   42.046752]  __arm64_sys_write+0x1c/0x28
>>> [   42.050673]  invoke_syscall.constprop.0+0x50/0xe4
>>> [   42.055378]  do_el0_svc+0x40/0xc4
>>> [   42.058691]  el0_svc+0x40/0x15c
>>> [   42.061834]  el0t_64_sync_handler+0xa0/0xe4
>>> [   42.066015]  el0t_64_sync+0x198/0x19c
>>> [   42.069680] Code: aa0603e0 d65f03c0 f9400000 8b214000 (b9400000)
>>>
>>>>
>>>>>
>>>>> TIFS powers off the hardware during deep sleep regardless, since it
>>>>> was never informed to keep the domain active. On resume, because the
>>>>> domain's genpd status is ON, no get_device is issued. The driver
>>>>> then accesses registers of a powered-off domain, causing a
>>>>> synchronous external abort (AXI bus error, ESR 0x96000010).
>>>>
>>>> Hmm, if something is wakeup source, I would expect even TIFS/DM not to
>>>> turn if off, else module wakeup wouldn't work.
>>>>
>>>
>>> I tested UART as a wakeup source and I couldn't reproduce this issue. My
>>> understanding is that UART has its own TI SCI domain and device_may_wakeup()
>>> is
>>> true directly on that domain device, so the set_device_constraint fires
>>> correctly and DM keeps it powered.
>>>
>>> Here is my tracking of the issue:
>>>
>>> Wi-Fi driver registers as wakeup source:
>>> device_init_wakeup(mmc0:0001)
>>>
>>> During suspend/resume.
>>> dpm_suspend()
>>> ->genpd_suspend_dev(fa20000.mmc)
>>>      ->ti_sci_pd_suspend(fa20000.mmc)
>>>         ->ti_sci_pd_set_wkup_constraint(fa20000.mmc)
>>>           device_may_wakeup(fa20000.mmc)  = false
>>>           set_device_constraint never sent to DM
>>>
>>>
>>> dpm_suspend_noirq()
>>> ->genpd_finish_suspend(fa20000.mmc)
>>>     ->device_awake_path(fa20000.mmc) = true
>>>     ->GENPD_FLAG_ACTIVE_WAKEUP = true
>>>       genpd status = GENPD_STATE_ON
>>>       skip power_off (ti_sci_pd_power_off)
>>>
>>> On deep sleep entry, DM powers off fa20000.mmc independently.
>>> It received no set_device_constraint nor ti_sci_pd_power_off.
>>
>> In AM62P fa20000.mmc is part of main domain. During deepsleep the entire
>> main domain is turned off by the DM, that is why you see the failures.
>>
>> In-order to debug this we need to check why pd off and pd on call is not
>> getting called for fa20000.mmc during suspend and resume.
> 
> This is an expected behavior from genpd. On suspend, ti_sci_pd_power_off is not
> called because genpd_finish_suspend() takes an early return when both
> device_awake_path() and GENPD_FLAG_ACTIVE_WAKEUP are true.
>                                                                 
> On resume, ti_sci_pd_power_on is not called because genpd sees the domain status
> as GENPD_STATE_ON (it was never cleared) and skips the power-on entirely.
>
>>
>>>
>>> I attempted to fix this by calling set_device_constraint when
>>> device_wakeup_path() is true but it prevented the system from entering deep
>>> sleep entirely.
>>
>> In AM62P the DM manager selects the low power mode to enter based on the
>> constrains set. The mode selection logic will ensure that if a
>> constraint is set on the device, it will select a low power mode in
>> which the device is kept on or can wake the system up. the MMC is part
>> of main domain and there is no low power mode in which the MMC can stay
>> alive or generate a wake up interrupt. so when a constraint is set of
>> MMC, we cannot enter any low power mode. that why you see a failure.
>>
> 
> This is consistent with what we observed. I am open to suggestions if there is a
> better way to handle this.

We looked into this issue and found the root-cause. What we have 
identified is that, there was bug in our ti_sci driver. In ti_sci driver 
we were not handling the devices in which the child device can act as a 
wake up source, we only handled the cases in which the device can act as 
a wake up source. Kendall is working on fix for this. soon she will post 
the fix.

With the fix, if you suspend it will fail because the wake up is enabled 
for the WiFi, as a constraint will be sent to the DM firmware and DM 
will NACK the suspend because it cannot suspend with WiFi acting as a 
wake up source. But you can disable the wake up of WiFi from the sysfs 
entry and try suspend then suspend and resume will work.

Thankyou very much for digging deep into this issue and proposing a 
solution.

Thanks
Sebin

> 
> Thanks,
> Vitor Soares
> 

Re: [PATCH v1] pmdomain: ti_sci: re-sync TIFS with genpd on resume

[Vignesh Raghavendra]

On 06/05/26 19:56, Sebin Francis wrote:
> Hi Vitor
> 
> On 04/05/26 11:56, Vitor Soares wrote:
>> Hello Sebin
>>
>> On Thu, 2026-04-30 at 16:17 +0530, Sebin Francis wrote:
>>> Hi Vitor,
>>>
>>> On 29/04/26 21:56, Vitor Soares wrote:
>>>> Hi Vignesh
>>>>
>>>> Thank you for the review.
>>>>
>>>> On Wed, 2026-04-29 at 10:03 +0530, Vignesh Raghavendra wrote:
>>>>> Hi Vitor
>>>>>
>>>>> On 27/04/26 13:18, Vitor Soares wrote:
>>>>>> From: Vitor Soares <vitor.soares@toradex.com>
>>>>>>
>>>>>> When a device in a TI SCI power domain is on the wakeup path of a
>>>>>> wakeup-capable child, the suspend path skips genpd_sync_power_off().
>>>>>> No put_device is sent to TIFS and the domain's genpd status remains
>>>>>> ON.
>>>>>
>>>>> Correction of terminologies: TIFS is Root of trust component and is
>>>>> not
>>>>> usually involved in power management, that would be DM (Device
>>>>> Manager)
>>>>>
>>>>
>>>> Thank you for the clarification. I will address this on v2. Also, I was
>>>> thinking
>>>> to replace put_device/get_device with ti_sci_pd_power_off/
>>>> ti_sci_pd_power_on
>>>> if
>>>> that makes more clear the content.
>>>>
>>>>> But to be really sure who is doing what, Could you provide an example
>>>>> and the platform on which you see the issue / external abort?
>>>>>
>>>>
>>>> This was reproduced on our Toradex Verdin AM62P WB and the driver
>>>> for our
>>>> Wi-Fi
>>>> module on the SDIO bus calls device_init_wakeup() during the
>>>> initialization.
>>>>
>>>> After enter in suspend, it show the following error resume path:
>>>>
>>>>
>>>> [   41.759341] Internal error: synchronous external abort:
>>>> 0000000096000010
>>>> [#1]
>>>> SMP
>>>> [   41.843286] CPU: 0 UID: 0 PID: 933 Comm: rtcwake Tainted: G  
>>>> M       O
>>>> 6.18.21-dirty #3 PREEMPT
>>>> [   41.852762] Tainted: [M]=MACHINE_CHECK, [O]=OOT_MODULE
>>>> [   41.857891] Hardware name: Toradex Verdin AM62P WB on Verdin
>>>> Development
>>>> Board (DT)
>>>> [   41.865537] pstate: 200000c5 (nzCv daIF -PAN -UAO -TCO -DIT -SSBS
>>>> BTYPE=-
>>>> -)
>>>> [   41.872492] pc : regmap_mmio_read32le+0x8/0x20
>>>> [   41.876941] lr : regmap_mmio_read+0x44/0x70
>>>> [   41.881120] sp : ffff800081fdb8e0
>>>> [   41.884428] x29: ffff800081fdb8e0 x28: 0000000000000000 x27:
>>>> ffffa95bb64aa9c8
>>>> [   41.891563] x26: 0000000000000000 x25: 0000000000000000 x24:
>>>> 0000000000000000
>>>> [   41.898697] x23: 0000000080000000 x22: ffff000002df5c00 x21:
>>>> ffff800081fdb9b4
>>>> [   41.905831] x20: 0000000000000100 x19: ffff000001286400 x18:
>>>> 0000000000000000
>>>> [   41.912965] x17: 2d69696d67722f79 x16: 687020726f662067 x15:
>>>> ffff00007fb74f40
>>>> [   41.920100] x14: 00000000000002ea x13: 000000000000031f x12:
>>>> 0000000000000000
>>>> [   41.927234] x11: 00000000000000c0 x10: 00000000000009e0 x9 :
>>>> ffff800081fdb7a0
>>>> [   41.934368] x8 : ffff00007fb6ce00 x7 : 0000000000000000 x6 :
>>>> 0000000000000000
>>>> [   41.941502] x5 : ffffa95bb57948d8 x4 : 0000000000000100 x3 :
>>>> 0000000000000100
>>>> [   41.948636] x2 : ffffa95bb5795034 x1 : 0000000000000100 x0 :
>>>> ffff80008025d100
>>>> [   41.955770] Call trace:
>>>> [   41.958211]  regmap_mmio_read32le+0x8/0x20 (P)
>>>> [   41.962655]  _regmap_bus_reg_read+0x70/0xb0
>>>> [   41.966839]  _regmap_read+0x64/0xdc
>>>> [   41.970327]  _regmap_update_bits+0xf4/0x140
>>>> [   41.974509]  regmap_update_bits_base+0x64/0x98
>>>> [   41.978952]  sdhci_am654_runtime_resume+0x138/0x208
>>>> [   41.983830]  pm_generic_runtime_resume+0x2c/0x44
>>>> [   41.988445]  __genpd_runtime_resume+0x30/0x7c
>>>> [   41.992804]  genpd_runtime_resume+0xdc/0x2e8
>>>> [   41.997073]  pm_runtime_force_resume+0x68/0xf4
>>>> [   42.001517]  dpm_run_callback+0x8c/0x14c
>>>> [   42.005439]  device_resume+0x11c/0x34c
>>>> [   42.009188]  dpm_resume+0x178/0x1f0
>>>> [   42.012673]  dpm_resume_end+0x18/0x34
>>>> [   42.016332]  suspend_devices_and_enter+0x4a4/0x668
>>>> [   42.021123]  pm_suspend+0x170/0x2dc
>>>> [   42.024610]  state_store+0x80/0x104
>>>> [   42.028096]  kobj_attr_store+0x18/0x2c
>>>> [   42.031845]  sysfs_kf_write+0x7c/0x94
>>>> [   42.035508]  kernfs_fop_write_iter+0x130/0x1fc
>>>> [   42.039949]  vfs_write+0x200/0x370
>>>> [   42.043351]  ksys_write+0x6c/0x100
>>>> [   42.046752]  __arm64_sys_write+0x1c/0x28
>>>> [   42.050673]  invoke_syscall.constprop.0+0x50/0xe4
>>>> [   42.055378]  do_el0_svc+0x40/0xc4
>>>> [   42.058691]  el0_svc+0x40/0x15c
>>>> [   42.061834]  el0t_64_sync_handler+0xa0/0xe4
>>>> [   42.066015]  el0t_64_sync+0x198/0x19c
>>>> [   42.069680] Code: aa0603e0 d65f03c0 f9400000 8b214000 (b9400000)
>>>>
>>>>>
>>>>>>
>>>>>> TIFS powers off the hardware during deep sleep regardless, since it
>>>>>> was never informed to keep the domain active. On resume, because the
>>>>>> domain's genpd status is ON, no get_device is issued. The driver
>>>>>> then accesses registers of a powered-off domain, causing a
>>>>>> synchronous external abort (AXI bus error, ESR 0x96000010).
>>>>>
>>>>> Hmm, if something is wakeup source, I would expect even TIFS/DM not to
>>>>> turn if off, else module wakeup wouldn't work.
>>>>>
>>>>
>>>> I tested UART as a wakeup source and I couldn't reproduce this
>>>> issue. My
>>>> understanding is that UART has its own TI SCI domain and
>>>> device_may_wakeup()
>>>> is
>>>> true directly on that domain device, so the set_device_constraint fires
>>>> correctly and DM keeps it powered.
>>>>
>>>> Here is my tracking of the issue:
>>>>
>>>> Wi-Fi driver registers as wakeup source:
>>>> device_init_wakeup(mmc0:0001)
>>>>
>>>> During suspend/resume.
>>>> dpm_suspend()
>>>> ->genpd_suspend_dev(fa20000.mmc)
>>>>      ->ti_sci_pd_suspend(fa20000.mmc)
>>>>         ->ti_sci_pd_set_wkup_constraint(fa20000.mmc)
>>>>           device_may_wakeup(fa20000.mmc)  = false
>>>>           set_device_constraint never sent to DM
>>>>
>>>>
>>>> dpm_suspend_noirq()
>>>> ->genpd_finish_suspend(fa20000.mmc)
>>>>     ->device_awake_path(fa20000.mmc) = true
>>>>     ->GENPD_FLAG_ACTIVE_WAKEUP = true
>>>>       genpd status = GENPD_STATE_ON
>>>>       skip power_off (ti_sci_pd_power_off)
>>>>
>>>> On deep sleep entry, DM powers off fa20000.mmc independently.
>>>> It received no set_device_constraint nor ti_sci_pd_power_off.
>>>
>>> In AM62P fa20000.mmc is part of main domain. During deepsleep the entire
>>> main domain is turned off by the DM, that is why you see the failures.
>>>
>>> In-order to debug this we need to check why pd off and pd on call is not
>>> getting called for fa20000.mmc during suspend and resume.
>>
>> This is an expected behavior from genpd. On suspend,
>> ti_sci_pd_power_off is not
>> called because genpd_finish_suspend() takes an early return when both
>> device_awake_path() and GENPD_FLAG_ACTIVE_WAKEUP are true.
>>                                                                 On
>> resume, ti_sci_pd_power_on is not called because genpd sees the domain
>> status
>> as GENPD_STATE_ON (it was never cleared) and skips the power-on entirely.
>>
>>>
>>>>
>>>> I attempted to fix this by calling set_device_constraint when
>>>> device_wakeup_path() is true but it prevented the system from
>>>> entering deep
>>>> sleep entirely.
>>>
>>> In AM62P the DM manager selects the low power mode to enter based on the
>>> constrains set. The mode selection logic will ensure that if a
>>> constraint is set on the device, it will select a low power mode in
>>> which the device is kept on or can wake the system up. the MMC is part
>>> of main domain and there is no low power mode in which the MMC can stay
>>> alive or generate a wake up interrupt. so when a constraint is set of
>>> MMC, we cannot enter any low power mode. that why you see a failure.
>>>
>>
>> This is consistent with what we observed. I am open to suggestions if
>> there is a
>> better way to handle this.
> 
> We looked into this issue and found the root-cause. What we have
> identified is that, there was bug in our ti_sci driver. In ti_sci driver
> we were not handling the devices in which the child device can act as a
> wake up source, we only handled the cases in which the device can act as
> a wake up source. Kendall is working on fix for this. soon she will post
> the fix.
> 
> With the fix, if you suspend it will fail because the wake up is enabled
> for the WiFi, as a constraint will be sent to the DM firmware and DM
> will NACK the suspend because it cannot suspend with WiFi acting as a
> wake up source. But you can disable the wake up of WiFi from the sysfs
> entry and try suspend then suspend and resume will work.
> 

To avoid above and still have wakeup from WiFi supported (if needed), 
this should use IO Daisy chain as the wakeup source and not SDIO 
directly (MMC IP isnt capable of waking up the system but any activity 
on the IOPAD can wakeup the system (similar to Main UART)

https://software-dl.ti.com/processor-sdk-linux/esd/AM62PX/latest/exports/docs/linux/Foundational_Components/Power_Management/pm_wakeup_sources.html#main-i-o-daisy-chain

[...]

-- 
Regards
Vignesh
https://ti.com/opensource