[PATCH v1] pmdomain: mediatek: Fix MT8183 hang on boot

[PATCH v1] pmdomain: mediatek: Fix MT8183 hang on boot

[Dmitry Osipenko]

Depending on firmware, part of the MFG domains may be left ON at boot
leaving only some MFG cores powered, to let the ACP to prefetch the GPU
region when the display controller is brought up. This doesn't play well
with an eventual delay in probing Panfrost when the display controller is
fully set up, as that would make genpd's sync_state() to power off the
domain while ACP tries to prefetch: this is causing an AXI stall,
effectively freezing the AP indefinitely. In order to prevent this from
happening, the sync_state() functionality must be obliterated on all of
the MFG domains: while this guarantees a power leakage if the bootloader
boots the kernel with MFG PDs partially powered on, this is the only way
to ensure stable operation of the SoC during boot on devices with such
firmware because, of course, those will never officially receive a
firmware update.

Fixes Kappa Chromebook hanging during system boot.

Fixes: 0e789b491ba0 ("pmdomain: core: Leave powered-on genpds on until sync_state")
Fixes: 13a4b7fb6260 ("pmdomain: core: Leave powered-on genpds on until late_initcall_sync")
Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
---
 drivers/pmdomain/mediatek/mt8183-pm-domains.h | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/pmdomain/mediatek/mt8183-pm-domains.h b/drivers/pmdomain/mediatek/mt8183-pm-domains.h
index 3742782a2702..5e33b8628e85 100644
--- a/drivers/pmdomain/mediatek/mt8183-pm-domains.h
+++ b/drivers/pmdomain/mediatek/mt8183-pm-domains.h
@@ -47,7 +47,7 @@ static const struct scpsys_domain_data scpsys_domain_data_mt8183[] = {
 		.pwr_sta2nd_offs = 0x0184,
 		.sram_pdn_bits = 0,
 		.sram_pdn_ack_bits = 0,
-		.caps = MTK_SCPD_DOMAIN_SUPPLY,
+		.caps = MTK_SCPD_DOMAIN_SUPPLY | MTK_SCPD_KEEP_DEFAULT_OFF,
 	},
 	[MT8183_POWER_DOMAIN_MFG] = {
 		.name = "mfg",
@@ -57,7 +57,7 @@ static const struct scpsys_domain_data scpsys_domain_data_mt8183[] = {
 		.pwr_sta2nd_offs = 0x0184,
 		.sram_pdn_bits = GENMASK(8, 8),
 		.sram_pdn_ack_bits = GENMASK(12, 12),
-		.caps = MTK_SCPD_DOMAIN_SUPPLY,
+		.caps = MTK_SCPD_DOMAIN_SUPPLY | MTK_SCPD_KEEP_DEFAULT_OFF,
 	},
 	[MT8183_POWER_DOMAIN_MFG_CORE0] = {
 		.name = "mfg_core0",
@@ -67,6 +67,7 @@ static const struct scpsys_domain_data scpsys_domain_data_mt8183[] = {
 		.pwr_sta2nd_offs = 0x0184,
 		.sram_pdn_bits = GENMASK(8, 8),
 		.sram_pdn_ack_bits = GENMASK(12, 12),
+		.caps = MTK_SCPD_KEEP_DEFAULT_OFF,
 	},
 	[MT8183_POWER_DOMAIN_MFG_CORE1] = {
 		.name = "mfg_core1",
@@ -76,6 +77,7 @@ static const struct scpsys_domain_data scpsys_domain_data_mt8183[] = {
 		.pwr_sta2nd_offs = 0x0184,
 		.sram_pdn_bits = GENMASK(8, 8),
 		.sram_pdn_ack_bits = GENMASK(12, 12),
+		.caps = MTK_SCPD_KEEP_DEFAULT_OFF,
 	},
 	[MT8183_POWER_DOMAIN_MFG_2D] = {
 		.name = "mfg_2d",
@@ -85,6 +87,7 @@ static const struct scpsys_domain_data scpsys_domain_data_mt8183[] = {
 		.pwr_sta2nd_offs = 0x0184,
 		.sram_pdn_bits = GENMASK(8, 8),
 		.sram_pdn_ack_bits = GENMASK(12, 12),
+		.caps = MTK_SCPD_KEEP_DEFAULT_OFF,
 		.bp_cfg = {
 			BUS_PROT_WR(INFRA,
 				    MT8183_TOP_AXI_PROT_EN_1_MFG,
-- 
2.54.0

Re: [PATCH v1] pmdomain: mediatek: Fix MT8183 hang on boot

[AngeloGioacchino Del Regno]

On 7/22/26 15:27, Dmitry Osipenko wrote:
> Depending on firmware, part of the MFG domains may be left ON at boot
> leaving only some MFG cores powered, to let the ACP to prefetch the GPU
> region when the display controller is brought up. This doesn't play well
> with an eventual delay in probing Panfrost when the display controller is
> fully set up, as that would make genpd's sync_state() to power off the
> domain while ACP tries to prefetch: this is causing an AXI stall,
> effectively freezing the AP indefinitely. In order to prevent this from
> happening, the sync_state() functionality must be obliterated on all of
> the MFG domains: while this guarantees a power leakage if the bootloader
> boots the kernel with MFG PDs partially powered on, this is the only way
> to ensure stable operation of the SoC during boot on devices with such
> firmware because, of course, those will never officially receive a
> firmware update.
> 
> Fixes Kappa Chromebook hanging during system boot.
> 
> Fixes: 0e789b491ba0 ("pmdomain: core: Leave powered-on genpds on until sync_state")
> Fixes: 13a4b7fb6260 ("pmdomain: core: Leave powered-on genpds on until late_initcall_sync")
> Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>

Reviewed-by: AngeloGioacchino Del Regno <angelogioacchino.delregno@collabora.com>

Re: [PATCH v1] pmdomain: mediatek: Fix MT8183 hang on boot

[Ulf Hansson]

On Wed, Jul 22, 2026 at 3:28 PM Dmitry Osipenko
<dmitry.osipenko@collabora.com> wrote:
>
> Depending on firmware, part of the MFG domains may be left ON at boot
> leaving only some MFG cores powered, to let the ACP to prefetch the GPU
> region when the display controller is brought up. This doesn't play well
> with an eventual delay in probing Panfrost when the display controller is
> fully set up, as that would make genpd's sync_state() to power off the
> domain while ACP tries to prefetch: this is causing an AXI stall,
> effectively freezing the AP indefinitely. In order to prevent this from

I haven't made any detailed investigation, but to me, it sounds like
there is a missing consumer somewhere.

Should the ACP/panfrost devices be a consumer of the MFG domains as
well? Or perhaps, if the ACP/panfrost should rely on the display
controller to be probed and powered-on, to be able to probe?

> happening, the sync_state() functionality must be obliterated on all of
> the MFG domains: while this guarantees a power leakage if the bootloader
> boots the kernel with MFG PDs partially powered on, this is the only way
> to ensure stable operation of the SoC during boot on devices with such
> firmware because, of course, those will never officially receive a
> firmware update.
>
> Fixes Kappa Chromebook hanging during system boot.
>
> Fixes: 0e789b491ba0 ("pmdomain: core: Leave powered-on genpds on until sync_state")
> Fixes: 13a4b7fb6260 ("pmdomain: core: Leave powered-on genpds on until late_initcall_sync")
> Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>

Kind regards
Uffe

> ---
>  drivers/pmdomain/mediatek/mt8183-pm-domains.h | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/pmdomain/mediatek/mt8183-pm-domains.h b/drivers/pmdomain/mediatek/mt8183-pm-domains.h
> index 3742782a2702..5e33b8628e85 100644
> --- a/drivers/pmdomain/mediatek/mt8183-pm-domains.h
> +++ b/drivers/pmdomain/mediatek/mt8183-pm-domains.h
> @@ -47,7 +47,7 @@ static const struct scpsys_domain_data scpsys_domain_data_mt8183[] = {
>                 .pwr_sta2nd_offs = 0x0184,
>                 .sram_pdn_bits = 0,
>                 .sram_pdn_ack_bits = 0,
> -               .caps = MTK_SCPD_DOMAIN_SUPPLY,
> +               .caps = MTK_SCPD_DOMAIN_SUPPLY | MTK_SCPD_KEEP_DEFAULT_OFF,
>         },
>         [MT8183_POWER_DOMAIN_MFG] = {
>                 .name = "mfg",
> @@ -57,7 +57,7 @@ static const struct scpsys_domain_data scpsys_domain_data_mt8183[] = {
>                 .pwr_sta2nd_offs = 0x0184,
>                 .sram_pdn_bits = GENMASK(8, 8),
>                 .sram_pdn_ack_bits = GENMASK(12, 12),
> -               .caps = MTK_SCPD_DOMAIN_SUPPLY,
> +               .caps = MTK_SCPD_DOMAIN_SUPPLY | MTK_SCPD_KEEP_DEFAULT_OFF,
>         },
>         [MT8183_POWER_DOMAIN_MFG_CORE0] = {
>                 .name = "mfg_core0",
> @@ -67,6 +67,7 @@ static const struct scpsys_domain_data scpsys_domain_data_mt8183[] = {
>                 .pwr_sta2nd_offs = 0x0184,
>                 .sram_pdn_bits = GENMASK(8, 8),
>                 .sram_pdn_ack_bits = GENMASK(12, 12),
> +               .caps = MTK_SCPD_KEEP_DEFAULT_OFF,
>         },
>         [MT8183_POWER_DOMAIN_MFG_CORE1] = {
>                 .name = "mfg_core1",
> @@ -76,6 +77,7 @@ static const struct scpsys_domain_data scpsys_domain_data_mt8183[] = {
>                 .pwr_sta2nd_offs = 0x0184,
>                 .sram_pdn_bits = GENMASK(8, 8),
>                 .sram_pdn_ack_bits = GENMASK(12, 12),
> +               .caps = MTK_SCPD_KEEP_DEFAULT_OFF,
>         },
>         [MT8183_POWER_DOMAIN_MFG_2D] = {
>                 .name = "mfg_2d",
> @@ -85,6 +87,7 @@ static const struct scpsys_domain_data scpsys_domain_data_mt8183[] = {
>                 .pwr_sta2nd_offs = 0x0184,
>                 .sram_pdn_bits = GENMASK(8, 8),
>                 .sram_pdn_ack_bits = GENMASK(12, 12),
> +               .caps = MTK_SCPD_KEEP_DEFAULT_OFF,
>                 .bp_cfg = {
>                         BUS_PROT_WR(INFRA,
>                                     MT8183_TOP_AXI_PROT_EN_1_MFG,
> --
> 2.54.0
>

Re: [PATCH v1] pmdomain: mediatek: Fix MT8183 hang on boot

[Brady Norander]

On 7/22/26 09:27, Dmitry Osipenko wrote:
> Depending on firmware, part of the MFG domains may be left ON at boot
> leaving only some MFG cores powered, to let the ACP to prefetch the GPU
> region when the display controller is brought up. This doesn't play well
> with an eventual delay in probing Panfrost when the display controller is
> fully set up, as that would make genpd's sync_state() to power off the
> domain while ACP tries to prefetch: this is causing an AXI stall,
> effectively freezing the AP indefinitely. In order to prevent this from
> happening, the sync_state() functionality must be obliterated on all of
> the MFG domains: while this guarantees a power leakage if the bootloader
> boots the kernel with MFG PDs partially powered on, this is the only way
> to ensure stable operation of the SoC during boot on devices with such
> firmware because, of course, those will never officially receive a
> firmware update.
> 
> Fixes Kappa Chromebook hanging during system boot.

I also saw this same issue on my MT8192 Hayato Chromebook, and I also 
saw an issue on my MT8183 and MT8186 Chromebooks where the audio was 
broken. I fixed all of those issues by setting the GENPD_FLAG_NO_STAY_ON 
flag on all domains. I held off from submitting that change as I was 
unsure if it was the "correct" way to handle it, but perhaps it is. 
Either way, this issue is not specific to only MT8183.

Re: [PATCH v1] pmdomain: mediatek: Fix MT8183 hang on boot

[AngeloGioacchino Del Regno]

On 7/25/26 07:11, Brady Norander wrote:
> On 7/22/26 09:27, Dmitry Osipenko wrote:
>> Depending on firmware, part of the MFG domains may be left ON at boot
>> leaving only some MFG cores powered, to let the ACP to prefetch the GPU
>> region when the display controller is brought up. This doesn't play well
>> with an eventual delay in probing Panfrost when the display controller is
>> fully set up, as that would make genpd's sync_state() to power off the
>> domain while ACP tries to prefetch: this is causing an AXI stall,
>> effectively freezing the AP indefinitely. In order to prevent this from
>> happening, the sync_state() functionality must be obliterated on all of
>> the MFG domains: while this guarantees a power leakage if the bootloader
>> boots the kernel with MFG PDs partially powered on, this is the only way
>> to ensure stable operation of the SoC during boot on devices with such
>> firmware because, of course, those will never officially receive a
>> firmware update.
>>
>> Fixes Kappa Chromebook hanging during system boot.
> 
> I also saw this same issue on my MT8192 Hayato Chromebook, and I also saw an issue 
> on my MT8183 and MT8186 Chromebooks where the audio was broken. I fixed all of 
> those issues by setting the GENPD_FLAG_NO_STAY_ON flag on all domains. I held off 
> from submitting that change as I was unsure if it was the "correct" way to handle 
> it, but perhaps it is. Either way, this issue is not specific to only MT8183.

Audio is something a bit different I believe.

For MFG on MT8183 specifically (and potentially same generation or even slightly
older) there's an issue that is very specific and described in the description
of this commit.

So, specifically for audio, I think that there may be something else that is
wrong if you're seeing such a behavior - as in, some dependencies may be missing
from somewhere (some devicetree node), or something else.

In any case, I'm always open for discussion, of course.

Re: [PATCH v1] pmdomain: mediatek: Fix MT8183 hang on boot

[AngeloGioacchino Del Regno]

On 7/24/26 17:01, Ulf Hansson wrote:
> On Wed, Jul 22, 2026 at 3:28 PM Dmitry Osipenko
> <dmitry.osipenko@collabora.com> wrote:
>>
>> Depending on firmware, part of the MFG domains may be left ON at boot
>> leaving only some MFG cores powered, to let the ACP to prefetch the GPU
>> region when the display controller is brought up. This doesn't play well
>> with an eventual delay in probing Panfrost when the display controller is
>> fully set up, as that would make genpd's sync_state() to power off the
>> domain while ACP tries to prefetch: this is causing an AXI stall,
>> effectively freezing the AP indefinitely. In order to prevent this from
> 
> I haven't made any detailed investigation, but to me, it sounds like
> there is a missing consumer somewhere.
> 
> Should the ACP/panfrost devices be a consumer of the MFG domains as
> well? Or perhaps, if the ACP/panfrost should rely on the display
> controller to be probed and powered-on, to be able to probe?
> 

This is about how the bootloader is leaving the display controller, the ACP,
and the MFG registers before booting the kernel.

The kernel drivers themselves will reset this weird state when the HW gets
to full probe so there won't be any more issue(s) from that point onwards.

Actually, there would be A LOT to say about this, but I'll be very brief...

Basically, before booting the kernel, the bootloader is leaving a half-assed
setup to potentially support continuous splash functionality, but without it
being actually used on the device(s) that we're talking about here...

Again, I should put a remark on every word of that last paragraph, but this
is the gist of it.

If you're curious or need more information, of course I can expand a bit,
there's no secret involved here, it's just that I'm avoiding to write a wall
of text around that...

P.S.: We've been discussing that internally and chasing this issue for months
before deciding to "solve" it like this.

Cheers,
Angelo

>> happening, the sync_state() functionality must be obliterated on all of
>> the MFG domains: while this guarantees a power leakage if the bootloader
>> boots the kernel with MFG PDs partially powered on, this is the only way
>> to ensure stable operation of the SoC during boot on devices with such
>> firmware because, of course, those will never officially receive a
>> firmware update.
>>
>> Fixes Kappa Chromebook hanging during system boot.
>>
>> Fixes: 0e789b491ba0 ("pmdomain: core: Leave powered-on genpds on until sync_state")
>> Fixes: 13a4b7fb6260 ("pmdomain: core: Leave powered-on genpds on until late_initcall_sync")
>> Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
> 
> Kind regards
> Uffe
> 
>> ---
>>   drivers/pmdomain/mediatek/mt8183-pm-domains.h | 7 +++++--
>>   1 file changed, 5 insertions(+), 2 deletions(-)
>>
>> diff --git a/drivers/pmdomain/mediatek/mt8183-pm-domains.h b/drivers/pmdomain/mediatek/mt8183-pm-domains.h
>> index 3742782a2702..5e33b8628e85 100644
>> --- a/drivers/pmdomain/mediatek/mt8183-pm-domains.h
>> +++ b/drivers/pmdomain/mediatek/mt8183-pm-domains.h
>> @@ -47,7 +47,7 @@ static const struct scpsys_domain_data scpsys_domain_data_mt8183[] = {
>>                  .pwr_sta2nd_offs = 0x0184,
>>                  .sram_pdn_bits = 0,
>>                  .sram_pdn_ack_bits = 0,
>> -               .caps = MTK_SCPD_DOMAIN_SUPPLY,
>> +               .caps = MTK_SCPD_DOMAIN_SUPPLY | MTK_SCPD_KEEP_DEFAULT_OFF,
>>          },
>>          [MT8183_POWER_DOMAIN_MFG] = {
>>                  .name = "mfg",
>> @@ -57,7 +57,7 @@ static const struct scpsys_domain_data scpsys_domain_data_mt8183[] = {
>>                  .pwr_sta2nd_offs = 0x0184,
>>                  .sram_pdn_bits = GENMASK(8, 8),
>>                  .sram_pdn_ack_bits = GENMASK(12, 12),
>> -               .caps = MTK_SCPD_DOMAIN_SUPPLY,
>> +               .caps = MTK_SCPD_DOMAIN_SUPPLY | MTK_SCPD_KEEP_DEFAULT_OFF,
>>          },
>>          [MT8183_POWER_DOMAIN_MFG_CORE0] = {
>>                  .name = "mfg_core0",
>> @@ -67,6 +67,7 @@ static const struct scpsys_domain_data scpsys_domain_data_mt8183[] = {
>>                  .pwr_sta2nd_offs = 0x0184,
>>                  .sram_pdn_bits = GENMASK(8, 8),
>>                  .sram_pdn_ack_bits = GENMASK(12, 12),
>> +               .caps = MTK_SCPD_KEEP_DEFAULT_OFF,
>>          },
>>          [MT8183_POWER_DOMAIN_MFG_CORE1] = {
>>                  .name = "mfg_core1",
>> @@ -76,6 +77,7 @@ static const struct scpsys_domain_data scpsys_domain_data_mt8183[] = {
>>                  .pwr_sta2nd_offs = 0x0184,
>>                  .sram_pdn_bits = GENMASK(8, 8),
>>                  .sram_pdn_ack_bits = GENMASK(12, 12),
>> +               .caps = MTK_SCPD_KEEP_DEFAULT_OFF,
>>          },
>>          [MT8183_POWER_DOMAIN_MFG_2D] = {
>>                  .name = "mfg_2d",
>> @@ -85,6 +87,7 @@ static const struct scpsys_domain_data scpsys_domain_data_mt8183[] = {
>>                  .pwr_sta2nd_offs = 0x0184,
>>                  .sram_pdn_bits = GENMASK(8, 8),
>>                  .sram_pdn_ack_bits = GENMASK(12, 12),
>> +               .caps = MTK_SCPD_KEEP_DEFAULT_OFF,
>>                  .bp_cfg = {
>>                          BUS_PROT_WR(INFRA,
>>                                      MT8183_TOP_AXI_PROT_EN_1_MFG,
>> --
>> 2.54.0
>>

Re: [PATCH v1] pmdomain: mediatek: Fix MT8183 hang on boot

[Ulf Hansson]

On Mon, Jul 27, 2026 at 1:04 PM AngeloGioacchino Del Regno
<angelogioacchino.delregno@collabora.com> wrote:
>
> On 7/24/26 17:01, Ulf Hansson wrote:
> > On Wed, Jul 22, 2026 at 3:28 PM Dmitry Osipenko
> > <dmitry.osipenko@collabora.com> wrote:
> >>
> >> Depending on firmware, part of the MFG domains may be left ON at boot
> >> leaving only some MFG cores powered, to let the ACP to prefetch the GPU
> >> region when the display controller is brought up. This doesn't play well
> >> with an eventual delay in probing Panfrost when the display controller is
> >> fully set up, as that would make genpd's sync_state() to power off the
> >> domain while ACP tries to prefetch: this is causing an AXI stall,
> >> effectively freezing the AP indefinitely. In order to prevent this from
> >
> > I haven't made any detailed investigation, but to me, it sounds like
> > there is a missing consumer somewhere.
> >
> > Should the ACP/panfrost devices be a consumer of the MFG domains as
> > well? Or perhaps, if the ACP/panfrost should rely on the display
> > controller to be probed and powered-on, to be able to probe?
> >
>
> This is about how the bootloader is leaving the display controller, the ACP,
> and the MFG registers before booting the kernel.

Right.

>
> The kernel drivers themselves will reset this weird state when the HW gets
> to full probe so there won't be any more issue(s) from that point onwards.
>
> Actually, there would be A LOT to say about this, but I'll be very brief...
>
> Basically, before booting the kernel, the bootloader is leaving a half-assed
> setup to potentially support continuous splash functionality, but without it
> being actually used on the device(s) that we're talking about here...

So does that mean that the print in scpsys_add_one_domain() ("A
default off power domain has been ON") gets printed for these PM
domains?

If so, perhaps it would be better to turn off the PM domains (if
possible) before registering them with genpd?

>
> Again, I should put a remark on every word of that last paragraph, but this
> is the gist of it.
>
> If you're curious or need more information, of course I can expand a bit,
> there's no secret involved here, it's just that I'm avoiding to write a wall
> of text around that...
>
> P.S.: We've been discussing that internally and chasing this issue for months
> before deciding to "solve" it like this.

Huh, I see, thanks for sharing.

I think it would be nice to clarify a little bit more in the commit
message, as it all kind of indicates to me that there is a missing
consumer or that a parent needs to be runtime resumed before a child
is probed.

Or another way of looking at it, that perhaps makes better sense.
Based on your latest information, it seems like the PM domain isn't
really fully powered-on by the bootloader, but still we are telling
genpd that it is, leading to an inconsistent state and errors.

[...]

Kind regards
Uffe

Re: [PATCH v1] pmdomain: mediatek: Fix MT8183 hang on boot

[Brady Norander]

On 7/27/26 06:57, AngeloGioacchino Del Regno wrote:
> On 7/25/26 07:11, Brady Norander wrote:
>> On 7/22/26 09:27, Dmitry Osipenko wrote:
>>> Depending on firmware, part of the MFG domains may be left ON at boot
>>> leaving only some MFG cores powered, to let the ACP to prefetch the GPU
>>> region when the display controller is brought up. This doesn't play well
>>> with an eventual delay in probing Panfrost when the display 
>>> controller is
>>> fully set up, as that would make genpd's sync_state() to power off the
>>> domain while ACP tries to prefetch: this is causing an AXI stall,
>>> effectively freezing the AP indefinitely. In order to prevent this from
>>> happening, the sync_state() functionality must be obliterated on all of
>>> the MFG domains: while this guarantees a power leakage if the bootloader
>>> boots the kernel with MFG PDs partially powered on, this is the only way
>>> to ensure stable operation of the SoC during boot on devices with such
>>> firmware because, of course, those will never officially receive a
>>> firmware update.
>>>
>>> Fixes Kappa Chromebook hanging during system boot.
>>
>> I also saw this same issue on my MT8192 Hayato Chromebook, and I also 
>> saw an issue on my MT8183 and MT8186 Chromebooks where the audio was 
>> broken. I fixed all of those issues by setting the 
>> GENPD_FLAG_NO_STAY_ON flag on all domains. I held off from submitting 
>> that change as I was unsure if it was the "correct" way to handle it, 
>> but perhaps it is. Either way, this issue is not specific to only MT8183.
> 
> Audio is something a bit different I believe.
> 
> For MFG on MT8183 specifically (and potentially same generation or even 
> slightly
> older) there's an issue that is very specific and described in the 
> description
> of this commit.
> 

I do believe that the issue I'm seeing on MT8192 is very similar to the 
issue described on MT8183. On my MT8192 Hayato, I get the following 
sync_state warnings due to these two display components not having 
drivers which bind to them:

   [   25.828186] mtk-power-controller 10006000.syscon:power-controller: 
sync_state() pending due to 1400d000.postmask
   [   25.838380] mtk-power-controller 10006000.syscon:power-controller: 
sync_state() pending due to 1400e000.dither

This device boots just fine up until I reach the point where my display 
manager starts, where I get a full SoC hang. If I disable my display 
manager from starting at boot, the device boots to a shell just fine. 
While MT8192 doesn't have ACP to prefetch anything from the GPU, the 
display manager (which uses 3D accel) still appears to trigger an AXI 
stall (I don't know how to verify this is actually what happens, but it 
seems likely) and I would assume the bootloader also leaves the MFG 
domains partially enabled on MT8192 based on this behavior.

> So, specifically for audio, I think that there may be something else 
> that is
> wrong if you're seeing such a behavior - as in, some dependencies may be 
> missing
> from somewhere (some devicetree node), or something else.
> 

The audio issues I see on MT8183 and MT8186 also seem to have the same 
root cause, the bootloader leaving domains partially enabled. On my 
MT8183 krane I get these warnings:

   [   21.705672] mtk-power-controller 10006000.syscon:power-controller: 
sync_state() pending due to 14005000.dma-controller
   [   21.705677] mtk-power-controller 10006000.syscon:power-controller: 
sync_state() pending due to 14006000.mdp3-wdma
   [   21.705688] mtk-power-controller 10006000.syscon:power-controller: 
sync_state() pending due to 14012000.dither

And I get this error when attempting to play audio:

   [  217.282793] mt8183-audio 11220000.audio-controller:mt8183-afe-pcm: 
mtk_afe_pcm_pointer hw_ptr err

In both the MT8192 GPU case and the MT8183/8186 audio case, the issue 
was fixed with the following diff to not use the sync_state system to 
keep domains powered on:

diff --git a/drivers/pmdomain/mediatek/mtk-pm-domains.c 
b/drivers/pmdomain/mediatek/mtk-pm-domains.c
index 9c9323c8c93a..b0868e24e241 100644
--- a/drivers/pmdomain/mediatek/mtk-pm-domains.c
+++ b/drivers/pmdomain/mediatek/mtk-pm-domains.c
@@ -639,6 +639,7 @@ generic_pm_domain *scpsys_add_one_domain(struct 
scpsys *scpsys, struct device_no

  	pd->genpd.power_off = scpsys_power_off;
  	pd->genpd.power_on = scpsys_power_on;
+	pd->genpd.flags |= GENPD_FLAG_NO_STAY_ON;

  	if (MTK_SCPD_CAPS(pd, MTK_SCPD_ACTIVE_WAKEUP))
  		pd->genpd.flags |= GENPD_FLAG_ACTIVE_WAKEUP;

What isn't clear to me is why leaving display domains powered on is 
preventing audio from working. Maybe there is some dependency that isn't 
clear to me? The other thing which isn't clear to me is why I never 
experienced the boot hang on MT8183. Either way, I'm experiencing issues 
on multiple SoCs which are fixed by allowing the domains to power off 
and back on.

> In any case, I'm always open for discussion, of course.

Re: [PATCH v1] pmdomain: mediatek: Fix MT8183 hang on boot

[AngeloGioacchino Del Regno]

On 7/27/26 18:48, Ulf Hansson wrote:
> On Mon, Jul 27, 2026 at 1:04 PM AngeloGioacchino Del Regno
> <angelogioacchino.delregno@collabora.com> wrote:
>>
>> On 7/24/26 17:01, Ulf Hansson wrote:
>>> On Wed, Jul 22, 2026 at 3:28 PM Dmitry Osipenko
>>> <dmitry.osipenko@collabora.com> wrote:
>>>>
>>>> Depending on firmware, part of the MFG domains may be left ON at boot
>>>> leaving only some MFG cores powered, to let the ACP to prefetch the GPU
>>>> region when the display controller is brought up. This doesn't play well
>>>> with an eventual delay in probing Panfrost when the display controller is
>>>> fully set up, as that would make genpd's sync_state() to power off the
>>>> domain while ACP tries to prefetch: this is causing an AXI stall,
>>>> effectively freezing the AP indefinitely. In order to prevent this from
>>>
>>> I haven't made any detailed investigation, but to me, it sounds like
>>> there is a missing consumer somewhere.
>>>
>>> Should the ACP/panfrost devices be a consumer of the MFG domains as
>>> well? Or perhaps, if the ACP/panfrost should rely on the display
>>> controller to be probed and powered-on, to be able to probe?
>>>
>>
>> This is about how the bootloader is leaving the display controller, the ACP,
>> and the MFG registers before booting the kernel.
> 
> Right.
> 
>>
>> The kernel drivers themselves will reset this weird state when the HW gets
>> to full probe so there won't be any more issue(s) from that point onwards.
>>
>> Actually, there would be A LOT to say about this, but I'll be very brief...
>>
>> Basically, before booting the kernel, the bootloader is leaving a half-assed
>> setup to potentially support continuous splash functionality, but without it
>> being actually used on the device(s) that we're talking about here...
> 
> So does that mean that the print in scpsys_add_one_domain() ("A
> default off power domain has been ON") gets printed for these PM
> domains?
> 

Yeah...

> If so, perhaps it would be better to turn off the PM domains (if
> possible) before registering them with genpd?
> 

Eh, that's not possible :-(

Doing that would bring us back to square 1. The SoC will lockup.... ACP is still
trying to constantly prefetch, and the issue here is going totally unpowered, as
explained in the commit description - because that's the HW condition that causes
the stall eventually.

To make things "funnier" (in this case) for everyone, the ACP is always-on, of
course, so there's no cleaner way around it.

>>
>> Again, I should put a remark on every word of that last paragraph, but this
>> is the gist of it.
>>
>> If you're curious or need more information, of course I can expand a bit,
>> there's no secret involved here, it's just that I'm avoiding to write a wall
>> of text around that...
>>
>> P.S.: We've been discussing that internally and chasing this issue for months
>> before deciding to "solve" it like this.
> 
> Huh, I see, thanks for sharing.
> 
> I think it would be nice to clarify a little bit more in the commit
> message, as it all kind of indicates to me that there is a missing
> consumer or that a parent needs to be runtime resumed before a child
> is probed.
> 

Eh for both me and Dmitry the commit message looked clear, but yeah, it's surely
just because we bashed our heads over this issue, and our knowledge makes things
look obvious when they are definitely *not*, as you just pointed out - so thanks
for that!

So yeah, Dmitry, if you can please send a v2 with some more clarification in the
commit message (practically, add the information that I shared in this topic),
that would be awesome.

> Or another way of looking at it, that perhaps makes better sense.
> Based on your latest information, it seems like the PM domain isn't
> really fully powered-on by the bootloader, but still we are telling
> genpd that it is, leading to an inconsistent state and errors.
> 

Yeah, the bootloader is doing random things, leaving the PDs only partially powered
ON (just the SRAM, which is half of the MFG_ASYNC setup, and 2D accelerator, but
not the rest, which makes little sense because there's another parent in the middle
that is not powered at all...) and that's why this issue is experienced.

I think this is really just a bootloader bug in this case, so if we lived in a
parallel universe where the MediaTek kernel drivers supported continuous splash on
upstream... I'm mostly sure that this issue would still be there, and that this SoC
would still randomly and spectacularly freeze without this commit.

Cheers,
Angelo


> [...]
> 
> Kind regards
> Uffe

Re: [PATCH v1] pmdomain: mediatek: Fix MT8183 hang on boot

[AngeloGioacchino Del Regno]

On 7/27/26 19:38, Brady Norander wrote:
> On 7/27/26 06:57, AngeloGioacchino Del Regno wrote:
>> On 7/25/26 07:11, Brady Norander wrote:
>>> On 7/22/26 09:27, Dmitry Osipenko wrote:
>>>> Depending on firmware, part of the MFG domains may be left ON at boot
>>>> leaving only some MFG cores powered, to let the ACP to prefetch the GPU
>>>> region when the display controller is brought up. This doesn't play well
>>>> with an eventual delay in probing Panfrost when the display controller is
>>>> fully set up, as that would make genpd's sync_state() to power off the
>>>> domain while ACP tries to prefetch: this is causing an AXI stall,
>>>> effectively freezing the AP indefinitely. In order to prevent this from
>>>> happening, the sync_state() functionality must be obliterated on all of
>>>> the MFG domains: while this guarantees a power leakage if the bootloader
>>>> boots the kernel with MFG PDs partially powered on, this is the only way
>>>> to ensure stable operation of the SoC during boot on devices with such
>>>> firmware because, of course, those will never officially receive a
>>>> firmware update.
>>>>
>>>> Fixes Kappa Chromebook hanging during system boot.
>>>
>>> I also saw this same issue on my MT8192 Hayato Chromebook, and I also saw an 
>>> issue on my MT8183 and MT8186 Chromebooks where the audio was broken. I fixed 
>>> all of those issues by setting the GENPD_FLAG_NO_STAY_ON flag on all domains. I 
>>> held off from submitting that change as I was unsure if it was the "correct" way 
>>> to handle it, but perhaps it is. Either way, this issue is not specific to only 
>>> MT8183.
>>
>> Audio is something a bit different I believe.
>>
>> For MFG on MT8183 specifically (and potentially same generation or even slightly
>> older) there's an issue that is very specific and described in the description
>> of this commit.
>>
> 
> I do believe that the issue I'm seeing on MT8192 is very similar to the issue 
> described on MT8183. On my MT8192 Hayato, I get the following sync_state warnings 
> due to these two display components not having drivers which bind to them:
> 
>    [   25.828186] mtk-power-controller 10006000.syscon:power-controller: 
> sync_state() pending due to 1400d000.postmask
>    [   25.838380] mtk-power-controller 10006000.syscon:power-controller: 
> sync_state() pending due to 1400e000.dither
> 
> This device boots just fine up until I reach the point where my display manager 
> starts, where I get a full SoC hang. If I disable my display manager from starting 
> at boot, the device boots to a shell just fine.

Eeeeeeh....

 > While MT8192 doesn't have ACP to > prefetch anything from the GPU,

....well..... drivers/soc/mediatek/mtk-infracfg.c - read that driver: are you still
sure about what you're saying now? :-P

P.S.: We tried the same trick with MT8183, but that needs more than just that, and
it's anyway a very suboptimal (and ugly) solution anyway.


> the display manager (which uses 3D accel) still 
> appears to trigger an AXI stall (I don't know how to verify this is actually what 
> happens, but it seems likely) and I would assume the bootloader also leaves the MFG 
> domains partially enabled on MT8192 based on this behavior.
> 

That can be a real possibility. Try to apply the same patch on 8192 domains, and
see if and which power domains are kept on in the MFlexGraphics island (you'll see
a print saying that a default off domain "has been on").


>> So, specifically for audio, I think that there may be something else that is
>> wrong if you're seeing such a behavior - as in, some dependencies may be missing
>> from somewhere (some devicetree node), or something else.
>>
> 
> The audio issues I see on MT8183 and MT8186 also seem to have the same root cause, 
> the bootloader leaving domains partially enabled. On my MT8183 krane I get these 
> warnings:
> 
>    [   21.705672] mtk-power-controller 10006000.syscon:power-controller: 
> sync_state() pending due to 14005000.dma-controller
>    [   21.705677] mtk-power-controller 10006000.syscon:power-controller: 
> sync_state() pending due to 14006000.mdp3-wdma
>    [   21.705688] mtk-power-controller 10006000.syscon:power-controller: 
> sync_state() pending due to 14012000.dither
> 
> And I get this error when attempting to play audio:
> 
>    [  217.282793] mt8183-audio 11220000.audio-controller:mt8183-afe-pcm: 
> mtk_afe_pcm_pointer hw_ptr err
> 
> In both the MT8192 GPU case and the MT8183/8186 audio case, the issue was fixed 
> with the following diff to not use the sync_state system to keep domains powered on:
> 

In the audio case, since there's no stall, a cleaner solution could be to trigger a
full reset on the ADSP and on the AFE/PCM/whatever.

There should be a reset register for that.

Alternatively, the drivers are assuming that the device starts from a clean known
state, so some registers are not set as they are left to the initial default state:
adding writes to force those in default state could also be an option... but then
that depends on how many, because otherwise just asserting reset is simply a better
option.

> diff --git a/drivers/pmdomain/mediatek/mtk-pm-domains.c b/drivers/pmdomain/ 
> mediatek/mtk-pm-domains.c
> index 9c9323c8c93a..b0868e24e241 100644
> --- a/drivers/pmdomain/mediatek/mtk-pm-domains.c
> +++ b/drivers/pmdomain/mediatek/mtk-pm-domains.c
> @@ -639,6 +639,7 @@ generic_pm_domain *scpsys_add_one_domain(struct scpsys *scpsys, 
> struct device_no
> 
>       pd->genpd.power_off = scpsys_power_off;
>       pd->genpd.power_on = scpsys_power_on;
> +    pd->genpd.flags |= GENPD_FLAG_NO_STAY_ON;
> 
>       if (MTK_SCPD_CAPS(pd, MTK_SCPD_ACTIVE_WAKEUP))
>           pd->genpd.flags |= GENPD_FLAG_ACTIVE_WAKEUP;
> 
> What isn't clear to me is why leaving display domains powered on is preventing 
> audio from working. Maybe there is some dependency that isn't clear to me?

It's probably due to how MediaTek SoCs are internally structured: I didn't check
on 8186 or 83 specifically, but it's likely that the multimedia IP block (which
also contains multimedia clock controllers, which are powered by multimedia power
domains!!!!) is providing some clock source to some audio related gate (or PLL
even).

When the clock source for the whatever audio related IP is turned off, the IP is
getting in a unclocked state (or even unpowered, depending on what's actually going
on), so when it gets re-clocked, said IP restarts like it was reset with register
defaults - then you get a "known and clean" (is it really clean if this is what is
happening? ugh!) state for the initialization sequence to go on Just Fine (TM).

That could be an explanation... but then please take all this with a grain of salt,
I'm doing an information dump here hoping to help you somehow with this, but I can
not be certain of what's going on unless I really analyze the specific issue on HW.

> The 
> other thing which isn't clear to me is why I never experienced the boot hang on 
> MT8183.

Luck and just the right defconfig, where the right drivers never -EPROBE_DEFER,
and the whole boot process doesn't take "too much time". That's why.

For Dmitry the hang didn't happen all the time either, it did depend on both how
much time it was required for booting and on probe deferrals.

> Either way, I'm experiencing issues on multiple SoCs which are fixed by 
> allowing the domains to power off and back on.
> 

Multiple SoCs... MediaTek? Qualcomm? NXP? And which models?

This thing, anyway, all started since sync_state() was introduced in power domains:
it wasn't an issue before, and AFAIK that actually made more platforms to regress.

I'm not blaming anyone though... this only means that many platforms had some
"hidden" bug (or an easter egg, if you wish? :-P) that was power domain related...
..when you have such situations, it's only a matter of time until something goes
wrong... in this case it all started going wrong with sync_state, but it could've
been worst, or could've happened in other racy ways which would've been way harder
to debug, etc.

Cheers,
Angelo

>> In any case, I'm always open for discussion, of course.
> 

Re: [PATCH v1] pmdomain: mediatek: Fix MT8183 hang on boot

[Brady Norander]

On 7/28/26 04:53, AngeloGioacchino Del Regno wrote:
> On 7/27/26 19:38, Brady Norander wrote:
>> On 7/27/26 06:57, AngeloGioacchino Del Regno wrote:
>>> On 7/25/26 07:11, Brady Norander wrote:
>>>> On 7/22/26 09:27, Dmitry Osipenko wrote:
>>>>> Depending on firmware, part of the MFG domains may be left ON at boot
>>>>> leaving only some MFG cores powered, to let the ACP to prefetch the 
>>>>> GPU
>>>>> region when the display controller is brought up. This doesn't play 
>>>>> well
>>>>> with an eventual delay in probing Panfrost when the display 
>>>>> controller is
>>>>> fully set up, as that would make genpd's sync_state() to power off the
>>>>> domain while ACP tries to prefetch: this is causing an AXI stall,
>>>>> effectively freezing the AP indefinitely. In order to prevent this 
>>>>> from
>>>>> happening, the sync_state() functionality must be obliterated on 
>>>>> all of
>>>>> the MFG domains: while this guarantees a power leakage if the 
>>>>> bootloader
>>>>> boots the kernel with MFG PDs partially powered on, this is the 
>>>>> only way
>>>>> to ensure stable operation of the SoC during boot on devices with such
>>>>> firmware because, of course, those will never officially receive a
>>>>> firmware update.
>>>>>
>>>>> Fixes Kappa Chromebook hanging during system boot.
>>>>
>>>> I also saw this same issue on my MT8192 Hayato Chromebook, and I 
>>>> also saw an issue on my MT8183 and MT8186 Chromebooks where the 
>>>> audio was broken. I fixed all of those issues by setting the 
>>>> GENPD_FLAG_NO_STAY_ON flag on all domains. I held off from 
>>>> submitting that change as I was unsure if it was the "correct" way 
>>>> to handle it, but perhaps it is. Either way, this issue is not 
>>>> specific to only MT8183.
>>>
>>> Audio is something a bit different I believe.
>>>
>>> For MFG on MT8183 specifically (and potentially same generation or 
>>> even slightly
>>> older) there's an issue that is very specific and described in the 
>>> description
>>> of this commit.
>>>
>>
>> I do believe that the issue I'm seeing on MT8192 is very similar to 
>> the issue described on MT8183. On my MT8192 Hayato, I get the 
>> following sync_state warnings due to these two display components not 
>> having drivers which bind to them:
>>
>>    [   25.828186] mtk-power-controller 10006000.syscon:power- 
>> controller: sync_state() pending due to 1400d000.postmask
>>    [   25.838380] mtk-power-controller 10006000.syscon:power- 
>> controller: sync_state() pending due to 1400e000.dither
>>
>> This device boots just fine up until I reach the point where my 
>> display manager starts, where I get a full SoC hang. If I disable my 
>> display manager from starting at boot, the device boots to a shell 
>> just fine.
> 
> Eeeeeeh....
> 
>  > While MT8192 doesn't have ACP to > prefetch anything from the GPU,
> 
> ....well..... drivers/soc/mediatek/mtk-infracfg.c - read that driver: 
> are you still
> sure about what you're saying now? :-P

Poor wording choice. I meant ACP gets disabled on MT8192, so it wouldn't 
have that issue. As I'm writing this now I'm considering the case where 
ACP tries to prefetch before being disabled so it still *may* be an issue.

> 
> P.S.: We tried the same trick with MT8183, but that needs more than just 
> that, and
> it's anyway a very suboptimal (and ugly) solution anyway.
> 
> 
>> the display manager (which uses 3D accel) still appears to trigger an 
>> AXI stall (I don't know how to verify this is actually what happens, 
>> but it seems likely) and I would assume the bootloader also leaves the 
>> MFG domains partially enabled on MT8192 based on this behavior.
>>
> 
> That can be a real possibility. Try to apply the same patch on 8192 
> domains, and
> see if and which power domains are kept on in the MFlexGraphics island 
> (you'll see
> a print saying that a default off domain "has been on").

This does indeed fix the issue I saw on MT8192.

> 
> 
>>> So, specifically for audio, I think that there may be something else 
>>> that is
>>> wrong if you're seeing such a behavior - as in, some dependencies may 
>>> be missing
>>> from somewhere (some devicetree node), or something else.
>>>
>>
>> The audio issues I see on MT8183 and MT8186 also seem to have the same 
>> root cause, the bootloader leaving domains partially enabled. On my 
>> MT8183 krane I get these warnings:
>>
>>    [   21.705672] mtk-power-controller 10006000.syscon:power- 
>> controller: sync_state() pending due to 14005000.dma-controller
>>    [   21.705677] mtk-power-controller 10006000.syscon:power- 
>> controller: sync_state() pending due to 14006000.mdp3-wdma
>>    [   21.705688] mtk-power-controller 10006000.syscon:power- 
>> controller: sync_state() pending due to 14012000.dither
>>
>> And I get this error when attempting to play audio:
>>
>>    [  217.282793] mt8183-audio 11220000.audio-controller:mt8183-afe- 
>> pcm: mtk_afe_pcm_pointer hw_ptr err
>>
>> In both the MT8192 GPU case and the MT8183/8186 audio case, the issue 
>> was fixed with the following diff to not use the sync_state system to 
>> keep domains powered on:
>>
> 
> In the audio case, since there's no stall, a cleaner solution could be 
> to trigger a
> full reset on the ADSP and on the AFE/PCM/whatever.
> 
> There should be a reset register for that.
> 
> Alternatively, the drivers are assuming that the device starts from a 
> clean known
> state, so some registers are not set as they are left to the initial 
> default state:
> adding writes to force those in default state could also be an option... 
> but then
> that depends on how many, because otherwise just asserting reset is 
> simply a better
> option.

I do agree that your proposed solution is cleaner than what I had. I 
still have the feeling that there are more of these hidden bugs caused 
by sync_state in genpd that are waiting to be found. In both the MT8183 
and MT8192 cases, the sync_state warnings were generated because there 
are certain devices which do not have a driver which binds to them. 
Maybe a cleaner solution would be to give these devices a driver so the 
sync_state logic in genpd can work properly.

> 
>> diff --git a/drivers/pmdomain/mediatek/mtk-pm-domains.c b/drivers/ 
>> pmdomain/ mediatek/mtk-pm-domains.c
>> index 9c9323c8c93a..b0868e24e241 100644
>> --- a/drivers/pmdomain/mediatek/mtk-pm-domains.c
>> +++ b/drivers/pmdomain/mediatek/mtk-pm-domains.c
>> @@ -639,6 +639,7 @@ generic_pm_domain *scpsys_add_one_domain(struct 
>> scpsys *scpsys, struct device_no
>>
>>       pd->genpd.power_off = scpsys_power_off;
>>       pd->genpd.power_on = scpsys_power_on;
>> +    pd->genpd.flags |= GENPD_FLAG_NO_STAY_ON;
>>
>>       if (MTK_SCPD_CAPS(pd, MTK_SCPD_ACTIVE_WAKEUP))
>>           pd->genpd.flags |= GENPD_FLAG_ACTIVE_WAKEUP;
>>
>> What isn't clear to me is why leaving display domains powered on is 
>> preventing audio from working. Maybe there is some dependency that 
>> isn't clear to me?
> 
> It's probably due to how MediaTek SoCs are internally structured: I 
> didn't check
> on 8186 or 83 specifically, but it's likely that the multimedia IP block 
> (which
> also contains multimedia clock controllers, which are powered by 
> multimedia power
> domains!!!!) is providing some clock source to some audio related gate 
> (or PLL
> even).
> 
> When the clock source for the whatever audio related IP is turned off, 
> the IP is
> getting in a unclocked state (or even unpowered, depending on what's 
> actually going
> on), so when it gets re-clocked, said IP restarts like it was reset with 
> register
> defaults - then you get a "known and clean" (is it really clean if this 
> is what is
> happening? ugh!) state for the initialization sequence to go on Just 
> Fine (TM).
> 
> That could be an explanation... but then please take all this with a 
> grain of salt,
> I'm doing an information dump here hoping to help you somehow with this, 
> but I can
> not be certain of what's going on unless I really analyze the specific 
> issue on HW.
> 
>> The other thing which isn't clear to me is why I never experienced the 
>> boot hang on MT8183.
> 
> Luck and just the right defconfig, where the right drivers never - 
> EPROBE_DEFER,
> and the whole boot process doesn't take "too much time". That's why.
> 
> For Dmitry the hang didn't happen all the time either, it did depend on 
> both how
> much time it was required for booting and on probe deferrals.
> 
>> Either way, I'm experiencing issues on multiple SoCs which are fixed 
>> by allowing the domains to power off and back on.
>>
> 
> Multiple SoCs... MediaTek? Qualcomm? NXP? And which models?

All of the MediaTek SoCs which I have mentioned (MT8183, MT8192, MT8186) 
and potentially more which I have not personally tested (hardware is 
expensive and you can never have enough of it).

> 
> This thing, anyway, all started since sync_state() was introduced in 
> power domains:
> it wasn't an issue before, and AFAIK that actually made more platforms 
> to regress.
> 
> I'm not blaming anyone though... this only means that many platforms had 
> some
> "hidden" bug (or an easter egg, if you wish? :-P) that was power domain 
> related...
> ..when you have such situations, it's only a matter of time until 
> something goes
> wrong... in this case it all started going wrong with sync_state, but it 
> could've
> been worst, or could've happened in other racy ways which would've been 
> way harder
> to debug, etc.
> 
> Cheers,
> Angelo
> 
>>> In any case, I'm always open for discussion, of course.
>>