ucsi_acpi: intermittent "PPM init failed" at boot is never retried, Type-C event handling stays dead for the session

ucsi_acpi: intermittent "PPM init failed" at boot is never retried, Type-C event handling stays dead for the session

[Jacob Riff]

Hi,

On a Lenovo ThinkPad X1 Carbon Gen 14 (21V7CTO1WW, Panther Lake), UCSI
initialization intermittently fails at boot, and because the failure is
never retried, no typec ports are registered for the rest of the
session. Most visible consequence: after unplugging and replugging the
USB-C charger (a PD monitor), the machine silently never resumes
charging. Notably, DisplayPort alt mode on the same port continues to
work across replugs in this state, and charging does work if the
charger was already attached at boot (EC autonomous) - it is
specifically resumption of charging after a replug that is lost, which
makes the failure easy to miss until the battery is unexpectedly
drained.

Failure rate: 5 of 8 boots on one day of testing. Reproduced on two
firmware versions including the latest (BIOS N4OET49W/1.12 and
N4OET51W/1.14, EC 1.09 and 1.10). No clear correlation with whether
anything is attached at boot: two boots one minute apart with the same
setup split ok/fail.

Kernel: 7.1.4 (Arch Linux, unpatched in this area)

Two failure flavors seen:

  ucsi_acpi USBC000:00: error -ENODEV: PPM init failed

and:

  ucsi_acpi USBC000:00: possible UCSI driver bug 2
  ucsi_acpi USBC000:00: error -EINVAL: PPM init failed

Both appear ~1s after the typec ports bind. On failed boots
/sys/class/typec/ stays empty.

The part that suggests a driver-side improvement: recovery is trivial.
Reloading the module seconds later has succeeded on every attempt so
far (double digits by now):

  modprobe -r ucsi_acpi && sleep 2 && modprobe ucsi_acpi

after which connectors register and charging renegotiates immediately,
no replug needed.

ucsi_init_work() currently only requeues the init work for
-EPROBE_DEFER (up to UCSI_ROLE_SWITCH_WAIT_COUNT). Given that an
immediate retry reliably succeeds here, would it be reasonable to also
retry a few times on other errors (-ENODEV/-EINVAL) before giving up?
The PPM on these machines appears to simply not be ready to answer
during a window around when init runs.

Possibly related prior reports of Lenovo PPMs being slow/unready at
init: the "usb: typec: ucsi: increase timeout for PPM reset operations"
RFC (Feb 2025) and Ubuntu bug #2054928 (ThinkPad E490, -ENODEV).

Happy to test patches on this hardware.

Thanks,
Jacob Riff

Re: ucsi_acpi: intermittent "PPM init failed" at boot is never retried, Type-C event handling stays dead for the session

[Heikki Krogerus]

On Sat, Jul 25, 2026 at 02:31:53PM -0700, Jacob Riff wrote:
> Hi,
> 
> On a Lenovo ThinkPad X1 Carbon Gen 14 (21V7CTO1WW, Panther Lake), UCSI
> initialization intermittently fails at boot, and because the failure is
> never retried, no typec ports are registered for the rest of the
> session. Most visible consequence: after unplugging and replugging the
> USB-C charger (a PD monitor), the machine silently never resumes
> charging. Notably, DisplayPort alt mode on the same port continues to
> work across replugs in this state, and charging does work if the
> charger was already attached at boot (EC autonomous) - it is
> specifically resumption of charging after a replug that is lost, which
> makes the failure easy to miss until the battery is unexpectedly
> drained.
> 
> Failure rate: 5 of 8 boots on one day of testing. Reproduced on two
> firmware versions including the latest (BIOS N4OET49W/1.12 and
> N4OET51W/1.14, EC 1.09 and 1.10). No clear correlation with whether
> anything is attached at boot: two boots one minute apart with the same
> setup split ok/fail.
> 
> Kernel: 7.1.4 (Arch Linux, unpatched in this area)
> 
> Two failure flavors seen:
> 
>   ucsi_acpi USBC000:00: error -ENODEV: PPM init failed
> 
> and:
> 
>   ucsi_acpi USBC000:00: possible UCSI driver bug 2
>   ucsi_acpi USBC000:00: error -EINVAL: PPM init failed
> 
> Both appear ~1s after the typec ports bind. On failed boots
> /sys/class/typec/ stays empty.
> 
> The part that suggests a driver-side improvement: recovery is trivial.
> Reloading the module seconds later has succeeded on every attempt so
> far (double digits by now):
> 
>   modprobe -r ucsi_acpi && sleep 2 && modprobe ucsi_acpi
> 
> after which connectors register and charging renegotiates immediately,
> no replug needed.
> 
> ucsi_init_work() currently only requeues the init work for
> -EPROBE_DEFER (up to UCSI_ROLE_SWITCH_WAIT_COUNT). Given that an
> immediate retry reliably succeeds here, would it be reasonable to also
> retry a few times on other errors (-ENODEV/-EINVAL) before giving up?
> The PPM on these machines appears to simply not be ready to answer
> during a window around when init runs.
> 
> Possibly related prior reports of Lenovo PPMs being slow/unready at
> init: the "usb: typec: ucsi: increase timeout for PPM reset operations"
> RFC (Feb 2025) and Ubuntu bug #2054928 (ThinkPad E490, -ENODEV).
> 
> Happy to test patches on this hardware.

I'm sorry to keep you waiting. I'm just letting you know that this
issue is in my queue, but right now I don't have time. I will try
to take a closer look at this later this month.

Thanks,

-- 
heikki

Re: ucsi_acpi: intermittent "PPM init failed" at boot is never retried, Type-C event handling stays dead for the session

[Heikki Krogerus]

On Wed, Aug 05, 2026 at 02:11:43PM +0200, Heikki Krogerus wrote:
> On Sat, Jul 25, 2026 at 02:31:53PM -0700, Jacob Riff wrote:
> > Hi,
> > 
> > On a Lenovo ThinkPad X1 Carbon Gen 14 (21V7CTO1WW, Panther Lake), UCSI
> > initialization intermittently fails at boot, and because the failure is
> > never retried, no typec ports are registered for the rest of the
> > session. Most visible consequence: after unplugging and replugging the
> > USB-C charger (a PD monitor), the machine silently never resumes
> > charging. Notably, DisplayPort alt mode on the same port continues to
> > work across replugs in this state, and charging does work if the
> > charger was already attached at boot (EC autonomous) - it is
> > specifically resumption of charging after a replug that is lost, which
> > makes the failure easy to miss until the battery is unexpectedly
> > drained.
> > 
> > Failure rate: 5 of 8 boots on one day of testing. Reproduced on two
> > firmware versions including the latest (BIOS N4OET49W/1.12 and
> > N4OET51W/1.14, EC 1.09 and 1.10). No clear correlation with whether
> > anything is attached at boot: two boots one minute apart with the same
> > setup split ok/fail.
> > 
> > Kernel: 7.1.4 (Arch Linux, unpatched in this area)
> > 
> > Two failure flavors seen:
> > 
> >   ucsi_acpi USBC000:00: error -ENODEV: PPM init failed
> > 
> > and:
> > 
> >   ucsi_acpi USBC000:00: possible UCSI driver bug 2
> >   ucsi_acpi USBC000:00: error -EINVAL: PPM init failed
> > 
> > Both appear ~1s after the typec ports bind. On failed boots
> > /sys/class/typec/ stays empty.
> > 
> > The part that suggests a driver-side improvement: recovery is trivial.
> > Reloading the module seconds later has succeeded on every attempt so
> > far (double digits by now):
> > 
> >   modprobe -r ucsi_acpi && sleep 2 && modprobe ucsi_acpi
> > 
> > after which connectors register and charging renegotiates immediately,
> > no replug needed.
> > 
> > ucsi_init_work() currently only requeues the init work for
> > -EPROBE_DEFER (up to UCSI_ROLE_SWITCH_WAIT_COUNT). Given that an
> > immediate retry reliably succeeds here, would it be reasonable to also
> > retry a few times on other errors (-ENODEV/-EINVAL) before giving up?
> > The PPM on these machines appears to simply not be ready to answer
> > during a window around when init runs.
> > 
> > Possibly related prior reports of Lenovo PPMs being slow/unready at
> > init: the "usb: typec: ucsi: increase timeout for PPM reset operations"
> > RFC (Feb 2025) and Ubuntu bug #2054928 (ThinkPad E490, -ENODEV).
> > 
> > Happy to test patches on this hardware.
> 
> I'm sorry to keep you waiting. I'm just letting you know that this
> issue is in my queue, but right now I don't have time. I will try
> to take a closer look at this later this month.

Can you check does this improve the situation:
https://lore.kernel.org/linux-usb/20260805085725.389761-1-huangwei@kylinos.cn/

Thanks,

-- 
heikki

Re: ucsi_acpi: intermittent "PPM init failed" at boot is never retried, Type-C event handling stays dead for the session

[Jacob Riff]

Hi Heikki,

I tested Huang Wei's patch on the affected machine (applied to 7.1.4's
ucsi.c, confirmed active via srcversion): no improvement.

8 boots with the patch: 4x "PPM init failed", 4 ok. That is the same
~50-60% failure rate as the unpatched baseline (5 of 8).

That matches what the error flavors suggest: this doesn't look like a
slow PPM. The -ENODEV case is GET_CAPABILITY completing but reporting
zero connectors, and the -EINVAL case comes with "possible UCSI driver
bug 2", i.e. the PPM rejecting a standard command. These are prompt
wrong answers rather than missing ones, so a longer completion wait
doesn't change the outcome.

One additional data point from this round: all 8 test boots had
nothing attached to any Type-C port, and the failure rate was
unchanged. So it doesn't appear related to PD contract negotiation
racing the handshake either. The PPM simply seems not ready to answer
correctly for a window during boot.

Which still leaves the observation from my original mail: reloading
ucsi_acpi a couple of seconds later has succeeded on every attempt
(well into double digits now), and the machine then behaves fully
normally. Would retrying ucsi_init() a few times on -ENODEV/-EINVAL,
similar to the existing -EPROBE_DEFER requeue, be an acceptable
direction? Happy to test such a patch on this hardware.

- Jacob Riff
Mobile: 760-840-6060

On Wed, Aug 5, 2026 at 5:46 AM Heikki Krogerus
<heikki.krogerus@linux.intel.com> wrote:
>
> On Wed, Aug 05, 2026 at 02:11:43PM +0200, Heikki Krogerus wrote:
> > On Sat, Jul 25, 2026 at 02:31:53PM -0700, Jacob Riff wrote:
> > > Hi,
> > >
> > > On a Lenovo ThinkPad X1 Carbon Gen 14 (21V7CTO1WW, Panther Lake), UCSI
> > > initialization intermittently fails at boot, and because the failure is
> > > never retried, no typec ports are registered for the rest of the
> > > session. Most visible consequence: after unplugging and replugging the
> > > USB-C charger (a PD monitor), the machine silently never resumes
> > > charging. Notably, DisplayPort alt mode on the same port continues to
> > > work across replugs in this state, and charging does work if the
> > > charger was already attached at boot (EC autonomous) - it is
> > > specifically resumption of charging after a replug that is lost, which
> > > makes the failure easy to miss until the battery is unexpectedly
> > > drained.
> > >
> > > Failure rate: 5 of 8 boots on one day of testing. Reproduced on two
> > > firmware versions including the latest (BIOS N4OET49W/1.12 and
> > > N4OET51W/1.14, EC 1.09 and 1.10). No clear correlation with whether
> > > anything is attached at boot: two boots one minute apart with the same
> > > setup split ok/fail.
> > >
> > > Kernel: 7.1.4 (Arch Linux, unpatched in this area)
> > >
> > > Two failure flavors seen:
> > >
> > >   ucsi_acpi USBC000:00: error -ENODEV: PPM init failed
> > >
> > > and:
> > >
> > >   ucsi_acpi USBC000:00: possible UCSI driver bug 2
> > >   ucsi_acpi USBC000:00: error -EINVAL: PPM init failed
> > >
> > > Both appear ~1s after the typec ports bind. On failed boots
> > > /sys/class/typec/ stays empty.
> > >
> > > The part that suggests a driver-side improvement: recovery is trivial.
> > > Reloading the module seconds later has succeeded on every attempt so
> > > far (double digits by now):
> > >
> > >   modprobe -r ucsi_acpi && sleep 2 && modprobe ucsi_acpi
> > >
> > > after which connectors register and charging renegotiates immediately,
> > > no replug needed.
> > >
> > > ucsi_init_work() currently only requeues the init work for
> > > -EPROBE_DEFER (up to UCSI_ROLE_SWITCH_WAIT_COUNT). Given that an
> > > immediate retry reliably succeeds here, would it be reasonable to also
> > > retry a few times on other errors (-ENODEV/-EINVAL) before giving up?
> > > The PPM on these machines appears to simply not be ready to answer
> > > during a window around when init runs.
> > >
> > > Possibly related prior reports of Lenovo PPMs being slow/unready at
> > > init: the "usb: typec: ucsi: increase timeout for PPM reset operations"
> > > RFC (Feb 2025) and Ubuntu bug #2054928 (ThinkPad E490, -ENODEV).
> > >
> > > Happy to test patches on this hardware.
> >
> > I'm sorry to keep you waiting. I'm just letting you know that this
> > issue is in my queue, but right now I don't have time. I will try
> > to take a closer look at this later this month.
>
> Can you check does this improve the situation:
> https://lore.kernel.org/linux-usb/20260805085725.389761-1-huangwei@kylinos.cn/
>
> Thanks,
>
> --
> heikki

[RFC PATCH] usb: typec: ucsi: retry init when the PPM answers early commands incorrectly

[Jacob Riff]

On some platforms the PPM is not ready to answer commands correctly
for a short window during boot. On the Lenovo ThinkPad X1 Carbon
Gen 14 (21V7, tested on BIOS 1.12 and 1.14) roughly half of all boots
fail ucsi_init() with either -ENODEV (GET_CAPABILITY completes but
reports zero connectors) or -EINVAL (a standard command is rejected,
logged as "possible UCSI driver bug"). The failure is not a timeout:
increasing the sync command completion wait does not change the rate.

Since ucsi_init_work() only requeues on -EPROBE_DEFER, a single bad
answer during that window leaves UCSI dead for the whole session, so
Type-C events are never handled again; most visibly, the machine
silently never resumes charging after the charger is replugged.
Manually reloading ucsi_acpi a few seconds later has succeeded on
every attempt observed, which suggests simply retrying is enough.

Retry -ENODEV and -EINVAL the same way as the role switch wait, log
the retries at debug level, keep the loud report for the case where
the retries are exhausted, and note when init only succeeded after
retrying.

Tested on the affected machine: across 8 consecutive boots with this
patch, 5 hit the failure (matching the historical ~50-60% rate) and
all 5 recovered on the first retry ("PPM init succeeded after 2
attempts"). 0 of 8 boots ended with UCSI unusable, where ~5 of 8
would have without the patch.

Signed-off-by: Jacob Riff <jacob@riff.dk>
---
This is the failure previously reported in the thread
"ucsi_acpi: intermittent PPM init failed at boot is never retried"
and reproduces on the latest firmware for the machine. Happy to test
alternative approaches on this hardware.

--- a/drivers/usb/typec/ucsi/ucsi.c
+++ b/drivers/usb/typec/ucsi/ucsi.c
@@ -2211,18 +2211,36 @@
 	int ret;
 
 	ret = ucsi_init(ucsi);
-	if (ret)
-		dev_err_probe(ucsi->dev, ret, "PPM init failed\n");
+	if (!ret) {
+		if (ucsi->work_count)
+			dev_info(ucsi->dev,
+				 "PPM init succeeded after %u attempts\n",
+				 ucsi->work_count + 1);
+		return;
+	}
 
-	if (ret == -EPROBE_DEFER) {
-		if (ucsi->work_count++ > UCSI_ROLE_SWITCH_WAIT_COUNT) {
-			dev_err(ucsi->dev, "PPM init failed, stop trying\n");
+	/*
+	 * On some platforms the PPM is not ready to answer commands
+	 * correctly for a short window during boot: standard commands are
+	 * rejected or GET_CAPABILITY reports zero connectors, seen as
+	 * -EINVAL or -ENODEV from ucsi_init(), and a retry moments later
+	 * succeeds (observed on Lenovo ThinkPad X1 Carbon Gen 14, where
+	 * this affects roughly half of all boots). Retry those like the
+	 * USB role switch wait instead of giving up on the first attempt.
+	 */
+	if (ret == -EPROBE_DEFER || ret == -ENODEV || ret == -EINVAL) {
+		if (ucsi->work_count++ < UCSI_ROLE_SWITCH_WAIT_COUNT) {
+			dev_dbg(ucsi->dev, "PPM init failed (%pe), retrying\n",
+				ERR_PTR(ret));
+			queue_delayed_work(system_dfl_long_wq, &ucsi->work,
+					   UCSI_ROLE_SWITCH_INTERVAL);
 			return;
 		}
-
-		queue_delayed_work(system_dfl_long_wq, &ucsi->work,
-				   UCSI_ROLE_SWITCH_INTERVAL);
+		dev_err(ucsi->dev, "PPM init failed, stop trying\n");
+		return;
 	}
+
+	dev_err_probe(ucsi->dev, ret, "PPM init failed\n");
 }
 
 /**

Re: [RFC PATCH] usb: typec: ucsi: retry init when the PPM answers early commands incorrectly

[Heikki Krogerus]

On Fri, Aug 14, 2026 at 05:57:24PM -0700, Jacob Riff wrote:
> On some platforms the PPM is not ready to answer commands correctly
> for a short window during boot. On the Lenovo ThinkPad X1 Carbon
> Gen 14 (21V7, tested on BIOS 1.12 and 1.14) roughly half of all boots
> fail ucsi_init() with either -ENODEV (GET_CAPABILITY completes but
> reports zero connectors) or -EINVAL (a standard command is rejected,
> logged as "possible UCSI driver bug"). The failure is not a timeout:
> increasing the sync command completion wait does not change the rate.
> 
> Since ucsi_init_work() only requeues on -EPROBE_DEFER, a single bad
> answer during that window leaves UCSI dead for the whole session, so
> Type-C events are never handled again; most visibly, the machine
> silently never resumes charging after the charger is replugged.
> Manually reloading ucsi_acpi a few seconds later has succeeded on
> every attempt observed, which suggests simply retrying is enough.
> 
> Retry -ENODEV and -EINVAL the same way as the role switch wait, log
> the retries at debug level, keep the loud report for the case where
> the retries are exhausted, and note when init only succeeded after
> retrying.
> 
> Tested on the affected machine: across 8 consecutive boots with this
> patch, 5 hit the failure (matching the historical ~50-60% rate) and
> all 5 recovered on the first retry ("PPM init succeeded after 2
> attempts"). 0 of 8 boots ended with UCSI unusable, where ~5 of 8
> would have without the patch.
> 
> Signed-off-by: Jacob Riff <jacob@riff.dk>
> ---
> This is the failure previously reported in the thread
> "ucsi_acpi: intermittent PPM init failed at boot is never retried"
> and reproduces on the latest firmware for the machine. Happy to test
> alternative approaches on this hardware.

Thanks for the patch. I don't have currently any better ideas so we
can go ahead with this.

Thanks,

> --- a/drivers/usb/typec/ucsi/ucsi.c
> +++ b/drivers/usb/typec/ucsi/ucsi.c
> @@ -2211,18 +2211,36 @@
>  	int ret;
>  
>  	ret = ucsi_init(ucsi);
> -	if (ret)
> -		dev_err_probe(ucsi->dev, ret, "PPM init failed\n");
> +	if (!ret) {
> +		if (ucsi->work_count)
> +			dev_info(ucsi->dev,
> +				 "PPM init succeeded after %u attempts\n",
> +				 ucsi->work_count + 1);
> +		return;
> +	}
>  
> -	if (ret == -EPROBE_DEFER) {
> -		if (ucsi->work_count++ > UCSI_ROLE_SWITCH_WAIT_COUNT) {
> -			dev_err(ucsi->dev, "PPM init failed, stop trying\n");
> +	/*
> +	 * On some platforms the PPM is not ready to answer commands
> +	 * correctly for a short window during boot: standard commands are
> +	 * rejected or GET_CAPABILITY reports zero connectors, seen as
> +	 * -EINVAL or -ENODEV from ucsi_init(), and a retry moments later
> +	 * succeeds (observed on Lenovo ThinkPad X1 Carbon Gen 14, where
> +	 * this affects roughly half of all boots). Retry those like the
> +	 * USB role switch wait instead of giving up on the first attempt.
> +	 */
> +	if (ret == -EPROBE_DEFER || ret == -ENODEV || ret == -EINVAL) {
> +		if (ucsi->work_count++ < UCSI_ROLE_SWITCH_WAIT_COUNT) {
> +			dev_dbg(ucsi->dev, "PPM init failed (%pe), retrying\n",
> +				ERR_PTR(ret));
> +			queue_delayed_work(system_dfl_long_wq, &ucsi->work,
> +					   UCSI_ROLE_SWITCH_INTERVAL);
>  			return;
>  		}
> -
> -		queue_delayed_work(system_dfl_long_wq, &ucsi->work,
> -				   UCSI_ROLE_SWITCH_INTERVAL);
> +		dev_err(ucsi->dev, "PPM init failed, stop trying\n");
> +		return;
>  	}
> +
> +	dev_err_probe(ucsi->dev, ret, "PPM init failed\n");
>  }
>  
>  /**

-- 
heikki

Re: [RFC PATCH] usb: typec: ucsi: retry init when the PPM answers early commands incorrectly

[Jacob Riff]

On Mon, 17 Aug 2026 11:19:24 +0800, huangwei wrote:
> There the commands do complete, but the EC takes longer than the 5s
> hardcoded wait in ucsi_sync_control_common(). The fix for that case is
> already reviewed [1] (thanks Heikki, Fedor), so both failure modes of
> the "PPM not ready during boot" window would be covered:
>
>   - slow answers  -> longer completion timeout [1]
>   - wrong answers -> init retry (this RFC)

Hi Huang Wei,

Thanks for taking a look, and good to see the two failure modes of
the boot window end up with complementary fixes.

> One small question on the -EINVAL retry: ucsi_read_error() also
> returns -EINVAL for UCSI_ERROR_INVALID_CON_NUM / UNREGONIZED_CMD /
> INVALID_CMD_ARGUMENT, which are logged as "possible UCSI driver bug".
> Retrying those is harmless in practice (debug-level logging, bounded
> attempts), but it delays the report of a genuine driver bug from the
> first occurrence to the 100th. Did you consider distinguishing the
> PPM-not-ready case from the real error case, or is the simplicity of
> retrying both worth that trade-off?

I considered it, but by error code the two cases are identical. On
this machine the not-ready PPM answers a valid standard init command
with UNREGONIZED_CMD / INVALID_CMD_ARGUMENT - exactly the codes
ucsi_read_error() maps to "possible UCSI driver bug". So the only
discriminators left are context and persistence, which is what the
patch relies on: the retry exists only in the ucsi_init_work() path
(a runtime -EINVAL is unaffected), the not-ready window has cleared
after a single retry in every boot observed here, and a genuine bug
fails deterministically, exhausts the bounded attempts (100 x 100ms
= 10s with the role switch constants) and still ends in the loud
"PPM init failed, stop trying".

On delaying the report: the "possible UCSI driver bug" message is
dev_err in ucsi_read_error() and this patch does not touch it, so it
still fires on the first occurrence. With the retries a deterministic
bug would print it on every attempt before the final error, so the
cost is a 10 second delay of the final verdict rather than a hidden
report. That seemed a fair price for reusing the existing role switch
retry machinery unchanged.

Best regards,
Jacob