[Bug 221842] ACPI: EC/battery AML deadlock also on HP ZBook Fury 16 G11; triggered at the end of a battery recharge

[Jeremy Rutman <jeremy.rutman@rutmanip.com>]

Hello,

I am seeing the AB/BA deadlock described in kernel Bugzilla #221842
(kec_query / acpi_ec_event_processor vs kacpi_notify /
acpi_battery_notify, reported on an HP ZBook Firefly 14 G10) on a
second HP model, an HP ZBook Fury 16 G11. Seven hangs in four weeks,
on two BIOS versions. Below are the wait graphs from the hung-task
traces and one new datum: the hangs fall at the end of a battery
recharge on AC. At the end are a suggested firmware fix, a suggested
kernel-side mitigation, and the workaround I am deploying.

Machine
-------
  HP ZBook Fury 16 G11 Mobile Workstation PC (board 8CA4)
  i9-13950HX, NVIDIA RTX 5000 Ada Laptop GPU
  BIOS W96 Ver. 01.03.00 (09/01/2025), then W96 Ver. 01.05.01
  (05/05/2026), the newest available to me; EC firmware 9.69
  Kernel 6.17.0-1032-oem (Ubuntu 26.04.1 LTS). The kernel is tainted
  OE by the NVIDIA open module 580.178.04; the deadlock involves only
  ACPI paths, and #221842 shows it on 6.17 and 7.0 generic kernels.
  Always on AC; long unattended batch jobs. Suspend is disabled.

Occurrences
-----------
  Start of hang      BIOS      Notes
  2026-09-16 09:59   01.03.00  hung-task report
  2026-09-17 03:31   01.03.00  hung-task report
  2026-09-18 20:04   01.03.00  hung-task report
  2026-10-01 03:29   01.03.00  hung-task report
  2026-10-06 02:53   01.05.01  hung-task report
  2026-10-07 ~19:41  01.05.01  hung-task report at 19:51
  2026-10-14 ~11:19  01.05.01  no hung-task report: an unrelated
                               I/O stall had used up
                               hung_task_warnings=10 the day before;
                               "Method reached maximum reentrancy
                               limit (255)" from 13:45 on

Once it starts it never clears; every one ended in a forced
power-off. Unrelated hung-task episodes on the same machine (USB
disk writeback stalls) cleared by themselves. The cap of 10 reports
matters: it hid the head of the chain on 10-06 and the whole hang
on 10-14.

Wait graph
----------
In every one of the six traced hangs, a kacpi_notify worker running
acpi_battery_notify() holds a lock that the others wait on, alongside
kec_query workers running EC _Qxx methods. The cleanest is
2026-10-07, a two-party cycle (trimmed stacks):

  task:kworker/7:1 pid:1556456 state:D
  Workqueue: kec_query acpi_ec_event_processor
   acpi_os_wait_semaphore
   acpi_ds_begin_method_execution      <- entering a Serialized method
   acpi_evaluate_object
   acpi_ec_event_processor

  task:kworker/7:2 pid:1798807 state:D
  Workqueue: kacpi_notify acpi_os_execute_deferred
   acpi_os_wait_semaphore
   acpi_ex_acquire_mutex                <- AML Acquire()
   acpi_evaluate_object
   acpi_battery_get_info                <- _BIX
   acpi_battery_notify
   acpi_ev_notify_dispatch
   acpi_os_execute_deferred

  INFO: task kworker/7:1:1556456 blocked on a semaphore likely last
        held by task kworker/7:2:1798807
  INFO: task kworker/7:2:1798807 blocked on a semaphore likely last
        held by task kworker/7:1:1556456

Everything else (thermald via int340x_thermal_get_zone_temp, upowerd
via acpi_battery_get_property, further kec_query workers, and later
PID 1, see below) queues behind these two.

The other five traces (09-16, 09-17, 09-18, 10-01, 10-06) show the
same holder: a kacpi_notify worker in acpi_battery_notify ->
acpi_battery_update. On 10-06 that battery worker is itself waiting
to enter a Serialized method, from acpi_battery_update ->
acpi_bus_get_status (_STA), held by another kacpi_notify worker
whose trace fell beyond the 10-report cap.

Reading of the AML pattern (inferred from the stacks; I have not yet
confirmed the method names against the DSDT):

  EC _Qxx (battery event):  Acquire(M) ... Notify(BATx, 0x80/0x81)
                            ... calls Serialized method S  -> blocks
  battery handler (_BIX/_BST/_STA, run by the kernel on kacpi_notify
  while the _Qxx is still running):
                            enters Serialized S ... Acquire(M) -> blocks

Since ACPICA enforces SyncLevel ordering both on Acquire()
(acpi_ex_acquire_mutex_object) and on entering a Serialized method
(acpi_ds_begin_method_execution, AE_AML_MUTEX_ORDER), M and S
presumably share a SyncLevel (likely 0), so nothing catches the
inversion.

Trigger: the end of a recharge
------------------------------
On AC this machine lets the battery drift from 100% to 94% while
reporting "fully-charged", then recharges. upowerd samples the battery
every 30 s while charging, and its history stops when upowerd blocks.
The two hangs that fall inside upower's history both hit about 20
minutes after the battery reached 99%, i.e. at charge termination:

  2026-10-07  18:49 94% -> charging   19:18 99%
              19:41:12 last upowerd sample; hung-task reports at 19:51
  2026-10-14  10:39 94% -> charging   10:58 98%
              11:19:12 last upowerd sample; 11:30 PID 1 blocked

Between 10-08 and 10-14, 15 recharges started; 14 completed normally
and the 15th deadlocked. So it is a race in the notifications around
charge termination (perhaps the full-charge-capacity update), hit
about once in 15 cycles.

Why it takes the whole machine down
-----------------------------------
Any later read of /sys/class/power_supply/AC/online blocks too. On
systemd systems, PID 1 reads it to evaluate ConditionACPower=, which
Ubuntu ships on anacron, apt-daily, apt-daily-upgrade, e2scrub_all,
logrotate, man-db and plocate-updatedb. The next such timer wedges
PID 1 (10-07 20:16 apt-daily; 10-14 ~11:30 anacron). After that no
unit or app can start, and "reboot" does nothing (tried twice on
10-14); only a forced power-off recovers. Running processes that do
not touch ACPI carry on, so it looks like a partial freeze.

Suggested fixes
---------------
1. Firmware (HP). Take M and S in one order on both paths: in the
   battery _Qxx, release M before calling S (or before the
   Notify()), or drop Serialized from S and protect its body with M
   so that every path takes M first. Giving S and M distinct
   SyncLevels would let the interpreter reject the inverted order
   (AE_AML_MUTEX_ORDER) instead of deadlocking.

2. Kernel (suggestion, not built or tested). The cycle needs the
   battery methods to run while the _Qxx that sent the Notify() is
   still executing. acpi_battery_notify() could wait for pending EC
   queries before touching the battery methods; untested sketch:

   --- a/drivers/acpi/battery.c
   +++ b/drivers/acpi/battery.c
   @@ static void acpi_battery_notify(acpi_handle handle, u32 event,
    	old = battery->bat;
   +	/*
   +	 * Some HP firmware sends battery notifications from an EC _Qxx
   +	 * that still holds an AML mutex the battery methods take, then
   +	 * enters a Serialized method they also use. Let the _Qxx finish
   +	 * first (bug 221842).
   +	 */
   +	acpi_ec_flush_work();
    	if (battery_notification_delay_ms > 0)
    		msleep(battery_notification_delay_ms);

   acpi_ec_flush_work() is declared in drivers/acpi/internal.h only
   under CONFIG_PM_SLEEP, so the declaration would need to move, and
   the change needs review for any path where an EC query waits on
   kacpi_notify. A narrower variant uses the existing mechanism: add
   the affected HP models to bat_dmi_table with
   battery_notification_delay_quirk (1000 ms). That only narrows
   the window, but needs no new code.

3. Workaround (userspace, no rebuild), deploying now. Unbind the
   battery driver at boot, so nothing evaluates the battery methods
   when the EC signals:
     echo -n PNP0C0A:00 > /sys/bus/acpi/drivers/battery/unbind
   This loses the battery in upower and GNOME; I replace upower's
   critical-battery action with a clean poweroff after 10 minutes
   without AC. Separately, drop-ins that reset ConditionACPower= keep
   PID 1 alive, and an EC-deadlock detector reboots cleanly. I will
   report whether the hangs stop. At about one hang a week, that
   takes a few weeks.

Available on request: full kernel journals for all seven boots,
upower's battery history, and acpidump output (not yet taken).

This analysis was prepared with the help of an AI assistant (Claude,
Anthropic) working from this machine's logs; the traces and timings
quoted are from the machine.

Thanks,
Jeremy Rutman