domU suspend issue - freeze processes failed - Linux 6.16

domU suspend issue - freeze processes failed - Linux 6.16

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 1913 bytes --]

Hi,

When suspending domU I get the following issue:

    Freezing user space processes
    Freezing user space processes failed after 20.004 seconds (1 tasks refusing to freeze, wq_busy=0):
    task:xl              state:D stack:0     pid:466   tgid:466   ppid:1      task_flags:0x400040 flags:0x00004006
    Call Trace:
     <TASK>
     __schedule+0x2f3/0x780
     schedule+0x27/0x80
     schedule_preempt_disabled+0x15/0x30
     __mutex_lock.constprop.0+0x49f/0x880
     unregister_xenbus_watch+0x216/0x230
     xenbus_write_watch+0xb9/0x220
     xenbus_file_write+0x131/0x1b0
     vfs_writev+0x26c/0x3d0
     ? do_writev+0xeb/0x110
     do_writev+0xeb/0x110
     do_syscall_64+0x84/0x2c0
     ? do_syscall_64+0x200/0x2c0
     ? generic_handle_irq+0x3f/0x60
     ? syscall_exit_work+0x108/0x140
     ? do_syscall_64+0x200/0x2c0
     ? __irq_exit_rcu+0x4c/0xe0
     entry_SYSCALL_64_after_hwframe+0x76/0x7e
    RIP: 0033:0x79b618138642
    RSP: 002b:00007fff9a192fc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000014
    RAX: ffffffffffffffda RBX: 00000000024fd490 RCX: 000079b618138642
    RDX: 0000000000000003 RSI: 00007fff9a193120 RDI: 0000000000000014
    RBP: 00007fff9a193000 R08: 0000000000000000 R09: 0000000000000000
    R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000014
    R13: 00007fff9a193120 R14: 0000000000000003 R15: 0000000000000000
     </TASK>
    OOM killer enabled.
    Restarting tasks: Starting
    Restarting tasks: Done
    xen:manage: do_suspend: freeze processes failed -16

The process in question is `xl devd` daemon. It's a domU serving a
xenvif backend.

I noticed it on 6.16.1, but looking at earlier test logs I see it with
6.16-rc6 already (but interestingly, not 6.16-rc2 yet? feels weird given
seemingly no relevant changes between rc2 and rc6).

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: domU suspend issue - freeze processes failed - Linux 6.16

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 2307 bytes --]

On Fri, Aug 22, 2025 at 04:39:33PM +0200, Marek Marczykowski-Górecki wrote:
> Hi,
> 
> When suspending domU I get the following issue:
> 
>     Freezing user space processes
>     Freezing user space processes failed after 20.004 seconds (1 tasks refusing to freeze, wq_busy=0):
>     task:xl              state:D stack:0     pid:466   tgid:466   ppid:1      task_flags:0x400040 flags:0x00004006
>     Call Trace:
>      <TASK>
>      __schedule+0x2f3/0x780
>      schedule+0x27/0x80
>      schedule_preempt_disabled+0x15/0x30
>      __mutex_lock.constprop.0+0x49f/0x880
>      unregister_xenbus_watch+0x216/0x230
>      xenbus_write_watch+0xb9/0x220
>      xenbus_file_write+0x131/0x1b0
>      vfs_writev+0x26c/0x3d0
>      ? do_writev+0xeb/0x110
>      do_writev+0xeb/0x110
>      do_syscall_64+0x84/0x2c0
>      ? do_syscall_64+0x200/0x2c0
>      ? generic_handle_irq+0x3f/0x60
>      ? syscall_exit_work+0x108/0x140
>      ? do_syscall_64+0x200/0x2c0
>      ? __irq_exit_rcu+0x4c/0xe0
>      entry_SYSCALL_64_after_hwframe+0x76/0x7e
>     RIP: 0033:0x79b618138642
>     RSP: 002b:00007fff9a192fc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000014
>     RAX: ffffffffffffffda RBX: 00000000024fd490 RCX: 000079b618138642
>     RDX: 0000000000000003 RSI: 00007fff9a193120 RDI: 0000000000000014
>     RBP: 00007fff9a193000 R08: 0000000000000000 R09: 0000000000000000
>     R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000014
>     R13: 00007fff9a193120 R14: 0000000000000003 R15: 0000000000000000
>      </TASK>
>     OOM killer enabled.
>     Restarting tasks: Starting
>     Restarting tasks: Done
>     xen:manage: do_suspend: freeze processes failed -16
> 
> The process in question is `xl devd` daemon. It's a domU serving a
> xenvif backend.
> 
> I noticed it on 6.16.1, but looking at earlier test logs I see it with
> 6.16-rc6 already (but interestingly, not 6.16-rc2 yet? feels weird given
> seemingly no relevant changes between rc2 and rc6).

I forgot to include link for (a little) more details:
https://github.com/QubesOS/qubes-linux-kernel/pull/1157

Especially, there is another call trace with panic_on_warn enabled -
slightly different, but looks related.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: domU suspend issue - freeze processes failed - Linux 6.16

[Jürgen Groß]

[-- Attachment #1.1.1: Type: text/plain, Size: 2741 bytes --]

On 22.08.25 16:42, Marek Marczykowski-Górecki wrote:
> On Fri, Aug 22, 2025 at 04:39:33PM +0200, Marek Marczykowski-Górecki wrote:
>> Hi,
>>
>> When suspending domU I get the following issue:
>>
>>      Freezing user space processes
>>      Freezing user space processes failed after 20.004 seconds (1 tasks refusing to freeze, wq_busy=0):
>>      task:xl              state:D stack:0     pid:466   tgid:466   ppid:1      task_flags:0x400040 flags:0x00004006
>>      Call Trace:
>>       <TASK>
>>       __schedule+0x2f3/0x780
>>       schedule+0x27/0x80
>>       schedule_preempt_disabled+0x15/0x30
>>       __mutex_lock.constprop.0+0x49f/0x880
>>       unregister_xenbus_watch+0x216/0x230
>>       xenbus_write_watch+0xb9/0x220
>>       xenbus_file_write+0x131/0x1b0
>>       vfs_writev+0x26c/0x3d0
>>       ? do_writev+0xeb/0x110
>>       do_writev+0xeb/0x110
>>       do_syscall_64+0x84/0x2c0
>>       ? do_syscall_64+0x200/0x2c0
>>       ? generic_handle_irq+0x3f/0x60
>>       ? syscall_exit_work+0x108/0x140
>>       ? do_syscall_64+0x200/0x2c0
>>       ? __irq_exit_rcu+0x4c/0xe0
>>       entry_SYSCALL_64_after_hwframe+0x76/0x7e
>>      RIP: 0033:0x79b618138642
>>      RSP: 002b:00007fff9a192fc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000014
>>      RAX: ffffffffffffffda RBX: 00000000024fd490 RCX: 000079b618138642
>>      RDX: 0000000000000003 RSI: 00007fff9a193120 RDI: 0000000000000014
>>      RBP: 00007fff9a193000 R08: 0000000000000000 R09: 0000000000000000
>>      R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000014
>>      R13: 00007fff9a193120 R14: 0000000000000003 R15: 0000000000000000
>>       </TASK>
>>      OOM killer enabled.
>>      Restarting tasks: Starting
>>      Restarting tasks: Done
>>      xen:manage: do_suspend: freeze processes failed -16
>>
>> The process in question is `xl devd` daemon. It's a domU serving a
>> xenvif backend.
>>
>> I noticed it on 6.16.1, but looking at earlier test logs I see it with
>> 6.16-rc6 already (but interestingly, not 6.16-rc2 yet? feels weird given
>> seemingly no relevant changes between rc2 and rc6).
> 
> I forgot to include link for (a little) more details:
> https://github.com/QubesOS/qubes-linux-kernel/pull/1157
> 
> Especially, there is another call trace with panic_on_warn enabled -
> slightly different, but looks related.
> 

I'm pretty sure the PV variant for suspending is just wrong: it is calling
dpm_suspend_start() from do_suspend() without taking the required
system_transition_mutex, resulting in the WARN() in pm_restrict_gfp_mask().

It might be as easy as just adding the mutex() call to do_suspend(), but I'm
really not sure that will be a proper fix.


Juergen

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3743 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: domU suspend issue - freeze processes failed - Linux 6.16

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 3143 bytes --]

On Fri, Aug 22, 2025 at 05:27:20PM +0200, Jürgen Groß wrote:
> On 22.08.25 16:42, Marek Marczykowski-Górecki wrote:
> > On Fri, Aug 22, 2025 at 04:39:33PM +0200, Marek Marczykowski-Górecki wrote:
> > > Hi,
> > > 
> > > When suspending domU I get the following issue:
> > > 
> > >      Freezing user space processes
> > >      Freezing user space processes failed after 20.004 seconds (1 tasks refusing to freeze, wq_busy=0):
> > >      task:xl              state:D stack:0     pid:466   tgid:466   ppid:1      task_flags:0x400040 flags:0x00004006
> > >      Call Trace:
> > >       <TASK>
> > >       __schedule+0x2f3/0x780
> > >       schedule+0x27/0x80
> > >       schedule_preempt_disabled+0x15/0x30
> > >       __mutex_lock.constprop.0+0x49f/0x880
> > >       unregister_xenbus_watch+0x216/0x230
> > >       xenbus_write_watch+0xb9/0x220
> > >       xenbus_file_write+0x131/0x1b0
> > >       vfs_writev+0x26c/0x3d0
> > >       ? do_writev+0xeb/0x110
> > >       do_writev+0xeb/0x110
> > >       do_syscall_64+0x84/0x2c0
> > >       ? do_syscall_64+0x200/0x2c0
> > >       ? generic_handle_irq+0x3f/0x60
> > >       ? syscall_exit_work+0x108/0x140
> > >       ? do_syscall_64+0x200/0x2c0
> > >       ? __irq_exit_rcu+0x4c/0xe0
> > >       entry_SYSCALL_64_after_hwframe+0x76/0x7e
> > >      RIP: 0033:0x79b618138642
> > >      RSP: 002b:00007fff9a192fc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000014
> > >      RAX: ffffffffffffffda RBX: 00000000024fd490 RCX: 000079b618138642
> > >      RDX: 0000000000000003 RSI: 00007fff9a193120 RDI: 0000000000000014
> > >      RBP: 00007fff9a193000 R08: 0000000000000000 R09: 0000000000000000
> > >      R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000014
> > >      R13: 00007fff9a193120 R14: 0000000000000003 R15: 0000000000000000
> > >       </TASK>
> > >      OOM killer enabled.
> > >      Restarting tasks: Starting
> > >      Restarting tasks: Done
> > >      xen:manage: do_suspend: freeze processes failed -16
> > > 
> > > The process in question is `xl devd` daemon. It's a domU serving a
> > > xenvif backend.
> > > 
> > > I noticed it on 6.16.1, but looking at earlier test logs I see it with
> > > 6.16-rc6 already (but interestingly, not 6.16-rc2 yet? feels weird given
> > > seemingly no relevant changes between rc2 and rc6).
> > 
> > I forgot to include link for (a little) more details:
> > https://github.com/QubesOS/qubes-linux-kernel/pull/1157
> > 
> > Especially, there is another call trace with panic_on_warn enabled -
> > slightly different, but looks related.
> > 
> 
> I'm pretty sure the PV variant for suspending is just wrong: it is calling
> dpm_suspend_start() from do_suspend() without taking the required
> system_transition_mutex, resulting in the WARN() in pm_restrict_gfp_mask().
> 
> It might be as easy as just adding the mutex() call to do_suspend(), but I'm
> really not sure that will be a proper fix.

Hm, this might explain the second call trace, but not the freeze failure
quoted here above, I think?


-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: domU suspend issue - freeze processes failed - Linux 6.16

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 4033 bytes --]

On Fri, Aug 22, 2025 at 08:42:30PM +0200, Marek Marczykowski-Górecki wrote:
> On Fri, Aug 22, 2025 at 05:27:20PM +0200, Jürgen Groß wrote:
> > On 22.08.25 16:42, Marek Marczykowski-Górecki wrote:
> > > On Fri, Aug 22, 2025 at 04:39:33PM +0200, Marek Marczykowski-Górecki wrote:
> > > > Hi,
> > > > 
> > > > When suspending domU I get the following issue:
> > > > 
> > > >      Freezing user space processes
> > > >      Freezing user space processes failed after 20.004 seconds (1 tasks refusing to freeze, wq_busy=0):
> > > >      task:xl              state:D stack:0     pid:466   tgid:466   ppid:1      task_flags:0x400040 flags:0x00004006
> > > >      Call Trace:
> > > >       <TASK>
> > > >       __schedule+0x2f3/0x780
> > > >       schedule+0x27/0x80
> > > >       schedule_preempt_disabled+0x15/0x30
> > > >       __mutex_lock.constprop.0+0x49f/0x880
> > > >       unregister_xenbus_watch+0x216/0x230
> > > >       xenbus_write_watch+0xb9/0x220
> > > >       xenbus_file_write+0x131/0x1b0
> > > >       vfs_writev+0x26c/0x3d0
> > > >       ? do_writev+0xeb/0x110
> > > >       do_writev+0xeb/0x110
> > > >       do_syscall_64+0x84/0x2c0
> > > >       ? do_syscall_64+0x200/0x2c0
> > > >       ? generic_handle_irq+0x3f/0x60
> > > >       ? syscall_exit_work+0x108/0x140
> > > >       ? do_syscall_64+0x200/0x2c0
> > > >       ? __irq_exit_rcu+0x4c/0xe0
> > > >       entry_SYSCALL_64_after_hwframe+0x76/0x7e
> > > >      RIP: 0033:0x79b618138642
> > > >      RSP: 002b:00007fff9a192fc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000014
> > > >      RAX: ffffffffffffffda RBX: 00000000024fd490 RCX: 000079b618138642
> > > >      RDX: 0000000000000003 RSI: 00007fff9a193120 RDI: 0000000000000014
> > > >      RBP: 00007fff9a193000 R08: 0000000000000000 R09: 0000000000000000
> > > >      R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000014
> > > >      R13: 00007fff9a193120 R14: 0000000000000003 R15: 0000000000000000
> > > >       </TASK>
> > > >      OOM killer enabled.
> > > >      Restarting tasks: Starting
> > > >      Restarting tasks: Done
> > > >      xen:manage: do_suspend: freeze processes failed -16
> > > > 
> > > > The process in question is `xl devd` daemon. It's a domU serving a
> > > > xenvif backend.
> > > > 
> > > > I noticed it on 6.16.1, but looking at earlier test logs I see it with
> > > > 6.16-rc6 already (but interestingly, not 6.16-rc2 yet? feels weird given
> > > > seemingly no relevant changes between rc2 and rc6).
> > > 
> > > I forgot to include link for (a little) more details:
> > > https://github.com/QubesOS/qubes-linux-kernel/pull/1157
> > > 
> > > Especially, there is another call trace with panic_on_warn enabled -
> > > slightly different, but looks related.
> > > 
> > 
> > I'm pretty sure the PV variant for suspending is just wrong: it is calling
> > dpm_suspend_start() from do_suspend() without taking the required
> > system_transition_mutex, resulting in the WARN() in pm_restrict_gfp_mask().
> > 
> > It might be as easy as just adding the mutex() call to do_suspend(), but I'm
> > really not sure that will be a proper fix.
> 
> Hm, this might explain the second call trace, but not the freeze failure
> quoted here above, I think?

While the patch I sent appears to fix this particular issue, it made me
wonder: is there any fundamental reason why do_suspend() is not using
pm_suspend() and register Xen-specific actions via platform_suspend_ops
(and maybe syscore_ops)? From a brief look at the code, it should
theoretically be possible, and should avoid issues like this.

I tried to do a quick&dirty attempt at that[1], and it failed (panic). I
surely made several mistakes there (and also left a ton of todo
comments). But before spending any more time at that, I'd like to ask
if this is a viable option at all.

[1] https://github.com/marmarek/linux/commit/47cfdb991c85566c9c333570511e67bf477a5da6
-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: domU suspend issue - freeze processes failed - Linux 6.16

[Grygorii Strashko]

On 22.09.25 13:09, Marek Marczykowski-Górecki wrote:
> On Fri, Aug 22, 2025 at 08:42:30PM +0200, Marek Marczykowski-Górecki wrote:
>> On Fri, Aug 22, 2025 at 05:27:20PM +0200, Jürgen Groß wrote:
>>> On 22.08.25 16:42, Marek Marczykowski-Górecki wrote:
>>>> On Fri, Aug 22, 2025 at 04:39:33PM +0200, Marek Marczykowski-Górecki wrote:
>>>>> Hi,
>>>>>
>>>>> When suspending domU I get the following issue:
>>>>>
>>>>>       Freezing user space processes
>>>>>       Freezing user space processes failed after 20.004 seconds (1 tasks refusing to freeze, wq_busy=0):
>>>>>       task:xl              state:D stack:0     pid:466   tgid:466   ppid:1      task_flags:0x400040 flags:0x00004006
>>>>>       Call Trace:
>>>>>        <TASK>
>>>>>        __schedule+0x2f3/0x780
>>>>>        schedule+0x27/0x80
>>>>>        schedule_preempt_disabled+0x15/0x30
>>>>>        __mutex_lock.constprop.0+0x49f/0x880
>>>>>        unregister_xenbus_watch+0x216/0x230
>>>>>        xenbus_write_watch+0xb9/0x220
>>>>>        xenbus_file_write+0x131/0x1b0
>>>>>        vfs_writev+0x26c/0x3d0
>>>>>        ? do_writev+0xeb/0x110
>>>>>        do_writev+0xeb/0x110
>>>>>        do_syscall_64+0x84/0x2c0
>>>>>        ? do_syscall_64+0x200/0x2c0
>>>>>        ? generic_handle_irq+0x3f/0x60
>>>>>        ? syscall_exit_work+0x108/0x140
>>>>>        ? do_syscall_64+0x200/0x2c0
>>>>>        ? __irq_exit_rcu+0x4c/0xe0
>>>>>        entry_SYSCALL_64_after_hwframe+0x76/0x7e
>>>>>       RIP: 0033:0x79b618138642
>>>>>       RSP: 002b:00007fff9a192fc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000014
>>>>>       RAX: ffffffffffffffda RBX: 00000000024fd490 RCX: 000079b618138642
>>>>>       RDX: 0000000000000003 RSI: 00007fff9a193120 RDI: 0000000000000014
>>>>>       RBP: 00007fff9a193000 R08: 0000000000000000 R09: 0000000000000000
>>>>>       R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000014
>>>>>       R13: 00007fff9a193120 R14: 0000000000000003 R15: 0000000000000000
>>>>>        </TASK>
>>>>>       OOM killer enabled.
>>>>>       Restarting tasks: Starting
>>>>>       Restarting tasks: Done
>>>>>       xen:manage: do_suspend: freeze processes failed -16
>>>>>
>>>>> The process in question is `xl devd` daemon. It's a domU serving a
>>>>> xenvif backend.
>>>>>
>>>>> I noticed it on 6.16.1, but looking at earlier test logs I see it with
>>>>> 6.16-rc6 already (but interestingly, not 6.16-rc2 yet? feels weird given
>>>>> seemingly no relevant changes between rc2 and rc6).
>>>>
>>>> I forgot to include link for (a little) more details:
>>>> https://github.com/QubesOS/qubes-linux-kernel/pull/1157
>>>>
>>>> Especially, there is another call trace with panic_on_warn enabled -
>>>> slightly different, but looks related.
>>>>
>>>
>>> I'm pretty sure the PV variant for suspending is just wrong: it is calling
>>> dpm_suspend_start() from do_suspend() without taking the required
>>> system_transition_mutex, resulting in the WARN() in pm_restrict_gfp_mask().
>>>
>>> It might be as easy as just adding the mutex() call to do_suspend(), but I'm
>>> really not sure that will be a proper fix.
>>
>> Hm, this might explain the second call trace, but not the freeze failure
>> quoted here above, I think?
> 
> While the patch I sent appears to fix this particular issue, it made me
> wonder: is there any fundamental reason why do_suspend() is not using
> pm_suspend() and register Xen-specific actions via platform_suspend_ops
> (and maybe syscore_ops)? From a brief look at the code, it should
> theoretically be possible, and should avoid issues like this.
> 
> I tried to do a quick&dirty attempt at that[1], and it failed (panic). I
> surely made several mistakes there (and also left a ton of todo
> comments). But before spending any more time at that, I'd like to ask
> if this is a viable option at all.

I think it might, but be careful with this, because there are two "System Low power" paths in Linux
1) Suspend2RAM and Co
2) Hybernation

While "Suspend2RAM and Co" path is relatively straight forward and expected to be always
started through pm_suspend(). In general, it's expected to happen
  - from sysfs (User space)
  - from autosuspend (wakelocks).

the "hibernation" path is more complicated:(
- Genuine Linux hybernation hibernate()/hibernate_quiet_exec()

I'm not sure what path Xen originally implemented :( It seems like "suspend2RAM",
but, at the same time "hybernation" specific staff is used, like PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE.
As result, Linux suspend/hybernation code moves forward while Xen stays behind and unsync.

So it sounds reasonable to avoid custom implementation, but may be not easy :(

Suspending Xen features can be split between suspend stages, but
not sure if platform_suspend_ops can be used.

Generic suspend stages list
- freeze
- prepare
- suspend
- suspend_late
- suspend_noirq (SPIs disabled, except wakeups)
   [most of Xen specific staff has to be suspended at this point]
- disable_secondary_cpus
- arch disable IRQ (from this point no IRQs allowed, no timers, no scheduling)
- syscore_suspend
   [rest here]
- platform->enter() (suspended)

You can't just overwrite platform_suspend_ops, because ARM64 is expected to enter
suspend through PSCI FW interface:
drivers/firmware/psci/psci.c
  static const struct platform_suspend_ops psci_suspend_ops = {

As an option, some Xen components could be converted to use syscore_ops (but not xenstore),
and some might need to use DD(dev_pm_ops).

> 
> [1] https://github.com/marmarek/linux/commit/47cfdb991c85566c9c333570511e67bf477a5da6

-- 
Best regards,
-grygorii

Re: domU suspend issue - freeze processes failed - Linux 6.16

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 7241 bytes --]

On Wed, Sep 24, 2025 at 01:17:15PM +0300, Grygorii Strashko wrote:
> 
> 
> On 22.09.25 13:09, Marek Marczykowski-Górecki wrote:
> > On Fri, Aug 22, 2025 at 08:42:30PM +0200, Marek Marczykowski-Górecki wrote:
> > > On Fri, Aug 22, 2025 at 05:27:20PM +0200, Jürgen Groß wrote:
> > > > On 22.08.25 16:42, Marek Marczykowski-Górecki wrote:
> > > > > On Fri, Aug 22, 2025 at 04:39:33PM +0200, Marek Marczykowski-Górecki wrote:
> > > > > > Hi,
> > > > > > 
> > > > > > When suspending domU I get the following issue:
> > > > > > 
> > > > > >       Freezing user space processes
> > > > > >       Freezing user space processes failed after 20.004 seconds (1 tasks refusing to freeze, wq_busy=0):
> > > > > >       task:xl              state:D stack:0     pid:466   tgid:466   ppid:1      task_flags:0x400040 flags:0x00004006
> > > > > >       Call Trace:
> > > > > >        <TASK>
> > > > > >        __schedule+0x2f3/0x780
> > > > > >        schedule+0x27/0x80
> > > > > >        schedule_preempt_disabled+0x15/0x30
> > > > > >        __mutex_lock.constprop.0+0x49f/0x880
> > > > > >        unregister_xenbus_watch+0x216/0x230
> > > > > >        xenbus_write_watch+0xb9/0x220
> > > > > >        xenbus_file_write+0x131/0x1b0
> > > > > >        vfs_writev+0x26c/0x3d0
> > > > > >        ? do_writev+0xeb/0x110
> > > > > >        do_writev+0xeb/0x110
> > > > > >        do_syscall_64+0x84/0x2c0
> > > > > >        ? do_syscall_64+0x200/0x2c0
> > > > > >        ? generic_handle_irq+0x3f/0x60
> > > > > >        ? syscall_exit_work+0x108/0x140
> > > > > >        ? do_syscall_64+0x200/0x2c0
> > > > > >        ? __irq_exit_rcu+0x4c/0xe0
> > > > > >        entry_SYSCALL_64_after_hwframe+0x76/0x7e
> > > > > >       RIP: 0033:0x79b618138642
> > > > > >       RSP: 002b:00007fff9a192fc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000014
> > > > > >       RAX: ffffffffffffffda RBX: 00000000024fd490 RCX: 000079b618138642
> > > > > >       RDX: 0000000000000003 RSI: 00007fff9a193120 RDI: 0000000000000014
> > > > > >       RBP: 00007fff9a193000 R08: 0000000000000000 R09: 0000000000000000
> > > > > >       R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000014
> > > > > >       R13: 00007fff9a193120 R14: 0000000000000003 R15: 0000000000000000
> > > > > >        </TASK>
> > > > > >       OOM killer enabled.
> > > > > >       Restarting tasks: Starting
> > > > > >       Restarting tasks: Done
> > > > > >       xen:manage: do_suspend: freeze processes failed -16
> > > > > > 
> > > > > > The process in question is `xl devd` daemon. It's a domU serving a
> > > > > > xenvif backend.
> > > > > > 
> > > > > > I noticed it on 6.16.1, but looking at earlier test logs I see it with
> > > > > > 6.16-rc6 already (but interestingly, not 6.16-rc2 yet? feels weird given
> > > > > > seemingly no relevant changes between rc2 and rc6).
> > > > > 
> > > > > I forgot to include link for (a little) more details:
> > > > > https://github.com/QubesOS/qubes-linux-kernel/pull/1157
> > > > > 
> > > > > Especially, there is another call trace with panic_on_warn enabled -
> > > > > slightly different, but looks related.
> > > > > 
> > > > 
> > > > I'm pretty sure the PV variant for suspending is just wrong: it is calling
> > > > dpm_suspend_start() from do_suspend() without taking the required
> > > > system_transition_mutex, resulting in the WARN() in pm_restrict_gfp_mask().
> > > > 
> > > > It might be as easy as just adding the mutex() call to do_suspend(), but I'm
> > > > really not sure that will be a proper fix.
> > > 
> > > Hm, this might explain the second call trace, but not the freeze failure
> > > quoted here above, I think?
> > 
> > While the patch I sent appears to fix this particular issue, it made me
> > wonder: is there any fundamental reason why do_suspend() is not using
> > pm_suspend() and register Xen-specific actions via platform_suspend_ops
> > (and maybe syscore_ops)? From a brief look at the code, it should
> > theoretically be possible, and should avoid issues like this.
> > 
> > I tried to do a quick&dirty attempt at that[1], and it failed (panic). I
> > surely made several mistakes there (and also left a ton of todo
> > comments). But before spending any more time at that, I'd like to ask
> > if this is a viable option at all.
> 
> I think it might, but be careful with this, because there are two "System Low power" paths in Linux
> 1) Suspend2RAM and Co
> 2) Hybernation
> 
> While "Suspend2RAM and Co" path is relatively straight forward and expected to be always
> started through pm_suspend(). In general, it's expected to happen
>  - from sysfs (User space)
>  - from autosuspend (wakelocks).
> 
> the "hibernation" path is more complicated:(
> - Genuine Linux hybernation hibernate()/hibernate_quiet_exec()

IIUC hibernation is very different as it puts Linux in charge of dumping
all the state to the disk. In case of Xen, the primary use case for
suspend is preparing VM for Xen toolstack serializing its state to disk
(or migrating to another host). 
Additionally, VM suspend may be used as preparation for host suspend
(this is what I actually do here). This is especially relevant if the VM
has some PCI passthrough - to properly suspend (and resume) devices
across host suspend.

> I'm not sure what path Xen originally implemented :( It seems like "suspend2RAM",
> but, at the same time "hybernation" specific staff is used, like PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE.
> As result, Linux suspend/hybernation code moves forward while Xen stays behind and unsync.

Yeah, I think it's supposed to be suspend2RAM. TBH the
PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE confuses me too and Qubes OS has a
patch[2] to switch it to PMSG_SUSPEND/PMSG_RESUME.

> So it sounds reasonable to avoid custom implementation, but may be not easy :(
> 
> Suspending Xen features can be split between suspend stages, but
> not sure if platform_suspend_ops can be used.
> 
> Generic suspend stages list
> - freeze
> - prepare
> - suspend
> - suspend_late
> - suspend_noirq (SPIs disabled, except wakeups)
>   [most of Xen specific staff has to be suspended at this point]
> - disable_secondary_cpus
> - arch disable IRQ (from this point no IRQs allowed, no timers, no scheduling)
> - syscore_suspend
>   [rest here]
> - platform->enter() (suspended)
> 
> You can't just overwrite platform_suspend_ops, because ARM64 is expected to enter
> suspend through PSCI FW interface:
> drivers/firmware/psci/psci.c
>  static const struct platform_suspend_ops psci_suspend_ops = {

Does this apply to a VM on ARM64 too? At least on x86, the VM is
supposed to make a hypercall to tell Xen it suspended (the hypercall
will return only on resume).

> As an option, some Xen components could be converted to use syscore_ops (but not xenstore),
> and some might need to use DD(dev_pm_ops).
> 
> > 
> > [1] https://github.com/marmarek/linux/commit/47cfdb991c85566c9c333570511e67bf477a5da6
> 
> -- 
> Best regards,
> -grygorii
> 

[2] https://github.com/QubesOS/qubes-linux-kernel/blob/main/xen-pm-use-suspend.patch

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: domU suspend issue - freeze processes failed - Linux 6.16

[Yann Sionneau]

On 9/24/25 15:30, Marek Marczykowski-Górecki wrote:
> On Wed, Sep 24, 2025 at 01:17:15PM +0300, Grygorii Strashko wrote:
>>
>>
>> On 22.09.25 13:09, Marek Marczykowski-Górecki wrote:
>>> On Fri, Aug 22, 2025 at 08:42:30PM +0200, Marek Marczykowski-Górecki wrote:
>>>> On Fri, Aug 22, 2025 at 05:27:20PM +0200, Jürgen Groß wrote:
>>>>> On 22.08.25 16:42, Marek Marczykowski-Górecki wrote:
>>>>>> On Fri, Aug 22, 2025 at 04:39:33PM +0200, Marek Marczykowski-Górecki wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> When suspending domU I get the following issue:
>>>>>>>
>>>>>>>        Freezing user space processes
>>>>>>>        Freezing user space processes failed after 20.004 seconds (1 tasks refusing to freeze, wq_busy=0):
>>>>>>>        task:xl              state:D stack:0     pid:466   tgid:466   ppid:1      task_flags:0x400040 flags:0x00004006
>>>>>>>        Call Trace:
>>>>>>>         <TASK>
>>>>>>>         __schedule+0x2f3/0x780
>>>>>>>         schedule+0x27/0x80
>>>>>>>         schedule_preempt_disabled+0x15/0x30
>>>>>>>         __mutex_lock.constprop.0+0x49f/0x880
>>>>>>>         unregister_xenbus_watch+0x216/0x230
>>>>>>>         xenbus_write_watch+0xb9/0x220
>>>>>>>         xenbus_file_write+0x131/0x1b0
>>>>>>>         vfs_writev+0x26c/0x3d0
>>>>>>>         ? do_writev+0xeb/0x110
>>>>>>>         do_writev+0xeb/0x110
>>>>>>>         do_syscall_64+0x84/0x2c0
>>>>>>>         ? do_syscall_64+0x200/0x2c0
>>>>>>>         ? generic_handle_irq+0x3f/0x60
>>>>>>>         ? syscall_exit_work+0x108/0x140
>>>>>>>         ? do_syscall_64+0x200/0x2c0
>>>>>>>         ? __irq_exit_rcu+0x4c/0xe0
>>>>>>>         entry_SYSCALL_64_after_hwframe+0x76/0x7e
>>>>>>>        RIP: 0033:0x79b618138642
>>>>>>>        RSP: 002b:00007fff9a192fc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000014
>>>>>>>        RAX: ffffffffffffffda RBX: 00000000024fd490 RCX: 000079b618138642
>>>>>>>        RDX: 0000000000000003 RSI: 00007fff9a193120 RDI: 0000000000000014
>>>>>>>        RBP: 00007fff9a193000 R08: 0000000000000000 R09: 0000000000000000
>>>>>>>        R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000014
>>>>>>>        R13: 00007fff9a193120 R14: 0000000000000003 R15: 0000000000000000
>>>>>>>         </TASK>
>>>>>>>        OOM killer enabled.
>>>>>>>        Restarting tasks: Starting
>>>>>>>        Restarting tasks: Done
>>>>>>>        xen:manage: do_suspend: freeze processes failed -16
>>>>>>>
>>>>>>> The process in question is `xl devd` daemon. It's a domU serving a
>>>>>>> xenvif backend.
>>>>>>>
>>>>>>> I noticed it on 6.16.1, but looking at earlier test logs I see it with
>>>>>>> 6.16-rc6 already (but interestingly, not 6.16-rc2 yet? feels weird given
>>>>>>> seemingly no relevant changes between rc2 and rc6).
>>>>>>
>>>>>> I forgot to include link for (a little) more details:
>>>>>> https://github.com/QubesOS/qubes-linux-kernel/pull/1157
>>>>>>
>>>>>> Especially, there is another call trace with panic_on_warn enabled -
>>>>>> slightly different, but looks related.
>>>>>>
>>>>>
>>>>> I'm pretty sure the PV variant for suspending is just wrong: it is calling
>>>>> dpm_suspend_start() from do_suspend() without taking the required
>>>>> system_transition_mutex, resulting in the WARN() in pm_restrict_gfp_mask().
>>>>>
>>>>> It might be as easy as just adding the mutex() call to do_suspend(), but I'm
>>>>> really not sure that will be a proper fix.
>>>>
>>>> Hm, this might explain the second call trace, but not the freeze failure
>>>> quoted here above, I think?
>>>
>>> While the patch I sent appears to fix this particular issue, it made me
>>> wonder: is there any fundamental reason why do_suspend() is not using
>>> pm_suspend() and register Xen-specific actions via platform_suspend_ops
>>> (and maybe syscore_ops)? From a brief look at the code, it should
>>> theoretically be possible, and should avoid issues like this.
>>>
>>> I tried to do a quick&dirty attempt at that[1], and it failed (panic). I
>>> surely made several mistakes there (and also left a ton of todo
>>> comments). But before spending any more time at that, I'd like to ask
>>> if this is a viable option at all.
>>
>> I think it might, but be careful with this, because there are two "System Low power" paths in Linux
>> 1) Suspend2RAM and Co
>> 2) Hybernation
>>
>> While "Suspend2RAM and Co" path is relatively straight forward and expected to be always
>> started through pm_suspend(). In general, it's expected to happen
>>   - from sysfs (User space)
>>   - from autosuspend (wakelocks).
>>
>> the "hibernation" path is more complicated:(
>> - Genuine Linux hybernation hibernate()/hibernate_quiet_exec()
>
> IIUC hibernation is very different as it puts Linux in charge of dumping
> all the state to the disk. In case of Xen, the primary use case for
> suspend is preparing VM for Xen toolstack serializing its state to disk
> (or migrating to another host).
> Additionally, VM suspend may be used as preparation for host suspend
> (this is what I actually do here). This is especially relevant if the VM
> has some PCI passthrough - to properly suspend (and resume) devices
> across host suspend.
>
>> I'm not sure what path Xen originally implemented :( It seems like "suspend2RAM",
>> but, at the same time "hybernation" specific staff is used, like PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE.
>> As result, Linux suspend/hybernation code moves forward while Xen stays behind and unsync.
>
> Yeah, I think it's supposed to be suspend2RAM. TBH the
> PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE confuses me too and Qubes OS has a
> patch[2] to switch it to PMSG_SUSPEND/PMSG_RESUME.
>
>> So it sounds reasonable to avoid custom implementation, but may be not easy :(
>>
>> Suspending Xen features can be split between suspend stages, but
>> not sure if platform_suspend_ops can be used.
>>
>> Generic suspend stages list
>> - freeze
>> - prepare
>> - suspend
>> - suspend_late
>> - suspend_noirq (SPIs disabled, except wakeups)
>>    [most of Xen specific staff has to be suspended at this point]
>> - disable_secondary_cpus
>> - arch disable IRQ (from this point no IRQs allowed, no timers, no scheduling)
>> - syscore_suspend
>>    [rest here]
>> - platform->enter() (suspended)
>>
>> You can't just overwrite platform_suspend_ops, because ARM64 is expected to enter
>> suspend through PSCI FW interface:
>> drivers/firmware/psci/psci.c
>>   static const struct platform_suspend_ops psci_suspend_ops = {
>
> Does this apply to a VM on ARM64 too? At least on x86, the VM is
> supposed to make a hypercall to tell Xen it suspended (the hypercall
> will return only on resume).
>
>> As an option, some Xen components could be converted to use syscore_ops (but not xenstore),
>> and some might need to use DD(dev_pm_ops).
>>
>>>
>>> [1] https://github.com/marmarek/linux/commit/47cfdb991c85566c9c333570511e67bf477a5da6
>>
>> --
>> Best regards,
>> -grygorii
>>
>
> [2] https://github.com/QubesOS/qubes-linux-kernel/blob/main/xen-pm-use-suspend.patch
>

On my setup I get a weird behavior when trying to suspend (s2idle) a
Linux guest.
Doing echo freeze > /sys/power/state in the guest seems to "freeze" the
guest for good, I could not unfreeze it afterward.
VCPU goes to 100% according to XenOrchestra
xl list shows state "r" but xl console blocks forever
xl shutdown would block for some time and then print:
Shutting down domain 721
?ibxl: error: libxl_domain.c:848:pvcontrol_cb: guest didn't acknowledge
control request: -9
shutdown failed (rc=-9)

Do you think it's related to your current issue?

Regards,

--


--
Yann Sionneau | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: domU suspend issue - freeze processes failed - Linux 6.16

[Grygorii Strashko]

On 24.09.25 16:30, Marek Marczykowski-Górecki wrote:
> On Wed, Sep 24, 2025 at 01:17:15PM +0300, Grygorii Strashko wrote:
>>
>>
>> On 22.09.25 13:09, Marek Marczykowski-Górecki wrote:
>>> On Fri, Aug 22, 2025 at 08:42:30PM +0200, Marek Marczykowski-Górecki wrote:
>>>> On Fri, Aug 22, 2025 at 05:27:20PM +0200, Jürgen Groß wrote:
>>>>> On 22.08.25 16:42, Marek Marczykowski-Górecki wrote:
>>>>>> On Fri, Aug 22, 2025 at 04:39:33PM +0200, Marek Marczykowski-Górecki wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> When suspending domU I get the following issue:
>>>>>>>
>>>>>>>        Freezing user space processes
>>>>>>>        Freezing user space processes failed after 20.004 seconds (1 tasks refusing to freeze, wq_busy=0):
>>>>>>>        task:xl              state:D stack:0     pid:466   tgid:466   ppid:1      task_flags:0x400040 flags:0x00004006
>>>>>>>        Call Trace:
>>>>>>>         <TASK>
>>>>>>>         __schedule+0x2f3/0x780
>>>>>>>         schedule+0x27/0x80
>>>>>>>         schedule_preempt_disabled+0x15/0x30
>>>>>>>         __mutex_lock.constprop.0+0x49f/0x880
>>>>>>>         unregister_xenbus_watch+0x216/0x230
>>>>>>>         xenbus_write_watch+0xb9/0x220
>>>>>>>         xenbus_file_write+0x131/0x1b0
>>>>>>>         vfs_writev+0x26c/0x3d0
>>>>>>>         ? do_writev+0xeb/0x110
>>>>>>>         do_writev+0xeb/0x110
>>>>>>>         do_syscall_64+0x84/0x2c0
>>>>>>>         ? do_syscall_64+0x200/0x2c0
>>>>>>>         ? generic_handle_irq+0x3f/0x60
>>>>>>>         ? syscall_exit_work+0x108/0x140
>>>>>>>         ? do_syscall_64+0x200/0x2c0
>>>>>>>         ? __irq_exit_rcu+0x4c/0xe0
>>>>>>>         entry_SYSCALL_64_after_hwframe+0x76/0x7e
>>>>>>>        RIP: 0033:0x79b618138642
>>>>>>>        RSP: 002b:00007fff9a192fc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000014
>>>>>>>        RAX: ffffffffffffffda RBX: 00000000024fd490 RCX: 000079b618138642
>>>>>>>        RDX: 0000000000000003 RSI: 00007fff9a193120 RDI: 0000000000000014
>>>>>>>        RBP: 00007fff9a193000 R08: 0000000000000000 R09: 0000000000000000
>>>>>>>        R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000014
>>>>>>>        R13: 00007fff9a193120 R14: 0000000000000003 R15: 0000000000000000
>>>>>>>         </TASK>
>>>>>>>        OOM killer enabled.
>>>>>>>        Restarting tasks: Starting
>>>>>>>        Restarting tasks: Done
>>>>>>>        xen:manage: do_suspend: freeze processes failed -16
>>>>>>>
>>>>>>> The process in question is `xl devd` daemon. It's a domU serving a
>>>>>>> xenvif backend.
>>>>>>>
>>>>>>> I noticed it on 6.16.1, but looking at earlier test logs I see it with
>>>>>>> 6.16-rc6 already (but interestingly, not 6.16-rc2 yet? feels weird given
>>>>>>> seemingly no relevant changes between rc2 and rc6).
>>>>>>
>>>>>> I forgot to include link for (a little) more details:
>>>>>> https://github.com/QubesOS/qubes-linux-kernel/pull/1157
>>>>>>
>>>>>> Especially, there is another call trace with panic_on_warn enabled -
>>>>>> slightly different, but looks related.
>>>>>>
>>>>>
>>>>> I'm pretty sure the PV variant for suspending is just wrong: it is calling
>>>>> dpm_suspend_start() from do_suspend() without taking the required
>>>>> system_transition_mutex, resulting in the WARN() in pm_restrict_gfp_mask().
>>>>>
>>>>> It might be as easy as just adding the mutex() call to do_suspend(), but I'm
>>>>> really not sure that will be a proper fix.
>>>>
>>>> Hm, this might explain the second call trace, but not the freeze failure
>>>> quoted here above, I think?
>>>
>>> While the patch I sent appears to fix this particular issue, it made me
>>> wonder: is there any fundamental reason why do_suspend() is not using
>>> pm_suspend() and register Xen-specific actions via platform_suspend_ops
>>> (and maybe syscore_ops)? From a brief look at the code, it should
>>> theoretically be possible, and should avoid issues like this.
>>>
>>> I tried to do a quick&dirty attempt at that[1], and it failed (panic). I
>>> surely made several mistakes there (and also left a ton of todo
>>> comments). But before spending any more time at that, I'd like to ask
>>> if this is a viable option at all.
>>
>> I think it might, but be careful with this, because there are two "System Low power" paths in Linux
>> 1) Suspend2RAM and Co
>> 2) Hybernation
>>
>> While "Suspend2RAM and Co" path is relatively straight forward and expected to be always
>> started through pm_suspend(). In general, it's expected to happen
>>   - from sysfs (User space)
>>   - from autosuspend (wakelocks).
>>
>> the "hibernation" path is more complicated:(
>> - Genuine Linux hybernation hibernate()/hibernate_quiet_exec()
> 
> IIUC hibernation is very different as it puts Linux in charge of dumping
> all the state to the disk. In case of Xen, the primary use case for
> suspend is preparing VM for Xen toolstack serializing its state to disk
> (or migrating to another host).
> Additionally, VM suspend may be used as preparation for host suspend
> (this is what I actually do here). This is especially relevant if the VM
> has some PCI passthrough - to properly suspend (and resume) devices
> across host suspend.
> 
>> I'm not sure what path Xen originally implemented :( It seems like "suspend2RAM",
>> but, at the same time "hybernation" specific staff is used, like PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE.
>> As result, Linux suspend/hybernation code moves forward while Xen stays behind and unsync.
> 
> Yeah, I think it's supposed to be suspend2RAM. TBH the
> PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE confuses me too and Qubes OS has a
> patch[2] to switch it to PMSG_SUSPEND/PMSG_RESUME.
> 
>> So it sounds reasonable to avoid custom implementation, but may be not easy :(
>>
>> Suspending Xen features can be split between suspend stages, but
>> not sure if platform_suspend_ops can be used.
>>
>> Generic suspend stages list
>> - freeze
>> - prepare
>> - suspend
>> - suspend_late
>> - suspend_noirq (SPIs disabled, except wakeups)
>>    [most of Xen specific staff has to be suspended at this point]
>> - disable_secondary_cpus
>> - arch disable IRQ (from this point no IRQs allowed, no timers, no scheduling)
>> - syscore_suspend
>>    [rest here]
>> - platform->enter() (suspended)
>>
>> You can't just overwrite platform_suspend_ops, because ARM64 is expected to enter
>> suspend through PSCI FW interface:
>> drivers/firmware/psci/psci.c
>>   static const struct platform_suspend_ops psci_suspend_ops = {
> 
> Does this apply to a VM on ARM64 too? At least on x86, the VM is
> supposed to make a hypercall to tell Xen it suspended (the hypercall
> will return only on resume).

On ARM64 Guest expected to trigger PSCI(SYSTEM_SUSPEND) (which is HVC - trap, similar to hypercall on x86)
PSCI is Arm Power State Coordination Interface which defines unified FW interface for PM.
So if you overwrite platform_suspend_ops (which is Singleton) it will break ARM Guest suspend.

> 
>> As an option, some Xen components could be converted to use syscore_ops (but not xenstore),
>> and some might need to use DD(dev_pm_ops).
>>
>>>
>>> [1] https://github.com/marmarek/linux/commit/47cfdb991c85566c9c333570511e67bf477a5da6
>>
>> -- 
>> Best regards,
>> -grygorii
>>
> 
> [2] https://github.com/QubesOS/qubes-linux-kernel/blob/main/xen-pm-use-suspend.patch
> 

-- 
Best regards,
-grygorii

Re: domU suspend issue - freeze processes failed - Linux 6.16

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 8462 bytes --]

On Wed, Sep 24, 2025 at 02:28:27PM +0000, Yann Sionneau wrote:
> On 9/24/25 15:30, Marek Marczykowski-Górecki wrote:
> > On Wed, Sep 24, 2025 at 01:17:15PM +0300, Grygorii Strashko wrote:
> >>
> >>
> >> On 22.09.25 13:09, Marek Marczykowski-Górecki wrote:
> >>> On Fri, Aug 22, 2025 at 08:42:30PM +0200, Marek Marczykowski-Górecki wrote:
> >>>> On Fri, Aug 22, 2025 at 05:27:20PM +0200, Jürgen Groß wrote:
> >>>>> On 22.08.25 16:42, Marek Marczykowski-Górecki wrote:
> >>>>>> On Fri, Aug 22, 2025 at 04:39:33PM +0200, Marek Marczykowski-Górecki wrote:
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> When suspending domU I get the following issue:
> >>>>>>>
> >>>>>>>        Freezing user space processes
> >>>>>>>        Freezing user space processes failed after 20.004 seconds (1 tasks refusing to freeze, wq_busy=0):
> >>>>>>>        task:xl              state:D stack:0     pid:466   tgid:466   ppid:1      task_flags:0x400040 flags:0x00004006
> >>>>>>>        Call Trace:
> >>>>>>>         <TASK>
> >>>>>>>         __schedule+0x2f3/0x780
> >>>>>>>         schedule+0x27/0x80
> >>>>>>>         schedule_preempt_disabled+0x15/0x30
> >>>>>>>         __mutex_lock.constprop.0+0x49f/0x880
> >>>>>>>         unregister_xenbus_watch+0x216/0x230
> >>>>>>>         xenbus_write_watch+0xb9/0x220
> >>>>>>>         xenbus_file_write+0x131/0x1b0
> >>>>>>>         vfs_writev+0x26c/0x3d0
> >>>>>>>         ? do_writev+0xeb/0x110
> >>>>>>>         do_writev+0xeb/0x110
> >>>>>>>         do_syscall_64+0x84/0x2c0
> >>>>>>>         ? do_syscall_64+0x200/0x2c0
> >>>>>>>         ? generic_handle_irq+0x3f/0x60
> >>>>>>>         ? syscall_exit_work+0x108/0x140
> >>>>>>>         ? do_syscall_64+0x200/0x2c0
> >>>>>>>         ? __irq_exit_rcu+0x4c/0xe0
> >>>>>>>         entry_SYSCALL_64_after_hwframe+0x76/0x7e
> >>>>>>>        RIP: 0033:0x79b618138642
> >>>>>>>        RSP: 002b:00007fff9a192fc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000014
> >>>>>>>        RAX: ffffffffffffffda RBX: 00000000024fd490 RCX: 000079b618138642
> >>>>>>>        RDX: 0000000000000003 RSI: 00007fff9a193120 RDI: 0000000000000014
> >>>>>>>        RBP: 00007fff9a193000 R08: 0000000000000000 R09: 0000000000000000
> >>>>>>>        R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000014
> >>>>>>>        R13: 00007fff9a193120 R14: 0000000000000003 R15: 0000000000000000
> >>>>>>>         </TASK>
> >>>>>>>        OOM killer enabled.
> >>>>>>>        Restarting tasks: Starting
> >>>>>>>        Restarting tasks: Done
> >>>>>>>        xen:manage: do_suspend: freeze processes failed -16
> >>>>>>>
> >>>>>>> The process in question is `xl devd` daemon. It's a domU serving a
> >>>>>>> xenvif backend.
> >>>>>>>
> >>>>>>> I noticed it on 6.16.1, but looking at earlier test logs I see it with
> >>>>>>> 6.16-rc6 already (but interestingly, not 6.16-rc2 yet? feels weird given
> >>>>>>> seemingly no relevant changes between rc2 and rc6).
> >>>>>>
> >>>>>> I forgot to include link for (a little) more details:
> >>>>>> https://github.com/QubesOS/qubes-linux-kernel/pull/1157
> >>>>>>
> >>>>>> Especially, there is another call trace with panic_on_warn enabled -
> >>>>>> slightly different, but looks related.
> >>>>>>
> >>>>>
> >>>>> I'm pretty sure the PV variant for suspending is just wrong: it is calling
> >>>>> dpm_suspend_start() from do_suspend() without taking the required
> >>>>> system_transition_mutex, resulting in the WARN() in pm_restrict_gfp_mask().
> >>>>>
> >>>>> It might be as easy as just adding the mutex() call to do_suspend(), but I'm
> >>>>> really not sure that will be a proper fix.
> >>>>
> >>>> Hm, this might explain the second call trace, but not the freeze failure
> >>>> quoted here above, I think?
> >>>
> >>> While the patch I sent appears to fix this particular issue, it made me
> >>> wonder: is there any fundamental reason why do_suspend() is not using
> >>> pm_suspend() and register Xen-specific actions via platform_suspend_ops
> >>> (and maybe syscore_ops)? From a brief look at the code, it should
> >>> theoretically be possible, and should avoid issues like this.
> >>>
> >>> I tried to do a quick&dirty attempt at that[1], and it failed (panic). I
> >>> surely made several mistakes there (and also left a ton of todo
> >>> comments). But before spending any more time at that, I'd like to ask
> >>> if this is a viable option at all.
> >>
> >> I think it might, but be careful with this, because there are two "System Low power" paths in Linux
> >> 1) Suspend2RAM and Co
> >> 2) Hybernation
> >>
> >> While "Suspend2RAM and Co" path is relatively straight forward and expected to be always
> >> started through pm_suspend(). In general, it's expected to happen
> >>   - from sysfs (User space)
> >>   - from autosuspend (wakelocks).
> >>
> >> the "hibernation" path is more complicated:(
> >> - Genuine Linux hybernation hibernate()/hibernate_quiet_exec()
> >
> > IIUC hibernation is very different as it puts Linux in charge of dumping
> > all the state to the disk. In case of Xen, the primary use case for
> > suspend is preparing VM for Xen toolstack serializing its state to disk
> > (or migrating to another host).
> > Additionally, VM suspend may be used as preparation for host suspend
> > (this is what I actually do here). This is especially relevant if the VM
> > has some PCI passthrough - to properly suspend (and resume) devices
> > across host suspend.
> >
> >> I'm not sure what path Xen originally implemented :( It seems like "suspend2RAM",
> >> but, at the same time "hybernation" specific staff is used, like PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE.
> >> As result, Linux suspend/hybernation code moves forward while Xen stays behind and unsync.
> >
> > Yeah, I think it's supposed to be suspend2RAM. TBH the
> > PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE confuses me too and Qubes OS has a
> > patch[2] to switch it to PMSG_SUSPEND/PMSG_RESUME.
> >
> >> So it sounds reasonable to avoid custom implementation, but may be not easy :(
> >>
> >> Suspending Xen features can be split between suspend stages, but
> >> not sure if platform_suspend_ops can be used.
> >>
> >> Generic suspend stages list
> >> - freeze
> >> - prepare
> >> - suspend
> >> - suspend_late
> >> - suspend_noirq (SPIs disabled, except wakeups)
> >>    [most of Xen specific staff has to be suspended at this point]
> >> - disable_secondary_cpus
> >> - arch disable IRQ (from this point no IRQs allowed, no timers, no scheduling)
> >> - syscore_suspend
> >>    [rest here]
> >> - platform->enter() (suspended)
> >>
> >> You can't just overwrite platform_suspend_ops, because ARM64 is expected to enter
> >> suspend through PSCI FW interface:
> >> drivers/firmware/psci/psci.c
> >>   static const struct platform_suspend_ops psci_suspend_ops = {
> >
> > Does this apply to a VM on ARM64 too? At least on x86, the VM is
> > supposed to make a hypercall to tell Xen it suspended (the hypercall
> > will return only on resume).
> >
> >> As an option, some Xen components could be converted to use syscore_ops (but not xenstore),
> >> and some might need to use DD(dev_pm_ops).
> >>
> >>>
> >>> [1] https://github.com/marmarek/linux/commit/47cfdb991c85566c9c333570511e67bf477a5da6
> >>
> >> --
> >> Best regards,
> >> -grygorii
> >>
> >
> > [2] https://github.com/QubesOS/qubes-linux-kernel/blob/main/xen-pm-use-suspend.patch
> >
> 
> On my setup I get a weird behavior when trying to suspend (s2idle) a
> Linux guest.
> Doing echo freeze > /sys/power/state in the guest seems to "freeze" the
> guest for good, I could not unfreeze it afterward.
> VCPU goes to 100% according to XenOrchestra
> xl list shows state "r" but xl console blocks forever
> xl shutdown would block for some time and then print:
> Shutting down domain 721
> ?ibxl: error: libxl_domain.c:848:pvcontrol_cb: guest didn't acknowledge
> control request: -9
> shutdown failed (rc=-9)
> 
> Do you think it's related to your current issue?

Maybe? Is it a HVM guest or PV?

Regarding s2idle, we have another patch:
https://github.com/QubesOS/qubes-linux-kernel/blob/main/xen-events-Add-wakeup-support-to-xen-pirq.patch
but it fixes wakeup, not really going to sleep.

It was posted also to this ML, but still waiting for review (see link
in the patch header).

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: domU suspend issue - freeze processes failed - Linux 6.16

[Jason Andryuk]

On 2025-09-24 10:28, Yann Sionneau wrote:
> On 9/24/25 15:30, Marek Marczykowski-Górecki wrote:
>> On Wed, Sep 24, 2025 at 01:17:15PM +0300, Grygorii Strashko wrote:
>>>
>>>
>>> On 22.09.25 13:09, Marek Marczykowski-Górecki wrote:
>>>> On Fri, Aug 22, 2025 at 08:42:30PM +0200, Marek Marczykowski-Górecki wrote:
>>>>> On Fri, Aug 22, 2025 at 05:27:20PM +0200, Jürgen Groß wrote:
>>>>>> On 22.08.25 16:42, Marek Marczykowski-Górecki wrote:
>>>>>>> On Fri, Aug 22, 2025 at 04:39:33PM +0200, Marek Marczykowski-Górecki wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> When suspending domU I get the following issue:
>>>>>>>>
>>>>>>>>         Freezing user space processes
>>>>>>>>         Freezing user space processes failed after 20.004 seconds (1 tasks refusing to freeze, wq_busy=0):
>>>>>>>>         task:xl              state:D stack:0     pid:466   tgid:466   ppid:1      task_flags:0x400040 flags:0x00004006
>>>>>>>>         Call Trace:
>>>>>>>>          <TASK>
>>>>>>>>          __schedule+0x2f3/0x780
>>>>>>>>          schedule+0x27/0x80
>>>>>>>>          schedule_preempt_disabled+0x15/0x30
>>>>>>>>          __mutex_lock.constprop.0+0x49f/0x880
>>>>>>>>          unregister_xenbus_watch+0x216/0x230
>>>>>>>>          xenbus_write_watch+0xb9/0x220
>>>>>>>>          xenbus_file_write+0x131/0x1b0
>>>>>>>>          vfs_writev+0x26c/0x3d0
>>>>>>>>          ? do_writev+0xeb/0x110
>>>>>>>>          do_writev+0xeb/0x110
>>>>>>>>          do_syscall_64+0x84/0x2c0
>>>>>>>>          ? do_syscall_64+0x200/0x2c0
>>>>>>>>          ? generic_handle_irq+0x3f/0x60
>>>>>>>>          ? syscall_exit_work+0x108/0x140
>>>>>>>>          ? do_syscall_64+0x200/0x2c0
>>>>>>>>          ? __irq_exit_rcu+0x4c/0xe0
>>>>>>>>          entry_SYSCALL_64_after_hwframe+0x76/0x7e
>>>>>>>>         RIP: 0033:0x79b618138642
>>>>>>>>         RSP: 002b:00007fff9a192fc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000014
>>>>>>>>         RAX: ffffffffffffffda RBX: 00000000024fd490 RCX: 000079b618138642
>>>>>>>>         RDX: 0000000000000003 RSI: 00007fff9a193120 RDI: 0000000000000014
>>>>>>>>         RBP: 00007fff9a193000 R08: 0000000000000000 R09: 0000000000000000
>>>>>>>>         R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000014
>>>>>>>>         R13: 00007fff9a193120 R14: 0000000000000003 R15: 0000000000000000
>>>>>>>>          </TASK>
>>>>>>>>         OOM killer enabled.
>>>>>>>>         Restarting tasks: Starting
>>>>>>>>         Restarting tasks: Done
>>>>>>>>         xen:manage: do_suspend: freeze processes failed -16
>>>>>>>>
>>>>>>>> The process in question is `xl devd` daemon. It's a domU serving a
>>>>>>>> xenvif backend.
>>>>>>>>
>>>>>>>> I noticed it on 6.16.1, but looking at earlier test logs I see it with
>>>>>>>> 6.16-rc6 already (but interestingly, not 6.16-rc2 yet? feels weird given
>>>>>>>> seemingly no relevant changes between rc2 and rc6).
>>>>>>>
>>>>>>> I forgot to include link for (a little) more details:
>>>>>>> https://github.com/QubesOS/qubes-linux-kernel/pull/1157
>>>>>>>
>>>>>>> Especially, there is another call trace with panic_on_warn enabled -
>>>>>>> slightly different, but looks related.
>>>>>>>
>>>>>>
>>>>>> I'm pretty sure the PV variant for suspending is just wrong: it is calling
>>>>>> dpm_suspend_start() from do_suspend() without taking the required
>>>>>> system_transition_mutex, resulting in the WARN() in pm_restrict_gfp_mask().
>>>>>>
>>>>>> It might be as easy as just adding the mutex() call to do_suspend(), but I'm
>>>>>> really not sure that will be a proper fix.
>>>>>
>>>>> Hm, this might explain the second call trace, but not the freeze failure
>>>>> quoted here above, I think?
>>>>
>>>> While the patch I sent appears to fix this particular issue, it made me
>>>> wonder: is there any fundamental reason why do_suspend() is not using
>>>> pm_suspend() and register Xen-specific actions via platform_suspend_ops
>>>> (and maybe syscore_ops)? From a brief look at the code, it should
>>>> theoretically be possible, and should avoid issues like this.
>>>>
>>>> I tried to do a quick&dirty attempt at that[1], and it failed (panic). I
>>>> surely made several mistakes there (and also left a ton of todo
>>>> comments). But before spending any more time at that, I'd like to ask
>>>> if this is a viable option at all.
>>>
>>> I think it might, but be careful with this, because there are two "System Low power" paths in Linux
>>> 1) Suspend2RAM and Co
>>> 2) Hybernation
>>>
>>> While "Suspend2RAM and Co" path is relatively straight forward and expected to be always
>>> started through pm_suspend(). In general, it's expected to happen
>>>    - from sysfs (User space)
>>>    - from autosuspend (wakelocks).
>>>
>>> the "hibernation" path is more complicated:(
>>> - Genuine Linux hybernation hibernate()/hibernate_quiet_exec()
>>
>> IIUC hibernation is very different as it puts Linux in charge of dumping
>> all the state to the disk. In case of Xen, the primary use case for
>> suspend is preparing VM for Xen toolstack serializing its state to disk
>> (or migrating to another host).
>> Additionally, VM suspend may be used as preparation for host suspend
>> (this is what I actually do here). This is especially relevant if the VM
>> has some PCI passthrough - to properly suspend (and resume) devices
>> across host suspend.
>>
>>> I'm not sure what path Xen originally implemented :( It seems like "suspend2RAM",
>>> but, at the same time "hybernation" specific staff is used, like PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE.
>>> As result, Linux suspend/hybernation code moves forward while Xen stays behind and unsync.
>>
>> Yeah, I think it's supposed to be suspend2RAM. TBH the
>> PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE confuses me too and Qubes OS has a
>> patch[2] to switch it to PMSG_SUSPEND/PMSG_RESUME.
>>
>>> So it sounds reasonable to avoid custom implementation, but may be not easy :(
>>>
>>> Suspending Xen features can be split between suspend stages, but
>>> not sure if platform_suspend_ops can be used.
>>>
>>> Generic suspend stages list
>>> - freeze
>>> - prepare
>>> - suspend
>>> - suspend_late
>>> - suspend_noirq (SPIs disabled, except wakeups)
>>>     [most of Xen specific staff has to be suspended at this point]
>>> - disable_secondary_cpus
>>> - arch disable IRQ (from this point no IRQs allowed, no timers, no scheduling)
>>> - syscore_suspend
>>>     [rest here]
>>> - platform->enter() (suspended)
>>>
>>> You can't just overwrite platform_suspend_ops, because ARM64 is expected to enter
>>> suspend through PSCI FW interface:
>>> drivers/firmware/psci/psci.c
>>>    static const struct platform_suspend_ops psci_suspend_ops = {
>>
>> Does this apply to a VM on ARM64 too? At least on x86, the VM is
>> supposed to make a hypercall to tell Xen it suspended (the hypercall
>> will return only on resume).
>>
>>> As an option, some Xen components could be converted to use syscore_ops (but not xenstore),
>>> and some might need to use DD(dev_pm_ops).
>>>
>>>>
>>>> [1] https://github.com/marmarek/linux/commit/47cfdb991c85566c9c333570511e67bf477a5da6
>>>
>>> -- 
>>> Best regards,
>>> -grygorii
>>>
>>
>> [2] https://github.com/QubesOS/qubes-linux-kernel/blob/main/xen-pm-use-suspend.patch
>>
> 
> On my setup I get a weird behavior when trying to suspend (s2idle) a
> Linux guest.
> Doing echo freeze > /sys/power/state in the guest seems to "freeze" the
> guest for good, I could not unfreeze it afterward.
> VCPU goes to 100% according to XenOrchestra
> xl list shows state "r" but xl console blocks forever
> xl shutdown would block for some time and then print:
> Shutting down domain 721
> ?ibxl: error: libxl_domain.c:848:pvcontrol_cb: guest didn't acknowledge
> control request: -9
> shutdown failed (rc=-9)
> 
> Do you think it's related to your current issue?

idle=halt on the Linux command line addresses the 100% CPU usage.  Or 
alternatively C2 needs to be implemented for guest vcpus.  I forget 
preceisely, but I think the 100% CPU is because there are no C-states 
available and Linux/cpuidle won't use halt by default.

To wake up, you need a wake up source.  The ACPI buttons presses will do 
that:
xl trigger $dom power
xl trigger $dom sleep

However, I think without changes, domU s2idle/S3 will detach all its PV 
devices.  Naturally they don't get reconnected on resume.  You can hack 
around that to skip the detach.

Actually, maybe we just need:
--- i/drivers/xen/xenbus/xenbus_probe_frontend.c
+++ w/drivers/xen/xenbus/xenbus_probe_frontend.c
@@ -148,8 +148,6 @@ static void xenbus_frontend_dev_shutdown(struct 
device *_dev)
  }

  static const struct dev_pm_ops xenbus_pm_ops = {
-       .suspend        = xenbus_dev_suspend,
-       .resume         = xenbus_frontend_dev_resume,
         .freeze         = xenbus_dev_suspend,
         .thaw           = xenbus_dev_cancel,
-       .restore        = xenbus_dev_resume,
+       .restore        = xenbus_frontend_dev_resume,
  };

b3e96c0c7562 ("xen: use freeze/restore/thaw PM events for 
suspend/resume/chkpt") changed from PMSG_SUSPEND/PMSG_RESUME to 
PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE, but the suspend/resume callbacks 
remained.  But freeze and suspend being identical doesn't seem correct.

This would leave xl save/restore/migrate using the hibernate 
freeze/thaw/resume.  S3/s2idle would no touch the PV devices, so they 
would still be present on resume.  Maybe there are cases I am not 
thinking of though.

Regards,
Jason

Re: domU suspend issue - freeze processes failed - Linux 6.16

[Yann Sionneau]

On 10/6/25 16:28, Jason Andryuk wrote:
> On 2025-09-24 10:28, Yann Sionneau wrote:
>> On 9/24/25 15:30, Marek Marczykowski-Górecki wrote:
>>> On Wed, Sep 24, 2025 at 01:17:15PM +0300, Grygorii Strashko wrote:
>>>>
>>>>
>>>> On 22.09.25 13:09, Marek Marczykowski-Górecki wrote:
>>>>> On Fri, Aug 22, 2025 at 08:42:30PM +0200, Marek Marczykowski-
>>>>> Górecki wrote:
>>>>>> On Fri, Aug 22, 2025 at 05:27:20PM +0200, Jürgen Groß wrote:
>>>>>>> On 22.08.25 16:42, Marek Marczykowski-Górecki wrote:
>>>>>>>> On Fri, Aug 22, 2025 at 04:39:33PM +0200, Marek Marczykowski-
>>>>>>>> Górecki wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> When suspending domU I get the following issue:
>>>>>>>>>
>>>>>>>>>         Freezing user space processes
>>>>>>>>>         Freezing user space processes failed after 20.004
>>>>>>>>> seconds (1 tasks refusing to freeze, wq_busy=0):
>>>>>>>>>         task:xl              state:D stack:0     pid:466
>>>>>>>>> tgid:466   ppid:1      task_flags:0x400040 flags:0x00004006
>>>>>>>>>         Call Trace:
>>>>>>>>>          <TASK>
>>>>>>>>>          __schedule+0x2f3/0x780
>>>>>>>>>          schedule+0x27/0x80
>>>>>>>>>          schedule_preempt_disabled+0x15/0x30
>>>>>>>>>          __mutex_lock.constprop.0+0x49f/0x880
>>>>>>>>>          unregister_xenbus_watch+0x216/0x230
>>>>>>>>>          xenbus_write_watch+0xb9/0x220
>>>>>>>>>          xenbus_file_write+0x131/0x1b0
>>>>>>>>>          vfs_writev+0x26c/0x3d0
>>>>>>>>>          ? do_writev+0xeb/0x110
>>>>>>>>>          do_writev+0xeb/0x110
>>>>>>>>>          do_syscall_64+0x84/0x2c0
>>>>>>>>>          ? do_syscall_64+0x200/0x2c0
>>>>>>>>>          ? generic_handle_irq+0x3f/0x60
>>>>>>>>>          ? syscall_exit_work+0x108/0x140
>>>>>>>>>          ? do_syscall_64+0x200/0x2c0
>>>>>>>>>          ? __irq_exit_rcu+0x4c/0xe0
>>>>>>>>>          entry_SYSCALL_64_after_hwframe+0x76/0x7e
>>>>>>>>>         RIP: 0033:0x79b618138642
>>>>>>>>>         RSP: 002b:00007fff9a192fc8 EFLAGS: 00000246 ORIG_RAX:
>>>>>>>>> 0000000000000014
>>>>>>>>>         RAX: ffffffffffffffda RBX: 00000000024fd490 RCX:
>>>>>>>>> 000079b618138642
>>>>>>>>>         RDX: 0000000000000003 RSI: 00007fff9a193120 RDI:
>>>>>>>>> 0000000000000014
>>>>>>>>>         RBP: 00007fff9a193000 R08: 0000000000000000 R09:
>>>>>>>>> 0000000000000000
>>>>>>>>>         R10: 0000000000000000 R11: 0000000000000246 R12:
>>>>>>>>> 0000000000000014
>>>>>>>>>         R13: 00007fff9a193120 R14: 0000000000000003 R15:
>>>>>>>>> 0000000000000000
>>>>>>>>>          </TASK>
>>>>>>>>>         OOM killer enabled.
>>>>>>>>>         Restarting tasks: Starting
>>>>>>>>>         Restarting tasks: Done
>>>>>>>>>         xen:manage: do_suspend: freeze processes failed -16
>>>>>>>>>
>>>>>>>>> The process in question is `xl devd` daemon. It's a domU serving a
>>>>>>>>> xenvif backend.
>>>>>>>>>
>>>>>>>>> I noticed it on 6.16.1, but looking at earlier test logs I see
>>>>>>>>> it with
>>>>>>>>> 6.16-rc6 already (but interestingly, not 6.16-rc2 yet? feels
>>>>>>>>> weird given
>>>>>>>>> seemingly no relevant changes between rc2 and rc6).
>>>>>>>>
>>>>>>>> I forgot to include link for (a little) more details:
>>>>>>>> https://github.com/QubesOS/qubes-linux-kernel/pull/1157
>>>>>>>>
>>>>>>>> Especially, there is another call trace with panic_on_warn
>>>>>>>> enabled -
>>>>>>>> slightly different, but looks related.
>>>>>>>>
>>>>>>>
>>>>>>> I'm pretty sure the PV variant for suspending is just wrong: it
>>>>>>> is calling
>>>>>>> dpm_suspend_start() from do_suspend() without taking the required
>>>>>>> system_transition_mutex, resulting in the WARN() in
>>>>>>> pm_restrict_gfp_mask().
>>>>>>>
>>>>>>> It might be as easy as just adding the mutex() call to
>>>>>>> do_suspend(), but I'm
>>>>>>> really not sure that will be a proper fix.
>>>>>>
>>>>>> Hm, this might explain the second call trace, but not the freeze
>>>>>> failure
>>>>>> quoted here above, I think?
>>>>>
>>>>> While the patch I sent appears to fix this particular issue, it
>>>>> made me
>>>>> wonder: is there any fundamental reason why do_suspend() is not using
>>>>> pm_suspend() and register Xen-specific actions via
>>>>> platform_suspend_ops
>>>>> (and maybe syscore_ops)? From a brief look at the code, it should
>>>>> theoretically be possible, and should avoid issues like this.
>>>>>
>>>>> I tried to do a quick&dirty attempt at that[1], and it failed
>>>>> (panic). I
>>>>> surely made several mistakes there (and also left a ton of todo
>>>>> comments). But before spending any more time at that, I'd like to ask
>>>>> if this is a viable option at all.
>>>>
>>>> I think it might, but be careful with this, because there are two
>>>> "System Low power" paths in Linux
>>>> 1) Suspend2RAM and Co
>>>> 2) Hybernation
>>>>
>>>> While "Suspend2RAM and Co" path is relatively straight forward and
>>>> expected to be always
>>>> started through pm_suspend(). In general, it's expected to happen
>>>>    - from sysfs (User space)
>>>>    - from autosuspend (wakelocks).
>>>>
>>>> the "hibernation" path is more complicated:(
>>>> - Genuine Linux hybernation hibernate()/hibernate_quiet_exec()
>>>
>>> IIUC hibernation is very different as it puts Linux in charge of dumping
>>> all the state to the disk. In case of Xen, the primary use case for
>>> suspend is preparing VM for Xen toolstack serializing its state to disk
>>> (or migrating to another host).
>>> Additionally, VM suspend may be used as preparation for host suspend
>>> (this is what I actually do here). This is especially relevant if the VM
>>> has some PCI passthrough - to properly suspend (and resume) devices
>>> across host suspend.
>>>
>>>> I'm not sure what path Xen originally implemented :( It seems like
>>>> "suspend2RAM",
>>>> but, at the same time "hybernation" specific staff is used, like
>>>> PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE.
>>>> As result, Linux suspend/hybernation code moves forward while Xen
>>>> stays behind and unsync.
>>>
>>> Yeah, I think it's supposed to be suspend2RAM. TBH the
>>> PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE confuses me too and Qubes OS has a
>>> patch[2] to switch it to PMSG_SUSPEND/PMSG_RESUME.
>>>
>>>> So it sounds reasonable to avoid custom implementation, but may be
>>>> not easy :(
>>>>
>>>> Suspending Xen features can be split between suspend stages, but
>>>> not sure if platform_suspend_ops can be used.
>>>>
>>>> Generic suspend stages list
>>>> - freeze
>>>> - prepare
>>>> - suspend
>>>> - suspend_late
>>>> - suspend_noirq (SPIs disabled, except wakeups)
>>>>     [most of Xen specific staff has to be suspended at this point]
>>>> - disable_secondary_cpus
>>>> - arch disable IRQ (from this point no IRQs allowed, no timers, no
>>>> scheduling)
>>>> - syscore_suspend
>>>>     [rest here]
>>>> - platform->enter() (suspended)
>>>>
>>>> You can't just overwrite platform_suspend_ops, because ARM64 is
>>>> expected to enter
>>>> suspend through PSCI FW interface:
>>>> drivers/firmware/psci/psci.c
>>>>    static const struct platform_suspend_ops psci_suspend_ops = {
>>>
>>> Does this apply to a VM on ARM64 too? At least on x86, the VM is
>>> supposed to make a hypercall to tell Xen it suspended (the hypercall
>>> will return only on resume).
>>>
>>>> As an option, some Xen components could be converted to use
>>>> syscore_ops (but not xenstore),
>>>> and some might need to use DD(dev_pm_ops).
>>>>
>>>>>
>>>>> [1] https://github.com/marmarek/linux/
>>>>> commit/47cfdb991c85566c9c333570511e67bf477a5da6
>>>>
>>>> --
>>>> Best regards,
>>>> -grygorii
>>>>
>>>
>>> [2] https://github.com/QubesOS/qubes-linux-kernel/blob/main/xen-pm-
>>> use-suspend.patch
>>>
>>
>> On my setup I get a weird behavior when trying to suspend (s2idle) a
>> Linux guest.
>> Doing echo freeze > /sys/power/state in the guest seems to "freeze" the
>> guest for good, I could not unfreeze it afterward.
>> VCPU goes to 100% according to XenOrchestra
>> xl list shows state "r" but xl console blocks forever
>> xl shutdown would block for some time and then print:
>> Shutting down domain 721
>> ?ibxl: error: libxl_domain.c:848:pvcontrol_cb: guest didn't acknowledge
>> control request: -9
>> shutdown failed (rc=-9)
>>
>> Do you think it's related to your current issue?
>
> idle=halt on the Linux command line addresses the 100% CPU usage.  Or
> alternatively C2 needs to be implemented for guest vcpus.  I forget
> preceisely, but I think the 100% CPU is because there are no C-states
> available and Linux/cpuidle won't use halt by default.
>
> To wake up, you need a wake up source.  The ACPI buttons presses will do
> that:
> xl trigger $dom power
> xl trigger $dom sleep
>
> However, I think without changes, domU s2idle/S3 will detach all its PV
> devices.  Naturally they don't get reconnected on resume.  You can hack
> around that to skip the detach.
>
> Actually, maybe we just need:
> --- i/drivers/xen/xenbus/xenbus_probe_frontend.c
> +++ w/drivers/xen/xenbus/xenbus_probe_frontend.c
> @@ -148,8 +148,6 @@ static void xenbus_frontend_dev_shutdown(struct
> device *_dev)
>   }
>
>   static const struct dev_pm_ops xenbus_pm_ops = {
> -       .suspend        = xenbus_dev_suspend,
> -       .resume         = xenbus_frontend_dev_resume,
>          .freeze         = xenbus_dev_suspend,
>          .thaw           = xenbus_dev_cancel,
> -       .restore        = xenbus_dev_resume,
> +       .restore        = xenbus_frontend_dev_resume,
>   };
>
> b3e96c0c7562 ("xen: use freeze/restore/thaw PM events for suspend/
> resume/chkpt") changed from PMSG_SUSPEND/PMSG_RESUME to PMSG_FREEZE/
> PMSG_THAW/PMSG_RESTORE, but the suspend/resume callbacks remained.  But
> freeze and suspend being identical doesn't seem correct.
>
> This would leave xl save/restore/migrate using the hibernate freeze/
> thaw/resume.  S3/s2idle would no touch the PV devices, so they would
> still be present on resume.  Maybe there are cases I am not thinking of
> though.
>
> Regards,
> Jason
>

Hi all,

I finally took time to try what you guyz advised, thanks for all the
answers!

So, I tried this on a clean Debian 13 VM install on a XCP-ng 8.3 (Xen
4.17.5 with local patches) host.

1/ Just booting with idle=halt/nomwait/poll didn't help with waking up
the VM, it kept being in a weird unwakable state. Even providing xl
trigger power/sleep events. Although I reckon you were saying it would
help with the CPU@100% and indeed I don't see this anymore.

2/ Just applying QubesOS patch
https://github.com/QubesOS/qubes-linux-kernel/blob/main/xen-events-Add-wakeup-support-to-xen-pirq.patch
did help *a bit*. Hitting a key would wake the VM, then show again the
Debian desktop, with some flickering, allowing to move the mouse and bit
and hitting some keys on the keyboard ... then it would very quickly
completely freeze and stay dead.

3/ on top of previous patch, applying your modification (removing
.suspend, .resume and changing .restore to xenbus_frontend_dev_resume):
then tadaaa I can wake the VM and it stays alive afterward. It seems it
fixes the issue (I did not test for very very long though).

Thanks for the help on this!
What do you think we should do to move this forward? Submit those
changes to upstream Linux? That will take long to end up in distros...

--


--
Yann Sionneau | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: domU suspend issue - freeze processes failed - Linux 6.16

[Jason Andryuk]

On 2025-11-13 10:55, Yann Sionneau wrote:
> On 10/6/25 16:28, Jason Andryuk wrote:
>> On 2025-09-24 10:28, Yann Sionneau wrote:
>>> On 9/24/25 15:30, Marek Marczykowski-Górecki wrote:
>>>> On Wed, Sep 24, 2025 at 01:17:15PM +0300, Grygorii Strashko wrote:
>>>>>
>>>>>
>>>>> On 22.09.25 13:09, Marek Marczykowski-Górecki wrote:
>>>>>> On Fri, Aug 22, 2025 at 08:42:30PM +0200, Marek Marczykowski-
>>>>>> Górecki wrote:
>>>>>>> On Fri, Aug 22, 2025 at 05:27:20PM +0200, Jürgen Groß wrote:
>>>>>>>> On 22.08.25 16:42, Marek Marczykowski-Górecki wrote:
>>>>>>>>> On Fri, Aug 22, 2025 at 04:39:33PM +0200, Marek Marczykowski-
>>>>>>>>> Górecki wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> When suspending domU I get the following issue:
>>>>>>>>>>
>>>>>>>>>>          Freezing user space processes
>>>>>>>>>>          Freezing user space processes failed after 20.004
>>>>>>>>>> seconds (1 tasks refusing to freeze, wq_busy=0):
>>>>>>>>>>          task:xl              state:D stack:0     pid:466
>>>>>>>>>> tgid:466   ppid:1      task_flags:0x400040 flags:0x00004006
>>>>>>>>>>          Call Trace:
>>>>>>>>>>           <TASK>
>>>>>>>>>>           __schedule+0x2f3/0x780
>>>>>>>>>>           schedule+0x27/0x80
>>>>>>>>>>           schedule_preempt_disabled+0x15/0x30
>>>>>>>>>>           __mutex_lock.constprop.0+0x49f/0x880
>>>>>>>>>>           unregister_xenbus_watch+0x216/0x230
>>>>>>>>>>           xenbus_write_watch+0xb9/0x220
>>>>>>>>>>           xenbus_file_write+0x131/0x1b0
>>>>>>>>>>           vfs_writev+0x26c/0x3d0
>>>>>>>>>>           ? do_writev+0xeb/0x110
>>>>>>>>>>           do_writev+0xeb/0x110
>>>>>>>>>>           do_syscall_64+0x84/0x2c0
>>>>>>>>>>           ? do_syscall_64+0x200/0x2c0
>>>>>>>>>>           ? generic_handle_irq+0x3f/0x60
>>>>>>>>>>           ? syscall_exit_work+0x108/0x140
>>>>>>>>>>           ? do_syscall_64+0x200/0x2c0
>>>>>>>>>>           ? __irq_exit_rcu+0x4c/0xe0
>>>>>>>>>>           entry_SYSCALL_64_after_hwframe+0x76/0x7e
>>>>>>>>>>          RIP: 0033:0x79b618138642
>>>>>>>>>>          RSP: 002b:00007fff9a192fc8 EFLAGS: 00000246 ORIG_RAX:
>>>>>>>>>> 0000000000000014
>>>>>>>>>>          RAX: ffffffffffffffda RBX: 00000000024fd490 RCX:
>>>>>>>>>> 000079b618138642
>>>>>>>>>>          RDX: 0000000000000003 RSI: 00007fff9a193120 RDI:
>>>>>>>>>> 0000000000000014
>>>>>>>>>>          RBP: 00007fff9a193000 R08: 0000000000000000 R09:
>>>>>>>>>> 0000000000000000
>>>>>>>>>>          R10: 0000000000000000 R11: 0000000000000246 R12:
>>>>>>>>>> 0000000000000014
>>>>>>>>>>          R13: 00007fff9a193120 R14: 0000000000000003 R15:
>>>>>>>>>> 0000000000000000
>>>>>>>>>>           </TASK>
>>>>>>>>>>          OOM killer enabled.
>>>>>>>>>>          Restarting tasks: Starting
>>>>>>>>>>          Restarting tasks: Done
>>>>>>>>>>          xen:manage: do_suspend: freeze processes failed -16
>>>>>>>>>>
>>>>>>>>>> The process in question is `xl devd` daemon. It's a domU serving a
>>>>>>>>>> xenvif backend.
>>>>>>>>>>
>>>>>>>>>> I noticed it on 6.16.1, but looking at earlier test logs I see
>>>>>>>>>> it with
>>>>>>>>>> 6.16-rc6 already (but interestingly, not 6.16-rc2 yet? feels
>>>>>>>>>> weird given
>>>>>>>>>> seemingly no relevant changes between rc2 and rc6).
>>>>>>>>>
>>>>>>>>> I forgot to include link for (a little) more details:
>>>>>>>>> https://github.com/QubesOS/qubes-linux-kernel/pull/1157
>>>>>>>>>
>>>>>>>>> Especially, there is another call trace with panic_on_warn
>>>>>>>>> enabled -
>>>>>>>>> slightly different, but looks related.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I'm pretty sure the PV variant for suspending is just wrong: it
>>>>>>>> is calling
>>>>>>>> dpm_suspend_start() from do_suspend() without taking the required
>>>>>>>> system_transition_mutex, resulting in the WARN() in
>>>>>>>> pm_restrict_gfp_mask().
>>>>>>>>
>>>>>>>> It might be as easy as just adding the mutex() call to
>>>>>>>> do_suspend(), but I'm
>>>>>>>> really not sure that will be a proper fix.
>>>>>>>
>>>>>>> Hm, this might explain the second call trace, but not the freeze
>>>>>>> failure
>>>>>>> quoted here above, I think?
>>>>>>
>>>>>> While the patch I sent appears to fix this particular issue, it
>>>>>> made me
>>>>>> wonder: is there any fundamental reason why do_suspend() is not using
>>>>>> pm_suspend() and register Xen-specific actions via
>>>>>> platform_suspend_ops
>>>>>> (and maybe syscore_ops)? From a brief look at the code, it should
>>>>>> theoretically be possible, and should avoid issues like this.
>>>>>>
>>>>>> I tried to do a quick&dirty attempt at that[1], and it failed
>>>>>> (panic). I
>>>>>> surely made several mistakes there (and also left a ton of todo
>>>>>> comments). But before spending any more time at that, I'd like to ask
>>>>>> if this is a viable option at all.
>>>>>
>>>>> I think it might, but be careful with this, because there are two
>>>>> "System Low power" paths in Linux
>>>>> 1) Suspend2RAM and Co
>>>>> 2) Hybernation
>>>>>
>>>>> While "Suspend2RAM and Co" path is relatively straight forward and
>>>>> expected to be always
>>>>> started through pm_suspend(). In general, it's expected to happen
>>>>>     - from sysfs (User space)
>>>>>     - from autosuspend (wakelocks).
>>>>>
>>>>> the "hibernation" path is more complicated:(
>>>>> - Genuine Linux hybernation hibernate()/hibernate_quiet_exec()
>>>>
>>>> IIUC hibernation is very different as it puts Linux in charge of dumping
>>>> all the state to the disk. In case of Xen, the primary use case for
>>>> suspend is preparing VM for Xen toolstack serializing its state to disk
>>>> (or migrating to another host).
>>>> Additionally, VM suspend may be used as preparation for host suspend
>>>> (this is what I actually do here). This is especially relevant if the VM
>>>> has some PCI passthrough - to properly suspend (and resume) devices
>>>> across host suspend.
>>>>
>>>>> I'm not sure what path Xen originally implemented :( It seems like
>>>>> "suspend2RAM",
>>>>> but, at the same time "hybernation" specific staff is used, like
>>>>> PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE.
>>>>> As result, Linux suspend/hybernation code moves forward while Xen
>>>>> stays behind and unsync.
>>>>
>>>> Yeah, I think it's supposed to be suspend2RAM. TBH the
>>>> PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE confuses me too and Qubes OS has a
>>>> patch[2] to switch it to PMSG_SUSPEND/PMSG_RESUME.
>>>>
>>>>> So it sounds reasonable to avoid custom implementation, but may be
>>>>> not easy :(
>>>>>
>>>>> Suspending Xen features can be split between suspend stages, but
>>>>> not sure if platform_suspend_ops can be used.
>>>>>
>>>>> Generic suspend stages list
>>>>> - freeze
>>>>> - prepare
>>>>> - suspend
>>>>> - suspend_late
>>>>> - suspend_noirq (SPIs disabled, except wakeups)
>>>>>      [most of Xen specific staff has to be suspended at this point]
>>>>> - disable_secondary_cpus
>>>>> - arch disable IRQ (from this point no IRQs allowed, no timers, no
>>>>> scheduling)
>>>>> - syscore_suspend
>>>>>      [rest here]
>>>>> - platform->enter() (suspended)
>>>>>
>>>>> You can't just overwrite platform_suspend_ops, because ARM64 is
>>>>> expected to enter
>>>>> suspend through PSCI FW interface:
>>>>> drivers/firmware/psci/psci.c
>>>>>     static const struct platform_suspend_ops psci_suspend_ops = {
>>>>
>>>> Does this apply to a VM on ARM64 too? At least on x86, the VM is
>>>> supposed to make a hypercall to tell Xen it suspended (the hypercall
>>>> will return only on resume).
>>>>
>>>>> As an option, some Xen components could be converted to use
>>>>> syscore_ops (but not xenstore),
>>>>> and some might need to use DD(dev_pm_ops).
>>>>>
>>>>>>
>>>>>> [1] https://github.com/marmarek/linux/
>>>>>> commit/47cfdb991c85566c9c333570511e67bf477a5da6
>>>>>
>>>>> -- 
>>>>> Best regards,
>>>>> -grygorii
>>>>>
>>>>
>>>> [2] https://github.com/QubesOS/qubes-linux-kernel/blob/main/xen-pm-
>>>> use-suspend.patch
>>>>
>>>
>>> On my setup I get a weird behavior when trying to suspend (s2idle) a
>>> Linux guest.
>>> Doing echo freeze > /sys/power/state in the guest seems to "freeze" the
>>> guest for good, I could not unfreeze it afterward.
>>> VCPU goes to 100% according to XenOrchestra
>>> xl list shows state "r" but xl console blocks forever
>>> xl shutdown would block for some time and then print:
>>> Shutting down domain 721
>>> ?ibxl: error: libxl_domain.c:848:pvcontrol_cb: guest didn't acknowledge
>>> control request: -9
>>> shutdown failed (rc=-9)
>>>
>>> Do you think it's related to your current issue?
>>
>> idle=halt on the Linux command line addresses the 100% CPU usage.  Or
>> alternatively C2 needs to be implemented for guest vcpus.  I forget
>> preceisely, but I think the 100% CPU is because there are no C-states
>> available and Linux/cpuidle won't use halt by default.
>>
>> To wake up, you need a wake up source.  The ACPI buttons presses will do
>> that:
>> xl trigger $dom power
>> xl trigger $dom sleep
>>
>> However, I think without changes, domU s2idle/S3 will detach all its PV
>> devices.  Naturally they don't get reconnected on resume.  You can hack
>> around that to skip the detach.
>>
>> Actually, maybe we just need:
>> --- i/drivers/xen/xenbus/xenbus_probe_frontend.c
>> +++ w/drivers/xen/xenbus/xenbus_probe_frontend.c
>> @@ -148,8 +148,6 @@ static void xenbus_frontend_dev_shutdown(struct
>> device *_dev)
>>    }
>>
>>    static const struct dev_pm_ops xenbus_pm_ops = {
>> -       .suspend        = xenbus_dev_suspend,
>> -       .resume         = xenbus_frontend_dev_resume,
>>           .freeze         = xenbus_dev_suspend,
>>           .thaw           = xenbus_dev_cancel,
>> -       .restore        = xenbus_dev_resume,
>> +       .restore        = xenbus_frontend_dev_resume,
>>    };
>>
>> b3e96c0c7562 ("xen: use freeze/restore/thaw PM events for suspend/
>> resume/chkpt") changed from PMSG_SUSPEND/PMSG_RESUME to PMSG_FREEZE/
>> PMSG_THAW/PMSG_RESTORE, but the suspend/resume callbacks remained.  But
>> freeze and suspend being identical doesn't seem correct.
>>
>> This would leave xl save/restore/migrate using the hibernate freeze/
>> thaw/resume.  S3/s2idle would no touch the PV devices, so they would
>> still be present on resume.  Maybe there are cases I am not thinking of
>> though.
>>
>> Regards,
>> Jason
>>
> 
> Hi all,
> 
> I finally took time to try what you guyz advised, thanks for all the
> answers!
> 
> So, I tried this on a clean Debian 13 VM install on a XCP-ng 8.3 (Xen
> 4.17.5 with local patches) host.
> 
> 1/ Just booting with idle=halt/nomwait/poll didn't help with waking up
> the VM, it kept being in a weird unwakable state. Even providing xl
> trigger power/sleep events. Although I reckon you were saying it would
> help with the CPU@100% and indeed I don't see this anymore.

Yes, just 100% CPU.

> 2/ Just applying QubesOS patch
> https://github.com/QubesOS/qubes-linux-kernel/blob/main/xen-events-Add-wakeup-support-to-xen-pirq.patch
> did help *a bit*. Hitting a key would wake the VM, then show again the
> Debian desktop, with some flickering, allowing to move the mouse and bit
> and hitting some keys on the keyboard ... then it would very quickly
> completely freeze and stay dead.

I think this is needed for dom0 s2idle, but not for domU.

> 3/ on top of previous patch, applying your modification (removing
> .suspend, .resume and changing .restore to xenbus_frontend_dev_resume):
> then tadaaa I can wake the VM and it stays alive afterward. It seems it
> fixes the issue (I did not test for very very long though).
> 
> Thanks for the help on this!
> What do you think we should do to move this forward? Submit those
> changes to upstream Linux? That will take long to end up in distros...

I submitted the patch for #3.  Yes, it'll take a little while to get to 
distros, but I don't see any other choice here.

Regards,
Jason