Forcing devices into idle

Forcing devices into idle

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 1544 bytes --]

Hi Rafael, all,

I've been looking at adding support for VPR on Tegra. This is a special
type of protected memory (video protection region) and since it can be
used for potentially large video frames, it can grow fairly large. For
certain devices, this is prohibitively large, so the VPR needs to be
resizable at runtime.

One of the prerequisites for runtime resizing of the VPR is that certain
devices need to go into idle mode. Essentially they need to be suspended
before the VPR resize and resumed after.

Instead of adding some custom API to achieve this, I was hoping I could
use the PM framework for this. I can hook up devices with VPR using DT
properties, and using the reserved memory infrastructure I can get
access to all the necessary struct device:s for the devices that are
impacted by this.

Unfortunately, I can't seem to find a good way to force devices into an
idle mode using the existing APIs. I've had some success by running
pm_runtime_put_sync() and pm_runtime_get_sync() before and after the
resize on these devices, but that feels wrong, since these APIs are more
targeted at being called from the owning drivers themselves.

Any thoughts on how to solve this? Is the pm_runtime_{put,get}_sync()
method acceptable? If not, are there other alternatives to achieve the
same thing that I'm not aware of? Would it be useful to add a new set of
APIs to force devices into an idle state (which could be semantically
different from runtime suspend)? Or is this all too specific for any
kind of generic API?

Thanks,
Thierry

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Forcing devices into idle

[Oliver Neukum]

On 03.07.25 12:08, Thierry Reding wrote:

> Any thoughts on how to solve this? Is the pm_runtime_{put,get}_sync()
> method acceptable? If not, are there other alternatives to achieve the
> same thing that I'm not aware of? Would it be useful to add a new set of
> APIs to force devices into an idle state (which could be semantically
> different from runtime suspend)? Or is this all too specific for any
> kind of generic API?

Basically what you need is what happens when the system prepares to
do a snapshot for S4. However, if you just perform FREEZE and then THAW,
devices will assume that user space has been frozen. You need a way
to substitute for that assumption.

Runtime power management is unlikely to be a fruitful approach to your
needs.

	HTH
		Oliver

Re: Forcing devices into idle

[Rafael J. Wysocki]

On Thu, Jul 3, 2025 at 12:33 PM Oliver Neukum <oneukum@suse.com> wrote:
>
> On 03.07.25 12:08, Thierry Reding wrote:
>
> > Any thoughts on how to solve this? Is the pm_runtime_{put,get}_sync()
> > method acceptable? If not, are there other alternatives to achieve the
> > same thing that I'm not aware of? Would it be useful to add a new set of
> > APIs to force devices into an idle state (which could be semantically
> > different from runtime suspend)? Or is this all too specific for any
> > kind of generic API?
>
> Basically what you need is what happens when the system prepares to
> do a snapshot for S4. However, if you just perform FREEZE and then THAW,
> devices will assume that user space has been frozen. You need a way
> to substitute for that assumption.

Well, you just need to freeze user space beforehand.

> Runtime power management is unlikely to be a fruitful approach to your
> needs.

Agreed.

Re: Forcing devices into idle

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 1996 bytes --]

On Thu, Jul 03, 2025 at 01:12:15PM +0200, Rafael J. Wysocki wrote:
> On Thu, Jul 3, 2025 at 12:33 PM Oliver Neukum <oneukum@suse.com> wrote:
> >
> > On 03.07.25 12:08, Thierry Reding wrote:
> >
> > > Any thoughts on how to solve this? Is the pm_runtime_{put,get}_sync()
> > > method acceptable? If not, are there other alternatives to achieve the
> > > same thing that I'm not aware of? Would it be useful to add a new set of
> > > APIs to force devices into an idle state (which could be semantically
> > > different from runtime suspend)? Or is this all too specific for any
> > > kind of generic API?
> >
> > Basically what you need is what happens when the system prepares to
> > do a snapshot for S4. However, if you just perform FREEZE and then THAW,
> > devices will assume that user space has been frozen. You need a way
> > to substitute for that assumption.
> 
> Well, you just need to freeze user space beforehand.

Freezing userspace seems a bit heavy-handed. There's only a very few
devices that need to be put into reset (such as the GPU), so most of
userspace should be fine to continue to run. For the GPU the idea is
to block all incoming requests while the device is forced into idle,
and then to resume processing these requests after the VPR resize.

But maybe freezing userspace isn't the heavy operation that I think it
is. Ideally we do not want to suspend things for too long to avoid
stuttering on the userspace side.

Also, I think we'd want the freezing to be triggered by the VPR driver
because userspace ideally doesn't know when the resizing happens. The
DMA BUF heap API that I'm trying to use is too simple for that, and
only the VPR driver knows when a resize needs to happen.

Is it possible to trigger the freeze from a kernel driver? Or localize
the freezing of userspace to only the processes that are accessing a
given device?

Other than that, freeze() and thaw() seem like the right callbacks for
this.

Thierry

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Forcing devices into idle

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 2986 bytes --]

On Thu, Jul 03, 2025 at 02:06:00PM +0200, Thierry Reding wrote:
> On Thu, Jul 03, 2025 at 01:12:15PM +0200, Rafael J. Wysocki wrote:
> > On Thu, Jul 3, 2025 at 12:33 PM Oliver Neukum <oneukum@suse.com> wrote:
> > >
> > > On 03.07.25 12:08, Thierry Reding wrote:
> > >
> > > > Any thoughts on how to solve this? Is the pm_runtime_{put,get}_sync()
> > > > method acceptable? If not, are there other alternatives to achieve the
> > > > same thing that I'm not aware of? Would it be useful to add a new set of
> > > > APIs to force devices into an idle state (which could be semantically
> > > > different from runtime suspend)? Or is this all too specific for any
> > > > kind of generic API?
> > >
> > > Basically what you need is what happens when the system prepares to
> > > do a snapshot for S4. However, if you just perform FREEZE and then THAW,
> > > devices will assume that user space has been frozen. You need a way
> > > to substitute for that assumption.
> > 
> > Well, you just need to freeze user space beforehand.
> 
> Freezing userspace seems a bit heavy-handed. There's only a very few
> devices that need to be put into reset (such as the GPU), so most of
> userspace should be fine to continue to run. For the GPU the idea is
> to block all incoming requests while the device is forced into idle,
> and then to resume processing these requests after the VPR resize.
> 
> But maybe freezing userspace isn't the heavy operation that I think it
> is. Ideally we do not want to suspend things for too long to avoid
> stuttering on the userspace side.
> 
> Also, I think we'd want the freezing to be triggered by the VPR driver
> because userspace ideally doesn't know when the resizing happens. The
> DMA BUF heap API that I'm trying to use is too simple for that, and
> only the VPR driver knows when a resize needs to happen.
> 
> Is it possible to trigger the freeze from a kernel driver? Or localize
> the freezing of userspace to only the processes that are accessing a
> given device?
> 
> Other than that, freeze() and thaw() seem like the right callbacks for
> this.

I've prototyped this using the sledgehammer freeze_processes() and
thaw_processes() functions and the entire process seems to be pretty
quick. I can get through most of it in ~30 ms. This is on a mostly
idle test system, so I expect this to go up significantly if there
is a high load.

On the other hand, this will drastically simplify the GPU driver
implementation, because by the time ->freeze() is called, all userspace
will be frozen, so there's no need to do any blocking on the kernel
side.

What I have now is roughly this:

	freeze_processes();

	for each VPR device dev:
		pm_generic_freeze(dev);

	resize_vpr();

	for each VPR device dev:
		pm_generic_thaw(dev);

	thaw_processes()

I still can't shake the feeling that this is sketchy, but it seems to
work. Is there anything blatantly wrong about this?

Thierry

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Forcing devices into idle

[Rafael J. Wysocki]

On Thu, Jul 3, 2025 at 3:32 PM Thierry Reding <thierry.reding@gmail.com> wrote:
>
> On Thu, Jul 03, 2025 at 02:06:00PM +0200, Thierry Reding wrote:
> > On Thu, Jul 03, 2025 at 01:12:15PM +0200, Rafael J. Wysocki wrote:
> > > On Thu, Jul 3, 2025 at 12:33 PM Oliver Neukum <oneukum@suse.com> wrote:
> > > >
> > > > On 03.07.25 12:08, Thierry Reding wrote:
> > > >
> > > > > Any thoughts on how to solve this? Is the pm_runtime_{put,get}_sync()
> > > > > method acceptable? If not, are there other alternatives to achieve the
> > > > > same thing that I'm not aware of? Would it be useful to add a new set of
> > > > > APIs to force devices into an idle state (which could be semantically
> > > > > different from runtime suspend)? Or is this all too specific for any
> > > > > kind of generic API?
> > > >
> > > > Basically what you need is what happens when the system prepares to
> > > > do a snapshot for S4. However, if you just perform FREEZE and then THAW,
> > > > devices will assume that user space has been frozen. You need a way
> > > > to substitute for that assumption.
> > >
> > > Well, you just need to freeze user space beforehand.
> >
> > Freezing userspace seems a bit heavy-handed. There's only a very few
> > devices that need to be put into reset (such as the GPU), so most of
> > userspace should be fine to continue to run. For the GPU the idea is
> > to block all incoming requests while the device is forced into idle,
> > and then to resume processing these requests after the VPR resize.
> >
> > But maybe freezing userspace isn't the heavy operation that I think it
> > is. Ideally we do not want to suspend things for too long to avoid
> > stuttering on the userspace side.
> >
> > Also, I think we'd want the freezing to be triggered by the VPR driver
> > because userspace ideally doesn't know when the resizing happens. The
> > DMA BUF heap API that I'm trying to use is too simple for that, and
> > only the VPR driver knows when a resize needs to happen.
> >
> > Is it possible to trigger the freeze from a kernel driver? Or localize
> > the freezing of userspace to only the processes that are accessing a
> > given device?
> >
> > Other than that, freeze() and thaw() seem like the right callbacks for
> > this.
>
> I've prototyped this using the sledgehammer freeze_processes() and
> thaw_processes() functions and the entire process seems to be pretty
> quick. I can get through most of it in ~30 ms. This is on a mostly
> idle test system, so I expect this to go up significantly if there
> is a high load.
>
> On the other hand, this will drastically simplify the GPU driver
> implementation, because by the time ->freeze() is called, all userspace
> will be frozen, so there's no need to do any blocking on the kernel
> side.
>
> What I have now is roughly this:
>
>         freeze_processes();
>
>         for each VPR device dev:
>                 pm_generic_freeze(dev);
>
>         resize_vpr();
>
>         for each VPR device dev:
>                 pm_generic_thaw(dev);
>
>         thaw_processes()
>
> I still can't shake the feeling that this is sketchy, but it seems to
> work. Is there anything blatantly wrong about this?

There are a few things to take into consideration.

First, there are 4 tiers of "freeze" callbacks (->prepare, ->freeze,
->freeze_late, ->freeze_noirq), and analogously for "thaw" callbacks,
but you only use one of them.  This may be fine in a particular case,
but you need to ensure that the other tiers are not needed and, in
particular, the _noirq ones need not be involved.  Also ensure that
they don't assume that PM notifiers have run (or that they will run on
the resume side).

Second, if there are dependencies between the devices being frozen and
other devices, they will have to be taken into account.

Also note that kernel threads are generally not affected by
freeze_processes(), but I guess this is not a problem in your use
case.

Thanks!

Re: Forcing devices into idle

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 5397 bytes --]

On Thu, Jul 03, 2025 at 03:46:03PM +0200, Rafael J. Wysocki wrote:
> On Thu, Jul 3, 2025 at 3:32 PM Thierry Reding <thierry.reding@gmail.com> wrote:
> >
> > On Thu, Jul 03, 2025 at 02:06:00PM +0200, Thierry Reding wrote:
> > > On Thu, Jul 03, 2025 at 01:12:15PM +0200, Rafael J. Wysocki wrote:
> > > > On Thu, Jul 3, 2025 at 12:33 PM Oliver Neukum <oneukum@suse.com> wrote:
> > > > >
> > > > > On 03.07.25 12:08, Thierry Reding wrote:
> > > > >
> > > > > > Any thoughts on how to solve this? Is the pm_runtime_{put,get}_sync()
> > > > > > method acceptable? If not, are there other alternatives to achieve the
> > > > > > same thing that I'm not aware of? Would it be useful to add a new set of
> > > > > > APIs to force devices into an idle state (which could be semantically
> > > > > > different from runtime suspend)? Or is this all too specific for any
> > > > > > kind of generic API?
> > > > >
> > > > > Basically what you need is what happens when the system prepares to
> > > > > do a snapshot for S4. However, if you just perform FREEZE and then THAW,
> > > > > devices will assume that user space has been frozen. You need a way
> > > > > to substitute for that assumption.
> > > >
> > > > Well, you just need to freeze user space beforehand.
> > >
> > > Freezing userspace seems a bit heavy-handed. There's only a very few
> > > devices that need to be put into reset (such as the GPU), so most of
> > > userspace should be fine to continue to run. For the GPU the idea is
> > > to block all incoming requests while the device is forced into idle,
> > > and then to resume processing these requests after the VPR resize.
> > >
> > > But maybe freezing userspace isn't the heavy operation that I think it
> > > is. Ideally we do not want to suspend things for too long to avoid
> > > stuttering on the userspace side.
> > >
> > > Also, I think we'd want the freezing to be triggered by the VPR driver
> > > because userspace ideally doesn't know when the resizing happens. The
> > > DMA BUF heap API that I'm trying to use is too simple for that, and
> > > only the VPR driver knows when a resize needs to happen.
> > >
> > > Is it possible to trigger the freeze from a kernel driver? Or localize
> > > the freezing of userspace to only the processes that are accessing a
> > > given device?
> > >
> > > Other than that, freeze() and thaw() seem like the right callbacks for
> > > this.
> >
> > I've prototyped this using the sledgehammer freeze_processes() and
> > thaw_processes() functions and the entire process seems to be pretty
> > quick. I can get through most of it in ~30 ms. This is on a mostly
> > idle test system, so I expect this to go up significantly if there
> > is a high load.
> >
> > On the other hand, this will drastically simplify the GPU driver
> > implementation, because by the time ->freeze() is called, all userspace
> > will be frozen, so there's no need to do any blocking on the kernel
> > side.
> >
> > What I have now is roughly this:
> >
> >         freeze_processes();
> >
> >         for each VPR device dev:
> >                 pm_generic_freeze(dev);
> >
> >         resize_vpr();
> >
> >         for each VPR device dev:
> >                 pm_generic_thaw(dev);
> >
> >         thaw_processes()
> >
> > I still can't shake the feeling that this is sketchy, but it seems to
> > work. Is there anything blatantly wrong about this?
> 
> There are a few things to take into consideration.
> 
> First, there are 4 tiers of "freeze" callbacks (->prepare, ->freeze,
> ->freeze_late, ->freeze_noirq), and analogously for "thaw" callbacks,
> but you only use one of them.  This may be fine in a particular case,
> but you need to ensure that the other tiers are not needed and, in
> particular, the _noirq ones need not be involved.  Also ensure that
> they don't assume that PM notifiers have run (or that they will run on
> the resume side).

I think that's something we can accomodate in this case. The primary
device that needs this in the integrated GPU on Tegra, so it's very a
narrow set.

> Second, if there are dependencies between the devices being frozen and
> other devices, they will have to be taken into account.

Fortunately, as far as I know the only dependency, if any, would be via
the userspace. There's no relationship between the devices from a PM
point of view.

There might be a video decoder that decodes images into the VPR and the
GPU would then read data out of the VPR and composite onto the screen.
So as long as userspace is frozen, there should be no issue.

> Also note that kernel threads are generally not affected by
> freeze_processes(), but I guess this is not a problem in your use
> case.

I had read through some of the documentation around this, and yes, the
kernel threads shouldn't be an issue.

In any case I'll make sure to add comments where necessary to point out
the peculiarities of the whole thing.

So this sounds like a workable solution. If for whatever reason freezing
the entire userspace doesn't work out, would it still be possible to
"abuse" the freeze() and thaw() callbacks like this? The way I imagine
this would be for freeze() to include code that effectively suspends all
userspace submissions to the GPU (i.e. effectively suspend all GPU
related operations).

Thierry

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Forcing devices into idle

[Rafael J. Wysocki]

On Thu, Jul 3, 2025 at 4:14 PM Thierry Reding <thierry.reding@gmail.com> wrote:
>
> On Thu, Jul 03, 2025 at 03:46:03PM +0200, Rafael J. Wysocki wrote:
> > On Thu, Jul 3, 2025 at 3:32 PM Thierry Reding <thierry.reding@gmail.com> wrote:
> > >
> > > On Thu, Jul 03, 2025 at 02:06:00PM +0200, Thierry Reding wrote:
> > > > On Thu, Jul 03, 2025 at 01:12:15PM +0200, Rafael J. Wysocki wrote:
> > > > > On Thu, Jul 3, 2025 at 12:33 PM Oliver Neukum <oneukum@suse.com> wrote:
> > > > > >
> > > > > > On 03.07.25 12:08, Thierry Reding wrote:
> > > > > >
> > > > > > > Any thoughts on how to solve this? Is the pm_runtime_{put,get}_sync()
> > > > > > > method acceptable? If not, are there other alternatives to achieve the
> > > > > > > same thing that I'm not aware of? Would it be useful to add a new set of
> > > > > > > APIs to force devices into an idle state (which could be semantically
> > > > > > > different from runtime suspend)? Or is this all too specific for any
> > > > > > > kind of generic API?
> > > > > >
> > > > > > Basically what you need is what happens when the system prepares to
> > > > > > do a snapshot for S4. However, if you just perform FREEZE and then THAW,
> > > > > > devices will assume that user space has been frozen. You need a way
> > > > > > to substitute for that assumption.
> > > > >
> > > > > Well, you just need to freeze user space beforehand.
> > > >
> > > > Freezing userspace seems a bit heavy-handed. There's only a very few
> > > > devices that need to be put into reset (such as the GPU), so most of
> > > > userspace should be fine to continue to run. For the GPU the idea is
> > > > to block all incoming requests while the device is forced into idle,
> > > > and then to resume processing these requests after the VPR resize.
> > > >
> > > > But maybe freezing userspace isn't the heavy operation that I think it
> > > > is. Ideally we do not want to suspend things for too long to avoid
> > > > stuttering on the userspace side.
> > > >
> > > > Also, I think we'd want the freezing to be triggered by the VPR driver
> > > > because userspace ideally doesn't know when the resizing happens. The
> > > > DMA BUF heap API that I'm trying to use is too simple for that, and
> > > > only the VPR driver knows when a resize needs to happen.
> > > >
> > > > Is it possible to trigger the freeze from a kernel driver? Or localize
> > > > the freezing of userspace to only the processes that are accessing a
> > > > given device?
> > > >
> > > > Other than that, freeze() and thaw() seem like the right callbacks for
> > > > this.
> > >
> > > I've prototyped this using the sledgehammer freeze_processes() and
> > > thaw_processes() functions and the entire process seems to be pretty
> > > quick. I can get through most of it in ~30 ms. This is on a mostly
> > > idle test system, so I expect this to go up significantly if there
> > > is a high load.
> > >
> > > On the other hand, this will drastically simplify the GPU driver
> > > implementation, because by the time ->freeze() is called, all userspace
> > > will be frozen, so there's no need to do any blocking on the kernel
> > > side.
> > >
> > > What I have now is roughly this:
> > >
> > >         freeze_processes();
> > >
> > >         for each VPR device dev:
> > >                 pm_generic_freeze(dev);
> > >
> > >         resize_vpr();
> > >
> > >         for each VPR device dev:
> > >                 pm_generic_thaw(dev);
> > >
> > >         thaw_processes()
> > >
> > > I still can't shake the feeling that this is sketchy, but it seems to
> > > work. Is there anything blatantly wrong about this?
> >
> > There are a few things to take into consideration.
> >
> > First, there are 4 tiers of "freeze" callbacks (->prepare, ->freeze,
> > ->freeze_late, ->freeze_noirq), and analogously for "thaw" callbacks,
> > but you only use one of them.  This may be fine in a particular case,
> > but you need to ensure that the other tiers are not needed and, in
> > particular, the _noirq ones need not be involved.  Also ensure that
> > they don't assume that PM notifiers have run (or that they will run on
> > the resume side).
>
> I think that's something we can accomodate in this case. The primary
> device that needs this in the integrated GPU on Tegra, so it's very a
> narrow set.
>
> > Second, if there are dependencies between the devices being frozen and
> > other devices, they will have to be taken into account.
>
> Fortunately, as far as I know the only dependency, if any, would be via
> the userspace. There's no relationship between the devices from a PM
> point of view.
>
> There might be a video decoder that decodes images into the VPR and the
> GPU would then read data out of the VPR and composite onto the screen.
> So as long as userspace is frozen, there should be no issue.
>
> > Also note that kernel threads are generally not affected by
> > freeze_processes(), but I guess this is not a problem in your use
> > case.
>
> I had read through some of the documentation around this, and yes, the
> kernel threads shouldn't be an issue.
>
> In any case I'll make sure to add comments where necessary to point out
> the peculiarities of the whole thing.
>
> So this sounds like a workable solution. If for whatever reason freezing
> the entire userspace doesn't work out, would it still be possible to
> "abuse" the freeze() and thaw() callbacks like this? The way I imagine
> this would be for freeze() to include code that effectively suspends all
> userspace submissions to the GPU (i.e. effectively suspend all GPU
> related operations).

If you can effectively prevent user space from interacting with any of
the affected devices, directly or indirectly, even via mmapped memory
regions or some such, then this should work.

The freezing of tasks is basically all about getting user space out of the way.