PM QoS dynamic resource manager

PM QoS dynamic resource manager

[Chidambaram, Praveen]

Hi,

I am looking for some extensible features for PM QoS. Specifically, 
dynamic runtime parameters that my architecture could define at 
runtime and the drivers can make use of. 

What are the plans in this regard?

Thanks,
Praveen

Re: PM QoS dynamic resource manager

[mark gross]

On Fri, May 21, 2010 at 03:37:39PM -0600, Chidambaram, Praveen wrote:
> Hi,
> 
> I am looking for some extensible features for PM QoS. Specifically, 
> dynamic runtime parameters that my architecture could define at 
> runtime and the drivers can make use of. 
> 
> What are the plans in this regard?

we've talked about it a few times in the past.  

It's come down to how do we prevent platform independent code from
depending on such platform independent dynamic prarams in a sane manner.

I've alway's pushed for coming up with more reasonable abstractions that
could be re-used and adding them (like the bus thing) as opposed to
having drivers make / use new parrameters dynamically.

--mgross

> 
> Thanks,
> Praveen
> _______________________________________________
> linux-pm mailing list
> linux-pm@lists.linux-foundation.org
> https://lists.linux-foundation.org/mailman/listinfo/linux-pm

Re: PM QoS dynamic resource manager

[Chidambaram, Praveen]

Thanks Mark.

I do understand your concern about having static paramters for the 
system, makes it more usuable across architectures. 

My requirement is drivers would want something like what PM QoS 
provides for their driver - (aggregation/max/min) of requests and pass 
on the final value to the driver for that specific driver. Writing it 
out for every driver is a duplication of code when PM QoS already 
provides this as a framework.

If a subsytem wants to use the driver they can use PM QoS with the 
published driver-specific QoS param. If the driver does not exist, ie, 
the param is not defined then they get the error code back. I do have 
something that has been written for arch/arm/mach-msm (it is arch 
agnostic though). I would like to invite you to take a look at the 
link below.

https://www.codeaurora.org/gitweb/quic/la/?p=kernel/msm.git;a=blob;f=Documentation/arm/msm/npa.txt;h=b5134041be1e6109b86179085db8aee19960c5dc;hb=android-msm-2.6.32
The implementation is in arch/arm/mach-msm/npa.c

I would like to bring in some key features that NPA has into PM QoS. 
NPA complements PM QoS in a way that it is completely dynamic and 
supports multiple params (known as resources/nodes). The essence of 
the framework is same as PM QoS - requests can be made by sub-systems 
and the framework aggregates or calculates the final value (the 
function to do this can be driver specific or framework pre-defined 
library) and the result is then passed on to the driver. In addition, 
the NPA framework supports aliasing of params (a 'system bus' on one 
architecture can mean one thing and on an other can reference another 
resource) and would be abstracted from the users. Dependencies between 
params can also be specified. The document also specifies 'remoting', 
which we can ignore it as it is just a layer over the framework.

I am interested in bringing in some of these features into PM QoS. Let 
me know what you think.

Thanks,
Praveen




On Tue, 25 May 2010 8:52am, mark gross <640e9920@gmail.com> wrote:
> On Fri, May 21, 2010 at 03:37:39PM -0600, Chidambaram, Praveen wrote:
> > Hi,
> > 
> > I am looking for some extensible features for PM QoS. Specifically, 
> > dynamic runtime parameters that my architecture could define at 
> > runtime and the drivers can make use of. 
> > 
> > What are the plans in this regard?
> 
> we've talked about it a few times in the past.  
> 
> It's come down to how do we prevent platform independent code from
> depending on such platform independent dynamic prarams in a sane manner.
> 
> I've alway's pushed for coming up with more reasonable abstractions that
> could be re-used and adding them (like the bus thing) as opposed to
> having drivers make / use new parrameters dynamically.
> 
> --mgross
> 
> > 
> > Thanks,
> > Praveen
> > _______________________________________________
> > linux-pm mailing list
> > linux-pm@lists.linux-foundation.org
> > https://lists.linux-foundation.org/mailman/listinfo/linux-pm
> 

Re: PM QoS dynamic resource manager

[Mike Chan]

On Tue, May 25, 2010 at 7:52 AM, mark gross <640e9920@gmail.com> wrote:
> On Fri, May 21, 2010 at 03:37:39PM -0600, Chidambaram, Praveen wrote:
>> Hi,
>>
>> I am looking for some extensible features for PM QoS. Specifically,
>> dynamic runtime parameters that my architecture could define at
>> runtime and the drivers can make use of.
>>
>> What are the plans in this regard?
>
> we've talked about it a few times in the past.
>
> It's come down to how do we prevent platform independent code from
> depending on such platform independent dynamic prarams in a sane manner.
>
> I've alway's pushed for coming up with more reasonable abstractions that
> could be re-used and adding them (like the bus thing) as opposed to
> having drivers make / use new parrameters dynamically.
>

Would it be possible to elaborate on the bus resource ideas you had? I
couldn't find anything in the archives. Sounds like a similar that we
(Android / embedded world) are having (or will) on multiple platforms
(omap / msm/ tegra).

-- Mike

> --mgross
>
>>
>> Thanks,
>> Praveen
>> _______________________________________________
>> linux-pm mailing list
>> linux-pm@lists.linux-foundation.org
>> https://lists.linux-foundation.org/mailman/listinfo/linux-pm
> _______________________________________________
> linux-pm mailing list
> linux-pm@lists.linux-foundation.org
> https://lists.linux-foundation.org/mailman/listinfo/linux-pm
>

Re: PM QoS dynamic resource manager

[mark gross]

On Mon, Jun 07, 2010 at 05:18:25PM -0700, Mike Chan wrote:
> On Tue, May 25, 2010 at 7:52 AM, mark gross <640e9920@gmail.com> wrote:
> > On Fri, May 21, 2010 at 03:37:39PM -0600, Chidambaram, Praveen wrote:
> >> Hi,
> >>
> >> I am looking for some extensible features for PM QoS. Specifically,
> >> dynamic runtime parameters that my architecture could define at
> >> runtime and the drivers can make use of.
> >>
> >> What are the plans in this regard?
> >
> > we've talked about it a few times in the past.
> >
> > It's come down to how do we prevent platform independent code from
> > depending on such platform independent dynamic prarams in a sane manner.
> >
> > I've alway's pushed for coming up with more reasonable abstractions that
> > could be re-used and adding them (like the bus thing) as opposed to
> > having drivers make / use new parrameters dynamically.
> >
> 
> Would it be possible to elaborate on the bus resource ideas you had? I
> couldn't find anything in the archives. Sounds like a similar that we
> (Android / embedded world) are having (or will) on multiple platforms
> (omap / msm/ tegra).
> 
> -- Mike

Its been 5 months since the last time this basic discussion happened,
and I'll do some searching for the thread later but;

But the bus thing we did post a patch to get the code aurora idea
included, but for some reason that patch didn't stick.
http://lkml.org/lkml/2010/4/22/213  (I guess the details are in the
archives)  I'm happy to re-visit it.

WRT ideas I've had, that went unsupported (well, rejected) by the domain
experts at that time, where around throttling the GPU and / or it access
to the memory it used.

Most of the ideas where around clock throttling of buses and memory.
Things like throttling the graphics BW to whatever memory it was using,
or clocking down the GPU when idle.  My thought was a pm_qos hint could
be handy, but I was assured that modern hardware auto-throttled and
didn't need this.  So I dropped it.  (At that time I also thought it
would apply to HD audio to allow better DAC throttling.  I also dropped
this one too for the same reasons.)

One problem was what type of units to use when talking about the qos of
the graphics was not obvious.  It needs to be something meaningful to
be portable or provide a good abstraction.

Bus bandwidth in KBS is ok for somethings but its not meaningful to an
application that wants to request a specific minimum graphics
performance.  As I type this I suppose a bandwidth for blts per
second would work for the current Android graphics but for os stacks
that more fully use the GL engine or games running on android I don't
think that its a meaningful metric for describing GPU performance
throttling. 

what ideas do you have?  I would like to seem more pm_qos users.

--mgross

Re: PM QoS dynamic resource manager

[Chidambaram, Praveen]

> Most of the ideas where around clock throttling of buses and memory.
> Things like throttling the graphics BW to whatever memory it was using,
> or clocking down the GPU when idle.  My thought was a pm_qos hint could
> be handy, but I was assured that modern hardware auto-throttled and
> didn't need this.  So I dropped it.  (At that time I also thought it
> would apply to HD audio to allow better DAC throttling.  I also dropped
> this one too for the same reasons.)

Yes, auto throttling by tying the Bus clock in relation to the CPU is 
a fairly good idea, but cannot be the only solution. The drivers know 
best their requirements and some drivers would need arbirated 
bandwidth that no one else can encroach on.

> One problem was what type of units to use when talking about the qos of
> the graphics was not obvious.  It needs to be something meaningful to  
> be portable or provide a good abstraction.

With PM QoS you can specify one parameter. However, a single parameter 
is quite insufficient. To specify bus bandwidth (say MBPS), we would 
need 2 dimensional values. Buses are getting complex and the arbiters 
that sit around them can make good decisions when they are fed the 
right data. The clock speed is just one of them. To allocate a 
bandwidth to a specific client, you would need to specify the amount 
of data to be transferred and when transferred, the amount of data in 
burst that need to be transferred would help in specifying the speed 
for each transaction. The clock speed can be deduced by the 
arbiter/driver from these two inputs.

> One problem was what type of units to use when talking about the qos of
> the graphics was not obvious.  It needs to be something meaningful to
> be portable or provide a good abstraction.

The units as we understand better are MBPS. They are something that 
drivers can calculate before they make a request and request the 
Bandwidth and the burst bandwidth to the arbiter.

What do you think about vector data inputs?

Thanks,
Praveen

Re: PM QoS dynamic resource manager

[mark gross]

On Tue, Jun 08, 2010 at 08:44:30AM -0600, Chidambaram, Praveen wrote:
> > Most of the ideas where around clock throttling of buses and memory.
> > Things like throttling the graphics BW to whatever memory it was using,
> > or clocking down the GPU when idle.  My thought was a pm_qos hint could
> > be handy, but I was assured that modern hardware auto-throttled and
> > didn't need this.  So I dropped it.  (At that time I also thought it
> > would apply to HD audio to allow better DAC throttling.  I also dropped
> > this one too for the same reasons.)
> 
> Yes, auto throttling by tying the Bus clock in relation to the CPU is 
> a fairly good idea, but cannot be the only solution. The drivers know 
> best their requirements and some drivers would need arbirated 
> bandwidth that no one else can encroach on.
> 
> > One problem was what type of units to use when talking about the qos of
> > the graphics was not obvious.  It needs to be something meaningful to  
> > be portable or provide a good abstraction.
> 
> With PM QoS you can specify one parameter. However, a single parameter 
> is quite insufficient. To specify bus bandwidth (say MBPS), we would 
> need 2 dimensional values. Buses are getting complex and the arbiters 
> that sit around them can make good decisions when they are fed the 
> right data. The clock speed is just one of them. To allocate a 
> bandwidth to a specific client, you would need to specify the amount 
> of data to be transferred and when transferred, the amount of data in 
> burst that need to be transferred would help in specifying the speed 
> for each transaction. The clock speed can be deduced by the 
> arbiter/driver from these two inputs.
> 
> > One problem was what type of units to use when talking about the qos of
> > the graphics was not obvious.  It needs to be something meaningful to
> > be portable or provide a good abstraction.
> 
> The units as we understand better are MBPS. They are something that 
> drivers can calculate before they make a request and request the 
> Bandwidth and the burst bandwidth to the arbiter.
> 
> What do you think about vector data inputs?
>
I think functional (as in functional programming) as pm_qos constraints
are an interesting science experiment and, mostly doable for simple
functions.  But on face level feel pretty complicated to me, and need a
good, easy to understand application or example to get the idea across.

I can imagine some multi-dimensional constraints, (mostly in the space of
limited power budgets and graceful degradation under limited load /
brown out situations.)  But that stuff is not QOS as much as an
optimization zero sum game problem.

So I think vector data inputs into a qos parameter needs clear
motivation.   yet interesting to think about.

--mgross


 

Re: PM QoS dynamic resource manager

[Mike Chan]

On Mon, Jun 7, 2010 at 8:33 PM, mark gross <640e9920@gmail.com> wrote:
> On Mon, Jun 07, 2010 at 05:18:25PM -0700, Mike Chan wrote:
>> On Tue, May 25, 2010 at 7:52 AM, mark gross <640e9920@gmail.com> wrote:
>> > On Fri, May 21, 2010 at 03:37:39PM -0600, Chidambaram, Praveen wrote:
>> >> Hi,
>> >>
>> >> I am looking for some extensible features for PM QoS. Specifically,
>> >> dynamic runtime parameters that my architecture could define at
>> >> runtime and the drivers can make use of.
>> >>
>> >> What are the plans in this regard?
>> >
>> > we've talked about it a few times in the past.
>> >
>> > It's come down to how do we prevent platform independent code from
>> > depending on such platform independent dynamic prarams in a sane manner.
>> >
>> > I've alway's pushed for coming up with more reasonable abstractions that
>> > could be re-used and adding them (like the bus thing) as opposed to
>> > having drivers make / use new parrameters dynamically.
>> >
>>
>> Would it be possible to elaborate on the bus resource ideas you had? I
>> couldn't find anything in the archives. Sounds like a similar that we
>> (Android / embedded world) are having (or will) on multiple platforms
>> (omap / msm/ tegra).
>>
>> -- Mike
>
> Its been 5 months since the last time this basic discussion happened,
> and I'll do some searching for the thread later but;
>
> But the bus thing we did post a patch to get the code aurora idea
> included, but for some reason that patch didn't stick.
> http://lkml.org/lkml/2010/4/22/213  (I guess the details are in the
> archives)  I'm happy to re-visit it.
>

Interesting patch, it looks like having a "system wide bus" doesn't
easily apply to msm and tegra platforms.
An example of some things I would like to be able to control are i2c
and memory bus.

I'm tempted to suggest adding two types memory and i2c but I'm not
sure how future proof this will be given the growing complexity in the
embedded hardware road-map.
What about the possibility of registering not one but several buses?
You could add a bus qos param, with a type enum, or bind to some
platform_driver or bus_driver

Then there's the issue of having to deal with platform specific buses,
do you add this type to pm qos with only one user? Or have some
platform bus types defined somewhere. The generic code of min / max
for resource X can be useful so everyone doesn't spin their own
resource framework in their own architecture.

-- Mike

> WRT ideas I've had, that went unsupported (well, rejected) by the domain
> experts at that time, where around throttling the GPU and / or it access
> to the memory it used.
>
> Most of the ideas where around clock throttling of buses and memory.
> Things like throttling the graphics BW to whatever memory it was using,
> or clocking down the GPU when idle.  My thought was a pm_qos hint could
> be handy, but I was assured that modern hardware auto-throttled and
> didn't need this.  So I dropped it.  (At that time I also thought it
> would apply to HD audio to allow better DAC throttling.  I also dropped
> this one too for the same reasons.)
>
> One problem was what type of units to use when talking about the qos of
> the graphics was not obvious.  It needs to be something meaningful to
> be portable or provide a good abstraction.
>
> Bus bandwidth in KBS is ok for somethings but its not meaningful to an
> application that wants to request a specific minimum graphics
> performance.  As I type this I suppose a bandwidth for blts per
> second would work for the current Android graphics but for os stacks
> that more fully use the GL engine or games running on android I don't
> think that its a meaningful metric for describing GPU performance
> throttling.
>
> what ideas do you have?  I would like to seem more pm_qos users.
>
> --mgross
>
>

Re: PM QoS dynamic resource manager

[Bryan Huntsman]

>> http://lkml.org/lkml/2010/4/22/213  (I guess the details are in the
>> archives)  I'm happy to re-visit it.
>>
> 
> Interesting patch, it looks like having a "system wide bus" doesn't
> easily apply to msm and tegra platforms.
> An example of some things I would like to be able to control are i2c
> and memory bus.
> 
> I'm tempted to suggest adding two types memory and i2c but I'm not
> sure how future proof this will be given the growing complexity in the
> embedded hardware road-map.
> What about the possibility of registering not one but several buses?
> You could add a bus qos param, with a type enum, or bind to some
> platform_driver or bus_driver
> 
> Then there's the issue of having to deal with platform specific buses,
> do you add this type to pm qos with only one user? Or have some
> platform bus types defined somewhere. The generic code of min / max
> for resource X can be useful so everyone doesn't spin their own
> resource framework in their own architecture.
> 
> -- Mike

Mike, one idea I'm exploring is having platform-specific busses with QoS 
constraints specified via runtime_pm as part of the LDM.  Adding dynamic 
class creation within pm_qos, or a type enum as you suggest, would work. 
  However, I think this kind of behavior would fit nicely within runtime_pm.

- Bryan

Re: PM QoS dynamic resource manager

[Mike Chan]

On Tue, Jun 8, 2010 at 4:03 PM, Bryan Huntsman <bryanh@codeaurora.org> wrote:
>>> http://lkml.org/lkml/2010/4/22/213  (I guess the details are in the
>>> archives)  I'm happy to re-visit it.
>>>
>>
>> Interesting patch, it looks like having a "system wide bus" doesn't
>> easily apply to msm and tegra platforms.
>> An example of some things I would like to be able to control are i2c
>> and memory bus.
>>
>> I'm tempted to suggest adding two types memory and i2c but I'm not
>> sure how future proof this will be given the growing complexity in the
>> embedded hardware road-map.
>> What about the possibility of registering not one but several buses?
>> You could add a bus qos param, with a type enum, or bind to some
>> platform_driver or bus_driver
>>
>> Then there's the issue of having to deal with platform specific buses,
>> do you add this type to pm qos with only one user? Or have some
>> platform bus types defined somewhere. The generic code of min / max
>> for resource X can be useful so everyone doesn't spin their own
>> resource framework in their own architecture.
>>
>> -- Mike
>
> Mike, one idea I'm exploring is having platform-specific busses with QoS
> constraints specified via runtime_pm as part of the LDM.  Adding dynamic
> class creation within pm_qos, or a type enum as you suggest, would work.
>  However, I think this kind of behavior would fit nicely within runtime_pm.
>

I'm not too familiar with the current work in runtime pm and LDM.
However platform specific buses sounds like a good thing, at least
more future proof. This works in the embedded SOC world but I'm
wondering what happens when you have re-configurable hardware and now
your same peripheral is sourcing a different bus?

Does runtime pm hook into pm qos similar to how cpuidle uses pm qos?
So the platform-specific buses would be aa pm rumtime or a pm qos
addition?

-- Mike

> - Bryan
>

Re: PM QoS dynamic resource manager

[Bryan Huntsman]

>> Mike, one idea I'm exploring is having platform-specific busses with QoS
>> constraints specified via runtime_pm as part of the LDM.  Adding dynamic
>> class creation within pm_qos, or a type enum as you suggest, would work.
>>  However, I think this kind of behavior would fit nicely within runtime_pm.
>>
> 
> I'm not too familiar with the current work in runtime pm and LDM.
> However platform specific buses sounds like a good thing, at least
> more future proof. This works in the embedded SOC world but I'm
> wondering what happens when you have re-configurable hardware and now
> your same peripheral is sourcing a different bus?

My current thought is to have the board file enumerate the device on the 
proper bus.  This configuration is target-specific.  Each driver would 
register on as many buses as it needs to.  Some of the driver/bus 
registrations would be extraneous but the driver/device binding would 
only be possible on one bus.

> Does runtime pm hook into pm qos similar to how cpuidle uses pm qos?
> So the platform-specific buses would be aa pm rumtime or a pm qos
> addition?
> 
> -- Mike

Not to my knowledge.  I'm considering modeling the runtime_pm 
enhancements after pm_qos.  Meaning, there would be additional 
add/update/remove_requirements() callbacks added to struct dev_pm_ops 
for runtime_pm.  Instead of having a system-wide QoS sink per QoS class, 
the requirements would be passed up the LDM tree.  This way, a bus 
driver would be able to receive QoS constraints from all of it's devices 
and, combined with the active state of it's children from runtime_pm, be 
able to do some useful power management.  The entire bus could be 
throttled or idled in this manner.  This scheme would require the SW 
device topology to match the HW topology and not just have everything 
hang off the platform bus.

- Bryan

Re: PM QoS dynamic resource manager

[mark gross]

On Tue, Jun 08, 2010 at 04:03:20PM -0700, Bryan Huntsman wrote:
> >>http://lkml.org/lkml/2010/4/22/213  (I guess the details are in the
> >>archives)  I'm happy to re-visit it.
> >>
> >
> >Interesting patch, it looks like having a "system wide bus" doesn't
> >easily apply to msm and tegra platforms.
> >An example of some things I would like to be able to control are i2c
> >and memory bus.
> >
> >I'm tempted to suggest adding two types memory and i2c but I'm not
> >sure how future proof this will be given the growing complexity in the
> >embedded hardware road-map.
> >What about the possibility of registering not one but several buses?
> >You could add a bus qos param, with a type enum, or bind to some
> >platform_driver or bus_driver
> >
> >Then there's the issue of having to deal with platform specific buses,
> >do you add this type to pm qos with only one user? Or have some
> >platform bus types defined somewhere. The generic code of min / max
> >for resource X can be useful so everyone doesn't spin their own
> >resource framework in their own architecture.
> >
> >-- Mike
> 
> Mike, one idea I'm exploring is having platform-specific busses with
> QoS constraints specified via runtime_pm as part of the LDM.  Adding
> dynamic class creation within pm_qos, or a type enum as you suggest,
> would work.  However, I think this kind of behavior would fit nicely
> within runtime_pm.
>

Something like that is what Kevin Hilman was thinking too.  It would
bring a qos concept to the LDM for each bus driver object.  Picking what
qos parameters to use (I recommend latency and bandwidth) and how "local"
the parameters effect of these bus_qos interfaces are.  

They are not the same as the more global system wide pm_qos parameters,
and they would be unlikely things ever exposed to usermode.

Yes I think something like this is inevitable and will happen.  But we
need some good applications to roll out the idea with.  (I think.)

--mgross
 

Re: PM QoS dynamic resource manager

[Mike Chan]

On Tue, Jun 8, 2010 at 8:05 PM, mark gross <640e9920@gmail.com> wrote:
> On Tue, Jun 08, 2010 at 04:03:20PM -0700, Bryan Huntsman wrote:
>> >>http://lkml.org/lkml/2010/4/22/213  (I guess the details are in the
>> >>archives)  I'm happy to re-visit it.
>> >>
>> >
>> >Interesting patch, it looks like having a "system wide bus" doesn't
>> >easily apply to msm and tegra platforms.
>> >An example of some things I would like to be able to control are i2c
>> >and memory bus.
>> >
>> >I'm tempted to suggest adding two types memory and i2c but I'm not
>> >sure how future proof this will be given the growing complexity in the
>> >embedded hardware road-map.
>> >What about the possibility of registering not one but several buses?
>> >You could add a bus qos param, with a type enum, or bind to some
>> >platform_driver or bus_driver
>> >
>> >Then there's the issue of having to deal with platform specific buses,
>> >do you add this type to pm qos with only one user? Or have some
>> >platform bus types defined somewhere. The generic code of min / max
>> >for resource X can be useful so everyone doesn't spin their own
>> >resource framework in their own architecture.
>> >
>> >-- Mike
>>
>> Mike, one idea I'm exploring is having platform-specific busses with
>> QoS constraints specified via runtime_pm as part of the LDM.  Adding
>> dynamic class creation within pm_qos, or a type enum as you suggest,
>> would work.  However, I think this kind of behavior would fit nicely
>> within runtime_pm.
>>
>
> Something like that is what Kevin Hilman was thinking too.  It would
> bring a qos concept to the LDM for each bus driver object.  Picking what
> qos parameters to use (I recommend latency and bandwidth) and how "local"
> the parameters effect of these bus_qos interfaces are.
>

Are you thinking of having a (possible) pm qos constraint for each
struct device_driver? Or struct bus_type ? This would probably work if
for something like i2c. I'm not sure how this would work for memory
bus. If you did not want to tie memory bus performance to cpu speeds,
since (at least from what I"ve seen in omap / msm / tegra) there's no
device_driver for a memory bus clock, but I could be wrong so someone
correct me if I'm mistaken.

Typically I've seen (on msm / tegra / omap) if cpu is running at
frequency X, then set mem bus clock to Y. Which leads to a bunch of
hacks with drivers requesting frequency X, when really they need the
faster memory speed.

Perhaps both per bus-type pm qos parameter as well as a new global
memory bus (per cpu for numa systems?) parameter.

I'm worried about trying to over-engineer a solution here for
non-existing (or non-interested) customers. Ideally something that
will fit our needs with Android on msm / omap / tegra platforms but
still flexible enough for non-SOC.

-- Mike

> They are not the same as the more global system wide pm_qos parameters,
> and they would be unlikely things ever exposed to usermode.
>

I don't think we care about exposing this to userspace for our needs.

> Yes I think something like this is inevitable and will happen.  But we
> need some good applications to roll out the idea with.  (I think.)
>



> --mgross
>
>

Re: PM QoS dynamic resource manager

[Bryan Huntsman]

> Are you thinking of having a (possible) pm qos constraint for each
> struct device_driver? Or struct bus_type ? This would probably work if
> for something like i2c. I'm not sure how this would work for memory
> bus. If you did not want to tie memory bus performance to cpu speeds,
> since (at least from what I"ve seen in omap / msm / tegra) there's no
> device_driver for a memory bus clock, but I could be wrong so someone
> correct me if I'm mistaken.

I'm imagining QoS classes similar to pm_qos.  Bandwidth, cpu latency, 
etc.  Each driver decides what to do for each type, if anything.  This 
gets passed up the tree and aggregated by a parent, which I think would 
be a bus driver but could be anyone who cares.  The QoS sink, for lack 
of a better term, decides what it wants to do with that data.  For a 
memory bus, it would be possible to add up all bandwidth requests and 
scale the bus clock accordingly.  It could also look at the active 
status of it's children, decide that the bus is inactive, and completely 
shut it off.  This is where having arch-specific bus drivers is 
necessary.  What each QoS sink does will vary drastically from platform 
to platform, but the input data should be somewhat consistent.  That's 
why I'm looking into extending runtime_pm in this manner.

> Typically I've seen (on msm / tegra / omap) if cpu is running at
> frequency X, then set mem bus clock to Y. Which leads to a bunch of
> hacks with drivers requesting frequency X, when really they need the
> faster memory speed.

Picking the right QoS constraints will always be a problem.  It's 
possible to abuse pm_qos today in the same manner.

> Perhaps both per bus-type pm qos parameter as well as a new global
> memory bus (per cpu for numa systems?) parameter.
> 
> I'm worried about trying to over-engineer a solution here for
> non-existing (or non-interested) customers. Ideally something that
> will fit our needs with Android on msm / omap / tegra platforms but
> still flexible enough for non-SOC.
> 
> -- Mike

In general, I share your dislike for over-engineering.  This doesn't 
feel like that quite yet.  From a high level at least, it would seem to 
fit in nicely with runtime_pm.  From Mark's comments it sounds like 
others are having similar thoughts.

- Bryan

Re: PM QoS dynamic resource manager

[mark gross]

On Wed, Jun 09, 2010 at 05:53:53PM -0700, Mike Chan wrote:
> On Tue, Jun 8, 2010 at 8:05 PM, mark gross <640e9920@gmail.com> wrote:
> > On Tue, Jun 08, 2010 at 04:03:20PM -0700, Bryan Huntsman wrote:
> >> >>http://lkml.org/lkml/2010/4/22/213  (I guess the details are in the
> >> >>archives)  I'm happy to re-visit it.
> >> >>
> >> >
> >> >Interesting patch, it looks like having a "system wide bus" doesn't
> >> >easily apply to msm and tegra platforms.
> >> >An example of some things I would like to be able to control are i2c
> >> >and memory bus.
> >> >
> >> >I'm tempted to suggest adding two types memory and i2c but I'm not
> >> >sure how future proof this will be given the growing complexity in the
> >> >embedded hardware road-map.
> >> >What about the possibility of registering not one but several buses?
> >> >You could add a bus qos param, with a type enum, or bind to some
> >> >platform_driver or bus_driver
> >> >
> >> >Then there's the issue of having to deal with platform specific buses,
> >> >do you add this type to pm qos with only one user? Or have some
> >> >platform bus types defined somewhere. The generic code of min / max
> >> >for resource X can be useful so everyone doesn't spin their own
> >> >resource framework in their own architecture.
> >> >
> >> >-- Mike
> >>
> >> Mike, one idea I'm exploring is having platform-specific busses with
> >> QoS constraints specified via runtime_pm as part of the LDM.  Adding
> >> dynamic class creation within pm_qos, or a type enum as you suggest,
> >> would work.  However, I think this kind of behavior would fit nicely
> >> within runtime_pm.
> >>
> >
> > Something like that is what Kevin Hilman was thinking too.  It would
> > bring a qos concept to the LDM for each bus driver object.  Picking what
> > qos parameters to use (I recommend latency and bandwidth) and how "local"
> > the parameters effect of these bus_qos interfaces are.
> >
> 
> Are you thinking of having a (possible) pm qos constraint for each
> struct device_driver? Or struct bus_type ? This would probably work if

I'm thinking ever struct bus_type at the moment, but really this is
somehting hat has been getting handwaving at so far.

> for something like i2c. I'm not sure how this would work for memory
> bus. If you did not want to tie memory bus performance to cpu speeds,
> since (at least from what I"ve seen in omap / msm / tegra) there's no
> device_driver for a memory bus clock, but I could be wrong so someone
> correct me if I'm mistaken.

it hasn't been thought out yet by me yet, WRT dependency expressions
that have more global scope than a given bus like what I think you are
talking about.

--mgross

> 
> Typically I've seen (on msm / tegra / omap) if cpu is running at
> frequency X, then set mem bus clock to Y. Which leads to a bunch of
> hacks with drivers requesting frequency X, when really they need the
> faster memory speed.
> 
> Perhaps both per bus-type pm qos parameter as well as a new global
> memory bus (per cpu for numa systems?) parameter.
> 
> I'm worried about trying to over-engineer a solution here for
> non-existing (or non-interested) customers. Ideally something that
> will fit our needs with Android on msm / omap / tegra platforms but
> still flexible enough for non-SOC.
> 
> -- Mike
> 
> > They are not the same as the more global system wide pm_qos parameters,
> > and they would be unlikely things ever exposed to usermode.
> >
> 
> I don't think we care about exposing this to userspace for our needs.
> 
> > Yes I think something like this is inevitable and will happen.  But we
> > need some good applications to roll out the idea with.  (I think.)
> >
> 
> 
> 
> > --mgross
> >
> >

Re: PM QoS dynamic resource manager

[Kevin Hilman]

On Tue, 2010-06-08 at 20:05 -0700, mark gross wrote:
> On Tue, Jun 08, 2010 at 04:03:20PM -0700, Bryan Huntsman wrote:
> > 
> > Mike, one idea I'm exploring is having platform-specific busses with
> > QoS constraints specified via runtime_pm as part of the LDM.  Adding
> > dynamic class creation within pm_qos, or a type enum as you suggest,
> > would work.  However, I think this kind of behavior would fit nicely
> > within runtime_pm.
> >
> 
> Something like that is what Kevin Hilman was thinking too.  It would
> bring a qos concept to the LDM for each bus driver object.  

Yes, I definitely have some thoughts/ideas in this direction and is what
we are doing on OMAP.  I've been on the road, so I will chime in more on
this next week when I get back...

Kevin

Re: PM QoS dynamic resource manager

[Kevin Hilman]

Kevin Hilman <khilman@deeprootsystems.com> writes:

> On Tue, 2010-06-08 at 20:05 -0700, mark gross wrote:
>> On Tue, Jun 08, 2010 at 04:03:20PM -0700, Bryan Huntsman wrote:
>> > 
>> > Mike, one idea I'm exploring is having platform-specific busses with
>> > QoS constraints specified via runtime_pm as part of the LDM.  Adding
>> > dynamic class creation within pm_qos, or a type enum as you suggest,
>> > would work.  However, I think this kind of behavior would fit nicely
>> > within runtime_pm.
>> >
>> 
>> Something like that is what Kevin Hilman was thinking too.  It would
>> bring a qos concept to the LDM for each bus driver object.  
>
> Yes, I definitely have some thoughts/ideas in this direction and is what
> we are doing on OMAP.  I've been on the road, so I will chime in more on
> this next week when I get back...

[apologies for the delay, travel has made it difficult to keep up with mail]

So below is an overview of how we are managing device-specific
latency/throughput on OMAP.  We are currently managing in an
OMAP-specific way, but I certainly see the value in generalizing this
into something managed at the 'struct device' level.

So this is obviously being used for device-specific constraints, but
on OMAP, we're also modeling the internal bus(ses) as special
"omap_device(s)" and we use constraints the bus "devices" for
throughput.

Bryan mentioned aggregating the constraints up to some parent where
platform-specific decisions can be made, and I am totally in support
of that.  On OMAP, we're aggregating the device latency constraints at
the time of a device idle (triggered by pm_runtime_idle().)  More
details on that in the below overview.

It would be great if we could all get together at one of the
conferences this year to talk through some of our ideas in more
detail.  I plan to be at ELC Europe (October in UK) and Linux Plumbers
(Nov in Boston) with a remote possibility of making it to LinuxCon
(Aug in Boston, which includes a PM mini-summit on Monday.)


-------------------------------------------------------------------

What we currently have in OMAP is a layer called omap_device.  A
struct omap_device contains a platform_device which contains a struct
device.

In the omap_device layer, each omap_device tracks it's actual wakeup
latency and also has a configurable maximum wakeup latency.  Drivers
or subsystems can set and change the maximum wakeup latency
dynamically, and the actual wakeup latency can be manually set when
the omap_device is registered (based on HW specs), or can be
dyamically measured at runtime (since we know how reliable said HW
specs can be.)

On OMAP, drivers usage of runtime PM will eventually trickle down to
calling omap_device_enable() or omap_device_idle().  These calls in
the omap_device API make the final decisions based comparing the
actual and requested wakeup latencies.

For example, runtime_pm_idle() will eventually call omap_device_idle()
for an omap_device.  In omap_device_idle(), if the wakeup latency of
that device is less than the currently requested maximum, the
'deactivate' hook for that latency level is called (which leads to the
HW-level magic to disable that device at the HW level.)  And of
course, if the wakeup latency for that device is longer than the
currently requested maximum, the deactivate hooks is not called.
Similarily on enable, if the devices current wakeup latency already
meets the requested maximum, nothing needs to change.  If it doesn't
the 'activate' hook is called so that it can meet the requested
maximum.

Thats a brief summary, but is a bit of an oversimplification.  The
actual omap_device code supports multiple latency levels
per-omap_device.  In both the _enable() and _idle() paths, the various
levels are traversed until one is found which meets the maximum wakeup
latency requirements.

Why would a device want multiple per-device latency levels?  As an
simple example, one level might simply be disabling the devices
clocks.  Another might also disable the PLL generating those clocks.
Another might be turning off the voltage regulator supplying power to
that device, etc. etc.  For each latency level, there are
configurable, platform-specific, activate and deactivate hooks that
can be used to do the actual work.

I hope that helps explain things, and feel free to peruse the code.
It's currently in mainline and part of the OMAP arch-specific code:

   arch/arm/plat-omap/omap_device.c
   arch/arm/plat-omap/include/plat/omap_device.h

Kevin

Re: PM QoS dynamic resource manager

[Huntsman, Bryan]

> It would be great if we could all get together at one of the
> conferences this year to talk through some of our ideas in more
> detail.  I plan to be at ELC Europe (October in UK) and Linux Plumbers
> (Nov in Boston) with a remote possibility of making it to LinuxCon
> (Aug in Boston, which includes a PM mini-summit on Monday.)

I'm planning on attending LinuxCon and Plumbers.  I'll also be up at the Symposium.  A meetup at the PM mini-summit would be great if Mike and Brian S can make it.  Thanks.

- Bryan

Re: PM QoS dynamic resource manager

[mark gross]

On Thu, Jun 17, 2010 at 10:57:46PM -0700, Huntsman, Bryan wrote:
> > It would be great if we could all get together at one of the
> > conferences this year to talk through some of our ideas in more
> > detail.  I plan to be at ELC Europe (October in UK) and Linux Plumbers
> > (Nov in Boston) with a remote possibility of making it to LinuxCon
> > (Aug in Boston, which includes a PM mini-summit on Monday.)
> 
> I'm planning on attending LinuxCon and Plumbers.  I'll also be up at
> the Symposium.  A meetup at the PM mini-summit would be great if Mike
> and Brian S can make it.  Thanks.
>

I just checked with my boss at work.  It looks like I can get travel to
Plumbers this fall.  Lets try to have a couple of PM tracts around this
and the excitement over suspend blocking at plumbers.


--mgross
 

Re: PM QoS dynamic resource manager

[mark gross]

On Thu, Jun 17, 2010 at 10:57:46PM -0700, Huntsman, Bryan wrote:
> > It would be great if we could all get together at one of the
> > conferences this year to talk through some of our ideas in more
> > detail.  I plan to be at ELC Europe (October in UK) and Linux Plumbers
> > (Nov in Boston) with a remote possibility of making it to LinuxCon
> > (Aug in Boston, which includes a PM mini-summit on Monday.)
> 
> I'm planning on attending LinuxCon and Plumbers.  I'll also be up at
> the Symposium.  A meetup at the PM mini-summit would be great if Mike
> and Brian S can make it.  Thanks.
> 
> - Bryan

I will be going to plumbers I think a working session to get this type
of generalization of qos-like ideas into the LDM would be an interesting
thing to try to do.  I won't be able to lead or drive this topic
however.  one of you guys need to make the proposal for the working
session.

http://www.linuxplumbersconf.org/2010/07/07/linux-plumbers-conference-call-for-working-sessions-submissions

oh, we are running up against a July 19 deadline.

--mgross

Re: PM QoS dynamic resource manager

[James Bottomley]

On Sun, 2010-07-18 at 05:57 -0700, mark gross wrote:
> On Thu, Jun 17, 2010 at 10:57:46PM -0700, Huntsman, Bryan wrote:
> > > It would be great if we could all get together at one of the
> > > conferences this year to talk through some of our ideas in more
> > > detail.  I plan to be at ELC Europe (October in UK) and Linux Plumbers
> > > (Nov in Boston) with a remote possibility of making it to LinuxCon
> > > (Aug in Boston, which includes a PM mini-summit on Monday.)
> > 
> > I'm planning on attending LinuxCon and Plumbers.  I'll also be up at
> > the Symposium.  A meetup at the PM mini-summit would be great if Mike
> > and Brian S can make it.  Thanks.
> > 
> > - Bryan
> 
> I will be going to plumbers I think a working session to get this type
> of generalization of qos-like ideas into the LDM would be an interesting
> thing to try to do.  I won't be able to lead or drive this topic
> however.  one of you guys need to make the proposal for the working
> session.
> 
> http://www.linuxplumbersconf.org/2010/07/07/linux-plumbers-conference-call-for-working-sessions-submissions
> 
> oh, we are running up against a July 19 deadline.

For working sessions, this isn't quite true ... the 19 July deadline is
only for Refereed Presentations (i.e. the standard conference talks).
The deadline for Working Sessions (Or MicroConferences) is up to the MC
Session Leader (although in practise, the earlier you submit, the better
your chances) reflecting the fact that Linux is fast moving and things
may change at the last minute.  If you want a separate MC on PM_QOS,
then submit now ... we only have slots for about 12 MCs at plumbers, and
once those are filled we won't really be able to add additional ones.

James

Re: PM QoS dynamic resource manager

[Rafael J. Wysocki]

On Sunday, July 18, 2010, James Bottomley wrote:
> On Sun, 2010-07-18 at 05:57 -0700, mark gross wrote:
> > On Thu, Jun 17, 2010 at 10:57:46PM -0700, Huntsman, Bryan wrote:
> > > > It would be great if we could all get together at one of the
> > > > conferences this year to talk through some of our ideas in more
> > > > detail.  I plan to be at ELC Europe (October in UK) and Linux Plumbers
> > > > (Nov in Boston) with a remote possibility of making it to LinuxCon
> > > > (Aug in Boston, which includes a PM mini-summit on Monday.)
> > > 
> > > I'm planning on attending LinuxCon and Plumbers.  I'll also be up at
> > > the Symposium.  A meetup at the PM mini-summit would be great if Mike
> > > and Brian S can make it.  Thanks.
> > > 
> > > - Bryan
> > 
> > I will be going to plumbers I think a working session to get this type
> > of generalization of qos-like ideas into the LDM would be an interesting
> > thing to try to do.  I won't be able to lead or drive this topic
> > however.  one of you guys need to make the proposal for the working
> > session.
> > 
> > http://www.linuxplumbersconf.org/2010/07/07/linux-plumbers-conference-call-for-working-sessions-submissions
> > 
> > oh, we are running up against a July 19 deadline.
> 
> For working sessions, this isn't quite true ... the 19 July deadline is
> only for Refereed Presentations (i.e. the standard conference talks).
> The deadline for Working Sessions (Or MicroConferences) is up to the MC
> Session Leader

OK, how do I figure out who's the one for the PM MC?

Rafael

Re: PM QoS dynamic resource manager

[James Bottomley]

On Mon, 2010-07-19 at 03:46 +0200, Rafael J. Wysocki wrote:
> On Sunday, July 18, 2010, James Bottomley wrote:
> > On Sun, 2010-07-18 at 05:57 -0700, mark gross wrote:
> > > On Thu, Jun 17, 2010 at 10:57:46PM -0700, Huntsman, Bryan wrote:
> > > > > It would be great if we could all get together at one of the
> > > > > conferences this year to talk through some of our ideas in more
> > > > > detail.  I plan to be at ELC Europe (October in UK) and Linux Plumbers
> > > > > (Nov in Boston) with a remote possibility of making it to LinuxCon
> > > > > (Aug in Boston, which includes a PM mini-summit on Monday.)
> > > > 
> > > > I'm planning on attending LinuxCon and Plumbers.  I'll also be up at
> > > > the Symposium.  A meetup at the PM mini-summit would be great if Mike
> > > > and Brian S can make it.  Thanks.
> > > > 
> > > > - Bryan
> > > 
> > > I will be going to plumbers I think a working session to get this type
> > > of generalization of qos-like ideas into the LDM would be an interesting
> > > thing to try to do.  I won't be able to lead or drive this topic
> > > however.  one of you guys need to make the proposal for the working
> > > session.
> > > 
> > > http://www.linuxplumbersconf.org/2010/07/07/linux-plumbers-conference-call-for-working-sessions-submissions
> > > 
> > > oh, we are running up against a July 19 deadline.
> > 
> > For working sessions, this isn't quite true ... the 19 July deadline is
> > only for Refereed Presentations (i.e. the standard conference talks).
> > The deadline for Working Sessions (Or MicroConferences) is up to the MC
> > Session Leader
> 
> OK, how do I figure out who's the one for the PM MC?

They're supposed to have Track Lead as their excerpt ... apparently
Matthew Garrett (who's the Session Leader) doesn't.

James

Re: PM QoS dynamic resource manager

[Matthew Garrett]

On Mon, Jul 19, 2010 at 08:38:33AM +0200, James Bottomley wrote:

> They're supposed to have Track Lead as their excerpt ... apparently
> Matthew Garrett (who's the Session Leader) doesn't.

Sorry about that - I've updated it.

I'll send out a mail about the kind of things I think would be helpful 
to discuss at Plumbers.

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: PM QoS dynamic resource manager

[Kevin Hilman]

On Thu, Jun 17, 2010 at 3:32 PM, Kevin Hilman
<khilman@deeprootsystems.com> wrote:
> Kevin Hilman <khilman@deeprootsystems.com> writes:
>
>> On Tue, 2010-06-08 at 20:05 -0700, mark gross wrote:
>>> On Tue, Jun 08, 2010 at 04:03:20PM -0700, Bryan Huntsman wrote:
>>> >
>>> > Mike, one idea I'm exploring is having platform-specific busses with
>>> > QoS constraints specified via runtime_pm as part of the LDM.  Adding
>>> > dynamic class creation within pm_qos, or a type enum as you suggest,
>>> > would work.  However, I think this kind of behavior would fit nicely
>>> > within runtime_pm.
>>> >
>>>
>>> Something like that is what Kevin Hilman was thinking too.  It would
>>> bring a qos concept to the LDM for each bus driver object.
>>

[...]

> It would be great if we could all get together at one of the
> conferences this year to talk through some of our ideas in more
> detail.  I plan to be at ELC Europe (October in UK) and Linux Plumbers
> (Nov in Boston) with a remote possibility of making it to LinuxCon
> (Aug in Boston, which includes a PM mini-summit on Monday.)

FYI... I've added a proposal to the PM mini-conf at Plumbers to have
more detailed discussion on per-device constraints:

   http://www.linuxplumbersconf.org/2010/ocw/events/LPC2010MC/proposals

Kevin