* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-04 21:56 ` Stefano Stabellini
@ 2014-09-09 10:19 ` Oleksandr Dmytryshyn
2014-09-09 21:52 ` Stefano Stabellini
2014-09-09 10:32 ` Ian Campbell
2014-09-09 13:25 ` Vitaly Chernooky
2 siblings, 1 reply; 57+ messages in thread
From: Oleksandr Dmytryshyn @ 2014-09-09 10:19 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Ian Campbell, jinsong.liu, Tim Deegan, xen-devel, Jan Beulich,
jacob.shin, Andrii Tseglytskyi
On Fri, Sep 5, 2014 at 12:56 AM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> On Thu, 4 Sep 2014, Oleksandr Dmytryshyn wrote:
>> Hi to all.
>>
>> I want to implement cpufreq driver in the next way:
>> 1. Cpufreq governor will be implemented in the Xen
>> 2. dom0 will only change cpu frequency and voltage of the physical cpus
>>
>> But there are some nuances:
>> 1. dom0 driver should read an information about operation points
>> (frequencies and voltages) and cpu supply source from the device tree for each
>> physical cpu. In the omap processor case this driver suspects that
>> those settings
>> located in the /cpus/cpu@0/ node. But hypervisor creates an cpu node
>> for each vcpu
>> for kernel dom0 in the device tree and those information is lost in the dom0.
>> 2. What about this case if we will have some physical cpus with different
>> operation points (for example 2 cpus) and we give only one cpu for dom0?
>>
>> How should I transfer all information from the original cpu@0..cpu@n nodes
>> about all physical cpus to the kernel dom0 driver? Maybe an additional
>> nodes should be created by the hypervisor in the device tree for dom0
>> and named as pcpu@0..pcpu@n?
>
> If we do that, wouldn't we require changes to the core OMAP drivers or
> cpu initialization code in Linux (to parse "pcpu" instead of "cpu"
> nodes)? I don't expect they would be easy to upstream or maintain going
> forward.
>
> I am trying to think of an alternative, such as passing the real cpu
> nodes to dom0 but then adding status = "disabled", but I am not sure
> whether Linux checks the status for cpu nodes. In addition this scheme
> wouldn't support the case where dom0 has more vcpus than pcpus on the
> system. Granted it is not very common and might even be detrimental for
> performances, but we should be able to support it.
>
> Ian, what do you think about this?
>
>
>
>> Oleksandr Dmytryshyn | Product Engineering and Development
>> GlobalLogic
>> M +38.067.382.2525
>> www.globallogic.com
>>
>> http://www.globallogic.com/email_disclaimer.txt
>>
>>
>> On Tue, Sep 2, 2014 at 9:46 PM, Andrii Tseglytskyi
>> <andrii.tseglytskyi@globallogic.com> wrote:
>> >
>> > Hi Stefano,
>> >
>> > Thank you for explanation.
>> > I think this requires more and deeper investigation, but for sure dom0
>> > must be able to do this.
>> > Let us investigate this.
>> >
>> > Thank you,
>> >
>> > Regards,
>> > Andrii
>> >
>> > On Tue, Sep 2, 2014 at 9:39 PM, Stefano Stabellini
>> > <stefano.stabellini@eu.citrix.com> wrote:
>> > > On Tue, 2 Sep 2014, Andrii Tseglytskyi wrote:
>> > >> On Tue, Sep 2, 2014 at 4:00 AM, Stefano Stabellini
>> > >> <stefano.stabellini@eu.citrix.com> wrote:
>> > >> > On Fri, 29 Aug 2014, Andrii Tseglytskyi wrote:
>> > >> >> Hi,
>> > >> >>
>> > >> >> Stefano, Ian,
>> > >> >>
>> > >> >> Could you please clarify the following point:
>> > >> >>
>> > >> >> I agree that decision about frequency change should be taken by Xen
>> > >> >> hypervisor. But what about hardware frequency changing?
>> > >> >> In general when frequency changed to bigger value (for example from 1
>> > >> >> GHz to 1.5 GHz) for ARM kernels sequence looks like the following:
>> > >> >>
>> > >> >> 1) cpufreq governor decides that frequency should be changed. This
>> > >> >> decision is taken after analysing of CPU performance data taking in
>> > >> >> account governor policy.
>> > >> >> 2) cpufreq governor asks cpufreq driver about new frequency.
>> > >> >> 3) cpufreq driver compares current and target frequencies and asks
>> > >> >> cpufreq regulator about voltage change.
>> > >> >> 4) cpufreq regulator send i2c command to standalone microchip, which
>> > >> >> is responsible for voltage changing.
>> > >> >> 5) cpufreq driver asks clock framework about new frequency for CPU clock
>> > >> >> 6) clock framework performs frequency sanity checks, taking in account
>> > >> >> clock parents and clock divider settings, and call platform specific
>> > >> >> "set_frequency" callback.
>> > >> >> 7) platform specific callback performs proper HW registers
>> > >> >> configuration for newly selected frequency
>> > >> >>
>> > >> >> Also there are some special cases - for example for OMAP5+ when
>> > >> >> frequency is changed to 1.5 GHz+, two additional HW IPs should be
>> > >> >> triggered (ABB and DCC, if someone is familiar with OMAP5+ )
>> > >> >>
>> > >> >> So, for generic ARM kernel we have 3 entities to change frequency:
>> > >> >>
>> > >> >> - cpufreq governor
>> > >> >> - cpufreq driver
>> > >> >> - cpufreq regulator
>> > >> >>
>> > >> >> + 2 additional IP for OMAP5+
>> > >> >> - ABB
>> > >> >> - DCC
>> > >> >>
>> > >> >> Taking in account all above, it looks like it would be better to
>> > >> >> implement only Xen cpufreq governor. Xen will take a decision about
>> > >> >> new frequency, and kernel dom0 will perform other steps. Dom0 contains
>> > >> >> all generic and platform specific frameworks, needed for frequency
>> > >> >> changing.
>> > >> >>
>> > >> >> What do you think ?
>> > >> >
>> > >> > Keep in mind that the architecture must be able to handle the case where
>> > >> > dom0 has only 1 or 2 vcpus on a 4 or 8 cores system with multiple
>> > >> > physical cpus.
>> > >> > Could dom0 change the frequency of a physical core or a physical cpu is
>> > >> > not even running on? If that is not a problem, because cpus and
>> > >> > frequency changing are decoupled enough in Linux to allow it, then I am
>> > >> > OK with it. But I suspect they are not.
>> > >> >
>> > >>
>> > >> Not sure that I got your point correctly - dom0 will change frequency
>> > >> on physical CPU.
>> > >> And in case of OMAP - this changing affects on both ARM physical cpus
>> > >> - changing is coupled.
>> > >> In case of other ARM platforms - changing may be not coupled (I've
>> > >> heard that Snapdragon can change cpu freqs independently on each
>> > >> physical cpu)
>> > >
>> > > Let me explain with a concrete example.
>> > >
>> > > Let's suppose that the platform has 2 physical cpus, each cpu has 4
>> > > cores. Let's also supposed that dom0 has only 2 vcpus, currently
>> > > running on core0 and core1 of cpu0.
>> > >
>> > > In this case would dom0 be able to change the frequency of core3 of
>> > > cpu1, given that is not even running on it?
>> > > If it can be done without any hacks, then we can go ahead with this
>> > > approach.
>> >
>> >
>> >
>> > --
>> >
>> > Andrii Tseglytskyi | Embedded Dev
>> > GlobalLogic
>> > www.globallogic.com
>>
Hi to all.
I've done next work to check how dom0 can change the frequency of the cpus.
1. I've written a small HACK in the hypervisor which copies all settings
from the physical cpu0 to the vcpu0.
Thus kernel can read information about OPP states and power supply.
2. I've turned on the cpufreq driver in kernel dom0.
3. I've turned on all needed regulators.
4. I've reverted a patch whitch disables CPUFREQ in kernel dom0 when
it is running under the xen.
Now kernel dom0 has 2 virtual cpus and board has 2 physical cpus.
CPUFREQ driver uses vcpus as source to calculate frequency and changes this
frequency on the physical cpus.
I've checked that in this case kernel dom0 can change the frequency of the cpus.
I want to disable CPUFREQ governor driver in dom0 and leave only cpu0-cpufreq
driver (driver for OMAP processors). CPUFREQ governor driver will be implemented
in the xen.
Here are some questions:
1. How implement an mechanism which allows the hypervisor to give
commands to the
CPUFREQ driver in the dom0 (i. e. 'set frequency')?
2. Could we use a limitation when the number of VCPUs is domain0 must
be equal to
the number of physical CPUs, and the domain0 VCPU must be pinned to
the respective
physical CPU? This limitation us used for the existing Domain0 based cpufreq.
3. If we will use this limitation can I simply copy all settings from physical
cpus to the appropriate vcpus in the device tree in xen? In this case
the existing
kernel code will not be modified and simple CPUFREQ driver will be written
which will receive commands from xen and redirect them to the cpu0-cpufreq
driver (or another registered cpufreq driver for any cpu).
Oleksandr Dmytryshyn | Product Engineering and Development
GlobalLogic
M +38.067.382.2525
www.globallogic.com
http://www.globallogic.com/email_disclaimer.txt
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-09 10:19 ` Oleksandr Dmytryshyn
@ 2014-09-09 21:52 ` Stefano Stabellini
0 siblings, 0 replies; 57+ messages in thread
From: Stefano Stabellini @ 2014-09-09 21:52 UTC (permalink / raw)
To: Oleksandr Dmytryshyn
Cc: Ian Campbell, Stefano Stabellini, jinsong.liu, Tim Deegan,
xen-devel, Jan Beulich, jacob.shin, Andrii Tseglytskyi
On Tue, 9 Sep 2014, Oleksandr Dmytryshyn wrote:
> On Fri, Sep 5, 2014 at 12:56 AM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
> > On Thu, 4 Sep 2014, Oleksandr Dmytryshyn wrote:
> >> Hi to all.
> >>
> >> I want to implement cpufreq driver in the next way:
> >> 1. Cpufreq governor will be implemented in the Xen
> >> 2. dom0 will only change cpu frequency and voltage of the physical cpus
> >>
> >> But there are some nuances:
> >> 1. dom0 driver should read an information about operation points
> >> (frequencies and voltages) and cpu supply source from the device tree for each
> >> physical cpu. In the omap processor case this driver suspects that
> >> those settings
> >> located in the /cpus/cpu@0/ node. But hypervisor creates an cpu node
> >> for each vcpu
> >> for kernel dom0 in the device tree and those information is lost in the dom0.
> >> 2. What about this case if we will have some physical cpus with different
> >> operation points (for example 2 cpus) and we give only one cpu for dom0?
> >>
> >> How should I transfer all information from the original cpu@0..cpu@n nodes
> >> about all physical cpus to the kernel dom0 driver? Maybe an additional
> >> nodes should be created by the hypervisor in the device tree for dom0
> >> and named as pcpu@0..pcpu@n?
> >
> > If we do that, wouldn't we require changes to the core OMAP drivers or
> > cpu initialization code in Linux (to parse "pcpu" instead of "cpu"
> > nodes)? I don't expect they would be easy to upstream or maintain going
> > forward.
> >
> > I am trying to think of an alternative, such as passing the real cpu
> > nodes to dom0 but then adding status = "disabled", but I am not sure
> > whether Linux checks the status for cpu nodes. In addition this scheme
> > wouldn't support the case where dom0 has more vcpus than pcpus on the
> > system. Granted it is not very common and might even be detrimental for
> > performances, but we should be able to support it.
> >
> > Ian, what do you think about this?
> >
> >
> >
> >> Oleksandr Dmytryshyn | Product Engineering and Development
> >> GlobalLogic
> >> M +38.067.382.2525
> >> www.globallogic.com
> >>
> >> http://www.globallogic.com/email_disclaimer.txt
> >>
> >>
> >> On Tue, Sep 2, 2014 at 9:46 PM, Andrii Tseglytskyi
> >> <andrii.tseglytskyi@globallogic.com> wrote:
> >> >
> >> > Hi Stefano,
> >> >
> >> > Thank you for explanation.
> >> > I think this requires more and deeper investigation, but for sure dom0
> >> > must be able to do this.
> >> > Let us investigate this.
> >> >
> >> > Thank you,
> >> >
> >> > Regards,
> >> > Andrii
> >> >
> >> > On Tue, Sep 2, 2014 at 9:39 PM, Stefano Stabellini
> >> > <stefano.stabellini@eu.citrix.com> wrote:
> >> > > On Tue, 2 Sep 2014, Andrii Tseglytskyi wrote:
> >> > >> On Tue, Sep 2, 2014 at 4:00 AM, Stefano Stabellini
> >> > >> <stefano.stabellini@eu.citrix.com> wrote:
> >> > >> > On Fri, 29 Aug 2014, Andrii Tseglytskyi wrote:
> >> > >> >> Hi,
> >> > >> >>
> >> > >> >> Stefano, Ian,
> >> > >> >>
> >> > >> >> Could you please clarify the following point:
> >> > >> >>
> >> > >> >> I agree that decision about frequency change should be taken by Xen
> >> > >> >> hypervisor. But what about hardware frequency changing?
> >> > >> >> In general when frequency changed to bigger value (for example from 1
> >> > >> >> GHz to 1.5 GHz) for ARM kernels sequence looks like the following:
> >> > >> >>
> >> > >> >> 1) cpufreq governor decides that frequency should be changed. This
> >> > >> >> decision is taken after analysing of CPU performance data taking in
> >> > >> >> account governor policy.
> >> > >> >> 2) cpufreq governor asks cpufreq driver about new frequency.
> >> > >> >> 3) cpufreq driver compares current and target frequencies and asks
> >> > >> >> cpufreq regulator about voltage change.
> >> > >> >> 4) cpufreq regulator send i2c command to standalone microchip, which
> >> > >> >> is responsible for voltage changing.
> >> > >> >> 5) cpufreq driver asks clock framework about new frequency for CPU clock
> >> > >> >> 6) clock framework performs frequency sanity checks, taking in account
> >> > >> >> clock parents and clock divider settings, and call platform specific
> >> > >> >> "set_frequency" callback.
> >> > >> >> 7) platform specific callback performs proper HW registers
> >> > >> >> configuration for newly selected frequency
> >> > >> >>
> >> > >> >> Also there are some special cases - for example for OMAP5+ when
> >> > >> >> frequency is changed to 1.5 GHz+, two additional HW IPs should be
> >> > >> >> triggered (ABB and DCC, if someone is familiar with OMAP5+ )
> >> > >> >>
> >> > >> >> So, for generic ARM kernel we have 3 entities to change frequency:
> >> > >> >>
> >> > >> >> - cpufreq governor
> >> > >> >> - cpufreq driver
> >> > >> >> - cpufreq regulator
> >> > >> >>
> >> > >> >> + 2 additional IP for OMAP5+
> >> > >> >> - ABB
> >> > >> >> - DCC
> >> > >> >>
> >> > >> >> Taking in account all above, it looks like it would be better to
> >> > >> >> implement only Xen cpufreq governor. Xen will take a decision about
> >> > >> >> new frequency, and kernel dom0 will perform other steps. Dom0 contains
> >> > >> >> all generic and platform specific frameworks, needed for frequency
> >> > >> >> changing.
> >> > >> >>
> >> > >> >> What do you think ?
> >> > >> >
> >> > >> > Keep in mind that the architecture must be able to handle the case where
> >> > >> > dom0 has only 1 or 2 vcpus on a 4 or 8 cores system with multiple
> >> > >> > physical cpus.
> >> > >> > Could dom0 change the frequency of a physical core or a physical cpu is
> >> > >> > not even running on? If that is not a problem, because cpus and
> >> > >> > frequency changing are decoupled enough in Linux to allow it, then I am
> >> > >> > OK with it. But I suspect they are not.
> >> > >> >
> >> > >>
> >> > >> Not sure that I got your point correctly - dom0 will change frequency
> >> > >> on physical CPU.
> >> > >> And in case of OMAP - this changing affects on both ARM physical cpus
> >> > >> - changing is coupled.
> >> > >> In case of other ARM platforms - changing may be not coupled (I've
> >> > >> heard that Snapdragon can change cpu freqs independently on each
> >> > >> physical cpu)
> >> > >
> >> > > Let me explain with a concrete example.
> >> > >
> >> > > Let's suppose that the platform has 2 physical cpus, each cpu has 4
> >> > > cores. Let's also supposed that dom0 has only 2 vcpus, currently
> >> > > running on core0 and core1 of cpu0.
> >> > >
> >> > > In this case would dom0 be able to change the frequency of core3 of
> >> > > cpu1, given that is not even running on it?
> >> > > If it can be done without any hacks, then we can go ahead with this
> >> > > approach.
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> > Andrii Tseglytskyi | Embedded Dev
> >> > GlobalLogic
> >> > www.globallogic.com
> >>
>
> Hi to all.
>
> I've done next work to check how dom0 can change the frequency of the cpus.
> 1. I've written a small HACK in the hypervisor which copies all settings
> from the physical cpu0 to the vcpu0.
> Thus kernel can read information about OPP states and power supply.
> 2. I've turned on the cpufreq driver in kernel dom0.
> 3. I've turned on all needed regulators.
> 4. I've reverted a patch whitch disables CPUFREQ in kernel dom0 when
> it is running under the xen.
>
> Now kernel dom0 has 2 virtual cpus and board has 2 physical cpus.
> CPUFREQ driver uses vcpus as source to calculate frequency and changes this
> frequency on the physical cpus.
>
> I've checked that in this case kernel dom0 can change the frequency of the cpus.
>
> I want to disable CPUFREQ governor driver in dom0 and leave only cpu0-cpufreq
> driver (driver for OMAP processors). CPUFREQ governor driver will be implemented
> in the xen.
>
> Here are some questions:
>
> 1. How implement an mechanism which allows the hypervisor to give
> commands to the
> CPUFREQ driver in the dom0 (i. e. 'set frequency')?
We could set up an event channel for that. Dom0 would receive an
evtchn_irq interrupt and check what needs to be done.
> 2. Could we use a limitation when the number of VCPUs is domain0 must
> be equal to
> the number of physical CPUs, and the domain0 VCPU must be pinned to
> the respective
> physical CPU? This limitation us used for the existing Domain0 based cpufreq.
I think that the limitation would drastically limit the usefulness of the
solution. I wouldn't want to have to resort to this.
Nowadays it is extremely common to have dom0 running with fewer vcpus
that the amount of pcpus in the system.
Unless there is a way to tell Linux to leave the vcpus idle all the time.
> 3. If we will use this limitation can I simply copy all settings from physical
> cpus to the appropriate vcpus in the device tree in xen? In this case
> the existing
> kernel code will not be modified and simple CPUFREQ driver will be written
> which will receive commands from xen and redirect them to the cpu0-cpufreq
> driver (or another registered cpufreq driver for any cpu).
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-04 21:56 ` Stefano Stabellini
2014-09-09 10:19 ` Oleksandr Dmytryshyn
@ 2014-09-09 10:32 ` Ian Campbell
2014-09-09 21:41 ` Stefano Stabellini
2014-09-09 13:25 ` Vitaly Chernooky
2 siblings, 1 reply; 57+ messages in thread
From: Ian Campbell @ 2014-09-09 10:32 UTC (permalink / raw)
To: Stefano Stabellini
Cc: jinsong.liu, Oleksandr Dmytryshyn, Tim Deegan, xen-devel,
Jan Beulich, jacob.shin, Andrii Tseglytskyi
On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
> I am trying to think of an alternative, such as passing the real cpu
> nodes to dom0 but then adding status = "disabled", but I am not sure
> whether Linux checks the status for cpu nodes.
status = "disabled" is defined to have a specific (i.e. non-default)
meaning for cpu nodes, Julien mentioned this when I tried to add a
similar patch to Xen to ignore them. I think it basically means "present
but not running, you should start them!".
> In addition this scheme
> wouldn't support the case where dom0 has more vcpus than pcpus on the
> system. Granted it is not very common and might even be detrimental for
> performances, but we should be able to support it.
It's a bit of an edge case, for sure. I guess it wouldn't be totally
unreasonable to say that if you use this sort of configuration you may
not get cpufreq support.
> Ian, what do you think about this?
All the options suck in one way or another AFAICT. I think we are going
to be looking for the least bad solution not necessarily a good one.
Fundamentally are we trying to avoid having to have a i2c subsystem etc
in the hypervisor to be be able to change the voltages before/after
changing the frequency?
We can't just say "that's part of the cpufreq driver" since different
boards using the same SoC might use different voltage regulators, over
i2c or some other bus etc, so we end up with a matrix.
It's arguable that we should be letting dom0 poke at that regulator
functionality anyway, at least not all of it. Taking that ability away
would necessarily imply more platform specific functionality in the
hypervisor.
How does this stuff work on x86?
Sorry, more questions than answers here.
Ian.
> > Oleksandr Dmytryshyn | Product Engineering and Development
> > GlobalLogic
> > M +38.067.382.2525
> > www.globallogic.com
> >
> > http://www.globallogic.com/email_disclaimer.txt
> >
> >
> > On Tue, Sep 2, 2014 at 9:46 PM, Andrii Tseglytskyi
> > <andrii.tseglytskyi@globallogic.com> wrote:
> > >
> > > Hi Stefano,
> > >
> > > Thank you for explanation.
> > > I think this requires more and deeper investigation, but for sure dom0
> > > must be able to do this.
> > > Let us investigate this.
> > >
> > > Thank you,
> > >
> > > Regards,
> > > Andrii
> > >
> > > On Tue, Sep 2, 2014 at 9:39 PM, Stefano Stabellini
> > > <stefano.stabellini@eu.citrix.com> wrote:
> > > > On Tue, 2 Sep 2014, Andrii Tseglytskyi wrote:
> > > >> On Tue, Sep 2, 2014 at 4:00 AM, Stefano Stabellini
> > > >> <stefano.stabellini@eu.citrix.com> wrote:
> > > >> > On Fri, 29 Aug 2014, Andrii Tseglytskyi wrote:
> > > >> >> Hi,
> > > >> >>
> > > >> >> Stefano, Ian,
> > > >> >>
> > > >> >> Could you please clarify the following point:
> > > >> >>
> > > >> >> I agree that decision about frequency change should be taken by Xen
> > > >> >> hypervisor. But what about hardware frequency changing?
> > > >> >> In general when frequency changed to bigger value (for example from 1
> > > >> >> GHz to 1.5 GHz) for ARM kernels sequence looks like the following:
> > > >> >>
> > > >> >> 1) cpufreq governor decides that frequency should be changed. This
> > > >> >> decision is taken after analysing of CPU performance data taking in
> > > >> >> account governor policy.
> > > >> >> 2) cpufreq governor asks cpufreq driver about new frequency.
> > > >> >> 3) cpufreq driver compares current and target frequencies and asks
> > > >> >> cpufreq regulator about voltage change.
> > > >> >> 4) cpufreq regulator send i2c command to standalone microchip, which
> > > >> >> is responsible for voltage changing.
> > > >> >> 5) cpufreq driver asks clock framework about new frequency for CPU clock
> > > >> >> 6) clock framework performs frequency sanity checks, taking in account
> > > >> >> clock parents and clock divider settings, and call platform specific
> > > >> >> "set_frequency" callback.
> > > >> >> 7) platform specific callback performs proper HW registers
> > > >> >> configuration for newly selected frequency
> > > >> >>
> > > >> >> Also there are some special cases - for example for OMAP5+ when
> > > >> >> frequency is changed to 1.5 GHz+, two additional HW IPs should be
> > > >> >> triggered (ABB and DCC, if someone is familiar with OMAP5+ )
> > > >> >>
> > > >> >> So, for generic ARM kernel we have 3 entities to change frequency:
> > > >> >>
> > > >> >> - cpufreq governor
> > > >> >> - cpufreq driver
> > > >> >> - cpufreq regulator
> > > >> >>
> > > >> >> + 2 additional IP for OMAP5+
> > > >> >> - ABB
> > > >> >> - DCC
> > > >> >>
> > > >> >> Taking in account all above, it looks like it would be better to
> > > >> >> implement only Xen cpufreq governor. Xen will take a decision about
> > > >> >> new frequency, and kernel dom0 will perform other steps. Dom0 contains
> > > >> >> all generic and platform specific frameworks, needed for frequency
> > > >> >> changing.
> > > >> >>
> > > >> >> What do you think ?
> > > >> >
> > > >> > Keep in mind that the architecture must be able to handle the case where
> > > >> > dom0 has only 1 or 2 vcpus on a 4 or 8 cores system with multiple
> > > >> > physical cpus.
> > > >> > Could dom0 change the frequency of a physical core or a physical cpu is
> > > >> > not even running on? If that is not a problem, because cpus and
> > > >> > frequency changing are decoupled enough in Linux to allow it, then I am
> > > >> > OK with it. But I suspect they are not.
> > > >> >
> > > >>
> > > >> Not sure that I got your point correctly - dom0 will change frequency
> > > >> on physical CPU.
> > > >> And in case of OMAP - this changing affects on both ARM physical cpus
> > > >> - changing is coupled.
> > > >> In case of other ARM platforms - changing may be not coupled (I've
> > > >> heard that Snapdragon can change cpu freqs independently on each
> > > >> physical cpu)
> > > >
> > > > Let me explain with a concrete example.
> > > >
> > > > Let's suppose that the platform has 2 physical cpus, each cpu has 4
> > > > cores. Let's also supposed that dom0 has only 2 vcpus, currently
> > > > running on core0 and core1 of cpu0.
> > > >
> > > > In this case would dom0 be able to change the frequency of core3 of
> > > > cpu1, given that is not even running on it?
> > > > If it can be done without any hacks, then we can go ahead with this
> > > > approach.
> > >
> > >
> > >
> > > --
> > >
> > > Andrii Tseglytskyi | Embedded Dev
> > > GlobalLogic
> > > www.globallogic.com
> >
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-09 10:32 ` Ian Campbell
@ 2014-09-09 21:41 ` Stefano Stabellini
2014-09-10 9:42 ` Ian Campbell
0 siblings, 1 reply; 57+ messages in thread
From: Stefano Stabellini @ 2014-09-09 21:41 UTC (permalink / raw)
To: Ian Campbell
Cc: Stefano Stabellini, jinsong.liu, Oleksandr Dmytryshyn, jacob.shin,
Tim Deegan, Jan Beulich, xen-devel, Andrii Tseglytskyi
On Tue, 9 Sep 2014, Ian Campbell wrote:
> On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
> > I am trying to think of an alternative, such as passing the real cpu
> > nodes to dom0 but then adding status = "disabled", but I am not sure
> > whether Linux checks the status for cpu nodes.
>
> status = "disabled" is defined to have a specific (i.e. non-default)
> meaning for cpu nodes, Julien mentioned this when I tried to add a
> similar patch to Xen to ignore them. I think it basically means "present
> but not running, you should start them!".
>
> > In addition this scheme
> > wouldn't support the case where dom0 has more vcpus than pcpus on the
> > system. Granted it is not very common and might even be detrimental for
> > performances, but we should be able to support it.
>
> It's a bit of an edge case, for sure. I guess it wouldn't be totally
> unreasonable to say that if you use this sort of configuration you may
> not get cpufreq support.
>
> > Ian, what do you think about this?
>
> All the options suck in one way or another AFAICT. I think we are going
> to be looking for the least bad solution not necessarily a good one.
>
> Fundamentally are we trying to avoid having to have a i2c subsystem etc
> in the hypervisor to be be able to change the voltages before/after
> changing the frequency?
>
> We can't just say "that's part of the cpufreq driver" since different
> boards using the same SoC might use different voltage regulators, over
> i2c or some other bus etc, so we end up with a matrix.
>
> It's arguable that we should be letting dom0 poke at that regulator
> functionality anyway, at least not all of it. Taking that ability away
> would necessarily imply more platform specific functionality in the
> hypervisor.
Right.
I am afraid that in order to avoid more code in Xen, we end up with an
unmaintainable interface and unupstreamable hacks in dom0.
> How does this stuff work on x86?
As far as I can tell, after an initial version based on Dom0 doing the
work, the functionality has been moved into the hypervisor.
Of course doing that is easier on x86 where differences across
platforms are more limited.
> Sorry, more questions than answers here.
>
> Ian.
>
> > > Oleksandr Dmytryshyn | Product Engineering and Development
> > > GlobalLogic
> > > M +38.067.382.2525
> > > www.globallogic.com
> > >
> > > http://www.globallogic.com/email_disclaimer.txt
> > >
> > >
> > > On Tue, Sep 2, 2014 at 9:46 PM, Andrii Tseglytskyi
> > > <andrii.tseglytskyi@globallogic.com> wrote:
> > > >
> > > > Hi Stefano,
> > > >
> > > > Thank you for explanation.
> > > > I think this requires more and deeper investigation, but for sure dom0
> > > > must be able to do this.
> > > > Let us investigate this.
> > > >
> > > > Thank you,
> > > >
> > > > Regards,
> > > > Andrii
> > > >
> > > > On Tue, Sep 2, 2014 at 9:39 PM, Stefano Stabellini
> > > > <stefano.stabellini@eu.citrix.com> wrote:
> > > > > On Tue, 2 Sep 2014, Andrii Tseglytskyi wrote:
> > > > >> On Tue, Sep 2, 2014 at 4:00 AM, Stefano Stabellini
> > > > >> <stefano.stabellini@eu.citrix.com> wrote:
> > > > >> > On Fri, 29 Aug 2014, Andrii Tseglytskyi wrote:
> > > > >> >> Hi,
> > > > >> >>
> > > > >> >> Stefano, Ian,
> > > > >> >>
> > > > >> >> Could you please clarify the following point:
> > > > >> >>
> > > > >> >> I agree that decision about frequency change should be taken by Xen
> > > > >> >> hypervisor. But what about hardware frequency changing?
> > > > >> >> In general when frequency changed to bigger value (for example from 1
> > > > >> >> GHz to 1.5 GHz) for ARM kernels sequence looks like the following:
> > > > >> >>
> > > > >> >> 1) cpufreq governor decides that frequency should be changed. This
> > > > >> >> decision is taken after analysing of CPU performance data taking in
> > > > >> >> account governor policy.
> > > > >> >> 2) cpufreq governor asks cpufreq driver about new frequency.
> > > > >> >> 3) cpufreq driver compares current and target frequencies and asks
> > > > >> >> cpufreq regulator about voltage change.
> > > > >> >> 4) cpufreq regulator send i2c command to standalone microchip, which
> > > > >> >> is responsible for voltage changing.
> > > > >> >> 5) cpufreq driver asks clock framework about new frequency for CPU clock
> > > > >> >> 6) clock framework performs frequency sanity checks, taking in account
> > > > >> >> clock parents and clock divider settings, and call platform specific
> > > > >> >> "set_frequency" callback.
> > > > >> >> 7) platform specific callback performs proper HW registers
> > > > >> >> configuration for newly selected frequency
> > > > >> >>
> > > > >> >> Also there are some special cases - for example for OMAP5+ when
> > > > >> >> frequency is changed to 1.5 GHz+, two additional HW IPs should be
> > > > >> >> triggered (ABB and DCC, if someone is familiar with OMAP5+ )
> > > > >> >>
> > > > >> >> So, for generic ARM kernel we have 3 entities to change frequency:
> > > > >> >>
> > > > >> >> - cpufreq governor
> > > > >> >> - cpufreq driver
> > > > >> >> - cpufreq regulator
> > > > >> >>
> > > > >> >> + 2 additional IP for OMAP5+
> > > > >> >> - ABB
> > > > >> >> - DCC
> > > > >> >>
> > > > >> >> Taking in account all above, it looks like it would be better to
> > > > >> >> implement only Xen cpufreq governor. Xen will take a decision about
> > > > >> >> new frequency, and kernel dom0 will perform other steps. Dom0 contains
> > > > >> >> all generic and platform specific frameworks, needed for frequency
> > > > >> >> changing.
> > > > >> >>
> > > > >> >> What do you think ?
> > > > >> >
> > > > >> > Keep in mind that the architecture must be able to handle the case where
> > > > >> > dom0 has only 1 or 2 vcpus on a 4 or 8 cores system with multiple
> > > > >> > physical cpus.
> > > > >> > Could dom0 change the frequency of a physical core or a physical cpu is
> > > > >> > not even running on? If that is not a problem, because cpus and
> > > > >> > frequency changing are decoupled enough in Linux to allow it, then I am
> > > > >> > OK with it. But I suspect they are not.
> > > > >> >
> > > > >>
> > > > >> Not sure that I got your point correctly - dom0 will change frequency
> > > > >> on physical CPU.
> > > > >> And in case of OMAP - this changing affects on both ARM physical cpus
> > > > >> - changing is coupled.
> > > > >> In case of other ARM platforms - changing may be not coupled (I've
> > > > >> heard that Snapdragon can change cpu freqs independently on each
> > > > >> physical cpu)
> > > > >
> > > > > Let me explain with a concrete example.
> > > > >
> > > > > Let's suppose that the platform has 2 physical cpus, each cpu has 4
> > > > > cores. Let's also supposed that dom0 has only 2 vcpus, currently
> > > > > running on core0 and core1 of cpu0.
> > > > >
> > > > > In this case would dom0 be able to change the frequency of core3 of
> > > > > cpu1, given that is not even running on it?
> > > > > If it can be done without any hacks, then we can go ahead with this
> > > > > approach.
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Andrii Tseglytskyi | Embedded Dev
> > > > GlobalLogic
> > > > www.globallogic.com
> > >
>
>
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-09 21:41 ` Stefano Stabellini
@ 2014-09-10 9:42 ` Ian Campbell
2014-09-10 10:19 ` Andrii Tseglytskyi
0 siblings, 1 reply; 57+ messages in thread
From: Ian Campbell @ 2014-09-10 9:42 UTC (permalink / raw)
To: Stefano Stabellini
Cc: jinsong.liu, Oleksandr Dmytryshyn, Tim Deegan, xen-devel,
Jan Beulich, jacob.shin, Andrii Tseglytskyi
On Tue, 2014-09-09 at 22:41 +0100, Stefano Stabellini wrote:
> On Tue, 9 Sep 2014, Ian Campbell wrote:
> > On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
> > > I am trying to think of an alternative, such as passing the real cpu
> > > nodes to dom0 but then adding status = "disabled", but I am not sure
> > > whether Linux checks the status for cpu nodes.
> >
> > status = "disabled" is defined to have a specific (i.e. non-default)
> > meaning for cpu nodes, Julien mentioned this when I tried to add a
> > similar patch to Xen to ignore them. I think it basically means "present
> > but not running, you should start them!".
> >
> > > In addition this scheme
> > > wouldn't support the case where dom0 has more vcpus than pcpus on the
> > > system. Granted it is not very common and might even be detrimental for
> > > performances, but we should be able to support it.
> >
> > It's a bit of an edge case, for sure. I guess it wouldn't be totally
> > unreasonable to say that if you use this sort of configuration you may
> > not get cpufreq support.
> >
> > > Ian, what do you think about this?
> >
> > All the options suck in one way or another AFAICT. I think we are going
> > to be looking for the least bad solution not necessarily a good one.
> >
> > Fundamentally are we trying to avoid having to have a i2c subsystem etc
> > in the hypervisor to be be able to change the voltages before/after
> > changing the frequency?
> >
> > We can't just say "that's part of the cpufreq driver" since different
> > boards using the same SoC might use different voltage regulators, over
> > i2c or some other bus etc, so we end up with a matrix.
> >
> > It's arguable that we should be letting dom0 poke at that regulator
> > functionality anyway, at least not all of it. Taking that ability away
> > would necessarily imply more platform specific functionality in the
> > hypervisor.
>
> Right.
> I am afraid that in order to avoid more code in Xen, we end up with an
> unmaintainable interface and unupstreamable hacks in dom0.
That's what I'm worried about to. Hence I'm wondering if we should just
do this in the hypervisor.
Although there are a myriad of them the parts used to do voltage control
tend to be fairly simple.
One concern I have is that i2c busses also tend to have other things on
them which dom0 might legitimately access (e.g. rtc), I'm not sure what
to suggest here.
> > How does this stuff work on x86?
>
> As far as I can tell, after an initial version based on Dom0 doing the
> work, the functionality has been moved into the hypervisor.
> Of course doing that is easier on x86 where differences across
> platforms are more limited.
This chimes with my limited understanding. I think we still need a dom0
component to feed stuff down from ACPI to let Xen do its work, is that
right? Or do we go under the hood and ignore ACPI PM somehow?
Ian.
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-10 9:42 ` Ian Campbell
@ 2014-09-10 10:19 ` Andrii Tseglytskyi
2014-09-10 18:35 ` Stefano Stabellini
0 siblings, 1 reply; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-09-10 10:19 UTC (permalink / raw)
To: Ian Campbell
Cc: Stefano Stabellini, jinsong.liu, Oleksandr Dmytryshyn, Tim Deegan,
xen-devel, Jan Beulich, jacob.shin
Hi,
On Wed, Sep 10, 2014 at 12:42 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>
> On Tue, 2014-09-09 at 22:41 +0100, Stefano Stabellini wrote:
> > On Tue, 9 Sep 2014, Ian Campbell wrote:
> > > On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
> > > > I am trying to think of an alternative, such as passing the real cpu
> > > > nodes to dom0 but then adding status = "disabled", but I am not sure
> > > > whether Linux checks the status for cpu nodes.
> > >
> > > status = "disabled" is defined to have a specific (i.e. non-default)
> > > meaning for cpu nodes, Julien mentioned this when I tried to add a
> > > similar patch to Xen to ignore them. I think it basically means "present
> > > but not running, you should start them!".
> > >
> > > > In addition this scheme
> > > > wouldn't support the case where dom0 has more vcpus than pcpus on the
> > > > system. Granted it is not very common and might even be detrimental for
> > > > performances, but we should be able to support it.
> > >
> > > It's a bit of an edge case, for sure. I guess it wouldn't be totally
> > > unreasonable to say that if you use this sort of configuration you may
> > > not get cpufreq support.
> > >
> > > > Ian, what do you think about this?
> > >
> > > All the options suck in one way or another AFAICT. I think we are going
> > > to be looking for the least bad solution not necessarily a good one.
> > >
> > > Fundamentally are we trying to avoid having to have a i2c subsystem etc
> > > in the hypervisor to be be able to change the voltages before/after
> > > changing the frequency?
> > >
> > > We can't just say "that's part of the cpufreq driver" since different
> > > boards using the same SoC might use different voltage regulators, over
> > > i2c or some other bus etc, so we end up with a matrix.
> > >
> > > It's arguable that we should be letting dom0 poke at that regulator
> > > functionality anyway, at least not all of it. Taking that ability away
> > > would necessarily imply more platform specific functionality in the
> > > hypervisor.
> >
> > Right.
> > I am afraid that in order to avoid more code in Xen, we end up with an
> > unmaintainable interface and unupstreamable hacks in dom0.
>
> That's what I'm worried about to. Hence I'm wondering if we should just
> do this in the hypervisor.
>
> Although there are a myriad of them the parts used to do voltage control
> tend to be fairly simple.
>
> One concern I have is that i2c busses also tend to have other things on
> them which dom0 might legitimately access (e.g. rtc), I'm not sure what
> to suggest here.
I would try to avoid i2c transactions in Xen. I2C driver is quite
complicated in Linux kernel. It consists of several parts - common
core + platform specific. I'm pretty sure Xen should not handle this.
I think that establishing of event channel for frequency changing is a
good idea. It would be good to try to implement this. In process of
implementation we will see what is need to be resolved. The only
question here is how to pass physical cpu to dom0.
Regarding x86.
I'm not sure but maybe ACPI interface encapsulate voltage changing as well?
Regards,
Andrii
--
Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-10 10:19 ` Andrii Tseglytskyi
@ 2014-09-10 18:35 ` Stefano Stabellini
2014-09-10 19:31 ` Konrad Rzeszutek Wilk
2014-09-11 9:43 ` Ian Campbell
0 siblings, 2 replies; 57+ messages in thread
From: Stefano Stabellini @ 2014-09-10 18:35 UTC (permalink / raw)
To: Andrii Tseglytskyi
Cc: Ian Campbell, Stefano Stabellini, jinsong.liu,
Oleksandr Dmytryshyn, Tim Deegan, xen-devel, Jan Beulich,
jacob.shin
On Wed, 10 Sep 2014, Andrii Tseglytskyi wrote:
> Hi,
>
> On Wed, Sep 10, 2014 at 12:42 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> >
> > On Tue, 2014-09-09 at 22:41 +0100, Stefano Stabellini wrote:
> > > On Tue, 9 Sep 2014, Ian Campbell wrote:
> > > > On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
> > > > > I am trying to think of an alternative, such as passing the real cpu
> > > > > nodes to dom0 but then adding status = "disabled", but I am not sure
> > > > > whether Linux checks the status for cpu nodes.
> > > >
> > > > status = "disabled" is defined to have a specific (i.e. non-default)
> > > > meaning for cpu nodes, Julien mentioned this when I tried to add a
> > > > similar patch to Xen to ignore them. I think it basically means "present
> > > > but not running, you should start them!".
> > > >
> > > > > In addition this scheme
> > > > > wouldn't support the case where dom0 has more vcpus than pcpus on the
> > > > > system. Granted it is not very common and might even be detrimental for
> > > > > performances, but we should be able to support it.
> > > >
> > > > It's a bit of an edge case, for sure. I guess it wouldn't be totally
> > > > unreasonable to say that if you use this sort of configuration you may
> > > > not get cpufreq support.
> > > >
> > > > > Ian, what do you think about this?
> > > >
> > > > All the options suck in one way or another AFAICT. I think we are going
> > > > to be looking for the least bad solution not necessarily a good one.
> > > >
> > > > Fundamentally are we trying to avoid having to have a i2c subsystem etc
> > > > in the hypervisor to be be able to change the voltages before/after
> > > > changing the frequency?
> > > >
> > > > We can't just say "that's part of the cpufreq driver" since different
> > > > boards using the same SoC might use different voltage regulators, over
> > > > i2c or some other bus etc, so we end up with a matrix.
> > > >
> > > > It's arguable that we should be letting dom0 poke at that regulator
> > > > functionality anyway, at least not all of it. Taking that ability away
> > > > would necessarily imply more platform specific functionality in the
> > > > hypervisor.
> > >
> > > Right.
> > > I am afraid that in order to avoid more code in Xen, we end up with an
> > > unmaintainable interface and unupstreamable hacks in dom0.
> >
> > That's what I'm worried about to. Hence I'm wondering if we should just
> > do this in the hypervisor.
> >
> > Although there are a myriad of them the parts used to do voltage control
> > tend to be fairly simple.
> >
> > One concern I have is that i2c busses also tend to have other things on
> > them which dom0 might legitimately access (e.g. rtc), I'm not sure what
> > to suggest here.
>
> I would try to avoid i2c transactions in Xen. I2C driver is quite
> complicated in Linux kernel. It consists of several parts - common
> core + platform specific. I'm pretty sure Xen should not handle this.
> I think that establishing of event channel for frequency changing is a
> good idea. It would be good to try to implement this. In process of
> implementation we will see what is need to be resolved.
OK, that's reasonable.
> The only question here is how to pass physical cpu to dom0.
We can use a device tree based interface to pass the information to
dom0, but requiring a number of dom0 vcpus equal to the number of
physical cpus and in addition to that having to pin the vcpus each to a
different pcpu is quite a stringent limitation. However I don't know the
frequency changing interfaces in Linux well enough to know how hard
would be to lift it.
> Regarding x86.
> I'm not sure but maybe ACPI interface encapsulate voltage changing as well?
I think so (but I am not an expert on that).
> Regards,
> Andrii
>
>
> --
>
> Andrii Tseglytskyi | Embedded Dev
> GlobalLogic
> www.globallogic.com
>
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-10 18:35 ` Stefano Stabellini
@ 2014-09-10 19:31 ` Konrad Rzeszutek Wilk
2014-09-16 13:49 ` Oleksandr Dmytryshyn
2014-09-11 9:43 ` Ian Campbell
1 sibling, 1 reply; 57+ messages in thread
From: Konrad Rzeszutek Wilk @ 2014-09-10 19:31 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Ian Campbell, jinsong.liu, Oleksandr Dmytryshyn, Tim Deegan,
xen-devel, Jan Beulich, jacob.shin, Andrii Tseglytskyi
On Wed, Sep 10, 2014 at 07:35:47PM +0100, Stefano Stabellini wrote:
> On Wed, 10 Sep 2014, Andrii Tseglytskyi wrote:
> > Hi,
> >
> > On Wed, Sep 10, 2014 at 12:42 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > >
> > > On Tue, 2014-09-09 at 22:41 +0100, Stefano Stabellini wrote:
> > > > On Tue, 9 Sep 2014, Ian Campbell wrote:
> > > > > On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
> > > > > > I am trying to think of an alternative, such as passing the real cpu
> > > > > > nodes to dom0 but then adding status = "disabled", but I am not sure
> > > > > > whether Linux checks the status for cpu nodes.
> > > > >
> > > > > status = "disabled" is defined to have a specific (i.e. non-default)
> > > > > meaning for cpu nodes, Julien mentioned this when I tried to add a
> > > > > similar patch to Xen to ignore them. I think it basically means "present
> > > > > but not running, you should start them!".
> > > > >
> > > > > > In addition this scheme
> > > > > > wouldn't support the case where dom0 has more vcpus than pcpus on the
> > > > > > system. Granted it is not very common and might even be detrimental for
> > > > > > performances, but we should be able to support it.
> > > > >
> > > > > It's a bit of an edge case, for sure. I guess it wouldn't be totally
> > > > > unreasonable to say that if you use this sort of configuration you may
> > > > > not get cpufreq support.
> > > > >
> > > > > > Ian, what do you think about this?
> > > > >
> > > > > All the options suck in one way or another AFAICT. I think we are going
> > > > > to be looking for the least bad solution not necessarily a good one.
> > > > >
> > > > > Fundamentally are we trying to avoid having to have a i2c subsystem etc
> > > > > in the hypervisor to be be able to change the voltages before/after
> > > > > changing the frequency?
> > > > >
> > > > > We can't just say "that's part of the cpufreq driver" since different
> > > > > boards using the same SoC might use different voltage regulators, over
> > > > > i2c or some other bus etc, so we end up with a matrix.
> > > > >
> > > > > It's arguable that we should be letting dom0 poke at that regulator
> > > > > functionality anyway, at least not all of it. Taking that ability away
> > > > > would necessarily imply more platform specific functionality in the
> > > > > hypervisor.
> > > >
> > > > Right.
> > > > I am afraid that in order to avoid more code in Xen, we end up with an
> > > > unmaintainable interface and unupstreamable hacks in dom0.
> > >
> > > That's what I'm worried about to. Hence I'm wondering if we should just
> > > do this in the hypervisor.
> > >
> > > Although there are a myriad of them the parts used to do voltage control
> > > tend to be fairly simple.
> > >
> > > One concern I have is that i2c busses also tend to have other things on
> > > them which dom0 might legitimately access (e.g. rtc), I'm not sure what
> > > to suggest here.
> >
> > I would try to avoid i2c transactions in Xen. I2C driver is quite
> > complicated in Linux kernel. It consists of several parts - common
> > core + platform specific. I'm pretty sure Xen should not handle this.
> > I think that establishing of event channel for frequency changing is a
> > good idea. It would be good to try to implement this. In process of
> > implementation we will see what is need to be resolved.
>
> OK, that's reasonable.
>
>
> > The only question here is how to pass physical cpu to dom0.
>
> We can use a device tree based interface to pass the information to
> dom0, but requiring a number of dom0 vcpus equal to the number of
> physical cpus and in addition to that having to pin the vcpus each to a
> different pcpu is quite a stringent limitation. However I don't know the
> frequency changing interfaces in Linux well enough to know how hard
> would be to lift it.
>
>
> > Regarding x86.
> > I'm not sure but maybe ACPI interface encapsulate voltage changing as well?
>
> I think so (but I am not an expert on that).
The usual states are P and C states. The P states is the closes to what you
are looking at:
struct acpi_processor_px {
u64 core_frequency; /* megahertz */
u64 power; /* milliWatts */
u64 transition_latency; /* microseconds */
u64 bus_master_latency; /* microseconds */
u64 control; /* control value */
u64 status; /* success indicator */
};
>
>
>
> > Regards,
> > Andrii
> >
> >
> > --
> >
> > Andrii Tseglytskyi | Embedded Dev
> > GlobalLogic
> > www.globallogic.com
> >
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 57+ messages in thread* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-10 19:31 ` Konrad Rzeszutek Wilk
@ 2014-09-16 13:49 ` Oleksandr Dmytryshyn
2014-09-17 17:06 ` Konrad Rzeszutek Wilk
2014-09-24 14:36 ` Stefano Stabellini
0 siblings, 2 replies; 57+ messages in thread
From: Oleksandr Dmytryshyn @ 2014-09-16 13:49 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk
Cc: Ian Campbell, Stefano Stabellini, jinsong.liu, Tim Deegan,
xen-devel, Jan Beulich, jacob.shin, Andrii Tseglytskyi
On Wed, Sep 10, 2014 at 10:31 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Wed, Sep 10, 2014 at 07:35:47PM +0100, Stefano Stabellini wrote:
>> On Wed, 10 Sep 2014, Andrii Tseglytskyi wrote:
>> > Hi,
>> >
>> > On Wed, Sep 10, 2014 at 12:42 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>> > >
>> > > On Tue, 2014-09-09 at 22:41 +0100, Stefano Stabellini wrote:
>> > > > On Tue, 9 Sep 2014, Ian Campbell wrote:
>> > > > > On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
>> > > > > > I am trying to think of an alternative, such as passing the real cpu
>> > > > > > nodes to dom0 but then adding status = "disabled", but I am not sure
>> > > > > > whether Linux checks the status for cpu nodes.
>> > > > >
>> > > > > status = "disabled" is defined to have a specific (i.e. non-default)
>> > > > > meaning for cpu nodes, Julien mentioned this when I tried to add a
>> > > > > similar patch to Xen to ignore them. I think it basically means "present
>> > > > > but not running, you should start them!".
>> > > > >
>> > > > > > In addition this scheme
>> > > > > > wouldn't support the case where dom0 has more vcpus than pcpus on the
>> > > > > > system. Granted it is not very common and might even be detrimental for
>> > > > > > performances, but we should be able to support it.
>> > > > >
>> > > > > It's a bit of an edge case, for sure. I guess it wouldn't be totally
>> > > > > unreasonable to say that if you use this sort of configuration you may
>> > > > > not get cpufreq support.
>> > > > >
>> > > > > > Ian, what do you think about this?
>> > > > >
>> > > > > All the options suck in one way or another AFAICT. I think we are going
>> > > > > to be looking for the least bad solution not necessarily a good one.
>> > > > >
>> > > > > Fundamentally are we trying to avoid having to have a i2c subsystem etc
>> > > > > in the hypervisor to be be able to change the voltages before/after
>> > > > > changing the frequency?
>> > > > >
>> > > > > We can't just say "that's part of the cpufreq driver" since different
>> > > > > boards using the same SoC might use different voltage regulators, over
>> > > > > i2c or some other bus etc, so we end up with a matrix.
>> > > > >
>> > > > > It's arguable that we should be letting dom0 poke at that regulator
>> > > > > functionality anyway, at least not all of it. Taking that ability away
>> > > > > would necessarily imply more platform specific functionality in the
>> > > > > hypervisor.
>> > > >
>> > > > Right.
>> > > > I am afraid that in order to avoid more code in Xen, we end up with an
>> > > > unmaintainable interface and unupstreamable hacks in dom0.
>> > >
>> > > That's what I'm worried about to. Hence I'm wondering if we should just
>> > > do this in the hypervisor.
>> > >
>> > > Although there are a myriad of them the parts used to do voltage control
>> > > tend to be fairly simple.
>> > >
>> > > One concern I have is that i2c busses also tend to have other things on
>> > > them which dom0 might legitimately access (e.g. rtc), I'm not sure what
>> > > to suggest here.
>> >
>> > I would try to avoid i2c transactions in Xen. I2C driver is quite
>> > complicated in Linux kernel. It consists of several parts - common
>> > core + platform specific. I'm pretty sure Xen should not handle this.
>> > I think that establishing of event channel for frequency changing is a
>> > good idea. It would be good to try to implement this. In process of
>> > implementation we will see what is need to be resolved.
>>
>> OK, that's reasonable.
>>
>>
>> > The only question here is how to pass physical cpu to dom0.
>>
>> We can use a device tree based interface to pass the information to
>> dom0, but requiring a number of dom0 vcpus equal to the number of
>> physical cpus and in addition to that having to pin the vcpus each to a
>> different pcpu is quite a stringent limitation. However I don't know the
>> frequency changing interfaces in Linux well enough to know how hard
>> would be to lift it.
>>
>>
>> > Regarding x86.
>> > I'm not sure but maybe ACPI interface encapsulate voltage changing as well?
>>
>> I think so (but I am not an expert on that).
>
> The usual states are P and C states. The P states is the closes to what you
> are looking at:
>
> struct acpi_processor_px {
> u64 core_frequency; /* megahertz */
> u64 power; /* milliWatts */
> u64 transition_latency; /* microseconds */
> u64 bus_master_latency; /* microseconds */
> u64 control; /* control value */
> u64 status; /* success indicator */
> };
>
>>
>>
>>
>> > Regards,
>> > Andrii
>> >
>> >
>> > --
>> >
>> > Andrii Tseglytskyi | Embedded Dev
>> > GlobalLogic
>> > www.globallogic.com
>> >
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
Cpufreq driver implementation.
____________
/ \
| xenpm tool |
\____________/
Dom0 kernel user-space
---------------------------------------------------------------------------
________________ _____
/ \ / \ CPU
| DevTree Parser | /->| ARM | driver
\________________/ | \_____/
Dom0 kernel | |
-----------------------------------------------------|-----|---------------
| |
_____________________________________ | |
| __________ ___________ | | |
| / \ / \ | | |
| | ondemand | | userspace | | | |
Registered | \__________/ \___________/ | | |
cpufreq | _____________ ___________ | | |
governor | / \ / \ | | |
| | performance | | powersave | | | |
| \_____________/ \___________/ | | |
|_____________________________________| | |
^ | |
| | |
______|_______ | |
/ \ | | Change
| cpufreq core |-------------/ | frequency
\______________/ set/get freq |
commands |
Xen |
-----------------------------------------------------------|--------------
Hardware __V__
| |
| CPU |
|_____|
Description of the implementation:
Cpufreq core and registered cpufreq governors are located in xen. Dom0
has CPU driver
which can only change frequency of the physical CPUs. In addition this driver
can change CPUs regulator voltage. I'll reuse some ACPI-specific
variables for ARM.
Thus I can make minimum modification in the xen cpufreq driver and all utilities
(as xenpm) will be working without modification if the xen code. In first
implementation xenpm tool won't show information about C-states, but it can show
information about P-states and can change cpufreq parameters and
change governor.
DevTree parser is a part of the CPU driver in Dom0 and it will read information
from /cpus/cpu@0/private_data path instead of the original /cpus path.
Steps of the initialization:
1. Xen copies all cpu@0..cpu@N nodes (from input device tree) with properties to
/cpus/cpu@0/private_data node (device tree for Dom0). Thus we can have
any number
of VCPUs in Dom0 and we give all information about all physical CPUs in
the private_data node.
2. Driver in Dom0 will parse /cpus/cpu@0/private_data path instead of the /cpus
path and give the information about CPUs parameters to the hypervisor via
XENPF_set_processor_pminfo hypercall. (Some parameters are calculated in the
Dom0 driver and can not be calculated in the hypervisor).
3. Cpufreq core driver in the hypervisor will communicate via some interface
with Dom0 (event channel can be used to notify Dom0) and give some commands
to the CPU driver in Dom0. Those command are set/get frequency, etc.
Can I implement cpufreq driver in this way?
Oleksandr Dmytryshyn | Product Engineering and Development
GlobalLogic
M +38.067.382.2525
www.globallogic.com
http://www.globallogic.com/email_disclaimer.txt
^ permalink raw reply [flat|nested] 57+ messages in thread* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-16 13:49 ` Oleksandr Dmytryshyn
@ 2014-09-17 17:06 ` Konrad Rzeszutek Wilk
2014-09-18 9:38 ` Oleksandr Dmytryshyn
2014-09-24 14:36 ` Stefano Stabellini
1 sibling, 1 reply; 57+ messages in thread
From: Konrad Rzeszutek Wilk @ 2014-09-17 17:06 UTC (permalink / raw)
To: Oleksandr Dmytryshyn
Cc: Ian Campbell, Stefano Stabellini, jinsong.liu, Tim Deegan,
xen-devel, Jan Beulich, jacob.shin, Andrii Tseglytskyi
On Tue, Sep 16, 2014 at 04:49:46PM +0300, Oleksandr Dmytryshyn wrote:
> On Wed, Sep 10, 2014 at 10:31 PM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
> > On Wed, Sep 10, 2014 at 07:35:47PM +0100, Stefano Stabellini wrote:
> >> On Wed, 10 Sep 2014, Andrii Tseglytskyi wrote:
> >> > Hi,
> >> >
> >> > On Wed, Sep 10, 2014 at 12:42 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> >> > >
> >> > > On Tue, 2014-09-09 at 22:41 +0100, Stefano Stabellini wrote:
> >> > > > On Tue, 9 Sep 2014, Ian Campbell wrote:
> >> > > > > On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
> >> > > > > > I am trying to think of an alternative, such as passing the real cpu
> >> > > > > > nodes to dom0 but then adding status = "disabled", but I am not sure
> >> > > > > > whether Linux checks the status for cpu nodes.
> >> > > > >
> >> > > > > status = "disabled" is defined to have a specific (i.e. non-default)
> >> > > > > meaning for cpu nodes, Julien mentioned this when I tried to add a
> >> > > > > similar patch to Xen to ignore them. I think it basically means "present
> >> > > > > but not running, you should start them!".
> >> > > > >
> >> > > > > > In addition this scheme
> >> > > > > > wouldn't support the case where dom0 has more vcpus than pcpus on the
> >> > > > > > system. Granted it is not very common and might even be detrimental for
> >> > > > > > performances, but we should be able to support it.
> >> > > > >
> >> > > > > It's a bit of an edge case, for sure. I guess it wouldn't be totally
> >> > > > > unreasonable to say that if you use this sort of configuration you may
> >> > > > > not get cpufreq support.
> >> > > > >
> >> > > > > > Ian, what do you think about this?
> >> > > > >
> >> > > > > All the options suck in one way or another AFAICT. I think we are going
> >> > > > > to be looking for the least bad solution not necessarily a good one.
> >> > > > >
> >> > > > > Fundamentally are we trying to avoid having to have a i2c subsystem etc
> >> > > > > in the hypervisor to be be able to change the voltages before/after
> >> > > > > changing the frequency?
> >> > > > >
> >> > > > > We can't just say "that's part of the cpufreq driver" since different
> >> > > > > boards using the same SoC might use different voltage regulators, over
> >> > > > > i2c or some other bus etc, so we end up with a matrix.
> >> > > > >
> >> > > > > It's arguable that we should be letting dom0 poke at that regulator
> >> > > > > functionality anyway, at least not all of it. Taking that ability away
> >> > > > > would necessarily imply more platform specific functionality in the
> >> > > > > hypervisor.
> >> > > >
> >> > > > Right.
> >> > > > I am afraid that in order to avoid more code in Xen, we end up with an
> >> > > > unmaintainable interface and unupstreamable hacks in dom0.
> >> > >
> >> > > That's what I'm worried about to. Hence I'm wondering if we should just
> >> > > do this in the hypervisor.
> >> > >
> >> > > Although there are a myriad of them the parts used to do voltage control
> >> > > tend to be fairly simple.
> >> > >
> >> > > One concern I have is that i2c busses also tend to have other things on
> >> > > them which dom0 might legitimately access (e.g. rtc), I'm not sure what
> >> > > to suggest here.
> >> >
> >> > I would try to avoid i2c transactions in Xen. I2C driver is quite
> >> > complicated in Linux kernel. It consists of several parts - common
> >> > core + platform specific. I'm pretty sure Xen should not handle this.
> >> > I think that establishing of event channel for frequency changing is a
> >> > good idea. It would be good to try to implement this. In process of
> >> > implementation we will see what is need to be resolved.
> >>
> >> OK, that's reasonable.
> >>
> >>
> >> > The only question here is how to pass physical cpu to dom0.
> >>
> >> We can use a device tree based interface to pass the information to
> >> dom0, but requiring a number of dom0 vcpus equal to the number of
> >> physical cpus and in addition to that having to pin the vcpus each to a
> >> different pcpu is quite a stringent limitation. However I don't know the
> >> frequency changing interfaces in Linux well enough to know how hard
> >> would be to lift it.
> >>
> >>
> >> > Regarding x86.
> >> > I'm not sure but maybe ACPI interface encapsulate voltage changing as well?
> >>
> >> I think so (but I am not an expert on that).
> >
> > The usual states are P and C states. The P states is the closes to what you
> > are looking at:
> >
> > struct acpi_processor_px {
> > u64 core_frequency; /* megahertz */
> > u64 power; /* milliWatts */
> > u64 transition_latency; /* microseconds */
> > u64 bus_master_latency; /* microseconds */
> > u64 control; /* control value */
> > u64 status; /* success indicator */
> > };
> >
> >>
> >>
> >>
> >> > Regards,
> >> > Andrii
> >> >
> >> >
> >> > --
> >> >
> >> > Andrii Tseglytskyi | Embedded Dev
> >> > GlobalLogic
> >> > www.globallogic.com
> >> >
> >>
> >> _______________________________________________
> >> Xen-devel mailing list
> >> Xen-devel@lists.xen.org
> >> http://lists.xen.org/xen-devel
>
>
> Cpufreq driver implementation.
> ____________
> / \
> | xenpm tool |
> \____________/
> Dom0 kernel user-space
> ---------------------------------------------------------------------------
>
> ________________ _____
> / \ / \ CPU
> | DevTree Parser | /->| ARM | driver
> \________________/ | \_____/
> Dom0 kernel | |
> -----------------------------------------------------|-----|---------------
> | |
> _____________________________________ | |
> | __________ ___________ | | |
> | / \ / \ | | |
> | | ondemand | | userspace | | | |
> Registered | \__________/ \___________/ | | |
> cpufreq | _____________ ___________ | | |
> governor | / \ / \ | | |
> | | performance | | powersave | | | |
> | \_____________/ \___________/ | | |
> |_____________________________________| | |
> ^ | |
> | | |
> ______|_______ | |
> / \ | | Change
> | cpufreq core |-------------/ | frequency
> \______________/ set/get freq |
> commands |
> Xen |
> -----------------------------------------------------------|--------------
> Hardware __V__
> | |
> | CPU |
> |_____|
>
>
> Description of the implementation:
> Cpufreq core and registered cpufreq governors are located in xen. Dom0
> has CPU driver
> which can only change frequency of the physical CPUs. In addition this driver
> can change CPUs regulator voltage. I'll reuse some ACPI-specific
> variables for ARM.
> Thus I can make minimum modification in the xen cpufreq driver and all utilities
> (as xenpm) will be working without modification if the xen code. In first
> implementation xenpm tool won't show information about C-states, but it can show
> information about P-states and can change cpufreq parameters and
> change governor.
> DevTree parser is a part of the CPU driver in Dom0 and it will read information
> from /cpus/cpu@0/private_data path instead of the original /cpus path.
>
> Steps of the initialization:
> 1. Xen copies all cpu@0..cpu@N nodes (from input device tree) with properties to
> /cpus/cpu@0/private_data node (device tree for Dom0). Thus we can have
> any number
> of VCPUs in Dom0 and we give all information about all physical CPUs in
> the private_data node.
>
> 2. Driver in Dom0 will parse /cpus/cpu@0/private_data path instead of the /cpus
> path and give the information about CPUs parameters to the hypervisor via
> XENPF_set_processor_pminfo hypercall. (Some parameters are calculated in the
> Dom0 driver and can not be calculated in the hypervisor).
Which driver? I presume it would be similar to the xen-acpi-processor.c driver
in drivers/xen?
>
> 3. Cpufreq core driver in the hypervisor will communicate via some interface
> with Dom0 (event channel can be used to notify Dom0) and give some commands
> to the CPU driver in Dom0. Those command are set/get frequency, etc.
Like the 'xenpm' which does that?
>
> Can I implement cpufreq driver in this way?
I don't see why not. Thought I am curious to what is the 'driver' you
are referring too. I presume it is the one that reads the voltage values
from something (what is that "Something" ?)?
>
> Oleksandr Dmytryshyn | Product Engineering and Development
> GlobalLogic
> M +38.067.382.2525
> www.globallogic.com
>
> http://www.globallogic.com/email_disclaimer.txt
^ permalink raw reply [flat|nested] 57+ messages in thread* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-17 17:06 ` Konrad Rzeszutek Wilk
@ 2014-09-18 9:38 ` Oleksandr Dmytryshyn
2014-09-18 14:59 ` Konrad Rzeszutek Wilk
0 siblings, 1 reply; 57+ messages in thread
From: Oleksandr Dmytryshyn @ 2014-09-18 9:38 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk
Cc: Ian Campbell, Stefano Stabellini, jinsong.liu, Tim Deegan,
xen-devel, Jan Beulich, jacob.shin, Andrii Tseglytskyi
On Wed, Sep 17, 2014 at 8:06 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
>
> On Tue, Sep 16, 2014 at 04:49:46PM +0300, Oleksandr Dmytryshyn wrote:
> > On Wed, Sep 10, 2014 at 10:31 PM, Konrad Rzeszutek Wilk
> > <konrad.wilk@oracle.com> wrote:
> > > On Wed, Sep 10, 2014 at 07:35:47PM +0100, Stefano Stabellini wrote:
> > >> On Wed, 10 Sep 2014, Andrii Tseglytskyi wrote:
> > >> > Hi,
> > >> >
> > >> > On Wed, Sep 10, 2014 at 12:42 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > >> > >
> > >> > > On Tue, 2014-09-09 at 22:41 +0100, Stefano Stabellini wrote:
> > >> > > > On Tue, 9 Sep 2014, Ian Campbell wrote:
> > >> > > > > On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
> > >> > > > > > I am trying to think of an alternative, such as passing the real cpu
> > >> > > > > > nodes to dom0 but then adding status = "disabled", but I am not sure
> > >> > > > > > whether Linux checks the status for cpu nodes.
> > >> > > > >
> > >> > > > > status = "disabled" is defined to have a specific (i.e. non-default)
> > >> > > > > meaning for cpu nodes, Julien mentioned this when I tried to add a
> > >> > > > > similar patch to Xen to ignore them. I think it basically means "present
> > >> > > > > but not running, you should start them!".
> > >> > > > >
> > >> > > > > > In addition this scheme
> > >> > > > > > wouldn't support the case where dom0 has more vcpus than pcpus on the
> > >> > > > > > system. Granted it is not very common and might even be detrimental for
> > >> > > > > > performances, but we should be able to support it.
> > >> > > > >
> > >> > > > > It's a bit of an edge case, for sure. I guess it wouldn't be totally
> > >> > > > > unreasonable to say that if you use this sort of configuration you may
> > >> > > > > not get cpufreq support.
> > >> > > > >
> > >> > > > > > Ian, what do you think about this?
> > >> > > > >
> > >> > > > > All the options suck in one way or another AFAICT. I think we are going
> > >> > > > > to be looking for the least bad solution not necessarily a good one.
> > >> > > > >
> > >> > > > > Fundamentally are we trying to avoid having to have a i2c subsystem etc
> > >> > > > > in the hypervisor to be be able to change the voltages before/after
> > >> > > > > changing the frequency?
> > >> > > > >
> > >> > > > > We can't just say "that's part of the cpufreq driver" since different
> > >> > > > > boards using the same SoC might use different voltage regulators, over
> > >> > > > > i2c or some other bus etc, so we end up with a matrix.
> > >> > > > >
> > >> > > > > It's arguable that we should be letting dom0 poke at that regulator
> > >> > > > > functionality anyway, at least not all of it. Taking that ability away
> > >> > > > > would necessarily imply more platform specific functionality in the
> > >> > > > > hypervisor.
> > >> > > >
> > >> > > > Right.
> > >> > > > I am afraid that in order to avoid more code in Xen, we end up with an
> > >> > > > unmaintainable interface and unupstreamable hacks in dom0.
> > >> > >
> > >> > > That's what I'm worried about to. Hence I'm wondering if we should just
> > >> > > do this in the hypervisor.
> > >> > >
> > >> > > Although there are a myriad of them the parts used to do voltage control
> > >> > > tend to be fairly simple.
> > >> > >
> > >> > > One concern I have is that i2c busses also tend to have other things on
> > >> > > them which dom0 might legitimately access (e.g. rtc), I'm not sure what
> > >> > > to suggest here.
> > >> >
> > >> > I would try to avoid i2c transactions in Xen. I2C driver is quite
> > >> > complicated in Linux kernel. It consists of several parts - common
> > >> > core + platform specific. I'm pretty sure Xen should not handle this.
> > >> > I think that establishing of event channel for frequency changing is a
> > >> > good idea. It would be good to try to implement this. In process of
> > >> > implementation we will see what is need to be resolved.
> > >>
> > >> OK, that's reasonable.
> > >>
> > >>
> > >> > The only question here is how to pass physical cpu to dom0.
> > >>
> > >> We can use a device tree based interface to pass the information to
> > >> dom0, but requiring a number of dom0 vcpus equal to the number of
> > >> physical cpus and in addition to that having to pin the vcpus each to a
> > >> different pcpu is quite a stringent limitation. However I don't know the
> > >> frequency changing interfaces in Linux well enough to know how hard
> > >> would be to lift it.
> > >>
> > >>
> > >> > Regarding x86.
> > >> > I'm not sure but maybe ACPI interface encapsulate voltage changing as well?
> > >>
> > >> I think so (but I am not an expert on that).
> > >
> > > The usual states are P and C states. The P states is the closes to what you
> > > are looking at:
> > >
> > > struct acpi_processor_px {
> > > u64 core_frequency; /* megahertz */
> > > u64 power; /* milliWatts */
> > > u64 transition_latency; /* microseconds */
> > > u64 bus_master_latency; /* microseconds */
> > > u64 control; /* control value */
> > > u64 status; /* success indicator */
> > > };
> > >
> > >>
> > >>
> > >>
> > >> > Regards,
> > >> > Andrii
> > >> >
> > >> >
> > >> > --
> > >> >
> > >> > Andrii Tseglytskyi | Embedded Dev
> > >> > GlobalLogic
> > >> > www.globallogic.com
> > >> >
> > >>
> > >> _______________________________________________
> > >> Xen-devel mailing list
> > >> Xen-devel@lists.xen.org
> > >> http://lists.xen.org/xen-devel
> >
> >
> > Cpufreq driver implementation.
> > ____________
> > / \
> > | xenpm tool |
> > \____________/
> > Dom0 kernel user-space
> > ---------------------------------------------------------------------------
> >
> > ________________ _____
> > / \ / \ CPU
> > | DevTree Parser | /->| ARM | driver
> > \________________/ | \_____/
> > Dom0 kernel | |
> > -----------------------------------------------------|-----|---------------
> > | |
> > _____________________________________ | |
> > | __________ ___________ | | |
> > | / \ / \ | | |
> > | | ondemand | | userspace | | | |
> > Registered | \__________/ \___________/ | | |
> > cpufreq | _____________ ___________ | | |
> > governor | / \ / \ | | |
> > | | performance | | powersave | | | |
> > | \_____________/ \___________/ | | |
> > |_____________________________________| | |
> > ^ | |
> > | | |
> > ______|_______ | |
> > / \ | | Change
> > | cpufreq core |-------------/ | frequency
> > \______________/ set/get freq |
> > commands |
> > Xen |
> > -----------------------------------------------------------|--------------
> > Hardware __V__
> > | |
> > | CPU |
> > |_____|
> >
> >
> > Description of the implementation:
> > Cpufreq core and registered cpufreq governors are located in xen. Dom0
> > has CPU driver
> > which can only change frequency of the physical CPUs. In addition this driver
> > can change CPUs regulator voltage. I'll reuse some ACPI-specific
> > variables for ARM.
> > Thus I can make minimum modification in the xen cpufreq driver and all utilities
> > (as xenpm) will be working without modification if the xen code. In first
> > implementation xenpm tool won't show information about C-states, but it can show
> > information about P-states and can change cpufreq parameters and
> > change governor.
> > DevTree parser is a part of the CPU driver in Dom0 and it will read information
> > from /cpus/cpu@0/private_data path instead of the original /cpus path.
> >
> > Steps of the initialization:
> > 1. Xen copies all cpu@0..cpu@N nodes (from input device tree) with properties to
> > /cpus/cpu@0/private_data node (device tree for Dom0). Thus we can have
> > any number
> > of VCPUs in Dom0 and we give all information about all physical CPUs in
> > the private_data node.
> >
> > 2. Driver in Dom0 will parse /cpus/cpu@0/private_data path instead of the /cpus
> > path and give the information about CPUs parameters to the hypervisor via
> > XENPF_set_processor_pminfo hypercall. (Some parameters are calculated in the
> > Dom0 driver and can not be calculated in the hypervisor).
>
> Which driver? I presume it would be similar to the xen-acpi-processor.c driver
> in drivers/xen?
I meant slightly modified cpufreq-cpu0 driver in Linux kernel - driver
which changes
frequency and voltage on the OMAP CPU. And this driver will use part of the
xen-acpi-processor.c driver to upload P-states for xen.
xen-acpi-processor.c driver only uploads the information about
P-states and C-states
to the xen and xen can change CPUs frequency and voltage via ACPI interface.
In this case xen works with ACPI interface directly (without Dom0 kernel).
In case the ARM architecture every processor has own non-standardized interface
to change frequency. And in addition we should change voltage on CPU when we
change frequency. Every ARM CPU has own external Power Management IC
which can change voltage. But Power Management IC has own non-standardized
interface and command sequence to change voltage. In my case voltage can
be changed via I2C commands. We can not use I2C in the same time in xen
and in Dom0 (in this case I2C should be virtualized). Thus if we want
that cpufreq
driver work in the same way for ARM as for x86 we should port to xen a lot of
the frameworks from Linux kernel (I2C, clock and regulator framework) and
we should virtualize each framework. But we can not do that.
> >
> > 3. Cpufreq core driver in the hypervisor will communicate via some interface
> > with Dom0 (event channel can be used to notify Dom0) and give some commands
> > to the CPU driver in Dom0. Those command are set/get frequency, etc.
>
> Like the 'xenpm' which does that?
Not exactly. 'xenpm' communicates via hypercalls with Xen. In this case
xen acts as slave. And all cpufreq drivers are inside xen. In my case
part of the
cpufreq driver will be implemented in the Linux kernel (Dom0) and xen should
act as master and give commands to the cpufreq CPU driver in Dom0
(this driver will only change frequency and voltage on physical CPUs).
> >
> > Can I implement cpufreq driver in this way?
>
> I don't see why not. Thought I am curious to what is the 'driver' you
> are referring too. I presume it is the one that reads the voltage values
> from something (what is that "Something" ?)?
I've written about this driver above. This driver (in Dom0) will read
information
about voltages and frequencies from Device Tree. This Device Tree is created
by xen for Dom0.
> >
> > Oleksandr Dmytryshyn | Product Engineering and Development
> > GlobalLogic
> > M +38.067.382.2525
> > www.globallogic.com
> >
> > http://www.globallogic.com/email_disclaimer.txt
Oleksandr Dmytryshyn | Product Engineering and Development
GlobalLogic
M +38.067.382.2525
www.globallogic.com
http://www.globallogic.com/email_disclaimer.txt
^ permalink raw reply [flat|nested] 57+ messages in thread* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-18 9:38 ` Oleksandr Dmytryshyn
@ 2014-09-18 14:59 ` Konrad Rzeszutek Wilk
2014-09-19 9:38 ` Oleksandr Dmytryshyn
0 siblings, 1 reply; 57+ messages in thread
From: Konrad Rzeszutek Wilk @ 2014-09-18 14:59 UTC (permalink / raw)
To: Oleksandr Dmytryshyn
Cc: Ian Campbell, Stefano Stabellini, jinsong.liu, Tim Deegan,
xen-devel, Jan Beulich, jacob.shin, Andrii Tseglytskyi
> > > 2. Driver in Dom0 will parse /cpus/cpu@0/private_data path instead of the /cpus
> > > path and give the information about CPUs parameters to the hypervisor via
> > > XENPF_set_processor_pminfo hypercall. (Some parameters are calculated in the
> > > Dom0 driver and can not be calculated in the hypervisor).
> >
> > Which driver? I presume it would be similar to the xen-acpi-processor.c driver
> > in drivers/xen?
> I meant slightly modified cpufreq-cpu0 driver in Linux kernel - driver
> which changes
> frequency and voltage on the OMAP CPU. And this driver will use part of the
OK, is that something you would upstream as well? Which of the ARM cpufreq
drivers is this?
> xen-acpi-processor.c driver to upload P-states for xen.
> xen-acpi-processor.c driver only uploads the information about
> P-states and C-states
> to the xen and xen can change CPUs frequency and voltage via ACPI interface.
> In this case xen works with ACPI interface directly (without Dom0 kernel).
> In case the ARM architecture every processor has own non-standardized interface
> to change frequency. And in addition we should change voltage on CPU when we
> change frequency. Every ARM CPU has own external Power Management IC
> which can change voltage. But Power Management IC has own non-standardized
> interface and command sequence to change voltage. In my case voltage can
> be changed via I2C commands. We can not use I2C in the same time in xen
> and in Dom0 (in this case I2C should be virtualized). Thus if we want
> that cpufreq
> driver work in the same way for ARM as for x86 we should port to xen a lot of
> the frameworks from Linux kernel (I2C, clock and regulator framework) and
> we should virtualize each framework. But we can not do that.
>
> > >
> > > 3. Cpufreq core driver in the hypervisor will communicate via some interface
> > > with Dom0 (event channel can be used to notify Dom0) and give some commands
> > > to the CPU driver in Dom0. Those command are set/get frequency, etc.
> >
> > Like the 'xenpm' which does that?
> Not exactly. 'xenpm' communicates via hypercalls with Xen. In this case
> xen acts as slave. And all cpufreq drivers are inside xen. In my case
> part of the
> cpufreq driver will be implemented in the Linux kernel (Dom0) and xen should
> act as master and give commands to the cpufreq CPU driver in Dom0
> (this driver will only change frequency and voltage on physical CPUs).
Oh, I must have misunderstood your picture. In it the cpufreq and the
lot seemed to be all inside Xen.
> I
> > >
> > > Can I implement cpufreq driver in this way?
> >
> > I don't see why not. Thought I am curious to what is the 'driver' you
> > are referring too. I presume it is the one that reads the voltage values
> > from something (what is that "Something" ?)?
> I've written about this driver above. This driver (in Dom0) will read
> information
> about voltages and frequencies from Device Tree. This Device Tree is created
> by xen for Dom0.
And this driver is based on some Linux driver right ? Which one?
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-18 14:59 ` Konrad Rzeszutek Wilk
@ 2014-09-19 9:38 ` Oleksandr Dmytryshyn
0 siblings, 0 replies; 57+ messages in thread
From: Oleksandr Dmytryshyn @ 2014-09-19 9:38 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk
Cc: Ian Campbell, Stefano Stabellini, jinsong.liu, Tim Deegan,
xen-devel, Jan Beulich, jacob.shin, Andrii Tseglytskyi
On Thu, Sep 18, 2014 at 5:59 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
>> > > 2. Driver in Dom0 will parse /cpus/cpu@0/private_data path instead of the /cpus
>> > > path and give the information about CPUs parameters to the hypervisor via
>> > > XENPF_set_processor_pminfo hypercall. (Some parameters are calculated in the
>> > > Dom0 driver and can not be calculated in the hypervisor).
>> >
>> > Which driver? I presume it would be similar to the xen-acpi-processor.c driver
>> > in drivers/xen?
>> I meant slightly modified cpufreq-cpu0 driver in Linux kernel - driver
>> which changes
>> frequency and voltage on the OMAP CPU. And this driver will use part of the
>
> OK, is that something you would upstream as well? Which of the ARM cpufreq
> drivers is this?
I'll try to upstream it. But at first I should write and check that
cpufreq mechanism
is fully working.
This driver is drivers/cpufreq/cpufreq-cpu0.c
>> xen-acpi-processor.c driver to upload P-states for xen.
>> xen-acpi-processor.c driver only uploads the information about
>> P-states and C-states
>> to the xen and xen can change CPUs frequency and voltage via ACPI interface.
>> In this case xen works with ACPI interface directly (without Dom0 kernel).
>> In case the ARM architecture every processor has own non-standardized interface
>> to change frequency. And in addition we should change voltage on CPU when we
>> change frequency. Every ARM CPU has own external Power Management IC
>> which can change voltage. But Power Management IC has own non-standardized
>> interface and command sequence to change voltage. In my case voltage can
>> be changed via I2C commands. We can not use I2C in the same time in xen
>> and in Dom0 (in this case I2C should be virtualized). Thus if we want
>> that cpufreq
>> driver work in the same way for ARM as for x86 we should port to xen a lot of
>> the frameworks from Linux kernel (I2C, clock and regulator framework) and
>> we should virtualize each framework. But we can not do that.
>>
>> > >
>> > > 3. Cpufreq core driver in the hypervisor will communicate via some interface
>> > > with Dom0 (event channel can be used to notify Dom0) and give some commands
>> > > to the CPU driver in Dom0. Those command are set/get frequency, etc.
>> >
>> > Like the 'xenpm' which does that?
>> Not exactly. 'xenpm' communicates via hypercalls with Xen. In this case
>> xen acts as slave. And all cpufreq drivers are inside xen. In my case
>> part of the
>> cpufreq driver will be implemented in the Linux kernel (Dom0) and xen should
>> act as master and give commands to the cpufreq CPU driver in Dom0
>> (this driver will only change frequency and voltage on physical CPUs).
>
> Oh, I must have misunderstood your picture. In it the cpufreq and the
> lot seemed to be all inside Xen.
Yes, but there is a 'CPU driver' on the picture and this driver is
drivers/cpufreq/cpufreq-cpu0.c
in the Linux kernel.
>
>> I
>> > >
>> > > Can I implement cpufreq driver in this way?
>> >
>> > I don't see why not. Thought I am curious to what is the 'driver' you
>> > are referring too. I presume it is the one that reads the voltage values
>> > from something (what is that "Something" ?)?
>> I've written about this driver above. This driver (in Dom0) will read
>> information
>> about voltages and frequencies from Device Tree. This Device Tree is created
>> by xen for Dom0.
>
> And this driver is based on some Linux driver right ? Which one?
This driver is drivers/cpufreq/cpufreq-cpu0.c and it reads all information
from the Device Tree. Also this driver can change voltage and frequency
on physical CPU (using 'clock' and 'regulator' frameworks).
Oleksandr Dmytryshyn | Product Engineering and Development
GlobalLogic
M +38.067.382.2525
www.globallogic.com
http://www.globallogic.com/email_disclaimer.txt
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-16 13:49 ` Oleksandr Dmytryshyn
2014-09-17 17:06 ` Konrad Rzeszutek Wilk
@ 2014-09-24 14:36 ` Stefano Stabellini
2014-09-24 14:46 ` Konrad Rzeszutek Wilk
2014-09-25 9:08 ` Oleksandr Dmytryshyn
1 sibling, 2 replies; 57+ messages in thread
From: Stefano Stabellini @ 2014-09-24 14:36 UTC (permalink / raw)
To: Oleksandr Dmytryshyn
Cc: Ian Campbell, Stefano Stabellini, jinsong.liu, Tim Deegan,
xen-devel, Jan Beulich, jacob.shin, Andrii Tseglytskyi
On Tue, 16 Sep 2014, Oleksandr Dmytryshyn wrote:
> On Wed, Sep 10, 2014 at 10:31 PM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
> > On Wed, Sep 10, 2014 at 07:35:47PM +0100, Stefano Stabellini wrote:
> >> On Wed, 10 Sep 2014, Andrii Tseglytskyi wrote:
> >> > Hi,
> >> >
> >> > On Wed, Sep 10, 2014 at 12:42 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> >> > >
> >> > > On Tue, 2014-09-09 at 22:41 +0100, Stefano Stabellini wrote:
> >> > > > On Tue, 9 Sep 2014, Ian Campbell wrote:
> >> > > > > On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
> >> > > > > > I am trying to think of an alternative, such as passing the real cpu
> >> > > > > > nodes to dom0 but then adding status = "disabled", but I am not sure
> >> > > > > > whether Linux checks the status for cpu nodes.
> >> > > > >
> >> > > > > status = "disabled" is defined to have a specific (i.e. non-default)
> >> > > > > meaning for cpu nodes, Julien mentioned this when I tried to add a
> >> > > > > similar patch to Xen to ignore them. I think it basically means "present
> >> > > > > but not running, you should start them!".
> >> > > > >
> >> > > > > > In addition this scheme
> >> > > > > > wouldn't support the case where dom0 has more vcpus than pcpus on the
> >> > > > > > system. Granted it is not very common and might even be detrimental for
> >> > > > > > performances, but we should be able to support it.
> >> > > > >
> >> > > > > It's a bit of an edge case, for sure. I guess it wouldn't be totally
> >> > > > > unreasonable to say that if you use this sort of configuration you may
> >> > > > > not get cpufreq support.
> >> > > > >
> >> > > > > > Ian, what do you think about this?
> >> > > > >
> >> > > > > All the options suck in one way or another AFAICT. I think we are going
> >> > > > > to be looking for the least bad solution not necessarily a good one.
> >> > > > >
> >> > > > > Fundamentally are we trying to avoid having to have a i2c subsystem etc
> >> > > > > in the hypervisor to be be able to change the voltages before/after
> >> > > > > changing the frequency?
> >> > > > >
> >> > > > > We can't just say "that's part of the cpufreq driver" since different
> >> > > > > boards using the same SoC might use different voltage regulators, over
> >> > > > > i2c or some other bus etc, so we end up with a matrix.
> >> > > > >
> >> > > > > It's arguable that we should be letting dom0 poke at that regulator
> >> > > > > functionality anyway, at least not all of it. Taking that ability away
> >> > > > > would necessarily imply more platform specific functionality in the
> >> > > > > hypervisor.
> >> > > >
> >> > > > Right.
> >> > > > I am afraid that in order to avoid more code in Xen, we end up with an
> >> > > > unmaintainable interface and unupstreamable hacks in dom0.
> >> > >
> >> > > That's what I'm worried about to. Hence I'm wondering if we should just
> >> > > do this in the hypervisor.
> >> > >
> >> > > Although there are a myriad of them the parts used to do voltage control
> >> > > tend to be fairly simple.
> >> > >
> >> > > One concern I have is that i2c busses also tend to have other things on
> >> > > them which dom0 might legitimately access (e.g. rtc), I'm not sure what
> >> > > to suggest here.
> >> >
> >> > I would try to avoid i2c transactions in Xen. I2C driver is quite
> >> > complicated in Linux kernel. It consists of several parts - common
> >> > core + platform specific. I'm pretty sure Xen should not handle this.
> >> > I think that establishing of event channel for frequency changing is a
> >> > good idea. It would be good to try to implement this. In process of
> >> > implementation we will see what is need to be resolved.
> >>
> >> OK, that's reasonable.
> >>
> >>
> >> > The only question here is how to pass physical cpu to dom0.
> >>
> >> We can use a device tree based interface to pass the information to
> >> dom0, but requiring a number of dom0 vcpus equal to the number of
> >> physical cpus and in addition to that having to pin the vcpus each to a
> >> different pcpu is quite a stringent limitation. However I don't know the
> >> frequency changing interfaces in Linux well enough to know how hard
> >> would be to lift it.
> >>
> >>
> >> > Regarding x86.
> >> > I'm not sure but maybe ACPI interface encapsulate voltage changing as well?
> >>
> >> I think so (but I am not an expert on that).
> >
> > The usual states are P and C states. The P states is the closes to what you
> > are looking at:
> >
> > struct acpi_processor_px {
> > u64 core_frequency; /* megahertz */
> > u64 power; /* milliWatts */
> > u64 transition_latency; /* microseconds */
> > u64 bus_master_latency; /* microseconds */
> > u64 control; /* control value */
> > u64 status; /* success indicator */
> > };
> >
> >>
> >>
> >>
> >> > Regards,
> >> > Andrii
> >> >
> >> >
> >> > --
> >> >
> >> > Andrii Tseglytskyi | Embedded Dev
> >> > GlobalLogic
> >> > www.globallogic.com
> >> >
> >>
> >> _______________________________________________
> >> Xen-devel mailing list
> >> Xen-devel@lists.xen.org
> >> http://lists.xen.org/xen-devel
>
>
> Cpufreq driver implementation.
> ____________
> / \
> | xenpm tool |
> \____________/
> Dom0 kernel user-space
> ---------------------------------------------------------------------------
>
> ________________ _____
> / \ / \ CPU
> | DevTree Parser | /->| ARM | driver
> \________________/ | \_____/
> Dom0 kernel | |
> -----------------------------------------------------|-----|---------------
> | |
> _____________________________________ | |
> | __________ ___________ | | |
> | / \ / \ | | |
> | | ondemand | | userspace | | | |
> Registered | \__________/ \___________/ | | |
> cpufreq | _____________ ___________ | | |
> governor | / \ / \ | | |
> | | performance | | powersave | | | |
> | \_____________/ \___________/ | | |
> |_____________________________________| | |
> ^ | |
> | | |
> ______|_______ | |
> / \ | | Change
> | cpufreq core |-------------/ | frequency
> \______________/ set/get freq |
> commands |
> Xen |
> -----------------------------------------------------------|--------------
> Hardware __V__
> | |
> | CPU |
> |_____|
>
>
> Description of the implementation:
> Cpufreq core and registered cpufreq governors are located in xen. Dom0
> has CPU driver
> which can only change frequency of the physical CPUs. In addition this driver
> can change CPUs regulator voltage. I'll reuse some ACPI-specific
> variables for ARM.
> Thus I can make minimum modification in the xen cpufreq driver and all utilities
> (as xenpm) will be working without modification if the xen code. In first
> implementation xenpm tool won't show information about C-states, but it can show
> information about P-states and can change cpufreq parameters and
> change governor.
> DevTree parser is a part of the CPU driver in Dom0 and it will read information
> from /cpus/cpu@0/private_data path instead of the original /cpus path.
>
> Steps of the initialization:
> 1. Xen copies all cpu@0..cpu@N nodes (from input device tree) with properties to
> /cpus/cpu@0/private_data node (device tree for Dom0). Thus we can have
> any number
> of VCPUs in Dom0 and we give all information about all physical CPUs in
> the private_data node.
>
> 2. Driver in Dom0 will parse /cpus/cpu@0/private_data path instead of the /cpus
> path and give the information about CPUs parameters to the hypervisor via
> XENPF_set_processor_pminfo hypercall. (Some parameters are calculated in the
> Dom0 driver and can not be calculated in the hypervisor).
>
> 3. Cpufreq core driver in the hypervisor will communicate via some interface
> with Dom0 (event channel can be used to notify Dom0) and give some commands
> to the CPU driver in Dom0. Those command are set/get frequency, etc.
>
> Can I implement cpufreq driver in this way?
The architecture looks sane to me. As Konrad pointed out, the difficulty
here is to be able to upstream the changes to the Linux driver in 2),
that you later in the thread identified as
drivers/cpufreq/cpufreq-cpu0.c.
If the changes are not invasive and you manage to upstream them in
Linux, I am all for this solution.
^ permalink raw reply [flat|nested] 57+ messages in thread* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-24 14:36 ` Stefano Stabellini
@ 2014-09-24 14:46 ` Konrad Rzeszutek Wilk
2014-09-25 9:13 ` Oleksandr Dmytryshyn
2014-09-25 9:08 ` Oleksandr Dmytryshyn
1 sibling, 1 reply; 57+ messages in thread
From: Konrad Rzeszutek Wilk @ 2014-09-24 14:46 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Ian Campbell, jinsong.liu, Oleksandr Dmytryshyn, Tim Deegan,
xen-devel, Jan Beulich, jacob.shin, Andrii Tseglytskyi
On Wed, Sep 24, 2014 at 03:36:15PM +0100, Stefano Stabellini wrote:
> On Tue, 16 Sep 2014, Oleksandr Dmytryshyn wrote:
> > On Wed, Sep 10, 2014 at 10:31 PM, Konrad Rzeszutek Wilk
> > <konrad.wilk@oracle.com> wrote:
> > > On Wed, Sep 10, 2014 at 07:35:47PM +0100, Stefano Stabellini wrote:
> > >> On Wed, 10 Sep 2014, Andrii Tseglytskyi wrote:
> > >> > Hi,
> > >> >
> > >> > On Wed, Sep 10, 2014 at 12:42 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > >> > >
> > >> > > On Tue, 2014-09-09 at 22:41 +0100, Stefano Stabellini wrote:
> > >> > > > On Tue, 9 Sep 2014, Ian Campbell wrote:
> > >> > > > > On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
> > >> > > > > > I am trying to think of an alternative, such as passing the real cpu
> > >> > > > > > nodes to dom0 but then adding status = "disabled", but I am not sure
> > >> > > > > > whether Linux checks the status for cpu nodes.
> > >> > > > >
> > >> > > > > status = "disabled" is defined to have a specific (i.e. non-default)
> > >> > > > > meaning for cpu nodes, Julien mentioned this when I tried to add a
> > >> > > > > similar patch to Xen to ignore them. I think it basically means "present
> > >> > > > > but not running, you should start them!".
> > >> > > > >
> > >> > > > > > In addition this scheme
> > >> > > > > > wouldn't support the case where dom0 has more vcpus than pcpus on the
> > >> > > > > > system. Granted it is not very common and might even be detrimental for
> > >> > > > > > performances, but we should be able to support it.
> > >> > > > >
> > >> > > > > It's a bit of an edge case, for sure. I guess it wouldn't be totally
> > >> > > > > unreasonable to say that if you use this sort of configuration you may
> > >> > > > > not get cpufreq support.
> > >> > > > >
> > >> > > > > > Ian, what do you think about this?
> > >> > > > >
> > >> > > > > All the options suck in one way or another AFAICT. I think we are going
> > >> > > > > to be looking for the least bad solution not necessarily a good one.
> > >> > > > >
> > >> > > > > Fundamentally are we trying to avoid having to have a i2c subsystem etc
> > >> > > > > in the hypervisor to be be able to change the voltages before/after
> > >> > > > > changing the frequency?
> > >> > > > >
> > >> > > > > We can't just say "that's part of the cpufreq driver" since different
> > >> > > > > boards using the same SoC might use different voltage regulators, over
> > >> > > > > i2c or some other bus etc, so we end up with a matrix.
> > >> > > > >
> > >> > > > > It's arguable that we should be letting dom0 poke at that regulator
> > >> > > > > functionality anyway, at least not all of it. Taking that ability away
> > >> > > > > would necessarily imply more platform specific functionality in the
> > >> > > > > hypervisor.
> > >> > > >
> > >> > > > Right.
> > >> > > > I am afraid that in order to avoid more code in Xen, we end up with an
> > >> > > > unmaintainable interface and unupstreamable hacks in dom0.
> > >> > >
> > >> > > That's what I'm worried about to. Hence I'm wondering if we should just
> > >> > > do this in the hypervisor.
> > >> > >
> > >> > > Although there are a myriad of them the parts used to do voltage control
> > >> > > tend to be fairly simple.
> > >> > >
> > >> > > One concern I have is that i2c busses also tend to have other things on
> > >> > > them which dom0 might legitimately access (e.g. rtc), I'm not sure what
> > >> > > to suggest here.
> > >> >
> > >> > I would try to avoid i2c transactions in Xen. I2C driver is quite
> > >> > complicated in Linux kernel. It consists of several parts - common
> > >> > core + platform specific. I'm pretty sure Xen should not handle this.
> > >> > I think that establishing of event channel for frequency changing is a
> > >> > good idea. It would be good to try to implement this. In process of
> > >> > implementation we will see what is need to be resolved.
> > >>
> > >> OK, that's reasonable.
> > >>
> > >>
> > >> > The only question here is how to pass physical cpu to dom0.
> > >>
> > >> We can use a device tree based interface to pass the information to
> > >> dom0, but requiring a number of dom0 vcpus equal to the number of
> > >> physical cpus and in addition to that having to pin the vcpus each to a
> > >> different pcpu is quite a stringent limitation. However I don't know the
> > >> frequency changing interfaces in Linux well enough to know how hard
> > >> would be to lift it.
> > >>
> > >>
> > >> > Regarding x86.
> > >> > I'm not sure but maybe ACPI interface encapsulate voltage changing as well?
> > >>
> > >> I think so (but I am not an expert on that).
> > >
> > > The usual states are P and C states. The P states is the closes to what you
> > > are looking at:
> > >
> > > struct acpi_processor_px {
> > > u64 core_frequency; /* megahertz */
> > > u64 power; /* milliWatts */
> > > u64 transition_latency; /* microseconds */
> > > u64 bus_master_latency; /* microseconds */
> > > u64 control; /* control value */
> > > u64 status; /* success indicator */
> > > };
> > >
> > >>
> > >>
> > >>
> > >> > Regards,
> > >> > Andrii
> > >> >
> > >> >
> > >> > --
> > >> >
> > >> > Andrii Tseglytskyi | Embedded Dev
> > >> > GlobalLogic
> > >> > www.globallogic.com
> > >> >
> > >>
> > >> _______________________________________________
> > >> Xen-devel mailing list
> > >> Xen-devel@lists.xen.org
> > >> http://lists.xen.org/xen-devel
> >
> >
> > Cpufreq driver implementation.
> > ____________
> > / \
> > | xenpm tool |
> > \____________/
> > Dom0 kernel user-space
> > ---------------------------------------------------------------------------
> >
> > ________________ _____
> > / \ / \ CPU
> > | DevTree Parser | /->| ARM | driver
> > \________________/ | \_____/
> > Dom0 kernel | |
> > -----------------------------------------------------|-----|---------------
> > | |
> > _____________________________________ | |
> > | __________ ___________ | | |
> > | / \ / \ | | |
> > | | ondemand | | userspace | | | |
> > Registered | \__________/ \___________/ | | |
> > cpufreq | _____________ ___________ | | |
> > governor | / \ / \ | | |
> > | | performance | | powersave | | | |
> > | \_____________/ \___________/ | | |
> > |_____________________________________| | |
> > ^ | |
> > | | |
> > ______|_______ | |
> > / \ | | Change
> > | cpufreq core |-------------/ | frequency
> > \______________/ set/get freq |
> > commands |
> > Xen |
> > -----------------------------------------------------------|--------------
> > Hardware __V__
> > | |
> > | CPU |
> > |_____|
> >
> >
> > Description of the implementation:
> > Cpufreq core and registered cpufreq governors are located in xen. Dom0
> > has CPU driver
> > which can only change frequency of the physical CPUs. In addition this driver
> > can change CPUs regulator voltage. I'll reuse some ACPI-specific
> > variables for ARM.
> > Thus I can make minimum modification in the xen cpufreq driver and all utilities
> > (as xenpm) will be working without modification if the xen code. In first
> > implementation xenpm tool won't show information about C-states, but it can show
> > information about P-states and can change cpufreq parameters and
> > change governor.
> > DevTree parser is a part of the CPU driver in Dom0 and it will read information
> > from /cpus/cpu@0/private_data path instead of the original /cpus path.
> >
> > Steps of the initialization:
> > 1. Xen copies all cpu@0..cpu@N nodes (from input device tree) with properties to
> > /cpus/cpu@0/private_data node (device tree for Dom0). Thus we can have
> > any number
> > of VCPUs in Dom0 and we give all information about all physical CPUs in
> > the private_data node.
> >
> > 2. Driver in Dom0 will parse /cpus/cpu@0/private_data path instead of the /cpus
> > path and give the information about CPUs parameters to the hypervisor via
> > XENPF_set_processor_pminfo hypercall. (Some parameters are calculated in the
> > Dom0 driver and can not be calculated in the hypervisor).
> >
> > 3. Cpufreq core driver in the hypervisor will communicate via some interface
> > with Dom0 (event channel can be used to notify Dom0) and give some commands
> > to the CPU driver in Dom0. Those command are set/get frequency, etc.
> >
> > Can I implement cpufreq driver in this way?
>
> The architecture looks sane to me. As Konrad pointed out, the difficulty
> here is to be able to upstream the changes to the Linux driver in 2),
> that you later in the thread identified as
> drivers/cpufreq/cpufreq-cpu0.c.
>
> If the changes are not invasive and you manage to upstream them in
> Linux, I am all for this solution.
Looking at the driver, you could make some of the drivers functionality
be a library (all of those 'voltage-tolerance', 'clock-latency', regulator,
etc).
And then disable the cpufreq API from working altogether (disable_cpufreq())
and have your own driver (drivers/xen/xen-cpufreq-cpu0.c) use the libraries
and upload the data to the hypervisor.
(or
^ permalink raw reply [flat|nested] 57+ messages in thread* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-24 14:46 ` Konrad Rzeszutek Wilk
@ 2014-09-25 9:13 ` Oleksandr Dmytryshyn
0 siblings, 0 replies; 57+ messages in thread
From: Oleksandr Dmytryshyn @ 2014-09-25 9:13 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk
Cc: Ian Campbell, Stefano Stabellini, jinsong.liu, Tim Deegan,
xen-devel, Jan Beulich, jacob.shin, Andrii Tseglytskyi
On Wed, Sep 24, 2014 at 5:46 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Wed, Sep 24, 2014 at 03:36:15PM +0100, Stefano Stabellini wrote:
>> On Tue, 16 Sep 2014, Oleksandr Dmytryshyn wrote:
>> > On Wed, Sep 10, 2014 at 10:31 PM, Konrad Rzeszutek Wilk
>> > <konrad.wilk@oracle.com> wrote:
>> > > On Wed, Sep 10, 2014 at 07:35:47PM +0100, Stefano Stabellini wrote:
>> > >> On Wed, 10 Sep 2014, Andrii Tseglytskyi wrote:
>> > >> > Hi,
>> > >> >
>> > >> > On Wed, Sep 10, 2014 at 12:42 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>> > >> > >
>> > >> > > On Tue, 2014-09-09 at 22:41 +0100, Stefano Stabellini wrote:
>> > >> > > > On Tue, 9 Sep 2014, Ian Campbell wrote:
>> > >> > > > > On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
>> > >> > > > > > I am trying to think of an alternative, such as passing the real cpu
>> > >> > > > > > nodes to dom0 but then adding status = "disabled", but I am not sure
>> > >> > > > > > whether Linux checks the status for cpu nodes.
>> > >> > > > >
>> > >> > > > > status = "disabled" is defined to have a specific (i.e. non-default)
>> > >> > > > > meaning for cpu nodes, Julien mentioned this when I tried to add a
>> > >> > > > > similar patch to Xen to ignore them. I think it basically means "present
>> > >> > > > > but not running, you should start them!".
>> > >> > > > >
>> > >> > > > > > In addition this scheme
>> > >> > > > > > wouldn't support the case where dom0 has more vcpus than pcpus on the
>> > >> > > > > > system. Granted it is not very common and might even be detrimental for
>> > >> > > > > > performances, but we should be able to support it.
>> > >> > > > >
>> > >> > > > > It's a bit of an edge case, for sure. I guess it wouldn't be totally
>> > >> > > > > unreasonable to say that if you use this sort of configuration you may
>> > >> > > > > not get cpufreq support.
>> > >> > > > >
>> > >> > > > > > Ian, what do you think about this?
>> > >> > > > >
>> > >> > > > > All the options suck in one way or another AFAICT. I think we are going
>> > >> > > > > to be looking for the least bad solution not necessarily a good one.
>> > >> > > > >
>> > >> > > > > Fundamentally are we trying to avoid having to have a i2c subsystem etc
>> > >> > > > > in the hypervisor to be be able to change the voltages before/after
>> > >> > > > > changing the frequency?
>> > >> > > > >
>> > >> > > > > We can't just say "that's part of the cpufreq driver" since different
>> > >> > > > > boards using the same SoC might use different voltage regulators, over
>> > >> > > > > i2c or some other bus etc, so we end up with a matrix.
>> > >> > > > >
>> > >> > > > > It's arguable that we should be letting dom0 poke at that regulator
>> > >> > > > > functionality anyway, at least not all of it. Taking that ability away
>> > >> > > > > would necessarily imply more platform specific functionality in the
>> > >> > > > > hypervisor.
>> > >> > > >
>> > >> > > > Right.
>> > >> > > > I am afraid that in order to avoid more code in Xen, we end up with an
>> > >> > > > unmaintainable interface and unupstreamable hacks in dom0.
>> > >> > >
>> > >> > > That's what I'm worried about to. Hence I'm wondering if we should just
>> > >> > > do this in the hypervisor.
>> > >> > >
>> > >> > > Although there are a myriad of them the parts used to do voltage control
>> > >> > > tend to be fairly simple.
>> > >> > >
>> > >> > > One concern I have is that i2c busses also tend to have other things on
>> > >> > > them which dom0 might legitimately access (e.g. rtc), I'm not sure what
>> > >> > > to suggest here.
>> > >> >
>> > >> > I would try to avoid i2c transactions in Xen. I2C driver is quite
>> > >> > complicated in Linux kernel. It consists of several parts - common
>> > >> > core + platform specific. I'm pretty sure Xen should not handle this.
>> > >> > I think that establishing of event channel for frequency changing is a
>> > >> > good idea. It would be good to try to implement this. In process of
>> > >> > implementation we will see what is need to be resolved.
>> > >>
>> > >> OK, that's reasonable.
>> > >>
>> > >>
>> > >> > The only question here is how to pass physical cpu to dom0.
>> > >>
>> > >> We can use a device tree based interface to pass the information to
>> > >> dom0, but requiring a number of dom0 vcpus equal to the number of
>> > >> physical cpus and in addition to that having to pin the vcpus each to a
>> > >> different pcpu is quite a stringent limitation. However I don't know the
>> > >> frequency changing interfaces in Linux well enough to know how hard
>> > >> would be to lift it.
>> > >>
>> > >>
>> > >> > Regarding x86.
>> > >> > I'm not sure but maybe ACPI interface encapsulate voltage changing as well?
>> > >>
>> > >> I think so (but I am not an expert on that).
>> > >
>> > > The usual states are P and C states. The P states is the closes to what you
>> > > are looking at:
>> > >
>> > > struct acpi_processor_px {
>> > > u64 core_frequency; /* megahertz */
>> > > u64 power; /* milliWatts */
>> > > u64 transition_latency; /* microseconds */
>> > > u64 bus_master_latency; /* microseconds */
>> > > u64 control; /* control value */
>> > > u64 status; /* success indicator */
>> > > };
>> > >
>> > >>
>> > >>
>> > >>
>> > >> > Regards,
>> > >> > Andrii
>> > >> >
>> > >> >
>> > >> > --
>> > >> >
>> > >> > Andrii Tseglytskyi | Embedded Dev
>> > >> > GlobalLogic
>> > >> > www.globallogic.com
>> > >> >
>> > >>
>> > >> _______________________________________________
>> > >> Xen-devel mailing list
>> > >> Xen-devel@lists.xen.org
>> > >> http://lists.xen.org/xen-devel
>> >
>> >
>> > Cpufreq driver implementation.
>> > ____________
>> > / \
>> > | xenpm tool |
>> > \____________/
>> > Dom0 kernel user-space
>> > ---------------------------------------------------------------------------
>> >
>> > ________________ _____
>> > / \ / \ CPU
>> > | DevTree Parser | /->| ARM | driver
>> > \________________/ | \_____/
>> > Dom0 kernel | |
>> > -----------------------------------------------------|-----|---------------
>> > | |
>> > _____________________________________ | |
>> > | __________ ___________ | | |
>> > | / \ / \ | | |
>> > | | ondemand | | userspace | | | |
>> > Registered | \__________/ \___________/ | | |
>> > cpufreq | _____________ ___________ | | |
>> > governor | / \ / \ | | |
>> > | | performance | | powersave | | | |
>> > | \_____________/ \___________/ | | |
>> > |_____________________________________| | |
>> > ^ | |
>> > | | |
>> > ______|_______ | |
>> > / \ | | Change
>> > | cpufreq core |-------------/ | frequency
>> > \______________/ set/get freq |
>> > commands |
>> > Xen |
>> > -----------------------------------------------------------|--------------
>> > Hardware __V__
>> > | |
>> > | CPU |
>> > |_____|
>> >
>> >
>> > Description of the implementation:
>> > Cpufreq core and registered cpufreq governors are located in xen. Dom0
>> > has CPU driver
>> > which can only change frequency of the physical CPUs. In addition this driver
>> > can change CPUs regulator voltage. I'll reuse some ACPI-specific
>> > variables for ARM.
>> > Thus I can make minimum modification in the xen cpufreq driver and all utilities
>> > (as xenpm) will be working without modification if the xen code. In first
>> > implementation xenpm tool won't show information about C-states, but it can show
>> > information about P-states and can change cpufreq parameters and
>> > change governor.
>> > DevTree parser is a part of the CPU driver in Dom0 and it will read information
>> > from /cpus/cpu@0/private_data path instead of the original /cpus path.
>> >
>> > Steps of the initialization:
>> > 1. Xen copies all cpu@0..cpu@N nodes (from input device tree) with properties to
>> > /cpus/cpu@0/private_data node (device tree for Dom0). Thus we can have
>> > any number
>> > of VCPUs in Dom0 and we give all information about all physical CPUs in
>> > the private_data node.
>> >
>> > 2. Driver in Dom0 will parse /cpus/cpu@0/private_data path instead of the /cpus
>> > path and give the information about CPUs parameters to the hypervisor via
>> > XENPF_set_processor_pminfo hypercall. (Some parameters are calculated in the
>> > Dom0 driver and can not be calculated in the hypervisor).
>> >
>> > 3. Cpufreq core driver in the hypervisor will communicate via some interface
>> > with Dom0 (event channel can be used to notify Dom0) and give some commands
>> > to the CPU driver in Dom0. Those command are set/get frequency, etc.
>> >
>> > Can I implement cpufreq driver in this way?
>>
>> The architecture looks sane to me. As Konrad pointed out, the difficulty
>> here is to be able to upstream the changes to the Linux driver in 2),
>> that you later in the thread identified as
>> drivers/cpufreq/cpufreq-cpu0.c.
>>
>> If the changes are not invasive and you manage to upstream them in
>> Linux, I am all for this solution.
>
> Looking at the driver, you could make some of the drivers functionality
> be a library (all of those 'voltage-tolerance', 'clock-latency', regulator,
> etc).
>
> And then disable the cpufreq API from working altogether (disable_cpufreq())
> and have your own driver (drivers/xen/xen-cpufreq-cpu0.c) use the libraries
> and upload the data to the hypervisor.
In this case I should copy original cpufreq-cpu0.c file to the
drivers/xen/xen-cpufreq-cpu0.c, change only one string -
path in the device tree
with the settings for the CPUs opp:
string
np = of_find_node_by_path("/cpus/cpu@0");
will changed to:
np = of_find_node_by_path("/cpus/cpu@0/private_data/cpu@0");
and write additional code. But I want to avoid copying code.
> (or
Oleksandr Dmytryshyn | Product Engineering and Development
GlobalLogic
M +38.067.382.2525
www.globallogic.com
http://www.globallogic.com/email_disclaimer.txt
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-24 14:36 ` Stefano Stabellini
2014-09-24 14:46 ` Konrad Rzeszutek Wilk
@ 2014-09-25 9:08 ` Oleksandr Dmytryshyn
2014-09-25 10:14 ` Stefano Stabellini
2014-09-26 18:13 ` Konrad Rzeszutek Wilk
1 sibling, 2 replies; 57+ messages in thread
From: Oleksandr Dmytryshyn @ 2014-09-25 9:08 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Ian Campbell, jinsong.liu, Tim Deegan, xen-devel, Jan Beulich,
jacob.shin, Andrii Tseglytskyi
On Wed, Sep 24, 2014 at 5:36 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> On Tue, 16 Sep 2014, Oleksandr Dmytryshyn wrote:
>> On Wed, Sep 10, 2014 at 10:31 PM, Konrad Rzeszutek Wilk
>> <konrad.wilk@oracle.com> wrote:
>> > On Wed, Sep 10, 2014 at 07:35:47PM +0100, Stefano Stabellini wrote:
>> >> On Wed, 10 Sep 2014, Andrii Tseglytskyi wrote:
>> >> > Hi,
>> >> >
>> >> > On Wed, Sep 10, 2014 at 12:42 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>> >> > >
>> >> > > On Tue, 2014-09-09 at 22:41 +0100, Stefano Stabellini wrote:
>> >> > > > On Tue, 9 Sep 2014, Ian Campbell wrote:
>> >> > > > > On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
>> >> > > > > > I am trying to think of an alternative, such as passing the real cpu
>> >> > > > > > nodes to dom0 but then adding status = "disabled", but I am not sure
>> >> > > > > > whether Linux checks the status for cpu nodes.
>> >> > > > >
>> >> > > > > status = "disabled" is defined to have a specific (i.e. non-default)
>> >> > > > > meaning for cpu nodes, Julien mentioned this when I tried to add a
>> >> > > > > similar patch to Xen to ignore them. I think it basically means "present
>> >> > > > > but not running, you should start them!".
>> >> > > > >
>> >> > > > > > In addition this scheme
>> >> > > > > > wouldn't support the case where dom0 has more vcpus than pcpus on the
>> >> > > > > > system. Granted it is not very common and might even be detrimental for
>> >> > > > > > performances, but we should be able to support it.
>> >> > > > >
>> >> > > > > It's a bit of an edge case, for sure. I guess it wouldn't be totally
>> >> > > > > unreasonable to say that if you use this sort of configuration you may
>> >> > > > > not get cpufreq support.
>> >> > > > >
>> >> > > > > > Ian, what do you think about this?
>> >> > > > >
>> >> > > > > All the options suck in one way or another AFAICT. I think we are going
>> >> > > > > to be looking for the least bad solution not necessarily a good one.
>> >> > > > >
>> >> > > > > Fundamentally are we trying to avoid having to have a i2c subsystem etc
>> >> > > > > in the hypervisor to be be able to change the voltages before/after
>> >> > > > > changing the frequency?
>> >> > > > >
>> >> > > > > We can't just say "that's part of the cpufreq driver" since different
>> >> > > > > boards using the same SoC might use different voltage regulators, over
>> >> > > > > i2c or some other bus etc, so we end up with a matrix.
>> >> > > > >
>> >> > > > > It's arguable that we should be letting dom0 poke at that regulator
>> >> > > > > functionality anyway, at least not all of it. Taking that ability away
>> >> > > > > would necessarily imply more platform specific functionality in the
>> >> > > > > hypervisor.
>> >> > > >
>> >> > > > Right.
>> >> > > > I am afraid that in order to avoid more code in Xen, we end up with an
>> >> > > > unmaintainable interface and unupstreamable hacks in dom0.
>> >> > >
>> >> > > That's what I'm worried about to. Hence I'm wondering if we should just
>> >> > > do this in the hypervisor.
>> >> > >
>> >> > > Although there are a myriad of them the parts used to do voltage control
>> >> > > tend to be fairly simple.
>> >> > >
>> >> > > One concern I have is that i2c busses also tend to have other things on
>> >> > > them which dom0 might legitimately access (e.g. rtc), I'm not sure what
>> >> > > to suggest here.
>> >> >
>> >> > I would try to avoid i2c transactions in Xen. I2C driver is quite
>> >> > complicated in Linux kernel. It consists of several parts - common
>> >> > core + platform specific. I'm pretty sure Xen should not handle this.
>> >> > I think that establishing of event channel for frequency changing is a
>> >> > good idea. It would be good to try to implement this. In process of
>> >> > implementation we will see what is need to be resolved.
>> >>
>> >> OK, that's reasonable.
>> >>
>> >>
>> >> > The only question here is how to pass physical cpu to dom0.
>> >>
>> >> We can use a device tree based interface to pass the information to
>> >> dom0, but requiring a number of dom0 vcpus equal to the number of
>> >> physical cpus and in addition to that having to pin the vcpus each to a
>> >> different pcpu is quite a stringent limitation. However I don't know the
>> >> frequency changing interfaces in Linux well enough to know how hard
>> >> would be to lift it.
>> >>
>> >>
>> >> > Regarding x86.
>> >> > I'm not sure but maybe ACPI interface encapsulate voltage changing as well?
>> >>
>> >> I think so (but I am not an expert on that).
>> >
>> > The usual states are P and C states. The P states is the closes to what you
>> > are looking at:
>> >
>> > struct acpi_processor_px {
>> > u64 core_frequency; /* megahertz */
>> > u64 power; /* milliWatts */
>> > u64 transition_latency; /* microseconds */
>> > u64 bus_master_latency; /* microseconds */
>> > u64 control; /* control value */
>> > u64 status; /* success indicator */
>> > };
>> >
>> >>
>> >>
>> >>
>> >> > Regards,
>> >> > Andrii
>> >> >
>> >> >
>> >> > --
>> >> >
>> >> > Andrii Tseglytskyi | Embedded Dev
>> >> > GlobalLogic
>> >> > www.globallogic.com
>> >> >
>> >>
>> >> _______________________________________________
>> >> Xen-devel mailing list
>> >> Xen-devel@lists.xen.org
>> >> http://lists.xen.org/xen-devel
>>
>>
>> Cpufreq driver implementation.
>> ____________
>> / \
>> | xenpm tool |
>> \____________/
>> Dom0 kernel user-space
>> ---------------------------------------------------------------------------
>>
>> ________________ _____
>> / \ / \ CPU
>> | DevTree Parser | /->| ARM | driver
>> \________________/ | \_____/
>> Dom0 kernel | |
>> -----------------------------------------------------|-----|---------------
>> | |
>> _____________________________________ | |
>> | __________ ___________ | | |
>> | / \ / \ | | |
>> | | ondemand | | userspace | | | |
>> Registered | \__________/ \___________/ | | |
>> cpufreq | _____________ ___________ | | |
>> governor | / \ / \ | | |
>> | | performance | | powersave | | | |
>> | \_____________/ \___________/ | | |
>> |_____________________________________| | |
>> ^ | |
>> | | |
>> ______|_______ | |
>> / \ | | Change
>> | cpufreq core |-------------/ | frequency
>> \______________/ set/get freq |
>> commands |
>> Xen |
>> -----------------------------------------------------------|--------------
>> Hardware __V__
>> | |
>> | CPU |
>> |_____|
>>
>>
>> Description of the implementation:
>> Cpufreq core and registered cpufreq governors are located in xen. Dom0
>> has CPU driver
>> which can only change frequency of the physical CPUs. In addition this driver
>> can change CPUs regulator voltage. I'll reuse some ACPI-specific
>> variables for ARM.
>> Thus I can make minimum modification in the xen cpufreq driver and all utilities
>> (as xenpm) will be working without modification if the xen code. In first
>> implementation xenpm tool won't show information about C-states, but it can show
>> information about P-states and can change cpufreq parameters and
>> change governor.
>> DevTree parser is a part of the CPU driver in Dom0 and it will read information
>> from /cpus/cpu@0/private_data path instead of the original /cpus path.
>>
>> Steps of the initialization:
>> 1. Xen copies all cpu@0..cpu@N nodes (from input device tree) with properties to
>> /cpus/cpu@0/private_data node (device tree for Dom0). Thus we can have
>> any number
>> of VCPUs in Dom0 and we give all information about all physical CPUs in
>> the private_data node.
>>
>> 2. Driver in Dom0 will parse /cpus/cpu@0/private_data path instead of the /cpus
>> path and give the information about CPUs parameters to the hypervisor via
>> XENPF_set_processor_pminfo hypercall. (Some parameters are calculated in the
>> Dom0 driver and can not be calculated in the hypervisor).
>>
>> 3. Cpufreq core driver in the hypervisor will communicate via some interface
>> with Dom0 (event channel can be used to notify Dom0) and give some commands
>> to the CPU driver in Dom0. Those command are set/get frequency, etc.
>>
>> Can I implement cpufreq driver in this way?
>
> The architecture looks sane to me. As Konrad pointed out, the difficulty
> here is to be able to upstream the changes to the Linux driver in 2),
> that you later in the thread identified as
> drivers/cpufreq/cpufreq-cpu0.c.
I'll write driver drivers/xen/xen-cpufreq.c and it replace original
drivers/cpufreq/cpufreq.c
And in the original cpufreq-cpu0 driver I'll chande only one string -
path in the device tree
with the settings for the CPUs opp:
string
np = of_find_node_by_path("/cpus/cpu@0");
will changed to:
np = of_find_node_by_path("/cpus/cpu@0/private_data/cpu@0");
> If the changes are not invasive and you manage to upstream them in
> Linux, I am all for this solution.
In Linux kernel I should make few changes:
1. Enable CONFIG_CPU_FREQ_TABLE
with disabled CONFIG_CPU_FREQ
2. Enable CONFIG_GENERIC_CPUFREQ_CPU0
with disabled CONFIG_CPU_FREQ
I mean make those configs dependent on
CONFIG_CPU_FREQ or CONFIX_XEN_DOM0
instead of
CONFIG_CPU_FREQ
Oleksandr Dmytryshyn | Product Engineering and Development
GlobalLogic
M +38.067.382.2525
www.globallogic.com
http://www.globallogic.com/email_disclaimer.txt
^ permalink raw reply [flat|nested] 57+ messages in thread* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-25 9:08 ` Oleksandr Dmytryshyn
@ 2014-09-25 10:14 ` Stefano Stabellini
2014-09-25 11:15 ` Oleksandr Dmytryshyn
2014-09-26 18:13 ` Konrad Rzeszutek Wilk
1 sibling, 1 reply; 57+ messages in thread
From: Stefano Stabellini @ 2014-09-25 10:14 UTC (permalink / raw)
To: Oleksandr Dmytryshyn
Cc: Ian Campbell, jinsong.liu, Stefano Stabellini, Tim Deegan,
xen-devel, Jan Beulich, jacob.shin, Andrii Tseglytskyi
On Thu, 25 Sep 2014, Oleksandr Dmytryshyn wrote:
> On Wed, Sep 24, 2014 at 5:36 PM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
> > On Tue, 16 Sep 2014, Oleksandr Dmytryshyn wrote:
> >> On Wed, Sep 10, 2014 at 10:31 PM, Konrad Rzeszutek Wilk
> >> <konrad.wilk@oracle.com> wrote:
> >> > On Wed, Sep 10, 2014 at 07:35:47PM +0100, Stefano Stabellini wrote:
> >> >> On Wed, 10 Sep 2014, Andrii Tseglytskyi wrote:
> >> >> > Hi,
> >> >> >
> >> >> > On Wed, Sep 10, 2014 at 12:42 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> >> >> > >
> >> >> > > On Tue, 2014-09-09 at 22:41 +0100, Stefano Stabellini wrote:
> >> >> > > > On Tue, 9 Sep 2014, Ian Campbell wrote:
> >> >> > > > > On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
> >> >> > > > > > I am trying to think of an alternative, such as passing the real cpu
> >> >> > > > > > nodes to dom0 but then adding status = "disabled", but I am not sure
> >> >> > > > > > whether Linux checks the status for cpu nodes.
> >> >> > > > >
> >> >> > > > > status = "disabled" is defined to have a specific (i.e. non-default)
> >> >> > > > > meaning for cpu nodes, Julien mentioned this when I tried to add a
> >> >> > > > > similar patch to Xen to ignore them. I think it basically means "present
> >> >> > > > > but not running, you should start them!".
> >> >> > > > >
> >> >> > > > > > In addition this scheme
> >> >> > > > > > wouldn't support the case where dom0 has more vcpus than pcpus on the
> >> >> > > > > > system. Granted it is not very common and might even be detrimental for
> >> >> > > > > > performances, but we should be able to support it.
> >> >> > > > >
> >> >> > > > > It's a bit of an edge case, for sure. I guess it wouldn't be totally
> >> >> > > > > unreasonable to say that if you use this sort of configuration you may
> >> >> > > > > not get cpufreq support.
> >> >> > > > >
> >> >> > > > > > Ian, what do you think about this?
> >> >> > > > >
> >> >> > > > > All the options suck in one way or another AFAICT. I think we are going
> >> >> > > > > to be looking for the least bad solution not necessarily a good one.
> >> >> > > > >
> >> >> > > > > Fundamentally are we trying to avoid having to have a i2c subsystem etc
> >> >> > > > > in the hypervisor to be be able to change the voltages before/after
> >> >> > > > > changing the frequency?
> >> >> > > > >
> >> >> > > > > We can't just say "that's part of the cpufreq driver" since different
> >> >> > > > > boards using the same SoC might use different voltage regulators, over
> >> >> > > > > i2c or some other bus etc, so we end up with a matrix.
> >> >> > > > >
> >> >> > > > > It's arguable that we should be letting dom0 poke at that regulator
> >> >> > > > > functionality anyway, at least not all of it. Taking that ability away
> >> >> > > > > would necessarily imply more platform specific functionality in the
> >> >> > > > > hypervisor.
> >> >> > > >
> >> >> > > > Right.
> >> >> > > > I am afraid that in order to avoid more code in Xen, we end up with an
> >> >> > > > unmaintainable interface and unupstreamable hacks in dom0.
> >> >> > >
> >> >> > > That's what I'm worried about to. Hence I'm wondering if we should just
> >> >> > > do this in the hypervisor.
> >> >> > >
> >> >> > > Although there are a myriad of them the parts used to do voltage control
> >> >> > > tend to be fairly simple.
> >> >> > >
> >> >> > > One concern I have is that i2c busses also tend to have other things on
> >> >> > > them which dom0 might legitimately access (e.g. rtc), I'm not sure what
> >> >> > > to suggest here.
> >> >> >
> >> >> > I would try to avoid i2c transactions in Xen. I2C driver is quite
> >> >> > complicated in Linux kernel. It consists of several parts - common
> >> >> > core + platform specific. I'm pretty sure Xen should not handle this.
> >> >> > I think that establishing of event channel for frequency changing is a
> >> >> > good idea. It would be good to try to implement this. In process of
> >> >> > implementation we will see what is need to be resolved.
> >> >>
> >> >> OK, that's reasonable.
> >> >>
> >> >>
> >> >> > The only question here is how to pass physical cpu to dom0.
> >> >>
> >> >> We can use a device tree based interface to pass the information to
> >> >> dom0, but requiring a number of dom0 vcpus equal to the number of
> >> >> physical cpus and in addition to that having to pin the vcpus each to a
> >> >> different pcpu is quite a stringent limitation. However I don't know the
> >> >> frequency changing interfaces in Linux well enough to know how hard
> >> >> would be to lift it.
> >> >>
> >> >>
> >> >> > Regarding x86.
> >> >> > I'm not sure but maybe ACPI interface encapsulate voltage changing as well?
> >> >>
> >> >> I think so (but I am not an expert on that).
> >> >
> >> > The usual states are P and C states. The P states is the closes to what you
> >> > are looking at:
> >> >
> >> > struct acpi_processor_px {
> >> > u64 core_frequency; /* megahertz */
> >> > u64 power; /* milliWatts */
> >> > u64 transition_latency; /* microseconds */
> >> > u64 bus_master_latency; /* microseconds */
> >> > u64 control; /* control value */
> >> > u64 status; /* success indicator */
> >> > };
> >> >
> >> >>
> >> >>
> >> >>
> >> >> > Regards,
> >> >> > Andrii
> >> >> >
> >> >> >
> >> >> > --
> >> >> >
> >> >> > Andrii Tseglytskyi | Embedded Dev
> >> >> > GlobalLogic
> >> >> > www.globallogic.com
> >> >> >
> >> >>
> >> >> _______________________________________________
> >> >> Xen-devel mailing list
> >> >> Xen-devel@lists.xen.org
> >> >> http://lists.xen.org/xen-devel
> >>
> >>
> >> Cpufreq driver implementation.
> >> ____________
> >> / \
> >> | xenpm tool |
> >> \____________/
> >> Dom0 kernel user-space
> >> ---------------------------------------------------------------------------
> >>
> >> ________________ _____
> >> / \ / \ CPU
> >> | DevTree Parser | /->| ARM | driver
> >> \________________/ | \_____/
> >> Dom0 kernel | |
> >> -----------------------------------------------------|-----|---------------
> >> | |
> >> _____________________________________ | |
> >> | __________ ___________ | | |
> >> | / \ / \ | | |
> >> | | ondemand | | userspace | | | |
> >> Registered | \__________/ \___________/ | | |
> >> cpufreq | _____________ ___________ | | |
> >> governor | / \ / \ | | |
> >> | | performance | | powersave | | | |
> >> | \_____________/ \___________/ | | |
> >> |_____________________________________| | |
> >> ^ | |
> >> | | |
> >> ______|_______ | |
> >> / \ | | Change
> >> | cpufreq core |-------------/ | frequency
> >> \______________/ set/get freq |
> >> commands |
> >> Xen |
> >> -----------------------------------------------------------|--------------
> >> Hardware __V__
> >> | |
> >> | CPU |
> >> |_____|
> >>
> >>
> >> Description of the implementation:
> >> Cpufreq core and registered cpufreq governors are located in xen. Dom0
> >> has CPU driver
> >> which can only change frequency of the physical CPUs. In addition this driver
> >> can change CPUs regulator voltage. I'll reuse some ACPI-specific
> >> variables for ARM.
> >> Thus I can make minimum modification in the xen cpufreq driver and all utilities
> >> (as xenpm) will be working without modification if the xen code. In first
> >> implementation xenpm tool won't show information about C-states, but it can show
> >> information about P-states and can change cpufreq parameters and
> >> change governor.
> >> DevTree parser is a part of the CPU driver in Dom0 and it will read information
> >> from /cpus/cpu@0/private_data path instead of the original /cpus path.
> >>
> >> Steps of the initialization:
> >> 1. Xen copies all cpu@0..cpu@N nodes (from input device tree) with properties to
> >> /cpus/cpu@0/private_data node (device tree for Dom0). Thus we can have
> >> any number
> >> of VCPUs in Dom0 and we give all information about all physical CPUs in
> >> the private_data node.
> >>
> >> 2. Driver in Dom0 will parse /cpus/cpu@0/private_data path instead of the /cpus
> >> path and give the information about CPUs parameters to the hypervisor via
> >> XENPF_set_processor_pminfo hypercall. (Some parameters are calculated in the
> >> Dom0 driver and can not be calculated in the hypervisor).
> >>
> >> 3. Cpufreq core driver in the hypervisor will communicate via some interface
> >> with Dom0 (event channel can be used to notify Dom0) and give some commands
> >> to the CPU driver in Dom0. Those command are set/get frequency, etc.
> >>
> >> Can I implement cpufreq driver in this way?
> >
> > The architecture looks sane to me. As Konrad pointed out, the difficulty
> > here is to be able to upstream the changes to the Linux driver in 2),
> > that you later in the thread identified as
> > drivers/cpufreq/cpufreq-cpu0.c.
> I'll write driver drivers/xen/xen-cpufreq.c and it replace original
> drivers/cpufreq/cpufreq.c
> And in the original cpufreq-cpu0 driver I'll chande only one string -
> path in the device tree
> with the settings for the CPUs opp:
> string
> np = of_find_node_by_path("/cpus/cpu@0");
> will changed to:
> np = of_find_node_by_path("/cpus/cpu@0/private_data/cpu@0");
There is no way that Linux upstream is going to accept code copied
like that. However if you refactor the code from
drivers/cpufreq/cpufreq-cpu0.c like Konrad suggested, moving the
functions we need to a separate file, then you can call into these
function from both cpufreq-cpu0.c and xen-cpufreq.c.
> > If the changes are not invasive and you manage to upstream them in
> > Linux, I am all for this solution.
> In Linux kernel I should make few changes:
> 1. Enable CONFIG_CPU_FREQ_TABLE
> with disabled CONFIG_CPU_FREQ
> 2. Enable CONFIG_GENERIC_CPUFREQ_CPU0
> with disabled CONFIG_CPU_FREQ
>
> I mean make those configs dependent on
> CONFIG_CPU_FREQ or CONFIX_XEN_DOM0
> instead of
> CONFIG_CPU_FREQ
^ permalink raw reply [flat|nested] 57+ messages in thread* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-25 10:14 ` Stefano Stabellini
@ 2014-09-25 11:15 ` Oleksandr Dmytryshyn
0 siblings, 0 replies; 57+ messages in thread
From: Oleksandr Dmytryshyn @ 2014-09-25 11:15 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Ian Campbell, jinsong.liu, Tim Deegan, xen-devel, Jan Beulich,
jacob.shin, Andrii Tseglytskyi
On Thu, Sep 25, 2014 at 1:14 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> On Thu, 25 Sep 2014, Oleksandr Dmytryshyn wrote:
>> On Wed, Sep 24, 2014 at 5:36 PM, Stefano Stabellini
>> <stefano.stabellini@eu.citrix.com> wrote:
>> > On Tue, 16 Sep 2014, Oleksandr Dmytryshyn wrote:
>> >> On Wed, Sep 10, 2014 at 10:31 PM, Konrad Rzeszutek Wilk
>> >> <konrad.wilk@oracle.com> wrote:
>> >> > On Wed, Sep 10, 2014 at 07:35:47PM +0100, Stefano Stabellini wrote:
>> >> >> On Wed, 10 Sep 2014, Andrii Tseglytskyi wrote:
>> >> >> > Hi,
>> >> >> >
>> >> >> > On Wed, Sep 10, 2014 at 12:42 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>> >> >> > >
>> >> >> > > On Tue, 2014-09-09 at 22:41 +0100, Stefano Stabellini wrote:
>> >> >> > > > On Tue, 9 Sep 2014, Ian Campbell wrote:
>> >> >> > > > > On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
>> >> >> > > > > > I am trying to think of an alternative, such as passing the real cpu
>> >> >> > > > > > nodes to dom0 but then adding status = "disabled", but I am not sure
>> >> >> > > > > > whether Linux checks the status for cpu nodes.
>> >> >> > > > >
>> >> >> > > > > status = "disabled" is defined to have a specific (i.e. non-default)
>> >> >> > > > > meaning for cpu nodes, Julien mentioned this when I tried to add a
>> >> >> > > > > similar patch to Xen to ignore them. I think it basically means "present
>> >> >> > > > > but not running, you should start them!".
>> >> >> > > > >
>> >> >> > > > > > In addition this scheme
>> >> >> > > > > > wouldn't support the case where dom0 has more vcpus than pcpus on the
>> >> >> > > > > > system. Granted it is not very common and might even be detrimental for
>> >> >> > > > > > performances, but we should be able to support it.
>> >> >> > > > >
>> >> >> > > > > It's a bit of an edge case, for sure. I guess it wouldn't be totally
>> >> >> > > > > unreasonable to say that if you use this sort of configuration you may
>> >> >> > > > > not get cpufreq support.
>> >> >> > > > >
>> >> >> > > > > > Ian, what do you think about this?
>> >> >> > > > >
>> >> >> > > > > All the options suck in one way or another AFAICT. I think we are going
>> >> >> > > > > to be looking for the least bad solution not necessarily a good one.
>> >> >> > > > >
>> >> >> > > > > Fundamentally are we trying to avoid having to have a i2c subsystem etc
>> >> >> > > > > in the hypervisor to be be able to change the voltages before/after
>> >> >> > > > > changing the frequency?
>> >> >> > > > >
>> >> >> > > > > We can't just say "that's part of the cpufreq driver" since different
>> >> >> > > > > boards using the same SoC might use different voltage regulators, over
>> >> >> > > > > i2c or some other bus etc, so we end up with a matrix.
>> >> >> > > > >
>> >> >> > > > > It's arguable that we should be letting dom0 poke at that regulator
>> >> >> > > > > functionality anyway, at least not all of it. Taking that ability away
>> >> >> > > > > would necessarily imply more platform specific functionality in the
>> >> >> > > > > hypervisor.
>> >> >> > > >
>> >> >> > > > Right.
>> >> >> > > > I am afraid that in order to avoid more code in Xen, we end up with an
>> >> >> > > > unmaintainable interface and unupstreamable hacks in dom0.
>> >> >> > >
>> >> >> > > That's what I'm worried about to. Hence I'm wondering if we should just
>> >> >> > > do this in the hypervisor.
>> >> >> > >
>> >> >> > > Although there are a myriad of them the parts used to do voltage control
>> >> >> > > tend to be fairly simple.
>> >> >> > >
>> >> >> > > One concern I have is that i2c busses also tend to have other things on
>> >> >> > > them which dom0 might legitimately access (e.g. rtc), I'm not sure what
>> >> >> > > to suggest here.
>> >> >> >
>> >> >> > I would try to avoid i2c transactions in Xen. I2C driver is quite
>> >> >> > complicated in Linux kernel. It consists of several parts - common
>> >> >> > core + platform specific. I'm pretty sure Xen should not handle this.
>> >> >> > I think that establishing of event channel for frequency changing is a
>> >> >> > good idea. It would be good to try to implement this. In process of
>> >> >> > implementation we will see what is need to be resolved.
>> >> >>
>> >> >> OK, that's reasonable.
>> >> >>
>> >> >>
>> >> >> > The only question here is how to pass physical cpu to dom0.
>> >> >>
>> >> >> We can use a device tree based interface to pass the information to
>> >> >> dom0, but requiring a number of dom0 vcpus equal to the number of
>> >> >> physical cpus and in addition to that having to pin the vcpus each to a
>> >> >> different pcpu is quite a stringent limitation. However I don't know the
>> >> >> frequency changing interfaces in Linux well enough to know how hard
>> >> >> would be to lift it.
>> >> >>
>> >> >>
>> >> >> > Regarding x86.
>> >> >> > I'm not sure but maybe ACPI interface encapsulate voltage changing as well?
>> >> >>
>> >> >> I think so (but I am not an expert on that).
>> >> >
>> >> > The usual states are P and C states. The P states is the closes to what you
>> >> > are looking at:
>> >> >
>> >> > struct acpi_processor_px {
>> >> > u64 core_frequency; /* megahertz */
>> >> > u64 power; /* milliWatts */
>> >> > u64 transition_latency; /* microseconds */
>> >> > u64 bus_master_latency; /* microseconds */
>> >> > u64 control; /* control value */
>> >> > u64 status; /* success indicator */
>> >> > };
>> >> >
>> >> >>
>> >> >>
>> >> >>
>> >> >> > Regards,
>> >> >> > Andrii
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> >
>> >> >> > Andrii Tseglytskyi | Embedded Dev
>> >> >> > GlobalLogic
>> >> >> > www.globallogic.com
>> >> >> >
>> >> >>
>> >> >> _______________________________________________
>> >> >> Xen-devel mailing list
>> >> >> Xen-devel@lists.xen.org
>> >> >> http://lists.xen.org/xen-devel
>> >>
>> >>
>> >> Cpufreq driver implementation.
>> >> ____________
>> >> / \
>> >> | xenpm tool |
>> >> \____________/
>> >> Dom0 kernel user-space
>> >> ---------------------------------------------------------------------------
>> >>
>> >> ________________ _____
>> >> / \ / \ CPU
>> >> | DevTree Parser | /->| ARM | driver
>> >> \________________/ | \_____/
>> >> Dom0 kernel | |
>> >> -----------------------------------------------------|-----|---------------
>> >> | |
>> >> _____________________________________ | |
>> >> | __________ ___________ | | |
>> >> | / \ / \ | | |
>> >> | | ondemand | | userspace | | | |
>> >> Registered | \__________/ \___________/ | | |
>> >> cpufreq | _____________ ___________ | | |
>> >> governor | / \ / \ | | |
>> >> | | performance | | powersave | | | |
>> >> | \_____________/ \___________/ | | |
>> >> |_____________________________________| | |
>> >> ^ | |
>> >> | | |
>> >> ______|_______ | |
>> >> / \ | | Change
>> >> | cpufreq core |-------------/ | frequency
>> >> \______________/ set/get freq |
>> >> commands |
>> >> Xen |
>> >> -----------------------------------------------------------|--------------
>> >> Hardware __V__
>> >> | |
>> >> | CPU |
>> >> |_____|
>> >>
>> >>
>> >> Description of the implementation:
>> >> Cpufreq core and registered cpufreq governors are located in xen. Dom0
>> >> has CPU driver
>> >> which can only change frequency of the physical CPUs. In addition this driver
>> >> can change CPUs regulator voltage. I'll reuse some ACPI-specific
>> >> variables for ARM.
>> >> Thus I can make minimum modification in the xen cpufreq driver and all utilities
>> >> (as xenpm) will be working without modification if the xen code. In first
>> >> implementation xenpm tool won't show information about C-states, but it can show
>> >> information about P-states and can change cpufreq parameters and
>> >> change governor.
>> >> DevTree parser is a part of the CPU driver in Dom0 and it will read information
>> >> from /cpus/cpu@0/private_data path instead of the original /cpus path.
>> >>
>> >> Steps of the initialization:
>> >> 1. Xen copies all cpu@0..cpu@N nodes (from input device tree) with properties to
>> >> /cpus/cpu@0/private_data node (device tree for Dom0). Thus we can have
>> >> any number
>> >> of VCPUs in Dom0 and we give all information about all physical CPUs in
>> >> the private_data node.
>> >>
>> >> 2. Driver in Dom0 will parse /cpus/cpu@0/private_data path instead of the /cpus
>> >> path and give the information about CPUs parameters to the hypervisor via
>> >> XENPF_set_processor_pminfo hypercall. (Some parameters are calculated in the
>> >> Dom0 driver and can not be calculated in the hypervisor).
>> >>
>> >> 3. Cpufreq core driver in the hypervisor will communicate via some interface
>> >> with Dom0 (event channel can be used to notify Dom0) and give some commands
>> >> to the CPU driver in Dom0. Those command are set/get frequency, etc.
>> >>
>> >> Can I implement cpufreq driver in this way?
>> >
>> > The architecture looks sane to me. As Konrad pointed out, the difficulty
>> > here is to be able to upstream the changes to the Linux driver in 2),
>> > that you later in the thread identified as
>> > drivers/cpufreq/cpufreq-cpu0.c.
>> I'll write driver drivers/xen/xen-cpufreq.c and it replace original
>> drivers/cpufreq/cpufreq.c
>> And in the original cpufreq-cpu0 driver I'll chande only one string -
>> path in the device tree
>> with the settings for the CPUs opp:
>> string
>> np = of_find_node_by_path("/cpus/cpu@0");
>> will changed to:
>> np = of_find_node_by_path("/cpus/cpu@0/private_data/cpu@0");
>
> There is no way that Linux upstream is going to accept code copied
> like that. However if you refactor the code from
> drivers/cpufreq/cpufreq-cpu0.c like Konrad suggested, moving the
> functions we need to a separate file, then you can call into these
> function from both cpufreq-cpu0.c and xen-cpufreq.c.
I'll see what can I do. At first I'll write and check the cpufreq
driver and then
I'll try to upstream patches for Linux and Xen hypervisor.
>
>> > If the changes are not invasive and you manage to upstream them in
>> > Linux, I am all for this solution.
>> In Linux kernel I should make few changes:
>> 1. Enable CONFIG_CPU_FREQ_TABLE
>> with disabled CONFIG_CPU_FREQ
>> 2. Enable CONFIG_GENERIC_CPUFREQ_CPU0
>> with disabled CONFIG_CPU_FREQ
>>
>> I mean make those configs dependent on
>> CONFIG_CPU_FREQ or CONFIX_XEN_DOM0
>> instead of
>> CONFIG_CPU_FREQ
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-25 9:08 ` Oleksandr Dmytryshyn
2014-09-25 10:14 ` Stefano Stabellini
@ 2014-09-26 18:13 ` Konrad Rzeszutek Wilk
2014-09-29 9:45 ` Oleksandr Dmytryshyn
1 sibling, 1 reply; 57+ messages in thread
From: Konrad Rzeszutek Wilk @ 2014-09-26 18:13 UTC (permalink / raw)
To: Oleksandr Dmytryshyn
Cc: Ian Campbell, Stefano Stabellini, jinsong.liu, Tim Deegan,
xen-devel, Jan Beulich, jacob.shin, Andrii Tseglytskyi
> > The architecture looks sane to me. As Konrad pointed out, the difficulty
> > here is to be able to upstream the changes to the Linux driver in 2),
> > that you later in the thread identified as
> > drivers/cpufreq/cpufreq-cpu0.c.
> I'll write driver drivers/xen/xen-cpufreq.c and it replace original
> drivers/cpufreq/cpufreq.c
> And in the original cpufreq-cpu0 driver I'll chande only one string -
> path in the device tree
> with the settings for the CPUs opp:
> string
> np = of_find_node_by_path("/cpus/cpu@0");
> will changed to:
> np = of_find_node_by_path("/cpus/cpu@0/private_data/cpu@0");
>
> > If the changes are not invasive and you manage to upstream them in
> > Linux, I am all for this solution.
> In Linux kernel I should make few changes:
> 1. Enable CONFIG_CPU_FREQ_TABLE
> with disabled CONFIG_CPU_FREQ
> 2. Enable CONFIG_GENERIC_CPUFREQ_CPU0
> with disabled CONFIG_CPU_FREQ
>
> I mean make those configs dependent on
> CONFIG_CPU_FREQ or CONFIX_XEN_DOM0
> instead of
> CONFIG_CPU_FREQ
Gosh no. Please make it work runtime.
>
> Oleksandr Dmytryshyn | Product Engineering and Development
> GlobalLogic
> M +38.067.382.2525
> www.globallogic.com
>
> http://www.globallogic.com/email_disclaimer.txt
^ permalink raw reply [flat|nested] 57+ messages in thread* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-26 18:13 ` Konrad Rzeszutek Wilk
@ 2014-09-29 9:45 ` Oleksandr Dmytryshyn
2014-09-29 15:18 ` Konrad Rzeszutek Wilk
0 siblings, 1 reply; 57+ messages in thread
From: Oleksandr Dmytryshyn @ 2014-09-29 9:45 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk
Cc: Ian Campbell, Stefano Stabellini, jinsong.liu, Tim Deegan,
xen-devel, Jan Beulich, jacob.shin, Andrii Tseglytskyi
On Fri, Sep 26, 2014 at 9:13 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
>
> > > The architecture looks sane to me. As Konrad pointed out, the difficulty
> > > here is to be able to upstream the changes to the Linux driver in 2),
> > > that you later in the thread identified as
> > > drivers/cpufreq/cpufreq-cpu0.c.
> > I'll write driver drivers/xen/xen-cpufreq.c and it replace original
> > drivers/cpufreq/cpufreq.c
> > And in the original cpufreq-cpu0 driver I'll chande only one string -
> > path in the device tree
> > with the settings for the CPUs opp:
> > string
> > np = of_find_node_by_path("/cpus/cpu@0");
> > will changed to:
> > np = of_find_node_by_path("/cpus/cpu@0/private_data/cpu@0");
> >
> > > If the changes are not invasive and you manage to upstream them in
> > > Linux, I am all for this solution.
> > In Linux kernel I should make few changes:
> > 1. Enable CONFIG_CPU_FREQ_TABLE
> > with disabled CONFIG_CPU_FREQ
> > 2. Enable CONFIG_GENERIC_CPUFREQ_CPU0
> > with disabled CONFIG_CPU_FREQ
> >
> > I mean make those configs dependent on
> > CONFIG_CPU_FREQ or CONFIX_XEN_DOM0
> > instead of
> > CONFIG_CPU_FREQ
>
> Gosh no. Please make it work runtime.
Sorry, Konrad, could You please explain what did You mean.
> >
> > Oleksandr Dmytryshyn | Product Engineering and Development
> > GlobalLogic
> > M +38.067.382.2525
> > www.globallogic.com
> >
> > http://www.globallogic.com/email_disclaimer.txt
Oleksandr Dmytryshyn | Product Engineering and Development
GlobalLogic
M +38.067.382.2525
www.globallogic.com
http://www.globallogic.com/email_disclaimer.txt
^ permalink raw reply [flat|nested] 57+ messages in thread* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-29 9:45 ` Oleksandr Dmytryshyn
@ 2014-09-29 15:18 ` Konrad Rzeszutek Wilk
2014-09-30 10:28 ` Oleksandr Dmytryshyn
0 siblings, 1 reply; 57+ messages in thread
From: Konrad Rzeszutek Wilk @ 2014-09-29 15:18 UTC (permalink / raw)
To: Oleksandr Dmytryshyn
Cc: Ian Campbell, Stefano Stabellini, jinsong.liu, Tim Deegan,
xen-devel, Jan Beulich, jacob.shin, Andrii Tseglytskyi
On Mon, Sep 29, 2014 at 12:45:36PM +0300, Oleksandr Dmytryshyn wrote:
> On Fri, Sep 26, 2014 at 9:13 PM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
> >
> > > > The architecture looks sane to me. As Konrad pointed out, the difficulty
> > > > here is to be able to upstream the changes to the Linux driver in 2),
> > > > that you later in the thread identified as
> > > > drivers/cpufreq/cpufreq-cpu0.c.
> > > I'll write driver drivers/xen/xen-cpufreq.c and it replace original
> > > drivers/cpufreq/cpufreq.c
> > > And in the original cpufreq-cpu0 driver I'll chande only one string -
> > > path in the device tree
> > > with the settings for the CPUs opp:
> > > string
> > > np = of_find_node_by_path("/cpus/cpu@0");
> > > will changed to:
> > > np = of_find_node_by_path("/cpus/cpu@0/private_data/cpu@0");
> > >
> > > > If the changes are not invasive and you manage to upstream them in
> > > > Linux, I am all for this solution.
> > > In Linux kernel I should make few changes:
> > > 1. Enable CONFIG_CPU_FREQ_TABLE
> > > with disabled CONFIG_CPU_FREQ
> > > 2. Enable CONFIG_GENERIC_CPUFREQ_CPU0
> > > with disabled CONFIG_CPU_FREQ
> > >
> > > I mean make those configs dependent on
> > > CONFIG_CPU_FREQ or CONFIX_XEN_DOM0
> > > instead of
> > > CONFIG_CPU_FREQ
> >
> > Gosh no. Please make it work runtime.
> Sorry, Konrad, could You please explain what did You mean.
Don't make XEN options disable other options.
Distributions want one kernel that can satisfi a variety of platforms - not just specifc ones.
^ permalink raw reply [flat|nested] 57+ messages in thread* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-29 15:18 ` Konrad Rzeszutek Wilk
@ 2014-09-30 10:28 ` Oleksandr Dmytryshyn
2014-09-30 17:37 ` Konrad Rzeszutek Wilk
0 siblings, 1 reply; 57+ messages in thread
From: Oleksandr Dmytryshyn @ 2014-09-30 10:28 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk
Cc: Ian Campbell, Stefano Stabellini, jinsong.liu, Tim Deegan,
xen-devel, Jan Beulich, jacob.shin, Andrii Tseglytskyi
On Mon, Sep 29, 2014 at 6:18 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
>
> On Mon, Sep 29, 2014 at 12:45:36PM +0300, Oleksandr Dmytryshyn wrote:
> > On Fri, Sep 26, 2014 at 9:13 PM, Konrad Rzeszutek Wilk
> > <konrad.wilk@oracle.com> wrote:
> > >
> > > > > The architecture looks sane to me. As Konrad pointed out, the difficulty
> > > > > here is to be able to upstream the changes to the Linux driver in 2),
> > > > > that you later in the thread identified as
> > > > > drivers/cpufreq/cpufreq-cpu0.c.
> > > > I'll write driver drivers/xen/xen-cpufreq.c and it replace original
> > > > drivers/cpufreq/cpufreq.c
> > > > And in the original cpufreq-cpu0 driver I'll chande only one string -
> > > > path in the device tree
> > > > with the settings for the CPUs opp:
> > > > string
> > > > np = of_find_node_by_path("/cpus/cpu@0");
> > > > will changed to:
> > > > np = of_find_node_by_path("/cpus/cpu@0/private_data/cpu@0");
> > > >
> > > > > If the changes are not invasive and you manage to upstream them in
> > > > > Linux, I am all for this solution.
> > > > In Linux kernel I should make few changes:
> > > > 1. Enable CONFIG_CPU_FREQ_TABLE
> > > > with disabled CONFIG_CPU_FREQ
> > > > 2. Enable CONFIG_GENERIC_CPUFREQ_CPU0
> > > > with disabled CONFIG_CPU_FREQ
> > > >
> > > > I mean make those configs dependent on
> > > > CONFIG_CPU_FREQ or CONFIX_XEN_DOM0
> > > > instead of
> > > > CONFIG_CPU_FREQ
> > >
> > > Gosh no. Please make it work runtime.
> > Sorry, Konrad, could You please explain what did You mean.
>
> Don't make XEN options disable other options.
>
> Distributions want one kernel that can satisfi a variety of platforms - not just specifc ones.
XEN option will not disable other options. On the contrary, it will
extend them. In my case XEN option
will allow to select more options.
Please, see example
diff --git a/drivers/cpufreq/Kconfig b/drivers/cpufreq/Kconfig
index cbcb21e..4531e04 100644
--- a/drivers/cpufreq/Kconfig
+++ b/drivers/cpufreq/Kconfig
@@ -15,11 +15,15 @@ config CPU_FREQ
If in doubt, say N.
-if CPU_FREQ
+if CPU_FREQ || XEN_DOM0
config CPU_FREQ_TABLE
tristate
@@ -184,6 +188,10 @@ config CPU_FREQ_GOV_CONSERVATIVE
If in doubt, say N.
+if CPU_FREQ || XEN_DOM0
+
config GENERIC_CPUFREQ_CPU0
tristate "Generic CPU0 cpufreq driver"
depends on HAVE_CLK && REGULATOR && PM_OPP && OF
^ permalink raw reply related [flat|nested] 57+ messages in thread* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-30 10:28 ` Oleksandr Dmytryshyn
@ 2014-09-30 17:37 ` Konrad Rzeszutek Wilk
0 siblings, 0 replies; 57+ messages in thread
From: Konrad Rzeszutek Wilk @ 2014-09-30 17:37 UTC (permalink / raw)
To: Oleksandr Dmytryshyn
Cc: Ian Campbell, Stefano Stabellini, jinsong.liu, Tim Deegan,
xen-devel, Jan Beulich, jacob.shin, Andrii Tseglytskyi
On Tue, Sep 30, 2014 at 01:28:42PM +0300, Oleksandr Dmytryshyn wrote:
> On Mon, Sep 29, 2014 at 6:18 PM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
> >
> > On Mon, Sep 29, 2014 at 12:45:36PM +0300, Oleksandr Dmytryshyn wrote:
> > > On Fri, Sep 26, 2014 at 9:13 PM, Konrad Rzeszutek Wilk
> > > <konrad.wilk@oracle.com> wrote:
> > > >
> > > > > > The architecture looks sane to me. As Konrad pointed out, the difficulty
> > > > > > here is to be able to upstream the changes to the Linux driver in 2),
> > > > > > that you later in the thread identified as
> > > > > > drivers/cpufreq/cpufreq-cpu0.c.
> > > > > I'll write driver drivers/xen/xen-cpufreq.c and it replace original
> > > > > drivers/cpufreq/cpufreq.c
> > > > > And in the original cpufreq-cpu0 driver I'll chande only one string -
> > > > > path in the device tree
> > > > > with the settings for the CPUs opp:
> > > > > string
> > > > > np = of_find_node_by_path("/cpus/cpu@0");
> > > > > will changed to:
> > > > > np = of_find_node_by_path("/cpus/cpu@0/private_data/cpu@0");
> > > > >
> > > > > > If the changes are not invasive and you manage to upstream them in
> > > > > > Linux, I am all for this solution.
> > > > > In Linux kernel I should make few changes:
> > > > > 1. Enable CONFIG_CPU_FREQ_TABLE
> > > > > with disabled CONFIG_CPU_FREQ
> > > > > 2. Enable CONFIG_GENERIC_CPUFREQ_CPU0
> > > > > with disabled CONFIG_CPU_FREQ
> > > > >
> > > > > I mean make those configs dependent on
> > > > > CONFIG_CPU_FREQ or CONFIX_XEN_DOM0
> > > > > instead of
> > > > > CONFIG_CPU_FREQ
> > > >
> > > > Gosh no. Please make it work runtime.
> > > Sorry, Konrad, could You please explain what did You mean.
> >
> > Don't make XEN options disable other options.
> >
> > Distributions want one kernel that can satisfi a variety of platforms - not just specifc ones.
> XEN option will not disable other options. On the contrary, it will
> extend them. In my case XEN option
> will allow to select more options.
>
> Please, see example
> diff --git a/drivers/cpufreq/Kconfig b/drivers/cpufreq/Kconfig
> index cbcb21e..4531e04 100644
> --- a/drivers/cpufreq/Kconfig
> +++ b/drivers/cpufreq/Kconfig
> @@ -15,11 +15,15 @@ config CPU_FREQ
>
> If in doubt, say N.
>
> -if CPU_FREQ
> +if CPU_FREQ || XEN_DOM0
>
> config CPU_FREQ_TABLE
> tristate
>
> @@ -184,6 +188,10 @@ config CPU_FREQ_GOV_CONSERVATIVE
>
> If in doubt, say N.
>
> +if CPU_FREQ || XEN_DOM0
> +
Ah, then it is fine. Sorry for the misunderstanding.
> config GENERIC_CPUFREQ_CPU0
> tristate "Generic CPU0 cpufreq driver"
> depends on HAVE_CLK && REGULATOR && PM_OPP && OF
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-10 18:35 ` Stefano Stabellini
2014-09-10 19:31 ` Konrad Rzeszutek Wilk
@ 2014-09-11 9:43 ` Ian Campbell
1 sibling, 0 replies; 57+ messages in thread
From: Ian Campbell @ 2014-09-11 9:43 UTC (permalink / raw)
To: Stefano Stabellini
Cc: jinsong.liu, Oleksandr Dmytryshyn, Tim Deegan, xen-devel,
Jan Beulich, jacob.shin, Andrii Tseglytskyi
On Wed, 2014-09-10 at 19:35 +0100, Stefano Stabellini wrote:
> > > One concern I have is that i2c busses also tend to have other things on
> > > them which dom0 might legitimately access (e.g. rtc), I'm not sure what
> > > to suggest here.
> >
> > I would try to avoid i2c transactions in Xen. I2C driver is quite
> > complicated in Linux kernel. It consists of several parts - common
> > core + platform specific. I'm pretty sure Xen should not handle this.
> > I think that establishing of event channel for frequency changing is a
> > good idea. It would be good to try to implement this. In process of
> > implementation we will see what is need to be resolved.
>
> OK, that's reasonable.
I don't think a single purpose in-Xen i2c would need quite so much
infrastructure as what Linux provides, so it could be a whole lot
simpler, I think, so I wouldn't necessarily rule it out without trying
it.
> > The only question here is how to pass physical cpu to dom0.
>
> We can use a device tree based interface to pass the information to
> dom0, but requiring a number of dom0 vcpus equal to the number of
> physical cpus and in addition to that having to pin the vcpus each to a
> different pcpu is quite a stringent limitation. However I don't know the
> frequency changing interfaces in Linux well enough to know how hard
> would be to lift it.
>
>
> > Regarding x86.
> > I'm not sure but maybe ACPI interface encapsulate voltage changing as well?
>
> I think so (but I am not an expert on that).
Likewise.
Ian.
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-04 21:56 ` Stefano Stabellini
2014-09-09 10:19 ` Oleksandr Dmytryshyn
2014-09-09 10:32 ` Ian Campbell
@ 2014-09-09 13:25 ` Vitaly Chernooky
2014-09-09 21:58 ` Stefano Stabellini
2 siblings, 1 reply; 57+ messages in thread
From: Vitaly Chernooky @ 2014-09-09 13:25 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Ian Campbell, jinsong.liu, Oleksandr Dmytryshyn, Tim Deegan,
xen-devel, Jan Beulich, jacob.shin, Andrii Tseglytskyi
[-- Attachment #1.1: Type: text/plain, Size: 7545 bytes --]
Hi All!
On Fri, Sep 5, 2014 at 12:56 AM, Stefano Stabellini <
stefano.stabellini@eu.citrix.com> wrote:
> On Thu, 4 Sep 2014, Oleksandr Dmytryshyn wrote:
> > Hi to all.
> >
> > I want to implement cpufreq driver in the next way:
> > 1. Cpufreq governor will be implemented in the Xen
> > 2. dom0 will only change cpu frequency and voltage of the physical cpus
> >
> > But there are some nuances:
> > 1. dom0 driver should read an information about operation points
> > (frequencies and voltages) and cpu supply source from the device tree
> for each
> > physical cpu. In the omap processor case this driver suspects that
> > those settings
> > located in the /cpus/cpu@0/ node. But hypervisor creates an cpu node
> > for each vcpu
> > for kernel dom0 in the device tree and those information is lost in the
> dom0.
> > 2. What about this case if we will have some physical cpus with different
> > operation points (for example 2 cpus) and we give only one cpu for dom0?
> >
> > How should I transfer all information from the original cpu@0..cpu@n
> nodes
> > about all physical cpus to the kernel dom0 driver? Maybe an additional
> > nodes should be created by the hypervisor in the device tree for dom0
> > and named as pcpu@0..pcpu@n?
>
> If we do that, wouldn't we require changes to the core OMAP drivers or
> cpu initialization code in Linux (to parse "pcpu" instead of "cpu"
> nodes)? I don't expect they would be easy to upstream or maintain going
> forward.
>
> I am trying to think of an alternative, such as passing the real cpu
> nodes to dom0 but then adding status = "disabled", but I am not sure
> whether Linux checks the status for cpu nodes. In addition this scheme
> wouldn't support the case where dom0 has more vcpus than pcpus on the
> system. Granted it is not very common and might even be detrimental for
> performances, but we should be able to support it.
>
In case where dom0 has more vcpus than pcpus on the
system, the dom0 kernel *is the most bug-prone place* for pcpu cpufreq
governor. So I still believe that separate driver domain with direct 1:1
vcpu:pcpu mapping is the best place for cpufreq governor. But it also
reasonable to run cpufreq governor as userspace daemon in dom0.
Also what do you think about PM QoS support? On bare metal cpufreq is
tightly integrated with PM QoS and intensively cooperate in frequency
scaling.
With best regards,
> Ian, what do you think about this?
>
>
>
> > Oleksandr Dmytryshyn | Product Engineering and Development
> > GlobalLogic
> > M +38.067.382.2525
> > www.globallogic.com
> >
> > http://www.globallogic.com/email_disclaimer.txt
> >
> >
> > On Tue, Sep 2, 2014 at 9:46 PM, Andrii Tseglytskyi
> > <andrii.tseglytskyi@globallogic.com> wrote:
> > >
> > > Hi Stefano,
> > >
> > > Thank you for explanation.
> > > I think this requires more and deeper investigation, but for sure dom0
> > > must be able to do this.
> > > Let us investigate this.
> > >
> > > Thank you,
> > >
> > > Regards,
> > > Andrii
> > >
> > > On Tue, Sep 2, 2014 at 9:39 PM, Stefano Stabellini
> > > <stefano.stabellini@eu.citrix.com> wrote:
> > > > On Tue, 2 Sep 2014, Andrii Tseglytskyi wrote:
> > > >> On Tue, Sep 2, 2014 at 4:00 AM, Stefano Stabellini
> > > >> <stefano.stabellini@eu.citrix.com> wrote:
> > > >> > On Fri, 29 Aug 2014, Andrii Tseglytskyi wrote:
> > > >> >> Hi,
> > > >> >>
> > > >> >> Stefano, Ian,
> > > >> >>
> > > >> >> Could you please clarify the following point:
> > > >> >>
> > > >> >> I agree that decision about frequency change should be taken by
> Xen
> > > >> >> hypervisor. But what about hardware frequency changing?
> > > >> >> In general when frequency changed to bigger value (for example
> from 1
> > > >> >> GHz to 1.5 GHz) for ARM kernels sequence looks like the
> following:
> > > >> >>
> > > >> >> 1) cpufreq governor decides that frequency should be changed.
> This
> > > >> >> decision is taken after analysing of CPU performance data taking
> in
> > > >> >> account governor policy.
> > > >> >> 2) cpufreq governor asks cpufreq driver about new frequency.
> > > >> >> 3) cpufreq driver compares current and target frequencies and
> asks
> > > >> >> cpufreq regulator about voltage change.
> > > >> >> 4) cpufreq regulator send i2c command to standalone microchip,
> which
> > > >> >> is responsible for voltage changing.
> > > >> >> 5) cpufreq driver asks clock framework about new frequency for
> CPU clock
> > > >> >> 6) clock framework performs frequency sanity checks, taking in
> account
> > > >> >> clock parents and clock divider settings, and call platform
> specific
> > > >> >> "set_frequency" callback.
> > > >> >> 7) platform specific callback performs proper HW registers
> > > >> >> configuration for newly selected frequency
> > > >> >>
> > > >> >> Also there are some special cases - for example for OMAP5+ when
> > > >> >> frequency is changed to 1.5 GHz+, two additional HW IPs should be
> > > >> >> triggered (ABB and DCC, if someone is familiar with OMAP5+ )
> > > >> >>
> > > >> >> So, for generic ARM kernel we have 3 entities to change
> frequency:
> > > >> >>
> > > >> >> - cpufreq governor
> > > >> >> - cpufreq driver
> > > >> >> - cpufreq regulator
> > > >> >>
> > > >> >> + 2 additional IP for OMAP5+
> > > >> >> - ABB
> > > >> >> - DCC
> > > >> >>
> > > >> >> Taking in account all above, it looks like it would be better to
> > > >> >> implement only Xen cpufreq governor. Xen will take a decision
> about
> > > >> >> new frequency, and kernel dom0 will perform other steps. Dom0
> contains
> > > >> >> all generic and platform specific frameworks, needed for
> frequency
> > > >> >> changing.
> > > >> >>
> > > >> >> What do you think ?
> > > >> >
> > > >> > Keep in mind that the architecture must be able to handle the
> case where
> > > >> > dom0 has only 1 or 2 vcpus on a 4 or 8 cores system with multiple
> > > >> > physical cpus.
> > > >> > Could dom0 change the frequency of a physical core or a physical
> cpu is
> > > >> > not even running on? If that is not a problem, because cpus and
> > > >> > frequency changing are decoupled enough in Linux to allow it,
> then I am
> > > >> > OK with it. But I suspect they are not.
> > > >> >
> > > >>
> > > >> Not sure that I got your point correctly - dom0 will change
> frequency
> > > >> on physical CPU.
> > > >> And in case of OMAP - this changing affects on both ARM physical
> cpus
> > > >> - changing is coupled.
> > > >> In case of other ARM platforms - changing may be not coupled (I've
> > > >> heard that Snapdragon can change cpu freqs independently on each
> > > >> physical cpu)
> > > >
> > > > Let me explain with a concrete example.
> > > >
> > > > Let's suppose that the platform has 2 physical cpus, each cpu has 4
> > > > cores. Let's also supposed that dom0 has only 2 vcpus, currently
> > > > running on core0 and core1 of cpu0.
> > > >
> > > > In this case would dom0 be able to change the frequency of core3 of
> > > > cpu1, given that is not even running on it?
> > > > If it can be done without any hacks, then we can go ahead with this
> > > > approach.
> > >
> > >
> > >
> > > --
> > >
> > > Andrii Tseglytskyi | Embedded Dev
> > > GlobalLogic
> > > www.globallogic.com
> >
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>
--
*Vitaly Chernooky | Senior Developer - Product Engineering and Development*
GlobalLogic
P *+380.44.4929695 ext.1136* M *+380.98.7920568* S cvv_2k
www.globallogic.com
http://www.globallogic.com/email_disclaimer.txt
[-- Attachment #1.2: Type: text/html, Size: 11085 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-09 13:25 ` Vitaly Chernooky
@ 2014-09-09 21:58 ` Stefano Stabellini
2014-09-10 10:15 ` Andrii Tseglytskyi
2014-09-10 14:28 ` Vitaly Chernooky
0 siblings, 2 replies; 57+ messages in thread
From: Stefano Stabellini @ 2014-09-09 21:58 UTC (permalink / raw)
To: Vitaly Chernooky
Cc: Ian Campbell, Stefano Stabellini, jinsong.liu,
Oleksandr Dmytryshyn, Tim Deegan, xen-devel, Jan Beulich,
Anthony.Perard, jacob.shin, Andrii Tseglytskyi
[-- Attachment #1: Type: text/plain, Size: 9159 bytes --]
On Tue, 9 Sep 2014, Vitaly Chernooky wrote:
> Hi All!
>
> On Fri, Sep 5, 2014 at 12:56 AM, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:
> On Thu, 4 Sep 2014, Oleksandr Dmytryshyn wrote:
> > Hi to all.
> >
> > I want to implement cpufreq driver in the next way:
> > 1. Cpufreq governor will be implemented in the Xen
> > 2. dom0 will only change cpu frequency and voltage of the physical cpus
> >
> > But there are some nuances:
> > 1. dom0 driver should read an information about operation points
> > (frequencies and voltages) and cpu supply source from the device tree for each
> > physical cpu. In the omap processor case this driver suspects that
> > those settings
> > located in the /cpus/cpu@0/ node. But hypervisor creates an cpu node
> > for each vcpu
> > for kernel dom0 in the device tree and those information is lost in the dom0.
> > 2. What about this case if we will have some physical cpus with different
> > operation points (for example 2 cpus) and we give only one cpu for dom0?
> >
> > How should I transfer all information from the original cpu@0..cpu@n nodes
> > about all physical cpus to the kernel dom0 driver? Maybe an additional
> > nodes should be created by the hypervisor in the device tree for dom0
> > and named as pcpu@0..pcpu@n?
>
> If we do that, wouldn't we require changes to the core OMAP drivers or
> cpu initialization code in Linux (to parse "pcpu" instead of "cpu"
> nodes)? I don't expect they would be easy to upstream or maintain going
> forward.
>
> I am trying to think of an alternative, such as passing the real cpu
> nodes to dom0 but then adding status = "disabled", but I am not sure
> whether Linux checks the status for cpu nodes. In addition this scheme
> wouldn't support the case where dom0 has more vcpus than pcpus on the
> system. Granted it is not very common and might even be detrimental for
> performances, but we should be able to support it.
>
>
> In case where dom0 has more vcpus than pcpus on the
> system, the dom0 kernel is the most bug-prone place for pcpu cpufreq governor. So I still believe that separate driver
> domain with direct 1:1 vcpu:pcpu mapping is the best place for cpufreq governor. But it also reasonable to run cpufreq
> governor as userspace daemon in dom0.
>
> Also what do you think about PM QoS support? On bare metal cpufreq is tightly integrated with PM QoS and intensively
> cooperate in frequency scaling.
Device PM needs to be done in Dom0.
CPU an Platform level PM architecturally belongs to Xen, but I do
understand that to do that in Xen we would need to add lots of code to
the hypervisor. There is no silver bullet here.
A driver domain with 1:1 vcpu:pcpu mapping could work, but what kernel
are you going to use for that? Linux? Wouldn't Linux be too big for a
cpufreq driver domain, especially in embedded deployments? I think it
would need at least 32MB to run.
> With best regards,
>
> Ian, what do you think about this?
>
>
>
> > Oleksandr Dmytryshyn | Product Engineering and Development
> > GlobalLogic
> > M +38.067.382.2525
> > www.globallogic.com
> >
> > http://www.globallogic.com/email_disclaimer.txt
> >
> >
> > On Tue, Sep 2, 2014 at 9:46 PM, Andrii Tseglytskyi
> > <andrii.tseglytskyi@globallogic.com> wrote:
> > >
> > > Hi Stefano,
> > >
> > > Thank you for explanation.
> > > I think this requires more and deeper investigation, but for sure dom0
> > > must be able to do this.
> > > Let us investigate this.
> > >
> > > Thank you,
> > >
> > > Regards,
> > > Andrii
> > >
> > > On Tue, Sep 2, 2014 at 9:39 PM, Stefano Stabellini
> > > <stefano.stabellini@eu.citrix.com> wrote:
> > > > On Tue, 2 Sep 2014, Andrii Tseglytskyi wrote:
> > > >> On Tue, Sep 2, 2014 at 4:00 AM, Stefano Stabellini
> > > >> <stefano.stabellini@eu.citrix.com> wrote:
> > > >> > On Fri, 29 Aug 2014, Andrii Tseglytskyi wrote:
> > > >> >> Hi,
> > > >> >>
> > > >> >> Stefano, Ian,
> > > >> >>
> > > >> >> Could you please clarify the following point:
> > > >> >>
> > > >> >> I agree that decision about frequency change should be taken by Xen
> > > >> >> hypervisor. But what about hardware frequency changing?
> > > >> >> In general when frequency changed to bigger value (for example from 1
> > > >> >> GHz to 1.5 GHz) for ARM kernels sequence looks like the following:
> > > >> >>
> > > >> >> 1) cpufreq governor decides that frequency should be changed. This
> > > >> >> decision is taken after analysing of CPU performance data taking in
> > > >> >> account governor policy.
> > > >> >> 2) cpufreq governor asks cpufreq driver about new frequency.
> > > >> >> 3) cpufreq driver compares current and target frequencies and asks
> > > >> >> cpufreq regulator about voltage change.
> > > >> >> 4) cpufreq regulator send i2c command to standalone microchip, which
> > > >> >> is responsible for voltage changing.
> > > >> >> 5) cpufreq driver asks clock framework about new frequency for CPU clock
> > > >> >> 6) clock framework performs frequency sanity checks, taking in account
> > > >> >> clock parents and clock divider settings, and call platform specific
> > > >> >> "set_frequency" callback.
> > > >> >> 7) platform specific callback performs proper HW registers
> > > >> >> configuration for newly selected frequency
> > > >> >>
> > > >> >> Also there are some special cases - for example for OMAP5+ when
> > > >> >> frequency is changed to 1.5 GHz+, two additional HW IPs should be
> > > >> >> triggered (ABB and DCC, if someone is familiar with OMAP5+ )
> > > >> >>
> > > >> >> So, for generic ARM kernel we have 3 entities to change frequency:
> > > >> >>
> > > >> >> - cpufreq governor
> > > >> >> - cpufreq driver
> > > >> >> - cpufreq regulator
> > > >> >>
> > > >> >> + 2 additional IP for OMAP5+
> > > >> >> - ABB
> > > >> >> - DCC
> > > >> >>
> > > >> >> Taking in account all above, it looks like it would be better to
> > > >> >> implement only Xen cpufreq governor. Xen will take a decision about
> > > >> >> new frequency, and kernel dom0 will perform other steps. Dom0 contains
> > > >> >> all generic and platform specific frameworks, needed for frequency
> > > >> >> changing.
> > > >> >>
> > > >> >> What do you think ?
> > > >> >
> > > >> > Keep in mind that the architecture must be able to handle the case where
> > > >> > dom0 has only 1 or 2 vcpus on a 4 or 8 cores system with multiple
> > > >> > physical cpus.
> > > >> > Could dom0 change the frequency of a physical core or a physical cpu is
> > > >> > not even running on? If that is not a problem, because cpus and
> > > >> > frequency changing are decoupled enough in Linux to allow it, then I am
> > > >> > OK with it. But I suspect they are not.
> > > >> >
> > > >>
> > > >> Not sure that I got your point correctly - dom0 will change frequency
> > > >> on physical CPU.
> > > >> And in case of OMAP - this changing affects on both ARM physical cpus
> > > >> - changing is coupled.
> > > >> In case of other ARM platforms - changing may be not coupled (I've
> > > >> heard that Snapdragon can change cpu freqs independently on each
> > > >> physical cpu)
> > > >
> > > > Let me explain with a concrete example.
> > > >
> > > > Let's suppose that the platform has 2 physical cpus, each cpu has 4
> > > > cores. Let's also supposed that dom0 has only 2 vcpus, currently
> > > > running on core0 and core1 of cpu0.
> > > >
> > > > In this case would dom0 be able to change the frequency of core3 of
> > > > cpu1, given that is not even running on it?
> > > > If it can be done without any hacks, then we can go ahead with this
> > > > approach.
> > >
> > >
> > >
> > > --
> > >
> > > Andrii Tseglytskyi | Embedded Dev
> > > GlobalLogic
> > > www.globallogic.com
> >
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>
>
>
>
> --
> Vitaly Chernooky | Senior Developer - Product Engineering and Development
> GlobalLogic
> P +380.44.4929695 ext.1136 M +380.98.7920568 S cvv_2k
> www.globallogic.com
>
> http://www.globallogic.com/email_disclaimer.txt
>
>
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-09 21:58 ` Stefano Stabellini
@ 2014-09-10 10:15 ` Andrii Tseglytskyi
2014-09-10 10:24 ` Vitaly Chernooky
2014-09-10 11:24 ` Vitaly Chernooky
2014-09-10 14:28 ` Vitaly Chernooky
1 sibling, 2 replies; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-09-10 10:15 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Vitaly Chernooky, Ian Campbell, jinsong.liu, Oleksandr Dmytryshyn,
Tim Deegan, xen-devel, Jan Beulich, Anthony.Perard, jacob.shin
Hi,
>> In case where dom0 has more vcpus than pcpus on the
>> system, the dom0 kernel is the most bug-prone place for pcpu cpufreq governor. So I still believe that separate driver
>> domain with direct 1:1 vcpu:pcpu mapping is the best place for cpufreq governor. But it also reasonable to run cpufreq
>> governor as userspace daemon in dom0.
>>
>> Also what do you think about PM QoS support? On bare metal cpufreq is tightly integrated with PM QoS and intensively
>> cooperate in frequency scaling.
>
> Device PM needs to be done in Dom0.
> CPU an Platform level PM architecturally belongs to Xen, but I do
> understand that to do that in Xen we would need to add lots of code to
> the hypervisor. There is no silver bullet here.
>
> A driver domain with 1:1 vcpu:pcpu mapping could work, but what kernel
> are you going to use for that? Linux? Wouldn't Linux be too big for a
> cpufreq driver domain, especially in embedded deployments? I think it
> would need at least 32MB to run.
>
I would suggest not to do this. Driver domain will need to share the
same HW with dom0, which is impossible to implement. Good examples are
I2C, which is needed for Display and for Cpufreq, and it cannot be
shared between domains. (At least in current Linux kernel it won't be
easy to implement). dom0 is the best place to do cpufreq.
Regards,
Andrii
--
Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-10 10:15 ` Andrii Tseglytskyi
@ 2014-09-10 10:24 ` Vitaly Chernooky
2014-09-10 11:18 ` Andrii Tseglytskyi
2014-09-10 18:37 ` Stefano Stabellini
2014-09-10 11:24 ` Vitaly Chernooky
1 sibling, 2 replies; 57+ messages in thread
From: Vitaly Chernooky @ 2014-09-10 10:24 UTC (permalink / raw)
To: Andrii Tseglytskyi
Cc: Ian Campbell, Stefano Stabellini, jinsong.liu,
Oleksandr Dmytryshyn, Tim Deegan, xen-devel, Jan Beulich,
Anthony.Perard, jacob.shin
[-- Attachment #1.1: Type: text/plain, Size: 1896 bytes --]
Hi Andrii,
On Wed, Sep 10, 2014 at 1:15 PM, Andrii Tseglytskyi <
andrii.tseglytskyi@globallogic.com> wrote:
> Hi,
>
> >> In case where dom0 has more vcpus than pcpus on the
> >> system, the dom0 kernel is the most bug-prone place for pcpu cpufreq
> governor. So I still believe that separate driver
> >> domain with direct 1:1 vcpu:pcpu mapping is the best place for cpufreq
> governor. But it also reasonable to run cpufreq
> >> governor as userspace daemon in dom0.
> >>
> >> Also what do you think about PM QoS support? On bare metal cpufreq is
> tightly integrated with PM QoS and intensively
> >> cooperate in frequency scaling.
> >
> > Device PM needs to be done in Dom0.
> > CPU an Platform level PM architecturally belongs to Xen, but I do
> > understand that to do that in Xen we would need to add lots of code to
> > the hypervisor. There is no silver bullet here.
> >
> > A driver domain with 1:1 vcpu:pcpu mapping could work, but what kernel
> > are you going to use for that? Linux? Wouldn't Linux be too big for a
> > cpufreq driver domain, especially in embedded deployments? I think it
> > would need at least 32MB to run.
> >
>
> I would suggest not to do this. Driver domain will need to share the
> same HW with dom0, which is impossible to implement. Good examples are
> I2C, which is needed for Display and for Cpufreq, and it cannot be
> shared between domains. (At least in current Linux kernel it won't be
> easy to implement). dom0 is the best place to do cpufreq.
>
What do you think about cpufreq in dom0 userspace?
With best regards,
> Regards,
> Andrii
>
>
> --
>
> Andrii Tseglytskyi | Embedded Dev
> GlobalLogic
> www.globallogic.com
>
--
*Vitaly Chernooky | Senior Developer - Product Engineering and Development*
GlobalLogic
P *+380.44.4929695 ext.1136* M *+380.98.7920568* S cvv_2k
www.globallogic.com
http://www.globallogic.com/email_disclaimer.txt
[-- Attachment #1.2: Type: text/html, Size: 3049 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-10 10:24 ` Vitaly Chernooky
@ 2014-09-10 11:18 ` Andrii Tseglytskyi
2014-09-10 18:37 ` Stefano Stabellini
1 sibling, 0 replies; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-09-10 11:18 UTC (permalink / raw)
To: Vitaly Chernooky
Cc: Ian Campbell, Stefano Stabellini, jinsong.liu,
Oleksandr Dmytryshyn, Tim Deegan, xen-devel, Jan Beulich,
Anthony.Perard, jacob.shin
On Wed, Sep 10, 2014 at 1:24 PM, Vitaly Chernooky
<vitalii.chernookyi@globallogic.com> wrote:
> Hi Andrii,
>
>
> On Wed, Sep 10, 2014 at 1:15 PM, Andrii Tseglytskyi
> <andrii.tseglytskyi@globallogic.com> wrote:
>>
>> Hi,
>>
>> >> In case where dom0 has more vcpus than pcpus on the
>> >> system, the dom0 kernel is the most bug-prone place for pcpu cpufreq
>> >> governor. So I still believe that separate driver
>> >> domain with direct 1:1 vcpu:pcpu mapping is the best place for cpufreq
>> >> governor. But it also reasonable to run cpufreq
>> >> governor as userspace daemon in dom0.
>> >>
>> >> Also what do you think about PM QoS support? On bare metal cpufreq is
>> >> tightly integrated with PM QoS and intensively
>> >> cooperate in frequency scaling.
>> >
>> > Device PM needs to be done in Dom0.
>> > CPU an Platform level PM architecturally belongs to Xen, but I do
>> > understand that to do that in Xen we would need to add lots of code to
>> > the hypervisor. There is no silver bullet here.
>> >
>> > A driver domain with 1:1 vcpu:pcpu mapping could work, but what kernel
>> > are you going to use for that? Linux? Wouldn't Linux be too big for a
>> > cpufreq driver domain, especially in embedded deployments? I think it
>> > would need at least 32MB to run.
>> >
>>
>> I would suggest not to do this. Driver domain will need to share the
>> same HW with dom0, which is impossible to implement. Good examples are
>> I2C, which is needed for Display and for Cpufreq, and it cannot be
>> shared between domains. (At least in current Linux kernel it won't be
>> easy to implement). dom0 is the best place to do cpufreq.
>
>
> What do you think about cpufreq in dom0 userspace?
>
What do you mean? Could you please explain with more details?
Regards,
Andrii
--
Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-10 10:24 ` Vitaly Chernooky
2014-09-10 11:18 ` Andrii Tseglytskyi
@ 2014-09-10 18:37 ` Stefano Stabellini
2014-09-11 7:51 ` Vitaly Chernooky
1 sibling, 1 reply; 57+ messages in thread
From: Stefano Stabellini @ 2014-09-10 18:37 UTC (permalink / raw)
To: Vitaly Chernooky
Cc: Ian Campbell, Stefano Stabellini, jinsong.liu,
Oleksandr Dmytryshyn, Tim Deegan, xen-devel, Jan Beulich,
Anthony.Perard, jacob.shin, Andrii Tseglytskyi
[-- Attachment #1: Type: text/plain, Size: 2382 bytes --]
On Wed, 10 Sep 2014, Vitaly Chernooky wrote:
> Hi Andrii,
>
>
> On Wed, Sep 10, 2014 at 1:15 PM, Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com> wrote:
> Hi,
>
> >> In case where dom0 has more vcpus than pcpus on the
> >> system, the dom0 kernel is the most bug-prone place for pcpu cpufreq governor. So I still believe that
> separate driver
> >> domain with direct 1:1 vcpu:pcpu mapping is the best place for cpufreq governor. But it also reasonable to
> run cpufreq
> >> governor as userspace daemon in dom0.
> >>
> >> Also what do you think about PM QoS support? On bare metal cpufreq is tightly integrated with PM QoS and
> intensively
> >> cooperate in frequency scaling.
> >
> > Device PM needs to be done in Dom0.
> > CPU an Platform level PM architecturally belongs to Xen, but I do
> > understand that to do that in Xen we would need to add lots of code to
> > the hypervisor. There is no silver bullet here.
> >
> > A driver domain with 1:1 vcpu:pcpu mapping could work, but what kernel
> > are you going to use for that? Linux? Wouldn't Linux be too big for a
> > cpufreq driver domain, especially in embedded deployments? I think it
> > would need at least 32MB to run.
> >
>
> I would suggest not to do this. Driver domain will need to share the
> same HW with dom0, which is impossible to implement. Good examples are
> I2C, which is needed for Display and for Cpufreq, and it cannot be
> shared between domains. (At least in current Linux kernel it won't be
> easy to implement). dom0 is the best place to do cpufreq.
>
>
> What do you think about cpufreq in dom0 userspace?
Would you write an i2c driver in userspace? How would you suggest to
do the frequency and voltage changes in dom0 userspace otherwise?
> With best regards,
>
> Regards,
> Andrii
>
>
> --
>
> Andrii Tseglytskyi | Embedded Dev
> GlobalLogic
> www.globallogic.com
>
>
>
>
> --
> Vitaly Chernooky | Senior Developer - Product Engineering and Development
> GlobalLogic
> P +380.44.4929695 ext.1136 M +380.98.7920568 S cvv_2k
> www.globallogic.com
>
> http://www.globallogic.com/email_disclaimer.txt
>
>
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-10 18:37 ` Stefano Stabellini
@ 2014-09-11 7:51 ` Vitaly Chernooky
0 siblings, 0 replies; 57+ messages in thread
From: Vitaly Chernooky @ 2014-09-11 7:51 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Ian Campbell, jinsong.liu, Oleksandr Dmytryshyn, Tim Deegan,
xen-devel, Jan Beulich, Anthony PERARD, Andrii Tseglytskyi
[-- Attachment #1.1: Type: text/plain, Size: 2916 bytes --]
It't not a problem. TI BSPs usually contain examples. There is another
problems which prevent me developing this direction.
With best regards,
On Wed, Sep 10, 2014 at 9:37 PM, Stefano Stabellini <
stefano.stabellini@eu.citrix.com> wrote:
> On Wed, 10 Sep 2014, Vitaly Chernooky wrote:
> > Hi Andrii,
> >
> >
> > On Wed, Sep 10, 2014 at 1:15 PM, Andrii Tseglytskyi <
> andrii.tseglytskyi@globallogic.com> wrote:
> > Hi,
> >
> > >> In case where dom0 has more vcpus than pcpus on the
> > >> system, the dom0 kernel is the most bug-prone place for pcpu
> cpufreq governor. So I still believe that
> > separate driver
> > >> domain with direct 1:1 vcpu:pcpu mapping is the best place for
> cpufreq governor. But it also reasonable to
> > run cpufreq
> > >> governor as userspace daemon in dom0.
> > >>
> > >> Also what do you think about PM QoS support? On bare metal
> cpufreq is tightly integrated with PM QoS and
> > intensively
> > >> cooperate in frequency scaling.
> > >
> > > Device PM needs to be done in Dom0.
> > > CPU an Platform level PM architecturally belongs to Xen, but I do
> > > understand that to do that in Xen we would need to add lots of
> code to
> > > the hypervisor. There is no silver bullet here.
> > >
> > > A driver domain with 1:1 vcpu:pcpu mapping could work, but what
> kernel
> > > are you going to use for that? Linux? Wouldn't Linux be too big
> for a
> > > cpufreq driver domain, especially in embedded deployments? I
> think it
> > > would need at least 32MB to run.
> > >
> >
> > I would suggest not to do this. Driver domain will need to share
> the
> > same HW with dom0, which is impossible to implement. Good examples
> are
> > I2C, which is needed for Display and for Cpufreq, and it cannot be
> > shared between domains. (At least in current Linux kernel it won't
> be
> > easy to implement). dom0 is the best place to do cpufreq.
> >
> >
> > What do you think about cpufreq in dom0 userspace?
>
> Would you write an i2c driver in userspace? How would you suggest to
> do the frequency and voltage changes in dom0 userspace otherwise?
>
>
> > With best regards,
> >
> > Regards,
> > Andrii
> >
> >
> > --
> >
> > Andrii Tseglytskyi | Embedded Dev
> > GlobalLogic
> > www.globallogic.com
> >
> >
> >
> >
> > --
> > Vitaly Chernooky | Senior Developer - Product Engineering and Development
> > GlobalLogic
> > P +380.44.4929695 ext.1136 M +380.98.7920568 S cvv_2k
> > www.globallogic.com
> >
> > http://www.globallogic.com/email_disclaimer.txt
> >
> >
>
--
*Vitaly Chernooky | Senior Developer - Product Engineering and Development*
GlobalLogic
P *+380.44.4929695 ext.1136* M *+380.98.7920568* S cvv_2k
www.globallogic.com
http://www.globallogic.com/email_disclaimer.txt
[-- Attachment #1.2: Type: text/html, Size: 4656 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-10 10:15 ` Andrii Tseglytskyi
2014-09-10 10:24 ` Vitaly Chernooky
@ 2014-09-10 11:24 ` Vitaly Chernooky
1 sibling, 0 replies; 57+ messages in thread
From: Vitaly Chernooky @ 2014-09-10 11:24 UTC (permalink / raw)
To: Andrii Tseglytskyi
Cc: Ian Campbell, Stefano Stabellini, jinsong.liu,
Oleksandr Dmytryshyn, Tim Deegan, xen-devel, Jan Beulich,
Anthony.Perard, jacob.shin
[-- Attachment #1.1: Type: text/plain, Size: 1893 bytes --]
On Wed, Sep 10, 2014 at 1:15 PM, Andrii Tseglytskyi <
andrii.tseglytskyi@globallogic.com> wrote:
> Hi,
>
> >> In case where dom0 has more vcpus than pcpus on the
> >> system, the dom0 kernel is the most bug-prone place for pcpu cpufreq
> governor. So I still believe that separate driver
> >> domain with direct 1:1 vcpu:pcpu mapping is the best place for cpufreq
> governor. But it also reasonable to run cpufreq
> >> governor as userspace daemon in dom0.
> >>
> >> Also what do you think about PM QoS support? On bare metal cpufreq is
> tightly integrated with PM QoS and intensively
> >> cooperate in frequency scaling.
> >
> > Device PM needs to be done in Dom0.
> > CPU an Platform level PM architecturally belongs to Xen, but I do
> > understand that to do that in Xen we would need to add lots of code to
> > the hypervisor. There is no silver bullet here.
> >
> > A driver domain with 1:1 vcpu:pcpu mapping could work, but what kernel
> > are you going to use for that? Linux? Wouldn't Linux be too big for a
> > cpufreq driver domain, especially in embedded deployments? I think it
> > would need at least 32MB to run.
> >
>
> I would suggest not to do this. Driver domain will need to share the
> same HW with dom0, which is impossible to implement. Good examples are
> I2C, which is needed for Display and for Cpufreq, and it cannot be
> shared between domains. (At least in current Linux kernel it won't be
> easy to implement). dom0 is the best place to do cpufreq.
>
Also how do you feel about providing virtualized i2c from driver domain to
dom0?
> Regards,
> Andrii
>
>
> --
>
> Andrii Tseglytskyi | Embedded Dev
> GlobalLogic
> www.globallogic.com
>
--
*Vitaly Chernooky | Senior Developer - Product Engineering and Development*
GlobalLogic
P *+380.44.4929695 ext.1136* M *+380.98.7920568* S cvv_2k
www.globallogic.com
http://www.globallogic.com/email_disclaimer.txt
[-- Attachment #1.2: Type: text/html, Size: 3025 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-09 21:58 ` Stefano Stabellini
2014-09-10 10:15 ` Andrii Tseglytskyi
@ 2014-09-10 14:28 ` Vitaly Chernooky
2014-09-10 18:41 ` Stefano Stabellini
1 sibling, 1 reply; 57+ messages in thread
From: Vitaly Chernooky @ 2014-09-10 14:28 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Ian Campbell, jinsong.liu, Oleksandr Dmytryshyn, Tim Deegan,
xen-devel, Jan Beulich, Anthony PERARD, jacob.shin,
Andrii Tseglytskyi
[-- Attachment #1.1: Type: text/plain, Size: 10026 bytes --]
I've intensively discussed my suggestions here and now it is transparent to
me that we should not try to use Cpufreq on ARM SoCs without direct 1:1
pcpu:vcpu mapping in dom0. So if someone want to break 1:1 mapping he
should forget Cpufreq.
With best regards,
On Wed, Sep 10, 2014 at 12:58 AM, Stefano Stabellini <
stefano.stabellini@eu.citrix.com> wrote:
> On Tue, 9 Sep 2014, Vitaly Chernooky wrote:
> > Hi All!
> >
> > On Fri, Sep 5, 2014 at 12:56 AM, Stefano Stabellini <
> stefano.stabellini@eu.citrix.com> wrote:
> > On Thu, 4 Sep 2014, Oleksandr Dmytryshyn wrote:
> > > Hi to all.
> > >
> > > I want to implement cpufreq driver in the next way:
> > > 1. Cpufreq governor will be implemented in the Xen
> > > 2. dom0 will only change cpu frequency and voltage of the
> physical cpus
> > >
> > > But there are some nuances:
> > > 1. dom0 driver should read an information about operation points
> > > (frequencies and voltages) and cpu supply source from the device
> tree for each
> > > physical cpu. In the omap processor case this driver suspects
> that
> > > those settings
> > > located in the /cpus/cpu@0/ node. But hypervisor creates an cpu
> node
> > > for each vcpu
> > > for kernel dom0 in the device tree and those information is lost
> in the dom0.
> > > 2. What about this case if we will have some physical cpus with
> different
> > > operation points (for example 2 cpus) and we give only one cpu
> for dom0?
> > >
> > > How should I transfer all information from the original cpu@0.
> .cpu@n nodes
> > > about all physical cpus to the kernel dom0 driver? Maybe an
> additional
> > > nodes should be created by the hypervisor in the device tree for
> dom0
> > > and named as pcpu@0..pcpu@n?
> >
> > If we do that, wouldn't we require changes to the core OMAP
> drivers or
> > cpu initialization code in Linux (to parse "pcpu" instead of "cpu"
> > nodes)? I don't expect they would be easy to upstream or maintain
> going
> > forward.
> >
> > I am trying to think of an alternative, such as passing the real
> cpu
> > nodes to dom0 but then adding status = "disabled", but I am not
> sure
> > whether Linux checks the status for cpu nodes. In addition this
> scheme
> > wouldn't support the case where dom0 has more vcpus than pcpus on
> the
> > system. Granted it is not very common and might even be
> detrimental for
> > performances, but we should be able to support it.
> >
> >
> > In case where dom0 has more vcpus than pcpus on the
> > system, the dom0 kernel is the most bug-prone place for pcpu cpufreq
> governor. So I still believe that separate driver
> > domain with direct 1:1 vcpu:pcpu mapping is the best place for cpufreq
> governor. But it also reasonable to run cpufreq
> > governor as userspace daemon in dom0.
> >
> > Also what do you think about PM QoS support? On bare metal cpufreq is
> tightly integrated with PM QoS and intensively
> > cooperate in frequency scaling.
>
> Device PM needs to be done in Dom0.
> CPU an Platform level PM architecturally belongs to Xen, but I do
> understand that to do that in Xen we would need to add lots of code to
> the hypervisor. There is no silver bullet here.
>
> A driver domain with 1:1 vcpu:pcpu mapping could work, but what kernel
> are you going to use for that? Linux? Wouldn't Linux be too big for a
> cpufreq driver domain, especially in embedded deployments? I think it
> would need at least 32MB to run.
>
>
> > With best regards,
> >
> > Ian, what do you think about this?
> >
> >
> >
> > > Oleksandr Dmytryshyn | Product Engineering and Development
> > > GlobalLogic
> > > M +38.067.382.2525
> > > www.globallogic.com
> > >
> > > http://www.globallogic.com/email_disclaimer.txt
> > >
> > >
> > > On Tue, Sep 2, 2014 at 9:46 PM, Andrii Tseglytskyi
> > > <andrii.tseglytskyi@globallogic.com> wrote:
> > > >
> > > > Hi Stefano,
> > > >
> > > > Thank you for explanation.
> > > > I think this requires more and deeper investigation, but for
> sure dom0
> > > > must be able to do this.
> > > > Let us investigate this.
> > > >
> > > > Thank you,
> > > >
> > > > Regards,
> > > > Andrii
> > > >
> > > > On Tue, Sep 2, 2014 at 9:39 PM, Stefano Stabellini
> > > > <stefano.stabellini@eu.citrix.com> wrote:
> > > > > On Tue, 2 Sep 2014, Andrii Tseglytskyi wrote:
> > > > >> On Tue, Sep 2, 2014 at 4:00 AM, Stefano Stabellini
> > > > >> <stefano.stabellini@eu.citrix.com> wrote:
> > > > >> > On Fri, 29 Aug 2014, Andrii Tseglytskyi wrote:
> > > > >> >> Hi,
> > > > >> >>
> > > > >> >> Stefano, Ian,
> > > > >> >>
> > > > >> >> Could you please clarify the following point:
> > > > >> >>
> > > > >> >> I agree that decision about frequency change should be
> taken by Xen
> > > > >> >> hypervisor. But what about hardware frequency changing?
> > > > >> >> In general when frequency changed to bigger value (for
> example from 1
> > > > >> >> GHz to 1.5 GHz) for ARM kernels sequence looks like the
> following:
> > > > >> >>
> > > > >> >> 1) cpufreq governor decides that frequency should be
> changed. This
> > > > >> >> decision is taken after analysing of CPU performance
> data taking in
> > > > >> >> account governor policy.
> > > > >> >> 2) cpufreq governor asks cpufreq driver about new
> frequency.
> > > > >> >> 3) cpufreq driver compares current and target
> frequencies and asks
> > > > >> >> cpufreq regulator about voltage change.
> > > > >> >> 4) cpufreq regulator send i2c command to standalone
> microchip, which
> > > > >> >> is responsible for voltage changing.
> > > > >> >> 5) cpufreq driver asks clock framework about new
> frequency for CPU clock
> > > > >> >> 6) clock framework performs frequency sanity checks,
> taking in account
> > > > >> >> clock parents and clock divider settings, and call
> platform specific
> > > > >> >> "set_frequency" callback.
> > > > >> >> 7) platform specific callback performs proper HW
> registers
> > > > >> >> configuration for newly selected frequency
> > > > >> >>
> > > > >> >> Also there are some special cases - for example for
> OMAP5+ when
> > > > >> >> frequency is changed to 1.5 GHz+, two additional HW IPs
> should be
> > > > >> >> triggered (ABB and DCC, if someone is familiar with
> OMAP5+ )
> > > > >> >>
> > > > >> >> So, for generic ARM kernel we have 3 entities to change
> frequency:
> > > > >> >>
> > > > >> >> - cpufreq governor
> > > > >> >> - cpufreq driver
> > > > >> >> - cpufreq regulator
> > > > >> >>
> > > > >> >> + 2 additional IP for OMAP5+
> > > > >> >> - ABB
> > > > >> >> - DCC
> > > > >> >>
> > > > >> >> Taking in account all above, it looks like it would be
> better to
> > > > >> >> implement only Xen cpufreq governor. Xen will take a
> decision about
> > > > >> >> new frequency, and kernel dom0 will perform other steps.
> Dom0 contains
> > > > >> >> all generic and platform specific frameworks, needed for
> frequency
> > > > >> >> changing.
> > > > >> >>
> > > > >> >> What do you think ?
> > > > >> >
> > > > >> > Keep in mind that the architecture must be able to handle
> the case where
> > > > >> > dom0 has only 1 or 2 vcpus on a 4 or 8 cores system with
> multiple
> > > > >> > physical cpus.
> > > > >> > Could dom0 change the frequency of a physical core or a
> physical cpu is
> > > > >> > not even running on? If that is not a problem, because
> cpus and
> > > > >> > frequency changing are decoupled enough in Linux to allow
> it, then I am
> > > > >> > OK with it. But I suspect they are not.
> > > > >> >
> > > > >>
> > > > >> Not sure that I got your point correctly - dom0 will change
> frequency
> > > > >> on physical CPU.
> > > > >> And in case of OMAP - this changing affects on both ARM
> physical cpus
> > > > >> - changing is coupled.
> > > > >> In case of other ARM platforms - changing may be not
> coupled (I've
> > > > >> heard that Snapdragon can change cpu freqs independently on
> each
> > > > >> physical cpu)
> > > > >
> > > > > Let me explain with a concrete example.
> > > > >
> > > > > Let's suppose that the platform has 2 physical cpus, each
> cpu has 4
> > > > > cores. Let's also supposed that dom0 has only 2 vcpus,
> currently
> > > > > running on core0 and core1 of cpu0.
> > > > >
> > > > > In this case would dom0 be able to change the frequency of
> core3 of
> > > > > cpu1, given that is not even running on it?
> > > > > If it can be done without any hacks, then we can go ahead
> with this
> > > > > approach.
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Andrii Tseglytskyi | Embedded Dev
> > > > GlobalLogic
> > > > www.globallogic.com
> > >
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
> >
> >
> >
> >
> > --
> > Vitaly Chernooky | Senior Developer - Product Engineering and Development
> > GlobalLogic
> > P +380.44.4929695 ext.1136 M +380.98.7920568 S cvv_2k
> > www.globallogic.com
> >
> > http://www.globallogic.com/email_disclaimer.txt
> >
> >
>
--
*Vitaly Chernooky | Senior Developer - Product Engineering and Development*
GlobalLogic
P *+380.44.4929695 ext.1136* M *+380.98.7920568* S cvv_2k
www.globallogic.com
http://www.globallogic.com/email_disclaimer.txt
[-- Attachment #1.2: Type: text/html, Size: 14830 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-10 14:28 ` Vitaly Chernooky
@ 2014-09-10 18:41 ` Stefano Stabellini
2014-09-11 7:48 ` Vitaly Chernooky
0 siblings, 1 reply; 57+ messages in thread
From: Stefano Stabellini @ 2014-09-10 18:41 UTC (permalink / raw)
To: Vitaly Chernooky
Cc: Ian Campbell, Stefano Stabellini, jinsong.liu,
Oleksandr Dmytryshyn, Tim Deegan, xen-devel, Jan Beulich,
Anthony PERARD, jacob.shin, Andrii Tseglytskyi
[-- Attachment #1: Type: text/plain, Size: 11387 bytes --]
The issue with that limitation is that it doesn't scale well on large
systems. You really wouldn't want dom0 to have 18 vcpus on a Xeon E5
because it would badly affect performances. Even on a 8 cores SoC it
would be best to assign less than 8 vcpus to dom0.
On Wed, 10 Sep 2014, Vitaly Chernooky wrote:
> I've intensively discussed my suggestions here and now it is transparent to me that we should not try to use Cpufreq on ARM
> SoCs without direct 1:1 pcpu:vcpu mapping in dom0. So if someone want to break 1:1 mapping he should forget Cpufreq.
> With best regards,
>
>
> On Wed, Sep 10, 2014 at 12:58 AM, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:
> On Tue, 9 Sep 2014, Vitaly Chernooky wrote:
> > Hi All!
> >
> > On Fri, Sep 5, 2014 at 12:56 AM, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:
> > On Thu, 4 Sep 2014, Oleksandr Dmytryshyn wrote:
> > > Hi to all.
> > >
> > > I want to implement cpufreq driver in the next way:
> > > 1. Cpufreq governor will be implemented in the Xen
> > > 2. dom0 will only change cpu frequency and voltage of the physical cpus
> > >
> > > But there are some nuances:
> > > 1. dom0 driver should read an information about operation points
> > > (frequencies and voltages) and cpu supply source from the device tree for each
> > > physical cpu. In the omap processor case this driver suspects that
> > > those settings
> > > located in the /cpus/cpu@0/ node. But hypervisor creates an cpu node
> > > for each vcpu
> > > for kernel dom0 in the device tree and those information is lost in the dom0.
> > > 2. What about this case if we will have some physical cpus with different
> > > operation points (for example 2 cpus) and we give only one cpu for dom0?
> > >
> > > How should I transfer all information from the original cpu@0..cpu@n nodes
> > > about all physical cpus to the kernel dom0 driver? Maybe an additional
> > > nodes should be created by the hypervisor in the device tree for dom0
> > > and named as pcpu@0..pcpu@n?
> >
> > If we do that, wouldn't we require changes to the core OMAP drivers or
> > cpu initialization code in Linux (to parse "pcpu" instead of "cpu"
> > nodes)? I don't expect they would be easy to upstream or maintain going
> > forward.
> >
> > I am trying to think of an alternative, such as passing the real cpu
> > nodes to dom0 but then adding status = "disabled", but I am not sure
> > whether Linux checks the status for cpu nodes. In addition this scheme
> > wouldn't support the case where dom0 has more vcpus than pcpus on the
> > system. Granted it is not very common and might even be detrimental for
> > performances, but we should be able to support it.
> >
> >
> > In case where dom0 has more vcpus than pcpus on the
> > system, the dom0 kernel is the most bug-prone place for pcpu cpufreq governor. So I still believe that
> separate driver
> > domain with direct 1:1 vcpu:pcpu mapping is the best place for cpufreq governor. But it also reasonable to run
> cpufreq
> > governor as userspace daemon in dom0.
> >
> > Also what do you think about PM QoS support? On bare metal cpufreq is tightly integrated with PM QoS and
> intensively
> > cooperate in frequency scaling.
>
> Device PM needs to be done in Dom0.
> CPU an Platform level PM architecturally belongs to Xen, but I do
> understand that to do that in Xen we would need to add lots of code to
> the hypervisor. There is no silver bullet here.
>
> A driver domain with 1:1 vcpu:pcpu mapping could work, but what kernel
> are you going to use for that? Linux? Wouldn't Linux be too big for a
> cpufreq driver domain, especially in embedded deployments? I think it
> would need at least 32MB to run.
>
>
> > With best regards,
> >
> > Ian, what do you think about this?
> >
> >
> >
> > > Oleksandr Dmytryshyn | Product Engineering and Development
> > > GlobalLogic
> > > M +38.067.382.2525
> > > www.globallogic.com
> > >
> > > http://www.globallogic.com/email_disclaimer.txt
> > >
> > >
> > > On Tue, Sep 2, 2014 at 9:46 PM, Andrii Tseglytskyi
> > > <andrii.tseglytskyi@globallogic.com> wrote:
> > > >
> > > > Hi Stefano,
> > > >
> > > > Thank you for explanation.
> > > > I think this requires more and deeper investigation, but for sure dom0
> > > > must be able to do this.
> > > > Let us investigate this.
> > > >
> > > > Thank you,
> > > >
> > > > Regards,
> > > > Andrii
> > > >
> > > > On Tue, Sep 2, 2014 at 9:39 PM, Stefano Stabellini
> > > > <stefano.stabellini@eu.citrix.com> wrote:
> > > > > On Tue, 2 Sep 2014, Andrii Tseglytskyi wrote:
> > > > >> On Tue, Sep 2, 2014 at 4:00 AM, Stefano Stabellini
> > > > >> <stefano.stabellini@eu.citrix.com> wrote:
> > > > >> > On Fri, 29 Aug 2014, Andrii Tseglytskyi wrote:
> > > > >> >> Hi,
> > > > >> >>
> > > > >> >> Stefano, Ian,
> > > > >> >>
> > > > >> >> Could you please clarify the following point:
> > > > >> >>
> > > > >> >> I agree that decision about frequency change should be taken by Xen
> > > > >> >> hypervisor. But what about hardware frequency changing?
> > > > >> >> In general when frequency changed to bigger value (for example from 1
> > > > >> >> GHz to 1.5 GHz) for ARM kernels sequence looks like the following:
> > > > >> >>
> > > > >> >> 1) cpufreq governor decides that frequency should be changed. This
> > > > >> >> decision is taken after analysing of CPU performance data taking in
> > > > >> >> account governor policy.
> > > > >> >> 2) cpufreq governor asks cpufreq driver about new frequency.
> > > > >> >> 3) cpufreq driver compares current and target frequencies and asks
> > > > >> >> cpufreq regulator about voltage change.
> > > > >> >> 4) cpufreq regulator send i2c command to standalone microchip, which
> > > > >> >> is responsible for voltage changing.
> > > > >> >> 5) cpufreq driver asks clock framework about new frequency for CPU clock
> > > > >> >> 6) clock framework performs frequency sanity checks, taking in account
> > > > >> >> clock parents and clock divider settings, and call platform specific
> > > > >> >> "set_frequency" callback.
> > > > >> >> 7) platform specific callback performs proper HW registers
> > > > >> >> configuration for newly selected frequency
> > > > >> >>
> > > > >> >> Also there are some special cases - for example for OMAP5+ when
> > > > >> >> frequency is changed to 1.5 GHz+, two additional HW IPs should be
> > > > >> >> triggered (ABB and DCC, if someone is familiar with OMAP5+ )
> > > > >> >>
> > > > >> >> So, for generic ARM kernel we have 3 entities to change frequency:
> > > > >> >>
> > > > >> >> - cpufreq governor
> > > > >> >> - cpufreq driver
> > > > >> >> - cpufreq regulator
> > > > >> >>
> > > > >> >> + 2 additional IP for OMAP5+
> > > > >> >> - ABB
> > > > >> >> - DCC
> > > > >> >>
> > > > >> >> Taking in account all above, it looks like it would be better to
> > > > >> >> implement only Xen cpufreq governor. Xen will take a decision about
> > > > >> >> new frequency, and kernel dom0 will perform other steps. Dom0 contains
> > > > >> >> all generic and platform specific frameworks, needed for frequency
> > > > >> >> changing.
> > > > >> >>
> > > > >> >> What do you think ?
> > > > >> >
> > > > >> > Keep in mind that the architecture must be able to handle the case where
> > > > >> > dom0 has only 1 or 2 vcpus on a 4 or 8 cores system with multiple
> > > > >> > physical cpus.
> > > > >> > Could dom0 change the frequency of a physical core or a physical cpu is
> > > > >> > not even running on? If that is not a problem, because cpus and
> > > > >> > frequency changing are decoupled enough in Linux to allow it, then I am
> > > > >> > OK with it. But I suspect they are not.
> > > > >> >
> > > > >>
> > > > >> Not sure that I got your point correctly - dom0 will change frequency
> > > > >> on physical CPU.
> > > > >> And in case of OMAP - this changing affects on both ARM physical cpus
> > > > >> - changing is coupled.
> > > > >> In case of other ARM platforms - changing may be not coupled (I've
> > > > >> heard that Snapdragon can change cpu freqs independently on each
> > > > >> physical cpu)
> > > > >
> > > > > Let me explain with a concrete example.
> > > > >
> > > > > Let's suppose that the platform has 2 physical cpus, each cpu has 4
> > > > > cores. Let's also supposed that dom0 has only 2 vcpus, currently
> > > > > running on core0 and core1 of cpu0.
> > > > >
> > > > > In this case would dom0 be able to change the frequency of core3 of
> > > > > cpu1, given that is not even running on it?
> > > > > If it can be done without any hacks, then we can go ahead with this
> > > > > approach.
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Andrii Tseglytskyi | Embedded Dev
> > > > GlobalLogic
> > > > www.globallogic.com
> > >
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
> >
> >
> >
> >
> > --
> > Vitaly Chernooky | Senior Developer - Product Engineering and Development
> > GlobalLogic
> > P +380.44.4929695 ext.1136 M +380.98.7920568 S cvv_2k
> > www.globallogic.com
> >
> > http://www.globallogic.com/email_disclaimer.txt
> >
> >
>
>
>
>
> --
> Vitaly Chernooky | Senior Developer - Product Engineering and Development
> GlobalLogic
> P +380.44.4929695 ext.1136 M +380.98.7920568 S cvv_2k
> www.globallogic.com
>
> http://www.globallogic.com/email_disclaimer.txt
>
>
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: cpufreq implementation for OMAP under xen hypervisor.
2014-09-10 18:41 ` Stefano Stabellini
@ 2014-09-11 7:48 ` Vitaly Chernooky
0 siblings, 0 replies; 57+ messages in thread
From: Vitaly Chernooky @ 2014-09-11 7:48 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Ian Campbell, jinsong.liu, Oleksandr Dmytryshyn, Tim Deegan,
xen-devel, Jan Beulich, Anthony PERARD, jacob.shin,
Andrii Tseglytskyi
[-- Attachment #1.1: Type: text/plain, Size: 11903 bytes --]
Stefano,
I don't know how to control voltages and frequency on x86 HW. May be there
is a *reasonable *way to implement Cpufreq with different numbers of pcpu
and vcpu. But there is definitely no *reasonable* ways to do if on TI and
Marvell ARM SoCs without 1:1 mapping. We need really huge amount of work on
ARM SoCs in steps to break 1:1 mapping dependency. I will be
really surprised if there is an ARM SOC not affected be 1:1 dependency.
With best regards,
On Wed, Sep 10, 2014 at 9:41 PM, Stefano Stabellini <
stefano.stabellini@eu.citrix.com> wrote:
> The issue with that limitation is that it doesn't scale well on large
> systems. You really wouldn't want dom0 to have 18 vcpus on a Xeon E5
> because it would badly affect performances. Even on a 8 cores SoC it
> would be best to assign less than 8 vcpus to dom0.
>
> On Wed, 10 Sep 2014, Vitaly Chernooky wrote:
> > I've intensively discussed my suggestions here and now it is transparent
> to me that we should not try to use Cpufreq on ARM
> > SoCs without direct 1:1 pcpu:vcpu mapping in dom0. So if someone want to
> break 1:1 mapping he should forget Cpufreq.
> > With best regards,
> >
> >
> > On Wed, Sep 10, 2014 at 12:58 AM, Stefano Stabellini <
> stefano.stabellini@eu.citrix.com> wrote:
> > On Tue, 9 Sep 2014, Vitaly Chernooky wrote:
> > > Hi All!
> > >
> > > On Fri, Sep 5, 2014 at 12:56 AM, Stefano Stabellini <
> stefano.stabellini@eu.citrix.com> wrote:
> > > On Thu, 4 Sep 2014, Oleksandr Dmytryshyn wrote:
> > > > Hi to all.
> > > >
> > > > I want to implement cpufreq driver in the next way:
> > > > 1. Cpufreq governor will be implemented in the Xen
> > > > 2. dom0 will only change cpu frequency and voltage of
> the physical cpus
> > > >
> > > > But there are some nuances:
> > > > 1. dom0 driver should read an information about
> operation points
> > > > (frequencies and voltages) and cpu supply source from
> the device tree for each
> > > > physical cpu. In the omap processor case this driver
> suspects that
> > > > those settings
> > > > located in the /cpus/cpu@0/ node. But hypervisor
> creates an cpu node
> > > > for each vcpu
> > > > for kernel dom0 in the device tree and those information
> is lost in the dom0.
> > > > 2. What about this case if we will have some physical
> cpus with different
> > > > operation points (for example 2 cpus) and we give only
> one cpu for dom0?
> > > >
> > > > How should I transfer all information from the original
> cpu@0..cpu@n nodes
> > > > about all physical cpus to the kernel dom0 driver? Maybe
> an additional
> > > > nodes should be created by the hypervisor in the device
> tree for dom0
> > > > and named as pcpu@0..pcpu@n?
> > >
> > > If we do that, wouldn't we require changes to the core
> OMAP drivers or
> > > cpu initialization code in Linux (to parse "pcpu" instead
> of "cpu"
> > > nodes)? I don't expect they would be easy to upstream or
> maintain going
> > > forward.
> > >
> > > I am trying to think of an alternative, such as passing
> the real cpu
> > > nodes to dom0 but then adding status = "disabled", but I
> am not sure
> > > whether Linux checks the status for cpu nodes. In addition
> this scheme
> > > wouldn't support the case where dom0 has more vcpus than
> pcpus on the
> > > system. Granted it is not very common and might even be
> detrimental for
> > > performances, but we should be able to support it.
> > >
> > >
> > > In case where dom0 has more vcpus than pcpus on the
> > > system, the dom0 kernel is the most bug-prone place for pcpu
> cpufreq governor. So I still believe that
> > separate driver
> > > domain with direct 1:1 vcpu:pcpu mapping is the best place for
> cpufreq governor. But it also reasonable to run
> > cpufreq
> > > governor as userspace daemon in dom0.
> > >
> > > Also what do you think about PM QoS support? On bare metal
> cpufreq is tightly integrated with PM QoS and
> > intensively
> > > cooperate in frequency scaling.
> >
> > Device PM needs to be done in Dom0.
> > CPU an Platform level PM architecturally belongs to Xen, but I do
> > understand that to do that in Xen we would need to add lots of code to
> > the hypervisor. There is no silver bullet here.
> >
> > A driver domain with 1:1 vcpu:pcpu mapping could work, but what kernel
> > are you going to use for that? Linux? Wouldn't Linux be too big for a
> > cpufreq driver domain, especially in embedded deployments? I think it
> > would need at least 32MB to run.
> >
> >
> > > With best regards,
> > >
> > > Ian, what do you think about this?
> > >
> > >
> > >
> > > > Oleksandr Dmytryshyn | Product Engineering and Development
> > > > GlobalLogic
> > > > M +38.067.382.2525
> > > > www.globallogic.com
> > > >
> > > > http://www.globallogic.com/email_disclaimer.txt
> > > >
> > > >
> > > > On Tue, Sep 2, 2014 at 9:46 PM, Andrii Tseglytskyi
> > > > <andrii.tseglytskyi@globallogic.com> wrote:
> > > > >
> > > > > Hi Stefano,
> > > > >
> > > > > Thank you for explanation.
> > > > > I think this requires more and deeper investigation, but for
> sure dom0
> > > > > must be able to do this.
> > > > > Let us investigate this.
> > > > >
> > > > > Thank you,
> > > > >
> > > > > Regards,
> > > > > Andrii
> > > > >
> > > > > On Tue, Sep 2, 2014 at 9:39 PM, Stefano Stabellini
> > > > > <stefano.stabellini@eu.citrix.com> wrote:
> > > > > > On Tue, 2 Sep 2014, Andrii Tseglytskyi wrote:
> > > > > >> On Tue, Sep 2, 2014 at 4:00 AM, Stefano Stabellini
> > > > > >> <stefano.stabellini@eu.citrix.com> wrote:
> > > > > >> > On Fri, 29 Aug 2014, Andrii Tseglytskyi wrote:
> > > > > >> >> Hi,
> > > > > >> >>
> > > > > >> >> Stefano, Ian,
> > > > > >> >>
> > > > > >> >> Could you please clarify the following point:
> > > > > >> >>
> > > > > >> >> I agree that decision about frequency change should be
> taken by Xen
> > > > > >> >> hypervisor. But what about hardware frequency changing?
> > > > > >> >> In general when frequency changed to bigger value (for
> example from 1
> > > > > >> >> GHz to 1.5 GHz) for ARM kernels sequence looks like
> the following:
> > > > > >> >>
> > > > > >> >> 1) cpufreq governor decides that frequency should be
> changed. This
> > > > > >> >> decision is taken after analysing of CPU performance
> data taking in
> > > > > >> >> account governor policy.
> > > > > >> >> 2) cpufreq governor asks cpufreq driver about new
> frequency.
> > > > > >> >> 3) cpufreq driver compares current and target
> frequencies and asks
> > > > > >> >> cpufreq regulator about voltage change.
> > > > > >> >> 4) cpufreq regulator send i2c command to standalone
> microchip, which
> > > > > >> >> is responsible for voltage changing.
> > > > > >> >> 5) cpufreq driver asks clock framework about new
> frequency for CPU clock
> > > > > >> >> 6) clock framework performs frequency sanity checks,
> taking in account
> > > > > >> >> clock parents and clock divider settings, and call
> platform specific
> > > > > >> >> "set_frequency" callback.
> > > > > >> >> 7) platform specific callback performs proper HW
> registers
> > > > > >> >> configuration for newly selected frequency
> > > > > >> >>
> > > > > >> >> Also there are some special cases - for example for
> OMAP5+ when
> > > > > >> >> frequency is changed to 1.5 GHz+, two additional HW
> IPs should be
> > > > > >> >> triggered (ABB and DCC, if someone is familiar with
> OMAP5+ )
> > > > > >> >>
> > > > > >> >> So, for generic ARM kernel we have 3 entities to
> change frequency:
> > > > > >> >>
> > > > > >> >> - cpufreq governor
> > > > > >> >> - cpufreq driver
> > > > > >> >> - cpufreq regulator
> > > > > >> >>
> > > > > >> >> + 2 additional IP for OMAP5+
> > > > > >> >> - ABB
> > > > > >> >> - DCC
> > > > > >> >>
> > > > > >> >> Taking in account all above, it looks like it would be
> better to
> > > > > >> >> implement only Xen cpufreq governor. Xen will take a
> decision about
> > > > > >> >> new frequency, and kernel dom0 will perform other
> steps. Dom0 contains
> > > > > >> >> all generic and platform specific frameworks, needed
> for frequency
> > > > > >> >> changing.
> > > > > >> >>
> > > > > >> >> What do you think ?
> > > > > >> >
> > > > > >> > Keep in mind that the architecture must be able to
> handle the case where
> > > > > >> > dom0 has only 1 or 2 vcpus on a 4 or 8 cores system
> with multiple
> > > > > >> > physical cpus.
> > > > > >> > Could dom0 change the frequency of a physical core or a
> physical cpu is
> > > > > >> > not even running on? If that is not a problem, because
> cpus and
> > > > > >> > frequency changing are decoupled enough in Linux to
> allow it, then I am
> > > > > >> > OK with it. But I suspect they are not.
> > > > > >> >
> > > > > >>
> > > > > >> Not sure that I got your point correctly - dom0 will
> change frequency
> > > > > >> on physical CPU.
> > > > > >> And in case of OMAP - this changing affects on both ARM
> physical cpus
> > > > > >> - changing is coupled.
> > > > > >> In case of other ARM platforms - changing may be not
> coupled (I've
> > > > > >> heard that Snapdragon can change cpu freqs independently
> on each
> > > > > >> physical cpu)
> > > > > >
> > > > > > Let me explain with a concrete example.
> > > > > >
> > > > > > Let's suppose that the platform has 2 physical cpus, each
> cpu has 4
> > > > > > cores. Let's also supposed that dom0 has only 2 vcpus,
> currently
> > > > > > running on core0 and core1 of cpu0.
> > > > > >
> > > > > > In this case would dom0 be able to change the frequency of
> core3 of
> > > > > > cpu1, given that is not even running on it?
> > > > > > If it can be done without any hacks, then we can go ahead
> with this
> > > > > > approach.
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Andrii Tseglytskyi | Embedded Dev
> > > > > GlobalLogic
> > > > > www.globallogic.com
> > > >
> > >
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xen.org
> > > http://lists.xen.org/xen-devel
> > >
> > >
> > >
> > >
> > > --
> > > Vitaly Chernooky | Senior Developer - Product Engineering and
> Development
> > > GlobalLogic
> > > P +380.44.4929695 ext.1136 M +380.98.7920568 S cvv_2k
> > > www.globallogic.com
> > >
> > > http://www.globallogic.com/email_disclaimer.txt
> > >
> > >
> >
> >
> >
> >
> > --
> > Vitaly Chernooky | Senior Developer - Product Engineering and Development
> > GlobalLogic
> > P +380.44.4929695 ext.1136 M +380.98.7920568 S cvv_2k
> > www.globallogic.com
> >
> > http://www.globallogic.com/email_disclaimer.txt
> >
> >
>
--
*Vitaly Chernooky | Senior Developer - Product Engineering and Development*
GlobalLogic
P *+380.44.4929695 ext.1136* M *+380.98.7920568* S cvv_2k
www.globallogic.com
http://www.globallogic.com/email_disclaimer.txt
[-- Attachment #1.2: Type: text/html, Size: 18172 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 57+ messages in thread