RFC: cpu frequency scaling

RFC: cpu frequency scaling

[Rik van Riel]

Right now, CPU frequency scaling in Xen does not work (right) for a
combination of reasons:
1) dom0 might not run on all physical CPUs
2) dom0 might not even have enough VCPUs to fill all physical CPUs
3) dom0 has no idea how busy the other guests are, so it does not
    know whether to scale the CPU frequencies up or down

I can see two ways to fix this:
1) move cpu frequency scaling into the hypervisor, since the hypervisor
    knows how busy all the guests are

2) make sure every physical CPU core has a dom0 VCPU pinned to it, and
    have dom0 poll the hypervisor to find out how busy each CPU is,
    scaling the frequencies of the CPUs as required

These approaches both have their drawbacks.  One requires porting over
a series of drivers from Linux to the hypervisor, maintaining two sets
into the future.  The second could mess up some of the credit
scheduler's balancing, and requires Linux modifications to poll the
hypervisor for the CPU load...

Also, it would mean the end of running a 1 vcpu small dom0, which some
people seem to like.

Which of these approaches would you prefer?

Is there a 3rd option?

-- 
Who do you trust?
The people with all the right answers?
Or the people with the right questions?

Re: RFC: cpu frequency scaling

[Keir Fraser]

On 25/10/06 7:04 pm, "Rik van Riel" <riel@redhat.com> wrote:

> These approaches both have their drawbacks.  One requires porting over
> a series of drivers from Linux to the hypervisor, maintaining two sets
> into the future.  The second could mess up some of the credit
> scheduler's balancing, and requires Linux modifications to poll the
> hypervisor for the CPU load...
> 
> Also, it would mean the end of running a 1 vcpu small dom0, which some
> people seem to like.
> 
> Which of these approaches would you prefer?

Frequency changes in dom0, please. The 'small dom0' issue is easily handled
-- create a driver domain to handle the frequency management. Or move the
potentially resource-intensive processes in dom0 out to a domU (amounts to
the same thing, really). It ought to take little enough resource that having
a VCPU per PCPU should not affect scheduling of more user-visible VMs.

We want to have the flexibility to implement cpufreq policy in a user-space
process anyway, so the feedback loop requires an interface to be punched
through between Xen and a cpufreq-managing guest.

 -- Keir

Re: RFC: cpu frequency scaling

[Rik van Riel]

Keir Fraser wrote:

> Frequency changes in dom0, please. The 'small dom0' issue is easily handled
> -- create a driver domain to handle the frequency management. Or move the
> potentially resource-intensive processes in dom0 out to a domU 

or have a cpufreq "driver domain"...

OK, I'll take a look at how we can do these things.

I'm guessing the way to start would be having either userspace
cpufreq code polling the hypervisor for CPU load, or have a
drivers/cpufreq/cpufreq_ondemand-xen.c that gets the CPU load
information from the hypervisor.

I'm guessing something like xentop already gets the right info
from the hypervisor in a relatively efficient way, or some of
the code in libvirt.

I'll dig around to see what code to hook into...

-- 
Who do you trust?
The people with all the right answers?
Or the people with the right questions?

RE: RFC: cpu frequency scaling

[Petersson, Mats]

 

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com 
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of 
> Rik van Riel
> Sent: 25 October 2006 22:13
> To: Keir Fraser
> Cc: xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] RFC: cpu frequency scaling
> 
> Keir Fraser wrote:
> 
> > Frequency changes in dom0, please. The 'small dom0' issue 
> is easily handled
> > -- create a driver domain to handle the frequency 
> management. Or move the
> > potentially resource-intensive processes in dom0 out to a domU 
> 
> or have a cpufreq "driver domain"...
> 
> OK, I'll take a look at how we can do these things.
> 
> I'm guessing the way to start would be having either userspace
> cpufreq code polling the hypervisor for CPU load, or have a
> drivers/cpufreq/cpufreq_ondemand-xen.c that gets the CPU load
> information from the hypervisor.
> 
> I'm guessing something like xentop already gets the right info
> from the hypervisor in a relatively efficient way, or some of
> the code in libvirt.
> 
> I'll dig around to see what code to hook into...

Consider that dual core CPU's (at least from AMD) will have to have the
same ("choose highest") CPU-speed both cores. This means that you need
to know which PHYSICAL CPU we're talking about. 

So you need to get the actual load of each physical CPU, then sort them
"per socket" and do any changes based on the highest (or lowest) load of
the actual socket. I'm not sure if all the necessary information is
available in Dom0 unless it's also running on all available cores -
however, neither am I sure if there's much point in pursuing the case
where all CPU's are NOT in Dom0... Allowing those cores that are not
Dom0-cores to have a driver-domain to control the speed would be
sensible solution - interesting things happen, however, if someone puts
core0 of a socket to Dom0, and Core1 of the same socket as DomU - you
can't really control that one even with a driver domain... 

--
Mats
> 
> -- 
> Who do you trust?
> The people with all the right answers?
> Or the people with the right questions?
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
> 
> 

Re: RFC: cpu frequency scaling

[Keir Fraser]

On 26/10/06 1:35 pm, "Petersson, Mats" <Mats.Petersson@amd.com> wrote:

> So you need to get the actual load of each physical CPU, then sort them
> "per socket" and do any changes based on the highest (or lowest) load of
> the actual socket. I'm not sure if all the necessary information is
> available in Dom0 unless it's also running on all available cores -
> however, neither am I sure if there's much point in pursuing the case
> where all CPU's are NOT in Dom0... Allowing those cores that are not
> Dom0-cores to have a driver-domain to control the speed would be
> sensible solution - interesting things happen, however, if someone puts
> core0 of a socket to Dom0, and Core1 of the same socket as DomU - you
> can't really control that one even with a driver domain...

We should state that as an assumption of the cpufreq domain (that it has a
one-to-one correspondence between VCPUs and PCPUs). We liekly also need an
interface to allow a domain to find out the 'topology' of its VCPUs, so that
the cpufreq drivers know which VCPUs share a socket, etc. They could also
work this out from cpuid in most cases I expect, but it's probably a better
fit with the Linux code to provide a generic 'topology info' interface and
people experimenting with NUMA will appreciate it too.

 -- Keir