From mboxrd@z Thu Jan 1 00:00:00 1970 From: George Dunlap Subject: Re: Xen credit scheduler question Date: Thu, 15 Nov 2012 19:52:53 +0000 Message-ID: <50A54815.9010402@eu.citrix.com> References: <50A53479.5050901@eu.citrix.com> <27449f60-0433-4e5f-b1fb-06914b84c6f1@default> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============5647842968013421696==" Return-path: In-Reply-To: <27449f60-0433-4e5f-b1fb-06914b84c6f1@default> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Michael Palmeter Cc: Ashok Aletty , Dario Faggioli , "xen-devel@lists.xen.org" List-Id: xen-devel@lists.xenproject.org --===============5647842968013421696== Content-Type: multipart/alternative; boundary="------------090604080201040700060707" --------------090604080201040700060707 Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 8bit On 15/11/12 19:03, Michael Palmeter wrote: > > Thank you for your answer, George. > > The origin of my question is more of a business concern than a > technical one. Many software products are licensed based on a cost > per processor core. It is desirable to sometimes allow customers to > pay a fraction of software license costs in exchange for running that > software using only a commensurate fraction of available compute power > (capacity sub-licensing). If the cap is a means of making a vCPU > more-or-less deterministic (in terms of its effective computational > capacity) then that would be useful as a programmatic means of > enabling capacity sub-licensing. My example below was based on a case > where I have a customer that would like to use ‘cap’ to constrain > their single vCPU VM to only ˝ of a core worth of compute capacity > (logically 1/32 of the compute power) in exchange for only paying 1/32 > of the license cost for the physical server. > Right -- I've seen the "limit cpu power for licensing purposes" thing before, but I think that only went down to cores, not sub-core. > Below you answered: > > “You can use ‘cap’ to make the VM in question get 50% of logical vcpu > time, which on an idle system will give it 0.5 of the capacity of a > physical core (if we don't consider Intel's Turbo Boost technology). > But if the system becomes busy, it will get less than 0.5 of the > processing capacity of a physical core.” > > Are you saying that cap would be able to CONSTRAIN a vCPU to an > effective compute capacity equal to 50% of a physical core, but it > does not GUARANTEE effective compute capacity equal to 50% of a > physical core? > Theoretically, a cap at 50 will give your single-vcpu VM 50% of the time of one hyperthread. So if C is "typicall throughput of a single non-hyperthreaded core running at standard requency", and we factor out Turbo Boost, then there are two cases to consider: * Other thread is idle. In that case, the VM will get 0.5C. * The other thread is busy. In this case, assuming a 0.7 factor, the VM will get 0.5 * (0.7 * C), or about 0.35C So the total computing power available to the VM should be <= 0.5C (satisfying the licensing requirements), but on a busy system it may be significantly less than 0.5C (perhaps not so satisfying to the owner of the VM). I don't think it should be terribly difficult to put a simple "shared hyperthread" multiplier on the credit burned -- if someone at Oracle wanted to help implement this, we'd be happy to point you in the right direction. :-) If you have Turbo Boost, then (as I understand it) the CPU can raise the clock speed of the processor when threads or cores are idle; the wikipedia article seems to think some processors can increase the clock speed up to 1.6x over the baseline frequency. That would throw a bit of a wrench in the works, as you might end up with 0.5 * 1.6 * C = 0.8 C > 0.5 C; however, looking at Intel's website, it looks like only 2- and 4-core processors have TurboBoost, so maybe on 8-core processors we can punt on that thorny issue for a little while yet. :-) > Can you offer any guidance regarding real-world scheduler overhead > (when cap>0 is used) and precision (how variable is actual compute > power for a vCPU with a cap of 100%, for example)? > I have not done extensive testing with the cap; I mainly know the mechanism by which it works. There is no extra accounting done in the scheduler for having a cap: all vcpus are assigned credit every 30ms according to their weight and cap. The difference is that if a non-capped vcpu uses up its credits, it is allowed to go negative; whereas a capped vcpu will be paused until it receives more credits. So there should be no extra hypervisor overhead from using a cap. The cap fundamentally works by locking out a vcpu for very small amounts of time within the 30ms accounting window. But this same effect might happen just by having other VMs competing for the cpu; so in theory shouldn't be any riskier than virtualizing in the first place. Executive summary: Factoring out Turbo Boost, "cap" should be able to set a sub-core upper-bound on processing power. But on a busy system, it may result in the VM getting less than its upper-bound in processing power. However, scheduling is a very complex and dynamic system, and like economics, very simple changes can have unpredictable results. So it's probably a good idea to do some testing before recommending it to customers. :-) BTW, are you familiar with Xen's cpupool functionality? The guys at Fujitsu wrote it so that a customer could rent a fixed number of cores to a customer, who could then run as many VMs on those cores as they wanted. I think licensing restrictions had something to do with that as well. More about that here, if you're interested: http://blog.xen.org/index.php/2012/04/23/xen-4-2-cpupools/ -George --------------090604080201040700060707 Content-Type: text/html; charset="windows-1252" Content-Transfer-Encoding: 8bit
On 15/11/12 19:03, Michael Palmeter wrote:

Thank you for your answer, George.

 

The origin of my question is more of a business concern than a technical one.  Many software products are licensed based on a cost per processor core.  It is desirable to sometimes allow customers to pay a fraction of software license costs in exchange for running that software using only a commensurate fraction of available compute power (capacity sub-licensing).  If the cap is a means of making a vCPU more-or-less deterministic (in terms of its effective computational capacity) then that would be useful as a programmatic means of enabling capacity sub-licensing.  My example below was based on a case where I have a customer that would like to use ‘cap’ to constrain their single vCPU VM to only ˝ of a core worth of compute capacity (logically 1/32 of the compute power) in exchange for only paying 1/32 of the license cost for the physical server.


Right -- I've seen the "limit cpu power for licensing purposes" thing before, but I think that only went down to cores, not sub-core.

 

Below you answered:

 

“You can use ‘cap’ to make the VM in question get 50% of logical vcpu time, which on an idle system will give it 0.5 of the capacity of a physical core (if we don't consider Intel's Turbo Boost technology).  But if the system becomes busy, it will get less than 0.5 of the processing capacity of a physical core.”

 

Are you saying that cap would be able to CONSTRAIN a vCPU to an effective compute capacity equal to 50% of a physical core, but it does not GUARANTEE effective compute capacity equal to 50% of a physical core?


Theoretically, a cap at 50 will give your single-vcpu VM 50% of the time of one hyperthread.

So if C is "typicall throughput of a single non-hyperthreaded core running at standard requency", and we factor out Turbo Boost, then there are two cases to consider:

* Other thread is idle.  In that case, the VM will get 0.5C.
* The other thread is busy.  In this case, assuming a 0.7 factor, the VM will get 0.5 * (0.7 * C), or about 0.35C

So the total computing power available to the VM should be <= 0.5C (satisfying the licensing requirements), but on a busy system it may be significantly less than 0.5C (perhaps not so satisfying to the owner of the VM).

I don't think it should be terribly difficult to put a simple "shared hyperthread" multiplier on the credit burned -- if someone at Oracle wanted to help implement this, we'd be happy to point you in the right direction. :-)

If you have Turbo Boost, then (as I understand it) the CPU can raise the clock speed of the processor when threads or cores are idle; the wikipedia article seems to think some processors can increase the clock speed up to 1.6x over the baseline frequency.  That would throw a bit of a wrench in the works, as you might end up with 0.5 * 1.6 * C = 0.8 C > 0.5 C; however, looking at Intel's website, it looks like only 2- and 4-core processors have TurboBoost, so maybe on 8-core processors we can punt on that thorny issue for a little while yet. :-)

Can you offer any guidance regarding real-world scheduler overhead (when cap>0 is used) and precision (how variable is actual compute power for a vCPU with a cap of 100%, for example)?


I have not done extensive testing with the cap; I mainly know the mechanism by which it works.  There is no extra accounting done in the scheduler for having a cap: all vcpus are assigned credit every 30ms according to their weight and cap.  The difference is that if a non-capped vcpu uses up its credits, it is allowed to go negative; whereas a capped vcpu will be paused until it receives more credits.  So there should be no extra hypervisor overhead from using a cap.

The cap fundamentally works by locking out a vcpu for very small amounts of time within the 30ms accounting window.  But this same effect might happen just by having other VMs competing for the cpu; so in theory shouldn't be any riskier than virtualizing in the first place.

Executive summary: Factoring out Turbo Boost, "cap" should be able to set a sub-core upper-bound on processing power.  But on a busy system, it may result in the VM getting less than its upper-bound in processing power.

However, scheduling is a very complex and dynamic system, and like economics, very simple changes can have unpredictable results.  So it's probably a good idea to do some testing before recommending it to customers. :-)

BTW, are you familiar with Xen's cpupool functionality?  The guys at Fujitsu wrote it so that a customer could rent a fixed number of cores to a customer, who could then run as many VMs on those cores as they wanted.  I think licensing restrictions had something to do with that as well.  More about that here, if you're interested:
 http://blog.xen.org/index.php/2012/04/23/xen-4-2-cpupools/

 -George
--------------090604080201040700060707-- --===============5647842968013421696== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============5647842968013421696==--