From mboxrd@z Thu Jan 1 00:00:00 1970 From: David Xu Subject: Re: Re: performance of credit2 on hybrid workload Date: Wed, 8 Jun 2011 17:43:18 -0400 Message-ID: References: <1306340309.21026.8524.camel@elijah> <1306401493.21026.8526.camel@elijah> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0568000276==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: George Dunlap Cc: xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org --===============0568000276== Content-Type: multipart/alternative; boundary=20cf305b0c267ec7c704a53a3924 --20cf305b0c267ec7c704a53a3924 Content-Type: text/plain; charset=ISO-8859-1 Hi George, Thanks for your reply. I have similar ideas to you, adding another parameter that indicates the required latency and then letting scheduler determine latency characteristics of a VM automatically. Firstly, adding another parameter and let users set its value in advance sounds similar to SEDF. But sometimes the configuration process is hard and inflexible when workloads in VM is complex. So in my opinion, a task-aware scheduler is better. However, manually configuration can help us to check out the effectiveness of the new parameter. For another hand, as you described, it is also not easy and accurate to make scheduler determine the latency characteristics of a VM automatically with some information we can get from hypervisor, for instance the delay interrupt. Therefore, the key point for me is to find and implement a scheduling helper to indicate which VM should be scheduled soon. For example, for TCP network, we can implement a tool similar to a packet sniffer to capture the packet and analyze its head information to infer the type of workload. Then the analysis result can help scheduler to make a decision. In fact, not all I/O-intensive workloads require low latency, some of them only require high-throughput. Of course, scheduling latency impact significantly the throughput (You handled this problem with boost mechanism to some extension). What I want to is to only reduce the latency of a VM which require low latency while postpone other VMs, and use other technology such as packet offloading to compensate their lost and improve their throughput. This is just my course idea and there are many problems as well. I hope I can often discuss with you and share our results. Thanks very much. Regards, Cong 2011/6/8 George Dunlap > On Tue, Jun 7, 2011 at 8:28 PM, David Xu wrote: > > Hi George, > > Could you share some ideas about how to addressed the "mixed workload" > > problem, where a single VM does both > > cpu-intensive and latency-sensitive workloads, even though you haven't > > implemented it yet? I am also working on it, maybe I can try some > methods > > and give you feedback. Thanks. > > Well the main thing to remember is that you can't give the VM any > *more* time. The amount of time it's allowed is defined by the > scheduler parameters (and the other VMs running). So all you can do > is change *when* the VM gets the time. So what you want the scheduler > to do is give the VM shorter timeslices *so that* it can get time more > frequently. > > For example, the credit1 scheduler will let a VM burn through 30ms of > credit. That means if its "fair share" is (say) 50%, then it has to > wait at least 30ms before being allowed to run again in order to > maintain fairness. If its "fair share" is 33%, then the VM has to > wait at least 60ms. If the scheduler were to preempt it after 5ms, > then the VM would only have to be delayed for 5ms or 10ms, > respectively; and if it were preempted after 1ms, it would only have > to be delayed 1s or 2s. > > So the real key to giving a VM with a mixed workload better latency > characteristics is not to wake it up sooner, but to preempt it sooner. > > The problem is, of course, that preempting workloads which are *not* > latency sensitive too soon adds scheduling overhead, and reduces cache > effectiveness. So the question becomes, how do I know how long to let > a VM run for? > > One solution would be to introduce a scheduling parameter that will > tell the scheduler how long to set the preemption timer for. Then if > an administrator knows he's running a mixed-workload VM, he can > shorten it down; or if he knows he's running a cpu-cruncher, he can > make it longer. This would also be useful in verifying the logic of > "shorter timeslices -> less latency for mixed workloads"; i.e,. we > could vary this number and see the effects. > > One issue with adding this to the credit1 scheduler is that the > credit1 scheduler is that there are only 3 priorities (BOOST, UNDER, > and OVER), and scheduling is round-robin within each priority. It's a > known issue with round-robin scheduling that tasks which yield (or are > preempted soon) are discriminated against compared to tasks which use > up their full timeslice (or are preempted less soon). So there > results may not be representative. > > The next step would be to try to get the scheduler to determine the > latency characteristics of a VM automatically. The key observation > here is that most of the time, latency-sensitive operations are > initiated with an interrupt; or to put it the other way, a pending > interrupt generally means that there is a latency sensitive operation > waiting to happen. My idea was to have the scheduler look at the > historical rate of interrupts and determine a preemption timeslice > based on those, such that on average, the VM's credit would be enough > to run just when the next interrupt arrived for it to handle. > > It occurs to me now that after a certain point, interrupts themselves > become inefficient and drivers sometimes go into "polling" mode, which > would look to the scheduler the same as cpu-bound. Hmm... bears > thinking about. :-) > > Anyway, that's where I got in my thinking on this. Let me know what > you think. :-) > > -George > --20cf305b0c267ec7c704a53a3924 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi George,

Thanks for your reply. I have similar ideas t= o you, adding another parameter that indicates the required latency and the= n letting scheduler determine=A0=A0determine the=A0latency characteristics of a VM automatically wit= h some information we can get from hypervisor, for instance the delay inter= rupt. Therefore, the key point for me is to find and implement a scheduling= helper to indicate which VM should be scheduled soon. For example, for TCP= network, we can implement a tool similar to a packet sniffer to capture th= e packet and analyze its head information to infer the type of workload. Th= en the analysis result can help scheduler to make a decision. In fact, not = all I/O-intensive workloads require low latency, some of them only require = high-throughput. Of course, scheduling latency impact significantly the thr= oughput (You handled this problem with boost mechanism to some extension). = What I want to is to only reduce the latency of a VM which require low late= ncy while postpone other VMs, and use other technology=A0such as packet offloading=A0to compensate their lost and improve= their throughput.

This is just my course idea and t= here are many problems as well. I hope I can often discuss with you and sha= re our results. Thanks very much.

Regards,
Cong

2011/6/8 George Dunlap <George.Dunlap= @eu.citrix.com>
On Tue, Jun 7, 2011 at 8:= 28 PM, David Xu <davidxu06@gmail.= com> wrote:
> Hi George,
> Could you share some ideas about how to addressed the=A0=A0"mixed= workload"
> problem,=A0=A0where a single VM does both
> cpu-intensive and latency-sensitive workloads, even though you haven&#= 39;t
> implemented it yet? =A0I am also working on it, maybe I can try some m= ethods
> and give you feedback. Thanks.

Well the main thing to remember is that you can't give the VM any=
*more* time. =A0The amount of time it's allowed is defined by the
scheduler parameters (and the other VMs running). =A0So all you can do
is change *when* the VM gets the time. =A0So what you want the scheduler to do is give the VM shorter timeslices *so that* it can get time more
frequently.

For example, the credit1 scheduler will let a VM burn through 30ms of
credit. =A0That means if its "fair share" is (say) 50%, then it h= as to
wait at least 30ms before being allowed to run again in order to
maintain fairness. =A0If its "fair share" is 33%, then the VM has= to
wait at least 60ms. =A0If the scheduler were to preempt it after 5ms,
then the VM would only have to be delayed for 5ms or 10ms,
respectively; and if it were preempted after 1ms, it would only have
to be delayed 1s or 2s.

So the real key to giving a VM with a mixed workload better latency
characteristics is not to wake it up sooner, but to preempt it sooner.

The problem is, of course, that preempting workloads which are *not*
latency sensitive too soon adds scheduling overhead, and reduces cache
effectiveness. =A0So the question becomes, how do I know how long to let a VM run for?

One solution would be to introduce a scheduling parameter that will
tell the scheduler how long to set the preemption timer for. =A0Then if
an administrator knows he's running a mixed-workload VM, he can
shorten it down; or if he knows he's running a cpu-cruncher, he can
make it longer. =A0This would also be useful in verifying the logic of
"shorter timeslices -> less latency for mixed workloads"; i.e,= . we
could vary this number and see the effects.

One issue with adding this to the credit1 scheduler is that the
credit1 scheduler is that there are only 3 priorities (BOOST, UNDER,
and OVER), and scheduling is round-robin within each priority. =A0It's = a
known issue with round-robin scheduling that tasks which yield (or are
preempted soon) are discriminated against compared to tasks which use
up their full timeslice (or are preempted less soon). =A0So there
results may not be representative.

The next step would be to try to get the scheduler to determine the
latency characteristics of a VM automatically. =A0The key observation
here is that most of the time, latency-sensitive operations are
initiated with an interrupt; or to put it the other way, a pending
interrupt generally means that there is a latency sensitive operation
waiting to happen. =A0My idea was to have the scheduler look at the
historical rate of interrupts and determine a preemption timeslice
based on those, such that on average, the VM's credit would be enough to run just when the next interrupt arrived for it to handle.

It occurs to me now that after a certain point, interrupts themselves
become inefficient and drivers sometimes go into "polling" mode, = which
would look to the scheduler the same as cpu-bound. =A0Hmm... bears
thinking about. :-)

Anyway, that's where I got in my thinking on this. Let me know what
you think. :-)

=A0-George

--20cf305b0c267ec7c704a53a3924-- --===============0568000276== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --===============0568000276==--