VCPU migration overhead conpensation ?

VCPU migration overhead conpensation ?

[Sisu Xi]

[-- Attachment #1.1: Type: text/plain, Size: 573 bytes --]

Hi, all:

I am studying the credit2 scheduler and found it would compensate VCPU
migration with 50 microseconds.
/* How much to "compensate" a vcpu for L2 migration */
#define CSCHED_MIGRATE_COMPENSATION MICROSECS(50)

Why 50us? Is this number based on some experiments? If so, could any one
kindly indicate how to perform the overhead measurement?

Thanks very much!

Sisu


-- 
Sisu Xi, PhD Candidate

http://www.cse.wustl.edu/~xis/
Department of Computer Science and Engineering
Campus Box 1045
Washington University in St. Louis
One Brookings Drive
St. Louis, MO 63130

[-- Attachment #1.2: Type: text/html, Size: 894 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: VCPU migration overhead conpensation ?

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1433 bytes --]

On ven, 2013-03-29 at 16:16 -0500, Sisu Xi wrote:
> Hi, all:
> 
Hi again Sisu,

> I am studying the credit2 scheduler and found it would compensate VCPU
> migration with 50 microseconds.
> /* How much to "compensate" a vcpu for L2 migration */
> #define CSCHED_MIGRATE_COMPENSATION MICROSECS(50)
> 
> 
> Why 50us? Is this number based on some experiments? 
>
This predates my joining of the Xen community, but yes, ISTR hearing
people talking about it coming from a mixture of common sense and
experience/measurements.

BTW, you may want to try, when sending e-mail to this list, adding in Cc
some of the people that are more involved with the code/feature you're
asking about. You can look at the MAINTAINERS file, in the source tree,
to have an idea of who to put there. For instance, for scheduling, the
absolute authority is George (which I added), and in fact MAINTAINERS
says: 

SCHEDULING
M:      George Dunlap <george.dunlap@eu.citrix.com>
S:      Supported
F:      xen/common/sched*

(however, feel free to Cc me as well... I do not master all the Xen
scheduler code yet, but am always curious about it. :-) )

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: VCPU migration overhead conpensation ?

[Sisu Xi]

[-- Attachment #1.1: Type: text/plain, Size: 2082 bytes --]

Hi, Dario:

Thanks for your reply and tips for sending emails!  :)

I am also performing some overhead measurement for the scheduler. If a VCPU
is migrated from one core to another, the overhead is around 2 microseconds
on my machine, which is much less than what is set in Credit2 (50
microseconds).

Thanks again for your help!

Sisu


On Mon, Apr 8, 2013 at 8:46 PM, Dario Faggioli <dario.faggioli@citrix.com>wrote:

> On ven, 2013-03-29 at 16:16 -0500, Sisu Xi wrote:
> > Hi, all:
> >
> Hi again Sisu,
>
> > I am studying the credit2 scheduler and found it would compensate VCPU
> > migration with 50 microseconds.
> > /* How much to "compensate" a vcpu for L2 migration */
> > #define CSCHED_MIGRATE_COMPENSATION MICROSECS(50)
> >
> >
> > Why 50us? Is this number based on some experiments?
> >
> This predates my joining of the Xen community, but yes, ISTR hearing
> people talking about it coming from a mixture of common sense and
> experience/measurements.
>
> BTW, you may want to try, when sending e-mail to this list, adding in Cc
> some of the people that are more involved with the code/feature you're
> asking about. You can look at the MAINTAINERS file, in the source tree,
> to have an idea of who to put there. For instance, for scheduling, the
> absolute authority is George (which I added), and in fact MAINTAINERS
> says:
>
> SCHEDULING
> M:      George Dunlap <george.dunlap@eu.citrix.com>
> S:      Supported
> F:      xen/common/sched*
>
> (however, feel free to Cc me as well... I do not master all the Xen
> scheduler code yet, but am always curious about it. :-) )
>
> Regards,
> Dario
>
> --
> <<This happens because I choose it to happen!>> (Raistlin Majere)
> -----------------------------------------------------------------
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
>
>


-- 
Sisu Xi, PhD Candidate

http://www.cse.wustl.edu/~xis/
Department of Computer Science and Engineering
Campus Box 1045
Washington University in St. Louis
One Brookings Drive
St. Louis, MO 63130

[-- Attachment #1.2: Type: text/html, Size: 3016 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: VCPU migration overhead conpensation ?

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1715 bytes --]

On mer, 2013-04-10 at 06:44 +0100, Sisu Xi wrote:
> Hi, Dario:
> 
Hi! :-)

> I am also performing some overhead measurement for the scheduler. If a
> VCPU is migrated from one core to another, the overhead is around 2
> microseconds on my machine, which is much less than what is set in
> Credit2 (50 microseconds). 
> 
Ok, and, finally, how did you decide to measure this? I mean, what is it
that you are doing for coming up with that 2us value?

When I was doing research on task scheduling, in "plain" OS, not
hypervisors, I remember having setup something like this: you take a
task (or a set of them), performing a known computation on a certain
CPU, and you measure how long it takes. After that, you force a
migration to another CPU, and keep monitoring the duration of the
computation. Well, the first samples that you get right  after the
migration, gives you an idea of the overhead caused by the migration
itself.

Of course, you can/should try different kind of migrations, i.e., across
cores (i.e., with a lot of shared caches), across sockets, etc.

It is an indirect method, but the nice thing is you can use real
programs and inspect the effect of migration on real workloads (e.g., a
video player playing the same video).

Of course, we have one more "layer of indirection" here (the vCPUs), but
I guess we can think of something... So, again, I'm curious, how are you
measuring it?

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: VCPU migration overhead conpensation ?

[George Dunlap]

(BTW, please don't top-post on xen-devel: respond in-line to the thing 
you want to respond to.)

On 10/04/13 06:44, Sisu Xi wrote:
> Hi, Dario:
>
> Thanks for your reply and tips for sending emails!  :)
>
> I am also performing some overhead measurement for the scheduler. If a 
> VCPU is migrated from one core to another, the overhead is around 2 
> microseconds on my machine, which is much less than what is set in 
> Credit2 (50 microseconds).

When you say "overhead", I assume you mean that's how long the whole 
migration takes?

The point of the compensation isn't so much for the actual migration 
itself, but for the lower performance the vcpu will get after the 
migration due to having cold caches.

  -George

Re: VCPU migration overhead conpensation ?

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1299 bytes --]

On mer, 2013-04-10 at 09:21 +0100, George Dunlap wrote
> On 10/04/13 06:44, Sisu Xi wrote:
> > I am also performing some overhead measurement for the scheduler. If a 
> > VCPU is migrated from one core to another, the overhead is around 2 
> > microseconds on my machine, which is much less than what is set in 
> > Credit2 (50 microseconds).
> 
> When you say "overhead", I assume you mean that's how long the whole 
> migration takes?
> 
> The point of the compensation isn't so much for the actual migration 
> itself, but for the lower performance the vcpu will get after the 
> migration due to having cold caches.
> 
Which is, BTW, right the effect that we were trying to measure
(although, again, it was Linux, not Xen at that time), with the
experiments I was describing in my e-mail...

Might have been obvious, but I think it's worth making it even more that
(thanks George :-) ), since I agree with George that _this_ is what we
should be concerned about, when it comes to migration.

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: VCPU migration overhead conpensation ?

[Sisu Xi]

[-- Attachment #1.1: Type: text/plain, Size: 3697 bytes --]

Hi, Dario and George:

Thanks for your reply. I am doing the measurement within Xen. I agree that
a more precise description would be " the time spent performing scheduling
and context switch ". and it does NOT consider cache. (I think the cache
effect should be measured in the application level. not in hypervisor.)

Basically I add code in trace.h, schedule.c, and domain.c (for
context_switch). I measure three time here:
1. The time spent in making scheduling decisions. defined as scheduling
latency. It includes grabbing the spinlock, call the scheduler dependent
do_scheduler() function, and release the spinlock;
2. If the scheduler make a decision to perform a context switch (prev !=
next), it will call context_switch() in domain.c. I measure the time spent,
right before the context_saved(prev);
3. The time spent in context_saved(prev), which is to insert the current
VCPU back to runq. If a global queue is used and protected by a spinlock,
this would include the time to grab the lock and so on.

I ran the experiments on a i7 6 core machine, hyper-thread disabled,
speedstep disabled. It has one socket. each core has dedicated L1 and L2
cache, and all of them share L3 cache. Host is CentOS 6.3 with Linux
3.4.35, Xen is 4.1.4.
Domain 0 boots with one VCPU and pinned to core 0. one guest VM boots with
5 VCPUs, and the cpumask set for the remaining 5 VCPUs. I am using a gedf
scheduler which I am working on. it is essentially a sedf scheduler sharing
one global queue.
In the guest VM, I just ran 5 cpu busy loops to make all the VCPUs busy.

The data is get via xentrace with -M option, and I measured it for 10
seconds.

The numbers are scheduler dependent, but the absolute number spent in
context_switch() should be scheduler independent. And in my measurement,
the worst case happens when we do context switch from a IDLE VCPU to a busy
VCPU (which is scheduled previously on a different core), and perform the
tlb_flush. It is around 2 microseconds on my machine.

I am attaching the cdf plots of these three measurements.

Any feedback is welcome.

Thanks.

Sisu







On Wed, Apr 10, 2013 at 3:25 AM, Dario Faggioli
<dario.faggioli@citrix.com>wrote:

> On mer, 2013-04-10 at 09:21 +0100, George Dunlap wrote
> > On 10/04/13 06:44, Sisu Xi wrote:
> > > I am also performing some overhead measurement for the scheduler. If a
> > > VCPU is migrated from one core to another, the overhead is around 2
> > > microseconds on my machine, which is much less than what is set in
> > > Credit2 (50 microseconds).
> >
> > When you say "overhead", I assume you mean that's how long the whole
> > migration takes?
> >
> > The point of the compensation isn't so much for the actual migration
> > itself, but for the lower performance the vcpu will get after the
> > migration due to having cold caches.
> >
> Which is, BTW, right the effect that we were trying to measure
> (although, again, it was Linux, not Xen at that time), with the
> experiments I was describing in my e-mail...
>
> Might have been obvious, but I think it's worth making it even more that
> (thanks George :-) ), since I agree with George that _this_ is what we
> should be concerned about, when it comes to migration.
>
> Regards,
> Dario
>
> --
> <<This happens because I choose it to happen!>> (Raistlin Majere)
> -----------------------------------------------------------------
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
>
>


-- 
Sisu Xi, PhD Candidate

http://www.cse.wustl.edu/~xis/
Department of Computer Science and Engineering
Campus Box 1045
Washington University in St. Louis
One Brookings Drive
St. Louis, MO 63130

[-- Attachment #1.2: Type: text/html, Size: 4963 bytes --]

[-- Attachment #2: gedf_one_nopin_context_saved.eps --]
[-- Type: application/postscript, Size: 13227 bytes --]

[-- Attachment #3: gedf_one_nopin_context_switch.eps --]
[-- Type: application/postscript, Size: 20994 bytes --]

[-- Attachment #4: gedf_one_nopin_sched_latency.eps --]
[-- Type: application/postscript, Size: 30746 bytes --]

[-- Attachment #5: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel