[RFC] [Design] Better XL support of RTDS scheduler for Xen 4.6

[RFC] [Design] Better XL support of RTDS scheduler for Xen 4.6

[Meng Xu]

Hi all,

This is for Xen 4.6: Enabling XL to set the parameters of each
individual VCPU of a domain for RTDS scheduler.

[Goal]
Right now, xl sched-rtds tool can only set the VCPUs of a domain to
the same parameter although the scheduler did support VCPUs with
different parameters. This work is to provide the ‘xl sched-rtds’ tool
the ability to configure the VCPUs of a domain with different
parameters.

After this has been implemented, an example of the functionality of XL
will look like this:

#xl sched-rtds
Cpupool Pool-0: sched=RTDS
Name                                ID VCPU Period Budget
Domain-0                             0    0  10000  10000
Domain-0                             0    1  20000  20000
Domain-0                             0    2  30000  30000
Domain-0                             0    3  10000  10000
litmus1                              1    0  10000   4000
litmus1                              1    1  10000   4000
//comment: VCPU is the index of the VCPUs in the domain. The
parameters of VCPUs in the same domain are different, while they are
the same in Xen 4.5.

//set the parameters of the vcpu 1 of domain litmus1:
# xl sched-rtds -d litmus1 -v 1 -p 20000 -b 10000

//domain litmus1's vcpu 1's parameters are changed, display each VCPU's
parameters separately:
# xl sched-rtds -d litmus1
Name                                ID VCPU Period Budget
litmus1                              1    0  10000   4000
litmus1                              1    1  20000  10000

[Design sketch]
We submitted the patch along with the RTDS scheduler before last year.
The thread can be found at
http://lists.xen.org/archives/html/xen-devel/2014-08/msg02245.html.

In order to implement the get function, which return the parameters of
all VCPUs of the same domain, we can bounce an array that has all
parameters of all VCPUs of the domain from hypervisor to userspace and
output it. (This part didn't cause much buzz. :-) )
The other choice is using a hypercall to get the information of each
VCPU. If there are x VCPUs in a domain, we will issue x hypercalls to
get all information of this domain. This choice causes more overhead
since the number of hypercalls is linear to the number of VCPUs in a
domain. The first choice only needs one hypercall for a domain to get
all needed information.

In order to implement the set function, which set the parameters of
one specific VCPU of a domain, we have two different approaches to
implement it:
Choice 1): When users set the parameters of k^th VCPUs of a domain,
which has x VCPUs in total, the toolstack passes "only" the modified
VCPUs’ information from userspace to hypervisor via hypercall;  So
this implementation does "not" need an array to bounce between
userspace and hypervisor. (The implementation in lilbxc was at
http://lists.xen.org/archives/html/xen-devel/2014-08/msg02248.html)
Choice 2): When users set the parameters of k^th VCPUs of a domain,
which has x VCPUs in total, the toolstack will create an array with
length x and set the information (period, budget) of the k^th element
in the array. The other elements in the array are marked as 0. After
the toolstack bounce the array into the hypervisor, it will only set
the parameters of the k^th VCPU.

My questions are:
1) Should we allow users to set the parameters of "several" VCPUs with
one command:
For example, should we allow users to set the parameters of VCPU 1 and
2 of domain litmus1 with one xl sched-rtds command?
# xl sched-rtds -d litmus1 -v 1 -p 20000 -b 10000 -v 2 -p 30000 -b 20000
IMO, we'd better not allow them to do this because 1) setting the
parameters of VCPUs of a domain should not be very frequent; 2) the
command will become quite long when users want to set the parameters
of many VCPUs of a domain.

2) Which design choice should we choose to implement the set function,
Choice 1 or Choice 2? or do you have better suggestion?

Thank you very much!

best,

Meng
-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [RFC] [Design] Better XL support of RTDS scheduler for Xen 4.6

[Meng Xu]

[Adding Denys and Andri from globallogic  for their advice about the
user interface and why we need such functionality]

2015-02-21 10:51 GMT-05:00 Meng Xu <xumengpanda@gmail.com>:
> Hi all,
>
> This is for Xen 4.6: Enabling XL to set the parameters of each
> individual VCPU of a domain for RTDS scheduler.
>
> [Goal]
> Right now, xl sched-rtds tool can only set the VCPUs of a domain to
> the same parameter although the scheduler did support VCPUs with
> different parameters. This work is to provide the ‘xl sched-rtds’ tool
> the ability to configure the VCPUs of a domain with different
> parameters.
>
> After this has been implemented, an example of the functionality of XL
> will look like this:
>
> #xl sched-rtds
> Cpupool Pool-0: sched=RTDS
> Name                                ID VCPU Period Budget
> Domain-0                             0    0  10000  10000
> Domain-0                             0    1  20000  20000
> Domain-0                             0    2  30000  30000
> Domain-0                             0    3  10000  10000
> litmus1                              1    0  10000   4000
> litmus1                              1    1  10000   4000
> //comment: VCPU is the index of the VCPUs in the domain. The
> parameters of VCPUs in the same domain are different, while they are
> the same in Xen 4.5.
>
> //set the parameters of the vcpu 1 of domain litmus1:
> # xl sched-rtds -d litmus1 -v 1 -p 20000 -b 10000
>
> //domain litmus1's vcpu 1's parameters are changed, display each VCPU's
> parameters separately:
> # xl sched-rtds -d litmus1
> Name                                ID VCPU Period Budget
> litmus1                              1    0  10000   4000
> litmus1                              1    1  20000  10000
>
> [Design sketch]
> We submitted the patch along with the RTDS scheduler before last year.
> The thread can be found at
> http://lists.xen.org/archives/html/xen-devel/2014-08/msg02245.html.
>
> In order to implement the get function, which return the parameters of
> all VCPUs of the same domain, we can bounce an array that has all
> parameters of all VCPUs of the domain from hypervisor to userspace and
> output it. (This part didn't cause much buzz. :-) )
> The other choice is using a hypercall to get the information of each
> VCPU. If there are x VCPUs in a domain, we will issue x hypercalls to
> get all information of this domain. This choice causes more overhead
> since the number of hypercalls is linear to the number of VCPUs in a
> domain. The first choice only needs one hypercall for a domain to get
> all needed information.
>
> In order to implement the set function, which set the parameters of
> one specific VCPU of a domain, we have two different approaches to
> implement it:
> Choice 1): When users set the parameters of k^th VCPUs of a domain,
> which has x VCPUs in total, the toolstack passes "only" the modified
> VCPUs’ information from userspace to hypervisor via hypercall;  So
> this implementation does "not" need an array to bounce between
> userspace and hypervisor. (The implementation in lilbxc was at
> http://lists.xen.org/archives/html/xen-devel/2014-08/msg02248.html)
> Choice 2): When users set the parameters of k^th VCPUs of a domain,
> which has x VCPUs in total, the toolstack will create an array with
> length x and set the information (period, budget) of the k^th element
> in the array. The other elements in the array are marked as 0. After
> the toolstack bounce the array into the hypervisor, it will only set
> the parameters of the k^th VCPU.
>
> My questions are:
> 1) Should we allow users to set the parameters of "several" VCPUs with
> one command:
> For example, should we allow users to set the parameters of VCPU 1 and
> 2 of domain litmus1 with one xl sched-rtds command?
> # xl sched-rtds -d litmus1 -v 1 -p 20000 -b 10000 -v 2 -p 30000 -b 20000
> IMO, we'd better not allow them to do this because 1) setting the
> parameters of VCPUs of a domain should not be very frequent; 2) the
> command will become quite long when users want to set the parameters
> of many VCPUs of a domain.
>
> 2) Which design choice should we choose to implement the set function,
> Choice 1 or Choice 2? or do you have better suggestion?
>
> Thank you very much!
>
> best,
>
> Meng
> -----------
> Meng Xu
> PhD Student in Computer and Information Science
> University of Pennsylvania

Meng

-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [RFC] [Design] Better XL support of RTDS scheduler for Xen 4.6

[Wei Liu]

Not an expert on scheduler, so bear with me if my questions / comments
are stupid.

On Sat, Feb 21, 2015 at 10:51:59AM -0500, Meng Xu wrote:
> Hi all,
> 
> This is for Xen 4.6: Enabling XL to set the parameters of each
> individual VCPU of a domain for RTDS scheduler.
> 
> [Goal]
> Right now, xl sched-rtds tool can only set the VCPUs of a domain to
> the same parameter although the scheduler did support VCPUs with
> different parameters. This work is to provide the ‘xl sched-rtds’ tool
> the ability to configure the VCPUs of a domain with different
> parameters.
> 
> After this has been implemented, an example of the functionality of XL
> will look like this:
> 
> #xl sched-rtds
> Cpupool Pool-0: sched=RTDS
> Name                                ID VCPU Period Budget
> Domain-0                             0    0  10000  10000
> Domain-0                             0    1  20000  20000
> Domain-0                             0    2  30000  30000
> Domain-0                             0    3  10000  10000
> litmus1                              1    0  10000   4000
> litmus1                              1    1  10000   4000
> //comment: VCPU is the index of the VCPUs in the domain. The
> parameters of VCPUs in the same domain are different, while they are
> the same in Xen 4.5.
> 
> //set the parameters of the vcpu 1 of domain litmus1:
> # xl sched-rtds -d litmus1 -v 1 -p 20000 -b 10000
> 
> //domain litmus1's vcpu 1's parameters are changed, display each VCPU's
> parameters separately:
> # xl sched-rtds -d litmus1
> Name                                ID VCPU Period Budget
> litmus1                              1    0  10000   4000
> litmus1                              1    1  20000  10000
> 
> [Design sketch]
> We submitted the patch along with the RTDS scheduler before last year.
> The thread can be found at
> http://lists.xen.org/archives/html/xen-devel/2014-08/msg02245.html.
> 
> In order to implement the get function, which return the parameters of
> all VCPUs of the same domain, we can bounce an array that has all
> parameters of all VCPUs of the domain from hypervisor to userspace and
> output it. (This part didn't cause much buzz. :-) )
> The other choice is using a hypercall to get the information of each
> VCPU. If there are x VCPUs in a domain, we will issue x hypercalls to
> get all information of this domain. This choice causes more overhead
> since the number of hypercalls is linear to the number of VCPUs in a
> domain. The first choice only needs one hypercall for a domain to get
> all needed information.
> 
> In order to implement the set function, which set the parameters of
> one specific VCPU of a domain, we have two different approaches to
> implement it:
> Choice 1): When users set the parameters of k^th VCPUs of a domain,
> which has x VCPUs in total, the toolstack passes "only" the modified
> VCPUs’ information from userspace to hypervisor via hypercall;  So

A new hypercall or existing hypercall?

> this implementation does "not" need an array to bounce between
> userspace and hypervisor. (The implementation in lilbxc was at
> http://lists.xen.org/archives/html/xen-devel/2014-08/msg02248.html)
> Choice 2): When users set the parameters of k^th VCPUs of a domain,
> which has x VCPUs in total, the toolstack will create an array with
> length x and set the information (period, budget) of the k^th element
> in the array. The other elements in the array are marked as 0. After
> the toolstack bounce the array into the hypervisor, it will only set
> the parameters of the k^th VCPU.
> 

One thing to consider is when you have hundreds of cpus the overhead
might be too big? What's the amount of data you need to pass down to
hypervisor?

> My questions are:
> 1) Should we allow users to set the parameters of "several" VCPUs with
> one command:
> For example, should we allow users to set the parameters of VCPU 1 and
> 2 of domain litmus1 with one xl sched-rtds command?
> # xl sched-rtds -d litmus1 -v 1 -p 20000 -b 10000 -v 2 -p 30000 -b 20000
> IMO, we'd better not allow them to do this because 1) setting the
> parameters of VCPUs of a domain should not be very frequent; 2) the
> command will become quite long when users want to set the parameters
> of many VCPUs of a domain.
> 
> 2) Which design choice should we choose to implement the set function,
> Choice 1 or Choice 2? or do you have better suggestion?
> 

I have some general questions. Here in this email you only mention xl. I
think in fact you mean xl + libxl? We surely want to enable other
toolstacks to control specific parameters of schedulers as well.

And I presume in the future other schedulers might also want to have
some per-vcpu tuning interface. Is it possible to abstract out common
interface in hypervisor and libxl to do that? Even if we can't sensible
come up with a hypervisor interface to do that, we might still want to
do that inside libxl. I.e. to provide some kind of common infrastructure
that individual scheduler can hook in.

This is all vague idea. If my idea is stupid feel free to disagree.

Wei.

> Thank you very much!
> 
> best,
> 
> Meng
> -----------
> Meng Xu
> PhD Student in Computer and Information Science
> University of Pennsylvania

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [RFC] [Design] Better XL support of RTDS scheduler for Xen 4.6

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 5872 bytes --]

> 2015-02-21 10:51 GMT-05:00 Meng Xu <xumengpanda@gmail.com>:
> > Hi all,
> >
Hey Meng... First of all, thanks for keeping up working on this, we really need it!

> > This is for Xen 4.6: Enabling XL to set the parameters of each
> > individual VCPU of a domain for RTDS scheduler.
> >
Right.

> > [Goal]
> > Right now, xl sched-rtds tool can only set the VCPUs of a domain to
> > the same parameter although the scheduler did support VCPUs with
> > different parameters. This work is to provide the ‘xl sched-rtds’ tool
> > the ability to configure the VCPUs of a domain with different
> > parameters.
> >
Yes, for RTDS, we need this, much more than for the other available
schedulers we have in Xen.

The main reason, IMO, is that I see it quite common for someone to want
to pin tasks to (v)cpus inside a real-time enabled VM, and if they do
this, they expect to be able to specify the scheduling parameters for
each one of the vcpus (or for a group of vcpus), depending on what tasks
inside the guest they are pinning where.

> > [Design sketch]
> > We submitted the patch along with the RTDS scheduler before last year.
> > The thread can be found at
> > http://lists.xen.org/archives/html/xen-devel/2014-08/msg02245.html.
> >
> > In order to implement the get function, which return the parameters of
> > all VCPUs of the same domain, we can bounce an array that has all
> > parameters of all VCPUs of the domain from hypervisor to userspace and
> > output it. (This part didn't cause much buzz. :-) )
> > The other choice is using a hypercall to get the information of each
> > VCPU. If there are x VCPUs in a domain, we will issue x hypercalls to
> > get all information of this domain. This choice causes more overhead
> > since the number of hypercalls is linear to the number of VCPUs in a
> > domain. The first choice only needs one hypercall for a domain to get
> > all needed information.
> >
ISTR that at some point we agreed on the fact that, even if getting and
setting vcpu params won't happen in hot paths (neither from the tools
nor from the hypervisor point of view), batching hypercall is always
good, an we should always batch as far as we can.

That of course does not mean that we the API, at all levels, need to
always deal with all vcpus. As you know very well, there is the hcall
interface, where we said we want to batch; then there is the libxc
interface, which can well diverge, but it's usually better map quite
closely to the hcall one; and then there is libxl, on top of libxl, and
things can be significantly different here.

So, I think your design proposal/code should explicitly state what the
plan is at each level.

> > In order to implement the set function, which set the parameters of
> > one specific VCPU of a domain, we have two different approaches to
> > implement it:
> > Choice 1): When users set the parameters of k^th VCPUs of a domain,
> > which has x VCPUs in total, the toolstack passes "only" the modified
> > VCPUs’ information from userspace to hypervisor via hypercall;  So
> > this implementation does "not" need an array to bounce between
> > userspace and hypervisor. (The implementation in lilbxc was at
> > http://lists.xen.org/archives/html/xen-devel/2014-08/msg02248.html)
> > Choice 2): When users set the parameters of k^th VCPUs of a domain,
> > which has x VCPUs in total, the toolstack will create an array with
> > length x and set the information (period, budget) of the k^th element
> > in the array. The other elements in the array are marked as 0. After
> > the toolstack bounce the array into the hypervisor, it will only set
> > the parameters of the k^th VCPU.
> >
Get and set interface should match, at each level, so we should decide
once and for both of them, bearing in mind all the pros and the cons of
each solution together, not split the two decisions.

> > My questions are:
> > 1) Should we allow users to set the parameters of "several" VCPUs with
> > one command:
> > For example, should we allow users to set the parameters of VCPU 1 and
> > 2 of domain litmus1 with one xl sched-rtds command?
> > # xl sched-rtds -d litmus1 -v 1 -p 20000 -b 10000 -v 2 -p 30000 -b 20000
> > IMO, we'd better not allow them to do this because 1) setting the
> > parameters of VCPUs of a domain should not be very frequent; 2) the
> > command will become quite long when users want to set the parameters
> > of many VCPUs of a domain.
> >
I actually see some value in supporting such syntax. It's not something
critical, though, I think. For sure it doesn't have much to do with the
decision on how the interface should look like at the various level.

This means you can just go ahead and send code implementing only one '-v
xxx -p yy y -b zzz', and we can extend it later, adding the support for
the list.

> > 2) Which design choice should we choose to implement the set function,
> > Choice 1 or Choice 2? or do you have better suggestion?
> >
If we batch for get, I think we should batch for set. As I said, I
personally see some value in an interface allowing setting all the
parameters of all the vcpus at the same time. In fact, this is probably
better, even as far as real-time theory is concerned: as I'm sure you
know better than me, parameter changing can be tricky for the algorithm,
so it's probably better to encourage the user to change the params all
together, synchronously, rather than having to deal with a stream of
adjustments scattered in time.

So, my opinion, choice 2).

If I can add something, the big challenge here is how to make the
interfaces good and homogeneous enough with what we already have in
place for other schedulers. I really suggest you to address this too in
this design conversation and/or in RFC/code.

Thanks and Regards,
Dario

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [RFC] [Design] Better XL support of RTDS scheduler for Xen 4.6

[Meng Xu]

2015-02-23 10:57 GMT-05:00 Wei Liu <wei.liu2@citrix.com>:
> Not an expert on scheduler, so bear with me if my questions / comments
> are stupid.

This is more about the toolstack (xl, libxl, libxc), so you are
definitely the expert. :-P

>
> On Sat, Feb 21, 2015 at 10:51:59AM -0500, Meng Xu wrote:
>> Hi all,
>>
>> This is for Xen 4.6: Enabling XL to set the parameters of each
>> individual VCPU of a domain for RTDS scheduler.
>>
>> [Goal]
>> Right now, xl sched-rtds tool can only set the VCPUs of a domain to
>> the same parameter although the scheduler did support VCPUs with
>> different parameters. This work is to provide the ‘xl sched-rtds’ tool
>> the ability to configure the VCPUs of a domain with different
>> parameters.
>>
>> After this has been implemented, an example of the functionality of XL
>> will look like this:
>>
>> #xl sched-rtds
>> Cpupool Pool-0: sched=RTDS
>> Name                                ID VCPU Period Budget
>> Domain-0                             0    0  10000  10000
>> Domain-0                             0    1  20000  20000
>> Domain-0                             0    2  30000  30000
>> Domain-0                             0    3  10000  10000
>> litmus1                              1    0  10000   4000
>> litmus1                              1    1  10000   4000
>> //comment: VCPU is the index of the VCPUs in the domain. The
>> parameters of VCPUs in the same domain are different, while they are
>> the same in Xen 4.5.
>>
>> //set the parameters of the vcpu 1 of domain litmus1:
>> # xl sched-rtds -d litmus1 -v 1 -p 20000 -b 10000
>>
>> //domain litmus1's vcpu 1's parameters are changed, display each VCPU's
>> parameters separately:
>> # xl sched-rtds -d litmus1
>> Name                                ID VCPU Period Budget
>> litmus1                              1    0  10000   4000
>> litmus1                              1    1  20000  10000
>>
>> [Design sketch]
>> We submitted the patch along with the RTDS scheduler before last year.
>> The thread can be found at
>> http://lists.xen.org/archives/html/xen-devel/2014-08/msg02245.html.
>>
>> In order to implement the get function, which return the parameters of
>> all VCPUs of the same domain, we can bounce an array that has all
>> parameters of all VCPUs of the domain from hypervisor to userspace and
>> output it. (This part didn't cause much buzz. :-) )
>> The other choice is using a hypercall to get the information of each
>> VCPU. If there are x VCPUs in a domain, we will issue x hypercalls to
>> get all information of this domain. This choice causes more overhead
>> since the number of hypercalls is linear to the number of VCPUs in a
>> domain. The first choice only needs one hypercall for a domain to get
>> all needed information.
>>
>> In order to implement the set function, which set the parameters of
>> one specific VCPU of a domain, we have two different approaches to
>> implement it:
>> Choice 1): When users set the parameters of k^th VCPUs of a domain,
>> which has x VCPUs in total, the toolstack passes "only" the modified
>> VCPUs’ information from userspace to hypervisor via hypercall;  So
>
> A new hypercall or existing hypercall?

We only need to change the current hypercall
(XEN_DOMCTL_SCHEDOP_setinfo) implementation:


>
>> this implementation does "not" need an array to bounce between
>> userspace and hypervisor. (The implementation in lilbxc was at
>> http://lists.xen.org/archives/html/xen-devel/2014-08/msg02248.html)
>> Choice 2): When users set the parameters of k^th VCPUs of a domain,
>> which has x VCPUs in total, the toolstack will create an array with
>> length x and set the information (period, budget) of the k^th element
>> in the array. The other elements in the array are marked as 0. After
>> the toolstack bounce the array into the hypervisor, it will only set
>> the parameters of the k^th VCPU.
>>
>
> One thing to consider is when you have hundreds of cpus the overhead
> might be too big? What's the amount of data you need to pass down to
> hypervisor?

Suppose a domain have k VCPUs, each VCPU has info: period (uint32_t),
budget(uint32_t), and index(uint32_t); So each hypercall will pass k *
(3*8Bytes) to the hypervisor.

>
>> My questions are:
>> 1) Should we allow users to set the parameters of "several" VCPUs with
>> one command:
>> For example, should we allow users to set the parameters of VCPU 1 and
>> 2 of domain litmus1 with one xl sched-rtds command?
>> # xl sched-rtds -d litmus1 -v 1 -p 20000 -b 10000 -v 2 -p 30000 -b 20000
>> IMO, we'd better not allow them to do this because 1) setting the
>> parameters of VCPUs of a domain should not be very frequent; 2) the
>> command will become quite long when users want to set the parameters
>> of many VCPUs of a domain.
>>
>> 2) Which design choice should we choose to implement the set function,
>> Choice 1 or Choice 2? or do you have better suggestion?
>>
>
> I have some general questions. Here in this email you only mention xl. I
> think in fact you mean xl + libxl? We surely want to enable other
> toolstacks to control specific parameters of schedulers as well.

Yes. I need to change xl + libxl + libxc + rt_dom_cntl() in hypervisor. :-)

I'm not sure if other schedulers need such functionality right now,
because the credit2 scheduler account for the credit based on each
domain instead of each VCPU. But if the scheduler will consider the
vcpu-level credit/budget accounting, this could be helpful.

>
> And I presume in the future other schedulers might also want to have
> some per-vcpu tuning interface. Is it possible to abstract out common
> interface in hypervisor and libxl to do that? Even if we can't sensible
> come up with a hypervisor interface to do that, we might still want to
> do that inside libxl. I.e. to provide some kind of common infrastructure
> that individual scheduler can hook in.

Good point! This should be possible. I think maybe we can do this step
by step: First have a concrete implementation for the RTDS scheduler,
and then we can extend it to a more general interface.

>
> This is all vague idea. If my idea is stupid feel free to disagree.

This is very useful and insightful idea, actually! :-)

Thank you very much!

best,

Meng

>
> Wei.
>
>> Thank you very much!
>>
>> best,
>>
>> Meng
>> -----------
>> Meng Xu
>> PhD Student in Computer and Information Science
>> University of Pennsylvania



-- 


-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [RFC] [Design] Better XL support of RTDS scheduler for Xen 4.6

[Meng Xu]

2015-02-23 12:27 GMT-05:00 Dario Faggioli <dario.faggioli@citrix.com>:
>> 2015-02-21 10:51 GMT-05:00 Meng Xu <xumengpanda@gmail.com>:
>> > Hi all,
>> >
> Hey Meng... First of all, thanks for keeping up working on this, we really need it!

Since we promised, we will definitely keep our words. :-)
My plan is to have this done soon, since globallogic guys is also
waiting for it for some experiments. :-)

>
>> > This is for Xen 4.6: Enabling XL to set the parameters of each
>> > individual VCPU of a domain for RTDS scheduler.
>> >
> Right.
>
>> > [Goal]
>> > Right now, xl sched-rtds tool can only set the VCPUs of a domain to
>> > the same parameter although the scheduler did support VCPUs with
>> > different parameters. This work is to provide the ‘xl sched-rtds’ tool
>> > the ability to configure the VCPUs of a domain with different
>> > parameters.
>> >
> Yes, for RTDS, we need this, much more than for the other available
> schedulers we have in Xen.
>
> The main reason, IMO, is that I see it quite common for someone to want
> to pin tasks to (v)cpus inside a real-time enabled VM, and if they do
> this, they expect to be able to specify the scheduling parameters for
> each one of the vcpus (or for a group of vcpus), depending on what tasks
> inside the guest they are pinning where.

Right! Especially when you want to have zero-schedule latency to
respond an interrupt. You can just have full capacity VCPU pinned to a
cpu and direct the interrupt to that VCPU.

>
>> > [Design sketch]
>> > We submitted the patch along with the RTDS scheduler before last year.
>> > The thread can be found at
>> > http://lists.xen.org/archives/html/xen-devel/2014-08/msg02245.html.
>> >
>> > In order to implement the get function, which return the parameters of
>> > all VCPUs of the same domain, we can bounce an array that has all
>> > parameters of all VCPUs of the domain from hypervisor to userspace and
>> > output it. (This part didn't cause much buzz. :-) )
>> > The other choice is using a hypercall to get the information of each
>> > VCPU. If there are x VCPUs in a domain, we will issue x hypercalls to
>> > get all information of this domain. This choice causes more overhead
>> > since the number of hypercalls is linear to the number of VCPUs in a
>> > domain. The first choice only needs one hypercall for a domain to get
>> > all needed information.
>> >
> ISTR that at some point we agreed on the fact that, even if getting and
> setting vcpu params won't happen in hot paths (neither from the tools
> nor from the hypervisor point of view), batching hypercall is always
> good, an we should always batch as far as we can.

Yes.

>
> That of course does not mean that we the API, at all levels, need to
> always deal with all vcpus. As you know very well, there is the hcall
> interface, where we said we want to batch; then there is the libxc
> interface, which can well diverge, but it's usually better map quite
> closely to the hcall one; and then there is libxl, on top of libxl, and
> things can be significantly different here.
>
> So, I think your design proposal/code should explicitly state what the
> plan is at each level.

I agree.
In my mind, my plan is to have all levels batching the request, from
xl to libxl to libxc to rt_dom_cntl() in sched_rt.c.

>
>> > In order to implement the set function, which set the parameters of
>> > one specific VCPU of a domain, we have two different approaches to
>> > implement it:
>> > Choice 1): When users set the parameters of k^th VCPUs of a domain,
>> > which has x VCPUs in total, the toolstack passes "only" the modified
>> > VCPUs’ information from userspace to hypervisor via hypercall;  So
>> > this implementation does "not" need an array to bounce between
>> > userspace and hypervisor. (The implementation in lilbxc was at
>> > http://lists.xen.org/archives/html/xen-devel/2014-08/msg02248.html)
>> > Choice 2): When users set the parameters of k^th VCPUs of a domain,
>> > which has x VCPUs in total, the toolstack will create an array with
>> > length x and set the information (period, budget) of the k^th element
>> > in the array. The other elements in the array are marked as 0. After
>> > the toolstack bounce the array into the hypervisor, it will only set
>> > the parameters of the k^th VCPU.
>> >
> Get and set interface should match, at each level, so we should decide
> once and for both of them, bearing in mind all the pros and the cons of
> each solution together, not split the two decisions.

I see. Then using batch seems a better choice to me.

>
>> > My questions are:
>> > 1) Should we allow users to set the parameters of "several" VCPUs with
>> > one command:
>> > For example, should we allow users to set the parameters of VCPU 1 and
>> > 2 of domain litmus1 with one xl sched-rtds command?
>> > # xl sched-rtds -d litmus1 -v 1 -p 20000 -b 10000 -v 2 -p 30000 -b 20000
>> > IMO, we'd better not allow them to do this because 1) setting the
>> > parameters of VCPUs of a domain should not be very frequent; 2) the
>> > command will become quite long when users want to set the parameters
>> > of many VCPUs of a domain.
>> >
> I actually see some value in supporting such syntax. It's not something
> critical, though, I think. For sure it doesn't have much to do with the
> decision on how the interface should look like at the various level.
>
> This means you can just go ahead and send code implementing only one '-v
> xxx -p yy y -b zzz', and we can extend it later, adding the support for
> the list.
>
>> > 2) Which design choice should we choose to implement the set function,
>> > Choice 1 or Choice 2? or do you have better suggestion?
>> >
> If we batch for get, I think we should batch for set. As I said, I
> personally see some value in an interface allowing setting all the
> parameters of all the vcpus at the same time. In fact, this is probably
> better, even as far as real-time theory is concerned: as I'm sure you
> know better than me, parameter changing can be tricky for the algorithm,
> so it's probably better to encourage the user to change the params all
> together, synchronously, rather than having to deal with a stream of
> adjustments scattered in time.
>
> So, my opinion, choice 2).

OK. Agree! :-P
I think this is also the reason why libxl and xl should use batch as
well, for consistence?

>
> If I can add something, the big challenge here is how to make the
> interfaces good and homogeneous enough with what we already have in
> place for other schedulers. I really suggest you to address this too in
> this design conversation and/or in RFC/code.

Actually, there is a basic question:
Should we back support the existing xl tool for RTDS? Right now, we
assume all VCPU of the same domain have the same parameter and the
toolstack (xl, libxl, libxc and rt_dom_cntl()) are implemented based
on such assumption.
If we have implemented the vcpu-level parameter control, it will be a
superset of the current 'xl sched-rtds'. Back-supporting the current
xl tool for RTDS scheduler (only) will just make the logic a bit more
complex.
I'm not sure if back-supporting for the experimental feature is a must?
If it's not a must, I tend to not back-support it for simplicity.
(since less code ==> less possible bugs :-) )
What do you think?

Thank you very much!

Best regards,

Meng

-- 


-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [RFC] [Design] Better XL support of RTDS scheduler for Xen 4.6

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 4504 bytes --]

On Mon, 2015-02-23 at 23:15 -0500, Meng Xu wrote:
> 2015-02-23 12:27 GMT-05:00 Dario Faggioli <dario.faggioli@citrix.com>:

> > That of course does not mean that we the API, at all levels, need to
> > always deal with all vcpus. As you know very well, there is the hcall
> > interface, where we said we want to batch; then there is the libxc
> > interface, which can well diverge, but it's usually better map quite
> > closely to the hcall one; and then there is libxl, on top of libxl, and
> > things can be significantly different here.
> >
> > So, I think your design proposal/code should explicitly state what the
> > plan is at each level.
> 
> I agree.
> In my mind, my plan is to have all levels batching the request, from
> xl to libxl to libxc to rt_dom_cntl() in sched_rt.c.
> 
xc and Xen, I agree. libxl, I'm not sure (a few more words on this
below), xl, it may make sense, but depending how you support specifying
the budget for multiple vcpus.

> > If we batch for get, I think we should batch for set. As I said, I
> > personally see some value in an interface allowing setting all the
> > parameters of all the vcpus at the same time. In fact, this is probably
> > better, even as far as real-time theory is concerned: as I'm sure you
> > know better than me, parameter changing can be tricky for the algorithm,
> > so it's probably better to encourage the user to change the params all
> > together, synchronously, rather than having to deal with a stream of
> > adjustments scattered in time.
> >
> > So, my opinion, choice 2).
> 
> OK. Agree! :-P
> I think this is also the reason why libxl and xl should use batch as
> well, for consistence?
> 
Not really. Consistency is important, but now between completely
different and independent things. Hypercalls, libxc calls, the libxl API
and xl commands have very different users, and the interface should be,
at each layer, the best one for the layer's users.

> > If I can add something, the big challenge here is how to make the
> > interfaces good and homogeneous enough with what we already have in
> > place for other schedulers. I really suggest you to address this too in
> > this design conversation and/or in RFC/code.
> 
> Actually, there is a basic question:
> Should we back support the existing xl tool for RTDS? Right now, we
> assume all VCPU of the same domain have the same parameter and the
> toolstack (xl, libxl, libxc and rt_dom_cntl()) are implemented based
> on such assumption.
>
Xen (so the hypercall), libxc and xl commands are not stable interfaces.
Add to this the fact that this feature is marked as experimental. So, in
in general, I would say that there's no need for ensuring backward
compatibility for these components. That being said, wrt to Xen and
libxc, whether you want to add a new hypercall and a (well, two) new xc
call, or you want to modify the existing ones, it's up to you and to the
hypervisor and tools maintainers' opinion.

I don't think it would be terrible to leave an interface to set the
parameters of all vcpus of the domain alone, as we have for all the
other schedulers, and add hcalls and xc calls to set per-vcpu values.
Actually, if we design this well, it could be useful in future for other
schedulers, or if existent schedulers want to support different per-vcpu
parameters.

OTOH, libxl API is stable, so we must not change the look of the
behavior of calls introduced in a release. As said above, I think it is
actually a good thing to continue having a mechanism for setting the
parameters of all the vcpus of the domain at the same time, so I will
introduce per-vcpu parametes support as an entirely new API.

> If we have implemented the vcpu-level parameter control, it will be a
> superset of the current 'xl sched-rtds'. Back-supporting the current
> xl tool for RTDS scheduler (only) will just make the logic a bit more
> complex.
> I'm not sure if back-supporting for the experimental feature is a must?
>
In xl, it's again not "a must". However, I would not kill the current
logic. In fact, how about the following:

# xl sched-rtds -d 4 -p 10000 -b 5000
 |
 --->  sets all the vcpus of domain 4 to 5000/10000

# xl sched-rtds -d 3 -v 2 -p 5000 -b 2000
 |
 ---> set vcpu 2 of domain 4 to 2000/5000

# xl sched-rtds -d 2 -v 0 -p 10000 -b 1000 -v 1 -p 5000 -b 2500
 |
 ---> set vcpu 0 of domain 2 to 1000/10000 and vcpu 1 of
      domain 2 to 2500/5000

Regards,
Dario

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [RFC] [Design] Better XL support of RTDS scheduler for Xen 4.6

[Ian Campbell]

On Mon, 2015-02-23 at 22:58 -0500, Meng Xu wrote:
> I'm not sure if other schedulers need such functionality right now,
> because the credit2 scheduler account for the credit based on each
> domain instead of each VCPU. But if the scheduler will consider the
> vcpu-level credit/budget accounting, this could be helpful.

We should take it as given that some other scheduler will want
vcpu-level parameters in the future and decide now how we want that
interface to look across multiple schedulers now, with the big question
I suppose being a large union struct vs individual structs (and perhaps
a KeyedUnion).

At the domain level param level we have libxl_domain_sched_params in
libxl_domain_build_info and via libxl_domain_sched_params_set/get.

We also have libxl_sched_credit_params and
libxl_sched_credit_params_get/set for the credit scheduler, but not for
other scheduler types.

I don't recall how we ended up with both, are the credit ones
deprecated/legacy and the combined one the One True Way or the other way
around?

In any case, we should decide what we want for both per-domain and
per-vcpu parameters, with one eye on the libxl API guarantees, and try
and have them at least be somewhat consistent.

Ian.

Re: [RFC] [Design] Better XL support of RTDS scheduler for Xen 4.6

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 2700 bytes --]

On Tue, 2015-02-24 at 10:37 +0000, Ian Campbell wrote:
> On Mon, 2015-02-23 at 22:58 -0500, Meng Xu wrote:
> > I'm not sure if other schedulers need such functionality right now,
> > because the credit2 scheduler account for the credit based on each
> > domain instead of each VCPU. But if the scheduler will consider the
> > vcpu-level credit/budget accounting, this could be helpful.
> 
> We should take it as given that some other scheduler will want
> vcpu-level parameters in the future and decide now how we want that
> interface to look across multiple schedulers now,
>
+1

> with the big question
> I suppose being a large union struct vs individual structs (and perhaps
> a KeyedUnion).
> 
Indeed.

> At the domain level param level we have libxl_domain_sched_params in
> libxl_domain_build_info and via libxl_domain_sched_params_set/get.
> 
> We also have libxl_sched_credit_params and
> libxl_sched_credit_params_get/set for the credit scheduler, but not for
> other scheduler types.
> 
> I don't recall how we ended up with both, are the credit ones
> deprecated/legacy and the combined one the One True Way or the other way
> around?
> 
sched_credit_params are system level parameters, i.e., they apply to an
instance of the credit scheduler as a whole (so, to all the host or to a
cpupool).

Other schedulers does not have similar knobs...yet. When/I case they
will, I expect the tweaks to be rather specific, and hence different
from the one offered by Credit (with, maybe, the exception of Credit2),
so it would not be too bad to have sched_rtds_params, or
sched_credit2_params as new and independent structs. In any case, this
is completely orthogonal to this discussion, IMO.

> In any case, we should decide what we want for both per-domain and
> per-vcpu parameters, with one eye on the libxl API guarantees, and try
> and have them at least be somewhat consistent.
> 
Exactly. My opinion would be to leave the per-domain interface alone as
it is (we probably don't have much alternatives to this, BTW, due to API
stability! :-D ) and introduce a new (set of) API(s) for per-vcpu level
parameters.

The former will still be preferred/used by the schedulers that does not
support different per-vcpu params, or by all the schedulers, if they
want to set all the params for all the vcpus to the same values (e.g.,
default values at domain creation time, if nothing is specified in the
config file).

The latter would be, of course, used by the schedulers that support
different per-vcpu parameters and, if someone tries to use it with one
of the others, we can return the usual whatever_NOTSUP_whatever.

Regards,
Dario

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [RFC] [Design] Better XL support of RTDS scheduler for Xen 4.6

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 4688 bytes --]

On Mon, 2015-02-23 at 22:58 -0500, Meng Xu wrote:
> 2015-02-23 10:57 GMT-05:00 Wei Liu <wei.liu2@citrix.com>:

> >> In order to implement the set function, which set the parameters of
> >> one specific VCPU of a domain, we have two different approaches to
> >> implement it:
> >> Choice 1): When users set the parameters of k^th VCPUs of a domain,
> >> which has x VCPUs in total, the toolstack passes "only" the modified
> >> VCPUs’ information from userspace to hypervisor via hypercall;  So
> >
> > A new hypercall or existing hypercall?
> 
> We only need to change the current hypercall
> (XEN_DOMCTL_SCHEDOP_setinfo) implementation:
> 
IMHO, it can be either way (add a new one or change the existing one). 

> >> Choice 2): When users set the parameters of k^th VCPUs of a domain,
> >> which has x VCPUs in total, the toolstack will create an array with
> >> length x and set the information (period, budget) of the k^th element
> >> in the array. The other elements in the array are marked as 0. After
> >> the toolstack bounce the array into the hypervisor, it will only set
> >> the parameters of the k^th VCPU.
> >>
> >
> > One thing to consider is when you have hundreds of cpus the overhead
> > might be too big? What's the amount of data you need to pass down to
> > hypervisor?
> 
> Suppose a domain have k VCPUs, each VCPU has info: period (uint32_t),
> budget(uint32_t), and index(uint32_t); So each hypercall will pass k *
> (3*8Bytes) to the hypervisor.
> 
Maybe I'm missing something but, if we pass the array, why do we need
the index? Oh, and in case we need the index, it can well be smaller
than 32 bits.

For Wei, we're talking about _v_cpus here (in your reply you're just
saying 'cpus', so I'm not sure whether you were referring to _v_ or _p_
cpus). If we have a guest with k > 100 vcpus, it's either:
 - pass a k ( > 100) element array down to Xen
 - issue k ( > 100) hypercalls back to back

OTOH, if we have such a large guest, and we only want to change the
params for one vcpu and we go for batching, we'd always have to pass the
big array, with only one meaningful value.

Allow me to say that the more common use case for this scheduler would
not include guests that big. Also, I expect most of the users wanting to
set the parameters of the various vcpus all together, and, if not once
and for all, at least quite rarely, rather than tweaking the values
every now and then.

Then, of course, users will do the weirdest things, so we really should
consider the overhead, but I thought it would be worth to at least
mention this.

> >> 2) Which design choice should we choose to implement the set function,
> >> Choice 1 or Choice 2? or do you have better suggestion?
> >>
> >
> > I have some general questions. Here in this email you only mention xl. I
> > think in fact you mean xl + libxl? We surely want to enable other
> > toolstacks to control specific parameters of schedulers as well.
> 
> Yes. I need to change xl + libxl + libxc + rt_dom_cntl() in hypervisor. :-)
> 
> I'm not sure if other schedulers need such functionality right now,
>
They don't, but they may in future... Or new schedulers we will be
introducing in 5 years time may need it.

> because the credit2 scheduler account for the credit based on each
> domain instead of each VCPU. 
>
Mmm... no, that's not the reason why they don't need this (but let's not
get into this, or we'll end up off topic :-D ).

> But if the scheduler will consider the
> vcpu-level credit/budget accounting, this could be helpful.
> 
Exactly. And while introducing new APIs, we really should think to that.

> > And I presume in the future other schedulers might also want to have
> > some per-vcpu tuning interface. Is it possible to abstract out common
> > interface in hypervisor and libxl to do that? Even if we can't sensible
> > come up with a hypervisor interface to do that, we might still want to
> > do that inside libxl. I.e. to provide some kind of common infrastructure
> > that individual scheduler can hook in.
> 
> Good point! This should be possible. I think maybe we can do this step
> by step: First have a concrete implementation for the RTDS scheduler,
> and then we can extend it to a more general interface.
>
Well, we of course can do things step by step, but at least the API
needs to be consistent and generic enough from day 0... but that should
not be too difficult, IMO. The big question (at all levels!) is still
whether we want a potentially large array of structs of single structs
leading to a potentially long stream of API-calls and hypercalls.

Regards,
Dario

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [RFC] [Design] Better XL support of RTDS scheduler for Xen 4.6

[Ian Campbell]

On Tue, 2015-02-24 at 10:52 +0000, Dario Faggioli wrote:
> On Tue, 2015-02-24 at 10:37 +0000, Ian Campbell wrote:
> > On Mon, 2015-02-23 at 22:58 -0500, Meng Xu wrote:
> > > I'm not sure if other schedulers need such functionality right now,
> > > because the credit2 scheduler account for the credit based on each
> > > domain instead of each VCPU. But if the scheduler will consider the
> > > vcpu-level credit/budget accounting, this could be helpful.
> > 
> > We should take it as given that some other scheduler will want
> > vcpu-level parameters in the future and decide now how we want that
> > interface to look across multiple schedulers now,
> >
> +1
> 
> > with the big question
> > I suppose being a large union struct vs individual structs (and perhaps
> > a KeyedUnion).
> > 
> Indeed.
> 
> > At the domain level param level we have libxl_domain_sched_params in
> > libxl_domain_build_info and via libxl_domain_sched_params_set/get.
> > 
> > We also have libxl_sched_credit_params and
> > libxl_sched_credit_params_get/set for the credit scheduler, but not for
> > other scheduler types.
> > 
> > I don't recall how we ended up with both, are the credit ones
> > deprecated/legacy and the combined one the One True Way or the other way
> > around?
> > 
> sched_credit_params are system level parameters, i.e., they apply to an
> instance of the credit scheduler as a whole (so, to all the host or to a
> cpupool).

Ah, ok, then yes these can be ignored for the purposes of this
conversation I think.

This means that the current interface is the one-big-struct type.

> > In any case, we should decide what we want for both per-domain and
> > per-vcpu parameters, with one eye on the libxl API guarantees, and try
> > and have them at least be somewhat consistent.
> > 
> Exactly. My opinion would be to leave the per-domain interface alone as
> it is (we probably don't have much alternatives to this, BTW, due to API
> stability! :-D )

We do have the option of introducing a new/better interface so long as
the old one also continues working.

>  and introduce a new (set of) API(s) for per-vcpu level
> parameters.

> The former will still be preferred/used by the schedulers that does not
> support different per-vcpu params, or by all the schedulers, if they
> want to set all the params for all the vcpus to the same values (e.g.,
> default values at domain creation time, if nothing is specified in the
> config file).
> 
> The latter would be, of course, used by the schedulers that support
> different per-vcpu parameters and,

You seem to be implying ("still be preferred") that there will be some
sort of either/or choice here between the per-domain and per-vcpu
interfaces.

Surely a scheduler can have both domain-wide and per-vcpu parameters and
a given scheduler might implement either or both as it needs.

>  if someone tries to use it with one
> of the others, we can return the usual whatever_NOTSUP_whatever.

Obviously.

Ian.

Re: [RFC] [Design] Better XL support of RTDS scheduler for Xen 4.6

[Ian Campbell]

On Tue, 2015-02-24 at 10:58 +0000, Dario Faggioli wrote:
> OTOH, if we have such a large guest, and we only want to change the
> params for one vcpu and we go for batching, we'd always have to pass the
> big array, with only one meaningful value.

Such an interface would then naturally want to include first and nr
params allowing you to pass in only an array slice, i.e. array[0]
applies to cpu[first].

Ian.

Re: [RFC] [Design] Better XL support of RTDS scheduler for Xen 4.6

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 2497 bytes --]

On Tue, 2015-02-24 at 11:06 +0000, Ian Campbell wrote:
> On Tue, 2015-02-24 at 10:52 +0000, Dario Faggioli wrote:

> > The former will still be preferred/used by the schedulers that does not
> > support different per-vcpu params, or by all the schedulers, if they
> > want to set all the params for all the vcpus to the same values (e.g.,
> > default values at domain creation time, if nothing is specified in the
> > config file).
> > 
> > The latter would be, of course, used by the schedulers that support
> > different per-vcpu parameters and,
> 
> You seem to be implying ("still be preferred") that there will be some
> sort of either/or choice here between the per-domain and per-vcpu
> interfaces.
> 
I was implying some kind of either/or logic, yes, although not in a
static and mutually exclusive way.

> Surely a scheduler can have both domain-wide and per-vcpu parameters and
> a given scheduler might implement either or both as it needs.
>
Oh, I see. well, of course, if there are parameters that are clearly
domain-wide and others that are clearly per-vcpu, they should live in
their respective structs, and the scheduler should implement things as
per its needs.

The problem is whether or not we allow overlap, i.e., whether the same
parameters can live in both interfaces, and what is the semantic of
that.

Let's take RTDS as an example. budget and period make sense at the vcpu
level so, as soon as we introduce the API for per-vcpu params, they
should live there. But then what about the budget and period field we
have in libxl_domain_sched_params? What's the meaning of them?

Semantically speaking, they just should be killed. OTOH, what I was
suggesting was this: if one calls libxl_domain_sched_params_set(), which
takes a libxl_domain_sched_params, the budget and priod there will be
interpreted as "for the whole domain", and applied to all the domain's
vcpus. It's in this sense that I see it an either/or:
 - at domain creation time, if the user does not say anything we'll use
   the domain-wide API for setting a default budget and period for all
   the vcpus.
 - if the user does say something in the config file (once this will be
   possible), or upon request, via the proper xl command, to modify the
   parameters of vcpu X, we'll use the vcpu-wide API.

Of course this applies to parameters that are part of both
structs/interfaces, which is something I was thinking to allow... were
you?

Regards,
Dario

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [RFC] [Design] Better XL support of RTDS scheduler for Xen 4.6

[Wei Liu]

On Tue, Feb 24, 2015 at 10:58:06AM +0000, Dario Faggioli wrote:
> On Mon, 2015-02-23 at 22:58 -0500, Meng Xu wrote:
> > 2015-02-23 10:57 GMT-05:00 Wei Liu <wei.liu2@citrix.com>:
> 
> > >> In order to implement the set function, which set the parameters of
> > >> one specific VCPU of a domain, we have two different approaches to
> > >> implement it:
> > >> Choice 1): When users set the parameters of k^th VCPUs of a domain,
> > >> which has x VCPUs in total, the toolstack passes "only" the modified
> > >> VCPUs’ information from userspace to hypervisor via hypercall;  So
> > >
> > > A new hypercall or existing hypercall?
> > 
> > We only need to change the current hypercall
> > (XEN_DOMCTL_SCHEDOP_setinfo) implementation:
> > 
> IMHO, it can be either way (add a new one or change the existing one). 
> 
> > >> Choice 2): When users set the parameters of k^th VCPUs of a domain,
> > >> which has x VCPUs in total, the toolstack will create an array with
> > >> length x and set the information (period, budget) of the k^th element
> > >> in the array. The other elements in the array are marked as 0. After
> > >> the toolstack bounce the array into the hypervisor, it will only set
> > >> the parameters of the k^th VCPU.
> > >>
> > >
> > > One thing to consider is when you have hundreds of cpus the overhead
> > > might be too big? What's the amount of data you need to pass down to
> > > hypervisor?
> > 
> > Suppose a domain have k VCPUs, each VCPU has info: period (uint32_t),
> > budget(uint32_t), and index(uint32_t); So each hypercall will pass k *
> > (3*8Bytes) to the hypervisor.
> > 
> Maybe I'm missing something but, if we pass the array, why do we need
> the index? Oh, and in case we need the index, it can well be smaller
> than 32 bits.
> 
> For Wei, we're talking about _v_cpus here (in your reply you're just
> saying 'cpus', so I'm not sure whether you were referring to _v_ or _p_
> cpus). If we have a guest with k > 100 vcpus, it's either:

Yes, I was talking about vcpus. AIUI the proposed hypercall is for
setting vcpu parameters. :-)

>  - pass a k ( > 100) element array down to Xen
>  - issue k ( > 100) hypercalls back to back
> 
> OTOH, if we have such a large guest, and we only want to change the
> params for one vcpu and we go for batching, we'd always have to pass the
> big array, with only one meaningful value.
> 
> Allow me to say that the more common use case for this scheduler would

The name of that hypercall XEN_DOMCTL_SCHEDOP_setinfo doesn't suggest it
is only used for this particular scheduler.

> not include guests that big. Also, I expect most of the users wanting to
> set the parameters of the various vcpus all together, and, if not once
> and for all, at least quite rarely, rather than tweaking the values
> every now and then.
> 

Right. This makes sense. Listing some common use cases can help us
establish expectation of a certain interface, so ...

> Then, of course, users will do the weirdest things, so we really should

when users do weird things we can say "it's not supposed to work like
that, your expectation is wrong, blah blah blah".

> consider the overhead, but I thought it would be worth to at least
> mention this.
> 

That would be good.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [RFC] [Design] Better XL support of RTDS scheduler for Xen 4.6

[Ian Campbell]

On Tue, 2015-02-24 at 11:30 +0000, Dario Faggioli wrote:
> The problem is whether or not we allow overlap, i.e., whether the same
> parameters can live in both interfaces, and what is the semantic of
> that.

Ah, I see.

> Let's take RTDS as an example. budget and period make sense at the vcpu
> level so, as soon as we introduce the API for per-vcpu params, they
> should live there. But then what about the budget and period field we
> have in libxl_domain_sched_params? What's the meaning of them?

> Semantically speaking, they just should be killed. OTOH, what I was
> suggesting was this: if one calls libxl_domain_sched_params_set(), which
> takes a libxl_domain_sched_params, the budget and priod there will be
> interpreted as "for the whole domain", and applied to all the domain's
> vcpus. It's in this sense that I see it an either/or:
>  - at domain creation time, if the user does not say anything we'll use
>    the domain-wide API for setting a default budget and period for all
>    the vcpus.
>  - if the user does say something in the config file (once this will be
>    possible), or upon request, via the proper xl command, to modify the
>    parameters of vcpu X, we'll use the vcpu-wide API.

The simplest (and IMHO least surprising) thing would be to have the
per-vcpu ones override the per-domain ones. IOW per-domain sets a
default used unless a per-vcpu one says otherwise for a given vcpu.

So at build time you would first call the per-domain set function with
whatever parameters were provided, followed by setting any per-vcpu
options which are configured.

At run time via xl commands I guess they would be separate commands for
domains vs vcpus, and if you call the per-domain one I'd probably expect
it to clobber any existing per-vcpu settings (i.e act like the vcpu
option given --all).

> Of course this applies to parameters that are part of both
> structs/interfaces, which is something I was thinking to allow... were
> you?

Yes, I think it makes sense to have them.

Ian.

Re: [RFC] [Design] Better XL support of RTDS scheduler for Xen 4.6

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 2225 bytes --]

On Tue, 2015-02-24 at 14:16 +0000, Ian Campbell wrote:
> On Tue, 2015-02-24 at 11:30 +0000, Dario Faggioli wrote:

> > Semantically speaking, they just should be killed. OTOH, what I was
> > suggesting was this: if one calls libxl_domain_sched_params_set(), which
> > takes a libxl_domain_sched_params, the budget and priod there will be
> > interpreted as "for the whole domain", and applied to all the domain's
> > vcpus. It's in this sense that I see it an either/or:
> >  - at domain creation time, if the user does not say anything we'll use
> >    the domain-wide API for setting a default budget and period for all
> >    the vcpus.
> >  - if the user does say something in the config file (once this will be
> >    possible), or upon request, via the proper xl command, to modify the
> >    parameters of vcpu X, we'll use the vcpu-wide API.
> 
> The simplest (and IMHO least surprising) thing would be to have the
> per-vcpu ones override the per-domain ones. 
>
I agree.

> So at build time you would first call the per-domain set function with
> whatever parameters were provided, followed by setting any per-vcpu
> options which are configured.
> 
Yep.

> At run time via xl commands I guess they would be separate commands for
> domains vs vcpus, 
>
Yes, either there will be separate commands, or the same command will
have a way of specifying whether the parameters are to be considered
domain-wide, or just apply to one (or more) vcpus.

Still using RTDS as an example, this is what I was suggesting, earlier
in this thread
(http://www.gossamer-threads.com/lists/xen/devel/368154#368154):

    # xl sched-rtds -d 4 -p 10000 -b 5000 
     | 
     ---> sets all the vcpus of domain 4 to 5000/10000 

    # xl sched-rtds -d 3 -v 2 -p 5000 -b 2000 
     | 
     ---> set vcpu 2 of domain 4 to 2000/5000 

    # xl sched-rtds -d 2 -v 0 -p 10000 -b 1000 -v 1 -p 5000 -b 2500 
     | 
     ---> set vcpu 0 of domain 2 to 1000/10000 and vcpu 1 of 
          domain 2 to 2500/5000 

> and if you call the per-domain one I'd probably expect
> it to clobber any existing per-vcpu settings (i.e act like the vcpu
> option given --all).
> 
Exactly

Regards,
Dario

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [RFC] [Design] Better XL support of RTDS scheduler for Xen 4.6

[Meng Xu]

Hi all,

2015-02-24 11:35 GMT-05:00 Dario Faggioli <dario.faggioli@citrix.com>:
> On Tue, 2015-02-24 at 14:16 +0000, Ian Campbell wrote:
>> On Tue, 2015-02-24 at 11:30 +0000, Dario Faggioli wrote:
>
>> > Semantically speaking, they just should be killed. OTOH, what I was
>> > suggesting was this: if one calls libxl_domain_sched_params_set(), which
>> > takes a libxl_domain_sched_params, the budget and priod there will be
>> > interpreted as "for the whole domain", and applied to all the domain's
>> > vcpus. It's in this sense that I see it an either/or:
>> >  - at domain creation time, if the user does not say anything we'll use
>> >    the domain-wide API for setting a default budget and period for all
>> >    the vcpus.
>> >  - if the user does say something in the config file (once this will be
>> >    possible), or upon request, via the proper xl command, to modify the
>> >    parameters of vcpu X, we'll use the vcpu-wide API.
>>
>> The simplest (and IMHO least surprising) thing would be to have the
>> per-vcpu ones override the per-domain ones.
>>
> I agree.

Sorry, I'm not so sure I understand the term "to have the per-vcpu
ones override the per-domain ones."
My understanding is:
At domain creation time, the default option is to use the per-domain
implementation (in libxl, libxc, and xen) to set the default budget
and period for all VCPUs.
If user specify the per-vcpu parameters or wants to change the
parameter with xl command after the domain is created, they can use
the vcpu-wide api to change it.

Am I correct?

>
>> So at build time you would first call the per-domain set function with
>> whatever parameters were provided, followed by setting any per-vcpu
>> options which are configured.
>>
> Yep.
>
>> At run time via xl commands I guess they would be separate commands for
>> domains vs vcpus,
>>
> Yes, either there will be separate commands, or the same command will
> have a way of specifying whether the parameters are to be considered
> domain-wide, or just apply to one (or more) vcpus.

So let me try to summarize: :-)

We are going to implement both per-domain and per-vcpu get/set method.

In hypervisor and libxc: we will use
XEN_DOMCTL_SCHEDOP_getinfo/setinfo as the hypercall for per-domain
operation(i.e., get/set method); We will introduce new hypercall, say
XEN_VCPUCTL_SCHEDOP_getinfo/setinfo, for per-vcpu operation.

In xl: we will support both per-domain and per-vcpu operation.
The semantics of xl  per-domain operation will be: xl sched-rtds -d 4
-p 10000 -b 5000 --all, which set all vcpus of domain 4 to have period
10ms and budget 5ms;
The semantics of xl per-vcpu operation will be what Dario describes as follows:

>     # xl sched-rtds -d 4 -p 10000 -b 5000
>      |
>      ---> sets all the vcpus of domain 4 to 5000/10000
>
>     # xl sched-rtds -d 3 -v 2 -p 5000 -b 2000
>      |
>      ---> set vcpu 2 of domain 4 to 2000/5000
>
>     # xl sched-rtds -d 2 -v 0 -p 10000 -b 1000 -v 1 -p 5000 -b 2500
>      |
>      ---> set vcpu 0 of domain 2 to 1000/10000 and vcpu 1 of
>           domain 2 to 2500/5000
>

In libxl: we will support both per-domain and per-vcpu operations. The
per-domain operations have been supported in the current Xen. So we
will add another two libraries for the per-vcpu get/set operations.
(Right now, the per-vcpu get/set operation in libxl will also use the
array-based implementation.)

As to the data structure, we will use the union structure to hold the
data of the RTDS per-vcpu parameters, so that when other schedulers
want to support per-vcpu operation, they can add the specific data
structure in the union structure.

Does the above summary make sense? :-)

BTW, in the array that hold all VCPUs' parameters, we don't need the
vcpu index because it's implicitly encoded as the index of the array
element.

Thank you very much for your advice!

Best,

Meng

-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania

Re: [RFC] [Design] Better XL support of RTDS scheduler for Xen 4.6

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 7260 bytes --]

On Tue, 2015-02-24 at 21:42 -0500, Meng Xu wrote:
> Hi all,
> 
Hello,

> 2015-02-24 11:35 GMT-05:00 Dario Faggioli <dario.faggioli@citrix.com>:
> > On Tue, 2015-02-24 at 14:16 +0000, Ian Campbell wrote:
> >> On Tue, 2015-02-24 at 11:30 +0000, Dario Faggioli wrote:
> >
> >> > Semantically speaking, they just should be killed. OTOH, what I was
> >> > suggesting was this: if one calls libxl_domain_sched_params_set(), which
> >> > takes a libxl_domain_sched_params, the budget and priod there will be
> >> > interpreted as "for the whole domain", and applied to all the domain's
> >> > vcpus. It's in this sense that I see it an either/or:
> >> >  - at domain creation time, if the user does not say anything we'll use
> >> >    the domain-wide API for setting a default budget and period for all
> >> >    the vcpus.
> >> >  - if the user does say something in the config file (once this will be
> >> >    possible), or upon request, via the proper xl command, to modify the
> >> >    parameters of vcpu X, we'll use the vcpu-wide API.
> >>
> >> The simplest (and IMHO least surprising) thing would be to have the
> >> per-vcpu ones override the per-domain ones.
> >>
> > I agree.
> 
> Sorry, I'm not so sure I understand the term "to have the per-vcpu
> ones override the per-domain ones."
>
It means that if one calls libxl_domain_sched_params_set(), with a
struct libxl_domain_sched_params as argument --which contains one single
budget value and one single period value-- we will apply those budget
and period to _all_ the vcpus of the domain.

> My understanding is:
> At domain creation time, the default option is to use the per-domain
> implementation (in libxl, libxc, and xen) to set the default budget
> and period for all VCPUs.
>
This is an implementation detail, while we're discussing the look and
the semantic of the interface. However, yes, it is very likely that
we'll be calling libxl_domain_sched_params_set(), with some default
budget and period values, at early stage during domain creation, to set
default values, in case the user does not specify anything.

> If user specify the per-vcpu parameters or wants to change the
> parameter with xl command after the domain is created, they can use
> the vcpu-wide api to change it.
> 
> Am I correct?
> 
Well, they can use both, can't they? If they use the domain-wide API,
that will set the parameters for all the vcpus, as said above. It they
use the per-vcpu API, that will set the parameters for the vcpus
specified in the arguments.

Does this make sense?

> So let me try to summarize: :-)
> 
> We are going to implement both per-domain and per-vcpu get/set method.
> 
The per-domain is already there, the per-vcpu, yes, we have to implement
it.

> In hypervisor and libxc: we will use
> XEN_DOMCTL_SCHEDOP_getinfo/setinfo as the hypercall for per-domain
> operation(i.e., get/set method); We will introduce new hypercall, say
> XEN_VCPUCTL_SCHEDOP_getinfo/setinfo, for per-vcpu operation.
> 
All the above is a lot more focused on libxl API than on the Xen one,
and they not necessarily have to match. That being said, yes, my opinion
would be to do something similar to the above at the hypercall level
too, i.e., to leave XEN_DOMCTL_SCHEDOP_{get,put}info alone, and add new
hypercalls for per-vcpu parameters manipulation.

These new hypercalls must be designed in such a way that, although
they'll be used by RTDS only for now, it will be possible for other
schedulers to use them in future. This is less critical than in libxl,
as this is not a stable interface, but if it's simple enough (as I think
it is), I see few reasons why not to do so.

About the name, I'd go for something like
XEN_DOMCTL_SCHEDOP_{get,put}vcpuinfo, i.e., let's retain the
XEN_DOMCTL_SCHEDOP_ prefix, and tell what specific operation is
afterwards, in the lowercase part of the name.

> In xl: we will support both per-domain and per-vcpu operation.
> The semantics of xl  per-domain operation will be: xl sched-rtds -d 4
> -p 10000 -b 5000 --all, which set all vcpus of domain 4 to have period
> 10ms and budget 5ms;
> The semantics of xl per-vcpu operation will be what Dario describes as follows:
> 
> >     # xl sched-rtds -d 4 -p 10000 -b 5000
> >      |
> >      ---> sets all the vcpus of domain 4 to 5000/10000
> >
> >     # xl sched-rtds -d 3 -v 2 -p 5000 -b 2000
> >      |
> >      ---> set vcpu 2 of domain 4 to 2000/5000
> >
> >     # xl sched-rtds -d 2 -v 0 -p 10000 -b 1000 -v 3 -p 5000 -b 2500
> >      |
> >      ---> set vcpu 0 of domain 2 to 1000/10000 and vcpu 3 of
> >           domain 2 to 2500/5000
> >
> 
Yes. The third variant can come as a future, incremental addition, it
does not have to be there in the first submission.

> In libxl: we will support both per-domain and per-vcpu operations. The
> per-domain operations have been supported in the current Xen. So we
> will add another two libraries for the per-vcpu get/set operations.
> (Right now, the per-vcpu get/set operation in libxl will also use the
> array-based implementation.)
> 
Yes. Only thing I'm not sure is, what do you mean with 'right now'?

> As to the data structure, we will use the union structure to hold the
> data of the RTDS per-vcpu parameters, so that when other schedulers
> want to support per-vcpu operation, they can add the specific data
> structure in the union structure.
> 
Exactly.

> BTW, in the array that hold all VCPUs' parameters, we don't need the
> vcpu index because it's implicitly encoded as the index of the array
> element.
> 
Well, actually, thinking more about this, taking Wei's and Ian's
comments into account, I think the index could be useful.

In fact, the best option is probably to pass around arrays as big as the
number of vcpus we want to modify the parameters of. That means we need
a 'number_of_elements' field in the struct, next to the array itself
(this is easily done in libxl_types.idl, and everywhere else), and a
vcpu index, within each array element, to figure out what vcpu that
element is about.

So, in the above example:

    # xl sched-rtds -d 4 -p 10000 -b 5000
      |
      ---> this will use the domain-wide API, so no array involved

    # xl sched-rtds -d 3 -v 2 -p 5000 -b 2000
      |
      ---> this will use the per-vcpu API. The array will only have one
           element, relative to vcpu 2:
            num_vcpu_params = 1
            vcpu_params[0].index = 2
            vcpu_params[0].period = 10000
            vcpu_params[0].budget = 5000

    # xl sched-rtds -d 2 -v 0 -p 10000 -b 1000 -v 3 -p 5000 -b 2500
      |
      ---> this will use the per-vcpu API. The array will only have two
           elements, relative to vcpu 0 and 1:
            num_vcpu_params = 2
            vcpu_params[0].index = 0
            vcpu_params[0].period = 10000
            vcpu_params[0].budget = 5000
            vcpu_params[1].index = 3
            vcpu_params[1].period = 10000
            vcpu_params[1].budget = 5000

This strategy still uses batching, but at the same time limits the
amount of data passed around to the minimum.

Would this be fine?

Regards,
Dario

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [RFC] [Design] Better XL support of RTDS scheduler for Xen 4.6

[Meng Xu]

Hi Dario,

In summary, I agree with you and will apply these comments discussed
in this thread to work out a RFC patch for it. Below is just some
reply to your question. :-)

2015-02-25 3:44 GMT-05:00 Dario Faggioli <dario.faggioli@citrix.com>:
> On Tue, 2015-02-24 at 21:42 -0500, Meng Xu wrote:
>> Hi all,
>>
> Hello,
>
>> 2015-02-24 11:35 GMT-05:00 Dario Faggioli <dario.faggioli@citrix.com>:
>> > On Tue, 2015-02-24 at 14:16 +0000, Ian Campbell wrote:
>> >> On Tue, 2015-02-24 at 11:30 +0000, Dario Faggioli wrote:
>> >
>> >> > Semantically speaking, they just should be killed. OTOH, what I was
>> >> > suggesting was this: if one calls libxl_domain_sched_params_set(), which
>> >> > takes a libxl_domain_sched_params, the budget and priod there will be
>> >> > interpreted as "for the whole domain", and applied to all the domain's
>> >> > vcpus. It's in this sense that I see it an either/or:
>> >> >  - at domain creation time, if the user does not say anything we'll use
>> >> >    the domain-wide API for setting a default budget and period for all
>> >> >    the vcpus.
>> >> >  - if the user does say something in the config file (once this will be
>> >> >    possible), or upon request, via the proper xl command, to modify the
>> >> >    parameters of vcpu X, we'll use the vcpu-wide API.
>> >>
>> >> The simplest (and IMHO least surprising) thing would be to have the
>> >> per-vcpu ones override the per-domain ones.
>> >>
>> > I agree.
>>
>> Sorry, I'm not so sure I understand the term "to have the per-vcpu
>> ones override the per-domain ones."
>>
> It means that if one calls libxl_domain_sched_params_set(), with a
> struct libxl_domain_sched_params as argument --which contains one single
> budget value and one single period value-- we will apply those budget
> and period to _all_ the vcpus of the domain.
>
>> My understanding is:
>> At domain creation time, the default option is to use the per-domain
>> implementation (in libxl, libxc, and xen) to set the default budget
>> and period for all VCPUs.
>>
> This is an implementation detail, while we're discussing the look and
> the semantic of the interface. However, yes, it is very likely that
> we'll be calling libxl_domain_sched_params_set(), with some default
> budget and period values, at early stage during domain creation, to set
> default values, in case the user does not specify anything.
>
>> If user specify the per-vcpu parameters or wants to change the
>> parameter with xl command after the domain is created, they can use
>> the vcpu-wide api to change it.
>>
>> Am I correct?
>>
> Well, they can use both, can't they? If they use the domain-wide API,
> that will set the parameters for all the vcpus, as said above. It they
> use the per-vcpu API, that will set the parameters for the vcpus
> specified in the arguments.
>
> Does this make sense?
>
>> So let me try to summarize: :-)
>>
>> We are going to implement both per-domain and per-vcpu get/set method.
>>
> The per-domain is already there, the per-vcpu, yes, we have to implement
> it.
>
>> In hypervisor and libxc: we will use
>> XEN_DOMCTL_SCHEDOP_getinfo/setinfo as the hypercall for per-domain
>> operation(i.e., get/set method); We will introduce new hypercall, say
>> XEN_VCPUCTL_SCHEDOP_getinfo/setinfo, for per-vcpu operation.
>>
> All the above is a lot more focused on libxl API than on the Xen one,
> and they not necessarily have to match. That being said, yes, my opinion
> would be to do something similar to the above at the hypercall level
> too, i.e., to leave XEN_DOMCTL_SCHEDOP_{get,put}info alone, and add new
> hypercalls for per-vcpu parameters manipulation.
>
> These new hypercalls must be designed in such a way that, although
> they'll be used by RTDS only for now, it will be possible for other
> schedulers to use them in future. This is less critical than in libxl,
> as this is not a stable interface, but if it's simple enough (as I think
> it is), I see few reasons why not to do so.
>
> About the name, I'd go for something like
> XEN_DOMCTL_SCHEDOP_{get,put}vcpuinfo, i.e., let's retain the
> XEN_DOMCTL_SCHEDOP_ prefix, and tell what specific operation is
> afterwards, in the lowercase part of the name.
>
>> In xl: we will support both per-domain and per-vcpu operation.
>> The semantics of xl  per-domain operation will be: xl sched-rtds -d 4
>> -p 10000 -b 5000 --all, which set all vcpus of domain 4 to have period
>> 10ms and budget 5ms;
>> The semantics of xl per-vcpu operation will be what Dario describes as follows:
>>
>> >     # xl sched-rtds -d 4 -p 10000 -b 5000
>> >      |
>> >      ---> sets all the vcpus of domain 4 to 5000/10000
>> >
>> >     # xl sched-rtds -d 3 -v 2 -p 5000 -b 2000
>> >      |
>> >      ---> set vcpu 2 of domain 4 to 2000/5000
>> >
>> >     # xl sched-rtds -d 2 -v 0 -p 10000 -b 1000 -v 3 -p 5000 -b 2500
>> >      |
>> >      ---> set vcpu 0 of domain 2 to 1000/10000 and vcpu 3 of
>> >           domain 2 to 2500/5000
>> >
>>
> Yes. The third variant can come as a future, incremental addition, it
> does not have to be there in the first submission.
>
>> In libxl: we will support both per-domain and per-vcpu operations. The
>> per-domain operations have been supported in the current Xen. So we
>> will add another two libraries for the per-vcpu get/set operations.
>> (Right now, the per-vcpu get/set operation in libxl will also use the
>> array-based implementation.)
>>
> Yes. Only thing I'm not sure is, what do you mean with 'right now'?

Ah, 'Right now' means the current design is blabla... ;-)

>
>> As to the data structure, we will use the union structure to hold the
>> data of the RTDS per-vcpu parameters, so that when other schedulers
>> want to support per-vcpu operation, they can add the specific data
>> structure in the union structure.
>>
> Exactly.
>
>> BTW, in the array that hold all VCPUs' parameters, we don't need the
>> vcpu index because it's implicitly encoded as the index of the array
>> element.
>>
> Well, actually, thinking more about this, taking Wei's and Ian's
> comments into account, I think the index could be useful.
>
> In fact, the best option is probably to pass around arrays as big as the
> number of vcpus we want to modify the parameters of. That means we need
> a 'number_of_elements' field in the struct, next to the array itself
> (this is easily done in libxl_types.idl, and everywhere else), and a
> vcpu index, within each array element, to figure out what vcpu that
> element is about.
>
> So, in the above example:
>
>     # xl sched-rtds -d 4 -p 10000 -b 5000
>       |
>       ---> this will use the domain-wide API, so no array involved
>
>     # xl sched-rtds -d 3 -v 2 -p 5000 -b 2000
>       |
>       ---> this will use the per-vcpu API. The array will only have one
>            element, relative to vcpu 2:
>             num_vcpu_params = 1
>             vcpu_params[0].index = 2
>             vcpu_params[0].period = 10000
>             vcpu_params[0].budget = 5000
>
>     # xl sched-rtds -d 2 -v 0 -p 10000 -b 1000 -v 3 -p 5000 -b 2500
>       |
>       ---> this will use the per-vcpu API. The array will only have two
>            elements, relative to vcpu 0 and 1:
>             num_vcpu_params = 2
>             vcpu_params[0].index = 0
>             vcpu_params[0].period = 10000
>             vcpu_params[0].budget = 5000
>             vcpu_params[1].index = 3
>             vcpu_params[1].period = 10000
>             vcpu_params[1].budget = 5000
>
> This strategy still uses batching, but at the same time limits the
> amount of data passed around to the minimum.
>
> Would this be fine?

Sure! This will be perfectly fine! I did implement this before and
submitted a version before.
I will incorporate these comments here and sent out a RFC patch this
week or next week so that we can make it concrete. :-)

Thank you very much!

best,

Meng



-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania