Introduce rtds real-time scheduler for Xen

Introduce rtds real-time scheduler for Xen

[Meng Xu]

This serie of patches adds rtds real-time scheduler to Xen.

In summary, It supports:
1) Preemptive Global Earliest Deadline First scheduling policy by using a global RunQ for the scheduler;
2) Assign/display VCPUs' parameters of each domain (All VCPUs of each domain have the same period and budget);
3) Supports CPU Pool
Note: 1)  Although the toolstack only allows users to set the paramters of all VCPUs of the same domain to the same number, the scheduler supports to schedule VCPUs with different parameters of the same domain. In Xen 4.6, we plan to support assign/display each VCPU's parameters of each domain.
      2)  Parameters of a domain at tool stack is in microsecond, instead of millisecond.

Compared with the PATCH v2, this set of patch has the following modifications:
    a) Split one global RunQ to two queues: runq and depletedq, instead of using flag_vcpu;
    b) Remove prv->cpus and use cpupool_scheduler_cpumask to get the valid cpus for the scheduler; (one scheduler per cpupool)
    c) Remove unnecessary printk in sched_rt.c;
    d) Change the scheduler's name from rt to rtds;
    e) Miscellous modification, such as removing trailing spaces.

[PATCH v3 1/4] xen: add real time scheduler rtds
[PATCH v3 2/4] libxc: add rtds scheduler
[PATCH v3 3/4] libxl: add rtds scheduler
[PATCH v3 4/4] xl: introduce rtds scheduler
-----------------------------------------------------------------------------------------------------------------------------
TODO after Xen 4.5:
    a) Burn budget in finer granularity instead of 1ms; [medium]
    b) Use separate timer per vcpu for each vcpu's budget replenishment, instead of scanning the full runqueue every now and then [medium]
    c) Handle time stolen from domU by hypervisor. When it runs on a machine with many sockets and lots of cores, the spin-lock for global RunQ used in rtds scheduler could eat up time from domU, which could make domU have less budget than it requires. [not sure about difficulty right now] (Thank Konrad Rzeszutek to point this out in the XenSummit. :-))
    d) Toolstack supports assiging/display each VCPU's parameters of each domain.

-----------------------------------------------------------------------------------------------------------------------------
The design of this rtds scheduler is as follows:

This scheduler follows the Preemptive Global Earliest Deadline First
(EDF) theory in real-time field.
At any scheduling point, the VCPU with earlier deadline has higher
priority. The scheduler always picks the highest priority VCPU to run on a
feasible PCPU.
A PCPU is feasible if the VCPU can run on this PCPU and (the PCPU is
idle or has a lower-priority VCPU running on it.)

Each VCPU has a dedicated period and budget.
The deadline of a VCPU is at the end of each period;
A VCPU has its budget replenished at the beginning of each period;
While scheduled, a VCPU burns its budget.
The VCPU needs to finish its budget before its deadline in each period;
The VCPU discards its unused budget at the end of each period.
If a VCPU runs out of budget in a period, it has to wait until next period.

Each VCPU is implemented as a deferable server.
When a VCPU has a task running on it, its budget is continuously burned;
When a VCPU has no task but with budget left, its budget is preserved.

Queue scheme:
A global runqueue and a global depletedq for each CPU pool.
The runqueue holds all runnable VCPUs with budget and sorted by deadline;
The depletedq holds all VCPUs without budget and unsorted.

Note: cpumask and cpupool is supported.

If you are intersted in the details of the design and evaluation of this rt scheduler, please refer to our paper "Real-Time Multi-Core Virtual Machine Scheduling in Xen" (http://www.cis.upenn.edu/~mengxu/emsoft14/emsoft14.pdf), which will be published in EMSOFT14. This paper has the following details:
    a) Desgin of this scheduler;
    b) Measurement of the implementation overhead, e.g., scheduler overhead, context switch overhead, etc. 
    c) Comparison of this rt scheduler and credit scheduler in terms of the real-time performance.

If you are interested in other real-time schedulers in Xen, please refer to the RT-Xen project's website (https://sites.google.com/site/realtimexen/). It also supports Preemptive Global Rate Monotonic schedulers.
-----------------------------------------------------------------------------------------------------------------------------
One scenario to show the functionality of this rtds scheduler is as follows:
//list the domains
#xl list
Name                                        ID   Mem VCPUs  State   Time(s)
Domain-0                                     0  3344     4     r-----     146.1
vm1                                          1   512     2     r-----     155.1

//list VCPUs' parameters of each domain in cpu pools using rtds scheduler
#xl sched-rtds
Cpupool Pool-0: sched=EDF
Name                                ID    Period    Budget
Domain-0                             0     10000      4000
vm1                                  1     10000      4000

//set VCPUs' parameters of each domain to new value
xl sched-rtds -d Domain-0 -p 20000 -b 10000
//Now all vcpus of Domain-0 have period 20000us and budget 10000us.
#xl sched-rtds
Cpupool Pool-0: sched=EDF
Name                                ID    Period    Budget
Domain-0                             0     20000     10000
vm1                                  1     10000      4000

// list cpupool information
#xl cpupool-list
Name               CPUs   Sched     Active   Domain count
Pool-0               4     rtds       y          2
#xl cpupool-list -c
Name               CPU list
Pool-0             0,1,2,3

//create a cpupool test
#xl cpupool-cpu-remove Pool-0 3
#xl cpupool-cpu-remove Pool-0 2
#xl cpupool-create name=\"test\" sched=\"rtds\"
#xl cpupool-cpu-add test 3
#xl cpupool-cpu-add test 2
#xl cpupool-list
Name               CPUs   Sched     Active   Domain count
Pool-0               2     rtds       y          2
test                 2     rtds       y          0

//migrate vm1 from cpupool Pool-0 to cpupool test.
#xl cpupool-migrate vm1 test

//now vm1 is in cpupool test
# xl sched-rtds
pupool Pool-0: sched=EDF
Name                                ID    Period    Budget
Domain-0                             0     20000     10000
Cpupool test: sched=EDF
Name                                ID    Period    Budget
vm1                                  1     10000      4000

-----------------------------------------------------------------------------------------------------------------------------
Any comment, question, and concerns are more than welcome! :-)

Thank you very much!

Best,

Meng

---
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

Re: Introduce rtds real-time scheduler for Xen

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 497 bytes --]

On Sun, 2014-09-14 at 17:37 -0400, Meng Xu wrote:

> //list VCPUs' parameters of each domain in cpu pools using rtds scheduler
> #xl sched-rtds
> Cpupool Pool-0: sched=EDF
>
This needs to become 'sched=RTDS'

Regards, Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Introduce rtds real-time scheduler for Xen

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 8872 bytes --]

On dom, 2014-09-14 at 17:37 -0400, Meng Xu wrote:
> This serie of patches adds rtds real-time scheduler to Xen.
> 
I gave this series some testing, and the behavior of the scheduler is as
expected, so again, Meng and Sisu, good work.

While doing it, I've also put the series in this git repo/branch:

  git://xenbits.xen.org/people/dariof/xen.git  sched/rt/rtds-v3
  http://xenbits.xen.org/gitweb/?p=people/dariof/xen.git;a=shortlog;h=refs/heads/sched/rt/rtds-v3


There are a couple of issue, though, one minor and one serious, which
I'd like you to fix, if possible, before freeze date. More info below.

> //list VCPUs' parameters of each domain in cpu pools using rtds scheduler
> #xl sched-rtds
> Cpupool Pool-0: sched=EDF
> Name                                ID    Period    Budget
> Domain-0                             0     10000      4000
> vm1                                  1     10000      4000
> 
So, when I boot Xen with sched=rtds, issueing this command (`xl
sched-rtds') produces a lot of printk on the serial console, basically
outputting the dump of the scheduler information.

I guess there is one call to rt_sched_dump() (or whatever that was) left
somewhere. Could you please check?

This is not a serious issue, but since you're resending anyway...

> //create a cpupool test
> #xl cpupool-cpu-remove Pool-0 3
> #xl cpupool-cpu-remove Pool-0 2
> #xl cpupool-create name=\"test\" sched=\"rtds\"
> #xl cpupool-cpu-add test 3
> #xl cpupool-cpu-add test 2
> #xl cpupool-list
> Name               CPUs   Sched     Active   Domain count
> Pool-0               2     rtds       y          2
> test                 2     rtds       y          0
> 
This works for me too.

Booting with sched=credit, creating an rtds cpupool and migruting
domains there also works here too.

However, booting with sched=rtds, and issuing the following commands:
# xl cpupool-cpu-remove Pool-0 20
# xl cpupool-cpu-remove Pool-0 21
# xl cpupool-create /etc/xen/be-cpupool

Where /etc/xen/be-cpupool looks like this:
 name = "be"
 sched = "credit"
 cpus = ["20", "21"]
 sched="credit"

Makes Xen *CRASH* with the following trace:

(XEN) ----[ Xen-4.5-unstable  x86_64  debug=y  Not tainted ]----
(XEN) ----[ Xen-4.5-unstable  x86_64  debug=y  Not tainted ]----
(XEN) CPU:    20
(XEN) CPU:    21
(XEN) RIP:    e008:[<ffff82d08012bb1e>]RIP:    e008:[<ffff82d08012bb1e>] check_lock+0x1e/0x3b check_lock+0x1e/0x3b
(XEN) RFLAGS: 0000000000010002   
(XEN) RFLAGS: 0000000000010002   CONTEXT: hypervisor
(XEN) CONTEXT: hypervisor
(XEN) rax: 0000000000000001   rbx: 0000000000000000   rcx: 0000000000000001
(XEN) rax: 0000000000000001   rbx: 0000000000000000   rcx: 0000000000000001
(XEN) rdx: 0000000000000001   rsi: ffff830917fc4c80   rdi: 0000000000000004
(XEN) rdx: 0000000000000001   rsi: ffff830917fc2c80   rdi: 0000000000000004
(XEN) rbp: ffff830917fb7e08   rsp: ffff830917fb7e08   r8:  0000001d52034e80
(XEN) rbp: ffff830917fafe08   rsp: ffff830917fafe08   r8:  0000000000000000
(XEN) r9:  ffff830828a47978   r10: 00000000deadbeef   r11: 0000000000000246
(XEN) r9:  ffff830917fe3ea8   r10: 00000000deadbeef   r11: 0000000000000246
(XEN) r12: 0000001d51efbd89   r13: ffff82d080320de0   r14: 0000000000000000
(XEN) r12: 0000001d51efbb99   r13: ffff82d080320de0   r14: 0000000000000000
(XEN) r15: 0000000000000014   cr0: 000000008005003b   cr4: 00000000000026f0
(XEN) r15: 0000000000000015   cr0: 000000008005003b   cr4: 00000000000026f0
(XEN) cr3: 00000000cf08f000   cr2: 0000000000000004
(XEN) cr3: 00000000cf08f000   cr2: 0000000000000004
(XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from rsp=ffff830917fb7e08:
(XEN)   Xen stack trace from rsp=ffff830917fafe08:
(XEN)    ffff830917fb7e20 ffff830917fafe20 ffff82d08012bbc4 ffff82d08012bbc4 ffff8300cf12d000 ffff8300cf12c000 ffff830917fb7eb0 ffff830917fafeb0
(XEN)   
(XEN)    ffff82d080128175 ffff82d080128175 ffff830917fb7e40 ffff830917fafe40 ffff82d0801879ef ffff82d0801879ef 0000001400fb7e60 0000001500fafe60
(XEN)   
(XEN)    ffff830917fc4060 ffff830917fc2060 ffff830917fb7e60 ffff830917fafe60 ffff830917fc4200 ffff830917fc2200 ffff830917fb7eb0 ffff830917fafeb0
(XEN)   
(XEN)    ffff82d08012e983 ffff82d08012e983 ffff830917fb7ef0 ffff830917fafef0 ffff82d0801aa941 ffff82d0801aa941 ffff830917fb7e90 ffff830917fafe90
(XEN)   
(XEN)    ffff82d0802f8980 ffff82d0802f8a00 ffff82d0802f7f80 ffff82d0802f7f80 ffffffffffffffff ffffffffffffffff ffff830917fb0000 ffff830917fa8000
(XEN)   
(XEN)    00000000000f4240 00000000000f4240 ffff830917fb7ee0 ffff830917fafee0 ffff82d08012b539 ffff82d08012b539 ffff830917fb0000 ffff830917fa8000
(XEN)   
(XEN)    0000001d51dfd294 0000001d51dfcb8f ffff8300cf12d000 ffff8300cf12c000 ffff83092d6a0990 ffff83092d6a0990 ffff830917fb7ef0 ffff830917fafef0
(XEN)   
(XEN)    ffff82d08012b591 ffff82d08012b591 ffff830917fb7f10 ffff830917faff10 ffff82d080160425 ffff82d080160425 ffff82d08012b591 ffff82d08012b591
(XEN)   
(XEN)    ffff8300cf12d000 ffff8300cf12c000 ffff830917fb7e10 ffff830917fafe10 0000000000000000 0000000000000000 ffff88003a0cbfd8 ffff88003a0edfd8
(XEN)   
(XEN)    ffff88003a0cbfd8 ffff88003a0edfd8 0000000000000007 0000000000000014 ffff88003a0cbec0 ffff88003a0edec0 0000000000000000 0000000000000000
(XEN)   
(XEN)    0000000000000246 0000000000000246 0000001c6462ff88 0000001c986bebc8 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)   
(XEN)    0000000000000000 0000000000000000 ffffffff810013aa ffffffff810013aa ffffffff81c31160 ffffffff81c31160 00000000deadbeef 00000000deadbeef
(XEN)   
(XEN)    00000000deadbeef 00000000deadbeef 0000010000000000 0000010000000000 ffffffff810013aa ffffffff810013aa 000000000000e033 000000000000e033
(XEN)   
(XEN)    0000000000000246 0000000000000246 ffff88003a0cbea8 ffff88003a0edea8 000000000000e02b 000000000000e02b c2c2c2c2c2c2c2c2 c2c2c2c2c2c2c2c2
(XEN)   
(XEN)    c2c2c2c2c2c2c2c2 c2c2c2c2c2c2c2c2 c2c2c2c2c2c2c2c2 c2c2c2c2c2c2c2c2 c2c2c2c2c2c2c2c2 c2c2c2c2c2c2c2c2 c2c2c2c200000014 c2c2c2c200000015
(XEN)   
(XEN)    ffff8300cf12d000 ffff8300cf12c000 0000003897ca3280 0000003897ca1280 c2c2c2c2c2c2c2c2 c2c2c2c2c2c2c2c2
(XEN) 
(XEN) Xen call trace:
(XEN) Xen call trace:
(XEN)    [<ffff82d08012bb1e>] check_lock+0x1e/0x3b
(XEN)    [<ffff82d08012bb1e>] check_lock+0x1e/0x3b
(XEN)    [<ffff82d08012bbc4>] _spin_lock_irq+0x1b/0x6c
(XEN)    [<ffff82d08012bbc4>] _spin_lock_irq+0x1b/0x6c
(XEN)    [<ffff82d080128175>] schedule+0xc0/0x5da
(XEN)    [<ffff82d080128175>] schedule+0xc0/0x5da
(XEN)    [<ffff82d08012b539>] __do_softirq+0x81/0x8c
(XEN)    [<ffff82d08012b539>] __do_softirq+0x81/0x8c
(XEN)    [<ffff82d08012b591>] do_softirq+0x13/0x15
(XEN)    [<ffff82d08012b591>] do_softirq+0x13/0x15
(XEN)    [<ffff82d080160425>] idle_loop+0x5e/0x6e
(XEN)    [<ffff82d080160425>] idle_loop+0x5e/0x6e
(XEN) 
(XEN) 
(XEN) Pagetable walk from 0000000000000004:
(XEN) Pagetable walk from 0000000000000004:
(XEN)  L4[0x000] = 000000092d6a4063 ffffffffffffffff
(XEN)  L4[0x000] = 000000092d6a4063 ffffffffffffffff
(XEN)  L3[0x000] = 000000092d6a3063 ffffffffffffffff
(XEN)  L3[0x000] = 000000092d6a3063 ffffffffffffffff
(XEN)  L2[0x000] = 000000092d6a2063 ffffffffffffffff 
(XEN)  L2[0x000] = 000000092d6a2063 ffffffffffffffff 
(XEN)  L1[0x000] = 0000000000000000 ffffffffffffffff
(XEN)  L1[0x000] = 0000000000000000 ffffffffffffffff
(XEN) 
(XEN) ****************************************
(XEN) Panic on CPU 20:
(XEN) FATAL PAGE FAULT
(XEN) [error_code=0000]
(XEN) Faulting linear address: 0000000000000004
(XEN) ****************************************
(XEN) 
(XEN) Manual reset required ('noreboot' specified)
(XEN) 
(XEN) ****************************************
(XEN) Panic on CPU 21:
(XEN) FATAL PAGE FAULT
(XEN) [error_code=0000]
(XEN) Faulting linear address: 0000000000000004
(XEN) ****************************************
(XEN) 
(XEN) Manual reset required ('noreboot' specified)

Can you please check, try to reproduce, and fix ASAP?

This, IMO, does not alter the prognosis, wrt 4.5 inclusion, at least not
per-se. In fact, it's ok for some bugs to be around at feature freeze
time, for the features we said we want. What we need to know is that
we're likely going to be able to fix them without making the release
slip.

So you should really either fix this, or provide here enough insights,
to convince people you're on the way to that. :-)

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Introduce rtds real-time scheduler for Xen

[Meng Xu]

[-- Attachment #1.1: Type: text/plain, Size: 9352 bytes --]

Hi Dario,

First, thank you so much for testing it. I really appreciate it. :-)

2014-09-17 10:15 GMT-04:00 Dario Faggioli <dario.faggioli@citrix.com>:

> On dom, 2014-09-14 at 17:37 -0400, Meng Xu wrote:
> > This serie of patches adds rtds real-time scheduler to Xen.
> >
> I gave this series some testing, and the behavior of the scheduler is as
> expected, so again, Meng and Sisu, good work.
>
> While doing it, I've also put the series in this git repo/branch:
>
>   git://xenbits.xen.org/people/dariof/xen.git  sched/rt/rtds-v3
>
> http://xenbits.xen.org/gitweb/?p=people/dariof/xen.git;a=shortlog;h=refs/heads/sched/rt/rtds-v3
>
>
> There are a couple of issue, though, one minor and one serious, which
> I'd like you to fix, if possible, before freeze date. More info below.
>
> > //list VCPUs' parameters of each domain in cpu pools using rtds scheduler
> > #xl sched-rtds
> > Cpupool Pool-0: sched=EDF
> > Name                                ID    Period    Budget
> > Domain-0                             0     10000      4000
> > vm1                                  1     10000      4000
> >
> So, when I boot Xen with sched=rtds, issueing this command (`xl
> sched-rtds') produces a lot of printk on the serial console, basically
> outputting the dump of the scheduler information.
>
> I guess there is one call to rt_sched_dump() (or whatever that was) left
> somewhere. Could you please check?
>
> This is not a serious issue, but since you're resending anyway...
>
> > //create a cpupool test
> > #xl cpupool-cpu-remove Pool-0 3
> > #xl cpupool-cpu-remove Pool-0 2
> > #xl cpupool-create name=\"test\" sched=\"rtds\"
> > #xl cpupool-cpu-add test 3
> > #xl cpupool-cpu-add test 2
> > #xl cpupool-list
> > Name               CPUs   Sched     Active   Domain count
> > Pool-0               2     rtds       y          2
> > test                 2     rtds       y          0
> >
> This works for me too.
>
> Booting with sched=credit, creating an rtds cpupool and migruting
> domains there also works here too.
>
> However, booting with sched=rtds, and issuing the following commands:
> # xl cpupool-cpu-remove Pool-0 20
> # xl cpupool-cpu-remove Pool-0 21
> # xl cpupool-create /etc/xen/be-cpupool
>
> Where /etc/xen/be-cpupool looks like this:
>  name = "be"
>  sched = "credit"
>  cpus = ["20", "21"]
>  sched="credit"
>
> Makes Xen *CRASH* with the following trace:
>
> (XEN) ----[ Xen-4.5-unstable  x86_64  debug=y  Not tainted ]----
> (XEN) ----[ Xen-4.5-unstable  x86_64  debug=y  Not tainted ]----
> (XEN) CPU:    20
> (XEN) CPU:    21
> (XEN) RIP:    e008:[<ffff82d08012bb1e>]RIP:    e008:[<ffff82d08012bb1e>]
> check_lock+0x1e/0x3b check_lock+0x1e/0x3b
> (XEN) RFLAGS: 0000000000010002
> (XEN) RFLAGS: 0000000000010002   CONTEXT: hypervisor
> (XEN) CONTEXT: hypervisor
> (XEN) rax: 0000000000000001   rbx: 0000000000000000   rcx: 0000000000000001
> (XEN) rax: 0000000000000001   rbx: 0000000000000000   rcx: 0000000000000001
> (XEN) rdx: 0000000000000001   rsi: ffff830917fc4c80   rdi: 0000000000000004
> (XEN) rdx: 0000000000000001   rsi: ffff830917fc2c80   rdi: 0000000000000004
> (XEN) rbp: ffff830917fb7e08   rsp: ffff830917fb7e08   r8:  0000001d52034e80
> (XEN) rbp: ffff830917fafe08   rsp: ffff830917fafe08   r8:  0000000000000000
> (XEN) r9:  ffff830828a47978   r10: 00000000deadbeef   r11: 0000000000000246
> (XEN) r9:  ffff830917fe3ea8   r10: 00000000deadbeef   r11: 0000000000000246
> (XEN) r12: 0000001d51efbd89   r13: ffff82d080320de0   r14: 0000000000000000
> (XEN) r12: 0000001d51efbb99   r13: ffff82d080320de0   r14: 0000000000000000
> (XEN) r15: 0000000000000014   cr0: 000000008005003b   cr4: 00000000000026f0
> (XEN) r15: 0000000000000015   cr0: 000000008005003b   cr4: 00000000000026f0
> (XEN) cr3: 00000000cf08f000   cr2: 0000000000000004
> (XEN) cr3: 00000000cf08f000   cr2: 0000000000000004
> (XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e010   cs: e008
> (XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e010   cs: e008
> (XEN) Xen stack trace from rsp=ffff830917fb7e08:
> (XEN)   Xen stack trace from rsp=ffff830917fafe08:
> (XEN)    ffff830917fb7e20 ffff830917fafe20 ffff82d08012bbc4
> ffff82d08012bbc4 ffff8300cf12d000 ffff8300cf12c000 ffff830917fb7eb0
> ffff830917fafeb0
> (XEN)
> (XEN)    ffff82d080128175 ffff82d080128175 ffff830917fb7e40
> ffff830917fafe40 ffff82d0801879ef ffff82d0801879ef 0000001400fb7e60
> 0000001500fafe60
> (XEN)
> (XEN)    ffff830917fc4060 ffff830917fc2060 ffff830917fb7e60
> ffff830917fafe60 ffff830917fc4200 ffff830917fc2200 ffff830917fb7eb0
> ffff830917fafeb0
> (XEN)
> (XEN)    ffff82d08012e983 ffff82d08012e983 ffff830917fb7ef0
> ffff830917fafef0 ffff82d0801aa941 ffff82d0801aa941 ffff830917fb7e90
> ffff830917fafe90
> (XEN)
> (XEN)    ffff82d0802f8980 ffff82d0802f8a00 ffff82d0802f7f80
> ffff82d0802f7f80 ffffffffffffffff ffffffffffffffff ffff830917fb0000
> ffff830917fa8000
> (XEN)
> (XEN)    00000000000f4240 00000000000f4240 ffff830917fb7ee0
> ffff830917fafee0 ffff82d08012b539 ffff82d08012b539 ffff830917fb0000
> ffff830917fa8000
> (XEN)
> (XEN)    0000001d51dfd294 0000001d51dfcb8f ffff8300cf12d000
> ffff8300cf12c000 ffff83092d6a0990 ffff83092d6a0990 ffff830917fb7ef0
> ffff830917fafef0
> (XEN)
> (XEN)    ffff82d08012b591 ffff82d08012b591 ffff830917fb7f10
> ffff830917faff10 ffff82d080160425 ffff82d080160425 ffff82d08012b591
> ffff82d08012b591
> (XEN)
> (XEN)    ffff8300cf12d000 ffff8300cf12c000 ffff830917fb7e10
> ffff830917fafe10 0000000000000000 0000000000000000 ffff88003a0cbfd8
> ffff88003a0edfd8
> (XEN)
> (XEN)    ffff88003a0cbfd8 ffff88003a0edfd8 0000000000000007
> 0000000000000014 ffff88003a0cbec0 ffff88003a0edec0 0000000000000000
> 0000000000000000
> (XEN)
> (XEN)    0000000000000246 0000000000000246 0000001c6462ff88
> 0000001c986bebc8 0000000000000000 0000000000000000 0000000000000000
> 0000000000000000
> (XEN)
> (XEN)    0000000000000000 0000000000000000 ffffffff810013aa
> ffffffff810013aa ffffffff81c31160 ffffffff81c31160 00000000deadbeef
> 00000000deadbeef
> (XEN)
> (XEN)    00000000deadbeef 00000000deadbeef 0000010000000000
> 0000010000000000 ffffffff810013aa ffffffff810013aa 000000000000e033
> 000000000000e033
> (XEN)
> (XEN)    0000000000000246 0000000000000246 ffff88003a0cbea8
> ffff88003a0edea8 000000000000e02b 000000000000e02b c2c2c2c2c2c2c2c2
> c2c2c2c2c2c2c2c2
> (XEN)
> (XEN)    c2c2c2c2c2c2c2c2 c2c2c2c2c2c2c2c2 c2c2c2c2c2c2c2c2
> c2c2c2c2c2c2c2c2 c2c2c2c2c2c2c2c2 c2c2c2c2c2c2c2c2 c2c2c2c200000014
> c2c2c2c200000015
> (XEN)
> (XEN)    ffff8300cf12d000 ffff8300cf12c000 0000003897ca3280
> 0000003897ca1280 c2c2c2c2c2c2c2c2 c2c2c2c2c2c2c2c2
> (XEN)
> (XEN) Xen call trace:
> (XEN) Xen call trace:
> (XEN)    [<ffff82d08012bb1e>] check_lock+0x1e/0x3b
> (XEN)    [<ffff82d08012bb1e>] check_lock+0x1e/0x3b
> (XEN)    [<ffff82d08012bbc4>] _spin_lock_irq+0x1b/0x6c
> (XEN)    [<ffff82d08012bbc4>] _spin_lock_irq+0x1b/0x6c
> (XEN)    [<ffff82d080128175>] schedule+0xc0/0x5da
> (XEN)    [<ffff82d080128175>] schedule+0xc0/0x5da
> (XEN)    [<ffff82d08012b539>] __do_softirq+0x81/0x8c
> (XEN)    [<ffff82d08012b539>] __do_softirq+0x81/0x8c
> (XEN)    [<ffff82d08012b591>] do_softirq+0x13/0x15
> (XEN)    [<ffff82d08012b591>] do_softirq+0x13/0x15
> (XEN)    [<ffff82d080160425>] idle_loop+0x5e/0x6e
> (XEN)    [<ffff82d080160425>] idle_loop+0x5e/0x6e
> (XEN)
> (XEN)
> (XEN) Pagetable walk from 0000000000000004:
> (XEN) Pagetable walk from 0000000000000004:
> (XEN)  L4[0x000] = 000000092d6a4063 ffffffffffffffff
> (XEN)  L4[0x000] = 000000092d6a4063 ffffffffffffffff
> (XEN)  L3[0x000] = 000000092d6a3063 ffffffffffffffff
> (XEN)  L3[0x000] = 000000092d6a3063 ffffffffffffffff
> (XEN)  L2[0x000] = 000000092d6a2063 ffffffffffffffff
> (XEN)  L2[0x000] = 000000092d6a2063 ffffffffffffffff
> (XEN)  L1[0x000] = 0000000000000000 ffffffffffffffff
> (XEN)  L1[0x000] = 0000000000000000 ffffffffffffffff
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 20:
> (XEN) FATAL PAGE FAULT
> (XEN) [error_code=0000]
> (XEN) Faulting linear address: 0000000000000004
> (XEN) ****************************************
> (XEN)
> (XEN) Manual reset required ('noreboot' specified)
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 21:
> (XEN) FATAL PAGE FAULT
> (XEN) [error_code=0000]
> (XEN) Faulting linear address: 0000000000000004
> (XEN) ****************************************
> (XEN)
> (XEN) Manual reset required ('noreboot' specified)
>
> Can you please check, try to reproduce, and fix ASAP?
>

​Sure! Start doing it now. I will try to reproduce it and let you know.​


>
> This, IMO, does not alter the prognosis, wrt 4.5 inclusion, at least not
> per-se. In fact, it's ok for some bugs to be around at feature freeze
> time, for the features we said we want. What we need to know is that
> we're likely going to be able to fix them without making the release
> slip.
>
> So you should really either fix this, or provide here enough insights,
> to convince people you're on the way to that. :-)
>

​Sure! Thanks!

Meng​




-- 


-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania

[-- Attachment #1.2: Type: text/html, Size: 11419 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Introduce rtds real-time scheduler for Xen

[Meng Xu]

Hi Dario,

I think I fixed the bug and could you please test it again. :-) I will
comment the error for the detailed explanation.

2014-09-17 10:15 GMT-04:00 Dario Faggioli <dario.faggioli@citrix.com>:
> On dom, 2014-09-14 at 17:37 -0400, Meng Xu wrote:
>> This serie of patches adds rtds real-time scheduler to Xen.
>>
> I gave this series some testing, and the behavior of the scheduler is as
> expected, so again, Meng and Sisu, good work.
>
> While doing it, I've also put the series in this git repo/branch:
>
>   git://xenbits.xen.org/people/dariof/xen.git  sched/rt/rtds-v3
>   http://xenbits.xen.org/gitweb/?p=people/dariof/xen.git;a=shortlog;h=refs/heads/sched/rt/rtds-v3
>
>
> There are a couple of issue, though, one minor and one serious, which
> I'd like you to fix, if possible, before freeze date. More info below.
>
>> //list VCPUs' parameters of each domain in cpu pools using rtds scheduler
>> #xl sched-rtds
>> Cpupool Pool-0: sched=EDF
>> Name                                ID    Period    Budget
>> Domain-0                             0     10000      4000
>> vm1                                  1     10000      4000
>>
> So, when I boot Xen with sched=rtds, issueing this command (`xl
> sched-rtds') produces a lot of printk on the serial console, basically
> outputting the dump of the scheduler information.
>
> I guess there is one call to rt_sched_dump() (or whatever that was) left
> somewhere. Could you please check?
>
> This is not a serious issue, but since you're resending anyway...
>
>> //create a cpupool test
>> #xl cpupool-cpu-remove Pool-0 3
>> #xl cpupool-cpu-remove Pool-0 2
>> #xl cpupool-create name=\"test\" sched=\"rtds\"
>> #xl cpupool-cpu-add test 3
>> #xl cpupool-cpu-add test 2
>> #xl cpupool-list
>> Name               CPUs   Sched     Active   Domain count
>> Pool-0               2     rtds       y          2
>> test                 2     rtds       y          0
>>
> This works for me too.
>
> Booting with sched=credit, creating an rtds cpupool and migruting
> domains there also works here too.
>
> However, booting with sched=rtds, and issuing the following commands:
> # xl cpupool-cpu-remove Pool-0 20
> # xl cpupool-cpu-remove Pool-0 21
> # xl cpupool-create /etc/xen/be-cpupool
>
> Where /etc/xen/be-cpupool looks like this:
>  name = "be"
>  sched = "credit"
>  cpus = ["20", "21"]
>  sched="credit"
>
> Makes Xen *CRASH* with the following trace:
>
> (XEN) ----[ Xen-4.5-unstable  x86_64  debug=y  Not tainted ]----
> (XEN) ----[ Xen-4.5-unstable  x86_64  debug=y  Not tainted ]----
> (XEN) CPU:    20
> (XEN) CPU:    21
> (XEN) RIP:    e008:[<ffff82d08012bb1e>]RIP:    e008:[<ffff82d08012bb1e>] check_lock+0x1e/0x3b check_lock+0x1e/0x3b
> (XEN) RFLAGS: 0000000000010002
> (XEN) RFLAGS: 0000000000010002   CONTEXT: hypervisor
> (XEN) CONTEXT: hypervisor
> (XEN) rax: 0000000000000001   rbx: 0000000000000000   rcx: 0000000000000001
> (XEN) rax: 0000000000000001   rbx: 0000000000000000   rcx: 0000000000000001
> (XEN) rdx: 0000000000000001   rsi: ffff830917fc4c80   rdi: 0000000000000004
> (XEN) rdx: 0000000000000001   rsi: ffff830917fc2c80   rdi: 0000000000000004
> (XEN) rbp: ffff830917fb7e08   rsp: ffff830917fb7e08   r8:  0000001d52034e80
> (XEN) rbp: ffff830917fafe08   rsp: ffff830917fafe08   r8:  0000000000000000
> (XEN) r9:  ffff830828a47978   r10: 00000000deadbeef   r11: 0000000000000246
> (XEN) r9:  ffff830917fe3ea8   r10: 00000000deadbeef   r11: 0000000000000246
> (XEN) r12: 0000001d51efbd89   r13: ffff82d080320de0   r14: 0000000000000000
> (XEN) r12: 0000001d51efbb99   r13: ffff82d080320de0   r14: 0000000000000000
> (XEN) r15: 0000000000000014   cr0: 000000008005003b   cr4: 00000000000026f0
> (XEN) r15: 0000000000000015   cr0: 000000008005003b   cr4: 00000000000026f0
> (XEN) cr3: 00000000cf08f000   cr2: 0000000000000004
> (XEN) cr3: 00000000cf08f000   cr2: 0000000000000004
> (XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e010   cs: e008
> (XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e010   cs: e008
> (XEN) Xen stack trace from rsp=ffff830917fb7e08:
> (XEN)   Xen stack trace from rsp=ffff830917fafe08:
> (XEN)    ffff830917fb7e20 ffff830917fafe20 ffff82d08012bbc4 ffff82d08012bbc4 ffff8300cf12d000 ffff8300cf12c000 ffff830917fb7eb0 ffff830917fafeb0
> (XEN)
> (XEN)    ffff82d080128175 ffff82d080128175 ffff830917fb7e40 ffff830917fafe40 ffff82d0801879ef ffff82d0801879ef 0000001400fb7e60 0000001500fafe60
> (XEN)
> (XEN)    ffff830917fc4060 ffff830917fc2060 ffff830917fb7e60 ffff830917fafe60 ffff830917fc4200 ffff830917fc2200 ffff830917fb7eb0 ffff830917fafeb0
> (XEN)
> (XEN)    ffff82d08012e983 ffff82d08012e983 ffff830917fb7ef0 ffff830917fafef0 ffff82d0801aa941 ffff82d0801aa941 ffff830917fb7e90 ffff830917fafe90
> (XEN)
> (XEN)    ffff82d0802f8980 ffff82d0802f8a00 ffff82d0802f7f80 ffff82d0802f7f80 ffffffffffffffff ffffffffffffffff ffff830917fb0000 ffff830917fa8000
> (XEN)
> (XEN)    00000000000f4240 00000000000f4240 ffff830917fb7ee0 ffff830917fafee0 ffff82d08012b539 ffff82d08012b539 ffff830917fb0000 ffff830917fa8000
> (XEN)
> (XEN)    0000001d51dfd294 0000001d51dfcb8f ffff8300cf12d000 ffff8300cf12c000 ffff83092d6a0990 ffff83092d6a0990 ffff830917fb7ef0 ffff830917fafef0
> (XEN)
> (XEN)    ffff82d08012b591 ffff82d08012b591 ffff830917fb7f10 ffff830917faff10 ffff82d080160425 ffff82d080160425 ffff82d08012b591 ffff82d08012b591
> (XEN)
> (XEN)    ffff8300cf12d000 ffff8300cf12c000 ffff830917fb7e10 ffff830917fafe10 0000000000000000 0000000000000000 ffff88003a0cbfd8 ffff88003a0edfd8
> (XEN)
> (XEN)    ffff88003a0cbfd8 ffff88003a0edfd8 0000000000000007 0000000000000014 ffff88003a0cbec0 ffff88003a0edec0 0000000000000000 0000000000000000
> (XEN)
> (XEN)    0000000000000246 0000000000000246 0000001c6462ff88 0000001c986bebc8 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN)
> (XEN)    0000000000000000 0000000000000000 ffffffff810013aa ffffffff810013aa ffffffff81c31160 ffffffff81c31160 00000000deadbeef 00000000deadbeef
> (XEN)
> (XEN)    00000000deadbeef 00000000deadbeef 0000010000000000 0000010000000000 ffffffff810013aa ffffffff810013aa 000000000000e033 000000000000e033
> (XEN)
> (XEN)    0000000000000246 0000000000000246 ffff88003a0cbea8 ffff88003a0edea8 000000000000e02b 000000000000e02b c2c2c2c2c2c2c2c2 c2c2c2c2c2c2c2c2
> (XEN)
> (XEN)    c2c2c2c2c2c2c2c2 c2c2c2c2c2c2c2c2 c2c2c2c2c2c2c2c2 c2c2c2c2c2c2c2c2 c2c2c2c2c2c2c2c2 c2c2c2c2c2c2c2c2 c2c2c2c200000014 c2c2c2c200000015
> (XEN)
> (XEN)    ffff8300cf12d000 ffff8300cf12c000 0000003897ca3280 0000003897ca1280 c2c2c2c2c2c2c2c2 c2c2c2c2c2c2c2c2
> (XEN)
> (XEN) Xen call trace:
> (XEN) Xen call trace:
> (XEN)    [<ffff82d08012bb1e>] check_lock+0x1e/0x3b
> (XEN)    [<ffff82d08012bb1e>] check_lock+0x1e/0x3b
> (XEN)    [<ffff82d08012bbc4>] _spin_lock_irq+0x1b/0x6c
> (XEN)    [<ffff82d08012bbc4>] _spin_lock_irq+0x1b/0x6c
> (XEN)    [<ffff82d080128175>] schedule+0xc0/0x5da
> (XEN)    [<ffff82d080128175>] schedule+0xc0/0x5da
> (XEN)    [<ffff82d08012b539>] __do_softirq+0x81/0x8c
> (XEN)    [<ffff82d08012b539>] __do_softirq+0x81/0x8c
> (XEN)    [<ffff82d08012b591>] do_softirq+0x13/0x15
> (XEN)    [<ffff82d08012b591>] do_softirq+0x13/0x15
> (XEN)    [<ffff82d080160425>] idle_loop+0x5e/0x6e
> (XEN)    [<ffff82d080160425>] idle_loop+0x5e/0x6e
> (XEN)
> (XEN)
> (XEN) Pagetable walk from 0000000000000004:
> (XEN) Pagetable walk from 0000000000000004:
> (XEN)  L4[0x000] = 000000092d6a4063 ffffffffffffffff
> (XEN)  L4[0x000] = 000000092d6a4063 ffffffffffffffff
> (XEN)  L3[0x000] = 000000092d6a3063 ffffffffffffffff
> (XEN)  L3[0x000] = 000000092d6a3063 ffffffffffffffff
> (XEN)  L2[0x000] = 000000092d6a2063 ffffffffffffffff
> (XEN)  L2[0x000] = 000000092d6a2063 ffffffffffffffff
> (XEN)  L1[0x000] = 0000000000000000 ffffffffffffffff
> (XEN)  L1[0x000] = 0000000000000000 ffffffffffffffff
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 20:
> (XEN) FATAL PAGE FAULT
> (XEN) [error_code=0000]
> (XEN) Faulting linear address: 0000000000000004
> (XEN) ****************************************
> (XEN)
> (XEN) Manual reset required ('noreboot' specified)
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 21:
> (XEN) FATAL PAGE FAULT
> (XEN) [error_code=0000]
> (XEN) Faulting linear address: 0000000000000004
> (XEN) ****************************************
> (XEN)
> (XEN) Manual reset required ('noreboot' specified)
>
> Can you please check, try to reproduce, and fix ASAP?

This is a lock/null pointer issue. When we remove a cpu from a
cpupool, rt_vcpu_remove() @xen/common/sched_rt.c will be called. In
that function, I just set the schedule_lock of schedule_data to NULL
in version 3' patch, which causes null pointer bug when schedule() @
xen/common/schedule.c tries to access the lock by "lock =
pcpu_schedule_lock_irq(cpu);".

I fixed this by changing the rt_vcpu_remove() @ sched_rt.c. The
changed rt_vcpu_remove() function is at LINE 436 at
https://github.com/PennPanda/xenproject/blob/rtxen-v1.90-patch-v4/xen/common/sched_rt.c

I also tested the scenario you gave as above on my 12cores machine. No bugs now

After booting up the system with rtds, I did:
# xl cpupool-cpu-remove Pool-0 11
# xl cpupool-cpu-remove Pool-0 10
# xl cpupool-create /etc/xen/be-cpupool

Where /etc/xen/be-cpupool looks like this:
    name = "be"
    sched = "credit"
    cpus = ["10", "11"]
    sched="credit"


Could you please pull the code and test on your machine to confirm it
works on your 24 cores machine as well?
The latest code is at https://github.com/PennPanda/xenproject ,
branch: rtxen-v1.90-patch-v4

(Of course, I can send a next version of patch with this fix, but I
hope to make sure this is working on your machine so that people won't
get too many patches. Then I "hope" the next patch set could get
committed. :-))

Once I receive the confirmation that the bug does not exist on your
machine any more, I will release the next version with all other
comments fixed.


>
> This, IMO, does not alter the prognosis, wrt 4.5 inclusion, at least not
> per-se. In fact, it's ok for some bugs to be around at feature freeze
> time, for the features we said we want. What we need to know is that
> we're likely going to be able to fix them without making the release
> slip.
>
> So you should really either fix this, or provide here enough insights,
> to convince people you're on the way to that. :-)

I think I figured out what happened, so the bug should have been solved. :-)

Thanks,

Meng


-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania

Introduce rtds real-time scheduler for Xen

[Meng Xu]

This serie of patches adds rtds real-time scheduler to Xen.

In summary, It supports:
1) Preemptive Global Earliest Deadline First scheduling policy by
using a global RunQ for the scheduler;
2) Assign/display VCPUs' parameters of each domain (All VCPUs of each
domain have the same period and budget);
3) Supports CPU Pool
Note:
 a)  Although the toolstack only allows users to set the paramters of
all VCPUs of the same domain to the same number, the scheduler
supports to schedule VCPUs with different parameters of the same
domain. In Xen 4.6, we plan to support assign/display each VCPU's
parameters of each domain.
 b)  Parameters of a domain at tool stack is in microsecond, instead
of millisecond.

The status of this patch set is as follows:
[PATCH v4 1/4] xen: add real time scheduler rtds
      Acked-by: Jan Beulich <jbeulich@suse.com>

[PATCH v4 2/4] libxc: add rtds scheduler
    Reviewed-by: Dario Faggioli <dario.faggioli@citrix.com>
    Acked-by: Ian Campbell <ian.campbell@citrix.com>
    Acked-by: George Dunlap <george.dunlap@eu.citrix.com>

[PATCH v4 3/4] libxl: add rtds scheduler
     Reviewed-by: Dario Faggioli <dario.faggioli@citrix.com>
     Reviewed-by: George Dunlap <george.dunlap@eu.citrix.com>

[PATCH v4 4/4] xl: introduce rtds scheduler
     Reviewed-by: Dario Faggioli <dario.faggioli@citrix.com>
     Reviewed-by: George Dunlap <george.dunlap@eu.citrix.com>
     Acked-by: Ian Campbell <ian.campbell@citrix.com>

-----------------------------------------------------------------------------------------------------------------------------
TODO after Xen 4.5:
    a) Burn budget in finer granularity instead of 1ms; [medium]
    b) Use separate timer per vcpu for each vcpu's budget
replenishment, instead of scanning the full runqueue every now and
then [medium]
    c) Handle time stolen from domU by hypervisor. When it runs on a
machine with many sockets and lots of cores, the spin-lock for global
RunQ used in rtds scheduler could eat up time from domU, which could
make domU have less budget than it requires. [not sure about
difficulty right now] (Thank Konrad Rzeszutek to point this out in the
XenSummit. :-))
    d) Toolstack supports assiging/display each VCPU's parameters of
each domain.

-----------------------------------------------------------------------------------------------------------------------------
The design of this rtds scheduler is as follows:

This scheduler follows the Preemptive Global Earliest Deadline First
(EDF) theory in real-time field.
At any scheduling point, the VCPU with earlier deadline has higher
priority. The scheduler always picks the highest priority VCPU to run on a
feasible PCPU.
A PCPU is feasible if the VCPU can run on this PCPU and (the PCPU is
idle or has a lower-priority VCPU running on it.)

Each VCPU has a dedicated period and budget.
The deadline of a VCPU is at the end of each period;
A VCPU has its budget replenished at the beginning of each period;
While scheduled, a VCPU burns its budget.
The VCPU needs to finish its budget before its deadline in each period;
The VCPU discards its unused budget at the end of each period.
If a VCPU runs out of budget in a period, it has to wait until next period.

Each VCPU is implemented as a deferable server.
When a VCPU has a task running on it, its budget is continuously burned;
When a VCPU has no task but with budget left, its budget is preserved.

Queue scheme:
A global runqueue and a global depletedq for each CPU pool.
The runqueue holds all runnable VCPUs with budget and sorted by deadline;
The depletedq holds all VCPUs without budget and unsorted.

Note: cpumask and cpupool is supported.

If you are intersted in the details of the design and evaluation of
this rt scheduler, please refer to our paper "Real-TimeMulti-Core
Virtual Machine Scheduling in Xen"
(http://www.cis.upenn.edu/~mengxu/emsoft14/emsoft14.pdf), which will
be published in EMSOFT14. This paper has the following details:
    a) Desgin of this scheduler;
    b) Measurement of the implementation overhead, e.g., scheduler
overhead, context switch overhead, etc.
    c) Comparison of this rt scheduler and credit scheduler in terms
of the real-time performance.

If you are interested in other real-time schedulers in Xen, please
refer to the RT-Xen project's website
(https://sites.google.com/site/realtimexen/). It also supports
Preemptive Global Rate Monotonic schedulers.
-----------------------------------------------------------------------------------------------------------------------------
We tested the following commands/scenarios:
#xl list
#xl sched-rtds
//set VCPUs' parameters of each domain to new value
#xl sched-rtds -d Domain-0 -p 20000 -b 10000

// list cpupool information
#xl cpupool-list

//create a cpupool test
#xl cpupool-cpu-remove Pool-0 3
#xl cpupool-cpu-remove Pool-0 2
#xl cpupool-create name=\"test\" sched=\"rtds\"
#xl cpupool-cpu-add test 3
#xl cpupool-cpu-add test 2

//migrate vm1 from cpupool Pool-0 to cpupool test.
#xl cpupool-migrate vm1 test

We also tested the scenarios:
1) System boots with rtds scheduler, create a cpupool with credit
scheduler, and migrate a domU vm1 to the new cpupool;
    #xl cpupool-cpu-remove Pool-0 3
    #xl cpupool-create name=\"test\" sched=\"credit\"
    #xl cpupool-cpu-add test 3
    #xl cpupool-migrate vm1 test
2) System boots with credit scheduler, create a cpupool with rtds
scheduler and migrate a domU vm1 to the new cpupool;
    #xl cpupool-cpu-remove Pool-0 3
    #xl cpupool-cpu-remove Pool-0 2
    #xl cpupool-create name=\"test\" sched=\"rtds\"
    #xl cpupool-cpu-add test 3
    #xl cpupool-cpu-add test 2
    #xl cpupool-migrate vm1 test

-----------------------------------------------------------------------------------------------------------------------------
Any comment, question, and concerns are more than welcome! :-)

Thank you very much!

Best,

Meng

---
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

Re: Introduce rtds real-time scheduler for Xen

[Meng Xu]

Now this patch set has each patch reviewed and acked by maintainers. I
updated the latest status in this email.


2014-09-20 18:16 GMT-04:00 Meng Xu <xumengpanda@gmail.com>:
> This serie of patches adds rtds real-time scheduler to Xen.
>
> In summary, It supports:
> 1) Preemptive Global Earliest Deadline First scheduling policy by
> using a global RunQ for the scheduler;
> 2) Assign/display VCPUs' parameters of each domain (All VCPUs of each
> domain have the same period and budget);
> 3) Supports CPU Pool
> Note:
>  a)  Although the toolstack only allows users to set the paramters of
> all VCPUs of the same domain to the same number, the scheduler
> supports to schedule VCPUs with different parameters of the same
> domain. In Xen 4.6, we plan to support assign/display each VCPU's
> parameters of each domain.
>  b)  Parameters of a domain at tool stack is in microsecond, instead
> of millisecond.
>
> The status of this patch set is as follows:
> [PATCH v4 1/4] xen: add real time scheduler rtds
>       Acked-by: Jan Beulich <jbeulich@suse.com>

Reviewed-and-Tested-by: Dario Faggioli <dario.faggioli@citrix.com>
Reviewed-by: George Dunlap <george.dunlap@eu.citrix.com>

>
> [PATCH v4 2/4] libxc: add rtds scheduler
>     Reviewed-by: Dario Faggioli <dario.faggioli@citrix.com>
>     Acked-by: Ian Campbell <ian.campbell@citrix.com>
>     Acked-by: George Dunlap <george.dunlap@eu.citrix.com>
>
> [PATCH v4 3/4] libxl: add rtds scheduler
>      Reviewed-by: Dario Faggioli <dario.faggioli@citrix.com>
>      Reviewed-by: George Dunlap <george.dunlap@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

>
> [PATCH v4 4/4] xl: introduce rtds scheduler
>      Reviewed-by: Dario Faggioli <dario.faggioli@citrix.com>
>      Reviewed-by: George Dunlap <george.dunlap@eu.citrix.com>
>      Acked-by: Ian Campbell <ian.campbell@citrix.com>
>

A quick question:
Do I need to do anything now?

Thanks,

-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania

Re: Introduce rtds real-time scheduler for Xen

[Ian Campbell]

On Tue, 2014-09-23 at 09:27 -0400, Meng Xu wrote:
> Do I need to do anything now?

No (not even this mail was strictly needed), I'm in the middle of my
precommit test script...

Ian.

Re: Introduce rtds real-time scheduler for Xen

[Ian Campbell]

On Sun, 2014-09-14 at 17:37 -0400, Meng Xu wrote:
> This serie of patches adds rtds real-time scheduler to Xen.

I've applied v4 of this series, which was apparently unthreaded and had
no introductory email (hence replying to what I think is v3's intro).

For the avoidance of doubt I've applied the patches with these
message-ids in this order:

<1411251228-2093-1-git-send-email-mengxu@cis.upenn.edu>
<1411251258-2131-1-git-send-email-mengxu@cis.upenn.edu>
<1411251283-2169-1-git-send-email-mengxu@cis.upenn.edu>
<1411251302-2207-1-git-send-email-mengxu@cis.upenn.edu>

I think that's right but I suggest you check that the end result in the
staging branch corresponds with what you thought you were sending. 

Note that tools/libxc/xenctrl.h moved to tools/libxc/include/xenctrl.h
since your patch was posted and I fixed that up as I went. Also, it
didn't build for arm32 due to:

        sched_rt.c: In function ‘burn_budget’:
        sched_rt.c:630:17: error: format ‘%ld’ expects argument of type ‘long int’, but argument 3 has type ‘s_time_t’ [-Werror=format]
        
I therefore folded this into the first patch:

diff --git a/xen/common/sched_rt.c b/xen/common/sched_rt.c
index 4c522bb..6c8faeb 100644
--- a/xen/common/sched_rt.c
+++ b/xen/common/sched_rt.c
@@ -626,7 +626,7 @@ burn_budget(const struct scheduler *ops, struct rt_vcpu *svc, s_time_t now)
      */
     if ( delta < 0 )
     {
-        printk("%s, ATTENTION: now is behind last_start! delta = %ld\n",
+        printk("%s, ATTENTION: now is behind last_start! delta=%"PRI_stime"\n",
                 __func__, delta);
         svc->last_start = now;
         return;

(this seemed easier than another round trip, I hope it is ok)

Apart from trying not to  forget the introductory mail and threading in
the future, please use the same $subject prefix on the introductory mail
as the rest of the series (e.g. '[PATCH for 4.5 v4 0/4]: Introduce rtds
real-time scheduler for Xen'), so I can figure out what goes with what.

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Introduce rtds real-time scheduler for Xen

[Meng Xu]

2014-09-23 9:50 GMT-04:00 Ian Campbell <Ian.Campbell@citrix.com>:
> On Sun, 2014-09-14 at 17:37 -0400, Meng Xu wrote:
>> This serie of patches adds rtds real-time scheduler to Xen.
>
> I've applied v4 of this series, which was apparently unthreaded and had
> no introductory email (hence replying to what I think is v3's intro).
>
> For the avoidance of doubt I've applied the patches with these
> message-ids in this order:
>
> <1411251228-2093-1-git-send-email-mengxu@cis.upenn.edu>
> <1411251258-2131-1-git-send-email-mengxu@cis.upenn.edu>
> <1411251283-2169-1-git-send-email-mengxu@cis.upenn.edu>
> <1411251302-2207-1-git-send-email-mengxu@cis.upenn.edu>
>
> I think that's right but I suggest you check that the end result in the
> staging branch corresponds with what you thought you were sending.

Yes, all of these are correct. I double-checked that and run them.

>
> Note that tools/libxc/xenctrl.h moved to tools/libxc/include/xenctrl.h
> since your patch was posted and I fixed that up as I went. Also, it
> didn't build for arm32 due to:
>
>         sched_rt.c: In function ‘burn_budget’:
>         sched_rt.c:630:17: error: format ‘%ld’ expects argument of type ‘long int’, but argument 3 has type ‘s_time_t’ [-Werror=format]
>
> I therefore folded this into the first patch:
>
> diff --git a/xen/common/sched_rt.c b/xen/common/sched_rt.c
> index 4c522bb..6c8faeb 100644
> --- a/xen/common/sched_rt.c
> +++ b/xen/common/sched_rt.c
> @@ -626,7 +626,7 @@ burn_budget(const struct scheduler *ops, struct rt_vcpu *svc, s_time_t now)
>       */
>      if ( delta < 0 )
>      {
> -        printk("%s, ATTENTION: now is behind last_start! delta = %ld\n",
> +        printk("%s, ATTENTION: now is behind last_start! delta=%"PRI_stime"\n",
>                  __func__, delta);
>          svc->last_start = now;
>          return;
>
> (this seemed easier than another round trip, I hope it is ok)

It's right! Thank you so much!

>
> Apart from trying not to  forget the introductory mail and threading in
> the future, please use the same $subject prefix on the introductory mail
> as the rest of the series (e.g. '[PATCH for 4.5 v4 0/4]: Introduce rtds
> real-time scheduler for Xen'), so I can figure out what goes with what.

I will remember that and avoid this. I used my gmail to send the
introductory email and used my school's email to send the patches,
that could be the reason why it's hard to find the introductory mail.
Anyway, I will avoid this and add the introductory mail as 0/4 as you
said.

Thank you very much!

Best,

Meng

>
> Ian.
>



-- 


-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Introduce rtds real-time scheduler for Xen

[Wei Liu]

On Wed, Sep 24, 2014 at 04:59:18PM -0400, Meng Xu wrote:
[...]
> 
> >
> > Apart from trying not to  forget the introductory mail and threading in
> > the future, please use the same $subject prefix on the introductory mail
> > as the rest of the series (e.g. '[PATCH for 4.5 v4 0/4]: Introduce rtds
> > real-time scheduler for Xen'), so I can figure out what goes with what.
> 
> I will remember that and avoid this. I used my gmail to send the
> introductory email and used my school's email to send the patches,
> that could be the reason why it's hard to find the introductory mail.
> Anyway, I will avoid this and add the introductory mail as 0/4 as you
> said.
> 

FWIW git format-patch has --cover option to nicely generate a cover
letter with some useful diffstat in it. Then you can add in your stuffs
in cover letter and send that along with other patches all in one go
with git send-email.

Wei.

> Thank you very much!
> 
> Best,
> 
> Meng
> 
> >
> > Ian.
> >
> 
> 
> 
> -- 
> 
> 
> -----------
> Meng Xu
> PhD Student in Computer and Information Science
> University of Pennsylvania
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Introduce rtds real-time scheduler for Xen

[Ian Campbell]

On Wed, 2014-09-24 at 22:14 +0100, Wei Liu wrote:
> On Wed, Sep 24, 2014 at 04:59:18PM -0400, Meng Xu wrote:
> [...]
> > 
> > >
> > > Apart from trying not to  forget the introductory mail and threading in
> > > the future, please use the same $subject prefix on the introductory mail
> > > as the rest of the series (e.g. '[PATCH for 4.5 v4 0/4]: Introduce rtds
> > > real-time scheduler for Xen'), so I can figure out what goes with what.
> > 
> > I will remember that and avoid this. I used my gmail to send the
> > introductory email and used my school's email to send the patches,
> > that could be the reason why it's hard to find the introductory mail.
> > Anyway, I will avoid this and add the introductory mail as 0/4 as you
> > said.
> > 
> 
> FWIW git format-patch has --cover option to nicely generate a cover
> letter with some useful diffstat in it. Then you can add in your stuffs
> in cover letter and send that along with other patches all in one go
> with git send-email.

Also FWIW you can use
         git send-email --in-reply-to=<message.id@domain.example>
to hook the series onto a 0/N mail which you sent manually.

Ian.