2.6.23.1-rt9 (and others)

2.6.23.1-rt9 (and others)

[Steven Rostedt]

This is a special announcement for the latest -rt patches. This is
actually announcing more than one tree (pay close attention to the
differences between -rt7, -rt8 and -rt9).

  2.6.23.1-rt6

   - Removed BUG_ON in exit (Steven Rostedt and Daniel Walker)

   -  Turn RCU preempt boost on by default (Steven Rostedt)
      (for when RCU PREEMPT is enabled)

   - Fixes for PowerPC (Paul McKenney)


  2.6.23.1-rt7

   - Found that there's a flaw in the PowerPC patch so
     it was pulled from the tree.

  2.6.23.1-rt8

   - More aggressive RT Balancing (Gregory Haskins)

  2.6.23.1-rt9

   - RT balancing by CPU priorities (Gregory Haskins)


Now benchmarks against 2.6.23.1-rt7 -rt8 and -rt9 would be greatly
appreciated.  These three are all present in

  http://www.kernel.org/pub/linux/kernel/projects/rt/

Gregory and I have been having disagreements on how to solve RT task
balancing among CPUS. Although we shared ideas back and forth, and both
our methods have been greatly influenced by each other, the real answer
comes from actual numbers. So these three versions are posted for your
convenience to see which actually do the best. I would be happy to tell
Gregory he's right, if the numbers prove it.

Currently, what we do to test RT latencies is to run Thomas Gleixner's
cyclictest
(http://git.kernel.org/?p=linux/kernel/git/tglx/rt-tests.git;a=summary)
as well as hackbench, to see what the maximum latencies we get are.

Other tests are welcomed too.


to build a 2.6.23.1-rt7 tree, the following patches should be applied:

  http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.23.1.tar.bz2 
  http://www.kernel.org/pub/linux/kernel/projects/rt/patch-2.6.23.1-rt7.bz2

to build a 2.6.23.1-rt8 tree, the following patches should be applied:

  http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.23.1.tar.bz2 
  http://www.kernel.org/pub/linux/kernel/projects/rt/patch-2.6.23.1-rt8.bz2

to build a 2.6.23.1-rt9 tree, the following patches should be applied:

  http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.23.1.tar.bz2 
  http://www.kernel.org/pub/linux/kernel/projects/rt/patch-2.6.23.1-rt9.bz2


And like always, my RT version of Matt Mackall's ketchup will get this
for you nicely:

  http://people.redhat.com/srostedt/rt/tools/ketchup-0.9.8-rt1

The broken out patches are also available.

Thanks!

-- Steve

Re: 2.6.23.1-rt9 (and others)

[Gregory Haskins]

>>> On Wed, Nov 7, 2007 at  9:17 AM, in message <4731C90E.7030509@gmx.net>, Dragan
Noveski <perodog@gmx.net> wrote: 

> hallo, i just tried to compile rt9 and rt7 and both times i get this 
> error at about the end of the 'make && make modules' step:
> 
>   GEN     .version
>   CHK     include/linux/compile.h
>   UPD     include/linux/compile.h
>   CC      init/version.o
>   LD      init/built-in.o
>   LD      .tmp_vmlinux1
> kernel/built-in.o: In function `sched_init':
> (.init.text+0x1af): undefined reference to `cpupri_init'
> make: *** [.tmp_vmlinux1] Fehler 1
> 
> 
> i am on uniprozessor machine here, IBM-thinkpad-r50e + debian testing.
> if you need parts of my config file, just feel free to tell me so and  i 
> ll try to provide you the information!

Doh!  Your error made me realize that I broke uniprocessor in -rt9.  Will fix right away.

As far as -rt7 is concerned, that doesn't make a lot of sense since cpupri isnt introduced until -rt9.  Perhaps your tree was dirtied from a previous application of -rt9?  Let me know if that doesn't appear to be the case.

Regards,
-Greg

Re: 2.6.23.1-rt9 (and others)

[Dragan Noveski]

Steven Rostedt wrote:
> This is a special announcement for the latest -rt patches. This is
> actually announcing more than one tree (pay close attention to the
> differences between -rt7, -rt8 and -rt9).
>
>   2.6.23.1-rt6
>
>    - Removed BUG_ON in exit (Steven Rostedt and Daniel Walker)
>
>    -  Turn RCU preempt boost on by default (Steven Rostedt)
>       (for when RCU PREEMPT is enabled)
>
>    - Fixes for PowerPC (Paul McKenney)
>
>
>   2.6.23.1-rt7
>
>    - Found that there's a flaw in the PowerPC patch so
>      it was pulled from the tree.
>
>   2.6.23.1-rt8
>
>    - More aggressive RT Balancing (Gregory Haskins)
>
>   2.6.23.1-rt9
>
>    - RT balancing by CPU priorities (Gregory Haskins)
>
>
> Now benchmarks against 2.6.23.1-rt7 -rt8 and -rt9 would be greatly
> appreciated.  These three are all present in
>
>   http://www.kernel.org/pub/linux/kernel/projects/rt/
>
> Gregory and I have been having disagreements on how to solve RT task
> balancing among CPUS. Although we shared ideas back and forth, and both
> our methods have been greatly influenced by each other, the real answer
> comes from actual numbers. So these three versions are posted for your
> convenience to see which actually do the best. I would be happy to tell
> Gregory he's right, if the numbers prove it.
>
> Currently, what we do to test RT latencies is to run Thomas Gleixner's
> cyclictest
> (http://git.kernel.org/?p=linux/kernel/git/tglx/rt-tests.git;a=summary)
> as well as hackbench, to see what the maximum latencies we get are.
>
> Other tests are welcomed too.
>
>
> to build a 2.6.23.1-rt7 tree, the following patches should be applied:
>
>   http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.23.1.tar.bz2 
>   http://www.kernel.org/pub/linux/kernel/projects/rt/patch-2.6.23.1-rt7.bz2
>
> to build a 2.6.23.1-rt8 tree, the following patches should be applied:
>
>   http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.23.1.tar.bz2 
>   http://www.kernel.org/pub/linux/kernel/projects/rt/patch-2.6.23.1-rt8.bz2
>
> to build a 2.6.23.1-rt9 tree, the following patches should be applied:
>
>   http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.23.1.tar.bz2 
>   http://www.kernel.org/pub/linux/kernel/projects/rt/patch-2.6.23.1-rt9.bz2
>
>
> And like always, my RT version of Matt Mackall's ketchup will get this
> for you nicely:
>
>   http://people.redhat.com/srostedt/rt/tools/ketchup-0.9.8-rt1
>
> The broken out patches are also available.
>
> Thanks!
>
> -- Steve
>
>   
hallo, i just tried to compile rt9 and rt7 and both times i get this 
error at about the end of the 'make && make modules' step:

  GEN     .version
  CHK     include/linux/compile.h
  UPD     include/linux/compile.h
  CC      init/version.o
  LD      init/built-in.o
  LD      .tmp_vmlinux1
kernel/built-in.o: In function `sched_init':
(.init.text+0x1af): undefined reference to `cpupri_init'
make: *** [.tmp_vmlinux1] Fehler 1


i am on uniprozessor machine here, IBM-thinkpad-r50e + debian testing.
if you need parts of my config file, just feel free to tell me so and  i 
ll try to provide you the information!

cheers,
doc

[PATCH] RT: fix uniprocessor build issue with new scheduler enhancements

[Gregory Haskins]

Primary issue is cpupri_init() is not defined, but also clean up
some warnings related to uniproc builds.

Signed-off-by: Gregory Haskins <ghaskins@novell.com>
CC: Dragan Noveski <perodog@gmx.net>
---

 kernel/sched.c        |    2 ++
 kernel/sched_cpupri.h |    5 +++++
 kernel/sched_rt.c     |    2 +-
 3 files changed, 8 insertions(+), 1 deletions(-)

diff --git a/kernel/sched.c b/kernel/sched.c
index 6f24aa0..365c987 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -863,9 +863,11 @@ static int balance_tasks(struct rq *this_rq, int this_cpu, struct rq *busiest,
 		      int *all_pinned, unsigned long *load_moved,
 		      int *this_best_prio, struct rq_iterator *iterator);
 
+#ifdef CONFIG_SMP
 static unsigned long source_load(int cpu, int type);
 static unsigned long target_load(int cpu, int type);
 static unsigned long cpu_avg_load_per_task(int cpu);
+#endif /* CONFIG_SMP */
 
 #include "sched_stats.h"
 #include "sched_rt.c"
diff --git a/kernel/sched_cpupri.h b/kernel/sched_cpupri.h
index 8cdd15d..2119495 100644
--- a/kernel/sched_cpupri.h
+++ b/kernel/sched_cpupri.h
@@ -5,6 +5,11 @@
 
 int  cpupri_find(struct task_struct *p, cpumask_t *lowest_mask);
 void cpupri_set(int cpu, int pri);
+
+#ifdef CONFIG_SMP
 void cpupri_init(void);
+#else
+#define cpupri_init() do { } while(0)
+#endif
 
 #endif /* _LINUX_CPUPRI_H */
diff --git a/kernel/sched_rt.c b/kernel/sched_rt.c
index 71ae9e6..0213aa2 100644
--- a/kernel/sched_rt.c
+++ b/kernel/sched_rt.c
@@ -207,9 +207,9 @@ yield_task_rt(struct rq *rq, struct task_struct *p)
 	requeue_task_rt(rq, p);
 }
 
+#ifdef CONFIG_SMP
 static int find_lowest_rq(struct task_struct *task);
 
-#ifdef CONFIG_SMP
 static int select_task_rq_rt(struct task_struct *p, int sync)
 {
 	struct rq *rq = task_rq(p);

Re: 2.6.23.1-rt9 (and others)

[Dragan Noveski]

Gregory Haskins wrote:
> Doh!  Your error made me realize that I broke uniprocessor in -rt9.  Will fix right away.
>
> As far as -rt7 is concerned, that doesn't make a lot of sense since cpupri isnt introduced until -rt9.  Perhaps your tree was dirtied from a previous application of -rt9?  Let me know if that doesn't appear to be the case.
>
> Regards,
> -Greg
>
>
>   
sorry, i am not sure if i did not done some missmatch by copying the 
config file into the tree, but i am always doing 'rm -r', and unpacking 
the tree before compiling.
i tried again the rt9 (100% without missmatching) but it does not work....
i ll give a try with the rt6 now.

very much thanks for the support and cheers,
doc

Re: 2.6.23.1-rt9 (and others)

[Steven Rostedt]

On Wed, 7 Nov 2007, Steven Rostedt wrote:

> This is a special announcement for the latest -rt patches. This is
> actually announcing more than one tree (pay close attention to the
> differences between -rt7, -rt8 and -rt9).
>

[...]

>   2.6.23.1-rt9
>
>    - RT balancing by CPU priorities (Gregory Haskins)

-rt9 has been found to break UP compilation, so -rt10 has been released
with the fix.

>
>
> Now benchmarks against 2.6.23.1-rt7 -rt8 and -rt9 would be greatly
> appreciated.  These three are all present in

So please compare -rt7, -rt8 and -rt10.

-- Steve

>
>   http://www.kernel.org/pub/linux/kernel/projects/rt/
>

Re: 2.6.23.1-rt9 (and others)

[Steven Rostedt]

On Wed, 7 Nov 2007, Dragan Noveski wrote:

> Gregory Haskins wrote:
> > Doh!  Your error made me realize that I broke uniprocessor in -rt9.  Will fix right away.
> >
> > As far as -rt7 is concerned, that doesn't make a lot of sense since cpupri isnt introduced until -rt9.  Perhaps your tree was dirtied from a previous application of -rt9?  Let me know if that doesn't appear to be the case.
> >
> > Regards,
> > -Greg
> >
> >
> >
> sorry, i am not sure if i did not done some missmatch by copying the
> config file into the tree, but i am always doing 'rm -r', and unpacking
> the tree before compiling.
> i tried again the rt9 (100% without missmatching) but it does not work....
> i ll give a try with the rt6 now.
>
> very much thanks for the support and cheers,


-rt6 is broken.

I'd recommend doing the following:

 wget -O /usr/local/bin/ketchup http://people.redhat.com/srostedt/rt/tools/ketchup-0.9.8-rt1

 mkdir tmp
 cd tmp
 ketchup -r -G 2.6.23.1-rt7

This will get you the 2.6.23.1-rt7 kernel and rename the "tmp" directory
to linux-2.6.23.1-rt7

After you compiled and install -rt7 while in the same directory you can do

 ketchup -r -G 2.6.23.1-rt10

and it will update that kernel tree to 2.6.23.1-rt10 and again rename that
directory. (2.6.23.1-rt10 which BTW has the compile fix).


-- Steve

Re: 2.6.23.1-rt9 (and others)

[Dragan Noveski]

Gregory Haskins wrote:
>
> As far as -rt7 is concerned, that doesn't make a lot of sense since cpupri isnt introduced until -rt9.  Perhaps your tree was dirtied from a previous application of -rt9?  Let me know if that doesn't appear to be the case.
>
> Regards,
> -Greg
>
>   
you were right, it was some missmatching from my side, so rt7 build 
successfully without adding any additional patch and rt8 bulding just now.

cheers,
doc

Re: 2.6.23.1-rt9 (and others)

[Dragan Noveski]

Gregory Haskins wrote:
>
> As far as -rt7 is concerned, that doesn't make a lot of sense since cpupri isnt introduced until -rt9.  Perhaps your tree was dirtied from a previous application of -rt9?  Let me know if that doesn't appear to be the case.
>
> Regards,
> -Greg
>
>   
you were right, it was some missmatching from my side, so rt7 build 
successfully without adding any additional patch and rt8 is bulding just 
now.

cheers,
doc

Re: 2.6.23.1-rt9 (and others)

[Steven Rostedt]

On Wed, 7 Nov 2007, Dragan Noveski wrote:

> Steven Rostedt wrote:
> >
> > Currently, what we do to test RT latencies is to run Thomas Gleixner's
> > cyclictest
> > (http://git.kernel.org/?p=linux/kernel/git/tglx/rt-tests.git;a=summary)
> > as well as hackbench, to see what the maximum latencies we get are.
> >
>
> hallo, can please someone suggest for how long time you are running
> cyclictest?
> if i run it for about 3 min. i get a feeling that the values will not
> change anymore, so is there any recommended time for how long to run it?

Actually, you want to run it for a long time, and under very heavy load.
I'm currently running it while doing a "make -j128" on a kernel tree. This
takes several hours to complete the make. So far with -rt11 I have a max
latency with one RT thread at the prio of 98 68us.

When you do run cyclic test, make sure you run it at a higher priority
than the irq threads. And also use the -n option since the default is to
test signals, and that part of the test isn't working so well.

Here's what I'm running right now:

./cyclictest -n -p98 -t1 -i250

While doing that "make -j120"

>
> another thing i do not know is, if i run cyclictest for lets say 3
> minutes, than stop ('ctrl+c') and than start again, i get often
> different values. so is it recommended to run it more times and than to
> pick up the best/worst cases.
>
> i have done some tests already with rt7,8 and 10 but i am not sure if
> the tests make any sense - thats why i would really appreciate some
> answers on the both questions.
>

Actually I didn't want to discourage you. For UP, -rt7,8 and 10 really
should be exactly the same (with some noise factures apart). The changes
that were made supposable only affect SMP boxes. But, I'm glad you are
doing the tests, since we also would like to know if anything creeped into
te UP side, and has caused any regressions.

Thanks, for taking the time to test this.

-- Steve

Re: 2.6.23.1-rt9 (and others)

[Dragan Noveski]

Steven Rostedt wrote:
>
> I'd recommend doing the following:
>
>  wget -O /usr/local/bin/ketchup http://people.redhat.com/srostedt/rt/tools/ketchup-0.9.8-rt1
>
>  mkdir tmp
>  cd tmp
>  ketchup -r -G 2.6.23.1-rt7
>
> This will get you the 2.6.23.1-rt7 kernel and rename the "tmp" directory
> to linux-2.6.23.1-rt7
>
> After you compiled and install -rt7 while in the same directory you can do
>
>  ketchup -r -G 2.6.23.1-rt10
>
> and it will update that kernel tree to 2.6.23.1-rt10 and again rename that
> directory. (2.6.23.1-rt10 which BTW has the compile fix).
>
>
> -- Steve
>
>   
hallo steve, very much thanks for this nice how-to, but i already
finished with rt7,8 and 10.

i ll keep this message and try another time.

cheers,
doc

Re: 2.6.23.1-rt9 (and others)

[Dragan Noveski]

Steven Rostedt wrote:
>
> Currently, what we do to test RT latencies is to run Thomas Gleixner's
> cyclictest
> (http://git.kernel.org/?p=linux/kernel/git/tglx/rt-tests.git;a=summary)
> as well as hackbench, to see what the maximum latencies we get are.
>   

hallo, can please someone suggest for how long time you are running
cyclictest?
if i run it for about 3 min. i get a feeling that the values will not
change anymore, so is there any recommended time for how long to run it?

another thing i do not know is, if i run cyclictest for lets say 3
minutes, than stop ('ctrl+c') and than start again, i get often
different values. so is it recommended to run it more times and than to
pick up the best/worst cases.

i have done some tests already with rt7,8 and 10 but i am not sure if
the tests make any sense - thats why i would really appreciate some
answers on the both questions.

cheers,
doc

Re: 2.6.23.1-rt9 (and others)

[Dragan Noveski]

Steven Rostedt wrote:
> On Wed, 7 Nov 2007, Dragan Noveski wrote:
>
>   
>> Steven Rostedt wrote:
>>     
>>> Currently, what we do to test RT latencies is to run Thomas Gleixner's
>>> cyclictest
>>> (http://git.kernel.org/?p=linux/kernel/git/tglx/rt-tests.git;a=summary)
>>> as well as hackbench, to see what the maximum latencies we get are.
>>>
>>>       
>> hallo, can please someone suggest for how long time you are running
>> cyclictest?
>> if i run it for about 3 min. i get a feeling that the values will not
>> change anymore, so is there any recommended time for how long to run it?
>>     
>
> Actually, you want to run it for a long time, and under very heavy load.
> I'm currently running it while doing a "make -j128" on a kernel tree. This
> takes several hours to complete the make. So far with -rt11 I have a max
> latency with one RT thread at the prio of 98 68us.
>
> When you do run cyclic test, make sure you run it at a higher priority
> than the irq threads. And also use the -n option since the default is to
> test signals, and that part of the test isn't working so well.
>
> Here's what I'm running right now:
>
> ./cyclictest -n -p98 -t1 -i250
>   

ok, i was trying like './cyclictest -n -t3 -p99' but i ll have a look 
into -t and -i options more deeper now. but i was running it always for 
about 3-4 minutes....
> While doing that "make -j120"
>
>   
>> another thing i do not know is, if i run cyclictest for lets say 3
>> minutes, than stop ('ctrl+c') and than start again, i get often
>> different values. so is it recommended to run it more times and than to
>> pick up the best/worst cases.
>>
>> i have done some tests already with rt7,8 and 10 but i am not sure if
>> the tests make any sense - thats why i would really appreciate some
>> answers on the both questions.
>>
>>     
>
> Actually I didn't want to discourage you. For UP, -rt7,8 and 10 really
> should be exactly the same (with some noise factures apart). The changes
> that were made supposable only affect SMP boxes. But, I'm glad you are
> doing the tests, since we also would like to know if anything creeped into
> te UP side, and has caused any regressions.
>   

i was expecting this part about 'smp boxes' but anyway, as now i know 
how to run cyclictest, i will be able to do again and again as the 
development goes on.
> Thanks, for taking the time to test this.
>
> -- Steve
>
>   
i thank you to for this very informative response!

cheers,
doc

Re: 2.6.23.1-rt9 (and others)

[Gregory Haskins]

>>> On Wed, Nov 7, 2007 at 10:41 AM, in message
<Pine.LNX.4.58.0711071040210.8572@gandalf.stny.rr.com>, Steven Rostedt
<rostedt@goodmis.org> wrote: 

> 
> So please compare -rt7, -rt8 and -rt10.

Here are my results from running:

sudo chrt -f 80 ./cyclictest -n -p 90 -t 8 -d 100 -i 100

with

while true; do make mrproper; make allmodconfig; make -j 128 > /dev/null; done
 on a plain kernel.org kernel

running in the background on an 8-way (4core x2) C2D Xeon 5335 system

23.1-rt7
---------------------------

  144.60 135.30 116.38 30/814 8184

  T: 0 ( 5990) P:90 I:100 C:11560415 Min:2 Act:  5 Avg: 4 Max:  83
  T: 1 ( 5991) P:89 I:200 C:5780208 Min: 2 Act:  5 Avg: 4 Max:  66
  T: 2 ( 5992) P:88 I:300 C:3853472 Min: 2 Act:  5 Avg: 5 Max:  89
  T: 3 ( 5993) P:87 I:400 C:2890104 Min: 2 Act:  6 Avg: 5 Max:  70
  T: 4 ( 5994) P:86 I:500 C:2312083 Min: 2 Act:  4 Avg: 5 Max:  91
  T: 5 ( 5995) P:85 I:600 C:1926736 Min: 2 Act:  5 Avg: 5 Max:  94
  T: 6 ( 5996) P:84 I:700 C:1651488 Min: 2 Act: 12 Avg: 5 Max: 115
  T: 7 ( 5997) P:83 I:800 C:1445052 Min: 2 Act:  6 Avg: 5 Max:  79

23.1-rt8
---------------------------
  119.95 106.56 107.98 37/811 18445

  T: 0 ( 5052) P:90 I:100 C:29592746 Min: 2 Act: 21 Avg: 4 Max:  78
  T: 1 ( 5053) P:89 I:200 C:14796374 Min: 2 Act:  6 Avg: 4 Max:  81
  T: 2 ( 5054) P:88 I:300 C:9864249 Min:  2 Act: 10 Avg: 4 Max:  88
  T: 3 ( 5055) P:87 I:400 C:7398187 Min:  2 Act:  6 Avg: 4 Max:  86
  T: 4 ( 5056) P:86 I:500 C:5918550 Min:  2 Act: 13 Avg: 9 Max:  69
  T: 5 ( 5057) P:85 I:600 C:4932125 Min:  2 Act: 11 Avg: 4 Max:  71
  T: 6 ( 5058) P:84 I:700 C:4227536 Min:  2 Act:  8 Avg: 5 Max:  65
  T: 7 ( 5059) P:83 I:800 C:3699094 Min:  2 Act:  4 Avg: 4 Max: 114

23.1-rt10
---------------------------
  143.39 123.49 117.92 22/791 5802

  T: 0 ( 5305) P:90 I:100 C:45332860 Min: 2 Act:  4 Avg: 4 Max: 89
  T: 1 ( 5306) P:89 I:200 C:22666431 Min: 2 Act:  3 Avg: 5 Max: 49
  T: 2 ( 5307) P:88 I:300 C:15110954 Min: 2 Act:  6 Avg: 5 Max: 76
  T: 3 ( 5308) P:87 I:400 C:11333216 Min: 2 Act:  7 Avg: 5 Max: 81
  T: 4 ( 5309) P:86 I:500 C:9066572 Min:  2 Act:  8 Avg: 5 Max: 57
  T: 5 ( 5310) P:85 I:600 C:7555477 Min:  2 Act:  8 Avg: 5 Max: 55
  T: 6 ( 5311) P:84 I:700 C:6476123 Min:  2 Act: 14 Avg: 6 Max: 73
  T: 7 ( 5312) P:83 I:800 C:5666608 Min:  2 Act:  5 Avg: 6 Max: 78

note that the rt10 image was running for a much longer duration than the other two...which generally will push the max higher.  I know in general if I let -rt7 go that long it will hit 120-150+.  This means these tests were biased towards -rt7 and -rt8 performing better, but they still came in with higher latencies.

I will officially test -rt11 next, tho in my small runs so far it looks great.

HTH

Regards,
-Greg