[patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Ingo Molnar]

i have released the -V0.7.46-01 Real-Time Preemption patch, which can be 
downloaded from the usual place:

    http://redhat.com/~mingo/realtime-preempt/

this includes fixes from Daniel Walker, which could fix the plist 
related slowdown bugs:

 - fix plist_del_init() argument order bug

 - fix plist related priority calculation bug in __up_mutex()

to build a -V0.7.46-01 tree, the following patches have to be applied:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.11.tar.bz2
   http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.12-rc3.bz2
   http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.12-rc3-V0.7.46-01

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Ingo Molnar]

> this includes fixes from Daniel Walker, which could fix the plist 
> related slowdown bugs:

there are still some problems remaining: i just ran Esben Nielsen's 
priority-inheritance validation testsuite, and the plist code gives a 
worst-case latency of 9.0 msecs.

I've reverted the plist changes for now and have uploaded -46-02 - this 
gives the expected 1.0 msec worst-case latencies. Diffing -01 against 
-02 should give you the latest plist snapshot.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Daniel Walker]

Is this the PriorityInversionTest (pi_test.tgz) or was there another one?


Daniel


On Fri, 22 Apr 2005, Ingo Molnar wrote:

> 
> > this includes fixes from Daniel Walker, which could fix the plist 
> > related slowdown bugs:
> 
> there are still some problems remaining: i just ran Esben Nielsen's 
> priority-inheritance validation testsuite, and the plist code gives a 
> worst-case latency of 9.0 msecs.
> 
> I've reverted the plist changes for now and have uploaded -46-02 - this 
> gives the expected 1.0 msec worst-case latencies. Diffing -01 against 
> -02 should give you the latest plist snapshot.
> 
> 	Ingo
> 

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Ingo Molnar]

* Daniel Walker <dwalker@mvista.com> wrote:

> Is this the PriorityInversionTest (pi_test.tgz) or was there another one?

yes, it's pi_test.tgz.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Daniel Walker]

What command line did you use ?

./test --samples 10000 --hertz 128 --tasks 0


Daniel

On Fri, 22 Apr 2005, Ingo Molnar wrote:

> 
> * Daniel Walker <dwalker@mvista.com> wrote:
> 
> > Is this the PriorityInversionTest (pi_test.tgz) or was there another one?
> 
> yes, it's pi_test.tgz.
> 
> 	Ingo
> 

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Ingo Molnar]

* Daniel Walker <dwalker@mvista.com> wrote:

> What command line did you use ?
> 
> ./test --samples 10000 --hertz 128 --tasks 0

i used:

  ./test --tasks 10 file.hist

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> 
> * Daniel Walker <dwalker@mvista.com> wrote:
> 
> > What command line did you use ?
> > 
> > ./test --samples 10000 --hertz 128 --tasks 0
> 
> i used:
> 
>   ./test --tasks 10 file.hist

but first i did:

chrt -f 98 -p `pidof 'IRQ 8'`

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Inaky Perez-Gonzalez]

>>>>> Ingo Molnar <mingo@elte.hu> writes:

>> this includes fixes from Daniel Walker, which could fix the plist
>> related slowdown bugs:

> there are still some problems remaining: i just ran Esben Nielsen's
> priority-inheritance validation testsuite, and the plist code gives
> a worst-case latency of 9.0 msecs.

With which machine is this? 

I tried to reproduce with V0.7.45-01 in my 2xPIII 0.9 Ghz I cannot get
more than 1.9ms when doing 2000 samples repeatedly (60 iterations and
counting). 

Did you use other parameters to 'test'?

# cnt=0 
# while true
> do echo -n "$cnt "
> ./test --samples 2000 file.hist 2>&1 | grep maximum; cnt=$(($cnt+1))
> done

With 2.6.12-rc3-V0.7.46-02 I get:

...
2 maximum cycle time: 0.003ms
3 maximum cycle time: 1.706ms
4 maximum cycle time: 1.538ms
5 maximum cycle time: 1.168ms
6 maximum cycle time: 0.003ms
7 maximum cycle time: 0.008ms
8 maximum cycle time: 1.821ms
9 maximum cycle time: 1.013ms
10 maximum cycle time: 1.043ms
11 maximum cycle time: 1.787ms
12 maximum cycle time: 1.519ms
13 ...

roughly the same as with 0.7.45-01:

0 maximum cycle time: 1.872ms
1 maximum cycle time: 0.008ms
2 maximum cycle time: 1.947ms
3 maximum cycle time: 1.902ms
4 maximum cycle time: 1.925ms
5 maximum cycle time: 0.002ms
6 maximum cycle time: 1.950ms
7 maximum cycle time: 0.008ms
8 maximum cycle time: 0.008ms
9 maximum cycle time: 0.008ms
10 maximum cycle time: 1.943ms
11 maximum cycle time: 1.173ms
12 maximum cycle time: 0.008ms
13 maximum cycle time: 0.008ms
...


-- 

Inaky

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Daniel Walker]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1025 bytes --]

t
On Fri, 22 Apr 2005, Inaky Perez-Gonzalez wrote:

> >>>>> Ingo Molnar <mingo@elte.hu> writes:
> 
> >> this includes fixes from Daniel Walker, which could fix the plist
> >> related slowdown bugs:
> 
> > there are still some problems remaining: i just ran Esben Nielsen's
> > priority-inheritance validation testsuite, and the plist code gives
> > a worst-case latency of 9.0 msecs.
> 
> With which machine is this? 
> 
> I tried to reproduce with V0.7.45-01 in my 2xPIII 0.9 Ghz I cannot get
> more than 1.9ms when doing 2000 samples repeatedly (60 iterations and
> counting). 
> 
> Did you use other parameters to 'test'?
> 
> # cnt=0 
> # while true
> > do echo -n "$cnt "
> > ./test --samples 2000 file.hist 2>&1 | grep maximum; cnt=$(($cnt+1))
> > done
> 
> With 2.6.12-rc3-V0.7.46-02 I get:

It's still to high , it should be under a millisecond .. 

 
I'm still testing but the times get better with this patch . I was 
initializing the lists to 0, instead of MAX_INT . Let me know if it 
changes anything.


Daniel


[-- Attachment #2: Type: TEXT/PLAIN, Size: 954 bytes --]

Index: linux-2.6.11/include/linux/plist.h
===================================================================
--- linux-2.6.11.orig/include/linux/plist.h	2005-04-22 19:36:54.000000000 +0000
+++ linux-2.6.11/include/linux/plist.h	2005-04-22 19:38:29.000000000 +0000
@@ -251,7 +251,7 @@
 static inline void plist_del_init(struct plist *pl, struct plist *plist)
 {
         plist_del (pl, plist);
-        plist_init(pl, 0);
+        plist_init(pl, INT_MAX);
 }
 
 /* Return the priority a pl node */
Index: linux-2.6.11/kernel/rt.c
===================================================================
--- linux-2.6.11.orig/kernel/rt.c	2005-04-22 15:05:33.000000000 +0000
+++ linux-2.6.11/kernel/rt.c	2005-04-22 19:36:29.000000000 +0000
@@ -939,6 +939,7 @@
 
 	set_task_state(task, TASK_UNINTERRUPTIBLE);
 
+	plist_init (&waiter.list, task->prio);
 	task_blocks_on_lock(&waiter, task, lock, eip);
 
 	TRACE_BUG_ON(!irqs_disabled());

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Daniel Walker]

Oops, I missed that you tested a non-plist kernel .. Maybe there are some 
lingering problems in current stuff..

Daniel


On Fri, 22 Apr 2005, Inaky Perez-Gonzalez wrote:


> With 2.6.12-rc3-V0.7.46-02 I get:
> 
> ...
> 2 maximum cycle time: 0.003ms
> 3 maximum cycle time: 1.706ms
> 4 maximum cycle time: 1.538ms
> 5 maximum cycle time: 1.168ms
> 6 maximum cycle time: 0.003ms
> 7 maximum cycle time: 0.008ms
> 8 maximum cycle time: 1.821ms
> 9 maximum cycle time: 1.013ms
> 10 maximum cycle time: 1.043ms
> 11 maximum cycle time: 1.787ms
> 12 maximum cycle time: 1.519ms
> 13 ...
> 

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Daniel Walker]

[-- Attachment #1: Type: text/plain, Size: 211 bytes --]

On Fri, 2005-04-22 at 08:55, Ingo Molnar wrote:

> i used:
> 
>   ./test --tasks 10 file.hist


This patch cleanup the delays on increased numbers of tasks. It goes on
to of the current plist snapshot. 

Daniel

[-- Attachment #2: fix_pi_list_init.patch --]
[-- Type: text/x-patch, Size: 1772 bytes --]

Index: linux-2.6.11/include/linux/plist.h
===================================================================
--- linux-2.6.11.orig/include/linux/plist.h	2005-04-22 19:36:54.000000000 +0000
+++ linux-2.6.11/include/linux/plist.h	2005-04-22 19:38:29.000000000 +0000
@@ -251,7 +251,7 @@
 static inline void plist_del_init(struct plist *pl, struct plist *plist)
 {
         plist_del (pl, plist);
-        plist_init(pl, 0);
+        plist_init(pl, INT_MAX);
 }
 
 /* Return the priority a pl node */
Index: linux-2.6.11/include/linux/rt_lock.h
===================================================================
--- linux-2.6.11.orig/include/linux/rt_lock.h	2005-04-21 17:49:26.000000000 +0000
+++ linux-2.6.11/include/linux/rt_lock.h	2005-04-22 22:35:37.000000000 +0000
@@ -95,12 +95,12 @@
 #ifdef CONFIG_RT_DEADLOCK_DETECT
 # define __RT_MUTEX_INITIALIZER(lockname) \
 	{ .wait_lock = RAW_SPIN_LOCK_UNLOCKED, \
-	.wait_list = PLIST_INIT((lockname).wait_list, 0),  \
+	.wait_list = PLIST_INIT((lockname).wait_list, MAX_PRIO),  \
 	.name = #lockname, .file = __FILE__, .line = __LINE__ }
 #else
 # define __RT_MUTEX_INITIALIZER(lockname) \
 	{ .wait_lock = RAW_SPIN_LOCK_UNLOCKED, \
-	PLIST_INIT((lockname).wait_list, 0) }
+	PLIST_INIT((lockname).wait_list, MAX_PRIO) }
 #endif
 /*
  * RW-semaphores are an RT mutex plus a reader-depth count.
Index: linux-2.6.11/kernel/rt.c
===================================================================
--- linux-2.6.11.orig/kernel/rt.c	2005-04-22 15:05:33.000000000 +0000
+++ linux-2.6.11/kernel/rt.c	2005-04-22 22:34:22.000000000 +0000
@@ -939,6 +939,7 @@
 
 	set_task_state(task, TASK_UNINTERRUPTIBLE);
 
+	plist_init (&waiter.list, task->prio);
 	task_blocks_on_lock(&waiter, task, lock, eip);
 
 	TRACE_BUG_ON(!irqs_disabled());

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Lee Revell]

[-- Attachment #1: Type: text/plain, Size: 798 bytes --]

On Fri, 2005-04-22 at 08:27 +0200, Ingo Molnar wrote:
> i have released the -V0.7.46-01 Real-Time Preemption patch, which can be 
> downloaded from the usual place:
> 
>     http://redhat.com/~mingo/realtime-preempt/

Ingo,

With Mingming's patch to resolve the ext3 allocate-with-reservation
latency, it looks like we are finally getting close to the lower limit
that can be achieved with PREEMPT_DESKTOP.  I've attached the trace of
the lowest latency over several days of testing with
2.6.12-rc3-RT-V0.7.46-02 + Mingming's patch.  It's only 127 usecs, and
IIRC you mentioned previously that this code path can't be made
preemptible in !PREEMPT_RT.

Since Mingming's patch will have to live in -mm for a while, can it be
added to the RT patchset as well?

http://lkml.org/lkml/2005/4/22/127

Lee

[-- Attachment #2: signal-delivery-trace.bz2 --]
[-- Type: application/x-bzip, Size: 1957 bytes --]

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Ingo Molnar]

* Lee Revell <rlrevell@joe-job.com> wrote:

> On Fri, 2005-04-22 at 08:27 +0200, Ingo Molnar wrote:
> > i have released the -V0.7.46-01 Real-Time Preemption patch, which can be 
> > downloaded from the usual place:
> > 
> >     http://redhat.com/~mingo/realtime-preempt/
> 
> Ingo,
> 
> With Mingming's patch to resolve the ext3 allocate-with-reservation 
> latency, it looks like we are finally getting close to the lower limit 
> that can be achieved with PREEMPT_DESKTOP.  I've attached the trace of 
> the lowest latency over several days of testing with 
> 2.6.12-rc3-RT-V0.7.46-02 + Mingming's patch.  It's only 127 usecs, and 
> IIRC you mentioned previously that this code path can't be made 
> preemptible in !PREEMPT_RT.

yeah, signal delivery will have to stay atomic in !PREEMPT_RT kernels.

> Since Mingming's patch will have to live in -mm for a while, can it be 
> added to the RT patchset as well?

i think so - i'll add it to the next patch.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Ingo Molnar]

* Daniel Walker <dwalker@mvista.com> wrote:

> > With 2.6.12-rc3-V0.7.46-02 I get:
> 
> It's still to high , it should be under a millisecond ..
>  
> I'm still testing but the times get better with this patch . I was 
> initializing the lists to 0, instead of MAX_INT . Let me know if it 
> changes anything.

could you send me a single patch that adds the plist stuff to the 
current tree? (that way i'll have your latest and i'll be able to 
add/remove it quicker)

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Lee Revell]

On Wed, 2005-05-04 at 10:22 +0200, Ingo Molnar wrote:
> * Lee Revell <rlrevell@joe-job.com> wrote:
> > it looks like we are finally getting close to the lower limit 
> > that can be achieved with PREEMPT_DESKTOP ... It's only 127 usecs, and 
> > IIRC you mentioned previously that this code path can't be made 
> > preemptible in !PREEMPT_RT.
> 
> yeah, signal delivery will have to stay atomic in !PREEMPT_RT kernels.
> 

OK.  I found a few more.

When umounting, shrink_dcache_sb() will produce a 3-4 ms. bump.  However
it's not clear this can be made preemptible.  AFAICT the whole thing
needs to be under the dcache_lock.  This one is pretty obvious so I'm
not attaching a trace.

There's also still one path in the VM related to page freeing that can
produce ~500 usec latencies but I don't have a trace handy now.

> > Since Mingming's patch will have to live in -mm for a while, can it be 
> > added to the RT patchset as well?
> 
> i think so - i'll add it to the next patch.

OK, great.

Lee

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Norval Watson]

I have tried to build 2.6.12-rc3 kernel with Ingo's
realtime-preempt-2.6.12-rc3-V0.7.46-02 on amd64

When I enabled option 4, Complete Realtime, make bzImage failed as
described in Lee Revell's Apr 12 lkml post:
http://lkml.org/lkml/2005/4/12/523

"HOSTCC scripts/bin2c 
CC arch/x86_64/kernel/asm-offsets.s 
CHK include/asm-x86_64/offset.h 
UPD include/asm-x86_64/offset.h 
CC init/main.o 
In file included from include/linux/rwsem.h:38, 
from include/linux/kobject.h:24, 
from include/linux/module.h:19, 
from init/main.c:16: 
include/asm/rwsem.h:55: error: redefinition of `struct rw_semaphore' 
etc etc......"

When I started again with option 3, Low Latency Desktop, the build got a
bit further before hanging: (Full ouput:
http://www.longforest.com/index.php?option=com_content&task=view&id=133&Itemid=2 )

"
  CC      arch/x86_64/kernel/head64.o
  CC      arch/x86_64/kernel/init_task.o
arch/x86_64/kernel/init_task.c:17: warning: implicit declaration of
function `__RWSEM_INITIALIZER'
arch/x86_64/kernel/init_task.c:17: warning: missing braces around
initializer

...etc etc etc...

 constant
arch/x86_64/kernel/init_task.c:17: error: (near initialization for
`init_mm.default_kioctx')
make[1]: *** [arch/x86_64/kernel/init_task.o] Error 1
make: *** [arch/x86_64/kernel] Error 2"

Please CC any advice (or abuse!),
I can supply further info if required
Cheers,
Norv

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc3-V0.7.46-01

[Ingo Molnar]

* Daniel Walker <dwalker@mvista.com> wrote:

> On Fri, 2005-04-22 at 08:55, Ingo Molnar wrote:
> 
> > i used:
> > 
> >   ./test --tasks 10 file.hist
> 
> This patch cleanup the delays on increased numbers of tasks. It goes 
> on to of the current plist snapshot.

ok, these fixes for priority init appear to have solved the pi-test 
latencies. I've uploaded the -rc4-RT-V0.7.47-00 patch with the plist 
code re-added again. It's looking good so far on my testboxes.

	Ingo