2.6.16-rt1

2.6.16-rt1

[Ingo Molnar]

i have released the 2.6.16-rt1 tree, which can be downloaded from the 
usual place:

   http://redhat.com/~mingo/realtime-preempt/

this is mostly a merge from -rc6 to 2.6.16-final, plus some smaller 
fixes.

to build a 2.6.16-rt1 tree, the following patches should be applied:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.16.tar.bz2
  http://redhat.com/~mingo/realtime-preempt/patch-2.6.16-rt1

	Ingo

Re: 2.6.16-rt1

[K.R. Foley]

Ingo Molnar wrote:
> i have released the 2.6.16-rt1 tree, which can be downloaded from the 
> usual place:
> 
>    http://redhat.com/~mingo/realtime-preempt/
> 
> this is mostly a merge from -rc6 to 2.6.16-final, plus some smaller 
> fixes.
> 
> to build a 2.6.16-rt1 tree, the following patches should be applied:
> 
>   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.16.tar.bz2
>   http://redhat.com/~mingo/realtime-preempt/patch-2.6.16-rt1
> 
> 	Ingo

Ingo,

I haven't had a chance to look into it yet but this combination brings
with it a significant latency regression, at least as measured by the
rtc histogram stuff.

-- 
   kr

Re: 2.6.16-rt1

[Ingo Molnar]

* K.R. Foley <kr@cybsft.com> wrote:

> I haven't had a chance to look into it yet but this combination brings 
> with it a significant latency regression, at least as measured by the 
> rtc histogram stuff.

can you also measure it via rtc_wakeup, a'ka:

 chrt -f -p 98 `pidof 'IRQ 8'`
 ./rtc_wakeup -f 8192 -t 100000

? I have just tried it, and -rt4 looks pretty good on the latency front 
(with all debugging disabled).

	Ingo

Re: 2.6.16-rt1

[K.R. Foley]

Ingo Molnar wrote:
> * K.R. Foley <kr@cybsft.com> wrote:
> 
>> I haven't had a chance to look into it yet but this combination brings 
>> with it a significant latency regression, at least as measured by the 
>> rtc histogram stuff.
> 
> can you also measure it via rtc_wakeup, a'ka:
> 
>  chrt -f -p 98 `pidof 'IRQ 8'`
>  ./rtc_wakeup -f 8192 -t 100000
> 
> ? I have just tried it, and -rt4 looks pretty good on the latency front 
> (with all debugging disabled).
> 
> 	Ingo
> 
Sorry I have been onsite and completely buried today. Am running an
initial test on both UP and SMP now with 2.6.16-rt1. UP doesn't look bad
at all. SMP on the other hand doesn't look so good. I will give -rt4 a
spin when these are done.

-- 
   kr

Re: 2.6.16-rt1

[Ingo Molnar]

* K.R. Foley <kr@cybsft.com> wrote:

> Sorry I have been onsite and completely buried today. Am running an 
> initial test on both UP and SMP now with 2.6.16-rt1. UP doesn't look 
> bad at all. SMP on the other hand doesn't look so good. I will give 
> -rt4 a spin when these are done.

thanks for the testing - i'll check SMP too.

	Ingo

Re: 2.6.16-rt1

[K.R. Foley]

Ingo Molnar wrote:
> * K.R. Foley <kr@cybsft.com> wrote:
> 
>> Sorry I have been onsite and completely buried today. Am running an 
>> initial test on both UP and SMP now with 2.6.16-rt1. UP doesn't look 
>> bad at all. SMP on the other hand doesn't look so good. I will give 
>> -rt4 a spin when these are done.
> 
> thanks for the testing - i'll check SMP too.
> 
> 	Ingo
> 
OK. On my dual 933 under heavy load I get the following with 2.6.16-rt4
and I get tons of missed interrupts. Running 2.6.15-rc16 I get a max of
88usec with most falling under 30usec. On my UP AthlonXP 1700 I get a
max of 19usec with 2.6.16-rt4 under load. What sort of results do you
see on SMP?

rtc latency histogram of {rtc_wakeup/7095, 17528666 samples}:
0 85
1 169
2 200
3 348
4 704
5 950
6 661
7 1123
8 1243
9 1217
10 1396
11 1932
12 1640
13 1136
14 839
15 9896
16 128030
17 653034
18 649117
19 980871
20 3100062
21 1568493
22 979036
23 811244
24 959543
25 732209
26 681492
27 856068
28 939870
29 649806
30 449958
31 367307
32 322118
33 238856
34 205575
35 184051
36 148624
37 147437
38 134690
39 126161
40 141023
41 192116
42 227147
43 199154
44 143090
45 101829
46 74295
47 55930
48 44070
49 35757
50 31143
51 28283
52 26081
53 24561
54 21771
55 20275
56 17621
57 14803
58 12567
59 10606
60 9132
61 8132
62 6978
63 5865
64 4751
65 4077
66 3367
67 3008
68 2718
69 2621
70 2504
71 2390
72 2071
73 1699
74 1430
75 1118
76 957
77 774
78 710
79 575
80 560
81 467
82 410
83 372
84 343
85 298
86 263
87 230
88 176
89 188
90 157
91 130
92 107
93 122
94 89
95 64
96 41
97 40
98 51
99 42
100 34
101 33
102 14
103 20
104 17
105 15
106 12
107 6
108 10
109 9
110 8
111 10
112 6
113 7
114 6
115 1
116 7
117 6
118 4
119 4
120 5
121 1
122 1
123 4
124 5
125 1
126 3
127 1
128 3
129 1
130 4
132 6
133 3
134 7
135 2
136 3
137 4
138 5
139 1
140 1
141 2
142 2
143 5
144 1
147 2
148 2
149 1
150 1
151 2
152 1
154 1
155 1
159 1
160 3
163 1
171 1
211 1
212 1
475 1
9187 1
9192 1
9219 1
9231 1
9259 1
9268 1


-- 
   kr

Re: 2.6.16-rt1

[K.R. Foley]

K.R. Foley wrote:
> Ingo Molnar wrote:
>> * K.R. Foley <kr@cybsft.com> wrote:
>>
>>> Sorry I have been onsite and completely buried today. Am running an 
>>> initial test on both UP and SMP now with 2.6.16-rt1. UP doesn't look 
>>> bad at all. SMP on the other hand doesn't look so good. I will give 
>>> -rt4 a spin when these are done.
>> thanks for the testing - i'll check SMP too.
>>
>> 	Ingo
>>
> OK. On my dual 933 under heavy load I get the following with 2.6.16-rt4
> and I get tons of missed interrupts. Running 2.6.15-rc16 I get a max of
> 88usec with most falling under 30usec. On my UP AthlonXP 1700 I get a
> max of 19usec with 2.6.16-rt4 under load. What sort of results do you
> see on SMP?
> 

Found something interesting. Having Wakeup latency timing turned on
makes a HUGE difference. I turned it off and recompiled and now I am
seeing numbers back in line with what I expected from 2.6.16-rt4. Sorry,
but I had no idea it would make that much difference. I don't have a
complete run yet, but I have seen enough to know that I am not seeing
tons of missed interrupts and the highest reported latency thus far is
61 usec.

-- 
   kr

Re: 2.6.16-rt1

[Daniel Walker]

On Wed, 2006-03-22 at 10:10 -0600, K.R. Foley wrote:

> Found something interesting. Having Wakeup latency timing turned on
> makes a HUGE difference. I turned it off and recompiled and now I am
> seeing numbers back in line with what I expected from 2.6.16-rt4. Sorry,
> but I had no idea it would make that much difference. I don't have a
> complete run yet, but I have seen enough to know that I am not seeing
> tons of missed interrupts and the highest reported latency thus far is
> 61 usec.

Just Wakeup latency timing , and not latency tracing ?

Daniel

Re: 2.6.16-rt1

[K.R. Foley]

Daniel Walker wrote:
> On Wed, 2006-03-22 at 10:10 -0600, K.R. Foley wrote:
> 
>> Found something interesting. Having Wakeup latency timing turned on
>> makes a HUGE difference. I turned it off and recompiled and now I am
>> seeing numbers back in line with what I expected from 2.6.16-rt4. Sorry,
>> but I had no idea it would make that much difference. I don't have a
>> complete run yet, but I have seen enough to know that I am not seeing
>> tons of missed interrupts and the highest reported latency thus far is
>> 61 usec.
> 
> Just Wakeup latency timing , and not latency tracing ?
> 
> Daniel
> 
> 

Unfortunately I can't say for sure because I don't remember turning it
off, but looking at the log it appears as though latency tracing was
turned on

-- 
   kr

Re: 2.6.16-rt1

[Steven Rostedt]

On Wed, 2006-03-22 at 10:10 -0600, K.R. Foley wrote:
> K.R. Foley wrote:
> > Ingo Molnar wrote:
> >> * K.R. Foley <kr@cybsft.com> wrote:
> >>
> >>> Sorry I have been onsite and completely buried today. Am running an 
> >>> initial test on both UP and SMP now with 2.6.16-rt1. UP doesn't look 
> >>> bad at all. SMP on the other hand doesn't look so good. I will give 
> >>> -rt4 a spin when these are done.
> >> thanks for the testing - i'll check SMP too.
> >>
> >> 	Ingo
> >>
> > OK. On my dual 933 under heavy load I get the following with 2.6.16-rt4
> > and I get tons of missed interrupts. Running 2.6.15-rc16 I get a max of
> > 88usec with most falling under 30usec. On my UP AthlonXP 1700 I get a
> > max of 19usec with 2.6.16-rt4 under load. What sort of results do you
> > see on SMP?
> > 
> 
> Found something interesting. Having Wakeup latency timing turned on
> makes a HUGE difference. I turned it off and recompiled and now I am
> seeing numbers back in line with what I expected from 2.6.16-rt4. Sorry,
> but I had no idea it would make that much difference. I don't have a
> complete run yet, but I have seen enough to know that I am not seeing
> tons of missed interrupts and the highest reported latency thus far is
> 61 usec.

Hmm, high wake up latency on SMP and not on UP...

Ingo, could this be due to the migrate task latency?  This was where I
saw the problem with the 50ms latency running hack bench.  I remember
there was a bug in the older latency tool that didn't catch this latency
before.

I'm just getting back to looking at the latest stuff.  I had some
customer deliveries lately and haven't had time to look at the new
goodies.

-- Steve

Re: 2.6.16-rt1

[Serge Noiraud]

lundi 20 Mars 2006 09:51, Ingo Molnar wrote/a écrit :
> i have released the 2.6.16-rt1 tree, which can be downloaded from the 
> usual place:
> 
>    http://redhat.com/~mingo/realtime-preempt/
> 
> this is mostly a merge from -rc6 to 2.6.16-final, plus some smaller 
> fixes.

I get the following error :
...
Kernel: arch/i386/boot/bzImage is ready  (#2)
  Building modules, stage 2.
  MODPOST
*** Warning: "mutex_destroy" [fs/xfs/xfs.ko] undefined!
*** Warning: "there_is_no_init_MUTEX_LOCKED_for_RT_semaphores" [fs/xfs/xfs.ko] undefined!
...
Is it a BUG ?

-- 
Serge Noiraud

Re: 2.6.16-rt1

[Jan Altenberg]

IIRC, I've seen this error already in a post for 2.6.16-rc6-rt1

> *** Warning: "mutex_destroy" [fs/xfs/xfs.ko] undefined!

mutex_destroy isn't defined for CONFIG_PREEMPT_RT.

In the past mutex_destroy was defined in fs/xfs/linux_26/mutex.h, but
this has been moved. Have a look at: include/linux/mutex-debug.h,
kernel/mutex-debug.c, include/linux/mutex.h


JAN

Re: 2.6.16-rt1

[Ingo Molnar]

* Serge Noiraud <serge.noiraud@bull.net> wrote:

> I get the following error :
> ...
> Kernel: arch/i386/boot/bzImage is ready  (#2)
>   Building modules, stage 2.
>   MODPOST
> *** Warning: "mutex_destroy" [fs/xfs/xfs.ko] undefined!
> *** Warning: "there_is_no_init_MUTEX_LOCKED_for_RT_semaphores" [fs/xfs/xfs.ko] undefined!
> ...
> Is it a BUG ?

could you check -rt2?

	Ingo

Re: 2.6.16-rt1

[Michal Piotrowski]

Hi Ingo,

On 21/03/06, Ingo Molnar <mingo@elte.hu> wrote:
>
> could you check -rt2?
>

Here is first oops http://www.stardust.webpages.pl/files/rt/2.6.16-rt2/b1.jpg
Here is second oops http://www.stardust.webpages.pl/files/rt/2.6.16-rt2/b2.jpg

Here is config http://www.stardust.webpages.pl/files/rt/2.6.16-rt2/rt-config

I can't boot that kernel, 2.6.16-rt1 was fine.

Regards,
Michal

--
Michal K. K. Piotrowski
LTG - Linux Testers Group
(http://www.stardust.webpages.pl/ltg/wiki/)

Re: 2.6.16-rt1

[Ingo Molnar]

* Michal Piotrowski <michal.k.k.piotrowski@gmail.com> wrote:

> Hi Ingo,
> 
> On 21/03/06, Ingo Molnar <mingo@elte.hu> wrote:
> >
> > could you check -rt2?
> >
> 
> Here is first oops http://www.stardust.webpages.pl/files/rt/2.6.16-rt2/b1.jpg
> Here is second oops http://www.stardust.webpages.pl/files/rt/2.6.16-rt2/b2.jpg
> 
> Here is config http://www.stardust.webpages.pl/files/rt/2.6.16-rt2/rt-config
> 
> I can't boot that kernel, 2.6.16-rt1 was fine.

ok, i broke irqs-off latency tracing in -rt2. Could you try -rt3 - it 
should be fixed there.

	Ingo

Re: 2.6.16-rt1

[Michal Piotrowski]

On 21/03/06, Ingo Molnar <mingo@elte.hu> wrote:
>
> * Michal Piotrowski <michal.k.k.piotrowski@gmail.com> wrote:
>
> > Hi Ingo,
> >
> > On 21/03/06, Ingo Molnar <mingo@elte.hu> wrote:
> > >
> > > could you check -rt2?
> > >
> >
> > Here is first oops http://www.stardust.webpages.pl/files/rt/2.6.16-rt2/b1.jpg
> > Here is second oops http://www.stardust.webpages.pl/files/rt/2.6.16-rt2/b2.jpg
> >
> > Here is config http://www.stardust.webpages.pl/files/rt/2.6.16-rt2/rt-config
> >
> > I can't boot that kernel, 2.6.16-rt1 was fine.
>
> ok, i broke irqs-off latency tracing in -rt2. Could you try -rt3 - it
> should be fixed there.
>
>         Ingo
>

Thanks!
Problem fixed.

Regards,
Michal

--
Michal K. K. Piotrowski
LTG - Linux Testers Group
(http://www.stardust.webpages.pl/ltg/wiki/)

Re: 2.6.16-rt1

[Sébastien Dugué]

  Hi,

  when testing 2.6.16-rt1, I noticed schedule_hrtimer() disappeared from
hrtimer.c. I was exporting and using it for a custom application so I had
to fall back to reimplementing the same functionality where it was needed.

  On a more general note, wouldn't it be usefull to have that kind of
function available to the rest of the kernel?

  Sébastien.

-- 
-----------------------------------------------------

  Sébastien Dugué                BULL/FREC:B1-247
  phone: (+33) 476 29 77 70      Bullcom: 229-7770

  mailto:sebastien.dugue@bull.net

  Linux POSIX AIO: http://www.bullopensource.org/posix
                   http://sourceforge.net/projects/paiol

-----------------------------------------------------