[patch] Real-Time Preemption, -RT-2.6.12-rc6-V0.7.47-29

[patch] Real-Time Preemption, -RT-2.6.12-rc6-V0.7.47-29

[Ingo Molnar]

i have released the -V0.7.47-29 Real-Time Preemption patch, which can be 
downloaded from the usual place:

    http://redhat.com/~mingo/realtime-preempt/

it's a fixes and speedups release. Changes since -47-20:

 - x64 fixes (Michal Schmidt)

 - cpufreq fix (Esben Nielsen)

 - CONFIG_RT_DEADLOCK_DETECT build fix (Michal Schmidt)

 - MAX_USER_RT_PRIO fix (Steven Rostedt)

there are more microoptimizations to the spin_lock/unlock hotpath:

 - the caching of mutex_getprio() priority in p->normal_prio

 - the mutex lock/unlock paths are now all fall-through. (Found a gcc
   bug, it mishandles __builtin_expect() in certain cases and produces 
   correct but suboptimal code - we are working it around now.)

 - reduced the amount of recursive preemption-counter bumps via the use 
   of raw spinlocks

 - rely on the preemption-counter instead of IRQs-off sections

These changes brought the PREEMPT_RT overhead significantly down on 
hackbench workloads (clearly a worst-case test for PREEMPT_RT overhead).  

Would be interesting to see what kind of system time overhead PREEMPT_RT 
now causes for e.g. jack_test workloads.

to build a -V0.7.47-29 tree, the following patches should to be applied:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.11.tar.bz2
   http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.12-rc6.bz2
   http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.12-rc6-V0.7.47-29

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc6-V0.7.47-29

[Daniel Walker]

On Tue, 2005-06-07 at 21:41 +0200, Ingo Molnar wrote:
> ere are more microoptimizations to the spin_lock/unlock hotpath:
> 
>  - the caching of mutex_getprio() priority in p->normal_prio
> 
>  - the mutex lock/unlock paths are now all fall-through. (Found a gcc
>    bug, it mishandles __builtin_expect() in certain cases and produces 
>    correct but suboptimal code - we are working it around now.)
> 
>  - reduced the amount of recursive preemption-counter bumps via the use 
>    of raw spinlocks
> 
>  - rely on the preemption-counter instead of IRQs-off sections

There is a local_irq_enable missing someplace in UP ..

BUG: scheduling with irqs disabled: khelper/0x00000000/5
caller is __down_mutex+0x276/0x440
 [<c03a88d6>] __down_mutex+0x276/0x440 (4)
 [<c03a75cc>] schedule+0xec/0x100 (4)
 [<c03a88d6>] __down_mutex+0x276/0x440 (12)
 [<c012b290>] remove_wait_queue+0x10/0x40 (64)
 [<c0130e02>] __spin_lock+0x22/0x40 (32)
 [<c012b290>] remove_wait_queue+0x10/0x40 (8)
 [<c0130e8c>] _spin_lock_irqsave+0xc/0x20 (12)
 [<c012b290>] remove_wait_queue+0x10/0x40 (8)
 [<c012b290>] remove_wait_queue+0x10/0x40 (4)
 [<c0126a2b>] worker_thread+0x14b/0x280 (20)
 [<c01265a0>] __call_usermodehelper+0x0/0x60 (16)
 [<c0112d20>] default_wake_function+0x0/0x40 (28)
 [<c0112d20>] default_wake_function+0x0/0x40 (32)
 [<c01268e0>] worker_thread+0x0/0x280 (36)
 [<c012af24>] kthread+0x84/0xc0 (4)
 [<c012aea0>] kthread+0x0/0xc0 (20)
 [<c010132d>] kernel_thread_helper+0x5/0x18 (16)

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc6-V0.7.47-29

[Ingo Molnar]

* Daniel Walker <dwalker@mvista.com> wrote:

> >  - rely on the preemption-counter instead of IRQs-off sections
> 
> There is a local_irq_enable missing someplace in UP ..
> 
> BUG: scheduling with irqs disabled: khelper/0x00000000/5
> caller is __down_mutex+0x276/0x440
>  [<c03a88d6>] __down_mutex+0x276/0x440 (4)

.config please.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc6-V0.7.47-29

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> > There is a local_irq_enable missing someplace in UP ..
> > 
> > BUG: scheduling with irqs disabled: khelper/0x00000000/5
> > caller is __down_mutex+0x276/0x440
> >  [<c03a88d6>] __down_mutex+0x276/0x440 (4)
> 
> .config please.

ok, managed to reproduce.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc6-V0.7.47-29

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> 
> * Ingo Molnar <mingo@elte.hu> wrote:
> 
> > > There is a local_irq_enable missing someplace in UP ..
> > > 
> > > BUG: scheduling with irqs disabled: khelper/0x00000000/5
> > > caller is __down_mutex+0x276/0x440
> > >  [<c03a88d6>] __down_mutex+0x276/0x440 (4)
> > 
> > .config please.
> 
> ok, managed to reproduce.

ok, fixed it and uploaded the -47-30 patch with the fix.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc6-V0.7.47-29

[Gene Heskett]

On Wednesday 08 June 2005 03:48, Ingo Molnar wrote:
>* Ingo Molnar <mingo@elte.hu> wrote:
>> * Ingo Molnar <mingo@elte.hu> wrote:
>> > > There is a local_irq_enable missing someplace in UP ..
>> > >
>> > > BUG: scheduling with irqs disabled: khelper/0x00000000/5
>> > > caller is __down_mutex+0x276/0x440
>> > >  [<c03a88d6>] __down_mutex+0x276/0x440 (4)
>> >
>> > .config please.
>>
>> ok, managed to reproduce.
>
>ok, fixed it and uploaded the -47-30 patch with the fix.
>
> Ingo

I just built this one, with RT but without the statistical stuff 
enabled.  While rc6 works fine with tvtime-0.99, this gives a no 
video blue screen, audio ok.

Its also found a phantom usb-serial-audio device and fouled up kmix 
with two devices to control.  This causes a slash thru the icon for 
kmix on the toolbar.  The lsusb -v output is many pages in length for 
this mis-identified device as I believe thats the vendor:product for 
a labtec webcam that is plugged in.  100% el-cheapo model.

config, dmesg saved if you'd like to see them.

>-
>To unsubscribe from this list: send the line "unsubscribe
> linux-kernel" in the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.35% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com and AOL/TW attorneys please note, additions to the above
message by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc6-V0.7.47-29

[Ingo Molnar]

* Gene Heskett <gene.heskett@verizon.net> wrote:

> I just built this one, with RT but without the statistical stuff 
> enabled.  While rc6 works fine with tvtime-0.99, this gives a no video 
> blue screen, audio ok.

could you check PREEMPT_DESKTOP too, with softirq/hardirq threading 
enabled/disabled?

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc6-V0.7.47-29

[Gene Heskett]

On Wednesday 08 June 2005 07:59, Ingo Molnar wrote:
>* Gene Heskett <gene.heskett@verizon.net> wrote:
>> I just built this one, with RT but without the statistical stuff
>> enabled.  While rc6 works fine with tvtime-0.99, this gives a no
>> video blue screen, audio ok.
>
>could you check PREEMPT_DESKTOP too, with softirq/hardirq threading
>enabled/disabled?
>
> Ingo

Ok, next build with preempt mode 3, hard & softirq's theaded, as is 
RCU.

Kmix now normal, tvtime still fubar.

Build with mode 2 and it looks as if its all working:

[root@coyote linux-2.6.12-rc6-RT-V0.7.47-30]# grep PREEMPT .config
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT_DESKTOP is not set
# CONFIG_PREEMPT_RT is not set
CONFIG_PREEMPT_SOFTIRQS=y
CONFIG_PREEMPT_HARDIRQS=y
# CONFIG_PREEMPT_BKL is not set

Now, I note that going to the #2 mode (voluntary) turned off threaded 
RCU's, so I'm going to leave that off and try a mode 3 again. BRB.  

And that makes tvtime's video fail with a blue screen, audio ok..

Mode 2, FWIW, makes for quite jerky card motions while playing 
AisleRiot, the solitaire game.

>-
>To unsubscribe from this list: send the line "unsubscribe
> linux-kernel" in the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.35% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com and AOL/TW attorneys please note, additions to the above
message by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc6-V0.7.47-29

[Ingo Molnar]

* Gene Heskett <gene.heskett@verizon.net> wrote:

> [root@coyote linux-2.6.12-rc6-RT-V0.7.47-30]# grep PREEMPT .config
> # CONFIG_PREEMPT_NONE is not set
> CONFIG_PREEMPT_VOLUNTARY=y
> # CONFIG_PREEMPT_DESKTOP is not set
> # CONFIG_PREEMPT_RT is not set
> CONFIG_PREEMPT_SOFTIRQS=y
> CONFIG_PREEMPT_HARDIRQS=y
> # CONFIG_PREEMPT_BKL is not set
> 
> Now, I note that going to the #2 mode (voluntary) turned off threaded 
> RCU's, so I'm going to leave that off and try a mode 3 again. BRB.

please try mode 3 and disable softirq/hardirq threading. If that fixes 
things, could you check which of the two options 
(CONFIG_PREEMPT_SOFTIRQS or CONFIG_PREEMPT_HARDIRQS) causes which type 
of regression?

> And that makes tvtime's video fail with a blue screen, audio ok..
>
> Mode 2, FWIW, makes for quite jerky card motions while playing 
> AisleRiot, the solitaire game.

this doesnt happen on PREEMPT_DESKTOP or PREEMPT_RT?

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc6-V0.7.47-29

[Gene Heskett]

On Wednesday 08 June 2005 10:59, Ingo Molnar wrote:
>* Gene Heskett <gene.heskett@verizon.net> wrote:
>> [root@coyote linux-2.6.12-rc6-RT-V0.7.47-30]# grep PREEMPT .config
>> # CONFIG_PREEMPT_NONE is not set
>> CONFIG_PREEMPT_VOLUNTARY=y
>> # CONFIG_PREEMPT_DESKTOP is not set
>> # CONFIG_PREEMPT_RT is not set
>> CONFIG_PREEMPT_SOFTIRQS=y
>> CONFIG_PREEMPT_HARDIRQS=y
>> # CONFIG_PREEMPT_BKL is not set
>>
>> Now, I note that going to the #2 mode (voluntary) turned off
>> threaded RCU's, so I'm going to leave that off and try a mode 3
>> again. BRB.
>
>please try mode 3 and disable softirq/hardirq threading. If that
> fixes things, could you check which of the two options
>(CONFIG_PREEMPT_SOFTIRQS or CONFIG_PREEMPT_HARDIRQS) causes which
> type of regression?

Ok, they are both off, mode 3 and no RCU threading:

[root@coyote linux-2.6.12-rc6-RT-V0.7.47-30]# grep PREEMPT .config
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
CONFIG_PREEMPT_DESKTOP=y
# CONFIG_PREEMPT_RT is not set
CONFIG_PREEMPT=y
# CONFIG_PREEMPT_SOFTIRQS is not set
# CONFIG_PREEMPT_HARDIRQS is not set
# CONFIG_PREEMPT_RCU is not set
CONFIG_PREEMPT_BKL=y
CONFIG_DEBUG_PREEMPT=y
CONFIG_PREEMPT_TRACE=y
# CONFIG_CRITICAL_PREEMPT_TIMING is not set


And tvtime seems normal.  However, when the video works (and this is a 
known seperate problem with the cx88 driver) then the audio is highly 
'aliased' making for very tinny sound until an internal restart with 
new stds (or 3, sometimes it comes back after a few seconds) is done.  
There is a cx88_video.c patch that went by a few days ago, but even 
thats not 100% "bulletproof" :-)

>> And that makes tvtime's video fail with a blue screen, audio ok..
>>
>> Mode 2, FWIW, makes for quite jerky card motions while playing
>> AisleRiot, the solitaire game.
>
>this doesnt happen on PREEMPT_DESKTOP or PREEMPT_RT?

Its subjectively better, but plain old rc6 is the best in this regard.  
And I note that as I play with the soft & hard irq threading off, in 
mode 3, it improves to be equ to rc6 after about 5 or 6 moves.  Like 
the cfq scheduler is dynamicly adjusting priorities or something.  
Thats my default scheduler for the last 2-3 months, should I be 
retrying the others?

On to make one with softirq threading turned on, BRB.
With this:
[root@coyote linux-2.6.12-rc6-RT-V0.7.47-30]# grep PREEMPT .config
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
CONFIG_PREEMPT_DESKTOP=y
# CONFIG_PREEMPT_RT is not set
CONFIG_PREEMPT=y
CONFIG_PREEMPT_SOFTIRQS=y
# CONFIG_PREEMPT_HARDIRQS is not set
# CONFIG_PREEMPT_RCU is not set
CONFIG_PREEMPT_BKL=y
CONFIG_DEBUG_PREEMPT=y
CONFIG_PREEMPT_TRACE=y
# CONFIG_CRITICAL_PREEMPT_TIMING is not set

tvtime seems normal.  So for the next build, turn on 
CONFIG_PREEMPT_RCU=y for effects since I note that its turned on 
along with hardirq threading by CONFIG_PREEMPT_RT=y.  BRB (well, 15 
minutes by the time I reboot).

Speaking of reboot, I saw a 'failed to contact something' message go 
flying by in the shutdown phase the last time around this loop.

With:
[root@coyote linux-2.6.12-rc6-RT-V0.7.47-30]# grep PREEMPT .config
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
CONFIG_PREEMPT_DESKTOP=y
# CONFIG_PREEMPT_RT is not set
CONFIG_PREEMPT=y
CONFIG_PREEMPT_SOFTIRQS=y
# CONFIG_PREEMPT_HARDIRQS is not set
CONFIG_PREEMPT_RCU=y
CONFIG_PREEMPT_BKL=y
CONFIG_DEBUG_PREEMPT=y
CONFIG_PREEMPT_TRACE=y
# CONFIG_CRITICAL_PREEMPT_TIMING is not set

tvtime works ok with RCU threading on.

But a "service ntpd restart" which failed during the boot, failed the 
2nd time befrore I did the startx, and here's the 3rd restart:
[root@coyote linux-2.6.12-rc6-RT-V0.7.47-30]# service ntpd restart
Shutting down ntpd:                                        [  OK  ]
ntpd: Synchronizing with time server:                      [FAILED]
Starting ntpd:  -U ntp -p /var/run/ntpd.pid -g             [  OK  ]

According to the lights on the router, it pounded on the server about 
10 times at 1 second intervals before it output the failed message.

So we are flushing out some bugs here.  The missus just came back, 
she's been out to the sawbones with her emphesema so my playtime 
today is about used up.  Many thanks for the quick responses.
>
> Ingo

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.35% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com and AOL/TW attorneys please note, additions to the above
message by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.

Re: [patch] Real-Time Preemption, -RT-2.6.12-rc6-V0.7.47-29

[Daniel Walker]

On Wed, 8 Jun 2005, Ingo Molnar wrote:

> > > .config please.
> > 
> > ok, managed to reproduce.
> 
> ok, fixed it and uploaded the -47-30 patch with the fix.

I'm glad cause I don't think I have the .config any more ..


Daniel