[patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Ingo Molnar]

i have released the -V0.7.32-0 Real-Time Preemption patch, which can be
downloaded from the usual place:

	http://redhat.com/~mingo/realtime-preempt/

this is a fixes-mostly release with one new feature:

implemented global RT-task balancing on SMP systems, which improves the
latency of RT tasks on SMP systems. The basic problem was that the 2.6
kernel has per-CPU runqueues. In the current design there is no
guarantee that if an RT task starves another, lower-prio (or same-prio)
RT task in a given local runqueue, that the starved task will ever be
migrated to another CPU: it has to wait for the higher-prio task to
finish. In short, task migration on SMP is fundamentally non-RT and
priority-insensitive. In particular the workloads and latencies reported
by Mark H. Johnson reflect such SMP scheduling artifacts.

the new global RT-task balancing feature solves this problem by tracking
the 'RT overload' situation (when there is more than one RT tasks on a
CPU) and makes other CPUs 'pull' RT tasks (and only RT tasks)
immediately when such a situation occurs.

To give an example, in the following scheduling scenario:

  CPU#0					CPU#1

  task-A SCHED_FIFO prio 30		task-C SCHED_FIFO prio 30 [curr]
  task-B SCHED_FIFO prio 40 [curr]

task-B is the currently executing task on CPU#0, task-C is the currently
executing task on CPU#1. Now on the vanilla 2.6 kernel, if task-C
blocks, there's no guarantee that task-A will be run there - if there's
a SCHED_NORMAL task on CPU#1's runqueue then it will run indefinitely. 
With global RT-balancing task-A will be scheduled on CPU#1 immediately
after task-C leaves it.

furthermore, if in the same scenario, if e.g. a RT-prio 35 task-D is
woken up on CPU#0, the vanilla 2.6 scheduler will not move it to CPU#1,
even though it could preempt the prio 30 task-C there. With global
RT-balancing this will happen and task-C will be preempted immediately
and task-D runs.

on low RT load (the common case) the scheduler behaves like the stock
scheduler - the new logic only kicks in if a CPU runqueue has 2 or more
RT tasks running at once.

anyway, while the feature is stable on my SMP testboxes, this is still a
nontrivial ~200-lines delta in the scheduler so there might be problems. 
UP should not be affected by this.

other changes since -V0.7.32-20:

 - local-APIC shutdown fix: this should solve some of the 'reboot hangs' 
   reports.

 - more tracing fixes - might fix the 'truncated traces' problems.

 - reduce the NMI watchdog frequency from 10 KHz to 1000 Hz.

 - dont report futex reschedules as atomicity violations

to create a -V0.7.32-0 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm2/2.6.10-rc2-mm2.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm2-V0.7.32-0

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Ingo Molnar]

> to create a -V0.7.32-0 tree from scratch, the patching order is:

that should read:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm3/2.6.10-rc2-mm3.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm3-V0.7.32-0

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Mark_H_Johnson]

Comparison of .32-0 and .31-15 results


      within 100 usec
       CPU loop (%)   Elapsed Time (sec)    2.4
Test   32-0  31-15     32-0   31-15  |   CPU  Elapsed
X     94.58  99.22      67      68   |  97.20   70
top   95.29  97.96      39      34   |  97.48   29
neto  94.24  99.98     360     360   |  96.23   36
neti  94.83  98.31     360     350   |  95.86   41
diskw 90.77  99.57     360     360 * |  77.64   29
diskc 93.47  97.49     360     360   |  84.12   77
diskr 93.49  98.35     320     180   |  90.66   86
total                 1866    1712   |         368
* wide variation in audio duration

Grr. This appears that .32-0 is MUCH WORSE than 31.15 at
keeping the relatively small (100 usec) latencies down.
Probably due to the CPU task switch that prevents the
longer ones from occurring. The MAX CPU latencies are down
in most cases (over 4 msec down to about 3 msec) which
is a good result.

The long elapsed times appear to indicate that we are
starving the "stress test" application (and likely
running the niced, non RT cpu_burn instead).

--Mark H Johnson
  <mailto:Mark_H_Johnson@raytheon.com>

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Rui Nuno Capela]

Ingo Molnar wrote:
>
> i have released the -V0.7.32-0 Real-Time Preemption patch, which can be
> downloaded from the usual place:
>

I have a bug to report, it shows on both of my machines (SMP and UP) now
running RT-V0.7.32-2. It seems to be present also on previous RT releases,
and don't even know if it's upstream.

When one usb-storage flash stick is first time unplugged:

usb 4-6: USB disconnect, address 4
slab error in kmem_cache_destroy(): cache `scsi_cmd_cache': Can't free all
objects
 [<c010361f>] dump_stack+0x23/0x25 (20)
 [<c01467d7>] kmem_cache_destroy+0x103/0x1aa (28)
 [<c026e77a>] scsi_destroy_command_freelist+0x97/0xa8 (28)
 [<c026f5b1>] scsi_host_dev_release+0x37/0xe1 (104)
 [<c023c6a9>] device_release+0x7a/0x7c (32)
 [<c01efd90>] kobject_cleanup+0x87/0x89 (28)
 [<c01f07eb>] kref_put+0x52/0xef (40)
 [<c01efdcc>] kobject_put+0x25/0x27 (16)
 [<f8cf1843>] usb_stor_release_resources+0x66/0xca [usb_storage] (16)
 [<f8cf1d93>] storage_disconnect+0x8e/0x9b [usb_storage] (16)
 [<f89ca117>] usb_unbind_interface+0x84/0x86 [usbcore] (28)
 [<c023d915>] device_release_driver+0x75/0x77 (28)
 [<c023db18>] bus_remove_device+0x53/0x82 (20)
 [<c023cae1>] device_del+0x4b/0x9b (20)
 [<f89d142a>] usb_disable_device+0xf5/0x10a [usbcore] (28)
 [<f89cc61c>] usb_disconnect+0xc8/0x164 [usbcore] (40)
 [<f89cd77e>] hub_port_connect_change+0x2ef/0x426 [usbcore] (56)
 [<f89cda7b>] hub_events+0x1c6/0x39d [usbcore] (56)
 [<f89cdc89>] hub_thread+0x37/0x109 [usbcore] (96)
 [<c01009b1>] kernel_thread_helper+0x5/0xb (142893076)


Then, just when the same is plugged in again:

usb 4-6: new high speed USB device using ehci_hcd and address 5
kmem_cache_create: duplicate cache scsi_cmd_cache
BUG at mm/slab.c:1447!
------------[ cut here ]------------
kernel BUG at mm/slab.c:1447!
invalid operand: 0000 [#1]
PREEMPT SMP
Modules linked in: nls_utf8 nls_cp860 vfat fat nls_base usb_storage
ohci1394 ieee1394 ehci_hcd usbhid uhci_hcd intel_mch_agp agpgart evdev
wacom snd_pcm_oss snd_mixer_oss snd_seq_midi snd_seq_midi_event snd_seq
snd_usb_usx2y snd_usb_lib snd_hwdep snd_cs46xx snd_rawmidi snd_seq_device
snd_intel8x0 snd_ac97_codec snd_pcm snd_timer snd soundcore snd_page_alloc
realtime commoncap w83781d i2c_sensor i2c_isa i2c_i801 i2c_core sk98lin
subfs dm_mod usbcore
CPU:    1
EIP:    0060:[<c01462e9>]    Not tainted VLI
EFLAGS: 00010286   (2.6.10-rc2-mm3-RT-V0.7.32-2.0smp)
EIP is at kmem_cache_create+0x4df/0x69c
eax: 00000017   ebx: f7cd9710   ecx: c05c5d00   edx: f77b8000
esi: c030973a   edi: c030973a   ebp: f77b9d10   esp: f77b9cd8
ds: 007b   es: 007b   ss: 0068   preempt: 00000001
Process khubd (pid: 1477, threadinfo=f77b8000 task=f777c2b0)
Stack: c02f2054 c02f63aa 000005a7 00000565 f77b9d00 f7cd10c0 c0000000
ffffff80
       00000001 f7cd1080 00000080 c03418a0 f7000400 f7000428 f77b9d3c
c026e662
       c030972b 00000180 00000080 00002000 00000000 00000000 f7000490
00000000
Call Trace:
 [<c01035f4>] show_stack+0xb4/0xbc (28)
 [<c010378a>] show_registers+0x169/0x1de (56)
 [<c010399c>] die+0x10b/0x191 (68)
 [<c0103e19>] do_invalid_op+0xba/0xc4 (204)
 [<c0103237>] error_code+0x2b/0x30 (116)
 [<c026e662>] scsi_setup_command_freelist+0x94/0x115 (44)
 [<c026f8ca>] scsi_host_alloc+0x26f/0x399 (124)
 [<f8cf173f>] usb_stor_acquire_resources+0x6c/0x10a [usb_storage] (28)
 [<f8cf1c64>] storage_probe+0x1b8/0x259 [usb_storage] (40)
 [<f89ca07a>] usb_probe_interface+0x62/0x7b [usbcore] (28)
 [<c023d723>] driver_probe_device+0x2f/0x70 (24)
 [<c023d7ae>] device_attach+0x4a/0xa5 (40)
 [<c023da89>] bus_add_device+0x4d/0x89 (28)
 [<c023c9a6>] device_add+0xb1/0x137 (40)
 [<f89d1ba6>] usb_set_configuration+0x3ca/0x444 [usbcore] (84)
 [<f89cc822>] usb_new_device+0xa5/0x164 [usbcore] (44)
 [<f89cd6b0>] hub_port_connect_change+0x221/0x426 [usbcore] (56)
 [<f89cda7b>] hub_events+0x1c6/0x39d [usbcore] (56)
 [<f89cdc89>] hub_thread+0x37/0x109 [usbcore] (96)
 [<c01009b1>] kernel_thread_helper+0x5/0xb (142893076)
Code: e8 57 5e fd ff b8 08 4f be c0 e8 c0 d1 fe ff c7 44 24 08 a7 05 00 00
c7 44 24 04 aa 63 2f c0 c7 04 24 54 20 2f c0 e8 31 5e fd ff <0f> 0b a7 05
aa 63 2f c0 eb b9 a1 44 1a 33 c0 c7 44 24 04 d0 00


After this, any USB related operation is doomed, if not the entire system.

Bye.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Ingo Molnar]

* Rui Nuno Capela <rncbc@rncbc.org> wrote:

> Ingo Molnar wrote:
> >
> > i have released the -V0.7.32-0 Real-Time Preemption patch, which can be
> > downloaded from the usual place:
> >
> 
> I have a bug to report, it shows on both of my machines (SMP and UP)
> now running RT-V0.7.32-2. It seems to be present also on previous RT
> releases, and don't even know if it's upstream.
> 
> When one usb-storage flash stick is first time unplugged:

hm, doesnt seem to be directly related to -RT. Could you try the vanilla
-rc2-mm3 kernel, does it trigger there too?

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Rui Nuno Capela]

Ingo Molnar wrote:
>
> Rui Nuno Capela wrote:
>
>> Ingo Molnar wrote:
>> >
>> > i have released the -V0.7.32-0 Real-Time Preemption patch, which can
>> > be downloaded from the usual place:
>> >
>>
>> I have a bug to report, it shows on both of my machines (SMP and UP)
>> now running RT-V0.7.32-2. It seems to be present also on previous RT
>> releases, and don't even know if it's upstream.
>>
>> When one usb-storage flash stick is first time unplugged:
>
> hm, doesnt seem to be directly related to -RT. Could you try the vanilla
> -rc2-mm3 kernel, does it trigger there too?
>

No, it doesn't. Just tested on 2.6.10-rc2-mm3 (vanilla) and nothing wrong
happens. I can plug and unplug the USB flashram stick, and again, and
again, and everything stays fine.

On 2.6.10-rc2-mm3-RT-V0.7.32-2 I get trouble after the first time I unplug
the thingie. From then on I rather not touch it back again ;)

Take care.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[K.R. Foley]

Ingo Molnar wrote:
> * Rui Nuno Capela <rncbc@rncbc.org> wrote:
> 
> 
>>Ingo Molnar wrote:
>>
>>>i have released the -V0.7.32-0 Real-Time Preemption patch, which can be
>>>downloaded from the usual place:
>>>
>>
>>I have a bug to report, it shows on both of my machines (SMP and UP)
>>now running RT-V0.7.32-2. It seems to be present also on previous RT
>>releases, and don't even know if it's upstream.
>>
>>When one usb-storage flash stick is first time unplugged:
> 
> 
> hm, doesnt seem to be directly related to -RT. Could you try the vanilla
> -rc2-mm3 kernel, does it trigger there too?
> 
> 	Ingo
> 

Gene Heskett reported a very similar problem yesterday with the subject: 
  Re:2.6.10-rc2-mm3-V0.7.31-19

kr

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Gene Heskett]

On Saturday 04 December 2004 17:32, Rui Nuno Capela wrote:
>Ingo Molnar wrote:
>> i have released the -V0.7.32-0 Real-Time Preemption patch, which
>> can be downloaded from the usual place:

I can confirm essentially this same bug, this rather voluminous output 
took place after I turned off my camera and unplugged the usb cable 
from it.  At that point all other usb stuff was spotty, even the 
mouse till I rebooted.

>I have a bug to report, it shows on both of my machines (SMP and UP)
> now running RT-V0.7.32-2. It seems to be present also on previous
> RT releases, and don't even know if it's upstream.
>
>When one usb-storage flash stick is first time unplugged:
>
>usb 4-6: USB disconnect, address 4
>slab error in kmem_cache_destroy(): cache `scsi_cmd_cache': Can't
> free all objects
> [<c010361f>] dump_stack+0x23/0x25 (20)
> [<c01467d7>] kmem_cache_destroy+0x103/0x1aa (28)
> [<c026e77a>] scsi_destroy_command_freelist+0x97/0xa8 (28)
> [<c026f5b1>] scsi_host_dev_release+0x37/0xe1 (104)
> [<c023c6a9>] device_release+0x7a/0x7c (32)
> [<c01efd90>] kobject_cleanup+0x87/0x89 (28)
> [<c01f07eb>] kref_put+0x52/0xef (40)
> [<c01efdcc>] kobject_put+0x25/0x27 (16)
> [<f8cf1843>] usb_stor_release_resources+0x66/0xca [usb_storage]
> (16) [<f8cf1d93>] storage_disconnect+0x8e/0x9b [usb_storage] (16)
> [<f89ca117>] usb_unbind_interface+0x84/0x86 [usbcore] (28)
> [<c023d915>] device_release_driver+0x75/0x77 (28)
> [<c023db18>] bus_remove_device+0x53/0x82 (20)
> [<c023cae1>] device_del+0x4b/0x9b (20)
> [<f89d142a>] usb_disable_device+0xf5/0x10a [usbcore] (28)
> [<f89cc61c>] usb_disconnect+0xc8/0x164 [usbcore] (40)
> [<f89cd77e>] hub_port_connect_change+0x2ef/0x426 [usbcore] (56)
> [<f89cda7b>] hub_events+0x1c6/0x39d [usbcore] (56)
> [<f89cdc89>] hub_thread+0x37/0x109 [usbcore] (96)
> [<c01009b1>] kernel_thread_helper+0x5/0xb (142893076)
>
>
>Then, just when the same is plugged in again:
>
>usb 4-6: new high speed USB device using ehci_hcd and address 5
>kmem_cache_create: duplicate cache scsi_cmd_cache
>BUG at mm/slab.c:1447!
>------------[ cut here ]------------
>kernel BUG at mm/slab.c:1447!
>invalid operand: 0000 [#1]
>PREEMPT SMP
>Modules linked in: nls_utf8 nls_cp860 vfat fat nls_base usb_storage
>ohci1394 ieee1394 ehci_hcd usbhid uhci_hcd intel_mch_agp agpgart
> evdev wacom snd_pcm_oss snd_mixer_oss snd_seq_midi
> snd_seq_midi_event snd_seq snd_usb_usx2y snd_usb_lib snd_hwdep
> snd_cs46xx snd_rawmidi snd_seq_device snd_intel8x0 snd_ac97_codec
> snd_pcm snd_timer snd soundcore snd_page_alloc realtime commoncap
> w83781d i2c_sensor i2c_isa i2c_i801 i2c_core sk98lin subfs dm_mod
> usbcore
>CPU:    1
>EIP:    0060:[<c01462e9>]    Not tainted VLI
>EFLAGS: 00010286   (2.6.10-rc2-mm3-RT-V0.7.32-2.0smp)
>EIP is at kmem_cache_create+0x4df/0x69c
>eax: 00000017   ebx: f7cd9710   ecx: c05c5d00   edx: f77b8000
>esi: c030973a   edi: c030973a   ebp: f77b9d10   esp: f77b9cd8
>ds: 007b   es: 007b   ss: 0068   preempt: 00000001
>Process khubd (pid: 1477, threadinfo=f77b8000 task=f777c2b0)
>Stack: c02f2054 c02f63aa 000005a7 00000565 f77b9d00 f7cd10c0
> c0000000 ffffff80
>       00000001 f7cd1080 00000080 c03418a0 f7000400 f7000428
> f77b9d3c c026e662
>       c030972b 00000180 00000080 00002000 00000000 00000000
> f7000490 00000000
>Call Trace:
> [<c01035f4>] show_stack+0xb4/0xbc (28)
> [<c010378a>] show_registers+0x169/0x1de (56)
> [<c010399c>] die+0x10b/0x191 (68)
> [<c0103e19>] do_invalid_op+0xba/0xc4 (204)
> [<c0103237>] error_code+0x2b/0x30 (116)
> [<c026e662>] scsi_setup_command_freelist+0x94/0x115 (44)
> [<c026f8ca>] scsi_host_alloc+0x26f/0x399 (124)
> [<f8cf173f>] usb_stor_acquire_resources+0x6c/0x10a [usb_storage]
> (28) [<f8cf1c64>] storage_probe+0x1b8/0x259 [usb_storage] (40)
> [<f89ca07a>] usb_probe_interface+0x62/0x7b [usbcore] (28)
> [<c023d723>] driver_probe_device+0x2f/0x70 (24)
> [<c023d7ae>] device_attach+0x4a/0xa5 (40)
> [<c023da89>] bus_add_device+0x4d/0x89 (28)
> [<c023c9a6>] device_add+0xb1/0x137 (40)
> [<f89d1ba6>] usb_set_configuration+0x3ca/0x444 [usbcore] (84)
> [<f89cc822>] usb_new_device+0xa5/0x164 [usbcore] (44)
> [<f89cd6b0>] hub_port_connect_change+0x221/0x426 [usbcore] (56)
> [<f89cda7b>] hub_events+0x1c6/0x39d [usbcore] (56)
> [<f89cdc89>] hub_thread+0x37/0x109 [usbcore] (96)
> [<c01009b1>] kernel_thread_helper+0x5/0xb (142893076)
>Code: e8 57 5e fd ff b8 08 4f be c0 e8 c0 d1 fe ff c7 44 24 08 a7 05
> 00 00 c7 44 24 04 aa 63 2f c0 c7 04 24 54 20 2f c0 e8 31 5e fd ff
> <0f> 0b a7 05 aa 63 2f c0 eb b9 a1 44 1a 33 c0 c7 44 24 04 d0 00
>
>
>After this, any USB related operation is doomed, if not the entire
> system.
>
>Bye.

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.30% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2004 by Maurice Eugene Heskett, all rights reserved.

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Gene Heskett]

On Saturday 04 December 2004 18:55, K.R. Foley wrote:
>Ingo Molnar wrote:
>> * Rui Nuno Capela <rncbc@rncbc.org> wrote:
>>>Ingo Molnar wrote:
>>>>i have released the -V0.7.32-0 Real-Time Preemption patch, which
>>>> can be downloaded from the usual place:
>>>
>>>I have a bug to report, it shows on both of my machines (SMP and
>>> UP) now running RT-V0.7.32-2. It seems to be present also on
>>> previous RT releases, and don't even know if it's upstream.
>>>
>>>When one usb-storage flash stick is first time unplugged:
>>
>> hm, doesnt seem to be directly related to -RT. Could you try the
>> vanilla -rc2-mm3 kernel, does it trigger there too?
>>
>>  Ingo
>
>Gene Heskett reported a very similar problem yesterday with the
> subject: Re:2.6.10-rc2-mm3-V0.7.31-19

Yes, and I can confirm that it does not do it right now, running
the UP version of 2.6.10-rc3.  I just tried it and this is all
I got in the log:
-----------------
Dec  4 22:06:38 coyote kernel: usb 3-2.2: new full speed USB device using ohci_hcd and address 7
Dec  4 22:06:38 coyote kernel: scsi0 : SCSI emulation for USB Mass Storage devices
Dec  4 22:06:44 coyote kernel:   Vendor: OLYMPUS   Model: C-3020ZOOM(U)     Rev: 1.00
Dec  4 22:06:44 coyote kernel:   Type:   Direct-Access                      ANSI SCSI revision: 02
Dec  4 22:06:44 coyote kernel: Attached scsi generic sg0 at scsi0, channel 0, id 0, lun 0,  type 0
Dec  4 22:06:44 coyote scsi.agent[26069]: disk at /devices/pci0000:00/0000:00:02.1/usb3/3-2/3-2.2/3-2.2:1.0/host0/target0:0:0/0:0:0:0
Dec  4 22:06:44 coyote kernel: SCSI device sda: 128000 512-byte hdwr sectors (66 MB)
Dec  4 22:06:44 coyote kernel: sda: assuming Write Enabled
Dec  4 22:06:44 coyote kernel: sda: assuming drive cache: write through
Dec  4 22:06:44 coyote kernel: SCSI device sda: 128000 512-byte hdwr sectors (66 MB)
Dec  4 22:06:44 coyote kernel: sda: assuming Write Enabled
Dec  4 22:06:44 coyote kernel: sda: assuming drive cache: write through
Dec  4 22:06:44 coyote kernel:  sda: sda1
Dec  4 22:06:44 coyote kernel: Attached scsi removable disk sda at scsi0, channel 0, id 0, lun 0
Dec  4 22:06:51 coyote kernel: usb 3-2.2: USB disconnect, address 7
-----------------
-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.30% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2004 by Maurice Eugene Heskett, all rights reserved.

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Esben Nielsen]

On Fri, 3 Dec 2004, Ingo Molnar wrote:

> 
> [...] 
> on low RT load (the common case) 

In the system I deal with on my day job, 50% of the CPU load is from
RT tasks!

In fact, I think that is quite normal for industrial applications. Those
tend to be PLC-like: You run your scan every 10 ms. First you read your
input, then you do your calculations then you set your output. You do
the same thing every time. You can thus safely tick around at a CPU
load close to 100%!!

This is quite opposit to how telecom real-time systems works: Those are
usually event based. I.e. they process things when they occur. The load
can vary a lot depending on how much traffic has to go through the device.

Esben

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Ingo Molnar]

* Esben Nielsen <simlo@phys.au.dk> wrote:

> On Fri, 3 Dec 2004, Ingo Molnar wrote:
> 
> > 
> > [...] 
> > on low RT load (the common case) 
> 
> In the system I deal with on my day job, 50% of the CPU load is from
> RT tasks!

i'm not sure what point you are trying to make, but 'low RT load' in
this context means up to a load of 1.0. (i.e. one or less tasks running
on average)

also, this only applies to SMP systems.

thirdly, even if the new balancing code kicks in, the overhead is not
that bad, and it's for a purpose: we rather want an RT task to run on a
free CPU.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Mark_H_Johnson]

>* Esben Nielsen <simlo@phys.au.dk> wrote:
>
>> On Fri, 3 Dec 2004, Ingo Molnar wrote:
>>
>> >
>> > [...]
>> > on low RT load (the common case)
>>
>> In the system I deal with on my day job, 50% of the CPU load is from
>> RT tasks!
>
>i'm not sure what point you are trying to make, but 'low RT load' in
>this context means up to a load of 1.0. (i.e. one or less tasks running
>on average)
I am not quite so sure that is a good assumption with our real time
system either. We use RT priorities based on execution rate with a
single (highest priority) RT task that wakes up the others at the
appropriate times. So an 80 Hz task has a higher RT priority than a
40 Hz task, both are greater than 10 Hz, and all greater than 1 Hz.
We do have a few other tasks for signal handling and other purposes
but most fit the above pattern.

At periodic rate (e.g., once per second) we basically start ALL RT tasks
and let the kernel figure out which one (two actually) should run first.
At that point in time, I may have 10-20 RT tasks ready to run with only
two CPU's to run them on. As the high priority tasks get done, the load
starts to drop, but a one Hz task may get delayed for some time and
interrupted several times during execution (later cycles of higher rate
RT tasks). FYI - I just checked and on our cluster head node, we have
31 RT tasks on a two CPU system.

With the real time load up, the one minute load average on 2.4 systems
oscillates in an odd fashion. For example, looking at 1 hour chart of
load average on my cluster shows the head node oscillating between zero
and five, even though the CPU usage is constant at 17.4%. I assume this
is a sampling error.

If the algorithm is OK for "small numbers" where small is under ten
times the number of CPU's, that is probably OK. [I'll allow that
SOMEONE will try to run a big RT simulation on a 256 CPU SMP machine
and complain later]

>also, this only applies to SMP systems.
Yes.

>thirdly, even if the new balancing code kicks in, the overhead is not
>that bad, and it's for a purpose: we rather want an RT task to run on a
>free CPU.
I agree though I wish it was possible to determine if the interrupt
preemption was "short" or "long" to decide to switch or not [the switch
DOES have costs...]. I believe this is why the max latency of my CPU
task went down (by about 1 msec) but the number of short latencies
(> 100 usec) went up.

But I am still seeing starvation of the non RT tasks that was not
present in 2.4. It appears that cpu_burn (non RT, nice) is taking CPU
cycles that should be allocated to my (non RT, not nice) stress test
programs. This distorts the elapsed time of my tests and to some extent
the latency results (since less I/O stress is on the system) of the tests.

--Mark H Johnson
  <mailto:Mark_H_Johnson@raytheon.com>

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Esben Nielsen]

On Mon, 6 Dec 2004, Ingo Molnar wrote:

> 
> * Esben Nielsen <simlo@phys.au.dk> wrote:
> 
> > On Fri, 3 Dec 2004, Ingo Molnar wrote:
> > 
> > > 
> > > [...] 
> > > on low RT load (the common case) 
> > 
> > In the system I deal with on my day job, 50% of the CPU load is from
> > RT tasks!
> 
> i'm not sure what point you are trying to make, but 'low RT load' in
> this context means up to a load of 1.0. (i.e. one or less tasks running
> on average)
>

The point I want to make is that RT tasks is not always of the type
which reacts to some interrupt, does a small job and then goes to
sleep. Sometimes you also want larger jobs done realtime. Forinstance an
application waking up every 10ms, running for 5ms before going to sleep
again. That ofcourse gives you a RT-load <= 1.0.

In more complicated systems you can have a task running every 5ms,
another at every 25ms and another at every 100ms. That gives you a
RT-load of up to 3! And it is a even common case because you try to
syncronize the tasks, i.e. you wake all 3 tasks up at the same time but
then you know that on your typical UP system they will be executed by
priority, i.e. first the 5ms one, then the 25ms one and last the 100ms.

So my point was: Low RT-load might not be the common case on specific
systems. It is on the desktop where you want to run some multimedia
multimedia task(s) and have fast CPU compared to the job you
need done in RT but in embedded devices you do not pick such a big CPU and
you often use a large part on it in you RT control-loops.
 
> also, this only applies to SMP systems.
> 
> thirdly, even if the new balancing code kicks in, the overhead is not
> that bad, and it's for a purpose: we rather want an RT task to run on a
> free CPU.
> 

Good to hear. I was going to ask if the overhead wasn't too big :-)

However, I do have a feeling that it is rather expensive to move tasks
around. If I was going to design a real-time application on a 2 CPU SMP
system, I think I would CPU lock my RT-tasks to make the system more
deterministic. I.e. I would probably "port" the above mentioned
application by putting the 5 ms task on one CPU, the 25ms on the 
other and the 100 ms on the whatever CPU has the less CPU load. Then the
other non-real-time tasks in the system can wander around as they please.

But that is ofcourse not something you can do unless you know the specific
hardware system you will run your application on.  That is not the case
when you make multimedia software for the PC. I.e. in that case moving
RT-tasks around makes perfect sense!


> 	Ingo
> 

Esben

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Ingo Molnar]

* Esben Nielsen <simlo@phys.au.dk> wrote:

> So my point was: Low RT-load might not be the common case on specific
> systems. [...]

i did not suggest it was. The reason why i mentioned it was to point out
that _non-RT usage_ does not see any overhead, i.e. ordinary Linux
boxes. (which nevertheless do run RT tasks occasionally.)

of course in the RT-specific it can be common - Mark's test is one such
workload. If it werent widespread i'd not try to solve the problem...

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Ingo Molnar]

* Mark_H_Johnson@raytheon.com <Mark_H_Johnson@raytheon.com> wrote:

> >i'm not sure what point you are trying to make, but 'low RT load' in
> >this context means up to a load of 1.0. (i.e. one or less tasks running
> >on average)

> I am not quite so sure that is a good assumption with our real time
> system either. [...]

I implemented this feature with your workload in mind. This is what i
wrote:

> > on low RT load (the common case) the scheduler behaves like the
> > stock scheduler - the new logic only kicks in if a CPU runqueue has
> > 2 or more RT tasks running at once.

'the common case' == ordinary (non-RT) Linux boxes! When i implement
scheduler features i'm always trying to make them as generic as
possible, i.e. this feature too is structured to be as upstream
mergeable as possible. For that purpose the change had to be
low-overhead in the common, non-RT case. It is easy to hack the
scheduler to fix some RT issue but break the generic scheduler - this
solution is not meant to be such a hack.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Mark_H_Johnson]

>> > on low RT load (the common case) the scheduler behaves like the
>> > stock scheduler - the new logic only kicks in if a CPU runqueue has
>> > 2 or more RT tasks running at once.
>
>'the common case' == ordinary (non-RT) Linux boxes! When i implement
>scheduler features i'm always trying to make them as generic as
>possible, i.e. this feature too is structured to be as upstream
>mergeable as possible. For that purpose the change had to be
>low-overhead in the common, non-RT case.
I truly appreciate that. However...

>It is easy to hack the
>scheduler to fix some RT issue but break the generic scheduler - this
>solution is not meant to be such a hack.
I agree but I see the big delay of running the RT task to be a symptom
that the current non RT scheduler is somehow broken. I've reported the
non RT starvation condition several times. Yes, the second CPU is busy,
but I really do want to bump cpu_burn (which is non RT & nice) to run my
(non RT and not nice) stress script / commands instead.

--Mark H Johnson
  <mailto:Mark_H_Johnson@raytheon.com>

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

[Ingo Molnar]

* Mark_H_Johnson@raytheon.com <Mark_H_Johnson@raytheon.com> wrote:

> > It is easy to hack the scheduler to fix some RT issue but break the 
> > generic scheduler - this solution is not meant to be such a hack.
>
> I agree but I see the big delay of running the RT task to be a symptom
> that the current non RT scheduler is somehow broken. I've reported the
> non RT starvation condition several times. Yes, the second CPU is
> busy, but I really do want to bump cpu_burn (which is non RT & nice)
> to run my (non RT and not nice) stress script / commands instead.

well ... doing global balancing on SMP gets really expensive, really
quick. It's simply not an option for 4 way or higher, and even on 2-way
it's measurable. In the RT case arguably latency is more important than
throughput, but i dont think the same case could be made for nice vs. 
non-nice.

	Ingo