public inbox for linux-kernel@vger.kernel.org
 help / color / mirror / Atom feed
* Re: [patch] Real-Time Preemption, -RT-2.6.10-rc1-mm3-V0.7.25-1
@ 2004-11-12 18:23 Mark_H_Johnson
  2004-11-12 20:19 ` Shane Shrybman
  0 siblings, 1 reply; 43+ messages in thread
From: Mark_H_Johnson @ 2004-11-12 18:23 UTC (permalink / raw)
  To: Shane Shrybman
  Cc: Amit Shah, Karsten Wiese, Bill Huey, Adam Heath, emann,
	Gunther Persoons, K.R. Foley, linux-kernel, Ingo Molnar,
	Florian Schmidt, Fernando Pablo Lopez-Lezcano, Lee Revell,
	Rui Nuno Capela, Thomas Gleixner, Michal Schmidt

>Typical example of the error message:
>
>kernel: hde: dma_timer_expiry: dma status == 0x24
>kernel: ALSA sound/core/pcm_native.c:1424: playback drain error (DMA or
IRQ trouble?)
>kernel: PDC202XX: Primary channel reset.
>kernel: hde: DMA interrupt recovery
>kernel: hde: lost interrupt
>
>This was on a Promise TX2 133 ide card with one IDE disk. The problem
>would show itself if using the RT patches and APIC. But the problem seems
>to have been resolved now.

I had errors like that one when the IDE IRQ was at a priority less than
the real time task. Since then, I run with all the IRQ's at max RT priority
and will continue to do so until I get a better assessment of what my real
application (not these audio tests...) needs for IRQ priorities.

This may have been fixed as a side effect of Ingo setting the IRQ threads
at
RT priorities in the 40's.

--Mark H Johnson
  <mailto:Mark_H_Johnson@raytheon.com>


^ permalink raw reply	[flat|nested] 43+ messages in thread
* Re: [patch] Real-Time Preemption, -RT-2.6.10-rc1-mm3-V0.7.25-1
@ 2004-11-15 18:40 Mark_H_Johnson
  2004-11-15 19:52 ` Ingo Molnar
  0 siblings, 1 reply; 43+ messages in thread
From: Mark_H_Johnson @ 2004-11-15 18:40 UTC (permalink / raw)
  To: Ingo Molnar
  Cc: linux-kernel, Lee Revell, Rui Nuno Capela, K.R. Foley, Bill Huey,
	Adam Heath, Florian Schmidt, Thomas Gleixner, Michal Schmidt,
	Fernando Pablo Lopez-Lezcano, Karsten Wiese, Gunther Persoons,
	emann, Shane Shrybman, Amit Shah

Didn't see an announcement for V0.7.26, so I'll post this summary
under this title.

Built today with V0.7.26-4 without any problems. System booted up
and telnet 5 was uneventful as well.

Ran two series of tests with latencytest and my stress tests with
the following results:

[1] Appear to have a new symptom of 200 usec delays at raw_read_unlock
which doesn't make any sense to me. Have included a latency trace
at the end of this message with an example. An occasional 100 usec
hit I understand (disk DMA) but I don't recall seeing this symptom
before.

[2] Still get the symptoms with truncated trace output and bad
ping responses. Refer to my previous messages for examples.

[3] The logging script (sleep 5 seconds, record data if slept for
over 10 seconds) was triggered about 30 times in an hour of testing.
None have the huge load average values reported last time but
several have 1 minute load averages above 15 (expect 6-8).

[4] All of the tests have bursts of long application level delays.
I'll be running another test program to see if I can find anything
with the user level tracing. Disk activity seems to make it worse
but all the tests had at least one CPU delay over a millisecond.
There seems to be a "short" (well > 500 usec) delay related to
disk reads and a longer one for disk writes.

[5] System after testing was done had a major "time shift"
as noted in the system log.

Nov 15 12:33:55 dws77 ntpd[2359]: synchronized to 192.52.216.4, stratum=3
Nov 15 12:33:57 dws77 ntpd[2359]: synchronized to 192.52.216.1, stratum=2
Nov 15 12:33:36 dws77 ntpd[2359]: time reset -21.466037 s
Nov 15 12:33:36 dws77 ntpd[2359]: frequency error -512 PPM exceeds
tolerance 500 PPM

No crashes nor any major stability problems.
  --Mark

--- 200 usec latency example ---
preemption latency trace v1.0.7 on 2.6.10-rc1-mm3-RT-V0.7.26-4
-------------------------------------------------------
 latency: 206 us, entries: 12 (12)   |   [VP:0 KP:1 SP:1 HP:1 #CPUS:2]
    -----------------
    | task: kjournald/1209, uid:0 nice:0 policy:0 rt_prio:0
    -----------------
 => started at: __down_mutex+0x3f/0x300 <c032d9af>
 => ended at:   __down_mutex+0x1a6/0x300 <c032db16>
=======>
 1209 80000000 0.000ms (+0.002ms): __down_mutex (__spin_lock)
 1209 80000000 0.002ms (+0.001ms): _raw_spin_lock (__down_mutex)
 1209 80000000 0.004ms (+0.000ms): _raw_spin_lock (__down_mutex)
 1209 80000000 0.004ms (+0.000ms): do_nmi (__down_mutex)
 1209 80000000 0.005ms (+0.000ms): do_nmi (mcount)
 1209 80000000 0.005ms (+0.000ms): do_nmi (<00200286>)
 1209 80000000 0.006ms (+0.000ms): profile_hook (profile_tick)
 1209 80000000 0.006ms (+0.000ms): _raw_read_lock (profile_hook)
 1209 80000000 0.007ms (+0.196ms): _raw_read_unlock (profile_tick)
 1209 80000000 0.204ms (+0.001ms): set_new_owner (__down_mutex)
 1209 80000000 0.205ms (+0.000ms): _raw_spin_unlock (__down_mutex)
 1209 80000000 0.205ms (+0.000ms): _raw_spin_unlock (__down_mutex)


^ permalink raw reply	[flat|nested] 43+ messages in thread
* Re: [patch] Real-Time Preemption, -RT-2.6.10-rc1-mm3-V0.7.25-1
@ 2004-11-12 16:58 Mark_H_Johnson
  2004-11-15 16:46 ` Ingo Molnar
  0 siblings, 1 reply; 43+ messages in thread
From: Mark_H_Johnson @ 2004-11-12 16:58 UTC (permalink / raw)
  To: Ingo Molnar
  Cc: linux-kernel, Lee Revell, Rui Nuno Capela, K.R. Foley, Bill Huey,
	Adam Heath, Florian Schmidt, Thomas Gleixner, Michal Schmidt,
	Fernando Pablo Lopez-Lezcano, Karsten Wiese, Gunther Persoons,
	emann, Shane Shrybman, Amit Shah

>i have released the -V0.7.25-1 Real-Time Preemption patch, which can be
>downloaded from the usual place:
>
>    http://redhat.com/~mingo/realtime-preempt/
>

This builds just fine and runs without any serious failures. The RT
performance still has rough edges with several bursts of long
delays in the CPU loop of latencytest. These tests are all on an
SMP system with CONFIG_PREEMPT_RT and full tracing enabled.

I have sent several log files to Ingo separately but the following
summarizes what I am seeing:

 [1] major network delays while latencytest is running (ping drops packets
or they get delayed by minutes). I did not see this on some previous tests
where I made more of the /0 and /1 tasks RT. May have to do that again.
 [2] display / keyboard / mouse will occasionally freeze or act much more
slowly than on a non PREEMPT_RT kernel.
 [3] I no longer see the major delays in events/0 and /1. This particular
live lock appears to be solved. I actually don't see any tasks or
applications taking a lot of CPU time except cpu_burn and latencytest
(both should). This is a little puzzling since with the poor response
to non RT / interactive activities, I would expect to see something
take up the CPU time.
 [4] the user latency traces show the RT CPU loop in latencytrace being
delayed by a variety of kernel activites. I also note that preempt_count
is zero during several periods of the trace so I'm surprised that we did
not continue to run the RT task (and do this stuff on the other CPU).
These delays have hundreds of entries with over 100 usec delay overall.
 [5] I can make the symptoms MUCH worse by simply running my cpu_burn
application as a non realtime application. (no I/O or system calls, just
a simple loop) This is run as a nice'd application so should run only
when nothing else is ready to run.
 [6] the latency trace may have some SMP race conditions where the entries
displayed do not match the header. Examples are a 100 usec trace header
followed by 8 entries that last about 4 usec.
 [7] both the wakeup and the preempt disable traces do not show any
significant periods of delays. The most I can get out of these is
roughly 100 usec which I believe correlates with disk DMA and the
particular motherboard chip set on the system under test. This
looks really good if [4] can be fixed.
 [8] Some samples of /proc/loadavg during my big test showed some
extremely large numbers. For example:
5.07 402.44 0.58 5/120 4448
6.35 195.67 1.63 7/122 4663
5.39 130.82 2.20 18/122 4705
2.10 43.17 3.00 8/122 5912
8.90 8.89 4.70 10/123 7780
8.33 8.52 4.95 6/124 7887
Not quite sure what a 5 minute loadavg of 402 means when I have
only a 120 tasks in the system. May be a symptom of some bug in
the load average calculations (and not PREEMPT_RT related) but
not sure.

  --Mark


^ permalink raw reply	[flat|nested] 43+ messages in thread
* [patch] Real-Time Preemption, -RT-2.6.9-mm1-U10
@ 2004-10-22 15:50 Ingo Molnar
  2004-10-22 17:56 ` [patch] Real-Time Preemption, -RT-2.6.9-mm1-U10.2 Ingo Molnar
  0 siblings, 1 reply; 43+ messages in thread
From: Ingo Molnar @ 2004-10-22 15:50 UTC (permalink / raw)
  To: linux-kernel
  Cc: Lee Revell, Rui Nuno Capela, Mark_H_Johnson, K.R. Foley,
	Bill Huey, Adam Heath, Florian Schmidt, Thomas Gleixner,
	Michal Schmidt, Fernando Pablo Lopez-Lezcano


i have released the -U10 Real-Time Preemption patch, which can be
downloaded from:
 
  http://redhat.com/~mingo/realtime-preempt/

this is purely a rebasing of -U9.3 to 2.6.9-mm1.

to create a -U10 tree from scratch, the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
 + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9/2.6.9-mm1/2.6.9-mm1.bz2
 + http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-mm1-U10

	Ingo

^ permalink raw reply	[flat|nested] 43+ messages in thread

end of thread, other threads:[~2004-11-16 12:06 UTC | newest]

Thread overview: 43+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2004-11-12 18:23 [patch] Real-Time Preemption, -RT-2.6.10-rc1-mm3-V0.7.25-1 Mark_H_Johnson
2004-11-12 20:19 ` Shane Shrybman
  -- strict thread matches above, loose matches on Subject: below --
2004-11-15 18:40 Mark_H_Johnson
2004-11-15 19:52 ` Ingo Molnar
2004-11-12 16:58 Mark_H_Johnson
2004-11-15 16:46 ` Ingo Molnar
2004-11-15 17:50   ` Ingo Molnar
2004-10-22 15:50 [patch] Real-Time Preemption, -RT-2.6.9-mm1-U10 Ingo Molnar
2004-10-22 17:56 ` [patch] Real-Time Preemption, -RT-2.6.9-mm1-U10.2 Ingo Molnar
2004-10-25 10:40   ` [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0 Ingo Molnar
2004-10-27  0:15     ` [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.3 Ingo Molnar
2004-11-03 10:58       ` [patch] Real-Time Preemption, -RT-2.6.10-rc1-mm2-V0.7.1 Ingo Molnar
2004-11-06 15:57         ` [patch] Real-Time Preemption, -RT-2.6.10-rc1-mm3-V0.7.18 Ingo Molnar
2004-11-08  9:16           ` [patch] Real-Time Preemption, -RT-2.6.10-rc1-mm3-V0.7.19 Ingo Molnar
2004-11-08 16:57             ` [patch] Real-Time Preemption, -RT-2.6.10-rc1-mm3-V0.7.21 Ingo Molnar
2004-11-09 16:05               ` [patch] Real-Time Preemption, -RT-2.6.10-rc1-mm3-V0.7.23 Ingo Molnar
2004-11-11 14:44                 ` [patch] Real-Time Preemption, -RT-2.6.10-rc1-mm3-V0.7.25-0 Ingo Molnar
2004-11-11 21:51                   ` [patch] Real-Time Preemption, -RT-2.6.10-rc1-mm3-V0.7.25-1 Ingo Molnar
2004-11-12  4:08                     ` Bill Huey
2004-11-12  5:03                       ` Bill Huey
2004-11-12  8:39                         ` Ingo Molnar
2004-11-12 10:52                           ` Bill Huey
2004-11-12 14:31                     ` Shane Shrybman
2004-11-12 17:27                       ` K.R. Foley
2004-11-12 17:50                         ` Shane Shrybman
2004-11-12 20:13                       ` Ingo Molnar
2004-11-12 22:15                         ` Shane Shrybman
2004-11-12 23:44                         ` Shane Shrybman
2004-11-14 12:51                           ` Ingo Molnar
2004-11-12 19:48                     ` Gunther Persoons
2004-11-12 20:19                       ` Ingo Molnar
2004-11-13 12:55                         ` Gunther Persoons
2004-11-13 14:36                         ` Gunther Persoons
2004-11-14 12:49                           ` Ingo Molnar
2004-11-14 14:25                             ` Gunther Persoons
2004-11-13 23:12                         ` Gunther Persoons
2004-11-14 12:38                           ` Ingo Molnar
2004-11-14 12:56                     ` Florian Schmidt
2004-11-14 13:26                       ` K.R. Foley
2004-11-14 13:35                         ` Florian Schmidt
2004-11-14 13:56                         ` K.R. Foley
2004-11-14 14:11                         ` Florian Schmidt
2004-11-14 14:15                       ` Ingo Molnar
2004-11-15  1:27                         ` Florian Schmidt
2004-11-15  2:22                           ` K.R. Foley
2004-11-15 15:15                         ` Florian Schmidt
2004-11-15 14:33                     ` Rui Nuno Capela
2004-11-15 15:40                       ` Ingo Molnar
2004-11-15 16:11                       ` Ingo Molnar
2004-11-15 16:52                         ` Rui Nuno Capela
     [not found]                         ` <33583.195.245.190.93.1100537554.squirrel@195.245.190.93>
2004-11-15 22:35                           ` Rui Nuno Capela
2004-11-16 10:41                             ` Ingo Molnar
2004-11-16 12:05                               ` Rui Nuno Capela

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox