Attempted summary of "RT patch acceptance" thread

Attempted summary of "RT patch acceptance" thread

[Paul E. McKenney]

Hello!

Midway through the recent "RT patch acceptance" thread, someone mentioned
that it might be good to summarize the various approaches.  The following
is an attempt to do just this, with an eye to providing a reasonable
framework for future discussion.

Thoughts?  Errors?  Omissions?

[Quickly donning the asbestos suit with tungsten pinstripes...]

						Thanx, Paul

------------------------------------------------------------------------

CONTENTS

A.	INTRODUCTION
B.	DESIRABLE PROPERTIES
C.	LINUX REALTIME APPROACHES
D.	SUMMARY

Search for a line beginning with the corresponding capital letter followed
by a period to jump to the corresponding section.


A.  INTRODUCTION

Common wisdom dictates that realtime operating systems, particularly
hard-realtime operating systems, must be designed from ground up; that
serious realtime support cannot be simply grafted onto an existing
general-purpose operating system.  Although this common wisdom was
not arrived at lightly, it is often worthwhile to look for important
exceptions to this sort of general rule of thumb.  Candidate exceptions
include:

1.	Many realtime applications use a very restricted subset of
	the services provided by a general-purpose OS like Linux.
	Some applications require realtime support only for scheduling
	user-mode code, for example, an application that directly accesses
	MMIO registers mapped into its address space.  This observation
	leads to the possibility of providing very limited realtime
	support.

2.	Computer performance and capacity has increased dramatically
	over the past few decades, quite literally by multiple orders
	of magnitude.  A small embedded system can easily be much more
	capable than a mid-70s supercomputer, for example, the vaunted
	Cray-1, introduced in 1976, ran at 160MFLOPs and sported 8MB of
	main memory.  In today's terms, this would be a modest embedded
	system -- and just you try running Linux on an 8MB system!
	This dramatic increase in performance permits some applications
	that would have required heavy-duty RTOS support in the 70s to
	run reasonably well on unmodified general-purpose OSes.

There are still limits to the degree of realtime support that one can
expect from a general-purpose OS -- there are some extremely demanding
applications that can be satisfied only by hand-coded assembly running
on bare metal.  In fact, there are applications that can be satisfied
only by custom hardware implementations.

Nevertheless, it is clear that Linux can support significant realtime
requirements, as it is already being used heavily in the realtime arena.
But how far should Linux extend its realtime support, and what is the
best way to extend Linux in this direction?  Can one approach to realtime
satisfy all reasonable requirements, or would it be better to support
multiple approaches, each with its area of applicability?

The answers to these questions are not yet clear, and have been the
subject of much spirited discussion, for example, see the more than
300 messages in the following LKML thread:

	http://lkml.org/lkml/2005/5/23/156
	http://marc.theaimsgroup.com/?l=linux-kernel&m=111689227213061&w=2

This document looks at some strategies that have been proposed for
realtime Linux, comparing and contrasting their capabilities.  But, to
evaluate these strategies, it is first necessary to determine what exactly
one might want in a realtime Linux.  If you would rather skip straight to
the comparing and contrasting, search for "LINUX REALTIME APPROACHES".


B.  DESIRABLE PROPERTIES

As usual, there are conflicting desires, at least they conflict given
the current state of the art.  These desires fall into the following
categories:

1.	Quality of service
2.	Amount of code that must be inspected to assure quality of service
3.	API provided
4.	Relative complexity of OS and applications
5.	Fault isolation: what non-RT failures endanger RT code?
6.	What hardware and software configurations are supported?

Each of these categories is expanded upon below, and later used to compare
a number of proposed realtime approaches for Linux.  The discussion does
go for some time, which is not surprising given that it is summarizing
many hundreds of email messages.  ;-)  Search for the corresponding
number at the beginning of a line to skip directly to the discussion of
a given category.


1.  Quality of Service

The traditional view is that the entire operating system is either
hard realtime, soft realtime, or non-realtime, but this viewpoint
is too coarse grained.  Different workloads have different needs,
and there is disagreement over the exact definitions of these three
categories of realtime.  For example, (at least) the following two
definitions of "hard realtime" are in use:

a.	In absence of hardware failures, software provably meets
	the specified deadlines.  This is fine and good, but many
	applications simply do not need this "diamond hard" realtime.

b.	Failure to meet the specified deadline results in application
	failure.  This is OK, but -only- if there is a corresponding
	required probability of success.  Otherwise, one could claim
	"hard realtime" by simply failing the application every time it
	tries to do anything, which is clearly not useful.

A better approach is to simply specified the required probability
of meeting the specified deadline in absence of hardware failure.
A probability of 1.0 is consistent with definition (a).  Other
applications will be satisfied with a probability such as 0.999999,
which might be sufficiently high that the probability of software
scheduling failure is "in the noise" compared with the probability
of hardware failure.  A recent LKML thread called this "metal hard"
realtime.  Or was it "ruby hard"?  ;-)

Of course, one can increase the reliability of hardware through
redundancy, but no hardware configuration provides perfect reliability.
For example, clusters can increase reliability, so that the probability
of failure of the cluster is p^n, where "p" is the probability of a
single node failing and "n" is the number of nodes.  Note that this
expression never reaches a probability of 1, no matter how large "n" is.
In addition, this mathematical expression assumes that the failover
software is perfectly reliable and perfectly configured.  This assumption
conflicts sharply with my own experience, in which there has always been
a point beyond which adding nodes -decreased- cluster reliability.

The timeframe is also critically important.  Any system can provide
hard realtime guarantees if the deadline is an infinite amount of time
in the future.  No computer system that I am aware of at this writing
is capable of meeting a 1-picosecond scheduling deadline for any task
of non-zero duration, but then neither can dedicated digital hardware.
Some applications have definite response-time goals, for example,
industrial process-control applications tend to have response-time
goals ranging from 100s of microseconds to small numbers of seconds.
Other applications can benefit from any improvement in response-time
goals -- faster is better, think in terms of Doom players -- but even
in these cases there is normally a point of diminishing returns.

The services used by the realtime application also figure in.  Given
current disk technology, it is not possible to meet a 100-microsecond
deadline for a 1MB synchronous write to disk.  Not even if you cheat and
supply the disk with a battery-backed-up DRAM.  However, many realtime
applications need only a few of the services that an operating system
might provide.  This list might include interrupt handling, process
scheduling, disk I/O, network I/O, process creation/destruction, VM
operations, and so on.  Keep in mind that many popular RTOSes provide
very little in the way of services!  They frequently leave the
complex stuff (e.g., web serving) to general-purpose operating systems.

Note that each service can have an associated deadline that it can
meet.  The interrupt system might be able to meet a 1-microsecond
deadline, the real-time process scheduler a 10-microsecond deadline,
the disk I/O system a 10-millisecond deadline for moderate-sized
I/Os, and so on.  The deadline that a service can meet might also
depend on the parameters, so that the disk-I/O system would be
expected to take longer for larger I/Os.

Furthermore, the probability might vary from service to service or with
the parameters to that service.  For example, the probability of network
I/O completing successfully in minimal time might well be a function
of the number of packets transmitted (to account for the probability of
packet loss) as well as of packet size (to account for bit-error rate).
To make things even more complicated, the probability of meeting the
deadline will vary depending on the length of time allowed.  Considering
the networking example, a very short deadline might not allow the data
transmission to complete, even if it proceeds at wire speed.  A longer
deadline might allow transmission to complete, but only if there are
no transmission errors.  An even longer deadline might allow time for
a limited number of retransmissions, in order to recover from packet
loss due to transmission errors.  Of course, a deadline infinitely far
into the future would allow guaranteed completion, but I for one am not
that patient.

Finally, the performance and scalability of both realtime and non-realtime
applications running on the system can be important.  Given the current
state of the art, one must pay a performance penalty for realtime support,
but the smaller the penalty, the better.

So, to sum up, here are the components of a quality-of-service metric
for realtime OSes:

a.	List of services for which realtime response is supported.

b.	For each service:

	i.	Probability of missing a deadline due to software,
		ranging from 0 to 1, with the value of 1 corresponding
		to the hardest possible hard realtime.

	ii.	Allowable deadline, measured from the time that
		the request is initiated to the time by which the
		response must be received.

c.	Performance and scalability provided to both realtime and
	non-realtime applications.


2.  Amount of Code Inspection Required

So you add a new feature to a realtime operating system.  How much of
the rest of the system must you inspect and understand in order to be
able to guarantee that your new feature provides the required level
of realtime response?  The smaller this amount of code, the easier it
is to add new features and fix bugs, and the greater the number of
people who will be able to contribute to the project.  In addition,
the smaller the amount of such code, the smaller the probability that
some well-intentioned bug fix will break realtime response.

Each of the following categories of code might need to be inspected:

a.	The low-level interrupt-handing code.

b.	The realtime process scheduler.

c.	Any code that disables interrupts.

d.	Any code that disables preemption.

e.	Any code that holds a lock, mutex, semaphore, or other resource
	that is needed by the code implementing your new feature.

Of course, use of automated tools could make such inspection much
more reliable and less onerous, but such tools would need to deal with
the very large number of CPU architectures and configuration options
that Linux supports.  The smaller the amount of code that must be
inspected, the less chance there is that such a tool will fall victim
to configuration-architecture combinatorial explosion.

Each of Linux realtime approaches uses a different strategy to
minimize the amount of code in these categories.  These differences
are surprisingly important, and will be discussed in more detail
when going over the various approaches to Linux realtime.


3.  API Provided

I never have learned to -really- like the POSIX API, with the gets()
primitive being a particular cause of heartburn, but given the huge
amount of software out there that relies on it and the equally huge
number of developers who are familiar with it, one should certainly
strive to provide it, or at least a sizeable subset of it.

Other popular APIs include the various Java runtime environments,
and of course the feared and loathed, but quite ubiquitous, Windows
API.

There are a lot of developers and a lot of software out there.  The
more of these existing developers and software your API supports,
the more successful your realtime facility is likely to be.


4.  Relative Complexity

How much realtime capability should be added to the operating system?
How much of this burden should the applications take on?  Is it better
to push some of the complexity into a nanokernel, hypervisor, or other
software or firmware layer?  Let's first look at the tradeoff between
OS and application.

For example, although it is certainly possible to program for separate
realtime and non-realtime operating-system instances, doing so adds
complexity to the application.  Complexity is particularly deadly in the
hard realtime arena, and can be literally so if human lives are at risk.

Balancing this consideration is the need for simplicity in the
operating-system kernel.  This balancing act must be carefully considered,
taking both the relative complexities and the number of uses into
account.  Some would argue that it is worthwhile adding 1,000 lines
to the OS if that saves 100 lines in each of 1,000 applications.
Others would disagree, perhaps citing the greater fault isolation
that might be provided by the separation.

But this balance clearly must be struck somewhere between writing the
application to bare metal on the one hand (but achieving a perfectly
simple zero-size operating system) and bloating the operating system
beyond the limits of maintainability on the other hand.

Similar arguments can be made for moving some functionality into a
hypervisor or nanokernel layer, though fault isolation also comes
into play here.

Many of the most vociferous arguments seem to revolve around this
complexity issue.


5.  Fault Isolation

Can a programming error in a non-realtime application or in a non-realtime
portion of the OS harm a realtime application?

Some applications do not care: in these cases, a failure anywhere
causes a user-visible failure, so it is not important to isolate
faults.  Of course, even in these cases, it may be valuable to isolate
faults in order to aid debugging, but, other than that, the fault
isolation does not help overall application reliability.

In other cases, the realtime portion of the application is protecting
someone's life and limb, but the non-realtime portion is only compiling
statistics and reports.  In this case, fault isolation can be of the
utmost importance.

What sorts of faults need isolating?

o	Excessive disabling of interrupts.

o	Excessive disabling of preemption.

o	Holding a lock, mutex, or semaphore for too long, when that
	resource must be acquired by realtime code.

o	Memory corruption, either via wild pointers or via wild DMA.

These faults might occur in the main kernel, in a loadable module, or in
some debugging tool, such as a kprobe procedure or a kernel-debugger
breakpoint script.  Though in the latter case, perhaps realtime
deadlines should not be guaranteed when actively debugging.  After all,
straightforward debugging techniques, such as use of kprint(), can cause
response-time problems even in non-realtime environments.


6.  Hardware and Software Configurations

Is SMP required?  If so, how many CPUs?  How many tasks?  How many
disks?  How many HBAs?

If all the code in the kernel were O(1), it might not matter, but the
Linux kernel has not yet reached this goal.  Therefore, some applications
may choose to restrict the software or the hardware configuration of
the platform in order to meet the realtime deadlines.  This approach is
consistent with traditional RTOS methodology -- RTOS vendors have been
known to restrict the configurations in which they will support hard
realtime guarantees.


C.  LINUX REALTIME APPROACHES

The following general approaches to Linux realtime have been proposed,
along with many variations on each of these themes:

1.	non-CONFIG_PREEMPT
2.	CONFIG_PREEMPT
3.	CONFIG_PREEMPT_RT
4.	Nested OS
5.	Dual-OS/Dual-Core
6.	Migration Between OSes
7.	Migration Within OS

Each of these general approaches is discussed in the following sections.
Each section ends with a brief (but perhaps controversial) summary of
the corresponding approach's strengths and weaknesses.  I do not address
"strength of community", even though this may well be the decisive factor.
After all, the technical comparision will provide sufficient flame-bait.

This document does not present measured comparisons among all of the
approaches, despite the fact that such comparisons would be extremely
useful.  The reason for this, aside from gross laziness, is that it is
wise to agree on the metrics beforehand.  Therefore, the comparisons
in this document are for the most part qualitative.  In some cases,
they are based on actual measurements, but these measurements were
taken by different people on different configurations using different
benchmarks.  This is a prime area for future improvement.


1.  non-CONFIG_PREEMPT

This is the stock kernel, not even using preemption.  Why would -anyone-
think of using stock 2.6 for a realtime task?  Because some realtime
applications have very forgiving scheduling deadlines.  One project
I worked on in the early 1980s had 2-second response-time deadlines.
This was quite a challenge, given that it was running on a 4MHz Z80 CPU --
though, to be fair, the Z80 was accompanied by a hardware floating-point
processor that was able to compute a 32-bit floating-point multiply in
well under a millisecond.  Modern hardware running a stock Linux 2.6 
kernel would have no problem with this application.  Hey, just having
32 address bits rather than only 16 would help a lot!

a.	Quality of service: soft realtime, with timeframe of 10s of
	milliseconds for most services.  Some I/O requests can take
	longer.  Provides full performance and scalability to both
	realtime and non-realtime applications.

b.	Amount of code that must be inspected to assure quality of service
	for a new feature: the entire kernel, every little bit of it,
	since the entire kernel runs with preemption disabled.

c.	API provided: POSIX with limited realtime extensions.
	Realtime and non-realtime applications can interact using
	the normal POSIX services.

d.	Relative complexity of OS and applications: everything is
	stock, and all the normal system calls operate as expected.

e.	Fault isolation: none.

f.	Hardware and software configurations supported: all of them.
	Larger hardware configurations and some device drivers can
	result in degraded response time.

Strengths:  Simplicity and robustness.  "Good enough" realtime support
	for undemanding realtime applications.  Excellent performance
	and scalability for both realtime and non-realtime applications.
	Applications and administrators see a single OS instance.

Weaknesses:  Poor realtime response, need to inspect the entire kernel
	to find issues that degrade realtime response.


2.  CONFIG_PREEMPT

The CONFIG_PREEMPT option renders much of the kernel code preemptible,
with the exception of spinlock critical sections, RCU read-side critical
sections, code with interrupts disabled, code that accesses per-CPU
variables, and other code that explicitly disables preemption.

a.	Quality of service: soft realtime, with timeframe of 100s of
	microseconds for task scheduling and interrupt handling.
	System services providing I/O, networking, task creation, and VM
	manipulation can take much longer.  A very small performance
	penalty is exacted, since spinlocks and RCU must suppress
	preemption.

b.	Amount of code that must be inspected to assure quality of service
	for a new feature:

	i.	The low-level interrupt-handing code.

	ii.	The process scheduler.

	iii.	Any code that disables interrupts, which includes
		all interrupt handlers, both hardware and softirq.

	iv.	Any code that disables preemption, including spinlock
		critical sections, RCU read-side critical sections,
		code with interrupts disabled, code that accesses
		per-CPU variables, and other code that explicitly
		disables preemption.

	v.	Any code that holds a lock, mutex, semaphore, or other
		resource that is needed by the code implementing the
		new feature.

c.	API provided: POSIX with limited realtime extensions.

d.	Relative complexity of OS and applications: all the normal system
	calls operate as expected, so realtime and non-realtime processes
	can interact normally.

e.	Fault isolation: none.

f.	Hardware and software configurations supported: all of them.
	Larger hardware configurations and some device drivers can
	result in degraded response time.

Strengths:  Simplicity.  Available now, even from distributions.
	Provides "good enough" realtime support for a large number
	of applications.  Applications and administrators see a
	single OS instance.

Weaknesses:  Limited testing, so that some robustness issues remain.
	Need to inspect large portions of the kernel in order
	to find issues that degrade realtime response.


3.  CONFIG_PREEMPT_RT

The CONFIG_PREEMPT_RT patch by Ingo Molnar introduces additional
preemption, allowing most spinlock (now "mutexes") critical sections,
RCU read-side critical sections, and interrupt handlers to be preempted.
Preemption of spinlock critical sections requires that priority
inheritance be added to prevent the "priority inversion" problem where
a low-priority task holding a lock is preempted by a medium-priority
task, while a high-priority task is blocked waiting on the lock.
The CONFIG_PREEMPT_RT patch addresses this via "priority inheritance",
where a task waiting on a lock "donates" its priority to the task holding
that lock, but only until it releases the lock.  In the example above,
the low-priority task would run at high priority until it released the
lock, preempting the medium-priority task, so that the high-priority
task gets the lock in a timely fashion.  Priority inheritance has been
used in a number of realtime OS environments over the past few decades,
so it is a well-tested concept.

One problem with priority inheritance is that it is difficult to implement
for reader-writer locks, where a high-priority writer might wish to
donate its high priority to a large number of low-priority readers.
The CONFIG_PREEMPT_RT patch addresses this by allowing only one task at
a time to read-acquire a reader-writer lock, although it is permitted
to do so recursively.

Note that a few critical spinlocks remain non-preemptible, using the
"raw spinlock" implementation.

a.	Quality of service: soft realtime, with timeframe of about 10
	microseconds for task scheduling and interrupt-handler entry.
	System services providing I/O, networking, task creation,
	and VM manipulation can take much longer.  Since spinlocks are
	replaced by blocking mutexes, some believe that the performance
	penalty is significant.  There is likely to be some performance
	penalty exacted from RCU, but, with luck, this penalty will be
	minimal.

b.	Amount of code that must be inspected to assure quality of service
	by a new feature:

	i.	The low-level interrupt-handing code.

	ii.	The process scheduler.

	iii.	Any code that disables interrupts, but -not- including
		interrupt handlers, which now run in process context.

	iv.	Any code that disables preemption, including raw-spinlock
		critical sections, code with interrupts disabled, code
		that accesses per-CPU variables, and other code that
		explicitly disables preemption.

	v.	Any code that holds a lock, mutex, semaphore, or other resource
		that is needed by the code implementing the new feature.

c.	API provided: POSIX with limited realtime extensions.

d.	Relative complexity of OS and applications: all the normal system
	calls operate as expected, so realtime and non-realtime processes
	can interact normally.

e.	Fault isolation: none.

f.	Hardware and software configurations supported: most of them.
	SMP support is a bit rough, and a number of drivers have not yet
	been upgraded to work properly in the CONFIG_PREEMPT_RT environment.
	It is likely that larger hardware configurations and some device
	drivers can result in degraded scheduling latency, but given that
	normal spinlocks are now preemptible, this effect should be much
	less of an issue than for CONFIG_PREEMPT.

Strengths:  Excellent scheduling latencies, potential for hard
	realtime for some services (e.g., user-mode execution) in
	some configurations.  A number of aspects of this approach
	might be incrementally added to Linux (e.g., priority
	inheritance for semaphores to prevent semaphore priority
	inversion).  Applications and administrators see a single
	OS instance.

Weaknesses:  Limited testing, so that robustness issues remain.
	Large patch to Linux (~18K lines of context diff).
	Both realtime and non-realtime applications pay performance
	and scalability penalties for the realtime service.


4.  Nested OS

The Linux instance runs as a user process in an enclosing RTOS.  Realtime
service is provided by the RTOS, and a richer set of non-realtime services
is provided by the Linux instance.

a.	Quality of service: hard realtime, with timeframe of about 10
	microseconds for services provided by the underlying RTOS.
	More complex services (I/O, task creation, and so on) will
	likely take longer to execute, which may impose a significant
	performance and scalability penalty.

b.	Amount of code that must be inspected to assure quality of service
	by a new feature:

	i.	All of the RTOS.  One would strive to keep the RTOS
		quite small, the greater the number of realtime services
		provided, the larger the RTOS must be.

	ii.	Any Linux-kernel code that disables interrupts.
		Note that in many implementations, the Linux kernel
		will be prevented from disabling interrupts, since
		any attempt to disable interrupts will trap into
		the RTOS.

		If the Linux kernel runs in privileged mode, however,
		all bets are off.  In this case, special care must be
		used to avoid disabling the real hardware interrupts,
		including such disabling within any kernel modules
		that might be loaded.

c.	API provided: Whatever the RTOS wants to provide, often a
	subset of POSIX with realtime extensions.

d.	Relative complexity of OS and applications: there are now two
	operating systems, both of which must be configured and
	administered.  Applications that contain both realtime and
	non-realtime components must be explicitly aware of both OS
	instances, and of their respective APIs.

e.	Fault isolation: the following faults may propagate from the
	Linux OS to the underlying RTOS, or not, depending on the
	implementation:

	i.	Excessive disabling of interrupts, if the Linux instance
		is permitted to disable them (hopefully not).

	ii.	Memory corruption, if the Linux instance is given direct
		access to the hardware MMU or to DMA-capable I/O devices.

f.	Hardware and software configurations supported: depends on
	the implementation, however, there are products with this
	architecture that support SMP and a reasonable variety of devices.
	Note that supporting a large variety of devices either requires
	that this support be present in the RTOS, or that Linux be
	granted access to the devices.  In the latter case, Linux will
	likely have the ability to DMA over the top of the RTOS.

Strengths:  Excellent scheduling latencies.  Hard-realtime support for
	some services in some configurations.  Reasonable fault isolation
	for some implementations.  Well-tested and robust implementations
	are available.

Weaknesses:  Realtime application software must deal with two separate
	OS instances and their respective APIs, with explicit
	communication.	Administrators must deal with two OS instances.
	Non-realtime applications are likely to suffer significant
	performance and scalability penalties.  The pair of cores
	will be more expensive than a single core, though one might
	use virtualization to emulate the two CPUs.


5.  Dual-OS/Dual-Core

Linux and RTOS instances run side-by-side on different CPUs in the
same system.  The CPUs might be different physical CPUs, different
hardware threads in the same CPU, or different virtual CPUs provided by
a virtualizing layer, such as Xen.  The two instances might or might not share
memory, and, if they do share memory, there might or might not be hardware
protection to prevent one OS from overwriting the other OS's memory.

a.	Quality of service: hard realtime, with timeframe of about 10
	microseconds for services provided by the RTOS.  More complex
	services (I/O, task creation, and so on) will likely take longer
	to execute.  Since the Linux instance runs on a separate core,
	there need not be any performance or scalability penalty for
	non-realtime tasks.

b.	Amount of code that must be inspected to assure quality of service
	by a new feature: all of the RTOS, but only the RTOS.  One would
	strive to keep the RTOS quite small, but the greater the number
	of realtime services provided, the larger the RTOS must be.

c.	API provided: Whatever the RTOS wants to provide, often a
	subset of POSIX with realtime extensions.

d.	Relative complexity of OS and applications: there are now two
	operating systems, both of which must be configured and
	administered.  Applications that contain both realtime and
	non-realtime components must be explicitly aware of both
	OS instances and APIs, and must also be aware of whatever
	hardware facility is used to communicate between the realtime
	and non-realtime CPUs.

e.	Fault isolation: the following faults may propagate from the
	Linux OS to the underlying RTOS, or not, depending on the
	implementation:

	i.	Memory corruption, but only if the Linux instance is given
		direct access to the RTOS's memory or to DMA-capable
		I/O devices that can access the RTOS's memory.

f.	Hardware and software configurations supported: depends on
	the implementation, however, there are products based on this
	approach that support SMP and a reasonable variety of devices.

Strengths:  Best possible scheduling latencies with the hardest reasonable
	realtime -- just as good as bare metal in some implementations.
	Best possible fault isolation for some implementations.
	Well-tested and robust implementations are available.
	Linux can be used as is, so full performance and scalability
	can be provided to non-realtime tasks.

Weaknesses:  Realtime application software must deal with two separate
	OS instances, with explicit communication.  Administrators must
	deal with two OS instances.  "RTOSes" that provide the best
	latencies offer the least services -- in extreme cases, the only
	service is execution of raw code on bare metal.


6.  Migration Between OSes

A Linux and RTOS instance run side-by-side in the same system.  The two
OSes might run on different physical CPUs, different hardware threads
in the same CPU, different virtual CPUs provided by a virtualizing
layer like Xen, or alternatively, the two OSes might use some sort
of interrupt-pipeline scheme (such as Adeos) to share a single CPU.

However, applications see a single unified environment.  Applications
run on the RTOS, but the RTOS provides Linux-compatible system calls and
memory layout.  If the application invokes a non-realtime system call,
the task is transparently migrated to the Linux OS instance for the
duration of that system call.  This differs from the other dual-OS
approaches, where the applications must be explicitly aware of the
different OSes.

At this writing, it appears that the two instances need to share memory,
since tasks can migrate from one OS to the other.

a.	Quality of service: hard realtime, with timeframe of about 10
	microseconds for services provided by the RTOS.  More complex
	services (I/O, task creation, and so on) will likely take longer
	to execute.  It is also possible for tasks to be "trapped"
	in the Linux instance, for example, if they are sleeping, but
	have not yet been given a chance to respond to some event that
	should wake them up.  The performance and scalability penalties
	to non-realtime tasks can be expected to depend on the amount
	of protection provided for realtime tasks against non-realtime
	misbehavior -- the greater the protection, the greater the
	expected penalty.  It may be possible to provide hardware
	support to improve this tradeoff.

b.	Amount of code that must be inspected to assure quality of service
	by a new feature:

	i.	All of the RTOS.  One would strive to keep the RTOS
		quite small, but the greater the number of realtime
		services provided, the larger the RTOS must be.

	ii.	Any Linux-kernel code that disables interrupts.
		Note that in many implementations, the Linux kernel
		will be prevented from disabling interrupts, since
		any attempt to disable interrupts will trap into
		the RTOS or into the underlying software/firmware
		layer (e.g., Xen or Adeos).

		If the Linux kernel runs in privileged mode, however,
		all bets are off.  In this case, special care must be
		used to avoid disabling the real hardware interrupts,
		including such disabling within any kernel modules
		that might be loaded.

c.	API provided: Full POSIX with realtime extensions.  Anytime
	a task running in the context of the RTOS attempts to execute
	a non-realtime system call, it is migrated to the Linux instance.

d.	Relative complexity of OS and applications: there are now two
	operating systems, both of which must be configured and
	administered.  However, applications can be written as if
	there was only one OS instance that provided the full set
	of services, some realtime and some not.

e.	Fault isolation: the following faults may propagate from the
	Linux OS to the underlying RTOS, or not, depending on the
	implementation:

	i	Excessive disabling of interrupts, if the Linux OS
		is permitted to disable hardware interrupts (hopefully
		not, though preventing this may require special hardware).

	ii.	Memory corruption, either due to wild pointer or
		via wild DMA.

f.	Hardware and software configurations supported: depends on
	the implementation, however, it is reasonable to believe that
	SMP and a reasonable variety of devices could be supported.
	Note that supporting a large variety of devices either requires
	that this support be present in the RTOS, or that Linux be
	granted access to the devices.  In the latter case, Linux will
	likely have the ability to DMA over the RTOS.

Strengths:  Excellent scheduling latencies.  Hard-realtime support for
	some services in some configurations.  Applications see a
	single OS.

Weaknesses:  Administrators must deal with two OS instances.
	The two OSes will be extremely sensitive to each other's
	version and patch level, since they access each other's
	data structures.


7.  Migration Within OS

A Linux instance runs on multiple CPUs, either different physical CPUs,
different hardware threads in the same CPU, or different virtual CPUs
provided by a virtualizing layer such as Xen.  Some (but not all!) of
the CPUs are designated as realtime CPUs.  If a task running on a
realtime CPU executes a trap or system call that contains non-deterministic
code sequences, the task is migrated to a non-realtime CPU to complete
execution of the trap or system call, then migrated back.  This prevents
any non-realtime execution of a given realtime task from interfering
with that of other realtime tasks.

Interrupts can be directed away from realtime CPUs.  Such interrupt
redirection is supported on a few architectures, and has in fact been
used for realtime support since at least the 2.4 kernel.

a.	Quality of service: unknown at present.  This approach is
	expected to provide hard realtime with timeframe of about 10
	microseconds for services provided on the realtime CPUs.  More
	complex services (I/O, task creation, and so on) will likely take
	longer to execute.  It is also possible for tasks to be "trapped"
	on the non-realtime CPUs, for example, if they are sleeping,
	but have not yet been given a chance to respond to some event
	that should wake them up.  Since a stock non-CONFIG_PREEMPT Linux
	may be used, there need be no performance or scalability penalty
	for non-realtime tasks, nor for realtime tasks that execute only
	realtime operations.  There can be a significant migration penalty
	when realtime tasks frequently execute non-realtime operations.

b.	Amount of code that must be inspected to assure quality of service
	by a new feature:

	i.	Any part of the Linux kernel that is permitted to execute
		on the realtime CPUs.  This would normally be only the
		realtime portions of the scheduler and the low-level
		interrupt and trap handling code (the actual interrupts
		and traps would be migrated, if necessary).

	ii.	Any critical section of any lock acquired by the portion
		of the Linux kernel that is permitted to execute on the
		realtime CPUs.

c.	API provided: Full POSIX with realtime extensions.

d.	Relative complexity of OS and applications: There is but
	one OS, though it has a bit of added complexity due to the
	migration capability.  Applications see only one OS.

e.	Fault isolation: the following faults may propagate from the
	non-realtime CPUs to the realtime CPUs:

	i	Holding a lock, mutex, or semaphore for too long, when
		that resource must be acquired by code that is permitted
		to run on the realtime CPUs.

	ii.	Memory corruption, either due to wild pointer or
		via wild DMA.

f.	Hardware and software configurations supported: all configurations,
	though single-CPU systems must have some sort of virtualizing
	facility so that the OS sees at least two virtual CPUs.

Strengths:  Excellent scheduling latencies.  Hard-realtime support for
	some services in some configurations.  Applications and
	administrators see a single OS and API.  Full performance and
	scalability for non-realtime and for pure-realtime tasks.

Weaknesses:  Migration overhead.  Strictly theoretical, as no implementations
	exist.  Requires multiple CPUs, either real or virtual.


D.  SUMMARY

At this point, it does not appear that any one approach can be all things
to all realtime applications.  It is therefore too early to pick a winner.
Advocates of a given approach are therefore advised to concentrate their
energy on implementations of their favorite approach, rather than engaging
in flamewars with advocates of other approaches.  ;-)

After all, in the end, the approaches that best meet the needs of the
user community will win out.  In fact, given that the Linux community
has come up with no fewer than seven classes of solutions to a problem
that is commonly thought to be unsolvable, it seems quite reasonable
to expect that yet more classes of solutions will yet appear.

So, which of these approaches can be combined?  The first three can
be thought of as elaborations on the general preemption theme, and
can be combined with each of the remaining four.  The nested-OS and
dual-OS/dual-core ideas can be combined by having one of the OSes
on one of the cores have another OS nested within it.   The
dual-core/dual-OS approach can be combined with either of the
migration approaches, simply by having one of the cores implement
the migration approach.  It should be possible to combine the two
migration approaches, though it is not clear that this is useful.

Regardless of whether Linux's direction ends up being a single one
of these approaches, a yet-as-unknown approach, or some combination,
realtime Linux looks to remain an exciting area.

Re: Attempted summary of "RT patch acceptance" thread

[Karim Yaghmour]

Paul E. McKenney wrote:
> [Quickly donning the asbestos suit with tungsten pinstripes...]

Haven't started going through it yet ... but I've just abandonned
my usual surroundings and quickly ran into my 10-feet-thick led
bunker ;)

Karim
-- 
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 1-866-677-4546

Re: Attempted summary of "RT patch acceptance" thread

[Paul E. McKenney]

On Tue, Jun 07, 2005 at 11:00:18PM -0400, Karim Yaghmour wrote:
> 
> Paul E. McKenney wrote:
> > [Quickly donning the asbestos suit with tungsten pinstripes...]
> 
> Haven't started going through it yet ... but I've just abandonned
> my usual surroundings and quickly ran into my 10-feet-thick led
> bunker ;)

Goes to show how naive I am -- I was only expecting the usual flamage,
not WMDs.  ;-)

						Thanx, Paul

Re: Attempted summary of "RT patch acceptance" thread

[Eric Piel]

Paul E. McKenney said:
> Hello!
> 
> Midway through the recent "RT patch acceptance" thread, someone mentioned
> that it might be good to summarize the various approaches.  The following
> is an attempt to do just this, with an eye to providing a reasonable
> framework for future discussion.
> 
> Thoughts?  Errors?  Omissions?

Hi Paul,

I haven't yet gone through all your email in detail but it seems very 
well documented and precise.

I'd like to add some information about your section "7.  Migration 
Within OS". I've now been working for more than a year on a project 
called ARTiS which precisely implements this approach. A announced was 
recently posted on the LKML :-) . You can find more information in this 
tread : http://lkml.org/lkml/2005/5/3/50 as well on our webpage 
www.lifl.fr/west/artis .

Concerning the QoS, we have been able to obtain hard realtime, at least 
very firm real-time. Tests were conducted over 8 hours on IA-64 and x86 
and gave respectively 105µs and 40µs of maximum latency. Not as good as 
you have mentioned but mostly of the same order :-)

Concerning the "e. fault isolation", on our implementation, holding a 
lock, mutex or semaphore will automatically migrate the task, therefore 
it's not a problem. Of course, some parts of the kernel that cannot be 
migrated might take a lock, namely the scheduler. For the scheduler, we 
modified most of the data structures requiring a lock so that they can 
be accessed locklessly (it's the hardest part of the implementation).

Concerning the weaknesses, one point that you didn't mention is the 
difficulty to fully load the realtime dedicated CPUs. Tasks tends to 
migrate more away from the RT CPUs than they come back. In ARTiS, a 
modification of the load-balancer permits to keep most of the power but 
there is still probably some loss. Concerning the migration overhead, 
there must be some, but it's not very big and quite difficult to 
measure. Actually, the migration itself is light in CPU usage, the 
problem is that it unschedules the task so that it might take some time 
before the task is scheduled again (but if there is enough load on the 
computer, we lose mostly nothing).

Finally, as you pointed out, one major requirement is obiously to have 
several CPUs. Luckily, SMT and dual core processors are more and more 
common (ARTiS was succesfully tested on Pentium HT). Still, in the 
embedded market this is not so usual, so that's a weakeness point if you 
target very devices. Our implementation is oriented toward big 
applications that requires both RT properties and high performance 
(that's why it was developped on IA-64).

That's my 2 cents for your summary :-)

Eric

PS: please CC me when replying to the lkml.

Re: Attempted summary of "RT patch acceptance" thread

[Karim Yaghmour]

Paul,

I've finished reading your summary and I must say that it's excellent.
I don't remember ever reading a non-partisan comparison of this level
on the issue of real-time and Linux. Thanks for writing _and_ having
the guts to post it :)

There is only one issue I would like to further highlight.

Note: None of the following should be in any way controversial, I'm
just providing further background.

Paul E. McKenney wrote:
> the corresponding approach's strengths and weaknesses.  I do not address
> "strength of community", even though this may well be the decisive factor.

Indeed what you state here is entirely true. While Adeos and RTAI
development has been very active for quite a few years now, it must
be said that this development has largely gone unnoticed to LKML
participants -- as was obvious by the amount of surprise caused by
the realization of the existence of key Adeos and RTAI features.

Part of this is historical. 10 years ago, Linux's state was such
that those who were interested in doing rt with it realized that
it wasn't about to become rt-capable any time soon. Hence, they
"went away" and did their own little thing. They had their mailing
lists, their own flame-wars, their own conferences, and there was
very little common shared with the mainstream LKML community.

In fact, for a very long time, most kernel developers I spoke to
about real-time would refer back to a single project, RTLinux. To
this day, actually, if you look in the MAINTAINERS file, it still
says:
> RTLINUX  REALTIME  LINUX
> P:      Victor Yodaiken
> M:      yodaiken@fsmlabs.com
> L:      rtl@rtlinux.org
> W:      www.rtlinux.org
> S:      Maintained
Yet, the days where RTLinux was _the_ real-time Linux extension
are long gone and www.rtlinux.org has been a redirect to a .com
site for quite some time now -- I've suggested in the past that
this entry be replaced by RTAI, but I was told that neither should
in fact be in there, which is fair-enough, but nothing came of
this suggestion and the entry is still in the maintainers file.

This state of things remained until May 2002 when I picked up on
a post by Andrea to point out a "few" problems the RTAI community
saw with the RTLinux project. The ensuing thread was remarkably
intense -- not for the faint of heart. Here's the root of it if
you're ever interested in reading a huge flame-fest:
http://marc.theaimsgroup.com/?l=linux-kernel&m=102227589127072&w=2
While that discussion did serve to put RTAI on the map for some
developers, it also highlighted problems with the RTAI project
that needed to be solved.

Part of the issues was the patent problem, and that was solved
with the introduction of Adeos. However, with this and other
problems solved, the RTAI developers went back the way they came
from: to their own separate mailing lists.

In the past few years, though, a new bread of real-time developers
have become interested in making Linux fit for real-time
applications. Unlike the previous generation, though, these folks
have concentrated their efforts on working within the framework
already agreed upon by existing kernel developers: the LKML. And
in that, they have achieved a level of awareness amongst the kernel
crowd that I think RTAI and Adeos have not yet reached.

I've tried to remedy to this situation as best I can, by pointing
out what was obvious to me when appropriate. However, it must be
said that I haven't been actively involved with either Adeos or
RTAI in quite some time. So while I did play a part in the
history of both projects, there are others that are in a much
better position than I am to present to the LKML the work done
by the RTAI and Adeos communities.

In essence, therefore, what I have to say is this:
- To those who are actively involved in the development of RTAI
and Adeos, now is the time to drop the historical tendency of
acting as an entirely separate community and to start sharing
your work on the LKML.
- To those who are actively involved in finding solutions to the
real-time issues in Linux, do not be fooled by the apparent lack
of activity in the Adeos or RTAI projects, they are both very
active and warrant consideration.

As you correctly state, "strength of community" is likely a decisive
factor. What is important here is not to confuse "apparent" strength
of community -- or lack thereof -- with "actual" strength of
community -- or lack thereof.

Thanks again for a great piece.

Karim Yaghmour
-- 
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 1-866-677-4546

Re: Attempted summary of "RT patch acceptance" thread

[Chris Friesen]

Paul E. McKenney wrote:

Nice writeup.

I think you mixed up the "weaknesses" section between "nested OS" and 
"dual-os/dual-core".

Chris

Re: Attempted summary of "RT patch acceptance" thread

[Paul E. McKenney]

On Wed, Jun 08, 2005 at 12:46:45PM -0600, Chris Friesen wrote:
> Paul E. McKenney wrote:
> 
> Nice writeup.

Glad you liked it!

> I think you mixed up the "weaknesses" section between "nested OS" and 
> "dual-os/dual-core".

Good catch!  I have moved the sentence:

	The pair of cores will be more expensive than a single core,
	though one might use virtualization to emulate the two CPUs.

from "nested OS" to "dual-os/dual-core".

							Thanx, Paul

Re: Attempted summary of "RT patch acceptance" thread

[Paul E. McKenney]

On Wed, Jun 08, 2005 at 05:54:31PM +0200, Eric Piel wrote:
> Paul E. McKenney said:
> >Hello!
> >
> >Midway through the recent "RT patch acceptance" thread, someone mentioned
> >that it might be good to summarize the various approaches.  The following
> >is an attempt to do just this, with an eye to providing a reasonable
> >framework for future discussion.
> >
> >Thoughts?  Errors?  Omissions?
> 
> Hi Paul,
> 
> I haven't yet gone through all your email in detail but it seems very 
> well documented and precise.

Thank you, glad you like it!

> I'd like to add some information about your section "7.  Migration 
> Within OS". I've now been working for more than a year on a project 
> called ARTiS which precisely implements this approach. A announced was 
> recently posted on the LKML :-) . You can find more information in this 
> tread : http://lkml.org/lkml/2005/5/3/50 as well on our webpage 
> www.lifl.fr/west/artis .

Aaack!!!  Don't know how I missed this, will review it and update the
document.  Hmmm...  Checking the date, I plead near-terminal jet lag.

> Concerning the QoS, we have been able to obtain hard realtime, at least 
> very firm real-time. Tests were conducted over 8 hours on IA-64 and x86 
> and gave respectively 105µs and 40µs of maximum latency. Not as good as 
> you have mentioned but mostly of the same order :-)

Quite impressive!  So, does this qualify as "ruby hard", or is it only
"metal hard"?  ;-)

The service measured was process scheduling, right?

> Concerning the "e. fault isolation", on our implementation, holding a 
> lock, mutex or semaphore will automatically migrate the task, therefore 
> it's not a problem. Of course, some parts of the kernel that cannot be 
> migrated might take a lock, namely the scheduler. For the scheduler, we 
> modified most of the data structures requiring a lock so that they can 
> be accessed locklessly (it's the hardest part of the implementation).

Are the non-migrateable portions of the scheduler small enough that
one could do a worst-case timing analysis?  Preferably aided by some
automation...

> Concerning the weaknesses, one point that you didn't mention is the 
> difficulty to fully load the realtime dedicated CPUs. Tasks tends to 
> migrate more away from the RT CPUs than they come back. In ARTiS, a 
> modification of the load-balancer permits to keep most of the power but 
> there is still probably some loss. Concerning the migration overhead, 
> there must be some, but it's not very big and quite difficult to 
> measure. Actually, the migration itself is light in CPU usage, the 
> problem is that it unschedules the task so that it might take some time 
> before the task is scheduled again (but if there is enough load on the 
> computer, we lose mostly nothing).

One approach would be to mark the migrated task so that it returns
to the realtime CPU as soon as it completes the realtime-unsafe
operation.

> Finally, as you pointed out, one major requirement is obiously to have 
> several CPUs. Luckily, SMT and dual core processors are more and more 
> common (ARTiS was succesfully tested on Pentium HT). Still, in the 
> embedded market this is not so usual, so that's a weakeness point if you 
> target very devices. Our implementation is oriented toward big 
> applications that requires both RT properties and high performance 
> (that's why it was developped on IA-64).

Another approach is to insert a virtualization layer (think in terms of
a very cut-down variant of Xen) that tells the OS that there are two CPUs.
This layer then gives realtime service to one, but not to the other.
That way, the OS thinks that it has two CPUs, and can still do the
migration tricks despite having only one real CPU.

Anyway, interesting approach!

						Thanx, Paul

> That's my 2 cents for your summary :-)
> 
> Eric
> 
> PS: please CC me when replying to the lkml.
> 

Re: Attempted summary of "RT patch acceptance" thread

[Paul E. McKenney]

On Wed, Jun 08, 2005 at 12:51:05PM -0400, Karim Yaghmour wrote:
> 
> Paul,
> 
> I've finished reading your summary and I must say that it's excellent.
> I don't remember ever reading a non-partisan comparison of this level
> on the issue of real-time and Linux. Thanks for writing _and_ having
> the guts to post it :)

Thank you for the kind words!  But who needs guts when you have
senility?  ;-)

> There is only one issue I would like to further highlight.
> 
> Note: None of the following should be in any way controversial, I'm
> just providing further background.

;-)  ;-)  ;-)

> Paul E. McKenney wrote:
> > the corresponding approach's strengths and weaknesses.  I do not address
> > "strength of community", even though this may well be the decisive factor.
> 
> Indeed what you state here is entirely true. While Adeos and RTAI
> development has been very active for quite a few years now, it must
> be said that this development has largely gone unnoticed to LKML
> participants -- as was obvious by the amount of surprise caused by
> the realization of the existence of key Adeos and RTAI features.
> 
> Part of this is historical. 10 years ago, Linux's state was such
> that those who were interested in doing rt with it realized that
> it wasn't about to become rt-capable any time soon. Hence, they
> "went away" and did their own little thing. They had their mailing
> lists, their own flame-wars, their own conferences, and there was
> very little common shared with the mainstream LKML community.
> 
> In fact, for a very long time, most kernel developers I spoke to
> about real-time would refer back to a single project, RTLinux. To
> this day, actually, if you look in the MAINTAINERS file, it still
> says:
> > RTLINUX  REALTIME  LINUX
> > P:      Victor Yodaiken
> > M:      yodaiken@fsmlabs.com
> > L:      rtl@rtlinux.org
> > W:      www.rtlinux.org
> > S:      Maintained
> Yet, the days where RTLinux was _the_ real-time Linux extension
> are long gone and www.rtlinux.org has been a redirect to a .com
> site for quite some time now -- I've suggested in the past that
> this entry be replaced by RTAI, but I was told that neither should
> in fact be in there, which is fair-enough, but nothing came of
> this suggestion and the entry is still in the maintainers file.
> 
> This state of things remained until May 2002 when I picked up on
> a post by Andrea to point out a "few" problems the RTAI community
> saw with the RTLinux project. The ensuing thread was remarkably
> intense -- not for the faint of heart. Here's the root of it if
> you're ever interested in reading a huge flame-fest:
> http://marc.theaimsgroup.com/?l=linux-kernel&m=102227589127072&w=2
> While that discussion did serve to put RTAI on the map for some
> developers, it also highlighted problems with the RTAI project
> that needed to be solved.
> 
> Part of the issues was the patent problem, and that was solved
> with the introduction of Adeos. However, with this and other
> problems solved, the RTAI developers went back the way they came
> from: to their own separate mailing lists.
> 
> In the past few years, though, a new bread of real-time developers
> have become interested in making Linux fit for real-time
> applications. Unlike the previous generation, though, these folks
> have concentrated their efforts on working within the framework
> already agreed upon by existing kernel developers: the LKML. And
> in that, they have achieved a level of awareness amongst the kernel
> crowd that I think RTAI and Adeos have not yet reached.
> 
> I've tried to remedy to this situation as best I can, by pointing
> out what was obvious to me when appropriate. However, it must be
> said that I haven't been actively involved with either Adeos or
> RTAI in quite some time. So while I did play a part in the
> history of both projects, there are others that are in a much
> better position than I am to present to the LKML the work done
> by the RTAI and Adeos communities.
> 
> In essence, therefore, what I have to say is this:
> - To those who are actively involved in the development of RTAI
> and Adeos, now is the time to drop the historical tendency of
> acting as an entirely separate community and to start sharing
> your work on the LKML.
> - To those who are actively involved in finding solutions to the
> real-time issues in Linux, do not be fooled by the apparent lack
> of activity in the Adeos or RTAI projects, they are both very
> active and warrant consideration.
> 
> As you correctly state, "strength of community" is likely a decisive
> factor. What is important here is not to confuse "apparent" strength
> of community -- or lack thereof -- with "actual" strength of
> community -- or lack thereof.

All good points!  I added a few sentences encouraging realtime folks to
participate in LKML discussions.

						Thanx, Paul

> Thanks again for a great piece.
> 
> Karim Yaghmour
> -- 
> Author, Speaker, Developer, Consultant
> Pushing Embedded and Real-Time Linux Systems Beyond the Limits
> http://www.opersys.com || karim@opersys.com || 1-866-677-4546
> 

Re: Attempted summary of "RT patch acceptance" thread

[Philippe Gerum]

Karim Yaghmour wrote:
 > Paul,
 >
 > I've finished reading your summary and I must say that it's excellent.
 > I don't remember ever reading a non-partisan comparison of this level
 > on the issue of real-time and Linux. Thanks for writing _and_ having
 > the guts to post it :)
 >

I second that. It's refreshing.

 > There is only one issue I would like to further highlight.
 >
 > Note: None of the following should be in any way controversial, I'm
 > just providing further background.
 >
 > Paul E. McKenney wrote:
 >
 >>the corresponding approach's strengths and weaknesses.  I do not address
 >>"strength of community", even though this may well be the decisive 
factor.
 >

[snip][snip][snip]

 >
 > In the past few years, though, a new bread of real-time developers
 > have become interested in making Linux fit for real-time
 > applications. Unlike the previous generation, though, these folks
 > have concentrated their efforts on working within the framework
 > already agreed upon by existing kernel developers: the LKML. And
 > in that, they have achieved a level of awareness amongst the kernel
 > crowd that I think RTAI and Adeos have not yet reached.
 >
 > I've tried to remedy to this situation as best I can, by pointing
 > out what was obvious to me when appropriate.

Ack, and you should be thanked for this. Especially since in some
occasions, RTAI, as a community project, did likely get on your
nerves, by not presenting its progress in this area on a more regular
basis, if it ever bothered presenting it at all. Well, ok, no excuse.

Btw, taking the risk to piss you off a little bit more :o), I don't
resist to announce here, two weeks late, that RTAI/fusion (0.7.5) has
been ported over ia64, thanks to the contribution of the HYADES
project (http://www.hyades-itea.org/).

 > However, it must be
 > said that I haven't been actively involved with either Adeos or
 > RTAI in quite some time. So while I did play a part in the
 > history of both projects, there are others that are in a much
 > better position than I am to present to the LKML the work done
 > by the RTAI and Adeos communities.
 >
 > In essence, therefore, what I have to say is this:
 > - To those who are actively involved in the development of RTAI
 > and Adeos, now is the time to drop the historical tendency of
 > acting as an entirely separate community and to start sharing
 > your work on the LKML.

Needless to remind you that some of us, in the Adeos/RTAI side of the
universe, finally acknowledged some time ago that not participating on
a regular basis to the LKML was a mistake, and FWIW, I'm one of these
people.

This said, the following might help understanding why it has not been
as "natural" as one would have thought, for us to have a regular
presence so far, in the discussions around the kernel framework:

1. Adeos is basically an enabler for real-time cores that are
steadfastly designed to run side-by-side with the Linux kernel, for
various reasons involving predictability, complexity, scalability and
performances; I won't rehash those here. Meanwhile, no consensus seems 
to exist among the kernel contributors and users, on the best way to
provide real-time support, either natively inside the kernel, or
cooperatively with, but outside of it, which is obviously a crucial
design issue with long-term implications. People don't even seem to
settle on the definition of "real-time", which indeed covers various 
application requirements in terms of predictability. Accordingly, /me, 
as an Adeos contributor, did not see the point in waving hands too much 
or submit patches for inclusion. Some may say that it's precisely 
because of the latter that Adeos gets no opportunity of ever being 
considered by the kernel people, but hey, nobody said I was a smart guy 
neither.

2. RTAI is a common client of the Adeos layer, and as such, it's an
external piece of code which in turn is loaded by the Linux kernel,
and runs embodied in plain kernel modules. For the most part, people
in the RTAI project usually don't write core kernel code, but are
using a limited subset of its documented interface "as is". For this
reason, these people tend to keep the noise / signal ratio as low as
possible, and do not appear that much on the LKML, just because their
contribution might be of no direct interest to the "vanilla" kernel
folks.

But,... fact is that RTAI is evolving, and that people working on the
development track called "fusion" have a year ago decided to reach a
seamless cooperation between the Linux kernel and the real-time core for
their users, so that people using RTAI are now able to keep the best
of both worlds. This approach preserves the common Linux programming 
model in user-space, and offers a consistent integration of the regular 
Linux services into the RTAI space. Hopefully, this cooperation of 
real-time designs will someday translate into the cooperation of people 
promoting these designs.

Accordingly, we already closely track what's going on the
LKML on a daily basis, don't doubt of it. All in all, being more
present in the kernel community for us is more about finding the best
way to submit the most useful information to the LKML, and perhaps get
over some shyness to do it. Anyway, we will learn.

 > - To those who are actively involved in finding solutions to the
 > real-time issues in Linux, do not be fooled by the apparent lack
 > of activity in the Adeos or RTAI projects, they are both very
 > active and warrant consideration.
 >
 > As you correctly state, "strength of community" is likely a decisive
 > factor. What is important here is not to confuse "apparent" strength
 > of community -- or lack thereof -- with "actual" strength of
 > community -- or lack thereof.
 >
 > Thanks again for a great piece.
 >
 > Karim Yaghmour

-- 

Philippe.

Re: Attempted summary of "RT patch acceptance" thread

[Tim Bird]

Paul E. McKenney wrote:
> Hello!
> 
> Midway through the recent "RT patch acceptance" thread, someone mentioned
> that it might be good to summarize the various approaches.  The following
> is an attempt to do just this, with an eye to providing a reasonable
> framework for future discussion.
> 
> Thoughts?  Errors?  Omissions?

I haven't finished it, but it looks great so far.  Are you planning to
repost it to LKML, or otherwise publish it somewhere, after incorporating
feedback?

If you don't mind, I would like to put a copy on the CELF wiki.  But
if it's going to have a home elsewhere, I don't want to create a
duplicate copy.

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Electronics
=============================

Re: Attempted summary of "RT patch acceptance" thread

[Paul E. McKenney]

On Thu, Jun 09, 2005 at 04:34:11PM -0700, Tim Bird wrote:
> Paul E. McKenney wrote:
> > Hello!
> > 
> > Midway through the recent "RT patch acceptance" thread, someone mentioned
> > that it might be good to summarize the various approaches.  The following
> > is an attempt to do just this, with an eye to providing a reasonable
> > framework for future discussion.
> > 
> > Thoughts?  Errors?  Omissions?
> 
> I haven't finished it, but it looks great so far.  Are you planning to
> repost it to LKML, or otherwise publish it somewhere, after incorporating
> feedback?

Glad you like it!

I figured on sending out an update sometime next week, after incorporating
feedback.

> If you don't mind, I would like to put a copy on the CELF wiki.  But
> if it's going to have a home elsewhere, I don't want to create a
> duplicate copy.

As long as questions, comments, and suggestions get pointed back to me,
feel free!  I may do a more formal writeup in a couple of months, if there
is enough added content of interest.

							Thanx, Paul

Re: Attempted summary of "RT patch acceptance" thread

[Lee Revell]

On Thu, 2005-06-09 at 16:50 -0700, Paul E. McKenney wrote:
> On Thu, Jun 09, 2005 at 04:34:11PM -0700, Tim Bird wrote:
> > Paul E. McKenney wrote:
> > > Hello!
> > > 
> > > Midway through the recent "RT patch acceptance" thread, someone mentioned
> > > that it might be good to summarize the various approaches.  The following
> > > is an attempt to do just this, with an eye to providing a reasonable
> > > framework for future discussion.
> > > 
> > > Thoughts?  Errors?  Omissions?
> > 
> > I haven't finished it, but it looks great so far.  Are you planning to
> > repost it to LKML, or otherwise publish it somewhere, after incorporating
> > feedback?
> 
> Glad you like it!
> 
> I figured on sending out an update sometime next week, after incorporating
> feedback.

I believe you are too friendly to vanilla + CONFIG_PREEMPT.  People are
still seeing tens or hundreds of ms bumps.

Does the LTP include an RT latency test yet?

Lee

Re: Attempted summary of "RT patch acceptance" thread

[Paul E. McKenney]

On Thu, Jun 09, 2005 at 10:59:48PM -0400, Lee Revell wrote:
> On Thu, 2005-06-09 at 16:50 -0700, Paul E. McKenney wrote:
> > On Thu, Jun 09, 2005 at 04:34:11PM -0700, Tim Bird wrote:
> > > Paul E. McKenney wrote:
> > > > Hello!
> > > > 
> > > > Midway through the recent "RT patch acceptance" thread, someone mentioned
> > > > that it might be good to summarize the various approaches.  The following
> > > > is an attempt to do just this, with an eye to providing a reasonable
> > > > framework for future discussion.
> > > > 
> > > > Thoughts?  Errors?  Omissions?
> > > 
> > > I haven't finished it, but it looks great so far.  Are you planning to
> > > repost it to LKML, or otherwise publish it somewhere, after incorporating
> > > feedback?
> > 
> > Glad you like it!
> > 
> > I figured on sending out an update sometime next week, after incorporating
> > feedback.
> 
> I believe you are too friendly to vanilla + CONFIG_PREEMPT.  People are
> still seeing tens or hundreds of ms bumps.

Good point -- I certainly need to add a disclaimer to the effect that
common hardware (such as VGA, last I checked some months ago) can
cause such bumps, as can long-running system calls.  Here is my updated
CONFIG_PREEMPT quality-of-service section:

a.	Quality of service: soft realtime, with timeframe of 100s of
	microseconds for task scheduling and interrupt handling, but
	-only- for very carefully restricted hardware configurations
	that exclude problematic devices and drivers (such as VGA)
	that can cause latency bumps of tens or even hundreds of
	milliseconds (-not- microseconds).  Furthermore, the software
	configuration of such systems must be carefully controlled,
	for example, doing a "kill -1" traverses the entire task list
	with tasklist_lock held (see kill_something_info()), which might
	result in disappointing latencies in systems with very large
	numbers of tasks.  System services providing I/O, networking,
	task creation, and VM manipulation can take much longer.  A very
	small performance penalty is exacted, since spinlocks and RCU
	must suppress preemption.

Does this help, or are there other CONFIG_PREEMPT latency issues that
need to be called out?

> Does the LTP include an RT latency test yet?

Not as far as I know.  I believe that LTP contains primarily pass-fail
rather than performance tests, but regardless of where RT latency
tests live, I believe that there needs to be a good home for them.

There are a number of RT latency tests that people run -- any nominations
for particularly good ones?  Should we be running lmbench at realtime
priority with 15 kernel builds in the background, measuring the
latency of each lmbench operation, in addition to running the existing
scheduler-latency tests?  (Sorry, couldn't resist!)

						Thanx, Paul

Re: Attempted summary of "RT patch acceptance" thread

[Andrea Arcangeli]

Hello Paul,

kudos for your very nice RT documents (as usual ;)

On Fri, Jun 10, 2005 at 08:47:46AM -0700, Paul E. McKenney wrote:
> Good point -- I certainly need to add a disclaimer to the effect that
> common hardware (such as VGA, last I checked some months ago) can

> a.	Quality of service: soft realtime, with timeframe of 100s of
> 	microseconds for task scheduling and interrupt handling, but
> 	-only- for very carefully restricted hardware configurations
> 	that exclude problematic devices and drivers (such as VGA)
> 	that can cause latency bumps of tens or even hundreds of
> 	milliseconds (-not- microseconds).  Furthermore, the software
> 	configuration of such systems must be carefully controlled,
> 	for example, doing a "kill -1" traverses the entire task list
> 	with tasklist_lock held (see kill_something_info()), which might
> 	result in disappointing latencies in systems with very large
> 	numbers of tasks.  System services providing I/O, networking,
> 	task creation, and VM manipulation can take much longer.  A very
> 	small performance penalty is exacted, since spinlocks and RCU
> 	must suppress preemption.
> 
> Does this help, or are there other CONFIG_PREEMPT latency issues that
> need to be called out?

You don't need to add it to the document, but as a further pratical
example of troublesome hardware besides VGA (could be a software issue
and not hardware issue though) I'd like to make the example of the irq
handler of the uhci usb1.1 controller that takes up to 8msec on a 1ghz
atlhon UP system, and there's nothing that PREEMPT can do about it since
it's an hard-irq. This latency keeps triggering a few times per second
on my firewall for the last few years.

preempt-RT _can_ do something about it but only _if_ people hacks the
drivers properly and makes sure to call local_irq_save_nort instead of
local_irq_save and other explicit changes like that, things that if
missing are noticeable only during measurements with preempt-RT config
option enabled (hence the metal-hard classification of preempt-RT and
not ruby-hard definition).

See the tg3 updates required to be safe with preempt-RT without breaking
hard-RT as a clear example of how preempt-RT is weak:

--- linux/drivers/net/tg3.c.orig
+++ linux/drivers/net/tg3.c
@@ -3229,9 +3229,9 @@ static int tg3_start_xmit(struct sk_buff
         * So we really do need to disable interrupts when taking
         * tx_lock here.
         */
-       local_irq_save(flags);
+       local_irq_save_nort(flags);
        if (!spin_trylock(&tp->tx_lock)) { 
-               local_irq_restore(flags);
+               local_irq_restore_nort(flags);
                return NETDEV_TX_LOCKED; 
        } 
 
There's no apparent reason why all those changes should be required to
get hard-RT.

Both RTAI and rtlinux _don't_ require to change all those drivers to get
the guarantee that the kernel will get out of the way within a certain
nanoseconds deadline interval.

Furthermore with the scheduler, mutex and context switch code into the
equation, it gets more and more difficult to calculate with math the max
latency that preempt-RT will provide, while it's almost trivial to do
that with RTAI/rtlinux given only the nanokernel code runs before the
hard-RT code is invoked and there are not many paths to test, so one has
to disable the cache and just measure the few possible nanokenrel paths.
(as usual when speaking about hard-RT I've robots in mind, and not audio
code that will call into the alsa ioctls)

This below is the kind of stuff where I wouldn't even dream to replace
a ruby-hard rtlinux/RTAI with a weaker metal-hard and possibly
underperformant (cause scheduling hard-irq in userland and scheduling
instead of spinning isn't going to be cheap in smp) preempt-RT solution:

http://linuxdevices.com/articles/AT7871136191.html

In the above RTAI should have made it as well as rtlinux of course.

Re: Attempted summary of "RT patch acceptance" thread

[Bill Huey]

On Fri, Jun 10, 2005 at 07:37:28PM +0200, Andrea Arcangeli wrote:
> See the tg3 updates required to be safe with preempt-RT without breaking
> hard-RT as a clear example of how preempt-RT is weak:
...
> There's no apparent reason why all those changes should be required to
> get hard-RT.

Some of it is lack of displine with how various drivers do locking in
that they overload the meaning of a spin_lock, etc... to also disable
preemption and side effects with preempt_count. Making all of this formal
is a good thing since it clarifies and un-ambiguates those code paths.
It's something that should have been done in the first place and I
consider using implicit masking like that to be rather sloppy.

> Both RTAI and rtlinux _don't_ require to change all those drivers to get
> the guarantee that the kernel will get out of the way within a certain
> nanoseconds deadline interval.

They obey the same constraints on driver in those nanokernel systems also
apply to the preempt patch. There is no difference other than the regular
driver layers in that they now need to be audited, duh. All RTOSs need to
do this.

> Furthermore with the scheduler, mutex and context switch code into the
> equation, it gets more and more difficult to calculate with math the max
> latency that preempt-RT will provide, while it's almost trivial to do
> that with RTAI/rtlinux given only the nanokernel code runs before the
> hard-RT code is invoked and there are not many paths to test, so one has
> to disable the cache and just measure the few possible nanokenrel paths.
> (as usual when speaking about hard-RT I've robots in mind, and not audio
> code that will call into the alsa ioctls)
> 
> This below is the kind of stuff where I wouldn't even dream to replace
> a ruby-hard rtlinux/RTAI with a weaker metal-hard and possibly
> underperformant (cause scheduling hard-irq in userland and scheduling
> instead of spinning isn't going to be cheap in smp) preempt-RT solution:

LynxOS is a hard real time OS similar to how the preemption patch works
in that it's a single kernel system. It's gets those times regularly and
is obviously deterministic because of careful coding conventions just like
it has always been. Single kernel determinism is not new. It's only new
to you.

bill

Re: Attempted summary of "RT patch acceptance" thread

[Lee Revell]

On Fri, 2005-06-10 at 12:39 -0700, Bill Huey wrote:
> Some of it is lack of displine with how various drivers do locking in
> that they overload the meaning of a spin_lock, etc... to also disable
> preemption and side effects with preempt_count. Making all of this
> formal
> is a good thing since it clarifies and un-ambiguates those code paths.
> It's something that should have been done in the first place 

Keep in mind there are large subsystems that are already RT safe like
ALSA.

Lee

Re: Attempted summary of "RT patch acceptance" thread

[Daniel Walker]

On Fri, 2005-06-10 at 19:37 +0200, Andrea Arcangeli wrote:

> 
> preempt-RT _can_ do something about it but only _if_ people hacks the
> drivers properly and makes sure to call local_irq_save_nort instead of
> local_irq_save and other explicit changes like that, things that if
> missing are noticeable only during measurements with preempt-RT config
> option enabled (hence the metal-hard classification of preempt-RT and
> not ruby-hard definition).


This is no longer an issue. We did an interrupt disable conversion so
now there is only a few places that actually disable interrupts. It
hasn't been tested fully, but you should be able to used whatever
drivers you want and still get the same interrupt latency. However,
preempt latency is still and issue for these drivers.

Daniel

Re: Attempted summary of "RT patch acceptance" thread

[Karim Yaghmour]

Bill Huey (hui) wrote:
> LynxOS is a hard real time OS similar to how the preemption patch works
> in that it's a single kernel system. It's gets those times regularly and
> is obviously deterministic because of careful coding conventions just like
> it has always been. Single kernel determinism is not new. It's only new
> to you.

Bill, I know PREEMPT_RT is something you root for strongly. FWIW, however,
such statements are usually counter-productive.

The existence of real-time monolithic kernels is not disputed. Note, though,
that LynxOS is controlled by a single corporation and its development
model could not be compared in any way to Linux. Because of Linux's
development model, what others are attempting to explain (Andrea included,
I think), the reality is that the auditing/upgrading you speak of is unlikely
to ever become part of the development philosophy.

I don't contest the fact that those promoting PREEMPT_RT intend to conduct
this auditing for the drivers that are of interest to their development.
That, though, doesn't mean that those who are maintaining existing driver
sets want to take part in that.

Karim
-- 
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 1-866-677-4546

Re: Attempted summary of "RT patch acceptance" thread

[Lee Revell]

On Fri, 2005-06-10 at 08:47 -0700, Paul E. McKenney wrote:
> > Does the LTP include an RT latency test yet?
> 
> Not as far as I know.  I believe that LTP contains primarily pass-fail
> rather than performance tests, but regardless of where RT latency
> tests live, I believe that there needs to be a good home for them.

Maybe I didn't mean LTP, I'm thinking of whatever that popular benchmark
suite is that people use to identify performance regressions.  OSDL
something or other?

The canonical test is "rtc_wakeup", it just sets up a stream of
interrupts from the RTC, polls on it and measures the delay.  Check the
list archives for the URL.

Lee

Re: Attempted summary of "RT patch acceptance" thread

[Lee Revell]

On Fri, 2005-06-10 at 19:37 +0200, Andrea Arcangeli wrote:
> You don't need to add it to the document, but as a further pratical
> example of troublesome hardware besides VGA (could be a software issue
> and not hardware issue though)  

The VGA problems were due to (X, not kernel!) driver bugs.  Recent
versions of X are thought to be OK.

It's the exact same issue described in this paper, scroll down to
section 4.5.

http://www.cs.utah.edu/~regehr/papers/hotos7/hotos7.html

There's absolutely nothing the kernel or PREEMPT_RT can do about this,
AFAICT even RTAI would be affected, because X lets userspace drivers
talk directly to the hardware including wedging the PCI bus.

Lee

Re: Attempted summary of "RT patch acceptance" thread

[Andrea Arcangeli]

On Fri, Jun 10, 2005 at 04:45:38PM -0400, Lee Revell wrote:
> AFAICT even RTAI would be affected, because X lets userspace drivers
> talk directly to the hardware including wedging the PCI bus.

Yes, I made the usb example exactly to show how latency bugs can be
longstanding in the kernel too without requiring X or hardware bugs
(this usb thing breaks kernel latency for years and yet nobody fixed it
simply because it just works fine in practice, I noticed because
apparently my PIT or tsc goes out of sync over time, and in turn my
system time was going into the future pretty quick with HZ=1000, or I
would have never noticed, of course I'm compiling the kernel with HZ=100
on that system to work around it). Those latency issues can showup in
random drivers all the time, and this one was of an extreme magnitude in
the millisecond range. The smaller the magnitude of the latency impact,
the more frequently you should find it in drivers.

Re: Attempted summary of "RT patch acceptance" thread

[Eric Piel]

06/09/2005 04:20 AM, Paul E. McKenney wrote/a écrit:
 >>Concerning the QoS, we have been able to obtain hard realtime, at least
 >>very firm real-time. Tests were conducted over 8 hours on IA-64 and x86
 >>and gave respectively 105µs and 40µs of maximum latency. Not as good as
 >>you have mentioned but mostly of the same order :-)
 >
 >
 > Quite impressive!  So, does this qualify as "ruby hard", or is it only
 > "metal hard"?  ;-)
Well, you have to consider that this is still full Linux running. All 
the best we can do is to not make it crash or hung more than the vanilla 
kernel, it's still vulnerable to any bug of any driver :-/ In addition, 
I highly doubt this approach can ever have an implementation were the 
maximum latency is theoritically proven. The best we have is just 
measurements of the system running with high loads during very long time.

 >
 > The service measured was process scheduling, right?
Yes, on IA64, from the hardware IRQ fireing to process scheduling (on 
x86 it's from kernel IRQ handling to process scheduling).

 >>Concerning the "e. fault isolation", on our implementation, holding a
 >>lock, mutex or semaphore will automatically migrate the task, therefore
 >>it's not a problem. Of course, some parts of the kernel that cannot be
 >>migrated might take a lock, namely the scheduler. For the scheduler, we
 >>modified most of the data structures requiring a lock so that they can
 >>be accessed locklessly (it's the hardest part of the implementation).
 >
 >
 > Are the non-migrateable portions of the scheduler small enough that
 > one could do a worst-case timing analysis?  Preferably aided by some
 > automation...
Well, ARTiS only modifies the schedule() function but there is probably 
too much possible interaction to really be able to prove anything (the 
fact that it's a SMP system doesn't help!).


 > One approach would be to mark the migrated task so that it returns
 > to the realtime CPU as soon as it completes the realtime-unsafe
 > operation.
We use a different approach: keep (small) statistics of the task doing 
often lock and the one that are more "computational". Then we migrate in 
priority the tasks that don't do locks. Your suggestion could be used 
at the same time but it might not be so efficient anymore. Additionally, 
in the current implementation, it's not so easy to know when a task 
which is running can go back to a RT CPU.



 > Another approach is to insert a virtualization layer (think in terms of
 > a very cut-down variant of Xen) that tells the OS that there are two 
CPUs.
 > This layer then gives realtime service to one, but not to the other.
 > That way, the OS thinks that it has two CPUs, and can still do the
 > migration tricks despite having only one real CPU.
Simulation of an SMP on a UP? This sounds quite heavy but it might be 
interesting to try :-)


 >
 > Anyway, interesting approach!
Thanks,

Eric

Re: Attempted summary of "RT patch acceptance" thread

[Bill Huey]

On Fri, Jun 10, 2005 at 11:06:14PM +0200, Andrea Arcangeli wrote:
> On Fri, Jun 10, 2005 at 04:45:38PM -0400, Lee Revell wrote:
> > AFAICT even RTAI would be affected, because X lets userspace drivers
> > talk directly to the hardware including wedging the PCI bus.
> 
> Yes, I made the usb example exactly to show how latency bugs can be
> longstanding in the kernel too without requiring X or hardware bugs
> (this usb thing breaks kernel latency for years and yet nobody fixed it
> simply because it just works fine in practice, I noticed because

...

That's because it puts much more stress on the Linux kernel than before.
It's going to take some time to log, audit and fix these issues, but
fixing them after discovery isn't really a big deal as you make it. The
hyperbole where drivers screw with latency is finite. This is a very
managable problem. Again, this a problem for all RTOS system and their
supporting drivers as well since they often come from BSD open source
community.

bill

Re: Attempted summary of "RT patch acceptance" thread

[Eric Piel]

I've read the whole summary and really appreciated it. It's clear, 
precise and objective.

Just a small change to "1 - QoS":

> b.	For each service:
> 
> 	i.	Probability of missing a deadline due to software,
> 		ranging from 0 to 1, with the value of 1 corresponding
> 		to the hardest possible hard realtime.

I think it should be (by reference to how you define probability at the 
beginning of the section):
Probability of not missing any deadline due to software

Thanks again for writing this summary,
Eric

Re: Attempted summary of "RT patch acceptance" thread

[Bill Huey]

On Fri, Jun 10, 2005 at 04:26:48PM -0400, Karim Yaghmour wrote:
> Bill, I know PREEMPT_RT is something you root for strongly. FWIW, however,
> such statements are usually counter-productive.

Some of the comments from various folks are just intolerably paranoid
and I'm frustrated by that.

> The existence of real-time monolithic kernels is not disputed. Note, though,
> that LynxOS is controlled by a single corporation and its development
> model could not be compared in any way to Linux. Because of Linux's

Well, as a dude that hangs with the LynxOS engineering staff in cubicals
next to me, I can say that development here is really no different than
any other kernel community that I've run into, open or closed. We're all
kernel engineers except with maybe a difference in emphasis on what is
important at any given time.

> development model, what others are attempting to explain (Andrea included,
> I think), the reality is that the auditing/upgrading you speak of is unlikely
> to ever become part of the development philosophy.

But it already has. This RT group doesn't need the whole community to jump
on it. We can do it ourselves and submit the changes to an upstream trees
where relevant.
 
> I don't contest the fact that those promoting PREEMPT_RT intend to conduct
> this auditing for the drivers that are of interest to their development.

Yeah, but that's just you and folks that have been friendly to this effort.
The rest is another story altogether.

> That, though, doesn't mean that those who are maintaining existing driver
> sets want to take part in that.

bill

Re: Attempted summary of "RT patch acceptance" thread

[Andrea Arcangeli]

On Fri, Jun 10, 2005 at 03:19:14PM -0700, Bill Huey wrote:
> managable problem. Again, this a problem for all RTOS system and their
> supporting drivers as well since they often come from BSD open source
> community.

Did you read Karim's answer at all?

Those RTOS don't have frenetic development in scheduler and all other
subsystems and they may be simpler as well, so perhaps you can disable
L2 and L3 and measure all possible paths that kernel can take in
invoking your rt code, but I doubt you're going to keep it up with linux
development in demonstrating the worst case deadline in every possible
hardware. Measuring all possible paths of a nanokernel is absolutely
trivial in comparison cause there are no different paths, so it's very
easy to give a worst case deadline to the RTAI/rtlinux designs, while it
sounds a total pain for preempt-RT, even after the local_irq_disable
issue is fixed (which is feasible as Daniel seems to have implemented
already).

metal hard solutions are fine where having an exact deadline isn't
required, or in places where missing the deadline results in a crashed
cellphone (even if I really hate it if my cellphone crashes, and yes
even the linux cellphone crashes by doing a cp /dev/sda /dev/null, the
usbstorage thingy has some problem in accessing the partition table of
the internal flash), but again if they would have picked a RTOS for the
project I linked on linuxdevices.com that has to react in 50usec, I
think they would have done a mistake when ruby-hard solutions like
RTAI/rtlinux are available on linux. So kudos to them for their IMHO
best correct choice of RT design.

RTOS may be fine for cellphones or other stuff where you may not even
need hard-RT at all, but you just want the lowest possible latency, but
I'd _never_ use it whenever I need a true deadline, since other more
reliable solutions are possible without much more complexity.

But this is me and my focus is on projects like the one quoted on
linuxdevices, cellphones may not even need hard-RT at all since I doubt
anybody would offload the gsm compression to the main linux cpu with the
risk of doing a buffer overflow during decompression and crashing
stations. You're free to use a metal-hard solution for problems that
requires ruby-hard, but I think that's very wrong choice after you're
aware it's not possible to math demonstrate a worst case latency with
metal-hard solutions.

About the fact the kernel can crash, and a driver can corrupt memory,
that's the difference between ruby and diamond.

I simply think RTOS is an awful solution to hard-RT, RTAI/rtlinux
designs are much more reliable. There are certainly areas where RTOS is
acceptable, and this is where hard-RT isn't required and you simply want
the lowest possible latency when invoking syscalls like alsa ioctls. But
when alsa kernel code is out of the equation I'd never use RTOS to get
the kernel out of the way.

Re: Attempted summary of "RT patch acceptance" thread

[Bill Huey]

On Fri, Jun 10, 2005 at 03:37:24PM -0700, Bill Huey wrote:
> On Fri, Jun 10, 2005 at 04:26:48PM -0400, Karim Yaghmour wrote:
> > Bill, I know PREEMPT_RT is something you root for strongly. FWIW, however,
> > such statements are usually counter-productive.
> 
> Some of the comments from various folks are just intolerably paranoid
> and I'm frustrated by that.

Oh god,

	http://linuxdevices.com/articles/AT4721170330.html

The original thread and the summary got posted to "linuxdevices" today.

bill

Re: Attempted summary of "RT patch acceptance" thread

[Daniel Walker]

On Sat, 2005-06-11 at 00:37 +0200, Andrea Arcangeli wrote:


> issue is fixed (which is feasible as Daniel seems to have implemented
> already).

It's in the RT patch as of 0.7.48 . Released early this week .

Daniel

Re: Attempted summary of "RT patch acceptance" thread

[Andrea Arcangeli]

On Fri, Jun 10, 2005 at 03:37:24PM -0700, Bill Huey wrote:
> Some of the comments from various folks are just intolerably paranoid

Just tell me how can you go to a customer and tell him that your
linux-RTOS has a guaranteed worst case latency of 50usec.  How can you
tell that? Did you exercise all possible scheduler paths with cache
disabled and calculated the worst case latency with rdtsc + math?

Why do you take risks when you can go with much more relaible solutions
like RTAI and rtlinux?

Using metal-hard RTOS in things that requires ruby-hard deadline
guarantee, seems just an unnecessary risk. I'd prefer you to be
more paranoid and less relaxed about these issues, there are enough bugs
and risks even when one does the best and is very paranoid.

Said that there are many problems where metal-hard solutions are
fine, every single app that don't require a worst-case guarantee of
the worst-case latency is fine with metal-hard, and every time one has
to call into the alsa ioctl or other complex syscalls also is fine with
metal hard since the first kmalloc or lock contention inside RT context,
will send your RTOS into water-hard state.

Re: Attempted summary of "RT patch acceptance" thread

[Bill Huey]

On Sat, Jun 11, 2005 at 12:37:51AM +0200, Andrea Arcangeli wrote:
> Did you read Karim's answer at all?

Reverse to youe did you read message from the original thread at all ?
Karim supports this work and knows this is possible.

> Those RTOS don't have frenetic development in scheduler and all other

SCHED_FIFO isn't effected by it, nor is SCHED_RR. That's another hysterical
claim on your part.

> subsystems and they may be simpler as well, so perhaps you can disable
> L2 and L3 and measure all possible paths that kernel can take in
> invoking your rt code, but I doubt you're going to keep it up with linux
> development in demonstrating the worst case deadline in every possible
> hardware. Measuring all possible paths of a nanokernel is absolutely

Not an issue. Folks do it all of the time here. There's no hysteria
regarding this here. Single image kernel do it all of the time such as
Mach, QNX, LynxOS and friends. 

> RTOS may be fine for cellphones or other stuff where you may not even
> need hard-RT at all, but you just want the lowest possible latency, but
> I'd _never_ use it whenever I need a true deadline, since other more
> reliable solutions are possible without much more complexity.

Read the original thread. Folks have intention of doing more than you
realize at this point. Trivial strawman claims of provable verses
measured latencies are just stupid. The problem is not as wacked you
make it.

[cellphone commentary stripped]
 
> About the fact the kernel can crash, and a driver can corrupt memory,
> that's the difference between ruby and diamond.

Nanokernels crash as well from lack of memory protection. So what ?

You make too many assumptions in domain you have little experience in
and turn it into some assumed fatal argument about single image RT which
is far from reality and the true of the matter.

> I simply think RTOS is an awful solution to hard-RT, RTAI/rtlinux
> designs are much more reliable. There are certainly areas where RTOS is

LynxOS runs on HP printers for real time guarantees all of the time and
is good for general purpose usages as well. Folks use single image kernel
for RT guarantees even before the existence RTAI and RT Linux. Your history
is reversed here. People have been doing this for ages under a single image.

> acceptable, and this is where hard-RT isn't required and you simply want
> the lowest possible latency when invoking syscalls like alsa ioctls. But
> when alsa kernel code is out of the equation I'd never use RTOS to get
> the kernel out of the way.

Hard RT is required in audio. It's require in processing DSP data coming
from those cards. ioctl() paths are short and pretty much directly go to
the driver.

bill

Re: Attempted summary of "RT patch acceptance" thread

[Paul E. McKenney]

On Sat, Jun 11, 2005 at 12:25:01AM +0200, Eric Piel wrote:
> I've read the whole summary and really appreciated it. It's clear, 
> precise and objective.

Glad you like it!

> Just a small change to "1 - QoS":
> 
> >b.	For each service:
> >
> >	i.	Probability of missing a deadline due to software,
> >		ranging from 0 to 1, with the value of 1 corresponding
> >		to the hardest possible hard realtime.
> 
> I think it should be (by reference to how you define probability at the 
> beginning of the section):
> Probability of not missing any deadline due to software

Good catch!  How about the following?

	i.      Probability of meeting a deadline due to software,
		ranging from 0 to 1, with the value of 1 corresponding
		to the hardest possible hard realtime.

Changing "missing" in the original to "meeting".

							Thanx, Paul

> Thanks again for writing this summary,
> Eric
> 

Re: Attempted summary of "RT patch acceptance" thread

[Andrea Arcangeli]

On Fri, Jun 10, 2005 at 04:01:21PM -0700, Bill Huey wrote:
> LynxOS runs on HP printers for real time guarantees all of the time and
> is good for general purpose usages as well. Folks use single image kernel
> for RT guarantees even before the existence RTAI and RT Linux. Your history
> is reversed here. People have been doing this for ages under a single image.

I don't care what folks did for ages, I care what I think is best to do
_now_ with linux, and I don't see why one should go with RTOS when much
simpler and more reliable and _more_performant_ solutions exists when
switching to linux, even for the printers.

> Hard RT is required in audio. It's require in processing DSP data coming
> from those cards. ioctl() paths are short and pretty much directly go to

Those are sort of problems where metal-hard may be simpler to deal with,
if there's a skip they'll record a second time, and you can just use
measurement and proabability there, without providing any guarantee.

Perhaps you can't handle valid criticism cause you have a conflict of
interest.

Re: Attempted summary of "RT patch acceptance" thread

[Bill Huey]

On Sat, Jun 11, 2005 at 12:52:31AM +0200, Andrea Arcangeli wrote:
> Just tell me how can you go to a customer and tell him that your
> linux-RTOS has a guaranteed worst case latency of 50usec.  How can you
> tell that? Did you exercise all possible scheduler paths with cache
> disabled and calculated the worst case latency with rdtsc + math?

Ask Ingo. I'm through with this track and your misinformed comments
about it. We just went through a 300+ plus thread discussing the issues
and you still apparently haven't read through the relevant portions of
it.
 
> Why do you take risks when you can go with much more relaible solutions
> like RTAI and rtlinux?

How are they more reliable ? explain that to me ?

> Said that there are many problems where metal-hard solutions are
> fine, every single app that don't require a worst-case guarantee of
> the worst-case latency is fine with metal-hard, and every time one has
> to call into the alsa ioctl or other complex syscalls also is fine with
> metal hard since the first kmalloc or lock contention inside RT context,
> will send your RTOS into water-hard state.

When you demonstrate a basic understanding of real time programming and
ask a relevant and informed question, then I'll be very happy to explain
this patch, single kernel RT design issues. Until then, I'm declaring
that I'm officially sick of reply to FUD generating comments from folks
that don't understand how this patch works.

bill

Re: Attempted summary of "RT patch acceptance" thread

[Paul E. McKenney]

On Fri, Jun 10, 2005 at 04:38:24PM -0400, Lee Revell wrote:
> On Fri, 2005-06-10 at 08:47 -0700, Paul E. McKenney wrote:
> > > Does the LTP include an RT latency test yet?
> > 
> > Not as far as I know.  I believe that LTP contains primarily pass-fail
> > rather than performance tests, but regardless of where RT latency
> > tests live, I believe that there needs to be a good home for them.
> 
> Maybe I didn't mean LTP, I'm thinking of whatever that popular benchmark
> suite is that people use to identify performance regressions.  OSDL
> something or other?
> 
> The canonical test is "rtc_wakeup", it just sets up a stream of
> interrupts from the RTC, polls on it and measures the delay.  Check the
> list archives for the URL.

OK, http://affenbande.org/~tapas/rtc_wakeup/ is the URL, right?

And LTP might well be a good home for it, I honestly don't know.

I am checking to see if this can be added to the set of tests run
in Martin's automated testing.  If you haven't looked at this, check
out:

http://www.kernel.org/pub/linux/kernel/people/mbligh/abat/regression_matrix.html

This doesn't cover all the various RT variants of Linux, but might
focus some attention on mainline latency issues.

						Thanx, Paul

Re: Attempted summary of "RT patch acceptance" thread

[Bill Huey]

On Sat, Jun 11, 2005 at 01:05:10AM +0200, Andrea Arcangeli wrote:
> I don't care what folks did for ages, I care what I think is best to do
> _now_ with linux, and I don't see why one should go with RTOS when much
> simpler and more reliable and _more_performant_ solutions exists when
> switching to linux, even for the printers.

Covered in the previous mega-flame-war RT acceptance thread. 

> Those are sort of problems where metal-hard may be simpler to deal with,
> if there's a skip they'll record a second time, and you can just use
> measurement and proabability there, without providing any guarantee.
> 
> Perhaps you can't handle valid criticism cause you have a conflict of
> interest.

Look, I don't like being pissy with you, but this has already been
covered in that monster thread from hell. Really, it was already covered.

bill

Re: Attempted summary of "RT patch acceptance" thread

[Paul E. McKenney]

On Fri, Jun 10, 2005 at 04:45:38PM -0400, Lee Revell wrote:
> On Fri, 2005-06-10 at 19:37 +0200, Andrea Arcangeli wrote:
> > You don't need to add it to the document, but as a further pratical
> > example of troublesome hardware besides VGA (could be a software issue
> > and not hardware issue though)  
> 
> The VGA problems were due to (X, not kernel!) driver bugs.  Recent
> versions of X are thought to be OK.
> 
> It's the exact same issue described in this paper, scroll down to
> section 4.5.
> 
> http://www.cs.utah.edu/~regehr/papers/hotos7/hotos7.html
> 
> There's absolutely nothing the kernel or PREEMPT_RT can do about this,
> AFAICT even RTAI would be affected, because X lets userspace drivers
> talk directly to the hardware including wedging the PCI bus.

Cute!

I suppose that Eric Piel's approach would avoid this problem if
implemented on a NUMA machine with all processes making use of video
being affinitied to the NUMA node containing the video card.  ;-)

Sounds like I need to add "antisocial hardware" to the list of
things that need to be inspected to validate realtime latencies.

						Thanx, Paul

Re: Attempted summary of "RT patch acceptance" thread

[Eric Piel]

06/11/2005 01:04 AM, Paul E. McKenney wrote/a écrit:
>>Just a small change to "1 - QoS":
>>
>>
>>>b.	For each service:
>>>
>>>	i.	Probability of missing a deadline due to software,
>>>		ranging from 0 to 1, with the value of 1 corresponding
>>>		to the hardest possible hard realtime.
>>
>>I think it should be (by reference to how you define probability at the 
>>beginning of the section):
>>Probability of not missing any deadline due to software
> 
> 
> Good catch!  How about the following?
> 
> 	i.      Probability of meeting a deadline due to software,
> 		ranging from 0 to 1, with the value of 1 corresponding
> 		to the hardest possible hard realtime.
> 
> Changing "missing" in the original to "meeting".

It sounds strange to me (but english is not my mother tongue), it's like 
hardware was not so good but software helped to recover the situation 
and, eventually, the deadline was met ;-)

What about using the way you wrote it at the beginning of the section:
"Probability of missing a deadline only because of a hardware failure"

Eric

Re: Attempted summary of "RT patch acceptance" thread

[Bill Huey]

On Fri, Jun 10, 2005 at 04:16:47PM -0700, Paul E. McKenney wrote:
> Sounds like I need to add "antisocial hardware" to the list of
> things that need to be inspected to validate realtime latencies.

And anti-social memory controllers (cough G5 Macs)

bill

Re: Attempted summary of "RT patch acceptance" thread

[Andrea Arcangeli]

On Fri, Jun 10, 2005 at 04:08:36PM -0700, Bill Huey wrote:
> On Sat, Jun 11, 2005 at 12:52:31AM +0200, Andrea Arcangeli wrote:
> > Just tell me how can you go to a customer and tell him that your
> > linux-RTOS has a guaranteed worst case latency of 50usec.  How can you
> > tell that? Did you exercise all possible scheduler paths with cache
> > disabled and calculated the worst case latency with rdtsc + math?
> 
> Ask Ingo. I'm through with this track and your misinformed comments

You didn't provide an answer yourself, and you fallback on somebody else
cause there's no valid answer to this question. All I've seen so far are
measurements backed by some statistical significance, that's very far
from the meaning I give to the word _guarantee_.

I fully acknowledge there are some problems that aren't more equally
easily solved with full userland code, for those problems keeping things
simpler by making the kernel more RT aware has a value.

I fully acknowledge that simulating infinite cpus has a value too in
discovering race conditions too ;).

But I personally dislike all things that works by luck, preempt-RT that
aims to provide guarantees about worst case latencies is no exception,
so you shoudln't be surprised if I'm not a RTOS backer for problems that
can be more easily solved with ruby-hard RT designs like RTAI and
rtlinux. I don't care how things have worked in the past on non-linux
OS. Most of the time even current not-RT linux will work fine if it's
idle as it would be on some embedded usages like the printer example.

This even ignoring the fact all those context switch will not be cheap,
kernel can execute a lot more hard-irqs than context switches per
second, and this is another fact. On a 4ghz cpu it doesn't matter, but
on a embedded it can matter, so the more reliable solution is obviously
higher performant too. Of the 50usec it takes to such project on
linuxdevices to execute probably a 10% of that is wasted in the irq
handling itself (ioapic hardware proper stuff).

Anyway I've more fun things to do on than to talk about hard-RT, which
I'm doing just with the aim to provide my little contribution in form of
IMHO valid criticism that you clearly don't appreciate.

Re: Attempted summary of "RT patch acceptance" thread

[Zwane Mwaikambo]

On Fri, 10 Jun 2005, Bill Huey wrote:

> On Fri, Jun 10, 2005 at 04:16:47PM -0700, Paul E. McKenney wrote:
> > Sounds like I need to add "antisocial hardware" to the list of
> > things that need to be inspected to validate realtime latencies.
> 
> And anti-social memory controllers (cough G5 Macs)

What's wrong with the memory controller on the G5?

Re: Attempted summary of "RT patch acceptance" thread

[Bill Huey]

On Fri, Jun 10, 2005 at 05:36:10PM -0600, Zwane Mwaikambo wrote:
> What's wrong with the memory controller on the G5?

DMA starvation and please don't ask where I got this information. :)

bill

Re: Attempted summary of "RT patch acceptance" thread

[Lee Revell]

On Fri, 2005-06-10 at 16:41 -0700, Bill Huey wrote:
> On Fri, Jun 10, 2005 at 05:36:10PM -0600, Zwane Mwaikambo wrote:
> > What's wrong with the memory controller on the G5?
> 
> DMA starvation

DMA starvation has also been observed with some PC chipsets during
PREEMPT_RT development and the cause was traced to the UDMA133
controller.  The symptom is latency traces that seem to go in extreme
"slow motion".  Check the archives for the hardware affected.

Lee 

Re: Attempted summary of "RT patch acceptance" thread

[Paul E. McKenney]

On Fri, Jun 10, 2005 at 07:37:28PM +0200, Andrea Arcangeli wrote:
> Hello Paul,
> 
> kudos for your very nice RT documents (as usual ;)

Thank you, glad you liked it!

> On Fri, Jun 10, 2005 at 08:47:46AM -0700, Paul E. McKenney wrote:
> > Good point -- I certainly need to add a disclaimer to the effect that
> > common hardware (such as VGA, last I checked some months ago) can
> 
> > a.	Quality of service: soft realtime, with timeframe of 100s of
> > 	microseconds for task scheduling and interrupt handling, but
> > 	-only- for very carefully restricted hardware configurations
> > 	that exclude problematic devices and drivers (such as VGA)
> > 	that can cause latency bumps of tens or even hundreds of
> > 	milliseconds (-not- microseconds).  Furthermore, the software
> > 	configuration of such systems must be carefully controlled,
> > 	for example, doing a "kill -1" traverses the entire task list
> > 	with tasklist_lock held (see kill_something_info()), which might
> > 	result in disappointing latencies in systems with very large
> > 	numbers of tasks.  System services providing I/O, networking,
> > 	task creation, and VM manipulation can take much longer.  A very
> > 	small performance penalty is exacted, since spinlocks and RCU
> > 	must suppress preemption.
> > 
> > Does this help, or are there other CONFIG_PREEMPT latency issues that
> > need to be called out?
> 
> You don't need to add it to the document, but as a further pratical
> example of troublesome hardware besides VGA (could be a software issue
> and not hardware issue though) I'd like to make the example of the irq
> handler of the uhci usb1.1 controller that takes up to 8msec on a 1ghz
> atlhon UP system, and there's nothing that PREEMPT can do about it since
> it's an hard-irq. This latency keeps triggering a few times per second
> on my firewall for the last few years.

After reading the ensuing thread, I am not sure what example to use!
Maybe I should just pick random examples and see if they tend to get
fixed?  ;-)

> preempt-RT _can_ do something about it but only _if_ people hacks the
> drivers properly and makes sure to call local_irq_save_nort instead of
> local_irq_save and other explicit changes like that, things that if
> missing are noticeable only during measurements with preempt-RT config
> option enabled (hence the metal-hard classification of preempt-RT and
> not ruby-hard definition).
> 
> See the tg3 updates required to be safe with preempt-RT without breaking
> hard-RT as a clear example of how preempt-RT is weak:
> 
> --- linux/drivers/net/tg3.c.orig
> +++ linux/drivers/net/tg3.c
> @@ -3229,9 +3229,9 @@ static int tg3_start_xmit(struct sk_buff
>          * So we really do need to disable interrupts when taking
>          * tx_lock here.
>          */
> -       local_irq_save(flags);
> +       local_irq_save_nort(flags);
>         if (!spin_trylock(&tp->tx_lock)) { 
> -               local_irq_restore(flags);
> +               local_irq_restore_nort(flags);
>                 return NETDEV_TX_LOCKED; 
>         } 
>  
> There's no apparent reason why all those changes should be required to
> get hard-RT.
> 
> Both RTAI and rtlinux _don't_ require to change all those drivers to get
> the guarantee that the kernel will get out of the way within a certain
> nanoseconds deadline interval.
> 
> Furthermore with the scheduler, mutex and context switch code into the
> equation, it gets more and more difficult to calculate with math the max
> latency that preempt-RT will provide, while it's almost trivial to do
> that with RTAI/rtlinux given only the nanokernel code runs before the
> hard-RT code is invoked and there are not many paths to test, so one has
> to disable the cache and just measure the few possible nanokenrel paths.
> (as usual when speaking about hard-RT I've robots in mind, and not audio
> code that will call into the alsa ioctls)
> 
> This below is the kind of stuff where I wouldn't even dream to replace
> a ruby-hard rtlinux/RTAI with a weaker metal-hard and possibly
> underperformant (cause scheduling hard-irq in userland and scheduling
> instead of spinning isn't going to be cheap in smp) preempt-RT solution:
> 
> http://linuxdevices.com/articles/AT7871136191.html
> 
> In the above RTAI should have made it as well as rtlinux of course.

As long as the nested-OS or migration-between-OSes approaches prevents
Linux from disabling interrupts, then interrupts and preemption-disabling
are not a problem.  As noted later in this thread, there is still the
possibility of hardware stalls, which seems to affect all the approaches,
with the possible exception of migration-within-OS between CPUs that
don't share the offending hardware.  Maybe a dual AMD?

						Thanx, Paul

Re: Attempted summary of "RT patch acceptance" thread

[Paul E. McKenney]

On Sat, Jun 11, 2005 at 01:23:34AM +0200, Eric Piel wrote:
> 06/11/2005 01:04 AM, Paul E. McKenney wrote/a écrit:
> >>Just a small change to "1 - QoS":
> >>
> >>
> >>>b.	For each service:
> >>>
> >>>	i.	Probability of missing a deadline due to software,
> >>>		ranging from 0 to 1, with the value of 1 corresponding
> >>>		to the hardest possible hard realtime.
> >>
> >>I think it should be (by reference to how you define probability at the 
> >>beginning of the section):
> >>Probability of not missing any deadline due to software
> >
> >
> >Good catch!  How about the following?
> >
> >	i.      Probability of meeting a deadline due to software,
> >		ranging from 0 to 1, with the value of 1 corresponding
> >		to the hardest possible hard realtime.
> >
> >Changing "missing" in the original to "meeting".
> 
> It sounds strange to me (but english is not my mother tongue), it's like 
> hardware was not so good but software helped to recover the situation 
> and, eventually, the deadline was met ;-)

Hmmm...  It would appear that English's being my mother tongue is not
helping me as much as one might hope.  ;-)

> What about using the way you wrote it at the beginning of the section:
> "Probability of missing a deadline only because of a hardware failure"

Good point, I may just need to invert the whole thing, so that it
becomes something like:

	i.	Probability of missing a deadline due to software,
		ranging from 0 to 1, with the value of 0 corresponding
		to the hardest possible hard realtime.

But then the "p^n" becomes "1-(1-p)^n".  Bleah.

OK, how about the following?

	i.	Probability of meeting a deadline in absence of hardware
		failure, ranging from 0 to 1, with the value of 1
		corresponding to the hardest possible hard realtime.

							Thanx, Paul

Re: Attempted summary of "RT patch acceptance" thread

[Paul E. McKenney]

On Fri, Jun 10, 2005 at 04:26:28PM -0700, Bill Huey wrote:
> On Fri, Jun 10, 2005 at 04:16:47PM -0700, Paul E. McKenney wrote:
> > Sounds like I need to add "antisocial hardware" to the list of
> > things that need to be inspected to validate realtime latencies.
> 
> And anti-social memory controllers (cough G5 Macs)

OK, the list now reads:

	Each of the following categories of code might need to be
	inspected:

	a.	The low-level interrupt-handing code.

	b.	The realtime process scheduler.

	c.	Any code that disables interrupts.

	d.	Any code that disables preemption.

	e.	Any code that holds a lock, mutex, semaphore, or other
		resource that is needed by the code implementing your
		new feature.

	f.	Any code that manipulates hardware that can stall the
		bus, delay interrupts, or otherwise interfere with
		forward progress.  Note that it is also necessary to
		inspect user-level code that directly manipulates such
		hardware.

I added point "f".  Does that cover it?

						Thanx, Paul

Re: Attempted summary of "RT patch acceptance" thread

[Eric Piel]

11.06.2005 02:59, Paul E. McKenney wrote/a écrit:
> On Sat, Jun 11, 2005 at 01:23:34AM +0200, Eric Piel wrote:
>>What about using the way you wrote it at the beginning of the section:
>>"Probability of missing a deadline only because of a hardware failure"
> 
> 
> Good point, I may just need to invert the whole thing, so that it
> becomes something like:
> 
> 	i.	Probability of missing a deadline due to software,
> 		ranging from 0 to 1, with the value of 0 corresponding
> 		to the hardest possible hard realtime.
> 
> But then the "p^n" becomes "1-(1-p)^n".  Bleah.
Yes, it seems language doesn't fit well with mathematics ;-)

> 
> OK, how about the following?
> 
> 	i.	Probability of meeting a deadline in absence of hardware
> 		failure, ranging from 0 to 1, with the value of 1
> 		corresponding to the hardest possible hard realtime.
> 
Sounds good!

Eric

Re: Attempted summary of "RT patch acceptance" thread

[Paul E. McKenney]

On Sat, Jun 11, 2005 at 01:29:55AM +0200, Andrea Arcangeli wrote:
> On Fri, Jun 10, 2005 at 04:08:36PM -0700, Bill Huey wrote:
> > On Sat, Jun 11, 2005 at 12:52:31AM +0200, Andrea Arcangeli wrote:
> > > Just tell me how can you go to a customer and tell him that your
> > > linux-RTOS has a guaranteed worst case latency of 50usec.  How can you
> > > tell that? Did you exercise all possible scheduler paths with cache
> > > disabled and calculated the worst case latency with rdtsc + math?
> > 
> > Ask Ingo. I'm through with this track and your misinformed comments
> 
> You didn't provide an answer yourself, and you fallback on somebody else
> cause there's no valid answer to this question. All I've seen so far are
> measurements backed by some statistical significance, that's very far
> from the meaning I give to the word _guarantee_.

I just -know- I am going to regret stepping into the middle of
this one...

[Hey, Karim!!!  Any extra room in that bunker of yours?]

All I have seen Ingo claim is that -some- hardware configurations running
-some- specific workloads had excellent -measured- maximum scheduling
latencies.  I have not seen Ingo claim that he has mathematically proven
that CONFIG_PREEMPT_RT's worst-case scheduling latency is bounded, let
alone make any claim of what such a mathematically proven bound might be.
Nor have I seen Ingo claim that CONFIG_PREEMPT_RT has passed any sort of
industry-standard acceptance test.  If I missed such a mathematical proof,
or such an industry-standard acceptance test, please send me the URL.

Ingo -has- greatly reduced the amount of kernel code that must
be inspected to guarantee realtime response, and this is a great
accomplishment and a great contribution.  I believe that his approach
is more than "good enough" for a great number of applications that are
traditionally thought of as "hard realtime".  But, unless I missed
something, it is not mathematically proven, nor is it in any way
"certified".

Some of the approaches involving separate RTOSes -can- legitimately claim
have mathematically proven worst-case scheduling latencies.  For example,
the dual-OS/dual-core approach can do so, particularly in the case where
the "RTOS" is a tight loop written on bare metal in hand-coded assembly.
One might consider this to be a trivial example, perhaps even a stupid
example, but there are a lot of apps out there that use exactly this
approach.

The migration-between-OSes approach, such as RTAI Fusion, might also be
able to mathematically guarantee RTOS scheduling latencies.  And, unlike
the dual-OS/dual-core approaches, it does provide a single environment
to applications.  However, it still results in two separate OS instances
to administer, and, as near as I can tell, the RTOS has to stick its
hands so deeply into Linux's internals that it might as well be part
of the Linux kernel.  Unless I am missing something, any sort of bug
in the Linux kernel has a reasonable chance of messing up the RTOS,
since the RTOS must paw through Linux's tasks.

Can we say that one of these approaches is definitely "good enough" for
all reasonable Linux RT work, and that we should therefore stop working on 
the other approach?

I might well be missing something, but I don't believe so, at least
not yet.

In the meantime, the more time anyone spends sending flames on LKML, the
less time that person is putting into improving his/her favorite approach.
So the greater number of flames one sends, the smaller one's favorite
approach's chance of success!

							Thanx, Paul

> I fully acknowledge there are some problems that aren't more equally
> easily solved with full userland code, for those problems keeping things
> simpler by making the kernel more RT aware has a value.
> 
> I fully acknowledge that simulating infinite cpus has a value too in
> discovering race conditions too ;).
> 
> But I personally dislike all things that works by luck, preempt-RT that
> aims to provide guarantees about worst case latencies is no exception,
> so you shoudln't be surprised if I'm not a RTOS backer for problems that
> can be more easily solved with ruby-hard RT designs like RTAI and
> rtlinux. I don't care how things have worked in the past on non-linux
> OS. Most of the time even current not-RT linux will work fine if it's
> idle as it would be on some embedded usages like the printer example.
> 
> This even ignoring the fact all those context switch will not be cheap,
> kernel can execute a lot more hard-irqs than context switches per
> second, and this is another fact. On a 4ghz cpu it doesn't matter, but
> on a embedded it can matter, so the more reliable solution is obviously
> higher performant too. Of the 50usec it takes to such project on
> linuxdevices to execute probably a 10% of that is wasted in the irq
> handling itself (ioapic hardware proper stuff).
> 
> Anyway I've more fun things to do on than to talk about hard-RT, which
> I'm doing just with the aim to provide my little contribution in form of
> IMHO valid criticism that you clearly don't appreciate.
> 

Re: Attempted summary of "RT patch acceptance" thread

[Paul E. McKenney]

On Sat, Jun 11, 2005 at 03:38:04AM +0200, Eric Piel wrote:
> 11.06.2005 02:59, Paul E. McKenney wrote/a écrit:
> >On Sat, Jun 11, 2005 at 01:23:34AM +0200, Eric Piel wrote:
> >>What about using the way you wrote it at the beginning of the section:
> >>"Probability of missing a deadline only because of a hardware failure"
> >
> >
> >Good point, I may just need to invert the whole thing, so that it
> >becomes something like:
> >
> >	i.	Probability of missing a deadline due to software,
> >		ranging from 0 to 1, with the value of 0 corresponding
> >		to the hardest possible hard realtime.
> >
> >But then the "p^n" becomes "1-(1-p)^n".  Bleah.
> Yes, it seems language doesn't fit well with mathematics ;-)

Hey, it could be worse.  I am just glad that it is not normally necessary
to do integration by parts, trig substitutions, or Laplace transforms
on English sentences!!!  ;-)

> >OK, how about the following?
> >
> >	i.	Probability of meeting a deadline in absence of hardware
> >		failure, ranging from 0 to 1, with the value of 1
> >		corresponding to the hardest possible hard realtime.
> >
> Sounds good!

OK, I made the change!  Thank you for catching this one, good eyes!

						Thanx, Paul

Re: Attempted summary of "RT patch acceptance" thread

[Karim Yaghmour]

Paul E. McKenney wrote:
> [Hey, Karim!!!  Any extra room in that bunker of yours?]

If you bring your own doritos and coffee, there just might be some room
to sleep beneath the computer-controlled mechanical saw bench I've set
up to evaluate the real-time responsiveness of Linux ...

Karim
-- 
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 1-866-677-4546

Re: Attempted summary of "RT patch acceptance" thread

[Paul E. McKenney]

On Fri, Jun 10, 2005 at 11:58:27PM +0200, Eric Piel wrote:
> 06/09/2005 04:20 AM, Paul E. McKenney wrote/a écrit:
> >>Concerning the QoS, we have been able to obtain hard realtime, at least
> >>very firm real-time. Tests were conducted over 8 hours on IA-64 and x86
> >>and gave respectively 105µs and 40µs of maximum latency. Not as good as
> >>you have mentioned but mostly of the same order :-)
> >
> > Quite impressive!  So, does this qualify as "ruby hard", or is it only
> > "metal hard"?  ;-)
> Well, you have to consider that this is still full Linux running. All 
> the best we can do is to not make it crash or hung more than the vanilla 
> kernel, it's still vulnerable to any bug of any driver :-/ In addition, 
> I highly doubt this approach can ever have an implementation were the 
> maximum latency is theoritically proven. The best we have is just 
> measurements of the system running with high loads during very long time.

Hmmm...  Might one make a theoretical proof that assumed the absense
of bugs?  Wholly unrealistic, I know, but seems like a reasonable
question to ask.  Besides, isn't theory -supposed- to be wholly
unrealistic?

> > The service measured was process scheduling, right?
> Yes, on IA64, from the hardware IRQ fireing to process scheduling (on 
> x86 it's from kernel IRQ handling to process scheduling).

Cool!

> >>Concerning the "e. fault isolation", on our implementation, holding a
> >>lock, mutex or semaphore will automatically migrate the task, therefore
> >>it's not a problem. Of course, some parts of the kernel that cannot be
> >>migrated might take a lock, namely the scheduler. For the scheduler, we
> >>modified most of the data structures requiring a lock so that they can
> >>be accessed locklessly (it's the hardest part of the implementation).
> >
> >
> > Are the non-migrateable portions of the scheduler small enough that
> > one could do a worst-case timing analysis?  Preferably aided by some
> > automation...
> Well, ARTiS only modifies the schedule() function but there is probably 
> too much possible interaction to really be able to prove anything (the 
> fact that it's a SMP system doesn't help!).

SMP does make it more complex, but not hopeless.  If you can bound the
number of threads, and if you can guarantee FIFO queuing on locks,
then the locks can have bounded wait times.  Not that any of the
high-performance spinlocks actually have this property...

> > One approach would be to mark the migrated task so that it returns
> > to the realtime CPU as soon as it completes the realtime-unsafe
> > operation.
> We use a different approach: keep (small) statistics of the task doing 
> often lock and the one that are more "computational". Then we migrate in 
> priority the tasks that don't do locks. Your suggestion could be used 
> at the same time but it might not be so efficient anymore. Additionally, 
> in the current implementation, it's not so easy to know when a task 
> which is running can go back to a RT CPU.

OK.  I suppose you could migrate it back as soon as it returned to
user-mode execution.  Might or might not be useful.  The other approach
would be once it dropped all of its locks, but, as you say, this could
result in a large number of migrations...

> > Another approach is to insert a virtualization layer (think in terms of
> > a very cut-down variant of Xen) that tells the OS that there are two 
> CPUs.
> > This layer then gives realtime service to one, but not to the other.
> > That way, the OS thinks that it has two CPUs, and can still do the
> > migration tricks despite having only one real CPU.
> Simulation of an SMP on a UP? This sounds quite heavy but it might be 
> interesting to try :-)

It can be quite light-weight -- you just have to carefully lie to the
OS about what CPU it is running on.  And no doubt fix a bunch of places
where it makes assumptions about the two CPUs being real CPUs.  :-(

						Thanx, Paul

> > Anyway, interesting approach!
> Thanks,
> 
> Eric
> 

Re: Attempted summary of "RT patch acceptance" thread

[Paul E. McKenney]

On Fri, Jun 10, 2005 at 09:50:55PM -0400, Karim Yaghmour wrote:
> 
> Paul E. McKenney wrote:
> > [Hey, Karim!!!  Any extra room in that bunker of yours?]
> 
> If you bring your own doritos and coffee, there just might be some room
> to sleep beneath the computer-controlled mechanical saw bench I've set
> up to evaluate the real-time responsiveness of Linux ...

Cool!  I will bring a few packages of hot dogs!!!

						Thanx, Paul

Re: Attempted summary of "RT patch acceptance" thread

[Ingo Molnar]

* Andrea Arcangeli <andrea@suse.de> wrote:

> On Fri, Jun 10, 2005 at 03:37:24PM -0700, Bill Huey wrote:
> > Some of the comments from various folks are just intolerably paranoid
> 
> Just tell me how can you go to a customer and tell him that your 
> linux-RTOS has a guaranteed worst case latency of 50usec. [...]

Andrea, please just stop these scare tactics, it's getting boring. First 
you came with incorrect statements about locks, then you came with the 
patent boogeyman, then you came with the driver bogosity, and now you 
come with "but ... [sputter] ... but ... where's the _guarantee_?".

Andrea, _please_ lean back and read back some of your earlier arguments.
To make it easier for you and to refresh your recollection, let me give
you a selection, just so that you can see why your sentiment creates
flames and personal responses:

  '[...] how can preempt-RT ever become hard-RT when a simple lock hangs 
   it.' (Andrea Arcangeli, May 31, 2005)

  'Exactly, they're simply not remotely comparable, a VM improvement may
   break preempt-RT anytime, it's just too easy to screw things up and
   invalidate all "measurements".' (Andrea Arcangeli, Jun 1, 2005)

  'Indeed, that's why I believe hard-RT with preempt-RT is just a 
   joke.' (Andrea Arcangeli, May 31, 2005)

  'Then I'm afraid preempt-RT infringe on the patent that they take 
   after years of doing that in linux. I'm not a lawyer but you may want 
   to check before investing too much on this for the next 15 years.' 
   (Andrea Arcangeli, Jun 1, 2005)

  'Why do you take risks when you can go with much more relaible 
   solutions like RTAI and rtlinux?' (Andrea Arcangeli, Jun 11, 2005)

I have to say, based on these statements you could soon become a fine 
replacement for Blake Stowell or the Iraqi Information Minister :-)

just to make it plain and obvious: there are two things that are true 
about the above snippets: 1) as far as they were technical comments they 
are all dead wrong, and 2) you never apologized for them. Wouldnt you 
yourself become a bit ... touchy if this happened to you on such a 
widespread basis? Please give me an answer, how much unjust accusations 
do i have to take before i have the _right_ to flame you, hm? :-)

thinking about the root causes of your behavior, it seems to me that you 
have a real inner trouble admitting mistakes (and that's not only true 
for this incident) - even though mistakes are human and i do at least a 
dozen mistakes every day and sometimes i screw up really bad. Like 3 
months ago when i criticised SECCOMP [your project] in an uninformed and 
thus unjust way. Perhaps we are getting this from you because you are 
still hurting from that SECCOMP incident? If it's about SECCOMP then i'd 
like to apologize again: i was wrong about SECCOMP and it's a fine and 
promising project. Okay?

if you review the PREEMPT_RT technology (with a cool mind) i do think 
you'll eventually come to the conclusion that it is actually a pretty 
nifty idea and implementation, with some pretty good potential :-) It's 
not trying to be the holy grail, it wont (nor does it want to) replace 
nanokernels, but it actually has some thinking behind it and is 
definitely useful to alot of people. And that is what matters. Your 
feelings about it or me dont really count.

	Ingo

Re: Attempted summary of "RT patch acceptance" thread

[Andrea Arcangeli]

On Fri, Jun 10, 2005 at 06:07:55PM -0700, Paul E. McKenney wrote:
> 	f.	Any code that manipulates hardware that can stall the
> 		bus, delay interrupts, or otherwise interfere with
> 		forward progress.  Note that it is also necessary to
> 		inspect user-level code that directly manipulates such
> 		hardware.
> 
> I added point "f".  Does that cover it?

Yes. F is really a bad problem if it really can cause DMA starvation on
the memory bus on some arch as stated on this thread. Hopefully
measurements are good enough to rule it out and there will be no corner
cases triggering once in a while.

Re: Attempted summary of "RT patch acceptance" thread

[Andrea Arcangeli]

On Fri, Jun 10, 2005 at 06:41:33PM -0700, Paul E. McKenney wrote:
> I just -know- I am going to regret stepping into the middle of
> this one...

Never mind, I keep regretting it myself. Perhaps it was better if I
avoided posting here, and by now people would think preempt-RT was the
holy grail of hard-RT and that would obsolete RTAI/rtlinux, when infact
it's an inferior solution from from a reliability/guarantee standpoint
and from a performance standpoint too (as some measurements showed
already too) [and local_irq_disable would still be not emulated in
software]

> All I have seen Ingo claim is that -some- hardware configurations running
> -some- specific workloads had excellent -measured- maximum scheduling
> latencies.  I have not seen Ingo claim that he has mathematically proven
> that CONFIG_PREEMPT_RT's worst-case scheduling latency is bounded, let
> alone make any claim of what such a mathematically proven bound might be.

Indeed. As I stated in my first relevant post to this thread on "Tue, 31
May 2005 18:11:57 +0200" _before_ most of the other quotes that I received
today, and before realizing the _irq part of the spin_lock_irq was
already emulated in software and before realizing that the rtlinux
patent doesn't forbid to emulate in software the _irq part of the
spin_lock_irq because I'm not really good at reading all the patent
claims but I focused only on the patent concept (and regardless there is
apparently somebody who rightfully paid lawyers for research in that area):

	For the determinism, you could do what Ingo did so far, that is to
	"measure" but there's no way a "measurement" can provide an hard-RT
	guarantee. The "measure" way is great for the lowlatency patches, and to
	try to eliminate the bad-latencies paths, but it _can't_ guarantee a 
	"worst-case-latency".

This is what I posted in my _second_ email to this thread. Of course
nobody is quoting the above, they picked other later emails, but if you
go back to the second email (and first relevant one) I posted here,
you'll see this text at the end.  Sorry for that mistake about the _irq
part and the related patent claims that I'm not good at reading cause
it's written by lawyers, but that mistake made no difference to the
bottom line cause local_irq_disable was still disable the hard irqs
without being emulated in software. I was reading local_irq_disable that
wasn't touched (local_irq_disable is the _first_ thing I check in every
RT patch on earth) and so I assumed no _irq disable was emulated. My bad
but it didn't change the bottom line.

Now even local_irq_disable can be emulated in software (incidentally
happened after my suggestions last week) but that still leaves the above
issue in my second email unsolved.

It's not like I made up this "guarante" of worst case latency argument
after realizing that _irq was emulated and after local_irq_disable has
been emulated too, as somebody apparently seems to have described today
by providing a list of quotes ignoring the above argument that was the
core of my first relevant email to this thread.

If you check the timestamp, I mentioned the above core issue, way
_before_ going into the _irq details.

> Ingo -has- greatly reduced the amount of kernel code that must
> be inspected to guarantee realtime response, and this is a great
> accomplishment and a great contribution.  I believe that his approach

Indeed.

> is more than "good enough" for a great number of applications that are
> traditionally thought of as "hard realtime".  But, unless I missed
> something, it is not mathematically proven, nor is it in any way
> "certified".

Yes, it could be that 99% of the app that traditionally claims to need
hard-RT will be fully covered by preempt-RT (assuming the cpu can keep
up with the slowdown), but when I talk about hard-RT I only think at
those apps like the linuxdevices article that beats at 50usec. To me
hard-RT is all about guaranteeing to provide a deadline, the rest I call
it lowlatency patches. preempt-RT is obviously a very great improvement
to the lowlatency soft-RT approach and I agree a lot of those apps
thought to need hard-RT will be fine with it.

But one should evaluate if an hard-RT guarantee is truly needed or not
before picking preempt-RT, the performance should be evaluated as well
cause irq handling is much faster than a context switch.

The slowdown numbers posted so far are quite impressive as well.

Think that in some recent research I'm doing, I'm looking to make the
semaphore a spinlock to boost performance, preempt-RT is the opposite ;)

> Some of the approaches involving separate RTOSes -can- legitimately claim
> have mathematically proven worst-case scheduling latencies.  For example,
> the dual-OS/dual-core approach can do so, particularly in the case where
> the "RTOS" is a tight loop written on bare metal in hand-coded assembly.
> One might consider this to be a trivial example, perhaps even a stupid
> example, but there are a lot of apps out there that use exactly this
> approach.
>
> The migration-between-OSes approach, such as RTAI Fusion, might also be
> able to mathematically guarantee RTOS scheduling latencies.  And, unlike
> the dual-OS/dual-core approaches, it does provide a single environment
> to applications.  However, it still results in two separate OS instances
> to administer, and, as near as I can tell, the RTOS has to stick its
> hands so deeply into Linux's internals that it might as well be part
> of the Linux kernel.  Unless I am missing something, any sort of bug
> in the Linux kernel has a reasonable chance of messing up the RTOS,
> since the RTOS must paw through Linux's tasks.

Yes. I'm a beliver in the dual-OS/dual-core model, as the only one who
can reasonably simply guarantee a deadline.

The single image RTOS has much larger pieces of complex code involved in
the RT equation (in your list you didn't mention the mutex code itself),
plus this codes like the scheduler changes all the time (it's not like
the irq code that pratically never changes in stable cycles), and it
slowsdown quite a bit as well at emulating infinite cpus and
overscheduling at every contention.

> Can we say that one of these approaches is definitely "good enough" for
> all reasonable Linux RT work, and that we should therefore stop working on 
> the other approach?
> 
> I might well be missing something, but I don't believe so, at least
> not yet.

I agree with you. Perhaps I didn't make my point of view clear enough on
the good things of preempt-RT. preempt-RT is a great feature for some
apps.

I _only_ criticise people who wants to put preempt-RT on par with
guaranteed hard-RT solutions like RTAI/rtlinux that can easily provide a
guaranteed deadline, while preempt-RT can only proviede measurements
backed by statistically signficance, which is quite far away from a
guarantee. Plus it's much slower as well. But apparently some people
here, wants you to believe that preempt-RT will be exactly guaranteed
hard-RT as RTAI/rtlinux, and they can't handle it when I say this is not
the case. This is the only thing I'm objecting as stated in my first
relevant email of the thread partly quoted above.

I'm sure I've ever said that preempt-RT is useless and there's no application
for it, infact I stated _multiple_ times there are cases like alsa where
preempt-RT is the very best that one can do.

> In the meantime, the more time anyone spends sending flames on LKML, the
> less time that person is putting into improving his/her favorite approach.

Well, I'm not in RT development anyway so that's not my case, but of
course I'm could spend my time much better anyway ;)

Thanks again for your appreciated RT efforts Paul!

Re: Attempted summary of "RT patch acceptance" thread

[Andrea Arcangeli]

On Sat, Jun 11, 2005 at 07:23:19AM +0200, Ingo Molnar wrote:
> Andrea, please just stop these scare tactics, it's getting boring. First 

These are not scare tactics, infact if you use RTOS for something where
a deadline has to be math guaranteed, and you certainly know that's not
possible to guarantee a deadline with preempt-RT if the OS is under
stress at the same time, and something goes wrong and I get a damage
from it, be sure I'd consider you liable for recommending me to use
solution that you know you can't demonstrate as correct, when fully
reliable solutions were available and you knew it (for free as well as
preempt-RT).

> you came with incorrect statements about locks, then you came with the 
> patent boogeyman, then you came with the driver bogosity, and now you 
> come with "but ... [sputter] ... but ... where's the _guarantee_?".

I raised the "guarantee" argument in the second email I posted to this
thread in explanataion of the statement "don't get the  idea of using
preempt-RT where hard-RT is required.". the staement "don't get the  
idea of using preempt-RT where hard-RT is required." comes from the very
_first_ email I posted to this thread.

I don't see why you attempt to reverse the truth and you try to picture
me as making up the next scare tactic of the day, when all posts are
public and with timestamps and when I raised the "_guarantee_" at the
very to of the thread, _before_ talking about _irq or local_irq_disable
or the patent.

The reason I got into this thread is not some economic interest that
some of the preempt-RT developers certainly has, nor any personal
feeling about you. I've said it because I wouldn't like to get near a
robot where the deadlines have to be met thanks to preempt-RT.

You've seen Bill answer when I've said I wouldn't even dream to use
preempt-RT in the project that needs the 50usec latency right? That's
the kind of thing that made me jump into this thread.

The rest of your email IMHO doesn't deserve comments except for this:

> if you review the PREEMPT_RT technology (with a cool mind) i do think 
> you'll eventually come to the conclusion that it is actually a pretty 
> nifty idea and implementation, with some pretty good potential :-) It's 

I've never stated otherwise, I've said _several_ times that in realtime
apps like audio and multimedia in general, preempt-RT is certainly the
best one can do if syscalls are involved anyways for a multitude of
reasons. If you were a musician and you were asking me for a realtime
solution I would sure point to preempt-RT, no argument about that. Even
if local_irq_disable would still do an hard cli, that's still better
than lowlatency alone, no? preempt can't get the irqs out of the way,
preempt-RT can, and that will help audio if you've a usb modem like mine
where the hardirq takes 8msec.

preempt-RT is a great feature at debugging as well, it is a clever
design as well.

> not trying to be the holy grail, it wont (nor does it want to) replace 
> nanokernels, but it actually has some thinking behind it and is 
> definitely useful to alot of people. And that is what matters. Your 

This is the point I tried to make all the way long. Some people here
clearly wants to use preempt-RT in all the domain of nanokernels and
that's wrong IMHO (again see Bill's answer when I've said I wouldn't
dream to use preempt-RT in the 50usec project on linuxdevices).

This is the source of all the flamewar as far as I can tell, and I'm
glad that even despite your insults, at the very end you agree too.

Re: Attempted summary of "RT patch acceptance" thread

[Paul E. McKenney]

On Sat, Jun 11, 2005 at 05:16:46PM +0200, Andrea Arcangeli wrote:
> On Fri, Jun 10, 2005 at 06:07:55PM -0700, Paul E. McKenney wrote:
> > 	f.	Any code that manipulates hardware that can stall the
> > 		bus, delay interrupts, or otherwise interfere with
> > 		forward progress.  Note that it is also necessary to
> > 		inspect user-level code that directly manipulates such
> > 		hardware.
> > 
> > I added point "f".  Does that cover it?
> 
> Yes. F is really a bad problem if it really can cause DMA starvation on
> the memory bus on some arch as stated on this thread. Hopefully
> measurements are good enough to rule it out and there will be no corner
> cases triggering once in a while.

Agreed!

But who knows?  Maybe we can convince some HW manufacturers to make
their memory and I/O systems better.  So it might also be good to have
tests that forced the conditions as well as tests that avoid them.
Though such tests do sound horribly hardware dependent...

						Thanx, Paul

Re: Attempted summary of "RT patch acceptance" thread

[Paul E. McKenney]

On Sat, Jun 11, 2005 at 05:54:59PM +0200, Andrea Arcangeli wrote:
> On Fri, Jun 10, 2005 at 06:41:33PM -0700, Paul E. McKenney wrote:
> > I just -know- I am going to regret stepping into the middle of
> > this one...
> 
> Never mind, I keep regretting it myself. Perhaps it was better if I
> avoided posting here, and by now people would think preempt-RT was the
> holy grail of hard-RT and that would obsolete RTAI/rtlinux, when infact
> it's an inferior solution from from a reliability/guarantee standpoint
> and from a performance standpoint too (as some measurements showed
> already too) [and local_irq_disable would still be not emulated in
> software]

Well, I probably should have edited the "TO" and "CC" lines to make it
clear that I was not singling you out.  Please accept my apologies for
making it appear that I was singling you out.

But at least we all know that both you and Bill care deeply about Linux!  ;-)

> > All I have seen Ingo claim is that -some- hardware configurations running
> > -some- specific workloads had excellent -measured- maximum scheduling
> > latencies.  I have not seen Ingo claim that he has mathematically proven
> > that CONFIG_PREEMPT_RT's worst-case scheduling latency is bounded, let
> > alone make any claim of what such a mathematically proven bound might be.
> 
> Indeed. As I stated in my first relevant post to this thread on "Tue, 31
> May 2005 18:11:57 +0200" _before_ most of the other quotes that I received
> today, and before realizing the _irq part of the spin_lock_irq was
> already emulated in software and before realizing that the rtlinux
> patent doesn't forbid to emulate in software the _irq part of the
> spin_lock_irq because I'm not really good at reading all the patent
> claims but I focused only on the patent concept (and regardless there is
> apparently somebody who rightfully paid lawyers for research in that area):

CONFIG_PREEMPT_RT continues to be a bit of a learning experience for me
as well.  As to patent claims, don't get me started...

> 	For the determinism, you could do what Ingo did so far, that is to
> 	"measure" but there's no way a "measurement" can provide an hard-RT
> 	guarantee. The "measure" way is great for the lowlatency patches, and to
> 	try to eliminate the bad-latencies paths, but it _can't_ guarantee a 
> 	"worst-case-latency".
> 
> This is what I posted in my _second_ email to this thread. Of course
> nobody is quoting the above, they picked other later emails, but if you
> go back to the second email (and first relevant one) I posted here,
> you'll see this text at the end.  Sorry for that mistake about the _irq
> part and the related patent claims that I'm not good at reading cause
> it's written by lawyers, but that mistake made no difference to the
> bottom line cause local_irq_disable was still disable the hard irqs
> without being emulated in software. I was reading local_irq_disable that
> wasn't touched (local_irq_disable is the _first_ thing I check in every
> RT patch on earth) and so I assumed no _irq disable was emulated. My bad
> but it didn't change the bottom line.
> 
> Now even local_irq_disable can be emulated in software (incidentally
> happened after my suggestions last week) but that still leaves the above
> issue in my second email unsolved.
> 
> It's not like I made up this "guarante" of worst case latency argument
> after realizing that _irq was emulated and after local_irq_disable has
> been emulated too, as somebody apparently seems to have described today
> by providing a list of quotes ignoring the above argument that was the
> core of my first relevant email to this thread.

Yep, I do remember several emails in the earlier thread that seemed
to claim hard-realtime response.  One might be able to create some
sort of software tool that scanned the required code to produce such
guarantee, but I have not heard of anyone actually doing this with
any Linux-based realtime scheme.

> If you check the timestamp, I mentioned the above core issue, way
> _before_ going into the _irq details.

This does indeed match my memory of the earlier thread.

> > Ingo -has- greatly reduced the amount of kernel code that must
> > be inspected to guarantee realtime response, and this is a great
> > accomplishment and a great contribution.  I believe that his approach
> 
> Indeed.
> 
> > is more than "good enough" for a great number of applications that are
> > traditionally thought of as "hard realtime".  But, unless I missed
> > something, it is not mathematically proven, nor is it in any way
> > "certified".
> 
> Yes, it could be that 99% of the app that traditionally claims to need
> hard-RT will be fully covered by preempt-RT (assuming the cpu can keep
> up with the slowdown), but when I talk about hard-RT I only think at
> those apps like the linuxdevices article that beats at 50usec. To me
> hard-RT is all about guaranteeing to provide a deadline, the rest I call
> it lowlatency patches. preempt-RT is obviously a very great improvement
> to the lowlatency soft-RT approach and I agree a lot of those apps
> thought to need hard-RT will be fine with it.
> 
> But one should evaluate if an hard-RT guarantee is truly needed or not
> before picking preempt-RT, the performance should be evaluated as well
> cause irq handling is much faster than a context switch.

Yep, much care is required when selecting a realtime OS approach,
whether on Linux or any other OS.  Seems that care in selecting hardware
is required as well -- the PCI issue with X was a new one for me!

> The slowdown numbers posted so far are quite impressive as well.

Agreed!

> Think that in some recent research I'm doing, I'm looking to make the
> semaphore a spinlock to boost performance, preempt-RT is the opposite ;)

;-)

> > Some of the approaches involving separate RTOSes -can- legitimately claim
> > have mathematically proven worst-case scheduling latencies.  For example,
> > the dual-OS/dual-core approach can do so, particularly in the case where
> > the "RTOS" is a tight loop written on bare metal in hand-coded assembly.
> > One might consider this to be a trivial example, perhaps even a stupid
> > example, but there are a lot of apps out there that use exactly this
> > approach.
> >
> > The migration-between-OSes approach, such as RTAI Fusion, might also be
> > able to mathematically guarantee RTOS scheduling latencies.  And, unlike
> > the dual-OS/dual-core approaches, it does provide a single environment
> > to applications.  However, it still results in two separate OS instances
> > to administer, and, as near as I can tell, the RTOS has to stick its
> > hands so deeply into Linux's internals that it might as well be part
> > of the Linux kernel.  Unless I am missing something, any sort of bug
> > in the Linux kernel has a reasonable chance of messing up the RTOS,
> > since the RTOS must paw through Linux's tasks.
> 
> Yes. I'm a beliver in the dual-OS/dual-core model, as the only one who
> can reasonably simply guarantee a deadline.

If your realtime application needs every scrap of realtime response that
the hardware can deliver, then I agree that the dual-OS/dual-core approach
is very likely the best choice.  But it depends on what OS services the
realtime portion of the application needs.  Beyond a certain point,
it might be better to accept some probability of scheduling failure
from Linux than to accept a perhaps-higher probability of bugs from a
less-throroughly-tested RTOS.  Or maybe not.  To decide, it is necessary
to look carefully at the details of the app, the RTOS, and Linux.

> The single image RTOS has much larger pieces of complex code involved in
> the RT equation (in your list you didn't mention the mutex code itself),
> plus this codes like the scheduler changes all the time (it's not like
> the irq code that pratically never changes in stable cycles), and it
> slowsdown quite a bit as well at emulating infinite cpus and
> overscheduling at every contention.

Excellent point, am adding "mutex code" to the list.  How does the
following look?

e.	Any code that holds a lock, mutex, semaphore, or other resource
	that is needed by the code implementing your new feature, as
	well as the code that actually implements the lock, mutex,
	semaphore, or other resource.

> > Can we say that one of these approaches is definitely "good enough" for
> > all reasonable Linux RT work, and that we should therefore stop working on 
> > the other approach?
> > 
> > I might well be missing something, but I don't believe so, at least
> > not yet.
> 
> I agree with you. Perhaps I didn't make my point of view clear enough on
> the good things of preempt-RT. preempt-RT is a great feature for some
> apps.

Or I could easily have misread or misunderstood your emails.

> I _only_ criticise people who wants to put preempt-RT on par with
> guaranteed hard-RT solutions like RTAI/rtlinux that can easily provide a
> guaranteed deadline, while preempt-RT can only proviede measurements
> backed by statistically signficance, which is quite far away from a
> guarantee. Plus it's much slower as well. But apparently some people
> here, wants you to believe that preempt-RT will be exactly guaranteed
> hard-RT as RTAI/rtlinux, and they can't handle it when I say this is not
> the case. This is the only thing I'm objecting as stated in my first
> relevant email of the thread partly quoted above.
> 
> I'm sure I've ever said that preempt-RT is useless and there's no application
> for it, infact I stated _multiple_ times there are cases like alsa where
> preempt-RT is the very best that one can do.

The big thing that I learned in reviewing the thread and in doing the
writeup is that there are a lot of different things that a realtime app
might want from a OS and from the hardware.  It seems to me that we
probably cannot do a simple linear ranking of the approaches -- some
will be better at one aspect, others will be better at another.

Though I would love to be proven wrong!  It is always nice from a user's
point of view if you can just use the "best" one and spend more time
worrying about getting your app right instead of having to spend a lot
of time worrying about what OS approach to use!  ;-)

> > In the meantime, the more time anyone spends sending flames on LKML, the
> > less time that person is putting into improving his/her favorite approach.
> 
> Well, I'm not in RT development anyway so that's not my case, but of
> course I'm could spend my time much better anyway ;)

If you were -anywhere- at all near the worst example of misdirected
effort, the pace of Linux development would be terrifyingly fast!  ;-)

> Thanks again for your appreciated RT efforts Paul!

Glad you liked it, and, again, please accept my apologies for appearing
to be singling you out.

							Thanx, Paul

Re: Attempted summary of "RT patch acceptance" thread

[Karim Yaghmour]

Paul E. McKenney wrote:
> The big thing that I learned in reviewing the thread and in doing the
> writeup is that there are a lot of different things that a realtime app
> might want from a OS and from the hardware.  It seems to me that we
> probably cannot do a simple linear ranking of the approaches -- some
> will be better at one aspect, others will be better at another.

During the earlier thread, I did suggest a third option which I think
could help integrate both PREEMPT_RT and the interrupt pipeline in
Adeos and RTAI-fusion _without_ impacting on the rest of the kernel
code. However, I haven't received any feedback whatsoever in that
regard. So here it is again. Given how much discussion space is taken
on "concepts", it may be worth entertaining a third option.

> <alternate proposal>
> Much like there is nothing precluding PREEMPT_RT to co-exist with
> the nanokernel approach (on which RTAI is based), it could be suggested
> the adding of a linux/hard-rt directory containing the (re?)implementation
> of services/abstractions required for hard-rt applications. You still
> get a single tree, but there's then a clear separation at many levels,
> including maintainership. As such, much of what RTAI-fusion is currently
> doing could find itself in linux/hard-rt. For example, RTAI-fusion
> transparently provides a hard-rt deterministic nanosleep(). This and
> other such replacements for kernel/*.c would live in hard-rt/ with
> no disturbance to the rest of the tree. In the same way, include/linux
> could be a symbolic link to either include/linux-hrt or include/linux-srt,
> with headers in include/linux-hrt referring back to include/linux-srt
> where appropriate. Again, zero cost for mainstream maintainers. If the
> hard-rt stuff breaks, only the rt folks get the pain. Note that I'm not
> suggesting creating duplicates like this for all directories. In fact,
> most of what's in arch/* and drivers/* would remain unchanged, and
> where appropriate, hard-rt/* and include/linux-hrt should reuse as much
> of what already exists as possible.
> 
> Sure, the hard-rt part wouldn't have all the bells and whistles of the
> mainstream part, but that's what we're going to have anyway if
> PREEMPT_RT is included (as is clearly acknowledged elsewhere in this
> thread by those backing it), unless there's a general consensus amongst
> all subsystem maintainers that Linux should become QNX-like ... which,
> to the best of my reading of this thread, most are not interested in.
> 
> The above suggestion doesn't solve the two-app vs. one-app dilemma, but
> it takes away the "oh, horror, we need to maintain two separate kernel
> trees for our application development" from those against the nanokernel
> approach _without_ imposing additional burden on mainstream maintainers.
> </alternate proposal>

Let me clarify what I say above to make it clear that linux/hard-rt
and include/linux-hrt/ could/would include a merged PREEMPT_RT, adeos, and
rtai-fusion. The combo patch put together by Philippe (which includes both
PREEMPT_RT and adeos) is already a good start in that direction. Like we
said earlier, both methods are not mutually exclusive.

Note that my purpose by posting this proposal is to invite further
discussion as to what is the best way to integrate any real-time approach
to the existing kernel structure. What approach eventually makes it into
this structure, whether it be PREEMPT_RT, Adeos, fusion, a combination of
these, or something entirely different, is not something I attempt to
resolve here. As it currently stands, for PREEMPT_RT at least, the
intrusiveness of the patch onto the mainstream code is something that
seems to make a lot of people uneasy.

Like I said earlier:
> ... so here goes, it's just an idea I'm throwing in the lion pit ...
> it clearly would require much more work and input ... so devour, tear,
> and crush at will ...

Karim
-- 
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 1-866-677-4546

Re: Attempted summary of "RT patch acceptance" thread

[Andrea Arcangeli]

On Sat, Jun 11, 2005 at 02:04:17PM -0700, Paul E. McKenney wrote:
> Well, I probably should have edited the "TO" and "CC" lines to make it
> clear that I was not singling you out.  Please accept my apologies for
> making it appear that I was singling you out.

No need of apologies, your email was very appreciated. I greatly regret
posting here only after reading the weird emails that some people from
some other company posts. Having to pass emails through the
politically-correct filter before sending them isn't very entertaining
work.

> as well.  As to patent claims, don't get me started...

Well, AFIK we're not supposed to be able to read them cause we're not
lawyers, so I don't feel guilty in not understanding them completely, I
try only to understand and remember the _concept_ of the patent, I don't
even try to understand or remember all the legal details of the claims
which seems completely unreadable for me even though I know it's the
most important part of any patent.

> sort of software tool that scanned the required code to produce such
> guarantee, but I have not heard of anyone actually doing this with
> any Linux-based realtime scheme.

With the nanokernel approach it's quite easy to provide a guarantee
because there are no million of possible paths that the software can
take. So demonstrating that with cold cache it takes less than 50usec to
reach the RT userland regardless whatever path is taken is quite easy
and feasible IMHO.

One still has the risk of the system crashing with memory corruption
from a bad device driver, but the hardware can crash too. At some point
one has to relay on stress testing, but starting from a weak design
doesn't sound correct to me, especially if later something goes wrong in
production (which from my point of view would make the designer liable
if the design was flakey and the proper math wasn't done, it's like a
bridge falling because proper math wasn't done, unless one tells the
customer to use the stuff at his own risk, which doesn't sound a great
form of "guarantee" in the first place ;).

With preempt-RT RTOS there are way too many possible paths to even
attempt at providing a guarantee IMHO.

> is required as well -- the PCI issue with X was a new one for me!

I knew pci hardware can stall the bus, but I seriously hoped core things
like memory controller in the northbridge or embedded in the cpu would
not risk to generate any starvation. So far I thought by ruling out
desktop graphics card or other fancy pci stuff that needs some extreme
performance, one would have been fairly safe after some benchmarking. I
hope on some embedded arch like arm people is careful to design
northbridge to be RT compliant.

> If your realtime application needs every scrap of realtime response that
> the hardware can deliver, then I agree that the dual-OS/dual-core approach
> is very likely the best choice.  But it depends on what OS services the
> realtime portion of the application needs.  Beyond a certain point,
> it might be better to accept some probability of scheduling failure
> from Linux than to accept a perhaps-higher probability of bugs from a
> less-throroughly-tested RTOS.  Or maybe not.  To decide, it is necessary
> to look carefully at the details of the app, the RTOS, and Linux.

Agreed.

I think using preempt-RT RTOS in problems where RTAI/rtlinux is equally
simple would be a big waste and an unnecessary lower reliability. RTOS
tries to keep the userland API as close as possible to the non-RT one,
by increasing the kernel complexity with relative slowdown. So the whole
system is certainly more complicated and slower even though the userland
part alone may be a bit simpler.

Things becomes different if the RT app isn't fully in userland (like
multimedia), once the RT app has to call deep into kernel code, then
RTAI quickly becomes useless (ok it can run nanosleep with fusion fine,
but I don't see how RTAI can run getdents or stuff like that which
requires shared non-RT aware locks). Clearly it's possible to design a
non-RT thread doing the non-RT stuff, and passing up the info to the
RTAI userland RT thread, but RTOS is better at that, since the kernel
locks will have RT awareness and because irqs will get out of the way.

But once somebody calls into the kernel for getdents, one can forget the
word hard-RT, the mutex will get out of the way in the userland, but not
in the kernel code, and kmalloc will swap.

Infact from my point of view preempt-RT and RTAI are fully orthogonal.

My view is still preempt-RT as a much much much improved soft-RT, and
RTAI/rtlinux as the only hard-RT with guaranteed deadline.

If it was me having to call stuff like getdents in userland from a RT
context, I could try using preempt-RT with a soft-RT thread passing up
the getdents results to a RTAI hard-RT thread, mixing RTAI and
preempt-RT in the same system.

As usual when I talk about hard-RT I think at the 1st kind of hard-RT
where missing a deadline is unacceptable. As said before, the hard-RT
type 2 is fine with current soft-RT linux too by just calling
gettimeofday after the deadline was supposed to be met (or by an
equivalent hack in the scheduler calling gettimeofday in kernel space
and killing the task or sending a signal, whatever).

> Excellent point, am adding "mutex code" to the list.  How does the
> following look?
> 
> e.	Any code that holds a lock, mutex, semaphore, or other resource
> 	that is needed by the code implementing your new feature, as
> 	well as the code that actually implements the lock, mutex,
> 	semaphore, or other resource.

Looks fine thanks!

> Glad you liked it, and, again, please accept my apologies for appearing
> to be singling you out.

Don't worry, I didn't feel singled out, and infact I greatly appreciated
your contribution, thanks.

Re: Attempted summary of "RT patch acceptance" thread

[Andrea Arcangeli]

On Sat, Jun 11, 2005 at 07:48:39PM -0400, Karim Yaghmour wrote:
> Let me clarify what I say above to make it clear that linux/hard-rt
> and include/linux-hrt/ could/would include a merged PREEMPT_RT, adeos, and
> rtai-fusion. The combo patch put together by Philippe (which includes both
> PREEMPT_RT and adeos) is already a good start in that direction. Like we
> said earlier, both methods are not mutually exclusive.

FWIW I definitely agree that merging PREEMPT_RT design and nanokernel
design in a single RT kernel makes lots of sense, and this confirms my
original belief that preempt-RT and nanokernel are orthogonal and they
don't cover the same domain of RT apps (or they'd overlap completely and
it would be pointless to have them at the same time in the same kernel
to use them at the same time).

Re: Attempted summary of "RT patch acceptance" thread

[Lee Revell]

On Sun, 2005-06-12 at 19:01 +0200, Andrea Arcangeli wrote:
> I seriously hoped core things
> like memory controller in the northbridge or embedded in the cpu would
> not risk to generate any starvation. 

Nforce4 chipsets have some problems that look like DMA starvation too,
the RME people claim to have traced it to the SATA controller.

http://www.rme-audio.de/english/techinfo/nforce4_tests.htm

Basically these all fall under common sense engineering, you don't use
buggy/unproved hardware for an RT or any mission critical system.  

Lee  

Re: Attempted summary of "RT patch acceptance" thread

[Bill Huey]

On Sun, Jun 12, 2005 at 02:43:02PM -0400, Lee Revell wrote:
> Nforce4 chipsets have some problems that look like DMA starvation too,
> the RME people claim to have traced it to the SATA controller.
> 
> http://www.rme-audio.de/english/techinfo/nforce4_tests.htm
> 
> Basically these all fall under common sense engineering, you don't use
> buggy/unproved hardware for an RT or any mission critical system.  

How did folks work around that ? by using another SATA PCI-e card ?

bill

Re: Attempted summary of "RT patch acceptance" thread

[Paul E. McKenney]

On Sat, Jun 11, 2005 at 07:48:39PM -0400, Karim Yaghmour wrote:
> 
> Paul E. McKenney wrote:
> > The big thing that I learned in reviewing the thread and in doing the
> > writeup is that there are a lot of different things that a realtime app
> > might want from a OS and from the hardware.  It seems to me that we
> > probably cannot do a simple linear ranking of the approaches -- some
> > will be better at one aspect, others will be better at another.
> 
> During the earlier thread, I did suggest a third option which I think
> could help integrate both PREEMPT_RT and the interrupt pipeline in
> Adeos and RTAI-fusion _without_ impacting on the rest of the kernel
> code. However, I haven't received any feedback whatsoever in that
> regard. So here it is again. Given how much discussion space is taken
> on "concepts", it may be worth entertaining a third option.

Good point -- I did "fly over" this one on earlier reading, partly because
I was somehow thinking that it was a dual-system-call-table approach so
that realtime processes would see deterministic implementations of system
calls (or get migrated/errored/whatever if there was no deterministic
implementation for that particular system call), while non-realtime
processes would see the standard stuff.

> > <alternate proposal>
> > Much like there is nothing precluding PREEMPT_RT to co-exist with
> > the nanokernel approach (on which RTAI is based), it could be suggested
> > the adding of a linux/hard-rt directory containing the (re?)implementation
> > of services/abstractions required for hard-rt applications. You still
> > get a single tree, but there's then a clear separation at many levels,
> > including maintainership. As such, much of what RTAI-fusion is currently
> > doing could find itself in linux/hard-rt. For example, RTAI-fusion
> > transparently provides a hard-rt deterministic nanosleep(). This and
> > other such replacements for kernel/*.c would live in hard-rt/ with
> > no disturbance to the rest of the tree. In the same way, include/linux
> > could be a symbolic link to either include/linux-hrt or include/linux-srt,
> > with headers in include/linux-hrt referring back to include/linux-srt
> > where appropriate. Again, zero cost for mainstream maintainers. If the
> > hard-rt stuff breaks, only the rt folks get the pain. Note that I'm not
> > suggesting creating duplicates like this for all directories. In fact,
> > most of what's in arch/* and drivers/* would remain unchanged, and
> > where appropriate, hard-rt/* and include/linux-hrt should reuse as much
> > of what already exists as possible.
> > 
> > Sure, the hard-rt part wouldn't have all the bells and whistles of the
> > mainstream part, but that's what we're going to have anyway if
> > PREEMPT_RT is included (as is clearly acknowledged elsewhere in this
> > thread by those backing it), unless there's a general consensus amongst
> > all subsystem maintainers that Linux should become QNX-like ... which,
> > to the best of my reading of this thread, most are not interested in.
> > 
> > The above suggestion doesn't solve the two-app vs. one-app dilemma, but
> > it takes away the "oh, horror, we need to maintain two separate kernel
> > trees for our application development" from those against the nanokernel
> > approach _without_ imposing additional burden on mainstream maintainers.
> > </alternate proposal>

This could potentially address the need for version-synchronization
between RTAI-Fusion and the Linux kernel.  Would you really want two
separate builds, or is there some reasonable way of producing a single
kernel binary that has both?  And if there is some reasonable way of
doing this, is it the right thing to do?

The single-binary approach could potentially reduce the
dual-OS-administration load associated with RTAI-Fusion.  However,
handling all the interactions between the deterministic and
non-deterministic system calls could get hairy.  No big deal for
scheduling primitives, but things could get interesting for I/O and
networking protocols.

> Let me clarify what I say above to make it clear that linux/hard-rt
> and include/linux-hrt/ could/would include a merged PREEMPT_RT, adeos, and
> rtai-fusion. The combo patch put together by Philippe (which includes both
> PREEMPT_RT and adeos) is already a good start in that direction. Like we
> said earlier, both methods are not mutually exclusive.

Agreed, most of the realtime approaches can be used together or combined.

> Note that my purpose by posting this proposal is to invite further
> discussion as to what is the best way to integrate any real-time approach
> to the existing kernel structure. What approach eventually makes it into
> this structure, whether it be PREEMPT_RT, Adeos, fusion, a combination of
> these, or something entirely different, is not something I attempt to
> resolve here. As it currently stands, for PREEMPT_RT at least, the
> intrusiveness of the patch onto the mainstream code is something that
> seems to make a lot of people uneasy.

So, one can use the following types of combination:

o	single source tree, multiple kernels (which is what I now
	think that you are getting at above).

o	straight merge, as between PREEMPT and PREEMPT_RT.

o	single kernel, multiple syscall implementations for
	some syscalls (deterministic vs. non-deterministic).

o	side-by-side combination, as with dual-OS/dual-core and
	pretty much any other approach.

Other combination approaches?

I have a paragraph in the updated version mentioning the possibility
of combinations, and if some of these get "cooked" enough, will add
more verbiage.

> Like I said earlier:
> > ... so here goes, it's just an idea I'm throwing in the lion pit ...
> > it clearly would require much more work and input ... so devour, tear,
> > and crush at will ...

						Thanx, Paul

Re: Attempted summary of "RT patch acceptance" thread

[Karim Yaghmour]

Paul E. McKenney wrote:
> This could potentially address the need for version-synchronization
> between RTAI-Fusion and the Linux kernel.  Would you really want two
> separate builds, or is there some reasonable way of producing a single
> kernel binary that has both?  And if there is some reasonable way of
> doing this, is it the right thing to do?

No, single build is what I'm looking for. Nothing precludes the
fusion parts from being built during the same kernel build ...
as modules. If you don't load 'em, you don't need to worry about
'em.

> The single-binary approach could potentially reduce the
> dual-OS-administration load associated with RTAI-Fusion.  However,
> handling all the interactions between the deterministic and
> non-deterministic system calls could get hairy.  No big deal for
> scheduling primitives, but things could get interesting for I/O and
> networking protocols.

Again, if you don't load 'em, you don't get 'em. If you use it
and it's broken, then you're doing rt and you need to sync up
with the maintainer. Nothing different here from the standard
run of the mill "I'm using subsystem X and it doesn't work"
posting to LKML.

> So, one can use the following types of combination:
> 
> o	single source tree, multiple kernels (which is what I now
> 	think that you are getting at above).
> 
> o	straight merge, as between PREEMPT and PREEMPT_RT.
> 
> o	single kernel, multiple syscall implementations for
> 	some syscalls (deterministic vs. non-deterministic).
> 
> o	side-by-side combination, as with dual-OS/dual-core and
> 	pretty much any other approach.

I'm not sure how you'd fit what I'm trying to suggest above, but
let me rephrase it with the above in mind:

What I'm suggesting is that all rt components be included, but
in separate directories within mainline. That may or may not
mean additional schedulers/services. In the case where the
new layout would include both PREEMPT_RT and fusion, what
we'd get is that the user would have access to these configs:
- Plain Linux, no PREEMPT_RT, no ipipe, no fusion.
- Linux with PREEMPT_RT, no fusion: ints are threaded and locks
  are mutexes like now (however without the code intrusiveness
  given the use of separate directories.) May or may not include
  ipipe.
- Linux with fusion, no PREEMPT_RT: the fusion modules are built
  and installed with the rest of the modules. ipipe must be
  enabled.
- Linux with fusion and PREEMPT_RT: combination of the previous
  two.
- Linux with ipipe, no fusion or PREEMPT_RT: the soft-cli stuff
  is built into the kernel and loaded drivers can get
  deterministic response times, but there are no fancy rt
  services offered to anyone.

Practically, linux/hard-rt/ would contain both the code for
PREEMPT_RT and the code for fusion. The actual layout in that
directory would still need to be detailed, but the desired
effect is that both PREEMPT_RT and fusion share as much code
as possible.

Hope this clarifies what I'm suggesting a little bit more. Of
course, all this would need to be rehashed a number of times,
and most importantly, the PREEMPT_RT folks and the fusion
effort would need to agree to join forces. From the fusion
POV, it's clear that the door is open for collaboration. As
proof, Philippe has been publishing combo patches with Adeos
and PREEMPT_RT for some time. I can't speak for the PREEMPT_RT
POV, though. I might be mistaken, but it seems that the feedback
I've seen from some PREEMPT_RT backers does seem to indicate
some openess. We'll see how things go.

Karim
-- 
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 1-866-677-4546

Re: Attempted summary of "RT patch acceptance" thread

[Paul E. McKenney]

On Sun, Jun 12, 2005 at 09:35:15PM -0400, Karim Yaghmour wrote:
> 
> Paul E. McKenney wrote:
> > This could potentially address the need for version-synchronization
> > between RTAI-Fusion and the Linux kernel.  Would you really want two
> > separate builds, or is there some reasonable way of producing a single
> > kernel binary that has both?  And if there is some reasonable way of
> > doing this, is it the right thing to do?
> 
> No, single build is what I'm looking for. Nothing precludes the
> fusion parts from being built during the same kernel build ...
> as modules. If you don't load 'em, you don't need to worry about
> 'em.

OK...  Then the idea is to dynamically redirect the symbolic link
to include/linux-srt or include/linux-srt that you mentioned in your
previous email, or is the symlink serving some other purpose?

> > The single-binary approach could potentially reduce the
> > dual-OS-administration load associated with RTAI-Fusion.  However,
> > handling all the interactions between the deterministic and
> > non-deterministic system calls could get hairy.  No big deal for
> > scheduling primitives, but things could get interesting for I/O and
> > networking protocols.
> 
> Again, if you don't load 'em, you don't get 'em. If you use it
> and it's broken, then you're doing rt and you need to sync up
> with the maintainer. Nothing different here from the standard
> run of the mill "I'm using subsystem X and it doesn't work"
> posting to LKML.

So your focus is on system calls that can have totally separate
realtime and non-realtime implementations?  Or am I missing some
trick here?

> > So, one can use the following types of combination:
> > 
> > o	single source tree, multiple kernels (which is what I now
> > 	think that you are getting at above).
> > 
> > o	straight merge, as between PREEMPT and PREEMPT_RT.
> > 
> > o	single kernel, multiple syscall implementations for
> > 	some syscalls (deterministic vs. non-deterministic).
> > 
> > o	side-by-side combination, as with dual-OS/dual-core and
> > 	pretty much any other approach.
> 
> I'm not sure how you'd fit what I'm trying to suggest above, but
> let me rephrase it with the above in mind:
> 
> What I'm suggesting is that all rt components be included, but
> in separate directories within mainline. That may or may not
> mean additional schedulers/services. In the case where the
> new layout would include both PREEMPT_RT and fusion, what
> we'd get is that the user would have access to these configs:
> - Plain Linux, no PREEMPT_RT, no ipipe, no fusion.
> - Linux with PREEMPT_RT, no fusion: ints are threaded and locks
>   are mutexes like now (however without the code intrusiveness
>   given the use of separate directories.) May or may not include
>   ipipe.
> - Linux with fusion, no PREEMPT_RT: the fusion modules are built
>   and installed with the rest of the modules. ipipe must be
>   enabled.
> - Linux with fusion and PREEMPT_RT: combination of the previous
>   two.
> - Linux with ipipe, no fusion or PREEMPT_RT: the soft-cli stuff
>   is built into the kernel and loaded drivers can get
>   deterministic response times, but there are no fancy rt
>   services offered to anyone.

Single kernel, multiple implementations for some syscalls, more
or less, anyway.

> Practically, linux/hard-rt/ would contain both the code for
> PREEMPT_RT and the code for fusion. The actual layout in that
> directory would still need to be detailed, but the desired
> effect is that both PREEMPT_RT and fusion share as much code
> as possible.
> 
> Hope this clarifies what I'm suggesting a little bit more. Of
> course, all this would need to be rehashed a number of times,
> and most importantly, the PREEMPT_RT folks and the fusion
> effort would need to agree to join forces. From the fusion
> POV, it's clear that the door is open for collaboration. As
> proof, Philippe has been publishing combo patches with Adeos
> and PREEMPT_RT for some time. I can't speak for the PREEMPT_RT
> POV, though. I might be mistaken, but it seems that the feedback
> I've seen from some PREEMPT_RT backers does seem to indicate
> some openess. We'll see how things go.

My guess is that there are enough people in the PREEMPT_RT camp that
it might not make sense to ascribe a single point of view to them  ;-)

How are you and Kristian looking to benchmark/compare the various
combinations you call out above?  Seems like one would have to look
at more than straight scheduling/interrupt latency to make a reasonable
comparison.

						Thanx, Paul

Re: Attempted summary of "RT patch acceptance" thread

[Karim Yaghmour]

Paul E. McKenney wrote:
> OK...  Then the idea is to dynamically redirect the symbolic link
> to include/linux-srt or include/linux-srt that you mentioned in your
> previous email, or is the symlink serving some other purpose?

What I'm suggesting is that rt patches shouldn't touch the existing
codebase. Instead, functionality having to do with rt should be
integrated in separate directories, and depending the way you
configure the kernel, include/linux would point to either
include/linux-srt or include/linux-hrt, much like include/asm
points to one of inclux/asm-*.

> So your focus is on system calls that can have totally separate
> realtime and non-realtime implementations?  Or am I missing some
> trick here?

There's no trick. It's just a layout thing. Hope the above
explains what I'm trying to say.

> How are you and Kristian looking to benchmark/compare the various
> combinations you call out above?  Seems like one would have to look
> at more than straight scheduling/interrupt latency to make a reasonable
> comparison.

I'm not sure that benchmarking would be relevant. This is just a
integration/layout/configuration/build suggestion. I don't think
that this organization will change anything to the benchmark
results.

Karim
-- 
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 1-866-677-4546

Re: Attempted summary of "RT patch acceptance" thread

[Daniel Walker]

On Mon, 2005-06-13 at 15:49 -0400, Karim Yaghmour wrote:
> Paul E. McKenney wrote:
> > OK...  Then the idea is to dynamically redirect the symbolic link
> > to include/linux-srt or include/linux-srt that you mentioned in your
> > previous email, or is the symlink serving some other purpose?
> 
> What I'm suggesting is that rt patches shouldn't touch the existing
> codebase. Instead, functionality having to do with rt should be
> integrated in separate directories, and depending the way you
> configure the kernel, include/linux would point to either
> include/linux-srt or include/linux-hrt, much like include/asm
> points to one of inclux/asm-*.


I think this is mistake. Projects that create separation like this are
begging for the community to reject them. I see this as a design for
one, instead of design for many mistake. From what I've seen, a project
would want to do as much clean integration as possible.

Daniel

Re: Attempted summary of "RT patch acceptance" thread

[Paul E. McKenney]

On Mon, Jun 13, 2005 at 03:49:08PM -0400, Karim Yaghmour wrote:
> 
> Paul E. McKenney wrote:
> > OK...  Then the idea is to dynamically redirect the symbolic link
> > to include/linux-srt or include/linux-srt that you mentioned in your
> > previous email, or is the symlink serving some other purpose?
> 
> What I'm suggesting is that rt patches shouldn't touch the existing
> codebase. Instead, functionality having to do with rt should be
> integrated in separate directories, and depending the way you
> configure the kernel, include/linux would point to either
> include/linux-srt or include/linux-hrt, much like include/asm
> points to one of inclux/asm-*.

I would guess that the end result would be a mixed strategy, where some
things (e.g., the existing CONFIG_PREEMPT) are intermingled with the
rest of the Linux code based, while other things (e.g., nanokernel
implementations) are in separate directories.  But this is quickly
getting -way- outside of my area.

So I must leave this aspect of the discussion to others.

> > So your focus is on system calls that can have totally separate
> > realtime and non-realtime implementations?  Or am I missing some
> > trick here?
> 
> There's no trick. It's just a layout thing. Hope the above
> explains what I'm trying to say.

OK.  However, should the discussion get to the point where something
like RTAI-Fusion has realtime versions of system calls that have
globally-visible side-effects (such as I/O, networking, IPC, ...),
then the issue of how to get the non-realtime and the realtime variants
to play nicely with each other will arise.

It may well be that system calls containing such side-effects need to be
Linux-only, or maybe someone will come up with the necessary tricks to
make it all work nicely.  Not particularly worried about it myself --
yet, anyway.  There are plenty of things to worry about before we get
to that point!

> > How are you and Kristian looking to benchmark/compare the various
> > combinations you call out above?  Seems like one would have to look
> > at more than straight scheduling/interrupt latency to make a reasonable
> > comparison.
> 
> I'm not sure that benchmarking would be relevant. This is just a
> integration/layout/configuration/build suggestion. I don't think
> that this organization will change anything to the benchmark
> results.

Sorry, side issue.

I was responding to your list of combinations of CONFIG_PREEMPT_RT, Adeos,
and Fusion, assuming (probably incorrectly) that you and Kristian were
looking to compare all the possible combinations.  If my assumption is
incorrect, then my question was irrelevant, and I apologize for the noise.

						Thanx, Paul

Re: Attempted summary of "RT patch acceptance" thread

[Paul E. McKenney]

On Mon, Jun 13, 2005 at 01:03:53PM -0700, Daniel Walker wrote:
> On Mon, 2005-06-13 at 15:49 -0400, Karim Yaghmour wrote:
> > Paul E. McKenney wrote:
> > > OK...  Then the idea is to dynamically redirect the symbolic link
> > > to include/linux-srt or include/linux-srt that you mentioned in your
> > > previous email, or is the symlink serving some other purpose?
> > 
> > What I'm suggesting is that rt patches shouldn't touch the existing
> > codebase. Instead, functionality having to do with rt should be
> > integrated in separate directories, and depending the way you
> > configure the kernel, include/linux would point to either
> > include/linux-srt or include/linux-hrt, much like include/asm
> > points to one of inclux/asm-*.
> 
> I think this is mistake. Projects that create separation like this are
> begging for the community to reject them. I see this as a design for
> one, instead of design for many mistake. From what I've seen, a project
> would want to do as much clean integration as possible.

It depends on the details of the situation, right?  For example, it
would not have made sense to integrate RCU into spinlock.h, where most
of the other Linux synchronization primitives live.  On the other hand,
spinlock variants, such as spin_lock_bh(), are integrated into the same
spinlock.h file as the the base spin_lock() functions.

For example, CONFIG_PREEMPT_RT separates out some of its realtime function
into separate files (e.g., kernel/rt.c and include/linux/rt_lock.h),
while other parts of its realtime function are integrated into the rest
of the kernel.

						Thanx, Paul

Re: Attempted summary of "RT patch acceptance" thread

[Lee Revell]

On Mon, 2005-06-13 at 16:26 -0400, Karim Yaghmour wrote:
> Daniel Walker wrote:
> > I think this is mistake. Projects that create separation like this are
> > begging for the community to reject them. I see this as a design for
> > one, instead of design for many mistake. From what I've seen, a project
> > would want to do as much clean integration as possible.
> 
> I understand what you're saying, but based on the feedback
> PREEMPT_RT has gotten up until now, and now outright suggestions
> that the debate is not even relevant to the LKML, I think that
> some people are trying to give those interested a hint: integration
> with mainline code is NOT on the agenda.

I think you're referring to one known flamer and hothead who most people
on LKML don't seem to take seriously anymore.  I suspect the silent
majority does not mind as long as it's not intrusive on the current code
base.

Lee

Re: Attempted summary of "RT patch acceptance" thread

[Karim Yaghmour]

Daniel Walker wrote:
> I think this is mistake. Projects that create separation like this are
> begging for the community to reject them. I see this as a design for
> one, instead of design for many mistake. From what I've seen, a project
> would want to do as much clean integration as possible.

I understand what you're saying, but based on the feedback
PREEMPT_RT has gotten up until now, and now outright suggestions
that the debate is not even relevant to the LKML, I think that
some people are trying to give those interested a hint: integration
with mainline code is NOT on the agenda.

Some may want to continue trying to force-feed mainstream
maintainers. I can't stop anyone from trying, that's for sure.
However, I think what I'm suggesting is a reasonable compromise:
mainstream maintainers don't need to care about RT on a day-to-
day basis and the RT folks get to be part of mainline.

That said, please make sure you've carefully read through what
I suggest. I'm not saying that everything inside the kernel needs
to be duplicated. I'm saying that what add-ons are should be
separate. For example, like I was first saying, headers in
include/linux-hrt would point back to include/linux-srt where
appropriate.

And like I said earlier, this suggestion would need to be refined.

Karim
-- 
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 1-866-677-4546

Re: Attempted summary of "RT patch acceptance" thread

[Daniel Walker]

On Mon, 2005-06-13 at 16:26 -0400, Karim Yaghmour wrote:
> Daniel Walker wrote:
> > I think this is mistake. Projects that create separation like this are
> > begging for the community to reject them. I see this as a design for
> > one, instead of design for many mistake. From what I've seen, a project
> > would want to do as much clean integration as possible.
> 
> I understand what you're saying, but based on the feedback
> PREEMPT_RT has gotten up until now, and now outright suggestions
> that the debate is not even relevant to the LKML, I think that
> some people are trying to give those interested a hint: integration
> with mainline code is NOT on the agenda.

I wouldn't work on RT if mainline integration wasn't on the agenda. 

> Some may want to continue trying to force-feed mainstream
> maintainers. I can't stop anyone from trying, that's for sure.
> However, I think what I'm suggesting is a reasonable compromise:
> mainstream maintainers don't need to care about RT on a day-to-
> day basis and the RT folks get to be part of mainline.

There is going to be positive , and negative discussion on this. I think
in the end the maintainers (Linus, and Andrew) don't want "people" to
get a patch or modification from the outside. It's best if the community
is not separated .. If you make a clean integration , and people want
what you are doing, there is no reason for it to be rejected.

Daniel

Re: Attempted summary of "RT patch acceptance" thread

[Bill Huey]

On Mon, Jun 13, 2005 at 01:10:46PM -0700, Paul E. McKenney wrote:
...
> It may well be that system calls containing such side-effects need to be
> Linux-only, or maybe someone will come up with the necessary tricks to
> make it all work nicely.  Not particularly worried about it myself --
> yet, anyway.  There are plenty of things to worry about before we get
> to that point!
...

I suggest not speculating about the needs of various RT apps with
respect to the kernel. (your speculation here will add more confusion
to parts of an already confused kernel community) Vendors and other
groups will layer them on as needed depending on specific usage.
The futex/fusyn work is an exception with regard to syscalls, but
don't speculate about projects like that if you haven't been carefully
tracking their progress and their end goals.

That's my suggestion.

bill

Re: Attempted summary of "RT patch acceptance" thread

[Karim Yaghmour]

Paul E. McKenney wrote:
> OK.  However, should the discussion get to the point where something
> like RTAI-Fusion has realtime versions of system calls that have
> globally-visible side-effects (such as I/O, networking, IPC, ...),
> then the issue of how to get the non-realtime and the realtime variants
> to play nicely with each other will arise.

Maybe so, but this will be a problem for the RT folks, not the
mainstream folks, and that's why I believe this strategy is
likely to be more feasible.

> I was responding to your list of combinations of CONFIG_PREEMPT_RT, Adeos,
> and Fusion, assuming (probably incorrectly) that you and Kristian were
> looking to compare all the possible combinations.  If my assumption is
> incorrect, then my question was irrelevant, and I apologize for the noise.

Sorry, there's only so much we can do. Currently, we are redoing
our earlier tests with what Ingo gave us.

Karim
-- 
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 1-866-677-4546

Re: Attempted summary of "RT patch acceptance" thread

[Bill Huey]

On Mon, Jun 13, 2005 at 01:03:53PM -0700, Daniel Walker wrote:
> On Mon, 2005-06-13 at 15:49 -0400, Karim Yaghmour wrote:
> >...
> > What I'm suggesting is that rt patches shouldn't touch the existing
> > codebase. Instead, functionality having to do with rt should be
> > integrated in separate directories, and depending the way you
> > configure the kernel, include/linux would point to either
> > include/linux-srt or include/linux-hrt, much like include/asm
> > points to one of inclux/asm-*.
> 
> I think this is mistake. Projects that create separation like this are
> begging for the community to reject them. I see this as a design for
> one, instead of design for many mistake. From what I've seen, a project
> would want to do as much clean integration as possible.

The problem with the patch at this time is that many of the headers,
including the one you mention, need clean up as well as some
reorganization. The problem with general sloppiness of the patch from
rapid patch inclusion is different than the big event of integrating
the patch into the mainline. Also, folks like me and you in our
respective companies are much more aggressive about doing things with
the patch which is the exception of Linux kernel users and not the
norm. A little more testing wouldn't hurt as well, but I think the
patch is certainly worth getting into an semi-experimental kernel.

IMO, the patch should go eventually, but after some significant clean
up. There will be other areas in the kernel that will force some
more proper abstraction and compartmentalization.

That's my view on the matter.

bill

Re: Attempted summary of "RT patch acceptance" thread

[Paul E. McKenney]

On Mon, Jun 13, 2005 at 01:31:00PM -0700, Bill Huey wrote:
> On Mon, Jun 13, 2005 at 01:10:46PM -0700, Paul E. McKenney wrote:
> ...
> > It may well be that system calls containing such side-effects need to be
> > Linux-only, or maybe someone will come up with the necessary tricks to
> > make it all work nicely.  Not particularly worried about it myself --
> > yet, anyway.  There are plenty of things to worry about before we get
> > to that point!
> ...
> 
> I suggest not speculating about the needs of various RT apps with
> respect to the kernel. (your speculation here will add more confusion
> to parts of an already confused kernel community) Vendors and other
> groups will layer them on as needed depending on specific usage.

Certainly good advice, but...

In this instance, I was speculating not about what various RT apps might
need, but instead about what Karim was getting at.  And I have seen cases
where people needed realtime (as in deterministic) I/O, networking stacks,
and the like, so that part is not speculation.

Whether or not deterministic implementations of I/O, networking stacks,
and the like should ever be included in Linux certainly -is- speculation
at this point, but, as I said in my earlier email, there are plenty of
things to worry about before we get to that point.

> The futex/fusyn work is an exception with regard to syscalls, but
> don't speculate about projects like that if you haven't been carefully
> tracking their progress and their end goals.

The futex/fusyn work is on my list, not for speculation, but rather
for looking at what they have, what it does, how it fits in, and how it
appears to be doing.  My reason for looking into it is that I have
also seen people saying that they need priority inheritance in user-level
mutexes.

I would of course be quite interested in your thoughts on futex/fusyn!

						Thanx, Paul

> That's my suggestion.
> 
> bill
> 
> 

Re: Attempted summary of "RT patch acceptance" thread

[Paul E. McKenney]

On Mon, Jun 13, 2005 at 04:34:15PM -0400, Karim Yaghmour wrote:
> 
> Paul E. McKenney wrote:
> > OK.  However, should the discussion get to the point where something
> > like RTAI-Fusion has realtime versions of system calls that have
> > globally-visible side-effects (such as I/O, networking, IPC, ...),
> > then the issue of how to get the non-realtime and the realtime variants
> > to play nicely with each other will arise.
> 
> Maybe so, but this will be a problem for the RT folks, not the
> mainstream folks, and that's why I believe this strategy is
> likely to be more feasible.

Unless correctly handling the side effects requires changes to the
mainstream as well as to the RT implementations, as it might well for
I/O, networking, and IPC.  Again, I believe that it might be some time
before we get to this point, so am not all that worried about it.

> > I was responding to your list of combinations of CONFIG_PREEMPT_RT, Adeos,
> > and Fusion, assuming (probably incorrectly) that you and Kristian were
> > looking to compare all the possible combinations.  If my assumption is
> > incorrect, then my question was irrelevant, and I apologize for the noise.
> 
> Sorry, there's only so much we can do. Currently, we are redoing
> our earlier tests with what Ingo gave us.

Yours and Kristian's benchmarking contribution is quite timely and
substantial, and I know that I am not the only one who very much
appreciates it!

							Thanx, Paul

Re: Attempted summary of "RT patch acceptance" thread

[Karim Yaghmour]

Daniel Walker wrote:
> I wouldn't work on RT if mainline integration wasn't on the agenda. 

Mainline integration IS what I'm talking about. It's just not done
the same way.

> There is going to be positive , and negative discussion on this. I think
> in the end the maintainers (Linus, and Andrew) don't want "people" to
> get a patch or modification from the outside. It's best if the community
> is not separated .. If you make a clean integration , and people want
> what you are doing, there is no reason for it to be rejected.

I'm not suggesting the separation of the community, I'm suggesting
a strategy of integration based on the fact that a large portion of
kernel contributors don't necessarily care about RT, and most don't
want to care about it in their day-to-day work (though I think most
would care that Linux could have an additional spade down its
sleeve, and would certainly try to help in as much they can from
time to time.)

I'm not suggesting asking "people" to get patches from the outside.
What I'm saying is that those developing mainstream code shouldn't
need to worry about anything real-time, including modifications to
locking primitives in headers (be they defined out or in).

In essence, what you ask can only hold if all kernel developers
intend for Linux to become QNX. Clearly this isn't going to happen.
Whatever changes are made to such core functionality as locking
primitives and interrupt handling can hardly be "transparent"
simply by wrapping #ifdef CONFIG_X around it in mainstream headers.

>From my point of view, determinism and best overall performance are
conflicting goals. Having separate derectories for something as
fundamentally different from best overall performance as determinism
is not too much to ask.

Karim
-- 
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 1-866-677-4546

Re: Attempted summary of "RT patch acceptance" thread

[Karim Yaghmour]

Karim Yaghmour wrote:
> I'm not suggesting the separation of the community, I'm suggesting

BTW, though I repeat that this wasn't what I was suggesting, it can
be easily argued that community separation already exists. The fact
of the matter is that few mainstream developers have been "asking"
for RT. And at this point, some seem genuinely worried about its
possible integration.

So one can keep ignoring the reality of the existing clash here,
but it may just be that backing off a little and thinking about it
in less absolute integration terms would help.

Karim
-- 
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 1-866-677-4546

Re: Attempted summary of "RT patch acceptance" thread

[Bill Huey]

On Mon, Jun 13, 2005 at 06:11:58PM -0400, Karim Yaghmour wrote:
> BTW, though I repeat that this wasn't what I was suggesting, it can
> be easily argued that community separation already exists. The fact
> of the matter is that few mainstream developers have been "asking"
> for RT. And at this point, some seem genuinely worried about its
> possible integration.

Ignore it. They are misinformed and ignorant about these issues.
 
> So one can keep ignoring the reality of the existing clash here,
> but it may just be that backing off a little and thinking about it
> in less absolute integration terms would help.

Be a leader, not a follower. :)

You're never going to get anything cool done if you are "beat down"
by easily offended folks that don't understand what you're doing.

bill

RE: Attempted summary of "RT patch acceptance" thread

[Saksena, Manas]

Karim Yaghmour wrote:
> In essence, what you ask can only hold if all kernel developers
> intend for Linux to become QNX. Clearly this isn't going to happen.

The needs that Linux and QNX (or, whatever your favorite RTOS is)
fulfill are not that separate. 

Keep in mind that Linux has been making inroads into traditional
RTOS markets for 4+ years. RTOSes have been used in many devices
and systems -- many of which do not need the "ruby/diamond" hard
variety of real-time -- preempt-rt would be hard-enough for a 
very large number of devices/systems that currently use an RTOS
(or non mainline Linux kernel). 

> From my point of view, determinism and best overall performance are
> conflicting goals. 

And, likewise SMP and large system scalability will often conflict
with desktop performance. Or, interactive performance goals conflict
with server throughput goals, and so on.... 

> Having separate derectories for something as
> fundamentally different from best overall performance as determinism
> is not too much to ask.   

It probably is too much to ask. In the end, it may very well be the 
case that some capabilities will naturally be partitioned into separate 
directories, but I don't think we are there yet. 

Manas

Re: Attempted summary of "RT patch acceptance" thread

[Karim Yaghmour]

Bill Huey (hui) wrote:
> Ignore it. They are misinformed and ignorant about these issues.

Actually, what often happens on the LKML is that there a handfull
who are ready to get flamed, and there's a pack in the back that
would rather get some work done and have others take the heat for
ideas they themselves share. So some may discount Andrea and
Christoph, but I have a hunch there are others out there who
think the same but just don't want to waste time feeding flames.
... after all, the patch isn't up for inclusion just yet ... :|

> Be a leader, not a follower. :)

Oh believe me, leading is not a problem here.

> You're never going to get anything cool done if you are "beat down"
> by easily offended folks that don't understand what you're doing.

Been there done that. This is not about leadership, it's about
equilibrium.

Karim
-- 
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 1-866-677-4546

Re: Attempted summary of "RT patch acceptance" thread

[Bill Huey]

On Mon, Jun 13, 2005 at 06:28:05PM -0400, Karim Yaghmour wrote:
> Been there done that. This is not about leadership, it's about
> equilibrium.

I'm tell'n you that there is no way that this patch is going to
die. The commercial vendors and the general community is going
use this once they figure out what it can do and there will be
no stopping it. You can bet on some body extremity that this
will happen. It's inevitable and ultimately the future of Linux
whether they like it or not.

bill

Re: Attempted summary of "RT patch acceptance" thread

[Karim Yaghmour]

Saksena, Manas wrote:
> The needs that Linux and QNX (or, whatever your favorite RTOS is)
> fulfill are not that separate. 

That point of view may not be shared by others.

> Keep in mind that Linux has been making inroads into traditional
> RTOS markets for 4+ years. RTOSes have been used in many devices
> and systems -- many of which do not need the "ruby/diamond" hard
> variety of real-time -- preempt-rt would be hard-enough for a 
> very large number of devices/systems that currently use an RTOS
> (or non mainline Linux kernel). 

Please, Manas, go teach someone else about how Linux has been doing
in the real-time world for the 10 years.

> And, likewise SMP and large system scalability will often conflict
> with desktop performance. Or, interactive performance goals conflict
> with server throughput goals, and so on.... 

Not at this scale.

Karim
-- 
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 1-866-677-4546

RE: Attempted summary of "RT patch acceptance" thread

[Saksena, Manas]

Bill Huey (hui) wrote:
> On Mon, Jun 13, 2005 at 06:20:10PM -0400, Saksena, Manas wrote:
>> Keep in mind that Linux has been making inroads into traditional RTOS
>> markets for 4+ years. RTOSes have been used in many devices and
>> systems -- many of which do not need the "ruby/diamond" hard variety
>> of real-time -- preempt-rt would be hard-enough for a very large
>> number of devices/systems that currently use an RTOS (or non mainline
>> Linux kernel).
> 
> It's better to use different terminology. The notion of real time is
> *not* a single dimensional vector that is either "more" or "less"
> than of any particular thing. It's much more complicated than that.  

I agree. But, I thought that it is better than soft/hard dichotomy
of real-time, which makes even less sense in reality. Or, worse the
dichotomy of non real-time and real-time -- which is the point I was
trying to make. And, even though, in practice we talk about real-time
operating systems vs non real-time operating systems -- the difference
is not that fundamental as some would like to believe. 

Manas

Re: Attempted summary of "RT patch acceptance" thread

[Bill Huey]

On Mon, Jun 13, 2005 at 06:20:10PM -0400, Saksena, Manas wrote:
> Keep in mind that Linux has been making inroads into traditional
> RTOS markets for 4+ years. RTOSes have been used in many devices
> and systems -- many of which do not need the "ruby/diamond" hard
> variety of real-time -- preempt-rt would be hard-enough for a 
> very large number of devices/systems that currently use an RTOS
> (or non mainline Linux kernel). 

It's better to use different terminology. The notion of real time
is *not* a single dimensional vector that is either "more" or "less"
than of any particular thing. It's much more complicated than that.

There's at least more than one definition of hard real time floating
around with various degrees of determinism. Deterministic is
deterministic and many RTOS are capable of that. Theoretically provable
anything is pushing it and is not RTOSes are typically constructed.

I recommend dropping that ridiculous terminology because how complex
this problem/domain is as well as how misleading it can be.

bill

Re: Attempted summary of "RT patch acceptance" thread

[Karim Yaghmour]

typo.

Karim Yaghmour wrote:
> Please, Manas, go teach someone else about how Linux has been doing
> in the real-time world for the 10 years.

Should read:
"for the past 10 years."

Karim
-- 
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 1-866-677-4546

Re: Attempted summary of "RT patch acceptance" thread

[Karim Yaghmour]

Bill Huey (hui) wrote:
> I'm tell'n you that there is no way that this patch is going to
> die. The commercial vendors and the general community is going
> use this once they figure out what it can do and there will be
> no stopping it. You can bet on some body extremity that this
> will happen. It's inevitable and ultimately the future of Linux
> whether they like it or not.

Please cool down. This is a rather extreme position.

The fact of the matter is that it is the community that is driving
Linux, not the vendors. If the community decides something isn't
going in, there isn't much any company can do about it, except
maintain it outside of Linux for years on end, which economically
speaking is just not feasible. That's why inclusion is so important.

Bare in mind that I wasn't even talking about whether or not
PREEMPT_RT was making it in. I was only talking about the logistics
of inclusion of any RT extension. Please people, get a grip, we're
just trying to figure out the best way to do this stuff.

Karim
-- 
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 1-866-677-4546

Re: Attempted summary of "RT patch acceptance" thread

[Nicolas Pitre]

On Mon, 13 Jun 2005, Karim Yaghmour wrote:

> Bare in mind that I wasn't even talking about whether or not
> PREEMPT_RT was making it in. I was only talking about the logistics
> of inclusion of any RT extension. Please people, get a grip, we're
> just trying to figure out the best way to do this stuff.

The best way to do this stuff has already been outlined by Ingo himself, 
and as far as I know he's still the man leading the show.  And no one 
objected to his strategy so far.

There is just no rush.  It doesn't have to be integrated ASAP, and 
actually not before a while, and then certainly not in a disturbing way.  
Lots of things have to be cleaned up in the patch, and even in the 
current stock kernel for that matter, and Ingo even agreed to that.

So any discussion about merging of PREEMPT_RT into mainline is simply 
hand waving at this point.  Let it evolve and resume this discussion 
when there is actually something being proposed for merging into 
mainline.


Nicolas

Re: Attempted summary of "RT patch acceptance" thread

[Karim Yaghmour]

Nicolas Pitre wrote:
> The best way to do this stuff has already been outlined by Ingo himself, 
> and as far as I know he's still the man leading the show.

Touchy we are I see. Sorry to disappoint you, but I'm not interested
in stealing anyone's show.  I've got my own workload thank you very
much.

>  And no one 
> objected to his strategy so far.

Objections have been voiced. Go back and read earlier postings.

> There is just no rush.  It doesn't have to be integrated ASAP, and 
> actually not before a while, and then certainly not in a disturbing way.  
> Lots of things have to be cleaned up in the patch, and even in the 
> current stock kernel for that matter, and Ingo even agreed to that.

Sorry, but go back and check who's asking for ASAP. Then go back a
few e-mails and read back what I said:
> So one can keep ignoring the reality of the existing clash here,
> but it may just be that backing off a little and thinking about it
> in less absolute integration terms would help.

Certainly you realize that "backing off" and "thinking" have little
to do with any sort of "rush".

> So any discussion about merging of PREEMPT_RT into mainline is simply 
> hand waving at this point.  Let it evolve and resume this discussion 
> when there is actually something being proposed for merging into 
> mainline.

So in your opinion, we should all put our brains on idle until someone
posts a patch containing the words "please apply"?

Understand that I've got no inclination to going into a flamewar.
I just made some rearrangement suggestion and all hell broke loose
with people retreating to their defense positions reiterating
POVs that had little to do with what I was suggesting.

You can disagree with what I'm saying, but please read the previous
posts for context and don't put words in my mouth that I haven't said.

Karim
-- 
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 1-866-677-4546

Re: Attempted summary of "RT patch acceptance" thread

[Nicolas Pitre]

On Mon, 13 Jun 2005, Karim Yaghmour wrote:

> 
> Nicolas Pitre wrote:
> > The best way to do this stuff has already been outlined by Ingo himself, 
> > and as far as I know he's still the man leading the show.
> 
> Touchy we are I see. Sorry to disappoint you, but I'm not interested
> in stealing anyone's show.  I've got my own workload thank you very
> much.

Wow...  Did I really offend you so much?

I pretty know who you are, in case you forgot that we know each other, 
and your reading into my words that I somehow might have been suggesting 
that you would do that work yourself is rather amusing.

> >  And no one 
> > objected to his strategy so far.
> 
> Objections have been voiced. Go back and read earlier postings.

I read them.  Admitedly some people are quite enthousiastic about 
PREEMPT_RT and maybe a little too anxious to see it appear into 
mainline.  This is however not INgo's case AFAICS.

> > There is just no rush.  It doesn't have to be integrated ASAP, and 
> > actually not before a while, and then certainly not in a disturbing way.  
> > Lots of things have to be cleaned up in the patch, and even in the 
> > current stock kernel for that matter, and Ingo even agreed to that.
> 
> Sorry, but go back and check who's asking for ASAP. Then go back a
> few e-mails and read back what I said:
> > So one can keep ignoring the reality of the existing clash here,
> > but it may just be that backing off a little and thinking about it
> > in less absolute integration terms would help.

Thing is that there is just no existing clash.  At least not yet.  The 
whole thing is not ready for merging as Ingo said and therefore any 
discussion on merging issues are rather premature at this point.

> > So any discussion about merging of PREEMPT_RT into mainline is simply 
> > hand waving at this point.  Let it evolve and resume this discussion 
> > when there is actually something being proposed for merging into 
> > mainline.
> 
> So in your opinion, we should all put our brains on idle until someone
> posts a patch containing the words "please apply"?

Exactly.  How often has it been said on this very list and elsewhere: 
"show me the patch"?  In other words, there is no point discussing any 
merging issues when the project leader himself says it is not ready and 
has not posted any patch labeled as merge candidate.  I somehow trust 
Ingo for posting patches when they are ready to be reviewed for 
integration into mainline +_if_ and _when_ it is time.  Until then loads 
of changes may happen in the PREEMPT_RT code randering any present 
discussion on merging rather moot.

There is no gain into getting too excited about things.
And this sentence is not directed at Karim in particular.
I'm sure the concerned people will recognize themselves.


Nicolas

Re: Attempted summary of "RT patch acceptance" thread

[Nicolas Pitre]

On Mon, 13 Jun 2005, Nicolas Pitre wrote:

> On Mon, 13 Jun 2005, Karim Yaghmour wrote:
> 
> > So in your opinion, we should all put our brains on idle until someone
> > posts a patch containing the words "please apply"?
> 
> Exactly.  [...]

... unless you wish to get involved in the development of that 
particular patch of course.  ;-)


Nicolas

Re: Attempted summary of "RT patch acceptance" thread

[Karim Yaghmour]

Nicolas Pitre wrote:
> Wow...  Did I really offend you so much?

People have been edgy on these thread, so I read your response in
light of that.

> I pretty know who you are, in case you forgot that we know each other, 
> and your reading into my words that I somehow might have been suggesting 
> that you would do that work yourself is rather amusing.

Yes, I'd say we know each other quite well, hence my surprise at
the reading I made of your tone. I'm glad that you've read this
must amusement, we'll laugh it off next month at OLS :)

> Thing is that there is just no existing clash.  At least not yet.  The 
> whole thing is not ready for merging as Ingo said and therefore any 
> discussion on merging issues are rather premature at this point.

I know Ingo's clearly stated that he isn't asking for it to be
included at this point. This I don't deny. Ingo, however, is
not alone in working on this, as you noted yourself ...

> Exactly.  How often has it been said on this very list and elsewhere: 
> "show me the patch"?  In other words, there is no point discussing any 
> merging issues when the project leader himself says it is not ready and 
> has not posted any patch labeled as merge candidate.  I somehow trust 
> Ingo for posting patches when they are ready to be reviewed for 
> integration into mainline +_if_ and _when_ it is time.  Until then loads 
> of changes may happen in the PREEMPT_RT code randering any present 
> discussion on merging rather moot.

Ah, well then please read my proposals in light of the sub-topic
that's been floating around in those threads regarding PREEMPT_RT
and fusion being orthogonal and how they could be used together.

As for "show me the patch", posting patches to LKML is by
definition asking for some measure of feedback. You're correct
in stating that until a proposal is made for inclusion, things
may change. Nothing precludes making suggestions though ...

In any case, I hope the above has clarified a few things. In
the mean time, I'll get back to the load of unrelated stuff that
I need to work on here.

Cheers,

Karim
-- 
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 1-866-677-4546

Re: Attempted summary of "RT patch acceptance" thread

[Eugeny S. Mints]

Karim Yaghmour wrote:
> Daniel Walker wrote:
> 
>>I wouldn't work on RT if mainline integration wasn't on the agenda. 
> 
> 
> Mainline integration IS what I'm talking about. It's just not done
> the same way.
> 
> 
>>There is going to be positive , and negative discussion on this. I think
>>in the end the maintainers (Linus, and Andrew) don't want "people" to
>>get a patch or modification from the outside. It's best if the community
>>is not separated .. If you make a clean integration , and people want
>>what you are doing, there is no reason for it to be rejected.
> 
> 
> I'm not suggesting the separation of the community, I'm suggesting
> a strategy of integration based on the fact that a large portion of
> kernel contributors don't necessarily care about RT, and most don't
> want to care about it in their day-to-day work 
I would suggest that "don't want to care about it" already led and could 
lead to more bugs similar to the bugs discovered due to RT 
enchancements. But the bottom line here is that these bugs are almost 
always bugs in SMP kernel as well about which each kernel developer have 
to worry about.
>(though I think most
> would care that Linux could have an additional spade down its
> sleeve, and would certainly try to help in as much they can from
> time to time.)
> 
> I'm not suggesting asking "people" to get patches from the outside.
> What I'm saying is that those developing mainstream code shouldn't
> need to worry about anything real-time, including modifications to
> locking primitives in headers (be they defined out or in).
> 
> In essence, what you ask can only hold if all kernel developers
> intend for Linux to become QNX. Clearly this isn't going to happen.
> Whatever changes are made to such core functionality as locking
> primitives and interrupt handling can hardly be "transparent"
> simply by wrapping #ifdef CONFIG_X around it in mainstream headers.
Do you hardly working on the kernel patched with RT patch in other 
configurations than PREEMPT_RT? But I'm working and can;t report any 
outstanding issues/degradation. These changes _are_ "transperent" due to 
preemtp modes configurability. So what do you have behind "can hardly be 
"transparent" simply by wrapping #ifdef CONFIG_X around it"?
> 
>>From my point of view, determinism and best overall performance are
> conflicting goals. Having separate derectories for something as
> fundamentally different from best overall performance as determinism
> is not too much to ask.
configurability of the kernel gives you possibily to choose what you 
want in your custom case. Source code organization is a matter of 
whatever else (readability, complexity, separation of completely 
different functionality (but this is not this case), desire for 
community acceptance, etc) but not that determinism and best overall 
performance are conflicting goals. Why don;t you object against SMP 
though it definitly could conflict with goals of desktop kernel 
configurations?

	Eugeny
> Karim

Re: Attempted summary of "RT patch acceptance" thread

[Gerrit Huizenga]

On Mon, 13 Jun 2005 18:00:10 EDT, Karim Yaghmour wrote:
> 
> Daniel Walker wrote:
> > I wouldn't work on RT if mainline integration wasn't on the agenda. 
> 
> Mainline integration IS what I'm talking about. It's just not done
> the same way.
> 
> > There is going to be positive , and negative discussion on this. I think
> > in the end the maintainers (Linus, and Andrew) don't want "people" to
> > get a patch or modification from the outside. It's best if the community
> > is not separated .. If you make a clean integration , and people want
> > what you are doing, there is no reason for it to be rejected.
> 
> I'm not suggesting the separation of the community, I'm suggesting
> a strategy of integration based on the fact that a large portion of
> kernel contributors don't necessarily care about RT, and most don't
> want to care about it in their day-to-day work (though I think most
> would care that Linux could have an additional spade down its
> sleeve, and would certainly try to help in as much they can from
> time to time.)
 
It is probably worth pointing out that when companies like IBM and
SGI (and many others that I don't want to slight) committed to
making Linux scalable, they faced very similar issues to what you
are facing now.  Specifically, the community at that time (and even in
many cases now) were not concerned about scalability _at all_.

Hence, the burdn was on those advocating scalability to find ways that
minimized the code impact, that minimized the need for distinct CONFIG
options, that allowed distros to build "one version" which worked
everywhere, and at the same time simplified, or, at least, did not
add substantial complexity.

I believe that any effort towards mainline support of RT has to follow
a similar set of guidelines.  And, I believe strongly that *most* of the
RT code should be crafted so that every single laptop user is running
most of the code *and* benefiting from it.  If most of the RT code goes
unused by most of the population, and the only way to get an RT kernel
of any reasonable level is to ask the distros to build yet another
configuration, RT will always be a poor, undertested, underutilized
ugly stepchild of Linux.

And, in fact, RT has some goals which are very much in common with
many Linux users.  Simply the focus on audio and video playback and
record capabilities and the realtime needs for those common applications
should provide the basis for commonality in approach and implementation
of RT code in the current mainline.

Of course, this leaves a not inconsiderable burden on the RT community
to bridge their knowledge with the overall community goals of simplicity,
code cleanliness, comprehensibility, maintainability, etc.  But such an
approach has a greater chance of success than replicating a "kernel in
a kernel" approach where a CONFIG option enables an entirely different set
of RT based characteristics from the mainline kernel that is most widely
used.

> I'm not suggesting asking "people" to get patches from the outside.
> What I'm saying is that those developing mainstream code shouldn't
> need to worry about anything real-time, including modifications to
> locking primitives in headers (be they defined out or in).
 
I would disagree a bit.  Making it *simple* to give the kernel RT
behavior in more and more subsystems will over time work much like
scalability does today.  If patches are agressively vetted, reviewed,
tested, and commented on as they come in, e.g. via the -mm tree, it
is possible to drive common development towards better and better RT
behavior.   Of course, this requires simple APIs, lots of education,
occasional tools like sparse, or simple patches to measure latency as
has been done many times in the past.

> In essence, what you ask can only hold if all kernel developers
> intend for Linux to become QNX. Clearly this isn't going to happen.
> Whatever changes are made to such core functionality as locking
> primitives and interrupt handling can hardly be "transparent"
> simply by wrapping #ifdef CONFIG_X around it in mainstream headers.

Not all kernel developers need to subscribe to any philosophy, ever.
However, by making it simple to validate patches for the capabilities
you wish to advocate, it *is* possible to reject patches which detract
from your capabilities or impede your ability to deploy those capabilities.
And, when rejecting such patches, the burden will fall to the RT community
to show *why* the patch won't work, as well as help identify better means
for accomplishing the original patches intent, while also enabling better
RT support.

No, this is not an easy path.  But I believe it is a path towards the
ultimate success of Linux being able to provide increasingly better RT
support over time.

> From my point of view, determinism and best overall performance are
> conflicting goals. Having separate derectories for something as
> fundamentally different from best overall performance as determinism
> is not too much to ask.

I remember a day when many claimed that desktop responsiveness and
scalability were conflicting goals.  I don't see those people as
active any more and further, I note that over time, scalability
improvements have in many cases furthered desktop responsiveness as well.
You've taken the first step which is to identify the apparent conflict
of goals.  The next step is to find solutions which actually can address
both of those goals, and do it with simple, clean, easy to maintain
code.  Not easy, but I know there are a lot of smart people out here who
can rise to that particular challenge.  ;)

gerrit

Re: Attempted summary of "RT patch acceptance" thread

[Gerrit Huizenga]

On Mon, 13 Jun 2005 18:55:52 EDT, Karim Yaghmour wrote:
> 
> Bill Huey (hui) wrote:
> > I'm tell'n you that there is no way that this patch is going to
> > die. The commercial vendors and the general community is going
> > use this once they figure out what it can do and there will be
> > no stopping it. You can bet on some body extremity that this
> > will happen. It's inevitable and ultimately the future of Linux
> > whether they like it or not.
> 
> Please cool down. This is a rather extreme position.
> 
> The fact of the matter is that it is the community that is driving
> Linux, not the vendors. If the community decides something isn't
> going in, there isn't much any company can do about it, except
> maintain it outside of Linux for years on end, which economically
> speaking is just not feasible. That's why inclusion is so important.
 
Everyone scratches their own itches - and if a vendor doesn't *become*
a member of the community, including understanding the ground rules, the
motivators and de-motivators, and does not become an advocate for those
community goals, they will always be on the outside looking in.
Being a member of the community can cost you your vendor soul.  I really
like the phrasing that Andrew used in his keynote at OLS last year.
Reread this and think hard about it:

http://www.zip.com.au/~akpm/linux/patches/stuff/ols-2004.txt

In particular, I think this portion is key:

	Well, I think a similar process is coming into play as corporate
	programmers are assigned to sit in their cubicles and work on
	Linux.	They come in as good little corporate people but some
	of them are subverted.	We end up taking over their brains
	and owning them.  They become members "of the community".
	They become Free software developers.

While this is a statement of what is happening, I think it is also an
important statement about what needs to happen for the RT community to
become part of the mainline development community.  Until you have been
subverted, you are going to keep talking about how to advocate *your*
agenda.  You need to advocate the *communities'* agenda, as a member of
the subverted community who holds onto *all* of the major driving forces
and goals for Linux.

> Bare in mind that I wasn't even talking about whether or not
> PREEMPT_RT was making it in. I was only talking about the logistics
> of inclusion of any RT extension. Please people, get a grip, we're
> just trying to figure out the best way to do this stuff.

It's easy:

	1) Change your allegiance to Linux first
	2) Do the right thing.

And, implicit in that is that I believe that RT for the community will
in the long term be The Right Thing.

gerrit

Re: Attempted summary of "RT patch acceptance" thread

[Andrea Arcangeli]

On Tue, Jun 14, 2005 at 09:09:39AM -0700, Gerrit Huizenga wrote:
> RT code should be crafted so that every single laptop user is running
> most of the code *and* benefiting from it.  If most of the RT code goes

The problem is that it's not feasible to enable it by default and
pretend not to hurt the larger part of the userbase.

Only the musicians and the videogame players may get a benefit from
preempt-RT (probably the videogames players may not get a noticeable
benefit either), everyone else would be hurt somehow.

I don't think it's correct to compare this to the smp scalability
effort, where even a small 2-way was getting benefit from every SMP
optimization done for a 16-way.

Also don't forget today UP systems still run with CONFIG_SMP disable to
keep performance of those systems higher (that may change once UP
systems becomes obsolete, but not because CONFIG_SMP is useless per se ;).

The RT design emulates infinite cpus, with the associated locking
contention that infinite cpus would generate (actually the number is
bound at some point, it's not really infinite but it depends on the
hardware/software combination). To make things even worse every
contention (even if short) generates overscheduling. So the performance
impact of some workload is probably going to be very measurable for a
small system that has no need of dealing with the infinite-cpus locking.
Some slowdown number have been posted already, but probably debug
options were enabled and there's room for further optimization, they
seemed excessive bad numbers so far which can be improved, but no idea
how much.

The mouse going faster kind of thing must be always verified with a
placebo to give it any statistical value (if it cannot be measured),
otherwise by applying the new feature it'll always go faster with the
new feature applied, it happened to me as well sometime, humans aren't
very reliable since our (or at least my) perception of time vary
depending on the feelings.

To make an example I'm 100% sure that I would only get downsides by
running preempt-RT in all my production systems (including laptops,
desktop and servers), so while I may run it on my test systems to help
testing, I would never consider to enable it on any of my production
systems where I try to keep an optimal setup. This has never been the
case with the smp scalability effort, I've always got benefits in all my
smp systems, and I had CONFIG_SMP=n in the UP systems.

This is why there's a config option for preempt-rt I think, which makes
perfect sense to me and I doubt it'll be enabled by default, at least
unless they find a way to reduce the slowdown to not-measurable levels,
which I strongly doubt, especially due the irq scheduling thingy.

Perhaps the only thing that I could benefit from is such horrible 8msec
latency of the usb irq in my firewall, but that's a bug that needs
fixing by offloading it to softirq or in some other way, preempt-RT
would only hide the real problem at the expense of overscheduling.

non-RT irqs are always supposed to be as quick as possible and to do the
processing in normal kernel context after a wakeup (or softirq context
after posting a tasklet or raising a softirq, or even a kevent).

Overall I doubt preempt-RT config option objective was to be enabled by
default in the kernel that gets installed on laptops, but I could be
wrong.

Re: Attempted summary of "RT patch acceptance" thread

[Bill Huey]

On Tue, Jun 14, 2005 at 09:41:22AM -0700, Gerrit Huizenga wrote:
> Everyone scratches their own itches - and if a vendor doesn't *become*
> a member of the community, including understanding the ground rules, the
> motivators and de-motivators, and does not become an advocate for those
> community goals, they will always be on the outside looking in.
...

I agree. You're one of the few folks from IBM and other large companies
to understand this. There's a real lack of technical leadership in this
community and it is a fear of change in general as well as to this patch
because of what seem to be an inability to think through various problems
like this and be comfortable with it. That's why I went to FreeBSD for a
while since the Linux VM system back in the day was doing nothing be
thrashing. Even in 2005 it's still got problems relative to the BSDs for
load handling it seems and this why I still track projects like DragonFly
BSD since they are much more friendly to heavy design changes unlike Linux.

> While this is a statement of what is happening, I think it is also an
> important statement about what needs to happen for the RT community to
> become part of the mainline development community.  Until you have been
> subverted, you are going to keep talking about how to advocate *your*
> agenda.  You need to advocate the *communities'* agenda, as a member of
> the subverted community who holds onto *all* of the major driving forces
> and goals for Linux.

In my case, I was already subverted by Linux and other open source projects.
I just happen to get hired by an RTOS vendor to continue this work in some
way.
 
> It's easy:
> 
> 	1) Change your allegiance to Linux first
> 	2) Do the right thing.
> 
> And, implicit in that is that I believe that RT for the community will
> in the long term be The Right Thing.

This implications of this patch are huge. It's transparent and will ultimately
change the userspace land scape for just about everything despite the paranoia
and disinformation spread by various arch-conservatives in this community.

bill

Re: Attempted summary of "RT patch acceptance" thread

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 338 bytes --]

On Tue, 14 Jun 2005 12:20:56 PDT, Bill Huey said:

> This implications of this patch are huge. It's transparent and will ultimately
> change the userspace land scape for just about everything despite the paranoia
> and disinformation spread by various arch-conservatives in this community.

Is this patch running for political office? :)

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Attempted summary of "RT patch acceptance" thread

[Gerrit Huizenga]

On Tue, 14 Jun 2005 12:20:56 PDT, Bill Huey wrote:
> On Tue, Jun 14, 2005 at 09:41:22AM -0700, Gerrit Huizenga wrote:
> > It's easy:
> > 
> > 	1) Change your allegiance to Linux first
> > 	2) Do the right thing.
> > 
> > And, implicit in that is that I believe that RT for the community will
> > in the long term be The Right Thing.
> 
> This implications of this patch are huge. It's transparent and will ultimately
> change the userspace land scape for just about everything despite the paranoia
> and disinformation spread by various arch-conservatives in this community.

Hmm.  I think a few people have misunderstood one aspect of my position
here which bears clarification:

I'm making __NO__ statement on the validity of the specific RT patch
under circulation.  Well, no statement on whether it should go into
mainline as it is.  I think it is a cool patch, from what I've seen and
read, it is way better than most things that have been kicked around.

However, what I __AM__ saying is:  Linux over time can tolerate changes
which will improve its ability to be used in various levels of Real Time
situations.  Those improvements may start out small, such as analyses
of latency problems, or locking improvements and such.  However, the
thing about understanding the community is to figure out *which* solutions
are the right solutions that move things closer to real time.  If this
patch fits all of the community goals and guidelines for simplicity,
easy to maintain, clear, and evolves in the right general direction, then,
yes, it is probably the right patch.  But even if it isn't, that doesn't
mean the end of RT support for Linux - it just means that the community
needs to continue to collaborate in the right direction and this particular
patch can be used as a yardstick for progress in some cases.

This isn't about defending a particular patch to the death (even if it
is the best one we've seen so far) but about evolving solutions to
resolve the core problem while not breaking anything else.  This patch
*might* be the next step in that evolution.  It *looks* like it could
be a good step in the right direction and useful to test in -mm or the
like for a while.  But odds are it will also introduce some problems and
need some continued evolution as well.

gerrit

Re: Attempted summary of "RT patch acceptance" thread

[Gene Heskett]

On Tuesday 14 June 2005 15:35, Valdis.Kletnieks@vt.edu wrote:
>On Tue, 14 Jun 2005 12:20:56 PDT, Bill Huey said:
>> This implications of this patch are huge. It's transparent and
>> will ultimately change the userspace land scape for just about
>> everything despite the paranoia and disinformation spread by
>> various arch-conservatives in this community.
>
>Is this patch running for political office? :)

That's entirely possible Valdis.  This patch has intrigued me for 
almost all the time that Ingo has been working on it, and I'm 
currently running 48-19.  Now he has converted a few others to both 
help and guide.  This cannot be anything but a plus for linux in the 
long view.  Unforch, I see, call it from the vantage point of an 
elder geek or whatever viewpoint, the digging in of heels and the 
speechifying against the changes proposed by those that in some sense 
consider themselves the 'guardian of the flame' or something.

I am somewhat reminded of Peter the Great, whose statement at the time 
was that he was going to bring Russia into the 20th century, kicking 
and screaming if neccessary.   I guess this is the kicking and 
screaming phase?

But, I also see this as a sign of health because I've read some pretty 
decent and very educational arguments here (I even half understood 
some of them!), with only minimal name-calling (I guess you all 
reserve that for JS maybe?) and have seen a few change their mind to 
become 'supporters of the cause'.

So lets all just keep on keeping on.  This WILL sort itself given 
enough time to get the application programmers onboard.  It is they 
who will find that they are the ones who need this, but they will 
also need to learn how to code to take advantage of it to its 
fullest.  For that, it needs a stable interface.  But its only going 
to become 'stable' by going mainline so it gets enough exposure and 
everything gets sorted eventually.

A sample app that needs help badly is kmail, any background processes 
involved with incoming mail sorting and filtering result in a frozen 
user interface for the duration of the background process.  So any 
kde people copying the mail here, sic 'em!  My guess is that this is 
exactly what you need to remove that particular user headache/itch.  
That's one of my personal itches...

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.35% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com and AOL/TW attorneys please note, additions to the above
message by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.