random ramblings on 8xx patches (long and tedious :-)

random ramblings on 8xx patches (long and tedious :-)

[Robert P. J. Day]

   i'd like to clarify a number of things regarding potential 8xx
patches, ask some questions, make some observations, etc.  (poor
wolfgang has already seen much of this via private e-mail so i expect
he'll skim it, if that.  i'm still working thru his replies so i'll be
repeating some of the stuff i already sent him.)

What's a "patch"?
-----------------

   from the mpc850 family user's manual, i have the parameter RAM
memory map table (USB, I2C, SCC2, SPI, and so on), and it's clear that
a major purpose of patching is to relocate some of these areas
elsewhere in DPRAM by patching the microcode (did i phrase that
correctly?)  in our case, we need to relocate SMC1 so we can use SCC3
for ethernet, and we already have that working under an ancient 2.4.22
kernel so, yes, we've succesfully applied a patch.

   does that completely cover the definition of a patch?  an array of
hex values to be written to, say, 2000, or 2f00 (or elsewhere)?  and
strictly for the purposes of parameter relocation?  if not, what else
would fall under the definition of "patch" that should be available to
the developer?

How many potential patches are there?
-------------------------------------

   currently, in the latest bk pull, there is only one patch selection
option: "I2C/SPI Microcode Patch".  but clearly there are other
choices.  even though that's the only selection listed in the config
menu, if you look at micropatch.c, there's mention of an SMC patch
(which you can apparently get only by editing the file, not by just
selecting an option).  there's also reference to a USB SOF patch.
why wouldn't these also be listed in the kernel config menu?  it seems
misleading to display a choice for only a single patch when, in fact,
there are *clearly* others (specifically, SMC1 relocation to use SCC3
for ethernet which is the one we need although, admittedly, that's
kind of specialized).

   rather than that single selection in the "MPC8xx CPM Options", it
would make far more sense to have *all* patches listed as subentries
under a patch menu, like:

   MPC8xx CPM Patches -->

akin to top-level config menu entries like:

   Device drivers -->
   File systems -->

and so on, with mutually-exclusive patches deactivating one another as
they're selected.  to present the user with a single I2C/SPI patch
option is just silly.

What's the limit on patches?
----------------------------

   again, based on my reading, it seems clear that you can apply only a
limited number of patches, as each patch appears to require a certain
number of "traps" in micropatch.c  for example, if you want to apply
the SMC patch, there's this excerpt in micropatch.c:

 	/* Enable the traps to get to it. */

         commproc->cp_cpmcr1 = 0x8080;
         commproc->cp_cpmcr2 = 0x808a;
         commproc->cp_cpmcr3 = 0x8028;
         commproc->cp_cpmcr4 = 0x802a;

similarly, if you want the IIC patch:

 	/* Enable the traps to get to it. */

         commproc->cp_cpmcr1 = 0x802a;
         commproc->cp_cpmcr2 = 0x8028;
         commproc->cp_cpmcr3 = 0x802e;
         commproc->cp_cpmcr4 = 0x802c;

which suggests that not only are the SMC and IIC patches mutually
exclusive, once you pick either one, you're finished in terms of
applying other patches.  is that accurate?

   (interestingly, there's nothing in micropatch.c that forces those
two patches to be mutually exclusive.  since IIC occurs further down
in the file, it would apparently override the earlier SMC patch.  not
good.)

   and what about the USB patch?  at first glance, it doesn't appear to
require any traps, but it does set "commproc->cp_rccr", so does that
make it mutually exclusive with IIC and SMC patches?

   in short, it would be useful to know:

   * what's the definition of a patch?
   * what are the list of possible patches?
   * what are the resources required for a patch to be applied, and
     what are the consequences for patch co-existence?

The I2C/SPI patch
-----------------

   perhaps showing my unbounded ignorance but, given that the parameter
RAM memory map table shows different addresses for I2C and SPI,
respectively, is it feasible to relocate one and not the other?  why
does all the literature i read insist on referring to the *combined*
I2C/SPI patch?  just curious.  would there be any value in relocating
just one and, if you could, would that cost only two traps, leaving
open the possibility of another patch?

The USB SOF patch
-----------------

   apparently, the onboard USB on the 850 is somewhat flaky, which is
why the USB SOF patch isn't even selectable in micropatch.c, is that
it?  note the excerpt from micropatch.c

#ifdef CONFIG_USB_MPC8xx
#define USE_USB_SOF_PATCH
#endif

   a recursive grep shows that there's nothing in the entire kernel
source tree that even defines CONFIG_USB_MPC8xx, so that patch is
pretty well a non-issue.

   (side note:  apparently, up to rev. b of the processor had this
problem.  we are using rev. c which allegedly has fixed the problem,
but rev. c is not widely available.  in any case, it might still be
nice to re-activate the potential US patch, albeit with a warning
that, unless you have rev. c, you're taking your chances.  just a
thought.)

Patch dependencies on selected features
---------------------------------------

   is it reasonable for selectable patches to depend on the underlying
features having already been selected?  for example, should you only
be allowed to apply the USB patch if you selected basic USB support in
the first place?  should you only be presented with the I2C patch if
you've selected I2C support?  at the moment, even if you don't pick
I2C support in the kernel, you can still apply the I2C/SPI patch.
does this make sense?  should the kernel config files be more rigorous
in terms of dependencies?

   (in our case, i can see being presented with a patch to relocate
SMC1 for SCC3 ethernet, but only if networking was selected to begin
with.  otherwise, what's the point?)

Patch menu entries -- policy versus implementation
--------------------------------------------------

   if one comes up with a more organized (read: sane) menu for patch
selection, it might be useful for newbies (read: me) if the text that
was presented didn't refer to implementation, but rather to policy.

   for instance, it didn't mean anything to me to be asked whether i
wanted to apply the I2C/SPI patch, since i had no clue what the heck
that meant.  what *would* have made sense is for the menu entry to
have explained what the effect would have been.

   as an example, consider two menu entries:

   [ ] SMC patch                                         (bad)
   [ ] Move SMC1 to free up SCC3 for Ethernet            (good)

even a complete noob like me would at least have understood what that
second choice was telling me.

Processor-dependent patches
---------------------------

   perhaps overly ambitious, but it would be *really* cool if the
patches presented to the developer were tied to the underlying
processor.  that is, once you've selected, say, RPXlite, the only
patches displayed in the menu should be those relevant to the rpxlite.
admittedly, this might be a bit much, but it seems that there should
be some kinds of safeguards that stop you from applying patches that
are hideously inappropriate for your selected platform (with, perhaps,
the possibility of overriding those safeguards).

   anyway, as i said, that's pretty ambitious.

"Help"
------

   yes, much more "Help" info during the kernel config process. lots of
help.  buckets of help.  no excuse for "There is no help available for
this option."  (unless it really is a stunningly obvious option, but
keep in mind that one man's "stunningly obvious" is another man's
"what the hell?").

   thoughts?  i'm willing to help with menu reorg -- my only
contribution to the 2.6 kernel was to thoroughly restructure the
"Filesystems" menu.  which means i like it, and i don't care what
*you* think. ;-)

rday


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: random ramblings on 8xx patches (long and tedious :-)

[Matt Porter]

On Thu, Jul 22, 2004 at 08:00:01AM -0400, Robert P. J. Day wrote:

[Remove long explanation]

>    thoughts?  i'm willing to help with menu reorg -- my only
> contribution to the 2.6 kernel was to thoroughly restructure the
> "Filesystems" menu.  which means i like it, and i don't care what
> *you* think. ;-)

You raise a lot of interesting concerns on usability for a newbie.
Descriptive menu options with useful help info are a good thing.
Please send a patch with your proposed changes so it can be reviewed.

-Matt

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: random ramblings on 8xx patches (long and tedious :-)

[Robert P. J. Day]

On Thu, 22 Jul 2004, Matt Porter wrote:

> On Thu, Jul 22, 2004 at 08:00:01AM -0400, Robert P. J. Day wrote:
>
> [Remove long explanation]
>
>>    thoughts?  i'm willing to help with menu reorg -- my only
>> contribution to the 2.6 kernel was to thoroughly restructure the
>> "Filesystems" menu.  which means i like it, and i don't care what
>> *you* think. ;-)
>
> You raise a lot of interesting concerns on usability for a newbie.
> Descriptive menu options with useful help info are a good thing.
> Please send a patch with your proposed changes so it can be reviewed.

the first addition i'd like to see is a set of extra conditionals
added to arch/ppc/config.in, to refine the type of processor.  in some
cases, it's not enough to know that you have an 8xx, and it's *too*
refined to know that you have, say, a TDM850M.  sometimes, you need to
know only that you have an 823, or 850, or whatever.

perhaps variables:

   CONFIG_8xx_823
   CONFIG_8xx_850
   CONFIG_8xx_860

or alternatively named

   CONFIG_PPC_823
   CONFIG_PPC_850

and so on.  i've already seen code in the kernel tree (can't remember
where, sadly) that goes through a tedious preprocessor check when all
it wanted to know was the actual processor.  and all of this extra
checking and setting could be done entirely within the
arch/ppc/config.in file, with no harm to other files or small animals.
(i suspect the same might hold true for 4xx, 6xx and others, but i
haven't looked at those yet.)

i can't design this patch just yet, as i'm not sure how many different
variants are worth keeping track of.  if there's a list someone could
point me at, that would be just ducky.

rday

p.s.  sadly, there's some inconsistency in the way the current
variable names were chosen.  in that same file, we have both of
CONFIG_8xx and CONFIG_PPC_5xxx.  even thought it's more verbose, it
might have been safer to keep that "PPC" internal string to avoid
potential conflicts.  so it's a tossup as to what variable name would
be the best choice to identify an 850:

   CONFIG_850
   CONFIG_PPC_850
   CONFIG_8xx_850

thoughts?  yes, this is just me being pedantic.  it gets worse. :-)


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: random ramblings on 8xx patches (long and tedious :-)

[Wolfgang Denk]

In message <Pine.LNX.4.60.0407230821050.4487@localhost.localdomain> you wrote:
>
>    CONFIG_850
>    CONFIG_PPC_850
>    CONFIG_8xx_850
>
> thoughts?  yes, this is just me being pedantic.  it gets worse. :-)

How far do you want to take this?

"850" may not  be  detailed  enough  -  is  it  a  MPC850,  MPC850DE,
MPC850DSL, or a MPC850SR ?


There is at least 40 differnt models of 8xx processors.  Go  and  add
82xx  and  85xx  and  52xx  and than start working down the 4xx, 7xx,
74xx, ... lines? I think this gives a LONG list...

Best regards,

Wolfgang Denk

--
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-4596-87  Fax: (+49)-8142-4596-88  Email: wd@denx.de
If something is different, it's either better or worse,  and  usually
both.                                                    - Larry Wall

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: random ramblings on 8xx patches (long and tedious :-)

[Robert P. J. Day]

On Fri, 23 Jul 2004, Wolfgang Denk wrote:

> In message <Pine.LNX.4.60.0407230821050.4487@localhost.localdomain> you wrote:
>>
>>    CONFIG_850
>>    CONFIG_PPC_850
>>    CONFIG_8xx_850
>>
>> thoughts?  yes, this is just me being pedantic.  it gets worse. :-)
>
> How far do you want to take this?
>
> "850" may not  be  detailed  enough  -  is  it  a  MPC850,  MPC850DE,
> MPC850DSL, or a MPC850SR ?
>
>
> There is at least 40 differnt models of 8xx processors.  Go  and  add
> 82xx  and  85xx  and  52xx  and than start working down the 4xx, 7xx,
> 74xx, ... lines? I think this gives a LONG list...

possibly, but there's no need for it to be infinitely detailed.  if
you look at arch/ppc/config.in, there's obviously a coarse
categorization with things like CONFIG_6xx, CONFIG_8xx and so on.
at the other extreme, there's specific models like CONFIG_ARCTIC2,
etc.

there's also *some* current refinement somewhere in the middle with
conditionals like:

if [ "$CONFIG_TQM823M" = "y" -o \
        "$CONFIG_TQM850M" = "y" -o \
        "$CONFIG_TQM855M" = "y" -o \
        "$CONFIG_TQM860M" = "y" -o \
        "$CONFIG_TQM862M" = "y" -o \
        "$CONFIG_NSCU"    = "y" ]; then
         define_bool     CONFIG_TQM8xxM  y

so *someone* must have thought this approach was a good idea at some
point. :-)

so the obvious question is, is there any value for more of this
mid-level categorization?  if not, then we can just forget the idea
and move on.

but if there is, no one says it has to be patched in all at once.
different people who work with different processors can submit patches
relative to their area to make future work easier.  if someone works
primarily with, say, 4xx processors, they can submit a patch to add to
the config.in file for that family refining the 4xx family.  whatever
makes their life easier. if no one cares about a particular family, no
big deal.

as a thought, what about additional variables of the form CONFIG_850,
CONFIG_7xx, CONFIG_74xx, etc.  (sadly, as i mentioned earlier, there's
already some inconsistency between names like CONFIG_8xx and
CONFIG_PPC_5xxx with that internal "PPC_".  technically, i would have
preferred variables of the form CONFIG_PPC_????, but i think the trend
has already been established.)

so, to start things off, here's a proposal.  if this kind of
refinement would have made/will make your life easier, email me
directly with the patch you'd like to see added.  i'll collect them
all, paste them together, make sure they're aesthetically consistent,
examine the final result as if i know what i'm doing, and submit the
whole thing as one big patch.  you don't need to be perfect with the
first attempt, you can always save further, more detailed refinement
for a later patch.

thoughts?

rday

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: random ramblings on 8xx patches (long and tedious :-)

[Wolfgang Denk]

In message <Pine.LNX.4.60.0407230947570.5108@localhost.localdomain> you wrote:
>
> > There is at least 40 differnt models of 8xx processors.  Go  and  add
> > 82xx  and  85xx  and  52xx  and than start working down the 4xx, 7xx,
> > 74xx, ... lines? I think this gives a LONG list...
>
> possibly, but there's no need for it to be infinitely detailed.  if
> you look at arch/ppc/config.in, there's obviously a coarse
> categorization with things like CONFIG_6xx, CONFIG_8xx and so on.
> at the other extreme, there's specific models like CONFIG_ARCTIC2,
> etc.

But if I understood you right you intend to  use  this  selection  to
prevent the kconfig system to present options to the user which don't
apply  for this processor. So you need to know if this is for example
a MPC860, MPC860T or MPC860P because the latter two have a FEC so you
must enable the FEC options in kconfig, while the first one does not.
Or you need to know if it's MPC850 which  has  only  one  SCC,  or  a
MPC850DE, MPC850SR, or MPC850DSL which have two.

If you want to be consequent, and only present such config options to
the user that really apply to his hardware, then you  must  bite  the
bullet.

> if [ "$CONFIG_TQM823M" = "y" -o \
>         "$CONFIG_TQM850M" = "y" -o \
>         "$CONFIG_TQM855M" = "y" -o \
>         "$CONFIG_TQM860M" = "y" -o \
>         "$CONFIG_TQM862M" = "y" -o \
>         "$CONFIG_NSCU"    = "y" ]; then
>          define_bool     CONFIG_TQM8xxM  y
>
> so *someone* must have thought this approach was a good idea at some
> point. :-)

This is (1) something different (because all these use the very  same
board),  and  (2)  insignificant,  as  it it not part of the official
kernel tree.

> so the obvious question is, is there any value for more of this
> mid-level categorization?  if not, then we can just forget the idea
> and move on.

It was you who started this, with the intention to prevent  the  user
from  selecting bogus config options. [My opinion is that this is not
a  high  priority  issue;  if  we  can  provide  a  working   default
configuration  for a board this is fine; if the user changes this for
his requirements we have to assume that he knows what he is doing.
"UNIX  was not designed to stop you from doing stupid things, because
that would also stop you from doing clever things." - Doug Gwyn

> but if there is, no one says it has to be patched in all at once.

Arghh... Please don't. If we had different approaches in  the  kernel
depoending  on which processor family you're working with it would be
just worse.

> so, to start things off, here's a proposal.  if this kind of
> refinement would have made/will make your life easier, email me
> directly with the patch you'd like to see added.  i'll collect them

Please don't expect a patch from me.

Best regards,

Wolfgang Denk

--
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-4596-87  Fax: (+49)-8142-4596-88  Email: wd@denx.de
The thing is, as you progress in the Craft,  you'll  learn  there  is
another rule... When you break rules, break 'em good and hard.
                                    - Terry Pratchett, _Wyrd Sisters_

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: random ramblings on 8xx patches (long and tedious :-)

[Mark Chambers]

> The family should be sufficient (say 8xx).
>
> Then everything should be automatically set up at run-time, based
> on probing code which should detect the rest.
>
> That's what OCP should do ideally when it is ready.
>
> In this day and age wasting 50k to kill every-single last
> ugly define in drivers or the setup code is worth it; again IMHO.
>

This sounds like a good idea, especially since the setup code can probably
be thrown away after startup.  However, does this fully solve the problem?
For instance if the user is deciding whether to include a UART or USB driver
for SCC3 doesn't he needs to know at that time whether it's on chip or not?
Perhaps udev handles this situation?

Mark Chambers


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: random ramblings on 8xx patches (long and tedious :-)

[Wolfgang Denk]

In message <410123EE.4000602@intracom.gr> you wrote:
>
> IMHO we shouldn't even bother.

oops???

> Then everything should be automatically set up at run-time, based
> on probing code which should detect the rest.

You must be joking. This is for embedded systems, and code  size  and
especially boot time are critical.

Also, this suggestion does not coder Robert's intention of preventint
the user from selecting bogus configuration options for  stuff  which
doesn't exist on his chip.

Best regards,

Wolfgang Denk

--
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-4596-87  Fax: (+49)-8142-4596-88  Email: wd@denx.de
"Life sucks, but it's better than the alternative."
- Peter da Silva

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Linux is not reliable enough?

[Kevin P. Dankwardt]

I am working with a team on a project where their customer is concerned
about the reliability of Linux. The customer wants to go with QNX because of
the belief that QNX Neutrino is inherently more reliable. This belief
revolves around the differences in design where drivers in QNX do not reside
in the same address space as the (micro-)kernel.

What the team was hoping to use is a MPC5200 based system and the ELDK.

The team needs to specifically address their customer's concern that a
single driver can crash the operating system in Linux, since the driver
resides in the same memory space as the kernel. They need to present
convincing arguments to the customer's Chief Software Architect.

Does anyone know of any good resources/references to address these concerns?
Any evidence, either way, that QNX Neutrino is more reliable?

Will the ELDK be adopting any of the Carrier Grade Linux requirements for
reliability? Any other projects like this of note?

Does anyone know of any embedded Linux projects where human lives really do
depend upon Linux to be robust and reliable?

Is UserMode Linux a possibility? Can one create custom drivers for UML and
mitigate risks that way?

Any clever ideas? Any clever, actually tested, ideas?

Despite all of the hype, would any of you be willing to look a customer in
the eye and say that an embedded Linux system can be reliable enough for
human lives to depend on it?

Thanks,
Kevin Dankwardt


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Linux is not reliable enough?

[Linh Dang]

Well, X server doesn't run in the same vm space as the
kernel. however, on my PC, X is the thing that would most likely to
put the system into an unusable state (frozen keyboard, mouse and
screen).

QNX's drivers run in their own process space but they still have
direct access to the hardware. A driver can screw up the hw regardless
whatever vm-sandbox in which it run.


my 0.02$

--
Linh Dang

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Linux is not reliable enough?

[Der Herr Hofrat]

>
> Well, X server doesn't run in the same vm space as the
> kernel. however, on my PC, X is the thing that would most likely to
> put the system into an unusable state (frozen keyboard, mouse and
> screen).

in prinzipal that is the way it should be but this is not quite true
as X utilizes user-space (SUID-ROOT) drivers quite a lot - so even though
X is in its on VM it is fiddling with low level resources that are normally
under kernel control (i.e. sti/cli). So this is actually the
same problem in Linux as you noted for QNX.

hofrat

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Linux is not reliable enough?

[Wolfgang Denk]

In message <wn5pt6mw1jc.fsf@linhd-2.ca.nortel.com> you wrote:
>
> Well, X server doesn't run in the same vm space as the
> kernel. however, on my PC, X is the thing that would most likely to
> put the system into an unusable state (frozen keyboard, mouse and
> screen).

This just means that X is crashed. I'm  pretty  sure  the  system  is
stillr  unning  fine, and you can for example login over the network.
This example has lttle relevance to embedded system.

> QNX's drivers run in their own process space but they still have
> direct access to the hardware. A driver can screw up the hw regardless
> whatever vm-sandbox in which it run.

Indeed.

Best regards,

Wolfgang Denk

--
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-4596-87  Fax: (+49)-8142-4596-88  Email: wd@denx.de
Dear Lord: I just want *one* one-armed manager so  I  never  have  to
hear "On the other hand", again.

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Linux is not reliable enough?

[Mark Chambers]

Kevin,

I suspect you have a political problem here.  Mr. Chief Software Architect
(which is a title for someone who doesn't actually *do* anything) is not
going to lose his job for choosing QNX.  My suggestion is this:  Point out
that the only way to prove reliability is with testing.  Linux is open
source, it won't cost anything to put it on a side by side test, and let
Linux speak for itself.

- Linux is open source, any potential bugs are theoretically fixable.  What
do you do if QNX develops a problem?
- You can hunt around for some of Linus's comments about microkernal
architecture.  He thinks they're stupid, only he's says it more poetically.
- Don't fall into this trap of software mysticism, that one operating system
is somehow intrinsically more reliable than another.  There's good and bad
software, to be sure, but even Windows can be reliable in certain carefully
constrained environments.  It's only ones and zeros.

My $.02

Mark Chambers

----- Original Message -----
From: "Kevin P. Dankwardt" <k@kcomputing.com>
To: "Embedded Linux PPC list" <linuxppc-embedded@lists.linuxppc.org>
Sent: Friday, July 23, 2004 5:06 PM
Subject: Linux is not reliable enough?


>
> I am working with a team on a project where their customer is concerned
> about the reliability of Linux. The customer wants to go with QNX because
of
> the belief that QNX Neutrino is inherently more reliable. This belief
> revolves around the differences in design where drivers in QNX do not
reside
> in the same address space as the (micro-)kernel.
>
> What the team was hoping to use is a MPC5200 based system and the ELDK.
>
<snip>


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

MPC8245 Error No. 26

[DeLaGarza, Robert]

With regard to "Error No. 26" (quoted below), has any work been done in
the kernel or serial drivers to compensate for this error?

I've observed problems only when both UARTs were enabled, so I suppose
one solution is to enable only one UART.



Error No. 26: Concurrent writes occur to the UART registers when
UAFR[0] is set

Detailed Description:
The UART Alternate Function Register (UAFR) enables software to write
concurrently to both UART1 and UART2 registers with the same write
operation. However, this action occurs when the UARTs are individually
written to, but in addition, concurrent writes to the UART registers
occur as a result of writing to other EUMBAR registers, including the
error injection registers, when UAFR[0] is set.

Projected Impact:
Concurrent writes occur to the UART registers when UAFR(0) is set and
EUMBAR registers are written to.

Work Arounds:
Set UAFR(0) only for concurrent writes to UART registers. Clear this bit
immediately afterwards.

Projected Solution:
No plans to fix.

http://www.freescale.com/files/32bit/doc/errata/MPC8245CE.pdf


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Linux is not reliable enough?

[Marius Groeger]

On Sat, 24 Jul 2004, Mark Chambers wrote:

> that the only way to prove reliability is with testing.  Linux is open
> source, it won't cost anything to put it on a side by side test, and let
> Linux speak for itself.

Getting to the point where you can run this side by side test *will*
cost money, and typically rather much, what's more. It is not likely
that Kevin's customer is going to pay the implementation for two OSes,
even if it is only to the prototype stage.

So, thinking about the right OS for the job in advance, as they do, is
a good idea. Only the thinking must be done right, of course :-)

Regards,
Marius

--
Marius Groeger <mgroeger@sysgo.com>
SYSGO AG                      Embedded and Real-Time Software
Voice: +49 6136 9948 0                  FAX: +49 6136 9948 10
www.sysgo.com | www.elinos.com | www.osek.de | www.imerva.com


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Linux is not reliable enough?

[Mark Chambers]

> On Sat, 24 Jul 2004, Mark Chambers wrote:
>
> > that the only way to prove reliability is with testing.  Linux is open
> > source, it won't cost anything to put it on a side by side test, and let
> > Linux speak for itself.
>
> Getting to the point where you can run this side by side test *will*
> cost money, and typically rather much, what's more. It is not likely
> that Kevin's customer is going to pay the implementation for two OSes,
> even if it is only to the prototype stage.
>

Yes, a good point.  But I'm speaking with a salesman voice.  For someone who
is an expert like Kevin he can no doubt prototype something fairly quickly,
and getting the customer to see something actually working is very powerful.
It puts the ball in the Chief Software Architect's (the CSA, hereafter :-)
court to justify the additional expense of QNX.

> So, thinking about the right OS for the job in advance, as they do, is
> a good idea. Only the thinking must be done right, of course :-)
>

Indeed.  I guess I should spell out what I think is wrong with the CSAs
apparent thinking:  He points out an aspect of linux, namely that drivers
can crash the system, as an issue that somehow makes linux intrinsically
unreliable.  But if you write drivers that don't crash the system then linux
is not unreliable.  The only operating system that doesn't allow a clever
programmer to crash is one that doesn't do anything.  Microkernels, they
say, allow you to do nifty things like replace the file system without
rebooting.  So that means you could swap in a buggy filesystem and destroy
the data on your disc/flash.  Without rebooting.  Which is good since you
won't be able to boot from your corrupted filesystem, which won't show up
until the next power failure, while the poor nurse with a flashlight talks
to a guy on the phone who assures her QNX can't fail.  So every OS, and
every feature, has its pro's and con's.  The question for any CSA is not 'is
this reliable' but 'can I make a reliable system using this component'?
Will the OS eat itself, or do I only have to worry about the mistakes I
make?  A carefully constructed linux system should be good for 5 or even 6
nines of reliability.

Mark Chambers


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Linux is not reliable enough?

[Der Herr Hofrat]

>
> > On Sat, 24 Jul 2004, Mark Chambers wrote:
> >
> > > that the only way to prove reliability is with testing.  Linux is open
> > > source, it won't cost anything to put it on a side by side test, and let
> > > Linux speak for itself.
> >
> > Getting to the point where you can run this side by side test *will*
> > cost money, and typically rather much, what's more. It is not likely
> > that Kevin's customer is going to pay the implementation for two OSes,
> > even if it is only to the prototype stage.
> >
>
> Yes, a good point.  But I'm speaking with a salesman voice.  For someone who
> is an expert like Kevin he can no doubt prototype something fairly quickly,
> and getting the customer to see something actually working is very powerful.
> It puts the ball in the Chief Software Architect's (the CSA, hereafter :-)
> court to justify the additional expense of QNX.
>

prototyping and testing only can proof things if you
can reliably reproduce the rare failure cases - which limits
this posibility seriously. I guess nobody doubts
that Linux is stable under typical load situations (what
ever those may be..)

> > So, thinking about the right OS for the job in advance, as they do, is
> > a good idea. Only the thinking must be done right, of course :-)
> >
>
> Indeed.  I guess I should spell out what I think is wrong with the CSAs
> apparent thinking:  He points out an aspect of linux, namely that drivers
> can crash the system, as an issue that somehow makes linux intrinsically
> unreliable.  But if you write drivers that don't crash the system then linux
> is not unreliable.  The only operating system that doesn't allow a clever
> programmer to crash is one that doesn't do anything.  Microkernels, they
> say, allow you to do nifty things like replace the file system without
> rebooting.  So that means you could swap in a buggy filesystem and destroy
> the data on your disc/flash.  Without rebooting.  Which is good since you
> won't be able to boot from your corrupted filesystem, which won't show up
> until the next power failure, while the poor nurse with a flashlight talks
> to a guy on the phone who assures her QNX can't fail.  So every OS, and
> every feature, has its pro's and con's.  The question for any CSA is not 'is
> this reliable' but 'can I make a reliable system using this component'?
> Will the OS eat itself, or do I only have to worry about the mistakes I
> make?  A carefully constructed linux system should be good for 5 or even 6
> nines of reliability.

The issue is more the presenting of convincing
safty cases - and in that area QNX most likely has a easier game than
embedded Linux - not because Linux has less potential but because
it does not have the trak record for safty critical apps (yet)
And with the development speed of the Linux kernel a real
evaluation of the kernel is a non-trivial task - in that respect a
microkernel does have serious advantages if one can isolate components
, that is gurarty error-containment within a component, in a way that
composability is maintained even in the error case - this definitly will
be hard to do for Linux and most likely for QNX core components a lot of
this work has allready been done.

to summarize the problem - a quote from Rich Cook:

Programming today is a race between software engineers striving to
build bigger and better idiot-proof programs, and the universe
striving to produce bigger and better idiots. So far, the universe is
winning.

hofrat

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Linux is not reliable enough?

[Marius Groeger]

On Mon, 26 Jul 2004, Mark Chambers wrote:

> to a guy on the phone who assures her QNX can't fail.  So every OS, and
> every feature, has its pro's and con's.  The question for any CSA is not 'is
> this reliable' but 'can I make a reliable system using this component'?

I agree: reliability is very strongly connected to the actual
components being used, and the overall system design.

One point pro Microkernel approaches, and a one that the CSA may have
been after: they allow you to decouple things. People advocating Linux
as a "solves-everything" sometimes fail to see that there are numerous
applications which want to embrace the typical strengths of Linux
(that is, the general purpose OS with a GUI, networking, POSIX shell)
for having a nice front-end, or for non-critical functions. They still
*need* to have the critical stuff running in a certified environment.

In other words: without serious work (read: big $$$), and most likely
many, many modifications and limitations that take away almost all
dynamic that open source software is known and loved for, we're not
going to see Linux in applications which require DO-178B Level A
certification.

Having said that, we may be far beyond that CSA's intentions here. But
I think this is an interesting (albeit OT) discussion, regardless.

Regards,
Marius

--
Marius Groeger <mgroeger@sysgo.com>
SYSGO AG                      Embedded and Real-Time Software
Voice: +49 6136 9948 0                  FAX: +49 6136 9948 10
www.sysgo.com | www.elinos.com | www.osek.de | www.imerva.com

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Linux is not reliable enough?

[Robert Kaiser]

Am Montag, 26. Juli 2004 15:46 schrieb Mark Chambers:
> But if you write drivers that don't crash the system then
> linux is not unreliable.

Its not just the drivers: every line of kernel code (and there are over a
million of those in linux) has the potential to crash the system. In order to
be *really* sure that the system is reliable, one would have to give all that
code a thorough examination. Depending on how "thorough examination" is
defined (and there are approved standards for this), this effort results in
costs that quickly make the question wether the OS's source code is available
for free, or will cost a few hundred kilobucks, a non-issue.

> The only operating system that doesn't allow a
> clever programmer to crash is one that doesn't do anything.  Microkernels,
> they say, allow you to do nifty things like replace the file system without
> rebooting.

This is not really a microkernel-specific feature. I believe Linux with its
kernel modules can do this as well.

The important thing about the microkernel approach is that it allows to build
OS functionality from components, where each component runs in its own
address space and only has access to the resources it needs to do its job. A
device driver only needs access to the registers of the device it is supposed
to handle, so it can only foul up this particular device (*). If such a
driver goes haywire, it *can* not affect, e.g. other driver's hardware or
memory: the bug remains local to the software that causes it. Such a failure
affects only the offending component itself and the software modules that
rely on the services this component offers.

The benefit of this approach for safety-critical systems is that one can
identify the components that are critical to the application. If a particular
application does not require a big deal of OS functionality, then only the
few components necessary to implement it need to be scrutinized. Other
components may well exist in the system, for example to support non-critical
parts of the application, because they can not affect the critical parts.

Don't get me wrong: I'm not saying a microkernel (or even QNX) is inherently
safer than Linux. However, if done right, it can give you the freedom to
trade functional complexity against functional safety.

> So that means you could swap in a buggy filesystem and destroy
> the data on your disc/flash.  Without rebooting.  Which is good since you
> won't be able to boot from your corrupted filesystem, which won't show up
> until the next power failure, while the poor nurse with a flashlight talks
> to a guy on the phone who assures her QNX can't fail.  So every OS, and
> every feature, has its pro's and con's.  The question for any CSA is not
> 'is this reliable' but 'can I make a reliable system using this component'?
> Will the OS eat itself, or do I only have to worry about the mistakes I
> make?  A carefully constructed linux system should be good for 5 or even 6
> nines of reliability.

This may be your gut feeling, but the CSA has to *prove* it for the OS he
chooses (at least he should have to, that is his responsibility).

(Honestly: would you fly in an aircraft whose steer-by-wire system is
controlled by Linux/QNX/any other OS (name please)?)


Rob

(*) There are some more issues here which I left out for brevity: If the
device being handled by a driver is capable of DMA, it *can* crash
everything. Therefore, such drivers need special consideration. Also, for
memory-mapped I/O, access permissions to device registers need to be enforced
by the MMU, so, if there are multiple devices with their registers within the
same physical page, they can not be protected from each other. Nevertheless,
the likelihood if a wild pointer causing spurious crashes is still greatly
reduced.

----------------------------------------------------------------
Robert Kaiser                         email: rkaiser@sysgo.com
SYSGO AG
Am Pfaffenstein 14                    phone: (49) 6136 9948-762
D-55270 Klein-Winternheim / Germany   fax:   (49) 6136 9948-10


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Linux is not reliable enough?

[Mark Chambers]

Ok, I promise (mostly to myself) that this is my last comment about this.
One point I was trying to make is that assuming the underlying hardware is
good, all software is theoretically perfect.  That is, given the same set of
input conditions it will always produce the same output.  So software can
only become unreliable when applied to some real world application, when the
deterministic outputs are not the ones we wanted.  So perhaps I'm being
overly pedantic here, but I think it's relevant to our discussion, because
people may make an apples to oranges comparison when comparing linux to
other OS's.  Other OS's may provide a shrink-wrapped solution which has been
extensively tested and is relatively guaranteed for some range of inputs.
Linux, on the other hand, is more a raw material.  But because linux is open
source it can be molded to your application.  The many flavors of hard
realtime or 'carrier grade' linux are perfect examples of this.  So it comes
down to evaluating what you need to do your job, and not applying some
mystical quality of 'reliability' to a piece of software.  I do understand
what we mean by 'reliable' as a practical matter, my point is just that you
can't say whether linux is reliable or not until you know what it's required
to do.  If I wanted to create a digital alarm clock, for an extreme example,
I bet I could write a linux app that would NEVER crash. (at least until
Y4K).

By the way, it's not my 'feeling' that linux can do 5 nines.  It's been
done.  Granted, the numbers come from clusters of PCs, so 100 PCs running
without failure for a year does not necessarily translate to one PC running
for 100 years.

And finally, this is all quite relevant to the discussion about dropping 2.4
for 2.6.  It's important for linux in general that well characterized
versions of the software are available.

So as Forrest Gump would say, that's all I have to say about that.

Mark Chambers


> Its not just the drivers: every line of kernel code (and there are over a
> million of those in linux) has the potential to crash the system. In order
to
> be *really* sure that the system is reliable, one would have to give all
that
> code a thorough examination. Depending on how "thorough examination" is
> defined (and there are approved standards for this), this effort results
in
> costs that quickly make the question wether the OS's source code is
available
> for free, or will cost a few hundred kilobucks, a non-issue.
>
> > The only operating system that doesn't allow a
> > clever programmer to crash is one that doesn't do anything.
Microkernels,
> > they say, allow you to do nifty things like replace the file system
without
> > rebooting.
>
> This is not really a microkernel-specific feature. I believe Linux with
its
> kernel modules can do this as well.
>
> The important thing about the microkernel approach is that it allows to
build
> OS functionality from components, where each component runs in its own
> address space and only has access to the resources it needs to do its job.
A
> device driver only needs access to the registers of the device it is
supposed
> to handle, so it can only foul up this particular device (*). If such a
> driver goes haywire, it *can* not affect, e.g. other driver's hardware or
> memory: the bug remains local to the software that causes it. Such a
failure
> affects only the offending component itself and the software modules that
> rely on the services this component offers.
>
> The benefit of this approach for safety-critical systems is that one can
> identify the components that are critical to the application. If a
particular
> application does not require a big deal of OS functionality, then only the
> few components necessary to implement it need to be scrutinized. Other
> components may well exist in the system, for example to support
non-critical
> parts of the application, because they can not affect the critical parts.
>
> Don't get me wrong: I'm not saying a microkernel (or even QNX) is
inherently
> safer than Linux. However, if done right, it can give you the freedom to
> trade functional complexity against functional safety.
>
> > So that means you could swap in a buggy filesystem and destroy
> > the data on your disc/flash.  Without rebooting.  Which is good since
you
> > won't be able to boot from your corrupted filesystem, which won't show
up
> > until the next power failure, while the poor nurse with a flashlight
talks
> > to a guy on the phone who assures her QNX can't fail.  So every OS, and
> > every feature, has its pro's and con's.  The question for any CSA is not
> > 'is this reliable' but 'can I make a reliable system using this
component'?
> > Will the OS eat itself, or do I only have to worry about the mistakes I
> > make?  A carefully constructed linux system should be good for 5 or even
6
> > nines of reliability.
>
> This may be your gut feeling, but the CSA has to *prove* it for the OS he
> chooses (at least he should have to, that is his responsibility).
>
> (Honestly: would you fly in an aircraft whose steer-by-wire system is
> controlled by Linux/QNX/any other OS (name please)?)
>
>
> Rob
>
> (*) There are some more issues here which I left out for brevity: If the
> device being handled by a driver is capable of DMA, it *can* crash
> everything. Therefore, such drivers need special consideration. Also, for
> memory-mapped I/O, access permissions to device registers need to be
enforced
> by the MMU, so, if there are multiple devices with their registers within
the
> same physical page, they can not be protected from each other.
Nevertheless,
> the likelihood if a wild pointer causing spurious crashes is still greatly
> reduced.
>
> ----------------------------------------------------------------
> Robert Kaiser                         email: rkaiser@sysgo.com
> SYSGO AG
> Am Pfaffenstein 14                    phone: (49) 6136 9948-762
> D-55270 Klein-Winternheim / Germany   fax:   (49) 6136 9948-10
>
>
>


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Linux is not reliable enough?

[Sylvain Munaut]

Kevin P. Dankwardt wrote:

 >
 > What the team was hoping to use is a MPC5200 based system and the
 > ELDK.
 >
 >
 > Does anyone know of any embedded Linux projects where human lives
 > really do depend upon Linux to be robust and reliable?

AFAIK, the MPC5200 is not to be used in life-critical systems ...
That's what the datahseet say anyway.

Sylvain


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Could 2_4_devel support RPXlite DW LCD panel?

[Song Sam]

Hi,

I want to port 2_4_devel on RPXlite DW with
framebuffer support and tried linuxppc_2_4_devel in
linuxppc official BK tree and DENX tree repectively
but failed.Could anyone try it before?Worked?

----[The config part of frame buffer support]----
...
CONFIG_RPXLITE=y
...
CONFIG_FB=y
CONFIG_DUMMY_CONSOLE=y
...
CONFIG_FB_8xx_LCD=y
...
CONFIG_FB_RPX_LCD_NEC=y   ---[Linuxppc BK tree]
CONFIG_NEC_NL6448AC33=y   ---[DENX CVS tree]
...
CONFIG_FBCON_ADVANCED=y
...
CONFIG_FBCON_CFB8=y
CONFIG_FBCON_CFB16=y
...
CONFIG_FBCON_FONTS=y
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
...
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
...

-----------------------------------------------

Anyinput would be appreciated!

Best regards,

Sam

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: Linux is not reliable enough?

[Wells, Charles]

Mark,

A couple of comments on your comments (sorry for keeping this going).

> One point I was trying to make is that assuming the underlying hardware
> is good, all software is theoretically perfect.

I can't imagine this statement being true.  It's true that if the hardware
is bad, the software may not operate correctly, but the converse isn't true.
The following code is incorrect, regardless of the state of the hardware it
runs on:

  int a[100], b = 123;
  a[b] = 0;

I guess I'm taking exception to your use of the phrase "all software".


> That is, given the same set of input conditions it will always produce
> the same output.

If ...

0. Asynchronous interrupts are enabled, or
1. Your code reads an A/D converter and acts on that data, or
2. Your code acts on operator input, or
3. One of several other normal situations hold,

then this statement, while true, just doesn't apply. In my experience,
real-world situations that allow the assumption of software determinism
are remarkably rare.


Ultimately what were talking about here is: who has to be convinced of
the reliability of the chosen OS?  I personally spent many years
designing and deploying hospital-grade medical monitors.  If human
life is at stake, there are regulatory agencies looking over your shoulder.

In the medical business, there is our own FDA as well as a number of other
agencies (including the German TUV (IMHO the toughest taskmaster of
them all)).  You simply aren't going to sell your device until you get
approval from the appropriate regulatory agency. It is the regulatory
agencies you need to convince.

What the agencies are looking for in your submission for approval to
sell your device is extensive test data that your company is willing
assert is accurate and that demonstrates this reliability.  This is
a huge task.  So, what you do is "pass the buck."  You find a vendor
of a commercial OS that already has done this testing and you include
their test data (and their assertions) in your submission to the
regulatory agencies.


I suppose I've wandered a bit off-topic here, but it seemed relevant.

Regards,
Charlie


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Linux is not reliable enough?

[Mark Chambers]

Hah! I'm breaking my promise!  See below.

> Mark,
>
> A couple of comments on your comments (sorry for keeping this going).
>
> > One point I was trying to make is that assuming the underlying hardware
> > is good, all software is theoretically perfect.
>
> I can't imagine this statement being true.  It's true that if the hardware
> is bad, the software may not operate correctly, but the converse isn't
true.
> The following code is incorrect, regardless of the state of the hardware
it
> runs on:
>
>   int a[100], b = 123;
>   a[b] = 0;
>
> I guess I'm taking exception to your use of the phrase "all software".
>
>

What I mean is, if &a = 0x10000, then a[b] will always write 0 to 0x101ec.
That may not be smart, may not be what you intended to do, but the uP will
always do the exact same thing.  (Does this mean 'C' is unreliable because
it lets you do things like that?)

> > That is, given the same set of input conditions it will always produce
> > the same output.
>
> If ...
>
> 0. Asynchronous interrupts are enabled, or
> 1. Your code reads an A/D converter and acts on that data, or
> 2. Your code acts on operator input, or
> 3. One of several other normal situations hold,
>
> then this statement, while true, just doesn't apply. In my experience,
> real-world situations that allow the assumption of software determinism
> are remarkably rare.
>

Yes, but they are different input conditions then.

>
> Ultimately what were talking about here is: who has to be convinced of
> the reliability of the chosen OS?  I personally spent many years
> designing and deploying hospital-grade medical monitors.  If human
> life is at stake, there are regulatory agencies looking over your
shoulder.
>
> In the medical business, there is our own FDA as well as a number of other
> agencies (including the German TUV (IMHO the toughest taskmaster of
> them all)).  You simply aren't going to sell your device until you get
> approval from the appropriate regulatory agency. It is the regulatory
> agencies you need to convince.
>
> What the agencies are looking for in your submission for approval to
> sell your device is extensive test data that your company is willing
> assert is accurate and that demonstrates this reliability.  This is
> a huge task.  So, what you do is "pass the buck."  You find a vendor
> of a commercial OS that already has done this testing and you include
> their test data (and their assertions) in your submission to the
> regulatory agencies.
>
>

I agree completely, and I think you're making my point, that some sort of
instrinsic reliability isn't the real issue, rather what tools you need to
get the job done.

> I suppose I've wandered a bit off-topic here, but it seemed relevant.
>

Sure, it's fun, and we could go round and round until we get real jobs :-)

> Regards,
> Charlie
>
>


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: Linux is not reliable enough?

[Mészáros Lajos]

> >   int a[100], b = 123;
> >   a[b] = 0;
> >
> > I guess I'm taking exception to your use of the phrase "all
> software".
> >
> >
>
> What I mean is, if &a = 0x10000, then a[b] will always write
> 0 to 0x101ec.
> That may not be smart, may not be what you intended to do,
> but the uP will
> always do the exact same thing.  (Does this mean 'C' is
> unreliable because
> it lets you do things like that?)

Yes, 'C' is unreliable because writing beyond the
"maxindex" lets you overwrite other's data, other's
code and DOES make backdoor for viruses.

On the other hand testing every index every time
for min and max slowes the executing.

So what?

    Ludwig

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Linux is not reliable enough?

[Oliver Korpilla]

Mészáros Lajos wrote:
>
>Yes, 'C' is unreliable because writing beyond the "maxindex" lets
>you overwrite other's data, other's code and DOES make backdoor for
>viruses.
>
>On the other hand testing every index every time for min and max slowes
>the executing.

QNX does not, and Linux does not, and with both C is as unreliable as ever.

However, a failure in a QNX in the driver level is not as potentially
malicious as in Linux. While this does not exclude failure, and does not
say a thing about the actual quality of QNX or Linux code, it's a nice
_additional_ feature related towards stability.

I guess Linux lacking proper certification for some applications is a
much bigger obstacle in the minds of managers, anyway.

But somehow this is getting offtopic, quickly, isn't it?

With kind regards,
Oliver Korpilla

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Linux is not reliable enough?

[Conn Clark]

Oliver Korpilla wrote:
>
> Mészáros Lajos wrote:
>
>>
>> Yes, 'C' is unreliable because writing beyond the "maxindex" lets
>> you overwrite other's data, other's code and DOES make backdoor for
>> viruses.
>>
>> On the other hand testing every index every time for min and max slowes
>> the executing.
>
>
> QNX does not, and Linux does not, and with both C is as unreliable as ever.
>
> However, a failure in a QNX in the driver level is not as potentially
> malicious as in Linux. While this does not exclude failure, and does not
> say a thing about the actual quality of QNX or Linux code, it's a nice
> _additional_ feature related towards stability.
>
> I guess Linux lacking proper certification for some applications is a
> much bigger obstacle in the minds of managers, anyway.
>
> But somehow this is getting offtopic, quickly, isn't it?
>
> With kind regards,
> Oliver Korpilla
>

	If C is not reliable enough then nothing is. In assembly(or even
handcoded machine language for the real hard core people) you can do
just about anything. Since all languages must resort to this at some
point, our foundation is built on sand and we are eternaly screwed.

	Only the 4 last words of the last sentence of this statement is false.
Its not the languages job to write good solid code or verify it, its the
programmers responsibility. The only code you can truly trust is code
you have total control over. For truly solid code, no funky error
checking computer science language will ever replace good practices,
thorough testing, and documentation.

	The reason for this is obvious, at some point higher level languages
must be built using a lower language to avoid the chicken/egg paradox.
Hence the quality of a higher level language can never be assured since
its foundation is made of sand.

Its a poor musician that blames their instrument.

-- Conn Clark

*****************************************************************
Give a man a match and you heat him for a moment. Set him on fire
and you'll heat him for life.
*****************************************************************

Conn Clark
Engineering Stooge				clark@esteem.com
Electronic Systems Technology Inc.		www.esteem.com


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/