Re: Breaking system configuration in stable kernels

Re: Breaking system configuration in stable kernels

[Bill Davidsen]

On Mon, 8 Oct 2001, Chris Siebenmann wrote:

> You write:
> |   I've beaten this dead horse before, but Linux will not look to
> | management like a viable candidate for default o/s until whoever releases
> | new versions of *stable* kernel series with cosmetic changes which break
> | existing systems running earlier releases of the same stable kernel
> | series.
> [...]
> |   I love getting problems like this on my vacation, [...]
> 
>  Although I sympathise, I have to ask: why are you rolling new kernels
> (or new anythings) onto production servers without testing and a
> reversion plan? In years of experience with all sorts of vendors as well
> as Linux, I have yet to see *anyone* be reliable about this (the worst
> offender I've ever had to deal with was SGI, who would release 'urgent'
> kernel patches with crash bugs).

I certainly had a reversion plan and backups, but I don't have in-house
hardware duplicating every client configuration. But there's only so much
testing you can do before you put user load on the machine. Also, existing
machines doesn't mean irreplacable mission critical machines, I don't
upgrade unless I have a reason, and piss-poor VM performance certainly is
a reason.
 
>  I strive to never put anything on our servers that I have not tested
> on an expendable machine that is configured as closely as possible to
> the server. And I also try not to be anywhere near the leading edge
> on production servers, unless there is some strong benefit to it.
> 
>  The only time I roll something out 'right now stat!' is when it is
> an urgent security fix.

I don't have that luxury. I can't afford to have a copy of most machines,
and after load and stability testing I have to try on production machines.
None of which addresses the issue that parameter name changes buy nothing
functional, and can cause serious problems. If the module in question was
needed for boot and in the initrd file the system wouldn't boot at all and
there would be little to tell why.

I'd like Linux to be more widely used, and things like this are not
invisible to the people who choose standards.

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: Breaking system configuration in stable kernels

[Richard B. Johnson]

On Tue, 9 Oct 2001, Bill Davidsen wrote:

> On Mon, 8 Oct 2001, Chris Siebenmann wrote:
> 
> > You write:
> > |   I've beaten this dead horse before, but Linux will not look to
> > | management like a viable candidate for default o/s until whoever releases

Then educate "management". Have them use another Operating System.
Try Microsoft Windows-2000/Professional. I've got it on my lap-top.
It has never crashed so it's pretty good by Microsoft standards.
However, I just finished downloading the latest Linux-kernel by
using FTP on that machine. I didn't want to tie up this machine.
I started the download at about 11:00 this morning. It finished
around 3:00 this afternoon. We have a 1.3 GHz fiber link and the LAN
is 100 MB/s. Pretty good for something that usually takes 5 minutes
on Linux.

As I see it, you get the chance to complain about some poor performance
because you don't have anything to compare present performance against.
So, you upgrade to a minimally-tested kernel and complain that it's
not a "viable candidate for default o/s..." -your words. Guess what,
this complaint will always fall upon deaf ears.

What will help get your bleeding-edge kernel stable, is a bug report,
a performance report, and sometimes even a "Hey... This one works great!".

Don't ever expect the kernel's internals to remain constant. This means
that you will have to modify any of your custom modules for each and
every kernel you download.

You can expect the API to remain constant. If it doesn't you have
a valid beef. Otherwise, you will be talking to the forest. 

Cheers,
Dick Johnson

Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).

    I was going to compile a list of innovations that could be
    attributed to Microsoft. Once I realized that Ctrl-Alt-Del
    was handled in the BIOS, I found that there aren't any.

Re: Breaking system configuration in stable kernels

[bill davidsen]

In article <Pine.LNX.3.95.1011009151209.3636A-100000@chaos.analogic.com>
	root@chaos.analogic.com wrote:

| As I see it, you get the chance to complain about some poor performance
| because you don't have anything to compare present performance against.
| So, you upgrade to a minimally-tested kernel and complain that it's
| not a "viable candidate for default o/s..." -your words. Guess what,
| this complaint will always fall upon deaf ears.
| 
| What will help get your bleeding-edge kernel stable, is a bug report,
| a performance report, and sometimes even a "Hey... This one works great!".
| 
| Don't ever expect the kernel's internals to remain constant. This means
| that you will have to modify any of your custom modules for each and
| every kernel you download.
| 
| You can expect the API to remain constant. If it doesn't you have
| a valid beef. Otherwise, you will be talking to the forest. 

  There seems to be confusion about the meaning of "stable." The stable
kernel series should be getting bug fixes, and the untested changes
should go into the development (like 2.5) series where no one would
expect it to be stable.

  There should be no bleeding edge stable kernels, my point exactly. As
long as Linux behaves like a hacker's dream it will be perceived as
such. When a stable kernel is released someone interested in reliable
operation should take it over. We should not have "oh, shit, don't use"
stable versions. For all the new features, the -ac kernels have
consistently been more stable, because, as Alan says, "I run these
kernels on my machines" and they behave like it.

| Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).
| 
|     I was going to compile a list of innovations that could be
|     attributed to Microsoft. Once I realized that Ctrl-Alt-Del
|     was handled in the BIOS, I found that there aren't any.

  Microsoft has actually had two good ideas; the software floating
point in the FORTRAN compiler for the 8080 CP/M system (one extra bit
of precision) and MS-CHAP to prevent clear text passwords from passing
on the net. Can't think of anything else which impressed me...

-- 
bill davidsen <davidsen@tmr.com>
 "If I were a diplomat, in the best case I'd go hungry.  In the worst
  case, people would die."
		-- Robert Lipe

Re: Breaking system configuration in stable kernels

[Doug McNaught]

davidsen@tmr.com (bill davidsen) writes:

>   Microsoft has actually had two good ideas; the software floating
> point in the FORTRAN compiler for the 8080 CP/M system (one extra bit
> of precision) and MS-CHAP to prevent clear text passwords from passing
> on the net. 

Kerberos predates the latter by quite a bit.

-Doug
-- 
Let us cross over the river, and rest under the shade of the trees.
   --T. J. Jackson, 1863

Re: Breaking system configuration in stable kernels

[Horst von Brand]

Doug McNaught <doug@wireboard.com> said:
> davidsen@tmr.com (bill davidsen) writes:
> 
> >   Microsoft has actually had two good ideas; the software floating
> > point in the FORTRAN compiler for the 8080 CP/M system (one extra bit
> > of precision) and MS-CHAP to prevent clear text passwords from passing
> > on the net. 
> 
> Kerberos predates the latter by quite a bit.

And numerical analysis types will tell you that extra precision in
intermediate results is *EVIL* ;-)
-- 
Horst von Brand                             vonbrand@sleipnir.valparaiso.cl
Casilla 9G, Vin~a del Mar, Chile                               +56 32 672616

Re: Breaking system configuration in stable kernels

[Bill Davidsen]

On Thu, 11 Oct 2001, Dan Maas wrote:

> >   Microsoft has actually had two good ideas; the software floating
> > point in the FORTRAN compiler for the 8080 CP/M system (one extra bit
> > of precision) and MS-CHAP to prevent clear text passwords from passing
> > on the net. Can't think of anything else which impressed me...
> 
> How about a 2D windowing system that doesn't whinge, wheeze, sputter, and
> choke a 1.4GHz machine when doing incredibly complex things like dragging a
> window around?

Is that supposed to be in the new version? Don't do Windows much. Or is
that with some accelerated driver which actaully does it on the video
board?

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Breaking system configuration in stable kernels

[Bill Davidsen]

  I've beaten this dead horse before, but Linux will not look to
management like a viable candidate for default o/s until whoever releases
new versions of *stable* kernel series with cosmetic changes which break
existing systems running earlier releases of the same stable kernel
series.

  The last time I complained, it was about changing the name of a module
itself, this time the names of the parameters of modules have been
changed. Couldn't that have waited for 2.5, or for-bloody-ever? The names
of the parameters to the cmpci module were changed, for example from
"fm_io" to "fmio" which prevents the module from loading with a newer
kernel. And if the "options" line in modules.conf is changed, then it
won't load with older kernels. Maybe I'm to only one who has to roll back
out of a new release?

  This is serious, because the module can't be loaded by hand if
modules.conf has invalid parameters. So loading would have to be moved to
rc.local, and done with correct parameters, which eliminates demand
loading of the module. PITA when sound is only loaded to play a message to
the operator to change the backup media.

  I love getting problems like this on my vacation, I'm pissed, and I
really think it indicates a lack of attention to detail. I think I saw
this in another module while doing a quick diff and grep, but it should be
in any modules. Cosmetic changes which impact users can wait, even
including fixes in spelling in error messages, since people *do* grep logs
for them. Grrr!

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: Breaking system configuration in stable kernels

[Kain X]

[-- Attachment #1: Type: text/plain, Size: 877 bytes --]

On Mon, 2001-10-08 at 09:54, Bill Davidsen wrote:
>   I've beaten this dead horse before, but Linux will not look to
> management like a viable candidate for default o/s until whoever releases
> new versions of *stable* kernel series with cosmetic changes which break
> existing systems running earlier releases of the same stable kernel
> series.
This is why there are *distributions*.  If you're going to be upgrading
straight from the devloper's mouth, you should be prepared to check for
these errata *when you upgrade*.  I don't recall seeing any kernel
document that guarantees stability for anything but base APIs in any
(pure) kernel distribution I have ever seen.			
-- 
The Internet interprets censorship as damage and routes around it.
 -- John Gilmore
**
Amateur Nuclear Specialist
Bryon Roche, Kain <kain@imperativesoultions.com>
<kain@kain.org>

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: Breaking system configuration in stable kernels

[Tim Moore]

--

Re: Breaking system configuration in stable kernels

[Tim Moore]

Bill Davidsen wrote:
> 
>   I've beaten this dead horse before, but Linux will not look to
> management like a viable candidate for default o/s until whoever releases
> new versions of *stable* kernel series with cosmetic changes which break
> existing systems running earlier releases of the same stable kernel
> series.
>
> [cmpci module stuff]
> 
>   I love getting problems like this on my vacation, I'm pissed, and I
> really think it indicates a lack of attention to detail. I think I saw

I wouldn't use a new release of Windows or OSF/1 or Digital Unix or
Solaris in a commercial situation and linux is no different.  In this
case [cmpci] it sounds like some user's desktop which is also asking for
problems.  Having it visible to management is just too much risk
regardless of the os. 

Stick with distribution OR 2.2.x (2.2.19pre2 or higher) kernels.

rgds,
tim.
--

Re: Breaking system configuration in stable kernels

[Bill Davidsen]

On Tue, 9 Oct 2001, Tim Moore wrote:

> Bill Davidsen wrote:
> > 
> >   I've beaten this dead horse before, but Linux will not look to
> > management like a viable candidate for default o/s until whoever releases
> > new versions of *stable* kernel series with cosmetic changes which break
> > existing systems running earlier releases of the same stable kernel
> > series.
 
> I wouldn't use a new release of Windows or OSF/1 or Digital Unix or
> Solaris in a commercial situation and linux is no different.  In this
> case [cmpci] it sounds like some user's desktop which is also asking for
> problems.  Having it visible to management is just too much risk
> regardless of the os. 

The bad performance of the VM in earlier kernels is really visible to
management, which is why going to something more capable is desirable (and
works well). The issue was that Linux has always taken too long to get the
development kernel series out, which encourages development work in the
"stable" kernel. Currently the -ac kernels tend to avoid flights of fancy
and "jackpot" bad VM bahaviour. Making a parameter name change just seems
a very odd thing.
 
> Stick with distribution OR 2.2.x (2.2.19pre2 or higher) kernels.

People who are sticking with obsolete kernels aren't on this list in
general, I don't upgrade until there's a reason, but much better
performance visible to the user is definitely a reason.

I would love to see Alan do the stable releases and Linus do the
development, and a development branch at the time of the first release
candidates for a new stable branch.

If Linux is going to continue to gain credibility as an everyday, every
application o/s, which I think everyone on the list believes is desirable,
then a little effort to behave a bit more like traditional development is
a good thing. World domination take some "low animal cunning*" as well as
a better technical product.

*E. B. Griswald, MIT 1961

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.