kernel changes

kernel changes

[Pavel Zaitsev]

[-- Attachment #1: Type: text/plain, Size: 979 bytes --]

Hey guys,
I have been watching development of 2.4 since 2.4.2, I wonder wether
there would be reversion to old process where kernel source will be
solidified before starting development branch. Sysadmins where I work
are uneasy to move to linux from solaris, because of erratic changes
to the kernel such as replacement of the hardware monitoring code, rewriting
of network card drivers - all of which broke some or other software,
or simply did not work. I myself update kernel as time permits,
usually every 0.0.2+ I have spent nearly two days tracing problem
of my network, that ended up in source code of the driver that have
been radically changed for "better". Now I don't trust 2.4 line
kernel to work *at all*, so cautiously keep all old kernels in the /boot,
when upgrading.
It seems I am not only one and wonder if something will be done about that.
Regards,
p.

-- 
Research causes cancer in rats.
110461387
http://gpg.md5.ca
http://perlpimp.com

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: kernel changes

[Mark Frazer]

The answer is to treat all linus/ac/aa/... kernels as development
kernels.  Don't treat anything as stable until it's been through
a real QA cycle.  I've heard Suse, RedHat and the like don't do a
bad job at this.

Re: kernel changes

[Alan Cox]

> been radically changed for "better". Now I don't trust 2.4 line
> kernel to work *at all*, so cautiously keep all old kernels in the /boot,
> when upgrading.
> It seems I am not only one and wonder if something will be done about that.

You certainly aren't the only one. 2.4.10 both really alarms me and doesn't
survive over night on my test box

Re: kernel changes

[Ben Greear]

Mark Frazer wrote:
> 
> The answer is to treat all linus/ac/aa/... kernels as development
> kernels.  Don't treat anything as stable until it's been through
> a real QA cycle.  I've heard Suse, RedHat and the like don't do a
> bad job at this.

I agree, except that the earlier 2.4.0 kernels they are using
are also buggy to one degree or another (for instance, the Tulip
driver didn't start working for my cards untill about 2.4.8, and
by then the VM seems to have gone berserk...)  Hell, many of the
official kernels won't even compile with certain options turned
on, which is really frightening, and makes it hard to run regression
tests...

It's almost like we need to fork off a 2.5 kernel just to let the
pressure for massive change off, and let less panic'ed changes go
into the 2.4 series...

The fact that there's a 20MB patch to get from Linus to AC is also
disconcerting!!

Ben

> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
Ben Greear <greearb@candelatech.com>          <Ben_Greear@excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear

Re: kernel changes

[Alan Cox]

> The answer is to treat all linus/ac/aa/... kernels as development
> kernels.  Don't treat anything as stable until it's been through
> a real QA cycle.  I've heard Suse, RedHat and the like don't do a
> bad job at this.

We try. If you want to QA your own kernel the cerberus test suite is
publically available - and indeed the VA guys are to thank for its origins.

Alan

Re: kernel changes

[Mika Liljeberg]

Pavel Zaitsev wrote:
> Now I don't trust 2.4 line
> kernel to work *at all*, so cautiously keep all old kernels in the /boot,
> when upgrading.

Well, it's a good idea to always keep a few old kernels in /boot but I
certainly identify with your point. I like to run the bleeding edge
kernels at home but lately I've been having doubts. I've been looking
for a stable kernel since 2.4.0-test9. While 2.4.0-test9 is not exactly
bug free either, any later kernel either reboots at random (all later
test versions and early 2.4.x versions) or locks up hard on my SMP
machine. This does not appear to have anything to do with load, indeed
it often happens with the machine completely idle. It can take hours or
days before this occurs but sooner or later it does. Of course, nothing
ever gets written to the logs. With 2.4.8 I almost thought I had hit
paydirt, but no such luck. 2.4.9 crashed right off the bat. 2.4.10 seems
more unstable than most. Enough so that I finally patched in ext3 and
installed journals on my file systems just to give them a semblance of
stability (and to speed up the random reboots).

That said, I might just have a hardware problem. However, something in
the new kernels seems to touch it off. Any ideas and tools to track this
down, besides intuition and brute force, would be appreciated. [Right
now I'm running with swap disabled, on somebody's suggestion. Let's see
what happens.]

Regards,

	Mika Liljeberg

Re: kernel changes

[Dan Maas]

> The answer is to treat all linus/ac/aa/... kernels as development
> kernels.  Don't treat anything as stable until it's been through
> a real QA cycle.

I definitely have to second what you guys are saying here... 2.2.x is the
stable kernel series, 2.4.x is for caffeine-fueled developers who read the
LKML at least once every day...

e.g. I consider it extremely embarrassing that fundamental drivers like
aic7xxx, emu10k1, tulip, etc. are breaking regularly in the mainline
kernels. I haven't had any trouble with things like this in Windows for
several years now... Sure the Windows drivers are probably a few percent
slower, but as Nathan Myers once wrote, "It is meaningless to compare the
efficiency of a running system against one which might have done some
operations faster if it had not crashed."

I think we all owe major thanks to Alan Cox, who does his best to keep the
house in order amidst the chaos of kernel development (kudos to Mr. Cox for
holding on to Rik's VM design long enough for it to stabilize!). If anything
I wish there were a third primary maintainer who would take an even more
conservative stance, hanging maybe 2 minor versions behind Linus and -ac,
and only picking up changes that have been tested widely. Hmm, the people
working on distro kernels are probably just about doing this already...
Maybe if they could combine efforts somehow?

Regards,
Dan

Re: kernel changes

[Ville Herva]

On Fri, Sep 28, 2001 at 02:32:05PM -0700, you [Pavel Zaitsev] said:
> I have been watching development of 2.4 since 2.4.2, I wonder wether there
> would be reversion to old process where kernel source will be solidified
> before starting development branch.

I think you can think of each new 2.4.x kernel as a candidate for
solification. The part of the linux community a like me (and perhaps you?)
that can't contribute directly to the kernel development in form of patches,
can help by testing the kernels with a great variety of setups.  Linus,
Alan, all the other guys are working their asses off getting the code as
good, clean and fast as possible. But you can't exactly expect them to
verify each change with thousands of different machines (think of the
variety of just PC hw, then think of all the other platforms linux runs on),
like MS does. Even the linux driver developers don't have access to all the
hw they support. MS doesn't release nearly as often and they have a huge QA
department running the test suites for each change set. They also have a
direct sopprt of hw vendors (who in most cases to the drivers for MS and use
all theK interim knowledge and analyzer equipment in that.)

I've always thought when Linux or alan puts out a new kernel, it's up to the
community to do the testing part.

If you need something stable, stick with 2.2 or something that has gone
through a test cycle like the RH kernels.

If you have the hardware, help testing and report bugs.

That's the way I always figured Linux community works, and IMHO considering
the facts, it does result in pretty impressive development pace and even
stability (much thanks to the pure skill the developers have).

It does, however, seem that in 2.4 the pace has been so rapid that for each
2.4.n bug found and fixed in 2.4.(n+1), there have been a few new bugs in
2.4.(n+1). I'd expect that to get better as 2.5 forks of and the 2.4 pace
slows down.

Just my two cents.


"Dan Maas" <dmaas@dcine.com> wrote:
> e.g. I consider it extremely embarrassing that fundamental drivers like
> aic7xxx, emu10k1, tulip, etc. are breaking regularly in the mainline
> kernels. I haven't had any trouble with things like this in Windows for
> several years now...

Well. I just installed the SRP (security rollup patch MS came up with
instead of releasing proper SP7) on a NT4SP6 box. The SRP erraneously
detected the box was SMP and installed SMP versions for 2 out of 6 kernel
files. Needless to say it didn't work and was a pain to trace and fix.

Similarly, I just updated to the newest NVidia display driver for NT4. It
didn't boot. Not even to VGA mode. BSOD right in start. It took me hours and
a non-NVidia display adapter to clean up the mess. And that driver had been
QA'd at NVidia for months.

So shit happens on (supposedly stable version of) windows as well. And it is
usually more of a pain to clean up, since MS gives you little tools to work
with in case of such problem.

Of course that's not an excuse and I'd certainly like to see 2.4 to be more
stable.


-- v --

v@iki.fi

Re: kernel changes

[Andrew Ebling]

On Fri, 2001-09-28 at 23:06, Alan Cox wrote:
> > been radically changed for "better". Now I don't trust 2.4 line
> > kernel to work *at all*, so cautiously keep all old kernels in the /boot,
> > when upgrading.
> > It seems I am not only one and wonder if something will be done about that.
> 
> You certainly aren't the only one. 2.4.10 both really alarms me and doesn't
> survive over night on my test box.

It would be better to be premature starting 2.5.x than to risk damaging
the reputation of Linux by pretending an unproven kernel is stable.

Lets start 2.5.x and let Alan sort 2.4.x out.

Andrew

Re: kernel changes

[John Alvord]

On Sat, 29 Sep 2001, Ville Herva wrote:

> On Fri, Sep 28, 2001 at 02:32:05PM -0700, you [Pavel Zaitsev] said:
> > I have been watching development of 2.4 since 2.4.2, I wonder wether there
> > would be reversion to old process where kernel source will be solidified
> > before starting development branch.
> 
> I think you can think of each new 2.4.x kernel as a candidate for
> solification. The part of the linux community a like me (and perhaps you?)

One aspect that bothers me is the absence of a success criteria. The
current competition for best VM is a good example. The fact is that every
operating system will fail with a high enough load. The best you can hope
for is a better degradation then the prior release. At the moment both
2.4.10 and 2.4.9-ac16 are better then 2.2.19. But people keep testing
under higher and higher loads and (surprise) they both fail... initiating
a search for better degradation logic.

Without a success criteria, this process cannot end.

Other examples are recent updates for multi-quad NUMA machines and changes
to handle locking problems on 8-way machines.

john alvord

Re: kernel changes

[arjan]

In article <Pine.LNX.4.20.0109290937510.18362-100000@otter.mbay.net> you wrote:

> One aspect that bothers me is the absence of a success criteria.

I disagree here. Red Hat uses "must pass the cerberus test" as one of the
criteria for kernels. The are other similar criteria, most are obvious (must
boot :). All other distributions have similar tests and a few even use the
cerberus testsuite as well.

Maybe your problem is "absence of tests before Linus releases", well 
even that isn't fully true as distros run these tests on -pre kernels as
well (or -ac kernels, which are mostly in sync with -pre kernels)...

> The current competition for best VM is a good example. The fact is that
> every operating system will fail with a high enough load. The best you can
> hope for is a better degradation then the prior release.

There are a few basic creteria here as well, and 2.4.10 fails on some of
them so far:

1) Must not kill processes as long as there is plenty of swap
   or (possibly dirty) cache memory
2) Must not deadlock (as that is a code-bug)
3) Must not livelock without any progress

Note that no 2.4 kernel so far really achieves 1) in the presence of
highmem; the obvious deadlocks are just pushed further by tuning.

> At the moment both 2.4.10 and 2.4.9-ac16 are better then 2.2.19. But
> people keep testing under higher and higher loads and (surprise) they both
> fail... initiating a search for better degradation logic.

2.4.10 isn't better than 2.2.19 given the criteria above. 2.4.10aa2 might
be though... and 2.4.9acX+Rik's patches are solid in testing. 

Greetings,
   Arjan van de Ven

Re: kernel changes

[John Alvord]

On Sat, 29 Sep 2001 18:23:15 +0100, arjan@fenrus.demon.nl wrote:

>In article <Pine.LNX.4.20.0109290937510.18362-100000@otter.mbay.net> you wrote:
>
>> One aspect that bothers me is the absence of a success criteria.
>
>I disagree here. Red Hat uses "must pass the cerberus test" as one of the
>criteria for kernels. The are other similar criteria, most are obvious (must
>boot :). All other distributions have similar tests and a few even use the
>cerberus testsuite as well.
I believe you. I was worrying about the developer transition from 2.4
to 2.5, not the excellent work that the distributors do.

>
>Maybe your problem is "absence of tests before Linus releases", well 
>even that isn't fully true as distros run these tests on -pre kernels as
>well (or -ac kernels, which are mostly in sync with -pre kernels)...
As mentioned, I am not worried about the distributions.

>
>> The current competition for best VM is a good example. The fact is that
>> every operating system will fail with a high enough load. The best you can
>> hope for is a better degradation then the prior release.
>
>There are a few basic creteria here as well, and 2.4.10 fails on some of
>them so far:
>
>1) Must not kill processes as long as there is plenty of swap
>   or (possibly dirty) cache memory
>2) Must not deadlock (as that is a code-bug)
>3) Must not livelock without any progress
Is this criteria explicity accepted by all parties? And is it the only
criteria?

>
>Note that no 2.4 kernel so far really achieves 1) in the presence of
>highmem; the obvious deadlocks are just pushed further by tuning.
>
>> At the moment both 2.4.10 and 2.4.9-ac16 are better then 2.2.19. But
>> people keep testing under higher and higher loads and (surprise) they both
>> fail... initiating a search for better degradation logic.
>
>2.4.10 isn't better than 2.2.19 given the criteria above. 2.4.10aa2 might
>be though... and 2.4.9acX+Rik's patches are solid in testing. 

I read all sorts of reports, many positive and some negative.

john alvord

Re: kernel changes

[Paul Larson]

On 28 Sep 2001 23:33:39 +0100, Alan Cox wrote:
> > The answer is to treat all linus/ac/aa/... kernels as development
> > kernels.  Don't treat anything as stable until it's been through
> > a real QA cycle.  I've heard Suse, RedHat and the like don't do a
> > bad job at this.
> 
> We try. If you want to QA your own kernel the cerberus test suite is
> publically available - and indeed the VA guys are to thank for its origins.
> 
> Alan

For Linux QA/Testing, you may also want to check out the Linux Test
Project.  We just had a new release that added about 400 new testcases
which is way up from what we had before.  We also have descriptions for
all the tests now too.

-Paul Larson

Re: kernel changes

[Martin J. Bligh]

--On Saturday, September 29, 2001 09:45 AM -0700 John Alvord 
<jalvo@mbay.net> wrote:
> ...
> Without a success criteria, this process cannot end.
>
> Other examples are recent updates for multi-quad NUMA machines

I'm not sure why these bother you - if you don't turn on
CONFIG_MULTIQUAD, the code impact is practically non-existant.

Martin.

Re: kernel changes

[bill davidsen]

In article <07f901c148b8$720a6230$1a01a8c0@allyourbase> dmaas@dcine.com wrote:
| > The answer is to treat all linus/ac/aa/... kernels as development
| > kernels.  Don't treat anything as stable until it's been through
| > a real QA cycle.
| 
| I definitely have to second what you guys are saying here... 2.2.x is the
| stable kernel series, 2.4.x is for caffeine-fueled developers who read the
| LKML at least once every day...

  Not really. I have found that 2.4 kernels are usefully stable if you
pick them carefully.

| e.g. I consider it extremely embarrassing that fundamental drivers like
| aic7xxx, emu10k1, tulip, etc. are breaking regularly in the mainline
| kernels. I haven't had any trouble with things like this in Windows for
| several years now... Sure the Windows drivers are probably a few percent
| slower, but as Nathan Myers once wrote, "It is meaningless to compare the
| efficiency of a running system against one which might have done some
| operations faster if it had not crashed."

  Again, given the choice of the o/s failing every few months or not
using your hardware, where do you go? I haven't found a 2.2 kernel which
like running multiple NICs at 85% of max most of the time, while several
of the 2.4.kernels, most recently 2.4.6 will.

| I think we all owe major thanks to Alan Cox, who does his best to keep the
| house in order amidst the chaos of kernel development (kudos to Mr. Cox for
| holding on to Rik's VM design long enough for it to stabilize!). If anything
| I wish there were a third primary maintainer who would take an even more
| conservative stance, hanging maybe 2 minor versions behind Linus and -ac,
| and only picking up changes that have been tested widely. Hmm, the people
| working on distro kernels are probably just about doing this already...
| Maybe if they could combine efforts somehow?

  A quick look at the c.o.l.development.system will show I said ages ago
that we should QA "stable" kernel releases before sending them out. I
offered to set up a group to do that for each release candidate, but
there was no interest.

  The VM work is really needed, but I wish the development was in "pre"
kernels, not releases. I've been playing while on vacation, and
2.4.9-ac14 looks much better, 2.4.9-ac16 has minor problems on a machine
I won't see until I "get back to work," and I haven't deceded if I want
to try 2.4.9-ac18 or not. Sadly, horror stories on 2.4.10 have suggested
that I let others try that.

  Given the load of totally new VM stuff dropped in, I admit I'd like to
see useful stuff which only takes effect if configured (the so-called
Athlon patch) in the kernel, since many systems simply will not run an
Athlon kernel without it. I only wish the preempt was being pushed as
hard as VM, it looks really good on loaded machines.

  Perhaps it's time for 2.5 indeed.

-- 
bill davidsen <davidsen@tmr.com>
 "If I were a diplomat, in the best case I'd go hungry.  In the worst
  case, people would die."
		-- Robert Lipe

Re: kernel changes

[John Alvord]

On Wed, 3 Oct 2001 20:36:01 -0400, davidsen@tmr.com (bill davidsen)
wrote:

>In article <07f901c148b8$720a6230$1a01a8c0@allyourbase> dmaas@dcine.com wrote:
>| > The answer is to treat all linus/ac/aa/... kernels as development
>| > kernels.  Don't treat anything as stable until it's been through
>| > a real QA cycle.
>| 
>| I definitely have to second what you guys are saying here... 2.2.x is the
>| stable kernel series, 2.4.x is for caffeine-fueled developers who read the
>| LKML at least once every day...
>
>  Not really. I have found that 2.4 kernels are usefully stable if you
>pick them carefully.

If you run kernels from the developers, you have joined the linux
development test team... whether you know it or not.  The bug reports
coming in are the main output of the test team. Positive reports are
nice but contain relatively little information for developers. You may
get productive use, and that is nice, but it is not the primary
purpose of the test team.

john alvord