Re: stable? quality assurance?

Martin Steigerwald
2010-09-04T19:33:32Z

[-- Attachment #1: Type: Text/Plain, Size: 7236 bytes --]

Hi again,

Am Samstag 04 September 2010 schrieb Willy Tarreau:
> On Sat, Sep 04, 2010 at 06:42:19PM +0200, Martin Steigerwald wrote:
> (...)
>
> > The main idea here is to have a two-staged freeze process and to
> > distribute the "I am only taking bug fixes" work to more people than
> > Linus.
> >
> > For this to work properly, I think at the time of the release of the
> > stable kernel subsystem maintainers and Andrew should branch their
> > trees. For example when 2.6.36 is released:
> >
> > - tree
> >
> > => 2.6.36-stable-tree
> > => tree, where 2.6.37 stuff will be going in
> >
> > Thus when subsystem maintainers take new stuff during the merge
> > window, it will be for the next kernel release already, not for the
> > current one. Except bugfix work. Whereas I think the criteria for
> > bug fix work should not be that strict than for the stable patches
> > Greg collects.
> >
> > Thus it needs to be clear: No new stuff for next kernel already two
> > weeks prior to release the current stable kernel.
>
> While I respect your beliefs on this matter (they once were mine too),
> I now realized I was wrong for several reasons :
> - most developers want to create. They (generally) test what they
> create, they believe it's flawless because it works for them. No need
> for more testing, go on with new features ; if you refuse to merge
> their new work for some time, they work on their own tree and push you
> more work at once next time.
>
> - developers need real world use cases. That means more testers.
> Developers are bad testers because they don't trigger the unexpected
> use cases. And how do you get good testers ? by motivating end users
> to test your code. Most testers will only test a new kernel to get a
> new feature. If it works for them, no need to push the tests further.
> So that means that the first RCs are the most tested, and that the
> later ones are the least tested. Thus at one point you can't hope to
> get bug reports anymore. When you see an -rc7 or -rc8, you think "hey,
> -rc4 was OK, let's wait for -final and install it".

That fits perfectly well. If the first rcs are nicely testing, then ideally
all major issues should be done, when rc7 or rc8 are reached. And thus
time can be spent on fixing the major remaining open regression. I guess
those who reported these regression are interested in testing a fix.

For me features have been number one reason to upgrade kernels as well,
but then its not a yes or no decision, but more a tuning on how much new
feature stuff each stable kernel release should have and a way to put a
little bit more attention to making a stable kernel release stable.

> - people concerned by stability don't test every release. They test
> when they can, precisely because they can't impact production. So they
> don't contribute bug reports in time. And as the 2.4 maintainer, I'm
> well aware of that, because when I break something, I only know about
> it 3-4 months later.

How does this affect my suggestion above? If as you say the first rcs are
tested better and if as I assume those who reported regressions have an
interest in testing their fixes, I think this can work out nicely.

Aside from that, I am not sure whether most people step in with rc1 or rc2
already. When I tested rc kernels - there have been some times - I usually
waited to rc3 or rc4 so I could be somewhat confident that really major
issues are fixed already.

> For this reason, I think the release rhythm can't much be changed.

I still object that for above given reasons. And cause I think that if
something does not work out perfectly it still can be improved. But I am
interested in your other suggestions as well, cause maybe its not so much
the release process but something else the issue here:

> I think that trying to evaluate and publish quality per developer or
> maintainer can have a better effect because everyone in the commit
> chain is responsible. But even doing that is hard because some changes
> touch everything and it's not obvious to say that Mr X or Y has done
> some crap.

And who judges on what is crap? Build failures could be tracked
automatically. Partly maybe even performance regression as the automated
tests from Phoronix show. Well boot failures or freezes are even more
important. But then, you are probably not judging the quality of the work
of the developer but the difficulty of the area he works on.

Nix pointed out that programming ATI Radeon cards can be quite
challenging. And I do have lots of respect for the Radeon KMS related
work. So I think it would be unfair to point at one of the Radeon KMS
developers and say to him "you did crap" for example.

I think crap does happen and am more concerned about how to handle it when
it does.

> In my opinion, reporting bugs is the most effective way of improving
> quality. If you report 10 bugs in a week on the same driver, there are
> chances that at one point this driver's author will want to take some
> time to audit his code and find other bugs before you next point your
> finger at him/her. As you see, the goal is not just to report bugs to
> get them fixed, but to educate bug authors.

Okay, my contribution then: I report bugs. I reported 4-5 kernels bugs in
the last time. I reported some before, but only occassionally. I didn't
face that many bugs prior to 2.6.34 which contributed to my admittedly
very subjective impression that kernel quality has lowered.

> I can tell you that I am an author of quite a number of bugs in another
> project (haproxy), and I absolutely hate it when a bug is detected in
> production (especially given the product's goal), to the point that the
> code is generally reworked 2, 3, 5, 10 times before being committed. Of
> course it is still not enough to catch all bugs, but since the product
> has got a widely accepted reputation of being rock solid, I think it
> works quite well afterall.

Interesting project, I am implementing a highly available active/passive
loadbalancer cluster using Corosync, Pacemaker and the IPVS frontend
Ldirectord at the moment currently at work.

> Last, developers must not betray their users' trust. When they're not
> certain of their code, this must be advertised (this is often the case
> but not always). That helps a lot end users select only reliable
> features and experience more stability.

Well for me a balance must be met: A kernel has to work good enough for me
to use it regularily. And currently 2.6.34 upto 2.6.36-rc2 on my ThinkPad
T42 simply do not fulfil that criterium. What annoys me most: Radeon KMS
already works perfectly stable on 2.6.33 for me. So the issue was not in
the initial version of Radeon KMS. It has been introduced afterwards. Thus
a supposedly more matured and stable version of it is working less stable
for me.

2.6.33-tp42-01231-g11b897c has been good to me so far. I am glad it had
not frozen yet. I better press send now.

Ciao,
--
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]