The most insane proposal in regard to the Linux kernel development

The most insane proposal in regard to the Linux kernel development

[Artem S. Tashkinov]

Hello all,

It's not a secret that there are two basic ways of running a Linux 
distribution on your hardware. Either you use a stable distro which has 
quite an outdated kernel release which might not support your hardware 
or you run the most recent stable version but you lose stability and you 
are prone to regressions.

This problem can be solved by decoupling drivers from the kernel and 
supplying them separately so that you could enjoy stable kernel version 
X with brand new drivers like it's done in most other proprietary OS'es. 
I've been thinking of asking Linus about this decoupling for years 
already but I'm hesitant 'cause I'm 99.99999% sure he will downright 
reject this proposal. Still I'm gonna risk asking 'cause there are 
multiple pluses with this proposal:

1) We might have truly stable really long term supported kernels (3-5 
years of more).
2) The kernel size will be reduced by two orders of magnitude.
3) The user will be free to try different kernel drivers versions 
without leaving his/her stable kernel.
4) Drivers will become easier to develop, debug and maintain (usually 
the developer will just have two kernel trees to target and test 
against).
5) There will be a sense of QA/QC and accountability (nothing like that 
exists at the moment as reflected by a very long list regressions for 
every kernel release).
6) Drivers regressions will be easier to spot ('cause you can be sure 
that no other kernel changes have had undesired consequences/conflicts - 
right now driver A might break and does occasionally break because 
unrelated feature B has been reworked/tweaked/etc.).
7) There will be a lot fewer kernel releases and no constant rush to 
update them.
8) Kernel release numbers will become meaningful again. Right now no one 
can quickly say what's the difference between kernel 4.5.0 and 4.1.0.

This way kernel development must be changed to accommodate this 
proposal:

1) Yeah, I know, you all hate that, but stable APIs and ABIs must be 
introduced and supported for, let's say, at least three to five years.
2) Like we used to have during 2.2.x, 2.4.x development cycles, unstable 
kernels with new APIs must be developed in parallel to stable ones.
3) Of course that means that drivers for every kernel tree 
(stable/unstable) must be developed in parallel. In the future, perhaps, 
several parallel drivers versions will have to be developed, e.g. 
drivers for kernels 1.0.x (stable), 1.2.x (next stable) and 1.3.x 
(unstable). However, taking into consideration that these three kernel 
releases span the range of 3..5 * 3 years = 9..15 years, older kernels 
will stop being supported eventually.

In short I'm offering a concept of Windows NT kernel development. They 
have very rare stable kernel releases (e.g. XP SP0, SP1, SP2, 2003, 2003 
R2 - all binary compatible), then Vista kernel began development and 
after its release six years later, hardware vendors had to support just 
two kernel releases. Not that is a big issue.

One very big justification of this proposal is that core Linux 
development (I'm talking about various subsystems like mm/ ipc/ and 
interfaces under block/ fs/ security/ sound/ etc. ) has slowed down 
significantly over the past years so radical changes which warrant new 
kernel API/ABI are less likely to be introduced.

Please, share your opinion.

-- 
Best regards,

Artem

Re: The most insane proposal in regard to the Linux kernel development

[Al Viro]

On Sat, Apr 02, 2016 at 05:43:47PM +0500, Artem S. Tashkinov wrote:

> 1) Yeah, I know, you all hate that, but stable APIs and ABIs must be
> introduced and supported for, let's say, at least three to five
> years.

"Must"?  Whatever for?

> One very big justification of this proposal is that core Linux
> development (I'm talking about various subsystems like mm/ ipc/ and
> interfaces under block/ fs/ security/ sound/ etc. ) has slowed down
> significantly over the past years so radical changes which warrant
> new kernel API/ABI are less likely to be introduced.

Has it?

> Please, share your opinion.

Your mail setup is a disgrace - it has managed to delay your AFD posting
until the next evening.

Re: The most insane proposal in regard to the Linux kernel development

[Greg KH]

On Sat, Apr 02, 2016 at 05:43:47PM +0500, Artem S. Tashkinov wrote:
> One very big justification of this proposal is that core Linux development
> (I'm talking about various subsystems like mm/ ipc/ and interfaces under
> block/ fs/ security/ sound/ etc. ) has slowed down significantly over the
> past years so radical changes which warrant new kernel API/ABI are less
> likely to be introduced.

That's not true at all, the change is constant, and increasing, just
look at the tree for proof of that.

> Please, share your opinion.

Please read Documentation/stable_api_nonsense.txt for my opinion, and
that of the current developers.

If you don't agree with this, that's fine, you are welcome to fork the
kernel at any specific point and keep that api stable, just like many
companies do and make money from it (SuSE, Red Hat, etc.)

best of luck with your kernel project,

greg k-h

Re: The most insane proposal in regard to the Linux kernel development

[Artem S. Tashkinov]

On 2016-04-07 01:05, Greg KH wrote:
> On Sat, Apr 02, 2016 at 05:43:47PM +0500, Artem S. Tashkinov wrote:
>> One very big justification of this proposal is that core Linux 
>> development
>> (I'm talking about various subsystems like mm/ ipc/ and interfaces 
>> under
>> block/ fs/ security/ sound/ etc. ) has slowed down significantly over 
>> the
>> past years so radical changes which warrant new kernel API/ABI are 
>> less
>> likely to be introduced.
> 
> That's not true at all, the change is constant, and increasing, just
> look at the tree for proof of that.
> 
>> Please, share your opinion.
> 
> Please read Documentation/stable_api_nonsense.txt for my opinion, and
> that of the current developers.
> 
> If you don't agree with this, that's fine, you are welcome to fork the
> kernel at any specific point and keep that api stable, just like many
> companies do and make money from it (SuSE, Red Hat, etc.)
> 
> best of luck with your kernel project,
> 

Tell me, why no one in the Linux kernel dev team is concerned that:

1) There is up to a hundred regressions in each kernel release where a 
big chunk of them are caused by internal API changes?
2) API changes sometimes require drastic changes in every related 
hardware driver and since there's no way you can realistically test the 
code or the hardware, people later discover that their hardware has 
stopped working?
3) The core kernel developers do not have enough expertise to correctly 
update the entire kernel source tree so little things get broken?
4) Developing drivers for a moving target is a Herculean job?
5) You cannot easily bisect kernel regressions because regressions are 
often caused by things _outside_ of the problem you're experiencing.
6) You cannot use new drivers for you hardware on your old kernel, 
because new drivers are incompatible with an old source tree (don't 
remind me of RHEL's kernel - it's a rare exception and they usually port 
only the drivers their respective clients use).
7) Tech unsavvy people cannot realistically debug the kernel.

Hey, please, do not tell me that you're doing a great job following 
postings in LKML or resolving bugs files in bugzilla. You do a very 
lousy job indeed - multiple postings in LKML get zero replies because a 
corresponding developer is either not subscribed to LKML at all, or he 
has missed the message. There are literally hundreds(!) of bugs in 
bugzilla which have ZERO replies. What's more, a great number of kernel 
developers do not have accounts in bugzilla and they don't read 
corresponding mailing lists.

What the hell is wrong with you guys? You're developing the kernel like 
it's your toy project.

1) There's no accountability whatsoever.
2) There are no unit tests. Not a single one.
3) There's no surefire way to contact developers who have commited "bad" 
code.
4) There's no sense of direction.
5) There's no easy way to debug the kernel.

For instance, let's talk about the revoke() call. Right now, if a 
certain IO device is removed while files on it are still open (there are 
multiple ways of opening files in Linux, starting from fopen() and 
ending with mmap()), the kernel state is basically undefined(!). Great! 
The corresponding mount point cannot be reused(!). Whatever program, 
which has its files' descriptors on this accidentally removed device, 
usually cannot gracefully quit or continue working. How on Earth this 
syscall doesn't get the utmost attention?

Then we have bug 12309(1). My last comment to this bug gives a very 
simple way of reproducing it on all Android devices.

Then we have bug 15875(2) which will probably take just ten man hours to 
be resolved, yet there is no interest at all, yet thousands of people 
have very real problems due to it.

Tell me, are you really proud of yourselves?
Tell me, do you develop the kernel for your amusement, ego, your 
employee or for average people to use?
Tell me, are you really interested in more people migrating from stable 
long term supported OSes to Linux?

I want some truly honest answers. And let's not repeat this mantra "we 
don't have enough resources". You have enough resources to break 
API/ABIs in a huge way, you have enough resources to introduce 
regressions - you only don't have enough resources to have any 
resemblance of a responsible development process.

Best regards,

Artem

1) https://bugzilla.kernel.org/show_bug.cgi?id=12309
2) https://bugzilla.kernel.org/show_bug.cgi?id=15875

Re: The most insane proposal in regard to the Linux kernel development

[Greg KH]

On Thu, Apr 07, 2016 at 02:01:03PM +0500, Artem S. Tashkinov wrote:
> On 2016-04-07 01:05, Greg KH wrote:
> > On Sat, Apr 02, 2016 at 05:43:47PM +0500, Artem S. Tashkinov wrote:
> > > One very big justification of this proposal is that core Linux
> > > development
> > > (I'm talking about various subsystems like mm/ ipc/ and interfaces
> > > under
> > > block/ fs/ security/ sound/ etc. ) has slowed down significantly
> > > over the
> > > past years so radical changes which warrant new kernel API/ABI are
> > > less
> > > likely to be introduced.
> > 
> > That's not true at all, the change is constant, and increasing, just
> > look at the tree for proof of that.
> > 
> > > Please, share your opinion.
> > 
> > Please read Documentation/stable_api_nonsense.txt for my opinion, and
> > that of the current developers.
> > 
> > If you don't agree with this, that's fine, you are welcome to fork the
> > kernel at any specific point and keep that api stable, just like many
> > companies do and make money from it (SuSE, Red Hat, etc.)
> > 
> > best of luck with your kernel project,
> > 
> 
> Tell me, why no one in the Linux kernel dev team is concerned that:
> 
> 1) There is up to a hundred regressions in each kernel release where a big
> chunk of them are caused by internal API changes?

Really?  Citation please.

> 2) API changes sometimes require drastic changes in every related hardware
> driver and since there's no way you can realistically test the code or the
> hardware, people later discover that their hardware has stopped working?

Specific examples would be nice to show here, sometimes things fall
through the cracks, but usually that's just an oversight and they get
fixed once they get reported.

> 3) The core kernel developers do not have enough expertise to correctly
> update the entire kernel source tree so little things get broken?

Citation please.

> 4) Developing drivers for a moving target is a Herculean job?

Somehow more people do it than happens for any other operating system so
perhaps you just need to take the time to learn how to do it properly?
Yes, there are ways of doing this wrong, and correctly.  How have you
been doing this?

Also note, Linux drivers are, on average, 1/3 smaller than other
operating system's drivers _because_ of our development model.  You wish
to waste more driver developer time, and bloat your system, just because
you can't keep up with the rest of us?

> 5) You cannot easily bisect kernel regressions because regressions are often
> caused by things _outside_ of the problem you're experiencing.

You do know about 'git bisect skip', right?

> 6) You cannot use new drivers for you hardware on your old kernel, because
> new drivers are incompatible with an old source tree (don't remind me of
> RHEL's kernel - it's a rare exception and they usually port only the drivers
> their respective clients use).

Never use old kernels, that's your first mistake.  If you are stuck with
an old kernel, then go get support from the company that is forcing you
to do such a horrible thing, as you are paying them for this privilege.

> 7) Tech unsavvy people cannot realistically debug the kernel.

And they can debug a kernel from other operating systems?  That makes no
sense at all.

> Hey, please, do not tell me that you're doing a great job following postings
> in LKML or resolving bugs files in bugzilla. You do a very lousy job indeed
> - multiple postings in LKML get zero replies because a corresponding
> developer is either not subscribed to LKML at all, or he has missed the
> message.

You should never just blindly email lkml, that's why we have 100+ other
subsystem mailing lists, and the MAINTAINERS file, for you to figure out
who to contact.

> There are literally hundreds(!) of bugs in bugzilla which have ZERO
> replies.

Most development groups in the kernel do not use bugzilla, rather they
use the aforementioned mailing list as that scales much better.

> What's more, a great number of kernel developers do not have
> accounts in bugzilla and they don't read corresponding mailing lists.

True about bugzilla (it doesn't scale, known thing), but not true about
mailing lists.

> What the hell is wrong with you guys? You're developing the kernel like it's
> your toy project.

Really?  What other operating system is being developed better?  No one
is forcing you to use Linux, if you don't like our development model, go
use one that you like, it's that simple.

> 1) There's no accountability whatsoever.

Define "accountability" please?  Lots of companies will be glad to take
your money for that privilege.  Perhaps you should work with one of
them.

> 2) There are no unit tests. Not a single one.

You haven't looked very hard have you?  That's a totally incorrect
statement. (hint, 'make kselftest')

> 3) There's no surefire way to contact developers who have commited "bad"
> code.

Yes there is, their emails are right there in the patch, much easier to
contact than any other operating system's developers.

> 4) There's no sense of direction.

What direction do you want?

> 5) There's no easy way to debug the kernel.

Define "easy"?  Are other operating system's kernels somehow easier to
debug?  What did you try and fail doing with Linux?

> For instance, let's talk about the revoke() call. Right now, if a certain IO
> device is removed while files on it are still open (there are multiple ways
> of opening files in Linux, starting from fopen() and ending with mmap()),
> the kernel state is basically undefined(!).

If you wish to implement revoke(), please do so,  However it's
non-trivial, which is why no other operating system has it either for
all devices (BSD has it only for a limited set of devices, the trivial
ones.)

> Great! The corresponding mount
> point cannot be reused(!). Whatever program, which has its files'
> descriptors on this accidentally removed device, usually cannot gracefully
> quit or continue working. How on Earth this syscall doesn't get the utmost
> attention?

Because in real life it's not an issue.  The resources are properly
freed, userspace is notified correctly, and everyone continues happily
along with the new device descriptor.  You have written your userspace
code to properly handle this issue, right?  :)

> Then we have bug 12309(1). My last comment to this bug gives a very simple
> way of reproducing it on all Android devices.

Great, work with those developers through email please.

> Then we have bug 15875(2) which will probably take just ten man hours to be
> resolved, yet there is no interest at all, yet thousands of people have very
> real problems due to it.

If you wish to have a new feature implemented, you have to implement it
yourself, or pay someone to implement it for you.  You can't tell other
people what to do for you, which seems to be your major objection here.
That's just a matter of how the world works, no matter what group you
work with, or operating system you use.  Sorry you have to learn it this
way.

> Tell me, are you really proud of yourselves?
> Tell me, do you develop the kernel for your amusement, ego, your employee or
> for average people to use?
> Tell me, are you really interested in more people migrating from stable long
> term supported OSes to Linux?

If someone is happy with other operating systems, great, we don't care,
that works well for them, we aren't forcing anyone to use Linux.

> I want some truly honest answers. And let's not repeat this mantra "we don't
> have enough resources". You have enough resources to break API/ABIs in a
> huge way, you have enough resources to introduce regressions - you only
> don't have enough resources to have any resemblance of a responsible
> development process.

We don't have a way for people to tell other people what to do, yes, you
are correct.  Neither does any other operating system for that manner.
But if you can find such a system, great, I highly recommend you go use
that one.

Best of luck,

greg k-h