Re: [F.A.Q.] the advantages of a shared tool/kernel Git repository, tools/perf/ and tools/kvm/

Ingo Molnar
2011-11-09T08:25:14Z

* Ted Ts'o <tytso@mit.edu> wrote:

> On Tue, Nov 08, 2011 at 01:55:09PM +0100, Ingo Molnar wrote:
>
> > I guess you can do well with a split project as well - my main
> > claim is that good compatibility comes *naturally* with
> > integration.
>
> Here I have to disagree; my main worry is that integration makes it
> *naturally* easy for people to skip the hard work needed to keep a
> stable kernel/userspace interface.

There's two observations i have:

Firstly, how come that this has not actually happened in practice in
the case of perf? Looks like the (random) version compatibility
experiment i conducted yesterday should have failed spectacularly.

Secondly, within the kernel we don't have a stable ABI - we don't
even have stable APIs, and still it's a 15 MLOC project that is
thriving.

I argue that it is thriving in large part *BECAUSE* we don't have a
stable API of any sort: if stuff is broken and the whole world needs
to be fixed then we fix the whole world.

One could even make the argument that in the special case of deeply
kernel integrated tools a stable kernel/userspace interface for those
special, Linux-specific ABIs is *too expensive* and results in an
inferior end result.

I'd really love it if people started thinking outside the box a bit.
Why do people assume that *all* of the kernel project's code *has* to
run in kernel mode? It's not a valid technical restriction *at all*.

"It has been done like this for 30 years" is not a valid technical
restriction. Splitting deeply kernel related tools away from the
kernel was a valid decision 15 years ago due to kernel image size and
similar resource considerations. Today it's less and less true and we
are *actively hurting* from tools being split away from the kernel
proper.

Graphics, storage and user-space suspend are good examples i think of
separation gone bad: and the resulting mess has cost Linux distros
*the desktop market*. Think about it, the price we pay for this
inferior end result is huge.

ext4tools is an example of separation gone good. I think it's the
exception that strengthens the rule.

Why was the 2.4 to 2.6 migration so difficult? I can tell you the
distro side story: mainly because the release took too long and tools
broke left and right which created stop-ship situations. We had a
much larger ABI cross section than we could sanely handle with the
testing power we had. So we got into a negative feedback loop: the
reduction in 2.3 testers further delayed the release, which moved the
(independently evolving ...) tools further away from the to-be-2.6
kernel, which further reduced the effective testing. It was not a
sustainable. We addressed many of the problems by shortening the
release cycle to 3 months, but IMHO we have not addressed the
underlying problem of lack of integration.

Responsible release engineering is actually *easier* if you don't
have a moving target and if you have the ability to fix stuff that
breaks without being bound to an external project.

Deeply kernel integrated tools could come in the initrd and could be
offered by the kernel, statically linked images made available via
/proc/sbin or such. We could even swap them out on demand so there's
no RAM overhead. There's no technical barrier.

I'd even argue that that C library is obviously something the kernel
should offer as well - so klibc is the way to go and would help us
further streamline this and keep Linux quality high.

We could actually keep the kernel and such tools tightly integrated,
reducing the compatibility matrix. The kernel would upgrade with
these tools but it *already* upgrades with some user-space components
like the vdso so it's not a true technical barrier.

> The other worry which I've mentioned, but which I haven't seen
> addressed, is that the even if you can use a perf from a newer
> kernel with an older kernel, this causes distributions a huge
> amount of pain, since they have to package two different kernel
> source packages, and only compile perf from the newer kernel source
> package. This leads to all sorts of confusion from a distribution
> packaging point of view.
>
> For example, assume that RHEL 5, which is using 2.6.32 or something
> like that, wants to use a newer e2fsck that does a better job
> fixing file system corruptions. [...]

Firstly, it's not a big issue: if a tool comes with the kernel
package then it's part of the regular backporting flow: if you
backport a new tool to an old kernel then you do the same as if you
backported a new kernel feature to an older enterprise kernel.
Happens all the time, it's a technological problem with technological
solutions. Enterprise distros explicitly do not support
cross-distro-version package installs, so backporting will be done
anyway.

Secondly, i actually think that the obsession with using obsolete
kernel versions is silly technologically - and it has evolved that
way partly *BECAUSE* we are not integrated enough and distros fear
kernel upgrades because it had the bad habit of *breaking tools*.

The answer to that problem is to reduce the external cross section of
the kernel and make sure that tools upgrade nicely together with the
kernel - and integrating tools is a valid way to achieve that.

> > Did you consider it a possibility that out of tree projects that
> > have deep ties to the kernel technically seem to be at a relative
> > disadvantage to in-kernel projects because separation is
> > technically costly with the costs of separation being larger than
> > the advantages of separation?
>
> As the e2fsprogs developer, I live with the costs all the time; I
> can testify to the facy that they are very slight. [...]

Seriously, how can you tell that: you've never tried the integrated
approach. I testified to the fact from the first hand experience of
having tried both models of development.

> > But note that there are several OS projects that succeeded doing
> > the equivalent of a 'whole world' single Git repo, so i don't
> > think we have the basis to claim that it *cannot* work.
>
> There have indeed, and there has speculation that this was one of
> many contributions to why they lost out in the popularity and
> adoption competition with Linux. [...]

I don't see Android having "lost out" in any way, do you? I actually
see Android as being an obviously more successful approach to Linux
on the desktop than anything else seen so far. We should at minimum
stop and think about that fact, observe it, learn and adapt.

iOS also has not 'lost out' to Linux in any way.

> > But why do you have to think in absolutes and extremes all the
> > time? Why not excercise some good case by case judgement about
> > the merits of integration versus separation?
>
> I agree that there are tradeoffs to both approaches, and I agree
> that case by case judgement is something that should be done. One
> of the reasons why I've spent a lot of time pointing out the
> downsides of integration and the shortcomings in the integration
> position is that I've seen advocates claiming that the fact that
> was perf was integrated was a precedent that meant that choice for
> kvm-tool was something that should not be questioned since
> tools/perf justified anything they wanted to do, and that if we
> wanted to argue about whether kvm-tool should have been bundled
> into the kernel, we should made different decisions about perf.

I don't think Pekka claimed 'anything goes' at all when he asked
tools/kvm to be merged upstream - why are you using that strawman
argument? He listed numerous valid technological reasons why they
decided to work in the tools/kvm/ space and the results speak for
themselves.

> [...] (Specifically, the reasoning goes that the need to package up
> the kernel plus userspace meant that we had distributions in the
> Linux ecosystem, and the competition kept everyone honest. If one
> distribution started making insane decisions, whether it's forcing
> Unity on everyone, or forcing GNOME 3 on everyone, it's always
> possible to switch to another distribution. The *BSD systems
> didn't have that safety valve....)

I don't think your argument makes much sense: how come Linux, a 15
MLOC monster project running for 20 years has not been destroyed by
the "lack of the safety valve" problem? Why would adding the at most
1 MLOC deeply kernel related Linux tool and library space to the
kernel repo affect the dynamics negatively? We added more code to the
kernel last year alone.

Fact is, competition thrives within the Linux kernel as well. Why is
a coherent, unified, focused project management an impediment to a
good technological result? Especially when it comes to desktop
computers / tablets / smartphones, where having a unified project is
a *must*, so extreme are the requirements of users to get a coherent
experience.

Think about this plain fact: there's not a single successful
smartphone OS on the market that does not have unified project
management. Yes, correlation is not causation and such, but still,
think about it for a moment.

Thanks,

Ingo