Re: [F.A.Q.] the advantages of a shared tool/kernel Git repository, tools/perf/ and tools/kvm/

Ingo Molnar
2011-11-08T12:57:06Z

* Theodore Tso <tytso@MIT.EDU> wrote:

>
> On Nov 8, 2011, at 4:32 AM, Ingo Molnar wrote:
> >
> > No ifs and when about it, these are the plain facts:
> >
> > - Better features, better ABIs: perf maintainers can enforce clean,
> > functional and usable tooling support *before* committing to an
> > ABI on the kernel side.
>
> "We don't have to be careful about breaking interface compatibility
> while we are developing new features".

See my other mail titled:

[F.A.Q.] perf ABI backwards and forwards compatibility

the compatibility process works surprisingly well, given the
complexity and the flux of changes.

>From the experience i have with other ABI and feature extension
efforts, perf ABI compatibility works comparably better, because the
changes always go together so people can review and notice any ABI
problems a lot easier than with an artificially fragmented
tooling/kernel maintenance setup.

I guess you can do well with a split project as well - my main claim
is that good compatibility comes *naturally* with integration.

Btw., this might explain why iOS and Android is surprisingly
compatible as well, despite the huge complexity and the huge flux of
changes on both platforms - versus modular approaches like Windows or
Linux distros.

> The flip side of this is that it's not obvious when an interface is
> stable, and when it is still subject to change. [...]

... actual results seem to belie that expectation, right?

> [...] It makes life much harder for any userspace code that
> doesn't live in the kernel. [...]

So *that* is the real argument? As long as compatibility is good, i
don't think why that should be the case.

Did you consider it a possibility that out of tree projects that have
deep ties to the kernel technically seem to be at a relative
disadvantage to in-kernel projects because separation is technically
costly with the costs of separation being larger than the advantages
of separation?

> [...] And I think we do agree that moving all of userspace into a
> single git tree makes no sense, right?

I'm inclined to agree that applications that have no connection and
affinity to the kernel (technically or socially) should not live in
the kernel repo. (In fact i argue that they should be sandboxed but
that's another topic .)

But note that there are several OS projects that succeeded doing the
equivalent of a 'whole world' single Git repo, so i don't think we
have the basis to claim that it *cannot* work.

> > - We have a shared Git tree with unified, visible version control. I
> > can see kernel feature commits followed by tooling support, in a
> > single flow of related commits:
> >
> > perf probe: Update perf-probe document
> > perf probe: Support --del option
> > trace-kprobe: Support delete probe syntax
> >
> > With two separate Git repositories this kind of connection between
> > the tool and the kernel is inevitably weakened or lost.
>
> "We don't have to clearly document new interfaces between kernel
> and userspace, and instead rely on git commit order for people to
> figure out what's going on with some new interface"

It does not prevent the creation of documentation at all - but i
argue that the actual *working commits* are more valuable information
than the documentation.

That inevitably leads to the conclusion that you cannot destroy the
more valuable information just to artificially promote the creation
of the less valuable piece of information, right?

> > - Easier development, easier testing: if you work on a kernel
> > feature and on matching tooling support then it's *much* easier to
> > work in a single tree than working in two or more trees in
> > parallel. I have worked on multi-tree features before, and except
> > special exceptions they are generally a big pain to develop.
>
> I've developed in the split tree systems, and it's really not that
> hard. It does mean you have to be explicit about designing
> interfaces up front, and then you have to have a good, robust way
> of negotiating what features are in the kernel, and what features
> are supposed by the userspace --- but if you don't do that then
> having good backwards and forwards compatibility between different
> versions of the tool simply doesn't exist.

I actually think that ext4 is a good example at ABI design - and we
borrowed heavily from that positive experience in the perf.data
handling code.

But i also worked in other projects where the split design worked a
lot less smoothly, and arguably ext4 would be *dead* if it had a
messy interface design: a persistent filesystem cannot under any
circumstance be messy to survive in the long run.

Other ABIs, not so much, and we are hurting from that.

> So at the end of the day it question is whether you want to be able
> to (for example) update e2fsck to get better ability to fix more
> file system corruptions, without needing to upgrade the kernel. If
> you want to be able to use a newer, better e2fsck with an older,
> enterprise kernel, then you have use certain programming
> disciplines. That's where the work is, not in whether you have to
> maintain two git trees or a single git tree.

I demonstrated how this actually works with perf (albeit the
compatibility requirements are a lot less severe on perf than with a
persistent, on-disk filesystem), do you accept that example as proof?

> > - We are using and enforcing established quality control and
> > coding principles of the kernel project. If we mess up then
> > Linus pushes back on us at the last line of defense - and has
> > pushed back on us in the past. I think many of the currently
> > external kernel utilities could benefit from the resulting rise
> > in quality. I've seen separate tool projects degrade into
> > barely usable tinkerware - that i think cannot happen to perf,
> > regardless of who maintains it in the future.
>
> That's basically saying that if you don't have someone competent
> managing the git tree and providing quality assurance, life gets
> hard. [...]

No, it says that we want to *guarantee* that someone competent is
maintaining it. If me, Peter and Arnaldo gets hit by the same bus or
crashes with the same airplane then i'm pretty confident that life
will go on just fine and capable people will pick it up.

With an external project i wouldn't be nearly as sure about that - it
could be abandonware or could degrade into tinkerware.

Working in groups and structuring that way and relying on the
infrastructure of a large project is an *advantage* of Linux, why
should this surprise *you* of all people, hm? :-)

> [...] Sure. But at the same time, does it scale to move all of
> userspace under one git tree and depending on Linus to push back?

We don't depend on Linus for every single commit, that would be silly
and it would not scale.

We depend on Linus depending on someone who depends on someone else
who depends on someone else. 3 people along that chain would have to
make the same bad mistake for crap to get to Linus and while it
happens, we try to keep it as rare as humanly possible.

> I mean, it would have been nice to move all of GNOME 3 under the
> Linux kernel, so Linus could have pushed back on behalf of all of
> us power users, [...]

You are starting to make sense ;-)

> [...] but as much as many of us would have appreciated someone
> being able to push back against the insanity which is the GNOME
> design process, is that really a good enough excuse to move all of
> GNOME 3 into the kernel source tree? :-)

Why not? </joking>

Seriously, if someone gave me a tools/term/ tool that has rudimentary
xterm functionality with tabbing support, written in pure libdri and
starting off a basic fbcon console and taking over the full screen,
i'd switch to it within about 0.5 nanoseconds and would do most of my
daily coding there and would help out with extending it to more apps
(starting with a sane mail client perhaps).

I'd not expect the Gnome people to move there against their own good
judgement - i have no right to do that. (Nor do i think would it be
possible technically and socially: the culture friction between those
projects is way too large IMO so it's clearly one of the clear
'HELL NO!' cases for integration.)

But why do you have to think in absolutes and extremes all the time?
Why not excercise some good case by case judgement about the merits
of integration versus separation?

> > - Better debuggability: sometimes a combination of a perf
> > change in combination with a kernel change causes a breakage. I
> > have bisected the shared tree a couple of times already, instead
> > of having to bisect a (100,000 commits x 10,000 commits) combined
> > space which much harder to debug …
>
> What you are describing happens when someone hasn't been careful
> about their kernel/userspace interfaces.

What i'm describing is what happens when there are complex bugs that
interact in unforeseen ways.

> If you have been rigorous with your interfaces, this isn't really
> an issue. When's the last time we've had to do a NxM exhaustive
> testing to find a broken sys call ABI between (for example) the
> kernel and MySQL?

MySQL relies on very little on complex kernel facilities.

perf on the other hand uses a very complex interface to the kernel
and extracts way more structured information from the kernel than
MySQL does.

That's where the whole "is a tool deeply related to the kernel or
not" judgement call starts mattering.

Also, i think we have a very clear example of split projects *NOT*
working very well when it comes to NxMxO testing matrix: the whole
graphics stack ...

You *really* need to acknowledge those very real complications and
uglies as well when you argue in favor of separation ...

> > - Code reuse: we can and do share source code between the kernel
> > and the tool where it makes sense. Both the tooling and the
> > kernel side code improves from this. (Often explicit
> > librarization makes little sense due to the additional
> > maintenance overhead of a split library project and the
> > impossibly long latency of how the kernel can rely on the ready
> > existence of such a newly created library project.)
>
> How much significant code really can get shared? [...]

It's relatively minor right now, but there's possibilities:

> [...] Memory allocation is different between kernel and userspace
> code, how you do I/O is different, error reporting conventions are
> generally different, etc. You might have some serialization and
> deserialization code which is in common, but (surprise!) that's
> generally part of your interface, which is hopefully relatively
> stable especially once the tool and the interface has matured.

The KVM tool would like to utilize lockdep for example, to cover
user-space locks as well. It already uses the semantics of the kernel
locking primitives:

disk/qcow.c: mutex_lock(&q->mutex);
disk/qcow.c: mutex_unlock(&q->mutex);
disk/qcow.c: mutex_unlock(&q->mutex);
disk/qcow.c: mutex_unlock(&q->mutex);
disk/qcow.c: mutex_unlock(&q->mutex);
disk/qcow.c: mutex_lock(&q->mutex);
disk/qcow.c: mutex_unlock(&q->mutex);
disk/qcow.c: mutex_unlock(&q->mutex);
disk/qcow.c: mutex_unlock(&q->mutex);
disk/qcow.c: mutex_unlock(&q->mutex);
disk/qcow.c: mutex_lock(&q->mutex);

... and lockdep would certainly make sense for such type of
"user-space that emulates hardware" while i don't think we'd ever
want to go to the overhead of outright librarizing lockdep in an
external way.

Thanks,

Ingo