Re: [GIT PULL] Squashfs pull request for 2.6.29

[Ingo Molnar <mingo@elte.hu>]

* Andrew Morton <akpm@linux-foundation.org> wrote:

> More importantly, the filesystem driver has to be able to read older 
> filesystem instances.  This is a userspace-visible binary interface! A 
> really complex one.
> 
> If for some reason we wish to change the on-disk format then that could 
> be done now.  But once the code is merged, such changes could only be 
> done in a back-compatible way.

IMHO what makes squashfs special is that:

 1) it's read-only: i.e. we dont actually _generate_ this data structure.
    It comes from the outside.

 2) it's a the "cat is already out of the bag" situation.
    It's in the field, it's used.

And as such it's pretty much an externality already to Linux: the 
filesystem is in use, distros patch it in, devices ship with it, etc. etc.

So the main technical question at this stage is whether we want to support 
that binary format _at all_, whether we want to support that kind of data 
protocol.

Also, i think your interpretation of 'ABI' stretches it quite far: we dont 
control the lowlevel bits at all - we dont generate the data, we just 
interpret it in the kernel in a read-only way. We dont provide the raw 
lowlevel bits to user-space either - we provide VFS bindings to it and 
that is generalized and not touched by squashfs.

As such squashfs has little to no classic ABI bindings - other than the 
trivial "do we support this data format" question. (which is not a classic 
ABI question.)

[ Often such read-only formats can even be iterated slightly incompatibly 
  without much fuss, as the userspace side goes together with the kernel
  anyway so it's easy to stay in touch. ]

And the thing is, i think we never before in the kernel said "no" to any 
support for a data format externality. [Let me qualify that: 'unpure' data 
protocols with active 'code' components were always an exception to that: 
ACPI, reiser4, etc. - but read-only access to a passive medium is far more 
clear-cut.]

This is a nice filesystem, and we support far, far worse data formats 
already. Far, far, far, far worse data formats, formats that are less used 
and were developed completely regardless of Linux. We support data formats 
and physical transports that should not even be mentioned with squashfs in 
the same paragraph.

And the thing is, if in the mainline kernel we were at the forefront of FS 
R&D, we would and could prototype such filesystems in the mainline kernel 
and we would have efficient and prompt influence over lowlevel format 
decisions.

We are not at the forefront - distros tend to apply new filesystem patches 
far earlier than we do and it takes forever to get stuff upstream - for 
better or worse. And we (the upstream community) had little active role in 
developing this thing _at all_.

( This is not a criticism it's just an observation of current reality: it 
  is _fundamentally hard_ to do active high-flux R&D for persistent 
  formats upstream and still be reasonably compatible at the same time.
  So developing out-of-tree - or via FUSE - might as well be the right 
  model here. )

Generally when a filesystem driver comes to us, its lowlevel format is 
pretty much a done deal already - it's out in the wild and we should say 
'no' only as an exception mechanism for clearly unacceptable crap.

Instead of trying to flex our muscle and steer the big red firetruck way 
after the fire has been put out already - by others.

Saying 'no' at this stage comes at a great and largely unnecessary cost to 
everyone involved. I believe we force ourselves into the R&D flow at an 
inappropriately late stage - while at the same time we are unreceptive to 
early adopter projects who'd like to avoid that. We cannot have the cake 
and eat it too.

At least IMHO.

( What could _perhaps_ change the picture a bit IMO is drivers/staging/ i 
  think - we could take a far more active role in certain types of 
  projects that have been done out of tree typically, with no formal
  promise for compatibility - or something like that. )

	Ingo