Re: silent semantic changes with reiser4

Spam
2004-08-29T16:53:29Z

> Spam <spam@tnonline.net> said:
>> Horst von Brand <vonbrand@inf.utfsm.cl> said:
>> > Hans Reiser <reiser@namesys.com> said:
>> >> Andrew Morton wrote:

> [...]

>> >> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem
>> >> for Apple,
>>
>> > Says who?
>>
>> At least I as a user see the benefits largely. The workstation /
>> desktop user has much more need for such features, than perhaps the
>> normal Linux sysadmin.

> Show benefits, show there are no downsides. Show that the same can't
> possibly be done in userland. Then I'll consider buying it.

I am not saying it couldn't be done in userland. Most things can.

The problem with userland is that it doesn't provide generic access
across most applications. For example GnomeVFS is limited to Gnome
applications only. The same for KDE. to be able to coordinate
application developer on anything is just almost impossible.

Still. Why do you oppose plugins, streams and meta files? The could
be valuable and easy to use tools for many purposes. One could
be would be advanced ACLs defined as a meta-file using XML format.

> AFAICS: Smallish benefits for some narrow application niche. Mayor problems
> (no decent multiuser way of using it, extremely complex stuff placed in the
> kernel). Userland solutions are all around us right now, and work just
> fine. No, placing the same gunk in the kernel won't make it magically use
> no time or memory, more like exactly the opposite. Besides, all the visions
> waved around here have very little in common (except the idea that they
> make all restrictions magically dissapear).

> Get a list of required features, show concrete ways of using it, outline
> possible implementations, discuss resource usage. I.e., do "engineering".
> Then we have something to discuss.

I am not sure what you refer to as "required features". Do you mean
example of usage and contents for streams, plugins and meta files?

>> >> and that means we need to put search engine and database
>> >> functionality into the filesystem.

>> > Please don't. Unix works and is extremely popular because it _doesn't_ try
>> > to be smart (policy vs mechanism distinction, simple abstractions that can
>> > be combined endlessly). If you add this to the filesystems, you either redo
>> > _all_ userland to understand _one_ particular way of doing "smart things"
>> > (which will turn out to be almost exactly the dumbest possible for some
>> > uses, and then you are stuck), or get lots of shards from broken apps (and
>> > users, and sysadmins).

>> Indeed. But you forget one thing. Combining lots of small userland
>> tools can actually make things more complex - at least for the user.

> So? Placing lots of little things in kernel does the same, plus screws over
> sysadmins, and makes kernel hacker's job harder. I prefer the former alone.

Difference being that it could be transparent in a way otherwise not
possible.

>> That said, I do not see that anyone has suggested to remove this way
>> of working with applications.

> Excuse me, if files suddenly become directories, and you can't have some
> "reserved" file/directory names, and... you are breaking lots of stuff.

>> Are the new and old features mutually exclusive?

> Yep.

>> >> It takes 11 years of serious
>> >> research to build a clean storage layer able to handle doing that.
>>
>> > Great! Build an experimental OS showing how to use it, and through that see
>> > if the ideas hold any water _in real, day-to-day, down-to-earth, practice_.

>> There are many uses already.

> Where?

>> It could be encryption, compression,
>> meta-data, file streams, sorting, searching and a multitude of uses
>> and ideas that haven't been discovered yet.

> "Could", "multitude ... haven't been discovered yet" is _not_ "many current
> uses".

>> >> Reiser4 has done that, finally. None of the other Linux filesystems
>> >> have.

>> > How come nobody before you in the almost 30 years of Unix has ever seen the
>> > need for this indispensable feature?

>> I, as a user, have for a long time. Do not drag everyone over the
>> same edge. Users have different needs. We should try to be as
>> forthcoming as possible to provide the tools and options for them.

> Why? Breaking the basic paradigms for some user whim is not good OS design
> in my book, sorry. Many "bright ideas" have been thrown out over the years
> for just the same reasons.

>> >> The next major release of ReiserFS is going to be bursting with
>> >> semantic enhancements, because the prerequisites for them are in place
>> >> now. None of the other Linux filesystems have those prerequisites.

>> > To me that would mean that ReiserFS has no place in Linux.

>> Why do you say this? I thought Linux embraced advancements and new
>> technologies.

> If they are right. I think breaking the base of the OS is not a good
> idea. Perhaps ReiserFS has a place elsewhere (in MacOS, or Windows, where
> this is supposedly so so badly needed?), but not here.

You mean in Windows and MacOS where much of this functionality
already exists?

>> >> They won't be able to keep up with the semantic enhancements. This
>> >> metafiles and file-directories stuff is actually fairly trivial stuff.

>> > Maybe. But the question of it being of any use aren't trivially answered
>> > "yes". My gut feeling is that the answer is a resounding "no", but that's
>> > just me. Direcories are directories, if you want to ship directory-like
>> > stuff around, use directories (or tarfiles, or whatever). Just look what
>> > happened to the much, much easier stuff of splitting SUID/SGID into
>> > capabilities: Nothing at all whatsoever, because they have no decent place
>> > to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be
>> > able to _define_ APIs to follow, but even so it isn't at all easy to do so.

>> Exactly. The problem I see in this discussion is the lack of just
>> this - APIs. Because the lack of good APIs we cannot make any larger
>> changes or bring new features because old applications will break.

I'll chop this up...

> No. There is a complete lack of answers to why it should be done ("I think
> it would be nice", "MSFT is trying to do it", "Look at MacOS" is far from
> enough)

No perhaps not by itself. But it certainly should be considered.
These two OS'es have most of the market share. They have great
resources trying to make things that work for their users.

Users are more important than you think. Linux and any OS can do
what it wants, but if it wants to keep users then it has to listen
to them.

> , on what the alternatives are ("Sorry, it must be in the kernel
> because it is impossible to get application writers to agree on how to do
> it"

This is a perfectly valid reason. If it is necessary to move
something into the core of Linux because it couldn't otherwise be
done then it should seriously be considered.

> will just add another incompatible way of "doing almost the same", this
> time in the kernel),

People on this list are actually trying to make it to be fully
compatible with old legacy applications.

If you hadn't noticed, Reiser4 does work. The meta files do work.
The discussion is wether parts of Reiser4 should be made generic
and fit into the VFS.

> and last but not least what the impact is (performance
> wise, application breakage, user/sysadmin expectation breakage,
> incompatibility with other systems, kernel development impact, to list a
> few). Hash this out, and we can talk.

You paint up a doomsday picture here. Nothing of this magnitude has
happened and won't. besides, what is wrong if sysadmins would have
to learn something new?

>> I think that Linux should make new API that will completely separate
>> applications from the storage media. This way we will not as easily
>> break applications when new features are developed. This should be
>> the main priority.

> The "files + directories" stuff is a nice abstraction, and works today on
> all random-access storage media. You don't care much if it is on a CD, a
> floppy, a disk, a RAID array, or a flash card, do you? It won't work on
> tapes, or printers; but that is to be expected.

you didn't actually comment on what I said. I mentioned API's
nothing about directories and files. API being the keyword here.

If there are good API's that applications can use, for example to
access filesystems, then it will be easier and much more possible to
introduce new things into the filesystem without breaking existing
applications. One example of preventing application breakage could
be to have a versioning system. Applications could query which
version is supported, or vice versa.

From what I understand it is much harder to do anything today
because of the fixed ways. This is why we have this discussion.

> [...]

>> Application breakage has happened before between virtually every
>> kernel change. It isn't new. Just take devfs and sysfs.

> Oops, now emacs stopped working because devfs was added? gcc needed a mayor
> overhaul because of /sys vs /proc? Gimme a break. That was _basic system tool_
> breakage, not all-around application breakage.

Emacs seem to work fine with the file-as-dir on my Reiser4
partition. And so does actually any other application I have used on
it. Be it in Gnome, KDE or in plain shell's.

>> >> Look guys, in 1993 I anticipated the battle would be here, and I build
>> >> the foundation for a defensive tower right at the spot MS and Apple are
>> >> now maneuvering towards.
>>
>> > Why place a protective tower around the pit into which they are trying
>> > desperately to throw themselves? ;-)
>>
>> It is easy to say this. But the fact is that users can actually use
>> these new features for good things.

> And can abuse them for bad things, and the features aren't free. Balance
> benefits (very minor) against costs (quite large).

Abuse depends on permissions and security policies which is your job
as admin to look after.

It would be possible to limit, for example), the types of plugins
that a user can either use or load.

Do you give your users access to modprobe or tools like that?

>> >> Help me get the next level on the tower before
>> >> they get here. It is one hell of a foundation, they won't be able to
>> >> shake it, their trees are not as powerful. Don't move reiser4 into vfs,
>> >> use reiser4 as the vfs. Don't write filesystems, write file plugins and
>> >> disk format plugins and all the other kinds of plugins, and you won't be
>> >> missing any expressive power that you really want....
>>
>> > Better write libraries handling whatever weird format you have in mind for
>> > each use. Even applications. I don't expect to be able to use emacs to edit
>> > a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit
>> > a poem. The current trend (which I heartily agree with) is to take stuff
>> > _out_ of the kernel (for flexibility, access, development ease, even
>> > performance).

>> If I want to be able to quickly find files by a certain author I
>> could search the meta-data fields (if they are used) with normal
>> tools such as grep.

> Iff the metadata is there. It won't (everybody would have to agree on
> placing that on the files for it to work), so it is useless.

No it is not useless. It just means the file has no meta
information. Just as it is today. No change.

>> Compression or encryption is another example.
>>
>> Would you rather patch every application there is for Linux to
>> enable support for reading these files and directories?

> No. But an encrypted or compressed filessytem doesn't need weird files.
> Easiest on applications is just to serve the decrypted/uncompressed files
> for use via the traditional APIs. Next easiest is provide programs that
> handle cryptography and compression and can be combined with whatever
> unaware program you care to use. Funny, that is exactly the way things work
> now...

So you can setup a directory that has some files compressed and some
other encrypted and yet they will be available to use with tar, cp,
ls, mc, Nautilus, Emacs etc etc?