Re: silent semantic changes with reiser4

[Jamie Lokier <jamie@shareable.org>]

Linus Torvalds wrote:
> Well, you _could_ make it do a "tar", but what's the point? That really 
> doesn't add any value as far as user space is concerned.
> 
> Think of it this way: if the directory and the "default stream" associated 
> with it aren't independent, then user space already has the information, 
> and the stream is pointless. In fact, the stream is _worse_ than 
> pointless, because it hides the fact that there is no independent 
> information.
> 
> See my point? 

I see your point, but I agree 50%.

(1) One of the uses of "streams" is to have extra information which
    accompanies a file, and is separate from the flat file's information.

    For example, metadata like "this C source file is in UTF-8",
    "the author is Jamie (nowhere to put it in the data file)" and
    "Jamie's signature".

(2) Filesystem settings like "compress this", and perhaps "permissions".

(3) Filesystem information like "this file has been compressed by 59%,
    uses 5 1k blocks".

(4) Another use, which is more akin to containers, is to _present_ the
    information of the flat file in a different way.

    For example, the id tags and comments of any media file, the component
    parts of any media file, license (of course :), and expanded or
    decoded views such as inside an archive file, filesystem image, or
    other structured storage file (.doc :).

    Those alternate views are useful just because they can be accessed
    using ordinary unix tools, no matter what the obscure codec format
    provided it's supported.  With a bit of effort a standard
    presentation for certain kinds of data may arise.

(5) There is also data which is _calculated_ from the flat file or its
    extra metadata, and which the kernel guarantees is either
    consistent with the file, not set at all, or at least is
    invalidated if the file is changed.  It may be calculated on demand
    through the kernel interface, or applications may simply have to
    calculate it themselves when they don't have an up to date value.

    For example MD5 or SHA1 digests of the flat file, or of its parts.
    The key thing is that these are always up to date, not dependent
    on weak tests like mtime.

    Partial checksums for speeding up rsync and other delta algorithms.
    Computed thumbnails, MP3 average power levels, character encoding
    extracted from parsing old HTML META tags.  Anything really that
    can be calculated and usefully cached.

    This is similar to (4), but I'm emphasising things that
    may take a while to compute, may be cached with the filesystem
    itself helping to decide when to evict, and writing them won't
    affect the data they're calculated from, if writing is allowed at all.

    The only really important of these are MD5 or SHA1 digests, or
    some other way of tracking modifications like a strong serial number.

    All other computed cached data can be maintained by applications
    elsewhere, if they know when the data has been modified since the
    last time.  (Though there are advantages to making the data
    disappear if the file is deleted, or move if it's renamed).


The thing with these different categories is that we want different
behaviour when serialising the data for a backup or to copy it.

We probably always (1) to be copied.

(2) is questionable: it's like unix file permissions, you might want
to copy it; you might not.

(3) Shouldn't be copied or stored except for special reasons.  It's a
bit like /proc: it's not meant to be stored on your backup CD and it's
definitely not meant to be written during a restore. :)

(4) Should never be copied if you're copying the flat file, because
it's exactly the same information, just accessed differently.  You
might want to copy it by itself for some reason though, and you
certainly would like to work with it from scripts and applications.
These streams are special in another way: they might themselves
decompose into further streams.  Just think of a AVI file containing
an OGG file containing a FLAC containing metadata and data streams, or
something like that.  They happen.

(5) shouldn't be copied if you're transmitting a file, as a matter of
data integrity among other things.  You might want it on the backup,
then again maybe only some of the values.  Probably never the crypto digests.


To summarise: most data inside a directory-as-file should _not_
usually be copied and transported.  A little of it, (1), should be
copied if the serialised form will have it, and some of the other
metadata should be copied under special circumstances.

I've suggested that we could simply not have (1), and recommend that
only reproducible, non-essential data appears inside the directory.
Everything essential would be part of the serialised, flat file.

However, you seem to like (1), metadata which should be serialised and
copied around, and which is not part of the flat file.

Note that (1) will only be copied around on systems which handle
"extra forks" anyway -- things like Samba and MacOS compatibility.
FTP and HTTP obviously won't copy the metadata of (1), although HTTP
WebDAV might allow it to be visible as properties.

All that suggests a common structure for the directory inside a
"container" -- one where the metadata to be copied by default is
clearly distinguished from the metadata which should never by copied,
or which is cached, or which is a view.  Possibly it should be
distinguished by name, although permission bits or (not sure I like
this) meta-metadata could serve the same role without being name
dependent.

-- Jamie