Re: silent semantic changes with reiser4

[Jamie Lokier <jamie@shareable.org>]

Andrew Morton wrote:
> No.  All of the applications which you initially identified can be
> implemented by putting the various bits of data into a single file and
> getting applications to agree on the format of that file.
> 
> For example, some image file formats already support embedded metadata, do
> they not?

Same for MP3s and their metadata tags.  _That_ kind of data should be
part of the contents of the file: it is intended to be transported.

It is useful to auto-extract it from the file's data stream, using
format-specific programs, and offer that metadata in separate files
where it is easy to use e.g. by shell scripts and indexing programs,
which can then work independently of file formats.

That kind of metadata should be _derived_ from the file's data stream,
and ideally the extraction should be automatic, with userspace
assistance.  It may be useful for the filesystem to record what's
extracted, for fast access and indexing.

The same applies to thumbnails.  The only difference is that these may
take longer to compute.  But they're the same thing: something
calculated from the file's data stream, which may (or may not) be
stored in the filesystem so it doesn't have to be recomputed each
time.  It might even be worth expiring the metadata from disk if it
hasn't been used for a while and can be recomputed.

These are the same as looking into archives or structured files.  The
only difference is that the unpacked views might not be stored on the
filesystem (but they might be, if the filesystem knows how to cache
views for faster future access).[*]

Those are all examples of data derived from the data stream of a flat
file.  They are useful applications, and all work fine with FTP, cp,
cat <a >b, and email attachments, _no application changes required_.

There's another kind of metadata which isn't intended for transport.
That would be things like the classic attributes: permissions,
timestamps.  It would also include things like author information when
the file format cannot carry it.

-- Jamie


[*] - A fancier variant, which is hard without good fs support, it
writing to structured files through a view.  I.e. modifying files
inside a .tar.gz, or (this is useful) inside an
{MS,Open,Gnome,K}Office container file or *SQL database table.  To do
this efficiently, the file's main data stream is regenerated on demand
from the view's data.  As I said, it requires good, special fs support
to do it efficiently (fancy caching).  With appropriate userspace
libraries you can make apps work with container files on all other
OSes (just like .tar.gz and *Office containers at the moment) but are
significantly faster to work with on Linux (no need to serialise and
(de-)compress except on demand) and more visible to the standard unix
scripting tools.