Iron files

[Gregory Maxwell <gmaxwell@gmail.com>]

So the post about file system failure modes made me think of something
interesting...

We'd discussed in the past that it would be interesting to store
cryptographic hashes of files as metadata for facilitating
applications which require hashes as well as data integrity.  Of
course, the challenge is making it perform well.. tree hashes make it
possible but still messy.

Another though on the subject, when we're using the compression plugin
it's quite likely that many blocks will be shrunk quite a bit on
write. We could at that time add a strong checksum (or cryptographic
hash)...  It could just be stored as though it were part of the
compressed data, the cost partly saved by the gains of compression. 
It would probably be useful to include the file identity and position
offset into the hash for each sub part of the file, so that if an
upper level data structure were corrupted in the FS that you'd never
end up with a part of another file silently sitting in the middle of
another file.

This would enable a policy where files could never be silently
corrupted.  Protection could be controlled on a file by file basis
just like compression and optionally operate in mode where check data
is only written but not tested (no substantial performance loss on
read, but a risk of returning corrupted data to the application).

Just another thought for the never ending list...