Data redundancy in tabled

Data redundancy in tabled

[Pete Zaitcev]

I have considered how I want to implement the first cut of data redundancy
in tabled, and came with the following plan:

 - Checksum for data is to be stored in both the db4 entry and
   the metadata in chunk.

 - tabled gets a new "thread" (probably not really a thread)
   that walks keys in db4 by record number, then asks all listed
   chunkservers to verify if the OID is present (actually asks
   to fetch the metadata). If not present, it updates the db4.
   If the reported checksum mismatches the one in db4, it tells
   chunkserver to drop that object (optionally).

   If in the end the object's redundancy is insufficient,
   replication is scheduled.

 - chunkservers run checksums by themselves, without a command
   from tabled, verify against checksums in the metadata that
   they have. If object fails, it's removed (or maked dead,
   I haven't decided yet).

 - a separate process running at tabled processes scheduled
   replications. The usual applies: control bandwidth and
   prevent hogging up the network and servers, batch up OIDs
   between same chunkservers, etc.

This all looks nicely parallel except for the db4 scanner,
thanks to decoupling of checksumming from tabled's affairs.
I decided not to do anything about the db4 at this stage, because
any improvements need a new database. I have a vague plan for
that as well, but it must wait. The above is what will make
tabled useable for the general public.

-- Pete