[Lustre-devel] changelog for whole filesystem?

[Eric Barton <eeb@whamcloud.com>]

Nathan wrote...

> This same "initial population of a database from objects" problem
> occurs when trying to replicate a Lustre filesystem using the
> changelog.

That must be a great test case.  If we can reliably reconstruct an
exact replica using a "from empty" changelog, we must have got it
right :)

> The problem is actually more complicated even for a single changelog
> consumer: since iterating through the virtual changelog takes non-0
> time, you're not sure if a virtual record was created before or
> after an actual changelog entry.  E.g. you might try to resolve the
> name of a file for a virtual record either before or after a rename,
> and then later you would see the rename in the changelog, leading to
> an inconsistent view of the namespace.
> 
> If you don't care about having the exact right name, then it's easy
> enough to ignore inapplicable changelog records.

I was trying to hint at the need to filter changes for individual
consumers depending on how far through the initial "from empty"
iteration they are in my original post.  Andreas stated it more
explicitly.  I think you just need to enumerate the cases - e.g. for
rename...

Iterated over yet?               Record
Source Inode   Target Inode      emitted
no             no                none
no             yes               delete target
yes            yes               rename source + delete target
yes            no                rename source

The final case just looks like a rename where the target name didn't
exist already.  In any case, the only real requirement on the stream
of changelog records constructed in the initial iteration is that
consistent filesystem state be reconstructed after the last record
is consumed.

-- 

                Cheers,
                        Eric

Eric Barton
CTO Whamcloud, Inc.
Tel: +44 117 330 1575
Mob: +44 7920 797 273