Re: Issues with using fanotify for a filesystem indexer

[Alexander Larsson <alexl@redhat.com>]

On Thu, 2009-04-02 at 16:54 +0200, Jan Kara wrote:

>   Some time ago I was trying to solve a similar problem and I've come up
> with a solution which I've called recursive mtime. The general idea is
> that with each directory, kernel additionally keeps a flag and a
> timestamp. When a directory is modified, we do:
>   dir = changed dir;
>   while dir has flag set do
>     update timestamp to current time
>     clear flag
>     dir = parent dir
> 
>   When a file is modified, you just start with a parent directory of
> that file. With this scheme, you are able to find reasonably quickly
> (without looking at unchanged directories) what has changed since
> you've looked last time (you look for timestamps newer than the time
> when you started last scan and you set flags as you go). Also the scheme
> is quite cheap to maintain and has no problems with overflowing event
> queues etc. (observe that the scheme works perfectly fine for several
> independent scanners in parallel). As a bonus, if you store the flag +
> timestamp persistently on disk, you can use this scheme to speedup things
> like rsync.
>   What gets nasty (but solvable) are hardlinks and bind mounts. I was
> writing a library to handle these last summer but then had to work on
> something else and didn't get back to it yet.

Another potential issue with this is that every change bubbles up to the
top, modifying the recursive mtime of that. This will become very
contented, and may imply a partial serialization of fs activity, which
is kinda costly.