Re: Beagle and logging inotify events

["Jon Smirl" <jonsmirl@gmail.com>]

On 11/15/07, Jan Kara <jack@suse.cz> wrote:
> On Wed 14-11-07 14:38:05, J. Bruce Fields wrote:
> > On Wed, Nov 14, 2007 at 12:32:45PM -0700, Andreas Dilger wrote:
> > > On Nov 14, 2007  11:32 -0500, Chuck Lever wrote:
> > > > I disagree: we don't need a "bullet-proof" log.  We can get a significant
> > > > performance improvement even with a permanent dnotify log implemented in
> > > > user-space.  We already have well-defined fallback behavior if such a log
> > > > is missing or incomplete.
> > > >
> > > > The problem with a permanent inotify log is that it can become unmanageably
> > > > enormous, and a performance problem to boot.  Recording at that level of
> > > > detail makes it more likely that the logger won't be able to keep up with
> > > > file system activity.
> > > >
> > > > A lightweight solution gets us most of the way there, is simple to
> > > > implement, and doesn't introduce many new issues.  As long as it can tell
> > > > us precisely where the holes are, it shouldn't be a problem.
> > >
> > > Jan Kara is working on a patch for ext4 which would store a recursive
> > > timestamp for each directory that gives the latest time that a file in
> > > that directory was modified.  ZFS has a similar mechanism by virtue of
> > > doing full-tree updates during COW of all the metadata blocks and storing
> > > the most recent transaction number in each block.  I suspect btrfs could
> > > do the same thing easily.
> > >
> > > That would allow recursive-descent filesystem traversal to be much more
> > > efficient because whole chunks of the filesystem tree can be ignored during
> > > scans.
> >
> > The problem is that people may not be happy with the random behavior of
> > hardlinks, right?
>   The kernel part has this non-determinism with hardlinks but it can be
> worked-around in userspace (and actually if you watch the whole filesystem
> you don't care about the non-determinism at all because you are guaranteed
> there is *at least one* path which indicates the file was modified). I'm
> planning to write a userspace library which would mostly hide the
> problems with hardlinks (and also problems with the fact that scanner may
> not have enough rights to set inode flags pointed out by Ted) from
> applications...
>
>                                                                 Honza
>
> BTW: Where did this discussion started? Googling the subject gives me just
> one news message...

Linus made comments about it on the git list which restarted it.
Beagle's writing to the xattrs makes git think the file has been
changed. It was really a fs question not git so I forwarded it to this
group. Git wouldn't have a problem if there was a way for Beagle to
avoid using the xattrs.

Discussion of the general problem have been going on for a long time
and has caused the creation of dnofity and inotify. But the problem is
still not completely solved since there are these windows where files
can change and the tracker (beagle or other apps) is not aware of the
change.

My thoughts are that the kernel implementation of inotify should be
extended to support checkpoints and replay of events after the
checkpoint. How the file systems or kernel implement this is a harder
problem.

I do know for sure that many people are greatly annoyed by Beagle
beating away on their disk after each boot.


> --
> Jan Kara <jack@suse.cz>
> SUSE Labs, CR
>


-- 
Jon Smirl
jonsmirl@gmail.com