inotify support, nearly there

inotify support, nearly there

[Duy Nguyen]

Just a quick update for the enthusiasts. My branch file-watcher [1]
has got working per-user inotify support. It's a 20 patch series so
I'll refrain from spamming git@vger for a while, even though it hurts
your eyes a lot less than what I have posted so far. The test suite
ran fine with it so it's not that buggy. It has new tests too, even
though real inotify is not tested in the new tests. Documentation is
there, either in .txt or comments. Using it is simple:

$ mkdir ~/.watcher
$ git file-watcher --detach ~/.watcher
$ git config --global filewatcher.path $HOME/.watcher

There's still some polishing work to do. But I think the core logic is
done. I have some ideas what to be polished, but I'd appreciate
feedback if anyone uses it. We may need to make lookup code faster
later.

MacOS, FreeBSD and Windows contributors. If you have time and are
interested, have a look at the protocol, which is basically documented
in file-watcher.c:handle_command(), and see if something is
incompatible with your file notification mechanism. MacOS and FreeBSD
may reuse code from file-watcher.c, at least the unix socket part. I'm
not so sure about Windows. It probably needs a brand new daemon
because little could be shared, I don't know. I deliberately design
the daemon dumb so writing a completely new one won't be so hard. My
plan is focus on inotify and get it merged first, then new OS support
can come later (with refactoring if needed, but should not change the
protocol drastically).

[1] git clone https://github.com/pclouds/git.git file-watcher
--
Duy

Re: inotify support, nearly there

[Mike Hommey]

On Wed, Jan 29, 2014 at 01:47:30PM +0700, Duy Nguyen wrote:
> Just a quick update for the enthusiasts. My branch file-watcher [1]
> has got working per-user inotify support. It's a 20 patch series so
> I'll refrain from spamming git@vger for a while, even though it hurts
> your eyes a lot less than what I have posted so far. The test suite
> ran fine with it so it's not that buggy. It has new tests too, even
> though real inotify is not tested in the new tests. Documentation is
> there, either in .txt or comments. Using it is simple:
> 
> $ mkdir ~/.watcher
> $ git file-watcher --detach ~/.watcher
> $ git config --global filewatcher.path $HOME/.watcher
> 
> There's still some polishing work to do. But I think the core logic is
> done. I have some ideas what to be polished, but I'd appreciate
> feedback if anyone uses it. We may need to make lookup code faster
> later.
> 
> MacOS, FreeBSD and Windows contributors. If you have time and are
> interested, have a look at the protocol, which is basically documented
> in file-watcher.c:handle_command(), and see if something is
> incompatible with your file notification mechanism. MacOS and FreeBSD
> may reuse code from file-watcher.c, at least the unix socket part. I'm
> not so sure about Windows. It probably needs a brand new daemon
> because little could be shared, I don't know. I deliberately design
> the daemon dumb so writing a completely new one won't be so hard. My
> plan is focus on inotify and get it merged first, then new OS support
> can come later (with refactoring if needed, but should not change the
> protocol drastically).
> 
> [1] git clone https://github.com/pclouds/git.git file-watcher

Haven't looked at the code, so I don't know if you've done that, but in
case you haven't, it would be nice to have an environment variable or a
config option to make git use the file-watcher *and* normal lstat
operations, to check consistency.

Mike

Re: inotify support, nearly there

[Duy Nguyen]

On Wed, Jan 29, 2014 at 2:44 PM, Mike Hommey <mh@glandium.org> wrote:
> Haven't looked at the code, so I don't know if you've done that, but in
> case you haven't, it would be nice to have an environment variable or a
> config option to make git use the file-watcher *and* normal lstat
> operations, to check consistency.

No I haven't. So far I depend on my manual tests and the test suite to
check for consistency. Will do.
-- 
Duy

Re: inotify support, nearly there

[Juan P]

Duy Nguyen <pclouds <at> gmail.com> writes:

> 
> Just a quick update for the enthusiasts. My branch file-watcher [1]
> has got working per-user inotify support. It's a 20 patch series so
> I'll refrain from spamming git <at> vger for a while, even though it hurts
> your eyes a lot less than what I have posted so far. The test suite
> ran fine with it so it's not that buggy. It has new tests too, even
> though real inotify is not tested in the new tests. Documentation is
> there, either in .txt or comments. Using it is simple:
> 
> $ mkdir ~/.watcher
> $ git file-watcher --detach ~/.watcher
> $ git config --global filewatcher.path $HOME/.watcher
> 

Will this mean that Git would work faster with repositories with large 
number of files or commits? I am new into this topic, but I am trying to 
understand, any pointers are appreciated.

Re: inotify support, nearly there

[Ronnie Sahlberg]

On Sun, Jul 20, 2014 at 10:10 PM, Juan P <juandavid1707@gmail.com> wrote:
> Duy Nguyen <pclouds <at> gmail.com> writes:
>
>>
>> Just a quick update for the enthusiasts. My branch file-watcher [1]
>> has got working per-user inotify support. It's a 20 patch series so
>> I'll refrain from spamming git <at> vger for a while, even though it hurts
>> your eyes a lot less than what I have posted so far. The test suite
>> ran fine with it so it's not that buggy. It has new tests too, even
>> though real inotify is not tested in the new tests. Documentation is
>> there, either in .txt or comments. Using it is simple:
>>
>> $ mkdir ~/.watcher
>> $ git file-watcher --detach ~/.watcher
>> $ git config --global filewatcher.path $HOME/.watcher
>>
>
> Will this mean that Git would work faster with repositories with large
> number of files or commits? I am new into this topic, but I am trying to
> understand, any pointers are appreciated.
>

That is the idea. Instead of having to recursively lstat() every file
to find which ones have changed this will allow to query a database
for "tell me all files that match this criteria xyz" which could speed
up many things.

Duy, keep up the good work.
Once you are ready I am more than happy to help out reviewing the patches.

regards
ronnie sahlberg

Re: inotify support, nearly there

[Duy Nguyen]

On Tue, Jul 22, 2014 at 11:39 PM, Ronnie Sahlberg <sahlberg@google.com> wrote:
> Duy, keep up the good work.
> Once you are ready I am more than happy to help out reviewing the patches.

The road I'm taking is:

1) split-index to speed up index writing time
2) add index-helper daemon to speed up index reading time
3) speed up .gitignore processing by caching the results and validate
using directory stat info
4) make sure 3) works (e.g. index extension data..) with another
source than directory stat info, like inotify
5) finally put inotify in place to kill lstat in index refresh and .gitignore

1) is entering next (or master) now. I need to submit 2) again and get
Windows guys' opinion (up to 3 it should work on Windows). As you can
see it'll like take a lot more months to get to 5). I feel a bit
guilty of "blocking" David Turner's inotify support series, which does
not require so many changes like this (but only benefits *nix),
because I _think_ this road is better. If you have time, maybe you can
have a look at his series and push it forward?
-- 
Duy