Re: Watchman/inotify support and other ways to speed up git status

[David Turner <dturner@twopensource.com>]

On Thu, 2015-10-22 at 07:59 +0200, Christian Couder wrote:
> Hi everyone,
> 
> I am starting to investigate ways to speed up git status and other git
> commands for Booking.com (thanks to AEvar) and I'd be happy to discuss
> the current status or be pointed to relevant documentation or mailing
> list threads.
> 
> From the threads below ([0], [1], [2], [3], [4], [5], [6], [7], [8]) I
> understand that the status is roughly the following:
> 
> - instead of working on inotify support it's better to work on using a
> cross platform tool like Watchman
> 
> - instead of working on Watchman support it is better to work first on
> caching information in the index
> 
> - git update-index --untracked-cache has been developed by Duy and
> others and merged to master in May 2015 to cache untracked status in
> the index; it is still considered experimental
> 
> - git index-helper has been worked on by Duy but its status is not
> clear (at least to me)
> 
> Is that correct?
> What are the possible/planned next steps in this area? improving

We're using Watchman at Twitter.  A week or two ago posted a dump of our
code to github, but I would advise waiting a day or two to use it, as
I'm about to pull a large number of bugfixes into it (I'll update this
thread and provide a link once I do so).  

It's good, but it's not great.  One major problem is a bug on OS X[1]
that causes missed updates.  Another is that wide changes end up being
quite inefficient when querying watchman.  This means that we do some
hackery to manually update the fs_cache during various large git
operations.

I agree that in general it would be better to store or all some of this
information in the index, and the untracked-cache is a good step on
that. But with it enabled and watchman disabled, there still appears to
be 1 lstat per file (plus one stat per dir).  The stats per-directory
alone are a large issue for Twitter because we have a relatively deep
and bushy directory structure (an average dir has about 3 or 4 entries
in it).  As a result, git status with watchman is almost twice as fast
as with the untracked cache (on my particular machine).


[1] https://github.com/facebook/watchman/issues/172