Re: Watchman/inotify support and other ways to speed up git status

[Christian Couder <christian.couder@gmail.com>]

On Wed, Oct 28, 2015 at 12:54 AM, David Turner <dturner@twopensource.com> wrote:
>
> On Thu, 2015-10-22 at 07:59 +0200, Christian Couder wrote:
>> Hi everyone,
>>
>> I am starting to investigate ways to speed up git status and other git
>> commands for Booking.com (thanks to AEvar) and I'd be happy to discuss
>> the current status or be pointed to relevant documentation or mailing
>> list threads.
>>
>> From the threads below ([0], [1], [2], [3], [4], [5], [6], [7], [8]) I
>> understand that the status is roughly the following:
>>
>> - instead of working on inotify support it's better to work on using a
>> cross platform tool like Watchman
>>
>> - instead of working on Watchman support it is better to work first on
>> caching information in the index
>>
>> - git update-index --untracked-cache has been developed by Duy and
>> others and merged to master in May 2015 to cache untracked status in
>> the index; it is still considered experimental
>>
>> - git index-helper has been worked on by Duy but its status is not
>> clear (at least to me)
>>
>> Is that correct?
>> What are the possible/planned next steps in this area? improving
>
> We're using Watchman at Twitter.  A week or two ago posted a dump of our
> code to github, but I would advise waiting a day or two to use it, as
> I'm about to pull a large number of bugfixes into it (I'll update this
> thread and provide a link once I do so).

Great, I will have a look at it then!

> It's good, but it's not great.  One major problem is a bug on OS X[1]
> that causes missed updates.  Another is that wide changes end up being
> quite inefficient when querying watchman.  This means that we do some
> hackery to manually update the fs_cache during various large git
> operations.
>
> I agree that in general it would be better to store or all some of this
> information in the index, and the untracked-cache is a good step on
> that. But with it enabled and watchman disabled, there still appears to
> be 1 lstat per file (plus one stat per dir).  The stats per-directory
> alone are a large issue for Twitter because we have a relatively deep
> and bushy directory structure (an average dir has about 3 or 4 entries
> in it).  As a result, git status with watchman is almost twice as fast
> as with the untracked cache (on my particular machine).

Thanks for this detailled description.

> [1] https://github.com/facebook/watchman/issues/172