Re: [PATCH v3 03/16] index-helper: new daemon for caching index and related stuff

[David Turner <dturner@twopensource.com>]

On Thu, 2016-04-14 at 11:12 -0700, Junio C Hamano wrote:
> Duy Nguyen <pclouds@gmail.com> writes:
> 
> > On Wed, Apr 13, 2016 at 1:05 AM, Junio C Hamano <gitster@pobox.com>
> > wrote:
> > > > >  - access is slow (unless cached, but we can't be sure)
> > > > 
> > > > We could solve this (and the other problem) with mlock.
> > > 
> > > Probably you meant madvise(2)?
> > > 
> > > For something of a size comparable to the index file held by
> > > index-helper-daemon in-core, I'd expect we wouldn't page too
> > > badly.
> > 
> > I had a look at linux implementation of madvise(MADV_WILLNEED). All
> > it
> > does is force populating the entire memory region, which is good.
> > But
> > I suspect when memory is under pressure, some pages may be
> > reclaimed.
> 
> I share that suspicion.  Why is such a reclamation bad thing, though?
> 
> > index files in monster repo case can go up to a few hundred
> > megabytes,
> > chances of being reclaimed rise accordingly. But we can reconsider
> > mlock() later when/if real problems happen.
> 
> Holding onto "a few hundred megabytes" just so that occasional Git
> operations will not stall with the daemon and slowing down the
> overall work of the user by panalizing other elements in the user's
> workflow does not sound like a good trade-off to me.  Wouldn't the
> user better off by not using the daemon at that point, which would
> give the few hundred megabytes back to the system for better uses?

By running git index-helper, the user has signaled their intent about
memory usage.  It's probably reasonable to honor this intent (even if
the user might be mistaken about what the right trade-off is).