Re: [PATCH] read-cache: call verify_hdr() in a background thread

[Jeff Hostetler <git@jeffhostetler.com>]

On 3/24/2017 11:36 AM, Jeff King wrote:
> On Fri, Mar 24, 2017 at 01:27:51PM +0000, git@jeffhostetler.com wrote:
>
>> From: Jeff Hostetler <jeffhost@microsoft.com>
>>
>> Teash do_read_index() in read-cache.c to call verify_hdr()
>> in a background thread while the forground thread parses
>> the index and builds the_index.
>>
>> This is a performance optimization to reduce the overall
>> time required to get the index into memory.
>>
>> Testing on Windows (using the OpenSSL SHA1 routine) showed
>> that parsing the index and computing the SHA1 take almost
>> equal time, so this patch effectively reduces the startup
>> time by 1/2.
>
> Have you considered just skipping the sha1 check on read (possibly with
> an option)?
>
> Its purpose is to detect disk bit-rot. Checking it for every single
> operation may be a bit excessive, especially because it gets rewritten a
> lot. Nobody really cared until now because they don't have index files
> that are hundreds of megabytes.

Yes, we have done that in a version of the client customized
for the virtual file system effort and have been using it
internally without incident for quite a while.

And we can get a patch for that instead (or include it with
this one) if there's interest.

I tried to limit my proposal here to increasing performance while
preserving the existing behavior.

Jeff