[PATCH] read-cache: increase write buffer size

[PATCH] read-cache: increase write buffer size

[David Turner]

Each write() has syscall overhead, and writing a large index entails
many such calls.  A larger write buffer reduces the overhead,
leading to increased performance.

On my repo, which has an index size of 30m, this saves about 10ms of
time writing the index.

Signed-off-by: David Turner <dturner@twopensource.com>
---
 read-cache.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/read-cache.c b/read-cache.c
index d9fb78b..09ebe08 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -1714,7 +1714,7 @@ int unmerged_index(const struct index_state *istate)
 	return 0;
 }
 
-#define WRITE_BUFFER_SIZE 8192
+#define WRITE_BUFFER_SIZE 131072
 static unsigned char write_buffer[WRITE_BUFFER_SIZE];
 static unsigned long write_buffer_len;
 
-- 
2.4.2.767.g62658d5-twtrsrc

Re: [PATCH] read-cache: increase write buffer size

[Torsten Bögershausen]

On 2016-03-19 02.19, David Turner wrote:
> Each write() has syscall overhead, and writing a large index entails
> many such calls.  A larger write buffer reduces the overhead,
> leading to increased performance.
> 
> On my repo, which has an index size of 30m, this saves about 10ms of
> time writing the index.
> 
> Signed-off-by: David Turner <dturner@twopensource.com>
> ---
>  read-cache.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/read-cache.c b/read-cache.c
> index d9fb78b..09ebe08 100644
> --- a/read-cache.c
> +++ b/read-cache.c
> @@ -1714,7 +1714,7 @@ int unmerged_index(const struct index_state *istate)
>  	return 0;
>  }
>  
> -#define WRITE_BUFFER_SIZE 8192
> +#define WRITE_BUFFER_SIZE 131072
>  static unsigned char write_buffer[WRITE_BUFFER_SIZE];
>  static unsigned long write_buffer_len;
>  
> 
Do I read that right, it saves 10 milliseconds ?
What happens to small system (like Raspberry PI), when you
want 128K write buffer ?
Could the buffer size be turned into a makefile variable, defaulting to 8192 ?

Re: [PATCH] read-cache: increase write buffer size

[Pranit Bauva]

On Sat, Mar 19, 2016 at 1:35 PM, Torsten Bögershausen <tboegi@web.de> wrote:
> On 2016-03-19 02.19, David Turner wrote:
>> Each write() has syscall overhead, and writing a large index entails
>> many such calls.  A larger write buffer reduces the overhead,
>> leading to increased performance.
>>
>> On my repo, which has an index size of 30m, this saves about 10ms of
>> time writing the index.
>>
>> Signed-off-by: David Turner <dturner@twopensource.com>
>> ---
>>  read-cache.c | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/read-cache.c b/read-cache.c
>> index d9fb78b..09ebe08 100644
>> --- a/read-cache.c
>> +++ b/read-cache.c
>> @@ -1714,7 +1714,7 @@ int unmerged_index(const struct index_state *istate)
>>       return 0;
>>  }
>>
>> -#define WRITE_BUFFER_SIZE 8192
>> +#define WRITE_BUFFER_SIZE 131072
>>  static unsigned char write_buffer[WRITE_BUFFER_SIZE];
>>  static unsigned long write_buffer_len;
>>
>>
> Do I read that right, it saves 10 milliseconds ?
> What happens to small system (like Raspberry PI), when you
> want 128K write buffer ?
> Could the buffer size be turned into a makefile variable, defaulting to 8192 ?

Maybe for all ARM devices not just Raspberry Pi.

There is one way, though its not that good. This is just a sample
code. I am unsure about if using bash in makefile is a good idea

in Makefile,
uname -a | grep "arm" >out
if [ -s out ]; then
    export ARM=arm
fi
rm out

in read-cache.c,
#define ARCH (getenv("ARCH"))
#ifdef ARCH,
    #define WRITE_BUFFER_SIZE 128000
#endif

Re: [PATCH] read-cache: increase write buffer size

[Duy Nguyen]

On Sat, Mar 19, 2016 at 8:19 AM, David Turner <dturner@twopensource.com> wrote:
> Each write() has syscall overhead, and writing a large index entails
> many such calls.  A larger write buffer reduces the overhead,
> leading to increased performance.
>
> On my repo, which has an index size of 30m, this saves about 10ms of
> time writing the index.

I wonder if split-index does not work as intended. Because if it does,
you should rarely need to write that such big index files (or is this
30mb the small part and the base index is even bigger?).

But I agree with Torsten that we should make this configurable,
preferably via config file, if not we can still move this define back
in Makefile, overridable using config.mak
-- 
Duy

Re: [PATCH] read-cache: increase write buffer size

[David Turner]

On Sat, 2016-03-19 at 18:18 +0700, Duy Nguyen wrote:
> On Sat, Mar 19, 2016 at 8:19 AM, David Turner <
> dturner@twopensource.com> wrote:
> > Each write() has syscall overhead, and writing a large index
> > entails
> > many such calls.  A larger write buffer reduces the overhead,
> > leading to increased performance.
> > 
> > On my repo, which has an index size of 30m, this saves about 10ms
> > of
> > time writing the index.
> 
> I wonder if split-index does not work as intended. Because if it
> does,
> you should rarely need to write that such big index files (or is this
> 30mb the small part and the base index is even bigger?).
> 
> But I agree with Torsten that we should make this configurable,
> preferably via config file, if not we can still move this define back
> in Makefile, overridable using config.mak

Oh, right, split index.  Yeah, I should just turn that on.  Nevermind.