Re: Better value for chunk_size when threaded

["Jon Smirl" <jonsmirl@gmail.com>]

On 12/6/07, Nicolas Pitre <nico@cam.org> wrote:
> On Thu, 6 Dec 2007, Jon Smirl wrote:
>
> > I tried some various ideas out for chunk_size and the best strategy I
> > found was to simply set it to a constant. How does 20,000 work on
> > other CPUs?
>
> That depends on the object size.  If you have a repo with big objects
> but only 1000 of them for example, then the constant doesn't work.

How about defaulting it to 20,000 and allowing an override? It's not
fatal if we guess wrong, we just want to most common cases to work out
of the box. 20,000 is definitely better than the current window *
1000.

> Ideally I'd opt for a value that tend towards around 5 seconds worth of
> work per segment, or something like that.  Maybe using the actual
> objects size could be another way.
>
> > I'd turn on default threaded support with this change. With threads=1
> > versus non-threaded there is no appreciable difference in the time.
>
> Would need a way to determine pthreads availability from Makefile.

configure knows if pthreads is there.

> > Is there an API to ask how many CPUs are in the system? It would be
> > nice to default the number of threads equal to the number of CPUs and
> > only use pack.threads=X to override.
>
> If there is one besides futzing with /proc/cpuinfo I'd like to know
> about it.  Bonus points if it is portable.
>
>
> Nicolas
>


-- 
Jon Smirl
jonsmirl@gmail.com