Re: Performance problem, long run of identical hashes

["Jon Smirl" <jonsmirl@gmail.com>]

On 12/10/07, Nicolas Pitre <nico@cam.org> wrote:
> On Mon, 10 Dec 2007, Jon Smirl wrote:
>
> > Running oprofile during my gcc repack shows this loop as the hottest
> > place in the code by far.
>
> Well, that is kind of expected.
>
> > I added some debug printfs which show that I
> > have a 100,000+ run of identical hash entries. Processing the 100,000
> > entries also causes RAM consumption to explode.
>
> That is impossible.  If you look at the code where those hash entries
> are created in create_delta_index(), you'll notice a hard limit of
> HASH_LIMIT (currently 64) is imposed on the number of identical hash
> entries.

On 12/10/07, Jon Smirl <jonsmirl@gmail.com> wrote:
> On 12/9/07, Jon Smirl <jonsmirl@gmail.com> wrote:
> > > +               if (victim) {
> > > +                       sub_size = victim->remaining / 2;
> > > +                       list = victim->list + victim->list_size - sub_size;
> > > +                       while (sub_size && list[0]->hash &&
> > > +                              list[0]->hash == list[-1]->hash) {
> > > +                               list++;
> >
> > I think you needed to copy sub_size to another variable for this loop
>
> Copying sub_size was wrong. I believe you are checking for deltas on
> the same file. It's probably that chain of 103,817 deltas that can't
> be broken up.

At the end of multi-threaded repack one thread ends up with 45 minutes
of work after all the other threads have exited. That's because it
hits this loop and can't spit the list any more.

If the lists can't be over 64 identical entries, why do I get caught
in this loop for 50,000+ iterations? If remove this loop the threads
are balanced right to the end.

-- 
Jon Smirl
jonsmirl@gmail.com


-- 
Jon Smirl
jonsmirl@gmail.com