git-repack: Outof memory

git-repack: Outof memory

[Dongsheng Song]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi all:


I'm import from subversion. The problem appears to be git-repack phase using too many memory:

$ git-repack -a -f -d --window=64 --depth=64
Generating pack...
Done counting 123497 objects.
Deltifying 123497 objects.
  24% (29677/123497) done

$ top

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 3572 www-data  18   0 2591m 1.9g  528 R   13 94.8  81:48.98 git-pack-object


$ free
             total       used       free     shared    buffers     cached
Mem:       2076308    2029824      46484          0       2760       6800
- -/+ buffers/cache:    2020264      56044
Swap:      1028152     684032     344120


How to compute memory usage of git-repack ?

Thanks and regards,

Dongsheng

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (MingW32)

iD8DBQFFDKVh90pbDJCgbHoRAlweAJ45DhTXI+bb+nb2Y+JlbIBoFusK8wCgk0U2
XcUc9K/chYiUYq3ZLychzcU=
=NzXf
-----END PGP SIGNATURE-----

Re: git-repack: Outof memory

[Shawn Pearce]

Dongsheng Song <dongsheng.song@gmail.com> wrote:
> I'm import from subversion. The problem appears to be git-repack phase using too many memory:
> 
> $ git-repack -a -f -d --window=64 --depth=64
> Generating pack...
> Done counting 123497 objects.
> Deltifying 123497 objects.
>   24% (29677/123497) done
> 
> $ top
> 
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>  3572 www-data  18   0 2591m 1.9g  528 R   13 94.8  81:48.98 git-pack-object

*ouch* You are probably running out of address space if you are on
a 32 bit architecture.  A 2.5 GiB virtual address space is pretty
close to the maximum allowed on most OSes.

The code that I'm sitting on but haven't yet completed rebasing
onto current Git would probably help here.

Do you have any existing .pack files in .git/objects/pack?  How big
are they and their corresponding .idx files?

git-repack will need to mmap every .pack and .idx in
.git/objects/pack, plus it needs working memory for each object
(123,497 of 'em) but as I recall its pretty frugal on its per-object
allocation.  It can easily work with as many as 2 million objects on
a 32 bit system, assuming the .pack and .idx files aren't too large.
 
-- 
Shawn.

Re: git-repack: Outof memory

[Shawn Pearce]

Dongsheng Song <dongsheng.song@gmail.com> wrote:
> finished, thanks a lot.
> 
> $ ls -l .git/objects/pack/
> total 1675466
> -rw-r--r-- 1 www-data www-data    2964992 Sep 17 19:17
> pack-b09c57c25e4459f6365b5d27139abfd93bf1c86f.idx
> -rw-r--r-- 1 www-data www-data 1711037142 Sep 17 19:17
> pack-b09c57c25e4459f6365b5d27139abfd93bf1c86f.pack
> 
> If the pack files larger than 2.5g,  how can  I repack it on i686 ?

*youch* A 1.5 GiB pack file?

Your files apparently do not delta compress very well.  I fear that
you are going to bump up against address space limitations soon
on 32 bit systems.  Then you will bump up against the 4 GiB pack
file size limit.  Which means you will need to use several packs
and avoid the '-a' flag when calling git-repack.

Nico was suggesting using the default window size (rather than
--window=64) as larger windows requires more memory during repack.
But you may also need the mmap window code I'm working on.  I better
hurry up and get that into Junio's testing branches.  :-)

-- 
Shawn.

Re: git-repack: Outof memory

[Junio C Hamano]

Shawn Pearce <spearce@spearce.org> writes:

> ...  I fear that
> you are going to bump up against address space limitations soon
> on 32 bit systems.  Then you will bump up against the 4 GiB pack
> file size limit.  Which means you will need to use several packs
> and avoid the '-a' flag when calling git-repack.

I presume that you are hinting that we would need to update
git-repack so that it is still useful without --all.

Which means "pack-objects --unpacked=active-pack" would need to
be pushed out so that git-repack can be updated to do the
"archived ones and repacking the active pack" we talked about
earlier.

I think pack-objects --unpacked=active-pack is ready, so I'll
push it out.

Re: git-repack: Outof memory

[Shawn Pearce]

Junio C Hamano <junkio@cox.net> wrote:
> Shawn Pearce <spearce@spearce.org> writes:
> 
> > ...  I fear that
> > you are going to bump up against address space limitations soon
> > on 32 bit systems.  Then you will bump up against the 4 GiB pack
> > file size limit.  Which means you will need to use several packs
> > and avoid the '-a' flag when calling git-repack.
> 
> I presume that you are hinting that we would need to update
> git-repack so that it is still useful without --all.

Yes.  :-)

Apparently the Porcelain level is still open for debate.  Someone
just needs to post a reasonable implementation to get discussion
(or lack there of) going.  I'm too tired right now to try to roll
something up right, even though it looks to be rather simple.

As it is I'm having a difficult time navigating in vi to write this
email and normally vi navigation is as easy as breathing for me...

> I think pack-objects --unpacked=active-pack is ready, so I'll
> push it out.

Thanks; I noticed that in your "What's in git.git" announcement.

-- 
Shawn.

Re: git-repack: Outof memory

[Nicolas Pitre]

On Sun, 17 Sep 2006, Dongsheng Song wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Hi all:
> 
> 
> I'm import from subversion. The problem appears to be git-repack phase using too many memory:
> 
> $ git-repack -a -f -d --window=64 --depth=64

Try with that --window=64 argument removed.


Nicolas