git repack -a -d is not safe

git repack -a -d is not safe

[Matthias Lederhofer]

First the object is unpacked, start something using read_sha1_file.
read_sha1_file: try packed object (fails)
run git repack -a -d
read_sha1_file: try to read the loose object (fails)
read_sha1_file: reread list of packs
run git repack -a -d so it will create a new pack
read_sha1_file: tries to open the old packs, not the new one, and
                fails

Anyway this is quite improbable because normally a read_sha1_file is
much faster than git repack -a -d.

gdb commands to do that:
shell git init
shell seq 1 10 > a
shell git add a
shell git commit -m first
file git-cat-file
break read_sha1_file
run -p HEAD:a
next
next
shell git repack -a -d
next
next
next
shell seq 11 20 >> a
shell git add a
shell git commit -m second
shell git repack -a -d
continue

gdb session running the commands:
GNU gdb 6.4.90-debian
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i486-linux-gnu".
$ shell git init
Initialized empty Git repository in .git/
$ shell seq 1 10 > a
$ shell git add a
$ shell git commit -m first
Created initial commit c520a6d82591b2c6654abe65418a2223302f927b
 1 files changed, 10 insertions(+), 0 deletions(-)
 create mode 100644 a
$ file git-cat-file
Reading symbols from /home/matled/local/bin/git-cat-file...done.
Using host libthread_db library "/lib/tls/i686/cmov/libthread_db.so.1".
$ break read_sha1_file
Breakpoint 1 at 0x8098959: file sha1_file.c, line 1483.
$ run -p HEAD:a
Starting program: /home/matled/local/bin/git-cat-file -p HEAD:a
warning: Lowest section in system-supplied DSO at 0xffffe000 is .hash at ffffe0b4

Breakpoint 1, read_sha1_file (sha1=0xffcc0db8 "Å ¦Ø%\221²ÆeJ¾eA\212\"#0/\222{", type=0xffcc0dd0 "ÿÿÿÿ", size=0xffcc0dcc)
    at sha1_file.c:1483
1483            buf = read_packed_sha1(sha1, type, size);
$ next
1484            if (buf)
$ next
1486            map = map_sha1_file(sha1, &mapsize);
$ shell git repack -a -d
Generating pack...
Done counting 3 objects.
Deltifying 3 objects.
 100% (3/3) done
Writing 3 objects.
 100% (3/3) done
Total 3 (delta 0), reused 0 (delta 0)
Pack pack-fcbc60879b35995a9b532543a3b072b83a50b510 created.
Removing unused objects 100%...
Done.
$ next
1487            if (map) {
$ next
1492            reprepare_packed_git();
$ next
1493            return read_packed_sha1(sha1, type, size);
$ shell seq 11 20 >> a
$ shell git add a
$ shell git commit -m second
Created commit 0bc9af2dfb8538abf9247ae282d0a2bd0cfc7b0d
 1 files changed, 10 insertions(+), 0 deletions(-)
$ shell git repack -a -d
Generating pack...
Done counting 6 objects.
Deltifying 6 objects.
 100% (6/6) done
Writing 6 objects.
 100% (6/6) done
Total 6 (delta 0), reused 3 (delta 0)
Pack pack-ad55ad59fa078c3261550ebd9adf0035cf89fa63 created.
Removing unused objects 100%...
Done.
$ continue 
Continuing.
fatal: packfile .git/objects/pack/pack-fcbc60879b35995a9b532543a3b072b83a50b510.pack cannot be opened

Program exited with code 0200.
$ 

Re: git repack -a -d is not safe

[Shawn O. Pearce]

Matthias Lederhofer <matled@gmx.net> wrote:
> First the object is unpacked, start something using read_sha1_file.
> read_sha1_file: try packed object (fails)
> run git repack -a -d
> read_sha1_file: try to read the loose object (fails)
> read_sha1_file: reread list of packs
> run git repack -a -d so it will create a new pack
> read_sha1_file: tries to open the old packs, not the new one, and
>                 fails

My 4 patch series that I just posted should (partially) fix this
problem.  We cannot close it completely however as there always
exists the potential of the OS to starve the reader and allow 2
full repacks between the time the reader notices a .idx file and
the time the reader could open the .pack.

At least now with my additional changes Git will report that a
packfile is not usable, and then that the object cannot be found,
and die.

All I can say is, wait at least a few minutes between repacks?  ;-)

-- 
Shawn.

Re: git repack -a -d is not safe

[Johannes Schindelin]

Hi,

On Thu, 1 Feb 2007, Shawn O. Pearce wrote:

> Matthias Lederhofer <matled@gmx.net> wrote:
> > First the object is unpacked, start something using read_sha1_file.
> > read_sha1_file: try packed object (fails)
> > run git repack -a -d
> > read_sha1_file: try to read the loose object (fails)
> > read_sha1_file: reread list of packs
> > run git repack -a -d so it will create a new pack
> > read_sha1_file: tries to open the old packs, not the new one, and
> >                 fails
> 
> My 4 patch series that I just posted should (partially) fix this
> problem.  We cannot close it completely however as there always
> exists the potential of the OS to starve the reader and allow 2
> full repacks between the time the reader notices a .idx file and
> the time the reader could open the .pack.

I may be misunderstanding the problem, but can't we just keep looking for 
a new pack until either there is _no_ new pack to be mapped, or the object 
is found?

Ciao,
Dscho