git-fast-import crashes

git-fast-import crashes

[Shun Kei Leung]

Hi all,

I am using git 1.5.3.4.206.g58ba4-dirty on Mac OS X 10.4. When I tried
to run `git-p4 rebase', it failed with a broken pipe to
`git-fast-import'. I gather the following from GDB. I am kind of
stuck. Does anyone have any idea what's going on?


Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x64617469
in_window (win=0x5004d0, offset=3501) at sha1_file.c:701
701             off_t win_off = win->offset;
(gdb) bt
#0  in_window (win=0x5004d0, offset=3501) at sha1_file.c:701
#1  0x0000a7b0 in use_pack (p=0x500740, w_cursor=0xbfffd328,
offset=3501, left=0xbfffd2c8) at sha1_file.c:728
#2  0x0000aaa0 in unpack_object_header (p=0x64617461, w_curs=0x0,
curpos=0xbfffd330, sizep=0xbffff4ec) at sha1_file.c:1329
#3  0x0000e17c in unpack_entry (p=0x500740, obj_offset=3501,
type=0xbffff4e8, sizep=0xbffff4ec) at sha1_file.c:1591
#4  0x0000dc8c in read_packed_sha1 (sha1=0x64617461 <Address
0x64617461 out of bounds>, type=0xbffff4e8, size=0xbffff4ec) at
sha1_file.c:1811
#5  0x0000dd90 in read_sha1_file (sha1=0xbffff4f0
")S??=?P\r?6[?w?S[\021\004??", type=0xbffff4e8, size=0xbffff4ec) at
sha1_file.c:1877
#6  0x0000e020 in read_object_with_reference (sha1=0x100950b
")S??=?P\r?6[?w?S[\021\004??", required_type_name=0x0,
size=0xbffff578, actual_sha1_return=0x100950b
")S??=?P\r?6[?w?S[\021\004??") at sha1_file.c:1906
#7  0x0000450c in cmd_from_existing (b=0x10094c0) at fast-import.c:1922
#8  0x00004b3c in cmd_from (b=0x10094c0) at fast-import.c:1965
#9  0x000078a0 in cmd_new_commit () at fast-import.c:2044
#10 0x000088f0 in main (argc=213252, argv=0xbffff838) at fast-import.c:2329
(gdb) print win
$1 = (struct pack_window *) 0x5004d0
(gdb) print *win
$2 = {
  next = 0x64617461,
  base = 0x20333936 <Address 0x20333936 out of bounds>,
  offset = 22523564414626158,
  len = 1685026675,
  last_used = 795894075,
  inuse_cnt = 0
}


Regards,
Kevin Leung

Re: git-fast-import crashes

[Shawn O. Pearce]

Shun Kei Leung <kevinlsk@gmail.com> wrote:
> Program received signal EXC_BAD_ACCESS, Could not access memory.
> Reason: KERN_INVALID_ADDRESS at address: 0x64617469
> in_window (win=0x5004d0, offset=3501) at sha1_file.c:701
> 701             off_t win_off = win->offset;
...
> (gdb) print win
> $1 = (struct pack_window *) 0x5004d0
> (gdb) print *win
> $2 = {
>   next = 0x64617461,
>   base = 0x20333936 <Address 0x20333936 out of bounds>,
>   offset = 22523564414626158,
>   len = 1685026675,
>   last_used = 795894075,
>   inuse_cnt = 0
> }

Wow.  There's no way that struct pack_window is valid anymore.
The base isn't a valid address.  The offset cannot possibly be
correct (you don't have that big of a packfile, do you?!

What does `git count-objects -v` give you?  I'm specifically
interested in how many packfiles you have.  The other thing that
may be interesting to see is the value of pack_open_windows and
peak_pack_open_windows (file scope in sha1_file.c).

Then again, maybe that isn't interesting.  This looks like it is
memory corruption (e.g. someone overwriting a free'd segment),
but that sort of memory corruption is very hard to track down.

-- 
Shawn.

Re: git-fast-import crashes

[Shawn O. Pearce]

"Shawn O. Pearce" <spearce@spearce.org> wrote:
> Shun Kei Leung <kevinlsk@gmail.com> wrote:
> > I am using git 1.5.3.4.206.g58ba4-dirty on Mac OS X 10.4.
...
> > Program received signal EXC_BAD_ACCESS, Could not access memory.
> > Reason: KERN_INVALID_ADDRESS at address: 0x64617469
...
> This looks like it is
> memory corruption (e.g. someone overwriting a free'd segment),
> but that sort of memory corruption is very hard to track down.

OK, so the version you have (58ba4) is the latest fast-import after
the strbuf.c series went in.  The one immediately before that series
was 4bf538 and is probably actually stable.

So I wonder, can you test 4bf538 and then if it is good bisect
between those two commits?  There must be a memory corruption
introduced by one of the strbuf changes...

-- 
Shawn.

Re: git-fast-import crashes

[Pierre Habouzit]

[-- Attachment #1: Type: text/plain, Size: 1166 bytes --]

On Sat, Oct 13, 2007 at 03:34:07AM +0000, Shawn O. Pearce wrote:
> "Shawn O. Pearce" <spearce@spearce.org> wrote:
> > Shun Kei Leung <kevinlsk@gmail.com> wrote:
> > > I am using git 1.5.3.4.206.g58ba4-dirty on Mac OS X 10.4.
> ....
> > > Program received signal EXC_BAD_ACCESS, Could not access memory.
> > > Reason: KERN_INVALID_ADDRESS at address: 0x64617469
> ....
> > This looks like it is
> > memory corruption (e.g. someone overwriting a free'd segment),
> > but that sort of memory corruption is very hard to track down.
> 
> OK, so the version you have (58ba4) is the latest fast-import after
> the strbuf.c series went in.  The one immediately before that series
> was 4bf538 and is probably actually stable.
> 
> So I wonder, can you test 4bf538 and then if it is good bisect
> between those two commits?  There must be a memory corruption
> introduced by one of the strbuf changes...

  Gasp, if you get the offending sha1 commit, don't forget to Cc: me.

-- 
·O·  Pierre Habouzit
··O                                                madcoder@debian.org
OOO                                                http://www.madism.org

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: git-fast-import crashes

[Pierre Habouzit]

[-- Attachment #1: Type: text/plain, Size: 1732 bytes --]

On sam, oct 13, 2007 at 07:36:40 +0000, Pierre Habouzit wrote:
> On Sat, Oct 13, 2007 at 03:34:07AM +0000, Shawn O. Pearce wrote:
> > "Shawn O. Pearce" <spearce@spearce.org> wrote:
> > > Shun Kei Leung <kevinlsk@gmail.com> wrote:
> > > > I am using git 1.5.3.4.206.g58ba4-dirty on Mac OS X 10.4.
> > ....
> > > > Program received signal EXC_BAD_ACCESS, Could not access memory.
> > > > Reason: KERN_INVALID_ADDRESS at address: 0x64617469
> > ....
> > > This looks like it is
> > > memory corruption (e.g. someone overwriting a free'd segment),
> > > but that sort of memory corruption is very hard to track down.
> > 
> > OK, so the version you have (58ba4) is the latest fast-import after
> > the strbuf.c series went in.  The one immediately before that series
> > was 4bf538 and is probably actually stable.
> > 
> > So I wonder, can you test 4bf538 and then if it is good bisect
> > between those two commits?  There must be a memory corruption
> > introduced by one of the strbuf changes...
> 
>   Gasp, if you get the offending sha1 commit, don't forget to Cc: me.

  Okay, given that fast-import uses quote_c_style, I believe this is the
same but that the one that was reported already. I've read the full
`git diff 4bf53833dbca666f61b5177977e96d453527db20.. -- fast-import.c`
and nothing alarming shows up.

  Please try to apply:
http://git.madism.org/?p=git.git;a=commit;h=7406e83342cd445ac38c1753c5fce75377737e2f

  And see if that fixes the issue for you. Else a bisection would be
much appreciated. Thanks.

-- 
·O·  Pierre Habouzit
··O                                                madcoder@debian.org
OOO                                                http://www.madism.org

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: git-fast-import crashes

[Johannes Schindelin]

Hi,

On Fri, 12 Oct 2007, Shawn O. Pearce wrote:

> Shun Kei Leung <kevinlsk@gmail.com> wrote:
> > Program received signal EXC_BAD_ACCESS, Could not access memory.
> > Reason: KERN_INVALID_ADDRESS at address: 0x64617469
> > in_window (win=0x5004d0, offset=3501) at sha1_file.c:701
> > 701             off_t win_off = win->offset;
> ...
> > (gdb) print win
> > $1 = (struct pack_window *) 0x5004d0
> > (gdb) print *win
> > $2 = {
> >   next = 0x64617461,
> >   base = 0x20333936 <Address 0x20333936 out of bounds>,
> >   offset = 22523564414626158,
> >   len = 1685026675,
> >   last_used = 795894075,
> >   inuse_cnt = 0
> > }
> 
> Wow.  There's no way that struct pack_window is valid anymore.
>
> [...] 
>
> This looks like it is memory corruption (e.g. someone overwriting a 
> free'd segment), but that sort of memory corruption is very hard to 
> track down.

I found valgrind invaluable to find such errors.

Ciao,
Dscho

Re: git-fast-import crashes

[Shun Kei Leung]

Hi,

Sorry for the late reply. I was away from my computer in the weekend.


Hi Pierre,

I didn't try:
http://git.madism.org/?p=git.git;a=commit;h=7406e83342cd445ac38c1753c5fce75377737e2f

because the bad commit turns out to be b449f4c according to `git bisect'.


Hi Shawn,

I include the output of `git count-objects -v' for your information:

count: 104
size: 552
in-pack: 10652
packs: 12
prune-packable: 0
garbage: 0


Regards,
Kevin Leung

Re: git-fast-import crashes

[Pierre Habouzit]

[-- Attachment #1: Type: text/plain, Size: 916 bytes --]

On Mon, Oct 15, 2007 at 04:53:38AM +0000, Shun Kei Leung wrote:
> Hi,
> 
> Sorry for the late reply. I was away from my computer in the weekend.
> 
> 
> Hi Pierre,
> 
> I didn't try:
> http://git.madism.org/?p=git.git;a=commit;h=7406e83342cd445ac38c1753c5fce75377737e2f
> 
> because the bad commit turns out to be b449f4c according to `git bisect'.

  I don't get the reason for your "because" but so be it. The commit you
show is not obviously broken to me, especially not in fast-import.c, so
I'll need more input.  Could you please run your test in valgrind and
report the output please? Or if the data to reproduce the bug are online
or shareable, it'd be great to share, so that I can reproduce the issue
here.


-- 
·O·  Pierre Habouzit
··O                                                madcoder@debian.org
OOO                                                http://www.madism.org

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: git-fast-import crashes

[Shun Kei Leung]

>   I don't get the reason for your "because" but so be it.

Well, my reasoning was that the commit didn't touch the convert.c. But
after re-reading the patch, I think I should apply and test with the
patch again.

...
> Or if the data to reproduce the bug are online
> or shareable, it'd be great to share, so that I can reproduce the issue
> here.

The repository is private, and it is in maintenance mode for the rest
of today. I will get back to you tomorrow with updates.


Thanks & regards,
Kevin Leung

Re: git-fast-import crashes

[Shun Kei Leung]

Hi Pierre,

I have tested with your patch applied, and it still crashed badly.

I compiled the b449f4c version and I got a different backtrace from
GDB. I am still not sure about how the error occurs. I attached the
memory usage from various command line tools.

Regards,
Kevin Leung



GDB output
===========

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x756c7463
0x90018d0c in szone_realloc ()
(gdb) bt
#0  0x90018d0c in szone_realloc ()
#1  0x90018898 in realloc ()
#2  0x0001f8c8 in strbuf_grow (sb=0x75ef8, extra=0) at git-compat-util.h:223
#3  0x0001fc14 in read_line (sb=0x75ef8, fp=0xa0001b9c, term=10) at strbuf.c:107
#4  0x00002da8 in read_next_command () at fast-import.c:1599
#5  0x0000742c in cmd_new_commit () at fast-import.c:2118
#6  0x00007e80 in main (argc=1, argv=0xbffff828) at fast-import.c:2390
(gdb) frame 3
#3  0x0001fc14 in read_line (sb=0x75ef8, fp=0xa0001b9c, term=10) at strbuf.c:107
107                     strbuf_grow(sb, 1);
(gdb) print *sb
$1 = {
  alloc = 60,
  len = 23,
  eof = 0,
  buf = 0x5020b0 "D windows/packages/cdc/ces/adm"
}
(gdb) frame 2
#2  0x0001f8c8 in strbuf_grow (sb=0x75ef8, extra=0) at git-compat-util.h:223
223             void *ret = realloc(ptr, size);
(gdb) print (char *) ptr
$2 = 0x5020b0 "D windows/packages/cdc/ces/adm"
(gdb) print (char *)ret
$3 = 0xd <Address 0xd out of bounds>
(gdb)


The output of `leaks'
=================

Process 27075: 88 nodes malloced for 2335 KB
Process 27075: 6 leaks for 624 total leaked bytes.
Leak: 0x005012b0  size=208      string 'EOT'
Leak: 0x00501e60  size=128      string 'EOT'
Leak: 0x00501c80  size=128      string 'EOT'
Leak: 0x00500f10  size=64       string 'EOT'
Leak: 0x00501fa0  size=64       string 'EOT'
Leak: 0x00501f60  size=32
        0x00000000 0x02008000 0x00000000 0x01000000     ................
        0x02000000 0x0000001a 0x00000000 0x00000000     ................



Below is the output of `malloc_history'
==============================

Deallocation of 5020b0 - no corresponding allocation
67 calls for 2874 bytes: thread_a000ed88 |0x0 | start | _start | main
| cmd_new_commit | read_next_command | read_line | strbuf_grow

12 calls for 2016 bytes: thread_a000ed88 |0x0 | start | _start | main
| prepare_packed_git | prepare_packed_git_one | add_packed_git

8 calls for 831 bytes: thread_a000ed88 |0x0 | start | _start | main |
cmd_new_commit | cmd_data | read_line | strbuf_grow

4 calls for 160 bytes: thread_a000ed88 |0x0 | start | _start |
__keymgr_dwarf2_register_sections | _dyld_register_func_for_add_image
| dyld::registerAddCallback(void (*)(mach_header const*, long)) |
ImageLoader::runNotification(ImageLoader::LinkContext const&,
unsigned) | ImageLoader::recursiveImageNotification(ImageLoader::LinkContext
const&, unsigned) | dyld::imageNotification(ImageLoader*, unsigned) |
dwarf2_unwind_dyld_add_image_hook

3 calls for 1976 bytes: thread_a000ed88 |0x0 | start | _start | main |
cmd_new_commit | tree_content_set | tree_content_set | load_tree |
read_sha1_file | read_packed_sha1 | unpack_entry | unpack_entry |
patch_delta

3 calls for 84 bytes: thread_a000ed88 |0x0 | start | _start | main |
cmd_new_commit | tree_content_set | tree_content_set | load_tree |
read_sha1_file | read_packed_sha1 | unpack_entry |
unpack_object_header | use_pack

2 calls for 1200 bytes: thread_a000ed88 |0x0 | start | _start | main |
cmd_new_commit | tree_content_set | load_tree | read_sha1_file |
read_packed_sha1 | unpack_entry | unpack_entry | unpack_entry |
unpack_entry | patch_delta

2 calls for 1200 bytes: thread_a000ed88 |0x0 | start | _start | main |
cmd_new_commit | tree_content_set | load_tree | read_sha1_file |
read_packed_sha1 | unpack_entry | unpack_entry | unpack_entry |
patch_delta

2 calls for 1200 bytes: thread_a000ed88 |0x0 | start | _start | main |
cmd_new_commit | tree_content_set | load_tree | read_sha1_file |
read_packed_sha1 | unpack_entry | unpack_entry | patch_delta

2 calls for 80 bytes: thread_a000ed88 |0x0 | start | _start |
__keymgr_dwarf2_register_sections | _dyld_register_func_for_add_image
| dyld::registerAddCallback(void (*)(mach_header const*, long)) |
ImageLoader::runNotification(ImageLoader::LinkContext const&,
unsigned) | ImageLoader::recursiveImageNotification(ImageLoader::LinkContext
const&, unsigned) | dyld::imageNotification(ImageLoader*, unsigned) |
dwarf2_unwind_dyld_add_image_hook

2 calls for 56 bytes: thread_a000ed88 |0x0 | start | _start | main |
cmd_new_commit | tree_content_set | load_tree | read_sha1_file |
read_packed_sha1 | unpack_entry | unpack_object_header | use_pack

1 calls for 2097152 bytes: thread_a000ed88 |0x0 | start | _start |
main | pool_calloc | pool_alloc

1 calls for 160012 bytes: thread_a000ed88 |0x0 | start | _start | main
| alloc_objects

1 calls for 52000 bytes: thread_a000ed88 |0x0 | start | _start | main
| cmd_new_commit | load_tree | new_tree_entry

1 calls for 17804 bytes: thread_a000ed88 |0x0 | start | _start | main

1 calls for 8192 bytes: thread_a000ed88 |0x0 | start | _start | main |
cmd_new_commit | cmd_data

1 calls for 4156 bytes: thread_a000ed88 |0x0 | start | _start | main

1 calls for 4096 bytes: thread_a000ed88 |0x0 | start | _start | main |
cmd_new_commit | store_tree | store_tree | store_tree | mktree |
size_dbuf

1 calls for 4096 bytes: thread_a000ed88 |0x0 | start | _start | main |
read_next_command | read_line | fgetc | __srget | __srefill |
__smakebuf

1 calls for 3072 bytes: thread_a000ed88 |0x0 | start | _start | main |
cmd_new_commit | store_tree | store_tree | store_tree | mktree |
size_dbuf

1 calls for 2048 bytes: thread_a000ed88 |0x0 | start | _start | main |
cmd_new_commit | size_dbuf

1 calls for 1149 bytes: thread_a000ed88 |0x0 | start | _start | main |
cmd_new_commit | tree_content_set | tree_content_set | load_tree |
read_sha1_file | read_packed_sha1 | unpack_entry | unpack_entry |
unpack_entry | patch_delta

1 calls for 1149 bytes: thread_a000ed88 |0x0 | start | _start | main |
cmd_new_commit | tree_content_set | tree_content_set | load_tree |
read_sha1_file | read_packed_sha1 | unpack_entry | unpack_entry |
unpack_entry | unpack_entry | unpack_compressed_entry

1 calls for 1032 bytes: thread_a000ed88 |0x0 | start | _start | main |
cmd_new_commit | tree_content_set | load_tree | read_sha1_file |
read_packed_sha1 | unpack_entry | unpack_entry | unpack_entry |
unpack_entry | unpack_entry | unpack_entry | unpack_compressed_entry

1 calls for 822 bytes: thread_a000ed88 |0x0 | start | _start | main |
cmd_new_commit | tree_content_set | tree_content_set |
tree_content_set | load_tree | read_sha1_file | read_packed_sha1 |
unpack_entry | unpack_entry | unpack_compressed_entry

1 calls for 522 bytes: thread_a000ed88 |0x0 | start | _start | main |
cmd_new_commit | tree_content_set | load_tree | read_sha1_file |
read_packed_sha1 | unpack_entry | unpack_entry |
unpack_compressed_entry

1 calls for 436 bytes: thread_a000ed88 |0x0 | start | _start | main |
cmd_new_commit | tree_content_set | tree_content_set |
tree_content_set | tree_content_set | tree_content_set |
tree_content_set | load_tree | read_sha1_file | read_packed_sha1 |
unpack_entry | unpack_entry | unpack_compressed_entry

1 calls for 400 bytes: thread_a000ed88 |0x0 | start | _start | main

1 calls for 285 bytes: thread_a000ed88 |0x0 | start | _start | main |
cmd_new_commit | tree_content_set | tree_content_set |
tree_content_set | tree_content_set | load_tree | read_sha1_file |
read_packed_sha1 | unpack_entry | unpack_entry |
unpack_compressed_entry

1 calls for 258 bytes: thread_a000ed88 |0x0 | start | _start | main |
start_packfile | mkstemp | arc4random

1 calls for 212 bytes: thread_a000ed88 |0x0 | start | _start | main |
cmd_new_commit | tree_content_set | tree_content_set |
tree_content_set | tree_content_set | tree_content_set | load_tree |
read_sha1_file | read_packed_sha1 | unpack_entry | unpack_entry |
unpack_compressed_entry

1 calls for 168 bytes: thread_a000ed88 |0x0 | start | _start | main |
cmd_new_commit | tree_content_set | load_tree | read_sha1_file |
read_packed_sha1 | unpack_entry | unpack_entry | unpack_entry |
unpack_entry | unpack_entry | unpack_entry | patch_delta

1 calls for 168 bytes: thread_a000ed88 |0x0 | start | _start | main |
cmd_new_commit | tree_content_set | load_tree | read_sha1_file |
read_packed_sha1 | unpack_entry | unpack_entry | unpack_entry |
unpack_entry | unpack_entry | unpack_entry | unpack_entry |
unpack_entry | patch_delta

1 calls for 168 bytes: thread_a000ed88 |0x0 | start | _start | main |
cmd_new_commit | tree_content_set | load_tree | read_sha1_file |
read_packed_sha1 | unpack_entry | unpack_entry | unpack_entry |
unpack_entry | unpack_entry | unpack_entry | unpack_entry |
unpack_entry | unpack_entry | patch_delta

1 calls for 168 bytes: thread_a000ed88 |0x0 | start | _start | main |
cmd_new_commit | tree_content_set | load_tree | read_sha1_file |
read_packed_sha1 | unpack_entry | unpack_entry | unpack_entry |
unpack_entry | unpack_entry | unpack_entry | unpack_entry |
unpack_entry | unpack_entry | unpack_entry | unpack_compressed_entry

1 calls for 168 bytes: thread_a000ed88 |0x0 | start | _start | main |
cmd_new_commit | tree_content_set | load_tree | read_sha1_file |
read_packed_sha1 | unpack_entry | unpack_entry | unpack_entry |
unpack_entry | unpack_entry | unpack_entry | unpack_entry |
patch_delta

1 calls for 129 bytes: thread_a000ed88 |0x0 | start | _start | main |
start_packfile

1 calls for 122 bytes: thread_a000ed88 |0x0 | start | _start | main |
cmd_new_commit | tree_content_set | tree_content_set | load_tree |
read_sha1_file | read_packed_sha1 | unpack_entry | unpack_entry |
unpack_compressed_entry

1 calls for 64 bytes: thread_a000ed88 |0x0 | start | _start | main |
git_config | git_path | get_git_dir | setup_git_env | sprintf$LDBL128
| __vfprintf$LDBL128 | localeconv_l

1 calls for 61 bytes: thread_a000ed88 |0x0 | start | _start | main |
cmd_new_commit | parse_ident

1 calls for 60 bytes: thread_a000ed88 |0x0 | start | _start |
__keymgr_dwarf2_register_sections | _dyld_register_func_for_add_image
| dyld::registerAddCallback(void (*)(mach_header const*, long)) |
ImageLoader::runNotification(ImageLoader::LinkContext const&,
unsigned) | dyld::imageNotification(ImageLoader*, unsigned) |
dwarf2_unwind_dyld_add_image_hook |
_keymgr_get_and_lock_processwide_ptr_2 | get_or_create_key_element

1 calls for 52 bytes: thread_a000ed88 |0x0 | start | _start | main |
git_config | git_path | get_git_dir | setup_git_env | strdup

1 calls for 48 bytes: thread_a000ed88 |0x0 | start | _start | main |
git_config | git_path | get_git_dir | setup_git_env

1 calls for 46 bytes: thread_a000ed88 |0x0 | start | _start | main |
git_config | git_path | get_git_dir | setup_git_env

1 calls for 45 bytes: thread_a000ed88 |0x0 | start | _start | main |
git_config | git_path | get_git_dir | setup_git_env

1 calls for 40 bytes: thread_a000ed88 |0x0 | start | _start |
__keymgr_dwarf2_register_sections | _dyld_register_func_for_add_image
| dyld::registerAddCallback(void (*)(mach_header const*, long)) |
ImageLoader::runNotification(ImageLoader::LinkContext const&,
unsigned) | dyld::imageNotification(ImageLoader*, unsigned) |
dwarf2_unwind_dyld_add_image_hook

1 calls for 40 bytes: thread_a000ed88 |0x0 | start | _start |
__keymgr_dwarf2_register_sections | _dyld_register_func_for_add_image
| dyld::registerAddCallback(void (*)(mach_header const*, long)) |
ImageLoader::runNotification(ImageLoader::LinkContext const&,
unsigned) | ImageLoader::recursiveImageNotification(ImageLoader::LinkContext
const&, unsigned) | dyld::imageNotification(ImageLoader*, unsigned) |
dwarf2_unwind_dyld_add_image_hook

1 calls for 28 bytes: thread_a000ed88 |0x0 | start | _start | main |
cmd_new_commit | tree_content_set | tree_content_set |
tree_content_set | tree_content_set | tree_content_set |
tree_content_set | load_tree | read_sha1_file | read_packed_sha1 |
unpack_entry | unpack_object_header | use_pack

1 calls for 18 bytes: thread_a000ed88 |0x0 | start | _start |
__keymgr_dwarf2_register_sections | _dyld_register_func_for_add_image
| dyld::registerAddCallback(void (*)(mach_header const*, long)) |
ImageLoader::runNotification(ImageLoader::LinkContext const&,
unsigned) | dyld::imageNotification(ImageLoader*, unsigned) |
dwarf2_unwind_dyld_add_image_hook | calloc

1 calls for 4 bytes: thread_a000ed88 |0x0 | start | _start | main |
start_packfile


Output of `vmmap'
===============

Virtual Memory Map of process 27270 (git-fast-import)
Output report format:  2.0

==== Non-writable regions for process 27270
__PAGEZERO             00000000-00001000 [    4K] ---/--- SM=NUL
/Users/developer/bin/git-fast-import
__TEXT                 00001000-00034000 [  204K] r-x/rwx SM=COW
/Users/developer/bin/git-fast-import
__LINKEDIT             00080000-000f3000 [  460K] r--/rwx SM=COW
/Users/developer/bin/git-fast-import
mapped file            000f7000-000f8000 [    4K] r--/rwx SM=COW
/Volumes/data/kevin/test_gitp4/cdc/.git/objects/pack/pack-b7cda23e237f38b466d2f7b54f665a7d0c8a32ed.idx
mapped file            000f8000-000f9000 [    4K] r--/rwx SM=COW
/Volumes/data/kevin/test_gitp4/cdc/.git/objects/pack/pack-5f301a4b08a7f7d44b288f2a96d33f3c3efbf7fb.idx
mapped file            000f9000-000fa000 [    4K] r--/rwx SM=COW
/Volumes/data/kevin/test_gitp4/cdc/.git/objects/pack/pack-ee6298d97a1154422169e66d6655071a49cb8686.idx
__TEXT                 00205000-00216000 [   68K] r-x/rwx SM=COW
/opt/local/lib/libz.1.dylib
__LINKEDIT             00217000-00219000 [    8K] r--/rwx SM=COW
/opt/local/lib/libz.1.dylib
__TEXT                 00219000-00313000 [ 1000K] r-x/rwx SM=COW
/opt/local/lib/libiconv.2.dylib
__LINKEDIT             00314000-0031e000 [   40K] r--/rwx SM=COW
/opt/local/lib/libiconv.2.dylib
__TEXT                 0031e000-00421000 [ 1036K] r-x/rwx SM=COW
/opt/local/lib/libcrypto.0.9.8.dylib
__LINKEDIT             00438000-0048d000 [  340K] r--/rwx SM=COW
/opt/local/lib/libcrypto.0.9.8.dylib
__TEXT                 0048d000-004c8000 [  236K] r-x/rwx SM=COW
/opt/local/lib/libssl.0.9.8.dylib
__LINKEDIT             004cc000-004dd000 [   68K] r--/rwx SM=COW
/opt/local/lib/libssl.0.9.8.dylib
mapped file            004dd000-004df000 [    8K] r--/rwx SM=COW
/Volumes/data/kevin/test_gitp4/cdc/.git/objects/pack/pack-5f301a4b08a7f7d44b288f2a96d33f3c3efbf7fb.pack
mapped file            004ec000-004ed000 [    4K] r--/rwx SM=COW
/Volumes/data/kevin/test_gitp4/cdc/.git/objects/pack/pack-ea2fe30b68a19c9cd6a4bf1dd345377b4dd77df0.idx
mapped file            004ed000-004ee000 [    4K] r--/rwx SM=COW
/Volumes/data/kevin/test_gitp4/cdc/.git/objects/pack/pack-97e365a44e20cb37576362c81afdaa879a871a10.idx
mapped file            004ee000-004ef000 [    4K] r--/rwx SM=COW
/Volumes/data/kevin/test_gitp4/cdc/.git/objects/pack/pack-f1bcc0635a35caff73d8ccac60ec6656f0527a10.idx
mapped file            004ef000-004f0000 [    4K] r--/rwx SM=COW
/Volumes/data/kevin/test_gitp4/cdc/.git/objects/pack/pack-f1bcc0635a35caff73d8ccac60ec6656f0527a10.pack
mapped file            004f0000-004f1000 [    4K] r--/rwx SM=COW
/Volumes/data/kevin/test_gitp4/cdc/.git/objects/pack/pack-60dd3fcceddf3eacbe2279ef191f87668b97fb57.idx
mapped file            004f1000-004f2000 [    4K] r--/rwx SM=COW
/Volumes/data/kevin/test_gitp4/cdc/.git/objects/pack/pack-8a733b7ccad3eb85b4286a011eccdff20441d9e4.idx
mapped file            004f2000-004f7000 [   20K] r--/rwx SM=COW
/Volumes/data/kevin/test_gitp4/cdc/.git/objects/pack/pack-8acbc2151777eb8a2093e5229897d5711dab175b.idx
mapped file            004f7000-004fa000 [   12K] r--/rwx SM=COW
/Volumes/data/kevin/test_gitp4/cdc/.git/objects/pack/pack-3d264397b577400c2277f1b89f2065e0cc9846e3.idx
mapped file            004fa000-004fc000 [    8K] r--/rwx SM=COW
/Volumes/data/kevin/test_gitp4/cdc/.git/objects/pack/pack-010e1c06a07ef38c31f8e01782eeda433b007709.idx
mapped file            004fc000-00500000 [   16K] r--/rwx SM=COW
/Volumes/data/kevin/test_gitp4/cdc/.git/objects/pack/pack-ea2fe30b68a19c9cd6a4bf1dd345377b4dd77df0.pack
mapped file            006ad000-006e4000 [  220K] r--/rwx SM=COW
/Volumes/data/kevin/test_gitp4/cdc/.git/objects/pack/pack-3294c669f264ef7a21e2280bfd934e7459140022.idx
mapped file            01208000-01363000 [ 1388K] r--/rwx SM=COW
/Volumes/data/kevin/test_gitp4/cdc/.git/objects/pack/pack-8acbc2151777eb8a2093e5229897d5711dab175b.pack
mapped file            01363000-0146c000 [ 1060K] r--/rwx SM=COW
/Volumes/data/kevin/test_gitp4/cdc/.git/objects/pack/pack-010e1c06a07ef38c31f8e01782eeda433b007709.pack
mapped file            02008000-052fe000 [52184K] r--/rwx SM=COW
/Volumes/data/kevin/test_gitp4/cdc/.git/objects/pack/pack-3294c669f264ef7a21e2280bfd934e7459140022.pack
__TEXT                 8fe00000-8fe01000 [    4K] r-x/rwx SM=COW  /usr/lib/dyld
__TEXT                 8fe01000-8fe02000 [    4K] r-x/rwx SM=COW  /usr/lib/dyld
__TEXT                 8fe02000-8fe53000 [  324K] r-x/rwx SM=COW  /usr/lib/dyld
__LINKEDIT             8fe5d000-8fe71000 [   80K] r--/rwx SM=COW  /usr/lib/dyld
__TEXT                 90000000-901bd000 [ 1780K] r-x/r-x SM=COW
/usr/lib/libSystem.B.dylib
__LINKEDIT             901bd000-90214000 [  348K] r--/r-- SM=COW
/usr/lib/libSystem.B.dylib
__TEXT                 90214000-9021a000 [   24K] r-x/r-x SM=COW
/usr/lib/system/libmathCommon.A.dylib
__LINKEDIT             9021a000-9021b000 [    4K] r--/r-- SM=COW
/usr/lib/system/libmathCommon.A.dylib
__TEXT                 91433000-9143f000 [   48K] r-x/r-x SM=COW
/usr/lib/libgcc_s.1.dylib
__LINKEDIT             9143f000-91443000 [   16K] r--/r-- SM=COW
/usr/lib/libgcc_s.1.dylib
__DATA                 a0214000-a0215000 [    4K] r--/r-- SM=COW
/usr/lib/system/libmathCommon.A.dylib
STACK GUARD            fffec000-ffff0000 [   16K] ---/rwx SM=NUL
system                 ffff8000-ffffa000 [    8K] r--/r-- SM=SHM
commpage [libSystem.B.dylib]

==== Writable regions for process 27270
__DATA                 00034000-00035000 [    4K] rw-/rwx SM=COW
/Users/developer/bin/git-fast-import
__DATA                 00035000-00080000 [  300K] rw-/rwx SM=PRV
/Users/developer/bin/git-fast-import
VM_ALLOCATE ?          000fa000-000fb000 [    4K] rw-/rwx SM=NUL
MALLOC_LARGE           000fb000-00100000 [   20K] rw-/rwx SM=PRV
DefaultMallocZone_0x500000
MALLOC_TINY (freed?)   00100000-00102000 [    8K] rw-/rwx SM=COW
MALLOC_TINY (freed?)   00102000-00205000 [ 1036K] rw-/rwx SM=ZER
__DATA                 00216000-00217000 [    4K] rw-/rwx SM=COW
/opt/local/lib/libz.1.dylib
__DATA                 00313000-00314000 [    4K] rw-/rwx SM=COW
/opt/local/lib/libiconv.2.dylib
__DATA                 00421000-00435000 [   80K] rw-/rwx SM=COW
/opt/local/lib/libcrypto.0.9.8.dylib
__DATA                 00435000-00438000 [   12K] rw-/rwx SM=NUL
/opt/local/lib/libcrypto.0.9.8.dylib
__DATA                 004c8000-004cc000 [   16K] rw-/rwx SM=COW
/opt/local/lib/libssl.0.9.8.dylib
MALLOC_LARGE           004df000-004ec000 [   52K] rw-/rwx SM=PRV
DefaultMallocZone_0x500000
MALLOC_TINY            00500000-00605000 [ 1044K] rw-/rwx SM=COW
DefaultMallocZone_0x500000
Performance tool data  00605000-00685000 [  512K] rw-/rwx SM=COW
MALLOC_LARGE           00685000-006ad000 [  160K] rw-/rwx SM=PRV
DefaultMallocZone_0x500000
Performance tool data  006e4000-006ed000 [   36K] rw-/rwx SM=COW
MALLOC (freed?)        00800000-01008000 [ 8224K] rw-/rwx SM=PRV
MALLOC_LARGE           01008000-01208000 [ 2048K] rw-/rwx SM=PRV
DefaultMallocZone_0x500000
MALLOC                 01800000-02008000 [ 8224K] rw-/rwx SM=COW
DefaultMallocZone_0x500000
__DATA                 8fe53000-8fe54000 [    4K] rw-/rwx SM=COW  /usr/lib/dyld
__DATA                 8fe54000-8fe5a000 [   24K] rw-/rwx SM=COW  /usr/lib/dyld
__DATA                 8fe5a000-8fe5d000 [   12K] rw-/rwx SM=PRV  /usr/lib/dyld
__DATA                 a0000000-a0002000 [    8K] rw-/rw- SM=COW
/usr/lib/libSystem.B.dylib
__DATA                 a0002000-a0003000 [    4K] rw-/rw- SM=COW
/usr/lib/libSystem.B.dylib
__DATA                 a0003000-a0004000 [    4K] rw-/rw- SM=COW
/usr/lib/libSystem.B.dylib
__DATA                 a0004000-a0005000 [    4K] rw-/rw- SM=COW
/usr/lib/libSystem.B.dylib
__DATA                 a0005000-a000b000 [   24K] rw-/rw- SM=COW
/usr/lib/libSystem.B.dylib
__DATA                 a000b000-a0010000 [   20K] rw-/rw- SM=ZER
/usr/lib/libSystem.B.dylib
__DATA                 a0010000-a0011000 [    4K] rw-/rw- SM=COW
/usr/lib/libSystem.B.dylib
__DATA                 a0011000-a0012000 [    4K] rw-/rw- SM=ZER
/usr/lib/libSystem.B.dylib
__DATA                 a1433000-a1434000 [    4K] rw-/rw- SM=COW
/usr/lib/libgcc_s.1.dylib
Stack                  bf800000-bffff000 [ 8188K] rw-/rwx SM=PRV
Stack                  bffff000-c0000000 [    4K] rw-/rwx SM=COW  thread 0

==== Legend
SM=sharing mode:
        COW=copy_on_write PRV=private NUL=empty ALI=aliased
        SHM=shared ZER=zero_filled S/A=shared_alias

==== Summary for process 27270
ReadOnly portion of Libraries: Total=6092KB resident=2944KB(48%)
swapped_out_or_unallocated=3148KB(52%)
Writable regions: Total=30036KB written=504KB(2%) resident=968KB(3%)
swapped_out=0KB(0%) unallocated=29068KB(97%)

REGION TYPE             [ VIRTUAL]
===========             [ =======]
MALLOC                  [  20816K]
Performance tool data   [    548K]
STACK GUARD             [     16K]
Stack                   [   8192K]
VM_ALLOCATE ?           [      4K]
__DATA                  [    540K]
__LINKEDIT              [   1364K]
__PAGEZERO              [      4K]
__TEXT                  [   4728K]
mapped file             [  54952K]
system                  [      8K]

Re: git-fast-import crashes

[Pierre Habouzit]

[-- Attachment #1: Type: text/plain, Size: 3133 bytes --]

On Tue, Oct 16, 2007 at 07:13:31AM +0000, Shun Kei Leung wrote:
> Hi Pierre,
> 
> I have tested with your patch applied, and it still crashed badly.

  okay

> I compiled the b449f4c version and I got a different backtrace from
> GDB. I am still not sure about how the error occurs. I attached the
> memory usage from various command line tools.

  I believe you should rather test master as any of the strbuf stages is
broken for you, and that there has been some issues with strbuf fixed
lately (not only the patch I asked you to try, but also an issue with
empty strubufs, and you _need_ that patch).

> GDB output
> ===========
> 
> Program received signal EXC_BAD_ACCESS, Could not access memory.
> Reason: KERN_INVALID_ADDRESS at address: 0x756c7463
> 0x90018d0c in szone_realloc ()
> (gdb) bt
> #0  0x90018d0c in szone_realloc ()
> #1  0x90018898 in realloc ()
> #2  0x0001f8c8 in strbuf_grow (sb=0x75ef8, extra=0) at git-compat-util.h:223
> #3  0x0001fc14 in read_line (sb=0x75ef8, fp=0xa0001b9c, term=10) at strbuf.c:107
> #4  0x00002da8 in read_next_command () at fast-import.c:1599
> #5  0x0000742c in cmd_new_commit () at fast-import.c:2118
> #6  0x00007e80 in main (argc=1, argv=0xbffff828) at fast-import.c:2390
> (gdb) frame 3
> #3  0x0001fc14 in read_line (sb=0x75ef8, fp=0xa0001b9c, term=10) at strbuf.c:107
> 107                     strbuf_grow(sb, 1);
> (gdb) print *sb
> $1 = {
>   alloc = 60,
>   len = 23,
>   eof = 0,
>   buf = 0x5020b0 "D windows/packages/cdc/ces/adm"
> }
> (gdb) frame 2
> #2  0x0001f8c8 in strbuf_grow (sb=0x75ef8, extra=0) at git-compat-util.h:223
> 223             void *ret = realloc(ptr, size);
> (gdb) print (char *) ptr
> $2 = 0x5020b0 "D windows/packages/cdc/ces/adm"
> (gdb) print (char *)ret
> $3 = 0xd <Address 0xd out of bounds>

  This does not make sense, ptr should be either valid or NULL.
> The output of `leaks'
> =================
> 
> Process 27075: 88 nodes malloced for 2335 KB
> Process 27075: 6 leaks for 624 total leaked bytes.
> Leak: 0x005012b0  size=208      string 'EOT'
> Leak: 0x00501e60  size=128      string 'EOT'
> Leak: 0x00501c80  size=128      string 'EOT'
> Leak: 0x00500f10  size=64       string 'EOT'
> Leak: 0x00501fa0  size=64       string 'EOT'
> Leak: 0x00501f60  size=32
>         0x00000000 0x02008000 0x00000000 0x01000000     ................
>         0x02000000 0x0000001a 0x00000000 0x00000000     ................

  What I _really_ need is that you run your test into valgrind (the easy
way is to rename git-fast-import into git-fast-import.bin, and make
git-fast-import be a shell-script doing:

  exec valgrind --log-file /tmp/git-fast-import git-fast-import "$@"

  and please send to me the /tmp/git-fast-import.<pid> from the one that
crashes.

  And please do that on the most recent tip of master so that you have a
(at least believe to be) full and sane strbuf series. Thanks.

-- 
·O·  Pierre Habouzit
··O                                                madcoder@debian.org
OOO                                                http://www.madism.org

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: git-fast-import crashes

[Shun Kei Leung]

>   What I _really_ need is that you run your test into valgrind (the easy
> way is to rename git-fast-import into git-fast-import.bin, and make
> git-fast-import be a shell-script doing:
...

valgrind is not yet ported to Mac OS X, sadly enough.

Regards,
Kevin Leung

Re: git-fast-import crashes

[Pierre Habouzit]

[-- Attachment #1: Type: text/plain, Size: 1045 bytes --]

On Tue, Oct 16, 2007 at 08:01:18AM +0000, Shun Kei Leung wrote:
> >   What I _really_ need is that you run your test into valgrind (the easy
> > way is to rename git-fast-import into git-fast-import.bin, and make
> > git-fast-import be a shell-script doing:
> ....
> 
> valgrind is not yet ported to Mac OS X, sadly enough.

  hmmm I see. hmmm isn't there _any_ chance that you can have access to
a linux box to see if that fails in the same conditions ? I mean I
believe qemu (or virtual pc or …) on your macos would be just fine.

  But this realloc() that returns 0xd looks fishy. I would be tempted to
think of a stack corruption, but the frame just before still looks
correct so I'm confused. And valgrind is definitely the tool to use for
stack smashing and other memory related corruptions :/

  Using gcc -Wstack-protector may also help.

-- 
·O·  Pierre Habouzit
··O                                                madcoder@debian.org
OOO                                                http://www.madism.org

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]