git binary size...

git binary size...

[Linus Torvalds]

Guess what the difference is here?

	[torvalds@g5 ~]$ du -sh bin/
	14M     bin/
	[torvalds@g5 ~]$ du -sh bin/
	5.8M    bin/

Give up?

In one case, "git" was compiled with the default options in the git 
Makefile. In the other one, the "-g" was removed.

Now, maybe this is extra visible with PowerPC (32-bit) binaries, and it's 
not as bad on x86, but it's still a bit distressing.

That "-g" doesn't buy users much of anything, and I doubt most developers 
care that deeply most of the time either (and can easily add it when they 
do care). It's left-over from long ago when it was much more useful.

So I'd suggest just removing it.

		Linus

Re: git binary size...

[Andreas Ericsson]

Linus Torvalds wrote:
> Guess what the difference is here?
> 
> 	[torvalds@g5 ~]$ du -sh bin/
> 	14M     bin/
> 	[torvalds@g5 ~]$ du -sh bin/
> 	5.8M    bin/
> 
> Give up?
> 
> In one case, "git" was compiled with the default options in the git 
> Makefile. In the other one, the "-g" was removed.
> 
> Now, maybe this is extra visible with PowerPC (32-bit) binaries, and it's 
> not as bad on x86, but it's still a bit distressing.
> 
> That "-g" doesn't buy users much of anything, and I doubt most developers 
> care that deeply most of the time either (and can easily add it when they 
> do care). It's left-over from long ago when it was much more useful.
> 
> So I'd suggest just removing it.
> 

I'd suggest adding

strip:
	strip $(PROGRAMS)

install: strip

to Makefile instead. That way people working on various git-tools won't 
have to remember to override the CFLAGS when debugging new stuff.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Re: git binary size...

[H. Peter Anvin]

Andreas Ericsson wrote:
> 
> I'd suggest adding
> 
> strip:
>     strip $(PROGRAMS)
> 
> install: strip
> 
> to Makefile instead. That way people working on various git-tools won't 
> have to remember to override the CFLAGS when debugging new stuff.
> 

I disagree with both of these.  Most users will not install via "make 
install", but rather via a package manager.  Package managers have long 
since dealt with this problem; in the case of rpm, the debug information 
is stripped off into a separate package.  Having "make install" strip 
would break that -- although defining $(STRIP) makes it a bit easier to 
deal with (STRIP=: make install).

However, if the concern is "the average user" I really think there is no 
reason do bother with this at all -- the average user should to "yum 
install git" or whatever is the equivalent for their distribution.  This 
ain't 1990.

	-hpa

Re: git binary size...

[Junio C Hamano]

Andreas Ericsson <ae@op5.se> writes:

> I'd suggest adding
>
> strip:
> 	strip $(PROGRAMS)
>
> install: strip
>
> to Makefile instead.

That sounds sane.  Binary distribution _might_ hate us if we did
so, though.  IIRC Debian wanted to build with -g and strip
before installation themselves.  I do not know how this
interacts with rpmbuild.

Re: git binary size...

[Linus Torvalds]

On Wed, 11 Jan 2006, Andreas Ericsson wrote:
> 
> strip:
> 	strip $(PROGRAMS)
> 
> install: strip

Well, that ends up shaving some more from the binaries, but at a much 
bigger cost than just removing "-g".

With stripped binaries, you can't really do _anything_. You get a 
core-file, and you're screwed. 

With non-stripped binaries you can at least see the function the SIGSEGV 
happened in, and you usually even get a half-way decent backtrace etc.

		Linus

Re: git binary size...

[Andreas Ericsson]

H. Peter Anvin wrote:
> 
> I disagree with both of these.  Most users will not install via "make 
> install", but rather via a package manager.  Package managers have long 
> since dealt with this problem; in the case of rpm, the debug information 
> is stripped off into a separate package.  Having "make install" strip 
> would break that -- although defining $(STRIP) makes it a bit easier to 
> deal with (STRIP=: make install).
> 

Fair point. So how about a strip: target for just stripping the binaries 
in the directory? That way one can do "make strip install", but 
everything else will work as always. Better yet, make it

strip:
	strip $(STRIP_OPTS) $(PROGRAMS)

so Linus can set STRIP_OPTS=--strip-debug and everybody's happy.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Re: git binary size...

[H. Peter Anvin]

Andreas Ericsson wrote:
> H. Peter Anvin wrote:
> 
>>
>> I disagree with both of these.  Most users will not install via "make 
>> install", but rather via a package manager.  Package managers have 
>> long since dealt with this problem; in the case of rpm, the debug 
>> information is stripped off into a separate package.  Having "make 
>> install" strip would break that -- although defining $(STRIP) makes it 
>> a bit easier to deal with (STRIP=: make install).
>>
> 
> Fair point. So how about a strip: target for just stripping the binaries 
> in the directory? That way one can do "make strip install", but 
> everything else will work as always. Better yet, make it
> 
> strip:
>     strip $(STRIP_OPTS) $(PROGRAMS)
> 
> so Linus can set STRIP_OPTS=--strip-debug and everybody's happy.
> 

The proper way to do this is:

strip:
	$(STRIP) $(STRIP_OPTS) $(PROGRAMS)

For "strip", this is frequently omitted, though.  Since "strip" is 
architecture-dependent, though, this is necessary for cross-compilation.

	-hpa

Re: git binary size...

[H. Peter Anvin]

Linus Torvalds wrote:
> 
> Well, that ends up shaving some more from the binaries, but at a much 
> bigger cost than just removing "-g".
> 
> With stripped binaries, you can't really do _anything_. You get a 
> core-file, and you're screwed. 
> 
> With non-stripped binaries you can at least see the function the SIGSEGV 
> happened in, and you usually even get a half-way decent backtrace etc.
> 

That's the really nice bit with the way e.g. RPM does it.  You don't 
have to have the debug information around most of the time, but you can 
do "yum install git-debuginfo" and suddenly you have it all -- and yet 
the binary is unchanged, so your random core file actually matches the 
debuginfo.

	-hpa

Re: git binary size...

[Andreas Ericsson]

H. Peter Anvin wrote:
> Andreas Ericsson wrote:
>>
>> strip:
>>     strip $(STRIP_OPTS) $(PROGRAMS)
>>
>> so Linus can set STRIP_OPTS=--strip-debug and everybody's happy.
>>
> 
> The proper way to do this is:
> 
> strip:
>     $(STRIP) $(STRIP_OPTS) $(PROGRAMS)
> 
> For "strip", this is frequently omitted, though.  Since "strip" is 
> architecture-dependent, though, this is necessary for cross-compilation.
> 

Damn! I almost made it through the entire day without learning anything, 
and now you come along and spoil it all.

Oh well, new game tomorrow. ;)

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Re: git binary size...

[Coywolf Qi Hunt]

2006/1/12, Linus Torvalds <torvalds@osdl.org>:
>
>
> On Wed, 11 Jan 2006, Andreas Ericsson wrote:
> >
> > strip:
> >       strip $(PROGRAMS)
> >
> > install: strip
>
> Well, that ends up shaving some more from the binaries, but at a much
> bigger cost than just removing "-g".
>
> With stripped binaries, you can't really do _anything_. You get a
> core-file, and you're screwed.

Are you sure?

gemini:~> file `which mke2fs`
/sbin/mke2fs: ELF 32-bit LSB executable, Intel 80386, version 1
(SYSV), for GNU/Linux 2.2.0, dynamically linked (uses shared libs),
stripped

gemini:~> file /lib/libext2fs.so.2
/lib/libext2fs.so.2: symbolic link to `libext2fs.so.2.4'

gemini:~> file /lib/libext2fs.so.2.4
/lib/libext2fs.so.2.4: ELF 32-bit LSB shared object, Intel 80386,
version 1 (SYSV), stripped


In gdb:

No symbol table is loaded.  Use the "file" command.
(gdb) bt
#0  0xb7f16445 in ext2fs_mark_generic_bitmap () from /lib/libext2fs.so.2
#1  0xb7f110ed in ext2fs_reserve_super_and_bgd () from /lib/libext2fs.so.2
#2  0xb7f18353 in ext2fs_initialize () from /lib/libext2fs.so.2
#3  0x0804b461 in ?? ()
#4  0xbf84b9ad in ?? ()
#5  0x00000000 in ?? ()

So with stripped binary, I still get the backtrace to locate the buggy
function. IMO, Debian packages are build with -g.

So I suggest to let git go with `-g and striped' like all other packages do.

And if we use gnu autoconf, which provides site config ability, we
could easily get (1) "-g, striped" for distros, (2) "not striped" for
Linus and (3) "-g3, -O0, no striped" for some git developers.

To make git not tight to Linux, but cross platform, consider autoconf.

>
> With non-stripped binaries you can at least see the function the SIGSEGV
> happened in, and you usually even get a half-way decent backtrace etc.
>

False, see above.

>                 Linus

If I missed something in Debian, correct me.

--
Coywolf Qi Hunt

Re: git binary size...

[Andreas Ericsson]

Coywolf Qi Hunt wrote:
> 2006/1/12, Linus Torvalds <torvalds@osdl.org>:
> 
>>
>>On Wed, 11 Jan 2006, Andreas Ericsson wrote:
>>
>>>strip:
>>>      strip $(PROGRAMS)
>>>
>>>install: strip
>>
>>Well, that ends up shaving some more from the binaries, but at a much
>>bigger cost than just removing "-g".
>>
>>With stripped binaries, you can't really do _anything_. You get a
>>core-file, and you're screwed.
> 
> 
> Are you sure?
> 
> gemini:~> file `which mke2fs`
> /sbin/mke2fs: ELF 32-bit LSB executable, Intel 80386, version 1
> (SYSV), for GNU/Linux 2.2.0, dynamically linked (uses shared libs),
> stripped
> 
> gemini:~> file /lib/libext2fs.so.2.4
> /lib/libext2fs.so.2.4: ELF 32-bit LSB shared object, Intel 80386,
> version 1 (SYSV), stripped
> 
> (gdb) bt
> #0  0xb7f16445 in ext2fs_mark_generic_bitmap () from /lib/libext2fs.so.2
> #1  0xb7f110ed in ext2fs_reserve_super_and_bgd () from /lib/libext2fs.so.2
> #2  0xb7f18353 in ext2fs_initialize () from /lib/libext2fs.so.2
> #3  0x0804b461 in ?? ()
> #4  0xbf84b9ad in ?? ()
> #5  0x00000000 in ?? ()
> 
> So with stripped binary, I still get the backtrace to locate the buggy
> function. IMO, Debian packages are build with -g.
> 

No, you don't. The last three stack-frames resolve to no symbol. 
Libraries always contain symbol names. You wouldn't be able to use them 
if they didn't, because the dynamic linker uses those symbols to look up 
the address of the function to call.

> 
> To make git not tight to Linux, but cross platform, consider autoconf.
> 

git is already fairly portable without the autoconf hackery. It's easy 
enough to move some of the conditional stuff out of the Makefile without 
autoconf, but it would still require GNU Make, so there's no real point 
in doing so.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Re: git binary size...

[H. Peter Anvin]

Andreas Ericsson wrote:
>>
>> To make git not tight to Linux, but cross platform, consider autoconf.
> 
> git is already fairly portable without the autoconf hackery. It's easy 
> enough to move some of the conditional stuff out of the Makefile without 
> autoconf, but it would still require GNU Make, so there's no real point 
> in doing so.
> 

The problem is that one really at some point end up reinventing a whole 
lot of autoconf.

Now, a lot of autoconf ugliness comes from two sources:

  - Not abstracting appropriately (#ifdef mess)
  - Not using GNU make features

Once you require GNU make, you can have a top-level Makefile and have 
configure generating MCONFIG and config.h, and have conventional 
dependencies on those rules.  Pretty much the whole autoconstipation 
should be confined to those two files.

I might eventually try to make a clean patch for autoconf with git, and 
hopefully show that when done correctly, it's probably cleaner than the 
increasing Makefile mess, plus it's automatic.

	-hpa

Re: git binary size...

[Linus Torvalds]

On Thu, 12 Jan 2006, Coywolf Qi Hunt wrote:
> >
> > With stripped binaries, you can't really do _anything_. You get a
> > core-file, and you're screwed.
> 
> Are you sure?

I'm sure.

> In gdb:
> 
> No symbol table is loaded.  Use the "file" command.
> (gdb) bt
> #0  0xb7f16445 in ext2fs_mark_generic_bitmap () from /lib/libext2fs.so.2

Note where the debug info comes from. It comes from the _library_.

> So with stripped binary, I still get the backtrace to locate the buggy
> function.

No you don't. With the _non-stripped_ library you get the backtrace. 
There's no backtrace at all for the binary itself.

		Linus

Re: git binary size...

[Linus Torvalds]

On Thu, 12 Jan 2006, Andreas Ericsson wrote:
> 
> git is already fairly portable without the autoconf hackery. It's easy enough
> to move some of the conditional stuff out of the Makefile without autoconf,
> but it would still require GNU Make, so there's no real point in doing so.

More importantly, autoconf is a worse hack than _any_ of the config hacks 
we do currently.

Repeat after me: "autoconf is crap".

		Linus

Re: git binary size...

[Johannes Schindelin]

Hi,

On Thu, 12 Jan 2006, Linus Torvalds wrote:

> Repeat after me: "autoconf is crap".

I tried to hold my breath for that mail, but you disappointed me. What 
took you so long?

Ciao,
Dscho

P.S.: Note that I am not at all detesting autoconf, but I know Linus does.

Re: git binary size...

[Linus Torvalds]

On Thu, 12 Jan 2006, Linus Torvalds wrote:
> 
> Repeat after me: "autoconf is crap".

.. which is not to say that some _other_ autoconf-like thing might not be 
good.

The problem I have with autoconf is that it adds absolutely horrendous 
#ifdef's etc all over the place, and the resulting makefile (and the 
config file itself) is just completely unreadable.

The reason autoconf sucks *ss is that it doesn't try to abstract out any 
of the differences between systems, it tries to basically "fix up" the 
differences.

A real abstraction library would be a lot more preferable than autoconf. 
It's kind of the way the git stuff works (ie using things like 
"gitstrcasestr()" and "gitfakemmap()"), but for many of the same reasons 
that autoconf never did a good job, git itself doesn't do a good job (it 
uses "#if" hackery to then do things like "#define mmap gitfakemmap").

But I think the git kind of hackish #ifdef thing is better than the 
_insitutionalized_ horrible autoconf hackery.

There are real abstraction layers out there, but they usually do a lot 
more than just simple autoconf things. They do full system abstraction, 
usually with support for graphical GUI stuff too: layers like Qt, 
wxVidgets, whatever..

Now THAT is a good approach. 

Sadly, for the git kind of area, I don't know of any sane toolkits like 
that. The "gnulib" project could have been something, but it has been 
corrupted by the autoconf disease, as far as I can tell.

Anyway, it _should_ be possible to make a nice "extended posix wrapper" 
with just a single

	#include "libposix.h"

and having a per-system "posix header" that resolves all the stupid header 
file differences, and a "posix library" that adds (or fixes) all the posix 
problems for each platform.

Instead, people still use autoconf ;(

                 Linus

Re: git binary size...

[H. Peter Anvin]

Linus Torvalds wrote:
> 
>>Repeat after me: "autoconf is crap".
> 
> .. which is not to say that some _other_ autoconf-like thing might not be 
> good.
> 
> The problem I have with autoconf is that it adds absolutely horrendous 
> #ifdef's etc all over the place, and the resulting makefile (and the 
> config file itself) is just completely unreadable.
> 
> The reason autoconf sucks *ss is that it doesn't try to abstract out any 
> of the differences between systems, it tries to basically "fix up" the 
> differences.
> 
> A real abstraction library would be a lot more preferable than autoconf. 
> It's kind of the way the git stuff works (ie using things like 
> "gitstrcasestr()" and "gitfakemmap()"), but for many of the same reasons 
> that autoconf never did a good job, git itself doesn't do a good job (it 
> uses "#if" hackery to then do things like "#define mmap gitfakemmap").
> 
> But I think the git kind of hackish #ifdef thing is better than the 
> _insitutionalized_ horrible autoconf hackery.
> 

You can use autoconf in this way, though.  See my previous post on the 
matter.

	-hpa

Re: git binary size...

[Jon Loeliger]

On Thu, 2006-01-12 at 12:13, Linus Torvalds wrote:

> More importantly, autoconf is a worse hack than _any_ of the config hacks 
> we do currently.
> 
> Repeat after me: "autoconf is crap"

Autoconf is also extraordinarily painful.

If there is actually enough configuration stuff needed,
would there be interest in a Kconfig-like mechanism?
I could spare time that into existence, perhaps...?

jdl