Git 1.3.2 on Solaris

Git 1.3.2 on Solaris

[Stefan Pfetzing]

Hi,

I've been trying to get git to work on the latest Solaris Express
release (with the help of NetBSD's pkgsrc).

It mostly miserabely fails because of common "shell commands" being
used with GNU options. (like xargs, diff, tr and prob. some more) On
my box (and thats AFAIK the default when you install gnu coreutils on
Solaris) the commands do have a g prefix.

So there are 2 possible solutions to get git working on Solaris.

1.  fix every single shellscript automatically during the build phase
2.  setup a dir which contains symlinks to the "right" binaries and
put that dir into PATH.

No matter what solution is chosen to be the best, I'm volunteering to
create a patch for it. :)

(although I personally prefer the second, because its easier...)

bye

Stefan
-- 
        http://www.dreamind.de/
Oroborus and Debian GNU/Linux Developer.

Re: Git 1.3.2 on Solaris

[Jason Riedy]

And "Stefan Pfetzing" writes:
 - I've been trying to get git to work on the latest Solaris Express
 - release (with the help of NetBSD's pkgsrc).

I've been using it on Solaris 8 and 9 with the GNU tools
in pkgsrc for quite a while, as well as on AIX with the
GNU tools available as modules (but I haven't compiled a
new AIX version for a month or two).

 - It mostly miserabely fails because of common "shell commands" being
 - used with GNU options. (like xargs, diff, tr and prob. some more) On
 - my box (and thats AFAIK the default when you install gnu coreutils on
 - Solaris) the commands do have a g prefix.

In your pkgsrc mk.conf, use:
GNU_PROGRAM_PREFIX=
GTAR_PROGRAM_PREFIX=

I tried your first suggestion (patch all the commands) back
in February.  It's pretty fragile against future changes, and
I wouldn't recommend it.

 - 2.  setup a dir which contains symlinks to the "right" binaries and
 - put that dir into PATH.

Setting a GIT_COMPAT_PATH in the Makefile and prepending
it to the path in git.c and git-sh-setup.sh might be more
sane.  A fragment like the following in git.c before adding
GIT_EXEC_PATH:
#ifdef GIT_COMPAT_PATH
	/* Search for sane external utilities */
	prepend_to_path(GIT_COMPAT_PATH, strlen(GIT_COMPAT_PATH));
#endif

And maybe in git-sh-setup.sh to help those of us who
use git-foo rather than git foo:
if [ ! -z "@GIT_COMPAT_PATH@" ] ; then
	PATH="@GIT_COMPAT_PATH@:${PATH}"
	export PATH
fi

Plus Makefile fun.

Jason

Re: Git 1.3.2 on Solaris

[Linus Torvalds]

[ Junio - see the "grep" issue ]

On Wed, 17 May 2006, Stefan Pfetzing wrote:
> 
> So there are 2 possible solutions to get git working on Solaris.
> 
> 1.  fix every single shellscript automatically during the build phase
> 2.  setup a dir which contains symlinks to the "right" binaries and
> put that dir into PATH.

If the biggest issue is git depending on some GNU extensions, I'd really 
suggest
 (a) install all the normal GNU binaries, and put them in the path before 
     git just to get it working (and don't try to change git at all)
 (b) help send in patches that just remove the dependency entirely.

I've been - on and off - trying to libify most of the core git sources, so 
that the shell scripts can be re-written to be just plain C. Most of the 
time it's not actually even a huge amount of work, it's just somewhat 
boring.

Writing them as C usually gets rid of any dependencies on any GNU tools, 
and hopefully even cygwin. For example, we got rid of one "xargs -0" in 
the development branch pretty recently, thanks to making "git grep" a 
built-in.

Of course, I don't think anybody tried the new "git grep" on Solaris, and 
I think the solaris "grep" lacks the "-H" flag, for example. But that 
should be easy to fix (for example, replace the use of "--" and "-H" with 
putting a "/dev/null" as the first filename).

I don't think it's worth it trying to add some compatibility layer for the 
shell-scripts. We really do want to get rid of them, and the more people 
that help, the merrier.

In many ways, the libification effort isn't even needed. It's perfectly ok 
to turn a stupid shell-script (and they really all _are_ pretty stupid) 
into a builtin-cmd.c C file that just does something really easy like a 
"fork + execve()" translation of the original shell script.

The complete libification will take some time, and in the meantime, a few 
silly C files that hard-code the shell logic is probably much preferable 
to using the shell and all the problems that involves (like the whole 
problem with quoting arguments - just _gone_ when you do it as a execve() 
in a simple C program).

So anybody can help with this. If you know shell (and the git 
shell-scripts aren't even _advanced_ shell), and know some basic C, you're 
all set to do a trivial conversion from one to the other. And when the 
libification gets further, your conversion will probably help that (ie 
maybe libificaiton isn't complete, but a _part_ of the thing can be 
written to use the library interfaces instead of spawning an external 
program).

There aren't _that_ many shell programs, and a lot of them are really 
really trivial (ie they parse the arguments, and then do just a couple of 
external git commands).

			Linus

Re: Git 1.3.2 on Solaris

[Jason Riedy]

And Linus Torvalds writes:
 - 
 - The complete libification will take some time, and in the meantime, a few 
 - silly C files that hard-code the shell logic is probably much preferable 
 - to using the shell and all the problems that involves (like the whole 
 - problem with quoting arguments - just _gone_ when you do it as a execve() 
 - in a simple C program).

But for recommending and using git on these systems _now_...
Simply translating the shell script into C with execs doesn't
help if you're execing one of the known problems, or if the
script has embedded, non-trivial Perl.  git-clone is the major
blocker; a trivial translation would be a great step but won't 
let people without GNU utilities clone repos.

Plus, alas, Perl modules and Python version drift can be a bit
of a problem on the same semi-pristine (or unmaintained, or
too-stable) systems, so shell isn't the only thing that needs to
go.  And that'll take a good deal of effort.

Note that my code snippets weren't a suggested patch.  I wouldn't
want the easy way out to impede progress on the right thing.

But some local installations may find it much easier to patch git
than to instruct users to change their utilities to match what git
expects, especially if users have old scripts that would break
if they changed their path globally.  Luckily, git makes it really
easy to keep those patches locally...

Jason

Re: Git 1.3.2 on Solaris

[Linus Torvalds]

On Tue, 16 May 2006, Jason Riedy wrote:
> 
> But for recommending and using git on these systems _now_...

Yes. For that, I would literally suggest having people install the GNU 
tools (and/or a recent enough perl) somewhere early in the path.

If you use the git wrapper, for example, you can already depend on the 
fact that it will prepend the git installation directory to the path, so 
while the GNU tools might not _normally_ be on the path, if you put them 
in the same directory as your git install, you'll automatically get them 
as long as you use the "git cmd" format (rather than the "git-cmd" 
format).

		Linus

Re: Git 1.3.2 on Solaris

[Ryan Anderson]

On Tue, May 16, 2006 at 08:26:24PM -0700, Jason Riedy wrote:
> Plus, alas, Perl modules and Python version drift can be a bit
> of a problem on the same semi-pristine (or unmaintained, or
> too-stable) systems, so shell isn't the only thing that needs to
> go.  And that'll take a good deal of effort.

The Perl used in core-git is pretty forgiving of older versions of Perl,
back to at least 5.6.  (Going back to 5.005.003 is rather painful,
however, to be honest.)

The only major tool I can think of that has embedded Perl in the shell
script is format-patch.  That could probably be redone in pure Perl if
it would help.

Re: Git 1.3.2 on Solaris

[Stefan Pfetzing]

Hi Linus,

2006/5/17, Linus Torvalds <torvalds@osdl.org>:
>
> On Tue, 16 May 2006, Jason Riedy wrote:
> >
> > But for recommending and using git on these systems _now_...
>
> Yes. For that, I would literally suggest having people install the GNU
> tools (and/or a recent enough perl) somewhere early in the path.
>
> If you use the git wrapper, for example, you can already depend on the
> fact that it will prepend the git installation directory to the path, so
> while the GNU tools might not _normally_ be on the path, if you put them
> in the same directory as your git install, you'll automatically get them
> as long as you use the "git cmd" format (rather than the "git-cmd"
> format).

Well I guess for my pkgsrc environment this won't work.
I already (quite some time ago) tried to have gnu coreutils, findutils and
diffutils installed without the g prefix.
This broke several things on NetBSD and on Solaris.

So I'd prefer a solution where one could set one flag for the Makefile of git,
and git would check for the g prefix, create somewhere a directory with
symlinks to the "real" gnu binaries and put it into $PATH upon startup of
every git c-program or shellscript.

I suggest having these gnu "tools" dependancies removed can only be a long
term goal.

bye

dreamind

P.S.: I had to re-sent this mail, somehow gmail did put html crap into it.
--
       http://www.dreamind.de/
Oroborus and Debian GNU/Linux Developer.

Re: Git 1.3.2 on Solaris

[Junio C Hamano]

Ryan Anderson <ryan@michonline.com> writes:

> The only major tool I can think of that has embedded Perl in the shell
> script is format-patch.  That could probably be redone in pure Perl if
> it would help.

Actually, that one is in the process of migrating all C.

Re: Git 1.3.2 on Solaris

[Junio C Hamano]

"Stefan Pfetzing" <stefan.pfetzing@gmail.com> writes:

> 1.  fix every single shellscript automatically during the build phase
> 2.  setup a dir which contains symlinks to the "right" binaries and
> put that dir into PATH.

You forgot 3.

  3.  rewrite scripts so that they would require only POSIX;
      for ones that do need GNU extended coreutils to do in
      shell, find other ways, perhaps rewriting the stuff in C.

I am not looking forward to do the g- prefix in the main
Makefile.  The approach to have symlink forest under gitexecdir
(<Pine.LNX.4.64.0605162047380.10823@g5.osdl.org> by Linus) is
more palatable, and I am not opposed to host a script to do so
under contrib/notgnu perhaps.

Re: Git 1.3.2 on Solaris

[Junio C Hamano]

Linus Torvalds <torvalds@osdl.org> writes:

> [ Junio - see the "grep" issue ]
> ...
> Of course, I don't think anybody tried the new "git grep" on Solaris,...

I haven't tried the new grep on Solaris myself, as the Solaris
box I have easy access is badly maintained (unmaintained is
probably a better wording).

> ...and 
> I think the solaris "grep" lacks the "-H" flag, for example. But that 
> should be easy to fix (for example, replace the use of "--" and "-H" with 
> putting a "/dev/null" as the first filename).

You mean like this, I presume.

But I think this approach breaks -L; I do not think Solaris
supports -L, so it does not matter there, but on platforms that
knows how to do -L it does.

-- >8 --
[PATCH] builtin-grep: give /dev/null at the beginning instead of -H

---
diff --git a/builtin-grep.c b/builtin-grep.c
index 66111de..ff3c1f7 100644
--- a/builtin-grep.c
+++ b/builtin-grep.c
@@ -453,7 +453,6 @@ static int external_grep(struct grep_opt
 
 	len = nr = 0;
 	push_arg("grep");
-	push_arg("-H");
 	if (opt->fixed)
 		push_arg("-F");
 	if (opt->linenum)
@@ -503,7 +502,7 @@ static int external_grep(struct grep_opt
 		push_arg("-e");
 		push_arg(p->pattern);
 	}
-	push_arg("--");
+	push_arg("/dev/null");
 
 	hit = 0;
 	argc = nr;

Re: Git 1.3.2 on Solaris

[Stefan Pfetzing]

Hi Junio,

2006/5/17, Junio C Hamano <junkio@cox.net>:
> "Stefan Pfetzing" <stefan.pfetzing@gmail.com> writes:
>
> > 1.  fix every single shellscript automatically during the build phase
> > 2.  setup a dir which contains symlinks to the "right" binaries and
> > put that dir into PATH.
>
> You forgot 3.
>
>   3.  rewrite scripts so that they would require only POSIX;
>       for ones that do need GNU extended coreutils to do in
>       shell, find other ways, perhaps rewriting the stuff in C.

Yes, thats right, but this can only be a long term goal, because I guess
this will take significantly longer. - even "tr" and "diff" behave different
on Solaris.

> I am not looking forward to do the g- prefix in the main
> Makefile.  The approach to have symlink forest under gitexecdir
> (<Pine.LNX.4.64.0605162047380.10823@g5.osdl.org> by Linus) is
> more palatable, and I am not opposed to host a script to do so
> under contrib/notgnu perhaps.

Hm, gitexecdir is also the path where git is installed, right? So if I'd
install git with pkgsrc it will be /usr/pkg/bin, right? - If so,
putting symlinks
there _will_ break pkgsrc.

bye

Stefan
-- 
       http://www.dreamind.de/
Oroborus and Debian GNU/Linux Developer.

Re: Git 1.3.2 on Solaris

[Junio C Hamano]

"Stefan Pfetzing" <stefan.pfetzing@gmail.com> writes:

> Hm, gitexecdir is also the path where git is installed, right? So if I'd
> install git with pkgsrc it will be /usr/pkg/bin, right? - If so,
> putting symlinks
> there _will_ break pkgsrc.

If you look at our Makefile, you will see bindir does not have
to be gitexecdir.  The suggestion by Linus is that you set
bindir to /usr/local/bin or whereever your distribution's
packaging scheme wants the locally installed software to be that
is on user's PATH, and gitexecdir to /usr/local/libexec/git
(again, whereever), _and_ have:

	ln -s /usr/bin/gtr /usr/local/libexec/git/tr
	ln -s /usr/bin/gxargs /usr/local/libexec/git/xargs
        ...

Then:

	(1) git and gitk are available in /usr/local/bin;

        (2) while git and gitk runs, /usr/local/libexec/git will
            be prepended to the PATH, so when they want xargs,
            they will get gxargs;

        (3) but your users will _not_ have /usr/local/libexec/git
            on their PATH, so when they type xargs they will get
            the one that barfs on -0 option.

and train your users and user's scripts to use the officially
sanctioned way to refer to git subprograms.  From interactive
sessions, say "git foo", not "git-foo".  If your script _really_
cares about extra exec git wrapper does, use "git --exec-path"
upfront in the script to obtain correct gitexecpath, export
GIT_EXEC_PATH environment variable with that value, and prepend
it to PATH so that it can find "git-foo" executable (you would
probably need to do both, so that git-foo can find git-bar and
its friends).


	

[PATCH] builtin-grep: workaround for non GNU grep.

[Junio C Hamano]

Some implementations do not know what to do with -H; define
NO_H_OPTION_IN_GREP when you build git if your grep lacks -H.

Most of the time, it can be worked around by prepending
/dev/null to the argument list, but that causes -L and -c to
slightly misbehave (they both expose /dev/null is given), so
when these options are given, do not run external grep that does
not understand -H.

Signed-off-by: Junio C Hamano <junkio@cox.net>
---

 Junio C Hamano <junkio@cox.net> writes:

 > But I think this approach breaks -L; I do not think Solaris
 > supports -L, so it does not matter there, but on platforms that
 > knows how to do -L it does.

 So this is an updated version.  I am not proud of the handling
 of the new Makefile variable, although I like the C code that
 does not need #ifdef thanks to it.

 Makefile       |   11 +++++++++++
 builtin-grep.c |   22 +++++++++++++++++++---
 2 files changed, 30 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index 9ba608c..c67108d 100644
--- a/Makefile
+++ b/Makefile
@@ -46,6 +46,8 @@ # Patrick Mauritz).
 #
 # Define NO_MMAP if you want to avoid mmap.
 #
+# Define NO_H_OPTION_IN_GREP if your grep does not understand -H.
+#
 # Define WITH_OWN_SUBPROCESS_PY if you want to use with python 2.3.
 #
 # Define NO_IPV6 if you lack IPv6 support and getaddrinfo().
@@ -444,6 +446,12 @@ ifdef NO_ACCURATE_DIFF
 	ALL_CFLAGS += -DNO_ACCURATE_DIFF
 endif
 
+ifdef NO_H_OPTION_IN_GREP
+	NO_H_OPTION_IN_GREP=1
+else
+	NO_H_OPTION_IN_GREP=0
+endif
+
 # Shell quote (do not use $(call) to accomodate ancient setups);
 
 SHA1_HEADER_SQ = $(subst ','\'',$(SHA1_HEADER))
@@ -526,6 +534,9 @@ git$X git.spec \
 %.o: %.S
 	$(CC) -o $*.o -c $(ALL_CFLAGS) $<
 
+builtin-grep.o: builtin-grep.c
+	$(CC) -o $*.o -c $(ALL_CFLAGS) -DNO_H_OPTION_IN_GREP=$(NO_H_OPTION_IN_GREP) $<
+
 exec_cmd.o: exec_cmd.c
 	$(CC) -o $*.o -c $(ALL_CFLAGS) '-DGIT_EXEC_PATH="$(gitexecdir_SQ)"' $<
 
diff --git a/builtin-grep.c b/builtin-grep.c
index 66111de..36512d8 100644
--- a/builtin-grep.c
+++ b/builtin-grep.c
@@ -453,7 +453,6 @@ static int external_grep(struct grep_opt
 
 	len = nr = 0;
 	push_arg("grep");
-	push_arg("-H");
 	if (opt->fixed)
 		push_arg("-F");
 	if (opt->linenum)
@@ -503,7 +502,13 @@ static int external_grep(struct grep_opt
 		push_arg("-e");
 		push_arg(p->pattern);
 	}
-	push_arg("--");
+
+	if (NO_H_OPTION_IN_GREP)
+		push_arg("/dev/null");
+	else {
+		push_arg("-H");
+		push_arg("--");
+	}
 
 	hit = 0;
 	argc = nr;
@@ -535,8 +540,19 @@ #ifdef __unix__
 	 * Use the external "grep" command for the case where
 	 * we grep through the checked-out files. It tends to
 	 * be a lot more optimized
+	 *
+	 * Some grep implementations do not understand -H nor --
+	 * but /dev/null can be used as a substitution in most
+	 * cases.
+	 *
+	 * However -L and -c would slightly misbehave (-L would
+	 * list /dev/null as a hit, and -c would report 0 hits
+	 * from /dev/null); so do not use the external one on
+	 * such platforms.
 	 */
-	if (!cached) {
+	if (!cached &&
+	    (!NO_H_OPTION_IN_GREP ||
+	     (!opt->count && !opt->unmatch_name_only))) {
 		hit = external_grep(opt, paths, cached);
 		if (hit >= 0)
 			return hit;
-- 
1.3.3.g8a24

Re: Git 1.3.2 on Solaris

[Stefan Pfetzing]

Hi Junio,

2006/5/17, Junio C Hamano <junkio@cox.net>:
> If you look at our Makefile, you will see bindir does not have
> to be gitexecdir.  The suggestion by Linus is that you set
> bindir to /usr/local/bin or whereever your distribution's
> packaging scheme wants the locally installed software to be that
> is on user's PATH, and gitexecdir to /usr/local/libexec/git
> (again, whereever), _and_ have:
>
>         ln -s /usr/bin/gtr /usr/local/libexec/git/tr
>         ln -s /usr/bin/gxargs /usr/local/libexec/git/xargs
>         ...

Nice, that looks like a solution, but will this also "fix" the tr usage for
the git tests? If so, I'll write a small shellscript to create the links and
so on, and test it on Solaris later today.

bye

dreamind
-- 
       http://www.dreamind.de/
Oroborus and Debian GNU/Linux Developer.

Re: [PATCH] builtin-grep: workaround for non GNU grep.

[Linus Torvalds]

No, please don't do it this way.

On Wed, 17 May 2006, Junio C Hamano wrote:
>
> +	 * Some grep implementations do not understand -H nor --
> +	 * but /dev/null can be used as a substitution in most
> +	 * cases.
> +	 *
> +	 * However -L and -c would slightly misbehave (-L would
> +	 * list /dev/null as a hit, and -c would report 0 hits
> +	 * from /dev/null); so do not use the external one on
> +	 * such platforms.
>  	 */
> -	if (!cached) {
> +	if (!cached &&
> +	    (!NO_H_OPTION_IN_GREP ||
> +	     (!opt->count && !opt->unmatch_name_only))) {
>  		hit = external_grep(opt, paths, cached);
>  		if (hit >= 0)
>  			return hit;

That's the ugliest test ever, and at all the wrong levels.

Just make "external_grep()" test for the cases that it cannot handle, and 
return -1. That's how it's designed to work.

		Linus

Re: Git 1.3.2 on Solaris

[Linus Torvalds]

On Wed, 17 May 2006, Stefan Pfetzing wrote:
> 
> So I'd prefer a solution where one could set one flag for the Makefile of git,
> and git would check for the g prefix, create somewhere a directory with
> symlinks to the "real" gnu binaries and put it into $PATH upon startup of
> every git c-program or shellscript.

So let me just quote the thing you quoted but apparently didn't read:

> > If you use the git wrapper, for example, you can already depend on the
> > fact that it will prepend the git installation directory to the path, so
> > while the GNU tools might not _normally_ be on the path, if you put them
> > in the same directory as your git install, you'll automatically get them
> > as long as you use the "git cmd" format (rather than the "git-cmd"
> > format).

There already _is_ such a directory. It's your "prefix=" directory plus 
"bin".

So what you can do is make sure you compile with

	make prefix=/my/git/installation/prefix

and then install the GNU tools in /my/git/installation/prefix/bin, and 
you're all set.

At most you might have to make some of the tests use "git xyzzy" instead 
of "git-xyzzy", and run "make install" before "make test". 

It wouldn't be wonderful, but hey, I've given alternatives (like using the 
GNU tools by default, or helping make git more portable in the first 
place). So it's a hack. 

		Linus

Re: Git 1.3.2 on Solaris

[Stefan Pfetzing]

Hi Linus,

2006/5/17, Linus Torvalds <torvalds@osdl.org>:
>
> So let me just quote the thing you quoted but apparently didn't read:

[snip]

I did read that.

> There already _is_ such a directory. It's your "prefix=" directory plus
> "bin".
>
> So what you can do is make sure you compile with
>
>         make prefix=/my/git/installation/prefix
>
> and then install the GNU tools in /my/git/installation/prefix/bin, and
> you're all set.

Ok, if I would do so, my prefix would be /usr/pkg, and the bindir would be
/usr/pkg/bin. So I would need to have an xargs and so on symlink in
/usr/pkg/bin.
But this is simply not acceptable, because it breaks other NetBSD
pkgsrc scripts.

Besides that, installing git to a different location is not an option
for me, because
I want to have git packaged by pkgsrc.

I suggest Junio's solution will work (gitexecdir) but I have to try
that later today.

> At most you might have to make some of the tests use "git xyzzy" instead
> of "git-xyzzy", and run "make install" before "make test".
>
> It wouldn't be wonderful, but hey, I've given alternatives (like using the
> GNU tools by default, or helping make git more portable in the first
> place). So it's a hack.

Yes I know, as far as I can, I'm willing to help with this.

bye

dreamind

-- 
       http://www.dreamind.de/
Oroborus and Debian GNU/Linux Developer.

Re: [PATCH] builtin-grep: workaround for non GNU grep.

[Bertrand Jacquin]

On 5/17/06, Junio C Hamano <junkio@cox.net> wrote:

> +ifdef NO_H_OPTION_IN_GREP
> +       NO_H_OPTION_IN_GREP=1
> +else
> +       NO_H_OPTION_IN_GREP=0
> +endif

> +       if (NO_H_OPTION_IN_GREP)
> +               push_arg("/dev/null");
> +       else {
> +               push_arg("-H");
> +               push_arg("--");
> +       }

Sorry, maybe a C code beginner question but while you define
NO_H_OPTION_IN_GREP in Makefile, why don't use a build time ``if''
instead of a runtime one ?

Like :

#if NO_H_OPTION_IN_GREP
               push_arg("/dev/null");
#else
               push_arg("-H");
               push_arg("--");
#fi

-- 
Beber
#e.fr@freenode

Re: Git 1.3.2 on Solaris

[Linus Torvalds]

On Wed, 17 May 2006, Stefan Pfetzing wrote:
> 
> Ok, if I would do so, my prefix would be /usr/pkg, and the bindir would be
> /usr/pkg/bin. So I would need to have an xargs and so on symlink in
> /usr/pkg/bin.
> But this is simply not acceptable, because it breaks other NetBSD
> pkgsrc scripts.

DON'T USE /usr/pkg then.

Use /usr/pkg/git-core/share/ or something that is normally not on your 
path.

And then install _just_ the "git" binary in /usr/pkg/bin.

That must be allowable by whatever solaris packaging rules: it's not like 
other projects don't have their own internal library files.

Then you install the GNU symlinks under that same

	/usr/pkg/git-core/share/bin

and you're all set. The only binary you can _see_ is "git", and when that 
executes any scripts or other git binaries, it will set up the path to 
include that magic hidden directory.

> Besides that, installing git to a different location is not an option
> for me, because I want to have git packaged by pkgsrc.

Now, I'm told pkgsrc is horrible, but it can't be so horrid as to not 
allow private directories?

		Linus

Re: Git 1.3.2 on Solaris

[Jason Riedy]

And Linus Torvalds writes:
 - 
 - Now, I'm told pkgsrc is horrible, but it can't be so horrid as to not 
 - allow private directories?

Works fine.
  1) Depend on the GNU tools through the buildlink, um, stuff.
  2) Add a config.mak via a local patch that sets gitexecprefix.
  3) Add another local patch that sets up links within that
     gitexecprefix to the GNU tools.  Remember to check if the
     GNU tools were installed without the silly g prefix.

And pkgsrc itself works just fine without the silly g prefix,
or at least does for me as a mere user (and as well as it does
work).  But if you intend on adding the package upstream, it'll
need something to cope with the g.  And pkgsrc handles local
patches...

Jason

Re: [PATCH] builtin-grep: workaround for non GNU grep.

[Junio C Hamano]

Linus Torvalds <torvalds@osdl.org> writes:

>> -	if (!cached) {
>> +	if (!cached &&
>> +	    (!NO_H_OPTION_IN_GREP ||
>> +	     (!opt->count && !opt->unmatch_name_only))) {
>>  		hit = external_grep(opt, paths, cached);
>>  		if (hit >= 0)
>>  			return hit;
>
> That's the ugliest test ever, and at all the wrong levels.
>
> Just make "external_grep()" test for the cases that it cannot handle, and 
> return -1. That's how it's designed to work.

Ah....  *BLUSH*  I was not thinking when I saw that "if (hit >= 0)"
stuff.  Yes, you made it to fall back.

Re: [PATCH] builtin-grep: workaround for non GNU grep.

[Junio C Hamano]

"Bertrand Jacquin" <beber.mailing@gmail.com> writes:

> Sorry, maybe a C code beginner question but while you define
> NO_H_OPTION_IN_GREP in Makefile, why don't use a build time ``if''
> instead of a runtime one ?
>
> Like :
>
> #if NO_H_OPTION_IN_GREP
>               push_arg("/dev/null");
> #else
>               push_arg("-H");
>               push_arg("--");
> #fi

Exactly because I wanted to avoid conditional compilation using
C preprocessor directive (#if).

Re: [PATCH] builtin-grep: workaround for non GNU grep.

[Linus Torvalds]

On Wed, 17 May 2006, Junio C Hamano wrote:
>
> "Bertrand Jacquin" <beber.mailing@gmail.com> writes:
> >
> > #if NO_H_OPTION_IN_GREP
> >               push_arg("/dev/null");
> > #else
> >               push_arg("-H");
> >               push_arg("--");
> > #fi
> 
> Exactly because I wanted to avoid conditional compilation using
> C preprocessor directive (#if).

I think this is portable and correct.

Of course, it still ignores the fact that not all grep's support some of 
the flags like -F/-L/-A/-C etc, but for those cases, the external grep 
itself will happily just say "unrecognized option -F" or similar.

So with this change, "git grep" should handle all the flags the native 
grep handles, which is really quite fine. We don't _need_ to expose 
anything more, and if you do want our extensions, you can get them with 
"--uncached" and an up-to-date index.

No configuration necessary, and we automatically take advantage of any 
native grep we have, if possible.

		Linus

---
diff --git a/builtin-grep.c b/builtin-grep.c
index 66111de..d09ddf0 100644
--- a/builtin-grep.c
+++ b/builtin-grep.c
@@ -453,7 +453,6 @@ static int external_grep(struct grep_opt
 
 	len = nr = 0;
 	push_arg("grep");
-	push_arg("-H");
 	if (opt->fixed)
 		push_arg("-F");
 	if (opt->linenum)
@@ -503,17 +502,35 @@ static int external_grep(struct grep_opt
 		push_arg("-e");
 		push_arg(p->pattern);
 	}
-	push_arg("--");
+
+	/*
+	 * To make sure we get the header printed out when we want it,
+	 * add /dev/null to the paths to grep.  This is unnecessary
+	 * (and wrong) with "-l" or "-L", which always print out the
+	 * name anyway.
+	 *
+	 * GNU grep has "-H", but this is portable.
+	 */
+	if (!opt->name_only && !opt->unmatch_name_only)
+		push_arg("/dev/null");
 
 	hit = 0;
 	argc = nr;
 	for (i = 0; i < active_nr; i++) {
 		struct cache_entry *ce = active_cache[i];
+		const char *name;
 		if (ce_stage(ce) || !S_ISREG(ntohl(ce->ce_mode)))
 			continue;
 		if (!pathspec_matches(paths, ce->name))
 			continue;
-		argv[argc++] = ce->name;
+		name = ce->name;
+		if (name[0] == '-') {
+			int len = ce_namelen(ce);
+			name = xmalloc(len + 3);
+			memcpy(name, "./", 2);
+			memcpy(name + 2, ce->name, len + 1);
+		}
+		argv[argc++] = name;
 		if (argc < MAXARGS)
 			continue;
 		hit += exec_grep(argc, argv);

Re: [PATCH] builtin-grep: workaround for non GNU grep.

[Junio C Hamano]

Linus Torvalds <torvalds@osdl.org> writes:

> I think this is portable and correct.
>
> Of course, it still ignores the fact that not all grep's support some of 
> the flags like -F/-L/-A/-C etc, but for those cases, the external grep 
> itself will happily just say "unrecognized option -F" or similar.
>
> So with this change, "git grep" should handle all the flags the native 
> grep handles, which is really quite fine. We don't _need_ to expose 
> anything more, and if you do want our extensions, you can get them with 
> "--uncached" and an up-to-date index.
>
> No configuration necessary, and we automatically take advantage of any 
> native grep we have, if possible.

This makes -c misbehave in a subtle way.

	git grep -c -e no-such-string-anywhere | head -n 1

But I do not think we care.

Re: [PATCH] builtin-grep: workaround for non GNU grep.

[Linus Torvalds]

On Wed, 17 May 2006, Junio C Hamano wrote:
> 
> This makes -c misbehave in a subtle way.
> 
> 	git grep -c -e no-such-string-anywhere | head -n 1
> 
> But I do not think we care.

Ahh, yes. That appears to be unfixable without using special GNU 
extensions.

I agree that we probably don't care, though.

		Linus

Re: Git 1.3.2 on Solaris

[Stefan Pfetzing]

Hi Jason,

2006/5/17, Jason Riedy <ejr@eecs.berkeley.edu>:
> And pkgsrc itself works just fine without the silly g prefix,
> or at least does for me as a mere user (and as well as it does
> work).  But if you intend on adding the package upstream, it'll
> need something to cope with the g.  And pkgsrc handles local
> patches...

Well I had some problems on NetBSD without the g prefix for the
gnu coreutils - since then I always used that prefix.

But now I have a completely different problem with the tests on
solaris. It seems on solaris access() always returns 0 if a file is
existant and the effective uid is 0.

so:
--- snip ---
#include <stdio.h>
#include <unistd.h>

int
main (int argc, char **argv)
{
  printf ("access: %d\n", access("/etc/motd", X_OK));
  return 0;
}
--- snap ---

will return 0 on solaris - when run as root, even though /etc/motd
is not executeable. This seems to break hooks on Solaris - but
I'm not sure if this is only a Solaris Express bug. (I have no Solaris
10 system to verify it)

bye

Stefan

-- 
       http://www.dreamind.de/
Oroborus and Debian GNU/Linux Developer.

Re: Git 1.3.2 on Solaris

[Jason Riedy]

And "Stefan Pfetzing" writes:
 -   printf ("access: %d\n", access("/etc/motd", X_OK));
[...]
 - will return 0 on solaris - when run as root, even though /etc/motd
 - is not executeable.

This is explicitly allowed by the SUS, even for non-root:
  http://www.opengroup.org/onlinepubs/000095399/functions/access.html
For non-root, some ACL systems could allow you to execute
the file even if there are no execute bits.  What a joy
ACLs are.  Or NFS uid mappings could play tricks on you,
or...  And as you've noticed, this kills [ -x ].  (Failing
to run the hooks in receive-pack.c is noisy but not fatal.
It's the shell scripts that stop.)

I think you're stuck.  To disable the hooks for all possible
users, OSes, file systems, etc., you need to remove them.

Or just don't run as root, and hope that the OS isn't 
completely insane.

BTW, ERR_RUN_COMMAND_EXEC is never returned.  Any failure
to exec will produce an exit code of 128 from die.  This
will be an issue when commit becomes a built-in, right?

Jason

Re: Git 1.3.2 on Solaris

[Stefan Pfetzing]

Hi Jason,

2006/5/23, Jason Riedy <ejr@eecs.berkeley.edu>:
> This is explicitly allowed by the SUS, even for non-root:
>   http://www.opengroup.org/onlinepubs/000095399/functions/access.html
> For non-root, some ACL systems could allow you to execute
> the file even if there are no execute bits.  What a joy
> ACLs are.  Or NFS uid mappings could play tricks on you,
> or...  And as you've noticed, this kills [ -x ].  (Failing
> to run the hooks in receive-pack.c is noisy but not fatal.
> It's the shell scripts that stop.)

Yup, but this breaks test t5400 - I think because of the post-update
hook is failing.

> I think you're stuck.  To disable the hooks for all possible
> users, OSes, file systems, etc., you need to remove them.
>
> Or just don't run as root, and hope that the OS isn't
> completely insane.

As non-root it works fine.

> BTW, ERR_RUN_COMMAND_EXEC is never returned.  Any failure
> to exec will produce an exit code of 128 from die.  This
> will be an issue when commit becomes a built-in, right?

Think so.

bye

Stefan

-- 
       http://www.dreamind.de/
Oroborus and Debian GNU/Linux Developer.

Re: Git 1.3.2 on Solaris

[Linus Torvalds]

On Mon, 22 May 2006, Jason Riedy wrote:

> And "Stefan Pfetzing" writes:
>  -   printf ("access: %d\n", access("/etc/motd", X_OK));
> [...]
>  - will return 0 on solaris - when run as root, even though /etc/motd
>  - is not executeable.
> 
> This is explicitly allowed by the SUS, even for non-root:

What kind of CRAP has Solaris become?

Yes, it's allowed. That doesn't mean that a quality implementation should 
do it.

SunOS used to be the best system around - it was, compared to others, 
_nice_ to work with. It wasn't about what was "allowed by the standards", 
that was the HP-SUX and AIX's excuses. It was the highest-quality 
implementation.

Now, Solaris had some serious problems early on (yes, I was there when 
they switched, and yes, I hated it), but I thought they had fixed their 
stuff long long ago.

Have Sun people forgotten the difference between "quality" and "crap that 
passes standards tests"? 

Not doing a reasonable job on "access()" is a joke. It's like VMS being 
"posix". Sure, it's the letter of the law, but it's still not _unix_.

Btw, even SuS says:

    "The sentence concerning appropriate privileges and execute permission 
     bits reflects the two possibilities implemented by historical 
     implementations when checking superuser access for X_OK.

     New implementations are discouraged from returning X_OK unless at 
     least one execution permission bit is set."

which clearly says "Solaris is CRAP" to me.

What the heck is going on? First the totally broken stdio that doesn't 
retry on EINTR, now access(). And these people think they can compete?

Somebody hit some Solaris engineers with a 2x4 clue-stick, please.

		Linus

Re: Git 1.3.2 on Solaris

[Edgar Toernig]

Linus Torvalds wrote:
>
> >  -   printf ("access: %d\n", access("/etc/motd", X_OK));
> > [...]
> >  - will return 0 on solaris - when run as root, even though /etc/motd
> >  - is not executeable.
> > 
> > This is explicitly allowed by the SUS, even for non-root:
> 
>      New implementations are discouraged from returning X_OK unless at 
>      least one execution permission bit is set."
> 
> which clearly says "Solaris is CRAP" to me.

Just for the record: firefox's download manager performs exactly this
test to decide whether you can 'open with' a file (pretty silly because
the test is done on the freshly downloaded file in the temp dir which
never has an x-bit set).  But I was hit by this effect on my system
which is - surprise surprise - Linux :-)   Ok, it's a pretty old one
with a 2.0 kernel and libc 5.  But nevertheless, access(2) is not the
right function to portably test the x-bit.

Ciao, ET.

Re: Git 1.3.2 on Solaris

[Linus Torvalds]

On Tue, 23 May 2006, Edgar Toernig wrote:
> 
> But I was hit by this effect on my system which is - surprise surprise - 
> Linux :-)  Ok, it's a pretty old one with a 2.0 kernel and libc 5.

Yes, we've had that bug too, and yes, I was hit by a clue-stick, and still 
have the bruise. That's how you teach people.

[ And how the heck does anybody still run 2.0, btw? ]

		Linus

Re: Git 1.3.2 on Solaris

[Jason Riedy]

And Linus Torvalds writes:
 - 
 - What kind of CRAP has Solaris become?

Become?  heh.  Check mount's output; it's "mountpoint
on device".  Always has been.  I think there might be
a reason why certain other OSes have eaten their lunch,
and it ain't the price.

 - It wasn't about what was "allowed by the standards", 
 - that was the HP-SUX and AIX's excuses.

No, AIX's excuse is that it's "dictated by these three
standards over here and disallowed by those two, so
clearly we have to support both behaviors depending
on some footnote in our 1e9 page manual."  wheee...

 - Have Sun people forgotten the difference between "quality" and "crap that 
 - passes standards tests"? 

As far as I've been told, Sun's more interested in
near-perfect backwards compatibility than external
standards tests.  It worked for Intel, right? ;)

 - Btw, even SuS says:
[...]
 -      New implementations are discouraged from returning X_OK unless at 
 -      least one execution permission bit is set."

Now there is one possible, cross-OS problem that I
haven't tested.  You can chmod a-x and then use
setfacl to grant one person execute access.  I'm not
sure if access works in that case, but that might
possibly just say that current ACL systems are crap.

Hmm.  Does access handle SELinux or the other systems?
That might be interesting for a public git server, but
I don't know enough about it.

 - Somebody hit some Solaris engineers with a 2x4 clue-stick, please.

I think you're targetting the wrong department...
Their hands are tied.

Jason, wondering if you could resist the SUS bait...

Re: Git 1.3.2 on Solaris

[Linus Torvalds]

On Tue, 23 May 2006, Jason Riedy wrote:
>
>  - Btw, even SuS says:
> [...]
>  -      New implementations are discouraged from returning X_OK unless at 
>  -      least one execution permission bit is set."
> 
> Now there is one possible, cross-OS problem that I
> haven't tested.  You can chmod a-x and then use
> setfacl to grant one person execute access.  I'm not
> sure if access works in that case, but that might
> possibly just say that current ACL systems are crap.

I absolutely agree. That is why the OS has a "access()" system call. It's 
there to ask the OS whether the file is executable (or readable/writable).

Otherwise, we'd just do

   static inline int executable(const char *path)
   {
	struct stat st;
	return  !stat(pathname, &st) &&
		S_ISREG(st.st_mode) &&
		(st.st_mode & 0111) != 0;
   }

and be done with it. But exactly because the OS knows what "executable" 
means, we ask it. We don't know about all the ACL's etc, the OS does.

(Similar issues are true for writability too - the file may be "writable" 
in the sense that the write permission bits are on, but if the filesystem 
is mounted read-only, it sure as hell ain't W_OK _anyway_).

> Hmm.  Does access handle SELinux or the other systems?

Yup. 

Modulo bugs, of course, but yes, access() on linux should check both 
POSIX ACL's and SELinux security extensions. It uses exactly the same 
code-paths that open()/execve() does: it uses the "vfs_permission()" 
function which is also what execve() uses.

Now, I think access() actually misses a no-exec mount (it doesn't seem to 
check MNT_NOEXEC for X_OK), and that looks like it might actually be a 
real bug.

		Linus

Re: Git 1.3.2 on Solaris

[Edgar Toernig]

Linus Torvalds wrote:
>
> [ And how the heck does anybody still run 2.0, btw? ]

Why not?  There was never the need for an upgrade.
All connected hardware is well supported.  Sure,
there are no binary packages for it and you have to
compile from sources but most apps can be made to
run on it (on the way you learn to hate autoconf).
And I wish the 2.6 system were as reliable as the
2.0 one ...

Ciao, ET.

Re: Git 1.3.2 on Solaris

[Linus Torvalds]

On Tue, 23 May 2006, Linus Torvalds wrote:
> 
> I absolutely agree. That is why the OS has a "access()" system call. It's 
> there to ask the OS whether the file is executable (or readable/writable).

Side note: I'm not claiming that "access()" is a wonderful thing. I do 
agree that we might want to replace it with something else inside of git, 
if only because of portability concerns.

So I'm really just ranting my normal "standards lawyerese doesn't mean 
much" rant..

(access() also has other isses: X_OK obviously means different things for 
directories and for regular files, so quite often you need to do a stat() 
on the thing _anyway_ just to determine whether it's "executable in the 
'execve()' sense" or "executable in the 'path lookup' sense").

		Linus

Re: Git 1.3.2 on Solaris

[Stefan Pfetzing]

Hi list,

2006/5/23, Stefan Pfetzing <stefan.pfetzing@gmail.com>:

> 2006/5/17, Jason Riedy <ejr@eecs.berkeley.edu>:
> > And pkgsrc itself works just fine without the silly g prefix,
> > or at least does for me as a mere user (and as well as it does
> > work).  But if you intend on adding the package upstream, it'll
> > need something to cope with the g.  And pkgsrc handles local
> > patches...
>
> Well I had some problems on NetBSD without the g prefix for the
> gnu coreutils - since then I always used that prefix.

...

Well finally - after some patching around access() and after figuring
out "merge" was broken in pkgsrc (and still is - I had to open a
problem report) - I got all tests to complete successfully.

bye

Stefan

P.S.: merge from devel/rcs uses /bin/diff3 on solaris which somehow
breaks merge.
-- 
       http://www.dreamind.de/
Oroborus and Debian GNU/Linux Developer.