MinGW port usable

MinGW port usable

[Johannes Sixt]

I'd like to point interested parties to the MinGW port at

 git://repo.or.cz/git/mingw.git

which is now in a usable state, methinks. I'm using it with git-gui and gitk 
on a (almost) production repository.

The README.MinGW at 
http://repo.or.cz/w/git/mingw.git?a=blob_plain;f=README.MinGW;hb=master gives 
an overview on the state. The transfer via native git protocol does not work 
and cannot be made working without major surgery(*). Theoretically, using 
netcat (nc) as GIT_PROXY_COMMAND should work, but not in my setup for some 
reason that I still do not know.

(*) The reason is that on Windows read() and write() cannot operate on 
descriptors created by socket(). A work-around is to implement a (threaded) 
proxy, but that's almost the same as if netcat were used as 
GIT_PROXY_COMMAND.

Junio, you may like to cherry-pick these two non-critical commits from my 
repository:

8c8bb94f94f1d972c7ffadda4744cf343fac6f34 gitk: Use peek-remote instead of 
ls-remote.
46580d2192d79a469f8b40fc1081db9116ad5517 Add a missing fork() error check.

-- Hannes

Re: MinGW port usable

[Shawn O. Pearce]

Johannes Sixt <johannes.sixt@telecom.at> wrote:
> I'd like to point interested parties to the MinGW port at
> 
>  git://repo.or.cz/git/mingw.git
> 
> which is now in a usable state, methinks. I'm using it with git-gui and gitk 
> on a (almost) production repository.

Well, that's pretty sweet that git-gui is working in a MinGW case.

Last night I pushed out a number of changes to avoid cygpath when
we don't have it in PATH (commit 20ddfcaa).  Were you testing with
that version, or something earlier?

-- 
Shawn.

Re: MinGW port usable

[Christian MICHON]

On 1/29/07, Shawn O. Pearce <spearce@spearce.org> wrote:
> Johannes Sixt <johannes.sixt@telecom.at> wrote:
> > I'd like to point interested parties to the MinGW port at
> >
> >  git://repo.or.cz/git/mingw.git
> >
> > which is now in a usable state, methinks. I'm using it with git-gui and gitk
> > on a (almost) production repository.
>
> Well, that's pretty sweet that git-gui is working in a MinGW case.
>
> Last night I pushed out a number of changes to avoid cygpath when
> we don't have it in PATH (commit 20ddfcaa).  Were you testing with
> that version, or something earlier?
>

my last compilation is less than 8 hours old, and I do not have cygwin
installed. What is it you want to test actually ?

git-gui and gitk have been working for a bit more than a week, if I
recall well. I'd like to see git-blame latest patch work with git-gui.

:)

-- 
Christian

Re: MinGW port usable

[Shawn O. Pearce]

Christian MICHON <christian.michon@gmail.com> wrote:
> my last compilation is less than 8 hours old, and I do not have cygwin
> installed. What is it you want to test actually ?

Try creating a desktop icon (Repository->Create Desktop Icon) and
start git-gui from the resulting .bat file.  It *should* come up
right on MinGW, but I'm only guessing here.

The other one that I'm curious about is if fetch/push/merge work.
merge requires git-merge right now, which is a shell script.
fetch is the same, but push is pure C so it should work.  But I
don't think the Tcl environment will make it into the child, which
means things like SSH_AUTH_SOCK don't get used.

But I really should just install the git-mingw port on one of
my windows systems and play with it.  If its faster than Cygwin
then it's worthwhile.
 
> git-gui and gitk have been working for a bit more than a week, if I
> recall well. I'd like to see git-blame latest patch work with git-gui.

The blame feature is in git-gui (went in last night).  It requires
the --incremental patches from Linus&Junio, which are now in git.git
master.  It also needs a big display, as the interface is horrid.  :)

-- 
Shawn.

Re: MinGW port usable

[Christian MICHON]

On 1/30/07, Shawn O. Pearce <spearce@spearce.org> wrote:
> Try creating a desktop icon (Repository->Create Desktop Icon) and
> start git-gui from the resulting .bat file.  It *should* come up
> right on MinGW, but I'm only guessing here.

ok noted for tomorrow's tests :)

> The other one that I'm curious about is if fetch/push/merge work.
> merge requires git-merge right now, which is a shell script.
> fetch is the same, but push is pure C so it should work.  But I
> don't think the Tcl environment will make it into the child, which
> means things like SSH_AUTH_SOCK don't get used.

I'll see what I can do.

> But I really should just install the git-mingw port on one of
> my windows systems and play with it.  If its faster than Cygwin
> then it's worthwhile.

one of the hassles with this mingw thingy is that you need to
install it, and whatever you compile gets inside it, like libs,
includes, etc... Whenever I come out with a stable product
that can be sfx'ed inside a 7zip executable, I do it, to avoid
installing/uninstalling or even modifying it.

it works like a charm with msys/mingw, and I guess git/gitk/git-gui
will all work in this way too. Once this port is ready for mass
testing, I guess we can either show how to compile it (actually
some patches are still needed, like no symlinks exist on
msys/mingw, so we should replace those links by bash scripts),
or provide a 7z sfx binary that can help <<pushing>> git
to the mass. On this last possibility I can help out...

your call, guys. :)

> The blame feature is in git-gui (went in last night).  It requires
> the --incremental patches from Linus&Junio, which are now in git.git
> master.  It also needs a big display, as the interface is horrid.  :)

I'll find a big one then.

-- 
Christian

Re: MinGW port usable

[Christian MICHON]

On 1/30/07, Shawn O. Pearce <spearce@spearce.org> wrote:
> Try creating a desktop icon (Repository->Create Desktop Icon) and
> start git-gui from the resulting .bat file.  It *should* come up
> right on MinGW, but I'm only guessing here.

ok, yes it's working. :)
but call gitk from inside it fail :(

"error in startup script"
( git-peek-remote is not working from git-gui, could be path related.
It's hard to tell why yet)

-- 
Christian

Re: MinGW port usable

[Christian MICHON]

On 1/30/07, Christian MICHON <christian.michon@gmail.com> wrote:
> On 1/30/07, Shawn O. Pearce <spearce@spearce.org> wrote:
> > Try creating a desktop icon (Repository->Create Desktop Icon) and
> > start git-gui from the resulting .bat file.  It *should* come up
> > right on MinGW, but I'm only guessing here.
>
> ok, yes it's working. :)
> but call gitk from inside it fail :(
>
> "error in startup script"
> ( git-peek-remote is not working from git-gui, could be path related.
> It's hard to tell why yet)
>
> --
> Christian
>

just to be more clear: when using the .bat shortcut, calling gitk
from within git-gui fails.

But if I call git-gui from a msys rxvt+bash, I can invoke all tools,
and calling gitk from within git-gui works.

I think this shortcut stuff may not be the best approach yet.

-- 
Christian

Re: MinGW port usable

[Johannes Schindelin]

Hi,

On Tue, 30 Jan 2007, Christian MICHON wrote:

> On 1/30/07, Christian MICHON <christian.michon@gmail.com> wrote:
> > On 1/30/07, Shawn O. Pearce <spearce@spearce.org> wrote:
> > > Try creating a desktop icon (Repository->Create Desktop Icon) and
> > > start git-gui from the resulting .bat file.  It *should* come up
> > > right on MinGW, but I'm only guessing here.
> > 
> > ok, yes it's working. :)
> > but call gitk from inside it fail :(
> > 
> > "error in startup script"
> > ( git-peek-remote is not working from git-gui, could be path related.
> > It's hard to tell why yet)
> > 
> > --
> > Christian
> > 
> 
> just to be more clear: when using the .bat shortcut, calling gitk
> > from within git-gui fails.

I guess you need to change the .bat shortcut, so that the tcl script is 
called via bash instead of directly from cmd.

Otherwise git-gui does not have a chance to find the MSYS environment.

Ciao,
Dscho

Re: MinGW port usable

[Shawn O. Pearce]

Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> On Tue, 30 Jan 2007, Christian MICHON wrote:
> > just to be more clear: when using the .bat shortcut, calling gitk
> > > from within git-gui fails.
> 
> I guess you need to change the .bat shortcut, so that the tcl script is 
> called via bash instead of directly from cmd.
> 
> Otherwise git-gui does not have a chance to find the MSYS environment.

Hmmph.  The [is_Cygwin] test within git-gui causes it to setup the
.bat shortcut using:

  $(cygpath --windows /bin/sh) --login -c 'git git-gui'

for exactly that reason.  I didn't think that with MSYS there was
environment to hand down through a shell's init scripts.  I thought
it was all coming from Windows itself.  Oops.

When starting gitk, git-gui assumes (rightly or wrongly) that gitk
is a Tcl/Tk script and launches it with the same Tcl interpreter
that is running git-gui.  I did that to bypass the unnecessary
fork/exec of a shell just to enter a wish process.  I had assumed
that would also work well on the MSYS case.  Apparently not.

Patches to git-gui would be welcome.  It will probably be a few weeks
before I can get an MSYS+Git configuration working that I can test
git-gui against.  I'm just swamped with too many tasks right now,
and this, although interesting enough to fix, is likely to take a
back seat.

-- 
Shawn.

Re: MinGW port usable

[Johannes Sixt]

Christian MICHON wrote:
> 
> On 1/30/07, Christian MICHON <christian.michon@gmail.com> wrote:
> > but call gitk from inside it fail :(
> >
> > "error in startup script"
> > ( git-peek-remote is not working from git-gui, could be path related.
> > It's hard to tell why yet)
> 
> just to be more clear: when using the .bat shortcut, calling gitk
> from within git-gui fails.

Calling gitk from git-gui works here, even if git-gui was invoked from
the .bat file. But maybe just because I have all git tools in my $PATH.
And I have this CMD script in my $PATH as well:

D:\>cat D:\MSYS\1.0\git\bin\gitk.cmd
@echo off
start wish84 D:/MSYS/1.0/git/bin/gitk %1 %2 %3 %4 %5 %6 %7 %8 %9

(and, btw, a similar script for git-gui).

-- Hannes

Re: MinGW port usable

[Johannes Sixt]

"Shawn O. Pearce" wrote:
> Last night I pushed out a number of changes to avoid cygpath when
> we don't have it in PATH (commit 20ddfcaa).  Were you testing with
> that version, or something earlier?

yes, I tested that version. The new incremental blame works, too.

-- Hannes

Re: MinGW port usable

[Linus Torvalds]

On Mon, 29 Jan 2007, Johannes Sixt wrote:

> I'd like to point interested parties to the MinGW port at
> 
>  git://repo.or.cz/git/mingw.git
> 
> which is now in a usable state, methinks. I'm using it with git-gui and gitk 
> on a (almost) production repository.

Can you elaborate about any performance differences, especially with 
cygwin? Does this make git perform reasonably on Windows, or are the 
performance problems as bad as with cygwin?

> Junio, you may like to cherry-pick these two non-critical commits from my 
> repository:
> 
> 8c8bb94f94f1d972c7ffadda4744cf343fac6f34 gitk: Use peek-remote instead of  ls-remote.

gitk really shouldn't use either. It should probably use

	git-show-ref -h -d

instead, which has the same output format (modulo a space vs tab issue), 
and is entirely local, with no silly unnecessary remote connext.

Something like this patch.

Does that work on mingw too?

		Linus
----
diff --git a/gitk b/gitk
index 031c829..6b4a4ac 100755
--- a/gitk
+++ b/gitk
@@ -309,9 +309,9 @@ proc readrefs {} {
     foreach v {tagids idtags headids idheads otherrefids idotherrefs} {
 	catch {unset $v}
     }
-    set refd [open [list | git ls-remote [gitdir]] r]
+    set refd [open [list | git show-ref -h -d] r]
     while {0 <= [set n [gets $refd line]]} {
-	if {![regexp {^([0-9a-f]{40})	refs/([^^]*)$} $line \
+	if {![regexp {^([0-9a-f]{40}) refs/([^^]*)$} $line \
 	    match id path]} {
 	    continue
 	}

Re: MinGW port usable

[Christian MICHON]

On 1/29/07, Linus Torvalds <torvalds@linux-foundation.org> wrote:
> Can you elaborate about any performance differences, especially with
> cygwin? Does this make git perform reasonably on Windows, or are the
> performance problems as bad as with cygwin?
>

I do not know about cygwin recently (else than it was slow), but I
cloned my packed git repository of all 117 2.6 linux kernels and
the "git diff" rocks on it.

git-unpack and git-repack was slow as hell, though. Maybe this
is the part to be improved. Mostly bad I/Os...

> gitk really shouldn't use either. It should probably use
>
>         git-show-ref -h -d
>
> instead, which has the same output format (modulo a space vs tab issue),
> and is entirely local, with no silly unnecessary remote connext.
>
> Something like this patch.
>
> Does that work on mingw too?
>

I'll try that tomorrow, but you'll have other reporting about it, for sure.

-- 
Christian

Re: MinGW port usable

[Johannes Schindelin]

Hi,

On Tue, 30 Jan 2007, Christian MICHON wrote:

> git-unpack and git-repack was slow as hell, though. Maybe this is the 
> part to be improved. Mostly bad I/Os...

How do you want to improve bad I/O on Windows?

Ciao,
Dscho

Re: MinGW port usable

[Christian MICHON]

On 1/30/07, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> How do you want to improve bad I/O on Windows?
>

upgrade to Vista ? :lol:

seriously, no clue here. I want first to fairly compare mingw-git with
git inside colinux inside a shared cofs directory.

-- 
Christian

Re: MinGW port usable

[Johannes Sixt]

Linus Torvalds wrote:
> Can you elaborate about any performance differences, especially with
> cygwin? Does this make git perform reasonably on Windows, or are the
> performance problems as bad as with cygwin?

Performance is horrid, although better than I initially expected. I
don't (want to) have cygwin installed to compare it (that's why I did
the MinGW port in the first place ;) Maybe others can comment on this?

The test suite takes ~15min to complete on my box, which is an oooold
800MHz with a slow HD. (And it's still Win2K, if that matters.)

Quite frankly, the "performance boost" that I expect from this port is
that it allows me the workflow that I want, instead of constantly
banging my head against the walls of CVS/SVN/you-name-it.

> gitk really shouldn't use either. It should probably use
> 
>         git-show-ref -h -d
> 
> instead, which has the same output format (modulo a space vs tab issue),
> and is entirely local, with no silly unnecessary remote connext.

Thanks, this works (I tested Junio's version). The problem with
ls-remote was that it is a shell script, and for some reason it dumps
its output into a cmd.exe that opens and closes right away instead of to
the pipe.

-- Hannes

Re: MinGW port usable

[Johannes Schindelin]

Hi,

On Mon, 29 Jan 2007, Linus Torvalds wrote:

> Can you elaborate about any performance differences, especially with 
> cygwin? Does this make git perform reasonably on Windows, or are the 
> performance problems as bad as with cygwin?

On the git repository, using MinGW:

	$ time git-rev-list next > /dev/null

	real    0m0.320s
	user    0m0.031s
	sys     0m0.000s

	$ time git-rev-list next > /dev/null

	real    0m0.320s
	user    0m0.015s
	sys     0m0.015s

	$ time git-rev-list next > /dev/null

	real    0m0.352s
	user    0m0.015s
	sys     0m0.000s

Using cygwin:

	$ time git-rev-list next > /dev/null

	real    0m0.750s
	user    0m0.421s
	sys     0m0.140s

	$ time git-rev-list next > /dev/null

	real    0m0.750s
	user    0m0.374s
	sys     0m0.233s

	$ time git-rev-list next > /dev/null

	real    0m0.750s
	user    0m0.374s
	sys     0m0.218s

IOW MinGW wins hands down with an average 0.33s vs cygwin with an average 
0.75s. (I am fairly certain that you must not trust sys times.)

I have no idea why, as the big show-stopper fork() is not even involved.

Now, for something different: git-log. On MinGW:

	$ time git-log next > /dev/null

	real    0m1.594s
	user    0m0.015s
	sys     0m0.015s

	$ time git-log next > /dev/null

	real    0m1.562s
	user    0m0.031s
	sys     0m0.000s

	$ time git-log next > /dev/null

	real    0m1.610s
	user    0m0.015s
	sys     0m0.015s

On Cygwin:

	$ time git-log next > /dev/null

	real    0m2.063s
	user    0m0.827s
	sys     0m1.031s

	$ time git-log next > /dev/null

	real    0m2.085s
	user    0m0.780s
	sys     0m1.061s

	$ time git-log next > /dev/null

	real    0m2.047s
	user    0m0.686s
	sys     0m1.171s

That's a bit less obvious: 1.589s for MinGW vs. 2.065s for Cygwin.

These are all hot-cache numbers, since the cold-cache performance 
of Windows is not easily triggered. (FWIW the first git-log on Cygwin -- 
which was cold-cache -- took over 10 seconds.)

Ciao,
Dscho

Re: MinGW port usable

[Shawn O. Pearce]

Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> On the git repository, using MinGW:
> Using cygwin:
> 
> IOW MinGW wins hands down with an average 0.33s vs cygwin with an average 
> 0.75s. (I am fairly certain that you must not trust sys times.)

Just curious, but were these tests run with mmap(), or NO_MMAP?

Clearly, for a packed repository most of the overhead should be
in getting to the pack data itself (as everything else is pure
userspace).

-- 
Shawn.

Re: MinGW port usable

[Johannes Schindelin]

Hi,

On Tue, 30 Jan 2007, Shawn O. Pearce wrote:

> Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > On the git repository, using MinGW:
> > Using cygwin:
> > 
> > IOW MinGW wins hands down with an average 0.33s vs cygwin with an average 
> > 0.75s. (I am fairly certain that you must not trust sys times.)
> 
> Just curious, but were these tests run with mmap(), or NO_MMAP?

Both with NO_MMAP=YesPlease.

> Clearly, for a packed repository most of the overhead should be in 
> getting to the pack data itself (as everything else is pure userspace).

Right. That's why I did the test on the _same_ repository. (Not just both 
git repositories, but in the same path on the same machine.)

That said, I find the numbers quite convincing. On a (much weaker 
equipped) Linux bux, it takes 0.65s and 1.2s, respectively.

Ciao,
Dscho

Re: MinGW port usable

[Shawn O. Pearce]

Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> On Tue, 30 Jan 2007, Shawn O. Pearce wrote:
> > Just curious, but were these tests run with mmap(), or NO_MMAP?
> 
> Both with NO_MMAP=YesPlease.

I wonder what the difference is between NO_MMAP=Yes and NO_MMAP=
on Windows.  This is something I've never tried, but probably should
do.  I run about ~25 developers on NO_MMAP= on Windows (all NTFS)
without issue, but now I'm curious if there's an actual performance
difference for some operations.  There's no real rationale behind
my NO_MMAP= setting, other than that's how we've always used it...
 
> That said, I find the numbers quite convincing. On a (much weaker 
> equipped) Linux bux, it takes 0.65s and 1.2s, respectively.

The time difference between the MinGW and Cygwin ports of Git
is certainly an interesting one, and says good things about the
MinGW work.

We're paying a price for POSIX compatibility on Windows, and its a
pretty high one it seems.  I can't blame the Cygwin folks, they've
done a great job at trying to make the most of VMS.  But sometimes
you just need Linux (or *BSD, or Solaris).  :-)

-- 
Shawn.

Re: MinGW port usable

[Johannes Schindelin]

Hi,

On Tue, 30 Jan 2007, Shawn O. Pearce wrote:

> Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > On Tue, 30 Jan 2007, Shawn O. Pearce wrote:
> > > Just curious, but were these tests run with mmap(), or NO_MMAP?
> > 
> > Both with NO_MMAP=YesPlease.
> 
> I wonder what the difference is between NO_MMAP=Yes and NO_MMAP=
> on Windows.

Cygwin:

	$ time git-rev-list next >/dev/null

	real    0m0.812s
	user    0m0.358s
	sys     0m0.203s

	$ time git-rev-list next >/dev/null

	real    0m0.797s
	user    0m0.358s
	sys     0m0.234s

	$ time git-rev-list next >/dev/null

	real    0m0.845s
	user    0m0.390s
	sys     0m0.265s

and

	$ time git-log next >/dev/null

	real    0m2.094s
	user    0m0.718s
	sys     0m1.077s

	$ time git-log next >/dev/null

	real    0m2.078s
	user    0m0.718s
	sys     0m1.062s

	$ time git-log next >/dev/null

	real    0m2.124s
	user    0m0.796s
	sys     0m1.046s

IOW, the numbers are slightly worse (!) than with mmap().

MinGW does not even have mmap().

Ciao,
Dscho

Re: MinGW port usable

[Shawn O. Pearce]

Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > I wonder what the difference is between NO_MMAP=Yes and NO_MMAP=
> > on Windows.
> Cygwin:
> 	real    0m0.812s
> and
> 	real    0m2.094s
> 
> IOW, the numbers are slightly worse (!) than with mmap().

Slightly?  That's double the time!
 
> MinGW does not even have mmap().

But Windows has something almost there.  I just read a flame war
thread about implementing mmap in libiberty for MinGW by stealing
source from Cygwin, and how the FSF might feel about someone's
dirty feet after playing rugby.  Yea... Google is good.  :)

-- 
Shawn.

Re: MinGW port usable

[Johannes Schindelin]

Hi,

On Tue, 30 Jan 2007, Shawn O. Pearce wrote:

> Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > > I wonder what the difference is between NO_MMAP=Yes and NO_MMAP=
> > > on Windows.
> > Cygwin:
> > 	real    0m0.812s
> > and
> > 	real    0m2.094s
> > 
> > IOW, the numbers are slightly worse (!) than with mmap().
> 
> Slightly?  That's double the time!

Nope. What I meant is this: rev-list takes 0.8s without NO_MMAP, while it 
takes 0.75s with NO_MMAP, and similarly log takes 2.09s without NO_MMAP, 
while it takes 2.065 with NO_MMAP.

> > MinGW does not even have mmap().
> 
> But Windows has something almost there.

Yeah, CreateFile(). Very intuitive name.

>  I just read a flame war thread about implementing mmap in libiberty for 
> MinGW by stealing source from Cygwin, and how the FSF might feel about 
> someone's dirty feet after playing rugby.

;-)

> Yea... Google is good.  :)

Well, I had _horrible_ results in the last two weeks, slowly getting 
better.

Ciao,
Dscho

Re: MinGW port usable

[Daniel Barkalow]

On Mon, 29 Jan 2007, Johannes Sixt wrote:

> (*) The reason is that on Windows read() and write() cannot operate on 
> descriptors created by socket(). A work-around is to implement a (threaded) 
> proxy, but that's almost the same as if netcat were used as 
> GIT_PROXY_COMMAND.

Can you do

#define read(fd, buffer, len) recv(fd, buffer, len, 0)
#define write(fd, buffer, len) send(fd, buffer, len, 0)

in the appropriate file?

	-Daniel
*This .sig left intentionally blank*

Re: MinGW port usable

[Johannes Sixt]

Daniel Barkalow wrote:
> 
> On Mon, 29 Jan 2007, Johannes Sixt wrote:
> 
> > (*) The reason is that on Windows read() and write() cannot operate on
> > descriptors created by socket(). A work-around is to implement a (threaded)
> > proxy, but that's almost the same as if netcat were used as
> > GIT_PROXY_COMMAND.
> 
> Can you do
> 
> #define read(fd, buffer, len) recv(fd, buffer, len, 0)
> #define write(fd, buffer, len) send(fd, buffer, len, 0)
> 
> in the appropriate file?

I doubt that recv and send can operate on regular file descriptors, as
opened by _pipe(), open(), can they?

-- Hannes

Re: MinGW port usable

[Andreas Ericsson]

Johannes Sixt wrote:
> Daniel Barkalow wrote:
>> On Mon, 29 Jan 2007, Johannes Sixt wrote:
>>
>>> (*) The reason is that on Windows read() and write() cannot operate on
>>> descriptors created by socket(). A work-around is to implement a (threaded)
>>> proxy, but that's almost the same as if netcat were used as
>>> GIT_PROXY_COMMAND.
>> Can you do
>>
>> #define read(fd, buffer, len) recv(fd, buffer, len, 0)
>> #define write(fd, buffer, len) send(fd, buffer, len, 0)
>>
>> in the appropriate file?
> 
> I doubt that recv and send can operate on regular file descriptors, as
> opened by _pipe(), open(), can they?
> 

Hence "in the appropriate file", I guess.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Re: MinGW port usable

[Johannes Schindelin]

Hi,

On Tue, 30 Jan 2007, Johannes Sixt wrote:

> Daniel Barkalow wrote:
> > 
> > On Mon, 29 Jan 2007, Johannes Sixt wrote:
> > 
> > > (*) The reason is that on Windows read() and write() cannot operate on
> > > descriptors created by socket(). A work-around is to implement a (threaded)
> > > proxy, but that's almost the same as if netcat were used as
> > > GIT_PROXY_COMMAND.
> > 
> > Can you do
> > 
> > #define read(fd, buffer, len) recv(fd, buffer, len, 0)
> > #define write(fd, buffer, len) send(fd, buffer, len, 0)
> > 
> > in the appropriate file?
> 
> I doubt that recv and send can operate on regular file descriptors, as
> opened by _pipe(), open(), can they?

They can't. Thus, you'd break at least inetd mode.

Of course, you really could start a thread which gets two pipe()d fds, and 
the proper socket, and just shuffles things back & forth. Not that hard.

Ciao,
Dscho

Re: MinGW port usable

[Daniel Barkalow]

On Tue, 30 Jan 2007, Johannes Sixt wrote:

> Daniel Barkalow wrote:
> > 
> > On Mon, 29 Jan 2007, Johannes Sixt wrote:
> > 
> > > (*) The reason is that on Windows read() and write() cannot operate on
> > > descriptors created by socket(). A work-around is to implement a (threaded)
> > > proxy, but that's almost the same as if netcat were used as
> > > GIT_PROXY_COMMAND.
> > 
> > Can you do
> > 
> > #define read(fd, buffer, len) recv(fd, buffer, len, 0)
> > #define write(fd, buffer, len) send(fd, buffer, len, 0)
> > 
> > in the appropriate file?
> 
> I doubt that recv and send can operate on regular file descriptors, as
> opened by _pipe(), open(), can they?

I don't think so, but I think it should be possible to make 
packet_write/packet_read always get a socket, by calling receive-pack and 
upload-pack with a socket pair instead of a pair of pipes. Actually, IIRC, 
this would be beneficial for making programs not leak file descriptors or 
double-close them, because there's already the issue on *nix that it isn't 
consistant whether both directions are the same file descriptor.

	-Daniel
*This .sig left intentionally blank*

Re: MinGW port usable

[Johannes Schindelin]

Hi,

On Tue, 30 Jan 2007, Daniel Barkalow wrote:

> On Tue, 30 Jan 2007, Johannes Sixt wrote:
> 
> > Daniel Barkalow wrote:
> > > 
> > > On Mon, 29 Jan 2007, Johannes Sixt wrote:
> > > 
> > > > (*) The reason is that on Windows read() and write() cannot operate on
> > > > descriptors created by socket(). A work-around is to implement a (threaded)
> > > > proxy, but that's almost the same as if netcat were used as
> > > > GIT_PROXY_COMMAND.
> > > 
> > > Can you do
> > > 
> > > #define read(fd, buffer, len) recv(fd, buffer, len, 0)
> > > #define write(fd, buffer, len) send(fd, buffer, len, 0)
> > > 
> > > in the appropriate file?
> > 
> > I doubt that recv and send can operate on regular file descriptors, as
> > opened by _pipe(), open(), can they?
> 
> I don't think so, but I think it should be possible to make 
> packet_write/packet_read always get a socket, by calling receive-pack and 
> upload-pack with a socket pair instead of a pair of pipes.

As I already mentioned in this thread, that would break inetd support.

Ciao,
Dscho

Re: MinGW port usable

[Daniel Barkalow]

On Tue, 30 Jan 2007, Johannes Schindelin wrote:

> Hi,
> 
> On Tue, 30 Jan 2007, Daniel Barkalow wrote:
> 
> > On Tue, 30 Jan 2007, Johannes Sixt wrote:
> > 
> > > Daniel Barkalow wrote:
> > > > 
> > > > On Mon, 29 Jan 2007, Johannes Sixt wrote:
> > > > 
> > > > > (*) The reason is that on Windows read() and write() cannot operate on
> > > > > descriptors created by socket(). A work-around is to implement a (threaded)
> > > > > proxy, but that's almost the same as if netcat were used as
> > > > > GIT_PROXY_COMMAND.
> > > > 
> > > > Can you do
> > > > 
> > > > #define read(fd, buffer, len) recv(fd, buffer, len, 0)
> > > > #define write(fd, buffer, len) send(fd, buffer, len, 0)
> > > > 
> > > > in the appropriate file?
> > > 
> > > I doubt that recv and send can operate on regular file descriptors, as
> > > opened by _pipe(), open(), can they?
> > 
> > I don't think so, but I think it should be possible to make 
> > packet_write/packet_read always get a socket, by calling receive-pack and 
> > upload-pack with a socket pair instead of a pair of pipes.
> 
> As I already mentioned in this thread, that would break inetd support.

I was actually thinking of only using recv/send on mingw. So the rule 
could be: if git sets up the connection to a pkt_line-user itself, the 
connection is a socket; otherwise it might be a pair of pipes; if you're 
on mingw, pkt_line uses recv/send. Then everything should work except for 
inetd on mingw, and I don't think that's a plausible combination anyway.

	-Daniel
*This .sig left intentionally blank*

Re: MinGW port usable

[Johannes Sixt]

Daniel Barkalow wrote:
> I was actually thinking of only using recv/send on mingw. So the rule
> could be: if git sets up the connection to a pkt_line-user itself, the
> connection is a socket; otherwise it might be a pair of pipes; if you're
> on mingw, pkt_line uses recv/send. Then everything should work except for
> inetd on mingw, and I don't think that's a plausible combination anyway.

Except that it won't work. Because in the same program, say git-fetch-pack,
the packet_*() functions, where you intend to replace read/write by
recv/send, are sometimes called with pipes, and sometimes with sockets,
even on MinGW.

Call it object oriented programming with polymorphic behavior if you like ;)

-- Hannes

Re: MinGW port usable

[Junio C Hamano]

Johannes Sixt <johannes.sixt@telecom.at> writes:

> Junio, you may like to cherry-pick these two non-critical commits from my 
> repository:
>
> 8c8bb94f94f1d972c7ffadda4744cf343fac6f34 gitk: Use peek-remote instead of 
> ls-remote.
> 46580d2192d79a469f8b40fc1081db9116ad5517 Add a missing fork() error check.

Thanks; cherry-picked the fork() one.

Regarding gitk, I think the following patch would be better
regardless of the platform.

-- >8 --
[PATCH] gitk: Use show-ref instead of ls-remote

It used to be ls-remote on self was the only easy way to grab
the ref information.  Now we have show-ref which does not
involve fork and IPC, so use it.

Signed-off-by: Junio C Hamano <junkio@cox.net>
---
diff --git a/gitk b/gitk
index 31d0aad..67e6a64 100755
--- a/gitk
+++ b/gitk
@@ -309,9 +309,9 @@ proc readrefs {} {
     foreach v {tagids idtags headids idheads otherrefids idotherrefs} {
 	catch {unset $v}
     }
-    set refd [open [list | git ls-remote [gitdir]] r]
+    set refd [open [list | git show-ref] r]
     while {0 <= [set n [gets $refd line]]} {
-	if {![regexp {^([0-9a-f]{40})	refs/([^^]*)$} $line \
+	if {![regexp {^([0-9a-f]{40}) refs/([^^]*)$} $line \
 	    match id path]} {
 	    continue
 	}

Re: MinGW port usable

[H. Peter Anvin]

Johannes Sixt wrote:
> 
> The README.MinGW at 
> http://repo.or.cz/w/git/mingw.git?a=blob_plain;f=README.MinGW;hb=master gives 
> an overview on the state. The transfer via native git protocol does not work 
> and cannot be made working without major surgery(*). Theoretically, using 
> netcat (nc) as GIT_PROXY_COMMAND should work, but not in my setup for some 
> reason that I still do not know.
> 
> (*) The reason is that on Windows read() and write() cannot operate on 
> descriptors created by socket(). A work-around is to implement a (threaded) 
> proxy, but that's almost the same as if netcat were used as 
> GIT_PROXY_COMMAND.
> 

Actually, I believe it can for the NT series kernels (at least 2000 or 
later, not sure about the earlier ones), but not for the DOS-based ones.

The trick is to use _open_osfhandle() to convert the file handle (a 
WinAPI construct) to a file descriptor (which in Windows is a construct 
of the C library.)

In 9x/ME, a socket isn't represented by a file handle, so 
_open_osfhandle() doesn't work on a socket object.

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vccore98/html/_crt__open_osfhandle.asp

	-hpa

Re: MinGW port usable

[Johannes Sixt]

"H. Peter Anvin" wrote:
> 
> Johannes Sixt wrote:
> > (*) The reason is that on Windows read() and write() cannot operate on
> > descriptors created by socket(). A work-around is to implement a (threaded)
> > proxy, but that's almost the same as if netcat were used as
> > GIT_PROXY_COMMAND.
> >
> 
> Actually, I believe it can for the NT series kernels (at least 2000 or
> later, not sure about the earlier ones), but not for the DOS-based ones.
> 
> The trick is to use _open_osfhandle() to convert the file handle (a
> WinAPI construct) to a file descriptor (which in Windows is a construct
> of the C library.)

I tried this, but it doesn't seem to work. I get an EINVAL at the first
write() to the socket. I conclude that the things returned by socket()
are not WinAPI file handles that are valid for WriteFile(). :(

-- Hannes

Re: MinGW port usable

[H. Peter Anvin]

Johannes Sixt wrote:
> "H. Peter Anvin" wrote:
>> Johannes Sixt wrote:
>>> (*) The reason is that on Windows read() and write() cannot operate on
>>> descriptors created by socket(). A work-around is to implement a (threaded)
>>> proxy, but that's almost the same as if netcat were used as
>>> GIT_PROXY_COMMAND.
>>>
>> Actually, I believe it can for the NT series kernels (at least 2000 or
>> later, not sure about the earlier ones), but not for the DOS-based ones.
>>
>> The trick is to use _open_osfhandle() to convert the file handle (a
>> WinAPI construct) to a file descriptor (which in Windows is a construct
>> of the C library.)
> 
> I tried this, but it doesn't seem to work. I get an EINVAL at the first
> write() to the socket. I conclude that the things returned by socket()
> are not WinAPI file handles that are valid for WriteFile(). :(
> 

Except they are (for NT-based Windows), so you're doing something goofy. 
  This is a widely used construct, so it can't be that broken.

	-hpa

Re: MinGW port usable

[H. Peter Anvin]

H. Peter Anvin wrote:
> 
> Except they are (for NT-based Windows), so you're doing something goofy. 
>  This is a widely used construct, so it can't be that broken.
> 

Erf... I dug through this, and it seems that WriteFile only works on a 
socket if it has an OVERLAPPED argument now, because the socket is 
opened for overlapping I/O.  This must be new behaviour in XP-SP2, 
because this definitely wasn't the case when I last played with this 
stuff back in 2003.  The Internet is full of people using this 
technique, but I haven't found a way to get a socket which is *not* 
opened for overlapping I/O.

How typical of Microsoft to break an incredibly powerful unified 
paradigm, sort-of repair it, and then break it again.  There doesn't 
seem to be an obvious way to repair this, either, since MS DLLs won't 
let you override for example the write() function as called from inside 
the C runtime DLL.

"Some people are just a total waste of carbon..."

	-hpa

Re: MinGW port usable

[Davide Libenzi]

On Fri, 2 Feb 2007, H. Peter Anvin wrote:

> The Internet is full of people using this technique, but I haven't found a way
> to get a socket which is *not* opened for overlapping I/O.

Doesn't WSASocket() w/out WSA_FLAG_OVERLAPPED work for you? I use that and 
it works just fine with ReadFile/WriteFile (example, I use to to feed std 
handles to CreateProcess - and those are always accessed with ReadFile/WriteFile 
by childs).


- Davide

Re: MinGW port usable

[Johannes Sixt]

"H. Peter Anvin" wrote:
> Erf... I dug through this, and it seems that WriteFile only works on a
> socket if it has an OVERLAPPED argument now, because the socket is
> opened for overlapping I/O.

Guess, what? When I use WSASocket() instead of socket() to explicitly
create a non-overlapped socket, plus _open_osfhandle(), it works!

This does not seem to be that new a behavior, since I observed all the
failures on Windows 2000.

Thanks a lot for your tip!

-- Hannes

Re: MinGW port usable

[H. Peter Anvin]

H. Peter Anvin wrote:
> H. Peter Anvin wrote:
>>
>> Except they are (for NT-based Windows), so you're doing something 
>> goofy.  This is a widely used construct, so it can't be that broken.
>>
> 
> Erf... I dug through this, and it seems that WriteFile only works on a 
> socket if it has an OVERLAPPED argument now, because the socket is 
> opened for overlapping I/O.  This must be new behaviour in XP-SP2, 
> because this definitely wasn't the case when I last played with this 
> stuff back in 2003.  The Internet is full of people using this 
> technique, but I haven't found a way to get a socket which is *not* 
> opened for overlapping I/O.
> 
> How typical of Microsoft to break an incredibly powerful unified 
> paradigm, sort-of repair it, and then break it again.  There doesn't 
> seem to be an obvious way to repair this, either, since MS DLLs won't 
> let you override for example the write() function as called from inside 
> the C runtime DLL.
> 
> "Some people are just a total waste of carbon..."
> 

**HAH***

I found a workaround after all.  The trick is to use WSASocket() with a 
last argument 0 instead of socket().

I feel dirty now...


#include <winsock2.h>
#include <fcntl.h>
#include <io.h>

#define socket(a,b,c) sane_socket(a,b,c)

int sane_socket(int domain, int type, int protocol)
{
	SOCKET s;
	int fd;

	s = WSASocket(domain, type, protocol, NULL, 0, 0);
	if (s == INVALID_SOCKET) {
		/*
		 * WSAGetLastError() values seem to be mostly
		 * errno+10000; verify this or replace this with
		 * a switch statement...
		 */
		errno = WSAGetLastError() - 10000;
		return -1;
	}
	
	fd = _open_osfhandle(s, 0);
	if (fd < 0)
		closesocket(s);

	return fd;
}