git 0.99.7b doesn't build on Cygwin

git 0.99.7b doesn't build on Cygwin

[Peter TB Brett]

Hi folks,

I wanted to use git on a Windows-based project (yes, there are some
out there still), so I fired up my Cygwin xterm, untarred the git
sources and totally failed to succeed in building them:

                              ---- ----

$ make
gcc -o apply.o -c -g -O2 -Wall  '-DSHA1_HEADER=<openssl/sha.h>' apply.c
gcc -o blob.o -c -g -O2 -Wall  '-DSHA1_HEADER=<openssl/sha.h>' blob.c
gcc -o commit.o -c -g -O2 -Wall  '-DSHA1_HEADER=<openssl/sha.h>' commit.c
gcc -o connect.o -c -g -O2 -Wall  '-DSHA1_HEADER=<openssl/sha.h>' connect.c
connect.c: In function `git_tcp_connect':
connect.c:298: error: storage size of 'hints' isn't known
connect.c:322: warning: implicit declaration of function `getaddrinfo'
connect.c:324: warning: implicit declaration of function `gai_strerror'
connect.c:324: warning: format argument is not a pointer (arg 3)
connect.c:326: error: dereferencing pointer to incomplete type
connect.c:327: error: dereferencing pointer to incomplete type
connect.c:327: error: dereferencing pointer to incomplete type
connect.c:327: error: dereferencing pointer to incomplete type
connect.c:330: error: dereferencing pointer to incomplete type
connect.c:330: error: dereferencing pointer to incomplete type
connect.c:338: warning: implicit declaration of function `freeaddrinfo'
connect.c:298: warning: unused variable `hints'
make: *** [connect.o] Error 1

$ gcc --version
gcc (GCC) 3.4.4 (cygming special) (gdc 0.12, using dmd 0.125)
...

                              ---- ----

It looks like the sort of problems I get when I'm missing header
files, but all the headers #included by connect.c are present on my
system, so I'm really not sure what's going on there...

Peter


P.S. Please Cc: me on any replies, I'm not subscribed to the list.


-- 
Quake II build tools:  http://peter-b.co.uk/
Latest QuArK:          http://quark.sourceforge.net/LatestVersion

v2sw6YShw7ln5pr6ck3ma8u7Lw3+2m0l7CFi6e4+8t4Eb8Aen4g6Pa2Xs5MSr5p4
  hackerkey.com

Re: git 0.99.7b doesn't build on Cygwin

[Johannes Schindelin]

Hi,

On Fri, 23 Sep 2005, Peter TB Brett wrote:

> I wanted to use git on a Windows-based project (yes, there are some
> out there still), so I fired up my Cygwin xterm, untarred the git
> sources and totally failed to succeed in building them:

Also see my mail regarding cygwin:

http://www.gelato.unsw.edu.au/archives/git/0508/7956.html

> gcc -o connect.o -c -g -O2 -Wall  '-DSHA1_HEADER=<openssl/sha.h>' connect.c
> connect.c: In function `git_tcp_connect':
> connect.c:298: error: storage size of 'hints' isn't known
> connect.c:322: warning: implicit declaration of function `getaddrinfo'
> connect.c:324: warning: implicit declaration of function `gai_strerror'
> [...]

This is the IPv6 stuff. There are patches to cygwin to support IPv6 
somewhere, but they haven't made it into mainline.

As for the other problem I mentioned in my original mail:

It seems that the fixup of the mmap()ed regions after a fork() does not 
work properly in cygwin. Remember that cygwin just wraps the non-POSIX 
Win32API and tries to make it sort of POSIX compliant. The problem is that 
Win32API lacks a proper fork(). This is therefore emulated, and after 
that, all the mmap()ed regions have to be mapped again. That fails.

Sidenote: I ran it inside gdb, and it worked fine! So I tried to recompile 
the cygwin1.dll, but that wrecked my whole installation of cygwin and I 
spent 2 hours just to be able to "gcc -o" again.

Ciao,
Dscho

Re: git 0.99.7b doesn't build on Cygwin

[Peter TB Brett]

Johannes Schindelin wrote:

>> I wanted to use git on a Windows-based project (yes, there are some
>> out there still), so I fired up my Cygwin xterm, untarred the git
>> sources and totally failed to succeed in building them:
>
> Also see my mail regarding cygwin:
>
> http://www.gelato.unsw.edu.au/archives/git/0508/7956.html

Yes. I found that just after I'd spammed the list.

> This is the IPv6 stuff. There are patches to cygwin to support IPv6
> somewhere, but they haven't made it into mainline.
>
> As for the other problem I mentioned in my original mail:
>
> It seems that the fixup of the mmap()ed regions after a fork() does not
> work properly in cygwin. Remember that cygwin just wraps the non-POSIX
> Win32API and tries to make it sort of POSIX compliant. The problem is that
> Win32API lacks a proper fork(). This is therefore emulated, and after
> that, all the mmap()ed regions have to be mapped again. That fails.

Hmph. Sounds like I'm stuffed.

Ah well, I'll just have to use something else -- I know Mercurial works on
Windows.  To be honest, I'd prefer to use git though; I've used it on
other projects and it's really nice.

Ah well, can't have everything you want all of the time, neh?

Peter


-- 
Quake II build tools: http://peter-b.co.uk/
Latest QuArK:         http://quark.sourceforge.net/LatestVersion

v2sw6YShw7ln5pr6ck3ma8u7Lw3+2m0l7CFi6e4+8t4Eb8Aen4g6Pa2Xs5MSr5p4
  hackerkey.com

Re: git 0.99.7b doesn't build on Cygwin

[Martin Langhoff]

On 9/24/05, Peter TB Brett <peter@peter-b.co.uk> wrote:
> Ah well, I'll just have to use something else -- I know Mercurial works on
> Windows.  To be honest, I'd prefer to use git though; I've used it on
> other projects and it's really nice.

Petr Baudis was working on a Mercurial-GIT gateway which could be
useful, or perhaps you can use tailor.py. OTOH, if you have a unix
machine in the network, you can probably make cretive use of samba...

cheers,


martin

Re: git 0.99.7b doesn't build on Cygwin

[Petr Baudis]

Dear diary, on Sat, Sep 24, 2005 at 12:08:28AM CEST, I got a letter
where Martin Langhoff <martin.langhoff@gmail.com> told me that...
> On 9/24/05, Peter TB Brett <peter@peter-b.co.uk> wrote:
> > Ah well, I'll just have to use something else -- I know Mercurial works on
> > Windows.  To be honest, I'd prefer to use git though; I've used it on
> > other projects and it's really nice.
> 
> Petr Baudis was working on a Mercurial-GIT gateway which could be
> useful, or perhaps you can use tailor.py. OTOH, if you have a unix
> machine in the network, you can probably make cretive use of samba...

Clarification: that was a Monotone-GIT gateway.

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.

Re: git 0.99.7b doesn't build on Cygwin

[Linus Torvalds]

On Fri, 23 Sep 2005, Johannes Schindelin wrote:
> 
> It seems that the fixup of the mmap()ed regions after a fork() does not 
> work properly in cygwin. Remember that cygwin just wraps the non-POSIX 
> Win32API and tries to make it sort of POSIX compliant. The problem is that 
> Win32API lacks a proper fork(). This is therefore emulated, and after 
> that, all the mmap()ed regions have to be mapped again. That fails.

Now, I'm not a big fan of windows ("No, really? Tell us more!") but I'd 
actually like it if the _core_ git stuff worked in as wide a variety of 
situations as possible. Screw the shell scripts and the daemon or 
secondary things like that which windows users might as well generate 
their own stuff for, but I'd hope the really core stuff would work.

If I understood correctly, you said that "git-diff-tree" doesn't work due
to the fork/mmap issue. Now, I assume that means that it's the builtin
diff that has problems. 

As far as I can tell, we can solve that two ways:

 - make Windows always use the external diff program. That may be the 
   right thing to do, since then the fork() just turns into a regular 
   fork+exec, which is how windows works anyway.

 - look at doing the diff internally.

I'm wondering if there is some stupid way to turn a diff generated by 
diff_delta() into a line-based one? If you have the original file and the 
xdiff, I think we should be able to just walk the original file and output 
a unified diff.

Davide, maybe I'm being stupid, but I'm thinking that it might be possible
to generate a -u3 diff by basically walking the xdiff file in a linear
fashion: if the edits are in strictly ascending order, we could walk the
original file one line at a time, and keeping a buffer of the three last
lines. Then, when the file offset hits the next "edit" in the xdiff, we
start generating a line-based diff (and use the previous three lines as
the context).

Does that sound possible? Maybe somebody has even done it? Is it a stupid 
idea?

I realize that it might not generate the same diff as GNU diff would do, 
and maybe it's really nasty, but it sounds like it _could_ be a "cheap" 
way of generating diffs, considering that we have something that already 
generates xdiffs..

		Linus

Re: git 0.99.7b doesn't build on Cygwin

[Linus Torvalds]

On Fri, 23 Sep 2005, Linus Torvalds wrote:
> 
> Davide, maybe I'm being stupid, but I'm thinking that it might be possible
> to generate a -u3 diff by basically walking the xdiff file in a linear
> fashion: if the edits are in strictly ascending order

Ahh, no. Looking at the xdiff format, the edits are strictly ascending in 
the destination, but they may take stuff from the source in any order, so 
it's not like you can generate a diff from it by just walking it.

Oh, well. I guess we're better off just using the external diff command, 
even if it is slower to execve an external diff. 

The GNU diff sources are hard enough to read that I don't think we want to 
try to merge the unified diff generation from there.

		Linus

Re: git 0.99.7b doesn't build on Cygwin

[Junio C Hamano]

Linus Torvalds <torvalds@osdl.org> writes:

> The GNU diff sources are hard enough to read that I don't think we want to 
> try to merge the unified diff generation from there.

I was talking with GNU diff maintainer and his impression was
that CVS folks may have done enough libification -- I'll find
time to look at CVS code and see how much damage we are talking
about.

Implementing diff, was Re: git 0.99.7b doesn't build on Cygwin

[Johannes Schindelin]

Hi,

On Sun, 25 Sep 2005, Junio C Hamano wrote:

> Linus Torvalds <torvalds@osdl.org> writes:
> 
> > The GNU diff sources are hard enough to read that I don't think we want to 
> > try to merge the unified diff generation from there.
> 
> I was talking with GNU diff maintainer and his impression was
> that CVS folks may have done enough libification -- I'll find
> time to look at CVS code and see how much damage we are talking
> about.

I am not sure if it would be wise to completely do away with the current 
method: Often, I call git-diff with my own wdiff-helper. Also, options 
like "-b" to diff are very useful, and would have to be implemented, too.

Ciao,
Dscho

Re: Implementing diff, was Re: git 0.99.7b doesn't build on Cygwin

[Davide Libenzi]

On Sun, 25 Sep 2005, Johannes Schindelin wrote:

>> Linus Torvalds <torvalds@osdl.org> writes:
>>
>>> The GNU diff sources are hard enough to read that I don't think we want to
>>> try to merge the unified diff generation from there.
>>
>> I was talking with GNU diff maintainer and his impression was
>> that CVS folks may have done enough libification -- I'll find
>> time to look at CVS code and see how much damage we are talking
>> about.
>
> I am not sure if it would be wise to completely do away with the current
> method: Often, I call git-diff with my own wdiff-helper. Also, options
> like "-b" to diff are very useful, and would have to be implemented, too.

What you'd have to do, if you chose to use diffutils stuff, is to 
transform the main() of diff in diff_main(), use setjmp/longjmp to capture 
its exit()s, and make it use a proper allocator (if you want to avoid 
leaks upon aborts). You can see an example inside the diff/libgdiff 
directory of this packages:

https://www.cvshome.org
http://www.opencm.org

In that way, instead of executing "diff -u ...", you'd call diff_main() 
with the proper args array. The CVS one (the other project seems dead, 
and they lifted the thing from CVS anyway) should be readily usable.


- Davide

Re: Implementing diff, was Re: git 0.99.7b doesn't build on Cygwin

[Linus Torvalds]

On Sun, 25 Sep 2005, Davide Libenzi wrote:
>
> What you'd have to do, if you chose to use diffutils stuff, is to 
> transform the main() of diff in diff_main(), use setjmp/longjmp to capture 
> its exit()s, and make it use a proper allocator (if you want to avoid 
> leaks upon aborts).

I'd love to use libxdiff instead since you say it can do it, but quite
frankly, the man-page didn't much help me. Do you have an example of how
to generate a uni-diff with it? Something that mortal men can read and say 
"oh"?

		Linus

Re: Implementing diff, was Re: git 0.99.7b doesn't build on Cygwin

[Davide Libenzi]

On Sun, 25 Sep 2005, Linus Torvalds wrote:

> On Sun, 25 Sep 2005, Davide Libenzi wrote:
>>
>> What you'd have to do, if you chose to use diffutils stuff, is to
>> transform the main() of diff in diff_main(), use setjmp/longjmp to capture
>> its exit()s, and make it use a proper allocator (if you want to avoid
>> leaks upon aborts).
>
> I'd love to use libxdiff instead since you say it can do it, but quite
> frankly, the man-page didn't much help me. Do you have an example of how
> to generate a uni-diff with it? Something that mortal men can read and say
> "oh"?

Ahh, you looked at the docs. Don't do that ;) Take a look at the 
test/xdiff_test.c for an example on how to use its most important APIs. 
There is also a regression test (xregression) that create random files, 
renadomly changes them, and then try that (A-B)+B=A and (B-A)+A=B (for 
both text and binary). I'm dropping inline an example on how to text-diff ...



- Davide



#include "xmacros.h"
#include "xdiff.h"

#define XDLT_STD_BLKSIZE (1024 * 8)

static int xdlt_load_mmfile(char const *fname, mmfile_t *mf, int binmode) {
         char cc;
         int fd;
         long size, bsize;
         char *blk;

         if (xdl_init_mmfile(mf, XDLT_STD_BLKSIZE, XDL_MMF_ATOMIC) < 0) {

                 return -1;
         }
         if ((fd = open(fname, O_RDONLY)) == -1) {
                 perror(fname);
                 xdl_free_mmfile(mf);
                 return -1;
         }
         if ((size = bsize = lseek(fd, 0, SEEK_END)) > 0 && !binmode) {
                 if (lseek(fd, -1, SEEK_END) != (off_t) -1 &&
                     read(fd, &cc, 1) && cc != '\n')
                         bsize++;
         }
         lseek(fd, 0, SEEK_SET);
         if (!(blk = (char *) xdl_mmfile_writeallocate(mf, bsize))) {
                 xdl_free_mmfile(mf);
                 close(fd);
                 return -1;
         }
         if (read(fd, blk, (size_t) size) != (size_t) size) {
                 perror(fname);
                 xdl_free_mmfile(mf);
                 close(fd);
                 return -1;
         }
         close(fd);
         if (bsize > size)
                 blk[size] = '\n';
         return 0;
}

static int xdlt_outf(void *priv, mmbuffer_t *mb, int nbuf) {
         int i;

         for (i = 0; i < nbuf; i++)
                 if (!fwrite(mb[i].ptr, mb[i].size, 1, (FILE *) priv))
                         return -1;
         return 0;
}

static void *wrap_malloc(void *priv, unsigned int size) {

         return malloc(size);
}

static void wrap_free(void *priv, void *ptr) {

         free(ptr);
}

static void *wrap_realloc(void *priv, void *ptr, unsigned int size) {

         return realloc(ptr, size);
}

int sample_textdiff(char const *pre, char const *post, FILE *outf) {
         int error;
         memallocator_t malt;
         mmfile_t mf1, mf2;
         xpparam_t xpp;
         xdemitconf_t xecfg;
         xdemitcb_t ecb;

         malt.priv = NULL;
         malt.malloc = wrap_malloc;
         malt.free = wrap_free;
         malt.realloc = wrap_realloc;
         xdl_set_allocator(&malt);
         if (xdlt_load_mmfile(pre, &mf1, 0) < 0)
                 return -1;
         if (xdlt_load_mmfile(post, &mf2, 0) < 0) {
                 xdl_free_mmfile(&mf1);
                 return -1;
         }
         xpp.flags = 0;
         xecfg.ctxlen = 3;
         ecb.priv = outf;
         ecb.outf = xdlt_outf;
         error = xdl_diff(&mf1, &mf2, &xpp, &xecfg, &ecb);
         xdl_free_mmfile(&mf2);
         xdl_free_mmfile(&mf1);

         return error;
}

Re: git 0.99.7b doesn't build on Cygwin

[Johannes Schindelin]

Hi,

On Fri, 23 Sep 2005, Linus Torvalds wrote:

> On Fri, 23 Sep 2005, Johannes Schindelin wrote:
> > 
> > It seems that the fixup of the mmap()ed regions after a fork() does not 
> > work properly in cygwin. Remember that cygwin just wraps the non-POSIX 
> > Win32API and tries to make it sort of POSIX compliant. The problem is that 
> > Win32API lacks a proper fork(). This is therefore emulated, and after 
> > that, all the mmap()ed regions have to be mapped again. That fails.
> 
> Now, I'm not a big fan of windows ("No, really? Tell us more!") but I'd 
> actually like it if the _core_ git stuff worked in as wide a variety of 
> situations as possible.

It is sure worth to try to be as portable as possible. Just look at the 
bugs found by running git on x86_64 (for example by HPA), which were not 
apparent from x86 or PowerPC.

> Screw the shell scripts and the daemon or secondary things like that 
> which windows users might as well generate their own stuff for, but I'd 
> hope the really core stuff would work.

Whoa, slow! The shell scripts and the networking are important parts even 
of the core git suite. Without them, work is next to impossible.

> If I understood correctly, you said that "git-diff-tree" doesn't work due
> to the fork/mmap issue. Now, I assume that means that it's the builtin
> diff that has problems. 

No. It means that there is something weird going on inside cygwin1.dll. 
This library works perfectly when the program is run inside gdb. Which 
could well mean some timing issue. Unfortunately, I have problems 
rebuilding cygwin1.dll, and therefore cannot debug in detail.

BTW I am fairly convinced that the same issues would trouble a git-pull, 
once the networking is running, since the pack transfer relies on 
fork()ing.

> I'm wondering if there is some stupid way to turn a diff generated by 
> diff_delta() into a line-based one? If you have the original file and the 
> xdiff, I think we should be able to just walk the original file and output 
> a unified diff.

It sure would be nice to have a unified diff generator included, but I 
doubt that a reliable (=simple) one is easy to come by.

Ciao,
Dscho

Re: git 0.99.7b doesn't build on Cygwin

[Linus Torvalds]

On Sat, 24 Sep 2005, Johannes Schindelin wrote:
> 
> BTW I am fairly convinced that the same issues would trouble a git-pull, 
> once the networking is running, since the pack transfer relies on 
> fork()ing.

I'm not sure.

Almost all other fork() users end up doing a more-or-less immediate 
execve() after the fork. Yes, they do some other minor setup, but not a 
whole lot.

The diff.c fork() is somewhat different. It actually ends up doing malloc 
and stdio IO before it actually gets to the exec(), so that one is more 
likely to hit any bugs in the fork() implementation.

Actually, looking a bit closer, the create_pack_file() thing also does 
malloc inside the child, but at least there it would be trivial to move 
that argument setup code into the parent.

But looking at send_pack() or fetch_pack(), for example, they are both
_very_ traditional fork()+exec() calls, with just a few close() calls in
between.

Looking a bit closer at the diff() usage, I actually think that we could 
move the fork() closer to the exec - we'd just have to move it _into_ all 
the different cases (ie you'd have two different fork() calls: one for 
the "builtin" case, one for the external pgm case, but then the child in 
both cases would be very simple).

Oh. Actually, I wonder if we could mke them "vfork()" calls. Does anybody 
know if cygwin has an easier time with vfork() + eventual exec? That 
_should_ map better to a non-UNIX process model, so maybe we could do it 
that way?

> It sure would be nice to have a unified diff generator included, but I 
> doubt that a reliable (=simple) one is easy to come by.

Yeah, I looked at GNU diffutils, and I had to rinse out my eyes with soap 
and water.

		Linus

Re: git 0.99.7b doesn't build on Cygwin

[Junio C Hamano]

Linus Torvalds <torvalds@osdl.org> writes:

> Looking a bit closer at the diff() usage, I actually think that we could 
> move the fork() closer to the exec - we'd just have to move it _into_ all 
> the different cases (ie you'd have two different fork() calls: one for 
> the "builtin" case, one for the external pgm case, but then the child in 
> both cases would be very simple).

Looking back at what I did in the diff.c, I actually think the
part near fork() is a total crap ;-).

Originally I intended to do more work in the child process (this
is totally opposite of what is being proposed now), for example
running prepare_temp_file() after child forked, so that the
parent process does not have to worry about using memory for
expanded blob to be written out to the temporary file and then
later forgetting to free it ;-), but it seems the parent is
doing more work than I intended to.  I honestly think that the
part of the code is ancient enough to deserve a major facelift.

Re: git 0.99.7b doesn't build on Cygwin

[Davide Libenzi]

On Fri, 23 Sep 2005, Linus Torvalds wrote:

> But looking at send_pack() or fetch_pack(), for example, they are both
> _very_ traditional fork()+exec() calls, with just a few close() calls in
> between.
>
> Looking a bit closer at the diff() usage, I actually think that we could
> move the fork() closer to the exec - we'd just have to move it _into_ all
> the different cases (ie you'd have two different fork() calls: one for
> the "builtin" case, one for the external pgm case, but then the child in
> both cases would be very simple).
>
> Oh. Actually, I wonder if we could mke them "vfork()" calls. Does anybody
> know if cygwin has an easier time with vfork() + eventual exec? That
> _should_ map better to a non-UNIX process model, so maybe we could do it
> that way?

If you have only to run diff/patch, just use the native Win32 CreateProcess().
You abstract that on a git_exec(), and you use fork/exec on Unix and 
CreateProcess() on Winblows. If fork() is slow on Cygwin, fork+exec is 
pathetic. They do all that work to give you a fork(), and you throw it 
away with an exec().


- Davide

Re: git 0.99.7b doesn't build on Cygwin

[Linus Torvalds]

On Fri, 23 Sep 2005, Davide Libenzi wrote:
> 
> If you have only to run diff/patch, just use the native Win32 CreateProcess().
> You abstract that on a git_exec(), and you use fork/exec on Unix and 
> CreateProcess() on Winblows. If fork() is slow on Cygwin, fork+exec is 
> pathetic. They do all that work to give you a fork(), and you throw it 
> away with an exec().

CreateProcess doesn't work all that well, since we want to dup file 
descriptors around and close them in the child.

In general, CreateProcess() is a totally crap interface. I realize it's 
common (and especially in the VMS/Windows world it's how things are done), 
but hey, at that point it's better if somebody just waits until git is 
stable, and just makes a totally separate "git for windows" thing. The 
interfaces are certainly simple. There's no point in trying to maintain 
one tree.

However, vfork() really _is_ a nice interface. It's faster even on UNIX,
and at least in theory it should be possible to do an efficient vfork()  
implementation on top of crap like windows. Does cygwin support that well?

Yes, git uses lots of filesystem stuff, and they suck under windows. Maybe 
cygwin adds its own overhead, but from everything I've ever been able to 
tell, filesystem access sucks under Windows regardless of any cygwin 
stuff. Add to an already slow FS interface the fact that virus checkers 
tend to hook into it and make it _even_slower_, and hey, you have a truly 
sucky OS. 

But at least with pack-files, the filesystem access patterns are much 
less common. Opening one pack-file and mapping it gets the FS out of the 
way. So I don't think that's necessarily a huge problem.

		Linus

Re: git 0.99.7b doesn't build on Cygwin

[Davide Libenzi]

On Sat, 24 Sep 2005, Linus Torvalds wrote:

>
>
> On Fri, 23 Sep 2005, Davide Libenzi wrote:
>>
>> If you have only to run diff/patch, just use the native Win32 CreateProcess().
>> You abstract that on a git_exec(), and you use fork/exec on Unix and
>> CreateProcess() on Winblows. If fork() is slow on Cygwin, fork+exec is
>> pathetic. They do all that work to give you a fork(), and you throw it
>> away with an exec().
>
> CreateProcess doesn't work all that well, since we want to dup file
> descriptors around and close them in the child.

You can do that (dup stuff and pass them around) even with CreateProcess(),
if you want. Yes, the interface sucks (zillions of parameters/flags) ;)



> In general, CreateProcess() is a totally crap interface. I realize it's
> common (and especially in the VMS/Windows world it's how things are done),
> but hey, at that point it's better if somebody just waits until git is
> stable, and just makes a totally separate "git for windows" thing. The
> interfaces are certainly simple. There's no point in trying to maintain
> one tree.
>
> However, vfork() really _is_ a nice interface. It's faster even on UNIX,
> and at least in theory it should be possible to do an efficient vfork()
> implementation on top of crap like windows. Does cygwin support that well?
>
> Yes, git uses lots of filesystem stuff, and they suck under windows. Maybe
> cygwin adds its own overhead, but from everything I've ever been able to
> tell, filesystem access sucks under Windows regardless of any cygwin
> stuff. Add to an already slow FS interface the fact that virus checkers
> tend to hook into it and make it _even_slower_, and hey, you have a truly
> sucky OS.

I also realized that git plays/handles with unix permissions too, and this 
might make the "interface layer" not so small. Dunno about vfork() on 
Cygwin, but if you really care about performance on Windows, I'd rather 
remove the external program execution and use an in-process diff library.



- Davide

Re: git 0.99.7b doesn't build on Cygwin

[Junio C Hamano]

Making really really core part usable on Windows would not need
this, but there is another thing: .git/HEAD symlink.
 

Re: git 0.99.7b doesn't build on Cygwin

[Davide Libenzi]

On Sat, 24 Sep 2005, Junio C Hamano wrote:

> Making really really core part usable on Windows would not need
> this, but there is another thing: .git/HEAD symlink.

Starting from Win2k, they *finally* added:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/createhardlink.asp



- Davide

Re: git 0.99.7b doesn't build on Cygwin

[Junio C Hamano]

Davide Libenzi <davidel@xmailserver.org> writes:

> On Sat, 24 Sep 2005, Junio C Hamano wrote:
>
>> Making really really core part usable on Windows would not need
>> this, but there is another thing: .git/HEAD symlink.
>
> Starting from Win2k, they *finally* added:

Ah, that's good to know.

Re: git 0.99.7b doesn't build on Cygwin

[Junio C Hamano]

Davide Libenzi <davidel@xmailserver.org> writes:

> On Sat, 24 Sep 2005, Junio C Hamano wrote:
>
>> Making really really core part usable on Windows would not need
>> this, but there is another thing: .git/HEAD symlink.
>
> Starting from Win2k, they *finally* added:
>
> http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/createhardlink.asp

It talks about "a hard link".  Can we readlink it?

Re: git 0.99.7b doesn't build on Cygwin

[Davide Libenzi]

On Sat, 24 Sep 2005, Junio C Hamano wrote:

> Davide Libenzi <davidel@xmailserver.org> writes:
>
>> On Sat, 24 Sep 2005, Junio C Hamano wrote:
>>
>>> Making really really core part usable on Windows would not need
>>> this, but there is another thing: .git/HEAD symlink.
>>
>> Starting from Win2k, they *finally* added:
>>
>> http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/createhardlink.asp
>
> It talks about "a hard link".  Can we readlink it?

Nope. It's an hardlink (ala link(2)).


- Davide

Re: git 0.99.7b doesn't build on Cygwin

[Linus Torvalds]

On Sat, 24 Sep 2005, Davide Libenzi wrote:
> 
> http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/createhardlink.asp

Don't you mean

	http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/createsymboliclink.asp

rather?

It mentions longhorn.

		Linus

Re: git 0.99.7b doesn't build on Cygwin

[Linus Torvalds]

On Sat, 24 Sep 2005, Linus Torvalds wrote:
> 
> It mentions longhorn.

Anyway, regardless, we could certainly make HEAD be a regular file 
containing the name of the head instead.

It probably wouldn't even require a whole lot of changes. HEAD already 
ends up getting some special attention, since most of the things that look 
for refs only look inside the .git/refs directory.

		Linus

Re: git 0.99.7b doesn't build on Cygwin

[Linus Torvalds]

On Sat, 24 Sep 2005, Linus Torvalds wrote:
>
> Anyway, regardless, we could certainly make HEAD be a regular file 
> containing the name of the head instead.
> 
> It probably wouldn't even require a whole lot of changes. HEAD already 
> ends up getting some special attention, since most of the things that look 
> for refs only look inside the .git/refs directory.

This patch does some of it. I decided not to special-case HEAD, but to 
just improve "read_ref()" a bit.

Changing "read_ref()" was the trivial part - the bigger part that we had 
three different implementations of it, and this patch is thus bigger just 
because it collapses them all into "read_ref()" and makes the calling 
conventions acceptable to all.

NOTE! This makes "symbolic refs" usable in general, ie you can do

	echo "ref: refs/tags/v0.99.7" > .git/refs/tags/LATEST

and that essentially makes "LATEST" a symbolic ref that points to the 
v0.99.7 tag without using a filesystem symlink. But it does NOT mean that 
you can replace the HEAD symlink with a file containing "refs/tags/master" 
yet: there are _other_ parts of git that depend on it being a symlink. 

(For the most core example, the "write new head" logic depends on just
writing to HEAD, and that symlink will automatically change that write to 
the thing the HEAD _points_ to. That's the biggest one).

I'll change those too to accept a regular file, if people agree this is 
worthwhile. In theory there are even UNIXes out there that don't support 
symlinks, so maybe it's worth it. But maybe people dislike this.

In the meantime, you can test this out with

	echo "ref: HEAD" > .git/TEST_HEAD
	git-rev-parse HEAD TEST_HEAD master

which - if your HEAD points to master - should print out the same SHA1
three times ;)

		Linus

---
Subject: Allow reading "symbolic refs" that point to other refs

This extends the ref reading to understand a "symbolic ref": a ref file 
that starts with "ref: " and points to another ref file, and thus 
introduces the notion of ref aliases.

This is in preparation of allowing HEAD to eventually not be a symlink, 
but one of these symbolic refs instead.

Signed-off-by: Linus Torvalds <torvalds@osdl.org>
---
diff --git a/cache.h b/cache.h
--- a/cache.h
+++ b/cache.h
@@ -227,6 +227,7 @@ extern int has_pack_index(const unsigned
 extern int get_sha1(const char *str, unsigned char *sha1);
 extern int get_sha1_hex(const char *hex, unsigned char *sha1);
 extern char *sha1_to_hex(const unsigned char *sha1);	/* static buffer result! */
+extern int read_ref(const char *filename, unsigned char *sha1);
 
 /* General helper functions */
 extern void usage(const char *err) NORETURN;
diff --git a/refs.c b/refs.c
--- a/refs.c
+++ b/refs.c
@@ -2,17 +2,38 @@
 #include "cache.h"
 
 #include <errno.h>
+#include <ctype.h>
 
-static int read_ref(const char *refname, unsigned char *sha1)
+/* We allow "recursive" symbolic refs. Only within reason, though */
+#define MAXDEPTH 5
+
+int read_ref(const char *filename, unsigned char *sha1)
 {
-	int ret = -1;
-	int fd = open(git_path("%s", refname), O_RDONLY);
+	int depth = 0;
+	int ret = -1, fd;
+
+	while ((fd = open(filename, O_RDONLY)) >= 0) {
+		char buffer[256];
+		int len = read(fd, buffer, sizeof(buffer)-1);
 
-	if (fd >= 0) {
-		char buffer[60];
-		if (read(fd, buffer, sizeof(buffer)) >= 40)
-			ret = get_sha1_hex(buffer, sha1);
 		close(fd);
+		if (len < 0)
+			break;
+
+		buffer[len] = 0;
+		while (len && isspace(buffer[len-1]))
+			buffer[--len] = 0;
+
+		if (!strncmp(buffer, "ref: ", 5)) {
+			if (depth > MAXDEPTH)
+				break;
+			depth++;
+			filename = git_path("%s", buffer+5);
+			continue;
+		}
+		if (len >= 40)
+			ret = get_sha1_hex(buffer, sha1);
+		break;
 	}
 	return ret;
 }
@@ -54,7 +75,7 @@ static int do_for_each_ref(const char *b
 					break;
 				continue;
 			}
-			if (read_ref(path, sha1) < 0)
+			if (read_ref(git_path("%s", path), sha1) < 0)
 				continue;
 			if (!has_sha1_file(sha1))
 				continue;
@@ -71,7 +92,7 @@ static int do_for_each_ref(const char *b
 int head_ref(int (*fn)(const char *path, const unsigned char *sha1))
 {
 	unsigned char sha1[20];
-	if (!read_ref("HEAD", sha1))
+	if (!read_ref(git_path("HEAD"), sha1))
 		return fn("HEAD", sha1);
 	return 0;
 }
@@ -101,33 +122,14 @@ static char *ref_lock_file_name(const ch
 	return ret;
 }
 
-static int read_ref_file(const char *filename, unsigned char *sha1) {
-	int fd = open(filename, O_RDONLY);
-	char hex[41];
-	if (fd < 0) {
-		return error("Couldn't open %s\n", filename);
-	}
-	if ((read(fd, hex, 41) < 41) ||
-	    (hex[40] != '\n') ||
-	    get_sha1_hex(hex, sha1)) {
-		error("Couldn't read a hash from %s\n", filename);
-		close(fd);
-		return -1;
-	}
-	close(fd);
-	return 0;
-}
-
 int get_ref_sha1(const char *ref, unsigned char *sha1)
 {
-	char *filename;
-	int retval;
+	const char *filename;
+
 	if (check_ref_format(ref))
 		return -1;
-	filename = ref_file_name(ref);
-	retval = read_ref_file(filename, sha1);
-	free(filename);
-	return retval;
+	filename = git_path("refs/%s", ref);
+	return read_ref(filename, sha1);
 }
 
 static int lock_ref_file(const char *filename, const char *lock_filename,
@@ -140,7 +142,7 @@ static int lock_ref_file(const char *fil
 		return error("Couldn't open lock file for %s: %s",
 			     filename, strerror(errno));
 	}
-	retval = read_ref_file(filename, current_sha1);
+	retval = read_ref(filename, current_sha1);
 	if (old_sha1) {
 		if (retval) {
 			close(fd);
diff --git a/sha1_name.c b/sha1_name.c
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -119,21 +119,6 @@ static int get_short_sha1(const char *na
 	return -1;
 }
 
-static int get_sha1_file(const char *path, unsigned char *result)
-{
-	char buffer[60];
-	int fd = open(path, O_RDONLY);
-	int len;
-
-	if (fd < 0)
-		return -1;
-	len = read(fd, buffer, sizeof(buffer));
-	close(fd);
-	if (len < 40)
-		return -1;
-	return get_sha1_hex(buffer, result);
-}
-
 static int get_sha1_basic(const char *str, int len, unsigned char *sha1)
 {
 	static const char *prefix[] = {
@@ -150,7 +135,7 @@ static int get_sha1_basic(const char *st
 
 	for (p = prefix; *p; p++) {
 		char *pathname = git_path("%s/%.*s", *p, len, str);
-		if (!get_sha1_file(pathname, sha1))
+		if (!read_ref(pathname, sha1))
 			return 0;
 	}
 

Re: git 0.99.7b doesn't build on Cygwin

[Jon Loeliger]

On Sat, 2005-09-24 at 17:27, Linus Torvalds wrote:
> On Sat, 24 Sep 2005, Linus Torvalds wrote:
> > 
> > It mentions longhorn.
> 
> Anyway, regardless, we could certainly make HEAD be a regular file 
> containing the name of the head instead.

I cleverly removed a branch this morning (intentionally),
but what I failed to realize was that it was also the
current HEAD.  The next series of command were oddly
cryptic, only telling me the git-read-tree usage message.
Not a clue in the world why, of course.  After poking
around, reading some scripts and all, I discovered that
I had a dangling .git/HEAD symlink still point to my
removed branch.

Would it be worthwhile for me to re-discover this
Wonky Failure (Hi Linus! :-) and try to write it up
somewhere or dream up a better error message patch?

Thanks,
jdl

Re: git 0.99.7b doesn't build on Cygwin

[Junio C Hamano]

Jon Loeliger <jdl@freescale.com> writes:

> Would it be worthwhile for me to re-discover this
> Wonky Failure (Hi Linus! :-) and try to write it up
> somewhere or dream up a better error message patch?

Thanks.

Re: git 0.99.7b doesn't build on Cygwin

[Davide Libenzi]

On Sat, 24 Sep 2005, Linus Torvalds wrote:

>
>
> On Sat, 24 Sep 2005, Davide Libenzi wrote:
>>
>> http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/createhardlink.asp
>
> Don't you mean
>
> 	http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/createsymboliclink.asp
>
> rather?
>
> It mentions longhorn.

Hah, didn't know this one. Requiring LongHorn is pretty strict though ;)


- Davide

Re: git 0.99.7b doesn't build on Cygwin

[Giuseppe Bilotta]

On Sat, 24 Sep 2005 15:41:27 -0700 (PDT), Davide Libenzi wrote:

> On Sat, 24 Sep 2005, Linus Torvalds wrote:
> 
>>
>>
>> On Sat, 24 Sep 2005, Davide Libenzi wrote:
>>>
>>> http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/createhardlink.asp
>>
>> Don't you mean
>>
>> 	http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/createsymboliclink.asp
>>
>> rather?
>>
>> It mentions longhorn.
> 
> Hah, didn't know this one. Requiring LongHorn is pretty strict though ;)

However, it might be possible to use .lnk files, which would work on
both NTFS and FAT32, and even under Win9x.

-- 
Giuseppe "Oblomov" Bilotta

"I weep for our generation" -- Charlie Brown

Re: git 0.99.7b doesn't build on Cygwin

[Junio C Hamano]

Giuseppe Bilotta <bilotta78@hotpop.com> writes:

>> Hah, didn't know this one. Requiring LongHorn is pretty strict though ;)
>
> However, it might be possible to use .lnk files, which would work on
> both NTFS and FAT32, and even under Win9x.

Possibly, but it is a moot point now.

When textual "symbolic refs" support becomes mature, we will use
it on boxes without symbolic links to express .git/HEAD.

Re: git 0.99.7b doesn't build on Cygwin

[Davide Libenzi]

On Sun, 25 Sep 2005, Giuseppe Bilotta wrote:

> However, it might be possible to use .lnk files, which would work on
> both NTFS and FAT32, and even under Win9x.

The .lnk files are a shell thing, not an OS one. Try to open()+read() a 
.lnk file and look at what you get ...


- Davide

Re: git 0.99.7b doesn't build on Cygwin

[Giuseppe Bilotta]

On Sun, 25 Sep 2005 22:05:29 -0700 (PDT), Davide Libenzi wrote:

> On Sun, 25 Sep 2005, Giuseppe Bilotta wrote:
> 
>> However, it might be possible to use .lnk files, which would work on
>> both NTFS and FAT32, and even under Win9x.
> 
> The .lnk files are a shell thing, not an OS one. Try to open()+read() a 
> .lnk file and look at what you get ...

Well, sure. I wasn't thinking about just substituting .lnk files for
symlinks, but that's the closest thing you can get on Windows,
currently, so maybe supporting this kind of thing would be the best
approach.

-- 
Giuseppe "Oblomov" Bilotta

[W]hat country can preserve its liberties, if its rulers are not
warned from time to time that [the] people preserve the spirit of
resistance? Let them take arms...The tree of liberty must be
refreshed from time to time, with the blood of patriots and
tyrants.
	-- Thomas Jefferson, letter to Col. William S. Smith, 1787

Re: git 0.99.7b doesn't build on Cygwin

[H. Peter Anvin]

Davide Libenzi wrote:
> On Sun, 25 Sep 2005, Giuseppe Bilotta wrote:
> 
>> However, it might be possible to use .lnk files, which would work on
>> both NTFS and FAT32, and even under Win9x.
> 
> 
> The .lnk files are a shell thing, not an OS one. Try to open()+read() a 
> .lnk file and look at what you get ...
> 

Except that Cygwin uses them transparently, so if you do open() and 
read() under Cygwin they work as expected.

	-hpa

Re: git 0.99.7b doesn't build on Cygwin

[Davide Libenzi]

On Mon, 26 Sep 2005, H. Peter Anvin wrote:

> Davide Libenzi wrote:
>> On Sun, 25 Sep 2005, Giuseppe Bilotta wrote:
>> 
>>> However, it might be possible to use .lnk files, which would work on
>>> both NTFS and FAT32, and even under Win9x.
>> 
>> 
>> The .lnk files are a shell thing, not an OS one. Try to open()+read() a 
>> .lnk file and look at what you get ...
>> 
>
> Except that Cygwin uses them transparently, so if you do open() and read() 
> under Cygwin they work as expected.

With Cygwin you don't even need .lnk files, since it already supports all 
the Unix symlinks APIs/cmds. The discussion born thinking about a native 
Win32 interface, w/out the Cygwin crud in it.


- Davide

Re: git 0.99.7b doesn't build on Cygwin

[H. Peter Anvin]

Davide Libenzi wrote:
>>
>> Except that Cygwin uses them transparently, so if you do open() and 
>> read() under Cygwin they work as expected.
> 
> With Cygwin you don't even need .lnk files, since it already supports 
> all the Unix symlinks APIs/cmds. The discussion born thinking about a 
> native Win32 interface, w/out the Cygwin crud in it.
> 

Cygwin symbolic links are implemented as .lnk files on the underlying 
filesystem was my point.

	-hpaa

Re: git 0.99.7b doesn't build on Cygwin

[Daniel Barkalow]

On Sat, 24 Sep 2005, Junio C Hamano wrote:

> Making really really core part usable on Windows would not need
> this, but there is another thing: .git/HEAD symlink.

Cygwin supports symlinks without underlying filesystem support. It does 
basically the standard UNIX thing for symlinks, but inefficiently in 
userspace.

	-Daniel
*This .sig left intentionally blank*

Re: git 0.99.7b doesn't build on Cygwin

[Davide Libenzi]

On Fri, 23 Sep 2005, Linus Torvalds wrote:

>
>
> On Fri, 23 Sep 2005, Johannes Schindelin wrote:
>>
>> It seems that the fixup of the mmap()ed regions after a fork() does not
>> work properly in cygwin. Remember that cygwin just wraps the non-POSIX
>> Win32API and tries to make it sort of POSIX compliant. The problem is that
>> Win32API lacks a proper fork(). This is therefore emulated, and after
>> that, all the mmap()ed regions have to be mapped again. That fails.
>
> Now, I'm not a big fan of windows ("No, really? Tell us more!") but I'd
> actually like it if the _core_ git stuff worked in as wide a variety of
> situations as possible. Screw the shell scripts and the daemon or
> secondary things like that which windows users might as well generate
> their own stuff for, but I'd hope the really core stuff would work.
>
> If I understood correctly, you said that "git-diff-tree" doesn't work due
> to the fork/mmap issue. Now, I assume that means that it's the builtin
> diff that has problems.

Stay away from Cygwin if you use extensively fork(), since it becomes dog 
slow (the fork() implementation on Cygwin involves creating a new 
suspended task, *copy* - not COW - the *whole* VM space to the child, 
and resuming it). Actually, the whole Cygwin in general is dog slow, 
especially on FS operations (and git likes them). If you really like to 
have Windows support (uuu hoo WinTorvalds) you might be better using a 
small compat layer (should be fairly small for git).



> As far as I can tell, we can solve that two ways:
>
> - make Windows always use the external diff program. That may be the
>   right thing to do, since then the fork() just turns into a regular
>   fork+exec, which is how windows works anyway.
>
> - look at doing the diff internally.
>
> I'm wondering if there is some stupid way to turn a diff generated by
> diff_delta() into a line-based one? If you have the original file and the
> xdiff, I think we should be able to just walk the original file and output
> a unified diff.
>
> Davide, maybe I'm being stupid, but I'm thinking that it might be possible
> to generate a -u3 diff by basically walking the xdiff file in a linear
> fashion: if the edits are in strictly ascending order, we could walk the
> original file one line at a time, and keeping a buffer of the three last
> lines. Then, when the file offset hits the next "edit" in the xdiff, we
> start generating a line-based diff (and use the previous three lines as
> the context).
>
> Does that sound possible? Maybe somebody has even done it? Is it a stupid
> idea?
>
> I realize that it might not generate the same diff as GNU diff would do,
> and maybe it's really nasty, but it sounds like it _could_ be a "cheap"
> way of generating diffs, considering that we have something that already
> generates xdiffs..

Hehe, the same library from where Nicolas lifted the code for the binary 
diff, has a totally portable diff/patch APIs (on top of xdiff/xpach):

http://www.xmailserver.org/xdiff.html

Generating text diffs, unfortunately is quite more complex than binary 
ones. Libxdiff uses the same algorithm of GNU diff (Eugene W. Myers). The 
library has zero dependency other than ANSI C. Another alternative, IIRC 
someone made a library by wrapping the diffutil stuff, but I do not 
remeber where it was.



- Davide