Windows symlinks

Windows symlinks

[Patrick.Higgins]

It looks like one of the bigger (biggest?) hurdles for git adoption at my company is going to be handling symlinks on Windows. We may be able to sidestep the issue entirely by having developers run Linux in a virtual machine (or better yet, Windows in a VM) because we deploy to Linux and thus can make a very strong argument that developers should develop on Linux, too. That run into an image problem where people might start blaming git for needing to have an extra operating system around just for revision control.

We currently use Clearcase snapshot views on Windows and it has some kind of magic to be able to deal with symlinks. I haven't actually used the Windows version of Clearcase so I don't know how it does it, but my understanding is that it handles them with copies, and I guess it somehow knows to update all the copies when a change is made.

Has anyone thought about a way for git to handle symlinks? Vista seems to have added native symlinks, but you need have elevated privilege to create them. NTFS junction points seem helpful for older versions of Windows, but don't work for anything except directories, and seem to be dangerous to use with tools that do recursive deletes. Neither junction points nor native symlinks sound like great options.

Cygwin's clever symlink trick seems to work pretty well in practice. I'm not exactly sure what it's doing, but it seems to create a shortcut that it's own programs understand. Some other non-Cygwin programs seem to understand them, too, but Java does not which is a big problem for me.

I just don't see any good solutions here. Does anyone have any better ideas?

Thanks,
Patrick

Re: Windows symlinks

[Avery Pennarun]

On 6/24/08, Patrick.Higgins@cexp.com <Patrick.Higgins@cexp.com> wrote:
>  Cygwin's clever symlink trick seems to work pretty well in practice. I'm not exactly sure what
> it's doing, but it seems to create a shortcut that it's own programs understand. Some other
> non-Cygwin programs seem to understand them, too, but Java does not which is a big
> problem for me.

It seems somewhat odd that your developers are relying on an operating
system feature that doesn't even exist.  I might suggest one of:

- add an abstraction layer to your Java applet that understands
cygwin-style symlinks

- actually just check in the file multiple times; git will store it
efficiently anyway

- improve your Makefile to automatically copy the files and/or create
symlinks at build time rather than storing the symlinks in revision
control.

My understanding of Clearcase is that it actually loads a virtual
filesystem driver in Windows and does all kinds of insane magic -
slowly and inefficiently, too - behind the scenes.  Something tells me
git (and every other VCS :)) really doesn't want to go there.

Have fun,

Avery

Re: Windows symlinks

[Jakub Narebski]

Patrick Higgins <Patrick.Higgins@cexp.com> writes:

> It looks like one of the bigger (biggest?) hurdles for git adoption
> at my company is going to be handling symlinks on Windows. We may be
> able to sidestep the issue [...] by [...] run[ning] Linux
> in a virtual machine [...]

If only MS Windows supported other filesystems which have symlinks...

> Has anyone thought about a way for git to handle symlinks? Vista
> seems to have added native symlinks, but you need have elevated
> privilege to create them. NTFS junction points seem helpful for
> older versions of Windows, but don't work for anything except
> directories, and seem to be dangerous to use with tools that do
> recursive deletes. Neither junction points nor native symlinks sound
> like great options.
> 
> Cygwin's clever symlink trick seems to work pretty well in
> practice. I'm not exactly sure what it's doing, but it seems to
> create a shortcut that it's own programs understand. Some other
> non-Cygwin programs seem to understand them, too, but Java does not
> which is a big problem for me.

First, I think that both "git on Windows" solutions, namely Cygwin and
msysGit port, don't use symlinks either in installed programs, nor in
repository layout.

Second, the problem there can be _only_ if your repository contains
(or contained) symlinks, and then it is your own damn fault.  I don't
know how Cygwin, or msysGit deals with symlinks in a wirking
directory, but you can work around symlinks (although in a bit
unwieldy way) by using `core.symlinks' configuration variable;
see git-config(1):

 core.symlinks::
        If false, symbolic links are checked out as small plain files
        that contain the link text. git-update-index(1) and git-add(1)
        will not change the recorded type to regular file. Useful on
        filesystems like FAT that do not support symbolic links. True
        by default.

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: Windows symlinks

[Robin Rosenberg]

onsdagen den 25 juni 2008 00.42.49 skrev Patrick.Higgins@cexp.com:
> It looks like one of the bigger (biggest?) hurdles for git adoption at my company is going to be handling symlinks on Windows. We may be able to sidestep the issue entirely by having developers run Linux in a virtual machine (or better yet, Windows in a VM) because we deploy to Linux and thus can make a very strong argument that developers should develop on Linux, too. That run into an image problem where people might start blaming git for needing to have an extra operating system around just for revision control.

Right, they could use only Linux and not have an extra OS for email..
> 
> We currently use Clearcase snapshot views on Windows and it has some kind of magic to be able to deal with symlinks. I haven't actually used the Windows version of Clearcase so I don't know how it does it, but my understanding is that it handles them with copies, and I guess it somehow knows to update all the copies when a change is made.

It is one of the few sane things clearcase does. msysgit's approach is just unusable. I've not come up with the patches yet. Clearcase makes copies
instead of the symlinks and updates them on checkout/checkin and update.
Git could do one step better by updating copies when index is updated, avoiding an unnecessary commit (which is pretty annoying in CC).
Invalid symlinks, i.e. pointing at non-existent objects or outside then VOB (repository in Git parlance, don't show up at all in snapshot views. Git could do the same thereby making the workdir usable
when it contains and depends on symlinks.

> Has anyone thought about a way for git to handle symlinks? Vista seems to have added native symlinks, but you need have elevated privilege to create
> them. NTFS junction points seem helpful for older versions of Windows, but don't work for anything except directories, and seem to be dangerous to use with tools that do recursive deletes. Neither junction points nor native symlinks sound like great options. 
Obviously one could make this a switchable feature and enable it when the
file system supports it. As for junction points i'd say no too. What little testing I did on symlinks in vista they seem capable of most sane uses.

> Cygwin's clever symlink trick seems to work pretty well in practice. I'm not exactly sure what it's doing, but it seems to create a shortcut that it's own programs understand. Some other non-Cygwin programs seem to understand them, too, but Java does not which is a big problem for me.
> 
> I just don't see any good solutions here. Does anyone have any better ideas?

No, copies. If you have a patch I can test it and then implement it for jgit, which doesn't have symlinks either.

-- robin

RE: Windows symlinks

[Patrick.Higgins]

> From: Jakub Narebski [mailto:jnareb@gmail.com]
> 
> Patrick Higgins <Patrick.Higgins@cexp.com> writes:
> 
> Second, the problem there can be _only_ if your repository contains
> (or contained) symlinks, and then it is your own damn fault.

Agreed. The first thing we started working on was getting symlinks out of our repositories. Unfortunately, we chose to use them because we are using broken development tools that don't support proper modularity. We found the best way to get the IDE's compilation to work was to symlink shared code into each project. It's a nasty solution, but it was the only one that worked. Clearcase made this solution possible. As powerful and flexible as git is, it would have left us out in the cold in this situation.


> I don't know how Cygwin, or msysGit deals with symlinks in a wirking
> directory

msysGit just created a file with the target of the symlink as it's contents, but its tools didn't know how to make use of it. I think Cygwin did the same thing, but ls, cat, and all the other Cygwin tools were able to make use of it. I tried to run the Java compiler over a symlinked file and it didn't work, though.

Copying files at build time is an interesting idea, but since the point was to get an IDE's build system to work, I don't I'll have the level of control I would need to make this work.

Re: Windows symlinks

[Avery Pennarun]

On 6/24/08, Patrick.Higgins@cexp.com <Patrick.Higgins@cexp.com> wrote:
> Agreed. The first thing we started working on was getting symlinks out of our repositories.
> Unfortunately, we chose to use them because we are using broken development tools that
> don't support proper modularity. We found the best way to get the IDE's compilation to work
> was to symlink shared code into each project. It's a nasty solution, but it was the only one
> that worked. Clearcase made this solution possible. As powerful and flexible as git is, it would
> have left us out in the cold in this situation.

Perhaps git-submodule would do what you're looking for.

Avery

RE: Windows symlinks

[Patrick.Higgins]

> From: Avery Pennarun [mailto:apenwarr@gmail.com]
>
> > Agreed. The first thing we started working on was getting 
> symlinks out of our repositories.
> > Unfortunately, we chose to use them because we are using 
> broken development tools that
> > don't support proper modularity. We found the best way to 
> 
> Perhaps git-submodule would do what you're looking for.

We might be able to get by with them, but submodules appear to be significantly more complex than symlinks, and we sometimes symlink just a file or two if that's all we need. Splitting up submodules to that level of granularity would be hard to manage.

My understanding of the submodule workflow is:

Projects A, B, and C (we actually have about 17 of these) all share common code in project Common. Then, each of A, B, and C adds Common as a submodule. While working on project A, the need arises to modify Common, so the developer changes it there, commits, pushes the change to Common, then commits and pushes the change to A. To update B and C, they would have to change to each of those projects, run a git pull, then git submodule update, and compile and test.

Is that correct? If so, it's a lot more work than letting a symlink propagate the change to all the other projects. Of course, since Windows doesn't have symlinks...

Re: Windows symlinks

[Jay Soffian]

On Tue, Jun 24, 2008 at 7:18 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
> My understanding of Clearcase is that it actually loads a virtual
> filesystem driver in Windows and does all kinds of insane magic -

Just FYI, this is only the case if you're using so-called dynamic
views, which require the MVFS
driver, as you say. The OP said he was using snapshot views -- these
don't use MVFS and instead just checkout to the OS's native
filesystem.

j.

Re: Windows symlinks

[Andreas Ericsson]

Patrick.Higgins@cexp.com wrote:
>> From: Avery Pennarun [mailto:apenwarr@gmail.com]
>> 
>>> Agreed. The first thing we started working on was getting
>> symlinks out of our repositories.
>>> Unfortunately, we chose to use them because we are using
>> broken development tools that
>>> don't support proper modularity. We found the best way to
>> Perhaps git-submodule would do what you're looking for.
> 
> We might be able to get by with them, but submodules appear to be
> significantly more complex than symlinks, and we sometimes symlink
> just a file or two if that's all we need. Splitting up submodules to
> that level of granularity would be hard to manage.
> 
> My understanding of the submodule workflow is:
> 
> Projects A, B, and C (we actually have about 17 of these) all share
> common code in project Common. Then, each of A, B, and C adds Common
> as a submodule. While working on project A, the need arises to modify
> Common, so the developer changes it there, commits, pushes the change
> to Common, then commits and pushes the change to A. To update B and
> C, they would have to change to each of those projects, run a git
> pull, then git submodule update, and compile and test.
> 
> Is that correct? If so, it's a lot more work than letting a symlink
> propagate the change to all the other projects. Of course, since
> Windows doesn't have symlinks...

That is correct, but you only need to do the change if the projects
B, C, D, n... actually requires the new feature/bugfix introduced in
Common. Otherwise you can just leave it as-is.

Besides, when an important bugfix is added to Common, you probably
want to cut new releases from B, C, D, n... anyway, so it still
means doing stuff to those repositories.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231