symbolic link management in git-archive

symbolic link management in git-archive

[Sergio Callegari]

Hi,

I guess the answer is "no" or "not yet", but is there a way to tell the zip
backend of git-archive to follow symbolic links rather than to store them?

Sergio

Re: symbolic link management in git-archive

[Miklos Vajna]

[-- Attachment #1: Type: text/plain, Size: 363 bytes --]

On Thu, Mar 27, 2008 at 11:29:28AM +0000, Sergio Callegari <sergio.callegari@gmail.com> wrote:
> I guess the answer is "no" or "not yet", but is there a way to tell the zip
> backend of git-archive to follow symbolic links rather than to store them?

how would that handle a '. -> foo' symlink? following such a recursion
would lead to an infinite loop, i guess.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: symbolic link management in git-archive

[Johannes Schindelin]

Hi,

On Thu, 27 Mar 2008, Miklos Vajna wrote:

> On Thu, Mar 27, 2008 at 11:29:28AM +0000, Sergio Callegari <sergio.callegari@gmail.com> wrote:
> > I guess the answer is "no" or "not yet", but is there a way to tell the zip
> > backend of git-archive to follow symbolic links rather than to store them?
> 
> how would that handle a '. -> foo' symlink? following such a recursion
> would lead to an infinite loop, i guess.

Don't forget '/ -> foo'.

Ciao,
Dscho

Re: symbolic link management in git-archive

[Sergio Callegari]

Johannes Schindelin <Johannes.Schindelin <at> gmx.de> writes:

> 
> Hi,
> 
> On Thu, 27 Mar 2008, Miklos Vajna wrote:
> 
> > On Thu, Mar 27, 2008 at 11:29:28AM +0000, Sergio Callegari <sergio.callegari
<at> gmail.com> wrote:
> > > I guess the answer is "no" or "not yet", but is there a way to tell the zip
> > > backend of git-archive to follow symbolic links rather than to store them?
> > 
> > how would that handle a '. -> foo' symlink? following such a recursion
> > would lead to an infinite loop, i guess.
> 
> Don't forget '/ -> foo'.
> 
> Ciao,
> Dscho
> 
> 

My question was inspired by the fact that the unix version of the zip program
has a switch, precisely to decide whether to follow or to store links.
I believe that the reason why this switch exist is clear: for some mysterious
reason the world is populated by OSs that do not understand symlinks and while
someone finds out why it is sensible to have workarounds. Obviously zip archives
with stored links are completely useless on these OSs.
BTW, note that also tar has a similar switch.
 
I believe that cases like the foo -> . or the foo -> /  (or even foo -> ..) can
be solved by either:

- limiting the symlink dereferentiation only to symlinks pointing to plain files

or

- leaving it to the user not to ask for link following when he knows that he has
such types of links

Personally I prefer the second. In the end:
- If these types of link exist, it is clear that the git-managed-stuff is not
made for certain OSs, so the user should know that asking for link following
makes no sense
- The user should know that there are some commands that might be dangerous
(think rm -fr ~), just warn them in the manual.
- The maximum risk here is to have the command never stop and fill the disk.

Sergio

Re: symbolic link management in git-archive

[Junio C Hamano]

Sergio Callegari <sergio.callegari@gmail.com> writes:

> I guess the answer is "no" or "not yet", but is there a way to tell the zip
> backend of git-archive to follow symbolic links rather than to store them?

I am not sure what you mean.  Are you tracking a symbolic link X that
points at Y in your revision and expecting git-archive to include whatever
happens to be at Y (which may or may not even exist) when you run the
command?

If that is the case, the answer is "no" and "will never happen".  If you
are tracking a symbolic link X that points at Y, the information git
tracks is the fact that there is a symbolic link X that points at Y, and
not what Y happens to look like at a random moment.  Change to Y is not
tracked by git so why should you get different output from git-archive of
the same revision before and after you modify Y which is not part of the
revision to begin with?

If that is not what you are asking, please restate the question.

Re: symbolic link management in git-archive

[Sergio Callegari]

Junio C Hamano <gitster <at> pobox.com> writes:

> 
> Sergio Callegari <sergio.callegari <at> gmail.com> writes:
> 
> > I guess the answer is "no" or "not yet", but is there a way to tell the
> > zip
> > backend of git-archive to follow symbolic links rather than to store 
> > them?
> 
> I am not sure what you mean.  Are you tracking a symbolic link X that
> points at Y in your revision and expecting git-archive to include 
> whatever
> happens to be at Y (which may or may not even exist) when you run the
> command?

Yes, this is the case.

> If that is the case, the answer is "no" and "will never happen".  If you
> are tracking a symbolic link X that points at Y, the information git
> tracks is the fact that there is a symbolic link X that points at Y,

... I obviously agree with this!

> and
> not what Y happens to look like at a random moment.

Well, certainly if the zip backend of git-archive in addition to the -1 ... -9
switches also supported a --dereference switch, I would not run git-archive
--format=zip --dereference at "random moments".  I would run it when I am sure
that things pointed at by the symbolic links in the tracked project actually
point at sensible things.

> Change to Y is not
> tracked by git

Unless Y is also in the tracked project...

> so why should you get different output from git-archive of
> the same revision before and after you modify Y which is not part of the
> revision to begin with?

Basically, because if I make a zip archive instead of a git bundle, I want to
store (and probably give to someone else) a copy of what _I see_ at a certain
instant in time which might not exactly coincide with the tracked state.

Note that if Y is outside of the tracked project and I make an archive, and then
I give the archive to my friend X, Mr. X will see the same symbolic link, but
still a completely and randomly different content than I do, depending on where
he is unpacking the archive.

But to get to a more practical case, my situation is this:

I have a project where I need to have the same content in multiple places,
otherwise development tools are not happy.
Since I do most of the development on Linux, I use symbolic links. This is very
good, not just because I save space, but particularly because I am sure that the
content cannot loose coherency (which would be very bad) at the different
places.

Every now and then, to distribute snapshots of the project, I run git archive to
make a zip file, that I give to other people.  Unfortunately, some of them use
Windows, where the symbolic links appear as one-liner text files. And obviously
they cannot compile anything and they complain.

So I would like git archive to be able to make zip archives with the symbolic
links /resolved/.

what I am asking is not to modify the standard behaviour of git archive to
follow links, rather to have a switch to activate the dereferencing behaviour
only when needed.  In the end git archive is a nice shorthand for a checkout and
a successive run of zip or tar and both zip and tar have a switch to control
this dereferencing behaviour (BTW, zip on my distro dereferences by default, the
switch is to store symbolic links).

Sergio

Re: symbolic link management in git-archive

[Junio C Hamano]

Sergio Callegari <sergio.callegari@gmail.com> writes:

> Unless Y is also in the tracked project...
> ...
> Note that if Y is outside of the tracked project and I make an archive,
> and then I give the archive to my friend X, Mr. X will see the same
> symbolic link, but still a completely and randomly different content
> than I do, depending on where he is unpacking the archive.

If you _do_ keep track of Y in a separate repository, I think two archives
(the one that has a pointer to Y, and the other that is taken from the
repository Y _at the revision you are using_), would solve that more
naturally.  Then the version markers recorded in the archives would still
be valid.

	Side note.  If we ever teach git-archive to create a recursive
	tarball that contains a submodule, we should be doing something
	like that, not necessarily as two separate tarballs but possibly
	with a single tarball that has two comments that describe the
	revision of the toplevel and the submodule.

> ...  In the end git archive is a nice shorthand for a checkout and a
> successive run of zip or tar and both zip and tar have a switch to
> control this dereferencing behaviour (BTW, zip on my distro dereferences
> by default, the switch is to store symbolic links).

Under such an option, at least the comment in the archive (both for zip
and tar) that notes which revision the tarball was taken from should be
omitted.  As long as that is done, I think it is Ok to have such an
optional behaviour.

Re: symbolic link management in git-archive

[Sergio Callegari]

Junio C Hamano <gitster <at> pobox.com> writes:

> 
> Under such an option, at least the comment in the archive (both for zip
> and tar) that notes which revision the tarball was taken from should be
> omitted.  As long as that is done, I think it is Ok to have such an
> optional behaviour.
> 

yes, I overlooked this, but it makes perfect sense: if it is not an exact copy
of the tracked state, it should not pretend to be so by indicating a revision.

Re: symbolic link management in git-archive

[René Scharfe]

Sergio Callegari schrieb:
> I have a project where I need to have the same content in multiple places,
> otherwise development tools are not happy.
> Since I do most of the development on Linux, I use symbolic links. This is very
> good, not just because I save space, but particularly because I am sure that the
> content cannot loose coherency (which would be very bad) at the different
> places.
> 
> Every now and then, to distribute snapshots of the project, I run git archive to
> make a zip file, that I give to other people.  Unfortunately, some of them use
> Windows, where the symbolic links appear as one-liner text files. And obviously
> they cannot compile anything and they complain.
> 
> So I would like git archive to be able to make zip archives with the symbolic
> links /resolved/.

Windows 2000 and up has support for symbolic links; it's just strangely
restricted, e.g. on Windows 2000 you can only link to directories and
you have to use tools that aren't shipped with the OS to create them.
Microsoft even calls them differently; here's a good starting point for
more information: http://en.wikipedia.org/wiki/NTFS_junction_point

Arguably, your unzip program should create junction points for symlinks
in zip files.  I wouldn't be surprised if none of the existing ones
support that, though; junction points have been left undocumented for a
long time.  It's also possible that they'd understand a different zip
dialect for symlinks than the Info-Zip one produced by git-archive.

Would it be practical for you to distribute a junction point creation
tool like Mark Russinovich's Junction (except that Junction's EULA
forbids redistribution under most circumstances; see here:
http://www.microsoft.com/technet/sysinternals/FileAndDisk/Junction.mspx)
and a script that creates these symlinks for your audience?


It's harder for git-archive to support following symlinks than for e.g.
GNU tar.  The reason is that the former operates on git objects, not
files, directories or symlinks.  In order to follow a symlink it would
need to evaluate the symlink, follow it and then add actual files and
directories to the archive.

For your purposes, perhaps a slightly different implementation might be
sufficient: namely to follow only relative symlinks that point to
tracked objects.  That way you still get a repeatable result and (most
importantly) git-archive doesn't need to look at files and directories,
it can stay safely in git land.  Would such a way of operation be useful
to you?

Thanks,
René

Re: symbolic link management in git-archive

[Sergio Callegari]

René Scharfe wrote:

> Windows 2000 and up has support for symbolic links; it's just strangely
> restricted, [...] here's a good starting point for
> more information: http://en.wikipedia.org/wiki/NTFS_junction_point
>
> Arguably, your unzip program should create junction points for symlinks
> in zip files.  I wouldn't be surprised if none of the existing ones
> support that, though;
>
> Would it be practical for you to distribute a junction point creation
> tool like Mark Russinovich's Junction (except that Junction's EULA
> forbids redistribution under most circumstances; see here:
> http://www.microsoft.com/technet/sysinternals/FileAndDisk/Junction.mspx)
> and a script that creates these symlinks for your audience?
>   
Thank you very much for the detailed explanation. Unfortunately, I do 
not think that Junction can be an option.  My colleagues using Windows 
tend to be a bit "conservative" about the tools they use.  If they 
navigate filesystem contents with the graphical tool, and they 
look-at/expand the content of a zip file from explorer they expect that 
it should be immediately right, otherwise there must be something wrong 
in the way /I/ create zip files.  They would not appreciate having to 
unzip the file and then run an additional program on it to fix the 
unzipped stuff.
> It's harder for git-archive to support following symlinks than for e.g.
> GNU tar.  The reason is that the former operates on git objects, not
> files, directories or symlinks.  In order to follow a symlink it would
> need to evaluate the symlink, follow it and then add actual files and
> directories to the archive.
>
> For your purposes, perhaps a slightly different implementation might be
> sufficient: namely to follow only relative symlinks that point to
> tracked objects.  That way you still get a repeatable result and (most
> importantly) git-archive doesn't need to look at files and directories,
> it can stay safely in git land.  Would such a way of operation be useful
> to you?
>   
Absolutely positive.  This would be already a great improvement, 
fulfilling 99.9% of needs.

Sergio