How should I handle binary file with GIT

How should I handle binary file with GIT

[moreau francis]

Hi,

I'd like to use git to keep track of a documentation repository. This repo is
mainly composed by text file and the documenation is generated by asciidoc.
People who are using my repo and updating some docs which may include new
images can't send their whole work through patches.

For now they only send me the text updates through patch and attach new images
with the patch email. Then I do:

        $ git am < text_only_patch
        $ git reset --soft HEAD^
        $ git add <new images>
        $ git commit -a -C ORIG_HEAD

Now my question: is it the best way to achieve this process ?

thanks for you answers

Francis 


	

	
		
___________________________________________________________________________ 
Nouveau : téléphonez moins cher avec Yahoo! Messenger ! Découvez les tarifs exceptionnels pour appeler la France et l'international.
Téléchargez sur http://fr.messenger.yahoo.com

Re: How should I handle binary file with GIT

[Junio C Hamano]

moreau francis <francis_moreau2000@yahoo.fr> writes:

> For now they only send me the text updates through patch and attach new images
> with the patch email. Then I do:
>
>         $ git am < text_only_patch
>         $ git reset --soft HEAD^
>         $ git add <new images>
>         $ git commit -a -C ORIG_HEAD
>
> Now my question: is it the best way to achieve this process ?

If I were doing that today, I would be doing almost exactly the
above sequence, or:

	$ git am patch
        $ git add <new images>
        $ git commit -a --amend

It _might_ make sense to adopt a well-defined binary patch
format (or if there is no prior art, introduce our own) and
support that format with both git-diff-* brothers and git-apply,
but that would be a bit longer term project.


        

Re: How should I handle binary file with GIT

[moreau francis]

--- Junio C Hamano <junkio@cox.net> a écrit :

> It _might_ make sense to adopt a well-defined binary patch
> format (or if there is no prior art, introduce our own) and
> support that format with both git-diff-* brothers and git-apply,
> but that would be a bit longer term project.
> 

well maybe it's just stupid, but why not simply transforming binary files into
ascii files (maybe by using uuencode) before  using git-diff-* brothers and
git-apply ?

Francis


	

	
		
___________________________________________________________________________ 
Nouveau : téléphonez moins cher avec Yahoo! Messenger ! Découvez les tarifs exceptionnels pour appeler la France et l'international.
Téléchargez sur http://fr.messenger.yahoo.com

Re: How should I handle binary file with GIT

[Nicolas Pitre]

On Wed, 5 Apr 2006, Junio C Hamano wrote:

> It _might_ make sense to adopt a well-defined binary patch
> format (or if there is no prior art, introduce our own) and
> support that format with both git-diff-* brothers and git-apply,
> but that would be a bit longer term project.

What about simply using diff-delta and encoding its output with base64?


Nicolas

Re: How should I handle binary file with GIT

[moreau francis]

--- Junio C Hamano <junkio@cox.net> a écrit :

> If I were doing that today, I would be doing almost exactly the
> above sequence, or:
> 
> 	$ git am patch
>         $ git add <new images>
>         $ git commit -a --amend
> 

BTW, what does "--amend" option do ? It doesn't seem to be documented anywhere.

Francis


	

	
		
___________________________________________________________________________ 
Nouveau : téléphonez moins cher avec Yahoo! Messenger ! Découvez les tarifs exceptionnels pour appeler la France et l'international.
Téléchargez sur http://fr.messenger.yahoo.com

Re: How should I handle binary file with GIT

[Nicolas Pitre]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 723 bytes --]

On Wed, 5 Apr 2006, moreau francis wrote:

> 
> --- Junio C Hamano <junkio@cox.net> a écrit :
> 
> > It _might_ make sense to adopt a well-defined binary patch
> > format (or if there is no prior art, introduce our own) and
> > support that format with both git-diff-* brothers and git-apply,
> > but that would be a bit longer term project.
> > 
> 
> well maybe it's just stupid, but why not simply transforming binary files into
> ascii files (maybe by using uuencode) before  using git-diff-* brothers and
> git-apply ?

Imagine if the only difference between two versions of the same file is 
a single byte inserted at the very beginning.  The uuencode would then 
be totally different between the two files.


Nicolas

Re: How should I handle binary file with GIT

[moreau francis]

--- Nicolas Pitre <nico@cam.org> a écrit :

> On Wed, 5 Apr 2006, moreau francis wrote:
> > 
> > well maybe it's just stupid, but why not simply transforming binary files
> into
> > ascii files (maybe by using uuencode) before  using git-diff-* brothers and
> > git-apply ?
> 
> Imagine if the only difference between two versions of the same file is 
> a single byte inserted at the very beginning.  The uuencode would then 
> be totally different between the two files.
> 

ok uuencode was just a bad example for encoding...

Francis



	

	
		
___________________________________________________________________________ 
Nouveau : téléphonez moins cher avec Yahoo! Messenger ! Découvez les tarifs exceptionnels pour appeler la France et l'international.
Téléchargez sur http://fr.messenger.yahoo.com

Re: How should I handle binary file with GIT

[Jakub Narebski]

Junio C Hamano wrote:

> It _might_ make sense to adopt a well-defined binary patch
> format (or if there is no prior art, introduce our own) and
> support that format with both git-diff-* brothers and git-apply,
> but that would be a bit longer term project.

bsdiff? http://www.daemonology.net/bsdiff/
EDelta? http://www.diku.dk/~jacobg/edelta/
Xdelta? http://xdelta.blogspot.com/

IIRC bsdiff is used by Firefox to distribute binary software updates.
Xdelta is generic (not optimized for binaries like bsdiff and edelta), but
supposedly offers worse compression (bigger diffs).

-- 
Jakub Narebski
Warsaw, Poland

Re: How should I handle binary file with GIT

[Nicolas Pitre]

On Wed, 5 Apr 2006, Jakub Narebski wrote:

> Junio C Hamano wrote:
> 
> > It _might_ make sense to adopt a well-defined binary patch
> > format (or if there is no prior art, introduce our own) and
> > support that format with both git-diff-* brothers and git-apply,
> > but that would be a bit longer term project.
> 
> bsdiff? http://www.daemonology.net/bsdiff/
> EDelta? http://www.diku.dk/~jacobg/edelta/
> Xdelta? http://xdelta.blogspot.com/
> 
> IIRC bsdiff is used by Firefox to distribute binary software updates.
> Xdelta is generic (not optimized for binaries like bsdiff and edelta), but
> supposedly offers worse compression (bigger diffs).

We already have our own delta code for pack storage.


Nicolas

Re: How should I handle binary file with GIT

[Randal L. Schwartz]

>>>>> "Nicolas" == Nicolas Pitre <nico@cam.org> writes:

>> IIRC bsdiff is used by Firefox to distribute binary software updates.
>> Xdelta is generic (not optimized for binaries like bsdiff and edelta), but
>> supposedly offers worse compression (bigger diffs).

Nicolas> We already have our own delta code for pack storage.

I think the issue is related to being able to cherry-pick and merge
when binaries are involved.  I've been worried about that myself.
How well are binaries supported these days for all the operations
we're taking for granted?  When is a "diff" expected to be a real
"diff" and not just "binary files differ"?

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

Re: How should I handle binary file with GIT

[Shawn Pearce]

"Randal L. Schwartz" <merlyn@stonehenge.com> wrote:
> >>>>> "Nicolas" == Nicolas Pitre <nico@cam.org> writes:
> 
> >> IIRC bsdiff is used by Firefox to distribute binary software updates.
> >> Xdelta is generic (not optimized for binaries like bsdiff and edelta), but
> >> supposedly offers worse compression (bigger diffs).
> 
> Nicolas> We already have our own delta code for pack storage.
> 
> I think the issue is related to being able to cherry-pick and merge
> when binaries are involved.  I've been worried about that myself.
> How well are binaries supported these days for all the operations
> we're taking for granted?  When is a "diff" expected to be a real
> "diff" and not just "binary files differ"?

The clearly safe approach is to include the full SHA1 ID of the
old object the patch was created from and use the xdelta in the
patch only as a means of transporting a compressed form of the new
version of the object.  If git-diff starts to export say a base 64
encoding of the xdelta then it should also include the full SHA1
ID for binary files, even if --full-index wasn't given.

git-apply should only apply an xdelta patch to the exact same
old object.  If the tree currently has a different object at that
path then reject the patch entirely.

If a path has a different object then the patch was based on then
we can do one of two things to be ``nice'' to the human:

  - If the old blob exists in the repository (it just isn't the
  current version at that path) then generate a temporary merge
  file holding the old blob with the delta applied.  The user can
  then finish the merge with whatever tool understands that binary
  file format, or do the merge by hand.

  - Supply a ``do it anyway'' flag to git-apply.  If this flag is
  given on the command line then the binary file is patched even
  though the object versions differ.  For some binary file formats
  this may actually be a valid thing to do.  But it probably isn't
  for a very large percentage of known file formats.

I could see some cases where it might be nice to be able to perform
specialized merge handling of binary files via hooks or filters.

For example *.tar.gz, *.zip, *.jar - these files are all just
compressed trees.  They should be somewhat mergeable with the same
semantics as other trees in GIT.  Of course one could just unpack
these into a directory and let GIT track the directory instead,
but this is rather inconvenient in a Java project.  :-)

If I recall correctly OpenOffice document files are XML compressed
into ZIP archives.  The XML *might* diff/patch cleanly as plain text.
The other resources in that archive are typically binary graphic
files and the like, which of course wouldn't diff/patch nicely.
But being able to diff/patch the main content might be semi-useful.

-- 
Shawn.

Re: How should I handle binary file with GIT

[Nicolas Pitre]

On Wed, 5 Apr 2006, Randal L. Schwartz wrote:

> >>>>> "Nicolas" == Nicolas Pitre <nico@cam.org> writes:
> 
> >> IIRC bsdiff is used by Firefox to distribute binary software updates.
> >> Xdelta is generic (not optimized for binaries like bsdiff and edelta), but
> >> supposedly offers worse compression (bigger diffs).
> 
> Nicolas> We already have our own delta code for pack storage.
> 
> I think the issue is related to being able to cherry-pick and merge
> when binaries are involved.  I've been worried about that myself.
> How well are binaries supported these days for all the operations
> we're taking for granted?  When is a "diff" expected to be a real
> "diff" and not just "binary files differ"?

First of all, does cherry-picking binary patches is a sensible thing to 
do?

Do you expect, say, a Word document, a JPEG image, or an MP3 file to 
still be valid and error free if two binary patches modifying a 
different part of the same file (same revision) are successively 
applied?  I seriously doubt it.

And what do you do with conflicts?  Using diff3 might be sensible for 
text data, but for binaries you really need a tool that understands the 
type of data your binary contains, which means one tool for each 
possible type of binary data which is outside the scope of GIT.

For example, if you patch a .wav file adding some data, then you end up 
with the additional samples and a new length in the file header.  If 
another patch to that .wav is applied, then it is easy to find the 
"surrounding context" where the second patch is adding/removing some 
other samples, but then you really needs knowledge about the .wav format 
to handle the conflict that will occur on the .wav header modification.

And so on for all possible binary types.

So IMHO a binary patch format is only useful for easy _transport_ along 
with other text patches.  And the binary patch must either apply 
perfectly against the same source file or it must not apply at all.  
That's the only sensible accommodation we can do with a generic binary 
patch format.

When the patch doesn't apply to your tree, then nothing prevents you 
from hooking a dedicated tool that will pick up the original file, the 
reconstructed remote version according to the binary patch you received 
and your own modified version so that tool can process them and do the 
necessary changes with proper knowledge of the data format.


Nicolas

Re: How should I handle binary file with GIT

[Nicolas Pitre]

On Wed, 5 Apr 2006, Shawn Pearce wrote:

> The clearly safe approach is to include the full SHA1 ID of the
> old object the patch was created from and use the xdelta in the
> patch only as a means of transporting a compressed form of the new
> version of the object.  If git-diff starts to export say a base 64
> encoding of the xdelta then it should also include the full SHA1
> ID for binary files, even if --full-index wasn't given.
> 
> git-apply should only apply an xdelta patch to the exact same
> old object.  If the tree currently has a different object at that
> path then reject the patch entirely.

Amen.  Exactly what I just said.


Nicolas

Re: How should I handle binary file with GIT

[Junio C Hamano]

merlyn@stonehenge.com (Randal L. Schwartz) writes:

> I think the issue is related to being able to cherry-pick and merge
> when binaries are involved.  I've been worried about that myself.
> How well are binaries supported these days for all the operations
> we're taking for granted?  When is a "diff" expected to be a real
> "diff" and not just "binary files differ"?

First of all, binary files are handled by cherry-pick and merge
without needing to involve "diff"+"patch" (which is not so
useful for binary files anyway).  They use 3-way read-tree merge
which compares the object names and leave the index unmerged if
there are conflicting changes, so you should be able to sort it
out by running up to three "git-cat-file blob $sha1".

What involves "diff"+"patch" are rebases and processing mailed-in
patches as in the example by the original poster.

In our diff output, we record the blob object name of preimage
and postimage, along with filemode, on the "index" line.
git-apply does not do anything with it by default, but if:

 - --binary flag is given,

 - the postimage blob is already available locally, and,

 - the file the patch is being applied to is the same as the
   recorded preimage,

then the file is _replaced_ with the postimage.

This is good enough for git-rebase (which uses format-patch
piped to am) and is safe (we do not "apply delta" -- only
replace when the file "being patched" matches the recorded
preimage).  It does not do any good for transferring a postimage
that the person who applies the patch does not yet have.

I think "applying delta" to a binary file is not very useful
thing to do.  Depending on the nature of the file being patched,
it may produce a perfectly good result, but verifying if the
result makes sense by the end user and hand-fixing it if does
not, which can be done for text files, is near impossible for
binary files.  "replace with postimage only when you are
applying to the same preimage" rule would be the only practical,
sane thing.

If we wanted to use the patch+diff (i.e. "format-patch,
send-email, and then am" workflow) to transfer new version of
binary files to a recipient, which I think is useful in some
projects, the sanest way to handle this is probably to add
Nico's delta, going from preimage to postimage, encoded for
safer transport, to our diff output.  For safety and sanity, we
will not "apply" the patch unless the patched file exactly
matches the preimage that is recorded in the diff, and as long
as the recipient has the preimage, such a patch would be able to
reproduce the postimage and hopefully be smaller than
transferring the whole thing.

We've been trying to keep our diff output reversible (e.g. we
show what the filemode of the preimage is), so if we take the
above route, it probably should record deltas for both going
from preimage to postimage _and_ going the other way (unless
xdelta can be applied in-reverse, which I do not think is the
case).

Of course, to be _completely_ generic, you could include both
compressed then uuencoded preimage and postimage, and let the
recipient sort it out.  An advantage of that approach is that
the applicability of such a "patch" improves as the tools to
apply it improve, after the patch was originally generated.  I
however think that is only a theoretical advantage, not a very
practical one.

Re: How should I handle binary file with GIT

[Randal L. Schwartz]

>>>>> "Junio" == Junio C Hamano <junkio@cox.net> writes:

Junio> If we wanted to use the patch+diff (i.e. "format-patch,
Junio> send-email, and then am" workflow) to transfer new version of
Junio> binary files to a recipient, which I think is useful in some
Junio> projects, the sanest way to handle this is probably to add
Junio> Nico's delta, going from preimage to postimage, encoded for
Junio> safer transport, to our diff output.

This is what I was looking for, and thanks for confirming that at least within
a local respository, everything already works.  Yeay.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

Re: How should I handle binary file with GIT

[Marco Roeland]

On Wednesday April 5th 2006 moreau francis wrote:

> BTW, what does "--amend" option do ? It doesn't seem to be documented anywhere.

This is the original commit text that introduced it:

diff-tree b4019f045646b1770a80394da876b8a7c6b8ca7b (from d320a5437f8304cf9ea3ee1898e49d643e005738)
Author: Junio C Hamano <junkio@cox.net>
Date:   Thu Mar 2 21:04:05 2006 -0800

    git-commit --amend
    
    The new flag is used to amend the tip of the current branch.  Prepare
    the tree object you would want to replace the latest commit as usual
    (this includes the usual -i/-o and explicit paths), and the commit log
    editor is seeded with the commit message from the tip of the current
    branch.  The commit you create replaces the current tip -- if it was a
    merge, it will have the parents of the current tip as parents -- so the
    current top commit is discarded.
    
    It is a rough equivalent for:
    
    	$ git reset --soft HEAD^
    	$ ... do something else to come up with the right tree ...
    	$ git commit -c ORIG_HEAD
    
    but can be used to amend a merge commit.
    
    Signed-off-by: Junio C Hamano <junkio@cox.net>

So in the original context you can add separate binaries to a commit
of only text files that you just rescued from CVS or something and then
change the commit to include these binaries as well.

I've sent a separate patch for the documentation for git-commit using
Junio's clear explanation.
-- 
Marco Roeland

Re: How should I handle binary file with GIT

[Nicolas Pitre]

On Wed, 5 Apr 2006, Junio C Hamano wrote:

> If we wanted to use the patch+diff (i.e. "format-patch,
> send-email, and then am" workflow) to transfer new version of
> binary files to a recipient, which I think is useful in some
> projects, the sanest way to handle this is probably to add
> Nico's delta, going from preimage to postimage, encoded for
> safer transport, to our diff output.  For safety and sanity, we
> will not "apply" the patch unless the patched file exactly
> matches the preimage that is recorded in the diff, and as long
> as the recipient has the preimage, such a patch would be able to
> reproduce the postimage and hopefully be smaller than
> transferring the whole thing.

Exactly the point.

> We've been trying to keep our diff output reversible (e.g. we
> show what the filemode of the preimage is), so if we take the
> above route, it probably should record deltas for both going
> from preimage to postimage _and_ going the other way (unless
> xdelta can be applied in-reverse, which I do not think is the
> case).

You cannot reverse a delta.  However if you were able to apply a delta 
from preimage to postimage that means you must already have had preimage 
in your object store.  Therefore reverting such a patch would simply 
involve restoring preimage.

> Of course, to be _completely_ generic, you could include both
> compressed then uuencoded preimage and postimage, and let the
> recipient sort it out.

I think this is just too much and besides the point of a diff.  If the 
work flow is so convoluted such that the simple binary patch as a delta 
doesn't apply then it would probably be a better idea to simply transfer 
those binaries as email attachments.  In other words, if a binary patch 
transfer mechanism is added, it should cover the common case and leave 
the rest for a better process like git-fetch/pull.


Nicolas

Re: How should I handle binary file with GIT

[Junio C Hamano]

Nicolas Pitre <nico@cam.org> writes:

> On Wed, 5 Apr 2006, Junio C Hamano wrote:
>
>> We've been trying to keep our diff output reversible (e.g. we
>> show what the filemode of the preimage is), so if we take the
>> above route, it probably should record deltas for both going
>> from preimage to postimage _and_ going the other way (unless
>> xdelta can be applied in-reverse, which I do not think is the
>> case).
>
> You cannot reverse a delta.  However if you were able to apply a delta 
> from preimage to postimage that means you must already have had preimage 
> in your object store.  Therefore reverting such a patch would simply 
> involve restoring preimage.

The case I had in mind was where you shipped a tarball of the
tip to somebody (or "a shallow clone"), and after seeing him
having problems with that release, sending him a patch telling
him "reverting this might help, could you please give it a try?"

Of course you could be nicer to him and generate the reverse
diff on your end in such a case instead.