git and binary files

git and binary files

[Petko Manolov]

 	Hello there,

I've searched the web for an answer, but i didn't find it and decided to 
take the risk of being yelled at.  And post it here.

Some of my projects require having binary files (firmware and other stuff) 
somewhere in the tree structure.  Unfortunately these files are big - 50MB 
and more.  After a couple of new versions arrive (and get committed) i end 
up with a repository way too big than necessary.

The nature of these binary files is such that i don't care neither about 
their history nor older versions.  Hence the question:  is there an easy 
way to tell git not to bother about the history of these particular files 
and keep the most recent version only?


cheers,
Petko

Re: git and binary files

[David Symonds]

On Jan 16, 2008 9:34 PM, Petko Manolov <petkan@nucleusys.com> wrote:
>
> The nature of these binary files is such that i don't care neither about
> their history nor older versions.  Hence the question:  is there an easy
> way to tell git not to bother about the history of these particular files
> and keep the most recent version only?

If you don't care about versioning those files, why would you use a
version control system? Just store them somewhere else, and use
symlinks.


Dave.

Re: git and binary files

[Johannes Schindelin]

Hi,

On Wed, 16 Jan 2008, Petko Manolov wrote:

> Some of my projects require having binary files (firmware and other 
> stuff) somewhere in the tree structure.  Unfortunately these files are 
> big - 50MB and more.  After a couple of new versions arrive (and get 
> committed) i end up with a repository way too big than necessary.
> 
> The nature of these binary files is such that i don't care neither about 
> their history nor older versions.  Hence the question:  is there an easy 
> way to tell git not to bother about the history of these particular 
> files and keep the most recent version only?

Your subject is a little bit misleading, no?  It's not about the 
binariness (git handles binary files just fine, thankyouverymuch), but 
about the not-tracking them.

The answer is no.  You cannot ask git to have the newest version of 
something, but not the old ones.  It contradicts the distributedness of 
git, too.

Hth,
Dscho

Re: git and binary files

[Petko Manolov]

On Wed, 16 Jan 2008, David Symonds wrote:

> If you don't care about versioning those files, why would you use a
> version control system? Just store them somewhere else, and use
> symlinks.

That is certainly a way of doing it.  However, it will be much simpler and 
fast to be able to "git clone" and then "git pull" every once in a while. 
The alternative involves "cp -a" or most likely "scp -r" the binaries 
along with the repository and you can never be sure that both are in sync.


 		Petko

Re: git and binary files

[Petko Manolov]

On Wed, 16 Jan 2008, Johannes Schindelin wrote:

> Your subject is a little bit misleading, no?  It's not about the 
> binariness (git handles binary files just fine, thankyouverymuch), but 
> about the not-tracking them.

You're absobloodylutely correct.  I was too preoccupied defining my problem 
in a better way, which left the subject kind of dumb.  Well, quite dumb. 
:-)

> The answer is no.  You cannot ask git to have the newest version of 
> something, but not the old ones.  It contradicts the distributedness of 
> git, too.

I don't agree here.  Assume that whatever you're working on require 
firmware for a device that won't change during the lifetime of the 
software project.  The newest version of the said firmware is mostly 
bugfixes and you basically don't want to revert to the older ones. 
Consider the microcode for modern Pentiums, Core 2, etc.

What i am trying to suggest is that there might be cases when you need 
something in the repository, but you don't want GIT to keep it's history 
nor it's predecessors.  Leaving it out breaks the atomicity of such 
repository and makes the project management more complex.

There's a few examples out there that shows how to solve this, but it 
seems inconvenient and involves branching, cloning, etc.  Isn't it 
possible to add something like:

 	"git nohistory firmware.bin"

or
 	"git nohistory -i-understand-this-might-be-dangerous firmware.bin"



cheers,
Petko

Re: git and binary files

[Johannes Schindelin]

Hi,

On Wed, 16 Jan 2008, Petko Manolov wrote:

> On Wed, 16 Jan 2008, David Symonds wrote:
> 
> > If you don't care about versioning those files, why would you use a 
> > version control system? Just store them somewhere else, and use 
> > symlinks.
> 
> That is certainly a way of doing it.  However, it will be much simpler 
> and fast to be able to "git clone" and then "git pull" every once in a 
> while. The alternative involves "cp -a" or most likely "scp -r" the 
> binaries along with the repository and you can never be sure that both 
> are in sync.

I think that you're missing the point of version control.  It's not only 
about having an up-to-date source tree, but also about being able to go 
back to a certain revision.

What you want is most likely covered by "rsync -au".

Hth,
Dscho

Re: git and binary files

[Jakub Narebski]

Petko Manolov <petkan@nucleusys.com> writes:

> On Wed, 16 Jan 2008, Johannes Schindelin wrote:
> 
> > The answer is no.  You cannot ask git to have the newest version of
> > something, but not the old ones.  It contradicts the distributedness
> > of git, too.
> 
> I don't agree here.  Assume that whatever you're working on require
> firmware for a device that won't change during the lifetime of the
> software project.  The newest version of the said firmware is mostly
> bugfixes and you basically don't want to revert to the older
> ones. Consider the microcode for modern Pentiums, Core 2, etc.
> 
> What i am trying to suggest is that there might be cases when you need
> something in the repository, but you don't want GIT to keep it's
> history nor it's predecessors.  Leaving it out breaks the atomicity of
> such repository and makes the project management more complex.
> 
> There's a few examples out there that shows how to solve this, but it
> seems inconvenient and involves branching, cloning, etc.  Isn't it
> possible to add something like:
> 
>  	"git nohistory firmware.bin"
> 
> or
>  	"git nohistory -i-understand-this-might-be-dangerous firmware.bin"

You can always tag a blob (like junio-gpg-pub tag in git.git repository),
but it wouldn't be in a working directory. But it would get distributed
on clone.

BTW. if those large binary files doesn't differ much between version,
they should get well compressed even if you would store them normally,
all revisions.

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: git and binary files

[Jeff King]

On Wed, Jan 16, 2008 at 03:39:06PM +0200, Petko Manolov wrote:

> What i am trying to suggest is that there might be cases when you need  
> something in the repository, but you don't want GIT to keep it's history  
> nor it's predecessors.  Leaving it out breaks the atomicity of such  
> repository and makes the project management more complex.

But not versioning some files while versioning others breaks the
atomicity of project version, which is at the core of git's model. There
is no such thing as "this file is at revision X, but that one is at
revision Y." There is only "the project is at revision X."

> There's a few examples out there that shows how to solve this, but it  
> seems inconvenient and involves branching, cloning, etc.  Isn't it  
> possible to add something like:
>
> 	"git nohistory firmware.bin"
>
> or
> 	"git nohistory -i-understand-this-might-be-dangerous firmware.bin"

Not easily. It goes against the underlying data model at the core of
git.

How big are your firmware files? How often do they change, and how large
are the changes? IOW, have you confirmed that repacking does not produce
an acceptable delta, meaning you get versioning for very low space cost?

-Peff

Re: git and binary files

[Petko Manolov]

On Wed, 16 Jan 2008, Johannes Schindelin wrote:

> I think that you're missing the point of version control.  It's not only 
> about having an up-to-date source tree, but also about being able to go 
> back to a certain revision.

No contradiction here.  In my case old source code will work perfectly 
with new binaries/firmware.  That's why i don't _need_ the history, only 
the latest stuff in order to save space.

I do realize that what i am talking about is statistically microscopic 
scenario, but it does exist.  If there's no such feature then i don't have 
much choice, but stick with my current way of doing things.

> What you want is most likely covered by "rsync -au".

Yeah, just like in the old days when "git pull" didn't do everything for 
you.


thanks,
Petko

Re: git and binary files

[Petko Manolov]

On Wed, 16 Jan 2008, Jakub Narebski wrote:

> You can always tag a blob (like junio-gpg-pub tag in git.git repository),
> but it wouldn't be in a working directory. But it would get distributed
> on clone.

Hm, how does it work?

> BTW. if those large binary files doesn't differ much between version, 
> they should get well compressed even if you would store them normally, 
> all revisions.

Unfortunately this is not the case.  These binary blobs are already 
compressed and/or encrypted and adding even a few bytes ends up storing 
new version in full size.


cheers,
Petko

Re: git and binary files

[Johannes Schindelin]

Hi,

On Wed, 16 Jan 2008, Petko Manolov wrote:

> On Wed, 16 Jan 2008, Johannes Schindelin wrote:
> 
> > I think that you're missing the point of version control.  It's not 
> > only about having an up-to-date source tree, but also about being able 
> > to go back to a certain revision.
> 
> No contradiction here.  In my case old source code will work perfectly 
> with new binaries/firmware.  That's why i don't _need_ the history, only 
> the latest stuff in order to save space.

No, you _do_ miss the point here.  You might _think_ that they work 
perfectly, but with revision control you want to have _exactly_ the same 
setup.  You want to be able to go back to a certain _revision_ (including 
the then-current firmware).

And that's what you don't want.  So git is not for you.

Ciao,
Dscho

Re: git and binary files

[Petko Manolov]

On Wed, 16 Jan 2008, Jeff King wrote:

> On Wed, Jan 16, 2008 at 03:39:06PM +0200, Petko Manolov wrote:
>
>> What i am trying to suggest is that there might be cases when you need
>> something in the repository, but you don't want GIT to keep it's history
>> nor it's predecessors.  Leaving it out breaks the atomicity of such
>> repository and makes the project management more complex.
>
> But not versioning some files while versioning others breaks the
> atomicity of project version, which is at the core of git's model. There
> is no such thing as "this file is at revision X, but that one is at
> revision Y." There is only "the project is at revision X."

Sigh.  You are right.

However, the said project is kind of exception.  The binaries are there 
from the very beginning - they are indivisible part of the project and it 
won't work without them.  This is why i am not worried if i revert to 
previous source code version, but actually check-out fresh binary - in my 
case it won't break things.

>> There's a few examples out there that shows how to solve this, but it
>> seems inconvenient and involves branching, cloning, etc.  Isn't it
>> possible to add something like:
>>
>> 	"git nohistory firmware.bin"
>>
>> or
>> 	"git nohistory -i-understand-this-might-be-dangerous firmware.bin"
>
> Not easily. It goes against the underlying data model at the core of
> git.

Damn, i knew you'd say something like this. :-)

> How big are your firmware files? How often do they change, and how large 
> are the changes? IOW, have you confirmed that repacking does not produce 
> an acceptable delta, meaning you get versioning for very low space cost?

Changes don't happen too often, but the size of everything binary in the 
tree easily goes to about 100MB.  Three commits later it ends up at about 
300MB...


cheers,
Petko

Re: git and binary files

[Jeff King]

On Wed, Jan 16, 2008 at 04:14:30PM +0200, Petko Manolov wrote:

>> How big are your firmware files? How often do they change, and how large 
>> are the changes? IOW, have you confirmed that repacking does not produce 
>> an acceptable delta, meaning you get versioning for very low space cost?
>
> Changes don't happen too often, but the size of everything binary in the  
> tree easily goes to about 100MB.  Three commits later it ends up at about  
> 300MB...

Right, as loose objects. Did you try running "git-gc" to repack?

-Peff

Re: git and binary files

[Jakub Narebski]

Petko Manolov wrote:
> On Wed, 16 Jan 2008, Jakub Narebski wrote:
> 
> > You can always tag a blob (like junio-gpg-pub tag in git.git repository),
> > but it wouldn't be in a working directory. But it would get distributed
> > on clone.
> 
> Hm, how does it work?

You use git-hash-object to put file (-t blob) into the object database.
It would return sha1 of added object. Use git-tag to create tag to blob
(use returned sha1 for head). You can get file (to stdout) with 
"git cat-file blob tagname^{blob}".

The file would be in object database, but not in working directory
by default.

> > BTW. if those large binary files doesn't differ much between version, 
> > they should get well compressed even if you would store them normally, 
> > all revisions.
> 
> Unfortunately this is not the case.  These binary blobs are already 
> compressed and/or encrypted and adding even a few bytes ends up storing 
> new version in full size.

Can't you store them uncompressed?

-- 
Jakub Narebski
Poland

Re: git and binary files

[Petko Manolov]

On Wed, 16 Jan 2008, Johannes Schindelin wrote:

> No, you _do_ miss the point here.  You might _think_ that they work 
> perfectly, but with revision control you want to have _exactly_ the same 
> setup.  You want to be able to go back to a certain _revision_ 
> (including the then-current firmware).

I _know_ older code will work with new binaries.  I know because i've done 
it many times and the application is the sort that is not going to forgive 
any frivolity.

Unfortunately this is very specific to what i'm doing and does not apply 
for 99.99% of what people usually need.

> And that's what you don't want.  So git is not for you.

I use git for SCV from day one.  It's great.  I was just thinking aloud 
about something i've stumbled upon.  ;-)


 		Petko

Re: git and binary files

[Petko Manolov]

On Wed, 16 Jan 2008, Jeff King wrote:

> On Wed, Jan 16, 2008 at 04:14:30PM +0200, Petko Manolov wrote:
>
>>> How big are your firmware files? How often do they change, and how large
>>> are the changes? IOW, have you confirmed that repacking does not produce
>>> an acceptable delta, meaning you get versioning for very low space cost?
>>
>> Changes don't happen too often, but the size of everything binary in the
>> tree easily goes to about 100MB.  Three commits later it ends up at about
>> 300MB...
>
> Right, as loose objects. Did you try running "git-gc" to repack?

I did "git repack -f -a -d", but it didn't reduce the repository size. 
Those binaries are already compressed so any change adds up their size 
once again.


cheers,
Petko

Re: git and binary files

[Jeff King]

On Wed, Jan 16, 2008 at 04:25:28PM +0200, Petko Manolov wrote:

>> Right, as loose objects. Did you try running "git-gc" to repack?
>
> I did "git repack -f -a -d", but it didn't reduce the repository size.  
> Those binaries are already compressed so any change adds up their size  
> once again.

OK, that was the answer I was looking for; it looks like you are out
of luck.

BTW, the main space-saver in repacking is _not_ compression, but rather
finding deltas between similar objects (e.g., two versions of the same
file that, although large, differ only by a small amount). So even
compressed files can still produce space savings during a repack, though
perhaps not as well because of randomness introduced by the compression.

As an experiment, it might be worth trying to store the uncompressed
versions instead (git will delta _and_ compress them for you).

-Peff

Re: git and binary files

[Wincent Colaiuta]

El 16/1/2008, a las 14:58, Petko Manolov escribió:

> On Wed, 16 Jan 2008, Johannes Schindelin wrote:
>
>> I think that you're missing the point of version control.  It's not  
>> only about having an up-to-date source tree, but also about being  
>> able to go back to a certain revision.
>
> No contradiction here.  In my case old source code will work  
> perfectly with new binaries/firmware.  That's why i don't _need_ the  
> history, only the latest stuff in order to save space.

You may be interested in the history, but the entire purpose of any  
version control system (not just Git) is to record exactly that:  
history.

If the exact contents of these large binaries *really* don't matter,  
as you say they don't, than why don't you just commit one and never  
touch it again?

Cheers,
Wincent

Re: git and binary files

[Petko Manolov]

On Wed, 16 Jan 2008, Jeff King wrote:

> OK, that was the answer I was looking for; it looks like you are out
> of luck.

Story of my life. :-)

> As an experiment, it might be worth trying to store the uncompressed
> versions instead (git will delta _and_ compress them for you).

I don't have them uncompressed.

I can try to convert those files into ascii format and then save them in 
the repository.  Since most changes are incremental git should be able to 
generate relatively small delta, which should compress well enough.

Thanks for the hint.


 		Petko

Re: git and binary files

[Petko Manolov]

On Wed, 16 Jan 2008, Jakub Narebski wrote:

> Petko Manolov wrote:
>> On Wed, 16 Jan 2008, Jakub Narebski wrote:
>>
>>> You can always tag a blob (like junio-gpg-pub tag in git.git repository),
>>> but it wouldn't be in a working directory. But it would get distributed
>>> on clone.
>>
>> Hm, how does it work?
>
> You use git-hash-object to put file (-t blob) into the object database.
> It would return sha1 of added object. Use git-tag to create tag to blob
> (use returned sha1 for head). You can get file (to stdout) with
> "git cat-file blob tagname^{blob}".

Sounds like i'll have to play with the above.  Thanks for the tip.

> The file would be in object database, but not in working directory
> by default.

Not a big problem.

>> Unfortunately this is not the case.  These binary blobs are already
>> compressed and/or encrypted and adding even a few bytes ends up storing
>> new version in full size.
>
> Can't you store them uncompressed?

Not really, but i can convert them into ascii format and store only the 
delta.  This will admittedly increase the initial size of the repository, 
but hopefully not by much.


 		Petko

Re: git and binary files

[Petko Manolov]

On Wed, 16 Jan 2008, Wincent Colaiuta wrote:

> If the exact contents of these large binaries *really* don't matter, as 
> you say they don't, than why don't you just commit one and never touch 
> it again?

Unfortunately those binaries does change, although the process is slow and 
not very frequent.  And this is why it pokes me in the eye - for changing 
a few bytes i end up with much larger repository.


 		Petko

Re: git and binary files

[Nicolas Pitre]

On Wed, 16 Jan 2008, Petko Manolov wrote:

> On Wed, 16 Jan 2008, Jakub Narebski wrote:
> 
> > Petko Manolov wrote:
> > > 
> > > Unfortunately this is not the case.  These binary blobs are already
> > > compressed and/or encrypted and adding even a few bytes ends up storing
> > > new version in full size.
> > 
> > Can't you store them uncompressed?
> 
> Not really, but i can convert them into ascii format and store only the delta.

If you don't have the original uncompressed unencrypted file, what will 
converting them to ascii actually give you?


Nicolas

Re: git and binary files

[Rogan Dawes]

Petko Manolov wrote:
> On Wed, 16 Jan 2008, Jeff King wrote:
> 
>> OK, that was the answer I was looking for; it looks like you are out
>> of luck.
> 
> Story of my life. :-)
> 
>> As an experiment, it might be worth trying to store the uncompressed
>> versions instead (git will delta _and_ compress them for you).
> 
> I don't have them uncompressed.
> 
> I can try to convert those files into ascii format and then save them in 
> the repository.  Since most changes are incremental git should be able 
> to generate relatively small delta, which should compress well enough.
> 
> Thanks for the hint.
> 
> 
>         Petko

That is unlikely to help, since git can find deltas in binary files just 
as easily as in text files. All you are doing is changing the encoding.

Rogan

Re: git and binary files

[Petko Manolov]

On Wed, 16 Jan 2008, Nicolas Pitre wrote:

> On Wed, 16 Jan 2008, Petko Manolov wrote:
>
>> On Wed, 16 Jan 2008, Jakub Narebski wrote:
>>
>>> Petko Manolov wrote:
>>>>
>>>> Unfortunately this is not the case.  These binary blobs are already
>>>> compressed and/or encrypted and adding even a few bytes ends up storing
>>>> new version in full size.
>>>
>>> Can't you store them uncompressed?
>>
>> Not really, but i can convert them into ascii format and store only the delta.
>
> If you don't have the original uncompressed unencrypted file, what will
> converting them to ascii actually give you?

I hope that in the case of incremental changes (0 to 5MB file is the same, 
last 64KB are actually new) the delta will be small and should be able to 
compress well.

This won't work for random changes along the length of the whole file.


 		Petko

Re: git and binary files

[Nicolas Pitre]

On Wed, 16 Jan 2008, Petko Manolov wrote:

> On Wed, 16 Jan 2008, Nicolas Pitre wrote:
> 
> > On Wed, 16 Jan 2008, Petko Manolov wrote:
> > 
> > > On Wed, 16 Jan 2008, Jakub Narebski wrote:
> > > 
> > > > Petko Manolov wrote:
> > > > > 
> > > > > Unfortunately this is not the case.  These binary blobs are already
> > > > > compressed and/or encrypted and adding even a few bytes ends up
> > > > > storing
> > > > > new version in full size.
> > > > 
> > > > Can't you store them uncompressed?
> > > 
> > > Not really, but i can convert them into ascii format and store only the
> > > delta.
> > 
> > If you don't have the original uncompressed unencrypted file, what will
> > converting them to ascii actually give you?
> 
> I hope that in the case of incremental changes (0 to 5MB file is the same,
> last 64KB are actually new) the delta will be small and should be able to
> compress well.
> 
> This won't work for random changes along the length of the whole file.

But my question remains.

If you cannot create good deltas out of your binary files, converting 
those binaries into ascii will do nothing to compression performance.


Nicolas

Re: git and binary files

[Petko Manolov]

On Wed, 16 Jan 2008, Nicolas Pitre wrote:

> On Wed, 16 Jan 2008, Petko Manolov wrote:
>
>> On Wed, 16 Jan 2008, Nicolas Pitre wrote:
>>
>>> On Wed, 16 Jan 2008, Petko Manolov wrote:
>>>
>>>> On Wed, 16 Jan 2008, Jakub Narebski wrote:
>>>>
>>>>> Petko Manolov wrote:
>>>>>>
>>>>>> Unfortunately this is not the case.  These binary blobs are already
>>>>>> compressed and/or encrypted and adding even a few bytes ends up
>>>>>> storing
>>>>>> new version in full size.
>>>>>
>>>>> Can't you store them uncompressed?
>>>>
>>>> Not really, but i can convert them into ascii format and store only the
>>>> delta.
>>>
>>> If you don't have the original uncompressed unencrypted file, what will
>>> converting them to ascii actually give you?
>>
>> I hope that in the case of incremental changes (0 to 5MB file is the same,
>> last 64KB are actually new) the delta will be small and should be able to
>> compress well.
>>
>> This won't work for random changes along the length of the whole file.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

> But my question remains.
>
> If you cannot create good deltas out of your binary files, converting
> those binaries into ascii will do nothing to compression performance.

I did agree with you above.  Binary or ascii it doesn't matter.

The experiments so far shows that git is doing a good job finding deltas 
in binary files.  This may be a solution for certain part of my files...


 		Petko

Re: git and binary files

[Petko Manolov]

On Wed, 16 Jan 2008, Nicolas Pitre wrote:

> On Wed, 16 Jan 2008, Petko Manolov wrote:
>
>> On Wed, 16 Jan 2008, Jakub Narebski wrote:
>>
>>> Petko Manolov wrote:
>>>>
>>>> Unfortunately this is not the case.  These binary blobs are already
>>>> compressed and/or encrypted and adding even a few bytes ends up storing
>>>> new version in full size.
>>>
>>> Can't you store them uncompressed?
>>
>> Not really, but i can convert them into ascii format and store only the delta.
>
> If you don't have the original uncompressed unencrypted file, what will
> converting them to ascii actually give you?

Not much.  I was blindly hoping that bzip2 may compress such files better, 
though i haven't tested this hypothesis yet.  Will let you know if the 
result is different from the obvious one. :-)


 		Petko

Re: git and binary files

[Jakub Narebski]

On Wed, 16 January 2008, Petko Manolov wrote:
> On Wed, 16 Jan 2008, Jakub Narebski wrote:
>> Petko Manolov wrote:
>>> On Wed, 16 Jan 2008, Jakub Narebski wrote:
>>>
>>>> You can always tag a blob (like junio-gpg-pub tag in git.git repository),
>>>> but it wouldn't be in a working directory. But it would get distributed
>>>> on clone.
>>>
>>> Hm, how does it work?
>>
>> You use git-hash-object to put file (-t blob) into the object database.
>> It would return sha1 of added object. Use git-tag to create tag to blob
>> (use returned sha1 for head). You can get file (to stdout) with
>> "git cat-file blob tagname^{blob}".
> 
> Sounds like i'll have to play with the above.  Thanks for the tip.
> 
>> The file would be in object database, but not in working directory
>> by default.
> 
> Not a big problem.

Please read carefully section about retagging / changing a tag
in git-tag(1) manpage; you should take care about propagating change
if you ever change the binary blob.

Nothe that I haven't used the technique described.

-- 
Jakub Narebski
Poland

Re: git and binary files

[Florian Weimer]

* Petko Manolov:

>>> Unfortunately this is not the case.  These binary blobs are already
>>> compressed and/or encrypted and adding even a few bytes ends up storing
>>> new version in full size.
>>
>> Can't you store them uncompressed?
>
> Not really, but i can convert them into ascii format and store only
> the delta.  This will admittedly increase the initial size of the
> repository, but hopefully not by much.

If the encryption is halfway decent, a new IV/nonce will be chosen for
each new revision, and you can't tell that two ciphertexts share a
common prefix without fully decrypting them.

-- 
Florian Weimer                <fweimer@bfk.de>
BFK edv-consulting GmbH       http://www.bfk.de/
Kriegsstraße 100              tel: +49-721-96201-1
D-76133 Karlsruhe             fax: +49-721-96201-99

Re: git and binary files

[Junio C Hamano]

Petko Manolov <petkan@nucleusys.com> writes:

> On Wed, 16 Jan 2008, Wincent Colaiuta wrote:
>
>> If the exact contents of these large binaries *really* don't matter,
>> as you say they don't, than why don't you just commit one and never
>> touch it again?
>
> Unfortunately those binaries does change, although the process is slow
> and not very frequent.  And this is why it pokes me in the eye - for
> changing a few bytes i end up with much larger repository.

For changing a few bytes you get with much larger repository?
What happened to our packfiles?

Is it possible for us to take a look at two versions of such a
binary blob, one before the change and one after such a change
that touches only a few bytes?

Re: git and binary files

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Petko Manolov <petkan@nucleusys.com> writes:
>
>> On Wed, 16 Jan 2008, Wincent Colaiuta wrote:
>>
>>> If the exact contents of these large binaries *really* don't matter,
>>> as you say they don't, than why don't you just commit one and never
>>> touch it again?
>>
>> Unfortunately those binaries does change, although the process is slow
>> and not very frequent.  And this is why it pokes me in the eye - for
>> changing a few bytes i end up with much larger repository.
>
> For changing a few bytes you get with much larger repository?
> What happened to our packfiles?
>
> Is it possible for us to take a look at two versions of such a
> binary blob, one before the change and one after such a change
> that touches only a few bytes?

I replied before reading the rest of the thread.  Please ignore
me on this one.

Re: git and binary files

[David Brown]

On Wed, Jan 16, 2008 at 04:39:19PM +0200, Petko Manolov wrote:

>> As an experiment, it might be worth trying to store the uncompressed
>> versions instead (git will delta _and_ compress them for you).
>
> I don't have them uncompressed.
>
> I can try to convert those files into ascii format and then save them in 
> the repository.  Since most changes are incremental git should be able to 
> generate relatively small delta, which should compress well enough.

This won't help, only make things slightly worse.

The problem is when you compress the files, as soon as there is a
difference between the two files, the output of the compressor will be
entirely different for the remainder of the file.  Unless you changes
always are at the end of the file, you will end up pretty much entire
versions in the repo.

I'm using some repos that get regular 50MB tarballs checked into them.  I
convinced them to not compress the tarballs, and git does a fairly decent
job of doing delta compression between them (although it needs quite a bit
of RAM to do so).

Dave

git and binary files

[Jan Dittmer]

Hi,

I'm using git to keep track of directories full of jpegs and movies
(19 GB .git directory, 5825 files, 18+ GB).

I now that this is also possible with rsync and friends, but I like
to have full version control and history, so please no discussion
about that.

The problem I'm facing is, that 'git status' is very
very slow. They seem to read through every file in the directory
tree. Is it possible to tell git to just use mtime/size information for
an initial guess? I already tried to turn of rename detection with
diff.renames=false, but still the problem remains.

Any hint? git version is 1.7.1.

Thanks,

Jan

-- 
Jan Dittmer <jdi@l4x.org>

Re: git and binary files

[Matthieu Moy]

Jan Dittmer <jdi@l4x.org> writes:

> The problem I'm facing is, that 'git status' is very
> very slow. They seem to read through every file in the directory
> tree. Is it possible to tell git to just use mtime/size information for
> an initial guess?

This is what it does by default. You probably need an initial "git
status" to be slow to get this information right in the index, though.

Have a look at git-bigfiles too.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: git and binary files

[Avery Pennarun]

On Wed, Jul 28, 2010 at 10:29 AM, Matthieu Moy
<Matthieu.Moy@grenoble-inp.fr> wrote:
> Jan Dittmer <jdi@l4x.org> writes:
>> The problem I'm facing is, that 'git status' is very
>> very slow. They seem to read through every file in the directory
>> tree. Is it possible to tell git to just use mtime/size information for
>> an initial guess?
>
> This is what it does by default. You probably need an initial "git
> status" to be slow to get this information right in the index, though.

Judging from the description, it sounds like 'git status' is the
problem, not the solution.  Although at 18 GB, it was probably pretty
slow to run, so maybe Jan only ran it once and the second time it
would have been fast :)

Depending on what the requirements are (ie. just keeping multiple
versions for posterity/safety as opposed to actually checking out old
versions on a regular basis), bup might be more useful than git in
this case: http://github.com/apenwarr/bup .

Have fun,

Avery

Git and Binary Files

[ryanzec]

I am a long time user of SVN and while it has worked for me in the past, I
always felt like there were features I wasn't using properly. Branching is
one of those features. Most of the time when I created branches, I was
really creating tags because when I created branches, I never really used
them for merging, just more to take a snapshot of the current code. When I
did have to make changes to code in one branch and merge them to another, I
always just manually merged the files because it was usually 1 or two files
and merge with SVN from what I hear is not a straight forward process. I
have recently starting using merging at work with SVN and I now really
understanding why people try to avoid it in SVN. Another thing I do in SVN
that I fell is counter productive is I only commit when I am 100% complete
with a feature. It would be great to commit things in chunks instead of one
great big change. I have had multiple times where I was like "I wish I could
go back just a little bit in time" but of course I have to go back to the
initial version which may be a week old or modify the file manually to get
back to the state I want it at (which it was I usually do).

Thankfully I have been introduced to the world of DVCS, specifically git. My
work is in the process of porting over to it and as I have been learning it,
I am wondering why I am still using SVN. Git fixes the above issues as well
as also having a number of cool little features.

Now while it fixes a lot of things there is one feature that concerns me
with trying to using it for game development projects and that is how it
handles binary files. In my day job, that is not a huge concern as I do web
development and I don't deal with many binary files that need to be
versioned. Images are the only ones and they rarely change and are not
usually that big. Game development is a different story. Game development in
general has many binary files (images, 3d assets, sound files, etc...).
Since git generally has a copy of the entire binary file for each change
that happens to it and since these assets can change quite often and
sometimes be large in size, I have a feeling that it can make the repository
of project become huge even with the compression that git does with git
pack/git gc.

Does anyone use git with game development? What is the workflow for binary
assets? Do you just store them in git (if so, are they in the main
repository or a separate repository using submodules)? If you don't include
binaries in the git repository, what tool do you use to manage binary assets
and does it work well with git?

--
View this message in context: http://git.661346.n2.nabble.com/Git-and-Binary-Files-tp6313601p6313601.html
Sent from the git mailing list archive at Nabble.com.

RE: Git and Binary Files

[Peter Jönsson P]

Hi!

I don't work with game development but I work with embedded systems where one usually have at least some types of binary files. This can range from entire cross compiler suites to FPGA binaries which should be included in the resulting product. Anyway, the "solution" to your problem is to separate the source code from the binary data by using several different repos. This can be achieved using either the Google "repo" tool (used for Android platform development) or "plain" git sub modules [2]. There have been some dicussions on this list regarding some other approaches but I think these two are the most common ones. 

// Peter

[1] http://google-opensource.blogspot.com/2008/11/gerrit-and-repo-android-source.html
[2] http://www.kernel.org/pub/software/scm/git/docs/git-submodule.html


-----Original Message-----
From: git-owner@vger.kernel.org [mailto:git-owner@vger.kernel.org] On Behalf Of ryanzec
Sent: den 28 april 2011 17:11
To: git@vger.kernel.org
Subject: Git and Binary Files

I am a long time user of SVN and while it has worked for me in the past, I always felt like there were features I wasn't using properly. Branching is one of those features. Most of the time when I created branches, I was really creating tags because when I created branches, I never really used them for merging, just more to take a snapshot of the current code. When I did have to make changes to code in one branch and merge them to another, I always just manually merged the files because it was usually 1 or two files and merge with SVN from what I hear is not a straight forward process. I have recently starting using merging at work with SVN and I now really understanding why people try to avoid it in SVN. Another thing I do in SVN that I fell is counter productive is I only commit when I a
 m 100% complete with a feature. It would be great to commit things in chunks instead of one great big change. I have had multiple times where I was like "I wish I could go back just a little bit in time" but of course I have to go back to the initial version which may be a week old or modify the file manually to get back to the state I want it at (which it was I usually do).

Thankfully I have been introduced to the world of DVCS, specifically git. My work is in the process of porting over to it and as I have been learning it, I am wondering why I am still using SVN. Git fixes the above issues as well as also having a number of cool little features.

Now while it fixes a lot of things there is one feature that concerns me with trying to using it for game development projects and that is how it handles binary files. In my day job, that is not a huge concern as I do web development and I don't deal with many binary files that need to be versioned. Images are the only ones and they rarely change and are not usually that big. Game development is a different story. Game development in general has many binary files (images, 3d assets, sound files, etc...).
Since git generally has a copy of the entire binary file for each change that happens to it and since these assets can change quite often and sometimes be large in size, I have a feeling that it can make the repository of project become huge even with the compression that git does with git pack/git gc.

Does anyone use git with game development? What is the workflow for binary assets? Do you just store them in git (if so, are they in the main repository or a separate repository using submodules)? If you don't include binaries in the git repository, what tool do you use to manage binary assets and does it work well with git?

--
View this message in context: http://git.661346.n2.nabble.com/Git-and-Binary-Files-tp6313601p6313601.html
Sent from the git mailing list archive at Nabble.com.
--
To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html