Problems with large compressed binaries when converting from svn

Problems with large compressed binaries when converting from svn

[Øyvind Harboe]

I'm converting from svn and I've run into a
problem with tar.gz and tar.bz2 compressed files.

(This is a separate but only slightly related to previous post).

In subversion we committed large tar.bz2/gz files. These files would
change relatively rarely, but only very slightly.  The trouble with the tar.bz2
format is that if the first byte changes, then the rest of the file will also
be different. .zip does not have this problem, but .zip isn't a very friendly
format for our purposes.

Later on the tar.bz2/gz files started to change fairly often, but harddrives
get bigger much more quickly than the .svn repository grows so we just
kept doing things the same way rather than reeducate and reengineer
the procedures.

With .git we need to handle this differently somehow.

Does git have some capability to store diffs of compressed files efficiently?

The only other alternative I can think of is to commit uncompressed
.tar files which is a bit of a bump in the road, but I suppose could be
made to work.



-- 
Øyvind Harboe
http://www.zylin.com/zy1000.html
ARM7 ARM9 XScale Cortex
JTAG debugger and flash programmer

Re: Problems with large compressed binaries when converting from svn

[Alex Riesen]

2009/1/6 Øyvind Harboe <oyvind.harboe@zylin.com>:
> I'm converting from svn and I've run into a
> problem with tar.gz and tar.bz2 compressed files.
>
> (This is a separate but only slightly related to previous post).
>
> In subversion we committed large tar.bz2/gz files. These files would
> change relatively rarely, but only very slightly.  The trouble with the tar.bz2
> format is that if the first byte changes, then the rest of the file will also
> be different. .zip does not have this problem, but .zip isn't a very friendly
> format for our purposes.
>
> Later on the tar.bz2/gz files started to change fairly often, but harddrives
> get bigger much more quickly than the .svn repository grows so we just
> kept doing things the same way rather than reeducate and reengineer
> the procedures.
>
> With .git we need to handle this differently somehow.
>
> Does git have some capability to store diffs of compressed files efficiently?

No, but you can unpack the tarballs and include the toolchains as submodules
(aka subprojects) in the projects which need them.

See man page to git submodule, the user-manual.txt on "submodule" and
gitmodules.txt (submodule configuration formats and conventions).

Re: Problems with large compressed binaries when converting from svn

[Øyvind Harboe]

>> Does git have some capability to store diffs of compressed files efficiently?
>
> No, but you can unpack the tarballs and include the toolchains as submodules
> (aka subprojects) in the projects which need them.
>
> See man page to git submodule, the user-manual.txt on "submodule" and
> gitmodules.txt (submodule configuration formats and conventions).

I'll need the submodule stuff for sure, but in this particular case I was
trying to see if there was a way to keep the svn abuse patterns from
svn under git without a lot of retraining.



-- 
Øyvind Harboe
http://www.zylin.com/zy1000.html
ARM7 ARM9 XScale Cortex
JTAG debugger and flash programmer

Re: Problems with large compressed binaries when converting from svn

[Johan Herland]

On Tuesday 06 January 2009, Øyvind Harboe wrote:
> I'm converting from svn and I've run into a
> problem with tar.gz and tar.bz2 compressed files.
>
> (This is a separate but only slightly related to previous post).
>
> In subversion we committed large tar.bz2/gz files. These files would
> change relatively rarely, but only very slightly.  The trouble with the
> tar.bz2 format is that if the first byte changes, then the rest of the
> file will also be different. .zip does not have this problem, but .zip
> isn't a very friendly format for our purposes.
>
> Later on the tar.bz2/gz files started to change fairly often, but
> harddrives get bigger much more quickly than the .svn repository grows so
> we just kept doing things the same way rather than reeducate and
> reengineer the procedures.
>
> With .git we need to handle this differently somehow.
>
> Does git have some capability to store diffs of compressed files
> efficiently?
>
> The only other alternative I can think of is to commit uncompressed
> .tar files which is a bit of a bump in the road, but I suppose could be
> made to work.

Git can automate this for you. Take a look at the gitattributes(5) man page, 
specifically the "filter" attribute. You should be able to set up filter 
drivers for .tar.gz files that use "clean=gunzip" and "smudge=gzip" (and a 
similar filter driver for .tar.bz2 files).

If I've understood this right (I haven't used this myself) your checkouts 
should now have .tar.gz and .tar.bz2 files, even though Git only 
stores .tar files internally (thus improving compression across versions 
dramatically).


Have fun! :)

...Johan

-- 
Johan Herland, <johan@herland.net>
www.herland.net