About Summer of code idea -- better big-file support

About Summer of code idea -- better big-file support

[徐迪]

Hi, there

I am wondering start implementing the idea about better big-file support
during summer of code.
Here are my idea: often gigantic files is media file, such as rmvb, swf,
pdf, dll, etc. And those file themselves are already being compressed, as we
all know, git uses zlib to compress all the object in repository, and it is
the common sense that if we use zlib to compress those file, the compress
ratio will be unbelievable low, from my test if we compress 521MiB mkv file
it will be 520MiB after compressed, how ridiculous it is!
Also I test that if we use "git hash-object" to calculate SHA-1, and copy
all content to repository, it will costs 1 minutes, once we use "git
hash-object -w" it will costs 2 minutes. So we sacrificed a lot, but gained
a little.
So, how do you guys think about that we can test the file type, and then
decide to use zlib or not?
Cheer
Di

Re: About Summer of code idea -- better big-file support

[Nguyen Thai Ngoc Duy]

2011/3/19 徐迪 <xudifsd@gmail.com>:
> So, how do you guys think about that we can test the file type, and then
> decide to use zlib or not?

We already have core.bigFileThreshold, which can be used to determine
storage type (traditional one with zlib, or some fancy ones for big
files). If that's not enough, .gitattributes can be used to mark
certain files as 'big'.

See below link for recent discussions regarding big file support:

http://thread.gmane.org/gmane.comp.version-control.git/165389/focus=168956

On the store of big files, bup (at github) model might fit in as well
if large files change a small portion regularly.
-- 
Duy

Re: About Summer of code idea -- better big-file support

[徐迪]

2011/3/19, Nguyen Thai Ngoc Duy <pclouds@gmail.com>:
> 2011/3/19 徐迪 <xudifsd@gmail.com>:
>> So, how do you guys think about that we can test the file type, and then
>> decide to use zlib or not?
>
> We already have core.bigFileThreshold, which can be used to determine
> storage type (traditional one with zlib, or some fancy ones for big
> files). If that's not enough, .gitattributes can be used to mark
> certain files as 'big'.
>
> See below link for recent discussions regarding big file support:
>
> http://thread.gmane.org/gmane.comp.version-control.git/165389/focus=168956
>
> On the store of big files, bup (at github) model might fit in as well
> if large files change a small portion regularly.
> --
> Duy
>
But, why those features haven't merged into upstream developement?
Sorry for this question, but I'm fresh here.

Re: About Summer of code idea -- better big-file support

[Nguyen Thai Ngoc Duy]

2011/3/19 徐迪 <xudifsd@gmail.com>:
> 2011/3/19, Nguyen Thai Ngoc Duy <pclouds@gmail.com>:
>> See below link for recent discussions regarding big file support:
>>
>> http://thread.gmane.org/gmane.comp.version-control.git/165389/focus=168956
>>
>> On the store of big files, bup (at github) model might fit in as well
>> if large files change a small portion regularly.
>>
> But, why those features haven't merged into upstream developement?
> Sorry for this question, but I'm fresh here.

This is just my guess. For specific apps, you should pull their authors in.

All apps experiment different ways of dealing with big files and work
in certain situations. None is generic enough to be merged upstream.
Take bup as an example, it aims at big files _only_. A file is treated
as a directory of many smaller pieces. In order to get it merged, we
need to separate those big files from the rest because it's just too
different in handling them.
-- 
Duy

Re: About Summer of code idea -- better big-file support

[徐迪]

2011/3/19 Nguyen Thai Ngoc Duy <pclouds@gmail.com>:
> 2011/3/19 徐迪 <xudifsd@gmail.com>:
>> 2011/3/19, Nguyen Thai Ngoc Duy <pclouds@gmail.com>:
>>> See below link for recent discussions regarding big file support:
>>>
>>> http://thread.gmane.org/gmane.comp.version-control.git/165389/focus=168956
>>>
>>> On the store of big files, bup (at github) model might fit in as well
>>> if large files change a small portion regularly.
>>>
>> But, why those features haven't merged into upstream developement?
>> Sorry for this question, but I'm fresh here.
>
> This is just my guess. For specific apps, you should pull their authors in.
>
> All apps experiment different ways of dealing with big files and work
> in certain situations. None is generic enough to be merged upstream.
> Take bup as an example, it aims at big files _only_. A file is treated
> as a directory of many smaller pieces. In order to get it merged, we
> need to separate those big files from the rest because it's just too
> different in handling them.
> --
> Duy
>

Thanks for that timely link, I think I should do some more research now :P