.gitignore for large files?

.gitignore for large files?

[Philip Oakley]

Has there been any discussion in the past on a method for ignoring large 
files via the .gitignore process?

It does appear to be a moderately common problem for folk to accidentally 
commit a large file which bloats their repository and they want rid of it, 
which causes history re-writes and such palaver.

Perhaps a simple '>' and '<' option (the latter to cover null or minimal 
files?) with a --warn postfix may be possible. Just an initial thought.

Where would the 'right place' be for me to look at the git code if it was 
beneficial.

Philip Oakley
(UK: Scotland) 

Re: .gitignore for large files?

[Nguyen Thai Ngoc Duy]

On Sun, Jul 24, 2011 at 3:00 AM, Philip Oakley <philipoakley@iee.org> wrote:
> Has there been any discussion in the past on a method for ignoring large
> files via the .gitignore process?
>
> It does appear to be a moderately common problem for folk to accidentally
> commit a large file which bloats their repository and they want rid of it,
> which causes history re-writes and such palaver.

Once they are in, they cannot be ignored. Perhaps commit hooks at
server side is a better place?

> Perhaps a simple '>' and '<' option (the latter to cover null or minimal
> files?) with a --warn postfix may be possible. Just an initial thought.

Or you can make use of .gitattributes, more flexible syntax.

> Where would the 'right place' be for me to look at the git code if it was
> beneficial.

In dir.c, add_exlude() does the parsing, excluded_from_list() handles the logic.
-- 
Duy

Re: .gitignore for large files?

[Philip Oakley]

Duy,

The .git attributes is a good call. I'm thinking that it would be a global 
flag (i.e. is a max file limit in place at all on this repo), and then a set 
of limits, initially one for 'text' and one for 'binary' (and possibly a 
third for anything with an ext diff), and probably use a similar attribute 
format as used for setting diff attributes and its filters to set a per file 
type limit [if required]

The relevant function names gives me a place to start....

Philip
----- Original Message ----- 
From: "Nguyen Thai Ngoc Duy" <pclouds@gmail.com>
To: "Philip Oakley" <philipoakley@iee.org>
Cc: "Git List" <git@vger.kernel.org>
Sent: Sunday, July 24, 2011 3:59 AM
Subject: Re: .gitignore for large files?


> On Sun, Jul 24, 2011 at 3:00 AM, Philip Oakley <philipoakley@iee.org> 
> wrote:
>> Has there been any discussion in the past on a method for ignoring large
>> files via the .gitignore process?
>>
>> It does appear to be a moderately common problem for folk to accidentally
>> commit a large file which bloats their repository and they want rid of 
>> it,
>> which causes history re-writes and such palaver.
>
> Once they are in, they cannot be ignored. Perhaps commit hooks at
> server side is a better place?
>
>> Perhaps a simple '>' and '<' option (the latter to cover null or minimal
>> files?) with a --warn postfix may be possible. Just an initial thought.
>
> Or you can make use of .gitattributes, more flexible syntax.
>
>> Where would the 'right place' be for me to look at the git code if it was
>> beneficial.
>
> In dir.c, add_exlude() does the parsing, excluded_from_list() handles the 
> logic.
> -- 
> Duy
>
> ______________________________________________________________________
> This email has been scanned by the MessageLabs Email Security System.
> For more information please visit http://www.messagelabs.com/email
> ______________________________________________________________________
>
>
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 10.0.1390 / Virus Database: 1518/3783 - Release Date: 07/23/11
> 

Re: .gitignore for large files?

[Nguyen Thai Ngoc Duy]

On Mon, Jul 25, 2011 at 1:53 PM, Philip Oakley <philipoakley@iee.org> wrote:
> Duy,
>
> The .git attributes is a good call. I'm thinking that it would be a global
> flag (i.e. is a max file limit in place at all on this repo), and then a set
> of limits, initially one for 'text' and one for 'binary' (and possibly a
> third for anything with an ext diff), and probably use a similar attribute
> format as used for setting diff attributes and its filters to set a per file
> type limit [if required]

While .gitattributes looks like a better place, it does not have
"exclude" attribute equivalence to .gitignore. If I remember
correctly, the way .gitignore and .gitattributes are implemented makes
it very hard to turn .gitignore into part of .gitattributes
implementation (gitattr checks .gitattributes of current dir first,
then upward to parents, while .gitgnore follows the opposite
direction).

But the other way might be simpler: introduce a new attribute, then
make use of the attribute in exclude code like autogenerated
.gitignore. I don't know. I have not thought through this idea.

But of course you can stay away from those by checking new attributes
in builtin/add.c and refusing to execute if someone wants to add a
file larger than a limit. You can avoid git-add internal business by
checking entries in to-be-written index, at label "finish" in
cmd_add(). This is the simplest way (and getting very close to using
hooks).

Anyway, good luck. Come back with some patches ;-)
-- 
Duy

Re: .gitignore for large files?

[Junio C Hamano]

Nguyen Thai Ngoc Duy <pclouds@gmail.com> writes:

> While .gitattributes looks like a better place, it does not have
> "exclude" attribute equivalence to .gitignore. If I remember
> correctly, the way .gitignore and .gitattributes are implemented makes
> it very hard to turn .gitignore into part of .gitattributes
> implementation (gitattr checks .gitattributes of current dir first,
> then upward to parents, while .gitgnore follows the opposite
> direction).

While I do not think it is necessarily a good idea to invent yet another
way to exclude and add it to the attributes mechanism (unless we will be
dropping the support for gitignore, which is not the case), I do not know
why you think the direction of the scan matters.

A more important difference is that the attribute mechanism covers the
actual paths, not intermediate directories, unlike gitignore does.

Re: .gitignore for large files?

[Philip Oakley]

Subject: How to ignore large files?
> Nguyen Thai Ngoc Duy <pclouds@gmail.com> writes:
>
>> While .gitattributes looks like a better place, it does not have
>> "exclude" attribute equivalence to .gitignore. If I remember
>> correctly, the way .gitignore and .gitattributes are implemented makes
>> it very hard to turn .gitignore into part of .gitattributes
>> implementation (gitattr checks .gitattributes of current dir first,
>> then upward to parents, while .gitgnore follows the opposite
>> direction).
>
> While I do not think it is necessarily a good idea to invent yet another
> way to exclude and add it to the attributes mechanism (unless we will be
> dropping the support for gitignore, which is not the case), I do not know
> why you think the direction of the scan matters.
>
> A more important difference is that the attribute mechanism covers the
> actual paths, not intermediate directories, unlike gitignore does.

The choice of any actual implementation would depend on both feasability and 
usefulness. Duy was pointing out (to me?) the different approaches used for 
respecting the gitattributes and (multiple) gitignore files. My first 
thought had been to use the .gitignore file but I then realised the 
potential problems of adding more metacharacters ( > and < ).

I've also just seen that 1.7.6 introduced the core.bigFileThreshold for 
memory management reasons, while this suggestion is to help folk avoid the 
mistake of unintentional committing of very large files without being 
warned.

Does the concept (of warning/ignoring when files are 'large') have any 
merit?

Philip