Please consider extending .gitignore to support setting of a file size limit

Please consider extending .gitignore to support setting of a file size limit

[Roger Mendes]

Allow .gitignore to support setting a file size limit so that all
files over a certain size will by automatically ignored when
performing git add, commit -a...;
Exclusions to this size limit for files can be allowed by specifying
exceptions '!' syntax that already exists. It probably should be
considered to have a default limit already built into git of say 100MB
(used by github) which can be changed in the .gitignore file.

This would avoid accidental commits of large files which are known to
be poorly supported.

This seems like a fairly simple feature to add, I'm not sure why no
one has done this or thought of it given the number of complaints,
questions... with large file handling.

We can ignore by file name attribute, why not by file size?

Yes, I know I can add pre-commit scripts... to workaround this issue,
but there should be a better way (or maybe I'm missing something?).


Accidental commits of large files is a common mistake, and often
results in the commit taking an extremely long time to process and
sometimes if you kill the process, then you end up with corrupt git
repo (at least in jgit, which is used by Netbeans).

IMHO this would be a very useful feature.

Re: Please consider extending .gitignore to support setting of a file size limit

[Junio C Hamano]

Roger Mendes <sproger1@gmail.com> writes:

> Allow .gitignore to support setting a file size limit so that all
> files over a certain size will by automatically ignored when
> performing git add, commit -a...;

No.

I would not say that it would not make sense to have a method to
tell your Git to reject an attempt to add certain files to the
index, but .gitignore is make that decision based on paths and
is a bad place to do any other decision, so take the above "No"
as "No, we do not consider butchering .gitignore for that", not
as "No, you shouldn't base your decision on file size".

Also, once the path is tracked, then "git add" will keep tracking it
unless and until you "git rm --cached" it.  So you may "git add" a
1kB file, grow the file to 500MB and then the next "git add" should
register the updated contents to the index.  Whatever you do when
designing that new feature, you should need to take that into
account.

Having said all that, why is it that you do not want to track
certain files that happen to be large?  Don't the existing large
file support facility work for you (or perhaps you are not using
them)?  It might be that your time is better spent on improving
that, instead of working around the issue, no?

Re: Please consider extending .gitignore to support setting of a file size limit

[Thomas Koch]

On Thursday, May 07, 2015 11:31:12 PM Junio C Hamano wrote:
> Roger Mendes <sproger1@gmail.com> writes:
> > Allow .gitignore to support setting a file size limit so that all
> > files over a certain size will by automatically ignored when
> > performing git add, commit -a...;
> 
> No.

Please have a look into "git annex". It provides large file support without
blowing up the repository size entirely and has facilities to decide what to
do with files based on multiple criterias like file size.

Maybe some features from git annex have matured enough to be proposed
for inclusion into git?

Regards, Thomas Koch

Re: Please consider extending .gitignore to support setting of a file size limit

[Kevin Daudt]

On Thu, May 07, 2015 at 02:31:12PM -0700, Junio C Hamano wrote:
> Roger Mendes <sproger1@gmail.com> writes:
> 
> Having said all that, why is it that you do not want to track
> certain files that happen to be large?  Don't the existing large
> file support facility work for you (or perhaps you are not using
> them)?  It might be that your time is better spent on improving
> that, instead of working around the issue, no?
> 

My experience in #git is that people want to prevent other people
accidentelly adding large files, resulting in all sorts of problems,
which takes a lot of effort to fix if you're not an experienced user.

Re: Please consider extending .gitignore to support setting of a file size limit

[Fredrik Gustafsson]

On Fri, May 08, 2015 at 08:54:59AM +0200, Kevin Daudt wrote:
> My experience in #git is that people want to prevent other people
> accidentelly adding large files, resulting in all sorts of problems,
> which takes a lot of effort to fix if you're not an experienced user.

I second this. In #git@freenode we often get people trying to reduce
repo size since they or a collegue accidental added a huge binary file.

They do a filter branch and remove that file. However then they need to
do a forced push. And the person (the collegue) that didn't had enough
git experience to understand why it was a mistake to add a big file, now
ends up with having to sync with a rewritten remote.

So I can really see that there's a demand for such a feature. However, I
wonder if this shouldn't just be a pre-recieve hook?

If we want to inform the user, we might just add a pre-commit hook for
this.

The problem with pre-commit hooks is distribution of them. As it is with
making this a configuration option. There's really no good way to share
configuration and hooks between git repos today as I know of. (Yes, I'm
aware of this beeing a hard problem due to security concerns).

-- 
Fredrik Gustafsson

phone: +46 733-608274
e-mail: iveqy@iveqy.com
website: http://www.iveqy.com

Re: Please consider extending .gitignore to support setting of a file size limit

[Stefan Beller]

On Fri, May 8, 2015 at 12:09 AM, Fredrik Gustafsson <iveqy@iveqy.com> wrote:
> On Fri, May 08, 2015 at 08:54:59AM +0200, Kevin Daudt wrote:
>> My experience in #git is that people want to prevent other people
>> accidentelly adding large files, resulting in all sorts of problems,
>> which takes a lot of effort to fix if you're not an experienced user.
>
> I second this. In #git@freenode we often get people trying to reduce
> repo size since they or a collegue accidental added a huge binary file.
>
> They do a filter branch and remove that file. However then they need to
> do a forced push. And the person (the collegue) that didn't had enough
> git experience to understand why it was a mistake to add a big file, now
> ends up with having to sync with a rewritten remote.
>
> So I can really see that there's a demand for such a feature. However, I
> wonder if this shouldn't just be a pre-recieve hook?
>
> If we want to inform the user, we might just add a pre-commit hook for
> this.
>
> The problem with pre-commit hooks is distribution of them. As it is with
> making this a configuration option. There's really no good way to share
> configuration and hooks between git repos today as I know of. (Yes, I'm
> aware of this beeing a hard problem due to security concerns).
>

Any problem in computer science can be solved with another layer of indirection
they said. So:

Maybe we can just distribute some good defaults (or your internal distribution
for git can ship adapted versions of the default). And instead of just running
a hook, maybe the clone/fetch could tell the client to enable these
predefined hooks.
so maybe we'd need a fetch capability "recommended-hooks" which could look like

    recommended-hooks=<name>,<sha1>

The name would indicate what hook you're interested in such as
.git/hooks/pre-commit-warn-huge-size.sample and the sha1 would be the
sha1 of that default hook to ensure nobody has tampered with that script.

I am not sure if we need the sha1, maybe that's overengineered. I mean trusting
a default script is as much as trusting the git binary on your system, isn't it?

Re: Please consider extending .gitignore to support setting of a file size limit

[Philip Oakley]

From: "Kevin Daudt" <me@ikke.info>
> On Thu, May 07, 2015 at 02:31:12PM -0700, Junio C Hamano wrote:
>> Roger Mendes <sproger1@gmail.com> writes:
>>
>> Having said all that, why is it that you do not want to track
>> certain files that happen to be large?  Don't the existing large
>> file support facility work for you (or perhaps you are not using
>> them)?  It might be that your time is better spent on improving
>> that, instead of working around the issue, no?
>>
>
> My experience in #git is that people want to prevent other people
> accidentelly adding large files, resulting in all sorts of problems,
> which takes a lot of effort to fix if you're not an experienced user.
> --
I'd also support this option.

One issue is to locate a suitable method/character code for the 'magic 
pathspec' that would define a size limit that can be used for the 'git 
add .' etc., and possibly also for sparse checkout (definitely an area 
where a size limit may be appropriate for some users).

The ability to extend the ignore method could also, in the future be 
used for a possible narrow clone (when desired).
--
Philip