Git ignore help

Git ignore help

[mdconf]

Hello,
I am trying to setup my git ignore (resp. .git/info/exclude) so that I exclude
 all directories and files except the content of directories that I 
specifically include (incl. anything within them recursively).

I set the .git/info/exclude with the following content:

========
# Exclude everything
/*
# Except the below that we include
!/db/data/load/base/bootstraponly
!/db/data/load/base/safetoload
!/db/ddl
!/labels
!/reports/usrint
!/scripts
!/src/cmdsrc/usrint
========

However it does not do what I anticipated. It indeed excludes everything but 
the include part does not work - it only works for !/labels and !/scripts 
directories (i.e. the first level directories). All other are still ignored - 
so when I create file /db/data/load/base/bootstraponly/somefile.txt git still 
ignores it...

Any idea what I am doing wrong?

I tried all of the following combinations but still with the same result:
!/db/data/load/base/bootstraponly
!/db/data/load/base/bootstraponly/
!/db/data/load/base/bootstraponly/*
!/db/data/load/base/bootstraponly/**

When I add just "!/db" to exclude then it works. But that's not what I want - 
I really want to include just the /db/data/load/base/bootstraponly and its
all content...

Thank you,
Martin

Re: Git ignore help

[Eric Sunshine]

On Fri, Mar 20, 2015 at 6:36 AM,  <mdconf@seznam.cz> wrote:
> I am trying to setup my git ignore (resp. .git/info/exclude) so that I exclude
>  all directories and files except the content of directories that I
> specifically include (incl. anything within them recursively).
>
> I set the .git/info/exclude with the following content:
>
> ========
> # Exclude everything
> /*
> # Except the below that we include
> !/db/data/load/base/bootstraponly
> !/db/data/load/base/safetoload
> !/db/ddl
> !/labels
> !/reports/usrint
> !/scripts
> !/src/cmdsrc/usrint
> ========
>
> However it does not do what I anticipated. It indeed excludes everything but
> the include part does not work - it only works for !/labels and !/scripts
> directories (i.e. the first level directories). All other are still ignored -
> so when I create file /db/data/load/base/bootstraponly/somefile.txt git still
> ignores it...
>
> Any idea what I am doing wrong?

The fourth bullet point of the "Pattern Format" section of the
gitignore man page has this to say, which explains the behavior you're
seeing:

    An optional prefix "!" which negates the pattern; any matching
    file excluded by a previous pattern will become included again.
    It is not possible to re-include a file if a parent directory of
    that file is excluded. Git doesn’t list excluded directories for
    performance reasons, so any patterns on contained files have no
    effect, no matter where they are defined.

Re: Git ignore help

[Duy Nguyen]

On Sun, Mar 22, 2015 at 10:33 AM, Eric Sunshine <sunshine@sunshineco.com> wrote:
> On Fri, Mar 20, 2015 at 6:36 AM,  <mdconf@seznam.cz> wrote:
>> I am trying to setup my git ignore (resp. .git/info/exclude) so that I exclude
>>  all directories and files except the content of directories that I
>> specifically include (incl. anything within them recursively).
>>
>> I set the .git/info/exclude with the following content:
>>
>> ========
>> # Exclude everything
>> /*
>> # Except the below that we include
>> !/db/data/load/base/bootstraponly
>> !/db/data/load/base/safetoload
>> !/db/ddl
>> !/labels
>> !/reports/usrint
>> !/scripts
>> !/src/cmdsrc/usrint
>> ========
>>
>> However it does not do what I anticipated. It indeed excludes everything but
>> the include part does not work - it only works for !/labels and !/scripts
>> directories (i.e. the first level directories). All other are still ignored -
>> so when I create file /db/data/load/base/bootstraponly/somefile.txt git still
>> ignores it...
>>
>> Any idea what I am doing wrong?
>
> The fourth bullet point of the "Pattern Format" section of the
> gitignore man page has this to say, which explains the behavior you're
> seeing:
>
>     An optional prefix "!" which negates the pattern; any matching
>     file excluded by a previous pattern will become included again.
>     It is not possible to re-include a file if a parent directory of
>     that file is excluded. Git doesn’t list excluded directories for
>     performance reasons, so any patterns on contained files have no
>     effect, no matter where they are defined.

This is true. To elaborate, if we have to recurse in excluded
directories so that we can include some back, then the reason for
excluding is already defeated as we may need to traverse the entire
directory structure. However in this particular case where we do know
in advance that only certain directories may have "re-include" rules,
e.g. "db", "reports" or "scripts", we could keep going for a while. I
think I attempted to do this in the past and failed (don't remember
exactly why). Maybe I'll try again some time in future.

Another option is, if the user is willing to accept performance
degradation (in many small repos, it does not matter anyway), then we
could keep digging in.
-- 
Duy

Re: Git ignore help

[Eric Sunshine]

On Mon, Mar 23, 2015 at 9:00 PM, Duy Nguyen <pclouds@gmail.com> wrote:
> On Sun, Mar 22, 2015 at 10:33 AM, Eric Sunshine <sunshine@sunshineco.com> wrote:
>> On Fri, Mar 20, 2015 at 6:36 AM,  <mdconf@seznam.cz> wrote:
>>> Any idea what I am doing wrong?
>>
>> The fourth bullet point of the "Pattern Format" section of the
>> gitignore man page has this to say, which explains the behavior you're
>> seeing:
>>
>>     An optional prefix "!" which negates the pattern; any matching
>>     file excluded by a previous pattern will become included again.
>>     It is not possible to re-include a file if a parent directory of
>>     that file is excluded. Git doesn’t list excluded directories for
>>     performance reasons, so any patterns on contained files have no
>>     effect, no matter where they are defined.
>
> This is true. To elaborate, if we have to recurse in excluded
> directories so that we can include some back, then the reason for
> excluding is already defeated as we may need to traverse the entire
> directory structure. However in this particular case where we do know
> in advance that only certain directories may have "re-include" rules,
> e.g. "db", "reports" or "scripts", we could keep going for a while. I
> think I attempted to do this in the past and failed (don't remember
> exactly why). Maybe I'll try again some time in future.

I also was pretty sure that you had attempted this, but couldn't find
a reference to it, so I didn't mention it in my response. However,
with some more digging, I finally located it[1].

[1]: http://article.gmane.org/gmane.comp.version-control.git/259870

> Another option is, if the user is willing to accept performance
> degradation (in many small repos, it does not matter anyway), then we
> could keep digging in.
> --
> Duy

Re: Git ignore help

[Duy Nguyen]

On Tue, Mar 24, 2015 at 8:55 AM, Eric Sunshine <sunshine@sunshineco.com> wrote:
>> e.g. "db", "reports" or "scripts", we could keep going for a while. I
>> think I attempted to do this in the past and failed (don't remember
>> exactly why). Maybe I'll try again some time in future.
>
> I also was pretty sure that you had attempted this, but couldn't find
> a reference to it, so I didn't mention it in my response. However,
> with some more digging, I finally located it[1].
>
> [1]: http://article.gmane.org/gmane.comp.version-control.git/259870

Thank you. I only looked at my repo and no branch name suggested it
(if only there is google search for a git repository..). So I gave up
because of performance reasons again but that was for enabling it
unconditionaly. If we enable it via a config variable and the user is
made aware of the performance implications, I guess it would be ok. So
it's back in my back log.
-- 
Duy

Re: Git ignore help

[mdconf]

Hi Duy, Eric,

thx a lot. So net net - I can't really achieve this. It feels like something very basic and simple so pretty surprising but guess that's what it is :)

Normally, I'd expect that the functionality will behave like with similar other blacklist/whitelist functionalities in Linux - that the more specific definition overrides the less specific one. And when there are 2 on the same level then one (include or exclude) has the priority by default...

Anyways, what I am still not clear on is how come that including just the 'single level folder' work?

Duy, you wrote:

"This is true. To elaborate, if we have to recurse in excluded directories so that we can include some back, then the reason for excluding is already defeated as we may need to traverse the entire directory structure. However in this particular case where we do know in advance that only certain directories may have "re-include" rules, e.g. "db", "reports" or "scripts", we could keep going for a while."

... so according to that it sounds like including /db, /reports, /scripts should actually also NOT work. But it does work - i.e. when I add the following:

# exclude
/*

# except
!/db
!/reports
!/scripts

then any content within those 3 directories (and their sub directories) is included and not ignored...

It ONLY does not work when I add more levels - e.g.:

!/reports/something

In this case neither /reports nor /reports/something or any sub directory is included.

Martin


---------- Původní zpráva ----------
Od: Duy Nguyen 
Komu: Eric Sunshine 
Datum: 24. 3. 2015 10:40:33
Předmět: Re: Git ignore help

On Tue, Mar 24, 2015 at 8:55 AM, Eric Sunshine  wrote:
>> e.g. "db", "reports" or "scripts", we could keep going for a while. I
>> think I attempted to do this in the past and failed (don't remember
>> exactly why). Maybe I'll try again some time in future.
>
> I also was pretty sure that you had attempted this, but couldn't find
> a reference to it, so I didn't mention it in my response. However,
> with some more digging, I finally located it[1].
>
> [1]: http://article.gmane.org/gmane.comp.version-control.git/259870

Thank you. I only looked at my repo and no branch name suggested it
(if only there is google search for a git repository..). So I gave up
because of performance reasons again but that was for enabling it
unconditionaly. If we enable it via a config variable and the user is
made aware of the performance implications, I guess it would be ok. So
it's back in my back log.
-- 
Duy

Re: Git ignore help

[Duy Nguyen]

On Tue, Mar 24, 2015 at 7:46 PM,  <mdconf@seznam.cz> wrote:
> Duy, you wrote:
>
> "This is true. To elaborate, if we have to recurse in excluded directories so that we can include some back, then the reason for excluding is already defeated as we may need to traverse the entire directory structure. However in this particular case where we do know in advance that only certain directories may have "re-include" rules, e.g. "db", "reports" or "scripts", we could keep going for a while."
>
> ... so according to that it sounds like including /db, /reports, /scripts should actually also NOT work. But it does work - i.e. when I add the following:
>
> # exclude
> /*
>
> # except
> !/db
> !/reports
> !/scripts
>
> then any content within those 3 directories (and their sub directories) is included and not ignored...
>
> It ONLY does not work when I add more levels - e.g.:
>
> !/reports/something
>
> In this case neither /reports nor /reports/something or any sub directory is included.

Yes. It's the subtlety of optimizing ;-) If you read the man page
really carefully (*), "if the _parent_ directory of that file(**) is
excluded" and the parent of these three directories is _not_ excluded.

(*) I'm not saying this is a good thing. Only docs such as language
spec or RFCs need that level of attention. But I'm not a good document
writer myself, can't blame others. Improvements are welcome though.

(**) "that file" should be "that file or directory" but I guess
simplification here is ok
-- 
Duy

Re: Git ignore help

[Eric Sunshine]

On Tue, Mar 24, 2015 at 5:39 AM, Duy Nguyen <pclouds@gmail.com> wrote:
> On Tue, Mar 24, 2015 at 8:55 AM, Eric Sunshine <sunshine@sunshineco.com> wrote:
>>> e.g. "db", "reports" or "scripts", we could keep going for a while. I
>>> think I attempted to do this in the past and failed (don't remember
>>> exactly why). Maybe I'll try again some time in future.
>>
>> I also was pretty sure that you had attempted this, but couldn't find
>> a reference to it, so I didn't mention it in my response. However,
>> with some more digging, I finally located it[1].
>>
>> [1]: http://article.gmane.org/gmane.comp.version-control.git/259870
>
> Thank you. I only looked at my repo and no branch name suggested it
> (if only there is google search for a git repository..). So I gave up
> because of performance reasons again but that was for enabling it
> unconditionaly. If we enable it via a config variable and the user is
> made aware of the performance implications, I guess it would be ok. So
> it's back in my back log.

How much does a config variable actually help? In a sense, one could
argue that this is already an opt-in feature since it requires
crafting gitignore in a particular fashion. Existing projects which
have (properly) functioning gitignore rules won't trigger this
behavior. In many cases, Git already allows people to shoot themselves
in the foot if they desire, thus, as long as the potential performance
impact is properly documented, this could be considered another such
instance.

Re: Git ignore help

[Junio C Hamano]

Eric Sunshine <sunshine@sunshineco.com> writes:

> On Tue, Mar 24, 2015 at 5:39 AM, Duy Nguyen <pclouds@gmail.com> wrote:
>> On Tue, Mar 24, 2015 at 8:55 AM, Eric Sunshine <sunshine@sunshineco.com> wrote:
>>>> e.g. "db", "reports" or "scripts", we could keep going for a while. I
>>>> think I attempted to do this in the past and failed (don't remember
>>>> exactly why). Maybe I'll try again some time in future.
>>>
>>> I also was pretty sure that you had attempted this, but couldn't find
>>> a reference to it, so I didn't mention it in my response. However,
>>> with some more digging, I finally located it[1].
>>>
>>> [1]: http://article.gmane.org/gmane.comp.version-control.git/259870
>>
>> Thank you. I only looked at my repo and no branch name suggested it
>> (if only there is google search for a git repository..). So I gave up
>> because of performance reasons again but that was for enabling it
>> unconditionaly. If we enable it via a config variable and the user is
>> made aware of the performance implications, I guess it would be ok. So
>> it's back in my back log.
>
> How much does a config variable actually help? In a sense, one could
> argue that this is already an opt-in feature since it requires
> crafting gitignore in a particular fashion. Existing projects which
> have (properly) functioning gitignore rules won't trigger this
> behavior. In many cases, Git already allows people to shoot themselves
> in the foot if they desire, thus, as long as the potential performance
> impact is properly documented, this could be considered another such
> instance.

Yeah, as I re-read that old thread, I really do not think anything
wrong with the reasoning expressed in the proposed log message.  It
is pointless to hunt for "!do-not-exclude-me-please" in D/.gitignore
when "D/" appears in .gitignore in the higher level, but if these two
i.e.

	D/
        !D/do-not-exclude-me-please

appear together in .gitignore in the higher level, we can pay
attention to that and pick up that single path.  And doing so would
be a lot more intuitive to the end user.

My comment in the thread was only about the documentation being
unclear and not about the feature, but somehow we failed to
follow-up the topic to completion X-<.