git-status and git-diff now very slow in project with a submodule

git-status and git-diff now very slow in project with a submodule

[Andy Parkins]

Hello,

I've just upgraded from 1.6.something to 1.7.1.  All very nice.

The new submodule reporting is nice too; but I'd like to be able to turn it 
off :-)

The problem is that I have a (relatively) small project as the supermodule, 
and a linux kernel clone as a submodule and an ffmpeg clone as a submodule.  
Now I used to be able to do git-status or git-diff and it would be instant, 
it now takes a number of seconds to report.  I guess (but don't know), that 
it is the detection of "dirty" status in the submodule's that is slowing 
down the supermodule processing.

I wouldn't like to see the feature go, because in almost all circumstances 
it is exactly right; however, I'd like to be able to turn off dirty 
detection in submodules.  Is this already possible, and I've just missed the 
configuration option?

One additional small point: why do untracked files in a submodule make the 
module dirty?  I've often got a few "temp.ps" or "debug.log" or 
"backtrace.log" files lying around -- inappropriate to add to an ignore 
file, but they don't make my working directory dirty.  "Dirty" in a working 
directory means uncommitted changes to tracked files, why does it mean 
something different in a submodule?



Andy
-- 
Dr Andy Parkins
andyparkins@gmail.com

Re: git-status and git-diff now very slow in project with a submodule

[Stefan Naewe]

On 5/20/2010 12:01 PM, Andy Parkins wrote:
> Hello,
> 
> I've just upgraded from 1.6.something to 1.7.1.  All very nice.
> 
> The new submodule reporting is nice too; but I'd like to be able to turn it 
> off :-)
> 
> The problem is that I have a (relatively) small project as the supermodule, 
> and a linux kernel clone as a submodule and an ffmpeg clone as a submodule.  
> Now I used to be able to do git-status or git-diff and it would be instant, 
> it now takes a number of seconds to report.  I guess (but don't know), that 
> it is the detection of "dirty" status in the submodule's that is slowing 
> down the supermodule processing.
> 
> I wouldn't like to see the feature go, because in almost all circumstances 
> it is exactly right; however, I'd like to be able to turn off dirty 
> detection in submodules.  Is this already possible, and I've just missed the 
> configuration option?

Maybe:

   git config status.submodulesummary false

> One additional small point: why do untracked files in a submodule make the 
> module dirty?  I've often got a few "temp.ps" or "debug.log" or 
> "backtrace.log" files lying around -- inappropriate to add to an ignore 
> file, but they don't make my working directory dirty.  "Dirty" in a working 
> directory means uncommitted changes to tracked files, why does it mean 
> something different in a submodule?

That's IMHO a good point. I'd like to get an answer for that, too.

Regards,
  Stefan
-- 
----------------------------------------------------------------
/dev/random says: ... Clinton excuse #15: Hey - I just do what the wife says

Re: git-status and git-diff now very slow in project with a submodule

[Andy Parkins]

Stefan Naewe wrote:

>> circumstances it is exactly right; however, I'd like to be able to turn
>> off dirty
>> detection in submodules.  Is this already possible, and I've just missed
>> the configuration option?
> 
> Maybe:
> 
>    git config status.submodulesummary false

Hey! Thanks for the reply. Exactly the right option... except it doesn't work :-(

$ git --version
git version 1.7.1
$ git config status.submodulesummary
false
$ git status -uno
# On branch master
# Changed but not updated:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#   (commit or discard the untracked or modified content in submodules)
#
#       modified:   ffmpeg (modified content)
#
no changes added to commit (use "git add" and/or "git commit -a")




Andy

-- 
Dr Andy Parkins
andyparkins@gmail.com

Re: git-status and git-diff now very slow in project with a submodule

[Michael J Gruber]

Andy Parkins venit, vidit, dixit 20.05.2010 13:37:
> Stefan Naewe wrote:
> 
>>> circumstances it is exactly right; however, I'd like to be able to turn
>>> off dirty
>>> detection in submodules.  Is this already possible, and I've just missed
>>> the configuration option?
>>
>> Maybe:
>>
>>    git config status.submodulesummary false
> 
> Hey! Thanks for the reply. Exactly the right option... except it doesn't work :-(
> 
> $ git --version
> git version 1.7.1
> $ git config status.submodulesummary
> false
> $ git status -uno
> # On branch master
> # Changed but not updated:
> #   (use "git add <file>..." to update what will be committed)
> #   (use "git checkout -- <file>..." to discard changes in working directory)
> #   (commit or discard the untracked or modified content in submodules)
> #
> #       modified:   ffmpeg (modified content)
> #
> no changes added to commit (use "git add" and/or "git commit -a")

You see: No submodule summary here!
Try setting the variable to true and see the difference. False is the
default.

Git needs to check the submodule in order to produce the "modified" line
even when no summary is required. Stopping Git from looking at the
submodule at all is impossible, I think. One could only hope that it
stops scanning after the first modification.

Michael

Re: git-status and git-diff now very slow in project with a submodule

[Andy Parkins]

Michael J Gruber wrote:

> You see: No submodule summary here!
> Try setting the variable to true and see the difference. False is the
> default.

Quite so; I hadn't understood submodulesummary -- I just tried it when it 
was suggested.

> Git needs to check the submodule in order to produce the "modified" line
> even when no summary is required. Stopping Git from looking at the

I realise that -- what I was after is a return to the old behaviour -- under 
the control of an option.

> submodule at all is impossible, I think. One could only hope that it
> stops scanning after the first modification.

"Impossible" is a strong word for a behaviour that existed pre-1.7.

It's not that I want git not to look at the submodule at all; in fact it 
certainly should for those cases when the submodule commit has changed, and 
I guess that a check for a dirty index is pretty quick too; but scanning the 
whole submodule tree (which it has to do to find if anything was modified, 
even when nothing was modified) is a lot of extra time when the submodule is 
large.  That extra time is inconvenient when you're working on a small 
project that makes use of a large project as a submodule.  (Most of my 
personal use of submodule is embedding large projects that I want to be able 
to guarantee are at a particular version, but I don't really change them)




Andy

-- 
Dr Andy Parkins
andyparkins@gmail.com

Re: git-status and git-diff now very slow in project with a submodule

[Jens Lehmann]

Am 20.05.2010 19:45, schrieb Andy Parkins:
> (Most of my
> personal use of submodule is embedding large projects that I want to be able 
> to guarantee are at a particular version, but I don't really change them)

But to guarantee they are at a particular version they have to be checked
for local modifications (no matter if they happened accidentally or on
purpose), no?

Re: git-status and git-diff now very slow in project with a submodule

[Andy Parkins]

On Thursday 20 May 2010 18:49:41 Jens Lehmann wrote:
> Am 20.05.2010 19:45, schrieb Andy Parkins:
> > (Most of my
> > personal use of submodule is embedding large projects that I want to be
> > able to guarantee are at a particular version, but I don't really
> > change them)
> 
> But to guarantee they are at a particular version they have to be checked
> for local modifications (no matter if they happened accidentally or on
> purpose), no?

A valid point.

Surely though in my own .git/config I can be allowed to tell git that I 
don't care about that risk?

I've got a top level module that used to diff/status instantly; now git 
scans an entire Linux kernel checkout and an entire ffmpeg checkout.  
Painful.  I fully accept that it was my own choice to arrange my repository 
in this way, but in my defence, it was fine last week :-)



Andy

-- 
Dr Andy Parkins
andyparkins@gmail.com

Re: git-status and git-diff now very slow in project with a submodule

[Nguyen Thai Ngoc Duy]

On Thu, May 20, 2010 at 8:01 PM, Andy Parkins <andyparkins@gmail.com> wrote:
> On Thursday 20 May 2010 18:49:41 Jens Lehmann wrote:
>> Am 20.05.2010 19:45, schrieb Andy Parkins:
>> > (Most of my
>> > personal use of submodule is embedding large projects that I want to be
>> > able to guarantee are at a particular version, but I don't really
>> > change them)
>>
>> But to guarantee they are at a particular version they have to be checked
>> for local modifications (no matter if they happened accidentally or on
>> purpose), no?
>
> A valid point.
>
> Surely though in my own .git/config I can be allowed to tell git that I
> don't care about that risk?
>
> I've got a top level module that used to diff/status instantly; now git
> scans an entire Linux kernel checkout and an entire ffmpeg checkout.
> Painful.  I fully accept that it was my own choice to arrange my repository
> in this way, but in my defence, it was fine last week :-)

Sounds similar to the old "slow lstat" stuff. How about setting
assume-unchanged bit on submodules? I don't know if it works. If it
does not, maybe we could reuse the bit to ignore changes in
submodules.
-- 
Duy

Re: git-status and git-diff now very slow in project with a submodule

[Junio C Hamano]

Andy Parkins <andyparkins@gmail.com> writes:

> One additional small point: why do untracked files in a submodule make the 
> module dirty?  I've often got a few "temp.ps" or "debug.log" or 
> "backtrace.log" files lying around -- inappropriate to add to an ignore 
> file, but they don't make my working directory dirty.

"They don't make my working directory dirty" is something only you can
decide, until you tell git about that fact, isn't it?

The way to tell git about them is to use the ignore/exclude mechanism.
Why are they "inappropriate to add to an ignore file"?  At least you could
have "*.log" in your personal exclude $GIT_DIR/info/exclude, no?

As to the not-working-configuration I don't remember the codepath well, so
sorry but no answer from me right now.

Re: git-status and git-diff now very slow in project with a submodule

[Andy Parkins]

On Thursday 20 May 2010 14:28:35 Junio C Hamano wrote:

> > One additional small point: why do untracked files in a submodule make
> > the module dirty?  I've often got a few "temp.ps" or "debug.log" or
> > "backtrace.log" files lying around -- inappropriate to add to an
> > ignore file, but they don't make my working directory dirty.
> 
> "They don't make my working directory dirty" is something only you can
> decide, until you tell git about that fact, isn't it?

Perhaps I've misunderstood then; I have always understood that "dirty" was 
the name we give to the state when tracked files have changes in the working 
directory.  If not, then what word should be used to distinguish between 
tracked files unchecked in and untracked files?

Anyhoo; I don't mind.  Me starting a semantics debate isn't helpful is it?

> The way to tell git about them is to use the ignore/exclude mechanism.
> Why are they "inappropriate to add to an ignore file"?  At least you
> could have "*.log" in your personal exclude $GIT_DIR/info/exclude, no?

I think you've taken me too literally; I was trying to get across the idea 
that they are files that are made on the fly, and when I notice them they 
just get deleted.

Also, I don't want *.log, or *.ps -- neither of them is guaranteed to be an 
ignore pattern.  These throw away files have all sorts of names, made up on 
the spot as I'm working, adding them to an ignore file is overkill from my 
point of view.



Andy
-- 
Dr Andy Parkins
andyparkins@gmail.com

Re: git-status and git-diff now very slow in project with a submodule

[Junio C Hamano]

Andy Parkins <andyparkins@gmail.com> writes:

> Also, I don't want *.log, or *.ps -- neither of them is guaranteed to be an 
> ignore pattern.  These throw away files have all sorts of names, made up on 
> the spot as I'm working, adding them to an ignore file is overkill from my 
> point of view.

I've learned to give them names that begin with an unusual letter (in my
case, a colon or a plus sign), way before I started working on git, so the
above is not a very convincing argument at least to me.

In any case, I am a bit torn about this whole issue.

On one hand, scanning for untracked files are not about these *.log cruft,
but to catch mistakes that are caused by new paths that you forgot to add,
and in that sense, uncommitted modifications to a path that happens to be
tracked and a new path that you forgot to add have (semantically) similar
chance of being a mistake that you might want to catch by running "git
status".

On the other hand, adding new paths to an existing project is a rare
event, compared to modifications to existing paths (e.g. even for a
project as small and young as git.git, we have 10x as many revisions as we
have paths), so by definition the chance that you might break others'
builds by forgetting to commit a new file is much smaller than forgetting
to commit necessary changes to existing files.

But ideally you would want your tool to catch mistakes that are rarer, as
you would learn to avoid common mistakes on your own without help from
your tool over time.

At least we should be able to let the users say, with "git status -uno",
"I don't care about untracked and unignored paths; I don't make such a
mistake to forget adding new paths", and optimize the scanning of
submodule directories taking advantage of that statement.  Is there a
fundamental reason why things shouldn't work that way, or is it just a
bug in the current code?

Re: git-status and git-diff now very slow in project with a submodule

[Jens Lehmann]

Am 21.05.2010 00:59, schrieb Junio C Hamano:
> At least we should be able to let the users say, with "git status -uno",
> "I don't care about untracked and unignored paths; I don't make such a
> mistake to forget adding new paths", and optimize the scanning of
> submodule directories taking advantage of that statement.  Is there a
> fundamental reason why things shouldn't work that way, or is it just a
> bug in the current code?

It works that way since 3bfc450. "git status" (and the diff family when
comparing against the work tree) forks a "git status" for each submodule
that is populated. If the status command in the superproject is started
with "-uno" the same option is used for the "git status" forked in the
populated submodules, so no checking for untracked files is done in that
case.

But that doesn't speed up that process much, as the tracked files inside
the submodules have to be checked for modifications anyway, no matter if
"-uno" is used or not. Getting rid of the fork of a new "git status" by
using an alternate odb is on my to do list. Apart from gaining some time
by avoiding the fork (which is a per-submodule-constant) we could terminate
early in case we find a modification (instead of continuing as the current
approach does). But only dirty submodules would profit from that, a clean
submodule won't be scanned much faster that way AFAICS.

Re: git-status and git-diff now very slow in project with a submodule

[Leo Razoumov]

On 2010-05-20, Junio C Hamano <gitster@pobox.com> wrote:
> Andy Parkins <andyparkins@gmail.com> writes:
>
>  > One additional small point: why do untracked files in a submodule make the
>  > module dirty?  I've often got a few "temp.ps" or "debug.log" or
>  > "backtrace.log" files lying around -- inappropriate to add to an ignore
>  > file, but they don't make my working directory dirty.
>
>
> "They don't make my working directory dirty" is something only you can
>  decide, until you tell git about that fact, isn't it?
>
>  The way to tell git about them is to use the ignore/exclude mechanism.
>  Why are they "inappropriate to add to an ignore file"?  At least you could
>  have "*.log" in your personal exclude $GIT_DIR/info/exclude, no?

Please, correct me if I am wrong, but I always thought that repo's
dirty designation has to do with changed files _in_ the repo.
Untracked files are _not_ in the repo and, as such, are irrelevant for
repo's clean/dirty status.

Speaking of .gitignore and untracked files. Explicitly mentioning all
such untracked files in .gitignore is often unpractical. For example,
during build process some large projects autogenerate many temporary
*.c  *.h *.cpp files. Hunting all of them down and adding to
.gitignore is a waste of time and one cannot use globs *.c *.h for
obvious reasons.

I think that it is consistent and logical to drop untracked files from
repo's dirty/clean decision process.

I would be interested to know of any counter-example: that is, a
use-case where it makes logical sense to declare a repo dirty when it
gets an untracked file not mentioned in .gitignore.

--Leo--

Re: git-status and git-diff now very slow in project with a submodule

[Andreas Schwab]

Leo Razoumov <slonik.az@gmail.com> writes:

> Speaking of .gitignore and untracked files. Explicitly mentioning all
> such untracked files in .gitignore is often unpractical. For example,
> during build process some large projects autogenerate many temporary
> *.c  *.h *.cpp files. Hunting all of them down and adding to
> .gitignore is a waste of time and one cannot use globs *.c *.h for
> obvious reasons.

You can actually, since tracked files are never ignored.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: git-status and git-diff now very slow in project with a submodule

[Jens Lehmann]

Am 21.05.2010 19:54, schrieb Andreas Schwab:
> Leo Razoumov <slonik.az@gmail.com> writes:
> 
>> Speaking of .gitignore and untracked files. Explicitly mentioning all
>> such untracked files in .gitignore is often unpractical. For example,
>> during build process some large projects autogenerate many temporary
>> *.c  *.h *.cpp files. Hunting all of them down and adding to
>> .gitignore is a waste of time and one cannot use globs *.c *.h for
>> obvious reasons.
> 
> You can actually, since tracked files are never ignored.

Hm, but that would mean newly added files would never show up again
(the same .gitignore is used by the developers of that submodule when
they do work on it). So I wouldn't add *.c to .gitignore either ...

But I don't consider it a "waste of time" to get the .gitignore
straight for pretty much the same reasons I want to see no warnings
during compilation: So i can instantly see when something fishy
might be going on. I consider that a "best practice" instead.

And for those projects where you can't or don't want to change the
.gitignore: Just ignore when "git status" tells you a submodule has
"untracked content" (it shows that information since 9297f7).

Re: git-status and git-diff now very slow in project with a submodule

[Jens Lehmann]

Am 21.05.2010 14:52, schrieb Leo Razoumov:
> On 2010-05-20, Junio C Hamano <gitster@pobox.com> wrote:
> I would be interested to know of any counter-example: that is, a
> use-case where it makes logical sense to declare a repo dirty when it
> gets an untracked file not mentioned in .gitignore.

For submodules the presence of untracked files must be visible in the
superproject. When new files are added to a submodule but the user
forgot to commit them, the superproject might not even build at all
when another developer clones the superproject. And yes, this is a
real-life use case from my dayjob.

The alternative to these broader definitions about when to declare a
submodule dirty (new commits, modified content and untracked content)
would have been to handle all eight combinations of these states. In
all relevant parts of the toolchain. Which seems pretty insane. So
the submodule shows up as modified; and all but the short outputs of
diff and status also tell you /why/ it is considered modified. So the
user can decide what to do about that.