cygwin, 44k files: how to commit only index?

cygwin, 44k files: how to commit only index?

[Alex Riesen]

I have a kind of awkward project to work with (~44k files, many binaries).

The normal "git commit", which seem to be more than enough
for anything and anyone else, is a really annoying procedure
in my context. It spend too much time refreshing index and
generating list of the files for the commit message.

At first I stopped using git commit -a (doing only update-index),
now I'm about to start using write-tree/commit-tree/update-ref
directly. It helps, but sometimes I really miss -F/-C. It's also
ugly: I can (and almost did) commit an unchanged tree.

Is there any simple way to modify git commit for such a workflow?
Failing that, any simple and _fast_ way to find out if the index

Re: cygwin, 44k files: how to commit only index?

[Junio C Hamano]

"Alex Riesen" <raa.lkml@gmail.com> writes:

> I have a kind of awkward project to work with (~44k files, many binaries).
>
> The normal "git commit", which seem to be more than enough
> for anything and anyone else, is a really annoying procedure
> in my context. It spend too much time refreshing index and
> generating list of the files for the commit message.
>
> At first I stopped using git commit -a (doing only update-index),

I am not sure what you are trying.  Do you mean stat() is slow
on your filesystem?

> Is there any simple way to modify git commit for such a workflow?
> Failing that, any simple and _fast_ way to find out if the index
> is any different from HEAD? (so that I don't produce empty commits).

Maybe you want "assume unchanged"?

Re: cygwin, 44k files: how to commit only index?

[Shawn Pearce]

Junio C Hamano <junkio@cox.net> wrote:
> "Alex Riesen" <raa.lkml@gmail.com> writes:
> 
> > I have a kind of awkward project to work with (~44k files, many binaries).
> >
> > The normal "git commit", which seem to be more than enough
> > for anything and anyone else, is a really annoying procedure
> > in my context. It spend too much time refreshing index and
> > generating list of the files for the commit message.
> >
> > At first I stopped using git commit -a (doing only update-index),
> 
> I am not sure what you are trying.  Do you mean stat() is slow
> on your filesystem?

Its Cygwin/NTFS.  lstat() is slow.  readdir() is slow.  I have the
same problem on my Cygwin systems.
 
> > Is there any simple way to modify git commit for such a workflow?
> > Failing that, any simple and _fast_ way to find out if the index
> > is any different from HEAD? (so that I don't produce empty commits).
> 
> Maybe you want "assume unchanged"?

Yes, basically.  The Cygwin/NTFS issues Alex is pointing out are
exactly why git-gui has a "Trust File Modification Timestamp" option
on both a per-repository and global level.  My larger repositories
(~10k files) are difficult to work with without that option enabled.

-- 

Re: cygwin, 44k files: how to commit only index?

[Shawn Pearce]

Shawn Pearce <spearce@spearce.org> wrote:
> Its Cygwin/NTFS.  lstat() is slow.  readdir() is slow.  I have the
> same problem on my Cygwin systems.

Just to be clear, I'm not trying to blame Cygwin here.

Windows' dir command is slow.  Windows Explorer is slow while
browsing directories.  Eclipse chugs hard while doing any directory
scans (it normally runs very fast if its not rescanning the entire
directory structure).  The drive is just plain slow.

Yea, I know, get a faster disk... but some bean counters don't
believe that a $50 more expensive disk could ever save enough time
to warrant the extra $50 captial expenditure...

I spend at least an hour a week waiting for enough IO to finish so
that the mouse pointer will move again.  *sigh*

-- 

Re: cygwin, 44k files: how to commit only index?

[Christian MICHON]

On 12/7/06, Shawn Pearce <spearce@spearce.org> wrote:
> Shawn Pearce <spearce@spearce.org> wrote:
> > Its Cygwin/NTFS.  lstat() is slow.  readdir() is slow.  I have the
> > same problem on my Cygwin systems.
>
> Just to be clear, I'm not trying to blame Cygwin here.
>
> Windows' dir command is slow.  Windows Explorer is slow while
> browsing directories.  Eclipse chugs hard while doing any directory
> scans (it normally runs very fast if its not rescanning the entire
> directory structure).  The drive is just plain slow.
> (...)

before buying any new hardware, you could easily imagine the
following scenario (I'm also "stuck" with windows, so it's an idea
I've been toying around for a week or so).

There're virtualizers around, on which networking capabilities can
be activated. And we could easily create a vm with linux+git
inside, using ext2/ext3/ext4 fs virtual disks (you'd benefit from
windows cache actually...)

example: YTech_Subversion_Appliance_v1.1 (ubuntu + subversion).

I've no prototype yet, but I've 2 scenario possible:
1) use vmplayer and a minimal uclibc initramfs with git onboard
2) use qemu+kqemu and a similar mini-distro (but right now networking
is an issue on windows hosts: I'm exploring tunneling)

The 1st scenario is "easy". And I start to prefer this idea over
even mingw porting of git (I tried and it's hard, really).

But again, maybe jgit would be a better universal solution.

-- 

Re: cygwin, 44k files: how to commit only index?

[Torgil Svensson]

On 12/7/06, Christian MICHON <christian.michon@gmail.com> wrote:
> On 12/7/06, Shawn Pearce <spearce@spearce.org> wrote:
> > Shawn Pearce <spearce@spearce.org> wrote:
> > > Its Cygwin/NTFS.  lstat() is slow.  readdir() is slow.  I have the
> > > same problem on my Cygwin systems.
> >
> > Just to be clear, I'm not trying to blame Cygwin here.
> >
> > Windows' dir command is slow.  Windows Explorer is slow while
> > browsing directories.

I think this is a very common scenario costing hideous amounts of
money around the globe.

If you have lot's of files in a folder, don't even think of
accidentally touching those folders in Windows Explorer, if you do -
keep Process Explorer or similar ready. I've ended up using (even w/o
Cygwin) scripts, automatic compressing and even a database functioning
as directory cache - basically creating accessibility layers for a
disabled file-system.


>
> before buying any new hardware, you could easily imagine the
> following scenario (I'm also "stuck" with windows, so it's an idea
> I've been toying around for a week or so).
>
> There're virtualizers around, on which networking capabilities can
> be activated. And we could easily create a vm with linux+git
> inside, using ext2/ext3/ext4 fs virtual disks (you'd benefit from
> windows cache actually...)
>
> example: YTech_Subversion_Appliance_v1.1 (ubuntu + subversion).
>
> I've no prototype yet, but I've 2 scenario possible:
> 1) use vmplayer and a minimal uclibc initramfs with git onboard
> 2) use qemu+kqemu and a similar mini-distro (but right now networking
> is an issue on windows hosts: I'm exploring tunneling)
>
> The 1st scenario is "easy". And I start to prefer this idea over
> even mingw porting of git (I tried and it's hard, really).
>
> But again, maybe jgit would be a better universal solution.
>
> --
> Christian
> -

Very interesting!  Have you a time-frame for this?  Maybe even

Re: cygwin, 44k files: how to commit only index?

[Junio C Hamano]

Shawn Pearce <spearce@spearce.org> writes:

>> I am not sure what you are trying.  Do you mean stat() is slow
>> on your filesystem?
>
> Its Cygwin/NTFS.  lstat() is slow.  readdir() is slow.  I have the
> same problem on my Cygwin systems.
>  
>> > Is there any simple way to modify git commit for such a workflow?
>> > Failing that, any simple and _fast_ way to find out if the index
>> > is any different from HEAD? (so that I don't produce empty commits).
>> 
>> Maybe you want "assume unchanged"?
>
> Yes, basically.

Then maybe "git grep assume.unchanged" would help?

Re: cygwin, 44k files: how to commit only index?

[Shawn Pearce]

Junio C Hamano <junkio@cox.net> wrote:
> Shawn Pearce <spearce@spearce.org> writes:
> 
> >> I am not sure what you are trying.  Do you mean stat() is slow
> >> on your filesystem?
> >> 
> >> Maybe you want "assume unchanged"?
> >
> > Yes, basically.
> 
> Then maybe "git grep assume.unchanged" would help?

Hmm.  OK, maybe I should have answered "No"" to your first question.
I keep looking at the assume unchanaged feature of update-index,
but refuse to use it because I'm a lazy guy who will forget to tell
the index a file has been modified.  Consequently I'm going to miss
a change during a commit.

What may help (and without using assume unchanged) is:

 * skip the `update-index --refresh` part of git-status/git-commit
 * skip the status template in COMMIT_MSG when using the editor

As Git will still at least make sure a `commit -a` includes
everything that is dirty.

Files whose modification dates may have been messed with (but
whose content are unchanged) will just go through expensive SHA1
computation to arrive at the same value, which is fine.

Users skipping the first part are doing so under the assumption that
their modification dates are usually always correct, and that then
they aren't the SHA1 computation of a handful of files is cheap
compared to stat'ing the entire set of files.

Users skipping the second part are doing so under the assumption
that knowing the names of the files they are committing doesn't
really improve their odds of writing a good commit message.

-- 

Re: cygwin, 44k files: how to commit only index?

[Junio C Hamano]

Shawn Pearce <spearce@spearce.org> writes:

> What may help (and without using assume unchanged) is:
>
>  * skip the `update-index --refresh` part of git-status/git-commit
>  * skip the status template in COMMIT_MSG when using the editor
>
> As Git will still at least make sure a `commit -a` includes
> everything that is dirty.
>
> Files whose modification dates may have been messed with (but
> whose content are unchanged) will just go through expensive SHA1
> computation to arrive at the same value, which is fine.
>
> Users skipping the first part are doing so under the assumption that
> their modification dates are usually always correct, and that then
> they aren't the SHA1 computation of a handful of files is cheap
> compared to stat'ing the entire set of files.
>
> Users skipping the second part are doing so under the assumption
> that knowing the names of the files they are committing doesn't
> really improve their odds of writing a good commit message.

The second part is not about a good commit message but more
about a path that should have been updated but forgotten (the
same mistake you would be likely to make and that is the reason
assume-unchanged is not good for you).

I do not mind too much if you added a new --quick option to "git
commit" for this rather specialized need.

Re: cygwin, 44k files: how to commit only index?

[Alex Riesen]

Junio C Hamano, Thu, Dec 07, 2006 20:16:39 +0100:
> > I have a kind of awkward project to work with (~44k files, many binaries).
> >
> > The normal "git commit", which seem to be more than enough
> > for anything and anyone else, is a really annoying procedure
> > in my context. It spend too much time refreshing index and
> > generating list of the files for the commit message.
> >
> > At first I stopped using git commit -a (doing only update-index),
> 
> I am not sure what you are trying.  Do you mean stat() is slow
> on your filesystem?

incredibly slow. That and the matter of having 44000 files to process
with that slow stat().

> > Is there any simple way to modify git commit for such a workflow?
> > Failing that, any simple and _fast_ way to find out if the index
> > is any different from HEAD? (so that I don't produce empty commits).
> 
> Maybe you want "assume unchanged"?
> 

If that is core.ignoreState you mean, than maybe this is what I mean.
I haven't tried it yet (now I wonder myself why I haven't tried it).
But (I'm repeating myself, in <81b0412b0612060235l5d5f93d0hd1aaf34924f7783@mail.gmail.com>)
I do not really understand how it _can_ help: "I ask because it does
not ignore stat info, as the name implies. Because if it would,
there'd be no point of calling lstat at all, wouldn't it?" That last
question was about refresh_cache_entry - it calls lstat
unconditionally.

Still, I guess I'll have to try it.

But aside from me trying ignoreState, can anyone help me with that
question regarding checking if the index is any different from HEAD?
Because even on a very brocken filesystem and 40k files in a repo you
sometimes do want to call git-update-index --refresh just to be sure
you haven't missed anything. And than it'll quickly become annoying
flicking ignoreState back and forth.

Re: cygwin, 44k files: how to commit only index?

[Junio C Hamano]

fork0@t-online.de (Alex Riesen) writes:

> But aside from me trying ignoreState, can anyone help me with that
> question regarding checking if the index is any different from HEAD?

Comparing index and HEAD should be cheap on a system with slow
lstat(), I think, as "git-diff-index --cached HEAD" should just
ignore the working tree altogether.  Is that what you want?

Re: cygwin, 44k files: how to commit only index?

[Alex Riesen]

Junio C Hamano, Thu, Dec 07, 2006 23:29:54 +0100:
> > But aside from me trying ignoreState, can anyone help me with that
> > question regarding checking if the index is any different from HEAD?
> 
> Comparing index and HEAD should be cheap on a system with slow
> lstat(), I think, as "git-diff-index --cached HEAD" should just
> ignore the working tree altogether.  Is that what you want?
> 

yes, except that it'll compare the whole trees. Could I make it stop
at first mismatch? "-q|--quiet" for git-diff-index perhaps?
It's just not only stat, but also, open, read, mmap (yes, I try to use
it for packs) and close are really slow here as well.

Re: cygwin, 44k files: how to commit only index?

[Junio C Hamano]

fork0@t-online.de (Alex Riesen) writes:

> yes, except that it'll compare the whole trees. Could I make it stop
> at first mismatch? "-q|--quiet" for git-diff-index perhaps?
> It's just not only stat, but also, open, read, mmap (yes, I try to use
> it for packs) and close are really slow here as well.

That sounds like optimizing for a wrong case -- you expect the
index to match HEAD and trying to catch mistakes by detecting
a mismatch, right?

Having said that, I should point out that it is a low hanging
fruit to optimize "diff-index --cached" for cases where index
is expected to mostly match HEAD.

The current code for "diff-index --cached" reads the whole tree
into the index as stage #1 entries (diff-lib.c::run_diff_index),
and then compares stage #0 (from the original index contents)
and stage #1 (the tree parameter from the command line).  Even
if you stop at the first mismatch, you would already have paid
the overhead to open and read all tree objects before even
starting the comparison.

However, this code is from the ancient time before cache-tree
was introduced in the index.  If the index is expected to mostly
match HEAD, most of the cache-tree nodes are up-to-date, and
whole subtree can be skipped with a single comparison between
two tree SHA-1s at a shallower level of the directory tree.

In 'pu' (jc/diff topic), I have a very generic code to walk the
index, working tree and zero or more trees in parallel, taking
advantage of cache-tree.  If somebody is interested to learn the
internals of git, some of the code could be lifted from there
and simplified to walk just the index and a single tree, and I
think that would optimize "diff-index --cached" quite a bit.

A very unscientific test of running in the kernel repository I
just pulled (hot cache) on my box is:

$ /usr/bin/time git diff-index -r --cached --abbrev v2.6.19 >/tmp/1
0.91user 0.20system 0:01.12elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+10949minor)pagefaults 0swaps

while the para-walk to produce the moral equivalent is:

$ /usr/bin/time test-para --no-work v2.6.19 >/tmp/2
0.11user 0.02system 0:00.13elapsed 98%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+4524minor)pagefaults 0swaps

Re: cygwin, 44k files: how to commit only index?

[Alex Riesen]

On 12/8/06, Junio C Hamano <junkio@cox.net> wrote:
> > yes, except that it'll compare the whole trees. Could I make it stop
> > at first mismatch? "-q|--quiet" for git-diff-index perhaps?
> > It's just not only stat, but also, open, read, mmap (yes, I try to use
> > it for packs) and close are really slow here as well.
>
> That sounds like optimizing for a wrong case -- you expect the
> index to match HEAD and trying to catch mistakes by detecting
> a mismatch, right?

I expect the index to differ from HEAD. The test is to avoid the mistake
of doing an empty commit.

> Having said that, I should point out that it is a low hanging
> fruit to optimize "diff-index --cached" for cases where index
> is expected to mostly match HEAD.
>
> The current code for "diff-index --cached" reads the whole tree
> into the index as stage #1 entries (diff-lib.c::run_diff_index),
> and then compares stage #0 (from the original index contents)
> and stage #1 (the tree parameter from the command line).  Even
> if you stop at the first mismatch, you would already have paid
> the overhead to open and read all tree objects before even
> starting the comparison.

But I don't have to pay for the overhead of comparing all
entries, if I can stop at first mismatch and exit with non-0.
I think it'd make a difference (at least some difference).
But, if we could avoid loading of the entries which
will be never compared anyway, the speedup will be
of course more substantial...

> In 'pu' (jc/diff topic), I have a very generic code to walk the
> index, working tree and zero or more trees in parallel, taking
> advantage of cache-tree.  If somebody is interested to learn the
> internals of git, some of the code could be lifted from there
> and simplified to walk just the index and a single tree, and I
> think that would optimize "diff-index --cached" quite a bit.

Re: cygwin, 44k files: how to commit only index?

[Junio C Hamano]

"Alex Riesen" <raa.lkml@gmail.com> writes:

>> The current code for "diff-index --cached" reads the whole tree
>> into the index as stage #1 entries (diff-lib.c::run_diff_index),
>> and then compares stage #0 (from the original index contents)
>> and stage #1 (the tree parameter from the command line).  Even
>> if you stop at the first mismatch, you would already have paid
>> the overhead to open and read all tree objects before even
>> starting the comparison.
>
> But I don't have to pay for the overhead of comparing all
> entries, if I can stop at first mismatch and exit with non-0.

Bench it if you doubt me.

I'd bet that the time spent in comparison between stages inside
index (and remember, you are not generating textual diff, only
comparing the SHA-1) is dwarfed by the overhead of populating
the stage #1 of the index with what is read from all the tree
objects.

Re: cygwin, 44k files: how to commit only index?

[Alex Riesen]

On 12/8/06, Junio C Hamano <junkio@cox.net> wrote:
> >> The current code for "diff-index --cached" reads the whole tree
> >> into the index as stage #1 entries (diff-lib.c::run_diff_index),
> >> and then compares stage #0 (from the original index contents)
> >> and stage #1 (the tree parameter from the command line).  Even
> >> if you stop at the first mismatch, you would already have paid
> >> the overhead to open and read all tree objects before even
> >> starting the comparison.
> >
> > But I don't have to pay for the overhead of comparing all
> > entries, if I can stop at first mismatch and exit with non-0.
>
> Bench it if you doubt me.

I don't question that the overhead of comparing is very much
unnoticable. It just that it surely isn't zero, and it will grow with
the size of repo (linearly, right?)... And I am sure that this
repo will _only_ grow (typical corporate project).

> I'd bet that the time spent in comparison between stages inside
> index (and remember, you are not generating textual diff, only
> comparing the SHA-1) is dwarfed by the overhead of populating
> the stage #1 of the index with what is read from all the tree
> objects.

I already understood that. I just haven't found yet what can I do

Re: cygwin, 44k files: how to commit only index?

[Alex Riesen]

On 12/8/06, Alex Riesen <raa.lkml@gmail.com> wrote:
> On 12/8/06, Junio C Hamano <junkio@cox.net> wrote:
> > In 'pu' (jc/diff topic), I have a very generic code to walk the
> > index, working tree and zero or more trees in parallel, taking
> > advantage of cache-tree.  If somebody is interested to learn the
> > internals of git, some of the code could be lifted from there
> > and simplified to walk just the index and a single tree, and I
> > think that would optimize "diff-index --cached" quite a bit.
>
> Will try to look at it.
>

And now I'm playing with that (against test-para.c from pu).
I expect it to be broken by that webGmail, so it may not
apply to anything, but you'll get the idea. More clearly than
from me trying to explain.

commit 83642cdaca6dc1a2f94aa41923bc9e8f02d0e12f
Author: Alex Riesen <raa.lkml@gmail.com>
Date:   Fri Dec 8 09:38:18 2006 +0100

    add --quiet to test-para: stop at the first difference

diff --git a/test-para.c b/test-para.c
index bce5f0c..99d3792 100644
--- a/test-para.c
+++ b/test-para.c
@@ -21,6 +21,7 @@ int main(int ac, const char **av)
 	unsigned char trees[64][20];
 	int num_tree = 0, i, using_head, show_all = 0;
 	int index_wanted = 1, work_wanted = 1, tree_wanted = 0;
+	int quiet = 0;
 	const char *prefix;
 	const char **pathspec;

@@ -43,6 +44,8 @@ int main(int ac, const char **av)
 			work_wanted = 0;
 		else if (!strcmp(av[1] + 2, "no-index"))
 			index_wanted = 0;
+		else if (!strcmp(av[1] + 2, "quiet"))
+			quiet = 1;
 		else if (!av[1][2])
 			break;
 		else
@@ -118,7 +121,9 @@ int main(int ac, const char **av)
 				show_one(z, e->name, e->namelen,
 					 e->hash, e->mode);
 			}
-			else
+			else {
+				if (quiet)
+					exit(1);
 				for (i = 0; i < w.num_trees + 2; i++) {
 					char numbuf[10];

@@ -149,6 +154,7 @@ int main(int ac, const char **av)
 					show_one(z, e->name, e->namelen,
 						 e->hash, e->mode);
 				}
+			}
 		}

Re: cygwin, 44k files: how to commit only index?

[Alex Riesen]

On 12/7/06, Alex Riesen <fork0@t-online.de> wrote:
> > Maybe you want "assume unchanged"?
>
> If that is core.ignoreState you mean, than maybe this is what I mean.
> I haven't tried it yet (now I wonder myself why I haven't tried it).
> But (I'm repeating myself, in
> <81b0412b0612060235l5d5f93d0hd1aaf34924f7783@mail.gmail.com>)
> I do not really understand how it _can_ help: "I ask because it does
> not ignore stat info, as the name implies. Because if it would,
> there'd be no point of calling lstat at all, wouldn't it?" That last
> question was about refresh_cache_entry - it calls lstat
> unconditionally.
>
> Still, I guess I'll have to try it.
>

Tried. No noticeable difference:

$ git repo-config core.ignorestat true; time gup --refresh
real    0m8.004s
user    0m1.936s
sys     0m5.702s
$ git repo-config core.ignorestat false; time gup --refresh
real    0m7.787s
user    0m1.890s
sys     0m5.703s
$