git ls-files handles paths differently in Windows and Mac (probably Linux)

git ls-files handles paths differently in Windows and Mac (probably Linux)

[Kirill Likhodedov]

Hi all,

The behavior of ls-files is inconsistent in Windows and Mac.

I want to see which files were changed in the specific directory of my working tree, so I call ls-files:
> git ls-files -douvm --exclude-standard -- MYDIR

On Mac (and probably on Linux) git inspects the content of MYDIR and gives me the status of not-indexed changes in this directory.
On Windows git gives the same result (only changes in MYDIR), but it scans the whole repository!

I noticed it because it took a long time to scan a small directory, and then I confirmed it by using monitoring tools:
on Windows I used Process Monitor, on Mac I used FileMon - both give the list of accessed files in the real time.

To narrow the search scope to MYDIR I have to add "/**" and thus call:
> git ls-files -douvm --exclude-standard -- MYDIR/**
This works as expected - scans only MYDIR.

But I can't call it with several directories:
> git ls-files -douvm --exclude-standard -- MYDIR/** ANOTHERDIR/**
This doesn't work as expected and scans the whole repo.
So to scan two directories I have to call ls-files twice.

Looks like a bug. Maybe a bug of porting Git to windows, but the behavior is the same on msys-git and cygwin-git. 
It was tested with Git 1.7.1 on Mac, 
MsysGit 1.7.3.1 and 1.7.0.2,
CygwinGit 1.7.0.4.

Thanks.

----------------------------------
Kirill Likhodedov
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

Re: git ls-files handles paths differently in Windows and Mac (probably Linux)

[Alex Riesen]

On Tue, Oct 12, 2010 at 14:14, Kirill Likhodedov
<Kirill.Likhodedov@jetbrains.com> wrote:
> The behavior of ls-files is inconsistent in Windows and Mac.
>
> I want to see which files were changed in the specific directory of my working tree, so I call ls-files:
>> git ls-files -douvm --exclude-standard -- MYDIR
>

Well, it is an... unconventional way to do that. May I suggest you consider
using "git diff --name-status" or just "git status" next time?

> Looks like a bug. Maybe a bug of porting Git to windows, but the behavior is the same on msys-git and cygwin-git.

Maybe. Both ports share a fair amount of Windows-related code.

Re: git ls-files handles paths differently in Windows and Mac (probably Linux)

[Kirill Likhodedov]

>> The behavior of ls-files is inconsistent in Windows and Mac.
>> 
>> I want to see which files were changed in the specific directory of my working tree, so I call ls-files:
>>> git ls-files -douvm --exclude-standard -- MYDIR
>> 
> 
> Well, it is an... unconventional way to do that. May I suggest you consider
> using "git diff --name-status" or just "git status" next time?

My tests show that "git ls-files -douvm" is faster than "git diff --name-status".

"git status" doesn't fit, because it's porcelain (btw, git diff is also porcelain) and its output may change in time;
"git status --porcelain" appeared only in 1.7.0, so I can't use it to work with older Git.


----------------------------------
Kirill Likhodedov
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

Re: git ls-files handles paths differently in Windows and Mac (probably Linux)

[Kirill Likhodedov]

>>> I want to see which files were changed in the specific directory of my working tree, so I call ls-files:
>>>> git ls-files -douvm --exclude-standard -- MYDIR
>>> 
>> 
>> Well, it is an... unconventional way to do that. May I suggest you consider
>> using "git diff --name-status" or just "git status" next time?
> 
> My tests show that "git ls-files -douvm" is faster than "git diff --name-status".
> 
> "git status" doesn't fit, because it's porcelain (btw, git diff is also porcelain) and its output may change in time;
> "git status --porcelain" appeared only in 1.7.0, so I can't use it to work with older Git.
> 


And also "git diff --name-status" doesn't show unversioned files.
As I've found, "git ls-files" is the only command which shows unversioned files (except git status), isn't it?



----------------------------------
Kirill Likhodedov
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

Re: git ls-files handles paths differently in Windows and Mac (probably Linux)

[Alex Riesen]

On Tue, Oct 12, 2010 at 15:52, Kirill Likhodedov
<Kirill.Likhodedov@jetbrains.com> wrote:
> And also "git diff --name-status" doesn't show unversioned files.
> As I've found, "git ls-files" is the only command which shows unversioned files (except git status), isn't it?

Yes, you're right. What are you trying to do, BTW?

Re: git ls-files handles paths differently in Windows and Mac (probably Linux)

[Kirill Likhodedov]

12.10.2010, в 19:25, Alex Riesen написал(а):
> 
> Besides, if you have to just check if the tree under a path is changed
> you can always use
> --exit-code or --quiet to "git diff", it will speed them up.

Unfortunately, that's not enough: I have to know which files have changed and how (created, deleted, modified).
Also I feel that it's not the size of the output which is slow - the scan of directories itself is. 


>> And also "git diff --name-status" doesn't show unversioned files.
>> As I've found, "git ls-files" is the only command which shows unversioned files (except git status), isn't it?
> 
> Yes, you're right. What are you trying to do, BTW?

I'm developing a plugin for Git integration for IntelliJ IDEA (an IDE for Java and other languages).
And the task I'm working now is to get the status of index and working tree. I want to make it as fast as possible.
To work with a Git repository I start a process with git native commands, so every command to execute is additional cost.
(Maybe I could work with the filesystem and raw Git repository, but it would be an overhead).

If we forget about "git status --porcelain" there is no single command to get the whole status. I could use 
"git ls-files -ov" for unversioned files + "git diff --name-status" for all other.
OR 
"git ls-files -douvm" for the changes in the working tree + "git diff-index --cached" for the changes in the index.
OR
"git ls-files -ov" for unversioned + "git diff-files" for other files in the working tree + "git diff-index --cached" for the changes in the index.
or maybe something else

I've made a script which executes these commands (which allow to get the total status) on a large repository many times. 
My results show that "ls-files -douvm"+"git diff-index --cached" is the fastest combination.

Do you think I could achieve the same result in a simpler or faster way?

Thanks a lot.

----------------------------------
Kirill Likhodedov
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

Re: git ls-files handles paths differently in Windows and Mac (probably Linux)

[Alex Riesen]

2010/10/13 Kirill Likhodedov <Kirill.Likhodedov@jetbrains.com>:
>
> 12.10.2010, в 19:25, Alex Riesen написал(а):
>>
>> Besides, if you have to just check if the tree under a path is changed
>> you can always use
>> --exit-code or --quiet to "git diff", it will speed them up.
>
> Unfortunately, that's not enough: I have to know which files have changed and how (created, deleted, modified).
> Also I feel that it's not the size of the output which is slow - the scan of directories itself is.

Yeah, that's Windows for you.

> I've made a script which executes these commands (which allow to get the total status) on a large
> repository many times.
> My results show that "ls-files -douvm"+"git diff-index --cached" is the fastest combination.
>
> Do you think I could achieve the same result in a simpler or faster way?

No, don't think so. But you can run the commands in parallel, I suppose.

And of course a patch implementing a command that does this (or improving
the performance of "git diff") will be gladly considered here.

Re: git ls-files handles paths differently in Windows and Mac (probably Linux)

[Jakub Narebski]

Kirill Likhodedov <Kirill.Likhodedov@jetbrains.com> writes:
> 12.10.2010, в 19:25, Alex Riesen написал(а):

> > [..]. What are you trying to do, BTW?
> 
> I'm developing a plugin for Git integration for IntelliJ IDEA (an IDE for
> Java and other languages).
>
> And the task I'm working now is to get the status of index and working
> tree. I want to make it as fast as possible.  To work with a Git
> repository I start a process with git native commands, so every command to
> execute is additional cost.  (Maybe I could work with the filesystem and
> raw Git repository, but it would be an overhead).

Isn't IntelliJ IDEA written itself in Java?  Perhaps you could use JGit, the
Java implementation of Git (though it is not complete implementation yet, if
I remember correctly).  It is EDL (new-style BSD) licensed, and is used by
Eclipse and NetBeans.

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: git ls-files handles paths differently in Windows and Mac (probably Linux)

[Kirill Likhodedov]

> Isn't IntelliJ IDEA written itself in Java?  Perhaps you could use JGit, the
> Java implementation of Git (though it is not complete implementation yet, if
> I remember correctly).  It is EDL (new-style BSD) licensed, and is used by
> Eclipse and NetBeans.
> 

Yep, we thought about JGit. 
But people say that it's not stable enough, and not completely implemented. 
Personally I haven't tried it yet though.

----------------------------------
Kirill Likhodedov
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

Re: git ls-files handles paths differently in Windows and Mac (probably Linux)

[Alex Riesen]

On Tue, Oct 12, 2010 at 14:59, Kirill Likhodedov
<Kirill.Likhodedov@jetbrains.com> wrote:
>>> The behavior of ls-files is inconsistent in Windows and Mac.
>>>
>>> I want to see which files were changed in the specific directory of my working tree, so I call ls-files:
>>>> git ls-files -douvm --exclude-standard -- MYDIR
>>>
>>
>> Well, it is an... unconventional way to do that. May I suggest you consider
>> using "git diff --name-status" or just "git status" next time?
>
> My tests show that "git ls-files -douvm" is faster than "git diff --name-status".

I find it hard to believe (the tool was definitely not mean for this).

> "git status" doesn't fit, because it's porcelain (btw, git diff is also porcelain) and its output may change in time;

There is always "git diff-files", but I doubt the output of "git diff
--name-status" will change soon.
Besides, if you have to just check if the tree under a path is changed
you can always use
--exit-code or --quiet to "git diff", it will speed them up.

Re: git ls-files handles paths differently in Windows and Mac (probably Linux)

[Johannes Sixt]

Am 10/12/2010 14:14, schrieb Kirill Likhodedov:
> I want to see which files were changed in the specific directory of my
> working tree, so I call ls-files:
>> git ls-files -douvm --exclude-standard -- MYDIR
> 
> On Mac (and probably on Linux) git inspects the content of MYDIR and
> gives me the status of not-indexed changes in this directory. On
> Windows git gives the same result (only changes in MYDIR), but it scans
> the whole repository!

On Linux, this scans the whole repository, too. Therefore, at least the
Windows version should not be *that* wrong. ;-)

-- Hannes

Re: git ls-files handles paths differently in Windows and Mac (probably Linux)

[Kirill Likhodedov]

Hannes,

how did you figure that? I mean - which tool did you use to see which files are accessed in Linux? 



12.10.2010, в 17:11, Johannes Sixt написал(а):

> Am 10/12/2010 14:14, schrieb Kirill Likhodedov:
>> I want to see which files were changed in the specific directory of my
>> working tree, so I call ls-files:
>>> git ls-files -douvm --exclude-standard -- MYDIR
>> 
>> On Mac (and probably on Linux) git inspects the content of MYDIR and
>> gives me the status of not-indexed changes in this directory. On
>> Windows git gives the same result (only changes in MYDIR), but it scans
>> the whole repository!
> 
> On Linux, this scans the whole repository, too. Therefore, at least the
> Windows version should not be *that* wrong. ;-)
> 
> -- Hannes

----------------------------------
Kirill Likhodedov
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

Re: git ls-files handles paths differently in Windows and Mac (probably Linux)

[Johannes Sixt]

Please don't top-post.

Am 10/12/2010 15:24, schrieb Kirill Likhodedov:
> how did you figure that? I mean - which tool did you use to see which
> files are accessed in Linux?

strace -e stat,lstat -o /tmp/ls-files.strace git ls-files ...

Could work on MacOS as well.

-- Hannes