Efficient parsing of `status -z` output

Efficient parsing of `status -z` output

[Matthew Rothenberg]

I've been working on a utility that parses the output of `git status
--porcelain` as a fundamental part of its operation.

Since I would like for this tool to be as robust as possible (and
cross-platform compatibility is a goal), I am currently trying to
migrate it from parsing the output of `--porcelain` to using the
output of `-z`, to quote from the documentation:

    > There is also an alternate -z format recommended for machine parsing. In
    > that format, the status field is the same, but some other things change.
    > First, the -> is omitted from rename entries and the field order is
    > reversed (e.g from -> to becomes to from). Second, a NUL (ASCII 0) follows
    > each filename, replacing space as a field separator and the terminating
    > newline (but a space still separates the status field from the first
    > filename). Third, filenames containing special characters are not
    > specially formatted; no quoting or backslash-escaping is performed.

I am encountering some significant issues with using this because of
one detail.  In particular, parsing output using NUL as *both* the
entry terminator and the filename separator for entries that contain
multiple files is problematic. Because of this, one cannot know in
advance how many NULs to read from the buffer until considering an
entry to be in memory for parsing.

There are two workarounds I've considered:

 1. Reading the *entire* buffer into memory, and then using a regular
expression (yikes) to split the entries. This is something I would
obviously like to avoid for performance reasons.

 2. Read from buffer until the first NUL, parse the entry status
codes, and if the entry status code represents a status that *should*
have multiple filenames, read from buffer until a second NUL is found,
and then reparse that entry with both filenames. The issues I see with
this approach:
   a.) One has to know exactly which status code combinations will end
up with two filenames, and this list has to be exhaustive. As far as I
can tell, there is no canonical documentation for this?
   b.) It seems a bit brittle, because if the logic from the above is
wrong and we miss an extended entry or ask for one when it doesn't
exist we will leave the buffer an essentially corrupt state for future
reads.

My understanding is the goal of `-z` is to make machine parsing status
from a binary stream *more* reliable, so perhaps (likely!) I am
missing something obvious?

Thanks for any assistance!

Re: Efficient parsing of `status -z` output

[Junio C Hamano]

Matthew Rothenberg <mrothenberg@gmail.com> writes:

>  2. Read from buffer until the first NUL, parse the entry status
> codes, and if the entry status code represents a status that *should*
> have multiple filenames, read from buffer until a second NUL is found,
> and then reparse that entry with both filenames. The issues I see with
> this approach:
>    a.) One has to know exactly which status code combinations will end
> up with two filenames, and this list has to be exhaustive. As far as I
> can tell, there is no canonical documentation for this?
>    b.) It seems a bit brittle, because if the logic from the above is
> wrong and we miss an extended entry or ask for one when it doesn't
> exist we will leave the buffer an essentially corrupt state for future
> reads.

I think this is how -z was designed to be used, and if that isn't
clear, then the documentation must be updated to clarify.  Rename
and Copy are the only ones that needs two pathnames, and I suspect
that whoever did the original description of the short format in the
documentation knew Git too well that he forgot to mention it ;-)

Re: Efficient parsing of `status -z` output

[Matthew Rothenberg]

On Sun, Mar 8, 2015 at 4:14 AM, Junio C Hamano <gitster@pobox.com> wrote:
> I think this is how -z was designed to be used, and if that isn't
> clear, then the documentation must be updated to clarify.  Rename
> and Copy are the only ones that needs two pathnames, and I suspect
> that whoever did the original description of the short format in the
> documentation knew Git too well that he forgot to mention it ;-)

I see, thank you. But how would one ever get a copy operation to show
up in the output of `git status -z` to begin with? It appears copies
are only detected in `diff` and `show`, can be forced with the
--find-copies-harder option, but that `git status` does not appear to
take that option nor detect copies in any way that I can get it to
replicate and output that status code to me... a test case would be
great if you know one, thanks!

Re: Efficient parsing of `status -z` output

[Jeff King]

On Sun, Mar 08, 2015 at 09:41:08PM -0400, Matthew Rothenberg wrote:

> I see, thank you. But how would one ever get a copy operation to show
> up in the output of `git status -z` to begin with? It appears copies
> are only detected in `diff` and `show`, can be forced with the
> --find-copies-harder option, but that `git status` does not appear to
> take that option nor detect copies in any way that I can get it to
> replicate and output that status code to me... a test case would be
> great if you know one, thanks!

We don't turn on copy-detection in "git status" by default (only rename
detection), and I think you are right that there is currently no way to
turn it on manually. However, it would probably be sensible to handle
"C" diffs in your parser, if only to future-proof against a day when
that changes (and because it should fairly trivial once you build "R"
support).

I don't know if anybody is actively working on such a change, but
somebody expressed interest recently-ish:

  http://thread.gmane.org/gmane.comp.version-control.git/260381

-Peff

Re: Efficient parsing of `status -z` output

[Jeff King]

On Mon, Mar 09, 2015 at 02:19:20AM -0400, Jeff King wrote:

> We don't turn on copy-detection in "git status" by default (only rename
> detection), and I think you are right that there is currently no way to
> turn it on manually.

Actually, I take it back. We do break-detection in git-status, which can
lead to finding a copy:

  $ git init
  $ seq 1 1000 >file && git add file && git commit -m base
  $ mv file other
  $ echo foo >file
  $ git add .
  $ git status --short
  M  file
  C  file -> other

-Peff

Re: Efficient parsing of `status -z` output

[Matthew Rothenberg]

On Mon, Mar 9, 2015 at 2:49 AM, Jeff King <peff@peff.net> wrote:
>   $ git init
>   $ seq 1 1000 >file && git add file && git commit -m base
>   $ mv file other
>   $ echo foo >file
>   $ git add .
>   $ git status --short
>   M  file
>   C  file -> other

Fantastic, I am able to replicate with these steps and will build
tests around this case.

For future proofing, from the documentation for git status is appears
the other two codes I would want to check for in addition to 'C '
(which this test cases generates) may be 'CM' and 'CD'? And all of
those should always have the additional PATH2 column present?

Thank you for your help!

Re: Efficient parsing of `status -z` output

[Jeff King]

On Mon, Mar 09, 2015 at 07:40:43PM -0400, Matthew Rothenberg wrote:

> On Mon, Mar 9, 2015 at 2:49 AM, Jeff King <peff@peff.net> wrote:
> >   $ git init
> >   $ seq 1 1000 >file && git add file && git commit -m base
> >   $ mv file other
> >   $ echo foo >file
> >   $ git add .
> >   $ git status --short
> >   M  file
> >   C  file -> other
> 
> Fantastic, I am able to replicate with these steps and will build
> tests around this case.
> 
> For future proofing, from the documentation for git status is appears
> the other two codes I would want to check for in addition to 'C '
> (which this test cases generates) may be 'CM' and 'CD'? And all of
> those should always have the additional PATH2 column present?

Yes, you can trivially make CM and CD by changing or deleting "other" in
the example above. I don't think you can ever have 'C' or 'R' in the
second column; we don't do renames on working tree changes, since a
"new" file there is simply untracked.

-Peff