Difficulties of scripting git

Difficulties of scripting git

[Alan Mackenzie]

Hello, git.

I want to write a bash script (more precisely, correct an existing
script) which uses git, and it is DIFFICULT!!!

I need a git pull in the script.  Fine, but how does the script know if
it's worked or not?  I couldn't find any description of a return code in
the git-pull man page (or in the git-merge man page).

The big problem is when I have modified, uncommitted files in the target
repo, and those same files are to be updated in commits coming from the
source repo.  Sadly, git is unable to merge these changes.  It just
fails, putting an error message onto stderr, but doesn't tell the
calling script in any way that I can see.

One idea would be always to call git stash before doing the pull, then
git stash pop afterwards.  Trouble is, git stash is unreliable - it
doesn't always add a new stash entry, so the stash pop at the end would
sometimes/often pop off an entry it shouldn't.  git stash doesn't have a
--force argument.  git stash doesn't set a result code, either, that I
can see.  One way around this would be to do

    $ git stash list | wc -l

both before and after the git stash and compare the answers, but really?

So, next idea, feed the output from git status --porcelain through grep
before and after the git pull, so as to find out whether there are any
modified files before the git pull (thus making a stash necessary) and
any files with conflicts after the git stash pop.  Shouldn't be too
difficult.

Except, how does one recognise a file with conflicts from this git
status output?  The man page says that

    "   For paths with merge conflicts, `X' and `Y' show the modification
    states of each side of the merge.  For paths that do not have merge
    conflicts, `X' shows the status of the index, and `Y' shows the status
    of the work tree.  For untracked paths, `XY' are `??'.  Other status
    codes can be interpreted as follows:  ...."

I've spent nearly an hour trying to make sense of this bit of man page.
How is one meant to distinguish an XY of a merge conflict from the XY of
an index/work tree entry?  I can't find that key bit of information
anywhere.

What am I missing?  Why does writing scripts using git have to be so
hard?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Difficulties of scripting git

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 3647 bytes --]

On 2021-01-09 at 19:14:13, Alan Mackenzie wrote:
> Hello, git.
> 
> I want to write a bash script (more precisely, correct an existing
> script) which uses git, and it is DIFFICULT!!!
> 
> I need a git pull in the script.  Fine, but how does the script know if
> it's worked or not?  I couldn't find any description of a return code in
> the git-pull man page (or in the git-merge man page).
> 
> The big problem is when I have modified, uncommitted files in the target
> repo, and those same files are to be updated in commits coming from the
> source repo.  Sadly, git is unable to merge these changes.  It just
> fails, putting an error message onto stderr, but doesn't tell the
> calling script in any way that I can see.
> 
> One idea would be always to call git stash before doing the pull, then
> git stash pop afterwards.  Trouble is, git stash is unreliable - it
> doesn't always add a new stash entry, so the stash pop at the end would
> sometimes/often pop off an entry it shouldn't.  git stash doesn't have a
> --force argument.  git stash doesn't set a result code, either, that I
> can see.  One way around this would be to do
> 
>     $ git stash list | wc -l
> 
> both before and after the git stash and compare the answers, but really?

git stash doesn't consider there being nothing to stash to be an error,
so it doesn't set a nonzero error code.  Think of it as, "I want a clean
working tree," not, "I want to create a stash."  In that sense, what you
wanted is already done, so it's not an error.

You are, however, not the only person who's been unhappy with this
behavior.  A --force option may be a useful option to have if someone
wanted to create it.

> So, next idea, feed the output from git status --porcelain through grep
> before and after the git pull, so as to find out whether there are any
> modified files before the git pull (thus making a stash necessary) and
> any files with conflicts after the git stash pop.  Shouldn't be too
> difficult.

Using "git status --porcelain" before and after the stash is an easy way
to find out whether there was anything to stash.  If there's a
difference, then a stash was created.

> Except, how does one recognise a file with conflicts from this git
> status output?  The man page says that
> 
>     "   For paths with merge conflicts, `X' and `Y' show the modification
>     states of each side of the merge.  For paths that do not have merge
>     conflicts, `X' shows the status of the index, and `Y' shows the status
>     of the work tree.  For untracked paths, `XY' are `??'.  Other status
>     codes can be interpreted as follows:  ...."
> 
> I've spent nearly an hour trying to make sense of this bit of man page.
> How is one meant to distinguish an XY of a merge conflict from the XY of
> an index/work tree entry?  I can't find that key bit of information
> anywhere.

In the chart, the options in the first section are "normal" cases that
occur without a merge conflict.  The second section is things that
happen when there's a conflict (the "unmerged" states).

So what you're looking for to find conflicts is something maybe a little
like this:

  git status --porcelain | grep -E '^(AA|DD|[AUD]U|U[AUD])'

If you believe that you're likely to need to handle names with newlines,
then of course you'll need -z as well.

I do agree that this is a little unclear; I went ahead and simulated a
merge conflict to verify.  I'll try to send a patch making the
documentation reflect what I mentioned about the chart above.
-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

[PATCH] docs: add description of status output table

[brian m. carlson]

The table describing the porcelain format in git-status(1) is helpful,
but it's not completely clear what the three sections mean, even to
some contributors.  As a result, users are unable to find how to detect
common cases like merge conflicts programmatically.

Let's improve this situation by describing what each section means:
non-conflicted, conflicted, or untracked files.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/git-status.txt | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/Documentation/git-status.txt b/Documentation/git-status.txt
index 7731b45f07..63f13c201d 100644
--- a/Documentation/git-status.txt
+++ b/Documentation/git-status.txt
@@ -201,6 +201,10 @@ codes can be interpreted as follows:
 Ignored files are not listed, unless `--ignored` option is in effect,
 in which case `XY` are `!!`.
 
+In the table below, the first section indicates normal non-conflicted states for
+tracked files, the second indicates files where a merge conflict has occurred
+but not yet been resolved, and the third indicates files not tracked by Git.
+
 ....
 X          Y     Meaning
 -------------------------------------------------

Re: [PATCH] docs: add description of status output table

[Junio C Hamano]

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

> The table describing the porcelain format in git-status(1) is helpful,
> but it's not completely clear what the three sections mean, even to
> some contributors.  As a result, users are unable to find how to detect
> common cases like merge conflicts programmatically.

I agree that the addition clarifies, but it is a bit sad that we
already have a beginning of the explanation; I wonder if we should
improve the existing description in addition, even if it may not be
sufficient to eliminate the need for this new paragraph.  Here is
what we already have:

    For paths with merge conflicts, `X` and `Y` show the modification
    states of each side of the merge. For paths that do not have merge
    conflicts, `X` shows the status of the index, and `Y` shows the status
    of the work tree.  For untracked paths, `XY` are `??`.  Other status
    codes can be interpreted as follows:

This introductory text does sort-of hint that there are three
classes (merged paths, unmerged paths and untracked paths), but (1)
the order the three classes are described do not match that of the
table, and (2) the explanation of the untracked paths predates the
addition of ignored ones to the untracked class, so the description
is added after the legends as if an afterthought.

I am actually tempted to suggest rewriting the whole section,
starting from the paragraph above and ending at the table, with
something like this:

    Three different classes of paths are shown in the same format,
    but the meaning of `XY` are different:

    * For merged paths, `X` shows the status of the index, and `Y`
      shows the status of the working tree.

    * For unmerged paths, `X` and `Y` show the modification states
      of each side of the merge, relative to the common ancestor.

    * For untracked paths, `X` and `Y` do not convey different
      meaning (as, by definition, they are not known to the index);
      `??` is shown for untracked paths, and when `--ignored` option
      is in effect, ignored paths are shown with `!!`.

    In the following table, these three classes are shown in
    separate sections, and these characters are used for `X` and `Y`
    fields for the first two sections that show tracked paths:

    * ' ' = unmodified
    * 'M' = modified

       ...

    ....
    X        Y       Meaning
    ------------------------------------------------------------
            [AMD]    not updated

       ...

Hmm?

Re: [PATCH] docs: add description of status output table

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1855 bytes --]

On 2021-01-10 at 01:41:23, Junio C Hamano wrote:
> "brian m. carlson" <sandals@crustytoothpaste.net> writes:
> 
> > The table describing the porcelain format in git-status(1) is helpful,
> > but it's not completely clear what the three sections mean, even to
> > some contributors.  As a result, users are unable to find how to detect
> > common cases like merge conflicts programmatically.
> 
> I agree that the addition clarifies, but it is a bit sad that we
> already have a beginning of the explanation; I wonder if we should
> improve the existing description in addition, even if it may not be
> sufficient to eliminate the need for this new paragraph.  Here is
> what we already have:
> 
>     For paths with merge conflicts, `X` and `Y` show the modification
>     states of each side of the merge. For paths that do not have merge
>     conflicts, `X` shows the status of the index, and `Y` shows the status
>     of the work tree.  For untracked paths, `XY` are `??`.  Other status
>     codes can be interpreted as follows:
> 
> This introductory text does sort-of hint that there are three
> classes (merged paths, unmerged paths and untracked paths), but (1)
> the order the three classes are described do not match that of the
> table, and (2) the explanation of the untracked paths predates the
> addition of ignored ones to the untracked class, so the description
> is added after the legends as if an afterthought.
> 
> I am actually tempted to suggest rewriting the whole section,
> starting from the paragraph above and ending at the table, with
> something like this:

Sure, I can reroll with that.  I noticed that we're using a text diagram
instead of a table, so maybe I can fix that up as well in v2, depending
how the output looks.
-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: [PATCH] docs: add description of status output table

[Alan Mackenzie]

Hello, Junio and Brian.

Thanks for such rapid and helpful responses to my post.

On Sat, Jan 09, 2021 at 17:41:23 -0800, Junio C Hamano wrote:
> "brian m. carlson" <sandals@crustytoothpaste.net> writes:

> > The table describing the porcelain format in git-status(1) is helpful,
> > but it's not completely clear what the three sections mean, even to
> > some contributors.  As a result, users are unable to find how to detect
> > common cases like merge conflicts programmatically.

> I agree that the addition clarifies, but it is a bit sad that we
> already have a beginning of the explanation; I wonder if we should
> improve the existing description in addition, even if it may not be
> sufficient to eliminate the need for this new paragraph.  Here is
> what we already have:

>     For paths with merge conflicts, `X` and `Y` show the modification
>     states of each side of the merge. For paths that do not have merge
>     conflicts, `X` shows the status of the index, and `Y` shows the status
>     of the work tree.  For untracked paths, `XY` are `??`.  Other status
>     codes can be interpreted as follows:

Another problem I tripped over was that word "Other", which is
ambiguous.  For quite some time I read it wrongly as meaning "Other than
the codes already described in this paragraph" rather than its intended
meaning "Other than for untracked paths".  So I was searching through
the man page looking for a description of codes for conflicts.

> This introductory text does sort-of hint that there are three
> classes (merged paths, unmerged paths and untracked paths), but (1)
> the order the three classes are described do not match that of the
> table, and (2) the explanation of the untracked paths predates the
> addition of ignored ones to the untracked class, so the description
> is added after the legends as if an afterthought.

When you amend this documentation, would you please consider using the
term "conflicts" instead of, or as well as "unmerged".  "Conflicts" is
something definite and clear - "unmerged" feels a bit vague and
indefinite, to me at any rate.  Why might a file be "unmerged" still?

> I am actually tempted to suggest rewriting the whole section,
> starting from the paragraph above and ending at the table, with
> something like this:

>     Three different classes of paths are shown in the same format,
>     but the meaning of `XY` are different:

>     * For merged paths, `X` shows the status of the index, and `Y`
>       shows the status of the working tree.

>     * For unmerged paths, `X` and `Y` show the modification states
>       of each side of the merge, relative to the common ancestor.

Also missing from the current doc, I think, is a description of which
"side" of the difference is represented by X, and which by Y.  In my use
case (having conflicts after doing a git stash pop) those "sides" would
be the work tree and the repository.  In other scenarios (say a merge
conflict after git pull) I think they would be something else (though my
head is hurting a bit, here).

>     * For untracked paths, `X` and `Y` do not convey different
>       meaning (as, by definition, they are not known to the index);
>       `??` is shown for untracked paths, and when `--ignored` option
>       is in effect, ignored paths are shown with `!!`.

>     In the following table, these three classes are shown in
>     separate sections, and these characters are used for `X` and `Y`
>     fields for the first two sections that show tracked paths:

>     * ' ' = unmodified
>     * 'M' = modified

>        ...

>     ....
>     X        Y       Meaning
>     ------------------------------------------------------------
>             [AMD]    not updated

>        ...

> Hmm?

I would appreciate such a new formulation.  Thanks!

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [PATCH] docs: add description of status output table

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1918 bytes --]

On 2021-01-10 at 12:28:40, Alan Mackenzie wrote:
> On Sat, Jan 09, 2021 at 17:41:23 -0800, Junio C Hamano wrote:
> > I am actually tempted to suggest rewriting the whole section,
> > starting from the paragraph above and ending at the table, with
> > something like this:
> 
> >     Three different classes of paths are shown in the same format,
> >     but the meaning of `XY` are different:
> 
> >     * For merged paths, `X` shows the status of the index, and `Y`
> >       shows the status of the working tree.
> 
> >     * For unmerged paths, `X` and `Y` show the modification states
> >       of each side of the merge, relative to the common ancestor.
> 
> Also missing from the current doc, I think, is a description of which
> "side" of the difference is represented by X, and which by Y.  In my use
> case (having conflicts after doing a git stash pop) those "sides" would
> be the work tree and the repository.  In other scenarios (say a merge
> conflict after git pull) I think they would be something else (though my
> head is hurting a bit, here).

The two sides are the two heads that are being merged and don't include
the working tree at all.  I think the answer about which side is which
in a merge depends on whether you're merging or rebasing.  Rebases do
things the opposite way of how merges do them.

In your case, you're interested in the fact that there _is_ an
unresolved conflict, which means you'll either get DD, AA, or something
with a U in it.  That will happen regardless of how you do it, with git
pull, git merge, git rebase -m, or git cherry-pick, and it indicates
that the working tree is "broken" (i.e., has conflict markers) and the
index is in a conflicted state.  Since the state of the working tree and
the index are known, we use this field for listing the two heads
instead.
-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

[PATCH v2] docs: rephrase and clarify the git status --short format

[brian m. carlson]

The table describing the porcelain format in git-status(1) is helpful,
but it's not completely clear what the three sections mean, even to
some contributors.  As a result, users are unable to find how to detect
common cases like merge conflicts programmatically.

Let's improve this situation by rephrasing to be more explicit about
what each of the sections in the table means, to tell users in plain
language which cases are occurring, and to describe what "unmerged"
means.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
This uses text from Junio's email, so his sign-off will be required
here.  I assume that won't be a problem, but I can send a v3 if it is.

 Documentation/git-status.txt | 28 ++++++++++++++++++++--------
 1 file changed, 20 insertions(+), 8 deletions(-)

diff --git a/Documentation/git-status.txt b/Documentation/git-status.txt
index 7731b45f07..c0764e850a 100644
--- a/Documentation/git-status.txt
+++ b/Documentation/git-status.txt
@@ -184,11 +184,26 @@ characters, that field will be quoted in the manner of a C string
 literal: surrounded by ASCII double quote (34) characters, and with
 interior special characters backslash-escaped.
 
-For paths with merge conflicts, `X` and `Y` show the modification
-states of each side of the merge. For paths that do not have merge
-conflicts, `X` shows the status of the index, and `Y` shows the status
-of the work tree.  For untracked paths, `XY` are `??`.  Other status
-codes can be interpreted as follows:
+There are three different types of states that are shown using this format, and
+each one uses the `XY` syntax differently:
+
+* When a merge is occurring and the merge was successful, or outside of a merge
+	situation, `X` shows the status of the index and `Y` shows the status of the
+	working tree.
+* When a merge conflict has occurred and has not yet been resolved, `X` and `Y`
+	show the state introduced by each head of the merge, relative to the common
+	ancestor. These paths are said to be _unmerged_.
+* When a path is untracked, `X` and `Y` are always the same, since they are
+	unknown to the index. `??` is used for untracked paths. Ignored files are
+	not listed unless `--ignored` is used; if it is, ignored files are indicated
+	by `!!`.
+
+Note that the term _merge_ here also includes rebases using the default
+`--merge` strategy, cherry-picks, and anything else using the merge machinery.
+
+In the following table, these three classes are shown in separate sections, and
+these characters are used for `X` and `Y` fields for the first two sections that
+show tracked paths:
 
 * ' ' = unmodified
 * 'M' = modified
@@ -198,9 +213,6 @@ codes can be interpreted as follows:
 * 'C' = copied
 * 'U' = updated but unmerged
 
-Ignored files are not listed, unless `--ignored` option is in effect,
-in which case `XY` are `!!`.
-
 ....
 X          Y     Meaning
 -------------------------------------------------

Re: [PATCH v2] docs: rephrase and clarify the git status --short format

[Junio C Hamano]

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

> The table describing the porcelain format in git-status(1) is helpful,
> but it's not completely clear what the three sections mean, even to
> some contributors.  As a result, users are unable to find how to detect
> common cases like merge conflicts programmatically.
>
> Let's improve this situation by rephrasing to be more explicit about
> what each of the sections in the table means, to tell users in plain
> language which cases are occurring, and to describe what "unmerged"
> means.
>
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
> This uses text from Junio's email, so his sign-off will be required
> here.  I assume that won't be a problem, but I can send a v3 if it is.

Actually I find the text so vastly improved from "here is my
attempt" version in the discussion that, other than the paragraph
structure, there is nothing left that I can call my contribution.
I'll sign the resulting commit off anyway, though ;-)

> +There are three different types of states that are shown using this format, and
> +each one uses the `XY` syntax differently:
> ...
> +
> +Note that the term _merge_ here also includes rebases using the default
> +`--merge` strategy, cherry-picks, and anything else using the merge machinery.

Even if rebase uses the good-old "format-patch | am -3" pipeline, it
would result in an index with entries at higher stages.  So I am not
sure if this "Note that" helps the reader.

> +In the following table, these three classes are shown in separate sections, and

This iteration has improved the "Three different classes of paths
are shown" in the "here is my attempt" version to "Three different
types of states ..."; shouldn't we be doing the same here with
s/classes/types of states/?

> +these characters are used for `X` and `Y` fields for the first two sections that
> +show tracked paths:

Thanks.

Re: [PATCH v2] docs: rephrase and clarify the git status --short format

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1762 bytes --]

On 2021-01-11 at 20:22:10, Junio C Hamano wrote:
> "brian m. carlson" <sandals@crustytoothpaste.net> writes:
> 
> > This uses text from Junio's email, so his sign-off will be required
> > here.  I assume that won't be a problem, but I can send a v3 if it is.
> 
> Actually I find the text so vastly improved from "here is my
> attempt" version in the discussion that, other than the paragraph
> structure, there is nothing left that I can call my contribution.
> I'll sign the resulting commit off anyway, though ;-)

Well, I did use some of your text verbatim, though, and the sign-off is
necessary.  Not that it's easily distinguishable in our history since
you're the maintainer, but I also like to be meticulous about sign-offs
because I think it makes sure that people get due credit for their
contributions.

> > +There are three different types of states that are shown using this format, and
> > +each one uses the `XY` syntax differently:
> > ...
> > +
> > +Note that the term _merge_ here also includes rebases using the default
> > +`--merge` strategy, cherry-picks, and anything else using the merge machinery.
> 
> Even if rebase uses the good-old "format-patch | am -3" pipeline, it
> would result in an index with entries at higher stages.  So I am not
> sure if this "Note that" helps the reader.

I'll rephrase.

> > +In the following table, these three classes are shown in separate sections, and
> 
> This iteration has improved the "Three different classes of paths
> are shown" in the "here is my attempt" version to "Three different
> types of states ..."; shouldn't we be doing the same here with
> s/classes/types of states/?

Can do.
-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]