[suggestion] Fail cherry-pick if it overwrites untracked files with the matching file names

[suggestion] Fail cherry-pick if it overwrites untracked files with the matching file names

[Rupinderjeet Singh]

Hello there,

I have filled a bug-report and it is available at
https://github.com/rupinderjeet/git-cherry-pick-removes-untracked-files

Since your email only receives plain-text, I assume I can't send the
report by uploading it in this email. If there's something I can do
from my end, or if you need some more information, please let me know.

From,
Rupinderjeet Singh Hans.

Re: [suggestion] Fail cherry-pick if it overwrites untracked files with the matching file names

[Junio C Hamano]

Rupinderjeet Singh <rupinderjeet47@gmail.com> writes:

> Since your email only receives plain-text, I assume I can't send the
> report by uploading it in this email. If there's something I can do
> from my end, or if you need some more information, please let me know.

You do not have to assume that.

"git bugreport" gives a template to create a text file, which you
can paste/include to the body of your e-mail message without making
it an attachment.

Re: [suggestion] Fail cherry-pick if it overwrites untracked files with the matching file names

[Junio C Hamano]

If the situation is about cherry-picking a commit that adds a new
file F to a checkout of another commit that lacks the file F, I
think the command is working exactly as designed.

    $ git init
    $ git commit -m 'initial' --allow-empty
    $ git tag initial
    $ date >file
    $ git add file
    $ git commit -m 'add file'
    $ git tag added
    $ git checkout -b second initial
    ... at this point we are back to the original state
    ... without 'file'
    $ >file
    ... the file is untracked with precious contents
    ... and the presence of it stops a cherry-pick that clobbers it
    $ git cherry-pick added
    error: The following untracked working tree files would be overwritten by merge:
            file
    Please move or remove them before you merge.
    Aborting
    fatal: cherry-pick failed

Now continuing from the above, things get (slightly) interesting

    $ echo file >.gitignore
    $ git cherry-pick added

This will replace "file" with the one from the "added" commit, and
that is because the user marked that the "file" in the working tree
is expendable.

Files in a working tree controlled by git fall into one of three
classes.  Tracked files are those that are known to the index and
appear in "git ls-files" output.  Among the others, ignored files
are those that .gitignore mechanism says are expendable.  The rest
are "untracked", possibly containing valuable contents that should
not be lost as the user may choose to 'git add' them later.

Not just cherry-pick but any merge-related operations, including
"checkout", follow this semantics.  Untracked files are kept, but
ignored files are expendable and will be removed if they are in the
way to complete the operation the user asks.

    $ rm .gitignore
    $ git checkout master
    error: The following untracked working tree files would be overwritten by checkout:
            file
    Please move or remove them before you switch branches.
    Aborting

    $ echo file >.gitignore
    $ git checkout master
    ... this should succeed, removing "file" whose contents were
    ... marked expendable.

Of course, after switching to 'master' (or cherry-picking 'added'),
the project now cares about the 'file'.  After all, it bothered to
add it to keep track of the changes to its contents.  So it is
recommended that you would adjust the contents of .gitignore so that
it no longer is marked as expendable.

Re: [suggestion] Fail cherry-pick if it overwrites untracked files with the matching file names

[Rupinderjeet Singh]

Yes, thank you for being so through with the example. I understand now.

I want to ask if you would suggest that local configuration (or any
information required by the project) that is too sensitive to be
tracked in git, should either be kept as 'untracked' files or 'outside
of git repository'.

Since, any merge or similar action can do this, it should be
considered risky to keep files with important information in
.gitignore file.

As an example, when I am working on an android project, I usually add
firebase-integration info and keystore details in the .gitignore,
because I don't want to add them to version control (even by mistake).
I have seen it as a good practice suggested by many devs around me,
codelabs, online courses, and from experienced developers at google
too.

With your explanation, am I correct to think that only the following
kind of information is suitable to be put in .gitignore files?
1. that can be regenerated
2. that doesn't matter when it is lost
3. that isn't used by the files tracked in git repository

Honestly, the name .gitignore implies that git will completely ignore
these files. Overwriting an ignored file does seem like a violation of
the rule present in .gitignore file. But, if the implementation is
intended as you describe, I have definitely learned something new.

Thank you for your answer. My bad for not including the report-file as
text, it slipped my mind.

From,
Rupinderjeet Singh Hans.


On Sun, 16 Oct 2022 at 00:05, Junio C Hamano <gitster@pobox.com> wrote:
>
> If the situation is about cherry-picking a commit that adds a new
> file F to a checkout of another commit that lacks the file F, I
> think the command is working exactly as designed.
>
>     $ git init
>     $ git commit -m 'initial' --allow-empty
>     $ git tag initial
>     $ date >file
>     $ git add file
>     $ git commit -m 'add file'
>     $ git tag added
>     $ git checkout -b second initial
>     ... at this point we are back to the original state
>     ... without 'file'
>     $ >file
>     ... the file is untracked with precious contents
>     ... and the presence of it stops a cherry-pick that clobbers it
>     $ git cherry-pick added
>     error: The following untracked working tree files would be overwritten by merge:
>             file
>     Please move or remove them before you merge.
>     Aborting
>     fatal: cherry-pick failed
>
> Now continuing from the above, things get (slightly) interesting
>
>     $ echo file >.gitignore
>     $ git cherry-pick added
>
> This will replace "file" with the one from the "added" commit, and
> that is because the user marked that the "file" in the working tree
> is expendable.
>
> Files in a working tree controlled by git fall into one of three
> classes.  Tracked files are those that are known to the index and
> appear in "git ls-files" output.  Among the others, ignored files
> are those that .gitignore mechanism says are expendable.  The rest
> are "untracked", possibly containing valuable contents that should
> not be lost as the user may choose to 'git add' them later.
>
> Not just cherry-pick but any merge-related operations, including
> "checkout", follow this semantics.  Untracked files are kept, but
> ignored files are expendable and will be removed if they are in the
> way to complete the operation the user asks.
>
>     $ rm .gitignore
>     $ git checkout master
>     error: The following untracked working tree files would be overwritten by checkout:
>             file
>     Please move or remove them before you switch branches.
>     Aborting
>
>     $ echo file >.gitignore
>     $ git checkout master
>     ... this should succeed, removing "file" whose contents were
>     ... marked expendable.
>
> Of course, after switching to 'master' (or cherry-picking 'added'),
> the project now cares about the 'file'.  After all, it bothered to
> add it to keep track of the changes to its contents.  So it is
> recommended that you would adjust the contents of .gitignore so that
> it no longer is marked as expendable.

Re: [suggestion] Fail cherry-pick if it overwrites untracked files with the matching file names

[Junio C Hamano]

Rupinderjeet Singh <rupinderjeet47@gmail.com> writes:

> Yes, thank you for being so through with the example. I understand now.
>
> I want to ask if you would suggest that local configuration (or any
> information required by the project) that is too sensitive to be
> tracked in git, should either be kept as 'untracked' files or 'outside
> of git repository'.

I think it depends on the project and participants may not have
control over it if it is ingrained in the project's build structure,
but a separate place may likely be more appropriate, as it would
reduce the chance of accidental "git add ." adding everything.

> With your explanation, am I correct to think that only the following
> kind of information is suitable to be put in .gitignore files?
> 1. that can be regenerated

Yes.

> 2. that doesn't matter when it is lost

Natural consequence of the above

> 3. that isn't used by the files tracked in git repository

I do not get this, so I decline to comment ;) Is mylib.o that is
"ignored", created from tracked mylib.c source, used by mymain.c
source that is tracked, when mymain.o would not link without what
mylib.o gives it?

Some other SCMs have an extra class "precious" to handle exactly the
case you have in mind.  You do not accidentally let "git add ." to
slurp them into the committed history, but you do want them to be
left alone.  We don't have it, and that is not because we have any
reason to be against having it.  It is just it didn't happen.

And nobody bothered to explore the ramification of adding the new
class yet.  We know about "checkout" and other mergy operations and
"add", but there probably are trickier interactions with other parts
of the existing system that somebody need to carefully go through
and make sure it does not introduce funny inconsistencies.

Re: [suggestion] Fail cherry-pick if it overwrites untracked files with the matching file names

[Elijah Newren]

On Sat, Oct 15, 2022 at 12:23 PM Junio C Hamano <gitster@pobox.com> wrote:
[...]
> Files in a working tree controlled by git fall into one of three
> classes.  Tracked files are those that are known to the index and
> appear in "git ls-files" output.  Among the others, ignored files
> are those that .gitignore mechanism says are expendable.  The rest
> are "untracked", possibly containing valuable contents that should
> not be lost as the user may choose to 'git add' them later.
>
> Not just cherry-pick but any merge-related operations, including
> "checkout", follow this semantics.  Untracked files are kept, but
> ignored files are expendable and will be removed if they are in the
> way to complete the operation the user asks.
>
>     $ rm .gitignore
>     $ git checkout master
>     error: The following untracked working tree files would be overwritten by checkout:
>             file
>     Please move or remove them before you switch branches.
>     Aborting
>
>     $ echo file >.gitignore
>     $ git checkout master
>     ... this should succeed, removing "file" whose contents were
>     ... marked expendable.

There is a command line option meant to allow tweaking this behavior:
--[no-]overwrite-ignore.  The default, as Junio explains above, is
--overwrite-ignore.  The --no-overwrite-ignore option works for `git
checkout`.  It also can work for `git merge`, but *only* if the merge
results in a fast-forward.  Although there is code in unpack_trees()
to handle this alternate behavior by just setting a simple flag, we
don't bother setting that flag in most merge paths even when the user
specifies that option.  It was just never hooked up.  And several
other related commands (am, reset, stash, rebase) don't even accept
such a flag even though for consistency they probably should.

I've been meaning to get back to it and hook up this command line flag
for the occasional user that wants this alternative behavior.
However, it's not an issue for me or the direct users I support, and
complaints about this behavior from the community are quite rare too,
so I just haven't been too motivated to work on it.  If someone else
wants to take a stab, it should be pretty easy.  Just grep for
"preserve_ignored.*FIXME".  And maybe read up on these two commits for
context:
    04988c8d18 ("unpack-trees: introduce preserve_ignored to
unpack_trees_options", 2021-09-27)
    1b5f37334a ("Remove ignored files by default when they are in the
way", 2021-09-27)

Maybe I should just tag this as #leftoverbits to see if someone else
picks it up?

Re: [suggestion] Fail cherry-pick if it overwrites untracked files with the matching file names

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> There is a command line option meant to allow tweaking this behavior:
> --[no-]overwrite-ignore.

Ah, I totally forgot that relatively recent invention.  Thanks for
bringing it up.

I personally consider it a bit too blunt an instrument, and in the
longer term, it would probably be a better direction to introduce a
new class of paths that are not tracked, and call the class
"precious", which (1) is similar to "ignored" in that the paths in
the class do not get added with "git add ." but (2) unlike "ignored"
but like "untracked", they are not considered to be expendable.

Until that happens, though, treating all "ignored" paths as if they
are "precious" may be a usable workaround.