File extension conflict when working with git and latex

File extension conflict when working with git and latex

[Matthias Beyer]

[-- Attachment #1: Type: text/plain, Size: 1196 bytes --]

Hi,

I struggled with a really nasty issue today (and yesterday):

I work on a semester project paper which is written in latex and
therefor version-controlled with git. We compile the document using
pdflatex, we automate the compiling with make.

pdflatex generates some temporary files, which are required if you
call it the second and third time (it is a top-down compiler, which
needs to be called several times). One of the file extensions is
".idx", which (kind of) conflicts with file extensions in .git/.

Unfortunetely, we wrote our `make clean` task recursively. I think you
can imagine what went wrong: The clean-task corrupted the repository,
as it removed .idx files from within .git/.

I lost work because of this ugly name collision.

My question now:

I know, I can fix this by fixing the clean task in my Makefile. But
maybe someone somewhere on this world doesn't know the git internals
as good as me (and, of course, my coworker). Is there _any chance
at all_ that this gets mentioned somewhere, so others don't fall into
this pit?

-- 
Mit freundlichen Grüßen,
Kind regards,
Matthias Beyer

Proudly sent with mutt.
Happily signed with gnupg.

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: File extension conflict when working with git and latex

[Simon Ruderich]

On Fri, Mar 21, 2014 at 05:13:24PM +0100, Matthias Beyer wrote:
> Unfortunetely, we wrote our `make clean` task recursively. I think you
> can imagine what went wrong: The clean-task corrupted the repository,
> as it removed .idx files from within .git/.
>
> I lost work because of this ugly name collision.

Hello Matthias,

You can recreate the .idx files by running

    git index-pack .git/objects/pack/pack-<hash>.pack

for each pack file.

Regards
Simon
-- 
+ privacy is necessary
+ using gnupg http://gnupg.org
+ public key id: 0x92FEFDB7E44C32F9

Re: File extension conflict when working with git and latex

[Matthias Beyer]

[-- Attachment #1: Type: text/plain, Size: 910 bytes --]

On 21-03-2014 17:36:28, Simon Ruderich wrote:
> On Fri, Mar 21, 2014 at 05:13:24PM +0100, Matthias Beyer wrote:
> > Unfortunetely, we wrote our `make clean` task recursively. I think you
> > can imagine what went wrong: The clean-task corrupted the repository,
> > as it removed .idx files from within .git/.
> >
> > I lost work because of this ugly name collision.
>
> Hello Matthias,
>
> You can recreate the .idx files by running
>
>     git index-pack .git/objects/pack/pack-<hash>.pack
>
> for each pack file.

Hi Simon,

I think so. I executed:

    git fsck # reports N missing blobs, commits, trees and dangling stuff
    git index-pack ...
    git fsck # reports only dangling commits and blobs

I don't know if this means that the repository is fixed now?

-- 
Mit freundlichen Grüßen,
Kind regards,
Matthias Beyer

Proudly sent with mutt.
Happily signed with gnupg.

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: File extension conflict when working with git and latex

[Simon Ruderich]

On Fri, Mar 21, 2014 at 05:46:51PM +0100, Matthias Beyer wrote:
> Hi Simon,
>
> I think so. I executed:
>
>     git fsck # reports N missing blobs, commits, trees and dangling stuff
>     git index-pack ...
>     git fsck # reports only dangling commits and blobs
>
> I don't know if this means that the repository is fixed now?

Hello Matthias,

If nothing else was deleted, then yes, the repository should be
fine now.

Regards
Simon
-- 
+ privacy is necessary
+ using gnupg http://gnupg.org
+ public key id: 0x92FEFDB7E44C32F9

Re: File extension conflict when working with git and latex

[Junio C Hamano]

Matthias Beyer <mail@beyermatthias.de> writes:

> I know, I can fix this by fixing the clean task in my Makefile. But
> maybe someone somewhere on this world doesn't know the git internals
> as good as me (and, of course, my coworker). Is there _any chance
> at all_ that this gets mentioned somewhere, so others don't fall into
> this pit?

Surely, we are here to please ;-)  All of us want to make sure
newbies do not shoot themselves in the foot.

But the problem is what exactly should be mentioned.  With a fresh
wound with your LaTeX project still in your mind, you may be tempted
to special case ".idx", but other newbies may inflict the same kind
of hurt on themselves with different "find" patterns, e.g.

    $ find . -name '[0-9a-f]*[0-9a-f]' -type f -print | xargs rm -f

when they know their project creates hexadecimal-numbered temoprary
files, or whatever other pattern that match the files they do not
care about, that also happens to match whatever is in $GIT_DIR.  The
only common caution that helps us to make sure "others do not fall
into this pit" is "Files and directories in $GIT_DIR are used to
record your work; do not muck with them unless you know what you are
doing e.g. manually repairing a corrupt repository", but that is a
bit lame, isn't it?

It is tempting to suggest "git clean '*.idx'", but that is a good
fit in the Makefile only when you know everybody involved in the
project works in a checkout from Git, not from a tarball extract,
and does not apply to projects in general.

Re: File extension conflict when working with git and latex

[Michael Haggerty]

On 03/21/2014 07:32 PM, Junio C Hamano wrote:
> Matthias Beyer <mail@beyermatthias.de> writes:
> 
>> I know, I can fix this by fixing the clean task in my Makefile. But
>> maybe someone somewhere on this world doesn't know the git internals
>> as good as me (and, of course, my coworker). Is there _any chance
>> at all_ that this gets mentioned somewhere, so others don't fall into
>> this pit?
> 
> Surely, we are here to please ;-)  All of us want to make sure
> newbies do not shoot themselves in the foot.
> 
> But the problem is what exactly should be mentioned.  With a fresh
> wound with your LaTeX project still in your mind, you may be tempted
> to special case ".idx", but other newbies may inflict the same kind
> of hurt on themselves with different "find" patterns, e.g.
> 
>     $ find . -name '[0-9a-f]*[0-9a-f]' -type f -print | xargs rm -f
> 
> when they know their project creates hexadecimal-numbered temoprary
> files, or whatever other pattern that match the files they do not
> care about, that also happens to match whatever is in $GIT_DIR.  The
> only common caution that helps us to make sure "others do not fall
> into this pit" is "Files and directories in $GIT_DIR are used to
> record your work; do not muck with them unless you know what you are
> doing e.g. manually repairing a corrupt repository", but that is a
> bit lame, isn't it?

It might help some people to show the "find" incantation for skipping
files under .git, which I think would be something like

    find . -name .git -prune -o \( [...] \) -print0 | xargs -0 rm -f

where "[...]" is whatever condition the user would otherwise use.

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu
http://softwareswirl.blogspot.com/

Re: File extension conflict when working with git and latex

[Torsten Bögershausen]

On 2014-03-21 17.13, Matthias Beyer wrote:
> Hi,
> 
> I struggled with a really nasty issue today (and yesterday):
> 
> I work on a semester project paper which is written in latex and
> therefor version-controlled with git. We compile the document using
> pdflatex, we automate the compiling with make.
> 
> pdflatex generates some temporary files, which are required if you
> call it the second and third time (it is a top-down compiler, which
> needs to be called several times). One of the file extensions is
> ".idx", which (kind of) conflicts with file extensions in .git/.
> 
> Unfortunetely, we wrote our `make clean` task recursively. I think you
> can imagine what went wrong: The clean-task corrupted the repository,
> as it removed .idx files from within .git/.
> 
> I lost work because of this ugly name collision.
> 
> My question now:
> 
> I know, I can fix this by fixing the clean task in my Makefile. But
> maybe someone somewhere on this world doesn't know the git internals
> as good as me (and, of course, my coworker). Is there _any chance
> at all_ that this gets mentioned somewhere, so others don't fall into
> this pit?
> 
The idx files are mentioned here:
http://www.git-scm.com/book/en/Git-Internals-Packfiles

(Otherwise, some random thoughts:)
- stay out of the .git directory, whatever you do
   (I lost some data because my faulty scripts using sed to replace
   "s/this/that/" and went into the .git directory) 
- Don't delete "read-only" files
   (The .idx files are all  "r--r--r--")
  Or in other words, do not use "rf -f" in a script, unless you know exactly
  what you and the script are doing.
- Replace "rm -f" whith "mv /tmp" could be an option
- As mentioned before, "git clean" can be an option,
  I used it once when the "make clean", "make distclean", "make reallydistclean"
  did not work as expected by me,
  some auto-generated config files where still not deleted. 
- Make backups of your repo (frequently)
- push the repo to a backup repo (frequently)
- Use $GIT_DIR to move the repo out of your work tree:
  https://www.kernel.org/pub/software/scm/git/docs/git.html

- Prepare a patch that renames the *.idx files used by Git into
  *.gitidx, *.gidx, *.packidx, (and the *.pack into *.packdata ??) 

Hopefully you did not loose too much data, and if you have an idea how to improve
things, you may consider sending a patch, improvements are possible and welcome.