git fast-export/fast-import *facepalm*

git fast-export/fast-import *facepalm*

[Chris Packham]

Hi,

I've had a bit of a brain melt and have done something silly with git
fast-import and was hoping someone would be able to get me out of the
mess I've created.

Basically I've got a linux repository which for various reasons isn't
based of any of the kernel.org ones but reflects our kernel
development on top of vanilla kernel tarballs. When we update we take
the tarball and extract and commit it to a vendor branch then merge
that with our main. A little strange but it works for us (despite
having to go hunting on kernel.org when we want the external history).

My brain melt came when I wanted to get the set of changes between 2
versions from a public branch and import them into our repository.
Having just learned about git fast-export I decided that it was the
right tool for the job so I did the following

(cd linux-2.6.32.y; git fast-export v2.6.32.12..v2.6.32.14) | git fast-import

What I've ended up with is a repository with a detached set of changes i.e

o -o   l - l - l - l - l - l     o - master
     \                          /
      o - o - o - o - o - o

o = our commits
l = linux commits

Because the code is common textually I think what I really should have done is

(cd linux-2.6.32.y; git format-patch v2.6.32.12..v2.6.32.14) | git am

Which I'll give a try in a minute. In the meantime is there anyway for
me to safely remove the upstream linux commits without loosing our
commits in the process?

Thanks,
Chris

Re: git fast-export/fast-import *facepalm*

[Shawn O. Pearce]

Chris Packham <judge.packham@gmail.com> wrote:
> What I've ended up with is a repository with a detached set of changes i.e
> 
> o -o   l - l - l - l - l - l     o - master
>      \                          /
>       o - o - o - o - o - o
> 
> o = our commits
> l = linux commits
> 
> Because the code is common textually I think what I really should have done is
> 
> (cd linux-2.6.32.y; git format-patch v2.6.32.12..v2.6.32.14) | git am
> 
> Which I'll give a try in a minute. In the meantime is there anyway for
> me to safely remove the upstream linux commits without loosing our
> commits in the process?

If I read your diagram right, the l-l-l chain isn't connected at
all to your graph, so it should just get removed with `git gc`.

But if it is connected due to a merge with your master, lookup the
merge and find its parent which is your local stuff and `git reset
--hard` to that commit.

-- 
Shawn.

Re: git fast-export/fast-import *facepalm*

[Chris Packham]

On Thu, May 27, 2010 at 2:03 PM, Shawn O. Pearce <spearce@spearce.org> wrote:
> Chris Packham <judge.packham@gmail.com> wrote:
>> What I've ended up with is a repository with a detached set of changes i.e
>>
>> o -o   l - l - l - l - l - l     o - master
>>      \                          /
>>       o - o - o - o - o - o
>>
>> o = our commits
>> l = linux commits
>>
>> Because the code is common textually I think what I really should have done is
>>
>> (cd linux-2.6.32.y; git format-patch v2.6.32.12..v2.6.32.14) | git am
>>
>> Which I'll give a try in a minute. In the meantime is there anyway for
>> me to safely remove the upstream linux commits without loosing our
>> commits in the process?
>
> If I read your diagram right, the l-l-l chain isn't connected at
> all to your graph, so it should just get removed with `git gc`.
>

Well it _looks_ detached in gitk I can't see any merge commits. I've
tried git gc but no joy. Maybe I need to tell it to be a bit more
thorough.

> But if it is connected due to a merge with your master, lookup the
> merge and find its parent which is your local stuff and `git reset
> --hard` to that commit.
>
> --
> Shawn.
>

Re: git fast-export/fast-import *facepalm*

[Avery Pennarun]

On Thu, May 27, 2010 at 5:57 PM, Chris Packham <judge.packham@gmail.com> wrote:
> Well it _looks_ detached in gitk I can't see any merge commits. I've
> tried git gc but no joy. Maybe I need to tell it to be a bit more
> thorough.

Why is it showing up in gitk at all?  Does it have its own branch
name?  gitk will normally only show commits that are attached to
something, unless you list a particular commit id explicitly on its
command line.

Once the commits are "detached" correctly, saving the disk space is
kind of unnecessary.  (Most likely the old commits are being held in
the reflog at least, which usually expires in 30 days or so for
safety.)  Moreover, most of the space is probabably space occupied by
the actual files in the revisions, which will be needed when you port
the patches "correctly" anyhow, so there's no point in going through
extra effort to recover the space just so you can use it again.

Have fun,

Avery

Re: git fast-export/fast-import *facepalm*

[Chris Packham]

On Thu, May 27, 2010 at 3:11 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
> On Thu, May 27, 2010 at 6:06 PM, Chris Packham <judge.packham@gmail.com> wrote:
>> They're showing up with 'gitk --all'. I've made a backup so I can
>> expire the reflog and see if they go away.
>
> gitk --all shouldn't be showing stuff just because it's in the reflog.
>  Does it have a branchname tag next to it?  Have you exit/restarted
> gitk or are you just using its refresh button?  The latter doesn't
> work very well across branch deletion.
>
> Avery
>

The commits do have tags. I'll try deleting them and gc'ing again.

Re: git fast-export/fast-import *facepalm*

[Chris Packham]

On Thu, May 27, 2010 at 3:18 PM, Chris Packham <judge.packham@gmail.com> wrote:
> On Thu, May 27, 2010 at 3:11 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
>> On Thu, May 27, 2010 at 6:06 PM, Chris Packham <judge.packham@gmail.com> wrote:
>>> They're showing up with 'gitk --all'. I've made a backup so I can
>>> expire the reflog and see if they go away.
>>
>> gitk --all shouldn't be showing stuff just because it's in the reflog.
>>  Does it have a branchname tag next to it?  Have you exit/restarted
>> gitk or are you just using its refresh button?  The latter doesn't
>> work very well across branch deletion.
>>
>> Avery
>>
>
> The commits do have tags. I'll try deleting them and gc'ing again.
>

Deleting the tags seemed to do the trick as far as removing the
detached commits in gitk.

Now git gc seems to be stuck

~/linux> git --version
git version 1.7.1

~/linux> git gc
Counting objects: 20058 (went away made a coffee, came back still
stuck at 20058)

I've attached with gdb and have the following trace if its of any use
(sorry I don't have debug symbols for libz). I've still got the
debugger attached if anyone wants me to dump out anything else.

[Switching to Thread 0xb759e6c0 (LWP 7731)]
0xb7898685 in ?? () from /lib/libz.so.1

(gdb) bt
#0  0xb7898685 in ?? () from /lib/libz.so.1
#1  0xb7898f99 in ?? () from /lib/libz.so.1
#2  0xb7895ccc in ?? () from /lib/libz.so.1
#3  0xb7895ff8 in deflate () from /lib/libz.so.1
#4  0x080f5ade in write_loose_object (sha1=<value optimized out>,
hdr=<value optimized out>, hdrlen=<value optimized out>,
buf=0x8248570, len=1170, mtime=1274994118) at sha1_file.c:2314
#5  0x080f6c09 in force_object_loose (sha1=0xb64ed204
"»²D\v\030\005ªn\fØÂ\026\201\037EªÞãc{²JõØ`º\214\020IvæÐ`Æ0\2113\222ÿ»²R\rdZ;#q\f?\001ü\201#ZXr·»²U¯Jl²\to\022¬³\004\035\215Õd0|z»²[çÝþaD2Ê\206'ºL¾y\037WÕi»²^©§Ê\024H\025\003\200\006?\003L=\001¯?²dòäÒ\223Úc\234\202\224ÍfcpOì8?l\0226\021~óxTç¼y.iN¾¡\036»²\212·Y\227\211t\v\"3
\200]\"®û\227õ3»²¢\024Té«>\212.¡a@b'Ü[Ò=4"..., mtime=1274994118) at
sha1_file.c:2377
#6  0x08089d8d in get_object_list (ac=<value optimized out>, av=<value
optimized out>) at builtin/pack-objects.c:2050
#7  0x0808c4e6 in cmd_pack_objects (argc=10, argv=0xbfb43118,
prefix=0x0) at builtin/pack-objects.c:2317
#8  0x0804b42b in handle_internal_command (argc=<value optimized out>,
argv=<value optimized out>) at git.c:260
#9  0x0804b5fe in main (argc=10, argv=0xbfb43118) at git.c:458

Re: git fast-export/fast-import *facepalm*

[Chris Packham]

On Thu, May 27, 2010 at 3:57 PM, Chris Packham <judge.packham@gmail.com> wrote:
> On Thu, May 27, 2010 at 3:18 PM, Chris Packham <judge.packham@gmail.com> wrote:
>> On Thu, May 27, 2010 at 3:11 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
>>> On Thu, May 27, 2010 at 6:06 PM, Chris Packham <judge.packham@gmail.com> wrote:
>>>> They're showing up with 'gitk --all'. I've made a backup so I can
>>>> expire the reflog and see if they go away.
>>>
>>> gitk --all shouldn't be showing stuff just because it's in the reflog.
>>>  Does it have a branchname tag next to it?  Have you exit/restarted
>>> gitk or are you just using its refresh button?  The latter doesn't
>>> work very well across branch deletion.
>>>
>>> Avery
>>>
>>
>> The commits do have tags. I'll try deleting them and gc'ing again.
>>
>
> Deleting the tags seemed to do the trick as far as removing the
> detached commits in gitk.
>
> Now git gc seems to be stuck
>
> ~/linux> git --version
> git version 1.7.1
>
> ~/linux> git gc
> Counting objects: 20058 (went away made a coffee, came back still
> stuck at 20058)
>
> I've attached with gdb and have the following trace if its of any use
> (sorry I don't have debug symbols for libz). I've still got the
> debugger attached if anyone wants me to dump out anything else.
>

Looks like its come back now. I guess there was just one object that
was taking a while to pack.

Re: git fast-export/fast-import *facepalm*

[Avery Pennarun]

On Thu, May 27, 2010 at 4:46 PM, Chris Packham <judge.packham@gmail.com> wrote:
> My brain melt came when I wanted to get the set of changes between 2
> versions from a public branch and import them into our repository.
> Having just learned about git fast-export I decided that it was the
> right tool for the job so I did the following
>
> (cd linux-2.6.32.y; git fast-export v2.6.32.12..v2.6.32.14) | git fast-import

Personally what I would do is use git rebase instead:

# WARNING: untested.  back up your repo first!
cd my-repo
git fetch ../linux-2.6.32.y v2.6.32.14
git branch b FETCH_HEAD
git fetch ../linux-2.6.32.y v2.6.32.12
git branch a FETCH_HEAD
git checkout b
git rebase --onto master a
# now your branch b should be master + the patches from a..b

> Which I'll give a try in a minute. In the meantime is there anyway for
> me to safely remove the upstream linux commits without loosing our
> commits in the process?

Looks like Shawn answered this question.  In any case, the answer is git-reset.

Have fun,

Avery