Embedded Linux development with GIT

Embedded Linux development with GIT

[Sean Kelley]

Hi,

I have a situation where we have a local GIT repository that is based
on v2.6.17.  We initially added the source tarball to an empty
repository and then started applying changes to it.  Looking back,
that might not have been the best idea 400 commits later.

My goal is to eventually bring our repository closer to mainline
revisions so as to make it easier to actually contribute back to the
community.  So how can I fix my repository so as to give it visibility
to Linus' kernel?

Here is my initial thoughts:

1) Clone kernel.org kernel and it is Master
2) Create a local Head based on 2.6.17 and call it Local
3) Pull my existing heavily patched repository into the Local branch and merge

Is it possible then to see our 400 odd commits then in the Local
branch on top of 2.6.17 so that we can see not only our history but
also the history that came before?  Then as Master advances we can see
about backporting and bringing our code close enough to mainline
kernel to actually be able to contribute back to the community and
submit patches.  Is this realistic approach.  I am unsure of the GIT
commands that I need to do this?

Thanks,

Sean

Re: Embedded Linux development with GIT

[Dan Chokola]

On 7/5/07, Sean Kelley <svk.sweng@gmail.com> wrote:
> Hi,
>
> I have a situation where we have a local GIT repository that is based
> on v2.6.17.  We initially added the source tarball to an empty
> repository and then started applying changes to it.  Looking back,
> that might not have been the best idea 400 commits later.
>
> My goal is to eventually bring our repository closer to mainline
> revisions so as to make it easier to actually contribute back to the
> community.  So how can I fix my repository so as to give it visibility
> to Linus' kernel?
>
> Here is my initial thoughts:
>
> 1) Clone kernel.org kernel and it is Master
> 2) Create a local Head based on 2.6.17 and call it Local
> 3) Pull my existing heavily patched repository into the Local branch and merge
>
> Is it possible then to see our 400 odd commits then in the Local
> branch on top of 2.6.17 so that we can see not only our history but
> also the history that came before?  Then as Master advances we can see
> about backporting and bringing our code close enough to mainline
> kernel to actually be able to contribute back to the community and
> submit patches.  Is this realistic approach.  I am unsure of the GIT
> commands that I need to do this?
>
> Thanks,
>
> Sean

If I understand correctly, what you might want to do is keep your
current master branch, pull in the mainline kernel to a separate
branch, and rebase your master on the mainline branch.

git checkout master
git rebase mainline
will find the last common commit between two branches, then plop all
the mainline commits on top, then proceed to merge in your changes on
the master branch, one-by-one, stopping if there are any conflicts and
allowing you to resolve as they happen. Then your 400 commits will be
on top and you can rebase whenever and still see them on top.

-Dan "Puzzles" Chokola

Re: Embedded Linux development with GIT

[Alex Riesen]

On 7/5/07, Sean Kelley <svk.sweng@gmail.com> wrote:
> 1) Clone kernel.org kernel and it is Master
> 2) Create a local Head based on 2.6.17 and call it Local
> 3) Pull my existing heavily patched repository into the Local branch and merge

Better:

  $ (cd $OLDREPO && git format-patch --stdout -k first..) | git am -k

or, if it is in the same repo, assuming it starts with sha1 "old-beginning"

  $ git format-patch --stdout -k old-beginning.. | git am -k

Merging of unrelated histories is slow (but works).

> Is it possible then to see our 400 odd commits then in the Local
> branch on top of 2.6.17 so that we can see not only our history but
> also the history that came before?  Then as Master advances we can see
> about backporting and bringing our code close enough to mainline
> kernel to actually be able to contribute back to the community and
> submit patches.  Is this realistic approach.

Yes. Look at how OLPC (www.laptop.org) does this with their kernel.

>  I am unsure of the GIT
> commands that I need to do this?

aside from the already mentioned git format-patch and git am:
git apply, git log, git show, gitk, and come to think of it - all of the
rest too, except maybe for importing programs.

Re: Embedded Linux development with GIT

[Johannes Sixt]

Sean Kelley wrote:
> 
> Hi,
> 
> I have a situation where we have a local GIT repository that is based
> on v2.6.17.  We initially added the source tarball to an empty
> repository and then started applying changes to it.  Looking back,
> that might not have been the best idea 400 commits later.
> 
> My goal is to eventually bring our repository closer to mainline
> revisions so as to make it easier to actually contribute back to the
> community.  So how can I fix my repository so as to give it visibility
> to Linus' kernel?
> 
> Here is my initial thoughts:
> 
> 1) Clone kernel.org kernel and it is Master
> 2) Create a local Head based on 2.6.17 and call it Local
> 3) Pull my existing heavily patched repository into the Local branch and merge
> 
> Is it possible then to see our 400 odd commits then in the Local
> branch on top of 2.6.17 so that we can see not only our history but
> also the history that came before?  Then as Master advances we can see
> about backporting and bringing our code close enough to mainline
> kernel to actually be able to contribute back to the community and
> submit patches.  Is this realistic approach.  I am unsure of the GIT
> commands that I need to do this?

That is possible using a graft:

  $ echo "$x $(git rev-parse v2.6.17^0)" >> .git/info/grafts

where $x is the SHA1 of the first commit you made on top of the imported
tarball. This way you have spliced your history with Linus's history.
(This is a strictly _local_ matter! Every clone of your history must
repeat the game!)

Now, Linus will not be able to pull from your faked history because he
doesn't know about the graft. In order to fix that, you can run
git-filter-branch from current git's master branch to rewrite your
history:

  $ git filter-branch new-master v2.6.17..master

Read the man page of git-filter-branch, and understand the implications
before you publish the result.

-- Hannes

Re: Embedded Linux development with GIT

[Johannes Schindelin]

Hi,

On Thu, 5 Jul 2007, Johannes Sixt wrote:

> Sean Kelley wrote:
> > 
> > I have a situation where we have a local GIT repository that is based
> > on v2.6.17.  We initially added the source tarball to an empty
> > repository and then started applying changes to it.  Looking back,
> > that might not have been the best idea 400 commits later.
> 
> That is possible using a graft:
> 
>   $ echo "$x $(git rev-parse v2.6.17^0)" >> .git/info/grafts
> 
> where $x is the SHA1 of the first commit you made on top of the imported
> tarball.

Yes, this is also how I would do it.

> This way you have spliced your history with Linus's history. (This is a 
> strictly _local_ matter! Every clone of your history must repeat the 
> game!)

Now, here I disagree slightly.  If you merge just once, subsequent merges 
will be possible even without that graft.

So if you merge with some newer Linux version, all your cloners get the 
benefit.

> Now, Linus will not be able to pull from your faked history because he
> doesn't know about the graft.

Except if you merge with a more recent version of Linux.

However, I doubt that such a distant (in terms of time!) merge would 
appeal to Linus.  I guess you have to rebase on top of Linus' version 
_anyway_.

> In order to fix that, you can run git-filter-branch from current git's 
> master branch to rewrite your history:
> 
>   $ git filter-branch new-master v2.6.17..master
> 
> Read the man page of git-filter-branch, and understand the implications
> before you publish the result.

This is a way to fix your history, yes.  Note that filter-branch is not 
yet in an official release of Git, and so you either have to wait for 
1.5.3, or you get filter-branch from git.git's "next" branch (just picking 
this one script should work fine, if you have at least 1.5.1 installed).

Note that this rewrites the history, so all the disadvantages of 
rebasing with pulling apply here, too.

But as stated above, I think you have to rebase eventually anyway, if you 
go for inclusion in Linus' tree.  In that case, the filter-branch is 
unnecessary.

Ciao,
Dscho

Re: Embedded Linux development with GIT

[linux]

> Is it possible then to see our 400 odd commits then in the Local
> branch on top of 2.6.17 so that we can see not only our history but
> also the history that came before?  Then as Master advances we can see
> about backporting and bringing our code close enough to mainline
> kernel to actually be able to contribute back to the community and
> submit patches.  Is this realistic approach.  I am unsure of the GIT
> commands that I need to do this?

Yes.  There are two ways to do this:

1) Use the grafts feature.  You can put a line in your repository's
   .git/info/grafts which lists your initial commit and the ancestors you
   want it to have.
   E.g. note that Linus's v2.6.17 is 427abfa28afedffadfca9dd8b067eb6d36bac53f, with
   parent ce221982e0bef039d7047b0f667bb414efece5af.  So you can just include
   <your initial checkin> ce221982e0bef039d7047b0f667bb414efece5af
   in git/info/grafts, and boom.

   The advantage is that you don't have to change anything.

   The disadvantge here is that this is NOT automatically propagated
   by pull operations (it would be a security breach if it were), so
   everyone pulling from you has to do the same thing manually.
   (If you're only sending out patches, it doesn't matter as much.)

2) Rebase your changes.

   Import a standard Linux kernel into the repository, and rebase your
   changes from your original root import onto Linus' 2.6.17.

   The advantage here is that the commit IDs will be preserved when you
   share your changes, so if you have your changes pulled into the kernel
   mainline, it'll be easier to merge them and for you to merge back.

   The disadvantage is that everyone who has cloned from you has to manually
   re-fetch once.

Basically, option 1 has less immediate pain, but has an ongoing cost.
Option 2 gets it out of the way all at once.