[doc][git] playing with git, and netdev/libata-dev trees

[doc][git] playing with git, and netdev/libata-dev trees

[Jeff Garzik]

Hopefully, this email can quick-start some people on git.

One of the things Linus's new 'git' tool allows me to do is make public 
the 50+ repositories that were previously only available on my local 
workstation.  This should make it a lot easier for developers to see 
precisely what I have merged, and makes generating follow-up patches a 
whole lot easier.

When I merge a patch for drivers/net/forcedeth.c, I merge it into a 
brand new 'forcedeth' repository, a peer to the 40+ other such 
repository.  Under BitKeeper, I made these repositories available merged 
together into one big "netdev-2.6" repository because it was too time 
consuming to make the individual 50+ trees publicly available.  With 
git, developers have direct access to the individual trees.

I thought I would write up a quick guide describing how to mess around 
with the netdev and libata-dev trees, and with git in general.


1) installing git

git requires bootstrapping, since you must have git installed in order 
to check out git.git (git repo), and linux-2.6.git (kernel repo).  I 
have put together a bootstrap tarball of today's git repository.

Download tarball from:
http://www.kernel.org/pub/linux/kernel/people/jgarzik/git-20050526.tar.bz2

tarball build-deps:  zlib, libcurl

install tarball:  unpack && make && sudo make prefix=/usr/local install

jgarzik helper scripts, not in official git distribution:
http://www.kernel.org/pub/linux/kernel/people/jgarzik/git-switch-tree
http://www.kernel.org/pub/linux/kernel/people/jgarzik/git-new-branch
http://www.kernel.org/pub/linux/kernel/people/jgarzik/git-changes-script

After reading the rest of this document, come back and update your copy 
of git to the latest:
rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/git.git


2) download a linux kernel tree for the very first time

mkdir -p linux-2.6/.git
cd linux-2.6
rsync -a --delete --verbose --stats --progress \ 
rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git/ 
\          <- word-wrapped backslash; sigh
    .git/


3) download latest changes to on-disk local tree

cd linux-2.6
git-pull-script \
rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git


4) check out files from the git repository into the working directory

cd linux-2.6
git-read-tree -m HEAD && git-checkout-cache -q -f -u -a


5) check in your own modifications (e.g. apply a patch)

# go to repo
cd linux-2.6

# make some modifications
patch -sp1 < /tmp/my.patch
diffstat -p1 < /tmp/my.patch

# NOTE: add '--add' and/or '--remove' if files were added or removed
git-update-cache <list of all files changed>

# commit changes
GIT_AUTHOR_NAME="John Doe"		\
    GIT_AUTHOR_EMAIL="jdoe@foo.com"	\
    GIT_COMMITTER_NAME="Jeff Garzik"	\
    GIT_COMMITTER_EMAIL="jgarzik@pobox.com"	\
    git-commit-tree `git-write-tree`	\
    -p $(cat .git/HEAD )			\
    < changelog.txt			\
    > .git/HEAD


6) List all changes in working dir, in diff format.

git-diff-cache -p HEAD


7) List all changesets (i.e. show each cset's description text) in local 
tree that are not present in remote tree.

cd my-kernel-tree-2.6
git-changes-script -L ../linux-2.6 | less


8) List all changesets:

git-whatchanged


9) apply all patches in a Berkeley mbox-format file

First, download and add to your PATH Linus's git tools:
rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/git-tools.git

cd my-kernel-tree-2.6
dotest /path/to/mbox  # yes, Linus has no taste in naming useful scripts


10) don't forget to download tags from time to time.

git-pull-script only downloads sha1-indexed object data, and the 
requested remote head.  This misses updates to the .git/refs/tags/ and 
.git/refs/heads directories.  It is advisable to update your kernel .git 
directories periodically with a full rsync command, to make sure you got 
everything:

cd linux-2.6
rsync -a --delete --verbose --stats --progress \ 
rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git/ 
\          <- word-wrapped backslash; sigh
    .git/


11) [jg-specific] list all branches found in netdev-2.6 or libata-dev trees.

Download
rsync://rsync.kernel.org/pub/scm/linux/kernel/git/jgarzik/netdev-2.6.git
	or
rsync://rsync.kernel.org/pub/scm/linux/kernel/git/jgarzik/libata-dev.git


cd netdev-2.6
ls .git/refs/heads/

{ these are the current netdev-2.6 branches }
> 8139cp       forcedeth    master     qeth           smc91x         we18
> 8139too-iomap  for-linus    natsemi      r8169      smc91x-eeprom  wifi
> airo           hdlc         ns83820      register-netdev  starfire
> atmel          ieee80211    orinoco      remove-drivers   tlan
> chelsio        iff-running  orinoco-hch  sis900           veth
> dm9000         janitor      ppp          skge             viro



12) [jg-specific] make desired branch current in working directory

git-switch-tree $branch


13) [jg-specific] create a new branch, and make it current

git-new-branch $branch


14) [jg-specific] examine which branch is current

ls -l .git/HEAD


15) undo all local modifications (same as checkout):

git-read-tree -m HEAD && git-checkout-cache -q -f -u -a

Re: [doc]playing with git, and netdev/libata-dev trees

[Frank Sorenson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jeff Garzik wrote:
> Hopefully, this email can quick-start some people on git.

I think the quick-start is great.  Definitely needed to get people
up-to-speed with git.

> 1) installing git
> 
> git requires bootstrapping, since you must have git installed in order
> to check out git.git (git repo), and linux-2.6.git (kernel repo).  I
> have put together a bootstrap tarball of today's git repository.
> 
> Download tarball from:
> http://www.kernel.org/pub/linux/kernel/people/jgarzik/git-20050526.tar.bz2

A tarball of git will work, but it's a big bootstrap, and will need
periodic updating.

I'm curious whether we couldn't make a git-bootstrap that contained a
significantly stripped-down version that did nothing other than
bootstrap git.  A single program/file (perl script perhaps?) that...

mkdir git
cd git
rsync -rl rsync://rsync.kernel.org/pub/scm/git/git.git/ .git
echo rsync://rsync.kernel.org/pub/scm/git/git.git > .git/branches/origin
beginning with .git/HEAD, start extracting files

...and nothing more.  Then, we could tell people to just download
git-bootstrap and run it to create an up-to-date git repo.

Frank
- --
Frank Sorenson - KD7TZK
Systems Manager, Computer Science Department
Brigham Young University
frank@tuxrocks.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFClYaKaI0dwg4A47wRAp1MAKCcGK8vTWtw1gnTCjFbMFpbZkSO8QCff5RE
NC8Z7RVHFP4qcbKRMSJ2rzg=
=fXsr
-----END PGP SIGNATURE-----

Re: [doc]playing with git, and netdev/libata-dev trees

[James Purser]

On Thu, 2005-05-26 at 18:19, Frank Sorenson wrote:

> A tarball of git will work, but it's a big bootstrap, and will need
> periodic updating.
> 

I don't really see the problem with an initial install from a tarball.
If the tarball is a direct pull of the git.git tree .git/ and all then
all anyone has to do to keep it up to date is either grab a copy of
cogito, or set up their own git based scripts.


-- 
James Purser
http://k-sit.com

Re: [doc]playing with git, and netdev/libata-dev trees

[Dave Airlie]

> 
> When I merge a patch for drivers/net/forcedeth.c, I merge it into a
> brand new 'forcedeth' repository, a peer to the 40+ other such
> repository.  Under BitKeeper, I made these repositories available merged
> together into one big "netdev-2.6" repository because it was too time
> consuming to make the individual 50+ trees publicly available.  With
> git, developers have direct access to the individual trees.
> 
> I thought I would write up a quick guide describing how to mess around
> with the netdev and libata-dev trees, and with git in general.
> 

Thanks for this, I'm starting to get up to speed on git now...

Two questions,

1. when you want to publish your tree what do you do? just rsync it
onto kernel.org?
2. When you are taking things from your queue for Linus do you create
another tree and merge your branches into it or what?

Dave.

Re: [doc]playing with git, and netdev/libata-dev trees

[Jeff Garzik]

[-- Attachment #1: Type: text/plain, Size: 2660 bytes --]

Dave Airlie wrote:
>>When I merge a patch for drivers/net/forcedeth.c, I merge it into a
>>brand new 'forcedeth' repository, a peer to the 40+ other such
>>repository.  Under BitKeeper, I made these repositories available merged
>>together into one big "netdev-2.6" repository because it was too time
>>consuming to make the individual 50+ trees publicly available.  With
>>git, developers have direct access to the individual trees.
>>
>>I thought I would write up a quick guide describing how to mess around
>>with the netdev and libata-dev trees, and with git in general.
>>
> 
> 
> Thanks for this, I'm starting to get up to speed on git now...
> 
> Two questions,
> 
> 1. when you want to publish your tree what do you do? just rsync it
> onto kernel.org?

Basically.  I copy the attached script into each repo, customize the 
script for the upload destination.

When I publish the tree, I just cd to the toplevel dir on my local 
workstation, and run "./push"


> 2. When you are taking things from your queue for Linus do you create
> another tree and merge your branches into it or what?

Not quite sure what you're asking, but I'll attempt to answer anyway :)

When I prepare a submission for Linus, I will pull several branches into 
a single 'for-linus' conglomeration branch.  I do this with the 
following incantation, for each branch I am pushing upstream:

git-resolve-script $(cat .git/HEAD) $(cat .git/refs/heads/$branch) \
	`pwd` $branch

(Note that I modified my git-resolve-script to take an additional $4 
argument, which causes the string " branch $4" to be added to the merge 
cset's commit msg)

After I take care of all the merge conflicts[1] from the 
git-resolve-script run, I give the big pile of mush a final build and 
test, and then run "./push".

For the "please pull" email, I use diffstat, git-diff-tree, 
git-changes-script and the newly written git-shortlog.

	Jeff



[1] I'm still scared of conflicts in the merge process.  Simple and 
automatic merging works just fine, like it did under BitKeeper.  But if 
there are conflicts that cause git-pull-script/git-resolve-script to 
bail, then I bail as well:  I export a patch, run patch(1), and then 
handle the merge the Old Fashioned Way(tm) by looking at .rej files.

I really wish somebody would write a merge helper for git that places 
the conflicts side-by-side in the code [in the working dir].  BitKeeper 
and CVS both presented conflicts to you in this manner.

The "I resolved this conflict, now let's continue where we left off" 
process is still quite raw in git.  I suppose this is something that is 
left for others to script, above the plumbing, but hey...


[-- Attachment #2: push --]
[-- Type: text/plain, Size: 137 bytes --]

#!/bin/sh

rsync -e ssh --verbose --delete --stats --progress -az .git/ master.kernel.org:/pub/scm/linux/kernel/git/jgarzik/misc-2.6.git

Re: [doc]playing with git, and netdev/libata-dev trees

[Dave Airlie]

> > 1. when you want to publish your tree what do you do? just rsync it
> > onto kernel.org?
> 
> Basically.  I copy the attached script into each repo, customize the
> script for the upload destination.
> 
> When I publish the tree, I just cd to the toplevel dir on my local
> workstation, and run "./push"
> 
> 
> > 2. When you are taking things from your queue for Linus do you create
> > another tree and merge your branches into it or what?
> 
> Not quite sure what you're asking, but I'll attempt to answer anyway :)

Yes that's what I'm asking, mainly the pulling of multiple trees into
one tree for giving to Linus, for Andrew I'm quite happy to have him
pull multiple HEADs from the one tree assuming I don't have many
interdependencies between trees...

Say I want something like this one tree with

drm-2.6 - HEAD <- linus tree
            - drm-via < a via driver
            - drm-initmap 
                      - drm-savage <- a savage driver that depends on
the drm-initmap tree

How would I construct such a beast, how does it work out from where to
branch, can I branch a branch for something like drm-savage so I can
say send Linus the initmap branch and then have -mm pulling the savage
one...

if that makes any sense :-)

Dave.
> 
> 
> [1] I'm still scared of conflicts in the merge process.  Simple and
> automatic merging works just fine, like it did under BitKeeper.  But if
> there are conflicts that cause git-pull-script/git-resolve-script to
> bail, then I bail as well:  I export a patch, run patch(1), and then
> handle the merge the Old Fashioned Way(tm) by looking at .rej files.
> 
> I really wish somebody would write a merge helper for git that places
> the conflicts side-by-side in the code [in the working dir].  BitKeeper
> and CVS both presented conflicts to you in this manner.
> 
> The "I resolved this conflict, now let's continue where we left off"
> process is still quite raw in git.  I suppose this is something that is
> left for others to script, above the plumbing, but hey...
> 
> 
> 
> #!/bin/sh
> 
> rsync -e ssh --verbose --delete --stats --progress -az .git/ master.kernel.org:/pub/scm/linux/kernel/git/jgarzik/misc-2.6.git
> 
> 
>

Re: [doc]playing with git, and netdev/libata-dev trees

[Jeff Garzik]

Dave Airlie wrote:
>>>1. when you want to publish your tree what do you do? just rsync it
>>>onto kernel.org?
>>
>>Basically.  I copy the attached script into each repo, customize the
>>script for the upload destination.
>>
>>When I publish the tree, I just cd to the toplevel dir on my local
>>workstation, and run "./push"
>>
>>
>>
>>>2. When you are taking things from your queue for Linus do you create
>>>another tree and merge your branches into it or what?
>>
>>Not quite sure what you're asking, but I'll attempt to answer anyway :)
> 
> 
> Yes that's what I'm asking, mainly the pulling of multiple trees into
> one tree for giving to Linus, for Andrew I'm quite happy to have him
> pull multiple HEADs from the one tree assuming I don't have many
> interdependencies between trees...
> 
> Say I want something like this one tree with
> 
> drm-2.6 - HEAD <- linus tree
>             - drm-via < a via driver
>             - drm-initmap 
>                       - drm-savage <- a savage driver that depends on
> the drm-initmap tree
> 
> How would I construct such a beast, how does it work out from where to
> branch, can I branch a branch for something like drm-savage so I can
> say send Linus the initmap branch and then have -mm pulling the savage
> one...

You used to use BitKeeper, right?

Well, git works just fine for creating and maintaining separate repos, 
where each repo is its own branch.  Dependent branches are handled 
simply by pulling new changes from the parent (drm-initmap) repo into 
the child (drm-savage) repo.

If you just have a few branches, such as your example above, rather than 
the 50+ branches I have, then I would recommend _ignoring_ the advice in 
my git howto, and instead create a new branch by cloning:

	cd /repos
	mkdir drm-via
	cp -a linux-2.6/.git drm-via
	rm -rf drm-via/.git/objects
	cp -al linux-2.6/.git/objects drm-via/.git

Once created by the above recipes, or similar, the directory will 
function as a repo just like with BK.  When you wish to update it from 
upstream, update your local linux-2.6 repo and then

	cd /repos/drm-via
	git-pull-script /repos/linux-2.6/.git

or to pull new changes from initmap into savage:

	cd /repos/drm-initmap
	# { check in some changes }
	cd /repos/drm-savage
	git-pull-script /repos/drm-initmap/.git

Does all that make sense?  It's really quite similar to BK, if you 
ignore the "[jg-specific]" portions of the git howto I wrote.

	Jeff