Looking for a way to set up Git correctly

Looking for a way to set up Git correctly

[Dennis]

I have a situation.

I have started a web project (call it branch1), and have maintained it 
without a version control system for quite some time.
Then, I copied it to another folder (branch2) and while the project remained 
essentially the same, I have changed a few of internal paths and some 
variable names inside the files.
Then, a few months later on, I copied branch2 to a folder called branch3 and 
also modified some of the variable names and some of the internal structure 
of the files.

Thus I ended up with 3 folders on my local HDD with pretty much the same 
file names and folder structure and everything, and most of the file 
content, except those small deltas that made those files different for each 
branch.

I guess it's never too late, and now I want to put these 3 projects into a 
version control system, and I chose git.

Now, this can be either really simple or really complicated.  My first 
question is:  how do I set the repository up in the proper way where I could 
work on all 3 projects separately, with additional possibility of working on 
branch1 only and later committing my changes to branch2 and branch3.  (Since 
projects are virtually identical, a fix in one branch usually needs to be 
propagated to other branches)
First, I assume I will use a single repository for this.  Then, do I simply 
set up 3 branches and start using them, or is there a way to set git up to 
capitalize on the projects being nearly identical?

My second question is that each branch has a huge folder with image data. 
By huge I mean 1 to 4Gb, depending on the branch.  Since images are not 
directly relevant to the development work, is there a way to not include 
those folders in git?  To be honest though, I probably should include them, 
but I wanted to ask about this separately as git repository may be get 
large, since all 3 branches may grow to 9Gb or so.

Thus I am looking for a git way to handle my situation.  Is this simple or 
is is hard?
Are there any recommendations before I jump in?
Dennis 

Re: Looking for a way to set up Git correctly

[Alex Riesen]

On Thu, Nov 11, 2010 at 04:25, Dennis <denny@dennymagicsite.com> wrote:
> I have started a web project (call it branch1), and have maintained it
> without a version control system for quite some time.
> Then, I copied it to another folder (branch2) and while the project remained
> essentially the same, I have changed a few of internal paths and some
> variable names inside the files.
> Then, a few months later on, I copied branch2 to a folder called branch3 and
> also modified some of the variable names and some of the internal structure
> of the files.
>
> Thus I ended up with 3 folders on my local HDD with pretty much the same
> file names and folder structure and everything, and most of the file
> content, except those small deltas that made those files different for each
> branch.
>
> I guess it's never too late, and now I want to put these 3 projects into a
> version control system, and I chose git.
>
> Now, this can be either really simple or really complicated.  My first
> question is:  how do I set the repository up in the proper way where I could
> work on all 3 projects separately, with additional possibility of working on
> branch1 only and later committing my changes to branch2 and branch3.  (Since
> projects are virtually identical, a fix in one branch usually needs to be
> propagated to other branches)
> First, I assume I will use a single repository for this.  Then, do I simply
> set up 3 branches and start using them, or is there a way to set git up to
> capitalize on the projects being nearly identical?

Assuming I've got the relationships of your "branches" right:

$ cp -a branch1 branch && cd branch
$ git init
$ echo /huge-images/ >.gitignore
$ git add .gitignore; git add .; git commit; git branch branch1
$ git checkout -b branch2
$ cp -a ../branch2 .
$ git add .; git commit
$ git checkout -b branch3
$ cp -a ../branch3 .
$ git add .; git commit

> My second question is that each branch has a huge folder with image data. By
> huge I mean 1 to 4Gb, depending on the branch.  Since images are not
> directly relevant to the development work, is there a way to not include
> those folders in git?  To be honest though, I probably should include them,
> but I wanted to ask about this separately as git repository may be get
> large, since all 3 branches may grow to 9Gb or so.
>
> Thus I am looking for a git way to handle my situation.  Is this simple or
> is is hard?

If you add the images you will eventually run into problems (heavy
swapping, for one).
Git is not really setup to work with big binary files (a file must fit into
memory completely).

Re: Looking for a way to set up Git correctly

[Enrico Weigelt]

* Dennis <denny@dennymagicsite.com> wrote:

Hi,

> Now, this can be either really simple or really complicated.  My first 
> question is:  how do I set the repository up in the proper way where I 
> could work on all 3 projects separately, with additional possibility of 
> working on branch1 only and later committing my changes to branch2 and 
> branch3.  

As first step you could create 3 separate git repos in each directory
and add everything to it (git init, git add -A, git commit). Then 
rename the branches properly (so instead of "master", they'll be called
"branch1", "branch2", "branch2" or something like that). Create another
(maybe bare) repo elsewhere, add it as remote to the three other ones
and push their branches upwards. Now you have 4 repos, 3 for working
on the individual branches and another for collecting them all (hub model).
You could also choose to throw the first three away and only work in
the last one.

> (Since projects are virtually identical, a fix in one branch 
> usually needs to be propagated to other branches)

In your case, cherry-pick might be the right for you.
You could also do a little bit refactoring, making a 4th branch which
the other 3 are then rebased onto. Then you could do your fixes in that
branch and merged into or rebase the other 3 onto that one.

> My second question is that each branch has a huge folder with image data. 
> By huge I mean 1 to 4Gb, depending on the branch.  Since images are not 
> directly relevant to the development work, is there a way to not include 
> those folders in git?

see .gitignore file.
nevertheless it might be useful to also have all the images in the
repo for backup reasons.

BTW: if you're concerned about disk space, you could add the object dir
of the 4th (hub) repository to the 3 working repos (run git-gc in the
hub repo before that!). Next gc runs will remove the objects that are
already present in the hub. But beware! If you remove something in the
hub repo and run git-gc there, you could loose objects in the other repos!
(maybe it would be wise to add the 3 working repos as remotes in the
hub and always run an git remote update before git-gc in the hub).


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: Looking for a way to set up Git correctly

[Jonathan Nieder]

(+cc: Dennis again, Alex)

Hi,

Enrico Weigelt wrote:
> * Dennis <denny@dennymagicsite.com> wrote:

>> Now, this can be either really simple or really complicated.  My first 
>> question is:  how do I set the repository up in the proper way where I 
>> could work on all 3 projects separately, with additional possibility of 
>> working on branch1 only and later committing my changes to branch2 and 
>> branch3.  
>
> As first step you could create 3 separate git repos in each directory
> and add everything to it (git init, git add -A, git commit). Then 
> rename the branches properly (so instead of "master", they'll be called
> "branch1", "branch2", "branch2" or something like that). Create another
> (maybe bare) repo elsewhere, add it as remote to the three other ones
> and push their branches upwards.

So this looks like so:

	for i in project1 project2 project3
	do
		(
			cd "$i"
			git init
			git add .
			git commit
		)
	done
	git init main
	cd main
	for i in project1 project2 project3
	do
		git fetch ../$i master:$i
	done
	mv project1 project2 project3 away/

If you would like multiple worktrees (one for each branch, maybe) for
the main repo, you might want to look into the new-workdir script in
contrib/workdir (but do consider the caveats[1]).

>> (Since projects are virtually identical, a fix in one branch 
>> usually needs to be propagated to other branches)
>
> In your case, cherry-pick might be the right for you.

e.g., when project3 gets a new fix:

	git checkout project1
	git cherry-pick project3

> You could also do a little bit refactoring, making a 4th branch which
> the other 3 are then rebased onto.

Right, what is the actual relationship between these projects?  Do
they actually represent branches in the history of a single project?

Suppose project1 is historically an ancestor to project2, project3,
and project4, which are independent.  (Maybe project1 is the initial
version and projects 2,3,4 are ports to other platforms.)  You could
take this into account when initially setting up the branches, like
this:

	git init main
	cd main
	GIT_DIR=$(pwd)/.git; export GIT_DIR
	GIT_WORK_TREE=../project1 git add .
	GIT_WORK_TREE=../project1 git commit
	git branch -m project1
	for i in project2 project3 project4
	do
		git checkout -b $i project1
		GIT_WORK_TREE=../$i git add -A
		GIT_WORK_TREE=../$i git commit
	done

(and use gitk --all when done to make sure everything looks right)

Alternatively, you can rearrange the history afterwards:

	$ git cat-file commit project2 | tee project2
	tree 76db51024713f6ef191928a8445d48d39ab55434
	author Junio C Hamano <gitster@pobox.com> 1289324716 -0800
	committer Junio C Hamano <gitster@pobox.com> 1289324716 -0800

	project2: an excellent project
	$ git rev-parse project1
	$ vi project2
	... add a "parent <object id>" line
	    after the tree line,
	    where <object id> is the full object name rev-parse printed ...
	$ git hash-object -t commit -w project2
	$ git branch -f branch2 <the object name hash-object prints>
	... repeat for project3 and project4 ...
	$ gitk --all;		# to make sure everything looks right

This is less convenient than it ought to be.  It would be nice to add
a "git graft" command to automate this procedure, which

 - interacts well with "git replace"
 - doesn't interact poorly with "git fetch" like .git/info/grafts does
 - could be more convenient to use than .git/info/grafts.

As the gitworkflows man page mentions, if you make your fixes on the
oldest branch they apply to (project1) and then merge to all later
branches, then the fixes will propagate forward correctly.  See the
"Graduation" and "Merging upwards" sections of gitworkflows for details.

>> My second question is that each branch has a huge folder with image data. 
>> By huge I mean 1 to 4Gb, depending on the branch.  Since images are not 
>> directly relevant to the development work, is there a way to not include 
>> those folders in git?

I would suggest tracking a symlink to another repository (or to a
directory tracked through other means, like unison).

Hope that helps,
Jonathan

[1] If you have two worktrees for the same project with the
same branch checked out at a given moment, the results can be
confusing (changes made in one worktree will look like they have
been commited and undone in the other).

The "detached HEAD" feature (which git-checkout.1 explains) and
multiple worktrees do not interact so well: the need to preserve
commits while no branch was checked out in one worktree will not be
taken into account when "git gc" runs (explicitly or implicitly!) on
the other.  This can be very disconcerting.

Re: Looking for a way to set up Git correctly

[Jonathan Nieder]

denny@dennymagicsite.com wrote:

> I am still looking through your replies and getting familiar with
> git commands.

By the way, please ignore that GIT_WORK_TREE stuff I did.  It
probably works, but it's ugly. :)  That example could have been
written better as

	git init everything
	GIT_DIR=$(pwd)/everything/.git; export GIT_DIR
	(
		cd common-ancestor
		git add -A
		git commit
		git branch -m ancestor
	)
	(
		cd project1
		git checkout -b project1 ancestor
		git add -A
		git commit
	)
	... etc ..
	unset GIT_DIR

	cd everything
	git checkout project1

[...]
> From a developer's point of view, working on projectX means making
> some changes and committing them to the repo for that project.  The
> developer may not be aware of other pojects existing.

For concreteness, I am imagining these directories represent various
versions of the Almquist shell.  The common ancestor is the BSD4.3/Net-2
version and various projects may have built from there in different
directions: NetBSD sh, FreeBSD sh, dash.  (Yes, I am oversimplifying. :))

Now suppose they have diverged so wildly that it is never possible to
synchronize code with each other.  Instead, they can copy fixes, and
this is especially convenient when the fixes are phrased as diffs to
the common ancestor.

To facilitate this, Alice revives the BSD4.3/Net-2 sh project with a
"fixes only" policy.  Her daily work might look like this:

 $ git fetch netbsd
 $ git log netbsd/for-alice@{1}..netbsd/for-alice; # any good patches today?
 $ git cherry-pick -s 67fd89980; # a good patch.
 ... quick test ...
 $ git cherry-pick -s 897ac8; # another good patch.
 ... quick test ...
 ...
 $ git fetch freebsd
 ... and similarly for the rest of the patch submitters ...
 $ git am emailed-patch

Then to more thoroughly test the result:

 $ git checkout -b throwaway;	# new throw-away branch.[1]
 $ git merge netbsd/master;	# will the changes work for netbsd?
 ... thorough test ...
 $ git reset --keep master
 $ git merge freebsd/master;	# how about freebsd?
 ... etc ...

And finally she pushes the changes out.

> Without knowing anything about git for a moment, one ideal workflow
> is where a developer makes changes to projectX that touch the base
> and projectX specific features.  Then the developer commits them and
> pushes them to the main repo.  The main repo contains all projects.
> During the commit, chages to the base automagically get pushed to
> all projects that share that base

If it is a matter of what files are touched, then maybe the base is
actually something like a library, which should be managed as a
separate project.  See the "git submodule" manual if you would like to
try something like this but still keep the projects coupled.

On the other hand, remaining in the situation from before:

Suppose Sam is the NetBSD sh maintainer.  The first step in working on
a new release might be

 $ git fetch ancestor
 $ git log -p HEAD..FETCH_HEAD;	# fixes look okay?
 $ git pull ancestor

since Alice tends to include only safe, well tested fixes.

Many changes Sam makes are specific to his project, but today he comes
up with a fix that might be useful for other ash descendants.

So instead of commiting directly, he can try:

 $ git checkout for-alice;	# carry the fix to the for-alice branch
 ... test ...
 $ git commit -a;		# commit it.

If it is not an urgent fix, at this point he might do

 $ git checkout master;		# back to the main NetBSD branch, without the fix

and give the other projects some time to work on the patch and come up
with a better fix.  Or he might cherry-pick the commit from for-alice,
and even publish it and encourage others to cherry-pick directly from
him to get the fix out ASAP.

Notice that not all changes to the base files are necessarily useful
for other descendants of the ancestral program.  So in this example,
propagation of changes between projects is fairly explicit.

[1] "git checkout HEAD^0" would be more convenient.
See DETACHED HEAD in the git checkout manual if interested.