Locally manage user/branch setting files without pushing them remotely

Locally manage user/branch setting files without pushing them remotely

[Daniele Segato]

Hi,

I had a situation I don't know how to manage with Git.

The project has a lot of setting that are really binded to the user
and should not go pushed in a remote repository (example: database
connections parameters / filesystem paths)
We have a .template file for those settings but the actual settings
are really binded to the user environment.
I added those files to the .gitignore but now I can't keep an history
of them and I can't use different version of them in different
branches.

I tell you what's the real situation but I will simplify things a
little, suppose I only have two branches: master and experiments.

I've a java application and some ANT script to automatizations like
compiling, packaging and deploying stuffs.

On my system I had two application servers and two databases one for
"master" and the other for "experiment".
There is a property files with the path of the two application servers
and the parameters to connect at the DB.

The ant script automatically deploy on the right application server
and automatically set the properties to connect at the right database.

This allow me to clean my experimantal database everytimes I need to
and start over by a clean situation and keep the "master" one
populated with all the data so I can do stress test/update procedures
test and so on...

At the moment I manually modify that property files when I checkout
one of the two branches and I need to deploy it to test or whatever.



Is there a way with Git to automatically switch that file when i
switch through branches?
It would be really perfect if I could also keep version of that file
so that when I checkout an old commit I had the options I was using
when I was testing that commit.

Ideally that means that should be some files that are kept within each
commit but they are not pushed to a remote repository.
(Even if i'm talking about pushing I'm using git-svn because my
company still use SVN as versioning system so I don't push but i do
git svn dcommit when committing on a remote repository)

How do you manage situation like this?

thanks,
Daniele

Re: Locally manage user/branch setting files without pushing them remotely

[Yann Simon]

2009/11/11 Daniele Segato <daniele.bilug@gmail.com>:
> I had a situation I don't know how to manage with Git.
>
> The project has a lot of setting that are really binded to the user
> and should not go pushed in a remote repository (example: database
> connections parameters / filesystem paths)
> We have a .template file for those settings but the actual settings
> are really binded to the user environment.
> I added those files to the .gitignore but now I can't keep an history
> of them and I can't use different version of them in different
> branches.
>
> I tell you what's the real situation but I will simplify things a
> little, suppose I only have two branches: master and experiments.
>
> I've a java application and some ANT script to automatizations like
> compiling, packaging and deploying stuffs.
>
> On my system I had two application servers and two databases one for
> "master" and the other for "experiment".
> There is a property files with the path of the two application servers
> and the parameters to connect at the DB.
>
> The ant script automatically deploy on the right application server
> and automatically set the properties to connect at the right database.
>
> This allow me to clean my experimantal database everytimes I need to
> and start over by a clean situation and keep the "master" one
> populated with all the data so I can do stress test/update procedures
> test and so on...
>
> At the moment I manually modify that property files when I checkout
> one of the two branches and I need to deploy it to test or whatever.
>
>
>
> Is there a way with Git to automatically switch that file when i
> switch through branches?
> It would be really perfect if I could also keep version of that file
> so that when I checkout an old commit I had the options I was using
> when I was testing that commit.
>
> Ideally that means that should be some files that are kept within each
> commit but they are not pushed to a remote repository.
> (Even if i'm talking about pushing I'm using git-svn because my
> company still use SVN as versioning system so I don't push but i do
> git svn dcommit when committing on a remote repository)
>
> How do you manage situation like this?

I am in the same situation.
What I did is not to add these files to .gitignore.
On my "work" branch, I commited these files in a separate commit
"local changes".
When I want to push to svn, I switch to the branch "master", and
cherry-pick the commit I want to push.
$ git checkout master
$ git svn rebase
$ git cherry-pick work
$ git svn dcommit
$ git checkout work
$ git rebase master

This workflow could maybe be better. Suggestions welcome.

Yann

Re: Locally manage user/branch setting files without pushing them remotely

[Daniele Segato]

On Wed, Nov 11, 2009 at 1:14 PM, Yann Simon <yann.simon.fr@gmail.com> wrote:
> I am in the same situation.
> What I did is not to add these files to .gitignore.
> On my "work" branch, I commited these files in a separate commit
> "local changes".
> When I want to push to svn, I switch to the branch "master", and
> cherry-pick the commit I want to push.
> $ git checkout master
> $ git svn rebase
> $ git cherry-pick work
> $ git svn dcommit
> $ git checkout work
> $ git rebase master
>
> This workflow could maybe be better. Suggestions welcome.

hum
but I don't want to push those files


the idea of placing those files in a separate commit could be handy...
If I create a branch for each version of the file: example
experiments-local, master-local then I could keep version of the file
there and cherry pick them when needed

like this:

$ git checkout experiment
$ # do work
$ git commit -a -m "my recent work"
$ git cherry-pick experiment-local
$ # test / compile / whatever
$ git reset --hard HEAD~ # undo the cherry-pick because i don't want to push it
$ git svn dcommit / git chechout another-branch

but I could forgot to reset --hard HEAD~ and I had to be very careful...

instead I would place those files in separate stash called
"experiments", "master" and apply the right stash when I need and then
discard it before switching branch..
this is something I hadn't thinked about

$ git checkout experiment
$ # do work
$ git stash apply experiment-stash # don't know if i could use name to
reference stashes
# test / compile / whatever
$ rm mypropfile.properties
$ git add <files...>
$ git commit
$ git svn dcommit / git checkout another-branch

this way I still had to remove the file manually but since I never use
git commit -a I probably wouldn't add it to a commit by mistake.

This could be more handy then the way I act now...
still I hope there is a better way to handle situations like this

regards,
Daniele

Re: Locally manage user/branch setting files without pushing them remotely

[Yann Simon]

2009/11/11 Daniele Segato <daniele.bilug@gmail.com>:
> On Wed, Nov 11, 2009 at 1:14 PM, Yann Simon <yann.simon.fr@gmail.com> wrote:
>> I am in the same situation.
>> What I did is not to add these files to .gitignore.
>> On my "work" branch, I commited these files in a separate commit
>> "local changes".
>> When I want to push to svn, I switch to the branch "master", and
>> cherry-pick the commit I want to push.
>> $ git checkout master
>> $ git svn rebase
>> $ git cherry-pick work
>> $ git svn dcommit
>> $ git checkout work
>> $ git rebase master
>>
> hum
> but I don't want to push those files

I explained myself not well.
In my "work" branch, there is
"local changes", with the local configuration files
and then "commit xyz to push"
That's why I make a git cherry-pick work to get the last commit, the
one to push.

Yann

Re: Locally manage user/branch setting files without pushing them remotely

[Paolo Ciarrocchi]

On Wed, Nov 11, 2009 at 2:00 PM, Daniele Segato <daniele.bilug@gmail.com> wrote:
> On Wed, Nov 11, 2009 at 1:14 PM, Yann Simon <yann.simon.fr@gmail.com> wrote:
>> I am in the same situation.
>> What I did is not to add these files to .gitignore.
>> On my "work" branch, I commited these files in a separate commit
>> "local changes".
>> When I want to push to svn, I switch to the branch "master", and
>> cherry-pick the commit I want to push.
>> $ git checkout master
>> $ git svn rebase
>> $ git cherry-pick work
>> $ git svn dcommit
>> $ git checkout work
>> $ git rebase master
>>
>> This workflow could maybe be better. Suggestions welcome.
>
> hum
> but I don't want to push those files

If you follow Yann's advise you won't push to the svn repo the "private" files,
they will be /only/ stored in you local repo.

The cons of that approach is that you will have to cherrypick all the
commits you want to push to the svn repo from the "working branch" to
the master branch.

Ciao,
-- 
Paolo

Re: Locally manage user/branch setting files without pushing them remotely

[Nicolas Sebrecht]

The 11/11/09, Daniele Segato wrote:

> The project has a lot of setting that are really binded to the user
> and should not go pushed in a remote repository (example: database
> connections parameters / filesystem paths)
> We have a .template file for those settings but the actual settings
> are really binded to the user environment.

This looks to _not_ be a piece of the development project. Settings used
for the tests suites, company where you work, etc _are_ clones (from the
abstract POV, at least) of the maintainer public repository.

If you want to track changes of settings in a per-production-repo basis,
you should do it _out_ of your main development repository. The latter
should have a publishable template instead (eg. "user = user_login",
"password = set_your_password").

-- 
Nicolas Sebrecht

Re: Locally manage user/branch setting files without pushing them remotely

[Daniele Segato]

On Wed, Nov 11, 2009 at 8:24 PM, Nicolas Sebrecht <nicolas.s.dev@gmx.fr> wrote:
> This looks to _not_ be a piece of the development project. Settings used
> for the tests suites, company where you work, etc _are_ clones (from the
> abstract POV, at least) of the maintainer public repository.
>
> If you want to track changes of settings in a per-production-repo basis,
> you should do it _out_ of your main development repository. The latter
> should have a publishable template instead (eg. "user = user_login",
> "password = set_your_password").

you are probably right but even if I place this files outside my
repository I still have to modify them manually..
it's the same of having them into the repo but in .gitignore

we are customizing a third party product and I can't change the way it
is build up.

I don't think that this situation is so uncommon anyway.

Regards,
Daniele

Re: Locally manage user/branch setting files without pushing them remotely

[Nicolas Sebrecht]

The 12/11/09, Daniele Segato wrote:
> On Wed, Nov 11, 2009 at 8:24 PM, Nicolas Sebrecht <nicolas.s.dev@gmx.fr> wrote:
> > The 11/11/09, Daniele Segato wrote:
> >
> > If you want to track changes of settings in a per-production-repo basis,
> > you should do it _out_ of your main development repository. The latter
> > should have a publishable template instead (eg. "user = user_login",
> > "password = set_your_password").
> 
> you are probably right but even if I place this files outside my
> repository I still have to modify them manually..

I don't understand. You'll still have to manually these files regardless
_how_ they are tracked.

> it's the same of having them into the repo but in .gitignore

When I say, "out of the main development repository", I don't think "out
of the directory of the repository" but "out of the tracking git system".

I think you should look at 'git subtree' or 'git submodule' to track the
setting files out of the _main_ repository and _inside_ the main
development directory.

> we are customizing a third party product and I can't change the way it
> is build up.
> 
> I don't think that this situation is so uncommon anyway.

Yeah, it doesn't help to be unable to do everything we want! But I'm
pretty sure git can do something very convenient in your case.

-- 
Nicolas Sebrecht

Re: Locally manage user/branch setting files without pushing them remotely

[Daniele Segato]

On Thu, Nov 12, 2009 at 10:31 AM, Nicolas Sebrecht <nicolas.s.dev@gmx.fr> wrote:
> I don't understand. You'll still have to manually these files regardless
> _how_ they are tracked.

the first time i clone the repo I had to compile them manually...
but then I would like to keep different setting in my different branches and
I would like to find a way to "load" the right conf file automatically
when I switch repo

> When I say, "out of the main development repository", I don't think "out
> of the directory of the repository" but "out of the tracking git system".

of course, which is almost the same

> I think you should look at 'git subtree' or 'git submodule' to track the
> setting files out of the _main_ repository and _inside_ the main
> development directory.

will they work with git-svn?
I'm not very familiar with them anyway.

>> we are customizing a third party product and I can't change the way it
>> is build up.
>>
>> I don't think that this situation is so uncommon anyway.
>
> Yeah, it doesn't help to be unable to do everything we want! But I'm
> pretty sure git can do something very convenient in your case.

you are right..
but since this is not so uncommon I think other people has found a way
to handle it.

May be using an external/local repository for those files.
but then I would like to have a way to link them automatically.

may be this can be achieved with some hook script..

What I have in mind is something like this:

1. creating a local repo "confrepo" on the same directory where I have
the real repo
2. .gitignore for all my local config file in the real repo
3. using the confrepo to store my local scripts and replicate in it
the same branch structure of my real repo

4. may be then I can create some hook script that when I do

$ git checkout experimental

a) it try to do the same on the confrepo (checkout experimental), if
the branch doesn't exist it does nothing else
b) copy everythink in the confrepo to my realrepo (overwriting if needed)


that way I could have something automatic. The only think I couldn't
have is, looking back in history, a way to know which config was in
place for an old commit.


could you tell me if this is even possible?


I'm going a little off-topic here proposing an idea for a new feature
What about adding a git feature to "link" different git repository?
Or providing a way to have 2 (or more) git repository in the same working area.

an ipotetical usage:

$ mkdir realrepo && cd realrepo
$ git (svn) clone http://whatever... .
--> master

$ git link createrepo ../confrepo confs
--> Initialized repo ../confrepo
--> Linked repo confs to ../confrepo
--> Created branch master in ../confrepo

$ touch localconf.properties
$ touch real-repo-file.c
$ git add real-repo.file.c
--> added real-repo-file.c to stage (as usual)
$ git link add localconf.properties
--> Added local.conf.properties to confs stage

$ git commit
--> standard commit
--> added localconf.properties to .gitignore (or .git/info/exclude)
--> commit on confs

$ git checkout -b experimental # creating new branch
--> checkout -b experimental on confs

$ # modify many things and the localconf.properties

$ git status
--> show standard output and on new section "linked files":
localconf.properties on confs has been modified

$ git add . && git commit
--> localconf.properties is committed in "confs" repository branch
"experimental"

$ git checkout master
--> checkout master on confs
--> copy all files of the confs repo to the current real repo working area

I think you got the idea.

the "linked" repo could also keep a tracked files like this:

.gitlink-source

which could keep the commit SHA of the real repo for each commit in
the confs repo

so that you had a way to get the right "confs" commit associated with
each commit on the real repo.


I don't know git enough to see how this could be hard or which
drawback could have.

Re: Locally manage user/branch setting files without pushing them remotely

[Nicolas Sebrecht]

The 12/11/09, Daniele Segato wrote:
> 
> > I think you should look at 'git subtree' or 'git submodule' to track the
> > setting files out of the _main_ repository and _inside_ the main
> > development directory.
> 
> will they work with git-svn?
> I'm not very familiar with them anyway.

Not the way you'd like to, I think. This could mean having as many svn
repo as git subtree/submodule.

> May be using an external/local repository for those files.
> but then I would like to have a way to link them automatically.
> 
> may be this can be achieved with some hook script..

You could look at 'man githooks'.

> What I have in mind is something like this:
> 
> 1. creating a local repo "confrepo" on the same directory where I have
> the real repo
> 2. .gitignore for all my local config file in the real repo
> 3. using the confrepo to store my local scripts and replicate in it
> the same branch structure of my real repo
> 
> 4. may be then I can create some hook script that when I do
> 
> $ git checkout experimental
> 
> a) it try to do the same on the confrepo (checkout experimental), if
> the branch doesn't exist it does nothing else
> b) copy everythink in the confrepo to my realrepo (overwriting if needed)
> 
> 
> that way I could have something automatic. The only think I couldn't
> have is, looking back in history, a way to know which config was in
> place for an old commit.
> 
> 
> could you tell me if this is even possible?

I think, yes.

> I'm going a little off-topic here proposing an idea for a new feature
> What about adding a git feature to "link" different git repository?
> Or providing a way to have 2 (or more) git repository in the same working area.

Patches are welcome. That said, I guess it won't be needed. As you
pointed out, the flexibility has been implemented in the hooks system.
Consequently, we all have to write the appropriate scripts. But OTOH, it
is more flexible than any feature matching _one_ workflow.


-- 
Nicolas Sebrecht