How to manage parameter files and code separately using git?

How to manage parameter files and code separately using git?

[Tilo Schwarz]

Hi all,

maybe someone has an idea how to do what I'd like to do using git:

I use git for "normal" c coding, i.e. having branches like master, test,  
foo, etc. Built are executables (Linux), which need parameter text files  
to work. What I did up to now was to check in the parameter files in the  
same way I check in the code: when I change a parameter file, I do a  
commit on it, this way I always have a history of my parameter files.

This has one drawback: If I check out an older version, I also get the old  
parameter file, which is not what I want, because the parameters are  
determined by hardware settings. I.e., I would like to checkout an old  
commit, but still have the last version of the parameter file.

I tried to solve the problem by

1. creating a branch 'parameters', which contains only the parameter files
2. ignore the parameters files with extension *.m in .gitignore in the  
code branches (master etc.)

This seemed to work at first, but when I switch back from the parameters  
branch to master, all *.m files are removed, although .gitignore of master  
contains '*.m'.

So apparently git removes during branch switch first the tracked files and  
than populates the working dir with the new files of master.

Now my questions are

1. Is there a way to work around it?
2. Is there a maybe totally different way to solve my initial problem  
(separate code history from parameter file history) without using two git  
repos?


Thanks a lot!

     Tilo

Re: How to manage parameter files and code separately using git?

[Junio C Hamano]

"Tilo Schwarz" <tilo@tilo-schwarz.de> writes:

> I use git for "normal" c coding, i.e. having branches like master, test,  
> foo, etc. Built are executables (Linux), which need parameter text files  
> to work. What I did up to now was to check in the parameter files in the  
> same way I check in the code: when I change a parameter file, I do a  
> commit on it, this way I always have a history of my parameter files.
>
> This has one drawback: If I check out an older version, I also get the old  
> parameter file, which is not what I want, because the parameters are  
> determined by hardware settings. I.e., I would like to checkout an old  
> commit, but still have the last version of the parameter file.

This design constraint makes the issue not a version control problem but
more of a software engineering problem.  The changes to your parameter
file have its own history (e.g. you upgraded your machine last week with
different hardware---the software didn't change but the parameter has to)
and the history is largely independent of the software that reads the
file.  In that sense, you should _never_ tie its history with the history
of your software.

At the same time, the history of the parameter file is not completely
independent of that of the software.  You may have added a new feature and
code to read a new setting in the parameter file today and older parameter
file would not have a setting for that variable.

The method I used in my day-job project is something like this:

 - Include a "config.sample" file and track it in the history of the
   source code; the values it gives to the variables are the same as the
   default ones.  This file mostly serves as a documentation.

 - Design your parameter mechanism in a way that:

   - a parameter file can include another;

   - an unrecognized parameter setting is ignored;

   - a variable definition that happens later overrides an earlier
     definition.

 - Have a real parameter file that is _not_ tracked in the history of the
   source code.  Have it include parameter.sample and then define machine
   specific settings after that to override.

It is only required for the "real" parameter file not to be tracked in the
history of the source code---it is perfectly fine to track it in a
separate history.  The easiest way to do this would be to keep a parameter
working tree next door, and "ln -s ../params/config config" in the working
tree for the source code.  You could keep this symlink tracked in the
source history, but you do not have to.

You will try to read from ./config, which will include ./config.sample in
order to read the default, and then read the customized setting that
appear later in ./config.  Because of the symlink, you will actually be
reading from ../params/config next door.

Re: How to manage parameter files and code separately using git?

[Tilo Schwarz]

On Sat, 01 May 2010 19:18:55 +0200, Junio C Hamano <gitster@pobox.com>  
wrote:

Thank you for the detailed explanation!

> This design constraint makes the issue not a version control problem but
> more of a software engineering problem.  The changes to your parameter
> file have its own history (e.g. you upgraded your machine last week with
> different hardware---the software didn't change but the parameter has to)
> and the history is largely independent of the software that reads the
> file.  In that sense, you should _never_ tie its history with the history
> of your software.

Ok.

> At the same time, the history of the parameter file is not completely
> independent of that of the software.  You may have added a new feature  
> and code to read a new setting in the parameter file today and older  
> parameter file would not have a setting for that variable.

Exactly.

> The method I used in my day-job project is something like this:
>
>  - Include a "config.sample" file and track it in the history of the
>    source code; the values it gives to the variables are the same as the
>    default ones.  This file mostly serves as a documentation.

I actually did exactly that already (but forgot to mention it in my first  
mail, sorry!). The problem with this "one" config.sample is, that I need  
about three really different config files because of really different  
sensor setups on about five different computers. So when I push on a USB  
stick, walk to another computer, then pull and try to run the software -  
bummer, forgot to copy the current config file - walk back, copy ...
But maybe the solution below solves this too. Or I write a little script  
doing the push and parameter file copy.

>  - Design your parameter mechanism in a way that:
>
>    - a parameter file can include another;

Can't do that yet.

>    - an unrecognized parameter setting is ignored;

Can do.

>    - a variable definition that happens later overrides an earlier
>      definition.

Can do.

>  - Have a real parameter file that is _not_ tracked in the history of the
>    source code.  Have it include parameter.sample and then define machine
>    specific settings after that to override.

Good idea.

> It is only required for the "real" parameter file not to be tracked in  
> the history of the source code---it is perfectly fine to track it in a
> separate history.  The easiest way to do this would be to keep a  
> parameter working tree next door, and "ln -s ../params/config config" in  
> the working tree for the source code.  You could keep this symlink  
> tracked in the
> source history, but you do not have to.

I see. So if I want parameter file history the only proper solution is to  
have a separate parameter file git repo, right?

> You will try to read from ./config, which will include ./config.sample in
> order to read the default, and then read the customized setting that
> appear later in ./config.  Because of the symlink, you will actually be
> reading from ../params/config next door.

Ok, will give it a try.

-- 
Regards,

	Tilo

Re: How to manage parameter files and code separately using git?

[Junio C Hamano]

"Tilo Schwarz" <tilo@tilo-schwarz.de> writes:

> I see. So if I want parameter file history the only proper solution is to  
> have a separate parameter file git repo, right?

I wouldn't say it is "the" "only" proper solution, but under your design
constraint that dictates that the part that is left in "config" (which
includes the tracked-and-tied-to-the-software-version "config.sample")
must have a history that is independent from the software history, it is
one workable solution that would be the easiest.  I can imagine a more
elaborate implementation that stores the history of that config file on a
separate branch in the same repository and manipulate that file with
custom scripts, and that also would be another workable solution.