Metadata and checkin file date

Metadata and checkin file date

[Gerhard Wiesinger]

Hello,

I'm new to git and I'm looking for the following features:
1.) Metadata for
   a.) directory versioning (e.g. add/rm, mv)
   b.) rights (basic: chmod, chow, chgrp, extended: extended attributes 
like ACLs and selinux), necessary for versioning e.g. /etc
2.) Original file dates (checkin date) on clone and pull (and not checkout 
date)

Is this possible? Any plans if missing?

Thnx.

Ciao,
Gerhard

--
http://www.wiesinger.com/

Re: Metadata and checkin file date

[Andreas Ericsson]

On 04/27/2010 07:23 AM, Gerhard Wiesinger wrote:
> Hello,
> 
> I'm new to git and I'm looking for the following features:
> 1.) Metadata for
> a.) directory versioning (e.g. add/rm, mv)

If you're talking about empty directories, that feature doesn't
exist and I can't imagine why you'd want it to. If you'd care to
explain why you want it, I'm sure we can find a different way of
achieving your goal.

> b.) rights (basic: chmod, chow, chgrp, extended: extended attributes 
> like ACLs and selinux), necessary for versioning e.g. /etc

Sounds like you want a backup-program. Some projects have been
aimed towards this goal already. I'm sure google can provide
more information. AFAIR, most of them work with two hook-scripts
that update a regular file with the meta-data of all tracked
files. This makes committing and checking out slower than it
would otherwise be, but since it's doing more I suppose that's
to be expected.

Adding it to core git would mean re-designing git's basic data
model, which is obviously not something we're about to do on
a whim.

> 2.) Original file dates (checkin date) on clone and pull (and not 
> checkout date)
> 

I expect the solutions that work for 1b will also have this
"feature", or that it will be easy to patch for it. For a
source code management system though, this is a very bad
idea indeed since it messes with the fundamental rules of
building; A changed file must be rebuilt.

Seeing as this would also require a major change in git's
data model, this is another of those changes that I doubt
will be supported in the git core in the foreseeable future.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

Re: Metadata and checkin file date

[Jakub Narebski]

Gerhard Wiesinger <lists@wiesinger.com> writes:

> Hello,
> 
> I'm new to git and I'm looking for the following features:
> 1.) Metadata for
>    a.) directory versioning (e.g. add/rm, mv)
>    b.) rights (basic: chmod, chow, chgrp, extended: extended
> attributes like ACLs and selinux), necessary for versioning e.g. /etc
> 2.) Original file dates (checkin date) on clone and pull (and not
> checkout date)
> 
> Is this possible? Any plans if missing?

Git is distributed version control system (DVCS), not a backup system.
It is used mainly for distributed development of programs.  Therefore
it supports natively only those parts of metadata that make sense for
VCS, namely symlinks (with workaround for filesystems that do not have
support for symbolic links) and the executable permission for files.

File ownership does not make sense for VCS, as other people that clone
your repository do not have the same set of users that you have, and
might not have the same set of groups that you have.  Neverthemind that
their filesystem might not support notion of file ownership, not only
do not have support for extended attributes and the like.

If you really, really need this, you can use additional tools to
preserve metadata, like Metastore or git-cache-meta, or even ready
tools that use Git as bckend like IsiSetup or bup (well, bup use
git package format, not git itself...), see this Git Wiki page:
https://git.wiki.kernel.org/index.php/InterfacesFrontendsAndTools

HTH (Hope That Helps)
-- 
Jakub Narebski
Poland

Re: Metadata and checkin file date

[Gerhard Wiesinger]

On Tue, 27 Apr 2010, Andreas Ericsson wrote:

> On 04/27/2010 07:23 AM, Gerhard Wiesinger wrote:
>> Hello,
>>
>> I'm new to git and I'm looking for the following features:
>> 1.) Metadata for
>> a.) directory versioning (e.g. add/rm, mv)
>
> If you're talking about empty directories, that feature doesn't
> exist and I can't imagine why you'd want it to. If you'd care to
> explain why you want it, I'm sure we can find a different way of
> achieving your goal.

Git focuses on content but I think git should also focus on 
metadata. For example restructuring source code moves (git mv 
file1.c file2.c, git mv dir1 dir2) should be documented also in the 
repository like e.g. subversion and commercial SCM like clearcase do. 
Otherwise we are on "CVS" level.

Empty directories is a special case and sometimes you need just versioned 
empty directies.

>> b.) rights (basic: chmod, chow, chgrp, extended: extended attributes
>> like ACLs and selinux), necessary for versioning e.g. /etc
>
> Sounds like you want a backup-program. Some projects have been
> aimed towards this goal already. I'm sure google can provide
> more information. AFAIR, most of them work with two hook-scripts
> that update a regular file with the meta-data of all tracked
> files. This makes committing and checking out slower than it
> would otherwise be, but since it's doing more I suppose that's
> to be expected.
>
> Adding it to core git would mean re-designing git's basic data
> model, which is obviously not something we're about to do on
> a whim.

No, I'm NOT looking for a backup program. Every admin has the problem of 
versioning config files (for example /etc). Versioning of config files 
makes sense because one can track the changes and e.g. correlate to 
problems. A backup program doesn't have features like history, committer 
and comments on file changes. Therefore git would be a perfect tool also for 
versioning configuration. (Software development doesn't end with the build 
but typically also has deployment&configuration issues).

And when you think about a destroyed machine where you want to 
get your versioned configuration you also want to get the right 
permissions therefore. Otherwise you have for sure a security leak or a 
non working machine.

You can also think on machines where the same config should be used and 
tracked.

>> 2.) Original file dates (checkin date) on clone and pull (and not
>> checkout date)
>>
>
> I expect the solutions that work for 1b will also have this
> "feature", or that it will be easy to patch for it. For a
> source code management system though, this is a very bad
> idea indeed since it messes with the fundamental rules of
> building; A changed file must be rebuilt.
>
> Seeing as this would also require a major change in git's
> data model, this is another of those changes that I doubt
> will be supported in the git core in the foreseeable future.

I agree that a changed file must be rebuilt but in the normal "forward" 
cases that's also guaranteed with commit dates as they are "later". But 
when you get an old version you can't assume anything (e.g. dependencies 
have completly changed, even file structure) and therefore only a clean 
build (e.g. make clean) is IHMO valid.

I think git already has the commit date for the revision the user wants. 
Therefore only a call to "set_date_and_time" is necessary when a file 
is touched (e.g. by switch --use-commit-timestamps, environment variable 
or by config item). Therefor this should IHMO be easy to implement without 
any metadata changes.

Thnx.

Ciao,
Gerhard

Re: Metadata and checkin file date

[Gerhard Wiesinger]

On Tue, 27 Apr 2010, Jakub Narebski wrote:

> Gerhard Wiesinger <lists@wiesinger.com> writes:
>
>> Hello,
>>
>> I'm new to git and I'm looking for the following features:
>> 1.) Metadata for
>>    a.) directory versioning (e.g. add/rm, mv)
>>    b.) rights (basic: chmod, chow, chgrp, extended: extended
>> attributes like ACLs and selinux), necessary for versioning e.g. /etc
>> 2.) Original file dates (checkin date) on clone and pull (and not
>> checkout date)
>>
>> Is this possible? Any plans if missing?
>
> Git is distributed version control system (DVCS), not a backup system.
> It is used mainly for distributed development of programs.  Therefore
> it supports natively only those parts of metadata that make sense for
> VCS, namely symlinks (with workaround for filesystems that do not have
> support for symbolic links) and the executable permission for files.
>
> File ownership does not make sense for VCS, as other people that clone
> your repository do not have the same set of users that you have, and
> might not have the same set of groups that you have.  Neverthemind that
> their filesystem might not support notion of file ownership, not only
> do not have support for extended attributes and the like.

I would suggest that only with special switches like --preserve-chmod 
--preserver-owner --preserve-group , etc. where one can guarantee user, 
groups, etc. See also my second post for arguments.

> If you really, really need this, you can use additional tools to
> preserve metadata, like Metastore or git-cache-meta, or even ready
> tools that use Git as bckend like IsiSetup or bup (well, bup use
> git package format, not git itself...), see this Git Wiki page:
> https://git.wiki.kernel.org/index.php/InterfacesFrontendsAndTools
>

Will have a look at them.

Thnx.

Ciao,
Gerhard

--
http://www.wiesinger.com/

Re: Metadata and checkin file date

[Andreas Ericsson]

On 04/27/2010 09:38 PM, Gerhard Wiesinger wrote:
> On Tue, 27 Apr 2010, Andreas Ericsson wrote:
> 
>> On 04/27/2010 07:23 AM, Gerhard Wiesinger wrote:
>>> Hello,
>>>
>>> I'm new to git and I'm looking for the following features:
>>> 1.) Metadata for
>>> a.) directory versioning (e.g. add/rm, mv)
>>
>> If you're talking about empty directories, that feature doesn't
>> exist and I can't imagine why you'd want it to. If you'd care to
>> explain why you want it, I'm sure we can find a different way of
>> achieving your goal.
> 
> Git focuses on content but I think git should also focus on metadata. 
> For example restructuring source code moves (git mv file1.c file2.c, git 
> mv dir1 dir2) should be documented also in the repository like e.g. 
> subversion and commercial SCM like clearcase do. Otherwise we are on 
> "CVS" level.
> 
> Empty directories is a special case and sometimes you need just 
> versioned empty directies.
> 

This has been discussed to death several times before on this mailing
list. Browse the archives. There haven't been any new arguments the
last 14 times it came up, so I doubt you'll be able to come up with
a single good reason to track file renames explicitly.

>>> b.) rights (basic: chmod, chow, chgrp, extended: extended attributes
>>> like ACLs and selinux), necessary for versioning e.g. /etc
>>
>> Sounds like you want a backup-program. Some projects have been
>> aimed towards this goal already. I'm sure google can provide
>> more information. AFAIR, most of them work with two hook-scripts
>> that update a regular file with the meta-data of all tracked
>> files. This makes committing and checking out slower than it
>> would otherwise be, but since it's doing more I suppose that's
>> to be expected.
>>
>> Adding it to core git would mean re-designing git's basic data
>> model, which is obviously not something we're about to do on
>> a whim.
> 
> No, I'm NOT looking for a backup program. Every admin has the problem of 
> versioning config files (for example /etc). Versioning of config files 
> makes sense because one can track the changes and e.g. correlate to 
> problems. A backup program doesn't have features like history, committer 
> and comments on file changes. Therefore git would be a perfect tool also 
> for versioning configuration. (Software development doesn't end with the 
> build but typically also has deployment&configuration issues).
> 

Right. So you want a *fancy* backup program and not just any random
backup solution. There are solutions for this. Someone else already
mentioned them elsewhere in this thread, so I'll refrain from further
comments on this.

In short: What you want can be (and has been) done, but it's written
as addons and not integral parts of git.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

Re: Metadata and checkin file date

[Gerhard Wiesinger]

On Tue, 27 Apr 2010, Andreas Ericsson wrote:

> On 04/27/2010 09:38 PM, Gerhard Wiesinger wrote:
>> On Tue, 27 Apr 2010, Andreas Ericsson wrote:
>>
>>> On 04/27/2010 07:23 AM, Gerhard Wiesinger wrote:
>>>> Hello,
>>>>
>>>> I'm new to git and I'm looking for the following features:
>>>> 1.) Metadata for
>>>> a.) directory versioning (e.g. add/rm, mv)
>>>
>>> If you're talking about empty directories, that feature doesn't
>>> exist and I can't imagine why you'd want it to. If you'd care to
>>> explain why you want it, I'm sure we can find a different way of
>>> achieving your goal.
>>
>> Git focuses on content but I think git should also focus on metadata.
>> For example restructuring source code moves (git mv file1.c file2.c, git
>> mv dir1 dir2) should be documented also in the repository like e.g.
>> subversion and commercial SCM like clearcase do. Otherwise we are on
>> "CVS" level.
>>
>> Empty directories is a special case and sometimes you need just
>> versioned empty directies.
>>
>
> This has been discussed to death several times before on this mailing
> list. Browse the archives. There haven't been any new arguments the
> last 14 times it came up, so I doubt you'll be able to come up with
> a single good reason to track file renames explicitly.

As it pops up that often by git users I think it is an issue which 
shouldn't be ignored as users should be king :-)

>>>> b.) rights (basic: chmod, chow, chgrp, extended: extended attributes
>>>> like ACLs and selinux), necessary for versioning e.g. /etc
>>>
>>> Sounds like you want a backup-program. Some projects have been
>>> aimed towards this goal already. I'm sure google can provide
>>> more information. AFAIR, most of them work with two hook-scripts
>>> that update a regular file with the meta-data of all tracked
>>> files. This makes committing and checking out slower than it
>>> would otherwise be, but since it's doing more I suppose that's
>>> to be expected.
>>>
>>> Adding it to core git would mean re-designing git's basic data
>>> model, which is obviously not something we're about to do on
>>> a whim.
>>
>> No, I'm NOT looking for a backup program. Every admin has the problem of
>> versioning config files (for example /etc). Versioning of config files
>> makes sense because one can track the changes and e.g. correlate to
>> problems. A backup program doesn't have features like history, committer
>> and comments on file changes. Therefore git would be a perfect tool also
>> for versioning configuration. (Software development doesn't end with the
>> build but typically also has deployment&configuration issues).
>>
>
> Right. So you want a *fancy* backup program and not just any random
> backup solution. There are solutions for this. Someone else already
> mentioned them elsewhere in this thread, so I'll refrain from further
> comments on this.
>
> In short: What you want can be (and has been) done, but it's written
> as addons and not integral parts of git.

A first quick look on them: They are only a workaround to the real 
problem. For example subversion therefore has generic properties which 
also can be user defined (e.g. svn propset/propget) and I think such a 
concept should also be integrated part of git.

Only my 2 cents making git (one of the SCM where I think it is very 
powerful and has also potential in the future) even better than 
competition.

Ciao,
Gerhard

--
http://www.wiesinger.com/

Re: Metadata and checkin file date

[Jakub Narebski]

On Tue, 27 Apr 2010, Gerhard Wiesinger wrote:
> On Tue, 27 Apr 2010, Jakub Narebski wrote:
>> Gerhard Wiesinger <lists@wiesinger.com> writes:
>>
>>> I'm new to git and I'm looking for the following features:
>>> 1.) Metadata for
>>>    a.) directory versioning (e.g. add/rm, mv)
>>>    b.) rights (basic: chmod, chow, chgrp, extended: extended
>>> attributes like ACLs and selinux), necessary for versioning e.g. /etc
>>> 2.) Original file dates (checkin date) on clone and pull (and not
>>> checkout date)
>>>
>>> Is this possible? Any plans if missing?
>>
>> Git is distributed version control system (DVCS), not a backup system.
>> It is used mainly for distributed development of programs.  Therefore
>> it supports natively only those parts of metadata that make sense for
>> VCS, namely symlinks (with workaround for filesystems that do not have
>> support for symbolic links) and the executable permission for files.
>>
>> File ownership does not make sense for VCS, as other people that clone
>> your repository do not have the same set of users that you have, and
>> might not have the same set of groups that you have.  Neverthemind that
>> their filesystem might not support notion of file ownership, not only
>> do not have support for extended attributes and the like.
> 
> I would suggest that only with special switches like --preserve-chmod 
> --preserver-owner --preserve-group , etc. where one can guarantee user, 
> groups, etc. See also my second post for arguments.

Do *one* thing, and do it well.

Note that in 'tree' object (used to represent directories, and holding
names and executable permission of files) there is simply no place for
full ownership, for timestamp(s), for extended attributes.  And changing
repository format for such minor fringe usage is out of the question for
Git.

So you would have to use some .gitmetadata file to store additional
metadata.  And then you can utilize hooks mechanism, i.e. add this
functionality to extra tool like etckeeper or IsiSetup, just like patch
management is done with extra tools like StGit, Guilt or TopGit.

>> If you really, really need this, you can use additional tools to
>> preserve metadata, like Metastore or git-cache-meta, or even ready
>> tools that use Git as bckend like IsiSetup or bup (well, bup use
>> git package format, not git itself...), see this Git Wiki page:
>> https://git.wiki.kernel.org/index.php/InterfacesFrontendsAndTools
> 
> Will have a look at them.

-- 
Jakub Narebski
Poland

Re: Metadata and checkin file date

[Jakub Narebski]

Andreas Ericsson <ae@op5.se> writes:
> On 04/27/2010 09:38 PM, Gerhard Wiesinger wrote:
>> On Tue, 27 Apr 2010, Andreas Ericsson wrote:
>>> On 04/27/2010 07:23 AM, Gerhard Wiesinger wrote:
>>>>
>>>> I'm new to git and I'm looking for the following features:
>>>> 1.) Metadata for
>>>> a.) directory versioning (e.g. add/rm, mv)
>>>
>>> If you're talking about empty directories, that feature doesn't
>>> exist and I can't imagine why you'd want it to. If you'd care to
>>> explain why you want it, I'm sure we can find a different way of
>>> achieving your goal.
>> 
>> Git focuses on content but I think git should also focus on metadata. 
>> For example restructuring source code moves (git mv file1.c file2.c, git 
>> mv dir1 dir2) should be documented also in the repository like e.g. 
>> subversion and commercial SCM like clearcase do. Otherwise we are on 
>> "CVS" level.

"git mv file1.c file2.c" works in practice thanks to heuristic
similarity based rename detection that git uses.  (Note that it might
not work in simplistic tests.)

"git mv dir1 dir2" works via _file_ rename detection, except for one
use case, namely when on one branch one does "git mv dir1 dir2", 
and in other branch one does "git add dir1/file.c" (new file in old
directory name).  See also below.

>> Empty directories is a special case and sometimes you need just 
>> versioned empty directies.
> 
> This has been discussed to death several times before on this mailing
> list. Browse the archives. There haven't been any new arguments the
> last 14 times it came up, so I doubt you'll be able to come up with
> a single good reason to track file renames explicitly.

There are three issues conflated here:

1. Rename tracking.  This is no go.  The only idea that have hint of
   being perhaps possibly accepted is recording resolutions of
   tree-level conflict in git-rerere2 cache.

2. Wholesame directory rename detection.  Possible, there were even
   some proof of concept patches send to git mailing list, but it
   didn't get merged in.  Would need to be resurrected, but this is
   not easy task.

3. Support for versioning empty directories.  Possible, but it would
   require extending index (the staging area one) to be able to hold
   not only files but also directories.  Not very easy.

   Current workaround: empty .gitignore / .keepme files.

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: Metadata and checkin file date

[Jakub Narebski]

Gerhard Wiesinger <lists@wiesinger.com> writes:
> On Tue, 27 Apr 2010, Andreas Ericsson wrote:
>> On 04/27/2010 09:38 PM, Gerhard Wiesinger wrote:
>>> On Tue, 27 Apr 2010, Andreas Ericsson wrote:
>>>> On 04/27/2010 07:23 AM, Gerhard Wiesinger wrote:

>>>>> b.) rights (basic: chmod, chow, chgrp, extended: extended attributes
>>>>> like ACLs and selinux), necessary for versioning e.g. /etc
>>>>
>>>> Sounds like you want a backup-program. Some projects have been
>>>> aimed towards this goal already. [...]. AFAIR, most of them work
>>>> with two hook-scripts that update a regular file with the
>>>> meta-data of all tracked files. [...]
>>>>
>>>> Adding it to core git would mean re-designing git's basic data
>>>> model, which is obviously not something we're about to do on
>>>> a whim.
>>>
>>> No, I'm NOT looking for a backup program. Every admin has the problem of
>>> versioning config files (for example /etc). Versioning of config files
>>> makes sense because one can track the changes and e.g. correlate to
>>> problems. A backup program doesn't have features like history, committer
>>> and comments on file changes. Therefore git would be a perfect tool also
>>> for versioning configuration. (Software development doesn't end with the
>>> build but typically also has deployment&configuration issues).

See IsiSetup, etckeeper and other such tools.

[...]
>> In short: What you want can be (and has been) done, but it's written
>> as addons and not integral parts of git.
> 
> A first quick look on them: They are only a workaround to the real
> problem. For example subversion therefore has generic properties which
> also can be user defined (e.g. svn propset/propget) and I think such a
> concept should also be integrated part of git.
> 
> Only my 2 cents making git (one of the SCM where I think it is very
> powerful and has also potential in the future) even better than
> competition.

Actually using *properties* is one of MIS-designs of Subversion.
Storing generic sh*t in repository is not a good idea either.

-- 
Jakub Narebski
Poland
ShadeHawk on #git