The philosophy behind my directory proposal in a nutshell

The philosophy behind my directory proposal in a nutshell

[David Kastrup]

Ok, a big nut.

Many people have lost track since I developed the idea over dozens of
postings and so I have been explaining only details instead of the
actual picture.

Let us assume we habe the following files in the work tree (there is a
reason for starting off with / always in this discussion):

/i/never/saw/this
/i/never/should/have
/i/would/have

Now it is obvious that we _also_ have the directories (which I'll call
by their proposed git name):

/i/never/saw/.
/i/never/should/.
/i/never/.
/i/would/.
/i/.
/.

Git does right now not track any of them.  It _induces_ their
existence through trees in the repository.  Let us call the _trees_ in
the following way:

/i/never/saw/
/i/never/should/
/i/never/
/i/would/
/i/
/

[The last line here is the reason we prepended / everywhere: it makes
the root tree more apparent].

All those _trees_ are just string prefixes of file names, and those
prefixes are without a physical existence.  But if we check out the
files, we need to create directories to accommodate them.  And for
efficiency reasons, we call every prefix occuring in the flat file
list ending in / a "tree" and give it an SHA1 value.

Now when do we remove a directory that is in the work filesystem?
When the corresponding tree (=prefix) in the repository becomes empty,
namely the prefix becomes unused, the tree evaporates.  At that
moment, git has lost the necessity for keeping the directory around,
and attempts to remove it.  If the attempt does not succeed, something
outside of git is using the directory.  At this point removing the
directory becomes somebody else's problem: git will not try again.
The tree is gone from the repository, so there is nothing that makes
git even _think_ about that directory.

So far, so well.  Quite consistent, in some cases quite convenient,
but not in all.  There are situations where empty directories matter.

So we want git to be able to track them.  Let us take a simple example
first:

mkdir a
mkdir a/b
touch a/b/c
git add a/b/c
git commit
git rm a/b/c
git commit

The naive expectation would be that the directories made by the user
will not get touched by git.  Trying to implement anything like a "git
removes only directories it created itself" would be insane, and here
is why: the user actually has no _chance_ not to create those
directories before checking the files in.  And there is no way to
deduce whether he did it on behalf of git or because of some other
reason.  Also it is not important _how_ the directory got created: git
tracks states, not responsibilities.

So this case _must_ behave the same: after the first commit, we have
/a/b/c in the repository as the only file, and it induces the
existence of the trees

/a/b
/a
/

So when we remove /a/b first /a collapses and evaporates (if a tree
falls in the repository and there is nobody to hear it...), then /
collapses but is kept alive because we can't remove our current work
directory, not even if the repository is kept elsewhere (and actually,
/ _is_ special after all).

This behavior _must_ stay the same, regardless of what settings we
use.  Now let us assume that git can be told to track directories, and
that the user wants to keep a around even if empty, but not
necessarily b.  How can he achieve this?
mkdir a
git-add a
mkdir a/b
touch a/b/c
git-add a/b/c
git-commit
git-rm a/b/c
git-commit

Now directories come into play.  After the first git-add, the
repository contains the following non-volatile stuff:

/a/.

and in consequence it contains the trees

/a/
/

as well.  The second add adds /a/b/c but nothing else, so we have

/a/.
/a/b/c

and the trees

/a/b/
/a/
/

Ok, so now do the git-rm and following commit.  The tree then contains

/a/.

and /a/b/ collapses while /a/ and / are held up by /a/. perfectly
well.  The collapse of /a/b/ leads to the deletion of the
corresponding directory in the work tree (so the collapse of a tree in
the repository _does_ make a sound in the work directory as long as
somebody is there to hear it).

    [As a note aside: working directories get removed when their
    corresponding tree collapses and disappears from the
    repository.  I should think it only consistent if the same
    happened with git-rm: let the file by default disappear at
    the moment when it gets removed from the repository,
    implying --cached.  Namely, let

    git-rm some-file
    git-add some-file

    be a noop, regardless whether the file was registered
    previously.  This also implies that git-rm will always leave
    unregistered files alone.]

So when directory tracking is on, basically whenever adding
a directory explicitly by naming it or a parent directory in
the add or commit, the corresponding entry ending in "/."
gets added and keeps the tree alive even if no other file
does.  And while the tree is alive, git does not touch the
directory in the work tree.

So that are the basics for more or less manually adding and
removing directories when they are being tracked.  How do we
let git decide whether or not to track directories?  It
never tracks them if our git-add commands only mention
files: in that case, the behavior is indistinguishable from
previously.  But if we use git-add to add directories, then
not just files, but also directories make it to the index
and repository.

Of course, we can have a configuration variable to set
this.  But this is not enough.  Fortunately, all directories
carry a name ending with "/." in the repository, and thus we
can use the gitignore mechanism to make git ignore
directories.

Some commands accept "ignore this" or "don't ignore this" on
the command line, this takes highest priority.

Then .gitignore files come to play in the directory
hierarchy: those make it easy to exclude or include any
subtree in directory tracking by specifying the pattern "."
or "!.".  Then $GIT_DIR/info/exclude takes effect in a
similar way, another level lower core.excludesfile is
consulted, and at the very lowest level, if nobody else
cares (unlikely that a project would not set forth a policy
in $GIT_DIR/info/exclude), the users preference variable
core.trackdirs takes effect, defaulting to false for
compatibility until all projects wanting to keep directories
out of their repository have made their wish explicit in
their configuration.

I hope the concept now is somewhat clearer.

Now how does git _exactly_ deal with ignored files?  Ignored
files basically means that git skips them when they get
added (unless you use -f).  However, it _does_ notice when
they get deleted and updates the repository correspondingly.

What does this mean for interoperability of users with
different preferences of directory tracking?  First assume
that both have git versions capable of tracking directories.
One ignores ".", the other doesn't.  "." entries will in
this case be removed when the non-tracker commits a
directory with -a where the physical directory has
disappeared (and thus also all files beneath that).  That's
reasonably harmless.  "." entries will be added for a
directory and its subdirectories when the tracking person
commits a directory.

No problems here, little cause for violence.  Now what
happens with a "legacy" git?  The legacy git _never_ sees
anything corresponding to the weird directory entries in the
file system.  So with every commit of a tree with -a, the
tree gets "corrected" by removing the "strange" entries.
They will get added again when somebody else commits, but in
the mean time, empty directories might disappear.  If that
is a problem, you should not have given a person using an
old git commit access.  Could be worse.

So that basically wraps this up.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

git-rm semantics (was: The philosophy behind my directory proposal in a nutshell)

[David Kastrup]

Following up on the git-rm note:

>     [As a note aside: working directories get removed when their
>     corresponding tree collapses and disappears from the
>     repository.  I should think it only consistent if the same
>     happened with git-rm: let the file by default disappear at
>     the moment when it gets removed from the repository,
>     implying --cached.  Namely, let
>
>     git-rm some-file
>     git-add some-file
>
>     be a noop, regardless whether the file was registered
>     previously.  This also implies that git-rm will always leave
>     unregistered files alone.]

Ok, this is somewhat incorrect: if we call git-rm and the file still
exists, this is strictly speaking the same situation as calling
git-add when a file does not exist.

And that gives:
fatal: pathspec 'geo' did not match any files
whether or not the repository has an idea about the file.

So in analogy, git-rm on an existing file could possibly made to barf.

Anyway, the following appears inconsistent to me:

touch geo
git-add geo
git-commit -m xxx -a
git-rm geo
[Now geo is gone]
git-add geo
[geo is complained about as unknown]
git-checkout geo
[geo is complained about as unknown!]

Now the last complaint clearly is quite a nuisance, because
git-checkout .
_will_ resurrect geo.

So for consistency's sake, it would appear that git-rm should really
only schedule a removal, and pull through only at the moment where the
file is actually removed from the repository.  There is a slight
discordance with git-add in that git-add requires the existence of the
file at the time of the add.  That is inevitable since git records the
_contents_ of the file at the time of the add.  git-rm does not need
any such information.

Now what if a file disappears before we commit, namely
touch woozle
git add woozle
rm woozle
?

Woozle will in that case get committed if and only if the commit is
done without the -a option.

Makes sense.  So in a nutshell: git-rm should not look at the working
directory.  That's the business of the commit.  When the repository
copy gets removed, this is the right time to remove the corresponding
file, and this is perfectly equivalent with the case for trees.

What _is_ strange is that a git-rmed file can't be restored by naming
it for a checkout.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

A simpler approach to tracking directories (was: The philosophy behind my directory proposal in a nutshell)

[David Kastrup]

Ok, I've gotten enough flak for my first proposal that it is clear
that it is rather irrelevant whether it would work (of which I remain
convinced) or not.  Now I also have been of the opinion that it was
perfectly easy to understand and embrace, and the amount, content and
tone of responses have made very obvious that I have been quite wrong
about _that_ at least.

The good thing is that among all the flamage there have been actually
some people who went to the pain of actually reviewing the proposed
changes and give feedback about their relative feasibility in the
current code base.  This has helped more than the reflections on my
sanity, I am glad to say.

So here we go:

I think there has been some vague consensus with the notion that it
does not make sense to track whether or not a directory _is_ empty at
the time of checkin (when checking in files below it, it _can't_ be),
but rather whether it should be automatically added and removed by git
based on the files in it alone.

The current state of affairs with git is that directories are
exclusively a mechanism for accommodating hierarchical filenames.
Consequently, the index does not need to know about them.

Anyway, here is the basic idea: git is already capable of tracking
file permissions (though at the current point of time, this is
somewhat artificially limited to the u+x bit), so there are basic
mechanisms in place for passing the respective bits through.

Now my proposal basically boils down to using the u+x bit on
directory/tree entries for tracking "keep around when empty".

I think we are on the safe side to assume a directory with access
permissions zero (a-rwx) is something we never want to be able to
track with git.  So this special value, which I think is the current
default, can express "delete when empty".  Personally I believe that
git can be made without problems to track more permissions (possibly
by configuring an appropriate mask for the project) than it currently
does, but that's not relevant for now: it is sufficient to use
permissions a-rwx for an automatically managed (=evaporate when empty)
tree/directory, and 755 for a manually managed (=stay around)
tree/directory without actually calling lstat for now.

Apart from needing to accommodate directories in the index (since they
are now associated with 1 instead of 0 bits of information), this will
work with the current data structures and protocols.  Patches and
diffs can encode the change between tracked and untracked directories
as a change in the permissions of the directory.

So this obliterates the need to distinguish trees from directories
using actually separate and separately listed repository entries,
hopefully removing the main stumbling block.

It also addresses the concern of a tree's SHA1 changing when its
evaporate-when-empty-ness changes.  It also will not lead to dangerous
situations on systems attempting to replace or delete "." as a regular
file.

So it has several technical advantages over my previous proposal, and
I _think_ it may be easier to understand and lead to less
contraintuitive behavior.

Now for deciding about _when_ to track a directory or not, I think
that the following part of the previous proposal was workable:

A directory's permission should be tracked in the same situations that
a file's permissions get tracked: that is if either the directory has
_explicitly_ been added to a commit, or it is _implicitly_ part of a
complete tree added into a commit.  For implicitly indicated _files_,
whether or not they become tracked depends on the gitignore mechanism.
I think asking the gitignore mechanism whether the pattern "." in a
given directory is worth tracking is a good way of deciding implicit
inclusion, falling back onto a "core.dirtrack" or similar variable
that has some vague analogy to the "core.filemode" variable.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: A simpler approach to tracking directories (was: The philosophy behind my directory proposal in a nutshell)

[Theodore Tso]

On Sun, Jul 22, 2007 at 05:39:06PM +0200, David Kastrup wrote:
> Now my proposal basically boils down to using the u+x bit on
> directory/tree entries for tracking "keep around when empty".
> 
> I think we are on the safe side to assume a directory with access
> permissions zero (a-rwx) is something we never want to be able to
> track with git.

Ok, so I think you are still missing the fundamental issue about why
Linus is upset with your proposals.  One of the fundamental things
which falls out of the "Git Tracks Contents" mantra is that
information which you expect to be pushed forward future revisions (as
opposed to metadata which is specific to a commit, such as the Author
and Committer of a patch, the Commit log, etc.) *MUST* be information
which is realized in the working tree.

That way, if you diff between working trees, one of which has either
your "." entry or your "u+x" bit, and of which doesn't, they need to
be *different* when run "git diff" on the two working trees.

Another way of putting is this single bit of information meaning "keep
this directory around when empty" is something that ***must*** be
representable in three different places: in a git tree object, in the
index, *AND* in the working tree.  The problem with "." is that since
all directories contain the "." entry, you can't represent the bit of
information you want to record in the working tree in that way.

There is a same problem with using the u+x permission, for the same
reason.  Unless you want to make directories you want to keep around
with access permissions of zero in the working tree, you're *still*
not able to record presence or absence of the "keep around when empty
bit" in the working tree.

That's why the ".gitignore" entry is acceptable, where as your
proposals are not.  The absence or presence of the ".gitignore" entry
in the working tree is a natural way of representing the "keep around
when empty".  So would a ".do_not_erase" file in the directory ---
sort of like the "Do Not Erase" that professors would write on
whiteboards in order to request janitors to not erase them.  It seems
that you are objecting to having something tangible written on the
whiteboard --- or, in the directory, to indicate the "do not erase"
note.   

But the problem is that Linus very much wants the "do not erase" bit
to be visible in the working directory, since to do otherwise would
violate fundamental design assumptions all over the git source files.
Metadata such as the executable bit is actually stored in the working
directory.  But we can't store the "do not erase bit" by leaving the
executable permission off, since that will prevent the directory from
being useful.

So your new proposal suffers from the same fundamental flaw as your
previous one.  

Maybe you disagree with Linus's design constraint, but you've never
addressed his specific concern on-point, which is that since Git
Tracks Content, if you want something to be tracked across revisions,
it must be visible in the working tree.  That is, metadata in the
sense of data which is not visible in the working tree, is not allowed
to exist inside Git and carried across revisions.  So fundamentally,
Git does not currently today support "svn propset" in terms of setting
metadata on a particular file which isn't visible in the working tree.

With SVN, today you can use svn propset to what you could think of as
extended attributes.  So one proposal that you *could* try proposing
is using extended attributes to represent arbitrary properties,
including the one which you want, which is the "do not erase"
property.  And one of the arguments might be that this way we could
better preserve arbitrary properties currently set inside SVN, so that
such repository could be faithfully translated into a Git repository.  

The problem with such a proposal is that it now requires that the
filesystem used to store the working directory MUST support extended
attributes, and some filesystems, such as FAT filesystems, do not.
And Git already has been accused of not being Windows friendly enough,
and this would make things worse.

It's also a lot of hair for two very marginal features, namely being
able to support arbitrary SVN property values, and the "do not erase
when empty" directory bit.  Personally, I don't htink it's worth it,
but at least using filesystem xattrs to store that one bit of
information would at least be faithful to the fundamental git design
principal.

Regards,

						- Ted

Re: A simpler approach to tracking directories (was: The philosophy behind my directory proposal in a nutshell)

[Jakub Narebski]

Theodore Tso wrote:

> Another way of putting is this single bit of information meaning "keep
> this directory around when empty" is something that ***must*** be
> representable in three different places: in a git tree object, in the
> index, *AND* in the working tree.  The problem with "." is that since
> all directories contain the "." entry, you can't represent the bit of
> information you want to record in the working tree in that way.

I don't think the "do not delete when empty" bit must be somehow represented
in the working directory. It is enough to have this bit in index to deal
with working directory, and in the repository to have it persistent.

Compare with submodules, and with unmerged entries (which sometimes do not
have worktree representation).

-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: A simpler approach to tracking directories

[David Kastrup]

Theodore Tso <tytso@mit.edu> writes:

> On Sun, Jul 22, 2007 at 05:39:06PM +0200, David Kastrup wrote:
>> Now my proposal basically boils down to using the u+x bit on
>> directory/tree entries for tracking "keep around when empty".
>> 
>> I think we are on the safe side to assume a directory with access
>> permissions zero (a-rwx) is something we never want to be able to
>> track with git.
>
> Ok, so I think you are still missing the fundamental issue about why
> Linus is upset with your proposals.

Or I am not careful in communicating what I consider obvious in
addressing it.

> One of the fundamental things which falls out of the "Git Tracks
> Contents" mantra is that information which you expect to be pushed
> forward future revisions (as opposed to metadata which is specific
> to a commit, such as the Author and Committer of a patch, the Commit
> log, etc.) *MUST* be information which is realized in the working
> tree.

For every _file_ in the working tree, there is _one_ bit of
information in the repository that is not in the working tree: namely
whether git is tracking this file at all.

> That way, if you diff between working trees, one of which has either
> your "." entry or your "u+x" bit, and of which doesn't, they need to
> be *different* when run "git diff" on the two working trees.

If I have two identical working trees, and in one working tree a file
A has been added into git, and in the other it hasn't, then git diff
will report a difference.

> Another way of putting is this single bit of information meaning
> "keep this directory around when empty" is something that ***must***
> be representable in three different places: in a git tree object, in
> the index, *AND* in the working tree.  The problem with "." is that
> since all directories contain the "." entry, you can't represent the
> bit of information you want to record in the working tree in that
> way.

But if I have two identical working trees, where in one of them A has
been added into git and in the other it hasn't, there _is_ a bit of
difference in the repository.

I think the remaining problem boils down to the following: a file that
is not tracked by git has nothing in the repository which one would
associate with it.  That is easy to understand.  But trees in the
repository are right now a consequence of tracking files: a file needs
a tree, and git creates one.  So we have an object in the repository
which we naturally correlate with the directory in the working tree
that is required for holding the files in the working tree.  So the
relationship becomes somewhat less intuitive:

git tracks file -> file is in the repository
git does not track file -> file is not in the repository

That's easy.

git tracks directory -> tree permissions are not zero in the repository
git does not track directory -> no tree with non-zero permissions

So the problem in understanding is that in spite of git not tracking
the directory, there can be _still_ an _incidental_ tree whose
existence is solely to support the blobs in it.  This problem is
acerbated by git having to actually manipulate working file
directories even if it has no information about them.  So that git
_works_ with directories does not help to clarify that it does not
_track_ them.  At least one pointer that it does not track them,
however, is that they are not even entered into the index.  So since
no information flow into the repository happens, at the current point
of time, all directories can be considered untracked.  git _is_
accessing and manipulating them, but without any help from the
repository that would be specific to the directory.

> There is a same problem with using the u+x permission, for the same
> reason.  Unless you want to make directories you want to keep around
> with access permissions of zero in the working tree, you're *still*
> not able to record presence or absence of the "keep around when
> empty bit" in the working tree.

Like it is not encoded in the working tree whether or not git is
tracking a particular file.

> That's why the ".gitignore" entry is acceptable, where as your
> proposals are not.

But whether or not a file is tracked is not decided by a .gitignore
entry: gitignore comes into play only when one is adding entire
directories, and thus _implicitly_ adding files.

The explicit act or non-act of "git-add file" records a difference in
the repository that is nowhere reflected in the work directory.

> But the problem is that Linus very much wants the "do not erase" bit
> to be visible in the working directory, since to do otherwise would
> violate fundamental design assumptions all over the git source
> files.

Whether or not a file is tracked is _not_ encoded in the work
directory.  While the gitignore entries provide useful defaults, as
long as every git-add and git-rm is only naming single files, tracking
and not tracking remains a possibility for every single file, a
possibility nowhere reflected in the work directory.

> So your new proposal suffers from the same fundamental flaw as your
> previous one.

It pretty much behaves the same, yes.  After all, it has to represent
the difference between

mkdir a
touch a/b
git-add a/b
git-commit -m x
git-rm a/b
git-commit -m x

(where a should be removed) and

mkdir a
touch a/b
git-add a
git-commit -m x
git-rm a/b
git-commit -m x

(where a should be retained).  Feel free to address this.

> Maybe you disagree with Linus's design constraint, but you've never
> addressed his specific concern on-point, which is that since Git
> Tracks Content, if you want something to be tracked across
> revisions, it must be visible in the working tree.

Oh, I address it pretty much in every posting: whether or not a _file_
is being tracked is _not_ visible in the working directory, and nobody
sees a problem with that.  So this "design constraint" is very much
violated all the time, anyway.

> The problem with such a proposal is that it now requires that the
> filesystem used to store the working directory MUST support extended
> attributes, and some filesystems, such as FAT filesystems, do not.
> And Git already has been accused of not being Windows friendly
> enough, and this would make things worse.

Where is the relation to my proposal?  Am I asking for anything that
any file system could not represent?

> It's also a lot of hair for two very marginal features, namely being
> able to support arbitrary SVN property values, and the "do not erase
> when empty" directory bit.

Maybe we should call the bit by its meaning, not its effect.  Call it
the "git has been told to track this directory" bit.

Perhaps then it gets less offensive.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: A simpler approach to tracking directories

[J. Bruce Fields]

On Sun, Jul 22, 2007 at 08:45:37PM +0200, David Kastrup wrote:
> Theodore Tso <tytso@mit.edu> writes:
> > One of the fundamental things which falls out of the "Git Tracks
> > Contents" mantra is that information which you expect to be pushed
> > forward future revisions (as opposed to metadata which is specific
> > to a commit, such as the Author and Committer of a patch, the Commit
> > log, etc.) *MUST* be information which is realized in the working
> > tree.
> 
> For every _file_ in the working tree, there is _one_ bit of
> information in the repository that is not in the working tree: namely
> whether git is tracking this file at all.

Actually, no.  The index just stores data for a given set of files.  You
can think of the set of files for which data is stored in the index as
the set that is "tracked", but there's no independent
"tracked/untracked" bit, and no way of marking a path as tracked without
also adding content at that path.

--b.