RFC: working with sparse work-copy: where to start?

RFC: working with sparse work-copy: where to start?

[Michael S. Tsirkin]

Hi!
I've been thinking about working in a sparsely-populated work copy.
E.g. svn will let you do a checkout of a specific sub-directory
and do all your development there assuming you only care about
changes in that directory. Ideally, it would be possible to specify
a set of directories that I care about.

git does close to what I want if I do git-clone -n and then
git-checkout <subdirectory>.

However,
1. This re-creates the same directory structure that the repository has.
   Especially for when this directory resides deep in the original repo
   tree structure, it would be nice to be able relocate the directory anywhere.
2. The rest of the tree is checked-out whenever I pull from the central repo.
3. git-commit would list all files I didn't check out as removed.

I wonder how hard would it be to add such support?
I tried looking at 2 - is builtin-merge-file.c the only place where the checkout
is done on merge?

What other commands need to be fixed for this to work?

-- 
MST

Re: RFC: working with sparse work-copy: where to start?

[Junio C Hamano]

"Michael S. Tsirkin" <mst@mellanox.co.il> writes:

> Hi!
> I've been thinking about working in a sparsely-populated work copy.
> E.g. svn will let you do a checkout of a specific sub-directory
> and do all your development there assuming you only care about
> changes in that directory. Ideally, it would be possible to specify
> a set of directories that I care about.
> ...
> What other commands need to be fixed for this to work?

Pretty much everything needs to be changed, I suspect.

The point of git is to track the whole tree, and that is
ingrained to the design of the tool everywhere.  You need to
bend backwards to operate in the way you propose; I am not
saying it is impossible but I suspect it will be very hard or
inefficient or both.

Your commits, although your working tree will only have selected
subtrees in "wrong" places, should have the same "directory
structure" as everybody else.  This is an absolute requirement
for the result of your work in your repository to be usable by
others.

One design level thing you need to worry about is what to do
with merges, if you are pulling from more than one people and
they start their work in different version of "the whole tree".
You may not be interested in what happens in "other parts" of
the whole tree you are not checking out.  For your merge result
to be any useful for other people you later want to merge from
you, you still have to come up with and record the sensible
result when these "other parts" have conflicts.

This means the tree objects you will create as the final product
has to have the same directory structure as other people, while
your working tree will have only selected things, and in "wrong"
(from the tree's perspective) places.

One decision you need to make early is which side you place your
index on.

One approach would be to have the same directory structure in
the index.  The other possibility is to match the directory
structure recorded in the index with the structure of your
working tree.  I am not sure pros-and-cons offhand.

Regardless of which avenue you pursue, you need to have a way to
tell your version of git which subtree your repository is
interested in, and where in your working tree you want to check
them out.  And you would need to change affected commands to
take this new information into account when do their work.

The former approach means you do not have to touch write-tree
and read-tree, but your diff-index, diff-tree (when you want it
to work sensibly inside subdirectories), checkout-index and
update-index need to know about this.  The latter means you do
not have to touch diff-files, update-index and checkout-index,
but read-tree will be affected.  Any command that deals with
both tree and working directory at the same time (diff-index,
read-tree -u, merge-recursive, etc.) or works from a
subdirectory (pretty much all Porcelain commands -- think of
'git add .' in a subdirectory) will need to change regardless of
which approach is taken.

I think it would be a lot simpler if you do not mind deep
directory structure but still support a sparse checkout,
though.

Re: RFC: working with sparse work-copy: where to start?

[Michael S. Tsirkin]

> > Hi!
> > I've been thinking about working in a sparsely-populated work copy.
> > E.g. svn will let you do a checkout of a specific sub-directory
> > and do all your development there assuming you only care about
> > changes in that directory. Ideally, it would be possible to specify
> > a set of directories that I care about.
> > ...
> > What other commands need to be fixed for this to work?
> 
> Pretty much everything needs to be changed, I suspect.
> 
> The point of git is to track the whole tree, and that is
> ingrained to the design of the tool everywhere.  You need to
> bend backwards to operate in the way you propose; I am not
> saying it is impossible but I suspect it will be very hard or
> inefficient or both.
> 
> Your commits, although your working tree will only have selected
> subtrees in "wrong" places, should have the same "directory
> structure" as everybody else.  This is an absolute requirement
> for the result of your work in your repository to be usable by
> others.
> 
> One design level thing you need to worry about is what to do
> with merges, if you are pulling from more than one people and
> they start their work in different version of "the whole tree".
> You may not be interested in what happens in "other parts" of
> the whole tree you are not checking out.  For your merge result
> to be any useful for other people you later want to merge from
> you, you still have to come up with and record the sensible
> result when these "other parts" have conflicts.

Yes, I've been thinking about this problem. As far as the merge can be
done in-index it should be fine though. Maybe failing a merge if there are
merges that can't be done with the checked out subtree is the right thing to do.
But, this means that we should try harder to merge in-index
changes that touch non-checked-out files.

For example, a commit could touch a checked-out and non-checked out file,
and non-checked-out file could be index-mergeable but checked-out one
would need file level merge. Yes?

> This means the tree objects you will create as the final product
> has to have the same directory structure as other people, while
> your working tree will have only selected things, and in "wrong"
> (from the tree's perspective) places.
> 
> One decision you need to make early is which side you place your
> index on.
> 
> One approach would be to have the same directory structure in
> the index.  The other possibility is to match the directory
> structure recorded in the index with the structure of your
> working tree.  I am not sure pros-and-cons offhand.
> 
> Regardless of which avenue you pursue, you need to have a way to
> tell your version of git which subtree your repository is
> interested in, and where in your working tree you want to check
> them out.  And you would need to change affected commands to
> take this new information into account when do their work.
> 
> The former approach means you do not have to touch write-tree
> and read-tree, but your diff-index, diff-tree (when you want it
> to work sensibly inside subdirectories), checkout-index and
> update-index need to know about this.  The latter means you do
> not have to touch diff-files, update-index and checkout-index,
> but read-tree will be affected.  Any command that deals with
> both tree and working directory at the same time (diff-index,
> read-tree -u, merge-recursive, etc.) or works from a
> subdirectory (pretty much all Porcelain commands -- think of
> 'git add .' in a subdirectory) will need to change regardless of
> which approach is taken.
> 
> I think it would be a lot simpler if you do not mind deep
> directory structure but still support a sparse checkout,
> though.

OK, I don't mind the deep directory structure so much.

I'm sorry, Junio, but I'm a bit lost wrt which of your comments
are related to the directory structure change idea
(which it seems wise to drop) and which to the sparse checkout
support which might be more doable. Could you help please?

-- 
MST