[RFC PATCH 00/12] Sparse checkout

[RFC PATCH 00/12] Sparse checkout

[Nguyễn Thái Ngọc Duy]

I have not looked at non-builtin commands yet, but I think it's not
a big deal. We have several rounds before this series is good enough ;)
So in short, sparse prefix will be stored in config, core.sparsecheckout.
you have three new commands to enter/update/leave sparse checkout:

git clone --path=prefix       # clone with sparse checkout
git checkout --path=prefix    # limit/update checkout paths
git checkout --full           # stop sparse checkout

Other operations should be normal. User attempts to do anything outside
sparse checkout will get flagged. Git itself should not touch anything
outside sparse checkout.

One more thing. As files outside sparse checkout will not be checking
out, .gitignore and .gitattributes from parent directories (outside
sparse checkout) will be gone too. This may lead to surprise.

Comments are welcome.

Nguyễn Thái Ngọc Duy (12):
  git-grep: read config
  git-grep: support --no-external-grep
  Introduce sparse prefix
  Protect index with sparse prefix
  tests for sparse checkout, index protection
  Avoid accessing working directory outside sparse prefix
  tests for sparse checkout, worktree protection
  git-clone: support --path to do sparse clone
  tests for sparse clone
  git-checkout: support --full and --path to manipulate sparse checkout
  tests for checkout [--full|--path]
  git-status: print sparse checkout status

 builtin-add.c                              |    4 +-
 builtin-apply.c                            |    5 +
 builtin-checkout-index.c                   |    2 +
 builtin-checkout.c                         |   46 ++++++-
 builtin-clean.c                            |    3 +
 builtin-clone.c                            |   13 ++
 builtin-commit.c                           |   13 ++-
 builtin-grep.c                             |   36 +++--
 builtin-ls-files.c                         |    6 +-
 builtin-ls-tree.c                          |    2 +-
 builtin-merge-recursive.c                  |    6 +-
 builtin-mv.c                               |    2 +-
 builtin-read-tree.c                        |    5 +
 builtin-rev-parse.c                        |    4 +
 builtin-rm.c                               |    7 +-
 builtin-update-index.c                     |   37 ++++-
 cache.h                                    |   19 +++
 config.c                                   |    7 +
 diff-lib.c                                 |    5 +-
 diff.c                                     |    3 +-
 entry.c                                    |    3 +
 environment.c                              |  221 ++++++++++++++++++++++++++
 read-cache.c                               |  112 +++++++++++++-
 setup.c                                    |   90 +++++++++++-
 sha1_file.c                                |    3 +
 t/t2010-checkout-sparse.sh                 |   71 +++++++++
 t/t2300-sparse-index.sh                    |  156 +++++++++++++++++++
 t/t2300/diff-index-sub.expected            |    3 +
 t/t2300/log.expected                       |   64 ++++++++
 t/t2301-sparse-index-merge-recursive.sh    |  226 +++++++++++++++++++++++++++
 t/t2302-sparse-worktree.sh                 |  113 ++++++++++++++
 t/t2302/add-u.expected                     |    1 +
 t/t2302/add.expected                       |    2 +
 t/t2302/commit.expected                    |   14 ++
 t/t2302/diff-1.expected                    |    7 +
 t/t2302/diff-cc.expected                   |    9 +
 t/t2302/grep-work.expected                 |    2 +
 t/t2302/grep.expected                      |    2 +
 t/t2302/ls-files.expected                  |    2 +
 t/t2303-sparse-worktree-apply.sh           |   62 ++++++++
 t/t2303/apply-initial.patch                |   14 ++
 t/t2303/apply-inside.patch                 |    7 +
 t/t2303/apply-leading-dirs.patch           |    3 +
 t/t2303/apply-outside.patch                |    7 +
 t/t2303/apply-remove.patch                 |    7 +
 t/t2303/apply-rename.expected              |   13 ++
 t/t2303/apply-rename.patch                 |    4 +
 t/t2304-sparse-worktree-merge-recursive.sh |  233 ++++++++++++++++++++++++++++
 t/t5703-clone-sparse.sh                    |   40 +++++
 unpack-trees.c                             |  152 +++++++++++++++++--
 unpack-trees.h                             |    4 +-
 wt-status.c                                |   12 +-
 52 files changed, 1831 insertions(+), 53 deletions(-)
 create mode 100755 t/t2010-checkout-sparse.sh
 create mode 100755 t/t2300-sparse-index.sh
 create mode 100644 t/t2300/diff-index-sub.expected
 create mode 100644 t/t2300/log.expected
 create mode 100755 t/t2301-sparse-index-merge-recursive.sh
 create mode 100755 t/t2302-sparse-worktree.sh
 create mode 100644 t/t2302/add-u.expected
 create mode 100644 t/t2302/add.expected
 create mode 100644 t/t2302/commit.expected
 create mode 100644 t/t2302/diff-1.expected
 create mode 100644 t/t2302/diff-cc.expected
 create mode 100644 t/t2302/grep-work.expected
 create mode 100644 t/t2302/grep.expected
 create mode 100644 t/t2302/ls-files.expected
 create mode 100755 t/t2303-sparse-worktree-apply.sh
 create mode 100644 t/t2303/apply-initial.patch
 create mode 100644 t/t2303/apply-inside.patch
 create mode 100644 t/t2303/apply-leading-dirs.patch
 create mode 100644 t/t2303/apply-outside.patch
 create mode 100644 t/t2303/apply-remove.patch
 create mode 100644 t/t2303/apply-rename.expected
 create mode 100644 t/t2303/apply-rename.patch
 create mode 100755 t/t2304-sparse-worktree-merge-recursive.sh
 create mode 100755 t/t5703-clone-sparse.sh

Re: [RFC PATCH 00/12] Sparse checkout

[Nguyen Thai Ngoc Duy]

Sorry list. Forgot --thread in format-patch :(
-- 
Duy

Re: [RFC PATCH 00/12] Sparse checkout

[Johannes Schindelin]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 559 bytes --]

Hi,

On Wed, 23 Jul 2008, Nguyễn Thái Ngọc Duy wrote:

> So in short, sparse prefix will be stored in config, core.sparsecheckout.

Do you really think the prefix should be stored anywhere else than the 
index?

With core.sparseCheckout you have to introduce a _sh*tload_ of config 
loaders.

And with core.sparseCheckout you are at the whim of the user, since 
.git/config is _supposed_ to be user-editable.

>From a logical point of view, I'd say that the sparse prefix has nothing 
to do with the "configuration" of the local repository.

Ciao,
Dscho

Re: [RFC PATCH 00/12] Sparse checkout

[Nguyen Thai Ngoc Duy]

On 7/23/08, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> Hi,
>
>
>  On Wed, 23 Jul 2008, Nguyễn Thái Ngọc Duy wrote:
>
>  > So in short, sparse prefix will be stored in config, core.sparsecheckout.
>
>
> Do you really think the prefix should be stored anywhere else than the
>  index?
>
>  With core.sparseCheckout you have to introduce a _sh*tload_ of config
>  loaders.
>
>  And with core.sparseCheckout you are at the whim of the user, since
>  .git/config is _supposed_ to be user-editable.
>
>  From a logical point of view, I'd say that the sparse prefix has nothing
>  to do with the "configuration" of the local repository.

Well, whatever place. I chose .git/config because I did not want to
introduce a new config place. But then how about .git/sparsecheckout?

>
>  Ciao,
>  Dscho
>


-- 
Duy

Re: [RFC PATCH 00/12] Sparse checkout

[Johannes Schindelin]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1185 bytes --]

Hi,

On Wed, 23 Jul 2008, Nguyen Thai Ngoc Duy wrote:

> On 7/23/08, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
>
> >  On Wed, 23 Jul 2008, Nguyễn Thái Ngọc Duy wrote:
> >
> >  > So in short, sparse prefix will be stored in config, 
> >  > core.sparsecheckout.
> >
> > Do you really think the prefix should be stored anywhere else than the 
> > index?
> >
> > With core.sparseCheckout you have to introduce a _sh*tload_ of config 
> > loaders.
> >
> > And with core.sparseCheckout you are at the whim of the user, since 
> > .git/config is _supposed_ to be user-editable.
> >
> > From a logical point of view, I'd say that the sparse prefix has 
> > nothing to do with the "configuration" of the local repository.
> 
> Well, whatever place. I chose .git/config because I did not want to 
> introduce a new config place. But then how about .git/sparsecheckout?

No, I did mean the index.  This is an attribute of the index: either it is 
sparsely checked out or not.  You can even have multiple indices 
(switching between them by setting GIT_INDEX_FILE) which have different 
prefixes.

Ciao,
Dscho "who seems to recall that the first series was much less intrusive"

Re: [RFC PATCH 00/12] Sparse checkout

[Nguyen Thai Ngoc Duy]

On 7/23/08, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> Hi,
>
>
>  On Wed, 23 Jul 2008, Nguyen Thai Ngoc Duy wrote:
>
>  > On 7/23/08, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
>  >
>
> > >  On Wed, 23 Jul 2008, Nguyen Thai Ngoc Duy wrote:
>  > >
>  > >  > So in short, sparse prefix will be stored in config,
>  > >  > core.sparsecheckout.
>  > >
>  > > Do you really think the prefix should be stored anywhere else than the
>  > > index?
>  > >
>  > > With core.sparseCheckout you have to introduce a _sh*tload_ of config
>  > > loaders.
>  > >
>  > > And with core.sparseCheckout you are at the whim of the user, since
>  > > .git/config is _supposed_ to be user-editable.
>  > >
>  > > From a logical point of view, I'd say that the sparse prefix has
>  > > nothing to do with the "configuration" of the local repository.
>  >
>  > Well, whatever place. I chose .git/config because I did not want to
>  > introduce a new config place. But then how about .git/sparsecheckout?
>
>
> No, I did mean the index.  This is an attribute of the index: either it is
>  sparsely checked out or not.  You can even have multiple indices
>  (switching between them by setting GIT_INDEX_FILE) which have different
>  prefixes.

I don't think so. It's a mask for workdir, right? If you save it it
index, you can switch index and the prefix as well, but workdir only
has several subtrees that do not fit any other prefix than the
original prefix.

>  Ciao,
>  Dscho "who seems to recall that the first series was much less intrusive"

The first series only took care of index, so yes it's simpler.
-- 
Duy

Re: [RFC PATCH 00/12] Sparse checkout

[Johannes Schindelin]

Hi,

On Thu, 24 Jul 2008, Nguyen Thai Ngoc Duy wrote:

> On 7/23/08, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
>
> >  On Wed, 23 Jul 2008, Nguyen Thai Ngoc Duy wrote:
> >
> >  > On 7/23/08, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> >  >
> >  > > And with core.sparseCheckout you are at the whim of the user, 
> >  > > since .git/config is _supposed_ to be user-editable.
> >  >
> >  > Well, whatever place. I chose .git/config because I did not want to 
> >  > introduce a new config place. But then how about 
> >  > .git/sparsecheckout?
> >
> > No, I did mean the index.  This is an attribute of the index: either 
> > it is sparsely checked out or not.  You can even have multiple indices 
> > (switching between them by setting GIT_INDEX_FILE) which have 
> > different prefixes.
> 
> I don't think so. It's a mask for workdir, right? If you save it it 
> index, you can switch index and the prefix as well, but workdir only has 
> several subtrees that do not fit any other prefix than the original 
> prefix.

Ah, you adroitly avoided addressing the issue that the user can change the 
prefix without the index ever noticing.

Well, in any case, if you do not agree that the prefix is really an 
attribute of the index, then there is nothing else I have to say on this 
series.

Ciao,
Dscho

Re: [RFC PATCH 00/12] Sparse checkout

[Nguyen Thai Ngoc Duy]

Hi,

On 7/24/08, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
>  > > No, I did mean the index.  This is an attribute of the index: either
>  > > it is sparsely checked out or not.  You can even have multiple indices
>  > > (switching between them by setting GIT_INDEX_FILE) which have
>  > > different prefixes.
>  >
>  > I don't think so. It's a mask for workdir, right? If you save it it
>  > index, you can switch index and the prefix as well, but workdir only has
>  > several subtrees that do not fit any other prefix than the original
>  > prefix.
>
>
> Ah, you adroitly avoided addressing the issue that the user can change the
>  prefix without the index ever noticing.

Forgive my ignorance. I still do not get why index must notice prefix
change? The only reason I can think of is that we must make sure there
won't be any user-modification in index outside the prefix. But that
can be guarded from higher level (plumbings and porcelains) because
index is allowed to have modification outside sparse prefix
(auto-merged entries).
-- 
Duy

Re: [RFC PATCH 00/12] Sparse checkout

[Johannes Schindelin]

Hi,

On Thu, 24 Jul 2008, Nguyen Thai Ngoc Duy wrote:

> On 7/24/08, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> >  > > No, I did mean the index.  This is an attribute of the index: 
> >  > > either it is sparsely checked out or not.  You can even have 
> >  > > multiple indices (switching between them by setting 
> >  > > GIT_INDEX_FILE) which have different prefixes.
> >  >
> >  > I don't think so. It's a mask for workdir, right? If you save it it 
> >  > index, you can switch index and the prefix as well, but workdir 
> >  > only has several subtrees that do not fit any other prefix than the 
> >  > original prefix.
> >
> >
> > Ah, you adroitly avoided addressing the issue that the user can change 
> > the prefix without the index ever noticing.
> 
> Forgive my ignorance. I still do not get why index must notice prefix 
> change? The only reason I can think of is that we must make sure there 
> won't be any user-modification in index outside the prefix. But that can 
> be guarded from higher level (plumbings and porcelains) because index is 
> allowed to have modification outside sparse prefix (auto-merged 
> entries).

Why do you want to guard it from the outside?  When the obvious fix is to 
put together what belongs together?

Hth,
Dscho

Re: [RFC PATCH 00/12] Sparse checkout

[Nguyen Thai Ngoc Duy]

On 7/24/08, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> Hi,
>
>  On Thu, 24 Jul 2008, Nguyen Thai Ngoc Duy wrote:
>
>
> > On 7/24/08, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
>  > >  > > No, I did mean the index.  This is an attribute of the index:
>  > >  > > either it is sparsely checked out or not.  You can even have
>  > >  > > multiple indices (switching between them by setting
>  > >  > > GIT_INDEX_FILE) which have different prefixes.
>  > >  >
>  > >  > I don't think so. It's a mask for workdir, right? If you save it it
>  > >  > index, you can switch index and the prefix as well, but workdir
>  > >  > only has several subtrees that do not fit any other prefix than the
>  > >  > original prefix.
>  > >
>  > >
>  > > Ah, you adroitly avoided addressing the issue that the user can change
>  > > the prefix without the index ever noticing.
>  >
>  > Forgive my ignorance. I still do not get why index must notice prefix
>  > change? The only reason I can think of is that we must make sure there
>  > won't be any user-modification in index outside the prefix. But that can
>  > be guarded from higher level (plumbings and porcelains) because index is
>  > allowed to have modification outside sparse prefix (auto-merged
>  > entries).
>
>
> Why do you want to guard it from the outside?  When the obvious fix is to
>  put together what belongs together?

OK. I am lost here. I do not know how putting sparse prefix and index
together could fix "it" (I think you meant the index guarding). But we
are in -rc period now, probably should not put too much time on this.
I will take time to think about this.

-- 
Duy

Re: [RFC PATCH 00/12] Sparse checkout

[Avery Pennarun]

On 7/24/08, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:
> OK. I am lost here. I do not know how putting sparse prefix and index
>  together could fix "it" (I think you meant the index guarding).

I think what Dscho is saying is simply that it's natural to put
information about the currently-checked-out tree in the index, and
unnatural to put it in the config.

If I switch branches, or checkout different versions, or git add, or
git rm, it all affects the index, never the config.  The index stores
the attributes of which files are checked out, and can detect whether
those files are different from before.  Choosing which subtrees to
check out sounds a lot more like one of those operations than it does
like a configuration change.

Also, I don't know if git supports this right now, but I can imagine
situations where you'd want to have more than one index (and
associated working trees) sharing the exact same .git folder.  It
would be fine to share the config between these parallel checkouts,
but you certainly couldn't share the index.  And you probably wouldn't
want to check out exactly the same set of subtrees in every working
tree.

The information required to do a checkout is in the index.  And
"sparse checkout" is all about checking out :)

Have fun,

Avery

Re: [RFC PATCH 00/12] Sparse checkout

[Johannes Schindelin]

Hi,

On Thu, 24 Jul 2008, Avery Pennarun wrote:

> [a nice summary about why the index should know the sparse checkout 
>  prefix]

Thanks.

> Also, I don't know if git supports this right now, but I can imagine
> situations where you'd want to have more than one index (and
> associated working trees) sharing the exact same .git folder.

It is.  And I already mentioned it.  Note that it can make sense to work 
on an index only, without ever touching a working directory.

For example, I do that a lot when importing projects that use 
"tarball+patches" for version control.

Ciao,
Dscho

Re: [RFC PATCH 00/12] Sparse checkout

[Nguyen Thai Ngoc Duy]

On 7/24/08, Avery Pennarun <apenwarr@gmail.com> wrote:
> On 7/24/08, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:
>  > OK. I am lost here. I do not know how putting sparse prefix and index
>  >  together could fix "it" (I think you meant the index guarding).
>
> [snip]
>
>  The information required to do a checkout is in the index.  And
>  "sparse checkout" is all about checking out :)

That was better Dscho's questions ;) Thanks for the enlightenment.
-- 
Duy

Re: [RFC PATCH 00/12] Sparse checkout

[Jan Hudec]

Hello,

sorry for replying to an old thread, but I am a bit puzzled here,

On Wed, Jul 23, 2008 at 17:55:14 +0100, Johannes Schindelin wrote:
> On Wed, 23 Jul 2008, Nguyen Thai Ngoc Duy wrote:
> > On 7/23/08, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > >  On Wed, 23 Jul 2008, Nguyễn Thái Ngọc Duy wrote:
> > >
> > >  > So in short, sparse prefix will be stored in config, 
> > >  > core.sparsecheckout.
> > >
> > > Do you really think the prefix should be stored anywhere else than the 
> > > index?
> > >
> > > With core.sparseCheckout you have to introduce a _sh*tload_ of config 
> > > loaders.
> > >
> > > And with core.sparseCheckout you are at the whim of the user, since 
> > > .git/config is _supposed_ to be user-editable.
> > >
> > > From a logical point of view, I'd say that the sparse prefix has 
> > > nothing to do with the "configuration" of the local repository.
> > 
> > Well, whatever place. I chose .git/config because I did not want to 
> > introduce a new config place. But then how about .git/sparsecheckout?
> 
> No, I did mean the index.  This is an attribute of the index: either it is 
> sparsely checked out or not.  You can even have multiple indices 
> (switching between them by setting GIT_INDEX_FILE) which have different 
> prefixes.

Um, but does the prefix we want to use depend on what files are physically
present in the tree? That would however imply that it's /not/ an attribute of
the index, but the tree and therefore should be stored in a separate object.

The question whether it goes in the .git/config or .git/sparsecheckout is
than that of semantics -- if it's in .git/config, than user changes that by
editing the file or git config and no other way, while if it's in
.git/sparsecheckout, user changes it by running checkout with appropriate
arguments.

By the way, why is it *prefix*? Wouldn't a *path limit* be better?

-- 
						 Jan 'Bulb' Hudec <bulb@ucw.cz>

Re: [RFC PATCH 00/12] Sparse checkout

[Johannes Schindelin]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2401 bytes --]

Hi,

On Sun, 3 Aug 2008, Jan Hudec wrote:

> On Wed, Jul 23, 2008 at 17:55:14 +0100, Johannes Schindelin wrote:
> > On Wed, 23 Jul 2008, Nguyen Thai Ngoc Duy wrote:
> > > On 7/23/08, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > > >  On Wed, 23 Jul 2008, Nguyễn Thái Ngọc Duy wrote:
> > > >
> > > >  > So in short, sparse prefix will be stored in config, 
> > > >  > core.sparsecheckout.
> > > >
> > > > Do you really think the prefix should be stored anywhere else than 
> > > > the index?
> > > >
> > > > With core.sparseCheckout you have to introduce a _sh*tload_ of 
> > > > config loaders.
> > > >
> > > > And with core.sparseCheckout you are at the whim of the user, 
> > > > since .git/config is _supposed_ to be user-editable.
> > > >
> > > > From a logical point of view, I'd say that the sparse prefix has 
> > > > nothing to do with the "configuration" of the local repository.
> > > 
> > > Well, whatever place. I chose .git/config because I did not want to 
> > > introduce a new config place. But then how about 
> > > .git/sparsecheckout?
> > 
> > No, I did mean the index.  This is an attribute of the index: either 
> > it is sparsely checked out or not.  You can even have multiple indices 
> > (switching between them by setting GIT_INDEX_FILE) which have 
> > different prefixes.
> 
> Um, but does the prefix we want to use depend on what files are 
> physically present in the tree?

The thing is: in a checkout, the flow of data is

	HEAD -> index -> worktree

To check in, the flow is exactly opposite.

There are also legal workflows where you do not work on the worktree at 
all.

And there are legal workflows where you work with multiple 
indices/worktrees.

Now, it appears pretty obvious to me that the logical thing is to tell the 
(current) index what is supposed to be checked out and what not.

After all, a merge is done inside the index.  And if the merge touches 
parts outside of the sparse checkout, it is natural to have the index know 
about it.

Because the merge _has_ to complain then.

In any case, isn't it clearly the more elegant solution to just put the 
information into the index when you touch it _anyway_?

I am really puzzled why people think it is a good idea to separate the 
file metadata from the information what files were checked out to begin 
with, as if they were totally independent.

Because they are not.

Ciao,
Dscho

Re: [RFC PATCH 00/12] Sparse checkout

[Nguyen Thai Ngoc Duy]

On 8/4/08, Jan Hudec <bulb@ucw.cz> wrote:
> Hello,
>
>  sorry for replying to an old thread, but I am a bit puzzled here,
>
>
>  On Wed, Jul 23, 2008 at 17:55:14 +0100, Johannes Schindelin wrote:
>  > On Wed, 23 Jul 2008, Nguyen Thai Ngoc Duy wrote:
>  > > On 7/23/08, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
>  > > >  On Wed, 23 Jul 2008, Nguyễn Thái Ngọc Duy wrote:
>  > > >
>  > > >  > So in short, sparse prefix will be stored in config,
>  > > >  > core.sparsecheckout.
>  > > >
>  > > > Do you really think the prefix should be stored anywhere else than the
>  > > > index?
>  > > >
>  > > > With core.sparseCheckout you have to introduce a _sh*tload_ of config
>  > > > loaders.
>  > > >
>  > > > And with core.sparseCheckout you are at the whim of the user, since
>  > > > .git/config is _supposed_ to be user-editable.
>  > > >
>  > > > From a logical point of view, I'd say that the sparse prefix has
>  > > > nothing to do with the "configuration" of the local repository.
>  > >
>  > > Well, whatever place. I chose .git/config because I did not want to
>  > > introduce a new config place. But then how about .git/sparsecheckout?
>  >
>  > No, I did mean the index.  This is an attribute of the index: either it is
>  > sparsely checked out or not.  You can even have multiple indices
>  > (switching between them by setting GIT_INDEX_FILE) which have different
>  > prefixes.
>
>
> Um, but does the prefix we want to use depend on what files are physically
>  present in the tree? That would however imply that it's /not/ an attribute of
>  the index, but the tree and therefore should be stored in a separate object.
>
>  The question whether it goes in the .git/config or .git/sparsecheckout is
>  than that of semantics -- if it's in .git/config, than user changes that by
>  editing the file or git config and no other way, while if it's in
>  .git/sparsecheckout, user changes it by running checkout with appropriate
>  arguments.

Please see the post Junio talked about narrow/sparse checkout [1].
With that approach (that is finer grain checkout than my approach), it
becomes clear that "prefix" is an attribute of index.

[1] http://article.gmane.org/gmane.comp.version-control.git/90016/match=narrow+checkout

>  By the way, why is it *prefix*? Wouldn't a *path limit* be better?

Well, yes "path limiter" seems a better term. But it's irrelevant now
because this approach won't fly. I'm going to wait to see if anyone
takes the task as part of hackontest. Otherwise I will redo my series,
using assume unchanged bit.

>  --
>                                                  Jan 'Bulb' Hudec <bulb@ucw.cz>
>


-- 
Duy

Re: [RFC PATCH 00/12] Sparse checkout

[James Pickens]

Nguyễn Thái Ngọc Duy <pclouds <at> gmail.com> writes:

> I have not looked at non-builtin commands yet, but I think it's not
> a big deal. We have several rounds before this series is good enough ;)
> So in short, sparse prefix will be stored in config, core.sparsecheckout.
> you have three new commands to enter/update/leave sparse checkout:
> 
> git clone --path=prefix       # clone with sparse checkout
> git checkout --path=prefix    # limit/update checkout paths
> git checkout --full           # stop sparse checkout
>

First things first, thanks a lot for working on this feature.  I have an
enormous project in CVS (144GB repo, containing 65000 directories and
463000 files) that I've been wanting to convert to git for a while now,
and the lack of sparse checkouts was the only thing about git that was
standing in the way.  The project is so big that checking out the whole
tree all the time is unworkable, and I think my coworkers would hang me
if I tried to make them use submodules.  We already use sparse checkouts
in CVS to make it manageable, so sparse checkout support in git would
vastly simplify the transition.

I played around with the patch briefly, and I have a couple of comments
on the interface.

First, I would want a capability to checkout a directory non-recursively.
I.e., checkout directory A/B, without also checking out directory A/B/C.
Perhaps a modifier could be added to a path element to make it
non-recursive.

Second, I would want a capability to checkout and release directories
incrementally, similar to how we do it in cvs.  For example, I might do
the following in cvs:

$ cvs co -l A         # Checkout dir A non-recursively
$ cd A
$ cvs up -l -d B1 B2  # Checkout dirs A/B1 and A/B2 non-recursively
$ cd B1
$ cvs up -d C1 C2     # Checkout dirs A/B1/C1 and A/B1/C2 recursively
(Oops, didn't need C2)
$ cvs release -d C2

At this point I would have the following directory tree, assuming the C1
directory in the repo contained a D1 directory:

A/
A/B1/
A/B1/C1/
A/B1/C1/D1/
A/B2/

A similar capability in git would be much appreciated.

Finally, I noticed what I think is a bug: if you do a partial checkout of
a non-existing directory, you just get an empty tree.  I would expect to
get an error message in that case.

I hope this is helpful, and thanks again for working on this.

James

Re: [RFC PATCH 00/12] Sparse checkout

[Nguyen Thai Ngoc Duy]

On 7/24/08, James Pickens <jepicken@gmail.com> wrote:
> Nguyen Thai Ngoc Duy <pclouds <at> gmail.com> writes:
>
>  > I have not looked at non-builtin commands yet, but I think it's not
>  > a big deal. We have several rounds before this series is good enough ;)
>  > So in short, sparse prefix will be stored in config, core.sparsecheckout.
>  > you have three new commands to enter/update/leave sparse checkout:
>  >
>  > git clone --path=prefix       # clone with sparse checkout
>  > git checkout --path=prefix    # limit/update checkout paths
>  > git checkout --full           # stop sparse checkout
>  >
>
>
> First things first, thanks a lot for working on this feature.  I have an
>  enormous project in CVS (144GB repo, containing 65000 directories and
>  463000 files) that I've been wanting to convert to git for a while now,
>  and the lack of sparse checkouts was the only thing about git that was
>  standing in the way.  The project is so big that checking out the whole
>  tree all the time is unworkable, and I think my coworkers would hang me
>  if I tried to make them use submodules.  We already use sparse checkouts
>  in CVS to make it manageable, so sparse checkout support in git would
>  vastly simplify the transition.
>
>  I played around with the patch briefly, and I have a couple of comments
>  on the interface.
>
>  First, I would want a capability to checkout a directory non-recursively.
>  I.e., checkout directory A/B, without also checking out directory A/B/C.
>  Perhaps a modifier could be added to a path element to make it
>  non-recursive.

This one is difficult (and may probably produce more intrusive patch).
Let's see what I can do.

>  Second, I would want a capability to checkout and release directories
>  incrementally, similar to how we do it in cvs.  For example, I might do
>  the following in cvs:
>
>  $ cvs co -l A         # Checkout dir A non-recursively
>  $ cd A
>  $ cvs up -l -d B1 B2  # Checkout dirs A/B1 and A/B2 non-recursively
>  $ cd B1
>  $ cvs up -d C1 C2     # Checkout dirs A/B1/C1 and A/B1/C2 recursively
>  (Oops, didn't need C2)
>  $ cvs release -d C2
>  At this point I would have the following directory tree, assuming the C1
>  directory in the repo contained a D1 directory:
>
>  A/
>  A/B1/
>  A/B1/C1/
>  A/B1/C1/D1/
>  A/B2/
>
>  A similar capability in git would be much appreciated.

You can do that with "git checkout --path" (non-recursive checkout aside):

$ git checkout --path=A                     # checkout full A
$ git checkout --path=A/B1/C1               # no, limit to A/B1/C1 only
$ git checkout --path=A/B1/C1:A/B2          # extend to A/B2 too

>
>  Finally, I noticed what I think is a bug: if you do a partial checkout of
>  a non-existing directory, you just get an empty tree.  I would expect to
>  get an error message in that case.

Thanks.

>  I hope this is helpful, and thanks again for working on this.
>
>  James
>
>
>
>  --
>  To unsubscribe from this list: send the line "unsubscribe git" in
>  the body of a message to majordomo@vger.kernel.org
>  More majordomo info at  http://vger.kernel.org/majordomo-info.html
>


-- 
Duy

Re: [RFC PATCH 00/12] Sparse checkout

[James Pickens]

Nguyen Thai Ngoc Duy <pclouds <at> gmail.com> writes:

> This one is difficult (and may probably produce more intrusive patch).
> Let's see what I can do.

This is not a high priority for me, so if the added
difficulty/intrusiveness means it would take longer for the sparse
checkout capability to make it into mainline git, then I would drop this
request, or at least delay it until some later time.

> >  Second, I would want a capability to checkout and release directories
> >  incrementally, similar to how we do it in cvs.  For example, I might do
> >  the following in cvs:
> 
> You can do that with "git checkout --path" (non-recursive checkout aside):
> 
> $ git checkout --path=A                     # checkout full A
> $ git checkout --path=A/B1/C1               # no, limit to A/B1/C1 only
> $ git checkout --path=A/B1/C1:A/B2          # extend to A/B2 too

My point was not that incremental checkout is impossible, just that it
would be easier if I didn't have to re-type the full path list again.
Suppose I had 9 directories (A/B1 through A/B9) in my sparse checkout,
and I wanted to add a 10th directory (A/B10).  It would be much easier
to type something like

$ cd A
$ git checkout B10

instead of

$ git checkout --path=A/B1:A/B2:A/B3:A/B4:A/B5:A/B6:A/B7:A/B8:A/B9:A/B10


Another thing I thought of, which I'm not sure if it's sensible or not,
is to make sparse checkout a persistent branch attribute by creating a
.gitpaths (or whatever) file containing a list of directories that
should be checked out.  This would be analogous to the .gitmodules file.
The .gitpaths file could then be checked in, and applied automatically
any time the user checks out a branch containing such file.  This is
just an idea for discussion, not a feature request.

James

Re: [RFC PATCH 00/12] Sparse checkout

[Nguyen Thai Ngoc Duy]

On 7/25/08, James Pickens <jepicken@gmail.com> wrote:
> > You can do that with "git checkout --path" (non-recursive checkout aside):
>  >
>  > $ git checkout --path=A                     # checkout full A
>  > $ git checkout --path=A/B1/C1               # no, limit to A/B1/C1 only
>  > $ git checkout --path=A/B1/C1:A/B2          # extend to A/B2 too
>
>
> My point was not that incremental checkout is impossible, just that it
>  would be easier if I didn't have to re-type the full path list again.
>  Suppose I had 9 directories (A/B1 through A/B9) in my sparse checkout,
>  and I wanted to add a 10th directory (A/B10).  It would be much easier
>  to type something like
>
>  $ cd A
>  $ git checkout B10
>
>  instead of
>
>  $ git checkout --path=A/B1:A/B2:A/B3:A/B4:A/B5:A/B6:A/B7:A/B8:A/B9:A/B10

"git checkout B10" form has already been taken. How about "git
checkout --add-path" and "git checkout --remove-path"?
-- 
Duy

Re: [RFC PATCH 00/12] Sparse checkout

[James Pickens]

Nguyen Thai Ngoc Duy <pclouds <at> gmail.com> writes:

> "git checkout B10" form has already been taken. How about "git
> checkout --add-path" and "git checkout --remove-path"?

That would be fine.  The specific syntax is not that important to me, as long as
the capability exists.

James

Re: [RFC PATCH 00/12] Sparse checkout

[Jakub Narebski]

Nguyễn Thái Ngọc Duy <pclouds@gmail.com> writes:

> I have not looked at non-builtin commands yet, but I think it's not
> a big deal. We have several rounds before this series is good enough ;)
> So in short, sparse prefix will be stored in config, core.sparsecheckout.
> you have three new commands to enter/update/leave sparse checkout:
> 
> git clone --path=prefix       # clone with sparse checkout
> git checkout --path=prefix    # limit/update checkout paths
> git checkout --full           # stop sparse checkout
> 
> Other operations should be normal. User attempts to do anything outside
> sparse checkout will get flagged. Git itself should not touch anything
> outside sparse checkout.
> 
> One more thing. As files outside sparse checkout will not be checking
> out, .gitignore and .gitattributes from parent directories (outside
> sparse checkout) will be gone too. This may lead to surprise.
> 
> Comments are welcome.

A note: my comments here reflects what I have remember from reading
comments in this thread; I have not examined the code, though.


First, I think that 'sparse checkout' is a better idea than former
'subtree (partial) checkout'; and I guess it could be easier to code.


Second, I think you can simply special case .git* files (.gitignore,
.gitattributes, .gitmodules), and always check them out for all
intermediate directories (unless configured otherwise, of course).
So for example if you have the following directory structure:

  A/.gitignore
  A/a
  A/B1/.gitignore
  A/B1/b
  A/B2/.gitignore
  A/B2/c

and you are checking out only subdirectory 'B1' (and all files in it;
if subdirectories are checked out recursively it depends on
configuration), and if for example there is .gitignore in every
directory, then checked out tree would look like this:

  A/.gitignore
  A/B1/.gitignore
  A/B1/b

The ability to do this is one of advantages of 'sparse' checkout over
'subtree' checkout.


Third, about the place where to store information about which paths
are checked out, and which are not.  There are three possible places
to store this information:

  1. repository configuration, e.g. `core.sparsecheckout' variable
     (multivalued?), like for `core.worktree'

  2. some text file in $GIT_DIR, e.g. '.git/sparse', like for shallow
     clone ("git clone --depth <depth>") it is grafts-like
     $GIT_DIR/shallow (see Documentation/technical/shallow.txt).

  3. in the index itseld ($GIT_DIR/index), as proposed by Johannes
     Schindelin.

While I do think that some information about sparseness should be in
the index, for git to be able to commit from the index for example,
I don't think it is a good place as the only/main place to store
information about which paths are checked out; I think that because
IMVHO git commands should survive hosing (removing) index file.

Just my 2 eurocents.
-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: [RFC PATCH 00/12] Sparse checkout

[Nguyen Thai Ngoc Duy]

On 7/24/08, Jakub Narebski <jnareb@gmail.com> wrote:
>  Second, I think you can simply special case .git* files (.gitignore,
>  .gitattributes, .gitmodules), and always check them out for all
>  intermediate directories (unless configured otherwise, of course).
>  So for example if you have the following directory structure:
>
>   A/.gitignore
>   A/a
>   A/B1/.gitignore
>   A/B1/b
>   A/B2/.gitignore
>   A/B2/c
>
>  and you are checking out only subdirectory 'B1' (and all files in it;
>  if subdirectories are checked out recursively it depends on
>  configuration), and if for example there is .gitignore in every
>  directory, then checked out tree would look like this:
>
>   A/.gitignore
>   A/B1/.gitignore
>   A/B1/b
>
>  The ability to do this is one of advantages of 'sparse' checkout over
>  'subtree' checkout.

Or teach git to use index version of those files. Or collect all those
files, combine them and put the result to .git/info/exclude (and
similar places). Anyway well organized repos won't have this problem.

Checking some files out as read-only (like this case) may be
interesting. Though I do not how much complicated it can be.
-- 
Duy

Re: [RFC PATCH 00/12] Sparse checkout

[Jakub Narebski]

On Thu, 24 Jul 2008, Nguyen Thai Ngoc Duy wrote:
> On 7/24/08, Jakub Narebski <jnareb@gmail.com> wrote:
> >
> >  Second, I think you can simply special case .git* files (.gitignore,
> >  .gitattributes, .gitmodules), and always check them out for all
> >  intermediate directories (unless configured otherwise, of course).
> >  So for example if you have the following directory structure:
> >
> >   A/.gitignore
> >   A/a
> >   A/B1/.gitignore
> >   A/B1/b
> >   A/B2/.gitignore
> >   A/B2/c
> >
> >  and you are checking out only subdirectory 'B1' (and all files in it;
> >  if subdirectories are checked out recursively it depends on
> >  configuration), and if for example there is .gitignore in every
> >  directory, then checked out tree would look like this:
> >
> >   A/.gitignore
> >   A/B1/.gitignore
> >   A/B1/b
> >
> >  The ability to do this is one of advantages of 'sparse' checkout over
> >  'subtree' checkout.
> 
> Or teach git to use index version of those files. Or collect all those
> files, combine them and put the result to .git/info/exclude (and
> similar places). Anyway well organized repos won't have this problem.
> 
> Checking some files out as read-only (like this case) may be
> interesting. Though I do not how much complicated it can be.

I think teaching git to use index version of .git* files (.gitignore,
.gitattributes, .gitmodules) would be much more work than adding
default rule that .git* files in leading directories are by default
checked out, just like leading directories are checked out.  This
would limit modifying git code, I think, and chances for errors.

Having "leading" directories and files read-only would be a good idea,
I think.

I don't understand the sentence "well organized repos won't have this
problem". I think well organized repos _would_ have this problem,
because of maintained and distributed top-level .gitignore and
.gitattributes.

P.S. I hope that 'sparse checkout' feature would be ready for 1.7.0
-- 
Jakub Narebski
Poland

Re: [RFC PATCH 00/12] Sparse checkout

[Nguyen Thai Ngoc Duy]

On 7/25/08, Jakub Narebski <jnareb@gmail.com> wrote:
> On Thu, 24 Jul 2008, Nguyen Thai Ngoc Duy wrote:
>
> > On 7/24/08, Jakub Narebski <jnareb@gmail.com> wrote:
>  > >
>  > >  Second, I think you can simply special case .git* files (.gitignore,
>  > >  .gitattributes, .gitmodules), and always check them out for all
>  > >  intermediate directories (unless configured otherwise, of course).
>  > >  So for example if you have the following directory structure:
>  > >
>  > >   A/.gitignore
>  > >   A/a
>  > >   A/B1/.gitignore
>  > >   A/B1/b
>  > >   A/B2/.gitignore
>  > >   A/B2/c
>  > >
>  > >  and you are checking out only subdirectory 'B1' (and all files in it;
>  > >  if subdirectories are checked out recursively it depends on
>  > >  configuration), and if for example there is .gitignore in every
>  > >  directory, then checked out tree would look like this:
>  > >
>  > >   A/.gitignore
>  > >   A/B1/.gitignore
>  > >   A/B1/b
>  > >
>  > >  The ability to do this is one of advantages of 'sparse' checkout over
>  > >  'subtree' checkout.
>  >
>  > Or teach git to use index version of those files. Or collect all those
>  > files, combine them and put the result to .git/info/exclude (and
>  > similar places). Anyway well organized repos won't have this problem.
>  >
>  > Checking some files out as read-only (like this case) may be
>  > interesting. Though I do not how much complicated it can be.
>
>
> I think teaching git to use index version of .git* files (.gitignore,
>  .gitattributes, .gitmodules) would be much more work than adding
>  default rule that .git* files in leading directories are by default
>  checked out, just like leading directories are checked out.  This
>  would limit modifying git code, I think, and chances for errors.
>
>  Having "leading" directories and files read-only would be a good idea,
>  I think.
>
>  I don't understand the sentence "well organized repos won't have this
>  problem". I think well organized repos _would_ have this problem,
>  because of maintained and distributed top-level .gitignore and
>  .gitattributes.

I wrote that with svn repos in mind. If those repos are to be
partially checked out, .svnignore would be in subdirectories rather
than at toplevel. Anyway that may not be true.

>  P.S. I hope that 'sparse checkout' feature would be ready for 1.7.0
>
> --
>  Jakub Narebski
>  Poland
>


-- 
Duy

Re: [RFC PATCH 00/12] Sparse checkout

[Avery Pennarun]

On 7/24/08, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:
> I wrote that with svn repos in mind. If those repos are to be
>  partially checked out, .svnignore would be in subdirectories rather
>  than at toplevel. Anyway that may not be true.

As far as I know, this is a misstatement of how svn works.  It uses
svn:ignore properties, not .svnignore files, and they don't apply
recursively, so this potential inconsistency wouldn't happen with svn.
 Of course, in the name of this consistency, svn gave up recursive
ignores.

So anyway, back on topic, the definition of a "well-organized
repository" is different wth git than with svn for this reason.  I
would certainly expect most well-organized git repositories to have a
toplevel .gitignore for the most common stuff.

Have fun,

Avery

Re: [RFC PATCH 00/12] Sparse checkout

[Johannes Schindelin]

Hi,

On Thu, 24 Jul 2008, Jakub Narebski wrote:

> I think teaching git to use index version of .git* files (.gitignore,
> .gitattributes, .gitmodules) would be much more work than adding
> default rule that .git* files in leading directories are by default
> checked out, just like leading directories are checked out.

"Teaching" Git that would also directly contradict Git's primary motto: 
"Content is king".

Oh, by some convoluted reasoning you could argue that "content" hidden 
somewhere else than the working directory is "content", but nobody in her 
right mind would buy into that.

Ciao,
Dscho