[QUESTION] Selective fetch possible?

[QUESTION] Selective fetch possible?

[Filippo Zangheri]

Hi everybody,

Is it possible to git-fetch only a portion of the tree
of the specified repository, say, fetch only one directory or a
subset of files matching some regular expression? This is currently
- to my knowledge - only possible via wget iff the GIT repository
has gitweb enabled. But that's just a workaround.

Thanks for the great instrument that GIT is.
Best regards.

PS: please CC me in replies, because I'm not subscribed to git ML.

-- 
Filippo Zangheri

GPG key ID: 0xE1D879FA
Key fingerprint: 816B CE57 D43C 0A47 EF35 3378 EA5F A72A E1D8 79FA
Key server: pgp.mit.edu

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GE d- s+:- a-- C++ UL+++ P+ L+++ E-- W+ N* o-- K- w--- O-- M--
V- PS++ PE+ Y+ PGP++ t 5-- X++ R* tv b+ DI-- D---- G-- e++ h--
r++ z*
------END GEEK CODE BLOCK------

Re: [QUESTION] Selective fetch possible?

[Shawn O. Pearce]

Filippo Zangheri <filippo.zangheri@yahoo.it> wrote:
> Is it possible to git-fetch only a portion of the tree
> of the specified repository, say, fetch only one directory or a
> subset of files matching some regular expression? This is currently
> - to my knowledge - only possible via wget iff the GIT repository
> has gitweb enabled. But that's just a workaround.

No.

You can use a shallow clone to fetch only X commits back into
history on any branch, and you can also manually configure the
fetch specification in .git/config to only fetch specific branches,
but you must fetch the entire tree to get any of the files in it.

If the repository is available by git:// protocol you may be able
to use git-archive to obtain a tarfile for just the directory you
want (service has to be enabled on the remote side) but that is
just a raw UNIX tar; there is no Git repository and no ability to
commit/fetch/push/diff/apply/log/etc.
 
-- 
Shawn.

Re: [QUESTION] Selective fetch possible?

[Jakub Narebski]

"Shawn O. Pearce" <spearce@spearce.org> writes:

> Filippo Zangheri <filippo.zangheri@yahoo.it> wrote:
> >
> > Is it possible to git-fetch only a portion of the tree
> > of the specified repository, say, fetch only one directory or a
> > subset of files matching some regular expression? This is currently
> > - to my knowledge - only possible via wget iff the GIT repository
> > has gitweb enabled. But that's just a workaround.
> 
> No.
> 
> You can use a shallow clone to fetch only X commits back into
> history on any branch, and you can also manually configure the
> fetch specification in .git/config to only fetch specific branches,
> but you must fetch the entire tree to get any of the files in it.
> 
> If the repository is available by git:// protocol you may be able
> to use git-archive to obtain a tarfile for just the directory you
> want (service has to be enabled on the remote side) but that is
> just a raw UNIX tar; there is no Git repository and no ability to
> commit/fetch/push/diff/apply/log/etc.

Note that what you wanted is, I guess, something called partial
checkout or subtree checkout.  This feature appears now and then in
feature requests; lately Nguyen Thai Ngoc Duy (pclouds) offered to do
this in "on subtree checkout" thread:
  http://thread.gmane.org/gmane.comp.version-control.git/74915

The problem is twofold, as far as I understand it.  First, what to do
if there is merge conflicts outside checked out (selected) directory?
Second, how to make repository contain only relevant objects: git in
many places assumes full connectivity, and that if it has an object it
hass all objects depending on it.

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: [QUESTION] Selective fetch possible?

[Rogan Dawes]

Jakub Narebski wrote:
> "Shawn O. Pearce" <spearce@spearce.org> writes:
> 
>> Filippo Zangheri <filippo.zangheri@yahoo.it> wrote:
>>> Is it possible to git-fetch only a portion of the tree
>>> of the specified repository, say, fetch only one directory or a
>>> subset of files matching some regular expression? This is currently
>>> - to my knowledge - only possible via wget iff the GIT repository
>>> has gitweb enabled. But that's just a workaround.
>> No.
>>
>> You can use a shallow clone to fetch only X commits back into
>> history on any branch, and you can also manually configure the
>> fetch specification in .git/config to only fetch specific branches,
>> but you must fetch the entire tree to get any of the files in it.
>>
>> If the repository is available by git:// protocol you may be able
>> to use git-archive to obtain a tarfile for just the directory you
>> want (service has to be enabled on the remote side) but that is
>> just a raw UNIX tar; there is no Git repository and no ability to
>> commit/fetch/push/diff/apply/log/etc.
> 
> Note that what you wanted is, I guess, something called partial
> checkout or subtree checkout.  This feature appears now and then in
> feature requests; lately Nguyen Thai Ngoc Duy (pclouds) offered to do
> this in "on subtree checkout" thread:
>   http://thread.gmane.org/gmane.comp.version-control.git/74915

I still believe that it could be done fairly easily by simply recording 
the SHA1's of the files and directories that are *not* checked out 
somewhere in the .git directory, and just reusing those when checking 
the working tree. i.e. rather than stat-ing a tree that was never 
checked out, get the known SHA1 for that tree from where it was recorded 
on checkout.

> The problem is twofold, as far as I understand it.  First, what to do
> if there is merge conflicts outside checked out (selected) directory?

This is something that has been repeated many times, and I fail to see 
how it can be an issue. How can there be a conflict in a directory that 
is not, and never has been, checked out, and therefore cannot have been 
modified?

The only possibility that I can see is if the directory has been renamed 
elsewhere, but in that case, it *is* effectively checked out (just with 
a different directory name).

> Second, how to make repository contain only relevant objects: git in
> many places assumes full connectivity, and that if it has an object it
> hass all objects depending on it.
> 

Yes, this is the big problem as I see it.

Rogan

Re: [QUESTION] Selective fetch possible?

[Shawn O. Pearce]

Rogan Dawes <lists@dawes.za.net> wrote:
> Jakub Narebski wrote:
> >"Shawn O. Pearce" <spearce@spearce.org> writes:
> >
> >>Filippo Zangheri <filippo.zangheri@yahoo.it> wrote:
> >>>Is it possible to git-fetch only a portion of the tree
> >>>of the specified repository, say, fetch only one directory or a
> >>>subset of files matching some regular expression?
> 
> >The problem is twofold, as far as I understand it.  First, what to do
> >if there is merge conflicts outside checked out (selected) directory?
> 
> This is something that has been repeated many times, and I fail to see 
> how it can be an issue. How can there be a conflict in a directory that 
> is not, and never has been, checked out, and therefore cannot have been 
> modified?

Given two branches:

	code
	docs

and the code people checkout the "src/" subdirectory and the docs
people checkout the "Documentation/" subdirectory, and they *only*
every work in that subdirectory, things are fine.

Until one day some developer also checks out "Documentation/" and
fixes something in the documentation as part of the same commit
that makes a code change.  The push this to the code branch.

Someday in the future a documentation writer merges the code branch
over to the docs branch, "just keeping it current".

Now there arises a possiblity of a merge conflict in a part of the
tree that you do not have checked out.


If you want to say "don't ever modify stuff outside of your branch's
purpose" then why aren't you just using submodules (one for docs and
one for code) and using a supermodule to tie everything together into
a "release package"?

> >Second, how to make repository contain only relevant objects: git in
> >many places assumes full connectivity, and that if it has an object it
> >hass all objects depending on it.
> >
> 
> Yes, this is the big problem as I see it.

This is easy enough that if the above problem could be resolved
sufficiently to the git gurus' satisfaction you would be able
to get some advice on how to solve it.  Its not difficult, just
damn annoying.  We already do it (to some extent) with grafts and
shallow clones.

-- 
Shawn.

Re: [QUESTION] Selective fetch possible?

[Rogan Dawes]

Shawn O. Pearce wrote:
> Rogan Dawes <lists@dawes.za.net> wrote:
>> Jakub Narebski wrote:
>>> "Shawn O. Pearce" <spearce@spearce.org> writes:
>>>
>>>> Filippo Zangheri <filippo.zangheri@yahoo.it> wrote:
>>>>> Is it possible to git-fetch only a portion of the tree
>>>>> of the specified repository, say, fetch only one directory or a
>>>>> subset of files matching some regular expression?
>>> The problem is twofold, as far as I understand it.  First, what to do
>>> if there is merge conflicts outside checked out (selected) directory?
>> This is something that has been repeated many times, and I fail to see 
>> how it can be an issue. How can there be a conflict in a directory that 
>> is not, and never has been, checked out, and therefore cannot have been 
>> modified?
> 
> Given two branches:
> 
> 	code
> 	docs
> 
> and the code people checkout the "src/" subdirectory and the docs
> people checkout the "Documentation/" subdirectory, and they *only*
> every work in that subdirectory, things are fine.
> 
> Until one day some developer also checks out "Documentation/" and
> fixes something in the documentation as part of the same commit
> that makes a code change.  The push this to the code branch.
> 
> Someday in the future a documentation writer merges the code branch
> over to the docs branch, "just keeping it current".
> 
> Now there arises a possiblity of a merge conflict in a part of the
> tree that you do not have checked out.
> 
> 
> If you want to say "don't ever modify stuff outside of your branch's
> purpose" then why aren't you just using submodules (one for docs and
> one for code) and using a supermodule to tie everything together into
> a "release package"?

Ok, fair enough. Thanks for the example.

I think that one should not *expect* to be able to complete merges with 
only a partial checkout, though. It *may* work in cases where there are 
no conflicts, but I think it would be a perfectly valid error path to 
fail if there is a conflicting merge in a part of the tree that has not 
been checked out.

So, for a user working on partial trees, they would be able to modify 
their partial tree, and check in their changes, but merges would have to 
be done by someone with a complete checkout. For the given examples 
where partial trees make sense (documentation workers), this seems like 
a reasonable compromise.

>>> Second, how to make repository contain only relevant objects: git in
>>> many places assumes full connectivity, and that if it has an object it
>>> hass all objects depending on it.
>>>
>> Yes, this is the big problem as I see it.
> 
> This is easy enough that if the above problem could be resolved
> sufficiently to the git gurus' satisfaction you would be able
> to get some advice on how to solve it.  Its not difficult, just
> damn annoying.  We already do it (to some extent) with grafts and
> shallow clones.

How's my suggestion above?

Rogan

Re: [QUESTION] Selective fetch possible?

[Filippo Zangheri]

Rogan Dawes ha scritto:
(...)
> I think that one should not *expect* to be able to complete merges with 
> only a partial checkout, though. It *may* work in cases where there are 
> no conflicts, but I think it would be a perfectly valid error path to 
> fail if there is a conflicting merge in a part of the tree that has not 
> been checked out.
> 
> So, for a user working on partial trees, they would be able to modify 
> their partial tree, and check in their changes, but merges would have to 
> be done by someone with a complete checkout. For the given examples 
> where partial trees make sense (documentation workers), this seems like 
> a reasonable compromise.

I think this is what every reasonable developer should have in mind
when he's working on just a project subset :). And I also think,
this is not a valid reason for forbidding/not implementing such a
partial (or subtree) checkout.


-- 
Filippo Zangheri

GPG key ID: 0xE1D879FA
Key fingerprint: 816B CE57 D43C 0A47 EF35 3378 EA5F A72A E1D8 79FA
Key server: pgp.mit.edu

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GE d- s+:- a-- C++ UL+++ P+ L+++ E-- W+ N* o-- K- w--- O-- M--
V- PS++ PE+ Y+ PGP++ t 5-- X++ R* tv b+ DI-- D---- G-- e++ h--
r++ z*
------END GEEK CODE BLOCK------