* Re: Change set based shallow clone
From: Jakub Narebski @ 2006-09-07 20:21 UTC (permalink / raw)
To: git
In-Reply-To: <9e4733910609071252ree73effwb06358e9a22ba965@mail.gmail.com>
Jon Smirl wrote:
> If you haven't updated for six months when the server walks backwards
> for 10 change sets it's not going to find anything you have locally.
> When this situation is encountered the server needs to generate a
> delta just for you between one of the change sets it knows you have
> and one of the 10 change sets you want. By generating this one-off
> delta it lets you avoid the need to fetch all of the objects back to a
> common branch ancestor. The delta functions as a jump over the
> intervening space.
I don't understand. Git is _not_ patchset based (like GNU Arch, or
Mercurial, or CVS). It is snapshot based. So if you want to download
"skip", you need only for the local part of doenloader to make appropriate
grafts, like below
*--*--*--*--*--*--*--*--*--*--*--HEAD (server)
*--*--*...........*--*--*--*--*--HEAD (shallow/sparse clone)
But the part you were talking about is _easy_ part; the hard part is
merges including merging branch which was split off the trunk before
cutoff-point, history rewriting (c.f. 'pu' branch, and rebases), etc.
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
^ permalink raw reply
* Re: [PATCH] autoconf: Add checking for BSD-compatibile 'install' program
From: Junio C Hamano @ 2006-09-07 20:21 UTC (permalink / raw)
To: Jakub Narebski; +Cc: git
In-Reply-To: <200609071431.18343.jnareb@gmail.com>
Jakub Narebski <jnareb@gmail.com> writes:
> Uncomment `AC_PROG_INSTALL' macro in configure.ac, and setting
> `INSTALL' variable in config.mak.in
>
> Add `install-sh' shell script (required when using `AC_PROG_INSTALL'
> macro) scriptversion=2005-11-07.23, taken from KDE SDK 3.5.3.
>
> Signed-off-by: Jakub Narebski <jnareb@gmail.com>
> ---
> Perhaps `install-sh' script should be replaced by something simplier,
> for example by script which was sent to git mailing list, but AFAIK
> not accepted.
That somehow sounds like whining. I do not think it was never
proposed for inclusion but was just offered as a friendly
"somebody else might find it useful".
In any case, I have two issues, but one non-issue first. I am
not opposed to including install-sh that is maintained somewhere
upstream and GPLv2 compatible.
- I do not particularly like to have too many autoconf related
cruft at the toplevel, and I am wondering if we can move it
to say compat/install-sh.
- I suspect this fall-back would set "INSTALL = ./install-sh"
(if you have it in compat, "INSTALL = compat/install-sh"). I
wonder if it breaks builds in subdirectories
(i.e. Documentation/ and templates/).
^ permalink raw reply
* Re: Change set based shallow clone
From: Jon Smirl @ 2006-09-07 20:41 UTC (permalink / raw)
To: Jakub Narebski; +Cc: git
In-Reply-To: <edpuut$dns$1@sea.gmane.org>
On 9/7/06, Jakub Narebski <jnareb@gmail.com> wrote:
> I don't understand. Git is _not_ patchset based (like GNU Arch, or
I meant change set to refer to a commit plus trees plus blobs that
make it up. These may be present in full or delta form.
> Mercurial, or CVS). It is snapshot based. So if you want to download
> "skip", you need only for the local part of doenloader to make appropriate
> grafts, like below
>
>
> *--*--*--*--*--*--*--*--*--*--*--HEAD (server)
>
> *--*--*...........*--*--*--*--*--HEAD (shallow/sparse clone)
>
> But the part you were talking about is _easy_ part; the hard part is
> merges including merging branch which was split off the trunk before
> cutoff-point, history rewriting (c.f. 'pu' branch, and rebases), etc.
Does an average user do these things? The shallow clone is there to
address the casual user who gags at a five hour download to get an
initial check out Mozilla when they want to make a five line change or
just browse the source for a few minutes.
I would expect advanced users to have a full tree present.
I was going to have the dangling references from the shallow clone
point to 'not-present' objects. When you try to do any of the more
complex operations you would hit these objects and fault down more of
the tree.
There would also be a command to bring down all of the objects to
fully populate a sparse tree. You could do the shallow clone to begin
with and then do the full tree populate overnight or in the
background.
Maybe the answer is to build a shallow clone tool for casual use, and
then if you try to run anything too complex on it git just tells you
that you have to download the entire tree.
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: Change set based shallow clone
From: Jeff King @ 2006-09-07 21:33 UTC (permalink / raw)
To: Jon Smirl; +Cc: Jakub Narebski, git
In-Reply-To: <9e4733910609071341u7e430214j71ddcbefa26810ca@mail.gmail.com>
On Thu, Sep 07, 2006 at 04:41:20PM -0400, Jon Smirl wrote:
> Maybe the answer is to build a shallow clone tool for casual use, and
> then if you try to run anything too complex on it git just tells you
> that you have to download the entire tree.
Can you list which operations you'd like to do on the shallow clone?
Many operations that look at history are going to fail with a very short
history (e.g., finding a merge base). Some operations can work with a
short history, but the results are probably not useful (seeing the last
10 commits from git-log is probably not interesting). If you just want
the latest snapshot, the remote git-archive work going on right now will
probably take care of that.
-Peff
^ permalink raw reply
* Re: Change set based shallow clone
From: Jakub Narebski @ 2006-09-07 21:37 UTC (permalink / raw)
To: git
In-Reply-To: <9e4733910609071341u7e430214j71ddcbefa26810ca@mail.gmail.com>
Jon Smirl wrote:
> On 9/7/06, Jakub Narebski <jnareb@gmail.com> wrote:
>> I don't understand. Git is _not_ patchset based (like GNU Arch, or
>
> I meant change set to refer to a commit plus trees plus blobs that
> make it up. These may be present in full or delta form.
>
>> Mercurial, or CVS). It is snapshot based. So if you want to download
>> "skip", you need only for the local part of downloader to make
>> appropriate grafts, like below
>>
>>
>> *--*--*--*--*--*--*--*--*--*--*--HEAD (server)
>>
>> *--*--*...........*--*--*--*--*--HEAD (shallow/sparse clone)
>>
>> But the part you were talking about is _easy_ part; the hard part is
>> merges including merging branch which was split off the trunk before
>> cutoff-point, history rewriting (c.f. 'pu' branch, and rebases), etc.
>
> Does an average user do these things? The shallow clone is there to
> address the casual user who gags at a five hour download to get an
> initial check out Mozilla when they want to make a five line change or
> just browse the source for a few minutes.
The question is not if average user do these things, but do average user
encounter these things. I think that merging branches into 'master' branch
(trunk) is quite common, and merging trunk into branches for checking is
also common.
As to checking out latest revision... with git-archive --remote (or
git-tar-tree --remote as of now) you can download newest version as
archive; although without commit object I think.
> I would expect advanced users to have a full tree present.
>
> I was going to have the dangling references from the shallow clone
> point to 'not-present' objects. When you try to do any of the more
> complex operations you would hit these objects and fault down more of
> the tree.
Only if you consider git-log to be complex operation. It would be better I
think to cover laps using grafts.
In first post you are talking about downloading diff covering the lap/skip;
it would be important only if git was patchset based. You need to download
commits and referenced objects, _except_ cutting off some parents.
> There would also be a command to bring down all of the objects to
> fully populate a sparse tree. You could do the shallow clone to begin
> with and then do the full tree populate overnight or in the
> background.
And that should be done by marking "shallow"/"sparse" grafts and just delete
them (or update if needed).
> Maybe the answer is to build a shallow clone tool for casual use, and
> then if you try to run anything too complex on it git just tells you
> that you have to download the entire tree.
The shallow/sparse/lazy[*1*] clone proposal is one of recurring proposals
(besides subproject support) on git mailing list.
Footnotes:
[*1*] Lazy clone, also known as remote alternatives.
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
^ permalink raw reply
* Re: [PATCH] autoconf: Add checking for BSD-compatibile 'install' program
From: Jakub Narebski @ 2006-09-07 21:39 UTC (permalink / raw)
To: git
In-Reply-To: <7v8xkvxwa7.fsf@assigned-by-dhcp.cox.net>
Junio C Hamano wrote:
> Jakub Narebski <jnareb@gmail.com> writes:
>
>> Uncomment `AC_PROG_INSTALL' macro in configure.ac, and setting
>> `INSTALL' variable in config.mak.in
>>
>> Add `install-sh' shell script (required when using `AC_PROG_INSTALL'
>> macro) scriptversion=2005-11-07.23, taken from KDE SDK 3.5.3.
>>
>> Signed-off-by: Jakub Narebski <jnareb@gmail.com>
>> ---
>> Perhaps `install-sh' script should be replaced by something simplier,
>> for example by script which was sent to git mailing list, but AFAIK
>> not accepted.
>
> That somehow sounds like whining. I do not think it was never
> proposed for inclusion but was just offered as a friendly
> "somebody else might find it useful".
Sorry, my mistake. I thought that the install-sh script was proposed as
solution for platforms lacking install program...
> In any case, I have two issues, but one non-issue first. I am
> not opposed to including install-sh that is maintained somewhere
> upstream and GPLv2 compatible.
>
> - I do not particularly like to have too many autoconf related
> cruft at the toplevel, and I am wondering if we can move it
> to say compat/install-sh.
We can do this by adding AC_CONFIG_AUX_DIR(DIR) to configure.ac,
where DIR would be for example 'compat/' or 'compat/autoconf/'
> - I suspect this fall-back would set "INSTALL = ./install-sh"
> (if you have it in compat, "INSTALL = compat/install-sh"). I
> wonder if it breaks builds in subdirectories
> (i.e. Documentation/ and templates/).
Yes, it sets "INSTALL = ./install-sh -c". And it does make problems for
e.g. 'make install-doc'. I don't know how this should be resolved...
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
^ permalink raw reply
* Re: Change set based shallow clone
From: Jakub Narebski @ 2006-09-07 21:51 UTC (permalink / raw)
To: git
In-Reply-To: <20060907213306.GA10781@sigio.intra.peff.net>
Jeff King wrote:
> Many operations that look at history are going to fail with a very short
> history (e.g., finding a merge base). Some operations can work with a
> short history, but the results are probably not useful (seeing the last
> 10 commits from git-log is probably not interesting). If you just want
> the latest snapshot, the remote git-archive work going on right now will
> probably take care of that.
That is the source of the idea of _sparse_ clone, which would include e.g.
top 10 commits from each of selected branches, all commits pointed by
branches and tags, and all merge bases, and roots, joined using grafts.
E.g. if the full history looks like this.
-*.|.......... head
/ |
-*--*--*- -*--*--x--*-|-*--*--*-- head
/ \ / |
*--*--*--*--*--*--*--*--*--*--*--*--*-|-*--*--*-- head
\...............|.......... tag
|
\- [cut-off]
then the sparse clone history would look like that:
(---) denotes parentage, (...) denotes grafts below
|
,......x....|.*--*--*-- head
: |
*....................*--*.............|.*--*--*-- head
\...............|.......... tag
|
\- [cut-off]
of course assuming that we have no earlier commits (i.e. sparse clone, not
sparse fetch).
Of course, that solves only some of troubles...
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
^ permalink raw reply
* Re: Change set based shallow clone
From: Junio C Hamano @ 2006-09-07 22:07 UTC (permalink / raw)
To: Jon Smirl; +Cc: git
In-Reply-To: <9e4733910609071252ree73effwb06358e9a22ba965@mail.gmail.com>
I've been postponing thinking about shallow clones with a plan
to do so when I have absolutely nothing else to do for a solid
few days. Since my time is booked pretty much full with day job
and the ongoing pack-file topics these days, I will most likely
stick to that plan after I finish this message.
Not because I think it is an insignificant problem at the fringe
(I did not say "when I have nothing better to do"), but because
I feel this is something important (and hard) that requires
"think about it and only about it straight through for a few
days, dropping everything else" kind of thinking.
When you have a ref under .git/refs/ pointing at an object, you
are making this assertion:
The object store associated with this repository
(usually, .git/objects and alternates pointed at by
.git/objects/info/alternates) holds _all_ objects that
are recursively reachable from this object by following
tag reference, commit ancestry chain, commit to its
tree, and tree to its subtrees and blobs.
So by definition, a shallow clone is a broken/incomplete
repository. Unless we fix the breakage in some way.
One way to do so which has often been talked about is to use the
grafts facility. You cut off the history at some commit, and
cauterize the commit ancestry chain at that point by telling git
to pretend that the commit does not have any parent. While this
would work after setting it up, in the sense that "git log"
would stop at the commit and "git fsck-objects" would say
nothing is missing, it is cumbersome to set up, we do not have
an easy way to fetch (or clone -- but clone is a special case of
fetch where the downstream starts out from emptiness) from such
a repository, unplugging to get histories further back with
a later fetch is quite involved, and it is not clear what it
means to further fetch from such a shallow clone.
Where does the difficulty in fetching come from? Mostly because
the upload side and download side needs to agree on the altered
history.
Let's say the download side is empty and somehow tells the
upload side that it wants the latest 5-commit worth of stuff
starting from the tip:
- The upload side runs
rev-list --objects HEAD~5..HEAD | pack-objects --stdout
to give you the needed data. This is almost [*1*] doable by
telling it that you have HEAD~5.
- Also the upload side can say "pretend that this commit HEAD~4
has no parent, although cat-file on it would say otherwise".
- The downloader creates a suitable grafts file to cauterize
commit HEAD~4.
Is this enough? Hardly.
Consider:
o---o---o---o---o---*---*---*---*---*
\ ^ / ^
\ HEAD~5 / HEAD
\ /
o---o---*---*---*
^ ^
HEAD^^2~3 HEAD^^2
5-commit request should apply to send '*' commits, so the
original "HEAD~5..HEAD" was bogus to begin with. You have to
say "HEAD --not HEAD~5 HEAD^^2~3" to get the above.
Coming up with this restriction needs to be done on the upload
side. Doing so on the download side means it needs to retrieve
the recent "real" history to do that computation to come up with
where to cut things off, which is crazy.
However, it may be that the above graph is not what the usual
downloader wants. If the upload side is _the_ official
repository of some sort that merges in subsystem branches, and
if you are mostly interested in the official repository's
history, you might have meant to ask this with your 5-commit
request instead:
o---o---o---o---o---*---*---*---*---*
\ ^ / ^
\ HEAD~5 / HEAD
\ /
o---o---o---o---o
^
HEAD^^2
That is, you would want to cauterize the side branch. The pack
needs to be built with "HEAD --not HEAD~5 HEAD^^2", and the
cauterize instructions would say "HEAD~4 has no parent, HEAD^
has only one parent that is HEAD~2", instead of saying just
"HEAD~4 has no parent."
So whoever is computing this cauterization point has a policy
decision to make, because the meaning of "5-commits away" or
"recent 10 days" are very fuzzy when merges are involved.
It obviously is easy to declare that the policy decision is
solely made on the uploader side, from the implementation point
of view, but I do not offhand know what the ramification of that
decision is.
Oh, about "recent 10 days".
The side branch might be an very old development that were
merged recently into the mainline. From mainline's point of
view, everything that happened on the side branch is something
new to him (think of Linus pulled from Jeff who has been cooking
his stuff for quite some time). Looking at each individual
commit on the side branch, however, they have older commit and
author timestamps. If you pick "I want mainline's point of
view", that means you need to compute (in the above picture):
- notice commit timestamp of HEAD^ (the merge) is within the
recent 10-day window.
- using HEAD^^1 (supposedly the state of the mainline before
the merge happened [*2*]) and HEAD^^2 (side branch),
compute:
rev-list HEAD^^1..HEAD^^2
the results are the "new" ones that appeared when the merge
was made, so they are within the recent 10-day window from
the mainline's point of view. Include them.
What that means is, assuming that HEAD~5 is also the 10-day
cut-off point on the mainline, and HEAD^ is recent, we would
send this:
o---o---o---o---o---*---*---*---*---*
\ ^ / ^
\ HEAD~5 / HEAD
\ /
*---*---*---*---*
^
HEAD^^2
I'll leave it as an exercise to the readers to Come up with the
pack generation specification and cauterizing instructions to
give to the downloader, but you see how quickly this becomes
very complex, and we still haven't talked about the above when
multiple merge bases are involved.
Thinking about it a bit more, maybe "from mainline's point of
view" policy would include the same entire side branch when
limiting by depth of commits. After all the whole side branch
was something new when HEAD^ merge was created. But that is
another policy decision. And the downloader side cannot make
that decision before seeing what the true ancestry graph looks
like.
Let's say we somehow managed to make a shallow clone. The
downloader has '*' commits:
o---o---o---o---o---*---*---*---*---*
\ ^ / ^
\ HEAD~5 / HEAD
\ /
o---o---*---*---*
^ ^
HEAD^^2~3 HEAD^^2
In the meantime, the uploader side made some more commits:
x---x---x---x---x---x
/ \
/ \
/ \
o---o---o---F---o---*---*---*---*---H---x---X
\ /
\ /
\ /
o---o---*---*---*
Now the downloader wants to update. How does that go?
- The uploader says the tip is at X
- The downloader says it has old HEAD (H), H^, ...
- Suppose the development on the new side branch recently
merged did not touch most of the files since it forked from
the mainline at F, but the mainline made lot of changes that
were not touched by commits 'x', which is now part of the
downloader's history. The uploader, if it did not know the
downloader has altered history, would take "have H" from the
downloader to mean "the other side has everything reachable
from H", so it would compute X..H (commits that are not in H
but in X and trees and blobs needed to complete them). But
that can leave out the trees and blobs that existed in F (and
was part of commits on the new side branch).
- So the uploader side needs to know how the history on the
downloader's side is altered from reality when interpreting
"have H". It does not mean "H and all the history behind it"
to the uploader anymore.
One way to do so is to send grafts information from downloader
to uploader. If the uploader _knows_ that the altered history
ends at the leftmost '*' commit on the mainline, then it can
take it into account the fact that the downloader does not have
commit 'F'. That, however, introduces another problem. Where
should it stop? Obviously X needs to be sent, so is X^, but
how do we know which 'x' commit to stop at? Do we record the
initial cut-off criteria (remember, we started the clone with
5-commit cutoff) somewhere in the downloader's repository and
send that to the uploader so that the uploader can apply the
same rule to include only 5 commits from the new side branch?
What happens if there are less than 5 commits on that new side
branch? Would we end up including 'F', which the downloader
specifically did not want when making this shallow clone
initially?
I won't go into the details, but when you think about what needs
to happen when fetching _from_ a shallow clone (i.e. the
uploader side is incomplete), your head will explode ;-). It is
solvable in the same sense that you can solve the previous
"fetching to update" problem by trying to define a reasonable
semantics to the operation by answering the questions I posed in
the above paragraph (which obviously is no way an exhaustive
set), but the result will be quite complex. It would involve
sending graft information both from uploader and downloader
(after all the downloader side can also be shallow but with
different cut-off points from the uploader) and somehow coming
up with the intersection of both to arrive at a shared view of
altered history before starting the traversal to find what to
send.
Also often people would want to deepen history (i.e. fetching
older history than initially fetched) after "checking out only
the tip". It is a reasonable request, and is somewhat easier
problem to solve than the initial shallow clone problem. If you
have a working initial shallow clone, essentially you can:
- Find the earlier cut off points by looking at your own grafts
file;
- Ask the uploader to clone shallowly from the parents missing
from your repository.
and update the resulting grafts information.
Anyway, it is a useful feature, an important operation, and
it involves a lot of thinking (and hard work, obviously).
I will not think about this anymore, until I have absolutely
nothing else to do for a solid few days, but you probably
haven't read this far ;-).
-jc
*1* Currently there is no way to find out what HEAD~10 is during
the initial protocol handshake, but this message is mostly about
potential protocol update needs.
*2* First parent of a merge is _usually_ what the repository
owner saw, i.e. the mainline merging side branch into it, but
that is not universally true when fast forwards are involved.
^ permalink raw reply
* Re: Change set based shallow clone
From: Junio C Hamano @ 2006-09-07 22:14 UTC (permalink / raw)
To: Jon Smirl; +Cc: git
In-Reply-To: <9e4733910609071341u7e430214j71ddcbefa26810ca@mail.gmail.com>
"Jon Smirl" <jonsmirl@gmail.com> writes:
> Does an average user do these things? The shallow clone is there to
> address the casual user who gags at a five hour download to get an
> initial check out Mozilla when they want to make a five line change or
> just browse the source for a few minutes.
>...
> Maybe the answer is to build a shallow clone tool for casual use, and
> then if you try to run anything too complex on it git just tells you
> that you have to download the entire tree.
For that kind of thing, "git-tar-tree --remote" would suffice I
would imagine. The five line change can be tracked locally by
creating an initial commit from the tar-tree extract; such a
casual user will not be pushing or asking to pull but sending in
patches to upstream, no?
^ permalink raw reply
* Re: Change set based shallow clone
From: Jakub Narebski @ 2006-09-07 22:40 UTC (permalink / raw)
To: git
In-Reply-To: <7vpse7tjp0.fsf@assigned-by-dhcp.cox.net>
Junio C Hamano wrote:
> One way to do so is to send grafts information from downloader
> to uploader. If the uploader _knows_ that the altered history
> ends at the leftmost '*' commit on the mainline, then it can
> take it into account the fact that the downloader does not have
> commit 'F'. That, however, introduces another problem. Where
> should it stop? Obviously X needs to be sent, so is X^, but
> how do we know which 'x' commit to stop at? Do we record the
> initial cut-off criteria (remember, we started the clone with
> 5-commit cutoff) somewhere in the downloader's repository and
> send that to the uploader so that the uploader can apply the
> same rule to include only 5 commits from the new side branch?
> What happens if there are less than 5 commits on that new side
> branch? Would we end up including 'F', which the downloader
> specifically did not want when making this shallow clone
> initially?
>
> I won't go into the details, but when you think about what needs
> to happen when fetching _from_ a shallow clone (i.e. the
> uploader side is incomplete), your head will explode ;-). It is
> solvable in the same sense that you can solve the previous
> "fetching to update" problem by trying to define a reasonable
> semantics to the operation by answering the questions I posed in
> the above paragraph (which obviously is no way an exhaustive
> set), but the result will be quite complex. It would involve
> sending graft information both from uploader and downloader
> (after all the downloader side can also be shallow but with
> different cut-off points from the uploader) and somehow coming
> up with the intersection of both to arrive at a shared view of
> altered history before starting the traversal to find what to
> send.
The proposed solution was to send graft information and cutoff from
downloader to uploader, coming up with the effective graft points which
would be intersection of downloader and uploader grafts[*1*], and then use
this intersection as graft information when calculating what to send.
With sparse clone we would have:
#---o---#---o---o---*---*---*---*---*
\ ^ / ^
\ HEAD~5 / HEAD
\ /
o---o---*---*---*
^ ^
HEAD^^2~3 HEAD^^2
where # and * are the commit which the downloader has (# are additional
commits for sparse rather thatn shallow clone).
x---x---x---x---x---x
/ \
/ \
/ \
#---o---#---F---o---*---*---*---*---H---x---X
\ /
\ /
\ /
o---o---*---*---*
Now, when we are shallow fetching, we should have remembered somewhere the
cutoff used (or use the same cutoff). So we sould get then
x---*---*---*---*---*
/ \
/ \
/ \
#---o---#---#---o---*---*---*---*---*---*---*
\ /
\ /
\ /
o---o---*---*---*
Footnotes:
[*1*] It would be some kind of graph intersection. Note that the repository
we clone from might be shallow clone, shallower in parts than our
repository, or might be repository with historic repository grafted on
(grafts adding history, not simplyfying it).
[*2*] I agree that shallow/sparse clone is hard. Lazy clone (or remote
alternatives) is also hard: when we do download objects, do we
download-ahead, do we cache downloaded object or move them to local
repository, how often we clean cache etc.
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
^ permalink raw reply
* Re: Change set based shallow clone
From: Jon Smirl @ 2006-09-07 23:09 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7vlkovtjd1.fsf@assigned-by-dhcp.cox.net>
On 9/7/06, Junio C Hamano <junkio@cox.net> wrote:
> "Jon Smirl" <jonsmirl@gmail.com> writes:
>
> > Does an average user do these things? The shallow clone is there to
> > address the casual user who gags at a five hour download to get an
> > initial check out Mozilla when they want to make a five line change or
> > just browse the source for a few minutes.
> >...
> > Maybe the answer is to build a shallow clone tool for casual use, and
> > then if you try to run anything too complex on it git just tells you
> > that you have to download the entire tree.
>
> For that kind of thing, "git-tar-tree --remote" would suffice I
> would imagine. The five line change can be tracked locally by
> creating an initial commit from the tar-tree extract; such a
> casual user will not be pushing or asking to pull but sending in
> patches to upstream, no?
>From my observation the casual user does something like this:
get a shallow clone
look at it for a while
pull once a day to keep it up to date
decide to make some changes
start a local branch
commit changes on local branch
push these changes to someone else for review
maybe pull changes on the branch back from the other person
keep pulling updates from the main repository
merge these updates to the local branch
browse around on the recent logs to see who changed what
finally push the branch to a maintainer
pull it back down from the main repository after the maintainer puts it in
abandon work on local branch
Some people can't figure out the branch step and just copy the repository.
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: [PATCH 2/3] Move color option parsing out of diff.c and into color.[ch]
From: Junio C Hamano @ 2006-09-07 23:56 UTC (permalink / raw)
To: Jeff King; +Cc: git
In-Reply-To: <20060907063559.GB17083@coredump.intra.peff.net>
Jeff King <peff@peff.net> writes:
> +#include <stdarg.h>
> +
> +#define COLOR_RESET "\033[m"
> +
> +static int
> +parse_color(const char *name, int len)
> +{
Style (applies to all functions you moved to this new file).
Some school of programming teach us to start the function name
at the beginning of the line, separate from its type. They say
that would make "grep '^parse_color'" to work better.
That happens to be the way I was taught (actually it was a bit
more strange that return type was to be indented by one TAB, so
it looked like this:
static int
parse_color(const char *name, int len)
). But git style is the kernel style, and I refrain from doing
that myself.
static int parse_color(const char *name, int len)
{
...
> +int
> +git_config_colorbool(const char *var, const char *value)
> +{
> + if (!value)
> + return 1;
> + if (!strcasecmp(value, "auto")) {
> + if (isatty(1) || (pager_in_use && pager_use_color)) {
> + char *term = getenv("TERM");
> + if (term && strcmp(term, "dumb"))
> + return 1;
> + }
> + return 0;
> + }
> + if (!strcasecmp(value, "never"))
> + return 0;
> + if (!strcasecmp(value, "always"))
> + return 1;
> + return git_config_bool(var, value);
> +}
> +int
> +color_printf(const char *color, const char *fmt, ...) {
Style.
int color_printf(const char *color, const char *fmt, ...)
{
...
^ permalink raw reply
* Re: [PATCH 2/3] Move color option parsing out of diff.c and into color.[ch]
From: Jeff King @ 2006-09-08 0:11 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7vfyf3s01w.fsf@assigned-by-dhcp.cox.net>
On Thu, Sep 07, 2006 at 04:56:43PM -0700, Junio C Hamano wrote:
> Some school of programming teach us to start the function name
> at the beginning of the line, separate from its type. They say
> that would make "grep '^parse_color'" to work better.
Hmm. I don't like that style either, I was just using it because I
thought that was the git style (which must have been caused by seeing
some other function(s) with the same style -- but grepping through, it
looks like there are very few functions using it). I'll fix the patches,
but will wait to resend until I hear other comments.
-Peff
^ permalink raw reply
* Re: [PATCH 3/3] git-commit.sh: convert run_status to a C builtin
From: Junio C Hamano @ 2006-09-08 0:20 UTC (permalink / raw)
To: Jeff King; +Cc: git
In-Reply-To: <20060907063621.GC17083@coredump.intra.peff.net>
Jeff King <peff@peff.net> writes:
> diff --git a/Makefile b/Makefile
> index 78748cb..d0ba3b5 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -252,7 +252,7 @@ LIB_OBJS = \
> fetch-clone.o revision.o pager.o tree-walk.o xdiff-interface.o \
> write_or_die.o trace.o \
> alloc.o merge-file.o path-list.o help.o unpack-trees.o $(DIFF_OBJS) \
> - color.o
> + color.o status.o
>
> BUILTIN_OBJS = \
> builtin-add.o \
At some point (this does not have to be part of this series), we
should demote DIFF_OBJS to just ordinary LIB_OBJS. They used to
be special in that they are used only by handful special
commands, but these days more and more things are integrated and
it outlived its usefulness.
> +static const char runstatus_usage[] =
> +"git-runstatus [--color|--nocolor] [--amend] [--verbose]";
> +
> +int cmd_runstatus(int argc, const char **argv, const char *prefix)
> +{
> + struct status s;
> + int i;
> +
> + git_config(git_status_config);
> + status_prepare(&s);
> +
> + for (i = 1; i < argc; i++) {
> + if (!strcmp(argv[i], "--color"))
> + status_use_color = 1;
> + else if (!strcmp(argv[i], "--nocolor"))
> + status_use_color = 0;
> + else if (!strcmp(argv[i], "--amend")) {
> + s.amend = 1;
> + s.reference = "HEAD^1";
> + }
> + else if (!strcmp(argv[i], "--verbose"))
> + s.verbose = 1;
> + else
> + usage(runstatus_usage);
> + }
> +
> + status_print(&s);
> + return s.commitable ? 0 : 1;
> +}
Quite funny indentation style your MUA likes ;-).
> diff --git a/status.c b/status.c
> new file mode 100644
> index 0000000..82aa881
> --- /dev/null
> +++ b/status.c
> @@ -0,0 +1,283 @@
> +#include "status.h"
"status.h" and "struct status" somehow sounds too broad.
Granted, "object.h" is also broad, but in git context "object"
has a specific meaning.
Having said that I cannot come up with a good alternative name.
It is not "project status" nor "repository status". It is
"working tree status", but that sounds very loooooooooooooooong.
> + color_printf(color(STATUS_HEADER), "#\t");
> + switch (p->status) {
> + case DIFF_STATUS_ADDED:
> + color_printf(c, "new file: %s\n", p->one->path); break;
Very nicely done. Especially I liked that you are careful not
to paint leading '#\t' (which is noticeable when you use reverse
as an attribute).
> +static void
> +status_print_untracked(const struct status *s)
> +{
> + struct dir_struct dir;
> + const char *x;
> + int i;
> + int shown_header = 0;
> +
> + memset(&dir, 0, sizeof(dir));
> +
> + dir.exclude_per_dir = ".gitignore";
> + x = git_path("info/exclude");
> + if (file_exists(x))
> + add_excludes_from_file(&dir, x);
> +
> + read_directory(&dir, ".", "", 0);
> + for(i = 0; i < dir.nr; i++) {
> + /* check for matching entry, which is unmerged; lifted from
> + * builtin-ls-files:show_other_files */
> + struct dir_entry *ent = dir.entries[i];
> + int pos = cache_name_pos(ent->name, ent->len);
> + struct cache_entry *ce;
> + if (0 <= pos)
> + die("bug in status_print_untracked");
> + pos = -pos - 1;
> + if (pos < active_nr) {
> + ce = active_cache[pos];
> + if (ce_namelen(ce) == ent->len &&
> + !memcmp(ce->name, ent->name, ent->len))
> + continue;
> + }
> + if (!shown_header) {
> + status_print_header("Untracked files",
> + "use \"git add\" to add to commit");
> + shown_header = 1;
> + }
> + color_printf(color(STATUS_HEADER), "#\t");
> + color_printf(color(STATUS_UNTRACKED), "%.*s\n",
> + ent->len, ent->name);
> + }
> +}
Very nice code reuse. I do not mean sarcasm -- the part copied
and pasted from ls-files is almost trivial to bother factoring
out. What's nice is read_directory() does all what is needed to
deal with .gitignore files, which I forgot almost all about.
> +int status_foreach_cached(status_cb cb);
> +int status_foreach_updated(status_cb cb);
> +int status_foreach_changed(status_cb cb);
> +int status_foreach_untracked(status_cb cb);
I do not see them defined nor used...
I'll take only [1/3] for now but I am interested in 2 and 3.
^ permalink raw reply
* Re: Add git-archive [take #2]
From: Junio C Hamano @ 2006-09-08 0:37 UTC (permalink / raw)
To: Franck; +Cc: git
In-Reply-To: <450019C3.4030001@innova-card.com>
Franck Bui-Huu <vagabon.xyz@gmail.com> writes:
> I'm sending a new version of the patchset which allows 'git-archive'
> and 'git-upload-archive' command. I tried to take into account all
> feedbacks made by Junio and Rene, but there are still some open points.
>
> 1/ Allow 'git-upload-archive' command to enable/disable some
> formats. This should be done by 'git-upload-archive'.
Perhaps. I was thinking about the way how a site administrator
can configure such when upload-archive is spawned via git-daemon
(for users coming from ssh and spawn an upload-archive on their
own, it's their own process and upload-archive has no business
deciding what is allowed and what is forbidden). Not very many
clean ways I can think of unfortunately.
> 2/ Can I remove 'git-upload-tar' command ?
> 3/ Should I kill 'git-zip-tree' command ?
We do not deprecate commands that easily. Notice we have kept
git-resolve for a long time (we should remove it and by now it
should be safe)?
Especially tar-tree --remote and upload-archive talks different
protocols, so it is not like not removing it is making your life
more difficult. Perhaps after next release (1.4.3 or 1.5? I
dunno) in the new development cycle, we would start saying
"don't use tar-tree --remote, use archive --fmt=tar --remote",
and then the release after that (1.4.4 or 1.5.1? Again I dunno)
we might remove it. The same thing for zip-tree, although that
one has lived shorter so it might not be missed if we remove it
earlier.
In any case, don't make removal of them as part of the series
please. Let's make sure this new toy works well first, and then
start talking about removing things that have become obsolete.
> 4/ Progress indicator support. Junio wants to mimic upload-pack for
> that. But it will lead in a lot of duplicated code if we don't
> try to share code. Can we copy that code anyways and clean up
> later ?
Refactoring first is always preferred, since "later" tends to
come very late (or worse, never) for clean-up tasks than feature
enhancements.
> 5/ Should we use "struct tree_desc" based interface for tree parsing
> ? According to Rene it doesn't worth it as soon as you actually
> start to do something to the trees
That became a non-issue I agree. Whichever is easier.
^ permalink raw reply
* Re: [RFC] gitweb wishlist and TODO list
From: Jakub Narebski @ 2006-09-08 0:44 UTC (permalink / raw)
To: git
In-Reply-To: <200609021817.09296.jnareb@gmail.com>
Jakub Narebski wrote:
> * Committags support, i.e. extend hyperlinking commit sha1 to commit
> view to hyperlinking bug #nn/BUG(nn) to the proper bugtracker, in both
> commit message view and (harder to do properly) in commit
> title/subject line which is hyperlink itself. The problem is how to do
> this general enough...
>
> I have thought about structure somewhat similar to %feature hash, i.e.
> something like %commitfilter hash, with 'enabled', 'override', 'name',
> 'options' or something like that and of course 'sub' (as we probably
> would use $cgi->a(...) to construct hyperlink, and not simple regexp
> replacement.
I have very preliminary work, where gitweb uses @enabled_committags as list
of committags to use, and %committags_info for actual committags info.
Examples of committags includes current linking of commit sha1,
gitweb-xmms2 linking of BUG(n) and FEATURE(n) to site-wide based URL
(Mantis), and RELEASE x.y.z to site-wide based URL (Wiki); perhaps "bug n"
to site-wide/project-wide URL (Bugzilla)... any other ideas?
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
^ permalink raw reply
* Re: Change set based shallow clone
From: linux @ 2006-09-08 1:01 UTC (permalink / raw)
To: git, jonsmirl, linux
In-Reply-To: <9e4733910609071252ree73effwb06358e9a22ba965@mail.gmail.com>
> Here's a change set based shallow clone scheme I've been thinking
> about, does it have potential?
Well, let's look at the problems...
You might want to look at the explanation of the current git network
protocol at
http://www.spinics.net/lists/git/msg08899.html
I'm just understanding it myself, but as best I understant it,
it has four phases:
- First, the server lists all of the heads it has, by SHA1 and name.
- Second, the client tells it which ones it wants, by name.
- Third, an interactive protocol is entered in which the client
tells the server which commits it has in terms the server can
understand. This proceeds as follows:
- For each head the client has, it lists commit objects going
backwards in history.
- As soon as the server sees one that it has heard of, it sends
back "ACK <hash> continue". The client stops tracing that
branch and starts in on the next one.
- (Periodically, the client sends a ping to the server, which
responds "NAK" if it's still alive.)
- This continues until the client says "done".
At this point, the server knows a set of commits which the client has,
and a set of commits which the client wants. Then it invokes
git-rev-list --objects-edge <want1> <want2> ^<have1> ^<have2>
(In practice, there will be a lot more than two of each, but forgive
me if I simplify.)
This builds a list of commits that are ancestors of <want1> and <want2>
but not <have1> or <have2>. (A commit is considered an ancestor of
itself, so <want1> and <want2> are in the set, while <have1> and <have2>
are not.)
Then it enumerates the complete trees of each of those commits. Then it
subtracts from that object set all of the trees and blobs in the <have1>
and <have2> commits.
This is a good approximation to the set of objects that the client needs
to be sent. It does not search the ancestors of <have1> and <have2>,
so it is possible that a copy of some object exists in an older commit,
and the state in <want1> is a reversion to a pre-<have1> state, but it's
unlikely and not worth looking for. Sending it redundantly is harmless.
Then, this set of objects is piped to git-pack-objects, which finds them
and builds a pack file. If the objects are stored locally as deltas
relative to other objects which are either to be sent, or are in the
<have1> or <have2> commits (which information is included in the data
output by git-rev-list), the delta is copied to the pack file verbatim.
Otherwise, more CPU is spent to pack it.
Again, the server is allowed to check for deltas against ancestors
of <have1> or <have2>, but doesn't bother for efficiency.
Fourth, the pack file is sent to the client, which unpacks it (and
discards any duplicates).
> When the client wants a shallow clone it starts by telling the server
> all of the HEADs and how many change sets down each of those HEADs it
> has locally. That's a small amout of data to transmit and it can be
> easily tracked. Let's ignore merged branches for the moment.
When you say "change set", I'm going to assume you mean "commit object".
Okay. Now, the server hasn't heard of one or more of those commit
objects, because they're local changes. What then?
Another issue is that a client with a nearly-full copy has to do a full
walk of its history to determine the depth count that it has. That can
be more than 2500 commits down in the git repository, and worse in the
mozilla one. It's actually pretty quick (git-show-branch --more=99999
will do it), but git normally tries to avoid full traversals like the
plague
Oh, and was "for the moment" supposed to last past the end of your e-mail?
I don't see what to do if there's a merge in the history and the depth
on different sides is not equal. E.g. the history looks like:
...a---b---c---d---e---f
/ \
...w---x---y HEAD
/
...p---q---r---s
Where "..." means that there are ancestors, but they're missing.
> If you haven't updated for six months when the server walks backwards
> for 10 change sets it's not going to find anything you have locally.
> When this situation is encountered the server needs to generate a
> delta just for you between one of the change sets it knows you have
> and one of the 10 change sets you want. By generating this one-off
> delta it lets you avoid the need to fetch all of the objects back to a
> common branch ancestor. The delta functions as a jump over the
> intervening space.
Your choice of words keeps giving me the impression that you believe
that a "change set" is a monolithic object that includes all the changes
made to all the files. It's neither monolithic nor composed of changes.
A commit objects consists soley of metadata, and contains a pointer to
a tree object, which points recursively to the entire project state at
the time of the commit.
There is massive sharing of component objects between successive
commits, but they are NOT stored as deltas relative to one another.
The pack-forming heuristics tend to achieve that effect, but it is not
guaranteed or required by design.
Please understand that, deep in your bones: git is based on snapshots,
not deltas.
But okay, so we've sent the client the latest 10 commits, with a dangling
tail at the bottom. (The files may have been sent as deltas against the
old state, or just fresh compressed copies, but that doesn't matter.)
Then the heads like "origin" have been advanced.
So the old commit history is now unreferenced garbage; nothing points
to it, and it will be deleted next time git-prune is run. Is that
the intended behavior? Or should updates to an existing clone always
complete the connections?
^ permalink raw reply
* Re: [RFC] gitweb wishlist and TODO list
From: Junio C Hamano @ 2006-09-08 1:16 UTC (permalink / raw)
To: Jakub Narebski; +Cc: git
In-Reply-To: <edqeco$ums$1@sea.gmane.org>
Jakub Narebski <jnareb@gmail.com> writes:
> I have very preliminary work, where gitweb uses @enabled_committags as list
> of committags to use, and %committags_info for actual committags info.
> Examples of committags includes current linking of commit sha1,
> gitweb-xmms2 linking of BUG(n) and FEATURE(n) to site-wide based URL
> (Mantis), and RELEASE x.y.z to site-wide based URL (Wiki); perhaps "bug n"
> to site-wide/project-wide URL (Bugzilla)... any other ideas?
Use a regexp as %committags key and its value to be whatever
that takes the matched string and munge it into a URL form
perhaps? A site that does not have commit tags do not have any
element in that hash.
^ permalink raw reply
* Re: Change set based shallow clone
From: Jon Smirl @ 2006-09-08 2:23 UTC (permalink / raw)
To: linux@horizon.com; +Cc: git
In-Reply-To: <20060908010112.6962.qmail@science.horizon.com>
On 7 Sep 2006 21:01:12 -0400, linux@horizon.com <linux@horizon.com> wrote:
> > When the client wants a shallow clone it starts by telling the server
> > all of the HEADs and how many change sets down each of those HEADs it
> > has locally. That's a small amout of data to transmit and it can be
> > easily tracked. Let's ignore merged branches for the moment.
>
> When you say "change set", I'm going to assume you mean "commit object".
>
> Okay. Now, the server hasn't heard of one or more of those commit
> objects, because they're local changes. What then?
Toss them, if they don't exist on the server the server is going to be
able to send any objects for them.
> Another issue is that a client with a nearly-full copy has to do a full
> walk of its history to determine the depth count that it has. That can
> be more than 2500 commits down in the git repository, and worse in the
> mozilla one. It's actually pretty quick (git-show-branch --more=99999
> will do it), but git normally tries to avoid full traversals like the
> plague
Client would track this incrementally and not recompute it each time.
> Oh, and was "for the moment" supposed to last past the end of your e-mail?
> I don't see what to do if there's a merge in the history and the depth
> on different sides is not equal. E.g. the history looks like:
>
> ...a---b---c---d---e---f
> / \
> ...w---x---y HEAD
> /
> ...p---q---r---s
>
> Where "..." means that there are ancestors, but they're missing.
>
> > If you haven't updated for six months when the server walks backwards
> > for 10 change sets it's not going to find anything you have locally.
> > When this situation is encountered the server needs to generate a
> > delta just for you between one of the change sets it knows you have
> > and one of the 10 change sets you want. By generating this one-off
> > delta it lets you avoid the need to fetch all of the objects back to a
> > common branch ancestor. The delta functions as a jump over the
> > intervening space.
>
> Your choice of words keeps giving me the impression that you believe
> that a "change set" is a monolithic object that includes all the changes
> made to all the files. It's neither monolithic nor composed of changes.
> A commit objects consists soley of metadata, and contains a pointer to
> a tree object, which points recursively to the entire project state at
> the time of the commit.
I was using change set to refer to snapshot.
> There is massive sharing of component objects between successive
> commits, but they are NOT stored as deltas relative to one another.
Yes, most of the sharing occurs via the tree structures.
> The pack-forming heuristics tend to achieve that effect, but it is not
> guaranteed or required by design.
>
> Please understand that, deep in your bones: git is based on snapshots,
> not deltas.
>
>
> But okay, so we've sent the client the latest 10 commits, with a dangling
> tail at the bottom. (The files may have been sent as deltas against the
> old state, or just fresh compressed copies, but that doesn't matter.)
> Then the heads like "origin" have been advanced.
>
> So the old commit history is now unreferenced garbage; nothing points
> to it, and it will be deleted next time git-prune is run. Is that
> the intended behavior? Or should updates to an existing clone always
> complete the connections?
If you follow the links in what looks to be a dangling object sooner
or latter you will run into the root object or a 'not present' object.
If you hit one of those the objects are not dangling and should be
preserved.
Here is another way to look at the shallow clone problem. The only
public ids in a git tree are the head and tag pointers. Send these to
the client. Now let's modify the git tools to fault the full objects
in one by one from the server whenever a git operation needs the
object. Dangling references would point to 'not-present' objects.
For a typical user using a model like this, how much of the Mozilla
repository would end up being faulted into their machine? Mozilla has
2M objects and 250K commits in a 450MB pack. My estimate is that a
typical user is going to touch less than 200K of the objects and maybe
less than 100K.
Of course always faulting in full objects is wasteful. A smart scheme
would be to try and anticipate with some read ahead and figure out
ways to send deltas. Tools like gitk would need to only touch the
objects needed to draw the screen and not run the full commit chain at
startup.
This experiment can be done fairly easily. Put all of the kernel
source into a single pack file. Modify the git tools to set a bit in
the index file if an object is accessed. Use the pack for a few days
and then dump out the results.
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: [PATCH 1/4] Add git-archive
From: Junio C Hamano @ 2006-09-08 2:35 UTC (permalink / raw)
To: Franck Bui-Huu; +Cc: git
In-Reply-To: <11576347251776-git-send-email-vagabon.xyz@gmail.com>
Franck Bui-Huu <vagabon.xyz@gmail.com> writes:
> The main reason for making this API is to avoid using
> git-{tar,zip}-tree commands, hence making them useless. Maybe it's
> time for them to die ?
The answer is "not yet" and the above paragraph, at least the
last sentence and half, do not belong to the commit log message.
> It also implements remote operations by defining a very simple
> protocol: it first sends the name of the specific uploader followed
> the repository name (git-upload-tar git://example.org/repo.git).
> Then it sends options. It's done by sending a sequence of one
> argument per packet, with prefix "argument ", followed by a flush.
I haven't looked the existing code closely recently, but my
impression was that if you want to make the protocol operable
with both git-daemon, ssh target, and local pipe, it is easier
to make the request message exactly like you would invoke the
remote command over the ssh connection in the target repository
(see connect.c). I am not sure how well the above
"git-upload-tar reponame" would work. I would have expected it
to be "git-upload-archive reponame" with the first on-protocol
parameter being "--fmt=tar".
Does your code work well when you run the remote archive
fetching locally, i.e. "git-archive --remote=../other.git",
I wonder?
... goes on and reads the patch, notices that the protocol
command is git-upload-archive with the archive location. Which
is GOOD. It is just the above description is a tad stale.
> diff --git a/archive.h b/archive.h
> new file mode 100644
> index 0000000..f33398e
> --- /dev/null
> +++ b/archive.h
> @@ -0,0 +1,41 @@
> +#ifndef ARCHIVE_H
> +#define ARCHIVE_H
> +
> +#define MAX_EXTRA_ARGS 32
> +#define MAX_ARGS (MAX_EXTRA_ARGS + 32)
> +
> +struct archiver_args {
> + const char *base;
> + struct tree *tree;
> + const unsigned char *commit_sha1;
> + time_t time;
> + const char **pathspec;
> + void *extra;
> +};
> +
> +typedef int (*write_archive_fn_t)(struct archiver_args *);
> +
> +typedef void *(*parse_extra_args_fn_t)(int argc, const char **argv);
> +
> +struct archiver {
> + const char *name;
> + const char *remote;
> + struct archiver_args args;
> + write_archive_fn_t write_archive;
> + parse_extra_args_fn_t parse_extra;
> +};
> +
> +extern struct archiver archivers[];
I thought the reason for archiver_args (and archiver_args.extra)
was because we wanted to avoid storing per invocation parameters
in static variables, which would hamper reentrancy. If one
process is creating two archives, both format=tar, it might be
reasonable for the code (future archiver enhancement, not your
current implementation of git-archive driver) to parse two sets
of parameters first (to get separate archiver instances) and
call their write_archive, but if archivers[] list has the
per-invocation parameter args then we are back to square one,
aren't we?
Reentrancy may not matter, but in any case the above archiver_args
is not helping enough to improve the situation, I think.
Actually you may be able to get away by returning a copy of
archivers[] element from get_archiver() when we need reentrancy
in the future. Of course the caller needs to free() it when it
is done with it, since it is a per-invocation handle.
> +void parse_treeish_arg(const char **argv, struct archiver_args *ar_args,
> + const char *prefix)
> +{
> +...
> + //free(tree);
> + ar_args->tree = tree;
> + ar_args->commit_sha1 = commit_sha1;
> + ar_args->time = archive_time;
> +}
Stray comment...
Overall looks good, except you already know your issue #4 ;-).
I haven't had a chance to look at connect.c code but I have a
mild suspicion that full reuse of upload-pack code by moving
everything into connect.c is not possible; at least the initial
handshake to determine if sideband is to be used is specific to
upload-pack protocol which needed to bolt-it-on to an existing
protocol to support older clients and servers, and I do not
think we would want to carry that baggage for this new protocol.
The pipe setup code in upload-pack.c::create_pack_file() is
quite specific to upload-pack. I am not sure how much of it can
be reused by refactoring, but it may be worth a try.
The part that reads from fd 1 and 2 and to multiplex into the
stream on the uploader side (the main while() loop in the same
create_pack_file() function, and send_client_data() function),
and the code to setup and demultiplex the bands on the
downloader side (setup_sideband() in fetch-clone.c) should be
reusable as-is, I think. They are defined as static so you
would need to move the code around to make them available from
elsewhere.
^ permalink raw reply
* Re: Change set based shallow clone
From: Martin Langhoff @ 2006-09-08 3:54 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Jon Smirl, git
In-Reply-To: <7vpse7tjp0.fsf@assigned-by-dhcp.cox.net>
On 9/8/06, Junio C Hamano <junkio@cox.net> wrote:
...
> Anyway, it is a useful feature, an important operation, and
> it involves a lot of thinking (and hard work, obviously).
>
> I will not think about this anymore, until I have absolutely
> nothing else to do for a solid few days, but you probably
> haven't read this far ;-).
For when you come back, then. I agree this is rather complicated, so
much so that I suspect that we may be barking up the wrong tree.
People who want shallow clones are actually asking for a "light"
clone, in terms of "how much do I need to download". If everyone has
the whole commit chain (but may be missing olden blobs, and even
trees), the problem becomes a lot easier.
On the wire, all the client needs to say from root commit X to later
commits A, B, C, D, can I have all the relevant blobs? (Some may end
up being resent, but that's a tradeoff).
On the server side, keeping commits (and/or trees) in a segregated
pack makes sense as an optimization. On the client side, a similar
segregated pack for the 'known blobless' commits/trees may make sense
to be able to be able treat ops on the blobless part of history
differently.
{From a user interaction point of view, it is also easier to error out
saying "don't have it, but you'll get it if you ask for history from
[SHA1] onwards".}
just my .2
m
^ permalink raw reply
* Re: Change set based shallow clone
From: Aneesh Kumar K.V @ 2006-09-08 5:05 UTC (permalink / raw)
To: git
In-Reply-To: <7vpse7tjp0.fsf@assigned-by-dhcp.cox.net>
Junio C Hamano wrote:
> I've been postponing thinking about shallow clones with a plan
> to do so when I have absolutely nothing else to do for a solid
> few days. Since my time is booked pretty much full with day job
> and the ongoing pack-file topics these days, I will most likely
> stick to that plan after I finish this message.
>
> Not because I think it is an insignificant problem at the fringe
> (I did not say "when I have nothing better to do"), but because
> I feel this is something important (and hard) that requires
> "think about it and only about it straight through for a few
> days, dropping everything else" kind of thinking.
>
> When you have a ref under .git/refs/ pointing at an object, you
> are making this assertion:
>
> The object store associated with this repository
> (usually, .git/objects and alternates pointed at by
> .git/objects/info/alternates) holds _all_ objects that
> are recursively reachable from this object by following
> tag reference, commit ancestry chain, commit to its
> tree, and tree to its subtrees and blobs.
>
> So by definition, a shallow clone is a broken/incomplete
> repository. Unless we fix the breakage in some way.
>
> One way to do so which has often been talked about is to use the
> grafts facility. You cut off the history at some commit, and
> cauterize the commit ancestry chain at that point by telling git
> to pretend that the commit does not have any parent. While this
> would work after setting it up, in the sense that "git log"
> would stop at the commit and "git fsck-objects" would say
> nothing is missing, it is cumbersome to set up, we do not have
> an easy way to fetch (or clone -- but clone is a special case of
> fetch where the downstream starts out from emptiness) from such
> a repository, unplugging to get histories further back with
> a later fetch is quite involved, and it is not clear what it
> means to further fetch from such a shallow clone.
Pardon my ignorance, but won't having a http:// or ssh based url support in objects/info/alternates
will support such shallow clones. What i am wondering is can't we make the objects/info/alternates following code
traverse the network transport and make git log work like cvs log in case things are not in local repository. This
should make all the git operation slow, but if you are interested in speed one can clone the repository fully.
-aneesh
^ permalink raw reply
* Re: Change set based shallow clone
From: Junio C Hamano @ 2006-09-08 5:30 UTC (permalink / raw)
To: Martin Langhoff; +Cc: Jon Smirl, git
In-Reply-To: <46a038f90609072054u5ec8bc46x9878a601953b2c5d@mail.gmail.com>
"Martin Langhoff" <martin.langhoff@gmail.com> writes:
> On 9/8/06, Junio C Hamano <junkio@cox.net> wrote:
> ...
>> Anyway, it is a useful feature, an important operation, and
>> it involves a lot of thinking (and hard work, obviously).
>>
>> I will not think about this anymore, until I have absolutely
>> nothing else to do for a solid few days, but you probably
>> haven't read this far ;-).
>
> For when you come back, then. I agree this is rather complicated, so
> much so that I suspect that we may be barking up the wrong tree.
>
> People who want shallow clones are actually asking for a "light"
> clone, in terms of "how much do I need to download". If everyone has
> the whole commit chain (but may be missing olden blobs, and even
> trees), the problem becomes a lot easier.
No, I do not think so. You are just pushing the same problem to
another layer.
Reachability through commit ancestry chain is no more special
than reachability through commit-tree or tree-containment
relationships. The grafts mechanism happen to treat commit
ancestry more specially than others, but that is not inherent
property of git [*1*]. It is only an implementation detail (and
limitation -- grafts cannot express anything but commit
ancestry) of the current system.
But let's touch a slightly different but related topic first.
People do not ask for shallow clones. They just want faster
transfer of "everything they care about". Shallow and lazy
clones are implementation techniques to achieve the goal of
making everything they care about appear available, typically
taking advantage of the fact that people care about recent past
a lot more than they do about ancient history.
Shallow clones do so by explicitly saying "sorry you told me
earlier you do not care about things older than X so if you now
care about things older than X please do such and such to deepen
your history". What you are saying is a variant of shallow that
says "NAK; commits I have but trees and blobs associated with
such an old commit you have to ask me to retrieve from
elsewhere". Lazy clones do so by "faulting missing objects on
demand" [*2*]. That is essentially automating the "please do
such and such" part. So they are all not that different.
Now, first and foremost, while I would love to have a system
that gracefully operates with a "sparse" repository that lacks
objects that should exist from tag/commit/tree/blob reachability
point of view, it is an absolute requirement that I can tell why
objects are missing from a repository when I find some are
missing by running fsck-objects [*3*]. If repository is a
shallow clone, not having some object may be expected, but I
want to be able to tell repository corruption locally even in
that case, so "assume lack of object is perhaps it was not
downloaded by shallow cloning" is an unacceptable attitude, and
"when you find an object missing from your repository, you can
ask the server [*4*], and if the server does not have it then
you know your repository is corrupt otherwise it is Ok" is a
wrong answer.
I talked about the need of upload-pack protocol extension to
exchange grafts information between uploader and downloader to
come up with the resulting graft and update the grafts in
downloaded repository after objects tranfer is done. It is
needed because by having such cauterizing graft entries I can be
sure that the repository is not corrupt when fsck-objects that
does look at grafts says "nothing is missing". Jon talked about
"fault-in on demand and leave stub objects until downloading the
real thing"; those stub objects are natural extension of grafts
facility but extends to trees and blobs. Either of these would
allow me to validate the sparse repository locally.
Now I'll really shut up ;-).
[*1*] For example, rev-list --object knows that it needs to show
the tree-containment when given only a tree object without any commit.
[*2*] You probably could write a FUSE module that mount on
your .git/objects, respond to accesses to .git/objects/??
directory and files with 38-byte hexadecimal names under them by
talking to other repositories over the wire. No, I am not going
to do it, and I will probably not touch it even with ten-foot
pole for reasons I state elsewhere in this message.
[*3*] The real fsck-objects always looks at grafts. The
fictitious version of fsck-objects I am talking about here is
the one that uses parents recorded on the "parent " line of
commit objects, that is, the true object level reachability.
[*4*] In git, there is no inherent server vs client or upstream
vs downstream relationship between repositories. You may be
even fetching from many people and do not have a set upstream at
all. Or you are _the_ upstream, and your notebook has the
latest devevelopment history, and after pushing that latest
history to your mothership repository, you may decide you do not
want ancient development history on a puny notebook, and locally
cauterize the history on your notebook repository and prune
ancient stuff. The objects missing from the notebook repository
are retrievable from your mothership repository again if/when
needed and you as the user would know that (and if you are lucky
you may even remember that a few months later), but git doesn't,
and there is no reason for git to want to know it. If we want
to do "fault-in on demand", we need to add a way to say "it is
Ok for this repository to be incomplete -- objects that are
missing must be completed from that repository (or those
repositories)". But that's quite a special case of having a
fixed upstream.
^ permalink raw reply
* Re: [PATCH 3/3] git-commit.sh: convert run_status to a C builtin
From: Jeff King @ 2006-09-08 5:42 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7vzmdbqke3.fsf@assigned-by-dhcp.cox.net>
On Thu, Sep 07, 2006 at 05:20:20PM -0700, Junio C Hamano wrote:
> "status.h" and "struct status" somehow sounds too broad.
> Granted, "object.h" is also broad, but in git context "object"
> has a specific meaning.
I agree it is quite broad (as is git-runstatus). Conceptually it's
another type of diff format, but making it a diff argument doesn't
really makes much sense. We're at least not introducing any broadness,
since there is already git-status; are we interested in fixing that
name?
Names for similar concepts from other systems: tree-lint (arch),
inventory (arch), whatsnew (darcs, perhaps a little too close to
whatchanged), status (svn, hg), update -n (cvs, ugh!).
> Having said that I cannot come up with a good alternative name.
> It is not "project status" nor "repository status". It is
> "working tree status", but that sounds very loooooooooooooooong.
wt_status? ucu_status (updated, changed, untracked)?
> Very nicely done. Especially I liked that you are careful not
> to paint leading '#\t' (which is noticeable when you use reverse
> as an attribute).
Yes. I seem to recall some issues raised about color attributes
persisting over a newline, but I can't find any reference to it now.
Using color_printf makes sure every color is 'closed' but it sometimes
includes the newline in the colorized portion. Does anybody object to
that?
> Very nice code reuse. I do not mean sarcasm -- the part copied
> and pasted from ls-files is almost trivial to bother factoring
Yes, I don't like cutting and pasting, but it seemed more confusing to
try factoring it out. It's not a lot of *code*, it's just that
remembering to pick out unmerged entries (and the weird -pos bit) is
confusing.
> out. What's nice is read_directory() does all what is needed to
> deal with .gitignore files, which I forgot almost all about.
I just had to write a file_exists to check for .git/gitignore, which was
a little clumsy.
> > +int status_foreach_cached(status_cb cb);
> > +int status_foreach_updated(status_cb cb);
> > +int status_foreach_changed(status_cb cb);
> > +int status_foreach_untracked(status_cb cb);
> I do not see them defined nor used...
My fault, they were left in from a previous iteration.
> I'll take only [1/3] for now but I am interested in 2 and 3.
OK. Besides the things you mentioned, what improvements would you like
to see?
-Peff
^ permalink raw reply
* Re: [PATCH 3/3] git-commit.sh: convert run_status to a C builtin
From: Junio C Hamano @ 2006-09-08 5:56 UTC (permalink / raw)
To: Jeff King; +Cc: git
In-Reply-To: <20060908054226.GA19537@coredump.intra.peff.net>
Jeff King <peff@peff.net> writes:
> On Thu, Sep 07, 2006 at 05:20:20PM -0700, Junio C Hamano wrote:
>
>> "status.h" and "struct status" somehow sounds too broad.
>> Granted, "object.h" is also broad, but in git context "object"
>> has a specific meaning.
>
> I agree it is quite broad (as is git-runstatus). Conceptually it's
> another type of diff format, but making it a diff argument doesn't
> really makes much sense. We're at least not introducing any broadness,
> since there is already git-status; are we interested in fixing that
> name?
The command name is a name exposed to the end user. We do not
currently have a command to give "repository status", and even
if we had one such a specialized command for repository
administrators would be called git-repository-status so
git-status is definitely fine as is.
I just wanted to point it out because I felt the names to
programmers are slightly different matter.
> wt_status? ucu_status (updated, changed, untracked)?
Yeah, something along those lines.
>> Very nicely done. Especially I liked that you are careful not
>> to paint leading '#\t' (which is noticeable when you use reverse
>> as an attribute).
>
> Yes. I seem to recall some issues raised about color attributes
> persisting over a newline, but I can't find any reference to it now.
> Using color_printf makes sure every color is 'closed' but it sometimes
> includes the newline in the colorized portion. Does anybody object to
> that?
In a distant past I saw some terminals get confused near the
edge if you do that, but these days everybody is on some sort of
Xterm so it may not matter. But that would probably be nice to
fix.
> OK. Besides the things you mentioned, what improvements would you like
> to see?
Besides the things I mentioned? I dunno offhand -- otherwise I
would have mentioned them ;-).
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox