* Re: Summer of Code project ideas due this Friday
2011-03-09 21:58 ` Summer of Code project ideas due this Friday Jeff King
@ 2011-03-10 0:10 ` Jonathan Nieder
2011-03-10 16:30 ` Jeff King
` (3 more replies)
2011-03-10 0:19 ` Nguyen Thai Ngoc Duy
` (4 subsequent siblings)
5 siblings, 4 replies; 63+ messages in thread
From: Jonathan Nieder @ 2011-03-10 0:10 UTC (permalink / raw)
To: Jeff King
Cc: Shawn Pearce, Ramkumar Ramachandra, Jens Lehmann,
Christian Couder, Thomas Rast, git
Hi,
Jeff King wrote:
> If you have any ideas, please add them to the page!
Thanks for a pointer. Some ideas still at the "throw them against the
wall and see if they stick" stage: please feel free to add to the page
if you think you can find subsets with the right scope. Later today I
can look more carefully.
1. Cross-compilable msysgit: ideally, allowing someone to run
"make msysgit-installer" on Linux from an msysgit-source.git repo
to get an installer. Nice subsets to start with might be msys.dll
(I think there has been some work on that already) and the basic
msys utilities.
2. Merge libgit2 into git.git: polish the existing code in libgit2 so
that standard git can use it. Presumably this would happen by
porting one function at a time, not by making git link to libgit2
immediately. People familiar with libgit2 might be able to pick
out particularly interesting subsets to start with (diff
generation?).
3. Remote helpers: bidi git remote-svn (or one-way remote-hg, or
remote-cvs, or ...).
4. filter-branch killer: using fast-import's new features to implement
common filter-branch operations (--subdirectory-filter,
--prune-empty, obliterating certain files) faster.
5. rev-list: Coping with timeskew.
6. shallow clone: Push support.
7. fixing workdirs (and alternates), along the lines explained in the
thread <http://thread.gmane.org/gmane.comp.version-control.git/150559>.
I would be very much interested in this.
7. packfilev4.
^ permalink raw reply [flat|nested] 63+ messages in thread* Re: Summer of Code project ideas due this Friday
2011-03-10 0:10 ` Jonathan Nieder
@ 2011-03-10 16:30 ` Jeff King
2011-03-10 17:31 ` Shawn Pearce
2011-03-10 17:15 ` Thomas Rast
` (2 subsequent siblings)
3 siblings, 1 reply; 63+ messages in thread
From: Jeff King @ 2011-03-10 16:30 UTC (permalink / raw)
To: Jonathan Nieder
Cc: Shawn Pearce, Ramkumar Ramachandra, Jens Lehmann,
Christian Couder, Thomas Rast, git
On Wed, Mar 09, 2011 at 06:10:17PM -0600, Jonathan Nieder wrote:
> 2. Merge libgit2 into git.git: polish the existing code in libgit2 so
> that standard git can use it. Presumably this would happen by
> porting one function at a time, not by making git link to libgit2
> immediately. People familiar with libgit2 might be able to pick
> out particularly interesting subsets to start with (diff
> generation?).
That's definitely bigger than a SoC project at this point. However, one
smaller project might be to start implementing a subset of git commands
(probably plumbing) on top of libgit2, and then see how well the result
passes git.git tests.
> 4. filter-branch killer: using fast-import's new features to implement
> common filter-branch operations (--subdirectory-filter,
> --prune-empty, obliterating certain files) faster.
Yeah, I like that. Or maybe somebody can pick up git-sequencer, which in
theory could be used to filter-branch, too (or maybe sequencer can go on
top of fast-import? :) ).
> 5. rev-list: Coping with timeskew.
I don't think this is big enough for a SoC project. It's on my todo list
for the near-term, though.
> 7. packfilev4.
I suspect that is too complex for a SoC project. But you never know.
-Peff
^ permalink raw reply [flat|nested] 63+ messages in thread* Re: Summer of Code project ideas due this Friday
2011-03-10 16:30 ` Jeff King
@ 2011-03-10 17:31 ` Shawn Pearce
2011-03-10 21:43 ` Alexander Miseler
0 siblings, 1 reply; 63+ messages in thread
From: Shawn Pearce @ 2011-03-10 17:31 UTC (permalink / raw)
To: Jeff King
Cc: Jonathan Nieder, Ramkumar Ramachandra, Jens Lehmann,
Christian Couder, Thomas Rast, git
On Thu, Mar 10, 2011 at 08:30, Jeff King <peff@peff.net> wrote:
>
>> 7. packfilev4.
>
> I suspect that is too complex for a SoC project. But you never know.
It is. Nico is also probably working on it.
My big concern with pack v4 right now is the on-disk format of
commits/trees is different from the current wire protocol. If a pack
v2 only client connected to a server using pack v4, the server would
have to spend a lot of time to inflate/deflate about 60% of the
objects in the repository (commits and trees). That is a lot of CPU
time. So even if a SoC-er finished the work, it probably wouldn't be
merged because of this penalty.
--
Shawn.
^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: Summer of Code project ideas due this Friday
2011-03-10 17:31 ` Shawn Pearce
@ 2011-03-10 21:43 ` Alexander Miseler
0 siblings, 0 replies; 63+ messages in thread
From: Alexander Miseler @ 2011-03-10 21:43 UTC (permalink / raw)
To: Shawn Pearce
Cc: Jeff King, Jonathan Nieder, Ramkumar Ramachandra, Jens Lehmann,
Christian Couder, Thomas Rast, git
On 10.03.2011 18:31, Shawn Pearce wrote:
> On Thu, Mar 10, 2011 at 08:30, Jeff King <peff@peff.net> wrote:
>>
>>> 7. packfilev4.
>>
>> I suspect that is too complex for a SoC project. But you never know.
>
> It is. Nico is also probably working on it.
>
> My big concern with pack v4 right now is the on-disk format of
> commits/trees is different from the current wire protocol. If a pack
> v2 only client connected to a server using pack v4, the server would
> have to spend a lot of time to inflate/deflate about 60% of the
> objects in the repository (commits and trees). That is a lot of CPU
> time. So even if a SoC-er finished the work, it probably wouldn't be
> merged because of this penalty.
I'm very interested in this. Can anyone please point me towards further information on the packfile format and proposed/in-progress changes?
^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: Summer of Code project ideas due this Friday
2011-03-10 0:10 ` Jonathan Nieder
2011-03-10 16:30 ` Jeff King
@ 2011-03-10 17:15 ` Thomas Rast
2011-03-10 18:17 ` Santi Béjar
2011-03-10 18:46 ` Jeff King
2011-03-10 17:39 ` Jakub Narebski
2011-03-13 17:08 ` Summer of Code project ideas due this Friday Ramkumar Ramachandra
3 siblings, 2 replies; 63+ messages in thread
From: Thomas Rast @ 2011-03-10 17:15 UTC (permalink / raw)
To: Jonathan Nieder
Cc: Jeff King, Shawn Pearce, Ramkumar Ramachandra, Jens Lehmann,
Christian Couder, git
Jonathan Nieder wrote:
> Thanks for a pointer. Some ideas still at the "throw them against the
> wall and see if they stick" stage: please feel free to add to the page
> if you think you can find subsets with the right scope.
Some stuff off the top of my head, please apply a similar filter:
* Word-based merge helper
The existing merge algorithms are all tailored to line-based formats
such as code. Writing, e.g., LaTeX or even asciidoc requires
sticking to a strict word-wrapped format. Worse even, re-wrapping
leads to headaches if people work on the same areas a lot, much like
the effects of code reindents.
Something similar was already proposed in 2010, IIRC by Dscho:
https://git.wiki.kernel.org/index.php/SoC2010Ideas#Merge_helper_for_LaTeX_files
One possible angle of attack given --word-diff=porcelain would be:
1. Fix --word-diff to properly represent both sides of the diff at
least optionally. (It was observed in a list post some months
ago that it doesn't even represent *one* side faithfully.)
2. Use --word-diff=porcelain as input to some to-be-written merge
algorithm.
* Clean up add -p
git-add--interactive.perl became a bit of a mess. Partly due to my
own efforts with {checkout,stash,...} it has bolted-on interfaces to
other commands. There are some UI issues that simply fall out of
its design, e.g., you cannot go back from one file to another,
Ctrl-C stops applying to the current file but does not discard
earlier files, etc. And that's not saying anything about 'add -i'
which I don't really know.
This would probably not be a very fun project, but it could add a
little edge of usability to the tool and it's probably one of the
few pure-Perl ideas we can offer.
> 4. filter-branch killer: using fast-import's new features to implement
> common filter-branch operations (--subdirectory-filter,
> --prune-empty, obliterating certain files) faster.
I would phrase this as follows:
* fast-import-filter
Implement a utility that can execute certain transformations to
fast-import streams, such as:
- delete or rename entire files/directories
- split out subtrees
- ...
Ideally, this should be a thin command line interface around a set
of primitives (such as a Perl package) that allow a competent
scripter to go beyond the CLI. (There have been threads on this.)
Later on, make an interface that automatically does
git fast-export | fast-import-filter | git fast-import
with suitable options. Nice because it should be largely orthogonal
to everything except the fast-import stream format.
In addition, Jeff wrote:
> Yeah, I like that. Or maybe somebody can pick up git-sequencer, which in
> theory could be used to filter-branch, too (or maybe sequencer can go on
> top of fast-import? :) ).
I'd be interested to hear how that goes, because I think the tools are
fundamentally different. The rebase and thus sequencer family is
delta-based, and the fast-import and filter-branch families are
tree-based. Feel free to prove me wrong of course.
--
Thomas Rast
trast@{inf,student}.ethz.ch
^ permalink raw reply [flat|nested] 63+ messages in thread* Re: Summer of Code project ideas due this Friday
2011-03-10 17:15 ` Thomas Rast
@ 2011-03-10 18:17 ` Santi Béjar
2011-03-10 18:46 ` Jeff King
1 sibling, 0 replies; 63+ messages in thread
From: Santi Béjar @ 2011-03-10 18:17 UTC (permalink / raw)
To: Thomas Rast
Cc: Jonathan Nieder, Jeff King, Shawn Pearce, Ramkumar Ramachandra,
Jens Lehmann, Christian Couder, git
On Thu, Mar 10, 2011 at 6:15 PM, Thomas Rast <trast@student.ethz.ch> wrote:
> Jonathan Nieder wrote:
>> Thanks for a pointer. Some ideas still at the "throw them against the
>> wall and see if they stick" stage: please feel free to add to the page
>> if you think you can find subsets with the right scope.
>
> Some stuff off the top of my head, please apply a similar filter:
>
> * Word-based merge helper
>
> The existing merge algorithms are all tailored to line-based formats
> such as code. Writing, e.g., LaTeX or even asciidoc requires
> sticking to a strict word-wrapped format. Worse even, re-wrapping
> leads to headaches if people work on the same areas a lot, much like
> the effects of code reindents.
Not a complete/right solution (and not even extensively tested), but I
did a "trivial" merge program for LaTeX files that:
1.- Rewraps all files as the first one, using dwdiff
2.- Perform a normal line based merge
[wlmerge]
#!/bin/bash
tmp1=$(mktemp -t wlmerge.XXXXXXX)
tmp2=$(mktemp -t wlmerge.XXXXXXX)
trap "rm -f $tmp1 $tmp2" 0 1 2 3 15
[ ! -e "$1" ] && exit 1
file1=$1 && shift
[ ! -e "$1" ] && exit 1
orig=$1 && shift
[ ! -e "$1" ] && exit 1
file2=$1 && shift
dwdiff -2 -w "" -x "" "$orig" "$file1" > $tmp1
dwdiff -2 -w "" -x "" "$file2" "$file1" > $tmp2
merge "$file1" $tmp1 $tmp2
I even have a "wldiff", but nowadays I use --color-words exclusively!
HTH,
Santi
^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: Summer of Code project ideas due this Friday
2011-03-10 17:15 ` Thomas Rast
2011-03-10 18:17 ` Santi Béjar
@ 2011-03-10 18:46 ` Jeff King
2011-03-10 19:21 ` Junio C Hamano
2011-03-11 13:31 ` Thomas Rast
1 sibling, 2 replies; 63+ messages in thread
From: Jeff King @ 2011-03-10 18:46 UTC (permalink / raw)
To: Thomas Rast
Cc: Jonathan Nieder, Shawn Pearce, Ramkumar Ramachandra, Jens Lehmann,
Christian Couder, git
On Thu, Mar 10, 2011 at 06:15:23PM +0100, Thomas Rast wrote:
> * Clean up add -p
>
> git-add--interactive.perl became a bit of a mess. Partly due to my
> own efforts with {checkout,stash,...} it has bolted-on interfaces to
> other commands. There are some UI issues that simply fall out of
> its design, e.g., you cannot go back from one file to another,
> Ctrl-C stops applying to the current file but does not discard
> earlier files, etc. And that's not saying anything about 'add -i'
> which I don't really know.
>
> This would probably not be a very fun project, but it could add a
> little edge of usability to the tool and it's probably one of the
> few pure-Perl ideas we can offer.
One more wishlist item for this. I use "add -p" for almost all of my
adds these days, because I like the final review check. So after a
conflicted merge, I find myself doing "add -p" to stage my resolution.
The current behavior is that it shows the --cc diff and exits. It would
be cool to handle staging the resolution, which would involve converting
the combined diff into something that can be applied.
> I'd be interested to hear how that goes, because I think the tools are
> fundamentally different. The rebase and thus sequencer family is
> delta-based, and the fast-import and filter-branch families are
> tree-based. Feel free to prove me wrong of course.
Hmm, yeah, that is a fundamental difference. It might be possible to
overcome it, but I admit I haven't really given it any thought yet at
this point.
-Peff
^ permalink raw reply [flat|nested] 63+ messages in thread* Re: Summer of Code project ideas due this Friday
2011-03-10 18:46 ` Jeff King
@ 2011-03-10 19:21 ` Junio C Hamano
2011-03-10 19:28 ` Jeff King
2011-03-11 13:31 ` Thomas Rast
1 sibling, 1 reply; 63+ messages in thread
From: Junio C Hamano @ 2011-03-10 19:21 UTC (permalink / raw)
To: Jeff King
Cc: Thomas Rast, Jonathan Nieder, Shawn Pearce, Ramkumar Ramachandra,
Jens Lehmann, Christian Couder, git
Jeff King <peff@peff.net> writes:
> One more wishlist item for this. I use "add -p" for almost all of my
> adds these days, because I like the final review check. So after a
> conflicted merge, I find myself doing "add -p" to stage my resolution.
> The current behavior is that it shows the --cc diff and exits. It would
> be cool to handle staging the resolution, which would involve converting
> the combined diff into something that can be applied.
I think the end-result would be a nice feature. I suspect that it would
not involve conversion from --cc, but more like using the difference
between the HEAD and the working tree, generated as if there is no
multi-stage index.
^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: Summer of Code project ideas due this Friday
2011-03-10 19:21 ` Junio C Hamano
@ 2011-03-10 19:28 ` Jeff King
2011-03-10 20:54 ` Junio C Hamano
0 siblings, 1 reply; 63+ messages in thread
From: Jeff King @ 2011-03-10 19:28 UTC (permalink / raw)
To: Junio C Hamano
Cc: Thomas Rast, Jonathan Nieder, Shawn Pearce, Ramkumar Ramachandra,
Jens Lehmann, Christian Couder, git
On Thu, Mar 10, 2011 at 11:21:47AM -0800, Junio C Hamano wrote:
> Jeff King <peff@peff.net> writes:
>
> > One more wishlist item for this. I use "add -p" for almost all of my
> > adds these days, because I like the final review check. So after a
> > conflicted merge, I find myself doing "add -p" to stage my resolution.
> > The current behavior is that it shows the --cc diff and exits. It would
> > be cool to handle staging the resolution, which would involve converting
> > the combined diff into something that can be applied.
>
> I think the end-result would be a nice feature. I suspect that it would
> not involve conversion from --cc, but more like using the difference
> between the HEAD and the working tree, generated as if there is no
> multi-stage index.
The trouble is that I would like to see the combined diff, then say "OK"
and have it apply the result to the index. But because we work on a
per-hunk basis, you need to match the combined diff hunks to the regular
diff hunks, taking into account that hunks could be split. Which maybe
is straightforward, but I haven't convinced myself yet that there are no
corner cases where they don't line up.
I guess one could argue that I should be using mergetool, and making a
mergetool helper that is more like "add -p" (show hunks, say "choose
left, choose right, choose none"). But I generally fix up the conflict
using an editor and just want "add -p" as a final check.
I'll probably look into it more eventually, but it's not a high priority
at this point.
-Peff
^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: Summer of Code project ideas due this Friday
2011-03-10 19:28 ` Jeff King
@ 2011-03-10 20:54 ` Junio C Hamano
2011-03-10 21:42 ` Jeff King
0 siblings, 1 reply; 63+ messages in thread
From: Junio C Hamano @ 2011-03-10 20:54 UTC (permalink / raw)
To: Jeff King
Cc: Thomas Rast, Jonathan Nieder, Shawn Pearce, Ramkumar Ramachandra,
Jens Lehmann, Christian Couder, git
Jeff King <peff@peff.net> writes:
>> I think the end-result would be a nice feature. I suspect that it would
>> not involve conversion from --cc, but more like using the difference
>> between the HEAD and the working tree, generated as if there is no
>> multi-stage index.
>
> The trouble is that I would like to see the combined diff, then say "OK"
> and have it apply the result to the index. But because we work on a
> per-hunk basis, you need to match the combined diff hunks to the regular
> diff hunks, taking into account that hunks could be split. Which maybe
> is straightforward, but I haven't convinced myself yet that there are no
> corner cases where they don't line up.
Forgetting for now the implementation, I _think_ what you would want is
for "git add -p" to notice that you are resolving conflicts, and do not
bother you about cleanly merged parts (I take it is a given that you would
always want to add them to the index without even inspecting when running
"add -p"), and make the per-hunk selection loop ask only about the parts
that originally had conflicts.
But it is rather hard to arrange. Neither "--cc" nor "-c" during the
merge is about showing conflicts but is about showing the result with
respect to the two originals. If you resolved conflicts in your editor
already, there are no lines with "<<</>>>" markers that are different from
either original to cause the "conflicted parts" to appear in their output.
If your conflict resolution ended up in taking what one side did, "--cc"
will hide it as a non-event. So at least you would be using "-c" to
implement this.
Also, you have to remember that "add -p" is about adding the state you
deem Ok incrementally to the index, that once you add a path to the index
the higher stages for the path will collapse to stage #0, and that "-c"
and "--cc" make their comparison based on what you have in higher stages.
I would imagine that a workable implementation might look like this:
0. The solution introduces a new index extension, "PRSF" (partial
resolution so far)". This is a mapping from pathname to a blob object
name.
1. "git add -p" notices that you are in the middle of conflict
resolution, by noticing that you have higher stages to the path.
2. Look up the path from the PRSF extension. If there is no entry for
the path, recreate the content-level 3-way merge using the content of
three stages (i.e. the state immediately after the mergy operation
that caused the conflicts before you touched the corresponding file in
your working tree), and register this image (with the full glory of
"<<</>>>" conflict markers) to the extension. If there already is an
entry in the extension for the path, skip this step.
3. The patch the user will see in the interactive hunk selection is the
difference between the working tree and the PRSF image, not your
regular index (nor the HEAD version). This allows you to see how you
resolved the conflict incrementally so far.
4. Choosing a hunk to "apply" would not affect the index entry for the
path, as it would collapse its higher stages. It instead updates the
blob registered in the PRSF extension using the hunk you are applying.
If you are running "git add -p" incrementally (i.e. edit a little, review
with diff, then "add -p" the part you are sure about, and repeat the whole
thing), the next invocation of "git add -p" would notice that you still
have unresolved conflicts in PRSF. It will skip the step 2 above and the
step 3 will let you review the remaining difference between the PRSF image
(which you updated in step 4 during the last round) and what you have in
the working tree. The step 4 will update the PRSF image for the next
invocation of "git add -p". And you continue until you are done.
At the end (we would probably need a good way to detect the user might
want to declare "end" automatically for a good user experience), use the
PRSF image to collapse the higher stages for the path down to stage 0.
^ permalink raw reply [flat|nested] 63+ messages in thread* Re: Summer of Code project ideas due this Friday
2011-03-10 20:54 ` Junio C Hamano
@ 2011-03-10 21:42 ` Jeff King
2011-03-10 22:58 ` Junio C Hamano
0 siblings, 1 reply; 63+ messages in thread
From: Jeff King @ 2011-03-10 21:42 UTC (permalink / raw)
To: Junio C Hamano
Cc: Thomas Rast, Jonathan Nieder, Shawn Pearce, Ramkumar Ramachandra,
Jens Lehmann, Christian Couder, git
On Thu, Mar 10, 2011 at 12:54:29PM -0800, Junio C Hamano wrote:
> Forgetting for now the implementation, I _think_ what you would want is
> for "git add -p" to notice that you are resolving conflicts, and do not
> bother you about cleanly merged parts (I take it is a given that you would
> always want to add them to the index without even inspecting when running
> "add -p"), and make the per-hunk selection loop ask only about the parts
> that originally had conflicts.
Yes, I don't want to see cleanly merged parts. And "--cc" already does
what I want by not showing them. But of course they still need to be
added to the index.
So my thinking was more along the lines of:
1. Get "git diff HEAD file" and store its hunks.
2. Get "git diff --cc file" and stores its hunks.
3. For each hunk in (1), if it does not have an analagous hunk in (2),
mark it for staging without asking the user.
4. For the remaining hunks in (1), show the user the analagous --cc
hunk from (2), and mark the hunk from (1) for staging if requested.
5. Create the final patch from the marked hunks, apply it to
HEAD:file, and put that in the index.
There are two issues (and I think you know this, but it took me
reasoning out why this wouldn't work to quite understand what you were
saying in your email, so I'll elaborate here for the benefit of other
readers):
a. Step (3) glosses over the definition of "analagous hunk". In simple
cases the hunk headers match up (i.e., they start at the same
offsets). But the --cc diff is actually a diff against the merge
base, and the diff in step (1) is against HEAD. So actually we
would want step (1) and (5) to deal not with the file in HEAD, but
the file in the merge-base.
Or we can use "-c" in the first place, which gives us interesting
and uninteresting parts, and then suppress the uninteresting ones.
b. You can't stage part of a resolution. This is the "adding a path
collapses stages to #0" that you mentioned. So either you need an
index extension, or you need to make it all-or-nothing.
> [discussion of how this could be done right]
That description makes sense to me, but is way overkill for my workflow.
I think at its simplest, what I would like is to be shown the entire
--cc diff, have it say "do you want this and all of the cleanly merged
bits added?" and then either stage the whole thing or not.
Which really I could do with:
for i in `git diff-files --name-only --diff-filter=U`; do
git diff --cc $i
echo 'OK?'
read r
test "$r" = y && git add $i
done
It would just be a little nicer to have it integrated into the "git add
-p" loop.
Even though it is obviously a less featureful solution than what you
proposed, I am tempted to implement it anyway because the current
behavior is so bad (it shows the diff but doesn't consider it a
selectable hunk at all, so it just skips it, exiting immediately if
there are no other files).
And then if somebody wants to spend time allowing partial resolution
selection, it would be a nice improvement on top of that.
-Peff
^ permalink raw reply [flat|nested] 63+ messages in thread* Re: Summer of Code project ideas due this Friday
2011-03-10 21:42 ` Jeff King
@ 2011-03-10 22:58 ` Junio C Hamano
2011-03-10 23:09 ` Jeff King
0 siblings, 1 reply; 63+ messages in thread
From: Junio C Hamano @ 2011-03-10 22:58 UTC (permalink / raw)
To: Jeff King
Cc: Thomas Rast, Jonathan Nieder, Shawn Pearce, Ramkumar Ramachandra,
Jens Lehmann, Christian Couder, git
Jeff King <peff@peff.net> writes:
> Yes, I don't want to see cleanly merged parts. And "--cc" already does
> what I want by not showing them.
Are you sure about that?
What "--cc" would show largely depends on what you have in your working
tree. If your two branches fixed the same bug with very different
approaches, you may resolve the conflict favouring what one side did while
discarding everything the other side did. The file in your work tree might
be the same as "git checkout --ours $that_path" after a mergy operation.
"diff --cc" won't show anything to you in such a case.
> So my thinking was more along the lines of:
>
> 1. Get "git diff HEAD file" and store its hunks.
>
> 2. Get "git diff --cc file" and stores its hunks.
>
> 3. For each hunk in (1), if it does not have an analagous hunk in (2),
> mark it for staging without asking the user.
>
> 4. For the remaining hunks in (1), show the user the analagous --cc
> hunk from (2), and mark the hunk from (1) for staging if requested.
>
> 5. Create the final patch from the marked hunks, apply it to
> HEAD:file, and put that in the index.
Actually, the largest problem I see in the above approach is _when_ you
perform the first two steps, not the details of the later steps.
The user has complete freedom outside our control between the time the
mergy operation leaves conflicts in the index and the time "add -p" is
invoked (except obviously the user cannot do any "add" to register the
resolution for the whole file to the index, as there won't be any per-hunk
resolution recording left to do by "add -p").
And as I repeatedly said, grabbing "--cc file" must be done before the
user starts mucking with the file in the working tree for the approach to
be any useful.
>> [discussion of how this could be done right]
>
> That description makes sense to me, but is way overkill for my workflow.
Obviously I don't think so. I'd rather consider what it does an absolute
minimum (not the part that it stores in an index extension, but the part
that the reference it uses to compare against the working tree file is a
original, mechanical merge result). By the way, I just relized that an
index extension similar to PRSF might be a way to properly give index log
similar to the reflog you wanted to have the other day---every time you
"git add", you record the previous state of the index entry there, so that
it can be replayed in reverse.
> Which really I could do with:
>
> for i in `git diff-files --name-only --diff-filter=U`; do
> git diff --cc $i
> echo 'OK?'
As you already know, I disagree the usefulness of this approach (see the
"Are you sure about that?" discussion), hence I doubt the usefulness of
"have it integrated into the "git add -p" loop".
^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: Summer of Code project ideas due this Friday
2011-03-10 22:58 ` Junio C Hamano
@ 2011-03-10 23:09 ` Jeff King
0 siblings, 0 replies; 63+ messages in thread
From: Jeff King @ 2011-03-10 23:09 UTC (permalink / raw)
To: Junio C Hamano
Cc: Thomas Rast, Jonathan Nieder, Shawn Pearce, Ramkumar Ramachandra,
Jens Lehmann, Christian Couder, git
On Thu, Mar 10, 2011 at 02:58:07PM -0800, Junio C Hamano wrote:
> Jeff King <peff@peff.net> writes:
>
> > Yes, I don't want to see cleanly merged parts. And "--cc" already does
> > what I want by not showing them.
>
> Are you sure about that?
Well, no. But my experience has been that its output is useful.
> What "--cc" would show largely depends on what you have in your working
> tree. If your two branches fixed the same bug with very different
> approaches, you may resolve the conflict favouring what one side did while
> discarding everything the other side did. The file in your work tree might
> be the same as "git checkout --ours $that_path" after a mergy operation.
> "diff --cc" won't show anything to you in such a case.
Actually, I think diff --cc is doing what I want there. If I take one
side's content completely, then it is not interesting to me anymore.
What I am really concerned about is doing a tricky content-level merge
in my editor and then screwing up the result. Or _trying_ to take one
side of the merge, and then screwing it up. So the "diff --cc" output
after I have mucked is useful for both those cases.
> And as I repeatedly said, grabbing "--cc file" must be done before the
> user starts mucking with the file in the working tree for the approach to
> be any useful.
I'm not sure I agree. The output after I have mucked is useful to me,
and that is what this use-case is based on. So I think we are talking
about related but slightly different use cases.
> > Which really I could do with:
> >
> > for i in `git diff-files --name-only --diff-filter=U`; do
> > git diff --cc $i
> > echo 'OK?'
>
> As you already know, I disagree the usefulness of this approach (see the
> "Are you sure about that?" discussion), hence I doubt the usefulness of
> "have it integrated into the "git add -p" loop".
OK, let me put it this way. I am not volunteering to work on the
approach you outlined. If you are, great. But if not, then what should
be done? The current behavior to show the diff and then exit is quite
confusing. At the very least, we should say something to the user about
what happened (or even suppress the diff and just say "these paths are
unmerged, and we can't handle them in add -p").
-Peff
^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: Summer of Code project ideas due this Friday
2011-03-10 18:46 ` Jeff King
2011-03-10 19:21 ` Junio C Hamano
@ 2011-03-11 13:31 ` Thomas Rast
1 sibling, 0 replies; 63+ messages in thread
From: Thomas Rast @ 2011-03-11 13:31 UTC (permalink / raw)
To: Jeff King
Cc: Jonathan Nieder, Shawn Pearce, Ramkumar Ramachandra, Jens Lehmann,
Christian Couder, git
Jeff King wrote:
> On Thu, Mar 10, 2011 at 06:15:23PM +0100, Thomas Rast wrote:
>
> > * Clean up add -p
>
> One more wishlist item for this. I use "add -p" for almost all of my
> adds these days, because I like the final review check. So after a
> conflicted merge, I find myself doing "add -p" to stage my resolution.
> The current behavior is that it shows the --cc diff and exits. It would
> be cool to handle staging the resolution, which would involve converting
> the combined diff into something that can be applied.
Good idea, thanks.
I put this (with your idea) and the word-merge thing on the wiki.
I'll let Jonathan decide what to do about the fast-import stream
filtering.
--
Thomas Rast
trast@{inf,student}.ethz.ch
^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: Summer of Code project ideas due this Friday
2011-03-10 0:10 ` Jonathan Nieder
2011-03-10 16:30 ` Jeff King
2011-03-10 17:15 ` Thomas Rast
@ 2011-03-10 17:39 ` Jakub Narebski
2011-03-11 13:28 ` Thomas Rast
2011-03-12 0:20 ` History surgery with fast-import (Re: Summer of Code project ideas due this Friday) Jonathan Nieder
2011-03-13 17:08 ` Summer of Code project ideas due this Friday Ramkumar Ramachandra
3 siblings, 2 replies; 63+ messages in thread
From: Jakub Narebski @ 2011-03-10 17:39 UTC (permalink / raw)
To: Jonathan Nieder
Cc: Jeff King, Shawn Pearce, Ramkumar Ramachandra, Jens Lehmann,
Christian Couder, Thomas Rast, git
Jonathan Nieder <jrnieder@gmail.com> writes:
> 4. filter-branch killer: using fast-import's new features to implement
> common filter-branch operations (--subdirectory-filter,
> --prune-empty, obliterating certain files) faster.
How it would be different from existing reposurgeon tool by ESR
(cross-VC thanks to using fast-import format), or git_fast_filter by
Elijah Newren (more of a library than a ready tool)?
--
Jakub Narebski
Poland
ShadeHawk on #git
^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: Summer of Code project ideas due this Friday
2011-03-10 17:39 ` Jakub Narebski
@ 2011-03-11 13:28 ` Thomas Rast
2011-03-12 0:20 ` History surgery with fast-import (Re: Summer of Code project ideas due this Friday) Jonathan Nieder
1 sibling, 0 replies; 63+ messages in thread
From: Thomas Rast @ 2011-03-11 13:28 UTC (permalink / raw)
To: Jakub Narebski
Cc: Jonathan Nieder, Jeff King, Shawn Pearce, Ramkumar Ramachandra,
Jens Lehmann, Christian Couder, git
Jakub Narebski wrote:
> Jonathan Nieder <jrnieder@gmail.com> writes:
>
> > 4. filter-branch killer: using fast-import's new features to implement
> > common filter-branch operations (--subdirectory-filter,
> > --prune-empty, obliterating certain files) faster.
>
> How it would be different from existing reposurgeon tool by ESR
> (cross-VC thanks to using fast-import format), or git_fast_filter by
> Elijah Newren (more of a library than a ready tool)?
Git integration? I haven't looked at these tools for long enough to
say how much they fit the ticket and how fast they are, so for me it's
hard to say how much work it would be.
--
Thomas Rast
trast@{inf,student}.ethz.ch
^ permalink raw reply [flat|nested] 63+ messages in thread* History surgery with fast-import (Re: Summer of Code project ideas due this Friday)
2011-03-10 17:39 ` Jakub Narebski
2011-03-11 13:28 ` Thomas Rast
@ 2011-03-12 0:20 ` Jonathan Nieder
1 sibling, 0 replies; 63+ messages in thread
From: Jonathan Nieder @ 2011-03-12 0:20 UTC (permalink / raw)
To: Jakub Narebski
Cc: Jeff King, Shawn Pearce, Ramkumar Ramachandra, Jens Lehmann,
Christian Couder, Thomas Rast, git, David Barr, Elijah Newren
Jakub Narebski wrote:
> Jonathan Nieder <jrnieder@gmail.com> writes:
>> 4. filter-branch killer: using fast-import's new features to implement
>> common filter-branch operations (--subdirectory-filter,
>> --prune-empty, obliterating certain files) faster.
>
> How it would be different from existing reposurgeon tool by ESR
> (cross-VC thanks to using fast-import format), or git_fast_filter by
> Elijah Newren (more of a library than a ready tool)?
Good question.
. reposurgeon loads the whole history into a convenient format,
manipulates it, then dumps it as a separate step. So it is even more
flexible than filter-branch, but with a large history and especially
on a small machine it can be slower.
. git_fast_filter is what Thomas mentioned --- a filter to go between
fast-export and fast-import.
Both involve unpacking all trees. Which is not a huge expense, mind
you, but there is room to go faster.
What I was suggesting is instead a tool that relies on fast-import to
do the heavy lifting. So, for example, when you ask to delete
path/to/remove.txt from all commits, it would write:
commit refs/heads/master
mark :172
author A U Thor <author@example.com> ...
committer ...
data ...
from ...
merge ...
M 040000 [old tree name for that commit] ""
D path/to/remove.txt
commit refs/heads/mater
mark :173
...
The idea is inspired by
http://thread.gmane.org/gmane.comp.version-control.git/158375
I am interested in it because I imagine something like this could be
useful for splitting out branches from a naive import of an svn repo
(and similar surgeries).
^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: Summer of Code project ideas due this Friday
2011-03-10 0:10 ` Jonathan Nieder
` (2 preceding siblings ...)
2011-03-10 17:39 ` Jakub Narebski
@ 2011-03-13 17:08 ` Ramkumar Ramachandra
3 siblings, 0 replies; 63+ messages in thread
From: Ramkumar Ramachandra @ 2011-03-13 17:08 UTC (permalink / raw)
To: Jonathan Nieder
Cc: Jeff King, Shawn Pearce, Jens Lehmann, Christian Couder,
Thomas Rast, git
Hi Jonathan,
Jonathan Nieder writes:
> 3. Remote helpers: bidi git remote-svn (or one-way remote-hg, or
> remote-cvs, or ...).
I wrote a small introduction and created an entry for git-remote-svn
in the wiki [1]. Two minor concerns:
1. The student will have to work very closely with us, and pick up a
lot of scattered WIP patches. I hope he doesn't get confused
between what's merged and what's in-progress.
2. The project isn't as de-coupled as we'd ideally like. The student
has to learn about the new fast-import features, and the
information contained in an SVN replay dumpstream before diving in.
Other than these, I think most of the pending work is in the mapper.
> 4. filter-branch killer: using fast-import's new features to implement
> common filter-branch operations (--subdirectory-filter,
> --prune-empty, obliterating certain files) faster.
This is an interesting project that I'd also thought about: it might
get tangled up along with the Sequencer project though. Should we put
up this project on the wiki nevertheless?
p.s- Sorry about the late reply; I wasn't in station.
[1] SoC2011Ideas#Welcome and SoC2011Ideas#Remote_helper_for_Subversion
-- Ram
^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: Summer of Code project ideas due this Friday
2011-03-09 21:58 ` Summer of Code project ideas due this Friday Jeff King
2011-03-10 0:10 ` Jonathan Nieder
@ 2011-03-10 0:19 ` Nguyen Thai Ngoc Duy
2011-03-10 16:31 ` Jeff King
2011-03-10 21:40 ` Alexander Miseler
` (3 subsequent siblings)
5 siblings, 1 reply; 63+ messages in thread
From: Nguyen Thai Ngoc Duy @ 2011-03-10 0:19 UTC (permalink / raw)
To: Jeff King
Cc: Shawn Pearce, Ramkumar Ramachandra, Jonathan Nieder, Jens Lehmann,
Christian Couder, Thomas Rast, git
On Thu, Mar 10, 2011 at 4:58 AM, Jeff King <peff@peff.net> wrote:
> They will be looking at our ideas page at:
>
> https://git.wiki.kernel.org/index.php/SoC2011Ideas
>
> If you have any ideas, please add them to the page!
Jeff, how is the "support multiple transport (torrent, bundle, http..)
cloning" going? The one that you were working on from the "initial
clone from bundle" thread? Is it ok to make it a SoC project
(presumably with at least bundle clone support)?
--
Duy
^ permalink raw reply [flat|nested] 63+ messages in thread* Re: Summer of Code project ideas due this Friday
2011-03-10 0:19 ` Nguyen Thai Ngoc Duy
@ 2011-03-10 16:31 ` Jeff King
0 siblings, 0 replies; 63+ messages in thread
From: Jeff King @ 2011-03-10 16:31 UTC (permalink / raw)
To: Nguyen Thai Ngoc Duy
Cc: Shawn Pearce, Ramkumar Ramachandra, Jonathan Nieder, Jens Lehmann,
Christian Couder, Thomas Rast, git
On Thu, Mar 10, 2011 at 07:19:36AM +0700, Nguyen Thai Ngoc Duy wrote:
> On Thu, Mar 10, 2011 at 4:58 AM, Jeff King <peff@peff.net> wrote:
> > They will be looking at our ideas page at:
> >
> > https://git.wiki.kernel.org/index.php/SoC2011Ideas
> >
> > If you have any ideas, please add them to the page!
>
> Jeff, how is the "support multiple transport (torrent, bundle, http..)
> cloning" going? The one that you were working on from the "initial
> clone from bundle" thread? Is it ok to make it a SoC project
> (presumably with at least bundle clone support)?
I pushed it back in favor of other topics, but I will get to it
eventually. If you want to make a SoC proposal, go for it. Though I
think it might not be big enough by itself, and might need to be folded
into a larger project.
-Peff
^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: Summer of Code project ideas due this Friday
2011-03-09 21:58 ` Summer of Code project ideas due this Friday Jeff King
2011-03-10 0:10 ` Jonathan Nieder
2011-03-10 0:19 ` Nguyen Thai Ngoc Duy
@ 2011-03-10 21:40 ` Alexander Miseler
2011-03-10 22:18 ` Jeff King
2011-03-11 12:18 ` Alexander Miseler
` (2 subsequent siblings)
5 siblings, 1 reply; 63+ messages in thread
From: Alexander Miseler @ 2011-03-10 21:40 UTC (permalink / raw)
To: Jeff King
Cc: Shawn Pearce, Ramkumar Ramachandra, Jonathan Nieder, Jens Lehmann,
Christian Couder, Thomas Rast, git
Comments on the "Better big-file support" section:
"While git can handle arbitrary-sized binary content [...]"
This is very much not true. Git tries at many places to load the complete file into memory and usually fails with "out of memory" if it can't. With the 32bit msysGit client this places the upper file size limit, from purely empirical observation, at 600-700 MByte. When a file is to large git fails late, adding and committing works (as long as there are no filters or other complications), but you can forget about pushing, rebasing or otherwise manipulating that commit. Even worse yet, commits consisting of smaller files but with a combined size over the limit will also cause out-of-memories.
Thus a main focus should be the memory problem, e.g. by using stream-like file handling everywhere, since not working at all is orders of magnitude worse than working slowly :)
"In some cases, this may be as simple as having a "large file" codepath that avoids pulling whole files into memory (e.g., during "git add")."
Ironically git add is one of the few things that work with large files, as mentioned above. Presumably the stream-oriented zlib enforced/encouraged a steam-like handling here :)
Slow as hell though and of course it is usually not sensible to compress a 1.5 GByte file.
I'm very willing to work on this topic. Though I'm not a student and as a git code newbie I also don't have the skills for mentoring yet.
^ permalink raw reply [flat|nested] 63+ messages in thread* Re: Summer of Code project ideas due this Friday
2011-03-10 21:40 ` Alexander Miseler
@ 2011-03-10 22:18 ` Jeff King
2011-03-11 14:17 ` Alexander Miseler
0 siblings, 1 reply; 63+ messages in thread
From: Jeff King @ 2011-03-10 22:18 UTC (permalink / raw)
To: Alexander Miseler
Cc: Shawn Pearce, Ramkumar Ramachandra, Jonathan Nieder, Jens Lehmann,
Christian Couder, Thomas Rast, git
On Thu, Mar 10, 2011 at 10:40:01PM +0100, Alexander Miseler wrote:
> "While git can handle arbitrary-sized binary content [...]"
>
> This is very much not true. Git tries at many places to load the
> complete file into memory and usually fails with "out of memory" if it
> can't. With the 32bit msysGit client this places the upper file size
> limit, from purely empirical observation, at 600-700 MByte.
I think we are picking nits here. What I meant was two things:
1. The fundamental design of git does not prevent storing
arbitrary-sized binary data.
2. How big a piece of data the current implementation can handle
sanely is up to how much hardware you throw at it. On my 64-bit
machine with 8G of RAM running Linux, I can easily work with 2
gigabyte files. Some operations are slow, of course, but it works.
I'm willing to accept that 32-bit msysgit has more trouble with
a case like that.
But I think we are probably in agreement with what needs to be done to
make things better. Specifically, I am thinking of:
1. Streaming blobs wherever possible (e.g., add, filters, textconv).
2. Converting the diff code not to have in-memory files is probably
going to be quite difficult. But most of these files don't have
interesting diffs _anyway_. They're usually binary, and we don't
generate binary diffs by default. So what we need to focus on is
avoiding loading them when we can. Things like:
a. Using caching textconv, and when we do run the textconv
filter, streaming the blob to the filter.
b. Avoid loading the whole file to check whether it is binary. We
can already avoid this by marking it binary with
gitattributes, but there is no reason git can't just load the
first 4K or so to check for binary-ness, and get this
optimization automatically. We can also consider caching
binary-ness for large files so we don't have to look at them
at all after the first time.
c. Handle rename detection better. It may be a matter of saying
"this file is too big for detection". But we may also be able
to stream it through the spantree-hashing, and then possibly
also cache the resultant data.
3. The above deal with memory problems. There is also a storage
problem. If I have a 100G repo, right now I use at least 100G in
the .git directory and 100G in the working tree. That's a problem
for repos of that size. If I have a storage server on the LAN and
want to accept the latency hit, it would be nice to keep the
commits local and the giant blobs on the server. Especially coupled
with the optimizations in (2), we can possibly avoid even having to
touch those blobs at all in many cases, so the latency wouldn't be
a big deal.
4. A related storage problem is that we put big files in packs, and
the I/O on rewriting the pack becomes a problem. They would do
better loose or in their own packs.
And I'm sure there are more variations on those things. Part of the
project would be identifying the problem areas.
> Even worse yet, commits consisting of smaller files but with a
> combined size over the limit will also cause out-of-memories.
That generally should work OK. The diff and packing code tries to keep
memory usage reasonable, which generally equates to two times the
largest file. If you have a test case that shows problems, there may
very well be a bug.
> Thus a main focus should be the memory problem, e.g. by using
> stream-like file handling everywhere, since not working at all is
> orders of magnitude worse than working slowly :)
Agreed. I think they are sort of the same problem. Whether it works
slowly or not at all is simply a matter of how much memory you have. ;)
> Ironically git add is one of the few things that work with large
> files, as mentioned above. Presumably the stream-oriented zlib
> enforced/encouraged a steam-like handling here :) Slow as hell though
> and of course it is usually not sensible to compress a 1.5 GByte file.
I just tried "git add" on a 2G file of random bytes. It took about a
minute or so to calculate the sha1 and compress it, but the memory usage
did jump to 2G. So we could obviously do better on the memory, and there
is almost certainly no point in zlib compressing something that big. In
my case, it was obviously just random junk. But most files of that size
are already going to have some kind of lossy compression, so we are just
wasting CPU. You can always set core.compression, but I really just want
it off for certain files.
> I'm very willing to work on this topic. Though I'm not a student and
> as a git code newbie I also don't have the skills for mentoring yet.
It's on my agenda, too. We'll see if a student steps up for the GSoC
project. But don't let that stop you if you want to take a look at it;
I'm sure there is plenty of work to go around. :)
-Peff
^ permalink raw reply [flat|nested] 63+ messages in thread* Re: Summer of Code project ideas due this Friday
2011-03-10 22:18 ` Jeff King
@ 2011-03-11 14:17 ` Alexander Miseler
2011-03-12 19:47 ` Alexander Miseler
0 siblings, 1 reply; 63+ messages in thread
From: Alexander Miseler @ 2011-03-11 14:17 UTC (permalink / raw)
To: Jeff King
Cc: Shawn Pearce, Ramkumar Ramachandra, Jonathan Nieder, Jens Lehmann,
Christian Couder, Thomas Rast, git
On 10.03.2011 23:18, Jeff King wrote:
> 1. The fundamental design of git does not prevent storing
> arbitrary-sized binary data.
agreed
> But I think we are probably in agreement with what needs to be done to
> make things better. Specifically, I am thinking of:
agreed
>> Even worse yet, commits consisting of smaller files but with a
>> combined size over the limit will also cause out-of-memories.
>
> That generally should work OK. The diff and packing code tries to keep
> memory usage reasonable, which generally equates to two times the
> largest file. If you have a test case that shows problems, there may
> very well be a bug.
I will look into it.
> But don't let that stop you if you want to take a look at it;
> I'm sure there is plenty of work to go around. :)
Certainly ^_^
^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: Summer of Code project ideas due this Friday
2011-03-11 14:17 ` Alexander Miseler
@ 2011-03-12 19:47 ` Alexander Miseler
0 siblings, 0 replies; 63+ messages in thread
From: Alexander Miseler @ 2011-03-12 19:47 UTC (permalink / raw)
To: Jeff King
Cc: Shawn Pearce, Ramkumar Ramachandra, Jonathan Nieder, Jens Lehmann,
Christian Couder, Thomas Rast, git
>>> Even worse yet, commits consisting of smaller files but with a
>>> combined size over the limit will also cause out-of-memories.
>>
>> That generally should work OK. The diff and packing code tries to keep
>> memory usage reasonable, which generally equates to two times the
>> largest file. If you have a test case that shows problems, there may
>> very well be a bug.
>
> I will look into it.
I failed to create an artificial test case for this (thus probably misunderstanding the exact circumstances leading to this). I will investigate it when I get a real life test case again.
^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: Summer of Code project ideas due this Friday
2011-03-09 21:58 ` Summer of Code project ideas due this Friday Jeff King
` (2 preceding siblings ...)
2011-03-10 21:40 ` Alexander Miseler
@ 2011-03-11 12:18 ` Alexander Miseler
2011-03-11 12:52 ` Ilari Liusvaara
2011-03-11 12:43 ` Ævar Arnfjörð Bjarmason
2011-03-17 23:40 ` Summer of Code project ideas Jakub Narebski
5 siblings, 1 reply; 63+ messages in thread
From: Alexander Miseler @ 2011-03-11 12:18 UTC (permalink / raw)
To: Jeff King
Cc: Shawn Pearce, Ramkumar Ramachandra, Jonathan Nieder, Jens Lehmann,
Christian Couder, Thomas Rast, git
On 09.03.2011 22:58, Jeff King wrote:
> If you have any ideas, please add them to the page! Whether you are
> available to mentor the project, or simply think it would make a good
> project and want to inspire others to mentor it, it's appropriate to go
> there.
Added two new ideas:
Port Git to Android
Resumable clone
I'm new to the Git code and therefore don't necessarily have an idea what I'm talking about. Please feel free to change,
correct, mutilate and massacre this ideas. This are just two things that I would like to see, but don't want to work on
myself since I have BIGGER fishes to fry ^_^
^ permalink raw reply [flat|nested] 63+ messages in thread* Re: Summer of Code project ideas due this Friday
2011-03-11 12:18 ` Alexander Miseler
@ 2011-03-11 12:52 ` Ilari Liusvaara
2011-03-11 13:48 ` Nguyen Thai Ngoc Duy
0 siblings, 1 reply; 63+ messages in thread
From: Ilari Liusvaara @ 2011-03-11 12:52 UTC (permalink / raw)
To: Alexander Miseler
Cc: Jeff King, Shawn Pearce, Ramkumar Ramachandra, Jonathan Nieder,
Jens Lehmann, Christian Couder, Thomas Rast, git
On Fri, Mar 11, 2011 at 01:18:45PM +0100, Alexander Miseler wrote:
> On 09.03.2011 22:58, Jeff King wrote:
> >If you have any ideas, please add them to the page! Whether you are
> >available to mentor the project, or simply think it would make a good
> >project and want to inspire others to mentor it, it's appropriate to go
> >there.
>
> Resumable clone
This is very very hard. Not so much to implement but to design the way it
can be done without assuming things (like object sort orders) that aren't
stable.
-Ilari
^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: Summer of Code project ideas due this Friday
2011-03-11 12:52 ` Ilari Liusvaara
@ 2011-03-11 13:48 ` Nguyen Thai Ngoc Duy
2011-03-11 14:10 ` Alexander Miseler
0 siblings, 1 reply; 63+ messages in thread
From: Nguyen Thai Ngoc Duy @ 2011-03-11 13:48 UTC (permalink / raw)
To: Ilari Liusvaara
Cc: Alexander Miseler, Jeff King, Shawn Pearce, Ramkumar Ramachandra,
Jonathan Nieder, Jens Lehmann, Christian Couder, Thomas Rast, git,
Pranav Ravichandran
On Fri, Mar 11, 2011 at 7:52 PM, Ilari Liusvaara
<ilari.liusvaara@elisanet.fi> wrote:
> On Fri, Mar 11, 2011 at 01:18:45PM +0100, Alexander Miseler wrote:
>> On 09.03.2011 22:58, Jeff King wrote:
>> >If you have any ideas, please add them to the page! Whether you are
>> >available to mentor the project, or simply think it would make a good
>> >project and want to inspire others to mentor it, it's appropriate to go
>> >there.
>>
>
>> Resumable clone
>
> This is very very hard. Not so much to implement but to design the way it
> can be done without assuming things (like object sort orders) that aren't
> stable.
Yes it's hard. I have some experimental thing that nearly works [1],
although whether it is an acceptable approach is to be seen. If
anyone's interested, I'll post it some time
A simpler way to restartable clone is to facilitate bundles (Nicolas'
idea). Some glue is needed to teach git-fetch/git-daemon to use the
bundles, and git-push to automatically create bundles periodically (or
a new command that can be run from cron). I think this way fit in GSoC
scope better.
[1] The idea of my work above was mentioned elsewhere, history is cut
down by path. Each file/dir's history a very long chain of deltas. We
can stream deltas (in parallel if needed) over the wire, resuming
where the chain stops last time.
There are many problems. One is that a deep chain can make git run out
of stack, so chains have to be broken down before storing (not done).
Another one is that not many deltas can be reused, so it will consume
more power than normal clone. But once you clone this way, the cloned
repo have lots of delta suitable for another clone (but probably not
for anything else).
--
Duy
^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: Summer of Code project ideas due this Friday
2011-03-11 13:48 ` Nguyen Thai Ngoc Duy
@ 2011-03-11 14:10 ` Alexander Miseler
2011-03-11 14:27 ` Nguyen Thai Ngoc Duy
0 siblings, 1 reply; 63+ messages in thread
From: Alexander Miseler @ 2011-03-11 14:10 UTC (permalink / raw)
To: Nguyen Thai Ngoc Duy
Cc: Ilari Liusvaara, Jeff King, Shawn Pearce, Ramkumar Ramachandra,
Jonathan Nieder, Jens Lehmann, Christian Couder, Thomas Rast, git,
Pranav Ravichandran
On 11.03.2011 14:48, Nguyen Thai Ngoc Duy wrote:
> On Fri, Mar 11, 2011 at 7:52 PM, Ilari Liusvaara
> <ilari.liusvaara@elisanet.fi> wrote:
>> On Fri, Mar 11, 2011 at 01:18:45PM +0100, Alexander Miseler wrote:
>>> Resumable clone
>>
>> This is very very hard. Not so much to implement but to design the way it
>> can be done without assuming things (like object sort orders) that aren't
>> stable.
>
> Yes it's hard. I have some experimental thing that nearly works [1],
> although whether it is an acceptable approach is to be seen. If
> anyone's interested, I'll post it some time
>
> A simpler way to restartable clone is to facilitate bundles (Nicolas'
> idea). Some glue is needed to teach git-fetch/git-daemon to use the
> bundles, and git-push to automatically create bundles periodically (or
> a new command that can be run from cron). I think this way fit in GSoC
> scope better.
>
> [1] The idea of my work above was mentioned elsewhere, history is cut
> down by path. Each file/dir's history a very long chain of deltas. We
> can stream deltas (in parallel if needed) over the wire, resuming
> where the chain stops last time.
>
> There are many problems. One is that a deep chain can make git run out
> of stack, so chains have to be broken down before storing (not done).
> Another one is that not many deltas can be reused, so it will consume
> more power than normal clone. But once you clone this way, the cloned
> repo have lots of delta suitable for another clone (but probably not
> for anything else).
This may all be aiming to short. IMHO the best solution would be some generic way for the client to specify exactly what
it wants to get and to get just that. This would lay the groundwork for:
- lazy clones
- sparse clones
- resumable cloning
- resumable fetching
and probably quite a few other nifty tricks.
I guess that would be far beyond the scope of a SoC project though.
^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: Summer of Code project ideas due this Friday
2011-03-11 14:10 ` Alexander Miseler
@ 2011-03-11 14:27 ` Nguyen Thai Ngoc Duy
2011-03-11 22:42 ` Sam Vilain
2011-03-12 21:41 ` Alexander Miseler
0 siblings, 2 replies; 63+ messages in thread
From: Nguyen Thai Ngoc Duy @ 2011-03-11 14:27 UTC (permalink / raw)
To: Alexander Miseler
Cc: Ilari Liusvaara, Jeff King, Shawn Pearce, Ramkumar Ramachandra,
Jonathan Nieder, Jens Lehmann, Christian Couder, Thomas Rast, git,
Pranav Ravichandran
On Fri, Mar 11, 2011 at 9:10 PM, Alexander Miseler <alexander@miseler.de> wrote:
> This may all be aiming to short. IMHO the best solution would be some
> generic way for the client to specify exactly what it wants to get and to
> get just that.
But how do you define "what it wants to get"? Pros and cons are
different and depend on that defintion.
Git's way of saying now is "I have these commits, give me these
branches (cutting at certain depth)". A near future extension would be
"I have these commits _restricted to these paths only_, give me...".
If it's broken half way, start again.
The bundle way is basically Git's way except that the remote side says
"I've got this bundle, fetch it here, then you can ask us again for
the rest in a normal way".
mirrorsync (aka gittorrent) says "I have this commit, give me objects
so that I can have this commit", which leaves the (probably) huge
initial commit out of view.
My way is "give me a chain of deltas starting from this SHA-1". A big
blob can be considered as a history of smaller pieces.
The last way of saying is just "give me this object", which would be a
big waste of bandwidth.
Your move :)
--
Duy
^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: Summer of Code project ideas due this Friday
2011-03-11 14:27 ` Nguyen Thai Ngoc Duy
@ 2011-03-11 22:42 ` Sam Vilain
2011-03-12 21:41 ` Alexander Miseler
1 sibling, 0 replies; 63+ messages in thread
From: Sam Vilain @ 2011-03-11 22:42 UTC (permalink / raw)
To: Nguyen Thai Ngoc Duy
Cc: Alexander Miseler, Ilari Liusvaara, Jeff King, Shawn Pearce,
Ramkumar Ramachandra, Jonathan Nieder, Jens Lehmann,
Christian Couder, Thomas Rast, git, Pranav Ravichandran
On 12/03/11 03:27, Nguyen Thai Ngoc Duy wrote:
> On Fri, Mar 11, 2011 at 9:10 PM, Alexander Miseler <alexander@miseler.de> wrote:
>> This may all be aiming to short. IMHO the best solution would be some
>> generic way for the client to specify exactly what it wants to get and to
>> get just that.
> But how do you define "what it wants to get"? Pros and cons are
> different and depend on that defintion.
>
> Git's way of saying now is "I have these commits, give me these
> branches (cutting at certain depth)". A near future extension would be
> "I have these commits _restricted to these paths only_, give me...".
> If it's broken half way, start again.
>
> The bundle way is basically Git's way except that the remote side says
> "I've got this bundle, fetch it here, then you can ask us again for
> the rest in a normal way".
>
> mirrorsync (aka gittorrent) says "I have this commit, give me objects
> so that I can have this commit", which leaves the (probably) huge
> initial commit out of view.
>
> My way is "give me a chain of deltas starting from this SHA-1". A big
> blob can be considered as a history of smaller pieces.
>
> The last way of saying is just "give me this object", which would be a
> big waste of bandwidth.
>
> Your move :)
Just FWIW, I recently produced some diagrams and demonstrated the
resumable clone algorithm; speed is currently very limited by the
implementation at 15 minutes to slice up git.git; but I'm playing with
libgit2 to see if I can make a more optimal version.
See http://vilain.net/comp/git/gittorrent/commit_reel.html
Sam
^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: Summer of Code project ideas due this Friday
2011-03-11 14:27 ` Nguyen Thai Ngoc Duy
2011-03-11 22:42 ` Sam Vilain
@ 2011-03-12 21:41 ` Alexander Miseler
1 sibling, 0 replies; 63+ messages in thread
From: Alexander Miseler @ 2011-03-12 21:41 UTC (permalink / raw)
To: Nguyen Thai Ngoc Duy
Cc: Ilari Liusvaara, Jeff King, Shawn Pearce, Ramkumar Ramachandra,
Jonathan Nieder, Jens Lehmann, Christian Couder, Thomas Rast, git,
Pranav Ravichandran
> Your move :)
Heh, I'm still pretty hampered by having no clue ^_^
> My way is "give me a chain of deltas starting from this SHA-1". A big
> blob can be considered as a history of smaller pieces.
I originally missed the fact that it is completely irrelevant where in the chain the reference point is and that you can go easily from any arbitrary point in the chain to any other arbitrary point in the same chain. This makes it a lot more flexible than I thought. But to support more than just resumable cloning You would still need a more powerful way to negotiate "I have this; I want that" between client and server.
My point was mostly that lazy clone, sparse clone and resumable clone are closely related and should ideally be solved in one go. You currently seem to aim at only one of them, which is a shame.
For lazy clone the client would start with all blobs referenced by HEAD (and possibly other branches) and then it would walk backwards through the delta chains if/as far/whenever it wants. Though it would also need to detect when it needs to start reading new delta chains as it walks backwards through history. Sparse clone would use the same mechanism and just restrict it to certain paths.
^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: Summer of Code project ideas due this Friday
2011-03-09 21:58 ` Summer of Code project ideas due this Friday Jeff King
` (3 preceding siblings ...)
2011-03-11 12:18 ` Alexander Miseler
@ 2011-03-11 12:43 ` Ævar Arnfjörð Bjarmason
2011-03-11 14:24 ` code.sculptor
2011-03-17 23:40 ` Summer of Code project ideas Jakub Narebski
5 siblings, 1 reply; 63+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2011-03-11 12:43 UTC (permalink / raw)
To: Jeff King
Cc: Shawn Pearce, Ramkumar Ramachandra, Jonathan Nieder, Jens Lehmann,
Christian Couder, Thomas Rast, git
On Wed, Mar 9, 2011 at 22:58, Jeff King <peff@peff.net> wrote:
> If you have any ideas, please add them to the page!
Here's a simple one:
Currently we have a global UI color setting, but pretty much only
git-status/log/etc use it.
Change git-bisect, git-am etc. to also support color output. It would
be a easy but big UI win.
^ permalink raw reply [flat|nested] 63+ messages in thread* Re: Summer of Code project ideas due this Friday
2011-03-11 12:43 ` Ævar Arnfjörð Bjarmason
@ 2011-03-11 14:24 ` code.sculptor
0 siblings, 0 replies; 63+ messages in thread
From: code.sculptor @ 2011-03-11 14:24 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason, git-owner, Jeff King
Cc: Shawn Pearce, Ramkumar Ramachandra, Jonathan Nieder, Jens Lehmann,
Christian Couder, Thomas Rast, git
Great idea!
Sent from my BlackBerry device on the Rogers Wireless Network
-----Original Message-----
From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Sender: git-owner@vger.kernel.org
Date: Fri, 11 Mar 2011 13:43:39
To: Jeff King<peff@peff.net>
Cc: Shawn Pearce<spearce@spearce.org>; Ramkumar Ramachandra<artagnon@gmail.com>; Jonathan Nieder<jrnieder@gmail.com>; Jens Lehmann<Jens.Lehmann@web.de>; Christian Couder<chriscool@tuxfamily.org>; Thomas Rast<trast@student.ethz.ch>; git<git@vger.kernel.org>
Subject: Re: Summer of Code project ideas due this Friday
On Wed, Mar 9, 2011 at 22:58, Jeff King <peff@peff.net> wrote:
> If you have any ideas, please add them to the page!
Here's a simple one:
Currently we have a global UI color setting, but pretty much only
git-status/log/etc use it.
Change git-bisect, git-am etc. to also support color output. It would
be a easy but big UI win.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: Summer of Code project ideas
2011-03-09 21:58 ` Summer of Code project ideas due this Friday Jeff King
` (4 preceding siblings ...)
2011-03-11 12:43 ` Ævar Arnfjörð Bjarmason
@ 2011-03-17 23:40 ` Jakub Narebski
2011-03-22 20:31 ` Heiko Voigt
` (2 more replies)
5 siblings, 3 replies; 63+ messages in thread
From: Jakub Narebski @ 2011-03-17 23:40 UTC (permalink / raw)
To: Jeff King
Cc: Shawn Pearce, Ramkumar Ramachandra, Jonathan Nieder, Jens Lehmann,
Christian Couder, Thomas Rast, git, J.H., Pat Thoyts,
Paul Mackerras
Jeff King <peff@peff.net> writes:
> https://git.wiki.kernel.org/index.php/SoC2011Ideas
>
> If you have any ideas, please add them to the page!
A few project ideas I am not sure if they are feasible for GSoC:
* merging-in support for caching in gitweb, and benchmarking/profiling
gitweb in high load situation
J.H., you would probably want gitweb [output] caching to be merged-in
sooner that the end of Google Summer of Code 2011, isn't it?
* embedding graphical diff and graphical merge tool in git-gui, e.g. as
"git gui diff". I think that we can use xxdiff; the license is
compatibile.
Pat and Shawn, is it something worth doing? Does it look like a good
project for GSoC2011, or is it too small of a project for this? Would
we be able to find mentor for this idea?
* splitting gitk, common library (Tcl/Tk bindings) for gitk and git-gui
Pat and Paul, do you think it is right scope, or is it too large project
to put as an GSoC idea?
--
Jakub Narebski
Poland
ShadeHawk on #git
^ permalink raw reply [flat|nested] 63+ messages in thread* Re: Re: Summer of Code project ideas
2011-03-17 23:40 ` Summer of Code project ideas Jakub Narebski
@ 2011-03-22 20:31 ` Heiko Voigt
2011-03-22 22:55 ` J.H.
2011-03-25 1:11 ` Pat Thoyts
2 siblings, 0 replies; 63+ messages in thread
From: Heiko Voigt @ 2011-03-22 20:31 UTC (permalink / raw)
To: Jakub Narebski
Cc: Jeff King, Shawn Pearce, Ramkumar Ramachandra, Jonathan Nieder,
Jens Lehmann, Christian Couder, Thomas Rast, git, J.H.,
Pat Thoyts, Paul Mackerras
Hi,
On Thu, Mar 17, 2011 at 04:40:44PM -0700, Jakub Narebski wrote:
> * embedding graphical diff and graphical merge tool in git-gui, e.g. as
> "git gui diff". I think that we can use xxdiff; the license is
> compatibile.
>
> Pat and Shawn, is it something worth doing? Does it look like a good
> project for GSoC2011, or is it too small of a project for this? Would
> we be able to find mentor for this idea?
>
> * splitting gitk, common library (Tcl/Tk bindings) for gitk and git-gui
>
> Pat and Paul, do you think it is right scope, or is it too large project
> to put as an GSoC idea?
I would be interested in mentoring those projects. I do not know whether
we can reach consensus how to split up things. The last time we
discussed a possible approach[1] to share code between gitk and git gui
we did not reach one.
Cheers Heiko
[1] http://thread.gmane.org/gmane.comp.version-control.git/166663/focus=166695
^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: Summer of Code project ideas
2011-03-17 23:40 ` Summer of Code project ideas Jakub Narebski
2011-03-22 20:31 ` Heiko Voigt
@ 2011-03-22 22:55 ` J.H.
2011-03-25 1:11 ` Pat Thoyts
2 siblings, 0 replies; 63+ messages in thread
From: J.H. @ 2011-03-22 22:55 UTC (permalink / raw)
To: Jakub Narebski
Cc: Jeff King, Shawn Pearce, Ramkumar Ramachandra, Jonathan Nieder,
Jens Lehmann, Christian Couder, Thomas Rast, git, Pat Thoyts,
Paul Mackerras
On 03/17/2011 04:40 PM, Jakub Narebski wrote:
> Jeff King <peff@peff.net> writes:
>
>> https://git.wiki.kernel.org/index.php/SoC2011Ideas
>>
>> If you have any ideas, please add them to the page!
>
> A few project ideas I am not sure if they are feasible for GSoC:
>
> * merging-in support for caching in gitweb, and benchmarking/profiling
> gitweb in high load situation
>
> J.H., you would probably want gitweb [output] caching to be merged-in
> sooner that the end of Google Summer of Code 2011, isn't it?
I think the work needed to get gitweb-caching actually merged upstream
is going to be a slight bit messier than I'm willing to put a student
through, and couple that with the fact I'm going to be herding 2 or 3
students for kernel.org as already, I won't have enough time to try and
mentor a student through that quagmire.
Should a student be interested in the graphing stuff for gitweb, I'm
happy to make commentary and review things as they come, but that's at
best a co-mentor from me.
^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: Summer of Code project ideas
2011-03-17 23:40 ` Summer of Code project ideas Jakub Narebski
2011-03-22 20:31 ` Heiko Voigt
2011-03-22 22:55 ` J.H.
@ 2011-03-25 1:11 ` Pat Thoyts
2011-03-25 13:02 ` Jakub Narebski
2 siblings, 1 reply; 63+ messages in thread
From: Pat Thoyts @ 2011-03-25 1:11 UTC (permalink / raw)
To: Jakub Narebski
Cc: Jeff King, Shawn Pearce, Ramkumar Ramachandra, Jonathan Nieder,
Jens Lehmann, Christian Couder, Thomas Rast, git, J.H.,
Paul Mackerras
Jakub Narebski <jnareb@gmail.com> writes:
>Jeff King <peff@peff.net> writes:
>
>> https://git.wiki.kernel.org/index.php/SoC2011Ideas
>>
>> If you have any ideas, please add them to the page!
>
>A few project ideas I am not sure if they are feasible for GSoC:
>
>* merging-in support for caching in gitweb, and benchmarking/profiling
> gitweb in high load situation
>
> J.H., you would probably want gitweb [output] caching to be merged-in
> sooner that the end of Google Summer of Code 2011, isn't it?
>
>* embedding graphical diff and graphical merge tool in git-gui, e.g. as
> "git gui diff". I think that we can use xxdiff; the license is
> compatibile.
>
> Pat and Shawn, is it something worth doing? Does it look like a good
> project for GSoC2011, or is it too small of a project for this? Would
> we be able to find mentor for this idea?
There is also tkdiff for stealing from. I'm not sure about the worth -
there are lots of free merge tools around. But if someone wants to do
that then fine.
>* splitting gitk, common library (Tcl/Tk bindings) for gitk and git-gui
>
> Pat and Paul, do you think it is right scope, or is it too large project
> to put as an GSoC idea?
It shouldn't be too large. You are likely looking at doing Git.pm as a
Tcl package (without looking in much detail). Testing that gitk and
git-gui didn't get broken will probably be tedious. So adding some test
suite to each could help. The Tcl test package is quite capable of being
used to test Tk apps provided some tests get written.
--
Pat Thoyts http://www.patthoyts.tk/
PGP fingerprint 2C 6E 98 07 2C 59 C8 97 10 CE 11 E6 04 E0 B9 DD
^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: Summer of Code project ideas
2011-03-25 1:11 ` Pat Thoyts
@ 2011-03-25 13:02 ` Jakub Narebski
0 siblings, 0 replies; 63+ messages in thread
From: Jakub Narebski @ 2011-03-25 13:02 UTC (permalink / raw)
To: Pat Thoyts
Cc: Jeff King, Shawn Pearce, Ramkumar Ramachandra, Jonathan Nieder,
Jens Lehmann, Christian Couder, Thomas Rast, git, J.H.,
Paul Mackerras
On Fri, 25 Mar 2011, Pat Thoyts wrote:
> Jakub Narebski <jnareb@gmail.com> writes:
> > A few project ideas I am not sure if they are feasible for GSoC:
> > * embedding graphical diff and graphical merge tool in git-gui, e.g. as
> > "git gui diff". I think that we can use xxdiff; the license is
> > compatibile.
> >
> > Pat and Shawn, is it something worth doing? Does it look like a good
> > project for GSoC2011, or is it too small of a project for this? Would
> > we be able to find mentor for this idea?
>
> There is also tkdiff for stealing from. I'm not sure about the worth -
> there are lots of free merge tools around. But if someone wants to do
> that then fine.
I meant TkDiff, not xxdiff here; I'm sorry for the mistake.
TkDiff has the advantage that is already written in Tcl/Tk, so we can
borrow code (like e.g. "refining" diff), not only algorithms.
> > * splitting gitk, common library (Tcl/Tk bindings) for gitk and git-gui
> >
> > Pat and Paul, do you think it is right scope, or is it too large project
> > to put as an GSoC idea?
>
> It shouldn't be too large. You are likely looking at doing Git.pm as a
> Tcl package (without looking in much detail). Testing that gitk and
> git-gui didn't get broken will probably be tedious. So adding some test
> suite to each could help. The Tcl test package is quite capable of being
> used to test Tk apps provided some tests get written.
I thought that would be mainly re-using library part of git-gui in gitk;
tests are decidely a good idea, though I don't know if there are ready
tools to do tests for programs written in Tcl/Tk , and for testing UI.
--
Jakub Narebski
Poland
^ permalink raw reply [flat|nested] 63+ messages in thread