GSOC idea: build in scripts and cleanups

GSOC idea: build in scripts and cleanups

[Robert David]

[-- Attachment #1: Type: text/plain, Size: 922 bytes --]

Hi, 

I'm IT student from Czech Republic.
I'm long time user of git and very interested in development participation. 
Having passing your GSOC ideas, keep my eyes on scripts building in and also 
git add -p cleanups. Because I like this kind of work. 

Because there are more thinks to do in the cleanup and rewrite task, I would 
like to ask if there is some priority in these tasks? To let me focus on the 
most important parts. 

I'm experienced in C and shell ( i did a lot of bug fixing in company I worked 
part-time ) and beginner in PERL ( I wrote some scripts but nothing big ). I 
like PERL much, so it is not a problem for me to learn that better.

I was also thinking if there isn't PERL a better choice in rewriting shell 
scripts, due to planed porting (android, etc). Better than C. But I don't know 
android and other platform so much, so thats why I'm asking.

Greets,
Robert David.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: GSOC idea: build in scripts and cleanups

[Jonathan Nieder]

(+cc: various relevant people for ideas)
Hi,

Robert David wrote:

> I'm long time user of git and very interested in development participation. 

Welcome!

> Having passing your GSOC ideas, keep my eyes on scripts building in and also 
> git add -p cleanups. Because I like this kind of work. 
>
> Because there are more thinks to do in the cleanup and rewrite task, I would 
> like to ask if there is some priority in these tasks? To let me focus on the 
> most important parts. 

Looking at the git-add--interactive.perl source, I see:

	process_args();
	refresh();
	if ($patch_mode) {
		patch_update_cmd();
	}
[...]

and patch_update_cmd looks like

	my @all_mods = list_modified($patch_mode_flavour{FILTER});
	my @mods = grep { !($_->{BINARY}) } @all_mods;
[...]
	if ($patch_mode) {
		@them = @mods;
	}
[...]
	for (@them) {
		return 0 if patch_update_file($_->{VALUE});
	}

patch_update_file loops over hunks in the diff for that file and acts
on them one at a time.  So a natural step might be to eliminate this
outer "for (@them)" loop, so the person at the keyboard could decide
to go back to a previous file or list all files and choose one.

Others may have more to say.  If there are things you wished "git
checkout --patch" was able to do, that can also help.

> I was also thinking if there isn't PERL a better choice in rewriting shell 
> scripts, due to planed porting (android, etc). Better than C. But I don't know 
> android and other platform so much, so thats why I'm asking.

So far (on Windows and various Unixen) it seems that C is much easier
to work with as far as porting goes.[2]

Jonathan

[1] Android is an odd example because the platform uses Java heavily
(so JGit might be a better fit for it).  Perhaps the wish for android
support should have been put on the Eclipse ideas page[2] and a link
added to git's; I dunno.
[2] http://wiki.eclipse.org/Google_Summer_of_Code_2011_Ideas#Ideas_submission

Re: GSOC idea: build in scripts and cleanups

[Jeff King]

On Fri, Mar 25, 2011 at 09:14:35PM -0500, Jonathan Nieder wrote:

> > I was also thinking if there isn't PERL a better choice in rewriting shell 
> > scripts, due to planed porting (android, etc). Better than C. But I don't know 
> > android and other platform so much, so thats why I'm asking.
> 
> So far (on Windows and various Unixen) it seems that C is much easier
> to work with as far as porting goes.[2]

If I were considering cleaning up and porting add--interactive to C, I
think I would probably start with just porting the "-p" patch loop
first. I think it's the part that most people use, and most callers
don't support a generic "-i" but just the "-p" interface (e.g., you can
do "git add -i" or "git add -p", but only "git checkout -p"). And that
cuts down the size of the task somewhat.

As far as cleanup versus features, I think Thomas would have to comment
on that. He is the one who did the most work on patch-mode, and
therefore the one who most thinks it needs cleaned up. :)

> [1] Android is an odd example because the platform uses Java heavily
> (so JGit might be a better fit for it).  Perhaps the wish for android
> support should have been put on the Eclipse ideas page[2] and a link
> added to git's; I dunno.

Yeah, I'm not sure what an Android port would quite look like. In theory
I could probably build stock git for my rooted N1 using a
cross-compiler. But I can't imagine what I would use it for. A native
app seems like it would be more useful, and that pretty much requires
Java.

-Peff

Re: GSOC idea: build in scripts and cleanups

[Robert David]

[-- Attachment #1: Type: Text/Plain, Size: 2003 bytes --]

Dne sobota 26 března 2011 14:39:39 Jeff King napsal(a):
> On Fri, Mar 25, 2011 at 09:14:35PM -0500, Jonathan Nieder wrote:
> > > I was also thinking if there isn't PERL a better choice in rewriting
> > > shell scripts, due to planed porting (android, etc). Better than C.
> > > But I don't know android and other platform so much, so thats why I'm
> > > asking.
> > 
> > So far (on Windows and various Unixen) it seems that C is much easier
> > to work with as far as porting goes.[2]
> 
> If I were considering cleaning up and porting add--interactive to C, I
> think I would probably start with just porting the "-p" patch loop
> first. I think it's the part that most people use, and most callers
> don't support a generic "-i" but just the "-p" interface (e.g., you can
> do "git add -i" or "git add -p", but only "git checkout -p"). And that
> cuts down the size of the task somewhat.

I have read all the code in git-add--interactive. If I understand that 
correct, you mean to separate -i and -p int two different commands.
And make the -p option more like the -i (but just only for patching).

And consequently port them to C. 

> 
> As far as cleanup versus features, I think Thomas would have to comment
> on that. He is the one who did the most work on patch-mode, and
> therefore the one who most thinks it needs cleaned up. :)
> 

Is Thomas going to be a mentor in this task?

Robert David.


> > [1] Android is an odd example because the platform uses Java heavily
> > (so JGit might be a better fit for it).  Perhaps the wish for android
> > support should have been put on the Eclipse ideas page[2] and a link
> > added to git's; I dunno.
> 
> Yeah, I'm not sure what an Android port would quite look like. In theory
> I could probably build stock git for my rooted N1 using a
> cross-compiler. But I can't imagine what I would use it for. A native
> app seems like it would be more useful, and that pretty much requires
> Java.
> 
> -Peff

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: GSOC idea: build in scripts and cleanups

[Jeff King]

On Mon, Mar 28, 2011 at 10:55:22AM +0200, Robert David wrote:

> > If I were considering cleaning up and porting add--interactive to C, I
> > think I would probably start with just porting the "-p" patch loop
> > first. I think it's the part that most people use, and most callers
> > don't support a generic "-i" but just the "-p" interface (e.g., you can
> > do "git add -i" or "git add -p", but only "git checkout -p"). And that
> > cuts down the size of the task somewhat.
> 
> I have read all the code in git-add--interactive. If I understand that 
> correct, you mean to separate -i and -p int two different commands.
> And make the -p option more like the -i (but just only for patching).
> 
> And consequently port them to C. 

Sort of. The approach I would take would be:

  1. write patch loop code in C (i.e., the code that is going to replace
     the --patch code path from git-add--interactive.perl), and make it
     call-able from within other C programs

  2. write a new command, "git-add--interactive-patch", that behaves
     like "git-add--interactive --patch", but is based on your new C
     code.

  3. Port C callers of "git-add--interactive --patch" to the new
     interface from (1). This should be "add -p", "checkout -p", and
     "reset -p".

  4. Port non-C callers of "git-add--interactive --patch" to the new
     command from (2). This should be git-add--interactive.perl and
     git-stash.sh.

  5. Remove the perl patch code in git-add--interactive.perl; you can
     replace it with a call to git-add--interactive-patch.

But that doesn't mean that's the way it has to be done. Just what I
would do. :)

> > As far as cleanup versus features, I think Thomas would have to comment
> > on that. He is the one who did the most work on patch-mode, and
> > therefore the one who most thinks it needs cleaned up. :)
> 
> Is Thomas going to be a mentor in this task?

I hope so. I can also co-mentor if it helps.

-Peff

Re: GSOC idea: build in scripts and cleanups

[Thomas Rast]

Jeff King wrote:
> On Mon, Mar 28, 2011 at 10:55:22AM +0200, Robert David wrote:
> 
> > > As far as cleanup versus features, I think Thomas would have to comment
> > > on that. He is the one who did the most work on patch-mode, and
> > > therefore the one who most thinks it needs cleaned up. :)
> > 
> > Is Thomas going to be a mentor in this task?
> 
> I hope so. I can also co-mentor if it helps.

I'm certainly ready; I didn't propose any projects that I would not
also mentor.  (However, I won't mentor more than one...)

Note that while it's certainly a bonus, porting it to C probably makes
the project more difficult and time-consuming.  We'll have to see in
the proposal timelines however.

As for cleanup, my gut feeling right now is that the Perl code can
probably cope with incremental cleanups as required for each feature.
On the other hand, the C port should start from a clean redesign so as
to not rewrite it twice.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

Re: GSOC idea: build in scripts and cleanups

[Robert David]

[-- Attachment #1: Type: Text/Plain, Size: 1483 bytes --]

Dne středa 30 března 2011 17:39:12 Thomas Rast napsal(a):
> Jeff King wrote:
> > On Mon, Mar 28, 2011 at 10:55:22AM +0200, Robert David wrote:
> > > > As far as cleanup versus features, I think Thomas would have to
> > > > comment on that. He is the one who did the most work on patch-mode,
> > > > and therefore the one who most thinks it needs cleaned up. :)
> > > 
> > > Is Thomas going to be a mentor in this task?
> > 
> > I hope so. I can also co-mentor if it helps.
> 
> I'm certainly ready; I didn't propose any projects that I would not
> also mentor.  (However, I won't mentor more than one...)

Thank you for your attention.

> 
> Note that while it's certainly a bonus, porting it to C probably makes
> the project more difficult and time-consuming.  We'll have to see in
> the proposal timelines however.

It is not a big problem to me. I plan to begin full time work on this in half 
June.
Before that I plan some cleanups and fixes to simplify the upcoming porting.

> 
> As for cleanup, my gut feeling right now is that the Perl code can
> probably cope with incremental cleanups as required for each feature.
> On the other hand, the C port should start from a clean redesign so as
> to not rewrite it twice.

I agree that porting to C should be done from scratch. Thats why I would clean 
and extend the script first to make a "preview". Than when it will comply 
with requirements, I will port that from scratch to C.

Robert

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: GSOC idea: build in scripts and cleanups

[Robert David]

[-- Attachment #1: Type: Text/Plain, Size: 1446 bytes --]

Dne středa 30 března 2011 17:39:12 Thomas Rast napsal(a):
> Jeff King wrote:
> > On Mon, Mar 28, 2011 at 10:55:22AM +0200, Robert David wrote:
> > > > As far as cleanup versus features, I think Thomas would have to
> > > > comment on that. He is the one who did the most work on patch-mode,
> > > > and therefore the one who most thinks it needs cleaned up. :)
> > > 
> > > Is Thomas going to be a mentor in this task?
> > 
> > I hope so. I can also co-mentor if it helps.
> 
> I'm certainly ready; I didn't propose any projects that I would not
> also mentor.  (However, I won't mentor more than one...)
> 
> Note that while it's certainly a bonus, porting it to C probably makes
> the project more difficult and time-consuming.  We'll have to see in
> the proposal timelines however.
> 
> As for cleanup, my gut feeling right now is that the Perl code can
> probably cope with incremental cleanups as required for each feature.
> On the other hand, the C port should start from a clean redesign so as
> to not rewrite it twice.

Hi,

I have submitted the proposal on gsoc official site. I would like to ask for 
some comments on that. I don't know how you get access to student proposals to 
git. But I see that as proposal from "robertdavid" number 1.
I would like to focus myself only on this one proposal, so that is why I'm 
asking for comments, to extend the proposal as needed.

Thanks,
Robert David.


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: GSOC idea: build in scripts and cleanups

[Robert David]

[-- Attachment #1: Type: Text/Plain, Size: 4742 bytes --]

As a advice from Jonathan Nieder. I'm sending copy of my proposal to ml.

Robert.

####################################################
Abstract
Today git code consists of the base written in C and many helper shell or PERL 
scripts. While at a first time it is easier to write the script, final code is 
supposed to be in C. One of these scripts is git-add--interactive. 

Git-add--interactive is a helper script for git-add, which servers its options 
-i and -p. It definitely need to be integrated in git-add.
Which means, dividing the script in two parts: git-add -p and git-add -i. This 
involves usage of some code being written already in git.
Than writing some new functions common for both --patch and --interactive. And 
at last, fully integrating these options in git-add.

But before that, it is need to clean and extend the current git-add--
interactive, to serve user needs at the best. 
This means for example rewrite the main part of the way the patches are chosen 
by user, to let the user more flexibility.


Project goals
Main and final project goal is integrating fully git-add--interactive into 
current git-add code.
This task also include cleaning the functionality of this code, to make these 
functions more "standardized". 
This means consolidate the differences in these functions and make them more 
consistent in the user point of view.

How to consider this project has success? That is pretty easy, the already 
done functionality will be integrated in git-add and the user usage would be 
consistent.


Interfaces
As this is mainly part of git-add, that means that it will need to be changed 
at the most.
There are also another commands using this functionality now: git-am, git-
checkout, git-rebase. 
So there is possibility that there would be some changes needs to be done to 
adopt new interface.

I want to use as much code as possible from current git code-base, but this 
means further analysis to decide what exactly use and what not.


Time-line 
The official time-line consists of 12 coding week, starting 24th May. The mid-
evaluation is in the 8th week.
So the plan is written in week order beginning on the first coding week. 

1) Pre-coding time
I will read the documentation, analyze the git-add--interactive code and 
possibly change some small amount of code there. 
To clean thinks up for upcoming work. I will also seek the git code to find out 
where is what, for further rewriting analyze.

2) 1-3 week
Get the code of git-add--interactive cleaned and possibly written some of the 
consistency stuff.
Analyze the code with the focus on code already written in C.

3) 4-5 week
Get the cleaning and consistency stuff done. 
Collect the community feedback to the code, to get things improve where it is 
needed.

4) 6-7 week
Start the rewriting period, this will provide some architectonic basis to be 
included in git-add.

5) 8-11 week
Extend the C code to the state it should be.
Adopt other git commands to work with the new interface correctly.
Test extensively.
Update documentation where needed.

6) 12 week
Write more documentation, to document what was done and how.
Correct bugs and test.


About me
I'm a student of second year of bachelors study on Faculty of Information 
Technology, Czech Technical University in Prague, Czech Republic. 
I have some experience with C and script languages, because I did worked for 
company making client software for two years.

I have never contribute to open source projects, in the means of submitting 
patches (I did some bug-reporting in projects like midnight-commander, arch 
linux, debian linux). 
But as I love open source and use that for a long time, I realized I have to 
begin participating in development. Thus I see GSOC as a good startup for me.
My prior experience is doing shell and PERL scripts, because I do that as 
"every-week" work. I work also for Prague children free-time organization, 
learning children open source stuff and little bit programming, mainly some 
small scripts (www.ddmpraha.cz).

My git experience is purely user based. I use git for everyday work, 
administrating my computers and servers, keeping track of my school works, and 
my personal projects. In the past I wrote some scripts helping develop debian 
packages using purely git (as the software was not released yet I can't 
explain it further and I also don't work for that company any more. 
www.zonio.net). 
Because of GSOC start I wanted to find out more information about this proposal 
on git mailing list (http://article.gmane.org/gmane.comp.version-
control.git/170036)
####################################################

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: GSOC idea: build in scripts and cleanups

[Junio C Hamano]

Robert David <robert.david.public@gmail.com> writes:

> 1) Pre-coding time
> 2) 1-3 week
> 3) 4-5 week
> 4) 6-7 week
> 5) 8-11 week
> Extend the C code to the state it should be.
> Adopt other git commands to work with the new interface correctly.
> Test extensively.
> Update documentation where needed.
>
> 6) 12 week
> Write more documentation, to document what was done and how.
> Correct bugs and test.

I am afraid to say that the above schedule is too ambitious and does not
leave any time for reviews and re-rolls.  Please keep in mind that
historically patch series by more experienced contributors of substantial
size (comparable or even smaller scale than the topic you are proposing)
all typically took three or four review-reroll cycles, if not less, and we
don't automatically get extra review bandwidth just because GSoC is going
on.

I am starting to suspect that it might make sense to say "as far as GSoC
participation is concerned, we would call a topic "merged upstream" when
it hits 'next', even if it is not ready for 'master' at the end of the
term".

What do regular reviewers and potential mentors think?  Perhaps we have
more stringent quality requirements than other open source projects that
take "commit first, review and fix as necessary" cycle, and they may
declare success when "commit first" happens.  If that is the case, 'next',
whose definition is "without glaring design and implementation bugs and
fit enough for dogfooding, but needs extra polish", might be a better
success criteria to be fair for our students.

I am not in the mentor pool and I would rather not to be to stay neutral,
so I'll leave it up to the mentors.

Re: GSOC idea: build in scripts and cleanups

[Robert David]

[-- Attachment #1: Type: Text/Plain, Size: 2329 bytes --]

Dne pondělí 04 dubna 2011 20:09:00 Junio C Hamano napsal(a):
> Robert David <robert.david.public@gmail.com> writes:
> > 1) Pre-coding time
> > 2) 1-3 week
> > 3) 4-5 week
> > 4) 6-7 week
> > 5) 8-11 week
> > Extend the C code to the state it should be.
> > Adopt other git commands to work with the new interface correctly.
> > Test extensively.
> > Update documentation where needed.
> > 
> > 6) 12 week
> > Write more documentation, to document what was done and how.
> > Correct bugs and test.
> 
> I am afraid to say that the above schedule is too ambitious and does not
> leave any time for reviews and re-rolls.  Please keep in mind that
> historically patch series by more experienced contributors of substantial
> size (comparable or even smaller scale than the topic you are proposing)
> all typically took three or four review-reroll cycles, if not less, and we
> don't automatically get extra review bandwidth just because GSoC is going
> on.

Thanks, this is what I wanted to hear. 
I wrote the proposal from my point of view. I'm prepared to change the size of 
the task and schedule on mentors and developers comments.
I'm also trying to understand your development cycle, to get that more 
precise. But I want also say that I'm prepared for a lot of work. I have the 
time in this period.

If I understand correctly, you mean to divide this task in more terms? And do 
less more precise. Test more, etc.

Robert 

> 
> I am starting to suspect that it might make sense to say "as far as GSoC
> participation is concerned, we would call a topic "merged upstream" when
> it hits 'next', even if it is not ready for 'master' at the end of the
> term".
> 
> What do regular reviewers and potential mentors think?  Perhaps we have
> more stringent quality requirements than other open source projects that
> take "commit first, review and fix as necessary" cycle, and they may
> declare success when "commit first" happens.  If that is the case, 'next',
> whose definition is "without glaring design and implementation bugs and
> fit enough for dogfooding, but needs extra polish", might be a better
> success criteria to be fair for our students.
> 
> I am not in the mentor pool and I would rather not to be to stay neutral,
> so I'll leave it up to the mentors.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: GSOC idea: build in scripts and cleanups

[Jeff King]

On Mon, Apr 04, 2011 at 11:09:00AM -0700, Junio C Hamano wrote:

> Robert David <robert.david.public@gmail.com> writes:
> 
> > 1) Pre-coding time
> > 2) 1-3 week
> > 3) 4-5 week
> > 4) 6-7 week
> > 5) 8-11 week
> > Extend the C code to the state it should be.
> > Adopt other git commands to work with the new interface correctly.
> > Test extensively.
> > Update documentation where needed.
> >
> > 6) 12 week
> > Write more documentation, to document what was done and how.
> > Correct bugs and test.
> 
> I am afraid to say that the above schedule is too ambitious and does not
> leave any time for reviews and re-rolls.  Please keep in mind that
> historically patch series by more experienced contributors of substantial
> size (comparable or even smaller scale than the topic you are proposing)
> all typically took three or four review-reroll cycles, if not less, and we
> don't automatically get extra review bandwidth just because GSoC is going
> on.

I agree. I think it's important to take review-reroll cycles into
account. Not just in terms of allocating time, but also considering the
latency, and keeping the student's pipeline full of work while waiting
on review.

To that end, I think it's useful to break the problem up as much as
possible, and feed chunks to the list as quickly as possible. Even if
you end up with something like:

  Week 2: send refactoring of functionality X
  Week 3: build new functionality Y on top of X

Obviously "Y" is going to depend somewhat on the refactoring of "X". But
you can say in the RFC/PATCH for Y that X is still ongoing, and to
review with that in mind. And that keeps the student doing something
during week 3 while reviews for X flow in.

In the past, students haven't been very engaged with the list community,
and I think that has hurt us. The code gets dumped as a whole at the
end, and review is a lot harder, and GSoC is over, so the student ends
up busy with going back to school. And the student never really learns
the "normal" way that contributors interact with the community, which
makes them less likely to become long-time contributors.

So I'd really like to see this year's projects breaking contributions
into smaller, reviewable bits and submitting over the course of the
whole summer.

> I am starting to suspect that it might make sense to say "as far as GSoC
> participation is concerned, we would call a topic "merged upstream" when
> it hits 'next', even if it is not ready for 'master' at the end of the
> term".

Yeah, I think that is reasonable. Stuff that makes it into "next" is
usually of good quality, but just needs time to shake out bugs. Every
once in a while we find that something in next turns out be utter crap,
but usually it is obvious before it even gets merged there. Hopefully a
student will be available to fix minor bugs as the topic cooks in next,
but if not, it is probably something the mentor can do.

-Peff

Re: GSOC idea: build in scripts and cleanups

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Mon, Apr 04, 2011 at 11:09:00AM -0700, Junio C Hamano wrote:
>
>> I am afraid to say that the above schedule is too ambitious and does not
>> leave any time for reviews and re-rolls.  Please keep in mind that
>> historically patch series by more experienced contributors of substantial
>> size (comparable or even smaller scale than the topic you are proposing)
>> all typically took three or four review-reroll cycles, if not less, and we
>> don't automatically get extra review bandwidth just because GSoC is going
>> on.
>
> I agree. I think it's important to take review-reroll cycles into
> account. Not just in terms of allocating time, but also considering the
> latency, and keeping the student's pipeline full of work while waiting
> on review.

During the course of GSoC program, one will learn that other community
members do not have any obligation to keep one's pipeline full [*1*], and
learn how to cope with it.  Measuring the availability of others so that
one can make optimum use of others' time is a skill to be learned to be
successful in interacting with the open source community.

It takes people skills, not just "hacking" [*2*].

>   Week 3: build new functionality Y on top of X
>
> Obviously "Y" is going to depend somewhat on the refactoring of "X". But
> you can say in the RFC/PATCH for Y that X is still ongoing, and to
> review with that in mind. And that keeps the student doing something
> during week 3 while reviews for X flow in.

Yes.

> In the past, students haven't been very engaged with the list community,
> and I think that has hurt us. The code gets dumped as a whole at the
> end, and review is a lot harder, and GSoC is over, so the student ends
> up busy with going back to school.

While GSoC's "success" criteria (from stipend payment point of view) may
be to finish the defined task for the project, the program's larger
objective is to give an opportunity to students to learn how to work with
the open source community, so in that sense, I'd call that a total
failure.

I agree that we should help fixing that.

[Footnote]

*1* The mentor is special, has that obligation, and fulfils the obligation
by exercising his open source community skills and guiding the student to
break down the project into smaller subtasks to match the bandwidth of the
community.

*2* That incidentally is what I learned from working with Linus during the
early 2 months of git's life.  "People skill" is not just "being nice,
polite and diplomatic", none of which describes Linus ;-).

I think the most important skills include

 * being observant and know what others are doing, how busy they are, what
   patches are in flight that may have potential interactions with your
   work, and at what velocity these patches are progressing; and

 * clearly communicate what you are doing, in what order, for what reason,
   and where the boundaries of your goals are, and demonstrate that you
   are dedicated to the cause of the project.

The former would help you avoid duplicated work (if somebody is doing what
you can take advantage of and it is going at a reasonable pace, you can
spend your time on areas that do not depend on his work in the meantime
and wait) and unnecessary conflicts. The latter would make it more likely
that others would want to help you by correcting flaws in your design
assumptions and filling the gap that you are planning to punt, and help
these others to avoid duplicated work and unnecessary conflicts while
helping you.

Re: GSOC idea: build in scripts and cleanups

[Jeff King]

On Mon, Apr 04, 2011 at 09:43:09AM +0200, Robert David wrote:

> Today git code consists of the base written in C and many helper shell or PERL 
> scripts. While at a first time it is easier to write the script, final code is 
> supposed to be in C. One of these scripts is git-add--interactive.
> 
> Git-add--interactive is a helper script for git-add, which servers its options 
> -i and -p. It definitely need to be integrated in git-add.

Can you expand on "definitely" here? I.e., what are the motivations for
this change? I know what some of the arguments are, and I know how _I_
would answer the question, but I want to hear what _you_ think.

And I am not just trying to be pedantic. Understanding the motivations
for a change will help us figure out the right way to go about it, and
how to figure out if we are successful at making it.

> Interfaces
> As this is mainly part of git-add, that means that it will need to be changed 
> at the most.
> There are also another commands using this functionality now: git-am, git-
> checkout, git-rebase. 

I don't think this is right. "am" and "rebase" have interactive modes,
but the code and functionality are not shared at all with
add--interactive. But you are missing some other commands that do have
patch modes built on add--interactive.

-Peff

Re: GSOC idea: build in scripts and cleanups

[Robert David]

[-- Attachment #1: Type: Text/Plain, Size: 2339 bytes --]

Dne úterý 05 dubna 2011 18:52:12 Jeff King napsal(a):
> On Mon, Apr 04, 2011 at 09:43:09AM +0200, Robert David wrote:
> > Today git code consists of the base written in C and many helper shell or
> > PERL scripts. While at a first time it is easier to write the script,
> > final code is supposed to be in C. One of these scripts is
> > git-add--interactive.
> > 
> > Git-add--interactive is a helper script for git-add, which servers its
> > options -i and -p. It definitely need to be integrated in git-add.
> 
> Can you expand on "definitely" here? I.e., what are the motivations for
> this change? I know what some of the arguments are, and I know how _I_
> would answer the question, but I want to hear what _you_ think.


From my point of view it means to keep the code consistent in the whole 
project. This further means better porting, avoid duplications and thus easier 
maintenance. And at last, I think it is my personality that takes me to 
project like this. I like to polish things up. 

Even I like the PERL, I would not like to maintain project consisted in more 
programming languages for a long term.


> 
> And I am not just trying to be pedantic. Understanding the motivations
> for a change will help us figure out the right way to go about it, and
> how to figure out if we are successful at making it.
> 
> > Interfaces
> > As this is mainly part of git-add, that means that it will need to be
> > changed at the most.
> > There are also another commands using this functionality now: git-am,
> > git- checkout, git-rebase.
> 
> I don't think this is right. "am" and "rebase" have interactive modes,
> but the code and functionality are not shared at all with
> add--interactive. But you are missing some other commands that do have
> patch modes built on add--interactive.

Thanks for the notice, I will dive more deeply into it.


I was also thinking about the timeline of this project. And maybe another 
solution is to constantly and slowly improve git-add--interactive, to make it 
accepted in next (maybe master) on end of the SoC period. And in parallel 
write the C code, which would be prepared for more longer term testing and 
bugfixing to get in next. This also means for me a clear way, how to continue 
contributing to git.

Robert 

> 
> -Peff

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: GSOC idea: build in scripts and cleanups

[Robert David]

[-- Attachment #1: Type: Text/Plain, Size: 5248 bytes --]

I'm sending the updated proposal. 

I was thinking about that and realized to do the cleaning and update of git-
add--interactive during GSoC and try to get that upstream (master,next). 
And as second part, start rewriting it into the C as longer term project. 
Which will lead in reviews past the GSoC during the year.


Robert.

########################################################
Abstract

Today git code consists of the base written in C and many helper shell or PERL 
scripts. While at a first time it is easier to write the script, final code is 
supposed to be in C. One of these scripts is git-add--interactive. 

Git-add--interactive is a helper script for git-add, which servers its options 
-i and -p. It definitely need to be integrated in git-add.
Which means, dividing the script in two parts: git-add -p and git-add -i. This 
involves usage of some code being written already in git.
Than writing some new functions common for both --patch and --interactive. And 
at last, fully integrating these options in git-add.

But before that, it is need to clean and extend the current git-add--
interactive, to serve user needs at the best. 
This means for example rewrite the main part of the way the patches are chosen 
by use, to let the user more flexibility.


Project goals

Main and final project goal is integrating fully git-add--interactive into 
current git-add code.
This task also include cleaning the functionality of this code, to make these 
functions more "standardized". 
This means consolidate the differences in these functions and make them more 
consistent in the user point of view.
As this project is a bit longer term for GSoC period, 
the final from GSoC point of view will be doing the first part of the work 
completely (the script part) 
and prepare the C part as architectonic preview for ongoing review.

How to consider this project has success?
That is pretty easy, the cleaned and extended git-add--interactive script will 
be merged into the master branch.

Interfaces

As this is mainly part of git-add, that means that it will need to be changed 
at the most.
There are also another commands using this functionality now: git-stash. 
So there is possibility that there would be some changes needs to be done to 
adopt new interface.

I want to use as much code as possible from current git code-base, but this 
means further analysis to decide what exactly use and what not.


Time-line 

The official time-line consists of 12 coding week, starting 24th May. The mid-
evaluation is in the 8th week.
So the plan is written in week order beginning on the first coding week. 

1) Pre-coding time
I will read the documentation, analyze the git-add--interactive code and 
possibly change some small amount of code there. 
To clean thinks up for upcoming work. I will also seek the git code to find out 
where is what, for further rewriting analyze.

2) 1-3 week
Get the code of git-add--interactive cleaned and possibly written some of the 
consistency stuff.
Analyze the code with the focus on code already written in C.

3) 4-5 week
Get the cleaning and consistency stuff done. 
Collect the community feedback to the code, to get things improve where it is 
needed.

4) 6-8 week
Work on the extensions to the script. Test and repair bugs, that will occur. 
Start the rewriting period, this will provide some architectonic basis to be 
included in git-add.

5) 9-11 week
Collect reviews and solve bugs.
Continue with the rewriting work. 
Test extensively.
Update documentation where needed.

6) 12 week
Write more documentation, to document what was done and how.
Correct remaining bugs and test.


About me

I'm a student of second year of bachelors study on Faculty of Information 
Technology, Czech Technical University in Prague, Czech Republic. 
I have some experience with C and script languages, because I did worked for 
company making client software for two years.
I have never contribute to open source projects, in the means of submitting 
patches (I did some bug-reporting in projects like midnight-commander, arch 
linux, debian linux). 
But as I love open source and use that for a long time, I realized I have to 
begin participating in development. Thus I see GSOC as a good startup for me.
My prior experience is doing shell and PERL scripts, because I do that as 
"every-week" work. I work also for Prague children free-time organization, 
learning children open source stuff and little bit programming, mainly some 
small scripts (www.ddmpraha.cz).

My git experience is purely user based. I use git for everyday work, 
administrating my computers and servers, keeping track of my school works, and 
my personal projects. In the past I wrote some scripts helping develop debian 
packages using purely git (as the software was not released yet I cant explain 
it further and I also don't work for that company any more. www.zonio.net). 
Because of GSOC start I wanted to find out more information about this proposal 
on git mailing list (http://article.gmane.org/gmane.comp.version-
control.git/170036)
########################################################

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: GSOC idea: build in scripts and cleanups

[Junio C Hamano]

Robert David <robert.david.public@gmail.com> writes:

> This task also include cleaning the functionality of this code, to make these 
> functions more "standardized". 
> This means consolidate the differences in these functions and make them more 
> consistent in the user point of view.

Standardized, consolidate, consistent are all pretty words but at the same
time they are highly subjective. Even though you may not have to have the
detailed description of everything in the proposal, you would need to at
least have a handful of examples of what are wrong with the current code
in mind when making such a sweeping statement, but I cannot sense that
from this proposal.

While reading the current add-i code recently, I noticed that over time
the code quality deteriorated compared to the reasonably clean design in
the original version, perhaps due to excessive copying and mimicing,
especially when "Edit patch" feature was added and then more recently when
new modes to apply patches in different directions are added to allow
other commands like "git reset" and "git checkout" to implement their
"--patch" options.

An example of a clean-up in the right direction may be a patch I posted a
few days ago:

  http://thread.gmane.org/gmane.comp.version-control.git/170685/focus=171017 

Is that the kind of "consolidation" you have in mind?

Also I am a bit puzzled by the inconsistency between the various stages of
your goals.  A clearly stated primary goal is:

> Main and final project goal is integrating fully git-add--interactive into 
> current git-add code.

but the success criteria is only about the above "This task also include"?

Re: GSOC idea: build in scripts and cleanups

[Robert David]

[-- Attachment #1.1: Type: Text/Plain, Size: 4114 bytes --]

Dne pátek 08 dubna 2011 00:19:05 Junio C Hamano napsal(a):
> Robert David <robert.david.public@gmail.com> writes:
> > This task also include cleaning the functionality of this code, to make
> > these functions more "standardized".
> > This means consolidate the differences in these functions and make them
> > more consistent in the user point of view.
> 
> Standardized, consolidate, consistent are all pretty words but at the same
> time they are highly subjective. Even though you may not have to have the
> detailed description of everything in the proposal, you would need to at
> least have a handful of examples of what are wrong with the current code
> in mind when making such a sweeping statement, but I cannot sense that
> from this proposal.

While I have read the code, first thing I noticed is the inconsistence of 
general coding standarts used. Sometimes the code is pretty nice divided into 
subrutines to get the code understand fast, and sometimes it is a huge 
subrutine that does as much things as possible, and thus is no so good to read 
(also I don't speak here about comments missing in a lot of code, where 
sometimes there are some). With the lack of documentation in Documentation 
directory, that means that this code is something like black box. And hard to 
learn for newcomer. Tell me if Im wrong, but isnt the open-source demand also 
to let as many people ac possible to involve in development? 

The biggest example is the "patch_update_file". Which can be divided in few 
parts. For example the "if ($line) { ..... }" big big box.
This also mean for list_and_chose and list_modified (and maybe others).

Sometimes there are perl like statements:

"something if|for|while|etc something"
and common things are written next like 
"if (something) { something }"

Is there demand to write scripts (mainly perl) like in C to keep the code more 
readable for C people or just write it more in perl style (to keep the code 
cleaner and shorter)?

Another thing is parsing command line options, why not use Getopt::Long here?



> 
> While reading the current add-i code recently, I noticed that over time
> the code quality deteriorated compared to the reasonably clean design in
> the original version, perhaps due to excessive copying and mimicing,
> especially when "Edit patch" feature was added and then more recently when
> new modes to apply patches in different directions are added to allow
> other commands like "git reset" and "git checkout" to implement their
> "--patch" options.
> 
> An example of a clean-up in the right direction may be a patch I posted a
> few days ago:
> 
>  
> http://thread.gmane.org/gmane.comp.version-control.git/170685/focus=171017
> 
> Is that the kind of "consolidation" you have in mind?

I have read that conversation before. I also tried to reproduce that bug, but 
have no success (in version 1.7.4.3). But before I read the code and try to 
find something that this can occur, you send the patch. I'm not so fast as 
you:)


> 
> Also I am a bit puzzled by the inconsistency between the various stages of
> 
> your goals.  A clearly stated primary goal is:
> > Main and final project goal is integrating fully git-add--interactive
> > into current git-add code.
> 
> but the success criteria is only about the above "This task also include"?

I will try to clear that up today, to finalize that in the meaning of doing the 
cleanup and extension part as part of GSoC and prepare the rewriting part for 
further work. I still think I can handle that and rewrite that as part of 
GSoC, but I didn't do such a big open-source project before, so I'm going to 
adopt your style of coding standards and timings. 

I also attach a example perl code I'm now writing, it is a half done(40%) data 
-> animated graph converter (part of my seminar work). 

Robert.

> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #1.2: semestralka.pl --]
[-- Type: application/x-perl, Size: 5138 bytes --]

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: GSOC idea: build in scripts and cleanups

[Jonathan Nieder]

Hi,

Robert David wrote:

> I'm sending copy of my proposal to ml.

Thanks, and sorry for a slow response.

Full disclosure: I locally use a patch[1] to teach a --patience option
to add -p, checkout -p, etc (to use the "patience diff" algorithm).  I
never submitted it since it wasn't clear to me how to integrate it
(and other diff options) properly.  I will be very happy to see a
cleaner add--interactive.

So I'm a likely early consumer of your code, although I don't think
I'll have time to co-mentor.  Luckily there seems to be no shortage of
willing mentors.

Some quick impressions.  This is off-the-cuff; please feel free to let
me know if something sounds crazy.

> Today git code consists of the base written in C and many helper shell or PERL 
> scripts. While at a first time it is easier to write the script, final code is 
> supposed to be in C. One of these scripts is git-add--interactive. 
[...]
> But before that, it is need to clean and extend the current git-add--
> interactive, to serve user needs at the best. 

I see two goals in tension: (1) to integrate add--interactive as C
code, and (2) to clean it up and change its feature set.  Either one
could happen without the other, and for planning it would be useful to
know which is going to drive decisions (e.g., what if time is short
and something has to get cut?).

If (1) is the main goal, it might be easiest to first translate the
existing code, perhaps modulo small preparatory changes that make the
translation easier, into C and leave major changes for afterwards.
Tracking down bugs due to a major change (like switch in
implementation language) is a lot easier if the pre-change version is
well tested and well understood.

If (2) is the main goal, it might be easiest to rewrite small parts of
add--interactive in C where convenient rather than rewriting the whole
thing.  In that story, the result is a series of small patches without
any single world-changing patch. :)

[...]
> How to consider this project has success? That is pretty easy, the already 
> done functionality will be integrated in git-add and the user usage would be 
> consistent.

After each patch, the test suite should pass.  If some important
functionality is not exercised in the test suite, ideally it can be
added to the test suite.  (Though that's no replacement for trying the
changes in day-to-day use, of course.)

[...]
> The official time-line consists of 12 coding week, starting 24th May. The mid-
> evaluation is in the 8th week.
> So the plan is written in week order beginning on the first coding week. 

Jeff and Junio's advice seems sane to me.  More advice that might help
with writing a timeline: [2] and Christian's reply.

Because of the uncertainty I mentioned above, it's hard to give an
example, but an ideal proposal would include a timeline that gives a
technical plan for the summer.

Also during the bonding time or even earlier it would be nice to get
used to sending patches and reviewing them (as explained in
Documentation/SubmittingPatches) if you find time for it.

Thanks again, and hope that helps.
Jonathan

[1] http://bugs.debian.org/522361
[2] http://thread.gmane.org/gmane.comp.version-control.git/142623/focus=142877

Re: GSOC idea: build in scripts and cleanups

[Robert David]

[-- Attachment #1: Type: Text/Plain, Size: 4011 bytes --]

Dne pondělí 11 dubna 2011 08:34:44 Jonathan Nieder napsal(a):
> Hi,
> 
> Robert David wrote:
> > I'm sending copy of my proposal to ml.
> 
> Thanks, and sorry for a slow response.

I'm also sorry for the late response, I was off for holiday now.

> 
> Full disclosure: I locally use a patch[1] to teach a --patience option
> to add -p, checkout -p, etc (to use the "patience diff" algorithm).  I
> never submitted it since it wasn't clear to me how to integrate it
> (and other diff options) properly.  I will be very happy to see a
> cleaner add--interactive.
> 
> So I'm a likely early consumer of your code, although I don't think
> I'll have time to co-mentor.  Luckily there seems to be no shortage of
> willing mentors.
> 
> Some quick impressions.  This is off-the-cuff; please feel free to let
> me know if something sounds crazy.
> 
> > Today git code consists of the base written in C and many helper shell or
> > PERL scripts. While at a first time it is easier to write the script,
> > final code is supposed to be in C. One of these scripts is
> > git-add--interactive.
> 
> [...]
> 
> > But before that, it is need to clean and extend the current git-add--
> > interactive, to serve user needs at the best.
> 
> I see two goals in tension: (1) to integrate add--interactive as C
> code, and (2) to clean it up and change its feature set.  Either one
> could happen without the other, and for planning it would be useful to
> know which is going to drive decisions (e.g., what if time is short
> and something has to get cut?).
> 
> If (1) is the main goal, it might be easiest to first translate the
> existing code, perhaps modulo small preparatory changes that make the
> translation easier, into C and leave major changes for afterwards.
> Tracking down bugs due to a major change (like switch in
> implementation language) is a lot easier if the pre-change version is
> well tested and well understood.
> 
> If (2) is the main goal, it might be easiest to rewrite small parts of
> add--interactive in C where convenient rather than rewriting the whole
> thing.  In that story, the result is a series of small patches without
> any single world-changing patch. :)

As I updated the proposal, to focus mainly on the (2) way to go. I agree with 
your suggestions and I think it will be part of the second round, when the 
git-add--interactive will be done and tested enough.


> 
> [...]
> 
> > How to consider this project has success? That is pretty easy, the
> > already done functionality will be integrated in git-add and the user
> > usage would be consistent.
> 
> After each patch, the test suite should pass.  If some important
> functionality is not exercised in the test suite, ideally it can be
> added to the test suite.  (Though that's no replacement for trying the
> changes in day-to-day use, of course.)
> 
> [...]
> 
> > The official time-line consists of 12 coding week, starting 24th May. The
> > mid- evaluation is in the 8th week.
> > So the plan is written in week order beginning on the first coding week.
> 
> Jeff and Junio's advice seems sane to me.  More advice that might help
> with writing a timeline: [2] and Christian's reply.
> 
> Because of the uncertainty I mentioned above, it's hard to give an
> example, but an ideal proposal would include a timeline that gives a
> technical plan for the summer.
> 
> Also during the bonding time or even earlier it would be nice to get
> used to sending patches and reviewing them (as explained in
> Documentation/SubmittingPatches) if you find time for it.

I'm planing to submit some patches as soon as it will have more time to fully 
get in the code, I think it will be something like perl style cleanups.

Thanks for your attention,
Robert. 

> 
> Thanks again, and hope that helps.
> Jonathan
> 
> [1] http://bugs.debian.org/522361
> [2]
> http://thread.gmane.org/gmane.comp.version-control.git/142623/focus=142877

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 230 bytes --]