Re: GSOC Proposal.

Re: GSOC Proposal.

[Jakub Narebski]

On Tue, 27 Mar 2012, jaseem abid wrote:

> Hello everyone,
> 
> I am interested in participating in GSoC 2012. I have gone through the
> ideas page and  mailing list archives by now.  I would like to suggest a
> new approach for the git instaweb rather than just JavaScript rewrite,
> which i believe can improve the overall quality of the software.

First of all, it is _gitweb_ that is git web interface subsystem.
git-instaweb is the name of script that runs gitweb (using available
web server, including apache, lighttpd, webrick and plackup) and by
default also start web browser opening gitweb, for a repository you
are in.  This is important difference.
 
For the rest I'll assume s/instaweb/gitweb/.

Second, the proposed "Use JavaScript library / framework in gitweb"
project has the advantage of being possible to be done in incremental
improvements (even without expanding client-side scripting in gitweb,
i.e. adding new features possible with using JavaScript library),
rather than all-or-nothing complete rewrite.  The potential for failure
is much higher in "rewrite" project.


It is nice that there is a would-be prospective student that proposes
his own blue-sky idea of a project.  The projects listed on

  https://github.com/peff/git/wiki/SoC-2012-Ideas

were meant to be only _example_ ideas.


> Data and presentation are too tightly bound in the current code base of
> gitweb. CSS classes are hard coded in the Perl script and it makes
> further development difficult and is considered to be a bad practice
> generally.

Separation of content and presentation is considered a good practice,
but it would require in my opinion heavy rewrite to use one of existing
Perl web (micro)frameworks[1] rather than pure old CGI.pm and hand-crafted
code plus some templating engine[2].  This is IMVHO too much work to do
in a single Google Summer of Code.

Though rewriting gitweb using Perl web framework / creating new web
interface based on gitweb could be a good GSoC project.  The problem
might be with selecting self contained set of work that can be done
in time available for GSoC.

Also there is advantage in gitweb using only minimal number of non-core
required Perl modules.  There is always Gitalist, a git web interface
written using Catalyst MVC framework (based originally on some old gitweb
code, much extended since).

[1] For example one of modern PSGI-based microframeworks like Dancer
    or Mojolicious.
[2] For example Template::Toolkit, or HTML::Zoom, or Template::Declare.

> Presentable HTML is the only output from gitweb as of now that 
> can't be used by anything other than a browser.

It doesn't make much sense to use in something other than a browser.
_What_ other use do you think about?

YAGNI, and don't overengineer.

> Using Perl to generate raw 
> data in JSON format (XML will also do, but not preferred) and using
> JavaScript to render it in the browser will lead to the separation of the
> application into multiple modules.

NO.  Requiring JavaScript to be able to render gitweb output is totally
out of question.  Especially for intended audience, which is composed
heavily of programmers and hackers, both of which are prone to using text
web browsers and running NoScript-equivalents or just with JavaScript
turned off.

Note also that splitting gitweb into smaller modules would be not easy,
as it will require reworking current "longjump"-based (non-local goto
outside subroutine) error handling into e.g. exception-based one
(e.g. Try::Tiny + HTTP::Exception)... or porting gitweb to web framework
which does that for us.

Anyway using JSON for output and _client side_ JavaScript for generating
(X)HTML and CSS looks like monumentally stupid idea.

> A Perl hacker can improve JSON 
> generating modules someday without bothering what is being done to the data
> after it is generated.

If / when gitweb is properly split into separate modules, this can be
done by generating Perl data structures (OOP or not), rather than going
microkernel-like design of bogging down performance with marshalling
and unmarshalling to and from JSON.

Proper separation of responsibilities and proper API design and improving
contents side can be done independently on presentation side.

> A JavaScript programmer can use this data to do any 
> of the tasks listed efficiently without bothering how it is being
> generated. This potentially opens the project to a wide developer audience
> and most probably much more efficient code. Generating UI in Perl for non
> JavaScript browsers can be implemented as a 3rd module. The languages fit
> in the context very well.

<irony>
Yep, and adding new view, or new feature in an existing view will require
both knowledge of Perl (to get data) and JavaScript + JS library (to show
data).  Wonderfull way of enlarging pool of possible gitweb developers.
</irony>

> I have submitted a detailed proposal on the melange website. Mentors please
> be kind to go through the same.
> 
> http://www.google-melange.com/gsoc/proposal/review/google/gsoc2012/jaseemabid/1

-- 
Jakub Narebski
Poland

GSoC proposal

[work]

As I was strongly encouraged to submit my GSoC proposal, I'll post it 
here and CC to my possible mentor.
Please, provide with your feedback about my draft. You can also comment 
it right in the Google doc. Thanks in advance

Proposal: 
https://docs.google.com/document/d/1Hpu9FfD3wb7qgWgTiKtIAie41OXK3ufgnhnNuRaEH4E

GSOC proposal

[Fredrik Gustafsson]

Hi,
I'm interested in working as a student for git in the GSOC program this
year.

I'm running a lot of web projects with heavy use of git submodules (each
project has around 40 submodules) and therefore I'm very interested in
enchant the git submodule experience.

I'm asking for your oppinions and idées for my planned todolist for this
summer (if I get accepted of course).

== Preventing false "pointers" ==
It's far too easy to push a gitrepo pointing to a submodule version that
is not existing on the server. Prevent this by checking for available
submodule versions before acceptingt the push.

== Threat every module alike ==
When failing fetching a submodule, continue fetching the next one
instead of dying. There's no need to prevent fetching a submodule
beginning at 'z' just because a failing in fetching a submodule
beginning at 'a'. The submodules should not be alphabetically dependant
as they are now.

== Make submodule changes globally visible ==
Give git-log submodule support. A git log of showing a new version of a
module should (if choosen by --submodules or alike) also list the
changes to that submodule since the last version of the submodule was
commited.

== Move submodules into core ==
This last bit is excellent described in the link below.

So, what do you all think? Is it too much? Too little? Is the quests
relevant to git?

See also
https://git.wiki.kernel.org/index.php/SoC2011Ideas#Git_Submodules_Enhancements

-- 
Med vänliga hälsningar
Fredrik Gustafsson

tel: 0733-608274
e-post: iveqy@iveqy.com

Re: GSOC proposal

[Jens Lehmann]

Am 24.03.2011 23:01, schrieb Fredrik Gustafsson:
> Hi,
> I'm interested in working as a student for git in the GSOC program this
> year.
> 
> I'm running a lot of web projects with heavy use of git submodules (each
> project has around 40 submodules) and therefore I'm very interested in
> enchant the git submodule experience.

Great to hear that!

> I'm asking for your oppinions and idées for my planned todolist for this
> summer (if I get accepted of course).
> 
> == Preventing false "pointers" ==
> It's far too easy to push a gitrepo pointing to a submodule version that
> is not existing on the server. Prevent this by checking for available
> submodule versions before acceptingt the push.

Yes, requiring to force such a push is an issue raised quite often (I
think addressing this issue would be a good starting point for people
who want to get involved in enhancing the submodule experience).

> == Threat every module alike ==
> When failing fetching a submodule, continue fetching the next one
> instead of dying. There's no need to prevent fetching a submodule
> beginning at 'z' just because a failing in fetching a submodule
> beginning at 'a'. The submodules should not be alphabetically dependant
> as they are now.

I assume you are talking about the implicit fetch done by "git submodule
update" here. The recursive submodule fetch that "git fetch" learned
recently continues to fetch other submodules even if one or more fetches
failed. But you are right that "git submodule update" should attempt to
continue updating the remaining submodules too even if one of those
updates failed along the way (This should be achieved with even less
effort than the push issue mentioned above, so it would be an even
easier starting point for people who want to get involved).

> == Make submodule changes globally visible ==
> Give git-log submodule support. A git log of showing a new version of a
> module should (if choosen by --submodules or alike) also list the
> changes to that submodule since the last version of the submodule was
> commited.

Yeah, adding an option so the user is able to tell "git log" she wants
to see all submodule commits in detail too would also be very nice
(Also some other commands could profit from being able to tell them to
recurse into submodules).

> == Move submodules into core ==
> This last bit is excellent described in the link below.

Assuming that you are referring to placing the repository for each
populated submodule in the superproject's $GIT_DIR/modules, me thinks
that that is the core part of this GSoC project. So I'd be very glad
to have someone on board who wants to solve this issue.

> So, what do you all think? Is it too much? Too little? Is the quests
> relevant to git?

I like it! I think all these issues are relevant to achieve better
submodule support in git and I'm looking forward to see a student
(maybe you? ;-) starting to work on this.

(And, as every year, it's a good idea for a prospective student to get
involved in the git community before his application is accepted ...
sending some patches is a good way to do that, maybe regarding one of
the first two issues raised here? ;-)

Re: GSOC proposal

[Junio C Hamano]

Jens Lehmann <Jens.Lehmann@web.de> writes:

>> == Preventing false "pointers" ==
>> It's far too easy to push a gitrepo pointing to a submodule version that
>> is not existing on the server. Prevent this by checking for available
>> submodule versions before acceptingt the push.
>
> Yes, requiring to force such a push is an issue raised quite often (I
> think addressing this issue would be a good starting point for people
> who want to get involved in enhancing the submodule experience).

You need to be careful, though.

That check can only be sanely done at a hosting site that hosts _both_ the
superproject and the submodule repositories.  It might make more sense to
have this check on the side that pushes, which by definition should have
both superprojet and the submodule.  Fail, or prompt to confirm, a push
from the superproject when it is detected that the submodule commits bound
to the commits in the superproject have not been pushed out.

> (And, as every year, it's a good idea for a prospective student to get
> involved in the git community before his application is accepted ...
> sending some patches is a good way to do that, maybe regarding one of
> the first two issues raised here? ;-)

While I agree that it is a good idea, I think the advice is about a few
weeks too late for this year already.

Re: GSOC proposal

[Fredrik Gustafsson]

On Thu, Mar 24, 2011 at 04:47:41PM -0700, Junio C Hamano wrote:
> Jens Lehmann <Jens.Lehmann@web.de> writes:
> 
> >> == Preventing false "pointers" ==
> >> It's far too easy to push a gitrepo pointing to a submodule version that
> >> is not existing on the server. Prevent this by checking for available
> >> submodule versions before acceptingt the push.
> >
> > Yes, requiring to force such a push is an issue raised quite often (I
> > think addressing this issue would be a good starting point for people
> > who want to get involved in enhancing the submodule experience).
> 
> You need to be careful, though.
> 
> That check can only be sanely done at a hosting site that hosts _both_ the
> superproject and the submodule repositories.  It might make more sense to
> have this check on the side that pushes, which by definition should have
> both superprojet and the submodule.  Fail, or prompt to confirm, a push
> from the superproject when it is detected that the submodule commits bound
> to the commits in the superproject have not been pushed out.

I can see three different ways of doing this:
1. the reciever (server) checks for available submodules before
accepting a commit

2. the sender checks for available submodules on the server before
sending a commit.

3. each submodule contains a flag that tells if the last command was a
commit or a push. The core-repository wont allow a push if one (or more)
submodules has a commit as "last command".

Although alternative 1 is the only one that is certain of preventing the
problem with "false pointers", it has several other drawbacks. I believe
that alt. 3 is the proper way of handling this in git. Although we
doesn't guarantee a sane server-repo we does prevent the client from
doing stupid things by mistake.

-- 
Med vänliga hälsningar
Fredrik Gustafsson

tel: 0733-608274
e-post: iveqy@iveqy.com

Re: GSOC proposal

[Jens Lehmann]

Am 25.03.2011 11:06, schrieb Fredrik Gustafsson:
> On Thu, Mar 24, 2011 at 04:47:41PM -0700, Junio C Hamano wrote:
>> Jens Lehmann <Jens.Lehmann@web.de> writes:
>>
>>>> == Preventing false "pointers" ==
>>>> It's far too easy to push a gitrepo pointing to a submodule version that
>>>> is not existing on the server. Prevent this by checking for available
>>>> submodule versions before acceptingt the push.
>>>
>>> Yes, requiring to force such a push is an issue raised quite often (I
>>> think addressing this issue would be a good starting point for people
>>> who want to get involved in enhancing the submodule experience).
>>
>> You need to be careful, though.
>>
>> That check can only be sanely done at a hosting site that hosts _both_ the
>> superproject and the submodule repositories.  It might make more sense to
>> have this check on the side that pushes, which by definition should have
>> both superprojet and the submodule.  Fail, or prompt to confirm, a push
>> from the superproject when it is detected that the submodule commits bound
>> to the commits in the superproject have not been pushed out.
> 
> I can see three different ways of doing this:
> 1. the reciever (server) checks for available submodules before
> accepting a commit
> 
> 2. the sender checks for available submodules on the server before
> sending a commit.
> 
> 3. each submodule contains a flag that tells if the last command was a
> commit or a push. The core-repository wont allow a push if one (or more)
> submodules has a commit as "last command".
> 
> Although alternative 1 is the only one that is certain of preventing the
> problem with "false pointers", it has several other drawbacks. I believe
> that alt. 3 is the proper way of handling this in git. Although we
> doesn't guarantee a sane server-repo we does prevent the client from
> doing stupid things by mistake.

I concur with Junio and am in favor of the following solution, as I
think we already have all the information needed present on the side
that pushes:

Before doing a push in the superproject collect all submodule commits
that are recorded in the commits to be pushed out to the superproject's
remote. Then check that they all are contained in at least one remote
ref of the submodule they are recorded for. If not, error out and tell
the user he should push the submodule first or has to use the "-f" flag
to force the push if he really knows what he is doing. Commits of those
submodules that aren't checked out are ignored.

This approach would achieve the goal you stated in your last sentence,
which me thinks is a sane thing to do.

Re: GSOC proposal

[Fredrik Gustafsson]

[-- Attachment #1: Type: text/plain, Size: 1590 bytes --]

On Fri, Mar 25, 2011 at 12:27:20AM +0100, Jens Lehmann wrote:
> > == Threat every module alike ==
> > When failing fetching a submodule, continue fetching the next one
> > instead of dying. There's no need to prevent fetching a submodule
> > beginning at 'z' just because a failing in fetching a submodule
> > beginning at 'a'. The submodules should not be alphabetically dependant
> > as they are now.
> 
> I assume you are talking about the implicit fetch done by "git submodule
> update" here. The recursive submodule fetch that "git fetch" learned
> recently continues to fetch other submodules even if one or more fetches
> failed. But you are right that "git submodule update" should attempt to
> continue updating the remaining submodules too even if one of those
> updates failed along the way (This should be achieved with even less
> effort than the push issue mentioned above, so it would be an even
> easier starting point for people who want to get involved).
> 
> /* snip */
> 
> (And, as every year, it's a good idea for a prospective student to get
> involved in the git community before his application is accepted ...
> sending some patches is a good way to do that, maybe regarding one of
> the first two issues raised here? ;-)

I've attached a patch solving the submodule update-problem. A thing I 
though about is if it's a good idéa to end the output with an error 
summary. A git submodule update with 40-50 modules would easily let 
an error go by undetected with this patch.

-- 
Med vänliga hälsningar
Fredrik Gustafsson

tel: 0733-608274
e-post: iveqy@iveqy.com

[-- Attachment #2: 0001-When-one-submodule-fails-continue-to-the-next.patch --]
[-- Type: text/x-diff, Size: 3542 bytes --]

>From 1bc9a5e69f7b3296bad6dd0449054de8501be5bd Mon Sep 17 00:00:00 2001
From: iveqy <iveqy@iveqy.com>
Date: Fri, 25 Mar 2011 10:43:42 +0100
Subject: [PATCH] When one submodule fails, continue to the next

---
 git-submodule.sh |   60 ++++++++++++++++++++++++++++++++++++------------------
 1 files changed, 40 insertions(+), 20 deletions(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 3a13397..ae3abd1 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -342,8 +342,11 @@ cmd_init()
 		test -z "$url" || continue
 
 		url=$(git config -f .gitmodules submodule."$name".url)
-		test -z "$url" &&
-		die "No url found for submodule path '$path' in .gitmodules"
+		if test -z "$url" 
+		then
+			say "No url found for submodule path '$path' in .gitmodules"
+			continue
+		fi
 
 		# Possibly a url relative to parent
 		case "$url" in
@@ -352,13 +355,17 @@ cmd_init()
 			;;
 		esac
 
-		git config submodule."$name".url "$url" ||
-		die "Failed to register url for submodule path '$path'"
+		if !(git config submodule."$name".url "$url")
+		then
+			say "Failed to register url for submodule path '$path'"
+		fi
 
 		upd="$(git config -f .gitmodules submodule."$name".update)"
-		test -z "$upd" ||
-		git config submodule."$name".update "$upd" ||
-		die "Failed to register update mode for submodule path '$path'"
+		if !(test -z "$upd" || git config submodule."$name".update "$upd")
+		then
+			say "Failed to register update mode for submodule path '$path'"
+			continue
+		fi
 
 		say "Submodule '$name' ($url) registered for path '$path'"
 	done
@@ -446,9 +453,11 @@ cmd_update()
 			cloned_modules="$cloned_modules;$name"
 			subsha1=
 		else
-			subsha1=$(clear_local_git_env; cd "$path" &&
-				git rev-parse --verify HEAD) ||
-			die "Unable to find current revision in submodule path '$path'"
+			if !(subsha1=$(clear_local_git_env; cd "$path" && git rev-parse --verify HEAD))
+			then
+				say "Unable to find current revision in submodule path '$path'"
+				continue
+			fi
 		fi
 
 		if ! test -z "$update"
@@ -466,9 +475,11 @@ cmd_update()
 
 			if test -z "$nofetch"
 			then
-				(clear_local_git_env; cd "$path" &&
-					git-fetch) ||
-				die "Unable to fetch in submodule path '$path'"
+				if !(clear_local_git_env; cd "$path" && git-fetch)
+				then
+					say "Unable to fetch in submodule path '$path'"
+					continue
+				fi
 			fi
 
 			# Is this something we just cloned?
@@ -496,15 +507,21 @@ cmd_update()
 				;;
 			esac
 
-			(clear_local_git_env; cd "$path" && $command "$sha1") ||
-			die "Unable to $action '$sha1' in submodule path '$path'"
+			if !(clear_local_git_env; cd "$path" && $command "$sha1")
+			then
+				say "Unable to $action '$sha1' in submodule path '$path'"
+				continue;
+			fi
 			say "Submodule path '$path': $msg '$sha1'"
 		fi
 
 		if test -n "$recursive"
 		then
-			(clear_local_git_env; cd "$path" && eval cmd_update "$orig_flags") ||
-			die "Failed to recurse into submodule path '$path'"
+			if !(clear_local_git_env; cd "$path" && eval cmd_update "$orig_flags") 
+			then
+				say "Failed to recurse into submodule path '$path'"
+				continue;
+			fi
 		fi
 	done
 }
@@ -790,13 +807,16 @@ cmd_status()
 
 		if test -n "$recursive"
 		then
-			(
+			if ! (
 				prefix="$displaypath/"
 				clear_local_git_env
 				cd "$path" &&
 				eval cmd_status "$orig_args"
-			) ||
-			die "Failed to recurse into submodule path '$path'"
+			) 
+			then
+				say "Failed to recurse into submodule path '$path'"
+				continue
+			fi
 		fi
 	done
 }
-- 
1.7.4.1.433.gcd306.dirty