Re: [spf:guess,iffy] Re: "Integrated Web Client for git" GSoC proposal

Re: [spf:guess,iffy] Re: "Integrated Web Client for git" GSoC proposal

[Jakub Narebski]

On Thu, 15 April 2010, Sam Vilain wrote:
> On Thu, 2010-04-15 at 02:04 +0200, Jakub Narebski wrote:

> > gitweb was and is meant to be simple, easy to install git web interface
> > (single script + some static files), with minimal dependencies outside
> > Perl core, and running with as old Perl as feasible (good Unicode support
> > is requirement that forces minimal Perl version).

By the way, is the above statement correct?  What should be the *goal*
for gitweb?  Is it running on oldest Perl possible (which IIRC is Perl
5.8.6, because gitweb needs good Unicode support)?  Is it minimal 
required non-core dependencies (core as of Perl 5.8.6)?  Is it being
easy to install (single monolithic script + some minimal number of static
files like CSS, JavaScript, images + perhaps extra files with optional
features like output caching) by hand?  Is it being easy to configure
by hand?

With local::lib it is easy to install Perl modules from CPAN locally
(in home directory), so perhaps we could ease on minimal dependencies
rule.  On the other hand gitweb is web app, and one would have to be
able to configure web server to use locally installed Perl modules,
which might be not possible; this swicthes the problem from "not being
able to install Perl modules as non-root" to "not being able to install
Perl modules that web server can use without help from sysadmin".


One of requirements is backwards compatibility of gitweb links (URLs),
both query string based, and path_info URLs.  Yet another is ability
to *scan* for projects (this is IIRC requirement for git.kernel.org),
rather than using project index file (perhaps (re)generated 
(semi)automatically).

> > 
> > Gitalist requires Catalyst web framework, which while it is (one of)
> > modern Perl web frameworks is not exactly lightweight.
> 
> Ok, I just re-checked the dependencies and my assertion that a pure-perl
> Gitalist would be possible is indeed wrong.  It certainly is though with
> very lightweight frameworks like CGI::Application.  If you are splitting
> the script up, all the best to you, but please do consider something
> like that rather than re-inventing the wheel!

There exists very many web frameworks in Perl[1][2][3][4]: Catalyst
(one of more popular, used by Gitalist), Jifty, CGI::Application (and
its offshot Titanium), Mojolicus, Dancer, Squatting, Web::Simple,...

Nowadays many (all mentioned) of those web frameworks are either built
on top of PSGI / Plack (Perl Superglue for Web Frameworks and Web Servers),
or have PSGI / Plack adapter (see http://plackperl.org).  So another
solution would be to use bare-bones Plack instead of CGI with help
of CGI.pm, perhaps also using one of URI dispatchers[5][6].  Plack/PSGI
looks like the future of Perl web scripting... but is currently quite new,
at version 0.9930.

[1]: http://www.perlfoundation.org/perl5/index.cgi?web_frameworks
[2]: http://plackperl.org/ ("Frameworks" section)
[3]: http://perl.worldken.org/search?q=web+framework
[4]: http://search.cpan.org/search?query=web+framework&mode=all

[5]: http://www.perlfoundation.org/perl5/index.cgi?web_application_dispatchers
[6]: http://github.com/miyagawa/plack-dispatching-samples


As to splitting gitweb into smaller modules (like e.g. SVN::Web)...
Gitweb isn't the largest script in git sources: gitk is.  But splitting
gitk should be much easier - git-gui is already split.  Close after
gitweb is git-svn, which probably also need splitting; and again it
would be easier to split parts of git-svn e.g into Git::SVN module,
as git-svn already depends on Git.pm.

The trouble with splitting is installing split web script.  I have no
idea how to do this in cross-webserver way, cross-distribution and
cross-system way (the filesystem hierarchy used by web server may
differ from distribution to distribution, and from operating system
to operating system).

-- 
Jakub Narebski
Poland

Re: [spf:guess,iffy] Re: "Integrated Web Client for git" GSoC proposal

[Sam Vilain]

On Fri, 2010-04-16 at 11:18 +0200, Jakub Narebski wrote: 
> > > gitweb was and is meant to be simple, easy to install git web interface
> > > (single script + some static files), with minimal dependencies outside
> > > Perl core, and running with as old Perl as feasible (good Unicode support
> > > is requirement that forces minimal Perl version).
> 
> By the way, is the above statement correct?  What should be the *goal*
> for gitweb?  Is it running on oldest Perl possible (which IIRC is Perl
> 5.8.6, because gitweb needs good Unicode support)?  Is it minimal 
> required non-core dependencies (core as of Perl 5.8.6)?  Is it being
> easy to install (single monolithic script + some minimal number of static
> files like CSS, JavaScript, images + perhaps extra files with optional
> features like output caching) by hand?  Is it being easy to configure
> by hand?
> 
> With local::lib it is easy to install Perl modules from CPAN locally
> (in home directory), so perhaps we could ease on minimal dependencies
> rule.  On the other hand gitweb is web app, and one would have to be
> able to configure web server to use locally installed Perl modules,
> which might be not possible; this swicthes the problem from "not being
> able to install Perl modules as non-root" to "not being able to install
> Perl modules that web server can use without help from sysadmin".

This is correct for XS modules, using DynaLoader to link in
arbitrary .so's may be restricted in a web server :).  But for pure Perl
modules it should be no problem.  Many modules have both XS and
Pure-Perl versions, for instance Template Toolkit has both and so is
suitable for this style of bundling.

> There exists very many web frameworks in Perl[1][2][3][4]: Catalyst
> (one of more popular, used by Gitalist), Jifty, CGI::Application (and
> its offshot Titanium), Mojolicus, Dancer, Squatting, Web::Simple,...
> 
> Nowadays many (all mentioned) of those web frameworks are either built
> on top of PSGI / Plack (Perl Superglue for Web Frameworks and Web Servers),
> or have PSGI / Plack adapter (see http://plackperl.org).  So another
> solution would be to use bare-bones Plack instead of CGI with help
> of CGI.pm, perhaps also using one of URI dispatchers[5][6].  Plack/PSGI
> looks like the future of Perl web scripting... but is currently quite new,
> at version 0.9930.

Ok so PSGI is the port of Python's WSGI to Perl.  Plack is the reference
implementation, and also quite heavy at 2.5MB tarball.

Titanium is an extension to CGI::Application and requires DBI for
instance.  That's probably not right.

Jifty, Mojolicious, will also have prohibitively difficult dependencies
for the run-anywhere case.

Squatting has a few XS dependencies, but perhaps they could be excised if
required.  Dancer however seems to stand out at only 94kB tarball, minimal
non-core dependencies and support for PSGI.  The HTTP::Server::Simple::PSGI
dependency should let it support the 'instaweb' case with pure perl.

> The trouble with splitting is installing split web script.  I have no
> idea how to do this in cross-webserver way, cross-distribution and
> cross-system way (the filesystem hierarchy used by web server may
> differ from distribution to distribution, and from operating system
> to operating system).

It should be possible for the script to figure out what filesystem path it
is being run from, perhaps find a local lib/ dir and then add that to @INC.
In shell scripts you just use FindBin, I don't know whether that is still
expected to work from eg mod_perl but there's bound to be a solution for
that.  So yeah I'd say just aim to ship lots of .pm files in a nearby dir
alongside the script...

Sam

Re: [spf:guess,iffy] Re: "Integrated Web Client for git" GSoC proposal

[Tatsuhiko Miyagawa]

Sam Vilain <sam <at> vilain.net> writes:

> > of CGI.pm, perhaps also using one of URI dispatchers[5][6].  Plack/PSGI
> > looks like the future of Perl web scripting... but is currently quite new,
> > at version 0.9930.

I admit that it's quite new - not old anyway, but don't get fooled by its
version number. It's already adapted by almost all perl frameworks and
used by lots of companies on production.
 
> Ok so PSGI is the port of Python's WSGI to Perl.  Plack is the reference
> implementation, and also quite heavy at 2.5MB tarball.

No, the tarball just contains 2.4MB of .jpeg file for large-file download
and upload testing. Its  dependencies are kept minimal - only requires
LWP, URI and some pure perl modules.

> required.  Dancer however seems to stand out at only 94kB tarball, minimal
> non-core dependencies and support for PSGI.  The HTTP::Server::Simple::PSGI
> dependency should let it support the 'instaweb' case with pure perl.

You can already use CGI::Emulate::PSGI (via Plack::App::CGIBin) to run
gitweb.cgi on Plack, or whatever web server that supports PSGI - like
HTTP::Server::Simple::PSGI you just mentioned.
http://gist.github.com/346068

> It should be possible for the script to figure out what filesystem path it
> is being run from, perhaps find a local lib/ dir and then add that to @INC.
> In shell scripts you just use FindBin, I don't know whether that is still
> expected to work from eg mod_perl but there's bound to be a solution for
> that.  So yeah I'd say just aim to ship lots of .pm files in a nearby dir
> alongside the script...

Or use App::FatPacker.

--
Tatsuhiko Miyagawa

Fwd: Re: "Integrated Web Client for git" GSoC proposal

[Pavan Kumar Sunkara]

On Sat, Apr 17, 2010 at 11:57 AM, Tatsuhiko Miyagawa
<miyagawa@bulknews.net> wrote:
>
> Sam Vilain <sam <at> vilain.net> writes:
>
> > > of CGI.pm, perhaps also using one of URI dispatchers[5][6].  Plack/PSGI
> > > looks like the future of Perl web scripting... but is currently quite new,
> > > at version 0.9930.
>
> I admit that it's quite new - not old anyway, but don't get fooled by its
> version number. It's already adapted by almost all perl frameworks and
> used by lots of companies on production.
>
> > Ok so PSGI is the port of Python's WSGI to Perl.  Plack is the reference
> > implementation, and also quite heavy at 2.5MB tarball.
>
> No, the tarball just contains 2.4MB of .jpeg file for large-file download
> and upload testing. Its  dependencies are kept minimal - only requires
> LWP, URI and some pure perl modules.
>
> > required.  Dancer however seems to stand out at only 94kB tarball, minimal
> > non-core dependencies and support for PSGI.  The HTTP::Server::Simple::PSGI
> > dependency should let it support the 'instaweb' case with pure perl.
>
> You can already use CGI::Emulate::PSGI (via Plack::App::CGIBin) to run
> gitweb.cgi on Plack, or whatever web server that supports PSGI - like
> HTTP::Server::Simple::PSGI you just mentioned.
> http://gist.github.com/346068
>
> > It should be possible for the script to figure out what filesystem path it
> > is being run from, perhaps find a local lib/ dir and then add that to @INC.
> > In shell scripts you just use FindBin, I don't know whether that is still
> > expected to work from eg mod_perl but there's bound to be a solution for
> > that.  So yeah I'd say just aim to ship lots of .pm files in a nearby dir
> > alongside the script...
>
> Or use App::FatPacker.
>
> --
> Tatsuhiko Miyagawa
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

We need to keep gitweb dependancies as less as possible. So, using
frameworks will be a bad idea, as everybody might don't want to use
them.

Coming to splitting of gitweb, as Jakub said installing a split web
script would be a problem, but we can have a solution the problem by
using proper algorithms. First, we parse the URL and get to know which
function to be implemented and then we include the necessary file and
call the function which will then execute and return the proper HTML
output.

This way, we can no longer worry about installing the split webscript.
All we need is to install the basic gitweb.pl file which does all this
basic parsing and calling other perl files. The other files just
needed to be in the path.

-pavan

Re: Fwd: Re: "Integrated Web Client for git" GSoC proposal

[Jakub Narebski]

Pavan Kumar Sunkara <pavan.sss1991@gmail.com> writes:

> We need to keep gitweb dependancies as less as possible. So, using
> frameworks will be a bad idea, as everybody might don't want to use
> them.

Well, hopefully in the future PSGI / Plack would be as widely
available as CGI.pm is currently...

> 
> Coming to splitting of gitweb, as Jakub said installing a split web
> script would be a problem, but we can have a solution the problem by
> using proper algorithms. First, we parse the URL and get to know which
> function to be implemented and then we include the necessary file and
> call the function which will then execute and return the proper HTML
> output.
> 
> This way, we can no longer worry about installing the split webscript.
> All we need is to install the basic gitweb.pl file which does all this
> basic parsing and calling other perl files. The other files just
> needed to be in the path.

Not a solution, I think.  If you could munge PATH for web server
[user], you could munge PERL5LIB and use locally (local::lib)
installed packages.

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: [spf:guess,iffy] Re: "Integrated Web Client for git" GSoC proposal

[Jakub Narebski]

Restored some of Cc list...

Tatsuhiko Miyagawa <miyagawa@bulknews.net> writes:

> Sam Vilain <sam <at> vilain.net> writes:

> > It should be possible for the script to figure out what filesystem path it
> > is being run from, perhaps find a local lib/ dir and then add that to @INC.

FindBin, Dir::Self (or just relevant code from it), and then "use lib <dir>".

> > In shell scripts you just use FindBin, I don't know whether that is still
> > expected to work from eg mod_perl but there's bound to be a solution for
> > that.  So yeah I'd say just aim to ship lots of .pm files in a nearby dir
> > alongside the script...
> 
> Or use App::FatPacker.

But try to make this build-time dependency optional.  I wonder also
what is performance cost of using App::FatPacker trick.


It is a bit pity that PAR is not in core.  We could use 

  $ git archive --format=zip --output=gitweb.par --prefix=lib/ \
    HEAD -- gitweb/lib

to generate packfile, and then

  use PAR 'gitweb.par';

in gitweb.

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: [spf:guess,iffy] Re: "Integrated Web Client for git" GSoC proposal

[Jakub Narebski]

On Sat, 17 Apr 2010, Sam Vilain wrote:
> On Fri, 2010-04-16 at 11:18 +0200, Jakub Narebski wrote: 

> > With local::lib it is easy to install Perl modules from CPAN locally
> > (in home directory), so perhaps we could ease on minimal dependencies
> > rule.  On the other hand gitweb is web app, and one would have to be
> > able to configure web server to use locally installed Perl modules,
> > which might be not possible; this swicthes the problem from "not being
> > able to install Perl modules as non-root" to "not being able to install
> > Perl modules that web server can use without help from sysadmin".
> 
> This is correct for XS modules, using DynaLoader to link in
> arbitrary .so's may be restricted in a web server :).  But for pure Perl
> modules it should be no problem.  Many modules have both XS and
> Pure-Perl versions, for instance Template Toolkit has both and so is
> suitable for this style of bundling.

Errr... how the above addresses "not being able to install Perl modules
that web server can use without help from sysadmin", or in other words
"not being able to tell web server where to find locally installed /
/ additional Perl modules"?  If you can't set PERL5LIB for 'apache'
or 'web' or 'nobody' user, and you can't edit web server configuration
files...

Well, perhaps

  use lib (grep { -d } split /:/, "++GITWEBPERLLIB++");

and associated change to gitweb/Makefile would help here.

> > There exists very many web frameworks in Perl[1][2][3][4]: Catalyst
> > (one of more popular, used by Gitalist), Jifty, CGI::Application (and
> > its offshot Titanium), Mojolicus, Dancer, Squatting, Web::Simple,...
> > 
> > Nowadays many (all mentioned) of those web frameworks are either built
> > on top of PSGI / Plack (Perl Superglue for Web Frameworks and Web Servers),
> > or have PSGI / Plack adapter (see http://plackperl.org).  So another
> > solution would be to use bare-bones Plack instead of CGI with help
> > of CGI.pm, perhaps also using one of URI dispatchers[5][6].  Plack/PSGI
> > looks like the future of Perl web scripting... but is currently quite new,
> > at version 0.9930.
> 
> Ok so PSGI is the port of Python's WSGI to Perl.  Plack is the reference
> implementation, and also quite heavy at 2.5MB tarball.

Tatsuhiko Miyagawa addressed the issue of "2.5MB tarball".  PSGI is
specification, Plack is utilities (and includes reference implementation).

Using PSGI / Plack / PSGI compatibile web framework would give us ability
to run gitweb on many web servers: CGI, FastCGI, mod_perl (all those via
adapters from Plack), mod_psgi, HTTP::Server::PSGI (included in Plack),...
and give us Test::Plack.

But not less important would be ability to use Plack middleware; for
example gitweb output caching could be as simple as

  use Plack::Builder;
  ...

  builder {
     enable "Cache", ...
     $app;
  };

> 
> Titanium is an extension to CGI::Application and requires DBI for
> instance.  That's probably not right.
> 
> Jifty, Mojolicious, will also have prohibitively difficult dependencies
> for the run-anywhere case.
> 
> Squatting has a few XS dependencies, but perhaps they could be excised if
> required.  Dancer however seems to stand out at only 94kB tarball, minimal
> non-core dependencies and support for PSGI.  

Thanks for the research on those Perl web frameworks.

> The HTTP::Server::Simple::PSGI 
> dependency should let it support the 'instaweb' case with pure perl.

Or HTTP::Server::PSGI from Plack.

> > The trouble with splitting is installing split web script.  I have no
> > idea how to do this in cross-webserver way, cross-distribution and
> > cross-system way (the filesystem hierarchy used by web server may
> > differ from distribution to distribution, and from operating system
> > to operating system).
> 
> It should be possible for the script to figure out what filesystem path it
> is being run from, perhaps find a local lib/ dir and then add that to @INC.
> In shell scripts you just use FindBin, I don't know whether that is still
> expected to work from eg mod_perl but there's bound to be a solution for
> that.  So yeah I'd say just aim to ship lots of .pm files in a nearby dir
> alongside the script...

Errr... isn't FindBin considered harmful, and current best practice is
using Self::Dir, or just

        # __DIR__ is taken from Dir::Self __DIR__ fragment
        sub __DIR__ () {
                File::Spec->rel2abs(join '', (File::Spec->splitpath(__FILE__))[0, 1]);
        }
        use lib __DIR__ . '/lib';


And there is also App::FatPacker and PAR solution; pity that neither is
in core (well, for App::FatPacker it would be build time dependency only).

-- 
Jakub Narebski
Poland

Re: "Integrated Web Client for git" GSoC proposal

[Jakub Narebski]

Dnia środa 14. kwietnia 2010 15:50, Junio C Hamano napisał:
> Petr Baudis <pasky@ucw.cz> writes:
> 
> > Well, it's not that noone maintains gitweb or noone reviews patches;
> > smaller changes do get merged relatively fine (I'd hope), but larger
> > changes get sidetracked in endless discussions and perpetual polishing.
> >
> > I think the problem is that we are extremely perfectionistic about
> > whatever gets into gitweb ...
> 
> Hmph, from time to time, I found review comments that were too picky, but
> I don't think gitweb reviews are particularly "perfectionistic".
> 
> What I find is more problematic is the lack of post-review follow up,
> especially when the submitter disappears without responding to the
> reviews.  To patches that are in the core-git area, I find myself doing
> the "finishing touches" in such a case if/when the topic deserves it.
> What gitweb needs may a dedicated shepherd or two.
> 
> > ... but other than that I think gitweb just
> > needs more of the "best is the enemy of good" mentality.
> 
> If that means I'd see more "These are a resend of the good enough version
> of patches with our Acks; apply them" from you and/or Jakub, or "In this
> repository, we have queued the good ones---pull them", which is even
> better, I am all for it ;-).

Well, I try to either save interesting gitweb patches (which doesn't help
much), or import them to my repository at
  http://repo.or.cz/w/git/jnareb-git.git

The interesting branches would be gitwe/* branches.  Some of them are
regularly rebased against 'master' branch of main git repository.

Those branches are:

gitweb/web::

  The branch that I am doing miscelanous gitweb development on.  Managed
  using StGit, not all patches applied (visible).  Regularly rebased 
  against 'origin' branch.

  Currently contains syntax highlighting support from Johannes Schindelin
  (based in turn on Daniel Svensson's and Sham Chukoury's work in
  gitweb-xmms2.git), plus my refactoring of that.

  Needs decision on names of subroutines, and on semantics of 'highlight'
  feature (on/off, or highlighter name).

gitweb/cache-kernel-pu::

  Simple file-based output caching for gitweb, based on work by John 
  'Warthog9' Hawley (J.H.), split into smaller patches for easier review.
  Much more configurable than original patch(es) by J.H., with pluggable
  caching engine (it works for example with Cache::FileCache).

  J.H. ideas were: no serialization and mtime based expiration of simple
  file-based cache, using locking to avoid 'cache miss stampede' problem,
  serving stale data when cache entry is being regenerated, generating
  cache in background, trick for non-AJAXy progress info, having everything
  related to cache in 'cache.pm' file.

  My ideas were: tests, separating simple file-based cache into separate
  package and making it easy to replace this cache engine by any
  Cache::Cache or CHI cache driver, select(FILEHANLE) based capturing,
  separating capturing output into package to be able to replace it by
  other captyring mechanism, daemonizing for background cache generation
  (for mod_perl), being able to 'require' the 'cache.pm' file, gathering
  options in %cache_options and adding configurability, documentation.

  This is work in progress, and a bit of mess currently.  I have posted
  information about current state of this branch to git mailing list.

  Rebased on top of 'origin'.

gitweb/cache-kernel-v3::
gitweb/cache-kernel-v2::
gitweb/cache-kernel::

  Gitweb caching series, as send to git mailing list.  Not modified, here
  for historical reasons.

gitweb/category::

  The "nitpicked to death" series by Sebastien Cevey.

  I have meant to fix the way list of projects is displayed first, to
  separate filtering list of projects and printing list of projects,
  but never did managed to get to it.

  Rebased on top of 'origin'.  Series is stalled.

gitweb/committags::

  Series by Marcel M. Cary.

  I have meant to comment on this series, but forgot about it.  The major
  problem is deciding how to configure this feature... and how to name it.
  It is decision about API (configuration), so it is important to get it
  right the first time.

  Needs review.  Series is stalled.

gitweb/cache::

  Old caching (pre-GSoC2009 gitweb caching project even) of Perl data,
  for projects_list page only, by Petr Baudis.  Some version of it might
  be used on http://repo.or.cz

  Abandoned for now.  Needs checking which method of caching is better.

-- 
Jakub Narebski
Poland