[PATCHv5] Add Gitweb support for XZ compressed snapshots

[PATCHv5] Add Gitweb support for XZ compressed snapshots

[Mark A Rada]

 >Note that for me the above results are not aligned in table.
 >This is a cosmetic issue.

The table formatting issue was due to a bad habit of mixing tabs and  
spaces,
I decided to go with spaces this time :)


 >One thing that would concern me greatly, is not so much the CPU time  
(though that's a *huge* change in comparison to gz) but the memory  
usage.  Where gzip and bzip2 are chewing 4M and 13M respectively, xz  
chews 102M.  >From a 'beefy' server perspective chewing up that much  
memory per snapshot for that long could be bad.  This is likely  
something that needs to have some sort of enable/disable switch if  
it's going to be included.

True, and there are two solutions I can think of for this "problem".

	1. My tests were at the default compression level, the XZ documentation
	says that at lower levels you will get resource usage and compression
	ratios that are comparable to BZip2. However, I'm not sure where you
	would change the compression level variable for this (globally for the
	system, somewhere in $GITWEB_CONFIG, a git config variable). Does
	someone know the correct answer here?

	2. Implement snapshot caching for Gitweb.


--
Mark A Rada (ferrous26)
marada@uwaterloo.ca


From: Mark Rada <marada@uwaterloo.ca>
Date: Thu, 30 Jul 2009 08:56:42 -0400
Subject: [PATCH] Add Gitweb support for XZ compressed snapshots

The XZ compression format uses the LZMA2 compression algorithm, which
often yields higher compression ratios than both GZip and BZip2 at the
cost of using more CPU time and RAM. Though, while XZ is the slowest
for compression, it is much faster than BZip2 for decompression, almost
comparable to GZip (see benchmarks below).

You can enable XZ compressed snapshots by adding 'txz' to the list of
default options for snapshots in your $GITWEB_CONFIG.

I did some simple benchmarks, starting with an already tarballed
archive of the repos listed below. Memory usage seemed to be consistent
for any given algorithm at their default compression level. Timings were
gathered using the `time' command.

CPU: AMD Sempron 3400+ (1 core @ 1.8GHz with 256K L2 cache)
Virtual Memory Usage
	GZip: 4152K	BZip2: 13352K	XZ: 102M

Linux 2.6 series (f5886c7f96f2542382d3a983c5f13e03d7fc5259)   349M
gzip    23.70s user   0.47s system  99% cpu   24.227 total    76M
gunzip  3.74s user    0.74s system  94% cpu   4.741 total
bzip2   130.96s user  0.53s system  99% cpu   2:11.97 total   59M
bunzip2 31.05s user   1.02s system  99% cpu   32.355 total
xz      448.78s user  0.91s system  99% cpu   7:31.28 total   51M
unxz    7.67s user    0.80s system  98% cpu   8.607 total

Git (0a53e9ddeaddad63ad106860237bbf53411d11a7)                11M
gzip    0.77s user    0.03s system  99% cpu   0.792 total     2.5M
gunzip  0.12s user    0.02s system  98% cpu   0.142 total
bzip2   3.42s user    0.02s system  99% cpu   3.454 total     2.1M
bunzip2 0.95s user    0.03s system  99% cpu   0.984 total
xz      12.88s user   0.14s system  98% cpu   13.239 total    1.9M
unxz    0.27s user    0.03s system  99% cpu   0.298 total

XZ (669413bb2db954bbfde3c4542fddbbab53891eb4)                 1.8M
gzip    0.12s user    0.00s system  95% cpu   0.132 total     442K
gunzip  0.02s user    0.00s system  97% cpu   0.027 tota
bzip2   1.28s user    0.01s system  99% cpu   1.298 total     363K
bunzip2 0.15s user    0.01s system  100% cpu  0.157 total
xz      1.62s user    0.03s system  99% cpu   1.652 total     347K
unxz    0.05s user    0.00s system  99% cpu   0.058 total

Purely from a time and memory perspective, nothing compares to GZip in
each of the three tests. Though, if you have an average upload speed of
20KB/s, it would take ~400 seconds longer to transfer the kernel
snapshot that was BZip2 compressed than it would the XZ compressed
snapshot, the transfer time difference is even greater when compared to
the GZip compressed snapshot. The wall clock time savings are
relatively the same for all test cases, but less dramatic for the
smaller repositories.

The obvious downside for XZ compressed snapshots is the large CPU and
memory load put on the server to actualy generate the snapshot. Though
XZ will eventually have good threading support, and I suspect then that
the wall clock time for making an XZ compressed snapshot would go down
considerably if the server had a beefy multi-core CPU.

I have not enabled XZ compression by default because the current default
is GZip, and XZ is only really competitive with BZip2. Also, the XZ
format is still fairly new (the format was declared stable about 6
months ago), and there have been no "stable" releases of the utils yet.

Signed-off-by: Mark Rada <marada@uwaterloo.ca>
---
  gitweb/gitweb.perl |    8 ++++++++
  1 files changed, 8 insertions(+), 0 deletions(-)

diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 7fbd5ff..3398163 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -176,6 +176,13 @@ our %known_snapshot_formats = (
  		'format' => 'tar',
  		'compressor' => ['bzip2']},

+	'txz' => {
+		'display' => 'tar.xz',
+		'type' => 'application/x-xz',
+		'suffix' => '.tar.xz',
+		'format' => 'tar',
+		'compressor' => ['xz']},
+
  	'zip' => {
  		'display' => 'zip',
  		'type' => 'application/x-zip',
@@ -188,6 +195,7 @@ our %known_snapshot_formats = (
  our %known_snapshot_format_aliases = (
  	'gzip'  => 'tgz',
  	'bzip2' => 'tbz2',
+	'xz'    => 'txz',

  	# backward compatibility: legacy gitweb config support
  	'x-gzip' => undef, 'gz' => undef,
-- 
1.6.4

Re: [PATCHv5] Add Gitweb support for XZ compressed snapshots

[J.H.]

Mark A Rada wrote:
> 
>  >Note that for me the above results are not aligned in table.
>  >This is a cosmetic issue.
> 
> The table formatting issue was due to a bad habit of mixing tabs and 
> spaces,
> I decided to go with spaces this time :)
> 
> 
>  >One thing that would concern me greatly, is not so much the CPU time 
> (though that's a *huge* change in comparison to gz) but the memory 
> usage.  Where gzip and bzip2 are chewing 4M and 13M respectively, xz 
> chews 102M.  >From a 'beefy' server perspective chewing up that much 
> memory per snapshot for that long could be bad.  This is likely 
> something that needs to have some sort of enable/disable switch if it's 
> going to be included.
> 
> True, and there are two solutions I can think of for this "problem".
> 
>     1. My tests were at the default compression level, the XZ documentation
>     says that at lower levels you will get resource usage and compression
>     ratios that are comparable to BZip2. However, I'm not sure where you
>     would change the compression level variable for this (globally for the
>     system, somewhere in $GITWEB_CONFIG, a git config variable). Does
>     someone know the correct answer here?

Well you can always call xz with -[1-9] to change the compression level 
(same as gzip and bzip2) though I think a full disabling would be 'more' 
preferable, though I'm not sure I like Jakub's suggestion of just 
deleting it after the fact, it would work.

>     2. Implement snapshot caching for Gitweb.

I think it's slightly broken in my version (binary files don't work 
right apparently, it's on my todo list to fix in my upcoming update) but 
both Lea and I have done this long ago (caching layers in Gitweb), which 
would be an acceptable workaround for this, create once and serve many - 
though this has the downside of trading cpu for diskspace.  At least 
with xz there's less diskspace used ;-)

I think more my concern is more what's enabled by default, and since xz 
is still new (as was pointed out) it's probably worth only enabling if 
the admin selects it to be enabled.

- John 'Warthog9' Hawley


> -- 
> Mark A Rada (ferrous26)
> marada@uwaterloo.ca
> 
> 
> From: Mark Rada <marada@uwaterloo.ca>
> Date: Thu, 30 Jul 2009 08:56:42 -0400
> Subject: [PATCH] Add Gitweb support for XZ compressed snapshots
> 
> The XZ compression format uses the LZMA2 compression algorithm, which
> often yields higher compression ratios than both GZip and BZip2 at the
> cost of using more CPU time and RAM. Though, while XZ is the slowest
> for compression, it is much faster than BZip2 for decompression, almost
> comparable to GZip (see benchmarks below).
> 
> You can enable XZ compressed snapshots by adding 'txz' to the list of
> default options for snapshots in your $GITWEB_CONFIG.
> 
> I did some simple benchmarks, starting with an already tarballed
> archive of the repos listed below. Memory usage seemed to be consistent
> for any given algorithm at their default compression level. Timings were
> gathered using the `time' command.
> 
> CPU: AMD Sempron 3400+ (1 core @ 1.8GHz with 256K L2 cache)
> Virtual Memory Usage
>     GZip: 4152K    BZip2: 13352K    XZ: 102M
> 
> Linux 2.6 series (f5886c7f96f2542382d3a983c5f13e03d7fc5259)   349M
> gzip    23.70s user   0.47s system  99% cpu   24.227 total    76M
> gunzip  3.74s user    0.74s system  94% cpu   4.741 total
> bzip2   130.96s user  0.53s system  99% cpu   2:11.97 total   59M
> bunzip2 31.05s user   1.02s system  99% cpu   32.355 total
> xz      448.78s user  0.91s system  99% cpu   7:31.28 total   51M
> unxz    7.67s user    0.80s system  98% cpu   8.607 total
> 
> Git (0a53e9ddeaddad63ad106860237bbf53411d11a7)                11M
> gzip    0.77s user    0.03s system  99% cpu   0.792 total     2.5M
> gunzip  0.12s user    0.02s system  98% cpu   0.142 total
> bzip2   3.42s user    0.02s system  99% cpu   3.454 total     2.1M
> bunzip2 0.95s user    0.03s system  99% cpu   0.984 total
> xz      12.88s user   0.14s system  98% cpu   13.239 total    1.9M
> unxz    0.27s user    0.03s system  99% cpu   0.298 total
> 
> XZ (669413bb2db954bbfde3c4542fddbbab53891eb4)                 1.8M
> gzip    0.12s user    0.00s system  95% cpu   0.132 total     442K
> gunzip  0.02s user    0.00s system  97% cpu   0.027 tota
> bzip2   1.28s user    0.01s system  99% cpu   1.298 total     363K
> bunzip2 0.15s user    0.01s system  100% cpu  0.157 total
> xz      1.62s user    0.03s system  99% cpu   1.652 total     347K
> unxz    0.05s user    0.00s system  99% cpu   0.058 total
> 
> Purely from a time and memory perspective, nothing compares to GZip in
> each of the three tests. Though, if you have an average upload speed of
> 20KB/s, it would take ~400 seconds longer to transfer the kernel
> snapshot that was BZip2 compressed than it would the XZ compressed
> snapshot, the transfer time difference is even greater when compared to
> the GZip compressed snapshot. The wall clock time savings are
> relatively the same for all test cases, but less dramatic for the
> smaller repositories.
> 
> The obvious downside for XZ compressed snapshots is the large CPU and
> memory load put on the server to actualy generate the snapshot. Though
> XZ will eventually have good threading support, and I suspect then that
> the wall clock time for making an XZ compressed snapshot would go down
> considerably if the server had a beefy multi-core CPU.
> 
> I have not enabled XZ compression by default because the current default
> is GZip, and XZ is only really competitive with BZip2. Also, the XZ
> format is still fairly new (the format was declared stable about 6
> months ago), and there have been no "stable" releases of the utils yet.
> 
> Signed-off-by: Mark Rada <marada@uwaterloo.ca>
> ---
>  gitweb/gitweb.perl |    8 ++++++++
>  1 files changed, 8 insertions(+), 0 deletions(-)
> 
> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
> index 7fbd5ff..3398163 100755
> --- a/gitweb/gitweb.perl
> +++ b/gitweb/gitweb.perl
> @@ -176,6 +176,13 @@ our %known_snapshot_formats = (
>          'format' => 'tar',
>          'compressor' => ['bzip2']},
> 
> +    'txz' => {
> +        'display' => 'tar.xz',
> +        'type' => 'application/x-xz',
> +        'suffix' => '.tar.xz',
> +        'format' => 'tar',
> +        'compressor' => ['xz']},
> +
>      'zip' => {
>          'display' => 'zip',
>          'type' => 'application/x-zip',
> @@ -188,6 +195,7 @@ our %known_snapshot_formats = (
>  our %known_snapshot_format_aliases = (
>      'gzip'  => 'tgz',
>      'bzip2' => 'tbz2',
> +    'xz'    => 'txz',
> 
>      # backward compatibility: legacy gitweb config support
>      'x-gzip' => undef, 'gz' => undef,

Re: [PATCHv5] Add Gitweb support for XZ compressed snapshots

[demerphq]

2009/8/1 J.H. <warthog19@eaglescrag.net>:
> Mark A Rada wrote:
>>
>>  >Note that for me the above results are not aligned in table.
>>  >This is a cosmetic issue.
>>
>> The table formatting issue was due to a bad habit of mixing tabs and
>> spaces,
>> I decided to go with spaces this time :)
>>
>>
>>  >One thing that would concern me greatly, is not so much the CPU time
>> (though that's a *huge* change in comparison to gz) but the memory usage.
>>  Where gzip and bzip2 are chewing 4M and 13M respectively, xz chews 102M.
>>  >From a 'beefy' server perspective chewing up that much memory per snapshot
>> for that long could be bad.  This is likely something that needs to have
>> some sort of enable/disable switch if it's going to be included.
>>
>> True, and there are two solutions I can think of for this "problem".
>>
>>    1. My tests were at the default compression level, the XZ documentation
>>    says that at lower levels you will get resource usage and compression
>>    ratios that are comparable to BZip2. However, I'm not sure where you
>>    would change the compression level variable for this (globally for the
>>    system, somewhere in $GITWEB_CONFIG, a git config variable). Does
>>    someone know the correct answer here?
>
> Well you can always call xz with -[1-9] to change the compression level
> (same as gzip and bzip2) though I think a full disabling would be 'more'
> preferable, though I'm not sure I like Jakub's suggestion of just deleting
> it after the fact, it would work.
>
>>    2. Implement snapshot caching for Gitweb.
>
> I think it's slightly broken in my version (binary files don't work right
> apparently, it's on my todo list to fix in my upcoming update) but both Lea
> and I have done this long ago (caching layers in Gitweb), which would be an
> acceptable workaround for this, create once and serve many - though this has
> the downside of trading cpu for diskspace.  At least with xz there's less
> diskspace used ;-)
>
> I think more my concern is more what's enabled by default, and since xz is
> still new (as was pointed out) it's probably worth only enabling if the
> admin selects it to be enabled.

FWIW the perl project ripped out all the snapshot generation logic
from gitweb, and replaced it with a tool that generates snapshots
correctly for our requirements (if the build process needs additional
files /currently/ git-archive does not support adding them), this
includes a disk level cache for the snapshots since creating the tar,
adding the additional files, then gziping is quite slow.

If its interesting to people I can post it and the other changes here,
although its not a "nice" change, as I literally ripped out the
existing code.

cheers,
Yves





-- 
perl -Mre=debug -e "/just|another|perl|hacker/"

Re: [PATCHv5] Add Gitweb support for XZ compressed snapshots

[Jakub Narebski]

"J.H." <warthog19@eaglescrag.net> writes:

> Well you can always call xz with -[1-9] to change the compression
> level (same as gzip and bzip2) though I think a full disabling would
> be 'more' preferable, though I'm not sure I like Jakub's suggestion of
> just deleting it after the fact, it would work.
[...]

The problem is that 'keys %known_snapshot_formats' serves also as list
of allowed snapshot formats, if project specific override is enabled.
We can add another optional flag ('disabled' => 1) if you don't want
to delete from %known_snapshot_formats in $GITWEB_CONFIG, though I
don't know if it is worth it.  Anyway such mechanism can be added, and
IMHO should be added, in a separate commit.

> I think more my concern is more what's enabled by default, and since
> xz is still new (as was pointed out) it's probably worth only enabling
> if the admin selects it to be enabled.

By default (i.e. as in gitweb in git.git) 'snapshot' feature has
disabled projects specific override.  Which means only 'tgz' snapshot
is enabled / used.

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: [PATCHv5] Add Gitweb support for XZ compressed snapshots

[Jakub Narebski]

demerphq <demerphq@gmail.com> writes:

> FWIW the perl project ripped out all the snapshot generation logic
> from gitweb, and replaced it with a tool that generates snapshots
> correctly for our requirements (if the build process needs additional
> files /currently/ git-archive does not support adding them), this
> includes a disk level cache for the snapshots since creating the tar,
> adding the additional files, then gziping is quite slow.
> 
> If its interesting to people I can post it and the other changes here,
> although its not a "nice" change, as I literally ripped out the
> existing code.

Do you mean gitweb serving http://perl5.git.perl.org/ ?

Well, at least publish this version of gitweb somewhere (snapshot or
better git repository), and add it to the list of gitweb forks at
http://git.or.cz/gitwiki/Gitweb wiki page.

Sidenote: snapshot support appeared, as far as I know, first at fork of
gitweb that had been used by XMMS2, but the snapshot was generated by
a separate snapshot.cgi script... written in Python.  (Now they use
stock (if old) gitweb, I think.)

P.S. Perhaps you or other Perl Mongers would be interested in creating
yet another git web interface, in Perl, but contrary to (current)
gitweb modular and using more of CPAN modules?  For example convert
SVN::Web or Insurrection (or other Perl module) from Subversion (or
other SCM) to Git (and rename it to Git::Web, or ???).

P.P.S. Could you per chance post announcement of Git User's Survey 2009
on http://perl5.git.perl.org, just like it is done on http://repo.or.cz
and http://git.kernel.org?  TIA.

-- 
Jakub Narebski
ShadeHawk on #git

Git User's Survey 2009
http://tinyurl.com/GitSurvey2009

Re: [PATCHv5] Add Gitweb support for XZ compressed snapshots

[demerphq]

2009/8/1 Jakub Narebski <jnareb@gmail.com>:
> demerphq <demerphq@gmail.com> writes:
>
>> FWIW the perl project ripped out all the snapshot generation logic
>> from gitweb, and replaced it with a tool that generates snapshots
>> correctly for our requirements (if the build process needs additional
>> files /currently/ git-archive does not support adding them), this
>> includes a disk level cache for the snapshots since creating the tar,
>> adding the additional files, then gziping is quite slow.
>>
>> If its interesting to people I can post it and the other changes here,
>> although its not a "nice" change, as I literally ripped out the
>> existing code.
>
> Do you mean gitweb serving http://perl5.git.perl.org/ ?

Yes. I updated it to a relatively recent version just the other week.

> Well, at least publish this version of gitweb somewhere (snapshot or
> better git repository), and add it to the list of gitweb forks at
> http://git.or.cz/gitwiki/Gitweb wiki page.

Sure, we have been discussing doing that recently.

I need to cleanup how I set up the repository for it tho.

Also probably there are one or two patches that should be pushed back to you.

> Sidenote: snapshot support appeared, as far as I know, first at fork of
> gitweb that had been used by XMMS2, but the snapshot was generated by
> a separate snapshot.cgi script... written in Python.  (Now they use
> stock (if old) gitweb, I think.)

Ah, I'm not generating it with a separate cgi script, I just replaced
the call to git-archive to be a call to my own tool.

> P.S. Perhaps you or other Perl Mongers would be interested in creating
> yet another git web interface, in Perl, but contrary to (current)
> gitweb modular and using more of CPAN modules?  For example convert
> SVN::Web or Insurrection (or other Perl module) from Subversion (or
> other SCM) to Git (and rename it to Git::Web, or ???).

I think that there is some interest in doing that and likely over time
there will be more. It has come up at least a few times in our
discussion forums. I think you could easily make a public call for
support on places like Perlmonks to get more action tho.

A better git api toolset for perl would make things a lot easier.
Especially an XS one.

>
> P.P.S. Could you per chance post announcement of Git User's Survey 2009
> on http://perl5.git.perl.org, just like it is done on http://repo.or.cz
> and http://git.kernel.org?  TIA.

Sure.

cheers,
Yves
-- 
perl -Mre=debug -e "/just|another|perl|hacker/"

Re: [PATCHv5] Add Gitweb support for XZ compressed snapshots

[Jakub Narebski]

On Sat, 1 Aug 2009, demerphq wrote:
> 2009/8/1 Jakub Narebski <jnareb@gmail.com>:
>> demerphq <demerphq@gmail.com> writes:
>>
>>> FWIW the perl project ripped out all the snapshot generation logic
>>> from gitweb, and replaced it with a tool that generates snapshots
>>> correctly for our requirements (if the build process needs additional
>>> files /currently/ git-archive does not support adding them), this
>>> includes a disk level cache for the snapshots since creating the tar,
>>> adding the additional files, then gziping is quite slow.
>>>
>>> If its interesting to people I can post it and the other changes here,
>>> although its not a "nice" change, as I literally ripped out the
>>> existing code.
>>
>> Do you mean gitweb serving http://perl5.git.perl.org/ ?
> 
> Yes. I updated it to a relatively recent version just the other week.

By the way, if you don't mind, how do you keep your changes on top
of changing target, i.e. on top of updating baseline to recent 
versions?

> 
>> Well, at least publish this version of gitweb somewhere (snapshot or
>> better git repository), and add it to the list of gitweb forks at
>> http://git.or.cz/gitwiki/Gitweb wiki page.
> 
> Sure, we have been discussing doing that recently.
> 
> I need to cleanup how I set up the repository for it tho.

O.K.

> 
> Also probably there are one or two patches that should be pushed back
> to you. 

What do those patches consist of?
 
>> Sidenote: snapshot support appeared, as far as I know, first at fork of
>> gitweb that had been used by XMMS2, but the snapshot was generated by
>> a separate snapshot.cgi script... written in Python.  (Now they use
>> stock (if old) gitweb, I think.)
> 
> Ah, I'm not generating it with a separate cgi script, I just replaced
> the call to git-archive to be a call to my own tool.

O.K.  

By the way, this is one of the only two places where we have to
use quote_command and 3-argument form of open, instead of list for of
magic open.  It is because of pipeline, piping git-archive output
into compressor.  The other such place is git_object, to redirect error
stream to /dev/null (to discard stderr).

I thought about replacing it by list form of open somewhat (you can
find it in git mailing list archive), but it is not easy.  And IPC::Run
is IMVHO a bit of overkill, especially for "minimal dependencies"
gitweb (perhaps for Git::Web?).

> 
>> P.S. Perhaps you or other Perl Mongers would be interested in creating
>> yet another git web interface, in Perl, but contrary to (current)
>> gitweb modular and using more of CPAN modules?  For example convert
>> SVN::Web or Insurrection (or other Perl module) from Subversion (or
>> other SCM) to Git (and rename it to Git::Web, or ???).
> 
> I think that there is some interest in doing that and likely over time
> there will be more. It has come up at least a few times in our
> discussion forums. I think you could easily make a public call for
> support on places like Perlmonks to get more action tho.

Thanks in advance.

> 
> A better git api toolset for perl would make things a lot easier.
> Especially an XS one.

Well, in the beginnings of Git.pm there was XS interface (to 
git-hash-object or git-cat-file), but it was dropped because it
used -fPIC, which is not portable enough for Git (this is not present
in perl/Git.pm history in git.git repository).  Git.pm was created
by Petr 'Pasky' Baudis, author of Cogito porcelain, and creator and
admin of repo.or.cz (and also creator of first Git homepage).  
But he is not a Perl hacker (correct me if I am wrong, Pasky); both
Cogito and repo.or.cz duct tape (under name of Girocco) are written
in bash, not in Perl.  Which can be seen for example by using 
Error::Simple in Git.pm...

Also I don't think that Pasky has time for maintaining Git.pm, 
nevermind modernizing it / adding new features.

There was another attempt to make Perl interface for Git, more 
object-oriented, by Lea Wiemann during GSoC 2008 project 'gitweb caching'.
You can see results in Lea repository at repo.or.cz[1], and also in 
git.kernel.org gitweb code[2].

[1]: http://repo.or.cz/w/git/gitweb-caching.git
[2]: http://git.kernel.org/?p=git/warthog9/gitweb.git
 
>>
>> P.P.S. Could you per chance post announcement of Git User's Survey 2009
>> on http://perl5.git.perl.org, just like it is done on http://repo.or.cz
>> and http://git.kernel.org?  TIA.
> 
> Sure.

Hmmm... I don't see "Git User's Survey 2009" announcement at 
http://perl5.git.perl.org/.  Neither at Planet Perl Iron Man, nor at
Perlsphere

-- 
Jakub Narebski

Git User's Survey 2009:
http://tinyurl.com/GitSurvey2009

Re: [PATCHv5] Add Gitweb support for XZ compressed snapshots

[demerphq]

2009/8/3 Jakub Narebski <jnareb@gmail.com>:
> On Sat, 1 Aug 2009, demerphq wrote:
>> 2009/8/1 Jakub Narebski <jnareb@gmail.com>:
>>> demerphq <demerphq@gmail.com> writes:
>>>
>>>> FWIW the perl project ripped out all the snapshot generation logic
>>>> from gitweb, and replaced it with a tool that generates snapshots
>>>> correctly for our requirements (if the build process needs additional
>>>> files /currently/ git-archive does not support adding them), this
>>>> includes a disk level cache for the snapshots since creating the tar,
>>>> adding the additional files, then gziping is quite slow.
>>>>
>>>> If its interesting to people I can post it and the other changes here,
>>>> although its not a "nice" change, as I literally ripped out the
>>>> existing code.
>>>
>>> Do you mean gitweb serving http://perl5.git.perl.org/ ?
>>
>> Yes. I updated it to a relatively recent version just the other week.
>
> By the way, if you don't mind, how do you keep your changes on top
> of changing target, i.e. on top of updating baseline to recent
> versions?

Things broke. I fixed them. :-)


>> Also probably there are one or two patches that should be pushed back
>> to you.
>
> What do those patches consist of?

One has to do with layout problems in the blame, and I think one deals
with broken line number anchor links.

Ill look closer when i have more time.

> By the way, this is one of the only two places where we have to
> use quote_command and 3-argument form of open, instead of list for of
> magic open.  It is because of pipeline, piping git-archive output
> into compressor.  The other such place is git_object, to redirect error
> stream to /dev/null (to discard stderr).
>
> I thought about replacing it by list form of open somewhat (you can
> find it in git mailing list archive), but it is not easy.  And IPC::Run
> is IMVHO a bit of overkill, especially for "minimal dependencies"
> gitweb (perhaps for Git::Web?).

Lately I have been playing with Capture::Tiny, which seems to fill
some nice niches in this respect.

>> A better git api toolset for perl would make things a lot easier.
>> Especially an XS one.
>
> Well, in the beginnings of Git.pm there was XS interface (to
> git-hash-object or git-cat-file), but it was dropped because it
> used -fPIC, which is not portable enough for Git (this is not present
> in perl/Git.pm history in git.git repository).  Git.pm was created
> by Petr 'Pasky' Baudis, author of Cogito porcelain, and creator and
> admin of repo.or.cz (and also creator of first Git homepage).
> But he is not a Perl hacker (correct me if I am wrong, Pasky); both
> Cogito and repo.or.cz duct tape (under name of Girocco) are written
> in bash, not in Perl.  Which can be seen for example by using
> Error::Simple in Git.pm...
>
> Also I don't think that Pasky has time for maintaining Git.pm,
> nevermind modernizing it / adding new features.
>
> There was another attempt to make Perl interface for Git, more
> object-oriented, by Lea Wiemann during GSoC 2008 project 'gitweb caching'.
> You can see results in Lea repository at repo.or.cz[1], and also in
> git.kernel.org gitweb code[2].
>
> [1]: http://repo.or.cz/w/git/gitweb-caching.git
> [2]: http://git.kernel.org/?p=git/warthog9/gitweb.git

Yeah I recall that being discussed. I see that Git.pm has recevied
some attention since the last time I looked. Which is good to see.

Having lurked on the list I understand that there is some kind of
effort for a git library.  Perhaps at some point that will end up with
an XS interface.

>>>
>>> P.P.S. Could you per chance post announcement of Git User's Survey 2009
>>> on http://perl5.git.perl.org, just like it is done on http://repo.or.cz
>>> and http://git.kernel.org?  TIA.
>>
>> Sure.
>
> Hmmm... I don't see "Git User's Survey 2009" announcement at
> http://perl5.git.perl.org/.  Neither at Planet Perl Iron Man, nor at
> Perlsphere

Ah, sorry, I posted it to the perl5-porters mailing list and asked the
relevant people to forward the request on or post it on their site. I
know at least one perl committer already did it. However tomorrow the
main european perl conference starts (YAPC::EU in Lisbon) and I think
a big chunk of the "active" Perl community will be there, so probably
there wont be much action from them until they get back.

To be honest I'm not familiar with Planet Perl Iron Man nor
Perlsphere. Ill have to check them out i guess. :-)

I will try to remember to put up a note on the website as soon as possible.

cheers,
Yves

-- 
perl -Mre=debug -e "/just|another|perl|hacker/"

Re: [PATCHv5] Add Gitweb support for XZ compressed snapshots

[Jakub Narebski]

On Mon, 3 August 2009, demerphq wrote:
> 2009/8/3 Jakub Narebski <jnareb@gmail.com>:
>> On Sat, 1 Aug 2009, demerphq wrote:
>>> 2009/8/1 Jakub Narebski <jnareb@gmail.com>:

>>>> Do you mean gitweb serving http://perl5.git.perl.org/ ?
>>>
>>> Yes. I updated it to a relatively recent version just the other week.
>>
>> By the way, if you don't mind, how do you keep your changes on top
>> of changing target, i.e. on top of updating baseline to recent
>> versions?
> 
> Things broke. I fixed them. :-)

Err... I meant here to ask if you use merging into your branch, rebasing
your branch on top of upstream, using some patch management interface like
StGIT, Guilt ot Quilt, or topic branch management tool namely TopGit?
 
>>> Also probably there are one or two patches that should be pushed back
>>> to you.
>>
>> What do those patches consist of?
> 
> One has to do with layout problems in the blame, and I think one deals
> with broken line number anchor links.
> 
> Ill look closer when i have more time.

About the one with broken 'linenr' links: you should look at patch which
makes gitweb use information in "previous" header from git-blame -p, 
which was send to git mailing list and is currently in 'pu':
  [PATCHv2 03/10] gitweb: Use "previous" header of git-blame -p in 'blame' view
  Message-Id: <1248475450-5668-4-git-send-email-jnareb@gmail.com>
  http://article.gmane.org/gmane.comp.version-control.git/123961


What layout problems in 'blame' view did you notice? 

>> By the way, this is one of the only two places where we have to
>> use quote_command and 3-argument form of open, instead of list for of
>> magic open.  It is because of pipeline, piping git-archive output
>> into compressor.  The other such place is git_object, to redirect error
>> stream to /dev/null (to discard stderr).
>>
>> I thought about replacing it by list form of open somewhat (you can
>> find it in git mailing list archive), but it is not easy.  And IPC::Run
>> is IMVHO a bit of overkill, especially for "minimal dependencies"
>> gitweb (perhaps for Git::Web?).
> 
> Lately I have been playing with Capture::Tiny, which seems to fill
> some nice niches in this respect.

I don't quite see how you plan to use Capture::Tiny here.  The output
of git-archive has to be passed to compressor, and then dumped to stdout,
not captured.

-- 
Jakub Narebski
Poland

Re: Working on gitweb (was: [PATCHv5] Add Gitweb support for XZ compressed snapshots)

[Jakub Narebski]

Dnia sobota 1. sierpnia 2009 13:13, demerphq napisał:

> One thing i notice is that most/all of the existing repos are forks of git.
> 
> And the changes are to gitweb.perl, persumably on the assumption that
> people upgrade using the install process.
> 
> That doesn't make it easy to track changes that are made to the
> production version ("installed" version), unless im missing something.
> 
> How does one set up gitweb so that one can hack its sources and track
> them in a sane way when you are not using make install-gitweb
> (whatever) all the time?

There are at least two possible options. First (that is the one I use)
is to create script to update gitweb, which would run "make gitweb/gitweb.cgi",
with appropriate option, then copy files, like gitweb-update.sh script below.

Second option is to do like t/t9500* gitweb test, which means providing
config file by using GITWEB_CONFIG environment variable, and set all 
required options/variables like $GIT via config file instead of via 
build time configuration.  See also gitweb-run.sh script below.

-- [gitweb-update.sh] --
#!/bin/bash

#BINDIR="/usr/bin"
BINDIR="/home/local/git"

function make_gitweb()
{
	pushd "/home/jnareb/git/"

	make GITWEB_PROJECTROOT="/home/local/scm" \
	     GITWEB_CSS="/gitweb/gitweb.css" \
	     GITWEB_LOGO="/gitweb/git-logo.png" \
	     GITWEB_FAVICON="/gitweb/git-favicon.png" \
	     GITWEB_BLAMEJS="/gitweb/blame.js" \
	     GITWEB_HOMETEXT="/home/local/scm/indextext.html" \
	     bindir=$BINDIR \
	     gitweb/gitweb.cgi

	popd
}

function copy_gitweb()
{
	cp -fv ~/git/gitweb/gitweb.{cgi,css} /home/local/gitweb/
}

make_gitweb
copy_gitweb

# end of gitweb-update.sh

-- [gitweb-run.sh] --
#!/bin/bash

export GATEWAY_INTERFACE="CGI/1.1"
export HTTP_ACCEPT="*/*"
export REQUEST_METHOD="GET"
export QUERY_STRING=""$1""
export PATH_INFO=""$2""

export GITWEB_CONFIG="~/git/gitweb/gitweb_config.perl"

perl -- ~/git/gitweb/gitweb.perl

-- [gitweb_config.perl] --
#!/usr/bin/perl

# gitweb configuration

our $version = "current";
#our $GIT = "/usr/bin/git";
our $GIT = "/home/local/git/git";
our $projectroot = "/home/local/scm";
our $home_link_str = "projects";
our $site_name = "[localhost]";
our $site_header = "";
our $site_footer = "";
our $home_text = "indextext.html";
our @stylesheets = ("file:///home/user/git/gitweb/gitweb.css");
our $logo = "file:///home/user/git/gitweb/git-logo.png";
our $favicon = "file:///home/user/git/gitweb/git-favicon.png";
our $blamejs = "file:///home/user/git/gitweb/blame.js";
our $projects_list = "";
our $export_ok = "";
our $strict_export = "";
our $project_maxdepth = 2007;
our @git_base_url_list = ("/home/user/git");

$feature{'blame'}{'default'} = [1];
$feature{'grep'}{'default'} = [1];
$feature{'pickaxe'}{'default'} = [1];

# end of gitweb_config.perl
-- 
Jakub Narebski
Poland