Add Gitweb support for LZMA compressed snapshots

Add Gitweb support for LZMA compressed snapshots

[Mark A Rada]

[-- Attachment #1: Type: text/plain, Size: 326 bytes --]

Hi,

I thought I would submit this little patch I made to my gitweb. I am  
on a relatively slow
connection, and so LZMA compression time is less of a  concern than  
bandwidth---I'm
guessing that I am not the only person who suffers from slow internet  
connection
syndrome.


--
Mark A Rada (ferrous26)
marada@uwaterloo.ca


[-- Attachment #2: 0001-Add-Gitweb-support-for-LZMA-compressed-snapshots.patch --]
[-- Type: application/octet-stream, Size: 1115 bytes --]

From a080ce5e8467317a3f20c6de8814549e5c4f46f2 Mon Sep 17 00:00:00 2001
From: Mark Rada <marada@uwaterloo.ca>
Date: Wed, 29 Jul 2009 21:27:32 -0400
Subject: [PATCH] Add Gitweb support for LZMA compressed snapshots

Signed-off-by: Mark Rada <marada@uwaterloo.ca>
---
 gitweb/gitweb.perl |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)

diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 38492d0..e131ea4 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -184,6 +184,13 @@ our %known_snapshot_formats = (
 		'format' => 'tar',
 		'compressor' => ['bzip2']},
 
+	'txz' => {
+		'display' => 'tar.lzma',
+		'type' => 'application/x-lzma',
+		'suffix' => '.lzma',
+		'format' => 'tar',
+		'compressor' => ['lzma']},
+
 	'zip' => {
 		'display' => 'zip',
 		'type' => 'application/x-zip',
@@ -196,6 +203,7 @@ our %known_snapshot_formats = (
 our %known_snapshot_format_aliases = (
 	'gzip'  => 'tgz',
 	'bzip2' => 'tbz2',
+	'lzma'  => 'txz',
 
 	# backward compatibility: legacy gitweb config support
 	'x-gzip' => undef, 'gz' => undef,
-- 
1.6.4


[-- Attachment #3: Type: text/plain, Size: 3 bytes --]

Re: Add Gitweb support for LZMA compressed snapshots

[J.H.]

Mark A Rada wrote:
> Hi,
> 
> I thought I would submit this little patch I made to my gitweb. I am on 
> a relatively slow
> connection, and so LZMA compression time is less of a  concern than 
> bandwidth---I'm
> guessing that I am not the only person who suffers from slow internet 
> connection
> syndrome.
> 
> 
> -- 
> Mark A Rada (ferrous26)
> marada@uwaterloo.ca

Don't use 'lzma' the command, use 'xz' ( http://tukaani.org/xz/ ) as it 
uses the lzma2 format which is, by far, preferable to what 'lzma' 
outputs.  Same compression (lzma) just different file format.

- John 'Warthog9' Hawley

Re: Add Gitweb support for LZMA compressed snapshots

[Alex Riesen]

On Thu, Jul 30, 2009 at 09:44, J.H.<warthog19@eaglescrag.net> wrote:
> Don't use 'lzma' the command, use 'xz' ( http://tukaani.org/xz/ ) as it uses

The host seems to be down.

> the lzma2 format which is, by far, preferable to what 'lzma' outputs.  Same
> compression (lzma) just different file format.

And the whole project seems to be very young (listen on ohloh since 2007).
And it is not in current stable distributions. Exotic thing.

Re: Add Gitweb support for LZMA compressed snapshots

[Dmitry Potapov]

On Sat, Aug 01, 2009 at 09:43:39AM +0200, Alex Riesen wrote:
> 
> And the whole project seems to be very young (listen on ohloh since 2007).
> And it is not in current stable distributions. Exotic thing.

The project seems to be it is developed together with Igor Pavlov, who
is the author of 7-Zip (and now 7-Zip also supports LZMA2 and XZ
format). Accordingly to Igor Pavlov:

   LZMA2 provides the following advantages over LZMA:
   1) Better compression ratio for data than can't be compressed.
   It can store such blocks of data in uncompressed form.
   Also it decompresses such data faster.
   2) Better multithreading support. If you compress big file, LZMA2 can split
   that file to chunks and compress these chunks in multiple threads.

Clearly XZ is not very popular yet, but I would not call it as exotic.
At least, I expect it to be supported by future distributives.


Dmitry

Re: Add Gitweb support for LZMA compressed snapshots

[André Goddard Rosa]

On Sat, Aug 1, 2009 at 4:43 AM, Alex Riesen<raa.lkml@gmail.com> wrote:
> On Thu, Jul 30, 2009 at 09:44, J.H.<warthog19@eaglescrag.net> wrote:
>> Don't use 'lzma' the command, use 'xz' ( http://tukaani.org/xz/ ) as it uses
>
> The host seems to be down.
>

You can clone it here: git://ctrl.tukaani.org/xz.git

Best regards,
Andre

Re: Add Gitweb support for LZMA compressed snapshots

[Jim Meyering]

Alex Riesen wrote:
> On Thu, Jul 30, 2009 at 09:44, J.H.<warthog19@eaglescrag.net> wrote:
>> Don't use 'lzma' the command, use 'xz' ( http://tukaani.org/xz/ ) as it uses
>
> The host seems to be down.
>
>> the lzma2 format which is, by far, preferable to what 'lzma' outputs.  Same
>> compression (lzma) just different file format.
>
> And the whole project seems to be very young (listen on ohloh since 2007).
> And it is not in current stable distributions. Exotic thing.

XZ is young, but worthy, imho.
It was added to Fedora 11 in the last few days.

Re: Add Gitweb support for LZMA compressed snapshots

[Alex Riesen]

On Sat, Aug 1, 2009 at 16:58, Jim Meyering<jim@meyering.net> wrote:
> Alex Riesen wrote:
>> And it is not in current stable distributions. Exotic thing.
>
> XZ is young, but worthy, imho.
> It was added to Fedora 11 in the last few days.

Well, then maybe we should do by its good example and
support xz. In addition to .lzma, .7z, .cpio, etc.

Re: Add Gitweb support for LZMA compressed snapshots

[Johannes Schindelin]

Hi,

On Thu, 30 Jul 2009, Mark A Rada wrote:

> I thought I would submit this little patch I made to my gitweb. I am on 
> a relatively slow connection, and so LZMA compression time is less of a 
> concern than bandwidth---I'm guessing that I am not the only person who 
> suffers from slow internet connection syndrome.

That's great!

FWIW on this mailing list, we prefer to have patches inlined so that it is 
easier for reviewers to comment (it is almost impossible to get any patch 
into git.git without review, and because reviewing is such an ungratifying 
job you want to make it easier).

The patch is pretty straight-forward, but I'd love to see some information 
in the commit message about:

- performance numbers on the _server_ (i.e. how does the RAM and CPU load 
  compare to, say, bzip2?)

- how to disable/enable it (yes, it's in the documentation, but you can 
  make things easier still)

- whether it is turned on by default, and why.

Thanks,
Dscho

Re: Add Gitweb support for LZMA compressed snapshots

[Jakub Narebski]

Mark A Rada <marada@uwaterloo.ca> writes:

> I thought I would submit this little patch I made to my gitweb. I am
> on a relatively slow connection, and so LZMA compression time is
> less of a concern than bandwidth---I'm guessing that I am not the
> only person who suffers from slow internet connection syndrome.

First, Documentation/SubmittingPatches states that we prefer inline
patches, with additional comments (like the one above) either between
"---\n" line and diffstat, or like I did here before patch with patch
separated by comment by e.g. "-- >8 --\n" (scissors) line.

If posting patch inline isn't possible, and you have to use
attachement (for example because mailer you use wraps lines), please
at least use 'text/plain' mimetype so patch can be viewed without need
to save it in separate file (this might require changing suffix from
'.patch' to '.txt').


Second, if you are using 'txz' as internal name for new snapshot
format, why not use 'XZ utils' instead of 'LZMA Utils', and use
'display' => 'txz', 'type' => 'application/x-xz' and 
'suffix' => '.txz' (BTW. shouldn't suffix in your case be '.tar.lzma'
and not only '.lzma'?) and of course 'compressor' => ['xz']?


BTW. I wonder if it would be good idea to add support for this format
directly to git-archive...  OTOH it would mean additional dependency.

-- 
Jakub Narebski
Git User's Survey 2009: http://tinyurl.com/GitSurvey2009

-- >8 --
From: Mark Rada <marada@uwaterloo.ca>
Subject: [PATCH] Add Gitweb support for LZMA compressed snapshots

Signed-off-by: Mark Rada <marada@uwaterloo.ca>
---
 gitweb/gitweb.perl |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)

diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 38492d0..e131ea4 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -184,6 +184,13 @@ our %known_snapshot_formats = (
 		'format' => 'tar',
 		'compressor' => ['bzip2']},
 
+	'txz' => {
+		'display' => 'tar.lzma',
+		'type' => 'application/x-lzma',
+		'suffix' => '.lzma',
+		'format' => 'tar',
+		'compressor' => ['lzma']},
+
 	'zip' => {
 		'display' => 'zip',
 		'type' => 'application/x-zip',
@@ -196,6 +203,7 @@ our %known_snapshot_formats = (
 our %known_snapshot_format_aliases = (
 	'gzip'  => 'tgz',
 	'bzip2' => 'tbz2',
+	'lzma'  => 'txz',
 
 	# backward compatibility: legacy gitweb config support
 	'x-gzip' => undef, 'gz' => undef,
-- 
1.6.4

Re: Add Gitweb support for LZMA compressed snapshots

[Johannes Schindelin]

Hi,

On Thu, 30 Jul 2009, Jakub Narebski wrote:

> BTW. I wonder if it would be good idea to add support for this format
> directly to git-archive...  OTOH it would mean additional dependency.

I don't think it would be a good idea; we do not have bzip2 support 
either.

The only reason we have inbuilt gzip and zip support is because the format 
is so similar to Git's own compression.

Ciao,
Dscho

Re: Add Gitweb support for LZMA compressed snapshots

[Felipe Contreras]

On Thu, Jul 30, 2009 at 8:43 AM, Johannes
Schindelin<Johannes.Schindelin@gmx.de> wrote:
> Hi,
>
> On Thu, 30 Jul 2009, Jakub Narebski wrote:
>
>> BTW. I wonder if it would be good idea to add support for this format
>> directly to git-archive...  OTOH it would mean additional dependency.
>
> I don't think it would be a good idea; we do not have bzip2 support
> either.

bzip2 has no advantages whatsoever. AFAIK xz is superior to other
formats and it would be nice to see git make a technological stance
encouraging xz.

> The only reason we have inbuilt gzip and zip support is because the format
> is so similar to Git's own compression.

Personally I don't see the point of having zip support.

-- 
Felipe Contreras

Re: Add Gitweb support for LZMA compressed snapshots

[Johannes Schindelin]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1234 bytes --]

Hi,

On Fri, 31 Jul 2009, Felipe Contreras wrote:

> On Thu, Jul 30, 2009 at 8:43 AM, Johannes
> Schindelin<Johannes.Schindelin@gmx.de> wrote:
> > Hi,
> >
> > On Thu, 30 Jul 2009, Jakub Narebski wrote:
> >
> >> BTW. I wonder if it would be good idea to add support for this format
> >> directly to git-archive...  OTOH it would mean additional dependency.
> >
> > I don't think it would be a good idea; we do not have bzip2 support
> > either.
> 
> bzip2 has no advantages whatsoever.

Bzzzzt.  Wrong.  Just because you cannot see them does not mean it has no 
advantages.

> AFAIK xz is superior to other formats and it would be nice to see git 
> make a technological stance encouraging xz.

Bzzzt. Wrong again.  Git's mission in life is not to encourage one 
compression over another.

If at all, the only compression Git actually does promote in a sense is 
zlib compression.

> > The only reason we have inbuilt gzip and zip support is because the 
> > format is so similar to Git's own compression.
> 
> Personally I don't see the point of having zip support.

Personally, I see the point of having zip support.  It makes things easy 
for Windows users.  And it's an established format, much more so than 
tar.gz.

Ciao,
Dscho

Re: Add Gitweb support for LZMA compressed snapshots

[Felipe Contreras]

On Sat, Aug 1, 2009 at 12:34 PM, Johannes
Schindelin<Johannes.Schindelin@gmx.de> wrote:
> Hi,
>
> On Fri, 31 Jul 2009, Felipe Contreras wrote:
>
>> On Thu, Jul 30, 2009 at 8:43 AM, Johannes
>> Schindelin<Johannes.Schindelin@gmx.de> wrote:
>> > Hi,
>> >
>> > On Thu, 30 Jul 2009, Jakub Narebski wrote:
>> >
>> >> BTW. I wonder if it would be good idea to add support for this format
>> >> directly to git-archive...  OTOH it would mean additional dependency.
>> >
>> > I don't think it would be a good idea; we do not have bzip2 support
>> > either.
>>
>> bzip2 has no advantages whatsoever.
>
> Bzzzzt.  Wrong.  Just because you cannot see them does not mean it has no
> advantages.

All right, most the time you compress once, and multiple people
uncompress multiple times. Therefore the path that should be optimized
is the decompression, that leaves bzip2 out of the picture.

You can think of advantages of bzip2 in more esoteric cases where the
compression time is also important. I don't think that's the case
here. Actually it might be if gitweb is not caching the tarballs, but
if that's the case I wouldn't say it's an advantage of bzip2.

>> AFAIK xz is superior to other formats and it would be nice to see git
>> make a technological stance encouraging xz.
>
> Bzzzt. Wrong again.  Git's mission in life is not to encourage one
> compression over another.

Git's mission in life is not promoting asciidot either, but it's doing
that unintentionally by merely using it.

> If at all, the only compression Git actually does promote in a sense is
> zlib compression.
>
>> > The only reason we have inbuilt gzip and zip support is because the
>> > format is so similar to Git's own compression.
>>
>> Personally I don't see the point of having zip support.
>
> Personally, I see the point of having zip support.  It makes things easy
> for Windows users.  And it's an established format, much more so than
> tar.gz.

Windows can't extract .tar.gz?

-- 
Felipe Contreras

Re: Add Gitweb support for LZMA compressed snapshots

[Erik Faye-Lund]

On Sat, Aug 1, 2009 at 3:10 PM, Felipe
Contreras<felipe.contreras@gmail.com> wrote:
> Windows can't extract .tar.gz?

Not without installing 3. party software. Zip-support is built into
the operating system.

-- 
Erik "kusma" Faye-Lund
kusmabite@gmail.com
(+47) 986 59 656

Re: Add Gitweb support for LZMA compressed snapshots

[Mark A Rada]

On 1-Aug-09, at 10:04 AM, Erik Faye-Lund wrote:

> On Sat, Aug 1, 2009 at 3:10 PM, Felipe
> Contreras<felipe.contreras@gmail.com> wrote:
>> Windows can't extract .tar.gz?
>
> Not without installing 3. party software. Zip-support is built into
> the operating system.
>
> -- 
> Erik "kusma" Faye-Lund
> kusmabite@gmail.com
> (+47) 986 59 656
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

FWIW, Windows doesn't seem to come with a whole lot, and you need
3rd party software for a number of things, BUT that is also one of the
primary reasons that I need to be able to  quickly get snapshots from
my Git repositories at home, and using zip format is convenient if I am
in an extremely restrictive environment and cannot get a smaller XZ
compressed snapshot :).


--
Mark A Rada (ferrous26)
marada@uwaterloo.ca

Re: Add Gitweb support for LZMA compressed snapshots

[Erik Faye-Lund]

On Sat, Aug 1, 2009 at 6:07 PM, Mark A Rada<marada@uwaterloo.ca> wrote:
> FWIW, Windows doesn't seem to come with a whole lot, and you need
> 3rd party software for a number of things

Tell that to the US Department of Justice ;)

-- 
Erik "kusma" Faye-Lund
kusmabite@gmail.com
(+47) 986 59 656

Re: Add Gitweb support for LZMA compressed snapshots

[Dmitry Potapov]

On Fri, Jul 31, 2009 at 03:45:02PM +0000, Felipe Contreras wrote:
> 
> bzip2 has no advantages whatsoever. AFAIK xz is superior to other
> formats and it would be nice to see git make a technological stance
> encouraging xz.

git is not about encouraging any compression. If you have not noticed,
git-archive produces tar and not tgz (though it would not be difficult
to implement it, because we use zlib anyway). If you like xz and it is
installed on your system, you can always compress tar with it:

git archive HEAD | xz > my-archive.tar.xz

So, I see no reason to have this compression coupled with git.

> 
> > The only reason we have inbuilt gzip and zip support is because the format
> > is so similar to Git's own compression.
> 
> Personally I don't see the point of having zip support.

zip is still a popular format on Windows. If git archive was not able to
produce then creating would require creating tar first, extracting it to
some temporary directory and then creating a zip file. It is not very
efficient process and requires extra efforts, so because we use zlib
compression anyway, it makes perfect sense that we can create zip file
directly.

Dmitry