Gitweb 1.7.5 and Textconv Configuration

Gitweb 1.7.5 and Textconv Configuration

[Al Haraka]

All,

I am sorry to bother the list with this question, but I have spent a
lot of time searching and have never really found what I was looking
for.  No one had answers to my question on IRC either.  I am running
git/gitweb 1.7.5 on shared hosting (Dreamhost, if that matters).  I
compiled it and followed a tutorial
(https://github.com/tmacam/private-git-on-dreamhost) on setting it up
in my environment.  It worked great, but I have some stuff that is not
source code, namely PDF's and ODP/ODT/ODS files I want to see changes
from beyond knowing they are binary files.  Enter the official wiki
page explaining textconv
(https://git.wiki.kernel.org/index.php/GitTips#How_to_use_git_to_track_OpenDocument_.28OpenOffice.2C_Koffice.29_files.3F).

So, I tried different variations of this, specifically for repos.
Now, I have tried it globally as that was tedious and it was not
working anyway. That works some of the time if I try commands over the
console (git log  -p for quick testing), but never in Gitweb.  It
continues to only say they are differing binary files.  Does Gitweb
1.7.5 support the use of this?  Is there something special I must do?
Below is my current config:

[dreamhost-server]$ git config --list --global
diff.odf.textconv=odt2txt
diff.odf.binary=false
diff.pdf.textconv=pdftotext
diff.pdf.binary=false
core.attributesfile=~/.gitattributes
[dreamhost-server]$ more ~/.gitattributes
*.odf diff=odf
*.odt diff=odf
*.odp diff=odf
*.pdf diff=pdf
[jabbah]$ more ~/repos/gitweb_config.perl
# where is the git binary?
$GIT = "/home/dreamhost_account/opt/bin/git";
# where are our git project repositories?
$projectroot = $ENV{'GIT_PROJECT_ROOT'};
# what do we call our projects in the gitweb UI?
$home_link_str = "~";
#  where are the files we need for gitweb to display?
@stylesheets = ("gitweb.css");
$logo = "git-logo.png";
$favicon = "/favicon.png";
# what do we call this site?
$site_name = "my personal git repos";

The diff.<driver>.binary addition is new.  I tried with or without
them, hoping mention of this on a mailing list thread from a while
back might help.  It was all in vain.  I also tried fiddling with
$diff_opts for Gitweb after scanning the man page for git-diff, but I
am not sure if there is anything helpful I missed.  I no this is a
newbish question, I am sure, so I apologize in advance.

Best,
_AJS

Re: Gitweb 1.7.5 and Textconv Configuration

[Jeff King]

On Sat, Jul 02, 2011 at 12:50:12PM +0300, Al Haraka wrote:

> So, I tried different variations of this, specifically for repos.
> Now, I have tried it globally as that was tedious and it was not
> working anyway. That works some of the time if I try commands over the
> console (git log  -p for quick testing), but never in Gitweb.  It
> continues to only say they are differing binary files.  Does Gitweb
> 1.7.5 support the use of this?  Is there something special I must do?
> Below is my current config:

The default gitweb configuration doesn't allow textconv (or external
diff) for security reasons; it would mean repo owners could execute
arbitrary commands as the gitweb user.

If that's OK for your site, you can probably enable it with:

  push @diff_opts, '--allow-textconv';

I don't know that anybody has tested that, though. It will use textconv
_everywhere_ in gitweb, including the "format-patch" invocation for
showing raw patches of commits. It might make sense in the long run for
gitweb to have another variable like @diff_opts that is used for
human-readable output (e.g., the HTML output, but not anything you might
import to a tool).

-Peff

Re: Gitweb 1.7.5 and Textconv Configuration

[Jeff King]

On Sat, Jul 02, 2011 at 06:43:13AM -0400, Jeff King wrote:

> The default gitweb configuration doesn't allow textconv (or external
> diff) for security reasons; it would mean repo owners could execute
> arbitrary commands as the gitweb user.
> 
> If that's OK for your site, you can probably enable it with:
> 
>   push @diff_opts, '--allow-textconv';

Oops, this option is just called "--textconv". I got it confused with
the flag in the code, which is ALLOW_TEXTCONV.

-Peff

Re: Gitweb 1.7.5 and Textconv Configuration

[Al Haraka]

On Sat, Jul 2, 2011 at 1:44 PM, Jeff King <peff@peff.net> wrote:
> On Sat, Jul 02, 2011 at 06:43:13AM -0400, Jeff King wrote:
>
>> The default gitweb configuration doesn't allow textconv (or external
>> diff) for security reasons; it would mean repo owners could execute
>> arbitrary commands as the gitweb user.
>>
>> If that's OK for your site, you can probably enable it with:
>>
>>   push @diff_opts, '--allow-textconv';

Just tried it.

$ more ~/repos/gitweb_config.perl
# where is the git binary?
$GIT = "/home/dreamhost_account/opt/bin/git";
# where are our git project repositories?
$projectroot = $ENV{'GIT_PROJECT_ROOT'};
# what do we call our projects in the gitweb UI?
$home_link_str = "~";
#  where are the files we need for gitweb to display?
@stylesheets = ("gitweb.css");
$logo = "git-logo.png";
$favicon = "/favicon.png";
# what do we call this site?
$site_name = "my personal git repos";
# Enable textconv
push @diff_opts, '--textconv';

Reloaded the webpage; still does *not* show text diffs.  Any way to
debug what is going wrong?

> Oops, this option is just called "--textconv". I got it confused with
> the flag in the code, which is ALLOW_TEXTCONV.

No worries.  You are steering my in the right direction, typos or not.

> -Peff
>



-- 
Alexander J. Stein
Cell:  +974 70013750
Email: alharaka@gmail.com
Skype: alexander.j.stein

Re: Gitweb 1.7.5 and Textconv Configuration

[Jeff King]

On Sat, Jul 02, 2011 at 02:14:30PM +0300, Al Haraka wrote:

> # Enable textconv
> push @diff_opts, '--textconv';
> 
> Reloaded the webpage; still does *not* show text diffs.  Any way to
> debug what is going wrong?

You could try setting "$ENV{GIT_TRACE} = 1", which will dump all of the
git commands being called to stderr (and presumaby your log). That would
double-check that the option is being passed.

I think what might be happening, though, is that git does not do a good
job checking .gitattributes files inside trees; it only checks the
.gitattributes in the working tree, even if you are diffing a tree. But
in the case of a bare repository, we don't even have a working tree at
all.

You can try working around it like this:

  cd /path/to/bare/repo.git
  git show HEAD:.gitattributes >info/attributes

which will make the repository-wide non-version-controlled gitattributes
the same as the last committed version. The problem is that it won't be
automatically updated as you commit and push changes to .gitattributes.

So I think it's a bug in git. It seems like the tree being diffed is an
obvious place for git to load gitattributes from. Though I'm not sure of
the precedence rules for interacting with the version of .gitattributes
in the worktree (if any). In a situation like this:

  $ git checkout master
  $ git show master~50

You would probably want the more recent worktree attributes to take
precedence over the ones in the older tree.

But in this:

  $ git checkout master~50
  $ git show master

you would probably want the more recent tree attributes to take
precedence over what's in the working tree.

-Peff

Re: Gitweb 1.7.5 and Textconv Configuration

[Al Haraka]

On Sat, Jul 2, 2011 at 9:57 PM, Jeff King <peff@peff.net> wrote:
> You could try setting "$ENV{GIT_TRACE} = 1", which will dump all of the
> git commands being called to stderr (and presumaby your log). That would
> double-check that the option is being passed.

Great.  Thanks for the tip.

> I think what might be happening, though, is that git does not do a good
> job checking .gitattributes files inside trees; it only checks the
> .gitattributes in the working tree, even if you are diffing a tree. But
> in the case of a bare repository, we don't even have a working tree at
> all.
>
> You can try working around it like this:
>
>  cd /path/to/bare/repo.git
>  git show HEAD:.gitattributes >info/attributes

This did not work, for what I think might be an obvious reason; I do
not have a .gitattributes file *in* the repo or any repos anymore, but
a global one.  See below.

[dreamhost-server]$ git show HEAD:.gitattributes >info/attributes
fatal: Path '.gitattributes' exists on disk, but not in 'HEAD'.

> which will make the repository-wide non-version-controlled gitattributes
> the same as the last committed version. The problem is that it won't be
> automatically updated as you commit and push changes to .gitattributes.

I thought my plan was to try and avoid this by using the
core.attributesfile directive, forcing this stuff to operate system
(well, account, besides the point here) wide on all repos with
specifying a .gitattributes (or, since it base bare, as you pointed
out yourself, $GIT_REPO_DIR/info/attributes) every single time.  Did I
misunderstand the mailing list thread that mentioned this a while
back?

[dreamhost-server]$ git config --list --global
diff.odf.textconv=odt2txt
diff.odf.binary=false
diff.pdf.textconv=pdftotext
diff.pdf.binary=false
core.attributesfile=~/.gitattributes

Inside one of the troublesome repos (so resultant collection of global
and repo-specific config):

[dreamhost-server]$ git config --list
diff.odf.textconv=odt2txt
diff.odf.binary=false
diff.pdf.textconv=pdftotext
diff.pdf.binary=false
core.attributesfile=~/.gitattributes
core.repositoryformatversion=0
core.filemode=true
core.bare=true
gitweb.owner=alharaka@gmail.com
diff.odf.textconv=odt2txt

This is the reason I went through the trouble of compiling an updated
version in my account (as opposed to the installed version on the
Dreamhost box; they are stuck at version 1.7.1.1; I saw this mentioned
on a thread somewhere and wanted to get the "latest" (well latest
stable version) to avoid this kind of problem?  Was that the right
thing to do?  Will it even work in this case?  I get the feeling from
your response I was expecting a lot with RTFM'ing more.  I have a
script, as described in the Github HOWTO, that creates the bare repo
as desired for me.  So, if this will not work, I will just modify my
version of the bash script to handle this.  I just am obviously not
sure what the "proper" approach or best practice is anymore.

Thanks again for your help.  I really appreciate it.  I am new to git,
but LOVE what you guys have being doing.  I never used any DRCS before
this, personally or professionally, but read snippets of howtos and so
much good press I decided it was time I mooch off you guys to organize
my life.  :-)

Re: Gitweb 1.7.5 and Textconv Configuration

[Jeff King]

On Sat, Jul 02, 2011 at 10:18:55PM +0300, Al Haraka wrote:

> > which will make the repository-wide non-version-controlled gitattributes
> > the same as the last committed version. The problem is that it won't be
> > automatically updated as you commit and push changes to .gitattributes.
> 
> I thought my plan was to try and avoid this by using the
> core.attributesfile directive, forcing this stuff to operate system
> (well, account, besides the point here) wide on all repos with
> specifying a .gitattributes (or, since it base bare, as you pointed
> out yourself, $GIT_REPO_DIR/info/attributes) every single time.  Did I
> misunderstand the mailing list thread that mentioned this a while
> back?

Ah, I see. That seems like a reasonable solution. Are you sure that the
user running gitweb as a CGI is the same as the user you log in as? That
is, are you sure that ~/.gitconfig is being parsed when it is called as
a CGI, and it's not looking in ~www/.gitconfig or something?

It would depend how your hosting is set up.

> This is the reason I went through the trouble of compiling an updated
> version in my account (as opposed to the installed version on the
> Dreamhost box; they are stuck at version 1.7.1.1; I saw this mentioned
> on a thread somewhere and wanted to get the "latest" (well latest
> stable version) to avoid this kind of problem?  Was that the right
> thing to do?  Will it even work in this case?  I get the feeling from
> your response I was expecting a lot with RTFM'ing more.

It sounds like it should work to me, but I've not tested it (nor do I
even run gitweb; I just have an interest in textconv).

> Thanks again for your help.  I really appreciate it.  I am new to git,
> but LOVE what you guys have being doing.  I never used any DRCS before
> this, personally or professionally, but read snippets of howtos and so
> much good press I decided it was time I mooch off you guys to organize
> my life.  :-)

You're welcome. Hopefully we can resolve your problem and get the answer
at least in the list archive. And then you will have made the world a
better place. :)

-Peff

Re: Gitweb 1.7.5 and Textconv Configuration

[Jakub Narebski]

Jeff King <peff@peff.net> writes:
> On Sat, Jul 02, 2011 at 10:18:55PM +0300, Al Haraka wrote:
> 
> > > which will make the repository-wide non-version-controlled gitattributes
> > > the same as the last committed version. The problem is that it won't be
> > > automatically updated as you commit and push changes to .gitattributes.
> > 
> > I thought my plan was to try and avoid this by using the
> > core.attributesfile directive, forcing this stuff to operate system
> > (well, account, besides the point here) wide on all repos with
> > specifying a .gitattributes (or, since it base bare, as you pointed
> > out yourself, $GIT_REPO_DIR/info/attributes) every single time.  Did I
> > misunderstand the mailing list thread that mentioned this a while
> > back?
> 
> Ah, I see. That seems like a reasonable solution. Are you sure that the
> user running gitweb as a CGI is the same as the user you log in as? That
> is, are you sure that ~/.gitconfig is being parsed when it is called as
> a CGI, and it's not looking in ~www/.gitconfig or something?
> 
> It would depend how your hosting is set up.

Well, there is also system wide $(prefix)/etc/gitconfig file...
 
> > This is the reason I went through the trouble of compiling an updated
> > version in my account (as opposed to the installed version on the
> > Dreamhost box; they are stuck at version 1.7.1.1; I saw this mentioned
> > on a thread somewhere and wanted to get the "latest" (well latest
> > stable version) to avoid this kind of problem?  Was that the right
> > thing to do?  Will it even work in this case?  I get the feeling from
> > your response I was expecting a lot with RTFM'ing more.
> 
> It sounds like it should work to me, but I've not tested it (nor do I
> even run gitweb; I just have an interest in textconv).

The question is if --textconv works with git-diff-tree, because that
is what gitweb uses.

BTW. we could use --textconv in 'blob' and 'blame' views (it is
documented that git-cat-file supports --textconv, and it is checked in
git testsuite but not documented that git-blame supports --textconv).
But it would require changes to gitweb.


Hoping that this email will made it...
-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: Gitweb 1.7.5 and Textconv Configuration

[Jeff King]

On Sat, Jul 02, 2011 at 01:23:38PM -0700, Jakub Narebski wrote:

> > Ah, I see. That seems like a reasonable solution. Are you sure that the
> > user running gitweb as a CGI is the same as the user you log in as? That
> > is, are you sure that ~/.gitconfig is being parsed when it is called as
> > a CGI, and it's not looking in ~www/.gitconfig or something?
> > 
> > It would depend how your hosting is set up.
> 
> Well, there is also system wide $(prefix)/etc/gitconfig file...

Good point. Though if he's on shared hosting, that might not be an
option.

> The question is if --textconv works with git-diff-tree, because that
> is what gitweb uses.

It does. It just defaults to "--no-textconv", but you can enable it on
the command-line.

-Peff

Re: Gitweb 1.7.5 and Textconv Configuration

[Al Haraka]

On Sat, Jul 2, 2011 at 11:23 PM, Jakub Narebski <jnareb@gmail.com> wrote:
> Jeff King <peff@peff.net> writes:
>> On Sat, Jul 02, 2011 at 10:18:55PM +0300, Al Haraka wrote:
>>
>> > > which will make the repository-wide non-version-controlled gitattributes
>> > > the same as the last committed version. The problem is that it won't be
>> > > automatically updated as you commit and push changes to .gitattributes.
>> >
>> > I thought my plan was to try and avoid this by using the
>> > core.attributesfile directive, forcing this stuff to operate system
>> > (well, account, besides the point here) wide on all repos with
>> > specifying a .gitattributes (or, since it base bare, as you pointed
>> > out yourself, $GIT_REPO_DIR/info/attributes) every single time.  Did I
>> > misunderstand the mailing list thread that mentioned this a while
>> > back?
>>
>> Ah, I see. That seems like a reasonable solution. Are you sure that the
>> user running gitweb as a CGI is the same as the user you log in as? That
>> is, are you sure that ~/.gitconfig is being parsed when it is called as
>> a CGI, and it's not looking in ~www/.gitconfig or something?
>>
>> It would depend how your hosting is set up.
>
> Well, there is also system wide $(prefix)/etc/gitconfig file...

Unfortunately, this is on Dreamhost.  I do not have root access, and
this is a kludge.  I know what I am asking for is a very exclusive use
case, but I would love to have this kind of feature until people tell
me it cannot be done.  Haha.

>> > This is the reason I went through the trouble of compiling an updated
>> > version in my account (as opposed to the installed version on the
>> > Dreamhost box; they are stuck at version 1.7.1.1; I saw this mentioned
>> > on a thread somewhere and wanted to get the "latest" (well latest
>> > stable version) to avoid this kind of problem?  Was that the right
>> > thing to do?  Will it even work in this case?  I get the feeling from
>> > your response I was expecting a lot with RTFM'ing more.
>>
>> It sounds like it should work to me, but I've not tested it (nor do I
>> even run gitweb; I just have an interest in textconv).
>
> The question is if --textconv works with git-diff-tree, because that
> is what gitweb uses.

It does work on git-dif-tree when I just checked.  So, I presume that
this means Jeff is right, and it must be not pulling the config from
the right location?

Jeff, on that note, I fear you are right about the appropriate user
executing the script:

http://wiki.dreamhost.com/Suexec

I get the feeling this is the case because in several instances I
thought I was clever with my use of ~ or $HOME, or the tutorial writer
had been himself.  In those instances, using full paths and not
user-specific environment vars fixed the issues.  So, I guess I need
to figure a way to pass the config vars appropriately?  I already
changed the core.attributesfile from ~/.gitattributes to
/home/dreamhost_user/.gitattributes in anticipation, but I know that
is only the first step.  I need to go to head out for a bit.  I will
revisit the issue tomorrow morning when I have time to think about the
right way to fix passing the environment variables for suexec.  I am
an idiot.

> BTW. we could use --textconv in 'blob' and 'blame' views (it is
> documented that git-cat-file supports --textconv, and it is checked in
> git testsuite but not documented that git-blame supports --textconv).
> But it would require changes to gitweb.
>
>
> Hoping that this email will made it...

It made it indeed.  Much appreciated for both your efforts so far.

> --
> Jakub Narebski
> Poland
> ShadeHawk on #git
>



-- 
Alexander J. Stein
Cell:  +974 70013750
Email: alharaka@gmail.com
Skype: alexander.j.stein

Re: Gitweb 1.7.5 and Textconv Configuration

[Jakub Narebski]

On Sat, 2 Jul 2011, Jeff King wrote:
> On Sat, Jul 02, 2011 at 01:23:38PM -0700, Jakub Narebski wrote:
> 
> > > Ah, I see. That seems like a reasonable solution. Are you sure that the
> > > user running gitweb as a CGI is the same as the user you log in as? That
> > > is, are you sure that ~/.gitconfig is being parsed when it is called as
> > > a CGI, and it's not looking in ~www/.gitconfig or something?
> > > 
> > > It would depend how your hosting is set up.
> > 
> > Well, there is also system wide $(prefix)/etc/gitconfig file...
> 
> Good point. Though if he's on shared hosting, that might not be an
> option.

Though on the other hand if he is compiling git himself, $prefix can
be $HOME...

> > The question is if --textconv works with git-diff-tree, because that
> > is what gitweb uses.
> 
> It does. It just defaults to "--no-textconv", but you can enable it on
> the command-line.

Undocumented.

  $ git grep -e --textconv Documentation/
  Documentation/RelNotes/1.6.3.3.txt: * "git diff --textconv" leaked memory badly when the tex
  Documentation/git-cat-file.txt:'git cat-file' (-t | -s | -e | -p | <type> | --textconv ) <ob
  Documentation/git-cat-file.txt:object type, or '-s' is used to find the object size, or '--t
  Documentation/git-cat-file.txt:--textconv::

(on current 'master').

-- 
Jakub Narebski
Poland

Re: Gitweb 1.7.5 and Textconv Configuration

[Jakub Narebski]

Jeff King <peff@peff.net> writes:

> On Sat, Jul 02, 2011 at 12:50:12PM +0300, Al Haraka wrote:
> 
> > So, I tried different variations of this, specifically for repos.
> > Now, I have tried it globally as that was tedious and it was not
> > working anyway. That works some of the time if I try commands over the
> > console (git log  -p for quick testing), but never in Gitweb.  It
> > continues to only say they are differing binary files.  Does Gitweb
> > 1.7.5 support the use of this?  Is there something special I must do?
> > Below is my current config:
> 
> The default gitweb configuration doesn't allow textconv (or external
> diff) for security reasons; it would mean repo owners could execute
> arbitrary commands as the gitweb user.
> 
> If that's OK for your site, you can probably enable it with:
> 
>   push @diff_opts, '--allow-textconv';
> 
> I don't know that anybody has tested that, though. It will use textconv
> _everywhere_ in gitweb, including the "format-patch" invocation for
> showing raw patches of commits. It might make sense in the long run for
> gitweb to have another variable like @diff_opts that is used for
> human-readable output (e.g., the HTML output, but not anything you might
> import to a tool).

Actually I think neither git-format-patch (which is used by 'patch'
and 'patches' views), not git-diff-tree (used by 'commitdiff' and
'blobdiff' views, and their *_plain versions) supports --textconv,
and that it is supported only by git-diff.  But you better check that.

Nb documentation as of 1.7.5 is entirely unhelpful.  Only git-cat-file
is documented as using --textconv... though this option is examined
in diff.c, and there is test for git-blame supporting it.


By the way, I don't know if git read .gitattributes file (which
defines textconv... unless you use per repository instance
.git/info/attributes) from tree for given commit, or only from working
area.  

git-chec-attr (official script API) can check only state of
gitattributes in working area.


So I don't think that gitweb can support textconv in diffs; it should
be quite easy to add suppport for --textconv in 'blob' view, ...

... but it wouldn't be in gitweb 1.7.5 ;-)
-- 
Jakub Narebski
Poland
ShadeHawk on #git

[PATCH] docs: document --textconv diff option

[Jeff King]

On Wed, Jul 06, 2011 at 06:59:53AM -0700, Jakub Narebski wrote:

> Actually I think neither git-format-patch (which is used by 'patch'
> and 'patches' views), not git-diff-tree (used by 'commitdiff' and
> 'blobdiff' views, and their *_plain versions) supports --textconv,
> and that it is supported only by git-diff.  But you better check that.

I did. It works for both.

> Nb documentation as of 1.7.5 is entirely unhelpful.  Only git-cat-file
> is documented as using --textconv... though this option is examined
> in diff.c, and there is test for git-blame supporting it.

Patch below.

> By the way, I don't know if git read .gitattributes file (which
> defines textconv... unless you use per repository instance
> .git/info/attributes) from tree for given commit, or only from working
> area.

It's not currently read from the tree; see earlier in the thread for
some discussion.

-Peff

-- >8 --
Subject: [PATCH] docs: document --textconv diff option

This has been there since textconv existed, but was never
documented. There is some overlap with what's in
gitattributes(5), but it's important to warn in both places
that textconv diffs probably can't be applied.

Signed-off-by: Jeff King <peff@peff.net>
---
 Documentation/diff-options.txt |   11 +++++++++++
 1 files changed, 11 insertions(+), 0 deletions(-)

diff --git a/Documentation/diff-options.txt b/Documentation/diff-options.txt
index 4235302..b620b3a 100644
--- a/Documentation/diff-options.txt
+++ b/Documentation/diff-options.txt
@@ -422,6 +422,17 @@ endif::git-format-patch[]
 --no-ext-diff::
 	Disallow external diff drivers.
 
+--textconv::
+--no-textconv::
+	Allow (or disallow) external text conversion filters to be run
+	when comparing binary files. See linkgit:gitattributes[5] for
+	details. Because textconv filters are typically a one-way
+	conversion, the resulting diff is suitable for human
+	consumption, but cannot be applied. For this reason, textconv
+	filters are enabled by default only for linkgit:git-diff[1] and
+	linkgit:git-log[1], but not for linkgit:git-format-patch[1] or
+	diff plumbing commands.
+
 --ignore-submodules[=<when>]::
 	Ignore changes to submodules in the diff generation. <when> can be
 	either "none", "untracked", "dirty" or "all", which is the default
-- 
1.7.6.20.g45f3f.dirty