Question on textconv

Question on textconv

[rsbecker]

I experienced a change in how textconv works since about 2.50 and it is hard
to get past. I would appreciate advice:

When I define an external binary textconv, roughly like:

.gitattributes:
simple binary diff=enscr

.gitconfig
[diff "enscr"]
        textconv = run -debug ../../enscribe-conv --verbose
        binary = true

The supplied file going to the textconv program looks like
/tmp/git-blob-GFtIhK/simple
and is always empty regardless of the file contents.

When there is only one file named simple in the repository I can find it,
but otherwise
any ambiguity in the name makes textconv processing impractical. Somewhere
prior to this
I was supplied with the actual file in the working index instead of a temp
file.

Am I missing something?

Thanks,
Randall

Re: Question on textconv

[D. Ben Knoble]

Hi Randall,

On Tue, Aug 4, 2026 at 2:52 PM <rsbecker@nexbridge.com> wrote:
>
> I experienced a change in how textconv works since about 2.50 and it is hard
> to get past. I would appreciate advice:
>
> When I define an external binary textconv, roughly like:
>
> .gitattributes:
> simple binary diff=enscr
>
> .gitconfig
> [diff "enscr"]
>         textconv = run -debug ../../enscribe-conv --verbose
>         binary = true
>
> The supplied file going to the textconv program looks like
> /tmp/git-blob-GFtIhK/simple
> and is always empty regardless of the file contents.

That's strange. Over here (2.53.0), I get the temp file, but it has
the expected contents.

(For example, I can debug a bit by using the shell form 'textconv =
"f() { echo $@; cat $@; <stuff>; }; f"'. NB those $@ need quoted in
production, but for my little test case that was more hassle than it
was worth. We can also see the invocation using GIT_TRACE2=1.)

This was true for me even after setting "binary = true" in the diff
driver block, which I thought was a bit unusual.

> When there is only one file named simple in the repository I can find it,
> but otherwise
> any ambiguity in the name makes textconv processing impractical. Somewhere
> prior to this
> I was supplied with the actual file in the working index instead of a temp
> file.
>
> Am I missing something?
>
> Thanks,
> Randall

I'm not sure :/

-- 
D. Ben Knoble

RE: Question on textconv

[rsbecker]

On August 4, 2026 4:53 PM, D. Ben Knoble wrote:
> On Tue, Aug 4, 2026 at 2:52 PM <rsbecker@nexbridge.com> wrote:
> >
> > I experienced a change in how textconv works since about 2.50 and it
> > is hard to get past. I would appreciate advice:
> >
> > When I define an external binary textconv, roughly like:
> >
> > .gitattributes:
> > simple binary diff=enscr
> >
> > .gitconfig
> > [diff "enscr"]
> >         textconv = run -debug ../../enscribe-conv --verbose
> >         binary = true
> >
> > The supplied file going to the textconv program looks like
> > /tmp/git-blob-GFtIhK/simple and is always empty regardless of the file
> > contents.
> 
> That's strange. Over here (2.53.0), I get the temp file, but it has the expected
> contents.
> 
> (For example, I can debug a bit by using the shell form 'textconv =
> "f() { echo $@; cat $@; <stuff>; }; f"'. NB those $@ need quoted in production, but
> for my little test case that was more hassle than it was worth. We can also see the
> invocation using GIT_TRACE2=1.)
> 
> This was true for me even after setting "binary = true" in the diff driver block, which
> I thought was a bit unusual.
> 
> > When there is only one file named simple in the repository I can find
> > it, but otherwise any ambiguity in the name makes textconv processing
> > impractical. Somewhere prior to this I was supplied with the actual
> > file in the working index instead of a temp file.
> >
> > Am I missing something?
> >
> > Thanks,
> > Randall
> 
> I'm not sure :/

Curiously, I get this only on binary files. It is almost as if leading nulls is causing an issue.
If I try on a text file or an ELF format file, the above works - ELF does not begin with a NULL.

Re: Question on textconv

[Jeff King]

On Tue, Aug 04, 2026 at 02:44:22PM -0400, rsbecker@nexbridge.com wrote:

> The supplied file going to the textconv program looks like
> /tmp/git-blob-GFtIhK/simple
> and is always empty regardless of the file contents.

I can't reproduce the problem here, even for files with embedded NULs.
However...

> When there is only one file named simple in the repository I can find
> it, but otherwise any ambiguity in the name makes textconv processing
> impractical. Somewhere prior to this I was supplied with the actual
> file in the working index instead of a temp file.

This part I can explain. We sometimes try to reuse the working tree
instead of generating a tempfile, as an optimization. We can only do
this when the working tree file is clean. But we also only bother to try
when one of the diff endpoints is the index. So if we set up a sample
textconv like:

  git config diff.foo.textconv 'echo >&2 "got: $*" && tr a-z A-Z <'
  echo "file diff=foo" >.gitattributes

  echo one >file && git add file && git commit -m one
  echo two >file && git add file && git commit -m two

The running either "git diff HEAD^" or "git diff --cached HEAD^" will
convert the copy in the working tree, and you'll get:

  got: /tmp/git-blob-0CLCMr/file
  got: file
  diff --git a/file b/file
  index 5626abf..f719efd 100644
  --- a/file
  +++ b/file
  @@ -1 +1 @@
  -ONE
  +TWO

but if you do "git show HEAD", you'll get two tempfiles:

  got: /tmp/git-blob-w1binM/file
  got: /tmp/git-blob-1nu2Rm/file
  [same diff]

even though this is the same diff! We _could_ try harder to reuse the
working tree copy here by checking whether the path has the same sha1 in
the tree and the index (and that the index entry is clean). But it only
helps in a few special cases, and it's not something users should rely
on (we might choose to create a tempfile anyway if the index is
stat-dirty).

-Peff

Re: Question on textconv

[D. Ben Knoble]

On Wed, Aug 5, 2026 at 12:50 AM Jeff King <peff@peff.net> wrote:
>
> On Tue, Aug 04, 2026 at 02:44:22PM -0400, rsbecker@nexbridge.com wrote:
>
> > The supplied file going to the textconv program looks like
> > /tmp/git-blob-GFtIhK/simple
> > and is always empty regardless of the file contents.
>
> I can't reproduce the problem here, even for files with embedded NULs.

Yeah, same:

    dd if=/dev/zero of=file count=1
    echo 'file diff=debug' >.gitattributes
    git config diff.debug.textconv 'echo >&2 "got $*"; xxd'

shows (with "git add --intent-to-add file; git diff file") the xxd
dump of zeros from the working tree. Then

    git commit file -mwip
    dd if=/dev/zero of=file count=2
    git diff file

shows a diff between a temp file and the working tree (but still
correctly showing the new block of zeros).

RE: Question on textconv

[rsbecker]

On August 5, 2026 12:50 AM, Jeff King wrote:
> On Tue, Aug 04, 2026 at 02:44:22PM -0400, rsbecker@nexbridge.com wrote:
> 
> > The supplied file going to the textconv program looks like
> > /tmp/git-blob-GFtIhK/simple and is always empty regardless of the file
> > contents.
> 
> I can't reproduce the problem here, even for files with embedded NULs.
> However...
> 
> > When there is only one file named simple in the repository I can find
> > it, but otherwise any ambiguity in the name makes textconv processing
> > impractical. Somewhere prior to this I was supplied with the actual
> > file in the working index instead of a temp file.
> 
> This part I can explain. We sometimes try to reuse the working tree instead of
> generating a tempfile, as an optimization. We can only do this when the working
> tree file is clean. But we also only bother to try when one of the diff endpoints is
> the index. So if we set up a sample textconv like:
> 
>   git config diff.foo.textconv 'echo >&2 "got: $*" && tr a-z A-Z <'
>   echo "file diff=foo" >.gitattributes
> 
>   echo one >file && git add file && git commit -m one
>   echo two >file && git add file && git commit -m two
> 
> The running either "git diff HEAD^" or "git diff --cached HEAD^" will convert the
> copy in the working tree, and you'll get:
> 
>   got: /tmp/git-blob-0CLCMr/file
>   got: file
>   diff --git a/file b/file
>   index 5626abf..f719efd 100644
>   --- a/file
>   +++ b/file
>   @@ -1 +1 @@
>   -ONE
>   +TWO
> 
> but if you do "git show HEAD", you'll get two tempfiles:
> 
>   got: /tmp/git-blob-w1binM/file
>   got: /tmp/git-blob-1nu2Rm/file
>   [same diff]
> 
> even though this is the same diff! We _could_ try harder to reuse the working tree
> copy here by checking whether the path has the same sha1 in the tree and the
> index (and that the index entry is clean). But it only helps in a few special cases, and
> it's not something users should rely on (we might choose to create a tempfile
> anyway if the index is stat-dirty).

Could we extend textconv to support %f (the original path) if specified in the
textconv configuration? That would solve the ambiguity of what is being supplied.

Re: Question on textconv

[Jeff King]

On Wed, Aug 05, 2026 at 10:05:52AM -0400, rsbecker@nexbridge.com wrote:

> Could we extend textconv to support %f (the original path) if specified in the
> textconv configuration? That would solve the ambiguity of what is being supplied.

In theory, yes. But there is one gotcha: there's a system for caching
textconv output in git-notes, and it uses only the original blob id as
the cache key.

So I'm not opposed to adding something like %f, as long as the patch to
do so handles the caching problem (even if it just refuses to cache,
that would be much better than returning possibly-wrong results).

That said, it sounds like you just want %f to work around a bug where
the content is not provided. Probably fixing the bug is a better path
forward. Looking at the working tree file to get the contents will not
always be correct (e.g., if you're diffing an old tree).

-Peff