Re: Make "git am" properly unescape lines matching ">>*From "

["H. Peter Anvin" <hpa@zytor.com>]

On 06/08/2010 12:57 PM, Carl Worth wrote:
> I'm adding support to notmuch[1] to more easily pipe a thread full of
> patches to "git am". So I added support for notmuch to format a thread
> (or any search) as an mbox.
> 
> When I did that, I was careful to escape lines from the bodies of email
> messages that begin with zero or more '>' characters followed
> immediately by "From " (From_ lines) by adding an initial '>'. [2]
> 
> But I noticed that "git am" wasn't removing any of these added '>'
> characters, so I was getting corrupted commit messages.
> 
> I'll follow up this message with a patch that fixes that by making
> git-mailsplit un-escape these lines. It's careful to do this only when
> processing an actual mbox, using the existing detection of a bare email
> message and not doing any un-escaping in that case.
> 
> I'll also follow up with a new test for both cases, (using "git am" with
> both an mbox with escaped From_ lines and an email message without
> escaped From_ lines).
> 

The problem with that is that it is not universally applied.  For what
I've seen, some mbox-based programs simply rely on there being a
Content-Length: header and don't need From lines to be escaped at all
(and don't do anything useful if they are), some do the leading > trick
(usually not reversably at all).

As far as I can tell, the Content-Length: is the most reliably handled
format and probably is what we should use.  This is the "mboxcl2" format
in your list.[*]  Unfortunately "mboxcl2" and "mboxrd" cannot be
distinguished from each other by inspection, which is a major defect of
both formats.

The statement that "the entire "mbox" family of mailbox formats is
gradually becoming irrelevant, and of only historical interest" is also
pretty silly -- mbox is still the preferred format for moving groups of
email from MUA to MUA, even if it is no longer used for active live
spool storage.  But, of course, you knew that already.

	-hpa

[*] There are apparently some MTA/MUAs which simply bypass the entire
problem by base64-encoding any email that contains /^From /, just as if
it contained NUL bytes.  It's a heavyweight, but thoroughly unambiguous
way of dealing with the problem.