git imap-send converting my patches to CRLF line endings?

git imap-send converting my patches to CRLF line endings?

[Michael Mc Donnell]

Hi

I'm using git imap-send to send patches to wine-patches, and it seems
like it converts all my patches to have CRLF line endings?

I can see it when I download the patch from the Gmail drafts folder.
Git complains about white space when I apply the downloaded patch. It
works fine if I just use git to create the patch and then apply it on
a new branch. Is it git imap-send or just Gmail that's the problem?

Is there any way to disable the conversion?

I'm using git 1.7.5.4

Thanks,
Michael Mc Donnell

Re: git imap-send converting my patches to CRLF line endings?

[Jeff King]

On Fri, Jun 17, 2011 at 03:35:04PM +0200, Michael Mc Donnell wrote:

> I'm using git imap-send to send patches to wine-patches, and it seems
> like it converts all my patches to have CRLF line endings?

The canonical line ending for mail is CRLF. So yes, it will convert your
patch to CRLF for storage. But anything pulling it out of the IMAP
folder should convert it back to native line endings.

> I can see it when I download the patch from the Gmail drafts folder.
> Git complains about white space when I apply the downloaded patch. It
> works fine if I just use git to create the patch and then apply it on
> a new branch. Is it git imap-send or just Gmail that's the problem?

How do you download and apply the patch exactly? If you are speaking
imap to gmail, generally the client would strip out the CR's from the
mail.

-Peff

Re: git imap-send converting my patches to CRLF line endings?

[Michael Mc Donnell]

On Fri, Jun 17, 2011 at 4:14 PM, Jeff King <peff@peff.net> wrote:
> On Fri, Jun 17, 2011 at 03:35:04PM +0200, Michael Mc Donnell wrote:
>
>> I'm using git imap-send to send patches to wine-patches, and it seems
>> like it converts all my patches to have CRLF line endings?
>
> The canonical line ending for mail is CRLF. So yes, it will convert your
> patch to CRLF for storage. But anything pulling it out of the IMAP
> folder should convert it back to native line endings.

Ok, so it's the clients responsibility to convert it back?

>> I can see it when I download the patch from the Gmail drafts folder.
>> Git complains about white space when I apply the downloaded patch. It
>> works fine if I just use git to create the patch and then apply it on
>> a new branch. Is it git imap-send or just Gmail that's the problem?
>
> How do you download and apply the patch exactly? If you are speaking
> imap to gmail, generally the client would strip out the CR's from the
> mail.

I'm just downloading it with Chrome.

Steps to reproduce:
1. Upload patch via:
$ git format-patch --stdout --keep-subject --attach origin | git imap-send
2. Open Gmail in Chrome.
3. Open email in drafts folder.
4. Click attachment download link
5. Apply patch on a fresh branch with git apply.

Git complains about the white space, which indicates that the
downloaded version has CRLF line endings.

I guess there's not much to do if the fault lies with Gmail?

Thanks for your reply.

Re: git imap-send converting my patches to CRLF line endings?

[Brandon Casey]

On 06/17/2011 09:45 AM, Michael Mc Donnell wrote:
> On Fri, Jun 17, 2011 at 4:14 PM, Jeff King <peff@peff.net> wrote:
>> On Fri, Jun 17, 2011 at 03:35:04PM +0200, Michael Mc Donnell wrote:

>> How do you download and apply the patch exactly? If you are speaking
>> imap to gmail, generally the client would strip out the CR's from the
>> mail.
> 
> I'm just downloading it with Chrome.
> 
> Steps to reproduce:
> 1. Upload patch via:
> $ git format-patch --stdout --keep-subject --attach origin | git imap-send
> 2. Open Gmail in Chrome.
> 3. Open email in drafts folder.
> 4. Click attachment download link
> 5. Apply patch on a fresh branch with git apply.
                                        ^^^^^^^^^

Ok, I suspected that.  The thing that you download from your gmail
drafts folder is an email, not a patch.  It may contain many inline
patches though.  You need to use 'git am' which will extract the
patches from the email and apply them.


A word of caution about using imap and gmail:

Unless something has changed recently, and I don't think it has, if you
_send_ the email using gmail's web interface, it will add newlines at
the 72nd character, and corrupt your patch.  So, even though you
uploaded the patch using 'git imap-send', you still have to select it
from your drafts folder and click "send" from gmail's web interface.
So, gmail's imap interface is pretty useless for sending patches.  You
should be able to use 'git send-email' and configure gmail as your smtp
server though. 

-Brandon

Re: git imap-send converting my patches to CRLF line endings?

[Brandon Casey]

On 06/17/2011 10:08 AM, Brandon Casey wrote:
> On 06/17/2011 09:45 AM, Michael Mc Donnell wrote:
>> On Fri, Jun 17, 2011 at 4:14 PM, Jeff King <peff@peff.net> wrote:
>>> On Fri, Jun 17, 2011 at 03:35:04PM +0200, Michael Mc Donnell wrote:
> 
>>> How do you download and apply the patch exactly? If you are speaking
>>> imap to gmail, generally the client would strip out the CR's from the
>>> mail.
>>
>> I'm just downloading it with Chrome.
>>
>> Steps to reproduce:
>> 1. Upload patch via:
>> $ git format-patch --stdout --keep-subject --attach origin | git imap-send

Wait a second.  You used --attach.

>> 2. Open Gmail in Chrome.
>> 3. Open email in drafts folder.
>> 4. Click attachment download link

Then you downloaded the attachment, which should be a _patch_.

>> 5. Apply patch on a fresh branch with git apply.

Well, scratch what I said before, you were correct in using
git apply.

Shouldn't the attachment have it's content preserved exactly?  Maybe
the fault does belong to gmail.

-Brandon

Re: git imap-send converting my patches to CRLF line endings?

[Jeff King]

On Fri, Jun 17, 2011 at 10:37:54AM -0500, Brandon Casey wrote:

> >> $ git format-patch --stdout --keep-subject --attach origin | git imap-send
> 
> Wait a second.  You used --attach.
> 
> >> 2. Open Gmail in Chrome.
> >> 3. Open email in drafts folder.
> >> 4. Click attachment download link
> 
> Then you downloaded the attachment, which should be a _patch_.

Yeah, but if it is text/*, then according to rfc2046, it must be
represented with CRLF as the line break. And especially if we are
including it unencoded in a message, it is going to need CR's added.

> >> 5. Apply patch on a fresh branch with git apply.
> 
> Well, scratch what I said before, you were correct in using
> git apply.
>
> Shouldn't the attachment have it's content preserved exactly?  Maybe
> the fault does belong to gmail.

Is it gmail's fault, or the browser's?  If gmail is handing back a
text/* content-type, then my reading of rfc2046 is that it should have
CRLF line breaks.  And it would be the browser's responsibility to
convert to native line endings.  But that's the MIME spec, and was
written with mail in mind; I don't know what's normal for HTTP in these
situations. But if the problem is not "strip CR" but "convert to native
line endings" (which I think it is), then how could gmail know the
user's native line ending preference, anyway?

-Peff

Re: git imap-send converting my patches to CRLF line endings?

[Brandon Casey]

On 06/17/2011 10:50 AM, Jeff King wrote:
> On Fri, Jun 17, 2011 at 10:37:54AM -0500, Brandon Casey wrote:
> 
>>>> $ git format-patch --stdout --keep-subject --attach origin | git imap-send
>>
>> Wait a second.  You used --attach.
>>
>>>> 2. Open Gmail in Chrome.
>>>> 3. Open email in drafts folder.
>>>> 4. Click attachment download link
>>
>> Then you downloaded the attachment, which should be a _patch_.
> 
> Yeah, but if it is text/*,

It is.

> then according to rfc2046, it must be
> represented with CRLF as the line break. And especially if we are
> including it unencoded in a message, it is going to need CR's added.
> 
>>>> 5. Apply patch on a fresh branch with git apply.
>>
>> Well, scratch what I said before, you were correct in using
>> git apply.
>>
>> Shouldn't the attachment have it's content preserved exactly?  Maybe
>> the fault does belong to gmail.
> 
> Is it gmail's fault, or the browser's?  If gmail is handing back a
> text/* content-type, then my reading of rfc2046 is that it should have
> CRLF line breaks.  And it would be the browser's responsibility to
> convert to native line endings.  But that's the MIME spec, and was
> written with mail in mind; I don't know what's normal for HTTP in these
> situations. But if the problem is not "strip CR" but "convert to native
> line endings" (which I think it is), then how could gmail know the
> user's native line ending preference, anyway?

So it's the same issue of line ending ambiguity that affects patches
sent inline in the body of the email message.  What we really want
is the _original_ line ending, not necessarily the native line ending
of the platform, but since any text/* content returned from or sent
to the mail server must have CRLF line endings, it is impossible to
determine whether or not the original content really had LF line
endings or not.  Currently, mailsplit chooses to assume the original
line ending was LF, based on the assumption that that's the line
ending that most projects use.

There doesn't seem to be any advantage to using --attach then, over
just including the patch inline.  Maybe attachments should always be
base64 encoded?  I get the eerie feeling that this topic has already
been hashed to death.

-Brandon

Re: git imap-send converting my patches to CRLF line endings?

[Michael Mc Donnell]

On Fri, Jun 17, 2011 at 6:54 PM, Brandon Casey
<brandon.casey.ctr@nrlssc.navy.mil> wrote:
> On 06/17/2011 10:50 AM, Jeff King wrote:
>> On Fri, Jun 17, 2011 at 10:37:54AM -0500, Brandon Casey wrote:
>>
>>>>> $ git format-patch --stdout --keep-subject --attach origin | git imap-send
>>>
>>> Wait a second.  You used --attach.
>>>
>>>>> 2. Open Gmail in Chrome.
>>>>> 3. Open email in drafts folder.
>>>>> 4. Click attachment download link
>>>
>>> Then you downloaded the attachment, which should be a _patch_.
>>
>> Yeah, but if it is text/*,
>
> It is.
>
>> then according to rfc2046, it must be
>> represented with CRLF as the line break. And especially if we are
>> including it unencoded in a message, it is going to need CR's added.
>>
>>>>> 5. Apply patch on a fresh branch with git apply.
>>>
>>> Well, scratch what I said before, you were correct in using
>>> git apply.
>>>
>>> Shouldn't the attachment have it's content preserved exactly?  Maybe
>>> the fault does belong to gmail.
>>
>> Is it gmail's fault, or the browser's?  If gmail is handing back a
>> text/* content-type, then my reading of rfc2046 is that it should have
>> CRLF line breaks.  And it would be the browser's responsibility to
>> convert to native line endings.  But that's the MIME spec, and was
>> written with mail in mind; I don't know what's normal for HTTP in these
>> situations. But if the problem is not "strip CR" but "convert to native
>> line endings" (which I think it is), then how could gmail know the
>> user's native line ending preference, anyway?
>
> So it's the same issue of line ending ambiguity that affects patches
> sent inline in the body of the email message.  What we really want
> is the _original_ line ending, not necessarily the native line ending
> of the platform, but since any text/* content returned from or sent
> to the mail server must have CRLF line endings, it is impossible to
> determine whether or not the original content really had LF line
> endings or not.  Currently, mailsplit chooses to assume the original
> line ending was LF, based on the assumption that that's the line
> ending that most projects use.
>
> There doesn't seem to be any advantage to using --attach then, over
> just including the patch inline.  Maybe attachments should always be
> base64 encoded?  I get the eerie feeling that this topic has already
> been hashed to death.

It sounds like its a big job to implement all those RFCs correctly. Is
there a library that could be used for handling imap?

Re: git imap-send converting my patches to CRLF line endings?

[Brandon Casey]

On 06/17/2011 09:14 AM, Jeff King wrote:
> On Fri, Jun 17, 2011 at 03:35:04PM +0200, Michael Mc Donnell wrote:
> 
>> I'm using git imap-send to send patches to wine-patches, and it seems
>> like it converts all my patches to have CRLF line endings?
> 
> The canonical line ending for mail is CRLF. So yes, it will convert your
> patch to CRLF for storage. But anything pulling it out of the IMAP
> folder should convert it back to native line endings.

Not always.  Modern thunderbird (3.1.10, is that modern? I haven't
checked), saves mail using CRLF.  I don't have access to gmail at the
moment, but I'm pretty sure gmail does the same thing, i.e. when you
select "view original", and then use your browser to "save as...".

mailsplit was modified to strip CRLF when splitting mail here:

  c2ca1d7 Allow mailsplit (and hence git-am) to handle mails with CRLF line-endings

which should have first appeared in git v1.6.5.

>> I can see it when I download the patch from the Gmail drafts folder.
>> Git complains about white space when I apply the downloaded patch. It
>> works fine if I just use git to create the patch and then apply it on
>> a new branch. Is it git imap-send or just Gmail that's the problem?
> 
> How do you download and apply the patch exactly? If you are speaking
> imap to gmail, generally the client would strip out the CR's from the
> mail.

Michael, how are you applying the "email"?  Are you using 'git am'? or
possibly are you trying to use 'git apply'?  You need to use 'git am'.

Also, as I mentioned above, you should be using git more recent than
v1.6.5 so that you have a mailsplit that will strip out the CRLF line
endings.

-Brandon

Re: git imap-send converting my patches to CRLF line endings?

[Jeff King]

On Fri, Jun 17, 2011 at 09:47:17AM -0500, Brandon Casey wrote:

> > The canonical line ending for mail is CRLF. So yes, it will convert your
> > patch to CRLF for storage. But anything pulling it out of the IMAP
> > folder should convert it back to native line endings.
> 
> Not always.  Modern thunderbird (3.1.10, is that modern? I haven't
> checked), saves mail using CRLF.  I don't have access to gmail at the
> moment, but I'm pretty sure gmail does the same thing, i.e. when you
> select "view original", and then use your browser to "save as...".
> 
> mailsplit was modified to strip CRLF when splitting mail here:
> 
>   c2ca1d7 Allow mailsplit (and hence git-am) to handle mails with CRLF line-endings
> 
> which should have first appeared in git v1.6.5.

Ah, I forgot about that. I am used to unix-y tools like mutt. But it is
obviously sensible for git to accept canonical mail via am, given that
some clients produce it.

> > How do you download and apply the patch exactly? If you are speaking
> > imap to gmail, generally the client would strip out the CR's from the
> > mail.
> 
> Michael, how are you applying the "email"?  Are you using 'git am'? or
> possibly are you trying to use 'git apply'?  You need to use 'git am'.

In another reply that crossed paths with yours, he wrote:

  5. Apply patch on a fresh branch with git apply.

So yeah, I think that is the problem.

-Peff