[PATCH b4] Reorder headers for `git imap-send` compatibility

[PATCH b4] Reorder headers for `git imap-send` compatibility

[Philippe Blain]

When downloading messages from a public-inbox instance, reorder the
"From", "Date" and "Subject" headers in that specific order. This allows
the output from 'b4 mbox -o- $url' to be piped to 'git imap-send', which
expects these headers in that specific order.

An easy place to implement that is in 'mailsplit_bytes', which is called
by 'get_pi_thread_by_url' and already loops over the messages in the 'git
mailsplit'-ed mbox file to populate the message list.

Note that 'mailsplit_bytes' is also called by 'get_msgs' directly when
piping in a local mailbox, i.e. when '-' is used as the value of
'--use-local-mbox', but it does no harm to also reorder the headers in
that case.

Signed-off-by: Philippe Blain <levraiphilippeblain@gmail.com>
---

Notes:
    Hi, I suggested that feature in [1] last December and just recently took the
    time to implement it.
    
    Konstantin suggested in his response to [1] to try using isync/mbsync instead,
    but I would prefer not having to change my workflow.  I've been using 'git
    imap-send' for a while with my custom script [2] when hitting emails that were
    not created with 'git send-email'/'git format-patch' and I did not hit any
    corner case where 'git imap-send' was not able to send the email.
    
    [1] https://lore.kernel.org/tools/7e6329fa-e5fa-c5af-002d-a8fbb60e0724@gmail.com/
    [2] https://gist.github.com/phil-blain/d350e91959efa6e7afce60e74bf7e4a8

 b4/__init__.py | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/b4/__init__.py b/b4/__init__.py
index b45e26f..baea003 100644
--- a/b4/__init__.py
+++ b/b4/__init__.py
@@ -2162,6 +2162,13 @@ def get_strict_thread(msgs, msgid):
 
     return strict
 
+def reorder_headers(msg):
+    """ Reorder "From", "Date" and "Subject" headers in the order `git imap-send` expects"""
+    for header in ['From', 'Date', 'Subject']:
+        header_content = msg[header]
+        del msg[header]
+        msg[header] = header_content
+    return msg
 
 def mailsplit_bytes(bmbox: bytes, outdir: str) -> list:
     logger.debug('Mailsplitting the mbox into %s', outdir)
@@ -2174,7 +2181,7 @@ def mailsplit_bytes(bmbox: bytes, outdir: str) -> list:
     # Read in the files
     for msg in os.listdir(outdir):
         with open(os.path.join(outdir, msg), 'rb') as fh:
-            msgs.append(email.message_from_binary_file(fh))
+            msgs.append(reorder_headers(email.message_from_binary_file(fh)))
     return msgs
 
 

base-commit: 45ef591d2fb245a06600514d8df57ab64d519fd8
-- 
2.29.2

Re: [PATCH b4] Reorder headers for `git imap-send` compatibility

[Konstantin Ryabitsev]

On Tue, Aug 10, 2021 at 10:41:32PM -0400, Philippe Blain wrote:
> When downloading messages from a public-inbox instance, reorder the
> "From", "Date" and "Subject" headers in that specific order. This allows
> the output from 'b4 mbox -o- $url' to be piped to 'git imap-send', which
> expects these headers in that specific order.
> 
> An easy place to implement that is in 'mailsplit_bytes', which is called
> by 'get_pi_thread_by_url' and already loops over the messages in the 'git
> mailsplit'-ed mbox file to populate the message list.
> 
> Note that 'mailsplit_bytes' is also called by 'get_msgs' directly when
> piping in a local mailbox, i.e. when '-' is used as the value of
> '--use-local-mbox', but it does no harm to also reorder the headers in
> that case.

I've decided not to take this patch because it will rearrange headers on all
processed messages just to work around a deficiency in a single tool
(git-imap-send). The order of headers in an RFC-2822 message should not
matter, so any tool that expects them in a specific arrangement is what needs
fixing.

Furthermore, reordering headers may actually cause DKIM validation errors in
the case where there are multiple From, Date or Subject headers. I know this
sounds contrived, but I've learned not to consider anything too bizarre of a
corner case when it comes to email.

For your particular case, I suggest you simply make a tiny python command that
takes an mbox on stdin and spits it out on stdout with headers in the desired
order, which should be about 20 lines of Python at most. This way you can do
`b4 mbox -o- $url | reorder.py | git imap-send`.

-K

Re: [PATCH b4] Reorder headers for `git imap-send` compatibility

[Philippe Blain]

Hi Konstantin,

Le 2021-08-26 à 15:40, Konstantin Ryabitsev a écrit :
> On Tue, Aug 10, 2021 at 10:41:32PM -0400, Philippe Blain wrote:
>> When downloading messages from a public-inbox instance, reorder the
>> "From", "Date" and "Subject" headers in that specific order. This allows
>> the output from 'b4 mbox -o- $url' to be piped to 'git imap-send', which
>> expects these headers in that specific order.
>>
>> An easy place to implement that is in 'mailsplit_bytes', which is called
>> by 'get_pi_thread_by_url' and already loops over the messages in the 'git
>> mailsplit'-ed mbox file to populate the message list.
>>
>> Note that 'mailsplit_bytes' is also called by 'get_msgs' directly when
>> piping in a local mailbox, i.e. when '-' is used as the value of
>> '--use-local-mbox', but it does no harm to also reorder the headers in
>> that case.
> 
> I've decided not to take this patch because it will rearrange headers on all
> processed messages just to work around a deficiency in a single tool
> (git-imap-send). The order of headers in an RFC-2822 message should not
> matter, so any tool that expects them in a specific arrangement is what needs
> fixing.
> 
> Furthermore, reordering headers may actually cause DKIM validation errors in
> the case where there are multiple From, Date or Subject headers. I know this
> sounds contrived, but I've learned not to consider anything too bizarre of a
> corner case when it comes to email.

Fair enough :)


> For your particular case, I suggest you simply make a tiny python command that
> takes an mbox on stdin and spits it out on stdout with headers in the desired
> order, which should be about 20 lines of Python at most. This way you can do
> `b4 mbox -o- $url | reorder.py | git imap-send`.

That is what I'll do!

Thanks, Philippe.