[patatt][PATCH v2] Handle MIME encoded-word & other header manglings

[patatt][PATCH v2] Handle MIME encoded-word & other header manglings

[Paul Barker]

When testing patatt with patches sent to a sr.ht hosted mailing list, it
was found that long header lines (such as the X-Developer-Signature
line) were re-encoded using the MIME encoded-word syntax (RFC 2047) when
an mbox archive is generated, causing patatt to choke on the resulting
text which looks like this:

    X-Developer-Signature: v=1; a=openpgp-sha256; l=672; h=from:subject;
     bh=C40yOKgIfnNIUP+OW9WyPdBfljkZPpfUL1NepOODlx8=; =?utf-8?q?b=3DowGbwMvMwCF2?=
     =?utf-8?q?w7xIXuiX9CvG02pJDAmb67lTNi0+IeF97TL76vtKD7xjSjaluz0o/KfmZLX8rMi7_?=
     =?utf-8?q?l3M6O0pZGMQ4GGTFFFl2z951+fqDJVt7b0gHw8xhZQIZwsDFKQATydFhZJi+fFfvJ?=
     =?utf-8?q?8+0MF7GrfzWnP?=
     K7mAM/3n/r/UC+bprf6/g114QYGdbHcsaK7b1nanfA4IeZi1V0lL26cruXUWxgSEnNDP1FrAA=

Avoiding this issue by neatly wrapping the X-Developer-Signature header
before sending doesn't appear to be possible without making invasive
changes to git-send-email and/or the Net::SMTP perl module. The header
content generated by patatt is wrapped at 78 characters as can be seen
here from a locally signed patch file:

    X-Developer-Signature: v=1; a=openpgp-sha256; l=672; h=from:subject;
    bh=C40yOKgIfnNIUP+OW9WyPdBfljkZPpfUL1NepOODlx8=;
    b=owGbwMvMwCF2w7xIXuiX9CvG02pJDAmbN1xO2bT4hIT3tcvsq+8rPfCOKdmU7vag8J+ak9XysyLv
    Xs7p7ChlYRDjYJAVU2TZPXvX5esPlmztvSEdDDOHlQlkCAMXpwBMpG0Dw/9Kpzgpc8UsQwOPK/taW6
    dFnZyy5QlXPfNCC4WTc76ft9ZnZJjI37a17fP7sxvclKJ1tm36EhITcK62Pphje9KrmOxMJg4A

Running `git send-email --smtp-debug=1 0001.patch` shows that this is
joined into a single long line before the message is sent:

    Net::SMTP::_SSL=GLOB(0x5646fbdc3ac8)>>> X-Developer-Signature: v=1; a=openpgp-sha256; l=672; h=from:subject; bh=C40yOKgIfnNIUP+OW9WyPdBfljkZPpfUL1NepOODlx8=; b=owGbwMvMwCF2w7xIXuiX9CvG02pJDAmb571P2bT4hIT3tcvsq+8rPfCOKdmU7vag8J+ak9XysyLv Xs7p7ChlYRDjYJAVU2TZPXvX5esPlmztvSEdDDOHlQlkCAMXpwBM5JA3I8O5hP6Tqm7lJst0rldcux 1V7M4q8T5o1fPU6Zs+hxj+SjvN8D/DK3rn8b0m34/Xy388Yeu8jvFdJf/c6Y6LDU7Hulj01nAAAA==

So we need to accept that the X-Developer-Signature line may be quite
long and so may be re-encoded by a mail server or archiver.

The Python email.header module provides the decode_header() and
make_header() functions which can be used to handle MIME encoded-word
syntax or other header manglings which may occur. The decode_header()
function requires a str argument so we must decode our bytes before
using this function. Thankfully, RFC 2822 makes life easy here as it
says that all header content must be composed of US-ASCII characters
(see section 2.2 of the RFC) so decoding is straightforward. The header
content is re-encoded into bytes after un-mangling to avoid having to
modify every other location in patatt where the header content is
accessed.

Signed-off-by: Paul Barker <paul@pbarker.dev>
---
v1 -> v2:
    * Avoid unnecessary header decoding/parsing/re-encoding if it's not
      needed.
    * Ignore errors when decoding the header content into an ASCII
      string.
    * Tidy up the comment so things are easier to read.

 patatt/__init__.py | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/patatt/__init__.py b/patatt/__init__.py
index 460d282..b4018ab 100644
--- a/patatt/__init__.py
+++ b/patatt/__init__.py
@@ -91,7 +91,7 @@ class DevsigHeader:
 
     def from_bytes(self, hval: bytes) -> None:
         self.hval = DevsigHeader._dkim_canonicalize_header(hval)
-        hval = re.sub(rb'\s*', b'', hval)
+        hval = re.sub(rb'\s*', b'', self.hval)
         for chunk in hval.split(b';'):
             parts = chunk.split(b'=', 1)
             if len(parts) < 2:
@@ -392,6 +392,15 @@ class DevsigHeader:
 
     @staticmethod
     def _dkim_canonicalize_header(hval: bytes) -> bytes:
+        # Handle MIME encoded-word syntax or other types of header encoding if
+        # present. The decode_header() function requires a str argument (not
+        # bytes) so we must decode our bytes first, this is easy as RFC2822 (sec
+        # 2.2) says header fields must be composed of US-ASCII characters. The
+        # resulting string is re-encoded to allow further processing.
+        if b'?q?' in hval:
+            hval = hval.decode('ascii', errors='ignore')
+            hval = str(email.header.make_header(email.header.decode_header(hval)))
+            hval = hval.encode('utf-8')
         # We only do relaxed for headers
         #    o  Unfold all header field continuation lines as described in
         #       [RFC5322]; in particular, lines with terminators embedded in
-- 
2.31.1

Re: [patatt][PATCH v2] Handle MIME encoded-word & other header manglings

[Konstantin Ryabitsev]

On Mon, 31 May 2021 15:05:39 +0100, Paul Barker wrote:
> When testing patatt with patches sent to a sr.ht hosted mailing list, it
> was found that long header lines (such as the X-Developer-Signature
> line) were re-encoded using the MIME encoded-word syntax (RFC 2047) when
> an mbox archive is generated, causing patatt to choke on the resulting
> text which looks like this:
> 
>     X-Developer-Signature: v=1; a=openpgp-sha256; l=672; h=from:subject;
>      bh=C40yOKgIfnNIUP+OW9WyPdBfljkZPpfUL1NepOODlx8=; =?utf-8?q?b=3DowGbwMvMwCF2?=
>      =?utf-8?q?w7xIXuiX9CvG02pJDAmb67lTNi0+IeF97TL76vtKD7xjSjaluz0o/KfmZLX8rMi7_?=
>      =?utf-8?q?l3M6O0pZGMQ4GGTFFFl2z951+fqDJVt7b0gHw8xhZQIZwsDFKQATydFhZJi+fFfvJ?=
>      =?utf-8?q?8+0MF7GrfzWnP?=
>      K7mAM/3n/r/UC+bprf6/g114QYGdbHcsaK7b1nanfA4IeZi1V0lL26cruXUWxgSEnNDP1FrAA=
> 
> [...]

Applied, thanks!

[1/1] Handle MIME encoded-word & other header manglings
      commit: 30840a6acae935ebc5332d08d61222cebe0b518b

Best regards,
-- 
Konstantin Ryabitsev <konstantin.ryabitsev@linux.dev>