[mlmmj] 8BITMIME

[mlmmj] 8BITMIME

[Paul Fariello]

Hi all,

first I'd like to thank you for the great work you have done on mlmmj. It's
really joyful to use it !

Unfortunately I have an issue with some of my mail that contains
utf-8 chararcters.

My MTA (OpenSMTPd) is masking data with 0x7f when mail are not coming with the
8BITMIME flag. And it seems that mlmmj is not using the 8BITMIME EHLO command
when sending mail to the list. So it ends up with messy characters like "C'"
instead of "ç".

I'm not aware of all specificity of 8BITMIME, may be my mail are missing some
header. Here is an example of the mail that fails :


> Delivered-To: mlmmj-test@lists.paulfariello.fr
> Received: from chimay.paulfariello.fr (chimay.paulfariello.fr [109.190.142.210]);
> 	by chimay.paulfariello.fr (OpenSMTPD) with ESMTPSA id 2e35f84e;
> 	TLS version=TLSv1/SSLv3 cipherìDHE-RSA-AES256-GCM-SHA384 bits%6 verify=NO;
> 	for <mlmmj-test@lists.paulfariello.fr>;
> 	Tue, 5 Nov 2013 10:30:32 +0100 (CET)
> Date: Tue, 5 Nov 2013 10:30:31 +0100
> From: Paul Fariello <paul@fariello.eu>
> To: mlmmj-test@lists.paulfariello.fr
> Subject: Re: [Test] test mlmmj-send
> Message-ID: <20131105093031.GM11270@chimay.paulfariello.fr>
> References: <20131105091721.GB11270@chimay.paulfariello.fr>
> List-Id: <test.lists.paulfariello.fr>
> List-Post: <mailto:mlmmj-test@lists.paulfariello.fr>
> List-Help: <mailto:mlmmj-test+help@lists.paulfariello.fr>
> List-Subscribe: <mailto:mlmmj-test+subscribe@lists.paulfariello.fr>
> List-Unsubscribe: <mailto:mlmmj-test+unsubscribe@lists.paulfariello.fr>
> List-Owner: <mailto:mlmmj-test+owner@lists.paulfariello.fr>
> MIME-Version: 1.0
> Content-Type: text/plain; charset=utf-8
> Content-Disposition: inline
> Content-Transfer-Encoding: 8bit
> In-Reply-To: <20131105091721.GB11270@chimay.paulfariello.fr>
> User-Agent: Mutt/1.5.21 (2010-09-15)
> 
> ça va ?

Any idea of what I should do ? Patch mlmnj to set "EHLO" is an option ?

Regards,

-- 
Paul Fariello

[mlmmj] Re: 8BITMIME

[Paul Fariello]

[-- Attachment #1: Type: text/plain, Size: 2224 bytes --]

On Tue, Nov 05, 2013 at 02:35:45PM +0100, Paul Fariello wrote:
> Hi all,
> 
> first I'd like to thank you for the great work you have done on mlmmj. It's
> really joyful to use it !
> 
> Unfortunately I have an issue with some of my mail that contains
> utf-8 chararcters.
> 
> My MTA (OpenSMTPd) is masking data with 0x7f when mail are not coming with the
> 8BITMIME flag. And it seems that mlmmj is not using the 8BITMIME EHLO command
> when sending mail to the list. So it ends up with messy characters like "C'"
> instead of "ç".
> 
> I'm not aware of all specificity of 8BITMIME, may be my mail are missing some
> header. Here is an example of the mail that fails :
> 
> 
> > Delivered-To: mlmmj-test@lists.paulfariello.fr
> > Received: from chimay.paulfariello.fr (chimay.paulfariello.fr [109.190.142.210]);
> > 	by chimay.paulfariello.fr (OpenSMTPD) with ESMTPSA id 2e35f84e;
> > 	TLS version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO;
> > 	for <mlmmj-test@lists.paulfariello.fr>;
> > 	Tue, 5 Nov 2013 10:30:32 +0100 (CET)
> > Date: Tue, 5 Nov 2013 10:30:31 +0100
> > From: Paul Fariello <paul@fariello.eu>
> > To: mlmmj-test@lists.paulfariello.fr
> > Subject: Re: [Test] test mlmmj-send
> > Message-ID: <20131105093031.GM11270@chimay.paulfariello.fr>
> > References: <20131105091721.GB11270@chimay.paulfariello.fr>
> > List-Id: <test.lists.paulfariello.fr>
> > List-Post: <mailto:mlmmj-test@lists.paulfariello.fr>
> > List-Help: <mailto:mlmmj-test+help@lists.paulfariello.fr>
> > List-Subscribe: <mailto:mlmmj-test+subscribe@lists.paulfariello.fr>
> > List-Unsubscribe: <mailto:mlmmj-test+unsubscribe@lists.paulfariello.fr>
> > List-Owner: <mailto:mlmmj-test+owner@lists.paulfariello.fr>
> > MIME-Version: 1.0
> > Content-Type: text/plain; charset=utf-8
> > Content-Disposition: inline
> > Content-Transfer-Encoding: 8bit
> > In-Reply-To: <20131105091721.GB11270@chimay.paulfariello.fr>
> > User-Agent: Mutt/1.5.21 (2010-09-15)
> > 
> > ça va ?
> 
> Any idea of what I should do ? Patch mlmnj to set "EHLO" is an option ?
> 
> Regards,
> 
> -- 
> Paul Fariello

Finally, I decided to patch mlmmj to add minimalistic support for 8BITMIME.

Here are my patches.

Regards,

-- 
Paul Fariello

[-- Attachment #2: patch-src_mail-function_c --]
[-- Type: text/plain, Size: 358 bytes --]

--- src/mail-functions.c.orig	Tue Nov 19 09:49:33 2013
+++ src/mail-functions.c	Tue Nov 19 09:49:52 2013
@@ -47,7 +47,7 @@
 	
 	if((helo = mymalloc(len)) == 0)
 		return errno;
-	snprintf(helo, len, "HELO %s\r\n", hostname);
+	snprintf(helo, len, "EHLO %s\r\n", hostname);
 	len = strlen(helo);
 #if 0
 	fprintf(stderr, "\nwrite_helo, helo = [%s]\n", helo);

[-- Attachment #3: patch-src_checkwait_smtpreply_c --]
[-- Type: text/plain, Size: 444 bytes --]

--- src/checkwait_smtpreply.c.orig	Tue Nov 19 14:58:41 2013
+++ src/checkwait_smtpreply.c	Tue Nov 19 14:59:08 2013
@@ -35,7 +35,11 @@
 {
 	char *smtpreply;
 
-	smtpreply = mygetline(sockfd);
+	// Consume all 8BITMIME XXX- reply
+	do {
+		smtpreply = mygetline(sockfd);
+	} while (strlen(smtpreply) > 3 && smtpreply[3] == '-');
+
 #if 0
 	printf("replytype = [%d], smtpreply = [%s]\n", replytype, smtpreply);
 	fprintf(stderr, "%s", smtpreply);

Re: [mlmmj] Re: 8BITMIME

[Ben Schmidt]

>> My MTA (OpenSMTPd) is masking data with 0x7f when mail are not coming
>> with the 8BITMIME flag. And it seems that mlmmj is not using the
>> 8BITMIME EHLO command when sending mail to the list. So it ends up
>> with messy characters like "C'" instead of "รง".
>>
>> I'm not aware of all specificity of 8BITMIME, may be my mail are
>> missing some header. Here is an example of the mail that fails :

[snip]

Your mail headers, etc. look fine at a quick glance.

> Finally, I decided to patch mlmmj to add minimalistic support for
> 8BITMIME.

Thank you for investing the time to look into this and contribute.

I haven't had time to investigate this fully, which is why I have
offered no reply to your earlier email.

It is awkward, because the specs for this kind of thing are not well
implemented. Out in the wild, things don't work the way the spec says
they should, and best practice often isn't what the spec says.

Consequently, I have a few questions for you.

1. Could you please open an enhancement request on the bug tracker
detailing this issue and attach your patch to it? This will help it not
to get lost/forgotten. If you could put your answers to these questions
in the request, too, that would be helpful.

2. You only patched HELO to be EHLO, and nothing else. In particular,
you didn't include the 8BITMIME 'tag' in the MAIL FROM line that the
spec requires. Nevertheless, can you confirm that you found it was
sufficient to just use EHLO for your MTA?

3. As you say, this is a minimalist patch. Would you be willing to put
in some time to improve it? Here are some items to address:

- You have introduced a memory leak, because mygetline() allocates
   memory which you don't free with myfree() (yes, I know this really
   needs to be documented...). Since mlmmj-send may make many connections
   to SMTP servers during a run, this could be significant.

- Just in case some server out there still implements plain SMTP (and
   there are probably a few, particularly special-use ones, e.g. for
   proxying), we shouldn't offer only EHLO. The program should check the
   error condition, and upon receiving a 500, should fall back to HELO
   (in line with the spec!).

- The responsees from EHLO should always begin with "250-" until the
   last; doing strncmp with that would be more efficient than using
   strlen (on the whole string) and then checking a character.
   Negligible, I know, but why not do it a little more efficiently? It's
   a more thorough check, too, which can't hurt.

Thanks again for your work.

Best regards,

Ben.

Re: [mlmmj] Re: 8BITMIME

[Paul Fariello]

Thank for your quick reply.

On Thu, Nov 21, 2013 at 08:22:35AM +1100, Ben Schmidt wrote:
> >>My MTA (OpenSMTPd) is masking data with 0x7f when mail are not coming
> >>with the 8BITMIME flag. And it seems that mlmmj is not using the
> >>8BITMIME EHLO command when sending mail to the list. So it ends up
> >>with messy characters like "C'" instead of "รง".
> >>
> >>I'm not aware of all specificity of 8BITMIME, may be my mail are
> >>missing some header. Here is an example of the mail that fails :
> 
> [snip]
> 
> Your mail headers, etc. look fine at a quick glance.
> 
> >Finally, I decided to patch mlmmj to add minimalistic support for
> >8BITMIME.
> 
> Thank you for investing the time to look into this and contribute.
> 
> I haven't had time to investigate this fully, which is why I have
> offered no reply to your earlier email.
> 
> It is awkward, because the specs for this kind of thing are not well
> implemented. Out in the wild, things don't work the way the spec says
> they should, and best practice often isn't what the spec says.
> 
> Consequently, I have a few questions for you.
> 
> 1. Could you please open an enhancement request on the bug tracker
> detailing this issue and attach your patch to it? This will help it not
> to get lost/forgotten. If you could put your answers to these questions
> in the request, too, that would be helpful.

You can find the corresponding enhancement request at :
http://mlmmj.org/bugs/bug.php?idI

> 2. You only patched HELO to be EHLO, and nothing else. In particular,
> you didn't include the 8BITMIME 'tag' in the MAIL FROM line that the
> spec requires. Nevertheless, can you confirm that you found it was
> sufficient to just use EHLO for your MTA?

I confirm that my patch is sufficient.

> 3. As you say, this is a minimalist patch. Would you be willing to put
> in some time to improve it? Here are some items to address:

Of course :)

> - You have introduced a memory leak, because mygetline() allocates
>   memory which you don't free with myfree() (yes, I know this really
>   needs to be documented...). Since mlmmj-send may make many connections
>   to SMTP servers during a run, this could be significant.

Will be addressed in next patch.

> - Just in case some server out there still implements plain SMTP (and
>   there are probably a few, particularly special-use ones, e.g. for
>   proxying), we shouldn't offer only EHLO. The program should check the
>   error condition, and upon receiving a 500, should fall back to HELO
>   (in line with the spec!).

I'll try to do that. Do you now a public MTA that only support HELO command so I
can test it.

> - The responsees from EHLO should always begin with "250-" until the
>   last; doing strncmp with that would be more efficient than using
>   strlen (on the whole string) and then checking a character.
>   Negligible, I know, but why not do it a little more efficiently? It's
>   a more thorough check, too, which can't hurt.

You are right, I'll improve it in next patch.

Regards,

-- 
Paul Fariello