Re: footer isn't appended to multipart messages

Re: footer isn't appended to multipart messages

[Casey Allen Shobe]

On Tuesday 05 July 2005 15:10, Patrick Bennett wrote:
> Ahah!  You're right, the footer *is* there on the "Rich Text"
> formatted raw source, but is not appearing in the email viewer.
>  I assume this is because the footer is being appended in an
> inappropriate place/way for an HTML mail, so the email viewer is
> not interpretting the footer properly.  Will this be fixed?

Well, HTML mail is generally strongly discouraged on mailing lists.  
But it's not exactly "HTML mail", but any sort of MIME-compliant 
multipart message.  In multipart messages, the mail contains more 
than one part.  Typically that's one HTML version, and one text 
version of the message; but inconsiderate people send HTML-only 
messages, and multipart serves many other purposes.

> It isn't acceptable, IMHO, for a list recipient/subscriber to get 
> no ongoing notice of how to unsubscribe from the list... is
> there, perhaps, a subscription "reminder" function in mlmmj as
> there are in some other mailing list softwares?

There is only one thing more annoying on mailing lists than 
appending stupid user text to every single message, and that is the 
monthly password reminders that mailman sends out.  I get spammed 
with about 50 of them at about the same time every month, so I 
wrote a filter to automatically delete them to retain sanity.

(Exception:  your mailing lists are used for outgoing marketing 
purposes only, in which case you can add unsubscription 
instructions yourself since you're the only one sending mail.)

That being said, List-Subscribe, List-Unsubcribe, etc. headers 
should be present, and they don't appear to be on this list.  Does 
mlmmj support them yet?

(this is not to say that your feature request/bug report is invalid 
- it is in fact useful in some cases and should work as you expect, 
but the example you provided should be discouraged and it's also 
not the easiest thing to implement due to the varying nature of 
what the multipart messages might contain, how the HTML is 
structured, etc.)

Cheers,
-- 
Casey Allen Shobe | http://casey.shobe.info
cshobe@seattleserver.com | cell 425-443-4653
AIM & Yahoo:  SomeLinuxGuy | ICQ:  1494523
SeattleServer.com, Inc. | http://www.seattleserver.com

Re: footer isn't appended to multipart messages

[Sven 'Darkman' Michels]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Casey Allen Shobe wrote:
| There is only one thing more annoying on mailing lists than
| appending stupid user text to every single message, and that is the
| monthly password reminders that mailman sends out.  I get spammed
| with about 50 of them at about the same time every month, so I
| wrote a filter to automatically delete them to retain sanity.

Full ACK. I got flooded by them, too. Never mind, everything can
be filtered ;)


| That being said, List-Subscribe, List-Unsubcribe, etc. headers
| should be present, and they don't appear to be on this list.  Does
| mlmmj support them yet?

Well, yes and no. Yes: you can add them to Custom headers file,
No: they're not added automaticly (maybe this could be implemented
but i think it's ok if you run a lot of lists to generate the
customheaders file by a script or so ;)


| (this is not to say that your feature request/bug report is invalid
| - it is in fact useful in some cases and should work as you expect,
| but the example you provided should be discouraged and it's also
| not the easiest thing to implement due to the varying nature of
| what the multipart messages might contain, how the HTML is
| structured, etc.)

correctly. I think the 'append a footer' feature is ok, too. Just
disable html mail and stuff and it's nearly fine. The List- Headers
should be always present (not hard to implement ;)
Implementing appending of stuff for multipartmessages just suck
cause you have so many ways clients do 'their' encoding and
maybe some gateways who add shitty 'this is confidential mail'
footers, too, so you may end up in a 4 or more part mail which
only contains a one liner...

Regards,
Sven
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)

iD8DBQFCywUDQoCguWUBzBwRAtdcAJ9o5ozqcY/H69Sn0kJhvDekr/TcEwCeJESi
bM+5Uw0+deHZHAnN30gddRw=EHej
-----END PGP SIGNATURE-----

Re: footer isn't appended to multipart messages (part II: reality check!)

[Patrick Bennett]

----- Original Message ----- 
From: "Sven 'Darkman' Michels" <sven@darkman.de>
To: <mlmmj@mmj.dk>
Sent: Tuesday, July 05, 2005 3:09 PM
Subject: Re: footer isn't appended to multipart messages


> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Casey Allen Shobe wrote:
> | There is only one thing more annoying on mailing lists than
> | appending stupid user text to every single message, and that is the
> | monthly password reminders that mailman sends out.  I get spammed
> | with about 50 of them at about the same time every month, so I
> | wrote a filter to automatically delete them to retain sanity.
>
> Full ACK. I got flooded by them, too. Never mind, everything can
> be filtered ;)

This is absolutely true- you (that is to say, *we*, that is to say 
techie/programmer types) find this stuff very annoying.  However, mailing 
lists are used by knitting clubs too, and I, for one, DO NOT want the 
nightmare of administrating a knitting club mailing list that has absolutely 
no dependable notification of "how to do it yourself".


>
>
> | That being said, List-Subscribe, List-Unsubcribe, etc. headers
> | should be present, and they don't appear to be on this list.  Does
> | mlmmj support them yet?
>
> Well, yes and no. Yes: you can add them to Custom headers file,
> No: they're not added automaticly (maybe this could be implemented
> but i think it's ok if you run a lot of lists to generate the
> customheaders file by a script or so ;)
>
>
> | (this is not to say that your feature request/bug report is invalid
> | - it is in fact useful in some cases and should work as you expect,
> | but the example you provided should be discouraged and it's also
> | not the easiest thing to implement due to the varying nature of
> | what the multipart messages might contain, how the HTML is
> | structured, etc.)
>

not sure which "example" you mean, but if it's monthly nags (a.k.a. mailing 
list subscribe/unsubscribe howto's), then, yes, it should be discouraged 
*for appropriate lists and when there is no reliable way to append footer 
text*.  For other lists, on the contrary, it should be *encouraged* (see 
above)!


> correctly. I think the 'append a footer' feature is ok, too. Just
> disable html mail and stuff and it's nearly fine. The List- Headers
> should be always present (not hard to implement ;)
> Implementing appending of stuff for multipartmessages just suck
> cause you have so many ways clients do 'their' encoding and
> maybe some gateways who add shitty 'this is confidential mail'
> footers, too, so you may end up in a 4 or more part mail which
> only contains a one liner...
>

1)  I haven't found the documentation on how to enforce text-only mailing 
lists (it's one of the first bits of info i looked for when i figured out 
what the problem was; 2) not sure what happens on a text-only mlmmj list 
when a multi-part msg is sent, but whatever it is, for the "knitting club", 
it *can't* be a reject msg- a mailing list that fits the "knitting club's" 
needs (and I realize this may never be what mlmmj is, though I wish it could 
be, because I've very much liked the software otherwise) has to handle the 
default settings of widespread (though not necessarily popular with us 
programmer/techies) email clients (eg. outlook express).

Regards,
> Sven
>

    -Patrick 

Re: footer isn't appended to multipart messages (part II: reality check!)

[Casey Allen Shobe]

On Tuesday 05 July 2005 23:32, you wrote:
> This is absolutely true- you (that is to say, *we*, that is to
> say techie/programmer types) find this stuff very annoying. 
> However, mailing lists are used by knitting clubs too, and I, for
> one, DO NOT want the nightmare of administrating a knitting club
> mailing list that has absolutely no dependable notification of
> "how to do it yourself".

Reality check indeed:  knitting club members aren't going to have a 
clue wtf the reminder mail or appended text means.  They don't 
understand what "unsubscribing from a mailing list" is, they just 
use a particular address to talk to other club members.  If they 
want off the list, no matter how much you spam them with 
instructions, they will ask the head of the knitting club to do it.  
I'm on some lists which append unsubscription messages to every 
mail, and guess what I see posted there all the time?  empty 
messages with "unsubscribe" subjects (often with unsubscribe 
misspelled, not that it matters since that's not how the lists 
work).

Reality is, nontechy users have no interest in doing it themselves, 
and techy users know right well how to find out.  Most anyone in 
between will look up the website where they signed up to try to 
figure it out.

Also, even though Outhouse Express is not yet one of them, many mail 
clients have built-in support for List-Unsubscribe header which 
enables a button for unsubscribing from the list.

> not sure which "example" you mean

The example you asked about:  appending footers with 
subscribe/unsubscribe information.

> For other lists, on the contrary, it should be *encouraged* (see
> above)! 

I could not disagree with this statement more.  You really think 
that when I (the idiot user in this example) feel like 
unsubscribing that I'm going to wait around for the next monthly 
mail or go digging around for an old one?  No, I'm going to whine 
to the list owner or whomever I think is in charge if I can't 
figure things out, and probably just the whole list to make sure 
somebody hears me.

> 1)  I haven't found the documentation on how to enforce text-only
> mailing lists

You could filter incoming messages with something like maildrop or 
your own script.  Not sure how you'd go about mingling this with 
mlmmj, perhaps somebody else can answer that.

> 2) not sure what happens on a text-only mlmmj list when a
> multi-part msg is sent, but [it] has to handle the
> default settings of widespread [...] email clients [...].

Short answer, if you want HTML mail to reach the list, then don't 
block it.  If you care about people seeing signatures on every 
single mail (even the HTML ones), then don't allow HTML.  You can't 
(easily) have your cake and eat it too.

Long answer, you *can* have your cake and eat it too, but it will 
require substantial work on your part to reduce the messages (which 
I don't think you're actually willing to do or you might have just 
implemented multipart footer appending yourself):

* Outlook Express sends HTML and text multipart messages by default 
I believe.  You could reduce this to plain text with some parsing, 
removing the HTML part and leaving the plain text.

* At least Thunderbird sends HTML-only mail by default.  It's not 
that hard to go through and strip out HTML tags, since 99.9% of the 
time, end users never use any custom HTML anyways, and if they do, 
it's just a font or color and doesn't affect the legibility of the 
message (except for the worse).

* You will run into messages with lots of parts, where people think 
it's a good idea to tack on a VCF, HTML message, text message, 
separate parts for their spam filtering mechanism's signature, one 
for saying the mail passed their lame virus scan, and an image of 
the company's logo.  Good luck with those.

Realistic answer, chances are you will never make your knitting club 
members happy, because the majority of them cannot and will not 
understand what a mailing list even is.  You might come closest 
with the least effort by just going and signing up for yahoogroups, 
which even gives you the option to convert all incoming messages to 
HTML (*shudder*) and appends the dummy footer to every message, and 
gives them an ad-filled web-interface that they can use instead of 
email if they want to boot.

Another answer, if you just want a monthly/weekly/whatever reminder, 
just mail the list from a cron job.

Cheers,
-- 
Casey Allen Shobe | http://casey.shobe.info
cshobe@seattleserver.com | cell 425-443-4653
AIM & Yahoo:  SomeLinuxGuy | ICQ:  1494523
SeattleServer.com, Inc. | http://www.seattleserver.com

Fw: footer isn't appended to multipart messages (part II: reality check!)

[Patrick Bennett]

oops, forgot to cc the list. apologies.

    -Patrick

----- Original Message ----- 
From: "Patrick Bennett" <patrick@pebcomputing.com>
To: "Casey Allen Shobe" <lists@seattleserver.com>
Sent: Tuesday, July 05, 2005 11:14 PM
Subject: Re: footer isn't appended to multipart messages (part II: reality 
check!)


> Alright, first, not intending to have started a flame war whatsoever, 
> but.... well, anyway, hoping I'm being read in a tone that takes it down a 
> notch and injects a bit of humor....
>
> I think we (ie. "free/open source" software developers {you, I presume}, 
> administrators {me} and users {lots of oft strident and fanatic folk out 
> there}), often miss the boat by insisting our preferences must be foisted 
> as "the one true way" on all the other innocent users out there (and by 
> "users" I mean not just end-users, but purchasing decision makers, pointy 
> haired bosses, systems administrators, etc.).  This, imho, is one large 
> factor that hinders open source software adoption by "the stupid users", 
> the business world, pointy haired bosses, ad infinitum.  Don't we come off 
> as a bit kooky with such unbending sloganeering?  We do propose to create 
> in the bazaar, rather than the cathedral, no?  I thought the point was to 
> create high quality, highly flexible software and share it with the world 
> for everyone's mutual benefit..... or maybe I got that wrong and it is 
> instead to create a "free as in beer" fourth reich.
>
>
> ----- Original Message ----- 
> From: "Casey Allen Shobe" <lists@seattleserver.com>
> To: <mlmmj@mmj.dk>
> Sent: Tuesday, July 05, 2005 7:38 PM
> Subject: Re: footer isn't appended to multipart messages (part II: reality 
> check!)
>
>
>> On Tuesday 05 July 2005 23:32, you wrote:
>>> This is absolutely true- you (that is to say, *we*, that is to
>>> say techie/programmer types) find this stuff very annoying.
>>> However, mailing lists are used by knitting clubs too, and I, for
>>> one, DO NOT want the nightmare of administrating a knitting club
>>> mailing list that has absolutely no dependable notification of
>>> "how to do it yourself".
>>
>> Reality check indeed:  knitting club members aren't going to have a
>> clue wtf the reminder mail or appended text means.  They don't
>> understand what "unsubscribing from a mailing list" is, they just
>> use a particular address to talk to other club members.  If they
>> want off the list, no matter how much you spam them with
>> instructions, they will ask the head of the knitting club to do it.
>> I'm on some lists which append unsubscription messages to every
>> mail, and guess what I see posted there all the time?  empty
>> messages with "unsubscribe" subjects (often with unsubscribe
>> misspelled, not that it matters since that's not how the lists
>> work).
>>
>
> You're right, my hypothetical "knitting club" was probably too far on the 
> other end of the spectrum and thus a poor example, mea maxima culpa.  My 
> target users are actually in between- comfortable with technology, but not 
> professionals nor geeks.  They desire obviousness and convenience and more 
> or less easily understand and are eager to follow a terse "unsubscribe 
> footer" or regular reminder; but are frustrated by enforced ignorance. 
> The damn shame of wasting a few hundred bits of bandwidth a day on a 
> helpful (to them, but obviously not to you) footer, or a few hundred bits 
> of bandwidth a month on a helpful (to them, but obviously not to you) 
> how-to-be-a-good-mailing-listizen email is utterly lost on them.  Some of 
> them, only a few mind you, might even actually have a background jpg 
> embedded into their email.  {Gasp!}
>
>> Also, even though Outhouse Express is not yet one of them, many mail
>> clients have built-in support for List-Unsubscribe header which
>> enables a button for unsubscribing from the list.
>>
>
> I wasn't aware of this feature.  Cool.  I'll see if I can get it to work. 
> Any further info on how to implement this?
>
>>> not sure which "example" you mean
>>
>> The example you asked about:  appending footers with
>> subscribe/unsubscribe information.
>>
>>> For other lists, on the contrary, it should be *encouraged* (see
>>> above)!
>>
>> I could not disagree with this statement more.  You really think
>> that when I (the idiot user in this example) feel like
>> unsubscribing that I'm going to wait around for the next monthly
>> mail or go digging around for an old one?  No, I'm going to whine
>> to the list owner or whomever I think is in charge if I can't
>> figure things out, and probably just the whole list to make sure
>> somebody hears me.
>>
>>> 1)  I haven't found the documentation on how to enforce text-only
>>> mailing lists
>>
>> You could filter incoming messages with something like maildrop or
>> your own script.  Not sure how you'd go about mingling this with
>> mlmmj, perhaps somebody else can answer that.
>>
> I had inferred (apparently incorrectly; a thousand apologies) from 
> something another respondent had said that there was a mlmmj built-in 
> method for enforcing text-only posting.  Thus, my comment about not being 
> able to find any documentation on it.
>
>>> 2) not sure what happens on a text-only mlmmj list when a
>>> multi-part msg is sent, but [it] has to handle the
>>> default settings of widespread [...] email clients [...].
>>
>> Short answer, if you want HTML mail to reach the list, then don't
>> block it.  If you care about people seeing signatures on every
>> single mail (even the HTML ones), then don't allow HTML.  You can't
>> (easily) have your cake and eat it too.
>>
>> Long answer, you *can* have your cake and eat it too, but it will
>> require substantial work on your part to reduce the messages (which
>> I don't think you're actually willing to do or you might have just
>> implemented multipart footer appending yourself):
>>
> If I understand correctly, and I may well be mistaken, other mailing list 
> softwares out there in the open source world (gnu mailman comes to mind as 
> an example), are able to handle appending footer messages to a wide 
> variety of incoming email formats.  No suggestion that, coding wise, this 
> is accomplished *easily*, but just that it would be cool (and, in 
> particular, useful to *me*) if mlmmj could do it as well.  Besides, I 
> doubt a corrupted final email (that is to say, a multi-part mime mail with 
> a plain-text footer appended to it inappropriately) is the goal of the 
> mlmmj crafters.  If I were both 1) an experienced programmer, and 2) an 
> expert in email protocol, formats, etc., you're right, I could implement 
> this my own damned self, couldn't I?!?  As it is, I've just wasted every 
> one's time with my unworthyness and rubish opinions.
>
>
>> Another answer, if you just want a monthly/weekly/whatever reminder,
>> just mail the list from a cron job.
>>
>
> Now *there* is a useful suggestion that I might well steal.  Even though, 
> again, I couldn't find any documentation of it, I had just hoped and 
> half-way suspected (since it seems like a common feature, even if you 
> personally are offended by it, in many mailing list managers) that mlmmj 
> had something built-in to handle this already.
>
>
>> Cheers,
>> -- 
>> Casey Allen Shobe | http://casey.shobe.info
>> cshobe@seattleserver.com | cell 425-443-4653
>> AIM & Yahoo:  SomeLinuxGuy | ICQ:  1494523
>> SeattleServer.com, Inc. | http://www.seattleserver.com
>>
>>
>
> Eeek! Is that a foote... nay, inline-spam!!!?!??!??
>
>    -Patrick 

Re: footer isn't appended to multipart messages

[Mads Martin Joergensen]

* Sven 'Darkman' Michels <sven@darkman.de> [Jul 06. 2005 00:09]:
> | That being said, List-Subscribe, List-Unsubcribe, etc. headers
> | should be present, and they don't appear to be on this list.  Does
> | mlmmj support them yet?
> 
> Well, yes and no. Yes: you can add them to Custom headers file,
> No: they're not added automaticly (maybe this could be implemented
> but i think it's ok if you run a lot of lists to generate the
> customheaders file by a script or so ;)

It's meant to be added to customheaders. Plenty of people are using this
for newsletters, etc. where they want control over stuff. If we add such
stuff automatically people cannot take it away, while if we don't it's
easy to add.

-- 
Mads Martin Joergensen, http://mmj.dk
"Why make things difficult, when it is possible to make them cryptic
 and totally illogical, with just a little bit more effort?"
                                -- A. P. J.

Re: footer isn't appended to multipart messages (part II: reality check!)

[Mads Martin Joergensen]

* Patrick Bennett <patrick@pebcomputing.com> [Jul 06. 2005 01:37]:
> >Full ACK. I got flooded by them, too. Never mind, everything can
> >be filtered ;)
> 
> This is absolutely true- you (that is to say, *we*, that is to say 
> techie/programmer types) find this stuff very annoying.  However, mailing 
> lists are used by knitting clubs too, and I, for one, DO NOT want the 
> nightmare of administrating a knitting club mailing list that has 
> absolutely no dependable notification of "how to do it yourself".

If your users are that clueless I'm not sure mlmmj is the right list to
use.

> >| That being said, List-Subscribe, List-Unsubcribe, etc. headers
> >| should be present, and they don't appear to be on this list.  Does
> >| mlmmj support them yet?
> >
> >Well, yes and no. Yes: you can add them to Custom headers file,
> >No: they're not added automaticly (maybe this could be implemented
> >but i think it's ok if you run a lot of lists to generate the
> >customheaders file by a script or so ;)
> >
> >
> >| (this is not to say that your feature request/bug report is invalid
> >| - it is in fact useful in some cases and should work as you expect,
> >| but the example you provided should be discouraged and it's also
> >| not the easiest thing to implement due to the varying nature of
> >| what the multipart messages might contain, how the HTML is
> >| structured, etc.)
> 
> not sure which "example" you mean, but if it's monthly nags (a.k.a. mailing 
> list subscribe/unsubscribe howto's), then, yes, it should be discouraged 
> *for appropriate lists and when there is no reliable way to append footer 
> text*.  For other lists, on the contrary, it should be *encouraged* (see 
> above)!

If you want monthly notification mails, please consider using mailman.
mlmmj is not going to have it.

> >correctly. I think the 'append a footer' feature is ok, too. Just
> >disable html mail and stuff and it's nearly fine. The List- Headers
> >should be always present (not hard to implement ;)
> >Implementing appending of stuff for multipartmessages just suck
> >cause you have so many ways clients do 'their' encoding and
> >maybe some gateways who add shitty 'this is confidential mail'
> >footers, too, so you may end up in a 4 or more part mail which
> >only contains a one liner...
> 
> 1)  I haven't found the documentation on how to enforce text-only mailing 
> lists (it's one of the first bits of info i looked for when i figured out 
> what the problem was; 2) not sure what happens on a text-only mlmmj list 
> when a multi-part msg is sent, but whatever it is, for the "knitting club", 
> it *can't* be a reject msg- a mailing list that fits the "knitting club's" 
> needs (and I realize this may never be what mlmmj is, though I wish it 
> could be, because I've very much liked the software otherwise) has to 
> handle the default settings of widespread (though not necessarily popular 
> with us programmer/techies) email clients (eg. outlook express).

One of the very first examples in README.access:

 deny !^Content-Type: text/plain
 allow

Put that in DIR/control/access, and a reject mail will be sent for
everyone not sending a text/plain message. Now whether or not this
reject mails fit the knitting club is up to you. Every list has it's own
text dir, so you can simply modify DIR/text/access to fit what you need.
Might even carry intructions on how to turn off html mail in outlook if
you want. Or a link to a webpage explainging it.

mlmmj allows for heavy customization.

-- 
Mads Martin Joergensen, http://mmj.dk
"Why make things difficult, when it is possible to make them cryptic
 and totally illogical, with just a little bit more effort?"
                                -- A. P. J.

Re: footer isn't appended to multipart messages (part II: reality check!)

[Mads Martin Joergensen]

* Casey Allen Shobe <lists@seattleserver.com> [Jul 06. 2005 04:39]:
> > 1)  I haven't found the documentation on how to enforce text-only
> > mailing lists
> 
> You could filter incoming messages with something like maildrop or 
> your own script.  Not sure how you'd go about mingling this with 
> mlmmj, perhaps somebody else can answer that.

http://mlmmj.mmj.dk/files/README.access has it, as mentioned in another
reply.

> Another answer, if you just want a monthly/weekly/whatever reminder,
> just mail the list from a cron job.

Yeah, couln't agree more.

-- 
Mads Martin Joergensen, http://mmj.dk
"Why make things difficult, when it is possible to make them cryptic
 and totally illogical, with just a little bit more effort?"
                                -- A. P. J.

Re: Fw: footer isn't appended to multipart messages (part II: reality check!)

[Morten K. Poulsen]

On Tue, Jul 05, 2005 at 11:17:10PM -0700, Patrick Bennett wrote:
> I thought the point was to create high quality, highly flexible software and
> share it with the world for everyone's mutual benefit.....

No no no, that's a byproduct. The reason for free software is to give other
people the OPTION to make the software fit their particular purpose. And when
you have done that (made the software fit your purpose), please send us the
patch. We would appreciate that.

Morten

-- 
Morten K. Poulsen <morten@afdelingp.dk>
http://www.afdelingp.dk/

Re: Fw: footer isn't appended to multipart messages (part II: reality

[Jakob Hirsch]

Patrick Bennett wrote:

> oops, forgot to cc the list. apologies.

you could have sent the same mail again to the list, so threading and
quoting wouldn't have been broken.

>> If I understand correctly, and I may well be mistaken, other mailing
>> list softwares out there in the open source world (gnu mailman comes
>> to mind as an example), are able to handle appending footer messages
>> to a wide variety of incoming email formats.  No suggestion that,
>> coding wise, this is accomplished *easily*, but just that it would be
>> cool (and, in particular, useful to *me*) if mlmmj could do it as

After playing around a little with mailman, I think it's not too hard to
do it, you just have to handle the possible cases correctly.

Mads, would you accept a patch?

Re: Fw: footer isn't appended to multipart messages (part II: reality check!)

[Mads Martin Joergensen]

* Jakob Hirsch <jh@plonk.de> [Jul 07. 2005 17:31]:
> >> If I understand correctly, and I may well be mistaken, other mailing
> >> list softwares out there in the open source world (gnu mailman comes
> >> to mind as an example), are able to handle appending footer messages
> >> to a wide variety of incoming email formats.  No suggestion that,
> >> coding wise, this is accomplished *easily*, but just that it would be
> >> cool (and, in particular, useful to *me*) if mlmmj could do it as
> 
> After playing around a little with mailman, I think it's not too hard to
> do it, you just have to handle the possible cases correctly.
> 
> Mads, would you accept a patch?

Of course :-)

-- 
Mads Martin Joergensen, http://mmj.dk
"Why make things difficult, when it is possible to make them cryptic
 and totally illogical, with just a little bit more effort?"
                                -- A. P. J.

Re: Fw: footer isn't appended to multipart messages (part II: reality

[Jakob Hirsch]

[-- Attachment #1: Type: text/plain, Size: 285 bytes --]

Mads Martin Joergensen wrote:

>>Mads, would you accept a patch?
> Of course :-)

Hah, I was not so sure, the patch is somewhat invasive and I'm not
totally happy with all the things, but it works and I think it takes
measures not to break things. Corrections and suggestions welcome.

[-- Attachment #2: mlmmj-footer-mime.patch --]
[-- Type: text/plain, Size: 4723 bytes --]

--- mlmmj-1.2.8/src/do_all_the_voodo_here.c	2005-05-09 14:50:15.000000000 +0200
+++ mlmmj-1.2.8-jh1/src/do_all_the_voodo_here.c	2005-07-09 19:57:43.000000000 +0200
@@ -82,6 +82,9 @@
 	char *hdrline, *subject, *unqp;
 	int hdrsadded = 0;
 	int subject_present = 0;
+	int mime = 0;
+	char *content_orig = NULL;  /* original content-type header */
+	char *boundary = NULL;
 
 	allhdrs->count = 0;
 	allhdrs->strs = NULL;
@@ -160,6 +163,51 @@
 			}
 		}
 		
+		/* Content-Type */
+		if(footfd>=0 && !strncasecmp(hdrline, "Content-Type:", 13)) {
+			char* p;
+			int len;
+			/* search first non-LWSP */
+			for (p = hdrline+13; *p==' ' || *p=='\t'; p++);
+
+			if (!strncasecmp(p, "multipart/mixed", 15)) {
+				mime = 1;
+                
+				/* get boundary string */
+				p = strstr(p, "boundary");
+				p = index(p, '=');
+				p = index(p, '"') + 1;
+				if (p == NULL) {
+	                log_error(LOG_ARGS, "MIME multipart but no boundary");
+	                mime = 0;
+				} else {
+					/* find closing quote */
+					/* for(len=0; p[len]!='"' && p[len]!=0; ++len); */
+					len = index(p, '"') - p;
+					if(p[len] == 0) {
+	                    log_error(LOG_ARGS, "no closing quote in boundary");
+	                    mime = 0;
+					} else {
+						boundary = mymalloc(len + 1);
+						strncpy(boundary, p, len);
+						boundary[len] = 0;
+					}
+				}
+			} else if (!strncasecmp(p, "multipart/alternative", 21)) {
+				mime = 2;
+				/* create new boundary for MIME-encapsulation */
+				boundary = random_str();
+				
+				/* save for later use */
+				content_orig = hdrline; 
+				/* and create new content-type header */
+				hdrline = concatstr(3,
+					"Content-Type: multipart/mixed; boundary=\"", 
+				    boundary, "\"\n");
+
+			}
+		}
+
 		/* Should it be stripped? */
 		if(delhdrs) {
 			if(!findit(hdrline, delhdrs))
@@ -171,18 +219,102 @@
 		myfree(hdrline);
 	}
 
-	/* Just print the rest of the mail */
-	if(dumpfd2fd(infd, outfd) < 0) {
-		log_error(LOG_ARGS, "Error when dumping rest of mail");
-		return -1;
-	}
+    /* MIME-Handling */
+	switch (mime) {
+		char* s;
+		unsigned int i;
+		
+		case 0 : /* no MIME, plain message */
+            log_error(LOG_ARGS, "no MIME, plain message"); /* debug */
+			/* Just print the rest of the mail */
+			if(dumpfd2fd(infd, outfd) < 0) {
+				log_error(LOG_ARGS, "Error when dumping rest of mail");
+				return -1;
+			}
 
-	/* No more, lets add the footer if one */
-	if(footfd >= 0)
-		if(dumpfd2fd(footfd, outfd) < 0) {
-			log_error(LOG_ARGS, "Error when adding footer");
-			return -1;
-		}
+			/* No more, lets add the footer if one */
+			if(footfd >= 0)
+				if(dumpfd2fd(footfd, outfd) < 0) {
+					log_error(LOG_ARGS, "Error when adding footer");
+					return -1;
+				}
+				
+			break;
+			
+		case 1 : /* multipart/mixed */
+
+			if(dumpfd2fd(infd, outfd) < 0) {
+				log_error(LOG_ARGS, "Error when dumping rest of mail");
+				return -1;
+			}
+
+			/* seek back to the end of the close-delimiter to kill
+			  the last two hyphens. hopefully nobody puts too
+			  much garbage in the epilogue or else this will fail... */
+			i = 2 * strlen(boundary) - 1;
+			lseek(outfd, -i, SEEK_END);
+			s = mymalloc(i+1);
+			read(outfd, s, i);
+			s[i] = 0;
+			i -= strstr(s, boundary) - s + strlen(boundary);
+			myfree(s);
+			if (i>2*strlen(boundary)) {
+				log_error(LOG_ARGS, "Error seeking MIME close-delimiter");
+			} else {
+				lseek(outfd, -i, SEEK_END);
+
+				s = concatstr(2, "\nContent-Type: text/plain\n",
+					"Content-Disposition: inline\n\n");
+				writen(outfd, s, strlen(s));
+				myfree(s);
+
+				if(dumpfd2fd(footfd, outfd) < 0) {
+					log_error(LOG_ARGS, "Error when adding footer");
+					myfree(boundary);
+					return -1;
+				}
+
+				s = concatstr(3, "\n--", boundary, "--\n");
+				writen(outfd, s, strlen(s));
+				myfree(s);
+			}
+			myfree(boundary);
+
+			break;
+
+		case 2 : /* multipart/alternative */
+			s = concatstr(5,
+				"This is a multi-part message in MIME format.\n\n--",
+				boundary, "\n",
+				content_orig, "\n");
+			myfree(content_orig);
+			writen(outfd, s, strlen(s));
+			myfree(s);
+
+			if(dumpfd2fd(infd, outfd) < 0) {
+				log_error(LOG_ARGS, "Error when dumping rest of mail");
+				return -1;
+			}
+
+		    s = concatstr(4,
+		        "\n--", boundary,
+		        "\nContent-Type: text/plain\n",
+		        "Content-Disposition: inline\n\n");
+			writen(outfd, s, strlen(s));
+			myfree(s);
+			
+			if(dumpfd2fd(footfd, outfd) < 0) {
+				log_error(LOG_ARGS, "Error when adding footer");
+				return -1;
+			}
+
+			s = concatstr(3, "\n--", boundary, "--\n");
+			myfree(boundary);
+			writen(outfd, s, strlen(s));
+			myfree(s);
+
+			break;
+	}
 
 	fsync(outfd);
 

Re: Fw: footer isn't appended to multipart messages (part II: reality check!)

[Mads Martin Joergensen]

* Jakob Hirsch <jh@plonk.de> [Jul 09. 2005 20:10]:
> Mads Martin Joergensen wrote:
> 
> >>Mads, would you accept a patch?
> > Of course :-)
> 
> Hah, I was not so sure, the patch is somewhat invasive and I'm not
> totally happy with all the things, but it works and I think it takes
> measures not to break things. Corrections and suggestions welcome.

Thanks for the patch!

I'll have a look next week, when I'm back from vacation.

-- 
Mads Martin Joergensen, http://mmj.dk
"Why make things difficult, when it is possible to make them cryptic
 and totally illogical, with just a little bit more effort?"
                                -- A. P. J.

Re: Fw: footer isn't appended to multipart messages (part II: reality

[Jakob Hirsch]

[-- Attachment #1: Type: text/plain, Size: 184 bytes --]

Mads Martin Joergensen wrote:

> Thanks for the patch!

de nada.
But better take the attached one. :)

> I'll have a look next week, when I'm back from vacation.

Have a nice holiday!

[-- Attachment #2: mlmmj-footer-mime2.patch --]
[-- Type: text/plain, Size: 4712 bytes --]

--- mlmmj-1.2.8/src/do_all_the_voodo_here.c	2005-05-09 14:50:15.000000000 +0200
+++ mlmmj-1.2.8-jh1/src/do_all_the_voodo_here.c	2005-07-16 13:53:00.000000000 +0200
@@ -82,6 +82,9 @@
 	char *hdrline, *subject, *unqp;
 	int hdrsadded = 0;
 	int subject_present = 0;
+	int mime = 0;
+	char *content_orig = NULL;  /* original content-type header */
+	char *boundary = NULL;
 
 	allhdrs->count = 0;
 	allhdrs->strs = NULL;
@@ -160,6 +163,55 @@
 			}
 		}
 		
+		/* Content-Type */
+		if(footfd>=0 && !mime && !strncasecmp(hdrline, "Content-Type:", 13)) {
+			char* p;
+			int len;
+			/* search first non-LWSP */
+			for (p = hdrline+13; *p==' ' || *p=='\t'; p++);
+
+			if (!strncasecmp(p, "multipart/mixed", 15)) {
+				mime = 1;
+                
+				/* get boundary string */
+				p = index(p, ';');
+				if (p != NULL)
+					p = strstr(p, "boundary");
+				if (p != NULL)
+					p = index(p, '=');
+				if (p != NULL)
+					p = index(p, '"') + 1;
+				if (p == NULL) {
+	                log_error(LOG_ARGS, "MIME multipart but no boundary");
+	                mime = 0;
+				} else {
+					/* find closing quote */
+					for(len=0; p[len]!='"' && p[len]!=0; ++len);
+					if(p[len] == 0) {
+	                    log_error(LOG_ARGS, "no closing quote in boundary");
+	                    mime = 0;
+					} else {
+						boundary = mymalloc(len + 1);
+						strncpy(boundary, p, len);
+						boundary[len] = 0;
+					}
+				}
+			} else if (!strncasecmp(p, "multipart/alternative", 21)) {
+				mime = 2;
+
+				/* create new boundary for MIME-encapsulation */
+				boundary = random_str();
+				
+				/* save for later use */
+				content_orig = hdrline; 
+				/* and create new content-type header */
+				hdrline = concatstr(3,
+					"Content-Type: multipart/mixed; boundary=\"", 
+				    boundary, "\"\n");
+
+			}
+		}
+
 		/* Should it be stripped? */
 		if(delhdrs) {
 			if(!findit(hdrline, delhdrs))
@@ -171,18 +223,101 @@
 		myfree(hdrline);
 	}
 
-	/* Just print the rest of the mail */
-	if(dumpfd2fd(infd, outfd) < 0) {
-		log_error(LOG_ARGS, "Error when dumping rest of mail");
-		return -1;
-	}
+    /* MIME-Handling */
+	switch (mime) {
+		char* s;
+		unsigned int i;
+		
+		case 0 : /* no MIME, plain message */
+			/* Just print the rest of the mail */
+			if(dumpfd2fd(infd, outfd) < 0) {
+				log_error(LOG_ARGS, "Error when dumping rest of mail");
+				return -1;
+			}
 
-	/* No more, lets add the footer if one */
-	if(footfd >= 0)
-		if(dumpfd2fd(footfd, outfd) < 0) {
-			log_error(LOG_ARGS, "Error when adding footer");
-			return -1;
-		}
+			/* No more, lets add the footer if one */
+			if(footfd >= 0)
+				if(dumpfd2fd(footfd, outfd) < 0) {
+					log_error(LOG_ARGS, "Error when adding footer");
+					return -1;
+				}
+				
+			break;
+			
+		case 1 : /* multipart/mixed */
+
+			if(dumpfd2fd(infd, outfd) < 0) {
+				log_error(LOG_ARGS, "Error when dumping rest of mail");
+				return -1;
+			}
+
+			/* seek back to the end of the close-delimiter to kill
+			  the last two hyphens. hopefully nobody puts too
+			  much garbage in the epilogue or else this will fail... */
+			i = 2 * strlen(boundary) - 1;
+			lseek(outfd, -i, SEEK_END);
+			s = mymalloc(i+1);
+			read(outfd, s, i);
+			s[i] = 0;
+			i -= strstr(s, boundary) - s + strlen(boundary);
+			myfree(s);
+			if (i>2*strlen(boundary)) {
+				log_error(LOG_ARGS, "Error seeking MIME close-delimiter");
+			} else {
+				lseek(outfd, -i, SEEK_END);
+
+				s = concatstr(2, "\nContent-Type: text/plain\n",
+					"Content-Disposition: inline\n\n");
+				writen(outfd, s, strlen(s));
+				myfree(s);
+
+				if(dumpfd2fd(footfd, outfd) < 0) {
+					log_error(LOG_ARGS, "Error when adding footer");
+					myfree(boundary);
+					return -1;
+				}
+
+				s = concatstr(3, "\n--", boundary, "--\n");
+				writen(outfd, s, strlen(s));
+				myfree(s);
+			}
+			myfree(boundary);
+
+			break;
+
+		case 2 : /* multipart/alternative */
+			s = concatstr(5,
+				"This is a multi-part message in MIME format.\n\n--",
+				boundary, "\n",
+				content_orig, "\n");
+			myfree(content_orig);
+			writen(outfd, s, strlen(s));
+			myfree(s);
+
+			if(dumpfd2fd(infd, outfd) < 0) {
+				log_error(LOG_ARGS, "Error when dumping rest of mail");
+				return -1;
+			}
+
+		    s = concatstr(4,
+		        "\n--", boundary,
+		        "\nContent-Type: text/plain\n",
+		        "Content-Disposition: inline\n\n");
+			writen(outfd, s, strlen(s));
+			myfree(s);
+			
+			if(dumpfd2fd(footfd, outfd) < 0) {
+				log_error(LOG_ARGS, "Error when adding footer");
+				return -1;
+			}
+
+			s = concatstr(3, "\n--", boundary, "--\n");
+			myfree(boundary);
+			writen(outfd, s, strlen(s));
+			myfree(s);
+
+			break;
+	}
 
 	fsync(outfd);
 

Re: Fw: footer isn't appended to multipart messages (part II: reality check!)

[Mads Martin Joergensen]

* Jakob Hirsch <jh@plonk.de> [Jul 16. 2005 15:13]:
> Mads Martin Joergensen wrote:
> 
> > Thanks for the patch!
> 
> de nada.
> But better take the attached one. :)
> 
> > I'll have a look next week, when I'm back from vacation.
> 
> Have a nice holiday!

Thanks :-)

> --- mlmmj-1.2.8/src/do_all_the_voodo_here.c	2005-05-09 14:50:15.000000000 +0200
> +++ mlmmj-1.2.8-jh1/src/do_all_the_voodo_here.c	2005-07-16 13:53:00.000000000 +0200
> @@ -82,6 +82,9 @@
>  	char *hdrline, *subject, *unqp;
>  	int hdrsadded = 0;
>  	int subject_present = 0;
> +	int mime = 0;
> +	char *content_orig = NULL;  /* original content-type header */
> +	char *boundary = NULL;

Couldn't you read it from mlmmj-process.c ?

        struct mailhdr readhdrs[] = {
                { "From:", 0, NULL },
                { "To:", 0, NULL },
                { "Cc:", 0, NULL },
                { "Return-Path:", 0, NULL },
                { "Delivered-To:", 0, NULL },
                { NULL, 0, NULL }
        };

Just add it here, and you can access it directly. These headers are
collected anyway.

Have a look at main() in mlmmj-process.c.

-- 
Mads Martin Joergensen, http://mmj.dk
"Why make things difficult, when it is possible to make them cryptic
 and totally illogical, with just a little bit more effort?"
                                -- A. P. J.

Re: Fw: footer isn't appended to multipart messages (part II: reality

[Jakob Hirsch]

Mads Martin Joergensen wrote:

>>--- mlmmj-1.2.8/src/do_all_the_voodo_here.c	2005-05-09 14:50:15.000000000 +0200
>>+	int mime = 0;
>>+	char *content_orig = NULL;  /* original content-type header */
>>+	char *boundary = NULL;
> Couldn't you read it from mlmmj-process.c ?

I'm not sure if I understand what you mean.

I have to modify the Content-Type header and parts of the message, which
can only be done in do_all_the_voodo_here(), AFAICT. The Content-Type is
not needed outside of do_all_the_voodo_here(), so I did not mess with it
in mlmmj-process.c.

Or did I miss something here?

Re: Fw: footer isn't appended to multipart messages (part II: reality check!)

[Mads Martin Joergensen]

* Jakob Hirsch <jh@plonk.de> [Jul 18. 2005 14:20]:
> >>--- mlmmj-1.2.8/src/do_all_the_voodo_here.c	2005-05-09 14:50:15.000000000 +0200
> >>+	int mime = 0;
> >>+	char *content_orig = NULL;  /* original content-type header */
> >>+	char *boundary = NULL;
> > Couldn't you read it from mlmmj-process.c ?
> 
> I'm not sure if I understand what you mean.
> 
> I have to modify the Content-Type header and parts of the message, which
> can only be done in do_all_the_voodo_here(), AFAICT. The Content-Type is
> not needed outside of do_all_the_voodo_here(), so I did not mess with it
> in mlmmj-process.c.
> 
> Or did I miss something here?

You're right. I was thinking somewhere else in the code--but indeed; in
do_all_the_voodo_here() is when that stuff is read.

Is it just me, or is the second patch much better than the first one?

How much testing have this seen?

-- 
Mads Martin Joergensen, http://mmj.dk
"Why make things difficult, when it is possible to make them cryptic
 and totally illogical, with just a little bit more effort?"
                                -- A. P. J.

Re: Fw: footer isn't appended to multipart messages (part II: reality

[Jakob Hirsch]

Mads Martin Joergensen wrote:

> Is it just me, or is the second patch much better than the first one?

Of course it's better, else I wouldn't have posted it. :)
But the difference is not that big. I took out one forgotten debug
message and made it a little more reliable. (Seems that the local Samba
Festival and the Tour de France passing through was a little too much
distraction :)

> How much testing have this seen?

I tested the most common cases: no content-type, text/plain,
multipart/mixed (i.e. attachment) and multipart/alternative (i.e. html
and text, with or without attachment). Everybody feel free to send
something else to test@list.ymmv.de.

Possible problems I see:

- as I did no full parsing of the Content-Type header, the recognition
of the boundary is not 100% reliable, but it's very unlikely that
;boundary=" is something else.

- I only handle multipart/mixed and multipart/alternative, everything
else is assumed to be plain text, which is probably not true but it's
also the former behaviour. Maybe it's better to re-encapsulate
everything that is not already multipart/mixed. What do you think?

- Finding the close delimiter of multipart/mixed messages will not work
if there is (too much) text after the closing delimiter. This should
normally not happen, as this text will not be displayed by mail clients
(which was the why this came up at all). I think this is a minor issue.

- memory leaks if something goes wrong (i.e., at every return -1). Also
not important IMHO, as the mlmmj-process will immediatly exit then, but
I can fix this if you want.

- "voodo" is spelled "voodoo", btw. :)

Re: Fw: footer isn't appended to multipart messages (part II: reality check!)

[Mads Martin Joergensen]

* Jakob Hirsch <jh@plonk.de> [Jul 19. 2005 01:27]:
> > Is it just me, or is the second patch much better than the first one?
> 
> Of course it's better, else I wouldn't have posted it. :)
> But the difference is not that big. I took out one forgotten debug
> message and made it a little more reliable. (Seems that the local Samba
> Festival and the Tour de France passing through was a little too much
> distraction :)

Hehe :)

> > How much testing have this seen?
> 
> I tested the most common cases: no content-type, text/plain,
> multipart/mixed (i.e. attachment) and multipart/alternative (i.e. html
> and text, with or without attachment). Everybody feel free to send
> something else to test@list.ymmv.de.

People, please send all kinds of potential mails breaking this to this
list. I would really like to have this tested.

> Possible problems I see:
> 
> - as I did no full parsing of the Content-Type header, the recognition
> of the boundary is not 100% reliable, but it's very unlikely that
> ;boundary=" is something else.

How hard would it be to parse it completely?

> - I only handle multipart/mixed and multipart/alternative, everything
> else is assumed to be plain text, which is probably not true but it's
> also the former behaviour. Maybe it's better to re-encapsulate
> everything that is not already multipart/mixed. What do you think?

But if it's assumed to be text/plain, then the behaviour is as before,
no more, no less?

> - Finding the close delimiter of multipart/mixed messages will not work
> if there is (too much) text after the closing delimiter. This should
> normally not happen, as this text will not be displayed by mail clients
> (which was the why this came up at all). I think this is a minor issue.

What does "too much text" mean in this case?

> - memory leaks if something goes wrong (i.e., at every return -1). Also
> not important IMHO, as the mlmmj-process will immediatly exit then, but
> I can fix this if you want.

Don't.

> - "voodo" is spelled "voodoo", btw. :)

Yeah--mlmmj have a history of misspelled words. Look at mlmmj-recieve
:-)

-- 
Mads Martin Joergensen, http://mmj.dk
"Why make things difficult, when it is possible to make them cryptic
 and totally illogical, with just a little bit more effort?"
                                -- A. P. J.

Re: Fw: footer isn't appended to multipart messages (part II: reality

[Jakob Hirsch]

Mads Martin Joergensen wrote:

>>and text, with or without attachment). Everybody feel free to send
>>something else to test@list.ymmv.de.
> People, please send all kinds of potential mails breaking this to this
> list. I would really like to have this tested.

Found 1 non-working case already: Pine uses an all upper-case
Content-Type line, strstr("boundary") fails. There is a strchrstr() on
my system, but it seems to be a GNUism (even without having a man-page),
so I have to invent my own.

>>Possible problems I see:
>>
>>- as I did no full parsing of the Content-Type header, the recognition
>>of the boundary is not 100% reliable, but it's very unlikely that
>>;boundary=" is something else.
> How hard would it be to parse it completely?

Depends on what can possibly appear in the Content-Type line. Until now,
all I ever saw there was 'multipart/something; boundary="...."' at most.
I have to investigate.

>>- I only handle multipart/mixed and multipart/alternative, everything
>>else is assumed to be plain text, which is probably not true but it's
>>also the former behaviour. Maybe it's better to re-encapsulate
>>everything that is not already multipart/mixed. What do you think?
> But if it's assumed to be text/plain, then the behaviour is as before,
> no more, no less?

Yes. In most cases you'll get the old behaviour if something goes wrong.
Worst case is no footer at all, but then the user won't have seen it anyway.

>>- Finding the close delimiter of multipart/mixed messages will not work
>>if there is (too much) text after the closing delimiter. This should
>>normally not happen, as this text will not be displayed by mail clients
>>(which was the why this came up at all). I think this is a minor issue.
> What does "too much text" mean in this case?

The code goes back 2*strlen(boundary)-1, so the garbage has to be less
than strlen(boundary), which is about 10..30 chars. I could read a
bigger buffer or even consecutively read back overlapping blocks. Hmm,
not too hard to do, so maybe I should stick to reliability.

>>- "voodo" is spelled "voodoo", btw. :)
> Yeah--mlmmj have a history of misspelled words. Look at mlmmj-recieve

Stumbled across that already :)

Re: Fw: footer isn't appended to multipart messages (part II: reality check!)

[Mads Martin Joergensen]

* Jakob Hirsch <jh@plonk.de> [Jul 19. 2005 09:59]:
> >>and text, with or without attachment). Everybody feel free to send
> >>something else to test@list.ymmv.de.
> > People, please send all kinds of potential mails breaking this to this
> > list. I would really like to have this tested.
> 
> Found 1 non-working case already: Pine uses an all upper-case
> Content-Type line, strstr("boundary") fails. There is a strchrstr() on
> my system, but it seems to be a GNUism (even without having a man-page),
> so I have to invent my own.

Look at findit(). It uses strncasecmp().

> >>Possible problems I see:
> >>
> >>- as I did no full parsing of the Content-Type header, the recognition
> >>of the boundary is not 100% reliable, but it's very unlikely that
> >>;boundary=" is something else.
> > How hard would it be to parse it completely?
> 
> Depends on what can possibly appear in the Content-Type line. Until now,
> all I ever saw there was 'multipart/something; boundary="...."' at most.
> I have to investigate.

It doesn't really matter what one have ever seen, it only matters what
is possible according to standards.

> >>- I only handle multipart/mixed and multipart/alternative, everything
> >>else is assumed to be plain text, which is probably not true but it's
> >>also the former behaviour. Maybe it's better to re-encapsulate
> >>everything that is not already multipart/mixed. What do you think?
> > But if it's assumed to be text/plain, then the behaviour is as before,
> > no more, no less?
> 
> Yes. In most cases you'll get the old behaviour if something goes wrong.
> Worst case is no footer at all, but then the user won't have seen it anyway.

Fine. If only it's not a regression.

> >>- Finding the close delimiter of multipart/mixed messages will not work
> >>if there is (too much) text after the closing delimiter. This should
> >>normally not happen, as this text will not be displayed by mail clients
> >>(which was the why this came up at all). I think this is a minor issue.
> > What does "too much text" mean in this case?
> 
> The code goes back 2*strlen(boundary)-1, so the garbage has to be less
> than strlen(boundary), which is about 10..30 chars. I could read a
> bigger buffer or even consecutively read back overlapping blocks. Hmm,
> not too hard to do, so maybe I should stick to reliability.

mlmmj is all about reliability :-)

-- 
Mads Martin Joergensen, http://mmj.dk
"Why make things difficult, when it is possible to make them cryptic
 and totally illogical, with just a little bit more effort?"
                                -- A. P. J.

Re: Fw: footer isn't appended to multipart messages (part II: reality

[Jakob Hirsch]

Mads Martin Joergensen wrote:

>>Depends on what can possibly appear in the Content-Type line. Until now,
>>all I ever saw there was 'multipart/something; boundary="...."' at most.
>>I have to investigate.
> It doesn't really matter what one have ever seen, it only matters what
> is possible according to standards.

That's what I meant. According to RFC1341, multipart/mixed has only the
boundary parameter. But this is not an issue anymore (see below).

>>The code goes back 2*strlen(boundary)-1, so the garbage has to be less
>>than strlen(boundary), which is about 10..30 chars. I could read a
>>bigger buffer or even consecutively read back overlapping blocks. Hmm,
>>not too hard to do, so maybe I should stick to reliability.
> mlmmj is all about reliability :-)

Sure, that's one the reasons we use it. :)

To make it really reliable, I will take the easy way: Create a new MIME
envelope with multipart/mixed for every Content-Type (not only for
multipart/alternative, as now). It's not even the easiest way to do it,
it may also be the best one. Should have done it this way right from the
start.
We can even think about doing this with all messages, so the behaviour
and appearance is consistent. But that should probably be controlled by
a switch in the control directory (like "footer_always_mime").


btw, what happens with log_error messages? I thought they appear in
mlmmj.operation.log...

Re: Fw: footer isn't appended to multipart messages (part II: reality

[Jakob Hirsch]

[-- Attachment #1: Type: text/plain, Size: 592 bytes --]

Jakob Hirsch wrote:

> To make it really reliable, I will take the easy way: Create a new MIME
> envelope with multipart/mixed for every Content-Type (not only for

done. Blatantly simple, compared with the original approach.

> We can even think about doing this with all messages, so the behaviour
> and appearance is consistent. But that should probably be controlled by
> a switch in the control directory (like "footer_always_mime").

ok, maybe later... :)

It could also be used for specifying a charset, but then an encoding
would also be needed (or we assume something, 8bit or qp).


[-- Attachment #2: mlmmj-footer-mime3.patch --]
[-- Type: text/plain, Size: 3621 bytes --]

--- mlmmj-1.2.8/src/do_all_the_voodo_here.c	2005-05-09 14:50:15.000000000 +0200
+++ mlmmj-1.2.8-jh4/src/do_all_the_voodo_here.c	2005-07-19 23:55:32.000000000 +0200
@@ -79,9 +79,12 @@
 		 const char **delhdrs, struct mailhdr *readhdrs,
 		 struct strlist *allhdrs, const char *prefix)
 {
-	char *hdrline, *subject, *unqp;
+	char *hdrline, *subject, *unqp, *buf;
 	int hdrsadded = 0;
 	int subject_present = 0;
+	char *content_type_orig = NULL;	/* original Content-Type header */
+	char *content_te_orig = NULL;   /* original Content-Transfer-Encoding */
+	char *boundary = NULL;
 
 	allhdrs->count = 0;
 	allhdrs->strs = NULL;
@@ -116,7 +119,22 @@
 					myfree(subject);
 					subject_present = 1;
 				}
-
+				
+				/* time to add Content-Type */
+				if (content_type_orig || content_te_orig) {
+					/* create new boundary for MIME-encapsulation */
+					buf = random_str();
+					boundary = concatstr(3, "=_=", buf, "=_=");
+					myfree(buf);
+
+					/* and create new content-type header */
+					buf = concatstr(3,
+						"Content-Type: multipart/mixed; boundary=\"",
+						boundary, "\"\n");
+					writen(outfd, buf, strlen(buf));
+					myfree(buf);
+				}
+				
 				if(writen(outfd, hdrline, strlen(hdrline))
 						< 0) {
 					myfree(hdrline);
@@ -159,7 +177,25 @@
 				myfree(unqp);
 			}
 		}
-		
+
+		if (footfd>=0) {
+			/* save MIME header for later use */
+			if (!strncasecmp(hdrline, "Content-Type:", 13)) {
+				if (!content_type_orig)
+					content_type_orig = hdrline;
+				else
+					myfree(hdrline); /* drop surplus */
+				continue;
+			} else if (!strncasecmp(hdrline, 
+					"Content-Transfer-Encoding:", 26)) {
+				if (!content_te_orig)
+				    content_te_orig = hdrline;
+				else
+				    myfree(hdrline); /* drop surplus */
+				continue;
+			}
+		}
+
 		/* Should it be stripped? */
 		if(delhdrs) {
 			if(!findit(hdrline, delhdrs))
@@ -170,20 +206,64 @@
 
 		myfree(hdrline);
 	}
+	
+	if (content_type_orig || content_te_orig) {
 
-	/* Just print the rest of the mail */
-	if(dumpfd2fd(infd, outfd) < 0) {
-		log_error(LOG_ARGS, "Error when dumping rest of mail");
-		return -1;
-	}
+		buf = concatstr(3,
+			"This is a multi-part message in MIME format.\n\n--",
+			boundary, "\n");
+		writen(outfd, buf, strlen(buf));
+		myfree(buf);
+
+		if (content_type_orig) {
+			writen(outfd, content_type_orig, strlen(content_type_orig));
+			myfree(content_type_orig);
+		}
+        
+		if (content_te_orig) {
+			writen(outfd, content_te_orig, strlen(content_te_orig));
+			myfree(content_te_orig);
+		}
+		
+		writen(outfd, "\n", 1);
 
-	/* No more, lets add the footer if one */
-	if(footfd >= 0)
+		/* Just print the rest of the mail */
+		if(dumpfd2fd(infd, outfd) < 0) {
+			log_error(LOG_ARGS, "Error when dumping rest of mail");
+			return -1;
+		}
+
+		buf = concatstr(3, "\n--", boundary,
+			"\nContent-Type: text/plain\nContent-Disposition: inline\n\n");
+		writen(outfd, buf, strlen(buf));
+ 		myfree(buf);
+
+		/* No more, lets add the footer if one */
 		if(dumpfd2fd(footfd, outfd) < 0) {
 			log_error(LOG_ARGS, "Error when adding footer");
 			return -1;
 		}
 
+		buf = concatstr(3, "\n--", boundary, "--\n");
+		myfree(boundary);
+		writen(outfd, buf, strlen(buf));
+		myfree(buf);
+		
+	} else {
+		/* Just print the rest of the mail */
+		if(dumpfd2fd(infd, outfd) < 0) {
+			log_error(LOG_ARGS, "Error when dumping rest of mail");
+			return -1;
+		}
+
+		/* No more, lets add the footer if one */
+		if(footfd >= 0)
+			if(dumpfd2fd(footfd, outfd) < 0) {
+				log_error(LOG_ARGS, "Error when adding footer");
+				return -1;
+			}
+	}
+
 	fsync(outfd);
 
 	return 0;

Re: Fw: footer isn't appended to multipart messages (part II: reality check!)

[Mads Martin Joergensen]

* Jakob Hirsch <jh@plonk.de> [Jul 19. 2005 20:44]:

[...]

> btw, what happens with log_error messages? I thought they appear in
> mlmmj.operation.log...

No, that ends up in mail syslog, /var/log/maillog on a FreeBSD system,
/var/log/mail on SUSE LINUX.

I know--mlmmj logging is horrible. Some people have promised in
the past to clean it all up, so we had nice proper logging using syslog,
but so far it failed.

mlmmj.operations.log is log_oper().
-- 
Mads Martin Joergensen, http://mmj.dk
"Why make things difficult, when it is possible to make them cryptic
 and totally illogical, with just a little bit more effort?"
                                -- A. P. J.

Re: Fw: footer isn't appended to multipart messages (part II: reality check!)

[Mads Martin Joergensen]

* Jakob Hirsch <jh@plonk.de> [Oct 09. 2005 21:11]:
> > Also those who are running patched versions, please please download this
> > tar-ball and make sure your patches are folded in.
> 
> Um, you didn't like my footer-mime patch? I didn't get any comments,
> so I wonder what's wrong about it.

Well, if someone wants to add a footer, then every single mail will be
"massaged" even in the case of it not being needed. I really would like
it to happen only when the mail is not a text mail.

I also would like to see some checking of the writen() return values. If
any one these writes fail, it just continues on.

* Jakob Hirsch <jh@plonk.de> [Jul 20. 2005 00:09]:
> --- mlmmj-1.2.8/src/do_all_the_voodo_here.c	2005-05-09 14:50:15.000000000 +0200
> +++ mlmmj-1.2.8-jh4/src/do_all_the_voodo_here.c	2005-07-19 23:55:32.000000000 +0200
> @@ -79,9 +79,12 @@
>  		 const char **delhdrs, struct mailhdr *readhdrs,
>  		 struct strlist *allhdrs, const char *prefix)
>  {
> -	char *hdrline, *subject, *unqp;
> +	char *hdrline, *subject, *unqp, *buf;
>  	int hdrsadded = 0;
>  	int subject_present = 0;
> +	char *content_type_orig = NULL;	/* original Content-Type header */
> +	char *content_te_orig = NULL;   /* original Content-Transfer-Encoding */
> +	char *boundary = NULL;
>  
>  	allhdrs->count = 0;
>  	allhdrs->strs = NULL;
> @@ -116,7 +119,22 @@
>  					myfree(subject);
>  					subject_present = 1;
>  				}
> -
> +				
> +				/* time to add Content-Type */
> +				if (content_type_orig || content_te_orig) {
> +					/* create new boundary for MIME-encapsulation */
> +					buf = random_str();
> +					boundary = concatstr(3, "=_=", buf, "=_=");
> +					myfree(buf);
> +
> +					/* and create new content-type header */
> +					buf = concatstr(3,
> +						"Content-Type: multipart/mixed; boundary=\"",
> +						boundary, "\"\n");
> +					writen(outfd, buf, strlen(buf));
> +					myfree(buf);
> +				}
> +				
>  				if(writen(outfd, hdrline, strlen(hdrline))
>  						< 0) {
>  					myfree(hdrline);
> @@ -159,7 +177,25 @@
>  				myfree(unqp);
>  			}
>  		}
> -		
> +
> +		if (footfd>=0) {
> +			/* save MIME header for later use */
> +			if (!strncasecmp(hdrline, "Content-Type:", 13)) {
> +				if (!content_type_orig)
> +					content_type_orig = hdrline;
> +				else
> +					myfree(hdrline); /* drop surplus */
> +				continue;
> +			} else if (!strncasecmp(hdrline, 
> +					"Content-Transfer-Encoding:", 26)) {
> +				if (!content_te_orig)
> +				    content_te_orig = hdrline;
> +				else
> +				    myfree(hdrline); /* drop surplus */
> +				continue;
> +			}
> +		}
> +
>  		/* Should it be stripped? */
>  		if(delhdrs) {
>  			if(!findit(hdrline, delhdrs))
> @@ -170,20 +206,64 @@
>  
>  		myfree(hdrline);
>  	}
> +	
> +	if (content_type_orig || content_te_orig) {
>  
> -	/* Just print the rest of the mail */
> -	if(dumpfd2fd(infd, outfd) < 0) {
> -		log_error(LOG_ARGS, "Error when dumping rest of mail");
> -		return -1;
> -	}
> +		buf = concatstr(3,
> +			"This is a multi-part message in MIME format.\n\n--",
> +			boundary, "\n");
> +		writen(outfd, buf, strlen(buf));
> +		myfree(buf);
> +
> +		if (content_type_orig) {
> +			writen(outfd, content_type_orig, strlen(content_type_orig));
> +			myfree(content_type_orig);
> +		}
> +        
> +		if (content_te_orig) {
> +			writen(outfd, content_te_orig, strlen(content_te_orig));
> +			myfree(content_te_orig);
> +		}
> +		
> +		writen(outfd, "\n", 1);
>  
> -	/* No more, lets add the footer if one */
> -	if(footfd >= 0)
> +		/* Just print the rest of the mail */
> +		if(dumpfd2fd(infd, outfd) < 0) {
> +			log_error(LOG_ARGS, "Error when dumping rest of mail");
> +			return -1;
> +		}
> +
> +		buf = concatstr(3, "\n--", boundary,
> +			"\nContent-Type: text/plain\nContent-Disposition: inline\n\n");
> +		writen(outfd, buf, strlen(buf));
> + 		myfree(buf);
> +
> +		/* No more, lets add the footer if one */
>  		if(dumpfd2fd(footfd, outfd) < 0) {
>  			log_error(LOG_ARGS, "Error when adding footer");
>  			return -1;
>  		}
>  
> +		buf = concatstr(3, "\n--", boundary, "--\n");
> +		myfree(boundary);
> +		writen(outfd, buf, strlen(buf));
> +		myfree(buf);
> +		
> +	} else {
> +		/* Just print the rest of the mail */
> +		if(dumpfd2fd(infd, outfd) < 0) {
> +			log_error(LOG_ARGS, "Error when dumping rest of mail");
> +			return -1;
> +		}
> +
> +		/* No more, lets add the footer if one */
> +		if(footfd >= 0)
> +			if(dumpfd2fd(footfd, outfd) < 0) {
> +				log_error(LOG_ARGS, "Error when adding footer");
> +				return -1;
> +			}
> +	}
> +
>  	fsync(outfd);
>  
>  	return 0;

-- 
Mads Martin Joergensen, http://mmj.dk
"Why make things difficult, when it is possible to make them cryptic
 and totally illogical, with just a little bit more effort?"
                                 -- A. P. J.

Re: Fw: footer isn't appended to multipart messages (part II: reality

[Jakob Hirsch]

Mads Martin Joergensen wrote:

>>so I wonder what's wrong about it.
> Well, if someone wants to add a footer, then every single mail will be
> "massaged" even in the case of it not being needed. I really would like
> it to happen only when the mail is not a text mail.

I considered this, but thought it would be an advantage to have constant
behaviour. There's no real downside in always having a MIME mail, or is
it? Mailing list software always modify messages and all current MUAs
are MIME-capable. Checking for text/plain adds only a little complexity,
so if you insist on it, I'll do it. It's probably a good idea to make
this configurable (footer_force_mime or something).

> I also would like to see some checking of the writen() return values. If
> any one these writes fail, it just continues on.

Should we abort if a critical writen() fails?
There are already some writen() without checking...

Re: Fw: footer isn't appended to multipart messages (part II: reality check!)

[Neale Pickett]

On Monday 10 October 2005 03:41 pm, you wrote:
> There's no real downside in always having a MIME mail, or is
> it?

I would consider this behaviour undesirable.  YMMV.  If this does get added I 
hope it's optional.

Re: Fw: footer isn't appended to multipart messages (part II: reality

[Jakob Hirsch]

Neale Pickett wrote:

>>There's no real downside in always having a MIME mail, or is it?
> I would consider this behaviour undesirable.  YMMV.  

I don't want to raise another "MIME - bad or not?" discussion, but why?
MIME was considered a bad back in the old days, when bandwidth was
expensive and many MUAs couldn't handle it. Things have changed a lot
since then.

> If this does get added I hope it's optional.

So there should be an option footer_never_mime? Don't think that makes
much sense. If the mail has a multipart content-type, the footer will be
lost. If it's already MIME, we only meet the standards.

Re: Fw: footer isn't appended to multipart messages (part II: reality check!)

[Mads Martin Joergensen]

* Jakob Hirsch <jh@plonk.de> [Oct 11. 2005 00:08]:
> > If this does get added I hope it's optional.
> 
> So there should be an option footer_never_mime? Don't think that makes
> much sense. If the mail has a multipart content-type, the footer will
> be lost. If it's already MIME, we only meet the standards.

So how about meeting the standars when and only when it's already MIME?

And BTW--if you don't configure mlmmj to do it, you can have a mailing
list manager that doesn't touch the mails whatsoever.

-- 
Mads Martin Joergensen, http://mmj.dk
"Why make things difficult, when it is possible to make them cryptic
 and totally illogical, with just a little bit more effort?"
                                 -- A. P. J.

Re: Fw: footer isn't appended to multipart messages (part II: reality check!)

[Neale Pickett]

On Monday 10 October 2005 06:15 pm, Mads Martin Joergensen wrote:
> * Jakob Hirsch <jh@plonk.de> [Oct 11. 2005 00:08]:
>
> > If it's already MIME, we only meet the standards.
>
> So how about meeting the standars when and only when it's already MIME?

+1 for this.  There's no need to make a message multipart if there's only one 
part in the first place (unless that one part isn't text/plain, I guess).

Re: Fw: footer isn't appended to multipart messages (part II:

[Jakob Hirsch]

Neale Pickett wrote:

>> > If it's already MIME, we only meet the standards.
>> So how about meeting the standars when and only when it's already MIME?

Content-Type and Content-Transfer-Encoding _are_ MIME headers, so messages
containing these headers are MIME messages and should be treated as such.

> +1 for this.  There's no need to make a message multipart if there's only
> one part in the first place (unless that one part isn't text/plain, I
> guess).

This is only valid if this one part's transfer-encoding is 7/8bit or QP.
Even then it's closer to the "MIME spirit" to make it multipart and add
things in it's own part. This also gives you the possibility to use your
own charset etc. (though this is not supported ATM).

If somebody cares about message modification (and does MTA debugging, like
I also do) he shouldn't use a footer in the first place, so there is no
modification at all.
But if you have to, MIME gives you the opportunity to reconstruct a
message as it was, since the original body is preserved in a MIME
envelope. Simply appending a footer is more tampering.

But ok, I yield to the overwhelming majority. :)
I'll add a switch so people can disable MIME handling. Or should the
default be "off"?

Re: Fw: footer isn't appended to multipart messages (part II: reality check!)

[Mads Martin Joergensen]

* Jakob Hirsch <jh@plonk.de> [Oct 11. 2005 09:52]:
> But ok, I yield to the overwhelming majority. :)
> I'll add a switch so people can disable MIME handling. Or should the
> default be "off"?

I think the default should be off, so we're keeping compatibility, and
old installations will never all of a sudden change behaviour.

DIR/control/mimeencapsulate seems like the TUNABLE for it :-)

-- 
Mads Martin Joergensen, http://mmj.dk
"Why make things difficult, when it is possible to make them cryptic
 and totally illogical, with just a little bit more effort?"
                                 -- A. P. J.

Re: Fw: footer isn't appended to multipart messages (part II:

[Jakob Hirsch]

Mads Martin Joergensen wrote:

>> But ok, I yield to the overwhelming majority. :)
>> I'll add a switch so people can disable MIME handling. Or should the
>> default be "off"?
> I think the default should be off, so we're keeping compatibility, and
> old installations will never all of a sudden change behaviour.

sounds sensible.

> DIR/control/mimeencapsulate seems like the TUNABLE for it :-)

something like footer_use_mime sounded logical for me, as it's only used
if there's a footer, but I don't really care about that.


About the writen()s you mentioned: abort if they fail?

Re: Fw: footer isn't appended to multipart messages (part II: reality check!)

[Mads Martin Joergensen]

* Jakob Hirsch <jh@plonk.de> [Oct 11. 2005 11:30]:
> >> But ok, I yield to the overwhelming majority. :)
> >> I'll add a switch so people can disable MIME handling. Or should the
> >> default be "off"?
> > I think the default should be off, so we're keeping compatibility,
> > and old installations will never all of a sudden change behaviour.
> 
> sounds sensible.
> 
> > DIR/control/mimeencapsulate seems like the TUNABLE for it :-)
> 
> something like footer_use_mime sounded logical for me, as it's only
> used if there's a footer, but I don't really care about that.

/me neither. You're making the patch, call it what you want.

> About the writen()s you mentioned: abort if they fail?

Like this first writen in mlmmj-process.c:

    if(writen(queuefd, tmp, strlen(tmp)) < 0)
            log_error(LOG_ARGS, "Could not write line for "
                                 "moderatemail");

You mention you've found other writen() calls that there's no check for.
It might very well be the case, but it's a bug then. Albeit a minor one,
since writen() catches EINTR and ensures that all bytes one wants to
write are written by calling write() subsequently.

-- 
Mads Martin Joergensen, http://mmj.dk
"Why make things difficult, when it is possible to make them cryptic
 and totally illogical, with just a little bit more effort?"
                                 -- A. P. J.

Re: Fw: footer isn't appended to multipart messages (part II: reality check!)

[Neale Pickett]

On Tuesday 11 October 2005 03:30 am, Jakob Hirsch wrote:

> > DIR/control/mimeencapsulate seems like the TUNABLE for it :-)
>
> something like footer_use_mime sounded logical for me, as it's only used
> if there's a footer, but I don't really care about that.

What if you call it control/mime_footer, which would supercede control/footer 
and contain a full MIME part?  For instance:

  Content-Type: text/plain

  This is the snot mail list.  To unsubscribe, send a message with
  the subject "UNSUBSCIBE" to every member of the list, individually.

The advantage here would be that you could make the footer anything you like: 
it could be HTML, or a PNG, or even a multipart/alternative with all three.

If you wanted to be super spiff about it, you could check to make sure there's 
a header block in the file and if not prepend the Content-Type: text/plain 
part.

Call me crazy,

Neale