Proposal for richer list text

Proposal for richer list text

[Ben Schmidt]

Hi,

I'm wondering if this sounds acceptable. I'll go ahead and have a shot 
implementing it, if so.

I'm thinking of extending the listtext/prepstdreply mechanism to, rather than just 
expect and include a subject header, to allow other headers at the top of 
listtexts, that would/could replace the default headers, and to allow greater 
amounts of the original mail to be included, and random and subject substitutions.

The first part would involve representing the default headers (To, From, Reply-To, 
MIME-Version, Content-Type, etc.) in a list/structure, then parsing each header 
line of the listtext, removing existing headers from the structure if they match 
the new one, and then adding the new one. Then the structure would be reduced to a 
single string and output.

The second part would involve moving the code for including original mail out of 
substitute_one() and into prepstdreply(). It would require the $originalmail$ 
substitution to appear alone on a line, or merely with preceding whitespace, to be 
recognised: this shouldn't be a problem as it is senseless to do anything else! 
The move of the code to prepstdreply() is so that instead of buffering the content 
of the mail in a string, it can be copied directly from the mailfile into the 
queuefile, which will be important for large mails. To specify how much of the 
mail to include, I suggest using $originalmailNNNNN$ where NNNNN is a number. The 
default, if the number were omitted, would be 100, so current behaviour would be 
unchanged, and to include the whole mail, a ridiculously large number could be 
given (1000000000 or something). Any whitespace preceding $originalmail$ would be 
prepended to each line of the mail also. The current single-space indent could be 
achieved with a single space, then, but that could be avoided, too.

The third part would require searching for Subject: lines in mails that arrive to 
mlmmj-process and then allowing $subject$ to be replaced by it in the listtext, as 
well as substitutions $randomN$ which would generate random strings like the 
cookies generate, and which could be reused by including the same number N.

The result of all this is that you could construct MIME mails for your listtexts, 
including properly randomised boundaries, and attach the original mail message to 
them in entirety rather than have a text dump of its initial lines. This could 
pave the way to more nice-looking listtexts (even HTML for those so inclined) and 
easier moderation with more meaningful subject lines including the subjects of the 
messages being moderated.

Does this seem like an OK extension, and a suitable way to implement it?

Ben.

Re: Proposal for richer list text

[Christoph Wilke]

Hej,

On Tue, January 26, 2010 14:06, Ben Schmidt wrote:
>
> Hi,
>
> I'm wondering if this sounds acceptable. I'll go ahead and have a shot
> implementing it, if so.
>
[...]
> The second part would involve moving the code for including original mail out
> of
> substitute_one() and into prepstdreply(). It would require the $originalmail$
> substitution to appear alone on a line, or merely with preceding whitespace,
> to be
> recognised: this shouldn't be a problem as it is senseless to do anything
> else!
> The move of the code to prepstdreply() is so that instead of buffering the
> content
> of the mail in a string, it can be copied directly from the mailfile into the
> queuefile, which will be important for large mails. To specify how much of the
> mail to include, I suggest using $originalmailNNNNN$ where NNNNN is a number.
> The
> default, if the number were omitted, would be 100, so current behaviour would
> be
> unchanged, and to include the whole mail, a ridiculously large number could be
> given (1000000000 or something). Any whitespace preceding $originalmail$ would
> be
> prepended to each line of the mail also. The current single-space indent could
> be
> achieved with a single space, then, but that could be avoided, too.

That would be really great! I had a moderated newsletter service running for
some time and needed to change the code for a) appending the whole eMail and
b) removing that (in my opinion useless) space indent.
So I vote for that.

>
> The third part would require searching for Subject: lines in mails that arrive
> to
> mlmmj-process and then allowing $subject$ to be replaced by it in the
> listtext, as
> well as substitutions $randomN$ which would generate random strings like the
> cookies generate, and which could be reused by including the same number N.
>
> The result of all this is that you could construct MIME mails for your
> listtexts,
> including properly randomised boundaries, and attach the original mail message
> to
> them in entirety rather than have a text dump of its initial lines. This could
> pave the way to more nice-looking listtexts (even HTML for those so inclined)
> and
> easier moderation with more meaningful subject lines including the subjects of
> the
> messages being moderated.

The randomN has my vote too, since this would editing a moderated message
much more comfortable (as MIME mail).

Regards
Christoph

Re: Proposal for richer list text

[Robin H. Johnson]

On Wed, Jan 27, 2010 at 12:06:05AM +1100, Ben Schmidt wrote:
> I'm wondering if this sounds acceptable. I'll go ahead and have a
> shot implementing it, if so.
> 
> I'm thinking of extending the listtext/prepstdreply mechanism to,
> rather than just expect and include a subject header, to allow other
> headers at the top of listtexts, that would/could replace the
> default headers, and to allow greater amounts of the original mail
> to be included, and random and subject substitutions.
> 
> The first part would involve representing the default headers (To,
> From, Reply-To, MIME-Version, Content-Type, etc.) in a
> list/structure, then parsing each header line of the listtext,
> removing existing headers from the structure if they match the new
> one, and then adding the new one. Then the structure would be
> reduced to a single string and output.
Heavily in favour of supporting custom headers in listtexts.

I just found out the hard way that subconf mails do NOT have any of the
customheaders on them, and thus ended up getting us in a blacklist for a
while.

Can we get that first part implemented ASAP? One thing I am concerned
about with it, is the ability to append vs. replace headers.

For the moment, I say we just append, not replace.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Re: Proposal for richer list text

[Ben Schmidt]

On 3/02/10 9:36 AM, Robin H. Johnson wrote:
> On Wed, Jan 27, 2010 at 12:06:05AM +1100, Ben Schmidt wrote:
>> I'm wondering if this sounds acceptable. I'll go ahead and have a
>> shot implementing it, if so.
>>
>> I'm thinking of extending the listtext/prepstdreply mechanism to,
>> rather than just expect and include a subject header, to allow other
>> headers at the top of listtexts, that would/could replace the
>> default headers, and to allow greater amounts of the original mail
>> to be included, and random and subject substitutions.
>>
>> The first part would involve representing the default headers (To,
>> From, Reply-To, MIME-Version, Content-Type, etc.) in a
>> list/structure, then parsing each header line of the listtext,
>> removing existing headers from the structure if they match the new
>> one, and then adding the new one. Then the structure would be
>> reduced to a single string and output.
> Heavily in favour of supporting custom headers in listtexts.
>
> I just found out the hard way that subconf mails do NOT have any of the
> customheaders on them, and thus ended up getting us in a blacklist for a
> while.
>
> Can we get that first part implemented ASAP? One thing I am concerned
> about with it, is the ability to append vs. replace headers.
>
> For the moment, I say we just append, not replace.

Replacement is necessary to be able to change the content type which is
one of my main goals.

I can see it could be a problem if wanting to add multiple headers of
one kind, though.

How about it removes an mlmmj default header if it finds one, but
ensures it never replaces headers in the listtext with each other, i.e.
all the headers in the listtext will definitely end up in the mail?

For reference, the default headers mlmmj generates are:

From: for moderation emails, list+owner@...
To: for moderation emails, list-moderators@...
Reply-To: for moderation emails, list+moderate-blahblahblah@...
Date: ...
Message-ID: ...
Subject: from listtext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Encoding: 8bit

Actually, this last one is a bug...that should be
'Content-Transfer-Encoding:'; there is no such MIME header as
'Content-Encoding:'. The same bug is there when digests are constructed.
That bugfix will be included in the patch when I've made it. As well as
another one I just spotted where the listtext file isn't closed if
digests don't have a thread summary.

At any rate, these are the only headers that mlmmj adds, so the only
candidates for replacement, and I can imagine it being useful to be able
to replace all of them except 'Date', 'Message-ID' and possibly
'Reply-To'. Simply appending wouldn't work, particularly for what I want
to achieve.

Does this slight modification sound OK to you?

Ben.