[mlmmj] Problems with microsoft

[mlmmj] Problems with microsoft

[Christian Gleerup]

Dear List.
 
I use mlmmj to send newsletters, and I have the following problem, but now more serious since Microsoft have blocked the server.
 
Problem 1.
Our mail recipients sometimes receives a bunch of mails with the following message "some messaged could not be delivered. If you see this things are back to normal."
They get many of these, so they complain.
I understand that they get one mail telling them that everything Works, but why 10-40?

 
Is it a configuration problem between mlmmj and postfix?
I have the following settings for postfix and mlmmj that I think is relevant for the problem. But i don't really understand how they interact, could there be some configuration error so mlmmj fills postfix with a queue due to lack of respone.


* /etc/postfix/main.cf
bounce_queue_lifetime = 2d
minimal_backoff_time = 1800s


* in  'tunables' bouncelife
2592000
(30 days)


Problem 2
Microsoft think I am doing 'namespace mining', I know I don't. but maybe it is somehow connected to the problem above ?


Best Regards and I hope you can help me.
/Christian

Re: [mlmmj] Problems with microsoft

[Ben Schmidt]

Hi, Christian,

Here are some thoughts:

> I use mlmmj to send newsletters, and I have the following problem, but
> now more serious since Microsoft have blocked the server.

That's annoying. :-)

> Problem 1.
> Our mail recipients sometimes receives a bunch of mails with the
> following message "some messaged could not be delivered. If you see
> this things are back to normal." They get many of these, so they
> complain.
> I understand that they get one mail telling them that everything
> Works, but why 10-40?

It would be helpful to find out where those mails originate--is Mlmmj
sending a lot of mail to Postfix, or is Postfix sending a lot of mail to
the next server? There are a few things to look at: (1) your mail logs;
you should be able to find the messages going out, (2)
mlmmj.operations.log in the relevant listdir, (3)
mlmmj-maintd.lastrun.log in the relevant listdir (if you get to it
quickly enough after it happens), (4) the Message-ID and other headers
of the received 'duplicate' messages.

Perhaps you could furnish us with some of that information. De-identify
it by making some small modifications to the email/IP/list addresses in
it if necessary.

> Is it a configuration problem between mlmmj and postfix?

Possibly.

> I have the following settings for postfix and mlmmj that I think is
> relevant for the problem. But i don't really understand how they
> interact, could there be some configuration error so mlmmj fills
> postfix with a queue due to lack of respone.
>
> * /etc/postfix/main.cf
> bounce_queue_lifetime = 2d
> minimal_backoff_time = 1800s

This could possibly be relevant if Postfix is trying to send mail to
mlmmj-receive, succeeding, but receiving a failure response; Postfix
will keep trying for 2 days to deliver the bounce message to Mlmmj,
which will keep receiving it and keep thinking the address is bouncing,
and keep sending bounce probes.

You should be able to determine from your mail logs if this is happening
(you will see a lot of failed messages from Postfix itself--postmaster,
or mail_daemon or whatever it uses--to list+bounces addresses).

It would also be helpful to know how Postfix and Mlmmj are linked? What
do you have in your config files to facilitate delivery of messages to
Mlmmj?

> * in  'tunables' bouncelife
> 2592000
> (30 days)

This shouldn't be too relevant; it's how long Mlmmj waits before giving
up and unsubscribing the user. If you changed this, bounce probes would
just turn into unsubscriptions; the cause of the problems wouldn't be
addressed.

> Problem 2
> Microsoft think I am doing 'namespace mining', I know I don't. but
> maybe it is somehow connected to the problem above ?

If this is truly the case, it is probably unrelated to the problem
above. If you are namespace mining, you are trying lots of addresses
@hotmail.com (or wherever), hoping to find real ones. In fact, you are
probably actually getting a lot of bounces for nonexistent addresses.
So, if, as a responsible mail host, I want to detect if you're namespace
mining, I would use the number of bounces due to nonexistent addresses
as a heuristic, and block you if you get a lot of them.

A nonexistent address can't receive a lot of probe messages! It can't
receive anything. So it's probably not related to the problem above.

However, it could be related to your bouncelife tunable. If an address
ceases to exist, because it's deleted; or in some cases, if an address
with wrong spelling is added to the list (e.g. without requiring
confirmation), Mlmmj is going to receive a bounce message about the
non-existent address. However, Mlmmj doesn't know whether that bounce
message is a permanent error or a temporary error (and in fact,
sometimes, due to misconfiguration, errors that seem permanent are
actually temporary, so best retried anyway). Therefore Mlmmj will keep
retrying the address--possibly every 2 hours (however often mlmmj-maintd
runs; I'm not sure if Mlmmj throttles delivery or not) for *30 days*,
and every time receive a bounce message due to the non-existent address.
That many bounces for non-existent addresses would definitely make you
look like you're namespace mining (if the watchdog software that uses
the heuristic isn't smart enough to realise they're all for the same
address or few addresses).

You can check whether this is happening by looking in your listdir. The
last bounce for each currently-bouncing address is stored in the bounce
subdir, so you can read them, and see how many addresses (including how
many from Microsoft) are bouncing, and why.

One reason for doing automatic bounce processing is to minimise
unnecessary bounces, by unsubscribing users before bounces to them
become suspicious or waste too much bandwidth; by making your bouncelife
so high, you've reduced the effectiveness of the feature.

> Best Regards and I hope you can help me.

No trouble. If you need more help, please furnish us with more
information: mail logs, mlmmj logs, message headers, configuration. All
this information is useful and necessary for properly tracking down
these problems.

Ben.

Re: [mlmmj] Problems with microsoft

[Richard Mortimer]

Hi,

Have you tried looking up your mlmmj server's IP address in the various 
blocklists. The following will do a lookup and that might help you to 
see if it is just Microsoft or whether others think your server is a 
problem.

http://www.dnsbl.info/

That will at least give you an idea as to why the blocking may be happening.

Regards

Richard

On 24/11/2013 15:57, Christian Gleerup wrote:
> Hi Ben and List,
> I'm been going through the logs, but I am not getting much wiser.
>
> I hope the information provided here is the correct one, ontherwise please direct me as to what else to look for.
>
>
> Here a snippet from /var/log/mail.log.1
> ---- BEGIN SNIP -----------------------------------------------------
> Nov 10 06:27:22 lists postfix/smtp[8385]: 1465545E8A: to=<someone@hotmail.com>, relay=mx1.hotmail.com[65.55.37.72]:25, delay=0.94, delays=0.16/0.01/0.59/0.19, dsn=5.0.0, status=bounced (host mx1.hotmail.com[65.55.37.72] said: 550 SC-002 (COL0-MC1-F1) Unfortunately, messages from ----IP---- weren't sent. Please contact your Internet service provider since part of their network is on our block list. You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors. (in reply to MAIL FROM command))
> Nov 10 06:27:22 lists postfix/smtp[8385]: 1465545E8A: lost connection with mx1.hotmail.com[65.55.37.72] while sending RCPT TO
> Nov 10 06:27:22 lists postfix/cleanup[12147]: 108F1467C2: message-id=<20131110052722.108F1467C2@lists.DOMAIN>
> Nov 10 06:27:22 lists postfix/cleanup[11855]: 0B321467BD: message-id=<1384061241-12392-mlmmj-5f63aeea@lists.DOMAIN>
> Nov 10 06:27:22 lists postfix/smtp[8334]: 50697467AF: to=<someone@live.dk>, relay=mx4.hotmail.com[65.55.92.136]:25, delay=0.73, delays=0.1/0.01/0.47/0.15, dsn=5.0.0, status=bounced (host mx4.hotmail.com[65.55.92.136] said: 550 SC-002 (SNT0-MC1-F19) Unfortunately, messages from ----IP---- weren't sent. Please contact your Internet service provider since part of their network is on our block list. You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors. (in reply to MAIL FROM command))
> Nov 10 06:27:22 lists postfix/smtp[8334]: 50697467AF: lost connection with mx4.hotmail.com[65.55.92.136] while sending RCPT TO
> Nov 10 06:27:22 lists postfix/cleanup[12193]: 17834467C3: message-id=<20131110052722.17834467C3@lists.DOMAIN>

Re: [mlmmj] Problems with microsoft

[Ben Schmidt]

OK, so obviously after being blocked wasn't an ideal time to look at
bounce messages and logs, because it's just telling you what you already
know: you're blocked. To find out why, we need earlier data.

So once you've jumped through the hoops to get Microsoft to unblock you,
you should pay careful attention to what's in "bounce" for MS addresses
and see what's going on. That may shed light on what caused you to get
blocked. Also you should probably reduce your bouncelife tunable, as
discussed. Also, to avoid a massive backlog of mail for MS addresses,
you should perhaps remove them from the retry queue (clear out the
bounces from Mlmmj's list dir and confirm with mail logs Mlmmj is no
longer getting a stack of failed deliveries). That might help you avoid
getting your unblock request denied, or getting blocked again straight
away because Mlmmj is continuously trying to send probes. It would also
avoid the addresses all being unsubscribed (though I guess actually it
may be too late for that; O well; I suppose you can always put them back
as long as you know what they were).

You didn't seem to include any logs or other information to do with the
other issue (people receiving multiple bounce probes). You may need to
track down an instance of that occurring, and see what you can find in
the logs. Getting samples of the repeat messages would be a good first
step, looking at the headers, and trying to trace the deliveries back
through your logs (mail server logs, then back to Mlmmj if necessary).

Nevertheless, your logs and bounce files have confirmed that Mlmmj is
wired up to Postfix OK, and that bounces are making it all the way back
to Mlmmj.

Regarding addresses on the list that should have already been removed:
the most likely cause of this is bad permissions. Mlmmj's error checking
isn't always brilliant, and sometimes you don't get an informative
message when things go wrong (we are working on fixing this). Anyway,
check that mlmmj-maintd runs with permission to access the listdir, and
that all the files in the listdir have appropriate permissions for
mlmmj-maintd, mlmmj when invoked via Postfix, and mlmmj when invoked any
other ways you use (commandline/web interfaces). You may need to add
users to groups, etc. to make it work. I notice you're using "nobody"
for mlmmj, which isn't a good idea. See the tutorial/readme in the docs
about integrating Mlmmj with Postfix for details of why, and how to do
it better. Anyway, if one method of subscribing users creates files that
mlmmj-maintd can't modify, it won't be able to unsubscribe them.

I hope this helps,

Ben.



On 25/11/13 2:57 AM, Christian Gleerup wrote:
> Hi Ben and List,
> I'm been going through the logs, but I am not getting much wiser.
>
> I hope the information provided here is the correct one, ontherwise please direct me as to what else to look for.
>
>
> Here a snippet from /var/log/mail.log.1
> ---- BEGIN SNIP -----------------------------------------------------
> Nov 10 06:27:22 lists postfix/smtp[8385]: 1465545E8A: to=<someone@hotmail.com>, relay=mx1.hotmail.com[65.55.37.72]:25, delay=0.94, delays=0.16/0.01/0.59/0.19, dsn=5.0.0, status=bounced (host mx1.hotmail.com[65.55.37.72] said: 550 SC-002 (COL0-MC1-F1) Unfortunately, messages from ----IP---- weren't sent. Please contact your Internet service provider since part of their network is on our block list. You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors. (in reply to MAIL FROM command))
> Nov 10 06:27:22 lists postfix/smtp[8385]: 1465545E8A: lost connection with mx1.hotmail.com[65.55.37.72] while sending RCPT TO
> Nov 10 06:27:22 lists postfix/cleanup[12147]: 108F1467C2: message-id=<20131110052722.108F1467C2@lists.DOMAIN>
> Nov 10 06:27:22 lists postfix/cleanup[11855]: 0B321467BD: message-id=<1384061241-12392-mlmmj-5f63aeea@lists.DOMAIN>
> Nov 10 06:27:22 lists postfix/smtp[8334]: 50697467AF: to=<someone@live.dk>, relay=mx4.hotmail.com[65.55.92.136]:25, delay=0.73, delays=0.1/0.01/0.47/0.15, dsn=5.0.0, status=bounced (host mx4.hotmail.com[65.55.92.136] said: 550 SC-002 (SNT0-MC1-F19) Unfortunately, messages from ----IP---- weren't sent. Please contact your Internet service provider since part of their network is on our block list. You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors. (in reply to MAIL FROM command))
> Nov 10 06:27:22 lists postfix/smtp[8334]: 50697467AF: lost connection with mx4.hotmail.com[65.55.92.136] while sending RCPT TO
> Nov 10 06:27:22 lists postfix/cleanup[12193]: 17834467C3: message-id=<20131110052722.17834467C3@lists.DOMAIN>
> Nov 10 06:27:22 lists postfix/smtpd[12070]: disconnect from localhost[127.0.0.1]
> Nov 10 06:27:22 lists postfix/bounce[12246]: 1465545E8A: sender non-delivery notification: 108F1467C2
> Nov 10 06:27:22 lists postfix/qmgr[19489]: 17834467C3: from=<>, size464, nrcpt=1 (queue active)
> Nov 10 06:27:22 lists postfix/bounce[12248]: 50697467AF: sender non-delivery notification: 17834467C3
> Nov 10 06:27:22 lists postfix/qmgr[19489]: 1465545E8A: removed
> Nov 10 06:27:22 lists postfix/qmgr[19489]: 50697467AF: removed
> Nov 10 06:27:22 lists postfix/qmgr[19489]: 108F1467C2: from=<>, size524, nrcpt=1 (queue active)
> Nov 10 06:27:22 lists postfix/qmgr[19489]: 0B321467BD: from=<listname+bounces-probe-someone=hotmail.com@lists.DOMAIN>, size\x1015, nrcpt=1 (queue active)
> Nov 10 06:27:22 lists postfix/smtpd[12070]: connect from localhost[127.0.0.1]
> Nov 10 06:27:22 lists postfix/smtpd[12070]: 42F9545E8A: client=localhost[127.0.0.1]
> Nov 10 06:27:22 lists postfix/cleanup[12331]: 42F9545E8A: message-id=<1384061242-12397-mlmmj-70337e4e@lists.DOMAIN>
>
> Nov 10 06:27:22 lists postfix/pipe[11679]: 108F1467C2: to=<listname+bounces-probe-someone=hotmail.com@lists.DOMAIN>, relay=mlmmj, delay=0.41, delays=0.17/0.01/0/0.23, dsn=2.0.0, status=sent (delivered via mlmmj service)
> Nov 10 06:27:22 lists postfix/qmgr[19489]: 108F1467C2: removed
> Nov 10 06:27:22 lists postfix/pipe[12334]: 17834467C3: to=<listname+bounces-probe-someone=live.dk@lists.DOMAIN>, relay=mlmmj, delay=0.38, delays=0.09/0.06/0/0.23, dsn=2.0.0, status=sent (delivered via mlmmj service)
> Nov 10 06:27:22 lists postfix/qmgr[19489]: 17834467C3: removed
> Nov 10 06:27:22 lists postfix/qmgr[19489]: 42F9545E8A: from=<listname+bounces-probe-someone=hotmail.com@lists.DOMAIN>, size\x1009, nrcpt=1 (queue active)
> Nov 10 06:27:22 lists postfix/smtpd[12070]: disconnect from localhost[127.0.0.1]
>
> Nov 10 06:27:22 lists postfix/smtp[8370]: 78F99467B9: to=<someone@hotmail.com>, relay=mx3.hotmail.com[65.54.188.126]:25, delay=0.88, delays=0.15/0/0.55/0.18, dsn=5.0.0, status=bounced (host mx3.hotmail.com[65.54.188.126] said: 550 SC-002 (BAY0-MC4-F51) Unfortunately, messages from ----IP---- weren't sent. Please contact your Internet service provider since part of their network is on our block list. You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors. (in reply to MAIL FROM command))
> Nov 10 06:27:22 lists postfix/smtp[8370]: 78F99467B9: lost connection with mx3.hotmail.com[65.54.188.126] while sending RCPT TO
> Nov 10 06:27:22 lists postfix/cleanup[11855]: 7AB27467B3: message-id=<20131110052722.7AB27467B3@lists.DOMAIN>
> Nov 10 06:27:22 lists postfix/qmgr[19489]: 7AB27467B3: from=<>, size514, nrcpt=1 (queue active)
> Nov 10 06:27:22 lists postfix/smtpd[12070]: connect from localhost[127.0.0.1]
> Nov 10 06:27:22 lists postfix/bounce[12246]: 78F99467B9: sender non-delivery notification: 7AB27467B3
> Nov 10 06:27:22 lists postfix/qmgr[19489]: 78F99467B9: removed
> Nov 10 06:27:22 lists postfix/smtpd[12070]: 91402467B9: client=localhost[127.0.0.1]
> Nov 10 06:27:22 lists postfix/smtp[8390]: C159A467CE: to=<someone@hotmail.com>, relay=mx1.hotmail.com[65.55.92.152]:25, delay=0.8, delays=0.07/0.09/0.48/0.15, dsn=5.0.0, status=bounced (host mx1.hotmail.com[65.55.92.152] said: 550 SC-002 (SNT0-MC2-F38) Unfortunately, messages from ----IP---- weren't sent. Please contact your Internet service provider since part of their network is on our block list. You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors. (in reply to MAIL FROM command))
> Nov 10 06:27:22 lists postfix/smtp[8390]: C159A467CE: lost connection with mx1.hotmail.com[65.55.92.152] while sending RCPT TO
> Nov 10 06:27:22 lists postfix/cleanup[12193]: 9A555467C1: message-id=<20131110052722.9A555467C1@lists.DOMAIN>
> Nov 10 06:27:22 lists postfix/cleanup[12147]: 91402467B9: message-id=<1384061242-12402-mlmmj-2801aaca@lists.DOMAIN>
> Nov 10 06:27:22 lists postfix/pipe[12302]: 7AB27467B3: to=<listname+bounces-probe-someone=hotmail.com@lists.DOMAIN>, relay=mlmmj, delay=0.14, delays=0.08/0.01/0/0.05, dsn=2.0.0, status=sent (delivered via mlmmj service)
> Nov 10 06:27:22 lists postfix/qmgr[19489]: 7AB27467B3: removed
> Nov 10 06:27:22 lists postfix/qmgr[19489]: 9A555467C1: from=<>, size520, nrcpt=1 (queue active)
> Nov 10 06:27:22 lists postfix/bounce[12233]: C159A467CE: sender non-delivery notification: 9A555467C1
> Nov 10 06:27:22 lists postfix/qmgr[19489]: 91402467B9: from=<listname+bounces-probe-someone=hotmail.com@lists.DOMAIN>, size\x1009, nrcpt=1 (queue active)
> ---- END SNIP -------------------------------------------------------
> To me all I see here is that I have been blocked by hotmail and friends.
> (We are currently in the process of signing up for the junk mail program.)
> last time I recieved something on my own microsoft account was on the 24'th of Oktober,
> so I guess that today mlmmj will unsubscribe all the hotmail and live adresses :/
> (since I have bouncelife = 2592000)
> but on the other hand, I can see i still have some email adresses subscribed, where the domain was taken down somewehre arround the end of 2009!
> see next log snippet.
> ---- BEGIN SNIP -----------------------------------------------------
> Nov 24 13:05:49 lists postfix/qmgr[19489]: F2A1246824: from=<listname+bounces-probe-SOMEONE=tele2adsl.dk@lists.DOMAIN>, size—4, nrcpt=1 (queue active)
> Nov 24 13:05:49 lists postfix/qmgr[19489]: F1A2B46879: from=<listname+bounces-probe-SOMEONE=tele2adsl.dk@lists.DOMAIN>, size—5, nrcpt=1 (queue active)
> Nov 24 13:05:49 lists postfix/qmgr[19489]: F1B5246827: from=<listname+bounces-probe-SOMEONE=tele2adsl.dk@lists.DOMAIN>, size˜2, nrcpt=1 (queue active)
> Nov 24 13:05:49 lists postfix/qmgr[19489]: 3F30446978: from=<listname+bounces-probe-SOMEONE=lifecare.as@lists.DOMAIN>, size–4, nrcpt=1 (queue active)
> Nov 24 13:05:49 lists postfix/qmgr[19489]: 3F4AD467BE: from=<listname+bounces-probe-SOMEONE=tele2adsl.dk@lists.DOMAIN>, size—2, nrcpt=1 (queue active)
> Nov 24 13:05:49 lists postfix/error[617]: F2A1246824: to=<SOMEONE@tele2adsl.dk>, relay=none, delay\x176327, delays\x176327/0.06/0/0.04, dsn=4.4.3, statusÞferred (delivery temporarily suspended: Host or domain name not found. Name service error for name=tele2adsl.dk type=MX: Host not found, try again)
> Nov 24 13:05:49 lists postfix/qmgr[19489]: 3A13146A9C: from=<listname+bounces-probe-SOMEONE@lists.DOMAIN>, size–4, nrcpt=1 (queue active)
> Nov 24 13:05:49 lists postfix/error[618]: F1A2B46879: to=<SOMEONE@tele2adsl.dk>, relay=none, delay–500, delays–500/0.03/0/0.05, dsn=4.4.3, statusÞferred (delivery temporarily suspended: Host or domain name not found. Name service error for name=tele2adsl.dk type=MX: Host not found, try again)
> ---- END SNIP -------------------------------------------------------
> How is this possible?
> besides from me adding someone with these adresses recently, but I have arround 80 adresses from this domain, so it seems unlikely.
> (I add emails without validation, the adresses comes from lottery where the email adress is written on a piece of paper)
> This is the lines from mlmmj.operation.log for today, only suspecies thing here for me is the russian requesting help for the list.
> ---- BEGIN SNIP -----------------------------------------------------
> Sun Nov 24 02:02:26 2013 mlmmj-maintd: SOMEONE@inco.dk unsubscribed due to bouncing since  Thu Oct 24 13:51:45 2013
> Sun Nov 24 02:02:27 2013 mlmmj-recieve: sending mail from listname+bounces-help@lists.DOMAIN to owner
> Sun Nov 24 08:06:05 2013 mlmmj-sub: request for regular subscription from SOMEONE@youseeme.dk
> Sun Nov 24 09:41:35 2013 mlmmj-sub: SOMEONE@youseeme.dk confirmed subscription to regular list
> Sun Nov 24 09:41:35 2013 mlmmj-recieve: sending mail from listname+bounces-help@lists.DOMAIN to owner
> Sun Nov 24 10:17:03 2013 mlmmj-sub: request for regular subscription from SOMEONE@stofanet.dk
> Sun Nov 24 10:18:09 2013 mlmmj-sub: SOMEONE@stofanet.dk confirmed subscription to regular list
> Sun Nov 24 10:18:09 2013 mlmmj-recieve: sending mail from listname+bounces-help@lists.DOMAIN to owner
> Sun Nov 24 12:08:33 2013 mlmmj-unsub: SOMEONE@gmail.com requests unsubscribe from regular list
> Sun Nov 24 13:42:45 2013 SOMEONE@10ge.ru requested help
> ---- END SNIP -------------------------------------------------------
>
> And the content of mlmmj-maintd.lastrun.log
>
> ---- BEGIN SNIP -----------------------------------------------------
> Starting maintenance run at Sun Nov 24 14:00:01 2013
> clean_moderation(/var/spool/mlmmj/listname);
> clean_discarded(/var/spool/mlmmj/listname);
> clean_subconf(/var/spool/mlmmj/listname);
> clean_unsubconf(/var/spool/mlmmj/listname);
> resend_queue(/var/spool/mlmmj/listname, /usr/bin/mlmmj-send);
> resend_requeue(/var/spool/mlmmj/listname, /usr/bin/mlmmj-send);
> clean_nolongerbouncing(/var/spool/mlmmj/listname);
> unsub_bouncers(/var/spool/mlmmj/listname, /usr/bin/mlmmj-unsub);
> probe_bouncers(/var/spool/mlmmj/listname, /usr/bin/mlmmj-bounce);
> run_digests(/var/spool/mlmmj/listname, /usr/bin/mlmmj-send);
> ---- END SNIP -------------------------------------------------------
>
> I can provide you with the full log files if you wan't if so, do you have any ideay how to anonymise the files?
> in /etc/postfix/master.cf
>
> ---- BEGIN SNIP -----------------------------------------------------
> mlmmj unix - n n - - pipe
>          flags¿X
>          user=nobody:nogroup
>          argv=/usr/bin/mlmmj-recieve -L /var/spool/mlmmj/listname/ -s ${sender$
> ---- END SNIP -------------------------------------------------------
>
> in /etc/postfix/main.cf
>
> ---- BEGIN SNIP -----------------------------------------------------
> transport_maps >          proxy:hash:/etc/postfix/mlmmj_maillists
> virtual_mailbox_base = /var/mail
> virtual_mailbox_domains >          proxy:hash:/etc/postfix/mlmmj_domains
> virtual_mailbox_maps >          proxy:hash:/etc/postfix/mlmmj_maillists
> ---- END SNIP -------------------------------------------------------
>
> looking at the bouncelist is not a pretty sight :/ every hotmail and live address is there
> basically this confirms that hotmail and live are the majority in the bounce folder.
> I can't go thorough all the hotmail.lastmsg, but those I have looked at are there due to being blocked.
>
> I guess the first thing to do about the Microsoft blockade is to follow up what I will learn from the Microsoft 'junk mail program'
>
>
> since gmail is as big as it is, I looked in the bounce folder for gmail.com bounces, and luckily there is only one :)
>
> The diagnostics code code in the last message is
> Diagnostic-Code: smtp; 550-5.1.1 The email account that you tried to reach does
>      not exist. Please try 550-5.1.1 double-checking the recipient's email
>      address for typos or 550-5.1.1 unnecessary spaces. Learn more at 550 5.1.1
>      http://support.google.com/mail/bin/answer.py?answere96
>      pz10si9331677lbb.105 - gsmtp
>
> Would it be possible (and reasonable) to configure mlmmj to unsubsscribe in such cases immediately?
>   Any help is greatly appreciated
> kind Regards.
> Christian
> -----Original Message-----
>> From: "Ben Schmidt" <mail_ben_schmidt@yahoo.com.au>
>> To: "Christian Gleerup" <christian.gleerup@swn.nu>, mlmmj@mlmmj.org
>> Date: 23/11/2013 00:48
>> Subject: Re: [mlmmj] Problems with microsoft
>>
>> Hi, Christian,
>>
>> Here are some thoughts:
>>
>>> I use mlmmj to send newsletters, and I have the following problem, but
>>> now more serious since Microsoft have blocked the server.
>>
>> That's annoying. :-)
>>
>>> Problem 1.
>>> Our mail recipients sometimes receives a bunch of mails with the
>>> following message "some messaged could not be delivered. If you see
>>> this things are back to normal." They get many of these, so they
>>> complain.
>>> I understand that they get one mail telling them that everything
>>> Works, but why 10-40?
>>
>> It would be helpful to find out where those mails originate--is Mlmmj
>> sending a lot of mail to Postfix, or is Postfix sending a lot of mail to
>> the next server? There are a few things to look at: (1) your mail logs;
>> you should be able to find the messages going out, (2)
>> mlmmj.operations.log in the relevant listdir, (3)
>> mlmmj-maintd.lastrun.log in the relevant listdir (if you get to it
>> quickly enough after it happens), (4) the Message-ID and other headers
>> of the received 'duplicate' messages.
>>
>> Perhaps you could furnish us with some of that information. De-identify
>> it by making some small modifications to the email/IP/list addresses in
>> it if necessary.
>>
>>> Is it a configuration problem between mlmmj and postfix?
>>
>> Possibly.
>>
>>> I have the following settings for postfix and mlmmj that I think is
>>> relevant for the problem. But i don't really understand how they
>>> interact, could there be some configuration error so mlmmj fills
>>> postfix with a queue due to lack of respone.
>>>
>>> * /etc/postfix/main.cf
>>> bounce_queue_lifetime = 2d
>>> minimal_backoff_time = 1800s
>>
>> This could possibly be relevant if Postfix is trying to send mail to
>> mlmmj-receive, succeeding, but receiving a failure response; Postfix
>> will keep trying for 2 days to deliver the bounce message to Mlmmj,
>> which will keep receiving it and keep thinking the address is bouncing,
>> and keep sending bounce probes.
>>
>> You should be able to determine from your mail logs if this is happening
>> (you will see a lot of failed messages from Postfix itself--postmaster,
>> or mail_daemon or whatever it uses--to list+bounces addresses).
>>
>> It would also be helpful to know how Postfix and Mlmmj are linked? What
>> do you have in your config files to facilitate delivery of messages to
>> Mlmmj?
>>
>>> * in  'tunables' bouncelife
>>> 2592000
>>> (30 days)
>>
>> This shouldn't be too relevant; it's how long Mlmmj waits before giving
>> up and unsubscribing the user. If you changed this, bounce probes would
>> just turn into unsubscriptions; the cause of the problems wouldn't be
>> addressed.
>>
>>> Problem 2
>>> Microsoft think I am doing 'namespace mining', I know I don't. but
>>> maybe it is somehow connected to the problem above ?
>>
>> If this is truly the case, it is probably unrelated to the problem
>> above. If you are namespace mining, you are trying lots of addresses
>> @hotmail.com (or wherever), hoping to find real ones. In fact, you are
>> probably actually getting a lot of bounces for nonexistent addresses.
>> So, if, as a responsible mail host, I want to detect if you're namespace
>> mining, I would use the number of bounces due to nonexistent addresses
>> as a heuristic, and block you if you get a lot of them.
>>
>> A nonexistent address can't receive a lot of probe messages! It can't
>> receive anything. So it's probably not related to the problem above.
>>
>> However, it could be related to your bouncelife tunable. If an address
>> ceases to exist, because it's deleted; or in some cases, if an address
>> with wrong spelling is added to the list (e.g. without requiring
>> confirmation), Mlmmj is going to receive a bounce message about the
>> non-existent address. However, Mlmmj doesn't know whether that bounce
>> message is a permanent error or a temporary error (and in fact,
>> sometimes, due to misconfiguration, errors that seem permanent are
>> actually temporary, so best retried anyway). Therefore Mlmmj will keep
>> retrying the address--possibly every 2 hours (however often mlmmj-maintd
>> runs; I'm not sure if Mlmmj throttles delivery or not) for *30 days*,
>> and every time receive a bounce message due to the non-existent address.
>> That many bounces for non-existent addresses would definitely make you
>> look like you're namespace mining (if the watchdog software that uses
>> the heuristic isn't smart enough to realise they're all for the same
>> address or few addresses).
>>
>> You can check whether this is happening by looking in your listdir. The
>> last bounce for each currently-bouncing address is stored in the bounce
>> subdir, so you can read them, and see how many addresses (including how
>> many from Microsoft) are bouncing, and why.
>>
>> One reason for doing automatic bounce processing is to minimise
>> unnecessary bounces, by unsubscribing users before bounces to them
>> become suspicious or waste too much bandwidth; by making your bouncelife
>> so high, you've reduced the effectiveness of the feature.
>>
>>> Best Regards and I hope you can help me.
>>
>> No trouble. If you need more help, please furnish us with more
>> information: mail logs, mlmmj logs, message headers, configuration. All
>> this information is useful and necessary for properly tracking down
>> these problems.
>>
>> Ben.
>
>
>
>
>
>

Re: [mlmmj] Problems with microsoft

[Ben Schmidt]

O, and also, to state the obvious. If you're not running the latest
version of Mlmmj, upgrading is not a dumb idea. Bugs do get fixed in
every release (and between releases for that matter, but I can't expect
you to be running development versions!). I can't remember anything
recently that exactly addresses your problems, but maybe there was
something. We are due for a new release soon, as a few bugs have been
fixed since the last one. I will probably step into gear and get that
done in the next 2 months or so.

Ben.



On 27/11/13 8:02 AM, Ben Schmidt wrote:
> OK, so obviously after being blocked wasn't an ideal time to look at
> bounce messages and logs, because it's just telling you what you already
> know: you're blocked. To find out why, we need earlier data.
>
> So once you've jumped through the hoops to get Microsoft to unblock you,
> you should pay careful attention to what's in "bounce" for MS addresses
> and see what's going on. That may shed light on what caused you to get
> blocked. Also you should probably reduce your bouncelife tunable, as
> discussed. Also, to avoid a massive backlog of mail for MS addresses,
> you should perhaps remove them from the retry queue (clear out the
> bounces from Mlmmj's list dir and confirm with mail logs Mlmmj is no
> longer getting a stack of failed deliveries). That might help you avoid
> getting your unblock request denied, or getting blocked again straight
> away because Mlmmj is continuously trying to send probes. It would also
> avoid the addresses all being unsubscribed (though I guess actually it
> may be too late for that; O well; I suppose you can always put them back
> as long as you know what they were).
>
> You didn't seem to include any logs or other information to do with the
> other issue (people receiving multiple bounce probes). You may need to
> track down an instance of that occurring, and see what you can find in
> the logs. Getting samples of the repeat messages would be a good first
> step, looking at the headers, and trying to trace the deliveries back
> through your logs (mail server logs, then back to Mlmmj if necessary).
>
> Nevertheless, your logs and bounce files have confirmed that Mlmmj is
> wired up to Postfix OK, and that bounces are making it all the way back
> to Mlmmj.
>
> Regarding addresses on the list that should have already been removed:
> the most likely cause of this is bad permissions. Mlmmj's error checking
> isn't always brilliant, and sometimes you don't get an informative
> message when things go wrong (we are working on fixing this). Anyway,
> check that mlmmj-maintd runs with permission to access the listdir, and
> that all the files in the listdir have appropriate permissions for
> mlmmj-maintd, mlmmj when invoked via Postfix, and mlmmj when invoked any
> other ways you use (commandline/web interfaces). You may need to add
> users to groups, etc. to make it work. I notice you're using "nobody"
> for mlmmj, which isn't a good idea. See the tutorial/readme in the docs
> about integrating Mlmmj with Postfix for details of why, and how to do
> it better. Anyway, if one method of subscribing users creates files that
> mlmmj-maintd can't modify, it won't be able to unsubscribe them.
>
> I hope this helps,
>
> Ben.
>
>
>
> On 25/11/13 2:57 AM, Christian Gleerup wrote:
>> Hi Ben and List,
>> I'm been going through the logs, but I am not getting much wiser.
>>
>> I hope the information provided here is the correct one, ontherwise please
>> direct me as to what else to look for.
>>
>>
>> Here a snippet from /var/log/mail.log.1
>> ---- BEGIN SNIP -----------------------------------------------------
>> Nov 10 06:27:22 lists postfix/smtp[8385]: 1465545E8A: to=<someone@hotmail.com>,
>> relay=mx1.hotmail.com[65.55.37.72]:25, delay=0.94, delays=0.16/0.01/0.59/0.19,
>> dsn=5.0.0, status=bounced (host mx1.hotmail.com[65.55.37.72] said: 550 SC-002
>> (COL0-MC1-F1) Unfortunately, messages from ----IP---- weren't sent. Please
>> contact your Internet service provider since part of their network is on our
>> block list. You can also refer your provider to
>> http://mail.live.com/mail/troubleshooting.aspx#errors. (in reply to MAIL FROM
>> command))
>> Nov 10 06:27:22 lists postfix/smtp[8385]: 1465545E8A: lost connection with
>> mx1.hotmail.com[65.55.37.72] while sending RCPT TO
>> Nov 10 06:27:22 lists postfix/cleanup[12147]: 108F1467C2:
>> message-id=<20131110052722.108F1467C2@lists.DOMAIN>
>> Nov 10 06:27:22 lists postfix/cleanup[11855]: 0B321467BD:
>> message-id=<1384061241-12392-mlmmj-5f63aeea@lists.DOMAIN>
>> Nov 10 06:27:22 lists postfix/smtp[8334]: 50697467AF: to=<someone@live.dk>,
>> relay=mx4.hotmail.com[65.55.92.136]:25, delay=0.73, delays=0.1/0.01/0.47/0.15,
>> dsn=5.0.0, status=bounced (host mx4.hotmail.com[65.55.92.136] said: 550 SC-002
>> (SNT0-MC1-F19) Unfortunately, messages from ----IP---- weren't sent. Please
>> contact your Internet service provider since part of their network is on our
>> block list. You can also refer your provider to
>> http://mail.live.com/mail/troubleshooting.aspx#errors. (in reply to MAIL FROM
>> command))
>> Nov 10 06:27:22 lists postfix/smtp[8334]: 50697467AF: lost connection with
>> mx4.hotmail.com[65.55.92.136] while sending RCPT TO
>> Nov 10 06:27:22 lists postfix/cleanup[12193]: 17834467C3:
>> message-id=<20131110052722.17834467C3@lists.DOMAIN>
>> Nov 10 06:27:22 lists postfix/smtpd[12070]: disconnect from localhost[127.0.0.1]
>> Nov 10 06:27:22 lists postfix/bounce[12246]: 1465545E8A: sender non-delivery
>> notification: 108F1467C2
>> Nov 10 06:27:22 lists postfix/qmgr[19489]: 17834467C3: from=<>, size464,
>> nrcpt=1 (queue active)
>> Nov 10 06:27:22 lists postfix/bounce[12248]: 50697467AF: sender non-delivery
>> notification: 17834467C3
>> Nov 10 06:27:22 lists postfix/qmgr[19489]: 1465545E8A: removed
>> Nov 10 06:27:22 lists postfix/qmgr[19489]: 50697467AF: removed
>> Nov 10 06:27:22 lists postfix/qmgr[19489]: 108F1467C2: from=<>, size524,
>> nrcpt=1 (queue active)
>> Nov 10 06:27:22 lists postfix/qmgr[19489]: 0B321467BD:
>> from=<listname+bounces-probe-someone=hotmail.com@lists.DOMAIN>, size\x1015,
>> nrcpt=1 (queue active)
>> Nov 10 06:27:22 lists postfix/smtpd[12070]: connect from localhost[127.0.0.1]
>> Nov 10 06:27:22 lists postfix/smtpd[12070]: 42F9545E8A: client=localhost[127.0.0.1]
>> Nov 10 06:27:22 lists postfix/cleanup[12331]: 42F9545E8A:
>> message-id=<1384061242-12397-mlmmj-70337e4e@lists.DOMAIN>
>>
>> Nov 10 06:27:22 lists postfix/pipe[11679]: 108F1467C2:
>> to=<listname+bounces-probe-someone=hotmail.com@lists.DOMAIN>, relay=mlmmj,
>> delay=0.41, delays=0.17/0.01/0/0.23, dsn=2.0.0, status=sent (delivered via mlmmj
>> service)
>> Nov 10 06:27:22 lists postfix/qmgr[19489]: 108F1467C2: removed
>> Nov 10 06:27:22 lists postfix/pipe[12334]: 17834467C3:
>> to=<listname+bounces-probe-someone=live.dk@lists.DOMAIN>, relay=mlmmj,
>> delay=0.38, delays=0.09/0.06/0/0.23, dsn=2.0.0, status=sent (delivered via mlmmj
>> service)
>> Nov 10 06:27:22 lists postfix/qmgr[19489]: 17834467C3: removed
>> Nov 10 06:27:22 lists postfix/qmgr[19489]: 42F9545E8A:
>> from=<listname+bounces-probe-someone=hotmail.com@lists.DOMAIN>, size\x1009,
>> nrcpt=1 (queue active)
>> Nov 10 06:27:22 lists postfix/smtpd[12070]: disconnect from localhost[127.0.0.1]
>>
>> Nov 10 06:27:22 lists postfix/smtp[8370]: 78F99467B9: to=<someone@hotmail.com>,
>> relay=mx3.hotmail.com[65.54.188.126]:25, delay=0.88, delays=0.15/0/0.55/0.18,
>> dsn=5.0.0, status=bounced (host mx3.hotmail.com[65.54.188.126] said: 550 SC-002
>> (BAY0-MC4-F51) Unfortunately, messages from ----IP---- weren't sent. Please
>> contact your Internet service provider since part of their network is on our
>> block list. You can also refer your provider to
>> http://mail.live.com/mail/troubleshooting.aspx#errors. (in reply to MAIL FROM
>> command))
>> Nov 10 06:27:22 lists postfix/smtp[8370]: 78F99467B9: lost connection with
>> mx3.hotmail.com[65.54.188.126] while sending RCPT TO
>> Nov 10 06:27:22 lists postfix/cleanup[11855]: 7AB27467B3:
>> message-id=<20131110052722.7AB27467B3@lists.DOMAIN>
>> Nov 10 06:27:22 lists postfix/qmgr[19489]: 7AB27467B3: from=<>, size514,
>> nrcpt=1 (queue active)
>> Nov 10 06:27:22 lists postfix/smtpd[12070]: connect from localhost[127.0.0.1]
>> Nov 10 06:27:22 lists postfix/bounce[12246]: 78F99467B9: sender non-delivery
>> notification: 7AB27467B3
>> Nov 10 06:27:22 lists postfix/qmgr[19489]: 78F99467B9: removed
>> Nov 10 06:27:22 lists postfix/smtpd[12070]: 91402467B9: client=localhost[127.0.0.1]
>> Nov 10 06:27:22 lists postfix/smtp[8390]: C159A467CE: to=<someone@hotmail.com>,
>> relay=mx1.hotmail.com[65.55.92.152]:25, delay=0.8, delays=0.07/0.09/0.48/0.15,
>> dsn=5.0.0, status=bounced (host mx1.hotmail.com[65.55.92.152] said: 550 SC-002
>> (SNT0-MC2-F38) Unfortunately, messages from ----IP---- weren't sent. Please
>> contact your Internet service provider since part of their network is on our
>> block list. You can also refer your provider to
>> http://mail.live.com/mail/troubleshooting.aspx#errors. (in reply to MAIL FROM
>> command))
>> Nov 10 06:27:22 lists postfix/smtp[8390]: C159A467CE: lost connection with
>> mx1.hotmail.com[65.55.92.152] while sending RCPT TO
>> Nov 10 06:27:22 lists postfix/cleanup[12193]: 9A555467C1:
>> message-id=<20131110052722.9A555467C1@lists.DOMAIN>
>> Nov 10 06:27:22 lists postfix/cleanup[12147]: 91402467B9:
>> message-id=<1384061242-12402-mlmmj-2801aaca@lists.DOMAIN>
>> Nov 10 06:27:22 lists postfix/pipe[12302]: 7AB27467B3:
>> to=<listname+bounces-probe-someone=hotmail.com@lists.DOMAIN>, relay=mlmmj,
>> delay=0.14, delays=0.08/0.01/0/0.05, dsn=2.0.0, status=sent (delivered via mlmmj
>> service)
>> Nov 10 06:27:22 lists postfix/qmgr[19489]: 7AB27467B3: removed
>> Nov 10 06:27:22 lists postfix/qmgr[19489]: 9A555467C1: from=<>, size520,
>> nrcpt=1 (queue active)
>> Nov 10 06:27:22 lists postfix/bounce[12233]: C159A467CE: sender non-delivery
>> notification: 9A555467C1
>> Nov 10 06:27:22 lists postfix/qmgr[19489]: 91402467B9:
>> from=<listname+bounces-probe-someone=hotmail.com@lists.DOMAIN>, size\x1009,
>> nrcpt=1 (queue active)
>> ---- END SNIP -------------------------------------------------------
>> To me all I see here is that I have been blocked by hotmail and friends.
>> (We are currently in the process of signing up for the junk mail program.)
>> last time I recieved something on my own microsoft account was on the 24'th of
>> Oktober,
>> so I guess that today mlmmj will unsubscribe all the hotmail and live adresses :/
>> (since I have bouncelife = 2592000)
>> but on the other hand, I can see i still have some email adresses subscribed,
>> where the domain was taken down somewehre arround the end of 2009!
>> see next log snippet.
>> ---- BEGIN SNIP -----------------------------------------------------
>> Nov 24 13:05:49 lists postfix/qmgr[19489]: F2A1246824:
>> from=<listname+bounces-probe-SOMEONE=tele2adsl.dk@lists.DOMAIN>, size—4,
>> nrcpt=1 (queue active)
>> Nov 24 13:05:49 lists postfix/qmgr[19489]: F1A2B46879:
>> from=<listname+bounces-probe-SOMEONE=tele2adsl.dk@lists.DOMAIN>, size—5,
>> nrcpt=1 (queue active)
>> Nov 24 13:05:49 lists postfix/qmgr[19489]: F1B5246827:
>> from=<listname+bounces-probe-SOMEONE=tele2adsl.dk@lists.DOMAIN>, size˜2,
>> nrcpt=1 (queue active)
>> Nov 24 13:05:49 lists postfix/qmgr[19489]: 3F30446978:
>> from=<listname+bounces-probe-SOMEONE=lifecare.as@lists.DOMAIN>, size–4,
>> nrcpt=1 (queue active)
>> Nov 24 13:05:49 lists postfix/qmgr[19489]: 3F4AD467BE:
>> from=<listname+bounces-probe-SOMEONE=tele2adsl.dk@lists.DOMAIN>, size—2,
>> nrcpt=1 (queue active)
>> Nov 24 13:05:49 lists postfix/error[617]: F2A1246824: to=<SOMEONE@tele2adsl.dk>,
>> relay=none, delay\x176327, delays\x176327/0.06/0/0.04, dsn=4.4.3, statusÞferred
>> (delivery temporarily suspended: Host or domain name not found. Name service
>> error for name=tele2adsl.dk type=MX: Host not found, try again)
>> Nov 24 13:05:49 lists postfix/qmgr[19489]: 3A13146A9C:
>> from=<listname+bounces-probe-SOMEONE@lists.DOMAIN>, size–4, nrcpt=1 (queue
>> active)
>> Nov 24 13:05:49 lists postfix/error[618]: F1A2B46879: to=<SOMEONE@tele2adsl.dk>,
>> relay=none, delay–500, delays–500/0.03/0/0.05, dsn=4.4.3, statusÞferred
>> (delivery temporarily suspended: Host or domain name not found. Name service
>> error for name=tele2adsl.dk type=MX: Host not found, try again)
>> ---- END SNIP -------------------------------------------------------
>> How is this possible?
>> besides from me adding someone with these adresses recently, but I have arround
>> 80 adresses from this domain, so it seems unlikely.
>> (I add emails without validation, the adresses comes from lottery where the
>> email adress is written on a piece of paper)
>> This is the lines from mlmmj.operation.log for today, only suspecies thing here
>> for me is the russian requesting help for the list.
>> ---- BEGIN SNIP -----------------------------------------------------
>> Sun Nov 24 02:02:26 2013 mlmmj-maintd: SOMEONE@inco.dk unsubscribed due to
>> bouncing since  Thu Oct 24 13:51:45 2013
>> Sun Nov 24 02:02:27 2013 mlmmj-recieve: sending mail from
>> listname+bounces-help@lists.DOMAIN to owner
>> Sun Nov 24 08:06:05 2013 mlmmj-sub: request for regular subscription from
>> SOMEONE@youseeme.dk
>> Sun Nov 24 09:41:35 2013 mlmmj-sub: SOMEONE@youseeme.dk confirmed subscription
>> to regular list
>> Sun Nov 24 09:41:35 2013 mlmmj-recieve: sending mail from
>> listname+bounces-help@lists.DOMAIN to owner
>> Sun Nov 24 10:17:03 2013 mlmmj-sub: request for regular subscription from
>> SOMEONE@stofanet.dk
>> Sun Nov 24 10:18:09 2013 mlmmj-sub: SOMEONE@stofanet.dk confirmed subscription
>> to regular list
>> Sun Nov 24 10:18:09 2013 mlmmj-recieve: sending mail from
>> listname+bounces-help@lists.DOMAIN to owner
>> Sun Nov 24 12:08:33 2013 mlmmj-unsub: SOMEONE@gmail.com requests unsubscribe
>> from regular list
>> Sun Nov 24 13:42:45 2013 SOMEONE@10ge.ru requested help
>> ---- END SNIP -------------------------------------------------------
>>
>> And the content of mlmmj-maintd.lastrun.log
>>
>> ---- BEGIN SNIP -----------------------------------------------------
>> Starting maintenance run at Sun Nov 24 14:00:01 2013
>> clean_moderation(/var/spool/mlmmj/listname);
>> clean_discarded(/var/spool/mlmmj/listname);
>> clean_subconf(/var/spool/mlmmj/listname);
>> clean_unsubconf(/var/spool/mlmmj/listname);
>> resend_queue(/var/spool/mlmmj/listname, /usr/bin/mlmmj-send);
>> resend_requeue(/var/spool/mlmmj/listname, /usr/bin/mlmmj-send);
>> clean_nolongerbouncing(/var/spool/mlmmj/listname);
>> unsub_bouncers(/var/spool/mlmmj/listname, /usr/bin/mlmmj-unsub);
>> probe_bouncers(/var/spool/mlmmj/listname, /usr/bin/mlmmj-bounce);
>> run_digests(/var/spool/mlmmj/listname, /usr/bin/mlmmj-send);
>> ---- END SNIP -------------------------------------------------------
>>
>> I can provide you with the full log files if you wan't if so, do you have any
>> ideay how to anonymise the files?
>> in /etc/postfix/master.cf
>>
>> ---- BEGIN SNIP -----------------------------------------------------
>> mlmmj unix - n n - - pipe
>>          flags¿X
>>          user=nobody:nogroup
>>          argv=/usr/bin/mlmmj-recieve -L /var/spool/mlmmj/listname/ -s ${sender$
>> ---- END SNIP -------------------------------------------------------
>>
>> in /etc/postfix/main.cf
>>
>> ---- BEGIN SNIP -----------------------------------------------------
>> transport_maps >>          proxy:hash:/etc/postfix/mlmmj_maillists
>> virtual_mailbox_base = /var/mail
>> virtual_mailbox_domains >>          proxy:hash:/etc/postfix/mlmmj_domains
>> virtual_mailbox_maps >>          proxy:hash:/etc/postfix/mlmmj_maillists
>> ---- END SNIP -------------------------------------------------------
>>
>> looking at the bouncelist is not a pretty sight :/ every hotmail and live
>> address is there
>> basically this confirms that hotmail and live are the majority in the bounce
>> folder.
>> I can't go thorough all the hotmail.lastmsg, but those I have looked at are
>> there due to being blocked.
>>
>> I guess the first thing to do about the Microsoft blockade is to follow up what
>> I will learn from the Microsoft 'junk mail program'
>>
>>
>> since gmail is as big as it is, I looked in the bounce folder for gmail.com
>> bounces, and luckily there is only one :)
>>
>> The diagnostics code code in the last message is
>> Diagnostic-Code: smtp; 550-5.1.1 The email account that you tried to reach does
>>      not exist. Please try 550-5.1.1 double-checking the recipient's email
>>      address for typos or 550-5.1.1 unnecessary spaces. Learn more at 550 5.1.1
>>      http://support.google.com/mail/bin/answer.py?answere96
>>      pz10si9331677lbb.105 - gsmtp
>>
>> Would it be possible (and reasonable) to configure mlmmj to unsubsscribe in such
>> cases immediately?
>>   Any help is greatly appreciated
>> kind Regards.
>> Christian
>> -----Original Message-----
>>> From: "Ben Schmidt" <mail_ben_schmidt@yahoo.com.au>
>>> To: "Christian Gleerup" <christian.gleerup@swn.nu>, mlmmj@mlmmj.org
>>> Date: 23/11/2013 00:48
>>> Subject: Re: [mlmmj] Problems with microsoft
>>>
>>> Hi, Christian,
>>>
>>> Here are some thoughts:
>>>
>>>> I use mlmmj to send newsletters, and I have the following problem, but
>>>> now more serious since Microsoft have blocked the server.
>>>
>>> That's annoying. :-)
>>>
>>>> Problem 1.
>>>> Our mail recipients sometimes receives a bunch of mails with the
>>>> following message "some messaged could not be delivered. If you see
>>>> this things are back to normal." They get many of these, so they
>>>> complain.
>>>> I understand that they get one mail telling them that everything
>>>> Works, but why 10-40?
>>>
>>> It would be helpful to find out where those mails originate--is Mlmmj
>>> sending a lot of mail to Postfix, or is Postfix sending a lot of mail to
>>> the next server? There are a few things to look at: (1) your mail logs;
>>> you should be able to find the messages going out, (2)
>>> mlmmj.operations.log in the relevant listdir, (3)
>>> mlmmj-maintd.lastrun.log in the relevant listdir (if you get to it
>>> quickly enough after it happens), (4) the Message-ID and other headers
>>> of the received 'duplicate' messages.
>>>
>>> Perhaps you could furnish us with some of that information. De-identify
>>> it by making some small modifications to the email/IP/list addresses in
>>> it if necessary.
>>>
>>>> Is it a configuration problem between mlmmj and postfix?
>>>
>>> Possibly.
>>>
>>>> I have the following settings for postfix and mlmmj that I think is
>>>> relevant for the problem. But i don't really understand how they
>>>> interact, could there be some configuration error so mlmmj fills
>>>> postfix with a queue due to lack of respone.
>>>>
>>>> * /etc/postfix/main.cf
>>>> bounce_queue_lifetime = 2d
>>>> minimal_backoff_time = 1800s
>>>
>>> This could possibly be relevant if Postfix is trying to send mail to
>>> mlmmj-receive, succeeding, but receiving a failure response; Postfix
>>> will keep trying for 2 days to deliver the bounce message to Mlmmj,
>>> which will keep receiving it and keep thinking the address is bouncing,
>>> and keep sending bounce probes.
>>>
>>> You should be able to determine from your mail logs if this is happening
>>> (you will see a lot of failed messages from Postfix itself--postmaster,
>>> or mail_daemon or whatever it uses--to list+bounces addresses).
>>>
>>> It would also be helpful to know how Postfix and Mlmmj are linked? What
>>> do you have in your config files to facilitate delivery of messages to
>>> Mlmmj?
>>>
>>>> * in  'tunables' bouncelife
>>>> 2592000
>>>> (30 days)
>>>
>>> This shouldn't be too relevant; it's how long Mlmmj waits before giving
>>> up and unsubscribing the user. If you changed this, bounce probes would
>>> just turn into unsubscriptions; the cause of the problems wouldn't be
>>> addressed.
>>>
>>>> Problem 2
>>>> Microsoft think I am doing 'namespace mining', I know I don't. but
>>>> maybe it is somehow connected to the problem above ?
>>>
>>> If this is truly the case, it is probably unrelated to the problem
>>> above. If you are namespace mining, you are trying lots of addresses
>>> @hotmail.com (or wherever), hoping to find real ones. In fact, you are
>>> probably actually getting a lot of bounces for nonexistent addresses.
>>> So, if, as a responsible mail host, I want to detect if you're namespace
>>> mining, I would use the number of bounces due to nonexistent addresses
>>> as a heuristic, and block you if you get a lot of them.
>>>
>>> A nonexistent address can't receive a lot of probe messages! It can't
>>> receive anything. So it's probably not related to the problem above.
>>>
>>> However, it could be related to your bouncelife tunable. If an address
>>> ceases to exist, because it's deleted; or in some cases, if an address
>>> with wrong spelling is added to the list (e.g. without requiring
>>> confirmation), Mlmmj is going to receive a bounce message about the
>>> non-existent address. However, Mlmmj doesn't know whether that bounce
>>> message is a permanent error or a temporary error (and in fact,
>>> sometimes, due to misconfiguration, errors that seem permanent are
>>> actually temporary, so best retried anyway). Therefore Mlmmj will keep
>>> retrying the address--possibly every 2 hours (however often mlmmj-maintd
>>> runs; I'm not sure if Mlmmj throttles delivery or not) for *30 days*,
>>> and every time receive a bounce message due to the non-existent address.
>>> That many bounces for non-existent addresses would definitely make you
>>> look like you're namespace mining (if the watchdog software that uses
>>> the heuristic isn't smart enough to realise they're all for the same
>>> address or few addresses).
>>>
>>> You can check whether this is happening by looking in your listdir. The
>>> last bounce for each currently-bouncing address is stored in the bounce
>>> subdir, so you can read them, and see how many addresses (including how
>>> many from Microsoft) are bouncing, and why.
>>>
>>> One reason for doing automatic bounce processing is to minimise
>>> unnecessary bounces, by unsubscribing users before bounces to them
>>> become suspicious or waste too much bandwidth; by making your bouncelife
>>> so high, you've reduced the effectiveness of the feature.
>>>
>>>> Best Regards and I hope you can help me.
>>>
>>> No trouble. If you need more help, please furnish us with more
>>> information: mail logs, mlmmj logs, message headers, configuration. All
>>> this information is useful and necessary for properly tracking down
>>> these problems.
>>>
>>> Ben.
>>
>>
>>
>>
>>
>>
>
>
>

Re: [mlmmj] Problems with microsoft

[Ben Schmidt]

On 29/11/13 4:58 AM, Christian Gleerup wrote:
> I am considering just copying the 8 binaries
> (mlmmj-(bounce,list,maintd,process, recieve, send, sub, unsub)) in
> ~/mlmmj-1.2.18.0/src/ to /usr/bin/ where my old mlmmj files are.
>
> is there something I should be aware of or should it just Work?
> (I will give the files same owership, I will delete mlmmj-make-ml and mlmmj-make-ml.sh)

I never used a version as old as 1.2.16, so I'm not 100% sure.

However...

Take a bit of time to update your list texts as Mlmmj 1.2.18 uses a new
list text mechanism. As well as list-specific ones, there are
system-wide ones (can't quite remember where; maybe
/usr/local/share/mlmmj) which Mlmmj falls back to when it can't find
list-specific ones (and IIRC, it doesn't fall back in the most optimal
way, so it's best to update both).

Also note how mlmmj-receive has had the spelling fixed. "make install"
installs a symlink with the old name, I think, so old entries in mail
server config continue to work. You might prefer to update your mail
server config.

Apart from that, I think replacing the binaries will work. But no
guarantees!

> (I have upgraded the installation so I now have 768MB of ram, so I
> hope there will be no more 'log blackouts')

Cool. I had similar problems at one stage, where processes were killed
due to lack of memory. In fact, that's one reason I got into Mlmmj in
the first place, as its minimalist design makes it suitable for
constrained-memory applications. I have also found the software Monit
(single-host M/Monit) very helpful to keep essential services running
despite the kernel killing things (or other process death). By
installing it into inittab, Monit can be made to never die (or at least
be guaranteed a resurrection if it does).

Ben.

Re: [mlmmj] Problems with microsoft

[Ben Schmidt]

On 4/12/13 6:35 AM, Christian Gleerup wrote:
> I have now updated to 1.2.18, and id did take some time to get the
> list texts right, but it seems to be worken fine now. (I did a make
> install, and then moved the installed binaries since the old binaries
> was installed in a different location)

Perhaps you wanted to give the --prefix argument to configure or
something. Nevertheless, I'm sure what you did will have worked fine.

> But obviously I still have the problem with sending to hotmail
> accounts since that is a different problem... but as i noted in
> another mail, hotmail did, when they accepted mails from me, tell me
> to take it easy with sending that many mails in a little period of
> time.
>
> so the question is, how can i tell mlmmj+postfix to only deliver x
> mails pr. hour to the hotmail domain?

I think this is something that should be done at the MTA level, not via
Mlmmj. Multiple mailing lists and any other mail routed through the
server will all contribute to Hotmail's limit, so only Postfix as the
central point of control has enough information to enforce rate
limiting. A quick Google turns up this, which should give you a starting
point for investigation:

http://steam.io/2013/04/01/postfix-rate-limiting/

> Also, It will be interesting to see if mlmmj unsubscribes those
> subscribers after bouncelife (\x15 days)  with the accounts from the
> domain that was taken down 2009.

Indeed, it will. If not, something is broken somewhere, and we will have
to investigate further....

Smiles,

Ben.

Re: [mlmmj] Problems with microsoft

[Ben Schmidt]

> http://steam.io/2013/04/01/postfix-rate-limiting/

P.S. Although the post is dated April 1, I don't think it's an April
Fool's joke. The information looks right and aligns with the Postfix
documentation as far as I can tell at a quick glance.

Smiles,

Ben.

Re: [mlmmj] Problems with microsoft

[Ben Schmidt]

Hi, Christian,

Mlmmj-maintd should clean it; it will send a probe, wait a while in case
a bounce comes back, and if it doesn't, it will remove the bounce. If
everything is working properly....

Ben.



On 14/12/13 5:11 AM, Christian Gleerup wrote:
> iw_mail p{margin:0;padding:0;}
> Hello list
>
> I can gladly inform you that my server has finally been allowed to send to Microsoft Again.
>
> Yesterday i manually removed a bunch of email addresses where Microsoft servers had responded 550.
> I removed via command line.
>
> but now when i look in the '.../list/bounce' folder, all the addresses are still there.
> is mlmmj going to clean up here or should I remove them manually?
>
> I can see postfix is still trying to deliver the mails
>
> kind regards
> Christian
>
> -----Original Message-----
>> From: "Ben Schmidt" <mail_ben_schmidt@yahoo.com.au>
>> To: "Christian Gleerup" <christian.gleerup@swn.nu>, mlmmj@mlmmj.org
>> Date: 03/12/2013 22:19
>> Subject: Re: [mlmmj] Problems with microsoft
>>
>> On 4/12/13 6:35 AM, Christian Gleerup wrote:
>>> I have now updated to 1.2.18, and id did take some time to get the
>>> list texts right, but it seems to be worken fine now. (I did a make
>>> install, and then moved the installed binaries since the old binaries
>>> was installed in a different location)
>>
>> Perhaps you wanted to give the --prefix argument to configure or
>> something. Nevertheless, I'm sure what you did will have worked fine.
>>
>>> But obviously I still have the problem with sending to hotmail
>>> accounts since that is a different problem... but as i noted in
>>> another mail, hotmail did, when they accepted mails from me, tell me
>>> to take it easy with sending that many mails in a little period of
>>> time.
>>>
>>> so the question is, how can i tell mlmmj+postfix to only deliver x
>>> mails pr. hour to the hotmail domain?
>>
>> I think this is something that should be done at the MTA level, not via
>> Mlmmj. Multiple mailing lists and any other mail routed through the
>> server will all contribute to Hotmail's limit, so only Postfix as the
>> central point of control has enough information to enforce rate
>> limiting. A quick Google turns up this, which should give you a starting
>> point for investigation:
>>
>> http://steam.io/2013/04/01/postfix-rate-limiting/
>>
>>> Also, It will be interesting to see if mlmmj unsubscribes those
>>> subscribers after bouncelife (\x15 days)  with the accounts from the
>>> domain that was taken down 2009.
>>
>> Indeed, it will. If not, something is broken somewhere, and we will have
>> to investigate further....
>>
>> Smiles,
>>
>> Ben.
>
>
>
>
>
>

Re: [mlmmj] Problems with microsoft

[Ben Schmidt]

mlmmj-maintd should leave the bounce in the bounce subdir, and send a probe. If it 
gets no bounce for the probe for a while, it will remove the bounce assuming the 
user is no longer bouncing. If it does get a bounce to the probe, it will continue 
probing for a while, but eventually unsubscribe the user for bouncing for too long.

I think some of the amounts of time it takes to do these things are currently 
hard-coded. One day this will be fixed, but it's fairly low on the priority list.

It is possible that if you unsubscribe a user manually, Mlmmj will keep probing 
them. If that does happen, it's a bug, I suppose, and should be fixed.

Ben.



On 15/12/13 9:44 PM, Christian Gleerup wrote:
 > Hello Ben
 >
 > I was a bit impation, and manually deleted those in the bounce folder. (currently
 > a bit paranoid about getting trappend in Microsofts spam list again).
 >
 > Regarding mlmmj-maintd, it is scheduled to run every second hour, and I also tried
 > to run it manually (without a lot of waiting tough)
 > mlmmj-maind is setup in cron.d/mlmmj almost identical to the sample on mlmmj's
 > website.
 >
 > 0 */2 * * * root /usr/bin/test -x /usr/bin/mlmmj-maintd && /usr/bin/mlmmj-maintd
 > -F -d /var/spool/mlmmj
 >
 > On the other hand, I don't really understand how mlmmj-maintd works in the
 > following case:
 >
 > 1) mlmmj sends a mail to a some subscriber
 >
 > 2) the recipent sends 550, non-existing user.
 >
 > 3) I manually parse the 'bounce files' and identify the '550, non-existing user'
 >
 > 4) I manully unsubscribe the 'non-existing user' from command line
 >
 > ---> at this point, I would expect no more mails getting queued since the user is
 > no longer subscriber?
 >
 > * I would expect mlmmj to either remove the mail from the bounce list when
 > unsubscribing, or to remove it on the Next mlmmj-maintd run since it is no longer
 > a subscriber.
 >
 > if mlmmj sends a probe for 'removal' detection, isn't it going to stay in the
 > bounce directory as long as the server responds with 550?
 >
 > Is there something I am missing?
 >
 > kind regards
 >
 > Christian
 >
 > -----Original Message-----
 >  > From: "Ben Schmidt" <mail_ben_schmidt@yahoo.com.au>
 >  > To: "Christian Gleerup" <christian.gleerup@swn.nu>
 >  > Date: 15/12/2013 02:09
 >  > Subject: Re: [mlmmj] Problems with microsoft
 >  >
 >  > Hi, Christian,
 >  >
 >  > Mlmmj-maintd should clean it; it will send a probe, wait a while in case
 >  > a bounce comes back, and if it doesn't, it will remove the bounce. If
 >  > everything is working properly....
 >  >
 >  > Ben.
 >  >
 >  >
 >  >
 >  > On 14/12/13 5:11 AM, Christian Gleerup wrote:
 >  > > iw_mail p{margin:0;padding:0;}
 >  > > Hello list
 >  > >
 >  > > I can gladly inform you that my server has finally been allowed to send to
 > Microsoft Again.
 >  > >
 >  > > Yesterday i manually removed a bunch of email addresses where Microsoft
 > servers had responded 550.
 >  > > I removed via command line.
 >  > >
 >  > > but now when i look in the '.../list/bounce' folder, all the addresses are
 > still there.
 >  > > is mlmmj going to clean up here or should I remove them manually?
 >  > >
 >  > > I can see postfix is still trying to deliver the mails
 >  > >
 >  > > kind regards
 >  > > Christian
 >  > >
 >  > > -----Original Message-----
 >  > >> From: "Ben Schmidt" <mail_ben_schmidt@yahoo.com.au
 > <mailto:mail_ben_schmidt@yahoo.com.au>>
 >  > >> To: "Christian Gleerup" <christian.gleerup@swn.nu
 > <mailto:christian.gleerup@swn.nu>>, mlmmj@mlmmj.org <mailto:mlmmj@mlmmj.org>
 >  > >> Date: 03/12/2013 22:19
 >  > >> Subject: Re: [mlmmj] Problems with microsoft
 >  > >>
 >  > >> On 4/12/13 6:35 AM, Christian Gleerup wrote:
 >  > >>> I have now updated to 1.2.18, and id did take some time to get the
 >  > >>> list texts right, but it seems to be worken fine now. (I did a make
 >  > >>> install, and then moved the installed binaries since the old binaries
 >  > >>> was installed in a different location)
 >  > >>
 >  > >> Perhaps you wanted to give the --prefix argument to configure or
 >  > >> something. Nevertheless, I'm sure what you did will have worked fine.
 >  > >>
 >  > >>> But obviously I still have the problem with sending to hotmail
 >  > >>> accounts since that is a different problem... but as i noted in
 >  > >>> another mail, hotmail did, when they accepted mails from me, tell me
 >  > >>> to take it easy with sending that many mails in a little period of
 >  > >>> time.
 >  > >>>
 >  > >>> so the question is, how can i tell mlmmj+postfix to only deliver x
 >  > >>> mails pr. hour to the hotmail domain?
 >  > >>
 >  > >> I think this is something that should be done at the MTA level, not via
 >  > >> Mlmmj. Multiple mailing lists and any other mail routed through the
 >  > >> server will all contribute to Hotmail's limit, so only Postfix as the
 >  > >> central point of control has enough information to enforce rate
 >  > >> limiting. A quick Google turns up this, which should give you a starting
 >  > >> point for investigation:
 >  > >>
 >  > >> http://steam.io/2013/04/01/postfix-rate-limiting/
 >  > >>
 >  > >>> Also, It will be interesting to see if mlmmj unsubscribes those
 >  > >>> subscribers after bouncelife (\x15 days)  with the accounts from the
 >  > >>> domain that was taken down 2009.
 >  > >>
 >  > >> Indeed, it will. If not, something is broken somewhere, and we will have
 >  > >> to investigate further....
 >  > >>
 >  > >> Smiles,
 >  > >>
 >  > >> Ben.
 >  > >
 >  > >
 >  > >
 >  > >
 >  > >
 >  > >

Re: [mlmmj] Problems with microsoft

[Ben Schmidt]

On 19/12/13 5:32 AM, Christian Gleerup wrote:
> Hi Ben and list,
>
> I was wondering, what if I used the -B option to simulate a confirmed
> unsubscription, would that do the trick, or would this problem need a
> code patch to be solved?

I'm not sure what -B option you're talking about. The option that seems
most relevant would be the -R option to mlmmj-unsub, but I'm not sure
that it would work without a confirmation request first having been
sent. It might, though. It's certainly not intended to be used manually,
though. You can always unconditionally unsubscribe the user, without
notifying anybody, with -q -s on the commandline.

> I was thinking that, at the place where the unsubscription is finally
> effectuated. The function should do a quick cleanup in the bounce
> folder for the particullar email adress being unsubscribed.

It certainly wouldn't hurt for bounces, etc. to be cleaned up when users
are unsubscribed, if this isn't already done.

> I looked a bit at the code, but erhm, it is a  bit hard to read for
> me, if someone can point me in the right direction, I can spare some
> time to try to fix this.

The code does have some quirks, but once you get a feel for it, it's
quite straightforward.

The place to look for unsubscription logic and action, I believe, is in
the main() function for mlmmj-unsub, which you will find at the end of
mlmmj-unsub.c.

Cheers,

Ben.