korgalore delivering duplicates

korgalore delivering duplicates

[Ard Biesheuvel]

L.S.,

First of all, thanks for the work on korgalore - it is an awesome tool and it is already making my [work] life better.

As discussed on #kernel IRC, I am seeing some issues in combination with fastmail. I am using my @kernel.org as the email address in fastmail, using kernel.org for outgoing SMTP and an alias to ard@workofard.com for incoming email.

I am seeing duplicate messages in 2 cases:

- messages received directly that are also pulled by korgalore, potentially multiple times if they were cross-posted to multiple mailing lists;
- outgoing messages that are cc'ed to multiple mailing lists, which are then looped back and delivered by korgalore in the same manner.

Happy to test/play around/do whatever else could be helpful in finding a way to work around this. Thanks a lot.

Re: korgalore delivering duplicates

[Konstantin Ryabitsev]

On Tue, Jan 27, 2026, at 13:18, Ard Biesheuvel wrote:
> - messages received directly that are also pulled by korgalore,
>   potentially multiple times if they were cross-posted to multiple
>   mailing lists;

Properly dealing with this would require us looking into each target
folder to check which messages are already there. I'm not sure how much
pain it would be to deal with this situation -- we definitely don't
want to list all messages in a folder to compare message-id's, but
maybe jmap provides an API mechanism for this. I'll follow up once I
have a better picture.

For those deliveries that we perform via korgalore, at least, we can
track which message-id's we've seen per target and dedupe at that stage
-- we just need to make sure that we properly apply labels per each
deduped message we process.

-K

Re: korgalore delivering duplicates

[Konstantin Ryabitsev]

On Tue, Jan 27, 2026 at 07:18:18PM +0100, Ard Biesheuvel wrote:
> - messages received directly that are also pulled by korgalore, potentially
>   multiple times if they were cross-posted to multiple mailing lists;
> - outgoing messages that are cc'ed to multiple mailing lists, which are then
>   looped back and delivered by korgalore in the same manner.

Okay, for JMAP and IMAP targets, korgalore now checks if a message with the
same Message-ID already exists in the target folder before attempting to
import it. This uses JMAP's Email/query with a header filter for JMAP targets,
and IMAP's SEARCH HEADER for IMAP targets. Gmail already deduplicates
automatically via its API, so no changes were needed there.

Not doing anything for Maildir targets for now, because I don't want to
iterate through potentially thousands of messages.

I've also added an X-Korgalore-Trace header to all delivered messages, e.g.:

    X-Korgalore-Trace: from feed=lei:myquery for delivery=fastmail
      by korgalore/0.5-dev; Mon, 27 Jan 2026 17:10:09 -0500

This should help you identify which messages came through korgalore and
from which feed, at least.

The changes are in the 0.5 master branch and need more testing, if you are
able to do that via the git repo directly.

Best wishes,
-K