korgalore: a tool to put lore/lei messages directly into gmail

korgalore: a tool to put lore/lei messages directly into gmail

[Konstantin Ryabitsev]

Hello:

TL;DR: korgalore lets you pull mailing lists directly from lore and put them
into your gmail inbox via gmail API. It can do the same with lei searches.

Docs are here:
https://korgalore.docs.kernel.org/

More info
---------

Gmail has recently announced that they will be shutting down the ability to
pull messages via POP3:

https://support.google.com/mail/answer/16604719?hl=en

This is how some people (including myself) have been following high-volume
mailing lists, because directly subscribing to something like netdev, mm, or
LKML lists almost immediately runs over your gmail incoming messages quota.

As a replacement, I wrote a small tool that can follow mailing lists or lei
query results and put them directly into your gmail inbox using gmail's own
API.

The good:

- It works well -- I've been using it for 2 weeks now with much more reliable
  deliveries than SMTP or even POP3, because now patches don't get randomly
  tagged as spam.
- You can follow any public-inbox remote without needing to configure anything
  locally. The tool pulls underlying git repositories and is very light on the
  remote servers, resulting in quick operation.
- I'll probably add support for JMAP (Fastmail) and generic IMAP, so the tool
  can be used with providers other than gmail.

The bad:

Getting Gmail API credentials and then authenticating is HORRENDOUSLY
COMPLICATED. You have to enable developer mode, create your own project, and
then enable the Gmail API just to get the credentials.json file. Then you have
to have a web browser on your workstation to authenticate. It's mind-blowing
how hostile Google is to the idea of people using their own gmail account for
something as straightforward as accessing their own inbox.

So, unfortunately, this tool is not a drop-in replacement for most people who
want to subscribe to LKML or other high-volume lists. However, if clicking
through 10 screens doesn't scare you, this tool can be a good alternative to
SMTP-based subscriptions and will likely get you the list mail faster and more
reliably.

-K

Re: korgalore: a tool to put lore/lei messages directly into gmail

[Paul Moore]

On Wed, Nov 5, 2025 at 1:38 PM Konstantin Ryabitsev
<konstantin@linuxfoundation.org> wrote:
>
> Hello:
>
> TL;DR: korgalore lets you pull mailing lists directly from lore and put them
> into your gmail inbox via gmail API. It can do the same with lei searches.
>
> Docs are here:
> https://korgalore.docs.kernel.org/

Thanks for this, if gmail throttling continues to worsen it's good to
know there is an alternative.

I just did a quick scan of the docs and while I didn't see it in the
Quickstart or the Usage section, I wanted to check and see if there is
support (planned or otherwise) for pulling/importing specific messages
instead of an entire list?

Often I don't want to subscribe, pull, or otherwise consume an entire
list, but I do want to comment on a specific thread or patch.  With
lore it is easy to read the discussion, but difficult to reply with
gmail as there is no reply-to header support.  I have my own hacky way
of handling this that I don't particularly love, but if I had simple
tool that I could run, e.g. 'kgl pull -M <msgid>', which would import
a specific message-id into my gmail inbox, I would be very happy.

... and yes, I'd be willing to click through the eighty different
auth/creds pages to set it up on my system.  Sadly there are a number
things that require similar auth hacks, and for better or worse I've
gotten accustomed to the extra steps.

-- 
paul-moore.com

Re: korgalore: a tool to put lore/lei messages directly into gmail

[Jason Gunthorpe]

On Wed, Nov 05, 2025 at 01:36:12PM -0500, Konstantin Ryabitsev wrote:
> So, unfortunately, this tool is not a drop-in replacement for most people who
> want to subscribe to LKML or other high-volume lists. However, if clicking
> through 10 screens doesn't scare you, this tool can be a good alternative to
> SMTP-based subscriptions and will likely get you the list mail faster and more
> reliably.

This doesn't have to be so bad. My mail tool:

https://github.com/jgunthorpe/cloud_mdir_sync

Comes with a built in oauth cookie, and a built in webserver to manage
oauth.

The tool auto-launches the web browser to localhost. The first time
you run it you will have a one screen to accept the application to
your account. From then on it generally doesn't require any
interaction as the token that comes back is pretty long lived, and if
not the refresh through a browser with an already logged in tab is
totally invisible - again by invoking the web browser to a localhost
URL that automatically generates the required redirects to get the
authentication - without interaction.

You can use the above purely as a capable oauth broker, and a tool
like you are proposing here can just use a callout like:

  cms-oauth --cms_sock=cms.sock --proto=IMAP --user user@domain --output=token

Which will rely on the running daemon to acquire and retain all the
require tokens. The various documentation pages linked from above are
quite helpful :)

This can also be wired into the various smtp programs so it gives a
comprehensive and easy oauth based solution for mail. Works on
Microsoft O365 OAUTH too.

IMHO OAUTH is good, we have very bad software support in Linux, but
that doesn't mean there is anything wrong with OAUTH. It's just
kerberos reinvented to use public key crypto, JSON and "web
technologies".

Jason

Re: korgalore: a tool to put lore/lei messages directly into gmail

[Marc Herbert]

> On Wed, Nov 5, 2025 at 1:38 PM Konstantin Ryabitsev
> <konstantin@linuxfoundation.org> wrote:

>> It's mind-blowing how hostile Google is to the idea of people using
>> their own gmail account for something as straightforward as accessing
>> their own inbox.

I suspect you dropped an important keyword: "...hostile to accessing
their own inbox _programmatically_". Just a guess. Also: paranoid
security? Gmail is the passwords and identity manager for many Android
users.


Paul Moore <paul@paul-moore.com> writes:

> Often I don't want to subscribe, pull, or otherwise consume an entire
> list, but I do want to comment on a specific thread or patch.  With
> lore it is easy to read the discussion, but difficult to reply with
> gmail as there is no reply-to header support.  I have my own hacky way
> of handling this that I don't particularly love, but if I had simple
> tool that I could run, e.g. 'kgl pull -M <msgid>', which would import
> a specific message-id into my gmail inbox, I would be very happy.

This problem has been solved by NNTP in 1986 - and maybe UUCP even
before that? I just did exactly that thanks to:
https://subspace.kernel.org/subscribing.html#subscribing-via-nntp
Thank you SO MUCH kernel.org for it!

Mailing-lists are just not fine-grained enough. The LKML is actually a
database in email disguise. That's probably the most understated reason
why forges have been successful: because it's extremely easy to
participate at the _thread level_. Fine-grained (un)subscribe buttons
everywhere. The debates about web interfaces obscure this critical
point. We live in the era of information overload and attention
merchants, so this matters a lot.

Unfortunately, NNTP clients have become rare. I can't understand why
because it's the right tool for the job. I mean your problem statement
above is literally the design rationale of NNTP (and why I could not
resist replying). Also, NNTP still works. I know because I just used it to
fetch this message. Brief reality check, you never know.
Note I am not using NNTP to post this reply, just regular SMTP.

Thunderbird still does NNTP but it's a bit heavy and slow. I just
started using "gnus" again, it's fast and crazy powerful cause it runs
in EmacsOS but... it runs in EmacsOS which means everything freezes when
there is a network or proxy glitch, a very large download or some
bug. So I also just started running two Emacs instances as a workaround,
yuck :-( Also, don't even think about it if you're not already familiar
with Emacs.

If I really depended on mailing lists a lot then I would probably look
into "leafnode" or similar. "Airplane mode": check. Could something like
leafnode "bridge" NNTP and email so everyone can keep using their
favorite email client? Not an expert but I don't see why not.  Bonus
points for: vendor-neutral RFCs and no hard dependency on any cloud
provider.


PS:
https://lars.ingebrigtsen.no/2020/01/06/whatever-happened-to-news-gmane-org/

Re: korgalore: a tool to put lore/lei messages directly into gmail

[Konstantin Ryabitsev]

On Wed, Nov 05, 2025 at 02:12:44PM -0500, Paul Moore wrote:
> Thanks for this, if gmail throttling continues to worsen it's good to
> know there is an alternative.
> 
> I just did a quick scan of the docs and while I didn't see it in the
> Quickstart or the Usage section, I wanted to check and see if there is
> support (planned or otherwise) for pulling/importing specific messages
> instead of an entire list?

Yeah, okay, here you go:
https://korgalore.docs.kernel.org/en/latest/quickstart.html#one-off-message-import-yank

Note, that this won't (yet) follow any updates in the thread. I may add this
in the future, though, something like "yank and follow until there are no
updates for a week."

-K

Re: korgalore: a tool to put lore/lei messages directly into gmail

[Paul Moore]

On Wed, Nov 5, 2025 at 3:54 PM Konstantin Ryabitsev
<konstantin@linuxfoundation.org> wrote:
> On Wed, Nov 05, 2025 at 02:12:44PM -0500, Paul Moore wrote:
> > Thanks for this, if gmail throttling continues to worsen it's good to
> > know there is an alternative.
> >
> > I just did a quick scan of the docs and while I didn't see it in the
> > Quickstart or the Usage section, I wanted to check and see if there is
> > support (planned or otherwise) for pulling/importing specific messages
> > instead of an entire list?
>
> Yeah, okay, here you go:
> https://korgalore.docs.kernel.org/en/latest/quickstart.html#one-off-message-import-yank

Excellent, thank you.  I'll give it a shot this week and let you know
how it goes.

> Note, that this won't (yet) follow any updates in the thread. I may add this
> in the future, though, something like "yank and follow until there are no
> updates for a week."

For my selfish use case this isn't likely a big deal, if I need to
pull/import a follow-up that I wasn't CC'd on I can always "yank" that
message.

-- 
paul-moore.com

Re: korgalore: a tool to put lore/lei messages directly into gmail

[Geert Uytterhoeven]

Hi Paul,

On Wed, 5 Nov 2025 at 20:14, Paul Moore <paul@paul-moore.com> wrote:
> Often I don't want to subscribe, pull, or otherwise consume an entire
> list, but I do want to comment on a specific thread or patch.  With
> lore it is easy to read the discussion, but difficult to reply with
> gmail as there is no reply-to header support.  I have my own hacky way
> of handling this that I don't particularly love, but if I had simple
> tool that I could run, e.g. 'kgl pull -M <msgid>', which would import
> a specific message-id into my gmail inbox, I would be very happy.

You can also reply using git send-email.
Lore.kernel.org even tells you how to do that, but only when using
a URL that does not include "/T"...

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: korgalore: a tool to put lore/lei messages directly into gmail

[Rob Herring]

Hi Konstantin,

On Wed, Nov 5, 2025 at 12:36 PM Konstantin Ryabitsev
<konstantin@linuxfoundation.org> wrote:
>
> Hello:
>
> TL;DR: korgalore lets you pull mailing lists directly from lore and put them
> into your gmail inbox via gmail API. It can do the same with lei searches.
>
> Docs are here:
> https://korgalore.docs.kernel.org/

I'm trying to get a lei query to work with this. The gmail setup is
all good as yank works fine.

The first issue is that the 'master' branch is hardcoded, but git has
switched the default branch to 'main' in newer versions (I'm on 2.51).
I just did a s/master/main/ in korgalore to work-around that.

After that, I still don't get any messages.

I create this query:

lei q -t -o v2:/home/rob/lei/my-patches
--only=https://lore.kernel.org/all --dedupe=mid '(dfn:drivers OR
dfn:sound OR dfn:tools OR dfn:kernel OR dfn:arch OR dfn:Documentation
OR dfn:include OR dfn:scripts) AND f:robh@kernel.org AND
rt:3.month.ago..'

Then run 'kgl pull':

debug: Loading config from /home/rob/.config/korgalore/korgalore.toml
debug: Config loaded with 1 targets and 1 sources
debug: Data directory: /home/rob/.local/share/korgalore
debug: Processing list: lei-my-patches
debug: Running git command: git -C /home/rob/lei/my-patches/git/0.git show-ref
debug: Epoch 0 refdata: 9cf28456372cc07cf84f337ae4ee066f94b2072d refs/heads/main
debug: Running git command: git -C /home/rob/lei/my-patches/git/0.git
rev-list -n 1 main
debug: Running git command: git -C /home/rob/lei/my-patches/git/0.git
show -s --format=%ci 9cf28456372cc07cf84f337ae4ee066f94b2072d
debug: Running git command: git -C /home/rob/lei/my-patches/git/0.git
show 9cf28456372cc07cf84f337ae4ee066f94b2072d:m
Initialized: lei-my-patches.
Pull complete with no updates.

Then I do a 'lei edit-search' to get more messages because I won't get
new messages otherwise, and run pull again:

debug: Loading config from /home/rob/.config/korgalore/korgalore.toml
debug: Config loaded with 1 targets and 1 sources
debug: Data directory: /home/rob/.local/share/korgalore
debug: Processing list: lei-my-patches
debug: Running git command: git -C /home/rob/lei/my-patches/git/0.git show-ref
debug: Epoch 0 refdata: 9cf28456372cc07cf84f337ae4ee066f94b2072d refs/heads/main
debug: Running lei-up on list: lei-my-patches
debug: Running git command: git -C /home/rob/lei/my-patches/git/0.git show-ref
debug: Epoch 0 refdata: 42c36f6d0cde3902e679fec019007639b2bdf6aa refs/heads/main
debug: Running git command: git -C /home/rob/lei/my-patches/git/0.git
cat-file -e 9cf28456372cc07cf84f337ae4ee066f94b2072d^
debug: Running git command: git -C /home/rob/lei/my-patches/git/0.git
rev-list --reverse --ancestry-path
9cf28456372cc07cf84f337ae4ee066f94b2072d..main
debug: Found 131 new commits for list lei-my-patches
critical: Failed to process LEI list "lei-my-patches": 'NoneType'
object has no attribute 'get_message_at_commit'
Pull complete with no updates.

This is fixed with this change:

diff --git a/src/korgalore/cli.py b/src/korgalore/cli.py
index a578e4d93a52..0b62c67bc217 100644
--- a/src/korgalore/cli.py
+++ b/src/korgalore/cli.py
@@ -122,7 +122,7 @@ def process_commits(listname: str, commits:
List[str], gitdir: Path,
         logger.info('Limiting to %d messages as requested', max_count)
         commits = commits[-max_count:]

-    ls = ctx.obj['lore']
+    ls = ctx.obj['lei']
     cfg = ctx.obj.get('config', {})

     details = cfg['sources'][listname]


But I still don't have anything from the original query.


Finally, is there a way to get a maildir from the same query directly
from lei? I suppose I could just use offlineimap from gmail or just
setup a second identical query, but neither of those seem ideal.

Rob

Re: korgalore: a tool to put lore/lei messages directly into gmail

[Konstantin Ryabitsev]

On Wed, Dec 10, 2025 at 09:17:45PM -0600, Rob Herring wrote:
> The first issue is that the 'master' branch is hardcoded, but git has
> switched the default branch to 'main' in newer versions (I'm on 2.51).
> I just did a s/master/main/ in korgalore to work-around that.

Glad to see someone else try korgalore. :)

I didn't even know lei follows the defaultBranch -- the latest master should
have the fix for this.

> After that, I still don't get any messages.

That's a bit disorienting, I agree -- the first run simply initializes things
and doesn't actually import any mail. The main reason for this is that a new
lei query can return a LOT of results and we don't necessarily want to stick
them all into the remote.

I'll think about how we can make this less confusing.

> Then I do a 'lei edit-search' to get more messages because I won't get
> new messages otherwise, and run pull again:
> 
> debug: Loading config from /home/rob/.config/korgalore/korgalore.toml
> debug: Config loaded with 1 targets and 1 sources
> debug: Data directory: /home/rob/.local/share/korgalore
> debug: Processing list: lei-my-patches
> debug: Running git command: git -C /home/rob/lei/my-patches/git/0.git show-ref
> debug: Epoch 0 refdata: 9cf28456372cc07cf84f337ae4ee066f94b2072d refs/heads/main
> debug: Running lei-up on list: lei-my-patches
> debug: Running git command: git -C /home/rob/lei/my-patches/git/0.git show-ref
> debug: Epoch 0 refdata: 42c36f6d0cde3902e679fec019007639b2bdf6aa refs/heads/main
> debug: Running git command: git -C /home/rob/lei/my-patches/git/0.git
> cat-file -e 9cf28456372cc07cf84f337ae4ee066f94b2072d^
> debug: Running git command: git -C /home/rob/lei/my-patches/git/0.git
> rev-list --reverse --ancestry-path
> 9cf28456372cc07cf84f337ae4ee066f94b2072d..main
> debug: Found 131 new commits for list lei-my-patches
> critical: Failed to process LEI list "lei-my-patches": 'NoneType'
> object has no attribute 'get_message_at_commit'

Thanks, this is because I didn't test the situation where there can be a lei
source but no lore sources. It should be fixed now.

> But I still don't have anything from the original query.

This is a bit more surprising. I'll run some tests and see if I can replicate
this.

> Finally, is there a way to get a maildir from the same query directly
> from lei? I suppose I could just use offlineimap from gmail or just
> setup a second identical query, but neither of those seem ideal.

Right, the goal is to eventually add multiple target types, including IMAP,
JMAP, mbox/maildir, etc. I started out with just gmail because it's my most
pressing need.

-K