[RFC Draft net-next] docs: netdev: add section on using lei to manage netdev mail volume

[RFC Draft net-next] docs: netdev: add section on using lei to manage netdev mail volume

[David Wei]

As a beginner to netdev I found the volume of mail to be overwhelming. I only
want to focus on core netdev changes and ignore most driver changes. I found a
way to do this using lei, filtering the mailing list using lore's query
language and writing the results into an IMAP server.

This patch is an RFC draft of updating the maintainer-netdev documentation with
this information in the hope of helping out others in the future.

Signed-off-by: David Wei <dw@davidwei.uk>
---
 Documentation/process/maintainer-netdev.rst | 39 +++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/Documentation/process/maintainer-netdev.rst b/Documentation/process/maintainer-netdev.rst
index 7feacc20835e..93851783de6f 100644
--- a/Documentation/process/maintainer-netdev.rst
+++ b/Documentation/process/maintainer-netdev.rst
@@ -33,6 +33,45 @@ Aside from subsystems like those mentioned above, all network-related
 Linux development (i.e. RFC, review, comments, etc.) takes place on
 netdev.
 
+Managing emails
+~~~~~~~~~~~~~~~
+
+netdev is a busy mailing list with on average over 200 emails received per day,
+which can be overwhelming to beginners. Rather than subscribing to the entire
+list, considering using ``lei`` to only subscribe to topics that you are
+interested in. Konstantin Ryabitsev wrote excellent tutorials on using ``lei``:
+
+ - https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
+ - https://people.kernel.org/monsieuricon/lore-lei-part-2-now-with-imap
+
+As a netdev beginner, you may want to filter out driver changes and only focus
+on core netdev changes. Try using the following query with ``lei q``::
+
+  lei q -o ~/Mail/netdev \
+    -I https://lore.kernel.org/all \
+    -t '(b:b/net/* AND tc:netdev@vger.kernel.org AND rt:2.week.ago..'
+
+This query will only match threads containing messages with patches that modify
+files in ``net/*``. For more information on the query language, see:
+
+  https://lore.kernel.org/linux-btrfs/_/text/help/
+
+By default ``lei`` will output to a Maildir, but it also supports Mbox and IMAP
+by adding a prefix to the output directory ``-o``. For a list of supported
+formats and prefix strings, see:
+
+  https://www.mankier.com/1/lei-q
+
+If you would like to use IMAP, Konstantin’s blog is slightly outdated and you
+no longer need to use here strings i.e. ``<<<`` or ``<<EOF``. You can simply
+point lei at an IMAP server e.g. ``imaps://imap.gmail.com``::
+
+  lei q -o imaps://imap.gmail.com/netdev \
+    -I https://lore.kernel.org/all \
+    -t '(b:b/net/* AND tc:netdev@vger.kernel.org AND rt:2.week.ago..'
+
+You need to set up ``git-credentials`` properly first.
+
 Development cycle
 -----------------
 
-- 
2.39.3

Re: [RFC Draft net-next] docs: netdev: add section on using lei to manage netdev mail volume

[Matthieu Baerts]

Hi David,

On 05/11/2023 19:50, David Wei wrote:
> As a beginner to netdev I found the volume of mail to be overwhelming. I only
> want to focus on core netdev changes and ignore most driver changes. I found a
> way to do this using lei, filtering the mailing list using lore's query
> language and writing the results into an IMAP server.

I agree that the volume of mail is too high with a variety of subjects.
That's why it is very important to CC the right people (as mentioned by
Patchwork [1] ;) )

[1]
https://patchwork.kernel.org/project/netdevbpf/patch/20231105185014.2523447-1-dw@davidwei.uk/

> This patch is an RFC draft of updating the maintainer-netdev documentation with
> this information in the hope of helping out others in the future.

Note that I'm also using lei to filter emails, e.g. to be notified when
someone sends a patch modifying this maintainer-netdev.rst file! [2]

But I don't think this issue of "busy mailing list" is specific to
netdev. It seems that "lei" is already mentioned in another part of the
doc [3]. Maybe this part can be improved? Or the netdev doc could add a
reference to the existing part?

(Maybe such info should be present elsewhere, e.g. on vger [4] or lore)

[2]
https://lore.kernel.org/netdev/?q=%28dfn%3ADocumentation%2Fnetworking%2Fnetdev-FAQ.rst+OR+dfn%3ADocumentation%2Fprocess%2Fmaintainer-netdev.rst%29+AND+rt%3A1.month.ago..
[3]
https://docs.kernel.org/maintainer/feature-and-driver-maintainers.html#mailing-list-participation
[4] http://vger.kernel.org/vger-lists.html

(Note: regarding the commit message here, each line should be limited to
max 72 chars ideally)

> Signed-off-by: David Wei <dw@davidwei.uk>
> ---
>  Documentation/process/maintainer-netdev.rst | 39 +++++++++++++++++++++
>  1 file changed, 39 insertions(+)
> 
> diff --git a/Documentation/process/maintainer-netdev.rst b/Documentation/process/maintainer-netdev.rst
> index 7feacc20835e..93851783de6f 100644
> --- a/Documentation/process/maintainer-netdev.rst
> +++ b/Documentation/process/maintainer-netdev.rst
> @@ -33,6 +33,45 @@ Aside from subsystems like those mentioned above, all network-related
>  Linux development (i.e. RFC, review, comments, etc.) takes place on
>  netdev.
>  
> +Managing emails
> +~~~~~~~~~~~~~~~
> +
> +netdev is a busy mailing list with on average over 200 emails received per day,
> +which can be overwhelming to beginners. Rather than subscribing to the entire
> +list, considering using ``lei`` to only subscribe to topics that you are
> +interested in. Konstantin Ryabitsev wrote excellent tutorials on using ``lei``:
> +
> + - https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
> + - https://people.kernel.org/monsieuricon/lore-lei-part-2-now-with-imap
> +
> +As a netdev beginner, you may want to filter out driver changes and only focus
> +on core netdev changes. Try using the following query with ``lei q``::
> +
> +  lei q -o ~/Mail/netdev \
> +    -I https://lore.kernel.org/all \
> +    -t '(b:b/net/* AND tc:netdev@vger.kernel.org AND rt:2.week.ago..'

Small optimisations:

- you can remove tc:netdev@vger.kernel.org and modify the '-I' to
restrict to netdev instead of querying 'all': -I
https://lore.kernel.org/netdev/

- In theory, 'dfn:' should help you to match a filename being modified.
But in your case, 'net' is too generic, and I don't think we can specify
"starting with 'net'". You can still omit some results after [5] but the
syntax doesn't look better :)

  dfn:net AND NOT dfn:drivers/net AND NOT dfn:selftests/net AND NOT
dfn:tools/net AND rt:2.week.ago..

[5]
https://lore.kernel.org/netdev/?q=dfn%3Anet+AND+NOT+dfn%3Adrivers%2Fnet+AND+NOT+dfn%3Aselftests%2Fnet+AND+NOT+dfn%3Atools%2Fnet+AND+rt%3A2.week.ago..

> +This query will only match threads containing messages with patches that modify
> +files in ``net/*``. For more information on the query language, see:
> +
> +  https://lore.kernel.org/linux-btrfs/_/text/help/

(if this is specific to 'netdev', best to use '/netdev/', not
'/linux-btrfs/')

> +By default ``lei`` will output to a Maildir, but it also supports Mbox and IMAP
> +by adding a prefix to the output directory ``-o``. For a list of supported
> +formats and prefix strings, see:
> +
> +  https://www.mankier.com/1/lei-q

Maybe safer to point to the official doc?

https://public-inbox.org/lei-q.html

(or 'man lei-q')

> +If you would like to use IMAP, Konstantin’s blog is slightly outdated and you
> +no longer need to use here strings i.e. ``<<<`` or ``<<EOF``.

I think we can still use them. In the part 1, they are not used. Maybe
best to contact Konstantin to update his blog post instead of mentioning
in the doc that the blog post is outdated?

> You can simply
> +point lei at an IMAP server e.g. ``imaps://imap.gmail.com``::

In Konstantin's blog post, he mentioned different servers with different
specificities. Maybe easier to just point to that instead of taking one
example without more explanations?

Cheers,
Matt

Re: [RFC Draft net-next] docs: netdev: add section on using lei to manage netdev mail volume

[David Wei]

On 2023-11-06 03:24, Matthieu Baerts wrote:
> Hi David,
> 
> On 05/11/2023 19:50, David Wei wrote:
>> As a beginner to netdev I found the volume of mail to be overwhelming. I only
>> want to focus on core netdev changes and ignore most driver changes. I found a
>> way to do this using lei, filtering the mailing list using lore's query
>> language and writing the results into an IMAP server.
> 
> I agree that the volume of mail is too high with a variety of subjects.
> That's why it is very important to CC the right people (as mentioned by
> Patchwork [1] ;) )
> 
> [1]
> https://patchwork.kernel.org/project/netdevbpf/patch/20231105185014.2523447-1-dw@davidwei.uk/

Sorry and noted, I've now CC'd maintainers mentioned by Patchwork.

> 
>> This patch is an RFC draft of updating the maintainer-netdev documentation with
>> this information in the hope of helping out others in the future.
> 
> Note that I'm also using lei to filter emails, e.g. to be notified when
> someone sends a patch modifying this maintainer-netdev.rst file! [2]
> 
> But I don't think this issue of "busy mailing list" is specific to
> netdev. It seems that "lei" is already mentioned in another part of the
> doc [3]. Maybe this part can be improved? Or the netdev doc could add a
> reference to the existing part?

I think "busy mailing list" is especially bad for netdev. There are many
tutorials for setting up lei, but my ideal goal is a copy + paste
command specifically for netdev that outputs into an IMAP server for
beginners to use. As opposed to writing something more generic.

> 
> (Maybe such info should be present elsewhere, e.g. on vger [4] or lore)
> 
> [2]
> https://lore.kernel.org/netdev/?q=%28dfn%3ADocumentation%2Fnetworking%2Fnetdev-FAQ.rst+OR+dfn%3ADocumentation%2Fprocess%2Fmaintainer-netdev.rst%29+AND+rt%3A1.month.ago..
> [3]
> https://docs.kernel.org/maintainer/feature-and-driver-maintainers.html#mailing-list-participation

This document is aimed at kernel maintainers. My concern is that
beginners would not find or read this document.

> [4] http://vger.kernel.org/vger-lists.html

It would be nice to add a link in the netdev list "Info" section. Do you
know how to update it?

How about keeping a netdev specific sample lei query in
maintainer-netdev and refer to it from [4]?

> 
> (Note: regarding the commit message here, each line should be limited to
> max 72 chars ideally)

Apologies, I may not have line wrap set up properly in my editor.

> 
>> Signed-off-by: David Wei <dw@davidwei.uk>
>> ---
>>  Documentation/process/maintainer-netdev.rst | 39 +++++++++++++++++++++
>>  1 file changed, 39 insertions(+)
>>
>> diff --git a/Documentation/process/maintainer-netdev.rst b/Documentation/process/maintainer-netdev.rst
>> index 7feacc20835e..93851783de6f 100644
>> --- a/Documentation/process/maintainer-netdev.rst
>> +++ b/Documentation/process/maintainer-netdev.rst
>> @@ -33,6 +33,45 @@ Aside from subsystems like those mentioned above, all network-related
>>  Linux development (i.e. RFC, review, comments, etc.) takes place on
>>  netdev.
>>  
>> +Managing emails
>> +~~~~~~~~~~~~~~~
>> +
>> +netdev is a busy mailing list with on average over 200 emails received per day,
>> +which can be overwhelming to beginners. Rather than subscribing to the entire
>> +list, considering using ``lei`` to only subscribe to topics that you are
>> +interested in. Konstantin Ryabitsev wrote excellent tutorials on using ``lei``:
>> +
>> + - https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
>> + - https://people.kernel.org/monsieuricon/lore-lei-part-2-now-with-imap
>> +
>> +As a netdev beginner, you may want to filter out driver changes and only focus
>> +on core netdev changes. Try using the following query with ``lei q``::
>> +
>> +  lei q -o ~/Mail/netdev \
>> +    -I https://lore.kernel.org/all \
>> +    -t '(b:b/net/* AND tc:netdev@vger.kernel.org AND rt:2.week.ago..'
> 
> Small optimisations:
> 
> - you can remove tc:netdev@vger.kernel.org and modify the '-I' to
> restrict to netdev instead of querying 'all': -I
> https://lore.kernel.org/netdev/

Thank you, this is great.

> 
> - In theory, 'dfn:' should help you to match a filename being modified.
> But in your case, 'net' is too generic, and I don't think we can specify
> "starting with 'net'". You can still omit some results after [5] but the
> syntax doesn't look better :)
> 
>   dfn:net AND NOT dfn:drivers/net AND NOT dfn:selftests/net AND NOT
> dfn:tools/net AND rt:2.week.ago..

I initially went with this as well, but found it tedious to add many AND
NOT statements. My metric was number of emails filtered and matching
using b:b/net/* produced the least number of emails :)

It would be ideal if we could express dfn:^net/*. I contacted the public
inbox folks and they said it is not supported :(

> 
> [5]
> https://lore.kernel.org/netdev/?q=dfn%3Anet+AND+NOT+dfn%3Adrivers%2Fnet+AND+NOT+dfn%3Aselftests%2Fnet+AND+NOT+dfn%3Atools%2Fnet+AND+rt%3A2.week.ago..
> 
>> +This query will only match threads containing messages with patches that modify
>> +files in ``net/*``. For more information on the query language, see:
>> +
>> +  https://lore.kernel.org/linux-btrfs/_/text/help/
> 
> (if this is specific to 'netdev', best to use '/netdev/', not
> '/linux-btrfs/')

Thank you, will fix this.

> 
>> +By default ``lei`` will output to a Maildir, but it also supports Mbox and IMAP
>> +by adding a prefix to the output directory ``-o``. For a list of supported
>> +formats and prefix strings, see:
>> +
>> +  https://www.mankier.com/1/lei-q
> 
> Maybe safer to point to the official doc?
> 
> https://public-inbox.org/lei-q.html
> 
> (or 'man lei-q')

Thanks, official manpages are best.

> 
>> +If you would like to use IMAP, Konstantin’s blog is slightly outdated and you
>> +no longer need to use here strings i.e. ``<<<`` or ``<<EOF``.
> 
> I think we can still use them. In the part 1, they are not used. Maybe
> best to contact Konstantin to update his blog post instead of mentioning
> in the doc that the blog post is outdated?

You're right, I've emailed Konstantin.

> 
>> You can simply
>> +point lei at an IMAP server e.g. ``imaps://imap.gmail.com``::
> 
> In Konstantin's blog post, he mentioned different servers with different
> specificities. Maybe easier to just point to that instead of taking one
> example without more explanations?

Will do!

> 
> Cheers,
> Matt

Re: [RFC Draft net-next] docs: netdev: add section on using lei to manage netdev mail volume

[Jakub Kicinski]

On Sun,  5 Nov 2023 10:50:14 -0800 David Wei wrote:
>  
> +Managing emails
> +~~~~~~~~~~~~~~~

How about adding this section before "Patch review" ?

> +netdev is a busy mailing list with on average over 200 emails received per day,
> +which can be overwhelming to beginners. Rather than subscribing to the entire
> +list, considering using ``lei`` to only subscribe to topics that you are
> +interested in. Konstantin Ryabitsev wrote excellent tutorials on using ``lei``:
> +
> + - https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
> + - https://people.kernel.org/monsieuricon/lore-lei-part-2-now-with-imap
> +
> +As a netdev beginner, you may want to filter out driver changes and only focus
> +on core netdev changes. Try using the following query with ``lei q``::
> +
> +  lei q -o ~/Mail/netdev \
> +    -I https://lore.kernel.org/all \
> +    -t '(b:b/net/* AND tc:netdev@vger.kernel.org AND rt:2.week.ago..'

Let's add a sentence pointing out the b:b/net hack and why it's needed.

Re: [RFC Draft net-next] docs: netdev: add section on using lei to manage netdev mail volume

[Matthieu Baerts]

Hi David,

On 06/11/2023 17:57, David Wei wrote:
> On 2023-11-06 03:24, Matthieu Baerts wrote:
>> On 05/11/2023 19:50, David Wei wrote:
>>> As a beginner to netdev I found the volume of mail to be overwhelming. I only
>>> want to focus on core netdev changes and ignore most driver changes. I found a
>>> way to do this using lei, filtering the mailing list using lore's query
>>> language and writing the results into an IMAP server.
>>
>> I agree that the volume of mail is too high with a variety of subjects.
>> That's why it is very important to CC the right people (as mentioned by
>> Patchwork [1] ;) )
>>
>> [1]
>> https://patchwork.kernel.org/project/netdevbpf/patch/20231105185014.2523447-1-dw@davidwei.uk/
> 
> Sorry and noted, I've now CC'd maintainers mentioned by Patchwork.

Thanks!

>>> This patch is an RFC draft of updating the maintainer-netdev documentation with
>>> this information in the hope of helping out others in the future.
>>
>> Note that I'm also using lei to filter emails, e.g. to be notified when
>> someone sends a patch modifying this maintainer-netdev.rst file! [2]
>>
>> But I don't think this issue of "busy mailing list" is specific to
>> netdev. It seems that "lei" is already mentioned in another part of the
>> doc [3]. Maybe this part can be improved? Or the netdev doc could add a
>> reference to the existing part?
> 
> I think "busy mailing list" is especially bad for netdev. There are many
> tutorials for setting up lei, but my ideal goal is a copy + paste
> command specifically for netdev that outputs into an IMAP server for
> beginners to use. As opposed to writing something more generic.

I see. I don't know if many people are in this case, but having this
example will certainly help people adapting it to their case!

>> (Maybe such info should be present elsewhere, e.g. on vger [4] or lore)
>>
>> [2]
>> https://lore.kernel.org/netdev/?q=%28dfn%3ADocumentation%2Fnetworking%2Fnetdev-FAQ.rst+OR+dfn%3ADocumentation%2Fprocess%2Fmaintainer-netdev.rst%29+AND+rt%3A1.month.ago..
>> [3]
>> https://docs.kernel.org/maintainer/feature-and-driver-maintainers.html#mailing-list-participation
> 
> This document is aimed at kernel maintainers. My concern is that
> beginners would not find or read this document.

Indeed.

>> [4] http://vger.kernel.org/vger-lists.html
> 
> It would be nice to add a link in the netdev list "Info" section. Do you
> know how to update it?

No, sorry. Maybe Jakub or DaveM can help?

> How about keeping a netdev specific sample lei query in
> maintainer-netdev and refer to it from [4]?

Fine by me, but best to check with Netdev maintainers :)

(...)

> It would be ideal if we could express dfn:^net/*. I contacted the public
> inbox folks and they said it is not supported :(

Thank you for having asked them and Konstantin. That's a shame we cannot
use regex. Maybe later.

Cheers,
Matt