[b4] initial "b4 dig" to supplant Link: trailers

[b4] initial "b4 dig" to supplant Link: trailers

[Konstantin Ryabitsev]

Hi, all:

I just committed the initial implementation of "b4 dig" that will hopefully
make it easier for maintainers to figure out where a commit came from now that
we're actively trying to phase out Link: trailers. For now, it only does
"b4 dig -c [commitish]", e.g. taking some random commits from today:

Matching the exact patch-id:

	$ b4 dig -c ce740955b238761ec1d8cf0590d7e6802d3a813a
	Digging into commit ce740955b238761ec1d8cf0590d7e6802d3a813a
	Attempting to match by exact patch-id...
	Trying to find matching series by patch-id 469ebf2cf560b32106f206e389752deb5f741b6d
	Grabbing search results from lore.kernel.org
	Found matching series by patch-id
	---
	This patch is present in the following series:
	---
	  v3: [PATCH v3] dt-bindings: bus: renesas-bsc: allow additional properties
		  https://lore.kernel.org/r/20251009183630.5451-2-wsa+renesas@sang-engineering.com

No match looking up by exact patch-id (and trying myers, then histogram, then
patience algorithms with no success):

	$ b4 dig -c a29ad21b988652dc60aa99c6d3b1e3d52dc69c30
	Digging into commit a29ad21b988652dc60aa99c6d3b1e3d52dc69c30
	Attempting to match by exact patch-id...
	Trying to find matching series by patch-id abc964d5d53674450f7e553bbe031edc61ac3308
	Grabbing search results from lore.kernel.org
	Nothing matching that query.
	Trying to find matching series by patch-id adf0918395a1a601a8d54d7d26446aadfb6101c2
	Grabbing search results from lore.kernel.org
	Nothing matching that query.
	Attempting to match by author and subject...
	Grabbing search results from lore.kernel.org
	Found 22 matching messages
	---
	This patch is present in the following series:
	---
	  v3: [PATCH v3] tpm: Prevent local DOS via tpm/tpm0/ppi/*operations
		  https://lore.kernel.org/r/20250829175152.9704-2-daleksan@redhat.com
	  v4: [PATCH v4] tpm: Prevent local DOS via tpm/tpm0/ppi/*operations
		  https://lore.kernel.org/r/20250902142429.14041-2-daleksan@redhat.com
	  v5: [PATCH v5] tpm: Prevent local DOS via tpm/tpm0/ppi/*operations
		  https://lore.kernel.org/r/20250915210829.6661-1-daleksan@redhat.com

There are known annoyances:

1. We don't currently weed out AUTOSEL/stable backports, so older commits may
   end up in lots of false positives for now. I've not yet figured out how to
   avoid this without hardcoding it to be kernel-specific.
2. Patches that were sent as part of one series and then as part of another
   series (by the same author) don't currently do the right thing.

I'll work on getting that fixed, but for now if you're missing the presence of
Link: trailers, please test it out.

-K

Re: [b4] initial "b4 dig" to supplant Link: trailers

[Greg KH]

On Fri, Oct 10, 2025 at 04:47:27PM -0400, Konstantin Ryabitsev wrote:
> Hi, all:
> 
> I just committed the initial implementation of "b4 dig" that will hopefully
> make it easier for maintainers to figure out where a commit came from now that
> we're actively trying to phase out Link: trailers. For now, it only does
> "b4 dig -c [commitish]", e.g. taking some random commits from today:
> 
> Matching the exact patch-id:
> 
> 	$ b4 dig -c ce740955b238761ec1d8cf0590d7e6802d3a813a
> 	Digging into commit ce740955b238761ec1d8cf0590d7e6802d3a813a
> 	Attempting to match by exact patch-id...
> 	Trying to find matching series by patch-id 469ebf2cf560b32106f206e389752deb5f741b6d
> 	Grabbing search results from lore.kernel.org
> 	Found matching series by patch-id
> 	---
> 	This patch is present in the following series:
> 	---
> 	  v3: [PATCH v3] dt-bindings: bus: renesas-bsc: allow additional properties
> 		  https://lore.kernel.org/r/20251009183630.5451-2-wsa+renesas@sang-engineering.com
> 
> No match looking up by exact patch-id (and trying myers, then histogram, then
> patience algorithms with no success):
> 
> 	$ b4 dig -c a29ad21b988652dc60aa99c6d3b1e3d52dc69c30
> 	Digging into commit a29ad21b988652dc60aa99c6d3b1e3d52dc69c30
> 	Attempting to match by exact patch-id...
> 	Trying to find matching series by patch-id abc964d5d53674450f7e553bbe031edc61ac3308
> 	Grabbing search results from lore.kernel.org
> 	Nothing matching that query.
> 	Trying to find matching series by patch-id adf0918395a1a601a8d54d7d26446aadfb6101c2
> 	Grabbing search results from lore.kernel.org
> 	Nothing matching that query.
> 	Attempting to match by author and subject...
> 	Grabbing search results from lore.kernel.org
> 	Found 22 matching messages
> 	---
> 	This patch is present in the following series:
> 	---
> 	  v3: [PATCH v3] tpm: Prevent local DOS via tpm/tpm0/ppi/*operations
> 		  https://lore.kernel.org/r/20250829175152.9704-2-daleksan@redhat.com
> 	  v4: [PATCH v4] tpm: Prevent local DOS via tpm/tpm0/ppi/*operations
> 		  https://lore.kernel.org/r/20250902142429.14041-2-daleksan@redhat.com
> 	  v5: [PATCH v5] tpm: Prevent local DOS via tpm/tpm0/ppi/*operations
> 		  https://lore.kernel.org/r/20250915210829.6661-1-daleksan@redhat.com
> 
> There are known annoyances:
> 
> 1. We don't currently weed out AUTOSEL/stable backports, so older commits may
>    end up in lots of false positives for now. I've not yet figured out how to
>    avoid this without hardcoding it to be kernel-specific.

Because it's not weeding out anything, it seems that any commit that I
pick that is backported just gets confused.

As an example:
	$ b4 dig -c  cfa1a2329a691ffd991fcf7248a57d752e712881 2>&1 | wc -l
	1293

And the end result of that:
	$ b4 dig -c  cfa1a2329a691ffd991fcf7248a57d752e712881 2>&1 | tail -n 5
	---
	This patch is present in the following series:
	---
	  v1: [PATCH 6.1 000/128] 6.1.97-rc1 review
	      https://lore.kernel.org/r/20240702170226.231899085@linuxfoundation.org

Did find the stable review thread, but not all of them?  Is that
intentional?  If you just report all threads it's found in, I'd be happy
as that's what I dig through all the time.

> I'll work on getting that fixed, but for now if you're missing the presence of
> Link: trailers, please test it out.

I'm already missing them :(

And thanks for working on this, if it does end up working well, that's
great.

greg k-h

Re: [b4] initial "b4 dig" to supplant Link: trailers

[Peter Zijlstra]

On Sat, Oct 11, 2025 at 02:39:24PM +0200, Greg KH wrote:

> > I'll work on getting that fixed, but for now if you're missing the presence of
> > Link: trailers, please test it out.
> 
> I'm already missing them :(

The tip.git tree, per the tip-robot needing it to know what email to
reply to, still has them, except we now stick them in the git-notes,
and Linus isn't pulling those.

But for anybody that wants them, you can fetch the notes for tip.git and
they should all be there.

Re: [b4] initial "b4 dig" to supplant Link: trailers

[Geert Uytterhoeven]

Hi Konstantin,

On Fri, 10 Oct 2025 at 23:25, Konstantin Ryabitsev
<konstantin@linuxfoundation.org> wrote:
> I just committed the initial implementation of "b4 dig" that will hopefully
> make it easier for maintainers to figure out where a commit came from now that
> we're actively trying to phase out Link: trailers. For now, it only does
> "b4 dig -c [commitish]", e.g. taking some random commits from today:

Thanks for your work!

> Matching the exact patch-id:
>
>         $ b4 dig -c ce740955b238761ec1d8cf0590d7e6802d3a813a
>         Digging into commit ce740955b238761ec1d8cf0590d7e6802d3a813a
>         Attempting to match by exact patch-id...
>         Trying to find matching series by patch-id 469ebf2cf560b32106f206e389752deb5f741b6d
>         Grabbing search results from lore.kernel.org
>         Found matching series by patch-id
>         ---
>         This patch is present in the following series:
>         ---
>           v3: [PATCH v3] dt-bindings: bus: renesas-bsc: allow additional properties
>                   https://lore.kernel.org/r/20251009183630.5451-2-wsa+renesas@sang-engineering.com
>
> No match looking up by exact patch-id (and trying myers, then histogram, then
> patience algorithms with no success):
>
>         $ b4 dig -c a29ad21b988652dc60aa99c6d3b1e3d52dc69c30
>         Digging into commit a29ad21b988652dc60aa99c6d3b1e3d52dc69c30
>         Attempting to match by exact patch-id...
>         Trying to find matching series by patch-id abc964d5d53674450f7e553bbe031edc61ac3308
>         Grabbing search results from lore.kernel.org
>         Nothing matching that query.
>         Trying to find matching series by patch-id adf0918395a1a601a8d54d7d26446aadfb6101c2
>         Grabbing search results from lore.kernel.org
>         Nothing matching that query.
>         Attempting to match by author and subject...
>         Grabbing search results from lore.kernel.org

How expensive are these fuzzy look-ups? (Yes they must support fuzzy,
as lousy patches may come with lousy submission practices)
Do we really want to spend all those cycles on finding something
that could have been found easily if a simple Link:-tag pointing to
https://patch.msgid.link/ would have been (auto)added?

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [b4] initial "b4 dig" to supplant Link: trailers

[Linus Torvalds]

On Fri, 10 Oct 2025 at 13:48, Konstantin Ryabitsev
<konstantin@linuxfoundation.org> wrote:
>
> 1. We don't currently weed out AUTOSEL/stable backports, so older commits may
>    end up in lots of false positives for now. I've not yet figured out how to
>    avoid this without hardcoding it to be kernel-specific.

Why don't you just weed things out by the commit date? You know that
any backports will have a date after that, and that anything that was
the source of the commit will have a date before.

The author date will be fuzzier - because it might come from the
email, but it might also come from the email *containing* a "Date:"
line. But the commit date is reliable.

And by "reliable", I obviously dismiss people having their clocks set
wrong etc, but that seems to be happening a lot less often these days
than it used to.

             Linus

Re: [b4] initial "b4 dig" to supplant Link: trailers

[Sasha Levin]

[-- Attachment #1: Type: text/plain, Size: 1689 bytes --]

On Sat, Oct 11, 2025 at 03:46:39PM -0700, Linus Torvalds wrote:
>On Fri, 10 Oct 2025 at 13:48, Konstantin Ryabitsev
><konstantin@linuxfoundation.org> wrote:
>>
>> 1. We don't currently weed out AUTOSEL/stable backports, so older commits may
>>    end up in lots of false positives for now. I've not yet figured out how to
>>    avoid this without hardcoding it to be kernel-specific.
>
>Why don't you just weed things out by the commit date? You know that
>any backports will have a date after that, and that anything that was
>the source of the commit will have a date before.
>
>The author date will be fuzzier - because it might come from the
>email, but it might also come from the email *containing* a "Date:"
>line. But the commit date is reliable.
>
>And by "reliable", I obviously dismiss people having their clocks set
>wrong etc, but that seems to be happening a lot less often these days
>than it used to.

We have plenty of headers in stable-related mails that you can filter by to
avoid that noise. Something like the attached patch?

It also fixes the issue that Greg reported earlier in the thread:

$ b4.sh dig -c  cfa1a2329a691ffd991fcf7248a57d752e712881
[...]
This patch is present in the following series:
---
   v1: [PATCH bpf 1/2] bpf: Fix overrunning reservations in ringbuf
       https://lore.kernel.org/r/20240620213435.16336-1-daniel@iogearbox.net
   v2: [PATCH bpf v2 1/2] bpf: Fix overrunning reservations in ringbuf
       https://lore.kernel.org/r/20240621122610.15083-1-daniel@iogearbox.net
   v3: [PATCH bpf v3 1/2] bpf: Fix overrunning reservations in ringbuf
       https://lore.kernel.org/r/20240621140828.18238-1-daniel@iogearbox.net

-- 
Thanks,
Sasha

[-- Attachment #2: no_stable.patch --]
[-- Type: text/x-diff, Size: 3443 bytes --]

diff --git a/src/b4/dig.py b/src/b4/dig.py
index e5577a2..71e4fc7 100644
--- a/src/b4/dig.py
+++ b/src/b4/dig.py
@@ -25,6 +25,55 @@ try_diff_algos: List[str] = [
 ]
 
 
+def has_stable_header(msg: EmailMessage) -> bool:
+    """Check if the message has an X-stable header."""
+    for header_name in msg.keys():
+        if header_name.lower() == 'x-stable':
+            return True
+    return False
+
+
+def filter_stable_messages(lmbx: b4.LoreMailbox) -> None:
+    """Remove messages with X-stable header from a LoreMailbox."""
+    # Filter msgid_map
+    stable_msgids = []
+    for msgid, lmsg in lmbx.msgid_map.items():
+        if has_stable_header(lmsg.msg):
+            logger.debug('Ignoring -stable message: %s', lmsg.full_subject)
+            stable_msgids.append(msgid)
+
+    for msgid in stable_msgids:
+        del lmbx.msgid_map[msgid]
+
+    # Filter followups
+    lmbx.followups = [lmsg for lmsg in lmbx.followups
+                      if not has_stable_header(lmsg.msg)]
+
+    # Filter unknowns
+    lmbx.unknowns = [lmsg for lmsg in lmbx.unknowns
+                     if not has_stable_header(lmsg.msg)]
+
+    # Filter covers
+    stable_cover_revs = []
+    for rev, lmsg in lmbx.covers.items():
+        if has_stable_header(lmsg.msg):
+            stable_cover_revs.append(rev)
+
+    for rev in stable_cover_revs:
+        del lmbx.covers[rev]
+
+    # Filter series patches
+    for rev, lser in list(lmbx.series.items()):
+        original_count = len([p for p in lser.patches if p is not None])
+        lser.patches = [p if p is None or not has_stable_header(p.msg) else None
+                        for p in lser.patches]
+        filtered_count = len([p for p in lser.patches if p is not None])
+
+        # Remove series if all patches were filtered out
+        if filtered_count == 0:
+            del lmbx.series[rev]
+
+
 def dig_commit(cmdargs: argparse.Namespace) -> None:
     config = b4.get_main_config()
     cfg_llval = config.get('linkmask', '')
@@ -101,6 +150,12 @@ def dig_commit(cmdargs: argparse.Namespace) -> None:
         lmbx = b4.get_series_by_patch_id(patch_id)
         if lmbx:
             logger.info('Found matching series by patch-id')
+            filter_stable_messages(lmbx)
+            # Check if we still have any series after filtering
+            if not lmbx.series:
+                logger.debug('All series filtered out due to X-stable header')
+                lmbx = None
+                continue
             break
 
     if not lmbx:
@@ -111,6 +166,9 @@ def dig_commit(cmdargs: argparse.Namespace) -> None:
             logger.info('Found %s matching messages', len(msgs))
             lmbx = b4.LoreMailbox()
             for msg in msgs:
+                if has_stable_header(msg):
+                    logger.debug('Ignoring -stable message: %s', msg.get('subject', ''))
+                    continue
                 lmbx.add_message(msg)
         else:
             logger.error('Could not find anything matching commit %s', commit)
@@ -147,6 +205,9 @@ def dig_commit(cmdargs: argparse.Namespace) -> None:
         q_msgs = b4.get_pi_search_results(q, full_threads=True)
         if q_msgs:
             for q_msg in q_msgs:
+                if has_stable_header(q_msg):
+                    logger.debug('Ignoring -stable message: %s', q_msg.get('subject', ''))
+                    continue
                 lmbx.add_message(q_msg)
         break
 

Re: [b4] initial "b4 dig" to supplant Link: trailers

[Linus Torvalds]

On Sat, 11 Oct 2025 at 16:38, Sasha Levin <sashal@kernel.org> wrote:
>
> We have plenty of headers in stable-related mails that you can filter by to
> avoid that noise. Something like the attached patch?

I was reacting to Konstantin not feeling like making it
kernel-specific, and I kind of agree. It's a generally usable feature
for any project that uses lore or something lore-like.

And the commit date seems like the obvious cut-off if you actually
want a "what led up to this commit?"

But it's not like having special cases for avoiding stable back-ports
is *wrong*, I just feel like "hey, you have the commit, you're asking
b4 to dig for the source of it, then the commit date should be your
*first* filter".

Of course, in other situations you don't want to ask for the source,
but for people talking about it after-the-fact, and then you obviously
don't want to filter by date, but that's a "command line switch and
default behavior" kind of question.

            Linus

Re: [b4] initial "b4 dig" to supplant Link: trailers

[Sasha Levin]

On Sat, Oct 11, 2025 at 04:44:00PM -0700, Linus Torvalds wrote:
>On Sat, 11 Oct 2025 at 16:38, Sasha Levin <sashal@kernel.org> wrote:
>>
>> We have plenty of headers in stable-related mails that you can filter by to
>> avoid that noise. Something like the attached patch?
>
>I was reacting to Konstantin not feeling like making it
>kernel-specific, and I kind of agree. It's a generally usable feature
>for any project that uses lore or something lore-like.

Ah, I understand.

>And the commit date seems like the obvious cut-off if you actually
>want a "what led up to this commit?"
>
>But it's not like having special cases for avoiding stable back-ports
>is *wrong*, I just feel like "hey, you have the commit, you're asking
>b4 to dig for the source of it, then the commit date should be your
>*first* filter".

I ran a quick test to see how many commits do the (obviously) wrong thing w.r.t
commit date, and we only have a few of those in the past year:

$ git log --since="1 year" --pretty=format:"%H %at %ct" | awk '$3 < $2' | while read hash author_ts commit_ts rest; do echo "$hash Author: $(date -d @$author_ts) Commit: $(date -d @$commit_ts)"; done

82ab67d762e922bb5df1cbb442e8d4f12c26a7ae Author: Fri Sep  5 05:20:44 AM EDT 2025 Commit: Sat Aug 30 04:38:55 AM EDT 2025
11f74f48c14c1f4fe16541900ea5944c42e30ccf Author: Wed Jul 30 08:49:06 AM EDT 2025 Commit: Wed Jul 30 08:48:43 AM EDT 2025
ca652cf0c2612add5d3c9283bbc742dabc704a77 Author: Tue Jul 22 11:18:41 AM EDT 2025 Commit: Tue Jul 22 07:41:12 AM EDT 2025
a099b09a3342a0b28ea330e405501b5b4d0424b4 Author: Fri Jun 13 11:18:38 AM EDT 2025 Commit: Fri Jun 13 05:38:25 AM EDT 2025
f1a3fac4095c7bc0b30e2aa9921c232af8faeae0 Author: Thu May 15 10:53:59 AM EDT 2025 Commit: Thu May 15 10:19:22 AM EDT 2025
856366dc924a9561dae39f252b45dfd6cc6895ce Author: Mon Feb  3 09:10:51 AM EST 2025 Commit: Mon Feb  3 09:05:02 AM EST 2025
4343af66b8e1df1d3a2e6f1f8612506cb45b2afd Author: Mon Feb  3 09:10:50 AM EST 2025 Commit: Mon Feb  3 09:05:01 AM EST 2025
320155a61f7fc810a915644e9e2a451bdcea90b1 Author: Mon Feb  3 09:10:49 AM EST 2025 Commit: Mon Feb  3 09:05:00 AM EST 2025
f0173cbe7fa79eafbdf32eed32337209f84ddacd Author: Mon Feb  3 09:10:48 AM EST 2025 Commit: Mon Feb  3 09:04:59 AM EST 2025
cbe37a4d2b3c25d2e2a94097e09b6d87461b8833 Author: Mon Feb  3 09:10:47 AM EST 2025 Commit: Mon Feb  3 09:04:58 AM EST 2025
e995c51903384be1c7aead246dc30cb5244179ac Author: Mon Feb  3 09:10:46 AM EST 2025 Commit: Mon Feb  3 09:04:57 AM EST 2025
b9fb91692af881736f8fa1741fa0dbadf07d99ee Author: Mon Feb  3 09:10:45 AM EST 2025 Commit: Mon Feb  3 09:04:56 AM EST 2025
79ebb596201c86712fe38b0ef73d25d07b932664 Author: Mon Feb  3 09:10:44 AM EST 2025 Commit: Mon Feb  3 09:04:55 AM EST 2025
7d92a38d67e5d937b64b20aa4fd14451ee1772f3 Author: Mon Feb  3 09:10:43 AM EST 2025 Commit: Mon Feb  3 09:04:54 AM EST 2025
e92f042642aed6f6206caace892d9df2d0166841 Author: Mon Feb  3 09:10:42 AM EST 2025 Commit: Mon Feb  3 09:04:53 AM EST 2025
dc561ab16d8be9cbe8f07a49a7b2f5428fbcfeea Author: Mon Feb  3 09:10:41 AM EST 2025 Commit: Mon Feb  3 09:04:52 AM EST 2025
0b12850ddfb0032376ef1be10b5b46be00bba4d4 Author: Thu Jan  9 07:22:16 AM EST 2025 Commit: Thu Jan  9 07:14:28 AM EST 2025
ef724707788325a53ffa4cf58fceb94654e4793a Author: Thu Jan  9 07:22:15 AM EST 2025 Commit: Thu Jan  9 07:14:27 AM EST 2025
3eede0fc99c684df6f3f35866761036dabf89d05 Author: Thu Jan  9 07:22:14 AM EST 2025 Commit: Thu Jan  9 07:14:26 AM EST 2025
aea305d28551bc213aab3a41a0f59412247ae2b4 Author: Thu Jan  9 07:22:13 AM EST 2025 Commit: Thu Jan  9 07:14:25 AM EST 2025
480d9bb9cfb7b774b22cf82ff21903eb50d64cb9 Author: Thu Jan  9 07:22:12 AM EST 2025 Commit: Thu Jan  9 07:14:24 AM EST 2025
0ca529926c5d9d0a3c0b1609fb7034ab870e4770 Author: Thu Jan  9 07:22:11 AM EST 2025 Commit: Thu Jan  9 07:14:23 AM EST 2025
94aa347d34e079859a5378272c9452c728e4183a Author: Thu Jan  9 07:22:10 AM EST 2025 Commit: Thu Jan  9 07:14:22 AM EST 2025
33228036ff655ebed1bc4bde9c9b6329b569b27b Author: Thu Jan  9 07:22:09 AM EST 2025 Commit: Thu Jan  9 07:14:21 AM EST 2025
e3146775f05d18c667a2e26082da3ac105f87d9f Author: Thu Jan  9 07:22:08 AM EST 2025 Commit: Thu Jan  9 07:14:20 AM EST 2025
e9ca3db9f01a7ce91ceab35cd5fa52f0c5aca174 Author: Thu Jan  9 07:22:07 AM EST 2025 Commit: Thu Jan  9 07:14:19 AM EST 2025
cf4d74256fe103ece7b2647550e6c063048e5682 Author: Thu Jan  9 07:22:06 AM EST 2025 Commit: Thu Jan  9 07:14:18 AM EST 2025
dbda5c35b88794f6e5efe1b5b20044b0b3a340d4 Author: Thu Jan  9 07:22:05 AM EST 2025 Commit: Thu Jan  9 07:14:18 AM EST 2025
bca0fa5f6b5e96c03daac1ed62b1e5c5057a2048 Author: Thu Jan  9 07:22:04 AM EST 2025 Commit: Thu Jan  9 07:14:17 AM EST 2025

-- 
Thanks,
Sasha

Re: [b4] initial "b4 dig" to supplant Link: trailers

[Jason Gunthorpe]

On Sat, Oct 11, 2025 at 04:44:00PM -0700, Linus Torvalds wrote:
> On Sat, 11 Oct 2025 at 16:38, Sasha Levin <sashal@kernel.org> wrote:
> >
> > We have plenty of headers in stable-related mails that you can filter by to
> > avoid that noise. Something like the attached patch?
> 
> I was reacting to Konstantin not feeling like making it
> kernel-specific, and I kind of agree. It's a generally usable feature
> for any project that uses lore or something lore-like.

It is pretty common for backports (in any project) to have 'upstream
commit XXXX' or very similar language in the email and commit message,
so if you dig commit XXX and find emails that contain the string XXX
they can probably be excluded.

Jason

Re: [b4] initial "b4 dig" to supplant Link: trailers

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 3294 bytes --]

On Fri, Oct 10, 2025 at 04:47:27PM -0400, Konstantin Ryabitsev wrote:

> I just committed the initial implementation of "b4 dig" that will hopefully
> make it easier for maintainers to figure out where a commit came from now that
> we're actively trying to phase out Link: trailers. For now, it only does
> "b4 dig -c [commitish]", e.g. taking some random commits from today:

If I specify:

   b4 dig -c regulator-v6.18

b4 complains:

   Digging into commit b320fce86c04862ff5548ebdd84740e36e79e6ac
   Merge commit detected, please specify a single-parent commit.

but that's not a merge commit, it's a signed tag on a regular commit.

It does also feel odd to have to specify -c, I'd have expected this to
work more like b4 mbox and just assume that an unknown parameter was a
commitish.  Not the end of the world though.

> 
> Matching the exact patch-id:
> 
> 	$ b4 dig -c ce740955b238761ec1d8cf0590d7e6802d3a813a
> 	Digging into commit ce740955b238761ec1d8cf0590d7e6802d3a813a
> 	Attempting to match by exact patch-id...
> 	Trying to find matching series by patch-id 469ebf2cf560b32106f206e389752deb5f741b6d
> 	Grabbing search results from lore.kernel.org
> 	Found matching series by patch-id
> 	---
> 	This patch is present in the following series:
> 	---
> 	  v3: [PATCH v3] dt-bindings: bus: renesas-bsc: allow additional properties
> 		  https://lore.kernel.org/r/20251009183630.5451-2-wsa+renesas@sang-engineering.com
> 
> No match looking up by exact patch-id (and trying myers, then histogram, then
> patience algorithms with no success):
> 
> 	$ b4 dig -c a29ad21b988652dc60aa99c6d3b1e3d52dc69c30
> 	Digging into commit a29ad21b988652dc60aa99c6d3b1e3d52dc69c30
> 	Attempting to match by exact patch-id...
> 	Trying to find matching series by patch-id abc964d5d53674450f7e553bbe031edc61ac3308
> 	Grabbing search results from lore.kernel.org
> 	Nothing matching that query.
> 	Trying to find matching series by patch-id adf0918395a1a601a8d54d7d26446aadfb6101c2
> 	Grabbing search results from lore.kernel.org
> 	Nothing matching that query.
> 	Attempting to match by author and subject...
> 	Grabbing search results from lore.kernel.org
> 	Found 22 matching messages
> 	---
> 	This patch is present in the following series:
> 	---
> 	  v3: [PATCH v3] tpm: Prevent local DOS via tpm/tpm0/ppi/*operations
> 		  https://lore.kernel.org/r/20250829175152.9704-2-daleksan@redhat.com
> 	  v4: [PATCH v4] tpm: Prevent local DOS via tpm/tpm0/ppi/*operations
> 		  https://lore.kernel.org/r/20250902142429.14041-2-daleksan@redhat.com
> 	  v5: [PATCH v5] tpm: Prevent local DOS via tpm/tpm0/ppi/*operations
> 		  https://lore.kernel.org/r/20250915210829.6661-1-daleksan@redhat.com
> 
> There are known annoyances:
> 
> 1. We don't currently weed out AUTOSEL/stable backports, so older commits may
>    end up in lots of false positives for now. I've not yet figured out how to
>    avoid this without hardcoding it to be kernel-specific.
> 2. Patches that were sent as part of one series and then as part of another
>    series (by the same author) don't currently do the right thing.
> 
> I'll work on getting that fixed, but for now if you're missing the presence of
> Link: trailers, please test it out.
> 
> -K
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [b4] initial "b4 dig" to supplant Link: trailers

[Linus Torvalds]

On Mon, 13 Oct 2025 at 10:20, Mark Brown <broonie@kernel.org> wrote:
>
> If I specify:
>
>    b4 dig -c regulator-v6.18
>
> b4 complains:
>
>    Digging into commit b320fce86c04862ff5548ebdd84740e36e79e6ac
>    Merge commit detected, please specify a single-parent commit.
>
> but that's not a merge commit, it's a signed tag on a regular commit.

Ahh. b4 does

    commit = b4.git_revparse_obj(cmdargs.commit_id, topdir)

without actually ever asking for the thing to be converted to a commit.

So a tag will stay a tag, and then later when using "git show" to pick
out the commit info, it all goes to mush.

The trivial fix is likely something like

    commit = b4.git_revparse_obj(f'{cmdargs.commit_id}^0', topdir)

although the "proper" way is to use the obj^{commit} syntax, but for
f-strings you'd apparently need to double up on the curly braces and
it turns kind of messy.

            Linus

Re: [b4] initial "b4 dig" to supplant Link: trailers

[Konstantin Ryabitsev]

On Fri, Oct 10, 2025 at 04:47:27PM -0400, Konstantin Ryabitsev wrote:

Seems working on this right before disappearing for the long weekend due to
Canadian Thanksgiving was not the best choice. :)

I've carefully read everyone's feedback, and made significant changes to how
"b4 dig" operates. Specifically:

"b4 dig -c [commitish]" will now output a single link to stdout (all other
output going to stderr), so that it can be piped to other commands. It should
be the latest matching patch received by lore prior and including the "author
date" of the commit. This weeds out all ensuing backports and cherry-picking
efforts.

E.g.:

    $ b4 dig -c 57ce9f7793b714fb14a97d502ce926162c3b96b1
    Digging into commit 57ce9f7793b714fb14a97d502ce926162c3b96b1
    Attempting to match by exact patch-id...
    Trying to find matching series by patch-id ec28d569f47e7a709cc4e8ca022480ded520d928 (myers)
    Grabbing search results from lore.kernel.org
    Found matching series by patch-id
    Will consider promising messages: 3
    Looking up https://lore.kernel.org/r/20251006190854.103483-2-pc@manguebit.org
    Looking up https://lore.kernel.org/r/20251006164220.67333-2-pc@manguebit.org
    Looking up https://lore.kernel.org/r/20251007192326.234467-2-pc@manguebit.org
    ---
    [PATCH v3 2/4] smb: client: fix missing timestamp updates after ftruncate(2)
    https://lore.kernel.org/r/20251007192326.234467-2-pc@manguebit.org

"b4 dig -c [commitish] --all-series" will now try to find all the series where
that patch belongs and try to get you links to all prior revisions, even if
they don't have that very patch (e.g. because it only appeared in v3).

E.g. for the same commit:

    $ b4 dig -c 57ce9f7793b714fb14a97d502ce926162c3b96b1 --all-series
    Digging into commit 57ce9f7793b714fb14a97d502ce926162c3b96b1
    Attempting to match by exact patch-id...
    Trying to find matching series by patch-id ec28d569f47e7a709cc4e8ca022480ded520d928 (myers)
    Found matching series by patch-id
    Will consider promising messages: 3
    Looking up https://lore.kernel.org/r/20251007192326.234467-2-pc@manguebit.org
    Looking up https://lore.kernel.org/r/20251006164220.67333-2-pc@manguebit.org
    Looking up https://lore.kernel.org/r/20251006190854.103483-2-pc@manguebit.org
    Grabbing search results from lore.kernel.org
    ---
    This patch belongs in the following series:
    ---
      v1: [PATCH 1/3] smb: client: fix missing timestamp updates with O_TRUNC
          https://lore.kernel.org/r/20251006164220.67333-2-pc@manguebit.org
      v2: [PATCH v2 1/3] smb: client: fix missing timestamp updates with O_TRUNC
          https://lore.kernel.org/r/20251006190854.103483-2-pc@manguebit.org
      v3: [PATCH v3 1/4] smb: client: fix missing timestamp updates with O_TRUNC
          https://lore.kernel.org/r/20251007192326.234467-2-pc@manguebit.org

This change addresses the following known annoyances:

> 1. We don't currently weed out AUTOSEL/stable backports, so older commits may
>    end up in lots of false positives for now. I've not yet figured out how to
>    avoid this without hardcoding it to be kernel-specific.

This should now be filtered out due to limiting results by commit date.

> 2. Patches that were sent as part of one series and then as part of another
>    series (by the same author) don't currently do the right thing.

This should be doing the right thing now, but needs more testing.

-K

Re: [b4] initial "b4 dig" to supplant Link: trailers

[Sasha Levin]

On Tue, Oct 14, 2025 at 05:17:43PM -0400, Konstantin Ryabitsev wrote:
>On Fri, Oct 10, 2025 at 04:47:27PM -0400, Konstantin Ryabitsev wrote:
>
>Seems working on this right before disappearing for the long weekend due to
>Canadian Thanksgiving was not the best choice. :)
>
>I've carefully read everyone's feedback, and made significant changes to how
>"b4 dig" operates. Specifically:
>
>"b4 dig -c [commitish]" will now output a single link to stdout (all other
>output going to stderr), so that it can be piped to other commands. It should
>be the latest matching patch received by lore prior and including the "author
>date" of the commit. This weeds out all ensuing backports and cherry-picking
>efforts.

Nice!

I've tested it a bit and ended up with a few suggestions :)


1. If the provided commit actually has a Link: trailer, we should just use
that(?).


2. Some maintainers re-send patches they've accepted. An example:

$ ~/b4/b4.sh dig -c 54b91e54b113d4f15ab023a44f508251db6e22e7 
Digging into commit 54b91e54b113d4f15ab023a44f508251db6e22e7
Attempting to match by exact patch-id...
Trying to find matching series by patch-id 340e7baae92a71dd5e62a4e8a52245348ff3a0a1 (myers)
Found matching series by patch-id
Will consider promising messages: 2
---
[for-linus][PATCH v2 2/2] tracing: Stop fortify-string from warning in tracing_mark_raw_write()
https://lore.kernel.org/r/20251011194257.341582199@kernel.org

Which doesn't end up pointing to the actual submission. I'm not really sure
whats the "correct" way of addressing this...


3. Date filtering seems to be broken:

$ ~/b4/b4.sh dig -c 41b1f9fcba62b06195e625bb88c1031102892439
Digging into commit 41b1f9fcba62b06195e625bb88c1031102892439
Attempting to match by exact patch-id...
Trying to find matching series by patch-id 3f5f81f85a73d73ff83e8c0cb9a4fe81d5ba4619 (myers)
Grabbing search results from lore.kernel.org
Nothing matching that query.
Attempting to match by author and subject...
Grabbing search results from lore.kernel.org
Nothing matching that query.
Could not find anything matching commit 41b1f9fcba62b06195e625bb88c1031102892439

It doesn't look like lore takes a range in the form of rt:..{date}. We probably need something like:

--- a/src/b4/dig.py
+++ b/src/b4/dig.py
@@ -138,7 +138,7 @@ def dig_commitish(cmdargs: argparse.Namespace) -> None:
          logger.info('Trying to find matching series by patch-id %s (%s)', patch_id, algo)
          # Limit lookup by date prior to the commit date, to weed out any false-positives from
          # backports or from erroneously resent series
-        extra_query = f'AND rt:..{cdate}'
+        extra_query = f'rt:{cdate}'
          logger.debug('extra_query=%s', extra_query)
          msgs = b4.get_msgs_by_patch_id(patch_id, nocache=cmdargs.nocache, extra_query=extra_query)
          if msgs:

At which point:

$ ~/b4/b4.sh dig -c 41b1f9fcba62b06195e625bb88c1031102892439
Digging into commit 41b1f9fcba62b06195e625bb88c1031102892439
Attempting to match by exact patch-id...
Trying to find matching series by patch-id 3f5f81f85a73d73ff83e8c0cb9a4fe81d5ba4619 (myers)
Grabbing search results from lore.kernel.org
Found matching series by patch-id
Will consider promising messages: 1
Grabbing thread from lore.kernel.org/all/20250901215241.14667-1-mwen@igalia.com/t.mbox.gz
---
[PATCH] drm/amd/display: remove output_tf_change flag
https://lore.kernel.org/r/20250901215241.14667-1-mwen@igalia.com


-- 
Thanks,
Sasha

Re: [b4] initial "b4 dig" to supplant Link: trailers

[Konstantin Ryabitsev]

On Tue, Oct 14, 2025 at 05:49:31PM -0400, Sasha Levin wrote:
> 1. If the provided commit actually has a Link: trailer, we should just use
> that(?).

We do, but we now do it better in the latest master.

> 2. Some maintainers re-send patches they've accepted. An example:
> 
> $ ~/b4/b4.sh dig -c 54b91e54b113d4f15ab023a44f508251db6e22e7 Digging into
> commit 54b91e54b113d4f15ab023a44f508251db6e22e7
> Attempting to match by exact patch-id...
> Trying to find matching series by patch-id 340e7baae92a71dd5e62a4e8a52245348ff3a0a1 (myers)
> Found matching series by patch-id
> Will consider promising messages: 2
> ---
> [for-linus][PATCH v2 2/2] tracing: Stop fortify-string from warning in tracing_mark_raw_write()
> https://lore.kernel.org/r/20251011194257.341582199@kernel.org
> 
> Which doesn't end up pointing to the actual submission. I'm not really sure
> whats the "correct" way of addressing this...

There isn't a good way to deal with this. In this particular case there's a
Link: trailer in the commit, so we can filter by that to find the right patch,
but failing that it's a pattern that will always "win" because it's newer and
from the same author.

> 3. Date filtering seems to be broken:

Rather, I found that using rt: is not always effective for some reason, so I
switched to using d:, which seems to be more reliable.

-K

Re: [b4] initial "b4 dig" to supplant Link: trailers

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 888 bytes --]

On Tue, Oct 14, 2025 at 05:17:43PM -0400, Konstantin Ryabitsev wrote:

> "b4 dig -c [commitish]" will now output a single link to stdout (all other
> output going to stderr), so that it can be piped to other commands. It should
> be the latest matching patch received by lore prior and including the "author
> date" of the commit. This weeds out all ensuing backports and cherry-picking
> efforts.

It would be really handy to have a --mbox (or something) option that
will download the mailbox like b4 mbox would, my main use for this (and
for the existing link tags) is to pull the message ID out of the link
and feed that into b4 mbox to get a mailbox which is what I'll actually
look at and work with.  Obviously that can be scripted locally but it
seems like it might be a common enough need to automate directly,
especially given that the scripting will be going right back into b4.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [b4] initial "b4 dig" to supplant Link: trailers

[Konstantin Ryabitsev]

On Tue, Oct 14, 2025 at 11:13:00PM +0100, Mark Brown wrote:
> On Tue, Oct 14, 2025 at 05:17:43PM -0400, Konstantin Ryabitsev wrote:
> 
> > "b4 dig -c [commitish]" will now output a single link to stdout (all other
> > output going to stderr), so that it can be piped to other commands. It should
> > be the latest matching patch received by lore prior and including the "author
> > date" of the commit. This weeds out all ensuing backports and cherry-picking
> > efforts.
> 
> It would be really handy to have a --mbox (or something) option that
> will download the mailbox like b4 mbox would, my main use for this (and
> for the existing link tags) is to pull the message ID out of the link
> and feed that into b4 mbox to get a mailbox which is what I'll actually
> look at and work with.  Obviously that can be scripted locally but it
> seems like it might be a common enough need to automate directly,
> especially given that the scripting will be going right back into b4.

I just added a --save-mbox option to use with -c. It is currently exclusive
with --all-series, but I can also make it be non-exclusive, so that someone
can save all revisions.

-K

Re: [b4] initial "b4 dig" to supplant Link: trailers

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 506 bytes --]

On Wed, Oct 15, 2025 at 09:44:57AM -0400, Konstantin Ryabitsev wrote:
> On Tue, Oct 14, 2025 at 11:13:00PM +0100, Mark Brown wrote:

> > It would be really handy to have a --mbox (or something) option that
> > will download the mailbox like b4 mbox would, my main use for this (and

> I just added a --save-mbox option to use with -c. It is currently exclusive
> with --all-series, but I can also make it be non-exclusive, so that someone
> can save all revisions.

That seems to be working great, thanks!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [b4] initial "b4 dig" to supplant Link: trailers

[Rob Herring]

On Wed, Oct 15, 2025 at 8:45 AM Konstantin Ryabitsev
<konstantin@linuxfoundation.org> wrote:
>
> On Tue, Oct 14, 2025 at 11:13:00PM +0100, Mark Brown wrote:
> > On Tue, Oct 14, 2025 at 05:17:43PM -0400, Konstantin Ryabitsev wrote:
> >
> > > "b4 dig -c [commitish]" will now output a single link to stdout (all other
> > > output going to stderr), so that it can be piped to other commands. It should
> > > be the latest matching patch received by lore prior and including the "author
> > > date" of the commit. This weeds out all ensuing backports and cherry-picking
> > > efforts.
> >
> > It would be really handy to have a --mbox (or something) option that
> > will download the mailbox like b4 mbox would, my main use for this (and
> > for the existing link tags) is to pull the message ID out of the link
> > and feed that into b4 mbox to get a mailbox which is what I'll actually
> > look at and work with.  Obviously that can be scripted locally but it
> > seems like it might be a common enough need to automate directly,
> > especially given that the scripting will be going right back into b4.
>
> I just added a --save-mbox option to use with -c. It is currently exclusive
> with --all-series, but I can also make it be non-exclusive, so that someone
> can save all revisions.

I would find that useful, and not just for 'b4 dig', but for 'b4
mbox'. My workflow is essentially get a msgid from patchwork and then
fetch the thread to read with 'b4 mbox'. A current shortcoming with
that is I don't see the prior versions. Best case is there's a link in
the cover, but often not and that's a switch from mutt to my browser.

Rob

Re: [b4] initial "b4 dig" to supplant Link: trailers

[Martin K. Petersen]

Hi Konstantin!

> "b4 dig -c [commitish] --all-series" will now try to find all the
> series where that patch belongs and try to get you links to all prior
> revisions, even if they don't have that very patch (e.g. because it
> only appeared in v3).

There is some funky stuff going on with the date range limit. b4 dig
currently misses some messages for which the patch-id can be queried on
lore:

    ecode, out = b4.git_run_command(
        topdir, ['show', '--no-patch', '--format=%as %ae %s', commit],
    )

The resulting %as cdate will be in the format "2025-10-07". However,
specifying a query date limit of "rt:..2025-10-07" means the patch mail
will be missed:

    Full query: patchid:3c67a3033590c511bc96819aa2fdc6376c2799e1 AND rt:..2025-10-07 (nocache=False)
    Grabbing search results from lore.kernel.org
    Nothing matching that query.

Whereas my own script finds it:

    $ check-lore 3c67a3033590c511bc96819aa2fdc6376c2799e1
    Found 3c67a3033590c511bc96819aa2fdc6376c2799e1 on lore.kernel.org

Bumping the cdate to 2025-10-08 makes b4 dig find the message.

I don't know anything about Xapian and how it indexes. But I assume
there might be some fun with time zones and DST lurking here...

-- 
Martin K. Petersen

Re: [b4] initial "b4 dig" to supplant Link: trailers

[Konstantin Ryabitsev]

On Tue, Oct 14, 2025 at 10:52:42PM -0400, Martin K. Petersen wrote:
> There is some funky stuff going on with the date range limit. b4 dig
> currently misses some messages for which the patch-id can be queried on
> lore:
> 
>     ecode, out = b4.git_run_command(
>         topdir, ['show', '--no-patch', '--format=%as %ae %s', commit],
>     )
> 
> The resulting %as cdate will be in the format "2025-10-07". However,
> specifying a query date limit of "rt:..2025-10-07" means the patch mail
> will be missed:
> 
>     Full query: patchid:3c67a3033590c511bc96819aa2fdc6376c2799e1 AND rt:..2025-10-07 (nocache=False)
>     Grabbing search results from lore.kernel.org
>     Nothing matching that query.
> 
> Whereas my own script finds it:
> 
>     $ check-lore 3c67a3033590c511bc96819aa2fdc6376c2799e1
>     Found 3c67a3033590c511bc96819aa2fdc6376c2799e1 on lore.kernel.org
> 
> Bumping the cdate to 2025-10-08 makes b4 dig find the message.
> 
> I don't know anything about Xapian and how it indexes. But I assume
> there might be some fun with time zones and DST lurking here...

Yes, this is probably what's going on. I tweaked the filtering to add 24 hours
to the author date just so we don't bump into timezone shenanigans.

-K

Re: [b4] initial "b4 dig" to supplant Link: trailers

[Martin K. Petersen]

Hi Konstantin!

>> I don't know anything about Xapian and how it indexes. But I assume
>> there might be some fun with time zones and DST lurking here...
>
> Yes, this is probably what's going on. I tweaked the filtering to add
> 24 hours to the author date just so we don't bump into timezone
> shenanigans.

OK. That fixed some of my false negatives.

Thanks!

-- 
Martin K. Petersen