* [RFC] new design of review prompts management
@ 2026-10-21 23:44 Chih-En Lin
2026-10-22 19:50 ` Roman Gushchin
0 siblings, 1 reply; 4+ messages in thread
From: Chih-En Lin @ 2026-10-21 23:44 UTC (permalink / raw)
To: sashiko, Roman Gushchin, Chris Mason; +Cc: shiyn.lin, rawat10, pfonseca
Hi all,
I would like to propose a new prompt management system. This will likely
require changes to the prompt structure and how prompts are managed across
both Sashiko and the review-prompts [1] project. Therefore, I would like to
open this up for discussion before beginning implementation.
Currently, prompts are maintained in Markdown format (including the indexed
version of review-prompts [2]), where all guidelines for a subsystem are
stored in Markdown within a single file.
While this format is convenient for human readers, it lacks the ability to
track at scale which specific parts of the review prompts are effective,
which ones provide incomplete guidance, or which ones cause noise.
Consequently, it is difficult to verify and demonstrate their concrete benefit.
To address this, the proposed prompt management system manages each specific
guideline as an atomic "invariant unit" in JSON format, tracking its usage and
feedback trails during reviews.
Here is the overview of our design in Sashiko:
Incoming Email (LKML)
|
v
+-----------------------+
| Is feedback to a |
| prev Sashiko review? |
+-----------+-----------+
/ \
Yes / \ No (New patch)
/ \
v v
+-----------------------+ +-----------------------+
| Ingest Feedback | | Review Engine |
| | | |
| - #sashiko-feedback | | - Evaluates patch |
| (explicit ack/nack) | | (record which unit |
| - LLM sentiment | | is used) |
| (natural replies) | +-----------+-----------+
| - Grace period expiry | |
| (unknown / timeout) | |
+-----------+-----------+ v
| +-----------------------+
| | Send feedbacks toLKML |
| | with unit-id |
| +-----------+-----------+
| |
| | Author / Maintainer
| | replies on LKML
| | (loops back to ingest)
| v
+-------------------------->+
|
v
+-----------------------+
| Update Trails DB |
| (nr_used, ack, nack) |
+-----------+-----------+
|
v
+-----------------------+
| Invariant Analysis & |
| Prompt Optimization |
| (Retire noisy rules, |
| tune high-value ones)|
+-----------------------+
## Design
The upcoming indexed version of prompts [2] already provides two key benefits:
1. Reduce context window size:
Retrieves only relevant guidelines instead of dumping entire subsystem prompts.
2. Automated synchronization:
Keeps subsystem guidelines synchronized across prompt bundles.
The proposed invariant unit based on it to achieve further improvements:
- Identify which guideline is used during review:
- Allows tracking which guidelines are useful in practice
- Provide metadata per unit to help Sashiko retrieve relevant invariants:
- For example, during a security review stage, Sashiko can selectively
retrieve only security-related invariant units.
Here is the prototype schema for an invariant unit:
{
"seqid": <...>, /* unique invariant sequence number within the file */
"title": <...>,
"type": <generic|concurrency|security|etc>,
"source": {
"type": <email-thread|commit|source-code|human-edited|etc>,
"value": <message-id|commit-SHA|source-code-version|name-of-human|etc>
},
"last-modified-time": <...>,
"context": <...>,
}
To cleanly identify which invariant Sashiko uses, we form a ID from the
filename and its seqid: <filename>-<id>. This also helps the indexed
version [2] achieve cleaner referencing (i.e., citing a invariant ID rather
than a file line number like <filename>.md:<number>).
For example, the mm-reclaim-1 invariant in mm-reclaim.json:
{
"seqid": 1,
"title": "...",
"type": "concurrency",
"source": {
"type": "email-thread",
"value": "XXXX-XXX@example.com"
},
"last-modified-time": "2026-10-21",
"context": "..."
}
Later, when reviewing a patch, Sashiko records which invariants were applied,
and labels each review finding with its ID.
For example, when Sashiko posts feedback, it will contain:
>
> <patch context>
>
[Severity: XXX][invariant: mm-reclaim-1, mm-reclaim-3]
... <sashiko feedback> ...
Maintainers (or authors) can confirm or refute findings in their email replies
using rule-targeted command tags to indicate which invariant was useful:
#sashiko-feedback: mm-reclaim-1 ack
#sashiko-feedback: mm-reclaim-3 nack
This allows Sashiko to collect direct feedback from developers. If there is
no explicit command, we can either categorize the finding as unknown after a
grace period (e.g., 24 hours or 7 days), or, if there is a natural reply
without an explicit command, categorize developer thought with LLM.
Sashiko then maintains a database aggregating all invariant metrics.
Here is an abstraction of the tracked telemetry:
"trails": {
"mm-reclaim-1": {
"nr_used": 18,
"ack": 15,
"nack": 1,
"unknown": 2
},
...
}
"nr_used" counts how many times this unit was cited by Sashiko in review feedback.
## Discussions
In addition to the above design, there are serveral things for discussion:
- What metadata should an invariant unit contain?
Depending on the prompt indexing design [2], if prompt files are rebuilt
from scratch for every kernel version, "source" and "last-modified-time"
fields may not be strictly necessary. We could also consider adding fields
to record which questions or checklists generated the unit. On the other
hand, if prompts are maintained and synced incrementally, preserving
provenance and persistent IDs allows us to maintain long-term telemetry
trails and perform filtering, ranking, and analysis on guidelines.
- Whose feedback (e.g., ack) should be counted?
The straightforward approach is to count feedback only from maintainers,
but this places extra overhead on them. An alternative is to consider both
patch authors and maintainers, perhaps with a time buffer before finalizing
feedback status so maintainers or other reviewers have an opportunity to
review or override it.
- How to let the human read the prompts after chaning to unit-based one?
We can provide the script to convert it to the Markdown format. But,
this means that either we need to have a copy in the repo or the one
who wants to read will have to download the repo and run the script
manually...
- How to interact with embargoed feedback?
Probably just hold until it is released, but need more consideration.
[1] https://github.com/masoncl/review-prompts.git
[2] https://lore.kernel.org/sashiko/91e21331-ed6d-4ac0-b237-b01aeadca2d1@app.fastmail.com/
--
2.43.0
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [RFC] new design of review prompts management
2026-10-21 23:44 [RFC] new design of review prompts management Chih-En Lin
@ 2026-10-22 19:50 ` Roman Gushchin
2026-10-23 1:16 ` Chih-En Lin
0 siblings, 1 reply; 4+ messages in thread
From: Roman Gushchin @ 2026-10-22 19:50 UTC (permalink / raw)
To: Chih-En Lin; +Cc: sashiko, Chris Mason, rawat10, pfonseca
Chih-En Lin <shiyn.lin@gmail.com> writes:
> Hi all,
Hello,
Thank you for the proposal!
Many ideas here resonate with what I was working on/was thinking of
recently. I'm about to publish a big rework of prompts which separates
them in auto-generated part and subjective small part for maintainers.
It's somewhat close to what you're proposing, but not fully. Let's maybe
restart this discussion after this?
I'm sure there will be a ton of room for further improvements...
Thanks
> I would like to propose a new prompt management system. This will likely
> require changes to the prompt structure and how prompts are managed across
> both Sashiko and the review-prompts [1] project. Therefore, I would like to
> open this up for discussion before beginning implementation.
>
> Currently, prompts are maintained in Markdown format (including the indexed
> version of review-prompts [2]), where all guidelines for a subsystem are
> stored in Markdown within a single file.
>
> While this format is convenient for human readers, it lacks the ability to
> track at scale which specific parts of the review prompts are effective,
> which ones provide incomplete guidance, or which ones cause noise.
> Consequently, it is difficult to verify and demonstrate their concrete benefit.
>
> To address this, the proposed prompt management system manages each specific
> guideline as an atomic "invariant unit" in JSON format, tracking its usage and
> feedback trails during reviews.
>
> Here is the overview of our design in Sashiko:
>
> Incoming Email (LKML)
> |
> v
> +-----------------------+
> | Is feedback to a |
> | prev Sashiko review? |
> +-----------+-----------+
> / \
> Yes / \ No (New patch)
> / \
> v v
> +-----------------------+ +-----------------------+
> | Ingest Feedback | | Review Engine |
> | | | |
> | - #sashiko-feedback | | - Evaluates patch |
> | (explicit ack/nack) | | (record which unit |
> | - LLM sentiment | | is used) |
> | (natural replies) | +-----------+-----------+
> | - Grace period expiry | |
> | (unknown / timeout) | |
> +-----------+-----------+ v
> | +-----------------------+
> | | Send feedbacks toLKML |
> | | with unit-id |
> | +-----------+-----------+
> | |
> | | Author / Maintainer
> | | replies on LKML
> | | (loops back to ingest)
> | v
> +-------------------------->+
> |
> v
> +-----------------------+
> | Update Trails DB |
> | (nr_used, ack, nack) |
> +-----------+-----------+
> |
> v
> +-----------------------+
> | Invariant Analysis & |
> | Prompt Optimization |
> | (Retire noisy rules, |
> | tune high-value ones)|
> +-----------------------+
>
> ## Design
>
> The upcoming indexed version of prompts [2] already provides two key benefits:
>
> 1. Reduce context window size:
> Retrieves only relevant guidelines instead of dumping entire subsystem prompts.
> 2. Automated synchronization:
> Keeps subsystem guidelines synchronized across prompt bundles.
>
> The proposed invariant unit based on it to achieve further improvements:
>
> - Identify which guideline is used during review:
> - Allows tracking which guidelines are useful in practice
> - Provide metadata per unit to help Sashiko retrieve relevant invariants:
> - For example, during a security review stage, Sashiko can selectively
> retrieve only security-related invariant units.
>
> Here is the prototype schema for an invariant unit:
>
> {
> "seqid": <...>, /* unique invariant sequence number within the file */
> "title": <...>,
> "type": <generic|concurrency|security|etc>,
> "source": {
> "type": <email-thread|commit|source-code|human-edited|etc>,
> "value": <message-id|commit-SHA|source-code-version|name-of-human|etc>
> },
> "last-modified-time": <...>,
> "context": <...>,
> }
>
> To cleanly identify which invariant Sashiko uses, we form a ID from the
> filename and its seqid: <filename>-<id>. This also helps the indexed
> version [2] achieve cleaner referencing (i.e., citing a invariant ID rather
> than a file line number like <filename>.md:<number>).
> For example, the mm-reclaim-1 invariant in mm-reclaim.json:
>
> {
> "seqid": 1,
> "title": "...",
> "type": "concurrency",
> "source": {
> "type": "email-thread",
> "value": "XXXX-XXX@example.com"
> },
> "last-modified-time": "2026-10-21",
> "context": "..."
> }
>
> Later, when reviewing a patch, Sashiko records which invariants were applied,
> and labels each review finding with its ID.
> For example, when Sashiko posts feedback, it will contain:
>
>>
>> <patch context>
>>
> [Severity: XXX][invariant: mm-reclaim-1, mm-reclaim-3]
> ... <sashiko feedback> ...
>
>
> Maintainers (or authors) can confirm or refute findings in their email replies
> using rule-targeted command tags to indicate which invariant was useful:
>
> #sashiko-feedback: mm-reclaim-1 ack
> #sashiko-feedback: mm-reclaim-3 nack
>
> This allows Sashiko to collect direct feedback from developers. If there is
> no explicit command, we can either categorize the finding as unknown after a
> grace period (e.g., 24 hours or 7 days), or, if there is a natural reply
> without an explicit command, categorize developer thought with LLM.
>
> Sashiko then maintains a database aggregating all invariant metrics.
> Here is an abstraction of the tracked telemetry:
>
> "trails": {
> "mm-reclaim-1": {
> "nr_used": 18,
> "ack": 15,
> "nack": 1,
> "unknown": 2
> },
> ...
> }
>
> "nr_used" counts how many times this unit was cited by Sashiko in review feedback.
>
> ## Discussions
>
> In addition to the above design, there are serveral things for discussion:
>
> - What metadata should an invariant unit contain?
> Depending on the prompt indexing design [2], if prompt files are rebuilt
> from scratch for every kernel version, "source" and "last-modified-time"
> fields may not be strictly necessary. We could also consider adding fields
> to record which questions or checklists generated the unit. On the other
> hand, if prompts are maintained and synced incrementally, preserving
> provenance and persistent IDs allows us to maintain long-term telemetry
> trails and perform filtering, ranking, and analysis on guidelines.
>
> - Whose feedback (e.g., ack) should be counted?
> The straightforward approach is to count feedback only from maintainers,
> but this places extra overhead on them. An alternative is to consider both
> patch authors and maintainers, perhaps with a time buffer before finalizing
> feedback status so maintainers or other reviewers have an opportunity to
> review or override it.
>
> - How to let the human read the prompts after chaning to unit-based one?
> We can provide the script to convert it to the Markdown format. But,
> this means that either we need to have a copy in the repo or the one
> who wants to read will have to download the repo and run the script
> manually...
>
> - How to interact with embargoed feedback?
> Probably just hold until it is released, but need more consideration.
>
>
> [1] https://github.com/masoncl/review-prompts.git
> [2] https://lore.kernel.org/sashiko/91e21331-ed6d-4ac0-b237-b01aeadca2d1@app.fastmail.com/
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [RFC] new design of review prompts management
2026-10-22 19:50 ` Roman Gushchin
@ 2026-10-23 1:16 ` Chih-En Lin
2026-10-23 1:26 ` Roman Gushchin
0 siblings, 1 reply; 4+ messages in thread
From: Chih-En Lin @ 2026-10-23 1:16 UTC (permalink / raw)
To: Roman Gushchin; +Cc: sashiko, Chris Mason, rawat10, pfonseca
On Thu, Oct 22, 2026 at 07:50:02PM +0000, Roman Gushchin wrote:
> Chih-En Lin <shiyn.lin@gmail.com> writes:
>
> > Hi all,
>
> Hello,
>
> Thank you for the proposal!
Hi,
Thanks for the reply!
> Many ideas here resonate with what I was working on/was thinking of
> recently. I'm about to publish a big rework of prompts which separates
> them in auto-generated part and subjective small part for maintainers.
I wonder if that are:
https://lore.kernel.org/sashiko/91e21331-ed6d-4ac0-b237-b01aeadca2d1@app.fastmail.com/
https://github.com/masoncl/review-prompts.git subsystem-build
> It's somewhat close to what you're proposing, but not fully. Let's maybe
> restart this discussion after this?
Sure :-)
> I'm sure there will be a ton of room for further improvements...
We actually have other more ideas (included and not included in [1]) to
improve this type of patch review system, but most of them are based on
the invariant unit. And, since the kernel community has one already, we
are happy to contribute.
Thanks,
Chih-En Lin
---
[1] https://arxiv.org/abs/2603.24825
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [RFC] new design of review prompts management
2026-10-23 1:16 ` Chih-En Lin
@ 2026-10-23 1:26 ` Roman Gushchin
0 siblings, 0 replies; 4+ messages in thread
From: Roman Gushchin @ 2026-10-23 1:26 UTC (permalink / raw)
To: Chih-En Lin; +Cc: sashiko, Chris Mason, rawat10, pfonseca
Chih-En Lin <shiyn.lin@gmail.com> writes:
> On Thu, Oct 22, 2026 at 07:50:02PM +0000, Roman Gushchin wrote:
>> Chih-En Lin <shiyn.lin@gmail.com> writes:
>>
>> > Hi all,
>>
>> Hello,
>>
>> Thank you for the proposal!
>
> Hi,
>
> Thanks for the reply!
>
>> Many ideas here resonate with what I was working on/was thinking of
>> recently. I'm about to publish a big rework of prompts which separates
>> them in auto-generated part and subjective small part for maintainers.
>
> I wonder if that are:
>
> https://lore.kernel.org/sashiko/91e21331-ed6d-4ac0-b237-b01aeadca2d1@app.fastmail.com/
> https://github.com/masoncl/review-prompts.git subsystem-build
Not quite that, but to some extent a step towards the same direction...
I'll share it soon.
>> It's somewhat close to what you're proposing, but not fully. Let's maybe
>> restart this discussion after this?
>
> Sure :-)
>
>> I'm sure there will be a ton of room for further improvements...
>
> We actually have other more ideas (included and not included in [1]) to
> improve this type of patch review system, but most of them are based on
> the invariant unit. And, since the kernel community has one already, we
> are happy to contribute.
And this is great!
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2026-10-23 1:26 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-10-21 23:44 [RFC] new design of review prompts management Chih-En Lin
2026-10-22 19:50 ` Roman Gushchin
2026-10-23 1:16 ` Chih-En Lin
2026-10-23 1:26 ` Roman Gushchin
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.