Re: [RFC] new design of review prompts management

[Roman Gushchin <roman.gushchin@linux.dev>]

Chih-En Lin <shiyn.lin@gmail.com> writes:

> Hi all,

Hello,

Thank you for the proposal!
Many ideas here resonate with what I was working on/was thinking of
recently. I'm about to publish a big rework of prompts which separates
them in auto-generated part and subjective small part for maintainers.
It's somewhat close to what you're proposing, but not fully. Let's maybe
restart this discussion after this?
I'm sure there will be a ton of room for further improvements...

Thanks

> I would like to propose a new prompt management system. This will likely
> require changes to the prompt structure and how prompts are managed across
> both Sashiko and the review-prompts [1] project. Therefore, I would like to
> open this up for discussion before beginning implementation.
>
> Currently, prompts are maintained in Markdown format (including the indexed
> version of review-prompts [2]), where all guidelines for a subsystem are
> stored in Markdown within a single file.
>
> While this format is convenient for human readers, it lacks the ability to
> track at scale which specific parts of the review prompts are effective,
> which ones provide incomplete guidance, or which ones cause noise.
> Consequently, it is difficult to verify and demonstrate their concrete benefit.
>
> To address this, the proposed prompt management system manages each specific
> guideline as an atomic "invariant unit" in JSON format, tracking its usage and
> feedback trails during reviews.
>
> Here is the overview of our design in Sashiko:
>
>                      Incoming Email (LKML)
>                                   |
>                                   v
>                       +-----------------------+
>                       | Is feedback to a      |
>                       | prev Sashiko review?  |
>                       +-----------+-----------+
>                           /               \
>                     Yes  /                 \  No (New patch)
>                         /                   \
>                        v                     v
>         +-----------------------+   +-----------------------+
>         | Ingest Feedback       |   | Review Engine         |
>         |                       |   |                       |
>         | - #sashiko-feedback   |   | - Evaluates patch     |
>         |   (explicit ack/nack) |   |   (record which unit  |
>         | - LLM sentiment       |   |    is used)           |
>         |   (natural replies)   |   +-----------+-----------+
>         | - Grace period expiry |               |
>         |   (unknown / timeout) |               |
>         +-----------+-----------+               v
>                     |               +-----------------------+
>                     |               | Send feedbacks toLKML |
>                     |               | with unit-id          |
>                     |               +-----------+-----------+
>                     |                           |
>                     |                           | Author / Maintainer
>                     |                           | replies on LKML
>                     |                           | (loops back to ingest)
>                     |                           v
>                     +-------------------------->+
>                                                 |
>                                                 v
>                                     +-----------------------+
>                                     | Update Trails DB      |
>                                     | (nr_used, ack, nack)  |
>                                     +-----------+-----------+
>                                                 |
>                                                 v
>                                     +-----------------------+
>                                     | Invariant Analysis &  |
>                                     | Prompt Optimization   |
>                                     | (Retire noisy rules,  |
>                                     |  tune high-value ones)|
>                                     +-----------------------+
>
> ## Design
>
> The upcoming indexed version of prompts [2] already provides two key benefits:
>
> 1. Reduce context window size:
>    Retrieves only relevant guidelines instead of dumping entire subsystem prompts.
> 2. Automated synchronization:
>    Keeps subsystem guidelines synchronized across prompt bundles.
>
> The proposed invariant unit based on it to achieve further improvements:
>
> - Identify which guideline is used during review:
>     - Allows tracking which guidelines are useful in practice
> - Provide metadata per unit to help Sashiko retrieve relevant invariants:
>     - For example, during a security review stage, Sashiko can selectively
>       retrieve only security-related invariant units.
>
> Here is the prototype schema for an invariant unit:
>
> {
>   "seqid": <...>,  /* unique invariant sequence number within the file */
>   "title": <...>,
>   "type": <generic|concurrency|security|etc>,
>   "source": {
>     "type": <email-thread|commit|source-code|human-edited|etc>,
>     "value": <message-id|commit-SHA|source-code-version|name-of-human|etc>
>   },
>   "last-modified-time": <...>,
>   "context": <...>,
> }
>
> To cleanly identify which invariant Sashiko uses, we form a ID from the
> filename and its seqid: <filename>-<id>.  This also helps the indexed
> version [2] achieve cleaner referencing (i.e., citing a invariant ID rather
> than a file line number like <filename>.md:<number>).
> For example, the mm-reclaim-1 invariant in mm-reclaim.json:
>
> {
>   "seqid": 1,
>   "title": "...",
>   "type": "concurrency",
>   "source": {
>     "type": "email-thread",
>     "value": "XXXX-XXX@example.com"
>   },
>   "last-modified-time": "2026-10-21",
>   "context": "..."
> }
>
> Later, when reviewing a patch, Sashiko records which invariants were applied,
> and labels each review finding with its ID.
> For example, when Sashiko posts feedback, it will contain:
>
>>
>> <patch context>
>>
> [Severity: XXX][invariant: mm-reclaim-1, mm-reclaim-3]
> ... <sashiko feedback> ...
>
>
> Maintainers (or authors) can confirm or refute findings in their email replies
> using rule-targeted command tags to indicate which invariant was useful:
>
> #sashiko-feedback: mm-reclaim-1 ack
> #sashiko-feedback: mm-reclaim-3 nack
>
> This allows Sashiko to collect direct feedback from developers. If there is
> no explicit command, we can either categorize the finding as unknown after a
> grace period (e.g., 24 hours or 7 days), or, if there is a natural reply
> without an explicit command, categorize developer thought with LLM.
>
> Sashiko then maintains a database aggregating all invariant metrics.
> Here is an abstraction of the tracked telemetry:
>
> "trails": {
>   "mm-reclaim-1": {
>     "nr_used": 18,
>     "ack": 15,
>     "nack": 1,
>     "unknown": 2
>   },
>   ...
> }
>
> "nr_used" counts how many times this unit was cited by Sashiko in review feedback.
>
> ## Discussions
>
> In addition to the above design, there are serveral things for discussion:
>
> - What metadata should an invariant unit contain?
>     Depending on the prompt indexing design [2], if prompt files are rebuilt
>     from scratch for every kernel version, "source" and "last-modified-time"
>     fields may not be strictly necessary. We could also consider adding fields
>     to record which questions or checklists generated the unit. On the other
>     hand, if prompts are maintained and synced incrementally, preserving
>     provenance and persistent IDs allows us to maintain long-term telemetry
>     trails and perform filtering, ranking, and analysis on guidelines.
>
> - Whose feedback (e.g., ack) should be counted?
>     The straightforward approach is to count feedback only from maintainers,
>     but this places extra overhead on them. An alternative is to consider both
>     patch authors and maintainers, perhaps with a time buffer before finalizing
>     feedback status so maintainers or other reviewers have an opportunity to
>     review or override it.
>
> - How to let the human read the prompts after chaning to unit-based one?
>     We can provide the script to convert it to the Markdown format. But,
>     this means that either we need to have a copy in the repo or the one
>     who wants to read will have to download the repo and run the script
>     manually...
>
> - How to interact with embargoed feedback?
>     Probably just hold until it is released, but need more consideration.
>
>
> [1] https://github.com/masoncl/review-prompts.git
> [2] https://lore.kernel.org/sashiko/91e21331-ed6d-4ac0-b237-b01aeadca2d1@app.fastmail.com/