[RFC] new design of review prompts management

[Chih-En Lin <shiyn.lin@gmail.com>]

Hi all,

I would like to propose a new prompt management system. This will likely
require changes to the prompt structure and how prompts are managed across
both Sashiko and the review-prompts [1] project. Therefore, I would like to
open this up for discussion before beginning implementation.

Currently, prompts are maintained in Markdown format (including the indexed
version of review-prompts [2]), where all guidelines for a subsystem are
stored in Markdown within a single file.

While this format is convenient for human readers, it lacks the ability to
track at scale which specific parts of the review prompts are effective,
which ones provide incomplete guidance, or which ones cause noise.
Consequently, it is difficult to verify and demonstrate their concrete benefit.

To address this, the proposed prompt management system manages each specific
guideline as an atomic "invariant unit" in JSON format, tracking its usage and
feedback trails during reviews.

Here is the overview of our design in Sashiko:

                     Incoming Email (LKML)
                                  |
                                  v
                      +-----------------------+
                      | Is feedback to a      |
                      | prev Sashiko review?  |
                      +-----------+-----------+
                          /               \
                    Yes  /                 \  No (New patch)
                        /                   \
                       v                     v
        +-----------------------+   +-----------------------+
        | Ingest Feedback       |   | Review Engine         |
        |                       |   |                       |
        | - #sashiko-feedback   |   | - Evaluates patch     |
        |   (explicit ack/nack) |   |   (record which unit  |
        | - LLM sentiment       |   |    is used)           |
        |   (natural replies)   |   +-----------+-----------+
        | - Grace period expiry |               |
        |   (unknown / timeout) |               |
        +-----------+-----------+               v
                    |               +-----------------------+
                    |               | Send feedbacks toLKML |
                    |               | with unit-id          |
                    |               +-----------+-----------+
                    |                           |
                    |                           | Author / Maintainer
                    |                           | replies on LKML
                    |                           | (loops back to ingest)
                    |                           v
                    +-------------------------->+
                                                |
                                                v
                                    +-----------------------+
                                    | Update Trails DB      |
                                    | (nr_used, ack, nack)  |
                                    +-----------+-----------+
                                                |
                                                v
                                    +-----------------------+
                                    | Invariant Analysis &  |
                                    | Prompt Optimization   |
                                    | (Retire noisy rules,  |
                                    |  tune high-value ones)|
                                    +-----------------------+

## Design

The upcoming indexed version of prompts [2] already provides two key benefits:

1. Reduce context window size:
   Retrieves only relevant guidelines instead of dumping entire subsystem prompts.
2. Automated synchronization:
   Keeps subsystem guidelines synchronized across prompt bundles.

The proposed invariant unit based on it to achieve further improvements:

- Identify which guideline is used during review:
    - Allows tracking which guidelines are useful in practice
- Provide metadata per unit to help Sashiko retrieve relevant invariants:
    - For example, during a security review stage, Sashiko can selectively
      retrieve only security-related invariant units.

Here is the prototype schema for an invariant unit:

{
  "seqid": <...>,  /* unique invariant sequence number within the file */
  "title": <...>,
  "type": <generic|concurrency|security|etc>,
  "source": {
    "type": <email-thread|commit|source-code|human-edited|etc>,
    "value": <message-id|commit-SHA|source-code-version|name-of-human|etc>
  },
  "last-modified-time": <...>,
  "context": <...>,
}

To cleanly identify which invariant Sashiko uses, we form a ID from the
filename and its seqid: <filename>-<id>.  This also helps the indexed
version [2] achieve cleaner referencing (i.e., citing a invariant ID rather
than a file line number like <filename>.md:<number>).
For example, the mm-reclaim-1 invariant in mm-reclaim.json:

{
  "seqid": 1,
  "title": "...",
  "type": "concurrency",
  "source": {
    "type": "email-thread",
    "value": "XXXX-XXX@example.com"
  },
  "last-modified-time": "2026-10-21",
  "context": "..."
}

Later, when reviewing a patch, Sashiko records which invariants were applied,
and labels each review finding with its ID.
For example, when Sashiko posts feedback, it will contain:

>
> <patch context>
>
[Severity: XXX][invariant: mm-reclaim-1, mm-reclaim-3]
... <sashiko feedback> ...


Maintainers (or authors) can confirm or refute findings in their email replies
using rule-targeted command tags to indicate which invariant was useful:

#sashiko-feedback: mm-reclaim-1 ack
#sashiko-feedback: mm-reclaim-3 nack

This allows Sashiko to collect direct feedback from developers. If there is
no explicit command, we can either categorize the finding as unknown after a
grace period (e.g., 24 hours or 7 days), or, if there is a natural reply
without an explicit command, categorize developer thought with LLM.

Sashiko then maintains a database aggregating all invariant metrics.
Here is an abstraction of the tracked telemetry:

"trails": {
  "mm-reclaim-1": {
    "nr_used": 18,
    "ack": 15,
    "nack": 1,
    "unknown": 2
  },
  ...
}

"nr_used" counts how many times this unit was cited by Sashiko in review feedback.

## Discussions

In addition to the above design, there are serveral things for discussion:

- What metadata should an invariant unit contain?
    Depending on the prompt indexing design [2], if prompt files are rebuilt
    from scratch for every kernel version, "source" and "last-modified-time"
    fields may not be strictly necessary. We could also consider adding fields
    to record which questions or checklists generated the unit. On the other
    hand, if prompts are maintained and synced incrementally, preserving
    provenance and persistent IDs allows us to maintain long-term telemetry
    trails and perform filtering, ranking, and analysis on guidelines.

- Whose feedback (e.g., ack) should be counted?
    The straightforward approach is to count feedback only from maintainers,
    but this places extra overhead on them. An alternative is to consider both
    patch authors and maintainers, perhaps with a time buffer before finalizing
    feedback status so maintainers or other reviewers have an opportunity to
    review or override it.

- How to let the human read the prompts after chaning to unit-based one?
    We can provide the script to convert it to the Markdown format. But,
    this means that either we need to have a copy in the repo or the one
    who wants to read will have to download the repo and run the script
    manually...

- How to interact with embargoed feedback?
    Probably just hold until it is released, but need more consideration.


[1] https://github.com/masoncl/review-prompts.git
[2] https://lore.kernel.org/sashiko/91e21331-ed6d-4ac0-b237-b01aeadca2d1@app.fastmail.com/


-- 
2.43.0