Re: [RFC] devtools: rewrite doc vs code check in Python

[Stephen Hemminger <stephen@networkplumber.org>]

On Thu, 24 Sep 2026 10:36:42 +0200
Thomas Monjalon <thomas@monjalon.net> wrote:

> 24/09/2026 09:33, Bruce Richardson:
> > On Wed, Sep 23, 2026 at 01:54:30PM -0700, Stephen Hemminger wrote:  
> > > On Wed, 23 Sep 2026 21:19:45 +0200
> > > Thomas Monjalon <thomas@monjalon.net> wrote:
> > >   
> > > > 23/09/2026 20:42, Stephen Hemminger:  
> > > > > The existing check-doc-vs-code.sh only compares rte_flow items and
> > > > > actions, and only for drivers whose directory matches the ini name,
> > > > > so none of the drivers under net/intel are checked.
> > > > > 
> > > > > Replace it and parse-flow-support.sh with a Python script covering
> > > > > the whole NIC feature matrix:    
> > > > [...]  
> > > > >  devtools/check-doc-vs-code.py          | 1132 ++++++++++++++++++++++++
> > > > >  devtools/check-doc-vs-code.sh          |   84 --
> > > > >  devtools/parse-flow-support.sh         |   92 --
> > > > >  doc/guides/contributing/new_driver.rst |    4 +-
> > > > >  doc/guides/contributing/patches.rst    |   27 +
> > > > >  doc/guides/nics/features.rst           |    5 +
> > > > >  8 files changed, 1169 insertions(+), 180 deletions(-)
> > > > >  create mode 100755 devtools/check-doc-vs-code.py
> > > > >  delete mode 100755 devtools/check-doc-vs-code.sh
> > > > >  delete mode 100755 devtools/parse-flow-support.sh    
> > > > 
> > > > Thanks for working on it.
> > > > 
> > > > My concern is how easy it is to maintain for all contributors
> > > > having to insert their rules and exceptions?
> > > > 
> > > > It is replacing less 200 lines with more than 1000 lines
> > > > so it looks a lot more complex.
> > > > It is probably fully generated by AI?
> > > > Can we make it simpler?
> > > > 
> > > >   
> > > 
> > > The other suggestion would be to git rid of the .ini file method
> > > of generating this feature matrix in doc and just have python script
> > > generate it.  Prefer a single source of truth, less work  
> > 
> > +1, I was just going to suggest that when I saw the discussion on this
> > script.
> > In case of autogeneration, for cases like "partial" support, we can have a
> > well-defined comment tag or similar in the code to mark it.  
> 
> I agree with this direction.
> 

The plan AI generated is:

# DPDK NIC feature matrix: generator concept

Handoff note for resuming in a new session.
Branch `doc`, worktree /home/shemminger/DPDK/doc.

## The idea

Today the NIC feature matrix is a **hand-maintained cache of facts that are
already knowable from the code**. That is what produced ~448 doc-vs-code
findings: the cache went stale.

Replace it. Instead of checking docs against code, **generate the doc output
from the code** using the same rules.

Split the current `devtools/check-doc-vs-code.py` into two tools with
genuinely different jobs:

1. **generator** — code -> RST table directly. No `.ini` files at all.
2. **`check-ethdev-ops`** — a linter for *driver code* self-consistency.
   Nothing to do with docs.

The second tool matters because many current "findings" are **not doc bugs and
cannot be fixed by editing docs**:

    ena:   stats_get without stats_reset
    ntnic: mac_addr_add without mac_addr_remove
    pfe:   allmulticast_enable without allmulticast_disable
    mlx5:  flow_ctrl_get only returns an error, leave it NULL
    nfb:   fec_set without fec_get_capability
    xsc:   Rx timestamp offload without read_clock

Those are driver defects. The `OP_PAIRS` and `CODE_IMPLIES` tables already in
the script are `check-ethdev-ops` in embryo — lift them out roughly as-is.

## Current pipeline

    code -> (75 hand-maintained .ini) -> conf.py -> RST table

- `doc/guides/conf.py:168` `generate_overview_table()`, called **22 times**
  across **8 device classes**.
- Feature dirs: nics, bbdevs, vdpadevs, regexdevs, compressdevs, gpus,
  cryptodevs, eventdevs.
- **Only nics has rules.** The other 7 classes have no code-derivation rules,
  so `conf.py` must keep the ini path for them. Two mechanisms will coexist
  unless that is also tackled. This is an open scoping question.

## Is the nics table fully derivable? Yes (measured)

    Features        78 rows:  68 via RULES + 10 platform via meson -> 0 uncovered
    rte_flow items  68 rows:  scan RTE_FLOW_ITEM_TYPE_* tokens
    rte_flow actions 66 rows: scan RTE_FLOW_ACTION_TYPE_* tokens

`check-doc-vs-code.py -g <driver>` already generates a full ini and runs
cleanly for all 75 drivers. The machinery largely exists.

## CRITICAL: naive generation REGRESSES the docs

Generated vs committed across all 75 inis: **only 1/75 match**.
Totals: **+811 rows / -165 rows / ~160 value changes.**

Three distinct causes, each needing a fix before output is publishable:

### 1. Platform rows (~300 bogus additions)
Generator adds `LoongArch64` to 66 drivers, `rv64` to 65, `Power8` to 61,
`ARMv7` to 54 — solely because meson does not *exclude* them.

**"Not forbidden to build" != "supported".** Publishing this asserts tested
support that does not exist. Counter-example already in tree: `af_xdp.ini`
deliberately lists only `x86-64` though meson allows every arch.

Fix: treat meson as an *upper bound* only; keep explicit per-driver platform
claims. Do not assert support from absence of exclusion.

### 2. Non-derivable rows silently deleted
`Usage doc` x38, `SR-IOV` x22 (also `Design doc`, `Perf doc`).
These are the script's `UNCHECKED` set — no code equivalent exists.
Pure information loss. **They need a home.** (Open question below.)

### 3. Partial support flattened: P -> Y, 160 times
Generator cannot express partial support:
`eth` P->Y x19, `vlan` P->Y x14, `Speed capabilities` P->Y x13,
`L4 checksum offload` P->Y x11.

`RULES` already models requirement groups and `support()` already computes
partial (some-but-not-all groups matched) — **generation just discards it.**
Fix: propagate P instead of flattening. Verified real case: igb `eth = P` is
correct and the generator would clobber it to `Y`.

## Open questions for the user

1. **Where do non-derivable facts live?** (`Usage doc`, `Design doc`,
   `Perf doc`, `SR-IOV`, tested-platform claims.) Options: small per-driver
   override file; annotation in driver source; or drop those rows entirely.
2. **Scope across the other 7 device classes** — nics-only generation leaves
   two mechanisms in `conf.py`.

## Also deferred: DRIVERS[] table is brittle (user-flagged)

Hardcoded ini-name -> source-path map in the script.
- `driver_for()` falls back to `(name, 'intel/'+name)` — only ONE vendor dir is
  special-cased. A new vendor subdir, or a driver moving into one, **silently
  stops being checked**: no error, just no coverage. Worst failure mode for a
  linter. `ipn3ke` only works today via that fallback (`intel/ipn3ke`).
- `all_dirs()` hardcodes the same `intel` special case.
- 24/27 entries exist only to disambiguate PF/VF sharing one directory.

Idea: derive the directory from meson/driver registration and recurse vendor
subdirs generically; keep `DRIVERS[]` only for genuine PF/VF ops-regex cases.
**If the table becomes generated documentation rather than a lint heuristic,
this must be solid first.**

## Verified facts worth not re-deriving

- `Code.__init__` walks the **whole** driver dir and concatenates all `.c`/`.h`,
  so multi-file drivers are handled by default. Blind spot is only the 9
  `DRIVERS[]` entries that narrow the *file set* via `files=`
  (e1000, igb, igb_vf, igc, enetc, enetc4, enetc4_vf, ice, ice_dcf).
  The other entries use `ops=` only, which still reads every file.
- **Rule adopted:** before deleting a doc row, grep the whole driver directory
  for the symbol, not just the globbed subset. If it exists outside the glob
  it is a script bug (defer); if absent everywhere the row is genuinely wrong.
- e1000/igb rte_flow was NOT a script bug: `flow_ops_get` and all 7 flow items
  exist only in igb (`igb_flow.c`); em has none. Rows were in the wrong file.
  The *shared* header `e1000_ethdev.h` declaring `eth_igb_tx_done_cleanup` is
  why e1000 wrongly claimed "Free Tx mbuf on demand" — multi-file layout was
  the *cause* of the doc errors, not an obstacle to finding them.

## Commit conventions (verified against DPDK's own checkers)

Always run: `./devtools/check-git-log.sh -nN && ./devtools/checkpatches.sh -nN`
- A title containing "fix" **requires** a `Fixes:` tag or check-git-log fails.
  Generate with:
  `git log -1 --abbrev=12 --format='Fixes: %h ("%s")' <sha>`
- Doc feature-matrix fixes in history also carry `Cc: stable@dpdk.org`.
- Avoiding the word "fix" (e.g. "doc/af_packet: update feature matrix") is
  legitimate when there is no single culprit commit to blame.
- Find the culprit for a doc row: `git log -S'<row text>' -- <ini path>`

## Work already committed on branch `doc` (9 patches, all checker-clean)

    7d85396ec1 net/af_packet: support reading device clock   <- real code fix
    41f385c391 doc/af_packet: update feature matrix
    cffcad6ad2 doc: fix e1000 and igb feature matrix
    3718164550 doc/axgbe: update feature matrix
               doc/mana, doc/memif, doc/pcap, doc/octeon_ep, doc/vhost

Drivers now reporting zero findings: afpacket, e1000, igb, axgbe, mana, memif,
pcap, octeon_ep, vhost.

Recommendation: **keep these.** The af_packet `read_clock` is a genuine code
fix, and the doc ones are correct under either design and shrink the eventual
generated diff.

Uncommitted: nothing. A partial batch (ena, nfb, thunderx, enetfec, pfe,
ntnic, bnx2x) was deliberately **not** applied pending this redesign.

## Constraints from the user

- Drivers with no ini today (bonding, null, ring, softnic): **do not create one.**
- Prefer updating docs over changing drivers; if both, one patch each.
- One patch per driver.
- First pass: fix what is clearly fixable, do not force ambiguous cases.