Re: [docs] [PATCH v2 5/5] tools: Add check-confusables pre-commit hook

["Antonin Godard" <antonin.godard@bootlin.com>]

Hi,

On Tue Jul 28, 2026 at 9:23 AM CEST, Niko Mauno via lists.yoctoproject.org wrote:
> From: Niko Mauno <niko.mauno@vaisala.com>
>
> Add a check-confusables script, in the same fashion as
> check-glossaries, that scans the documentation .rst sources for
> non-ASCII "confusable" characters (curly quotes, en/em dashes,
> horizontal ellipsis, non-breaking and zero-width spaces, etc.) and
> reports each occurrence with its location and suggested ASCII
> replacement, exiting non-zero if any are found. This guards against
> the class of breakage fixed in the preceding commit, e.g. curly quotes
> causing recipe ParseErrors.
>
> Legitimate non-ASCII such as box-drawing characters used in directory
> trees, accented letters in contributor names and CJK characters are
> intentionally left untouched.
>
> Wire it up both as a local pre-commit hook and in the Makefile "checks"
> target, alongside check-glossaries.
>
> Suggested-by: Quentin Schulz <quentin.schulz@cherry.de>
> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
> Signed-off-by: Niko Mauno <niko.mauno@vaisala.com>
> ---
>  .pre-commit-config.yaml               |  5 ++
>  documentation/Makefile                |  1 +
>  documentation/tools/check-confusables | 77 +++++++++++++++++++++++++++
>  3 files changed, 83 insertions(+)
>  create mode 100755 documentation/tools/check-confusables
>
> diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
> index f2b73a481..d6008b609 100644
> --- a/.pre-commit-config.yaml
> +++ b/.pre-commit-config.yaml
> @@ -6,3 +6,8 @@ repos:
>          entry: ./documentation/tools/check-glossaries
>          language: python
>          pass_filenames: false
> +      - id: check-confusables
> +        name: Check for non-ASCII confusable characters
> +        entry: ./documentation/tools/check-confusables
> +        language: python
> +        pass_filenames: false

I think we should pass filenames and check only the modified files, no need to
checking the entire tree each time?

> diff --git a/documentation/Makefile b/documentation/Makefile
> index fe0574537..87a6f8a8b 100644
> --- a/documentation/Makefile
> +++ b/documentation/Makefile
> @@ -37,6 +37,7 @@ clean:
>  
>  checks:
>  	$(SOURCEDIR)/tools/check-glossaries --docs-dir $(SOURCEDIR)
> +	$(SOURCEDIR)/tools/check-confusables --docs-dir $(SOURCEDIR)
>  
>  stylecheck:
>  	vale sync
> diff --git a/documentation/tools/check-confusables b/documentation/tools/check-confusables
> new file mode 100755
> index 000000000..f79ee046c
> --- /dev/null
> +++ b/documentation/tools/check-confusables
> @@ -0,0 +1,77 @@
> +#!/usr/bin/env python3
> +
> +import argparse
> +import sys
> +
> +from pathlib import Path
> +
> +
> +def parse_arguments() -> argparse.Namespace:
> +    parser = argparse.ArgumentParser(
> +        description="Check documentation sources for non-ASCII typographic "
> +                    "characters that should be plain ASCII")
> +
> +    parser.add_argument("-d", "--docs-dir",
> +                        type=Path,
> +                        default=Path(__file__).resolve().parent.parent,
> +                        help="Path to documentation/ directory in yocto-docs")
> +
> +    return parser.parse_args()
> +
> +
> +# Map of "confusable" characters that are frequently introduced by editors,
> +# word processors or copy-pasting, to their plain ASCII replacement. These
> +# look almost identical to regular ASCII but break tooling, e.g. a curly
> +# quote in a recipe example causes:
> +#
> +#   ERROR: ParseError ...: unparsed line: 'RDEPENDS:${PN} = "foo"'

I don't see the curly quote here

> +#
> +# Only these characters are flagged; legitimate non-ASCII such as box-drawing
> +# characters used in directory trees, accented letters in contributor names
> +# and CJK characters are intentionally left alone.
> +confusables = {
> +    "‘": "'",      # LEFT SINGLE QUOTATION MARK
> +    "’": "'",      # RIGHT SINGLE QUOTATION MARK
> +    "“": '"',      # LEFT DOUBLE QUOTATION MARK
> +    "”": '"',      # RIGHT DOUBLE QUOTATION MARK
> +    "′": "'",      # PRIME
> +    "″": '"',      # DOUBLE PRIME
> +    "–": "-",      # EN DASH
> +    "—": "--",     # EM DASH
> +    "‐": "-",      # HYPHEN
> +    "‑": "-",      # NON-BREAKING HYPHEN
> +    "−": "-",      # MINUS SIGN
> +    "…": "...",    # HORIZONTAL ELLIPSIS
> +    " ": " ",      # NO-BREAK SPACE

Actually, since the `tree` command uses no-break spaces, I think it's OK to keep
them otherwise we'd have to convert each tree output, which also might be
confusing.

> +    " ": " ",      # NARROW NO-BREAK SPACE
> +    "​": "",       # ZERO WIDTH SPACE
> +    "": "",       # ZERO WIDTH NO-BREAK SPACE / BOM
> +    "­": "",       # SOFT HYPHEN
> +}
> +
> +
> +def main():
> +
> +    args = parse_arguments()
> +    exit_code = 0
> +
> +    for rst_path in sorted(Path(args.docs_dir).rglob("*.rst")):
> +        rel = rst_path.relative_to(args.docs_dir)
> +
> +        with open(rst_path, "r", encoding="utf-8") as f:
> +            for lineno, line in enumerate(f, start=1):
> +                for col, char in enumerate(line, start=1):
> +                    if char in confusables:
> +                        replacement = confusables[char]
> +                        hint = (f"'{replacement}'" if replacement
> +                                else "(remove)")
> +                        print(f"WARNING: {rel}:{lineno}:{col}: non-ASCII "
> +                              f"character U+{ord(char):04X} should be "
> +                              f"replaced with {hint}")
> +                        exit_code = 1
> +
> +    sys.exit(exit_code)
> +
> +
> +if __name__ == "__main__":
> +    main()


Antonin