[PATCH v5 0/2] documentation: remove non-ASCII confusables, add a check

[PATCH v5 0/2] documentation: remove non-ASCII confusables, add a check

[niko.mauno]

From: Niko Mauno <niko.mauno@vaisala.com>

The three whitespace/operator fixes from the earlier revisions of this
series have been merged. The two remaining patches are rebased
on the current master tip: they remove the non-ASCII "confusable"
characters from the documentation sources and add a check to keep them
out.

  * documentation: replace non-ASCII confusables (curly quotes, en/em
    dashes, zero-width spaces, and no-break spaces outside tree output)
    with ASCII; e.g. a curly quote copied into a recipe causes a
    bitbake ParseError. Legitimate non-ASCII (box-drawing in directory
    trees, the no-break spaces the tree command emits there, accented
    contributor names, CJK) is left untouched.
  * tools: add the check-confusables script and wire it up as a
    pre-commit hook and in the Makefile "checks" target, in the same
    fashion as check-glossaries, to guard against regressions.

Changes since v4:
  * Rebased on current master; the three whitespace/operator fixes are
    dropped from the series, as they have been merged.
  * Five files that the confusables patch used to touch are no longer
    part of it, as master has since gained equivalent fixes for them
    (contributor-guide/identify-component.rst, dev-manual/layers.rst,
    ref-manual/faq.rst, ref-manual/qa-checks.rst and
    ref-manual/variables.rst).
  * Re-ran the check across the whole tree on the new base; the
    documentation added to master in the meantime is clean.
  * Rewrapped a ragged paragraph in the first commit message; no
    content changes to either patch otherwise.

Changes since v3:
  * Add a copyright and license (MIT) header to the check-confusables
    script (per review feedback).
  * Spell the confusable characters in the map, and the related
    constants, as \u escape sequences rather than literal glyphs, so
    the source itself is unambiguous ASCII (per review feedback).
  * Explain in the commit message why the script carries a short
    curated table of its own instead of depending on a general-purpose
    confusables package: such a package targets the full Unicode
    homoglyph table and would flag the accented names, CJK and
    box-drawing characters we deliberately keep, so an allow-list and
    replacement policy of our own would be needed regardless.

Changes since v2:
  * Keep the no-break spaces that the tree command emits in directory
    listings, and make the check tolerate them on lines containing
    box-drawing characters (per review feedback). No-break spaces in
    prose and command examples are still converted.
  * Make the pre-commit hook pass filenames and check only the changed
    files (per review feedback); the Makefile "checks" target still
    scans the whole tree.
  * Stop treating the horizontal ellipsis as a confusable; the
    truncated git hashes in the release notes are left as-is, as it is
    a cosmetic character rather than a tooling hazard.

Changes since v1:
  * Squashed the two confusable cleanups (recipe-style-guide and
    identify-component) into a single documentation-wide patch and
    extended it to all remaining occurrences.
  * Added the check-confusables tool, pre-commit hook and Makefile
    integration.

Niko Mauno (2):
  documentation: Replace non-ASCII confusable characters with ASCII
  tools: Add check-confusables pre-commit hook

 .pre-commit-config.yaml                       |   5 +
 documentation/Makefile                        |   1 +
 documentation/contributor-guide/index.rst     |   4 +-
 .../contributor-guide/recipe-style-guide.rst  |   8 +-
 .../contributor-guide/submit-changes.rst      |   4 +-
 documentation/dev-manual/build-quality.rst    |   2 +-
 documentation/dev-manual/debugging.rst        |  14 +-
 documentation/dev-manual/new-recipe.rst       |   4 +-
 documentation/kernel-dev/common.rst           |  10 +-
 .../migration-guides/release-notes-4.1.rst    |   2 +-
 documentation/ref-manual/classes.rst          |   4 +-
 .../ref-manual/devtool-reference.rst          |   4 +-
 .../yocto-project-supported-features.rst      |  10 +-
 .../test-manual/yocto-project-compatible.rst  |   2 +-
 documentation/tools/check-confusables         | 123 ++++++++++++++++++
 .../transitioning-to-a-custom-environment.rst |   2 +-
 documentation/what-i-wish-id-known.rst        |  14 +-
 17 files changed, 171 insertions(+), 42 deletions(-)
 create mode 100755 documentation/tools/check-confusables


base-commit: 41dae3c3da3ada1745fc60228ff6269c64ee2361
-- 
2.47.3

[PATCH v5 1/2] documentation: Replace non-ASCII confusable characters with ASCII

[niko.mauno]

From: Niko Mauno <niko.mauno@vaisala.com>

Several documentation sources contain non-ASCII "confusable" characters
such as curly quotes, en and em dashes, and non-breaking or zero-width
spaces. These look almost identical to their plain ASCII counterparts
but break tooling and mislead readers who copy text out of the manuals.

For example, the curly double quotes present in the recipe style guide
example caused parse errors when copied into an actual recipe:

  ERROR: ParseError at .../foobar.bb:158: unparsed line: 'RDEPENDS:${PN} = "foo"'

and dash substitutions silently corrupt command-line options, turning
"--help" and "--dump-signatures" into unusable non-ASCII look-alikes.

Replace all such occurrences across the documentation with their
regular ASCII equivalents. Legitimate non-ASCII content is left
untouched: the box-drawing characters and no-break spaces that the
tree command emits in directory listings, accented letters in
contributor names, and CJK characters.

These occurrences were detected with the
documentation/tools/check-confusables script introduced in the
following commit.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Niko Mauno <niko.mauno@vaisala.com>
---
 documentation/contributor-guide/index.rst          |  4 ++--
 .../contributor-guide/recipe-style-guide.rst       |  8 ++++----
 documentation/contributor-guide/submit-changes.rst |  4 ++--
 documentation/dev-manual/build-quality.rst         |  2 +-
 documentation/dev-manual/debugging.rst             | 14 +++++++-------
 documentation/dev-manual/new-recipe.rst            |  4 ++--
 documentation/kernel-dev/common.rst                | 10 +++++-----
 .../migration-guides/release-notes-4.1.rst         |  2 +-
 documentation/ref-manual/classes.rst               |  4 ++--
 documentation/ref-manual/devtool-reference.rst     |  4 ++--
 .../yocto-project-supported-features.rst           | 10 +++++-----
 .../test-manual/yocto-project-compatible.rst       |  2 +-
 .../transitioning-to-a-custom-environment.rst      |  2 +-
 documentation/what-i-wish-id-known.rst             | 14 +++++++-------
 14 files changed, 42 insertions(+), 42 deletions(-)

diff --git a/documentation/contributor-guide/index.rst b/documentation/contributor-guide/index.rst
index a83216945..b374cc1c4 100644
--- a/documentation/contributor-guide/index.rst
+++ b/documentation/contributor-guide/index.rst
@@ -7,8 +7,8 @@ Yocto Project and OpenEmbedded Contributor Guide
 The Yocto Project and OpenEmbedded are open-source, community-based projects so
 contributions are very welcome, it is how the code evolves and everyone can
 effect change. Contributions take different forms, if you have a fix for an
-issue you’ve run into, a patch is the most appropriate way to contribute it.
-If you run into an issue but don’t have a solution, opening a defect in
+issue you've run into, a patch is the most appropriate way to contribute it.
+If you run into an issue but don't have a solution, opening a defect in
 :yocto_bugs:`Bugzilla <>` or asking questions on the mailing lists might be
 more appropriate. This guide intends to point you in the right direction to
 this.
diff --git a/documentation/contributor-guide/recipe-style-guide.rst b/documentation/contributor-guide/recipe-style-guide.rst
index f81ffb2ab..84c6bb14e 100644
--- a/documentation/contributor-guide/recipe-style-guide.rst
+++ b/documentation/contributor-guide/recipe-style-guide.rst
@@ -192,11 +192,11 @@ Custom tasks should be sorted similarly.
 
 Package specific variables are typically grouped together, e.g.::
 
-    RDEPENDS:${PN} = “foo”
-    RDEPENDS:${PN}-libs = “bar”
+    RDEPENDS:${PN} = "foo"
+    RDEPENDS:${PN}-libs = "bar"
 
-    RRECOMMENDS:${PN} = “one”
-    RRECOMMENDS:${PN}-libs = “two”
+    RRECOMMENDS:${PN} = "one"
+    RRECOMMENDS:${PN}-libs = "two"
 
 Recipe License Fields
 ---------------------
diff --git a/documentation/contributor-guide/submit-changes.rst b/documentation/contributor-guide/submit-changes.rst
index ae7a097ad..48c8af2fe 100644
--- a/documentation/contributor-guide/submit-changes.rst
+++ b/documentation/contributor-guide/submit-changes.rst
@@ -17,7 +17,7 @@ Both Yocto Project and OpenEmbedded have many key components that are
 maintained by patches being submitted on mailing lists. We appreciate this
 approach does look a little old fashioned when other workflows are available
 through web technology such as GitHub, GitLab and others. Since we are often
-asked this question, we’ve decided to document the reasons for using mailing
+asked this question, we've decided to document the reasons for using mailing
 lists.
 
 One significant factor is that we value peer review. When a change is proposed
@@ -40,7 +40,7 @@ of key people from the project, which would again be to its detriment.
 
 The projects are acutely aware that potential new contributors find the
 mailing list approach off-putting and would prefer a web-based GUI.
-Since we don’t believe that can work for us, the project is aiming to ensure
+Since we don't believe that can work for us, the project is aiming to ensure
 `patchwork <https://patchwork.yoctoproject.org/>`__ is available to help track
 patch status and also looking at how tooling can provide more feedback to users
 about patch status. We are looking at improving tools such as ``patchtest`` to
diff --git a/documentation/dev-manual/build-quality.rst b/documentation/dev-manual/build-quality.rst
index c2e46b69c..e9b066c44 100644
--- a/documentation/dev-manual/build-quality.rst
+++ b/documentation/dev-manual/build-quality.rst
@@ -358,7 +358,7 @@ You can examine build history output from the command line or from a web
 interface.
 
 To see any changes that have occurred (assuming you have
-:term:`BUILDHISTORY_COMMIT` = "1"),
+:term:`BUILDHISTORY_COMMIT` = "1"),
 you can simply use any Git command that allows you to view the history
 of a repository. Here is one method::
 
diff --git a/documentation/dev-manual/debugging.rst b/documentation/dev-manual/debugging.rst
index 21fb10537..3afc17ad4 100644
--- a/documentation/dev-manual/debugging.rst
+++ b/documentation/dev-manual/debugging.rst
@@ -152,7 +152,7 @@ Shell functions and tasks can also be inspected with the same mechanism::
 
 For Python functions and tasks, ``bitbake -e recipename`` can be used instead.
 
-Moreover, the output of the ``bitbake -e`` and ``bitbake -e`` recipe commands
+Moreover, the output of the ``bitbake -e`` and ``bitbake -e`` recipe commands
 includes the following information:
 
 -  The output starts with a tree listing all configuration files and
@@ -187,7 +187,7 @@ Here are a few of the available ``oe-pkgdata-util`` subcommands.
    that have been built, optionally limiting the match to packages that
    match pattern.
 
--  ``oe-pkgdata-util list-pkg-files package ...``: Lists the
+-  ``oe-pkgdata-util list-pkg-files package ...``: Lists the
    files and directories contained in the given packages.
 
    .. note::
@@ -205,7 +205,7 @@ Here are a few of the available ``oe-pkgdata-util`` subcommands.
       directory, make sure that :ref:`ref-classes-rm-work` is not
       enabled when you build the recipe.
 
--  ``oe-pkgdata-util find-path path ...``: Lists the names of
+-  ``oe-pkgdata-util find-path path ...``: Lists the names of
    the packages that contain the given paths. For example, the following
    tells us that ``/usr/share/man/man1/make.1`` is contained in the
    ``make-doc`` package::
@@ -213,7 +213,7 @@ Here are a few of the available ``oe-pkgdata-util`` subcommands.
       $ oe-pkgdata-util find-path /usr/share/man/man1/make.1
       make-doc: /usr/share/man/man1/make.1
 
--  ``oe-pkgdata-util lookup-recipe package ...``: Lists the name
+-  ``oe-pkgdata-util lookup-recipe package ...``: Lists the name
    of the recipes that produce the given packages.
 
 For more information on the ``oe-pkgdata-util`` command, use the help
@@ -362,7 +362,7 @@ You can also use BitBake to dump out the signature construction
 information without executing tasks by using either of the following
 BitBake command-line options::
 
-   ‐‐dump-signatures=SIGNATURE_HANDLER
+   --dump-signatures=SIGNATURE_HANDLER
    -S SIGNATURE_HANDLER
 
 
@@ -1072,7 +1072,7 @@ debugger.
       the actual image (e.g. ``gdb-cross-i586``). The suggestion is usually the
       actual name you want to use.
 
-#. *Set up the* ``debugfs``\ *:*
+#. *Set up the* ``debugfs``\ *:*
 
    Run the following commands to set up the ``debugfs``::
 
@@ -1106,7 +1106,7 @@ debugger.
 
    .. code-block:: shell
 
-      root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help
+      root@qemux86:~# gdbserver localhost:1234 /bin/gzip --help
 
    For
    additional gdbserver options, see the `GDB Server
diff --git a/documentation/dev-manual/new-recipe.rst b/documentation/dev-manual/new-recipe.rst
index 6888bb46a..d1c863117 100644
--- a/documentation/dev-manual/new-recipe.rst
+++ b/documentation/dev-manual/new-recipe.rst
@@ -99,14 +99,14 @@ command::
                        otherwise $EDITOR, otherwise vi.
    Use recipetool <subcommand> --help to get help on a specific command
 
-Running ``recipetool create -o OUTFILE`` creates the base recipe and
+Running ``recipetool create -o OUTFILE`` creates the base recipe and
 locates it properly in the layer that contains your source files.
 Here are some syntax examples:
 
  - Use this syntax to generate a recipe based on source. Once generated,
    the recipe resides in the existing source code layer::
 
-      recipetool create -o OUTFILE source
+      recipetool create -o OUTFILE source
 
  - Use this syntax to generate a recipe using code that
    you extract from source. The extracted code is placed in its own layer
diff --git a/documentation/kernel-dev/common.rst b/documentation/kernel-dev/common.rst
index 51e819699..948718433 100644
--- a/documentation/kernel-dev/common.rst
+++ b/documentation/kernel-dev/common.rst
@@ -577,7 +577,7 @@ applies the new configuration before building the kernel.
 For a detailed example showing how to configure the kernel, see the
 ":ref:`kernel-dev/common:configuring the kernel`" section.
 
-Using an "In-Tree"  ``defconfig`` File
+Using an "In-Tree"  ``defconfig`` File
 --------------------------------------
 
 It might be desirable to have kernel configuration fragment support
@@ -958,7 +958,7 @@ possible.
 For more information on kernel configuration, see the
 ":ref:`kernel-dev/common:changing the configuration`" section.
 
-Using  ``menuconfig``
+Using  ``menuconfig``
 ---------------------
 
 The easiest way to define kernel configurations is to set them through
@@ -1064,7 +1064,7 @@ kernel layer.
    Be sure to make a copy of the ``.config`` file and do not just rename it.
    The build system needs an existing ``.config`` file from which to work.
 
-Creating a  ``defconfig`` File
+Creating a  ``defconfig`` File
 ------------------------------
 
 A ``defconfig`` file in the context of the Yocto Project is often a
@@ -1693,12 +1693,12 @@ Showing a Particular Feature or Branch Change
 ---------------------------------------------
 
 Tags in the Yocto Project kernel tree divide changes for significant
-features or branches. The ``git show`` tag command shows changes based
+features or branches. The ``git show`` tag command shows changes based
 on a tag. Here is an example that shows ``systemtap`` changes::
 
    $ git show systemtap
 
-You can use the ``git branch --contains`` tag command to
+You can use the ``git branch --contains`` tag command to
 show the branches that contain a particular feature. This command shows
 the branches that contain the ``systemtap`` feature::
 
diff --git a/documentation/migration-guides/release-notes-4.1.rst b/documentation/migration-guides/release-notes-4.1.rst
index 81d541fac..4a3fb0d2c 100644
--- a/documentation/migration-guides/release-notes-4.1.rst
+++ b/documentation/migration-guides/release-notes-4.1.rst
@@ -191,7 +191,7 @@ Known Issues in 4.1
 ~~~~~~~~~~~~~~~~~~~
 
 - The change to :ref:`migration-4.1-complementary-deps` means that images
-  built with the ``ptest-pkgs`` :term:`IMAGE_FEATURES` don’t automatically
+  built with the ``ptest-pkgs`` :term:`IMAGE_FEATURES` don't automatically
   install ``ptest-runner``, as that package is a recommendation of the
   individual ``-ptest`` packages.  This will be resolved in the next point
   release, and can be worked around by explicitly installing ``ptest-runner``
diff --git a/documentation/ref-manual/classes.rst b/documentation/ref-manual/classes.rst
index 98dff1bac..f3d1aa64e 100644
--- a/documentation/ref-manual/classes.rst
+++ b/documentation/ref-manual/classes.rst
@@ -152,7 +152,7 @@ barebox, enabling tasks such as :ref:`ref-tasks-menuconfig` and
 
 The generated barebox binaries are deployed to
 :term:`DEPLOY_DIR_IMAGE` as well as installed to ``BAREBOX_INSTALL_PATH``
-(``/boot`` by default) making them part of the recipe’s base package.
+(``/boot`` by default) making them part of the recipe's base package.
 This setup supports both using the barebox binaries as independent artifacts
 and installing them into a rootfs.
 :term:`BAREBOX_BINARY` can be used to select a distinct binary to deploy and
@@ -3687,7 +3687,7 @@ for details.
 The :ref:`ref-classes-upstream-stable-release-point` class automatically
 generates the :term:`UPSTREAM_STABLE_RELEASE_REGEX` variable for recipes
 whose version uses a dot-separated scheme. This enables stable point release
-upgrades — version upgrades constrained to the same stable series (e.g.
+upgrades -- version upgrades constrained to the same stable series (e.g.
 ``1.4.2`` to ``1.4.3`` but not to ``1.5.0``).
 
 The class uses the :term:`STABLE_VERSION_PARTS` variable (defaults to ``"2"``)
diff --git a/documentation/ref-manual/devtool-reference.rst b/documentation/ref-manual/devtool-reference.rst
index 863825b06..6b21d302f 100644
--- a/documentation/ref-manual/devtool-reference.rst
+++ b/documentation/ref-manual/devtool-reference.rst
@@ -517,7 +517,7 @@ Deploying Your Software on the Target Machine
 Use the ``devtool deploy-target`` command to deploy the recipe's build
 output to the live target machine::
 
-   $ devtool deploy-target recipe target
+   $ devtool deploy-target recipe target
 
 The target is the address of the target machine, which must be running
 an SSH server (i.e. ``user@hostname[:destdir]``).
@@ -588,7 +588,7 @@ command to work, you must have previously used the
 ":ref:`devtool deploy-target <ref-manual/devtool-reference:deploying your software on the target machine>`"
 command::
 
-   $ devtool undeploy-target recipe target
+   $ devtool undeploy-target recipe target
 
 The target is the
 address of the target machine, which must be running an SSH server (i.e.
diff --git a/documentation/ref-manual/yocto-project-supported-features.rst b/documentation/ref-manual/yocto-project-supported-features.rst
index d7229f1c2..e209e54c4 100644
--- a/documentation/ref-manual/yocto-project-supported-features.rst
+++ b/documentation/ref-manual/yocto-project-supported-features.rst
@@ -164,7 +164,7 @@ to support, however there isn't wide enough interest and support to justify
 testing all incoming changes on it. There are however project member
 organisations and maintainers willing to run tests and review fixes.
 
-This category may be applicable as support/usage in an area develops and grows,
+This category may be applicable as support/usage in an area develops and grows,
 or as support/usage fades but we continue to have tests. It can also apply where
 resourcing isn't available for full primary support but there is
 member/maintainer support for running tests.
@@ -173,21 +173,21 @@ We therefore have the following criteria and policies for such items:
 
 -  It can be clearly isolated and defined by specific configuration.
 
--  There is a clear documented group of maintainers agreeing to maintain it.
+-  There is a clear documented group of maintainers agreeing to maintain it.
 
 -  Those maintainers are active and responsive.
 
 -  It is being actively and publicly tested (potentially using
-   the :ref:`Autobuilder <test-manual/intro:Yocto Project Autobuilder Overview>`
+   the :ref:`Autobuilder <test-manual/intro:Yocto Project Autobuilder Overview>`
    by agreement, or otherwise).
 
--  Testing would not be part of standard incoming change testing and regressions
+-  Testing would not be part of standard incoming change testing and regressions
    would not block incoming patches.
 
 -  The :yocto_wiki:`SWAT </Yocto_Build_Failure_Swat_Team>` team would not handle
    any test builds on the Autobuilder.
 
--  Test results can be submitted as part of the release process if desired.
+-  Test results can be submitted as part of the release process if desired.
 
 The Yocto Project :yocto_wiki:`Technical Steering Committee (TSC) </TSC>` makes
 decisions on features in this status and Autobuilder testing. Such support would
diff --git a/documentation/test-manual/yocto-project-compatible.rst b/documentation/test-manual/yocto-project-compatible.rst
index 7a2e00a99..b4da178ec 100644
--- a/documentation/test-manual/yocto-project-compatible.rst
+++ b/documentation/test-manual/yocto-project-compatible.rst
@@ -115,7 +115,7 @@ Here are key best practices the program tries to encourage:
    user changes a configuration setting to activate the layer, by selecting
    a :term:`MACHINE`, a :term:`DISTRO` or a :term:`DISTRO_FEATURES` setting.
 
--  Layers should be documenting where they don’t support normal "core"
+-  Layers should be documenting where they don't support normal "core"
    functionality such as where debug symbols are disabled or missing, where
    development headers and on-target library usage may not work or where
    functionality like the SDK/eSDK would not be expected to work.
diff --git a/documentation/transitioning-to-a-custom-environment.rst b/documentation/transitioning-to-a-custom-environment.rst
index 0fd7c950e..17938ebda 100644
--- a/documentation/transitioning-to-a-custom-environment.rst
+++ b/documentation/transitioning-to-a-custom-environment.rst
@@ -33,7 +33,7 @@ Transitioning to a custom environment for systems development
    Layer Index <>` to find and acquire the best BSP for your
    target board. The Yocto Project layer index BSPs are regularly validated. The
    best place to get your first BSP is from your silicon manufacturer or board
-   vendor – they can point you to their most qualified efforts. In general, for
+   vendor - they can point you to their most qualified efforts. In general, for
    Intel silicon use ``meta-intel``, for Texas Instruments use ``meta-ti``, and so
    forth. Choose a BSP that has been tested with the same Yocto Project release
    that you've downloaded. Be aware that some BSPs may not be immediately
diff --git a/documentation/what-i-wish-id-known.rst b/documentation/what-i-wish-id-known.rst
index e2ed9aa32..964b51b84 100644
--- a/documentation/what-i-wish-id-known.rst
+++ b/documentation/what-i-wish-id-known.rst
@@ -86,13 +86,13 @@ contact us with other suggestions.
    pinpoint where trouble is occurring and how the build is breaking. The
    workflow breaks down into the following steps:
 
-   #. Fetch – get the source code
-   #. Extract – unpack the sources
-   #. Patch – apply patches for bug fixes and new capability
-   #. Configure – set up your environment specifications
-   #. Build – compile and link
-   #. Install – copy files to target directories
-   #. Package – bundle files for installation
+   #. Fetch - get the source code
+   #. Extract - unpack the sources
+   #. Patch - apply patches for bug fixes and new capability
+   #. Configure - set up your environment specifications
+   #. Build - compile and link
+   #. Install - copy files to target directories
+   #. Package - bundle files for installation
 
    During "fetch", there may be an inability to find code. During "extract",
    there is likely an invalid zip or something similar. In other words, the
-- 
2.47.3

[PATCH v5 2/2] tools: Add check-confusables pre-commit hook

[niko.mauno]

From: Niko Mauno <niko.mauno@vaisala.com>

Add a check-confusables script, in the same fashion as
check-glossaries, that scans the documentation .rst sources for
non-ASCII "confusable" characters (curly quotes, en/em dashes,
non-breaking and zero-width spaces, etc.) and reports each occurrence
with its location and suggested ASCII replacement, exiting non-zero if
any are found. This guards against the class of breakage fixed in the
preceding commit, e.g. curly quotes causing recipe ParseErrors.

Legitimate non-ASCII such as box-drawing characters used in directory
trees, accented letters in contributor names and CJK characters are
intentionally left untouched. No-break spaces are likewise tolerated
on lines containing box-drawing characters, since the tree command
emits them as indentation in directory listings.

The set of flagged characters is intentionally small and curated
rather than exhaustive. A general-purpose dependency such as the
confusables PyPI package targets Unicode homoglyph detection against
the full confusables table; it would also flag the accented names, CJK
and box-drawing characters we deliberately keep, so we would still need
our own allow-list and replacement policy on top of it. A short,
dependency-free table kept in-tree matches check-glossaries and is
trivial to extend if a new problematic character shows up.

Wire it up both as a local pre-commit hook, which checks the changed
files, and in the Makefile "checks" target, which scans the whole
tree, alongside check-glossaries.

Suggested-by: Quentin Schulz <quentin.schulz@cherry.de>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Niko Mauno <niko.mauno@vaisala.com>
---
 .pre-commit-config.yaml               |   5 ++
 documentation/Makefile                |   1 +
 documentation/tools/check-confusables | 123 ++++++++++++++++++++++++++
 3 files changed, 129 insertions(+)
 create mode 100755 documentation/tools/check-confusables

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index f2b73a481..876546f9a 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -6,3 +6,8 @@ repos:
         entry: ./documentation/tools/check-glossaries
         language: python
         pass_filenames: false
+      - id: check-confusables
+        name: Check for non-ASCII confusable characters
+        entry: ./documentation/tools/check-confusables
+        language: python
+        files: \.rst$
diff --git a/documentation/Makefile b/documentation/Makefile
index fe0574537..87a6f8a8b 100644
--- a/documentation/Makefile
+++ b/documentation/Makefile
@@ -37,6 +37,7 @@ clean:
 
 checks:
 	$(SOURCEDIR)/tools/check-glossaries --docs-dir $(SOURCEDIR)
+	$(SOURCEDIR)/tools/check-confusables --docs-dir $(SOURCEDIR)
 
 stylecheck:
 	vale sync
diff --git a/documentation/tools/check-confusables b/documentation/tools/check-confusables
new file mode 100755
index 000000000..359386e7c
--- /dev/null
+++ b/documentation/tools/check-confusables
@@ -0,0 +1,123 @@
+#!/usr/bin/env python3
+#
+# Check documentation sources for non-ASCII typographic characters that
+# should be plain ASCII.
+#
+# Copyright (c) Vaisala Oyj. All rights reserved.
+#
+# SPDX-License-Identifier: MIT
+#
+
+import argparse
+import sys
+
+from pathlib import Path
+
+
+def parse_arguments() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description="Check documentation sources for non-ASCII typographic "
+                    "characters that should be plain ASCII")
+
+    parser.add_argument("files",
+                        nargs="*",
+                        type=Path,
+                        help="Specific files to check; if none are given, "
+                             "all *.rst files under --docs-dir are scanned")
+
+    parser.add_argument("-d", "--docs-dir",
+                        type=Path,
+                        default=Path(__file__).resolve().parent.parent,
+                        help="Path to documentation/ directory in yocto-docs")
+
+    return parser.parse_args()
+
+
+# Map of "confusable" characters that are frequently introduced by editors,
+# word processors or copy-pasting, to their plain ASCII replacement. These
+# look almost identical to regular ASCII but break tooling, e.g. a curly
+# quote in a recipe example causes:
+#
+#   ERROR: ParseError ...: unparsed line: 'RDEPENDS:${PN} = “foo”'
+#
+# Only these characters are flagged; legitimate non-ASCII such as box-drawing
+# characters used in directory trees, accented letters in contributor names
+# and CJK characters are intentionally left alone.
+confusables = {
+    "\u2018": "'",      # LEFT SINGLE QUOTATION MARK
+    "\u2019": "'",      # RIGHT SINGLE QUOTATION MARK
+    "\u201c": '"',      # LEFT DOUBLE QUOTATION MARK
+    "\u201d": '"',      # RIGHT DOUBLE QUOTATION MARK
+    "\u2032": "'",      # PRIME
+    "\u2033": '"',      # DOUBLE PRIME
+    "\u2013": "-",      # EN DASH
+    "\u2014": "--",     # EM DASH
+    "\u2010": "-",      # HYPHEN
+    "\u2011": "-",      # NON-BREAKING HYPHEN
+    "\u2212": "-",      # MINUS SIGN
+    "\u00a0": " ",      # NO-BREAK SPACE
+    "\u202f": " ",      # NARROW NO-BREAK SPACE
+    "\u200b": "",       # ZERO WIDTH SPACE
+    "\ufeff": "",       # ZERO WIDTH NO-BREAK SPACE / BOM
+    "\u00ad": "",       # SOFT HYPHEN
+}
+
+NO_BREAK_SPACE = "\u00a0"
+
+
+def is_box_drawing(char: str) -> bool:
+    # Box Drawing Unicode block (U+2500..U+257F), used for the directory
+    # trees rendered in the manuals.
+    return "\u2500" <= char <= "\u257f"
+
+
+def check_file(path: Path, display: str) -> bool:
+    found = False
+
+    with open(path, "r", encoding="utf-8") as f:
+        for lineno, line in enumerate(f, start=1):
+            # The tree(1) command indents its directory listings with
+            # no-break spaces; such listings are embedded verbatim in the
+            # manuals. A no-break space is therefore tolerated on any line
+            # that also contains box-drawing characters (i.e. inside a
+            # rendered directory tree), but still flagged elsewhere.
+            in_tree = any(is_box_drawing(c) for c in line)
+            for col, char in enumerate(line, start=1):
+                if char not in confusables:
+                    continue
+                if char == NO_BREAK_SPACE and in_tree:
+                    continue
+                replacement = confusables[char]
+                hint = f"'{replacement}'" if replacement else "(remove)"
+                print(f"WARNING: {display}:{lineno}:{col}: non-ASCII "
+                      f"character U+{ord(char):04X} should be "
+                      f"replaced with {hint}")
+                found = True
+
+    return found
+
+
+def main():
+
+    args = parse_arguments()
+
+    # When invoked with explicit files (e.g. by pre-commit, which passes the
+    # staged filenames) only those are checked; otherwise the whole tree of
+    # *.rst files under --docs-dir is scanned (e.g. by "make checks").
+    if args.files:
+        targets = [(path, str(path)) for path in args.files]
+    else:
+        docs_dir = Path(args.docs_dir)
+        targets = [(path, str(path.relative_to(docs_dir)))
+                   for path in sorted(docs_dir.rglob("*.rst"))]
+
+    exit_code = 0
+    for path, display in targets:
+        if check_file(path, display):
+            exit_code = 1
+
+    sys.exit(exit_code)
+
+
+if __name__ == "__main__":
+    main()
-- 
2.47.3

Re: [docs] [PATCH v5 2/2] tools: Add check-confusables pre-commit hook

[Antonin Godard]

Hi,

On Mon Aug 3, 2026 at 5:48 AM CEST, Niko Mauno via lists.yoctoproject.org wrote:
> From: Niko Mauno <niko.mauno@vaisala.com>
>
> Add a check-confusables script, in the same fashion as
> check-glossaries, that scans the documentation .rst sources for
> non-ASCII "confusable" characters (curly quotes, en/em dashes,
> non-breaking and zero-width spaces, etc.) and reports each occurrence
> with its location and suggested ASCII replacement, exiting non-zero if
> any are found. This guards against the class of breakage fixed in the
> preceding commit, e.g. curly quotes causing recipe ParseErrors.
>
> Legitimate non-ASCII such as box-drawing characters used in directory
> trees, accented letters in contributor names and CJK characters are
> intentionally left untouched. No-break spaces are likewise tolerated
> on lines containing box-drawing characters, since the tree command
> emits them as indentation in directory listings.
>
> The set of flagged characters is intentionally small and curated
> rather than exhaustive. A general-purpose dependency such as the
> confusables PyPI package targets Unicode homoglyph detection against
> the full confusables table; it would also flag the accented names, CJK
> and box-drawing characters we deliberately keep, so we would still need
> our own allow-list and replacement policy on top of it. A short,
> dependency-free table kept in-tree matches check-glossaries and is
> trivial to extend if a new problematic character shows up.
>
> Wire it up both as a local pre-commit hook, which checks the changed
> files, and in the Makefile "checks" target, which scans the whole
> tree, alongside check-glossaries.
>
> Suggested-by: Quentin Schulz <quentin.schulz@cherry.de>
> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
> Signed-off-by: Niko Mauno <niko.mauno@vaisala.com>
> ---
>  .pre-commit-config.yaml               |   5 ++
>  documentation/Makefile                |   1 +
>  documentation/tools/check-confusables | 123 ++++++++++++++++++++++++++
>  3 files changed, 129 insertions(+)
>  create mode 100755 documentation/tools/check-confusables
>
> diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
> index f2b73a481..876546f9a 100644
> --- a/.pre-commit-config.yaml
> +++ b/.pre-commit-config.yaml
> @@ -6,3 +6,8 @@ repos:
>          entry: ./documentation/tools/check-glossaries
>          language: python
>          pass_filenames: false
> +      - id: check-confusables
> +        name: Check for non-ASCII confusable characters
> +        entry: ./documentation/tools/check-confusables
> +        language: python
> +        files: \.rst$
> diff --git a/documentation/Makefile b/documentation/Makefile
> index fe0574537..87a6f8a8b 100644
> --- a/documentation/Makefile
> +++ b/documentation/Makefile
> @@ -37,6 +37,7 @@ clean:
>  
>  checks:
>  	$(SOURCEDIR)/tools/check-glossaries --docs-dir $(SOURCEDIR)
> +	$(SOURCEDIR)/tools/check-confusables --docs-dir $(SOURCEDIR)
>  
>  stylecheck:
>  	vale sync
> diff --git a/documentation/tools/check-confusables b/documentation/tools/check-confusables
> new file mode 100755
> index 000000000..359386e7c
> --- /dev/null
> +++ b/documentation/tools/check-confusables
> @@ -0,0 +1,123 @@
> +#!/usr/bin/env python3
> +#
> +# Check documentation sources for non-ASCII typographic characters that
> +# should be plain ASCII.
> +#
> +# Copyright (c) Vaisala Oyj. All rights reserved.
> +#
> +# SPDX-License-Identifier: MIT
> +#
> +
> +import argparse
> +import sys
> +
> +from pathlib import Path
> +
> +
> +def parse_arguments() -> argparse.Namespace:
> +    parser = argparse.ArgumentParser(
> +        description="Check documentation sources for non-ASCII typographic "
> +                    "characters that should be plain ASCII")
> +
> +    parser.add_argument("files",
> +                        nargs="*",
> +                        type=Path,
> +                        help="Specific files to check; if none are given, "
> +                             "all *.rst files under --docs-dir are scanned")
> +
> +    parser.add_argument("-d", "--docs-dir",
> +                        type=Path,
> +                        default=Path(__file__).resolve().parent.parent,
> +                        help="Path to documentation/ directory in yocto-docs")
> +
> +    return parser.parse_args()
> +
> +
> +# Map of "confusable" characters that are frequently introduced by editors,
> +# word processors or copy-pasting, to their plain ASCII replacement. These
> +# look almost identical to regular ASCII but break tooling, e.g. a curly
> +# quote in a recipe example causes:
> +#
> +#   ERROR: ParseError ...: unparsed line: 'RDEPENDS:${PN} = “foo”'
> +#
> +# Only these characters are flagged; legitimate non-ASCII such as box-drawing
> +# characters used in directory trees, accented letters in contributor names
> +# and CJK characters are intentionally left alone.
> +confusables = {

Nit, but this is defined in lowercase, but NO_BREAK_SPACE is uppercase, while
both are global variables. Maybe make those all uppercase?

> +    "\u2018": "'",      # LEFT SINGLE QUOTATION MARK
> +    "\u2019": "'",      # RIGHT SINGLE QUOTATION MARK
> +    "\u201c": '"',      # LEFT DOUBLE QUOTATION MARK
> +    "\u201d": '"',      # RIGHT DOUBLE QUOTATION MARK
> +    "\u2032": "'",      # PRIME
> +    "\u2033": '"',      # DOUBLE PRIME
> +    "\u2013": "-",      # EN DASH
> +    "\u2014": "--",     # EM DASH
> +    "\u2010": "-",      # HYPHEN
> +    "\u2011": "-",      # NON-BREAKING HYPHEN
> +    "\u2212": "-",      # MINUS SIGN
> +    "\u00a0": " ",      # NO-BREAK SPACE

You could reuse NO_BREAK_SPACE here, if you define it before.

> +    "\u202f": " ",      # NARROW NO-BREAK SPACE
> +    "\u200b": "",       # ZERO WIDTH SPACE
> +    "\ufeff": "",       # ZERO WIDTH NO-BREAK SPACE / BOM
> +    "\u00ad": "",       # SOFT HYPHEN
> +}
> +
> +NO_BREAK_SPACE = "\u00a0"
> +
> +
> +def is_box_drawing(char: str) -> bool:
> +    # Box Drawing Unicode block (U+2500..U+257F), used for the directory
> +    # trees rendered in the manuals.
> +    return "\u2500" <= char <= "\u257f"
> +
> +
> +def check_file(path: Path, display: str) -> bool:
> +    found = False
> +
> +    with open(path, "r", encoding="utf-8") as f:
> +        for lineno, line in enumerate(f, start=1):
> +            # The tree(1) command indents its directory listings with
> +            # no-break spaces; such listings are embedded verbatim in the
> +            # manuals. A no-break space is therefore tolerated on any line
> +            # that also contains box-drawing characters (i.e. inside a
> +            # rendered directory tree), but still flagged elsewhere.
> +            in_tree = any(is_box_drawing(c) for c in line)

Not a blocking change, but at this point you could probably continue if in_tree
is True and move onto the next line. Since we know we're in a box-drawing,
leaving confusables is fine since it's not a command to run?

> +            for col, char in enumerate(line, start=1):
> +                if char not in confusables:
> +                    continue
> +                if char == NO_BREAK_SPACE and in_tree:
> +                    continue
> +                replacement = confusables[char]
> +                hint = f"'{replacement}'" if replacement else "(remove)"
> +                print(f"WARNING: {display}:{lineno}:{col}: non-ASCII "
> +                      f"character U+{ord(char):04X} should be "
> +                      f"replaced with {hint}")
> +                found = True
> +
> +    return found
> +
> +
> +def main():
> +
> +    args = parse_arguments()
> +
> +    # When invoked with explicit files (e.g. by pre-commit, which passes the
> +    # staged filenames) only those are checked; otherwise the whole tree of
> +    # *.rst files under --docs-dir is scanned (e.g. by "make checks").
> +    if args.files:
> +        targets = [(path, str(path)) for path in args.files]

Those are Path objects, I don't think you need to store their string
representation separately to display them? Why not simply passing the Path
object when calling print() directly? It should get converted to strings
directly. And this would make check_file() simpler:

def check_file(path: Path) -> bool:

    ...

    print(f"WARNING: {path}:{lineno}:{col}: non-ASCII "
          f"character U+{ord(char):04X} should be "
          f"replaced with {hint}")

> +    else:
> +        docs_dir = Path(args.docs_dir)
> +        targets = [(path, str(path.relative_to(docs_dir)))
> +                   for path in sorted(docs_dir.rglob("*.rst"))]
> +
> +    exit_code = 0
> +    for path, display in targets:
> +        if check_file(path, display):
> +            exit_code = 1
> +
> +    sys.exit(exit_code)
> +
> +
> +if __name__ == "__main__":
> +    main()

Thanks,
Antonin

Re: [PATCH v5 1/2] documentation: Replace non-ASCII confusable characters with ASCII

[Antonin Godard]

Applied to master-next, thanks!

Antonin

[PATCH v6] tools: Add check-confusables pre-commit hook

[niko.mauno]

From: Niko Mauno <niko.mauno@vaisala.com>

Add a check-confusables script, in the same fashion as
check-glossaries, that scans the documentation .rst sources for
non-ASCII "confusable" characters (curly quotes, en/em dashes,
non-breaking and zero-width spaces, etc.) and reports each occurrence
with its location and suggested ASCII replacement, exiting non-zero if
any are found. This guards against the class of breakage fixed by the
preceding "documentation: Replace non-ASCII confusable characters
with ASCII" commit, e.g. curly quotes causing recipe ParseErrors.

Legitimate non-ASCII such as box-drawing characters used in directory
trees, accented letters in contributor names and CJK characters are
intentionally left untouched. No-break spaces are likewise tolerated
on lines containing box-drawing characters, since the tree command
emits them as indentation in directory listings.

The set of flagged characters is intentionally small and curated
rather than exhaustive. A general-purpose dependency such as the
confusables PyPI package targets Unicode homoglyph detection against
the full confusables table; it would also flag the accented names, CJK
and box-drawing characters we deliberately keep, so we would still need
our own allow-list and replacement policy on top of it. A short,
dependency-free table kept in-tree matches check-glossaries and is
trivial to extend if a new problematic character shows up.

Wire it up both as a local pre-commit hook, which checks the changed
files, and in the Makefile "checks" target, which scans the whole
tree, alongside check-glossaries.

Suggested-by: Quentin Schulz <quentin.schulz@cherry.de>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Niko Mauno <niko.mauno@vaisala.com>
---
This applies on top of master-next, which already carries the preceding
"documentation: Replace non-ASCII confusable characters with ASCII"
patch from v5.

Changes since v5:
  * Uppercase the CONFUSABLES map, so that both module-level globals
    follow the same convention (per review feedback).
  * Define NO_BREAK_SPACE ahead of the map and reuse it as the key for
    U+00A0 (per review feedback).
  * Pass the Path objects to check_file() directly and print them as
    such, dropping the separate display strings (per review feedback).
  * Realign the inline comments in the map, as one key is now a name.
  * Kept flagging the other confusables on box-drawing lines rather
    than skipping such lines wholesale: the no-break space is the only
    character the tree command is known to emit there, so a wider
    exemption would only create a blind spot.
  * Reword the commit message reference to the preceding patch, which
    is no longer part of this series.

 .pre-commit-config.yaml               |   5 ++
 documentation/Makefile                |   1 +
 documentation/tools/check-confusables | 121 ++++++++++++++++++++++++++
 3 files changed, 127 insertions(+)
 create mode 100755 documentation/tools/check-confusables

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index f2b73a481..876546f9a 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -6,3 +6,8 @@ repos:
         entry: ./documentation/tools/check-glossaries
         language: python
         pass_filenames: false
+      - id: check-confusables
+        name: Check for non-ASCII confusable characters
+        entry: ./documentation/tools/check-confusables
+        language: python
+        files: \.rst$
diff --git a/documentation/Makefile b/documentation/Makefile
index fe0574537..87a6f8a8b 100644
--- a/documentation/Makefile
+++ b/documentation/Makefile
@@ -37,6 +37,7 @@ clean:
 
 checks:
 	$(SOURCEDIR)/tools/check-glossaries --docs-dir $(SOURCEDIR)
+	$(SOURCEDIR)/tools/check-confusables --docs-dir $(SOURCEDIR)
 
 stylecheck:
 	vale sync
diff --git a/documentation/tools/check-confusables b/documentation/tools/check-confusables
new file mode 100755
index 000000000..36e8b22da
--- /dev/null
+++ b/documentation/tools/check-confusables
@@ -0,0 +1,121 @@
+#!/usr/bin/env python3
+#
+# Check documentation sources for non-ASCII typographic characters that
+# should be plain ASCII.
+#
+# Copyright (c) Vaisala Oyj. All rights reserved.
+#
+# SPDX-License-Identifier: MIT
+#
+
+import argparse
+import sys
+
+from pathlib import Path
+
+
+def parse_arguments() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description="Check documentation sources for non-ASCII typographic "
+                    "characters that should be plain ASCII")
+
+    parser.add_argument("files",
+                        nargs="*",
+                        type=Path,
+                        help="Specific files to check; if none are given, "
+                             "all *.rst files under --docs-dir are scanned")
+
+    parser.add_argument("-d", "--docs-dir",
+                        type=Path,
+                        default=Path(__file__).resolve().parent.parent,
+                        help="Path to documentation/ directory in yocto-docs")
+
+    return parser.parse_args()
+
+
+NO_BREAK_SPACE = "\u00a0"
+
+# Map of "confusable" characters that are frequently introduced by editors,
+# word processors or copy-pasting, to their plain ASCII replacement. These
+# look almost identical to regular ASCII but break tooling, e.g. a curly
+# quote in a recipe example causes:
+#
+#   ERROR: ParseError ...: unparsed line: 'RDEPENDS:${PN} = “foo”'
+#
+# Only these characters are flagged; legitimate non-ASCII such as box-drawing
+# characters used in directory trees, accented letters in contributor names
+# and CJK characters are intentionally left alone.
+CONFUSABLES = {
+    "\u2018": "'",        # LEFT SINGLE QUOTATION MARK
+    "\u2019": "'",        # RIGHT SINGLE QUOTATION MARK
+    "\u201c": '"',        # LEFT DOUBLE QUOTATION MARK
+    "\u201d": '"',        # RIGHT DOUBLE QUOTATION MARK
+    "\u2032": "'",        # PRIME
+    "\u2033": '"',        # DOUBLE PRIME
+    "\u2013": "-",        # EN DASH
+    "\u2014": "--",       # EM DASH
+    "\u2010": "-",        # HYPHEN
+    "\u2011": "-",        # NON-BREAKING HYPHEN
+    "\u2212": "-",        # MINUS SIGN
+    NO_BREAK_SPACE: " ",  # NO-BREAK SPACE
+    "\u202f": " ",        # NARROW NO-BREAK SPACE
+    "\u200b": "",         # ZERO WIDTH SPACE
+    "\ufeff": "",         # ZERO WIDTH NO-BREAK SPACE / BOM
+    "\u00ad": "",         # SOFT HYPHEN
+}
+
+
+def is_box_drawing(char: str) -> bool:
+    # Box Drawing Unicode block (U+2500..U+257F), used for the directory
+    # trees rendered in the manuals.
+    return "\u2500" <= char <= "\u257f"
+
+
+def check_file(path: Path) -> bool:
+    found = False
+
+    with open(path, "r", encoding="utf-8") as f:
+        for lineno, line in enumerate(f, start=1):
+            # The tree(1) command indents its directory listings with
+            # no-break spaces; such listings are embedded verbatim in the
+            # manuals. A no-break space is therefore tolerated on any line
+            # that also contains box-drawing characters (i.e. inside a
+            # rendered directory tree), but still flagged elsewhere.
+            in_tree = any(is_box_drawing(c) for c in line)
+            for col, char in enumerate(line, start=1):
+                if char not in CONFUSABLES:
+                    continue
+                if char == NO_BREAK_SPACE and in_tree:
+                    continue
+                replacement = CONFUSABLES[char]
+                hint = f"'{replacement}'" if replacement else "(remove)"
+                print(f"WARNING: {path}:{lineno}:{col}: non-ASCII "
+                      f"character U+{ord(char):04X} should be "
+                      f"replaced with {hint}")
+                found = True
+
+    return found
+
+
+def main():
+
+    args = parse_arguments()
+
+    # When invoked with explicit files (e.g. by pre-commit, which passes the
+    # staged filenames) only those are checked; otherwise the whole tree of
+    # *.rst files under --docs-dir is scanned (e.g. by "make checks").
+    if args.files:
+        targets = args.files
+    else:
+        targets = sorted(args.docs_dir.rglob("*.rst"))
+
+    exit_code = 0
+    for path in targets:
+        if check_file(path):
+            exit_code = 1
+
+    sys.exit(exit_code)
+
+
+if __name__ == "__main__":
+    main()

base-commit: ce9e3b121a258933c65397160fc5889e0436397c
-- 
2.47.3

Re: [docs] [PATCH v5 2/2] tools: Add check-confusables pre-commit hook

[Niko Mauno]

On 8/3/26 11:30 AM, Antonin Godard wrote:
> Hi,
> 
> On Mon Aug 3, 2026 at 5:48 AM CEST, Niko Mauno via lists.yoctoproject.org wrote:
>> From: Niko Mauno <niko.mauno@vaisala.com>
>>
>> Add a check-confusables script, in the same fashion as
>> check-glossaries, that scans the documentation .rst sources for
>> non-ASCII "confusable" characters (curly quotes, en/em dashes,
>> non-breaking and zero-width spaces, etc.) and reports each occurrence
>> with its location and suggested ASCII replacement, exiting non-zero if
>> any are found. This guards against the class of breakage fixed in the
>> preceding commit, e.g. curly quotes causing recipe ParseErrors.
>>
>> Legitimate non-ASCII such as box-drawing characters used in directory
>> trees, accented letters in contributor names and CJK characters are
>> intentionally left untouched. No-break spaces are likewise tolerated
>> on lines containing box-drawing characters, since the tree command
>> emits them as indentation in directory listings.
>>
>> The set of flagged characters is intentionally small and curated
>> rather than exhaustive. A general-purpose dependency such as the
>> confusables PyPI package targets Unicode homoglyph detection against
>> the full confusables table; it would also flag the accented names, CJK
>> and box-drawing characters we deliberately keep, so we would still need
>> our own allow-list and replacement policy on top of it. A short,
>> dependency-free table kept in-tree matches check-glossaries and is
>> trivial to extend if a new problematic character shows up.
>>
>> Wire it up both as a local pre-commit hook, which checks the changed
>> files, and in the Makefile "checks" target, which scans the whole
>> tree, alongside check-glossaries.
>>
>> Suggested-by: Quentin Schulz <quentin.schulz@cherry.de>
>> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
>> Signed-off-by: Niko Mauno <niko.mauno@vaisala.com>
>> ---
>>   .pre-commit-config.yaml               |   5 ++
>>   documentation/Makefile                |   1 +
>>   documentation/tools/check-confusables | 123 ++++++++++++++++++++++++++
>>   3 files changed, 129 insertions(+)
>>   create mode 100755 documentation/tools/check-confusables
>>
>> diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
>> index f2b73a481..876546f9a 100644
>> --- a/.pre-commit-config.yaml
>> +++ b/.pre-commit-config.yaml
>> @@ -6,3 +6,8 @@ repos:
>>           entry: ./documentation/tools/check-glossaries
>>           language: python
>>           pass_filenames: false
>> +      - id: check-confusables
>> +        name: Check for non-ASCII confusable characters
>> +        entry: ./documentation/tools/check-confusables
>> +        language: python
>> +        files: \.rst$
>> diff --git a/documentation/Makefile b/documentation/Makefile
>> index fe0574537..87a6f8a8b 100644
>> --- a/documentation/Makefile
>> +++ b/documentation/Makefile
>> @@ -37,6 +37,7 @@ clean:
>>   
>>   checks:
>>   	$(SOURCEDIR)/tools/check-glossaries --docs-dir $(SOURCEDIR)
>> +	$(SOURCEDIR)/tools/check-confusables --docs-dir $(SOURCEDIR)
>>   
>>   stylecheck:
>>   	vale sync
>> diff --git a/documentation/tools/check-confusables b/documentation/tools/check-confusables
>> new file mode 100755
>> index 000000000..359386e7c
>> --- /dev/null
>> +++ b/documentation/tools/check-confusables
>> @@ -0,0 +1,123 @@
>> +#!/usr/bin/env python3
>> +#
>> +# Check documentation sources for non-ASCII typographic characters that
>> +# should be plain ASCII.
>> +#
>> +# Copyright (c) Vaisala Oyj. All rights reserved.
>> +#
>> +# SPDX-License-Identifier: MIT
>> +#
>> +
>> +import argparse
>> +import sys
>> +
>> +from pathlib import Path
>> +
>> +
>> +def parse_arguments() -> argparse.Namespace:
>> +    parser = argparse.ArgumentParser(
>> +        description="Check documentation sources for non-ASCII typographic "
>> +                    "characters that should be plain ASCII")
>> +
>> +    parser.add_argument("files",
>> +                        nargs="*",
>> +                        type=Path,
>> +                        help="Specific files to check; if none are given, "
>> +                             "all *.rst files under --docs-dir are scanned")
>> +
>> +    parser.add_argument("-d", "--docs-dir",
>> +                        type=Path,
>> +                        default=Path(__file__).resolve().parent.parent,
>> +                        help="Path to documentation/ directory in yocto-docs")
>> +
>> +    return parser.parse_args()
>> +
>> +
>> +# Map of "confusable" characters that are frequently introduced by editors,
>> +# word processors or copy-pasting, to their plain ASCII replacement. These
>> +# look almost identical to regular ASCII but break tooling, e.g. a curly
>> +# quote in a recipe example causes:
>> +#
>> +#   ERROR: ParseError ...: unparsed line: 'RDEPENDS:${PN} = “foo”'
>> +#
>> +# Only these characters are flagged; legitimate non-ASCII such as box-drawing
>> +# characters used in directory trees, accented letters in contributor names
>> +# and CJK characters are intentionally left alone.
>> +confusables = {
> 
> Nit, but this is defined in lowercase, but NO_BREAK_SPACE is uppercase, while
> both are global variables. Maybe make those all uppercase?
> 
>> +    "\u2018": "'",      # LEFT SINGLE QUOTATION MARK
>> +    "\u2019": "'",      # RIGHT SINGLE QUOTATION MARK
>> +    "\u201c": '"',      # LEFT DOUBLE QUOTATION MARK
>> +    "\u201d": '"',      # RIGHT DOUBLE QUOTATION MARK
>> +    "\u2032": "'",      # PRIME
>> +    "\u2033": '"',      # DOUBLE PRIME
>> +    "\u2013": "-",      # EN DASH
>> +    "\u2014": "--",     # EM DASH
>> +    "\u2010": "-",      # HYPHEN
>> +    "\u2011": "-",      # NON-BREAKING HYPHEN
>> +    "\u2212": "-",      # MINUS SIGN
>> +    "\u00a0": " ",      # NO-BREAK SPACE
> 
> You could reuse NO_BREAK_SPACE here, if you define it before.
> 
>> +    "\u202f": " ",      # NARROW NO-BREAK SPACE
>> +    "\u200b": "",       # ZERO WIDTH SPACE
>> +    "\ufeff": "",       # ZERO WIDTH NO-BREAK SPACE / BOM
>> +    "\u00ad": "",       # SOFT HYPHEN
>> +}
>> +
>> +NO_BREAK_SPACE = "\u00a0"
>> +
>> +
>> +def is_box_drawing(char: str) -> bool:
>> +    # Box Drawing Unicode block (U+2500..U+257F), used for the directory
>> +    # trees rendered in the manuals.
>> +    return "\u2500" <= char <= "\u257f"
>> +
>> +
>> +def check_file(path: Path, display: str) -> bool:
>> +    found = False
>> +
>> +    with open(path, "r", encoding="utf-8") as f:
>> +        for lineno, line in enumerate(f, start=1):
>> +            # The tree(1) command indents its directory listings with
>> +            # no-break spaces; such listings are embedded verbatim in the
>> +            # manuals. A no-break space is therefore tolerated on any line
>> +            # that also contains box-drawing characters (i.e. inside a
>> +            # rendered directory tree), but still flagged elsewhere.
>> +            in_tree = any(is_box_drawing(c) for c in line)
> 
> Not a blocking change, but at this point you could probably continue if in_tree
> is True and move onto the next line. Since we know we're in a box-drawing,
> leaving confusables is fine since it's not a command to run?

Thanks Antonin, submitted v6 based on your feedback on other counts,
except the above one:

   ▎ On skipping the rest of the line when in_tree is true: I kept the
   ▎ narrower per-character rule. The no-break space is the only confusable
   ▎ the tree command is known to emit in directory listings, so exempting
   ▎ the whole line would also stop flagging curly quotes and dashes there,
   ▎ without a known false positive to justify it. Happy to change it if
   ▎ you would rather have the simpler loop.

-Niko


> 
>> +            for col, char in enumerate(line, start=1):
>> +                if char not in confusables:
>> +                    continue
>> +                if char == NO_BREAK_SPACE and in_tree:
>> +                    continue
>> +                replacement = confusables[char]
>> +                hint = f"'{replacement}'" if replacement else "(remove)"
>> +                print(f"WARNING: {display}:{lineno}:{col}: non-ASCII "
>> +                      f"character U+{ord(char):04X} should be "
>> +                      f"replaced with {hint}")
>> +                found = True
>> +
>> +    return found
>> +
>> +
>> +def main():
>> +
>> +    args = parse_arguments()
>> +
>> +    # When invoked with explicit files (e.g. by pre-commit, which passes the
>> +    # staged filenames) only those are checked; otherwise the whole tree of
>> +    # *.rst files under --docs-dir is scanned (e.g. by "make checks").
>> +    if args.files:
>> +        targets = [(path, str(path)) for path in args.files]
> 
> Those are Path objects, I don't think you need to store their string
> representation separately to display them? Why not simply passing the Path
> object when calling print() directly? It should get converted to strings
> directly. And this would make check_file() simpler:
> 
> def check_file(path: Path) -> bool:
> 
>      ...
> 
>      print(f"WARNING: {path}:{lineno}:{col}: non-ASCII "
>            f"character U+{ord(char):04X} should be "
>            f"replaced with {hint}")
> 
>> +    else:
>> +        docs_dir = Path(args.docs_dir)
>> +        targets = [(path, str(path.relative_to(docs_dir)))
>> +                   for path in sorted(docs_dir.rglob("*.rst"))]
>> +
>> +    exit_code = 0
>> +    for path, display in targets:
>> +        if check_file(path, display):
>> +            exit_code = 1
>> +
>> +    sys.exit(exit_code)
>> +
>> +
>> +if __name__ == "__main__":
>> +    main()
> 
> Thanks,
> Antonin