[PATCH 0/2] docs: Avoid duplicate labels with a sphinx extn

[PATCH 0/2] docs: Avoid duplicate labels with a sphinx extn

[Peter Maydell]

Sphinx requires that labels within documents are unique across the
whole manual.  This is because the "create a hyperlink" directive
specifies only the name of the label, not a filename+label.  Some
Sphinx versions will warn about duplicate labels, but even if there
is no warning there is still an ambiguity and no guarantee that the
hyperlink will be created to the intended target.

For QEMU this is awkward, because we have various .rst.inc fragments
which we include into multiple .rst files.  If you define a label in
the .rst.inc file then it will be a duplicate label.  We have mostly
worked around this by not putting labels into those .rst.inc files,
or by adding "insert a label" functionality into the hxtool extension
(see commit 1eeb432a953b0 "doc/sphinx/hxtool.py: add optional label
argument to SRST directive"). However, we let one into the codebase
without initially noticing, in commit 7f6314427e ("docs/devel: add a
codebase section"), because not all versions of Sphinx warn about
the duplicate labels.

This patchset resolves the problem by implementing a small Sphinx
extension. The extension lets you write in a .rst.inc:

  .. uniquelabel:: mylabel

and it will be as if you had written:

  .. _foo/bar-mylabel

where foo/bar.rst is the top level document that includes the
.rst.inc file.

Patch 1 is the extension; patch 2 is the use of it to fix the
problem in qemu-block-drivers.rst.inc. (Concretely, the result is
that instead of an ambiguous "nbd" label, we now have separate
"system/images-nbd" and "system/qemu-block-drivers-nbd" labels.
We want to link to the former, because the latter is in the
manpage, not the proper HTML manual.)

This patchset is a bit RFC quality -- I have not tested it
super thoroughly, and the extension itself is written based on
our existing ones, because I'm neither a Python nor a Sphinx
expert. I figured I'd send it out to see if people agreed that
it was the right way to solve this problem.

(In theory we could remove the SRST(label) functionality from
the hxtool extension and have the .hx files use uniquelabel.
Not sure that's worthwhile at this point.)

PS: I find that our extensions are confused about whether they
should set "required_arguments = 1" or "required_argument = 1";
probably the latter are all bugs that happen to have no bad
side effects...

thanks
-- PMM

Peter Maydell (2):
  docs: Create a uniquelabel Sphinx extension
  docs: Use uniquelabel in qemu-block-drivers.rst.inc

 docs/conf.py                           |  1 +
 docs/devel/codebase.rst                |  2 +-
 docs/sphinx/uniquelabel.py             | 74 ++++++++++++++++++++++++++
 docs/system/qemu-block-drivers.rst.inc |  2 +-
 4 files changed, 77 insertions(+), 2 deletions(-)
 create mode 100644 docs/sphinx/uniquelabel.py

-- 
2.43.0

[PATCH 1/2] docs: Create a uniquelabel Sphinx extension

[Peter Maydell]

Sphinx requires that labels within documents are unique across the
whole manual.  This is because the "create a hyperlink" directive
specifies only the name of the label, not a filename+label.  Some
Sphinx versions will warn about duplicate labels, but even if there
is no warning there is still an ambiguity and no guarantee that the
hyperlink will be created to the right target.

For QEMU this is awkward, because we have various .rst.inc fragments
which we include into multiple .rst files.  If you define a label in
the .rst.inc file then it will be a duplicate label.  We have mostly
worked around this by not putting labels into those .rst.inc files,
or by adding "insert a label" functionality into the hxtool extension
(see commit 1eeb432a953b0 "doc/sphinx/hxtool.py: add optional label
argument to SRST directive").

This Sphinx extension adds a "uniquelabel" directive, which creates a
label which is made unique by adding the name of the document to the
label.  The name of the generated label is "dir/file-labelname"; this
is patterned on the generated label names that the hxtool SRST(label)
directive creates.

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
---
 docs/conf.py               |  1 +
 docs/sphinx/uniquelabel.py | 74 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 75 insertions(+)
 create mode 100644 docs/sphinx/uniquelabel.py

diff --git a/docs/conf.py b/docs/conf.py
index 7b5712e122f..562db95bbf8 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -67,6 +67,7 @@
     'qapi_domain',
     'qapidoc',
     'qmp_lexer',
+    'uniquelabel',
 ]
 
 if sphinx.version_info[:3] > (4, 0, 0):
diff --git a/docs/sphinx/uniquelabel.py b/docs/sphinx/uniquelabel.py
new file mode 100644
index 00000000000..cc4ab7f31c0
--- /dev/null
+++ b/docs/sphinx/uniquelabel.py
@@ -0,0 +1,74 @@
+# coding=utf-8
+#
+# Copyright (c) 2025 Linaro
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# Sphinx extension to create a unique label by concatenating
+# the name of the origin document with the label text.
+#
+# Sphinx requires that labels within documents are unique across
+# the whole manual. This is because the "create a hyperlink" directive
+# specifies only the name of the label, not a filename+label.
+# Some Sphinx versions will warn about duplicate labels, but
+# even if there is no warning there is still an ambiguity and no
+# guarantee that the hyperlink will be created to the right target.
+#
+# For QEMU this is awkward, because we have various .rst.inc fragments
+# which we include into multiple .rst files. If you define a label in
+# the .rst.inc file then it will be a duplicate label.
+#
+# The uniquelabel directive is our fix for this: it creates a label
+# whose name includes the name of the top level .rst file. This is then
+# unique even if the .rst.inc file is included in multiple places, and
+# when we create a hyperlink we can explicitly specify which label we
+# are targeting.
+#
+# Concretely, if you have a foo/bar.rst and a foo/baz.rst that
+# both include wat.rst.inc, then in wat.rst.inc you can write
+# .. uniquelabel:: mylabel
+# and it will be as if you had written a reference label:
+# .. _foo/bar-mylabel
+# or
+# .. _foo/baz-mylabel
+# depending on which file included wat.rst.inc, and you can link to
+# whichever one you intend via any of the usual markup, e.g.
+# `documentation of the thing in bar <foo/bar-mylabel>`.
+
+"""uniquelabel is a Sphinx extension that implements the uniquelabel directive"""
+
+from docutils import nodes
+from docutils.statemachine import ViewList
+from docutils.parsers.rst import directives, Directive
+import sphinx
+
+__version__ = '1.0'
+
+class UniqueLabelDocDirective(Directive):
+    """Create a unique label by including the docname"""
+    required_arguments = 1
+    optional_arguments = 0
+    has_content = False
+
+    def run(self):
+        env = self.state.document.settings.env
+        label = self.arguments[0]
+
+        refline = ".. _" + env.docname + "-" + label + ":"
+
+        rstlist = ViewList()
+        rstlist.append(refline, "generated text", 0)
+
+        node = nodes.paragraph()
+        self.state.nested_parse(rstlist, 0, node)
+        return node.children
+
+def setup(app):
+    """ Register uniquelabel directive with Sphinx"""
+    app.add_directive('uniquelabel', UniqueLabelDocDirective)
+
+    return dict(
+        version = __version__,
+        parallel_read_safe = True,
+        parallel_write_safe = True
+    )
-- 
2.43.0

[PATCH 2/2] docs: Use uniquelabel in qemu-block-drivers.rst.inc

[Peter Maydell]

Use the uniquelabel directive for the 'nbd' label in
qemu-block-drivers.rst.inc.  This avoids a complaint from some Sphinx
versions about it being a duplicate label, and means that we can
reliably ensure that the link in codebase.rst goes to the document
file that we intend (i.e.  the section of the HTML manual on disk
images, not the HTML copy of the qemu-block-drivers manpage).

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
---
 docs/devel/codebase.rst                | 2 +-
 docs/system/qemu-block-drivers.rst.inc | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/docs/devel/codebase.rst b/docs/devel/codebase.rst
index 40273e7d31e..f3f133e9adb 100644
--- a/docs/devel/codebase.rst
+++ b/docs/devel/codebase.rst
@@ -116,7 +116,7 @@ yet, so sometimes the source code is all you have.
 * `monitor <https://gitlab.com/qemu-project/qemu/-/tree/master/monitor>`_:
   `Monitor <QEMU monitor>` implementation (HMP & QMP).
 * `nbd <https://gitlab.com/qemu-project/qemu/-/tree/master/nbd>`_:
-  QEMU `NBD (Network Block Device) <nbd>` server.
+  QEMU `NBD (Network Block Device) <system/images-nbd>` server.
 * `net <https://gitlab.com/qemu-project/qemu/-/tree/master/net>`_:
   Network (host) support.
 * `pc-bios <https://gitlab.com/qemu-project/qemu/-/tree/master/pc-bios>`_:
diff --git a/docs/system/qemu-block-drivers.rst.inc b/docs/system/qemu-block-drivers.rst.inc
index cfe1acb78ae..1a405e47823 100644
--- a/docs/system/qemu-block-drivers.rst.inc
+++ b/docs/system/qemu-block-drivers.rst.inc
@@ -500,7 +500,7 @@ What you should *never* do:
 - expect it to work when loadvm'ing
 - write to the FAT directory on the host system while accessing it with the guest system
 
-.. _nbd:
+.. uniquelabel:: nbd
 
 NBD access
 ~~~~~~~~~~
-- 
2.43.0

Re: [PATCH 0/2] docs: Avoid duplicate labels with a sphinx extn

[Peter Maydell]

Ping? Any opinions on this?

In the interim we've applied commit 82707dd4f0 to drop
the specific duplicate-label that is causing problems
right now, so patch 2 here will need the obvious trivial
update. But I do think this is a better approach than
forever avoiding defining labels in .rst.inc files...

thanks
-- PMM

On Tue, 29 Apr 2025 at 17:32, Peter Maydell <peter.maydell@linaro.org> wrote:
>
> Sphinx requires that labels within documents are unique across the
> whole manual.  This is because the "create a hyperlink" directive
> specifies only the name of the label, not a filename+label.  Some
> Sphinx versions will warn about duplicate labels, but even if there
> is no warning there is still an ambiguity and no guarantee that the
> hyperlink will be created to the intended target.
>
> For QEMU this is awkward, because we have various .rst.inc fragments
> which we include into multiple .rst files.  If you define a label in
> the .rst.inc file then it will be a duplicate label.  We have mostly
> worked around this by not putting labels into those .rst.inc files,
> or by adding "insert a label" functionality into the hxtool extension
> (see commit 1eeb432a953b0 "doc/sphinx/hxtool.py: add optional label
> argument to SRST directive"). However, we let one into the codebase
> without initially noticing, in commit 7f6314427e ("docs/devel: add a
> codebase section"), because not all versions of Sphinx warn about
> the duplicate labels.
>
> This patchset resolves the problem by implementing a small Sphinx
> extension. The extension lets you write in a .rst.inc:
>
>   .. uniquelabel:: mylabel
>
> and it will be as if you had written:
>
>   .. _foo/bar-mylabel
>
> where foo/bar.rst is the top level document that includes the
> .rst.inc file.
>
> Patch 1 is the extension; patch 2 is the use of it to fix the
> problem in qemu-block-drivers.rst.inc. (Concretely, the result is
> that instead of an ambiguous "nbd" label, we now have separate
> "system/images-nbd" and "system/qemu-block-drivers-nbd" labels.
> We want to link to the former, because the latter is in the
> manpage, not the proper HTML manual.)
>
> This patchset is a bit RFC quality -- I have not tested it
> super thoroughly, and the extension itself is written based on
> our existing ones, because I'm neither a Python nor a Sphinx
> expert. I figured I'd send it out to see if people agreed that
> it was the right way to solve this problem.
>
> (In theory we could remove the SRST(label) functionality from
> the hxtool extension and have the .hx files use uniquelabel.
> Not sure that's worthwhile at this point.)
>
> PS: I find that our extensions are confused about whether they
> should set "required_arguments = 1" or "required_argument = 1";
> probably the latter are all bugs that happen to have no bad
> side effects...
>
> thanks
> -- PMM
>
> Peter Maydell (2):
>   docs: Create a uniquelabel Sphinx extension
>   docs: Use uniquelabel in qemu-block-drivers.rst.inc
>
>  docs/conf.py                           |  1 +
>  docs/devel/codebase.rst                |  2 +-
>  docs/sphinx/uniquelabel.py             | 74 ++++++++++++++++++++++++++
>  docs/system/qemu-block-drivers.rst.inc |  2 +-
>  4 files changed, 77 insertions(+), 2 deletions(-)
>  create mode 100644 docs/sphinx/uniquelabel.py
>
> --

Re: [PATCH 0/2] docs: Avoid duplicate labels with a sphinx extn

[Peter Maydell]

Ping^2 on this one?

-- PMM

On Mon, 19 May 2025 at 15:32, Peter Maydell <peter.maydell@linaro.org> wrote:
>
> Ping? Any opinions on this?
>
> In the interim we've applied commit 82707dd4f0 to drop
> the specific duplicate-label that is causing problems
> right now, so patch 2 here will need the obvious trivial
> update. But I do think this is a better approach than
> forever avoiding defining labels in .rst.inc files...
>
> thanks
> -- PMM
>
> On Tue, 29 Apr 2025 at 17:32, Peter Maydell <peter.maydell@linaro.org> wrote:
> >
> > Sphinx requires that labels within documents are unique across the
> > whole manual.  This is because the "create a hyperlink" directive
> > specifies only the name of the label, not a filename+label.  Some
> > Sphinx versions will warn about duplicate labels, but even if there
> > is no warning there is still an ambiguity and no guarantee that the
> > hyperlink will be created to the intended target.
> >
> > For QEMU this is awkward, because we have various .rst.inc fragments
> > which we include into multiple .rst files.  If you define a label in
> > the .rst.inc file then it will be a duplicate label.  We have mostly
> > worked around this by not putting labels into those .rst.inc files,
> > or by adding "insert a label" functionality into the hxtool extension
> > (see commit 1eeb432a953b0 "doc/sphinx/hxtool.py: add optional label
> > argument to SRST directive"). However, we let one into the codebase
> > without initially noticing, in commit 7f6314427e ("docs/devel: add a
> > codebase section"), because not all versions of Sphinx warn about
> > the duplicate labels.
> >
> > This patchset resolves the problem by implementing a small Sphinx
> > extension. The extension lets you write in a .rst.inc:
> >
> >   .. uniquelabel:: mylabel
> >
> > and it will be as if you had written:
> >
> >   .. _foo/bar-mylabel
> >
> > where foo/bar.rst is the top level document that includes the
> > .rst.inc file.
> >
> > Patch 1 is the extension; patch 2 is the use of it to fix the
> > problem in qemu-block-drivers.rst.inc. (Concretely, the result is
> > that instead of an ambiguous "nbd" label, we now have separate
> > "system/images-nbd" and "system/qemu-block-drivers-nbd" labels.
> > We want to link to the former, because the latter is in the
> > manpage, not the proper HTML manual.)
> >
> > This patchset is a bit RFC quality -- I have not tested it
> > super thoroughly, and the extension itself is written based on
> > our existing ones, because I'm neither a Python nor a Sphinx
> > expert. I figured I'd send it out to see if people agreed that
> > it was the right way to solve this problem.
> >
> > (In theory we could remove the SRST(label) functionality from
> > the hxtool extension and have the .hx files use uniquelabel.
> > Not sure that's worthwhile at this point.)
> >
> > PS: I find that our extensions are confused about whether they
> > should set "required_arguments = 1" or "required_argument = 1";
> > probably the latter are all bugs that happen to have no bad
> > side effects...
> >
> > thanks
> > -- PMM
> >
> > Peter Maydell (2):
> >   docs: Create a uniquelabel Sphinx extension
> >   docs: Use uniquelabel in qemu-block-drivers.rst.inc
> >
> >  docs/conf.py                           |  1 +
> >  docs/devel/codebase.rst                |  2 +-
> >  docs/sphinx/uniquelabel.py             | 74 ++++++++++++++++++++++++++
> >  docs/system/qemu-block-drivers.rst.inc |  2 +-
> >  4 files changed, 77 insertions(+), 2 deletions(-)
> >  create mode 100644 docs/sphinx/uniquelabel.py
> >
> > --

Re: [PATCH 0/2] docs: Avoid duplicate labels with a sphinx extn

[Paolo Bonzini]

On 6/9/25 18:18, Peter Maydell wrote:
> Ping^2 on this one?

No objections if it's the easiest way to solve the issue.

Alternatively, is there a Sphinx way to write something in the spirit of

#ifdef DEFINE_THE_LABEL
.. _label
#endif

This way, you could have the nbd label enabled when including into 
system/images, but not when including into system/qemu-block-drivers.

Paolo

Re: [PATCH 0/2] docs: Avoid duplicate labels with a sphinx extn

[Peter Maydell]

On Mon, 9 Jun 2025 at 17:38, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> On 6/9/25 18:18, Peter Maydell wrote:
> > Ping^2 on this one?
>
> No objections if it's the easiest way to solve the issue.
>
> Alternatively, is there a Sphinx way to write something in the spirit of
>
> #ifdef DEFINE_THE_LABEL
> .. _label
> #endif

Unfortunately not. Sphinx's conditional-content stuff is a bit
odd, and only works for actual docs, not for sections, labels, etc:

https://www.sphinx-doc.org/en/master/usage/extensions/ifconfig.html
https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#including-content-based-on-tags

IIRC this is because under the hood the conditionalisation
only takes effect quite late, when it's about to emit
something: they really act just as filters on the output.
So you can't conditionalise a directive or anything that
needs to take effect at the parsing stage.

https://pypi.org/project/sphinx-selective-exclude/
is an extension that tries to work around this, but it
hasn't been updated in years and it probably doesn't work
with newer sphinx versions.

-- PMM