[PATCH v2 0/5] tools: ynl: add install target

[PATCH v2 0/5] tools: ynl: add install target

[Jan Stancek]

This series adds an install target for ynl. The python code
is moved to a subdirectory, so it can be used as a package
with flat layout, as well as directly from the tree.

To try the install as a non-root user you can run:
  $ mkdir /tmp/myroot
  $ make DESTDIR=/tmp/myroot install

  $ PATH="/tmp/myroot/usr/bin:$PATH" PYTHONPATH="$(ls -1d /tmp/myroot/usr/lib/python*/site-packages)" ynl --help

Proposed install layout is described in last patch.

Changes in v2:
- squashed 1st and 2nd patch together
- added symlinks for original user-facing scripts for backwards compatibility
- updated also Documentation and selftests references
  (tested Doc build and selftests ping.py test)

Jan Stancek (5):
  tools: ynl: move python code to separate sub-directory
  tools: ynl: provide symlinks to user-facing scripts for compatibility
  tools: ynl: add initial pyproject.toml for packaging
  tools: ynl: add install target for specs and docs
  tools: ynl: add main install target

 Documentation/Makefile                        |    2 +-
 Documentation/networking/multi-pf-netdev.rst  |    4 +-
 Documentation/networking/napi.rst             |    4 +-
 .../networking/netlink_spec/readme.txt        |    2 +-
 .../userspace-api/netlink/intro-specs.rst     |    8 +-
 tools/net/ynl/Makefile                        |   25 +-
 tools/net/ynl/cli.py                          |  120 +-
 tools/net/ynl/ethtool.py                      |  440 +--
 tools/net/ynl/generated/.gitignore            |    1 +
 tools/net/ynl/generated/Makefile              |   36 +-
 tools/net/ynl/lib/.gitignore                  |    1 -
 tools/net/ynl/lib/Makefile                    |    1 -
 tools/net/ynl/pyproject.toml                  |   26 +
 tools/net/ynl/pyynl/.gitignore                |    2 +
 tools/net/ynl/pyynl/__init__.py               |    0
 tools/net/ynl/pyynl/cli.py                    |  119 +
 tools/net/ynl/pyynl/ethtool.py                |  439 +++
 tools/net/ynl/{ => pyynl}/lib/__init__.py     |    0
 tools/net/ynl/{ => pyynl}/lib/nlspec.py       |    0
 tools/net/ynl/{ => pyynl}/lib/ynl.py          |    0
 tools/net/ynl/pyynl/ynl_gen_c.py              | 3018 ++++++++++++++++
 tools/net/ynl/pyynl/ynl_gen_rst.py            |  453 +++
 tools/net/ynl/ynl-gen-c.py                    | 3019 +----------------
 tools/net/ynl/ynl-gen-rst.py                  |  454 +--
 tools/net/ynl/ynl-regen.sh                    |    2 +-
 tools/testing/selftests/net/lib/py/ynl.py     |    4 +-
 tools/testing/selftests/net/ynl.mk            |    3 +-
 27 files changed, 4133 insertions(+), 4050 deletions(-)
 mode change 100755 => 120000 tools/net/ynl/cli.py
 mode change 100755 => 120000 tools/net/ynl/ethtool.py
 create mode 100644 tools/net/ynl/pyproject.toml
 create mode 100644 tools/net/ynl/pyynl/.gitignore
 create mode 100644 tools/net/ynl/pyynl/__init__.py
 create mode 100755 tools/net/ynl/pyynl/cli.py
 create mode 100755 tools/net/ynl/pyynl/ethtool.py
 rename tools/net/ynl/{ => pyynl}/lib/__init__.py (100%)
 rename tools/net/ynl/{ => pyynl}/lib/nlspec.py (100%)
 rename tools/net/ynl/{ => pyynl}/lib/ynl.py (100%)
 create mode 100755 tools/net/ynl/pyynl/ynl_gen_c.py
 create mode 100755 tools/net/ynl/pyynl/ynl_gen_rst.py
 mode change 100755 => 120000 tools/net/ynl/ynl-gen-c.py
 mode change 100755 => 120000 tools/net/ynl/ynl-gen-rst.py

-- 
2.43.0

[PATCH v2 1/5] tools: ynl: move python code to separate sub-directory

[Jan Stancek]

Move python code to a separate directory so it can be
packaged as a python module. Updates existing references
in selftests and docs.

Also rename ynl-gen-[c|rst] to ynl_gen_[c|rst], avoid
dashes as these prevent easy imports for entrypoints.

Signed-off-by: Jan Stancek <jstancek@redhat.com>
---
 Documentation/Makefile                                 | 2 +-
 Documentation/networking/multi-pf-netdev.rst           | 4 ++--
 Documentation/networking/napi.rst                      | 4 ++--
 Documentation/networking/netlink_spec/readme.txt       | 2 +-
 Documentation/userspace-api/netlink/intro-specs.rst    | 8 ++++----
 tools/net/ynl/Makefile                                 | 2 ++
 tools/net/ynl/generated/Makefile                       | 2 +-
 tools/net/ynl/lib/.gitignore                           | 1 -
 tools/net/ynl/lib/Makefile                             | 1 -
 tools/net/ynl/pyynl/.gitignore                         | 2 ++
 tools/net/ynl/pyynl/__init__.py                        | 0
 tools/net/ynl/{ => pyynl}/cli.py                       | 0
 tools/net/ynl/{ => pyynl}/ethtool.py                   | 0
 tools/net/ynl/{ => pyynl}/lib/__init__.py              | 0
 tools/net/ynl/{ => pyynl}/lib/nlspec.py                | 0
 tools/net/ynl/{ => pyynl}/lib/ynl.py                   | 0
 tools/net/ynl/{ynl-gen-c.py => pyynl/ynl_gen_c.py}     | 0
 tools/net/ynl/{ynl-gen-rst.py => pyynl/ynl_gen_rst.py} | 0
 tools/net/ynl/ynl-regen.sh                             | 2 +-
 tools/testing/selftests/net/lib/py/ynl.py              | 4 ++--
 tools/testing/selftests/net/ynl.mk                     | 3 ++-
 21 files changed, 20 insertions(+), 17 deletions(-)
 create mode 100644 tools/net/ynl/pyynl/.gitignore
 create mode 100644 tools/net/ynl/pyynl/__init__.py
 rename tools/net/ynl/{ => pyynl}/cli.py (100%)
 rename tools/net/ynl/{ => pyynl}/ethtool.py (100%)
 rename tools/net/ynl/{ => pyynl}/lib/__init__.py (100%)
 rename tools/net/ynl/{ => pyynl}/lib/nlspec.py (100%)
 rename tools/net/ynl/{ => pyynl}/lib/ynl.py (100%)
 rename tools/net/ynl/{ynl-gen-c.py => pyynl/ynl_gen_c.py} (100%)
 rename tools/net/ynl/{ynl-gen-rst.py => pyynl/ynl_gen_rst.py} (100%)

diff --git a/Documentation/Makefile b/Documentation/Makefile
index fa71602ec961..52c6c5a3efa9 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -104,7 +104,7 @@ quiet_cmd_sphinx = SPHINX  $@ --> file://$(abspath $(BUILDDIR)/$3/$4)
 YNL_INDEX:=$(srctree)/Documentation/networking/netlink_spec/index.rst
 YNL_RST_DIR:=$(srctree)/Documentation/networking/netlink_spec
 YNL_YAML_DIR:=$(srctree)/Documentation/netlink/specs
-YNL_TOOL:=$(srctree)/tools/net/ynl/ynl-gen-rst.py
+YNL_TOOL:=$(srctree)/tools/net/ynl/pyynl/ynl_gen_rst.py
 
 YNL_RST_FILES_TMP := $(patsubst %.yaml,%.rst,$(wildcard $(YNL_YAML_DIR)/*.yaml))
 YNL_RST_FILES := $(patsubst $(YNL_YAML_DIR)%,$(YNL_RST_DIR)%, $(YNL_RST_FILES_TMP))
diff --git a/Documentation/networking/multi-pf-netdev.rst b/Documentation/networking/multi-pf-netdev.rst
index 2cd25d81aaa7..2f5a5bb3ca9a 100644
--- a/Documentation/networking/multi-pf-netdev.rst
+++ b/Documentation/networking/multi-pf-netdev.rst
@@ -89,7 +89,7 @@ Observability
 =============
 The relation between PF, irq, napi, and queue can be observed via netlink spec::
 
-  $ ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/netdev.yaml --dump queue-get --json='{"ifindex": 13}'
+  $ ./tools/net/ynl/pyynl/cli.py --spec Documentation/netlink/specs/netdev.yaml --dump queue-get --json='{"ifindex": 13}'
   [{'id': 0, 'ifindex': 13, 'napi-id': 539, 'type': 'rx'},
    {'id': 1, 'ifindex': 13, 'napi-id': 540, 'type': 'rx'},
    {'id': 2, 'ifindex': 13, 'napi-id': 541, 'type': 'rx'},
@@ -101,7 +101,7 @@ The relation between PF, irq, napi, and queue can be observed via netlink spec::
    {'id': 3, 'ifindex': 13, 'napi-id': 542, 'type': 'tx'},
    {'id': 4, 'ifindex': 13, 'napi-id': 543, 'type': 'tx'}]
 
-  $ ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/netdev.yaml --dump napi-get --json='{"ifindex": 13}'
+  $ ./tools/net/ynl/pyynl/cli.py --spec Documentation/netlink/specs/netdev.yaml --dump napi-get --json='{"ifindex": 13}'
   [{'id': 543, 'ifindex': 13, 'irq': 42},
    {'id': 542, 'ifindex': 13, 'irq': 41},
    {'id': 541, 'ifindex': 13, 'irq': 40},
diff --git a/Documentation/networking/napi.rst b/Documentation/networking/napi.rst
index 02720dd71a76..6083210ab2a4 100644
--- a/Documentation/networking/napi.rst
+++ b/Documentation/networking/napi.rst
@@ -199,13 +199,13 @@ parameters mentioned above use hyphens instead of underscores:
 
 Per-NAPI configuration can be done programmatically in a user application
 or by using a script included in the kernel source tree:
-``tools/net/ynl/cli.py``.
+``tools/net/ynl/pyynl/cli.py``.
 
 For example, using the script:
 
 .. code-block:: bash
 
-  $ kernel-source/tools/net/ynl/cli.py \
+  $ kernel-source/tools/net/ynl/pyynl/cli.py \
             --spec Documentation/netlink/specs/netdev.yaml \
             --do napi-set \
             --json='{"id": 345,
diff --git a/Documentation/networking/netlink_spec/readme.txt b/Documentation/networking/netlink_spec/readme.txt
index 6763f99d216c..030b44aca4e6 100644
--- a/Documentation/networking/netlink_spec/readme.txt
+++ b/Documentation/networking/netlink_spec/readme.txt
@@ -1,4 +1,4 @@
 SPDX-License-Identifier: GPL-2.0
 
 This file is populated during the build of the documentation (htmldocs) by the
-tools/net/ynl/ynl-gen-rst.py script.
+tools/net/ynl/pyynl/ynl_gen_rst.py script.
diff --git a/Documentation/userspace-api/netlink/intro-specs.rst b/Documentation/userspace-api/netlink/intro-specs.rst
index bada89699455..a4435ae4628d 100644
--- a/Documentation/userspace-api/netlink/intro-specs.rst
+++ b/Documentation/userspace-api/netlink/intro-specs.rst
@@ -15,7 +15,7 @@ developing Netlink related code. The tool is implemented in Python
 and can use a YAML specification to issue Netlink requests
 to the kernel. Only Generic Netlink is supported.
 
-The tool is located at ``tools/net/ynl/cli.py``. It accepts
+The tool is located at ``tools/net/ynl/pyynl/cli.py``. It accepts
 a handul of arguments, the most important ones are:
 
  - ``--spec`` - point to the spec file
@@ -27,7 +27,7 @@ YAML specs can be found under ``Documentation/netlink/specs/``.
 
 Example use::
 
-  $ ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/ethtool.yaml \
+  $ ./tools/net/ynl/pyynl/cli.py --spec Documentation/netlink/specs/ethtool.yaml \
         --do rings-get \
 	--json '{"header":{"dev-index": 18}}'
   {'header': {'dev-index': 18, 'dev-name': 'eni1np1'},
@@ -75,7 +75,7 @@ the two marker lines like above to a file, add that file to git,
 and run the regeneration tool. Grep the tree for ``YNL-GEN``
 to see other examples.
 
-The code generation itself is performed by ``tools/net/ynl/ynl-gen-c.py``
+The code generation itself is performed by ``tools/net/ynl/pyynl/ynl_gen_c.py``
 but it takes a few arguments so calling it directly for each file
 quickly becomes tedious.
 
@@ -84,7 +84,7 @@ YNL lib
 
 ``tools/net/ynl/lib/`` contains an implementation of a C library
 (based on libmnl) which integrates with code generated by
-``tools/net/ynl/ynl-gen-c.py`` to create easy to use netlink wrappers.
+``tools/net/ynl/pyynl/ynl_gen_c.py`` to create easy to use netlink wrappers.
 
 YNL basics
 ----------
diff --git a/tools/net/ynl/Makefile b/tools/net/ynl/Makefile
index d1cdf2a8f826..5268b91bf7ed 100644
--- a/tools/net/ynl/Makefile
+++ b/tools/net/ynl/Makefile
@@ -21,5 +21,7 @@ clean distclean:
 		fi \
 	done
 	rm -f libynl.a
+	rm -rf pyynl/__pycache__
+	rm -rf pyynl/lib/__pycache__
 
 .PHONY: all clean distclean $(SUBDIRS)
diff --git a/tools/net/ynl/generated/Makefile b/tools/net/ynl/generated/Makefile
index 7db5240de58a..00af721b1571 100644
--- a/tools/net/ynl/generated/Makefile
+++ b/tools/net/ynl/generated/Makefile
@@ -12,7 +12,7 @@ include ../Makefile.deps
 YNL_GEN_ARG_ethtool:=--user-header linux/ethtool_netlink.h \
 	--exclude-op stats-get
 
-TOOL:=../ynl-gen-c.py
+TOOL:=../pyynl/ynl_gen_c.py
 
 GENS_PATHS=$(shell grep -nrI --files-without-match \
 		'protocol: netlink' \
diff --git a/tools/net/ynl/lib/.gitignore b/tools/net/ynl/lib/.gitignore
index 296c4035dbf2..a4383358ec72 100644
--- a/tools/net/ynl/lib/.gitignore
+++ b/tools/net/ynl/lib/.gitignore
@@ -1,2 +1 @@
-__pycache__/
 *.d
diff --git a/tools/net/ynl/lib/Makefile b/tools/net/ynl/lib/Makefile
index 94c49cca3dca..4b2b98704ff9 100644
--- a/tools/net/ynl/lib/Makefile
+++ b/tools/net/ynl/lib/Makefile
@@ -19,7 +19,6 @@ ynl.a: $(OBJS)
 
 clean:
 	rm -f *.o *.d *~
-	rm -rf __pycache__
 
 distclean: clean
 	rm -f *.a
diff --git a/tools/net/ynl/pyynl/.gitignore b/tools/net/ynl/pyynl/.gitignore
new file mode 100644
index 000000000000..b801cd2d016e
--- /dev/null
+++ b/tools/net/ynl/pyynl/.gitignore
@@ -0,0 +1,2 @@
+__pycache__/
+lib/__pycache__/
diff --git a/tools/net/ynl/pyynl/__init__.py b/tools/net/ynl/pyynl/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/net/ynl/cli.py b/tools/net/ynl/pyynl/cli.py
similarity index 100%
rename from tools/net/ynl/cli.py
rename to tools/net/ynl/pyynl/cli.py
diff --git a/tools/net/ynl/ethtool.py b/tools/net/ynl/pyynl/ethtool.py
similarity index 100%
rename from tools/net/ynl/ethtool.py
rename to tools/net/ynl/pyynl/ethtool.py
diff --git a/tools/net/ynl/lib/__init__.py b/tools/net/ynl/pyynl/lib/__init__.py
similarity index 100%
rename from tools/net/ynl/lib/__init__.py
rename to tools/net/ynl/pyynl/lib/__init__.py
diff --git a/tools/net/ynl/lib/nlspec.py b/tools/net/ynl/pyynl/lib/nlspec.py
similarity index 100%
rename from tools/net/ynl/lib/nlspec.py
rename to tools/net/ynl/pyynl/lib/nlspec.py
diff --git a/tools/net/ynl/lib/ynl.py b/tools/net/ynl/pyynl/lib/ynl.py
similarity index 100%
rename from tools/net/ynl/lib/ynl.py
rename to tools/net/ynl/pyynl/lib/ynl.py
diff --git a/tools/net/ynl/ynl-gen-c.py b/tools/net/ynl/pyynl/ynl_gen_c.py
similarity index 100%
rename from tools/net/ynl/ynl-gen-c.py
rename to tools/net/ynl/pyynl/ynl_gen_c.py
diff --git a/tools/net/ynl/ynl-gen-rst.py b/tools/net/ynl/pyynl/ynl_gen_rst.py
similarity index 100%
rename from tools/net/ynl/ynl-gen-rst.py
rename to tools/net/ynl/pyynl/ynl_gen_rst.py
diff --git a/tools/net/ynl/ynl-regen.sh b/tools/net/ynl/ynl-regen.sh
index a37304dcc88e..81b4ecd89100 100755
--- a/tools/net/ynl/ynl-regen.sh
+++ b/tools/net/ynl/ynl-regen.sh
@@ -1,7 +1,7 @@
 #!/bin/bash
 # SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
 
-TOOL=$(dirname $(realpath $0))/ynl-gen-c.py
+TOOL=$(dirname $(realpath $0))/pyynl/ynl_gen_c.py
 
 force=
 search=
diff --git a/tools/testing/selftests/net/lib/py/ynl.py b/tools/testing/selftests/net/lib/py/ynl.py
index a0d689d58c57..9c0e5a89eb12 100644
--- a/tools/testing/selftests/net/lib/py/ynl.py
+++ b/tools/testing/selftests/net/lib/py/ynl.py
@@ -13,14 +13,14 @@ try:
         SPEC_PATH = KSFT_DIR / "net/lib/specs"
 
         sys.path.append(tools_full_path.as_posix())
-        from net.lib.ynl.lib import YnlFamily, NlError
+        from net.lib.ynl.pyynl.lib import YnlFamily, NlError
     else:
         # Running in tree
         tools_full_path = KSRC / "tools"
         SPEC_PATH = KSRC / "Documentation/netlink/specs"
 
         sys.path.append(tools_full_path.as_posix())
-        from net.ynl.lib import YnlFamily, NlError
+        from net.ynl.pyynl.lib import YnlFamily, NlError
 except ModuleNotFoundError as e:
     ksft_pr("Failed importing `ynl` library from kernel sources")
     ksft_pr(str(e))
diff --git a/tools/testing/selftests/net/ynl.mk b/tools/testing/selftests/net/ynl.mk
index d43afe243779..12e7cae251be 100644
--- a/tools/testing/selftests/net/ynl.mk
+++ b/tools/testing/selftests/net/ynl.mk
@@ -31,7 +31,8 @@ $(OUTPUT)/libynl.a: $(YNL_SPECS) $(OUTPUT)/.libynl-$(YNL_GENS_HASH).sig
 	$(Q)cp $(top_srcdir)/tools/net/ynl/libynl.a $(OUTPUT)/libynl.a
 
 EXTRA_CLEAN += \
-	$(top_srcdir)/tools/net/ynl/lib/__pycache__ \
+	$(top_srcdir)/tools/net/ynl/pyynl/__pycache__ \
+	$(top_srcdir)/tools/net/ynl/pyynl/lib/__pycache__ \
 	$(top_srcdir)/tools/net/ynl/lib/*.[ado] \
 	$(OUTPUT)/.libynl-*.sig \
 	$(OUTPUT)/libynl.a
-- 
2.43.0

[PATCH v2 2/5] tools: ynl: provide symlinks to user-facing scripts for compatibility

[Jan Stancek]

For backwards compatibility provide also symlinks from original location
of user facing scripts.

Signed-off-by: Jan Stancek <jstancek@redhat.com>
---
 tools/net/ynl/cli.py         | 1 +
 tools/net/ynl/ethtool.py     | 1 +
 tools/net/ynl/ynl-gen-c.py   | 1 +
 tools/net/ynl/ynl-gen-rst.py | 1 +
 4 files changed, 4 insertions(+)
 create mode 120000 tools/net/ynl/cli.py
 create mode 120000 tools/net/ynl/ethtool.py
 create mode 120000 tools/net/ynl/ynl-gen-c.py
 create mode 120000 tools/net/ynl/ynl-gen-rst.py

diff --git a/tools/net/ynl/cli.py b/tools/net/ynl/cli.py
new file mode 120000
index 000000000000..c26fb97ae611
--- /dev/null
+++ b/tools/net/ynl/cli.py
@@ -0,0 +1 @@
+pyynl/cli.py
\ No newline at end of file
diff --git a/tools/net/ynl/ethtool.py b/tools/net/ynl/ethtool.py
new file mode 120000
index 000000000000..deea4569a939
--- /dev/null
+++ b/tools/net/ynl/ethtool.py
@@ -0,0 +1 @@
+pyynl/ethtool.py
\ No newline at end of file
diff --git a/tools/net/ynl/ynl-gen-c.py b/tools/net/ynl/ynl-gen-c.py
new file mode 120000
index 000000000000..716d34fa1257
--- /dev/null
+++ b/tools/net/ynl/ynl-gen-c.py
@@ -0,0 +1 @@
+pyynl/ynl_gen_c.py
\ No newline at end of file
diff --git a/tools/net/ynl/ynl-gen-rst.py b/tools/net/ynl/ynl-gen-rst.py
new file mode 120000
index 000000000000..b02558f540ec
--- /dev/null
+++ b/tools/net/ynl/ynl-gen-rst.py
@@ -0,0 +1 @@
+pyynl/ynl_gen_rst.py
\ No newline at end of file
-- 
2.43.0

[PATCH v2 3/5] tools: ynl: add initial pyproject.toml for packaging

[Jan Stancek]

Signed-off-by: Jan Stancek <jstancek@redhat.com>
---
 tools/net/ynl/pyproject.toml | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
 create mode 100644 tools/net/ynl/pyproject.toml

diff --git a/tools/net/ynl/pyproject.toml b/tools/net/ynl/pyproject.toml
new file mode 100644
index 000000000000..677ea8f4c185
--- /dev/null
+++ b/tools/net/ynl/pyproject.toml
@@ -0,0 +1,26 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "pyynl"
+authors = [
+    {name = "Donald Hunter", email = "donald.hunter@gmail.com"},
+    {name = "Jakub Kicinski", email = "kuba@kernel.org"},
+]
+description = "yaml netlink (ynl)"
+version = "0.0.1"
+requires-python = ">=3.9"
+dependencies = [
+    "pyyaml==6.*",
+    "jsonschema==4.*"
+]
+
+[tool.setuptools.packages.find]
+include = ["pyynl", "pyynl.lib"]
+
+[project.scripts]
+ynl = "pyynl.cli:main"
+ynl-ethtool = "pyynl.ethtool:main"
+ynl-gen-c = "pyynl.ynl_gen_c:main"
+ynl-gen-rst = "pyynl.ynl_gen_rst:main"
-- 
2.43.0

[PATCH v2 4/5] tools: ynl: add install target for specs and docs

[Jan Stancek]

Generate docs using ynl_gen_rst and add install target for
both specs and generates rst files.

Signed-off-by: Jan Stancek <jstancek@redhat.com>
---
 tools/net/ynl/generated/.gitignore |  1 +
 tools/net/ynl/generated/Makefile   | 34 +++++++++++++++++++++++++++---
 2 files changed, 32 insertions(+), 3 deletions(-)

diff --git a/tools/net/ynl/generated/.gitignore b/tools/net/ynl/generated/.gitignore
index ade488626d26..859a6fb446e1 100644
--- a/tools/net/ynl/generated/.gitignore
+++ b/tools/net/ynl/generated/.gitignore
@@ -1,2 +1,3 @@
 *-user.c
 *-user.h
+*.rst
diff --git a/tools/net/ynl/generated/Makefile b/tools/net/ynl/generated/Makefile
index 00af721b1571..d28509ac0573 100644
--- a/tools/net/ynl/generated/Makefile
+++ b/tools/net/ynl/generated/Makefile
@@ -7,12 +7,18 @@ ifeq ("$(DEBUG)","1")
   CFLAGS += -g -fsanitize=address -fsanitize=leak -static-libasan
 endif
 
+INSTALL	    ?= install
+prefix      ?= /usr
+datarootdir ?= $(prefix)/share
+docdir      ?= $(datarootdir)/doc
+
 include ../Makefile.deps
 
 YNL_GEN_ARG_ethtool:=--user-header linux/ethtool_netlink.h \
 	--exclude-op stats-get
 
 TOOL:=../pyynl/ynl_gen_c.py
+TOOL_RST:=../pyynl/ynl_gen_rst.py
 
 GENS_PATHS=$(shell grep -nrI --files-without-match \
 		'protocol: netlink' \
@@ -22,7 +28,11 @@ SRCS=$(patsubst %,%-user.c,${GENS})
 HDRS=$(patsubst %,%-user.h,${GENS})
 OBJS=$(patsubst %,%-user.o,${GENS})
 
-all: protos.a $(HDRS) $(SRCS) $(KHDRS) $(KSRCS) $(UAPI)
+SPECS_PATHS=$(wildcard ../../../../Documentation/netlink/specs/*.yaml)
+SPECS=$(patsubst ../../../../Documentation/netlink/specs/%.yaml,%,${SPECS_PATHS})
+RSTS=$(patsubst %,%.rst,${SPECS})
+
+all: protos.a $(HDRS) $(SRCS) $(KHDRS) $(KSRCS) $(UAPI) $(RSTS)
 
 protos.a: $(OBJS)
 	@echo -e "\tAR $@"
@@ -40,8 +50,12 @@ protos.a: $(OBJS)
 	@echo -e "\tCC $@"
 	@$(COMPILE.c) $(CFLAGS_$*) -o $@ $<
 
+%.rst: ../../../../Documentation/netlink/specs/%.yaml $(TOOL2)
+	@echo -e "\tGEN_RST $@"
+	@$(TOOL_RST) -o $@ -i $<
+
 clean:
-	rm -f *.o
+	rm -f *.o *.rst
 
 distclean: clean
 	rm -f *.c *.h *.a
@@ -49,5 +63,19 @@ distclean: clean
 regen:
 	@../ynl-regen.sh
 
-.PHONY: all clean distclean regen
+install-rsts:
+	@echo -e "\tINSTALL generated docs"
+	@$(INSTALL) -d $(DESTDIR)$(docdir)/ynl
+	@$(INSTALL) -m 0644 $(RSTS) $(DESTDIR)$(docdir)/ynl/
+
+install-specs:
+	@echo -e "\tINSTALL specs"
+	@$(INSTALL) -d $(DESTDIR)$(datarootdir)/ynl
+	@$(INSTALL) -m 0644 ../../../../Documentation/netlink/*.yaml $(DESTDIR)$(datarootdir)/ynl/
+	@$(INSTALL) -d $(DESTDIR)$(datarootdir)/ynl/specs
+	@$(INSTALL) -m 0644 ../../../../Documentation/netlink/specs/*.yaml $(DESTDIR)$(datarootdir)/ynl/specs/
+
+install: install-rsts install-specs
+
+.PHONY: all clean distclean regen install
 .DEFAULT_GOAL: all
-- 
2.43.0

[PATCH v2 5/5] tools: ynl: add main install target

[Jan Stancek]

This will install C library, specs, rsts and pyynl. The initial
structure is:

	$ mkdir /tmp/myroot
	$ make DESTDIR=/tmp/myroot install

	/usr
	/usr/lib64
	/usr/lib64/libynl.a
	/usr/lib/python3.XX/site-packages/pyynl/*
	/usr/lib/python3.XX/site-packages/pyynl-0.0.1.dist-info/*
	/usr/bin
	/usr/bin/ynl
	/usr/bin/ynl-ethtool
	/usr/bin/ynl-gen-c
	/usr/bin/ynl-gen-rst
	/usr/share
	/usr/share/doc
	/usr/share/doc/ynl
	/usr/share/doc/ynl/*.rst
	/usr/share/ynl
	/usr/share/ynl/genetlink-c.yaml
	/usr/share/ynl/genetlink-legacy.yaml
	/usr/share/ynl/genetlink.yaml
	/usr/share/ynl/netlink-raw.yaml
	/usr/share/ynl/specs
	/usr/share/ynl/specs/devlink.yaml
	/usr/share/ynl/specs/dpll.yaml
	/usr/share/ynl/specs/ethtool.yaml
	/usr/share/ynl/specs/fou.yaml
	/usr/share/ynl/specs/handshake.yaml
	/usr/share/ynl/specs/mptcp_pm.yaml
	/usr/share/ynl/specs/netdev.yaml
	/usr/share/ynl/specs/net_shaper.yaml
	/usr/share/ynl/specs/nfsd.yaml
	/usr/share/ynl/specs/nftables.yaml
	/usr/share/ynl/specs/nlctrl.yaml
	/usr/share/ynl/specs/ovs_datapath.yaml
	/usr/share/ynl/specs/ovs_flow.yaml
	/usr/share/ynl/specs/ovs_vport.yaml
	/usr/share/ynl/specs/rt_addr.yaml
	/usr/share/ynl/specs/rt_link.yaml
	/usr/share/ynl/specs/rt_neigh.yaml
	/usr/share/ynl/specs/rt_route.yaml
	/usr/share/ynl/specs/rt_rule.yaml
	/usr/share/ynl/specs/tcp_metrics.yaml
	/usr/share/ynl/specs/tc.yaml
	/usr/share/ynl/specs/team.yaml

Signed-off-by: Jan Stancek <jstancek@redhat.com>
---
 tools/net/ynl/Makefile | 23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)

diff --git a/tools/net/ynl/Makefile b/tools/net/ynl/Makefile
index 5268b91bf7ed..116a7fcfc540 100644
--- a/tools/net/ynl/Makefile
+++ b/tools/net/ynl/Makefile
@@ -1,5 +1,16 @@
 # SPDX-License-Identifier: GPL-2.0
 
+include ../../scripts/Makefile.arch
+
+INSTALL	?= install
+prefix  ?= /usr
+ifeq ($(LP64), 1)
+  libdir_relative = lib64
+else
+  libdir_relative = lib
+endif
+libdir  ?= $(prefix)/$(libdir_relative)
+
 SUBDIRS = lib generated samples
 
 all: $(SUBDIRS) libynl.a
@@ -23,5 +34,15 @@ clean distclean:
 	rm -f libynl.a
 	rm -rf pyynl/__pycache__
 	rm -rf pyynl/lib/__pycache__
+	rm -rf pyynl.egg-info
+	rm -rf build
+
+install: libynl.a
+	@echo -e "\tINSTALL libynl.a"
+	@$(INSTALL) -d $(DESTDIR)$(libdir)
+	@$(INSTALL) -m 0644 libynl.a $(DESTDIR)$(libdir)/libynl.a
+	@echo -e "\tINSTALL pyynl"
+	@pip install --prefix=$(DESTDIR)$(prefix) .
+	@make -C generated install
 
-.PHONY: all clean distclean $(SUBDIRS)
+.PHONY: all clean distclean install $(SUBDIRS)
-- 
2.43.0

Re: [PATCH v2 5/5] tools: ynl: add main install target

[Joe Damato]

On Mon, Dec 09, 2024 at 03:47:17PM +0100, Jan Stancek wrote:
> This will install C library, specs, rsts and pyynl. The initial
> structure is:
> 
> 	$ mkdir /tmp/myroot
> 	$ make DESTDIR=/tmp/myroot install
> 
> 	/usr
> 	/usr/lib64
> 	/usr/lib64/libynl.a

This is super useful thanks for doing this work. I could be missing
something, but it looks like the install target does not install the
generated C headers that user code can include at build time.

Am I reading that right? Is that intentional? I was thinking that it
would be really useful to have the headers installed, too.

> 	/usr/lib/python3.XX/site-packages/pyynl/*
> 	/usr/lib/python3.XX/site-packages/pyynl-0.0.1.dist-info/*
> 	/usr/bin
> 	/usr/bin/ynl
> 	/usr/bin/ynl-ethtool
> 	/usr/bin/ynl-gen-c
> 	/usr/bin/ynl-gen-rst
> 	/usr/share
> 	/usr/share/doc
> 	/usr/share/doc/ynl
> 	/usr/share/doc/ynl/*.rst
> 	/usr/share/ynl
> 	/usr/share/ynl/genetlink-c.yaml
> 	/usr/share/ynl/genetlink-legacy.yaml
> 	/usr/share/ynl/genetlink.yaml
> 	/usr/share/ynl/netlink-raw.yaml
> 	/usr/share/ynl/specs
> 	/usr/share/ynl/specs/devlink.yaml
> 	/usr/share/ynl/specs/dpll.yaml
> 	/usr/share/ynl/specs/ethtool.yaml
> 	/usr/share/ynl/specs/fou.yaml
> 	/usr/share/ynl/specs/handshake.yaml
> 	/usr/share/ynl/specs/mptcp_pm.yaml
> 	/usr/share/ynl/specs/netdev.yaml
> 	/usr/share/ynl/specs/net_shaper.yaml
> 	/usr/share/ynl/specs/nfsd.yaml
> 	/usr/share/ynl/specs/nftables.yaml
> 	/usr/share/ynl/specs/nlctrl.yaml
> 	/usr/share/ynl/specs/ovs_datapath.yaml
> 	/usr/share/ynl/specs/ovs_flow.yaml
> 	/usr/share/ynl/specs/ovs_vport.yaml
> 	/usr/share/ynl/specs/rt_addr.yaml
> 	/usr/share/ynl/specs/rt_link.yaml
> 	/usr/share/ynl/specs/rt_neigh.yaml
> 	/usr/share/ynl/specs/rt_route.yaml
> 	/usr/share/ynl/specs/rt_rule.yaml
> 	/usr/share/ynl/specs/tcp_metrics.yaml
> 	/usr/share/ynl/specs/tc.yaml
> 	/usr/share/ynl/specs/team.yaml
> 
> Signed-off-by: Jan Stancek <jstancek@redhat.com>

[...]

Re: [PATCH v2 5/5] tools: ynl: add main install target

[Jan Stancek]

On Mon, Dec 9, 2024 at 10:30 PM Joe Damato <jdamato@fastly.com> wrote:
>
> On Mon, Dec 09, 2024 at 03:47:17PM +0100, Jan Stancek wrote:
> > This will install C library, specs, rsts and pyynl. The initial
> > structure is:
> >
> >       $ mkdir /tmp/myroot
> >       $ make DESTDIR=/tmp/myroot install
> >
> >       /usr
> >       /usr/lib64
> >       /usr/lib64/libynl.a
>
> This is super useful thanks for doing this work. I could be missing
> something, but it looks like the install target does not install the
> generated C headers that user code can include at build time.
>
> Am I reading that right? Is that intentional? I was thinking that it
> would be really useful to have the headers installed, too.

It's not intentional, just noone asked for it yet. We can add those.
Would /usr/include/ynl/ work? Or do you/others have a different suggestion?

Regards,
Jan

>
> >       /usr/lib/python3.XX/site-packages/pyynl/*
> >       /usr/lib/python3.XX/site-packages/pyynl-0.0.1.dist-info/*
> >       /usr/bin
> >       /usr/bin/ynl
> >       /usr/bin/ynl-ethtool
> >       /usr/bin/ynl-gen-c
> >       /usr/bin/ynl-gen-rst
> >       /usr/share
> >       /usr/share/doc
> >       /usr/share/doc/ynl
> >       /usr/share/doc/ynl/*.rst
> >       /usr/share/ynl
> >       /usr/share/ynl/genetlink-c.yaml
> >       /usr/share/ynl/genetlink-legacy.yaml
> >       /usr/share/ynl/genetlink.yaml
> >       /usr/share/ynl/netlink-raw.yaml
> >       /usr/share/ynl/specs
> >       /usr/share/ynl/specs/devlink.yaml
> >       /usr/share/ynl/specs/dpll.yaml
> >       /usr/share/ynl/specs/ethtool.yaml
> >       /usr/share/ynl/specs/fou.yaml
> >       /usr/share/ynl/specs/handshake.yaml
> >       /usr/share/ynl/specs/mptcp_pm.yaml
> >       /usr/share/ynl/specs/netdev.yaml
> >       /usr/share/ynl/specs/net_shaper.yaml
> >       /usr/share/ynl/specs/nfsd.yaml
> >       /usr/share/ynl/specs/nftables.yaml
> >       /usr/share/ynl/specs/nlctrl.yaml
> >       /usr/share/ynl/specs/ovs_datapath.yaml
> >       /usr/share/ynl/specs/ovs_flow.yaml
> >       /usr/share/ynl/specs/ovs_vport.yaml
> >       /usr/share/ynl/specs/rt_addr.yaml
> >       /usr/share/ynl/specs/rt_link.yaml
> >       /usr/share/ynl/specs/rt_neigh.yaml
> >       /usr/share/ynl/specs/rt_route.yaml
> >       /usr/share/ynl/specs/rt_rule.yaml
> >       /usr/share/ynl/specs/tcp_metrics.yaml
> >       /usr/share/ynl/specs/tc.yaml
> >       /usr/share/ynl/specs/team.yaml
> >
> > Signed-off-by: Jan Stancek <jstancek@redhat.com>
>
> [...]
>

Re: [PATCH v2 5/5] tools: ynl: add main install target

[Joe Damato]

On Tue, Dec 10, 2024 at 02:56:05PM +0100, Jan Stancek wrote:
> On Mon, Dec 9, 2024 at 10:30 PM Joe Damato <jdamato@fastly.com> wrote:
> >
> > On Mon, Dec 09, 2024 at 03:47:17PM +0100, Jan Stancek wrote:
> > > This will install C library, specs, rsts and pyynl. The initial
> > > structure is:
> > >
> > >       $ mkdir /tmp/myroot
> > >       $ make DESTDIR=/tmp/myroot install
> > >
> > >       /usr
> > >       /usr/lib64
> > >       /usr/lib64/libynl.a
> >
> > This is super useful thanks for doing this work. I could be missing
> > something, but it looks like the install target does not install the
> > generated C headers that user code can include at build time.
> >
> > Am I reading that right? Is that intentional? I was thinking that it
> > would be really useful to have the headers installed, too.
> 
> It's not intentional, just noone asked for it yet. We can add those.
> Would /usr/include/ynl/ work? Or do you/others have a different suggestion?

/usr/include/ynl sounds good to me, but happy to see if other folks
have suggestions.

Thanks for being open to adding this; it'll make developing user
apps with libynl much easier.

Re: [PATCH v2 2/5] tools: ynl: provide symlinks to user-facing scripts for compatibility

[Jakub Kicinski]

On Mon,  9 Dec 2024 15:47:14 +0100 Jan Stancek wrote:
> For backwards compatibility provide also symlinks from original location
> of user facing scripts.

Did someone ask for this? Does everything work without the symlinks?
If the answers are "no", "yes" then let's try without this patch.
In tree users should be able to adjust.

Re: [PATCH v2 4/5] tools: ynl: add install target for specs and docs

[Jakub Kicinski]

On Mon,  9 Dec 2024 15:47:16 +0100 Jan Stancek wrote:
> +SPECS_PATHS=$(wildcard ../../../../Documentation/netlink/specs/*.yaml)

Maybe factor out:

SPECS_DIR := ../../../../Documentation/netlink/specs

? It's pretty long and we repeat it all over the place.

> +SPECS=$(patsubst ../../../../Documentation/netlink/specs/%.yaml,%,${SPECS_PATHS})
> +RSTS=$(patsubst %,%.rst,${SPECS})
> +
> +all: protos.a $(HDRS) $(SRCS) $(KHDRS) $(KSRCS) $(UAPI) $(RSTS)
>  
>  protos.a: $(OBJS)
>  	@echo -e "\tAR $@"
> @@ -40,8 +50,12 @@ protos.a: $(OBJS)
>  	@echo -e "\tCC $@"
>  	@$(COMPILE.c) $(CFLAGS_$*) -o $@ $<
>  
> +%.rst: ../../../../Documentation/netlink/specs/%.yaml $(TOOL2)

TOOL2 -> TOOL_RST ?

> +	@echo -e "\tGEN_RST $@"
> +	@$(TOOL_RST) -o $@ -i $<
> +
>  clean:
> -	rm -f *.o
> +	rm -f *.o *.rst

No strong preference but I'd count .rst as final artifacts so I'd clean
them up in distclean target only, not the clean target. The distinction
itself may be a local custom..

>  distclean: clean
>  	rm -f *.c *.h *.a
> @@ -49,5 +63,19 @@ distclean: clean
>  regen:
>  	@../ynl-regen.sh
>  
> -.PHONY: all clean distclean regen
> +install-rsts:
> +	@echo -e "\tINSTALL generated docs"
> +	@$(INSTALL) -d $(DESTDIR)$(docdir)/ynl
> +	@$(INSTALL) -m 0644 $(RSTS) $(DESTDIR)$(docdir)/ynl/
> +
> +install-specs:
> +	@echo -e "\tINSTALL specs"
> +	@$(INSTALL) -d $(DESTDIR)$(datarootdir)/ynl
> +	@$(INSTALL) -m 0644 ../../../../Documentation/netlink/*.yaml $(DESTDIR)$(datarootdir)/ynl/
> +	@$(INSTALL) -d $(DESTDIR)$(datarootdir)/ynl/specs
> +	@$(INSTALL) -m 0644 ../../../../Documentation/netlink/specs/*.yaml $(DESTDIR)$(datarootdir)/ynl/specs/
> +
> +install: install-rsts install-specs
> +
> +.PHONY: all clean distclean regen install

I think .PHONY needs install-rsts install-specs, too?

>  .DEFAULT_GOAL: all
-- 
pw-bot: cr

Re: [PATCH v2 2/5] tools: ynl: provide symlinks to user-facing scripts for compatibility

[Donald Hunter]

On Wed, 11 Dec 2024 at 03:26, Jakub Kicinski <kuba@kernel.org> wrote:
>
> On Mon,  9 Dec 2024 15:47:14 +0100 Jan Stancek wrote:
> > For backwards compatibility provide also symlinks from original location
> > of user facing scripts.
>
> Did someone ask for this? Does everything work without the symlinks?
> If the answers are "no", "yes" then let's try without this patch.
> In tree users should be able to adjust.

I asked for the symlinks for cli.py and ethtool.py to avoid surprising
people when they move. The ynl-gen- scripts are primarily used in-tree
via Makefiles so I didn't think they should be symlinked. Happy to go
with your suggestion to drop this if you'd prefer not to have any
symlinks.

Thanks,
Donald

Re: [PATCH v2 2/5] tools: ynl: provide symlinks to user-facing scripts for compatibility

[Jan Stancek]

On Wed, Dec 11, 2024 at 10:21 AM Donald Hunter <donald.hunter@gmail.com> wrote:
>
> On Wed, 11 Dec 2024 at 03:26, Jakub Kicinski <kuba@kernel.org> wrote:
> >
> > On Mon,  9 Dec 2024 15:47:14 +0100 Jan Stancek wrote:
> > > For backwards compatibility provide also symlinks from original location
> > > of user facing scripts.
> >
> > Did someone ask for this? Does everything work without the symlinks?
> > If the answers are "no", "yes" then let's try without this patch.
> > In tree users should be able to adjust.
>
> I asked for the symlinks for cli.py and ethtool.py to avoid surprising
> people when they move. The ynl-gen- scripts are primarily used in-tree
> via Makefiles so I didn't think they should be symlinked. Happy to go
> with your suggestion to drop this if you'd prefer not to have any
> symlinks.

I'll drop them, we can always add them later in case someone
_really_ needs original script locations.

>
> Thanks,
> Donald
>

Re: [PATCH v2 2/5] tools: ynl: provide symlinks to user-facing scripts for compatibility

[Jakub Kicinski]

On Wed, 11 Dec 2024 13:42:28 +0100 Jan Stancek wrote:
> > > Did someone ask for this? Does everything work without the symlinks?
> > > If the answers are "no", "yes" then let's try without this patch.
> > > In tree users should be able to adjust.  
> >
> > I asked for the symlinks for cli.py and ethtool.py to avoid surprising
> > people when they move. The ynl-gen- scripts are primarily used in-tree
> > via Makefiles so I didn't think they should be symlinked. Happy to go
> > with your suggestion to drop this if you'd prefer not to have any
> > symlinks.  
> 
> I'll drop them, we can always add them later in case someone
> _really_ needs original script locations.

FWIW that's my thinking, too.