[PATCH net-next] tools: ynl: try to avoid the very slow YAML loader

[PATCH net-next] tools: ynl: try to avoid the very slow YAML loader

[Jakub Kicinski]

Turns out Python YAML defaults to a pure Python loader for YAML
files which is a lot slower than the C loader (using libyaml).
Try to use the C one whenever possible.

The avg time to run:
  $ tools/net/ynl/pyynl/cli.py --family tc --no-schema
drops from 300+ ms to 115 ms with this change (40 samples).

We could drop the load time further to 85 ms if we "compiled"
the specs to JSON. Slightly tricky parts are that we don't
currently install the specs at all on make install, so it's
unclear where to put the conversion. Also JSON has questionable
support for comments and we need an SPDX line.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
---
CC: donald.hunter@gmail.com
---
 tools/net/ynl/pyynl/lib/nlspec.py | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/tools/net/ynl/pyynl/lib/nlspec.py b/tools/net/ynl/pyynl/lib/nlspec.py
index fcffeb5b7ba3..0469a0e270d0 100644
--- a/tools/net/ynl/pyynl/lib/nlspec.py
+++ b/tools/net/ynl/pyynl/lib/nlspec.py
@@ -439,6 +439,11 @@ import yaml as pyyaml
     # To be loaded dynamically as needed
     jsonschema = None
 
+    try:
+        _yaml_loader = pyyaml.CSafeLoader
+    except AttributeError:
+        _yaml_loader = pyyaml.SafeLoader
+
     def __init__(self, spec_path, schema_path=None, exclude_ops=None):
         with open(spec_path, "r", encoding='utf-8') as stream:
             prefix = '# SPDX-License-Identifier: '
@@ -448,7 +453,7 @@ import yaml as pyyaml
             self.license = first[len(prefix):]
 
             stream.seek(0)
-            spec = pyyaml.safe_load(stream)
+            spec = pyyaml.load(stream, Loader=self._yaml_loader)
 
         self.fixed_header = None
         self._resolution_list = []
@@ -464,7 +469,7 @@ import yaml as pyyaml
             schema_path = os.path.dirname(os.path.dirname(spec_path)) + f'/{self.proto}.yaml'
         if schema_path:
             with open(schema_path, "r", encoding='utf-8') as stream:
-                schema = pyyaml.safe_load(stream)
+                schema = pyyaml.load(stream, Loader=self._yaml_loader)
 
             if SpecFamily.jsonschema is None:
                 SpecFamily.jsonschema = importlib.import_module("jsonschema")
-- 
2.54.0

Re: [PATCH net-next] tools: ynl: try to avoid the very slow YAML loader

[Jacob Keller]

On 6/3/2026 2:08 PM, Jakub Kicinski wrote:
> Turns out Python YAML defaults to a pure Python loader for YAML
> files which is a lot slower than the C loader (using libyaml).
> Try to use the C one whenever possible.
> 
> The avg time to run:
>   $ tools/net/ynl/pyynl/cli.py --family tc --no-schema
> drops from 300+ ms to 115 ms with this change (40 samples).
> 
> We could drop the load time further to 85 ms if we "compiled"
> the specs to JSON. Slightly tricky parts are that we don't
> currently install the specs at all on make install, so it's
> unclear where to put the conversion. Also JSON has questionable
> support for comments and we need an SPDX line.
> > Signed-off-by: Jakub Kicinski <kuba@kernel.org>
> ---
> CC: donald.hunter@gmail.com
> ---
>  tools/net/ynl/pyynl/lib/nlspec.py | 9 +++++++--
>  1 file changed, 7 insertions(+), 2 deletions(-)
> 
> diff --git a/tools/net/ynl/pyynl/lib/nlspec.py b/tools/net/ynl/pyynl/lib/nlspec.py
> index fcffeb5b7ba3..0469a0e270d0 100644
> --- a/tools/net/ynl/pyynl/lib/nlspec.py
> +++ b/tools/net/ynl/pyynl/lib/nlspec.py
> @@ -439,6 +439,11 @@ import yaml as pyyaml
>      # To be loaded dynamically as needed
>      jsonschema = None
>  
> +    try:
> +        _yaml_loader = pyyaml.CSafeLoader
> +    except AttributeError:
> +        _yaml_loader = pyyaml.SafeLoader
> +
>      def __init__(self, spec_path, schema_path=None, exclude_ops=None):
>          with open(spec_path, "r", encoding='utf-8') as stream:
>              prefix = '# SPDX-License-Identifier: '
> @@ -448,7 +453,7 @@ import yaml as pyyaml
>              self.license = first[len(prefix):]
>  
>              stream.seek(0)
> -            spec = pyyaml.safe_load(stream)
> +            spec = pyyaml.load(stream, Loader=self._yaml_loader)
>  
Hmm. I was a bit confused at first by the switch from safe_load to
load.. but we're passing Loader as _yaml_loader which will either be
CSafeLoader *or* the default SafeLoader, so we'll get the appropriate
loader equivalent to what safe_load would have done, so there's no
change. Ok

Reviewed-by: Jacob Keller <jacob.e.keller@intel.com>

>          self.fixed_header = None
>          self._resolution_list = []
> @@ -464,7 +469,7 @@ import yaml as pyyaml
>              schema_path = os.path.dirname(os.path.dirname(spec_path)) + f'/{self.proto}.yaml'
>          if schema_path:
>              with open(schema_path, "r", encoding='utf-8') as stream:
> -                schema = pyyaml.safe_load(stream)
> +                schema = pyyaml.load(stream, Loader=self._yaml_loader)
>  
>              if SpecFamily.jsonschema is None:
>                  SpecFamily.jsonschema = importlib.import_module("jsonschema")

Re: [PATCH net-next] tools: ynl: try to avoid the very slow YAML loader

[Jakub Kicinski]

On Wed, 3 Jun 2026 15:08:46 -0700 Jacob Keller wrote:
> Hmm. I was a bit confused at first by the switch from safe_load to
> load.. but we're passing Loader as _yaml_loader which will either be
> CSafeLoader *or* the default SafeLoader, so we'll get the appropriate
> loader equivalent to what safe_load would have done, so there's no
> change. Ok

Maybe I should have mentioned this in the commit msg. I was also super
confused by these APIs but IDK how much is this me not knowing Python
and how much it's special. AFAIU basically:

somewhere in pyyaml...

	def safe_load(file):
		return load(file, Loader=SafeLoader)

so safe_load() is just a "shorthand" for using SafeLoader, which unlike
load()s default loader doesn't allow constructing real/binary Python
objects ?

Why it doesn't default to the C one is beyond my understanding.

Re: [PATCH net-next] tools: ynl: try to avoid the very slow YAML loader

[Jacob Keller]

On 6/3/2026 4:35 PM, Jakub Kicinski wrote:
> On Wed, 3 Jun 2026 15:08:46 -0700 Jacob Keller wrote:
>> Hmm. I was a bit confused at first by the switch from safe_load to
>> load.. but we're passing Loader as _yaml_loader which will either be
>> CSafeLoader *or* the default SafeLoader, so we'll get the appropriate
>> loader equivalent to what safe_load would have done, so there's no
>> change. Ok
> 
> Maybe I should have mentioned this in the commit msg. I was also super
> confused by these APIs but IDK how much is this me not knowing Python
> and how much it's special. AFAIU basically:
> 

This is a pyyaml issue not a generic python one I think.

> somewhere in pyyaml...
> 
> 	def safe_load(file):
> 		return load(file, Loader=SafeLoader)
> 
> so safe_load() is just a "shorthand" for using SafeLoader, which unlike
> load()s default loader doesn't allow constructing real/binary Python
> objects ?

load() by default uses the generic Loader that supports the full spec
and because of that is "unsafe":

https://github.com/yaml/pyyaml/wiki/PyYAML-yaml.load(input)-Deprecation

Basically, loading the full YAML spec on untrusted input is unsafe since
it allows arbitrary execution.
> 
> Why it doesn't default to the C one is beyond my understanding.

Right. I'm not super familiar here as to how it ends up not defaulting
to CSafeLoader, but probably its because its not always available.

Re: [PATCH net-next] tools: ynl: try to avoid the very slow YAML loader

[Nicolai Buchwitz]

On 4.6.2026 02:17, Jacob Keller wrote:
> On 6/3/2026 4:35 PM, Jakub Kicinski wrote:
>> On Wed, 3 Jun 2026 15:08:46 -0700 Jacob Keller wrote:
>>> Hmm. I was a bit confused at first by the switch from safe_load to
>>> load.. but we're passing Loader as _yaml_loader which will either be
>>> CSafeLoader *or* the default SafeLoader, so we'll get the appropriate
>>> loader equivalent to what safe_load would have done, so there's no
>>> change. Ok
>> 
>> Maybe I should have mentioned this in the commit msg. I was also super
>> confused by these APIs but IDK how much is this me not knowing Python
>> and how much it's special. AFAIU basically:
>> 
> 
> This is a pyyaml issue not a generic python one I think.
> 
>> somewhere in pyyaml...
>> 
>> 	def safe_load(file):
>> 		return load(file, Loader=SafeLoader)
>> 
>> so safe_load() is just a "shorthand" for using SafeLoader, which 
>> unlike
>> load()s default loader doesn't allow constructing real/binary Python
>> objects ?
> 
> load() by default uses the generic Loader that supports the full spec
> and because of that is "unsafe":
> 
> https://github.com/yaml/pyyaml/wiki/PyYAML-yaml.load(input)-Deprecation
> 
> Basically, loading the full YAML spec on untrusted input is unsafe 
> since
> it allows arbitrary execution.
>> 
>> Why it doesn't default to the C one is beyond my understanding.
> 
> Right. I'm not super familiar here as to how it ends up not defaulting
> to CSafeLoader, but probably its because its not always available.

That's correct. CSafeLoader is only available if libyaml-dev was present 
when
pyyaml was built. Auto-enabling the libyaml bindings was requested a few 
years
ago, but it stalled due to internals [1]. It would also require 
reworking
safe_load() and others, which are atm hardcoded. Hence the usual 
try/except
import pattern.

Anyway, the changes match the usual patterns.

Reviewed-by: Nicolai Buchwitz <nb@tipi-net.de>

[1] https://github.com/yaml/pyyaml/issues/437

Regards
Nicolai

Re: [PATCH net-next] tools: ynl: try to avoid the very slow YAML loader

[Donald Hunter]

Jakub Kicinski <kuba@kernel.org> writes:

> Turns out Python YAML defaults to a pure Python loader for YAML
> files which is a lot slower than the C loader (using libyaml).
> Try to use the C one whenever possible.
>
> The avg time to run:
>   $ tools/net/ynl/pyynl/cli.py --family tc --no-schema
> drops from 300+ ms to 115 ms with this change (40 samples).
>
> We could drop the load time further to 85 ms if we "compiled"
> the specs to JSON. Slightly tricky parts are that we don't
> currently install the specs at all on make install, so it's
> unclear where to put the conversion. Also JSON has questionable
> support for comments and we need an SPDX line.
>
> Signed-off-by: Jakub Kicinski <kuba@kernel.org>
> ---
> CC: donald.hunter@gmail.com

Reviewed-by: Donald Hunter <donald.hunter@gmail.com>