* [PATCH 0/1] Improving setuptools performance in Python
@ 2020-06-23 22:58 Trevor Gamblin
2020-06-23 22:58 ` [PATCH] python3-setuptools: patch entrypoints for faster initialization Trevor Gamblin
2020-06-23 23:02 ` ✗ patchtest: failure for " Patchwork
0 siblings, 2 replies; 3+ messages in thread
From: Trevor Gamblin @ 2020-06-23 22:58 UTC (permalink / raw)
To: openembedded-core
As explained in https://github.com/pypa/setuptools/issues/510, the
pkg_resources module in setuptools is very slow to initialize scripts
due to how it scans the Python path for each module that would import
pkg_resources. My understanding of upstream Python's response is that
as of 3.7 and later, importlib.resources is available for replacing
this functionality in new scripts, but existing code that uses
pkg_resources will still have to be re-written to use it (or something
like python3-fastentrypoints.
I've done some performance testing with this patch using the following,
very simple steps:
1) Add "IMAGE_INSTALL_append = " python3-setuptools" to local.conf;
2) bitbake core-image-full-cmdline
3) mkdir -p test/minimal;
4) In test/minimal/__init__.py, add:
def main():
print('hello world')
if __name__ == '__main__':
main()
5) In test/setup.py:
from setuptools import setup, find_packages
setup(
name='minimal',
version='0.1',
python_requires='>=3.4',
packages=['minimal'],
entry_points={ 'console_scripts': [ 'minimal = minimal:main' ] },
)
6) Run "python3 setup.py install"
7) Run "time /usr/bin/minimal"
8) Note the drastic differences in run time, e.g. with
core-image-full-cmdline:
core-image-full-cmdline, WITHOUT setuptools ScriptWriter patch:
root@qemux86-64:~# time /usr/bin/minimal
hello world
real 0m0.198s
user 0m0.174s
sys 0m0.023s
core-image-full-cmdline, WITH setuptools ScriptWriter patch:
root@qemux86-64:~# time /usr/bin/minimal
hello world
real 0m0.034s
user 0m0.024s
sys 0m0.010s
I've also tried to ensure that this has no adverse effects on the larger
python ecosystem in Yocto, by running ptests on all modules in
meta-python that have ptests available and use setuptools, with this
patch installed. I saw no failures related to setuptools, although it is
still possible that this could have unforeseen consequences on other
modules.
Finally, some (truncated) profiling comparisons to emphasize the gains
in performance, using the same example module before and after the patch.
There are three orders of magnitude difference in the number of calls
required, including much more recursion. What is most apparent from
these profiling results is how much more time the unpatched version
spends parsing entry points when that effort isn't required to achieve
the same end. As can be seen from the simple example used here, the
time savings at initialization are significant enough even for tiny
scripts that this fix should be incorporated in the project at the
setuptools level, rather than requiring manual application, or
modification of existing scripts on the user's end.
WITHOUT PATCH:
134641 function calls (130543 primitive calls) in 0.723 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
88/1 0.002 0.000 0.723 0.723 {built-in method builtins.exec}
1 0.000 0.000 0.723 0.723 minimal:3(<module>)
100/2 0.002 0.000 0.704 0.352 <frozen importlib._bootstrap>:986(_find_and_load)
100/2 0.002 0.000 0.703 0.352 <frozen importlib._bootstrap>:956(_find_and_load_unlocked)
93/2 0.002 0.000 0.700 0.350 <frozen importlib._bootstrap>:650(_load_unlocked)
67/1 0.001 0.000 0.699 0.699 <frozen importlib._bootstrap_external>:777(exec_module)
115/1 0.000 0.000 0.698 0.698 <frozen importlib._bootstrap>:211(_call_with_frames_removed)
1 0.000 0.000 0.698 0.698 __init__.py:2(<module>)
30/19 0.000 0.000 0.445 0.023 {built-in method builtins.__import__}
82 0.002 0.000 0.372 0.005 re.py:289(_compile)
75 0.000 0.000 0.369 0.005 re.py:250(compile)
73 0.002 0.000 0.367 0.005 sre_compile.py:759(compile)
1 0.000 0.000 0.262 0.262 requirements.py:4(<module>)
73 0.001 0.000 0.259 0.004 sre_parse.py:937(parse)
308/73 0.007 0.000 0.255 0.003 sre_parse.py:435(_parse_sub)
457/77 0.072 0.000 0.253 0.003 sre_parse.py:493(_parse)
283/280 0.009 0.000 0.222 0.001 {built-in method builtins.__build_class__}
9/7 0.000 0.000 0.159 0.023 <frozen importlib._bootstrap>:613(_load_backward_compatible)
6 0.000 0.000 0.159 0.026 __init__.py:35(load_module)
1 0.001 0.001 0.145 0.145 pyparsing.py:26(<module>)
26 0.001 0.000 0.137 0.005 pyparsing.py:2779(__init__)
73 0.001 0.000 0.104 0.001 sre_compile.py:598(_code)
1 0.000 0.000 0.097 0.097 parser.py:5(<module>)
1 0.000 0.000 0.095 0.095 feedparser.py:5(<module>)
1 0.000 0.000 0.095 0.095 specifiers.py:4(<module>)
770/73 0.032 0.000 0.091 0.001 sre_compile.py:71(_compile)
1 0.000 0.000 0.086 0.086 _policybase.py:1(<module>)
11488 0.049 0.000 0.086 0.000 sre_parse.py:254(get)
1 0.000 0.000 0.083 0.083 specifiers.py:275(Specifier)
1 0.000 0.000 0.073 0.073 pyparsing.py:5399(pyparsing_common)
98 0.003 0.000 0.063 0.001 <frozen importlib._bootstrap>:890(_find_spec)
1 0.000 0.000 0.058 0.058 utils.py:5(<module>)
WITH PATCH:
300 function calls (299 primitive calls) in 0.003 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
2/1 0.000 0.000 0.003 0.003 {built-in method builtins.exec}
1 0.000 0.000 0.003 0.003 minimal:2(<module>)
1 0.000 0.000 0.003 0.003 <frozen importlib._bootstrap>:986(_find_and_load)
1 0.000 0.000 0.002 0.002 <frozen importlib._bootstrap>:956(_find_and_load_unlocked)
1 0.000 0.000 0.002 0.002 <frozen importlib._bootstrap>:890(_find_spec)
1 0.000 0.000 0.002 0.002 <frozen importlib._bootstrap_external>:1334(find_spec)
1 0.000 0.000 0.002 0.002 <frozen importlib._bootstrap_external>:1302(_get_spec)
2 0.000 0.000 0.001 0.001 <frozen importlib._bootstrap_external>:1431(find_spec)
1 0.000 0.000 0.001 0.001 <frozen importlib._bootstrap>:650(_load_unlocked)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1479(_fill_cache)
18 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:62(_path_join)
1 0.000 0.000 0.000 0.000 {built-in method posix.listdir}
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:777(exec_module)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:849(get_code)
11 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:90(_path_is_mode_type)
9 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:99(_path_isfile)
15 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:80(_path_stat)
18 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:64(<listcomp>)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:549(module_from_spec)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:477(_init_module_attrs)
2 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1265(_path_importer_cache)
15 0.000 0.000 0.000 0.000 {built-in method posix.stat}
2 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:294(cache_from_source)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1252(_path_hooks)
2 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:376(cached)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:424(_get_cached)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1520(path_hook_for_FileFinder)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:969(get_data)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:147(__enter__)
38 0.000 0.000 0.000 0.000 {method 'rstrip' of 'str' objects}
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1394(__init__)
2 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:104(_path_isdir)
Trevor Gamblin (1):
python3-setuptools: patch entrypoints for performance boost
.../python/python-setuptools.inc | 4 +-
...nt-usr-bin-wrappers-signoff-included.patch | 60 +++++++++++++++++++
2 files changed, 63 insertions(+), 1 deletion(-)
create mode 100644 meta/recipes-devtools/python/python3-setuptools/0001-ScriptWriter-create-more-efficient-usr-bin-wrappers-signoff-included.patch
--
2.24.1
^ permalink raw reply [flat|nested] 3+ messages in thread* [PATCH] python3-setuptools: patch entrypoints for faster initialization 2020-06-23 22:58 [PATCH 0/1] Improving setuptools performance in Python Trevor Gamblin @ 2020-06-23 22:58 ` Trevor Gamblin 2020-06-23 23:02 ` ✗ patchtest: failure for " Patchwork 1 sibling, 0 replies; 3+ messages in thread From: Trevor Gamblin @ 2020-06-23 22:58 UTC (permalink / raw) To: openembedded-core setuptools' pkg_resources module has major performance issues with how it loads entry points (e.g. the console_script entry point, which sets up a module as a command-line executable), leading even the simplest "hello world" scripts to take on the order of 150ms to run if pkg_resources is incorporated. This is prohibitive for code that needs to run quickly, and so we patch setuptools to reduce this time. As of Python 3.7, importlib.resources is available and intended to replace much of the functionality that causes this sluggishness, but since many projects still utilize the legacy setuptools modules, a patch is still required. Note that python3-fastentrypoints (which is available in the meta-virtualization layer) is also intended to help alleviate the problem, but since it must be added to existing projects it has the same disadvantage as resorting to importlib.resources, requiring manual additions to existing code to see the performance gains. The intent here is to patch easy_install to load module entry points directly with the installed setuptools, rather than importing pkg_resources and having it search out the entry points itself. This leads to a drastic performance improvement - the changes in this patch have been shown to result in load time ~6-8x lower, depending on the complexity of the code it is tested with. A simple "hello world" example on core-image-full-cmdline gave these results with and without the patch: core-image-full-cmdline, without setuptools ScriptWriter patch: root@qemux86-64:~# time /usr/bin/minimal hello world real 0m0.198s user 0m0.174s sys 0m0.023s core-image-full-cmdline, with setuptools ScriptWriter patch: root@qemux86-64:~# time /usr/bin/minimal hello world real 0m0.034s user 0m0.024s sys 0m0.010s More details on the pkg_resources issue are available at: https://github.com/pypa/setuptools/issues/510 Signed-off-by: Trevor Gamblin <trevor.gamblin@windriver.com> --- .../python/python-setuptools.inc | 4 +- ...nt-usr-bin-wrappers-signoff-included.patch | 60 +++++++++++++++++++ 2 files changed, 63 insertions(+), 1 deletion(-) create mode 100644 meta/recipes-devtools/python/python3-setuptools/0001-ScriptWriter-create-more-efficient-usr-bin-wrappers-signoff-included.patch diff --git a/meta/recipes-devtools/python/python-setuptools.inc b/meta/recipes-devtools/python/python-setuptools.inc index 9322ca9a05..3222de7079 100644 --- a/meta/recipes-devtools/python/python-setuptools.inc +++ b/meta/recipes-devtools/python/python-setuptools.inc @@ -10,7 +10,9 @@ inherit pypi SRC_URI_append_class-native = " file://0001-conditionally-do-not-fetch-code-by-easy_install.patch" -SRC_URI += "file://0001-change-shebang-to-python3.patch" +SRC_URI += "file://0001-change-shebang-to-python3.patch \ + file://0001-ScriptWriter-create-more-efficient-usr-bin-wrappers-signoff-included.patch \ + " SRC_URI[md5sum] = "6e9de90b242fdd60ef59f497424ce13a" SRC_URI[sha256sum] = "145fa62b9d7bb544fce16e9b5a9bf4ab2032d2f758b7cd674af09a92736aff74" diff --git a/meta/recipes-devtools/python/python3-setuptools/0001-ScriptWriter-create-more-efficient-usr-bin-wrappers-signoff-included.patch b/meta/recipes-devtools/python/python3-setuptools/0001-ScriptWriter-create-more-efficient-usr-bin-wrappers-signoff-included.patch new file mode 100644 index 0000000000..81b98a749a --- /dev/null +++ b/meta/recipes-devtools/python/python3-setuptools/0001-ScriptWriter-create-more-efficient-usr-bin-wrappers-signoff-included.patch @@ -0,0 +1,60 @@ +From aae8cd3de3f289cea3db01212579913c925191e8 Mon Sep 17 00:00:00 2001 +From: Lauri Tirkkonen <lauri.tirkkonen.ext@nokia.com> +Date: Thu, 26 Mar 2020 14:24:25 +0000 +Subject: [PATCH] ScriptWriter: create more efficient /usr/bin wrappers + +Upstream setuptools writes scripts to /usr/bin that do insanely much +stuff at runtime. https://github.com/pypa/setuptools/issues/510 + +Since the script entry points are already known at build time, we can +just write those directly into the /usr/bin wrapper, avoiding the +expensive 'pkg_resources' import at runtime. The idea is from +https://github.com/ninjaaron/fast-entry_points but patched directly into +the native build of setuptools here, so that all Python modules under +bitbake automatically use it without needing additional build time +dependencies. + +Signed-off-by: Lauri Tirkkonen <lauri.tirkkonen.ext@nokia.com> +Signed-off-by: Trevor Gamblin <trevor.gamblin@windriver.com> +--- + setuptools/command/easy_install.py | 14 ++++++-------- + 1 file changed, 6 insertions(+), 8 deletions(-) + +diff --git a/setuptools/command/easy_install.py b/setuptools/command/easy_install.py +index 8fba7b41..03a72714 100755 +--- a/setuptools/command/easy_install.py ++++ b/setuptools/command/easy_install.py +@@ -2023,17 +2023,12 @@ class ScriptWriter(object): + """ + + template = textwrap.dedent(r""" +- # EASY-INSTALL-ENTRY-SCRIPT: %(spec)r,%(group)r,%(name)r +- __requires__ = %(spec)r +- import re + import sys +- from pkg_resources import load_entry_point ++ ++ from %(module)s import %(ep0)s + + if __name__ == '__main__': +- sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) +- sys.exit( +- load_entry_point(%(spec)r, %(group)r, %(name)r)() +- ) ++ sys.exit(%(entrypoint)s()) + """).lstrip() + + command_spec_class = CommandSpec +@@ -2068,6 +2063,9 @@ class ScriptWriter(object): + for type_ in 'console', 'gui': + group = type_ + '_scripts' + for name, ep in dist.get_entry_map(group).items(): ++ module = ep.module_name ++ ep0 = ep.attrs[0] ++ entrypoint = '.'.join(ep.attrs) + cls._ensure_safe_name(name) + script_text = cls.template % locals() + args = cls._get_script_args(type_, name, header, script_text) +-- +2.24.1 + -- 2.26.2 ^ permalink raw reply related [flat|nested] 3+ messages in thread
* ✗ patchtest: failure for python3-setuptools: patch entrypoints for faster initialization 2020-06-23 22:58 [PATCH 0/1] Improving setuptools performance in Python Trevor Gamblin 2020-06-23 22:58 ` [PATCH] python3-setuptools: patch entrypoints for faster initialization Trevor Gamblin @ 2020-06-23 23:02 ` Patchwork 1 sibling, 0 replies; 3+ messages in thread From: Patchwork @ 2020-06-23 23:02 UTC (permalink / raw) To: Trevor Gamblin; +Cc: openembedded-core == Series Details == Series: python3-setuptools: patch entrypoints for faster initialization Revision: 1 URL : https://patchwork.openembedded.org/series/24809/ State : failure == Summary == Thank you for submitting this patch series to OpenEmbedded Core. This is an automated response. Several tests have been executed on the proposed series by patchtest resulting in the following failures: * Issue Added patch file is missing Upstream-Status in the header [test_upstream_status_presence_format] Suggested fix Add Upstream-Status: <Valid status> to the header of meta/recipes-devtools/python/python3-setuptools/0001-ScriptWriter-create-more-efficient-usr-bin-wrappers-signoff-included.patch Standard format Upstream-Status: <Valid status> Valid status Pending, Accepted, Backport, Denied, Inappropriate [reason], Submitted [where] If you believe any of these test results are incorrect, please reply to the mailing list (openembedded-core@lists.openembedded.org) raising your concerns. Otherwise we would appreciate you correcting the issues and submitting a new version of the patchset if applicable. Please ensure you add/increment the version number when sending the new version (i.e. [PATCH] -> [PATCH v2] -> [PATCH v3] -> ...). --- Guidelines: https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest Test suite: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe ^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2020-06-23 23:02 UTC | newest] Thread overview: 3+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2020-06-23 22:58 [PATCH 0/1] Improving setuptools performance in Python Trevor Gamblin 2020-06-23 22:58 ` [PATCH] python3-setuptools: patch entrypoints for faster initialization Trevor Gamblin 2020-06-23 23:02 ` ✗ patchtest: failure for " Patchwork
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox