[PATCH V2] setuptools3-base.bbclass: override default subprocess timeout

[PATCH V2] setuptools3-base.bbclass: override default subprocess timeout

[Hongxu Jia]

The environment variable SETUPTOOLS_SCM_SUBPROCESS_TIMEOUT allows to override
the subprocess timeout. The default is 40 seconds and should work for most
needs.[1] However, it was not enough while using git shallow tarball and starting
multiple Yocto world builds in one host.

|   File "tmp/work/x86_64-linux/python3-scancode-native/32.1.0/recipe-sysroot-
native/usr/lib/python3.13/subprocess.py", line 1263, in _check_timeout
|     raise TimeoutExpired(
|     ...<2 lines>...
|             stderr=b''.join(stderr_seq) if stderr_seq else None)
| subprocess.TimeoutExpired: Command '['git', '--git-dir', 'tmp/work/x86_64-
linux/python3-scancode-native/32.1.0/git/.git', 'status', '--porcelain',
'--untracked-files=no']' timed out after 40 seconds

Explicitly set variable SETUPTOOLS_SCM_SUBPROCESS_TIMEOUT to 600s in bbclass,
and we could override it in local.conf

[1] https://github.com/pypa/setuptools-scm/blob/main/docs/overrides.md

Issue: LINCD-21838
(LOCAL REV: NOT UPSTREAM) -- Sent to oe-core on 20250217

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/classes-recipe/setuptools3-base.bbclass | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/classes-recipe/setuptools3-base.bbclass b/meta/classes-recipe/setuptools3-base.bbclass
index 27af6abc58..190d9e6e3a 100644
--- a/meta/classes-recipe/setuptools3-base.bbclass
+++ b/meta/classes-recipe/setuptools3-base.bbclass
@@ -23,6 +23,10 @@ export CCSHARED  = "-fPIC -DPIC"
 # the python executable
 export LINKFORSHARED = "${SECURITY_CFLAGS} -Xlinker -export-dynamic"
 
+# The environment variable SETUPTOOLS_SCM_SUBPROCESS_TIMEOUT allows
+# to override the subprocess timeout.
+export SETUPTOOLS_SCM_SUBPROCESS_TIMEOUT ??= "600"
+
 FILES:${PN} += "${PYTHON_SITEPACKAGES_DIR}"
 FILES:${PN}-staticdev += "${PYTHON_SITEPACKAGES_DIR}/*.a"
 FILES:${PN}-dev += "${PYTHON_SITEPACKAGES_DIR}/*.la"
-- 
2.34.1

Re: [OE-core] [PATCH V2] setuptools3-base.bbclass: override default subprocess timeout

[Yoann Congal]

[-- Attachment #1: Type: text/plain, Size: 2733 bytes --]

Hello,


Le lun. 17 févr. 2025 à 06:54, hongxu via lists.openembedded.org
<hongxu.jia=eng.windriver.com@lists.openembedded.org> a écrit :

> The environment variable SETUPTOOLS_SCM_SUBPROCESS_TIMEOUT allows to
> override
> the subprocess timeout. The default is 40 seconds and should work for most
> needs.[1] However, it was not enough while using git shallow tarball and
> starting
> multiple Yocto world builds in one host.
>
> |   File
> "tmp/work/x86_64-linux/python3-scancode-native/32.1.0/recipe-sysroot-
> native/usr/lib/python3.13/subprocess.py", line 1263, in _check_timeout
> |     raise TimeoutExpired(
> |     ...<2 lines>...
> |             stderr=b''.join(stderr_seq) if stderr_seq else None)
> | subprocess.TimeoutExpired: Command '['git', '--git-dir',
> 'tmp/work/x86_64-
> linux/python3-scancode-native/32.1.0/git/.git', 'status', '--porcelain',
> '--untracked-files=no']' timed out after 40 seconds
>
> Explicitly set variable SETUPTOOLS_SCM_SUBPROCESS_TIMEOUT to 600s in
> bbclass,
> and we could override it in local.conf
>
> [1] https://github.com/pypa/setuptools-scm/blob/main/docs/overrides.md
>
> Issue: LINCD-21838
> (LOCAL REV: NOT UPSTREAM) -- Sent to oe-core on 20250217

That looks like an internal tag that should not be part of the patch sent
to oe-core.

Regards,


> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
> ---
>  meta/classes-recipe/setuptools3-base.bbclass | 4 ++++
>  1 file changed, 4 insertions(+)
>
> diff --git a/meta/classes-recipe/setuptools3-base.bbclass
> b/meta/classes-recipe/setuptools3-base.bbclass
> index 27af6abc58..190d9e6e3a 100644
> --- a/meta/classes-recipe/setuptools3-base.bbclass
> +++ b/meta/classes-recipe/setuptools3-base.bbclass
> @@ -23,6 +23,10 @@ export CCSHARED  = "-fPIC -DPIC"
>  # the python executable
>  export LINKFORSHARED = "${SECURITY_CFLAGS} -Xlinker -export-dynamic"
>
> +# The environment variable SETUPTOOLS_SCM_SUBPROCESS_TIMEOUT allows
> +# to override the subprocess timeout.
> +export SETUPTOOLS_SCM_SUBPROCESS_TIMEOUT ??= "600"
> +
>  FILES:${PN} += "${PYTHON_SITEPACKAGES_DIR}"
>  FILES:${PN}-staticdev += "${PYTHON_SITEPACKAGES_DIR}/*.a"
>  FILES:${PN}-dev += "${PYTHON_SITEPACKAGES_DIR}/*.la"
> --
> 2.34.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#211513):
> https://lists.openembedded.org/g/openembedded-core/message/211513
> Mute This Topic: https://lists.openembedded.org/mt/111227231/4316185
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> yoann.congal@smile.fr]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 4601 bytes --]

[PATCH V3] setuptools3-base.bbclass: override default subprocess timeout

[Hongxu Jia]

The environment variable SETUPTOOLS_SCM_SUBPROCESS_TIMEOUT allows to override
the subprocess timeout. The default is 40 seconds and should work for most
needs.[1] However, it was not enough while using git shallow tarball and starting
multiple Yocto world builds in one host.

|   File "tmp/work/x86_64-linux/python3-scancode-native/32.1.0/recipe-sysroot-
native/usr/lib/python3.13/subprocess.py", line 1263, in _check_timeout
|     raise TimeoutExpired(
|     ...<2 lines>...
|             stderr=b''.join(stderr_seq) if stderr_seq else None)
| subprocess.TimeoutExpired: Command '['git', '--git-dir', 'tmp/work/x86_64-
linux/python3-scancode-native/32.1.0/git/.git', 'status', '--porcelain',
'--untracked-files=no']' timed out after 40 seconds

Explicitly set variable SETUPTOOLS_SCM_SUBPROCESS_TIMEOUT to 600s in bbclass,
and we could override it in local.conf

[1] https://github.com/pypa/setuptools-scm/blob/main/docs/overrides.md

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/classes-recipe/setuptools3-base.bbclass | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/classes-recipe/setuptools3-base.bbclass b/meta/classes-recipe/setuptools3-base.bbclass
index 27af6abc58..190d9e6e3a 100644
--- a/meta/classes-recipe/setuptools3-base.bbclass
+++ b/meta/classes-recipe/setuptools3-base.bbclass
@@ -23,6 +23,10 @@ export CCSHARED  = "-fPIC -DPIC"
 # the python executable
 export LINKFORSHARED = "${SECURITY_CFLAGS} -Xlinker -export-dynamic"
 
+# The environment variable SETUPTOOLS_SCM_SUBPROCESS_TIMEOUT allows
+# to override the subprocess timeout.
+export SETUPTOOLS_SCM_SUBPROCESS_TIMEOUT ??= "600"
+
 FILES:${PN} += "${PYTHON_SITEPACKAGES_DIR}"
 FILES:${PN}-staticdev += "${PYTHON_SITEPACKAGES_DIR}/*.a"
 FILES:${PN}-dev += "${PYTHON_SITEPACKAGES_DIR}/*.la"
-- 
2.34.1

Re: [OE-core] [PATCH V3] setuptools3-base.bbclass: override default subprocess timeout

[Ross Burton]

On 17 Feb 2025, at 07:46, hongxu via lists.openembedded.org <hongxu.jia=eng.windriver.com@lists.openembedded.org> wrote:
> 
> The environment variable SETUPTOOLS_SCM_SUBPROCESS_TIMEOUT allows to override
> the subprocess timeout. The default is 40 seconds and should work for most
> needs.[1] However, it was not enough while using git shallow tarball and starting
> multiple Yocto world builds in one host.
> 
> |   File "tmp/work/x86_64-linux/python3-scancode-native/32.1.0/recipe-sysroot-
> native/usr/lib/python3.13/subprocess.py", line 1263, in _check_timeout
> |     raise TimeoutExpired(
> |     ...<2 lines>...
> |             stderr=b''.join(stderr_seq) if stderr_seq else None)
> | subprocess.TimeoutExpired: Command '['git', '--git-dir', 'tmp/work/x86_64-
> linux/python3-scancode-native/32.1.0/git/.git', 'status', '--porcelain',
> '--untracked-files=no']' timed out after 40 seconds

Can you define “multiple”?  If a ‘git status’ is taking more than 40 seconds to complete then it sounds like your machine is being utterly hammered beyond reasonable measures.

Ross

Re: [PATCH V3] setuptools3-base.bbclass: override default subprocess timeout

[hongxu]

[-- Attachment #1: Type: text/plain, Size: 285 bytes --]

It happens occasionally, in our build farm, similar Yocto Autobuilder, the build test controller  starts up multiple containers for the build on the host, and each container is allocated for the world build.
It should be some kind of limit testing or stress testing for yocto build

[-- Attachment #2: Type: text/html, Size: 311 bytes --]

Re: [OE-core] [PATCH V3] setuptools3-base.bbclass: override default subprocess timeout

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1038 bytes --]

But why should this be set in core for everyone, and to ten minutes? Your
farm is misconfigured if 40 seconds is not enough got ‘git status’.

Alex

On Tue 18. Feb 2025 at 0.39, hongxu via lists.openembedded.org <hongxu.jia=
eng.windriver.com@lists.openembedded.org> wrote:

> It happens occasionally, in our build farm, similar Yocto Autobuilder, the
> build test controller  starts up multiple containers for the build on  the
> host, and each container is allocated for the world build.
> It should be some kind of limit testing or stress testing for yocto build
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#211558):
> https://lists.openembedded.org/g/openembedded-core/message/211558
> Mute This Topic: https://lists.openembedded.org/mt/111228031/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 1992 bytes --]