[yocto-docs PATCH] set_versions.py: use backward-compatible python argument in run

[yocto-docs PATCH] set_versions.py: use backward-compatible python argument in run

[Antonin Godard]

Some workers on the autobuilder reported the following error:

  File "./set_versions.py", line 102, in <module>
    subprocess.run(["git", "show", "yocto-%s" % release_series[activereleases[0]]], capture_output=True, check=True)
  File "/usr/lib64/python3.6/subprocess.py", line 423, in run
    with Popen(*popenargs, **kwargs) as process:
  TypeError: __init__() got an unexpected keyword argument 'capture_output'

See https://valkyrie.yoctoproject.org/#/builders/34/builds/86.

This is because capture_output was introduced in Python 3.7, and some of
the support distributions are still on Python 3.6. Since capture_output
is essentially just setting stdout and stderr to PIPE
(https://github.com/python/cpython/blob/3.13/Lib/subprocess.py#L547), do
it manually here to be compatible with older python versions.

This is also the case for the "text" parameter, introduced in 3.7 to
alias the universal_newlines parameter. Use "universal_newlines" to be
backward-compatible.

[ YOCTO #15687 ]

Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
---
Test run here: https://valkyrie.yoctoproject.org/#/builders/34/builds/94.
---
 documentation/set_versions.py | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/documentation/set_versions.py b/documentation/set_versions.py
index 376337e7b544829e7f65cb5dcacda635980ee28d..5c55f470d782bb613e619dfb01ca28dea5b54d59 100755
--- a/documentation/set_versions.py
+++ b/documentation/set_versions.py
@@ -99,12 +99,12 @@ docconfver = None
 
 # Test tags exist and inform the user to fetch if not
 try:
-    subprocess.run(["git", "show", "yocto-%s" % release_series[activereleases[0]]], capture_output=True, check=True)
+    subprocess.run(["git", "show", "yocto-%s" % release_series[activereleases[0]]], stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=True)
 except subprocess.CalledProcessError:
     sys.exit("Please run 'git fetch --tags' before building the documentation")
 
 # Try and figure out what we are
-tags = subprocess.run(["git", "tag", "--points-at", "HEAD"], capture_output=True, text=True).stdout
+tags = subprocess.run(["git", "tag", "--points-at", "HEAD"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True).stdout
 for t in tags.split():
     if t.startswith("yocto-"):
         ourversion = t[6:]
@@ -122,7 +122,7 @@ if ourversion:
                 bitbakeversion = bitbake_mapping[i]
 else:
     # We're floating on a branch
-    branch = subprocess.run(["git", "branch", "--show-current"], capture_output=True, text=True).stdout.strip()
+    branch = subprocess.run(["git", "branch", "--show-current"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True).stdout.strip()
     ourbranch = branch
     if branch != "master" and branch not in release_series:
         # We're not on a known release branch so we have to guess. Compare the numbers of commits
@@ -130,7 +130,7 @@ else:
         possible_branch = None
         branch_count = 0
         for b in itertools.chain(release_series.keys(), ["master"]):
-            result = subprocess.run(["git", "log", "--format=oneline",  "HEAD..origin/" + b], capture_output=True, text=True)
+            result = subprocess.run(["git", "log", "--format=oneline",  "HEAD..origin/" + b], stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
             if result.returncode == 0:
                 count = result.stdout.count('\n')
                 if not possible_branch or count < branch_count:
@@ -153,9 +153,9 @@ else:
     else:
         sys.exit("Unknown series for branch %s" % branch)
 
-    previoustags = subprocess.run(["git", "tag", "--merged", "HEAD"], capture_output=True, text=True).stdout
+    previoustags = subprocess.run(["git", "tag", "--merged", "HEAD"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True).stdout
     previoustags = [t[6:] for t in previoustags.split() if t.startswith("yocto-" + release_series[ourseries])]
-    futuretags = subprocess.run(["git", "tag", "--merged", ourbranch], capture_output=True, text=True).stdout
+    futuretags = subprocess.run(["git", "tag", "--merged", ourbranch], stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True).stdout
     futuretags = [t[6:] for t in futuretags.split() if t.startswith("yocto-" + release_series[ourseries])]
 
     # Append .999 against the last known version
@@ -228,7 +228,7 @@ with open("sphinx-static/switchers.js.in", "r") as r, open("sphinx-static/switch
             for branch in activereleases + ([ourseries] if ourseries not in activereleases else []):
                 if branch == devbranch:
                     continue
-                branch_versions = subprocess.run('git tag --list yocto-%s*' % (release_series[branch]), shell=True, capture_output=True, text=True).stdout.split()
+                branch_versions = subprocess.run('git tag --list yocto-%s*' % (release_series[branch]), shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True).stdout.split()
                 branch_versions = sorted([v.replace("yocto-" +  release_series[branch] + ".", "").replace("yocto-" +  release_series[branch], "0") for v in branch_versions], key=int)
                 if not branch_versions:
                     continue
@@ -273,7 +273,7 @@ def tag_to_semver_like(v):
     v_maj, v_min, v_patch = v_semver.groups('0')
     return int("{:0>2}{:0>2}{:0>2}".format(v_maj, v_min, v_patch), 10)
 
-yocto_tags = subprocess.run(["git", "tag", "--list", "--sort=version:refname", "yocto-*"], capture_output=True, text=True).stdout
+yocto_tags = subprocess.run(["git", "tag", "--list", "--sort=version:refname", "yocto-*"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True).stdout
 yocto_tags = sorted(yocto_tags.split() + missing_tags, key=tag_to_semver_like)
 tags = [tag[6:] for tag in yocto_tags]
 

---
base-commit: ee16c96202e5027d1a8d7e89e11c25f127c78326
change-id: 20241216-capture-output-compat-639e50f53be9

Best regards,
-- 
Antonin Godard <antonin.godard@bootlin.com>

Re: [docs] [yocto-docs PATCH] set_versions.py: use backward-compatible python argument in run

[Quentin Schulz]

Hi Antonin,

On 12/17/24 11:35 AM, Antonin Godard via lists.yoctoproject.org wrote:
> Some workers on the autobuilder reported the following error:
> 
>    File "./set_versions.py", line 102, in <module>
>      subprocess.run(["git", "show", "yocto-%s" % release_series[activereleases[0]]], capture_output=True, check=True)
>    File "/usr/lib64/python3.6/subprocess.py", line 423, in run
>      with Popen(*popenargs, **kwargs) as process:
>    TypeError: __init__() got an unexpected keyword argument 'capture_output'
> 
> See https://valkyrie.yoctoproject.org/#/builders/34/builds/86.
> 
> This is because capture_output was introduced in Python 3.7, and some of
> the support distributions are still on Python 3.6. Since capture_output
> is essentially just setting stdout and stderr to PIPE
> (https://github.com/python/cpython/blob/3.13/Lib/subprocess.py#L547), do
> it manually here to be compatible with older python versions.
> 
> This is also the case for the "text" parameter, introduced in 3.7 to
> alias the universal_newlines parameter. Use "universal_newlines" to be
> backward-compatible.
> 
> [ YOCTO #15687 ]
> 

Do we really want to open that can of worms?

https://docs.yoctoproject.org/ref-manual/system-requirements.html#required-git-tar-python-make-and-gcc-versions 
clearly states the minimal supported Python version is 3.8.0 so maybe we 
should just stop building the docs on those distros that are still on 
old Python? I'm actually surprised building YP on AlmaLinux 8 works 
then, is it because of uninative?

Looks ok to me otherwise.

Cheers,
Quentin

Re: [docs] [yocto-docs PATCH] set_versions.py: use backward-compatible python argument in run

[Antonin Godard]

Hi Quentin,

On Tue Dec 17, 2024 at 12:05 PM CET, Quentin Schulz wrote:
> Hi Antonin,
>
> On 12/17/24 11:35 AM, Antonin Godard via lists.yoctoproject.org wrote:
>> Some workers on the autobuilder reported the following error:
>> 
>>    File "./set_versions.py", line 102, in <module>
>>      subprocess.run(["git", "show", "yocto-%s" % release_series[activereleases[0]]], capture_output=True, check=True)
>>    File "/usr/lib64/python3.6/subprocess.py", line 423, in run
>>      with Popen(*popenargs, **kwargs) as process:
>>    TypeError: __init__() got an unexpected keyword argument 'capture_output'
>> 
>> See https://valkyrie.yoctoproject.org/#/builders/34/builds/86.
>> 
>> This is because capture_output was introduced in Python 3.7, and some of
>> the support distributions are still on Python 3.6. Since capture_output
>> is essentially just setting stdout and stderr to PIPE
>> (https://github.com/python/cpython/blob/3.13/Lib/subprocess.py#L547), do
>> it manually here to be compatible with older python versions.
>> 
>> This is also the case for the "text" parameter, introduced in 3.7 to
>> alias the universal_newlines parameter. Use "universal_newlines" to be
>> backward-compatible.
>> 
>> [ YOCTO #15687 ]
>> 
>
> Do we really want to open that can of worms?
>
> https://docs.yoctoproject.org/ref-manual/system-requirements.html#required-git-tar-python-make-and-gcc-versions 
> clearly states the minimal supported Python version is 3.8.0 so maybe we 
> should just stop building the docs on those distros that are still on 
> old Python? I'm actually surprised building YP on AlmaLinux 8 works 
> then, is it because of uninative?

We still support 3.6 on Kirkstone. The docs build uses a buildtools tarball to
get a recent Sphinx version, rsvg-convert, etc. I wouldn't know how this is done
for other builds, though.

> Looks ok to me otherwise.

Thank you,
Antonin

-- 
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] [yocto-docs PATCH] set_versions.py: use backward-compatible python argument in run

[Quentin Schulz]

Hi Antonin,

On 12/18/24 9:19 AM, Antonin Godard wrote:
> Hi Quentin,
> 
> On Tue Dec 17, 2024 at 12:05 PM CET, Quentin Schulz wrote:
>> Hi Antonin,
>>
>> On 12/17/24 11:35 AM, Antonin Godard via lists.yoctoproject.org wrote:
>>> Some workers on the autobuilder reported the following error:
>>>
>>>     File "./set_versions.py", line 102, in <module>
>>>       subprocess.run(["git", "show", "yocto-%s" % release_series[activereleases[0]]], capture_output=True, check=True)
>>>     File "/usr/lib64/python3.6/subprocess.py", line 423, in run
>>>       with Popen(*popenargs, **kwargs) as process:
>>>     TypeError: __init__() got an unexpected keyword argument 'capture_output'
>>>
>>> See https://valkyrie.yoctoproject.org/#/builders/34/builds/86.
>>>
>>> This is because capture_output was introduced in Python 3.7, and some of
>>> the support distributions are still on Python 3.6. Since capture_output
>>> is essentially just setting stdout and stderr to PIPE
>>> (https://github.com/python/cpython/blob/3.13/Lib/subprocess.py#L547), do
>>> it manually here to be compatible with older python versions.
>>>
>>> This is also the case for the "text" parameter, introduced in 3.7 to
>>> alias the universal_newlines parameter. Use "universal_newlines" to be
>>> backward-compatible.
>>>
>>> [ YOCTO #15687 ]
>>>
>>
>> Do we really want to open that can of worms?
>>
>> https://docs.yoctoproject.org/ref-manual/system-requirements.html#required-git-tar-python-make-and-gcc-versions
>> clearly states the minimal supported Python version is 3.8.0 so maybe we
>> should just stop building the docs on those distros that are still on
>> old Python? I'm actually surprised building YP on AlmaLinux 8 works
>> then, is it because of uninative?
> 
> We still support 3.6 on Kirkstone. The docs build uses a buildtools tarball to

I see, so the patch should be merged at least for the branches which do 
still support 3.6. For the others, not necessarily. The script isn't 
that big and is not that complex (in terms of Python code, the logic.... 
is another story :) ) so maybe it's fine. Your decision as maintainer. 
The diff looks sound to me (and I used it for building on openSUSE 
within containers), so I guess:

Reviewed-by: Quentin Schulz <quentin.schulz@cherry.de>
Tested-by: Quentin Schulz <quentin.schulz@cherry.de> # openSUSE Leap 
15.4-15.6

Thanks!
Quentin

Re: [docs] [yocto-docs PATCH] set_versions.py: use backward-compatible python argument in run

[Antonin Godard]

Hi Quentin,

On Wed Dec 18, 2024 at 2:39 PM CET, Quentin Schulz wrote:
> Hi Antonin,
>
> On 12/18/24 9:19 AM, Antonin Godard wrote:
>> Hi Quentin,
>> 
>> On Tue Dec 17, 2024 at 12:05 PM CET, Quentin Schulz wrote:
>>> Hi Antonin,
>>>
>>> On 12/17/24 11:35 AM, Antonin Godard via lists.yoctoproject.org wrote:
>>>> Some workers on the autobuilder reported the following error:
>>>>
>>>>     File "./set_versions.py", line 102, in <module>
>>>>       subprocess.run(["git", "show", "yocto-%s" % release_series[activereleases[0]]], capture_output=True, check=True)
>>>>     File "/usr/lib64/python3.6/subprocess.py", line 423, in run
>>>>       with Popen(*popenargs, **kwargs) as process:
>>>>     TypeError: __init__() got an unexpected keyword argument 'capture_output'
>>>>
>>>> See https://valkyrie.yoctoproject.org/#/builders/34/builds/86.
>>>>
>>>> This is because capture_output was introduced in Python 3.7, and some of
>>>> the support distributions are still on Python 3.6. Since capture_output
>>>> is essentially just setting stdout and stderr to PIPE
>>>> (https://github.com/python/cpython/blob/3.13/Lib/subprocess.py#L547), do
>>>> it manually here to be compatible with older python versions.
>>>>
>>>> This is also the case for the "text" parameter, introduced in 3.7 to
>>>> alias the universal_newlines parameter. Use "universal_newlines" to be
>>>> backward-compatible.
>>>>
>>>> [ YOCTO #15687 ]
>>>>
>>>
>>> Do we really want to open that can of worms?
>>>
>>> https://docs.yoctoproject.org/ref-manual/system-requirements.html#required-git-tar-python-make-and-gcc-versions
>>> clearly states the minimal supported Python version is 3.8.0 so maybe we
>>> should just stop building the docs on those distros that are still on
>>> old Python? I'm actually surprised building YP on AlmaLinux 8 works
>>> then, is it because of uninative?
>> 
>> We still support 3.6 on Kirkstone. The docs build uses a buildtools tarball to
>
> I see, so the patch should be merged at least for the branches which do 
> still support 3.6. For the others, not necessarily. The script isn't 
> that big and is not that complex (in terms of Python code, the logic.... 
> is another story :) ) so maybe it's fine. Your decision as maintainer. 
> The diff looks sound to me (and I used it for building on openSUSE 
> within containers), so I guess:

Unfortunately I think it's not that simple - the autobuilder
checks out the set_versions.py script from master for every branch it builds
(see the run-docs-build script on the yocto-autobuilder-helper repo).

I think merging this for every branch is fine though.

> Reviewed-by: Quentin Schulz <quentin.schulz@cherry.de>
> Tested-by: Quentin Schulz <quentin.schulz@cherry.de> # openSUSE Leap 
> 15.4-15.6

Thank you!
Antonin

-- 
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com