[PATCH 1/2] scripts/install-buildtools: add an option to preserve downloads

[PATCH 1/2] scripts/install-buildtools: add an option to preserve downloads

[Alexander Kanavin]

From: Alexander Kanavin <alex@linutronix.de>

By default the script puts everything it downloads into a temporary
directory and erases it after unpacking and installing the buildtools.

This isn't great for traceability and reproducibility of builds
(being able to see what was downloaded exactly, and being able
to reproduce setting up a build, especially if the buildtools
download location isn't available for whatever reason).

This commit adds an option to download items into a specified directory
and keep them there. I would particularly like to use it
with bitbake-setup, where an optional script to install the buildtools
would ensure the tarball remains available on local disk.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 scripts/install-buildtools | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/scripts/install-buildtools b/scripts/install-buildtools
index 639ebb12d78..1a68aad53e7 100755
--- a/scripts/install-buildtools
+++ b/scripts/install-buildtools
@@ -142,6 +142,9 @@ def main():
                         default=DEFAULT_INSTALL_DIR,
                         help='directory where buildtools SDK will be installed (optional)',
                         action='store')
+    parser.add_argument('--keep-downloads-directory',
+                        help='use this directory for tarball/checksum downloads and do not erase them (default is a temporary directory which is deleted after unpacking and installing the buildtools)',
+                        action='store')
     parser.add_argument('-r', '--release',
                         default=DEFAULT_RELEASE,
                         help='Yocto Project release string for SDK which will be '
@@ -235,11 +238,11 @@ def main():
                 safe_filename = quote(filename)
                 buildtools_url = "%s/%s/buildtools/%s" % (base_url, args.release, safe_filename)
 
-    tmpsdk_dir = tempfile.mkdtemp()
+    sdk_dir = args.keep_downloads_directory or tempfile.mkdtemp()
     try:
         # Fetch installer
         logger.info("Fetching buildtools installer")
-        tmpbuildtools = os.path.join(tmpsdk_dir, filename)
+        tmpbuildtools = os.path.join(sdk_dir, filename)
         ret = subprocess.call("wget -q -O %s %s" %
                               (tmpbuildtools, buildtools_url), shell=True)
         if ret != 0:
@@ -252,7 +255,7 @@ def main():
             checksum_type = "sha256sum"
             check_url = "{}.{}".format(buildtools_url, checksum_type)
             checksum_filename = "{}.{}".format(filename, checksum_type)
-            tmpbuildtools_checksum = os.path.join(tmpsdk_dir, checksum_filename)
+            tmpbuildtools_checksum = os.path.join(sdk_dir, checksum_filename)
             ret = subprocess.call("wget -q -O %s %s" %
                                     (tmpbuildtools_checksum, check_url), shell=True)
             if ret != 0:
@@ -347,7 +350,8 @@ def main():
 
     finally:
         # cleanup tmp directory
-        shutil.rmtree(tmpsdk_dir)
+        if not args.keep_downloads_directory:
+            shutil.rmtree(sdk_dir)
 
 
 if __name__ == '__main__':
-- 
2.39.5

[PATCH 2/2] scripts/install-buildtools: write download URLs to files for traceability

[Alexander Kanavin]

From: Alexander Kanavin <alex@linutronix.de>

This extends the previous commit, so that download URLs are preserved
in addition to actual artefacts. By default it's all written to a temporary
directory and erased together with artefacts, but users can choose
to preserve both:

$ install-buildtools -D --keep-downloads-directory=.
...

$ ls -1
buildtools_url
check_url
x86_64-buildtools-extended-nativesdk-standalone-5.1.2.sh
x86_64-buildtools-extended-nativesdk-standalone-5.1.2.sh.sha256sum

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 scripts/install-buildtools | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/scripts/install-buildtools b/scripts/install-buildtools
index 1a68aad53e7..7bd6556edf6 100755
--- a/scripts/install-buildtools
+++ b/scripts/install-buildtools
@@ -243,6 +243,8 @@ def main():
         # Fetch installer
         logger.info("Fetching buildtools installer")
         tmpbuildtools = os.path.join(sdk_dir, filename)
+        with open(os.path.join(sdk_dir, 'buildtools_url'), 'w') as f:
+            f.write(buildtools_url)
         ret = subprocess.call("wget -q -O %s %s" %
                               (tmpbuildtools, buildtools_url), shell=True)
         if ret != 0:
@@ -256,6 +258,8 @@ def main():
             check_url = "{}.{}".format(buildtools_url, checksum_type)
             checksum_filename = "{}.{}".format(filename, checksum_type)
             tmpbuildtools_checksum = os.path.join(sdk_dir, checksum_filename)
+            with open(os.path.join(sdk_dir, 'check_url'), 'w') as f:
+                f.write(check_url)
             ret = subprocess.call("wget -q -O %s %s" %
                                     (tmpbuildtools_checksum, check_url), shell=True)
             if ret != 0:
-- 
2.39.5

Re: [OE-core] [PATCH 2/2] scripts/install-buildtools: write download URLs to files for traceability

[Richard Purdie]

On Tue, 2025-03-11 at 11:41 +0100, Alexander Kanavin via lists.openembedded.org wrote:
> From: Alexander Kanavin <alex@linutronix.de>
> 
> This extends the previous commit, so that download URLs are preserved
> in addition to actual artefacts. By default it's all written to a temporary
> directory and erased together with artefacts, but users can choose
> to preserve both:
> 
> $ install-buildtools -D --keep-downloads-directory=.
> ...
> 
> $ ls -1
> buildtools_url
> check_url
> x86_64-buildtools-extended-nativesdk-standalone-5.1.2.sh
> x86_64-buildtools-extended-nativesdk-standalone-5.1.2.sh.sha256sum
> 
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>  scripts/install-buildtools | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/scripts/install-buildtools b/scripts/install-buildtools
> index 1a68aad53e7..7bd6556edf6 100755
> --- a/scripts/install-buildtools
> +++ b/scripts/install-buildtools
> @@ -243,6 +243,8 @@ def main():
>          # Fetch installer
>          logger.info("Fetching buildtools installer")
>          tmpbuildtools = os.path.join(sdk_dir, filename)
> +        with open(os.path.join(sdk_dir, 'buildtools_url'), 'w') as f:
> +            f.write(buildtools_url)
>          ret = subprocess.call("wget -q -O %s %s" %
>                                (tmpbuildtools, buildtools_url), shell=True)
>          if ret != 0:
> @@ -256,6 +258,8 @@ def main():
>              check_url = "{}.{}".format(buildtools_url, checksum_type)
>              checksum_filename = "{}.{}".format(filename, checksum_type)
>              tmpbuildtools_checksum = os.path.join(sdk_dir, checksum_filename)
> +            with open(os.path.join(sdk_dir, 'check_url'), 'w') as f:
> +                f.write(check_url)
>              ret = subprocess.call("wget -q -O %s %s" %
>                                      (tmpbuildtools_checksum, check_url), shell=True)
>              if ret != 0:
> 


The naming of "check_url" needs to be changed. I appreciate you've used
a variable name from the code however when you see this file in
isolation in a directory, you'd have no real idea what it is (a
checksum for the downloaded file). "buildtools_url.checksum" might be
even clearer?

Cheers,

Richard

Patchtest results for [PATCH 1/2] scripts/install-buildtools: add an option to preserve downloads

[patchtest]

[-- Attachment #1: Type: text/plain, Size: 3187 bytes --]

Thank you for your submission. Patchtest identified one
or more issues with the patch. Please see the log below for
more information:

---
Testing patch /home/patchtest/share/mboxes/1-2-scripts-install-buildtools-add-an-option-to-preserve-downloads.patch

FAIL: test max line length: Patch line too long (current length 201, maximum is 200) (test_metadata.TestMetadata.test_max_line_length)

PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence)
PASS: test commit message user tags (test_mbox.TestMbox.test_commit_message_user_tags)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)
PASS: test target mailing list (test_mbox.TestMbox.test_target_mailing_list)

SKIP: pretest pylint: No python related patches, skipping test (test_python_pylint.PyLint.pretest_pylint)
SKIP: pretest src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE check ignore: No modified recipes or older target branch, skipping test (test_metadata.TestMetadata.test_cve_check_ignore)
SKIP: test CVE tag format: No new CVE patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new CVE patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new CVE patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test pylint: No python related patches, skipping test (test_python_pylint.PyLint.test_pylint)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)

---

Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this
email with an explanation of why the patch should be accepted. If you
believe these results are due to an error in patchtest, please submit a
bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category
under 'Yocto Project Subprojects'). For more information on specific
failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank
you!

Re: [OE-core] [PATCH 1/2] scripts/install-buildtools: add an option to preserve downloads

[Antonin Godard]

Hi Alex,

On Tue Mar 11, 2025 at 11:41 AM CET, Alexander Kanavin via lists.openembedded.org wrote:
> From: Alexander Kanavin <alex@linutronix.de>
>
> By default the script puts everything it downloads into a temporary
> directory and erases it after unpacking and installing the buildtools.
>
> This isn't great for traceability and reproducibility of builds
> (being able to see what was downloaded exactly, and being able
> to reproduce setting up a build, especially if the buildtools
> download location isn't available for whatever reason).
>
> This commit adds an option to download items into a specified directory
> and keep them there. I would particularly like to use it
> with bitbake-setup, where an optional script to install the buildtools
> would ensure the tarball remains available on local disk.
>
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>  scripts/install-buildtools | 12 ++++++++----
>  1 file changed, 8 insertions(+), 4 deletions(-)
>
> diff --git a/scripts/install-buildtools b/scripts/install-buildtools
> index 639ebb12d78..1a68aad53e7 100755
> --- a/scripts/install-buildtools
> +++ b/scripts/install-buildtools
> @@ -142,6 +142,9 @@ def main():
>                          default=DEFAULT_INSTALL_DIR,
>                          help='directory where buildtools SDK will be installed (optional)',
>                          action='store')
> +    parser.add_argument('--keep-downloads-directory',
> +                        help='use this directory for tarball/checksum downloads and do not erase them (default is a temporary directory which is deleted after unpacking and installing the buildtools)',
> +                        action='store')
>      parser.add_argument('-r', '--release',
>                          default=DEFAULT_RELEASE,
>                          help='Yocto Project release string for SDK which will be '
> @@ -235,11 +238,11 @@ def main():
>                  safe_filename = quote(filename)
>                  buildtools_url = "%s/%s/buildtools/%s" % (base_url, args.release, safe_filename)
>  
> -    tmpsdk_dir = tempfile.mkdtemp()
> +    sdk_dir = args.keep_downloads_directory or tempfile.mkdtemp()

How about also creating the directory if args.keep_downloads_directory is set?

Antonin

-- 
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [PATCH 1/2] scripts/install-buildtools: add an option to preserve downloads

[Richard Purdie]

On Tue, 2025-03-11 at 11:41 +0100, Alexander Kanavin via lists.openembedded.org wrote:
> From: Alexander Kanavin <alex@linutronix.de>
> 
> By default the script puts everything it downloads into a temporary
> directory and erases it after unpacking and installing the buildtools.
> 
> This isn't great for traceability and reproducibility of builds
> (being able to see what was downloaded exactly, and being able
> to reproduce setting up a build, especially if the buildtools
> download location isn't available for whatever reason).
> 
> This commit adds an option to download items into a specified directory
> and keep them there. I would particularly like to use it
> with bitbake-setup, where an optional script to install the buildtools
> would ensure the tarball remains available on local disk.
> 
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>  scripts/install-buildtools | 12 ++++++++----
>  1 file changed, 8 insertions(+), 4 deletions(-)
> 
> diff --git a/scripts/install-buildtools b/scripts/install-buildtools
> index 639ebb12d78..1a68aad53e7 100755
> --- a/scripts/install-buildtools
> +++ b/scripts/install-buildtools
> @@ -142,6 +142,9 @@ def main():
>                          default=DEFAULT_INSTALL_DIR,
>                          help='directory where buildtools SDK will be installed (optional)',
>                          action='store')
> +    parser.add_argument('--keep-downloads-directory',
> +                        help='use this directory for tarball/checksum downloads and do not erase them (default is a temporary directory which is deleted after unpacking and installing the buildtools)',
> +                        action='store')
>      parser.add_argument('-r', '--release',
>                          default=DEFAULT_RELEASE,
>                          help='Yocto Project release string for SDK which will be '

I think this option should be renamed to simply "--downloads-
directory". The "keep" element made me think it was a boolean value
which when you read the code/description, it is not, it specifies a
download directory to use. I appreciate how you came to that naming in
the way it changes the code behaviour but I don't think users reading
the help text will understand it.

Cheers,

Richard