[PATCH 1/4] install-buildtools: support buildtools-make-tarball and update to 4.1

[PATCH 1/4] install-buildtools: support buildtools-make-tarball and update to 4.1

[Aleksandar Nikolic]

From: Paul Eggleton <paul.eggleton@microsoft.com>

Support installing buildtools-make-tarball that is built in version 4.1
and later for build hosts with a broken make version. Also update the
default version values to 4.1.

Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com>
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 5d539268d0c7b8fad1ba9352c7f2d4b81e78b75c)
Signed-off-by: Aleksandar Nikolic <aleksandar.nikolic@zeiss.com>
---
 scripts/install-buildtools | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/scripts/install-buildtools b/scripts/install-buildtools
index 10c3d043de..2218f3ffac 100755
--- a/scripts/install-buildtools
+++ b/scripts/install-buildtools
@@ -57,8 +57,8 @@ logger = scriptutils.logger_create(PROGNAME, stream=sys.stdout)
 
 DEFAULT_INSTALL_DIR = os.path.join(os.path.split(scripts_path)[0],'buildtools')
 DEFAULT_BASE_URL = 'http://downloads.yoctoproject.org/releases/yocto'
-DEFAULT_RELEASE = 'yocto-3.4'
-DEFAULT_INSTALLER_VERSION = '3.4'
+DEFAULT_RELEASE = 'yocto-4.1'
+DEFAULT_INSTALLER_VERSION = '4.1'
 DEFAULT_BUILDDATE = '202110XX'
 
 # Python version sanity check
@@ -154,6 +154,8 @@ def main():
     group.add_argument('--without-extended-buildtools', action='store_false',
                        dest='with_extended_buildtools',
                        help='disable extended buildtools (traditional buildtools tarball)')
+    group.add_argument('--make-only', action='store_true',
+                       help='only install make tarball')
     group = parser.add_mutually_exclusive_group()
     group.add_argument('-c', '--check', help='enable checksum validation',
                         default=True, action='store_true')
@@ -170,6 +172,9 @@ def main():
 
     args = parser.parse_args()
 
+    if args.make_only:
+        args.with_extended_buildtools = False
+
     if args.debug:
         logger.setLevel(logging.DEBUG)
     elif args.quiet:
@@ -197,7 +202,10 @@ def main():
                 if not args.build_date:
                     logger.error("Milestone installers require --build-date")
                 else:
-                    if args.with_extended_buildtools:
+                    if args.make_only:
+                        filename = "%s-buildtools-make-nativesdk-standalone-%s-%s.sh" % (
+                            arch, args.installer_version, args.build_date)
+                    elif args.with_extended_buildtools:
                         filename = "%s-buildtools-extended-nativesdk-standalone-%s-%s.sh" % (
                             arch, args.installer_version, args.build_date)
                     else:
@@ -207,6 +215,8 @@ def main():
                     buildtools_url = "%s/milestones/%s/buildtools/%s" % (base_url, args.release, safe_filename)
             # regular release SDK
             else:
+                if args.make_only:
+                    filename = "%s-buildtools-make-nativesdk-standalone-%s.sh" % (arch, args.installer_version)
                 if args.with_extended_buildtools:
                     filename = "%s-buildtools-extended-nativesdk-standalone-%s.sh" % (arch, args.installer_version)
                 else:
@@ -303,7 +313,9 @@ def main():
         if args.with_extended_buildtools and not m:
             logger.info("Ignoring --with-extended-buildtools as filename "
                         "does not contain 'extended'")
-        if args.with_extended_buildtools and m:
+        if args.make_only:
+            tool = 'make'
+        elif args.with_extended_buildtools and m:
             tool = 'gcc'
         else:
             tool = 'tar'
-- 
2.25.1

[PATCH 2/4] install-buildtools: remove md5 checksum validation

[Aleksandar Nikolic]

From: Aleksandar Nikolic <aleksandar.nikolic@zeiss.com>

No need to validate with the md5 checksum, as the file is not even
uploaded to the Yocto release webpage (the download never failed due
to a wrong indentation of an else statement). For validation purposes,
use the sha256 checksum only.

Signed-off-by: Aleksandar Nikolic <aleksandar.nikolic@zeiss.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit b740d2f9d40aef1e18c022d1e82b4fb2c5c1fc22)
Signed-off-by: Aleksandar Nikolic <aleksandar.nikolic@zeiss.com>
---
 scripts/install-buildtools | 27 ++++++++++-----------------
 1 file changed, 10 insertions(+), 17 deletions(-)

diff --git a/scripts/install-buildtools b/scripts/install-buildtools
index 2218f3ffac..a34474ea84 100755
--- a/scripts/install-buildtools
+++ b/scripts/install-buildtools
@@ -238,19 +238,15 @@ def main():
         # Verify checksum
         if args.check:
             logger.info("Fetching buildtools installer checksum")
-            checksum_type = ""
-            for checksum_type in ["md5sum", "sha256sum"]:
-                check_url = "{}.{}".format(buildtools_url, checksum_type)
-                checksum_filename = "{}.{}".format(filename, checksum_type)
-                tmpbuildtools_checksum = os.path.join(tmpsdk_dir, checksum_filename)
-                ret = subprocess.call("wget -q -O %s %s" %
-                                      (tmpbuildtools_checksum, check_url), shell=True)
-                if ret == 0:
-                    break
-            else:
-                if ret != 0:
-                    logger.error("Could not download file from %s" % check_url)
-                    return ret
+            checksum_type = "sha256sum"
+            check_url = "{}.{}".format(buildtools_url, checksum_type)
+            checksum_filename = "{}.{}".format(filename, checksum_type)
+            tmpbuildtools_checksum = os.path.join(tmpsdk_dir, checksum_filename)
+            ret = subprocess.call("wget -q -O %s %s" %
+                                    (tmpbuildtools_checksum, check_url), shell=True)
+            if ret != 0:
+                logger.error("Could not download file from %s" % check_url)
+                return ret
             regex = re.compile(r"^(?P<checksum>[0-9a-f]+)\s+(?P<path>.*/)?(?P<filename>.*)$")
             with open(tmpbuildtools_checksum, 'rb') as f:
                 original = f.read()
@@ -263,10 +259,7 @@ def main():
                     logger.error("Filename does not match name in checksum")
                     return 1
                 checksum = m.group('checksum')
-            if checksum_type == "md5sum":
-                checksum_value = md5_file(tmpbuildtools)
-            else:
-                checksum_value = sha256_file(tmpbuildtools)
+            checksum_value = sha256_file(tmpbuildtools)
             if checksum == checksum_value:
                     logger.info("Checksum success")
             else:
-- 
2.25.1

[PATCH 3/4] install-buildtools: fix "test installation" step

[Aleksandar Nikolic]

From: Aleksandar Nikolic <aleksandar.nikolic@zeiss.com>

The "Test installation" step fails with some harmless error messages
(see [1]). This can however make a user think that the buildtools
have not been installed correctly.

Two reasons for the error messages:
- some envvars in the environment-setup-<arch>-pokysdk-linux file
  start and end with double quotes (e.g., PATH) and are as such
  written into python os.environ. This leads that their usage is
  not valid later when testing the installation. This patch removes
  the double quotes before writing, if they are present.
- if installation directory (install_dir), given through the option
  --directory, is given as a relative path, checking if the path to
  a tool (e.g., gcc) in buildtools starts it will always fail. This
  patch converts the install_dir variable to an absolute path.

[1]
ERROR: Something went wrong: tar not found in ./build-tools
ERROR: Something went wrong: installation failed

Signed-off-by: Aleksandar Nikolic <aleksandar.nikolic@zeiss.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit e4eb0b14ecf9bd2fba13260441c9d86eb348f41e)
Signed-off-by: Aleksandar Nikolic <aleksandar.nikolic@zeiss.com>
---
 scripts/install-buildtools | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/scripts/install-buildtools b/scripts/install-buildtools
index a34474ea84..4f85fe87d4 100755
--- a/scripts/install-buildtools
+++ b/scripts/install-buildtools
@@ -102,6 +102,16 @@ def sha256_file(filename):
     import hashlib
     return _hasher(hashlib.sha256(), filename)
 
+def remove_quotes(var):
+    """
+    If a variable starts and ends with double quotes, remove them.
+    Assumption: if a variable starts with double quotes, it must also
+    end with them.
+    """
+    if var[0] == '"':
+        var = var[1:-1]
+    return var
+
 
 def main():
     global DEFAULT_INSTALL_DIR
@@ -273,7 +283,7 @@ def main():
         os.chmod(tmpbuildtools, st.st_mode | stat.S_IEXEC)
         logger.debug(os.stat(tmpbuildtools))
         if args.directory:
-            install_dir = args.directory
+            install_dir = os.path.abspath(args.directory)
             ret = subprocess.call("%s -d %s -y" %
                                   (tmpbuildtools, install_dir), shell=True)
         else:
@@ -294,7 +304,7 @@ def main():
                 if match:
                     env_var = match.group('env_var')
                     logger.debug("env_var: %s" % env_var)
-                    env_val = match.group('env_val')
+                    env_val = remove_quotes(match.group('env_val'))
                     logger.debug("env_val: %s" % env_val)
                     os.environ[env_var] = env_val
 
-- 
2.25.1

[PATCH 4/4] scripts/install-buildtools: Update to 4.0.21

[Aleksandar Nikolic]

From: Aleksandar Nikolic <aleksandar.nikolic@zeiss.com>

Update to the 4.0.21 release of the 4.0 series for buildtools.

Signed-off-by: Aleksandar Nikolic <aleksandar.nikolic@zeiss.com>
---
 scripts/install-buildtools | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/scripts/install-buildtools b/scripts/install-buildtools
index 4f85fe87d4..9054eb5f36 100755
--- a/scripts/install-buildtools
+++ b/scripts/install-buildtools
@@ -56,9 +56,9 @@ PROGNAME = 'install-buildtools'
 logger = scriptutils.logger_create(PROGNAME, stream=sys.stdout)
 
 DEFAULT_INSTALL_DIR = os.path.join(os.path.split(scripts_path)[0],'buildtools')
-DEFAULT_BASE_URL = 'http://downloads.yoctoproject.org/releases/yocto'
-DEFAULT_RELEASE = 'yocto-4.1'
-DEFAULT_INSTALLER_VERSION = '4.1'
+DEFAULT_BASE_URL = 'https://downloads.yoctoproject.org/releases/yocto'
+DEFAULT_RELEASE = 'yocto-4.0.21'
+DEFAULT_INSTALLER_VERSION = '4.0.21'
 DEFAULT_BUILDDATE = '202110XX'
 
 # Python version sanity check
-- 
2.25.1

Re: [OE-core] [PATCH 4/4] scripts/install-buildtools: Update to 4.0.21

[Jose Quaresma]

[-- Attachment #1: Type: text/plain, Size: 1893 bytes --]

Hi Aleksandar,

Aleksandar Nikolic via lists.openembedded.org <aleksandar.nikolic010=
gmail.com@lists.openembedded.org> escreveu (segunda, 23/09/2024 à(s) 10:00):

> From: Aleksandar Nikolic <aleksandar.nikolic@zeiss.com>
>
> Update to the 4.0.21 release of the 4.0 series for buildtools.
>
> Signed-off-by: Aleksandar Nikolic <aleksandar.nikolic@zeiss.com>
> ---
>  scripts/install-buildtools | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/scripts/install-buildtools b/scripts/install-buildtools
> index 4f85fe87d4..9054eb5f36 100755
> --- a/scripts/install-buildtools
> +++ b/scripts/install-buildtools
> @@ -56,9 +56,9 @@ PROGNAME = 'install-buildtools'
>  logger = scriptutils.logger_create(PROGNAME, stream=sys.stdout)
>
>  DEFAULT_INSTALL_DIR =
> os.path.join(os.path.split(scripts_path)[0],'buildtools')
> -DEFAULT_BASE_URL = 'http://downloads.yoctoproject.org/releases/yocto'
> -DEFAULT_RELEASE = 'yocto-4.1'
> -DEFAULT_INSTALLER_VERSION = '4.1'
> +DEFAULT_BASE_URL = 'https://downloads.yoctoproject.org/releases/yocto'
> +DEFAULT_RELEASE = 'yocto-4.0.21'
> +DEFAULT_INSTALLER_VERSION = '4.0.21'
>

This first patch in the series updates this version to 4.1.
What is the reason to downgrade this now on the last patch?

Jose


>  DEFAULT_BUILDDATE = '202110XX'
>
>  # Python version sanity check
> --
> 2.25.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#204799):
> https://lists.openembedded.org/g/openembedded-core/message/204799
> Mute This Topic: https://lists.openembedded.org/mt/108604377/5052612
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> quaresma.jose@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

-- 
Best regards,

José Quaresma

[-- Attachment #2: Type: text/html, Size: 3539 bytes --]

Re: [PATCH 4/4] scripts/install-buildtools: Update to 4.0.21

[Aleksandar Nikolic]

[-- Attachment #1: Type: text/plain, Size: 1990 bytes --]

Hi Jose,

On Mon, Sep 23, 2024 at 11:06 AM, Jose Quaresma wrote:

> 
> Hi Aleksandar,
> 
> Aleksandar Nikolic via lists.openembedded.org (
> http://lists.openembedded.org ) <aleksandar.nikolic010= gmail.com@lists.openembedded.org
> > escreveu (segunda, 23/09/2024 à(s) 10:00):
> 
>> From: Aleksandar Nikolic < aleksandar.nikolic@zeiss.com >
>> 
>> Update to the 4.0.21 release of the 4.0 series for buildtools.
>> 
>> Signed-off-by: Aleksandar Nikolic < aleksandar.nikolic@zeiss.com >
>> ---
>> scripts/install-buildtools | 6 +++---
>> 1 file changed, 3 insertions(+), 3 deletions(-)
>> 
>> diff --git a/scripts/install-buildtools b/scripts/install-buildtools
>> index 4f85fe87d4..9054eb5f36 100755
>> --- a/scripts/install-buildtools
>> +++ b/scripts/install-buildtools
>> @@ -56,9 +56,9 @@ PROGNAME = 'install-buildtools'
>> logger = scriptutils.logger_create(PROGNAME, stream=sys.stdout)
>> 
>> DEFAULT_INSTALL_DIR =
>> os.path.join(os.path.split(scripts_path)[0],'buildtools')
>> -DEFAULT_BASE_URL = ' http://downloads.yoctoproject.org/releases/yocto '
>> -DEFAULT_RELEASE = 'yocto-4.1'
>> -DEFAULT_INSTALLER_VERSION = '4.1'
>> +DEFAULT_BASE_URL = ' https://downloads.yoctoproject.org/releases/yocto '
>> +DEFAULT_RELEASE = 'yocto-4.0.21'
>> +DEFAULT_INSTALLER_VERSION = '4.0.21'
> 
> 
> This first patch in the series updates this version to 4.1.
> What is the reason to downgrade this now on the last patch?
> 
> 

I was thinking since 4.1 (and higher versions) are EOL and since this is the kirkstone branch, the buildtools should also reflect the latest available release on the kirkstone branch, which is 4.0.21. But I am not really sure what's the right solution here, if someone else could chime in, I'd be thankful.

Aleksandar

> 
> 
> Jose
> 
> 
>> DEFAULT_BUILDDATE = '202110XX'
>> 
>> # Python version sanity check
>> --
>> 2.25.1
>> 
>> 
>> 
>> 
> 
> 
> 
> 
> --
> Best regards,
> 
> José Quaresma
>

[-- Attachment #2: Type: text/html, Size: 3334 bytes --]

Re: [OE-core] [PATCH 4/4] scripts/install-buildtools: Update to 4.0.21

[Ross Burton]

On 24 Sep 2024, at 08:57, Aleksandar Nikolic via lists.openembedded.org <aleksandar.nikolic010=gmail.com@lists.openembedded.org> wrote:
>    I was thinking since 4.1 (and higher versions) are EOL and since this is the kirkstone branch, the buildtools should also reflect the latest available release on the kirkstone branch, which is 4.0.21. But I am not really sure what's the right solution here, if someone else could chime in, I'd be thankful.

If these patches are for a stable branch then please make that obvious by putting [kirkstone] in the subject.

Ross

Re: [PATCH 4/4] scripts/install-buildtools: Update to 4.0.21

[Aleksandar Nikolic]

[-- Attachment #1: Type: text/plain, Size: 863 bytes --]

On Tue, Sep 24, 2024 at 01:30 PM, Ross Burton wrote:

> 
> On 24 Sep 2024, at 08:57, Aleksandar Nikolic via lists.openembedded.org
> <aleksandar.nikolic010=gmail.com@lists.openembedded.org> wrote:
> 
>> I was thinking since 4.1 (and higher versions) are EOL and since this is
>> the kirkstone branch, the buildtools should also reflect the latest
>> available release on the kirkstone branch, which is 4.0.21. But I am not
>> really sure what's the right solution here, if someone else could chime
>> in, I'd be thankful.
> 
> If these patches are for a stable branch then please make that obvious by
> putting [kirkstone] in the subject.

I forgot to do that and only saw it after the patches were sent out (did it correctly for scarthgap later). Should I send a new patch series (v2) with [kirkstone] in the subject?

Aleksandar

> 
> Ross

[-- Attachment #2: Type: text/html, Size: 970 bytes --]