[Buildroot] [PATCH v2 0/5] Misc utils/genrandconfig improvements

[Buildroot] [PATCH v2 0/5] Misc utils/genrandconfig improvements

[Thomas Petazzoni via buildroot]

Hello,

My starting point for this series was that I wanted to create some
configurations that have BR2_BACKUP_SITE="", so that we do not use the
Buildroot backup mirror, to ensure that upstream locations are still
around.

While implementing this, I realized that all the semi-random
configuration tweaks that we do in gen_config() are in fact completely
overridden by "make randconfig", when genrandconfig is used without
any toolchain CSV.

So I thought it was time for a bit of clean-up:

- Drop some fixups related to toolchains that are no longer tested
  (patch 1)

- Completely remove the support for using a toolchain CSV file. "make
  randconfig" support has been around for quite a while, it gives good
  results, so let's use that as the default now and only use fully
  random configurations (patch 2)

- Adapt the fine-tuning logic to the fact that "make randconfig"
  overwrites everything: move the fine-tuning logic *after* "make
  randconfig", but keep only what makes sense, since "make randconfig"
  already randomizes everything (patch 3)

- Implement what I wanted: set BR2_BACKUP_SITE="" for some builds
  (patch 4)

- Rework the logging of genrandconfig, as it is quite noisy right now,
  but only with irrelevant messages (patch 5)

Changes since v1:

- Fix PATCH 3/5, which was breaking the randomness. Indeed the tuning
  done *after* randconfig was done by overwriting the .config
  entirely, instead of appending to it (open mode was "w+", while it
  should have been "a")

Let me know what you think.

Thomas

Thomas Petazzoni (5):
  utils/genrandconfig: remove fixups related to untested CT-NG
    toolchains
  utils/genrandconfig: remove support for toolchain CSV
  utils/genrandconfig: rework fine-tuning logic
  utils/genrandconfig: do not use BR2_BACKUP_SITE for some builds
  utils/genrandconfig: improve logging

 utils/genrandconfig | 268 ++++++++------------------------------------
 1 file changed, 44 insertions(+), 224 deletions(-)

-- 
2.46.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH v2 1/5] utils/genrandconfig: remove fixups related to untested CT-NG toolchains

[Thomas Petazzoni via buildroot]

We have accumulated a whole bunch of very old fixups to avoid issues
with super old CT-NG toolchains, which we are not testing anymore, so
remove those fixups.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 utils/genrandconfig | 69 ---------------------------------------------
 1 file changed, 69 deletions(-)

diff --git a/utils/genrandconfig b/utils/genrandconfig
index 216e733799..cefc3dca12 100755
--- a/utils/genrandconfig
+++ b/utils/genrandconfig
@@ -181,43 +181,8 @@ async def fixup_config(sysinfo, configfile):
 
     ROOTFS_SIZE = '5G'
 
-    BR2_TOOLCHAIN_EXTERNAL_URL = 'BR2_TOOLCHAIN_EXTERNAL_URL="http://autobuild.buildroot.org/toolchains/tarballs/'
-
     if "BR2_NEEDS_HOST_JAVA=y\n" in configlines and not await sysinfo.has("java"):
         return False
-    # The ctng toolchain is affected by PR58854
-    if 'BR2_PACKAGE_LTTNG_TOOLS=y\n' in configlines and \
-       BR2_TOOLCHAIN_EXTERNAL_URL + 'armv5-ctng-linux-gnueabi.tar.xz"\n' in configlines:
-        return False
-    # The ctng toolchain tigger an assembler error with guile package when compiled with -Os (same issue as for CS ARM 2014.05-29)
-    if 'BR2_PACKAGE_GUILE=y\n' in configlines and \
-       'BR2_OPTIMIZE_S=y\n' in configlines and \
-       BR2_TOOLCHAIN_EXTERNAL_URL + 'armv5-ctng-linux-gnueabi.tar.xz"\n' in configlines:
-        return False
-    # The ctng toolchain is affected by PR58854
-    if 'BR2_PACKAGE_LTTNG_TOOLS=y\n' in configlines and \
-       BR2_TOOLCHAIN_EXTERNAL_URL + 'armv6-ctng-linux-uclibcgnueabi.tar.xz"\n' in configlines:
-        return False
-    # The ctng toolchain is affected by PR58854
-    if 'BR2_PACKAGE_LTTNG_TOOLS=y\n' in configlines and \
-       BR2_TOOLCHAIN_EXTERNAL_URL + 'armv7-ctng-linux-gnueabihf.tar.xz"\n' in configlines:
-        return False
-    # The ctng toolchain is affected by PR60155
-    if 'BR2_PACKAGE_SDL=y\n' in configlines and \
-       BR2_TOOLCHAIN_EXTERNAL_URL + 'powerpc-ctng-linux-uclibc.tar.xz"\n' in configlines:
-        return False
-    # The ctng toolchain is affected by PR60155
-    if 'BR2_PACKAGE_LIBMPEG2=y\n' in configlines and \
-       BR2_TOOLCHAIN_EXTERNAL_URL + 'powerpc-ctng-linux-uclibc.tar.xz"\n' in configlines:
-        return False
-    # This MIPS toolchain uses eglibc-2.18 which lacks SYS_getdents64
-    if 'BR2_PACKAGE_STRONGSWAN=y\n' in configlines and \
-       BR2_TOOLCHAIN_EXTERNAL_URL + 'mips64el-ctng_n64-linux-gnu.tar.xz"\n' in configlines:
-        return False
-    # This MIPS toolchain uses eglibc-2.18 which lacks SYS_getdents64
-    if 'BR2_PACKAGE_PYTHON3=y\n' in configlines and \
-       BR2_TOOLCHAIN_EXTERNAL_URL + 'mips64el-ctng_n64-linux-gnu.tar.xz"\n' in configlines:
-        return False
     # libffi not available on ARMv7-M, but propagating libffi arch
     # dependencies in Buildroot is really too much work, so we handle
     # this here.
@@ -232,40 +197,6 @@ async def fixup_config(sysinfo, configfile):
     if 'BR2_PACKAGE_SUNXI_BOARDS=y\n' in configlines:
         configlines.remove('BR2_PACKAGE_SUNXI_BOARDS_FEX_FILE=""\n')
         configlines.append('BR2_PACKAGE_SUNXI_BOARDS_FEX_FILE="a10/hackberry.fex"\n')
-    # This MIPS uClibc toolchain fails to build the gdb package
-    if 'BR2_PACKAGE_GDB=y\n' in configlines and \
-       BR2_TOOLCHAIN_EXTERNAL_URL + 'mipsel-ctng-linux-uclibc.tar.xz"\n' in configlines:
-        return False
-    # This MIPS uClibc toolchain fails to build the rt-tests package
-    if 'BR2_PACKAGE_RT_TESTS=y\n' in configlines and \
-       BR2_TOOLCHAIN_EXTERNAL_URL + 'mipsel-ctng-linux-uclibc.tar.xz"\n' in configlines:
-        return False
-    # This MIPS uClibc toolchain fails to build the civetweb package
-    if 'BR2_PACKAGE_CIVETWEB=y\n' in configlines and \
-       BR2_TOOLCHAIN_EXTERNAL_URL + 'mipsel-ctng-linux-uclibc.tar.xz"\n' in configlines:
-        return False
-    # This MIPS ctng toolchain fails to build the python3 package
-    if 'BR2_PACKAGE_PYTHON3=y\n' in configlines and \
-       BR2_TOOLCHAIN_EXTERNAL_URL + 'mips64el-ctng_n64-linux-gnu.tar.xz"\n' in configlines:
-        return False
-    # This MIPS uClibc toolchain fails to build the strace package
-    if 'BR2_PACKAGE_STRACE=y\n' in configlines and \
-       BR2_TOOLCHAIN_EXTERNAL_URL + 'mipsel-ctng-linux-uclibc.tar.xz"\n' in configlines:
-        return False
-    # This MIPS uClibc toolchain fails to build the cdrkit package
-    if 'BR2_PACKAGE_CDRKIT=y\n' in configlines and \
-       'BR2_STATIC_LIBS=y\n' in configlines and \
-       BR2_TOOLCHAIN_EXTERNAL_URL + 'mipsel-ctng-linux-uclibc.tar.xz"\n' in configlines:
-        return False
-    # uClibc vfork static linking issue
-    if 'BR2_PACKAGE_ALSA_LIB=y\n' in configlines and \
-       'BR2_STATIC_LIBS=y\n' in configlines and \
-       BR2_TOOLCHAIN_EXTERNAL_URL + 'i486-ctng-linux-uclibc.tar.xz"\n' in configlines:
-        return False
-    # This MIPS uClibc toolchain fails to build the weston package
-    if 'BR2_PACKAGE_WESTON=y\n' in configlines and \
-       BR2_TOOLCHAIN_EXTERNAL_URL + 'mipsel-ctng-linux-uclibc.tar.xz"\n' in configlines:
-        return False
     # No C library for internal toolchain
     if 'BR2_TOOLCHAIN_BUILDROOT_NONE=y\n' in configlines:
         return False
-- 
2.46.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH v2 2/5] utils/genrandconfig: remove support for toolchain CSV

[Thomas Petazzoni via buildroot]

Now that the support for generating a fully random configuration has
been well-tested, the whole mechanism based on a toolchain CSV isn't
really useful anymore, so let's drop it to simplify the logic.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 utils/genrandconfig | 99 +--------------------------------------------
 1 file changed, 2 insertions(+), 97 deletions(-)

diff --git a/utils/genrandconfig b/utils/genrandconfig
index cefc3dca12..ee094824fd 100755
--- a/utils/genrandconfig
+++ b/utils/genrandconfig
@@ -20,7 +20,6 @@
 
 from binascii import hexlify
 import asyncio
-import csv
 import os
 from random import randint
 import sys
@@ -100,73 +99,6 @@ class SystemInfo:
         return not missing_requirements
 
 
-def get_toolchain_configs(toolchains_csv, buildrootdir):
-    """Fetch and return the possible toolchain configurations
-
-    This function returns an array of toolchain configurations. Each
-    toolchain configuration is itself an array of lines of the defconfig.
-    """
-
-    with open(toolchains_csv) as r:
-        # filter empty lines and comments
-        lines = [t for t in r.readlines() if len(t.strip()) > 0 and t[0] != '#']
-        toolchains = lines
-    configs = []
-
-    (_, _, _, _, hostarch) = os.uname()
-    # ~2015 distros report x86 when on a 32bit install
-    if hostarch == 'i686' or hostarch == 'i386' or hostarch == 'x86':
-        hostarch = 'x86'
-
-    for row in csv.reader(toolchains):
-        config = {}
-        configfile = row[0]
-        config_hostarch = row[1]
-        keep = False
-
-        # Keep all toolchain configs that work regardless of the host
-        # architecture
-        if config_hostarch == "any":
-            keep = True
-
-        # Keep all toolchain configs that can work on the current host
-        # architecture
-        if hostarch == config_hostarch:
-            keep = True
-
-        # Assume that x86 32 bits toolchains work on x86_64 build
-        # machines
-        if hostarch == 'x86_64' and config_hostarch == "x86":
-            keep = True
-
-        if not keep:
-            continue
-
-        if not os.path.isabs(configfile):
-            configfile = os.path.join(buildrootdir, configfile)
-
-        with open(configfile) as r:
-            config = r.readlines()
-        configs.append(config)
-    return configs
-
-
-async def is_toolchain_usable(configfile, config):
-    """Check if the toolchain is actually usable."""
-
-    with open(configfile) as configf:
-        configlines = configf.readlines()
-
-    # Check that the toolchain configuration is still present
-    for toolchainline in config:
-        if toolchainline not in configlines:
-            print("WARN: toolchain can't be used", file=sys.stderr)
-            print("      Missing: %s" % toolchainline.strip(), file=sys.stderr)
-            return False
-
-    return True
-
-
 async def fixup_config(sysinfo, configfile):
     """Finalize the configuration and reject any problematic combinations
 
@@ -581,24 +513,11 @@ async def fixup_config(sysinfo, configfile):
 
 async def gen_config(args):
     """Generate a new random configuration
-
-    This function generates the configuration, by choosing a random
-    toolchain configuration and then generating a random selection of
-    packages.
     """
 
     sysinfo = SystemInfo()
 
-    if args.toolchains_csv:
-        # Select a random toolchain configuration
-        configs = get_toolchain_configs(args.toolchains_csv, args.buildrootdir)
-
-        i = randint(0, len(configs) - 1)
-        toolchainconfig = configs[i]
-    else:
-        toolchainconfig = []
-
-    configlines = list(toolchainconfig)
+    configlines = list()
 
     # Combine with the minimal configuration
     minimalconfigfile = os.path.join(args.buildrootdir,
@@ -658,9 +577,6 @@ async def gen_config(args):
     if ret:
         return ret
 
-    if not await is_toolchain_usable(configfile, toolchainconfig):
-        return 2
-
     # Now, generate the random selection of packages, and fixup
     # things if needed.
     # Safe-guard, in case we can not quickly come to a valid
@@ -676,7 +592,7 @@ async def gen_config(args):
             "make", "O=%s" % args.outputdir, "-C", args.buildrootdir,
             "KCONFIG_SEED=0x%s" % hexlify(os.urandom(4)).decode("ascii").upper(),
             "KCONFIG_PROBABILITY=%d" % randint(1, 20),
-            "randpackageconfig" if args.toolchains_csv else "randconfig")
+            "randconfig")
         ret = await proc.wait()
         if ret:
             return ret
@@ -711,17 +627,6 @@ if __name__ == '__main__':
                         help="Buildroot directory (relative to current directory)",
                         type=str, default='.')
 
-    toolchains_csv = parser.add_mutually_exclusive_group(required=False)
-    toolchains_csv.add_argument("--toolchains-csv",
-                                dest="toolchains_csv",
-                                help="Path of the toolchain configuration file",
-                                type=str)
-    toolchains_csv.add_argument("--no-toolchains-csv",
-                                dest="toolchains_csv",
-                                help="Generate random toolchain configuration",
-                                action='store_false')
-    parser.set_defaults(toolchains_csv="support/config-fragments/autobuild/toolchain-configs.csv")
-
     args = parser.parse_args()
 
     # We need the absolute path to use with O=, because the relative
-- 
2.46.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH v2 3/5] utils/genrandconfig: rework fine-tuning logic

[Thomas Petazzoni via buildroot]

Before calling randpackageconfig/randconfig, we were pre-generating a
snippet of .config with:

 (1) minimal.config
 (2) BR2_CURL/BR2_WGET settings
 (3) some random selection of init system, debug, runtime debug, etc
 (4) enabling BR2_REPRODUCIBLE=y when diffoscope was found

Now that we only use randconfig, this whole fine-tuning is completely
irrelevant, as it gets overridden by "make randconfig".

(1) and (3) above are useless, as randconfig does all the
randomization that is needed.

However, we want to preserve (2) and (4) above, so we re-implement
those fixups, but *after* randconfig has done its job.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
Changes since v1:

- Restore proper randomness. Indeed the tuning done *after* randconfig
  was done by overwriting the .config entirely, instead of appending
  to it (open mode was "w+", while it should have been "a")

 utils/genrandconfig | 71 ++++++++++++---------------------------------
 1 file changed, 18 insertions(+), 53 deletions(-)

diff --git a/utils/genrandconfig b/utils/genrandconfig
index ee094824fd..0ccb6d83a6 100755
--- a/utils/genrandconfig
+++ b/utils/genrandconfig
@@ -517,65 +517,15 @@ async def gen_config(args):
 
     sysinfo = SystemInfo()
 
-    configlines = list()
-
-    # Combine with the minimal configuration
-    minimalconfigfile = os.path.join(args.buildrootdir,
-                                     'support/config-fragments/minimal.config')
-    with open(minimalconfigfile) as minimalf:
-        configlines += minimalf.readlines()
-
-    # Allow hosts with old certificates to download over https
-    configlines.append("BR2_WGET=\"wget -nd -t 3 --no-check-certificate\"\n")
-    configlines.append("BR2_CURL=\"curl --ftp-pasv --retry 3 --insecure\"\n")
-
-    # Per-package folder
-    if randint(0, 15) == 0:
-        configlines.append("BR2_PER_PACKAGE_DIRECTORIES=y\n")
-
-    # Amend the configuration with a few things.
-    if randint(0, 20) == 0:
-        configlines.append("BR2_ENABLE_DEBUG=y\n")
-    if randint(0, 20) == 0:
-        configlines.append("BR2_ENABLE_RUNTIME_DEBUG=y\n")
-    if randint(0, 1) == 0:
-        configlines.append("BR2_INIT_BUSYBOX=y\n")
-    elif randint(0, 15) == 0:
-        configlines.append("BR2_INIT_SYSTEMD=y\n")
-    elif randint(0, 10) == 0:
-        configlines.append("BR2_ROOTFS_DEVICE_CREATION_DYNAMIC_EUDEV=y\n")
-    if randint(0, 20) == 0:
-        configlines.append("BR2_STATIC_LIBS=y\n")
-    if randint(0, 20) == 0:
-        configlines.append("BR2_PACKAGE_PYTHON3_PY_ONLY=y\n")
-    if randint(0, 5) == 0:
-        configlines.append("BR2_OPTIMIZE_2=y\n")
-    if randint(0, 4) == 0:
-        configlines.append("BR2_SYSTEM_ENABLE_NLS=y\n")
-    if randint(0, 4) == 0:
-        configlines.append("BR2_FORTIFY_SOURCE_2=y\n")
-
-    # Randomly enable BR2_REPRODUCIBLE 10% of times
-    # also enable tar filesystem images for testing
-    if await sysinfo.has("diffoscope") and randint(0, 10) == 0:
-        configlines.append("BR2_REPRODUCIBLE=y\n")
-        configlines.append("BR2_TARGET_ROOTFS_TAR=y\n")
-
-    # Write out the configuration file
+    # Create output directory
     if not os.path.exists(args.outputdir):
         os.makedirs(args.outputdir)
+
+    # Calculate path to config file
     if args.outputdir == os.path.abspath(os.path.join(args.buildrootdir, "output")):
         configfile = os.path.join(args.buildrootdir, ".config")
     else:
         configfile = os.path.join(args.outputdir, ".config")
-    with open(configfile, "w+") as configf:
-        configf.writelines(configlines)
-
-    proc = await asyncio.create_subprocess_exec(
-        "make", "O=%s" % args.outputdir, "-C", args.buildrootdir, "olddefconfig")
-    ret = await proc.wait()
-    if ret:
-        return ret
 
     # Now, generate the random selection of packages, and fixup
     # things if needed.
@@ -600,6 +550,21 @@ async def gen_config(args):
         if await fixup_config(sysinfo, configfile):
             break
 
+    configlines = list()
+
+    # Allow hosts with old certificates to download over https
+    configlines.append("BR2_WGET=\"wget -nd -t 3 --no-check-certificate\"\n")
+    configlines.append("BR2_CURL=\"curl --ftp-pasv --retry 3 --insecure\"\n")
+
+    # Randomly enable BR2_REPRODUCIBLE 10% of times
+    # also enable tar filesystem images for testing
+    if await sysinfo.has("diffoscope") and randint(0, 10) == 0:
+        configlines.append("BR2_REPRODUCIBLE=y\n")
+        configlines.append("BR2_TARGET_ROOTFS_TAR=y\n")
+
+    with open(configfile, "a") as configf:
+        configf.writelines(configlines)
+
     proc = await asyncio.create_subprocess_exec(
         "make", "O=%s" % args.outputdir, "-C", args.buildrootdir, "olddefconfig")
     ret = await proc.wait()
-- 
2.46.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH v2 4/5] utils/genrandconfig: do not use BR2_BACKUP_SITE for some builds

[Thomas Petazzoni via buildroot]

In order to test that upstream sites are still working, we need to NOT
fallback to sources.buildroot.net for some builds.

As there is anyway a local cache in the autobuilder instances, we need
to do quite a lot of builds without any BR2_BACKUP_SITE configured to
have a chance to catch issues, which is why a 50% chance is used to
unset BR2_BACKUP_SITE.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 utils/genrandconfig | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/utils/genrandconfig b/utils/genrandconfig
index 0ccb6d83a6..66296e1954 100755
--- a/utils/genrandconfig
+++ b/utils/genrandconfig
@@ -562,6 +562,11 @@ async def gen_config(args):
         configlines.append("BR2_REPRODUCIBLE=y\n")
         configlines.append("BR2_TARGET_ROOTFS_TAR=y\n")
 
+    # From time to time, ignore sources.buildroot.net to really fetch
+    # from upstream
+    if randint(0, 1) == 0:
+        configlines.append("""BR2_BACKUP_SITE=""\n""")
+
     with open(configfile, "a") as configf:
         configf.writelines(configlines)
 
-- 
2.46.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH v2 5/5] utils/genrandconfig: improve logging

[Thomas Petazzoni via buildroot]

Right now, genrandconfig just spits out the random messages from the
different make invocations, which isn't terribly useful. Instead,
let's redirect the output of make invocations to oblivion, and add
some more high level logging.

As part of this logging, we're interested to see how many iterations
were needed to find a valid configuration, so changed the loop logic
to count from 0 to 100 instead of from 100 to 0 so that we can easily
show the iteration number.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 utils/genrandconfig | 28 +++++++++++++++++++++-------
 1 file changed, 21 insertions(+), 7 deletions(-)

diff --git a/utils/genrandconfig b/utils/genrandconfig
index 66296e1954..7ff47b12cc 100755
--- a/utils/genrandconfig
+++ b/utils/genrandconfig
@@ -531,23 +531,27 @@ async def gen_config(args):
     # things if needed.
     # Safe-guard, in case we can not quickly come to a valid
     # configuration: allow at most 100 (arbitrary) iterations.
-    bounded_loop = 100
+    loop = 0
     while True:
-        if bounded_loop == 0:
+        if loop == 100:
             print("ERROR: cannot generate random configuration after 100 iterations",
                   file=sys.stderr)
             return 1
-        bounded_loop -= 1
+        print("Generating configuration, loop %d" % loop)
+        loop += 1
         proc = await asyncio.create_subprocess_exec(
             "make", "O=%s" % args.outputdir, "-C", args.buildrootdir,
             "KCONFIG_SEED=0x%s" % hexlify(os.urandom(4)).decode("ascii").upper(),
             "KCONFIG_PROBABILITY=%d" % randint(1, 20),
-            "randconfig")
+            "randconfig",
+            stdout=asyncio.subprocess.DEVNULL,
+            stderr=asyncio.subprocess.DEVNULL)
         ret = await proc.wait()
         if ret:
             return ret
 
         if await fixup_config(sysinfo, configfile):
+            print("  configuration valid")
             break
 
     configlines = list()
@@ -570,20 +574,30 @@ async def gen_config(args):
     with open(configfile, "a") as configf:
         configf.writelines(configlines)
 
+    print("  olddefconfig")
     proc = await asyncio.create_subprocess_exec(
-        "make", "O=%s" % args.outputdir, "-C", args.buildrootdir, "olddefconfig")
+        "make", "O=%s" % args.outputdir, "-C", args.buildrootdir, "olddefconfig",
+        stdout=asyncio.subprocess.DEVNULL,
+        stderr=asyncio.subprocess.DEVNULL)
     ret = await proc.wait()
     if ret:
         return ret
 
+    print("  savedefconfig")
     proc = await asyncio.create_subprocess_exec(
-        "make", "O=%s" % args.outputdir, "-C", args.buildrootdir, "savedefconfig")
+        "make", "O=%s" % args.outputdir, "-C", args.buildrootdir, "savedefconfig",
+        stdout=asyncio.subprocess.DEVNULL,
+        stderr=asyncio.subprocess.DEVNULL)
+
     ret = await proc.wait()
     if ret:
         return ret
 
+    print("  dependencies")
     proc = await asyncio.create_subprocess_exec(
-        "make", "O=%s" % args.outputdir, "-C", args.buildrootdir, "dependencies")
+        "make", "O=%s" % args.outputdir, "-C", args.buildrootdir, "dependencies",
+        stdout=asyncio.subprocess.DEVNULL,
+        stderr=asyncio.subprocess.DEVNULL)
     return await proc.wait()
 
 
-- 
2.46.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 0/5] Misc utils/genrandconfig improvements

[Yann E. MORIN]

Thomas, All,

On 2024-08-18 11:03 +0200, Thomas Petazzoni via buildroot spake thusly:
> Hello,
> 
> My starting point for this series was that I wanted to create some
> configurations that have BR2_BACKUP_SITE="", so that we do not use the
> Buildroot backup mirror, to ensure that upstream locations are still
> around.
> 
> While implementing this, I realized that all the semi-random
> configuration tweaks that we do in gen_config() are in fact completely
> overridden by "make randconfig", when genrandconfig is used without
> any toolchain CSV.

Series applied to master, thanks. I did some changes on patch 2, so I'll
reply there as well..

Regards,
Yann E. MORIN.

> So I thought it was time for a bit of clean-up:
> 
> - Drop some fixups related to toolchains that are no longer tested
>   (patch 1)
> 
> - Completely remove the support for using a toolchain CSV file. "make
>   randconfig" support has been around for quite a while, it gives good
>   results, so let's use that as the default now and only use fully
>   random configurations (patch 2)
> 
> - Adapt the fine-tuning logic to the fact that "make randconfig"
>   overwrites everything: move the fine-tuning logic *after* "make
>   randconfig", but keep only what makes sense, since "make randconfig"
>   already randomizes everything (patch 3)
> 
> - Implement what I wanted: set BR2_BACKUP_SITE="" for some builds
>   (patch 4)
> 
> - Rework the logging of genrandconfig, as it is quite noisy right now,
>   but only with irrelevant messages (patch 5)
> 
> Changes since v1:
> 
> - Fix PATCH 3/5, which was breaking the randomness. Indeed the tuning
>   done *after* randconfig was done by overwriting the .config
>   entirely, instead of appending to it (open mode was "w+", while it
>   should have been "a")
> 
> Let me know what you think.
> 
> Thomas
> 
> Thomas Petazzoni (5):
>   utils/genrandconfig: remove fixups related to untested CT-NG
>     toolchains
>   utils/genrandconfig: remove support for toolchain CSV
>   utils/genrandconfig: rework fine-tuning logic
>   utils/genrandconfig: do not use BR2_BACKUP_SITE for some builds
>   utils/genrandconfig: improve logging
> 
>  utils/genrandconfig | 268 ++++++++------------------------------------
>  1 file changed, 44 insertions(+), 224 deletions(-)
> 
> -- 
> 2.46.0
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 2/5] utils/genrandconfig: remove support for toolchain CSV

[Yann E. MORIN]

Thomas, All,

On 2024-08-18 11:03 +0200, Thomas Petazzoni via buildroot spake thusly:
> Now that the support for generating a fully random configuration has
> been well-tested, the whole mechanism based on a toolchain CSV isn't
> really useful anymore, so let's drop it to simplify the logic.

As discussed on IRC, the autobuilder code still uses
--{,no-}toolchains-csv, so dopping those flags would instaly break the
autobuilders.

So, I've kept them and docuemnted them to be legacy.

Applied to master, thanks.

Regards,
Yann E. MORIN.

> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  utils/genrandconfig | 99 +--------------------------------------------
>  1 file changed, 2 insertions(+), 97 deletions(-)
> 
> diff --git a/utils/genrandconfig b/utils/genrandconfig
> index cefc3dca12..ee094824fd 100755
> --- a/utils/genrandconfig
> +++ b/utils/genrandconfig
> @@ -20,7 +20,6 @@
>  
>  from binascii import hexlify
>  import asyncio
> -import csv
>  import os
>  from random import randint
>  import sys
> @@ -100,73 +99,6 @@ class SystemInfo:
>          return not missing_requirements
>  
>  
> -def get_toolchain_configs(toolchains_csv, buildrootdir):
> -    """Fetch and return the possible toolchain configurations
> -
> -    This function returns an array of toolchain configurations. Each
> -    toolchain configuration is itself an array of lines of the defconfig.
> -    """
> -
> -    with open(toolchains_csv) as r:
> -        # filter empty lines and comments
> -        lines = [t for t in r.readlines() if len(t.strip()) > 0 and t[0] != '#']
> -        toolchains = lines
> -    configs = []
> -
> -    (_, _, _, _, hostarch) = os.uname()
> -    # ~2015 distros report x86 when on a 32bit install
> -    if hostarch == 'i686' or hostarch == 'i386' or hostarch == 'x86':
> -        hostarch = 'x86'
> -
> -    for row in csv.reader(toolchains):
> -        config = {}
> -        configfile = row[0]
> -        config_hostarch = row[1]
> -        keep = False
> -
> -        # Keep all toolchain configs that work regardless of the host
> -        # architecture
> -        if config_hostarch == "any":
> -            keep = True
> -
> -        # Keep all toolchain configs that can work on the current host
> -        # architecture
> -        if hostarch == config_hostarch:
> -            keep = True
> -
> -        # Assume that x86 32 bits toolchains work on x86_64 build
> -        # machines
> -        if hostarch == 'x86_64' and config_hostarch == "x86":
> -            keep = True
> -
> -        if not keep:
> -            continue
> -
> -        if not os.path.isabs(configfile):
> -            configfile = os.path.join(buildrootdir, configfile)
> -
> -        with open(configfile) as r:
> -            config = r.readlines()
> -        configs.append(config)
> -    return configs
> -
> -
> -async def is_toolchain_usable(configfile, config):
> -    """Check if the toolchain is actually usable."""
> -
> -    with open(configfile) as configf:
> -        configlines = configf.readlines()
> -
> -    # Check that the toolchain configuration is still present
> -    for toolchainline in config:
> -        if toolchainline not in configlines:
> -            print("WARN: toolchain can't be used", file=sys.stderr)
> -            print("      Missing: %s" % toolchainline.strip(), file=sys.stderr)
> -            return False
> -
> -    return True
> -
> -
>  async def fixup_config(sysinfo, configfile):
>      """Finalize the configuration and reject any problematic combinations
>  
> @@ -581,24 +513,11 @@ async def fixup_config(sysinfo, configfile):
>  
>  async def gen_config(args):
>      """Generate a new random configuration
> -
> -    This function generates the configuration, by choosing a random
> -    toolchain configuration and then generating a random selection of
> -    packages.
>      """
>  
>      sysinfo = SystemInfo()
>  
> -    if args.toolchains_csv:
> -        # Select a random toolchain configuration
> -        configs = get_toolchain_configs(args.toolchains_csv, args.buildrootdir)
> -
> -        i = randint(0, len(configs) - 1)
> -        toolchainconfig = configs[i]
> -    else:
> -        toolchainconfig = []
> -
> -    configlines = list(toolchainconfig)
> +    configlines = list()
>  
>      # Combine with the minimal configuration
>      minimalconfigfile = os.path.join(args.buildrootdir,
> @@ -658,9 +577,6 @@ async def gen_config(args):
>      if ret:
>          return ret
>  
> -    if not await is_toolchain_usable(configfile, toolchainconfig):
> -        return 2
> -
>      # Now, generate the random selection of packages, and fixup
>      # things if needed.
>      # Safe-guard, in case we can not quickly come to a valid
> @@ -676,7 +592,7 @@ async def gen_config(args):
>              "make", "O=%s" % args.outputdir, "-C", args.buildrootdir,
>              "KCONFIG_SEED=0x%s" % hexlify(os.urandom(4)).decode("ascii").upper(),
>              "KCONFIG_PROBABILITY=%d" % randint(1, 20),
> -            "randpackageconfig" if args.toolchains_csv else "randconfig")
> +            "randconfig")
>          ret = await proc.wait()
>          if ret:
>              return ret
> @@ -711,17 +627,6 @@ if __name__ == '__main__':
>                          help="Buildroot directory (relative to current directory)",
>                          type=str, default='.')
>  
> -    toolchains_csv = parser.add_mutually_exclusive_group(required=False)
> -    toolchains_csv.add_argument("--toolchains-csv",
> -                                dest="toolchains_csv",
> -                                help="Path of the toolchain configuration file",
> -                                type=str)
> -    toolchains_csv.add_argument("--no-toolchains-csv",
> -                                dest="toolchains_csv",
> -                                help="Generate random toolchain configuration",
> -                                action='store_false')
> -    parser.set_defaults(toolchains_csv="support/config-fragments/autobuild/toolchain-configs.csv")
> -
>      args = parser.parse_args()
>  
>      # We need the absolute path to use with O=, because the relative
> -- 
> 2.46.0
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot