* [Buildroot] [PATCH 1/2] support/scripts: add check-dotconfig.py
@ 2020-04-20 9:17 Romain Naour
2020-04-20 9:17 ` [Buildroot] [PATCH 2/2] gitlab-ci: check generated config files Romain Naour
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: Romain Naour @ 2020-04-20 9:17 UTC (permalink / raw)
To: buildroot
For the same reason as for 50b747f212be2c9c0f7cf10c674ed488d042715c,
we need to check if the generated configuration file (.config)
contains all symbols present in the defconfig file.
If not there is an issue with the defconfig.
This script will be used in .gitlab-ci.yml.
Inspired by is_toolchain_usable() function from genrandconfig:
https://git.busybox.net/buildroot/tree/utils/genrandconfig?h=2020.02#n164
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
support/scripts/check-dotconfig.py | 37 ++++++++++++++++++++++++++++++
1 file changed, 37 insertions(+)
create mode 100755 support/scripts/check-dotconfig.py
diff --git a/support/scripts/check-dotconfig.py b/support/scripts/check-dotconfig.py
new file mode 100755
index 0000000000..9e60810c1d
--- /dev/null
+++ b/support/scripts/check-dotconfig.py
@@ -0,0 +1,37 @@
+#!/usr/bin/env python3
+
+# This script check if the .config is contains all lines present in the defconfig.
+
+import sys
+
+
+def main():
+ if not (len(sys.argv) == 3):
+ print("Error: incorrect number of arguments")
+ print("""Usage: check-dotconfig <configfile> <defconfig>""")
+ sys.exit(1)
+
+ configfile = sys.argv[1]
+ defconfig = sys.argv[2]
+
+ with open(configfile) as configf:
+ configlines = configf.readlines()
+
+ defconfiglines = []
+ with open(defconfig) as defconfigf:
+ for line in defconfigf.readlines():
+ if line.startswith("BR2_"):
+ defconfiglines.append(line)
+
+ # Check that all the defconfig lines are still present
+ for defconfigline in defconfiglines:
+ if defconfigline not in configlines:
+ print("WARN: defconfig can't be used\n")
+ print(" Missing: %s\n" % defconfigline.strip())
+ return False
+
+ return True
+
+
+if __name__ == "__main__":
+ main()
--
2.25.3
^ permalink raw reply related [flat|nested] 6+ messages in thread* [Buildroot] [PATCH 2/2] gitlab-ci: check generated config files
2020-04-20 9:17 [Buildroot] [PATCH 1/2] support/scripts: add check-dotconfig.py Romain Naour
@ 2020-04-20 9:17 ` Romain Naour
2020-04-20 11:52 ` [Buildroot] [PATCH 1/2] support/scripts: add check-dotconfig.py Thomas Petazzoni
2020-04-20 13:06 ` Yann E. MORIN
2 siblings, 0 replies; 6+ messages in thread
From: Romain Naour @ 2020-04-20 9:17 UTC (permalink / raw)
To: buildroot
Use the script added by the previous patch to check
generated config files.
Tested on gitlab:
https://gitlab.com/kubu93/buildroot/pipelines/137597966
Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
No particular issues was found on existing defconfigs.
---
.gitlab-ci.yml | 1 +
.gitlab-ci.yml.in | 1 +
2 files changed, 2 insertions(+)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index fa8e077a07..0d06a1b7cf 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -47,6 +47,7 @@ check-package:
script:
- echo "Configure Buildroot for ${DEFCONFIG_NAME}"
- make ${DEFCONFIG_NAME}
+ - ./support/scripts/check-dotconfig.py .config ./configs/${DEFCONFIG_NAME}
- echo 'Build buildroot'
- |
make > >(tee build.log |grep '>>>') 2>&1 || {
diff --git a/.gitlab-ci.yml.in b/.gitlab-ci.yml.in
index 6b09730a65..413c6d2956 100644
--- a/.gitlab-ci.yml.in
+++ b/.gitlab-ci.yml.in
@@ -47,6 +47,7 @@ check-package:
script:
- echo "Configure Buildroot for ${DEFCONFIG_NAME}"
- make ${DEFCONFIG_NAME}
+ - ./support/scripts/check-dotconfig.py .config ./configs/${DEFCONFIG_NAME}
- echo 'Build buildroot'
- |
make > >(tee build.log |grep '>>>') 2>&1 || {
--
2.25.3
^ permalink raw reply related [flat|nested] 6+ messages in thread* [Buildroot] [PATCH 1/2] support/scripts: add check-dotconfig.py
2020-04-20 9:17 [Buildroot] [PATCH 1/2] support/scripts: add check-dotconfig.py Romain Naour
2020-04-20 9:17 ` [Buildroot] [PATCH 2/2] gitlab-ci: check generated config files Romain Naour
@ 2020-04-20 11:52 ` Thomas Petazzoni
2020-04-20 12:19 ` Romain Naour
2020-04-20 13:06 ` Yann E. MORIN
2 siblings, 1 reply; 6+ messages in thread
From: Thomas Petazzoni @ 2020-04-20 11:52 UTC (permalink / raw)
To: buildroot
Hello,
On Mon, 20 Apr 2020 11:17:56 +0200
Romain Naour <romain.naour@gmail.com> wrote:
> For the same reason as for 50b747f212be2c9c0f7cf10c674ed488d042715c,
> we need to check if the generated configuration file (.config)
> contains all symbols present in the defconfig file.
Then can we also make the runtime-test infrastructure use this
check-dotconfig.py script ?
No need to do it right now, in this patch or patch series, but just
something that would be good to have.
Thanks!
Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 6+ messages in thread
* [Buildroot] [PATCH 1/2] support/scripts: add check-dotconfig.py
2020-04-20 11:52 ` [Buildroot] [PATCH 1/2] support/scripts: add check-dotconfig.py Thomas Petazzoni
@ 2020-04-20 12:19 ` Romain Naour
2020-04-20 12:38 ` Thomas Petazzoni
0 siblings, 1 reply; 6+ messages in thread
From: Romain Naour @ 2020-04-20 12:19 UTC (permalink / raw)
To: buildroot
Hello Thomas,
Le 20/04/2020 ? 13:52, Thomas Petazzoni a ?crit?:
> Hello,
>
> On Mon, 20 Apr 2020 11:17:56 +0200
> Romain Naour <romain.naour@gmail.com> wrote:
>
>> For the same reason as for 50b747f212be2c9c0f7cf10c674ed488d042715c,
>> we need to check if the generated configuration file (.config)
>> contains all symbols present in the defconfig file.
>
> Then can we also make the runtime-test infrastructure use this
> check-dotconfig.py script ?
Not with the current implementation of the runtime-test infrastructure since the
defconfig is bundled in the test python script.
The defconfig is written to .config but then we run make olddefconfig to
generate the real .config. So we can't compare the .config file with
check-dotconfig.py without creating a defconfig file.
>
> No need to do it right now, in this patch or patch series, but just
> something that would be good to have.
This can be easily be done in a followup patch.
Best regards,
Romain
>
> Thanks!
>
> Thomas
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* [Buildroot] [PATCH 1/2] support/scripts: add check-dotconfig.py
2020-04-20 12:19 ` Romain Naour
@ 2020-04-20 12:38 ` Thomas Petazzoni
0 siblings, 0 replies; 6+ messages in thread
From: Thomas Petazzoni @ 2020-04-20 12:38 UTC (permalink / raw)
To: buildroot
On Mon, 20 Apr 2020 14:19:55 +0200
Romain Naour <romain.naour@gmail.com> wrote:
> Not with the current implementation of the runtime-test infrastructure since the
> defconfig is bundled in the test python script.
>
> The defconfig is written to .config but then we run make olddefconfig to
> generate the real .config. So we can't compare the .config file with
> check-dotconfig.py without creating a defconfig file.
Well, we can always have the defconfig written to a different file,
then copied to .config, that's not a big change to introduce.
> This can be easily be done in a followup patch.
Excellent, thanks!
Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 6+ messages in thread
* [Buildroot] [PATCH 1/2] support/scripts: add check-dotconfig.py
2020-04-20 9:17 [Buildroot] [PATCH 1/2] support/scripts: add check-dotconfig.py Romain Naour
2020-04-20 9:17 ` [Buildroot] [PATCH 2/2] gitlab-ci: check generated config files Romain Naour
2020-04-20 11:52 ` [Buildroot] [PATCH 1/2] support/scripts: add check-dotconfig.py Thomas Petazzoni
@ 2020-04-20 13:06 ` Yann E. MORIN
2 siblings, 0 replies; 6+ messages in thread
From: Yann E. MORIN @ 2020-04-20 13:06 UTC (permalink / raw)
To: buildroot
Romain, All,
On 2020-04-20 11:17 +0200, Romain Naour spake thusly:
> For the same reason as for 50b747f212be2c9c0f7cf10c674ed488d042715c,
> we need to check if the generated configuration file (.config)
> contains all symbols present in the defconfig file.
>
> If not there is an issue with the defconfig.
>
> This script will be used in .gitlab-ci.yml.
>
> Inspired by is_toolchain_usable() function from genrandconfig:
> https://git.busybox.net/buildroot/tree/utils/genrandconfig?h=2020.02#n164
>
> Signed-off-by: Romain Naour <romain.naour@gmail.com>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
> support/scripts/check-dotconfig.py | 37 ++++++++++++++++++++++++++++++
> 1 file changed, 37 insertions(+)
> create mode 100755 support/scripts/check-dotconfig.py
>
> diff --git a/support/scripts/check-dotconfig.py b/support/scripts/check-dotconfig.py
> new file mode 100755
> index 0000000000..9e60810c1d
> --- /dev/null
> +++ b/support/scripts/check-dotconfig.py
> @@ -0,0 +1,37 @@
> +#!/usr/bin/env python3
> +
> +# This script check if the .config is contains all lines present in the defconfig.
> +
> +import sys
> +
> +
> +def main():
> + if not (len(sys.argv) == 3):
> + print("Error: incorrect number of arguments")
> + print("""Usage: check-dotconfig <configfile> <defconfig>""")
> + sys.exit(1)
> +
> + configfile = sys.argv[1]
> + defconfig = sys.argv[2]
> +
> + with open(configfile) as configf:
> + configlines = configf.readlines()
> +
> + defconfiglines = []
> + with open(defconfig) as defconfigf:
> + for line in defconfigf.readlines():
> + if line.startswith("BR2_"):
> + defconfiglines.append(line)
And what about lines that match '# foo is not set' ?
We also want to ensure those are still present too.
> + # Check that all the defconfig lines are still present
> + for defconfigline in defconfiglines:
> + if defconfigline not in configlines:
> + print("WARN: defconfig can't be used\n")
> + print(" Missing: %s\n" % defconfigline.strip())
> + return False
This only reports a single missing variable, while there can more than
one.
Here is a proposal for an updated script:
#!/usr/bin/env python3
# This scripts check that all lines present in the defconfig are
# still present in the .config
import sys
def main():
if not (len(sys.argv) == 3):
print("Error: incorrect number of arguments")
print("""Usage: check-dotconfig <configfile> <defconfig>""")
sys.exit(1)
configfile = sys.argv[1]
defconfig = sys.argv[2]
# strip() to get rid of trailing \n
with open(configfile) as configf:
configlines = [l.strip() for l in configf.readlines()]
defconfiglines = []
with open(defconfig) as defconfigf:
for line in defconfigf.readlines():
# strip() to get rid of trailing \n
line = line.strip()
if line.startswith("BR2_"):
defconfiglines.append(line)
elif line.startswith('# BR2_') and line.endswith(' is not set'):
defconfiglines.append(line)
# Check that all the defconfig lines are still present
missing = [defconfigline.strip() for defconfigline in defconfiglines
if defconfigline not in configlines]
if len(missing):
print("WARN: defconfig {} can't be used:".format(defconfig))
for m in missing:
print(" Missing: {}".format(m))
sys.exit(1 if len(missing) else 0)
if __name__ == "__main__":
main()
And with this script, I ran:
$ for cfg in $(make list-defconfigs |sed -r -e '/ - /!d; s/ (.*) - .*/\1/'); do
make "${cfg}" >/dev/null 2>&1
support/scripts/check-dotconfig.py .config configs/"${cfg}"
done
WARN: defconfig configs/amarula_a64_relic_defconfig can't be used:
Missing: BR2_PACKAGE_HOST_ANDROID_TOOLS_FASTBOOT=y
WARN: defconfig configs/beaglebone_qt5_defconfig can't be used:
Missing: BR2_PACKAGE_QT5=y
Missing: BR2_PACKAGE_QT5BASE_EXAMPLES=y
Missing: BR2_PACKAGE_QT5BASE_EGLFS=y
Missing: BR2_PACKAGE_QT5BASE_DEFAULT_QPA="wayland"
Missing: BR2_PACKAGE_QT5QUICKCONTROLS=y
Missing: BR2_PACKAGE_QT5WAYLAND=y
Missing: BR2_PACKAGE_QT5WAYLAND_COMPOSITOR=y
WARN: defconfig configs/engicam_imx6qdl_icore_qt5_defconfig can't be used:
Missing: BR2_PACKAGE_QT5=y
Missing: BR2_PACKAGE_QT5BASE_LICENSE_APPROVED=y
Missing: BR2_PACKAGE_QT5BASE_OPENGL_LIB=y
Missing: BR2_PACKAGE_QT5BASE_LINUXFB=y
Missing: BR2_PACKAGE_QT5BASE_FONTCONFIG=y
Missing: BR2_PACKAGE_QT5BASE_GIF=y
Missing: BR2_PACKAGE_QT5BASE_JPEG=y
Missing: BR2_PACKAGE_GLMARK2=y
Missing: BR2_PACKAGE_QT5CINEX=y
Missing: BR2_PACKAGE_QT5CINEX_HD=y
WARN: defconfig configs/freescale_imx28evk_defconfig can't be used:
Missing: BR2_TARGET_ROOTFS_EXT4=y
WARN: defconfig configs/imx23evk_defconfig can't be used:
Missing: BR2_TARGET_ROOTFS_EXT4=y
WARN: defconfig configs/imx6-sabresd_qt5_defconfig can't be used:
Missing: BR2_PACKAGE_QT5=y
Missing: BR2_PACKAGE_QT5BASE_LICENSE_APPROVED=y
Missing: BR2_PACKAGE_QT5BASE_OPENGL_LIB=y
Missing: BR2_PACKAGE_QT5BASE_LINUXFB=y
Missing: BR2_PACKAGE_QT5BASE_FONTCONFIG=y
Missing: BR2_PACKAGE_QT5BASE_GIF=y
Missing: BR2_PACKAGE_QT5BASE_JPEG=y
Missing: BR2_PACKAGE_QT5CINEX=y
Missing: BR2_PACKAGE_QT5CINEX_HD=y
WARN: defconfig configs/minnowboard_max-graphical_defconfig can't be used:
Missing: BR2_PACKAGE_GLMARK2=y
Missing: BR2_PACKAGE_MESA3D_DEMOS=y
WARN: defconfig configs/nanopi_r1_defconfig can't be used:
Missing: BR2_TARGET_UBOOT_BOARD_DEFCONFIG="nanopi_r1"
WARN: defconfig configs/olimex_a20_olinuxino_lime2_defconfig can't be used:
Missing: BR2_PACKAGE_SUNXI_MALI_MAINLINE=y
Missing: BR2_PACKAGE_SUNXI_MALI_MAINLINE_DRIVER=y
WARN: defconfig configs/olimex_a20_olinuxino_lime_defconfig can't be used:
Missing: BR2_PACKAGE_SUNXI_MALI_MAINLINE=y
Missing: BR2_PACKAGE_SUNXI_MALI_MAINLINE_DRIVER=y
WARN: defconfig configs/olimex_imx233_olinuxino_defconfig can't be used:
Missing: BR2_TARGET_ROOTFS_EXT4=y
WARN: defconfig configs/qemu_ppc_virtex_ml507_defconfig can't be used:
Missing: BR2_SOFT_FLOAT=y
WARN: defconfig configs/qemu_riscv32_virt_defconfig can't be used:
Missing: BR2_TARGET_OPENSBI_USE_PLAT=y
WARN: defconfig configs/qemu_riscv64_virt_defconfig can't be used:
Missing: BR2_TARGET_OPENSBI_USE_PLAT=y
WARN: defconfig configs/raspberrypi3_qt5we_defconfig can't be used:
Missing: BR2_PACKAGE_QT5=y
Missing: BR2_PACKAGE_QT5BASE_EXAMPLES=y
Missing: BR2_PACKAGE_QT5BASE_GIF=y
Missing: BR2_PACKAGE_QT5BASE_JPEG=y
Missing: BR2_PACKAGE_QT5BASE_PNG=y
Missing: BR2_PACKAGE_QT5WEBENGINE=y
Missing: BR2_PACKAGE_QT5WEBENGINE_PROPRIETARY_CODECS=y
> + return True
> +
> +
> +if __name__ == "__main__":
> + main()
> --
> 2.25.3
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-04-20 13:06 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-04-20 9:17 [Buildroot] [PATCH 1/2] support/scripts: add check-dotconfig.py Romain Naour
2020-04-20 9:17 ` [Buildroot] [PATCH 2/2] gitlab-ci: check generated config files Romain Naour
2020-04-20 11:52 ` [Buildroot] [PATCH 1/2] support/scripts: add check-dotconfig.py Thomas Petazzoni
2020-04-20 12:19 ` Romain Naour
2020-04-20 12:38 ` Thomas Petazzoni
2020-04-20 13:06 ` Yann E. MORIN
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox