From mboxrd@z Thu Jan 1 00:00:00 1970 From: Thomas Petazzoni Date: Tue, 3 Apr 2018 14:09:37 +0200 Subject: [Buildroot] Buildroot hackathon day 3 highlights Message-ID: <20180403140937.3625de90@windsurf> List-Id: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: buildroot@busybox.net Hello, The Buildroot hackathon continued and finished on its third day, which was shorter than the others as most of the participants headed home towards the end of the afternoon/beginning of the evening. Nonetheless, some useful progress has been made: - By far and large, the most significant achievement of the day, and perhaps of the entire hackathon, is the merge of the patch series switching to per-package folders for the download directory, and the introduction of caching for Git downloads. This is the result of work done by Maxime Hadjinlian and Yann E. Morin, started during the Buildroot Summer Camp 2017, and which had seen several iterations since then, including one from Peter Seiderer. Yann and Maxime worked hard on this last day of the hackathon to iron out the remaining issues, sending several iterations. I'll try to send a separate e-mail to describe in more details what it is and what is the visible impact for the user. - The iqvlinux bump was sitting in patchwork for a while, due to the fact that upstream names the tarball iqvlinux.tar.gz (with no version included), which doesn't work well with our download/hash infrastructure. Carlos proposal was to make a direct call to the download wrapper in a pre-download hook to be able to rename the file during the download, but the fact that it was using internals of the download infrastructure was not very nice, and indeed such internals were being changed at the same time by Yann and Maxime for the Git caching. Thefore, after discussion with Peter, we opted for hosting an iqvlinux-.tar.gz tarball on sources.buildroot.net, and let the package use that. - Arnout continued his work on the Qt5 patches, and committed the bump to Qt 5.10, together with some preparation patches: qt5script: correct license information package: add libnss host package package: add libnpsr host package qt5: bump latest version to 5.10.1 - Arnout and Thomas had a look at the patch series from Carlos Santos adding support for nmap in ncat, netcf and libvirt. As a result of this, Thomas reworked the nmap patch up to the point where it could be comitted, and sent a number of review comments for the netcf and libvirt patches. - Peter investigated two separate hash mismatch issues, which we discussed quite at length. The first one happened on tarballs that we generate after doing a Git clone, and was exhibited by the recently merged gst1-interpipe package. This package has long file paths, which are encoded slightly differently in tar versions < 1.27 and tar >= 1.27. To fix this, we raised the minimum required tar version to 1.27, which means that machines having a tar older than 1.27 will now build their own host-tar. Ideally, this should only be done when Buildroot does git/bzr/hg/cvs/svn downloads, but that could be optimized later. The second one happened with Github-generated tarballs, which have changed subtly over time (it was noticed on the "runc" package). Unfortunately, since the tarball is generated by Github, there is not much we can do. sources.buildroot.net has a backup of the original tarball with the correct hash, so Buildroot properly falls back on sources.buildroot.net and the build succeeds, but it isn't really nice. One solution that was proposed by Thomas is that for Github downloads we extract the tarball, re-create it ourself, and store the hash of this new tarball. This way, we control how the tarball is created. - In a related discussion, we looked at https://bugs.busybox.net/show_bug.cgi?id=10511, which is a problem with wget 1.19.2 automatically uncompressing tarballs served by some HTTP servers. Since the problem has been fixed in wget 1.19.3, and therefore only affects 1.19.2 precisely, and that anyway the build succeeds thanks to the fallback to sources.buildroot.net, we decided to not change anything in Buildroot to cover what really is a wget bug. - Romain worked on the LLVM patch series, taking into account (some of) the comments made by Thomas the day before. Thomas and Romain then did a live review of the new version, and things are going in the right direction. - We also had some discussion about the fact that Linux 4.16 now requires bison and flex on the host. Bernd initially proposed new options "Needs bison" and "Needs flex", very much like we already have the "Needs OpenSSL" and "Needs libelf" options in the Linux kernel menu. However, while OpenSSL and libelf on the host are only needed for very specific kernel configurations, host-bison and host-flex will now be needed to build any kernel >= 4.16, so we suggested that those dependencies should be made mandatory. However, building host-flex and host-bison takes time, and since they are needed for the kernel kconfig code, building those dependencies will be needed even to run "make linux-menuconfig". This is made even worse with the fact that host-flex needs to be autoreconf'ed and gettextize'd, which requires building a whole bunch of dependencies, including the infamous host-gettext. Arnout suggested that perhaps we should handle host-bison and host-flex like host-cmake: only build them if the host doesn't provide them. This is obviously an interesting idea, but it reduces the "reproducibility" in that users would now be using different versions of flex and bison. - Arnout and Thomas also had a lengthy discussion about http://patchwork.ozlabs.org/patch/863021/, an apparently simple patch, but that is papering over a more serious problem: our handling of compiler flags is messy. Some of the flags are encoded in the wrapper, some are passed in the CFLAGS environment variable, and there is real consistency. Arnout and Thomas went through a detailed review of what we are doing today, and sketched out a plan to clean this up. Thomas has some (relatively messy) notes about this, that he'll try to clean up and post to the mailing list. - Thomas continued to go through the patchwork backlog, applying a number of patches, old and new, on various random topics. In total, we applied 222 patches during this 3-day hackathon, and we reduced the patchwork backlog from ~350 patches to ~175, roughly cutting it in half. We merged some significant patch series: rework of the filesystem image generation, support for Git caching, Golang package infrastructure, Qt5 bump to 5.10. We would like to thank once again Online and Scaleway for hosting our event at La Maison, in Paris, where we had really excellent hosting conditions. We look forward to organizing a similar hackathon in the future. Thomas -- Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons) Embedded Linux and Kernel engineering https://bootlin.com