From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 45A09C433F5 for ; Thu, 26 May 2022 16:46:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ZqCxFz3zTL/O4V1+g1x/LIo0Hl/JXrTc2qtQwH6nAR4=; b=UL2HabEGZuP1Zc QyCo01lUr0mIE9H5LCD6KbFFCA0LCwdjyJEoQScJpwRxXNDsHitaQJdeE5VLfQ6a/3dBsV9kxIXjy H6XAdXvnrDhdFQ34ToBb2FcCWAusTDtB2ovL6qTY37YeCx2ogG6lCbKyeMkRkE1vOY4FySr2+SF+N o/yfsJkIgOmAXxfiixsqH896l30Kf6QCZg043kPbCwgJpKEdpvuA2CQwg+9l+QnEedckbP1jHPPt/ ZhAVijfYnTMJkGJ/tfmWF9xO1dcfVQINzdVBjKVy0YD2rjNLP5wJlXU/JEDpLLb+JLhCc6/cr+z0r jHwODIHY0DpV4qje/Ktg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nuGbu-00FR0j-4f; Thu, 26 May 2022 16:45:18 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nuGbp-00FQzu-Tg for linux-arm-kernel@lists.infradead.org; Thu, 26 May 2022 16:45:16 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 56ED0150C; Thu, 26 May 2022 09:45:09 -0700 (PDT) Received: from monolith.localdoman (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 85F563F70D; Thu, 26 May 2022 09:45:07 -0700 (PDT) Date: Thu, 26 May 2022 17:45:18 +0100 From: Alexandru Elisei To: Andre Przywara Cc: will@kernel.org, julien.thierry.kdev@gmail.com, maz@kernel.org, suzuki.poulose@arm.com, james.morse@arm.com, vladimir.murzin@arm.com, anup@brainfault.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu Subject: Re: [PATCH kvmtool 0/2] Add libfdt library Message-ID: References: <20220526153058.123290-1-alexandru.elisei@arm.com> <20220526165907.0e34c075@donnerap.cambridge.arm.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20220526165907.0e34c075@donnerap.cambridge.arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220526_094514_100689_348CF518 X-CRM114-Status: GOOD ( 51.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi, On Thu, May 26, 2022 at 04:59:07PM +0100, Andre Przywara wrote: > On Thu, 26 May 2022 16:30:56 +0100 > Alexandru Elisei wrote: > > Hi Alex, > > > I found it very difficult to cross-build kvmtool for all arches because for > > some architectures I couldn't get libfdt to install: either dtc wasn't > > compiling (x86 machine, Arch Linux, missing yaml.h include when building > > for arm and powerpc), or libfdt was not being found by Makefile after > > installing it (x86 machine, Debian 11, building for powerpc). It was > > possible to compile only libfdt and teach Makefile where to look for it, > > which is what I ended up doing, but in my opinion, that's not an ideal user > > experience. > > Meh. Nobody said that userland cross compilation would be a walk in the > park, and this approach (cross-compiling and putting in SYSROOT) is the > canonical approach. When using a multiarch capable distro, this is easy > enough. Except when it doesn't work. On debian 11 I was not able to make it work for powerpc by following the instructions, while I was able to make it work for aarch64 following the same instructions. That's the kind of thing that I would like to avoid. > But I see that this is some major pain point for an otherwise rather "low > maintenance" tool, and libfdt is designed to be embedded, so I guess this > is fine, with the caveat mentioned below. > > > This is my attempt at making compilation easier by pulling in the libfdt > > library and having kvmtool compile it when an architecture requires it. All > > the user needs to do now is to find a suitable cross-compiler toolchain > > with glibc included [1]*. > > > > When making lkvm-static, the static version of libfdt is built (libfdt.a); > > for all other make targets the shared library is built. > > So does that mean that the .so gets shipped with kvmtool? And it should be Err.. I screwed up badly here, I just checked with ldd and lkvm was still using the system proved libfdt.so, that's why it was still working. Uninstalling the libfdt package broke lkvm. > installed as part of "make install"? I think that might bring up more > questions and problems, so do we want to leave the user some choice? > - The user brings their own libfdt. This should be the case for any clean > userland cross compilation, where you install target libraries in SYSROOT, > just for compilation, and have the .so file also on the actual target > system. This would also be best for native compilation. Current situation. > - The user decides to use the built-in libfdt, in which case that gets > always statically linked into the lkvm binary (which might still be > dynamically linked against libc), to avoid any conflict between the just > built version (.so), and the installed library on the target system. > > Some projects choose this approach, where it gets decided at build time > which version to use ("system library" vs. built-in version). This is > either done automatically, because a system library is not found, or using > some user provided flag for configure/make. > > If that sounds too involved, we could just always link in our libfdt copy > statically. And later revisit this when people complain. I would prefer this approach, as it keeps things the Makefile simple and it makes the build reproducible. > Can you check by how much an otherwise dynamically linked kvmtool binary > grows when we link against libfdt.a? I wrote a patch to always link against libfdt.a, here are the results of running make on a rockpro64 (I renamed the executables to tell which is which). Built from current master: $ /usr/bin/ls -l lkvm-libfdt-dynamically-linked -rwxr-xr-x 1 alex users 2127112 May 26 16:31 lkvm-libfdt-dynamically-linked $ ldd lkvm-libfdt-dynamically-linked linux-vdso.so.1 (0x0000ffffb52ae000) libz.so.1 => /usr/lib/libz.so.1 (0x0000ffffb5140000) libaio.so.1 => /usr/lib/libaio.so.1 (0x0000ffffb5120000) libfdt.so.1 => /usr/lib/libfdt.so.1 (0x0000ffffb5100000) libc.so.6 => /usr/lib/libc.so.6 (0x0000ffffb4f50000) /lib/ld-linux-aarch64.so.1 => /usr/lib/ld-linux-aarch64.so.1 (0x0000ffffb5275000) Built with Makefile modified to link against the static version: $ /usr/bin/ls -l lkvm-libfdt-statically-linked -rwxr-xr-x 1 alex users 2238920 May 26 16:32 lkvm-libfdt-statically-linked $ ldd lkvm-libfdt-statically-linked linux-vdso.so.1 (0x0000ffff99a6d000) libz.so.1 => /usr/lib/libz.so.1 (0x0000ffff99900000) libaio.so.1 => /usr/lib/libaio.so.1 (0x0000ffff998e0000) libc.so.6 => /usr/lib/libc.so.6 (0x0000ffff99730000) /lib/ld-linux-aarch64.so.1 => /usr/lib/ld-linux-aarch64.so.1 (0x0000ffff99a34000) A difference of 109KB (111808 bytes to be more exact). I'll send the changes to the Makefile in case people want to play with it. Thanks, Alex > > Cheers, > Andre > > > This only happens > > for those arches that require libfdt (arm, arm64, riscv, powerpc); libfdt > > is not built for MIPS and x86. > > > > I am pretty happy with how libfdt has been integrated with the kvmtool > > build system. The only niggle is that the libfdt_clean target echoes the > > commands to remove the build artifacts: > > > > $ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- clean > > Makefile:378: Skipping optional libraries: bfd zlib aio > > CLEAN (libfdt) > > rm -f libfdt/*.o libfdt/libfdt.a libfdt/libfdt-1.6.1.so > > rm -f libfdt/libfdt.so.1 > > CLEAN > > > > but I believe that's impossible to change without modifying the libfdt > > Makefile, which I don't want to do to keep it as easy as possible to update > > libfdt. > > > > Made some quick measurement about how libfdt affects the build time when > > building from a clean directory (incremental builds do not rebuild libfdt): > > > > - On a rockpro64, on the little core (Cortex-A53), running > > > > $ time taskset -c 0 make > > > > with libfdt takes an average of 90.721s (averaged over 3 runs, standard > > deviation 0.040s); without libfdt it takes 84.003s (averaged over 3 runs, > > stddev of 0.105s). That's a 6.718s absolute increase, or a 7.997% relative > > increase. > > > > - On the same rockpro64, on the big core (Cortex-A72), running > > > > $ time taskset -c 5 make > > > > with libfdt takes an average of 45.078s (averaged over 3 runs, stddev of > > 0.060s); without libfdt it takes 41.836s (averaged over 3 runs, stddev of > > 0.065). That's a 3.242s absolute increase, or a 7.749% relative increase. > > > > I believe that the ease of use more than justifies the increased > > compilation time. > > > > Testing > > ======= > > > > On x86 host, compiled for x86, MIPS, arm, arm64, powerpc, riscv. Booted an > > x86 VM. > > > > On arm64 host, compiled for arm64 and arm. I tried to cross-compile for > > other targets, but I couldn't find a cross-compiler toolchain for arm64 > > hosts that included a libc implementation. The arm cross-compiler was > > downloaded from the developer.arm.com [2]. Ran kvm-unit-tests for arm64 and > > arm (on the arm64 version of kvmtool), and also booted a kernel. > > > > *only for x86 hosts. > > > > [1] https://toolchains.bootlin.com/ > > [2] https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/downloads > > > > Alexandru Elisei (2): > > Add libfdt library > > INSTALL: Remove all mentions of libfdt > > > > INSTALL | 27 +- > > Makefile | 53 +- > > libfdt/.gitignore | 1 + > > libfdt/Makefile.libfdt | 18 + > > libfdt/TODO | 3 + > > libfdt/fdt.c | 335 ++++++ > > libfdt/fdt.h | 66 ++ > > libfdt/fdt_addresses.c | 101 ++ > > libfdt/fdt_check.c | 93 ++ > > libfdt/fdt_empty_tree.c | 38 + > > libfdt/fdt_overlay.c | 867 +++++++++++++++ > > libfdt/fdt_ro.c | 859 +++++++++++++++ > > libfdt/fdt_rw.c | 500 +++++++++ > > libfdt/fdt_strerror.c | 60 ++ > > libfdt/fdt_sw.c | 384 +++++++ > > libfdt/fdt_wip.c | 94 ++ > > libfdt/libfdt.h | 2147 ++++++++++++++++++++++++++++++++++++++ > > libfdt/libfdt_env.h | 96 ++ > > libfdt/libfdt_internal.h | 192 ++++ > > libfdt/meson.build | 55 + > > libfdt/version.lds | 83 ++ > > 21 files changed, 6033 insertions(+), 39 deletions(-) > > create mode 100644 libfdt/.gitignore > > create mode 100644 libfdt/Makefile.libfdt > > create mode 100644 libfdt/TODO > > create mode 100644 libfdt/fdt.c > > create mode 100644 libfdt/fdt.h > > create mode 100644 libfdt/fdt_addresses.c > > create mode 100644 libfdt/fdt_check.c > > create mode 100644 libfdt/fdt_empty_tree.c > > create mode 100644 libfdt/fdt_overlay.c > > create mode 100644 libfdt/fdt_ro.c > > create mode 100644 libfdt/fdt_rw.c > > create mode 100644 libfdt/fdt_strerror.c > > create mode 100644 libfdt/fdt_sw.c > > create mode 100644 libfdt/fdt_wip.c > > create mode 100644 libfdt/libfdt.h > > create mode 100644 libfdt/libfdt_env.h > > create mode 100644 libfdt/libfdt_internal.h > > create mode 100644 libfdt/meson.build > > create mode 100644 libfdt/version.lds > > > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel