Re: [meta-virtualization][PATCH v2] vhost-device: add recipes for vhost-device daemons

["Erik Schilling" <erik.schilling@linaro.org>]

On Wed Aug 16, 2023 at 4:19 PM CEST, Bruce Ashfield wrote:
> On Wed, Aug 16, 2023 at 10:14 AM Erik Schilling
> <erik.schilling@linaro.org> wrote:
> >
> > On Wed Aug 16, 2023 at 3:34 PM CEST, Bruce Ashfield wrote:
> > > On Wed, Aug 16, 2023 at 4:32 AM Erik Schilling
> > > <erik.schilling@linaro.org> wrote:
> > > >
> > > > On Wed Aug 16, 2023 at 10:11 AM CEST, Mikko Rapeli wrote:
> > > > > Hi,
> > > > >
> > > > > On Mon, Aug 14, 2023 at 10:04:06AM +0200, Erik Schilling wrote:
> > > > > > This adds recipes for the first tagged release of the vhost-device
> > > > > > daemons of the rust-vmm project.
> > > > > >
> > > > > > While the initial release was done for all daemons at the same time,
> > > > > > the daemons all have indepentend version numbers and will be released
> > > > > > on their own schedules in the future. Therefore, I splitted them into
> > > > > > independent recipes.
> > > > > >
> > > > > > Signed-off-by: Erik Schilling <erik.schilling@linaro.org>
> > > > > > ---
> > > > > > These are a bunch of daemons that implement various vhost-user virtio
> > > > > > devices. Currently, they are mostly tested with QEMU, but we are also
> > > > > > working on Xen support (most bits are upstream already).
> > > > > > ---
> > > > > > Changes in v2:
> > > > > > - Added README.md to explain generation of dependency .inc files
> > > > > > - Link to v1: https://lore.kernel.org/r/20230728-vhost-device-v1-1-d9f81124b66a@linaro.org
> > > > > > ---
> > > > > >  README.md                                          |  14 ++
> > > > > >  .../vhost-device/vhost-device-gpio-crates.inc      | 184 +++++++++++++++
> > > > > >  .../vhost-device/vhost-device-gpio_0.1.0.bb        |  20 ++
> > > > > >  .../vhost-device/vhost-device-i2c-crates.inc       | 126 ++++++++++
> > > > > >  .../vhost-device/vhost-device-i2c_0.1.0.bb         |  16 ++
> > > > > >  .../vhost-device/vhost-device-rng-crates.inc       | 158 +++++++++++++
> > > > > >  .../vhost-device/vhost-device-rng_0.1.0.bb         |  17 ++
> > > > > >  .../vhost-device/vhost-device-scsi-crates.inc      | 166 +++++++++++++
> > > > > >  .../vhost-device/vhost-device-scsi_0.1.0.bb        |  16 ++
> > > > > >  .../vhost-device/vhost-device-vsock-crates.inc     | 258 +++++++++++++++++++++
> > > > > >  .../vhost-device/vhost-device-vsock_0.1.0.bb       |  16 ++
> > > > > >  11 files changed, 991 insertions(+)
> > > > > <snip>
> > > > > > +++ b/recipes-extended/vhost-device/vhost-device-gpio_0.1.0.bb
> > > > > > @@ -0,0 +1,20 @@
> > > > > > +SUMMARY = "vhost gpio backend device"
> > > > > > +DESCRIPTION = "A vhost-user backend that emulates a VirtIO GPIO device"
> > > > > > +HOMEPAGE = "https://github.com/rust-vmm/vhost-device"
> > > > > > +LICENSE = "Apache-2.0 | BSD-3-Clause"
> > > > > > +LIC_FILES_CHKSUM = "\
> > > > > > +    file://LICENSE-APACHE;md5=3b83ef96387f14655fc854ddc3c6bd57 \
> > > > > > +    file://LICENSE-BSD-3-Clause;md5=2489db1359f496fff34bd393df63947e \
> > > > > > +"
> > > > > > +DEPENDS += "libgpiod"
> > > > > > +# libgpiod-sys generates bindings using bindgen, which depends on clang
> > > > > > +DEPENDS += "clang-native"
> > > > >
> > > > > As Richard mentioned on #yocto, this recipe adds a new layer dependency from
> > > > > meta-virtualization to meta-clang. This could be documented and test build jobs
> > > > > changed to include meta-clang, or these recipes could be moved to build conditionally
> > > > > only when meta-clang is available by moving the recipes to directory
> > > > > dynamic-layers/meta-clang/recipes-extended
> > > > >
> > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/121/builds/1528/steps/23/logs/stdio
> > > >
> > > > Hm... I guess moving vhost-device-gpio to dynamic-layers/ is the least
> > > > invasive option?
> > > >
> > > > @Bruce: Which option do you prefer? I can draft a patch.
> > >
> > > Definitely not a dynamic layer :)
> > >
> > > Can you expand on the dependency a bit more for me ? I find it odd that libgpiod
> > > is in meta-oe, but there's no mention (that I can see) about any clang
> > > dependencies.
> > >
> > > I know very little about the way rust is stitching the bits together.
> > >
> > > Should libgpod really be in the DEPENDS ? Is libgpiod-sys a clang thing ? And
> > > libgpiod is somehow conditionally using it ?
> >
> > Yeah, this is a bit confusing...
> >
> > Here is a (hopefully) better attempt at explaining what is going on:
> >
> > vhost-device-gpio depends on libgpiod (the Rust crate, not the C lib).
> > which in its turn depends on libgpiod-sys - a helper crate that provides
> > binding to the libgpiod (C) lib.
> >
> > Now, libgpiod-sys does not really consist of any _real_ code. Instead,
> > it is a recipe to generate the bindings at build time. This happens
> > by reading the pkg-config info from libgpiod (the C lib), parsing the
> > headers and linking to it.
> >
> > So: libgpiod (the C lib) - as it is in meta-oe - does not depend on
> > clang-native, only libgpiod-sys (the Rust lib) does.
> >
> > The final result is two dependencies being required for building
> > libgpiod-sys: libclang (for bindgen) and libgpiod (the C lib) (for
> > pkg-config, the headers and linking).
> >
> > In Rust - in contrast to how C libs are typically packaged - libgpiod
> > (the Rust lib) and libgpiod-sys (Rust lib again) are built and linked
> > into vhost-device-gpio all during the build of this recipe. So the
> > dependencies are required here.
> >
> > The final result is:
> >
> > ┌──────────────────────────────────────┐
> > │          vhost-device-gpio           │
> > │                                      │
> > │ libgpiod (Rust), libgpiod-sys (Rust) │ ─────────────►  libgpiod (C)
> > │           (static linking)           │ (dynamic link)      ▲
> > └──────────────────────────────────────┘                     │
> >                    ▲                           the usual libgpiod recipe
> >                    │
> >    this is the build result of this recipe
> >
>
> Aha. Yes, I see why my dependency investigation got tangled up
> in the "normal" libgpiod in meta-openembedded.
>
> >
> > Overall, the generation of bindings at build time is fairly common in
> > the Rust ecosystem. parsec in meta-security, for example, also does the
> > same thing. There the situation seems to be solved by putting parsec
> > into it's own layer within the meta-security repo.
>
> Layers with a single or few recipes are a pet peeve of mine :)
>
> >
> > > Either way, we used to do this same dance with meta-security, and we did that
> > > via SKIP_RECIPE. So that's how I'd solve this. But obviously none of the core
> > > packagesgroups, etc,  in meta-virt can have the vhost* packages if that is the
> > > case. Barring that, we add the dependency, if it becomes a widespread use case,
> > > it'll end up in core anyway.
> >
> > This would require users of the layer to undo the SKIP_RECIPE? Or am I
> > misunderstanding what you mean? Just trying to understand the proposal,
> > I got no opinions on the best solution to resolve this. I am happy with
> > what seems best for the greater world.
>
> It is similar to a layer, but it keeps everything centralized. It'll
> automatically
> be undone if meta-clang is available (with the appropriate changes of
> course).
>
> We used to have this for selinux and security:
>
> SKIP_RECIPE[cri-o] ?= "${@bb.utils.contains('BBFILE_COLLECTIONS',
> 'security', bb.utils.contains('BBFILE_COLLECTIONS', 'selinux', '',
> 'Depends on libselinux from meta-selinux which is not included', d),
> 'Depends on libseccomp from meta-security which is not included', d)}"
>
> SKIP_RECIPE[cri-o] ?= "${@bb.utils.contains('BBFILE_COLLECTIONS',
> 'selinux', '', 'Depends on libselinux from meta-selinux which is not
> included', d)}"

Ah. Did not know this works within a recipe! Patch posted:

https://lore.kernel.org/r/20230816-vhost-device-clang-fix-v1-1-bbbd47b13e5b@linaro.org

Thanks!

- Erik