Central repo for VirtIO conformance tests?

Central repo for VirtIO conformance tests?

[Alex Bennée]

Hi,

We've been working with Panasonic to expand the testing of VirtIO across
a range of hypervisors and VMMs. We've tackled this with two approaches:

  - simple unikernel to verify features and basic functions common
  - rootfs images to exercise the whole device

The unikernel utilizes rcore-os's no_std VirtIO drivers to discover and
initialize a range of VirtIO devices. The tests mostly focus on checking
no unknown feature bits are advertised as well as ensuring features
dependencies are properly realizable. This is useful for those
interested in moving workloads between hypervisors to ensure you don't
end up relying on a proprietary feature that aren't available elsewhere.

The virgl test also does some very basic blob mapping to check the
underlying mechanics are working.

The unikernel outputs a TAP stream to make integrating into a test
harness easier.

You can find the current state here:

  https://git.codelinaro.org/manos.pitsidianakis/virtio-tests

While we only build an aarch64 unikernel the upstream drivers have
examples for riscv and x86_64 as well.

As more complex VirtIO devices like GPUs tend to have a significant
user-space component, we have also started building Linux rootfs images
to exercise those. The images themselves are built against a baseline
architecture so they can be used on as wide a range of hardware as
possible. They have been built with buildroot to make them lightweight
and as close to the upstream projects as possible without relying on
particular distro support. You may have seen the recent aarch64 GPU
image being added to QEMU's functional tests recently:

  https://gitlab.com/qemu-project/qemu/-/blob/master/tests/functional/test_aarch64_virt_gpu.py

I'm currently working on a similar image utilizing a subset of the
blktests project to exercise the VirtIO block devices. The various
recipes can be found here:

  https://gitlab.com/stsquad/buildroot/-/tree/adding-blktests?ref_type=heads

although the intention is for all the recipes and basic QEMU based tests
to be up-streamed into buildroot in due course. While I will no doubt
expand the functional tests in QEMU over time to utilize these images,
there is a wider question of where would be a good place to host a more
comprehensive VirtIO test suite? While useful for validating proprietary
hypervisors there are also a bunch of other VMMs and VirtIO backends
other than QEMU:

  - rust-vmm's vhost-device collection of vhost-user backends
  - CrosVM
  - Cloud Hypervisor
  - libkrun

As well as using VirtIO backends with other hypervisors such as Xen,
Gunyah and WHPX.

So this brings me to the question posed in the subject. Where would a
good place be to host these conformance tests?

My initial thought was to see if this is something OASIS could host as
part of the specification however I'm not sure OASIS is set up for such a
thing.

We could host it as part of the QEMU project as a service to the wider
community. While it should be pretty easy to expand QEMU's own tests to
work with multiple hypervisors, its test machinery isn't really setup
for non-QEMU VMMs. Ideally we would want the core repository to be able
to run on multiple hypervisors and use different VMMs and backends
depending on where they are being run.

The other option I considered was hosting with the rust-vmm project -
although maybe that just makes sense for the unikernel tests as they are
rust based. We certainly need more automated testing of the vhost-device
repository which can serve as a backend to multiple VMMs and
hypervisors.

So what do people think? Where would be a good place for common test
repository to live?

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: Central repo for VirtIO conformance tests?

[Stefan Hajnoczi]

On Mon, Mar 31, 2025 at 6:39 AM Alex Bennée <alex.bennee@linaro.org> wrote:
> So what do people think? Where would be a good place for common test
> repository to live?

Maintaining the tests alongside the VIRTIO spec seems like a good fit
to me. Here is information on how to create a GitHub repo under the
OASIS VIRTIO Technical Committee:
https://www.oasis-open.org/open-repositories/

I don't see a reason to maintain the tests under QEMU or rust-vmm
unless they provide some unique project infrastructure that an
independent project would lack or need to reinvent.

If it is not possible to host the tests under OASIS then they could be
hosted independently (as a separate organization on Codeberg, GitLab,
GitHub, etc) and not on any contributor company's infrastructure to
make them more sustainable and neutral for the long term.

Stefan

Re: Central repo for VirtIO conformance tests?

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1371 bytes --]

On Mon, Mar 31, 2025 at 01:14:33PM +0100, Daniel P. Berrangé wrote:
> On Mon, Mar 31, 2025 at 07:52:33AM -0400, Stefan Hajnoczi wrote:
> > On Mon, Mar 31, 2025 at 6:39 AM Alex Bennée <alex.bennee@linaro.org> wrote:
> > > So what do people think? Where would be a good place for common test
> > > repository to live?
> > 
> > Maintaining the tests alongside the VIRTIO spec seems like a good fit
> > to me. Here is information on how to create a GitHub repo under the
> > OASIS VIRTIO Technical Committee:
> > https://www.oasis-open.org/open-repositories/
> > 
> > I don't see a reason to maintain the tests under QEMU or rust-vmm
> > unless they provide some unique project infrastructure that an
> > independent project would lack or need to reinvent.
> 
> IMHO opening that very link above illustrates the reason why *NOT* to
> host this under OASIS. Any potential contributor is faced with a 4 page
> long blurb of all sorts of rules & legal requirements, including
> 
>   "Each TC Open Repository shall be subject to a Contributor
>    License Agreement (“CLA”) by which all persons making repo
>    contributions into it are bound."
> 
> Hosting anywhere else would be better from the POV of removing barriers
> to potential contribution.

I didn't notice the CLA requirement. I agree, it's better to host it
elsewhere.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Central repo for VirtIO conformance tests?

[Alex Bennée]

Stefan Hajnoczi <stefanha@gmail.com> writes:

> On Mon, Mar 31, 2025 at 6:39 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>> The unikernel utilizes rcore-os's no_std VirtIO drivers to discover and
>> initialize a range of VirtIO devices.
>
> https://github.com/rcore-os/virtio-drivers
>
> I noticed that VIRTIO_F_VERSION_1, VIRTIO_F_RING_PACKED, and hardware
> device support are missing according to the readme. Those are
> important features for real-world test coverage.

At the moment most of the testing is around feature negotiation so we
don't get to the point that we are need packed virtqs. However it would
be useful to upstream such support for their use generally - I know
there are some downstream projects that have stuff to upstream in due
course.

The original idea for the feature bit tests was to utilise the kernels
kunit framework. However that was outside the scope of what kunit which
is really focused on testing kernel internals rather than the HW
interface.

We did consider expanding the kvm-unit-test uni-kernels but that would
either involve importing kernel bits or a fairly big C re-implementation
of it. At least by expanding the rcore-os drivers we will have two
different driver stacks in two different languages.

> Or is the idea to use the unikernel for low-level device
> initialization and the Linux image for tests that do I/O?

So far the rootfs images get by far the largest coverage because they
can more easily trigger a whole range of conditions. While VirtIO is
intended to be OS and hypervisor agnostic Linux remains the system with
the widest and most functional coverage of the interfaces.

>
> Stefan

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: Central repo for VirtIO conformance tests?

[Manos Pitsidianakis]

On Mon, Mar 31, 2025 at 1:38 PM Alex Bennée <alex.bennee@linaro.org> wrote:
>
>
> Hi,
>
> We've been working with Panasonic to expand the testing of VirtIO across
> a range of hypervisors and VMMs. We've tackled this with two approaches:
>
>   - simple unikernel to verify features and basic functions common
>   - rootfs images to exercise the whole device
>
> The unikernel utilizes rcore-os's no_std VirtIO drivers to discover and
> initialize a range of VirtIO devices. The tests mostly focus on checking
> no unknown feature bits are advertised as well as ensuring features
> dependencies are properly realizable. This is useful for those
> interested in moving workloads between hypervisors to ensure you don't
> end up relying on a proprietary feature that aren't available elsewhere.
>
> The virgl test also does some very basic blob mapping to check the
> underlying mechanics are working.
>
> The unikernel outputs a TAP stream to make integrating into a test
> harness easier.
>
> You can find the current state here:
>
>   https://git.codelinaro.org/manos.pitsidianakis/virtio-tests
>
> While we only build an aarch64 unikernel the upstream drivers have
> examples for riscv and x86_64 as well.
>
> As more complex VirtIO devices like GPUs tend to have a significant
> user-space component, we have also started building Linux rootfs images
> to exercise those. The images themselves are built against a baseline
> architecture so they can be used on as wide a range of hardware as
> possible. They have been built with buildroot to make them lightweight
> and as close to the upstream projects as possible without relying on
> particular distro support. You may have seen the recent aarch64 GPU
> image being added to QEMU's functional tests recently:
>
>   https://gitlab.com/qemu-project/qemu/-/blob/master/tests/functional/test_aarch64_virt_gpu.py
>
> I'm currently working on a similar image utilizing a subset of the
> blktests project to exercise the VirtIO block devices. The various
> recipes can be found here:
>
>   https://gitlab.com/stsquad/buildroot/-/tree/adding-blktests?ref_type=heads
>
> although the intention is for all the recipes and basic QEMU based tests
> to be up-streamed into buildroot in due course. While I will no doubt
> expand the functional tests in QEMU over time to utilize these images,
> there is a wider question of where would be a good place to host a more
> comprehensive VirtIO test suite? While useful for validating proprietary
> hypervisors there are also a bunch of other VMMs and VirtIO backends
> other than QEMU:
>
>   - rust-vmm's vhost-device collection of vhost-user backends
>   - CrosVM
>   - Cloud Hypervisor
>   - libkrun
>
> As well as using VirtIO backends with other hypervisors such as Xen,
> Gunyah and WHPX.
>
> So this brings me to the question posed in the subject. Where would a
> good place be to host these conformance tests?
>
> My initial thought was to see if this is something OASIS could host as
> part of the specification however I'm not sure OASIS is set up for such a
> thing.
>
> We could host it as part of the QEMU project as a service to the wider
> community. While it should be pretty easy to expand QEMU's own tests to
> work with multiple hypervisors, its test machinery isn't really setup
> for non-QEMU VMMs. Ideally we would want the core repository to be able
> to run on multiple hypervisors and use different VMMs and backends
> depending on where they are being run.
>
> The other option I considered was hosting with the rust-vmm project -
> although maybe that just makes sense for the unikernel tests as they are
> rust based. We certainly need more automated testing of the vhost-device
> repository which can serve as a backend to multiple VMMs and
> hypervisors.
>
> So what do people think? Where would be a good place for common test
> repository to live?
>
> --
> Alex Bennée
> Virtualisation Tech Lead @ Linaro

Pinging this thread to ask the OASIS people: Would it be possible to
host a software project (this testsuite) under OASIS but with a
lighter contribution process? We expect everyone to be contributing to
them, from open source/proprietary hypervisor developers, to kernel/OS
developers, hobbyists and professionals alike. So I think a low
barrier of entry would be reasonable here.

Seeing as a test suite is essentially tied to the published VIRTIO
spec itself it also makes it a good fit semantically. Having a suite
with multiple VIRTIO implementations including slim images with Linux
would also give a wider frame of reference for downstream consumers of
the spec. Currently, the Linux kernel is the de facto (because it's
also open source) VIRTIO implementation on the frontend side and
oftentimes we end up looking at what Linux does to implement backends.

We'd like to potentially:

- Expand the array of rootfs images with more Linux userspace tests
for more devices
- Expand the array of rootfs images with other open source OSes as
they gain VIRTIO functionalities
- Implement missing VIRTIO spec features in our unikernel test and
also exercise realistic but basic VIRTIO command use scenarios for
more devices
- (This is on my personal wishlist) write a VIRTIO fuzzer framework,
similar in spirit to Google's syzkaller project (unsupervised
coverage-guided kernel fuzzer)
- As a stretch goal, provide a reference test runner framework to run
the images with QEMU and rust-vmm vhost-user backends for people to
adapt to their setups.

PS: The git repositories Alex linked are temporarily inaccessible due
to internal unrelated issues, and will be public again.

-- 
Manos Pitsidianakis
Emulation and Virtualization Engineer at Linaro Ltd

Re: Central repo for VirtIO conformance tests?

[Michael S. Tsirkin]

On Mon, May 19, 2025 at 10:43:33AM +0300, Manos Pitsidianakis wrote:
> Pinging this thread to ask the OASIS people: Would it be possible to
> host a software project (this testsuite) under OASIS but with a
> lighter contribution process? We expect everyone to be contributing to
> them, from open source/proprietary hypervisor developers, to kernel/OS
> developers, hobbyists and professionals alike. So I think a low
> barrier of entry would be reasonable here.

Take a look here pls:
https://www.oasis-open.org/open-repositories/#licensingRules



> Seeing as a test suite is essentially tied to the published VIRTIO
> spec itself it also makes it a good fit semantically. Having a suite
> with multiple VIRTIO implementations including slim images with Linux
> would also give a wider frame of reference for downstream consumers of
> the spec. Currently, the Linux kernel is the de facto (because it's
> also open source) VIRTIO implementation on the frontend side and
> oftentimes we end up looking at what Linux does to implement backends.
> 
> We'd like to potentially:
> 
> - Expand the array of rootfs images with more Linux userspace tests
> for more devices
> - Expand the array of rootfs images with other open source OSes as
> they gain VIRTIO functionalities
> - Implement missing VIRTIO spec features in our unikernel test and
> also exercise realistic but basic VIRTIO command use scenarios for
> more devices
> - (This is on my personal wishlist) write a VIRTIO fuzzer framework,
> similar in spirit to Google's syzkaller project (unsupervised
> coverage-guided kernel fuzzer)
> - As a stretch goal, provide a reference test runner framework to run
> the images with QEMU and rust-vmm vhost-user backends for people to
> adapt to their setups.
> 
> PS: The git repositories Alex linked are temporarily inaccessible due
> to internal unrelated issues, and will be public again.


At the moment, from the above link, and if you want it tied to OASIS
open repositories, it has to be one of:

BSD-3-Clause License (which shall apply if the TC makes no license selection in its approval action); Apache License v 2.0; CC-BY 2.0; CC-BY 4.0; Eclipse Public License v 1.0.


I am not a lawyer, but can full OS images we licensed under one of
these? If not, you are better off creating the repository itself outside
the confines of OASIS.

You can still use e.g. virtio-dev mailing list for communication.

> -- 
> Manos Pitsidianakis
> Emulation and Virtualization Engineer at Linaro Ltd

Re: Central repo for VirtIO conformance tests?

[Manos Pitsidianakis]

Hello Michael, thanks for the reply,

On Mon, May 19, 2025 at 10:54 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Mon, May 19, 2025 at 10:43:33AM +0300, Manos Pitsidianakis wrote:
> > Pinging this thread to ask the OASIS people: Would it be possible to
> > host a software project (this testsuite) under OASIS but with a
> > lighter contribution process? We expect everyone to be contributing to
> > them, from open source/proprietary hypervisor developers, to kernel/OS
> > developers, hobbyists and professionals alike. So I think a low
> > barrier of entry would be reasonable here.
>
> Take a look here pls:
> https://www.oasis-open.org/open-repositories/#licensingRules
>
>
>
> > Seeing as a test suite is essentially tied to the published VIRTIO
> > spec itself it also makes it a good fit semantically. Having a suite
> > with multiple VIRTIO implementations including slim images with Linux
> > would also give a wider frame of reference for downstream consumers of
> > the spec. Currently, the Linux kernel is the de facto (because it's
> > also open source) VIRTIO implementation on the frontend side and
> > oftentimes we end up looking at what Linux does to implement backends.
> >
> > We'd like to potentially:
> >
> > - Expand the array of rootfs images with more Linux userspace tests
> > for more devices
> > - Expand the array of rootfs images with other open source OSes as
> > they gain VIRTIO functionalities
> > - Implement missing VIRTIO spec features in our unikernel test and
> > also exercise realistic but basic VIRTIO command use scenarios for
> > more devices
> > - (This is on my personal wishlist) write a VIRTIO fuzzer framework,
> > similar in spirit to Google's syzkaller project (unsupervised
> > coverage-guided kernel fuzzer)
> > - As a stretch goal, provide a reference test runner framework to run
> > the images with QEMU and rust-vmm vhost-user backends for people to
> > adapt to their setups.
> >
> > PS: The git repositories Alex linked are temporarily inaccessible due
> > to internal unrelated issues, and will be public again.
>
>
> At the moment, from the above link, and if you want it tied to OASIS
> open repositories, it has to be one of:
>
> BSD-3-Clause License (which shall apply if the TC makes no license selection in its approval action); Apache License v 2.0; CC-BY 2.0; CC-BY 4.0; Eclipse Public License v 1.0.

OK that's doable and completely reasonable.

> I am not a lawyer, but can full OS images we licensed under one of
> these? If not, you are better off creating the repository itself outside
> the confines of OASIS.

We'd be hosting essentially build recipes for the images, for tests
where we use other people's code. The build recipes would have the
same license.

If I understand the CLA requirement correctly, even if the
contribution is merged by a maintainer who has signed the CLA, the
original contributor who has no write access to the repository, has to
sign the CLA as well?

> You can still use e.g. virtio-dev mailing list for communication.
>
> > --
> > Manos Pitsidianakis
> > Emulation and Virtualization Engineer at Linaro Ltd
>

Re: Central repo for VirtIO conformance tests?

[Michael S. Tsirkin]

On Mon, May 19, 2025 at 11:03:43AM +0300, Manos Pitsidianakis wrote:
> Hello Michael, thanks for the reply,
> 
> On Mon, May 19, 2025 at 10:54 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Mon, May 19, 2025 at 10:43:33AM +0300, Manos Pitsidianakis wrote:
> > > Pinging this thread to ask the OASIS people: Would it be possible to
> > > host a software project (this testsuite) under OASIS but with a
> > > lighter contribution process? We expect everyone to be contributing to
> > > them, from open source/proprietary hypervisor developers, to kernel/OS
> > > developers, hobbyists and professionals alike. So I think a low
> > > barrier of entry would be reasonable here.
> >
> > Take a look here pls:
> > https://www.oasis-open.org/open-repositories/#licensingRules
> >
> >
> >
> > > Seeing as a test suite is essentially tied to the published VIRTIO
> > > spec itself it also makes it a good fit semantically. Having a suite
> > > with multiple VIRTIO implementations including slim images with Linux
> > > would also give a wider frame of reference for downstream consumers of
> > > the spec. Currently, the Linux kernel is the de facto (because it's
> > > also open source) VIRTIO implementation on the frontend side and
> > > oftentimes we end up looking at what Linux does to implement backends.
> > >
> > > We'd like to potentially:
> > >
> > > - Expand the array of rootfs images with more Linux userspace tests
> > > for more devices
> > > - Expand the array of rootfs images with other open source OSes as
> > > they gain VIRTIO functionalities
> > > - Implement missing VIRTIO spec features in our unikernel test and
> > > also exercise realistic but basic VIRTIO command use scenarios for
> > > more devices
> > > - (This is on my personal wishlist) write a VIRTIO fuzzer framework,
> > > similar in spirit to Google's syzkaller project (unsupervised
> > > coverage-guided kernel fuzzer)
> > > - As a stretch goal, provide a reference test runner framework to run
> > > the images with QEMU and rust-vmm vhost-user backends for people to
> > > adapt to their setups.
> > >
> > > PS: The git repositories Alex linked are temporarily inaccessible due
> > > to internal unrelated issues, and will be public again.
> >
> >
> > At the moment, from the above link, and if you want it tied to OASIS
> > open repositories, it has to be one of:
> >
> > BSD-3-Clause License (which shall apply if the TC makes no license selection in its approval action); Apache License v 2.0; CC-BY 2.0; CC-BY 4.0; Eclipse Public License v 1.0.
> 
> OK that's doable and completely reasonable.
> 
> > I am not a lawyer, but can full OS images we licensed under one of
> > these? If not, you are better off creating the repository itself outside
> > the confines of OASIS.
> 
> We'd be hosting essentially build recipes for the images, for tests
> where we use other people's code. The build recipes would have the
> same license.
> 
> If I understand the CLA requirement correctly, even if the
> contribution is merged by a maintainer who has signed the CLA, the
> original contributor who has no write access to the repository, has to
> sign the CLA as well?


I think is would be the OASIS equivalent of the Linux DCO:
	Each person making a repo contribution must be bound to the terms of the
	CLA, by obtaining their signature (which may be an equivalent electronic
	assent)

If you like I can reach out to OASIS to confirm.

> > You can still use e.g. virtio-dev mailing list for communication.
> >
> > > --
> > > Manos Pitsidianakis
> > > Emulation and Virtualization Engineer at Linaro Ltd
> >

Re: Central repo for VirtIO conformance tests?

[Manos Pitsidianakis]

On Mon, May 19, 2025 at 11:15 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Mon, May 19, 2025 at 11:03:43AM +0300, Manos Pitsidianakis wrote:
> > Hello Michael, thanks for the reply,
> >
> > On Mon, May 19, 2025 at 10:54 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Mon, May 19, 2025 at 10:43:33AM +0300, Manos Pitsidianakis wrote:
> > > > Pinging this thread to ask the OASIS people: Would it be possible to
> > > > host a software project (this testsuite) under OASIS but with a
> > > > lighter contribution process? We expect everyone to be contributing to
> > > > them, from open source/proprietary hypervisor developers, to kernel/OS
> > > > developers, hobbyists and professionals alike. So I think a low
> > > > barrier of entry would be reasonable here.
> > >
> > > Take a look here pls:
> > > https://www.oasis-open.org/open-repositories/#licensingRules
> > >
> > >
> > >
> > > > Seeing as a test suite is essentially tied to the published VIRTIO
> > > > spec itself it also makes it a good fit semantically. Having a suite
> > > > with multiple VIRTIO implementations including slim images with Linux
> > > > would also give a wider frame of reference for downstream consumers of
> > > > the spec. Currently, the Linux kernel is the de facto (because it's
> > > > also open source) VIRTIO implementation on the frontend side and
> > > > oftentimes we end up looking at what Linux does to implement backends.
> > > >
> > > > We'd like to potentially:
> > > >
> > > > - Expand the array of rootfs images with more Linux userspace tests
> > > > for more devices
> > > > - Expand the array of rootfs images with other open source OSes as
> > > > they gain VIRTIO functionalities
> > > > - Implement missing VIRTIO spec features in our unikernel test and
> > > > also exercise realistic but basic VIRTIO command use scenarios for
> > > > more devices
> > > > - (This is on my personal wishlist) write a VIRTIO fuzzer framework,
> > > > similar in spirit to Google's syzkaller project (unsupervised
> > > > coverage-guided kernel fuzzer)
> > > > - As a stretch goal, provide a reference test runner framework to run
> > > > the images with QEMU and rust-vmm vhost-user backends for people to
> > > > adapt to their setups.
> > > >
> > > > PS: The git repositories Alex linked are temporarily inaccessible due
> > > > to internal unrelated issues, and will be public again.
> > >
> > >
> > > At the moment, from the above link, and if you want it tied to OASIS
> > > open repositories, it has to be one of:
> > >
> > > BSD-3-Clause License (which shall apply if the TC makes no license selection in its approval action); Apache License v 2.0; CC-BY 2.0; CC-BY 4.0; Eclipse Public License v 1.0.
> >
> > OK that's doable and completely reasonable.
> >
> > > I am not a lawyer, but can full OS images we licensed under one of
> > > these? If not, you are better off creating the repository itself outside
> > > the confines of OASIS.
> >
> > We'd be hosting essentially build recipes for the images, for tests
> > where we use other people's code. The build recipes would have the
> > same license.
> >
> > If I understand the CLA requirement correctly, even if the
> > contribution is merged by a maintainer who has signed the CLA, the
> > original contributor who has no write access to the repository, has to
> > sign the CLA as well?
>
>
> I think is would be the OASIS equivalent of the Linux DCO:
>         Each person making a repo contribution must be bound to the terms of the
>         CLA, by obtaining their signature (which may be an equivalent electronic
>         assent)

So just a Signed-off-by? That massively simplifies things!

> If you like I can reach out to OASIS to confirm.

Yes, that'd be very helpful, thank you.


-- 
Manos Pitsidianakis
Emulation and Virtualization Engineer at Linaro Ltd

Re: Central repo for VirtIO conformance tests?

[Michael S. Tsirkin]

On Mon, May 19, 2025 at 11:18:36AM +0300, Manos Pitsidianakis wrote:
> On Mon, May 19, 2025 at 11:15 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Mon, May 19, 2025 at 11:03:43AM +0300, Manos Pitsidianakis wrote:
> > > Hello Michael, thanks for the reply,
> > >
> > > On Mon, May 19, 2025 at 10:54 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Mon, May 19, 2025 at 10:43:33AM +0300, Manos Pitsidianakis wrote:
> > > > > Pinging this thread to ask the OASIS people: Would it be possible to
> > > > > host a software project (this testsuite) under OASIS but with a
> > > > > lighter contribution process? We expect everyone to be contributing to
> > > > > them, from open source/proprietary hypervisor developers, to kernel/OS
> > > > > developers, hobbyists and professionals alike. So I think a low
> > > > > barrier of entry would be reasonable here.
> > > >
> > > > Take a look here pls:
> > > > https://www.oasis-open.org/open-repositories/#licensingRules
> > > >
> > > >
> > > >
> > > > > Seeing as a test suite is essentially tied to the published VIRTIO
> > > > > spec itself it also makes it a good fit semantically. Having a suite
> > > > > with multiple VIRTIO implementations including slim images with Linux
> > > > > would also give a wider frame of reference for downstream consumers of
> > > > > the spec. Currently, the Linux kernel is the de facto (because it's
> > > > > also open source) VIRTIO implementation on the frontend side and
> > > > > oftentimes we end up looking at what Linux does to implement backends.
> > > > >
> > > > > We'd like to potentially:
> > > > >
> > > > > - Expand the array of rootfs images with more Linux userspace tests
> > > > > for more devices
> > > > > - Expand the array of rootfs images with other open source OSes as
> > > > > they gain VIRTIO functionalities
> > > > > - Implement missing VIRTIO spec features in our unikernel test and
> > > > > also exercise realistic but basic VIRTIO command use scenarios for
> > > > > more devices
> > > > > - (This is on my personal wishlist) write a VIRTIO fuzzer framework,
> > > > > similar in spirit to Google's syzkaller project (unsupervised
> > > > > coverage-guided kernel fuzzer)
> > > > > - As a stretch goal, provide a reference test runner framework to run
> > > > > the images with QEMU and rust-vmm vhost-user backends for people to
> > > > > adapt to their setups.
> > > > >
> > > > > PS: The git repositories Alex linked are temporarily inaccessible due
> > > > > to internal unrelated issues, and will be public again.
> > > >
> > > >
> > > > At the moment, from the above link, and if you want it tied to OASIS
> > > > open repositories, it has to be one of:
> > > >
> > > > BSD-3-Clause License (which shall apply if the TC makes no license selection in its approval action); Apache License v 2.0; CC-BY 2.0; CC-BY 4.0; Eclipse Public License v 1.0.
> > >
> > > OK that's doable and completely reasonable.
> > >
> > > > I am not a lawyer, but can full OS images we licensed under one of
> > > > these? If not, you are better off creating the repository itself outside
> > > > the confines of OASIS.
> > >
> > > We'd be hosting essentially build recipes for the images, for tests
> > > where we use other people's code. The build recipes would have the
> > > same license.
> > >
> > > If I understand the CLA requirement correctly, even if the
> > > contribution is merged by a maintainer who has signed the CLA, the
> > > original contributor who has no write access to the repository, has to
> > > sign the CLA as well?
> >
> >
> > I think is would be the OASIS equivalent of the Linux DCO:
> >         Each person making a repo contribution must be bound to the terms of the
> >         CLA, by obtaining their signature (which may be an equivalent electronic
> >         assent)
> 
> So just a Signed-off-by? That massively simplifies things!


Well we need to make sure people read CLA and know what
it implies.
Which unfortunately I can not tell you what it is since
CLA at OASIS site is down:
https://www.oasis-open.org/resources/open-repositories/cla/individual-cla


I will reach out to OASIS to figure it out.

> > If you like I can reach out to OASIS to confirm.
> 
> Yes, that'd be very helpful, thank you.
> 
> 
> -- 
> Manos Pitsidianakis
> Emulation and Virtualization Engineer at Linaro Ltd

Re: Central repo for VirtIO conformance tests?

[Michael S. Tsirkin]

On Mon, May 19, 2025 at 11:18:36AM +0300, Manos Pitsidianakis wrote:
> On Mon, May 19, 2025 at 11:15 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Mon, May 19, 2025 at 11:03:43AM +0300, Manos Pitsidianakis wrote:
> > > Hello Michael, thanks for the reply,
> > >
> > > On Mon, May 19, 2025 at 10:54 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Mon, May 19, 2025 at 10:43:33AM +0300, Manos Pitsidianakis wrote:
> > > > > Pinging this thread to ask the OASIS people: Would it be possible to
> > > > > host a software project (this testsuite) under OASIS but with a
> > > > > lighter contribution process? We expect everyone to be contributing to
> > > > > them, from open source/proprietary hypervisor developers, to kernel/OS
> > > > > developers, hobbyists and professionals alike. So I think a low
> > > > > barrier of entry would be reasonable here.
> > > >
> > > > Take a look here pls:
> > > > https://www.oasis-open.org/open-repositories/#licensingRules
> > > >
> > > >
> > > >
> > > > > Seeing as a test suite is essentially tied to the published VIRTIO
> > > > > spec itself it also makes it a good fit semantically. Having a suite
> > > > > with multiple VIRTIO implementations including slim images with Linux
> > > > > would also give a wider frame of reference for downstream consumers of
> > > > > the spec. Currently, the Linux kernel is the de facto (because it's
> > > > > also open source) VIRTIO implementation on the frontend side and
> > > > > oftentimes we end up looking at what Linux does to implement backends.
> > > > >
> > > > > We'd like to potentially:
> > > > >
> > > > > - Expand the array of rootfs images with more Linux userspace tests
> > > > > for more devices
> > > > > - Expand the array of rootfs images with other open source OSes as
> > > > > they gain VIRTIO functionalities
> > > > > - Implement missing VIRTIO spec features in our unikernel test and
> > > > > also exercise realistic but basic VIRTIO command use scenarios for
> > > > > more devices
> > > > > - (This is on my personal wishlist) write a VIRTIO fuzzer framework,
> > > > > similar in spirit to Google's syzkaller project (unsupervised
> > > > > coverage-guided kernel fuzzer)
> > > > > - As a stretch goal, provide a reference test runner framework to run
> > > > > the images with QEMU and rust-vmm vhost-user backends for people to
> > > > > adapt to their setups.
> > > > >
> > > > > PS: The git repositories Alex linked are temporarily inaccessible due
> > > > > to internal unrelated issues, and will be public again.
> > > >
> > > >
> > > > At the moment, from the above link, and if you want it tied to OASIS
> > > > open repositories, it has to be one of:
> > > >
> > > > BSD-3-Clause License (which shall apply if the TC makes no license selection in its approval action); Apache License v 2.0; CC-BY 2.0; CC-BY 4.0; Eclipse Public License v 1.0.
> > >
> > > OK that's doable and completely reasonable.
> > >
> > > > I am not a lawyer, but can full OS images we licensed under one of
> > > > these? If not, you are better off creating the repository itself outside
> > > > the confines of OASIS.
> > >
> > > We'd be hosting essentially build recipes for the images, for tests
> > > where we use other people's code. The build recipes would have the
> > > same license.
> > >
> > > If I understand the CLA requirement correctly, even if the
> > > contribution is merged by a maintainer who has signed the CLA, the
> > > original contributor who has no write access to the repository, has to
> > > sign the CLA as well?
> >
> >
> > I think is would be the OASIS equivalent of the Linux DCO:
> >         Each person making a repo contribution must be bound to the terms of the
> >         CLA, by obtaining their signature (which may be an equivalent electronic
> >         assent)
> 
> So just a Signed-off-by? That massively simplifies things!
> 
> > If you like I can reach out to OASIS to confirm.
> 
> Yes, that'd be very helpful, thank you.


Well OASIS got back to us with the following:

	they have a tool that requires you to create an account then you file
	a CLA. If using email maintainer needs to check CLA in place.
	If using github pull requests, it checks automatically.
	Additionally each employer also has to file a different employer CLA.
	This last one for now has to just be emailed but at some point
	they will develop a tool to do that. They said "shortly" but no
	promises.
	Until they do, maintainer has to remember to check employer CLA
	is on file when merging.

If you want to bother with all this let me know I will get you set up.

	Or another option is to just create a repo elsewhere.  If we
	really want to we can make people accept CLA, then if at some point
	OASIS has better infra, transition to that.  I'd then use BSD-3-Clause
	License if that's ok, in case we do want to move to OASIS later.


	How do you intend to work, pull requests or email?






> 
> -- 
> Manos Pitsidianakis
> Emulation and Virtualization Engineer at Linaro Ltd