Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

["Michael S. Tsirkin" <mst@redhat.com>]

On Fri, Jul 01, 2022 at 01:20:30PM +0530, Ani Sinha wrote:
> On Fri, Jul 1, 2022 at 1:08 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Fri, Jul 01, 2022 at 12:58:33PM +0530, Ani Sinha wrote:
> > > On Fri, Jul 1, 2022 at 12:23 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Fri, Jul 01, 2022 at 06:12:14AM +0200, Thomas Huth wrote:
> > > > > I even wouldn't mind if you put your python stuff in a new directory like
> > > > > tests/pytests/ for example, as long as it downloads your binaries separately
> > > > > - as I wrote in another mail, the avocado framework rather looks like an
> > > > > oddball in our test framework nowadays since it uses a separate test runner
> > > > > and not the meson test harness, so having a new approach for python-based
> > > > > tests is maybe even a good idea. I just really want to avoid that this goes
> > > > > into tests/qtest (since it really does not belong there), and please don't
> > > > > add more external stuff via git submodules, that's really the wrong approach
> > > > > for this.
> > > >
> > > > I get it, people hate submodules with passion.  I think trying another
> > > > approach for testing that is neither avocado nor qtest is
> > > > not too bad. As long as this is not user visible, we can
> > > > allow ourselves space to experiment.
> > > >
> > > > OK so, how about this:
> > > > - put it in a new directory: tests/roms?
> > > > - create repo for a fork of biosbits under git.qemu.org
> > > > - roll our own analog to git submodules: a script
> > > >   that clones the repo
> > >
> > > No need to clone the whole repo. We can simply download the binaries
> > > that the girlab CI job would generate from the bits sources in that
> > > repo.
> > > We need to clone if we are always building bits from source for every
> > > test. That is not necessary IMHO since much of the bits package would
> > > remain as is without modification.
> >
> > IMHO CI job idea isn't great since isn't versioned at all, is it?
> 
> bits is versioned yes, in a crude way. every time you make a commit in
> the top level repo, the version would increment by one.

Is it easy to find out which source was this generated from?
And is there a promise to keep these around indefinitely?

> > Also as long as test passes, sure. But if it fails one will
> > need the sources to investigate.
> 
> sources might also be needed to write the tests.
> 
> >
> > Let's start with building things from source.
> 
> hmm. bitys uses old autotools, not ninja and takes about 10/15 mins to
> build depending on parallelity and build host.

Right. But whoever wants to use these just needs to do it once.


> Add an option
> > of prebuilt binaries as an optimization once things
> > stabilize.
> >
> >
> > > > - new target make check-roms,
> > >
> > > I think make pytest or some such is better and more generic if other
> > > such tests in other areas follow suit.
> >
> > The name is not critical in my mind, but I think we need to decide
> > what exactly differentiates it from other tests.
> >
> >
> > >
> > > if the clone exists locally -
> > > >   run the test, if not - skip it
> > >
> > > if download of the bits binaries fail, skip it.
> >
> > You seem to be recreating either git or avocado or both here.
> >
> > Personally I want something that works offline.
> >
> >
> >
> > > > - as for using pre-generates ISOs as an optimization,
> > > >   I'm not sure how important that is, if yes -
> > > >   we can add another repo and another make target along the
> > > >   same lines
> > > >
> > > >
> > > >
> > > > --
> > > > MST
> > > >
> >