Re: [Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[Cleber Rosa <crosa@redhat.com>]

On Fri, Sep 06, 2019 at 12:14:15PM -0300, Eduardo Habkost wrote:
> On Fri, Sep 06, 2019 at 10:52:02AM -0400, Cleber Rosa wrote:
> > On Fri, Sep 06, 2019 at 11:43:58AM -0300, Eduardo Habkost wrote:
> > > On Fri, Sep 06, 2019 at 10:34:16AM +1000, David Gibson wrote:
> > > > On Thu, Sep 05, 2019 at 09:38:24AM -0400, Cleber Rosa wrote:
> > > > > On Thu, Sep 05, 2019 at 12:09:24PM +1000, David Gibson wrote:
> > > > > > On Wed, Sep 04, 2019 at 03:57:17PM -0400, Cleber Rosa wrote:
> > > > > > > 
> > > > > > > Hi David,
> > > > > > > 
> > > > > > > I've pushed a branch here (most of the commits have already been sent
> > > > > > > separately):
> > > > > > > 
> > > > > > >    https://github.com/clebergnu/qemu/tree/ppc64
> > > > > > > 
> > > > > > > I've tested on a RHEL 8 ppc64le Power 9, and it seems to work for me.
> > > > > > > My steps for building QEMU:
> > > > > > > 
> > > > > > >   - Configured with: '/home/cleber/src/qemu/configure' '--enable-slirp=git' '--python=/usr/bin/python3' '--target-list=x86_64-softmmu,ppc64-softmmu,arm-softmmu,aarch64-softmmu,mips-softmmu,mipsel-softmmu,mips64-softmmu,mips64el-softmmu,sh4-softmmu,s390x-softmmu,alpha-softmmu,m68k-softmmu,riscv64-softmmu'
> > > > > > >   - make
> > > > > > >   - make check-acceptance
> > > > > > > 
> > > > > > > Would you be able to test if that branch works smoothly for you?
> > > > > > 
> > > > > > So, with this tree I'm no longer getting problems if openssl-devel is
> > > > > > not installed, so that much looks good.
> > > > > > 
> > > > > > I am getting some different errors - I was seeing this before (with
> > > > > > openssl-devel installed) sometimes, but only sometimes.  I haven't yet
> > > > > > worked out a pattern for when they appeared.  They also don't appear
> > > > > > to be fatal, the rest of the tests seem to be running ok.  Any ideas?
> > > > > > 
> > > > > >  VENV    /home/dwg/qemu/build/rhel8/tests/venv
> > > > > >   PIP     /home/dwg/qemu/tests/requirements.txt
> > > > > >   AVOCADO tests/acceptance
> > > > > > Error running method "run" of plugin "html": 'Namespace' object has no attribute 'get'
> > > > > > Error running method "run" of plugin "varianter_cit": 'Namespace' object has no attribute 'get'
> > > > > 
> > > > > Hi David,
> > > > > 
> > > > > Yes, those should not be fatal.  Anyway, they are caused by two things:
> > > > > 
> > > > >  1) A second set of Avocado plugins installation with different versions
> > > > >     than the Avocado installation on QEMU's tests venv
> > > > 
> > > > Ah!  That'll be from the Avocado setup that Lukaš set up on there for
> > > > some different tests.
> > > > 
> > > > >  2) The tests' venv, which is created with "--system-site-packages",
> > > > >     option that is described as "Give the virtual environment access
> > > > >     to the system site-packages dir."
> > > > > 
> > > > > The motivation to have "--system-site-packages" is that, if the user
> > > > > had any of the "requirements.txt" depedencies installed system wide,
> > > > > the venv would simply use it and skip downloads.  Maybe we should make
> > > > > that an optional feature, disabled by default, so that the test's venv
> > > > > is better isolated and more predictable by default.
> > > > 
> > > > If pip doesn't have good enough version tracking that it's verifying
> > > > the correct versions of the installed packages as well as the
> > > > presence, then yes I tend to agree that keeping it better isolated
> > > > would be preferable.
> > > 
> > > pip ensures the right version of packages we request are
> > > installed, but Avocado plugins are separate packages and we don't
> > > tell pip anything about the plugins we are not using (but could be
> > > already installed in the system).
> > 
> > Right.
> > 
> > > 
> > > Cleber, does Avocado have any kind of plugin API versioning, so
> > > it could detect incompatible plugins before using them?
> > >
> > 
> > It does not.  And while making an exact version match requirement
> > between plugins and core Avocado would be easy, it would put unfair
> > (IMO) requirements on plugin writers.  It's quite possible that
> > an external plugin would work for many different avocado versions,
> > and the author wouldn't feel like updating it just to satisfy the
> > version check.
> 
> I agree that tying a plugin to a specific Avocado version seems
> too strict.  Are there any rules about API stability that plugin
> writers can expect Avocado to follow?
>

TBH the plugin interfaces[1] are quite stable (have changed very
little), but:

 1) They're still part of the "avocado.core" namespace, which means
    the stability is a best effort, and not a promise/guarantee

 2) Most plugins make use of other "avocado.core" modules, which
    follow the same logic as #1

It's a known fact that it'd be good to have a supportable plugin
API[2], but we haven't had the resources to implement that.

> > 
> > I'm not quite sure we should be trying to solve this problem here, but
> > I'm open to suggestions.
> 
> This will probably become a minor problem once we stop using
> --system-site-packages by default.
>

Exactly, I think at this moment, this is the simple and best course of
action.

> -- 
> Eduardo


Thanks,
- Cleber.

[1] - https://github.com/avocado-framework/avocado/blob/master/avocado/core/plugin_interfaces.py
[2] - https://trello.com/c/rKIfx7rY/1285-create-a-sufficient-and-supportable-plugin-api