From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E0BA7C7EE22 for ; Thu, 11 May 2023 15:53:42 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1px8bd-0005fF-DI; Thu, 11 May 2023 11:53:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1px8bb-0005er-3x for qemu-devel@nongnu.org; Thu, 11 May 2023 11:53:23 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1px8bZ-0004lP-9K for qemu-devel@nongnu.org; Thu, 11 May 2023 11:53:22 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1683820400; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=FnjzAjrd5HkJ402FDaP9KjXSX3hhMiKxf8QMyQhpYn0=; b=CUXT4/y+sNKczQM1r3vhALbYYdqDoxY0L0p9xcfs79iehOf86SCj3AGg/m2BkCoEw8/60i VGtwyvjLVlCYEn64XI0qz3dsFqd3mfKnEC+1tEgBMPXcH92DG6MXaAx1oaR5/fM+4xeaMd GdZuEGTaVqp5BT7EF1jtGenBd/XheB0= Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-631-hcegDCfPMB6VoqOlwmPEFA-1; Thu, 11 May 2023 11:53:18 -0400 X-MC-Unique: hcegDCfPMB6VoqOlwmPEFA-1 Received: by mail-pf1-f198.google.com with SMTP id d2e1a72fcca58-6439bc1e3aaso5203810b3a.2 for ; Thu, 11 May 2023 08:53:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683820398; x=1686412398; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=FnjzAjrd5HkJ402FDaP9KjXSX3hhMiKxf8QMyQhpYn0=; b=X4SNWUj7V6d0WQhS/SHuXmOBpBcqQP1cmBf2jABrp7K6WlLYSRhIG8LHf+3tIKEJ1m A3jsbX4eOtSa2bdOfcJrl1WhiSXT/ZjlYPBGA2FjUp611ACOIiu+gvNxOCSaPLMDLQP+ kjt3QIkZzNYzhrcdo6VDiY2iyLwB6/YyAGXXDZ5fyBrNgpmo9NB20cPtyYEyBX9clCA/ jmTbLgpamWtdA42un5nA8WdF1RWFdyPakzS0BarNbt5/rkzFCYgbtOXpmW4dQMmJCqDk 5Mw/yg6rB7ZUzn1qow4IvNYDTXXBNaFgMSr2WHJ4yVDzFbIzAagofEUXzAo9RYKidjyV NhZw== X-Gm-Message-State: AC+VfDzl4/p3WEBI/9XzXgKNVzSg4N71eC7mdbjx686PVKqdra9H4hWt cIoZnRJw4ZaZ2vJV0J0oQORRxtcMDfl1lZfoF6szk/DzMXxXAJTlGqpjm+jciEah7S03d3UupKW mqwKFBO2897wQW/IX4NciXmoAKYmYhUU= X-Received: by 2002:a05:6a20:2452:b0:103:558c:516 with SMTP id t18-20020a056a20245200b00103558c0516mr7213886pzc.55.1683820397824; Thu, 11 May 2023 08:53:17 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4sfKrCssOV4yuN3P04FH06Gdzl/x2YDpNS+0D6z1uOCt6m3QSvxiMt8x4QYLBjooK+tkDjWt23MJYVemCLsDE= X-Received: by 2002:a05:6a20:2452:b0:103:558c:516 with SMTP id t18-20020a056a20245200b00103558c0516mr7213863pzc.55.1683820397537; Thu, 11 May 2023 08:53:17 -0700 (PDT) MIME-Version: 1.0 References: <20230511035435.734312-1-jsnow@redhat.com> <20230511035435.734312-8-jsnow@redhat.com> In-Reply-To: From: John Snow Date: Thu, 11 May 2023 11:53:06 -0400 Message-ID: Subject: Re: [PATCH 07/27] mkvenv: add diagnose() method for ensure() failures To: Paolo Bonzini Cc: qemu-devel , Warner Losh , Peter Maydell , =?UTF-8?Q?Daniel_P=2E_Berrang=C3=A9?= , Ani Sinha , Beraldo Leal , Markus Armbruster , Ryo ONODERA , Kyle Evans , =?UTF-8?B?QWxleCBCZW5uw6ll?= , Michael Roth , Reinoud Zandijk , =?UTF-8?B?TWFyYy1BbmRyw6kgTHVyZWF1?= , Cleber Rosa , Thomas Huth , "Michael S. Tsirkin" , =?UTF-8?Q?Philippe_Mathieu=2DDaud=C3=A9?= , Wainer dos Santos Moschetta Content-Type: multipart/alternative; boundary="000000000000a940f105fb6cfc8c" Received-SPF: pass client-ip=170.10.129.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org --000000000000a940f105fb6cfc8c Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, May 11, 2023, 2:53 AM Paolo Bonzini wrote: > On 5/11/23 05:54, John Snow wrote: > > This is a routine that is designed to print some usable info for human > > beings back out to the terminal if/when "mkvenv ensure" fails to locate > > or install a package during configure time, such as meson or sphinx. > > > > Since we are requiring that "meson" and "sphinx" are installed to the > > same Python environment as QEMU is configured to build with, this can > > produce some surprising failures when things are mismatched. This metho= d > > is here to try and ease that sting by offering some actionable > > diagnosis. > > I think this is a bit too verbose/scary, especially the "Ouch" for > what was a totally normal occurrence before (no "--enable-docs" and sphin= x > absent or too old) and the "ERROR" from "pip install --no-index". > > Here is an attempt to tone it down: > > diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py > index 8e097e4759e3..5d30174a9aff 100644 > --- a/python/scripts/mkvenv.py > +++ b/python/scripts/mkvenv.py > @@ -74,6 +74,7 @@ > Iterator, > Optional, > Sequence, > + Tuple, > Union, > ) > import venv > @@ -594,7 +595,7 @@ def diagnose( > online: bool, > wheels_dir: Optional[Union[str, Path]], > prog: Optional[str], > -) -> str: > +) -> Tuple[str, bool]: > """ > Offer a summary to the user as to why a package failed to be > installed. > > @@ -610,6 +611,9 @@ def diagnose( > """ > # pylint: disable=3Dtoo-many-branches > > + # Some errors are not particularly serious > + bad =3D False > + > pkg_name =3D pkgname_from_depspec(dep_spec) > pkg_version =3D None > > @@ -654,11 +658,11 @@ def diagnose( > "No suitable version found in, or failed to install from" > f" '{wheels_dir}'." > ) > - else: > - lines.append("No local package directory was searched.") > + bad =3D True > > if online: > lines.append("A suitable version could not be obtained from > PyPI.") > + bad =3D True > else: > lines.append( > "mkvenv was configured to operate offline and did not check > PyPI." > @@ -675,10 +679,14 @@ def diagnose( > f"Typically this means that '{prog}' has been installed= " > "against a different Python interpreter on your system.= " > ) > + bad =3D True > > lines =3D [f" =E2=80=A2 {line}" for line in lines] > - lines.insert(0, f"Could not ensure availability of '{dep_spec}':") > - return os.linesep.join(lines) > + if bad: > + lines.insert(0, f"Could not ensure availability of '{dep_spec}':= ") > + else: > + lines.insert(0, f"'{dep_spec}' not found:") > + return os.linesep.join(lines), bad > > > def pip_install( > @@ -731,7 +739,7 @@ def _do_ensure( > dep_specs, > online=3DFalse, > wheels_dir=3Dwheels_dir, > - devnull=3Donline and not wheels_dir, > + devnull=3Dnot wheels_dir, > ) > # (A) or (B) happened. Success. > except subprocess.CalledProcessError: > @@ -778,7 +786,10 @@ def ensure( > _do_ensure(dep_specs, online, wheels_dir) > except subprocess.CalledProcessError as exc: > # Well, that's not good. > - raise Ouch(diagnose(dep_specs[0], online, wheels_dir, prog)) fro= m > exc > + msg, bad =3D diagnose(dep_specs[0], online, wheels_dir, prog) > + if bad: > + raise Ouch(msg) from exc > + print("", msg, "\n", sep=3D"\n", file=3Dsys.stderr) > > > def post_venv_setup() -> None: > > > Paolo > You're right, in the "optional" case for sphinx the error isn't really *that* bad or serious. I'll try to work this or something very similar to it in. I was thinking it could be up to the caller to discard the input, but I suppose we can also route the semantics down into the tool, too. I'll play with it. --js > --000000000000a940f105fb6cfc8c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Thu, May 11, 2023, 2:53 AM Paolo Bonzini <pbonzini@redhat.com> wrote:
On 5/11/23 05:54, John Snow wrote:
> This is a routine that is designed to print some usable info for human=
> beings back out to the terminal if/when "mkvenv ensure" fail= s to locate
> or install a package during configure time, such as meson or sphinx. >
> Since we are requiring that "meson" and "sphinx" a= re installed to the
> same Python environment as QEMU is configured to build with, this can<= br> > produce some surprising failures when things are mismatched. This meth= od
> is here to try and ease that sting by offering some actionable
> diagnosis.

I think this is a bit too verbose/scary, especially the "Ouch" fo= r
what was a totally normal occurrence before (no "--enable-docs" a= nd sphinx
absent or too old) and the "ERROR" from "pip install --no-in= dex".

Here is an attempt to tone it down:

diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py
index 8e097e4759e3..5d30174a9aff 100644
--- a/python/scripts/mkvenv.py
+++ b/python/scripts/mkvenv.py
@@ -74,6 +74,7 @@
=C2=A0 =C2=A0 =C2=A0 Iterator,
=C2=A0 =C2=A0 =C2=A0 Optional,
=C2=A0 =C2=A0 =C2=A0 Sequence,
+=C2=A0 =C2=A0 Tuple,
=C2=A0 =C2=A0 =C2=A0 Union,
=C2=A0 )
=C2=A0 import venv
@@ -594,7 +595,7 @@ def diagnose(
=C2=A0 =C2=A0 =C2=A0 online: bool,
=C2=A0 =C2=A0 =C2=A0 wheels_dir: Optional[Union[str, Path]],
=C2=A0 =C2=A0 =C2=A0 prog: Optional[str],
-) -> str:
+) -> Tuple[str, bool]:
=C2=A0 =C2=A0 =C2=A0 """
=C2=A0 =C2=A0 =C2=A0 Offer a summary to the user as to why a package failed= to be installed.

@@ -610,6 +611,9 @@ def diagnose(
=C2=A0 =C2=A0 =C2=A0 """
=C2=A0 =C2=A0 =C2=A0 # pylint: disable=3Dtoo-many-branches

+=C2=A0 =C2=A0 # Some errors are not particularly serious
+=C2=A0 =C2=A0 bad =3D False
+
=C2=A0 =C2=A0 =C2=A0 pkg_name =3D pkgname_from_depspec(dep_spec)
=C2=A0 =C2=A0 =C2=A0 pkg_version =3D None

@@ -654,11 +658,11 @@ def diagnose(
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 "No suitable version = found in, or failed to install from"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 f" '{wheels_dir}&= #39;."
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 )
-=C2=A0 =C2=A0 else:
-=C2=A0 =C2=A0 =C2=A0 =C2=A0 lines.append("No local package directory = was searched.")
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 bad =3D True

=C2=A0 =C2=A0 =C2=A0 if online:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 lines.append("A suitable version co= uld not be obtained from PyPI.")
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 bad =3D True
=C2=A0 =C2=A0 =C2=A0 else:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 lines.append(
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 "mkvenv was configure= d to operate offline and did not check PyPI."
@@ -675,10 +679,14 @@ def diagnose(
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 f"Typic= ally this means that '{prog}' has been installed "
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 "agains= t a different Python interpreter on your system."
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 )
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 bad =3D True

=C2=A0 =C2=A0 =C2=A0 lines =3D [f" =E2=80=A2 {line}" for line in = lines]
-=C2=A0 =C2=A0 lines.insert(0, f"Could not ensure availability of '= ;{dep_spec}':")
-=C2=A0 =C2=A0 return os.linesep.join(lines)
+=C2=A0 =C2=A0 if bad:
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 lines.insert(0, f"Could not ensure availa= bility of '{dep_spec}':")
+=C2=A0 =C2=A0 else:
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 lines.insert(0, f"'{dep_spec}' no= t found:")
+=C2=A0 =C2=A0 return os.linesep.join(lines), bad


=C2=A0 def pip_install(
@@ -731,7 +739,7 @@ def _do_ensure(
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 dep_specs,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 online=3DFalse,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 wheels_dir=3Dwheels_dir, -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 devnull=3Donline and not wheels_= dir,
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 devnull=3Dnot wheels_dir,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 )
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # (A) or (B) happened. Success.
=C2=A0 =C2=A0 =C2=A0 except subprocess.CalledProcessError:
@@ -778,7 +786,10 @@ def ensure(
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 _do_ensure(dep_specs, online, wheels_dir= )
=C2=A0 =C2=A0 =C2=A0 except subprocess.CalledProcessError as exc:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # Well, that's not good.
-=C2=A0 =C2=A0 =C2=A0 =C2=A0 raise Ouch(diagnose(dep_specs[0], online, whee= ls_dir, prog)) from exc
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 msg, bad =3D diagnose(dep_specs[0], online, wh= eels_dir, prog)
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 if bad:
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 raise Ouch(msg) from exc
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 print("", msg, "\n", sep= =3D"\n", file=3Dsys.stderr)


=C2=A0 def post_venv_setup() -> None:


Paolo

You're right, in the "optional" case for sphinx the erro= r isn't really *that* bad or serious. I'll try to work this or some= thing very similar to it in.

I was thinking it could be up to the caller to discard the input, but = I suppose we can also route the semantics down into the tool, too.

I'll play with it.

--js
--000000000000a940f105fb6cfc8c--