From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C60F9C77B73 for ; Thu, 20 Apr 2023 16:55:35 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ppXYj-0006Ut-BU; Thu, 20 Apr 2023 12:55:01 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ppXYc-0006UV-Su for qemu-devel@nongnu.org; Thu, 20 Apr 2023 12:54:55 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ppXYY-00054P-B0 for qemu-devel@nongnu.org; Thu, 20 Apr 2023 12:54:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1682009682; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=1Mt7hNeizwniavCNQpnxrcNDkMWiOCKLmcw3DXQnItw=; b=RhGIn3dnIJaVZdmsnHb+WnxtHg3PPURzlL0BagAsmmGb422cIZvsIZT+xf/6UkF114ytts 2BJFYup30lqLQA5rV2NTuedl11rt1YNenxgiOkNIbGRG426rsQ2ya0TxHeUu88/d5RaL/U fA1SEr/gZdkmXWTI8gzPt1nbuwbNNpE= Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-68-pxunA3ieMu-Un003tYAQoQ-1; Thu, 20 Apr 2023 12:54:38 -0400 X-MC-Unique: pxunA3ieMu-Un003tYAQoQ-1 Received: by mail-pg1-f200.google.com with SMTP id 41be03b00d2f7-5187bd3d365so1082981a12.3 for ; Thu, 20 Apr 2023 09:54:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682009678; x=1684601678; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=1Mt7hNeizwniavCNQpnxrcNDkMWiOCKLmcw3DXQnItw=; b=I073t+f0Sbd+yfbpCns6J8xXzoiuDE26R+XxttLhupSvpPBeaEAgXMaZwwUZVnnLLY cyIp0XCwCLrEjK06YzBrF5Wh5SsQfT9TiBU0plMUKt87Q6VoG9FT2DpVkqHpphjPcXPd yO1yNUMOBXJDSlW/dmc41bU0MCICBN4q7MeluFFHeRSM9/DSzpIyTLy4XwczCBwtu3Ns LpPJOtQPPb/lUsems8tXCGo5zVCIvzEO+R7OCzBXKoCgahXH385txxcWDPLt8dpVxvMa gtKTnFl6fru7tQyXFaovXSzWHrLbRYZIkTlUFlpqU+TOh1TGzzBexbc6PlOL1s0PztOB LDag== X-Gm-Message-State: AAQBX9ev4NQeiTy4QEQt/TjZ61Ac9V8/SSXTvbTS/lkxcxSr/crCxxFF RYVdZIl7iC2eFtMcaDdMrxinyurEicUbvse+Q/BCdcjkErg7NIxiInsCwmMrKA6rJkNaGvSKU6d PxSmoBWDMKcovcbhOKMKlUCayU7xKVaY= X-Received: by 2002:a05:6a21:6d84:b0:ee:84a2:4ad0 with SMTP id wl4-20020a056a216d8400b000ee84a24ad0mr3317304pzb.22.1682009677671; Thu, 20 Apr 2023 09:54:37 -0700 (PDT) X-Google-Smtp-Source: AKy350Y7SyTAOV+E5UNWTlAd9im0v9n5xrGWmU6JHYs+uCzZQllPNLmuYOWrogcgOP4z+4CAO/o4f4iIwsYBoEaSNLQ= X-Received: by 2002:a05:6a21:6d84:b0:ee:84a2:4ad0 with SMTP id wl4-20020a056a216d8400b000ee84a24ad0mr3317279pzb.22.1682009677386; Thu, 20 Apr 2023 09:54:37 -0700 (PDT) MIME-Version: 1.0 References: <20230417134321.3627231-1-alex.bennee@linaro.org> <20230417134321.3627231-2-alex.bennee@linaro.org> <87pm82e74d.fsf@linaro.org> <74be23e6-015b-cb1d-967e-65fc32347553@linaro.org> In-Reply-To: <74be23e6-015b-cb1d-967e-65fc32347553@linaro.org> From: John Snow Date: Thu, 20 Apr 2023 12:54:26 -0400 Message-ID: Subject: Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0 To: =?UTF-8?Q?Philippe_Mathieu=2DDaud=C3=A9?= Cc: =?UTF-8?B?QWxleCBCZW5uw6ll?= , Cleber Rosa , Jan Richter , qemu-devel , Peter Maydell , Bastian Koppelmann , Markus Armbruster , Beraldo Leal , Thomas Huth , Stefan Hajnoczi , =?UTF-8?B?TWFyYy1BbmRyw6kgTHVyZWF1?= , =?UTF-8?Q?Daniel_P=2E_Berrang=C3=A9?= , Radoslaw Biernacki , Leif Lindholm , Paolo Bonzini , Wainer dos Santos Moschetta , qemu-arm , Kautuk Consul , Hariharan T S Content-Type: multipart/alternative; boundary="00000000000054679405f9c76538" Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org --00000000000054679405f9c76538 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Apr 20, 2023, 5:19 AM Philippe Mathieu-Daud=C3=A9 wrote: > On 17/4/23 19:44, Alex Benn=C3=A9e wrote: > > > > John Snow writes: > > > >> On Mon, Apr 17, 2023 at 9:43=E2=80=AFAM Alex Benn=C3=A9e > wrote: > >>> > >>> From: Kautuk Consul > >>> > >>> Avocado version 101.0 has a fix to re-compute the checksum > >>> of an asset file if the algorithm used in the *-CHECKSUM > >>> file isn't the same as the one being passed to it by the > >>> avocado user (i.e. the avocado_qemu python module). > >>> In the earlier avocado versions this fix wasn't there due > >>> to which if the checksum wouldn't match the earlier > >>> checksum (calculated by a different algorithm), the avocado > >>> code would start downloading a fresh image from the internet > >>> URL thus making the test-cases take longer to execute. > >>> > >>> Bump up the avocado-framework version to 101.0. > >>> > >>> Signed-off-by: Kautuk Consul > >>> Tested-by: Hariharan T S > >>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com> > >>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org> > >>> > >>> --- > >>> v2 > >>> - limit --max-parallel-tasks $(JOBS_OPTION:-j%=3D%) > >>> --- > >>> tests/Makefile.include | 18 +++++++++++------- > >>> tests/requirements.txt | 2 +- > >>> 2 files changed, 12 insertions(+), 8 deletions(-) > >>> > >>> diff --git a/tests/Makefile.include b/tests/Makefile.include > >>> index 9422ddaece..a4de0ad5a2 100644 > >>> --- a/tests/Makefile.include > >>> +++ b/tests/Makefile.include > >>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv > >>> # download all vm images, according to defined targets > >>> get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, > $(FEDORA_31_DOWNLOAD)) > >>> > >>> +JOBS_OPTION=3D$(lastword -j1 $(filter-out -j, $(filter > -j%,$(MAKEFLAGS)))) > >>> + > >>> check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images > >>> - $(call quiet-command, \ > >>> - $(TESTS_PYTHON) -m avocado \ > >>> - --show=3D$(AVOCADO_SHOW) run > --job-results-dir=3D$(TESTS_RESULTS_DIR) \ > >>> - $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \ > >>> - --filter-by-tags-include-empty-key) \ > >>> - $(AVOCADO_CMDLINE_TAGS) \ > >>> - $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ > >>> + $(call quiet-command, > \ > >>> + $(TESTS_PYTHON) -m avocado > \ > >>> + --show=3D$(AVOCADO_SHOW) run > --job-results-dir=3D$(TESTS_RESULTS_DIR) \ > >>> + $(if $(AVOCADO_TAGS),, > \ > >>> + --filter-by-tags-include-empty > \ > >>> + --filter-by-tags-include-empty-key) > \ > >>> + --max-parallel-tasks $(JOBS_OPTION:-j%=3D%) > \ > >>> + $(AVOCADO_CMDLINE_TAGS) > \ > >>> + $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), > \ > >>> "AVOCADO", "tests/avocado") > >>> > >>> check-acceptance-deprecated-warning: > >>> diff --git a/tests/requirements.txt b/tests/requirements.txt > >>> index 0ba561b6bd..a6f73da681 100644 > >>> --- a/tests/requirements.txt > >>> +++ b/tests/requirements.txt > >>> @@ -2,5 +2,5 @@ > >>> # in the tests/venv Python virtual environment. For more info, > >>> # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 > >>> # Note that qemu.git/python/ is always implicitly installed. > >>> -avocado-framework=3D=3D88.1 > >>> +avocado-framework=3D=3D101.0 > >>> pycdlib=3D=3D1.11.0 > >>> -- > >>> 2.39.2 > >>> > >> > >> I thought there were test failures that prohibited us from bumping the > >> Avocado-Framework version. Did those get rectified recently? > > No, still not working on Darwin. At this point I'm thinking at sending > a patch clarifying testing QEMU with Avocado is not supported on Darwin. > What's broken on Darwin? Is it broken with avocado-framework=3D=3D88.1 too? (Is this a regression?) We might need to consider entirely separate issues: (1) We need to upgrade avocado-framework to a supported LTS version >=3D v9= 0 for dependency harmony with qemu.git/python testing (2) We need to upgrade avocado-framework to >=3D v101.0 to fix the stated checksum issue in this patch (3) We would like avocado tests to work on Darwin. (Have they ever worked? When did they break? etc.) My understanding is that this patch helps with 1 and 2 but not 3, but if Darwin support isn't a regression, this is still a step forward. > > I'm not seeing any with your patch applied. > > Are you using running the Cirrus-CI jobs? > > --00000000000054679405f9c76538 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Thu, Apr 20, 2023, 5:19 AM Philippe Mathieu-Daud=C3= =A9 <philmd@linaro.org> wrot= e:
On 17/4/23 19:44, Alex Benn=C3= =A9e wrote:
>
> John Snow <jsnow@redhat.com> writes:
>
>> On Mon, Apr 17, 2023 at 9:43=E2=80=AFAM Alex Benn=C3=A9e <a= lex.bennee@linaro.org> wrote:
>>>
>>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com= >
>>>
>>> Avocado version 101.0 has a fix to re-compute the checksum
>>> of an asset file if the algorithm used in the *-CHECKSUM
>>> file isn't the same as the one being passed to it by the >>> avocado user (i.e. the avocado_qemu python module).
>>> In the earlier avocado versions this fix wasn't there due<= br> >>> to which if the checksum wouldn't match the earlier
>>> checksum (calculated by a different algorithm), the avocado >>> code would start downloading a fresh image from the internet >>> URL thus making the test-cases take longer to execute.
>>>
>>> Bump up the avocado-framework version to 101.0.
>>>
>>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ib= m.com>
>>> Tested-by: Hariharan T S <hariharan.ts@linux.v= net.ibm.com>
>>> Message-Id: <2023032711503= 0.3418323-2-kconsul@linux.vnet.ibm.com>
>>> Message-Id: <20230330101141.301= 99-10-alex.bennee@linaro.org>
>>>
>>> ---
>>> v2
>>>=C2=A0 =C2=A0 - limit --max-parallel-tasks $(JOBS_OPTION:-j%=3D= %)
>>> ---
>>>=C2=A0 =C2=A0tests/Makefile.include | 18 +++++++++++-------
>>>=C2=A0 =C2=A0tests/requirements.txt |=C2=A0 2 +-
>>>=C2=A0 =C2=A02 files changed, 12 insertions(+), 8 deletions(-)<= br> >>>
>>> diff --git a/tests/Makefile.include b/tests/Makefile.include >>> index 9422ddaece..a4de0ad5a2 100644
>>> --- a/tests/Makefile.include
>>> +++ b/tests/Makefile.include
>>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
>>>=C2=A0 =C2=A0# download all vm images, according to defined tar= gets
>>>=C2=A0 =C2=A0get-vm-images: check-venv $(patsubst %,get-vm-imag= e-fedora-31-%, $(FEDORA_31_DOWNLOAD))
>>>
>>> +JOBS_OPTION=3D$(lastword -j1 $(filter-out -j, $(filter -j%,$(= MAKEFLAGS))))
>>> +
>>>=C2=A0 =C2=A0check-avocado: check-venv $(TESTS_RESULTS_DIR) get= -vm-images
>>> -=C2=A0 =C2=A0 =C2=A0 =C2=A0$(call quiet-command, \
>>> -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 $(TESTS_PYTHON) -m = avocado \
>>> -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 --show=3D$(AVOCADO_= SHOW) run --job-results-dir=3D$(TESTS_RESULTS_DIR) \
>>> -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 $(if $(AVOCADO_TAGS= ),, --filter-by-tags-include-empty \
>>> -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0--filter-by-tags-include-empty-key) \
>>> -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 $(AVOCADO_CMDLINE_T= AGS) \
>>> -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 $(if $(GITLAB_CI),,= --failfast) $(AVOCADO_TESTS), \
>>> +=C2=A0 =C2=A0 =C2=A0 =C2=A0$(call quiet-command,=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0\
>>> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 $(TESTS_PYTHON) -m = avocado=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0\
>>> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 --show=3D$(AVOCADO_= SHOW) run --job-results-dir=3D$(TESTS_RESULTS_DIR)=C2=A0 \
>>> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 $(if $(AVOCADO_TAGS= ),,=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0\
>>> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0--filter-by-tags-include-empty=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 \
>>> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0--filter-by-tags-include-empty-key)=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0\
>>> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0--max-= parallel-tasks $(JOBS_OPTION:-j%=3D%)=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0\
>>> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 $(AVOCADO_CMDLINE_T= AGS)=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 \
>>> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 $(if $(GITLAB_CI),,= --failfast) $(AVOCADO_TESTS),=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0\
>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"AV= OCADO", "tests/avocado")
>>>
>>>=C2=A0 =C2=A0check-acceptance-deprecated-warning:
>>> diff --git a/tests/requirements.txt b/tests/requirements.txt >>> index 0ba561b6bd..a6f73da681 100644
>>> --- a/tests/requirements.txt
>>> +++ b/tests/requirements.txt
>>> @@ -2,5 +2,5 @@
>>>=C2=A0 =C2=A0# in the tests/venv Python virtual environment. Fo= r more info,
>>>=C2=A0 =C2=A0# refer to: https:/= /pip.pypa.io/en/stable/user_guide/#id1
>>>=C2=A0 =C2=A0# Note that qemu.git/python/ is always implicitly = installed.
>>> -avocado-framework=3D=3D88.1
>>> +avocado-framework=3D=3D101.0
>>>=C2=A0 =C2=A0pycdlib=3D=3D1.11.0
>>> --
>>> 2.39.2
>>>
>>
>> I thought there were test failures that prohibited us from bumping= the
>> Avocado-Framework version. Did those get rectified recently?

No, still not working on Darwin. At this point I'm thinking at sending<= br> a patch clarifying testing QEMU with Avocado is not supported on Darwin.

Wha= t's broken on Darwin?=C2=A0

Is it broken with avocado-framework=3D=3D88.1 too? (Is this a reg= ression?)

We might need = to consider entirely separate issues:

(1) We need to upgrade avocado-framework to a supported LTS v= ersion >=3D v90 for dependency harmony with qemu.git/python testing

(2) We need to upgrade avoca= do-framework to >=3D v101.0 to fix the stated checksum issue in this pat= ch

(3) We would like avo= cado tests to work on Darwin. (Have they ever worked? When did they break? = etc.)

My understanding i= s that this patch helps with 1 and 2 but not 3, but if Darwin support isn&#= 39;t a regression, this is still a step forward.

> I'm not seeing any with your patch applied.

Are you using running the Cirrus-CI jobs?

--00000000000054679405f9c76538--