From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 379A2C4332F for ; Fri, 21 Oct 2022 12:29:15 +0000 (UTC) Received: from localhost ([::1] helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1olr9G-00065D-5G for qemu-devel@archiver.kernel.org; Fri, 21 Oct 2022 08:29:14 -0400 Received: from [::1] (helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1olr2w-0003TK-8v for qemu-devel@archiver.kernel.org; Fri, 21 Oct 2022 08:22:42 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1olr2o-0003Sw-4H for qemu-devel@nongnu.org; Fri, 21 Oct 2022 08:22:35 -0400 Received: from mail-io1-xd33.google.com ([2607:f8b0:4864:20::d33]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1olr2l-0005C6-CN for qemu-devel@nongnu.org; Fri, 21 Oct 2022 08:22:33 -0400 Received: by mail-io1-xd33.google.com with SMTP id l127so2100856iof.12 for ; Fri, 21 Oct 2022 05:22:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=anisinha-ca.20210112.gappssmtp.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=TkiXdeeKeu9lj6pSmlQeIg2v9YmzLtvNV1HG3iVU5AA=; b=gXOPR/Dmgd4aZUm7YzNdar5ul4KfimJYzTeecwEr+4537xUHyXyAiWh+PAyg8UunZZ ND0lzYCWXvVP/8b50VmNmw5p9fWK1ftXupdRVkyftq3pUQBWH3+rvILeR48+Vai8prfK poDzsHzwiUZ1jB32fAdvLPfOesIu3GMxVDGM+RW91WBQN3D+wXPFrE9s99AH1aOdJbeR og6hKm0HAH7Tipfv4BhLeF3OqF+rga2MqsRm+G16EXcExOGL6RzxSPP0gjmp2fakmJN2 fKbI/1rg30+hQy0Hlj3AVCrOoYxV1KN2kX162LWGgZuikD0qwJ1MpMMDWuJuvPzfzXEf UebQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=TkiXdeeKeu9lj6pSmlQeIg2v9YmzLtvNV1HG3iVU5AA=; b=FAC4K6S9cWJ+gmYdJXQ0EnODRO4j3u5Fj44bir+elCbijbROlczj+8TljK8FJkRbz0 u8+66wFbaZlVallluhWIfOGQbcjdXatCNFsbm2/ybHy3/1vvfsmsvEeJBskL8aCGiuXc D9BYEyEfee1XzEVEKDOdNYDWTpoYD1HWRiBkZBOvup+0kT7/7AvZv+YGgk9HFqrtoW1a o9AiRvbRSz7ECOs6oUI+EjMUeM4TQ/LJ7XIQUdvvIBxpUh5PKq6Z1vG3ah5O0Uop9KS4 JRmhxoXGdIy2ycCsDSR55Js9RiRfcixskr8dKMFhuhPCfZjyCNJHDbJOA969/Gp8hTBj moTw== X-Gm-Message-State: ACrzQf2OWUnkYR2sUnbFT1xeXbhzi7xRSnqmr+6jLYirHOF3hsD4P3ko U+0ugTS4QZyhwUExHCd8P9xo9qn9UUhcvzH56epaEw== X-Google-Smtp-Source: AMsMyM6M14mDYdGG9f6T7hlCWdyBnT5Q37ls9TiQdT+Ab+C6Wh8cyg1wi+wYlJNrf+yF+Vi8MUpFQcJkaUptI0nF5Xc= X-Received: by 2002:a6b:8f91:0:b0:6bb:ee69:2ffe with SMTP id r139-20020a6b8f91000000b006bbee692ffemr14021771iod.34.1666354949738; Fri, 21 Oct 2022 05:22:29 -0700 (PDT) MIME-Version: 1.0 References: <20221020083810-mutt-send-email-mst@kernel.org> <20221020084311-mutt-send-email-mst@kernel.org> <20221020150857-mutt-send-email-mst@kernel.org> <20221021042449-mutt-send-email-mst@kernel.org> <87k04t7ca6.fsf@linaro.org> <20221021053828-mutt-send-email-mst@kernel.org> <87bkq575m8.fsf@linaro.org> In-Reply-To: <87bkq575m8.fsf@linaro.org> From: Ani Sinha Date: Fri, 21 Oct 2022 17:52:17 +0530 Message-ID: Subject: Re: [PATCH v6 00/10] Introduce new acpi/smbios avocado tests using biosbits To: =?UTF-8?B?QWxleCBCZW5uw6ll?= Cc: "Michael S. Tsirkin" , Beraldo Leal , Cleber Rosa , =?UTF-8?Q?Daniel_P_=2E_Berrang=C3=A9?= , Igor Mammedov , John Snow , Maydell Peter , Paolo Bonzini , =?UTF-8?Q?Philippe_Mathieu=2DDaud=C3=A9?= , Qemu Devel , Thomas Huth , Wainer dos Santos Moschetta Content-Type: multipart/alternative; boundary="000000000000d9681705eb8a7eb4" Received-SPF: none client-ip=2607:f8b0:4864:20::d33; envelope-from=ani@anisinha.ca; helo=mail-io1-xd33.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" --000000000000d9681705eb8a7eb4 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Fri, 21 Oct, 2022, 5:26 pm Alex Benn=C3=A9e, wr= ote: > > Ani Sinha writes: > > > On Fri, Oct 21, 2022 at 3:10 PM Michael S. Tsirkin > wrote: > >> > >> On Fri, Oct 21, 2022 at 10:30:09AM +0100, Alex Benn=C3=A9e wrote: > >> > > >> > Ani Sinha writes: > >> > > >> > > On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin > wrote: > >> > >> > >> > >> On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: > >> > >> > And have multiple platform specific branches in bits that have > fixes for those > >> > >> > platforms so that bits can run there. Plus the existing test ca= n > be enhanced to > >> > >> > pull in binaries from those branches based on the platform on > which it is being > >> > >> > run. > >> > >> > > >> > >> > >> > >> What a mess. > >> > >> Who is going to be testing all these million platforms? > >> > > > >> > > I am not talking about branches in QEMU but branches in bits. > >> > > If you are going to test multiple platforms, you do need to build > bits > >> > > binaries for them. There is no way around it. > >> > > bits is not all platform independent python. It does have binary > executables. > >> > > > >> > > Currently bits is built only for the x86 platform. Other platforms > are > >> > > not tested. I doubt if anyone even tried building bits for arm or > >> > > mips. > >> > > >> > I'm not worried about test bits on other targets, but we do run x86 > >> > targets on a number of hosts. The current reliance on a special > patched > >> > host build tool for only one architecture is the problem. If we jus= t > >> > download the iso that problem goes away. > >> > >> =F0=9F=91=8Dwhat he said. > > > > Yes, in that case the problem is that upstream bits does not pass all > > the test out of the box. Hence we are taking this approach of keeping > > some test scripts in QEMU repo and modifying them. Then generating the > > iso with the modified scripts. It also helps developers who want to > > write new tests or make enhancements to existing tests. > > If modifications need to be made to tests, they need to be versioned. > > We have gone through the route of not using submodules and I am not > > going to open that can of worms again. > > We have added a mirror of biosbits to the QEMU project so there is no > reason why we can't track changes and modifications there (we do this > for TestFloat which is forked from the upstream SoftFloat code). > The whole idea was that say an acpi developer added support for a new table in QEMU, he should write a corresponding test for bits so that the same table is exercised during run time. Making those changes from a single repo (either directly or through a submodule) makes things lit simpler and also keeps things in sync with each other. If we use separate repos for acpi bits test, it will be another mess when comes to developers adding changes and keeping things in sync. Anyways these things should have been brought up earlier. I'm out of the debate. I've sent v7 , incremental work over the last 6 months in my spare time without getting any pay. So take it or scrap it. =F0=9F=98=8A > Maintaining and occasionally re-based "vendor" branch shouldn't be too > hard and would track the changes we've made for QEMU's purposes. > > > We also have no consensus on where to keep the one time built iso that > > we can download for this test you are proposing. > > How big is the eventual ISO? If it's small we could just enable some CI > steps and serve the ISOs directly as tagged build artefacts from GitLab. > > > So let's figure out the above first. Programmatically downloading an > > iso and running tests within a VM would be a much simpler test than > > the one I wrote. We can add a subtest or a brand new test anytime if > > we can figure out the above logistics. > > > >> > >> > > It makes sense to try things incrementally once we have something > going. > >> > > > >> > > Lets discuss this on a separate thread. > >> > > > >> > >> All this does nothing at all to help developers avoid > >> > >> bugs and when they do trigger debug the issue. Which is > >> > >> after all why we have testing. > >> > >> Yes once in a very long while we are going to tweak > >> > >> something in the tests, and for that rare occurence > >> > >> it makes sense to periodically rebuild everything, > >> > >> otherwise code bitrots. > >> > >> > >> > >> But the test is supposed to run within a VM anyway, let's > >> > >> have an image and be done with it. > >> > >> > >> > >> -- > >> > >> MST > >> > >> > >> > > >> > > >> > -- > >> > Alex Benn=C3=A9e > >> > > > -- > Alex Benn=C3=A9e > --000000000000d9681705eb8a7eb4 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Fri, 21 Oct, 2022, 5:26 pm Alex Benn=C3=A9e, <alex.bennee@linaro.org> wrot= e:

Ani Sinha <ani@anisinha.ca> writes:

> On Fri, Oct 21, 2022 at 3:10 PM Michael S. Tsirkin <mst@redhat.com&= gt; wrote:
>>
>> On Fri, Oct 21, 2022 at 10:30:09AM +0100, Alex Benn=C3=A9e wrote:<= br> >> >
>> > Ani Sinha <ani@anisinha.ca> writes:
>> >
>> > > On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin <<= a href=3D"mailto:mst@redhat.com" target=3D"_blank" rel=3D"noreferrer">mst@r= edhat.com> wrote:
>> > >>
>> > >> On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha = wrote:
>> > >> > And have multiple platform specific branches in= bits that have fixes for those
>> > >> > platforms so that bits can run there. Plus the = existing test can be enhanced to
>> > >> > pull in binaries from those branches based on t= he platform on which it is being
>> > >> > run.
>> > >> >
>> > >>
>> > >> What a mess.
>> > >> Who is going to be testing all these million platfor= ms?
>> > >
>> > > I am not talking about branches in QEMU but branches in = bits.
>> > > If you are going to test multiple platforms, you do need= to build bits
>> > > binaries for them. There is no way around it.
>> > > bits is not all platform independent python. It does hav= e binary executables.
>> > >
>> > > Currently bits is built only for the x86 platform. Other= platforms are
>> > > not tested. I doubt if anyone even tried building bits f= or arm or
>> > > mips.
>> >
>> > I'm not worried about test bits on other targets, but we = do run x86
>> > targets on a number of hosts. The current reliance on a speci= al patched
>> > host build tool for only one architecture is the problem. If= =C2=A0 we just
>> > download the iso that problem goes away.
>>
>> =F0=9F=91=8Dwhat he said.
>
> Yes, in that case the problem is that upstream bits does not pass all<= br> > the test out of the box. Hence we are taking this approach of keeping<= br> > some test scripts in QEMU repo and modifying them. Then generating the=
> iso with the modified scripts. It also helps developers who want to > write new tests or make enhancements to existing tests.
> If modifications need to be made to tests, they need to be versioned.<= br> > We have gone through the route of not using submodules and I am not > going to open that can of worms again.

We have added a mirror of biosbits to the QEMU project so there is no
reason why we can't track changes and modifications there (we do this for TestFloat which is forked from the upstream SoftFloat code).

The whole i= dea was that say an acpi developer added support for a new table in QEMU, h= e should write a corresponding test for bits so that the same table is exer= cised during run time. Making those changes from a single repo (either dire= ctly or through a submodule)=C2=A0 makes things lit simpler and also keeps = things in sync with each other. If we use separate repos for acpi bits test= , it will be another mess when comes to developers adding changes and keepi= ng things in sync.=C2=A0

Anyways these things should have been brought up earlier. I'm out of t= he debate.=C2=A0

I'v= e sent v7 , incremental work over the last 6 months in my spare time withou= t getting any pay. So take it or scrap it.=C2=A0
=F0=9F=98=8A
Maintaining and occasionally re-based "vendor" branch shouldn'= ;t be too
hard and would track the changes we've made for QEMU's purposes.
> We also have no consensus on where to keep the one time built iso that=
> we can download for this test you are proposing.

How big is the eventual ISO? If it's small we could just enable some CI=
steps and serve the ISOs directly as tagged build artefacts from GitLab.
> So let's figure out the above first. Programmatically downloading = an
> iso and running tests within a VM would be a much simpler test than > the one I wrote. We can add a subtest or a brand new test anytime if > we can figure out the above logistics.
>
>>
>> > > It makes sense to try things incrementally once we have = something going.
>> > >
>> > > Lets discuss this on a separate thread.
>> > >
>> > >> All this does nothing at all to help developers avoi= d
>> > >> bugs and when they do trigger debug the issue. Which= is
>> > >> after all why we have testing.
>> > >> Yes once in a very long while we are going to tweak<= br> >> > >> something in the tests, and for that rare occurence<= br> >> > >> it makes sense to periodically rebuild everything, >> > >> otherwise code bitrots.
>> > >>
>> > >> But the test is supposed to run within a VM anyway, = let's
>> > >> have an image and be done with it.
>> > >>
>> > >> --
>> > >> MST
>> > >>
>> >
>> >
>> > --
>> > Alex Benn=C3=A9e
>>


--
Alex Benn=C3=A9e
--000000000000d9681705eb8a7eb4--