Re: Rust Oe-Selftest implementation V15 Testing

Re: Rust Oe-Selftest implementation V15 Testing

[Richard Purdie]

On Mon, 2023-07-17 at 16:08 +0100, Richard Purdie wrote:
> On Mon, 2023-07-17 at 16:34 +0200, Alexandre Belloni wrote:
> > Hello,
> > 
> > I got some feedback from RP:
> > 
> > http://autobuilder.yocto.io/pub/non-release/20230716-18/testresults/testresult-report.txt
> > 
> > This shows warnings for duplicate tests and he also asks being where the
> > qemuarm64 went, both issues being probably related.
> 
> I had a look and you probably need to add a:
> 
> @OETestTag("toolchain-user")
> @OETestTag("runqemu")
> class RustSelfTestBase(RustSelfTestSystemEmulated):
>     def test_check(self):
>         self.test_rust()
> 
> section to the tests to allow the tests to run for the non-IA
> architectures that use usermode emulation for the other tests which
> rust can't.
> 
> That then just brings the question of why there are duplicate tests
> results being reported. This is where the result for an ID is being
> reported more than once. I haven't looked into why that might be
> happening.

I really do want to get this rust test suite issue resolved so I went
digging into the code to find out what is really going on.

Firstly, the duplicate test results. The issue is that you defined the
core class like this:

 class RustSelfTestSystemEmulated(OESelftestTestCase, OEPTestResultTestCase):
     def test_rust

and python unittest has a convention where anything starting "test_" is
a test.

This meant that the rust test ran unguarded in all the oe-selftest
targets on the autobuilder and not just in the toolchain-system
filtered section.

The easiest fix is to drop the RustSelfTestBase class and move the
toolchain-system decorator to RustSelfTestSystemEmulated. That will
resolve the duplicate test warnings and ensure things run where they
should. You could have worked out this issue by finding that there were
rust test results in oe-selftest-* testresults.json files, e.g. here:

http://autobuilder.yocto.io/pub/non-release/20230716-18/testresults/oe-selftest-centos/oeqa/testresults.json

Moving on, the test result names really don't look good with the 
"[ui]  " and similar prefixes in the results file. I've patched a tweak
in to drop that.

I also noticed that there were no skipped tests being reported in the
results. This was due to "SKIP" being used instead of "SKIPPED" which
resulttool looks for.

We need to also add the toolchain-user decorator to make sure that the
tests run for the "user" architectures since we don't have any user
mode rust test equivalent.

I've rolled all these changes into a patch on master-next:

https://git.yoctoproject.org/poky/commit/?h=master-next&id=46ab84785da15ac156ee0b4a693ce8bb5ccf8c22

which I'll put into testing.

Looking to the future, I have concerns about the ease of maintenance of
this huge patch to rust to disable failing tests. I'd propose we change
this to a hardcoded list of tests to ignore in the result parsing code
which will be easier to maintain in the future.

We may want to take that as a subsequent follow up patch.

Cheers,

Richard

Re: [OE-core] Rust Oe-Selftest implementation V15 Testing

[Alex Kiernan]

On Wed, Jul 19, 2023 at 11:49 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Mon, 2023-07-17 at 16:08 +0100, Richard Purdie wrote:
> > On Mon, 2023-07-17 at 16:34 +0200, Alexandre Belloni wrote:
> > > Hello,
> > >
> > > I got some feedback from RP:
> > >
> > > http://autobuilder.yocto.io/pub/non-release/20230716-18/testresults/testresult-report.txt
> > >
> > > This shows warnings for duplicate tests and he also asks being where the
> > > qemuarm64 went, both issues being probably related.
> >
> > I had a look and you probably need to add a:
> >
> > @OETestTag("toolchain-user")
> > @OETestTag("runqemu")
> > class RustSelfTestBase(RustSelfTestSystemEmulated):
> >     def test_check(self):
> >         self.test_rust()
> >
> > section to the tests to allow the tests to run for the non-IA
> > architectures that use usermode emulation for the other tests which
> > rust can't.
> >
> > That then just brings the question of why there are duplicate tests
> > results being reported. This is where the result for an ID is being
> > reported more than once. I haven't looked into why that might be
> > happening.
>
> I really do want to get this rust test suite issue resolved so I went
> digging into the code to find out what is really going on.
>
> Firstly, the duplicate test results. The issue is that you defined the
> core class like this:
>
>  class RustSelfTestSystemEmulated(OESelftestTestCase, OEPTestResultTestCase):
>      def test_rust
>
> and python unittest has a convention where anything starting "test_" is
> a test.
>
> This meant that the rust test ran unguarded in all the oe-selftest
> targets on the autobuilder and not just in the toolchain-system
> filtered section.
>
> The easiest fix is to drop the RustSelfTestBase class and move the
> toolchain-system decorator to RustSelfTestSystemEmulated. That will
> resolve the duplicate test warnings and ensure things run where they
> should. You could have worked out this issue by finding that there were
> rust test results in oe-selftest-* testresults.json files, e.g. here:
>
> http://autobuilder.yocto.io/pub/non-release/20230716-18/testresults/oe-selftest-centos/oeqa/testresults.json
>
> Moving on, the test result names really don't look good with the
> "[ui]  " and similar prefixes in the results file. I've patched a tweak
> in to drop that.
>
> I also noticed that there were no skipped tests being reported in the
> results. This was due to "SKIP" being used instead of "SKIPPED" which
> resulttool looks for.
>
> We need to also add the toolchain-user decorator to make sure that the
> tests run for the "user" architectures since we don't have any user
> mode rust test equivalent.
>
> I've rolled all these changes into a patch on master-next:
>
> https://git.yoctoproject.org/poky/commit/?h=master-next&id=46ab84785da15ac156ee0b4a693ce8bb5ccf8c22
>
> which I'll put into testing.
>
> Looking to the future, I have concerns about the ease of maintenance of
> this huge patch to rust to disable failing tests. I'd propose we change
> this to a hardcoded list of tests to ignore in the result parsing code
> which will be easier to maintain in the future.
>

That feels far more maintainable - the scale of the  patch really
concerned me. Also I guess if it's just a list, then running with
everything enabled on upgrade to see what passes/fails would be a
relatively easy option.


> We may want to take that as a subsequent follow up patch.
>
> Cheers,
>
> Richard
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#184576): https://lists.openembedded.org/g/openembedded-core/message/184576
> Mute This Topic: https://lists.openembedded.org/mt/100232853/3618097
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kiernan@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>


--
Alex Kiernan

Re: [OE-core] Rust Oe-Selftest implementation V15 Testing

[Kokkonda, Sundeep]

[-- Attachment #1: Type: text/plain, Size: 4833 bytes --]



________________________________
From: Alex Kiernan <alex.kiernan@gmail.com>
Sent: 19 July 2023 17:28
To: Richard Purdie <richard.purdie@linuxfoundation.org>
Cc: Alexandre Belloni <alexandre.belloni@bootlin.com>; Shinde, Yash <Yash.Shinde@windriver.com>; openembedded-core <openembedded-core@lists.openembedded.org>; MacLeod, Randy <Randy.MacLeod@windriver.com>; Kokkonda, Sundeep <Sundeep.Kokkonda@windriver.com>; Gowda, Naveen <Naveen.Gowda@windriver.com>
Subject: Re: [OE-core] Rust Oe-Selftest implementation V15 Testing

CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.

On Wed, Jul 19, 2023 at 11:49 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Mon, 2023-07-17 at 16:08 +0100, Richard Purdie wrote:
> > On Mon, 2023-07-17 at 16:34 +0200, Alexandre Belloni wrote:
> > > Hello,
> > >
> > > I got some feedback from RP:
> > >
> > > http://autobuilder.yocto.io/pub/non-release/20230716-18/testresults/testresult-report.txt
> > >
> > > This shows warnings for duplicate tests and he also asks being where the
> > > qemuarm64 went, both issues being probably related.
> >
> > I had a look and you probably need to add a:
> >
> > @OETestTag("toolchain-user")
> > @OETestTag("runqemu")
> > class RustSelfTestBase(RustSelfTestSystemEmulated):
> >     def test_check(self):
> >         self.test_rust()
> >
> > section to the tests to allow the tests to run for the non-IA
> > architectures that use usermode emulation for the other tests which
> > rust can't.
> >
> > That then just brings the question of why there are duplicate tests
> > results being reported. This is where the result for an ID is being
> > reported more than once. I haven't looked into why that might be
> > happening.
>
> I really do want to get this rust test suite issue resolved so I went
> digging into the code to find out what is really going on.
>
> Firstly, the duplicate test results. The issue is that you defined the
> core class like this:
>
>  class RustSelfTestSystemEmulated(OESelftestTestCase, OEPTestResultTestCase):
>      def test_rust
>
> and python unittest has a convention where anything starting "test_" is
> a test.
>
> This meant that the rust test ran unguarded in all the oe-selftest
> targets on the autobuilder and not just in the toolchain-system
> filtered section.
>
> The easiest fix is to drop the RustSelfTestBase class and move the
> toolchain-system decorator to RustSelfTestSystemEmulated. That will
> resolve the duplicate test warnings and ensure things run where they
> should. You could have worked out this issue by finding that there were
> rust test results in oe-selftest-* testresults.json files, e.g. here:
>
> http://autobuilder.yocto.io/pub/non-release/20230716-18/testresults/oe-selftest-centos/oeqa/testresults.json
>
> Moving on, the test result names really don't look good with the
> "[ui]  " and similar prefixes in the results file. I've patched a tweak
> in to drop that.
>
> I also noticed that there were no skipped tests being reported in the
> results. This was due to "SKIP" being used instead of "SKIPPED" which
> resulttool looks for.
>
> We need to also add the toolchain-user decorator to make sure that the
> tests run for the "user" architectures since we don't have any user
> mode rust test equivalent.
>
> I've rolled all these changes into a patch on master-next:
>
> https://git.yoctoproject.org/poky/commit/?h=master-next&id=46ab84785da15ac156ee0b4a693ce8bb5ccf8c22
>
> which I'll put into testing.
>
> Looking to the future, I have concerns about the ease of maintenance of
> this huge patch to rust to disable failing tests. I'd propose we change
> this to a hardcoded list of tests to ignore in the result parsing code
> which will be easier to maintain in the future.
>

That feels far more maintainable - the scale of the  patch really
concerned me. Also I guess if it's just a list, then running with
everything enabled on upgrade to see what passes/fails would be a
relatively easy option.

Hello Alex, Hello Richard,

During next rebase to rust 1.71.0 we will update this.



> We may want to take that as a subsequent follow up patch.
>
> Cheers,
>
> Richard
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#184576): https://lists.openembedded.org/g/openembedded-core/message/184576
> Mute This Topic: https://lists.openembedded.org/mt/100232853/3618097
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kiernan@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>


--
Alex Kiernan

[-- Attachment #2: Type: text/html, Size: 9041 bytes --]

Re: [OE-core] Rust Oe-Selftest implementation V15 Testing

[Richard Purdie]

On Wed, 2023-07-19 at 12:58 +0100, Alex Kiernan wrote:
> On Wed, Jul 19, 2023 at 11:49 AM Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > 
> > On Mon, 2023-07-17 at 16:08 +0100, Richard Purdie wrote:
> > > On Mon, 2023-07-17 at 16:34 +0200, Alexandre Belloni wrote:
> > > > Hello,
> > > > 
> > > > I got some feedback from RP:
> > > > 
> > > > http://autobuilder.yocto.io/pub/non-release/20230716-18/testresults/testresult-report.txt
> > > > 
> > > > This shows warnings for duplicate tests and he also asks being where the
> > > > qemuarm64 went, both issues being probably related.
> > > 
> > > I had a look and you probably need to add a:
> > > 
> > > @OETestTag("toolchain-user")
> > > @OETestTag("runqemu")
> > > class RustSelfTestBase(RustSelfTestSystemEmulated):
> > >     def test_check(self):
> > >         self.test_rust()
> > > 
> > > section to the tests to allow the tests to run for the non-IA
> > > architectures that use usermode emulation for the other tests which
> > > rust can't.
> > > 
> > > That then just brings the question of why there are duplicate tests
> > > results being reported. This is where the result for an ID is being
> > > reported more than once. I haven't looked into why that might be
> > > happening.
> > 
> > I really do want to get this rust test suite issue resolved so I went
> > digging into the code to find out what is really going on.
> > 
> > Firstly, the duplicate test results. The issue is that you defined the
> > core class like this:
> > 
> >  class RustSelfTestSystemEmulated(OESelftestTestCase, OEPTestResultTestCase):
> >      def test_rust
> > 
> > and python unittest has a convention where anything starting "test_" is
> > a test.
> > 
> > This meant that the rust test ran unguarded in all the oe-selftest
> > targets on the autobuilder and not just in the toolchain-system
> > filtered section.
> > 
> > The easiest fix is to drop the RustSelfTestBase class and move the
> > toolchain-system decorator to RustSelfTestSystemEmulated. That will
> > resolve the duplicate test warnings and ensure things run where they
> > should. You could have worked out this issue by finding that there were
> > rust test results in oe-selftest-* testresults.json files, e.g. here:
> > 
> > http://autobuilder.yocto.io/pub/non-release/20230716-18/testresults/oe-selftest-centos/oeqa/testresults.json
> > 
> > Moving on, the test result names really don't look good with the
> > "[ui]  " and similar prefixes in the results file. I've patched a tweak
> > in to drop that.
> > 
> > I also noticed that there were no skipped tests being reported in the
> > results. This was due to "SKIP" being used instead of "SKIPPED" which
> > resulttool looks for.
> > 
> > We need to also add the toolchain-user decorator to make sure that the
> > tests run for the "user" architectures since we don't have any user
> > mode rust test equivalent.
> > 
> > I've rolled all these changes into a patch on master-next:
> > 
> > https://git.yoctoproject.org/poky/commit/?h=master-next&id=46ab84785da15ac156ee0b4a693ce8bb5ccf8c22
> > 
> > which I'll put into testing.
> > 
> > Looking to the future, I have concerns about the ease of maintenance of
> > this huge patch to rust to disable failing tests. I'd propose we change
> > this to a hardcoded list of tests to ignore in the result parsing code
> > which will be easier to maintain in the future.
> > 
> 
> That feels far more maintainable - the scale of the  patch really
> concerned me. Also I guess if it's just a list, then running with
> everything enabled on upgrade to see what passes/fails would be a
> relatively easy option.

I've gone ahead and merged the rust selftest changes since we had a
successful test run on the autobuilder and I think that patch series
has gone on long enough. I pushed my tweaks in a follow up commit.

I'd happily take a patch changing the current patch for the approach
mentioned above as a next step. I appreciate it will mean we need to
tweak the 1.71.0 update but felt getting the tests in was probably the
next logical step.

Cheers,

Richard