[PATCH] util-linux: use the system binaries when running ptest

[PATCH] util-linux: use the system binaries when running ptest

[Ross Burton]

Unless told otherwise, the util-linux test suite assumes that it is
running in a build tree and looks for binaries to run.

In ptest we're on the target, so it needs to run the system binaries
instead.

This reduces ~180 skipped test cases down to tens, and should fix weird
behaviour. There are some new failures that were previously skipped
which will be investigated shortly.

Also pass --parsable to reduce the amount of noise in the log ouput.

[ YOCTO #14244 ]

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta/recipes-core/util-linux/util-linux/run-ptest | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-core/util-linux/util-linux/run-ptest b/meta/recipes-core/util-linux/util-linux/run-ptest
index e135ee583bd..86f9fef24bd 100644
--- a/meta/recipes-core/util-linux/util-linux/run-ptest
+++ b/meta/recipes-core/util-linux/util-linux/run-ptest
@@ -29,7 +29,7 @@ res=0
 count=0
 for ts in $comps; 
 do 
-   $ts | sed -u '{        
+   $ts --use-system-commands --parsable | sed -u '{
       s/^\(.*\):\(.*\) \.\.\. OK$/PASS: \1:\2/                              
       s/^\(.*\):\(.*\) \.\.\. FAILED \(.*\)$/FAIL: \1:\2 \3/                
       s/^\(.*\):\(.*\) \.\.\. SKIPPED \(.*\)$/SKIP: \1:\2 \3/               
-- 
2.25.1

Re: [OE-core] [PATCH] util-linux: use the system binaries when running ptest

[Richard Purdie]

On Mon, 2021-06-28 at 14:01 +0100, Ross Burton wrote:
> Unless told otherwise, the util-linux test suite assumes that it is
> running in a build tree and looks for binaries to run.
> 
> In ptest we're on the target, so it needs to run the system binaries
> instead.
> 
> This reduces ~180 skipped test cases down to tens, and should fix weird
> behaviour. There are some new failures that were previously skipped
> which will be investigated shortly.
> 
> Also pass --parsable to reduce the amount of noise in the log ouput.
> 
> [ YOCTO #14244 ]
> 
> Signed-off-by: Ross Burton <ross.burton@arm.com>
> ---
>  meta/recipes-core/util-linux/util-linux/run-ptest | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/recipes-core/util-linux/util-linux/run-ptest b/meta/recipes-core/util-linux/util-linux/run-ptest
> index e135ee583bd..86f9fef24bd 100644
> --- a/meta/recipes-core/util-linux/util-linux/run-ptest
> +++ b/meta/recipes-core/util-linux/util-linux/run-ptest
> @@ -29,7 +29,7 @@ res=0
>  count=0
>  for ts in $comps; 
>  do 
> -   $ts | sed -u '{        
> +   $ts --use-system-commands --parsable | sed -u '{
>        s/^\(.*\):\(.*\) \.\.\. OK$/PASS: \1:\2/                              
>        s/^\(.*\):\(.*\) \.\.\. FAILED \(.*\)$/FAIL: \1:\2 \3/                
>        s/^\(.*\):\(.*\) \.\.\. SKIPPED \(.*\)$/SKIP: \1:\2 \3/               

This is a good fix, thanks. Makes me worry about regression identification.
FWIW testing showed this on x86 (arm still going):

https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2227

Looks like a build from Alex did have results for arm:
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1942/steps/12/logs/stdio

Cheers,

Richard

Re: [OE-core] [PATCH] util-linux: use the system binaries when running ptest

[Alexandre Belloni]

On 28/06/2021 23:51:13+0100, Richard Purdie wrote:
> On Mon, 2021-06-28 at 14:01 +0100, Ross Burton wrote:
> > Unless told otherwise, the util-linux test suite assumes that it is
> > running in a build tree and looks for binaries to run.
> > 
> > In ptest we're on the target, so it needs to run the system binaries
> > instead.
> > 
> > This reduces ~180 skipped test cases down to tens, and should fix weird
> > behaviour. There are some new failures that were previously skipped
> > which will be investigated shortly.
> > 
> > Also pass --parsable to reduce the amount of noise in the log ouput.
> > 
> > [ YOCTO #14244 ]
> > 
> > Signed-off-by: Ross Burton <ross.burton@arm.com>
> > ---
> >  meta/recipes-core/util-linux/util-linux/run-ptest | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/meta/recipes-core/util-linux/util-linux/run-ptest b/meta/recipes-core/util-linux/util-linux/run-ptest
> > index e135ee583bd..86f9fef24bd 100644
> > --- a/meta/recipes-core/util-linux/util-linux/run-ptest
> > +++ b/meta/recipes-core/util-linux/util-linux/run-ptest
> > @@ -29,7 +29,7 @@ res=0
> >  count=0
> >  for ts in $comps; 
> >  do 
> > -   $ts | sed -u '{        
> > +   $ts --use-system-commands --parsable | sed -u '{
> >        s/^\(.*\):\(.*\) \.\.\. OK$/PASS: \1:\2/                              
> >        s/^\(.*\):\(.*\) \.\.\. FAILED \(.*\)$/FAIL: \1:\2 \3/                
> >        s/^\(.*\):\(.*\) \.\.\. SKIPPED \(.*\)$/SKIP: \1:\2 \3/               
> 
> This is a good fix, thanks. Makes me worry about regression identification.
> FWIW testing showed this on x86 (arm still going):
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2227
> 

Indeed, this one has the same issue:
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2228/steps/12/logs/stdio

> Looks like a build from Alex did have results for arm:
> https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1942/steps/12/logs/stdio
> 

-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [PATCH] util-linux: use the system binaries when running ptest

[Ross Burton]

Will this be blocked on fixing those?

Ross

On Mon, 28 Jun 2021 at 23:51, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Mon, 2021-06-28 at 14:01 +0100, Ross Burton wrote:
> > Unless told otherwise, the util-linux test suite assumes that it is
> > running in a build tree and looks for binaries to run.
> >
> > In ptest we're on the target, so it needs to run the system binaries
> > instead.
> >
> > This reduces ~180 skipped test cases down to tens, and should fix weird
> > behaviour. There are some new failures that were previously skipped
> > which will be investigated shortly.
> >
> > Also pass --parsable to reduce the amount of noise in the log ouput.
> >
> > [ YOCTO #14244 ]
> >
> > Signed-off-by: Ross Burton <ross.burton@arm.com>
> > ---
> >  meta/recipes-core/util-linux/util-linux/run-ptest | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/meta/recipes-core/util-linux/util-linux/run-ptest b/meta/recipes-core/util-linux/util-linux/run-ptest
> > index e135ee583bd..86f9fef24bd 100644
> > --- a/meta/recipes-core/util-linux/util-linux/run-ptest
> > +++ b/meta/recipes-core/util-linux/util-linux/run-ptest
> > @@ -29,7 +29,7 @@ res=0
> >  count=0
> >  for ts in $comps;
> >  do
> > -   $ts | sed -u '{
> > +   $ts --use-system-commands --parsable | sed -u '{
> >        s/^\(.*\):\(.*\) \.\.\. OK$/PASS: \1:\2/
> >        s/^\(.*\):\(.*\) \.\.\. FAILED \(.*\)$/FAIL: \1:\2 \3/
> >        s/^\(.*\):\(.*\) \.\.\. SKIPPED \(.*\)$/SKIP: \1:\2 \3/
>
> This is a good fix, thanks. Makes me worry about regression identification.
> FWIW testing showed this on x86 (arm still going):
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2227
>
> Looks like a build from Alex did have results for arm:
> https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1942/steps/12/logs/stdio
>
> Cheers,
>
> Richard
>
>
>
>

Re: [OE-core] [PATCH] util-linux: use the system binaries when running ptest

[Richard Purdie]

On Tue, 2021-06-29 at 12:25 +0100, Ross Burton wrote:
> Will this be blocked on fixing those?

It means every ptest build would have to be triaged, so yes, it does really 
block on that unfortunately :(

Cheers,

Richard