[PATCH 1/2] qemux86/conf: TEST ONLY. Drop smp boot

[PATCH 1/2] qemux86/conf: TEST ONLY. Drop smp boot

[bruce.ashfield]

From: Bruce Ashfield <bruce.ashfield@gmail.com>

We are seeing intermittent boot hangs with x86. Once of the last
messages in failing logs is secondary cpu startup.

Dropping us down to smp for testing to see if the hangs start to
go away with smp removed.

Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
---
 meta/conf/machine/include/x86/qemuboot-x86.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/conf/machine/include/x86/qemuboot-x86.inc b/meta/conf/machine/include/x86/qemuboot-x86.inc
index 6ae03633ae..54a12119a7 100644
--- a/meta/conf/machine/include/x86/qemuboot-x86.inc
+++ b/meta/conf/machine/include/x86/qemuboot-x86.inc
@@ -1,6 +1,6 @@
 # For runqemu
 IMAGE_CLASSES += "qemuboot"
-QB_SMP ?= "-smp 4"
+QB_SMP ?= ""
 QB_CPU:x86 ?= "-cpu IvyBridge -machine q35,i8042=off"
 QB_CPU_KVM:x86 ?= "-cpu IvyBridge -machine q35,i8042=off"
 
-- 
2.39.2

[PATCH 2/2] qemux86*: TEST only set preferred provider to linux-yocto-dev

[bruce.ashfield]

From: Bruce Ashfield <bruce.ashfield@gmail.com>

We are seeing intermittent boot hangs on x86. To try and determine
if this is a 6.1-stable only issue, set the prferred provider for
qemux86 to linux-yocto-dev.

Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
---
 meta/conf/machine/include/qemu.inc | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/conf/machine/include/qemu.inc b/meta/conf/machine/include/qemu.inc
index 0d71bcbbad..121e8474c1 100644
--- a/meta/conf/machine/include/qemu.inc
+++ b/meta/conf/machine/include/qemu.inc
@@ -21,6 +21,8 @@ RDEPENDS:${KERNEL_PACKAGE_NAME}-base = ""
 
 # Use a common kernel recipe for all QEMU machines
 PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto"
+PREFERRED_PROVIDER_virtual/kernel:qemux86 = "linux-yocto-dev"
+PREFERRED_PROVIDER_virtual/kernel:qemux86-64 = "linux-yocto-dev"
 
 EXTRA_IMAGEDEPENDS += "qemu-system-native qemu-helper-native:do_addto_recipe_sysroot"
 
-- 
2.39.2

Re: [PATCH 2/2] qemux86*: TEST only set preferred provider to linux-yocto-dev

[Richard Purdie]

On Thu, 2023-06-08 at 19:39 +0000, bruce.ashfield@gmail.com wrote:
> From: Bruce Ashfield <bruce.ashfield@gmail.com>
> 
> We are seeing intermittent boot hangs on x86. To try and determine
> if this is a 6.1-stable only issue, set the prferred provider for
> qemux86 to linux-yocto-dev.
> 
> Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
> ---
>  meta/conf/machine/include/qemu.inc | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/meta/conf/machine/include/qemu.inc b/meta/conf/machine/include/qemu.inc
> index 0d71bcbbad..121e8474c1 100644
> --- a/meta/conf/machine/include/qemu.inc
> +++ b/meta/conf/machine/include/qemu.inc
> @@ -21,6 +21,8 @@ RDEPENDS:${KERNEL_PACKAGE_NAME}-base = ""
>  
>  # Use a common kernel recipe for all QEMU machines
>  PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto"
> +PREFERRED_PROVIDER_virtual/kernel:qemux86 = "linux-yocto-dev"
> +PREFERRED_PROVIDER_virtual/kernel:qemux86-64 = "linux-yocto-dev"
>  
>  EXTRA_IMAGEDEPENDS += "qemu-system-native qemu-helper-native:do_addto_recipe_sysroot"
>  

Unfortunately this has other issues:

https://autobuilder.yoctoproject.org/typhoon/#/builders/45/builds/7276/steps/13/logs/stdio

Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  [Previous line repeated 2 more times]
  File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/runtime/cases/kernelmodule.py", line 48, in test_kernel_module
    self.assertEqual(status, 0, msg='\n'.join([cmd, output]))
AssertionError: 1 != 0 : cd /tmp && insmod hellomod.ko
insmod: ERROR: could not insert module hellomod.ko: Invalid module format

But also:

https://autobuilder.yoctoproject.org/typhoon/#/builders/109/builds/6105/steps/13/logs/stdio

which shows ping failures, as if the networking isn't working. I
checked the boot logs and the kernel is booting but for some reason
networking doesn't become live and ping fails.

I can see messages saying the "IP-Config: Complete:" with appropriate
network config in the kernel boot log so it is all the more puzzling:

https://autobuilder.yocto.io/pub/failed-builds-data/qemu_boot_log.20230609064146

The ping failures repeated in many other targets in that build.

Cheers,

Richard

Re: [OE-core] [PATCH 2/2] qemux86*: TEST only set preferred provider to linux-yocto-dev

[Richard Purdie]

On Fri, 2023-06-09 at 08:17 +0100, Richard Purdie via
lists.openembedded.org wrote:
> On Thu, 2023-06-08 at 19:39 +0000, bruce.ashfield@gmail.com wrote:
> > From: Bruce Ashfield <bruce.ashfield@gmail.com>
> > 
> > We are seeing intermittent boot hangs on x86. To try and determine
> > if this is a 6.1-stable only issue, set the prferred provider for
> > qemux86 to linux-yocto-dev.
> > 
> > Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
> > ---
> >  meta/conf/machine/include/qemu.inc | 2 ++
> >  1 file changed, 2 insertions(+)
> > 
> > diff --git a/meta/conf/machine/include/qemu.inc b/meta/conf/machine/include/qemu.inc
> > index 0d71bcbbad..121e8474c1 100644
> > --- a/meta/conf/machine/include/qemu.inc
> > +++ b/meta/conf/machine/include/qemu.inc
> > @@ -21,6 +21,8 @@ RDEPENDS:${KERNEL_PACKAGE_NAME}-base = ""
> >  
> >  # Use a common kernel recipe for all QEMU machines
> >  PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto"
> > +PREFERRED_PROVIDER_virtual/kernel:qemux86 = "linux-yocto-dev"
> > +PREFERRED_PROVIDER_virtual/kernel:qemux86-64 = "linux-yocto-dev"
> >  
> >  EXTRA_IMAGEDEPENDS += "qemu-system-native qemu-helper-native:do_addto_recipe_sysroot"
> >  
> 
> Unfortunately this has other issues:
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/45/builds/7276/steps/13/logs/stdio
> 
> Traceback (most recent call last):
>   File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
>     return func(*args, **kwargs)
>            ^^^^^^^^^^^^^^^^^^^^^
>   File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
>     return func(*args, **kwargs)
>            ^^^^^^^^^^^^^^^^^^^^^
>   File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
>     return func(*args, **kwargs)
>            ^^^^^^^^^^^^^^^^^^^^^
>   [Previous line repeated 2 more times]
>   File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/runtime/cases/kernelmodule.py", line 48, in test_kernel_module
>     self.assertEqual(status, 0, msg='\n'.join([cmd, output]))
> AssertionError: 1 != 0 : cd /tmp && insmod hellomod.ko
> insmod: ERROR: could not insert module hellomod.ko: Invalid module format
> 
> But also:
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/109/builds/6105/steps/13/logs/stdio
> 
> which shows ping failures, as if the networking isn't working. I
> checked the boot logs and the kernel is booting but for some reason
> networking doesn't become live and ping fails.
> 
> I can see messages saying the "IP-Config: Complete:" with appropriate
> network config in the kernel boot log so it is all the more puzzling:
> 
> https://autobuilder.yocto.io/pub/failed-builds-data/qemu_boot_log.20230609064146
> 
> The ping failures repeated in many other targets in that build.

There is also a perf buildpath warning:

https://autobuilder.yoctoproject.org/typhoon/#/builders/109/builds/6105/steps/12/logs/warnings

which we'll likely have to fix at some point.

Cheers,

Richard

Re: [OE-core] [PATCH 2/2] qemux86*: TEST only set preferred provider to linux-yocto-dev

[Bruce Ashfield]

On Fri, Jun 9, 2023 at 3:19 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Fri, 2023-06-09 at 08:17 +0100, Richard Purdie via
> lists.openembedded.org wrote:
> > On Thu, 2023-06-08 at 19:39 +0000, bruce.ashfield@gmail.com wrote:
> > > From: Bruce Ashfield <bruce.ashfield@gmail.com>
> > >
> > > We are seeing intermittent boot hangs on x86. To try and determine
> > > if this is a 6.1-stable only issue, set the prferred provider for
> > > qemux86 to linux-yocto-dev.
> > >
> > > Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
> > > ---
> > >  meta/conf/machine/include/qemu.inc | 2 ++
> > >  1 file changed, 2 insertions(+)
> > >
> > > diff --git a/meta/conf/machine/include/qemu.inc b/meta/conf/machine/include/qemu.inc
> > > index 0d71bcbbad..121e8474c1 100644
> > > --- a/meta/conf/machine/include/qemu.inc
> > > +++ b/meta/conf/machine/include/qemu.inc
> > > @@ -21,6 +21,8 @@ RDEPENDS:${KERNEL_PACKAGE_NAME}-base = ""
> > >
> > >  # Use a common kernel recipe for all QEMU machines
> > >  PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto"
> > > +PREFERRED_PROVIDER_virtual/kernel:qemux86 = "linux-yocto-dev"
> > > +PREFERRED_PROVIDER_virtual/kernel:qemux86-64 = "linux-yocto-dev"
> > >
> > >  EXTRA_IMAGEDEPENDS += "qemu-system-native qemu-helper-native:do_addto_recipe_sysroot"
> > >
> >
> > Unfortunately this has other issues:
> >
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/45/builds/7276/steps/13/logs/stdio
> >
> > Traceback (most recent call last):
> >   File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
> >     return func(*args, **kwargs)
> >            ^^^^^^^^^^^^^^^^^^^^^
> >   File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
> >     return func(*args, **kwargs)
> >            ^^^^^^^^^^^^^^^^^^^^^
> >   File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
> >     return func(*args, **kwargs)
> >            ^^^^^^^^^^^^^^^^^^^^^
> >   [Previous line repeated 2 more times]
> >   File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/runtime/cases/kernelmodule.py", line 48, in test_kernel_module
> >     self.assertEqual(status, 0, msg='\n'.join([cmd, output]))
> > AssertionError: 1 != 0 : cd /tmp && insmod hellomod.ko
> > insmod: ERROR: could not insert module hellomod.ko: Invalid module format
> >

I'll fix this up as part of my -dev kernel work to get it ready as a
reference kernel.

> > But also:
> >
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/109/builds/6105/steps/13/logs/stdio
> >
> > which shows ping failures, as if the networking isn't working. I
> > checked the boot logs and the kernel is booting but for some reason
> > networking doesn't become live and ping fails.
> >
> > I can see messages saying the "IP-Config: Complete:" with appropriate
> > network config in the kernel boot log so it is all the more puzzling:
> >
> > https://autobuilder.yocto.io/pub/failed-builds-data/qemu_boot_log.20230609064146
> >
> > The ping failures repeated in many other targets in that build.

This is a strange one. Not seeing that here, but I'll see if some sort
of networking configuration needs tweaking.

>
> There is also a perf buildpath warning:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/109/builds/6105/steps/12/logs/warnings
>
> which we'll likely have to fix at some point.

I'll fix that right away.

Bruce

>
> Cheers,
>
> Richard



-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [PATCH 1/2] qemux86/conf: TEST ONLY. Drop smp boot

[Richard Purdie]

On Thu, 2023-06-08 at 19:39 +0000, bruce.ashfield@gmail.com wrote:
> From: Bruce Ashfield <bruce.ashfield@gmail.com>
> 
> We are seeing intermittent boot hangs with x86. Once of the last
> messages in failing logs is secondary cpu startup.
> 
> Dropping us down to smp for testing to see if the hangs start to
> go away with smp removed.
> 
> Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
> ---
>  meta/conf/machine/include/x86/qemuboot-x86.inc | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/conf/machine/include/x86/qemuboot-x86.inc b/meta/conf/machine/include/x86/qemuboot-x86.inc
> index 6ae03633ae..54a12119a7 100644
> --- a/meta/conf/machine/include/x86/qemuboot-x86.inc
> +++ b/meta/conf/machine/include/x86/qemuboot-x86.inc
> @@ -1,6 +1,6 @@
>  # For runqemu
>  IMAGE_CLASSES += "qemuboot"
> -QB_SMP ?= "-smp 4"
> +QB_SMP ?= ""
>  QB_CPU:x86 ?= "-cpu IvyBridge -machine q35,i8042=off"
>  QB_CPU_KVM:x86 ?= "-cpu IvyBridge -machine q35,i8042=off"
> 
> 

FWIW, this patch didn't change much. With it applied we still see:

https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/5168/steps/12/logs/stdio

[    0.634735] Freeing SMP alternatives memory: 52K
[   12.275510] smpboot: CPU0: Intel Xeon E3-12xx v2 (Ivy Bridge) (family: 0x6, model: 0x3a, stepping: 0x9)
[274491.955544] cblist_init_generic: Setting adjustable number of callback queues.
[274492.008510] cblist_init_generic: Setting shift to 0 and lim to 1.
[279820.791509] cblist_init_generic: Setting shift to 0 and lim to 1.
[282104.735510] cblist_init_generic: Setting shift to 0 and lim to 1.
[386831.154510] Performance Events: unsupported p6 CPU model 58 no PMU driver, software events only.
[386850.632510] rcu: Hierarchical SRCU implementation.
[386850.668510] rcu: 	Max phase no-delay instances is 400.
[386852.898509] smp: Bringing up secondary CPUs ...
[386852.946510] smp: Brought up 1 node, 1 CPU
[386852.992510] smpboot: Max logical packages: 1
[386853.018509] smpboot: Total of 1 processors activated (5199.99 BogoMIPS)
[386855.257509] devtmpfs: initialized
[386857.982509] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[386858.041510] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[386858.691509] pinctrl core: initialized pinctrl subsystem
[386862.836510] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[386864.720510] thermal_sys: Registered thermal governor 'step_wise'
[386864.725509] thermal_sys: Registered thermal governor 'user_space'
[386864.864510] cpuidle: using governor menu
[386865.662509] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xb0000000-0xbfffffff] (base 0xb0000000)
[386865.693510] PCI: MMCONFIG at [mem 0xb0000000-0xbfffffff] reserved in E820
[386865.900509] PCI: Using configuration type 1 for base access
[386878.550509] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.

Cheers,

Richard

Re: [PATCH 1/2] qemux86/conf: TEST ONLY. Drop smp boot

[Bruce Ashfield]

On Fri, Jun 9, 2023 at 2:06 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Thu, 2023-06-08 at 19:39 +0000, bruce.ashfield@gmail.com wrote:
> > From: Bruce Ashfield <bruce.ashfield@gmail.com>
> >
> > We are seeing intermittent boot hangs with x86. Once of the last
> > messages in failing logs is secondary cpu startup.
> >
> > Dropping us down to smp for testing to see if the hangs start to
> > go away with smp removed.
> >
> > Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
> > ---
> >  meta/conf/machine/include/x86/qemuboot-x86.inc | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/meta/conf/machine/include/x86/qemuboot-x86.inc b/meta/conf/machine/include/x86/qemuboot-x86.inc
> > index 6ae03633ae..54a12119a7 100644
> > --- a/meta/conf/machine/include/x86/qemuboot-x86.inc
> > +++ b/meta/conf/machine/include/x86/qemuboot-x86.inc
> > @@ -1,6 +1,6 @@
> >  # For runqemu
> >  IMAGE_CLASSES += "qemuboot"
> > -QB_SMP ?= "-smp 4"
> > +QB_SMP ?= ""
> >  QB_CPU:x86 ?= "-cpu IvyBridge -machine q35,i8042=off"
> >  QB_CPU_KVM:x86 ?= "-cpu IvyBridge -machine q35,i8042=off"
> >
> >
>
> FWIW, this patch didn't change much. With it applied we still see:


ok. so at least we can rule out it being specific to secondary cpu startup.

Bruce


>
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/5168/steps/12/logs/stdio
>
> [    0.634735] Freeing SMP alternatives memory: 52K
> [   12.275510] smpboot: CPU0: Intel Xeon E3-12xx v2 (Ivy Bridge) (family: 0x6, model: 0x3a, stepping: 0x9)
> [274491.955544] cblist_init_generic: Setting adjustable number of callback queues.
> [274492.008510] cblist_init_generic: Setting shift to 0 and lim to 1.
> [279820.791509] cblist_init_generic: Setting shift to 0 and lim to 1.
> [282104.735510] cblist_init_generic: Setting shift to 0 and lim to 1.
> [386831.154510] Performance Events: unsupported p6 CPU model 58 no PMU driver, software events only.
> [386850.632510] rcu: Hierarchical SRCU implementation.
> [386850.668510] rcu:    Max phase no-delay instances is 400.
> [386852.898509] smp: Bringing up secondary CPUs ...
> [386852.946510] smp: Brought up 1 node, 1 CPU
> [386852.992510] smpboot: Max logical packages: 1
> [386853.018509] smpboot: Total of 1 processors activated (5199.99 BogoMIPS)
> [386855.257509] devtmpfs: initialized
> [386857.982509] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
> [386858.041510] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
> [386858.691509] pinctrl core: initialized pinctrl subsystem
> [386862.836510] NET: Registered PF_NETLINK/PF_ROUTE protocol family
> [386864.720510] thermal_sys: Registered thermal governor 'step_wise'
> [386864.725509] thermal_sys: Registered thermal governor 'user_space'
> [386864.864510] cpuidle: using governor menu
> [386865.662509] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xb0000000-0xbfffffff] (base 0xb0000000)
> [386865.693510] PCI: MMCONFIG at [mem 0xb0000000-0xbfffffff] reserved in E820
> [386865.900509] PCI: Using configuration type 1 for base access
> [386878.550509] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
>
> Cheers,
>
> Richard
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [PATCH 1/2] qemux86/conf: TEST ONLY. Drop smp boot

[Paul Gortmaker]

[Re: [PATCH 1/2] qemux86/conf: TEST ONLY. Drop smp boot] On 09/06/2023 (Fri 10:08) Bruce Ashfield wrote:

> On Fri, Jun 9, 2023 at 2:06???AM Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> >

[...]

> > FWIW, this patch didn't change much. With it applied we still see:
> 
> 
> ok. so at least we can rule out it being specific to secondary cpu startup.

I'll own some of that.  I was looking forward in the dmesg where
secondary CPU init was, and ignoring looking backwards.

I have a new thought.  I still could be completely off base.  I updated
the case with my theory.

https://bugzilla.yoctoproject.org/show_bug.cgi?id=15138

Might totally be another wild goose chase.  Who knows...

Paul.
--

Re: [PATCH 1/2] qemux86/conf: TEST ONLY. Drop smp boot

[Richard Purdie]

On Sat, 2023-06-10 at 22:08 -0400, Paul Gortmaker wrote:
> [Re: [PATCH 1/2] qemux86/conf: TEST ONLY. Drop smp boot] On 09/06/2023 (Fri 10:08) Bruce Ashfield wrote:
> 
> > On Fri, Jun 9, 2023 at 2:06???AM Richard Purdie
> > <richard.purdie@linuxfoundation.org> wrote:
> > > 
> 
> [...]
> 
> > > FWIW, this patch didn't change much. With it applied we still see:
> > 
> > 
> > ok. so at least we can rule out it being specific to secondary cpu startup.
> 
> I'll own some of that.  I was looking forward in the dmesg where
> secondary CPU init was, and ignoring looking backwards.
> 
> I have a new thought.  I still could be completely off base.  I updated
> the case with my theory.
> 
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=15138
> 
> Might totally be another wild goose chase.  Who knows...

I was leaning toward leaving these on as it tests things as most users
would end up using them. Whether this is the right thing to do, not
sure.

As an extra data point, we have this failure with 6.1.28:

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5262/steps/14/logs/stdio

which means the issue is probably between 6.1.26 and 6.1.28. I'm
testing with 6.1.27 and haven't seen it there yet.

Cheers,

Richard