Fix for the APIC hangs in qemux86-64

Fix for the APIC hangs in qemux86-64

[Richard Purdie]

Hi Bruce,

https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301

We've root caused the hangs we're seeing in qemux86-64 on our
autobuilder workers. We've identified a commit on 4.15-rc3 which fixes
the issue but backporting it to the autobuilder's host kernels will be
a pain.

As a workaround I'd like to enable x2apic on qemux86-64. Since that
works via MSRs it avoids the problem. 

Would it be a big issue to set CONFIG_X86_X2APIC=y for qemux86-64?

Cheers,

Richard

Re: Fix for the APIC hangs in qemux86-64

[Bruce Ashfield]

On 12/13/2017 07:34 AM, Richard Purdie wrote:
> Hi Bruce,
> 
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301
> 
> We've root caused the hangs we're seeing in qemux86-64 on our
> autobuilder workers. We've identified a commit on 4.15-rc3 which fixes
> the issue but backporting it to the autobuilder's host kernels will be
> a pain.
> 
> As a workaround I'd like to enable x2apic on qemux86-64. Since that
> works via MSRs it avoids the problem.
> 
> Would it be a big issue to set CONFIG_X86_X2APIC=y for qemux86-64?

Nope. No big deal at all. I've used the same config i the past
when doing work with rt-KVM, so I know it well.

I'll prep a commit, and send it along as part of a rebased set
of kernel patches.

Cheers,

Bruce

> 
> Cheers,
> 
> Richard
> 
> 
> 

Re: Fix for the APIC hangs in qemux86-64

[Richard Purdie]

On Wed, 2017-12-13 at 08:38 -0500, Bruce Ashfield wrote:
> On 12/13/2017 07:34 AM, Richard Purdie wrote:
> > 
> > Hi Bruce,
> > 
> > https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301
> > 
> > We've root caused the hangs we're seeing in qemux86-64 on our
> > autobuilder workers. We've identified a commit on 4.15-rc3 which
> > fixes
> > the issue but backporting it to the autobuilder's host kernels will
> > be
> > a pain.
> > 
> > As a workaround I'd like to enable x2apic on qemux86-64. Since that
> > works via MSRs it avoids the problem.
> > 
> > Would it be a big issue to set CONFIG_X86_X2APIC=y for qemux86-64?
> Nope. No big deal at all. I've used the same config i the past
> when doing work with rt-KVM, so I know it well.
> 
> I'll prep a commit, and send it along as part of a rebased set
> of kernel patches.

Thanks, much appreciated.

Did we get to the bottom of the 4.12 selftest failures?

We may also need to backport the defconfig change back to the older
releases so we don't get the hangs in those builds either...

Cheers,

Richard

Re: Fix for the APIC hangs in qemux86-64

[Bruce Ashfield]

On 12/13/2017 09:05 AM, Richard Purdie wrote:
> On Wed, 2017-12-13 at 08:38 -0500, Bruce Ashfield wrote:
>> On 12/13/2017 07:34 AM, Richard Purdie wrote:
>>>
>>> Hi Bruce,
>>>
>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301
>>>
>>> We've root caused the hangs we're seeing in qemux86-64 on our
>>> autobuilder workers. We've identified a commit on 4.15-rc3 which
>>> fixes
>>> the issue but backporting it to the autobuilder's host kernels will
>>> be
>>> a pain.
>>>
>>> As a workaround I'd like to enable x2apic on qemux86-64. Since that
>>> works via MSRs it avoids the problem.
>>>
>>> Would it be a big issue to set CONFIG_X86_X2APIC=y for qemux86-64?
>> Nope. No big deal at all. I've used the same config i the past
>> when doing work with rt-KVM, so I know it well.
>>
>> I'll prep a commit, and send it along as part of a rebased set
>> of kernel patches.
> 
> Thanks, much appreciated.
> 
> Did we get to the bottom of the 4.12 selftest failures?
> 

Not yet no. I was distracted for a day, but need to learn how
to run them myself. I was able to log in via serial in my normal
boot flow, so that didn't get me anywhere.

Is the selftest HOWO in the yocto docs ? I'll google it up this
morning.

> We may also need to backport the defconfig change back to the older
> releases so we don't get the hangs in those builds either...

Not a problem either. I'll prepare several.

Bruce

> 
> Cheers,
> 
> Richard
> 

Re: Fix for the APIC hangs in qemux86-64

[Richard Purdie]

On Wed, 2017-12-13 at 09:07 -0500, Bruce Ashfield wrote:
> On 12/13/2017 09:05 AM, Richard Purdie wrote:
> > 
> > On Wed, 2017-12-13 at 08:38 -0500, Bruce Ashfield wrote:
> > > 
> > > On 12/13/2017 07:34 AM, Richard Purdie wrote:
> > > > 
> > > > 
> > > > Hi Bruce,
> > > > 
> > > > https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301
> > > > 
> > > > We've root caused the hangs we're seeing in qemux86-64 on our
> > > > autobuilder workers. We've identified a commit on 4.15-rc3
> > > > which
> > > > fixes
> > > > the issue but backporting it to the autobuilder's host kernels
> > > > will
> > > > be
> > > > a pain.
> > > > 
> > > > As a workaround I'd like to enable x2apic on qemux86-64. Since
> > > > that
> > > > works via MSRs it avoids the problem.
> > > > 
> > > > Would it be a big issue to set CONFIG_X86_X2APIC=y for qemux86-
> > > > 64?
> > > Nope. No big deal at all. I've used the same config i the past
> > > when doing work with rt-KVM, so I know it well.
> > > 
> > > I'll prep a commit, and send it along as part of a rebased set
> > > of kernel patches.
> > Thanks, much appreciated.
> > 
> > Did we get to the bottom of the 4.12 selftest failures?
> > 
> Not yet no. I was distracted for a day, but need to learn how
> to run them myself. I was able to log in via serial in my normal
> boot flow, so that didn't get me anywhere.
> 
> Is the selftest HOWO in the yocto docs ? I'll google it up this
> morning.

No HOWTO but I did give you the command. In a poky checkout run:

oe-selftest -r runqemu.RunqemuTests.test_boot_deploy

which runs a single test that fails (setup DL_DIR/SSTATE_MIRRORS first
to speed it up a bit).

It may tell you to clear SANITY_TESTED_DISTROS, I think it auto adds
meta-selftest now. You may also need to set DISPLAY to something (e.g.
a dummy vncserver).

Not sure what is different about the serial qemu is using verses your
own tests...

> > We may also need to backport the defconfig change back to the older
> > releases so we don't get the hangs in those builds either...
> Not a problem either. I'll prepare several.> 

Thanks!

Cheers,

Richard
> 

Re: Fix for the APIC hangs in qemux86-64

[Bruce Ashfield]

On 12/13/2017 09:14 AM, Richard Purdie wrote:
> On Wed, 2017-12-13 at 09:07 -0500, Bruce Ashfield wrote:
>> On 12/13/2017 09:05 AM, Richard Purdie wrote:
>>>
>>> On Wed, 2017-12-13 at 08:38 -0500, Bruce Ashfield wrote:
>>>>
>>>> On 12/13/2017 07:34 AM, Richard Purdie wrote:
>>>>>
>>>>>
>>>>> Hi Bruce,
>>>>>
>>>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301
>>>>>
>>>>> We've root caused the hangs we're seeing in qemux86-64 on our
>>>>> autobuilder workers. We've identified a commit on 4.15-rc3
>>>>> which
>>>>> fixes
>>>>> the issue but backporting it to the autobuilder's host kernels
>>>>> will
>>>>> be
>>>>> a pain.
>>>>>
>>>>> As a workaround I'd like to enable x2apic on qemux86-64. Since
>>>>> that
>>>>> works via MSRs it avoids the problem.
>>>>>
>>>>> Would it be a big issue to set CONFIG_X86_X2APIC=y for qemux86-
>>>>> 64?
>>>> Nope. No big deal at all. I've used the same config i the past
>>>> when doing work with rt-KVM, so I know it well.
>>>>
>>>> I'll prep a commit, and send it along as part of a rebased set
>>>> of kernel patches.
>>> Thanks, much appreciated.
>>>
>>> Did we get to the bottom of the 4.12 selftest failures?
>>>
>> Not yet no. I was distracted for a day, but need to learn how
>> to run them myself. I was able to log in via serial in my normal
>> boot flow, so that didn't get me anywhere.
>>
>> Is the selftest HOWO in the yocto docs ? I'll google it up this
>> morning.
> 
> No HOWTO but I did give you the command. In a poky checkout run:
> 
> oe-selftest -r runqemu.RunqemuTests.test_boot_deploy

Yah, that's where it went off the rails for me, so I figure that
I'm doing something stupid:

yow-bashfiel-d4 [/home/bruc...poky/build]> oe-selftest -r 
runqemu.RunqemuTests.test_boot_deploy
Traceback (most recent call last):
   File "/home/bruce/poky/scripts/oe-selftest", line 43, in <module>
     from oeqa.utils import load_test_components
ImportError: bad magic number in 'oeqa': b'\x03\xf3\r\n'

> 
> which runs a single test that fails (setup DL_DIR/SSTATE_MIRRORS first
> to speed it up a bit).
> 
> It may tell you to clear SANITY_TESTED_DISTROS, I think it auto adds
> meta-selftest now. You may also need to set DISPLAY to something (e.g.
> a dummy vncserver).

I'll keep smashing on it.

Bruce

> 
> Not sure what is different about the serial qemu is using verses your
> own tests...
> 
>>   > We may also need to backport the defconfig change back to the older
>>   > releases so we don't get the hangs in those builds either...
>>   Not a problem either. I'll prepare several.>
> 
> Thanks!
> 
> Cheers,
> 
> Richard
>>

Re: Fix for the APIC hangs in qemux86-64

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 608 bytes --]

On 13 December 2017 at 14:17, Bruce Ashfield <bruce.ashfield@windriver.com>
wrote:

> yow-bashfiel-d4 [/home/bruc...poky/build]> oe-selftest -r
> runqemu.RunqemuTests.test_boot_deploy
> Traceback (most recent call last):
>   File "/home/bruce/poky/scripts/oe-selftest", line 43, in <module>
>     from oeqa.utils import load_test_components
> ImportError: bad magic number in 'oeqa': b'\x03\xf3\r\n'


You've been switching between master and a pre-py3 branch, and Python is
stupid so the py3 binary is trying to run py2 compiled binaries.

find poky -name \*.pyc -delete will fix it.

Ross

[-- Attachment #2: Type: text/html, Size: 1176 bytes --]

Re: Fix for the APIC hangs in qemux86-64

[Bruce Ashfield]

On 12/13/2017 09:24 AM, Burton, Ross wrote:
> On 13 December 2017 at 14:17, Bruce Ashfield 
> <bruce.ashfield@windriver.com <mailto:bruce.ashfield@windriver.com>> wrote:
> 
>     yow-bashfiel-d4 [/home/bruc...poky/build]> oe-selftest -r
>     runqemu.RunqemuTests.test_boot_deploy
>     Traceback (most recent call last):
>        File "/home/bruce/poky/scripts/oe-selftest", line 43, in <module>
>          from oeqa.utils import load_test_components
>     ImportError: bad magic number in 'oeqa': b'\x03\xf3\r\n'
> 
> 
> You've been switching between master and a pre-py3 branch, and Python is 
> stupid so the py3 binary is trying to run py2 compiled binaries.


Yah. My build directory is three years old :D

> 
> find poky -name \*.pyc -delete will fix it.

Thanks. I'd have *never* come up with that!!!

Bruce

> 
> Ross

Re: Fix for the APIC hangs in qemux86-64

[Bruce Ashfield]

On 2017-12-13 9:14 AM, Richard Purdie wrote:
> On Wed, 2017-12-13 at 09:07 -0500, Bruce Ashfield wrote:
>> On 12/13/2017 09:05 AM, Richard Purdie wrote:
>>>
>>> On Wed, 2017-12-13 at 08:38 -0500, Bruce Ashfield wrote:
>>>>
>>>> On 12/13/2017 07:34 AM, Richard Purdie wrote:
>>>>>
>>>>>
>>>>> Hi Bruce,
>>>>>
>>>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301
>>>>>
>>>>> We've root caused the hangs we're seeing in qemux86-64 on our
>>>>> autobuilder workers. We've identified a commit on 4.15-rc3
>>>>> which
>>>>> fixes
>>>>> the issue but backporting it to the autobuilder's host kernels
>>>>> will
>>>>> be
>>>>> a pain.
>>>>>
>>>>> As a workaround I'd like to enable x2apic on qemux86-64. Since
>>>>> that
>>>>> works via MSRs it avoids the problem.
>>>>>
>>>>> Would it be a big issue to set CONFIG_X86_X2APIC=y for qemux86-
>>>>> 64?
>>>> Nope. No big deal at all. I've used the same config i the past
>>>> when doing work with rt-KVM, so I know it well.
>>>>
>>>> I'll prep a commit, and send it along as part of a rebased set
>>>> of kernel patches.
>>> Thanks, much appreciated.
>>>
>>> Did we get to the bottom of the 4.12 selftest failures?
>>>
>> Not yet no. I was distracted for a day, but need to learn how
>> to run them myself. I was able to log in via serial in my normal
>> boot flow, so that didn't get me anywhere.
>>
>> Is the selftest HOWO in the yocto docs ? I'll google it up this
>> morning.
> 
> No HOWTO but I did give you the command. In a poky checkout run:
> 
> oe-selftest -r runqemu.RunqemuTests.test_boot_deploy
> 
> which runs a single test that fails (setup DL_DIR/SSTATE_MIRRORS first
> to speed it up a bit).
> 
> It may tell you to clear SANITY_TESTED_DISTROS, I think it auto adds
> meta-selftest now. You may also need to set DISPLAY to something (e.g.
> a dummy vncserver).
> 

With the cleanup that Ross gave me, I'm now able to see the issue.

I'll start debugging on Thursday.

One additional question: what's the right way to find the logs of that
run ?

I'm getting a bit of noise from:

----------------
      0155:            else:
Exception: FileNotFoundError: [Errno 2] No such file or directory: 
'/home/bruce/poky/build/tmp/sysroots-components/x86_64/python3-native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt-1.pyc' 
-> 
'/home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6-r0/recipe-sysroot-native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt-1.pyc'

ERROR: gnu-efi-3.0.6-r0 do_package: Function failed: extend_recipe_sysroot
ERROR: Logfile of failure stored in: 
/home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6-r0/temp/log.do_package.10495

--------------

So I'm not seeing where to find it.

Bruce

> Not sure what is different about the serial qemu is using verses your
> own tests...
> 
>>   > We may also need to backport the defconfig change back to the older
>>   > releases so we don't get the hangs in those builds either...
>>   Not a problem either. I'll prepare several.>
> 
> Thanks!
> 
> Cheers,
> 
> Richard
>>

Re: Fix for the APIC hangs in qemux86-64

[Richard Purdie]

On Wed, 2017-12-13 at 19:01 -0500, Bruce Ashfield wrote:
> On 2017-12-13 9:14 AM, Richard Purdie wrote:
> > 
> > On Wed, 2017-12-13 at 09:07 -0500, Bruce Ashfield wrote:
> > > 
> > > On 12/13/2017 09:05 AM, Richard Purdie wrote:
> > > > 
> > > > 
> > > > On Wed, 2017-12-13 at 08:38 -0500, Bruce Ashfield wrote:
> > > > > 
> > > > > 
> > > > > On 12/13/2017 07:34 AM, Richard Purdie wrote:
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > Hi Bruce,
> > > > > > 
> > > > > > https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301
> > > > > > 
> > > > > > We've root caused the hangs we're seeing in qemux86-64 on
> > > > > > our
> > > > > > autobuilder workers. We've identified a commit on 4.15-rc3
> > > > > > which
> > > > > > fixes
> > > > > > the issue but backporting it to the autobuilder's host
> > > > > > kernels
> > > > > > will
> > > > > > be
> > > > > > a pain.
> > > > > > 
> > > > > > As a workaround I'd like to enable x2apic on qemux86-64.
> > > > > > Since
> > > > > > that
> > > > > > works via MSRs it avoids the problem.
> > > > > > 
> > > > > > Would it be a big issue to set CONFIG_X86_X2APIC=y for
> > > > > > qemux86-
> > > > > > 64?
> > > > > Nope. No big deal at all. I've used the same config i the
> > > > > past
> > > > > when doing work with rt-KVM, so I know it well.
> > > > > 
> > > > > I'll prep a commit, and send it along as part of a rebased
> > > > > set
> > > > > of kernel patches.
> > > > Thanks, much appreciated.
> > > > 
> > > > Did we get to the bottom of the 4.12 selftest failures?
> > > > 
> > > Not yet no. I was distracted for a day, but need to learn how
> > > to run them myself. I was able to log in via serial in my normal
> > > boot flow, so that didn't get me anywhere.
> > > 
> > > Is the selftest HOWO in the yocto docs ? I'll google it up this
> > > morning.
> > No HOWTO but I did give you the command. In a poky checkout run:
> > 
> > oe-selftest -r runqemu.RunqemuTests.test_boot_deploy
> > 
> > which runs a single test that fails (setup DL_DIR/SSTATE_MIRRORS
> > first
> > to speed it up a bit).
> > 
> > It may tell you to clear SANITY_TESTED_DISTROS, I think it auto
> > adds
> > meta-selftest now. You may also need to set DISPLAY to something
> > (e.g.
> > a dummy vncserver).
> > 
> With the cleanup that Ross gave me, I'm now able to see the issue.
> 
> I'll start debugging on Thursday.
> 
> One additional question: what's the right way to find the logs of
> that
> run ?
> 
> I'm getting a bit of noise from:
> 
> ----------------
>       0155:            else:
> Exception: FileNotFoundError: [Errno 2] No such file or directory: 
> '/home/bruce/poky/build/tmp/sysroots-components/x86_64/python3-
> native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt-
> 1.pyc' 
> -> 
> '/home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6-
> r0/recipe-sysroot-
> native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt-
> 1.pyc'
> 
> ERROR: gnu-efi-3.0.6-r0 do_package: Function failed:
> extend_recipe_sysroot
> ERROR: Logfile of failure stored in: 
> /home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6-
> r0/temp/log.do_package.10495

You've deleted pyc files from python itself which is bad. I'd
cleansstate python-native and rebuilt it to fix that. Ross meant onto
to run it on the poky meta directory...

Cheers,

Richard

Re: Fix for the APIC hangs in qemux86-64

[Bruce Ashfield]

On 2017-12-13 7:04 PM, Richard Purdie wrote:
> On Wed, 2017-12-13 at 19:01 -0500, Bruce Ashfield wrote:
>> On 2017-12-13 9:14 AM, Richard Purdie wrote:
>>>
>>> On Wed, 2017-12-13 at 09:07 -0500, Bruce Ashfield wrote:
>>>>
>>>> On 12/13/2017 09:05 AM, Richard Purdie wrote:
>>>>>
>>>>>
>>>>> On Wed, 2017-12-13 at 08:38 -0500, Bruce Ashfield wrote:
>>>>>>
>>>>>>
>>>>>> On 12/13/2017 07:34 AM, Richard Purdie wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hi Bruce,
>>>>>>>
>>>>>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301
>>>>>>>
>>>>>>> We've root caused the hangs we're seeing in qemux86-64 on
>>>>>>> our
>>>>>>> autobuilder workers. We've identified a commit on 4.15-rc3
>>>>>>> which
>>>>>>> fixes
>>>>>>> the issue but backporting it to the autobuilder's host
>>>>>>> kernels
>>>>>>> will
>>>>>>> be
>>>>>>> a pain.
>>>>>>>
>>>>>>> As a workaround I'd like to enable x2apic on qemux86-64.
>>>>>>> Since
>>>>>>> that
>>>>>>> works via MSRs it avoids the problem.
>>>>>>>
>>>>>>> Would it be a big issue to set CONFIG_X86_X2APIC=y for
>>>>>>> qemux86-
>>>>>>> 64?
>>>>>> Nope. No big deal at all. I've used the same config i the
>>>>>> past
>>>>>> when doing work with rt-KVM, so I know it well.
>>>>>>
>>>>>> I'll prep a commit, and send it along as part of a rebased
>>>>>> set
>>>>>> of kernel patches.
>>>>> Thanks, much appreciated.
>>>>>
>>>>> Did we get to the bottom of the 4.12 selftest failures?
>>>>>
>>>> Not yet no. I was distracted for a day, but need to learn how
>>>> to run them myself. I was able to log in via serial in my normal
>>>> boot flow, so that didn't get me anywhere.
>>>>
>>>> Is the selftest HOWO in the yocto docs ? I'll google it up this
>>>> morning.
>>> No HOWTO but I did give you the command. In a poky checkout run:
>>>
>>> oe-selftest -r runqemu.RunqemuTests.test_boot_deploy
>>>
>>> which runs a single test that fails (setup DL_DIR/SSTATE_MIRRORS
>>> first
>>> to speed it up a bit).
>>>
>>> It may tell you to clear SANITY_TESTED_DISTROS, I think it auto
>>> adds
>>> meta-selftest now. You may also need to set DISPLAY to something
>>> (e.g.
>>> a dummy vncserver).
>>>
>> With the cleanup that Ross gave me, I'm now able to see the issue.
>>
>> I'll start debugging on Thursday.
>>
>> One additional question: what's the right way to find the logs of
>> that
>> run ?
>>
>> I'm getting a bit of noise from:
>>
>> ----------------
>>        0155:            else:
>> Exception: FileNotFoundError: [Errno 2] No such file or directory:
>> '/home/bruce/poky/build/tmp/sysroots-components/x86_64/python3-
>> native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt-
>> 1.pyc'
>> ->
>> '/home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6-
>> r0/recipe-sysroot-
>> native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt-
>> 1.pyc'
>>
>> ERROR: gnu-efi-3.0.6-r0 do_package: Function failed:
>> extend_recipe_sysroot
>> ERROR: Logfile of failure stored in:
>> /home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6-
>> r0/temp/log.do_package.10495
> 
> You've deleted pyc files from python itself which is bad. I'd
> cleansstate python-native and rebuilt it to fix that. Ross meant onto
> to run it on the poky meta directory...

Yah. I clued into that right as I hit send. I'll see if I can save the
build now :D

Bruce

> 
> Cheers,
> 
> Richard
> 
> 

Re: Fix for the APIC hangs in qemux86-64

[Bruce Ashfield]

On 2017-12-13 7:04 PM, Richard Purdie wrote:
> On Wed, 2017-12-13 at 19:01 -0500, Bruce Ashfield wrote:
>> On 2017-12-13 9:14 AM, Richard Purdie wrote:
>>>
>>> On Wed, 2017-12-13 at 09:07 -0500, Bruce Ashfield wrote:
>>>>
>>>> On 12/13/2017 09:05 AM, Richard Purdie wrote:
>>>>>
>>>>>
>>>>> On Wed, 2017-12-13 at 08:38 -0500, Bruce Ashfield wrote:
>>>>>>
>>>>>>
>>>>>> On 12/13/2017 07:34 AM, Richard Purdie wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hi Bruce,
>>>>>>>
>>>>>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301
>>>>>>>
>>>>>>> We've root caused the hangs we're seeing in qemux86-64 on
>>>>>>> our
>>>>>>> autobuilder workers. We've identified a commit on 4.15-rc3
>>>>>>> which
>>>>>>> fixes
>>>>>>> the issue but backporting it to the autobuilder's host
>>>>>>> kernels
>>>>>>> will
>>>>>>> be
>>>>>>> a pain.
>>>>>>>
>>>>>>> As a workaround I'd like to enable x2apic on qemux86-64.
>>>>>>> Since
>>>>>>> that
>>>>>>> works via MSRs it avoids the problem.
>>>>>>>
>>>>>>> Would it be a big issue to set CONFIG_X86_X2APIC=y for
>>>>>>> qemux86-
>>>>>>> 64?
>>>>>> Nope. No big deal at all. I've used the same config i the
>>>>>> past
>>>>>> when doing work with rt-KVM, so I know it well.
>>>>>>
>>>>>> I'll prep a commit, and send it along as part of a rebased
>>>>>> set
>>>>>> of kernel patches.
>>>>> Thanks, much appreciated.
>>>>>
>>>>> Did we get to the bottom of the 4.12 selftest failures?
>>>>>
>>>> Not yet no. I was distracted for a day, but need to learn how
>>>> to run them myself. I was able to log in via serial in my normal
>>>> boot flow, so that didn't get me anywhere.
>>>>
>>>> Is the selftest HOWO in the yocto docs ? I'll google it up this
>>>> morning.
>>> No HOWTO but I did give you the command. In a poky checkout run:
>>>
>>> oe-selftest -r runqemu.RunqemuTests.test_boot_deploy
>>>
>>> which runs a single test that fails (setup DL_DIR/SSTATE_MIRRORS
>>> first
>>> to speed it up a bit).
>>>
>>> It may tell you to clear SANITY_TESTED_DISTROS, I think it auto
>>> adds
>>> meta-selftest now. You may also need to set DISPLAY to something
>>> (e.g.
>>> a dummy vncserver).
>>>
>> With the cleanup that Ross gave me, I'm now able to see the issue.
>>
>> I'll start debugging on Thursday.
>>
>> One additional question: what's the right way to find the logs of
>> that
>> run ?
>>
>> I'm getting a bit of noise from:
>>
>> ----------------
>>        0155:            else:
>> Exception: FileNotFoundError: [Errno 2] No such file or directory:
>> '/home/bruce/poky/build/tmp/sysroots-components/x86_64/python3-
>> native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt-
>> 1.pyc'
>> ->
>> '/home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6-
>> r0/recipe-sysroot-
>> native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt-
>> 1.pyc'
>>
>> ERROR: gnu-efi-3.0.6-r0 do_package: Function failed:
>> extend_recipe_sysroot
>> ERROR: Logfile of failure stored in:
>> /home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6-
>> r0/temp/log.do_package.10495
> 
> You've deleted pyc files from python itself which is bad. I'd
> cleansstate python-native and rebuilt it to fix that. Ross meant onto
> to run it on the poky meta directory...
> 

Is this what I should expect for the error ?

It looked like qemu timed out, but that is probably just the log
in issue, correct ?

A manual log into the same image worked!

Bruce

-------------------
32105  3290 python3 /home/bruce/poky/scripts/oe-selftest -r 
runqemu.RunqemuTests.test_boot_deploy
32520 32105 /bin/sh -c runqemu nographic 
/home/bruce/poky/build/tmp/deploy/images/qemux86-64 tcpserial=51081 
bootparams="console=tty1 console=ttyS0,115200n8 printk.time=1" 
qemuparams="-serial tcp:127.0.0.1:44545 -pidfile pidfile_32105"
32521 32520 python3 /home/bruce/poky/scripts/runqemu nographic 
/home/bruce/poky/build/tmp/deploy/images/qemux86-64 tcpserial=51081 
bootparams=console=tty1 console=ttyS0,115200n8 printk.time=1 
qemuparams=-serial tcp:127.0.0.1:44545 -pidfile pidfile_32105
32522 32105 python3 /home/bruce/poky/scripts/oe-selftest -r 
runqemu.RunqemuTests.test_boot_deploy
32527 32521 /bin/sh -c sudo /home/bruce/poky/scripts/runqemu-ifup 1000 
1000 
/home/bruce/poky/build/tmp/work/x86_64-linux/qemu-helper-native/1.0-r1/recipe-sysroot-native/usr/bin
32528 32527 sudo /home/bruce/poky/scripts/runqemu-ifup 1000 1000 
/home/bruce/poky/build/tmp/work/x86_64-linux/qemu-helper-native/1.0-r1/recipe-sysroot-native/usr/bin
32529     2 [kworker/u8:1]
32530 32105 ps axww -o pid,ppid,command

Qemu ended unexpectedly, dump data from host is in 
/tmp/oe-saved-tests/201712131950_qemu
Sending SIGTERM to runqemu
Sending SIGKILL to runqemu
Output from runqemu:
runqemu - INFO - Continuing with the following parameters:

runqemu - INFO - Setting up tap interface under sudo

----------------------------------------------------------------------
2017-12-13 19:52:57,550 - oe-selftest - INFO - Ran 1 test in 257.941s
2017-12-13 19:52:57,550 - oe-selftest - INFO - FAILED
2017-12-13 19:52:57,550 - oe-selftest - INFO -  (errors=1)
2017-12-13 19:52:57,550 - oe-selftest - INFO - RESULTS:
2017-12-13 19:52:57,551 - oe-selftest - INFO - RESULTS - 
runqemu.RunqemuTests.test_boot_deploy - Testcase 2007: ERROR
2017-12-13 19:52:57,551 - oe-selftest - INFO - SUMMARY:
2017-12-13 19:52:57,551 - oe-selftest - INFO - oe-selftest () - Ran 1 
test in 257.946s
2017-12-13 19:52:57,551 - oe-selftest - INFO - oe-selftest - FAIL - 
Required tests failed


> Cheers,
> 
> Richard
> 
> 

Re: Fix for the APIC hangs in qemux86-64

[Bruce Ashfield]

On 12/13/2017 07:04 PM, Richard Purdie wrote:
> On Wed, 2017-12-13 at 19:01 -0500, Bruce Ashfield wrote:
>> On 2017-12-13 9:14 AM, Richard Purdie wrote:
>>>
>>> On Wed, 2017-12-13 at 09:07 -0500, Bruce Ashfield wrote:
>>>>
>>>> On 12/13/2017 09:05 AM, Richard Purdie wrote:
>>>>>
>>>>>
>>>>> On Wed, 2017-12-13 at 08:38 -0500, Bruce Ashfield wrote:
>>>>>>
>>>>>>
>>>>>> On 12/13/2017 07:34 AM, Richard Purdie wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hi Bruce,
>>>>>>>
>>>>>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301
>>>>>>>
>>>>>>> We've root caused the hangs we're seeing in qemux86-64 on
>>>>>>> our
>>>>>>> autobuilder workers. We've identified a commit on 4.15-rc3
>>>>>>> which
>>>>>>> fixes
>>>>>>> the issue but backporting it to the autobuilder's host
>>>>>>> kernels
>>>>>>> will
>>>>>>> be
>>>>>>> a pain.
>>>>>>>
>>>>>>> As a workaround I'd like to enable x2apic on qemux86-64.
>>>>>>> Since
>>>>>>> that
>>>>>>> works via MSRs it avoids the problem.
>>>>>>>
>>>>>>> Would it be a big issue to set CONFIG_X86_X2APIC=y for
>>>>>>> qemux86-
>>>>>>> 64?
>>>>>> Nope. No big deal at all. I've used the same config i the
>>>>>> past
>>>>>> when doing work with rt-KVM, so I know it well.
>>>>>>
>>>>>> I'll prep a commit, and send it along as part of a rebased
>>>>>> set
>>>>>> of kernel patches.
>>>>> Thanks, much appreciated.
>>>>>
>>>>> Did we get to the bottom of the 4.12 selftest failures?
>>>>>
>>>> Not yet no. I was distracted for a day, but need to learn how
>>>> to run them myself. I was able to log in via serial in my normal
>>>> boot flow, so that didn't get me anywhere.
>>>>
>>>> Is the selftest HOWO in the yocto docs ? I'll google it up this
>>>> morning.
>>> No HOWTO but I did give you the command. In a poky checkout run:
>>>
>>> oe-selftest -r runqemu.RunqemuTests.test_boot_deploy
>>>
>>> which runs a single test that fails (setup DL_DIR/SSTATE_MIRRORS
>>> first
>>> to speed it up a bit).
>>>
>>> It may tell you to clear SANITY_TESTED_DISTROS, I think it auto
>>> adds
>>> meta-selftest now. You may also need to set DISPLAY to something
>>> (e.g.
>>> a dummy vncserver).
>>>
>> With the cleanup that Ross gave me, I'm now able to see the issue.
>>
>> I'll start debugging on Thursday.
>>
>> One additional question: what's the right way to find the logs of
>> that
>> run ?
>>
>> I'm getting a bit of noise from:
>>
>> ----------------
>>        0155:            else:
>> Exception: FileNotFoundError: [Errno 2] No such file or directory:
>> '/home/bruce/poky/build/tmp/sysroots-components/x86_64/python3-
>> native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt-
>> 1.pyc'
>> ->
>> '/home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6-
>> r0/recipe-sysroot-
>> native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt-
>> 1.pyc'
>>
>> ERROR: gnu-efi-3.0.6-r0 do_package: Function failed:
>> extend_recipe_sysroot
>> ERROR: Logfile of failure stored in:
>> /home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6-
>> r0/temp/log.do_package.10495
> 
> You've deleted pyc files from python itself which is bad. I'd
> cleansstate python-native and rebuilt it to fix that. Ross meant onto
> to run it on the poky meta directory...
> 

I'm fully up and running now. I did a boot last night, and did
see a failure in the selftest.

I then booted it manually (runqemu directly), logged in .. and now
I'm seeing passing selftests. grr.

https://pastebin.com/iA0vxrmD
https://pastebin.com/ved4xvbQ

Bruce

> Cheers,
> 
> Richard
> 
> 

Re: Fix for the APIC hangs in qemux86-64

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 3987 bytes --]

Whoops.  Maybe I should have been clearer... Sorry!

Ross

On 14 December 2017 at 00:04, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Wed, 2017-12-13 at 19:01 -0500, Bruce Ashfield wrote:
> > On 2017-12-13 9:14 AM, Richard Purdie wrote:
> > >
> > > On Wed, 2017-12-13 at 09:07 -0500, Bruce Ashfield wrote:
> > > >
> > > > On 12/13/2017 09:05 AM, Richard Purdie wrote:
> > > > >
> > > > >
> > > > > On Wed, 2017-12-13 at 08:38 -0500, Bruce Ashfield wrote:
> > > > > >
> > > > > >
> > > > > > On 12/13/2017 07:34 AM, Richard Purdie wrote:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Hi Bruce,
> > > > > > >
> > > > > > > https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301
> > > > > > >
> > > > > > > We've root caused the hangs we're seeing in qemux86-64 on
> > > > > > > our
> > > > > > > autobuilder workers. We've identified a commit on 4.15-rc3
> > > > > > > which
> > > > > > > fixes
> > > > > > > the issue but backporting it to the autobuilder's host
> > > > > > > kernels
> > > > > > > will
> > > > > > > be
> > > > > > > a pain.
> > > > > > >
> > > > > > > As a workaround I'd like to enable x2apic on qemux86-64.
> > > > > > > Since
> > > > > > > that
> > > > > > > works via MSRs it avoids the problem.
> > > > > > >
> > > > > > > Would it be a big issue to set CONFIG_X86_X2APIC=y for
> > > > > > > qemux86-
> > > > > > > 64?
> > > > > > Nope. No big deal at all. I've used the same config i the
> > > > > > past
> > > > > > when doing work with rt-KVM, so I know it well.
> > > > > >
> > > > > > I'll prep a commit, and send it along as part of a rebased
> > > > > > set
> > > > > > of kernel patches.
> > > > > Thanks, much appreciated.
> > > > >
> > > > > Did we get to the bottom of the 4.12 selftest failures?
> > > > >
> > > > Not yet no. I was distracted for a day, but need to learn how
> > > > to run them myself. I was able to log in via serial in my normal
> > > > boot flow, so that didn't get me anywhere.
> > > >
> > > > Is the selftest HOWO in the yocto docs ? I'll google it up this
> > > > morning.
> > > No HOWTO but I did give you the command. In a poky checkout run:
> > >
> > > oe-selftest -r runqemu.RunqemuTests.test_boot_deploy
> > >
> > > which runs a single test that fails (setup DL_DIR/SSTATE_MIRRORS
> > > first
> > > to speed it up a bit).
> > >
> > > It may tell you to clear SANITY_TESTED_DISTROS, I think it auto
> > > adds
> > > meta-selftest now. You may also need to set DISPLAY to something
> > > (e.g.
> > > a dummy vncserver).
> > >
> > With the cleanup that Ross gave me, I'm now able to see the issue.
> >
> > I'll start debugging on Thursday.
> >
> > One additional question: what's the right way to find the logs of
> > that
> > run ?
> >
> > I'm getting a bit of noise from:
> >
> > ----------------
> >       0155:            else:
> > Exception: FileNotFoundError: [Errno 2] No such file or directory:
> > '/home/bruce/poky/build/tmp/sysroots-components/x86_64/python3-
> > native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt-
> > 1.pyc'
> > ->
> > '/home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6-
> > r0/recipe-sysroot-
> > native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt-
> > 1.pyc'
> >
> > ERROR: gnu-efi-3.0.6-r0 do_package: Function failed:
> > extend_recipe_sysroot
> > ERROR: Logfile of failure stored in:
> > /home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6-
> > r0/temp/log.do_package.10495
>
> You've deleted pyc files from python itself which is bad. I'd
> cleansstate python-native and rebuilt it to fix that. Ross meant onto
> to run it on the poky meta directory...
>
> Cheers,
>
> Richard
>
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 6049 bytes --]