* Fix for the APIC hangs in qemux86-64 @ 2017-12-13 12:34 Richard Purdie 2017-12-13 13:38 ` Bruce Ashfield 0 siblings, 1 reply; 14+ messages in thread From: Richard Purdie @ 2017-12-13 12:34 UTC (permalink / raw) To: Ashfield, Bruce; +Cc: Lock, Joshua, openembedded-core Hi Bruce, https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301 We've root caused the hangs we're seeing in qemux86-64 on our autobuilder workers. We've identified a commit on 4.15-rc3 which fixes the issue but backporting it to the autobuilder's host kernels will be a pain. As a workaround I'd like to enable x2apic on qemux86-64. Since that works via MSRs it avoids the problem. Would it be a big issue to set CONFIG_X86_X2APIC=y for qemux86-64? Cheers, Richard ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Fix for the APIC hangs in qemux86-64 2017-12-13 12:34 Fix for the APIC hangs in qemux86-64 Richard Purdie @ 2017-12-13 13:38 ` Bruce Ashfield 2017-12-13 14:05 ` Richard Purdie 0 siblings, 1 reply; 14+ messages in thread From: Bruce Ashfield @ 2017-12-13 13:38 UTC (permalink / raw) To: Richard Purdie; +Cc: Lock, Joshua, openembedded-core On 12/13/2017 07:34 AM, Richard Purdie wrote: > Hi Bruce, > > https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301 > > We've root caused the hangs we're seeing in qemux86-64 on our > autobuilder workers. We've identified a commit on 4.15-rc3 which fixes > the issue but backporting it to the autobuilder's host kernels will be > a pain. > > As a workaround I'd like to enable x2apic on qemux86-64. Since that > works via MSRs it avoids the problem. > > Would it be a big issue to set CONFIG_X86_X2APIC=y for qemux86-64? Nope. No big deal at all. I've used the same config i the past when doing work with rt-KVM, so I know it well. I'll prep a commit, and send it along as part of a rebased set of kernel patches. Cheers, Bruce > > Cheers, > > Richard > > > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Fix for the APIC hangs in qemux86-64 2017-12-13 13:38 ` Bruce Ashfield @ 2017-12-13 14:05 ` Richard Purdie 2017-12-13 14:07 ` Bruce Ashfield 0 siblings, 1 reply; 14+ messages in thread From: Richard Purdie @ 2017-12-13 14:05 UTC (permalink / raw) To: Bruce Ashfield; +Cc: Lock, Joshua, openembedded-core On Wed, 2017-12-13 at 08:38 -0500, Bruce Ashfield wrote: > On 12/13/2017 07:34 AM, Richard Purdie wrote: > > > > Hi Bruce, > > > > https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301 > > > > We've root caused the hangs we're seeing in qemux86-64 on our > > autobuilder workers. We've identified a commit on 4.15-rc3 which > > fixes > > the issue but backporting it to the autobuilder's host kernels will > > be > > a pain. > > > > As a workaround I'd like to enable x2apic on qemux86-64. Since that > > works via MSRs it avoids the problem. > > > > Would it be a big issue to set CONFIG_X86_X2APIC=y for qemux86-64? > Nope. No big deal at all. I've used the same config i the past > when doing work with rt-KVM, so I know it well. > > I'll prep a commit, and send it along as part of a rebased set > of kernel patches. Thanks, much appreciated. Did we get to the bottom of the 4.12 selftest failures? We may also need to backport the defconfig change back to the older releases so we don't get the hangs in those builds either... Cheers, Richard ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Fix for the APIC hangs in qemux86-64 2017-12-13 14:05 ` Richard Purdie @ 2017-12-13 14:07 ` Bruce Ashfield 2017-12-13 14:14 ` Richard Purdie 0 siblings, 1 reply; 14+ messages in thread From: Bruce Ashfield @ 2017-12-13 14:07 UTC (permalink / raw) To: Richard Purdie; +Cc: Lock, Joshua, openembedded-core On 12/13/2017 09:05 AM, Richard Purdie wrote: > On Wed, 2017-12-13 at 08:38 -0500, Bruce Ashfield wrote: >> On 12/13/2017 07:34 AM, Richard Purdie wrote: >>> >>> Hi Bruce, >>> >>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301 >>> >>> We've root caused the hangs we're seeing in qemux86-64 on our >>> autobuilder workers. We've identified a commit on 4.15-rc3 which >>> fixes >>> the issue but backporting it to the autobuilder's host kernels will >>> be >>> a pain. >>> >>> As a workaround I'd like to enable x2apic on qemux86-64. Since that >>> works via MSRs it avoids the problem. >>> >>> Would it be a big issue to set CONFIG_X86_X2APIC=y for qemux86-64? >> Nope. No big deal at all. I've used the same config i the past >> when doing work with rt-KVM, so I know it well. >> >> I'll prep a commit, and send it along as part of a rebased set >> of kernel patches. > > Thanks, much appreciated. > > Did we get to the bottom of the 4.12 selftest failures? > Not yet no. I was distracted for a day, but need to learn how to run them myself. I was able to log in via serial in my normal boot flow, so that didn't get me anywhere. Is the selftest HOWO in the yocto docs ? I'll google it up this morning. > We may also need to backport the defconfig change back to the older > releases so we don't get the hangs in those builds either... Not a problem either. I'll prepare several. Bruce > > Cheers, > > Richard > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Fix for the APIC hangs in qemux86-64 2017-12-13 14:07 ` Bruce Ashfield @ 2017-12-13 14:14 ` Richard Purdie 2017-12-13 14:17 ` Bruce Ashfield 2017-12-14 0:01 ` Bruce Ashfield 0 siblings, 2 replies; 14+ messages in thread From: Richard Purdie @ 2017-12-13 14:14 UTC (permalink / raw) To: Bruce Ashfield; +Cc: Lock, Joshua, openembedded-core On Wed, 2017-12-13 at 09:07 -0500, Bruce Ashfield wrote: > On 12/13/2017 09:05 AM, Richard Purdie wrote: > > > > On Wed, 2017-12-13 at 08:38 -0500, Bruce Ashfield wrote: > > > > > > On 12/13/2017 07:34 AM, Richard Purdie wrote: > > > > > > > > > > > > Hi Bruce, > > > > > > > > https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301 > > > > > > > > We've root caused the hangs we're seeing in qemux86-64 on our > > > > autobuilder workers. We've identified a commit on 4.15-rc3 > > > > which > > > > fixes > > > > the issue but backporting it to the autobuilder's host kernels > > > > will > > > > be > > > > a pain. > > > > > > > > As a workaround I'd like to enable x2apic on qemux86-64. Since > > > > that > > > > works via MSRs it avoids the problem. > > > > > > > > Would it be a big issue to set CONFIG_X86_X2APIC=y for qemux86- > > > > 64? > > > Nope. No big deal at all. I've used the same config i the past > > > when doing work with rt-KVM, so I know it well. > > > > > > I'll prep a commit, and send it along as part of a rebased set > > > of kernel patches. > > Thanks, much appreciated. > > > > Did we get to the bottom of the 4.12 selftest failures? > > > Not yet no. I was distracted for a day, but need to learn how > to run them myself. I was able to log in via serial in my normal > boot flow, so that didn't get me anywhere. > > Is the selftest HOWO in the yocto docs ? I'll google it up this > morning. No HOWTO but I did give you the command. In a poky checkout run: oe-selftest -r runqemu.RunqemuTests.test_boot_deploy which runs a single test that fails (setup DL_DIR/SSTATE_MIRRORS first to speed it up a bit). It may tell you to clear SANITY_TESTED_DISTROS, I think it auto adds meta-selftest now. You may also need to set DISPLAY to something (e.g. a dummy vncserver). Not sure what is different about the serial qemu is using verses your own tests... > > We may also need to backport the defconfig change back to the older > > releases so we don't get the hangs in those builds either... > Not a problem either. I'll prepare several.> Thanks! Cheers, Richard > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Fix for the APIC hangs in qemux86-64 2017-12-13 14:14 ` Richard Purdie @ 2017-12-13 14:17 ` Bruce Ashfield 2017-12-13 14:24 ` Burton, Ross 2017-12-14 0:01 ` Bruce Ashfield 1 sibling, 1 reply; 14+ messages in thread From: Bruce Ashfield @ 2017-12-13 14:17 UTC (permalink / raw) To: Richard Purdie; +Cc: Lock, Joshua, openembedded-core On 12/13/2017 09:14 AM, Richard Purdie wrote: > On Wed, 2017-12-13 at 09:07 -0500, Bruce Ashfield wrote: >> On 12/13/2017 09:05 AM, Richard Purdie wrote: >>> >>> On Wed, 2017-12-13 at 08:38 -0500, Bruce Ashfield wrote: >>>> >>>> On 12/13/2017 07:34 AM, Richard Purdie wrote: >>>>> >>>>> >>>>> Hi Bruce, >>>>> >>>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301 >>>>> >>>>> We've root caused the hangs we're seeing in qemux86-64 on our >>>>> autobuilder workers. We've identified a commit on 4.15-rc3 >>>>> which >>>>> fixes >>>>> the issue but backporting it to the autobuilder's host kernels >>>>> will >>>>> be >>>>> a pain. >>>>> >>>>> As a workaround I'd like to enable x2apic on qemux86-64. Since >>>>> that >>>>> works via MSRs it avoids the problem. >>>>> >>>>> Would it be a big issue to set CONFIG_X86_X2APIC=y for qemux86- >>>>> 64? >>>> Nope. No big deal at all. I've used the same config i the past >>>> when doing work with rt-KVM, so I know it well. >>>> >>>> I'll prep a commit, and send it along as part of a rebased set >>>> of kernel patches. >>> Thanks, much appreciated. >>> >>> Did we get to the bottom of the 4.12 selftest failures? >>> >> Not yet no. I was distracted for a day, but need to learn how >> to run them myself. I was able to log in via serial in my normal >> boot flow, so that didn't get me anywhere. >> >> Is the selftest HOWO in the yocto docs ? I'll google it up this >> morning. > > No HOWTO but I did give you the command. In a poky checkout run: > > oe-selftest -r runqemu.RunqemuTests.test_boot_deploy Yah, that's where it went off the rails for me, so I figure that I'm doing something stupid: yow-bashfiel-d4 [/home/bruc...poky/build]> oe-selftest -r runqemu.RunqemuTests.test_boot_deploy Traceback (most recent call last): File "/home/bruce/poky/scripts/oe-selftest", line 43, in <module> from oeqa.utils import load_test_components ImportError: bad magic number in 'oeqa': b'\x03\xf3\r\n' > > which runs a single test that fails (setup DL_DIR/SSTATE_MIRRORS first > to speed it up a bit). > > It may tell you to clear SANITY_TESTED_DISTROS, I think it auto adds > meta-selftest now. You may also need to set DISPLAY to something (e.g. > a dummy vncserver). I'll keep smashing on it. Bruce > > Not sure what is different about the serial qemu is using verses your > own tests... > >> > We may also need to backport the defconfig change back to the older >> > releases so we don't get the hangs in those builds either... >> Not a problem either. I'll prepare several.> > > Thanks! > > Cheers, > > Richard >> ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Fix for the APIC hangs in qemux86-64 2017-12-13 14:17 ` Bruce Ashfield @ 2017-12-13 14:24 ` Burton, Ross 2017-12-13 14:25 ` Bruce Ashfield 0 siblings, 1 reply; 14+ messages in thread From: Burton, Ross @ 2017-12-13 14:24 UTC (permalink / raw) To: Bruce Ashfield; +Cc: Lock, Joshua, openembedded-core [-- Attachment #1: Type: text/plain, Size: 608 bytes --] On 13 December 2017 at 14:17, Bruce Ashfield <bruce.ashfield@windriver.com> wrote: > yow-bashfiel-d4 [/home/bruc...poky/build]> oe-selftest -r > runqemu.RunqemuTests.test_boot_deploy > Traceback (most recent call last): > File "/home/bruce/poky/scripts/oe-selftest", line 43, in <module> > from oeqa.utils import load_test_components > ImportError: bad magic number in 'oeqa': b'\x03\xf3\r\n' You've been switching between master and a pre-py3 branch, and Python is stupid so the py3 binary is trying to run py2 compiled binaries. find poky -name \*.pyc -delete will fix it. Ross [-- Attachment #2: Type: text/html, Size: 1176 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Fix for the APIC hangs in qemux86-64 2017-12-13 14:24 ` Burton, Ross @ 2017-12-13 14:25 ` Bruce Ashfield 0 siblings, 0 replies; 14+ messages in thread From: Bruce Ashfield @ 2017-12-13 14:25 UTC (permalink / raw) To: Burton, Ross; +Cc: Lock, Joshua, openembedded-core On 12/13/2017 09:24 AM, Burton, Ross wrote: > On 13 December 2017 at 14:17, Bruce Ashfield > <bruce.ashfield@windriver.com <mailto:bruce.ashfield@windriver.com>> wrote: > > yow-bashfiel-d4 [/home/bruc...poky/build]> oe-selftest -r > runqemu.RunqemuTests.test_boot_deploy > Traceback (most recent call last): > File "/home/bruce/poky/scripts/oe-selftest", line 43, in <module> > from oeqa.utils import load_test_components > ImportError: bad magic number in 'oeqa': b'\x03\xf3\r\n' > > > You've been switching between master and a pre-py3 branch, and Python is > stupid so the py3 binary is trying to run py2 compiled binaries. Yah. My build directory is three years old :D > > find poky -name \*.pyc -delete will fix it. Thanks. I'd have *never* come up with that!!! Bruce > > Ross ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Fix for the APIC hangs in qemux86-64 2017-12-13 14:14 ` Richard Purdie 2017-12-13 14:17 ` Bruce Ashfield @ 2017-12-14 0:01 ` Bruce Ashfield 2017-12-14 0:04 ` Richard Purdie 1 sibling, 1 reply; 14+ messages in thread From: Bruce Ashfield @ 2017-12-14 0:01 UTC (permalink / raw) To: Richard Purdie; +Cc: Lock, Joshua, openembedded-core On 2017-12-13 9:14 AM, Richard Purdie wrote: > On Wed, 2017-12-13 at 09:07 -0500, Bruce Ashfield wrote: >> On 12/13/2017 09:05 AM, Richard Purdie wrote: >>> >>> On Wed, 2017-12-13 at 08:38 -0500, Bruce Ashfield wrote: >>>> >>>> On 12/13/2017 07:34 AM, Richard Purdie wrote: >>>>> >>>>> >>>>> Hi Bruce, >>>>> >>>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301 >>>>> >>>>> We've root caused the hangs we're seeing in qemux86-64 on our >>>>> autobuilder workers. We've identified a commit on 4.15-rc3 >>>>> which >>>>> fixes >>>>> the issue but backporting it to the autobuilder's host kernels >>>>> will >>>>> be >>>>> a pain. >>>>> >>>>> As a workaround I'd like to enable x2apic on qemux86-64. Since >>>>> that >>>>> works via MSRs it avoids the problem. >>>>> >>>>> Would it be a big issue to set CONFIG_X86_X2APIC=y for qemux86- >>>>> 64? >>>> Nope. No big deal at all. I've used the same config i the past >>>> when doing work with rt-KVM, so I know it well. >>>> >>>> I'll prep a commit, and send it along as part of a rebased set >>>> of kernel patches. >>> Thanks, much appreciated. >>> >>> Did we get to the bottom of the 4.12 selftest failures? >>> >> Not yet no. I was distracted for a day, but need to learn how >> to run them myself. I was able to log in via serial in my normal >> boot flow, so that didn't get me anywhere. >> >> Is the selftest HOWO in the yocto docs ? I'll google it up this >> morning. > > No HOWTO but I did give you the command. In a poky checkout run: > > oe-selftest -r runqemu.RunqemuTests.test_boot_deploy > > which runs a single test that fails (setup DL_DIR/SSTATE_MIRRORS first > to speed it up a bit). > > It may tell you to clear SANITY_TESTED_DISTROS, I think it auto adds > meta-selftest now. You may also need to set DISPLAY to something (e.g. > a dummy vncserver). > With the cleanup that Ross gave me, I'm now able to see the issue. I'll start debugging on Thursday. One additional question: what's the right way to find the logs of that run ? I'm getting a bit of noise from: ---------------- 0155: else: Exception: FileNotFoundError: [Errno 2] No such file or directory: '/home/bruce/poky/build/tmp/sysroots-components/x86_64/python3-native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt-1.pyc' -> '/home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6-r0/recipe-sysroot-native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt-1.pyc' ERROR: gnu-efi-3.0.6-r0 do_package: Function failed: extend_recipe_sysroot ERROR: Logfile of failure stored in: /home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6-r0/temp/log.do_package.10495 -------------- So I'm not seeing where to find it. Bruce > Not sure what is different about the serial qemu is using verses your > own tests... > >> > We may also need to backport the defconfig change back to the older >> > releases so we don't get the hangs in those builds either... >> Not a problem either. I'll prepare several.> > > Thanks! > > Cheers, > > Richard >> ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Fix for the APIC hangs in qemux86-64 2017-12-14 0:01 ` Bruce Ashfield @ 2017-12-14 0:04 ` Richard Purdie 2017-12-14 0:05 ` Bruce Ashfield ` (3 more replies) 0 siblings, 4 replies; 14+ messages in thread From: Richard Purdie @ 2017-12-14 0:04 UTC (permalink / raw) To: Bruce Ashfield; +Cc: Lock, Joshua, openembedded-core On Wed, 2017-12-13 at 19:01 -0500, Bruce Ashfield wrote: > On 2017-12-13 9:14 AM, Richard Purdie wrote: > > > > On Wed, 2017-12-13 at 09:07 -0500, Bruce Ashfield wrote: > > > > > > On 12/13/2017 09:05 AM, Richard Purdie wrote: > > > > > > > > > > > > On Wed, 2017-12-13 at 08:38 -0500, Bruce Ashfield wrote: > > > > > > > > > > > > > > > On 12/13/2017 07:34 AM, Richard Purdie wrote: > > > > > > > > > > > > > > > > > > > > > > > > Hi Bruce, > > > > > > > > > > > > https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301 > > > > > > > > > > > > We've root caused the hangs we're seeing in qemux86-64 on > > > > > > our > > > > > > autobuilder workers. We've identified a commit on 4.15-rc3 > > > > > > which > > > > > > fixes > > > > > > the issue but backporting it to the autobuilder's host > > > > > > kernels > > > > > > will > > > > > > be > > > > > > a pain. > > > > > > > > > > > > As a workaround I'd like to enable x2apic on qemux86-64. > > > > > > Since > > > > > > that > > > > > > works via MSRs it avoids the problem. > > > > > > > > > > > > Would it be a big issue to set CONFIG_X86_X2APIC=y for > > > > > > qemux86- > > > > > > 64? > > > > > Nope. No big deal at all. I've used the same config i the > > > > > past > > > > > when doing work with rt-KVM, so I know it well. > > > > > > > > > > I'll prep a commit, and send it along as part of a rebased > > > > > set > > > > > of kernel patches. > > > > Thanks, much appreciated. > > > > > > > > Did we get to the bottom of the 4.12 selftest failures? > > > > > > > Not yet no. I was distracted for a day, but need to learn how > > > to run them myself. I was able to log in via serial in my normal > > > boot flow, so that didn't get me anywhere. > > > > > > Is the selftest HOWO in the yocto docs ? I'll google it up this > > > morning. > > No HOWTO but I did give you the command. In a poky checkout run: > > > > oe-selftest -r runqemu.RunqemuTests.test_boot_deploy > > > > which runs a single test that fails (setup DL_DIR/SSTATE_MIRRORS > > first > > to speed it up a bit). > > > > It may tell you to clear SANITY_TESTED_DISTROS, I think it auto > > adds > > meta-selftest now. You may also need to set DISPLAY to something > > (e.g. > > a dummy vncserver). > > > With the cleanup that Ross gave me, I'm now able to see the issue. > > I'll start debugging on Thursday. > > One additional question: what's the right way to find the logs of > that > run ? > > I'm getting a bit of noise from: > > ---------------- > 0155: else: > Exception: FileNotFoundError: [Errno 2] No such file or directory: > '/home/bruce/poky/build/tmp/sysroots-components/x86_64/python3- > native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt- > 1.pyc' > -> > '/home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6- > r0/recipe-sysroot- > native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt- > 1.pyc' > > ERROR: gnu-efi-3.0.6-r0 do_package: Function failed: > extend_recipe_sysroot > ERROR: Logfile of failure stored in: > /home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6- > r0/temp/log.do_package.10495 You've deleted pyc files from python itself which is bad. I'd cleansstate python-native and rebuilt it to fix that. Ross meant onto to run it on the poky meta directory... Cheers, Richard ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Fix for the APIC hangs in qemux86-64 2017-12-14 0:04 ` Richard Purdie @ 2017-12-14 0:05 ` Bruce Ashfield 2017-12-14 1:12 ` Bruce Ashfield ` (2 subsequent siblings) 3 siblings, 0 replies; 14+ messages in thread From: Bruce Ashfield @ 2017-12-14 0:05 UTC (permalink / raw) To: Richard Purdie; +Cc: Lock, Joshua, openembedded-core On 2017-12-13 7:04 PM, Richard Purdie wrote: > On Wed, 2017-12-13 at 19:01 -0500, Bruce Ashfield wrote: >> On 2017-12-13 9:14 AM, Richard Purdie wrote: >>> >>> On Wed, 2017-12-13 at 09:07 -0500, Bruce Ashfield wrote: >>>> >>>> On 12/13/2017 09:05 AM, Richard Purdie wrote: >>>>> >>>>> >>>>> On Wed, 2017-12-13 at 08:38 -0500, Bruce Ashfield wrote: >>>>>> >>>>>> >>>>>> On 12/13/2017 07:34 AM, Richard Purdie wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> Hi Bruce, >>>>>>> >>>>>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301 >>>>>>> >>>>>>> We've root caused the hangs we're seeing in qemux86-64 on >>>>>>> our >>>>>>> autobuilder workers. We've identified a commit on 4.15-rc3 >>>>>>> which >>>>>>> fixes >>>>>>> the issue but backporting it to the autobuilder's host >>>>>>> kernels >>>>>>> will >>>>>>> be >>>>>>> a pain. >>>>>>> >>>>>>> As a workaround I'd like to enable x2apic on qemux86-64. >>>>>>> Since >>>>>>> that >>>>>>> works via MSRs it avoids the problem. >>>>>>> >>>>>>> Would it be a big issue to set CONFIG_X86_X2APIC=y for >>>>>>> qemux86- >>>>>>> 64? >>>>>> Nope. No big deal at all. I've used the same config i the >>>>>> past >>>>>> when doing work with rt-KVM, so I know it well. >>>>>> >>>>>> I'll prep a commit, and send it along as part of a rebased >>>>>> set >>>>>> of kernel patches. >>>>> Thanks, much appreciated. >>>>> >>>>> Did we get to the bottom of the 4.12 selftest failures? >>>>> >>>> Not yet no. I was distracted for a day, but need to learn how >>>> to run them myself. I was able to log in via serial in my normal >>>> boot flow, so that didn't get me anywhere. >>>> >>>> Is the selftest HOWO in the yocto docs ? I'll google it up this >>>> morning. >>> No HOWTO but I did give you the command. In a poky checkout run: >>> >>> oe-selftest -r runqemu.RunqemuTests.test_boot_deploy >>> >>> which runs a single test that fails (setup DL_DIR/SSTATE_MIRRORS >>> first >>> to speed it up a bit). >>> >>> It may tell you to clear SANITY_TESTED_DISTROS, I think it auto >>> adds >>> meta-selftest now. You may also need to set DISPLAY to something >>> (e.g. >>> a dummy vncserver). >>> >> With the cleanup that Ross gave me, I'm now able to see the issue. >> >> I'll start debugging on Thursday. >> >> One additional question: what's the right way to find the logs of >> that >> run ? >> >> I'm getting a bit of noise from: >> >> ---------------- >> 0155: else: >> Exception: FileNotFoundError: [Errno 2] No such file or directory: >> '/home/bruce/poky/build/tmp/sysroots-components/x86_64/python3- >> native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt- >> 1.pyc' >> -> >> '/home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6- >> r0/recipe-sysroot- >> native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt- >> 1.pyc' >> >> ERROR: gnu-efi-3.0.6-r0 do_package: Function failed: >> extend_recipe_sysroot >> ERROR: Logfile of failure stored in: >> /home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6- >> r0/temp/log.do_package.10495 > > You've deleted pyc files from python itself which is bad. I'd > cleansstate python-native and rebuilt it to fix that. Ross meant onto > to run it on the poky meta directory... Yah. I clued into that right as I hit send. I'll see if I can save the build now :D Bruce > > Cheers, > > Richard > > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Fix for the APIC hangs in qemux86-64 2017-12-14 0:04 ` Richard Purdie 2017-12-14 0:05 ` Bruce Ashfield @ 2017-12-14 1:12 ` Bruce Ashfield 2017-12-14 15:00 ` Bruce Ashfield 2017-12-15 12:34 ` Burton, Ross 3 siblings, 0 replies; 14+ messages in thread From: Bruce Ashfield @ 2017-12-14 1:12 UTC (permalink / raw) To: Richard Purdie; +Cc: Lock, Joshua, openembedded-core On 2017-12-13 7:04 PM, Richard Purdie wrote: > On Wed, 2017-12-13 at 19:01 -0500, Bruce Ashfield wrote: >> On 2017-12-13 9:14 AM, Richard Purdie wrote: >>> >>> On Wed, 2017-12-13 at 09:07 -0500, Bruce Ashfield wrote: >>>> >>>> On 12/13/2017 09:05 AM, Richard Purdie wrote: >>>>> >>>>> >>>>> On Wed, 2017-12-13 at 08:38 -0500, Bruce Ashfield wrote: >>>>>> >>>>>> >>>>>> On 12/13/2017 07:34 AM, Richard Purdie wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> Hi Bruce, >>>>>>> >>>>>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301 >>>>>>> >>>>>>> We've root caused the hangs we're seeing in qemux86-64 on >>>>>>> our >>>>>>> autobuilder workers. We've identified a commit on 4.15-rc3 >>>>>>> which >>>>>>> fixes >>>>>>> the issue but backporting it to the autobuilder's host >>>>>>> kernels >>>>>>> will >>>>>>> be >>>>>>> a pain. >>>>>>> >>>>>>> As a workaround I'd like to enable x2apic on qemux86-64. >>>>>>> Since >>>>>>> that >>>>>>> works via MSRs it avoids the problem. >>>>>>> >>>>>>> Would it be a big issue to set CONFIG_X86_X2APIC=y for >>>>>>> qemux86- >>>>>>> 64? >>>>>> Nope. No big deal at all. I've used the same config i the >>>>>> past >>>>>> when doing work with rt-KVM, so I know it well. >>>>>> >>>>>> I'll prep a commit, and send it along as part of a rebased >>>>>> set >>>>>> of kernel patches. >>>>> Thanks, much appreciated. >>>>> >>>>> Did we get to the bottom of the 4.12 selftest failures? >>>>> >>>> Not yet no. I was distracted for a day, but need to learn how >>>> to run them myself. I was able to log in via serial in my normal >>>> boot flow, so that didn't get me anywhere. >>>> >>>> Is the selftest HOWO in the yocto docs ? I'll google it up this >>>> morning. >>> No HOWTO but I did give you the command. In a poky checkout run: >>> >>> oe-selftest -r runqemu.RunqemuTests.test_boot_deploy >>> >>> which runs a single test that fails (setup DL_DIR/SSTATE_MIRRORS >>> first >>> to speed it up a bit). >>> >>> It may tell you to clear SANITY_TESTED_DISTROS, I think it auto >>> adds >>> meta-selftest now. You may also need to set DISPLAY to something >>> (e.g. >>> a dummy vncserver). >>> >> With the cleanup that Ross gave me, I'm now able to see the issue. >> >> I'll start debugging on Thursday. >> >> One additional question: what's the right way to find the logs of >> that >> run ? >> >> I'm getting a bit of noise from: >> >> ---------------- >> 0155: else: >> Exception: FileNotFoundError: [Errno 2] No such file or directory: >> '/home/bruce/poky/build/tmp/sysroots-components/x86_64/python3- >> native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt- >> 1.pyc' >> -> >> '/home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6- >> r0/recipe-sysroot- >> native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt- >> 1.pyc' >> >> ERROR: gnu-efi-3.0.6-r0 do_package: Function failed: >> extend_recipe_sysroot >> ERROR: Logfile of failure stored in: >> /home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6- >> r0/temp/log.do_package.10495 > > You've deleted pyc files from python itself which is bad. I'd > cleansstate python-native and rebuilt it to fix that. Ross meant onto > to run it on the poky meta directory... > Is this what I should expect for the error ? It looked like qemu timed out, but that is probably just the log in issue, correct ? A manual log into the same image worked! Bruce ------------------- 32105 3290 python3 /home/bruce/poky/scripts/oe-selftest -r runqemu.RunqemuTests.test_boot_deploy 32520 32105 /bin/sh -c runqemu nographic /home/bruce/poky/build/tmp/deploy/images/qemux86-64 tcpserial=51081 bootparams="console=tty1 console=ttyS0,115200n8 printk.time=1" qemuparams="-serial tcp:127.0.0.1:44545 -pidfile pidfile_32105" 32521 32520 python3 /home/bruce/poky/scripts/runqemu nographic /home/bruce/poky/build/tmp/deploy/images/qemux86-64 tcpserial=51081 bootparams=console=tty1 console=ttyS0,115200n8 printk.time=1 qemuparams=-serial tcp:127.0.0.1:44545 -pidfile pidfile_32105 32522 32105 python3 /home/bruce/poky/scripts/oe-selftest -r runqemu.RunqemuTests.test_boot_deploy 32527 32521 /bin/sh -c sudo /home/bruce/poky/scripts/runqemu-ifup 1000 1000 /home/bruce/poky/build/tmp/work/x86_64-linux/qemu-helper-native/1.0-r1/recipe-sysroot-native/usr/bin 32528 32527 sudo /home/bruce/poky/scripts/runqemu-ifup 1000 1000 /home/bruce/poky/build/tmp/work/x86_64-linux/qemu-helper-native/1.0-r1/recipe-sysroot-native/usr/bin 32529 2 [kworker/u8:1] 32530 32105 ps axww -o pid,ppid,command Qemu ended unexpectedly, dump data from host is in /tmp/oe-saved-tests/201712131950_qemu Sending SIGTERM to runqemu Sending SIGKILL to runqemu Output from runqemu: runqemu - INFO - Continuing with the following parameters: runqemu - INFO - Setting up tap interface under sudo ---------------------------------------------------------------------- 2017-12-13 19:52:57,550 - oe-selftest - INFO - Ran 1 test in 257.941s 2017-12-13 19:52:57,550 - oe-selftest - INFO - FAILED 2017-12-13 19:52:57,550 - oe-selftest - INFO - (errors=1) 2017-12-13 19:52:57,550 - oe-selftest - INFO - RESULTS: 2017-12-13 19:52:57,551 - oe-selftest - INFO - RESULTS - runqemu.RunqemuTests.test_boot_deploy - Testcase 2007: ERROR 2017-12-13 19:52:57,551 - oe-selftest - INFO - SUMMARY: 2017-12-13 19:52:57,551 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 257.946s 2017-12-13 19:52:57,551 - oe-selftest - INFO - oe-selftest - FAIL - Required tests failed > Cheers, > > Richard > > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Fix for the APIC hangs in qemux86-64 2017-12-14 0:04 ` Richard Purdie 2017-12-14 0:05 ` Bruce Ashfield 2017-12-14 1:12 ` Bruce Ashfield @ 2017-12-14 15:00 ` Bruce Ashfield 2017-12-15 12:34 ` Burton, Ross 3 siblings, 0 replies; 14+ messages in thread From: Bruce Ashfield @ 2017-12-14 15:00 UTC (permalink / raw) To: Richard Purdie; +Cc: Lock, Joshua, openembedded-core On 12/13/2017 07:04 PM, Richard Purdie wrote: > On Wed, 2017-12-13 at 19:01 -0500, Bruce Ashfield wrote: >> On 2017-12-13 9:14 AM, Richard Purdie wrote: >>> >>> On Wed, 2017-12-13 at 09:07 -0500, Bruce Ashfield wrote: >>>> >>>> On 12/13/2017 09:05 AM, Richard Purdie wrote: >>>>> >>>>> >>>>> On Wed, 2017-12-13 at 08:38 -0500, Bruce Ashfield wrote: >>>>>> >>>>>> >>>>>> On 12/13/2017 07:34 AM, Richard Purdie wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> Hi Bruce, >>>>>>> >>>>>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301 >>>>>>> >>>>>>> We've root caused the hangs we're seeing in qemux86-64 on >>>>>>> our >>>>>>> autobuilder workers. We've identified a commit on 4.15-rc3 >>>>>>> which >>>>>>> fixes >>>>>>> the issue but backporting it to the autobuilder's host >>>>>>> kernels >>>>>>> will >>>>>>> be >>>>>>> a pain. >>>>>>> >>>>>>> As a workaround I'd like to enable x2apic on qemux86-64. >>>>>>> Since >>>>>>> that >>>>>>> works via MSRs it avoids the problem. >>>>>>> >>>>>>> Would it be a big issue to set CONFIG_X86_X2APIC=y for >>>>>>> qemux86- >>>>>>> 64? >>>>>> Nope. No big deal at all. I've used the same config i the >>>>>> past >>>>>> when doing work with rt-KVM, so I know it well. >>>>>> >>>>>> I'll prep a commit, and send it along as part of a rebased >>>>>> set >>>>>> of kernel patches. >>>>> Thanks, much appreciated. >>>>> >>>>> Did we get to the bottom of the 4.12 selftest failures? >>>>> >>>> Not yet no. I was distracted for a day, but need to learn how >>>> to run them myself. I was able to log in via serial in my normal >>>> boot flow, so that didn't get me anywhere. >>>> >>>> Is the selftest HOWO in the yocto docs ? I'll google it up this >>>> morning. >>> No HOWTO but I did give you the command. In a poky checkout run: >>> >>> oe-selftest -r runqemu.RunqemuTests.test_boot_deploy >>> >>> which runs a single test that fails (setup DL_DIR/SSTATE_MIRRORS >>> first >>> to speed it up a bit). >>> >>> It may tell you to clear SANITY_TESTED_DISTROS, I think it auto >>> adds >>> meta-selftest now. You may also need to set DISPLAY to something >>> (e.g. >>> a dummy vncserver). >>> >> With the cleanup that Ross gave me, I'm now able to see the issue. >> >> I'll start debugging on Thursday. >> >> One additional question: what's the right way to find the logs of >> that >> run ? >> >> I'm getting a bit of noise from: >> >> ---------------- >> 0155: else: >> Exception: FileNotFoundError: [Errno 2] No such file or directory: >> '/home/bruce/poky/build/tmp/sysroots-components/x86_64/python3- >> native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt- >> 1.pyc' >> -> >> '/home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6- >> r0/recipe-sysroot- >> native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt- >> 1.pyc' >> >> ERROR: gnu-efi-3.0.6-r0 do_package: Function failed: >> extend_recipe_sysroot >> ERROR: Logfile of failure stored in: >> /home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6- >> r0/temp/log.do_package.10495 > > You've deleted pyc files from python itself which is bad. I'd > cleansstate python-native and rebuilt it to fix that. Ross meant onto > to run it on the poky meta directory... > I'm fully up and running now. I did a boot last night, and did see a failure in the selftest. I then booted it manually (runqemu directly), logged in .. and now I'm seeing passing selftests. grr. https://pastebin.com/iA0vxrmD https://pastebin.com/ved4xvbQ Bruce > Cheers, > > Richard > > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Fix for the APIC hangs in qemux86-64 2017-12-14 0:04 ` Richard Purdie ` (2 preceding siblings ...) 2017-12-14 15:00 ` Bruce Ashfield @ 2017-12-15 12:34 ` Burton, Ross 3 siblings, 0 replies; 14+ messages in thread From: Burton, Ross @ 2017-12-15 12:34 UTC (permalink / raw) To: Richard Purdie; +Cc: Bruce Ashfield, Lock, Joshua, openembedded-core [-- Attachment #1: Type: text/plain, Size: 3987 bytes --] Whoops. Maybe I should have been clearer... Sorry! Ross On 14 December 2017 at 00:04, Richard Purdie < richard.purdie@linuxfoundation.org> wrote: > On Wed, 2017-12-13 at 19:01 -0500, Bruce Ashfield wrote: > > On 2017-12-13 9:14 AM, Richard Purdie wrote: > > > > > > On Wed, 2017-12-13 at 09:07 -0500, Bruce Ashfield wrote: > > > > > > > > On 12/13/2017 09:05 AM, Richard Purdie wrote: > > > > > > > > > > > > > > > On Wed, 2017-12-13 at 08:38 -0500, Bruce Ashfield wrote: > > > > > > > > > > > > > > > > > > On 12/13/2017 07:34 AM, Richard Purdie wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Bruce, > > > > > > > > > > > > > > https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301 > > > > > > > > > > > > > > We've root caused the hangs we're seeing in qemux86-64 on > > > > > > > our > > > > > > > autobuilder workers. We've identified a commit on 4.15-rc3 > > > > > > > which > > > > > > > fixes > > > > > > > the issue but backporting it to the autobuilder's host > > > > > > > kernels > > > > > > > will > > > > > > > be > > > > > > > a pain. > > > > > > > > > > > > > > As a workaround I'd like to enable x2apic on qemux86-64. > > > > > > > Since > > > > > > > that > > > > > > > works via MSRs it avoids the problem. > > > > > > > > > > > > > > Would it be a big issue to set CONFIG_X86_X2APIC=y for > > > > > > > qemux86- > > > > > > > 64? > > > > > > Nope. No big deal at all. I've used the same config i the > > > > > > past > > > > > > when doing work with rt-KVM, so I know it well. > > > > > > > > > > > > I'll prep a commit, and send it along as part of a rebased > > > > > > set > > > > > > of kernel patches. > > > > > Thanks, much appreciated. > > > > > > > > > > Did we get to the bottom of the 4.12 selftest failures? > > > > > > > > > Not yet no. I was distracted for a day, but need to learn how > > > > to run them myself. I was able to log in via serial in my normal > > > > boot flow, so that didn't get me anywhere. > > > > > > > > Is the selftest HOWO in the yocto docs ? I'll google it up this > > > > morning. > > > No HOWTO but I did give you the command. In a poky checkout run: > > > > > > oe-selftest -r runqemu.RunqemuTests.test_boot_deploy > > > > > > which runs a single test that fails (setup DL_DIR/SSTATE_MIRRORS > > > first > > > to speed it up a bit). > > > > > > It may tell you to clear SANITY_TESTED_DISTROS, I think it auto > > > adds > > > meta-selftest now. You may also need to set DISPLAY to something > > > (e.g. > > > a dummy vncserver). > > > > > With the cleanup that Ross gave me, I'm now able to see the issue. > > > > I'll start debugging on Thursday. > > > > One additional question: what's the right way to find the logs of > > that > > run ? > > > > I'm getting a bit of noise from: > > > > ---------------- > > 0155: else: > > Exception: FileNotFoundError: [Errno 2] No such file or directory: > > '/home/bruce/poky/build/tmp/sysroots-components/x86_64/python3- > > native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt- > > 1.pyc' > > -> > > '/home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6- > > r0/recipe-sysroot- > > native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt- > > 1.pyc' > > > > ERROR: gnu-efi-3.0.6-r0 do_package: Function failed: > > extend_recipe_sysroot > > ERROR: Logfile of failure stored in: > > /home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6- > > r0/temp/log.do_package.10495 > > You've deleted pyc files from python itself which is bad. I'd > cleansstate python-native and rebuilt it to fix that. Ross meant onto > to run it on the poky meta directory... > > Cheers, > > Richard > > > -- > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core > [-- Attachment #2: Type: text/html, Size: 6049 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2017-12-15 12:34 UTC | newest] Thread overview: 14+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2017-12-13 12:34 Fix for the APIC hangs in qemux86-64 Richard Purdie 2017-12-13 13:38 ` Bruce Ashfield 2017-12-13 14:05 ` Richard Purdie 2017-12-13 14:07 ` Bruce Ashfield 2017-12-13 14:14 ` Richard Purdie 2017-12-13 14:17 ` Bruce Ashfield 2017-12-13 14:24 ` Burton, Ross 2017-12-13 14:25 ` Bruce Ashfield 2017-12-14 0:01 ` Bruce Ashfield 2017-12-14 0:04 ` Richard Purdie 2017-12-14 0:05 ` Bruce Ashfield 2017-12-14 1:12 ` Bruce Ashfield 2017-12-14 15:00 ` Bruce Ashfield 2017-12-15 12:34 ` Burton, Ross
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox