Re: Yocto Project Status WW47’17

[Richard Purdie <richard.purdie@linuxfoundation.org>]

[-- Attachment #1: Type: text/plain, Size: 1667 bytes --]

On Mon, 2017-11-20 at 20:22 -0500, Randy MacLeod wrote:
> > o   Issues with 4.13.10 host kernels booting kvm x86 guests on
> > Tumbleweed (Suse) and Fedora 26 (attempting to see if 4.13.12
> > helps)
>  
> Robert, can you test Fedora 26. It would help to have a defect open
> with steps to reproduce or
> something about the typical workflow/ build time/ day of the week/
> phase of the moon. 

We have some further data:

a) The issue occurs on 4.13.12

rpurdie@tumbleweed:/tmp> cat /proc/version 
Linux version 4.13.12-1-vanilla (geeko@buildhost) (gcc version 7.2.1 20171020 [gcc-7-branch revision 253932] (SUSE Linux)) #1 SMP PREEMPT Wed Nov 8 11:21:09 UTC 2017 (9151c66)

b) The hang usually occurs at the TIMER line in the kernel logs but can
occur after booting into userspace around the udevd line, or
occasionally later in the boot process.

c) The similarity between this and the ppc bug I worked on make me
strongly suspect qemu's timers are stopping firing and the guest is
sitting in the idle loop.

d) I do now have a way to brute force the hangs at will. The attached
"runqemu-parallel.py" script runs 50 qemus in parallel. In around 45s I
had 10 hung on the autobuilder. I can provide more info on using that
script if its not obvious. It does assume my recent master changes to
the qemuconf files so we don't need to run bitbake -e to run runqemu.

This could well be the same kind of locking issue we saw on ppc. I'll
continue to look into that.

Hopefully this extra information will put us on a good track to
resolving it now. It is continuing to break builds and stop patch
merging.

Cheers,

Richard

[-- Attachment #2: runqemu-parallel.py --]
[-- Type: text/x-python, Size: 2190 bytes --]

#!/usr/bin/env python3

import shutil
import time
import subprocess
import os

procs = list(range(1, 50))

logs = "/tmp/runqemu-log"
errlogs = "/tmp/zrunqemu-log"
logfds = {}
errlogfds = {}
processes = {}
image = "/media/build1/poky/build/tmp-musl/deploy/images/qemuppc/core-image-sato-qemuppc.qemuboot.conf"
kernel = "tmp-musl/deploy/images/qemuppc/vmlinux-qemuppc.bin"
machine = "qemuppc"

image = "/media/build1/poky/build/tmp-musl/deploy/images/qemux86-64/core-image-sato-qemux86-64.qemuboot.conf"
kernel = "tmp-musl/deploy/images/qemux86-64/bzImage-qemux86-64.bin"
machine = "qemux86-64"

env = os.environ.copy()
env['MACHINE'] = machine

def start(p):
    logfds[p] = open(logs + ".%d" % p, "w")
    errlogfds[p] = open(errlogs + ".%d" % p, "w")
    #debugopts = "-d unimp,guest_errors,int,out_asm,op,op_opt,op_ind,mmu,pcall,cpu_reset,nochain"
    #exec,cpu,in_asm,
    debugopts = "-d unimp,guest_errors,int"
    debugopts += " -D /tmp/qemu.%d -monitor pty" % p
    processes[p] = subprocess.Popen(["runqemu",machine,"nographic", "snapshot", "kvm", kernel, image, "qemuparams=-pidfile /tmp/zzqemu.%d.pid %s" % (p, debugopts)], stdout=logfds[p], stderr=errlogfds[p], env=env)
    print("Started %d" % p)

for p in procs:
    start(p)

lastsize = {}

while procs:
    sizes = {}
    time.sleep(0.5)

    for p in procs:
        logfile = logs + ".%d" % p
        pidfn = "/tmp/zzqemu.%d.pid" % p
        with open(logfile, "r") as lf:
            for l in lf:
                if "login:" in l:
                    with open(pidfn) as pidfile:
                        pid = int(pidfile.readline().strip())
                    print("Killing %d with pid %d" % (p, pid))
                    os.kill(pid, 9)
                    procs.remove(p)
                    if p in lastsize:
                        del lastsize[p]
                    time.sleep(0.1)
                    logfds[p].close()
                    errlogfds[p].close()
                    start(p)
                    procs.append(p)
        sizes[p] = os.stat(logfile).st_size
        if p in lastsize and sizes[p]:
           if not sizes[p] > lastsize[p]:
               print("%d stalled?" % p)
    lastsize = sizes