[PATCH 1/6] runqemu: Ensure we process all tap devices

[PATCH 1/6] runqemu: Ensure we process all tap devices

[Richard Purdie]

The regexp in the script misses some tap devices, e.g. we see output like:

runqemu - INFO - Acquiring lockfile /tmp/qemu-tap-locks/tap25.lock failed: [Errno 11] Resource temporarily unavailable
runqemu - INFO - Acquiring lockfile /tmp/qemu-tap-locks/tap26.lock failed: [Errno 11] Resource temporarily unavailable
runqemu - INFO - Acquiring lockfile /tmp/qemu-tap-locks/tap27.lock failed: [Errno 11] Resource temporarily unavailable
runqemu - INFO - Acquiring lockfile /tmp/qemu-tap-locks/tap28.lock failed: [Errno 11] Resource temporarily unavailable
runqemu - INFO - Acquiring lockfile /tmp/qemu-tap-locks/tap40.lock failed: [Errno 11] Resource temporarily unavailable
runqemu - INFO - Acquiring lockfile /tmp/qemu-tap-locks/tap41.lock failed: [Errno 11] Resource temporarily unavailable

What happened to tap29 to tap39?

The issue is was we were missing devices with '0' in the number,
like "10:" and so on in the output from "ip link".

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 scripts/runqemu | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/runqemu b/scripts/runqemu
index fe4459d..ad88da3 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -923,7 +923,7 @@ class BaseConfig(object):
         logger.debug('Running %s...' % cmd)
         ip_link = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE).stdout.read().decode('utf-8')
         # Matches line like: 6: tap0: <foo>
-        possibles = re.findall('^[1-9]+: +(tap[0-9]+): <.*', ip_link, re.M)
+        possibles = re.findall('^[0-9]+: +(tap[0-9]+): <.*', ip_link, re.M)
         tap = ""
         for p in possibles:
             lockfile = os.path.join(lockdir, p)
-- 
2.7.4

[PATCH 2/6] qemu: Add patch to avoid qemuppc boot hangs

[Richard Purdie]

qemuppc boots are occasionally hanging on the autobuilder. This adds a
patch which fixes the issue in local testing. Its being discussed with
upstream qemu.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/recipes-devtools/qemu/qemu/ppc_locking.patch | 105 ++++++++++++++++++++++
 meta/recipes-devtools/qemu/qemu_2.10.1.bb         |   1 +
 2 files changed, 106 insertions(+)
 create mode 100644 meta/recipes-devtools/qemu/qemu/ppc_locking.patch

diff --git a/meta/recipes-devtools/qemu/qemu/ppc_locking.patch b/meta/recipes-devtools/qemu/qemu/ppc_locking.patch
new file mode 100644
index 0000000..6f72243
--- /dev/null
+++ b/meta/recipes-devtools/qemu/qemu/ppc_locking.patch
@@ -0,0 +1,105 @@
+I've tracked down what I think is a problem causing qemu-system-ppc
+to hang whilst booting images.
+
+I believe the decrementer timer stops receiving interrupts so
+tasks in our images hang indefinitely as the timer stopped. 
+
+It can be summed up with this line of debug:
+
+ppc_set_irq: 0x55b4e0d562f0 n_IRQ 8 level 1 => pending 00000100req 00000004
+
+It should normally read:
+
+ppc_set_irq: 0x55b4e0d562f0 n_IRQ 8 level 1 => pending 00000100req 00000002
+
+The question is why CPU_INTERRUPT_EXITTB ends up being set when the
+lines above this log message clearly sets CPU_INTERRUPT_HARD (via 
+cpu_interrupt() ).
+
+I note in cpu.h:
+
+    /* updates protected by BQL */
+    uint32_t interrupt_request;
+
+(for struct CPUState)
+
+The ppc code does "cs->interrupt_request |= CPU_INTERRUPT_EXITTB" in 5
+places, 3 in excp_helper.c and 2 in helper_regs.h. In all cases,  
+g_assert(qemu_mutex_iothread_locked()); fails. If I do something like:
+
+if (!qemu_mutex_iothread_locked()) {
+    qemu_mutex_lock_iothread();
+    cpu_interrupt(cs, CPU_INTERRUPT_EXITTB);
+    qemu_mutex_unlock_iothread();
+} else {
+    cpu_interrupt(cs, CPU_INTERRUPT_EXITTB);
+}
+
+in these call sites then I can no longer lock qemu up with my test
+case.
+
+I suspect the _HARD setting gets overwritten which stops the 
+decrementer interrupts being delivered.
+
+Upstream-Status: Submitted [Issue discussed on qemu mailing list 2017/11/20]
+RP 2017/11/20
+
+Index: qemu-2.10.1/target/ppc/excp_helper.c
+===================================================================
+--- qemu-2.10.1.orig/target/ppc/excp_helper.c
++++ qemu-2.10.1/target/ppc/excp_helper.c
+@@ -207,7 +207,9 @@ static inline void powerpc_excp(PowerPCC
+                         "Entering checkstop state\n");
+             }
+             cs->halted = 1;
+-            cs->interrupt_request |= CPU_INTERRUPT_EXITTB;
++            qemu_mutex_lock_iothread();
++            cpu_interrupt(cs, CPU_INTERRUPT_EXITTB);
++            qemu_mutex_unlock_iothread();
+         }
+         if (env->msr_mask & MSR_HVB) {
+             /* ISA specifies HV, but can be delivered to guest with HV clear
+@@ -940,7 +942,9 @@ void helper_store_msr(CPUPPCState *env,
+ 
+     if (excp != 0) {
+         CPUState *cs = CPU(ppc_env_get_cpu(env));
+-        cs->interrupt_request |= CPU_INTERRUPT_EXITTB;
++        qemu_mutex_lock_iothread();
++        cpu_interrupt(cs, CPU_INTERRUPT_EXITTB);
++        qemu_mutex_unlock_iothread();
+         raise_exception(env, excp);
+     }
+ }
+@@ -995,7 +999,9 @@ static inline void do_rfi(CPUPPCState *e
+     /* No need to raise an exception here,
+      * as rfi is always the last insn of a TB
+      */
+-    cs->interrupt_request |= CPU_INTERRUPT_EXITTB;
++    qemu_mutex_lock_iothread();
++    cpu_interrupt(cs, CPU_INTERRUPT_EXITTB);
++    qemu_mutex_unlock_iothread();
+ 
+     /* Reset the reservation */
+     env->reserve_addr = -1;
+Index: qemu-2.10.1/target/ppc/helper_regs.h
+===================================================================
+--- qemu-2.10.1.orig/target/ppc/helper_regs.h
++++ qemu-2.10.1/target/ppc/helper_regs.h
+@@ -114,11 +114,15 @@ static inline int hreg_store_msr(CPUPPCS
+     }
+     if (((value >> MSR_IR) & 1) != msr_ir ||
+         ((value >> MSR_DR) & 1) != msr_dr) {
+-        cs->interrupt_request |= CPU_INTERRUPT_EXITTB;
++        qemu_mutex_lock_iothread();
++        cpu_interrupt(cs, CPU_INTERRUPT_EXITTB);
++        qemu_mutex_unlock_iothread();
+     }
+     if ((env->mmu_model & POWERPC_MMU_BOOKE) &&
+         ((value >> MSR_GS) & 1) != msr_gs) {
+-        cs->interrupt_request |= CPU_INTERRUPT_EXITTB;
++        qemu_mutex_lock_iothread();
++        cpu_interrupt(cs, CPU_INTERRUPT_EXITTB);
++        qemu_mutex_unlock_iothread();
+     }
+     if (unlikely((env->flags & POWERPC_FLAG_TGPR) &&
+                  ((value ^ env->msr) & (1 << MSR_TGPR)))) {
diff --git a/meta/recipes-devtools/qemu/qemu_2.10.1.bb b/meta/recipes-devtools/qemu/qemu_2.10.1.bb
index 6e9b68b..8e3ca3c 100644
--- a/meta/recipes-devtools/qemu/qemu_2.10.1.bb
+++ b/meta/recipes-devtools/qemu/qemu_2.10.1.bb
@@ -24,6 +24,7 @@ SRC_URI = "http://wiki.qemu-project.org/download/${BP}.tar.bz2 \
            file://0003-Introduce-condition-in-TPM-backend-for-notification.patch \
            file://0004-Add-support-for-VM-suspend-resume-for-TPM-TIS-v2.9.patch \
            file://apic-fixup-fallthrough-to-PIC.patch \
+           file://ppc_locking.patch \
            "
 UPSTREAM_CHECK_REGEX = "qemu-(?P<pver>\d+\..*)\.tar"
 
-- 
2.7.4

[PATCH 3/6] classes/cross: Add addto_recipe_sysroot task to cross recipes

[Richard Purdie]

This is particularly useful if you want to use gdb-cross as there
is no other good way to access it now with RSS.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/cross.bbclass | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/meta/classes/cross.bbclass b/meta/classes/cross.bbclass
index d217717..e9fafed 100644
--- a/meta/classes/cross.bbclass
+++ b/meta/classes/cross.bbclass
@@ -92,3 +92,8 @@ export STRIP = "${BUILD_STRIP}"
 export NM = "${BUILD_NM}"
 
 inherit nopackages
+
+python do_addto_recipe_sysroot () {
+    bb.build.exec_func("extend_recipe_sysroot", d)
+}
+addtask addto_recipe_sysroot after do_populate_sysroot
-- 
2.7.4

[PATCH 4/6] runqemu: Improve relative path handling in qemuconf files

[Richard Purdie]

If a variable starts with "../", its likely its a path and we want to
set it to an absolute path relative to the qemuconf file.

This means we don't have to use bitbake as often to figure out variables.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 scripts/runqemu | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/scripts/runqemu b/scripts/runqemu
index ad88da3..b09cf1c 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -726,6 +726,8 @@ class BaseConfig(object):
         cf.read(self.qemuboot)
         for k, v in cf.items('config_bsp'):
             k_upper = k.upper()
+            if v.startswith("../"):
+                v = os.path.abspath(os.path.dirname(self.qemuboot) + "/" + v)
             self.set(k_upper, v)
 
     def validate_paths(self):
-- 
2.7.4

[PATCH 5/6] qemuboot: Improve relative path handling

[Richard Purdie]

qemuconf files are currently written relative to TOPDIR. What
makes more sense is to write paths relative to the location of the
file. This makes moving them around and decoding the end paths in
runqemu much easier.

The effect of this should allow less use of bitbake to determine
variables and allow us to simplify runqemu.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/qemuboot.bbclass | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/meta/classes/qemuboot.bbclass b/meta/classes/qemuboot.bbclass
index 7243cc5..15a9e63 100644
--- a/meta/classes/qemuboot.bbclass
+++ b/meta/classes/qemuboot.bbclass
@@ -85,7 +85,8 @@ python do_write_qemuboot_conf() {
 
     qemuboot = "%s/%s.qemuboot.conf" % (d.getVar('IMGDEPLOYDIR'), d.getVar('IMAGE_NAME'))
     qemuboot_link = "%s/%s.qemuboot.conf" % (d.getVar('IMGDEPLOYDIR'), d.getVar('IMAGE_LINK_NAME'))
-    topdir="%s/"%(d.getVar('TOPDIR')).replace("//","/")
+    finalpath = d.getVar("DEPLOY_DIR_IMAGE")
+    topdir = d.getVar('TOPDIR')
     cf = configparser.ConfigParser()
     cf.add_section('config_bsp')
     for k in sorted(qemuboot_vars(d)):
@@ -98,7 +99,8 @@ python do_write_qemuboot_conf() {
             val = d.getVar(k)
         # we only want to write out relative paths so that we can relocate images
         # and still run them
-        val=val.replace(topdir,"")
+        if val.startswith(topdir):
+            val = os.path.relpath(val, finalpath)
         cf.set('config_bsp', k, '%s' % val)
 
     # QB_DEFAULT_KERNEL's value of KERNEL_IMAGETYPE is the name of a symlink
@@ -108,7 +110,7 @@ python do_write_qemuboot_conf() {
     kernel = os.path.realpath(kernel_link)
     # we only want to write out relative paths so that we can relocate images
     # and still run them
-    kernel=kernel.replace(topdir,"")
+    kernel = os.path.relpath(kernel, finalpath)
     cf.set('config_bsp', 'QB_DEFAULT_KERNEL', kernel)
 
     bb.utils.mkdirhier(os.path.dirname(qemuboot))
-- 
2.7.4

[PATCH 6/6] qemurunner: Simplify binary data handling

[Richard Purdie]

I have concerns that bad timing of the flow of data from the logger
might corrupt the output due to the way binary strings are handled
in qemurunner.

This simplifies the code to do the same thing it did before but much
more safely.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/lib/oeqa/utils/qemurunner.py | 20 ++++++++------------
 1 file changed, 8 insertions(+), 12 deletions(-)

diff --git a/meta/lib/oeqa/utils/qemurunner.py b/meta/lib/oeqa/utils/qemurunner.py
index 82335d8..0631d43 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -275,7 +275,7 @@ class QemuRunner:
         reachedlogin = False
         stopread = False
         qemusock = None
-        bootlog = ''
+        bootlog = b''
         data = b''
         while time.time() < endtime and not stopread:
             try:
@@ -292,17 +292,13 @@ class QemuRunner:
                 else:
                     data = data + sock.recv(1024)
                     if data:
-                        try:
-                            data = data.decode("utf-8", errors="surrogateescape")
-                            bootlog += data
-                            data = b''
-                            if re.search(".* login:", bootlog):
-                                self.server_socket = qemusock
-                                stopread = True
-                                reachedlogin = True
-                                self.logger.debug("Reached login banner")
-                        except UnicodeDecodeError:
-                            continue
+                        bootlog += data
+                        data = b''
+                        if b' login:' in bootlog:
+                            self.server_socket = qemusock
+                            stopread = True
+                            reachedlogin = True
+                            self.logger.debug("Reached login banner")
                     else:
                         socklist.remove(sock)
                         sock.close()
-- 
2.7.4

✗ patchtest: failure for "runqemu: Ensure we process all..." and 5 more

[Patchwork]

== Series Details ==

Series: "runqemu: Ensure we process all..." and 5 more
Revision: 1
URL   : https://patchwork.openembedded.org/series/9898/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
  Suggested fix    Rebase your series on top of targeted branch
  Targeted branch  master (currently at 1e87283e92)

* Issue             A patch file has been added, but does not have a Signed-off-by tag [test_signed_off_by_presence] 
  Suggested fix    Sign off the added patch file (meta/recipes-devtools/qemu/qemu/ppc_locking.patch)



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe