[Buildroot] [PATCH v2 1/3] support/testing/infra/emulator.py: change the shell prompt before running tests

* [Buildroot] [PATCH v2 1/3] support/testing/infra/emulator.py: change the shell prompt before running tests
@ 2023-06-11 10:45 Julien Olivain
  2023-06-26 20:00 ` [Buildroot] [PATCH v2 2/3] support/testing/tests/package/test_dieharder.py: new runtime test Julien Olivain
                   ` (2 more replies)
  0 siblings, 3 replies; 6+ messages in thread
From: Julien Olivain @ 2023-06-11 10:45 UTC (permalink / raw)
  To: buildroot; +Cc: Julien Olivain, Ricardo Martincoski

If a program has the string '# ' (i.e. the default shell prompt) in
its output, the test execution in the Buildroot runtime test infra is
failing.

This can be reproduced by adding a single line in a package test
script:

    self.assertRunOk("echo ### this is a string with hashes ###")

When executed with, for example, the command:

    support/testing/run-tests \
        -d dl -o output_folder tests.package.test_sometest

The test will fail with output:

    Traceback (most recent call last):
      File "/buildroot/support/testing/tests/package/test_sometest.py", line 20, in test_run
        self.assertRunOk("echo ### this is a string with hashes ###")
      File "/buildroot/support/testing/infra/basetest.py", line 94, in assertRunOk
        out, exit_code = self.emulator.run(cmd, timeout)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/buildroot/support/testing/infra/emulator.py", line 121, in run
        exit_code = int(exit_code)
                    ^^^^^^^^^^^^^^
    ValueError: invalid literal for int() with base 10: ''

This issue has been seen with real softwares, for example while
writing tests for the dieharder package, which creates report tables
formatted with hash characters. See:
https://github.com/eddelbuettel/dieharder/blob/3.31.1.4/libdieharder/version.c#L34
See also the output produced by a program invocation with command
"dieharder -d 0". This issue has also been seen with dmidecode,
producing comments in its output (when simply invoked with
"dmidecode"). See:
https://git.savannah.gnu.org/cgit/dmidecode.git/tree/dmioutput.c?h=dmidecode-3-5#n30

This issue could technically be seen with program producing MarkDown
formatted output (for example "# Header 1").

This issue happen because the test infra emulator.run() method expect
the prompt "# " after running a command and while getting the return
code. See:
https://git.buildroot.org/buildroot/tree/support/testing/infra/emulator.py?h=2023.05#n113

Since the string "# " is quite common, this patch changes the prompt
after the emulator.login(), by setting the PS1 variable to a string
which is less likely to appear in a normal program output. A small
caveat: since there is a command echo, the command setting the new
prompt needs to be protected to make sure it will not be detected as
an actual shell prompt. The prompt is encoded by single-quoting
each character (e.g. abc -> 'a''b''c').

Signed-off-by: Julien Olivain <ju.o@free.fr>
---
Changes v1 -> v2:
- reworded commit log, to mention this issue was also seen while writing
  a test for the dmidecode package
- the patch series also introduce the new test for dmidecode
---
 support/testing/infra/emulator.py | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/support/testing/infra/emulator.py b/support/testing/infra/emulator.py
index 02cf486128..390c582e9d 100644
--- a/support/testing/infra/emulator.py
+++ b/support/testing/infra/emulator.py
@@ -13,6 +13,7 @@ class Emulator(object):
         # can take a long time to run the emulator. Use a timeout multiplier
         # when running the tests to avoid sporadic failures.
         self.timeout_multiplier = timeout_multiplier
+        self.shell_prompt = "#BRTEST# "
 
     # Start Qemu to boot the system
     #
@@ -100,6 +101,15 @@ class Emulator(object):
         index = self.qemu.expect(["# ", pexpect.TIMEOUT])
         if index != 0:
             raise SystemError("Cannot login")
+        # Set a special shell prompt while testing. Since the standard
+        # prompt '# ' is quite generic, a normal process output could
+        # contain that string and confuse expect. When changing the
+        # prompt, we also need to encode or escape it in some way to
+        # make sure the command echo will not be seen as a prompt
+        # itself. The prompt is encoded by single-quoting each
+        # character (e.g. abc -> 'a''b''c').
+        encoded_prompt = "''".join(self.shell_prompt)
+        self.run("export PS1='{}'".format(encoded_prompt))
         self.run("dmesg -n 1")
         # Prevent the shell from wrapping the commands at 80 columns.
         self.run("stty columns 29999")
@@ -110,13 +120,13 @@ class Emulator(object):
         self.qemu.sendline(cmd)
         if timeout != -1:
             timeout *= self.timeout_multiplier
-        self.qemu.expect("# ", timeout=timeout)
+        self.qemu.expect(self.shell_prompt, timeout=timeout)
         # Remove double carriage return from qemu stdout so str.splitlines()
         # works as expected.
         output = self.qemu.before.replace("\r\r", "\r").splitlines()[1:]
 
         self.qemu.sendline("echo $?")
-        self.qemu.expect("# ")
+        self.qemu.expect(self.shell_prompt)
         exit_code = self.qemu.before.splitlines()[2]
         exit_code = int(exit_code)
 
-- 
2.41.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

^ permalink raw reply related	[flat|nested] 6+ messages in thread