[Buildroot] Various runtime test failures on Gitlab CI

[Buildroot] Various runtime test failures on Gitlab CI

[Thomas Petazzoni]

Ricardo, Fran?ois, Yegor, Yann,

The last Gitlab CI pipeline on master
(https://gitlab.com/buildroot.org/buildroot/pipelines/75533041/failures)
shows a number of failures in our runtime tests.

Here are the failing tests:

  tests.package.test_python_treq.TestPythonPy3Treq
  tests.package.test_python_treq.TestPythonPy2Treq
  tests.package.test_python_crossbar.TestPythonPy3Crossbar
  tests.package.test_perl_io_socket_ssl.TestPerlIOSocketSSL
  tests.package.test_lua_http.TestLuajitHttp
  tests.package.test_lua_http.TestLuaHttp
  tests.init.test_systemd.TestInitSystemSystemdRwFull

And now, the details of the build failures:

 - For tests.package.test_python_treq.TestPythonPy3Treq and
   tests.package.test_python_treq.TestPythonPy2Treq, we're getting:

Traceback (most recent call last):
  File "/builds/buildroot.org/buildroot/support/testing/tests/package/test_python.py", line 125, in test_run
    self.run_sample_scripts()
  File "/builds/buildroot.org/buildroot/support/testing/tests/package/test_python_treq.py", line 10, in run_sample_scripts
    self.assertIn("Connection refused", output[0])
AssertionError: 'Connection refused' not found in ":0: UserWarning: You do not have a working installation of the service_identity module: 'cannot import name 'verify_ip_address' from 'service_identity.pyopenssl' (/usr/lib/python3.7/site-packages/service_identity/pyopenssl.pyc)'.  Please install it from <https://pypi.python.org/pypi/service_identity> and make sure all of its dependencies are satisfied.  Without the service_identity module, Twisted can perform only rudimentary TLS client hostname verification.  Many valid certificate/hostname mappings may be rejected."

 - For tests.package.test_python_crossbar.TestPythonPy3Crossbar, we're
   getting

Traceback (most recent call last):
  File "/builds/buildroot.org/buildroot/support/testing/tests/package/test_python.py", line 125, in test_run
    self.run_sample_scripts()
  File "/builds/buildroot.org/buildroot/support/testing/tests/package/test_python.py", line 120, in run_sample_scripts
    self.assertEqual(exit_code, 0)
AssertionError: 1 != 0

 - For tests.package.test_perl_io_socket_ssl.TestPerlIOSocketSSL,
   tests.package.test_lua_http.TestLuajitHttp and
   tests.package.test_lua_http.TestLuaHttp, we're getting:

======================================================================
ERROR: test_run (tests.package.test_perl_io_socket_ssl.TestPerlIOSocketSSL)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builds/buildroot.org/buildroot/support/testing/tests/package/test_perl_io_socket_ssl.py", line 21, in test_run
    self.module_test("IO::Socket::SSL")
  File "/builds/buildroot.org/buildroot/support/testing/tests/package/test_perl.py", line 22, in module_test
    _, exit_code = self.emulator.run(cmd)
  File "/builds/buildroot.org/buildroot/support/testing/infra/emulator.py", line 109, in run
    self.qemu.expect("# ", timeout=timeout)
  File "/usr/lib/python2.7/dist-packages/pexpect/spawnbase.py", line 321, in expect
    timeout, searchwindowsize, async)
  File "/usr/lib/python2.7/dist-packages/pexpect/spawnbase.py", line 345, in expect_list
    return exp.expect_loop(timeout)
  File "/usr/lib/python2.7/dist-packages/pexpect/expect.py", line 107, in expect_loop
    return self.timeout(e)
  File "/usr/lib/python2.7/dist-packages/pexpect/expect.py", line 70, in timeout
    raise TIMEOUT(msg)
TIMEOUT: Timeout exceeded.


 - For tests.init.test_systemd.TestInitSystemSystemdRwFull, we're
   getting:

Traceback (most recent call last):
  File "/builds/buildroot.org/buildroot/support/testing/tests/init/test_systemd.py", line 157, in test_run
    self.start_emulator("ext2", "zImage", "vexpress-v2p-ca9")
  File "/builds/buildroot.org/buildroot/support/testing/tests/init/base.py", line 38, in start_emulator
    self.emulator.login()
  File "/builds/buildroot.org/buildroot/support/testing/infra/emulator.py", line 92, in login
    raise SystemError("System does not boot")
SystemError: System does not boot

Could you have a look and fix those issues ?

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] Various runtime test failures on Gitlab CI

[Yann E. MORIN]

Thomas, All,

On 2019-08-10 15:19 +0200, Thomas Petazzoni spake thusly:
> The last Gitlab CI pipeline on master
> (https://gitlab.com/buildroot.org/buildroot/pipelines/75533041/failures)
> shows a number of failures in our runtime tests.
[--SNIP--]
>  - For tests.init.test_systemd.TestInitSystemSystemdRwFull, we're
>    getting:
> 
> Traceback (most recent call last):
>   File "/builds/buildroot.org/buildroot/support/testing/tests/init/test_systemd.py", line 157, in test_run
>     self.start_emulator("ext2", "zImage", "vexpress-v2p-ca9")
>   File "/builds/buildroot.org/buildroot/support/testing/tests/init/base.py", line 38, in start_emulator
>     self.emulator.login()
>   File "/builds/buildroot.org/buildroot/support/testing/infra/emulator.py", line 92, in login
>     raise SystemError("System does not boot")
> SystemError: System does not boot

This one works OK here:

    $ ./support/testing/run-tests -o $(pwd)/runtests -k
    tests.init.test_systemd.TestInitSystemSystemdRwFull
    15:25:25 TestInitSystemSystemdRwFull              Starting
    15:25:26 TestInitSystemSystemdRwFull              Building
    15:37:55 TestInitSystemSystemdRwFull              Building done
    15:38:34 TestInitSystemSystemdRwFull              Cleaning up
    .
    ----------------------------------------------------------------------
    Ran 1 test in 788.838s

    OK

Excerpt of the run log:

    10009000.uart: ttyAMA0 at MMIO 0x10009000 (irq = 38, base_baud = 0) is a PL011 rev1
    console [ttyAMA0] enabled
    [...]
    [   ***] (2 of 2) A start job is running for /dev/ttyAMA0 (16s / 1min 31s)
    [...]
    [  OK  ] Found device /dev/ttyAMA0.
    [  OK  ] Started Serial Getty on ttyAMA0.

while the one on gitlab fails to find the serial:

    10009000.uart: ttyAMA0 at MMIO 0x10009000 (irq = 38, base_baud = 0) is a PL011 rev1
    console [ttyAMA0] enabled
    [...]
    [    **] (1 of 2) A start job is running for /dev/ttyAMA0 (1min 28s / 1min 30s)
    [...]
    [ TIME ] Timed out waiting for device /dev/ttyAMA0.
    [DEPEND] Dependency failed for Serial Getty on ttyAMA0.

So, it looks like on my machine, it took abouts 16s for the ttyAMA0
systemd slice to be up (wft? 16s?), while on the gitlab-ci it was not
found...

I'll investigate what makes it appear so late...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] Various runtime test failures on Gitlab CI

[Yann E. MORIN]

Thomas, All,

On 2019-08-10 15:52 +0200, Yann E. MORIN spake thusly:
> On 2019-08-10 15:19 +0200, Thomas Petazzoni spake thusly:
> > The last Gitlab CI pipeline on master
> > (https://gitlab.com/buildroot.org/buildroot/pipelines/75533041/failures)
> > shows a number of failures in our runtime tests.
> [--SNIP--]
> >  - For tests.init.test_systemd.TestInitSystemSystemdRwFull, we're
> >    getting:
> > 
> > Traceback (most recent call last):
> >   File "/builds/buildroot.org/buildroot/support/testing/tests/init/test_systemd.py", line 157, in test_run
> >     self.start_emulator("ext2", "zImage", "vexpress-v2p-ca9")
> >   File "/builds/buildroot.org/buildroot/support/testing/tests/init/base.py", line 38, in start_emulator
> >     self.emulator.login()
> >   File "/builds/buildroot.org/buildroot/support/testing/infra/emulator.py", line 92, in login
> >     raise SystemError("System does not boot")
> > SystemError: System does not boot
> 
> This one works OK here:
>     [  OK  ] Found device /dev/ttyAMA0.
>     [  OK  ] Started Serial Getty on ttyAMA0.

I was however able to reproduce a similar issue with another runtime
test, tests.init.test_systemd.TestInitSystemSystemdRwNetworkd which
failed because it too was unable to find ttyAMA0...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] Various runtime test failures on Gitlab CI

[Yann E. MORIN]

All,

On 2019-08-10 19:01 +0200, Yann E. MORIN spake thusly:
> On 2019-08-10 15:52 +0200, Yann E. MORIN spake thusly:
> > On 2019-08-10 15:19 +0200, Thomas Petazzoni spake thusly:
> > >  - For tests.init.test_systemd.TestInitSystemSystemdRwFull, we're
> > >    getting:
[--SNIP--]
> > > SystemError: System does not boot
> > This one works OK here:
[--SNIP--]
> I was however able to reproduce a similar issue with another runtime
> test, tests.init.test_systemd.TestInitSystemSystemdRwNetworkd which
> failed because it too was unable to find ttyAMA0...

And restarting it again no longer fails...

So, it really is a timing issue, it seems. Meh...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] Various runtime test failures on Gitlab CI

[Ricardo Martincoski]

Hello,

I will send a small series to fix the python-*, lua-* and perl-* tests.

On Sat, Aug 10, 2019 at 10:19 AM, Thomas Petazzoni wrote:

[snip]
>  - For tests.package.test_python_treq.TestPythonPy3Treq and
>    tests.package.test_python_treq.TestPythonPy2Treq, we're getting:
> 
> Traceback (most recent call last):
>   File "/builds/buildroot.org/buildroot/support/testing/tests/package/test_python.py", line 125, in test_run
>     self.run_sample_scripts()
>   File "/builds/buildroot.org/buildroot/support/testing/tests/package/test_python_treq.py", line 10, in run_sample_scripts
>     self.assertIn("Connection refused", output[0])
> AssertionError: 'Connection refused' not found in ":0: UserWarning: You do not have a working installation of the service_identity module: 'cannot import name 'verify_ip_address' from 'service_identity.pyopenssl' (/usr/lib/python3.7/site-packages/service_identity/pyopenssl.pyc)'.  Please install it from <https://pypi.python.org/pypi/service_identity> and make sure all of its dependencies are satisfied.  Without the service_identity module, Twisted can perform only rudimentary TLS client hostname verification.  Many valid certificate/hostname mappings may be rejected."

The new warning message from treq at runtime confuses the test. I will improve
the test code to ignore it.

>  - For tests.package.test_python_crossbar.TestPythonPy3Crossbar, we're
>    getting
> 
> Traceback (most recent call last):
>   File "/builds/buildroot.org/buildroot/support/testing/tests/package/test_python.py", line 125, in test_run
>     self.run_sample_scripts()
>   File "/builds/buildroot.org/buildroot/support/testing/tests/package/test_python.py", line 120, in run_sample_scripts
>     self.assertEqual(exit_code, 0)
> AssertionError: 1 != 0

python-autobahn now defaults to use msgpack when CPython is used. But it
also still supports umsgpack when the user explicitly selects it using an
environment variable.
python-crossbar selects python-u-msgpack on menuconfig.
I will change the test to explicitly use umsgpack.

>  - For tests.package.test_perl_io_socket_ssl.TestPerlIOSocketSSL,
>    tests.package.test_lua_http.TestLuajitHttp and
>    tests.package.test_lua_http.TestLuaHttp, we're getting:
> 
> ======================================================================
> ERROR: test_run (tests.package.test_perl_io_socket_ssl.TestPerlIOSocketSSL)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/builds/buildroot.org/buildroot/support/testing/tests/package/test_perl_io_socket_ssl.py", line 21, in test_run
>     self.module_test("IO::Socket::SSL")
>   File "/builds/buildroot.org/buildroot/support/testing/tests/package/test_perl.py", line 22, in module_test
>     _, exit_code = self.emulator.run(cmd)
>   File "/builds/buildroot.org/buildroot/support/testing/infra/emulator.py", line 109, in run
>     self.qemu.expect("# ", timeout=timeout)
>   File "/usr/lib/python2.7/dist-packages/pexpect/spawnbase.py", line 321, in expect
>     timeout, searchwindowsize, async)
>   File "/usr/lib/python2.7/dist-packages/pexpect/spawnbase.py", line 345, in expect_list
>     return exp.expect_loop(timeout)
>   File "/usr/lib/python2.7/dist-packages/pexpect/expect.py", line 107, in expect_loop
>     return self.timeout(e)
>   File "/usr/lib/python2.7/dist-packages/pexpect/expect.py", line 70, in timeout
>     raise TIMEOUT(msg)
> TIMEOUT: Timeout exceeded.

All of these need entropy and are currently using armv7 kernel. I will switch to
use armv5 builtin kernel that already provides entropy.


Regards,
Ricardo