From mboxrd@z Thu Jan 1 00:00:00 1970 From: Stephen Warren Date: Thu, 19 Nov 2015 10:00:32 -0700 Subject: [U-Boot] [PATCH] Implement pytest-based test infrastructure In-Reply-To: References: <1447570381-1361-1-git-send-email-swarren@wwwdotorg.org> Message-ID: <564E0030.9090808@wwwdotorg.org> List-Id: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: u-boot@lists.denx.de On 11/19/2015 07:45 AM, Simon Glass wrote: > Hi Stephen, > > On 14 November 2015 at 23:53, Stephen Warren wrote: >> This tool aims to test U-Boot by executing U-Boot shell commands using the >> console interface. A single top-level script exists to execute or attach >> to the U-Boot console, run the entire script of tests against it, and >> summarize the results. Advantages of this approach are: >> >> - Testing is performed in the same way a user or script would interact >> with U-Boot; there can be no disconnect. >> - There is no need to write or embed test-related code into U-Boot itself. >> It is asserted that writing test-related code in Python is simpler and >> more flexible that writing it all in C. >> - It is reasonably simple to interact with U-Boot in this way. >> >> A few simple tests are provided as examples. Soon, we should convert as >> many as possible of the other tests in test/* and test/cmd_ut.c too. > > It's great to see this and thank you for putting in the effort! > > It looks like a good way of doing functional tests. I still see a role > for unit tests and things like test/dm. But if we can arrange to call > all U-Boot tests (unit and functional) from one 'test.py' command that > would be a win. > > I'll look more when I can get it to work - see below. ... > I get this on my Ubuntu 64-bit machine (14.04.3) > > $ ./test/py/test.py --bd sandbox --buildTraceback (most recent call last): > File "./test/py/test.py", line 12, in > os.execvp("py.test", args) > File "/usr/lib/python2.7/os.py", line 344, in execvp > _execvpe(file, args) > File "/usr/lib/python2.7/os.py", line 380, in _execvpe > func(fullname, *argrest) > OSError: [Errno 2] No such file or directory "py.test" isn't in your $PATH. Did you install it? See the following in test/py/README.md: > ## Requirements > > The test suite is implemented using pytest. Interaction with the U-Boot > console uses pexpect. Interaction with real hardware uses the tools of your > choice; you get to implement various "hook" scripts that are called by the > test suite at the appropriate time. > > On Debian or Debian-like distributions, the following packages are required. > Similar package names should exist in other distributions. > > | Package | Version tested (Ubuntu 14.04) | > | -------------- | ----------------------------- | > | python | 2.7.5-5ubuntu3 | > | python-pytest | 2.5.1-1 | > | python-pexpect | 3.1-1ubuntu0.1 | In the main Python code, I trapped at least one exception location and made it print a message about checking the docs for missing requirements. I can probably patch the top-level test.py to do the same.