[U-Boot] [PATCH V2 1/7] test/py: Implement pytest infrastructure

[Stephen Warren <swarren@wwwdotorg.org>]

On 12/16/2015 11:32 AM, Michal Simek wrote:
>
>
> 2015-12-16 19:09 GMT+01:00 Stephen Warren <swarren@wwwdotorg.org
> <mailto:swarren@wwwdotorg.org>>:
>
>     On 12/16/2015 10:43 AM, Michal Simek wrote:
>
>         Hi Stephen,
>
>         2015-12-16 17:27 GMT+01:00 Stephen Warren <swarren@wwwdotorg.org
>         <mailto:swarren@wwwdotorg.org>
>         <mailto:swarren at wwwdotorg.org <mailto:swarren@wwwdotorg.org>>>:
>
>
>              On 12/16/2015 08:11 AM, Michal Simek wrote:
>
>                  On 9.12.2015 17:32, Stephen Warren wrote:
>
>                      On 12/02/2015 03:18 PM, Stephen Warren wrote:
>
>                          This tool aims to test U-Boot by executing
>         U-Boot shell
>                          commands using
>                          the
>                          console interface. A single top-level script
>         exists to
>                          execute or attach
>                          to the U-Boot console, run the entire script of
>         tests
>                          against it, and
>                          summarize the results. Advantages of this
>         approach are:
>
>                          - Testing is performed in the same way a user
>         or script
>                          would interact
>                               with U-Boot; there can be no disconnect.
>                          - There is no need to write or embed
>         test-related code
>                          into U-Boot
>                          itself.
>                               It is asserted that writing test-related
>         code in
>                          Python is simpler and
>                               more flexible that writing it all in C.
>                          - It is reasonably simple to interact with
>         U-Boot in
>                          this way.
>
>                          A few simple tests are provided as examples.
>         Soon, we
>                          should convert as
>                          many as possible of the other tests in test/* and
>                          test/cmd_ut.c too.
>
>                          In the future, I hope to publish (out-of-tree)
>         the hook
>                          scripts, relay
>                          control utilities, and udev rules I will use
>         for my own
>                          HW setup.
>
>
>                      I finally got permission to publish these. Examples
>         are at:
>         https://github.com/swarren/uboot-test-hooks
>
>
>                  Interesting. What's the normal setup which you have for
>         the board?
>                  I see from your description that you use numato usb
>         relay - I
>                  expect one
>                  with more channels for reset.
>                  Then you are able to control boot mode. Is it also via
>         the same
>                  relay?
>                  How do you power up the board?
>
>
>              In my current setup I leave the board on all the time (or
>         rather,
>              manually turn on the power when I'm about to run the tests).
>              Automating control of the power source is a step I'll take
>         later.
>
>
>         ok.
>
>
>              For Tegra, there are two important signals: reset and
>         "force recovery".
>
>
>
>         Do you mean that these both signals are just connected out of chip?
>
>
>     Yes. Reset is typically driven into the PMIC, and the signal to
>     request force recovery is driven into Tegra itself.
>
>     Typically there are push-buttons on development boards to control
>     those two signals. I've simply wired my relays across those buttons
>     to simulate the button press.
>
>
> ok I see.
>
>
>              Each of these has a separate relay, so the system currently
>         uses 2
>              relays per target board. The numato relay board I own has 8
>         relays,
>              although there are a number of different models.
>
>
>         ok
>
>
>              On Tegra, when reset is pulsed:
>
>              - If force-recovery is connected, the SoC enters USB
>         recovery mode.
>              In this state, SW can be downloaded over USB into RAM and
>         executed.
>
>
>         Is this bootrom feature?
>
>
>     Yes.
>
>         For xilinx boards there is all the time jtag available. It means
>         download can be done via jtag instead.
>
>
>     That sounds plausible. The only issue might be general system state;
>     can you reset everything to POR defaults via JTAG before the download?
>
>
>
> There is cpu reset, Soc reset on the board which can be used but I have
> IP power switch. It means I can handle power which ensure correct state.
>
>     If not, perhaps e.g. the eMMC controller was partially initialized
>     by previous code, which might interfere with assumptions made by the
>     new code that's downloaded?
>
>
> I think my power switch solves this without any problem.
>
>
>              - If force-recovery is not connected, the SoC boots
>         normally, from
>              SW stored in flash (eMMC, SPI, ...)
>
>
>              The example scripts always use recovery mode to download
>         U-Boot into
>              RAM rather than writing it to flash first and then
>         resetting. This
>              saves wear cycles on the flash, but does mean the download
>         happens
>              in the "reset" rather than "flash" script, which may make the
>              examples a bit different than for some other SoCs.
>
>
>         Are you testing all boot modes? Because I expect these needs to be
>         tested too. Do you use SPL? If yes, are you going to test it in
>         this way?
>
>
>     With those example scripts, cold boot isn't being tested. However,
>     (a) I could define a new board ID (or pick up environment variables)
>     to cause that to be tested sometimes (b) I don't recall having seen
>     any differences between cold boot and recovery mode boot in the
>     past; we get a lot of quicker/lower-wear test coverage this way
>     without too much additional risk.
>
>
> ok.
>
>
>     SPL is in use. However, SPL on Tegra has a bit of a different job
>     than it has on some other chips. The boot ROM always initializes
>     SDRAM, and SPL actually runs on a different CPU and primarily has
>     the job of booting the main CPU where the main U-Boot binary runs.
>     For more information, see:
>
>     ftp://download.nvidia.com/tegra-public-appnotes/index.html
>
>
> Interesting.
>
>
>              Finally, the example scripts support two boards; my
>         home/laptop dev
>              setup that uses a Numato relay board to control the signals
>         to the
>              board I use there, and my work desktop dev setup that uses our
>              "PM342" debug board to controll the signals. The latter works
>              logically the same as the numato relay board, except it
>         contains
>              electronic switches driven by an FTDI chip.
>
>         I expect this is FTDI chip on the target right?
>
>
>     It's actually a separate common debug board. Most/all of our
>     development boards (and perhaps some production boards) have a
>     standardized connector into which the common debug board plugs.
>
>
>
> ok.
> I think my setup is not that far from what you are using and I expect
> that others SoCs will be very similar.
> Do you have any other testcases which you are running and you haven't sent?

Not at present.

As an FYI, I typically publish my local work-in-progress branch at:
git://github.com/swarren/u-boot.git tegra_dev