From mboxrd@z Thu Jan  1 00:00:00 1970
From: Curt Brune <curt@cumulusnetworks.com>
Date: Wed, 9 Oct 2013 09:22:56 -0700
Subject: [U-Boot] U-Boot Regression Testing
Message-ID: <20131009162255.GG31348@cumulusnetworks.com>
List-Id: <u-boot.lists.denx.de>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
To: u-boot@lists.denx.de

Hello,

I have some questions about how U-Boot regression testing works.  I am
assuming some regression testing happens during the release period
across some representative sample of boards and architectures.

I know people check for compilation failures with MAKEALL, but I am
wondering about runtime testing.  I also understand that testing
platform specific boot code is rather difficult (or easy depending on
your perspective) - it boots or it does not boot.

To be concrete -- how are core U-Boot commands and features tested?
For example how do you test that FIT image support is not broken or
that the 'env' command and all its options work properly?

Googling did not turn up much on how this is done.

On the social side -- is that something the community helps out with
or something DENX does, or a mix?

Are you using a test framework of some kind, either home grown or open
source?  These things tend to become home grown over time :)

This kind of testing usually takes the form of 'chat' scripts
communicating over serial consoles.  Perhaps you are using expect,
pexpect, python nose?

We have a project, of which U-Boot is a part, that is starting to span
multiple boards and architectures.  We make a few modifications to
U-Boot and I want to start automated regression testing as the number
of boards increases.

If an existing framework exists that folks are happy with I would love
to hear about it.  Equally, I am interested to hear about what did
*not* work for people.  Whatever method our project uses the scripts
will be publicly available.

Cheers,
Curt