[U-Boot] RFC: Testing U-Boot Part 1

[Marek Vasut <marek.vasut@gmail.com>]

On Thursday, August 25, 2011 02:58:00 PM Simon Glass wrote:
> Hi,
> 
> Summary: I am quite keen on improving the test infrastructure in
> U-Boot. I would like to have a test suite that can run in a minute or
> two on a Linux PC and test all non-platform code.
> 
> Detail
> ======
> We can break the U-Boot code base into two parts:
> 
> 1. Platform code, which is SOC-specifc. At present this is the CPU
> init (arch/xxx/cpu/...) and SOC-specific drivers (mostly in the
> drivers directory). There is also a small amount of generic CPU code
> in arch/xxx/lib and some board-specific code/drivers (e.g. drivers not
> within the SOC).
> 
> 2. Portable/Generic U-Boot code, which is cross-platform. This
> includes many drivers, the various commands, file system support,
> things like the MMC, USB and network stacks, etc.
> 
> My focus in this email the the second part of the code base - all the
> code which is not platform-specific, but can still have bugs.
> 
> Proposal
> ========
> To a large extent, testing of this part of the code base could simply
> be built on one or more of the available platforms. We could then run
> the tests on that platform, and announce that the code base works fine
> on that platform. Obviously the platform needs to support all possible
> features of U-Boot.
> 
> However, this approach fails to take advantage of two useful
> properties of this code:
> 
> - It is cross-platform, and even runs on x86
> - It is standalone, requiring no OS libraries to run
> 
> For speed, debugging and convenience, it would be nice to run U-Boot
> under a generic Linux environment on a workstation, and test all the
> generic non-platform code. The basic problem with this is that the
> non-platform code requires the platform code to operate. Even the x86
> platform code is designed for standalone operation on a particular x86
> board, and is not suitable for running under x86 Linux.
> 
> To get around this I propose that we create a new ?native?
> architecture. We write code in ?arch/native? which can run under
> Linux. Since all the non-platform code will happily run under this new
> ?architecture?, we can then write tests which run quickly under x86
> Linux (or another Linux for that matter). This U-Boot 'architecture'
> should build natively on any 32/64-bit Linux machine since it just
> uses standard Linux system calls. Calls to Linux would be entirely
> within this arch/native subdirectory.
> 
> Benefit
> =======
> What will this buy us? Well we can do things like:
> 
> - Create a test SPI flash device, which is file-backed. Use this to
> write a test of the cmd_sf layer by issuing a series of commands and
> making sure that the correct SPI flash contents is obtained
> 
> - Create a test MMC or IDE device, backed by a file. Use this to issue
> ext2 and fat commands to manipulate the filesystem. Then loopback
> mount the file and check from Linux that U-Boot did the right thing
> 
> - Create a test Ethernet device with a mocked remote host attached.
> Use this to issue network commands like bootp and tftp and check that
> the correct results are obtained.
> 
> Ultimately (one day) we might have a set of unit tests for U-Boot
> which can run to very quickly check that a patch does not break the
> core U-Boot code.
> 
> Comments
> ========
> At this early stage I am looking for comments on the concept - how
> useful it is and how best to implement it.

Hi Simon, 

sounds just awesome. I was planning on doing a similar thing, but "from the 
other side". I'm sure you're aware of the POST test framework. I was planning to 
either extend it or write new self-standing implementation that'd allow debuging 
platform-dependent drivers.

I can CC you on that if you have some interest.

> 
> Regards,
> Simon
> _______________________________________________
> U-Boot mailing list
> U-Boot at lists.denx.de
> http://lists.denx.de/mailman/listinfo/u-boot