From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jeremy Rosen Date: Tue, 30 Jun 2015 18:06:05 +0200 (CEST) Subject: [Buildroot] Buildroot runtime test infrastructure prototype In-Reply-To: <20150628115001.098430d1@free-electrons.com> Message-ID: <1591189778.2367426.1435680365451.JavaMail.root@openwide.fr> List-Id: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: buildroot@busybox.net I am replying to an earlier mail of Thomas because there were unanswered questions in there. see below... ----- Mail original ----- > Jeremy, > > On Fri, 26 Jun 2015 18:20:33 +0200 (CEST), Jeremy Rosen wrote: > > > more answers in the mail below, but we have a trainee working on an > > automated test framework based on "Robot Framework" this summer and > > we planned to upstream it when we have something to show... > > In fact, when I started working on this runtime test infrastructure, > I > also posted on Google Plus about the available frameworks, and one of > the answers was Robot Framework. So I had a quick look, and > definitely > could not understand how it can work: you are apparently not writing > the tests in some real programming language, but in some sort of > weird > "tabular" format. I really don't understand how you can express > complicated test scenarios with such a limited language. > There is the possibility to do loops in RFW (see the "continue for loop" and related keywords in the builtin library) but complicated things are usually implemented as python modules (or other languages, but mainly python). Note that the philosophy of RFW is to implement building blocks as libraries and assamble them into actual tests using the tabular language. Tests suites tend to be lots of very similar tests with slight variation and the tabular language is designed to make that kind of tests easy to read an write The existing libraries are pretty handy to implement all the common tasks of system testing, like starting/stoping/checking processes, opening network connections and crafting packets, using a SSH collection, dealing with XML to talk to an application etc... Python can do that too, of course (RFW is coded in python after all) but RFW does all the heavy lifting for us. Using raw python or a lower level python framework would probably mean reinventing the wheel in that regard... The point is that the high level language is very readable, esp for non- coders, which allows autogenerated documentation of test. > The Python unittest stuff just runs Python code, so you can express > whatever complicated test logic you want. > > > Our approch was mainly to have each package (on the buildroot side > > have its own RFW scriptlet and have buildroot assemble these > > scriptlets at make time to provide a test for the resulting system > > that would only contain the tests for whatever package was actually > > compiled in. > > At some point I indeed though of having some per-package test cases > directly in package//, for each package. But you also anyway > need > to describe a complete Buildroot configuration for each test, in > order > to have a complete system that actually boots and make sense. > yes and no... your approch is targeting the autobuild system, I was more thinking of providing tests for a user's own configuration or a semi-randomly generated config. Easily testing on real hardware was also part of the idea. If we want to combine the two, there would be a need for a base, bootable config (for example based on qemu) with a randomized overlay. The per-package part would help mainly with the randomly added packages. That would not prevent to also have a package independant base of test for the most basic functionalities. Per-Package also has the advantage that the tests will be maintained with the packages, thus by the people with the knowhow of what is being tested. overall I don't really see a reason not to have both approch. they are not really exclusive... now, one of the big difference is that you have host tests and target tests intermixed. That is a bit problematic if one wants to use your framework on a real board since the part responsible for starting the card and login is intermixed with the test. Again, my use-case is slightly different from yours. Having host-side tests is interesting and something we hadn't thought of. It should not be very hard to add with a RFW approch if we go in that direction separating the target and host test (whatever approch is taken) seems like a good idea to me, a way to override/disable the boot/login part would allow users to run your tests on real hardware. That seems like a good idea independantly of the framework used. > And in fact, testing packages is not the only target of this "runtime > test infrastructure". I also want to validate core features like > rootfs > overlay, post-build script, users/permission/device table, the myriad > possibilities of Linux kernel building, global patch directories, > etc. > Ok, so you want to test not just the image generated by buildroot but also buildroot itself. That is different than what we have in mind, but I can see the need for that. A complete framework would have to deal with both. Again, I think it is important to separate target testing from host/buildroot testing. The former can be qemu but might also be real hardware. In the case of real hardware buildroot can't guess how to upload and boot the image, but there is still things to do. Either by sending the tests to the target or by running them through a remote connection (serial or ssh) > > > At this point it would be possible for us to switch back to any > > test > > framework if RFW is not the one the buildroot project wants to use > > but I personally think that RFW is really a good framework for high > > level system testing (opposed to single-software unit testing) it > > is very simple to learn by looking at examples, the keyword approch > > is very readable and adding our own modules in python is fairly > > easy > > Do you have examples on what the RFW test cases for Buildroot look > like? > > I've released my code with the intent that others can have a look and > get a clear view of what the prototype looks like. Without seeing how > it looks with RFW, I can't really make up my mind on whether it is a > good alternate solution or not. > Denis reimplemented an infrastructure to redo your tests. The complete infrastructure can be found here : https://github.com/etkaDT/Rfw-buildroot-tests Note that this is still a quick and dirty job that is not integrated into buildroot, but it can start the custom qemu to test things on the target. A better integration could lead to easier keywords etc. Denis also included the output of running the python package test in the output/ subdirectory. The RFW specific XML is quite interesting because it contains the complete decomposition of keywords and how they failed (there is a failed test in the example) (Denis is the Open Wide trainee working on buildroot this summer, he is the one pushing the scanpypy patch and the related robotframework package) We hope this helps with the overall debate. > Best regards, > > Thomas > -- > Thomas Petazzoni, CTO, Free Electrons > Embedded Linux, Kernel and Android engineering > http://free-electrons.com >