From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jeremy Rosen Date: Wed, 1 Jul 2015 09:57:15 +0200 (CEST) Subject: [Buildroot] Buildroot runtime test infrastructure prototype In-Reply-To: <5593970D.3000707@andin.de> Message-ID: <239070435.2387566.1435737435416.JavaMail.root@openwide.fr> List-Id: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: buildroot@busybox.net ----- Mail original ----- > Hi Jeremy, > > Am 30.06.2015 um 18:06 schrieb Jeremy Rosen: > > >>> Our approch was mainly to have each package (on the buildroot > >>> side > >>> have its own RFW scriptlet and have buildroot assemble these > >>> scriptlets at make time to provide a test for the resulting > >>> system > >>> that would only contain the tests for whatever package was > >>> actually > >>> compiled in. > >> > >> At some point I indeed though of having some per-package test > >> cases > >> directly in package//, for each package. But you also anyway > >> need > >> to describe a complete Buildroot configuration for each test, in > >> order > >> to have a complete system that actually boots and make sense. > >> > > > > yes and no... your approch is targeting the autobuild system, I was > > more > > thinking of providing tests for a user's own configuration or a > > semi-randomly generated config. Easily testing on real hardware was > > also part of the idea. > > > > If we want to combine the two, there would be a need for a base, > > bootable > > config (for example based on qemu) with a randomized overlay. > > > > The per-package part would help mainly with the randomly added > > packages. > > That would not prevent to also have a package independant base of > > test > > for the most basic functionalities. Per-Package also has the > > advantage > > that the tests will be maintained with the packages, thus by the > > people > > with the knowhow of what is being tested. > > > > overall I don't really see a reason not to have both approch. they > > are > > not really exclusive... > > > > now, one of the big difference is that you have host tests and > > target > > tests intermixed. That is a bit problematic if one wants to use > > your > > framework on a real board since the part responsible for starting > > the card and login is intermixed with the test. Again, my use-case > > is > > slightly different from yours. Having host-side tests is > > interesting > > and something we hadn't thought of. It should not be very hard to > > add > > with a RFW approch if we go in that direction > > I agree with most things above and indeed, what to test is not > exactly > that same in our approach and buildroots. But I think it's very > possible > and beneficial to combine the efforts. > E.g. the deploy routines for our boards differ quite a bit. We solved > this by implementing the same keyword in different (target-specific) > resources, so the testcases using it doesnt need to change. When > running > the tests, we supply a DUT-specific variable file which then takes > care > of importing the correct resources with the applicable deploy step > (and > setting HW capabilities, and so on). > your RFW/buildroot integration seems more advanced than ours at this point, is it possible to share it ? > > > > separating the target and host test (whatever approch is taken) > > seems > > like a good idea to me, a way to override/disable the boot/login > > part > > would allow users to run your tests on real hardware. That seems > > like > > a good idea independantly of the framework used. > > This is something that can easily be done via tags and then just run > --include host* or --include target* > > > >> I've released my code with the intent that others can have a look > >> and > >> get a clear view of what the prototype looks like. Without seeing > >> how > >> it looks with RFW, I can't really make up my mind on whether it is > >> a > >> good alternate solution or not. > >> > > > > Denis reimplemented an infrastructure to redo your tests. The > > complete > > infrastructure can be found here : > > > > https://github.com/etkaDT/Rfw-buildroot-tests > > > > Note that this is still a quick and dirty job that is not > > integrated > > into buildroot, but it can start the custom qemu to test things on > > the target. A better integration could lead to easier keywords etc. > > > > Denis also included the output of running the python package test > > in > > the output/ subdirectory. The RFW specific XML is quite interesting > > because it contains the complete decomposition of keywords and how > > they failed (there is a failed test in the example) > > > > (Denis is the Open Wide trainee working on buildroot this summer, > > he is the one pushing the scanpypy patch and the related > > robotframework package) > > > > We hope this helps with the overall debate. > > Well, great to have another RFW implementation to look at and learn! > Remarkable also that you guys implemented the qemu part while I did > the > building part without even talking about it :) > One obvious difference is that you create a large number of small > almost > atomic testcases while my testcase contain more of the steps that you > probably would put into the prepare step. But I like the much more > detailed report this approach leads to. Since RFW provides only for > one > keyword in the prepare step, this probably leads to having one > prepare-keyword for almost every suite. > So far I'm using a global keyword 'Connect And Login' in all my > suites, > but it may be helpful to change that. > > Btw, what Editor did you use for the creation? > I'll let Denis answer that one :) > > regards, > Andreas > > > > > > > > >> Best regards, > >> > >> Thomas > >> -- > >> Thomas Petazzoni, CTO, Free Electrons > >> Embedded Linux, Kernel and Android engineering > >> http://free-electrons.com > >> > > >