[LTP] q-rig: qemu ltprun on armhf and arm64

[Cyril Hrubis <chrubis@suse.cz>]

Hi!
> I've been toying with Travis CI, qemu and LTP recently. My
> effort is here: https://github.com/q-rig. It runs LTP on top
> of the latest linux -rc on Debian sid, inside qemu on armhf and
> arm64. Binaries are built when needed. You can find them here:
> https://github.com/q-rig/qemu-ltprun-results.
> 
> Activities are run in Travis CI [0]. LTP tests are split so to finish
> in less than 49 minutes (Travis CI constraint). The LPT results are
> put again in https://github.com/q-rig/qemu-ltprun-results, which is
> probably the wrong place.

Interesting work, although I'm not sure that travis is right tool for
the job. On the other hand I've been thinking of similar misuse as
well...

> Now I'd like to tackle the aggregation of the data generated by LTP. The
> best I could do is just generating also the .html page and upload it
> to github [1] but I think more is actually desirable here.

I would like to be able to do more with the LTP results as well, for
instance compare two testruns for the purpose of automatic pre-release
testing which would allow us to do much shorter release cycles.

Unfortunately the best the legacy LTP runltp script + ltp-pan can do is
to write a list of failed and skipped tests, it also can't run the open
posix testcases, cannot kill tests on timeout, etc.

This are the reasons I've started to experiment with writing a new test
runner that would allow for writing different output formats among
others some kind of machine parseable output (I was thinking of json but
anything structured would do) which would allow us to produce much more
interesting output than the current html page. Proof of concet is done
in my private GitHub repo[1], but it's not complete enough to be usable
and it can produce only html[2] page as an output so far, but hacking in
json formatter would be fairly easy.

[1] https://github.com/metan-ucw/ltp
    https://github.com/metan-ucw/ltp/tree/master/tools/runltp-ng

[2] http://metan.ucw.cz/outgoing/ltp-openposix.html
    http://metan.ucw.cz/outgoing/ltp-syscalls.html

    The latest version also includes hardware and sofware version but
    apart from that it's more or less it.

> Is there any documentation I'm expected to read before asking for
> support here? :)

There is probably none or close to that hence asking here is the best
course of action.

-- 
Cyril Hrubis
chrubis@suse.cz