[LTP] q-rig: qemu ltprun on armhf and arm64

[LTP] q-rig: qemu ltprun on armhf and arm64

[Domenico Andreoli]

Hi all,

  this is me, Domenico (cavokz on github.com, cavok at debian.org).

I've been toying with Travis CI, qemu and LTP recently. My
effort is here: https://github.com/q-rig. It runs LTP on top
of the latest linux -rc on Debian sid, inside qemu on armhf and
arm64. Binaries are built when needed. You can find them here:
https://github.com/q-rig/qemu-ltprun-results.

Activities are run in Travis CI [0]. LTP tests are split so to finish
in less than 49 minutes (Travis CI constraint). The LPT results are
put again in https://github.com/q-rig/qemu-ltprun-results, which is
probably the wrong place.

All this is quite brute force cpu cycles harvesting generously proved
by Travis CI and github.

Now I'd like to tackle the aggregation of the data generated by LTP. The
best I could do is just generating also the .html page and upload it
to github [1] but I think more is actually desirable here.

Is there any documentation I'm expected to read before asking for
support here? :)

Thanks,
Domenico

[0] https://travis-ci.org/q-rig/qemu-ltprun
[1] https://github.com/q-rig/qemu-ltprun-results/tree/master/ltp/20170411-3.10-linux-4.11.0-rc6-ltp-20170116-arm64

-- 
3B10 0CA1 8674 ACBA B4FE  FCD2 CE5B CF17 9960 DE13

[LTP] q-rig: qemu ltprun on armhf and arm64

[Cyril Hrubis]

Hi!
> I've been toying with Travis CI, qemu and LTP recently. My
> effort is here: https://github.com/q-rig. It runs LTP on top
> of the latest linux -rc on Debian sid, inside qemu on armhf and
> arm64. Binaries are built when needed. You can find them here:
> https://github.com/q-rig/qemu-ltprun-results.
> 
> Activities are run in Travis CI [0]. LTP tests are split so to finish
> in less than 49 minutes (Travis CI constraint). The LPT results are
> put again in https://github.com/q-rig/qemu-ltprun-results, which is
> probably the wrong place.

Interesting work, although I'm not sure that travis is right tool for
the job. On the other hand I've been thinking of similar misuse as
well...

> Now I'd like to tackle the aggregation of the data generated by LTP. The
> best I could do is just generating also the .html page and upload it
> to github [1] but I think more is actually desirable here.

I would like to be able to do more with the LTP results as well, for
instance compare two testruns for the purpose of automatic pre-release
testing which would allow us to do much shorter release cycles.

Unfortunately the best the legacy LTP runltp script + ltp-pan can do is
to write a list of failed and skipped tests, it also can't run the open
posix testcases, cannot kill tests on timeout, etc.

This are the reasons I've started to experiment with writing a new test
runner that would allow for writing different output formats among
others some kind of machine parseable output (I was thinking of json but
anything structured would do) which would allow us to produce much more
interesting output than the current html page. Proof of concet is done
in my private GitHub repo[1], but it's not complete enough to be usable
and it can produce only html[2] page as an output so far, but hacking in
json formatter would be fairly easy.

[1] https://github.com/metan-ucw/ltp
    https://github.com/metan-ucw/ltp/tree/master/tools/runltp-ng

[2] http://metan.ucw.cz/outgoing/ltp-openposix.html
    http://metan.ucw.cz/outgoing/ltp-syscalls.html

    The latest version also includes hardware and sofware version but
    apart from that it's more or less it.

> Is there any documentation I'm expected to read before asking for
> support here? :)

There is probably none or close to that hence asking here is the best
course of action.

-- 
Cyril Hrubis
chrubis@suse.cz