Re: [PATCH 7/9] tests/performance: Add nightly tests

[Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 7279 bytes --]

On Wednesday, September 2, 2020, Alex Bennée <alex.bennee@linaro.org> wrote:

>
> Ahmed Karaman <ahmedkhaledkaraman@gmail.com> writes:
>
> > A nightly performance testing system to monitor any change in QEMU
> > performance across seventeen different targets.
> >
> > The system includes eight different benchmarks to provide a variety
> > of testing workloads.
> >
> > dijkstra_double:
> > Find the shortest path between the source node and all other nodes
> > using Dijkstra’s algorithm. The graph contains n nodes where all nxn
> > distances are double values. The value of n can be specified using
> > the -n flag. The default value is 2000.
> >
> > dijkstra_int32:
> > Find the shortest path between the source node and all other nodes
> > using Dijkstra’s algorithm. The graph contains n nodes where all nxn
> > distances are int32 values. The value of n can be specified using
> > the -n flag. The default value is 2000.
> >
> > matmult_double:
> > Standard matrix multiplication of an n*n matrix of randomly generated
> > double numbers from 0 to 100. The value of n is passed as an argument
> > with the -n flag. The default value is 200.
> >
> > matmult_int32:
> > Standard matrix multiplication of an n*n matrix of randomly generated
> > integer numbers from 0 to 100. The value of n is passed as an
> > argument with the -n flag. The default value is 200.
> >
> > qsort_double:
> > Quick sort of an array of n randomly generated double numbers from 0
> > to 1000. The value of n is passed as an argument with the -n flag.
> > The default value is 300000.
> >
> > qsort_int32:
> > Quick sort of an array of n randomly generated integer numbers from 0
> > to 50000000. The value of n is passed as an argument with the -n
> > flag.The default value is 300000.
> >
> > qsort_string:
> > Quick sort of an array of 10000 randomly generated strings of size 8
> > (including null terminating character). The sort process is repeated
> > n number of times. The value of n is passed as an argument with the
> > -n flag. The default value is 20.
> >
> > search_string:
> > Search for the occurrence of a small string in a much larger random
> > string (“needle in a hay”). The search process is repeated n number
> > of times and each time, a different large random string (“hay”) is
> > generated. The value of n can be specified using the -n flag. The
> > default value is 20.
> >
> > Syntax:
> >     nightly_tests_core.py [-h] [-r REF]
> >     Optional arguments:
> >         -h, --help            Show this help message and exit
> >         -r REF, --reference REF
> >                         Reference QEMU version - Default is v5.1.0
> >     Example of usage:
> >         nightly_tests_core.py -r v5.1.0 2>log.txt
> >
> > The following report includes detailed setup and execution details
> > of the system:
> > https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/
> QEMU-Nightly-Performance-Tests/
> >
> > Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> > ---
> >  tests/performance/nightly-tests/README.md     | 243 +++++
> >  .../source/dijkstra_double/dijkstra_double.c  | 194 ++++
> >  .../source/dijkstra_int32/dijkstra_int32.c    | 192 ++++
> >  .../source/matmult_double/matmult_double.c    | 123 +++
> >  .../source/matmult_int32/matmult_int32.c      | 121 +++
> >  .../source/qsort_double/qsort_double.c        | 104 ++
> >  .../source/qsort_int32/qsort_int32.c          | 103 ++
> >  .../source/qsort_string/qsort_string.c        | 122 +++
> >  .../source/search_string/search_string.c      | 110 +++
> >  .../scripts/nightly_tests_core.py             | 920 ++++++++++++++++++
> >  .../scripts/run_nightly_tests.py              | 135 +++
> >  .../nightly-tests/scripts/send_email.py       |  56 ++
> >  12 files changed, 2423 insertions(+)
> >  create mode 100644 tests/performance/nightly-tests/README.md
> >  create mode 100644 tests/performance/nightly-tests/benchmarks/source/
> dijkstra_double/dijkstra_double.c
> >  create mode 100644 tests/performance/nightly-tests/benchmarks/source/
> dijkstra_int32/dijkstra_int32.c
> >  create mode 100644 tests/performance/nightly-tests/benchmarks/source/
> matmult_double/matmult_double.c
> >  create mode 100644 tests/performance/nightly-tests/benchmarks/source/
> matmult_int32/matmult_int32.c
> >  create mode 100644 tests/performance/nightly-
> tests/benchmarks/source/qsort_double/qsort_double.c
> >  create mode 100644 tests/performance/nightly-
> tests/benchmarks/source/qsort_int32/qsort_int32.c
> >  create mode 100644 tests/performance/nightly-
> tests/benchmarks/source/qsort_string/qsort_string.c
> >  create mode 100644
> > tests/performance/nightly-tests/benchmarks/source/
> search_string/search_string.c
>
> Perhaps we could compress these paths down to:
>
>   tests/tcg/benchmarks/foo.c
>   tests/tcg/benchmarks/bar.c
>
> and then we can also ensure they are built using the existing TCG tests
> cross compile framework.
>
>
Hi,

May I just bring some alternative views on this topic of
benchmarks/multiple cross-compiling/nightly tests?

Having a known source code of any benchmark (and also license-compatible
with QEMU) is, for sure, a good thing. However, for actual test execution,
source code is not relevant, but the executables built for a diverse set of
targets. Expectation that a test bed should be able to build let's say 17
version for each benchmark is a tall order - and, in fact, unnecessary
self-imposed limitation.

My suggestion is that 17x8 executables needed for nightly tests in its
current form should be prebuilt - and kept permanently in a separate
repository (just not to burden QEMU repository). They are anyway built only
once, there is no need to update them at all.

This will allow nightly test execution on any system that can execute qemu,
without any prerequisite for cross-compiling.

Their source code could and should be integrated into QEMU tree, at the
place of choice, but the test beds should not be subject to any burden of
cross-compiling, simply because there is no need for that.

Yours,
Aleksandar




> >  create mode 100755 tests/performance/nightly-
> tests/scripts/nightly_tests_core.py
> >  create mode 100755 tests/performance/nightly-tests/scripts/run_nightly_
> tests.py
> >  create mode 100644 tests/performance/nightly-
> tests/scripts/send_email.py
> >
> > diff --git a/tests/performance/nightly-tests/README.md
> b/tests/performance/nightly-tests/README.md
> > new file mode 100644
> > index 0000000000..6db3b351b3
> > --- /dev/null
> > +++ b/tests/performance/nightly-tests/README.md
> > @@ -0,0 +1,243 @@
> > +### QEMU Nightly Tests
> > +
> > +**Required settings:**
> > +
> > +Update the `GMAIL_USER` object in `send_email.py` with your credentials.
> > +
> > +For more details on how the system works, please check the [eighth
> > report](https://ahmedkrmn.github.io/TCG-Continuous-
> Benchmarking/QEMU-Nightly-Performance-Tests/)
> > of the "TCG Continuos Benchmarking" series.
>
> As external URLs are potentially unstable I think we want to distil the
> details into a rst do in docs/devel/
>
> <snip>
>
> --
> Alex Bennée
>

[-- Attachment #2: Type: text/html, Size: 9003 bytes --]