From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.3 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E214C433E2 for ; Tue, 15 Sep 2020 16:44:05 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5E2F0206B5 for ; Tue, 15 Sep 2020 16:44:04 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="pqmcqWU3" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5E2F0206B5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:50768 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kIE3n-000827-3z for qemu-devel@archiver.kernel.org; Tue, 15 Sep 2020 12:44:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37228) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kIDzf-0005Oq-PG for qemu-devel@nongnu.org; Tue, 15 Sep 2020 12:39:47 -0400 Received: from mail-wr1-x441.google.com ([2a00:1450:4864:20::441]:42233) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kIDzd-00076B-CB for qemu-devel@nongnu.org; Tue, 15 Sep 2020 12:39:47 -0400 Received: by mail-wr1-x441.google.com with SMTP id c18so4019277wrm.9 for ; Tue, 15 Sep 2020 09:39:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=VAVwptGPTZh8EsjK9BtVG0lVM8GWJY52hf6+eD9P9lg=; b=pqmcqWU32DwvPQ6x2Wun6WZ3vhocGPnTlc4PTZ4CNaEApQpIEoTlimQc/Sm6cBBo2X xWiL6nWz8w75Gcl0YuX/9SHpfCqmgYEuBjwTlxU5MbbYR6EqyUzXAOzt4R1LwvP9QrQU EhZmHB3+BcpgXLrIDKo59GogloUjGVNWc3ZFGbFAhFeyLF6mu0VlTN6bCRBAwVjvSzII /mpFyeHW2gRJEA/PsKE+elzLwm1RibZhP0cc4gVYt7oYBVGAboK71KQj0JvgVKa04/Qu UNceS30TDhMPBNTVbYIGlFoBrtWTDiaE1VFTIbXaq2jw5iKAHh7oyXL8xR9XqEfzC5Rz LqDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=VAVwptGPTZh8EsjK9BtVG0lVM8GWJY52hf6+eD9P9lg=; b=ganKM3M7ybBaSiY1nNLJoplTA8amAvAw3XzDmI5QHU4+/bIES5t0f/WLZLcAhStjSv eknldCyP+GFh4jSIm+3d0QIDMTvKkq9jcp3spcMR7Rsdsy0oYahoTjGhha2xIzgEi0oU Ei/Jy9CnXoqTHEZhS3eQ4IqNzvm8MsksvZuRrVw4hrdXMZ2ff83kugkinAMHQ3ChfEVm OAkt5DxmfSpU91hx0kEMJyc4kFLi0FIZZvpBztolQ/gOruhFuXsRz7nnPZ7oV6A6EZE2 3q3d0Z0rPn+g7/iwp7DDwuUSJcpw0xjj4JU8NLg749l1OPuhLldo4yeZ5DzwVNh8kJoc XoGQ== X-Gm-Message-State: AOAM531LP3oK4WiSng+/f75OmD+I7vC4XppJoN0sJiKyT53XUG2HCp1j pvFbIBfs1pml2V6nMh6+HqBK+EICkG1+pfomP98= X-Google-Smtp-Source: ABdhPJzuwfOsJzB76SzMSIFXtFkKSn4/kVXgHiGgjY30NVH54esNO+0jICzINnS/Y9TOrk9WmCR3KWWJ+ibW/+JGfYE= X-Received: by 2002:a5d:4c4c:: with SMTP id n12mr9103398wrt.162.1600187983245; Tue, 15 Sep 2020 09:39:43 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a1c:7504:0:0:0:0:0 with HTTP; Tue, 15 Sep 2020 09:39:42 -0700 (PDT) In-Reply-To: <87zh6848e2.fsf@linaro.org> References: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com> <20200828104102.4490-8-ahmedkhaledkaraman@gmail.com> <87zh6848e2.fsf@linaro.org> From: Aleksandar Markovic Date: Tue, 15 Sep 2020 18:39:42 +0200 Message-ID: Subject: Re: [PATCH 7/9] tests/performance: Add nightly tests To: =?UTF-8?B?QWxleCBCZW5uw6ll?= Content-Type: multipart/alternative; boundary="00000000000050c23e05af5ccc59" Received-SPF: pass client-ip=2a00:1450:4864:20::441; envelope-from=aleksandar.qemu.devel@gmail.com; helo=mail-wr1-x441.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "ldoktor@redhat.com" , "ehabkost@redhat.com" , "philmd@redhat.com" , "qemu-devel@nongnu.org" , Ahmed Karaman , "jsnow@redhat.com" , "crosa@redhat.com" , "rth@twiddle.net" Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" --00000000000050c23e05af5ccc59 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wednesday, September 2, 2020, Alex Benn=C3=A9e = wrote: > > Ahmed Karaman writes: > > > A nightly performance testing system to monitor any change in QEMU > > performance across seventeen different targets. > > > > The system includes eight different benchmarks to provide a variety > > of testing workloads. > > > > dijkstra_double: > > Find the shortest path between the source node and all other nodes > > using Dijkstra=E2=80=99s algorithm. The graph contains n nodes where al= l nxn > > distances are double values. The value of n can be specified using > > the -n flag. The default value is 2000. > > > > dijkstra_int32: > > Find the shortest path between the source node and all other nodes > > using Dijkstra=E2=80=99s algorithm. The graph contains n nodes where al= l nxn > > distances are int32 values. The value of n can be specified using > > the -n flag. The default value is 2000. > > > > matmult_double: > > Standard matrix multiplication of an n*n matrix of randomly generated > > double numbers from 0 to 100. The value of n is passed as an argument > > with the -n flag. The default value is 200. > > > > matmult_int32: > > Standard matrix multiplication of an n*n matrix of randomly generated > > integer numbers from 0 to 100. The value of n is passed as an > > argument with the -n flag. The default value is 200. > > > > qsort_double: > > Quick sort of an array of n randomly generated double numbers from 0 > > to 1000. The value of n is passed as an argument with the -n flag. > > The default value is 300000. > > > > qsort_int32: > > Quick sort of an array of n randomly generated integer numbers from 0 > > to 50000000. The value of n is passed as an argument with the -n > > flag.The default value is 300000. > > > > qsort_string: > > Quick sort of an array of 10000 randomly generated strings of size 8 > > (including null terminating character). The sort process is repeated > > n number of times. The value of n is passed as an argument with the > > -n flag. The default value is 20. > > > > search_string: > > Search for the occurrence of a small string in a much larger random > > string (=E2=80=9Cneedle in a hay=E2=80=9D). The search process is repea= ted n number > > of times and each time, a different large random string (=E2=80=9Chay= =E2=80=9D) is > > generated. The value of n can be specified using the -n flag. The > > default value is 20. > > > > Syntax: > > nightly_tests_core.py [-h] [-r REF] > > Optional arguments: > > -h, --help Show this help message and exit > > -r REF, --reference REF > > Reference QEMU version - Default is v5.1.0 > > Example of usage: > > nightly_tests_core.py -r v5.1.0 2>log.txt > > > > The following report includes detailed setup and execution details > > of the system: > > https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/ > QEMU-Nightly-Performance-Tests/ > > > > Signed-off-by: Ahmed Karaman > > --- > > tests/performance/nightly-tests/README.md | 243 +++++ > > .../source/dijkstra_double/dijkstra_double.c | 194 ++++ > > .../source/dijkstra_int32/dijkstra_int32.c | 192 ++++ > > .../source/matmult_double/matmult_double.c | 123 +++ > > .../source/matmult_int32/matmult_int32.c | 121 +++ > > .../source/qsort_double/qsort_double.c | 104 ++ > > .../source/qsort_int32/qsort_int32.c | 103 ++ > > .../source/qsort_string/qsort_string.c | 122 +++ > > .../source/search_string/search_string.c | 110 +++ > > .../scripts/nightly_tests_core.py | 920 ++++++++++++++++++ > > .../scripts/run_nightly_tests.py | 135 +++ > > .../nightly-tests/scripts/send_email.py | 56 ++ > > 12 files changed, 2423 insertions(+) > > create mode 100644 tests/performance/nightly-tests/README.md > > create mode 100644 tests/performance/nightly-tests/benchmarks/source/ > dijkstra_double/dijkstra_double.c > > create mode 100644 tests/performance/nightly-tests/benchmarks/source/ > dijkstra_int32/dijkstra_int32.c > > create mode 100644 tests/performance/nightly-tests/benchmarks/source/ > matmult_double/matmult_double.c > > create mode 100644 tests/performance/nightly-tests/benchmarks/source/ > matmult_int32/matmult_int32.c > > create mode 100644 tests/performance/nightly- > tests/benchmarks/source/qsort_double/qsort_double.c > > create mode 100644 tests/performance/nightly- > tests/benchmarks/source/qsort_int32/qsort_int32.c > > create mode 100644 tests/performance/nightly- > tests/benchmarks/source/qsort_string/qsort_string.c > > create mode 100644 > > tests/performance/nightly-tests/benchmarks/source/ > search_string/search_string.c > > Perhaps we could compress these paths down to: > > tests/tcg/benchmarks/foo.c > tests/tcg/benchmarks/bar.c > > and then we can also ensure they are built using the existing TCG tests > cross compile framework. > > Hi, May I just bring some alternative views on this topic of benchmarks/multiple cross-compiling/nightly tests? Having a known source code of any benchmark (and also license-compatible with QEMU) is, for sure, a good thing. However, for actual test execution, source code is not relevant, but the executables built for a diverse set of targets. Expectation that a test bed should be able to build let's say 17 version for each benchmark is a tall order - and, in fact, unnecessary self-imposed limitation. My suggestion is that 17x8 executables needed for nightly tests in its current form should be prebuilt - and kept permanently in a separate repository (just not to burden QEMU repository). They are anyway built only once, there is no need to update them at all. This will allow nightly test execution on any system that can execute qemu, without any prerequisite for cross-compiling. Their source code could and should be integrated into QEMU tree, at the place of choice, but the test beds should not be subject to any burden of cross-compiling, simply because there is no need for that. Yours, Aleksandar > > create mode 100755 tests/performance/nightly- > tests/scripts/nightly_tests_core.py > > create mode 100755 tests/performance/nightly-tests/scripts/run_nightly= _ > tests.py > > create mode 100644 tests/performance/nightly- > tests/scripts/send_email.py > > > > diff --git a/tests/performance/nightly-tests/README.md > b/tests/performance/nightly-tests/README.md > > new file mode 100644 > > index 0000000000..6db3b351b3 > > --- /dev/null > > +++ b/tests/performance/nightly-tests/README.md > > @@ -0,0 +1,243 @@ > > +### QEMU Nightly Tests > > + > > +**Required settings:** > > + > > +Update the `GMAIL_USER` object in `send_email.py` with your credential= s. > > + > > +For more details on how the system works, please check the [eighth > > report](https://ahmedkrmn.github.io/TCG-Continuous- > Benchmarking/QEMU-Nightly-Performance-Tests/) > > of the "TCG Continuos Benchmarking" series. > > As external URLs are potentially unstable I think we want to distil the > details into a rst do in docs/devel/ > > > > -- > Alex Benn=C3=A9e > --00000000000050c23e05af5ccc59 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

On Wednesday, September 2, 2020, Alex Benn=C3=A9e <alex.bennee@linaro.org> wrote:

Ahmed Karaman <ahmedkhal= edkaraman@gmail.com> writes:

> A nightly performance testing system to monitor any change in QEMU
> performance across seventeen different targets.
>
> The system includes eight different benchmarks to provide a variety > of testing workloads.
>
> dijkstra_double:
> Find the shortest path between the source node and all other nodes
> using Dijkstra=E2=80=99s algorithm. The graph contains n nodes where a= ll nxn
> distances are double values. The value of n can be specified using
> the -n flag. The default value is 2000.
>
> dijkstra_int32:
> Find the shortest path between the source node and all other nodes
> using Dijkstra=E2=80=99s algorithm. The graph contains n nodes where a= ll nxn
> distances are int32 values. The value of n can be specified using
> the -n flag. The default value is 2000.
>
> matmult_double:
> Standard matrix multiplication of an n*n matrix of randomly generated<= br> > double numbers from 0 to 100. The value of n is passed as an argument<= br> > with the -n flag. The default value is 200.
>
> matmult_int32:
> Standard matrix multiplication of an n*n matrix of randomly generated<= br> > integer numbers from 0 to 100. The value of n is passed as an
> argument with the -n flag. The default value is 200.
>
> qsort_double:
> Quick sort of an array of n randomly generated double numbers from 0 > to 1000. The value of n is passed as an argument with the -n flag.
> The default value is 300000.
>
> qsort_int32:
> Quick sort of an array of n randomly generated integer numbers from 0<= br> > to 50000000. The value of n is passed as an argument with the -n
> flag.The default value is 300000.
>
> qsort_string:
> Quick sort of an array of 10000 randomly generated strings of size 8 > (including null terminating character). The sort process is repeated > n number of times. The value of n is passed as an argument with the > -n flag. The default value is 20.
>
> search_string:
> Search for the occurrence of a small string in a much larger random > string (=E2=80=9Cneedle in a hay=E2=80=9D). The search process is repe= ated n number
> of times and each time, a different large random string (=E2=80=9Chay= =E2=80=9D) is
> generated. The value of n can be specified using the -n flag. The
> default value is 20.
>
> Syntax:
>=C2=A0 =C2=A0 =C2=A0nightly_tests_core.py [-h] [-r REF]
>=C2=A0 =C2=A0 =C2=A0Optional arguments:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-h, --help=C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 Show this help message and exit
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-r REF, --reference REF
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0Reference QEMU version - Default is v5.1.0
>=C2=A0 =C2=A0 =C2=A0Example of usage:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0nightly_tests_core.py -r v5.1.0 2>= log.txt
>
> The following report includes detailed setup and execution details
> of the system:
> https://ahmedkrmn.github.io= /TCG-Continuous-Benchmarking/QEMU-Nightly-Performance-Tests/=
>
> Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> ---
>=C2=A0 tests/performance/nightly-tests/README.md=C2=A0 =C2=A0 =C2= =A0| 243 +++++
>=C2=A0 .../source/dijkstra_double/dijkstra_double.c=C2=A0 | 194 ++= ++
>=C2=A0 .../source/dijkstra_int32/dijkstra_int32.c=C2=A0 =C2=A0 | 1= 92 ++++
>=C2=A0 .../source/matmult_double/matmult_double.c=C2=A0 =C2=A0 | 1= 23 +++
>=C2=A0 .../source/matmult_int32/matmult_int32.c=C2=A0 =C2=A0 =C2= =A0 | 121 +++
>=C2=A0 .../source/qsort_double/qsort_double.c=C2=A0 =C2=A0 =C2=A0 = =C2=A0 | 104 ++
>=C2=A0 .../source/qsort_int32/qsort_int32.c=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 | 103 ++
>=C2=A0 .../source/qsort_string/qsort_string.c=C2=A0 =C2=A0 =C2=A0 = =C2=A0 | 122 +++
>=C2=A0 .../source/search_string/search_string.c=C2=A0 =C2=A0 =C2= =A0 | 110 +++
>=C2=A0 .../scripts/nightly_tests_core.py=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0| 920 ++++++++++++++++++
>=C2=A0 .../scripts/run_nightly_tests.py=C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 | 135 +++
>=C2=A0 .../nightly-tests/scripts/send_email.py=C2=A0 =C2=A0 =C2=A0= =C2=A0|=C2=A0 56 ++
>=C2=A0 12 files changed, 2423 insertions(+)
>=C2=A0 create mode 100644 tests/performance/nightly-tests/README.m= d
>=C2=A0 create mode 100644 tests/performance/nightly-tests/benchmar= ks/source/dijkstra_double/dijkstra_double.c
>=C2=A0 create mode 100644 tests/performance/nightly-tests/benchmar= ks/source/dijkstra_int32/dijkstra_int32.c
>=C2=A0 create mode 100644 tests/performance/nightly-tests/benchmar= ks/source/matmult_double/matmult_double.c
>=C2=A0 create mode 100644 tests/performance/nightly-tests/benchmar= ks/source/matmult_int32/matmult_int32.c
>=C2=A0 create mode 100644 tests/performance/nightly-tests/benchmar= ks/source/qsort_double/qsort_double.c
>=C2=A0 create mode 100644 tests/performance/nightly-tests/benchmar= ks/source/qsort_int32/qsort_int32.c
>=C2=A0 create mode 100644 tests/performance/nightly-tests/benchmar= ks/source/qsort_string/qsort_string.c
>=C2=A0 create mode 100644
> tests/performance/nightly-tests/benchmarks/source/search_str= ing/search_string.c

Perhaps we could compress these paths down to:

=C2=A0 tests/tcg/benchmarks/foo.c
=C2=A0 tests/tcg/benchmarks/bar.c

and then we can also ensure they are built using the existing TCG tests
cross compile framework.


Hi,

May I jus= t bring some alternative views on this topic of benchmarks/multiple cross-c= ompiling/nightly tests?

Having a known source code= of any benchmark (and also license-compatible with QEMU) is, for sure, a g= ood thing. However, for actual test execution, source code is not relevant,= but the executables built for a diverse set of targets. Expectation that a= test bed should be able to build let's say 17 version for each benchma= rk is a tall order - and, in fact, unnecessary self-imposed limitation.

My suggestion is that 17x8 executables needed for nig= htly tests in its current form should be prebuilt - and kept permanently in= a separate repository (just not to burden QEMU repository). They are anywa= y built only once, there is no need to update them at all.

This will allow nightly test execution on any system that can exec= ute qemu, without any prerequisite for cross-compiling.

Their source code could and should be integrated into QEMU tree, at t= he place of choice, but the test beds should not be subject to any burden o= f cross-compiling, simply because there is no need for that.

=
Yours,
Aleksandar


<= div>=C2=A0
>=C2=A0 create mode 100755 tests/performance/nightly-tests/scripts/= nightly_tests_core.py
>=C2=A0 create mode 100755 tests/performance/nightly-tests/scripts/= run_nightly_tests.py
>=C2=A0 create mode 100644 tests/performance/nightly-tests/scripts/= send_email.py
>
> diff --git a/tests/performance/nightly-tests/README.md b/tests/pe= rformance/nightly-tests/README.md
> new file mode 100644
> index 0000000000..6db3b351b3
> --- /dev/null
> +++ b/tests/performance/nightly-tests/README.md
> @@ -0,0 +1,243 @@
> +### QEMU Nightly Tests
> +
> +**Required settings:**
> +
> +Update the `GMAIL_USER` object in `send_email.py` with your credentia= ls.
> +
> +For more details on how the system works, please check the [eighth > report](https://ahmedkrmn.<= wbr>github.io/TCG-Continuous-Benchmarking/QEMU-Nightly-Performanc= e-Tests/)
> of the "TCG Continuos Benchmarking" series.

As external URLs are potentially unstable I think we want to distil the
details into a rst do in docs/devel/

<snip>

--
Alex Benn=C3=A9e
--00000000000050c23e05af5ccc59--