From: "Alex Bennée" <alex.bennee@linaro.org>
To: Peter Maydell <peter.maydell@linaro.org>
Cc: "Philippe Mathieu-Daudé" <f4bug@amsat.org>,
"Daniel P. Berrange" <berrange@redhat.com>,
"QEMU Developers" <qemu-devel@nongnu.org>,
"Cleber Rosa" <crosa@redhat.com>
Subject: Re: "make check-acceptance" takes way too long
Date: Sat, 31 Jul 2021 19:41:03 +0100 [thread overview]
Message-ID: <87sfzuwchl.fsf@linaro.org> (raw)
In-Reply-To: <CAFEAcA9cMZoj18gq7Ksv5PRoU1wRmXvW_e9UE73C_MEB7wTroQ@mail.gmail.com>
Peter Maydell <peter.maydell@linaro.org> writes:
> "make check-acceptance" takes way way too long. I just did a run
> on an arm-and-aarch64-targets-only debug build and it took over
> half an hour, and this despite it skipping or cancelling 26 out
> of 58 tests!
>
> I think that ~10 minutes runtime is reasonable. 30 is not;
> ideally no individual test would take more than a minute or so.
>
> Output saying where the time went. The first two tests take
> more than 10 minutes *each*. I think a good start would be to find
> a way of testing what they're testing that is less heavyweight.
>
> (01/58) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg_gicv2:
> PASS (629.74 s)
> (02/58) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg_gicv3:
> PASS (628.75 s)
> (03/58) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_kvm:
> CANCEL: kvm accelerator does not seem to be available (1.18 s)
For these tests which purport to exercise the various GIC configurations
I think we would be much better served by running kvm-unit-tests which
at least try and exercise all the features rather than rely on the side
effect of booting an entire OS.
> (04/58) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_aarch64_virt:
> PASS (3.53 s)
> (05/58) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_aarch64_xlnx_versal_virt:
> PASS (41.13 s)
> (06/58) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_virt:
> PASS (5.22 s)
> (07/58) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_emcraft_sf2:
> PASS (18.88 s)
> (08/58) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_raspi2_uart0:
> PASS (11.30 s)
> (09/58) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_raspi2_initrd:
> PASS (22.66 s)
> (10/58) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_exynos4210_initrd:
> PASS (31.89 s)
> (11/58) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_cubieboard_initrd:
> PASS (27.86 s)
> (12/58) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_cubieboard_sata:
> PASS (27.19 s)
> (13/58) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_quanta_gsj:
> SKIP: Test might timeout
> (14/58) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_quanta_gsj_initrd:
> PASS (22.53 s)
> (15/58) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi:
> PASS (4.86 s)
> (16/58) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_initrd:
> PASS (39.85 s)
> (17/58) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_sd:
> PASS (53.57 s)
> (18/58) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_bionic_20_08:
> SKIP: storage limited
> (19/58) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_uboot_netbsd9:
> SKIP: storage limited
> (20/58) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf:
> PASS (1.50 s)
> (21/58) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_vexpressa9:
> PASS (10.74 s)
> (22/58) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_ast2400_palmetto_openbmc_v2_9_0:
> PASS (39.43 s)
> (23/58) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_ast2500_romulus_openbmc_v2_9_0:
> PASS (54.01 s)
> (24/58) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_ast2600_debian:
> PASS (40.60 s)
> (25/58) tests/acceptance/boot_xen.py:BootXen.test_arm64_xen_411_and_dom0:
> PASS (20.22 s)
> (26/58) tests/acceptance/boot_xen.py:BootXen.test_arm64_xen_414_and_dom0:
> PASS (17.37 s)
> (27/58) tests/acceptance/boot_xen.py:BootXen.test_arm64_xen_415_and_dom0:
> PASS (23.82 s)
> (28/58) tests/acceptance/empty_cpu_model.py:EmptyCPUModel.test:
> CANCEL: No QEMU binary defined or found in the build tree (0.00 s)
> (29/58) tests/acceptance/info_usernet.py:InfoUsernet.test_hostfwd:
> CANCEL: No QEMU binary defined or found in the build tree (0.00 s)
> (30/58) tests/acceptance/machine_arm_canona1100.py:CanonA1100Machine.test_arm_canona1100:
> PASS (0.20 s)
> (31/58) tests/acceptance/machine_arm_integratorcp.py:IntegratorMachine.test_integratorcp_console:
> SKIP: untrusted code
> (32/58) tests/acceptance/machine_arm_integratorcp.py:IntegratorMachine.test_framebuffer_tux_logo:
> SKIP: Python NumPy not installed
> (33/58) tests/acceptance/machine_arm_n8x0.py:N8x0Machine.test_n800:
> SKIP: untrusted code
> (34/58) tests/acceptance/machine_arm_n8x0.py:N8x0Machine.test_n810:
> SKIP: untrusted code
> (35/58) tests/acceptance/migration.py:Migration.test_migration_with_tcp_localhost:
> CANCEL: No QEMU binary defined or found in the build tree (0.00 s)
> (36/58) tests/acceptance/migration.py:Migration.test_migration_with_unix:
> CANCEL: No QEMU binary defined or found in the build tree (0.00 s)
> (37/58) tests/acceptance/migration.py:Migration.test_migration_with_exec:
> CANCEL: No QEMU binary defined or found in the build tree (0.00 s)
> (38/58) tests/acceptance/multiprocess.py:Multiprocess.test_multiprocess_aarch64:
> CANCEL: kvm accelerator does not seem to be available (0.06 s)
> (39/58) tests/acceptance/replay_kernel.py:ReplayKernelNormal.test_aarch64_virt:
> PASS (19.59 s)
> (40/58) tests/acceptance/replay_kernel.py:ReplayKernelNormal.test_arm_virt:
> PASS (28.73 s)
> (41/58) tests/acceptance/replay_kernel.py:ReplayKernelNormal.test_arm_cubieboard_initrd:
> PASS (52.00 s)
> (42/58) tests/acceptance/replay_kernel.py:ReplayKernelNormal.test_arm_vexpressa9:
> PASS (25.69 s)
> (43/58) tests/acceptance/reverse_debugging.py:ReverseDebugging_AArch64.test_aarch64_virt:
> PASS (2.16 s)
> (44/58) tests/acceptance/smmu.py:SMMU.test_smmu_noril: CANCEL: kvm
> accelerator does not seem to be available (0.90 s)
> (45/58) tests/acceptance/smmu.py:SMMU.test_smmu_noril_passthrough:
> CANCEL: kvm accelerator does not seem to be available (0.70 s)
> (46/58) tests/acceptance/smmu.py:SMMU.test_smmu_noril_nostrict:
> CANCEL: kvm accelerator does not seem to be available (1.02 s)
> (47/58) tests/acceptance/smmu.py:SMMU.test_smmu_ril: CANCEL: kvm
> accelerator does not seem to be available (0.68 s)
> (48/58) tests/acceptance/smmu.py:SMMU.test_smmu_ril_passthrough:
> CANCEL: kvm accelerator does not seem to be available (0.98 s)
> (49/58) tests/acceptance/smmu.py:SMMU.test_smmu_ril_nostrict: CANCEL:
> kvm accelerator does not seem to be available (1.00 s)
> (50/58) tests/acceptance/tcg_plugins.py:PluginKernelNormal.test_aarch64_virt_insn:
> PASS (12.19 s)
> (51/58) tests/acceptance/tcg_plugins.py:PluginKernelNormal.test_aarch64_virt_insn_icount:
> PASS (12.35 s)
> (52/58) tests/acceptance/tcg_plugins.py:PluginKernelNormal.test_aarch64_virt_mem_icount:
> PASS (10.21 s)
> (53/58) tests/acceptance/version.py:Version.test_qmp_human_info_version:
> CANCEL: No QEMU binary defined or found in the build tree (0.00 s)
> (54/58) tests/acceptance/virtio_check_params.py:VirtioMaxSegSettingsCheck.test_machine_types:
> SKIP: break multi-arch CI
> (55/58) tests/acceptance/vnc.py:Vnc.test_no_vnc: CANCEL: No QEMU
> binary defined or found in the build tree (0.00 s)
> (56/58) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password:
> CANCEL: No QEMU binary defined or found in the build tree (0.00 s)
> (57/58) tests/acceptance/vnc.py:Vnc.test_change_password_requires_a_password:
> CANCEL: No QEMU binary defined or found in the build tree (0.00 s)
> (58/58) tests/acceptance/vnc.py:Vnc.test_change_password: CANCEL: No
> QEMU binary defined or found in the build tree (0.00 s)
> RESULTS : PASS 32 | ERROR 0 | FAIL 0 | SKIP 8 | WARN 0 | INTERRUPT
> 0 | CANCEL 18
> JOB TIME : 1967.44 s
>
> thanks
> -- PMM
--
Alex Bennée
next prev parent reply other threads:[~2021-07-31 18:44 UTC|newest]
Thread overview: 46+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-07-30 15:12 "make check-acceptance" takes way too long Peter Maydell
2021-07-30 15:41 ` Philippe Mathieu-Daudé
2021-07-30 15:42 ` Peter Maydell
2021-07-30 22:04 ` Cleber Rosa
2021-07-31 6:39 ` Thomas Huth
2021-07-31 17:58 ` Cleber Rosa
2021-07-31 18:41 ` Alex Bennée [this message]
2021-07-31 20:32 ` Peter Maydell
2021-08-02 22:55 ` Cleber Rosa
2021-08-02 8:38 ` Daniel P. Berrangé
2021-08-02 12:47 ` Alex Bennée
2021-08-02 12:59 ` Daniel P. Berrangé
2021-08-02 12:55 ` Alex Bennée
2021-08-02 13:00 ` Peter Maydell
2021-08-02 13:04 ` Daniel P. Berrangé
2021-08-02 13:25 ` Thomas Huth
2021-08-02 13:00 ` Daniel P. Berrangé
2021-08-02 13:27 ` Thomas Huth
2021-08-02 13:43 ` Gerd Hoffmann
2022-01-20 15:13 ` Peter Maydell
2022-01-20 15:35 ` Philippe Mathieu-Daudé via
2022-01-21 7:56 ` Thomas Huth
2022-01-21 10:50 ` Markus Armbruster
2022-01-21 11:33 ` Peter Maydell
2022-01-21 12:23 ` Alex Bennée
2022-01-21 12:41 ` Thomas Huth
2022-01-21 15:21 ` Daniel P. Berrangé
2022-01-25 9:20 ` Gerd Hoffmann
2022-02-01 6:31 ` Stefano Brivio
2022-02-01 7:49 ` Gerd Hoffmann
2022-02-01 9:06 ` Daniel P. Berrangé
2022-02-01 10:27 ` Stefano Brivio
2022-02-01 11:17 ` Alex Bennée
2022-02-01 16:01 ` Cleber Rosa
2022-02-01 16:19 ` Daniel P. Berrangé
2022-02-01 17:47 ` Cleber Rosa
2022-02-01 18:03 ` Alex Bennée
2022-02-01 19:04 ` Cleber Rosa
2022-02-01 18:35 ` Stefano Brivio
2022-02-01 17:59 ` Cédric Le Goater
2022-02-01 11:06 ` Kashyap Chamarthy
2022-02-01 15:54 ` Cleber Rosa
2022-02-01 5:29 ` Cleber Rosa
2022-02-01 17:01 ` Daniel P. Berrangé
2022-02-01 17:59 ` Cleber Rosa
2022-02-15 18:14 ` Alex Bennée
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87sfzuwchl.fsf@linaro.org \
--to=alex.bennee@linaro.org \
--cc=berrange@redhat.com \
--cc=crosa@redhat.com \
--cc=f4bug@amsat.org \
--cc=peter.maydell@linaro.org \
--cc=qemu-devel@nongnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).