All of lore.kernel.org
 help / color / mirror / Atom feed
From: Christian Schoenebeck <qemu_oss@crudebyte.com>
To: qemu-devel@nongnu.org, Thomas Huth <thuth@redhat.com>
Cc: Greg Kurz <groug@kaod.org>,
	Peter Maydell <peter.maydell@linaro.org>,
	"open list:RISC-V" <qemu-riscv@nongnu.org>
Subject: Re: [PULL 2/9] tests/9p-race: add 9p race test suite (race FID-path mutations)
Date: Mon, 26 Oct 2026 17:38:58 +0100	[thread overview]
Message-ID: <24565308.6Emhk5qWAg@weasel> (raw)
In-Reply-To: <97f60010-c6fb-4d21-92fd-754d2c791928@redhat.com>

On Monday, 26 October 2026 10:35:12 CET Thomas Huth wrote:
> On 26/10/2026 09.44, Thomas Huth wrote:
> > On 19/10/2026 16.27, Christian Schoenebeck wrote:
> >> Add a qtest-based stress test suite racing 9pfs server-side state.
> >> 
> >> This patch adds the first race test "race/fid-path" for this new
> >> test suite: it races (per round) one Tlcreate (which is mutating
> >> the FID path) against a number of reader requests (Twalk and
> >> Txattrwalk) on the same FID.
> >> 
> >> This very first 9pfs race test guards CVE-2026-93834 (fixed by
> >> commit fe9e30899f "hw/9pfs: Use read-write lock for protecting fid
> >> path.").
> >> 
> >> Hitting the small race window is timing dependant, so this is a
> >> probabilistic test, not a deterministic one. During my tests the
> >> (unpatched) vulnerability triggered sometimes after few hundred
> >> rounds, and sometimes after >100k rounds. Furthermore, it expects
> >> QEMU being compiled with a sanitizer (e.g. ASan) which is
> >> supposed to detect the actual race (e.g. by memory violation):
> >> 
> >>    ../configure --enable-virtfs --enable-debug \
> >>    --extra-cflags="-fsanitize=address -fno-omit-frame-pointer" \
> >>    --extra-ldflags="-fsanitize=address" ...
> >> 
> >> Even though this is a new test suite, in the codebase separate
> >> from the already existing deterministic 9pfs test cases (while
> >> 9p test client code being shared between the two test suites),
> >> this new race test suite is run with the same binary as the
> >> other qtests. However being a very slow test, it is registered
> >> as such, so the -m slow argument is required for this race test
> >> to run (as it is skipped by default):
> >> 
> >>    ASAN_OPTIONS=detect_leaks=0 \
> >>    tests/qtest/qos-test -m slow
> >> 
> >> Or if you only want to run this single race test (e.g. on x86_64):
> >> 
> >>    ASAN_OPTIONS=detect_leaks=0 \
> >>    tests/qtest/qos-test -m slow -p \
> >>    /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-9p-pci/virtio-9p/
> >> virtio-9p-tests/race/fid-path
> >> 
> >> On other systems, use `tests/qtest/qos-test -l` to get the correct
> >> (arch dependent) full test name.
> >> 
> >> `ASAN_OPTIONS=detect_leaks=0` is currently required, it suppresses
> >> a memory leak error at the end of each race test, which is a known
> >> QEMU shutdown memory leak, unrelated to 9p server or this race test
> >> suite. The official QEMU CI pipeline sets
> >> `ASAN_OPTIONS=detect_leaks=0` by default for this reason.
> >> 
> >> There are some knobs that can be controlled via environment
> >> variables:
> >> 
> >>    V9FS_RACE: skip test suite by setting to 0 or force running by
> >>               setting to 1
> >> 
> >>    V9FS_RACE_ROUNDS: total amount of rounds to run
> >>                      (default: 20000)
> >> 
> >>    V9FS_RACE_NCONCURRENT: amount of concurrent reader requests
> >>                           (default: 4)
> >> 
> >> A full run with 20000 rounds took around 7 minutes on my test
> >> machine. If you can't hit the (narrow) race window with
> >> unpatched QEMU, it might help to either add some load on the
> >> host system and/or turning the two knobs above.
> >> 
> >> Link: https://lore.kernel.org/qemu-> >> devel/4a0eff0539f16434b48728cd8e5a89f906c293d8.1791368962.git.qemu_oss@cr
> >> udebyte.com Signed-off-by: Christian Schoenebeck <qemu_oss@crudebyte.com>
> >> ---
> >>   MAINTAINERS                       |   1 +
> >>   tests/qtest/meson.build           |   1 +
> >>   tests/qtest/virtio-9p-race-test.c | 575 ++++++++++++++++++++++++++++++
> >>   3 files changed, 577 insertions(+)
> >>   create mode 100644 tests/qtest/virtio-9p-race-test.c
> >> 
> >   Hi Christian!
> > 
> > Since this has been merged, I'm now seeing timeouts of the qos-test when I
> > run "make -j8 check-qtest SPEED=thorough" ... can we please fix this
> > somehow, e.g. by increasing the timeout setting of the qos-test, or by
> > decreasing the default number of rounds that this test is trying to do?
> > WDYT?
> FWIW, when I allow the tests to run longer, these are the current testing
> times on my machine:
> 
>   qemu:qtest-arm/qos-test               : 373.06s   167 subtests passed
>   qemu:qtest-aarch64/qos-test           : 391.52s   148 subtests passed
>   qemu:qtest-i386/qos-test              : 407.43s   154 subtests passed
>   qemu:qtest-loongarch64/qos-test       : 405.79s   146 subtests passed
>   qemu:qtest-ppc64/qos-test             : 407.26s   134 subtests passed
>   qemu:qtest-x86_64/qos-test            : 393.14s   154 subtests passed
>   qemu:qtest-riscv32/qos-test           : 767.21s   216 subtests passed
>   qemu:qtest-riscv64/qos-test           : 684.54s   216 subtests passed

Right, I missed when you run the thorough/slow tests trough make, then the 
meson wrapped qos-test call is guarded by a timeout, which currently is set to 
120s for the *entire* qos-test suite (tests/qtest/meson.build), which is fine 
for regular tests, but insufficient for these new slow 9p race tests.

I thought timeouts were just a thing for the official in-cloud QEMU CI 
pipeline - which doesn't run any slow test at all, so no problem there.
That was actually the point why I explictly registered these new race tests as 
"slow" ones in the first place.

This is more of a project management question; should these extremely slow 9p 
race tests either:

a) be skipped entirely on `make check-qtest SPEED=thorough` or
b) run shorter (i.e. less rounds, very low probability to hit) or
c) increase the meson timeout for qos-test binary?

Personally I don't care too much. These 9p race tests are useful for me, but I 
also understand if other unrelated developers don't want to spend like 10+ 
mins on each test run for this. I still run them directly via `tests/qtest/
qos-test -m slow` ; last line of defense, etc.

Technically it is easy to address:

a) -> set envvar V9FS_RACE=0
b) -> set envvar V9FS_RACE_ROUNDS=n
c) -> s/'qos-test': 120/'qos-test': n/

all in tests/qtest/meson.build. Your call.

/Christian




  reply	other threads:[~2026-10-26 16:39 UTC|newest]

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-10-19 14:27 [PULL 0/9] 9p queue 2026-10-19 Christian Schoenebeck
2026-10-19 14:27 ` [PULL 6/9] hw/9pfs: fix dirent stream protection for legacy (9p2000.u) Tread Christian Schoenebeck
2026-10-19 14:27 ` [PULL 5/9] tests/9p-race: add Treaddir race test Christian Schoenebeck
2026-10-19 14:27 ` [PULL 4/9] tests/9p-client: add Tstat / Rstat test client functions Christian Schoenebeck
2026-10-19 14:27 ` [PULL 8/9] tests/9p-client: track the negotiated 9p version for session Christian Schoenebeck
2026-10-19 14:27 ` [PULL 2/9] tests/9p-race: add 9p race test suite (race FID-path mutations) Christian Schoenebeck
2026-10-26  8:44   ` Thomas Huth
2026-10-26  9:35     ` Thomas Huth
2026-10-26 16:38       ` Christian Schoenebeck [this message]
2026-10-27  7:27         ` Thomas Huth
2026-10-27 12:59           ` Christian Schoenebeck
2026-10-19 14:27 ` [PULL 7/9] tests/9p-client: add Tcreate / Rcreate test client functions Christian Schoenebeck
2026-10-19 14:27 ` [PULL 1/9] tests/9p-client: add Txattrwalk / Rxattrwalk " Christian Schoenebeck
2026-10-19 14:27 ` [PULL 3/9] tests/9p-client: add Rerror function (legacy 9p2000.u) Christian Schoenebeck
2026-10-19 14:27 ` [PULL 9/9] tests/9p-race: add Tread-on-dir race test Christian Schoenebeck
2026-10-20  6:42 ` [PULL 0/9] 9p queue 2026-10-19 Richard Henderson

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=24565308.6Emhk5qWAg@weasel \
    --to=qemu_oss@crudebyte.com \
    --cc=groug@kaod.org \
    --cc=peter.maydell@linaro.org \
    --cc=qemu-devel@nongnu.org \
    --cc=qemu-riscv@nongnu.org \
    --cc=thuth@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.