Re: [PULL 00/13] 9p queue 2020-10-23

[Christian Schoenebeck <qemu_oss@crudebyte.com>]

On Dienstag, 27. Oktober 2020 11:26:53 CET Dr. David Alan Gilbert wrote:
> * Christian Schoenebeck (qemu_oss@crudebyte.com) wrote:
> > On Dienstag, 27. Oktober 2020 10:06:53 CET Dr. David Alan Gilbert wrote:
> > > * Greg Kurz (groug@kaod.org) wrote:
> > > > On Mon, 26 Oct 2020 13:48:37 +0100
> > > > 
> > > > Christian Schoenebeck <qemu_oss@crudebyte.com> wrote:
> > > > > On Montag, 26. Oktober 2020 11:33:42 CET Peter Maydell wrote:
> > > > > > On Fri, 23 Oct 2020 at 12:46, Christian Schoenebeck
> > > > > > 
> > > > > > <qemu_oss@crudebyte.com> wrote:
> > > > > > > The following changes since commit
> > 
> > 4c5b97bfd0dd54dc27717ae8d1cd10e14eef1430:
> > > > > > >   Merge remote-tracking branch
> > > > > > >   'remotes/kraxel/tags/modules-20201022-pull-request' into
> > > > > > >   staging
> > > > > > >   (2020-10-22 12:33:21 +0100)>
> > > > > > > 
> > > > > > > are available in the Git repository at:
> > > > > > >   https://github.com/cschoenebeck/qemu.git tags/pull-9p-20201023
> > > > > > > 
> > > > > > > for you to fetch changes up to
> > 
> > ee01926a11b1f9bffcd6cdec0961dd9d1882da71:
> > > > > > >   tests/9pfs: add local Tunlinkat hard link test (2020-10-22
> > > > > > >   20:26:33
> > > > > > >   +0200)
> > > > > > > 
> > > > > > > ----------------------------------------------------------------
> > > > > > > 9pfs: more tests using local fs driver
> > > > > > > 
> > > > > > > Only 9pfs test case changes this time:
> > > > > > > 
> > > > > > > * Refactor: Rename functions to make top-level test functions
> > > > > > > fs_*()
> > > > > > > 
> > > > > > >   easily distinguishable from utility test functions do_*().
> > > > > > > 
> > > > > > > * Refactor: Drop unnecessary function arguments in utility test
> > > > > > > 
> > > > > > >   functions.
> > > > > > > 
> > > > > > > * More test cases using the 9pfs 'local' filesystem driver
> > > > > > > backend,
> > > > > > > 
> > > > > > >   namely for the following 9p requests: Tunlinkat, Tlcreate,
> > > > > > >   Tsymlink
> > > > > > >   and Tlink.
> > > > > > > 
> > > > > > > ----------------------------------------------------------------
> > > > > > 
> > > > > > I get a 'make check' failure on x86-64 Linux host:
> > > > > > 
> > > > > > PASS 54 qtest-x86_64: qos-test
> > > > > > /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-9p-pci/virtio-
> > > > > > 9p/v
> > > > > > irtio- 9p-tests/local/config PASS 55 qtest-x86_64: qos-test
> > > > > > /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-9p-pci/virtio-
> > > > > > 9p/v
> > > > > > irtio- 9p-tests/local/create_dir PASS 56 qtest-x86_64: qos-test
> > > > > > /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-9p-pci/virtio-
> > > > > > 9p/v
> > > > > > irtio- 9p-tests/local/unlinkat_dir PASS 57 qtest-x86_64: qos-test
> > > > > > /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-9p-pci/virtio-
> > > > > > 9p/v
> > > > > > irtio- 9p-tests/local/create_file PASS 58 qtest-x86_64: qos-test
> > > > > > /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-9p-pci/virtio-
> > > > > > 9p/v
> > > > > > irtio- 9p-tests/local/unlinkat_file PASS 59 qtest-x86_64: qos-test
> > > > > > /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-9p-pci/virtio-
> > > > > > 9p/v
> > > > > > irtio- 9p-tests/local/symlink_file Received response 7 (RLERROR)
> > > > > > instead of 73 (RMKDIR)
> > > > > > Rlerror has errno 2 (No such file or directory)
> > > > > > **
> > > > > > ERROR:../../tests/qtest/virtio-9p-test.c:300:v9fs_req_recv:
> > > > > > assertion
> > > > > > failed (hdr.id == id): (7 == 73)
> > > > 
> > > > Not sure this is related to this PR actually. Dave Gilbert reported on
> > > > irc
> > > > that he encountered a similar issue with 'make -j check', likely
> > > > without
> > > > these patches.
> > > 
> > > I was running on current master as of yesterday; no 9p specific patches.
> > > 
> > > Dave
> > 
> > They might be related as the "local/create_dir" test is already merged,
> > but
> > hard to say reliably without any data.
> > 
> > How is reproducibility, sometimes / always?
> 
> I think I was seeing a few different errors; but I was running make
> check -j 32 ish
> 
> Dave
> 

Ok, I understand, but how frequently are you able to trigger one of the test 
failures there? Does it happen like every time, or rather just once every xth 
run or so?

I'm running the qtests multi-threaded in a loop for several hours now, but so 
far I was unable to hit any test failure:

#/bin/sh
i=0
while true; do
  i=$[$i+1]
  echo '**************************************************'
  echo "* RUN $i                                         *"
  echo '**************************************************'
  make check-qtest -j32 V=1
  if [[ "$?" -ne 0 ]]; then
    echo "Aborted after $i runs due to failure"
    break
  fi
done

If you say it only happens once in a while, then I let it go for a day or 
more. However if it happens there quite frequently, then I guess I have to 
look into another aspect instead, like e.g. differences in the glib version.

> > > > > > ERROR qtest-x86_64: qos-test - Bail out!
> > > > > > ERROR:../../tests/qtest/virtio-9ptest.c:300:v9fs_req_recv:
> > > > > > assertion
> > > > > > failed (hdr.id == id): (7 == 73)
> > > > > > Makefile.mtest:3953: recipe for target 'run-test-492' failed
> > > > > > 
> > > > > > 
> > > > > > thanks
> > > > > > -- PMM
> > > > > 
> > > > > So the 9p server is already failing to create the test case
> > > > > directory
> > > > > "./qtest-9p-local/05/" relative to your current working directory.
> > > > > 
> > > > > I would appreciate to get more info when you have some free cycles,
> > > > > as
> > > > > I'm
> > > > > unable to reproduce this on any system unfortunately. But no hurry
> > > > > as
> > > > > these tests only become relevant actually for QEMU 6.
> > > > > 
> > > > > What puzzles me is that the previous test cases succeeded there,
> > > > > which
> > > > > all
> > > > > 
> > > > > create their own test directory in the same way:
> > > > > 	./qtest-9p-local/01/
> > > > > 	./qtest-9p-local/02/  (<-- dir vanishes after that test completed)
> > > > > 	./qtest-9p-local/03/
> > > > > 	./qtest-9p-local/04/
> > > > > 	...
> > > > > 
> > > > > How does the "./qtest-9p-local/" directory look like after that
> > > > > "local/symlink_file" test failed there? You can use this shortcut:
> > > > > 
> > > > > export QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64
> > > > > cd build
> > > > > tests/qtest/qos-test --verbose
> > > > > ls -l qtest-9p-local
> > > > > 
> > > > > That latter qos-test run will also output the assembled qemu command
> > > > > line the 9p local tests would run with, which might also be helpful,
> > > > > e.g. the relevant output would be something like this:
> > > > > 
> > > > > GTest: run:
> > > > > /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-9p-pci/virtio-9p
> > > > > /vi
> > > > > rtio-9p-tests/local/config (MSG: starting QEMU: exec
> > > > > x86_64-softmmu/qemu-system-x86_64 -qtest unix:/tmp/qtest-7428.sock
> > > > > -qtest-log /dev/null -chardev
> > > > > socket,path=/tmp/qtest-7428.qmp,id=char0
> > > > > -mon chardev=char0,mode=control -display none -M pc  -fsdev
> > > > > local,id=fsdev0,path='/home/me/git/qemu/build/qtest-9p-local',securi
> > > > > ty_
> > > > > model=mapped-xattr -device
> > > > > virtio-9p-pci,fsdev=fsdev0,addr=04.0,mount_tag=qtest -accel qtest)
> > > > > 
> > > > > Would probably the test succeed if run alone?
> > > > > 
> > > > > tests/qtest/qos-test -p
> > > > > /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-9p-pci/virtio-9p
> > > > > /vi
> > > > > rtio-9p-tests/local/symlink_file
> > > > > 
> > > > > Best regards,
> > > > > Christian Schoenebeck
> > 
> > Best regards,
> > Christian Schoenebeck