Re: [PULL 00/30] Next patches

[Juan Quintela <quintela@redhat.com>]

Richard Henderson <richard.henderson@linaro.org> wrote:
> On 6/26/23 00:01, Juan Quintela wrote:
>> Richard Henderson <richard.henderson@linaro.org> wrote:
>>> On 6/22/23 18:54, Juan Quintela wrote:
>>>> The following changes since commit b455ce4c2f300c8ba47cba7232dd03261368a4cb:
>>>>     Merge tag 'q800-for-8.1-pull-request'
>>>> ofhttps://github.com/vivier/qemu-m68k  into staging (2023-06-22
>>>> 10:18:32 +0200)
>>>> are available in the Git repository at:
>>>>     https://gitlab.com/juan.quintela/qemu.git  tags/next-pull-request
>>>> for you to fetch changes up to
>>>> 23e4307eadc1497bd0a11ca91041768f15963b68:
>>>>     migration/rdma: Split qemu_fopen_rdma() into input/output
>>>> functions (2023-06-22 18:11:58 +0200)
>>>> ----------------------------------------------------------------
>>>> Migration Pull request (20230621) take 2
>>>> In this pull request the only change is fixing 32 bits complitaion
>>>> issue.
>>>> Please apply.
>>>> [take 1]
>>>> - fix for multifd thread creation (fabiano)
>>>> - dirtylimity (hyman)
>>>>     * migration-test will go on next PULL request, as it has failures.
>>>> - Improve error description (tejus)
>>>> - improve -incoming and set parameters before calling incoming (wei)
>>>> - migration atomic counters reviewed patches (quintela)
>>>> - migration-test refacttoring reviewed (quintela)
>>>
>>> New failure with check-cfi-x86_64:
>>>
>>> https://gitlab.com/qemu-project/qemu/-/jobs/4527202764#L188
>> First of all, is there a way to get to the test log?  In particular,
>> I
>> am interested in knowing at least what test has failed (yes,
>> migration-test don't tell you much more).
>> After a bit more wrestling, I have been able to get things compiling
>> with this command:
>> $ /mnt/code/qemu/full/configure --enable-cfi
>> --target-list=x86_64-softmmu --enable-cfi-debug --cc=clang --cxx=clang++
>> --disable-docs --enable-safe-stack --disable-slirp
>> It should basically be the one that check-cfi-x86_64 is using if I
>> understand the build recipes correctly (that is a BIG IF).
>> And it passes for me with flying colors.
>> Here I have Fedora38, builder has F37.
>> 
>>> /builds/qemu-project/qemu/build/pyvenv/bin/meson test  --no-rebuild -t
>>> 0  --num-processes 1 --print-errorlogs
>>>    1/350 qemu:qtest+qtest-x86_64 / qtest-x86_64/qom-test
>>>    OK 6.55s   8 subtests passed
>>> ▶   2/350 ERROR:../tests/qtest/migration-test.c:320:check_guests_ram:
>>> assertion failed: (bad == 0) ERROR
>>>    2/350 qemu:qtest+qtest-x86_64 / qtest-x86_64/migration-test
>>>    ERROR 151.99s   killed by signal 6 SIGABRT
>>>>>>
>>>      G_TEST_DBUS_DAEMON=/builds/qemu-project/qemu/tests/dbus-vmstate-daemon.sh
>>>      MALLOC_PERTURB_=3 QTEST_QEMU_IMG=./qemu-img
>>>      QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon
>>>      QTEST_QEMU_BINARY=./qemu-system-x86_64
>>>      /builds/qemu-project/qemu/build/tests/qtest/migration-test --tap
>>>     -k
>>> ――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
>>> stderr:
>>> qemu-system-x86_64: Unable to read from socket: Connection reset by peer
>> This is the interesting bit, why is the conection closed.
>> 
>>> Memory content inconsistency at 4f65000 first_byte = 30 last_byte = 2f
>>> current = 88 hit_edge = 1
>>> **
>>> ERROR:../tests/qtest/migration-test.c:320:check_guests_ram: assertion failed: (bad == 0)
>>>
>>> (test program exited with status code -6)
>> This makes zero sense, except if we haven't migrated all the guest
>> state, that it is what it has happened.
>> Is there a place on the web interface to see the full logs?  Or that
>> is
>> the only thing that the CI system stores?
>
> The "full logs" are
>
> https://gitlab.com/qemu-project/qemu/-/jobs/4527202764/artifacts/download?file_type=trace

Not useful.  I was hoping that there is something like when one runs
./tests/qtest/migration-test

Anyways, to make things faster:

- created
  /mnt/code/qemu/full/configure --enable-cfi
    --target-list=x86_64-softmmu --enable-cfi-debug --cc=clang --cxx=clang++
    --disable-docs --enable-safe-stack --disable-slirp

  worked as a charm.

- Your test run:

   qemu-system-x86_64: Unable to read from socket: Connection reset by peer
   one of the sides die, so anything else after that don't matter.

  And I don't understand what CFI is (and I don't rule out that
  posibility) or I can't understand how checking indirect functions call
  can make migration-test die without a single CFI error message?

- I tried myself CI pipeline, some exact source:

  https://gitlab.com/juan.quintela/qemu/-/commit/23e4307eadc1497bd0a11ca91041768f15963b68/pipelines?ref=sent%2Fmigration-20230621b

This is what fails:

https://gitlab.com/juan.quintela/qemu/-/jobs/4527782025
16/395 ERROR:../tests/qtest/qos-test.c:191:subprocess_run_one_test: child process (/x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-net-pci/virtio-net/virtio-net-tests/vhost-user/reconnect/subprocess [4569]) failed unexpectedly ERROR         
 16/395 qemu:qtest+qtest-x86_64 / qtest-x86_64/qos-test                    ERROR          27.46s   killed by signal 6 SIGABRT
>>> MALLOC_PERTURB_=92 QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon QTEST_QEMU_BINARY=./qemu-system-x86_64 QTEST_QEMU_IMG=./qemu-img G_TEST_DBUS_DAEMON=/builds/juan.quintela/qemu/tests/dbus-vmstate-daemon.sh /builds/juan.quintela/qemu/build/tests/qtest/qos-test --tap -k
――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
stderr:
Vhost user backend fails to broadcast fake RARP
qemu-system-x86_64: -chardev socket,id=chr-reconnect,path=/tmp/vhost-test-8XUX61/reconnect.sock,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-8XUX61/reconnect.sock,server=on
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
**
ERROR:../tests/qtest/vhost-user-test.c:890:wait_for_rings_started: assertion failed (ctpop64(s->rings) == count): (1 == 2)
**
ERROR:../tests/qtest/qos-test.c:191:subprocess_run_one_test: child
process
(/x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-net-pci/virtio-net/virtio-net-tests/vhost-user/reconnect/subprocess
[4569]) failed unexpectedly

vhost?  virtio-queue?  in a non migration test?

I don't know what is going on, but this is weird.

Do we have a way to run on that image:

./tests/qtest/migration-test

in a loop until it fails, and at least see what test is failing?

Later, Juan.