Re: GICv3 for MTTCG

[Andrey Shinkevich <andrey.shinkevich@huawei.com>]

On 6/17/21 8:44 PM, shashi.mallela@linaro.org wrote:
> Hi Andrey,
> 
> The issue doesnt seem related to ITS patchset as the implementation has
> no changes around MTTCG or vCPU configurations.
> 
> if this patchset were not applied(with only commit 3e9f48b),do you
> still see the hang issue?

No, I don't. Even with the patchset applied, the 'gic-version=2' turns 
the guest to normal running.
With the 'gic-version=3' and '-smp 3' (1,2 or 3 vCPUs), the guest starts 
and runs OK as well.

Andrey

> 
> Thanks
> Shashi
> 
> 
> On Thu, 2021-06-17 at 16:43 +0000, Andrey Shinkevich wrote:
>> Dear Shashi,
>>
>> I have applied the version 4 of the series "GICv3 LPI and ITS
>> feature
>> implementation" right after the commit 3e9f48b as before (because
>> the
>> GCCv7.5 is unavailable in the YUM repository for CentOS-7.9).
>>
>> The guest OS still hangs at its start when QEMU is configured with 4
>> or
>> more vCPUs (with 1 to 3 vCPUs the guest starts and runs OK and the
>> MTTCG
>> works properly):
>>
>> Welcome to EulerOS 2.0 ... (Initramfs)!
>>
>> …
>>
>> [  OK  ] Mounted Kernel Configuration File System.
>>
>> [  OK  ] Started udev Coldplug all Devices.
>>
>> [  OK  ] Reached target System Initialization.
>>
>> [  OK  ] Reached target Basic System.
>>
>>
>>
>> IT HANGS HERE
>>    (with 4 or more vCPUs)!!!
>>
>>
>> [  OK  ] Found device /dev/mapper/euleros-root.
>>
>> [  OK  ] Reached target Initrd Root Device.
>>
>> [  OK  ] Started dracut initqueue hook.
>>
>>            Starting File System Check on /dev/mapper/euleros-root...
>>
>> [  OK  ] Reached target Remote File Systems (Pre).
>>
>> [  OK  ] Reached target Remote File Systems.
>>
>> [  OK  ] Started File System Check on /dev/mapper/euleros-root.
>>
>>            Mounting /sysroot...
>>
>> [  OK  ] Mounted /sysroot.
>>
>> …
>>
>>
>> The back trace of threads in QEMU looks like a dead lock in MTTCG,
>> doesn't it?
>>
>> Thread 7 (Thread 0x7f476e489700 (LWP 24967)):
>>
...
>>
>> Thread 5 (Thread 0x7f461f9ff700 (LWP 24971)):
>>
...
>>
>> Thread 4 (Thread 0x7f461f1fe700 (LWP 24972)):
>>
...
>>
>> Thread 3 (Thread 0x7f461e9fd700 (LWP 24973)...
>>
>> Thread 2 (Thread 0x7f461e1fc700 (LWP 24974)):
>>
>> #0  0x00007f477c59ca35 in pthread_cond_wait@@GLIBC_2.3.2 () at
>> /lib64/libpthread.so.0
>>
>> ---Type <return> to continue, or q <return> to quit---
>>
>> #1  0x000055747d419b1d in qemu_cond_wait_impl (cond=0x55747fa626c0,
>> mutex=0x55747e04dc00 <qemu_global_mutex>, file=0x55747d5dbe5c
>> "../softmmu/cpus.c", line=417) at ../util/qemu-thread-posix.c:174
>>
>> #2  0x000055747d20ae36 in qemu_wait_io_event
>> (cpu=cpu@entry=0x55747f9fcf00) at ../softmmu/cpus.c:417
>>
>> #3  0x000055747d18d6a1 in mttcg_cpu_thread_fn
>> (arg=arg@entry=0x55747f9fcf00) at ../accel/tcg/tcg-accel-ops-
>> mttcg.c:98
>>
>> #4  0x000055747d419406 in qemu_thread_start (args=<optimized out>)
>> at
>> ../util/qemu-thread-posix.c:521
>>
>> #5  0x00007f477c598ea5 in start_thread () at /lib64/libpthread.so.0
>>
>> #6  0x00007f477c2c19fd in clone () at /lib64/libc.so.6
>>
>>
>> Thread 1 (Thread 0x7f4781db4d00 (LWP 24957)):
>>
...
>>
>> (gdb)
>>
>>
>> I run QEMU with virt-manager as this:
>>
>> qemu      7311     1 70 19:15 ?        00:00:05
>> /usr/local/bin/qemu-system-aarch64 -name
>> guest=EulerOS-2.8-Rich,debug-threads=on -S -object
>> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-95-
>> EulerOS-2.8-Rich/master-key.aes
>> -machine virt-6.1,accel=tcg,usb=off,dump-guest-core=off,gic-
>> version=3
>> -cpu max -drive
>> file=/usr/share/AAVMF/AAVMF_CODE.fd,if=pflash,format=raw,unit=0,reado
>> nly=on
>> -drive
>> file=/var/lib/libvirt/qemu/nvram/EulerOS-2.8-
>> Rich_VARS.fd,if=pflash,format=raw,unit=1
>> -m 4096 -smp 4,sockets=4,cores=1,threads=1
...
>>
>> The issue is reproducible and persists.
>> 1. Do you think that applying the series results in the dead lock in
>> MTTCG? Or it may be other reason?
>> 2. Which piece of QEMU source code should I investigate to locate the
>> issue?
>>
>> Best regards,
>> Andrey Shinkevich
>>
...
> 
>