Core dump happened when starting a VM on arm64 server

Core dump happened when starting a VM on arm64 server

[Haibo Xu]

Hi,

I met a qemu core dump issue when starting a VM with cpu feature
"pmu=on" on an arm server.
The commands to start the machine is:

  ./qemu-system-aarch64 \
           -cpu host,pmu=on -M virt,accel=kvm,gic-version=3 -nographic
-m 2048M \
           -kernel ./Image \
           -initrd /boot/initrd.img-5.6.0-rc2+ \
           -append "root=/dev/vda rw console=ttyAMA0" -nodefaults -serial stdio\
           -drive if=none,file=./xenial.rootfs.ext4,id=hd0,format=raw \
           -device virtio-blk-device,drive=hd0


And here is the stack dump:

 Core was generated by `./qemu-system-aarch64 -cpu host,pmu=on -M
virt,accel=kvm,gic-version=3 -nograph'.
 Program terminated with signal SIGSEGV, Segmentation fault.
 #0  kvm_ioctl (s=0x0, type=type@entry=44547) at
/root/Downloads/qemu-git/accel/kvm/kvm-all.c:2509
 2509        ret = ioctl(s->fd, type, arg);
 [Current thread is 1 (Thread 0xffffa5108010 (LWP 22057))]
 (gdb) bt
 #0  0x0000aaaadc432950 in kvm_ioctl (s=0x0, type=type@entry=44547) at
/root/Downloads/qemu-git/accel/kvm/kvm-all.c:2509
 #1  0x0000aaaadc432adc in kvm_check_extension (s=<optimized out>,
extension=extension@entry=126) at
/root/Downloads/qemu-git/accel/kvm/kvm-all.c:866
 #2  0x0000aaaadc541ff0 in kvm_arm_pmu_supported (cpu=<optimized out>)
at /root/Downloads/qemu-git/target/arm/kvm.c:212
 #3  0x0000aaaadc53a08c in arm_set_pmu (obj=<optimized out>,
value=<optimized out>, errp=0xfffff2fba6b0) at
/root/Downloads/qemu-git/target/arm/cpu.c:1113
 #4  0x0000aaaadc88facc in property_set_bool (obj=0xaaab0b61a0f0,
v=<optimized out>, name=<optimized out>, opaque=0xaaab0b627690,
errp=0xfffff2fba6b0)
     at /root/Downloads/qemu-git/qom/object.c:2162
 #5  0x0000aaaadc892af4 in object_property_parse
(obj=obj@entry=0xaaab0b61a0f0, string=<optimized out>,
name=0xaaab0b527d00 "pmu", errp=errp@entry=0xfffff2fba6b0)
     at /root/Downloads/qemu-git/qom/object.c:1552
 #6  0x0000aaaadc892bbc in object_apply_global_props
(obj=0xaaab0b61a0f0, props=0xaaab0b473a60, errp=0xaaaadd003930
<error_fatal>)
     at /root/Downloads/qemu-git/qom/object.c:410
 #7  0x0000aaaadc891a64 in object_post_init_with_type
(ti=0xaaab0b20e9f0, obj=0xaaab0b61a0f0) at
/root/Downloads/qemu-git/qom/object.c:383
 #8  0x0000aaaadc891a64 in object_initialize_with_type
(data=data@entry=0xaaab0b61a0f0, size=<optimized out>,
type=type@entry=0xaaab0b212a40)
     at /root/Downloads/qemu-git/qom/object.c:517
 #9  0x0000aaaadc891ba4 in object_new_with_type (type=0xaaab0b212a40)
at /root/Downloads/qemu-git/qom/object.c:681
 #10 0x0000aaaadc4bfd10 in machvirt_init (machine=0xaaaadd003930
<error_fatal>) at /root/Downloads/qemu-git/hw/arm/virt.c:1804
 #11 0x0000aaaadc69ec5c in machine_run_board_init
(machine=0xaaab0b47e950) at
/root/Downloads/qemu-git/hw/core/machine.c:1132
 #12 0x0000aaaadc51f50c in qemu_init (argc=<optimized out>,
argv=<optimized out>, envp=<optimized out>) at
/root/Downloads/qemu-git/softmmu/vl.c:4347
 #13 0x0000aaaadc3d2abc in main (argc=<optimized out>, argv=<optimized
out>, envp=<optimized out>) at
/root/Downloads/qemu-git/softmmu/main.c:48
 (gdb)


The root cause is in the arm_get_pmu() operation which was introduced
in ae502508f83.
After deleting the KVM feature probe operation in this function, the
issue can be fixed.

 diff --git a/target/arm/cpu.c b/target/arm/cpu.c
 index 3801e25b79..ff18db8fd4 100644
 --- a/target/arm/cpu.c
 +++ b/target/arm/cpu.c
 @@ -1110,10 +1110,6 @@ static void arm_set_pmu(Object *obj, bool
value, Error **errp)
      ARMCPU *cpu = ARM_CPU(obj);

      if (value) {
 -        if (kvm_enabled() && !kvm_arm_pmu_supported(CPU(cpu))) {
 -            error_setg(errp, "'pmu' feature not supported by KVM on
this host");
 -            return;
 -        }
          set_feature(&cpu->env, ARM_FEATURE_PMU);
      } else {
          unset_feature(&cpu->env, ARM_FEATURE_PMU);


According to the Qemu document(docs/system/arm/cpu-features.rst), the
pmu is turned on by default when using KVM mode on a V8 host machine,
which means the pmu=on is redundant when starting a VM with PMU support.

  target/arm/kvm64.c
  672     /*
  673      * We can assume any KVM supporting CPU is at least a v8
  674      * with VFPv4+Neon; this in turn implies most of the other
  675      * feature bits.
  676      */
  677     features |= 1ULL << ARM_FEATURE_V8;
  678     features |= 1ULL << ARM_FEATURE_NEON;
  679     features |= 1ULL << ARM_FEATURE_AARCH64;
  680     features |= 1ULL << ARM_FEATURE_PMU;
  681     features |= 1ULL << ARM_FEATURE_GENERIC_TIMER;

But I think we need a better way to handle this when "pmu=on" is
present in the command line, not just trigger a core dump(qemu binary
was built from the current master branch).
Any comments?

Regards,
Haibo

Re: Core dump happened when starting a VM on arm64 server

[Andrew Jones]

On Thu, Jun 11, 2020 at 04:46:45PM +0800, Haibo Xu wrote:
> Hi,
> 
> I met a qemu core dump issue when starting a VM with cpu feature
> "pmu=on" on an arm server.
> The commands to start the machine is:
> 
>   ./qemu-system-aarch64 \
>            -cpu host,pmu=on -M virt,accel=kvm,gic-version=3 -nographic
> -m 2048M \
>            -kernel ./Image \
>            -initrd /boot/initrd.img-5.6.0-rc2+ \
>            -append "root=/dev/vda rw console=ttyAMA0" -nodefaults -serial stdio\
>            -drive if=none,file=./xenial.rootfs.ext4,id=hd0,format=raw \
>            -device virtio-blk-device,drive=hd0
> 
> 
> And here is the stack dump:
> 
>  Core was generated by `./qemu-system-aarch64 -cpu host,pmu=on -M
> virt,accel=kvm,gic-version=3 -nograph'.
>  Program terminated with signal SIGSEGV, Segmentation fault.
>  #0  kvm_ioctl (s=0x0, type=type@entry=44547) at

s=0x0 means cpu->kvm_state is NULL

> The root cause is in the arm_get_pmu() operation which was introduced
> in ae502508f83.

Actually the root cause is d70c996df23f ("target/arm/kvm: Use
CPUState::kvm_state in kvm_arm_pmu_supported()"). ae502508f83 used
the machine kvm_state, not the cpu kvm_state, and that allows pmu=on
to work. d70c996df23f changed that saying that "KVMState is already
accessible via CPUState::kvm_state, use it.", but I'm not sure why,
since kvm_init_vcpu() doesn't run until the vcpu thread is created.

Philippe?

Thanks,
drew

Re: Core dump happened when starting a VM on arm64 server

[Philippe Mathieu-Daudé]

On 6/11/20 11:14 AM, Andrew Jones wrote:
> On Thu, Jun 11, 2020 at 04:46:45PM +0800, Haibo Xu wrote:
>> Hi,
>>
>> I met a qemu core dump issue when starting a VM with cpu feature
>> "pmu=on" on an arm server.
>> The commands to start the machine is:
>>
>>   ./qemu-system-aarch64 \
>>            -cpu host,pmu=on -M virt,accel=kvm,gic-version=3 -nographic
>> -m 2048M \
>>            -kernel ./Image \
>>            -initrd /boot/initrd.img-5.6.0-rc2+ \
>>            -append "root=/dev/vda rw console=ttyAMA0" -nodefaults -serial stdio\
>>            -drive if=none,file=./xenial.rootfs.ext4,id=hd0,format=raw \
>>            -device virtio-blk-device,drive=hd0
>>
>>
>> And here is the stack dump:
>>
>>  Core was generated by `./qemu-system-aarch64 -cpu host,pmu=on -M
>> virt,accel=kvm,gic-version=3 -nograph'.
>>  Program terminated with signal SIGSEGV, Segmentation fault.
>>  #0  kvm_ioctl (s=0x0, type=type@entry=44547) at
> 
> s=0x0 means cpu->kvm_state is NULL
> 
>> The root cause is in the arm_get_pmu() operation which was introduced
>> in ae502508f83.
> 
> Actually the root cause is d70c996df23f ("target/arm/kvm: Use
> CPUState::kvm_state in kvm_arm_pmu_supported()"). ae502508f83 used
> the machine kvm_state, not the cpu kvm_state, and that allows pmu=on
> to work. d70c996df23f changed that saying that "KVMState is already
> accessible via CPUState::kvm_state, use it.", but I'm not sure why,
> since kvm_init_vcpu() doesn't run until the vcpu thread is created.
> 
> Philippe?

Sorry for some reason I missed this email. I'll look at this today.

> 
> Thanks,
> drew
> 

Re: Core dump happened when starting a VM on arm64 server

[Philippe Mathieu-Daudé]

On 6/17/20 10:23 AM, Philippe Mathieu-Daudé wrote:
> On 6/11/20 11:14 AM, Andrew Jones wrote:
>> On Thu, Jun 11, 2020 at 04:46:45PM +0800, Haibo Xu wrote:
>>> Hi,
>>>
>>> I met a qemu core dump issue when starting a VM with cpu feature
>>> "pmu=on" on an arm server.
>>> The commands to start the machine is:
>>>
>>>   ./qemu-system-aarch64 \
>>>            -cpu host,pmu=on -M virt,accel=kvm,gic-version=3 -nographic
>>> -m 2048M \
>>>            -kernel ./Image \
>>>            -initrd /boot/initrd.img-5.6.0-rc2+ \
>>>            -append "root=/dev/vda rw console=ttyAMA0" -nodefaults -serial stdio\
>>>            -drive if=none,file=./xenial.rootfs.ext4,id=hd0,format=raw \
>>>            -device virtio-blk-device,drive=hd0
>>>
>>>
>>> And here is the stack dump:
>>>
>>>  Core was generated by `./qemu-system-aarch64 -cpu host,pmu=on -M
>>> virt,accel=kvm,gic-version=3 -nograph'.
>>>  Program terminated with signal SIGSEGV, Segmentation fault.
>>>  #0  kvm_ioctl (s=0x0, type=type@entry=44547) at
>>
>> s=0x0 means cpu->kvm_state is NULL
>>
>>> The root cause is in the arm_get_pmu() operation which was introduced
>>> in ae502508f83.
>>
>> Actually the root cause is d70c996df23f ("target/arm/kvm: Use
>> CPUState::kvm_state in kvm_arm_pmu_supported()"). ae502508f83 used
>> the machine kvm_state, not the cpu kvm_state, and that allows pmu=on
>> to work. d70c996df23f changed that saying that "KVMState is already
>> accessible via CPUState::kvm_state, use it.", but I'm not sure why,
>> since kvm_init_vcpu() doesn't run until the vcpu thread is created.
>>
>> Philippe?
> 
> Sorry for some reason I missed this email. I'll look at this today.

Quick reproducer:

$ qemu-system-aarch64 -cpu host,pmu=on -M virt,accel=kvm,gic-version=3
Segmentation fault (core dumped)

Eduardo, I thought we had a 'machine' qtest testing for various
combination of properties, but I can't find it, do you remember?
Or maybe it was Thomas? Or Markus? =)

> 
>>
>> Thanks,
>> drew
>>
> 

Re: Core dump happened when starting a VM on arm64 server

[Thomas Huth]

On 17/06/2020 12.32, Philippe Mathieu-Daudé wrote:
> On 6/17/20 10:23 AM, Philippe Mathieu-Daudé wrote:
>> On 6/11/20 11:14 AM, Andrew Jones wrote:
>>> On Thu, Jun 11, 2020 at 04:46:45PM +0800, Haibo Xu wrote:
>>>> Hi,
>>>>
>>>> I met a qemu core dump issue when starting a VM with cpu feature
>>>> "pmu=on" on an arm server.
>>>> The commands to start the machine is:
>>>>
>>>>   ./qemu-system-aarch64 \
>>>>            -cpu host,pmu=on -M virt,accel=kvm,gic-version=3 -nographic
>>>> -m 2048M \
>>>>            -kernel ./Image \
>>>>            -initrd /boot/initrd.img-5.6.0-rc2+ \
>>>>            -append "root=/dev/vda rw console=ttyAMA0" -nodefaults -serial stdio\
>>>>            -drive if=none,file=./xenial.rootfs.ext4,id=hd0,format=raw \
>>>>            -device virtio-blk-device,drive=hd0
>>>>
>>>>
>>>> And here is the stack dump:
>>>>
>>>>  Core was generated by `./qemu-system-aarch64 -cpu host,pmu=on -M
>>>> virt,accel=kvm,gic-version=3 -nograph'.
>>>>  Program terminated with signal SIGSEGV, Segmentation fault.
>>>>  #0  kvm_ioctl (s=0x0, type=type@entry=44547) at
>>>
>>> s=0x0 means cpu->kvm_state is NULL
>>>
>>>> The root cause is in the arm_get_pmu() operation which was introduced
>>>> in ae502508f83.
>>>
>>> Actually the root cause is d70c996df23f ("target/arm/kvm: Use
>>> CPUState::kvm_state in kvm_arm_pmu_supported()"). ae502508f83 used
>>> the machine kvm_state, not the cpu kvm_state, and that allows pmu=on
>>> to work. d70c996df23f changed that saying that "KVMState is already
>>> accessible via CPUState::kvm_state, use it.", but I'm not sure why,
>>> since kvm_init_vcpu() doesn't run until the vcpu thread is created.
>>>
>>> Philippe?
>>
>> Sorry for some reason I missed this email. I'll look at this today.
> 
> Quick reproducer:
> 
> $ qemu-system-aarch64 -cpu host,pmu=on -M virt,accel=kvm,gic-version=3
> Segmentation fault (core dumped)
> 
> Eduardo, I thought we had a 'machine' qtest testing for various
> combination of properties, but I can't find it, do you remember?
> Or maybe it was Thomas? Or Markus? =)

You probably remember the scripts/device-crash-test script? ... that's
a) only testing -device options as far as I know, and b) is not run
automatically during "make check", so it certainly is not suitable to
detect these kind of errors automatically.

 Thomas

Re: Core dump happened when starting a VM on arm64 server

[Andrew Jones]

On Wed, Jun 17, 2020 at 12:32:09PM +0200, Philippe Mathieu-Daudé wrote:
> On 6/17/20 10:23 AM, Philippe Mathieu-Daudé wrote:
> > On 6/11/20 11:14 AM, Andrew Jones wrote:
> >> On Thu, Jun 11, 2020 at 04:46:45PM +0800, Haibo Xu wrote:
> >>> Hi,
> >>>
> >>> I met a qemu core dump issue when starting a VM with cpu feature
> >>> "pmu=on" on an arm server.
> >>> The commands to start the machine is:
> >>>
> >>>   ./qemu-system-aarch64 \
> >>>            -cpu host,pmu=on -M virt,accel=kvm,gic-version=3 -nographic
> >>> -m 2048M \
> >>>            -kernel ./Image \
> >>>            -initrd /boot/initrd.img-5.6.0-rc2+ \
> >>>            -append "root=/dev/vda rw console=ttyAMA0" -nodefaults -serial stdio\
> >>>            -drive if=none,file=./xenial.rootfs.ext4,id=hd0,format=raw \
> >>>            -device virtio-blk-device,drive=hd0
> >>>
> >>>
> >>> And here is the stack dump:
> >>>
> >>>  Core was generated by `./qemu-system-aarch64 -cpu host,pmu=on -M
> >>> virt,accel=kvm,gic-version=3 -nograph'.
> >>>  Program terminated with signal SIGSEGV, Segmentation fault.
> >>>  #0  kvm_ioctl (s=0x0, type=type@entry=44547) at
> >>
> >> s=0x0 means cpu->kvm_state is NULL
> >>
> >>> The root cause is in the arm_get_pmu() operation which was introduced
> >>> in ae502508f83.
> >>
> >> Actually the root cause is d70c996df23f ("target/arm/kvm: Use
> >> CPUState::kvm_state in kvm_arm_pmu_supported()"). ae502508f83 used
> >> the machine kvm_state, not the cpu kvm_state, and that allows pmu=on
> >> to work. d70c996df23f changed that saying that "KVMState is already
> >> accessible via CPUState::kvm_state, use it.", but I'm not sure why,
> >> since kvm_init_vcpu() doesn't run until the vcpu thread is created.
> >>
> >> Philippe?
> > 
> > Sorry for some reason I missed this email. I'll look at this today.
> 
> Quick reproducer:
> 
> $ qemu-system-aarch64 -cpu host,pmu=on -M virt,accel=kvm,gic-version=3
> Segmentation fault (core dumped)
> 
> Eduardo, I thought we had a 'machine' qtest testing for various
> combination of properties, but I can't find it, do you remember?
> Or maybe it was Thomas? Or Markus? =)

For arm cpu properties we have tests/qtest/arm-cpu-features. We can
add a regression test for this and other properties there. I just
tried it. I needed to add a new macro (see below), but otherwise it
worked to reproduce the problem.

Thanks,
drew

diff --git a/tests/qtest/arm-cpu-features.c b/tests/qtest/arm-cpu-features.c
index 469217367661..d6bdbc171893 100644
--- a/tests/qtest/arm-cpu-features.c
+++ b/tests/qtest/arm-cpu-features.c
@@ -159,16 +159,35 @@ static bool resp_get_feature(QDict *resp, const char *feature)
     qobject_unref(_resp);                                              \
 })
 
-#define assert_feature(qts, cpu_type, feature, expected_value)         \
+#define resp_assert_feature(resp, feature, expected_value)             \
 ({                                                                     \
-    QDict *_resp, *_props;                                             \
+    QDict *_props;                                                     \
                                                                        \
-    _resp = do_query_no_props(qts, cpu_type);                          \
     g_assert(_resp);                                                   \
     g_assert(resp_has_props(_resp));                                   \
     _props = resp_get_props(_resp);                                    \
     g_assert(qdict_get(_props, feature));                              \
     g_assert(qdict_get_bool(_props, feature) == (expected_value));     \
+})
+
+#define assert_feature(qts, cpu_type, feature, expected_value)         \
+({                                                                     \
+    QDict *_resp;                                                      \
+                                                                       \
+    _resp = do_query_no_props(qts, cpu_type);                          \
+    g_assert(_resp);                                                   \
+    resp_assert_feature(_resp, feature, expected_value);               \
+    qobject_unref(_resp);                                              \
+})
+
+#define assert_set_feature(qts, cpu_type, feature, value)              \
+({                                                                     \
+    const char *_fmt = (value) ? "{ %s: true }" : "{ %s: false }";     \
+    QDict *_resp;                                                      \
+                                                                       \
+    _resp = do_query(qts, cpu_type, _fmt, feature);                    \
+    g_assert(_resp);                                                   \
+    resp_assert_feature(_resp, feature, value);                        \
     qobject_unref(_resp);                                              \
 })
 
@@ -464,7 +483,10 @@ static void test_query_cpu_model_expansion_kvm(const void *data)
         return;
     }
 
+    /* Enabling and disabling kvm-no-adjvtime should always work. */
     assert_has_feature_disabled(qts, "host", "kvm-no-adjvtime");
+    assert_set_feature(qts, "host", "kvm-no-adjvtime", true);
+    assert_set_feature(qts, "host", "kvm-no-adjvtime", false);
 
     if (g_str_equal(qtest_get_arch(), "aarch64")) {
         bool kvm_supports_sve;
@@ -475,7 +497,11 @@ static void test_query_cpu_model_expansion_kvm(const void *data)
         char *error;
 
         assert_has_feature_enabled(qts, "host", "aarch64");
+
+        /* Enabling and disabling pmu should always work. */
         assert_has_feature_enabled(qts, "host", "pmu");
+        assert_set_feature(qts, "host", "pmu", true);
+        assert_set_feature(qts, "host", "pmu", false);
 
         assert_error(qts, "cortex-a15",
             "We cannot guarantee the CPU type 'cortex-a15' works "

Re: Core dump happened when starting a VM on arm64 server

[Philippe Mathieu-Daudé]

On 6/17/20 12:32 PM, Philippe Mathieu-Daudé wrote:
> On 6/17/20 10:23 AM, Philippe Mathieu-Daudé wrote:
>> On 6/11/20 11:14 AM, Andrew Jones wrote:
>>> On Thu, Jun 11, 2020 at 04:46:45PM +0800, Haibo Xu wrote:
>>>> Hi,
>>>>
>>>> I met a qemu core dump issue when starting a VM with cpu feature
>>>> "pmu=on" on an arm server.
>>>> The commands to start the machine is:
>>>>
>>>>   ./qemu-system-aarch64 \
>>>>            -cpu host,pmu=on -M virt,accel=kvm,gic-version=3 -nographic
>>>> -m 2048M \
>>>>            -kernel ./Image \
>>>>            -initrd /boot/initrd.img-5.6.0-rc2+ \
>>>>            -append "root=/dev/vda rw console=ttyAMA0" -nodefaults -serial stdio\
>>>>            -drive if=none,file=./xenial.rootfs.ext4,id=hd0,format=raw \
>>>>            -device virtio-blk-device,drive=hd0
>>>>
>>>>
>>>> And here is the stack dump:
>>>>
>>>>  Core was generated by `./qemu-system-aarch64 -cpu host,pmu=on -M
>>>> virt,accel=kvm,gic-version=3 -nograph'.
>>>>  Program terminated with signal SIGSEGV, Segmentation fault.
>>>>  #0  kvm_ioctl (s=0x0, type=type@entry=44547) at
>>>
>>> s=0x0 means cpu->kvm_state is NULL
>>>
>>>> The root cause is in the arm_get_pmu() operation which was introduced
>>>> in ae502508f83.
>>>
>>> Actually the root cause is d70c996df23f ("target/arm/kvm: Use
>>> CPUState::kvm_state in kvm_arm_pmu_supported()"). ae502508f83 used
>>> the machine kvm_state, not the cpu kvm_state, and that allows pmu=on
>>> to work. d70c996df23f changed that saying that "KVMState is already
>>> accessible via CPUState::kvm_state, use it.", but I'm not sure why,
>>> since kvm_init_vcpu() doesn't run until the vcpu thread is created.
>>>
>>> Philippe?
>>
>> Sorry for some reason I missed this email. I'll look at this today.
> 
> Quick reproducer:
> 
> $ qemu-system-aarch64 -cpu host,pmu=on -M virt,accel=kvm,gic-version=3
> Segmentation fault (core dumped)

Fix sent:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg713249.html