[Qemu-devel] QEMU + KVM PSCI and VCPU creation / destruction

[Qemu-devel] QEMU + KVM PSCI and VCPU creation / destruction

[Claudio Fontana]

Hello,

while heavily testing PSCI on QEMU+KVM during OSv enablement, I encountered, among others, the following issue:

I am running a test in which I boot an OS at EL1 under KVM, then boot a secondary VCPU,
then immediately call PSCI for a SYSTEM_RESET (reboot).

This loops over infinitely, or, as a matter of fact, until I run out of memory in the Foundation Model.

Now, before submitting another support request for the Model, I checked the code for the handling of PSCI, and it turns out that KVM handles the HVC and then sets an exit reason for QEMU to check,
which again sets the system_reset_requested to true, which causes a qemu_system_reset.

Now in there I see the call to qemu_devices_reset() and cpu_synchronize_all_post_reset(),
but are actually the VCPU destroyed? Is the VM destroyed? Or are new resources allocated at the next boot whenever PSCI asks for another VCPU to be booted via KVM_CREATE_VCPU etc?

If the resources associated to the VCPU (and VM?) are not freed, isn't this always going to cause leak in the host?

After around 3 hours of continuous PSCI secondary boot followed by SYSTEM_RESET I run out of memory on the host.

Thanks,

Claudio

Re: [Qemu-devel] QEMU + KVM PSCI and VCPU creation / destruction

[Peter Maydell]

On 29 June 2015 at 18:51, Claudio Fontana <claudio.fontana@huawei.com> wrote:
> but are actually the VCPU destroyed? Is the VM destroyed? Or are new
> resources allocated at the next boot whenever PSCI asks for another
> VCPU to be booted via KVM_CREATE_VCPU etc?

We don't destroy either the VCPUs or the VM. We will call
KVM_ARM_VCPU_INIT on the VCPUs to ask the kernel to reset them to
a clean state for us.

> If the resources associated to the VCPU (and VM?) are not freed,
> isn't this always going to cause leak in the host?
>
> After around 3 hours of continuous PSCI secondary boot followed by
> SYSTEM_RESET I run out of memory on the host.

This does sound like it's probably a leak somewhere, but you'll
need to track down what exactly is being leaked...

thanks
-- PMM

Re: [Qemu-devel] QEMU + KVM PSCI and VCPU creation / destruction

[Andrew Jones]

On Mon, Jun 29, 2015 at 07:51:15PM +0200, Claudio Fontana wrote:
> Hello,
> 
> while heavily testing PSCI on QEMU+KVM during OSv enablement, I encountered, among others, the following issue:
> 
> I am running a test in which I boot an OS at EL1 under KVM, then boot a secondary VCPU,
> then immediately call PSCI for a SYSTEM_RESET (reboot).
> 
> This loops over infinitely, or, as a matter of fact, until I run out of memory in the Foundation Model.
> 
> Now, before submitting another support request for the Model, I checked the code for the handling of PSCI, and it turns out that KVM handles the HVC and then sets an exit reason for QEMU to check,
> which again sets the system_reset_requested to true, which causes a qemu_system_reset.
> 
> Now in there I see the call to qemu_devices_reset() and cpu_synchronize_all_post_reset(),
> but are actually the VCPU destroyed? Is the VM destroyed? Or are new resources allocated at the next boot whenever PSCI asks for another VCPU to be booted via KVM_CREATE_VCPU etc?
> 
> If the resources associated to the VCPU (and VM?) are not freed, isn't this always going to cause leak in the host?
> 
> After around 3 hours of continuous PSCI secondary boot followed by SYSTEM_RESET I run out of memory on the host.

I can reproduce this with kvm-unit-tests. I don't see much sign of a
leak using tcg with x86 for the host, nor with using kvm on an aarch64
host. However using tcg on an aarch64 host shows memory getting used
pretty quickly. I didn't test with arm, only aarch64. I would just
provide the test, which is only

int main(void)
{
    assert(cpumask_weight(&cpu_present_mask) > 1);
    smp_boot_secondary(1, halt);
    psci_sys_reset();
    return 0;
}

but, in order to cleanly handle system reset, I ended up tweaking the
framework in several places. So I've created a branch for this, which
is here

https://github.com/rhdrjones/kvm-unit-tests/commits/arm/test-reset

On an aarch64 host, just do
./configure
make
export QEMU=<path-to-qemu>
$QEMU -machine virt,accel=tcg \
-device virtio-serial-device \
-device virtconsole,chardev=ctd \
-chardev testdev,id=ctd \
-display none -serial stdio \
-kernel arm/secondary-leak.flat \
-smp 2

I've expanded the command line because

  arm/run arm/secondary-leak.flat -smp 2

would use kvm by default on the aarch64 host, and we need tcg.
(maybe I'll add an arm/run command line switch for that someday)

The test will never output anything and never exit.  While it's
running just check 'free' several times in another term to see
that memory is going away, and then hit ^C on the test whenever
you want to quit.

drew