[Qemu-devel] Savevm and loadvm not working

[Qemu-devel] Savevm and loadvm not working

[sridhar kulkarni]

Hi,
I am new bee to snapshot feature and how to use it correctly. My requirement is simple, in that I want to snapshot the running VM and save the snapshot file. Using the snapshot file, I want to boot the VM directly to snapshot state.I came across the qemu monitors "savevm" and "loadvm" commands. The following are the steps I follow to create snapshot and load the snapshot,Step 1: Launch VM using below command, 
qemu-system-arm -M arm-machine -m 512M -name my_name -kernel main.rbx -serial pty -serial vc -serial vc -serial vc -drive if=none,format=qcow2,file=/home/sridhar/qemu_disk_image/dummy.qcow2 

Step 2: Save the snapshot (issue savevm from qemu monitor)
        --> savevm my_snapshot

Step 3 Launch the VM using snapshot image saved in step 2,
qemu-system-arm -M arm-machine -m 512M -name my_name -kernel main.rbx -serial pty -serial vc -serial vc -serial vc -drive if=none,format=qcow2,file=/home/sridhar/qemu_disk_image/dummy.qcow2 -loadvm my_snapshot
With this approach, I am seeing that RAM contents are not getting saved when I issue "savevm" command. I have copied the part of the file, when the function "ram_control_save_page" returns "RAM_SAVE_CONTROL_NOT_SUPP" and hence the page is not getting saved.

size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset,
                             ram_addr_t offset, size_t size,
                             uint64_t *bytes_sent)
{
    if (f->hooks && f->hooks->save_page) {
        int ret = f->hooks->save_page(f, f->opaque, block_offset,
                                      offset, size, bytes_sent);

        if (ret != RAM_SAVE_CONTROL_DELAYED) {
            if (bytes_sent && *bytes_sent > 0) {
                qemu_update_position(f, *bytes_sent);
            } else if (ret < 0) {
                qemu_file_set_error(f, ret);
            }
        }

        return ret;
    }

    return RAM_SAVE_CONTROL_NOT_SUPP;
}

Is there anything that I am missing here in the understanding "savevm" and "loadvm" commands? 

Thanks

Re: [Qemu-devel] Savevm and loadvm not working

[Peter Xu]

On Tue, Jan 30, 2018 at 10:34:31AM +0000, sridhar kulkarni via Qemu-devel wrote:
> Hi,
> I am new bee to snapshot feature and how to use it correctly. My requirement is simple, in that I want to snapshot the running VM and save the snapshot file. Using the snapshot file, I want to boot the VM directly to snapshot state.I came across the qemu monitors "savevm" and "loadvm" commands. The following are the steps I follow to create snapshot and load the snapshot,Step 1: Launch VM using below command, 
> qemu-system-arm -M arm-machine -m 512M -name my_name -kernel main.rbx -serial pty -serial vc -serial vc -serial vc -drive if=none,format=qcow2,file=/home/sridhar/qemu_disk_image/dummy.qcow2 
> 
> Step 2: Save the snapshot (issue savevm from qemu monitor)
>         --> savevm my_snapshot
> 
> Step 3 Launch the VM using snapshot image saved in step 2,
> qemu-system-arm -M arm-machine -m 512M -name my_name -kernel main.rbx -serial pty -serial vc -serial vc -serial vc -drive if=none,format=qcow2,file=/home/sridhar/qemu_disk_image/dummy.qcow2 -loadvm my_snapshot
> With this approach, I am seeing that RAM contents are not getting saved when I issue "savevm" command.

Could I ask how do you know that RAM contents are not saved?  Is there
any error happened after your loadvm operation?

> I have copied the part of the file, when the function "ram_control_save_page" returns "RAM_SAVE_CONTROL_NOT_SUPP" and hence the page is not getting saved.
> 
> size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset,
>                              ram_addr_t offset, size_t size,
>                              uint64_t *bytes_sent)
> {
>     if (f->hooks && f->hooks->save_page) {
>         int ret = f->hooks->save_page(f, f->opaque, block_offset,
>                                       offset, size, bytes_sent);
> 
>         if (ret != RAM_SAVE_CONTROL_DELAYED) {
>             if (bytes_sent && *bytes_sent > 0) {
>                 qemu_update_position(f, *bytes_sent);
>             } else if (ret < 0) {
>                 qemu_file_set_error(f, ret);
>             }
>         }
> 
>         return ret;
>     }
> 
>     return RAM_SAVE_CONTROL_NOT_SUPP;

Here IMHO as long as you are not using RDMA, this function should
always return with RAM_SAVE_CONTROL_NOT_SUPP.

And I do think the name is slightly misleading.

> }
> 
> Is there anything that I am missing here in the understanding "savevm" and "loadvm" commands? 
> 
> Thanks

-- 
Peter Xu

Re: [Qemu-devel] Savevm and loadvm not working

[sridhar kulkarni]

 

    On Tuesday, January 30, 2018 5:25 PM, Peter Xu <peterx@redhat.com> wrote:
 

 On Tue, Jan 30, 2018 at 10:34:31AM +0000, sridhar kulkarni via Qemu-devel wrote:
> Hi,
> I am new bee to snapshot feature and how to use it correctly. My requirement is simple, in that I want to snapshot the running VM and save the snapshot file. Using the snapshot file, I want to boot the VM directly to snapshot state.I came across the qemu monitors "savevm" and "loadvm" commands. The following are the steps I follow to create snapshot and load the snapshot,Step 1: Launch VM using below command, 
> qemu-system-arm -M arm-machine -m 512M -name my_name -kernel main.rbx -serial pty -serial vc -serial vc -serial vc -drive if=none,format=qcow2,file=/home/sridhar/qemu_disk_image/dummy.qcow2 
> 
> Step 2: Save the snapshot (issue savevm from qemu monitor)
>         --> savevm my_snapshot
> 
> Step 3 Launch the VM using snapshot image saved in step 2,
> qemu-system-arm -M arm-machine -m 512M -name my_name -kernel main.rbx -serial pty -serial vc -serial vc -serial vc -drive if=none,format=qcow2,file=/home/sridhar/qemu_disk_image/dummy.qcow2 -loadvm my_snapshot
> With this approach, I am seeing that RAM contents are not getting saved when I issue "savevm" command.

Could I ask how do you know that RAM contents are not saved?  Is there
any error happened after your loadvm operation?
I thought that RAM contents are not saved because of return value RAM_SAVE_CONTROL_NOT_SUPP. Now that you clarified about this, I debugged this further. After doing loadvm operation, I am getting following error,qemu-system-arm: error while loading state for instance 0x0 of device 'cpu'
qemu-system-arm: Error -1 while loading VM state
I figured out that, this error is because of following function returning FALSE. "write_raw_cp_reg" writes a value and "read_raw_cp_reg" reads the value back. There is mismatch between what we write and what is being read back, and hence function returns FALSE.
Between I want to mention that, I am trying the snapshot on a VM which runs ARM 7 core. I am not sure if this snapshot feature is fully function for ARM targets.

bool write_list_to_cpustate(ARMCPU *cpu)
{
    int i;
    bool ok = true;

    for (i = 0; i < cpu->cpreg_array_len; i++) {
        uint32_t regidx = kvm_to_cpreg_id(cpu->cpreg_indexes[i]);
        uint64_t v = cpu->cpreg_values[i];
        const ARMCPRegInfo *ri;

        ri = get_arm_cp_reginfo(cpu->cp_regs, regidx);
        if (!ri) {
            ok = false;
            continue;
        }
        if (ri->type & ARM_CP_NO_RAW) {
            continue;
        }
        /* Write value and confirm it reads back as written
         * (to catch read-only registers and partially read-only
         * registers where the incoming migration value doesn't match)
         */
        write_raw_cp_reg(&cpu->env, ri, v);
        if (read_raw_cp_reg(&cpu->env, ri) != v) {
            ok = false;
        }
    }
    return ok;

> I have copied the part of the file, when the function "ram_control_save_page" returns "RAM_SAVE_CONTROL_NOT_SUPP" and hence the page is not getting saved.
> 
> size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset,
>                              ram_addr_t offset, size_t size,
>                              uint64_t *bytes_sent)
> {
>     if (f->hooks && f->hooks->save_page) {
>         int ret = f->hooks->save_page(f, f->opaque, block_offset,
>                                       offset, size, bytes_sent);
> 
>         if (ret != RAM_SAVE_CONTROL_DELAYED) {
>             if (bytes_sent && *bytes_sent > 0) {
>                 qemu_update_position(f, *bytes_sent);
>             } else if (ret < 0) {
>                 qemu_file_set_error(f, ret);
>             }
>         }
> 
>         return ret;
>     }
> 
>     return RAM_SAVE_CONTROL_NOT_SUPP;

Here IMHO as long as you are not using RDMA, this function should
always return with RAM_SAVE_CONTROL_NOT_SUPP.

And I do think the name is slightly misleading.

> }
> 
> Is there anything that I am missing here in the understanding "savevm" and "loadvm" commands? 
> 
> Thanks

-- 
Peter Xu


   

Re: [Qemu-devel] Savevm and loadvm not working

[Peter Maydell]

On 1 February 2018 at 12:19, sridhar kulkarni via Qemu-devel
<qemu-devel@nongnu.org> wrote:
> I thought that RAM contents are not saved because of return value RAM_SAVE_CONTROL_NOT_SUPP. Now that you clarified about this, I debugged this further. After doing loadvm operation, I am getting following error,qemu-system-arm: error while loading state for instance 0x0 of device 'cpu'
> qemu-system-arm: Error -1 while loading VM state
> I figured out that, this error is because of following function returning FALSE. "write_raw_cp_reg" writes a value and "read_raw_cp_reg" reads the value back. There is mismatch between what we write and what is being read back, and hence function returns FALSE.
> Between I want to mention that, I am trying the snapshot on a VM which runs ARM 7 core. I am not sure if this snapshot feature is fully function for ARM targets.

It is supposed to work for ARM, and it does for
the CPUs I've tested it with. Which exact CPU
model are you using? (Do you really mean "ARM 7"?
We don't support that. We have ARMv7 architecture
CPUs like the cortex-a9 -- did you mean one of those?)

Are you using an unmodified upstream QEMU? We don't
have an "arm-machine" board model which is what your
command line is using with "-M arm-machine", so if
you're using a modified QEMU it's possible those
modifications haven't been written to support VM
state saving.

The next step in debugging would be to run QEMU under
gdb or add tracing printfs so that you can identify
which emulated system register it is that is causing the
failure.

thanks
-- PMM

Re: [Qemu-devel] Savevm and loadvm not working

[sridhar kulkarni]

 

    On Thursday, February 1, 2018 5:56 PM, Peter Maydell <peter.maydell@linaro.org> wrote:
 

 On 1 February 2018 at 12:19, sridhar kulkarni via Qemu-devel
<qemu-devel@nongnu.org> wrote:
> I thought that RAM contents are not saved because of return value RAM_SAVE_CONTROL_NOT_SUPP. Now that you clarified about this, I debugged this further. After doing loadvm operation, I am getting following error,qemu-system-arm: error while loading state for instance 0x0 of device 'cpu'
> qemu-system-arm: Error -1 while loading VM state
> I figured out that, this error is because of following function returning FALSE. "write_raw_cp_reg" writes a value and "read_raw_cp_reg" reads the value back. There is mismatch between what we write and what is being read back, and hence function returns FALSE.
> Between I want to mention that, I am trying the snapshot on a VM which runs ARM 7 core. I am not sure if this snapshot feature is fully function for ARM targets.

It is supposed to work for ARM, and it does for
the CPUs I've tested it with. Which exact CPU
model are you using? (Do you really mean "ARM 7"?
We don't support that. We have ARMv7 architecture
CPUs like the cortex-a9 -- did you mean one of those?)

Are you using an unmodified upstream QEMU? We don't
have an "arm-machine" board model which is what your
command line is using with "-M arm-machine", so if
you're using a modified QEMU it's possible those
modifications haven't been written to support VM
state saving.

The next step in debugging would be to run QEMU under
gdb or add tracing printfs so that you can identify
which emulated system register it is that is causing the
failure.

Sorry I meant to say arm a7. We are using a custom board "arm-machine" which is based on Vexpress-a15 but we use a7 core. I learnt reading through web sources that A7 emulation is not supported in qemu. Is this causing this behaviour? But in general we are able to run all our code on this virtual board without any issues. While doing snapshot, this problem is seen.

thanks
-- PMM

   

Re: [Qemu-devel] Savevm and loadvm not working

[Peter Maydell]

On 1 February 2018 at 17:36, sridhar kulkarni <sridhar_kulk@yahoo.com> wrote:
> Sorry I meant to say arm a7. We are using a custom board "arm-machine" which
> is based on Vexpress-a15 but we use a7 core. I learnt reading through web
> sources that A7 emulation is not supported in qemu. Is this causing this
> behaviour? But in general we are able to run all our code on this virtual
> board without any issues. While doing snapshot, this problem is seen.

I can't really help if you're using a custom QEMU. If you
can reproduce this with upstream QEMU then we can certainly
take a look at it. Otherwise, the problem is very likely
in whatever the custom modifications are. It should not
be too hard for whoever is responsible for those changes
to track down the problem (which is probably a new coprocessor
register definition that doesn't have the right flags on it),
so I would talk to them about this issue.

thanks
-- PMM