From: Keir Fraser <keir.xen@gmail.com>
To: Junjie Wei <junjie.wei@oracle.com>, xen-devel@lists.xen.org
Subject: Re: VM save/restore
Date: Sat, 18 Aug 2012 07:38:44 +0100 [thread overview]
Message-ID: <CC54F704.3C5C9%keir.xen@gmail.com> (raw)
In-Reply-To: <502EB799.9040002@oracle.com>
On 17/08/2012 22:28, "Junjie Wei" <junjie.wei@oracle.com> wrote:
> Hello,
>
> There is a problem in Xen-4.1.2 and early versions with VM save/restore.
> When a VM is configured with VCPUs > 64, it can be started or stopped,
> but cannot be saved. It happens to both PVM and HVM guests.
>
> # xm vcpu-list 3 | grep OVM_OL5U7_X86_64_PVM_10GB | wc -l
> 65
>
> # xm save 3 vm.save
> Error: /usr/lib64/xen/bin/xc_save 27 3 0 0 0 failed
>
> /var/log/xen/xend.log: INFO (XendCheckpoint:416)
> xc: error: Too many VCPUS in guest!: Internal error
>
> It was caused by a hard-coded limit in tools/libxc/xc_domain_save.c:
>
> if ( info.max_vcpu_id >= 64 )
> {
> ERROR("Too many VCPUS in guest!");
> goto out;
> }
>
> And also in tools/libxc/xc_domain_restore.c:
>
> case XC_SAVE_ID_VCPU_INFO:
> buf->new_ctxt_format = 1;
> if ( RDEXACT(fd, &buf->max_vcpu_id, sizeof(buf->max_vcpu_id)) ||
> buf->max_vcpu_id >= 64 || RDEXACT(fd, &buf->vcpumap,
> sizeof(uint64_t)) ) {
> PERROR("Error when reading max_vcpu_id");
> return -1;
> }
>
> The code above is in both xen-4.1.2 and xen-unstable.
>
> I think if a VM can be successfully started, then save/restore should
> also work. So I made a patch and did some testing.
The check for 64 VCPUs is to cover the fact we only save/restore a 64-bit
vcpumap. That would need fixing too surely, ot CPUs > 64 would be offline
after restore I would imagine.
And what is a PVM guest?
-- Keir
> The above problem is gone but there are new ones.
>
> Let me summarize the result here.
>
> With the patch, save/restore works fine as long as it can be started,
> except two cases.
>
> 1) 32-bit guests can be configured with VCPUs > 32 and started,
> but the guest can only make use of 32 of them.
>
> 2) 32-bit PVM guests can be configured with VCPUs > 64 and started,
> but `xm save' does not work.
>
> See the testing below for details.The limit of 128 VCPUs for HVM
> guests is already considered.
>
> Could you please review the patch and help with these two cases?
>
>
> Thanks,
> Junjie
>
> -= Test environment =-
>
> [root@ovs087 HVM_X86_64]# cat /etc/ovs-release
> Oracle VM server release 3.2.1
>
> [root@ovs087 HVM_X86_64]# uname -a
> Linux ovs087 2.6.39-200.30.1.el5uek #1 SMP Thu Jul 12 21:47:09 EDT 2012
> x86_64 x86_64 x86_64 GNU/Linux
>
> [root@ovs087 HVM_X86_64]# rpm -qa | grep xen
> xenpvboot-0.1-8.el5
> xen-devel-4.1.2-39
> xen-tools-4.1.2-39
> xen-4.1.2-39
>
> -= PVM x86_64, 128 VCPUs =-
>
> [root@ovs087 PVM_X86_64]# xm list
> Name ID Mem VCPUs State
> Time(s)
> Domain-0 0 511 8 r----- 6916.9
> OVM_OL5U7_X86_64_PVM_10GB 9 2048 128 r----- 48.1
>
> [root@ovs087 PVM_X86_64]# xm save 9 vm.save
>
> [root@ovs087 PVM_X86_64]# xm restore vm.save
>
> [root@ovs087 PVM_X86_64]# xm list
> Name ID Mem VCPUs State
> Time(s)
> Domain-0 0 511 8 r----- 7076.7
> OVM_OL5U7_X86_64_PVM_10GB 10 2048 128 r----- 51.6
>
> -= PVM x86_64, 256 VCPUs =-
>
> [root@ovs087 PVM_X86_64]# xm list
> Name ID Mem VCPUs State Time(s)
> Domain-0 0 511 8 r----- 10398.1
> OVM_OL5U7_X86_64_PVM_10GB 35 2048 256 r----- 30.4
>
> [root@ovs087 PVM_X86_64]# xm save 35 vm.save
>
> [root@ovs087 PVM_X86_64]# xm restore vm.save
>
> [root@ovs087 PVM_X86_64]# xm list
> Name ID Mem VCPUs State Time(s)
> Domain-0 0 511 8 r----- 10572.1
> OVM_OL5U7_X86_64_PVM_10GB 36 2048 256 r----- 1466.9
>
> -= HVM x86_64, 128 VCPUs =-
>
> [root@ovs087 HVM_X86_64]# xm list
> Name ID Mem VCPUs State
> Time(s)
> Domain-0 0 511 8 r----- 8017.4
> OVM_OL5U7_X86_64_PVHVM_10GB 19 2048 128 r----- 343.7
>
> [root@ovs087 HVM_X86_64]# xm save 19 vm.save
>
> [root@ovs087 HVM_X86_64]# xm restore vm.save
>
> [root@ovs087 HVM_X86_64]# xm list
> Name ID Mem VCPUs State
> Time(s)
> Domain-0 0 511 8 r----- 8241.1
> OVM_OL5U7_X86_64_PVHVM_10GB 20 2048 128 r----- 121.7
>
> -= PVM x86, 64 VCPUs =-
>
> [root@ovs087 PVM_X86]# xm list
> Name ID Mem VCPUs State
> Time(s)
> Domain-0 0 511 8 r----- 36798.0
> OVM_OL5U7_X86_PVM_10GB 54 2048 32 r----- 92.8
>
> [root@ovs087 PVM_X86]# xm vcpu-list 54 | grep OVM_OL5U7_X86_PVM_10GB | wc -l
> 64
>
> [root@ovs087 PVM_X86]# xm save 54 vm.save
>
> [root@ovs087 PVM_X86]# xm restore vm.save
>
> [root@ovs087 PVM_X86]# xm list
> Name ID Mem VCPUs State
> Time(s)
> Domain-0 0 511 8 r----- 36959.3
> OVM_OL5U7_X86_PVM_10GB 55 2048 32 r----- 51.0
>
> [root@ovs087 PVM_X86]# xm vcpu-list 55 | grep OVM_OL5U7_X86_PVM_10GB | wc -l
> 64
>
> 32-bit PVM, 65 VCPUs:
>
> [root@ovs087 PVM_X86]# xm list
> Name ID Mem VCPUs State
> Time(s)
> Domain-0 0 511 8 r----- 36975.9
> OVM_OL5U7_X86_PVM_10GB 56 2048 32 r----- 8.6
>
> [root@ovs087 PVM_X86]# xm vcpu-list 56 | grep OVM_OL5U7_X86_PVM_10GB | wc -l
> 65
>
> [root@ovs087 PVM_X86]# xm list
> Name ID Mem VCPUs State
> Time(s)
> Domain-0 0 511 8 r----- 36977.7
> OVM_OL5U7_X86_PVM_10GB 56 2048 32 r----- 24.8
>
> [root@ovs087 PVM_X86]# xm vcpu-list 56 | grep OVM_OL5U7_X86_PVM_10GB | wc -l
> 65
>
> [root@ovs087 PVM_X86]# xm save 56 vm.save
> Error: /usr/lib64/xen/bin/xc_save 26 56 0 0 0 failed
>
> /var/log/xen/xend.log: INFO (XendCheckpoint:416)
> xc: error: No context for VCPU64 (61 = No data available): Internal error
>
> -= HVM x86, 64 VCPUs =-
>
> [root@ovs087 HVM_X86]# xm list
> Name ID Mem VCPUs State
> Time(s)
> Domain-0 0 511 8 r----- 36506.1
> OVM_OL5U7_X86_PVHVM_10GB 52 2048 32 r----- 68.6
>
> [root@ovs087 HVM_X86]# xm vcpu-list 52 | grep OVM_OL5U7_X86_PVHVM_10GB |
> wc -l
> 64
>
> [root@ovs087 HVM_X86]# xm save 52 vm.save
>
> [root@ovs087 HVM_X86]# xm restore vm.save
>
> [root@ovs087 HVM_X86]# xm list
> Name ID Mem VCPUs State
> Time(s)
> Domain-0 0 511 8 r----- 36730.5
> OVM_OL5U7_X86_PVHVM_10GB 53 2048 32 r----- 19.8
>
> [root@ovs087 HVM_X86]# xm vcpu-list 53 | grep OVM_OL5U7_X86_PVHVM_10GB |
> wc -l
> 64
>
> -= HVM x86, 128 VCPUs =-
>
> [root@ovs087 HVM_X86]# xm list
> Name ID Mem VCPUs State
> Time(s)
> Domain-0 0 511 8 r----- 36261.1
> OVM_OL5U7_X86_PVHVM_10GB 50 2048 32 r----- 34.9
>
> [root@ovs087 HVM_X86]# xm vcpu-list 50 | grep OVM_OL5U7_X86_PVHVM_10GB |
> wc -l
> 128
>
> [root@ovs087 HVM_X86]# xm save 50 vm.save
>
> [root@ovs087 HVM_X86]# xm restore vm.save
>
> [root@ovs087 HVM_X86]# xm list
> Name ID Mem VCPUs State
> Time(s)
> Domain-0 0 511 8 r----- 36480.5
> OVM_OL5U7_X86_PVHVM_10GB 51 2048 32 r----- 20.3
>
> [root@ovs087 HVM_X86]# xm vcpu-list 51 | grep OVM_OL5U7_X86_PVHVM_10GB |
> wc -l
> 128
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
next prev parent reply other threads:[~2012-08-18 6:38 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-08-17 21:28 VM save/restore Junjie Wei
2012-08-18 6:38 ` Keir Fraser [this message]
2012-08-18 7:34 ` Keir Fraser
2012-08-20 20:54 ` Junjie Wei
2012-08-20 21:05 ` Junjie Wei
2012-08-20 20:58 ` Junjie Wei
2012-08-22 21:17 ` Keir Fraser
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CC54F704.3C5C9%keir.xen@gmail.com \
--to=keir.xen@gmail.com \
--cc=junjie.wei@oracle.com \
--cc=xen-devel@lists.xen.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.