From: Junjie Wei <junjie.wei@oracle.com>
To: xen-devel@lists.xen.org
Subject: VM save/restore
Date: Fri, 17 Aug 2012 17:28:57 -0400 [thread overview]
Message-ID: <502EB799.9040002@oracle.com> (raw)
[-- Attachment #1: Type: text/plain, Size: 7412 bytes --]
Hello,
There is a problem in Xen-4.1.2 and early versions with VM save/restore.
When a VM is configured with VCPUs > 64, it can be started or stopped,
but cannot be saved. It happens to both PVM and HVM guests.
# xm vcpu-list 3 | grep OVM_OL5U7_X86_64_PVM_10GB | wc -l
65
# xm save 3 vm.save
Error: /usr/lib64/xen/bin/xc_save 27 3 0 0 0 failed
/var/log/xen/xend.log: INFO (XendCheckpoint:416)
xc: error: Too many VCPUS in guest!: Internal error
It was caused by a hard-coded limit in tools/libxc/xc_domain_save.c:
if ( info.max_vcpu_id >= 64 )
{
ERROR("Too many VCPUS in guest!");
goto out;
}
And also in tools/libxc/xc_domain_restore.c:
case XC_SAVE_ID_VCPU_INFO:
buf->new_ctxt_format = 1;
if ( RDEXACT(fd, &buf->max_vcpu_id, sizeof(buf->max_vcpu_id)) ||
buf->max_vcpu_id >= 64 || RDEXACT(fd, &buf->vcpumap,
sizeof(uint64_t)) ) {
PERROR("Error when reading max_vcpu_id");
return -1;
}
The code above is in both xen-4.1.2 and xen-unstable.
I think if a VM can be successfully started, then save/restore should
also work. So I made a patch and did some testing.
The above problem is gone but there are new ones.
Let me summarize the result here.
With the patch, save/restore works fine as long as it can be started,
except two cases.
1) 32-bit guests can be configured with VCPUs > 32 and started,
but the guest can only make use of 32 of them.
2) 32-bit PVM guests can be configured with VCPUs > 64 and started,
but `xm save' does not work.
See the testing below for details.The limit of 128 VCPUs for HVM
guests is already considered.
Could you please review the patch and help with these two cases?
Thanks,
Junjie
-= Test environment =-
[root@ovs087 HVM_X86_64]# cat /etc/ovs-release
Oracle VM server release 3.2.1
[root@ovs087 HVM_X86_64]# uname -a
Linux ovs087 2.6.39-200.30.1.el5uek #1 SMP Thu Jul 12 21:47:09 EDT 2012
x86_64 x86_64 x86_64 GNU/Linux
[root@ovs087 HVM_X86_64]# rpm -qa | grep xen
xenpvboot-0.1-8.el5
xen-devel-4.1.2-39
xen-tools-4.1.2-39
xen-4.1.2-39
-= PVM x86_64, 128 VCPUs =-
[root@ovs087 PVM_X86_64]# xm list
Name ID Mem VCPUs State
Time(s)
Domain-0 0 511 8 r----- 6916.9
OVM_OL5U7_X86_64_PVM_10GB 9 2048 128 r----- 48.1
[root@ovs087 PVM_X86_64]# xm save 9 vm.save
[root@ovs087 PVM_X86_64]# xm restore vm.save
[root@ovs087 PVM_X86_64]# xm list
Name ID Mem VCPUs State
Time(s)
Domain-0 0 511 8 r----- 7076.7
OVM_OL5U7_X86_64_PVM_10GB 10 2048 128 r----- 51.6
-= PVM x86_64, 256 VCPUs =-
[root@ovs087 PVM_X86_64]# xm list
Name ID Mem VCPUs State Time(s)
Domain-0 0 511 8 r----- 10398.1
OVM_OL5U7_X86_64_PVM_10GB 35 2048 256 r----- 30.4
[root@ovs087 PVM_X86_64]# xm save 35 vm.save
[root@ovs087 PVM_X86_64]# xm restore vm.save
[root@ovs087 PVM_X86_64]# xm list
Name ID Mem VCPUs State Time(s)
Domain-0 0 511 8 r----- 10572.1
OVM_OL5U7_X86_64_PVM_10GB 36 2048 256 r----- 1466.9
-= HVM x86_64, 128 VCPUs =-
[root@ovs087 HVM_X86_64]# xm list
Name ID Mem VCPUs State
Time(s)
Domain-0 0 511 8 r----- 8017.4
OVM_OL5U7_X86_64_PVHVM_10GB 19 2048 128 r----- 343.7
[root@ovs087 HVM_X86_64]# xm save 19 vm.save
[root@ovs087 HVM_X86_64]# xm restore vm.save
[root@ovs087 HVM_X86_64]# xm list
Name ID Mem VCPUs State
Time(s)
Domain-0 0 511 8 r----- 8241.1
OVM_OL5U7_X86_64_PVHVM_10GB 20 2048 128 r----- 121.7
-= PVM x86, 64 VCPUs =-
[root@ovs087 PVM_X86]# xm list
Name ID Mem VCPUs State
Time(s)
Domain-0 0 511 8 r----- 36798.0
OVM_OL5U7_X86_PVM_10GB 54 2048 32 r----- 92.8
[root@ovs087 PVM_X86]# xm vcpu-list 54 | grep OVM_OL5U7_X86_PVM_10GB | wc -l
64
[root@ovs087 PVM_X86]# xm save 54 vm.save
[root@ovs087 PVM_X86]# xm restore vm.save
[root@ovs087 PVM_X86]# xm list
Name ID Mem VCPUs State
Time(s)
Domain-0 0 511 8 r----- 36959.3
OVM_OL5U7_X86_PVM_10GB 55 2048 32 r----- 51.0
[root@ovs087 PVM_X86]# xm vcpu-list 55 | grep OVM_OL5U7_X86_PVM_10GB | wc -l
64
32-bit PVM, 65 VCPUs:
[root@ovs087 PVM_X86]# xm list
Name ID Mem VCPUs State
Time(s)
Domain-0 0 511 8 r----- 36975.9
OVM_OL5U7_X86_PVM_10GB 56 2048 32 r----- 8.6
[root@ovs087 PVM_X86]# xm vcpu-list 56 | grep OVM_OL5U7_X86_PVM_10GB | wc -l
65
[root@ovs087 PVM_X86]# xm list
Name ID Mem VCPUs State
Time(s)
Domain-0 0 511 8 r----- 36977.7
OVM_OL5U7_X86_PVM_10GB 56 2048 32 r----- 24.8
[root@ovs087 PVM_X86]# xm vcpu-list 56 | grep OVM_OL5U7_X86_PVM_10GB | wc -l
65
[root@ovs087 PVM_X86]# xm save 56 vm.save
Error: /usr/lib64/xen/bin/xc_save 26 56 0 0 0 failed
/var/log/xen/xend.log: INFO (XendCheckpoint:416)
xc: error: No context for VCPU64 (61 = No data available): Internal error
-= HVM x86, 64 VCPUs =-
[root@ovs087 HVM_X86]# xm list
Name ID Mem VCPUs State
Time(s)
Domain-0 0 511 8 r----- 36506.1
OVM_OL5U7_X86_PVHVM_10GB 52 2048 32 r----- 68.6
[root@ovs087 HVM_X86]# xm vcpu-list 52 | grep OVM_OL5U7_X86_PVHVM_10GB |
wc -l
64
[root@ovs087 HVM_X86]# xm save 52 vm.save
[root@ovs087 HVM_X86]# xm restore vm.save
[root@ovs087 HVM_X86]# xm list
Name ID Mem VCPUs State
Time(s)
Domain-0 0 511 8 r----- 36730.5
OVM_OL5U7_X86_PVHVM_10GB 53 2048 32 r----- 19.8
[root@ovs087 HVM_X86]# xm vcpu-list 53 | grep OVM_OL5U7_X86_PVHVM_10GB |
wc -l
64
-= HVM x86, 128 VCPUs =-
[root@ovs087 HVM_X86]# xm list
Name ID Mem VCPUs State
Time(s)
Domain-0 0 511 8 r----- 36261.1
OVM_OL5U7_X86_PVHVM_10GB 50 2048 32 r----- 34.9
[root@ovs087 HVM_X86]# xm vcpu-list 50 | grep OVM_OL5U7_X86_PVHVM_10GB |
wc -l
128
[root@ovs087 HVM_X86]# xm save 50 vm.save
[root@ovs087 HVM_X86]# xm restore vm.save
[root@ovs087 HVM_X86]# xm list
Name ID Mem VCPUs State
Time(s)
Domain-0 0 511 8 r----- 36480.5
OVM_OL5U7_X86_PVHVM_10GB 51 2048 32 r----- 20.3
[root@ovs087 HVM_X86]# xm vcpu-list 51 | grep OVM_OL5U7_X86_PVHVM_10GB |
wc -l
128
[-- Attachment #2: skip-max-vcpu-id-check.patch --]
[-- Type: text/x-patch, Size: 1212 bytes --]
Index: tools/libxc/xc_domain_restore.c
===================================================================
--- tools/libxc/xc_domain_restore.c (revision 3415)
+++ tools/libxc/xc_domain_restore.c (working copy)
@@ -771,8 +771,7 @@
case XC_SAVE_ID_VCPU_INFO:
buf->new_ctxt_format = 1;
if ( RDEXACT(fd, &buf->max_vcpu_id, sizeof(buf->max_vcpu_id)) ||
- buf->max_vcpu_id >= 64 || RDEXACT(fd, &buf->vcpumap,
- sizeof(uint64_t)) ) {
+ RDEXACT(fd, &buf->vcpumap, sizeof(uint64_t)) ) {
PERROR("Error when reading max_vcpu_id");
return -1;
}
Index: tools/libxc/xc_domain_save.c
===================================================================
--- tools/libxc/xc_domain_save.c (revision 3415)
+++ tools/libxc/xc_domain_save.c (working copy)
@@ -1566,12 +1566,6 @@
uint64_t vcpumap;
} chunk = { XC_SAVE_ID_VCPU_INFO, info.max_vcpu_id };
- if ( info.max_vcpu_id >= 64 )
- {
- ERROR("Too many VCPUS in guest!");
- goto out;
- }
-
for ( i = 1; i <= info.max_vcpu_id; i++ )
{
xc_vcpuinfo_t vinfo;
[-- Attachment #3: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
next reply other threads:[~2012-08-17 21:28 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-08-17 21:28 Junjie Wei [this message]
2012-08-18 6:38 ` VM save/restore Keir Fraser
2012-08-18 7:34 ` Keir Fraser
2012-08-20 20:54 ` Junjie Wei
2012-08-20 21:05 ` Junjie Wei
2012-08-20 20:58 ` Junjie Wei
2012-08-22 21:17 ` Keir Fraser
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=502EB799.9040002@oracle.com \
--to=junjie.wei@oracle.com \
--cc=xen-devel@lists.xen.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).