* "X86_PV_VCPU_MSRS record truncated" during domain restore (was: Re: [qubes-users] DispVM Doesn't work) [not found] ` <345d27df-0424-c7bd-a099-d933b0a45c18@gmail.com> @ 2016-07-20 23:20 ` Marek Marczykowski-Górecki [not found] ` <20160720232009.GA24847@mail-itl> 1 sibling, 0 replies; 6+ messages in thread From: Marek Marczykowski-Górecki @ 2016-07-20 23:20 UTC (permalink / raw) To: Massimo Colombi; +Cc: xen-devel -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On Wed, Jul 20, 2016 at 02:33:20PM +0200, Massimo Colombi wrote: > I retried (it's not the first time) to regenerate new savefile, but DispVM > doesn't work. > I attach the results. For me it looks like a bug in savefile handling. Moving to xen-devel. Any idea? Background info: it's about restoring a domain through libvirt->libxl (virsh restore equivalent). Xen 4.6.1. Full error: > 2016-07-20 14:23:01 CEST xc: error: X86_PV_VCPU_MSRS record truncated: length 8, min 9: Internal error > 2016-07-20 14:23:01 CEST xc: error: Restore failed (0 = Success): Internal error > 2016-07-20 14:23:01 CEST libxl: error: libxl_stream_read.c:749:libxl__xc_domain_restore_done: restoring domain: Successo > 2016-07-20 14:23:01 CEST libxl: error: libxl_create.c:1145:domcreate_rebuild_done: cannot (re-)build domain: -3 If relevant, here is fragment of /proc/cpuinfo (just one core): processor : 0 vendor_id : AuthenticAMD cpu family : 22 model : 48 model name : AMD A8-6410 APU with AMD Radeon R5 Graphics stepping : 1 microcode : 0x7030105 cpu MHz : 1996.290 cache size : 2048 KB physical id : 0 siblings : 4 core id : 0 cpu cores : 4 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu de tsc msr pae mce cx8 apic mca cmov pat clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm constant_tsc rep_good nopl nonstop_tsc extd_apicid eagerfpu pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch perfctr_nb bpext perfctr_l2 arat cpb hw_pstate vmmcall bmi1 xsaveopt bugs : fxsave_leak bogomips : 3992.58 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: ts ttp tm 100mhzsteps hwpstate cpb [12] [13] > Best regards, > Massimo > > On 07/20/2016 01:37 PM, Marek Marczykowski-Górecki wrote: > > Try `qvm-create-default-dvm --default-template` to regenerate new savefile. > > It looks like your savefile is somehow broken. > > [user@dom0 logs]$ qvm-create-default-dvm --default-template > --> Creating volatile image: /var/lib/qubes/appvms/fedora-23-dvm/volatile.img... > --> Loading the VM (type = AppVM)... > --> Starting Qubes DB... > --> Setting Qubes DB info for the VM... > --> Updating firewall rules... > --> Starting the VM... > --> Starting Qubes GUId... > Connecting to VM's GUI agent: ..................connected > Waiting for DVM fedora-23-dvm ... > /qubes-used-mem > Disco scollegato correttamente > > DVM boot complete, memory used=303380. Saving image... > > > > Domain fedora-23-dvm saved to /var/lib/qubes/appvms/fedora-23-dvm/dvm-savefile > > DVM savefile created successfully. > [user@dom0 logs]$ echo xterm | /usr/lib/qubes/qfile-daemon-dvm qubes.VMShell dom0 DEFAULT red > time=1469017378.68, qfile-daemon-dvm init > time=1469017378.92, creating DispVM > time=1469017380.48, collection loaded > time=1469017380.49, VM created > time=1469017380.59, VM starting > time=1469017380.6, creating config file > time=1469017380.86, calling restore > Traceback (most recent call last): > File "/usr/lib/qubes/qfile-daemon-dvm", line 200, in <module> > main() > File "/usr/lib/qubes/qfile-daemon-dvm", line 188, in main > dispvm = qfile.get_dvm() > File "/usr/lib/qubes/qfile-daemon-dvm", line 150, in get_dvm > return self.do_get_dvm() > File "/usr/lib/qubes/qfile-daemon-dvm", line 103, in do_get_dvm > dispvm.start() > File "/usr/lib64/python2.7/site-packages/qubes/modules/01QubesDisposableVm.py", line 193, in start > domain_config, libvirt.VIR_DOMAIN_SAVE_PAUSED) > File "/usr/lib64/python2.7/site-packages/libvirt.py", line 4405, in restoreFlags > if ret == -1: raise libvirtError ('virDomainRestoreFlags() failed', conn=self) > libvirt.libvirtError: internal error: libxenlight failed to restore domain 'disp1' > - -- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBCAAGBQJXkActAAoJENuP0xzK19csHZ4IAJF35BwBKbKzNFo0yOcTJxng 2NqVIUmHg3hgVZuNdkoNm09L7f5yIUai+0AqsFvM8BbPmwC9C6EBcu7FICMub/lT NUbfyf4rW5YRuEVG+OB7+Zge4mz3++kb1cLqobn2vA5Z9ayCDW32Yq5yYFff9zd7 /qMtjuj35oG25fKEs1PIZbDtMkdnq2ef1rg7KDj695SpDSt0g3AadtTjnJVh7f2V v7K6aUKR6O2xcHWADWhqxLNaKEBA8cd2yHeGYmbQwD9cICqQNVFwH/jIHWsAqGun d/IKvRetZnLajFm42X7862UIgQWE1qTpqDXV6OCWT/KNh/cL4SLwn+0RW8RbfSQ= =gdks -----END PGP SIGNATURE----- _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 6+ messages in thread
[parent not found: <20160720232009.GA24847@mail-itl>]
* Re: "X86_PV_VCPU_MSRS record truncated" during domain restore [not found] ` <20160720232009.GA24847@mail-itl> @ 2016-07-21 0:10 ` Andrew Cooper 2016-07-21 8:39 ` Massimo Colombi 0 siblings, 1 reply; 6+ messages in thread From: Andrew Cooper @ 2016-07-21 0:10 UTC (permalink / raw) To: Marek Marczykowski-Górecki, Massimo Colombi; +Cc: xen-devel [-- Attachment #1.1.1: Type: text/plain, Size: 2461 bytes --] On 21/07/2016 00:20, Marek Marczykowski-Górecki wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > On Wed, Jul 20, 2016 at 02:33:20PM +0200, Massimo Colombi wrote: >> I retried (it's not the first time) to regenerate new savefile, but DispVM >> doesn't work. >> I attach the results. > For me it looks like a bug in savefile handling. Moving to xen-devel. Any > idea? > Background info: it's about restoring a domain through libvirt->libxl > (virsh restore equivalent). Xen 4.6.1. Full error: > >> 2016-07-20 14:23:01 CEST xc: error: X86_PV_VCPU_MSRS record truncated: length 8, min 9: Internal error >> 2016-07-20 14:23:01 CEST xc: error: Restore failed (0 = Success): Internal error [EDIT] While writing the explanation below, I have spotted the bug. The restore side found a record in the stream of type X86_PV_VCPU_MSRS and 8 bytes long. The X86_PV_VCPU_MSRS record is an 8 byte header, followed by 1 or more bytes which are treated as an opaque blob, and handed back to Xen. As such, 8 bytes is insufficient to contain a nonzero sized payload. Is it possible to do an `xl save` equivalent on the domain, and run tools/python/scripts/verify-stream-v2 against the resulting file? That should identify whether it is a malformed X86_PV_VCPU_MSRS record but otherwise intact stream, or whether the stream becomes corrupted elsewhere? If not, another line if inquiry would be to instrument tools/libxc/xc_sr_save_x86_pv.c write_one_vcpu_msrs() to identify what XEN_DOMCTL_get_vcpu_msrs is returning on the source side (although this function does deliberately check for a zero-sized payload and omit the record). The hypervisor side implementation is in xen/arch/x86/domctl.c at the XEN_DOMCTL_get_vcpu_msrs case label. Re-reading the write_one_vcpu_msrs(), there is an error. We first query Xen for the maximum number of MSRs it might potentially give to us, which will return 4 on this hardware. We then actually ask for the MSR content, but Xen only writes non-zero MSRs into payload. A domain which isn't using Debug Extensions will return 4 for the first query (maximum number of MSRs), then 0 for the second query (actual MSR content), and write a X86_PV_VCPU_MSRS record with 0 payload into the stream, which the receiving side is objecting to. I will make a patch series tomorrow to address this issue. I think there are similar issues for other records as well. ~Andrew [-- Attachment #1.2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 834 bytes --] [-- Attachment #2: Type: text/plain, Size: 127 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: "X86_PV_VCPU_MSRS record truncated" during domain restore 2016-07-21 0:10 ` "X86_PV_VCPU_MSRS record truncated" during domain restore Andrew Cooper @ 2016-07-21 8:39 ` Massimo Colombi 2016-07-21 8:53 ` Andrew Cooper 0 siblings, 1 reply; 6+ messages in thread From: Massimo Colombi @ 2016-07-21 8:39 UTC (permalink / raw) To: Andrew Cooper, Marek Marczykowski-Górecki; +Cc: xen-devel This is the output of verify-stream-v2: [user@dom0 scripts]$ sudo xl save fedora-23-dvm /fedora-23-dvm-savefile Saving to /fedora-23-dvm-savefile new xl format (info 0x3/0x0/1991) xc: info: Saving domain 4, type x86 PV xc: Frames: 912384/912384 100% xc: End of stream: 0/0 0% [user@dom0 scripts]$ ./verify-stream-v2 -f xl -i /fedora-23-dvm-savefile Stream Error: Traceback (most recent call last): File "./verify-stream-v2", line 82, in read_stream VerifyLibxl(info, stream_read).verify() File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py", line 82, in verify while self.verify_record() != REC_TYPE_end: File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py", line 136, in verify_record record_verifiers[rtype](self, content[:length]) File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py", line 155, in verify_record_libxc_context VerifyLibxc(self.info, self.read).verify() File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py", line 132, in verify while self.verify_record() != REC_TYPE_end: File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py", line 227, in verify_record record_verifiers[rtype](self, content[:length]) File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py", line 429, in <lambda> VerifyLibxc.verify_record_x86_pv_vcpu_generic(s, x, "msrs"), File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py", line 323, in verify_record_x86_pv_vcpu_generic " bytes long" % (name, minsz)) RecordError: X86_PV_VCPU_msrs record length must be at least 8 bytes long On 07/21/2016 02:10 AM, Andrew Cooper wrote: > Is it possible to do an `xl save` equivalent on the domain, and run > tools/python/scripts/verify-stream-v2 against the resulting file? That > should identify whether it is a malformed X86_PV_VCPU_MSRS record but > otherwise intact stream, or whether the stream becomes corrupted elsewhere? Thanks for your explanation of the bug. Best regards, Massimo _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: "X86_PV_VCPU_MSRS record truncated" during domain restore 2016-07-21 8:39 ` Massimo Colombi @ 2016-07-21 8:53 ` Andrew Cooper 2016-07-21 9:21 ` Massimo Colombi 2016-07-22 15:07 ` Massimo Colombi 0 siblings, 2 replies; 6+ messages in thread From: Andrew Cooper @ 2016-07-21 8:53 UTC (permalink / raw) To: Massimo Colombi, Marek Marczykowski-Górecki; +Cc: xen-devel On 21/07/2016 09:39, Massimo Colombi wrote: > This is the output of verify-stream-v2: > > [user@dom0 scripts]$ sudo xl save fedora-23-dvm /fedora-23-dvm-savefile > Saving to /fedora-23-dvm-savefile new xl format (info 0x3/0x0/1991) > xc: info: Saving domain 4, type x86 PV > xc: Frames: 912384/912384 100% > xc: End of stream: 0/0 0% > > [user@dom0 scripts]$ ./verify-stream-v2 -f xl -i /fedora-23-dvm-savefile > Stream Error: > Traceback (most recent call last): > File "./verify-stream-v2", line 82, in read_stream > VerifyLibxl(info, stream_read).verify() > File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py", > line 82, in verify > while self.verify_record() != REC_TYPE_end: > File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py", > line 136, in verify_record > record_verifiers[rtype](self, content[:length]) > File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py", > line 155, in verify_record_libxc_context > VerifyLibxc(self.info, self.read).verify() > File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py", > line 132, in verify > while self.verify_record() != REC_TYPE_end: > File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py", > line 227, in verify_record > record_verifiers[rtype](self, content[:length]) > File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py", > line 429, in <lambda> > VerifyLibxc.verify_record_x86_pv_vcpu_generic(s, x, "msrs"), > File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py", > line 323, in verify_record_x86_pv_vcpu_generic > " bytes long" % (name, minsz)) > RecordError: X86_PV_VCPU_msrs record length must be at least 8 bytes long > > > On 07/21/2016 02:10 AM, Andrew Cooper wrote: >> Is it possible to do an `xl save` equivalent on the domain, and run >> tools/python/scripts/verify-stream-v2 against the resulting file? That >> should identify whether it is a malformed X86_PV_VCPU_MSRS record but >> otherwise intact stream, or whether the stream becomes corrupted >> elsewhere? > > Thanks for your explanation of the bug. I should also improve the reported error message. Do you mind rerunning with an extra -v to dump a list of the records found in the stream? ~Andrew _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: "X86_PV_VCPU_MSRS record truncated" during domain restore 2016-07-21 8:53 ` Andrew Cooper @ 2016-07-21 9:21 ` Massimo Colombi 2016-07-22 15:07 ` Massimo Colombi 1 sibling, 0 replies; 6+ messages in thread From: Massimo Colombi @ 2016-07-21 9:21 UTC (permalink / raw) To: Andrew Cooper, Marek Marczykowski-Górecki; +Cc: xen-devel [-- Attachment #1: Type: text/plain, Size: 171 bytes --] OK. I attach verbose output. On 07/21/2016 10:53 AM, Andrew Cooper wrote: > Do you mind rerunning with an extra -v to dump a list of the records > found in the stream? [-- Attachment #2: stream.txt --] [-- Type: text/plain, Size: 2047 bytes --] [user@dom0 scripts]$ ./verify-stream-v2 -f xl -v -i /tmp-savefile Processed xl header Libxl Header: little endian Libxl Record: Libxc context, length 0 Libxc Image Header: little endian Domain Header: x86 PV from Xen 4.6 Libxc Record: x86 PV info, length 8 64bit guest, 4 levels of pagetables Libxc Record: x86 PV P2M frames, length 14264 Start pfn 0x0, End 0xdebff Squashed 891 Page Data records together Libxc Record: TSC info, length 24 Mode 2, 0 kHz, 0 ns, incarnation 1 Libxc Record: Shared info, length 4096 Libxc Record: x86 PV vcpu basic, length 5176 vcpu0 basic context, 5168 bytes Libxc Record: x86 PV vcpu extended, length 64 vcpu0 extended context, 56 bytes Libxc Record: x86 PV vcpu xsave, length 856 vcpu0 xsave context, 848 bytes Libxc Record: x86 PV vcpu msrs, length 8 Stream Error: Traceback (most recent call last): File "./verify-stream-v2", line 82, in read_stream VerifyLibxl(info, stream_read).verify() File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py", line 82, in verify while self.verify_record() != REC_TYPE_end: File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py", line 136, in verify_record record_verifiers[rtype](self, content[:length]) File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py", line 155, in verify_record_libxc_context VerifyLibxc(self.info, self.read).verify() File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py", line 132, in verify while self.verify_record() != REC_TYPE_end: File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py", line 227, in verify_record record_verifiers[rtype](self, content[:length]) File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py", line 429, in <lambda> VerifyLibxc.verify_record_x86_pv_vcpu_generic(s, x, "msrs"), File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py", line 323, in verify_record_x86_pv_vcpu_generic " bytes long" % (name, minsz)) RecordError: X86_PV_VCPU_msrs record length must be at least 8 bytes long [-- Attachment #3: Type: text/plain, Size: 127 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: "X86_PV_VCPU_MSRS record truncated" during domain restore 2016-07-21 8:53 ` Andrew Cooper 2016-07-21 9:21 ` Massimo Colombi @ 2016-07-22 15:07 ` Massimo Colombi 1 sibling, 0 replies; 6+ messages in thread From: Massimo Colombi @ 2016-07-22 15:07 UTC (permalink / raw) To: Andrew Cooper, Marek Marczykowski-Górecki; +Cc: xen-devel The patches (that are attached to "[PATCH RFC 0/4] Fix issues with zero-length records in migration v2") work! I patched locally qubes-builder to import your patches and recreate rpm files. These patches also work on Xen 4.6.1. Best regards, Massimo On 07/21/2016 10:53 AM, Andrew Cooper wrote: > On 21/07/2016 09:39, Massimo Colombi wrote: >> This is the output of verify-stream-v2: >> >> [user@dom0 scripts]$ sudo xl save fedora-23-dvm /fedora-23-dvm-savefile >> Saving to /fedora-23-dvm-savefile new xl format (info 0x3/0x0/1991) >> xc: info: Saving domain 4, type x86 PV >> xc: Frames: 912384/912384 100% >> xc: End of stream: 0/0 0% >> >> [user@dom0 scripts]$ ./verify-stream-v2 -f xl -i /fedora-23-dvm-savefile >> Stream Error: >> Traceback (most recent call last): >> File "./verify-stream-v2", line 82, in read_stream >> VerifyLibxl(info, stream_read).verify() >> File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py", >> line 82, in verify >> while self.verify_record() != REC_TYPE_end: >> File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py", >> line 136, in verify_record >> record_verifiers[rtype](self, content[:length]) >> File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py", >> line 155, in verify_record_libxc_context >> VerifyLibxc(self.info, self.read).verify() >> File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py", >> line 132, in verify >> while self.verify_record() != REC_TYPE_end: >> File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py", >> line 227, in verify_record >> record_verifiers[rtype](self, content[:length]) >> File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py", >> line 429, in <lambda> >> VerifyLibxc.verify_record_x86_pv_vcpu_generic(s, x, "msrs"), >> File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py", >> line 323, in verify_record_x86_pv_vcpu_generic >> " bytes long" % (name, minsz)) >> RecordError: X86_PV_VCPU_msrs record length must be at least 8 bytes long >> >> >> On 07/21/2016 02:10 AM, Andrew Cooper wrote: >>> Is it possible to do an `xl save` equivalent on the domain, and run >>> tools/python/scripts/verify-stream-v2 against the resulting file? That >>> should identify whether it is a malformed X86_PV_VCPU_MSRS record but >>> otherwise intact stream, or whether the stream becomes corrupted >>> elsewhere? >> Thanks for your explanation of the bug. > I should also improve the reported error message. > > Do you mind rerunning with an extra -v to dump a list of the records > found in the stream? > > ~Andrew _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2016-07-22 15:07 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <nmncv2$m3$1@ger.gmane.org>
[not found] ` <20160720113752.GT1161@mail-itl>
[not found] ` <345d27df-0424-c7bd-a099-d933b0a45c18@gmail.com>
2016-07-20 23:20 ` "X86_PV_VCPU_MSRS record truncated" during domain restore (was: Re: [qubes-users] DispVM Doesn't work) Marek Marczykowski-Górecki
[not found] ` <20160720232009.GA24847@mail-itl>
2016-07-21 0:10 ` "X86_PV_VCPU_MSRS record truncated" during domain restore Andrew Cooper
2016-07-21 8:39 ` Massimo Colombi
2016-07-21 8:53 ` Andrew Cooper
2016-07-21 9:21 ` Massimo Colombi
2016-07-22 15:07 ` Massimo Colombi
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).