From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:57267) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1b2y3U-0002nh-Ua for qemu-devel@nongnu.org; Wed, 18 May 2016 05:46:18 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1b2y3R-0002rN-NZ for qemu-devel@nongnu.org; Wed, 18 May 2016 05:46:16 -0400 Received: from mx1.parallels.com ([199.115.104.18]:57445) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1b2y3R-0002lp-DI for qemu-devel@nongnu.org; Wed, 18 May 2016 05:46:13 -0400 References: From: "Denis V. Lunev" Message-ID: <573C39AB.5010008@virtuozzo.com> Date: Wed, 18 May 2016 12:45:15 +0300 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] Regression with windows 7 VMs and VGA CVE-2016-3712 fix (2.6.0 and 2.5.1.1) List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Thomas Lamprecht , qemu-devel@nongnu.org, Gerd Hoffmann On 05/15/2016 02:13 AM, Thomas Lamprecht wrote: > Hi all, > > I recently ran into Problems when trying to install some Windows VMs > this was after an update to QEMU 2.5.1.1, the VM shows Windows loading > files for the installation, then the "Starting Windows" screen appears > here it hangs and never continues. > > Changing the "-vga" option to cirrus solves this, the installation can > proceed and finish. When changing back to std (or also qxl, vmware) the > installed VM also hangs on the "Starting Windows" screen while qemu > showing a little but no excessive load. > > This phenomena appears also with QEMU 2.6.0 but not with 2.6.0-rc4, a > git bisect shows fd3c136b3e1482cd0ec7285d6bc2a3e6a62c38d7 (vga: make > sure vga register setup for vbe stays intact (CVE-2016-3712)) as the > culprit for this regression, as its a fix for a DoS its not an option to > just revert it, I guess. > The (short) bisect log is: > > git bisect start > # bad: [bfc766d38e1fae5767d43845c15c79ac8fa6d6af] Update version for v2.6.0 release > git bisect bad bfc766d38e1fae5767d43845c15c79ac8fa6d6af > # good: [975eb6a547f809608ccb08c221552f666611af25] Update version for v2.6.0-rc4 release > git bisect good 975eb6a547f809608ccb08c221552f666611af25 > # good: [2068192dcccd8a80dddfcc8df6164cf9c26e0fc4] vga: update vga register setup on vbe changes > git bisect good 2068192dcccd8a80dddfcc8df6164cf9c26e0fc4 > # bad: [53db932604dfa7bb9241d132e0173894cf54261c] Merge remote-tracking branch 'remotes/kraxel/tags/pull-vga-20160509-1' into staging > git bisect bad 53db932604dfa7bb9241d132e0173894cf54261c > > I could reproduce that with QEMU 2.5.1 and QEMU 2.6 on a Debian derivate > (Promox VE) with 4.4 Kernel and also with QEMU 2.6 on an Arch Linux > System with a 4.5 Kernel, so it should not be host distro depended. Both > machines have Intel x86_64 processors. > The problem should be reproducible with said Versions or a build from > git including the above mentioned commit (fd3c136) by starting a VM with > an Windows 7 ISO, e.g.: > > Hanging installation > ./x86_64-softmmu/qemu-system-x86_64 -boot d -cdrom win7.iso -m 1024 > > Working installation: > ./x86_64-softmmu/qemu-system-x86_64 -boot d -cdrom win7.iso -m 1024 -vga cirrus > > Noteworthy may be that Windows 10 is working, I do not had time to get > other Windows versions and test them, I'll do that as soon as possible. > Various Linux system also seems to work fine, at least I did not ran > into an issue there yet. > > I also tried testing with SeaBIOS and OVMF, as initially I had no idea > what broke, both lead to the same result - without the CVE-2016-3712 fix > they both work, with not. > Further, KVM enabled and disabled does not make any difference. > > If I can take any further step, e.g. open a bug report at another place > or help with testing I'd glad to do so. > > best regards, > Thomas > > we have (may be) similar stuff with win2k3. With these patches merged we have win2k3 crashed with the bug check below. We are using qemu-kvm-2.3.0-31.2.13 There are a bunch of VMs in this state and semi-reliable reproduction. At the first glance here this looks like return of https://bugzilla.redhat.com/show_bug.cgi?id=693993 Den 0: kd> !analyze -v ******************************************************************************* * * * Bugcheck Analysis * * * ******************************************************************************* THREAD_STUCK_IN_DEVICE_DRIVER (ea) The device driver is spinning in an infinite loop, most likely waiting for hardware to become idle. This usually indicates problem with the hardware itself or with the device driver programming the hardware incorrectly. If the kernel debugger is connected and running when watchdog detects a timeout condition then DbgBreakPoint() will be called instead of KeBugCheckEx() and detailed message including bugcheck arguments will be printed to the debugger. This way we can identify an offending thread, set breakpoints in it, and hit go to return to the spinning code to debug it further. Because KeBugCheckEx() is not called the .bugcheck directive will not return bugcheck information in this case. The arguments are already printed out to the kernel debugger. You can also retrieve them from a global variable via "dd watchdog!g_WdBugCheckData l5" (use dq on NT64). On MP machines (OS builds <= 3790) it is possible to hit a timeout when the spinning thread is interrupted by hardware interrupt and ISR or DPC routine is running at the time of the bugcheck (this is because the timeout's work item can be delivered and handled on the second CPU and the same time). If this is the case you will have to look deeper at the offending thread's stack (e.g. using dds) to determine spinning code which caused the timeout to occur. Arguments: Arg1: fffffadfe6fd3bf0, Pointer to a stuck thread object. Do .thread then kb on it to find the hung location. Arg2: fffffadfe6f45f00, Pointer to a DEFERRED_WATCHDOG object. Arg3: fffffadfe7030350, Pointer to offending driver name. Arg4: 0000000000000010, Number of times this error occurred. If a debugger is attached, this error is not always fatal -- see DESCRIPTION below. On the blue screen, this will always equal 1. Debugging Details: ------------------ ERROR - could not read driver name for bugcheck parameter 3 FAULTING_THREAD: fffffadfe6fd3bf0 FAULTING_IP: vga!bEnableHardware+89 fffff97f`ff489609 85c0 test eax,eax IMAGE_NAME: vga.dll DEBUG_FLR_IMAGE_TIMESTAMP: 42438b7b MODULE_NAME: vga FAULTING_MODULE: fffff97fff487000 vga DEFAULT_BUCKET_ID: GRAPHICS_DRIVER_FAULT BUGCHECK_STR: 0xEA PROCESS_NAME: csrss.exe CURRENT_IRQL: 1 LAST_CONTROL_TRANSFER: from fffffadfe2fe3f03 to fffff8000102e950 STACK_TEXT: fffffadf`e419c428 fffffadf`e2fe3f03 : 00000000`000000ea fffffadf`e6fd3bf0 fffffadf`e6f45f00 fffffadf`e49acae0 : nt!KeBugCheckEx fffffadf`e419c430 fffff800`01027f81 : fffffadf`e419cdd8 fffffadf`e419cdd4 fffffadf`e70e8bf0 fffffadf`e3c9a158 : VIDEOPRT!WdpKernelApc+0x303 fffffadf`e419c9d0 fffff800`01027b8d : fffff800`00828cd0 00000000`00000000 00000000`00000000 fffff800`00828cd0 : nt!KiDeliverApc+0x215 fffffadf`e419ca70 fffff800`00815c54 : 00000000`00000000 fffff800`00815d02 00000000`00000010 00000000`00000246 : nt!KiApcInterrupt+0xdd fffffadf`e419cc00 fffff800`0081b36a : 00000000`00000010 fffff800`00816627 00000000`00000010 00000000`00000282 : hal!x86BiosWriteIoSpace+0xc4 fffffadf`e419cc40 fffff800`00816683 : 00000000`001a0018 fffff800`00828cd0 00000000`00000000 fffff800`00800000 : hal!XmOutsOp+0xda fffffadf`e419cc70 fffff800`0080e189 : fffff800`00828cd0 fffff800`01028090 00000000`00000000 fffffadf`e419cce0 : hal!XmEmulateStream+0x133 fffffadf`e419ccc0 fffffadf`e2fed0ac : 00000000`00000000 fffff800`00000000 fffffadf`e419cd00 00000000`00000000 : hal!HalCallBios+0x119 fffffadf`e419cd50 fffffadf`e45706db : 00000000`00000007 fffffadf`e76bf0c0 fffffadf`e71ec5b8 fffffadf`e4575510 : VIDEOPRT!VideoPortInt10+0x9c fffffadf`e419cda0 fffffadf`e45729ea : fffffadf`e741b400 fffffadf`e76bf0c0 00000000`00000000 fffffadf`e419cf08 : vgapnp!VgaSetMode+0x13b fffffadf`e419cdf0 fffffadf`e2ff7777 : 00000000`00000000 fffffadf`00000000 fffffadf`e741b400 fffffadf`e71ec1d0 : vgapnp!VgaStartIO+0x29a fffffadf`e419ce60 fffff97f`ff08676e : fffffadf`e71ec080 fffffadf`e741b3d0 00000000`c0000002 fffffadf`e71ec080 : VIDEOPRT!pVideoPortDispatch+0xb67 fffffadf`e419cf80 fffff97f`ff086671 : 00000000`00000000 00000000`00000001 000001e0`00000280 fffff97f`ff0ab843 : win32k!GreDeviceIoControl+0xdc fffffadf`e419d040 fffff97f`ff489609 : 00000000`00000000 00000000`ffffffff 00000000`00000001 fffff97f`ff0c66f9 : win32k!EngDeviceIoControl+0x37 fffffadf`e419d090 fffff97f`ff489304 : 00000000`00000010 00000000`00000000 00000000`0b050033 fffffa80`007a5350 : vga!bEnableHardware+0x89 fffffadf`e419d160 fffff97f`ff004ec9 : 000001e0`00000280 fffffa80`00455620 00000000`00000000 fffffa80`007a5350 : vga!DrvEnableSurface+0xc4 fffffadf`e419d1a0 fffff97f`ff2d3a51 : 00000000`64776447 fffffa80`008a2010 fffffa80`0045e000 fffffa80`00955010 : win32k!WatchdogDrvEnableSurface+0x85 fffffadf`e419d1e0 fffff97f`ff0ac414 : fffffa80`0045e000 00000000`00000000 00000000`00000000 fffffa80`0045e000 : win32k!PanEnableSurface+0x61 fffffadf`e419d2a0 fffff97f`ff007011 : fffffa80`0045e000 fffffa80`00949838 fffffa80`0078d630 fffffa80`008a2010 : win32k!PDEVOBJ::bMakeSurface+0x43 fffffadf`e419d2f0 fffff97f`ff2304f1 : fffffa80`00303a30 fffffa80`00338b90 fffffa80`0078d630 00000000`00000001 : win32k!hCreateHDEV+0x5fc fffffadf`e419d3e0 fffff97f`ff00a7ac : 00000000`00000004 fffffadf`e419d490 00000000`00000000 00000000`00000000 : win32k!DrvCreateMDEV+0xb7a fffffadf`e419d740 fffff97f`ff008dcd : fffffadf`ffffffff 00000000`00000000 00000000`01bc8000 00000000`00000000 : win32k!DrvChangeDisplaySettings+0x394 fffffadf`e419d970 fffff97f`ff00bb28 : 00000000`01bc8000 00000000`00000000 00000000`00000003 00000000`00000000 : win32k!InitVideo+0x4c fffffadf`e419d9e0 fffff97f`ff00b646 : 00000000`00000060 fffffadf`e419dcf0 00000000`00000003 00000000`0000005c : win32k!UserInitialize+0x63c fffffadf`e419dc20 fffff800`0102e3fd : fffffadf`000001e0 00000000`00000001 fffffadf`e6fd3bf0 00000000`00000020 : win32k!NtUserInitialize+0x15f fffffadf`e419dc70 000007ff`7c7de86a : 000007ff`7c7deaa9 00000000`001657f0 00000000`001657f0 00000000`00000003 : nt!KiSystemServiceCopyEnd+0x3 00000000`0015fbc8 000007ff`7c7deaa9 : 00000000`001657f0 00000000`001657f0 00000000`00000003 00000000`00000000 : winsrv!NtUserInitialize+0xa 00000000`0015fbd0 000007ff`7c5b3eb8 : 00000000`00000000 00000000`00000001 00000000`001657f0 00000000`00000000 : winsrv!UserServerDllInitialization+0x229 00000000`0015fc50 000007ff`7c5b38c0 : 00000000`00163746 00000000`00000000 00000000`0016372a 00000000`00163480 : CSRSRV!CsrLoadServerDll+0x228 00000000`0015fd10 000007ff`7c5b3a10 : 00000000`00163450 00000000`00000000 00000000`00163450 00000000`00000000 : CSRSRV!CsrParseServerCommandLine+0x340 00000000`0015fee0 00000000`4a681121 : 00000000`00110000 00000000`0015ff88 00000000`00163450 00000000`0000000a : CSRSRV!CsrServerInitialization+0xa0 00000000`0015ff10 00000000`4a6815bb : 00000000`0000000b 00000000`00000000 00000000`0000010f 00000000`00163570 : csrss!main+0x41 00000000`0015ff60 00000000`00000000 : 00000000`00000001 00000000`00000000 00000000`00000004 00000000`00000005 : csrss!NtProcessStartup+0x2fb