VirtIO and BSOD On Windows Server 2003

VirtIO and BSOD On Windows Server 2003

[Aaron Clausen]

I've been merrily running a Windows Server 2003 with Exchange vm under
an older version of KVM (0.12.5) under Debian 6 (Squeeze), for over a
year.

I recently built a new kvm server with Debian Wheezy which comes with
KVM 1.1.2 and when I moved this guest over, I immediately started
getting BSODs (0x0000007). I disabled virtio block driver and then
attempted to upgrade to the latest with no luck.

Right now I have it running under IDE, and because the new server is
pretty spunky, I don't see any performance issues, but I have another
Server 2003 vm that is a file server to move over and I'm concerned
that I start getting a few IDE emulated guests and things will start
to get ugly.

Oddly enough, I have a Server 2012 (x64) running on virtio that moved
over without a hitch, so this is clearly a Server 2003 issue.

The most obvious solution at the moment is to downgrade to Debian
Squeeze, and that's the course I may take for the time being, but
that's not much of a long term solution.

I've done some research and this does seem to be an issue with Windows
XP/Server 2003  guests but clearly the issue here is not just virtio
drivers, but interaction between the drivers and a newer version of
kvm.

Does anybody have any thoughts or workarounds?

--
Aaron Clausen
mightymartianca@gmail.com

Re: VirtIO and BSOD On Windows Server 2003

[Stefan Hajnoczi]

On Mon, Jun 03, 2013 at 09:56:41AM -0700, Aaron Clausen wrote:
> I recently built a new kvm server with Debian Wheezy which comes with
> KVM 1.1.2 and when I moved this guest over, I immediately started
> getting BSODs (0x0000007). I disabled virtio block driver and then
> attempted to upgrade to the latest with no luck.

Stop code 0x7b "Inaccessible boot device"?

How did you create the guest on the new server?  Perhaps the hardware
configuration changed - I suggest trying to make it as close to the
original guest as possible (including the same PCI slots).

Stefan

Re: VirtIO and BSOD On Windows Server 2003

[Vadim Rozenfeld]

----- Original Message -----
From: "Stefan Hajnoczi" <stefanha@gmail.com>
To: "Aaron Clausen" <mightymartianca@gmail.com>
Cc: kvm@vger.kernel.org, vrozenfe@redhat.com
Sent: Tuesday, June 4, 2013 10:10:50 PM
Subject: Re: VirtIO and BSOD On Windows Server 2003

On Mon, Jun 03, 2013 at 09:56:41AM -0700, Aaron Clausen wrote:
> I recently built a new kvm server with Debian Wheezy which comes with
> KVM 1.1.2 and when I moved this guest over, I immediately started
> getting BSODs (0x0000007). I disabled virtio block driver and then
> attempted to upgrade to the latest with no luck.

Stop code 0x7b "Inaccessible boot device"?

How did you create the guest on the new server?  Perhaps the hardware
configuration changed - I suggest trying to make it as close to the
original guest as possible (including the same PCI slots).

Stefan

It usually happens when system unable to find a bootable device.
Check qemu options, you probably missed something.
Vadim.

Fwd: VirtIO and BSOD On Windows Server 2003

[Aaron Clausen]

On Tue, Jun 4, 2013 at 7:29 AM, Vadim Rozenfeld <vrozenfe@redhat.com> wrote:
>
> If IDE works fine, try adding another disk as virtio and see it the secondary
> disk works smoothly as well.

That's what I did, and the instant the virtio block driver initialized
in BSODed on me with 0x0000007f (reason code 0x805000f). According to
Microsoft "this problem occurs because the NTFS driver incorrectly
locks the resource when the NTFS driver tries to access the resource."

I've attempted to start the guest with the virtio driver cache
disabled, but with no success.

--
Aaron Clausen
mightymartianca@gmail.com


--
Aaron Clausen
mightymartianca@gmail.com

Fwd: VirtIO and BSOD On Windows Server 2003

[Aaron Clausen]

On Tue, Jun 4, 2013 at 7:55 AM, Aaron Clausen <mightymartianca@gmail.com> wrote:
> On Tue, Jun 4, 2013 at 7:29 AM, Vadim Rozenfeld <vrozenfe@redhat.com> wrote:
>>
>> If IDE works fine, try adding another disk as virtio and see it the secondary
>> disk works smoothly as well.
>
> That's what I did, and the instant the virtio block driver initialized
> in BSODed on me with 0x0000007f (reason code 0x805000f). According to
> Microsoft "this problem occurs because the NTFS driver incorrectly
> locks the resource when the NTFS driver tries to access the resource."
>
> I've attempted to start the guest with the virtio driver cache
> disabled, but with no success.

As a further bit of disclosure, I'm using raw images right now. I'm
going to convert one of my Server 2003 guests to qcow2 and see if that
makes a difference.


--
Aaron Clausen
mightymartianca@gmail.com

Re: VirtIO and BSOD On Windows Server 2003

[Aaron Clausen]

On Wed, Jun 5, 2013 at 3:56 PM, Vadim Rozenfeld <vrozenfe@redhat.com> wrote:
>
>
> ----- Original Message -----
> From: "Aaron Clausen" <mightymartianca@gmail.com>
> To: "Vadim Rozenfeld" <vrozenfe@redhat.com>
> Sent: Thursday, June 6, 2013 12:45:14 AM
> Subject: Re: VirtIO and BSOD On Windows Server 2003
>
> On Tue, Jun 4, 2013 at 7:11 PM, Vadim Rozenfeld <vrozenfe@redhat.com> wrote:
>>
>>
>> ----- Original Message -----
>> From: "Aaron Clausen" <mightymartianca@gmail.com>
>> To: "Vadim Rozenfeld" <vrozenfe@redhat.com>
>> Sent: Wednesday, June 5, 2013 10:05:03 AM
>> Subject: Re: VirtIO and BSOD On Windows Server 2003
>>
>> On Tue, Jun 4, 2013 at 4:37 PM, Vadim Rozenfeld <vrozenfe@redhat.com> wrote:
>>
>>> Yes, we have it WHQL certified as a SCSI adapter for all OS'es ,
>>> except for XP, and it should be available as a part of virtio-win
>>> package from fedoraproject site.
>>> --
>>
>> Okay, some further information. The stop code was 0x0000007f, with the
>> parameter 0x0000000d;  UNEXPECTED_KERNEL_MODE_TRAP
>>
>> Do you have any other VM, like Win7/W2K8, to check with?
>
> No, unfortunately. I do have a Server 2012 install (just working as a
> domain controller/authenticator at the moment), so I know the x64
> virtio drivers have no problem. I'm pretty sure this isn't a problem
> with the driver itself. Doing some googling reveals that there are
> some problems with some x86 guests, and it's probable that Debian has
> run with a kernel that doesn't possess the appropriate patches (not
> for the first time), and unless I want to go to testing (not what I
> would consider a good idea on a production machine), I'll be stuck
> with IDE.
>
>
> the difference between WS2K3 and WS2012 is that the first one is only
> working in IRQ mode, while more modern OS'es operate in MSI interrupt mode.
> I would start with checking bios version, maybe it worse to download
> the recent Seabios and build it by yourself.
>
>
>>
>> I'm building the new server and maybe I'll throw CentOS or Fedora on
>> it. I'm more a Debian fan myself, but while these guests are running,
>> ide emulation is slow.
>>
>> Yes, according to our performance team the recent viostor driver was
>> 2..6 times faster than IDE, depending on load scenario.
>> Vadim.
>
> That didn't make me feel better :)
>
> I'm going to try out Fedora today. If the Windows guests work in it,
> then I'll move to the platform. I like Debian, but running under IDE,
> particularly for the MS-Exchange server, is just not pleasant.

Okay. I set up a CentOS 6.4 x64. Fired up a Windows Server 2003 x64
guest and the second I installed the virtio drivers, I got a BSOD with
0x00000007e (a different error, yes, but won't boot until I yank the
viostor.sys file).

This is on completely different hardware and it's a qcow2 rather than
a raw image, so at this point I have to believe that there is
something very broken in the virtio drivers. I'm thinking at this time
because I need these guests running in a production environment that
I'm going to go back to an earlier Debian release. I' m pretty
disappointed here. KVM has been rock solid for me for two years, and
this is the first trouble I've had.

--
Aaron Clausen
mightymartianca@gmail.com

Re: VirtIO and BSOD On Windows Server 2003

[Vadim Rozenfeld]

Well, that's really something new for me.
Our most recent official build was build-49,
all viostor drivers have passed WHQL, functional, 
autotest and performance tests on rhel6.4. 
More recent builds are targeted at rhel6.5
and some of them can be broken. I don't know how
virtio-win package has been structures on 
fedoraproject site, but our recommendation was
to keep at least two releases available - one
for the latest stable drivers (the same binaries 
we ship to our customers, but not MS signed), and
one is the latest build - mostly for preview and/or
quick fix engineering. Pleas try downgrading to build 49.
I don't know whether we trace CentOS virtio-win bugs 
bugzilla but we definitely maintain virtio-win
bugs in Fedora.
I will try to take a look into the problem during the week.

Best regards,
Vadim. 
  

----- Original Message -----
From: "Aaron Clausen" <mightymartianca@gmail.com>
To: "Vadim Rozenfeld" <vrozenfe@redhat.com>
Cc: kvm@vger.kernel.org
Sent: Monday, June 10, 2013 2:20:45 AM
Subject: Re: VirtIO and BSOD On Windows Server 2003

On Wed, Jun 5, 2013 at 3:56 PM, Vadim Rozenfeld <vrozenfe@redhat.com> wrote:
>
>
> ----- Original Message -----
> From: "Aaron Clausen" <mightymartianca@gmail.com>
> To: "Vadim Rozenfeld" <vrozenfe@redhat.com>
> Sent: Thursday, June 6, 2013 12:45:14 AM
> Subject: Re: VirtIO and BSOD On Windows Server 2003
>
> On Tue, Jun 4, 2013 at 7:11 PM, Vadim Rozenfeld <vrozenfe@redhat.com> wrote:
>>
>>
>> ----- Original Message -----
>> From: "Aaron Clausen" <mightymartianca@gmail.com>
>> To: "Vadim Rozenfeld" <vrozenfe@redhat.com>
>> Sent: Wednesday, June 5, 2013 10:05:03 AM
>> Subject: Re: VirtIO and BSOD On Windows Server 2003
>>
>> On Tue, Jun 4, 2013 at 4:37 PM, Vadim Rozenfeld <vrozenfe@redhat.com> wrote:
>>
>>> Yes, we have it WHQL certified as a SCSI adapter for all OS'es ,
>>> except for XP, and it should be available as a part of virtio-win
>>> package from fedoraproject site.
>>> --
>>
>> Okay, some further information. The stop code was 0x0000007f, with the
>> parameter 0x0000000d;  UNEXPECTED_KERNEL_MODE_TRAP
>>
>> Do you have any other VM, like Win7/W2K8, to check with?
>
> No, unfortunately. I do have a Server 2012 install (just working as a
> domain controller/authenticator at the moment), so I know the x64
> virtio drivers have no problem. I'm pretty sure this isn't a problem
> with the driver itself. Doing some googling reveals that there are
> some problems with some x86 guests, and it's probable that Debian has
> run with a kernel that doesn't possess the appropriate patches (not
> for the first time), and unless I want to go to testing (not what I
> would consider a good idea on a production machine), I'll be stuck
> with IDE.
>
>
> the difference between WS2K3 and WS2012 is that the first one is only
> working in IRQ mode, while more modern OS'es operate in MSI interrupt mode.
> I would start with checking bios version, maybe it worse to download
> the recent Seabios and build it by yourself.
>
>
>>
>> I'm building the new server and maybe I'll throw CentOS or Fedora on
>> it. I'm more a Debian fan myself, but while these guests are running,
>> ide emulation is slow.
>>
>> Yes, according to our performance team the recent viostor driver was
>> 2..6 times faster than IDE, depending on load scenario.
>> Vadim.
>
> That didn't make me feel better :)
>
> I'm going to try out Fedora today. If the Windows guests work in it,
> then I'll move to the platform. I like Debian, but running under IDE,
> particularly for the MS-Exchange server, is just not pleasant.

Okay. I set up a CentOS 6.4 x64. Fired up a Windows Server 2003 x64
guest and the second I installed the virtio drivers, I got a BSOD with
0x00000007e (a different error, yes, but won't boot until I yank the
viostor.sys file).

This is on completely different hardware and it's a qcow2 rather than
a raw image, so at this point I have to believe that there is
something very broken in the virtio drivers. I'm thinking at this time
because I need these guests running in a production environment that
I'm going to go back to an earlier Debian release. I' m pretty
disappointed here. KVM has been rock solid for me for two years, and
this is the first trouble I've had.

--
Aaron Clausen
mightymartianca@gmail.com

Re: VirtIO and BSOD On Windows Server 2003

[Vadim Rozenfeld]

----- Original Message -----
From: "Aaron Clausen" <mightymartianca@gmail.com>
To: "Vadim Rozenfeld" <vrozenfe@redhat.com>, kvm@vger.kernel.org
Sent: Monday, June 10, 2013 3:37:53 AM
Subject: Re: VirtIO and BSOD On Windows Server 2003

On Jun 9, 2013 9:20 AM, "Aaron Clausen" <mightymartianca@gmail.com> wrote:
> Okay. I set up a CentOS 6.4 x64. Fired up a Windows Server 2003 x64
> guest and the second I installed the virtio drivers, I got a BSOD with
> 0x00000007e (a different error, yes, but won't boot until I yank the
> viostor.sys file).
>
> This is on completely different hardware and it's a qcow2 rather than
> a raw image, so at this point I have to believe that there is
> something very broken in the virtio drivers. I'm thinking at this time
> because I need these guests running in a production environment that
> I'm going to go back to an earlier Debian release. I' m pretty
> disappointed here. KVM has been rock solid for me for two years, and
> this is the first trouble I've had.

I've done a bit of googling, and it sure looks like there have been some
seabios issues. Any word of that from your end?

There were so many bochs and seabios issues during last several years
that I stopped tracking them at some point. Try taking the most recent 
one from git repository and build it by yourself.