Windows SMP

Windows SMP

[Venefax]

[-- Attachment #1.1: Type: text/plain, Size: 951 bytes --]

I think that we need to tackle the absurd loss of performance that any
Windows HVM incurs with more than one processor. I have measured it and it
is amazing. I have an application that opens 15 independent network clients,
and with one single processor and the Standard PC Hal, it takes a little
over a minute to load the 15 clients and also to register with the SIP
provider. If I leave that unchanged, I mean "ceteris paribus" , and only
change the HAL to ACPI and add 8 processors, then the same operation takes
over 10 minutes. Hello!!! I am using the GPLVL drivers from James Harper.
Right now I am about to test the MPS -Non-ACPI HAL, but the driver still
create a BSOD. Maybe the problem is only ACPI, and not MPS, who knows. In
any case, what can we do so any Windows virtual machine can achieve a
"normal" performance, I mean, a linear performance when using more
processors and not this declining curve that can kill any host.

Federico

 


[-- Attachment #1.2: Type: text/html, Size: 2679 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Windows SMP

[Randy McAnally]

[-- Attachment #1.1: Type: text/plain, Size: 1786 bytes --]

I know this is probably not what you want to hear... but for our critical Windows applications we ended up switching to Citrix XenServer with the included PV drivers.  The performance and stability is night and day -- a year later we have still had zero issues since the switch.  It is truely production ready and performs very well in SMP.

Although the open source PV drivers for XenSource have come a long way, they are still far from production-ready (even a year now since we tried them!) and should only be used in development or personal setups.

-- 
Randy 
www.FastServ.com

---------- Original Message -----------
From: "Venefax" <venefax@gmail.com> 
To: <xen-devel@lists.xensource.com> 
Sent: Sat, 27 Dec 2008 22:01:29 -0500 
Subject: [Xen-devel] Windows SMP

> I think that we need to tackle the absurd loss of performance that any Windows HVM incurs with more than one processor. I have measured it and it is amazing. I have an application that opens 15 independent network clients, and with one single processor and the Standard PC Hal, it takes a little over a minute to load the 15 clients and also to register with the SIP provider. If I leave that unchanged, I mean “ceteris paribus” , and only change the HAL to ACPI and add 8 processors, then the same operation takes over 10 minutes. Hello!!! I am using the GPLVL drivers from James Harper. Right now I am about to test the MPS –Non-ACPI HAL, but the driver still create a BSOD. Maybe the problem is only ACPI, and not MPS, who knows. In any case, what can we do so any Windows virtual machine 
 can achieve a “normal” performance, I mean, a linear performance when using more processors and not this declining curve that can kill any host.

> Federico

>

 

------- End of Original Message -------

 

[-- Attachment #1.2: Type: text/html, Size: 2265 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Windows SMP

[Dirk Utterback]

[-- Attachment #1.1: Type: text/plain, Size: 1471 bytes --]

   It seems something wrong. It is hard to achieve linear performance with
more processors, but it should be at least something "normal".  Another
choice might be halsign turbogate drivers. I used it half a year ago, and it
is stable and performance is really nice.

2008/12/27 Venefax <venefax@gmail.com>

>  I think that we need to tackle the absurd loss of performance that any
> Windows HVM incurs with more than one processor. I have measured it and it
> is amazing. I have an application that opens 15 independent network clients,
> and with one single processor and the Standard PC Hal, it takes a little
> over a minute to load the 15 clients and also to register with the SIP
> provider. If I leave that unchanged, I mean "ceteris paribus" , and only
> change the HAL to ACPI and add 8 processors, then the same operation takes
> over 10 minutes. Hello!!! I am using the GPLVL drivers from James Harper.
> Right now I am about to test the MPS –Non-ACPI HAL, but the driver still
> create a BSOD. Maybe the problem is only ACPI, and not MPS, who knows. In
> any case, what can we do so any Windows virtual machine can achieve a
> "normal" performance, I mean, a linear performance when using more
> processors and not this declining curve that can kill any host.
>
> Federico
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
>

[-- Attachment #1.2: Type: text/html, Size: 1968 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Windows SMP

[Venefax]

[-- Attachment #1.1: Type: text/plain, Size: 1785 bytes --]

The problem is not SMP, is ACPI. I installed a non-ACPI Hal, but SMP
capable, and the performance went right up with 4 virtual processors.

I hope the developers can look into this mess.

Federico

 

From: Dirk Utterback [mailto:dirk.utterback@gmail.com] 
Sent: Sunday, December 28, 2008 9:40 PM
To: Venefax
Cc: xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] Windows SMP

 

   It seems something wrong. It is hard to achieve linear performance with
more processors, but it should be at least something "normal".  Another
choice might be halsign turbogate drivers. I used it half a year ago, and it
is stable and performance is really nice.

2008/12/27 Venefax <venefax@gmail.com>

I think that we need to tackle the absurd loss of performance that any
Windows HVM incurs with more than one processor. I have measured it and it
is amazing. I have an application that opens 15 independent network clients,
and with one single processor and the Standard PC Hal, it takes a little
over a minute to load the 15 clients and also to register with the SIP
provider. If I leave that unchanged, I mean "ceteris paribus" , and only
change the HAL to ACPI and add 8 processors, then the same operation takes
over 10 minutes. Hello!!! I am using the GPLVL drivers from James Harper.
Right now I am about to test the MPS -Non-ACPI HAL, but the driver still
create a BSOD. Maybe the problem is only ACPI, and not MPS, who knows. In
any case, what can we do so any Windows virtual machine can achieve a
"normal" performance, I mean, a linear performance when using more
processors and not this declining curve that can kill any host.

Federico

 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

 


[-- Attachment #1.2: Type: text/html, Size: 5073 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Windows SMP

[James Harper]

> 
> The problem is not SMP, is ACPI. I installed a non-ACPI Hal, but SMP
> capable, and the performance went right up with 4 virtual processors.
> 
> I hope the developers can look into this mess.
> 

Can you have ACPI enabled but APIC disabled, or is that not a valid
configuration?

Or the other way around, can you have ACPI disabled but APIC enabled?

Maybe the APIC emulation is causing a performance loss?

James

RE: Windows SMP

[Venefax]

I had to disable both, and PAE. Only APIC=0 would not make any difference. I
will some further testing with Citrix Xenserver 5, using the same virtual
machine and another copy with their vmpd drivers. I bet that there is no
difference in performance. It seems to be a Xen architectural issue. Any
ideas?

-----Original Message-----
From: James Harper [mailto:james.harper@bendigoit.com.au] 
Sent: Sunday, December 28, 2008 9:53 PM
To: Venefax; Dirk Utterback
Cc: xen-devel@lists.xensource.com
Subject: RE: [Xen-devel] Windows SMP

> 
> The problem is not SMP, is ACPI. I installed a non-ACPI Hal, but SMP
> capable, and the performance went right up with 4 virtual processors.
> 
> I hope the developers can look into this mess.
> 

Can you have ACPI enabled but APIC disabled, or is that not a valid
configuration?

Or the other way around, can you have ACPI disabled but APIC enabled?

Maybe the APIC emulation is causing a performance loss?

James

Re: Windows SMP

[Dirk Utterback]

[-- Attachment #1.1: Type: text/plain, Size: 1243 bytes --]

  What is your Xen and Windows version?  I have been using CentOS5, and
Windows XP as the guest.
It works fine with 2 vcpu for Windows in my setup(apic=1, acpi=1).

Dirk

On Sun, Dec 28, 2008 at 6:59 PM, Venefax <venefax@gmail.com> wrote:

> I had to disable both, and PAE. Only APIC=0 would not make any difference.
> I
> will some further testing with Citrix Xenserver 5, using the same virtual
> machine and another copy with their vmpd drivers. I bet that there is no
> difference in performance. It seems to be a Xen architectural issue. Any
> ideas?
>
> -----Original Message-----
> From: James Harper [mailto:james.harper@bendigoit.com.au]
> Sent: Sunday, December 28, 2008 9:53 PM
> To: Venefax; Dirk Utterback
> Cc: xen-devel@lists.xensource.com
> Subject: RE: [Xen-devel] Windows SMP
>
> >
> > The problem is not SMP, is ACPI. I installed a non-ACPI Hal, but SMP
> > capable, and the performance went right up with 4 virtual processors.
> >
> > I hope the developers can look into this mess.
> >
>
> Can you have ACPI enabled but APIC disabled, or is that not a valid
> configuration?
>
> Or the other way around, can you have ACPI disabled but APIC enabled?
>
> Maybe the APIC emulation is causing a performance loss?
>
> James
>
>

[-- Attachment #1.2: Type: text/html, Size: 1772 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Windows SMP

[Keir Fraser]

On 29/12/2008 02:59, "Venefax" <venefax@gmail.com> wrote:

> I had to disable both, and PAE. Only APIC=0 would not make any difference. I
> will some further testing with Citrix Xenserver 5, using the same virtual
> machine and another copy with their vmpd drivers. I bet that there is no
> difference in performance. It seems to be a Xen architectural issue. Any
> ideas?

The problem is almost certainly APIC related. APIC=0 actually has no effect
for a multi-processor HVM guest, since APICs are architecturally absolutely
required in x86 multi-processor systems.

The problem is most likely lots of emulated APIC TPR writes slowing things
down. Possible fixes:
 1. Run a Windows guest with the 'lazy TPR' optimisation -- w2k3sp2+, w2k8,
vista. Or run 64-bit Windows which will write TPR in a different way which
most Intel/AMD CPUs can virtualise efficiently.
 2. Run a new enough Intel processor which has automatic TPR handling even
for 32-bit Windows guests.
 3. Run Citrix drivers which patch Windows to avoid TPR writes.

 -- Keir

RE: Windows SMP

[James Harper]

> > I had to disable both, and PAE. Only APIC=0 would not make any
> difference. I
> > will some further testing with Citrix Xenserver 5, using the same
> virtual
> > machine and another copy with their vmpd drivers. I bet that there
is no
> > difference in performance. It seems to be a Xen architectural issue.
Any
> > ideas?
> 
> The problem is almost certainly APIC related. APIC=0 actually has no
> effect
> for a multi-processor HVM guest, since APICs are architecturally
> absolutely
> required in x86 multi-processor systems.
> 
> The problem is most likely lots of emulated APIC TPR writes slowing
things
> down. Possible fixes:
>  1. Run a Windows guest with the 'lazy TPR' optimisation -- w2k3sp2+,
> w2k8,
> vista. Or run 64-bit Windows which will write TPR in a different way
which
> most Intel/AMD CPUs can virtualise efficiently.
>  2. Run a new enough Intel processor which has automatic TPR handling
even
> for 32-bit Windows guests.
>  3. Run Citrix drivers which patch Windows to avoid TPR writes.
> 

Can you elaborate on that last point? Does that pass WHQL?

I think that 'venefax' is running Windows 2003sp2... does that exclude
'lazy TPR' fix as an option or does it need enabling somehow?

Thanks

James

Re: Windows SMP

[Keir Fraser]

On 29/12/2008 08:20, "James Harper" <james.harper@bendigoit.com.au> wrote:

>>  3. Run Citrix drivers which patch Windows to avoid TPR writes.

> Can you elaborate on that last point? Does that pass WHQL?

The WHQL tests are oblivious to it. It's just a patching of mmio writes to
the APIC TPR register.

> I think that 'venefax' is running Windows 2003sp2... does that exclude
> 'lazy TPR' fix as an option or does it need enabling somehow?

If he's running sp2 then I think TPR problems can be discounted.

 -- Keir

RE: Windows SMP

[James Harper]

> On 29/12/2008 08:20, "James Harper" <james.harper@bendigoit.com.au>
wrote:
> 
> >>  3. Run Citrix drivers which patch Windows to avoid TPR writes.
> 
> > Can you elaborate on that last point? Does that pass WHQL?
> 
> The WHQL tests are oblivious to it. It's just a patching of mmio
writes to
> the APIC TPR register.
> 

Looking at the way KVM does it, it appears to detect writes to the TPR
register when they are trapped, and then give the DomU (or whatever KVM
calls it) the address of the instruction so that the DomU driver can
then patch it. Is that what Citrix is doing? Does the current xensource
tree have such a mechanism in it?

Thanks

James

Re: Windows SMP

[Keir Fraser]

On 29/12/2008 08:47, "James Harper" <james.harper@bendigoit.com.au> wrote:

>> The WHQL tests are oblivious to it. It's just a patching of mmio
>> writes to the APIC TPR register.
> 
> Looking at the way KVM does it, it appears to detect writes to the TPR
> register when they are trapped, and then give the DomU (or whatever KVM
> calls it) the address of the instruction so that the DomU driver can
> then patch it. Is that what Citrix is doing? Does the current xensource
> tree have such a mechanism in it?

The result is the same, but there's no hypervisor component, so none of it
is open sourced.

You could get similar results by putting static Windows-kernel-specific
fixup tables in your drivers, or I'd be open to having a KVM type of
interaction between Xen and your GPLPV drivers. Putting the payload in the
generic virtual BIOS seemed kind of gross to me.

 -- Keir

RE: Windows SMP

[James Harper]

> On 29/12/2008 08:47, "James Harper" <james.harper@bendigoit.com.au>
wrote:
> 
> >> The WHQL tests are oblivious to it. It's just a patching of mmio
> >> writes to the APIC TPR register.
> >
> > Looking at the way KVM does it, it appears to detect writes to the
TPR
> > register when they are trapped, and then give the DomU (or whatever
KVM
> > calls it) the address of the instruction so that the DomU driver can
> > then patch it. Is that what Citrix is doing? Does the current
xensource
> > tree have such a mechanism in it?
> 
> The result is the same, but there's no hypervisor component, so none
of it
> is open sourced.

:)

> You could get similar results by putting static
Windows-kernel-specific
> fixup tables in your drivers,

As in 'write bytes to offset x into kernel function y', with x depending
on the exact kernel build? Wouldn't the rootkit detectors complain about
that?

> or I'd be open to having a KVM type of
> interaction between Xen and your GPLPV drivers. Putting the payload in
the
> generic virtual BIOS seemed kind of gross to me.

Is it possible for a virtualised DomU to trap the MMIO write itself, or
can it only be trapped by the hypervisor?

Btw, is it the vmexit that is slow about these TPR writes, or is it the
writes themselves?

Thanks

James

Re: Windows SMP

[Keir Fraser]

On 29/12/2008 09:03, "James Harper" <james.harper@bendigoit.com.au> wrote:

>> You could get similar results by putting static
> Windows-kernel-specific
>> fixup tables in your drivers,
> 
> As in 'write bytes to offset x into kernel function y', with x depending
> on the exact kernel build? Wouldn't the rootkit detectors complain about
> that?

I'm not actually sure on rootkit-detector compatibility with this approach.
I know that the built-in tripwire capabilities of some 64-bit Windows
versions would be a problem, but then they tend to have lazy TPR, and also
they write to the TPR with MOV CR8, which can usually be handled efficiently
by HVM-capable CPUs anyway, so no patching is required.

Any software-based approach is going to involve patching, so I'd cross the
rootkit-detector bridge when/if we come to it. I haven't heard or read of
any complaints in this regard so far.

>> or I'd be open to having a KVM type of
>> interaction between Xen and your GPLPV drivers. Putting the payload in
> the
>> generic virtual BIOS seemed kind of gross to me.
> 
> Is it possible for a virtualised DomU to trap the MMIO write itself, or
> can it only be trapped by the hypervisor?

It could, by trapping all accesses to the APIC page (remove mapping, hook
page-fault handler). It's going to be much easier to get help from the
hypervisor!

> Btw, is it the vmexit that is slow about these TPR writes, or is it the
> writes themselves?

Both. As well as the VMEXIT you have a run through the instruction emulator
and into the apic device model. It sucks pretty bad if you do it often.

 -- Keir

RE: Windows SMP

[James Harper]

> > Is it possible for a virtualised DomU to trap the MMIO write itself,
or
> > can it only be trapped by the hypervisor?
> 
> It could, by trapping all accesses to the APIC page (remove mapping,
hook
> page-fault handler). It's going to be much easier to get help from the
> hypervisor!

Hmmm... maybe easier with the hypervisor but by trapping writes I could
do it all in my drivers.

Assuming the hypervisor route, if I understand correctly it could go
like this:
. develop a mechanism for qemu's TPR write emulation to also signal my
drivers somehow. This signalling mechanism would have to be low
overhead, but could also be pretty lazy (if these TPR writes are
happening often then it doesn't matter if we miss a few as we'll patch
them all pretty quickly)
. the PV drivers get the signal, perform some validation on the
instruction that caused the TPR write to make sure it's patchable, and
then overwrite the instruction with a call to my own routine, making
sure that this is done in an SMP safe way and taking care of caching
etc...
. My own routine does ??? (I haven't figured this out exactly, but there
is plenty of code out there that does it already)

Thinking about this some more... could it work the other way around
instead:
. My drivers publish to qemu (somehow... io space write maybe???) the
required patch for the mmio write instruction, eg a patch request might
look like:
  <01> (magic number indicating a TPR write patch)
  <05> (length)
  <01><02><03><04><05> (code to replace the TPR write instruction with)
. Qemu stores the above in a table, and every time an mmio write
instruction for the TPR register occurs which is exactly 5 bytes in
length, patch the instruction with the code given

> > Btw, is it the vmexit that is slow about these TPR writes, or is it
the
> > writes themselves?
> 
> Both. As well as the VMEXIT you have a run through the instruction
> emulator
> and into the apic device model. It sucks pretty bad if you do it
often.

I see. It's the sum of a whole load of tiny overheads that adds up...

Thanks

James

Re: Windows SMP

[Keir Fraser]

On 29/12/2008 09:40, "James Harper" <james.harper@bendigoit.com.au> wrote:

> Assuming the hypervisor route, if I understand correctly it could go
> like this:

Either of your described approaches could work. This doesn't involve qemu at
all, since APIC emulation is done within Xen. So it'll be a new hvm_op
hypercall either way. Getting Xen to do the patching isn't a bad idea. Might
also be worth looking at how KVM does it to get some further ideas.

I don't think your patched routine has to be any more complex than filtering
out TPR writes which don't change the TPR value. Again you could check KVM
for this. And of course the TPR write in your patch routine should be
skipped by the trap-and-patch mechanism. :-)

>> Both. As well as the VMEXIT you have a run through the instruction
>> emulator
>> and into the apic device model. It sucks pretty bad if you do it
> often.
> 
> I see. It's the sum of a whole load of tiny overheads that adds up...

Two fairly big overheads.

 -- Keir

RE: Windows SMP

[James Harper]

> On 29/12/2008 09:40, "James Harper" <james.harper@bendigoit.com.au>
wrote:
> 
> > Assuming the hypervisor route, if I understand correctly it could go
> > like this:
> 
> Either of your described approaches could work. This doesn't involve
qemu
> at
> all, since APIC emulation is done within Xen. So it'll be a new hvm_op
> hypercall either way. Getting Xen to do the patching isn't a bad idea.
> Might
> also be worth looking at how KVM does it to get some further ideas.
> 

Oh. I had a quick look at hw/apic.c and assumed that's where the action
was... guess I'll look in xen then!

Thanks

James

Re: Windows SMP

[Wei Huang]

James,

Travis Betak released a Windows driver which patches Windows TPR 
accesses. You can find it from http://sourceforge.net/projects/amdvopt/. 
The code would be useful as a reference. src link:

svn co https://amdvopt.svn.sourceforge.net/svnroot/amdvopt amdvopt

Thanks,

-Wei


James Harper wrote:
>> On 29/12/2008 08:20, "James Harper" <james.harper@bendigoit.com.au>
> wrote:
>>>>  3. Run Citrix drivers which patch Windows to avoid TPR writes.
>>> Can you elaborate on that last point? Does that pass WHQL?
>> The WHQL tests are oblivious to it. It's just a patching of mmio
> writes to
>> the APIC TPR register.
>>
> 
> Looking at the way KVM does it, it appears to detect writes to the TPR
> register when they are trapped, and then give the DomU (or whatever KVM
> calls it) the address of the instruction so that the DomU driver can
> then patch it. Is that what Citrix is doing? Does the current xensource
> tree have such a mechanism in it?
> 
> Thanks
> 
> James
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

Re: Windows SMP

[Keir Fraser]

That looks like it could be a nice drop-in piece of code!

 -- Keir

On 30/12/2008 16:58, "Wei Huang" <wei.huang2@amd.com> wrote:

> James,
> 
> Travis Betak released a Windows driver which patches Windows TPR
> accesses. You can find it from http://sourceforge.net/projects/amdvopt/.
> The code would be useful as a reference. src link:
> 
> svn co https://amdvopt.svn.sourceforge.net/svnroot/amdvopt amdvopt
> 
> Thanks,
> 
> -Wei
> 
> 
> James Harper wrote:
>>> On 29/12/2008 08:20, "James Harper" <james.harper@bendigoit.com.au>
>> wrote:
>>>>>  3. Run Citrix drivers which patch Windows to avoid TPR writes.
>>>> Can you elaborate on that last point? Does that pass WHQL?
>>> The WHQL tests are oblivious to it. It's just a patching of mmio
>> writes to
>>> the APIC TPR register.
>>> 
>> 
>> Looking at the way KVM does it, it appears to detect writes to the TPR
>> register when they are trapped, and then give the DomU (or whatever KVM
>> calls it) the address of the instruction so that the DomU driver can
>> then patch it. Is that what Citrix is doing? Does the current xensource
>> tree have such a mechanism in it?
>> 
>> Thanks
>> 
>> James
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>> 
> 

RE: Windows SMP

[James Harper]

> > James,
> >
> > Travis Betak released a Windows driver which patches Windows TPR
> > accesses. You can find it from
http://sourceforge.net/projects/amdvopt/.
> > The code would be useful as a reference. src link:
> >
> > svn co https://amdvopt.svn.sourceforge.net/svnroot/amdvopt amdvopt> 
>
> That looks like it could be a nice drop-in piece of code!
> 

He is taking a table-based approach, which would require ongoing
maintenance. I was hoping to develop a more dynamic version, but given
that someone else has already put the patch tables together, doing it
dynamically suddenly seems like a lot of work :)

Is there a similar approach that would work on an Intel system?

Are there any other operations like TPR write's that are done frequently
that could benefit from patching?

James

RE: Windows SMP

[Tian, Kevin]

>From: James Harper
>Sent: Wednesday, December 31, 2008 4:22 PM
>> > James,
>> >
>> > Travis Betak released a Windows driver which patches Windows TPR
>> > accesses. You can find it from
>http://sourceforge.net/projects/amdvopt/.
>> > The code would be useful as a reference. src link:
>> >
>> > svn co https://amdvopt.svn.sourceforge.net/svnroot/amdvopt 
>amdvopt> 
>>
>> That looks like it could be a nice drop-in piece of code!
>> 
>
>He is taking a table-based approach, which would require ongoing
>maintenance. I was hoping to develop a more dynamic version, but given
>that someone else has already put the patch tables together, doing it
>dynamically suddenly seems like a lot of work :)
>
>Is there a similar approach that would work on an Intel system?

On Intel CPU with FlexPriority support, you don't need patching
guest since TPR accesses would be recognized  by hardware 
for acceleration automatically.

But on CPUs without h/w acceleration support, you may expect
borrow that overall idea, but instead of patching with LOCK MOV
CR0, you would replace it with a piece of code lines to emulate
similar acceleration as what h/w is assumed to do.

Thanks,
Kevin

RE: Windows SMP

[James Harper]

> >Is there a similar approach that would work on an Intel system?
> 
> On Intel CPU with FlexPriority support, you don't need patching
> guest since TPR accesses would be recognized  by hardware
> for acceleration automatically.
> 
> But on CPUs without h/w acceleration support, you may expect
> borrow that overall idea, but instead of patching with LOCK MOV
> CR0, you would replace it with a piece of code lines to emulate
> similar acceleration as what h/w is assumed to do.
> 

Do you have an example :)

One thing Keir suggested would be to install the patch to jump to some
code which compared the value being written to the TPR register with the
value last written, and only perform the actual write if the values are
different. I can do that without too much fuss but if there is something
faster then even better.

Thanks

James

RE: Windows SMP

[Tian, Kevin]

>From: James Harper [mailto:james.harper@bendigoit.com.au] 
>Sent: Wednesday, December 31, 2008 4:35 PM
>
>> >Is there a similar approach that would work on an Intel system?
>> 
>> On Intel CPU with FlexPriority support, you don't need patching
>> guest since TPR accesses would be recognized  by hardware
>> for acceleration automatically.
>> 
>> But on CPUs without h/w acceleration support, you may expect
>> borrow that overall idea, but instead of patching with LOCK MOV
>> CR0, you would replace it with a piece of code lines to emulate
>> similar acceleration as what h/w is assumed to do.
>> 
>
>Do you have an example :)
>
>One thing Keir suggested would be to install the patch to jump to some
>code which compared the value being written to the TPR 
>register with the
>value last written, and only perform the actual write if the values are

That's basically what I meant, and also what KVM does today. 
VM-exit in such case is only proactively requested by vmcall
in inserted lines if Xen emulation logic has to be involved.

>different. I can do that without too much fuss but if there is 
>something
>faster then even better.
>

If you compare to VM-exit overhead for every TPR access, above
is already far far faster. Of course fewer memory accesses used
in inserted lines, less overhead you'll see then.:-)

Thanks,
Kevin

Re: Windows SMP

[Keir Fraser]

On 31/12/2008 08:50, "Tian, Kevin" <kevin.tian@intel.com> wrote:

>> different. I can do that without too much fuss but if there is
>> something
>> faster then even better.
>> 
> 
> If you compare to VM-exit overhead for every TPR access, above
> is already far far faster. Of course fewer memory accesses used
> in inserted lines, less overhead you'll see then.:-)

It'll be plenty good enough, as you'll be saving probably on the order of
5000 cycles by not exiting and emulating. You'll be hard pushed to be worse
than 5% of that.

 -- Keir

RE: Windows SMP

[Tian, Kevin]

>From: James Harper
>Sent: Monday, December 29, 2008 4:48 PM
>
>> On 29/12/2008 08:20, "James Harper" <james.harper@bendigoit.com.au>
>wrote:
>> 
>> >>  3. Run Citrix drivers which patch Windows to avoid TPR writes.
>> 
>> > Can you elaborate on that last point? Does that pass WHQL?
>> 
>> The WHQL tests are oblivious to it. It's just a patching of mmio
>writes to
>> the APIC TPR register.
>> 
>
>Looking at the way KVM does it, it appears to detect writes to the TPR
>register when they are trapped, and then give the DomU (or whatever KVM
>calls it) the address of the instruction so that the DomU driver can
>then patch it. Is that what Citrix is doing? Does the current xensource
>tree have such a mechanism in it?
>

IIRC, based on my understanding months ago when KVM PTR
patching was introduced, it only works for UP windows guest.
One tricky issue for dynamic instruction patching is how to encode
vcpu ID in register, since vcpu ID is the index into per-vcpu shadow
vTPR value. Use memory accesses to retrieve vcpu ID would lower
down the performance gain. KVM borrows only one bit from vTR (TI?)
as vcpu id.

Thanks,
Kevin

Re: Windows SMP

[Andrew Lyon]

On Mon, Dec 29, 2008 at 8:14 AM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:
> On 29/12/2008 02:59, "Venefax" <venefax@gmail.com> wrote:
>
>> I had to disable both, and PAE. Only APIC=0 would not make any difference. I
>> will some further testing with Citrix Xenserver 5, using the same virtual
>> machine and another copy with their vmpd drivers. I bet that there is no
>> difference in performance. It seems to be a Xen architectural issue. Any
>> ideas?
>
> The problem is almost certainly APIC related. APIC=0 actually has no effect
> for a multi-processor HVM guest, since APICs are architecturally absolutely
> required in x86 multi-processor systems.
>
> The problem is most likely lots of emulated APIC TPR writes slowing things
> down. Possible fixes:
>  1. Run a Windows guest with the 'lazy TPR' optimisation -- w2k3sp2+, w2k8,
> vista. Or run 64-bit Windows which will write TPR in a different way which
> most Intel/AMD CPUs can virtualise efficiently.
>  2. Run a new enough Intel processor which has automatic TPR handling even
> for 32-bit Windows guests.

Does "PTR handling" have a cpu feature flag I can grep for in
/proc/cpuinfo, or perhaps there is a list of cpus that support that
feature?

I run Xen on two systems, a dual Xeon 5420 and a Core 2 Quad, neither
seem to suffer from bad performance with SMP Windows hvm's but most of
the windows hvm's I run are not heavily loaded so I would like to know
if my system is likely to suffer from this problem as it would
probably become noticible once load increases.

thx
Andy

>  3. Run Citrix drivers which patch Windows to avoid TPR writes.
>
>  -- Keir
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: Windows SMP

[Keir Fraser]

On 29/12/2008 21:40, "Andrew Lyon" <andrew.lyon@gmail.com> wrote:

> Does "PTR handling" have a cpu feature flag I can grep for in
> /proc/cpuinfo, or perhaps there is a list of cpus that support that
> feature?

No it's not a CPUID feature and hence not visible via /proc/cpuinfo. Xen
could print out if it detects the feature, but it doesn't currently. :-)

 -- Keir

Re: Windows SMP

[Andrew Lyon]

On Mon, Dec 29, 2008 at 9:57 PM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:
> On 29/12/2008 21:40, "Andrew Lyon" <andrew.lyon@gmail.com> wrote:
>
>> Does "PTR handling" have a cpu feature flag I can grep for in
>> /proc/cpuinfo, or perhaps there is a list of cpus that support that
>> feature?
>
> No it's not a CPUID feature and hence not visible via /proc/cpuinfo. Xen
> could print out if it detects the feature, but it doesn't currently. :-)

Hmm, that would be very useful.

As its not a cpuid feature I guess I will  have to check the Intel
manuals for each cpu model, do you know what the feature is described
as in Intel documentation?

>
>  -- Keir
>
>
>

Re: Windows SMP

[Keir Fraser]

On 30/12/2008 10:34, "Andrew Lyon" <andrew.lyon@gmail.com> wrote:

>> No it's not a CPUID feature and hence not visible via /proc/cpuinfo. Xen
>> could print out if it detects the feature, but it doesn't currently. :-)
> 
> Hmm, that would be very useful.
> 
> As its not a cpuid feature I guess I will  have to check the Intel
> manuals for each cpu model, do you know what the feature is described
> as in Intel documentation?

In the architecture manual it is known as apic-access virtualisation. I
don't know whether it would be a headline feature in a processor's data
sheet.

 -- Keir

Re: Windows SMP

[Andrew Lyon]

On Tue, Dec 30, 2008 at 11:04 AM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:
> On 30/12/2008 10:34, "Andrew Lyon" <andrew.lyon@gmail.com> wrote:
>
>>> No it's not a CPUID feature and hence not visible via /proc/cpuinfo. Xen
>>> could print out if it detects the feature, but it doesn't currently. :-)
>>
>> Hmm, that would be very useful.
>>
>> As its not a cpuid feature I guess I will  have to check the Intel
>> manuals for each cpu model, do you know what the feature is described
>> as in Intel documentation?
>
> In the architecture manual it is known as apic-access virtualisation. I
> don't know whether it would be a headline feature in a processor's data
> sheet.

Can you point me to the code where apic-access virtualisation is used?
Displaying a message might be a nice small improvement for me to
figure out and contribute.

>
>  -- Keir
>
>
>

Re: Windows SMP

[Keir Fraser]

On 30/12/2008 12:41, "Andrew Lyon" <andrew.lyon@gmail.com> wrote:

>> In the architecture manual it is known as apic-access virtualisation. I
>> don't know whether it would be a headline feature in a processor's data
>> sheet.
> 
> Can you point me to the code where apic-access virtualisation is used?
> Displaying a message might be a nice small improvement for me to
> figure out and contribute.

You can make use of the macro cpu_has_vmx_virtualize_apic_accesses to
control a printk() at the end of
xen/arch/x86/hvm/vmx/vmcs.c:vmx_init_vmcs_config().

 -- Keir

Re: Windows SMP

[Andrew Lyon]

On Tue, Dec 30, 2008 at 2:41 PM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:
> On 30/12/2008 12:41, "Andrew Lyon" <andrew.lyon@gmail.com> wrote:
>
>>> In the architecture manual it is known as apic-access virtualisation. I
>>> don't know whether it would be a headline feature in a processor's data
>>> sheet.
>>
>> Can you point me to the code where apic-access virtualisation is used?
>> Displaying a message might be a nice small improvement for me to
>> figure out and contribute.
>
> You can make use of the macro cpu_has_vmx_virtualize_apic_accesses to
> control a printk() at the end of
> xen/arch/x86/hvm/vmx/vmcs.c:vmx_init_vmcs_config().
>

Thanks Keir, I added a couple of printk's as you suggested and I can
now see if the feature is supported:

Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz = (XEN) APIC Access virtualized
Intel(R) Core(TM)2 CPU          6600  @ 2.40GHz = (XEN) APIC Access emulated

I didnt expect my Xeon system to support it, I've read some pdf's
about Intel virtualizaton features and they seemed to suggest it was a
new feature on 7xxx Xeons.

The results fit the performance I've seen, a windows xp 32 bit hvm
with 2 cpu's runs a lot faster on the Xeon 2.5 than on the Core 2.4.

Andy


>  -- Keir
>
>
>

Re: Windows SMP

[Keir Fraser]

On 31/12/2008 10:08, "Andrew Lyon" <andrew.lyon@gmail.com> wrote:

> 
> Thanks Keir, I added a couple of printk's as you suggested and I can
> now see if the feature is supported:
> 
> Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz = (XEN) APIC Access
> virtualized
> Intel(R) Core(TM)2 CPU          6600  @ 2.40GHz = (XEN) APIC Access emulated
> 
> I didnt expect my Xeon system to support it, I've read some pdf's
> about Intel virtualizaton features and they seemed to suggest it was a
> new feature on 7xxx Xeons.
> 
> The results fit the performance I've seen, a windows xp 32 bit hvm
> with 2 cpu's runs a lot faster on the Xeon 2.5 than on the Core 2.4.

I think it is probably worth printing out. I'll add a patch to xen-unstable.

 Thanks,
 Keir

RE: Windows SMP

[Venefax]

Dear Gentlemen
I have a machine with Intel 7350, 4 quad core. I can test since I have two
virtual machines with 8 vcpus, showing a gross overhead that makes them
unsuited for business. I use SLES 10 SP2, and I don't know how to apply the
patch, but if somebody can log in and apply it, we can see the results
immediately. The issue is affecting me directly. My two VM's have a VOIP
application, very intensive in network and CPU usage.
Federico
-----Original Message-----
From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] 
Sent: Wednesday, December 31, 2008 5:10 AM
To: Andrew Lyon
Cc: Venefax; James Harper; Dirk Utterback; xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] Windows SMP

On 31/12/2008 10:08, "Andrew Lyon" <andrew.lyon@gmail.com> wrote:

> 
> Thanks Keir, I added a couple of printk's as you suggested and I can
> now see if the feature is supported:
> 
> Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz = (XEN) APIC Access
> virtualized
> Intel(R) Core(TM)2 CPU          6600  @ 2.40GHz = (XEN) APIC Access
emulated
> 
> I didnt expect my Xeon system to support it, I've read some pdf's
> about Intel virtualizaton features and they seemed to suggest it was a
> new feature on 7xxx Xeons.
> 
> The results fit the performance I've seen, a windows xp 32 bit hvm
> with 2 cpu's runs a lot faster on the Xeon 2.5 than on the Core 2.4.

I think it is probably worth printing out. I'll add a patch to xen-unstable.

 Thanks,
 Keir

Re: Windows SMP

[Steve Ofsthun]

Venefax wrote:
> Dear Gentlemen

> I have a machine with Intel 7350, 4 quad core. I can test since I have two
> virtual machines with 8 vcpus, showing a gross overhead that makes them
> unsuited for business. I use SLES 10 SP2, and I don't know how to apply the
> patch, but if somebody can log in and apply it, we can see the results
> immediately. The issue is affecting me directly. My two VM's have a VOIP
> application, very intensive in network and CPU usage.

If your problem isn't the previously discussed TPR issues ...

What Windows version are you running in your guests?

Windows 2000 SMP has serious problems due to the CPU wasting idle loop.  We avoid this with a special idler daemon running in the guest.


Are you over-committing VCPUS at all (more than one active VCPU per CPU)?

We have noticed significant performance degradation with SMP windows once you over-commit VCPUS on the host.  This seems to be due to excessive guest spinlock overhead caused by the spinning VCPUS wasting their entire quantum looping for the guest lock at the same time they are preventing the guest lock holding VCPU from running.  We have been experimenting with new CPU features for exiting on PAUSE instructions.  This can possibly be used to detect the CPU wasting spinners.


Have you tried using the Novell shim (Vista/2008 guests)?

I'm not sure the shim provides locking enhancements, but we have seen benefits for certain workloads.


Are you running PV on HVM drivers?

One last area of concern may be distributing callback interrupts to all VCPUS.  We still run Xen (SLES10 SP2 based) with a patch to force all guest callback interrupts to VCPU 0.  This has consistently improved our SMP guest I/O performance while running PV-on-HVM drivers.


Steve

> Federico
> -----Original Message-----
> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] 
> Sent: Wednesday, December 31, 2008 5:10 AM
> To: Andrew Lyon
> Cc: Venefax; James Harper; Dirk Utterback; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] Windows SMP
> 
> On 31/12/2008 10:08, "Andrew Lyon" <andrew.lyon@gmail.com> wrote:
> 
>> Thanks Keir, I added a couple of printk's as you suggested and I can
>> now see if the feature is supported:
>>
>> Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz = (XEN) APIC Access
>> virtualized
>> Intel(R) Core(TM)2 CPU          6600  @ 2.40GHz = (XEN) APIC Access
> emulated
>> I didnt expect my Xeon system to support it, I've read some pdf's
>> about Intel virtualizaton features and they seemed to suggest it was a
>> new feature on 7xxx Xeons.
>>
>> The results fit the performance I've seen, a windows xp 32 bit hvm
>> with 2 cpu's runs a lot faster on the Xeon 2.5 than on the Core 2.4.
> 
> I think it is probably worth printing out. I'll add a patch to xen-unstable.
> 
>  Thanks,
>  Keir
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: Windows SMP

[Venefax]

I use only Windows 2003 R2 SP2 fully patched. Therefore that patch would not apply since it is for Vista (Novell shim (Vista/2008 guests). But I will ask Novell.
I would love to try that second patch that would "force all guest callback interrupts to VCPU 0". My setup is to use one core for xen, only, and the rest 15 cores for virtual machines, as instructed by Novell support. I am not over-committing CPU's. Even if I run one single virtual machine with 8 CPU's, the performance is awful. It takes  10 mins to load an application that loads in 1.5-2.0 mins with one single CPU. I switched all my 16 virtual machines to Standard PC Hal and that is the only way that the application works. However, it is an administrative nightmare. Instead of managing 16 independent windows machines via VNC, I should be able to manage two, maybe one.
Can you send me information regarding both patches mentioned?
Federico


-----Original Message-----
From: Steve Ofsthun [mailto:sofsthun@virtualiron.com] 
Sent: Monday, January 05, 2009 1:59 PM
To: Venefax
Cc: 'Keir Fraser'; 'Andrew Lyon'; 'James Harper'; 'Dirk Utterback'; xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] Windows SMP

Venefax wrote:
> Dear Gentlemen

> I have a machine with Intel 7350, 4 quad core. I can test since I have two
> virtual machines with 8 vcpus, showing a gross overhead that makes them
> unsuited for business. I use SLES 10 SP2, and I don't know how to apply the
> patch, but if somebody can log in and apply it, we can see the results
> immediately. The issue is affecting me directly. My two VM's have a VOIP
> application, very intensive in network and CPU usage.

If your problem isn't the previously discussed TPR issues ...

What Windows version are you running in your guests?

Windows 2000 SMP has serious problems due to the CPU wasting idle loop.  We avoid this with a special idler daemon running in the guest.


Are you over-committing VCPUS at all (more than one active VCPU per CPU)?

We have noticed significant performance degradation with SMP windows once you over-commit VCPUS on the host.  This seems to be due to excessive guest spinlock overhead caused by the spinning VCPUS wasting their entire quantum looping for the guest lock at the same time they are preventing the guest lock holding VCPU from running.  We have been experimenting with new CPU features for exiting on PAUSE instructions.  This can possibly be used to detect the CPU wasting spinners.


Have you tried using the Novell shim (Vista/2008 guests)?

I'm not sure the shim provides locking enhancements, but we have seen benefits for certain workloads.


Are you running PV on HVM drivers?

One last area of concern may be distributing callback interrupts to all VCPUS.  We still run Xen (SLES10 SP2 based) with a patch to force all guest callback interrupts to VCPU 0.  This has consistently improved our SMP guest I/O performance while running PV-on-HVM drivers.


Steve

> Federico
> -----Original Message-----
> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] 
> Sent: Wednesday, December 31, 2008 5:10 AM
> To: Andrew Lyon
> Cc: Venefax; James Harper; Dirk Utterback; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] Windows SMP
> 
> On 31/12/2008 10:08, "Andrew Lyon" <andrew.lyon@gmail.com> wrote:
> 
>> Thanks Keir, I added a couple of printk's as you suggested and I can
>> now see if the feature is supported:
>>
>> Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz = (XEN) APIC Access
>> virtualized
>> Intel(R) Core(TM)2 CPU          6600  @ 2.40GHz = (XEN) APIC Access
> emulated
>> I didnt expect my Xeon system to support it, I've read some pdf's
>> about Intel virtualizaton features and they seemed to suggest it was a
>> new feature on 7xxx Xeons.
>>
>> The results fit the performance I've seen, a windows xp 32 bit hvm
>> with 2 cpu's runs a lot faster on the Xeon 2.5 than on the Core 2.4.
> 
> I think it is probably worth printing out. I'll add a patch to xen-unstable.
> 
>  Thanks,
>  Keir
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Windows SMP

[Steve Ofsthun]

[-- Attachment #1: Type: text/plain, Size: 4565 bytes --]

Venefax wrote:
> I use only Windows 2003 R2 SP2 fully patched. Therefore that patch would not apply since it is for Vista (Novell shim (Vista/2008 guests). But I will ask Novell.
> I would love to try that second patch that would "force all guest callback interrupts to VCPU 0". My setup is to use one core for xen, only, and the rest 15 cores for virtual machines, as instructed by Novell support. I am not over-committing CPU's. Even if I run one single virtual machine with 8 CPU's, the performance is awful. It takes  10 mins to load an application that loads in 1.5-2.0 mins with one single CPU. I switched all my 16 virtual machines to Standard PC Hal and that is the only way that the application works. However, it is an administrative nightmare. Instead of managing 16 independent windows machines via VNC, I should be able to manage two, maybe one.
> Can you send me information regarding both patches mentioned?

I'll have to check into the state of the PAUSE exit patch.  We are doing the work on an Intel Tylersburg machine.  I don't think these are generally available yet.

I have attached the referenced callback IRQ routing patch.

Whose Windows PV-on-HVM drivers are you running?

Steve

> Federico
> 
> 
> -----Original Message-----
> From: Steve Ofsthun [mailto:sofsthun@virtualiron.com] 
> Sent: Monday, January 05, 2009 1:59 PM
> To: Venefax
> Cc: 'Keir Fraser'; 'Andrew Lyon'; 'James Harper'; 'Dirk Utterback'; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] Windows SMP
> 
> Venefax wrote:
>> Dear Gentlemen
> 
>> I have a machine with Intel 7350, 4 quad core. I can test since I have two
>> virtual machines with 8 vcpus, showing a gross overhead that makes them
>> unsuited for business. I use SLES 10 SP2, and I don't know how to apply the
>> patch, but if somebody can log in and apply it, we can see the results
>> immediately. The issue is affecting me directly. My two VM's have a VOIP
>> application, very intensive in network and CPU usage.
> 
> If your problem isn't the previously discussed TPR issues ...
> 
> What Windows version are you running in your guests?
> 
> Windows 2000 SMP has serious problems due to the CPU wasting idle loop.  We avoid this with a special idler daemon running in the guest.
> 
> 
> Are you over-committing VCPUS at all (more than one active VCPU per CPU)?
> 
> We have noticed significant performance degradation with SMP windows once you over-commit VCPUS on the host.  This seems to be due to excessive guest spinlock overhead caused by the spinning VCPUS wasting their entire quantum looping for the guest lock at the same time they are preventing the guest lock holding VCPU from running.  We have been experimenting with new CPU features for exiting on PAUSE instructions.  This can possibly be used to detect the CPU wasting spinners.
> 
> 
> Have you tried using the Novell shim (Vista/2008 guests)?
> 
> I'm not sure the shim provides locking enhancements, but we have seen benefits for certain workloads.
> 
> 
> Are you running PV on HVM drivers?
> 
> One last area of concern may be distributing callback interrupts to all VCPUS.  We still run Xen (SLES10 SP2 based) with a patch to force all guest callback interrupts to VCPU 0.  This has consistently improved our SMP guest I/O performance while running PV-on-HVM drivers.
> 
> 
> Steve
> 
>> Federico
>> -----Original Message-----
>> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] 
>> Sent: Wednesday, December 31, 2008 5:10 AM
>> To: Andrew Lyon
>> Cc: Venefax; James Harper; Dirk Utterback; xen-devel@lists.xensource.com
>> Subject: Re: [Xen-devel] Windows SMP
>>
>> On 31/12/2008 10:08, "Andrew Lyon" <andrew.lyon@gmail.com> wrote:
>>
>>> Thanks Keir, I added a couple of printk's as you suggested and I can
>>> now see if the feature is supported:
>>>
>>> Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz = (XEN) APIC Access
>>> virtualized
>>> Intel(R) Core(TM)2 CPU          6600  @ 2.40GHz = (XEN) APIC Access
>> emulated
>>> I didnt expect my Xeon system to support it, I've read some pdf's
>>> about Intel virtualizaton features and they seemed to suggest it was a
>>> new feature on 7xxx Xeons.
>>>
>>> The results fit the performance I've seen, a windows xp 32 bit hvm
>>> with 2 cpu's runs a lot faster on the Xeon 2.5 than on the Core 2.4.
>> I think it is probably worth printing out. I'll add a patch to xen-unstable.
>>
>>  Thanks,
>>  Keir
>>
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
> 


[-- Attachment #2: xen-vioapic-deliver-single.patch --]
[-- Type: text/x-patch, Size: 1079 bytes --]

diff -r 60570c3d7ea7 xen/arch/x86/hvm/vioapic.c
--- a/xen/arch/x86/hvm/vioapic.c
+++ b/xen/arch/x86/hvm/vioapic.c
@@ -341,21 +341,18 @@ void vioapic_deliver(struct hvm_hw_vioap
         return;
     }
 
+#ifdef IRQ0_SPECIAL_ROUTING
+    /* Force round-robin to pick VCPU 0 */
+    if ( ((irq == hvm_isa_irq_to_gsi(0)) && pit_channel0_enabled()) ||
+	 is_hvm_callback_irq(vioapic, irq) )
+	deliver_bitmask = (uint32_t)1;
+#endif
+
     switch ( delivery_mode )
     {
     case dest_LowestPrio:
     {
-#ifdef IRQ0_SPECIAL_ROUTING
-        /* Force round-robin to pick VCPU 0 */
-        if ( ((irq == hvm_isa_irq_to_gsi(0)) && pit_channel0_enabled()) ||
-                    is_hvm_callback_irq(vioapic, irq) )
-        {
-            v = vioapic_domain(vioapic)->vcpu[0];
-            target = v ? vcpu_vlapic(v) : NULL;
-        }
-        else
-#endif
-            target = apic_round_robin(vioapic_domain(vioapic),
+	target = apic_round_robin(vioapic_domain(vioapic),
                                       vector, deliver_bitmask);
         if ( target != NULL )
         {

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Windows SMP

[Venefax]

I am using two version of the drivers, with identical limitations. James' GPLPV and the Novell drivers.
Federico

-----Original Message-----
From: Steve Ofsthun [mailto:sofsthun@virtualiron.com] 
Sent: Monday, January 05, 2009 3:05 PM
To: Venefax
Cc: 'Keir Fraser'; 'Andrew Lyon'; 'James Harper'; 'Dirk Utterback'; xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] Windows SMP

Venefax wrote:
> I use only Windows 2003 R2 SP2 fully patched. Therefore that patch would not apply since it is for Vista (Novell shim (Vista/2008 guests). But I will ask Novell.
> I would love to try that second patch that would "force all guest callback interrupts to VCPU 0". My setup is to use one core for xen, only, and the rest 15 cores for virtual machines, as instructed by Novell support. I am not over-committing CPU's. Even if I run one single virtual machine with 8 CPU's, the performance is awful. It takes  10 mins to load an application that loads in 1.5-2.0 mins with one single CPU. I switched all my 16 virtual machines to Standard PC Hal and that is the only way that the application works. However, it is an administrative nightmare. Instead of managing 16 independent windows machines via VNC, I should be able to manage two, maybe one.
> Can you send me information regarding both patches mentioned?

I'll have to check into the state of the PAUSE exit patch.  We are doing the work on an Intel Tylersburg machine.  I don't think these are generally available yet.

I have attached the referenced callback IRQ routing patch.

Whose Windows PV-on-HVM drivers are you running?

Steve

> Federico
> 
> 
> -----Original Message-----
> From: Steve Ofsthun [mailto:sofsthun@virtualiron.com]
> Sent: Monday, January 05, 2009 1:59 PM
> To: Venefax
> Cc: 'Keir Fraser'; 'Andrew Lyon'; 'James Harper'; 'Dirk Utterback'; 
> xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] Windows SMP
> 
> Venefax wrote:
>> Dear Gentlemen
> 
>> I have a machine with Intel 7350, 4 quad core. I can test since I 
>> have two virtual machines with 8 vcpus, showing a gross overhead that 
>> makes them unsuited for business. I use SLES 10 SP2, and I don't know 
>> how to apply the patch, but if somebody can log in and apply it, we 
>> can see the results immediately. The issue is affecting me directly. 
>> My two VM's have a VOIP application, very intensive in network and CPU usage.
> 
> If your problem isn't the previously discussed TPR issues ...
> 
> What Windows version are you running in your guests?
> 
> Windows 2000 SMP has serious problems due to the CPU wasting idle loop.  We avoid this with a special idler daemon running in the guest.
> 
> 
> Are you over-committing VCPUS at all (more than one active VCPU per CPU)?
> 
> We have noticed significant performance degradation with SMP windows once you over-commit VCPUS on the host.  This seems to be due to excessive guest spinlock overhead caused by the spinning VCPUS wasting their entire quantum looping for the guest lock at the same time they are preventing the guest lock holding VCPU from running.  We have been experimenting with new CPU features for exiting on PAUSE instructions.  This can possibly be used to detect the CPU wasting spinners.
> 
> 
> Have you tried using the Novell shim (Vista/2008 guests)?
> 
> I'm not sure the shim provides locking enhancements, but we have seen benefits for certain workloads.
> 
> 
> Are you running PV on HVM drivers?
> 
> One last area of concern may be distributing callback interrupts to all VCPUS.  We still run Xen (SLES10 SP2 based) with a patch to force all guest callback interrupts to VCPU 0.  This has consistently improved our SMP guest I/O performance while running PV-on-HVM drivers.
> 
> 
> Steve
> 
>> Federico
>> -----Original Message-----
>> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
>> Sent: Wednesday, December 31, 2008 5:10 AM
>> To: Andrew Lyon
>> Cc: Venefax; James Harper; Dirk Utterback; 
>> xen-devel@lists.xensource.com
>> Subject: Re: [Xen-devel] Windows SMP
>>
>> On 31/12/2008 10:08, "Andrew Lyon" <andrew.lyon@gmail.com> wrote:
>>
>>> Thanks Keir, I added a couple of printk's as you suggested and I can 
>>> now see if the feature is supported:
>>>
>>> Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz = (XEN) APIC Access
>>> virtualized
>>> Intel(R) Core(TM)2 CPU          6600  @ 2.40GHz = (XEN) APIC Access
>> emulated
>>> I didnt expect my Xeon system to support it, I've read some pdf's 
>>> about Intel virtualizaton features and they seemed to suggest it was 
>>> a new feature on 7xxx Xeons.
>>>
>>> The results fit the performance I've seen, a windows xp 32 bit hvm 
>>> with 2 cpu's runs a lot faster on the Xeon 2.5 than on the Core 2.4.
>> I think it is probably worth printing out. I'll add a patch to xen-unstable.
>>
>>  Thanks,
>>  Keir
>>
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
> 

RE: Windows SMP

[Venefax]

Dear Gentlemen
One silly question, maybe, how do I apply the patch supplied in SLES SP2? I have not compiled anything from sources.
Federico

Re: Windows SMP

[Steve Ofsthun]

[-- Attachment #1: Type: text/plain, Size: 1006 bytes --]

Venefax wrote:
> Dear Gentlemen
> One silly question, maybe, how do I apply the patch supplied in SLES SP2? I have not compiled anything from sources.
> Federico

I believe the proper sequence is:

download the SLES10 SDK iso from you Novell support account:

SLE-10-SP2-SDK-DVD-x86_64-GM-DVD1.iso

Add this ISO as a yast installation source.

install the xen-3.2.0_16718_14-0.4.src.rpm rpm via yast or:

# rpm -i xen-3.2.0_16718_14-0.4.src.rpm

Note that this is the .src.rpm not the binary rpm.

Prep the source tree using:

# rpmbuild -bc /usr/src/packages/SPECS/xen.spec

This should build, patch and compile the package.  If it fails due to dependencies, add the missing RPMs using yast.

Once it succeeds, apply the attached patch (I attached the wrong one previously):

# cd /usr/src/packages/BUILD/xen-3.2-testing
# patch -p0 < ~/xen-vioapic-callback-routing.patch

Build the hypervisor:

# cd /usr/src/packages/BUILD/xen-3.2-testing
# make xen

try out the resulting xen/xen.gz manually.

Steve



[-- Attachment #2: xen-vioapic-callback-routing.patch --]
[-- Type: text/x-patch, Size: 2117 bytes --]

--- xen/arch/x86/hvm/vioapic.c.orig	2008-01-16 15:19:06.000000000 -0500
+++ xen/arch/x86/hvm/vioapic.c	2009-01-06 16:00:10.000000000 -0500
@@ -44,6 +44,7 @@
 #define IRQ0_SPECIAL_ROUTING 1
 
 static void vioapic_deliver(struct hvm_hw_vioapic *vioapic, int irq);
+static int is_hvm_callback_irq(struct hvm_hw_vioapic *vioapic,int irq);
 
 static unsigned long vioapic_read_indirect(struct hvm_hw_vioapic *vioapic,
                                            unsigned long addr,
@@ -334,7 +335,8 @@
     {
 #ifdef IRQ0_SPECIAL_ROUTING
         /* Force round-robin to pick VCPU 0 */
-        if ( (irq == hvm_isa_irq_to_gsi(0)) && pit_channel0_enabled() )
+        if ( ((irq == hvm_isa_irq_to_gsi(0)) && pit_channel0_enabled()) ||
+             is_hvm_callback_irq(vioapic, irq) )
         {
             v = vioapic_domain(vioapic)->vcpu[0];
             target = v ? vcpu_vlapic(v) : NULL;
@@ -366,7 +368,8 @@
             deliver_bitmask &= ~(1 << bit);
 #ifdef IRQ0_SPECIAL_ROUTING
             /* Do not deliver timer interrupts to VCPU != 0 */
-            if ( (irq == hvm_isa_irq_to_gsi(0)) && pit_channel0_enabled() )
+            if ( ((irq == hvm_isa_irq_to_gsi(0)) && pit_channel0_enabled()) ||
+                 is_hvm_callback_irq(vioapic, irq) )
                 v = vioapic_domain(vioapic)->vcpu[0];
             else
 #endif
@@ -544,3 +547,29 @@
     xfree(d->arch.hvm_domain.vioapic);
     d->arch.hvm_domain.vioapic = NULL;
 }
+
+static int is_hvm_callback_irq(struct hvm_hw_vioapic *vioapic,int irq)
+{
+   unsigned int gsi = VIOAPIC_NUM_PINS;
+   struct domain *d = vioapic_domain(vioapic);
+   struct hvm_irq *hvm_irq = &d->arch.hvm_domain.irq;
+
+
+   switch ( hvm_irq->callback_via_type )
+    {
+    case HVMIRQ_callback_gsi:
+        gsi = hvm_irq->callback_via.gsi;
+        break;
+    case HVMIRQ_callback_pci_intx:
+        gsi = hvm_pci_intx_gsi(hvm_irq->callback_via.pci.dev,
+                                hvm_irq->callback_via.pci.intx);
+        break;
+    default:
+        break;
+    }
+
+    if (gsi == irq)  /*Event Channel Interrupt*/
+      return(1);
+
+    return(0);
+}

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Windows SMP

[Venefax]

Dear Steve
I went as far as "make xen". The question is now what do I do to use this new, patched hypervisor, I mean, how do I put in production? A simple reboot? Please advise.
Federico

-----Original Message-----
From: Steve Ofsthun [mailto:sofsthun@virtualiron.com] 
Sent: Tuesday, January 06, 2009 4:20 PM
To: Venefax
Cc: 'Keir Fraser'; 'Andrew Lyon'; 'James Harper'; 'Dirk Utterback'; xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] Windows SMP

Venefax wrote:
> Dear Gentlemen
> One silly question, maybe, how do I apply the patch supplied in SLES SP2? I have not compiled anything from sources.
> Federico

I believe the proper sequence is:

download the SLES10 SDK iso from you Novell support account:

SLE-10-SP2-SDK-DVD-x86_64-GM-DVD1.iso

Add this ISO as a yast installation source.

install the xen-3.2.0_16718_14-0.4.src.rpm rpm via yast or:

# rpm -i xen-3.2.0_16718_14-0.4.src.rpm

Note that this is the .src.rpm not the binary rpm.

Prep the source tree using:

# rpmbuild -bc /usr/src/packages/SPECS/xen.spec

This should build, patch and compile the package.  If it fails due to dependencies, add the missing RPMs using yast.

Once it succeeds, apply the attached patch (I attached the wrong one previously):

# cd /usr/src/packages/BUILD/xen-3.2-testing
# patch -p0 < ~/xen-vioapic-callback-routing.patch

Build the hypervisor:

# cd /usr/src/packages/BUILD/xen-3.2-testing
# make xen

try out the resulting xen/xen.gz manually.

Steve

Re: Windows SMP

[Steve Ofsthun]

Venefax wrote:

> I went as far as "make xen". The question is now what do I do to use this new, patched hypervisor, I mean, how do I put in production? A simple reboot? Please advise.

As we don't want to assume that this test xen will even boot, the safest thing is to edit your /boot/grub/menu.lst (or use yast [System -> Boot Loader]) to copy your current xen boot entry.  Leave everything identical between the two entries except for the hypervisor (change /boot/xen.gz to /boot/xen-test.gz) and the title line to indicate the test entry (add something like "- test" to the end of the title line).  If you use yast, the title entry is in the label column.

Copy your newly created xen.gz to /boot/xen-test.gz (be careful not to copy your new file on top of any existing /boot/xen.gz).

Reboot the node and manually select the test selection.  If the boot succeeds, try your experiment.  If the boot fails, just reboot normally and verify your changes to the menu.lst.

> Federico
> 
> -----Original Message-----
> From: Steve Ofsthun [mailto:sofsthun@virtualiron.com] 
> Sent: Tuesday, January 06, 2009 4:20 PM
> To: Venefax
> Cc: 'Keir Fraser'; 'Andrew Lyon'; 'James Harper'; 'Dirk Utterback'; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] Windows SMP
> 
> Venefax wrote:
>> Dear Gentlemen
>> One silly question, maybe, how do I apply the patch supplied in SLES SP2? I have not compiled anything from sources.
>> Federico
> 
> I believe the proper sequence is:
> 
> download the SLES10 SDK iso from you Novell support account:
> 
> SLE-10-SP2-SDK-DVD-x86_64-GM-DVD1.iso
> 
> Add this ISO as a yast installation source.
> 
> install the xen-3.2.0_16718_14-0.4.src.rpm rpm via yast or:
> 
> # rpm -i xen-3.2.0_16718_14-0.4.src.rpm
> 
> Note that this is the .src.rpm not the binary rpm.
> 
> Prep the source tree using:
> 
> # rpmbuild -bc /usr/src/packages/SPECS/xen.spec
> 
> This should build, patch and compile the package.  If it fails due to dependencies, add the missing RPMs using yast.
> 
> Once it succeeds, apply the attached patch (I attached the wrong one previously):
> 
> # cd /usr/src/packages/BUILD/xen-3.2-testing
> # patch -p0 < ~/xen-vioapic-callback-routing.patch
> 
> Build the hypervisor:
> 
> # cd /usr/src/packages/BUILD/xen-3.2-testing
> # make xen
> 
> try out the resulting xen/xen.gz manually.
> 
> Steve
> 
> 
> 

RE: Windows SMP

[Venefax]

It booted fine, but then I found out that I actually went back in my version. Please let me know if I should go ahead or not. I downloaded the source file that you instructed me, xen-3.2.0_16718_14-0.4.src.rpm, but it turns out that I was using 3.2.0_16718_18-0.3.src.rpm, which I could not find in Google or Novell. I applied the patch and after rebooting I found that I am of course using _14-0.4 instead of 18-0.3. So the question is, does it matter? Or is it better to use this lower version with the patch? If it matters indeed, how do I get the _18.03 source rpm?
The other question is: is it possible to update the whole xen + tools to 3.3.1rc4 in Suse 10 SP2 and is this patch still valid in that scenario?
Federico


-----Original Message-----
From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Steve Ofsthun
Sent: Wednesday, January 07, 2009 9:47 AM
To: Venefax
Cc: xen-devel@lists.xensource.com; 'Andrew Lyon'; 'Dirk Utterback'; 'Keir Fraser'; 'James Harper'
Subject: Re: [Xen-devel] Windows SMP

Venefax wrote:

> I went as far as "make xen". The question is now what do I do to use this new, patched hypervisor, I mean, how do I put in production? A simple reboot? Please advise.

As we don't want to assume that this test xen will even boot, the safest thing is to edit your /boot/grub/menu.lst (or use yast [System -> Boot Loader]) to copy your current xen boot entry.  Leave everything identical between the two entries except for the hypervisor (change /boot/xen.gz to /boot/xen-test.gz) and the title line to indicate the test entry (add something like "- test" to the end of the title line).  If you use yast, the title entry is in the label column.

Copy your newly created xen.gz to /boot/xen-test.gz (be careful not to copy your new file on top of any existing /boot/xen.gz).

Reboot the node and manually select the test selection.  If the boot succeeds, try your experiment.  If the boot fails, just reboot normally and verify your changes to the menu.lst.

> Federico
> 
> -----Original Message-----
> From: Steve Ofsthun [mailto:sofsthun@virtualiron.com] 
> Sent: Tuesday, January 06, 2009 4:20 PM
> To: Venefax
> Cc: 'Keir Fraser'; 'Andrew Lyon'; 'James Harper'; 'Dirk Utterback'; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] Windows SMP
> 
> Venefax wrote:
>> Dear Gentlemen
>> One silly question, maybe, how do I apply the patch supplied in SLES SP2? I have not compiled anything from sources.
>> Federico
> 
> I believe the proper sequence is:
> 
> download the SLES10 SDK iso from you Novell support account:
> 
> SLE-10-SP2-SDK-DVD-x86_64-GM-DVD1.iso
> 
> Add this ISO as a yast installation source.
> 
> install the xen-3.2.0_16718_14-0.4.src.rpm rpm via yast or:
> 
> # rpm -i xen-3.2.0_16718_14-0.4.src.rpm
> 
> Note that this is the .src.rpm not the binary rpm.
> 
> Prep the source tree using:
> 
> # rpmbuild -bc /usr/src/packages/SPECS/xen.spec
> 
> This should build, patch and compile the package.  If it fails due to dependencies, add the missing RPMs using yast.
> 
> Once it succeeds, apply the attached patch (I attached the wrong one previously):
> 
> # cd /usr/src/packages/BUILD/xen-3.2-testing
> # patch -p0 < ~/xen-vioapic-callback-routing.patch
> 
> Build the hypervisor:
> 
> # cd /usr/src/packages/BUILD/xen-3.2-testing
> # make xen
> 
> try out the resulting xen/xen.gz manually.
> 
> Steve
> 
> 
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Windows SMP

[Steve Ofsthun]

Venefax wrote:
> It booted fine, but then I found out that I actually went back in my version. Please let me know if I should go ahead or not. I downloaded the source file that you instructed me, xen-3.2.0_16718_14-0.4.src.rpm, but it turns out that I was using 3.2.0_16718_18-0.3.src.rpm, which I could not find in Google or Novell. I applied the patch and after rebooting I found that I am of course using _14-0.4 instead of 18-0.3. So the question is, does it matter? Or is it better to use this lower version with the patch? If it matters indeed, how do I get the _18.03 source rpm?

Obviously running the correct version would be best.  I'm guessing my version is the original shipping SP2 not the latest SP2 kernel/xen update.  You should be able to get the proper src.rpm from the same place you got the binary rpm from.

> The other question is: is it possible to update the whole xen + tools to 3.3.1rc4 in Suse 10 SP2 and is this patch still valid in that scenario?

The short answer is no.  This is a much bigger job.  The dom0 kernel is intimately tied to the underlying Xen version.  If you want to try a newer Xen, you would have to move to OpenSUSE 11.1 or gain access to the SLES11 Beta.

As far as the supplied patch, it should apply since the underlying code is still present in xen-unstable.

Steve

> Federico
> 
> 
> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Steve Ofsthun
> Sent: Wednesday, January 07, 2009 9:47 AM
> To: Venefax
> Cc: xen-devel@lists.xensource.com; 'Andrew Lyon'; 'Dirk Utterback'; 'Keir Fraser'; 'James Harper'
> Subject: Re: [Xen-devel] Windows SMP
> 
> Venefax wrote:
> 
>> I went as far as "make xen". The question is now what do I do to use this new, patched hypervisor, I mean, how do I put in production? A simple reboot? Please advise.
> 
> As we don't want to assume that this test xen will even boot, the safest thing is to edit your /boot/grub/menu.lst (or use yast [System -> Boot Loader]) to copy your current xen boot entry.  Leave everything identical between the two entries except for the hypervisor (change /boot/xen.gz to /boot/xen-test.gz) and the title line to indicate the test entry (add something like "- test" to the end of the title line).  If you use yast, the title entry is in the label column.
> 
> Copy your newly created xen.gz to /boot/xen-test.gz (be careful not to copy your new file on top of any existing /boot/xen.gz).
> 
> Reboot the node and manually select the test selection.  If the boot succeeds, try your experiment.  If the boot fails, just reboot normally and verify your changes to the menu.lst.
> 
>> Federico
>>
>> -----Original Message-----
>> From: Steve Ofsthun [mailto:sofsthun@virtualiron.com] 
>> Sent: Tuesday, January 06, 2009 4:20 PM
>> To: Venefax
>> Cc: 'Keir Fraser'; 'Andrew Lyon'; 'James Harper'; 'Dirk Utterback'; xen-devel@lists.xensource.com
>> Subject: Re: [Xen-devel] Windows SMP
>>
>> Venefax wrote:
>>> Dear Gentlemen
>>> One silly question, maybe, how do I apply the patch supplied in SLES SP2? I have not compiled anything from sources.
>>> Federico
>> I believe the proper sequence is:
>>
>> download the SLES10 SDK iso from you Novell support account:
>>
>> SLE-10-SP2-SDK-DVD-x86_64-GM-DVD1.iso
>>
>> Add this ISO as a yast installation source.
>>
>> install the xen-3.2.0_16718_14-0.4.src.rpm rpm via yast or:
>>
>> # rpm -i xen-3.2.0_16718_14-0.4.src.rpm
>>
>> Note that this is the .src.rpm not the binary rpm.
>>
>> Prep the source tree using:
>>
>> # rpmbuild -bc /usr/src/packages/SPECS/xen.spec
>>
>> This should build, patch and compile the package.  If it fails due to dependencies, add the missing RPMs using yast.
>>
>> Once it succeeds, apply the attached patch (I attached the wrong one previously):
>>
>> # cd /usr/src/packages/BUILD/xen-3.2-testing
>> # patch -p0 < ~/xen-vioapic-callback-routing.patch
>>
>> Build the hypervisor:
>>
>> # cd /usr/src/packages/BUILD/xen-3.2-testing
>> # make xen
>>
>> try out the resulting xen/xen.gz manually.
>>
>> Steve
>>
>>
>>
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

RE: Windows SMP

[Venefax]

I put the patch in production. Is there any way to know if it is working. I cannot observe any effect, either positive or negative. I have 16 windows VM with one virtual processor each, the Standard PC Hal, and an application that can bring the cpu usage to 100%. Heavy network usage. It is a telephony application. Does anybody have any other patch that would make it use less CPU?
Federico

-----Original Message-----
From: Steve Ofsthun [mailto:sofsthun@virtualiron.com] 
Sent: Wednesday, January 07, 2009 10:42 AM
To: Venefax
Cc: xen-devel@lists.xensource.com; 'Andrew Lyon'; 'Dirk Utterback'; 'Keir Fraser'
Subject: Re: [Xen-devel] Windows SMP

Venefax wrote:
> It booted fine, but then I found out that I actually went back in my version. Please let me know if I should go ahead or not. I downloaded the source file that you instructed me, xen-3.2.0_16718_14-0.4.src.rpm, but it turns out that I was using 3.2.0_16718_18-0.3.src.rpm, which I could not find in Google or Novell. I applied the patch and after rebooting I found that I am of course using _14-0.4 instead of 18-0.3. So the question is, does it matter? Or is it better to use this lower version with the patch? If it matters indeed, how do I get the _18.03 source rpm?

Obviously running the correct version would be best.  I'm guessing my version is the original shipping SP2 not the latest SP2 kernel/xen update.  You should be able to get the proper src.rpm from the same place you got the binary rpm from.

> The other question is: is it possible to update the whole xen + tools to 3.3.1rc4 in Suse 10 SP2 and is this patch still valid in that scenario?

The short answer is no.  This is a much bigger job.  The dom0 kernel is intimately tied to the underlying Xen version.  If you want to try a newer Xen, you would have to move to OpenSUSE 11.1 or gain access to the SLES11 Beta.

As far as the supplied patch, it should apply since the underlying code is still present in xen-unstable.

Steve

> Federico
> 
> 
> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Steve Ofsthun
> Sent: Wednesday, January 07, 2009 9:47 AM
> To: Venefax
> Cc: xen-devel@lists.xensource.com; 'Andrew Lyon'; 'Dirk Utterback'; 'Keir Fraser'; 'James Harper'
> Subject: Re: [Xen-devel] Windows SMP
> 
> Venefax wrote:
> 
>> I went as far as "make xen". The question is now what do I do to use this new, patched hypervisor, I mean, how do I put in production? A simple reboot? Please advise.
> 
> As we don't want to assume that this test xen will even boot, the safest thing is to edit your /boot/grub/menu.lst (or use yast [System -> Boot Loader]) to copy your current xen boot entry.  Leave everything identical between the two entries except for the hypervisor (change /boot/xen.gz to /boot/xen-test.gz) and the title line to indicate the test entry (add something like "- test" to the end of the title line).  If you use yast, the title entry is in the label column.
> 
> Copy your newly created xen.gz to /boot/xen-test.gz (be careful not to copy your new file on top of any existing /boot/xen.gz).
> 
> Reboot the node and manually select the test selection.  If the boot succeeds, try your experiment.  If the boot fails, just reboot normally and verify your changes to the menu.lst.
> 
>> Federico
>>
>> -----Original Message-----
>> From: Steve Ofsthun [mailto:sofsthun@virtualiron.com] 
>> Sent: Tuesday, January 06, 2009 4:20 PM
>> To: Venefax
>> Cc: 'Keir Fraser'; 'Andrew Lyon'; 'James Harper'; 'Dirk Utterback'; xen-devel@lists.xensource.com
>> Subject: Re: [Xen-devel] Windows SMP
>>
>> Venefax wrote:
>>> Dear Gentlemen
>>> One silly question, maybe, how do I apply the patch supplied in SLES SP2? I have not compiled anything from sources.
>>> Federico
>> I believe the proper sequence is:
>>
>> download the SLES10 SDK iso from you Novell support account:
>>
>> SLE-10-SP2-SDK-DVD-x86_64-GM-DVD1.iso
>>
>> Add this ISO as a yast installation source.
>>
>> install the xen-3.2.0_16718_14-0.4.src.rpm rpm via yast or:
>>
>> # rpm -i xen-3.2.0_16718_14-0.4.src.rpm
>>
>> Note that this is the .src.rpm not the binary rpm.
>>
>> Prep the source tree using:
>>
>> # rpmbuild -bc /usr/src/packages/SPECS/xen.spec
>>
>> This should build, patch and compile the package.  If it fails due to dependencies, add the missing RPMs using yast.
>>
>> Once it succeeds, apply the attached patch (I attached the wrong one previously):
>>
>> # cd /usr/src/packages/BUILD/xen-3.2-testing
>> # patch -p0 < ~/xen-vioapic-callback-routing.patch
>>
>> Build the hypervisor:
>>
>> # cd /usr/src/packages/BUILD/xen-3.2-testing
>> # make xen
>>
>> try out the resulting xen/xen.gz manually.
>>
>> Steve
>>
>>
>>
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

Re: Windows SMP

[Steve Ofsthun]

Venefax wrote:
> I put the patch in production. Is there any way to know if it is working. I cannot observe any effect, either positive or negative. I have 16 windows VM with one virtual processor each, the Standard PC Hal, and an application that can bring the cpu usage to 100%. Heavy network usage. It is a telephony application. Does anybody have any other patch that would make it use less CPU?

Now I am confused.  I thought your original problem was related to guests with multiple vcpus (as the subject implies).  The patch addresses additional latency added when more than 1 vcpu is involved in the interrupt delivery.  The patch will have no effect on 1-vcpu guests.  With only 1 vcpu, interrupts can only be delivered to one place already (vcpu 0).

Steve

> Federico
> 
> -----Original Message-----
> From: Steve Ofsthun [mailto:sofsthun@virtualiron.com] 
> Sent: Wednesday, January 07, 2009 10:42 AM
> To: Venefax
> Cc: xen-devel@lists.xensource.com; 'Andrew Lyon'; 'Dirk Utterback'; 'Keir Fraser'
> Subject: Re: [Xen-devel] Windows SMP
> 
> Venefax wrote:
>> It booted fine, but then I found out that I actually went back in my version. Please let me know if I should go ahead or not. I downloaded the source file that you instructed me, xen-3.2.0_16718_14-0.4.src.rpm, but it turns out that I was using 3.2.0_16718_18-0.3.src.rpm, which I could not find in Google or Novell. I applied the patch and after rebooting I found that I am of course using _14-0.4 instead of 18-0.3. So the question is, does it matter? Or is it better to use this lower version with the patch? If it matters indeed, how do I get the _18.03 source rpm?
> 
> Obviously running the correct version would be best.  I'm guessing my version is the original shipping SP2 not the latest SP2 kernel/xen update.  You should be able to get the proper src.rpm from the same place you got the binary rpm from.
> 
>> The other question is: is it possible to update the whole xen + tools to 3.3.1rc4 in Suse 10 SP2 and is this patch still valid in that scenario?
> 
> The short answer is no.  This is a much bigger job.  The dom0 kernel is intimately tied to the underlying Xen version.  If you want to try a newer Xen, you would have to move to OpenSUSE 11.1 or gain access to the SLES11 Beta.
> 
> As far as the supplied patch, it should apply since the underlying code is still present in xen-unstable.
> 
> Steve
> 
>> Federico
>>
>>
>> -----Original Message-----
>> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Steve Ofsthun
>> Sent: Wednesday, January 07, 2009 9:47 AM
>> To: Venefax
>> Cc: xen-devel@lists.xensource.com; 'Andrew Lyon'; 'Dirk Utterback'; 'Keir Fraser'; 'James Harper'
>> Subject: Re: [Xen-devel] Windows SMP
>>
>> Venefax wrote:
>>
>>> I went as far as "make xen". The question is now what do I do to use this new, patched hypervisor, I mean, how do I put in production? A simple reboot? Please advise.
>> As we don't want to assume that this test xen will even boot, the safest thing is to edit your /boot/grub/menu.lst (or use yast [System -> Boot Loader]) to copy your current xen boot entry.  Leave everything identical between the two entries except for the hypervisor (change /boot/xen.gz to /boot/xen-test.gz) and the title line to indicate the test entry (add something like "- test" to the end of the title line).  If you use yast, the title entry is in the label column.
>>
>> Copy your newly created xen.gz to /boot/xen-test.gz (be careful not to copy your new file on top of any existing /boot/xen.gz).
>>
>> Reboot the node and manually select the test selection.  If the boot succeeds, try your experiment.  If the boot fails, just reboot normally and verify your changes to the menu.lst.
>>
>>> Federico
>>>
>>> -----Original Message-----
>>> From: Steve Ofsthun [mailto:sofsthun@virtualiron.com] 
>>> Sent: Tuesday, January 06, 2009 4:20 PM
>>> To: Venefax
>>> Cc: 'Keir Fraser'; 'Andrew Lyon'; 'James Harper'; 'Dirk Utterback'; xen-devel@lists.xensource.com
>>> Subject: Re: [Xen-devel] Windows SMP
>>>
>>> Venefax wrote:
>>>> Dear Gentlemen
>>>> One silly question, maybe, how do I apply the patch supplied in SLES SP2? I have not compiled anything from sources.
>>>> Federico
>>> I believe the proper sequence is:
>>>
>>> download the SLES10 SDK iso from you Novell support account:
>>>
>>> SLE-10-SP2-SDK-DVD-x86_64-GM-DVD1.iso
>>>
>>> Add this ISO as a yast installation source.
>>>
>>> install the xen-3.2.0_16718_14-0.4.src.rpm rpm via yast or:
>>>
>>> # rpm -i xen-3.2.0_16718_14-0.4.src.rpm
>>>
>>> Note that this is the .src.rpm not the binary rpm.
>>>
>>> Prep the source tree using:
>>>
>>> # rpmbuild -bc /usr/src/packages/SPECS/xen.spec
>>>
>>> This should build, patch and compile the package.  If it fails due to dependencies, add the missing RPMs using yast.
>>>
>>> Once it succeeds, apply the attached patch (I attached the wrong one previously):
>>>
>>> # cd /usr/src/packages/BUILD/xen-3.2-testing
>>> # patch -p0 < ~/xen-vioapic-callback-routing.patch
>>>
>>> Build the hypervisor:
>>>
>>> # cd /usr/src/packages/BUILD/xen-3.2-testing
>>> # make xen
>>>
>>> try out the resulting xen/xen.gz manually.
>>>
>>> Steve
>>>
>>>
>>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>
> 

RE: Windows SMP

[Venefax]

Dear Steve
I am trying to apply the xen-vioapic-callback-routing.patch to xen 3.3 as supplied by Suse 11.1. The question is: is it necessary? I use both SMP (8 cores) para-virtualized Centos 5.2 and fully-virtualized Windows SMP (4 cores). Do you recommend that I apply this patch? Or the patch is already in Xen 3.3?
Yours
Federico

-----Original Message-----
From: Steve Ofsthun [mailto:sofsthun@virtualiron.com] 
Sent: Friday, January 09, 2009 9:30 AM
To: Venefax
Cc: xen-devel@lists.xensource.com; 'Andrew Lyon'; 'Dirk Utterback'; 'Keir Fraser'
Subject: Re: [Xen-devel] Windows SMP

Venefax wrote:
> I put the patch in production. Is there any way to know if it is working. I cannot observe any effect, either positive or negative. I have 16 windows VM with one virtual processor each, the Standard PC Hal, and an application that can bring the cpu usage to 100%. Heavy network usage. It is a telephony application. Does anybody have any other patch that would make it use less CPU?

Now I am confused.  I thought your original problem was related to guests with multiple vcpus (as the subject implies).  The patch addresses additional latency added when more than 1 vcpu is involved in the interrupt delivery.  The patch will have no effect on 1-vcpu guests.  With only 1 vcpu, interrupts can only be delivered to one place already (vcpu 0).

Steve

> Federico
> 
> -----Original Message-----
> From: Steve Ofsthun [mailto:sofsthun@virtualiron.com] 
> Sent: Wednesday, January 07, 2009 10:42 AM
> To: Venefax
> Cc: xen-devel@lists.xensource.com; 'Andrew Lyon'; 'Dirk Utterback'; 'Keir Fraser'
> Subject: Re: [Xen-devel] Windows SMP
> 
> Venefax wrote:
>> It booted fine, but then I found out that I actually went back in my version. Please let me know if I should go ahead or not. I downloaded the source file that you instructed me, xen-3.2.0_16718_14-0.4.src.rpm, but it turns out that I was using 3.2.0_16718_18-0.3.src.rpm, which I could not find in Google or Novell. I applied the patch and after rebooting I found that I am of course using _14-0.4 instead of 18-0.3. So the question is, does it matter? Or is it better to use this lower version with the patch? If it matters indeed, how do I get the _18.03 source rpm?
> 
> Obviously running the correct version would be best.  I'm guessing my version is the original shipping SP2 not the latest SP2 kernel/xen update.  You should be able to get the proper src.rpm from the same place you got the binary rpm from.
> 
>> The other question is: is it possible to update the whole xen + tools to 3.3.1rc4 in Suse 10 SP2 and is this patch still valid in that scenario?
> 
> The short answer is no.  This is a much bigger job.  The dom0 kernel is intimately tied to the underlying Xen version.  If you want to try a newer Xen, you would have to move to OpenSUSE 11.1 or gain access to the SLES11 Beta.
> 
> As far as the supplied patch, it should apply since the underlying code is still present in xen-unstable.
> 
> Steve
> 
>> Federico
>>
>>
>> -----Original Message-----
>> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Steve Ofsthun
>> Sent: Wednesday, January 07, 2009 9:47 AM
>> To: Venefax
>> Cc: xen-devel@lists.xensource.com; 'Andrew Lyon'; 'Dirk Utterback'; 'Keir Fraser'; 'James Harper'
>> Subject: Re: [Xen-devel] Windows SMP
>>
>> Venefax wrote:
>>
>>> I went as far as "make xen". The question is now what do I do to use this new, patched hypervisor, I mean, how do I put in production? A simple reboot? Please advise.
>> As we don't want to assume that this test xen will even boot, the safest thing is to edit your /boot/grub/menu.lst (or use yast [System -> Boot Loader]) to copy your current xen boot entry.  Leave everything identical between the two entries except for the hypervisor (change /boot/xen.gz to /boot/xen-test.gz) and the title line to indicate the test entry (add something like "- test" to the end of the title line).  If you use yast, the title entry is in the label column.
>>
>> Copy your newly created xen.gz to /boot/xen-test.gz (be careful not to copy your new file on top of any existing /boot/xen.gz).
>>
>> Reboot the node and manually select the test selection.  If the boot succeeds, try your experiment.  If the boot fails, just reboot normally and verify your changes to the menu.lst.
>>
>>> Federico
>>>
>>> -----Original Message-----
>>> From: Steve Ofsthun [mailto:sofsthun@virtualiron.com] 
>>> Sent: Tuesday, January 06, 2009 4:20 PM
>>> To: Venefax
>>> Cc: 'Keir Fraser'; 'Andrew Lyon'; 'James Harper'; 'Dirk Utterback'; xen-devel@lists.xensource.com
>>> Subject: Re: [Xen-devel] Windows SMP
>>>
>>> Venefax wrote:
>>>> Dear Gentlemen
>>>> One silly question, maybe, how do I apply the patch supplied in SLES SP2? I have not compiled anything from sources.
>>>> Federico
>>> I believe the proper sequence is:
>>>
>>> download the SLES10 SDK iso from you Novell support account:
>>>
>>> SLE-10-SP2-SDK-DVD-x86_64-GM-DVD1.iso
>>>
>>> Add this ISO as a yast installation source.
>>>
>>> install the xen-3.2.0_16718_14-0.4.src.rpm rpm via yast or:
>>>
>>> # rpm -i xen-3.2.0_16718_14-0.4.src.rpm
>>>
>>> Note that this is the .src.rpm not the binary rpm.
>>>
>>> Prep the source tree using:
>>>
>>> # rpmbuild -bc /usr/src/packages/SPECS/xen.spec
>>>
>>> This should build, patch and compile the package.  If it fails due to dependencies, add the missing RPMs using yast.
>>>
>>> Once it succeeds, apply the attached patch (I attached the wrong one previously):
>>>
>>> # cd /usr/src/packages/BUILD/xen-3.2-testing
>>> # patch -p0 < ~/xen-vioapic-callback-routing.patch
>>>
>>> Build the hypervisor:
>>>
>>> # cd /usr/src/packages/BUILD/xen-3.2-testing
>>> # make xen
>>>
>>> try out the resulting xen/xen.gz manually.
>>>
>>> Steve
>>>
>>>
>>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>
> 

RE: Windows SMP

[Ian Pratt]

> > As its not a cpuid feature I guess I will  have to check the Intel
> > manuals for each cpu model, do you know what the feature is
described
> > as in Intel documentation?
> 
> In the architecture manual it is known as apic-access virtualisation.
I
> don't know whether it would be a headline feature in a processor's
data
> sheet.

I believe Intel's marketing moniker for the feature is "FlexPriority".

Ian