SMP support on AMD64

SMP support on AMD64

[Lynn Kerby]

Hi KVM developers (and lurkers like myself).

Apologies in advance as this is probably not the right forum for this  
question.

I've had no success getting an SMP virtual machine running on my AMD  
Athlon64 X2 server.  The base system is Ubuntu 7.10 (recently  
upgraded from 7.04) with a 2.6.22 kernel and I've tried kvm releases  
49 & 50 with no success.  My memory of whether or not it worked  
months ago with kvm-28 on the Ubuntu 2.6.20-16 kernel is no longer  
with me :-(.  The recent failures have all been BIOS problems at  
rombios.c:10605.  I'm pretty sure I must have something wrong in my  
environment and have spent entirely too much of my spare time trying  
various things with no success.

Can someone that has actually has an SMP virtual machine running  
currently on an AMD processor (preferably a 64-bit guest) give me  
some details on the base system kernel, KVM related bits, and the  
guest OS?

Lynn Kerby
San Martin, CA

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: SMP support on AMD64

[Izik Eidus]

On Tue, 2007-11-06 at 18:31 -0800, Lynn Kerby wrote:
> Hi KVM developers (and lurkers like myself).
> 
> Apologies in advance as this is probably not the right forum for this  
> question.
> 
> I've had no success getting an SMP virtual machine running on my AMD  
> Athlon64 X2 server.  The base system is Ubuntu 7.10 (recently  
> upgraded from 7.04) with a 2.6.22 kernel and I've tried kvm releases  
> 49 & 50 with no success.  My memory of whether or not it worked  
> months ago with kvm-28 on the Ubuntu 2.6.20-16 kernel is no longer  
> with me :-(.  The recent failures have all been BIOS problems at  
> rombios.c:10605.  I'm pretty sure I must have something wrong in my  
> environment and have spent entirely too much of my spare time trying  
> various things with no success.

yes, we got some problem with smp on amd right now
i hope it will be fixed in the next month (but i cant promise that)


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: SMP support on AMD64

[Avi Kivity]

[-- Attachment #1: Type: text/plain, Size: 1683 bytes --]

Lynn Kerby wrote:
> Hi KVM developers (and lurkers like myself).
>
> Apologies in advance as this is probably not the right forum for this  
> question.
>   

It is certainly the right forum.

> I've had no success getting an SMP virtual machine running on my AMD  
> Athlon64 X2 server.  The base system is Ubuntu 7.10 (recently  
> upgraded from 7.04) with a 2.6.22 kernel and I've tried kvm releases  
> 49 & 50 with no success.  My memory of whether or not it worked  
> months ago with kvm-28 on the Ubuntu 2.6.20-16 kernel is no longer  
> with me :-(.  The recent failures have all been BIOS problems at  
> rombios.c:10605.  I'm pretty sure I must have something wrong in my  
> environment and have spent entirely too much of my spare time trying  
> various things with no success.
>   

Yes you've been doing something wrong -- you should have reported this 
quickly.  Worst case somebody would have pointed out something trivial 
you're doing wrong and you'd have felt foolish for a bit.  But in this 
particular case...

> Can someone that has actually has an SMP virtual machine running  
> currently on an AMD processor (preferably a 64-bit guest) give me  
> some details on the base system kernel, KVM related bits, and the  
> guest OS?
>
>   

Turns out kvm smp on amd is broken.  I committed a fix for kvm-51, and 
now I have FC6 running happily with two vcpus.

If you see problems, please report them soon.  I'd rather have noise on 
the list than problems going unreported.

I'm attaching the patch which fixes the problem, let us know if it helps 
or not (there may be more problems lurking).

-- 
error compiling committee.c: too many arguments to function


[-- Attachment #2: amd-smp.patch --]
[-- Type: text/x-patch, Size: 752 bytes --]

commit 157117844e1d228db58dfe7fbd51c5a225ceafe4
Author: Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org>
Date:   Wed Nov 7 12:57:23 2007 +0200

    KVM: SVM: Fix SMP with kernel apic
    
    AP processor needs to reset to the SIPI vector, not normal INIT.
    
    Signed-off-by: Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org>

diff --git a/kernel/svm.c b/kernel/svm.c
index 95a3489..ea2cb83 100644
--- a/kernel/svm.c
+++ b/kernel/svm.c
@@ -563,6 +563,12 @@ static int svm_vcpu_reset(struct kvm_vcpu *vcpu)
 
 	init_vmcb(svm->vmcb);
 
+	if (vcpu->vcpu_id != 0) {
+		svm->vmcb->save.rip = 0;
+		svm->vmcb->save.cs.base = svm->vcpu.sipi_vector << 12;
+		svm->vmcb->save.cs.selector = svm->vcpu.sipi_vector << 8;
+	}
+
 	return 0;
 }
 

[-- Attachment #3: Type: text/plain, Size: 314 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

[-- Attachment #4: Type: text/plain, Size: 186 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Re: SMP support on AMD64

[Lynn Kerby]

On Nov 7, 2007, at 3:06 AM, Avi Kivity wrote:

> Lynn Kerby wrote:
>> Hi KVM developers (and lurkers like myself).
>>
>> Apologies in advance as this is probably not the right forum for  
>> this  question.
>>
>
> It is certainly the right forum.

Good to know.  There doesn't seem to be a lot of "user" oriented info  
for KVM and I was wondering if I just couldn't find the forums or what.

>> I've had no success getting an SMP virtual machine running on my  
>> AMD  Athlon64 X2 server.  The base system is Ubuntu 7.10  
>> (recently  upgraded from 7.04) with a 2.6.22 kernel and I've tried  
>> kvm releases  49 & 50 with no success.  My memory of whether or  
>> not it worked  months ago with kvm-28 on the Ubuntu 2.6.20-16  
>> kernel is no longer  with me :-(.  The recent failures have all  
>> been BIOS problems at  rombios.c:10605.  I'm pretty sure I must  
>> have something wrong in my  environment and have spent entirely  
>> too much of my spare time trying  various things with no success.
>>
>
> Yes you've been doing something wrong -- you should have reported  
> this quickly.  Worst case somebody would have pointed out something  
> trivial you're doing wrong and you'd have felt foolish for a bit.   
> But in this particular case...

I'd assumed (there's *that* word) that core developer types were  
using KVM on all kinds of hardware and that basics like this would  
have been shaken out.  I set up a few KVM virtual machines about 6  
months ago with the intent of migrating my mail server and some other  
functions to them but never quite got it working to my satisfaction.   
Now I've got major problems with the 10+ year old hardware (and  
software) with that system and it is becoming urgent.

>> Can someone that has actually has an SMP virtual machine running   
>> currently on an AMD processor (preferably a 64-bit guest) give me   
>> some details on the base system kernel, KVM related bits, and the   
>> guest OS?
>>
>>
>
> Turns out kvm smp on amd is broken.  I committed a fix for kvm-51,  
> and now I have FC6 running happily with two vcpus.
>
> If you see problems, please report them soon.  I'd rather have  
> noise on the list than problems going unreported.
>
> I'm attaching the patch which fixes the problem, let us know if it  
> helps or not (there may be more problems lurking).

Yes, that patch works nicely for me as does the kvm-51 release.

Thanks,

Lynn Kerby
San Martin, CA

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: SMP support on AMD64

[Avi Kivity]

Lynn Kerby wrote:
> On Nov 7, 2007, at 3:06 AM, Avi Kivity wrote:
>
>   
>> Lynn Kerby wrote:
>>     
>>> Hi KVM developers (and lurkers like myself).
>>>
>>> Apologies in advance as this is probably not the right forum for  
>>> this  question.
>>>
>>>       
>> It is certainly the right forum.
>>     
>
> Good to know.  There doesn't seem to be a lot of "user" oriented info  
> for KVM and I was wondering if I just couldn't find the forums or what.
>
>   

Well, there's a lot of overlap between kvm and qemu, and there's little
setup or black magic needed to use kvm.  What particularly are you missing?

The intent is that kvm be very easy to use to a unix user proficient in
the command line, with the gui junkies served by what virt-manager aims
to be (or competing solutions).

>> Yes you've been doing something wrong -- you should have reported  
>> this quickly.  Worst case somebody would have pointed out something  
>> trivial you're doing wrong and you'd have felt foolish for a bit.   
>> But in this particular case...
>>     
>
> I'd assumed (there's *that* word) that core developer types were  
> using KVM on all kinds of hardware and that basics like this would  
> have been shaken out.  

Well, it is a reasonable assumption, but the sad fact is, we do little
regression testing on smp here, and Intel doesn't test AMD for some
reason.  It used to work but regressed.

That's something we need to improve on, and quickly.  I'm counting on
Anthony's plan to flood the known universe with kvm-testing nanobots.

> I set up a few KVM virtual machines about 6  
> months ago with the intent of migrating my mail server and some other  
> functions to them but never quite got it working to my satisfaction.   
> Now I've got major problems with the 10+ year old hardware (and  
> software) with that system and it is becoming urgent.
>   

Let us know what's not working and we will try to address it.


-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: SMP support on AMD64

[Alexey Eremenko]

Hi Avi,

Speaking of AMD host - with KVM-51 it doesn't crashes anymore !

But there is still fonts rendering problem with Fedora7/ 32-bit guests.

-- 
-Alexey Eremenko "Technologov"

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: SMP support on AMD64

[Avi Kivity]

Alexey Eremenko wrote:
> Hi Avi,
>
> Speaking of AMD host - with KVM-51 it doesn't crashes anymore !
>
> But there is still fonts rendering problem with Fedora7/ 32-bit guests.
>
>   

Can you update the relevant bug trackers?  Please add xpdyinfo output as
well.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: SMP support on AMD64

[Gildas]

> 2007/11/8, Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org>:
> > Lynn Kerby wrote:
[snip]
> > I'd assumed (there's *that* word) that core developer types were
> > using KVM on all kinds of hardware and that basics like this would
> > have been shaken out.
>
> Well, it is a reasonable assumption, but the sad fact is, we do little
> regression testing on smp here, and Intel doesn't test AMD for some
> reason.  It used to work but regressed.
>
> That's something we need to improve on, and quickly.  I'm counting on
> Anthony's plan to flood the known universe with kvm-testing nanobots.

Hi,

I've noted that there's less posts from the AMD guys lately (Wei
Huang, Joerg Roedel, Markus Rechberger), does it means that AMD
decided to stop supporting KVM?

If not, I think this is time for AMD to step in and use Anthony's
tools to provide a regression testing platform the same way Intel
does.

It would be really sad to see AMD CPUS being a second class citizen in KVM...

Just my 2 cents,
Gildas

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: SMP support on AMD64

[Dor Laor]

[-- Attachment #1.1: Type: text/plain, Size: 1950 bytes --]

Gildas wrote:
>
> > 2007/11/8, Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org>:
> > > Lynn Kerby wrote:
> [snip]
> > > I'd assumed (there's *that* word) that core developer types were
> > > using KVM on all kinds of hardware and that basics like this would
> > > have been shaken out.
> >
> > Well, it is a reasonable assumption, but the sad fact is, we do little
> > regression testing on smp here, and Intel doesn't test AMD for some
> > reason.  It used to work but regressed.
> >
> > That's something we need to improve on, and quickly.  I'm counting on
> > Anthony's plan to flood the known universe with kvm-testing nanobots.
>
> Hi,
>
> I've noted that there's less posts from the AMD guys lately (Wei
> Huang, Joerg Roedel, Markus Rechberger), does it means that AMD
> decided to stop supporting KVM?
>
> If not, I think this is time for AMD to step in and use Anthony's
> tools to provide a regression testing platform the same way Intel
> does.
>
> It would be really sad to see AMD CPUS being a second class citizen in 
> KVM...
>
> Just my 2 cents,
> Gildas
>
You're right about AMD status not matched Intel's at the moment.
There was some time it was a bit neglected.
AMD & Qumranet are working actively to fix it.
KVM-51 supports XP & 2003 but has some issues with other OSs.
We will soon solve all of these issues.
There is also work in progress wrt regression testing.
Thanks for the feedback,
Dor.
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> kvm-devel mailing list
> kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
> https://lists.sourceforge.net/lists/listinfo/kvm-devel
>


[-- Attachment #1.2: Type: text/html, Size: 3305 bytes --]

[-- Attachment #2: Type: text/plain, Size: 314 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

[-- Attachment #3: Type: text/plain, Size: 186 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/kvm-devel