KVM, AMD & Interrupts

KVM, AMD & Interrupts

[Matthew Hall]

Hi Avi,

I spoke to you a while ago on #kvm on freenode - I had a problem with 
ide ops on an amd svm host with kvm. The issues regarded the inability 
to access ida devices (hda/hdc) under qemu & kvm on my host - if this 
triggers your memory...

The last point I got to was that it was kvm and amd specific. Id est, 
the same image run under an intel vmx host with the *exact* same image, 
kvm version and command options (svn 'trunk' from the $date).
The issue seemed to be AMD and kvm specific (running with '-no-kvm' made 
everything work ok, but slower, as you'd expect).

I got word of a possible workaround, from one of your colleagues whilst 
you've been away, to use the boot options 'noapic nolapic' on the host 
o/s. Using these options solved my hd* op problems and I can boot ok now.
It seems disabling the local apic was the fundamental part (iirc), as 
without host support the guest would always start the dummy apic; so 
running the guest with any 'special' options (ie. '-no-apic') should not 
be required.

I'm think this problem is specific to my particular cpu/motherboard 
combination aswell.
I also have box with a recent intel cpu (with the vmx extention) which 
doesn't display this behaviour; and from talking with you it's not 
purely amd specific (I believe your tests on amd cpu worked).
Is it possible a slowdown in the interrupt controller between the guest 
and host is a contributing factor?
Iirc, with a 'dummy' apic, I experience a ~5% slowdown on disk and 
network operations, which isn't so bad. I've yet to think about smp usage.

Thought you might appreciate this info should you feel it deserve a wiki 
entry. Point me in the direction of the appropriate page should you feel 
I can put this info in myself.


Best regards,
Matt

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: KVM, AMD & Interrupts

[Anthony Liguori]

Matthew Hall wrote:
> Hi Avi,
>
> I spoke to you a while ago on #kvm on freenode - I had a problem with 
> ide ops on an amd svm host with kvm. The issues regarded the inability 
> to access ida devices (hda/hdc) under qemu & kvm on my host - if this 
> triggers your memory...
>
> The last point I got to was that it was kvm and amd specific. Id est, 
> the same image run under an intel vmx host with the *exact* same image, 
> kvm version and command options (svn 'trunk' from the $date).
> The issue seemed to be AMD and kvm specific (running with '-no-kvm' made 
> everything work ok, but slower, as you'd expect).
>
> I got word of a possible workaround, from one of your colleagues whilst 
> you've been away, to use the boot options 'noapic nolapic' on the host 
> o/s. Using these options solved my hd* op problems and I can boot ok now.
>   

I've seen a very similar problem on an IBM x460.  The main symptom I see 
is lost timer interrupts in the host.

I haven't spent much time looking into this but I suspect it's almost 
certainly related to the fact that KVM is using a different interrupt 
delivery method than Xen.  Xen doesn't seem to have an issue on this 
system.  I suspect that the interrupts being received that's causing a 
VMEXIT is not actually getting propagated to the host (the timer being 
the most frequent interrupt to cause a VMEXIT).  If we switch to not to 
acknowledging the interrupt before the VMEXIT and then regenerating it 
either with a software interrupt or delivering it directly in the host, 
it's possible this problem may go away.

This is all just speculation though.

regards,

Anthony Liguori

> It seems disabling the local apic was the fundamental part (iirc), as 
> without host support the guest would always start the dummy apic; so 
> running the guest with any 'special' options (ie. '-no-apic') should not 
> be required.
>
> I'm think this problem is specific to my particular cpu/motherboard 
> combination aswell.
> I also have box with a recent intel cpu (with the vmx extention) which 
> doesn't display this behaviour; and from talking with you it's not 
> purely amd specific (I believe your tests on amd cpu worked).
> Is it possible a slowdown in the interrupt controller between the guest 
> and host is a contributing factor?
> Iirc, with a 'dummy' apic, I experience a ~5% slowdown on disk and 
> network operations, which isn't so bad. I've yet to think about smp usage.
>
> Thought you might appreciate this info should you feel it deserve a wiki 
> entry. Point me in the direction of the appropriate page should you feel 
> I can put this info in myself.
>
>
> Best regards,
> Matt
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> kvm-devel mailing list
> kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
> https://lists.sourceforge.net/lists/listinfo/kvm-devel
>   


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: KVM, AMD & Interrupts

[Avi Kivity]

Matthew Hall wrote:
> Hi Avi,
>
> I spoke to you a while ago on #kvm on freenode - I had a problem with 
> ide ops on an amd svm host with kvm. The issues regarded the inability 
> to access ida devices (hda/hdc) under qemu & kvm on my host - if this 
> triggers your memory...
>
> The last point I got to was that it was kvm and amd specific. Id est, 
> the same image run under an intel vmx host with the *exact* same 
> image, kvm version and command options (svn 'trunk' from the $date).
> The issue seemed to be AMD and kvm specific (running with '-no-kvm' 
> made everything work ok, but slower, as you'd expect).
>
> I got word of a possible workaround, from one of your colleagues 
> whilst you've been away, to use the boot options 'noapic nolapic' on 
> the host o/s. Using these options solved my hd* op problems and I can 
> boot ok now.
> It seems disabling the local apic was the fundamental part (iirc), as 
> without host support the guest would always start the dummy apic; so 
> running the guest with any 'special' options (ie. '-no-apic') should 
> not be required.
>
> I'm think this problem is specific to my particular cpu/motherboard 
> combination aswell.
> I also have box with a recent intel cpu (with the vmx extention) which 
> doesn't display this behaviour; and from talking with you it's not 
> purely amd specific (I believe your tests on amd cpu worked).
> Is it possible a slowdown in the interrupt controller between the 
> guest and host is a contributing factor?
> Iirc, with a 'dummy' apic, I experience a ~5% slowdown on disk and 
> network operations, which isn't so bad. I've yet to think about smp 
> usage.

I'll have to refresh my memory of the AMD spec (I didn't do the AMD port 
so the details are fuzzy).  Perhaps the AMD folks have some insight.  
Maybe it's just another missing event to intercept.

>
> Thought you might appreciate this info should you feel it deserve a 
> wiki entry. Point me in the direction of the appropriate page should 
> you feel I can put this info in myself.
>
>

The best place for such things is the bug tracker (you can find a link 
on the wiki).


-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: KVM, AMD & Interrupts

[Joerg Roedel]

Hi Matthew,

On Thu, Jan 25, 2007 at 01:03:00AM +0000, Matthew Hall wrote:
> I spoke to you a while ago on #kvm on freenode - I had a problem with 
> ide ops on an amd svm host with kvm. The issues regarded the inability 
> to access ida devices (hda/hdc) under qemu & kvm on my host - if this 
> triggers your memory...

I didn't follow your discussion about the problem in the #kvm channel.
Could you provide more information about the problem itself and about
your test environment? If I am able to reproduce the problem, we may be
able to fix it.

-- 
Joerg Roedel
Operating System Research Center
AMD Saxony LLC & Co. KG



-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: KVM, AMD & Interrupts

[Matthew Hall]

[-- Attachment #1: Type: text/plain, Size: 1623 bytes --]

Joerg Roedel wrote:
> I didn't follow your discussion about the problem in the #kvm channel.
> Could you provide more information about the problem itself and about
> your test environment? If I am able to reproduce the problem, we may be
> able to fix it.

Sure.

My host is a Dell SC1435, with 2 dual core Opteron 2212 HE's @2GHz. 
There's 4Gb memory and a 120Gb Sata disk (sata_svw). The host is running 
   FC6 x86_64 with 'kernel-2.6.19-1.2906.fc7' (2.6.20-rc4 from fc7 
devel) and kvm-12.
(The fc kernel comes with the kvm drivers from an earlier version, i've 
deleted these and have been testing bith the trunk and release kvm code).

The symptoms I saw were that the guest was not able to access any hd* 
devices 'properly'.
The local boot iso images (I tried the fc6 installer and some live-cd's) 
booted the kernel, but when linux attempted to enumerate the hd devices 
(before init) it stalled and eventually printed errors like 'hda: lost 
interrupt' and 'drive ready seek command failed' et al.

I got to the point of turning on the debug messages from qemu/hw/ide.c 
(DEBUG_IDE*).
At the point where the guest stalled (just after printing the CHS of 
hda) the host printed 'ide: CMD=91' (then there was a significant pause 
before anything else was printed). The guest then prints 'hda: lost 
interrupt' and the next pause was just after 'ide: CMD=10' (it then 
carries on like this until I give up hope and ctrl-c qemu). I've 
attached the debug output from that test.


The box is not live atm - so I have carte blanche to run any tests 
should you not be able to reproduce this yourselves.


Thanks,
Matt

[-- Attachment #2: debug.log.bz2 --]
[-- Type: application/x-bzip, Size: 2758 bytes --]

[-- Attachment #3: Type: text/plain, Size: 347 bytes --]

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

[-- Attachment #4: Type: text/plain, Size: 186 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/kvm-devel