* IRQ affinity: Xen view different from pvops (konrad-pcifront) domU
@ 2010-11-10 1:18 Dante Cinco
2010-11-10 2:11 ` Jeremy Fitzhardinge
2010-11-10 8:13 ` Jan Beulich
0 siblings, 2 replies; 5+ messages in thread
From: Dante Cinco @ 2010-11-10 1:18 UTC (permalink / raw)
To: Xen-devel
I have a Fibre Channel HBA that I'm passing through to the pvops domU.
In domU its IRQ 127 and I've affinitized it to VCPU0 (out of 16) by
setting /proc/irq/127/smp_affinity to 0001 (default setting was FFFF).
However, when I checked the interrupt bindings in Xen, it still shows
IRQ 127 as being affinitized to all CPUs (all F's). I checked lspci in
both dom0 and domU and the address portion (00000000fee00000) of the
MSI address/data seem to match Xen's interrupt binding report (I
expected the digits to the right of 'fee' to be non-zero if it's
affinitized to one specific CPU).
I've tried using a recent Ubuntu HVM kernel for domU instead of
Konrad's pcifront pvops kernel and with the HVM kernel, Xen's
interrupt binding matches the IRQ smp_affinity in domU.
Am I not correctly interpreting the apparent affinity discrepancy
between Xen and pvops domU or is there a known limitation or problem
with affinitizing IRQs in Konrad's pcifront pvops domU kernel?
from Konrad's pcifront pvop domU:
lspci -vv -s 00:00.0
00:00.0 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
Interrupt: pin D routed to IRQ 127
Capabilities: [60] Message Signalled Interrupts: Mask- 64bit+
Queue=0/1 Enable+
Address: 00000000fee00000 Data: 4043
Capabilities: [b0] MSI-X: Enable- Mask- TabSize=9
Vector table: BAR=4 offset=00004100
PBA: BAR=4 offset=00004000
cat /proc/irq/127/smp_affinity
0001
uname -a
Linux kaan-40-dpm 2.6.36-rc7-pvops-kpcif-08-2-domu-5.8.dcinco-debug #1
SMP Tue Nov 9 10:36:45 PST 2010 x86_64 GNU/Linux
from 2.6.32.25 pvops dom0:
lspci -vv -s 11:00.3
11:00.3 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
Interrupt: pin D routed to IRQ 4426
Capabilities: [60] Message Signalled Interrupts: Mask- 64bit+
Queue=0/1 Enable+
Address: 00000000fee00000 Data: 4043
Capabilities: [b0] MSI-X: Enable- Mask- TabSize=9
Vector table: BAR=4 offset=00004100
PBA: BAR=4 offset=00004000
uname -a
Linux kaan-40 2.6.32.25-pvops-stable-dom0-5.7.dcinco-debug #1 SMP
PREEMPT Fri Nov 5 16:13:32 PDT 2010 x86_64 GNU/Linux
(XEN) *** Serial input -> Xen (type 'CTRL-a' three times to switch
input to DOM0)
(XEN) Guest interrupt information:
(XEN) IRQ: 67 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:43
type=PCI-MSI status=00000010 in-flight=0
domain-list=2:127(----),
xm list
Name ID Mem VCPUs State Time(s)
Domain-0 0 1024 1 r----- 43.6
domU 2 2048 16 -b---- 63.0
xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
Domain-0 0 0 0 r-- 44.7 0
domU 2 0 0 -b- 17.1 0
domU 2 1 1 -b- 4.9 1
domU 2 2 2 -b- 3.1 2
domU 2 3 3 -b- 4.7 3
domU 2 4 4 -b- 3.0 4
domU 2 5 5 -b- 3.9 5
domU 2 6 6 -b- 4.9 6
domU 2 7 7 -b- 1.9 7
domU 2 8 8 -b- 3.7 8
domU 2 9 9 -b- 1.8 9
domU 2 10 10 -b- 1.8 10
domU 2 11 11 -b- 4.3 11
domU 2 12 12 -b- 2.8 12
domU 2 13 13 -b- 2.7 13
domU 2 14 14 -b- 1.9 14
domU 2 15 15 -b- 3.9 15
xm info
release : 2.6.32.25-pvops-stable-dom0-5.7.dcinco-debug
version : #1 SMP PREEMPT Fri Nov 5 16:13:32 PDT 2010
machine : x86_64
hw_caps :
bfebfbff:2c100800:00000000:00001f40:029ee3ff:00000000:00000001:00000000
max_node_id : 1
xen_major : 4
xen_minor : 0
xen_extra : .2-rc1-pre
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_changeset : Fri Sep 17 17:06:57 2010 +0100 21350:6e0ffcd2d9e0
xen_commandline : dummy=dummy dom0_mem=1024M dom0_max_vcpus=1
dom0_vcpus_pin=true iommu=1,passthrough,no-intremap loglvl=all
loglvl_guest=all loglevl=10 debug apic=on apic_verbosity=verbose
extra_guest_irqs=80 com1=115200,8n1 console=com1 console_to_ring
noirqbalance xen-pciback.permissive acpi=force numa=on
- Dante
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: IRQ affinity: Xen view different from pvops (konrad-pcifront) domU
2010-11-10 1:18 IRQ affinity: Xen view different from pvops (konrad-pcifront) domU Dante Cinco
@ 2010-11-10 2:11 ` Jeremy Fitzhardinge
2010-11-10 8:13 ` Jan Beulich
1 sibling, 0 replies; 5+ messages in thread
From: Jeremy Fitzhardinge @ 2010-11-10 2:11 UTC (permalink / raw)
To: Dante Cinco; +Cc: Xen-devel
On 11/09/2010 05:18 PM, Dante Cinco wrote:
> I have a Fibre Channel HBA that I'm passing through to the pvops domU.
> In domU its IRQ 127 and I've affinitized it to VCPU0 (out of 16) by
> setting /proc/irq/127/smp_affinity to 0001 (default setting was FFFF).
> However, when I checked the interrupt bindings in Xen, it still shows
> IRQ 127 as being affinitized to all CPUs (all F's). I checked lspci in
> both dom0 and domU and the address portion (00000000fee00000) of the
> MSI address/data seem to match Xen's interrupt binding report (I
> expected the digits to the right of 'fee' to be non-zero if it's
> affinitized to one specific CPU).
>
> I've tried using a recent Ubuntu HVM kernel for domU instead of
> Konrad's pcifront pvops kernel and with the HVM kernel, Xen's
> interrupt binding matches the IRQ smp_affinity in domU.
>
> Am I not correctly interpreting the apparent affinity discrepancy
> between Xen and pvops domU or is there a known limitation or problem
> with affinitizing IRQs in Konrad's pcifront pvops domU kernel?
A VCPU is not a real physical CPU, and can run on any physical CPU from
moment to moment. Setting the affinity within the domU will cause the
interrupts to be handled by a specific VCPU, but that has no meaning to
the hardware as it knows nothing about VCPUs. Since the VCPU can run on
any PCPU, it makes sense that the hardware routing is for all PCPUs. In
principle I guess you could pin the VCPU to a particular PCPU and then
route only to that PCPU, but I'm not sure what that would achieve.
J
> from Konrad's pcifront pvop domU:
> lspci -vv -s 00:00.0
> 00:00.0 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
> Interrupt: pin D routed to IRQ 127
> Capabilities: [60] Message Signalled Interrupts: Mask- 64bit+
> Queue=0/1 Enable+
> Address: 00000000fee00000 Data: 4043
> Capabilities: [b0] MSI-X: Enable- Mask- TabSize=9
> Vector table: BAR=4 offset=00004100
> PBA: BAR=4 offset=00004000
> cat /proc/irq/127/smp_affinity
> 0001
> uname -a
> Linux kaan-40-dpm 2.6.36-rc7-pvops-kpcif-08-2-domu-5.8.dcinco-debug #1
> SMP Tue Nov 9 10:36:45 PST 2010 x86_64 GNU/Linux
>
>
> from 2.6.32.25 pvops dom0:
> lspci -vv -s 11:00.3
> 11:00.3 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
> Interrupt: pin D routed to IRQ 4426
> Capabilities: [60] Message Signalled Interrupts: Mask- 64bit+
> Queue=0/1 Enable+
> Address: 00000000fee00000 Data: 4043
> Capabilities: [b0] MSI-X: Enable- Mask- TabSize=9
> Vector table: BAR=4 offset=00004100
> PBA: BAR=4 offset=00004000
> uname -a
> Linux kaan-40 2.6.32.25-pvops-stable-dom0-5.7.dcinco-debug #1 SMP
> PREEMPT Fri Nov 5 16:13:32 PDT 2010 x86_64 GNU/Linux
>
>
> (XEN) *** Serial input -> Xen (type 'CTRL-a' three times to switch
> input to DOM0)
> (XEN) Guest interrupt information:
> (XEN) IRQ: 67 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:43
> type=PCI-MSI status=00000010 in-flight=0
> domain-list=2:127(----),
>
>
> xm list
> Name ID Mem VCPUs State Time(s)
> Domain-0 0 1024 1 r----- 43.6
> domU 2 2048 16 -b---- 63.0
> xm vcpu-list
> Name ID VCPU CPU State Time(s) CPU Affinity
> Domain-0 0 0 0 r-- 44.7 0
> domU 2 0 0 -b- 17.1 0
> domU 2 1 1 -b- 4.9 1
> domU 2 2 2 -b- 3.1 2
> domU 2 3 3 -b- 4.7 3
> domU 2 4 4 -b- 3.0 4
> domU 2 5 5 -b- 3.9 5
> domU 2 6 6 -b- 4.9 6
> domU 2 7 7 -b- 1.9 7
> domU 2 8 8 -b- 3.7 8
> domU 2 9 9 -b- 1.8 9
> domU 2 10 10 -b- 1.8 10
> domU 2 11 11 -b- 4.3 11
> domU 2 12 12 -b- 2.8 12
> domU 2 13 13 -b- 2.7 13
> domU 2 14 14 -b- 1.9 14
> domU 2 15 15 -b- 3.9 15
> xm info
> release : 2.6.32.25-pvops-stable-dom0-5.7.dcinco-debug
> version : #1 SMP PREEMPT Fri Nov 5 16:13:32 PDT 2010
> machine : x86_64
> hw_caps :
> bfebfbff:2c100800:00000000:00001f40:029ee3ff:00000000:00000001:00000000
> max_node_id : 1
> xen_major : 4
> xen_minor : 0
> xen_extra : .2-rc1-pre
> xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
> hvm-3.0-x86_32p hvm-3.0-x86_64
> xen_changeset : Fri Sep 17 17:06:57 2010 +0100 21350:6e0ffcd2d9e0
> xen_commandline : dummy=dummy dom0_mem=1024M dom0_max_vcpus=1
> dom0_vcpus_pin=true iommu=1,passthrough,no-intremap loglvl=all
> loglvl_guest=all loglevl=10 debug apic=on apic_verbosity=verbose
> extra_guest_irqs=80 com1=115200,8n1 console=com1 console_to_ring
> noirqbalance xen-pciback.permissive acpi=force numa=on
>
>
> - Dante
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: IRQ affinity: Xen view different from pvops (konrad-pcifront) domU
2010-11-10 1:18 IRQ affinity: Xen view different from pvops (konrad-pcifront) domU Dante Cinco
2010-11-10 2:11 ` Jeremy Fitzhardinge
@ 2010-11-10 8:13 ` Jan Beulich
2010-11-10 20:18 ` Dante Cinco
1 sibling, 1 reply; 5+ messages in thread
From: Jan Beulich @ 2010-11-10 8:13 UTC (permalink / raw)
To: Dante Cinco; +Cc: Xen-devel
>>> On 10.11.10 at 02:18, Dante Cinco <dantecinco@gmail.com> wrote:
> I have a Fibre Channel HBA that I'm passing through to the pvops domU.
> In domU its IRQ 127 and I've affinitized it to VCPU0 (out of 16) by
> setting /proc/irq/127/smp_affinity to 0001 (default setting was FFFF).
> However, when I checked the interrupt bindings in Xen, it still shows
> IRQ 127 as being affinitized to all CPUs (all F's). I checked lspci in
> both dom0 and domU and the address portion (00000000fee00000) of the
> MSI address/data seem to match Xen's interrupt binding report (I
> expected the digits to the right of 'fee' to be non-zero if it's
> affinitized to one specific CPU).
See -unstable c/s 21625:0695a5cdcb42.
Jan
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: IRQ affinity: Xen view different from pvops (konrad-pcifront) domU
2010-11-10 8:13 ` Jan Beulich
@ 2010-11-10 20:18 ` Dante Cinco
2010-11-11 13:22 ` Jan Beulich
0 siblings, 1 reply; 5+ messages in thread
From: Dante Cinco @ 2010-11-10 20:18 UTC (permalink / raw)
To: Jan Beulich, Jeremy Fitzhardinge; +Cc: Xen-devel
On Wed, Nov 10, 2010 at 12:13 AM, Jan Beulich <JBeulich@novell.com> wrote:
>>>> On 10.11.10 at 02:18, Dante Cinco <dantecinco@gmail.com> wrote:
>> I have a Fibre Channel HBA that I'm passing through to the pvops domU.
>> In domU its IRQ 127 and I've affinitized it to VCPU0 (out of 16) by
>> setting /proc/irq/127/smp_affinity to 0001 (default setting was FFFF).
>> However, when I checked the interrupt bindings in Xen, it still shows
>> IRQ 127 as being affinitized to all CPUs (all F's). I checked lspci in
>> both dom0 and domU and the address portion (00000000fee00000) of the
>> MSI address/data seem to match Xen's interrupt binding report (I
>> expected the digits to the right of 'fee' to be non-zero if it's
>> affinitized to one specific CPU).
>
> See -unstable c/s 21625:0695a5cdcb42.
>
> Jan
>
>
Jan,
After seeing your response (cs 21625, x86: IRQ affinity should track
vCPU affinity), I installed the latest xen-unstable-4.1 (cs 22382) and
the result is the same --- IRQ affinity in Xen is set to all F's after
explicitly setting the IRQ smp_affinity in domU to a specific CPU. I
do have the VCPUs assigned to domU pinned to PCPUs on a one-to-one
basis (VCPU0 -> PCPU0, VCPU1 -> PCPU1, etc.).
BTW, I actually have 16 of these devices PCI-passed through to domU
which has16 VCPUs and I'm affinitizing each device to its own
dedicated CPU. Before explicitly setting the IRQ smp_affinity, all 16
devices are set to 0001. See the explicit IRQ smp_affinity setting
below. If I look in the domU's /proc/interrupts, it shows the
interrupts for a given IRQ going only to the CPU it has been
affinitized to (no interrupts going to the other CPUs) which is
expected.
My system has 24 PCPUs (dual socket X5650 Xeon/6-core Westmere) but
I'm only assigning the first 16 CPUs to domU and the VCPUs are pinned
to their respective PCPU. The part I don't understand is if the IRQ
affinity per Xen (all F's) is correct, how does the interrupt get
handled if it is directed to a PCPU that is not assigned/pinned to a
VCPU in domU?
Jeremy, to address your question, pinning the VCPUs to specific PCPUs
and affinitizing the IRQ to specific VCPUs in domU has worked very
well for us on the HVM kernel. The I/O performance is significantly
better compared with no affinitization. We're trying to transition to
the pvops domU with the expectation that our affinitization strategy
will still be applicable and maintain or even possibly improve I/O
performance.
These are the IRQ smp_affinity settings in pvops domU:
cat /proc/irq/112/smp_affinity
8000
cat /proc/irq/113/smp_affinity
4000
cat /proc/irq/114/smp_affinity
2000
cat /proc/irq/115/smp_affinity
1000
cat /proc/irq/116/smp_affinity
0800
cat /proc/irq/117/smp_affinity
0400
cat /proc/irq/118/smp_affinity
0200
cat /proc/irq/119/smp_affinity
0100
cat /proc/irq/120/smp_affinity
0080
cat /proc/irq/121/smp_affinity
0040
cat /proc/irq/122/smp_affinity
0020
cat /proc/irq/123/smp_affinity
0010
cat /proc/irq/124/smp_affinity
0008
cat /proc/irq/125/smp_affinity
0004
cat /proc/irq/126/smp_affinity
0002
cat /proc/irq/127/smp_affinity
0001
(XEN) *** Serial input -> Xen (type 'CTRL-a' three times to switch
input to DOM0)
(XEN) Guest interrupt information:
(XEN) IRQ: 67 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:7a
type=PCI-MSI status=00000010 in-flight=0
domain-list=1:127(----),
(XEN) IRQ: 68 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:8a
type=PCI-MSI status=00000010 in-flight=0
domain-list=1:126(----),
(XEN) IRQ: 69 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:92
type=PCI-MSI status=00000010 in-flight=0
domain-list=1:125(----),
(XEN) IRQ: 70 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:9a
type=PCI-MSI status=00000010 in-flight=0
domain-list=1:124(----),
(XEN) IRQ: 71 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:a2
type=PCI-MSI status=00000010 in-flight=0
domain-list=1:123(----),
(XEN) IRQ: 72 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:aa
type=PCI-MSI status=00000010 in-flight=0
domain-list=1:122(----),
(XEN) IRQ: 73 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:b2
type=PCI-MSI status=00000010 in-flight=0
domain-list=1:121(----),
(XEN) IRQ: 74 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:ba
type=PCI-MSI status=00000010 in-flight=0
domain-list=1:120(----),
(XEN) IRQ: 75 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:c2
type=PCI-MSI status=00000010 in-flight=0
domain-list=1:119(----),
(XEN) IRQ: 76 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:ca
type=PCI-MSI status=00000010 in-flight=0
domain-list=1:118(----),
(XEN) IRQ: 77 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:d2
type=PCI-MSI status=00000010 in-flight=0
domain-list=1:117(----),
(XEN) IRQ: 78 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:da
type=PCI-MSI status=00000010 in-flight=0
domain-list=1:116(----),
(XEN) IRQ: 79 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:23
type=PCI-MSI status=00000010 in-flight=0
domain-list=1:115(----),
(XEN) IRQ: 80 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:2b
type=PCI-MSI status=00000010 in-flight=0
domain-list=1:114(----),
(XEN) IRQ: 81 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:33
type=PCI-MSI status=00000010 in-flight=0
domain-list=1:113(----),
(XEN) IRQ: 82 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:3b
type=PCI-MSI status=00000010 in-flight=0
domain-list=1:112(----),
xm info
xen_changeset : Tue Nov 09 20:37:46 2010 +0000 22382:a15b0a2dc276
xm list
Name ID Mem VCPUs State Time(s)
Domain-0 0 1024 1 r----- 50.3
domU 1 2048 16 -b---- 220.9
xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
Domain-0 0 0 0 r-- 50.6 0
domU 1 0 0 -b- 32.2 0
domU 1 1 1 -b- 19.8 1
domU 1 2 2 -b- 19.0 2
domU 1 3 3 -b- 13.9 3
domU 1 4 4 -b- 8.4 4
domU 1 5 5 -b- 16.6 5
domU 1 6 6 -b- 26.6 6
domU 1 7 7 -b- 8.0 7
domU 1 8 8 -b- 9.6 8
domU 1 9 9 -b- 16.5 9
domU 1 10 10 -b- 9.0 10
domU 1 11 11 -b- 8.2 11
domU 1 12 12 -b- 12.4 12
domU 1 13 13 -b- 11.6 13
domU 1 14 14 -b- 4.9 14
domU 1 15 15 -b- 4.5 15
- Dante
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: IRQ affinity: Xen view different from pvops (konrad-pcifront) domU
2010-11-10 20:18 ` Dante Cinco
@ 2010-11-11 13:22 ` Jan Beulich
0 siblings, 0 replies; 5+ messages in thread
From: Jan Beulich @ 2010-11-11 13:22 UTC (permalink / raw)
To: Dante Cinco; +Cc: Jeremy Fitzhardinge, Xen-devel
>>> On 10.11.10 at 21:18, Dante Cinco <dantecinco@gmail.com> wrote:
> After seeing your response (cs 21625, x86: IRQ affinity should track
> vCPU affinity), I installed the latest xen-unstable-4.1 (cs 22382) and
> the result is the same --- IRQ affinity in Xen is set to all F's after
> explicitly setting the IRQ smp_affinity in domU to a specific CPU. I
> do have the VCPUs assigned to domU pinned to PCPUs on a one-to-one
> basis (VCPU0 -> PCPU0, VCPU1 -> PCPU1, etc.).
You may need to do some debugging then to find out what's not
working in your case.
> My system has 24 PCPUs (dual socket X5650 Xeon/6-core Westmere) but
> I'm only assigning the first 16 CPUs to domU and the VCPUs are pinned
> to their respective PCPU. The part I don't understand is if the IRQ
> affinity per Xen (all F's) is correct, how does the interrupt get
> handled if it is directed to a PCPU that is not assigned/pinned to a
> VCPU in domU?
Since the guest handles event channel signals only, all it takes
is binding the event channel to the correct vCPU. When the
channel gets signaled, that vCPU will call the handler.
Jan
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2010-11-11 13:22 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-11-10 1:18 IRQ affinity: Xen view different from pvops (konrad-pcifront) domU Dante Cinco
2010-11-10 2:11 ` Jeremy Fitzhardinge
2010-11-10 8:13 ` Jan Beulich
2010-11-10 20:18 ` Dante Cinco
2010-11-11 13:22 ` Jan Beulich
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.