* preventing arbitrary virtual function assignment
@ 2012-03-23 15:59 Scott
2012-03-23 21:26 ` Alex Williamson
0 siblings, 1 reply; 2+ messages in thread
From: Scott @ 2012-03-23 15:59 UTC (permalink / raw)
To: kvm
I'm really stymied by this issue so thanks in advance for any help!
Problem statement:
The virtual functions of an SR-IOV NIC are not being assigned assigned as specified in the VM guest's XML definitions.
Data Points:
Server Information: HP ProLiant DL165 G7
Ethernet Card Information: Intel 82576
uname: Linux 3.0.0-16-server x86_64 GNU/Linux
OS Release: Ubuntu 11.10
KVM version: 0.14.1+noroms-0ubuntu6.2
Steps to Reproduce:
# ip link show:
2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000
link/ether 00:1b:21:7d:70:16 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 02:22:25:00:00:00
vf 1 MAC 02:22:25:00:00:01
vf 2 MAC 02:22:25:00:00:02
vf 3 MAC 02:22:25:00:00:03
vf 4 MAC 02:22:25:00:00:04
vf 5 MAC 02:22:25:00:00:05
vf 6 MAC 02:22:25:00:00:06
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN qlen 1000
link/ether 00:1b:21:7d:70:16 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 02:22:25:01:00:00
vf 1 MAC 02:22:25:01:00:01
vf 2 MAC 02:22:25:01:00:02
vf 3 MAC 02:22:25:01:00:03
vf 4 MAC 02:22:25:01:00:04
vf 5 MAC 02:22:25:01:00:05
vf 6 MAC 02:22:25:01:00:06
# lspci |grep 05
05:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
05:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
# virsh nodedev-list |grep 05
pci_0000_05_00_0
pci_0000_05_00_1
pci_0000_05_10_0
pci_0000_05_10_1
pci_0000_05_10_2
pci_0000_05_10_3
pci_0000_05_10_4
pci_0000_05_10_5
pci_0000_05_10_6
pci_0000_05_10_7
pci_0000_05_11_0
pci_0000_05_11_1
pci_0000_05_11_2
pci_0000_05_11_3
pci_0000_05_11_4
pci_0000_05_11_5
# virsh nodedev-dettach pci_0000_05_10_0
Device pci_0000_05_10_0 dettached
# virsh nodedev-dettach pci_0000_05_10_1
Device pci_0000_05_10_1 dettached
# virsh nodedev-dettach pci_0000_05_11_0
Device pci_0000_05_11_0 dettached
# virsh nodedev-dettach pci_0000_05_11_1
Device pci_0000_05_11_1 dettached
# virsh nodedev-dumpxml pci_0000_05_10_0|grep -e bus -e slot -e fun |head -3
<bus>5</bus>
<slot>16</slot>
<function>0</function>
# virsh nodedev-dumpxml pci_0000_05_10_1|grep -e bus -e slot -e fun |head -3
<bus>5</bus>
<slot>16</slot>
<function>1</function>
# virsh nodedev-dumpxml pci_0000_05_11_0|grep -e bus -e slot -e fun |head -3
<bus>5</bus>
<slot>17</slot>
<function>0</function>
# virsh nodedev-dumpxml pci_0000_05_11_1|grep -e bus -e slot -e fun |head -3
<bus>5</bus>
<slot>17</slot>
<function>1</function>
== HOST-A ==
XML:
...
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x05' slot='0x10' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x05' slot='0x11' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</hostdev>
...
dmesg:
[ 2.765683] Intel(R) Virtual Function Network Driver - version 1.0.8-k0
[ 2.765687] Copyright (c) 2009 - 2010 Intel Corporation.
[ 2.765809] igbvf 0000:00:04.0: setting latency timer to 64
[ 2.766050] igbvf 0000:00:04.0: irq 42 for MSI/MSI-X
[ 2.766073] igbvf 0000:00:04.0: irq 43 for MSI/MSI-X
[ 2.766094] igbvf 0000:00:04.0: irq 44 for MSI/MSI-X
[ 3.072509] igbvf 0000:00:04.0: Intel(R) 82576 Virtual Function
[ 3.072514] igbvf 0000:00:04.0: Address: 02:22:25:00:00:00
[ 3.072516] igbvf 0000:00:04.0: MAC: 1
[ 3.072658] igbvf 0000:00:05.0: setting latency timer to 64
[ 3.072902] igbvf 0000:00:05.0: irq 45 for MSI/MSI-X
[ 3.072924] igbvf 0000:00:05.0: irq 46 for MSI/MSI-X
[ 3.072946] igbvf 0000:00:05.0: irq 47 for MSI/MSI-X
[ 3.241325] igbvf 0000:00:05.0: Intel(R) 82576 Virtual Function
[ 3.241329] igbvf 0000:00:05.0: Address: 02:22:25:00:00:04
[ 3.241332] igbvf 0000:00:05.0: MAC: 1
[ 4.070069] igbvf 0000:00:04.0: Link is Up 1000 Mbps Full Duplex
[ 4.110094] igbvf 0000:00:05.0: Link is Up 1000 Mbps Full Duplex
[ 4.110127] igbvf 0000:00:04.0: Link is Up 1000 Mbps Full Duplex
ip link
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:22:25:00:00:00 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:22:25:00:00:04 brd ff:ff:ff:ff:ff:ff
== HOST-B ==
XML:
...
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x05' slot='0x10' function='0x2'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x05' slot='0x11' function='0x2'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</hostdev>
...
dmesg:
[ 2.235455] igbvf 0000:00:04.0: setting latency timer to 64
[ 2.235682] igbvf 0000:00:04.0: irq 42 for MSI/MSI-X
[ 2.235704] igbvf 0000:00:04.0: irq 43 for MSI/MSI-X
[ 2.235724] igbvf 0000:00:04.0: irq 44 for MSI/MSI-X
[ 2.540856] igbvf 0000:00:04.0: Intel(R) 82576 Virtual Function
[ 2.540861] igbvf 0000:00:04.0: Address: 02:22:25:01:00:00
[ 2.540864] igbvf 0000:00:04.0: MAC: 1
[ 2.541017] igbvf 0000:00:05.0: setting latency timer to 64
[ 2.541272] igbvf 0000:00:05.0: irq 45 for MSI/MSI-X
[ 2.541294] igbvf 0000:00:05.0: irq 46 for MSI/MSI-X
[ 2.541314] igbvf 0000:00:05.0: irq 47 for MSI/MSI-X
[ 2.800795] igbvf 0000:00:05.0: Intel(R) 82576 Virtual Function
[ 2.800799] igbvf 0000:00:05.0: Address: 02:22:25:01:00:04
[ 2.800802] igbvf 0000:00:05.0: MAC: 1
ip link:
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
link/ether 02:22:25:01:00:00 brd ff:ff:ff:ff:ff:ff
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
link/ether 02:22:25:01:00:04 brd ff:ff:ff:ff:ff:ff
Expected Results:
HOST-A should have the following assignment:
* eth0 should = eth0-vf0 (bus:5, slot:10, fun:0), mac: 02:22:25:00:00:00
* eth1 should = eth1-vf0 (bus:5, slot:11, fun:0), mac: 02:22:25:01:00:00
HOST-b should have the following assignment:
* eth0 should = eth0-vf1 (bus:5, slot:10, fun:1), mac: 02:22:25:00:00:01
* eth1 should = eth1-vf1 (bus:5, slot:11, fun:1), mac: 02:22:25:01:00:01
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: preventing arbitrary virtual function assignment
2012-03-23 15:59 preventing arbitrary virtual function assignment Scott
@ 2012-03-23 21:26 ` Alex Williamson
0 siblings, 0 replies; 2+ messages in thread
From: Alex Williamson @ 2012-03-23 21:26 UTC (permalink / raw)
To: Scott; +Cc: kvm
On Fri, 2012-03-23 at 11:59 -0400, Scott wrote:
> I'm really stymied by this issue so thanks in advance for any help!
>
> Problem statement:
> The virtual functions of an SR-IOV NIC are not being assigned assigned as specified in the VM guest's XML definitions.
>
> Data Points:
> Server Information: HP ProLiant DL165 G7
> Ethernet Card Information: Intel 82576
> uname: Linux 3.0.0-16-server x86_64 GNU/Linux
> OS Release: Ubuntu 11.10
> KVM version: 0.14.1+noroms-0ubuntu6.2
>
> Steps to Reproduce:
>
> # ip link show:
> 2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000
> link/ether 00:1b:21:7d:70:16 brd ff:ff:ff:ff:ff:ff
> vf 0 MAC 02:22:25:00:00:00
> vf 1 MAC 02:22:25:00:00:01
> vf 2 MAC 02:22:25:00:00:02
> vf 3 MAC 02:22:25:00:00:03
> vf 4 MAC 02:22:25:00:00:04
> vf 5 MAC 02:22:25:00:00:05
> vf 6 MAC 02:22:25:00:00:06
> 3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN qlen 1000
> link/ether 00:1b:21:7d:70:16 brd ff:ff:ff:ff:ff:ff
> vf 0 MAC 02:22:25:01:00:00
> vf 1 MAC 02:22:25:01:00:01
> vf 2 MAC 02:22:25:01:00:02
> vf 3 MAC 02:22:25:01:00:03
> vf 4 MAC 02:22:25:01:00:04
> vf 5 MAC 02:22:25:01:00:05
> vf 6 MAC 02:22:25:01:00:06
>
> # lspci |grep 05
> 05:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
> 05:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
> 05:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
> 05:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
> 05:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
> 05:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
> 05:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
> 05:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
> 05:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
> 05:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
> 05:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
> 05:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
> 05:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
> 05:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
> 05:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
> 05:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
>
> # virsh nodedev-list |grep 05
> pci_0000_05_00_0
> pci_0000_05_00_1
> pci_0000_05_10_0
> pci_0000_05_10_1
> pci_0000_05_10_2
> pci_0000_05_10_3
> pci_0000_05_10_4
> pci_0000_05_10_5
> pci_0000_05_10_6
> pci_0000_05_10_7
> pci_0000_05_11_0
> pci_0000_05_11_1
> pci_0000_05_11_2
> pci_0000_05_11_3
> pci_0000_05_11_4
> pci_0000_05_11_5
These are mapped as:
pci_0000_05_00_0 -> eth0
pci_0000_05_00_1 -> eth1
pci_0000_05_10_0 -> eth0-vf0
pci_0000_05_10_1 -> eth1-vf0
pci_0000_05_10_2 -> eth0-vf1
pci_0000_05_10_3 -> eth1-vf1
pci_0000_05_10_4 -> eth0-vf2
pci_0000_05_10_5 -> eth1-vf2
pci_0000_05_10_6 -> eth0-vf3
pci_0000_05_10_7 -> eth1-vf3
pci_0000_05_11_0 -> eth0-vf4
pci_0000_05_11_1 -> eth1-vf4
pci_0000_05_11_2 -> eth0-vf5
pci_0000_05_11_3 -> eth1-vf5
pci_0000_05_11_4 -> eth0-vf6
pci_0000_05_11_5 -> eth1-vf6
> # virsh nodedev-dettach pci_0000_05_10_0
> Device pci_0000_05_10_0 dettached
>
> # virsh nodedev-dettach pci_0000_05_10_1
> Device pci_0000_05_10_1 dettached
>
> # virsh nodedev-dettach pci_0000_05_11_0
> Device pci_0000_05_11_0 dettached
>
> # virsh nodedev-dettach pci_0000_05_11_1
> Device pci_0000_05_11_1 dettached
>
> # virsh nodedev-dumpxml pci_0000_05_10_0|grep -e bus -e slot -e fun |head -3
> <bus>5</bus>
> <slot>16</slot>
> <function>0</function>
>
> # virsh nodedev-dumpxml pci_0000_05_10_1|grep -e bus -e slot -e fun |head -3
> <bus>5</bus>
> <slot>16</slot>
> <function>1</function>
>
> # virsh nodedev-dumpxml pci_0000_05_11_0|grep -e bus -e slot -e fun |head -3
> <bus>5</bus>
> <slot>17</slot>
> <function>0</function>
>
> # virsh nodedev-dumpxml pci_0000_05_11_1|grep -e bus -e slot -e fun |head -3
> <bus>5</bus>
> <slot>17</slot>
> <function>1</function>
>
> == HOST-A ==
>
> XML:
> ...
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0x05' slot='0x10' function='0x0'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
> </hostdev>
eth0-vf0 (02:22:25:00:00:00)
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0x05' slot='0x11' function='0x0'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
> </hostdev>
eth0-vf4 (02:22:25:00:00:04)
>
> dmesg:
> [ 2.765683] Intel(R) Virtual Function Network Driver - version 1.0.8-k0
> [ 2.765687] Copyright (c) 2009 - 2010 Intel Corporation.
> [ 2.765809] igbvf 0000:00:04.0: setting latency timer to 64
> [ 2.766050] igbvf 0000:00:04.0: irq 42 for MSI/MSI-X
> [ 2.766073] igbvf 0000:00:04.0: irq 43 for MSI/MSI-X
> [ 2.766094] igbvf 0000:00:04.0: irq 44 for MSI/MSI-X
> [ 3.072509] igbvf 0000:00:04.0: Intel(R) 82576 Virtual Function
> [ 3.072514] igbvf 0000:00:04.0: Address: 02:22:25:00:00:00
> [ 3.072516] igbvf 0000:00:04.0: MAC: 1
> [ 3.072658] igbvf 0000:00:05.0: setting latency timer to 64
> [ 3.072902] igbvf 0000:00:05.0: irq 45 for MSI/MSI-X
> [ 3.072924] igbvf 0000:00:05.0: irq 46 for MSI/MSI-X
> [ 3.072946] igbvf 0000:00:05.0: irq 47 for MSI/MSI-X
> [ 3.241325] igbvf 0000:00:05.0: Intel(R) 82576 Virtual Function
> [ 3.241329] igbvf 0000:00:05.0: Address: 02:22:25:00:00:04
> [ 3.241332] igbvf 0000:00:05.0: MAC: 1
> [ 4.070069] igbvf 0000:00:04.0: Link is Up 1000 Mbps Full Duplex
> [ 4.110094] igbvf 0000:00:05.0: Link is Up 1000 Mbps Full Duplex
> [ 4.110127] igbvf 0000:00:04.0: Link is Up 1000 Mbps Full Duplex
>
> ip link
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
> link/ether 02:22:25:00:00:00 brd ff:ff:ff:ff:ff:ff
> 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
> link/ether 02:22:25:00:00:04 brd ff:ff:ff:ff:ff:ff
Looks right
> == HOST-B ==
>
> XML:
> ...
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0x05' slot='0x10' function='0x2'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
> </hostdev>
eth0-vf1 (02:22:25:00:00:01)
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0x05' slot='0x11' function='0x2'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
> </hostdev>
eth0-vf5 (02:22:25:00:00:05)
> dmesg:
> [ 2.235455] igbvf 0000:00:04.0: setting latency timer to 64
> [ 2.235682] igbvf 0000:00:04.0: irq 42 for MSI/MSI-X
> [ 2.235704] igbvf 0000:00:04.0: irq 43 for MSI/MSI-X
> [ 2.235724] igbvf 0000:00:04.0: irq 44 for MSI/MSI-X
> [ 2.540856] igbvf 0000:00:04.0: Intel(R) 82576 Virtual Function
> [ 2.540861] igbvf 0000:00:04.0: Address: 02:22:25:01:00:00
> [ 2.540864] igbvf 0000:00:04.0: MAC: 1
> [ 2.541017] igbvf 0000:00:05.0: setting latency timer to 64
> [ 2.541272] igbvf 0000:00:05.0: irq 45 for MSI/MSI-X
> [ 2.541294] igbvf 0000:00:05.0: irq 46 for MSI/MSI-X
> [ 2.541314] igbvf 0000:00:05.0: irq 47 for MSI/MSI-X
> [ 2.800795] igbvf 0000:00:05.0: Intel(R) 82576 Virtual Function
> [ 2.800799] igbvf 0000:00:05.0: Address: 02:22:25:01:00:04
> [ 2.800802] igbvf 0000:00:05.0: MAC: 1
>
> ip link:
> 2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
> link/ether 02:22:25:01:00:00 brd ff:ff:ff:ff:ff:ff
> 3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
> link/ether 02:22:25:01:00:04 brd ff:ff:ff:ff:ff:ff
Um... not expected
> Expected Results:
> HOST-A should have the following assignment:
> * eth0 should = eth0-vf0 (bus:5, slot:10, fun:0), mac: 02:22:25:00:00:00
> * eth1 should = eth1-vf0 (bus:5, slot:11, fun:0), mac: 02:22:25:01:00:00
Nope, eth1-vf0 would be 5:10.1. The even VF come from PF func 0, odd
from odd. You can unbind 5:00.1 from igb to see this, all the odd VFs
will go away. You can also find the VFs linked under the PF
in /sys/bus/pci/devices/0000:05:00.[01].
> HOST-b should have the following assignment:
> * eth0 should = eth0-vf1 (bus:5, slot:10, fun:1), mac: 02:22:25:00:00:01
> * eth1 should = eth1-vf1 (bus:5, slot:11, fun:1), mac: 02:22:25:01:00:01--
Wait a sec, you originally did a nodedev-dumpxml for 5:10.[01] and
5:11.[01]. 5:10.0 & 5:11.0 are used for HOST-A. The xmldump snippet
for HOST-B is using 5:10.2 & 5:11.2. Since you reference .1 again here,
I can only assume that means HOST-B is actually using 5:10.1 & 5:11.1
(aka eth1-vf0 & eth1-vf4), which means the guest output is correct and
expected. Thanks,
Alex
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2012-03-23 21:26 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-03-23 15:59 preventing arbitrary virtual function assignment Scott
2012-03-23 21:26 ` Alex Williamson
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox