From: Alex Williamson <alex.williamson@redhat.com>
To: Scott <ph3lpz@gmail.com>
Cc: kvm@vger.kernel.org
Subject: Re: preventing arbitrary virtual function assignment
Date: Fri, 23 Mar 2012 15:26:20 -0600 [thread overview]
Message-ID: <1332537980.19665.18.camel@ul30vt> (raw)
In-Reply-To: <8EDC5828-B801-41A7-8D3C-2A5B47A44D4A@gmail.com>
On Fri, 2012-03-23 at 11:59 -0400, Scott wrote:
> I'm really stymied by this issue so thanks in advance for any help!
>
> Problem statement:
> The virtual functions of an SR-IOV NIC are not being assigned assigned as specified in the VM guest's XML definitions.
>
> Data Points:
> Server Information: HP ProLiant DL165 G7
> Ethernet Card Information: Intel 82576
> uname: Linux 3.0.0-16-server x86_64 GNU/Linux
> OS Release: Ubuntu 11.10
> KVM version: 0.14.1+noroms-0ubuntu6.2
>
> Steps to Reproduce:
>
> # ip link show:
> 2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000
> link/ether 00:1b:21:7d:70:16 brd ff:ff:ff:ff:ff:ff
> vf 0 MAC 02:22:25:00:00:00
> vf 1 MAC 02:22:25:00:00:01
> vf 2 MAC 02:22:25:00:00:02
> vf 3 MAC 02:22:25:00:00:03
> vf 4 MAC 02:22:25:00:00:04
> vf 5 MAC 02:22:25:00:00:05
> vf 6 MAC 02:22:25:00:00:06
> 3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN qlen 1000
> link/ether 00:1b:21:7d:70:16 brd ff:ff:ff:ff:ff:ff
> vf 0 MAC 02:22:25:01:00:00
> vf 1 MAC 02:22:25:01:00:01
> vf 2 MAC 02:22:25:01:00:02
> vf 3 MAC 02:22:25:01:00:03
> vf 4 MAC 02:22:25:01:00:04
> vf 5 MAC 02:22:25:01:00:05
> vf 6 MAC 02:22:25:01:00:06
>
> # lspci |grep 05
> 05:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
> 05:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
> 05:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
> 05:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
> 05:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
> 05:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
> 05:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
> 05:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
> 05:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
> 05:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
> 05:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
> 05:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
> 05:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
> 05:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
> 05:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
> 05:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
>
> # virsh nodedev-list |grep 05
> pci_0000_05_00_0
> pci_0000_05_00_1
> pci_0000_05_10_0
> pci_0000_05_10_1
> pci_0000_05_10_2
> pci_0000_05_10_3
> pci_0000_05_10_4
> pci_0000_05_10_5
> pci_0000_05_10_6
> pci_0000_05_10_7
> pci_0000_05_11_0
> pci_0000_05_11_1
> pci_0000_05_11_2
> pci_0000_05_11_3
> pci_0000_05_11_4
> pci_0000_05_11_5
These are mapped as:
pci_0000_05_00_0 -> eth0
pci_0000_05_00_1 -> eth1
pci_0000_05_10_0 -> eth0-vf0
pci_0000_05_10_1 -> eth1-vf0
pci_0000_05_10_2 -> eth0-vf1
pci_0000_05_10_3 -> eth1-vf1
pci_0000_05_10_4 -> eth0-vf2
pci_0000_05_10_5 -> eth1-vf2
pci_0000_05_10_6 -> eth0-vf3
pci_0000_05_10_7 -> eth1-vf3
pci_0000_05_11_0 -> eth0-vf4
pci_0000_05_11_1 -> eth1-vf4
pci_0000_05_11_2 -> eth0-vf5
pci_0000_05_11_3 -> eth1-vf5
pci_0000_05_11_4 -> eth0-vf6
pci_0000_05_11_5 -> eth1-vf6
> # virsh nodedev-dettach pci_0000_05_10_0
> Device pci_0000_05_10_0 dettached
>
> # virsh nodedev-dettach pci_0000_05_10_1
> Device pci_0000_05_10_1 dettached
>
> # virsh nodedev-dettach pci_0000_05_11_0
> Device pci_0000_05_11_0 dettached
>
> # virsh nodedev-dettach pci_0000_05_11_1
> Device pci_0000_05_11_1 dettached
>
> # virsh nodedev-dumpxml pci_0000_05_10_0|grep -e bus -e slot -e fun |head -3
> <bus>5</bus>
> <slot>16</slot>
> <function>0</function>
>
> # virsh nodedev-dumpxml pci_0000_05_10_1|grep -e bus -e slot -e fun |head -3
> <bus>5</bus>
> <slot>16</slot>
> <function>1</function>
>
> # virsh nodedev-dumpxml pci_0000_05_11_0|grep -e bus -e slot -e fun |head -3
> <bus>5</bus>
> <slot>17</slot>
> <function>0</function>
>
> # virsh nodedev-dumpxml pci_0000_05_11_1|grep -e bus -e slot -e fun |head -3
> <bus>5</bus>
> <slot>17</slot>
> <function>1</function>
>
> == HOST-A ==
>
> XML:
> ...
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0x05' slot='0x10' function='0x0'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
> </hostdev>
eth0-vf0 (02:22:25:00:00:00)
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0x05' slot='0x11' function='0x0'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
> </hostdev>
eth0-vf4 (02:22:25:00:00:04)
>
> dmesg:
> [ 2.765683] Intel(R) Virtual Function Network Driver - version 1.0.8-k0
> [ 2.765687] Copyright (c) 2009 - 2010 Intel Corporation.
> [ 2.765809] igbvf 0000:00:04.0: setting latency timer to 64
> [ 2.766050] igbvf 0000:00:04.0: irq 42 for MSI/MSI-X
> [ 2.766073] igbvf 0000:00:04.0: irq 43 for MSI/MSI-X
> [ 2.766094] igbvf 0000:00:04.0: irq 44 for MSI/MSI-X
> [ 3.072509] igbvf 0000:00:04.0: Intel(R) 82576 Virtual Function
> [ 3.072514] igbvf 0000:00:04.0: Address: 02:22:25:00:00:00
> [ 3.072516] igbvf 0000:00:04.0: MAC: 1
> [ 3.072658] igbvf 0000:00:05.0: setting latency timer to 64
> [ 3.072902] igbvf 0000:00:05.0: irq 45 for MSI/MSI-X
> [ 3.072924] igbvf 0000:00:05.0: irq 46 for MSI/MSI-X
> [ 3.072946] igbvf 0000:00:05.0: irq 47 for MSI/MSI-X
> [ 3.241325] igbvf 0000:00:05.0: Intel(R) 82576 Virtual Function
> [ 3.241329] igbvf 0000:00:05.0: Address: 02:22:25:00:00:04
> [ 3.241332] igbvf 0000:00:05.0: MAC: 1
> [ 4.070069] igbvf 0000:00:04.0: Link is Up 1000 Mbps Full Duplex
> [ 4.110094] igbvf 0000:00:05.0: Link is Up 1000 Mbps Full Duplex
> [ 4.110127] igbvf 0000:00:04.0: Link is Up 1000 Mbps Full Duplex
>
> ip link
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
> link/ether 02:22:25:00:00:00 brd ff:ff:ff:ff:ff:ff
> 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
> link/ether 02:22:25:00:00:04 brd ff:ff:ff:ff:ff:ff
Looks right
> == HOST-B ==
>
> XML:
> ...
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0x05' slot='0x10' function='0x2'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
> </hostdev>
eth0-vf1 (02:22:25:00:00:01)
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0x05' slot='0x11' function='0x2'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
> </hostdev>
eth0-vf5 (02:22:25:00:00:05)
> dmesg:
> [ 2.235455] igbvf 0000:00:04.0: setting latency timer to 64
> [ 2.235682] igbvf 0000:00:04.0: irq 42 for MSI/MSI-X
> [ 2.235704] igbvf 0000:00:04.0: irq 43 for MSI/MSI-X
> [ 2.235724] igbvf 0000:00:04.0: irq 44 for MSI/MSI-X
> [ 2.540856] igbvf 0000:00:04.0: Intel(R) 82576 Virtual Function
> [ 2.540861] igbvf 0000:00:04.0: Address: 02:22:25:01:00:00
> [ 2.540864] igbvf 0000:00:04.0: MAC: 1
> [ 2.541017] igbvf 0000:00:05.0: setting latency timer to 64
> [ 2.541272] igbvf 0000:00:05.0: irq 45 for MSI/MSI-X
> [ 2.541294] igbvf 0000:00:05.0: irq 46 for MSI/MSI-X
> [ 2.541314] igbvf 0000:00:05.0: irq 47 for MSI/MSI-X
> [ 2.800795] igbvf 0000:00:05.0: Intel(R) 82576 Virtual Function
> [ 2.800799] igbvf 0000:00:05.0: Address: 02:22:25:01:00:04
> [ 2.800802] igbvf 0000:00:05.0: MAC: 1
>
> ip link:
> 2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
> link/ether 02:22:25:01:00:00 brd ff:ff:ff:ff:ff:ff
> 3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
> link/ether 02:22:25:01:00:04 brd ff:ff:ff:ff:ff:ff
Um... not expected
> Expected Results:
> HOST-A should have the following assignment:
> * eth0 should = eth0-vf0 (bus:5, slot:10, fun:0), mac: 02:22:25:00:00:00
> * eth1 should = eth1-vf0 (bus:5, slot:11, fun:0), mac: 02:22:25:01:00:00
Nope, eth1-vf0 would be 5:10.1. The even VF come from PF func 0, odd
from odd. You can unbind 5:00.1 from igb to see this, all the odd VFs
will go away. You can also find the VFs linked under the PF
in /sys/bus/pci/devices/0000:05:00.[01].
> HOST-b should have the following assignment:
> * eth0 should = eth0-vf1 (bus:5, slot:10, fun:1), mac: 02:22:25:00:00:01
> * eth1 should = eth1-vf1 (bus:5, slot:11, fun:1), mac: 02:22:25:01:00:01--
Wait a sec, you originally did a nodedev-dumpxml for 5:10.[01] and
5:11.[01]. 5:10.0 & 5:11.0 are used for HOST-A. The xmldump snippet
for HOST-B is using 5:10.2 & 5:11.2. Since you reference .1 again here,
I can only assume that means HOST-B is actually using 5:10.1 & 5:11.1
(aka eth1-vf0 & eth1-vf4), which means the guest output is correct and
expected. Thanks,
Alex
prev parent reply other threads:[~2012-03-23 21:26 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-03-23 15:59 preventing arbitrary virtual function assignment Scott
2012-03-23 21:26 ` Alex Williamson [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1332537980.19665.18.camel@ul30vt \
--to=alex.williamson@redhat.com \
--cc=kvm@vger.kernel.org \
--cc=ph3lpz@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox