bnx2x + SR-IOV, no internal L2 switching

Netdev List
 help / color / mirror / Atom feed

* bnx2x + SR-IOV, no internal L2 switching
@ 2014-02-12 13:33 Yoann Juet
  2014-02-12 14:38 ` Ariel Elior
  2014-02-12 23:10 ` Ben Hutchings
  0 siblings, 2 replies; 5+ messages in thread
From: Yoann Juet @ 2014-02-12 13:33 UTC (permalink / raw)
  To: netdev

[-- Attachment #1.1: Type: text/plain, Size: 1363 bytes --]

Hi all,

I'm conducting experiments on SR-IOV with Broadcom and Intel cards on 
debian/unstable with KVM hypervisor. On Broadcom cards (bnx2x module, 
BCM57810 devices), Virtual Functions (VFs) get running, Virtual Machines 
attached to such VFs inherit network connectivity with excellent 
performance.

However, VMs attached to VFs on the Broadcom Physical Functions (PFs) 
behave like they were connected to an ancient hub, not a L2 switch. It 
is as if there was no internal L2 switching on the Broadcom card to 
process VF <-> VF or VF <-> PF communications. As a result, a VM sees 
all inbound/outbound traffic from/to others VMs as well as traffic 
destined to the PF (for instance, the physical ethX has an IP address).

On the other hand, everything works like a charm with Intel cards (ixgbe 
module, 82599EB devices). Traffic between VFs or VF/PF is switched 
internally by the card.

I found very little literature about SR-IOV on Broadcom devices. I 
wonder if it's a normal behaviour, a misconfiguration on my side or 
perhaps a firmware/driver bug.

Have you seen this issue before ?

---
Kernel 3.12.9 (same behaviour with kernels 3.10.x)
	driver: bnx2x
	firmware-version: 7.8.17
Debian/unstable
	libvirt 1.2.1
	QEMU 1.7.0
---

Best regards,
-- 
Université de Nantes - Direction des Systèmes d'Information

[-- Attachment #1.2: yoann_juet.vcf --]
[-- Type: text/x-vcard, Size: 377 bytes --]

begin:vcard
fn:Yoann Juet
n:Juet;Yoann
org;quoted-printable;quoted-printable:Direction des Syst=C3=A8mes d'Information;P=C3=B4le R=C3=A9seau
adr;quoted-printable:BP 92208;;2 rue de la Houssini=C3=A8re;Nantes Cedex 3;;44322;France
email;internet:yoann.juet@univ-nantes.fr
tel;work:02.53.48.49.26
tel;fax:02.53.48.49.09
tel;cell:06.73.15.42.19
version:2.1
end:vcard

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3256 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread

* RE: bnx2x + SR-IOV, no internal L2 switching
  2014-02-12 13:33 bnx2x + SR-IOV, no internal L2 switching Yoann Juet
@ 2014-02-12 14:38 ` Ariel Elior
  2014-02-12 15:54   ` Yoann Juet
  2014-02-12 23:10 ` Ben Hutchings
  1 sibling, 1 reply; 5+ messages in thread
From: Ariel Elior @ 2014-02-12 14:38 UTC (permalink / raw)
  To: yoann.juet@univ-nantes.fr, netdev@vger.kernel.org

> -----Original Message-----
> From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org]
> On Behalf Of Yoann Juet
> Sent: Wednesday, February 12, 2014 3:34 PM
> To: netdev@vger.kernel.org
> Subject: bnx2x + SR-IOV, no internal L2 switching

Hi Yoan,
57810 device does support l2 switching.
I find your terminology a little bit confusing and I have some further questions besides:

[Q1]: Are you attaching physical functions to VMs? I.e. are you passing through PFs to the VMs, or only virtual functions?

[Q2]: Are you using VFs only on VMs, or directly from the Hypervisor too?

[Q3]: You wrote "for instance the physical ethX has an IP address". That in itself is no problem and no surprise. You gave this as an example to traffic arriving where is shouldn't. Please elaborate.

[Q4]: Do you have vlan filters configured anywhere in your topology? Are they configured from Host or from Guest?
(in this respect host tagging would be 'ip link set eth0 vf 5 vlan 100' on the hypervisor while guest tagging would be 'ip link add link eth4 type vlan id 100' or 'vconfig add eth4 100' where eth0 is a PF and eth4 is a VF).

[Q5]: In the case you mentioned where you saw in a VM traffic which was destined to another VM: Did both VMs contain VFs? Were the VFs created from the same Physical Function? If not, what were the BDFs pf the respective PFs? Which mac addresses did you give the VFs?

[Q6]: Please isolate a specific case where switching is not behaving as expected and describe it in more detail:
Please describe the topology (which PFs are involved and which VFs. Who is assigned to which VMs)
Where are you sending data from (a VF, a PF, some peer on the network) and where is it arriving (perhaps arriving in multiple places)
Please detail the behavior you expect and the behavior you observe.
Please supply:
Mac addresses, ip addresses and masks, configured vlans (if any), promiscuous setting, etc.
ip -d -d link show on hypervisor and each of the VMs involved
ethtool -d from Hypervisor PFs 
dmesg from hypervisor

Please note we recently submitted some fixes to our tx-switching behavior:
In e8379c79 "bnx2x: fix VLAN configuration for VFs" we fixed an issue where traffic with the wrong vlan could still end up in a VM configured to a different vlan (hence my questions on vlans).
In c14db202 "bnx2x: Correct default Tx switching behavior" we fixed a connectivity issue with pf to vf connectivity.
Depending on your answers to the above, perhaps these might be relevant to your case.

Thanks,
Ariel

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: bnx2x + SR-IOV, no internal L2 switching
  2014-02-12 14:38 ` Ariel Elior
@ 2014-02-12 15:54   ` Yoann Juet
  0 siblings, 0 replies; 5+ messages in thread
From: Yoann Juet @ 2014-02-12 15:54 UTC (permalink / raw)
  To: Ariel Elior, netdev@vger.kernel.org


[-- Attachment #1.1: Type: text/plain, Size: 12968 bytes --]

Hi Ariel,

> [Q1]: Are you attaching physical functions to VMs? I.e. are you passing through PFs to the VMs, or only virtual functions?

I'm only attaching VF to the VMs.

> [Q2]: Are you using VFs only on VMs, or directly from the Hypervisor too?

VF are only used on VMs.

> [Q3]: You wrote "for instance the physical ethX has an IP address". That in itself is no problem and no surprise. You gave this as an example to traffic arriving where is shouldn't. Please elaborate.

See below.

> [Q4]: Do you have vlan filters configured anywhere in your topology? Are they configured from Host or from Guest?

No, currently all my tests are conducted without vlan isolation.

> [Q5]: In the case you mentioned where you saw in a VM traffic which was destined to another VM: Did both VMs contain VFs? Were the VFs created from the same Physical Function? If not, what were the BDFs pf the respective PFs? Which mac addresses did you give the VFs?

Each VMs has it's own VF from the same PF. Before starting VMs, I set 
the MAC address of each VF with a command similar to:

"ip link set eth0 vf <vf_num> mac de:ad:be:ff:ff:<num>"

>
> [Q6]: Please isolate a specific case where switching is not behaving as expected and describe it in more detail:
> Please describe the topology (which PFs are involved and which VFs. Who is assigned to which VMs)

|PF| ----|VF0 de:ad:be:ff:ff:11| ---- |VM1|
       |--|VF1 de:ad:be:ff:ff:12| ---- |VM2|


----
HOST
----

the PF (eth0) and VFs attached to it (eth2 -> eth9)

# ip add
....
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP 
group default qlen 1000
     link/ether a4:1f:72:d0:b3:9f brd ff:ff:ff:ff:ff:ff
     inet 172.20.6.229/24 brd 172.20.6.255 scope global eth0
        valid_lft forever preferred_lft forever
     inet6 fe80::a61f:72ff:fed0:b39f/64 scope link
        valid_lft forever preferred_lft forever
...
30: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group 
default qlen 1000
     link/ether de:ad:be:ff:ff:11 brd ff:ff:ff:ff:ff:ff
31: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group 
default qlen 1000
     link/ether de:ad:be:ff:ff:12 brd ff:ff:ff:ff:ff:ff
32: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group 
default qlen 1000
     link/ether de:ad:be:ff:ff:13 brd ff:ff:ff:ff:ff:ff
33: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group 
default qlen 1000
     link/ether de:ad:be:ff:ff:14 brd ff:ff:ff:ff:ff:ff
34: eth6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group 
default qlen 1000
     link/ether de:ad:be:ff:ff:15 brd ff:ff:ff:ff:ff:ff
35: eth7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group 
default qlen 1000
     link/ether de:ad:be:ff:ff:16 brd ff:ff:ff:ff:ff:ff
36: eth8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group 
default qlen 1000
     link/ether de:ad:be:ff:ff:17 brd ff:ff:ff:ff:ff:ff
37: eth9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group 
default qlen 1000
     link/ether de:ad:be:ff:ff:18 brd ff:ff:ff:ff:ff:ff

# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode 
DEFAULT group default
     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0
2: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode 
DEFAULT group default qlen 1000
     link/ether a4:1f:72:d0:b3:9c brd ff:ff:ff:ff:ff:ff promiscuity 0
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP 
mode DEFAULT group default qlen 1000
     link/ether a4:1f:72:d0:b3:9f brd ff:ff:ff:ff:ff:ff promiscuity 0
     vf 0 MAC de:ad:be:ff:ff:11, tx rate 10000 (Mbps), spoof checking 
on, link-state auto
     vf 1 MAC de:ad:be:ff:ff:12, tx rate 10000 (Mbps), spoof checking 
on, link-state auto
     vf 2 MAC de:ad:be:ff:ff:13, tx rate 10000 (Mbps), spoof checking 
on, link-state auto
     vf 3 MAC de:ad:be:ff:ff:14, tx rate 10000 (Mbps), spoof checking 
on, link-state auto
     vf 4 MAC de:ad:be:ff:ff:15, tx rate 10000 (Mbps), spoof checking 
on, link-state auto
     vf 5 MAC de:ad:be:ff:ff:16, tx rate 10000 (Mbps), spoof checking 
on, link-state auto
     vf 6 MAC de:ad:be:ff:ff:17, tx rate 10000 (Mbps), spoof checking 
on, link-state auto
     vf 7 MAC de:ad:be:ff:ff:18, tx rate 10000 (Mbps), spoof checking 
on, link-state auto
4: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN 
mode DEFAULT group default
     link/ether 8e:31:e9:eb:89:b9 brd ff:ff:ff:ff:ff:ff promiscuity 0
     bond
32: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode 
DEFAULT group default qlen 1000
     link/ether de:ad:be:ff:ff:13 brd ff:ff:ff:ff:ff:ff promiscuity 0
33: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode 
DEFAULT group default qlen 1000
     link/ether de:ad:be:ff:ff:14 brd ff:ff:ff:ff:ff:ff promiscuity 0
34: eth6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode 
DEFAULT group default qlen 1000
     link/ether de:ad:be:ff:ff:15 brd ff:ff:ff:ff:ff:ff promiscuity 0
35: eth7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode 
DEFAULT group default qlen 1000
     link/ether de:ad:be:ff:ff:16 brd ff:ff:ff:ff:ff:ff promiscuity 0
36: eth8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode 
DEFAULT group default qlen 1000
     link/ether de:ad:be:ff:ff:17 brd ff:ff:ff:ff:ff:ff promiscuity 0
37: eth9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode 
DEFAULT group default qlen 1000
     link/ether de:ad:be:ff:ff:18 brd ff:ff:ff:ff:ff:ff promiscuity 0
38: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode 
DEFAULT group default qlen 1000
     link/ether de:ad:be:ff:ff:11 brd ff:ff:ff:ff:ff:ff promiscuity 0
39: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode 
DEFAULT group default qlen 1000
     link/ether de:ad:be:ff:ff:12 brd ff:ff:ff:ff:ff:ff promiscuity 0

# lspci
01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet (rev 10)
01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet (rev 10)
01:09.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet Virtual Function
01:09.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet Virtual Function
01:09.2 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet Virtual Function
01:09.3 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet Virtual Function
01:09.4 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet Virtual Function
01:09.5 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet Virtual Function
01:09.6 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet Virtual Function
01:09.7 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet Virtual Function

I'm using libvirt with <hostdev> XML blocks to assign VF0 to VM1 and VF1 
to VM2 (also tried <interface type='network'> and network pool, that 
does not make a difference):

VM1 is attached to VF0:
     <hostdev mode='subsystem' type='pci' managed='yes'>
       <source>
         <address domain='0x0000' bus='0x01' slot='0x09' function='0x0'/>
       </source>
     </hostdev>

VM2 is attached to VF1:
     <hostdev mode='subsystem' type='pci' managed='yes'>
       <source>
         <address domain='0x0000' bus='0x01' slot='0x09' function='0x1'/>
       </source>
     </hostdev>


---------

Once VM1 and VM2 are started, we can see:

----
HOST
----

# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode 
DEFAULT group default
     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0
2: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode 
DEFAULT group default qlen 1000
     link/ether a4:1f:72:d0:b3:9c brd ff:ff:ff:ff:ff:ff promiscuity 0
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP 
mode DEFAULT group default qlen 1000
     link/ether a4:1f:72:d0:b3:9f brd ff:ff:ff:ff:ff:ff promiscuity 0
     vf 0 MAC 00:00:00:00:00:00, tx rate 10000 (Mbps), spoof checking 
on, link-state auto
     vf 1 MAC 00:00:00:00:00:00, tx rate 10000 (Mbps), spoof checking 
on, link-state auto
     vf 2 MAC de:ad:be:ff:ff:13, tx rate 10000 (Mbps), spoof checking 
on, link-state auto
     vf 3 MAC de:ad:be:ff:ff:14, tx rate 10000 (Mbps), spoof checking 
on, link-state auto
     vf 4 MAC de:ad:be:ff:ff:15, tx rate 10000 (Mbps), spoof checking 
on, link-state auto
     vf 5 MAC de:ad:be:ff:ff:16, tx rate 10000 (Mbps), spoof checking 
on, link-state auto
     vf 6 MAC de:ad:be:ff:ff:17, tx rate 10000 (Mbps), spoof checking 
on, link-state auto
     vf 7 MAC de:ad:be:ff:ff:18, tx rate 10000 (Mbps), spoof checking 
on, link-state auto
4: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN 
mode DEFAULT group default
     link/ether 8e:31:e9:eb:89:b9 brd ff:ff:ff:ff:ff:ff promiscuity 0
     bond
32: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode 
DEFAULT group default qlen 1000
     link/ether de:ad:be:ff:ff:13 brd ff:ff:ff:ff:ff:ff promiscuity 0
33: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode 
DEFAULT group default qlen 1000
     link/ether de:ad:be:ff:ff:14 brd ff:ff:ff:ff:ff:ff promiscuity 0
34: eth6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode 
DEFAULT group default qlen 1000
     link/ether de:ad:be:ff:ff:15 brd ff:ff:ff:ff:ff:ff promiscuity 0
35: eth7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode 
DEFAULT group default qlen 1000
     link/ether de:ad:be:ff:ff:16 brd ff:ff:ff:ff:ff:ff promiscuity 0
36: eth8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode 
DEFAULT group default qlen 1000
     link/ether de:ad:be:ff:ff:17 brd ff:ff:ff:ff:ff:ff promiscuity 0
37: eth9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode 
DEFAULT group default qlen 1000
     link/ether de:ad:be:ff:ff:18 brd ff:ff:ff:ff:ff:ff promiscuity 0


---
VM1
---

# ip add
...
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP 
qlen 1000
     link/ether de:ad:be:ff:ff:11 brd ff:ff:ff:ff:ff:ff
     inet 172.20.6.221/24 brd 172.20.6.255 scope global eth0
        valid_lft forever preferred_lft forever
     inet6 fe80::dcad:beff:feff:ff11/64 scope link
        valid_lft forever preferred_lft forever

---
VM2
---

# ip add
...
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP 
qlen 1000
     link/ether de:ad:be:ff:ff:12 brd ff:ff:ff:ff:ff:ff
     inet 172.20.6.222/24 brd 172.20.6.255 scope global eth0
        valid_lft forever preferred_lft forever
     inet6 fe80::dcad:beff:feff:ff12/64 scope link
        valid_lft forever preferred_lft forever

 > Where are you sending data from (a VF, a PF, some peer on the 
network) and where is it arriving (perhaps arriving in multiple places)
 > Please detail the behavior you expect and the behavior you observe.

When I send packets from VM1 to the default router 172.20.6.1 (this is 
an external device) I observe that all the traffic is visible on VM2 
(tshark listening on eth0):

vm2# tshark -i eth0 icmp
...
   0.000000 172.20.6.221 -> 172.20.6.1   ICMP 98 Echo (ping) request 
id=0x09ee, seq=1/256, ttl=64
   0.000297   172.20.6.1 -> 172.20.6.221 ICMP 98 Echo (ping) reply 
id=0x09ee, seq=1/256, ttl=64

Same result when permuting VM1 and VM2. VM2 sees all traffic 
received/sent to VM1.

Another test, I send packets from the outside to the PF (172.20.6.229) 
-> VM1 and VM2 see inbound packets as well as responses from the PF.

 > Please supply:
 > Mac addresses, ip addresses and masks, configured vlans (if any), 
promiscuous setting, etc.
 > ip -d -d link show on hypervisor and each of the VMs involved
 > ethtool -d from Hypervisor PFs
 > dmesg from hypervisor

dmesg is attached. I send to you personally the output of ethtool 
command (>1MB).

>
> Please note we recently submitted some fixes to our tx-switching behavior:
> In e8379c79 "bnx2x: fix VLAN configuration for VFs" we fixed an issue where traffic with the wrong vlan could still end up in a VM configured to a different vlan (hence my questions on vlans).
> In c14db202 "bnx2x: Correct default Tx switching behavior" we fixed a connectivity issue with pf to vf connectivity.
> Depending on your answers to the above, perhaps these might be relevant to your case.

I don't think this is relevant for tx-switching since the issue is also 
with VF to VF traffic + concerns RX and TX switching.

Thanks for your assistance,
Regards,
-- 
Université de Nantes - Direction des Systèmes d'Information

[-- Attachment #1.2: dmesg.gz --]
[-- Type: application/gzip, Size: 18057 bytes --]

[-- Attachment #1.3: yoann_juet.vcf --]
[-- Type: text/x-vcard, Size: 377 bytes --]

begin:vcard
fn:Yoann Juet
n:Juet;Yoann
org;quoted-printable;quoted-printable:Direction des Syst=C3=A8mes d'Information;P=C3=B4le R=C3=A9seau
adr;quoted-printable:BP 92208;;2 rue de la Houssini=C3=A8re;Nantes Cedex 3;;44322;France
email;internet:yoann.juet@univ-nantes.fr
tel;work:02.53.48.49.26
tel;fax:02.53.48.49.09
tel;cell:06.73.15.42.19
version:2.1
end:vcard


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3256 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: bnx2x + SR-IOV, no internal L2 switching
  2014-02-12 13:33 bnx2x + SR-IOV, no internal L2 switching Yoann Juet
  2014-02-12 14:38 ` Ariel Elior
@ 2014-02-12 23:10 ` Ben Hutchings
  2014-02-13  9:42   ` Yoann Juet
  1 sibling, 1 reply; 5+ messages in thread
From: Ben Hutchings @ 2014-02-12 23:10 UTC (permalink / raw)
  To: yoann.juet; +Cc: netdev, Ariel Elior

[-- Attachment #1: Type: text/plain, Size: 1798 bytes --]

On Wed, 2014-02-12 at 14:33 +0100, Yoann Juet wrote:
> Hi all,
> 
> I'm conducting experiments on SR-IOV with Broadcom and Intel cards on 
> debian/unstable with KVM hypervisor. On Broadcom cards (bnx2x module, 
> BCM57810 devices), Virtual Functions (VFs) get running, Virtual Machines 
> attached to such VFs inherit network connectivity with excellent 
> performance.
> 
> However, VMs attached to VFs on the Broadcom Physical Functions (PFs) 
> behave like they were connected to an ancient hub, not a L2 switch. It 
> is as if there was no internal L2 switching on the Broadcom card to 
> process VF <-> VF or VF <-> PF communications. As a result, a VM sees 
> all inbound/outbound traffic from/to others VMs as well as traffic 
> destined to the PF (for instance, the physical ethX has an IP address).

Are you're using the ISC DHCP client, which puts the interface in
promiscuous mode?  If the Broadcom NIC supports promiscuous mode on VFs,
that may explain what you're seeing.

> On the other hand, everything works like a charm with Intel cards (ixgbe 
> module, 82599EB devices). Traffic between VFs or VF/PF is switched 
> internally by the card.

I think these VFs don't support promiscuous mode.  Anyway, the ixgbevf
driver silently ignores it.

Ben.

> I found very little literature about SR-IOV on Broadcom devices. I 
> wonder if it's a normal behaviour, a misconfiguration on my side or 
> perhaps a firmware/driver bug.
> 
> Have you seen this issue before ?
> 
> ---
> Kernel 3.12.9 (same behaviour with kernels 3.10.x)
> 	driver: bnx2x
> 	firmware-version: 7.8.17
> Debian/unstable
> 	libvirt 1.2.1
> 	QEMU 1.7.0
> ---
> 
> Best regards,

-- 
Ben Hutchings
If more than one person is responsible for a bug, no one is at fault.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: bnx2x + SR-IOV, no internal L2 switching
  2014-02-12 23:10 ` Ben Hutchings
@ 2014-02-13  9:42   ` Yoann Juet
  0 siblings, 0 replies; 5+ messages in thread
From: Yoann Juet @ 2014-02-13  9:42 UTC (permalink / raw)
  To: Ben Hutchings; +Cc: netdev, Ariel Elior


[-- Attachment #1.1: Type: text/plain, Size: 563 bytes --]

Hi Ben,

 > Are you're using the ISC DHCP client, which puts the interface in
> promiscuous mode?  If the Broadcom NIC supports promiscuous mode on VFs,
> that may explain what you're seeing.

No, I put the interface into promiscuous mode with Wireshark's tshark 
command line tool.

> I think these VFs don't support promiscuous mode.  Anyway, the ixgbevf
> driver silently ignores it.

This could be an explanation. Ariel Elior sent to me a patch. Test in 
progress.

Regards,
-- 
Université de Nantes - Direction des Systèmes d'Information

[-- Attachment #1.2: yoann_juet.vcf --]
[-- Type: text/x-vcard, Size: 377 bytes --]

begin:vcard
fn:Yoann Juet
n:Juet;Yoann
org;quoted-printable;quoted-printable:Direction des Syst=C3=A8mes d'Information;P=C3=B4le R=C3=A9seau
adr;quoted-printable:BP 92208;;2 rue de la Houssini=C3=A8re;Nantes Cedex 3;;44322;France
email;internet:yoann.juet@univ-nantes.fr
tel;work:02.53.48.49.26
tel;fax:02.53.48.49.09
tel;cell:06.73.15.42.19
version:2.1
end:vcard


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3256 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2014-02-13  9:41 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-02-12 13:33 bnx2x + SR-IOV, no internal L2 switching Yoann Juet
2014-02-12 14:38 ` Ariel Elior
2014-02-12 15:54   ` Yoann Juet
2014-02-12 23:10 ` Ben Hutchings
2014-02-13  9:42   ` Yoann Juet

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox