2.6.31 ARP related problems with multiple macvlan NICs

2.6.31 ARP related problems with multiple macvlan NICs

[Or Gerlitz]

Using multiple (e.g 2) macvlan devices set over the same uplink NIC
and 2.6.31-rc7 I can get only one of the macvlan devices to respond on
arp request where the same scheme works fine on 2.6.29.1 and 2.6.30.

The only devices for which the system responds on ARP request is the
first match in the routing table, e.g mv0 below. Next are the commands
I am using to set the environment that reproduces the problem.

Looking on net/ipv4/arp.c I do someting that may be related which
was commited for 2.6.30 and reverted for -stable and .31 so the
fact that this test actually works with 2.6.29.1/2.6.30 makes me
think that the problem I see is not directly connected to the "ipv4: arp
announce, arp_proxy and windows ip conflict verification" commit/revert.

The problem is also not directly related to macvlan, I think, e.g
I have the same issue when having multiple veth pairs connected
to a bridge, only ping to/through the first routing hit works.

Or.

--> setup things

ip link add link eth3 address 00:19:d1:29:d2:00 mv0 type macvlan
ip link add link eth3 address 00:19:d1:29:d2:01 mv1 type macvlan

ifconfig mv0 20.20.49.10/16 up
ifconfig mv1 20.20.49.11/16 up

sysctl -w net.ipv4.conf.eth3.arp_ignore=1
sysctl -w net.ipv4.conf.mv0.arp_ignore=1
sysctl -w net.ipv4.conf.mv1.arp_ignore=1

--> try to a ping remote node from either of the interfaces
--> only the ping that goes through the first routing hit (mv0) works

# ping -I mv0 20.20.49.1 -f -c 100 -q
100 packets transmitted, 100 received, 0% packet loss, time 3ms

# ping -I mv1 20.20.49.1 -f -c 100 -q
100 packets transmitted, 0 received, 100% packet loss, time 1187ms

--> try to ping both interfaces from the remote node, same problem
# ping 20.20.49.10 -f -c 100 -q
100 packets transmitted, 100 received, 0% packet loss, time 2ms

# ping 20.20.49.11 -f -c 100 -q
100 packets transmitted, 0 received, +3 errors, 100% packet loss, time 1187ms, pipe 3

Re: 2.6.31 ARP related problems with multiple macvlan NICs

[Eric W. Biederman]

Or Gerlitz <ogerlitz@voltaire.com> writes:

> Using multiple (e.g 2) macvlan devices set over the same uplink NIC
> and 2.6.31-rc7 I can get only one of the macvlan devices to respond on
> arp request where the same scheme works fine on 2.6.29.1 and 2.6.30.
>
> The only devices for which the system responds on ARP request is the
> first match in the routing table, e.g mv0 below. Next are the commands
> I am using to set the environment that reproduces the problem.
>
> Looking on net/ipv4/arp.c I do someting that may be related which
> was commited for 2.6.30 and reverted for -stable and .31 so the
> fact that this test actually works with 2.6.29.1/2.6.30 makes me
> think that the problem I see is not directly connected to the "ipv4: arp
> announce, arp_proxy and windows ip conflict verification" commit/revert.
>
> The problem is also not directly related to macvlan, I think, e.g
> I have the same issue when having multiple veth pairs connected
> to a bridge, only ping to/through the first routing hit works.

I just tested.  If the two macvlans are in separate network namespaces
all is well, so definitely not macvlan.

As you have observed there are no real changes to arp.c

Interesting I can reproduce this to machines on the same network
segment but not to machines on different network segments.

Weird.


Eric

Re: 2.6.31 ARP related problems

[Or Gerlitz]

Eric W. Biederman wrote:
> I just tested.  If the two macvlans are in separate network namespaces all is well,
> so definitely not macvlan. As you have observed there are no real changes to arp.c

Yes, it's not macvlan to blame, I just tested it in SR-IOV scheme with igb/igbvf and
I see the same problem, only ping that goes through / targeted to the IP address of the first
VF device routing hit works, which means SR-IOV isn't really usable with 2.6.31 when you want 
a multiple VMs scheme, each attached to a different VF and all VMs on the same network segment.


Or.

Re: 2.6.31 ARP related problems

[Alexander Duyck]

I don't suspect this has much of an effect on the Virtualization use
case for SR-IOV since the VFs are meant to be direct assigned as PCI
devices to the individual VMs so they won't even show up in the
routing table.  For the most part the igbvf driver typically will be
black listed for the host kernel since it already has the PF
interface, and the driver will only be loaded on the guests.

You can probably also reproduce the issue by placing multiple physical
network interfaces on the same network segment if you saw the same
effect on SR-IOV since that is essentially the effect the VFs create
due to the switching logic built into the 82576.

Thanks,

Alex

On Wed, Sep 2, 2009 at 5:20 AM, Or Gerlitz<ogerlitz@voltaire.com> wrote:
> Eric W. Biederman wrote:
>> I just tested.  If the two macvlans are in separate network namespaces all is well,
>> so definitely not macvlan. As you have observed there are no real changes to arp.c
>
> Yes, it's not macvlan to blame, I just tested it in SR-IOV scheme with igb/igbvf and
> I see the same problem, only ping that goes through / targeted to the IP address of the first
> VF device routing hit works, which means SR-IOV isn't really usable with 2.6.31 when you want
> a multiple VMs scheme, each attached to a different VF and all VMs on the same network segment.
>
>
> Or.
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: 2.6.31 ARP related problems

[Or Gerlitz]

Alexander Duyck wrote:
> I don't suspect this has much of an effect on the Virtualization use case for SR-IOV since the VFs are meant to be direct assigned as PCI devices to the individual VMs 

I understand that eventually there will be scheme when VFs will be
directly assigned to the VM, but there are/will be many occasions where
a VF will serve as a virtual NIC in a Linux system e.g one serving as a
host but also other purposes (think on macvlan as "software SR-IOV"
where with your HW its the real thing).

> You can probably also reproduce the issue by placing multiple physical network interfaces on the same network segment if you saw the same effect on SR-IOV since that is essentially the effect the VFs create due to the switching logic built into the 82576
Yes, as I managed to produce it with thee schemes: macvlan, veth+bridge
and SR-IOV, I believe something is just broken wrt to ARP replies in
2.6.31 which is now in its rc8! I will try to look on that, and
hopefully we can fix it at least for -stable.

Or.

I wasn't sure to understand your "the effect the VFs create due to the
switching logic built into the 82576" comment, can you elaborate more on
that?

RE: 2.6.31 ARP related problems

[Duyck, Alexander H]

Or Gerlitz wrote:
> Alexander Duyck wrote:
>> I don't suspect this has much of an effect on the Virtualization use
>> case for SR-IOV since the VFs are meant to be direct assigned as PCI
>> devices to the individual VMs  
> 
> I understand that eventually there will be scheme when VFs will be
> directly assigned to the VM, but there are/will be many occasions
> where a VF will serve as a virtual NIC in a Linux system e.g one
> serving as a host but also other purposes (think on macvlan as
> "software SR-IOV" where with your HW its the real thing).
> 
>> You can probably also reproduce the issue by placing multiple
>> physical network interfaces on the same network segment if you saw
>> the same effect on SR-IOV since that is essentially the effect the
>> VFs create due to the switching logic built into the 82576   
> Yes, as I managed to produce it with thee schemes: macvlan,
> veth+bridge and SR-IOV, I believe something is just broken wrt to ARP
> replies in 
> 2.6.31 which is now in its rc8! I will try to look on that, and
> hopefully we can fix it at least for -stable.
> 
> Or.
> 
> I wasn't sure to understand your "the effect the VFs create due to the
> switching logic built into the 82576" comment, can you elaborate more
> on that?

The way the VFs work is that there is an L2 switch in the 82576 that is routing traffic between the PF, VFs, and the physical port.  It behaves very much like if you had multiple NICs connected to an external L2 switch with a gigabit uplink.  As such if you were to connect multiple physical ports to the same switch and configure them with addresses as you did with the VFs you should also see the same behavior.

Thanks,

Alex