Re: [linux-4.1 test] 63030: regressions - FAIL

[Ian Campbell <ian.campbell@citrix.com>]

On Thu, 2015-10-22 at 15:41 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [Xen-devel] [linux-4.1 test] 63030: regressions
> - FAIL"):
> > On Thu, 2015-10-22 at 12:03 +0100, Wei Liu wrote:
> > > No, vif-bridge script has two runes for off-lining a vif
> > >   brctl delif $bridge $vif
> > >   ifconfig $vif down
> > > 
> > > Neither of these causes cache entry to be flushed.
> > 
> > $vif disappearing when netback finally deletes the device will though.
> > Or
> > it should/used to.
> > 
> > Maybe this is happening after the new guest has started and confusing
> > things somewhere?
> 
> 
> There is confusion here.  Someone used the phrase `arp cache'.  But
> there are actually two relevant runtime of MAC addresses:
> 
>  * Each host has a neighbour database mapping IPv4 addresses to MAC
>    addresses.  This is used when trying to pass on an IPv4 datagram to
>    a host on the same ethernet (same broadcast domain).  This database
>    is normally referred to as an `ARP cache'.  Addresses are added to
>    the table by both ARP requests and responses, and also in many
>    implementations entries are refreshed by ordinary traffic.
> 
>    In the test colo, the osstest VM is on the same (bridged) ethernet
>    as the test box so, the relevant arp cache is the one in the
>    osstest controller's kernel: the osstest controller wants to send
>    an ssh SYN to the guest, and needs to construct an ethernet frame
>    with the guest's MAC address.  This is done using the osstest
>    controller's ARP cache entry.
> 
>    The osstest controller's ARP cache is unaffected by the migration.
>    ARP cache entries do time out but only after a number of minutes,
>    and the guest will have been spoken to recently by the controller.
>    I have no reason to think that lack of an entry for the guest's
>    IPv4 address in the osstest controller's ARP cache is relevant.

I was talking about this kind of ARP cache, but the one in the (single,
since it is localhost migrate) dom0. That's because I had misread Wei's
earlier script as sshing to the guest from dom0, not from his workstation
(the "controller" in his scenario).

Sorry for the confusion.

FWIW I believe the source dom0's ARP entry will be dropped when the VIF
device is destroyed.

>  * Each bridge has a forwarding database mapping MAC addresses to its
>    outbound links.  This is normally referred to as the bridge
>    (switch) `learning', and the table as the `MAC address table'.  MAC
>    addresses are learned when switch sees incoming frames.  When the
>    bridge receives a frame for a destination MAC address for which it
>    has no entry, it forwards the frame out of all its ports.  Special
>    considerations apply to broadcast and multicast MAC addresses.
>    None of this involves IPv4 or IPv6 addresses.
> 
>    In the test colo in the migration test case, there are up to four
>    relevant bridges:
> 
>       * The source test box's dom0's software bridge.
>         This has (logically speaking) three `ports':
>           - the test box's physical network interface
>           - the dom0 itself
>           - the vif corresponding to the outbound guest
>           - in a single-host test, the vif corresponding to
>             the inbound guest
> 
>       * The physical switch connecting the test boxes and the VM host
>         (newcastle.test-lab.xenproject.org).  This has two or three
>         relevant physical ports, for the two or three relevant
>         physical machines.  (In fact there are VLANs involved but this
>         is not relevant.)
> 
>       * The software bridge in newcastle.  This has two relevant
>         ports:
>           - newcastle's physical interface
>           - the vif serving the osstest VM
>         
>       * In a two-machine test, rather than a localhost test, the
>         destination test box's dom0's software bridge, which parallels
>         the source test box's.
> 
>    When the guest stops running on the source (with its vif torn
>    down), and starts running on the destination:
> 
>      (a) The source test box software bridge should lose its MAC
>         address table entry for the guest, because the corresponding
>         port (the vif) is removed.  However I am not sure whether this
>         actually happens immediately in Linux.

For Linux bridging I believe it happens at the latest when the vif device
is deleted, or possibly when it is removed from the bridge (i.e. earlier).

IOW I do not believe that Linux bridge remembers old ports.

openvswitch might, I don't recall, but I don't think that is in the picture
here.

>         It may be that instead the MAC address table entry for the
>         guest remains present but points to the dead vif.  In this
>         case incoming frames from the wire, the for the guest will be
>         dropped.
> 
>      (b) The destination test box (if different) will come up without a
>         MAC address entry for the guest.

Given the above I think even if it is the same as the source box, since it
will have been forgotten by the "source" box when the original VIF
disappeared.

>   If a frame for the guest's
>         MAC address arrives at the physical interface, it will be
>         forwarded to all of the other interfaces enslaved to the
>         bridge: ie, to the dom0 (which will ignore it because it has
>         the wrong destination MAC address) and to the newly-created
>         guest.
> 
>      (c) In a two-host test, the physical switch connecting the two
>         test boxes will retain the wrong learnt switch port.  It will
>         forward frames for the guest (only) to the source test box,
>         rather than the destination test box, where they will be
>         discarded.
> 
>    It is (a) and (c) that the gratuitous ARP is supposed to fix.
> 
>    The guest is supposed to send, when its interface comes up after
>    migration, a single broadcast gratuitous ARP response containing
>    its own IPv4 and MAC addresses.
> 
>    The IPv4 address in this message is irrelevant.
> 
>    The purpose is to update the MAC address tables in all the switches
>    in the network.  Each switch which receives the gratuitous ARP
>    updates its MAC address table to map the guest's MAC address to the
>    port on which the gratuitous ARP was recevied.
> 
>    If this happens, then frames from everywhere on the ethernet, to
>    the guest, will be properly delivered.  If it doesn't then there
>    may be lost packets and/or low-level timeouts of various kinds.
> 
> 
> Ian.