network breakage, smoking gun

network breakage, smoking gun

[Ted Kaczmarek]

Reproduced the network breakage problem, easiest way to reproduce is to
try a live migration across the xen-br0, but from what I am seeing just
bringing domU's up causes it as well.

You know its broke when you see 

peth0: received packet with own address as source address

starts filling up the xen console.

Tried to set the peth0 hardware address no joy
[root@tarkus ~]# ifconfig peth0 hw ether FD:FF:FF:FF:FF:FF
SIOCSIFHWADDR: Cannot assign requested address

tried fd:ff:ff:ff:ff:ff as well.

               Gateway
                   |
      --------------------------  subnet A
         |                 |
     dom0-1              dom0-2   dom0's
        |                  |              subnet B -xen-br0
----------------------------------------------------  
|    |   |   |   |   |   |   |   |   |    |   |   | 
1-1 1-2 1-3 1-4 1-5 1-6 1-7 1-8 1-9 1-10 2-1 2-2 2-3 domU's 


Ian stated that peth0 should not be end point src
or destination of any packets, but I suspect some llc frames
may be the cause of this.

This has been lurking for a long time, actually thought this was
expected behavior :-)

The domO's each have three nics, but eth2 is not used at this time.

Based upon what I have seen, almost anything will bring this problem on.
I always find it because each dom0 has one domU running OpenNMS on it,
and they each start reporting the neighbor dom0 xen-br0 ethernet down.

It gets really funky, domU's on one dom0 can talk to all domU's, while
domu's on the other dom0 can only see local domU's. The d


This is the controller on both machines for xen-br0

########
 dom0-1
########

00:0c.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit
Ethernet Controller
        Subsystem: Intel Corporation PRO/1000 MT Desktop Adapter
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop-
ParErr- Stepping- SERR- FastB2B-
        Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
        Latency: 64 (63750ns min), Cache Line Size 10
        Interrupt: pin A routed to IRQ 16
        Region 0: Memory at f4060000 (32-bit, non-prefetchable)
[size=128K]
        Region 1: Memory at f4040000 (32-bit, non-prefetchable)
[size=128K]
        Region 2: I/O ports at 2000 [size=64]
        Capabilities: [dc] Power Management version 2
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0
+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [e4] PCI-X non-bridge device.
                Command: DPERE- ERO+ RBC=0 OST=0
                Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-,
DC=simple, DMMRBC=2, DMOST=0, DMCRS=0, RSCEM-        Capabilities: [f0]
Message Signalled Interrupts: 64bit+ Queue=0/0 Enable-
                Address: 0000000000000000  Data: 0000


########
 dom0-2
########

02:0b.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet
Controller (rev 02)
        Subsystem: Intel Corporation PRO/1000 MT Desktop Adapter
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop-
ParErr- Stepping- SERR+ FastB2B-
        Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-        Latency: 64 (63750ns min), Cache
Line Size 10
        Interrupt: pin A routed to IRQ 16
        Region 0: Memory at fe1a0000 (32-bit, non-prefetchable)
[size=128K]
        Region 1: Memory at fe180000 (32-bit, non-prefetchable)
[size=128K]
        Region 2: I/O ports at e840 [size=64]
        Expansion ROM at fe200000 [disabled] [size=128K]
        Capabilities: [dc] Power Management version 2
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0
+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [e4] PCI-X non-bridge device.
                Command: DPERE- ERO+ RBC=0 OST=0
                Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-,
DC=simple, DMMRBC=2, DMOST=0, DMCRS=1, RSCEM-
        Capabilities: [f0] Message Signalled Interrupts: 64bit+
Queue=0/0 Enable-

Regards,
Ted