Re: r8169 ethernet bonding problems

Re: r8169 ethernet bonding problems

[Tim Durack]

> See:
> http://www.fr.zoreil.com/linux/kernel/2.6.x/2.6.21-rc7/
>
> Please Cc: netdev on success/failure.

Applied suggested patchset to vanilla 2.6.21.

Compiled using .config from linux-image-2.6.21-rc7 pulled down from
http://kernel-archive.buildserver.net/debian-kernel

Compiles, boots.

Bonding does not work reliably. Simple config:

#auto bond0
iface bond0 inet manual
  up ip link set dev $IFACE up
  up echo +eth1 > /sys/class/net/$IFACE/bonding/slaves
  up echo +eth2 > /sys/class/net/$IFACE/bonding/slaves
  up ip link set dev eth1 up
  up ip link set dev eth2 up
  up ip addr add 172.21.255.5/29 dev $IFACE
  down ip addr del 172.21.255.5/29 dev $IFACE
  down ip link set dev eth2 down
  down ip link set dev eth1 down
  down echo -eth2 > /sys/class/net/$IFACE/bonding/slaves
  down echo -eth1 > /sys/class/net/$IFACE/bonding/slaves
  down ip link set dev $IFACE down

Bond comes up, cannot ping unless I manually force promisc on both member links:

ip link set dev eth1 promisc on
ip link set dev eth2 promisc off

Link failover does not work reliably. Carrier-detect appears to be
working, so I don't think that is an issue.


A disturbing problem is mac corruption after "modprobe -r r8169; modprobe r8169"

ip link sh:

2: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
    link/ether 00:00:00:00:68:93 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:30:18:ab:68:94 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
    link/ether 00:30:18:ab:68:95 brd ff:ff:ff:ff:ff:ff

eth5 mac was 00:30:18:ab:68:93 before the modprobe.

Not sure what's going on here, but I'm starting to think these
interfaces aren't the best in the world...

Tim:>

Re: r8169 ethernet bonding problems

[Jay Vosburgh]

Tim Durack <tdurack@gmail.com> wrote:

[...]
>Bond comes up, cannot ping unless I manually force promisc on both member links:
>
>ip link set dev eth1 promisc on
>ip link set dev eth2 promisc off

	From looking at the source code for r8169, but it appears that
the r8169 driver only reads the device MAC address at probe time, and
doesn't propogate changes to the MAC out to the device.  That might
explain the "promisc" problem if the device also does MAC filtering.  Am
I missing something in the driver?

	I don't actually have one of these to test with, I'm just
looking at the source and observing that dev_addr is only ever
referenced in the probe function, and that's only to read in the MAC
from the device.

	-J

---
	-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com

Re: r8169 ethernet bonding problems

[Tim Durack]

> >ip link set dev eth1 promisc on
> >ip link set dev eth2 promisc off

typo, "ip link set dev eth2 promisc on" of course

>
>        From looking at the source code for r8169, but it appears that
> the r8169 driver only reads the device MAC address at probe time, and
> doesn't propogate changes to the MAC out to the device.  That might
> explain the "promisc" problem if the device also does MAC filtering.  Am
> I missing something in the driver?

I have applied all patches from the set, including:

0012-r8169-mac-address-change-support.patch

which I assume adds the ability to reprogram MAC filtering.

Behaviour indicates the MAC filter doesn't get reprogrammed though.

>        I don't actually have one of these to test with, I'm just
> looking at the source and observing that dev_addr is only ever
> referenced in the probe function, and that's only to read in the MAC
> from the device.

You're not missing much, believe me :-|

Tim:>

Re: r8169 ethernet bonding problems

[Francois Romieu]

Tim Durack <tdurack@gmail.com> :
[...]
> Link failover does not work reliably. Carrier-detect appears to be
> working, so I don't think that is an issue.
> 
> 
> A disturbing problem is mac corruption after "modprobe -r r8169; modprobe 
> r8169"
> 
> ip link sh:
> 
> 2: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
>    link/ether 00:00:00:00:68:93 brd ff:ff:ff:ff:ff:ff
> 3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast qlen 1000
>    link/ether 00:30:18:ab:68:94 brd ff:ff:ff:ff:ff:ff
> 4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
>    link/ether 00:30:18:ab:68:95 brd ff:ff:ff:ff:ff:ff
> 
> eth5 mac was 00:30:18:ab:68:93 before the modprobe.

Can you revert patch #0012 and see if it changes this part of the problem ?

A tcpdump trace 1) before promiscuous mode 2) after promiscuous mode 3)
after link failure (i.e. failover does not work) would be welcome.

> Not sure what's going on here, but I'm starting to think these
> interfaces aren't the best in the world...

The maintainer/driver/hardware combination is a bit quirky at times
but it does not work too bad. :o)

-- 
Ueimor

Anybody got a battery for my Ultra 10 ?

Re: r8169 ethernet bonding problems

[Francois Romieu]

Francois Romieu <romieu@fr.zoreil.com> :
[...]
> A tcpdump trace 1) before promiscuous mode 2) after promiscuous mode 3)
> after link failure (i.e. failover does not work) would be welcome.

A 'lspci -vx' will be needed too. There are different kinds of 8169.

-- 
Ueimor

Anybody got a battery for my Ultra 10 ?

Re: r8169 ethernet bonding problems

[Tim Durack]

> Can you revert patch #0012 and see if it changes this part of the problem ?

Reverted. mac is still wrong - I assume the eprom got corrupted somehow.

~# modprobe r8169
r8169 Gigabit Ethernet driver 2.2LK-NAPI loaded
ACPI: PCI Interrupt 0000:00:09.0[A] -> GSI 18 (level, low) -> IRQ 16
eth1: RTL8169sc/8110sc at 0xf8838000, 00:00:00:00:68:93, IRQ 16
r8169 Gigabit Ethernet driver 2.2LK-NAPI loaded
ACPI: PCI Interrupt 0000:00:0b.0[A] -> GSI 19 (level, low) -> IRQ 17
eth2: RTL8169sc/8110sc at 0xf883a000, 00:30:18:ab:68:94, IRQ 17
r8169 Gigabit Ethernet driver 2.2LK-NAPI loaded
ACPI: PCI Interrupt 0000:00:0c.0[A] -> GSI 16 (level, low) -> IRQ 18
eth4: RTL8169sc/8110sc at 0xf8a76000, 00:30:18:ab:68:95, IRQ 18

I can work around this with an "ip links set dev eth5 addr" for now.

Strange things is, this group of interfaces used to get initialised as
eth0/1/2, in sequence. Now one of the interfaces is always out of
order.

> A tcpdump trace 1) before promiscuous mode 2) after promiscuous mode 3)
> after link failure (i.e. failover does not work) would be welcome.

Will get these shortly.

lspci -vx:

00:00.0 Host bridge: VIA Technologies, Inc. CN700/VN800/P4M800CE/Pro Host Bridge
	Subsystem: VIA Technologies, Inc. CN700/VN800/P4M800CE/Pro Host Bridge
	Flags: bus master, 66MHz, medium devsel, latency 8
	Memory at e8000000 (32-bit, prefetchable) [size=128M]
	Capabilities: [80] AGP version 3.5
	Capabilities: [50] Power Management version 2
00: 06 11 14 03 06 00 30 22 00 00 00 06 00 08 80 00
10: 08 00 00 e8 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 06 11 14 03
30: 00 00 00 00 80 00 00 00 00 00 00 00 00 00 00 00

00:00.1 Host bridge: VIA Technologies, Inc. CN700/VN800/P4M800CE/Pro Host Bridge
	Flags: bus master, medium devsel, latency 0
00: 06 11 14 13 06 00 00 02 00 00 00 06 00 00 00 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

00:00.2 Host bridge: VIA Technologies, Inc. CN700/VN800/P4M800CE/Pro Host Bridge
	Flags: bus master, medium devsel, latency 0
00: 06 11 14 23 06 00 00 02 00 00 00 06 00 00 00 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

00:00.3 Host bridge: VIA Technologies, Inc. PT890 Host Bridge
	Flags: bus master, medium devsel, latency 0
00: 06 11 08 32 06 00 00 02 00 00 00 06 00 00 00 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

00:00.4 Host bridge: VIA Technologies, Inc. CN700/VN800/P4M800CE/Pro Host Bridge
	Flags: bus master, medium devsel, latency 0
00: 06 11 14 43 06 00 00 02 00 00 00 06 00 00 00 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

00:00.7 Host bridge: VIA Technologies, Inc. CN700/VN800/P4M800CE/Pro Host Bridge
	Flags: bus master, medium devsel, latency 0
00: 06 11 14 73 06 00 00 02 00 00 00 06 00 00 00 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 a0 00 00 00 00 00 00 00 00 00 00 00

00:01.0 PCI bridge: VIA Technologies, Inc. VT8237 PCI Bridge (prog-if
00 [Normal decode])
	Flags: bus master, 66MHz, medium devsel, latency 0
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	I/O behind bridge: 0000d000-0000dfff
	Memory behind bridge: fb000000-fcffffff
	Prefetchable memory behind bridge: f4000000-f7ffffff
	Capabilities: [70] Power Management version 2
00: 06 11 98 b1 07 00 30 02 00 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 d0 d0 20 e2
20: 00 fb f0 fc 00 f4 f0 f7 00 00 00 00 00 00 00 00
30: 00 00 00 00 70 00 00 00 00 00 00 00 00 00 08 00

00:09.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8169
Gigabit Ethernet (rev 10)
	Subsystem: Jetway Information Co., Ltd. Unknown device 10ec
	Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 16
	I/O ports at f200 [size=256]
	Memory at fdfff000 (32-bit, non-prefetchable) [size=256]
	[virtual] Expansion ROM at 40000000 [disabled] [size=64K]
	Capabilities: [dc] Power Management version 2
00: ec 10 69 81 17 00 b0 02 10 00 00 02 08 40 00 00
10: 01 f2 00 00 00 f0 ff fd 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 f3 16 ec 10
30: 00 00 00 00 dc 00 00 00 00 00 00 00 0b 01 20 40

00:0a.0 FireWire (IEEE 1394): VIA Technologies, Inc. IEEE 1394 Host
Controller (rev 80) (prog-if 10 [OHCI])
	Subsystem: VIA Technologies, Inc. IEEE 1394 Host Controller
	Flags: bus master, stepping, medium devsel, latency 32, IRQ 17
	Memory at fdffe000 (32-bit, non-prefetchable) [size=2K]
	I/O ports at ff00 [size=128]
	Capabilities: [50] Power Management version 2
00: 06 11 44 30 87 00 10 02 80 10 00 0c 01 20 00 00
10: 00 e0 ff fd 01 ff 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 06 11 44 30
30: 00 00 00 00 50 00 00 00 00 00 00 00 05 01 00 20

00:0b.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8169
Gigabit Ethernet (rev 10)
	Subsystem: Jetway Information Co., Ltd. Unknown device 10ec
	Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 17
	I/O ports at f000 [size=256]
	Memory at fdffd000 (32-bit, non-prefetchable) [size=256]
	[virtual] Expansion ROM at 40010000 [disabled] [size=64K]
	Capabilities: [dc] Power Management version 2
00: ec 10 69 81 17 00 b0 02 10 00 00 02 08 40 00 00
10: 01 f0 00 00 00 d0 ff fd 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 f3 16 ec 10
30: 00 00 00 00 dc 00 00 00 00 00 00 00 05 01 20 40

00:0c.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8169
Gigabit Ethernet (rev 10)
	Subsystem: Jetway Information Co., Ltd. Unknown device 10ec
	Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 18
	I/O ports at ec00 [size=256]
	Memory at fdffc000 (32-bit, non-prefetchable) [size=256]
	[virtual] Expansion ROM at 40020000 [disabled] [size=64K]
	Capabilities: [dc] Power Management version 2
00: ec 10 69 81 17 00 b0 02 10 00 00 02 08 40 00 00
10: 01 ec 00 00 00 c0 ff fd 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 f3 16 ec 10
30: 00 00 00 00 dc 00 00 00 00 00 00 00 0a 01 20 40

00:0f.0 IDE interface: VIA Technologies, Inc. VIA VT6420 SATA RAID
Controller (rev 80) (prog-if 8f [Master SecP SecO PriP PriO])
	Subsystem: VIA Technologies, Inc. VIA VT6420 SATA RAID Controller
	Flags: bus master, medium devsel, latency 32, IRQ 21
	I/O ports at fe00 [size=8]
	I/O ports at fd00 [size=4]
	I/O ports at fc00 [size=8]
	I/O ports at fb00 [size=4]
	I/O ports at fa00 [size=16]
	I/O ports at ee00 [size=256]
	Capabilities: [c0] Power Management version 2
00: 06 11 49 31 07 00 90 02 80 8f 01 01 00 20 80 00
10: 01 fe 00 00 01 fd 00 00 01 fc 00 00 01 fb 00 00
20: 01 fa 00 00 01 ee 00 00 00 00 00 00 06 11 49 31
30: 00 00 00 00 c0 00 00 00 00 00 00 00 0b 02 00 00

00:0f.1 IDE interface: VIA Technologies, Inc.
VT82C586A/B/VT82C686/A/B/VT823x/A/C PIPC Bus Master IDE (rev 06)
(prog-if 8a [Master SecP PriP])
	Subsystem: VIA Technologies, Inc.
VT82C586/B/VT82C686/A/B/VT8233/A/C/VT8235 PIPC Bus Master IDE
	Flags: bus master, medium devsel, latency 32, IRQ 21
	[virtual] Memory at 000001f0 (32-bit, non-prefetchable) [size=8]
	[virtual] Memory at 000003f0 (type 3, non-prefetchable) [size=1]
	[virtual] Memory at 00000170 (32-bit, non-prefetchable) [size=8]
	[virtual] Memory at 00000370 (type 3, non-prefetchable) [size=1]
	I/O ports at f900 [size=16]
	Capabilities: [c0] Power Management version 2
00: 06 11 71 05 07 00 90 02 06 8a 01 01 00 20 00 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 01 f9 00 00 00 00 00 00 00 00 00 00 06 11 71 05
30: 00 00 00 00 c0 00 00 00 00 00 00 00 ff 01 00 00

00:10.0 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1
Controller (rev 81) (prog-if 00 [UHCI])
	Subsystem: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller
	Flags: bus master, medium devsel, latency 32, IRQ 19
	I/O ports at f800 [size=32]
	Capabilities: [80] Power Management version 2
00: 06 11 38 30 07 00 10 02 81 00 03 0c 01 20 80 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 01 f8 00 00 00 00 00 00 00 00 00 00 06 11 38 30
30: 00 00 00 00 80 00 00 00 00 00 00 00 0a 01 00 00

00:10.1 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1
Controller (rev 81) (prog-if 00 [UHCI])
	Subsystem: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller
	Flags: bus master, medium devsel, latency 32, IRQ 19
	I/O ports at f700 [size=32]
	Capabilities: [80] Power Management version 2
00: 06 11 38 30 07 00 10 02 81 00 03 0c 01 20 80 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 01 f7 00 00 00 00 00 00 00 00 00 00 06 11 38 30
30: 00 00 00 00 80 00 00 00 00 00 00 00 0a 01 00 00

00:10.2 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1
Controller (rev 81) (prog-if 00 [UHCI])
	Subsystem: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller
	Flags: bus master, medium devsel, latency 32, IRQ 19
	I/O ports at f600 [size=32]
	Capabilities: [80] Power Management version 2
00: 06 11 38 30 07 00 10 02 81 00 03 0c 01 20 80 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 01 f6 00 00 00 00 00 00 00 00 00 00 06 11 38 30
30: 00 00 00 00 80 00 00 00 00 00 00 00 0b 02 00 00

00:10.3 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1
Controller (rev 81) (prog-if 00 [UHCI])
	Subsystem: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller
	Flags: bus master, medium devsel, latency 32, IRQ 19
	I/O ports at f500 [size=32]
	Capabilities: [80] Power Management version 2
00: 06 11 38 30 07 00 10 02 81 00 03 0c 01 20 80 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 01 f5 00 00 00 00 00 00 00 00 00 00 06 11 38 30
30: 00 00 00 00 80 00 00 00 00 00 00 00 0b 02 00 00

00:10.4 USB Controller: VIA Technologies, Inc. USB 2.0 (rev 86)
(prog-if 20 [EHCI])
	Subsystem: VIA Technologies, Inc. USB 2.0
	Flags: bus master, medium devsel, latency 32, IRQ 19
	Memory at fdffb000 (32-bit, non-prefetchable) [size=256]
	Capabilities: [80] Power Management version 2
00: 06 11 04 31 16 00 10 02 86 20 03 0c 08 20 80 00
10: 00 b0 ff fd 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 06 11 04 31
30: 00 00 00 00 80 00 00 00 00 00 00 00 0b 03 00 00

00:11.0 ISA bridge: VIA Technologies, Inc. VT8237 ISA bridge
[KT600/K8T800/K8T890 South]
	Subsystem: VIA Technologies, Inc. DFI KT600-AL Motherboard
	Flags: bus master, stepping, medium devsel, latency 0
	Capabilities: [c0] Power Management version 2
00: 06 11 27 32 87 00 10 02 00 00 01 06 00 00 80 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 06 11 27 32
30: 00 00 00 00 c0 00 00 00 00 00 00 00 00 00 00 00

00:11.5 Multimedia audio controller: VIA Technologies, Inc.
VT8233/A/8235/8237 AC97 Audio Controller (rev 60)
	Subsystem: Jetway Information Co., Ltd. Unknown device 4170
	Flags: medium devsel, IRQ 22
	I/O ports at ea00 [size=256]
	Capabilities: [c0] Power Management version 2
00: 06 11 59 30 01 00 10 02 60 00 01 04 00 00 00 00
10: 01 ea 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 f3 16 70 41
30: 00 00 00 00 c0 00 00 00 00 00 00 00 0b 03 00 00

00:12.0 Ethernet controller: VIA Technologies, Inc. VT6102 [Rhine-II] (rev 78)
	Subsystem: VIA Technologies, Inc. VT6102 [Rhine II] Embeded Ethernet
Controller on VT8235
	Flags: bus master, stepping, medium devsel, latency 32, IRQ 20
	I/O ports at e800 [size=256]
	Memory at fdffa000 (32-bit, non-prefetchable) [size=256]
	Capabilities: [40] Power Management version 2
00: 06 11 65 30 87 00 10 02 78 00 00 02 01 20 00 00
10: 01 e8 00 00 00 a0 ff fd 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 06 11 02 01
30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 01 03 08

01:00.0 VGA compatible controller: VIA Technologies, Inc. UniChrome
Pro IGP (rev 01) (prog-if 00 [VGA])
	Subsystem: VIA Technologies, Inc. UniChrome Pro IGP
	Flags: bus master, 66MHz, medium devsel, latency 32, IRQ 10
	Memory at f4000000 (32-bit, prefetchable) [size=64M]
	Memory at fb000000 (32-bit, non-prefetchable) [size=16M]
	[virtual] Expansion ROM at fc000000 [disabled] [size=64K]
	Capabilities: [60] Power Management version 2
	Capabilities: [70] AGP version 3.0
00: 06 11 44 33 07 00 30 02 01 00 00 03 00 20 00 00
10: 08 00 00 f4 00 00 00 fb 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 06 11 44 33
30: 00 00 00 00 60 00 00 00 00 00 00 00 0a 01 02 00

Re: r8169 ethernet bonding problems

[Tim Durack]

[-- Attachment #1: Type: text/plain, Size: 269 bytes --]

> A tcpdump trace 1) before promiscuous mode 2) after promiscuous mode 3)
> after link failure (i.e. failover does not work) would be welcome.

See attached. Captures performed on bond0. Let me know if you prefer
captures against the physical member interfaces.

Tim:>

[-- Attachment #2: promisc-off.pcap --]
[-- Type: application/octet-stream, Size: 0 bytes --]



[-- Attachment #3: promisc-on.pcap --]
[-- Type: application/octet-stream, Size: 0 bytes --]



[-- Attachment #4: fail-eth1.pcap --]
[-- Type: application/octet-stream, Size: 0 bytes --]



[-- Attachment #5: recover-eth1-1.pcap --]
[-- Type: application/octet-stream, Size: 0 bytes --]



[-- Attachment #6: fail-eth2-1.pcap --]
[-- Type: application/octet-stream, Size: 0 bytes --]



[-- Attachment #7: recover-eth2.pcap --]
[-- Type: application/octet-stream, Size: 0 bytes --]

Re: r8169 ethernet bonding problems

[Tim Durack]

[-- Attachment #1: Type: text/plain, Size: 370 bytes --]

On 4/30/07, Tim Durack <tdurack@gmail.com> wrote:
> > A tcpdump trace 1) before promiscuous mode 2) after promiscuous mode 3)
> > after link failure (i.e. failover does not work) would be welcome.
>
> See attached. Captures performed on bond0. Let me know if you prefer
> captures against the physical member interfaces.

Hmm, see attached files with content this time.

[-- Attachment #2: promisc-off.pcap --]
[-- Type: application/octet-stream, Size: 1164 bytes --]

[-- Attachment #3: promisc-on.pcap --]
[-- Type: application/octet-stream, Size: 822 bytes --]

[-- Attachment #4: fail-eth1.pcap --]
[-- Type: application/octet-stream, Size: 1164 bytes --]

[-- Attachment #5: recover-eth1.pcap --]
[-- Type: application/octet-stream, Size: 16594 bytes --]

[-- Attachment #6: fail-eth2.pcap --]
[-- Type: application/octet-stream, Size: 898 bytes --]

[-- Attachment #7: recover-eth2.pcap --]
[-- Type: application/octet-stream, Size: 2304 bytes --]

Re: r8169 ethernet bonding problems

[Tim Durack]

> A tcpdump trace 1) before promiscuous mode 2) after promiscuous mode 3)
> after link failure (i.e. failover does not work) would be welcome.

Okay, after fixing STP forwarding delays on the switch, bonding
failover now works, as long as member links are forced to promisc
mode.

Still Have a problem with links randomly failing to achieve link on
boot. This then requires a "modprobe -r r8169; modprobe r8169" cycle
to get link. Side effect is the mac address corruption I have
described previously.

Tim:>

Re: r8169 ethernet bonding problems

[Francois Romieu]

Tim Durack <tdurack@gmail.com> :
[...]
> Still Have a problem with links randomly failing to achieve link on
> boot. This then requires a "modprobe -r r8169; modprobe r8169" cycle
> to get link. Side effect is the mac address corruption I have
> described previously.

Can you try to set the MAC address by hand with the completely patched
kernel (i.e. including the mac-address-change patch) ?

Given the last report in http://bugzilla.kernel.org/show_bug.cgi?id=6032,
it should be straightforward with your Debian system.

No clear idea why your three cards are not handled the same though.

-- 
Ueimor

Anybody got a battery for my Ultra 10 ?

Re: r8169 ethernet bonding problems

[Tim Durack]

> Can you try to set the MAC address by hand with the completely patched
> kernel (i.e. including the mac-address-change patch) ?

Yup. Works fine. The randomly changing interface order is proving to
be a challenge!

> Given the last report in http://bugzilla.kernel.org/show_bug.cgi?id=6032,
> it should be straightforward with your Debian system.
>
> No clear idea why your three cards are not handled the same though.

Re: r8169 ethernet bonding problems

[Francois Romieu]

Tim Durack <tdurack@gmail.com> :
> >Can you try to set the MAC address by hand with the completely patched
> >kernel (i.e. including the mac-address-change patch) ?
> 
> Yup. Works fine. The randomly changing interface order is proving to
> be a challenge!

udev should be your friend to do a BUS/ID based rename.

-- 
Ueimor

Anybody got a battery for my Ultra 10 ?

Re: r8169 ethernet bonding problems

[Tim Durack]

> > Yup. Works fine. The randomly changing interface order is proving to
> > be a challenge!
>
> udev should be your friend to do a BUS/ID based rename.

True, but so far I have to force link addrs, force promisc mode, and
now add a udev work-around ;-|

Strange thing is, when the interfaces get initialised as eth0/1/2 in
order, they all have the correct macs. When they appear out of order
(eth5/1/2, eth5/6/1 etc), at least one or more of the macs are messed
up.

Once yesterday I saw them initialise in order, with correct macs. Have
not been able to repeat or figure out if there is some way of
triggering this correct behaviour.

Any more driver patches I can beta test?

Tim:>

Re: r8169 ethernet bonding problems

[Tim Durack]

On 5/2/07, Tim Durack <tdurack@gmail.com> wrote:
> > > Yup. Works fine. The randomly changing interface order is proving to
> > > be a challenge!
> >
> > udev should be your friend to do a BUS/ID based rename.

Okay, the udev workaround isn't too painful.

Would be great if I could do something like
ATTRS{address}="00:30:18:ab:cd:ef" and have that force the mac during
initialisation.

Tim:>

Re: r8169 ethernet bonding problems

[Francois Romieu]

Tim Durack <tdurack@gmail.com> :
[snip]

Can you try 2.6.22-rc5 +
http://www.fr.zoreil.com/people/francois/misc/20070619-2.6.22-rc5-r8169-test.patch

It does not seem to work too bad here (tested with 2.6.22-rc5 + patch):

# grep bonding /etc/modprobe.conf
alias bond0 bonding
options bonding miimon=1000 mode=balance-rr

$ dmesg
[    0.000000] Linux version 2.6.22-rc5 (root@localhost.localdomain) (gcc version 4.1.1 20070105 (Red Hat 4.1.1-51)) #9 Tue Jun 19 00:19:24 CEST 2007
[...]
[   35.516210] r8169 Gigabit Ethernet driver 2.2LK-NAPI loaded
[   35.516251] ACPI: PCI Interrupt 0000:01:00.0[A] -> GSI 17 (level, low) -> IRQ 17
[   35.516279] PCI: Setting latency timer of device 0000:01:00.0 to 64
[   35.516632] eth0: RTL8168b/8111b at 0xf8a08000, 00:13:8f:ea:b1:5d, XID 38000000 IRQ 17
[   35.523173] r8169 Gigabit Ethernet driver 2.2LK-NAPI loaded
[   35.523204] ACPI: PCI Interrupt 0000:03:01.0[A] -> GSI 22 (level, low) -> IRQ 20
[   35.523473] eth1: RTL8110s at 0xf8a0a800, 00:09:5b:bd:c1:a5, XID 04000000 IRQ 20
[   35.526654] r8169 Gigabit Ethernet driver 2.2LK-NAPI loaded
[   35.526683] ACPI: PCI Interrupt 0000:03:02.0[A] -> GSI 23 (level, low) -> IRQ 21
[   35.526940] eth2: RTL8110s at 0xf8a0cc00, 00:09:5b:bd:c1:a5, XID 04000000 IRQ 21
[   35.581378] USB Universal Host Controller Interface driver v3.0
[   35.582010] ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 23 (level, low) -> IRQ 21
[   35.582028] PCI: Setting latency timer of device 0000:00:1d.0 to 64
[...]
[   46.300540] Ethernet Channel Bonding Driver: v3.1.2 (January 20, 2007)
[   46.307134] bonding: MII link monitoring set to 1000 ms
[   50.515819] r8169: eth1: link up
[   50.519218] r8169: eth1: link up
[   50.539460] bonding: bond0: enslaving eth1 as an active interface with an up link.
[   50.656913] r8169: eth2: link up
[   50.660296] r8169: eth2: link up
[   50.679294] bonding: bond0: enslaving eth2 as an active interface with an up link.

$ ifconfig
bond0     Link encap:Ethernet  HWaddr 00:09:5B:BD:C1:A5
          inet addr:192.168.1.3  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::209:5bff:febd:c1a5/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:319 errors:0 dropped:0 overruns:0 frame:0
          TX packets:329 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:31583 (30.8 KiB)  TX bytes:30884 (30.1 KiB)

eth1      Link encap:Ethernet  HWaddr 00:09:5B:BD:C1:A5
          inet6 addr: fe80::209:5bff:febd:c1a5/64 Scope:Link
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:158 errors:0 dropped:0 overruns:0 frame:0
          TX packets:165 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:16228 (15.8 KiB)  TX bytes:15418 (15.0 KiB)
          Interrupt:20 Base address:0xa800

eth2      Link encap:Ethernet  HWaddr 00:09:5B:BD:C1:A5
          inet6 addr: fe80::209:5bff:febd:c1a5/64 Scope:Link
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:161 errors:0 dropped:0 overruns:0 frame:0
          TX packets:164 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:15355 (14.9 KiB)  TX bytes:15466 (15.1 KiB)
          Interrupt:21 Base address:0xcc00

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:560 (560.0 b)  TX bytes:560 (560.0 b)

# ethtool -S eth1
NIC statistics:
     tx_packets: 205
     rx_packets: 195
     tx_errors: 0
     rx_errors: 0
     rx_missed: 0
     align_errors: 0
     tx_single_collisions: 0
     tx_multi_collisions: 0
     unicast: 192
     broadcast: 1
     multicast: 2
     tx_aborted: 0
     tx_underrun: 0

# ethtool -S eth2
NIC statistics:
     tx_packets: 205
     rx_packets: 198
     tx_errors: 0
     rx_errors: 0
     rx_missed: 0
     align_errors: 0
     tx_single_collisions: 0
     tx_multi_collisions: 0
     unicast: 195
     broadcast: 2
     multicast: 1
     tx_aborted: 0
     tx_underrun: 0

Let's play gently with the cables:

PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.276 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.184 ms
64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=0.195 ms
64 bytes from 192.168.1.1: icmp_seq=4 ttl=64 time=0.173 ms
64 bytes from 192.168.1.1: icmp_seq=5 ttl=64 time=0.212 ms
64 bytes from 192.168.1.1: icmp_seq=6 ttl=64 time=0.173 ms
64 bytes from 192.168.1.1: icmp_seq=7 ttl=64 time=0.198 ms
64 bytes from 192.168.1.1: icmp_seq=8 ttl=64 time=0.173 ms
64 bytes from 192.168.1.1: icmp_seq=9 ttl=64 time=0.200 ms
64 bytes from 192.168.1.1: icmp_seq=10 ttl=64 time=0.199 ms
64 bytes from 192.168.1.1: icmp_seq=11 ttl=64 time=0.185 ms
64 bytes from 192.168.1.1: icmp_seq=12 ttl=64 time=0.237 ms
64 bytes from 192.168.1.1: icmp_seq=13 ttl=64 time=0.217 ms
64 bytes from 192.168.1.1: icmp_seq=14 ttl=64 time=0.187 ms
64 bytes from 192.168.1.1: icmp_seq=15 ttl=64 time=0.222 ms
64 bytes from 192.168.1.1: icmp_seq=16 ttl=64 time=0.193 ms
64 bytes from 192.168.1.1: icmp_seq=17 ttl=64 time=0.189 ms
64 bytes from 192.168.1.1: icmp_seq=18 ttl=64 time=0.218 ms
64 bytes from 192.168.1.1: icmp_seq=19 ttl=64 time=0.207 ms
64 bytes from 192.168.1.1: icmp_seq=20 ttl=64 time=0.200 ms
64 bytes from 192.168.1.1: icmp_seq=21 ttl=64 time=0.193 ms
64 bytes from 192.168.1.1: icmp_seq=22 ttl=64 time=0.176 ms
64 bytes from 192.168.1.1: icmp_seq=23 ttl=64 time=0.216 ms
64 bytes from 192.168.1.1: icmp_seq=24 ttl=64 time=0.194 ms
64 bytes from 192.168.1.1: icmp_seq=25 ttl=64 time=0.218 ms
64 bytes from 192.168.1.1: icmp_seq=26 ttl=64 time=0.192 ms
64 bytes from 192.168.1.1: icmp_seq=27 ttl=64 time=0.173 ms
64 bytes from 192.168.1.1: icmp_seq=28 ttl=64 time=0.214 ms
64 bytes from 192.168.1.1: icmp_seq=29 ttl=64 time=0.236 ms
64 bytes from 192.168.1.1: icmp_seq=30 ttl=64 time=0.188 ms
64 bytes from 192.168.1.1: icmp_seq=31 ttl=64 time=0.186 ms
64 bytes from 192.168.1.1: icmp_seq=32 ttl=64 time=0.164 ms
64 bytes from 192.168.1.1: icmp_seq=33 ttl=64 time=0.213 ms
64 bytes from 192.168.1.1: icmp_seq=34 ttl=64 time=0.180 ms
64 bytes from 192.168.1.1: icmp_seq=35 ttl=64 time=0.211 ms
64 bytes from 192.168.1.1: icmp_seq=36 ttl=64 time=0.178 ms
64 bytes from 192.168.1.1: icmp_seq=37 ttl=64 time=0.209 ms
64 bytes from 192.168.1.1: icmp_seq=38 ttl=64 time=0.201 ms
64 bytes from 192.168.1.1: icmp_seq=39 ttl=64 time=0.205 ms
64 bytes from 192.168.1.1: icmp_seq=40 ttl=64 time=0.187 ms
64 bytes from 192.168.1.1: icmp_seq=41 ttl=64 time=0.190 ms
64 bytes from 192.168.1.1: icmp_seq=42 ttl=64 time=0.262 ms
64 bytes from 192.168.1.1: icmp_seq=43 ttl=64 time=0.198 ms

[...]
[   65.364228] eth2: no IPv6 routers present
[   65.815462] eth1: no IPv6 routers present
[ 1197.478365] r8169: eth2: link down
[ 1197.679547] bonding: bond0: link status definitely down for interface eth2, disabling it
[ 1201.812241] r8169: eth2: link up
[ 1202.726842] bonding: bond0: link status definitely up for interface eth2.
[ 1208.448707] r8169: eth1: link down
[ 1208.779295] bonding: bond0: link status definitely down for interface eth1, disabling it
[ 1211.483899] r8169: eth1: link up
[ 1211.810965] bonding: bond0: link status definitely up for interface eth1.

--
Ueimor