* Toggling link state breaks network connectivity
@ 2017-06-12 13:22 Mason
2017-06-12 16:38 ` Florian Fainelli
2017-06-13 9:39 ` Matthias May
0 siblings, 2 replies; 14+ messages in thread
From: Mason @ 2017-06-12 13:22 UTC (permalink / raw)
To: netdev; +Cc: Florian Fainelli, Andrew Lunn, Mans Rullgard, Thibaud Cornic
Hello,
I am using the following drivers for Ethernet connectivity.
drivers/net/ethernet/aurora/nb8800.c
drivers/net/phy/at803x.c
Pulling the cable and plugging it back works as expected.
(I can ping both before and after.)
However, if I toggle the link state in software (using ip link set),
the board loses network connectivity.
# Statically assign IP address
ip addr add 172.27.64.77/18 brd 172.27.127.255 dev eth0
# Set link state to "up"
ip link set eth0 up
# ping -c 3 172.27.64.1 > /tmp/v1
PING 172.27.64.1 (172.27.64.1): 56 data bytes
64 bytes from 172.27.64.1: seq=0 ttl=64 time=18.321 ms
--- 172.27.64.1 ping statistics ---
3 packets transmitted, 1 packets received, 66% packet loss
round-trip min/avg/max = 18.321/18.321/18.321 ms
172.27.64.1 is a desktop system.
Running
% tcpdump -n -i eth1-boards ether host 00:16:e8:4d:7f:c4
on the desktop, I get:
15:01:45.187346 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
15:01:45.187359 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
15:01:45.194633 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41219, seq 0, length 64
15:01:45.194662 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41219, seq 0, length 64
15:01:50.198564 ARP, Request who-has 172.27.64.77 tell 172.27.64.1, length 28
15:01:50.205929 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41219, seq 1, length 64
15:01:50.205951 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41219, seq 1, length 64
15:01:50.213217 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41219, seq 2, length 64
15:01:50.213232 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41219, seq 2, length 64
15:01:51.198563 ARP, Request who-has 172.27.64.77 tell 172.27.64.1, length 28
15:01:51.209586 ARP, Reply 172.27.64.77 is-at 00:16:e8:4d:7f:c4, length 46
15:01:51.209598 ARP, Reply 172.27.64.77 is-at 00:16:e8:4d:7f:c4, length 46
Packet #1: the board asks for the desktop's MAC address
Packet #2: the desktop replies instantly
Packet #3: the board sends the first ping
Packet #4: the desktop replies instantly
Then the board goes quiet for a long time (why???)
Packet #5: the desktop asks for the board's MAC address (doesn't it have it already?)
Packet #6: this seems to unwedge the board, which sends the second ping
Packet #7: the desktop replies instantly
Packet #8: the board sends the third ping
Packet #9: the desktop replies instantly
Packet #10: the desktop asks again for the board's MAC address
Packet #11 and #12: the board answers twice (for the old and new requests?)
Some oddities, but it seems to work.
Now toggle the link state:
% ip link set eth0 down
% ip link set eth0 up
% ping -c 3 172.27.64.1 > /tmp/v2
PING 172.27.64.1 (172.27.64.1): 56 data bytes
--- 172.27.64.1 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
On the desktop, I see
15:14:03.900162 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
15:14:03.900175 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
15:14:05.017189 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
15:14:05.017200 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
15:14:06.030531 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
15:14:06.030541 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
So basically, the board is asking the desktop for its MAC address,
and the desktop is answering immediately. But the board doesn't seem
to be getting the replies... Any ideas, or words of wisdom, as they say?
Regards.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Toggling link state breaks network connectivity
2017-06-12 13:22 Toggling link state breaks network connectivity Mason
@ 2017-06-12 16:38 ` Florian Fainelli
2017-06-12 18:58 ` Mason
2017-06-13 15:07 ` Mason
2017-06-13 9:39 ` Matthias May
1 sibling, 2 replies; 14+ messages in thread
From: Florian Fainelli @ 2017-06-12 16:38 UTC (permalink / raw)
To: Mason, netdev; +Cc: Andrew Lunn, Mans Rullgard, Thibaud Cornic
On 06/12/2017 06:22 AM, Mason wrote:
> Hello,
>
> I am using the following drivers for Ethernet connectivity.
> drivers/net/ethernet/aurora/nb8800.c
> drivers/net/phy/at803x.c
>
> Pulling the cable and plugging it back works as expected.
> (I can ping both before and after.)
>
> However, if I toggle the link state in software (using ip link set),
> the board loses network connectivity.
>
> # Statically assign IP address
> ip addr add 172.27.64.77/18 brd 172.27.127.255 dev eth0
> # Set link state to "up"
> ip link set eth0 up
> # ping -c 3 172.27.64.1 > /tmp/v1
>
> PING 172.27.64.1 (172.27.64.1): 56 data bytes
> 64 bytes from 172.27.64.1: seq=0 ttl=64 time=18.321 ms
This delay seems abnormally long unless you are purposely introducing
delay (e.g: with cls_netem) or this is a really remote host, does not
seem to be based on your traces later on.
>
> --- 172.27.64.1 ping statistics ---
> 3 packets transmitted, 1 packets received, 66% packet loss
> round-trip min/avg/max = 18.321/18.321/18.321 ms
>
>
> 172.27.64.1 is a desktop system.
> Running
> % tcpdump -n -i eth1-boards ether host 00:16:e8:4d:7f:c4
> on the desktop, I get:
>
> 15:01:45.187346 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
> 15:01:45.187359 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
> 15:01:45.194633 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41219, seq 0, length 64
> 15:01:45.194662 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41219, seq 0, length 64
> 15:01:50.198564 ARP, Request who-has 172.27.64.77 tell 172.27.64.1, length 28
> 15:01:50.205929 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41219, seq 1, length 64
> 15:01:50.205951 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41219, seq 1, length 64
> 15:01:50.213217 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41219, seq 2, length 64
> 15:01:50.213232 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41219, seq 2, length 64
> 15:01:51.198563 ARP, Request who-has 172.27.64.77 tell 172.27.64.1, length 28
> 15:01:51.209586 ARP, Reply 172.27.64.77 is-at 00:16:e8:4d:7f:c4, length 46
> 15:01:51.209598 ARP, Reply 172.27.64.77 is-at 00:16:e8:4d:7f:c4, length 46
>
> Packet #1: the board asks for the desktop's MAC address
> Packet #2: the desktop replies instantly
> Packet #3: the board sends the first ping
> Packet #4: the desktop replies instantly
> Then the board goes quiet for a long time (why???)
> Packet #5: the desktop asks for the board's MAC address (doesn't it have it already?)
> Packet #6: this seems to unwedge the board, which sends the second ping
> Packet #7: the desktop replies instantly
> Packet #8: the board sends the third ping
> Packet #9: the desktop replies instantly
> Packet #10: the desktop asks again for the board's MAC address
> Packet #11 and #12: the board answers twice (for the old and new requests?)
>
> Some oddities, but it seems to work.
>
> Now toggle the link state:
>
> % ip link set eth0 down
> % ip link set eth0 up
> % ping -c 3 172.27.64.1 > /tmp/v2
>
> PING 172.27.64.1 (172.27.64.1): 56 data bytes
>
> --- 172.27.64.1 ping statistics ---
> 3 packets transmitted, 0 packets received, 100% packet loss
>
>
> On the desktop, I see
>
> 15:14:03.900162 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
> 15:14:03.900175 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
> 15:14:05.017189 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
> 15:14:05.017200 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
> 15:14:06.030531 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
> 15:14:06.030541 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
>
> So basically, the board is asking the desktop for its MAC address,
> and the desktop is answering immediately. But the board doesn't seem
> to be getting the replies... Any ideas, or words of wisdom, as they say?
- check the Ethernet MAC counters to see if there is packet loss, or
error, or both
- consult with your HW engineers for possible flaws in your
ndo_open/ndo_close paths and possible interactions with the MAC/PHY
clocks, or reset etc.
- see if your PHY needs a complete re-init after an up/down sequence and
if you are doing this properly
--
Florian
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Toggling link state breaks network connectivity
2017-06-12 16:38 ` Florian Fainelli
@ 2017-06-12 18:58 ` Mason
2017-06-13 15:07 ` Mason
1 sibling, 0 replies; 14+ messages in thread
From: Mason @ 2017-06-12 18:58 UTC (permalink / raw)
To: Florian Fainelli, netdev; +Cc: Andrew Lunn, Mans Rullgard, Thibaud Cornic
Hello Florian,
On 12/06/2017 18:38, Florian Fainelli wrote:
> On 06/12/2017 06:22 AM, Mason wrote:
>
>> I am using the following drivers for Ethernet connectivity.
>> drivers/net/ethernet/aurora/nb8800.c
>> drivers/net/phy/at803x.c
>>
>> Pulling the cable and plugging it back works as expected.
>> (I can ping both before and after.)
>>
>> However, if I toggle the link state in software (using ip link set),
>> the board loses network connectivity.
>>
>> # Statically assign IP address
>> ip addr add 172.27.64.77/18 brd 172.27.127.255 dev eth0
>> # Set link state to "up"
>> ip link set eth0 up
>> # ping -c 3 172.27.64.1 > /tmp/v1
>>
>> PING 172.27.64.1 (172.27.64.1): 56 data bytes
>> 64 bytes from 172.27.64.1: seq=0 ttl=64 time=18.321 ms
>
> This delay seems abnormally long unless you are purposely introducing
> delay (e.g: with cls_netem) or this is a really remote host, does not
> seem to be based on your traces later on.
172.27.64.1 and 172.27.64.77 are connected to the
same switch. Purely local traffic. It seems to me
that the ARP request/reply could explain the delay.
Start op at 45.187346
Receive ICMP echo reply at 45.194662
Hmmm, that's only 7 ms
>> 172.27.64.1 is a desktop system.
>> Running
>> % tcpdump -n -i eth1-boards ether host 00:16:e8:4d:7f:c4
>> on the desktop, I get:
>>
>> 15:01:45.187346 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
>> 15:01:45.187359 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
>> 15:01:45.194633 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41219, seq 0, length 64
>> 15:01:45.194662 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41219, seq 0, length 64
>> 15:01:50.198564 ARP, Request who-has 172.27.64.77 tell 172.27.64.1, length 28
>> 15:01:50.205929 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41219, seq 1, length 64
>> 15:01:50.205951 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41219, seq 1, length 64
>> 15:01:50.213217 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41219, seq 2, length 64
>> 15:01:50.213232 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41219, seq 2, length 64
>> 15:01:51.198563 ARP, Request who-has 172.27.64.77 tell 172.27.64.1, length 28
>> 15:01:51.209586 ARP, Reply 172.27.64.77 is-at 00:16:e8:4d:7f:c4, length 46
>> 15:01:51.209598 ARP, Reply 172.27.64.77 is-at 00:16:e8:4d:7f:c4, length 46
>>
>> Packet #1: the board asks for the desktop's MAC address
>> Packet #2: the desktop replies instantly
>> Packet #3: the board sends the first ping
>> Packet #4: the desktop replies instantly
>> Then the board goes quiet for a long time (why???)
>> Packet #5: the desktop asks for the board's MAC address (doesn't it have it already?)
>> Packet #6: this seems to unwedge the board, which sends the second ping
>> Packet #7: the desktop replies instantly
>> Packet #8: the board sends the third ping
>> Packet #9: the desktop replies instantly
>> Packet #10: the desktop asks again for the board's MAC address
>> Packet #11 and #12: the board answers twice (for the old and new requests?)
>>
>> Some oddities, but it seems to work.
>>
>> Now toggle the link state:
>>
>> % ip link set eth0 down
>> % ip link set eth0 up
>> % ping -c 3 172.27.64.1 > /tmp/v2
>>
>> PING 172.27.64.1 (172.27.64.1): 56 data bytes
>>
>> --- 172.27.64.1 ping statistics ---
>> 3 packets transmitted, 0 packets received, 100% packet loss
>>
>>
>> On the desktop, I see
>>
>> 15:14:03.900162 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
>> 15:14:03.900175 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
>> 15:14:05.017189 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
>> 15:14:05.017200 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
>> 15:14:06.030531 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
>> 15:14:06.030541 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
>>
>> So basically, the board is asking the desktop for its MAC address,
>> and the desktop is answering immediately. But the board doesn't seem
>> to be getting the replies... Any ideas, or words of wisdom, as they say?
>
> - check the Ethernet MAC counters to see if there is packet loss, or
> error, or both
I'll take a look, but I don't expect any packet loss
(LAN traffic on an idle switch).
> - consult with your HW engineers for possible flaws in your
> ndo_open/ndo_close paths and possible interactions with the MAC/PHY
> clocks, or reset etc.
(The HW engineers have no knowledge of Linux use-cases.)
The crazy thing is that I can use the same driver on the
previous chip, and I don't see this behavior... Will
retest tomorrow to be sure. What does change between
the two chips are a few clock frequencies though.
So maybe some race is now consistently lost on the
new chip...
> - see if your PHY needs a complete re-init after an up/down sequence and
> if you are doing this properly
Thanks for these suggestions. I'll take a closer look
tomorrow.
Regards.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Toggling link state breaks network connectivity
2017-06-12 16:38 ` Florian Fainelli
2017-06-12 18:58 ` Mason
@ 2017-06-13 15:07 ` Mason
2017-06-13 15:32 ` Mason
2017-06-13 15:36 ` Florian Fainelli
1 sibling, 2 replies; 14+ messages in thread
From: Mason @ 2017-06-13 15:07 UTC (permalink / raw)
To: Florian Fainelli, netdev; +Cc: Andrew Lunn, Mans Rullgard, Thibaud Cornic
On 12/06/2017 18:38, Florian Fainelli wrote:
> On 06/12/2017 06:22 AM, Mason wrote:
>
>> I am using the following drivers for Ethernet connectivity.
>> drivers/net/ethernet/aurora/nb8800.c
>> drivers/net/phy/at803x.c
>>
>> Pulling the cable and plugging it back works as expected.
>> (I can ping both before and after.)
>>
>> However, if I toggle the link state in software (using ip link set),
>> the board loses network connectivity.
>>
>> # Statically assign IP address
>> ip addr add 172.27.64.77/18 brd 172.27.127.255 dev eth0
>> # Set link state to "up"
>> ip link set eth0 up
>> # ping -c 3 172.27.64.1 > /tmp/v1
>>
>> PING 172.27.64.1 (172.27.64.1): 56 data bytes
>> 64 bytes from 172.27.64.1: seq=0 ttl=64 time=18.321 ms
>
> This delay seems abnormally long unless you are purposely introducing
> delay (e.g: with cls_netem) or this is a really remote host, does not
> seem to be based on your traces later on.
I think the delay is due to calling ping before the link
is actually up. For example, if I ping immediately after
setting the link up, the first 4 packets are lost.
PING 172.27.64.1 (172.27.64.1): 56 data bytes
64 bytes from 172.27.64.1: seq=4 ttl=64 time=0.235 ms
64 bytes from 172.27.64.1: seq=5 ttl=64 time=0.142 ms
64 bytes from 172.27.64.1: seq=6 ttl=64 time=0.110 ms
64 bytes from 172.27.64.1: seq=7 ttl=64 time=0.095 ms
64 bytes from 172.27.64.1: seq=8 ttl=64 time=0.139 ms
64 bytes from 172.27.64.1: seq=9 ttl=64 time=0.120 ms
--- 172.27.64.1 ping statistics ---
10 packets transmitted, 6 packets received, 40% packet loss
round-trip min/avg/max = 0.095/0.140/0.235 ms
>> So basically, the board is asking the desktop for its MAC address,
>> and the desktop is answering immediately. But the board doesn't seem
>> to be getting the replies... Any ideas, or words of wisdom, as they say?
>
> - check the Ethernet MAC counters to see if there is packet loss, or
> error, or both
>
> - consult with your HW engineers for possible flaws in your
> ndo_open/ndo_close paths and possible interactions with the MAC/PHY
> clocks, or reset etc.
>
> - see if your PHY needs a complete re-init after an up/down sequence and
> if you are doing this properly
I'm using the following test script:
ip addr add 172.27.64.77/18 brd 172.27.127.255 dev eth0
ip link set eth0 up
sleep 3 ## hopefully autoneg is complete
ethtool -S eth0 > /tmp/s0
ping -c 10 172.27.64.1 > /tmp/v1
ethtool -S eth0 > /tmp/s1
diff -U0 /tmp/s0 /tmp/s1
ip link set eth0 down
sleep 1
ip link set eth0 up
sleep 1
ethtool -S eth0 > /tmp/s0
ping -c 10 172.27.64.1 > /tmp/v2
ethtool -S eth0 > /tmp/s1
diff -U0 /tmp/s0 /tmp/s1
Testing with a generic PHY driver (no Atheros 8035 support built).
Apparently, ethtool doesn't report any packet loss or error.
First time:
# diff -U0 /tmp/s0 /tmp/s1
--- /tmp/s0
+++ /tmp/s1
@@ -2,2 +2,2 @@
- rx_bytes_ok: 0
- rx_frames_ok: 0
+ rx_bytes_ok: 1084
+ rx_frames_ok: 11
@@ -6,2 +6,2 @@
- rx_64_byte_frames: 0
- rx_127_byte_frames: 0
+ rx_64_byte_frames: 1
+ rx_127_byte_frames: 10
@@ -22,6 +22,6 @@
- rx_bytes: 0
- rx_frames: 0
- tx_bytes_ok: 0
- tx_frames_ok: 0
- tx_64_byte_frames: 0
- tx_127_byte_frames: 0
+ rx_bytes: 1084
+ rx_frames: 11
+ tx_bytes_ok: 1084
+ tx_frames_ok: 11
+ tx_64_byte_frames: 1
+ tx_127_byte_frames: 10
@@ -33 +33 @@
- tx_broadcast_frames: 0
+ tx_broadcast_frames: 1
@@ -43,2 +43,2 @@
- tx_bytes: 0
- tx_frames: 0
+ tx_bytes: 1084
+ tx_frames: 11
Second time:
# diff -U0 /tmp/s0 /tmp/s1
--- /tmp/s0
+++ /tmp/s1
@@ -2,2 +2,2 @@
- rx_bytes_ok: 1276
- rx_frames_ok: 14
+ rx_bytes_ok: 1779
+ rx_frames_ok: 19
@@ -6 +6 @@
- rx_64_byte_frames: 4
+ rx_64_byte_frames: 8
@@ -8 +8 @@
- rx_255_byte_frames: 0
+ rx_255_byte_frames: 1
@@ -14 +14 @@
- rx_broadcast_frames: 0
+ rx_broadcast_frames: 1
@@ -22,5 +22,5 @@
- rx_bytes: 1276
- rx_frames: 14
- tx_bytes_ok: 1276
- tx_frames_ok: 14
- tx_64_byte_frames: 4
+ rx_bytes: 1779
+ rx_frames: 19
+ tx_bytes_ok: 1724
+ tx_frames_ok: 21
+ tx_64_byte_frames: 11
@@ -33 +33 @@
- tx_broadcast_frames: 1
+ tx_broadcast_frames: 8
@@ -43,2 +43,2 @@
- tx_bytes: 1276
- tx_frames: 14
+ tx_bytes: 1724
+ tx_frames: 21
I did note something that seems important.
If I toggle the link state in software, then connectivity breaks.
If I unplug the ethernet cable, and replug, connectivity remains.
The difference is that plugging/unplugging doesn't call the
.ndo_stop callback. But 'ip link set eth0 down' does call it.
Should the .ndo_stop callback be symmetric to the .ndo_open callback?
In other words, should .ndo_open(); .ndo_stop(); be a NOP?
Regards.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Toggling link state breaks network connectivity
2017-06-13 15:07 ` Mason
@ 2017-06-13 15:32 ` Mason
2017-06-13 15:36 ` Florian Fainelli
1 sibling, 0 replies; 14+ messages in thread
From: Mason @ 2017-06-13 15:32 UTC (permalink / raw)
To: Florian Fainelli, netdev
Cc: Andrew Lunn, Mans Rullgard, Thibaud Cornic, Matthias May,
Zefir Kurtisi
On 13/06/2017 17:07, Mason wrote:
> I did note something that seems important.
>
> If I toggle the link state in software, then connectivity breaks.
>
> If I unplug the ethernet cable, and replug, connectivity remains.
>
> The difference is that plugging/unplugging doesn't call the
> .ndo_stop callback. But 'ip link set eth0 down' does call it.
>
> Should the .ndo_stop callback be symmetric to the .ndo_open callback?
> In other words, should .ndo_open(); .ndo_stop(); be a NOP?
I changed the ndo_open callback to a wrapper that calls:
nb8800_open(dev);
nb8800_stop(dev);
nb8800_open(dev);
With this change, connectivity is broken from the start.
Valid ARP requests are correctly *sent* but the corresponding
ARP replies are not received.
I'm hoping this limits the scope of what needs to be investigated.
Regards.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Toggling link state breaks network connectivity
2017-06-13 15:07 ` Mason
2017-06-13 15:32 ` Mason
@ 2017-06-13 15:36 ` Florian Fainelli
2017-06-13 15:47 ` Mason
1 sibling, 1 reply; 14+ messages in thread
From: Florian Fainelli @ 2017-06-13 15:36 UTC (permalink / raw)
To: Mason, netdev; +Cc: Andrew Lunn, Mans Rullgard, Thibaud Cornic
On 06/13/2017 08:07 AM, Mason wrote:> I did note something that seems
important.
>
> If I toggle the link state in software, then connectivity breaks.
>
> If I unplug the ethernet cable, and replug, connectivity remains.
What does that actually mean? If you disconnect the cable a link state
should be notified and another link state should be notified, even if
that happens faster than the PHYLIB polling time (1s).
>
> The difference is that plugging/unplugging doesn't call the
> .ndo_stop callback. But 'ip link set eth0 down' does call it.
>
> Should the .ndo_stop callback be symmetric to the .ndo_open callback?
Yes obviously.
> In other words, should .ndo_open(); .ndo_stop(); be a NOP?
--
Florian
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Toggling link state breaks network connectivity
2017-06-13 15:36 ` Florian Fainelli
@ 2017-06-13 15:47 ` Mason
2017-06-13 15:50 ` Florian Fainelli
0 siblings, 1 reply; 14+ messages in thread
From: Mason @ 2017-06-13 15:47 UTC (permalink / raw)
To: Florian Fainelli, netdev; +Cc: Andrew Lunn, Mans Rullgard, Thibaud Cornic
On 13/06/2017 17:36, Florian Fainelli wrote:
> On 06/13/2017 08:07 AM, Mason wrote:
>
>> I did note something that seems important.
>> If I toggle the link state in software, then connectivity breaks.
>> If I unplug the ethernet cable, and replug, connectivity remains.
>
> What does that actually mean? If you disconnect the cable a link state
> should be notified and another link state should be notified, even if
> that happens faster than the PHYLIB polling time (1s).
Sorry for being unclear.
If I unplug/replug the cable, ping still works afterward.
(Packet RX appears to be *not* wedged)
If I toggle the link state in SW (set link down/set link up),
I can no longer ping afterward.
(Packet RX appears to be wedged, so ARP replies are not seen)
Maybe the two experiments are too unrelated to consider
the different results relevant in any way?
Regards.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Toggling link state breaks network connectivity
2017-06-13 15:47 ` Mason
@ 2017-06-13 15:50 ` Florian Fainelli
2017-06-14 14:49 ` Mason
0 siblings, 1 reply; 14+ messages in thread
From: Florian Fainelli @ 2017-06-13 15:50 UTC (permalink / raw)
To: Mason, netdev; +Cc: Andrew Lunn, Mans Rullgard, Thibaud Cornic
On 06/13/2017 08:47 AM, Mason wrote:
> On 13/06/2017 17:36, Florian Fainelli wrote:
>
>> On 06/13/2017 08:07 AM, Mason wrote:
>>
>>> I did note something that seems important.
>>> If I toggle the link state in software, then connectivity breaks.
>>> If I unplug the ethernet cable, and replug, connectivity remains.
>>
>> What does that actually mean? If you disconnect the cable a link state
>> should be notified and another link state should be notified, even if
>> that happens faster than the PHYLIB polling time (1s).
>
> Sorry for being unclear.
>
> If I unplug/replug the cable, ping still works afterward.
> (Packet RX appears to be *not* wedged)
>
> If I toggle the link state in SW (set link down/set link up),
> I can no longer ping afterward.
> (Packet RX appears to be wedged, so ARP replies are not seen)
>
> Maybe the two experiments are too unrelated to consider
> the different results relevant in any way?
No, they are not, this really tells you that whatever your ndo_open()
and ndo_stop() functions do, they are currently broken with your
particular HW. I have seen numerous problems on some of our older
platforms using bcmgenet where the PHY needs to be reset *before* any
MAC activity occurs, and any MAC activity even qualifies as a reset of
the MAC itself, we can perform a PHY reset there because the PHY is
internal, that may not be an option, and your HW is different anyway.
--
Florian
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Toggling link state breaks network connectivity
2017-06-13 15:50 ` Florian Fainelli
@ 2017-06-14 14:49 ` Mason
2017-06-15 10:19 ` Måns Rullgård
0 siblings, 1 reply; 14+ messages in thread
From: Mason @ 2017-06-14 14:49 UTC (permalink / raw)
To: Florian Fainelli, netdev; +Cc: Andrew Lunn, Mans Rullgard, Thibaud Cornic
On 13/06/2017 17:50, Florian Fainelli wrote:
> On 06/13/2017 08:47 AM, Mason wrote:
>
>> If I unplug/replug the cable, ping still works afterward.
>> (Packet RX appears to be *not* wedged)
>>
>> If I toggle the link state in SW (set link down/set link up),
>> I can no longer ping afterward.
>> (Packet RX appears to be wedged, so ARP replies are not seen)
>>
>> Maybe the two experiments are too unrelated to consider
>> the different results relevant in any way?
>
> No, they are not, this really tells you that whatever your ndo_open()
> and ndo_stop() functions do, they are currently broken with your
> particular HW. I have seen numerous problems on some of our older
> platforms using bcmgenet where the PHY needs to be reset *before* any
> MAC activity occurs, and any MAC activity even qualifies as a reset of
> the MAC itself, we can perform a PHY reset there because the PHY is
> internal, that may not be an option, and your HW is different anyway.
I have logged *all* register writes, to compare state after probe,
after open, and after close. I do see a few odd values...
I'm still deep in the maze, but I may find an exit eventually.
What state is probe supposed to leave the HW in?
What state is open supposed to leave the HW in?
I'm thinking that I could reset the entire block in close.
Regards.
Found ethernet phy : AR8035
[ 0.852746] libphy: Fixed MDIO Bus: probed
[ 0.856975] ++ETH++ gw8 0xf0026424 0x00 # ETH_SW_RESET
[ 0.870942] ++ETH++ gw8 0xf0026424 0x01 # ETH_SW_RESET
[ 0.874894] ++ETH++ gw16 0xf0026420 0x0050 # ETH_MDIO_CLK_DIV
[ 0.879101] libphy: nb8800-mii: probed
[ 0.885005] ++ETH++ gw8 0xf0026000 0x1c # ETH_MAC_CORE_TRANSMIT_CONTROL
[ 0.888967] ++ETH++ gw8 0xf0026001 0x05 # ETH_MAC_CORE_TRANSMIT_CONTROL
[ 0.892911] ++ETH++ gw8 0xf0026004 0x22 # ETH_MAC_CORE_RECEIVE_CONTROL
[ 0.896855] ++ETH++ gw8 0xf0026008 0x04 # ETH_MAC_CORE_RANDOM_SEED
[ 0.900815] ++ETH++ gw8 0xf0026014 0x0c # ETH_MAC_CORE_TRANSMIT_SINGLE_DEFERRAL
[ 0.904758] ++ETH++ gw8 0xf0026051 0x00 # ETH_MAC_CORE_THRESHOLDS
[ 0.908700] ++ETH++ gw8 0xf0026052 0xff # ETH_MAC_CORE_THRESHOLDS
[ 0.912643] ++ETH++ gw8 0xf0026054 0x40 # ETH_MAC_CORE_BUFFER_SIZE_TX_AND_ETH_MAC_CORE_FIFO_CONTROL
[ 0.916586] ++ETH++ gw32 0xf0026100 0x000002fe # TRANSMIT_CHANNEL_CONTROL
[ 0.921054] ++ETH++ gw32 0xf0026118 0x003cc4a4 # TRANSMIT_INTERRUPT_TIME
[ 0.925521] ++ETH++ gw32 0xf0026200 0x000002fe # RECEIVE_CHANNEL_CONTROL
[ 0.929990] ++ETH++ gw32 0xf0026218 0x00004dc8 # RECEIVE_INTERRUPT_TIME
[ 0.934456] ++ETH++ gw8 0xf0026060 0x00 # ETH_MAC_CORE_PAUSE_QUANTA
[ 0.938398] ++ETH++ gw8 0xf0026061 0xc3 # ETH_MAC_CORE_PAUSE_QUANTA
[ 0.942338] ++ETH++ gw8 0xf002602e 0x00 # ETH_MAC_CORE_MULTICAST_ADDRESS1
[ 0.946283] ++ETH++ gw8 0xf0026400 0x01 # ETH_GPIO_PAD_MODE
[ 0.950226] ++ETH++ gw32 0xf0026200 0x0000028e # RECEIVE_CHANNEL_CONTROL
[ 0.954695] ++ETH++ gw8 0xf002606a 0x00 # ETH_MAC_CORE_SOURCE_ADDRESS0
[ 0.958638] ++ETH++ gw8 0xf002606b 0x16 # ETH_MAC_CORE_SOURCE_ADDRESS0
[ 0.962581] ++ETH++ gw8 0xf002606c 0xe8 # ETH_MAC_CORE_SOURCE_ADDRESS1
[ 0.966523] ++ETH++ gw8 0xf002606d 0x4d # ETH_MAC_CORE_SOURCE_ADDRESS1
[ 0.970465] ++ETH++ gw8 0xf002606e 0x7f # ETH_MAC_CORE_SOURCE_ADDRESS1
[ 0.974408] ++ETH++ gw8 0xf002606f 0xc4 # ETH_MAC_CORE_SOURCE_ADDRESS1
[ 0.978349] ++ETH++ gw8 0xf002603c 0x00 # ETH_MAC_CORE_UNICAST_ADDRESS0
[ 0.982291] ++ETH++ gw8 0xf002603d 0x16 # ETH_MAC_CORE_UNICAST_ADDRESS0
[ 0.986234] ++ETH++ gw8 0xf002603e 0xe8 # ETH_MAC_CORE_UNICAST_ADDRESS0
[ 0.990176] ++ETH++ gw8 0xf002603f 0x4d # ETH_MAC_CORE_UNICAST_ADDRESS0
[ 0.994118] ++ETH++ gw8 0xf0026040 0x7f # ETH_MAC_CORE_UNICAST_ADDRESS1
[ 0.998060] ++ETH++ gw8 0xf0026041 0xc4 # ETH_MAC_CORE_UNICAST_ADDRESS1
[ 1.002435] nb8800 26000.ethernet eth0: MAC address 00:16:e8:4d:7f:c4
# ip link set eth0 up
[ 33.202058] ENTER nb8800_open
[ 33.205134] ++ETH++ gw32 0xf0026204 0x0000000f # RECEIVE_CHANNEL_STATUS
[ 33.209610] ++ETH++ gw32 0xf0026104 0x0000000f # TRANSMIT_CHANNEL_STATUS
[ 33.214626] ++ETH++ gw32 0xf002620c 0x9de36000 # RECEIVE_DESCRIPTOR_ADDRESS
[ 33.219138] ++ETH++ gw8 0xf0026004 0x23 # ETH_MAC_CORE_RECEIVE_CONTROL
[ 33.223097] ++ETH++ gw8 0xf0026000 0x1d # ETH_MAC_CORE_TRANSMIT_CONTROL
[ 33.227054] nb8800_mdio_write: reg=0 val=8000 # by genphy_soft_reset()
[ 33.339579] nb8800_mdio_write: reg=0 val=0000 # by at803x_resume()
[ 33.370911] nb8800_mdio_write: reg=18 val=ec00 # by at803x_config_intr()
[ 33.384357] ++ETH++ gw32 0xf0026200 0x0000028f # RECEIVE_CHANNEL_CONTROL
[ 33.393330] EXIT nb8800_open
[ 33.393332] ENTER nb8800_stop
[ 33.421731] nb8800_mdio_write: reg=9 val=0200 # by genphy_config_aneg()
[ 33.444062] nb8800_mdio_write: reg=0 val=1200 # by genphy_restart_aneg()
[ 33.466411] nb8800_mdio_write: reg=18 val=0000 # by at803x_config_intr() from phy_stop()
[ 33.488859] ++ETH++ gw8 0xf0026004 0x03 # ETH_MAC_CORE_RECEIVE_CONTROL
[ 33.492805] ++ETH++ gw8 0xf0026044 0x04 # ETH_MAC_CORE_MAC_MODE *WHY ??*
[ 33.496748] ++ETH++ gw32 0xf002610c 0x9dcac000 \
[ 33.501217] ++ETH++ gw32 0xf0026100 0x000002ff |
[ 33.505750] ++ETH++ gw32 0xf0026104 0x00000004 | IRQ?
[ 33.510213] ++ETH++ gw32 0xf0026204 0x00000004 |
[ 33.514674] ++ETH++ gw32 0xf0026218 0x003cc4a4 |
[ 33.519146] ++ETH++ gw32 0xf0026104 0x00000001 /
[ 33.605719] ++ETH++ gw32 0xf002610c 0x9dcac000 \
[ 33.610187] ++ETH++ gw32 0xf0026100 0x000002ff |
[ 33.614717] ++ETH++ gw32 0xf0026104 0x00000004 | ANOTHER IRQ?
[ 33.619178] ++ETH++ gw32 0xf0026204 0x00000004 |
[ 33.623637] ++ETH++ gw32 0xf0026218 0x003cc4a4 |
[ 33.628106] ++ETH++ gw32 0xf0026104 0x00000001 /
[ 33.714689] ++ETH++ gw32 0xf002610c 0x9dcac000 |
[ 33.719156] ++ETH++ gw32 0xf0026100 0x000002ff | ANOTHER IRQ?
[ 33.723686] ++ETH++ gw32 0xf0026104 0x00000004 |
[ 33.724625] ++ETH++ gw8 0xf0026004 0x23 # ETH_MAC_CORE_RECEIVE_CONTROL
[ 33.724627] ++ETH++ gw8 0xf0026044 0x00 # ETH_MAC_CORE_MAC_MODE *AGAIN ??*
[ 33.724640] ++ETH++ gw32 0xf002620c 0x9de36000
[ 33.724642] ++ETH++ gw8 0xf0026004 0x22 # ETH_MAC_CORE_RECEIVE_CONTROL AGAIN
[ 33.724644] ++ETH++ gw8 0xf0026000 0x1c # ETH_MAC_CORE_TRANSMIT_CONTROL
[ 33.724690] nb8800_mdio_write: reg=18 val=0000 # by at803x_config_intr() from phy_disable_interrupts()
[ 33.724913] nb8800_mdio_write: reg=0 val=1800 # by at803x_suspend() from phy_suspend()
[ 33.819562] ++ETH++ gw32 0xf0026204 0x00000004
[ 33.824021] ++ETH++ gw32 0xf0026218 0x003cc4a4
[ 33.828518] EXIT nb8800_stop
[ 33.831505] ENTER nb8800_open
[ 33.834491] ++ETH++ gw32 0xf0026204 0x0000000f
[ 33.838959] ++ETH++ gw32 0xf0026104 0x0000000f
[ 33.843970] ++ETH++ gw32 0xf002620c 0x9dc1e000
[ 33.848449] ++ETH++ gw8 0xf0026004 0x23
[ 33.852396] ++ETH++ gw8 0xf0026000 0x1d
[ 33.856343] nb8800_mdio_write: reg=0 val=8000
[ 33.969686] nb8800_mdio_write: reg=0 val=0000
[ 34.001128] nb8800_mdio_write: reg=18 val=ec00
[ 34.014631] ++ETH++ gw32 0xf0026200 0x06100a8f # RECEIVE_CHANNEL_CONTROL
[ 34.023592] EXIT nb8800_open
[ 34.023760] nb8800_mdio_write: reg=9 val=0200
[ 34.023890] nb8800_mdio_write: reg=0 val=1200
[ 34.023991] nb8800_mdio_write: reg=18 val=0000
[ 34.024122] nb8800_mdio_write: reg=18 val=ec00
[ 34.200493] ++ETH++ gw8 0xf002602e 0x00
[ 34.204473] ++ETH++ gw8 0xf002602e 0x00
[ 34.208418] ++ETH++ gw8 0xf0026028 0x01
[ 34.212363] ++ETH++ gw8 0xf0026029 0x00
[ 34.216305] ++ETH++ gw8 0xf002602a 0x5e
[ 34.220248] ++ETH++ gw8 0xf002602b 0x00
[ 34.224190] ++ETH++ gw8 0xf002602c 0x00
[ 34.228132] ++ETH++ gw8 0xf002602d 0x01
[ 34.232073] ++ETH++ gw8 0xf002602e 0xff
[ 36.738162] nb8800_mdio_write: reg=18 val=0000
[ 36.769615] nb8800_mdio_write: reg=18 val=ec00
[ 36.836941] ++ETH++ gw8 0xf0026050 0x01
[ 36.840884] ++ETH++ gw8 0xf002601c 0xff
[ 36.844829] ++ETH++ gw8 0xf0026044 0x81
[ 36.848780] nb8800 26000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[ 36.865544] nb8800_mdio_write: reg=18 val=ec00
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Toggling link state breaks network connectivity
2017-06-14 14:49 ` Mason
@ 2017-06-15 10:19 ` Måns Rullgård
2017-06-16 14:18 ` Mason
0 siblings, 1 reply; 14+ messages in thread
From: Måns Rullgård @ 2017-06-15 10:19 UTC (permalink / raw)
To: Mason; +Cc: Florian Fainelli, netdev, Andrew Lunn, Thibaud Cornic
Mason <slash.tmp@free.fr> writes:
> [ 33.393332] ENTER nb8800_stop
> [ 33.421731] nb8800_mdio_write: reg=9 val=0200 # by genphy_config_aneg()
> [ 33.444062] nb8800_mdio_write: reg=0 val=1200 # by genphy_restart_aneg()
> [ 33.466411] nb8800_mdio_write: reg=18 val=0000 # by at803x_config_intr() from phy_stop()
Those are from the phy_stop() call.
The following is the ridiculous dance required to stop rx dma in
nb8800_dma_stop():
> [ 33.488859] ++ETH++ gw8 0xf0026004 0x03 # ETH_MAC_CORE_RECEIVE_CONTROL
Disable address filter.
> [ 33.492805] ++ETH++ gw8 0xf0026044 0x04 # ETH_MAC_CORE_MAC_MODE *WHY ??*
Enable loopback.
> [ 33.496748] ++ETH++ gw32 0xf002610c 0x9dcac000 \
> [ 33.501217] ++ETH++ gw32 0xf0026100 0x000002ff |
Send frame.
> [ 33.505750] ++ETH++ gw32 0xf0026104 0x00000004 | IRQ?
> [ 33.510213] ++ETH++ gw32 0xf0026204 0x00000004 |
> [ 33.514674] ++ETH++ gw32 0xf0026218 0x003cc4a4 |
> [ 33.519146] ++ETH++ gw32 0xf0026104 0x00000001 /
Handle resulting interrupt(s).
> [ 33.605719] ++ETH++ gw32 0xf002610c 0x9dcac000 \
> [ 33.610187] ++ETH++ gw32 0xf0026100 0x000002ff |
Send frame.
> [ 33.614717] ++ETH++ gw32 0xf0026104 0x00000004 | ANOTHER IRQ?
> [ 33.619178] ++ETH++ gw32 0xf0026204 0x00000004 |
> [ 33.623637] ++ETH++ gw32 0xf0026218 0x003cc4a4 |
> [ 33.628106] ++ETH++ gw32 0xf0026104 0x00000001 /
Handle interrupt(s).
> [ 33.714689] ++ETH++ gw32 0xf002610c 0x9dcac000 |
> [ 33.719156] ++ETH++ gw32 0xf0026100 0x000002ff | ANOTHER IRQ?
Send frame.
> [ 33.723686] ++ETH++ gw32 0xf0026104 0x00000004 |
Handle tx interrupt.
> [ 33.724625] ++ETH++ gw8 0xf0026004 0x23 # ETH_MAC_CORE_RECEIVE_CONTROL
Enable address filter.
> [ 33.724627] ++ETH++ gw8 0xf0026044 0x00 # ETH_MAC_CORE_MAC_MODE *AGAIN ??*
Disable loopback.
> [ 33.724640] ++ETH++ gw32 0xf002620c 0x9de36000
>From nb8800_dma_reset().
> [ 33.724642] ++ETH++ gw8 0xf0026004 0x22 # ETH_MAC_CORE_RECEIVE_CONTROL AGAIN
Disable rx.
> [ 33.724644] ++ETH++ gw8 0xf0026000 0x1c # ETH_MAC_CORE_TRANSMIT_CONTROL
Disable tx.
> [ 33.724690] nb8800_mdio_write: reg=18 val=0000 # by at803x_config_intr() from phy_disable_interrupts()
> [ 33.724913] nb8800_mdio_write: reg=0 val=1800 # by at803x_suspend() from phy_suspend()
> [ 33.819562] ++ETH++ gw32 0xf0026204 0x00000004
> [ 33.824021] ++ETH++ gw32 0xf0026218 0x003cc4a4
Handle rx interrupt, probably from the last loopback frame sent earlier.
> [ 33.828518] EXIT nb8800_stop
Now I did notice one thing. When the interrupts from the loopback
frames are handled, the rx interrupt is all but disabled for NAPI poll
mode. Of course, NAPI isn't active, so the rx interrupt never gets
re-enabled. We should probably do this in ndo_open just to be sure.
Try this patch:
diff --git a/drivers/net/ethernet/aurora/nb8800.c b/drivers/net/ethernet/aurora/nb8800.c
index e078d8da978c..f968468076b9 100644
--- a/drivers/net/ethernet/aurora/nb8800.c
+++ b/drivers/net/ethernet/aurora/nb8800.c
@@ -969,6 +969,7 @@ static int nb8800_open(struct net_device *dev)
if (err)
goto err_free_dma;
+ nb8800_writel(priv, NB8800_RX_ITR, priv->rx_itr_irq);
nb8800_mac_rx(dev, true);
nb8800_mac_tx(dev, true);
--
Måns Rullgård
^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: Toggling link state breaks network connectivity
2017-06-15 10:19 ` Måns Rullgård
@ 2017-06-16 14:18 ` Mason
0 siblings, 0 replies; 14+ messages in thread
From: Mason @ 2017-06-16 14:18 UTC (permalink / raw)
To: Mans Rullgard; +Cc: Florian Fainelli, netdev, Andrew Lunn, Thibaud Cornic
On 15/06/2017 12:19, Måns Rullgård wrote:
> Now I did notice one thing. When the interrupts from the loopback
> frames are handled, the rx interrupt is all but disabled for NAPI poll
> mode. Of course, NAPI isn't active, so the rx interrupt never gets
> re-enabled. We should probably do this in ndo_open just to be sure.
I have tried everything I could think of (whether reasonable,
or just random). Setting the link down always hoses the board's
network connectivity, 100% reproducible on Vantage 1218.
The single HW difference that I'm aware of is that the Atheros
PHY used to be clocked at 25 MHz (I think) on 1172 and 1191
boards, but it is clocked at 100 MHz on this 1218 board.
I'll try tweaking this setting.
For the time being, my test script is:
ip addr add 172.27.64.77/18 brd 172.27.127.255 dev eth0
ip link set eth0 up
sleep 6 ## autoneg should be complete
ntpdate 172.27.64.1
sleep 1
date +%T && ping -c 3 172.27.64.1 > /tmp/ping_before
sleep 1
ip link set eth0 down
sleep 2
ip link set eth0 up
sleep 5
date +%T && ping -c 3 172.27.64.1 > /tmp/ping_after
# cat /tmp/ping_before
PING 172.27.64.1 (172.27.64.1): 56 data bytes
64 bytes from 172.27.64.1: seq=0 ttl=64 time=31.442 ms
64 bytes from 172.27.64.1: seq=1 ttl=64 time=31.359 ms
64 bytes from 172.27.64.1: seq=2 ttl=64 time=31.379 ms
--- 172.27.64.1 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 31.359/31.393/31.442 ms
=> These RTT numbers are crazy high for a LAN connection
through a switch. (I get 0.250 ms from my desktop.)
=> What the server saw was:
15:23:30.400066 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41987, seq 0, length 64
15:23:30.400094 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41987, seq 0, length 64
15:23:31.431479 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41987, seq 1, length 64
15:23:31.431500 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41987, seq 1, length 64
15:23:32.462844 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41987, seq 2, length 64
15:23:32.462867 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41987, seq 2, length 64
# cat /tmp/ping_after
PING 172.27.64.1 (172.27.64.1): 56 data bytes
--- 172.27.64.1 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
=> What the server saw was:
15:23:41.180068 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
15:23:41.180081 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
15:23:42.194409 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
15:23:42.194419 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
15:23:43.207700 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
15:23:43.207710 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
Below are the "raw" logs (of what I thought might be useful)
Regards.
[ 0.898538] libphy: Fixed MDIO Bus: probed
[ 0.902734] ENTER nb8800_probe
[ 0.905863] ++ETH++ gw8 0xf0026424 0x00
[ 0.919601] ++ETH++ gw8 0xf0026424 0x01
[ 0.923550] ++ETH++ gw16 0xf0026420 0x0050
[ 0.927749] libphy: nb8800-mii: probed
[ 0.933736] ENTER nb8800_hw_init
[ 0.936991] ++ETH++ gw8 0xf0026000 0x1c
[ 0.940952] ++ETH++ gw8 0xf0026001 0x05
[ 0.944897] ++ETH++ gw8 0xf0026004 0x22
[ 0.948839] ++ETH++ gw8 0xf0026008 0x04
[ 0.952792] ++ETH++ gw8 0xf0026014 0x0c
[ 0.956740] ++ETH++ gw8 0xf0026051 0x00
[ 0.960693] ++ETH++ gw8 0xf0026052 0xff
[ 0.964637] ++ETH++ gw8 0xf0026054 0x40
[ 0.968579] ++ETH++ gw32 0xf0026100 0x000002fe
[ 0.973058] ++ETH++ gw32 0xf0026118 0x003cc4a4
[ 0.977527] ++ETH++ gw32 0xf0026200 0x000002fe
[ 0.981994] ++ETH++ gw32 0xf0026218 0x00004dc8
[ 0.986473] ++ETH++ gw8 0xf002602e 0x00
[ 0.990417] EXIT nb8800_hw_init
[ 0.993576] ++ETH++ gw8 0xf0026400 0x01
[ 0.997518] ++ETH++ gw32 0xf0026200 0x0000028e
[ 1.001988] ++ETH++ gw8 0xf002606a 0x00
[ 1.005930] ++ETH++ gw8 0xf002606b 0x16
[ 1.009873] ++ETH++ gw8 0xf002606c 0xe8
[ 1.013815] ++ETH++ gw8 0xf002606d 0x4d
[ 1.017757] ++ETH++ gw8 0xf002606e 0x7f
[ 1.021699] ++ETH++ gw8 0xf002606f 0xc4
[ 1.025641] ++ETH++ gw8 0xf002603c 0x00
[ 1.029584] ++ETH++ gw8 0xf002603d 0x16
[ 1.033526] ++ETH++ gw8 0xf002603e 0xe8
[ 1.037468] ++ETH++ gw8 0xf002603f 0x4d
[ 1.041411] ++ETH++ gw8 0xf0026040 0x7f
[ 1.045354] ++ETH++ gw8 0xf0026041 0xc4
[ 1.049772] nb8800 26000.ethernet eth0: MAC address 00:16:e8:4d:7f:c4
[ 1.056259] EXIT nb8800_probe
# ip addr add 172.27.64.77/18 brd 172.27.127.255 dev eth0
# ip link set eth0 up
[ 16.172093] ENTER nb8800_open
[ 16.175096] ++ETH++ gw32 0xf0026204 0x0000000f
[ 16.179574] ++ETH++ gw32 0xf0026104 0x0000000f
[ 16.184063] ENTER nb8800_dma_init
[ 16.188000] ++ETH++ gw32 0xf002620c 0x9dc4c000
[ 16.192478] EXIT nb8800_dma_init
[ 16.195851] ++ETH++ gw32 0xf0026218 0x00004dc8
[ 16.200334] ++ETH++ gw8 0xf0026004 0x23
[ 16.204280] ++ETH++ gw8 0xf0026000 0x1d
[ 16.208234] nb8800_mdio_write: reg=0x00 val=8000
[ 16.269621] nb8800_mdio_read : reg=0x00 val=0000
[ 16.282954] nb8800_mdio_read : reg=0x01 val=7949
[ 16.287639] nb8800_mdio_read : reg=0x0f val=a000
[ 16.292324] nb8800_mdio_read : reg=0x00 val=0000
[ 16.296968] nb8800_mdio_write: reg=0x00 val=0000
[ 16.301714] nb8800_mdio_read : reg=0x13 val=7c40
[ 16.306399] nb8800_mdio_read : reg=0x12 val=0000
[ 16.311044] nb8800_mdio_write: reg=0x12 val=ec00
[ 16.315726] ++ETH++ gw32 0xf0026200 0x0000028f
[ 16.320246] nb8800_mdio_read : reg=0x04 val=11e1
[ 16.324932] nb8800_mdio_read : reg=0x01 val=7949
[ 16.329583] EXIT nb8800_open
[ 16.329586] ENTER nb8800_set_rx_mode
[ 16.329590] ++ETH++ gw8 0xf002602e 0x00
[ 16.329593] EXIT nb8800_set_rx_mode
[ 16.329627] ENTER nb8800_set_rx_mode
[ 16.329630] ++ETH++ gw8 0xf002602e 0x00
[ 16.329632] ++ETH++ gw8 0xf0026028 0x01
[ 16.329634] ++ETH++ gw8 0xf0026029 0x00
[ 16.329635] ++ETH++ gw8 0xf002602a 0x5e
[ 16.329637] ++ETH++ gw8 0xf002602b 0x00
[ 16.329638] ++ETH++ gw8 0xf002602c 0x00
[ 16.329639] ++ETH++ gw8 0xf002602d 0x01
[ 16.329641] ++ETH++ gw8 0xf002602e 0xff
[ 16.329647] EXIT nb8800_set_rx_mode
[ 16.382924] nb8800_mdio_read : reg=0x09 val=0300
[ 16.387566] nb8800_mdio_write: reg=0x09 val=0200
[ 16.392289] nb8800_mdio_read : reg=0x00 val=0000
[ 16.396931] nb8800_mdio_write: reg=0x00 val=1200
# sleep 6 ## autoneg should be complete
[ 19.677048] nb8800_mdio_read : reg=0x12 val=ec00
[ 19.681709] nb8800_mdio_write: reg=0x12 val=0000
[ 19.686437] nb8800_mdio_read : reg=0x13 val=7400
[ 19.691124] nb8800_mdio_read : reg=0x12 val=0000
[ 19.695772] nb8800_mdio_write: reg=0x12 val=ec00
[ 19.700503] nb8800_mdio_read : reg=0x01 val=796d
[ 19.705189] nb8800_mdio_read : reg=0x01 val=796d
[ 19.709873] nb8800_mdio_read : reg=0x0a val=3c00
[ 19.714556] nb8800_mdio_read : reg=0x09 val=0200
[ 19.719237] nb8800_mdio_read : reg=0x05 val=cde1
[ 19.723922] nb8800_mdio_read : reg=0x04 val=11e1
[ 19.728604] nb8800_mdio_read : reg=0x01 val=796d
[ 19.733258] ++ETH++ gw8 0xf0026050 0x01
[ 19.737204] ++ETH++ gw8 0xf002601c 0xff
[ 19.741149] ++ETH++ gw8 0xf0026044 0x81
[ 19.745090] IGNORE nb8800_pause_config
[ 19.748869] nb8800 26000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
#
# ntpdate 172.27.64.1
(snip ntpdate-induced network activity)
16 Jun 13:23:29 ntpdate[928]: step time server 172.27.64.1 offset 1497619379.702206 sec
# sleep 1
# date +%T && ping -c 3 172.27.64.1 > /tmp/ping_before
13:23:30
[ 30.738458] ++ETH++ gw32 0xf002610c 0x9f6f8150
[ 30.742929] ++ETH++ gw32 0xf0026100 0x00540aff
[ 30.747414] ++ETH++ gw32 0xf0026104 0x00000004
[ 30.751879] ++ETH++ gw32 0xf0026204 0x00000004
[ 30.756338] ++ETH++ gw32 0xf0026218 0x003cc4a4
[ 30.760814] ++ETH++ gw32 0xf0026104 0x00000001
[ 30.765326] ++ETH++ gw32 0xf0026218 0x00004dc8
[ 31.769882] ++ETH++ gw32 0xf002610c 0x9f6f8180
[ 31.774345] ++ETH++ gw32 0xf0026100 0x005c0aff
[ 31.778821] ++ETH++ gw32 0xf0026104 0x00000004
[ 31.783285] ++ETH++ gw32 0xf0026204 0x00000004
[ 31.787745] ++ETH++ gw32 0xf0026218 0x003cc4a4
[ 31.792219] ++ETH++ gw32 0xf0026104 0x00000001
[ 31.796713] ++ETH++ gw32 0xf0026218 0x00004dc8
[ 32.801249] ++ETH++ gw32 0xf002610c 0x9f6f81b0
[ 32.805711] ++ETH++ gw32 0xf0026100 0x005c0aff
[ 32.810186] ++ETH++ gw32 0xf0026104 0x00000004
[ 32.814650] ++ETH++ gw32 0xf0026204 0x00000004
[ 32.819110] ++ETH++ gw32 0xf0026218 0x003cc4a4
[ 32.823581] ++ETH++ gw32 0xf0026104 0x00000001
[ 32.828075] ++ETH++ gw32 0xf0026218 0x00004dc8
# sleep 1
# ip link set eth0 down
[ 33.858675] ENTER nb8800_set_rx_mode
[ 33.862286] ++ETH++ gw8 0xf002602e 0x00
[ 33.866237] ++ETH++ gw8 0xf0026028 0x01
[ 33.870185] ++ETH++ gw8 0xf0026029 0x00
[ 33.874128] ++ETH++ gw8 0xf002602a 0x5e
[ 33.878070] ++ETH++ gw8 0xf002602b 0x00
[ 33.882013] ++ETH++ gw8 0xf002602c 0x00
[ 33.885953] ++ETH++ gw8 0xf002602d 0x01
[ 33.889894] ++ETH++ gw8 0xf002602e 0xff
[ 33.893835] EXIT nb8800_set_rx_mode
[ 33.897506] ENTER nb8800_stop
[ 33.900535] nb8800_mdio_read : reg=0x12 val=ec00
[ 33.905193] nb8800_mdio_write: reg=0x12 val=0000
[ 33.909926] nb8800_mdio_read : reg=0x13 val=0000
[ 33.914580] ENTER nb8800_dma_stop
[ 33.917931] ++ETH++ gw8 0xf0026004 0x03
[ 33.921875] ++ETH++ gw8 0xf0026044 0x85
[ 33.925829] ++ETH++ gw32 0xf002610c 0x9f6f8000
[ 33.930301] ++ETH++ gw32 0xf0026100 0x005c0aff
[ 33.934787] ++ETH++ gw32 0xf0026104 0x00000004
[ 33.939251] ++ETH++ gw32 0xf0026204 0x00000004
[ 33.943711] ++ETH++ gw32 0xf0026218 0x003cc4a4
[ 33.948182] ++ETH++ gw32 0xf0026104 0x00000001
[ 34.034815] ++ETH++ gw32 0xf002610c 0x9f6f8000
[ 34.039284] ++ETH++ gw32 0xf0026100 0x005c0aff
[ 34.043756] ++ETH++ gw32 0xf0026104 0x00000004
[ 34.048220] ++ETH++ gw32 0xf0026204 0x00000004
[ 34.052681] ++ETH++ gw32 0xf0026218 0x003cc4a4
[ 34.057151] ++ETH++ gw32 0xf0026104 0x00000001
[ 34.143788] ++ETH++ gw32 0xf002610c 0x9f6f8000
[ 34.148256] ++ETH++ gw32 0xf0026100 0x005c0aff
[ 34.152727] ++ETH++ gw32 0xf0026104 0x00000004
[ 34.153726] ++ETH++ gw8 0xf0026004 0x23
[ 34.153728] ++ETH++ gw8 0xf0026044 0x81
[ 34.153743] ++ETH++ gw32 0xf002620c 0x9dc4c000
[ 34.153744] EXIT nb8800_dma_stop
[ 34.153746] ++ETH++ gw8 0xf0026004 0x22
[ 34.153748] ++ETH++ gw8 0xf0026000 0x1c
[ 34.153790] nb8800_mdio_read : reg=0x12 val=0000
[ 34.153792] nb8800_mdio_write: reg=0x12 val=0000
[ 34.153871] nb8800_mdio_read : reg=0x13 val=0000
[ 34.153928] nb8800_mdio_read : reg=0x12 val=0000
[ 34.153969] nb8800_mdio_read : reg=0x12 val=0000
[ 34.154009] nb8800_mdio_read : reg=0x00 val=1000
[ 34.154011] nb8800_mdio_write: reg=0x00 val=1800
[ 34.213124] ++ETH++ gw32 0xf0026204 0x00000004
[ 34.217584] ++ETH++ gw32 0xf0026218 0x003cc4a4
[ 34.222088] EXIT nb8800_stop
# sleep 2
# ip link set eth0 up
[ 36.258689] ENTER nb8800_open
[ 36.261695] ++ETH++ gw32 0xf0026204 0x0000000f
[ 36.266168] ++ETH++ gw32 0xf0026104 0x0000000f
[ 36.270640] ENTER nb8800_dma_init
[ 36.274560] ++ETH++ gw32 0xf002620c 0x9dc4e000
[ 36.279032] EXIT nb8800_dma_init
[ 36.282390] ++ETH++ gw32 0xf0026218 0x00004dc8
[ 36.286860] ++ETH++ gw8 0xf0026004 0x23
[ 36.290806] ++ETH++ gw8 0xf0026000 0x1d
[ 36.294778] nb8800_mdio_write: reg=0x00 val=8000
[ 36.352948] nb8800_mdio_read : reg=0x00 val=0000
[ 36.366281] nb8800_mdio_read : reg=0x01 val=7949
[ 36.370966] nb8800_mdio_read : reg=0x0f val=a000
[ 36.375659] nb8800_mdio_read : reg=0x00 val=0000
[ 36.380304] nb8800_mdio_write: reg=0x00 val=0000
[ 36.385040] nb8800_mdio_read : reg=0x13 val=0c00
[ 36.389726] nb8800_mdio_read : reg=0x12 val=0000
[ 36.394384] nb8800_mdio_write: reg=0x12 val=ec00
[ 36.399071] ++ETH++ gw32 0xf0026200 0x06100a8f
[ 36.403593] nb8800_mdio_read : reg=0x04 val=11e1
[ 36.408279] nb8800_mdio_read : reg=0x01 val=7949
[ 36.412927] EXIT nb8800_open
[ 36.412932] ENTER nb8800_set_rx_mode
[ 36.412936] ++ETH++ gw8 0xf002602e 0x00
[ 36.412939] EXIT nb8800_set_rx_mode
[ 36.412969] ENTER nb8800_set_rx_mode
[ 36.412971] ++ETH++ gw8 0xf002602e 0x00
[ 36.412973] ++ETH++ gw8 0xf0026028 0x01
[ 36.412975] ++ETH++ gw8 0xf0026029 0x00
[ 36.412976] ++ETH++ gw8 0xf002602a 0x5e
[ 36.412978] ++ETH++ gw8 0xf002602b 0x00
[ 36.412979] ++ETH++ gw8 0xf002602c 0x00
[ 36.412981] ++ETH++ gw8 0xf002602d 0x01
[ 36.412982] ++ETH++ gw8 0xf002602e 0xff
[ 36.412988] EXIT nb8800_set_rx_mode
[ 36.466286] nb8800_mdio_read : reg=0x09 val=0200
[ 36.471510] nb8800_mdio_write: reg=0x09 val=0200
[ 36.476396] nb8800_mdio_read : reg=0x00 val=0000
[ 36.481079] nb8800_mdio_read : reg=0x00 val=0000
[ 36.485738] nb8800_mdio_write: reg=0x00 val=1200
[ 39.341468] nb8800_mdio_read : reg=0x12 val=ec00
[ 39.346125] nb8800_mdio_write: reg=0x12 val=0000
[ 39.350849] nb8800_mdio_read : reg=0x13 val=1400
[ 39.355534] nb8800_mdio_read : reg=0x12 val=0000
[ 39.360182] nb8800_mdio_write: reg=0x12 val=ec00
[ 39.364911] nb8800_mdio_read : reg=0x01 val=796d
[ 39.369597] nb8800_mdio_read : reg=0x01 val=796d
[ 39.374281] nb8800_mdio_read : reg=0x0a val=3c00
[ 39.378964] nb8800_mdio_read : reg=0x09 val=0200
[ 39.383645] nb8800_mdio_read : reg=0x05 val=cde1
[ 39.388329] nb8800_mdio_read : reg=0x04 val=11e1
[ 39.393013] nb8800_mdio_read : reg=0x01 val=796d
[ 39.397662] IGNORE nb8800_pause_config
#
# date +%T && ping -c 3 172.27.64.1 > /tmp/ping_after
13:23:41
[ 41.518560] ++ETH++ gw32 0xf002610c 0x9de58000
[ 41.523029] ++ETH++ gw32 0xf0026100 0x00080aff
[ 41.527499] ++ETH++ gw32 0xf0026104 0x00000004
[ 41.531964] ++ETH++ gw32 0xf0026204 0x00000004
[ 41.536425] ++ETH++ gw32 0xf0026218 0x003cc4a4
[ 41.540898] ++ETH++ gw32 0xf0026104 0x00000001
[ 41.545375] ++ETH++ gw32 0xf0026218 0x00004dc8
[ 42.532916] ++ETH++ gw32 0xf002610c 0x9de58030
[ 42.537380] ++ETH++ gw32 0xf0026100 0x00240aff
[ 42.541849] ++ETH++ gw32 0xf0026104 0x00000004
[ 42.551844] ++ETH++ gw32 0xf0026104 0x00000001
[ 42.556312] ++ETH++ gw32 0xf0026218 0x00004dc8
[ 43.546246] ++ETH++ gw32 0xf002610c 0x9de58060
[ 43.550708] ++ETH++ gw32 0xf0026100 0x00240aff
[ 43.555175] ++ETH++ gw32 0xf0026104 0x00000004
[ 43.565173] ++ETH++ gw32 0xf0026104 0x00000001
[ 43.569641] ++ETH++ gw32 0xf0026218 0x00004dc8
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Toggling link state breaks network connectivity
2017-06-12 13:22 Toggling link state breaks network connectivity Mason
2017-06-12 16:38 ` Florian Fainelli
@ 2017-06-13 9:39 ` Matthias May
2017-06-13 14:20 ` Mason
1 sibling, 1 reply; 14+ messages in thread
From: Matthias May @ 2017-06-13 9:39 UTC (permalink / raw)
To: Mason, netdev
Cc: Florian Fainelli, Andrew Lunn, Mans Rullgard, Thibaud Cornic
On 12/06/17 15:22, Mason wrote:
> Hello,
>
> I am using the following drivers for Ethernet connectivity.
> drivers/net/ethernet/aurora/nb8800.c
> drivers/net/phy/at803x.c
>
> Pulling the cable and plugging it back works as expected.
> (I can ping both before and after.)
>
> However, if I toggle the link state in software (using ip link set),
> the board loses network connectivity.
>
> # Statically assign IP address
> ip addr add 172.27.64.77/18 brd 172.27.127.255 dev eth0
> # Set link state to "up"
> ip link set eth0 up
> # ping -c 3 172.27.64.1 > /tmp/v1
>
> PING 172.27.64.1 (172.27.64.1): 56 data bytes
> 64 bytes from 172.27.64.1: seq=0 ttl=64 time=18.321 ms
>
> --- 172.27.64.1 ping statistics ---
> 3 packets transmitted, 1 packets received, 66% packet loss
> round-trip min/avg/max = 18.321/18.321/18.321 ms
>
>
> 172.27.64.1 is a desktop system.
> Running
> % tcpdump -n -i eth1-boards ether host 00:16:e8:4d:7f:c4
> on the desktop, I get:
>
> 15:01:45.187346 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
> 15:01:45.187359 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
> 15:01:45.194633 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41219, seq 0, length 64
> 15:01:45.194662 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41219, seq 0, length 64
> 15:01:50.198564 ARP, Request who-has 172.27.64.77 tell 172.27.64.1, length 28
> 15:01:50.205929 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41219, seq 1, length 64
> 15:01:50.205951 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41219, seq 1, length 64
> 15:01:50.213217 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41219, seq 2, length 64
> 15:01:50.213232 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41219, seq 2, length 64
> 15:01:51.198563 ARP, Request who-has 172.27.64.77 tell 172.27.64.1, length 28
> 15:01:51.209586 ARP, Reply 172.27.64.77 is-at 00:16:e8:4d:7f:c4, length 46
> 15:01:51.209598 ARP, Reply 172.27.64.77 is-at 00:16:e8:4d:7f:c4, length 46
>
> Packet #1: the board asks for the desktop's MAC address
> Packet #2: the desktop replies instantly
> Packet #3: the board sends the first ping
> Packet #4: the desktop replies instantly
> Then the board goes quiet for a long time (why???)
> Packet #5: the desktop asks for the board's MAC address (doesn't it have it already?)
> Packet #6: this seems to unwedge the board, which sends the second ping
> Packet #7: the desktop replies instantly
> Packet #8: the board sends the third ping
> Packet #9: the desktop replies instantly
> Packet #10: the desktop asks again for the board's MAC address
> Packet #11 and #12: the board answers twice (for the old and new requests?)
>
> Some oddities, but it seems to work.
>
> Now toggle the link state:
>
> % ip link set eth0 down
> % ip link set eth0 up
> % ping -c 3 172.27.64.1 > /tmp/v2
>
> PING 172.27.64.1 (172.27.64.1): 56 data bytes
>
> --- 172.27.64.1 ping statistics ---
> 3 packets transmitted, 0 packets received, 100% packet loss
>
>
> On the desktop, I see
>
> 15:14:03.900162 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
> 15:14:03.900175 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
> 15:14:05.017189 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
> 15:14:05.017200 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
> 15:14:06.030531 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
> 15:14:06.030541 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
>
> So basically, the board is asking the desktop for its MAC address,
> and the desktop is answering immediately. But the board doesn't seem
> to be getting the replies... Any ideas, or words of wisdom, as they say?
>
> Regards.
>
Hi
You might want to read this thread:
https://www.mail-archive.com/netdev@vger.kernel.org/msg133896.html
The symptoms you describe are pretty much what we had at one point.
BR
Matthias
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Toggling link state breaks network connectivity
2017-06-13 9:39 ` Matthias May
@ 2017-06-13 14:20 ` Mason
2017-06-13 14:40 ` Matthias May
0 siblings, 1 reply; 14+ messages in thread
From: Mason @ 2017-06-13 14:20 UTC (permalink / raw)
To: Matthias May, Zefir Kurtisi
Cc: netdev, Florian Fainelli, Andrew Lunn, Mans Rullgard,
Thibaud Cornic
On 13/06/2017 11:39, Matthias May wrote:
> On 12/06/17 15:22, Mason wrote:
>> Hello,
>>
>> I am using the following drivers for Ethernet connectivity.
>> drivers/net/ethernet/aurora/nb8800.c
>> drivers/net/phy/at803x.c
>>
>> Pulling the cable and plugging it back works as expected.
>> (I can ping both before and after.)
>>
>> However, if I toggle the link state in software (using ip link set),
>> the board loses network connectivity.
>>
>> # Statically assign IP address
>> ip addr add 172.27.64.77/18 brd 172.27.127.255 dev eth0
>> # Set link state to "up"
>> ip link set eth0 up
>> # ping -c 3 172.27.64.1 > /tmp/v1
>>
>> PING 172.27.64.1 (172.27.64.1): 56 data bytes
>> 64 bytes from 172.27.64.1: seq=0 ttl=64 time=18.321 ms
>>
>> --- 172.27.64.1 ping statistics ---
>> 3 packets transmitted, 1 packets received, 66% packet loss
>> round-trip min/avg/max = 18.321/18.321/18.321 ms
>>
>>
>> 172.27.64.1 is a desktop system.
>> Running
>> % tcpdump -n -i eth1-boards ether host 00:16:e8:4d:7f:c4
>> on the desktop, I get:
>>
>> 15:01:45.187346 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
>> 15:01:45.187359 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
>> 15:01:45.194633 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41219, seq 0, length 64
>> 15:01:45.194662 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41219, seq 0, length 64
>> 15:01:50.198564 ARP, Request who-has 172.27.64.77 tell 172.27.64.1, length 28
>> 15:01:50.205929 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41219, seq 1, length 64
>> 15:01:50.205951 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41219, seq 1, length 64
>> 15:01:50.213217 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41219, seq 2, length 64
>> 15:01:50.213232 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41219, seq 2, length 64
>> 15:01:51.198563 ARP, Request who-has 172.27.64.77 tell 172.27.64.1, length 28
>> 15:01:51.209586 ARP, Reply 172.27.64.77 is-at 00:16:e8:4d:7f:c4, length 46
>> 15:01:51.209598 ARP, Reply 172.27.64.77 is-at 00:16:e8:4d:7f:c4, length 46
>>
>> Packet #1: the board asks for the desktop's MAC address
>> Packet #2: the desktop replies instantly
>> Packet #3: the board sends the first ping
>> Packet #4: the desktop replies instantly
>> Then the board goes quiet for a long time (why???)
>> Packet #5: the desktop asks for the board's MAC address (doesn't it have it already?)
>> Packet #6: this seems to unwedge the board, which sends the second ping
>> Packet #7: the desktop replies instantly
>> Packet #8: the board sends the third ping
>> Packet #9: the desktop replies instantly
>> Packet #10: the desktop asks again for the board's MAC address
>> Packet #11 and #12: the board answers twice (for the old and new requests?)
>>
>> Some oddities, but it seems to work.
>>
>> Now toggle the link state:
>>
>> % ip link set eth0 down
>> % ip link set eth0 up
>> % ping -c 3 172.27.64.1 > /tmp/v2
>>
>> PING 172.27.64.1 (172.27.64.1): 56 data bytes
>>
>> --- 172.27.64.1 ping statistics ---
>> 3 packets transmitted, 0 packets received, 100% packet loss
>>
>>
>> On the desktop, I see
>>
>> 15:14:03.900162 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
>> 15:14:03.900175 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
>> 15:14:05.017189 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
>> 15:14:05.017200 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
>> 15:14:06.030531 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
>> 15:14:06.030541 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
>>
>> So basically, the board is asking the desktop for its MAC address,
>> and the desktop is answering immediately. But the board doesn't seem
>> to be getting the replies... Any ideas, or words of wisdom, as they say?
>
> You might want to read this thread:
> https://www.mail-archive.com/netdev@vger.kernel.org/msg133896.html
> The symptoms you describe are pretty much what we had at one point.
Hello Matthias,
Yes, I remember discussing the issue with Zefir.
https://www.mail-archive.com/netdev@vger.kernel.org/msg147369.html
Do you think I should apply Zefir's patch?
In an attempt to remove the Atheros driver from the equation,
I used the generic PHY driver (I didn't build the Atheros
driver at all). But this had no impact on the issue.
Regards.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Toggling link state breaks network connectivity
2017-06-13 14:20 ` Mason
@ 2017-06-13 14:40 ` Matthias May
0 siblings, 0 replies; 14+ messages in thread
From: Matthias May @ 2017-06-13 14:40 UTC (permalink / raw)
To: Mason, Zefir Kurtisi
Cc: netdev, Florian Fainelli, Andrew Lunn, Mans Rullgard,
Thibaud Cornic
On 13/06/17 16:20, Mason wrote:
> On 13/06/2017 11:39, Matthias May wrote:
>> On 12/06/17 15:22, Mason wrote:
>>> Hello,
>>>
>>> I am using the following drivers for Ethernet connectivity.
>>> drivers/net/ethernet/aurora/nb8800.c
>>> drivers/net/phy/at803x.c
>>>
>>> Pulling the cable and plugging it back works as expected.
>>> (I can ping both before and after.)
>>>
>>> However, if I toggle the link state in software (using ip link set),
>>> the board loses network connectivity.
>>>
>>> # Statically assign IP address
>>> ip addr add 172.27.64.77/18 brd 172.27.127.255 dev eth0
>>> # Set link state to "up"
>>> ip link set eth0 up
>>> # ping -c 3 172.27.64.1 > /tmp/v1
>>>
>>> PING 172.27.64.1 (172.27.64.1): 56 data bytes
>>> 64 bytes from 172.27.64.1: seq=0 ttl=64 time=18.321 ms
>>>
>>> --- 172.27.64.1 ping statistics ---
>>> 3 packets transmitted, 1 packets received, 66% packet loss
>>> round-trip min/avg/max = 18.321/18.321/18.321 ms
>>>
>>>
>>> 172.27.64.1 is a desktop system.
>>> Running
>>> % tcpdump -n -i eth1-boards ether host 00:16:e8:4d:7f:c4
>>> on the desktop, I get:
>>>
>>> 15:01:45.187346 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
>>> 15:01:45.187359 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
>>> 15:01:45.194633 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41219, seq 0, length 64
>>> 15:01:45.194662 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41219, seq 0, length 64
>>> 15:01:50.198564 ARP, Request who-has 172.27.64.77 tell 172.27.64.1, length 28
>>> 15:01:50.205929 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41219, seq 1, length 64
>>> 15:01:50.205951 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41219, seq 1, length 64
>>> 15:01:50.213217 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41219, seq 2, length 64
>>> 15:01:50.213232 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41219, seq 2, length 64
>>> 15:01:51.198563 ARP, Request who-has 172.27.64.77 tell 172.27.64.1, length 28
>>> 15:01:51.209586 ARP, Reply 172.27.64.77 is-at 00:16:e8:4d:7f:c4, length 46
>>> 15:01:51.209598 ARP, Reply 172.27.64.77 is-at 00:16:e8:4d:7f:c4, length 46
>>>
>>> Packet #1: the board asks for the desktop's MAC address
>>> Packet #2: the desktop replies instantly
>>> Packet #3: the board sends the first ping
>>> Packet #4: the desktop replies instantly
>>> Then the board goes quiet for a long time (why???)
>>> Packet #5: the desktop asks for the board's MAC address (doesn't it have it already?)
>>> Packet #6: this seems to unwedge the board, which sends the second ping
>>> Packet #7: the desktop replies instantly
>>> Packet #8: the board sends the third ping
>>> Packet #9: the desktop replies instantly
>>> Packet #10: the desktop asks again for the board's MAC address
>>> Packet #11 and #12: the board answers twice (for the old and new requests?)
>>>
>>> Some oddities, but it seems to work.
>>>
>>> Now toggle the link state:
>>>
>>> % ip link set eth0 down
>>> % ip link set eth0 up
>>> % ping -c 3 172.27.64.1 > /tmp/v2
>>>
>>> PING 172.27.64.1 (172.27.64.1): 56 data bytes
>>>
>>> --- 172.27.64.1 ping statistics ---
>>> 3 packets transmitted, 0 packets received, 100% packet loss
>>>
>>>
>>> On the desktop, I see
>>>
>>> 15:14:03.900162 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
>>> 15:14:03.900175 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
>>> 15:14:05.017189 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
>>> 15:14:05.017200 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
>>> 15:14:06.030531 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
>>> 15:14:06.030541 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
>>>
>>> So basically, the board is asking the desktop for its MAC address,
>>> and the desktop is answering immediately. But the board doesn't seem
>>> to be getting the replies... Any ideas, or words of wisdom, as they say?
>>
>> You might want to read this thread:
>> https://www.mail-archive.com/netdev@vger.kernel.org/msg133896.html
>> The symptoms you describe are pretty much what we had at one point.
>
> Hello Matthias,
>
> Yes, I remember discussing the issue with Zefir.
> https://www.mail-archive.com/netdev@vger.kernel.org/msg147369.html
>
> Do you think I should apply Zefir's patch?
>
> In an attempt to remove the Atheros driver from the equation,
> I used the generic PHY driver (I didn't build the Atheros
> driver at all). But this had no impact on the issue.
>
> Regards.
>
Isn't this patch already accepted?
I sent the link more as reference to try to check what we checked to detect the underlying issue.
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2017-06-16 14:18 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-06-12 13:22 Toggling link state breaks network connectivity Mason
2017-06-12 16:38 ` Florian Fainelli
2017-06-12 18:58 ` Mason
2017-06-13 15:07 ` Mason
2017-06-13 15:32 ` Mason
2017-06-13 15:36 ` Florian Fainelli
2017-06-13 15:47 ` Mason
2017-06-13 15:50 ` Florian Fainelli
2017-06-14 14:49 ` Mason
2017-06-15 10:19 ` Måns Rullgård
2017-06-16 14:18 ` Mason
2017-06-13 9:39 ` Matthias May
2017-06-13 14:20 ` Mason
2017-06-13 14:40 ` Matthias May
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox