Re: Lost network connectivity in 4.0.x

Re: Lost network connectivity in 4.0.x

[Cong Wang]

(Please always Cc netdev for networking bugs.)

On Sat, May 23, 2015 at 8:29 PM, Ken Moffat <zarniwhoop@ntlworld.com> wrote:
> On Sun, May 24, 2015 at 03:43:52AM +0100, Ken Moffat wrote:
>> Anybody else suffering frm lost network connectivity in 4.0.x
>> kernels ?  A couple of times this week, vim on an nfs-3 mount hung
>> and I had to reboot.  Both of those occasions were on an AMD desktop
>> with the r8169 driver, running 4.0.3.  I thought it might be
>> specific to that machine.  For the last two or three days I've been
>> using an intel, and about 10 minutes ago it suffered the same problem
>> while running 4.0.4.  Using ping from another term showed that it
>> had no connectivity to the server on my local network.
>>
>> This is a bit hard to diagnose - nothing in the logs.
>>
> I forgot to add that this is with the released gcc-5.1 : I keep
> forgetting that some people use old compilers ;-)
>

Is there any way you can help to narrow down the problem?

For example:

1) What is your network setup? iptables? routes? etc.

2) Can you check the stats to see if there is any error?
  `ip -s -s li show`, `ethtool -S <DEV>`

3) Do a bisect?

Thanks!

Re: Lost network connectivity in 4.0.x

[Ken Moffat]

On Wed, May 27, 2015 at 10:53:00PM -0700, Cong Wang wrote:
> (Please always Cc netdev for networking bugs.)
> 
> On Sat, May 23, 2015 at 8:29 PM, Ken Moffat <zarniwhoop@ntlworld.com> wrote:
> > On Sun, May 24, 2015 at 03:43:52AM +0100, Ken Moffat wrote:
> >> Anybody else suffering frm lost network connectivity in 4.0.x
> >> kernels ?  A couple of times this week, vim on an nfs-3 mount hung
> >> and I had to reboot.  Both of those occasions were on an AMD desktop
> >> with the r8169 driver, running 4.0.3.  I thought it might be
> >> specific to that machine.  For the last two or three days I've been
> >> using an intel, and about 10 minutes ago it suffered the same problem
> >> while running 4.0.4.  Using ping from another term showed that it
> >> had no connectivity to the server on my local network.
> >>
> >> This is a bit hard to diagnose - nothing in the logs.
> >>
> > I forgot to add that this is with the released gcc-5.1 : I keep
> > forgetting that some people use old compilers ;-)
> >
> 
> Is there any way you can help to narrow down the problem?
> 

Thanks for the reply.  The problem is continuing to show up, but
irregularly and often only after the machine has been booted for a
long time (with s2ram, but I don't think I've used s2ram on every
occasion).

> For example:
> 
> 1) What is your network setup? iptables? routes? etc.
> 
I'm using iptables.  Ah, yes - it started dropping packets around
the time I last had a problem:

May 27 00:48:26 ac4tv dhclient: DHCPREQUEST on eth0 to 192.168.7.254
port 67
May 27 00:48:27 ac4tv dhclient: DHCPACK from 192.168.7.254
May 27 00:48:27 ac4tv dhclient: bound to 192.168.7.152 -- renewal in
1787 seconds.

 That address came from my router, and I had been getting the same
address for an hour, tbut then the dropped packet messages start
appearing - they are for a different address, one that would have
been offered by my server:

May 27 00:53:16 ac4tv kernel: [31922.316798] IPTABLES Packet
Dropped: IN=eth0 OUT= MAC=c8:60:00:97:07:35:bc:ae:c5:57:70:c5:08:00
SRC=192.168.7.11 DST=192.168.7.121 LEN=60 TOS=0x00 PREC=0x00 TTL=64
ID=0 DF PROTO=TCP SPT=2049 DPT=1005 WINDOW=28960 RES=0x00 ACK SYN
URGP=0 
May 27 00:53:17 ac4tv kernel: [31923.316612] IPTABLES Packet
Dropped: IN=eth0 OUT= MAC=c8:60:00:97:07:35:bc:ae:c5:57:70:c5:08:00
SRC=192.168.7.11 DST=192.168.7.121 LEN=60 TOS=0x00 PREC=0x00 TTL=64
ID=0 DF PROTO=TCP SPT=2049 DPT=1005 WINDOW=28960 RES=0x00 ACK SYN
URGP=0 

and those continued until I forced a reboot.

> 2) Can you check the stats to see if there is any error?
>   `ip -s -s li show`, `ethtool -S <DEV>`
> 

I don't have ethtool installed, and that ip command appears ok at
the moment:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
mode DEFAULT group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    RX: bytes  packets  errors  dropped overrun mcast   
    3964       66       0       0       0       0       
    RX errors: length   crc     frame   fifo    missed
               0        0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns 
    3964       66       0       0       0       0       
    TX errors: aborted  fifo   window heartbeat transns
               0        0       0       0       0       
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UP mode DEFAULT group default qlen 1000
    link/ether c8:60:00:97:07:35 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    224661061  277642   0       0       0       0       
    RX errors: length   crc     frame   fifo    missed
               0        0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns 
    278152429  370438   0       0       0       0       
    TX errors: aborted  fifo   window heartbeat transns
               0        0       0       0       6       

> 3) Do a bisect?
> 
> Thanks!

That doesn't seem very practical when the machine is ok for a couple
of days at a time.

ĸen
-- 
Nanny Ogg usually went to bed early. After all, she was an old lady.
Sometimes she went to bed as early as 6 a.m.

Re: Lost network connectivity in 4.0.x

[Ken Moffat]

On Thu, May 28, 2015 at 03:41:49PM +0100, Ken Moffat wrote:
> On Wed, May 27, 2015 at 10:53:00PM -0700, Cong Wang wrote:
> > (Please always Cc netdev for networking bugs.)
> > 
Sorry, didn't spot that.  But anyway
> 
> > For example:
> > 
> > 1) What is your network setup? iptables? routes? etc.
> > 
> I'm using iptables.  Ah, yes - it started dropping packets around
> the time I last had a problem:
> 
> May 27 00:48:26 ac4tv dhclient: DHCPREQUEST on eth0 to 192.168.7.254
> port 67
> May 27 00:48:27 ac4tv dhclient: DHCPACK from 192.168.7.254
> May 27 00:48:27 ac4tv dhclient: bound to 192.168.7.152 -- renewal in
> 1787 seconds.
> 
>  That address came from my router, and I had been getting the same
> address for an hour, tbut then the dropped packet messages start
> appearing - they are for a different address, one that would have
> been offered by my server:
> 
Now that I've had time to think about this and look a bit more
deeply, I can see that at one point I got a lease from my server,
but then after a random length of time the client tried to renew and
got a lease from the router.  Some time after that, it failed
because iptables rejected the nfs packets because they were "not for
me".

So, not a kernel problem, and the reason I'm (now) seeing this on
4.0+ kernels is that I have not recently booted a system with an old
(3.19 or earlier) kernel and kept it running for a long time.

Thanks again, sorry to waste everybody's bandwidth.

ĸen
-- 
Nanny Ogg usually went to bed early. After all, she was an old lady.
Sometimes she went to bed as early as 6 a.m.