net: mv643xx: interface does not transmit after some time

net: mv643xx: interface does not transmit after some time

[Thomas Schlöter]

Hello,

I am using mainline kernel 4.4 on a QNAP TS-412 (Marvell Kirkwood Feroceon 6281). The network interface mv643xx loses connection after a few minutes to some hours. Only bringing the interface down and up again helps.

Now I have a serial console for debugging. When I ping the dead interface and run ?watch ifconfig eth0? on the serial console, I can see the RX packets counting up while TX packets stays unchanged. Log files only show a message of some services showing a timeout or ?if_sendrawpacket: No buffer space available?.

How can I figure out what is wrong?

Regards,
Thomas

net: mv643xx: interface does not transmit after some time

[Andrew Lunn]

> How can I figure out what is wrong?

Hi Thomas

Can you find an easily way to reproduce it? iperf, or scp a big file
etc.

If you have a simple and reliable way to reproduce it, i would suggest
doing a git bisect to find out which change broke it.

      Andrew

net: mv643xx: interface does not transmit after some time

[Martin Michlmayr]

* Andrew Lunn <andrew@lunn.ch> [2016-02-06 19:34]:
> Can you find an easily way to reproduce it? iperf, or scp a big file
> etc.

FWIW, we had a similar bug report in Debian recently:
https://lists.debian.org/debian-arm/2016/01/msg00098.html

Turning off TCP RX and TX offloading make it go away:
https://lists.debian.org/debian-arm/2016/01/msg00100.html

Hopefully Thomas can do a bisect.

-- 
Martin Michlmayr
http://www.cyrius.com/

net: mv643xx: interface does not transmit after some time

[Adam Baker]

On 06/02/16 23:19, Martin Michlmayr wrote:
> * Andrew Lunn <andrew@lunn.ch> [2016-02-06 19:34]:
>> Can you find an easily way to reproduce it? iperf, or scp a big file
>> etc.
>
> FWIW, we had a similar bug report in Debian recently:
> https://lists.debian.org/debian-arm/2016/01/msg00098.html
>
> Turning off TCP RX and TX offloading make it go away:
> https://lists.debian.org/debian-arm/2016/01/msg00100.html
>
> Hopefully Thomas can do a bisect.
>

I observed a similar issue when performing a large NFSv4 transfer with a 
kirkwood based server. Unfortunately I'd just jumped from 3.14 to 4.4 so 
I could do with a better starting point to bisect from.

It failed several times while transferring a 4GB file over NFS but is 
stable in normal use.

Regards

Adam

net: mv643xx: interface does not transmit after some time

[Andrew Lunn]

On Sun, Feb 07, 2016 at 04:15:36PM +0000, Adam Baker wrote:
> On 06/02/16 23:19, Martin Michlmayr wrote:
> >* Andrew Lunn <andrew@lunn.ch> [2016-02-06 19:34]:
> >>Can you find an easily way to reproduce it? iperf, or scp a big file
> >>etc.
> >
> >FWIW, we had a similar bug report in Debian recently:
> >https://lists.debian.org/debian-arm/2016/01/msg00098.html
> >
> >Turning off TCP RX and TX offloading make it go away:
> >https://lists.debian.org/debian-arm/2016/01/msg00100.html
> >
> >Hopefully Thomas can do a bisect.
> >
> 
> I observed a similar issue when performing a large NFSv4 transfer
> with a kirkwood based server. Unfortunately I'd just jumped from
> 3.14 to 4.4 so I could do with a better starting point to bisect
> from.
> 
> It failed several times while transferring a 4GB file over NFS but
> is stable in normal use.

I tried iperf, 6Gbytes, both RX and TX and it worked. This is with
4.5-rc2 net-next.

I will try again with different window sizes etc, to see if i can
trigger it.

	Andrew

net: mv643xx: interface does not transmit after some time

[Andrew Lunn]

> > FWIW, we had a similar bug report in Debian recently:
> > https://lists.debian.org/debian-arm/2016/01/msg00098.html

Hi Thomas

I this thread, Ian Campbell mentions a patch. Please could you try
that patch and see if it fixes your problem.

Thanks
	Andrew

net: mv643xx: interface does not transmit after some time

[Thomas Schlöter]

Am 07.02.2016 um 21:35 schrieb Andrew Lunn <andrew@lunn.ch>:
> 
>>> FWIW, we had a similar bug report in Debian recently:
>>> https://lists.debian.org/debian-arm/2016/01/msg00098.html
> 
> Hi Thomas
> 
> I this thread, Ian Campbell mentions a patch. Please could you try
> that patch and see if it fixes your problem.
> 
> Thanks
> 	Andrew

Hi Andrew,

I just applied the patch and the NAS is now running it. I?ll try to crash it tonight and keep you informed whether it worked.

Thanks
	Thomas

net: mv643xx: interface does not transmit after some time

[Thomas Schlöter]

> Am 07.02.2016 um 22:07 schrieb Thomas Schl?ter <thomas@schloeter.net>:
> 
> Am 07.02.2016 um 21:35 schrieb Andrew Lunn <andrew@lunn.ch>:
>> 
>>>> FWIW, we had a similar bug report in Debian recently:
>>>> https://lists.debian.org/debian-arm/2016/01/msg00098.html
>> 
>> Hi Thomas
>> 
>> I this thread, Ian Campbell mentions a patch. Please could you try
>> that patch and see if it fixes your problem.
>> 
>> Thanks
>> 	Andrew
> 
> Hi Andrew,
> 
> I just applied the patch and the NAS is now running it. I?ll try to crash it tonight and keep you informed whether it worked.
> 
> Thanks
> 	Thomas

Hi Andrew,

the patch did not fix the problem. After 1.2 GiB RX and 950 MiB TX, the interface crashed again.

Now I switched off RX/TX offload just to make sure we are talking about the same problem. If we are, the interface should be stable without offload, right?

	Thomas

net: mv643xx: interface does not transmit after some time

[Thomas Schlöter]

> Am 08.02.2016 um 19:49 schrieb Thomas Schl?ter <thomas@schloeter.net>:
> 
> 
>> Am 07.02.2016 um 22:07 schrieb Thomas Schl?ter <thomas@schloeter.net>:
>> 
>> Am 07.02.2016 um 21:35 schrieb Andrew Lunn <andrew@lunn.ch>:
>>> 
>>>>> FWIW, we had a similar bug report in Debian recently:
>>>>> https://lists.debian.org/debian-arm/2016/01/msg00098.html
>>> 
>>> Hi Thomas
>>> 
>>> I this thread, Ian Campbell mentions a patch. Please could you try
>>> that patch and see if it fixes your problem.
>>> 
>>> Thanks
>>> 	Andrew
>> 
>> Hi Andrew,
>> 
>> I just applied the patch and the NAS is now running it. I?ll try to crash it tonight and keep you informed whether it worked.
>> 
>> Thanks
>> 	Thomas
> 
> Hi Andrew,
> 
> the patch did not fix the problem. After 1.2 GiB RX and 950 MiB TX, the interface crashed again.
> 
> Now I switched off RX/TX offload just to make sure we are talking about the same problem. If we are, the interface should be stable without offload, right?
> 
> 	Thomas

Okay, so I have installed ethtool and switched off all offload features available. Now the NAS is running rock solid for two days. I backed up my Mac using Time Machine / netatalk (450 GiB transferred) and some Linux machines via NFS (100 GiB total) without a problem.

How much code is used for mv643xx offload functionality?
Is it possible to debug things in the driver and figure out what happens during the crash?
Is the hardware offload interface proprietary or reverse engineered or is it a well known API that can be analyzed?

	Thomas

net: mv643xx: interface does not transmit after some time

[Andrew Lunn]

On Wed, Feb 10, 2016 at 07:40:54PM +0100, Thomas Schl?ter wrote:
> 
> > Am 08.02.2016 um 19:49 schrieb Thomas Schl?ter <thomas@schloeter.net>:
> > 
> > 
> >> Am 07.02.2016 um 22:07 schrieb Thomas Schl?ter <thomas@schloeter.net>:
> >> 
> >> Am 07.02.2016 um 21:35 schrieb Andrew Lunn <andrew@lunn.ch>:
> >>> 
> >>>>> FWIW, we had a similar bug report in Debian recently:
> >>>>> https://lists.debian.org/debian-arm/2016/01/msg00098.html
> >>> 
> >>> Hi Thomas
> >>> 
> >>> I this thread, Ian Campbell mentions a patch. Please could you try
> >>> that patch and see if it fixes your problem.
> >>> 
> >>> Thanks
> >>> 	Andrew
> >> 
> >> Hi Andrew,
> >> 
> >> I just applied the patch and the NAS is now running it. I???ll try to crash it tonight and keep you informed whether it worked.
> >> 
> >> Thanks
> >> 	Thomas
> > 
> > Hi Andrew,
> > 
> > the patch did not fix the problem. After 1.2 GiB RX and 950 MiB TX, the interface crashed again.
> > 
> > Now I switched off RX/TX offload just to make sure we are talking about the same problem. If we are, the interface should be stable without offload, right?
> > 
> > 	Thomas
> 
> Okay, so I have installed ethtool and switched off all offload features available. Now the NAS is running rock solid for two days. I backed up my Mac using Time Machine / netatalk (450 GiB transferred) and some Linux machines via NFS (100 GiB total) without a problem.
> 
> How much code is used for mv643xx offload functionality?
> Is it possible to debug things in the driver and figure out what happens during the crash?
> Is the hardware offload interface proprietary or reverse engineered or is it a well known API that can be analyzed?

Hi Thomas

Ezequiel Garcia probably knows this part of the driver and hardware
the best...

    Andrew

net: mv643xx: interface does not transmit after some time

[Ezequiel Garcia]

(let's expand the Cc a bit)

On 10 February 2016 at 19:57, Andrew Lunn <andrew@lunn.ch> wrote:
> On Wed, Feb 10, 2016 at 07:40:54PM +0100, Thomas Schl?ter wrote:
>>
>> > Am 08.02.2016 um 19:49 schrieb Thomas Schl?ter <thomas@schloeter.net>:
>> >
>> >
>> >> Am 07.02.2016 um 22:07 schrieb Thomas Schl?ter <thomas@schloeter.net>:
>> >>
>> >> Am 07.02.2016 um 21:35 schrieb Andrew Lunn <andrew@lunn.ch>:
>> >>>
>> >>>>> FWIW, we had a similar bug report in Debian recently:
>> >>>>> https://lists.debian.org/debian-arm/2016/01/msg00098.html
>> >>>
>> >>> Hi Thomas
>> >>>
>> >>> I this thread, Ian Campbell mentions a patch. Please could you try
>> >>> that patch and see if it fixes your problem.
>> >>>
>> >>> Thanks
>> >>>   Andrew
>> >>
>> >> Hi Andrew,
>> >>
>> >> I just applied the patch and the NAS is now running it. I???ll try to crash it tonight and keep you informed whether it worked.
>> >>
>> >> Thanks
>> >>    Thomas
>> >
>> > Hi Andrew,
>> >
>> > the patch did not fix the problem. After 1.2 GiB RX and 950 MiB TX, the interface crashed again.
>> >
>> > Now I switched off RX/TX offload just to make sure we are talking about the same problem. If we are, the interface should be stable without offload, right?
>> >
>> >     Thomas
>>
>> Okay, so I have installed ethtool and switched off all offload features available. Now the NAS is running rock solid for two days. I backed up my Mac using Time Machine / netatalk (450 GiB transferred) and some Linux machines via NFS (100 GiB total) without a problem.
>>
>> How much code is used for mv643xx offload functionality?
>> Is it possible to debug things in the driver and figure out what happens during the crash?
>> Is the hardware offload interface proprietary or reverse engineered or is it a well known API that can be analyzed?
>
> Hi Thomas
>
> Ezequiel Garcia probably knows this part of the driver and hardware
> the best...
>

The TCP segmentation offload (TSO) implemented in this driver is
mostly a software thing.

I'm CCing Karl and Philipp, who have fixed subtle issues in the TSO
path, and may be able to help figure this one out.

-- 
Ezequiel Garc?a, VanguardiaSur
www.vanguardiasur.com.ar

net: mv643xx: interface does not transmit after some time

[Adam Baker]

On 11/02/16 14:38, Ezequiel Garcia wrote:
> (let's expand the Cc a bit)
>
> On 10 February 2016 at 19:57, Andrew Lunn <andrew@lunn.ch> wrote:
>> On Wed, Feb 10, 2016 at 07:40:54PM +0100, Thomas Schl?ter wrote:
>>>
>>>> Am 08.02.2016 um 19:49 schrieb Thomas Schl?ter <thomas@schloeter.net>:
>>>>
>>>>
>>>>> Am 07.02.2016 um 22:07 schrieb Thomas Schl?ter <thomas@schloeter.net>:
>>>>>
>>>>> Am 07.02.2016 um 21:35 schrieb Andrew Lunn <andrew@lunn.ch>:
>>>>>>
>>>>>>>> FWIW, we had a similar bug report in Debian recently:
>>>>>>>> https://lists.debian.org/debian-arm/2016/01/msg00098.html
>>>>>>
>>>>>> Hi Thomas
>>>>>>
>>>>>> I this thread, Ian Campbell mentions a patch. Please could you try
>>>>>> that patch and see if it fixes your problem.
>>>>>>
>>>>>> Thanks
>>>>>>    Andrew
>>>>>
>>>>> Hi Andrew,
>>>>>
>>>>> I just applied the patch and the NAS is now running it. I???ll try to crash it tonight and keep you informed whether it worked.
>>>>>
>>>>> Thanks
>>>>>     Thomas
>>>>
>>>> Hi Andrew,
>>>>
>>>> the patch did not fix the problem. After 1.2 GiB RX and 950 MiB TX, the interface crashed again.
>>>>
>>>> Now I switched off RX/TX offload just to make sure we are talking about the same problem. If we are, the interface should be stable without offload, right?
>>>>
>>>>      Thomas
>>>
>>> Okay, so I have installed ethtool and switched off all offload features available. Now the NAS is running rock solid for two days. I backed up my Mac using Time Machine / netatalk (450 GiB transferred) and some Linux machines via NFS (100 GiB total) without a problem.
>>>
>>> How much code is used for mv643xx offload functionality?
>>> Is it possible to debug things in the driver and figure out what happens during the crash?
>>> Is the hardware offload interface proprietary or reverse engineered or is it a well known API that can be analyzed?
>>
>> Hi Thomas
>>
>> Ezequiel Garcia probably knows this part of the driver and hardware
>> the best...
>>
>
> The TCP segmentation offload (TSO) implemented in this driver is
> mostly a software thing.
>
> I'm CCing Karl and Philipp, who have fixed subtle issues in the TSO
> path, and may be able to help figure this one out.
>

Hi,

Had this issue occur again today. In my case it seems to be triggered by 
large NFSv4 transfers.

I'm running 4.4 plus Nicolas Schichan's patch at
https://patchwork.ozlabs.org/patch/573334/

There is a thread a http://forum.doozan.com/read.php?2,17404 suggesting 
that this has been broken since at least 3.16.

I first spotted the issue when upgrading from 3.11 to 4.4.

Looking at 
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/log/drivers/net/ethernet/marvell/mv643xx_eth.c 
I see 2014-05-22 as the date TSO support was first added which is 
shortly before the merge window opened for 3.16. I'm therefore guessing 
that TSO has been problematic since it's introduction.

Regards

Adam

net: mv643xx: interface does not transmit after some time

[Thomas Schlöter]

Am 07.02.2016 um 00:19 schrieb Martin Michlmayr <tbm@cyrius.com>:
> 
> * Andrew Lunn <andrew@lunn.ch> [2016-02-06 19:34]:
>> Can you find an easily way to reproduce it? iperf, or scp a big file
>> etc.
> 
> FWIW, we had a similar bug report in Debian recently:
> https://lists.debian.org/debian-arm/2016/01/msg00098.html
> 
> Turning off TCP RX and TX offloading make it go away:
> https://lists.debian.org/debian-arm/2016/01/msg00100.html

Sounds interesting. I will try that.

> Hopefully Thomas can do a bisect.

I have never used that feature before. As far as I understand, I have to mark the last known good revision good and the most recent bad. Then I try every version in between which has changef related code. Right?

Actually I don't have a last good version as I had the original QNAP fw installed before.

4.3, which I installed during my first try was bad and 4.4 did not fix it. What can I do now? Should I provide my complete kernel config file?
At the moment I am trying Ian Campbell's patch as suggested by Andrew.

Thomas

@Martin: Sorry for reposting to your address, I did not manage to send plain text mail from my iPhone.