Re: [PATCH 0/3]: net: dsa: mt7530: support MT7530 in the MT7621 SoC

[NeilBrown <neil@brown.name>]

[-- Attachment #1: Type: text/plain, Size: 1871 bytes --]

On Sun, Dec 16 2018, Florian Fainelli wrote:

> On December 16, 2018 3:19:22 PM PST, NeilBrown <neil@brown.name> wrote:
>>On Sun, Dec 16 2018, David Miller wrote:
>>
>>> From: NeilBrown <neil@brown.name>
>>> Date: Mon, 17 Dec 2018 09:08:54 +1100
>>>
>>>> In my 4.4 kernel, the build_skb() call in (the equivalent of)
>>>> mtk_poll_rx() takes about 1.2usec and the call to napi_gro_receive()
>>>> takes about 3usec.
>>>> 
>>>> In my 4.20 kernel, these calls take about 30 and 24 usec
>>respectively.
>>>> This easily explains the slowdown.
>>>
>>> That's a huge difference.
>>>
>>> Nothing jumps out as a possible cause except perhaps retpoline or
>>> something like that.
>>
>>I'll keep that in mind - thanks.
>>
>>My guess was CPU-cache invalidation.
>>I just checked and the other CPU core (there are two - each
>>hyper-threaded - "other" meaning not the one that handles ethernet
>>interrupts) gets several thousand "IPI resched" interrupts while
>>running a 10 second (226MByte) iperf3 receive test.
>>About 17KB transferred per IPI.
>>I cannot see where build_skb() would do cache invalidation though.
>
> It doesn't the driver is responsible for that. How is coherency maintained between cores?

I suspect so - yes.  Coherency only needs explicit management with DMA
is used.  This wasn't the problem.

Further investigation showed that the problem was that I had
CONFIG_SLUB_DEBUG set.  That was probably useful in some earlier
debugging exercise, but it clearly isn't useful when performance-testing
the network.
I removed that and I have much nicer numbers - not quite the consistent
900+ that I saw with 4.4, but a lot closer.

Thanks for the encouragement, and sorry of the noise.

NeilBrown


>
> The IPI could be due to receive packet steering, is the MAC multi queue aware on the RX path?
> -- 
> Florian

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]