mvpp2: incorrect max mtu?

mvpp2: incorrect max mtu?

[Russell King (Oracle)]

Hi all,

While testing out the 10G speeds on my Macchiatobin platforms, the first
thing I notice is that they only manage about 1Gbps at a MTU of 1500.
As expected, this increases when the MTU is increased - a MTU of 9000
works, and gives a useful performance boost.

Then comes the obvious question - what is the maximum MTU.

#define MVPP2_BM_JUMBO_FRAME_SIZE       10432   /* frame size 9856 */

So, one may assume that 9856 is the maximum. However:

# ip li set dev eth0 mtu 9888
# ip li set dev eth0 mtu 9889
Error: mtu greater than device maximum.

So, the maximum that userspace can set appears to be 9888. If this is
set, then, while running iperf3, we get:

mvpp2 f2000000.ethernet eth0: bad rx status 9202e510 (resource error), size=9888

So clearly this is too large, and we should not be allowing userspace
to set this large a MTU.

At this point, it seems to be impossible to regain the previous speed of
the interface by lowering the MTU. Here is a MTU of 9000:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.37 MBytes  11.5 Mbits/sec   40   17.5 KBytes       
[  5]   1.00-2.00   sec  1.25 MBytes  10.5 Mbits/sec   39   8.74 KBytes       
[  5]   2.00-3.00   sec  1.13 MBytes  9.45 Mbits/sec   36   17.5 KBytes       
[  5]   3.00-4.00   sec  1.13 MBytes  9.45 Mbits/sec   39   8.74 KBytes       
[  5]   4.00-5.00   sec  1.13 MBytes  9.45 Mbits/sec   36   17.5 KBytes       
[  5]   5.00-6.00   sec  1.28 MBytes  10.7 Mbits/sec   39   8.74 KBytes       
[  5]   6.00-7.00   sec  1.13 MBytes  9.45 Mbits/sec   36   17.5 KBytes       
[  5]   7.00-8.00   sec  1.25 MBytes  10.5 Mbits/sec   39   8.74 KBytes       
[  5]   8.00-9.00   sec  1.13 MBytes  9.45 Mbits/sec   36   17.5 KBytes       
[  5]   9.00-10.00  sec  1.13 MBytes  9.45 Mbits/sec   39   8.74 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.9 MBytes  9.99 Mbits/sec  379             sender
[  5]   0.00-10.00  sec  11.7 MBytes  9.80 Mbits/sec                  receiver

Whereas before the test, it was:

[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   729 MBytes  6.11 Gbits/sec
[  5]   1.00-2.00   sec   719 MBytes  6.03 Gbits/sec
[  5]   2.00-3.00   sec   773 MBytes  6.49 Gbits/sec
[  5]   3.00-4.00   sec   769 MBytes  6.45 Gbits/sec
[  5]   4.00-5.00   sec   779 MBytes  6.54 Gbits/sec
[  5]   5.00-6.00   sec   784 MBytes  6.58 Gbits/sec
[  5]   6.00-7.00   sec   777 MBytes  6.52 Gbits/sec
[  5]   7.00-8.00   sec   774 MBytes  6.50 Gbits/sec
[  5]   8.00-9.00   sec   769 MBytes  6.45 Gbits/sec
[  5]   9.00-10.00  sec   774 MBytes  6.49 Gbits/sec
[  5]  10.00-10.00  sec  3.07 MBytes  5.37 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  7.47 GBytes  6.41 Gbits/sec                  receiver

(this is on the server end of iperf3, the others are the client end,
but the results were pretty very similar to that.)

So, clearly something bad has happened to the buffer management as a
result of raising the MTU so high.

As the end which has suffered this issue is the mcbin VM host, I'm not
currently in a position I can reboot it without cause major disruption
to my network. However, thoughts on this (and... can others reproduce
it) would be useful.

Thanks.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

Re: mvpp2: incorrect max mtu?

[Marcin Wojtas]

Hi Russell,


pt., 14 maj 2021 o 15:00 Russell King (Oracle) <linux@armlinux.org.uk>
napisał(a):
>
> Hi all,
>
> While testing out the 10G speeds on my Macchiatobin platforms, the first
> thing I notice is that they only manage about 1Gbps at a MTU of 1500.
> As expected, this increases when the MTU is increased - a MTU of 9000
> works, and gives a useful performance boost.
>
> Then comes the obvious question - what is the maximum MTU.
>
> #define MVPP2_BM_JUMBO_FRAME_SIZE       10432   /* frame size 9856 */
>
> So, one may assume that 9856 is the maximum. However:
>
> # ip li set dev eth0 mtu 9888
> # ip li set dev eth0 mtu 9889
> Error: mtu greater than device maximum.
>
> So, the maximum that userspace can set appears to be 9888. If this is
> set, then, while running iperf3, we get:
>
> mvpp2 f2000000.ethernet eth0: bad rx status 9202e510 (resource error), size=9888
>
> So clearly this is too large, and we should not be allowing userspace
> to set this large a MTU.
>
> At this point, it seems to be impossible to regain the previous speed of
> the interface by lowering the MTU. Here is a MTU of 9000:
>
> [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
> [  5]   0.00-1.00   sec  1.37 MBytes  11.5 Mbits/sec   40   17.5 KBytes
> [  5]   1.00-2.00   sec  1.25 MBytes  10.5 Mbits/sec   39   8.74 KBytes
> [  5]   2.00-3.00   sec  1.13 MBytes  9.45 Mbits/sec   36   17.5 KBytes
> [  5]   3.00-4.00   sec  1.13 MBytes  9.45 Mbits/sec   39   8.74 KBytes
> [  5]   4.00-5.00   sec  1.13 MBytes  9.45 Mbits/sec   36   17.5 KBytes
> [  5]   5.00-6.00   sec  1.28 MBytes  10.7 Mbits/sec   39   8.74 KBytes
> [  5]   6.00-7.00   sec  1.13 MBytes  9.45 Mbits/sec   36   17.5 KBytes
> [  5]   7.00-8.00   sec  1.25 MBytes  10.5 Mbits/sec   39   8.74 KBytes
> [  5]   8.00-9.00   sec  1.13 MBytes  9.45 Mbits/sec   36   17.5 KBytes
> [  5]   9.00-10.00  sec  1.13 MBytes  9.45 Mbits/sec   39   8.74 KBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bitrate         Retr
> [  5]   0.00-10.00  sec  11.9 MBytes  9.99 Mbits/sec  379             sender
> [  5]   0.00-10.00  sec  11.7 MBytes  9.80 Mbits/sec                  receiver
>
> Whereas before the test, it was:
>
> [ ID] Interval           Transfer     Bitrate
> [  5]   0.00-1.00   sec   729 MBytes  6.11 Gbits/sec
> [  5]   1.00-2.00   sec   719 MBytes  6.03 Gbits/sec
> [  5]   2.00-3.00   sec   773 MBytes  6.49 Gbits/sec
> [  5]   3.00-4.00   sec   769 MBytes  6.45 Gbits/sec
> [  5]   4.00-5.00   sec   779 MBytes  6.54 Gbits/sec
> [  5]   5.00-6.00   sec   784 MBytes  6.58 Gbits/sec
> [  5]   6.00-7.00   sec   777 MBytes  6.52 Gbits/sec
> [  5]   7.00-8.00   sec   774 MBytes  6.50 Gbits/sec
> [  5]   8.00-9.00   sec   769 MBytes  6.45 Gbits/sec
> [  5]   9.00-10.00  sec   774 MBytes  6.49 Gbits/sec
> [  5]  10.00-10.00  sec  3.07 MBytes  5.37 Gbits/sec
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bitrate
> [  5]   0.00-10.00  sec  7.47 GBytes  6.41 Gbits/sec                  receiver
>
> (this is on the server end of iperf3, the others are the client end,
> but the results were pretty very similar to that.)
>
> So, clearly something bad has happened to the buffer management as a
> result of raising the MTU so high.
>
> As the end which has suffered this issue is the mcbin VM host, I'm not
> currently in a position I can reboot it without cause major disruption
> to my network. However, thoughts on this (and... can others reproduce
> it) would be useful.
>

Thank your for the information. I will take a look after the weekend.
To be aligned - what exactly kernel baseline are you using?

Best regards,
Marcin

Re: mvpp2: incorrect max mtu?

[Russell King (Oracle)]

On Fri, May 14, 2021 at 04:25:48PM +0200, Marcin Wojtas wrote:
> Thank your for the information. I will take a look after the weekend.
> To be aligned - what exactly kernel baseline are you using?

That was with 5.11 with these additional mvpp2 patches:

net: mvpp2: add TX FC firmware check
net: mvpp2: set 802.3x GoP Flow Control mode
net: mvpp2: add PPv23 RX FIFO flow control
net: mvpp2: add BM protection underrun feature support
net: mvpp2: add ethtool flow control configuration support
net: mvpp2: add RXQ flow control configurations
net: mvpp2: enable global flow control
net: mvpp2: add FCA RXQ non occupied descriptor threshold
net: mvpp2: add FCA periodic timer configurations
net: mvpp2: increase BM pool and RXQ size
net: mvpp2: add PPv23 version definition
net: mvpp2: always compare hw-version vs MVPP21
net: mvpp2: add CM3 SRAM memory map
dts: marvell: add CM3 SRAM memory to cp11x ethernet device tree
doc: marvell: add CM3 address space and PPv2.3 description
net: marvell: Fixed two spellings,controling to controlling and oen to one
net: mvpp2: prs: improve ipv4 parse flow

I'll also try to work out what's happening, but I think we need to find
out what the correct value for dev->max_mtu should be. That's all rather
convoluted:

	dev->max_mtu = MVPP2_BM_JUMBO_PKT_SIZE;

#define MVPP2_BM_JUMBO_PKT_SIZE MVPP2_RX_MAX_PKT_SIZE(MVPP2_BM_JUMBO_FRAME_SIZE)
#define MVPP2_BM_JUMBO_FRAME_SIZE       10432   /* frame size 9856 */
#define MVPP2_RX_MAX_PKT_SIZE(total_size) \
        ((total_size) - MVPP2_SKB_HEADROOM - MVPP2_SKB_SHINFO_SIZE)

The maximum settable MTU on eth0 (9888) disagrees with the comment
"frame size 9856" by 32 bytes.

I haven't checked to see whether 9856 works yet, because that will
first need me to reboot the machine... which I'll do over the weekend.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

Re: mvpp2: incorrect max mtu?

[Andrew Lunn]

On Fri, May 14, 2021 at 02:00:18PM +0100, Russell King (Oracle) wrote:
> Hi all,
> 
> While testing out the 10G speeds on my Macchiatobin platforms, the first
> thing I notice is that they only manage about 1Gbps at a MTU of 1500.
> As expected, this increases when the MTU is increased - a MTU of 9000
> works, and gives a useful performance boost.
> 
> Then comes the obvious question - what is the maximum MTU.
> 
> #define MVPP2_BM_JUMBO_FRAME_SIZE       10432   /* frame size 9856 */
> 
> So, one may assume that 9856 is the maximum. However:
> 
> # ip li set dev eth0 mtu 9888
> # ip li set dev eth0 mtu 9889
> Error: mtu greater than device maximum.

Hi Russell

It all seems inconsistent:

https://elixir.bootlin.com/linux/latest/source/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c#L6879

	/* MTU range: 68 - 9704 */
	dev->min_mtu = ETH_MIN_MTU;
	/* 9704 == 9728 - 20 and rounding to 8 */
	dev->max_mtu = MVPP2_BM_JUMBO_PKT_SIZE;

Maybe this comment is correct, the code is now wrong, and the MAX MTU
should be 9704?

      Andrew

Re: mvpp2: incorrect max mtu?

[Russell King (Oracle)]

On Fri, May 14, 2021 at 04:35:12PM +0200, Andrew Lunn wrote:
> On Fri, May 14, 2021 at 02:00:18PM +0100, Russell King (Oracle) wrote:
> > Hi all,
> > 
> > While testing out the 10G speeds on my Macchiatobin platforms, the first
> > thing I notice is that they only manage about 1Gbps at a MTU of 1500.
> > As expected, this increases when the MTU is increased - a MTU of 9000
> > works, and gives a useful performance boost.
> > 
> > Then comes the obvious question - what is the maximum MTU.
> > 
> > #define MVPP2_BM_JUMBO_FRAME_SIZE       10432   /* frame size 9856 */
> > 
> > So, one may assume that 9856 is the maximum. However:
> > 
> > # ip li set dev eth0 mtu 9888
> > # ip li set dev eth0 mtu 9889
> > Error: mtu greater than device maximum.
> 
> Hi Russell
> 
> It all seems inconsistent:
> 
> https://elixir.bootlin.com/linux/latest/source/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c#L6879
> 
> 	/* MTU range: 68 - 9704 */
> 	dev->min_mtu = ETH_MIN_MTU;
> 	/* 9704 == 9728 - 20 and rounding to 8 */
> 	dev->max_mtu = MVPP2_BM_JUMBO_PKT_SIZE;
> 
> Maybe this comment is correct, the code is now wrong, and the MAX MTU
> should be 9704?

Oh, there's more values given elsewhere that disagree, see my reply to
Marcin. I would not be surprised if this was all "confused" about
what the proper value is. Certainly the comment you mention above
disagrees with what is in mvpp2.h

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

RE: [EXT] mvpp2: incorrect max mtu?

[Stefan Chulski]

> -----Original Message-----
> From: Russell King <linux@armlinux.org.uk>
> Sent: Friday, May 14, 2021 4:00 PM
> To: Stefan Chulski <stefanc@marvell.com>
> Cc: Marcin Wojtas <mw@semihalf.com>; netdev@vger.kernel.org
> Subject: [EXT] mvpp2: incorrect max mtu?
> 
> External Email
> 
> ----------------------------------------------------------------------
> Hi all,
> 
> While testing out the 10G speeds on my Macchiatobin platforms, the first
> thing I notice is that they only manage about 1Gbps at a MTU of 1500.
> As expected, this increases when the MTU is increased - a MTU of 9000
> works, and gives a useful performance boost.
> 
> Then comes the obvious question - what is the maximum MTU.
> 
> #define MVPP2_BM_JUMBO_FRAME_SIZE       10432   /* frame size 9856 */
> 
> So, one may assume that 9856 is the maximum. However:
> 
> # ip li set dev eth0 mtu 9888
> # ip li set dev eth0 mtu 9889
> Error: mtu greater than device maximum.
> 
> So, the maximum that userspace can set appears to be 9888. If this is set,
> then, while running iperf3, we get:
> 
> mvpp2 f2000000.ethernet eth0: bad rx status 9202e510 (resource error),
> size=9888
> 
> So clearly this is too large, and we should not be allowing userspace to set
> this large a MTU.
> 
> At this point, it seems to be impossible to regain the previous speed of the
> interface by lowering the MTU. Here is a MTU of 9000:
> 
> [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
> [  5]   0.00-1.00   sec  1.37 MBytes  11.5 Mbits/sec   40   17.5 KBytes
> [  5]   1.00-2.00   sec  1.25 MBytes  10.5 Mbits/sec   39   8.74 KBytes
> [  5]   2.00-3.00   sec  1.13 MBytes  9.45 Mbits/sec   36   17.5 KBytes
> [  5]   3.00-4.00   sec  1.13 MBytes  9.45 Mbits/sec   39   8.74 KBytes
> [  5]   4.00-5.00   sec  1.13 MBytes  9.45 Mbits/sec   36   17.5 KBytes
> [  5]   5.00-6.00   sec  1.28 MBytes  10.7 Mbits/sec   39   8.74 KBytes
> [  5]   6.00-7.00   sec  1.13 MBytes  9.45 Mbits/sec   36   17.5 KBytes
> [  5]   7.00-8.00   sec  1.25 MBytes  10.5 Mbits/sec   39   8.74 KBytes
> [  5]   8.00-9.00   sec  1.13 MBytes  9.45 Mbits/sec   36   17.5 KBytes
> [  5]   9.00-10.00  sec  1.13 MBytes  9.45 Mbits/sec   39   8.74 KBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bitrate         Retr
> [  5]   0.00-10.00  sec  11.9 MBytes  9.99 Mbits/sec  379             sender
> [  5]   0.00-10.00  sec  11.7 MBytes  9.80 Mbits/sec                  receiver
> 
> Whereas before the test, it was:
> 
> [ ID] Interval           Transfer     Bitrate
> [  5]   0.00-1.00   sec   729 MBytes  6.11 Gbits/sec
> [  5]   1.00-2.00   sec   719 MBytes  6.03 Gbits/sec
> [  5]   2.00-3.00   sec   773 MBytes  6.49 Gbits/sec
> [  5]   3.00-4.00   sec   769 MBytes  6.45 Gbits/sec
> [  5]   4.00-5.00   sec   779 MBytes  6.54 Gbits/sec
> [  5]   5.00-6.00   sec   784 MBytes  6.58 Gbits/sec
> [  5]   6.00-7.00   sec   777 MBytes  6.52 Gbits/sec
> [  5]   7.00-8.00   sec   774 MBytes  6.50 Gbits/sec
> [  5]   8.00-9.00   sec   769 MBytes  6.45 Gbits/sec
> [  5]   9.00-10.00  sec   774 MBytes  6.49 Gbits/sec
> [  5]  10.00-10.00  sec  3.07 MBytes  5.37 Gbits/sec
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bitrate
> [  5]   0.00-10.00  sec  7.47 GBytes  6.41 Gbits/sec                  receiver
> 
> (this is on the server end of iperf3, the others are the client end, but the
> results were pretty very similar to that.)
> 
> So, clearly something bad has happened to the buffer management as a
> result of raising the MTU so high.
> 
> As the end which has suffered this issue is the mcbin VM host, I'm not
> currently in a position I can reboot it without cause major disruption to my
> network. However, thoughts on this (and... can others reproduce
> it) would be useful.
> 
> Thanks.

Look like PPv2 tried scatter frame since it was larger than Jumbo buffer size and it drained buffer pool(Buffers never released).
Received packet should be less than value set in MVPP2_POOL_BUF_SIZE_REG for long pool.

Stefan.
 

Re: [EXT] mvpp2: incorrect max mtu?

[Russell King (Oracle)]

On Tue, May 18, 2021 at 06:09:12AM +0000, Stefan Chulski wrote:
> Look like PPv2 tried scatter frame since it was larger than Jumbo buffer size and it drained buffer pool(Buffers never released).
> Received packet should be less than value set in MVPP2_POOL_BUF_SIZE_REG for long pool.

So this must mean that setting dev->max_mtu is incorrect.

From what I can see, the value programmed into that register would be
MVPP2_BM_JUMBO_PKT_SIZE which I believe is 9888. This is currently the
same value that dev->max_mtu is set to, but max_mtu is the data
payload size in the ethernet frame, which doesn't include the hardware
ethernet header.

So, should max_mtu be set to 14 bytes less? Or should it be set to
9856? Less 14 bytes? Or what?

It is really confusing that we have these definitions that state e.g.
that JUMBO_FRAME_SIZE is 10432 but the frame size comment says 9856.
It's not clear why it's different like that - why the additional 576
octets.

All of this could do with some explanation in the driver - would it be
possible to add some kind of documentation, or at least make the
definitions around packet and frame size more understandable please?

Thanks.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

RE: [EXT] mvpp2: incorrect max mtu?

[Stefan Chulski]

> -----Original Message-----
> From: Russell King <linux@armlinux.org.uk>
> Sent: Tuesday, May 18, 2021 12:42 PM
> To: Stefan Chulski <stefanc@marvell.com>
> Cc: Marcin Wojtas <mw@semihalf.com>; netdev@vger.kernel.org
> Subject: Re: [EXT] mvpp2: incorrect max mtu?
> 
> On Tue, May 18, 2021 at 06:09:12AM +0000, Stefan Chulski wrote:
> > Look like PPv2 tried scatter frame since it was larger than Jumbo buffer size
> and it drained buffer pool(Buffers never released).
> > Received packet should be less than value set in
> MVPP2_POOL_BUF_SIZE_REG for long pool.
> 
> So this must mean that setting dev->max_mtu is incorrect.
> 
> From what I can see, the value programmed into that register would be
> MVPP2_BM_JUMBO_PKT_SIZE which I believe is 9888. This is currently the
> same value that dev->max_mtu is set to, but max_mtu is the data payload
> size in the ethernet frame, which doesn't include the hardware ethernet
> header.
> 
> So, should max_mtu be set to 14 bytes less? Or should it be set to 9856? Less
> 14 bytes? Or what?

Yes, look like dev->max_mtuis incorrect. It should be 9856, MVPP2_POOL_BUF_SIZE_REG,  Pool Buffer Size is in 32 units.
But before changing it I prefer to test different sizes. Currently, I don't have a connected board. 
I will connect board and perform some tests. JUMBO_FRAME_SIZE is the size of the buffer, but packet offset should be taken into account.

> It is really confusing that we have these definitions that state e.g.
> that JUMBO_FRAME_SIZE is 10432 but the frame size comment says 9856.
> It's not clear why it's different like that - why the additional 576 octets.
> 
> All of this could do with some explanation in the driver - would it be possible
> to add some kind of documentation, or at least make the definitions around
> packet and frame size more understandable please?
> 
> Thanks.

Yes, definition or comments should be improved. 

Regards,
Stefan. 

RE: [EXT] mvpp2: incorrect max mtu?

[Stefan Chulski]

> > -----Original Message-----
> > From: Russell King <linux@armlinux.org.uk>
> > Sent: Tuesday, May 18, 2021 12:42 PM
> > To: Stefan Chulski <stefanc@marvell.com>
> > Cc: Marcin Wojtas <mw@semihalf.com>; netdev@vger.kernel.org
> > Subject: Re: [EXT] mvpp2: incorrect max mtu?
> >
> > On Tue, May 18, 2021 at 06:09:12AM +0000, Stefan Chulski wrote:
> > > Look like PPv2 tried scatter frame since it was larger than Jumbo
> > > buffer size
> > and it drained buffer pool(Buffers never released).
> > > Received packet should be less than value set in
> > MVPP2_POOL_BUF_SIZE_REG for long pool.
> >
> > So this must mean that setting dev->max_mtu is incorrect.
> >
> > From what I can see, the value programmed into that register would be
> > MVPP2_BM_JUMBO_PKT_SIZE which I believe is 9888. This is currently the
> > same value that dev->max_mtu is set to, but max_mtu is the data
> > payload size in the ethernet frame, which doesn't include the hardware
> > ethernet header.
> >
> > So, should max_mtu be set to 14 bytes less? Or should it be set to
> > 9856? Less
> > 14 bytes? Or what?
> 
> Yes, look like dev->max_mtuis incorrect. It should be 9856,
> MVPP2_POOL_BUF_SIZE_REG,  Pool Buffer Size is in 32 units.
> But before changing it I prefer to test different sizes. Currently, I don't have a
> connected board.
> I will connect board and perform some tests. JUMBO_FRAME_SIZE is the size
> of the buffer, but packet offset should be taken into account.
> 
> > It is really confusing that we have these definitions that state e.g.
> > that JUMBO_FRAME_SIZE is 10432 but the frame size comment says 9856.
> > It's not clear why it's different like that - why the additional 576 octets.
> >
> > All of this could do with some explanation in the driver - would it be
> > possible to add some kind of documentation, or at least make the
> > definitions around packet and frame size more understandable please?
> >
> > Thanks.
> 
> Yes, definition or comments should be improved.
> 
> Regards,
> Stefan.

We probably should split this to 3 issues.
1. BM drain issue. I can post patch witch should fix it. So after lowering the MTU you won't have issue.
2. Regarding MTU max, here we actually talking about max MRU not MTU.
MRU set by mvpp2_xlg_max_rx_size_set/mvpp2_gmac_max_rx_size_set port->pkt_size - MVPP2_MH_SIZE(2 Bytes).
port->pkt_size = MTU +  MVPP2_MH_SIZE(2 Bytes) + VLAN tag (4 Bytes) + ETH_HLEN(14 Bytes) + ETH_FCS_LEN(4 Bytes) and from some reason aligned to cache_line_size().
So if we set MTU = 9888, we actually get MRU 9918.
3. Fix all max MTU comments in code.

Regards,
Stefan.