Full 40Gbps with i40e for large packets?

Full 40Gbps with i40e for large packets?

[Srivats P]

Hi,

Using the xdpsock txonly benchmark in zero-copy mode with the i40e
driver, I get ~22 Mpps for 64 Byte packets.

However, for 1518 byte packets, although the theoretical pps rate for
40Gbps is ~3.25 Mpps (which is much lower than 22 Mpps), I am not able
to get the full rate - only around 39G or 39.5Gbps (i.e. < 3.25Mpps).
My assumption is AF_XDP performance is more pps dependent than packet
size related and so I thought full 40G should be achievable.

Settings:
 * tuned set to "throughput performance"
 * hardware tx ring size set to 512 descriptors - increasing it to up
to the max of 4096 doesn't have any noticeable changes

Increasing the packet size to 3000 or 4096 does bump up the Gbps rate
to around 39.9Gbps.

Is there a way to get the full 40Gbps (or close, say 39.9Gbps) tx rate
with 1518 byte packets?

Srivats

Re: Full 40Gbps with i40e for large packets?

[Willy Tarreau]

Hi,

On Fri, Jun 24, 2022 at 06:51:33PM +0530, Srivats P wrote:
> Hi,
> 
> Using the xdpsock txonly benchmark in zero-copy mode with the i40e
> driver, I get ~22 Mpps for 64 Byte packets.
> 
> However, for 1518 byte packets, although the theoretical pps rate for
> 40Gbps is ~3.25 Mpps (which is much lower than 22 Mpps), I am not able
> to get the full rate - only around 39G or 39.5Gbps (i.e. < 3.25Mpps).
> My assumption is AF_XDP performance is more pps dependent than packet
> size related and so I thought full 40G should be achievable.
> 
> Settings:
>  * tuned set to "throughput performance"
>  * hardware tx ring size set to 512 descriptors - increasing it to up
> to the max of 4096 doesn't have any noticeable changes
> 
> Increasing the packet size to 3000 or 4096 does bump up the Gbps rate
> to around 39.9Gbps.
> 
> Is there a way to get the full 40Gbps (or close, say 39.9Gbps) tx rate
> with 1518 byte packets?

While I don't know for the XDP part itself, you should really not speak
in terms of bps but in terms of pps exclusively. The reason is that it
is very difficult otherwise to know what layer you're measuring.

In addition, 40 Gbps is 4*10GbE which itself is exactly 10G on copper but
can be a bit more (10.3125G) on fiber depending on the type (and maybe a
bit less on single-mode as well though I don't remember having seen that
in a while), so taking that into account can make it even harder to observe
an exactly round value.

So assuming that you're running on copper with 40.000 Gbps, this is the
on-wire bit rate, that includes the 8-byte preamble, the visible eth frame,
the FCS (4 bytes) and the IFG (12 bytes), hence 1538 bytes for the regular
1514 bytes. This gives a maximum frame rate of 3250975 pps for such frames,
which is what you'd get if by 1518 you included the FCS, or 3242542 pps if
by 1518 you included a VLAN header (hence 1542 bytes on the wire).

If we take that last example that sounds like the most plausible, that
gives you a layer2 max bit rate of 3242542*1518*8 = 39.377 Gbps. As such,
depending on how and where you measure, it's very possible that you've
totally filled the wire.

Thus as a conclusion: forget about bit rate and only count packet rates.

Hoping this helps,
Willy

Re: Full 40Gbps with i40e for large packets?

[Srivats P]

> So assuming that you're running on copper with 40.000 Gbps, this is the
> on-wire bit rate, that includes the 8-byte preamble, the visible eth frame,
> the FCS (4 bytes) and the IFG (12 bytes), hence 1538 bytes for the regular
> 1514 bytes. This gives a maximum frame rate of 3250975 pps for such frames,
> which is what you'd get if by 1518 you included the FCS, or 3242542 pps if
> by 1518 you included a VLAN header (hence 1542 bytes on the wire).
>
> If we take that last example that sounds like the most plausible, that
> gives you a layer2 max bit rate of 3242542*1518*8 = 39.377 Gbps. As such,
> depending on how and where you measure, it's very possible that you've
> totally filled the wire.

Yes, I'm using 40G copper. 1518 byte is including FCS and no VLANs, so
the pps expected to fully fill the pipe as you calculate rightly is
3250975 or 3.25 Mpps.

The rate I get varies continuously between 3.19 and 3.22 Mpps and very
rarely goes up to 3.24 Mpps. The few times I've seen 3.24 Mpps is at
the immediate start of a run, but not afterwards - a behaviour I'm not
able to explain.

I'm trying to understand why I get a lower rate than 3.250975 Mpps and
what I need to do to get the full pps rate.

Srivats

Re: Full 40Gbps with i40e for large packets?

[Willy Tarreau]

On Fri, Jun 24, 2022 at 09:23:51PM +0530, Srivats P wrote:
> Yes, I'm using 40G copper. 1518 byte is including FCS and no VLANs, so
> the pps expected to fully fill the pipe as you calculate rightly is
> 3250975 or 3.25 Mpps.

OK.

> The rate I get varies continuously between 3.19 and 3.22 Mpps and very
> rarely goes up to 3.24 Mpps. The few times I've seen 3.24 Mpps is at
> the immediate start of a run, but not afterwards - a behaviour I'm not
> able to explain.
> 
> I'm trying to understand why I get a lower rate than 3.250975 Mpps and
> what I need to do to get the full pps rate.

In other contexts I've faced causes such as too large tx_irq delay,
causing the tx ring to be empty when the code was able to deliver
more, and too small PCIe payload sizes (I remember having had to
change it to 4096 in a machine's BIOS years ago for a 100G test).

Beyond this I have no idea :-)

Willy

Re: Full 40Gbps with i40e for large packets?

[Willy Tarreau]

On Fri, Jun 24, 2022 at 09:56:22PM +0400, Alexander Petrovsky wrote:
> ??, 24 ???? 2022 ?. ? 17:57, Willy Tarreau <w@1wt.eu>:
> 
> > Hi,
> >
> > On Fri, Jun 24, 2022 at 06:51:33PM +0530, Srivats P wrote:
> > > Hi,
> > >
> > > Using the xdpsock txonly benchmark in zero-copy mode with the i40e
> > > driver, I get ~22 Mpps for 64 Byte packets.
> > >
> > > However, for 1518 byte packets, although the theoretical pps rate for
> > > 40Gbps is ~3.25 Mpps (which is much lower than 22 Mpps), I am not able
> > > to get the full rate - only around 39G or 39.5Gbps (i.e. < 3.25Mpps).
> > > My assumption is AF_XDP performance is more pps dependent than packet
> > > size related and so I thought full 40G should be achievable.
> > >
> > > Settings:
> > >  * tuned set to "throughput performance"
> > >  * hardware tx ring size set to 512 descriptors - increasing it to up
> > > to the max of 4096 doesn't have any noticeable changes
> > >
> > > Increasing the packet size to 3000 or 4096 does bump up the Gbps rate
> > > to around 39.9Gbps.
> > >
> > > Is there a way to get the full 40Gbps (or close, say 39.9Gbps) tx rate
> > > with 1518 byte packets?
> >
> > While I don't know for the XDP part itself, you should really not speak
> > in terms of bps but in terms of pps exclusively. The reason is that it
> > is very difficult otherwise to know what layer you're measuring.
> >
> > In addition, 40 Gbps is 4*10GbE which itself is exactly 10G on copper but
> > can be a bit more (10.3125G) on fiber depending on the type (and maybe a
> > bit less on single-mode as well though I don't remember having seen that
> > in a while), so taking that into account can make it even harder to observe
> > an exactly round value.
> >
> > So assuming that you're running on copper with 40.000 Gbps, this is the
> > on-wire bit rate, that includes the 8-byte preamble, the visible eth frame,
> > the FCS (4 bytes) and the IFG (12 bytes), hence 1538 bytes for the regular
> > 1514 bytes. This gives a maximum frame rate of 3250975 pps for such frames,
> > which is what you'd get if by 1518 you included the FCS, or 3242542 pps if
> > by 1518 you included a VLAN header (hence 1542 bytes on the wire).
> >
> > If we take that last example that sounds like the most plausible, that
> > gives you a layer2 max bit rate of 3242542*1518*8 = 39.377 Gbps. As such,
> > depending on how and where you measure, it's very possible that you've
> > totally filled the wire.
> >
> 
> Can't catch your math here. 40Gbps it's a 5GB. In this case, 5368709120 /
> 1518 = 3536699 frames, but it's not match with your 3250975. Could you
> point please where I'm mistaken?

The on-wire frames are not 1518 but 1538 because there's a 8-byte preamble
and a 12-byte inter-frame-gap. In addition, network bandwidths do not use
powers-of-two but powers-of-ten. Thus 40 Gbps is 40.10^9 bits per second
hence 5.10^9 bytes per second. 5000000000 / 1538 = 3250975.29...

Usually you don't see the FCS so you're seeing ethernet frames of 1514
(or 1518 when VLANs are used or 1522 when q-in-q is used). Thus you just
have to add 24 bytes to whatever ethernet packet size you're considering.
Just keep in mind that the smallest frame from dest addr to FCS is 64B,
hence 84B including preamble and IFG. This explains why the max frame
rate on 10Gbps is 14.88 Mpps.

This same principle explains why when transfering a file using HTTP or FTP
over a GbE link from a server to your laptop, you're seeing a perfectly
stable rate of 117.68 or 118.66 MB/s. Your TCP frames carry 1448 data
bytes (when timestamps are enabled) or 1460 (when disabled), for 1538
on-wire bytes, hence an efficiency ratio of 1448/1538. These values are
easy to remember and convenient to immediately spot that something's
wrong when you don't reach them.

Regards,
Willy