[Intel-wired-lan] Intel E810-XXVDA2: Too high TX timestamp read latency

[Intel-wired-lan] Intel E810-XXVDA2: Too high TX timestamp read latency

[Martin Kulas]

Hello Intel Wired LAN experts,


My time-sensitive user space application reads out
the hardware TX timestamp via poll() and recvmsg(MSG_ERRQUEUE).
On an Intel E810-XXVDA2 (for SFP),  it takes roughly 700 microseconds 
until the TX timestamp is available in user space (Linux 6.7.4).
This latency of 700 microseconds is far too high for my application.
For comparision,  on an Intel X550 card (Twisted pair),
my application retrieves the TX timestamp in roughly 30 microseconds
which is acceptable.


The output of ethtool of the tested E810-XXVDA2:

# ethtool -i enp65s0f1np1
driver: ice
version: 6.7.4
firmware-version: 4.40 0x8001c96c 1.3534.0
expansion-rom-version:
bus-info: 0000:41:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
#

When searching for a solution about this high latency TX timestamp read,
I found the following thread from 2022:

https://patchwork.ozlabs.org/project/intel-wired-lan/patch/20220802145757.273270-1-karol.kolacinski@intel.com/#2963591

Is this patch already in the current Linux Kernel 6.7.4 or is it pending?

Can you recommend anything else in order to reduce
the TX timestamp read latency?

If you know a better place for this issue,  please let me now!


Kind regards,
Martin

Re: [Intel-wired-lan] Intel E810-XXVDA2: Too high TX timestamp read latency

[Paul Menzel]

[Cc: +Karol]

Dear Martin,


Am 16.02.24 um 15:18 schrieb Martin Kulas:

> My time-sensitive user space application reads out
> the hardware TX timestamp via poll() and recvmsg(MSG_ERRQUEUE).
> On an Intel E810-XXVDA2 (for SFP), it takes roughly 700 microseconds 
> until the TX timestamp is available in user space (Linux 6.7.4).
> This latency of 700 microseconds is far too high for my application.
> For comparison, on an Intel X550 card (Twisted pair),
> my application retrieves the TX timestamp in roughly 30 microseconds
> which is acceptable.
> 
> The output of ethtool of the tested E810-XXVDA2:
> 
> # ethtool -i enp65s0f1np1
> driver: ice
> version: 6.7.4
> firmware-version: 4.40 0x8001c96c 1.3534.0
> expansion-rom-version:
> bus-info: 0000:41:00.1
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: yes
> #
> 
> When searching for a solution about this high latency TX timestamp read,
> I found the following thread from 2022:
> 
> https://patchwork.ozlabs.org/project/intel-wired-lan/patch/20220802145757.273270-1-karol.kolacinski@intel.com/#2963591
> 
> Is this patch already in the current Linux Kernel 6.7.4 or is it pending?

In the metadata on the patchwork page it says *Accepted*, and it seems 
to be commit 1229b33973c7 (ice: Add low latency Tx timestamp read) added 
to Linux 6.1-rc1, so it should be in Linux 6.1.

Please build the Linux kernel from this commit to rule out any 
regression. (If that commit actually should support such low times that 
you require. The commit message says something about 20 ms before.)

> Can you recommend anything else in order to reduce the TX timestamp
> read latency?
> 
> If you know a better place for this issue,  please let me now!
I think this is the right place.


Kind regards,

Paul


[1]: 
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1229b33973c7b89e989945a3edae7a17b6f909da

Re: [Intel-wired-lan] Intel E810-XXVDA2: Too high TX timestamp read latency

[Martin Kulas]

Dear Paul,

On Fri, Feb 16, 2024 at 10:18:03PM +0100, Paul Menzel wrote:
> 
> Am 16.02.24 um 15:18 schrieb Martin Kulas:
> 
> > My time-sensitive user space application reads out
> > the hardware TX timestamp via poll() and recvmsg(MSG_ERRQUEUE).
> > On an Intel E810-XXVDA2 (for SFP), it takes roughly 700 microseconds
> > until the TX timestamp is available in user space (Linux 6.7.4).
> > This latency of 700 microseconds is far too high for my application.
> > For comparison, on an Intel X550 card (Twisted pair),
> > my application retrieves the TX timestamp in roughly 30 microseconds
> > which is acceptable.
> > 

[...]

> > 
> > When searching for a solution about this high latency TX timestamp read,
> > I found the following thread from 2022:
> > 
> > https://patchwork.ozlabs.org/project/intel-wired-lan/patch/20220802145757.273270-1-karol.kolacinski@intel.com/#2963591
> > 
> > Is this patch already in the current Linux Kernel 6.7.4 or is it pending?
> 
> In the metadata on the patchwork page it says *Accepted*, and it seems to be
> commit 1229b33973c7 (ice: Add low latency Tx timestamp read) added to Linux
> 6.1-rc1, so it should be in Linux 6.1.
> 
> Please build the Linux kernel from this commit to rule out any regression.
> (If that commit actually should support such low times that you require. The
> commit message says something about 20 ms before.)

After having applied that commit,  the TX timestamp read latency stays
unacceptable high (in the range of about 700 microseconds), i.e. no
regression.


Here you find details about the E810-XXVDA2 network card with 
high Tx timestamp read latency:

    $ ethtool enp65s0f1
    Settings for enp65s0f1:
    	Supported ports: [ FIBRE ]
    	Supported link modes:   1000baseT/Full
    	                        10000baseT/Full
    	                        25000baseCR/Full
    	                        25000baseSR/Full
    	                        1000baseX/Full
    	                        10000baseSR/Full
    	                        10000baseLR/Full
    	Supported pause frame use: Symmetric
    	Supports auto-negotiation: No
    	Supported FEC modes: None
    	Advertised link modes:  10000baseT/Full
    	Advertised pause frame use: No
    	Advertised auto-negotiation: No
    	Advertised FEC modes: None
    	Speed: 10000Mb/s
    	Duplex: Full
    	Auto-negotiation: off
    	Port: Direct Attach Copper
    	PHYAD: 0
    	Transceiver: internal
    	Supports Wake-on: d
    	Wake-on: d
            Current message level: 0x00000007 (7)
                                   drv probe link
    	Link detected: yes
    $ ethtool -i enp65s0f1
    driver: ice
    version: 6.0.0-rc4-kulas-1229+
    firmware-version: 4.40 0x8001c96c 1.3534.0
    expansion-rom-version:
    bus-info: 0000:41:00.1
    supports-statistics: yes
    supports-test: yes
    supports-eeprom-access: yes
    supports-register-dump: yes
    supports-priv-flags: yes
    $


With an other NIC that supports hardware TX timestamping (Intel X550),
the TX timestamp read latency is much smaller than on Intel E810-XXVDA2,
i.e. 7 microseconds.  I am puzzled that an older Intel network card 
provides the hardware TX timestamp about 100 times faster than 
an new Intel network card.  

Configuration of the X550 network card with 
low Tx timestamp read latency:

    $ ethtool eno3
    Settings for eno3:
    	Supported ports: [ TP ]
    	Supported link modes:   100baseT/Full
    	                        1000baseT/Full
    	                        10000baseT/Full
    	                        2500baseT/Full
    	                        5000baseT/Full
    	Supported pause frame use: Symmetric
    	Supports auto-negotiation: Yes
    	Supported FEC modes: Not reported
    	Advertised link modes:  100baseT/Full
    	                        1000baseT/Full
    	                        10000baseT/Full
    	Advertised pause frame use: Symmetric
    	Advertised auto-negotiation: Yes
    	Advertised FEC modes: Not reported
    	Speed: 10000Mb/s
    	Duplex: Full
    	Auto-negotiation: on
    	Port: Twisted Pair
    	PHYAD: 0
    	Transceiver: internal
    	MDI-X: Unknown
    	Supports Wake-on: umbg
    	Wake-on: g
            Current message level: 0x00000007 (7)
                                   drv probe link
    	Link detected: yes
    $ ethtool -i eno3
    driver: ixgbe
    version: 5.17.12-100.fc34.x86_64
    firmware-version: 0x80001685, 22.0.9
    expansion-rom-version: 
    bus-info: 0000:02:00.0
    supports-statistics: yes
    supports-test: yes
    supports-eeprom-access: yes
    supports-register-dump: yes
    supports-priv-flags: yes
    $


Do you find anything suspicious in the configuration?
Is it a problem of connector types, i.e. twisted pair vs. direct attach?


If you want to experiment by your own,  you find
a minimum program to read out hardware TX timestamp in my ownCloud
space:

https://owncloud.gwdg.de/index.php/s/q7F6IuoynzGGZis


Which tuning knobs exist that allows me to reduce the latency
for reading out the TX timestamp from the network card?


Kind regards,
Martin

Re: [Intel-wired-lan] Intel E810-XXVDA2: Too high TX timestamp read latency

[Kolacinski, Karol]

Dear Martin,

On Tue, Feb 20, 2024 at 2:17PM +0100, Martin Kulas wrote:
[...]
> > > 
> > > When searching for a solution about this high latency TX timestamp read,
> > > I found the following thread from 2022:
> > > 
> > > https://patchwork.ozlabs.org/project/intel-wired-lan/patch/20220802145757.273270-1-karol.kolacinski@intel.com/#2963591
> > > 
> > > Is this patch already in the current Linux Kernel 6.7.4 or is it pending?
> > 
> > In the metadata on the patchwork page it says *Accepted*, and it seems to be
> > commit 1229b33973c7 (ice: Add low latency Tx timestamp read) added to Linux
> > 6.1-rc1, so it should be in Linux 6.1.
> > 
> > Please build the Linux kernel from this commit to rule out any regression.
> > (If that commit actually should support such low times that you require. The
> > commit message says something about 20 ms before.)
> 
> After having applied that commit,  the TX timestamp read latency stays
> unacceptable high (in the range of about 700 microseconds), i.e. no
> regression.

This improvement utilises 'low latency read', where the driver writes
a timestamp request to a specific register, FW has to poll this
register every 0.6 ms and then the driver receives a second interrupt,
so that it can read the timestamp from this register.
Unfortunately, polling every 0.6 ms is the best we could do.

[...]
> With an other NIC that supports hardware TX timestamping (Intel X550),
> the TX timestamp read latency is much smaller than on Intel E810-XXVDA2,
> i.e. 7 microseconds.  I am puzzled that an older Intel network card 
> provides the hardware TX timestamp about 100 times faster than 
> an new Intel network card.  

Intel X550 had a direct PHY register access.
Intel E810 does not have this possibility due to HW design. Only FW can
access PHY registers directly.
Before low latency Tx timestamp patch, the driver used AdminQ message
to get the timestamp, but it was far too slow and inconsistent.

[...]

Kind regards,
Karol

Re: [Intel-wired-lan] Intel E810-XXVDA2: Too high TX timestamp read latency

[Martin Kulas]

[-- Attachment #1: Type: text/plain, Size: 1451 bytes --]

Dear Karol,

On Thu, Feb 22, 2024 at 05:30:09PM +0000, Kolacinski, Karol wrote:
[...]
> 
> On Tue, Feb 20, 2024 at 2:17PM +0100, Martin Kulas wrote:
> > 
> > After having applied that commit,  the TX timestamp read latency stays
> > unacceptable high (in the range of about 700 microseconds), i.e. no
> > regression.
> 
> This improvement utilises 'low latency read', where the driver writes
> a timestamp request to a specific register, FW has to poll this
> register every 0.6 ms and then the driver receives a second interrupt,
> so that it can read the timestamp from this register.
> Unfortunately, polling every 0.6 ms is the best we could do.


Thank you very much for your explanation!

The fact the the firmware (FW) polls the TX timestamp register every 600
microseconds explains the TX timestamp read latency of my application of
700 microseconds.

Since it is a firmware problem:  A updated firmware with customizable 
polling rate would enable low TX hardware timestamp read latencies.  Right?


Currently,  I am looking for other Ethernetcards with TX hardware
timestamping capability.  I am not sure whether the Intel X710-DA2 (i40e) 
supports TX hardware timestamps for all outgoing packets and not only
for PTP packets.  Do you know the timestamping capabilities of that card by
any chance?  The datasheet of that Ethernet card is not completely clear
about it.

[...]


Kind regards,
Martin


[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5414 bytes --]

Re: [Intel-wired-lan] Intel E810-XXVDA2: Too high TX timestamp read latency

[Jacob Keller]

On 2/23/2024 5:07 AM, Martin Kulas wrote:
> Dear Karol,
> 
> On Thu, Feb 22, 2024 at 05:30:09PM +0000, Kolacinski, Karol wrote:
> [...]
>>
>> On Tue, Feb 20, 2024 at 2:17PM +0100, Martin Kulas wrote:
>>>
>>> After having applied that commit,  the TX timestamp read latency stays
>>> unacceptable high (in the range of about 700 microseconds), i.e. no
>>> regression.
>>
>> This improvement utilises 'low latency read', where the driver writes
>> a timestamp request to a specific register, FW has to poll this
>> register every 0.6 ms and then the driver receives a second interrupt,
>> so that it can read the timestamp from this register.
>> Unfortunately, polling every 0.6 ms is the best we could do.
> 
> 
> Thank you very much for your explanation!
> 
> The fact the the firmware (FW) polls the TX timestamp register every 600
> microseconds explains the TX timestamp read latency of my application of
> 700 microseconds.
> 
> Since it is a firmware problem:  A updated firmware with customizable 
> polling rate would enable low TX hardware timestamp read latencies.  Right?
> 
> 
> Currently,  I am looking for other Ethernetcards with TX hardware
> timestamping capability.  I am not sure whether the Intel X710-DA2 (i40e) 
> supports TX hardware timestamps for all outgoing packets and not only
> for PTP packets.  Do you know the timestamping capabilities of that card by
> any chance?  The datasheet of that Ethernet card is not completely clear
> about it.
> 

The 700 series can timestamp any outgoing Tx frame as long as its not
part of a TCP Segmentation Offload. It is limited for Rx timestamping to
only frames it recognizes as PTP, but this limitation does not apply for Tx.

As with the 500 series cards, this device is limited to a single
outstanding Tx timestamp request per PF. You can't initiate a second Tx
timestamp until the first completes.

> [...]
> 
> 
> Kind regards,
> Martin
>