Re: [PATCH net-next RFC v4 2/5] net: Expose available time stamping layers to user space.

[Jakub Kicinski <kuba@kernel.org>]

On Fri, 12 May 2023 02:07:17 +0300 Vladimir Oltean wrote:
> On Thu, May 11, 2023 at 03:09:02PM -0700, Jakub Kicinski wrote:
> > Exactly, think Tx. For Rx hauling the TS in metadata from PHY/MAC to
> > descriptor is easy. For Tx device will buffer the packet so the DMA
> > completion happens before the packet actually left onto the wire.
> > 
> > Which is not to say that some devices may not generate the Rx timestamp
> > when the packet is DMA'ed out of laziness, too.
> >   
> > > timestamps is an alternate solution to the same problem as DMA
> > > timestamps, or whatever:
> > > https://lore.kernel.org/netdev/20221018010733.4765-1-muhammad.husaini.zulkifli@intel.com/  
> > 
> > What I was thinking was:
> > 
> >  - PHY - per spec, at the RS layer
> >  - MAC - "close to the wire" in the MAC, specifically the pipeline
> >    delay (PHY stamp vs MAC stamp) should be constant for all packets;
> >    there must be no variable-time buffering and (for Tx) the time
> >    stamping must be past the stage of the pipeline affected by pause
> >    frames
> >  - DMA - worst quality, variable delay timestamp, usually taken when
> >    packets DMA descriptors (Rx or completion) are being written
> > 
> > With these definitions MAC and PHY timestamps are pretty similar
> > from the perspective of accuracy.  
> 
> So if I add a call to ptp_clock_info :: gettimex64() where the
> skb_tx_timestamp() call is located in a driver, could that pass as
> a DMA timestamp?

Yes.

> The question is how much do we want to encourage these DMA timestamps:
> enough to make them a first-class citizen in the UAPI? Are users even
> happy with their existence?

I don't want to call anyone out, but multiple high speed devices
with the current, in tree drivers report DMA timestamps when you 
ask for HW timestamps. DMA stamps are still 2 orders of magnitude
better than SW stamping.

> I mean, I can't ignore the fact that there are NICs that can provide
> 2-step TX timestamps at line rate for all packets (not just PTP) for
> line rates exceeding 10G, in band with the descriptor in its TX
> completion queue. I don't want to give any names, just to point out
> that there isn't any inherent limitation in the technology.

Oh, you should tell me, maybe off-list then. 'Cause I don't know any.

> AFAIU from igc_ptp_tx_hwtstamp(), it's just that the igc DMA
> controller did not bother to transport the timestamps from the MAC
> back into the descriptor, leaving it up to software to do it out of
> band, which of course may cause correlation bugs and limits
> throughput. Surely they can do better.

It's not that simple. Any packet loss or diversion and you may end up
missing a completion.