Re: Hardware time stamping support for AF_XDP applications

[Jesper Dangaard Brouer <brouer@redhat.com>]

On Tue, 10 Nov 2020 23:53:41 +0000
"Patel, Vedang" <vedang.patel@intel.com> wrote:

> Hi Saeed, 
> 
> > On Nov 10, 2020, at 3:32 PM, Saeed Mahameed <saeed@kernel.org> wrote:
> > 
> > On Tue, 2020-11-10 at 22:44 +0000, Patel, Vedang wrote:  
> >> [Sorry if you got the email twice. Resending because it was rejected
> >> by netdev for containing HTML]
> >> 
> >> Hi Saeed/Jesper, 
> >> 
> >> I am working in the Time Sensitive Networking team at Intel. We work
> >> on implementing and upstreaming support for TSN related features for
> >> intel based NICs. Recently we have been adding support for XDP in
> >> i225. One of the features which we want to add support for is passing
> >> the hardware timestamp information to the userspace application
> >> running AF_XDP sockets (for both Tx and Rx). I came across the XDP
> >> Workshop[1] conducted in July 2020 and there you stated that you are
> >> already working on adding support for BTF based metadata to pass
> >> hardware hints for XDP Programs. My understanding (along with a few
> >> questions) of the current state is:  

Have the i225 XDP support been upstreamed?

Can I buy a i255 NIC for my server, or is this embedded NICs?

Ilias have played with PoC for TSN (on ARM) here:
 https://github.com/xdp-project/xdp-project/blob/master/areas/arm64/xdp_for_tsn.org

> > Hi Patel,
> >   
> >> * This feature is currently being maintained out of tree. I found
> >> that an RFC Series[2] was posted in June 2018. Are you planning to
> >> post an updated version to be merged in the mainline anytime soon?   
> > 
> > Yes hopefully in the coming couple of weeks.
> >   
>
> Sure! I will start testing/developing on top of your branch mentioned
> below for now.

I've also signed up for helping out on this effort.  Notice Andrii (cc)
have already pointed out something that can be improved, and even made
easier.


> >> * I am guessing hardware timestamp is one of the metadata fields
> >> which will be eventually supported? [3]  
> > 
> > With BTF formatted metadata it is up to the driver to advertise
> > whatever it can/want :)
> > so yes.  
>
> I have a very basic question here. From what I understand about BTF,
> I can generate a header file (using bpftool?) containing the BTF data
> format provided by the driver. If so, how can I design an application
> which can work with multiple NICs drivers without recompilation? I am
> guessing there is some sort of “master list” of HW hints the drivers
> will agree upon?

I recommend that you read Andrii's blogpost:
 https://facebookmicrosites.github.io/bpf/blog/2020/02/19/bpf-portability-and-co-re.html

I need to learn more about BTF myself, but the way I understand this:
We need to agree on the meaning of struct member names (e.g. rxhash32).
Then you compile BPF with a BTF struct that have this rxhash32 member
name, and at BPF load-time the kernel CO-RE infra will remap the offset
to the rxhash32 offset used by the specific driver.

> >   
> >> * The Metadata support will be extended to pass on the hardware hints
> >> to AF_XDP sockets. Are there any rough plans on what metadata will be
> >> transferred?  
> > 
> > AF_XDP is not part of my series, but supporting AF_XDP with metadata
> > offlaod is up to the driver to implement, should be straight forward
> > and identical to XDP.

The XDP data_meta area is also transferred into the AF_XDP frame, also
in the copy-mode version of AF_XDP.


> > what meta data to pass is up to the driver.  
>
> Alright, let me take a closer look at your latest code. I will come
> back will questions if I have any.
> > 
> >   
> >> * The current plan for Tx side only includes passing data from the
> >> application to the driver. Are there any plans to support passing
> >> information (like HW TX timestamp) from driver to the Application?
> >>   
> > 
> > you mean for AF_XDP ? i actually haven't thought about this, 
> > but we could use TX umem packet buffer headroom to pass TX completion
> > metadata to AF_XDP app, or extend the completion queue entries to host
> > metadata, i am sure that the 1st approach is preferred, but i am not
> > planing to support this in my initial series. 
> >   
> Yeah, I was thinking of using approach 1 as well for this. Let me
> first work on the Rx side. Then we can scope this one out.
>
> >> Finally, is there any way I can help in expediting the development
> >> and upstreaming of this feature? I have been working on studying how
> >> XDP works and can work on implementing some part of this feature if
> >> you would like.
> >>   
> > 
> > Sure,
> > Please feel free to clone and test the following branch if you add
> > support to  your driver and implement offloads for AF_XDP that would be
> > awesome, and i will append your patches to my series before submission.
> > 
> > it is always great to send new features with multiple use cases and
> > multi vendor support, this will differently expedite submission and
> > acceptance
> > 
> > My Latest work can be found at:
> > 
> > https://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git/log/?h=topic/xdp_metadata3
> > 
> > Please feel free to send me any questions about the code in private or
> > public.
>
> Thanks Saeed for all the information! This is really helpful. :)
> > 
> > Thanks,
> > Saeed.
> >   
> >> Thanks,
> >> Vedang Patel
> >> Software Engineer
> >> Intel Corporation
> >> 
> >> [1] - https://netdevconf.info/0x14/session.html?workshop-XDP
> >> [2] - 
> >> https://patchwork.ozlabs.org/project/netdev/cover/20180627024615.17856-1-saeedm@mellanox.com/
> >> [3] - 
> >> https://xdp-project.net/#outline-container-Important-medium-term-tasks  
> 



-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer