Re: AF_XDP metadata/hints

[John Fastabend <john.fastabend@gmail.com>]

Toke Høiland-Jørgensen wrote:
> Jesper Dangaard Brouer <brouer@redhat.com> writes:
> 
> > On Tue, 25 May 2021 21:51:22 -0700
> > John Fastabend <john.fastabend@gmail.com> wrote:
> >
> >> Separate the config of hardware from the BPF infrastructure these
> >> are two separate things.
> >
> > I fully agree.
> 
> +1. Another reason why is the case of multiple XDP programs on a single
> interface: When attaching these (using freplace as libxdp does it), the
> kernel can just check the dest interface when verifying the freplace
> program and any rewriting of the bytecode from the BTF format can happen
> at that point. Whereas if the BPF attach needs to have side effects,
> suddenly we have to copy over all the features to the dispatcher program
> and do some kind of set union operation; and what happens if an freplace
> program is attached after the fact (same thing with tail calls)?
> 
> So in my mind there's no doubt this needs to be:
> 
> driver is config'ed -> it changes its exposed BTF metadata -> program is
> attached -> verifier rewrites program to access metadata correctly

Well likely libbpf would do the rewrite I think.

> 
> > How should we handle existing config interfaces?
> >
> > Let me give some concrete examples. Today there are multiple existing
> > interfaces to enable/disable NIC hardware features that change what is
> > available to put in our BTF-layout.
> >
> > E.g. changing if VLAN is in descriptor:
> >  # ethtool -K ixgbe1 rx-vlan-offload off
> >  # ethtool -k ixgbe1 | grep vlan-offload
> >  rx-vlan-offload: off
> >  tx-vlan-offload: on
> >
> > The timestamping features can be listed by ethtool -T (see below
> > signature), but it is a socket option that enable[1] these
> > (see SO_TIMESTAMPNS or SOF_TIMESTAMPING_RX_HARDWARE).
> >
> > Or tuning RSS hash fields:
> >  [2] https://github.com/stackpath/rxtxcpu/blob/master/Documentation/case-studies/observing-rss-on-ixgbe-advanced-rss-configuration-rss-hash-fields.md
> >
> > I assume we need to stay compatible and respect the existing config
> > interfaces, right?

I'm not convinced its a strict requirement, rather its a nice to
have. These are low level ethtool hooks into the hardware its
fine IMO if the hardware just reports off and uses a more robust
configuration channel. In general we should try to get away from
this model where kernel devs are acting as the gate keepers for
all hardware offloads and we explicit add checkboxs that driver
writers can use. The result is the current state of things where
we have very flexible hardware that are not usable from Linux.

> >
> > Should we simple leverage existing interfaces?
> 
> Now that ethtool has moved to netlink it should be quite
> straight-forward to add a separate subset of commands for configuring
> metadata fields; and internally the kernel can map those to the existing
> config knobs, no?

Its unclear to me how you simple expose knobs to reconfigure hardware.
It looks to me that you need to push a blob down to the hardware to
reconfigure it for new parsers, new actions, etc. But, maybe the
folks working on current hardware can speak up.

> 
> E.g., if you tell the kernel you'd like to have the VLAN field as a
> metadata field that kinda implies that rx-vlan-offload should be turned
> on; etc. Any reason this would break down?
> 
> -Toke
> 

Agree driver should be able to map these back onto 'legacy' feature
sets.

I'll still have a basic question though. I've never invested much time
into the hints because its still not clear to me what the use case is?
What would we put in the hints and do we have any data to show it would be
a performance win.

If its a simple hash of the headers then how would we use it? The
map_lookup/updates use IP addrs for keys in Cilium. So I think the
suggestion is to offload the jhash operation? But that requires some
program changes to work. Could someone convince me?

Maybe packet timestamp?

Thanks,
John