Re: [RFC net-next 08/15] ipxlat: add translation engine and dispatch core

["Toke Høiland-Jørgensen" <toke@kernel.org>]

Ralf Lici <ralf@mandelbit.com> writes:

> On Mon, 22 Jun 2026 16:36:24 +0200, Toke Høiland-Jørgensen <toke@kernel.org> wrote:
>> >> > My second concern is that the SIIT boundary would be a property of
>> >> > rule and hook placement. That gives flexibility, but it also means the
>> >> > translation point has to be constrained and documented very carefully
>> >> > to avoid ambiguous TTL/Hop Limit, PMTU/ICMP, and hook-order behavior.
>> >> > For this use case I would rather have the route that matches the
>> >> > translation prefix also be the object that says: leave this family
>> >> > here and continue in the other one.
>> >>
>> >> Yeah, with flexibility comes the ability to shoot yourself in the foot.
>> >> But that's not really different from much of the other functionality we
>> >> have in the kernel today, is it? For netfilter in particular it's
>> >> certainly possible to configure a broken NAT configuration that leads to
>> >> packet drops (or just invalid packets being sent out on a network
>> >> device).
>> >>
>> >
>> > True, misconfiguration is always possible and that alone is not an
>> > argument against the netfilter model. But what do we actually gain in
>> > capability from that flexibility? I agree on the UX argument (an admin
>> > would look in nft first), but in terms of what the feature can do, I
>> > can't yet see what the nft model unlocks. More on this just below.
>> >
>> >> > After looking at the available kernel mechanisms again, I think the
>> >> > better model is probably LWT: routes carry an ipxlat encap referencing a
>> >> > named translator domain configured over netlink. That should represent
>> >> > the stateless, prefix-based and symmetric nature of ipxlat.
>> >>
>> >> I think this description actually hits the nail on the head: What are we
>> >> implementing here? Is it a product feature, or a building block for one?
>> >> The properties you mention wrt consistency, symmetry etc are properties
>> >> of the high-level feature (which is also generally the level things are
>> >> specified in RFCs). Whereas other packet mangling features in the kernel
>> >> are more in the "building block" category, where it's possible to
>> >> configure things to implement a particular feature set / compliance with
>> >> a particular RFC, but it's also possible to do things that are outside
>> >> of that.
>> >>
>> >> I think this relates to the "mechanism, not policy" approach that we
>> >> take to most things in the kernel: implement the building blocks to do
>> >> something in the most general way we can, and then leave it up to
>> >> userspace to configure things in a way that results in a consistent
>> >> high-level system behaviour.
>> >>
>> >
>> > That's a good point, and I agree that we should not bake a high-level
>> > product policy into the kernel if what we need is a reusable mechanism
>> > (the LWT idea was my attempt at exactly that). What I am still trying to
>> > understand is whether there is a useful generic trigger for stateless
>> > cross-family translation beyond the route/prefix/policy-routing cases.
>> >
>> > Routes and policy routing already cover the selectors I can make
>> > coherent for a stateless, per-packet translator: destination/source
>> > prefix, iif/oif/VRF, mark, TOS/DSCP, and so on. nft can of course match
>> > much more than that, but the additional selectors that would materially
>> > change the translation decision seem to be selectors such as L4 fields,
>> > payload state, or conntrack state. Those are exactly the selectors I am
>> > struggling to make correct for a stateless translator:
>> >
>> > - non-first fragments carry no L4 header at all, yet the translator must
>> >   rewrite every fragment (an nft ... tcp dport trigger cannot fire on
>> >   them);
>> >
>> > - ICMP errors must be translated too, but the flow identity lives in the
>> >   quoted inner header (reversed), not in anything an L4/ct match on the
>> >   error packet can see and there is no conntrack to associate them,
>> >   since this is stateless.
>>
>> True in principle, but if (say) you deploy this on a network that is
>> configured so it will never fragment packets, this won't be an issue in
>> practice.
>>
>> I.e., you're quite right that arbitrary matching criteria cannot be
>> guaranteed to result in coherent translation. But I think that goes into
>> the "use it wrong, get wrong results" bin. E.g., if you match on
>> something that results in only a subset of the packets of a flow being
>> translated, well, only that subset of the packets will make it to the
>> destination. The SIIT translator itself should not try to fix this, but
>> neither should it prevent it; that's what I mean by "building block" -
>> it's up to the builder using the blocks to make sure the building
>> doesn't collapse, that's out of scope for the block manufacturer to
>> worry about :)
>>
>
> I agree with that framing. The translation core should not try to prove
> that the surrounding policy describes a coherent SIIT deployment.

Cool!

>> > So an L4-conditional trigger does not look like a good primitive for
>> > correct stateless SIIT unless the action also defragments/refragments or
>> > uses conntrack-like state. Those may be valid mechanisms, but they move
>> > the design away from the stateless per-packet SIIT boundary this RFC is
>> > trying to model.
>> >
>> > So my first question is: is there a useful nft configuration this should
>> > enable that is not naturally expressible as route selection, while still
>> > remaining stateless SIIT rather than a NAT64-like stateful feature?
>> > Maybe there is a real use case there, but I cannot construct one yet.
>>
>> So the poster child for "match on arbitrary criteria" is of course BPF.
>> You can write BPF programs that match on arbitrary parts of the packet
>> header, custom encapsulation headers,or even on out of band things like
>> system state, phase of the moon, or what have you. And we should
>> certainly allow a BPF program to make the decision on whether to perform
>> the SIIT translation.
>>
>> Which... maybe is an argument to keep it as a device like you do in this
>> RFC series? Redirecting to a device is trivially supported from TC-BPF,
>> which also makes it possible to use the translation mechanism without
>> going through the routing subsystem at all, saving a bit of overhead.
>> Whereas making it a route action ties it very closely to the routing
>> subsystem.
>>
>> WDYT?
>>
>
> I see the netdevice appeal for this, especially as a BPF redirect
> target. But as we discussed earlier, the device model has some real
> problems: the device selected by the first route is not the real
> post-translation egress, so the model ends up doing translation and
> reinjection rather than normal transmission. Concretely:
>
> - it needs synthetic routing state purely to get things like MTU for
>   fragmentation, because the real post-translation nexthop is not known
>   at translation time;
>
> - TTL/Hop Limit handling gets harder to reason about because the packet
>   has effectively gone through two routing decisions;
>
> - rx/tx stats can't be made meaningful for a direction-agnostic device
>   whose ndo_start_xmit is really "translate and receive";
>
> - and the setup is not very obvious: create an interface, route packets
>   to it, then have them come back translated.
>
> None of these is fatal on its own, but together they make me think the
> abstraction does not quite fit.

Right, OK, you're right.

> On the BPF point specifically: I agree a BPF program should be able to
> decide whether to translate. What I am less sure about is whether
> redirecting to a netdevice is the best way to expose that. A TC action
> (yet another model, I know :)) gives you the same thing in-pipeline and
> more directly:
>
>     tc filter add dev wwan0 egress \
>         bpf obj match.o action ipxlat4to6 domain clat0
>
> Let BPF make the policy decision, with the native action doing the
> translation work that the current BPF CLAT implementations have trouble
> with: fragmentation, checksum corner cases, and ICMP error inner
> headers (as explained by Beniamino).
>
> So TC clsact looks like the natural in-kernel replacement for today's
> TC-BPF CLAT programs: no extra netdev, you attach to the existing
> uplink, direction is explicit, and on egress you sit on the real route
> dst, so the synthetic-dst and double-routing problems above just don't
> arise. The cost is more moving parts than a single bpf_redirect since
> userspace has to manage clsact, filters, priorities and action
> lifecycle/cleanup.

Hmm, so no one really uses the bpf filter mechanism, since you can just
do everything from an action anyway (and with TCX attachment, you can
even avoid the overhead of the TC filter/action infrastructure
entirely). However, point taken wrt how to integrate this with BPF. I
guess the most flexible thing would be to expose the functionality
directly (as a kfunc callable from a BPF program). Which also fits with
your point below:

> For a gateway translator, though, I still think a device-bound model is
> less natural. There the translation point is more like a forwarding
> decision across routes and nexthops, so a route/LWT attachment, or
> possibly a netfilter attachment seems easier to reason about. Also, as
> you already pointed out while discussing LWT, an admin setting up NAT64
> is more likely to reach for an nft rule than for a clsact filter on a
> specific device.
>
> Taking a step back, ipxlat is really a generic translation engine plus a
> thin harness around it. So rather than pick one attachment, it might be
> worth structuring the engine so different harnesses can drive it.
> There's interesting precedent for this shape:
>
> - ILA, again, is the closest sibling: stateless IPv6 address translation
>   with a shared core in ila_common.c, driven both by an LWT frontend in
>   ila_lwt.c and by an inline netfilter hook with a netlink-configured
>   mapping table in ila_xlat.c.
>
> - act_ct is the precedent for the TC side specifically: a TC action that
>   reuses the netfilter conntrack engine rather than reimplementing it.
>
> And act_nat is the cautionary counter-example: a standalone TC
> reimplementation of stateless NAT that shares no code with nf_nat, and
> carries a "would be nice to share code" comment :)
>
> So I am wondering whether the right direction is to factor the
> translation engine cleanly, land it with one harness first, and keep the
> other attachment points as follow-up work once the core semantics are
> settled.
>
> Does that direction seem reasonable to you?

Yes, reusable functionality that can be called from multiple places
sounds like a good fit; let's try to structure it that way!

As for which hook to start with, well, let's see if we hear back from
the netfilter devs, but either netfilter or the routing subsystem (LWT
style) would be OK for me I think.

-Toke