Re: [RFC PATCH bpf-next] Documentation/bpf: Add a description of "stable kfuncs"

["Toke Høiland-Jørgensen" <toke@redhat.com>]

Daniel Borkmann <daniel@iogearbox.net> writes:

> On 1/16/23 11:57 PM, Toke Høiland-Jørgensen wrote:
>> Following up on the discussion at the BPF office hours, this patch adds a
>> description of the (new) concept of "stable kfuncs", which are kfuncs that
>> offer a "more stable" interface than what we have now, but is still not
>> part of UAPI.
>> 
>> This is mostly meant as a straw man proposal to focus discussions around
>> stability guarantees. From the discussion, it seemed clear that there were
>> at least some people (myself included) who felt that there needs to be some
>> way to export functionality that we consider "stable" (in the sense of
>> "applications can rely on its continuing existence").
>> 
>> One option is to keep BPF helpers as the stable interface and implement
>> some technical solution for moving functionality from kfuncs to helpers
>> once it has stood the test of time and we're comfortable committing to it
>> as a stable API. Another is to freeze the helper definitions, and instead
>> use kfuncs for this purpose as well, by marking a subset of them as
>> "stable" in some way. Or we can do both and have multiple levels of "stable",
>> I suppose.
>> 
>> This patch is an attempt to describe what the "stable kfuncs" idea might look
>> like, as well as to formulate some criteria for what we mean by "stable", and
>> describe an explicit deprecation procedure. Feel free to critique any part
>> of this (including rejecting the notion entirely).
>> 
>> Some people mentioned (in the office hours) that should we decide to go in
>> this direction, there's some work that needs to be done in libbpf (and
>> probably the kernel too?) to bring the kfunc developer experience up to par
>> with helpers. Things like exporting kfunc definitions to vmlinux.h (to make
>> them discoverable), and having CO-RE support for using them, etc. I kinda
>> consider that orthogonal to what's described here, but I added a
>> placeholder reference indicating that this (TBD) functionality exists.
>
> Thanks for the writeup.. I did some edits to your sections to make some parts
> more clear and to leave out other parts (e.g. libbpf-related bits which are not
> relevant in here and it's one of many libs). I also edited some parts to leave
> us more flexibility. Here would be my take mixed in:

Edits LGTM, with just one nit, below:

> 3. API (in)stability of kfuncs
> ==============================
>
> By default, kfuncs exported to BPF programs are considered a kernel-internal
> interface that can change between kernel versions. In the extreme case that
> could also include removal of a kfunc. This means that BPF programs using
> kfuncs might need to adapt to changes between kernel versions. In other words,
> kfuncs are _not_ part of the kernel UAPI! Rather, these kfuncs can be thought
> of as being similar to internal kernel API functions exported using the
> ``EXPORT_SYMBOL_GPL`` macro. All new BPF kernel helper-like functionality must
> initially start out as kfuncs.
>
> 3.1 Promotion to "stable"
> -------------------------
>
> While kfuncs are by default considered unstable as described above, some kfuncs
> may warrant a stronger stability guarantee and could be marked as *stable*. The
> decision to move a kfunc to *stable* is taken on a case-by-case basis and has
> a high barrier, taking into account its usefulness under longer-term production
> deployment without any unforeseen API issues or limitations. In general, it is
> not expected that every kfunc will turn into a stable one - think of it as an
> exception rather than the norm. kfuncs which have been promoted to stable are
> then marked using the ``KF_STABLE`` tag. The possibility from a stable kfunc to
> a BPF helper addition is up to the maintainers to decide.
>
> 1. Stable kfuncs will not change their function signature or functionality in
>     a way that may cause incompatibilities for BPF programs calling the function.
>
> 2. The BPF community will make every reasonable effort to keep stable kfuncs
>     around as long as they continue to be useful to real-world BPF applications.
>
> 3. Should a stable kfunc turn out to be no longer useful, a deprecation procedure
>     might be implemented for them as outlined below.

"deprecation procedure might be implemented" could be interpreted as "we
may implement a deprecation procedure, or we may just remove it without
one". Which is presumably not what you meant? So maybe:

 3. Should a stable kfunc turn out to be no longer useful, the BPF
    community may decide to eventually remove it. In this case, before
    being removed that kfunc will go through a deprecation procedure as
    outlined below.

> 3.2 Deprecation of kfuncs
> -------------------------
>
> As described above, the community will make every reasonable effort to keep
> kfuncs available through future kernel versions once they are marked as stable.
> However, there may be the unforeseen case that BPF development moves in a
> direction where even a stable kfunc is no longer useful for program development.
> In this case, stable kfuncs can be marked as *deprecated* using the
> ``KF_DEPRECATED`` tag. Such deprecation request cannot be arbitrary and must
> explain why a given stable kfunc should be deprecated.
>
> 1. A deprecated stable kfunc will be kept in the kernel for a conservatively
>     chosen period of time after it got first marked as deprecated (usually
>     corresponding to a span of multiple years).
> 2. Deprecated functions will be documented in the kernel docs along with their
>     remaining lifespan and including a recommendation for new functionality that
>     can replace the usage of the deprecated function (or an explanation for why
>     no such replacement exists).
>
> 3. After the deprecation period, the kfunc will be removed and the function name
>     will be marked as invalid inside the kernel (to ensure that no new kfunc is
>     accidentally introduced with the same name in the future). After this
>     happens, BPF programs calling the kfunc will be refused by the verifier.