Re: [RFC PATCH bpf-next] Documentation/bpf: Add a description of "stable kfuncs"

["Toke Høiland-Jørgensen" <toke@redhat.com>]

Daniel Borkmann <daniel@iogearbox.net> writes:

> On 1/17/23 12:30 PM, Toke Høiland-Jørgensen wrote:
>> Daniel Borkmann <daniel@iogearbox.net> writes:
>> 
>>> On 1/16/23 11:57 PM, Toke Høiland-Jørgensen wrote:
>>>> Following up on the discussion at the BPF office hours, this patch adds a
>>>> description of the (new) concept of "stable kfuncs", which are kfuncs that
>>>> offer a "more stable" interface than what we have now, but is still not
>>>> part of UAPI.
>>>>
>>>> This is mostly meant as a straw man proposal to focus discussions around
>>>> stability guarantees. From the discussion, it seemed clear that there were
>>>> at least some people (myself included) who felt that there needs to be some
>>>> way to export functionality that we consider "stable" (in the sense of
>>>> "applications can rely on its continuing existence").
>>>>
>>>> One option is to keep BPF helpers as the stable interface and implement
>>>> some technical solution for moving functionality from kfuncs to helpers
>>>> once it has stood the test of time and we're comfortable committing to it
>>>> as a stable API. Another is to freeze the helper definitions, and instead
>>>> use kfuncs for this purpose as well, by marking a subset of them as
>>>> "stable" in some way. Or we can do both and have multiple levels of "stable",
>>>> I suppose.
>>>>
>>>> This patch is an attempt to describe what the "stable kfuncs" idea might look
>>>> like, as well as to formulate some criteria for what we mean by "stable", and
>>>> describe an explicit deprecation procedure. Feel free to critique any part
>>>> of this (including rejecting the notion entirely).
>>>>
>>>> Some people mentioned (in the office hours) that should we decide to go in
>>>> this direction, there's some work that needs to be done in libbpf (and
>>>> probably the kernel too?) to bring the kfunc developer experience up to par
>>>> with helpers. Things like exporting kfunc definitions to vmlinux.h (to make
>>>> them discoverable), and having CO-RE support for using them, etc. I kinda
>>>> consider that orthogonal to what's described here, but I added a
>>>> placeholder reference indicating that this (TBD) functionality exists.
>>>
>>> Thanks for the writeup.. I did some edits to your sections to make some parts
>>> more clear and to leave out other parts (e.g. libbpf-related bits which are not
>>> relevant in here and it's one of many libs). I also edited some parts to leave
>>> us more flexibility. Here would be my take mixed in:
>> 
>> Edits LGTM, with just one nit, below:
>> 
>>> 3. API (in)stability of kfuncs
>>> ==============================
>>>
>>> By default, kfuncs exported to BPF programs are considered a kernel-internal
>>> interface that can change between kernel versions. In the extreme case that
>>> could also include removal of a kfunc. This means that BPF programs using
>>> kfuncs might need to adapt to changes between kernel versions. In other words,
>>> kfuncs are _not_ part of the kernel UAPI! Rather, these kfuncs can be thought
>>> of as being similar to internal kernel API functions exported using the
>>> ``EXPORT_SYMBOL_GPL`` macro. All new BPF kernel helper-like functionality must
>>> initially start out as kfuncs.
>>>
>>> 3.1 Promotion to "stable"
>>> -------------------------
>>>
>>> While kfuncs are by default considered unstable as described above, some kfuncs
>>> may warrant a stronger stability guarantee and could be marked as *stable*. The
>>> decision to move a kfunc to *stable* is taken on a case-by-case basis and has
>>> a high barrier, taking into account its usefulness under longer-term production
>>> deployment without any unforeseen API issues or limitations. In general, it is
>
> Forgot, we should probably also add after "[...] or limitations.":
>
>    Such promotion request along with aforementioned argumentation on why a kfunc
>    is ready to be stabilized must be driven from developer-side.

What does "driven from developer-side" mean, exactly? And what kind of
developers (BPF app developers, or kernel devs)?

>>> not expected that every kfunc will turn into a stable one - think of it as an
>>> exception rather than the norm. kfuncs which have been promoted to stable are
>>> then marked using the ``KF_STABLE`` tag. The possibility from a stable kfunc to
>>> a BPF helper addition is up to the maintainers to decide.
>>>
>>> 1. Stable kfuncs will not change their function signature or functionality in
>>>      a way that may cause incompatibilities for BPF programs calling the function.
>>>
>>> 2. The BPF community will make every reasonable effort to keep stable kfuncs
>>>      around as long as they continue to be useful to real-world BPF applications.
>>>
>>> 3. Should a stable kfunc turn out to be no longer useful, a deprecation procedure
>>>      might be implemented for them as outlined below.
>> 
>> "deprecation procedure might be implemented" could be interpreted as "we
>> may implement a deprecation procedure, or we may just remove it without
>> one". Which is presumably not what you meant? So maybe:
>> 
>>   3. Should a stable kfunc turn out to be no longer useful, the BPF
>>      community may decide to eventually remove it. In this case, before
>>      being removed that kfunc will go through a deprecation procedure as
>>      outlined below.
>
> Yes, that sounds good to me.

Awesome!

-Toke