Re: Are BPF programs preemptible?

[Yonghong Song <yhs@meta.com>]

On 1/23/23 4:30 AM, Yaniv Agman wrote:
> Ok, thanks Jakub for the answer and references.
> I must say that I am very surprised though. First, most of the
> documentation for BPF says that preemption is disabled, like the
> reference I gave [1] and even the bpf-helpers man page [2] says "Note
> that all programs run with preemption disabled..." for the
> bpf_get_smp_processor_id() helper. I think this is something that
> deserves more attention since many eBPF developers are still under the
> assumption that eBPF programs are non-preemptible, and running their
> programs on newer kernels might be broken.

It would be great if you can send a patch to fix these
out-dated comments!

> 
> I'm trying to figure out how I can solve this issue in our case - is
> it correct to assume that no more than one preemption can happen
> during a run of my bpf program? If so, I can try to write a percpu

No. It is possible that you have more than one preemption during the
same prog run. There is no restriction on this.

> buffer with 2 entries, and give the second entry to the program that
> interrupted the first one. But even then, I will need to find a way to
> know if my program currently interrupts the run of another program -
> is there a way to do that? Maybe checking if the current context is of
> an interrupt, can this be done? Any other suggestions to solve this
> problem?
> 
> [1]: https://docs.cilium.io/en/latest/bpf/toolchain
> [2]: https://man7.org/linux/man-pages/man7/bpf-helpers.7.html
> 
> Thanks,
> Yaniv
> 
> ‫בתאריך יום ב׳, 23 בינו׳ 2023 ב-12:54 מאת ‪Jakub Sitnicki‬‏
> <‪jakub@cloudflare.com‬‏>:‬
>>
>> On Mon, Jan 23, 2023 at 11:21 AM +02, Yaniv Agman wrote:
>>> Hello!
>>>
>>> Several places state that eBPF programs cannot be preempted by the
>>> kernel (e.g. https://docs.cilium.io/en/latest/bpf/toolchain ), however,
>>> I did see a strange behavior where an eBPF percpu map gets overridden,
>>> and I'm trying to figure out if it's due to a bug in my program or
>>> some misunderstanding I have about eBPF. What caught my eye was a
>>> sentence in a LWN article (https://lwn.net/Articles/812503/ ) that
>>> says: "Alexei thankfully enlightened me recently over a beer that the
>>> real intent here is to guarantee that the program runs to completion
>>> on the same CPU where it started".
>>>
>>> So my question is - are BPF programs guaranteed to run from start to
>>> end without being interrupted at all or the only guarantee I get is
>>> that they run on the same CPU but IRQs (NMIs, soft irqs, whatever) can
>>> interrupt their run?
>>>
>>> If the only guarantee is no migration, it means that a percpu map
>>> cannot be safely used by two different BPF programs that can preempt
>>> each other (e.g. some kprobe and a network cgroup program).
>>
>> Since v5.7 BPF program runners use migrate_disable() instead of
>> preempt_disable(). See commit 2a916f2f546c ("bpf: Use
>> migrate_disable/enable in array macros and cgroup/lirc code.") [1].
>>
>> But at that time migrate_disable() was merely an alias for
>> preempt_disable() on !CONFIG_PREEMPT_RT kernels.
>>
>> Since v5.11 migrate_disable() does no longer disable preemption on
>> !CONFIG_PREEMPT_RT kernels. See commit 74d862b682f5 ("sched: Make
>> migrate_disable/enable() independent of RT") [2].
>>
>> So, yes, you are right, but it depends on the kernel version.
>>
>> PS. The migrate_disable vs per-CPU data problem is also covered in [3].
>>
>> [1]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=2a916f2f546ca1c1e3323e2a4269307f6d9890eb
>> [2]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=74d862b682f51e45d25b95b1ecf212428a4967b0
>> [3]: https://lwn.net/Articles/836503/