[MODERATED] Generic eBPF hardening

[MODERATED] Generic eBPF hardening

[Dave Hansen]

[-- Attachment #1: Type: text/plain, Size: 1349 bytes --]

I'm hoping Alexei is on the list now.

Andi Kleen and I were talking about ways to generally harden eBPF.  We
were a bit concerned that eBPF could leave around attacker-controlled
values on the stack that might allow later, speculatively-executed
kernel code to be exploited.

One thing I wanted to clarify: The eBPF stack (BPF_REG_FP) *is* the
kernel stack, correct?  I see this code getting generated by
emit_prologue(), for instance:

>         /* sub rsp, rounded_stack_depth + AUX_STACK_SPACE */
>         EMIT3_off32(0x48, 0x81, 0xEC,
>                     round_up(stack_depth, 8) + AUX_STACK_SPACE);

Or does that only apply to the JIT?

One cheap, simple, mitigation for that would be to zero any kernel stack
that could have eBPF data in it during BPF_EXIT.  I guess we could do
that after the 'leave' but before the 'ret' in the x86 JIT.

We could also zero out all of the eBPF contents in the hardware
registers at this point.  That would help reduce the gadgets in the
kernel that could be exploited.

For speculative store bypass, specifically, would it make sense to do an
MFENCE at BFP_EXIT for unprivileged programs that do writes to the
stack?  That way, the result of any BPF stores will be known before we
start running kernel code again.

We would not need any of these for privileged programs.

[MODERATED] Re: Generic eBPF hardening

[Alexei Starovoitov]

On Thu, May 17, 2018 at 01:55:38PM -0700, speck for Dave Hansen wrote:
> I'm hoping Alexei is on the list now.
> 
> Andi Kleen and I were talking about ways to generally harden eBPF.  We
> were a bit concerned that eBPF could leave around attacker-controlled
> values on the stack that might allow later, speculatively-executed
> kernel code to be exploited.
> 
> One thing I wanted to clarify: The eBPF stack (BPF_REG_FP) *is* the
> kernel stack, correct?

yes. bpf program stack is kernel stack,
but I don't see how this is useful.
There are plenty of ways to populate kernel stack with user controlled
bytes. Ex: set_task_comm.

[MODERATED] Re: Generic eBPF hardening

[Andi Kleen]

On Thu, May 17, 2018 at 03:22:35PM -0700, speck for Alexei Starovoitov wrote:
> On Thu, May 17, 2018 at 01:55:38PM -0700, speck for Dave Hansen wrote:
> > I'm hoping Alexei is on the list now.
> > 
> > Andi Kleen and I were talking about ways to generally harden eBPF.  We
> > were a bit concerned that eBPF could leave around attacker-controlled
> > values on the stack that might allow later, speculatively-executed
> > kernel code to be exploited.
> > 
> > One thing I wanted to clarify: The eBPF stack (BPF_REG_FP) *is* the
> > kernel stack, correct?
> 
> yes. bpf program stack is kernel stack,
> but I don't see how this is useful.
> There are plenty of ways to populate kernel stack with user controlled
> bytes. Ex: set_task_comm.

EBPF has quite a bit more control over the exact layout than other 
buffers. 

Over time we should probably migrate most people who do copy_to_user
to stack buffers to do so onto the heap. But that's a longer term
project.

Luckily stack buffers are usually quite limited in size 
so attacks are likely harder.

But with EBPF controlling the exact layout it should be easier.

Would be better to fix it, especially since the fixes should be cheap.

-Andi

[MODERATED] Re: Generic eBPF hardening

[Dave Hansen]

On 05/17/2018 03:22 PM, speck for Alexei Starovoitov wrote:
> On Thu, May 17, 2018 at 01:55:38PM -0700, speck for Dave Hansen wrote:
>> I'm hoping Alexei is on the list now.
>>
>> Andi Kleen and I were talking about ways to generally harden eBPF.  We
>> were a bit concerned that eBPF could leave around attacker-controlled
>> values on the stack that might allow later, speculatively-executed
>> kernel code to be exploited.
>>
>> One thing I wanted to clarify: The eBPF stack (BPF_REG_FP) *is* the
>> kernel stack, correct?
> yes. bpf program stack is kernel stack,
> but I don't see how this is useful.
> There are plenty of ways to populate kernel stack with user controlled
> bytes. Ex: set_task_comm.

In most places in the kernel, it's hard to both get data on the stack
that you control *and* execute code that you (mostly) control.  For the
vast majority of the kernel, you are entirely at the mercy of the
compiler to give you the code and the stack layout.

BPF currently gives you pretty good control of both, though.

Zeroing the stack would at least take away one source of gadgets: kernel
code that speculatively consumes uninitialized data from the stack.

[MODERATED] Re: Generic eBPF hardening

[Alexei Starovoitov]

On Thu, May 17, 2018 at 03:54:48PM -0700, speck for Andi Kleen wrote:
> On Thu, May 17, 2018 at 03:22:35PM -0700, speck for Alexei Starovoitov wrote:
> > On Thu, May 17, 2018 at 01:55:38PM -0700, speck for Dave Hansen wrote:
> > > I'm hoping Alexei is on the list now.
> > > 
> > > Andi Kleen and I were talking about ways to generally harden eBPF.  We
> > > were a bit concerned that eBPF could leave around attacker-controlled
> > > values on the stack that might allow later, speculatively-executed
> > > kernel code to be exploited.
> > > 
> > > One thing I wanted to clarify: The eBPF stack (BPF_REG_FP) *is* the
> > > kernel stack, correct?
> > 
> > yes. bpf program stack is kernel stack,
> > but I don't see how this is useful.
> > There are plenty of ways to populate kernel stack with user controlled
> > bytes. Ex: set_task_comm.
> 
> EBPF has quite a bit more control over the exact layout than other 
> buffers. 
> 
> Over time we should probably migrate most people who do copy_to_user
> to stack buffers to do so onto the heap. But that's a longer term
> project.
> 
> Luckily stack buffers are usually quite limited in size 
> so attacks are likely harder.
> 
> But with EBPF controlling the exact layout it should be easier.
> 
> Would be better to fix it, especially since the fixes should be cheap.

I disagree with 'cheap' assessment.
The main use case for unprivileged bpf is so_reuseport and socket filters.
It's critical path of networking receive side.
Adding lfence after every program won't be cheap.
Once all copy_from_user() are migrated to heap we can restart
this discussion. Right now it's a nack.
All paranoid security people already do kernel.unprivileged_bpf_disabled=1
I don't want to complicate things just for 'peace of mind'.

[MODERATED] Re: Generic eBPF hardening

[Alexei Starovoitov]

On Thu, May 17, 2018 at 04:13:32PM -0700, speck for Dave Hansen wrote:
> On 05/17/2018 03:22 PM, speck for Alexei Starovoitov wrote:
> > On Thu, May 17, 2018 at 01:55:38PM -0700, speck for Dave Hansen wrote:
> >> I'm hoping Alexei is on the list now.
> >>
> >> Andi Kleen and I were talking about ways to generally harden eBPF.  We
> >> were a bit concerned that eBPF could leave around attacker-controlled
> >> values on the stack that might allow later, speculatively-executed
> >> kernel code to be exploited.
> >>
> >> One thing I wanted to clarify: The eBPF stack (BPF_REG_FP) *is* the
> >> kernel stack, correct?
> > yes. bpf program stack is kernel stack,
> > but I don't see how this is useful.
> > There are plenty of ways to populate kernel stack with user controlled
> > bytes. Ex: set_task_comm.
> 
> In most places in the kernel, it's hard to both get data on the stack
> that you control *and* execute code that you (mostly) control.  For the
> vast majority of the kernel, you are entirely at the mercy of the
> compiler to give you the code and the stack layout.

just turn on jit and blinding (bpf_jit_harden) and the code will be random.
We can add more randomness to blinding, but imo it's good enough already.

[MODERATED] Re: Generic eBPF hardening

[Andi Kleen]

> I disagree with 'cheap' assessment.
> The main use case for unprivileged bpf is so_reuseport and socket filters.
> It's critical path of networking receive side.
> Adding lfence after every program won't be cheap.

I don't think we need any lfence at all. It makes no difference for 
stack reuse. 

All we need is stack clear for the locations that are script controlled.
That should be quite cheap.

Register clearing would be nice, and should be also cheap.

-Andi

[MODERATED] Re: Generic eBPF hardening

[Dave Hansen]

On 05/17/2018 04:21 PM, speck for Alexei Starovoitov wrote:
> The main use case for unprivileged bpf is so_reuseport and socket filters.
> It's critical path of networking receive side.
> Adding lfence after every program won't be cheap.

Alexei, do you have any "benchmarks" or tests for these paths that you
regularly run to look for performance regressions?  I'd definitely want
to run those as we look at potential mitigations.

[MODERATED] Re: Generic eBPF hardening

[Alexei Starovoitov]

On Fri, May 18, 2018 at 11:18:26AM -0700, speck for Dave Hansen wrote:
> On 05/17/2018 04:21 PM, speck for Alexei Starovoitov wrote:
> > The main use case for unprivileged bpf is so_reuseport and socket filters.
> > It's critical path of networking receive side.
> > Adding lfence after every program won't be cheap.
> 
> Alexei, do you have any "benchmarks" or tests for these paths that you
> regularly run to look for performance regressions?  I'd definitely want
> to run those as we look at potential mitigations.

There is a large set of bpf tests in selftests/bpf/
In particular test_progs checks both correctness and performance.
There is no public benchmark for so_reuseport+bpf yet.
It's used by our L7 load balancer.
specdown mitigations are bad enough for production already.
I'm allergic to any extra overhead.

Right now the whole bpf space is either root or unpriv.
Over time there will be more categories in the middle.
Like all networking progs today require root, but that's overkill.
We're closing down pointer leaking even for root, so we
can relax networking progs to cap_net_admin.
Other folks are asking to be able to run all bpf prog types
without being root. The use case is continuous integration.
Currenly these buildbots require root and broken bpf tests
affect all other tests that run on the same server.
Hence we're thinking of some sort of sandbox for bpf tracing
and bpf networking that can be unpriv and won't affect the rest.

The main point here that today unpriv bpf use cases are tiny comparing
to bpf with root, but it will change in the future and we
cannot slow down unpriv. Both should be fast.