[QUESTION] BPF kernel selftests failed in the LTS stable kernel 4.19.x

[QUESTION] BPF kernel selftests failed in the LTS stable kernel 4.19.x

[Tiezhu Yang]

Hi all,

When update the following LTS stable kernel 4.19.x,
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/log/?h=linux-4.19.y

and then run BPF selftests according to
https://www.kernel.org/doc/html/latest/bpf/bpf_devel_QA.html#q-how-to-run-bpf-selftests

$ cd tools/testing/selftests/bpf/
$ make
$ sudo ./test_verifier
$ sudo make run_tests

there exists many failures include verifier tests and run_tests,
(1) is it necessary to make sure that there are no any failures in the LTS
stable kernel 4.19.x?
(2) if yes, how to fix these failures in the LTS stable kernel 4.19.x?
(3) if no, how to test BPF in the LTS stable kernel 4.19.x, just use 
test_bpf.ko?


Here are some verifier tests failures:

#165/u PTR_TO_STACK store/load - out of bounds low FAIL
Unexpected error message!
     EXP: invalid stack off=-79992 size=8
     RES: 0: (bf) r1 = r10
1: (07) r1 += -80000
invalid stack off=-80000 size=1
R1 stack pointer arithmetic goes out of range, prohibited for !root

0: (bf) r1 = r10
1: (07) r1 += -80000
invalid stack off=-80000 size=1
R1 stack pointer arithmetic goes out of range, prohibited for !root
#165/p PTR_TO_STACK store/load - out of bounds low OK

#194/u unpriv: adding of fp FAIL
Failed to load prog 'Numerical result out of range'!
0: (b7) r0 = 0
1: (b7) r1 = 0
2: (0f) r1 += r10
R1 tried to add from different maps, paths, or prohibited types
#194/p unpriv: adding of fp OK


#423/u bounds checks mixing signed and unsigned FAIL
Unexpected error message!
     EXP: unbounded min value
     RES: 0: (7a) *(u64 *)(r10 -8) = 0
1: (bf) r2 = r10
2: (07) r2 += -8
3: (18) r1 = 0x0
5: (85) call bpf_map_lookup_elem#1
6: (15) if r0 == 0x0 goto pc+7
  R0=map_value(id=0,off=0,ks=8,vs=8,imm=0) R10=fp0,call_-1
7: (7a) *(u64 *)(r10 -16) = -8
8: (79) r1 = *(u64 *)(r10 -16)
9: (b7) r2 = -1
10: (2d) if r1 > r2 goto pc+3
  R0=map_value(id=0,off=0,ks=8,vs=8,imm=0) R1=inv(id=0) R2=inv-1 
R10=fp0,call_-1
11: (65) if r1 s> 0x1 goto pc+2
  R0=map_value(id=0,off=0,ks=8,vs=8,imm=0) R1=inv(id=0,smax_value=1) 
R2=inv-1 R10=fp0,call_-1
12: (0f) r0 += r1
R1 has unknown scalar with mixed signed bounds, pointer arithmetic with 
it prohibited for !root


0: (7a) *(u64 *)(r10 -8) = 0
1: (bf) r2 = r10
2: (07) r2 += -8
3: (18) r1 = 0x0
5: (85) call bpf_map_lookup_elem#1
6: (15) if r0 == 0x0 goto pc+7
  R0=map_value(id=0,off=0,ks=8,vs=8,imm=0) R10=fp0,call_-1
7: (7a) *(u64 *)(r10 -16) = -8
8: (79) r1 = *(u64 *)(r10 -16)
9: (b7) r2 = -1
10: (2d) if r1 > r2 goto pc+3
  R0=map_value(id=0,off=0,ks=8,vs=8,imm=0) R1=inv(id=0) R2=inv-1 
R10=fp0,call_-1
11: (65) if r1 s> 0x1 goto pc+2
  R0=map_value(id=0,off=0,ks=8,vs=8,imm=0) R1=inv(id=0,smax_value=1) 
R2=inv-1 R10=fp0,call_-1
12: (0f) r0 += r1
R1 has unknown scalar with mixed signed bounds, pointer arithmetic with 
it prohibited for !root
#423/p bounds checks mixing signed and unsigned OK


#439/u subtraction bounds (map value) variant 2 FAIL
Unexpected error message!
     EXP: R0 min value is negative, either use unsigned index or do a if 
(index >=0) check.
     RES: 0: (7a) *(u64 *)(r10 -8) = 0
1: (bf) r2 = r10
2: (07) r2 += -8
3: (18) r1 = 0x0
5: (85) call bpf_map_lookup_elem#1
6: (15) if r0 == 0x0 goto pc+8
  R0=map_value(id=0,off=0,ks=8,vs=8,imm=0) R10=fp0,call_-1
7: (71) r1 = *(u8 *)(r0 +0)
  R0=map_value(id=0,off=0,ks=8,vs=8,imm=0) R10=fp0,call_-1
8: (25) if r1 > 0xff goto pc+6
9: (71) r3 = *(u8 *)(r0 +1)
  R0=map_value(id=0,off=0,ks=8,vs=8,imm=0) 
R1=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R10=fp0,call_-1
10: (25) if r3 > 0xff goto pc+4
11: (1f) r1 -= r3
12: (0f) r0 += r1
R1 has unknown scalar with mixed signed bounds, pointer arithmetic with 
it prohibited for !root


0: (7a) *(u64 *)(r10 -8) = 0
1: (bf) r2 = r10
2: (07) r2 += -8
3: (18) r1 = 0x0
5: (85) call bpf_map_lookup_elem#1
6: (15) if r0 == 0x0 goto pc+8
  R0=map_value(id=0,off=0,ks=8,vs=8,imm=0) R10=fp0,call_-1
7: (71) r1 = *(u8 *)(r0 +0)
  R0=map_value(id=0,off=0,ks=8,vs=8,imm=0) R10=fp0,call_-1
8: (25) if r1 > 0xff goto pc+6
9: (71) r3 = *(u8 *)(r0 +1)
  R0=map_value(id=0,off=0,ks=8,vs=8,imm=0) 
R1=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R10=fp0,call_-1
10: (25) if r3 > 0xff goto pc+4
11: (1f) r1 -= r3
12: (0f) r0 += r1
R1 has unknown scalar with mixed signed bounds, pointer arithmetic with 
it prohibited for !root
#439/p subtraction bounds (map value) variant 2 OK


#452/u bounds check map access with off+size signed 32bit overflow. 
test2 FAIL
Unexpected error message!
     EXP: pointer offset 1073741822
     RES: 0: (7a) *(u64 *)(r10 -8) = 0
1: (bf) r2 = r10
2: (07) r2 += -8
3: (18) r1 = 0x0
5: (85) call bpf_map_lookup_elem#1
6: (55) if r0 != 0x0 goto pc+1
  R0=inv0 R10=fp0,call_-1
7: (95) exit

from 6 to 8: R0=map_value(id=0,off=0,ks=8,vs=8,imm=0) R10=fp0,call_-1
8: (07) r0 += 536870911
  R0_w=map_value(id=0,off=536870911,ks=8,vs=8,imm=0) R10=fp0,call_-1
invalid access to map value, value_size=8 off=536870911 size=1
R0 min value is outside of the array range
R0 pointer arithmetic of map value goes out of range, prohibited for !root

0: (7a) *(u64 *)(r10 -8) = 0
1: (bf) r2 = r10
2: (07) r2 += -8
3: (18) r1 = 0x0
5: (85) call bpf_map_lookup_elem#1
6: (55) if r0 != 0x0 goto pc+1
  R0=inv0 R10=fp0,call_-1
7: (95) exit

from 6 to 8: R0=map_value(id=0,off=0,ks=8,vs=8,imm=0) R10=fp0,call_-1
8: (07) r0 += 536870911
  R0_w=map_value(id=0,off=536870911,ks=8,vs=8,imm=0) R10=fp0,call_-1
invalid access to map value, value_size=8 off=536870911 size=1
R0 min value is outside of the array range
R0 pointer arithmetic of map value goes out of range, prohibited for !root
#452/p bounds check map access with off+size signed 32bit overflow. test2 OK


#462/u direct stack access with 32-bit wraparound. test3 FAIL
Unexpected error message!
     EXP: fp pointer offset 1073741822
     RES: 0: (bf) r1 = r10
1: (07) r1 += 536870911
invalid stack off=536870911 size=1
R1 stack pointer arithmetic goes out of range, prohibited for !root

0: (bf) r1 = r10
1: (07) r1 += 536870911
invalid stack off=536870911 size=1
R1 stack pointer arithmetic goes out of range, prohibited for !root
#462/p direct stack access with 32-bit wraparound. test3 OK


#531/u check deducing bounds from const, 1 FAIL
Unexpected error message!
     EXP: R0 tried to subtract pointer from scalar
     RES: 0: (b7) r0 = 1
1: (75) if r0 s>= 0x1 goto pc+0
2: (1f) r0 -= r1
R0 tried to sub from different maps, paths, or prohibited types

0: (b7) r0 = 1
1: (75) if r0 s>= 0x1 goto pc+0
2: (1f) r0 -= r1
R0 tried to sub from different maps, paths, or prohibited types
#531/p check deducing bounds from const, 1 OK

Re: [QUESTION] BPF kernel selftests failed in the LTS stable kernel 4.19.x

[Greg Kroah-Hartman]

On Thu, May 27, 2021 at 10:27:51AM +0800, Tiezhu Yang wrote:
> Hi all,
> 
> When update the following LTS stable kernel 4.19.x,
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/log/?h=linux-4.19.y
> 
> and then run BPF selftests according to
> https://www.kernel.org/doc/html/latest/bpf/bpf_devel_QA.html#q-how-to-run-bpf-selftests
> 
> $ cd tools/testing/selftests/bpf/
> $ make
> $ sudo ./test_verifier
> $ sudo make run_tests
> 
> there exists many failures include verifier tests and run_tests,
> (1) is it necessary to make sure that there are no any failures in the LTS
> stable kernel 4.19.x?

Yes, it would be nice if that did not happen.

> (2) if yes, how to fix these failures in the LTS stable kernel 4.19.x?

Can you find the offending commits by using `git bisect` and find the
upstream commits that resolve this and let us know so we can backport
them?

thanks,

greg k-h

Re: [QUESTION] BPF kernel selftests failed in the LTS stable kernel 4.19.x

[Tiezhu Yang]

On 05/27/2021 12:24 PM, Greg Kroah-Hartman wrote:
> On Thu, May 27, 2021 at 10:27:51AM +0800, Tiezhu Yang wrote:
>> Hi all,
>>
>> When update the following LTS stable kernel 4.19.x,
>> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/log/?h=linux-4.19.y
>>
>> and then run BPF selftests according to
>> https://www.kernel.org/doc/html/latest/bpf/bpf_devel_QA.html#q-how-to-run-bpf-selftests
>>
>> $ cd tools/testing/selftests/bpf/
>> $ make
>> $ sudo ./test_verifier
>> $ sudo make run_tests
>>
>> there exists many failures include verifier tests and run_tests,
>> (1) is it necessary to make sure that there are no any failures in the LTS
>> stable kernel 4.19.x?
> Yes, it would be nice if that did not happen.
>
>> (2) if yes, how to fix these failures in the LTS stable kernel 4.19.x?
> Can you find the offending commits by using `git bisect` and find the
> upstream commits that resolve this and let us know so we can backport
> them?
>
> thanks,
>
> greg k-h

I compared the related code in 4.19.y and upstream mainline, some failures
disappeared after add ".flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS,", but
other failures still exist, and I didn't have enough in-depth knowledge to
touch things elsewhere.

The failures can be easily reproduced, I would greatly appreciate it if 
anyone
is interested to fix them.

Re: [QUESTION] BPF kernel selftests failed in the LTS stable kernel 4.19.x

[Greg Kroah-Hartman]

On Thu, May 27, 2021 at 04:18:15PM +0800, Tiezhu Yang wrote:
> On 05/27/2021 12:24 PM, Greg Kroah-Hartman wrote:
> > On Thu, May 27, 2021 at 10:27:51AM +0800, Tiezhu Yang wrote:
> > > Hi all,
> > > 
> > > When update the following LTS stable kernel 4.19.x,
> > > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/log/?h=linux-4.19.y
> > > 
> > > and then run BPF selftests according to
> > > https://www.kernel.org/doc/html/latest/bpf/bpf_devel_QA.html#q-how-to-run-bpf-selftests
> > > 
> > > $ cd tools/testing/selftests/bpf/
> > > $ make
> > > $ sudo ./test_verifier
> > > $ sudo make run_tests
> > > 
> > > there exists many failures include verifier tests and run_tests,
> > > (1) is it necessary to make sure that there are no any failures in the LTS
> > > stable kernel 4.19.x?
> > Yes, it would be nice if that did not happen.
> > 
> > > (2) if yes, how to fix these failures in the LTS stable kernel 4.19.x?
> > Can you find the offending commits by using `git bisect` and find the
> > upstream commits that resolve this and let us know so we can backport
> > them?
> > 
> > thanks,
> > 
> > greg k-h
> 
> I compared the related code in 4.19.y and upstream mainline, some failures
> disappeared after add ".flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS,", but
> other failures still exist, and I didn't have enough in-depth knowledge to
> touch things elsewhere.
> 
> The failures can be easily reproduced, I would greatly appreciate it if
> anyone
> is interested to fix them.

Sounds like you are interested, why not work on this instead of waiting
for someone else?  It's not like many people are interested in 4.19 at
this point in time.

good luck!

greg k-h