Re: [PATCH v3 bpf-next] selftests/bpf: Add parallelism to test_progs

[Yonghong Song <yhs@fb.com>]

On 9/9/21 12:35 PM, Yucong Sun wrote:
> From: Yucong Sun <sunyucong@gmail.com>
> 
> This patch adds "-j" mode to test_progs, executing tests in multiple process.
> "-j" mode is optional, and works with all existing test selection mechanism, as
> well as "-v", "-l" etc.
> 
> In "-j" mode, main process use UDS/DGRAM to communicate to each forked worker,
> commanding it to run tests and collect logs. After all tests are finished, a
> summary is printed. main process use multiple competing threads to dispatch
> work to worker, trying to keep them all busy.
> 
> Example output:
> 
>    > ./test_progs -n 15-20 -j
>    [    8.584709] bpf_testmod: loading out-of-tree module taints kernel.
>    Launching 2 workers.
>    [0]: Running test 15.
>    [1]: Running test 16.
>    [1]: Running test 17.
>    [1]: Running test 18.
>    [1]: Running test 19.
>    [1]: Running test 20.
>    [1]: worker exit.
>    [0]: worker exit.
>    #15 btf_dump:OK
>    #16 btf_endian:OK
>    #17 btf_map_in_map:OK
>    #18 btf_module:OK
>    #19 btf_skc_cls_ingress:OK
>    #20 btf_split:OK
>    Summary: 6/20 PASSED, 0 SKIPPED, 0 FAILED

I tried the patch with latest bpf-next and
 
https://lore.kernel.org/bpf/20210909215658.hgqkvxvtjrvdnrve@revolver/T/#u
to avoid kernel warning.

My commandline is ./test_progs -j
my env is a 4 cpu qemu.
It seems the test is stuck and cannot finish:
...
Still waiting for thread 0 (test 0). 
 

Still waiting for thread 0 (test 0). 
 

Still waiting for thread 0 (test 0). 
 

Still waiting for thread 0 (test 0). 
 

Still waiting for thread 0 (test 0). 
 

 
 

[1]+  Stopped                 ./test_progs -j
root@arch-fb-vm1:~/net-next/bpf-next/tools/testing/selftests/bpf ps
   PID TTY          TIME CMD
   231 ttyS0    00:00:00 bash
   254 ttyS0    00:00:00 test_progs
   256 ttyS0    00:00:00 test_progs <defunct>
   257 ttyS0    00:00:12 new_name <defunct>
   258 ttyS0    00:01:03 test_progs <defunct>
   259 ttyS0    00:00:02 test_progs <defunct>
  1310 ttyS0    00:00:00 ps
root@arch-fb-vm1:~/net-next/bpf-next/tools/testing/selftests/bpf fg
./test_progs -j
Still waiting for thread 0 (test 0).
Still waiting for thread 0 (test 0).
Still waiting for thread 0 (test 0).
Still waiting for thread 0 (test 0).
Still waiting for thread 0 (test 0).
Still waiting for thread 0 (test 0).
...

I didn't further debug what is the issue.

> 
> Know issue:
> 
> Some tests fail when running concurrently, later patch will either
> fix the test or pin them to worker 0.
> 
> Signed-off-by: Yucong Sun <sunyucong@gmail.com>
> 
> V3 -> V2: fix missing outputs in commit messages.
> V2 -> V1: switch to UDS client/server model.
> ---
>   tools/testing/selftests/bpf/test_progs.c | 456 ++++++++++++++++++++++-
>   tools/testing/selftests/bpf/test_progs.h |  36 +-
>   2 files changed, 478 insertions(+), 14 deletions(-)
> 
[...]