Re: next-20190723: bpf/seccomp - systemd/journald issue?

[Yonghong Song <yhs@fb.com>]

On 7/27/19 12:36 AM, Sedat Dilek wrote:
> On Sat, Jul 27, 2019 at 4:24 AM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
>>
>> On Fri, Jul 26, 2019 at 2:19 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>>
>>> On Fri, Jul 26, 2019 at 11:10 PM Yonghong Song <yhs@fb.com> wrote:
>>>>
>>>>
>>>>
>>>> On 7/26/19 2:02 PM, Sedat Dilek wrote:
>>>>> On Fri, Jul 26, 2019 at 10:38 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>>>>>
>>>>>> Hi Yonghong Song,
>>>>>>
>>>>>> On Fri, Jul 26, 2019 at 5:45 PM Yonghong Song <yhs@fb.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 7/26/19 1:26 AM, Sedat Dilek wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I have opened a new issue in the ClangBuiltLinux issue tracker.
>>>>>>>
>>>>>>> Glad to know clang 9 has asm goto support and now It can compile
>>>>>>> kernel again.
>>>>>>>
>>>>>>
>>>>>> Yupp.
>>>>>>
>>>>>>>>
>>>>>>>> I am seeing a problem in the area bpf/seccomp causing
>>>>>>>> systemd/journald/udevd services to fail.
>>>>>>>>
>>>>>>>> [Fri Jul 26 08:08:43 2019] systemd[453]: systemd-udevd.service: Failed
>>>>>>>> to connect stdout to the journal socket, ignoring: Connection refused
>>>>>>>>
>>>>>>>> This happens when I use the (LLVM) LLD ld.lld-9 linker but not with
>>>>>>>> BFD linker ld.bfd on Debian/buster AMD64.
>>>>>>>> In both cases I use clang-9 (prerelease).
>>>>>>>
>>>>>>> Looks like it is a lld bug.
>>>>>>>
>>>>>>> I see the stack trace has __bpf_prog_run32() which is used by
>>>>>>> kernel bpf interpreter. Could you try to enable bpf jit
>>>>>>>      sysctl net.core.bpf_jit_enable = 1
>>>>>>> If this passed, it will prove it is interpreter related.
>>>>>>>
>>>>>>
>>>>>> After...
>>>>>>
>>>>>> sysctl -w net.core.bpf_jit_enable=1
>>>>>>
>>>>>> I can start all failed systemd services.
>>>>>>
>>>>>> systemd-journald.service
>>>>>> systemd-udevd.service
>>>>>> haveged.service
>>>>>>
>>>>>> This is in maintenance mode.
>>>>>>
>>>>>> What is next: Do set a permanent sysctl setting for net.core.bpf_jit_enable?
>>>>>>
>>>>>
>>>>> This is what I did:
>>>>
>>>> I probably won't have cycles to debug this potential lld issue.
>>>> Maybe you already did, I suggest you put enough reproducible
>>>> details in the bug you filed against lld so they can take a look.
>>>>
>>>
>>> I understand and will put the journalctl-log into the CBL issue
>>> tracker and update informations.
>>>
>>> Thanks for your help understanding the BPF correlations.
>>>
>>> Is setting 'net.core.bpf_jit_enable = 2' helpful here?
>>
>> jit_enable=1 is enough.
>> Or use CONFIG_BPF_JIT_ALWAYS_ON to workaround.
>>
>> It sounds like clang miscompiles interpreter.
>> modprobe test_bpf
>> should be able to point out which part of interpreter is broken.
> 
> Maybe we need something like...
> 
> "bpf: Disable GCC -fgcse optimization for ___bpf_prog_run()"
> 
> ...for clang?

Not sure how do you get conclusion it is gcse causing the problem.
But anyway, adding such flag in the kernel is not a good idea.
clang/llvm should be fixed instead. Esp. there is still time
for 9.0.0 release to fix bugs.

> 
> - Sedat -
> 
> [1] https://git.kernel.org/linus/3193c0836f203a91bef96d88c64cccf0be090d9c
>