Re: [RFC bpf-next 00/12] Use uapi kernel headers with vmlinux.h

[Yonghong Song <yhs@meta.com>]

On 11/11/22 1:55 PM, Eduard Zingerman wrote:
> On Fri, 2022-10-28 at 11:56 -0700, Yonghong Song wrote:
>>>> [...]
>>>
>>> Ok, could we change the problem to detecting if some type is defined.
>>> Would it be possible to have something like
>>>
>>> #if !__is_type_defined(struct abc)
>>> struct abc {
>>> };
>>> #endif
>>>
>>> I think we talked about this and there were problems with this
>>> approach, but I don't remember details and how insurmountable the
>>> problem is. Having a way to check whether some type is defined would
>>> be very useful even outside of -target bpf parlance, though, so maybe
>>> it's the problem worth attacking?
>>
>> Yes, we discussed this before. This will need to add additional work
>> in preprocessor. I just made a discussion topic in llvm discourse
>>
>> https://discourse.llvm.org/t/add-a-type-checking-macro-is-type-defined-type/66268
>>
>> Let us see whether we can get some upstream agreement or not.
> 
> I did a small investigation of this feature.
> 
> The main pre-requirement is construction of the symbol table during
> source code pre-processing, which implies necessity to parse the
> source code at the same time. It is technically possible in clang, as
> lexing, pre-processing and AST construction happens at the same time
> when in compilation mode.
> 
> The prototype is available here [1], it includes:
> - Change in the pre-processor that adds an optional callback
>    "IsTypeDefinedFn" & necessary parsing of __is_type_defined
>    construct.
> - Change in Sema module (responsible for parsing/AST & symbol table)
>    that installs the appropriate "IsTypeDefinedFn" in the pre-processor
>    instance.
> 
> However, this prototype builds a backward dependency between
> pre-processor and semantic analysis. There are currently no such
> dependencies in the clang code base.
> 
> This makes it impossible to do pre-processing and compilation
> separately, e.g. consider the following example:
> 
> $ cat test.c
> 
>    struct foo { int x; };
>    
>    #if __is_type_defined(foo)
>      const int x = 1;
>    #else
>      const int x = 2;
>    #endif
>    
> $ clang -cc1 -ast-print test.c -o -
> 
>    struct foo {
>        int x;
>    };
>    const int x = 1;
> 
> $ clang -E test.c -o -
> 
>    # ... some line directives ...
>    struct foo { int x; };
>    const int x = 2;

Is it any chance '-E' could output the same one as '-cc1 -ast-print'?
That is, even with -E we could do some semantics analysis
as well, using either current clang semantics analysis or creating
an minimal version of sema analysis in preprocessor itself?

> 
> Note that __is_type_defined is computed to different value in the
> first and second calls. This is so because semantic analysis (AST,
> symbol table) is not done for -E.
> 
> It also breaks that C11 standard which clearly separates
> pre-processing and semantic analysis phases, see [2] 5.1.1.2.
> 
> So, my conclusion is as follows: this is technically possible in clang
> but has no chance to reach llvm upstream.
> 
> Thanks,
> Eduard
> 
> [1] https://github.com/llvm/llvm-project/compare/main...eddyz87:llvm-project:is-type-defined-experiment
> [2] https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1548.pdf
> 
> 
>>
>>>
>>>>
>>>>>
>>>>> BTW, I suggest splitting libbpf btf_dedup and btf_dump changes into a
>>>>> separate series and sending them as non-RFC sooner. Those improvements
>>>>> are independent of all the header guards stuff, let's get them landed
>>>>> sooner.
>>>>>
>>>>>> After some discussion with Alexei and Yonghong I'd like to request
>>>>>> your comments regarding a somewhat brittle and partial solution to
>>>>>> this issue that relies on adding `#ifndef FOO_H ... #endif` guards in
>>>>>> the generated `vmlinux.h`.
>>>>>>
>>>>>
>>>>> [...]
>>>>>
>>>>>> Eduard Zingerman (12):
>>>>>>      libbpf: Deduplicate unambigous standalone forward declarations
>>>>>>      selftests/bpf: Tests for standalone forward BTF declarations
>>>>>>        deduplication
>>>>>>      libbpf: Support for BTF_DECL_TAG dump in C format
>>>>>>      selftests/bpf: Tests for BTF_DECL_TAG dump in C format
>>>>>>      libbpf: Header guards for selected data structures in vmlinux.h
>>>>>>      selftests/bpf: Tests for header guards printing in BTF dump
>>>>>>      bpftool: Enable header guards generation
>>>>>>      kbuild: Script to infer header guard values for uapi headers
>>>>>>      kbuild: Header guards for types from include/uapi/*.h in kernel BTF
>>>>>>      selftests/bpf: Script to verify uapi headers usage with vmlinux.h
>>>>>>      selftests/bpf: Known good uapi headers for test_uapi_headers.py
>>>>>>      selftests/bpf: script for infer_header_guards.pl testing
>>>>>>
>>>>>>     scripts/infer_header_guards.pl                | 191 +++++
>>>>>>     scripts/link-vmlinux.sh                       |  13 +-
>>>>>>     tools/bpf/bpftool/btf.c                       |   4 +-
>>>>>>     tools/lib/bpf/btf.c                           | 178 ++++-
>>>>>>     tools/lib/bpf/btf.h                           |   7 +-
>>>>>>     tools/lib/bpf/btf_dump.c                      | 232 +++++-
>>>>>>     .../selftests/bpf/good_uapi_headers.txt       | 677 ++++++++++++++++++
>>>>>>     tools/testing/selftests/bpf/prog_tests/btf.c  | 152 ++++
>>>>>>     .../selftests/bpf/prog_tests/btf_dump.c       |  11 +-
>>>>>>     .../bpf/progs/btf_dump_test_case_decl_tag.c   |  39 +
>>>>>>     .../progs/btf_dump_test_case_header_guards.c  |  94 +++
>>>>>>     .../bpf/test_uapi_header_guards_infer.sh      |  33 +
>>>>>>     .../selftests/bpf/test_uapi_headers.py        | 197 +++++
>>>>>>     13 files changed, 1816 insertions(+), 12 deletions(-)
>>>>>>     create mode 100755 scripts/infer_header_guards.pl
>>>>>>     create mode 100644 tools/testing/selftests/bpf/good_uapi_headers.txt
>>>>>>     create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.c
>>>>>>     create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.c
>>>>>>     create mode 100755 tools/testing/selftests/bpf/test_uapi_header_guards_infer.sh
>>>>>>     create mode 100755 tools/testing/selftests/bpf/test_uapi_headers.py
>>>>>>
>>>>>> --
>>>>>> 2.34.1
>>>>>>
>