[Question] bpf: about a new 'tools/bpf/bpf_dwarf2btf'

[Question] bpf: about a new 'tools/bpf/bpf_dwarf2btf'

[Taeung Song]

Hi,

Building bpf programs with .BTF section,
I thought it'd be better to convert dwarf info to .BTF by
a new tool such as 'tools/bpf/bpf_dwarf2btf' instead of pahole
in the future.

Currently for bpf binary that have .BTF section,
we need to use pahole from https://github.com/iamkafai/pahole/tree/btf
with the command line such as "pahole -J bpf_prog.o".

I think it is great but if implementing new 'bpf_dwarf2btf'
(dwarf parsing + btf encoder code written by Martin KaFai Lau on
the pahole project i.e. btf.h, btf_encoder.c, btf_encoder.h, libbtf.c, 
libbtf.h),
BPF developers would more easily use functionalities based on BTF.

What do you think about this ? Do you think this is needed ?
Or, already implementing something like this ?

If it is needed, I want to try to make 'tools/bpf/bpf_dwarf2btf'
based on the pahole code. I'd appreciate it, if you reply to this

Thanks,
Taeung

Re: pahole + BTF was: Re: [Question] bpf: about a new 'tools/bpf/bpf_dwarf2btf'

[Taeung Song]

Hi Arnaldo,

On 07/26/2018 02:52 AM, Arnaldo Carvalho de Melo wrote:
> Em Thu, Jul 26, 2018 at 02:23:32AM +0900, Taeung Song escreveu:
>> Hi,
>>
>> Building bpf programs with .BTF section,
>> I thought it'd be better to convert dwarf info to .BTF by
>> a new tool such as 'tools/bpf/bpf_dwarf2btf' instead of pahole
>> in the future.
>   
>> Currently for bpf binary that have .BTF section,
>> we need to use pahole from https://github.com/iamkafai/pahole/tree/btf
>> with the command line such as "pahole -J bpf_prog.o".
>   
>> I think it is great but if implementing new 'bpf_dwarf2btf'
>> (dwarf parsing + btf encoder code written by Martin KaFai Lau on
>> the pahole project i.e. btf.h, btf_encoder.c, btf_encoder.h,
>> libbtf.c, libbtf.h),
>> BPF developers would more easily use functionalities based on BTF.
> 
> What would be easier exactly? Not having to install a package but build
> it from the kernel sources?
> 
> Many kernel developers already have pahole installed for other uses, so
> no need to install anything.
> 

Understood, but I think there are many non-kernel developers
developing BPF programs and they mightn't have or use pahole.

So, if providing the 'dwarf2btf' feature on tools/bpf or tools/bpf/bpftool,
non-kernel developers can also more easily build bpf prog with .BPF, no ?

> BTW, Daniel, I just pushed to pahole's main repository at:
> 
>    git://git.kernel.org/pub/scm/devel/pahole/pahole.git
> 
> with the Martin's BTF patch, so no need to pull from the github one,
> I'll tag v1.12 and announce the release so that distro package
> maintainers can update their packages.
>   
>> What do you think about this ? Do you think this is needed ?
>> Or, already implementing something like this ?
>   
>> If it is needed, I want to try to make 'tools/bpf/bpf_dwarf2btf'
>> based on the pahole code. I'd appreciate it, if you reply to this
> 
> The way Martin took advantage of the work done a long time ago to
> support CTF out of the same DWARF reading codebase was really cool, not
> that much work to do, just add a new format to pahole's codebase making
> it more useful.
> 

I got it !

Thanks,
Taeung

> I was just so far overly picky with testing it and kept leaving for
> later to have a good documentation about testing it, vacation and perf
> maintainership duties kept making this take like forever, grumble :-\
> 
> - Arnaldo
> 

--
oh.. :'(

Re: pahole + BTF was: Re: [Question] bpf: about a new 'tools/bpf/bpf_dwarf2btf'

[Daniel Borkmann]

On 07/25/2018 08:27 PM, Taeung Song wrote:
> On 07/26/2018 02:52 AM, Arnaldo Carvalho de Melo wrote:
[...]
>> BTW, Daniel, I just pushed to pahole's main repository at:
>>
>>    git://git.kernel.org/pub/scm/devel/pahole/pahole.git
>>
>> with the Martin's BTF patch, so no need to pull from the github one,
>> I'll tag v1.12 and announce the release so that distro package
>> maintainers can update their packages.

Awesome, thanks so much Arnaldo!

Re: pahole + BTF was: Re: [Question] bpf: about a new 'tools/bpf/bpf_dwarf2btf'

[Taeung Song]

On 07/26/2018 03:27 AM, Taeung Song wrote:
> Hi Arnaldo,
> 
> On 07/26/2018 02:52 AM, Arnaldo Carvalho de Melo wrote:
>> Em Thu, Jul 26, 2018 at 02:23:32AM +0900, Taeung Song escreveu:
>>> Hi,
>>>
>>> Building bpf programs with .BTF section,
>>> I thought it'd be better to convert dwarf info to .BTF by
>>> a new tool such as 'tools/bpf/bpf_dwarf2btf' instead of pahole
>>> in the future.
>>> Currently for bpf binary that have .BTF section,
>>> we need to use pahole from https://github.com/iamkafai/pahole/tree/btf
>>> with the command line such as "pahole -J bpf_prog.o".
>>> I think it is great but if implementing new 'bpf_dwarf2btf'
>>> (dwarf parsing + btf encoder code written by Martin KaFai Lau on
>>> the pahole project i.e. btf.h, btf_encoder.c, btf_encoder.h,
>>> libbtf.c, libbtf.h),
>>> BPF developers would more easily use functionalities based on BTF.
>>
>> What would be easier exactly? Not having to install a package but build
>> it from the kernel sources?
>>
>> Many kernel developers already have pahole installed for other uses, so
>> no need to install anything.
>>
> 
> Understood, but I think there are many non-kernel developers
> developing BPF programs and they mightn't have or use pahole.
> 
> So, if providing the 'dwarf2btf' feature on tools/bpf or tools/bpf/bpftool,
> non-kernel developers can also more easily build bpf prog with .BPF, no ?
> 

Or, if tools/lib/bpf/ have the 'dwarf2btf' feature,
I think BPF developers can just use bpf programs that have dwarf info
after compiling with clang '-g' and llc '-mattr=dwarfris', even though 
not using pahole.
Isn't it good way ?

>> BTW, Daniel, I just pushed to pahole's main repository at:
>>
>>    git://git.kernel.org/pub/scm/devel/pahole/pahole.git
>>
>> with the Martin's BTF patch, so no need to pull from the github one,
>> I'll tag v1.12 and announce the release so that distro package
>> maintainers can update their packages.
>>> What do you think about this ? Do you think this is needed ?
>>> Or, already implementing something like this ?
>>> If it is needed, I want to try to make 'tools/bpf/bpf_dwarf2btf'
>>> based on the pahole code. I'd appreciate it, if you reply to this
>>
>> The way Martin took advantage of the work done a long time ago to
>> support CTF out of the same DWARF reading codebase was really cool, not
>> that much work to do, just add a new format to pahole's codebase making
>> it more useful.
>>
> 
> I got it !
> 
> Thanks,
> Taeung
> 
>> I was just so far overly picky with testing it and kept leaving for
>> later to have a good documentation about testing it, vacation and perf
>> maintainership duties kept making this take like forever, grumble :-\
>>
>> - Arnaldo
>>
> 
> -- 
> oh.. :'(

Re: pahole + BTF was: Re: [Question] bpf: about a new 'tools/bpf/bpf_dwarf2btf'

[Martin KaFai Lau]

On Thu, Jul 26, 2018 at 04:21:31AM +0900, Taeung Song wrote:
> 
> 
> On 07/26/2018 03:27 AM, Taeung Song wrote:
> > Hi Arnaldo,
> > 
> > On 07/26/2018 02:52 AM, Arnaldo Carvalho de Melo wrote:
> > > Em Thu, Jul 26, 2018 at 02:23:32AM +0900, Taeung Song escreveu:
> > > > Hi,
> > > > 
> > > > Building bpf programs with .BTF section,
> > > > I thought it'd be better to convert dwarf info to .BTF by
> > > > a new tool such as 'tools/bpf/bpf_dwarf2btf' instead of pahole
> > > > in the future.
> > > > Currently for bpf binary that have .BTF section,
> > > > we need to use pahole from https://github.com/iamkafai/pahole/tree/btf
> > > > with the command line such as "pahole -J bpf_prog.o".
> > > > I think it is great but if implementing new 'bpf_dwarf2btf'
> > > > (dwarf parsing + btf encoder code written by Martin KaFai Lau on
> > > > the pahole project i.e. btf.h, btf_encoder.c, btf_encoder.h,
> > > > libbtf.c, libbtf.h),
> > > > BPF developers would more easily use functionalities based on BTF.
> > > 
> > > What would be easier exactly? Not having to install a package but build
> > > it from the kernel sources?
> > > 
> > > Many kernel developers already have pahole installed for other uses, so
> > > no need to install anything.
> > > 
> > 
> > Understood, but I think there are many non-kernel developers
> > developing BPF programs and they mightn't have or use pahole.
> > 
> > So, if providing the 'dwarf2btf' feature on tools/bpf or tools/bpf/bpftool,
> > non-kernel developers can also more easily build bpf prog with .BPF, no ?
Some quick thoughts,
IMO, I suspect if it is in the distro's pahole package,  it should be easy
enough for kernel and non kernel developer to install.
BTF usage is still evolving,  we might re-evaluate going forward but at this
point I think leveraging pahole's existing capability is a good option.

> > 
> 
> Or, if tools/lib/bpf/ have the 'dwarf2btf' feature,
> I think BPF developers can just use bpf programs that have dwarf info
> after compiling with clang '-g' and llc '-mattr=dwarfris', even though not
> using pahole.
> Isn't it good way ?
> 
> > > BTW, Daniel, I just pushed to pahole's main repository at:
> > > 
> > >    git://git.kernel.org/pub/scm/devel/pahole/pahole.git
> > > 
> > > with the Martin's BTF patch, so no need to pull from the github one,
> > > I'll tag v1.12 and announce the release so that distro package
> > > maintainers can update their packages.
Awesome! Thanks, Arnaldo!


- Martin

Re: pahole + BTF was: Re: [Question] bpf: about a new 'tools/bpf/bpf_dwarf2btf'

[Daniel Borkmann]

On 07/25/2018 10:11 PM, Martin KaFai Lau wrote:
> On Thu, Jul 26, 2018 at 04:21:31AM +0900, Taeung Song wrote:
>> On 07/26/2018 03:27 AM, Taeung Song wrote:
>>> On 07/26/2018 02:52 AM, Arnaldo Carvalho de Melo wrote:
>>>> Em Thu, Jul 26, 2018 at 02:23:32AM +0900, Taeung Song escreveu:
>>>>> Hi,
>>>>>
>>>>> Building bpf programs with .BTF section,
>>>>> I thought it'd be better to convert dwarf info to .BTF by
>>>>> a new tool such as 'tools/bpf/bpf_dwarf2btf' instead of pahole
>>>>> in the future.
>>>>> Currently for bpf binary that have .BTF section,
>>>>> we need to use pahole from https://github.com/iamkafai/pahole/tree/btf
>>>>> with the command line such as "pahole -J bpf_prog.o".
>>>>> I think it is great but if implementing new 'bpf_dwarf2btf'
>>>>> (dwarf parsing + btf encoder code written by Martin KaFai Lau on
>>>>> the pahole project i.e. btf.h, btf_encoder.c, btf_encoder.h,
>>>>> libbtf.c, libbtf.h),
>>>>> BPF developers would more easily use functionalities based on BTF.
>>>>
>>>> What would be easier exactly? Not having to install a package but build
>>>> it from the kernel sources?
>>>>
>>>> Many kernel developers already have pahole installed for other uses, so
>>>> no need to install anything.
>>>
>>> Understood, but I think there are many non-kernel developers
>>> developing BPF programs and they mightn't have or use pahole.
>>>
>>> So, if providing the 'dwarf2btf' feature on tools/bpf or tools/bpf/bpftool,
>>> non-kernel developers can also more easily build bpf prog with .BPF, no ?
> Some quick thoughts,
> IMO, I suspect if it is in the distro's pahole package,  it should be easy
> enough for kernel and non kernel developer to install.
> BTF usage is still evolving,  we might re-evaluate going forward but at this
> point I think leveraging pahole's existing capability is a good option.

Agree, if there will be a future use-case where pahole might not be well-fitting,
we could add it to bpftool then so I wouldn't rule it out, but for the functionality
right now it seems good to reuse it. Presumably BPF developers have it installed
anyway to inspect struct padding from BPF obj files.

Thanks,
Daniel

Re: pahole + BTF was: Re: [Question] bpf: about a new 'tools/bpf/bpf_dwarf2btf'

[Taeung Song]

On 07/26/2018 09:26 AM, Daniel Borkmann wrote:
> On 07/25/2018 10:11 PM, Martin KaFai Lau wrote:
>> On Thu, Jul 26, 2018 at 04:21:31AM +0900, Taeung Song wrote:
>>> On 07/26/2018 03:27 AM, Taeung Song wrote:
>>>> On 07/26/2018 02:52 AM, Arnaldo Carvalho de Melo wrote:
>>>>> Em Thu, Jul 26, 2018 at 02:23:32AM +0900, Taeung Song escreveu:
>>>>>> Hi,
>>>>>>
>>>>>> Building bpf programs with .BTF section,
>>>>>> I thought it'd be better to convert dwarf info to .BTF by
>>>>>> a new tool such as 'tools/bpf/bpf_dwarf2btf' instead of pahole
>>>>>> in the future.
>>>>>> Currently for bpf binary that have .BTF section,
>>>>>> we need to use pahole from https://github.com/iamkafai/pahole/tree/btf
>>>>>> with the command line such as "pahole -J bpf_prog.o".
>>>>>> I think it is great but if implementing new 'bpf_dwarf2btf'
>>>>>> (dwarf parsing + btf encoder code written by Martin KaFai Lau on
>>>>>> the pahole project i.e. btf.h, btf_encoder.c, btf_encoder.h,
>>>>>> libbtf.c, libbtf.h),
>>>>>> BPF developers would more easily use functionalities based on BTF.
>>>>>
>>>>> What would be easier exactly? Not having to install a package but build
>>>>> it from the kernel sources?
>>>>>
>>>>> Many kernel developers already have pahole installed for other uses, so
>>>>> no need to install anything.
>>>>
>>>> Understood, but I think there are many non-kernel developers
>>>> developing BPF programs and they mightn't have or use pahole.
>>>>
>>>> So, if providing the 'dwarf2btf' feature on tools/bpf or tools/bpf/bpftool,
>>>> non-kernel developers can also more easily build bpf prog with .BPF, no ?
>> Some quick thoughts,
>> IMO, I suspect if it is in the distro's pahole package,  it should be easy
>> enough for kernel and non kernel developer to install.
>> BTF usage is still evolving,  we might re-evaluate going forward but at this
>> point I think leveraging pahole's existing capability is a good option.
> 
> Agree, if there will be a future use-case where pahole might not be well-fitting,
> we could add it to bpftool then so I wouldn't rule it out, but for the functionality
> right now it seems good to reuse it. Presumably BPF developers have it installed
> anyway to inspect struct padding from BPF obj files.
> 
> Thanks,
> Daniel
> 

OK, I got it, thanks for answering my question so detailedly

Thanks,
Taeung