Re: cross compiling perf

[Milian Wolff <milian.wolff@kdab.com>]

[-- Attachment #1: Type: text/plain, Size: 10615 bytes --]

On Montag, 27. Juni 2016 13:56:23 CEST Milian Wolff wrote:
> On Monday, June 20, 2016 9:56:09 AM CEST Wangnan (F) wrote:
> > On 2016/6/17 19:00, Arnaldo Carvalho de Melo wrote:
> > > Adding some folks to the CC list, that either worked on the make
> > > infrastructure or that I know works with cross compiling environments,
> > > maybe they can help.
> > > 
> > > More below.
> > > 
> > > Em Fri, Jun 17, 2016 at 11:49:28AM +0200, Milian Wolff escreveu:
> > >> On Donnerstag, 16. Juni 2016 13:11:11 CEST Arnaldo Carvalho de Melo
> 
> wrote:
> > >>> Em Wed, Jun 15, 2016 at 04:24:52PM +0200, Milian Wolff escreveu:
> > >>>> I'm trying to compile a more modern version of the user-space perf
> > >>>> tools
> > >>>> for an arm64 embedded target. So far, no cigar.
> > >>>> 
> > >>>> Neither tools/build/Documentation nor
> > >>>> tools/perf/Documentation/Build.txt
> > >>>> explain how this should be done. Right now, I'm trying the following
> > >>>> from
> > >>>> an SDK with an environment that already sets up CC, CFLAGS etc. pp.
> > >>>> 
> > >>>> [SDK] ~/milian/linux/tools/perf$ make ARCH=arm64
> > >>>> CROSS_COMPILE=/home/sdk/
> > >>>> sysroots/x86_64-oesdk-linux/usr/bin/aarch64-gnu-linux/aarch64-gnu-lin
> > >>>> ux
> > >>>> -
> > >>>> CFLAGS="--sysroot=/home/sdk/sysroots/aarch64-gnu-linux
> > >>>> -I/home/milian/target- prefix/include
> > >>>> -L/home/milian/target-prefix/lib
> > >>>> $CFLAGS"
> > >>>> 
> > >>>>    BUILD:   Doing 'make -j8' parallel build
> > >>>> 
> > >>>> Auto-detecting system features:
> > >>>> ...                         dwarf: [ OFF ]
> > >>>> ...            dwarf_getlocations: [ OFF ]
> > >>>> ...                         glibc: [ OFF ]
> > >>>> ...                          gtk2: [ OFF ]
> > >>>> ...                      libaudit: [ OFF ]
> > >>>> ...                        libbfd: [ OFF ]
> > >>>> ...                        libelf: [ OFF ]
> > >>>> ...                       libnuma: [ OFF ]
> > >>>> ...        numa_num_possible_cpus: [ OFF ]
> > >>>> ...                       libperl: [ OFF ]
> > >>>> ...                     libpython: [ OFF ]
> > >>>> ...                      libslang: [ OFF ]
> > >>>> ...                     libcrypto: [ OFF ]
> > >>>> ...                     libunwind: [ OFF ]
> > >>>> ...            libdw-dwarf-unwind: [ OFF ]
> > >>>> ...                          zlib: [ OFF ]
> > >>>> ...                          lzma: [ OFF ]
> > >>>> ...                     get_cpuid: [ OFF ]
> > >>>> ...                           bpf: [ OFF ]
> > >>>> 
> > >>>> config/Makefile:272: *** No gnu/libc-version.h found, please install
> > >>>> glibc-
> > >>>> dev[el].  Stop
> > 
> > I think this error message is missleading. Let's see source code:
> > 
> > ifdef NO_LIBELF
> > 
> >    ...
> > 
> > else
> > 
> >    ifeq ($(feature-libelf), 0)
> >    
> >      ifeq ($(feature-glibc), 1)
> >      
> >        LIBC_SUPPORT := 1
> >      
> >      endif
> >      ifeq ($(BIONIC),1)
> >      
> >        LIBC_SUPPORT := 1
> >      
> >      endif
> >      ifeq ($(LIBC_SUPPORT),1)
> >      
> >        ...
> >      
> >      else
> >      
> >        ifneq ($(filter s% -static%,$(LDFLAGS),),)
> >        
> >          msg := $(error No static glibc found, please install
> >          glibc-static);
> > 
> > else
> > 
> >          msg := $(error No gnu/libc-version.h found, please install
> > 
> > glibc-dev[el]);
> > 
> >        endif
> >      
> >      endif
> >    
> >    else
> >    
> >      ...
> >    
> >    endif # libelf support
> > 
> > endif # NO_LIBELF
> > 
> > So the 'libc-version.h' error message really means the failure of
> > feature-glibc detector.
> > There's many reasons cause libc detector fail. 'libc-version.h' in error
> > message provides
> > wrong clue, lead user to check this file instead of checking feature
> > check result.
> 
> OK, so if it's the wrong clue, can you tell me what else to look at then?
> 
> I'm seeing
> 
> $ echo $CC
> aarch64-gnu-linux-gcc
> --sysroot=/home/yocto/SDK/E013/sdk/sysroots/aarch64-gnu- linux
> $ echo $PWD
> path/to/linux/tools/build/feature
> $ $CC test-glib.c
> $ echo $?
> 0
> 
> The above looks good, but from a make run I get:
> 
> $ cat test-glibc.make.output
> test-glibc.c:1:20: fatal error: stdlib.h: No such file or directory
>  #include <stdlib.h>
>                     ^
> compilation terminated.
> 
> Can I somehow figure out how it tries to compile the test-glib.c file? I.e.
> can I invoke "make" in a verbose mode that outputs all commands?

I have finally found the time to dig into these issues and now found a way to 
cross compile perf for my target platform. Here are the issues I needed to 
overcome:

a) CC can only point to the executable
For some reason, the toolchain I have to work with defines

CC=aarch64-gnu-linux-gcc --sysroot=/home/yocto/SDK/E013/sdk/sysroots/aarch64-
gnu-linux
CFLAGS=-O2 -pipe -g -feliminate-unused-debug-types -fno-omit-frame-pointer -
march=armv8-a -funwind-tables

This works everywhere else (qmake, configure, cmake), but the kernel 
buildsystem is more picky and will silently ignore the sysroot parameter set 
in CC. To fix this, one has to move the sysroot parameter into CFLAGS. This 
was the reason for the "test-glib.c" feature failure

b) Manually enable/disable features

Once the above is fixed, the feature detection is still completely broken for 
me. It seems to blindly enable all features for some reason:

~~~~~~~~~~~~~
export CC="aarch64-gnu-linux-gcc"
export CFLAGS="--sysroot=/home/yocto/SDK/E013/sdk/sysroots/aarch64-gnu-linux -
O2 -pipe -g -feliminate-unused-debug-types -fno-omit-frame-pointer -
march=armv8-a -funwind-tables"
export ARCH=arm64
export CROSS_COMPILE=aarch64-gnu-linux-
make V=1 VF=1 EXTRA_CFLAGS="$CFLAGS"
  BUILD:   Doing 'make -j8' parallel build

Auto-detecting system features:
...                         dwarf: [ on  ]
...            dwarf_getlocations: [ on  ]
...                         glibc: [ on  ]
...                          gtk2: [ on  ]
...                      libaudit: [ on  ]
...                        libbfd: [ on  ]
...                        libelf: [ on  ]
...                       libnuma: [ on  ]
...        numa_num_possible_cpus: [ on  ]
...                       libperl: [ on  ]
...                     libpython: [ on  ]
...                      libslang: [ on  ]
...                     libcrypto: [ on  ]
...                     libunwind: [ on  ]
...            libdw-dwarf-unwind: [ on  ]
...                          zlib: [ on  ]
...                          lzma: [ on  ]
...                     get_cpuid: [ on  ]
...                           bpf: [ on  ]
...                     backtrace: [ on  ]
...                fortify-source: [ on  ]
...         sync-compare-and-swap: [ on  ]
...                  gtk2-infobar: [ on  ]
...             libelf-getphdrnum: [ on  ]
...           libelf-gelf_getnote: [ on  ]
...          libelf-getshdrstrndx: [ on  ]
...                   libelf-mmap: [ on  ]
...             libpython-version: [ on  ]
...                 libunwind-x86: [ on  ]
...              libunwind-x86_64: [ on  ]
...                 libunwind-arm: [ on  ]
...             libunwind-aarch64: [ on  ]
...   pthread-attr-setaffinity-np: [ on  ]
...            stackprotector-all: [ on  ]
...                       timerfd: [ on  ]
...                           sdt: [ on  ]

<snip>

In file included from /home/yocto/milian/linux/tools/perf/util/include/../../
ui/keysyms.h:4:0,
                 from /home/yocto/milian/linux/tools/perf/util/include/../
hist.h:409,
                 from /home/yocto/milian/linux/tools/perf/util/include/../
sort.h:23,
                 from ui/gtk/browser.c:4:
/home/yocto/milian/linux/tools/perf/util/include/../../ui/libslang.h:12:19: 
fatal error: slang.h: No such file or directory
 #include <slang.h>

... and many more errors
~~~~~~~~~~~~~

Looking at the faulty FEATURE-DUMP file in tools/build, I noticed not only 
features being enabled that I have on my host but not on the target (which 
would be bad enough), but also the following, which can only mean it blindly 
enabled everything:

~~~~~~~~~~~~~~
feature-libunwind=1
feature-libunwind-x86=1
feature-libunwind-x86_64=1
feature-libunwind-arm=1
feature-libunwind-aarch64=1
~~~~~~~~~~~~~~

I work-arounded this for now by copying the faulty FEATURE-DUMP file from 
tools/build/ to some other location, then hand-editing it to disable the 
features that I know are not available for my platform (slang, sdt, numa, gtk, 
x86 libunwind, ...). Then, I pass FEATURES_DUMP=path/to/my/FEATURE-DUMP to 
make.

c) Picking up dependencies from custom prefix

I've build libelf and other dependencies manually into a custom prefix outside 
of my sysroot. To pick those up, it's not enough to pass `prefix=path/to/
prefix` to make. You need to manually add `-Ipath/to/prefix/include` to 
EXTRA_CFLAGS and `-Lpath/to/prefix/lib` to LDFLAGS...

d) My final make invocation looks like this now:

export CC=aarch64-gnu-linux-gcc
export CFLAGS="--sysroot=/home/yocto/SDK/E013/sdk/sysroots/aarch64-gnu-linux 
-O2 -pipe -g -feliminate-unused-debug-types -fno-omit-frame-pointer -
march=armv8-a -funwind-tables"

make \
    FEATURES_DUMP=/home/yocto/milian/perf-FEATURE-DUMP \
    CROSS_COMPILE="aarch64-gnu-linux-" \
    EXTRA_CFLAGS="$CFLAGS -I/home/yocto/milian/target-prefix/include" \
    LDFLAGS="-L/home/yocto/milian/target-prefix/lib $LDFLAGS" \
    prefix=/home/yocto/milian/target-prefix

The features dump contains the following:

feature-backtrace=1
feature-dwarf=1
feature-dwarf_getlocations=1
feature-fortify-source=1
feature-sync-compare-and-swap=1
feature-glibc=1
feature-gtk2=0
feature-gtk2-infobar=0
feature-libaudit=0
feature-libbfd=0
feature-libelf=1
feature-libelf-getphdrnum=1
feature-libelf-gelf_getnote=1
feature-libelf-getshdrstrndx=1
feature-libelf-mmap=1
feature-libnuma=0
feature-numa_num_possible_cpus=0
feature-libperl=0
feature-libpython=0
feature-libpython-version=0
feature-libslang=0
feature-libcrypto=1
feature-libunwind=1
feature-libunwind-x86=0
feature-libunwind-x86_64=0
feature-libunwind-arm=1
feature-libunwind-aarch64=1
feature-pthread-attr-setaffinity-np=1
feature-stackprotector-all=1
feature-timerfd=1
feature-libdw-dwarf-unwind=1
feature-zlib=1
feature-lzma=1
feature-get_cpuid=1
feature-bpf=0
feature-sdt=0

I hope this helps others. I finally have a working up2date perf on my aarch64 
platform and it seems to work form my simple 5min tests so far.

Cheers

-- 
Milian Wolff | milian.wolff@kdab.com | Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5903 bytes --]