perf record is crashing on system with bpf progs are running

perf record is crashing on system with bpf progs are running

[RAJESH DASARI]

Hi ,

perf record is crashing on systems with bpf programs are running.

We are running on the 4.18 host kernel and we launch a container ,
inside the container we run perf record command, (bpf program is
running). Perf is built on top of the 5.4.228 kernel.

(gdb) bt
#0  0x000000000055becf in perf_event__synthesize_one_bpf_prog
(opts=0x94def8 <record+280>, event=0xabd910, fd=132, machine=0xaf6208,
process=0x421260 <process_synthesized_event>,
    session=0xaf6090) at util/bpf-event.c:242
#1  perf_event__synthesize_bpf_events (session=session@entry=0xaf6090,
process=process@entry=0x421260 <process_synthesized_event>,
machine=machine@entry=0xaf6208,
    opts=opts@entry=0x94def8 <record+280>) at util/bpf-event.c:333
#2  0x000000000041ff5c in record__synthesize (tail=tail@entry=false,
rec=0x94dde0 <record>) at builtin-record.c:1323
#3  0x0000000000422bc4 in __cmd_record (rec=0x94dde0 <record>,
argv=<optimized out>, argc=<optimized out>) at builtin-record.c:1465
#4  cmd_record (argc=<optimized out>, argv=<optimized out>) at
builtin-record.c:2474
#5  0x0000000000490769 in run_builtin (p=p@entry=0x9574b8
<commands+216>, argc=argc@entry=1, argv=argv@entry=0x7fffffffc120) at
perf.c:312
#6  0x000000000040c05a in handle_internal_command
(argv=0x7fffffffc120, argc=1) at perf.c:364
#7  run_argv (argv=<synthetic pointer>, argcp=<synthetic pointer>) at perf.c:408
#8  main (argc=1, argv=0x7fffffffc120) at perf.c:538

We tried running perf record --no-bpf-event but crash is still seen so
we added the below patch to skip processing bpf events when
no-bpf-event options is provided , but not sure if this is the actual
solution, please provide your inputs.

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 454e275..78e96ad 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -1320,11 +1320,12 @@ static int record__synthesize(struct record
*rec, bool tail)
                return err;
        }

-       err = perf_event__synthesize_bpf_events(session,
process_synthesized_event,
-                                               machine, opts);
-       if (err < 0)
-               pr_warning("Couldn't synthesize bpf events.\n");
-
+        if (!opts->no_bpf_event) {
+                err = perf_event__synthesize_bpf_events(session,
process_synthesized_event,
+                                                       machine, opts);
+               if (err < 0)
+                       pr_warning("Couldn't synthesize bpf events.\n");
+        }


Thanks,
Rajesh Dasari.

Re: perf record is crashing on system with bpf progs are running

[Ian Rogers]

On Thu, Feb 23, 2023 at 12:02 AM RAJESH DASARI <raajeshdasari@gmail.com> wrote:
>
> Hi ,
>
> perf record is crashing on systems with bpf programs are running.
>
> We are running on the 4.18 host kernel and we launch a container ,
> inside the container we run perf record command, (bpf program is
> running). Perf is built on top of the 5.4.228 kernel.

You mean you are running perf built from Linux 5.4? Could you try
something more recent like Linux 6.2?

Thanks,
Ian

> (gdb) bt
> #0  0x000000000055becf in perf_event__synthesize_one_bpf_prog
> (opts=0x94def8 <record+280>, event=0xabd910, fd=132, machine=0xaf6208,
> process=0x421260 <process_synthesized_event>,
>     session=0xaf6090) at util/bpf-event.c:242
> #1  perf_event__synthesize_bpf_events (session=session@entry=0xaf6090,
> process=process@entry=0x421260 <process_synthesized_event>,
> machine=machine@entry=0xaf6208,
>     opts=opts@entry=0x94def8 <record+280>) at util/bpf-event.c:333
> #2  0x000000000041ff5c in record__synthesize (tail=tail@entry=false,
> rec=0x94dde0 <record>) at builtin-record.c:1323
> #3  0x0000000000422bc4 in __cmd_record (rec=0x94dde0 <record>,
> argv=<optimized out>, argc=<optimized out>) at builtin-record.c:1465
> #4  cmd_record (argc=<optimized out>, argv=<optimized out>) at
> builtin-record.c:2474
> #5  0x0000000000490769 in run_builtin (p=p@entry=0x9574b8
> <commands+216>, argc=argc@entry=1, argv=argv@entry=0x7fffffffc120) at
> perf.c:312
> #6  0x000000000040c05a in handle_internal_command
> (argv=0x7fffffffc120, argc=1) at perf.c:364
> #7  run_argv (argv=<synthetic pointer>, argcp=<synthetic pointer>) at perf.c:408
> #8  main (argc=1, argv=0x7fffffffc120) at perf.c:538
>
> We tried running perf record --no-bpf-event but crash is still seen so
> we added the below patch to skip processing bpf events when
> no-bpf-event options is provided , but not sure if this is the actual
> solution, please provide your inputs.
>
> diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
> index 454e275..78e96ad 100644
> --- a/tools/perf/builtin-record.c
> +++ b/tools/perf/builtin-record.c
> @@ -1320,11 +1320,12 @@ static int record__synthesize(struct record
> *rec, bool tail)
>                 return err;
>         }
>
> -       err = perf_event__synthesize_bpf_events(session,
> process_synthesized_event,
> -                                               machine, opts);
> -       if (err < 0)
> -               pr_warning("Couldn't synthesize bpf events.\n");
> -
> +        if (!opts->no_bpf_event) {
> +                err = perf_event__synthesize_bpf_events(session,
> process_synthesized_event,
> +                                                       machine, opts);
> +               if (err < 0)
> +                       pr_warning("Couldn't synthesize bpf events.\n");
> +        }
>
>
> Thanks,
> Rajesh Dasari.

Re: perf record is crashing on system with bpf progs are running

[RAJESH DASARI]

Hi ,

I tried with perf build with Linux 6.2
I get the below warning , now the crash is not seen but i see the
Couldn't synthesize bpf events message on the output, I guess crash is
avoided by this commit
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/tools/perf/util/bpf-event.c?h=v6.2.2&id=1a096ae46e21b73f83a581e617f76326c1de592d

perf record
Couldn't synthesize bpf events.

perf record --no-bpf-event
Couldn't synthesize bpf events.

But when I pass --no-bpf-events option also I see this  Couldn't
synthesize bpf events message , I feel the warning message is
misleading as i am explicitly specifying no-bpf-event option , Please
provide your inputs.

Thanks,
Rajesh Dasari.

On Fri, Feb 24, 2023 at 5:45 AM Ian Rogers <irogers@google.com> wrote:
>
> On Thu, Feb 23, 2023 at 12:02 AM RAJESH DASARI <raajeshdasari@gmail.com> wrote:
> >
> > Hi ,
> >
> > perf record is crashing on systems with bpf programs are running.
> >
> > We are running on the 4.18 host kernel and we launch a container ,
> > inside the container we run perf record command, (bpf program is
> > running). Perf is built on top of the 5.4.228 kernel.
>
> You mean you are running perf built from Linux 5.4? Could you try
> something more recent like Linux 6.2?
>
> Thanks,
> Ian
>
> > (gdb) bt
> > #0  0x000000000055becf in perf_event__synthesize_one_bpf_prog
> > (opts=0x94def8 <record+280>, event=0xabd910, fd=132, machine=0xaf6208,
> > process=0x421260 <process_synthesized_event>,
> >     session=0xaf6090) at util/bpf-event.c:242
> > #1  perf_event__synthesize_bpf_events (session=session@entry=0xaf6090,
> > process=process@entry=0x421260 <process_synthesized_event>,
> > machine=machine@entry=0xaf6208,
> >     opts=opts@entry=0x94def8 <record+280>) at util/bpf-event.c:333
> > #2  0x000000000041ff5c in record__synthesize (tail=tail@entry=false,
> > rec=0x94dde0 <record>) at builtin-record.c:1323
> > #3  0x0000000000422bc4 in __cmd_record (rec=0x94dde0 <record>,
> > argv=<optimized out>, argc=<optimized out>) at builtin-record.c:1465
> > #4  cmd_record (argc=<optimized out>, argv=<optimized out>) at
> > builtin-record.c:2474
> > #5  0x0000000000490769 in run_builtin (p=p@entry=0x9574b8
> > <commands+216>, argc=argc@entry=1, argv=argv@entry=0x7fffffffc120) at
> > perf.c:312
> > #6  0x000000000040c05a in handle_internal_command
> > (argv=0x7fffffffc120, argc=1) at perf.c:364
> > #7  run_argv (argv=<synthetic pointer>, argcp=<synthetic pointer>) at perf.c:408
> > #8  main (argc=1, argv=0x7fffffffc120) at perf.c:538
> >
> > We tried running perf record --no-bpf-event but crash is still seen so
> > we added the below patch to skip processing bpf events when
> > no-bpf-event options is provided , but not sure if this is the actual
> > solution, please provide your inputs.
> >
> > diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
> > index 454e275..78e96ad 100644
> > --- a/tools/perf/builtin-record.c
> > +++ b/tools/perf/builtin-record.c
> > @@ -1320,11 +1320,12 @@ static int record__synthesize(struct record
> > *rec, bool tail)
> >                 return err;
> >         }
> >
> > -       err = perf_event__synthesize_bpf_events(session,
> > process_synthesized_event,
> > -                                               machine, opts);
> > -       if (err < 0)
> > -               pr_warning("Couldn't synthesize bpf events.\n");
> > -
> > +        if (!opts->no_bpf_event) {
> > +                err = perf_event__synthesize_bpf_events(session,
> > process_synthesized_event,
> > +                                                       machine, opts);
> > +               if (err < 0)
> > +                       pr_warning("Couldn't synthesize bpf events.\n");
> > +        }
> >
> >
> > Thanks,
> > Rajesh Dasari.

Re: perf record is crashing on system with bpf progs are running

[Ian Rogers]

On Mon, Mar 6, 2023 at 7:26 AM RAJESH DASARI <raajeshdasari@gmail.com> wrote:
>
> Hi ,
>
> I tried with perf build with Linux 6.2
> I get the below warning , now the crash is not seen but i see the
> Couldn't synthesize bpf events message on the output, I guess crash is
> avoided by this commit
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/tools/perf/util/bpf-event.c?h=v6.2.2&id=1a096ae46e21b73f83a581e617f76326c1de592d
>
> perf record
> Couldn't synthesize bpf events.
>
> perf record --no-bpf-event
> Couldn't synthesize bpf events.
>
> But when I pass --no-bpf-events option also I see this  Couldn't
> synthesize bpf events message , I feel the warning message is
> misleading as i am explicitly specifying no-bpf-event option , Please
> provide your inputs.

It sounds like kernel issues. You can add the -v verbose flag to see
what's happening but there is insufficient detail here to say what's
happening.

Thanks,
Ian

> Thanks,
> Rajesh Dasari.
>
> On Fri, Feb 24, 2023 at 5:45 AM Ian Rogers <irogers@google.com> wrote:
> >
> > On Thu, Feb 23, 2023 at 12:02 AM RAJESH DASARI <raajeshdasari@gmail.com> wrote:
> > >
> > > Hi ,
> > >
> > > perf record is crashing on systems with bpf programs are running.
> > >
> > > We are running on the 4.18 host kernel and we launch a container ,
> > > inside the container we run perf record command, (bpf program is
> > > running). Perf is built on top of the 5.4.228 kernel.
> >
> > You mean you are running perf built from Linux 5.4? Could you try
> > something more recent like Linux 6.2?
> >
> > Thanks,
> > Ian
> >
> > > (gdb) bt
> > > #0  0x000000000055becf in perf_event__synthesize_one_bpf_prog
> > > (opts=0x94def8 <record+280>, event=0xabd910, fd=132, machine=0xaf6208,
> > > process=0x421260 <process_synthesized_event>,
> > >     session=0xaf6090) at util/bpf-event.c:242
> > > #1  perf_event__synthesize_bpf_events (session=session@entry=0xaf6090,
> > > process=process@entry=0x421260 <process_synthesized_event>,
> > > machine=machine@entry=0xaf6208,
> > >     opts=opts@entry=0x94def8 <record+280>) at util/bpf-event.c:333
> > > #2  0x000000000041ff5c in record__synthesize (tail=tail@entry=false,
> > > rec=0x94dde0 <record>) at builtin-record.c:1323
> > > #3  0x0000000000422bc4 in __cmd_record (rec=0x94dde0 <record>,
> > > argv=<optimized out>, argc=<optimized out>) at builtin-record.c:1465
> > > #4  cmd_record (argc=<optimized out>, argv=<optimized out>) at
> > > builtin-record.c:2474
> > > #5  0x0000000000490769 in run_builtin (p=p@entry=0x9574b8
> > > <commands+216>, argc=argc@entry=1, argv=argv@entry=0x7fffffffc120) at
> > > perf.c:312
> > > #6  0x000000000040c05a in handle_internal_command
> > > (argv=0x7fffffffc120, argc=1) at perf.c:364
> > > #7  run_argv (argv=<synthetic pointer>, argcp=<synthetic pointer>) at perf.c:408
> > > #8  main (argc=1, argv=0x7fffffffc120) at perf.c:538
> > >
> > > We tried running perf record --no-bpf-event but crash is still seen so
> > > we added the below patch to skip processing bpf events when
> > > no-bpf-event options is provided , but not sure if this is the actual
> > > solution, please provide your inputs.
> > >
> > > diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
> > > index 454e275..78e96ad 100644
> > > --- a/tools/perf/builtin-record.c
> > > +++ b/tools/perf/builtin-record.c
> > > @@ -1320,11 +1320,12 @@ static int record__synthesize(struct record
> > > *rec, bool tail)
> > >                 return err;
> > >         }
> > >
> > > -       err = perf_event__synthesize_bpf_events(session,
> > > process_synthesized_event,
> > > -                                               machine, opts);
> > > -       if (err < 0)
> > > -               pr_warning("Couldn't synthesize bpf events.\n");
> > > -
> > > +        if (!opts->no_bpf_event) {
> > > +                err = perf_event__synthesize_bpf_events(session,
> > > process_synthesized_event,
> > > +                                                       machine, opts);
> > > +               if (err < 0)
> > > +                       pr_warning("Couldn't synthesize bpf events.\n");
> > > +        }
> > >
> > >
> > > Thanks,
> > > Rajesh Dasari.