Re: plumbers session on profiling?

Re: plumbers session on profiling?

[Jose E. Marchesi]

[Added linux-toolchains@vger in CC]

It would be interesting to have some discussion in the Toolchains track
on building the kernel with PGO/FDO.  I have seen a raise on interest on
the topic in several companies, but it would make very little sense if
no kernel hacker is interested in participating... anybody?

> Hi Nick,
>
> Apologies for the delay. It's been a busy week with gprofng related work.
>
>> If you're still considering attending Linux Plumbers conf, please
>> submit a proposal:
>> https://lpc.events/event/16/abstracts/
>> Please make sure to select "Toolchains Track" as the "Track" after
>> clicking on "Submit new abstract."
>
> Thanks for asking!
>
> Our presence largely depends on a discussion on kernel profiling and
> to see what we might be able to do with gprofng regarding this.
>
>>From our side, the main person will be Vladimir. He knows the code
> inside and out, but I will also try to attend such a session.
>
> Is there any news regarding such a session on kernel profiling?
>
> Kind regards, Ruud
>
>> 
>>> 
>>> Kind regards, Ruud
>>> 
>>>> On 24 May 2022, at 12:24, Jose E. Marchesi <jemarch@gnu.org> wrote:
>>>> 
>>>> 
>>>> I am adding Ruud van der Pas in CC. He works in gprofng and would be
>>>> willing to participate in a discussion on kernel profiling.
>>>> 
>>>>> (Re-sending with Vladamir's email addr fixed; sorry for the noise)
>>>>> 
>>>>> On Fri, Apr 15, 2022 at 10:54 AM Nick Desaulniers
>>>>> <ndesaulniers@google.com> wrote:
>>>>>> 
>>>>>> Hi Sami, Bill, Jose, and Vladamir,
>>>>>> Jose and I are currently in the planning process to put together a
>>>>>> Kernel+Toolchain microconference track at Linux Plumbers Conference
>>>>>> this year (Sept 12-14) in Dublin, Ireland.
>>>>>> 
>>>>>> Would you all be interested in leading a session discussing various
>>>>>> profiling related topics such as:
>>>>>> * gprofng
>>>>>> * PGO
>>>>>> * AutoFDO
>>>>>> 
>>>>>> Would others find such a discussion useful?
>>>>>> --
>>>>>> Thanks,
>>>>>> ~Nick Desaulniers
>>> 
>> 
>> 
>> -- 
>> Thanks,
>> ~Nick Desaulniers

Re: plumbers session on profiling?

[Bill Wendling]

I have experience in this area with our internal production kernel.

-bw

On Wed, Jun 29, 2022 at 2:00 AM Jose E. Marchesi <jemarch@gnu.org> wrote:
>
>
> [Added linux-toolchains@vger in CC]
>
> It would be interesting to have some discussion in the Toolchains track
> on building the kernel with PGO/FDO.  I have seen a raise on interest on
> the topic in several companies, but it would make very little sense if
> no kernel hacker is interested in participating... anybody?
>
> > Hi Nick,
> >
> > Apologies for the delay. It's been a busy week with gprofng related work.
> >
> >> If you're still considering attending Linux Plumbers conf, please
> >> submit a proposal:
> >> https://lpc.events/event/16/abstracts/
> >> Please make sure to select "Toolchains Track" as the "Track" after
> >> clicking on "Submit new abstract."
> >
> > Thanks for asking!
> >
> > Our presence largely depends on a discussion on kernel profiling and
> > to see what we might be able to do with gprofng regarding this.
> >
> >>From our side, the main person will be Vladimir. He knows the code
> > inside and out, but I will also try to attend such a session.
> >
> > Is there any news regarding such a session on kernel profiling?
> >
> > Kind regards, Ruud
> >
> >>
> >>>
> >>> Kind regards, Ruud
> >>>
> >>>> On 24 May 2022, at 12:24, Jose E. Marchesi <jemarch@gnu.org> wrote:
> >>>>
> >>>>
> >>>> I am adding Ruud van der Pas in CC. He works in gprofng and would be
> >>>> willing to participate in a discussion on kernel profiling.
> >>>>
> >>>>> (Re-sending with Vladamir's email addr fixed; sorry for the noise)
> >>>>>
> >>>>> On Fri, Apr 15, 2022 at 10:54 AM Nick Desaulniers
> >>>>> <ndesaulniers@google.com> wrote:
> >>>>>>
> >>>>>> Hi Sami, Bill, Jose, and Vladamir,
> >>>>>> Jose and I are currently in the planning process to put together a
> >>>>>> Kernel+Toolchain microconference track at Linux Plumbers Conference
> >>>>>> this year (Sept 12-14) in Dublin, Ireland.
> >>>>>>
> >>>>>> Would you all be interested in leading a session discussing various
> >>>>>> profiling related topics such as:
> >>>>>> * gprofng
> >>>>>> * PGO
> >>>>>> * AutoFDO
> >>>>>>
> >>>>>> Would others find such a discussion useful?
> >>>>>> --
> >>>>>> Thanks,
> >>>>>> ~Nick Desaulniers
> >>>
> >>
> >>
> >> --
> >> Thanks,
> >> ~Nick Desaulniers

Re: plumbers session on profiling?

[Peter Zijlstra]

On Tue, Jun 28, 2022 at 07:08:48PM +0200, Jose E. Marchesi wrote:
> 
> [Added linux-toolchains@vger in CC]
> 
> It would be interesting to have some discussion in the Toolchains track
> on building the kernel with PGO/FDO.  I have seen a raise on interest on
> the topic in several companies, but it would make very little sense if
> no kernel hacker is interested in participating... anybody?

I know there's been a lot of work in this area, but none of it seems to
have trickled down to be easy enough for me to use it.

Re: plumbers session on profiling?

[Bill Wendling]

On Fri, Jul 1, 2022 at 2:02 AM Peter Zijlstra <peterz@infradead.org> wrote:
>
> On Tue, Jun 28, 2022 at 07:08:48PM +0200, Jose E. Marchesi wrote:
> >
> > [Added linux-toolchains@vger in CC]
> >
> > It would be interesting to have some discussion in the Toolchains track
> > on building the kernel with PGO/FDO.  I have seen a raise on interest on
> > the topic in several companies, but it would make very little sense if
> > no kernel hacker is interested in participating... anybody?
>
> I know there's been a lot of work in this area, but none of it seems to
> have trickled down to be easy enough for me to use it.

We use an instrumented kernel to collect the data we need. It gives us
the best payoff, because the profiling data is more fine-grained and
accurate. (PGO does much more than make inlining decisions.)

If I recall correctly, you previously suggested using sampling data.
(Correct?) Is there a document or article that outlines that process?

-bw

Re: plumbers session on profiling?

[Peter Zijlstra]

On Fri, Jul 01, 2022 at 03:17:54AM -0700, Bill Wendling wrote:
> On Fri, Jul 1, 2022 at 2:02 AM Peter Zijlstra <peterz@infradead.org> wrote:
> >
> > On Tue, Jun 28, 2022 at 07:08:48PM +0200, Jose E. Marchesi wrote:
> > >
> > > [Added linux-toolchains@vger in CC]
> > >
> > > It would be interesting to have some discussion in the Toolchains track
> > > on building the kernel with PGO/FDO.  I have seen a raise on interest on
> > > the topic in several companies, but it would make very little sense if
> > > no kernel hacker is interested in participating... anybody?
> >
> > I know there's been a lot of work in this area, but none of it seems to
> > have trickled down to be easy enough for me to use it.
> 
> We use an instrumented kernel to collect the data we need. It gives us
> the best payoff, because the profiling data is more fine-grained and
> accurate. (PGO does much more than make inlining decisions.)
> 
> If I recall correctly, you previously suggested using sampling data.
> (Correct?) Is there a document or article that outlines that process?

IIRC Google has LBR sample driven PGO somewhere as well. ISTR that being
the whole motivation for that gruesome Zen3 BRS hack.

Google got me this: https://research.google.com/pubs/archive/45290.pdf

Re: plumbers session on profiling?

[Paul E. McKenney]

On Fri, Jul 01, 2022 at 01:49:06PM +0200, Peter Zijlstra wrote:
> On Fri, Jul 01, 2022 at 03:17:54AM -0700, Bill Wendling wrote:
> > On Fri, Jul 1, 2022 at 2:02 AM Peter Zijlstra <peterz@infradead.org> wrote:
> > >
> > > On Tue, Jun 28, 2022 at 07:08:48PM +0200, Jose E. Marchesi wrote:
> > > >
> > > > [Added linux-toolchains@vger in CC]
> > > >
> > > > It would be interesting to have some discussion in the Toolchains track
> > > > on building the kernel with PGO/FDO.  I have seen a raise on interest on
> > > > the topic in several companies, but it would make very little sense if
> > > > no kernel hacker is interested in participating... anybody?
> > >
> > > I know there's been a lot of work in this area, but none of it seems to
> > > have trickled down to be easy enough for me to use it.
> > 
> > We use an instrumented kernel to collect the data we need. It gives us
> > the best payoff, because the profiling data is more fine-grained and
> > accurate. (PGO does much more than make inlining decisions.)
> > 
> > If I recall correctly, you previously suggested using sampling data.
> > (Correct?) Is there a document or article that outlines that process?
> 
> IIRC Google has LBR sample driven PGO somewhere as well. ISTR that being
> the whole motivation for that gruesome Zen3 BRS hack.
> 
> Google got me this: https://research.google.com/pubs/archive/45290.pdf

Whatever else, please refrain from using PGO and friends to drive
data-value speculation!

							Thanx, Paul

Re: plumbers session on profiling?

[Paul E. McKenney]

On Fri, Jul 01, 2022 at 03:32:12PM +0000, Hongtao Yu wrote:
> 
> 
> ________________________________
> From: Paul E. McKenney <paulmck@kernel.org>
> Sent: Friday, July 1, 2022 6:50 AM
> To: Peter Zijlstra <peterz@infradead.org>
> Cc: Bill Wendling <morbo@google.com>; Jose E. Marchesi <jemarch@gnu.org>; Ruud van der Pas <ruud.vanderpas@oracle.com>; Nick Desaulniers <ndesaulniers@google.com>; Sami Tolvanen <samitolvanen@google.com>; Vladimir Mezentsev <vladimir.mezentsev@oracle.com>; clang-built-linux <llvm@lists.linux.dev>; LKML <linux-kernel@vger.kernel.org>; Yonghong Song <yhs@fb.com>; Wenlei He <wenlei@fb.com>; Hongtao Yu <hoy@fb.com>; Ingo Molnar <mingo@kernel.org>; linux-toolchains <linux-toolchains@vger.kernel.org>; elena.zannoni@oracle.com <elena.zannoni@oracle.com>
> Subject: Re: plumbers session on profiling?
> 
> On Fri, Jul 01, 2022 at 01:49:06PM +0200, Peter Zijlstra wrote:
> > On Fri, Jul 01, 2022 at 03:17:54AM -0700, Bill Wendling wrote:
> > > On Fri, Jul 1, 2022 at 2:02 AM Peter Zijlstra <peterz@infradead.org> wrote:
> > > >
> > > > On Tue, Jun 28, 2022 at 07:08:48PM +0200, Jose E. Marchesi wrote:
> > > > >
> > > > > [Added linux-toolchains@vger in CC]
> > > > >
> > > > > It would be interesting to have some discussion in the Toolchains track
> > > > > on building the kernel with PGO/FDO.  I have seen a raise on interest on
> > > > > the topic in several companies, but it would make very little sense if
> > > > > no kernel hacker is interested in participating... anybody?
> > > >
> > > > I know there's been a lot of work in this area, but none of it seems to
> > > > have trickled down to be easy enough for me to use it.
> > >
> > > We use an instrumented kernel to collect the data we need. It gives us
> > > the best payoff, because the profiling data is more fine-grained and
> > > accurate. (PGO does much more than make inlining decisions.)
> > >
> > > If I recall correctly, you previously suggested using sampling data.
> > > (Correct?) Is there a document or article that outlines that process?
> >
> > IIRC Google has LBR sample driven PGO somewhere as well. ISTR that being
> > the whole motivation for that gruesome Zen3 BRS hack.
> >
> > Google got me this: https://research.google.com/pubs/archive/45290.pdf
> 
> > Whatever else, please refrain from using PGO and friends to drive
> data-value speculation!
> 
>  Is there a particular reason for this?

Yes, there is.

Doing so breaks many concurrent algorithms.

Sure, the hardware can get away with this sort of thing.  But that is
only because it has access to cache and store-buffer state that allows it
to roll back and replay when needed.  The compiler does not have access
to this state [1], so it needs to avoid playing this game.

							Thanx, Paul

Re: plumbers session on profiling?

[Bill Wendling]

On Fri, Jul 1, 2022 at 4:49 AM Peter Zijlstra <peterz@infradead.org> wrote:
> On Fri, Jul 01, 2022 at 03:17:54AM -0700, Bill Wendling wrote:
> > On Fri, Jul 1, 2022 at 2:02 AM Peter Zijlstra <peterz@infradead.org> wrote:
> > >
> > > On Tue, Jun 28, 2022 at 07:08:48PM +0200, Jose E. Marchesi wrote:
> > > >
> > > > [Added linux-toolchains@vger in CC]
> > > >
> > > > It would be interesting to have some discussion in the Toolchains track
> > > > on building the kernel with PGO/FDO.  I have seen a raise on interest on
> > > > the topic in several companies, but it would make very little sense if
> > > > no kernel hacker is interested in participating... anybody?
> > >
> > > I know there's been a lot of work in this area, but none of it seems to
> > > have trickled down to be easy enough for me to use it.
> >
> > We use an instrumented kernel to collect the data we need. It gives us
> > the best payoff, because the profiling data is more fine-grained and
> > accurate. (PGO does much more than make inlining decisions.)
> >
> > If I recall correctly, you previously suggested using sampling data.
> > (Correct?) Is there a document or article that outlines that process?
>
> IIRC Google has LBR sample driven PGO somewhere as well. ISTR that being
> the whole motivation for that gruesome Zen3 BRS hack.
>
> Google got me this: https://research.google.com/pubs/archive/45290.pdf
>
Right. However, there's a chicken-and-egg issue with AutoFDO for the
production kernel. We can't release a kernel that hasn't been compiled
with PGO/FDO. We could only release it in a test environment, in which
case we could use AutoFDO. However, the document says that AutoFDO
only reaches ~90% of FDO. They list some reasons for this, but
nonetheless I suspect that the delta would be too severe for us to
release the kernel.

As for LBR, that will work with Intel/AMD, but I thought that LBR
doesn't exist for Arm processors (my knowledge could be out of date on
this).

What would make PGO (sample-based or instrumented) easy enough for you
to use? What're the key elements missing?

-bw

Re: plumbers session on profiling?

[Fangrui Song]

On 2022-07-01, Bill Wendling wrote:
>On Fri, Jul 1, 2022 at 4:49 AM Peter Zijlstra <peterz@infradead.org> wrote:
>> On Fri, Jul 01, 2022 at 03:17:54AM -0700, Bill Wendling wrote:
>> > On Fri, Jul 1, 2022 at 2:02 AM Peter Zijlstra <peterz@infradead.org> wrote:
>> > >
>> > > On Tue, Jun 28, 2022 at 07:08:48PM +0200, Jose E. Marchesi wrote:
>> > > >
>> > > > [Added linux-toolchains@vger in CC]
>> > > >
>> > > > It would be interesting to have some discussion in the Toolchains track
>> > > > on building the kernel with PGO/FDO.  I have seen a raise on interest on
>> > > > the topic in several companies, but it would make very little sense if
>> > > > no kernel hacker is interested in participating... anybody?
>> > >
>> > > I know there's been a lot of work in this area, but none of it seems to
>> > > have trickled down to be easy enough for me to use it.
>> >
>> > We use an instrumented kernel to collect the data we need. It gives us
>> > the best payoff, because the profiling data is more fine-grained and
>> > accurate. (PGO does much more than make inlining decisions.)
>> >
>> > If I recall correctly, you previously suggested using sampling data.
>> > (Correct?) Is there a document or article that outlines that process?
>>
>> IIRC Google has LBR sample driven PGO somewhere as well. ISTR that being
>> the whole motivation for that gruesome Zen3 BRS hack.
>>
>> Google got me this: https://research.google.com/pubs/archive/45290.pdf
>>

I very support that the mainline kernel adds instrumentation based PGO
but I vaguely recall that it was NAKed by Linus (because he thought
sample based is better).

>Right. However, there's a chicken-and-egg issue with AutoFDO for the
>production kernel. We can't release a kernel that hasn't been compiled
>with PGO/FDO. We could only release it in a test environment, in which
>case we could use AutoFDO. However, the document says that AutoFDO
>only reaches ~90% of FDO. They list some reasons for this, but
>nonetheless I suspect that the delta would be too severe for us to
>release the kernel.
>
>As for LBR, that will work with Intel/AMD, but I thought that LBR
>doesn't exist for Arm processors (my knowledge could be out of date on
>this).

Some folks try using Embedded Trace Macrocells.
I am not at all familiar with it but it seems that retrieving profiles
is not easy. The needed efforts seem even higher than using
instrumentation based PGO.

Instrumentation based PGO has the nice property that it works with all
architectures (the compiler supports) and does not rely on hardware
support. In addition, it collects indirect call targets and string
operation sizes which are very difficult/impossible for sample based
PGO.

>What would make PGO (sample-based or instrumented) easy enough for you
>to use? What're the key elements missing?
>
>-bw
>

Re: plumbers session on profiling?

[Peter Zijlstra]

On Fri, Jul 01, 2022 at 12:34:09PM -0700, Fangrui Song wrote:

> Instrumentation based PGO has the nice property that it works with all
> architectures (the compiler supports) and does not rely on hardware
> support. In addition, it collects indirect call targets and string
> operation sizes which are very difficult/impossible for sample based
> PGO.

LBR samples should provide the indirect call targets just fine.

Re: plumbers session on profiling?

[Peter Zijlstra]

On Fri, Jul 01, 2022 at 11:57:25AM -0700, Bill Wendling wrote:
> On Fri, Jul 1, 2022 at 4:49 AM Peter Zijlstra <peterz@infradead.org> wrote:

> > IIRC Google has LBR sample driven PGO somewhere as well. ISTR that being
> > the whole motivation for that gruesome Zen3 BRS hack.
> >
> > Google got me this: https://research.google.com/pubs/archive/45290.pdf
> >
> Right. However, there's a chicken-and-egg issue with AutoFDO for the
> production kernel. We can't release a kernel that hasn't been compiled
> with PGO/FDO. We could only release it in a test environment, in which
> case we could use AutoFDO. However, the document says that AutoFDO
> only reaches ~90% of FDO. They list some reasons for this, but
> nonetheless I suspect that the delta would be too severe for us to
> release the kernel.

The pertinent question seems to be what's missing? Where does that 10%
go.

> As for LBR, that will work with Intel/AMD, but I thought that LBR
> doesn't exist for Arm processors (my knowledge could be out of date on
> this).

Not totally up to date on the ARM thing either; but I believe you're
right in that they don't yet have such a feature.

> What would make PGO (sample-based or instrumented) easy enough for you
> to use? What're the key elements missing?

The key piece missing is how to feed a perf.data file back into the
compile cycle, something like:

  $ make O=build/ PERF=perf.data -j$lots

would be useful I suppose.

Re: plumbers session on profiling?

[Peter Zijlstra]

On Mon, Jul 04, 2022 at 04:01:15PM +0200, Peter Zijlstra wrote:
> On Fri, Jul 01, 2022 at 11:57:25AM -0700, Bill Wendling wrote:

> > As for LBR, that will work with Intel/AMD, but I thought that LBR
> > doesn't exist for Arm processors (my knowledge could be out of date on
> > this).
> 
> Not totally up to date on the ARM thing either; but I believe you're
> right in that they don't yet have such a feature.

Just found this in my inbox; I should've seen it before, but it must've
slipped my mind:

  https://lkml.kernel.org/r/20220613100119.684673-1-anshuman.khandual@arm.com

And of course, PowerPC also supports this.