Re: [PATCH v8 01/13] Documentation: x86: Add AMD Hardware Feedback Interface documentation

["Ilpo Järvinen" <ilpo.jarvinen@linux.intel.com>]

[-- Attachment #1: Type: text/plain, Size: 11582 bytes --]

On Thu, 20 Mar 2025, Mario Limonciello wrote:

> On 3/19/2025 09:01, Ilpo Järvinen wrote:
> > On Tue, 18 Feb 2025, Mario Limonciello wrote:
> > 
> > > From: Perry Yuan <Perry.Yuan@amd.com>
> > > 
> > > Introduce a new documentation file, `amd_hfi.rst`, which delves into the
> > > implementation details of the AMD Hardware Feedback Interface and its
> > > associated driver, `amd_hfi`. This documentation describes how the
> > > driver provides hint to the OS scheduling which depends on the capability
> > > of core performance and efficiency ranking data.
> > > 
> > > This documentation describes
> > > * The design of the driver
> > > * How the driver provides hints to the OS scheduling
> > > * How the driver interfaces with the kernel for efficiency ranking data.
> > > 
> > > Reviewed-by: Bagas Sanjaya <bagasdotme@gmail.com>
> > > Signed-off-by: Perry Yuan <Perry.Yuan@amd.com>
> > > Reviewed-by: Mario Limonciello <mario.limonciello@amd.com>
> > > Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
> > > ---
> > >   Documentation/arch/x86/amd-hfi.rst | 127 +++++++++++++++++++++++++++++
> > >   Documentation/arch/x86/index.rst   |   1 +
> > >   2 files changed, 128 insertions(+)
> > >   create mode 100644 Documentation/arch/x86/amd-hfi.rst
> > > 
> > > diff --git a/Documentation/arch/x86/amd-hfi.rst
> > > b/Documentation/arch/x86/amd-hfi.rst
> > > new file mode 100644
> > > index 0000000000000..5d204688470e3
> > > --- /dev/null
> > > +++ b/Documentation/arch/x86/amd-hfi.rst
> > > @@ -0,0 +1,127 @@
> > > +.. SPDX-License-Identifier: GPL-2.0
> > > +
> > > +======================================================================
> > > +Hardware Feedback Interface For Hetero Core Scheduling On AMD Platform
> > > +======================================================================
> > > +
> > > +:Copyright: 2024 Advanced Micro Devices, Inc. All Rights Reserved.
> > > +
> > > +:Author: Perry Yuan <perry.yuan@amd.com>
> > > +:Author: Mario Limonciello <mario.limonciello@amd.com>
> > > +
> > > +Overview
> > > +--------
> > > +
> > > +AMD Heterogeneous Core implementations are comprised of more than one
> > > +architectural class and CPUs are comprised of cores of various efficiency
> > > and
> > > +power capabilities: performance-oriented *classic cores* and
> > > power-efficient
> > > +*dense cores*. As such, power management strategies must be designed to
> > > +accommodate the complexities introduced by incorporating different core
> > > types.
> > > +Heterogeneous systems can also extend to more than two architectural
> > > classes as
> > > +well. The purpose of the scheduling feedback mechanism is to provide
> > > +information to the operating system scheduler in real time such that the
> > > +scheduler can direct threads to the optimal core.
> > > +
> > > +The goal of AMD's heterogeneous architecture is to attain power benefit
> > > by sending
> > > +background thread to the dense cores while sending high priority threads
> > > to the classic
> > > +cores. From a performance perspective, sending background threads to
> > > dense cores can free
> > > +up power headroom and allow the classic cores to optimally service
> > > demanding threads.
> > > +Furthermore, the area optimized nature of the dense cores allows for an
> > > increasing
> > > +number of physical cores. This improved core density will have positive
> > > multithreaded
> > > +performance impact.
> > 
> > Hi Mario,
> > 
> > Please fold these paragraphs to 80 characters so that they're easier to
> > read as textfiles (the table can obviously exceed that but there should be
> > no reason for the text paragraphs to have excessively long lines).
> > 
> > My apologies for taking so long to get to review this series. 
> 
> No problem.  Thanks for looking.  I'll get a new version ready to put out
> after the next merge window.
> 
> > Most of my
> > comments are quite minor but there's also 1-2 things that seem more
> > important. It seemed to me that there is some disconnetion between the
> > promises made in the Kconfig description and what is provided by the patch
> > series.
> 
> Some of the series was pared down to go in multiple parts to make it easier to
> review with follow ups for the dynamic stuff planned for the next iteration.
> 
> You see some artifacts of that comments and Kconfig.  I figured it was better
> to leave as is for those given they get to the intent, but I can change if you
> think it's better to adjust them when the next part lands instead.

Okay, I thought that might be because such a split to multiple series. I 
think you can leave those as is as I assume to intention is to immediately 
follow up with the other parts (and not like wait a few kernel releases 
or so)?

-- 
 i.

> 
> > 
> > --
> >   i.
> > 
> > > +
> > > +AMD Heterogeneous Core Driver
> > > +-----------------------------
> > > +
> > > +The ``amd_hfi`` driver delivers the operating system a performance and
> > > energy efficiency
> > > +capability data for each CPU in the system. The scheduler can use the
> > > ranking data
> > > +from the HFI driver to make task placement decisions.
> > > +
> > > +Thread Classification and Ranking Table Interaction
> > > +----------------------------------------------------
> > > +
> > > +The thread classification is used to select into a ranking table that
> > > describes
> > > +an efficiency and performance ranking for each classification.
> > > +
> > > +Threads are classified during runtime into enumerated classes. The
> > > classes represent
> > > +thread performance/power characteristics that may benefit from special
> > > scheduling behaviors.
> > > +The below table depicts an example of thread classification and a
> > > preference where a given thread
> > > +should be scheduled based on its thread class. The real time thread
> > > classification is consumed
> > > +by the operating system and is used to inform the scheduler of where the
> > > thread should be placed.
> > > +
> > > +Thread Classification Example Table
> > > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > ++----------+----------------+-------------------------------+---------------------+---------+
> > > +| class ID | Classification | Preferred scheduling behavior | Preemption
> > > priority | Counter |
> > > ++----------+----------------+-------------------------------+---------------------+---------+
> > > +| 0        | Default        | Performant                    | Highest
> > > |         |
> > > ++----------+----------------+-------------------------------+---------------------+---------+
> > > +| 1        | Non-scalable   | Efficient                     | Lowest
> > > | PMCx1A1 |
> > > ++----------+----------------+-------------------------------+---------------------+---------+
> > > +| 2        | I/O bound      | Efficient                     | Lowest
> > > | PMCx044 |
> > > ++----------+----------------+-------------------------------+---------------------+---------+
> > > +
> > > +Thread classification is performed by the hardware each time that the
> > > thread is switched out.
> > > +Threads that don't meet any hardware specified criteria will be
> > > classified as "default".
> > > +
> > > +AMD Hardware Feedback Interface
> > > +--------------------------------
> > > +
> > > +The Hardware Feedback Interface provides to the operating system
> > > information
> > > +about the performance and energy efficiency of each CPU in the system.
> > > Each
> > > +capability is given as a unit-less quantity in the range [0-255]. A
> > > higher
> > > +performance value indicates higher performance capability, and a higher
> > > +efficiency value indicates more efficiency. Energy efficiency and
> > > performance
> > > +are reported in separate capabilities in the shared memory based ranking
> > > table.
> > > +
> > > +These capabilities may change at runtime as a result of changes in the
> > > +operating conditions of the system or the action of external factors.
> > > +Power Management FW is responsible for detecting events that would
> > > require
> > > +a reordering of the performance and efficiency ranking. Table updates
> > > would
> > > +happen relatively infrequently and occur on the time scale of seconds or
> > > more.
> > > +
> > > +The following events trigger a table update:
> > > +    * Thermal Stress Events
> > > +    * Silent Compute
> > > +    * Extreme Low Battery Scenarios
> > > +
> > > +The kernel or a userspace policy daemon can use these capabilities to
> > > modify
> > > +task placement decisions. For instance, if either the performance or
> > > energy
> > > +capabilities of a given logical processor becomes zero, it is an
> > > indication that
> > > +the hardware recommends to the operating system to not schedule any tasks
> > > on
> > > +that processor for performance or energy efficiency reasons,
> > > respectively.
> > > +
> > > +Implementation details for Linux
> > > +--------------------------------
> > > +
> > > +The implementation of threads scheduling consists of the following steps:
> > > +
> > > +1. A thread is spawned and scheduled to the ideal core using the default
> > > +   heterogeneous scheduling policy.
> > > +2. The processor profiles thread execution and assigns an enumerated
> > > classification ID.
> > > +   This classification is communicated to the OS via logical processor
> > > scope MSR.
> > > +3. During the thread context switch out the operating system consumes the
> > > workload(WL)
> > > +   classification which resides in a logical processor scope MSR.
> > > +4. The OS triggers the hardware to clear its history by writing to an
> > > MSR,
> > > +   after consuming the WL classification and before switching in the new
> > > thread.
> > > +5. If due to the classification, ranking table, and processor
> > > availability,
> > > +   the thread is not on its ideal processor, the OS will then consider
> > > scheduling
> > > +   the thread on its ideal processor (if available).
> > > +
> > > +Ranking Table
> > > +-------------
> > > +The ranking table is a shared memory region that is used to communicate
> > > the
> > > +performance and energy efficiency capabilities of each CPU in the system.
> > > +
> > > +The ranking table design includes rankings for each APIC ID in the system
> > > and
> > > +rankings both for performance and efficiency for each workload
> > > classification.
> > > +
> > > +.. kernel-doc:: drivers/platform/x86/amd/hfi/hfi.c
> > > +   :doc: amd_shmem_info
> > > +
> > > +Ranking Table update
> > > +---------------------------
> > > +The power management firmware issues an platform interrupt after updating
> > > the ranking
> > > +table and is ready for the operating system to consume it. CPUs receive
> > > such interrupt
> > > +and read new ranking table from shared memory which PCCT table has
> > > provided, then
> > > +``amd_hfi`` driver parse the new table to provide new consume data for
> > > scheduling decisions.
> > > diff --git a/Documentation/arch/x86/index.rst
> > > b/Documentation/arch/x86/index.rst
> > > index 8ac64d7de4dc9..56f2923f52597 100644
> > > --- a/Documentation/arch/x86/index.rst
> > > +++ b/Documentation/arch/x86/index.rst
> > > @@ -43,3 +43,4 @@ x86-specific Documentation
> > >      features
> > >      elf_auxvec
> > >      xstate
> > > +   amd-hfi
> > > 
> > 
> 
>