Re: [PATCH V2 0/5] Uncore PMON discovery mechanism support

Kin Cho
2022-09-20T18:25:44Z

Hi Kan,

We're seeing the warning below from uncore_insert_box_info on SPR.
I added a debug print:

/* Parsing Unit Discovery State */
for (i = 0; i < global.max_units; i++) {
..
uncore_insert_box_info(&unit, die, *parsed);
>>        pr_info("%d 0x%llx\n", i, unit.ctl);

and here's the output:

[   17.758579] intel_uncore: 0 0x2fc0
[   17.763117] intel_uncore: 2 0x2010
..
[   17.935286] intel_uncore: 65 0x87e410a0
[   17.940308] intel_uncore: 66 0x87e21318
[   17.945331] ------------[ cut here ]------------
[   17.946305] WARNING: CPU: 65 PID: 1 at
arch/x86/events/intel/uncore_discovery.c:184
intel_uncore_has_discovery_tables+0x4c0/0x65c
..
[   18.161512] intel_uncore: 67 0x87e410a0
[   18.166533] intel_uncore: 68 0x87e21318
..

Any suggestions?

-kin

[   17.945331] ------------[ cut here ]------------
[   17.946305] WARNING: CPU: 65 PID: 1 at
arch/x86/events/intel/uncore_discovery.c:184
intel_uncore_has_discovery_tables+0x4c0/0x65c
[   17.946305] Modules linked in:
[   17.946305] CPU: 65 PID: 1 Comm: swapper/0 Not tainted
5.4.17-2136.313.1-X10-2c+ #4
[   17.946305] Hardware name: Oracle Corporation
sca-x102c-107-sp/PCA,MB,X10-2c, BIOS 79805101 09/13/2022
[   17.946305] RIP: 0010:intel_uncore_has_discovery_tables+0x4c0/0x65c
[   17.946305] Code: 38 48 63 f0 48 8d 3c b1 45 8b 04 b0 44 89 07 4c 8b
42 40 45 8b 04 b0 45 89 04 b1 0f b7 75 ca 3b 37 75 cf 4c 89 8d 68 ff ff
ff <0f> 0b 48 89 cf e8 c6 4f 2b 00 4c 8b 8d 68 ff ff ff 4c 89 cf e8 b7
[   17.946305] RSP: 0000:ff4b04f60006bd08 EFLAGS: 00010246
[   17.946305] RAX: 0000000000000002 RBX: 0000000000000044 RCX:
ff43a98a4ff1bb30
[   17.946305] RDX: ff43a98a4ff294e0 RSI: 0000000000000003 RDI:
ff43a98a4ff1bb38
[   17.946305] RBP: ff4b04f60006bdb0 R08: 0000000000018000 R09:
ff43a98a4ff1b310
[   17.946305] R10: 0000000000000005 R11: ff43a98c7f7fe000 R12:
ff43a98777d66000
[   17.946305] R13: 0000000000015240 R14: ff4b04f61b286000 R15:
0000000000000043
[   17.946305] FS:  0000000000000000(0000) GS:ff43a90a7f840000(0000)
knlGS:0000000000000000
[   17.946305] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   17.946305] CR2: 0000000000000000 CR3: 000000967e00a001 CR4:
0000000000761ee0
[   17.946305] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[   17.946305] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7:
0000000000000400
[   17.946305] PKRU: 55555554
[   17.946305] Call Trace:
[   17.946305]  ? uncore_types_init+0x25f/0x25f
[   17.946305]  intel_uncore_init+0x64/0x50c
[   17.946305]  ? perf_pmu_register+0x2cc/0x403
[   17.946305]  ? uncore_types_init+0x25f/0x25f
[   17.946305]  do_one_initcall+0x52/0x1e1
[   17.946305]  ? trace_event_define_fields_initcall_level+0x2a/0x36
[   17.946305]  kernel_init_freeable+0x1fc/0x2a7
[   17.946305]  ? loglevel+0x5d/0x5d
[   17.946305]  ? rest_init+0xb0/0xb0
[   17.946305]  kernel_init+0xe/0x123
[   17.946305]  ret_from_fork+0x24/0x36
[   17.946305] ---[ end trace d9131e47b8a615f4 ]---

On 3/17/21 10:59 AM, kan.liang@linux.intel.com wrote:
> From: Kan Liang <kan.liang@linux.intel.com>
>
> Changes since V1:
> - Use the generic rbtree functions, rb_add() and rb_find(). (Patch 1)
> - Add a module parameter, uncore_no_discover. If users don't want the
> discovery feature, they can set uncore_no_discover=true. (Patch 1)
>
>
> A mechanism of self-describing HW for the uncore PMOM has been
> introduced with the latest Intel platforms. By reading through an MMIO
> page worth of information, SW can ‘discover’ all the standard uncore
> PMON registers.
>
> With the discovery mechanism, Perf can
> - Retrieve the generic uncore unit information of all standard uncore
> blocks, e.g., the address of counters, the address of the counter
> control, the counter width, the access type, etc.
> Perf can provide basic uncore support based on this information.
> For a new platform, perf users will get basic uncore support even if
> the platform-specific enabling code is not ready yet.
> - Retrieve accurate uncore unit information, e.g., the number of uncore
> boxes. The number of uncore boxes may be different among machines.
> Current perf hard code the max number of the uncore blocks. On some
> machines, perf may create a PMU for an unavailable uncore block.
> Although there is no harm (always return 0 for the unavailable uncore
> block), it may confuse the users. The discovery mechanism can provide
> the accurate number of available uncore boxes on a machine.
>
> But, the discovery mechanism has some limits,
> - Rely on BIOS's support. If a BIOS doesn't support the discovery
> mechanism, the uncore driver will exit with -ENODEV. There is nothing
> changed.
> - Only provide the generic uncore unit information. The information for
> the advanced features, such as fixed counters, filters, and
> constraints, cannot be retrieved.
> - Only support the standard PMON blocks. Non-standard PMON blocks, e.g.,
> free-running counters, are not supported.
> - Only provide an ID for an uncore block. No meaningful name is
> provided. The uncore_type_&typeID_&boxID will be used as the name.
> - Enabling the PCI and MMIO type of uncore blocks rely on the NUMA support.
> These uncore blocks require the mapping information from a BUS to a
> die. The current discovery table doesn't provide the mapping
> information. The pcibus_to_node() from NUMA is used to retrieve the
> information. If NUMA is not supported, some uncore blocks maybe
> unavailable.
>
> To locate the MMIO page, SW has to find a PCI device with the unique
> capability ID 0x23 and retrieve its BAR address.
>
> The spec can be found at Snow Ridge or Ice Lake server's uncore document.
> https://cdrdv2.intel.com/v1/dl/getContent/611319
>
> Kan Liang (5):
> perf/x86/intel/uncore: Parse uncore discovery tables
> perf/x86/intel/uncore: Generic support for the MSR type of uncore
> blocks
> perf/x86/intel/uncore: Rename uncore_notifier to
> uncore_pci_sub_notifier
> perf/x86/intel/uncore: Generic support for the PCI type of uncore
> blocks
> perf/x86/intel/uncore: Generic support for the MMIO type of uncore
> blocks
>
> arch/x86/events/intel/Makefile | 2 +-
> arch/x86/events/intel/uncore.c | 188 ++++++++--
> arch/x86/events/intel/uncore.h | 10 +-
> arch/x86/events/intel/uncore_discovery.c | 622 +++++++++++++++++++++++++++++++
> arch/x86/events/intel/uncore_discovery.h | 131 +++++++
> 5 files changed, 922 insertions(+), 31 deletions(-)
> create mode 100644 arch/x86/events/intel/uncore_discovery.c
> create mode 100644 arch/x86/events/intel/uncore_discovery.h
>