Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Daniel Walker (danielwa)]

[-- Attachment #1: Type: text/plain, Size: 2549 bytes --]


Hi,

I bisected an issue of a missing pci device to commit 2841631 the commit title
in the subject line which was included in v6.1 stable branch.

There was a later fix for a similar missing pci device commit 36c676e2 which
appears to be for Goldmont/Apollo Lake. The hardware I'm using is
Goldmont/Denverton. This fix did not appear to change the behavior I'm seeing.

The pci device which is disappearing is a custom gpio device.

I tested v6.12-rc5-next to see if any other changes had fixed the issue, but there was
no change in behavior since commit 2841631 .

When booting up the device is shown in the pci boot messages but the device
doesn't end up making it to lspci once you get to a prompt.

Here are the cpu details,

/ # cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 95
model name      : Intel(R) Atom(TM) CPU C3558 @ 2.20GHz
stepping        : 1
microcode       : 0x32
cpu MHz         : 2200.000
cache size      : 2048 KB
physical id     : 0
siblings        : 4
core id         : 2
cpu cores       : 4
apicid          : 4
initial apicid  : 4
fpu             : yes
fpu_exception   : yes
cpuid level     : 21
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave rdrand lahf_lm 3dnowprefetch cpuid_fault epb cat_l2 ssbd ibrs ibpb stibp tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust smep erms mpx rdt_a rdseed smap clflushopt intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves dtherm arat pln pts vnmi md_clear arch_capabilities
vmx flags       : vnmi preemption_timer posted_intr invvpid ept_x_only ept_ad ept_1gb flexpriority apicv tsc_offset vtpr mtf vapic ept vpid unrestricted_guest vapic_reg vid ple shadow_vmcs ept_violation_ve
bugs            : spectre_v1 spectre_v2 spec_store_bypass rfds
bogomips        : 4400.00
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:


The vendor and device details for the pci device are 8086:19dd . I've attached a
boot log for a kernel where the pci device is missing with commit 2841631 included

Any thought on this?

Daniel

[-- Attachment #2: dmesg-cat9k-pci-denverton.txt.gz --]
[-- Type: application/gzip, Size: 9019 bytes --]

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Hans de Goede]

[-- Attachment #1: Type: text/plain, Size: 1262 bytes --]

Hi,

On 13-Nov-24 4:42 PM, Daniel Walker (danielwa) wrote:
> 
> Hi,
> 
> I bisected an issue of a missing pci device to commit 2841631 the commit title
> in the subject line which was included in v6.1 stable branch.
> 
> There was a later fix for a similar missing pci device commit 36c676e2 which
> appears to be for Goldmont/Apollo Lake. The hardware I'm using is
> Goldmont/Denverton. This fix did not appear to change the behavior I'm seeing.
> 
> The pci device which is disappearing is a custom gpio device.
> 
> I tested v6.12-rc5-next to see if any other changes had fixed the issue, but there was
> no change in behavior since commit 2841631 .
> 
> When booting up the device is shown in the pci boot messages but the device
> doesn't end up making it to lspci once you get to a prompt.

Please give the attached patch a try, this will hopefully fix things.

Once I have confirmation that this fixes things I'll post it to the list.

Note this will not backport to the 6.1 stable branch cleanly due to
changes in the x86_cpu_id macros in mainline. Backporting it should
be trivial. Please send a backport to stable@vger.kernel.org yourself
once this has been merged upstream.

If you backport this, please also backport 36c676e2 first.

Regards,

Hans


[-- Attachment #2: 0001-platform-x86-p2sb-Cache-correct-PCI-bar-for-P2SB-on-.patch --]
[-- Type: text/x-patch, Size: 1477 bytes --]

From fa351c7614dd97713b26342c96c61fc28699d8d4 Mon Sep 17 00:00:00 2001
From: Hans de Goede <hdegoede@redhat.com>
Date: Wed, 13 Nov 2024 17:14:10 +0100
Subject: [PATCH] platform/x86: p2sb: Cache correct PCI bar for P2SB on
 Denverton and Gemini Lake

Denverton and Gemini Lak are Goldmont derived designs and as such have
the P2SB at device.function 13.0, rather then at the default 31.1.

At a mapping to P2SB_DEVFN_GOLDMONT to p2sb_cpu_ids[] for these 2 Goldmont
variants so that the correct PCI bar gets cached.

This fixes P2SB unhiding not working on these devices which fixes
GPIO and SPI (Gemini Lake only) support not working on devices where
these were hidden by the BIOS.

Fixes: 2841631a0365 ("platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe")
Reported-by: Daniel Walker (danielwa) <danielwa@cisco.com>
Closes: https://lore.kernel.org/lkml/ZzTI+biIUTvFT6NC@goliath/
Signed-off-by: Hans de Goede <hdegoede@redhat.com>
---
 drivers/platform/x86/p2sb.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/platform/x86/p2sb.c b/drivers/platform/x86/p2sb.c
index 31f38309b389..beb94c492eec 100644
--- a/drivers/platform/x86/p2sb.c
+++ b/drivers/platform/x86/p2sb.c
@@ -25,6 +25,8 @@
 
 static const struct x86_cpu_id p2sb_cpu_ids[] = {
 	X86_MATCH_VFM(INTEL_ATOM_GOLDMONT, P2SB_DEVFN_GOLDMONT),
+	X86_MATCH_VFM(INTEL_ATOM_GOLDMONT_D, P2SB_DEVFN_GOLDMONT),
+	X86_MATCH_VFM(INTEL_ATOM_GOLDMONT_PLUS, P2SB_DEVFN_GOLDMONT),
 	{}
 };
 
-- 
2.47.0

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Hans de Goede]

Hi,

On 13-Nov-24 5:24 PM, Hans de Goede wrote:
> Hi,
> 
> On 13-Nov-24 4:42 PM, Daniel Walker (danielwa) wrote:
>>
>> Hi,
>>
>> I bisected an issue of a missing pci device to commit 2841631 the commit title
>> in the subject line which was included in v6.1 stable branch.
>>
>> There was a later fix for a similar missing pci device commit 36c676e2 which
>> appears to be for Goldmont/Apollo Lake. The hardware I'm using is
>> Goldmont/Denverton. This fix did not appear to change the behavior I'm seeing.
>>
>> The pci device which is disappearing is a custom gpio device.
>>
>> I tested v6.12-rc5-next to see if any other changes had fixed the issue, but there was
>> no change in behavior since commit 2841631 .
>>
>> When booting up the device is shown in the pci boot messages but the device
>> doesn't end up making it to lspci once you get to a prompt.
> 
> Please give the attached patch a try, this will hopefully fix things.
> 
> Once I have confirmation that this fixes things I'll post it to the list.
> 
> Note this will not backport to the 6.1 stable branch cleanly due to
> changes in the x86_cpu_id macros in mainline. Backporting it should
> be trivial. Please send a backport to stable@vger.kernel.org yourself
> once this has been merged upstream.
> 
> If you backport this, please also backport 36c676e2 first.

Never mind, self nack. This is correct for Gemini Lake which
has its SPI at device.function 13.2 like Apollo Lake.

But looking at the dmesg Denverton actually has it at 1f.1
aka 31.1 like most other Intel SoCs.

Which make me wonder why this does not work on Denverton.

It probably has something to do with these 2 messages:

pci 0000:00:1f.1: BAR 0 [mem 0xfd000000-0xfdffffff 64bit]: can't claim; no compatible bridge window
pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned

I'm guessing that this re-assignment is messing up
the p2sb BAR caching, after which things go wrong.

Daniel, are you seeing similar messages with a working kernel ?

Regards,

Hans

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Hans de Goede]

Hi,

On 13-Nov-24 5:33 PM, Hans de Goede wrote:
> Hi,
> 
> On 13-Nov-24 5:24 PM, Hans de Goede wrote:
>> Hi,
>>
>> On 13-Nov-24 4:42 PM, Daniel Walker (danielwa) wrote:
>>>
>>> Hi,
>>>
>>> I bisected an issue of a missing pci device to commit 2841631 the commit title
>>> in the subject line which was included in v6.1 stable branch.
>>>
>>> There was a later fix for a similar missing pci device commit 36c676e2 which
>>> appears to be for Goldmont/Apollo Lake. The hardware I'm using is
>>> Goldmont/Denverton. This fix did not appear to change the behavior I'm seeing.
>>>
>>> The pci device which is disappearing is a custom gpio device.
>>>
>>> I tested v6.12-rc5-next to see if any other changes had fixed the issue, but there was
>>> no change in behavior since commit 2841631 .
>>>
>>> When booting up the device is shown in the pci boot messages but the device
>>> doesn't end up making it to lspci once you get to a prompt.
>>
>> Please give the attached patch a try, this will hopefully fix things.
>>
>> Once I have confirmation that this fixes things I'll post it to the list.
>>
>> Note this will not backport to the 6.1 stable branch cleanly due to
>> changes in the x86_cpu_id macros in mainline. Backporting it should
>> be trivial. Please send a backport to stable@vger.kernel.org yourself
>> once this has been merged upstream.
>>
>> If you backport this, please also backport 36c676e2 first.
> 
> Never mind, self nack. This is correct for Gemini Lake which
> has its SPI at device.function 13.2 like Apollo Lake.
> 
> But looking at the dmesg Denverton actually has it at 1f.1
> aka 31.1 like most other Intel SoCs.
> 
> Which make me wonder why this does not work on Denverton.
> 
> It probably has something to do with these 2 messages:
> 
> pci 0000:00:1f.1: BAR 0 [mem 0xfd000000-0xfdffffff 64bit]: can't claim; no compatible bridge window
> pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned
> 
> I'm guessing that this re-assignment is messing up
> the p2sb BAR caching, after which things go wrong.
> 
> Daniel, are you seeing similar messages with a working kernel ?

p.s. Can you also provide a full dmesg log from a working kernel ?

Regards,
 
Hans

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Hans de Goede]

Hi,

On 13-Nov-24 5:33 PM, Hans de Goede wrote:
> Hi,
> 
> On 13-Nov-24 5:24 PM, Hans de Goede wrote:
>> Hi,
>>
>> On 13-Nov-24 4:42 PM, Daniel Walker (danielwa) wrote:
>>>
>>> Hi,
>>>
>>> I bisected an issue of a missing pci device to commit 2841631 the commit title
>>> in the subject line which was included in v6.1 stable branch.
>>>
>>> There was a later fix for a similar missing pci device commit 36c676e2 which
>>> appears to be for Goldmont/Apollo Lake. The hardware I'm using is
>>> Goldmont/Denverton. This fix did not appear to change the behavior I'm seeing.
>>>
>>> The pci device which is disappearing is a custom gpio device.
>>>
>>> I tested v6.12-rc5-next to see if any other changes had fixed the issue, but there was
>>> no change in behavior since commit 2841631 .
>>>
>>> When booting up the device is shown in the pci boot messages but the device
>>> doesn't end up making it to lspci once you get to a prompt.
>>
>> Please give the attached patch a try, this will hopefully fix things.
>>
>> Once I have confirmation that this fixes things I'll post it to the list.
>>
>> Note this will not backport to the 6.1 stable branch cleanly due to
>> changes in the x86_cpu_id macros in mainline. Backporting it should
>> be trivial. Please send a backport to stable@vger.kernel.org yourself
>> once this has been merged upstream.
>>
>> If you backport this, please also backport 36c676e2 first.
> 
> Never mind, self nack. This is correct for Gemini Lake which
> has its SPI at device.function 13.2 like Apollo Lake.
> 
> But looking at the dmesg Denverton actually has it at 1f.1
> aka 31.1 like most other Intel SoCs.
> 
> Which make me wonder why this does not work on Denverton.
> 
> It probably has something to do with these 2 messages:
> 
> pci 0000:00:1f.1: BAR 0 [mem 0xfd000000-0xfdffffff 64bit]: can't claim; no compatible bridge window
> pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned
> 
> I'm guessing that this re-assignment is messing up
> the p2sb BAR caching, after which things go wrong.

Hmm, but that should be fixed by 2c6370e66076 ("platform/x86: p2sb: Don't init until unassigned resources have been assigned")

and you are seeing this with 6.12, which has that.

Can you try adding a pr_info() to the top of p2sb_cache_resources()
with 6.12 and then collec a new dmesg ?

If that pr_info() is done after the:

pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned

message then that does not explain things.

Regards,

Hans

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Daniel Walker (danielwa)]

[-- Attachment #1: Type: text/plain, Size: 2478 bytes --]

On Wed, Nov 13, 2024 at 05:38:46PM +0100, Hans de Goede wrote:
> Hi,
> 
> On 13-Nov-24 5:33 PM, Hans de Goede wrote:
> > Hi,
> > 
> > On 13-Nov-24 5:24 PM, Hans de Goede wrote:
> >> Hi,
> >>
> >> On 13-Nov-24 4:42 PM, Daniel Walker (danielwa) wrote:
> >>>
> >>> Hi,
> >>>
> >>> I bisected an issue of a missing pci device to commit 2841631 the commit title
> >>> in the subject line which was included in v6.1 stable branch.
> >>>
> >>> There was a later fix for a similar missing pci device commit 36c676e2 which
> >>> appears to be for Goldmont/Apollo Lake. The hardware I'm using is
> >>> Goldmont/Denverton. This fix did not appear to change the behavior I'm seeing.
> >>>
> >>> The pci device which is disappearing is a custom gpio device.
> >>>
> >>> I tested v6.12-rc5-next to see if any other changes had fixed the issue, but there was
> >>> no change in behavior since commit 2841631 .
> >>>
> >>> When booting up the device is shown in the pci boot messages but the device
> >>> doesn't end up making it to lspci once you get to a prompt.
> >>
> >> Please give the attached patch a try, this will hopefully fix things.
> >>
> >> Once I have confirmation that this fixes things I'll post it to the list.
> >>
> >> Note this will not backport to the 6.1 stable branch cleanly due to
> >> changes in the x86_cpu_id macros in mainline. Backporting it should
> >> be trivial. Please send a backport to stable@vger.kernel.org yourself
> >> once this has been merged upstream.
> >>
> >> If you backport this, please also backport 36c676e2 first.
> > 
> > Never mind, self nack. This is correct for Gemini Lake which
> > has its SPI at device.function 13.2 like Apollo Lake.
> > 
> > But looking at the dmesg Denverton actually has it at 1f.1
> > aka 31.1 like most other Intel SoCs.
> > 
> > Which make me wonder why this does not work on Denverton.
> > 
> > It probably has something to do with these 2 messages:
> > 
> > pci 0000:00:1f.1: BAR 0 [mem 0xfd000000-0xfdffffff 64bit]: can't claim; no compatible bridge window
> > pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned
> > 
> > I'm guessing that this re-assignment is messing up
> > the p2sb BAR caching, after which things go wrong.
> > 
> > Daniel, are you seeing similar messages with a working kernel ?
> 
> p.s. Can you also provide a full dmesg log from a working kernel ?
> 

Attached is a working dmesg from v6.8-rc1.

Daniel

[-- Attachment #2: v6.8-cat9k-working-pci.txt --]
[-- Type: text/plain, Size: 31905 bytes --]

Linux version 6.8.0-rc1-00006-g416de0246f35 (dwalker@goliath) (gcc (Ubuntu 13.2.0-4ubuntu3) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.41) #3 SMP Fri Nov  1 14:29:31 PDT 2024
Command line: console=ttyS0,9600 acpi_osi=Linux DEBUG_CONF=/flash/debug.conf REAL_MGMTE_DEV= rdinit=/bin/ash BOARDID=28780 SR_BOOT=tftp://223.255.254.253//auto/tftp-sjc-users2/danielwa/bzImage2.bin
BIOS-provided physical RAM map:
BIOS-e820: [mem 0x0000000000000000-0x000000000005efff] usable
BIOS-e820: [mem 0x000000000005f000-0x000000000005ffff] reserved
BIOS-e820: [mem 0x0000000000060000-0x0000000000087fff] usable
BIOS-e820: [mem 0x0000000000088000-0x000000000009ffff] reserved
BIOS-e820: [mem 0x0000000000100000-0x000000007067efff] usable
BIOS-e820: [mem 0x000000007067f000-0x00000000706b9fff] reserved
BIOS-e820: [mem 0x00000000706ba000-0x0000000071f31fff] usable
BIOS-e820: [mem 0x0000000071f32000-0x0000000073031fff] reserved
BIOS-e820: [mem 0x0000000073032000-0x0000000078240fff] usable
BIOS-e820: [mem 0x0000000078241000-0x000000007ebbefff] reserved
BIOS-e820: [mem 0x000000007ebbf000-0x000000007efbefff] ACPI NVS
BIOS-e820: [mem 0x000000007efbf000-0x000000007effefff] ACPI data
BIOS-e820: [mem 0x000000007efff000-0x000000007effffff] usable
BIOS-e820: [mem 0x00000000e0000000-0x00000000efffffff] reserved
BIOS-e820: [mem 0x00000000fd000000-0x00000000fe7fffff] reserved
BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
BIOS-e820: [mem 0x00000000fec80000-0x00000000fed00fff] reserved
BIOS-e820: [mem 0x00000000fed10000-0x00000000fed17fff] reserved
BIOS-e820: [mem 0x00000000fed84000-0x00000000fed84fff] reserved
BIOS-e820: [mem 0x00000000ff7f0000-0x00000000fffeffff] reserved
BIOS-e820: [mem 0x0000000100000000-0x000000027fffffff] usable
NX (Execute Disable) protection: active
APIC: Static calls initialized
efi: EFI v2.4 by INSYDE Corp.
efi: SMBIOS=0x784a3000 ESRT=0x784a0d18 ACPI 2.0=0x7effe014
efi: Remove mem57: MMIO range=[0xe0000000-0xefffffff] (256MB) from e820 map
efi: Remove mem58: MMIO range=[0xfd000000-0xfe7fffff] (24MB) from e820 map
efi: Not removing mem59: MMIO range=[0xfec00000-0xfec00fff] (4KB) from e820 map
efi: Remove mem60: MMIO range=[0xfec80000-0xfed00fff] (0MB) from e820 map
efi: Not removing mem61: MMIO range=[0xfed10000-0xfed17fff] (32KB) from e820 map
efi: Not removing mem62: MMIO range=[0xfed84000-0xfed84fff] (4KB) from e820 map
efi: Remove mem63: MMIO range=[0xff7f0000-0xfffeffff] (8MB) from e820 map
SMBIOS 2.8 present.
DMI: Insyde Harrisonville/Type2 - Board Product Name1, BIOS 05.10.12.0026 07/25/2018
tsc: Detected 2200.000 MHz processor
last_pfn = 0x280000 max_arch_pfn = 0x400000000
MTRR map: 10 entries (5 fixed + 5 variable; max 25), built from 10 variable MTRRs
x86/PAT: PAT support disabled because CONFIG_X86_PAT is disabled in the kernel.
x86/PAT: Configuration [0-7]: WB  WT  UC- UC  WB  WT  UC- UC
last_pfn = 0x7f000 max_arch_pfn = 0x400000000
esrt: Reserving ESRT space from 0x00000000784a0d18 to 0x00000000784a0d50.
Using GB pages for direct mapping
Secure boot could not be determined
ACPI: Early table checksum verification disabled
ACPI: RSDP 0x000000007EFFE014 000024 (v02 INSYDE)
ACPI: XSDT 0x000000007EFFE120 0000AC (v01 INSYDE DNV      00000001      01000013)
ACPI: FACP 0x000000007EFF9000 00010C (v05 INSYDE DNV      00000001 ACPI 00040000)
ACPI: DSDT 0x000000007EFF0000 0048FB (v02 INSYDE DNV      00000000 ACPI 00040000)
ACPI: FACS 0x000000007EFBD000 000040
ACPI: UEFI 0x000000007EFFD000 000236 (v01 INSYDE DNV      00000001 ACPI 00040000)
ACPI: UEFI 0x000000007EFFC000 000042 (v01 INSYDE DNV      00000000 ACPI 00040000)
ACPI: BDAT 0x000000007EFFB000 000030 (v01 INSYDE DNV      00000000 ACPI 00040000)
ACPI: BOOT 0x000000007EFFA000 000028 (v01 INSYDE DNV      00000001 ACPI 00040000)
ACPI: HPET 0x000000007EFF8000 000038 (v01 INSYDE DNV      00000001 ACPI 00040000)
ACPI: APIC 0x000000007EFF7000 0000D2 (v03 INSYDE DNV      00000001 ACPI 00040000)
ACPI: MCFG 0x000000007EFF6000 00003C (v01 INSYDE DNV      00000001 ACPI 00040000)
ACPI: WDAT 0x000000007EFF5000 0001C4 (v01 INSYDE DNV      00000001 ACPI 00040000)
ACPI: SSDT 0x000000007EFED000 001901 (v02 INSYDE DNV      00003000 ACPI 00040000)
ACPI: DMAR 0x000000007EFEC000 000070 (v01 INSYDE DNV      00000001 ACPI 00040000)
ACPI: DBG2 0x000000007EFEB000 000061 (v00 INSYDE DNV      00000000 ACPI 00040000)
ACPI: HEST 0x000000007EFEA000 0000A8 (v01 INSYDE DNV      00000001 ACPI 00040000)
ACPI: ERST 0x000000007EFE9000 000230 (v01 INSYDE DNV      00000001 ACPI 00040000)
ACPI: BERT 0x000000007EFE8000 000030 (v01 INSYDE DNV      00000001 ACPI 00040000)
ACPI: EINJ 0x000000007EFE7000 000150 (v01 INSYDE DNV      00000001 ACPI 00040000)
ACPI: BGRT 0x000000007EFE6000 000038 (v01 INSYDE DNV      00000001 ACPI 00040000)
ACPI: Reserving FACP table memory at [mem 0x7eff9000-0x7eff910b]
ACPI: Reserving DSDT table memory at [mem 0x7eff0000-0x7eff48fa]
ACPI: Reserving FACS table memory at [mem 0x7efbd000-0x7efbd03f]
ACPI: Reserving UEFI table memory at [mem 0x7effd000-0x7effd235]
ACPI: Reserving UEFI table memory at [mem 0x7effc000-0x7effc041]
ACPI: Reserving BDAT table memory at [mem 0x7effb000-0x7effb02f]
ACPI: Reserving BOOT table memory at [mem 0x7effa000-0x7effa027]
ACPI: Reserving HPET table memory at [mem 0x7eff8000-0x7eff8037]
ACPI: Reserving APIC table memory at [mem 0x7eff7000-0x7eff70d1]
ACPI: Reserving MCFG table memory at [mem 0x7eff6000-0x7eff603b]
ACPI: Reserving WDAT table memory at [mem 0x7eff5000-0x7eff51c3]
ACPI: Reserving SSDT table memory at [mem 0x7efed000-0x7efee900]
ACPI: Reserving DMAR table memory at [mem 0x7efec000-0x7efec06f]
ACPI: Reserving DBG2 table memory at [mem 0x7efeb000-0x7efeb060]
ACPI: Reserving HEST table memory at [mem 0x7efea000-0x7efea0a7]
ACPI: Reserving ERST table memory at [mem 0x7efe9000-0x7efe922f]
ACPI: Reserving BERT table memory at [mem 0x7efe8000-0x7efe802f]
ACPI: Reserving EINJ table memory at [mem 0x7efe7000-0x7efe714f]
ACPI: Reserving BGRT table memory at [mem 0x7efe6000-0x7efe6037]
Zone ranges:
  DMA      [mem 0x0000000000001000-0x0000000000ffffff]
  DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
  Normal   [mem 0x0000000100000000-0x000000027fffffff]
Movable zone start for each node
Early memory node ranges
  node   0: [mem 0x0000000000001000-0x000000000005efff]
  node   0: [mem 0x0000000000060000-0x0000000000087fff]
  node   0: [mem 0x0000000000100000-0x000000007067efff]
  node   0: [mem 0x00000000706ba000-0x0000000071f31fff]
  node   0: [mem 0x0000000073032000-0x0000000078240fff]
  node   0: [mem 0x000000007efff000-0x000000007effffff]
  node   0: [mem 0x0000000100000000-0x000000027fffffff]
Initmem setup node 0 [mem 0x0000000000001000-0x000000027fffffff]
On node 0, zone DMA: 1 pages in unavailable ranges
On node 0, zone DMA: 1 pages in unavailable ranges
On node 0, zone DMA: 120 pages in unavailable ranges
On node 0, zone DMA32: 59 pages in unavailable ranges
On node 0, zone DMA32: 4352 pages in unavailable ranges
On node 0, zone DMA32: 28094 pages in unavailable ranges
On node 0, zone Normal: 4096 pages in unavailable ranges
ACPI: PM-Timer IO Port: 0x1808
ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-119
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: Using ACPI (MADT) for SMP configuration information
ACPI: HPET id: 0x8086a001 base: 0xfed00000
TSC deadline timer available
smpboot: Allowing 16 CPUs, 12 hotplug CPUs
[mem 0x7f000000-0xfebfffff] available for PCI devices
clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns
setup_percpu: NR_CPUS:16 nr_cpumask_bits:16 nr_cpu_ids:16 nr_node_ids:1
percpu: Embedded 54 pages/cpu s181032 r8192 d31960 u262144
Kernel command line: console=ttyS0,9600 acpi_osi=Linux DEBUG_CONF=/flash/debug.conf REAL_MGMTE_DEV= rdinit=/bin/ash BOARDID=28780 SR_BOOT=tftp://223.255.254.253//auto/tftp-sjc-users2/danielwa/bzImage2.bin
Unknown kernel command line parameters "DEBUG_CONF=/flash/debug.conf REAL_MGMTE_DEV= BOARDID=28780 SR_BOOT=tftp://223.255.254.253//auto/tftp-sjc-users2/danielwa/bzImage2.bin", will be passed to user space.
random: crng init done
Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, linear)
Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
Built 1 zonelists, mobility grouping on.  Total pages: 2028098
mem auto-init: stack:all(zero), heap alloc:off, heap free:off
software IO TLB: area num 16.
Memory: 7905864K/8241716K available (14336K kernel code, 1957K rwdata, 4868K rodata, 31740K init, 5112K bss, 335592K reserved, 0K cma-reserved)
SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=16, Nodes=1
kmemleak: Kernel memory leak detector disabled
ftrace: allocating 41440 entries in 162 pages
ftrace: allocated 162 pages with 3 groups
rcu: Hierarchical RCU implementation.
rcu:    RCU event tracing is enabled.
        Rude variant of Tasks RCU enabled.
        Tracing variant of Tasks RCU enabled.
rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.
NR_IRQS: 4352, nr_irqs: 2184, preallocated irqs: 16
rcu: srcu_init: Setting srcu_struct sizes based on contention.
Console: colour dummy device 80x25
printk: legacy console [ttyS0] enabled
ACPI: Core revision 20230628
clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635855245 ns
APIC: Switch to symmetric I/O mode setup
DMAR: Host address width 39
DMAR: DRHD base: 0x000000fed90000 flags: 0x1
DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap d2008c40660462 ecap f050da
DMAR: RMRR base: 0x0000007ab40000 end: 0x0000007ab5ffff
DMAR-IR: IOAPIC id 2 under DRHD base  0xfed90000 IOMMU 0
DMAR-IR: HPET id 0 under DRHD base 0xfed90000
DMAR-IR: Enabled IRQ remapping in xapic mode
APIC: Switched APIC routing to: physical flat
..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x1fb633008a4, max_idle_ns: 440795292230 ns
Calibrating delay loop (skipped), value calculated using timer frequency.. 4400.00 BogoMIPS (lpj=2200000)
CPU0: Thermal monitoring enabled (TM1)
process: using mwait in idle threads
Last level iTLB entries: 4KB 48, 2MB 0, 4MB 0
Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0
Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
Spectre V2 : Mitigation: Retpolines
Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
Spectre V2 : Spectre v2 / SpectreRSB : Filling RSB on VMEXIT
Spectre V2 : Enabling Restricted Speculation for firmware calls
Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier
Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl
x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
x86/fpu: Supporting XSAVE feature 0x008: 'MPX bounds registers'
x86/fpu: Supporting XSAVE feature 0x010: 'MPX CSR'
x86/fpu: xstate_offset[3]:  576, xstate_sizes[3]:   64
x86/fpu: xstate_offset[4]:  640, xstate_sizes[4]:   64
x86/fpu: Enabled xstate features 0x1b, context size is 704 bytes, using 'compacted' format.
Freeing SMP alternatives memory: 40K
pid_max: default: 32768 minimum: 301
LSM: initializing lsm=capability,yama,selinux,integrity
Yama: becoming mindful.
SELinux:  Initializing.
Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
smpboot: CPU0: Intel(R) Atom(TM) CPU C3558 @ 2.20GHz (family: 0x6, model: 0x5f, stepping: 0x1)
RCU Tasks Rude: Setting shift to 4 and lim to 1 rcu_task_cb_adjust=1.
RCU Tasks Trace: Setting shift to 4 and lim to 1 rcu_task_cb_adjust=1.
Performance Events: PEBS fmt3+, Goldmont events, 32-deep LBR, full-width counters, Intel PMU driver.
... version:                4
... bit width:              48
... generic registers:      4
... value mask:             0000ffffffffffff
... max period:             00007fffffffffff
... fixed-purpose events:   3
... event mask:             000000070000000f
signal: max sigframe size: 2032
Estimated ratio of average max frequency by base frequency (times 1024): 1024
rcu: Hierarchical SRCU implementation.
rcu:    Max phase no-delay instances is 400.
smp: Bringing up secondary CPUs ...
smpboot: x86: Booting SMP configuration:
.... node  #0, CPUs:        #1  #2  #3
smp: Brought up 1 node, 4 CPUs
smpboot: Max logical packages: 4
smpboot: Total of 4 processors activated (17600.00 BogoMIPS)
devtmpfs: initialized
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
futex hash table entries: 4096 (order: 6, 262144 bytes, linear)
NET: Registered PF_NETLINK/PF_ROUTE protocol family
audit: initializing netlink subsys (disabled)
audit: type=2000 audit(1730497012.094:1): state=initialized audit_enabled=0 res=1
thermal_sys: Registered thermal governor 'step_wise'
thermal_sys: Registered thermal governor 'user_space'
cpuidle: using governor ladder
Simple Boot Flag at 0x44 set to 0x1
ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
PCI: ECAM [mem 0xe0000000-0xefffffff] (base 0xe0000000) for domain 0000 [bus 00-ff]
PCI: not using ECAM ([mem 0xe0000000-0xefffffff] not reserved)
PCI: Using configuration type 1 for base access
kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages
HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page
HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages
HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page
cryptd: max_cpu_qlen set to 1000
ACPI: Added _OSI(Module Device)
ACPI: Added _OSI(Processor Device)
ACPI: Added _OSI(3.0 _SCP Extensions)
ACPI: Added _OSI(Processor Aggregator Device)
ACPI: Added _OSI(Linux)
ACPI: 2 ACPI AML tables successfully acquired and loaded
ACPI: [Firmware Bug]: BIOS _OSI(Linux) query honored via cmdline
ACPI: Dynamic OEM Table Load:
ACPI: SSDT 0xFFFF8A3080A1CB40 0000B3 (v02 PmRef  Cpu0Cst  00003001 INTL 20130117)
ACPI: Dynamic OEM Table Load:
ACPI: SSDT 0xFFFF8A3080A55000 0004B3 (v02 PmRef  Cpu0Ist  00003000 INTL 20130117)
ACPI: Dynamic OEM Table Load:
ACPI: SSDT 0xFFFF8A3080A56800 00063B (v02 PmRef  ApIst    00003000 INTL 20130117)
ACPI: Dynamic OEM Table Load:
ACPI: SSDT 0xFFFF8A308090F800 000231 (v02 PmRef  ApCst    00003000 INTL 20130117)
ACPI: _OSC evaluated successfully for all CPUs
ACPI: Interpreter enabled
ACPI: PM: (supports S0 S5)
ACPI: Using IOAPIC for interrupt routing
PCI: ECAM [mem 0xe0000000-0xefffffff] (base 0xe0000000) for domain 0000 [bus 00-ff]
PCI: ECAM [mem 0xe0000000-0xefffffff] reserved as ACPI motherboard resource
PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
PCI: Using E820 reservations for host bridge windows
ACPI: Enabled 4 GPEs in block 00 to 7F
ACPI: \_TZ_.FN00: New power resource
ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig Segments MSI HPX-Type3]
acpi PNP0A08:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI]
acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration
PCI host bridge to bus 0000:00
pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
pci_bus 0000:00: root bus resource [mem 0x80000000-0xdfffffff window]
pci_bus 0000:00: root bus resource [mem 0x280000000-0x7fffffffff window]
pci_bus 0000:00: root bus resource [mem 0xfe010000-0xfe010fff window]
pci_bus 0000:00: root bus resource [bus 00-ff]
pci 0000:00:00.0: [8086:1980] type 00 class 0x060000 conventional PCI endpoint
pci 0000:00:04.0: [8086:19a1] type 00 class 0x060000 PCIe Root Complex Integrated Endpoint
pci 0000:00:05.0: [8086:19a2] type 00 class 0x080700 PCIe Root Complex Event Collector
pci 0000:00:05.0: PME# supported from D0 D3hot D3cold
pci 0000:00:09.0: [8086:19a4] type 01 class 0x060400 PCIe Root Port
pci 0000:00:09.0: BAR 0 [mem 0x7ea0200000-0x7ea021ffff 64bit]
pci 0000:00:09.0: PCI bridge to [bus 01-06]
pci 0000:00:09.0:   bridge window [io  0x5000-0x5fff]
pci 0000:00:09.0:   bridge window [mem 0x89000000-0x8affffff]
pci 0000:00:09.0:   bridge window [mem 0x7e00000000-0x7ea01fffff 64bit pref]
pci 0000:00:09.0: PME# supported from D0 D3hot D3cold
pci 0000:00:0b.0: [8086:19a6] type 01 class 0x060400 PCIe Root Port
pci 0000:00:0b.0: BAR 0 [mem 0x7ea0220000-0x7ea023ffff 64bit]
pci 0000:00:0b.0: PCI bridge to [bus 07-0c]
pci 0000:00:0b.0:   bridge window [io  0x4000-0x4fff]
pci 0000:00:0b.0:   bridge window [mem 0x87000000-0x881fffff]
pci 0000:00:0b.0:   bridge window [mem 0x81000000-0x81ffffff 64bit pref]
pci 0000:00:0b.0: PME# supported from D0 D3hot D3cold
pci 0000:00:0e.0: [8086:19a8] type 01 class 0x060400 PCIe Root Port
pci 0000:00:0e.0: BAR 0 [mem 0x7ea0240000-0x7ea025ffff 64bit]
pci 0000:00:0e.0: PCI bridge to [bus 0d-12]
pci 0000:00:0e.0:   bridge window [io  0x3000-0x3fff]
pci 0000:00:0e.0:   bridge window [mem 0x85000000-0x86ffffff]
pci 0000:00:0e.0:   bridge window [mem 0x7f00000000-0x7fa01fffff 64bit pref]
pci 0000:00:0e.0: PME# supported from D0 D3hot D3cold
pci 0000:00:10.0: [8086:19aa] type 01 class 0x060400 PCIe Root Port
pci 0000:00:10.0: BAR 0 [mem 0x7ea0260000-0x7ea027ffff 64bit]
pci 0000:00:10.0: PCI bridge to [bus 13-18]
pci 0000:00:10.0:   bridge window [io  0x2000-0x2fff]
pci 0000:00:10.0:   bridge window [mem 0x83000000-0x840fffff]
pci 0000:00:10.0:   bridge window [mem 0x82000000-0x82ffffff 64bit pref]
pci 0000:00:10.0: PME# supported from D0 D3hot D3cold
pci 0000:00:12.0: [8086:19ac] type 00 class 0x088000 PCIe Root Complex Integrated Endpoint
pci 0000:00:12.0: BAR 0 [mem 0x7ea0290000-0x7ea02903ff 64bit]
pci 0000:00:15.0: [8086:19d0] type 00 class 0x0c0330 conventional PCI endpoint
pci 0000:00:15.0: BAR 0 [mem 0x7ea0280000-0x7ea028ffff 64bit]
pci 0000:00:15.0: PME# supported from D3hot D3cold
pci 0000:00:16.0: [8086:19d1] type 01 class 0x060400 PCIe Root Port
pci 0000:00:16.0: PCI bridge to [bus 19-1a]
pci 0000:00:16.0:   bridge window [mem 0x88700000-0x88bfffff]
pci 0000:00:16.0:   bridge window [mem 0x88200000-0x886fffff 64bit pref]
pci 0000:00:16.0: PME# supported from D0 D3hot D3cold
pci 0000:00:18.0: [8086:19d3] type 00 class 0x078000 conventional PCI endpoint
pci 0000:00:18.0: BAR 0 [mem 0x7ea0291000-0x7ea0291fff 64bit]
pci 0000:00:18.0: PME# supported from D3hot
pci 0000:00:1c.0: [8086:19db] type 00 class 0x080501 conventional PCI endpoint
pci 0000:00:1c.0: BAR 0 [mem 0x7ea0292000-0x7ea0292fff 64bit]
pci 0000:00:1c.0: BAR 2 [mem 0x7ea0293000-0x7ea0293fff 64bit]
pci 0000:00:1f.0: [8086:19dc] type 00 class 0x060100 conventional PCI endpoint
pci 0000:00:1f.1: [8086:19dd] type 00 class 0x058000 conventional PCI endpoint
pci 0000:00:1f.1: BAR 0 [mem 0xfd000000-0xfdffffff 64bit]
pci 0000:00:1f.2: [8086:19de] type 00 class 0x058000 conventional PCI endpoint
pci 0000:00:1f.2: BAR 0 [mem 0x88c00000-0x88c03fff]
pci 0000:00:1f.4: [8086:19df] type 00 class 0x0c0500 conventional PCI endpoint
pci 0000:00:1f.4: BAR 0 [mem 0x7ea0294000-0x7ea02940ff 64bit]
pci 0000:00:1f.4: BAR 4 [io  0x6000-0x601f]
pci 0000:00:1f.5: [8086:19e0] type 00 class 0x0c8000 conventional PCI endpoint
pci 0000:00:1f.5: BAR 0 [mem 0xfe010000-0xfe010fff]
pci 0000:01:00.0: [1137:0209] type 00 class 0x058000 PCIe Endpoint
pci 0000:01:00.0: BAR 0 [mem 0x7e80000000-0x7e9fffffff 64bit pref]
pci 0000:01:00.0: BAR 2 [mem 0x7ea0000000-0x7ea01fffff 64bit pref]
pci 0000:01:00.0: BAR 4 [mem 0x7e00000000-0x7e7fffffff 64bit pref]
pci 0000:00:09.0: PCI bridge to [bus 01-06]
pci 0000:07:00.0: [1137:3852] type 00 class 0x058000 PCIe Endpoint
pci 0000:07:00.0: BAR 0 [mem 0x88100000-0x881fffff]
pci 0000:07:00.0: BAR 1 [mem 0x88000000-0x880fffff]
pci 0000:07:00.0: PME# supported from D0 D3hot
pci 0000:07:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x1 link at 0000:00:0b.0 (capable of 8.000 Gb/s with 2.5 GT/s PCIe x4 link)
pci 0000:00:0b.0: PCI bridge to [bus 07-0c]
pci 0000:0d:00.0: [1137:0209] type 00 class 0x058000 PCIe Endpoint
pci 0000:0d:00.0: BAR 0 [mem 0x7f80000000-0x7f9fffffff 64bit pref]
pci 0000:0d:00.0: BAR 2 [mem 0x7fa0000000-0x7fa01fffff 64bit pref]
pci 0000:0d:00.0: BAR 4 [mem 0x7f00000000-0x7f7fffffff 64bit pref]
pci 0000:00:0e.0: PCI bridge to [bus 0d-12]
pci 0000:13:00.0: [8086:1539] type 00 class 0x020000 PCIe Endpoint
pci 0000:13:00.0: BAR 0 [mem 0x84000000-0x8401ffff]
pci 0000:13:00.0: BAR 2 [io  0x2000-0x201f]
pci 0000:13:00.0: BAR 3 [mem 0x84020000-0x84023fff]
pci 0000:13:00.0: PME# supported from D0 D3hot D3cold
pci 0000:00:10.0: PCI bridge to [bus 13-18]
pci 0000:19:00.0: [8086:15c4] type 00 class 0x020000 PCIe Endpoint
pci 0000:19:00.0: BAR 0 [mem 0x88400000-0x885fffff 64bit pref]
pci 0000:19:00.0: BAR 4 [mem 0x88604000-0x88607fff 64bit pref]
pci 0000:19:00.0: ROM [mem 0x88780000-0x887fffff pref]
pci 0000:19:00.0: PME# supported from D0 D3hot D3cold
pci 0000:19:00.0: VF BAR 0 [mem 0x88a00000-0x88a03fff 64bit]
pci 0000:19:00.0: VF BAR 0 [mem 0x88a00000-0x88afffff 64bit]: contains BAR 0 for 64 VFs
pci 0000:19:00.0: VF BAR 3 [mem 0x88b00000-0x88b03fff 64bit]
pci 0000:19:00.0: VF BAR 3 [mem 0x88b00000-0x88bfffff 64bit]: contains BAR 3 for 64 VFs
pci 0000:19:00.1: [8086:15c4] type 00 class 0x020000 PCIe Endpoint
pci 0000:19:00.1: BAR 0 [mem 0x88200000-0x883fffff 64bit pref]
pci 0000:19:00.1: BAR 4 [mem 0x88600000-0x88603fff 64bit pref]
pci 0000:19:00.1: ROM [mem 0x88700000-0x8877ffff pref]
pci 0000:19:00.1: PME# supported from D0 D3hot D3cold
pci 0000:19:00.1: VF BAR 0 [mem 0x88800000-0x88803fff 64bit]
pci 0000:19:00.1: VF BAR 0 [mem 0x88800000-0x888fffff 64bit]: contains BAR 0 for 64 VFs
pci 0000:19:00.1: VF BAR 3 [mem 0x88900000-0x88903fff 64bit]
pci 0000:19:00.1: VF BAR 3 [mem 0x88900000-0x889fffff 64bit]: contains BAR 3 for 64 VFs
pci 0000:00:16.0: PCI bridge to [bus 19-1a]
ACPI: PCI: Interrupt link LNKA configured for IRQ 0
ACPI: PCI: Interrupt link LNKA disabled
ACPI: PCI: Interrupt link LNKB configured for IRQ 0
ACPI: PCI: Interrupt link LNKB disabled
ACPI: PCI: Interrupt link LNKC configured for IRQ 0
ACPI: PCI: Interrupt link LNKC disabled
ACPI: PCI: Interrupt link LNKD configured for IRQ 0
ACPI: PCI: Interrupt link LNKD disabled
ACPI: PCI: Interrupt link LNKE configured for IRQ 0
ACPI: PCI: Interrupt link LNKE disabled
ACPI: PCI: Interrupt link LNKF configured for IRQ 0
ACPI: PCI: Interrupt link LNKF disabled
ACPI: PCI: Interrupt link LNKG configured for IRQ 0
ACPI: PCI: Interrupt link LNKG disabled
ACPI: PCI: Interrupt link LNKH configured for IRQ 0
ACPI: PCI: Interrupt link LNKH disabled
iommu: Default domain type: Translated
iommu: DMA domain TLB invalidation policy: lazy mode
SCSI subsystem initialized
ACPI: bus type USB registered
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
pps_core: LinuxPPS API ver. 1 registered
pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
PTP clock support registered
EDAC MC: Ver: 3.0.0
efivars: Registered efivars operations
PCI: Using ACPI for IRQ routing
pci 0000:00:1f.1: BAR 0 [mem 0xfd000000-0xfdffffff 64bit]: can't claim; no compatible bridge window
hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0
hpet0: 8 comparators, 64-bit 24.000000 MHz counter
clocksource: Switched to clocksource tsc-early
pnp: PnP ACPI init
system 00:00: [mem 0xe0000000-0xefffffff] has been reserved
system 00:00: [mem 0x7f800000-0x7fffffff] could not be reserved
system 00:00: [mem 0xfed60000-0xfed60fff] has been reserved
system 00:01: [io  0x1800-0x187f] has been reserved
system 00:01: [io  0x0500-0x05fe] has been reserved
system 00:01: [io  0x0680-0x069f] has been reserved
system 00:01: [mem 0xfed00000-0xfed003ff] has been reserved
system 00:01: [mem 0x000c0000-0x000dffff] has been reserved
system 00:01: [mem 0x000e0000-0x000fffff] could not be reserved
system 00:01: [mem 0xff7f0000-0xfffeffff] has been reserved
system 00:06: [io  0x1800-0x18fe] could not be reserved
system 00:06: [mem 0xfed03000-0xfed03fff] has been reserved
pnp: PnP ACPI: found 7 devices
clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
NET: Registered PF_INET protocol family
IP idents hash table entries: 131072 (order: 8, 1048576 bytes, linear)
tcp_listen_portaddr_hash hash table entries: 4096 (order: 4, 65536 bytes, linear)
Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
TCP established hash table entries: 65536 (order: 7, 524288 bytes, linear)
TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, linear)
TCP: Hash tables configured (established 65536 bind 65536)
UDP hash table entries: 4096 (order: 5, 131072 bytes, linear)
UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes, linear)
NET: Registered PF_UNIX/PF_LOCAL protocol family
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp-with-tls transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
NET: Registered PF_XDP protocol family
pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned
pci 0000:00:09.0: PCI bridge to [bus 01-06]
pci 0000:00:09.0:   bridge window [io  0x5000-0x5fff]
pci 0000:00:09.0:   bridge window [mem 0x89000000-0x8affffff]
pci 0000:00:09.0:   bridge window [mem 0x7e00000000-0x7ea01fffff 64bit pref]
pci 0000:00:0b.0: PCI bridge to [bus 07-0c]
pci 0000:00:0b.0:   bridge window [io  0x4000-0x4fff]
pci 0000:00:0b.0:   bridge window [mem 0x87000000-0x881fffff]
pci 0000:00:0b.0:   bridge window [mem 0x81000000-0x81ffffff 64bit pref]
pci 0000:00:0e.0: PCI bridge to [bus 0d-12]
pci 0000:00:0e.0:   bridge window [io  0x3000-0x3fff]
pci 0000:00:0e.0:   bridge window [mem 0x85000000-0x86ffffff]
pci 0000:00:0e.0:   bridge window [mem 0x7f00000000-0x7fa01fffff 64bit pref]
pci 0000:00:10.0: PCI bridge to [bus 13-18]
pci 0000:00:10.0:   bridge window [io  0x2000-0x2fff]
pci 0000:00:10.0:   bridge window [mem 0x83000000-0x840fffff]
pci 0000:00:10.0:   bridge window [mem 0x82000000-0x82ffffff 64bit pref]
pci 0000:00:16.0: PCI bridge to [bus 19-1a]
pci 0000:00:16.0:   bridge window [mem 0x88700000-0x88bfffff]
pci 0000:00:16.0:   bridge window [mem 0x88200000-0x886fffff 64bit pref]
pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
pci_bus 0000:00: resource 5 [io  0x0d00-0xffff window]
pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
pci_bus 0000:00: resource 7 [mem 0x80000000-0xdfffffff window]
pci_bus 0000:00: resource 8 [mem 0x280000000-0x7fffffffff window]
pci_bus 0000:00: resource 9 [mem 0xfe010000-0xfe010fff window]
pci_bus 0000:01: resource 0 [io  0x5000-0x5fff]
pci_bus 0000:01: resource 1 [mem 0x89000000-0x8affffff]
pci_bus 0000:01: resource 2 [mem 0x7e00000000-0x7ea01fffff 64bit pref]
pci_bus 0000:07: resource 0 [io  0x4000-0x4fff]
pci_bus 0000:07: resource 1 [mem 0x87000000-0x881fffff]
pci_bus 0000:07: resource 2 [mem 0x81000000-0x81ffffff 64bit pref]
pci_bus 0000:0d: resource 0 [io  0x3000-0x3fff]
pci_bus 0000:0d: resource 1 [mem 0x85000000-0x86ffffff]
pci_bus 0000:0d: resource 2 [mem 0x7f00000000-0x7fa01fffff 64bit pref]
pci_bus 0000:13: resource 0 [io  0x2000-0x2fff]
pci_bus 0000:13: resource 1 [mem 0x83000000-0x840fffff]
pci_bus 0000:13: resource 2 [mem 0x82000000-0x82ffffff 64bit pref]
pci_bus 0000:19: resource 1 [mem 0x88700000-0x88bfffff]
pci_bus 0000:19: resource 2 [mem 0x88200000-0x886fffff 64bit pref]
PCI: CLS 64 bytes, default 64
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
software IO TLB: mapped [mem 0x000000006b000000-0x000000006f000000] (64MB)
RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 655360 ms ovfl timer
RAPL PMU: hw unit of domain pp0-core 2^-14 Joules
RAPL PMU: hw unit of domain package 2^-14 Joules
RAPL PMU: hw unit of domain dram 2^-14 Joules
clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x1fb633008a4, max_idle_ns: 440795292230 ns
clocksource: Switched to clocksource tsc
Initialise system trusted keyrings
workingset: timestamp_bits=46 max_order=21 bucket_order=0
Key type asymmetric registered
Asymmetric key parser 'x509' registered
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)
io scheduler mq-deadline registered
io scheduler kyber registered
ACPI: \_PR_.CPU0: Found 2 idle states
ACPI: \_PR_.CPU1: Found 2 idle states
ACPI: \_PR_.CPU2: Found 2 idle states
ACPI: \_PR_.CPU3: Found 2 idle states
thermal LNXTHERM:00: registered as thermal_zone0
ACPI: thermal: Thermal Zone [TZ01] (27 C)
Serial: 8250/16550 driver, 2 ports, IRQ sharing disabled
00:03: ttyS0 at I/O 0x3f8 (irq = 16, base_baud = 115200) is a 16550A
00:04: ttyS1 at I/O 0x2f8 (irq = 17, base_baud = 115200) is a 16550A
brd: module loaded
loop: module loaded
tun: Universal TUN/TAP device driver, 1.6
xhci_hcd 0000:00:15.0: xHCI Host Controller
xhci_hcd 0000:00:15.0: new USB bus registered, assigned bus number 1
xhci_hcd 0000:00:15.0: hcc params 0x200077c1 hci version 0x100 quirks 0x0000000001109810
xhci_hcd 0000:00:15.0: xHCI Host Controller
xhci_hcd 0000:00:15.0: new USB bus registered, assigned bus number 2
xhci_hcd 0000:00:15.0: Host supports USB 3.0 SuperSpeed
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 4 ports detected
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 4 ports detected
usbcore: registered new interface driver uas
usbcore: registered new interface driver usb-storage
rtc_cmos 00:02: RTC can wake from S4
rtc_cmos 00:02: registered as rtc0
rtc_cmos 00:02: setting system clock to 2024-11-01T21:37:24 UTC (1730497044)
rtc_cmos 00:02: alarms up to one month, 242 bytes nvram
i2c_dev: i2c /dev entries driver
device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.
device-mapper: ioctl: 4.48.0-ioctl (2023-03-01) initialised: dm-devel@redhat.com
intel_pstate: CPU model not supported
ipip: IPv4 and MPLS over IPv4 tunneling driver
IPv4 over IPsec tunneling driver
Initializing XFRM netlink socket
IPsec XFRM device driver
NET: Registered PF_INET6 protocol family
Segment Routing with IPv6
In-situ OAM (IOAM) with IPv6
NET: Registered PF_PACKET protocol family
Bridge firewalling registered
8021q: 802.1Q VLAN Support v1.8
Key type dns_resolver registered
microcode: Current revision: 0x00000032
IPI shorthand broadcast: enabled
SSE version of gcm_enc/dec engaged.
sched_clock: Marking stable (8264001872, 10805508944)->(33659055178, -14589544362)
Loading compiled-in X.509 certificates
Loaded X.509 cert 'Build time autogenerated kernel key: f5b8cea8a4f0e856859664f338eb05cdfb1f52b8'
ima: No TPM chip found, activating TPM-bypass!
Loading compiled-in module X.509 certificates
Loaded X.509 cert 'Build time autogenerated kernel key: f5b8cea8a4f0e856859664f338eb05cdfb1f52b8'
ima: Allocated hash algorithm: sha1
ima: No architecture policies found
Freeing unused kernel image (initmem) memory: 31740K
Write protecting the kernel read-only data: 20480k
Freeing unused kernel image (rodata/data gap) memory: 1276K
x86/mm: Checked W+X mappings: passed, no W+X pages found.
Run /bin/ash as init process

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Daniel Walker (danielwa)]

On Wed, Nov 13, 2024 at 06:04:44PM +0100, Hans de Goede wrote:
> Hi,
> 
> On 13-Nov-24 5:33 PM, Hans de Goede wrote:
> > Hi,
> > 
> > On 13-Nov-24 5:24 PM, Hans de Goede wrote:
> >> Hi,
> >>
> >> On 13-Nov-24 4:42 PM, Daniel Walker (danielwa) wrote:
> >>>
> >>> Hi,
> >>>
> >>> I bisected an issue of a missing pci device to commit 2841631 the commit title
> >>> in the subject line which was included in v6.1 stable branch.
> >>>
> >>> There was a later fix for a similar missing pci device commit 36c676e2 which
> >>> appears to be for Goldmont/Apollo Lake. The hardware I'm using is
> >>> Goldmont/Denverton. This fix did not appear to change the behavior I'm seeing.
> >>>
> >>> The pci device which is disappearing is a custom gpio device.
> >>>
> >>> I tested v6.12-rc5-next to see if any other changes had fixed the issue, but there was
> >>> no change in behavior since commit 2841631 .
> >>>
> >>> When booting up the device is shown in the pci boot messages but the device
> >>> doesn't end up making it to lspci once you get to a prompt.
> >>
> >> Please give the attached patch a try, this will hopefully fix things.
> >>
> >> Once I have confirmation that this fixes things I'll post it to the list.
> >>
> >> Note this will not backport to the 6.1 stable branch cleanly due to
> >> changes in the x86_cpu_id macros in mainline. Backporting it should
> >> be trivial. Please send a backport to stable@vger.kernel.org yourself
> >> once this has been merged upstream.
> >>
> >> If you backport this, please also backport 36c676e2 first.
> > 
> > Never mind, self nack. This is correct for Gemini Lake which
> > has its SPI at device.function 13.2 like Apollo Lake.
> > 
> > But looking at the dmesg Denverton actually has it at 1f.1
> > aka 31.1 like most other Intel SoCs.
> > 
> > Which make me wonder why this does not work on Denverton.
> > 
> > It probably has something to do with these 2 messages:
> > 
> > pci 0000:00:1f.1: BAR 0 [mem 0xfd000000-0xfdffffff 64bit]: can't claim; no compatible bridge window
> > pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned
> > 
> > I'm guessing that this re-assignment is messing up
> > the p2sb BAR caching, after which things go wrong.
> 
> Hmm, but that should be fixed by 2c6370e66076 ("platform/x86: p2sb: Don't init until unassigned resources have been assigned")
> 
> and you are seeing this with 6.12, which has that.
> 
> Can you try adding a pr_info() to the top of p2sb_cache_resources()
> with 6.12 and then collec a new dmesg ?
> 
> If that pr_info() is done after the:
> 
> pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned
> 
> message then that does not explain things.
> 

I haven't testing adding a pr_info() but the messages seem to happen in the same
order in both working and non-working cases.

Does that matter?

Daniel

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Hans de Goede]

Hi,

On 13-Nov-24 6:41 PM, Daniel Walker (danielwa) wrote:
> On Wed, Nov 13, 2024 at 06:04:44PM +0100, Hans de Goede wrote:
>> Hi,
>>
>> On 13-Nov-24 5:33 PM, Hans de Goede wrote:
>>> Hi,
>>>
>>> On 13-Nov-24 5:24 PM, Hans de Goede wrote:
>>>> Hi,
>>>>
>>>> On 13-Nov-24 4:42 PM, Daniel Walker (danielwa) wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I bisected an issue of a missing pci device to commit 2841631 the commit title
>>>>> in the subject line which was included in v6.1 stable branch.
>>>>>
>>>>> There was a later fix for a similar missing pci device commit 36c676e2 which
>>>>> appears to be for Goldmont/Apollo Lake. The hardware I'm using is
>>>>> Goldmont/Denverton. This fix did not appear to change the behavior I'm seeing.
>>>>>
>>>>> The pci device which is disappearing is a custom gpio device.
>>>>>
>>>>> I tested v6.12-rc5-next to see if any other changes had fixed the issue, but there was
>>>>> no change in behavior since commit 2841631 .
>>>>>
>>>>> When booting up the device is shown in the pci boot messages but the device
>>>>> doesn't end up making it to lspci once you get to a prompt.
>>>>
>>>> Please give the attached patch a try, this will hopefully fix things.
>>>>
>>>> Once I have confirmation that this fixes things I'll post it to the list.
>>>>
>>>> Note this will not backport to the 6.1 stable branch cleanly due to
>>>> changes in the x86_cpu_id macros in mainline. Backporting it should
>>>> be trivial. Please send a backport to stable@vger.kernel.org yourself
>>>> once this has been merged upstream.
>>>>
>>>> If you backport this, please also backport 36c676e2 first.
>>>
>>> Never mind, self nack. This is correct for Gemini Lake which
>>> has its SPI at device.function 13.2 like Apollo Lake.
>>>
>>> But looking at the dmesg Denverton actually has it at 1f.1
>>> aka 31.1 like most other Intel SoCs.
>>>
>>> Which make me wonder why this does not work on Denverton.
>>>
>>> It probably has something to do with these 2 messages:
>>>
>>> pci 0000:00:1f.1: BAR 0 [mem 0xfd000000-0xfdffffff 64bit]: can't claim; no compatible bridge window
>>> pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned
>>>
>>> I'm guessing that this re-assignment is messing up
>>> the p2sb BAR caching, after which things go wrong.
>>
>> Hmm, but that should be fixed by 2c6370e66076 ("platform/x86: p2sb: Don't init until unassigned resources have been assigned")
>>
>> and you are seeing this with 6.12, which has that.
>>
>> Can you try adding a pr_info() to the top of p2sb_cache_resources()
>> with 6.12 and then collec a new dmesg ?
>>
>> If that pr_info() is done after the:
>>
>> pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned
>>
>> message then that does not explain things.
>>
> 
> I haven't testing adding a pr_info() but the messages seem to happen in the same
> order in both working and non-working cases.
> 
> Does that matter?

The working case does not do the bar caching, we want to know if the
bar caching in the non working case happens before or after the assignment:

pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned

It should happen after the assignment.

Regards,

Hans


p.s.

I'm going offline now for a long weekend. So I won't follow-up on
this email-thread until Monday.

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Andy Shevchenko]

On Wed, Nov 13, 2024 at 05:33:41PM +0100, Hans de Goede wrote:
> On 13-Nov-24 5:24 PM, Hans de Goede wrote:
> > On 13-Nov-24 4:42 PM, Daniel Walker (danielwa) wrote:

> >> I bisected an issue of a missing pci device to commit 2841631 the commit title
> >> in the subject line which was included in v6.1 stable branch.
> >>
> >> There was a later fix for a similar missing pci device commit 36c676e2 which
> >> appears to be for Goldmont/Apollo Lake. The hardware I'm using is
> >> Goldmont/Denverton. This fix did not appear to change the behavior I'm seeing.
> >>
> >> The pci device which is disappearing is a custom gpio device.
> >>
> >> I tested v6.12-rc5-next to see if any other changes had fixed the issue, but there was
> >> no change in behavior since commit 2841631 .
> >>
> >> When booting up the device is shown in the pci boot messages but the device
> >> doesn't end up making it to lspci once you get to a prompt.
> > 
> > Please give the attached patch a try, this will hopefully fix things.
> > 
> > Once I have confirmation that this fixes things I'll post it to the list.
> > 
> > Note this will not backport to the 6.1 stable branch cleanly due to
> > changes in the x86_cpu_id macros in mainline. Backporting it should
> > be trivial. Please send a backport to stable@vger.kernel.org yourself
> > once this has been merged upstream.
> > 
> > If you backport this, please also backport 36c676e2 first.
> 
> Never mind, self nack. This is correct for Gemini Lake which
> has its SPI at device.function 13.2 like Apollo Lake.
> 
> But looking at the dmesg Denverton actually has it at 1f.1
> aka 31.1 like most other Intel SoCs.
> 
> Which make me wonder why this does not work on Denverton.
> 
> It probably has something to do with these 2 messages:

> pci 0000:00:1f.1: BAR 0 [mem 0xfd000000-0xfdffffff 64bit]: can't claim; no compatible bridge window
> pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned

As I tried to explain in the very first commit that brings the whole driver
into the kernel the P2SB mimics PCI device but actually doesn't provide all
PCI capabilities. The BAR address there is basically a protocol between
firmware and OS which gives the OS the first (most significant) byte of the
address space window of 16Mb that P2SB responds to in HW. So, I haven't
tested if the relocation actually works for this device, esp. if it goes
over 4G boundary.

That said, messing up with that address is most likely a problematic there.

> I'm guessing that this re-assignment is messing up
> the p2sb BAR caching, after which things go wrong.
> 
> Daniel, are you seeing similar messages with a working kernel ?

-- 
With Best Regards,
Andy Shevchenko

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Shinichiro Kawasaki]

[-- Attachment #1: Type: text/plain, Size: 2003 bytes --]

On Nov 13, 2024 / 19:34, Hans de Goede wrote:
> Hi,
> 
> On 13-Nov-24 6:41 PM, Daniel Walker (danielwa) wrote:
> > On Wed, Nov 13, 2024 at 06:04:44PM +0100, Hans de Goede wrote:
> >> Hi,
> >>
> >> On 13-Nov-24 5:33 PM, Hans de Goede wrote:
> >>> Hi,
> >>>
> >>> On 13-Nov-24 5:24 PM, Hans de Goede wrote:
[...]
> >>> It probably has something to do with these 2 messages:
> >>>
> >>> pci 0000:00:1f.1: BAR 0 [mem 0xfd000000-0xfdffffff 64bit]: can't claim; no compatible bridge window
> >>> pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned
> >>>
> >>> I'm guessing that this re-assignment is messing up
> >>> the p2sb BAR caching, after which things go wrong.
> >>
> >> Hmm, but that should be fixed by 2c6370e66076 ("platform/x86: p2sb: Don't init until unassigned resources have been assigned")
> >>
> >> and you are seeing this with 6.12, which has that.
> >>
> >> Can you try adding a pr_info() to the top of p2sb_cache_resources()
> >> with 6.12 and then collec a new dmesg ?
> >>
> >> If that pr_info() is done after the:
> >>
> >> pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned
> >>
> >> message then that does not explain things.
> >>
> > 
> > I haven't testing adding a pr_info() but the messages seem to happen in the same
> > order in both working and non-working cases.
> > 
> > Does that matter?
> 
> The working case does not do the bar caching, we want to know if the
> bar caching in the non working case happens before or after the assignment:
> 
> pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned
> 
> It should happen after the assignment.

Hello Daniel,

It's my sorrow that the change cause this trouble. I have created a debug patch
for the kernel and attached to this e-mail. It adds some pr_info() to answer
the question from Hans. It will also show us a bit more things. Could you try it
on your system? It should apply to v6.12-rcX kernels without conflicts.


[-- Attachment #2: p2sb_debug.patch --]
[-- Type: text/plain, Size: 1769 bytes --]

diff --git a/drivers/platform/x86/p2sb.c b/drivers/platform/x86/p2sb.c
index 31f38309b389..e9038e8ebc87 100644
--- a/drivers/platform/x86/p2sb.c
+++ b/drivers/platform/x86/p2sb.c
@@ -89,6 +89,12 @@ static void p2sb_scan_and_cache_devfn(struct pci_bus *bus, unsigned int devfn)
 		return;
 
 	p2sb_read_bar0(pdev, &cache->res);
+
+	pr_info("%s: devfn=%x.%x\n", __func__,
+		PCI_SLOT(devfn), PCI_FUNC(devfn));
+	pr_info("%s: %llx-%llx: %lx\n", __func__,
+		cache->res.start, cache->res.end, cache->res.flags);
+
 	cache->bus_dev_id = bus->dev.id;
 
 	pci_stop_and_remove_bus_device(pdev);
@@ -130,6 +136,8 @@ static int p2sb_cache_resources(void)
 	u16 class;
 	int ret;
 
+	pr_info("%s\n", __func__);
+
 	/* Get devfn for P2SB device itself */
 	p2sb_get_devfn(&devfn_p2sb);
 
@@ -157,6 +165,9 @@ static int p2sb_cache_resources(void)
 	 * Unhide the P2SB device here, if needed.
 	 */
 	pci_bus_read_config_dword(bus, devfn_p2sb, P2SBC, &value);
+
+	pr_info("%s: P2SBC_HIDE=%lu\n", __func__, value & P2SBC_HIDE);
+
 	if (value & P2SBC_HIDE)
 		pci_bus_write_config_dword(bus, devfn_p2sb, P2SBC, 0);
 
@@ -189,6 +200,8 @@ int p2sb_bar(struct pci_bus *bus, unsigned int devfn, struct resource *mem)
 {
 	struct p2sb_res_cache *cache;
 
+	pr_info("%s\n", __func__);
+
 	bus = p2sb_get_bus(bus);
 	if (!bus)
 		return -ENODEV;
@@ -204,6 +217,12 @@ int p2sb_bar(struct pci_bus *bus, unsigned int devfn, struct resource *mem)
 		return -ENOENT;
 
 	memcpy(mem, &cache->res, sizeof(*mem));
+
+	pr_info("%s: devfn=%x.%x\n", __func__,
+		PCI_SLOT(devfn), PCI_FUNC(devfn));
+	pr_info("%s: %llx-%llx: %lx\n", __func__,
+		cache->res.start, cache->res.end, cache->res.flags);
+
 	return 0;
 }
 EXPORT_SYMBOL_GPL(p2sb_bar);

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Daniel Walker (danielwa)]

[-- Attachment #1: Type: text/plain, Size: 2247 bytes --]

On Fri, Nov 15, 2024 at 11:35:46AM +0000, Shinichiro Kawasaki wrote:
> On Nov 13, 2024 / 19:34, Hans de Goede wrote:
> > Hi,
> > 
> > On 13-Nov-24 6:41 PM, Daniel Walker (danielwa) wrote:
> > > On Wed, Nov 13, 2024 at 06:04:44PM +0100, Hans de Goede wrote:
> > >> Hi,
> > >>
> > >> On 13-Nov-24 5:33 PM, Hans de Goede wrote:
> > >>> Hi,
> > >>>
> > >>> On 13-Nov-24 5:24 PM, Hans de Goede wrote:
> [...]
> > >>> It probably has something to do with these 2 messages:
> > >>>
> > >>> pci 0000:00:1f.1: BAR 0 [mem 0xfd000000-0xfdffffff 64bit]: can't claim; no compatible bridge window
> > >>> pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned
> > >>>
> > >>> I'm guessing that this re-assignment is messing up
> > >>> the p2sb BAR caching, after which things go wrong.
> > >>
> > >> Hmm, but that should be fixed by 2c6370e66076 ("platform/x86: p2sb: Don't init until unassigned resources have been assigned")
> > >>
> > >> and you are seeing this with 6.12, which has that.
> > >>
> > >> Can you try adding a pr_info() to the top of p2sb_cache_resources()
> > >> with 6.12 and then collec a new dmesg ?
> > >>
> > >> If that pr_info() is done after the:
> > >>
> > >> pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned
> > >>
> > >> message then that does not explain things.
> > >>
> > > 
> > > I haven't testing adding a pr_info() but the messages seem to happen in the same
> > > order in both working and non-working cases.
> > > 
> > > Does that matter?
> > 
> > The working case does not do the bar caching, we want to know if the
> > bar caching in the non working case happens before or after the assignment:
> > 
> > pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned
> > 
> > It should happen after the assignment.
> 
> Hello Daniel,
> 
> It's my sorrow that the change cause this trouble. I have created a debug patch
> for the kernel and attached to this e-mail. It adds some pr_info() to answer
> the question from Hans. It will also show us a bit more things. Could you try it
> on your system? It should apply to v6.12-rcX kernels without conflicts.
> 

Ok.. The dmesg with the patch applied is attached.

Daniel



[-- Attachment #2: dmesg-debug.txt.gz --]
[-- Type: application/gzip, Size: 9079 bytes --]

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Hans de Goede]

Hi Andy,

On 13-Nov-24 8:17 PM, Andy Shevchenko wrote:
> On Wed, Nov 13, 2024 at 05:33:41PM +0100, Hans de Goede wrote:
>> On 13-Nov-24 5:24 PM, Hans de Goede wrote:
>>> On 13-Nov-24 4:42 PM, Daniel Walker (danielwa) wrote:
> 
>>>> I bisected an issue of a missing pci device to commit 2841631 the commit title
>>>> in the subject line which was included in v6.1 stable branch.
>>>>
>>>> There was a later fix for a similar missing pci device commit 36c676e2 which
>>>> appears to be for Goldmont/Apollo Lake. The hardware I'm using is
>>>> Goldmont/Denverton. This fix did not appear to change the behavior I'm seeing.
>>>>
>>>> The pci device which is disappearing is a custom gpio device.
>>>>
>>>> I tested v6.12-rc5-next to see if any other changes had fixed the issue, but there was
>>>> no change in behavior since commit 2841631 .
>>>>
>>>> When booting up the device is shown in the pci boot messages but the device
>>>> doesn't end up making it to lspci once you get to a prompt.
>>>
>>> Please give the attached patch a try, this will hopefully fix things.
>>>
>>> Once I have confirmation that this fixes things I'll post it to the list.
>>>
>>> Note this will not backport to the 6.1 stable branch cleanly due to
>>> changes in the x86_cpu_id macros in mainline. Backporting it should
>>> be trivial. Please send a backport to stable@vger.kernel.org yourself
>>> once this has been merged upstream.
>>>
>>> If you backport this, please also backport 36c676e2 first.
>>
>> Never mind, self nack. This is correct for Gemini Lake which
>> has its SPI at device.function 13.2 like Apollo Lake.
>>
>> But looking at the dmesg Denverton actually has it at 1f.1
>> aka 31.1 like most other Intel SoCs.
>>
>> Which make me wonder why this does not work on Denverton.
>>
>> It probably has something to do with these 2 messages:
> 
>> pci 0000:00:1f.1: BAR 0 [mem 0xfd000000-0xfdffffff 64bit]: can't claim; no compatible bridge window
>> pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned
> 
> As I tried to explain in the very first commit that brings the whole driver
> into the kernel the P2SB mimics PCI device but actually doesn't provide all
> PCI capabilities. The BAR address there is basically a protocol between
> firmware and OS which gives the OS the first (most significant) byte of the
> address space window of 16Mb that P2SB responds to in HW. So, I haven't
> tested if the relocation actually works for this device, esp. if it goes
> over 4G boundary.
> 
> That said, messing up with that address is most likely a problematic there.

The relocation also happens in the provided working dmesg. I agree that
the relocation is weird, but that does not appear to be the issue.

At least it was not an issue before we switched to caching the bar
returned by p2sb_bar() early on so that p2sb_bar() does not need
to take looks.

Regards,

Hans

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Andy Shevchenko]

On Sat, Nov 16, 2024 at 12:34:54PM +0100, Hans de Goede wrote:
> On 13-Nov-24 8:17 PM, Andy Shevchenko wrote:
> > On Wed, Nov 13, 2024 at 05:33:41PM +0100, Hans de Goede wrote:

...

> > That said, messing up with that address is most likely a problematic there.
> 
> The relocation also happens in the provided working dmesg. I agree that
> the relocation is weird, but that does not appear to be the issue.

Hmm... I'm then wondering why we can't just unhide the device once at the early
boot (via [x86 specific] PCI quirk) and live with that... My most worries were
about relocation, but currently reading the documentations I see that this
shouldn't be a problem as the lowest 3 bytes define the target address on the
backbone anyway. I.o.w. it shouldn't matter what is written in the bits 63..24.

And FTR, I haven't found any specifics for Denverton, it's just an early
generation of P2SB RTL, but documentation says it should work as the others
on a basic levels (like what we expect from it in the Linux kernel).

> At least it was not an issue before we switched to caching the bar
> returned by p2sb_bar() early on so that p2sb_bar() does not need
> to take looks.

Anyway, it's easy to check just by caching the initial value of 0xfd000000
and see if it helps.

Another point here is the power management. Is there any PM handling during
the resource (re-)assignment? It might put the whole piece to D3hot and make
it not working. But I'm purely speculating here...

-- 
With Best Regards,
Andy Shevchenko

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Shinichiro Kawasaki]

On Nov 15, 2024 / 14:57, Daniel Walker (danielwa) wrote:
> On Fri, Nov 15, 2024 at 11:35:46AM +0000, Shinichiro Kawasaki wrote:
> > On Nov 13, 2024 / 19:34, Hans de Goede wrote:
> > > Hi,
> > > 
> > > On 13-Nov-24 6:41 PM, Daniel Walker (danielwa) wrote:
> > > > On Wed, Nov 13, 2024 at 06:04:44PM +0100, Hans de Goede wrote:
> > > >> Hi,
> > > >>
> > > >> On 13-Nov-24 5:33 PM, Hans de Goede wrote:
> > > >>> Hi,
> > > >>>
> > > >>> On 13-Nov-24 5:24 PM, Hans de Goede wrote:
> > [...]
> > > >>> It probably has something to do with these 2 messages:
> > > >>>
> > > >>> pci 0000:00:1f.1: BAR 0 [mem 0xfd000000-0xfdffffff 64bit]: can't claim; no compatible bridge window
> > > >>> pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned
> > > >>>
> > > >>> I'm guessing that this re-assignment is messing up
> > > >>> the p2sb BAR caching, after which things go wrong.
> > > >>
> > > >> Hmm, but that should be fixed by 2c6370e66076 ("platform/x86: p2sb: Don't init until unassigned resources have been assigned")
> > > >>
> > > >> and you are seeing this with 6.12, which has that.
> > > >>
> > > >> Can you try adding a pr_info() to the top of p2sb_cache_resources()
> > > >> with 6.12 and then collec a new dmesg ?
> > > >>
> > > >> If that pr_info() is done after the:
> > > >>
> > > >> pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned
> > > >>
> > > >> message then that does not explain things.
> > > >>
> > > > 
> > > > I haven't testing adding a pr_info() but the messages seem to happen in the same
> > > > order in both working and non-working cases.
> > > > 
> > > > Does that matter?
> > > 
> > > The working case does not do the bar caching, we want to know if the
> > > bar caching in the non working case happens before or after the assignment:
> > > 
> > > pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned
> > > 
> > > It should happen after the assignment.
> > 
> > Hello Daniel,
> > 
> > It's my sorrow that the change cause this trouble. I have created a debug patch
> > for the kernel and attached to this e-mail. It adds some pr_info() to answer
> > the question from Hans. It will also show us a bit more things. Could you try it
> > on your system? It should apply to v6.12-rcX kernels without conflicts.
> > 
> 
> Ok.. The dmesg with the patch applied is attached.

Thank you. Here I quote the relevant part of the debug log.

--------------------------------------------------------------------------------
...
pci 0000:00:1f.0: [8086:19dc] type 00 class 0x060100 conventional PCI endpoint
pci 0000:00:1f.1: [8086:19dd] type 00 class 0x058000 conventional PCI endpoint ... [A]
pci 0000:00:1f.1: BAR 0 [mem 0xfd000000-0xfdffffff 64bit]
pci 0000:00:1f.2: [8086:19de] type 00 class 0x058000 conventional PCI endpoint
pci 0000:00:1f.2: BAR 0 [mem 0x88c00000-0x88c03fff]
...
PCI: Using ACPI for IRQ routing
pci 0000:00:1f.1: BAR 0 [mem 0xfd000000-0xfdffffff 64bit]: can't claim; no compatible bridge window ... [B]
hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0
...
NET: Registered PF_XDP protocol family
pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned ... [C]
pci 0000:00:09.0: PCI bridge to [bus 01-06]
...
PCI: CLS 64 bytes, default 64
p2sb_cache_resources
p2sb_cache_resources: P2SBC_HIDE=0  ... [D]
p2sb_scan_and_cache_devfn: devfn=1f.1
p2sb_scan_and_cache_devfn: 280000000-280ffffff: 140204 ... [E]
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
...
--------------------------------------------------------------------------------

Also, here I list my observations.

[A] The P2SB device was detected with DEVFN 1f.1, and device id 8086:19dd
[B] Failed to claim its resource
[C] Assigned new resource
[D] p2sb_cache_resource() was called after the new resource assignment.
    P2SBC_HIDE bit is not set.
[E] The new resource was cached. IORESOURCE flags: MEM_64,SIZE_ALIGN,MEM.

So it was confirmed that the p2sb_cache_resource() was called after the new
resource assignment, but Hans and Andy discuss that this order is not the
problem cause, probably.

One thing I noticed is that p2sb_bar() call is not recorded in the log. My
understanding is that all device drivers which use P2SB resource shouled call
p2sb_bar(). Daniel, you noted that "a custom gpio device" disappeared. Does its
device driver call p2sb_bar()?

On the other hand, Daniel noted that,

  "The vendor and device details for the pci device are 8086:19dd."

I think 8086:19dd is the P2SB device itself. When p2sb_cache_resource() is
called, pci_stop_and_remove_bus_device() is called for it, so I guess it is
expected the device 8086:19dd disappears. Before applying the commit
5913320eb0b3, this pci_stop_and_remove_bus_device() call happened when
p2sb_bar() was called. So, my mere guess is that Daniel's system's drivers do
not call p2sb_bar() during the boot process, then the 8086:19dd P2SB device was
still visible after boot.

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Hans de Goede]

Hi Shinichiro,

On 18-Nov-24 12:30 PM, Shinichiro Kawasaki wrote:
> On Nov 15, 2024 / 14:57, Daniel Walker (danielwa) wrote:
>> On Fri, Nov 15, 2024 at 11:35:46AM +0000, Shinichiro Kawasaki wrote:
>>> On Nov 13, 2024 / 19:34, Hans de Goede wrote:
>>>> Hi,
>>>>
>>>> On 13-Nov-24 6:41 PM, Daniel Walker (danielwa) wrote:
>>>>> On Wed, Nov 13, 2024 at 06:04:44PM +0100, Hans de Goede wrote:
>>>>>> Hi,
>>>>>>
>>>>>> On 13-Nov-24 5:33 PM, Hans de Goede wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> On 13-Nov-24 5:24 PM, Hans de Goede wrote:
>>> [...]
>>>>>>> It probably has something to do with these 2 messages:
>>>>>>>
>>>>>>> pci 0000:00:1f.1: BAR 0 [mem 0xfd000000-0xfdffffff 64bit]: can't claim; no compatible bridge window
>>>>>>> pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned
>>>>>>>
>>>>>>> I'm guessing that this re-assignment is messing up
>>>>>>> the p2sb BAR caching, after which things go wrong.
>>>>>>
>>>>>> Hmm, but that should be fixed by 2c6370e66076 ("platform/x86: p2sb: Don't init until unassigned resources have been assigned")
>>>>>>
>>>>>> and you are seeing this with 6.12, which has that.
>>>>>>
>>>>>> Can you try adding a pr_info() to the top of p2sb_cache_resources()
>>>>>> with 6.12 and then collec a new dmesg ?
>>>>>>
>>>>>> If that pr_info() is done after the:
>>>>>>
>>>>>> pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned
>>>>>>
>>>>>> message then that does not explain things.
>>>>>>
>>>>>
>>>>> I haven't testing adding a pr_info() but the messages seem to happen in the same
>>>>> order in both working and non-working cases.
>>>>>
>>>>> Does that matter?
>>>>
>>>> The working case does not do the bar caching, we want to know if the
>>>> bar caching in the non working case happens before or after the assignment:
>>>>
>>>> pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned
>>>>
>>>> It should happen after the assignment.
>>>
>>> Hello Daniel,
>>>
>>> It's my sorrow that the change cause this trouble. I have created a debug patch
>>> for the kernel and attached to this e-mail. It adds some pr_info() to answer
>>> the question from Hans. It will also show us a bit more things. Could you try it
>>> on your system? It should apply to v6.12-rcX kernels without conflicts.
>>>
>>
>> Ok.. The dmesg with the patch applied is attached.
> 
> Thank you. Here I quote the relevant part of the debug log.
> 
> --------------------------------------------------------------------------------
> ...
> pci 0000:00:1f.0: [8086:19dc] type 00 class 0x060100 conventional PCI endpoint
> pci 0000:00:1f.1: [8086:19dd] type 00 class 0x058000 conventional PCI endpoint ... [A]
> pci 0000:00:1f.1: BAR 0 [mem 0xfd000000-0xfdffffff 64bit]
> pci 0000:00:1f.2: [8086:19de] type 00 class 0x058000 conventional PCI endpoint
> pci 0000:00:1f.2: BAR 0 [mem 0x88c00000-0x88c03fff]
> ...
> PCI: Using ACPI for IRQ routing
> pci 0000:00:1f.1: BAR 0 [mem 0xfd000000-0xfdffffff 64bit]: can't claim; no compatible bridge window ... [B]
> hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0
> ...
> NET: Registered PF_XDP protocol family
> pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned ... [C]
> pci 0000:00:09.0: PCI bridge to [bus 01-06]
> ...
> PCI: CLS 64 bytes, default 64
> p2sb_cache_resources
> p2sb_cache_resources: P2SBC_HIDE=0  ... [D]
> p2sb_scan_and_cache_devfn: devfn=1f.1
> p2sb_scan_and_cache_devfn: 280000000-280ffffff: 140204 ... [E]
> PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
> ...
> --------------------------------------------------------------------------------
> 
> Also, here I list my observations.
> 
> [A] The P2SB device was detected with DEVFN 1f.1, and device id 8086:19dd
> [B] Failed to claim its resource
> [C] Assigned new resource
> [D] p2sb_cache_resource() was called after the new resource assignment.
>     P2SBC_HIDE bit is not set.
> [E] The new resource was cached. IORESOURCE flags: MEM_64,SIZE_ALIGN,MEM.
> 
> So it was confirmed that the p2sb_cache_resource() was called after the new
> resource assignment, but Hans and Andy discuss that this order is not the
> problem cause, probably.
> 
> One thing I noticed is that p2sb_bar() call is not recorded in the log. My
> understanding is that all device drivers which use P2SB resource shouled call
> p2sb_bar(). Daniel, you noted that "a custom gpio device" disappeared. Does its
> device driver call p2sb_bar()?
> 
> On the other hand, Daniel noted that,
> 
>   "The vendor and device details for the pci device are 8086:19dd."
> 
> I think 8086:19dd is the P2SB device itself. When p2sb_cache_resource() is
> called, pci_stop_and_remove_bus_device() is called for it, so I guess it is
> expected the device 8086:19dd disappears. Before applying the commit
> 5913320eb0b3, this pci_stop_and_remove_bus_device() call happened when
> p2sb_bar() was called. So, my mere guess is that Daniel's system's drivers do
> not call p2sb_bar() during the boot process, then the 8086:19dd P2SB device was
> still visible after boot.

Thank you that is a great analysis.

It sounds to me like calling pci_stop_and_remove_bus_device() on an unhidden
PCI device is the real problem here. And that we were not hitting that before
was more or less luck.

I wonder if we can cache the bar in some other way, or even delay retrieving
it till p2sb_bar() call time in the case when it is not hidden ?

Regards,

Hans

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Andy Shevchenko]

On Mon, Nov 18, 2024 at 12:42:18PM +0100, Hans de Goede wrote:
> On 18-Nov-24 12:30 PM, Shinichiro Kawasaki wrote:
> > On Nov 15, 2024 / 14:57, Daniel Walker (danielwa) wrote:
> >> On Fri, Nov 15, 2024 at 11:35:46AM +0000, Shinichiro Kawasaki wrote:
> >>> On Nov 13, 2024 / 19:34, Hans de Goede wrote:
> >>>> On 13-Nov-24 6:41 PM, Daniel Walker (danielwa) wrote:
> >>>>> On Wed, Nov 13, 2024 at 06:04:44PM +0100, Hans de Goede wrote:
> >>>>>> On 13-Nov-24 5:33 PM, Hans de Goede wrote:
> >>>>>>> On 13-Nov-24 5:24 PM, Hans de Goede wrote:

[...]

> >>>>>>> It probably has something to do with these 2 messages:
> >>>>>>>
> >>>>>>> pci 0000:00:1f.1: BAR 0 [mem 0xfd000000-0xfdffffff 64bit]: can't claim; no compatible bridge window
> >>>>>>> pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned
> >>>>>>>
> >>>>>>> I'm guessing that this re-assignment is messing up
> >>>>>>> the p2sb BAR caching, after which things go wrong.
> >>>>>>
> >>>>>> Hmm, but that should be fixed by 2c6370e66076 ("platform/x86: p2sb: Don't init until unassigned resources have been assigned")
> >>>>>>
> >>>>>> and you are seeing this with 6.12, which has that.
> >>>>>>
> >>>>>> Can you try adding a pr_info() to the top of p2sb_cache_resources()
> >>>>>> with 6.12 and then collec a new dmesg ?
> >>>>>>
> >>>>>> If that pr_info() is done after the:
> >>>>>>
> >>>>>> pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned
> >>>>>>
> >>>>>> message then that does not explain things.
> >>>>>
> >>>>> I haven't testing adding a pr_info() but the messages seem to happen in the same
> >>>>> order in both working and non-working cases.
> >>>>>
> >>>>> Does that matter?
> >>>>
> >>>> The working case does not do the bar caching, we want to know if the
> >>>> bar caching in the non working case happens before or after the assignment:
> >>>>
> >>>> pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned
> >>>>
> >>>> It should happen after the assignment.
> >>>
> >>> Hello Daniel,
> >>>
> >>> It's my sorrow that the change cause this trouble. I have created a debug patch
> >>> for the kernel and attached to this e-mail. It adds some pr_info() to answer
> >>> the question from Hans. It will also show us a bit more things. Could you try it
> >>> on your system? It should apply to v6.12-rcX kernels without conflicts.
> >>
> >> Ok.. The dmesg with the patch applied is attached.
> > 
> > Thank you. Here I quote the relevant part of the debug log.
> > 
> > --------------------------------------------------------------------------------
> > ...
> > pci 0000:00:1f.0: [8086:19dc] type 00 class 0x060100 conventional PCI endpoint
> > pci 0000:00:1f.1: [8086:19dd] type 00 class 0x058000 conventional PCI endpoint ... [A]
> > pci 0000:00:1f.1: BAR 0 [mem 0xfd000000-0xfdffffff 64bit]
> > pci 0000:00:1f.2: [8086:19de] type 00 class 0x058000 conventional PCI endpoint
> > pci 0000:00:1f.2: BAR 0 [mem 0x88c00000-0x88c03fff]
> > ...
> > PCI: Using ACPI for IRQ routing
> > pci 0000:00:1f.1: BAR 0 [mem 0xfd000000-0xfdffffff 64bit]: can't claim; no compatible bridge window ... [B]
> > hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0
> > ...
> > NET: Registered PF_XDP protocol family
> > pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned ... [C]
> > pci 0000:00:09.0: PCI bridge to [bus 01-06]
> > ...
> > PCI: CLS 64 bytes, default 64
> > p2sb_cache_resources
> > p2sb_cache_resources: P2SBC_HIDE=0  ... [D]
> > p2sb_scan_and_cache_devfn: devfn=1f.1
> > p2sb_scan_and_cache_devfn: 280000000-280ffffff: 140204 ... [E]
> > PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
> > ...
> > --------------------------------------------------------------------------------
> > 
> > Also, here I list my observations.
> > 
> > [A] The P2SB device was detected with DEVFN 1f.1, and device id 8086:19dd
> > [B] Failed to claim its resource
> > [C] Assigned new resource
> > [D] p2sb_cache_resource() was called after the new resource assignment.
> >     P2SBC_HIDE bit is not set.
> > [E] The new resource was cached. IORESOURCE flags: MEM_64,SIZE_ALIGN,MEM.
> > 
> > So it was confirmed that the p2sb_cache_resource() was called after the new
> > resource assignment, but Hans and Andy discuss that this order is not the
> > problem cause, probably.
> > 
> > One thing I noticed is that p2sb_bar() call is not recorded in the log. My
> > understanding is that all device drivers which use P2SB resource shouled call
> > p2sb_bar(). Daniel, you noted that "a custom gpio device" disappeared. Does its
> > device driver call p2sb_bar()?

FYI: if we are talking about on-die GPIO (that's usually covered by
the drivers/intel/pinctrl-denverton.c), the new kernels have the piece
of the code to enumerate it via p2sb (the code is located in the
drivers/mfd/lpc_ich.c).

> > On the other hand, Daniel noted that,
> > 
> >   "The vendor and device details for the pci device are 8086:19dd."
> > 
> > I think 8086:19dd is the P2SB device itself. When p2sb_cache_resource() is
> > called, pci_stop_and_remove_bus_device() is called for it, so I guess it is
> > expected the device 8086:19dd disappears. Before applying the commit
> > 5913320eb0b3, this pci_stop_and_remove_bus_device() call happened when
> > p2sb_bar() was called. So, my mere guess is that Daniel's system's drivers do
> > not call p2sb_bar() during the boot process, then the 8086:19dd P2SB device was
> > still visible after boot.
> 
> Thank you that is a great analysis.

+1 here, indeed a great analysis!

> It sounds to me like calling pci_stop_and_remove_bus_device() on an unhidden
> PCI device is the real problem here. And that we were not hitting that before
> was more or less luck.
> 
> I wonder if we can cache the bar in some other way, or even delay retrieving
> it till p2sb_bar() call time in the case when it is not hidden ?

-- 
With Best Regards,
Andy Shevchenko

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Daniel Walker (danielwa)]

On Mon, Nov 18, 2024 at 11:30:59AM +0000, Shinichiro Kawasaki wrote:
> 
> Thank you. Here I quote the relevant part of the debug log.
> 
> --------------------------------------------------------------------------------
> ...
> pci 0000:00:1f.0: [8086:19dc] type 00 class 0x060100 conventional PCI endpoint
> pci 0000:00:1f.1: [8086:19dd] type 00 class 0x058000 conventional PCI endpoint ... [A]
> pci 0000:00:1f.1: BAR 0 [mem 0xfd000000-0xfdffffff 64bit]
> pci 0000:00:1f.2: [8086:19de] type 00 class 0x058000 conventional PCI endpoint
> pci 0000:00:1f.2: BAR 0 [mem 0x88c00000-0x88c03fff]
> ...
> PCI: Using ACPI for IRQ routing
> pci 0000:00:1f.1: BAR 0 [mem 0xfd000000-0xfdffffff 64bit]: can't claim; no compatible bridge window ... [B]
> hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0
> ...
> NET: Registered PF_XDP protocol family
> pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned ... [C]
> pci 0000:00:09.0: PCI bridge to [bus 01-06]
> ...
> PCI: CLS 64 bytes, default 64
> p2sb_cache_resources
> p2sb_cache_resources: P2SBC_HIDE=0  ... [D]
> p2sb_scan_and_cache_devfn: devfn=1f.1
> p2sb_scan_and_cache_devfn: 280000000-280ffffff: 140204 ... [E]
> PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
> ...
> --------------------------------------------------------------------------------
> 
> Also, here I list my observations.
> 
> [A] The P2SB device was detected with DEVFN 1f.1, and device id 8086:19dd
> [B] Failed to claim its resource
> [C] Assigned new resource
> [D] p2sb_cache_resource() was called after the new resource assignment.
>     P2SBC_HIDE bit is not set.
> [E] The new resource was cached. IORESOURCE flags: MEM_64,SIZE_ALIGN,MEM.
> 
> So it was confirmed that the p2sb_cache_resource() was called after the new
> resource assignment, but Hans and Andy discuss that this order is not the
> problem cause, probably.
> 
> One thing I noticed is that p2sb_bar() call is not recorded in the log. My
> understanding is that all device drivers which use P2SB resource shouled call
> p2sb_bar(). Daniel, you noted that "a custom gpio device" disappeared. Does its
> device driver call p2sb_bar()?

No calls to p2sb_bar(). It maybe an Intel GPIO device actually. 0x8086 is Intel's vendor code.
I suspect it's something standard on Denverton.

> On the other hand, Daniel noted that,
> 
>   "The vendor and device details for the pci device are 8086:19dd."
> 
> I think 8086:19dd is the P2SB device itself. When p2sb_cache_resource() is
> called, pci_stop_and_remove_bus_device() is called for it, so I guess it is
> expected the device 8086:19dd disappears. Before applying the commit
> 5913320eb0b3, this pci_stop_and_remove_bus_device() call happened when
> p2sb_bar() was called. So, my mere guess is that Daniel's system's drivers do
> not call p2sb_bar() during the boot process, then the 8086:19dd P2SB device was
> still visible after boot.

The method I used for the bisect was to check lspci for 19dd. I find it odd that
the device which does exist was not in lspci. Also pci_get_device() fails for
this device and gpio operations can't happen.

Daniel

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Andy Shevchenko]

On Mon, Nov 18, 2024 at 12:40:16PM +0000, Daniel Walker (danielwa) wrote:
> On Mon, Nov 18, 2024 at 11:30:59AM +0000, Shinichiro Kawasaki wrote:
> > 
> > Thank you. Here I quote the relevant part of the debug log.
> > 
> > --------------------------------------------------------------------------------
> > ...
> > pci 0000:00:1f.0: [8086:19dc] type 00 class 0x060100 conventional PCI endpoint
> > pci 0000:00:1f.1: [8086:19dd] type 00 class 0x058000 conventional PCI endpoint ... [A]
> > pci 0000:00:1f.1: BAR 0 [mem 0xfd000000-0xfdffffff 64bit]
> > pci 0000:00:1f.2: [8086:19de] type 00 class 0x058000 conventional PCI endpoint
> > pci 0000:00:1f.2: BAR 0 [mem 0x88c00000-0x88c03fff]
> > ...
> > PCI: Using ACPI for IRQ routing
> > pci 0000:00:1f.1: BAR 0 [mem 0xfd000000-0xfdffffff 64bit]: can't claim; no compatible bridge window ... [B]
> > hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0
> > ...
> > NET: Registered PF_XDP protocol family
> > pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned ... [C]
> > pci 0000:00:09.0: PCI bridge to [bus 01-06]
> > ...
> > PCI: CLS 64 bytes, default 64
> > p2sb_cache_resources
> > p2sb_cache_resources: P2SBC_HIDE=0  ... [D]
> > p2sb_scan_and_cache_devfn: devfn=1f.1
> > p2sb_scan_and_cache_devfn: 280000000-280ffffff: 140204 ... [E]
> > PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
> > ...
> > --------------------------------------------------------------------------------
> > 
> > Also, here I list my observations.
> > 
> > [A] The P2SB device was detected with DEVFN 1f.1, and device id 8086:19dd
> > [B] Failed to claim its resource
> > [C] Assigned new resource
> > [D] p2sb_cache_resource() was called after the new resource assignment.
> >     P2SBC_HIDE bit is not set.
> > [E] The new resource was cached. IORESOURCE flags: MEM_64,SIZE_ALIGN,MEM.
> > 
> > So it was confirmed that the p2sb_cache_resource() was called after the new
> > resource assignment, but Hans and Andy discuss that this order is not the
> > problem cause, probably.
> > 
> > One thing I noticed is that p2sb_bar() call is not recorded in the log. My
> > understanding is that all device drivers which use P2SB resource shouled call
> > p2sb_bar(). Daniel, you noted that "a custom gpio device" disappeared. Does its
> > device driver call p2sb_bar()?
> 
> No calls to p2sb_bar(). It maybe an Intel GPIO device actually. 0x8086 is Intel's vendor code.
> I suspect it's something standard on Denverton.

Are you referring to LPC GPIO?

> > On the other hand, Daniel noted that,
> > 
> >   "The vendor and device details for the pci device are 8086:19dd."
> > 
> > I think 8086:19dd is the P2SB device itself. When p2sb_cache_resource() is
> > called, pci_stop_and_remove_bus_device() is called for it, so I guess it is
> > expected the device 8086:19dd disappears. Before applying the commit
> > 5913320eb0b3, this pci_stop_and_remove_bus_device() call happened when
> > p2sb_bar() was called. So, my mere guess is that Daniel's system's drivers do
> > not call p2sb_bar() during the boot process, then the 8086:19dd P2SB device was
> > still visible after boot.
> 
> The method I used for the bisect was to check lspci for 19dd. I find it odd that
> the device which does exist was not in lspci. Also pci_get_device() fails for
> this device and gpio operations can't happen.

So, it seems like we have avoid removing PCI devices if the P2SB has been
unhidden during the cache call. Still wondering if we simply unhide it early
(if it was hidden) and assume the p2sb device is always present. In this case
p2sb_bar() will become almost an equivalent to the pci_resource_start().

-- 
With Best Regards,
Andy Shevchenko

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Hans de Goede]

Hi,

On 18-Nov-24 2:24 PM, Andy Shevchenko wrote:
> On Mon, Nov 18, 2024 at 12:40:16PM +0000, Daniel Walker (danielwa) wrote:
>> On Mon, Nov 18, 2024 at 11:30:59AM +0000, Shinichiro Kawasaki wrote:
>>>
>>> Thank you. Here I quote the relevant part of the debug log.
>>>
>>> --------------------------------------------------------------------------------
>>> ...
>>> pci 0000:00:1f.0: [8086:19dc] type 00 class 0x060100 conventional PCI endpoint
>>> pci 0000:00:1f.1: [8086:19dd] type 00 class 0x058000 conventional PCI endpoint ... [A]
>>> pci 0000:00:1f.1: BAR 0 [mem 0xfd000000-0xfdffffff 64bit]
>>> pci 0000:00:1f.2: [8086:19de] type 00 class 0x058000 conventional PCI endpoint
>>> pci 0000:00:1f.2: BAR 0 [mem 0x88c00000-0x88c03fff]
>>> ...
>>> PCI: Using ACPI for IRQ routing
>>> pci 0000:00:1f.1: BAR 0 [mem 0xfd000000-0xfdffffff 64bit]: can't claim; no compatible bridge window ... [B]
>>> hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0
>>> ...
>>> NET: Registered PF_XDP protocol family
>>> pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned ... [C]
>>> pci 0000:00:09.0: PCI bridge to [bus 01-06]
>>> ...
>>> PCI: CLS 64 bytes, default 64
>>> p2sb_cache_resources
>>> p2sb_cache_resources: P2SBC_HIDE=0  ... [D]
>>> p2sb_scan_and_cache_devfn: devfn=1f.1
>>> p2sb_scan_and_cache_devfn: 280000000-280ffffff: 140204 ... [E]
>>> PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
>>> ...
>>> --------------------------------------------------------------------------------
>>>
>>> Also, here I list my observations.
>>>
>>> [A] The P2SB device was detected with DEVFN 1f.1, and device id 8086:19dd
>>> [B] Failed to claim its resource
>>> [C] Assigned new resource
>>> [D] p2sb_cache_resource() was called after the new resource assignment.
>>>     P2SBC_HIDE bit is not set.
>>> [E] The new resource was cached. IORESOURCE flags: MEM_64,SIZE_ALIGN,MEM.
>>>
>>> So it was confirmed that the p2sb_cache_resource() was called after the new
>>> resource assignment, but Hans and Andy discuss that this order is not the
>>> problem cause, probably.
>>>
>>> One thing I noticed is that p2sb_bar() call is not recorded in the log. My
>>> understanding is that all device drivers which use P2SB resource shouled call
>>> p2sb_bar(). Daniel, you noted that "a custom gpio device" disappeared. Does its
>>> device driver call p2sb_bar()?
>>
>> No calls to p2sb_bar(). It maybe an Intel GPIO device actually. 0x8086 is Intel's vendor code.
>> I suspect it's something standard on Denverton.
> 
> Are you referring to LPC GPIO?
> 
>>> On the other hand, Daniel noted that,
>>>
>>>   "The vendor and device details for the pci device are 8086:19dd."
>>>
>>> I think 8086:19dd is the P2SB device itself. When p2sb_cache_resource() is
>>> called, pci_stop_and_remove_bus_device() is called for it, so I guess it is
>>> expected the device 8086:19dd disappears. Before applying the commit
>>> 5913320eb0b3, this pci_stop_and_remove_bus_device() call happened when
>>> p2sb_bar() was called. So, my mere guess is that Daniel's system's drivers do
>>> not call p2sb_bar() during the boot process, then the 8086:19dd P2SB device was
>>> still visible after boot.
>>
>> The method I used for the bisect was to check lspci for 19dd. I find it odd that
>> the device which does exist was not in lspci. Also pci_get_device() fails for
>> this device and gpio operations can't happen.
> 
> So, it seems like we have avoid removing PCI devices if the P2SB has been
> unhidden during the cache call. Still wondering if we simply unhide it early
> (if it was hidden) and assume the p2sb device is always present. In this case
> p2sb_bar() will become almost an equivalent to the pci_resource_start().

That would basically open it up for normal PCI drivers to bind to. Which is not
necessarily a problem, but I wonder if we already have such drivers which then
all of a sudden will start binding to it ?

Note that e.g. the GPIO code in lpc_ich.c checks if the GPIO controller is not
ACPI enumerated before calling p2sb_bar(). Unconditionally unhiding the P2SB
would not mean that at least the GPIO part will be enumerated twice, once
through ACPI and once through PCI. I'm not sure if that is a good idea.

My vote would go to fix the problem of the PCI device being removed by
the p2sb.c code when it was not hidden.

Regards,

Hans

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Daniel Walker (danielwa)]

On Mon, Nov 18, 2024 at 03:24:20PM +0200, Andy Shevchenko wrote:
> On Mon, Nov 18, 2024 at 12:40:16PM +0000, Daniel Walker (danielwa) wrote:
> > On Mon, Nov 18, 2024 at 11:30:59AM +0000, Shinichiro Kawasaki wrote:
> > > 
> > > Thank you. Here I quote the relevant part of the debug log.
> > > 
> > > --------------------------------------------------------------------------------
> > > ...
> > > pci 0000:00:1f.0: [8086:19dc] type 00 class 0x060100 conventional PCI endpoint
> > > pci 0000:00:1f.1: [8086:19dd] type 00 class 0x058000 conventional PCI endpoint ... [A]
> > > pci 0000:00:1f.1: BAR 0 [mem 0xfd000000-0xfdffffff 64bit]
> > > pci 0000:00:1f.2: [8086:19de] type 00 class 0x058000 conventional PCI endpoint
> > > pci 0000:00:1f.2: BAR 0 [mem 0x88c00000-0x88c03fff]
> > > ...
> > > PCI: Using ACPI for IRQ routing
> > > pci 0000:00:1f.1: BAR 0 [mem 0xfd000000-0xfdffffff 64bit]: can't claim; no compatible bridge window ... [B]
> > > hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0
> > > ...
> > > NET: Registered PF_XDP protocol family
> > > pci 0000:00:1f.1: BAR 0 [mem 0x280000000-0x280ffffff 64bit]: assigned ... [C]
> > > pci 0000:00:09.0: PCI bridge to [bus 01-06]
> > > ...
> > > PCI: CLS 64 bytes, default 64
> > > p2sb_cache_resources
> > > p2sb_cache_resources: P2SBC_HIDE=0  ... [D]
> > > p2sb_scan_and_cache_devfn: devfn=1f.1
> > > p2sb_scan_and_cache_devfn: 280000000-280ffffff: 140204 ... [E]
> > > PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
> > > ...
> > > --------------------------------------------------------------------------------
> > > 
> > > Also, here I list my observations.
> > > 
> > > [A] The P2SB device was detected with DEVFN 1f.1, and device id 8086:19dd
> > > [B] Failed to claim its resource
> > > [C] Assigned new resource
> > > [D] p2sb_cache_resource() was called after the new resource assignment.
> > >     P2SBC_HIDE bit is not set.
> > > [E] The new resource was cached. IORESOURCE flags: MEM_64,SIZE_ALIGN,MEM.
> > > 
> > > So it was confirmed that the p2sb_cache_resource() was called after the new
> > > resource assignment, but Hans and Andy discuss that this order is not the
> > > problem cause, probably.
> > > 
> > > One thing I noticed is that p2sb_bar() call is not recorded in the log. My
> > > understanding is that all device drivers which use P2SB resource shouled call
> > > p2sb_bar(). Daniel, you noted that "a custom gpio device" disappeared. Does its
> > > device driver call p2sb_bar()?
> > 
> > No calls to p2sb_bar(). It maybe an Intel GPIO device actually. 0x8086 is Intel's vendor code.
> > I suspect it's something standard on Denverton.
> 
> Are you referring to LPC GPIO?
 
 I don't know the hardware well enough to say for certain. It's whatever device 8086:19dd is.

Daniel

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Andy Shevchenko]

On Mon, Nov 18, 2024 at 01:32:55PM +0000, Daniel Walker (danielwa) wrote:
> On Mon, Nov 18, 2024 at 03:24:20PM +0200, Andy Shevchenko wrote:
> > On Mon, Nov 18, 2024 at 12:40:16PM +0000, Daniel Walker (danielwa) wrote:

...

> > Are you referring to LPC GPIO?
>  
>  I don't know the hardware well enough to say for certain. It's whatever device 8086:19dd is.

This is device which represents p2sb. It's not a GPIO device you are talking
about for sure. You can send privately more detailed info in case this is shouldn't
be on public to me to understand what would be the best approach to fix your issue.

-- 
With Best Regards,
Andy Shevchenko

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Andy Shevchenko]

On Mon, Nov 18, 2024 at 02:29:42PM +0100, Hans de Goede wrote:
> On 18-Nov-24 2:24 PM, Andy Shevchenko wrote:
> > On Mon, Nov 18, 2024 at 12:40:16PM +0000, Daniel Walker (danielwa) wrote:
> >> On Mon, Nov 18, 2024 at 11:30:59AM +0000, Shinichiro Kawasaki wrote:

...

> > Still wondering if we simply unhide it early
> > (if it was hidden) and assume the p2sb device is always present. In this case
> > p2sb_bar() will become almost an equivalent to the pci_resource_start().
> 
> That would basically open it up for normal PCI drivers to bind to. Which is not
> necessarily a problem, but I wonder if we already have such drivers which then
> all of a sudden will start binding to it ?

AFAIK we have no P2SB device drivers in the kernel.

> Note that e.g. the GPIO code in lpc_ich.c checks if the GPIO controller is not
> ACPI enumerated before calling p2sb_bar(). Unconditionally unhiding the P2SB
> would not mean that at least the GPIO part will be enumerated twice, once
> through ACPI and once through PCI. I'm not sure if that is a good idea.

I agree, but how can it be in practice if p2sb is enumerated without creating
any devices behind it? I mean if we have the p2sb driver for solely purpose of
p2sb_bar() call.

> My vote would go to fix the problem of the PCI device being removed by
> the p2sb.c code when it was not hidden.

-- 
With Best Regards,
Andy Shevchenko

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Daniel Walker (danielwa)]

On Mon, Nov 18, 2024 at 03:49:32PM +0200, Andy Shevchenko wrote:
> On Mon, Nov 18, 2024 at 01:32:55PM +0000, Daniel Walker (danielwa) wrote:
> > On Mon, Nov 18, 2024 at 03:24:20PM +0200, Andy Shevchenko wrote:
> > > On Mon, Nov 18, 2024 at 12:40:16PM +0000, Daniel Walker (danielwa) wrote:
> 
> ...
> 
> > > Are you referring to LPC GPIO?
> >  
> >  I don't know the hardware well enough to say for certain. It's whatever device 8086:19dd is.
> 
> This is device which represents p2sb. It's not a GPIO device you are talking
> about for sure. You can send privately more detailed info in case this is shouldn't
> be on public to me to understand what would be the best approach to fix your issue.

Here's a comment,

/* INTEL Denverton GPIO registers are accessible using SBREG_BAR(bar 0) as base */

We have gpio wired to an FPGA and I believe the gpio line is used to reset the
fpga.

So the pci resources attached to 8086:19dd can be used to access gpio of some
type. 

I'm not a pci expert but on the 19bb device bar 0 we use the below offset to manipulate
the gpio,

#define INTEL_GPIO_REG_RESET_OFFSET          0xC50578

The comments suggest this is gpio 6. I would seems your reaction would be that
there is no gpio on the 19dd device. Maybe our driver is access gpio thru p2sb
or something like that.

Does the offset above make sense to you in the context of the p2sb ?

Daniel

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Andy Shevchenko]

On Mon, Nov 18, 2024 at 02:35:44PM +0000, Daniel Walker (danielwa) wrote:
> On Mon, Nov 18, 2024 at 03:49:32PM +0200, Andy Shevchenko wrote:
> > On Mon, Nov 18, 2024 at 01:32:55PM +0000, Daniel Walker (danielwa) wrote:
> > > On Mon, Nov 18, 2024 at 03:24:20PM +0200, Andy Shevchenko wrote:
> > > > On Mon, Nov 18, 2024 at 12:40:16PM +0000, Daniel Walker (danielwa) wrote:

...

> > > > Are you referring to LPC GPIO?
> > >  
> > >  I don't know the hardware well enough to say for certain. It's whatever device 8086:19dd is.
> > 
> > This is device which represents p2sb. It's not a GPIO device you are talking
> > about for sure. You can send privately more detailed info in case this is shouldn't
> > be on public to me to understand what would be the best approach to fix your issue.
> 
> Here's a comment,
> 
> /* INTEL Denverton GPIO registers are accessible using SBREG_BAR(bar 0) as base */
> 
> We have gpio wired to an FPGA and I believe the gpio line is used to reset the
> fpga.
> 
> So the pci resources attached to 8086:19dd can be used to access gpio of some
> type. 
> 
> I'm not a pci expert but on the 19bb device bar 0 we use the below offset to manipulate
> the gpio,
> 
> #define INTEL_GPIO_REG_RESET_OFFSET          0xC50578
> 
> The comments suggest this is gpio 6. I would seems your reaction would be that
> there is no gpio on the 19dd device. Maybe our driver is access gpio thru p2sb
> or something like that.
> 
> Does the offset above make sense to you in the context of the p2sb ?

Yes, everything makes sense. Please, enable lpc_ich driver and forget about
talking to the p2sb memory mapped IO.

/* Offset data for Denverton GPIO controllers */
static const resource_size_t dnv_gpio_offsets[DNV_GPIO_NR_RESOURCES] = {
	[DNV_GPIO_NORTH] = 0xc20000,
	[DNV_GPIO_SOUTH] = 0xc50000,
};

So, you are using a pin from the Community "South" of the on-die Denverton GPIO.

In EDS this called GPIO_6, while in the driver it's pin 88, i.e. SMB3_IE0_DATA.

You will need to
- enable lpc_ich driver (CONFIG_LPC_ICH)
- enable Intel Denverton pin control driver (CONFIG_PINCTRL_DENVERTON)
- replace your custom approach to:
  - GPIO lookup table
  - proper GPIO APIs, as gpiod_get() or alike

See how it was done in the current kernel code:

8241b55f1ded ("drm/i915/dsi: Replace poking of VLV GPIOs behind the driver's back")
a6c80bec3c93 ("leds: simatic-ipc-leds-gpio: Add GPIO version of Siemens driver")

Hans, there will be no need to fix anything if they implement correct access
to the GPIO, i.e. via driver and board code with GPIO lookup tables.

-- 
With Best Regards,
Andy Shevchenko

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Hans de Goede]

Hi,

On 18-Nov-24 4:55 PM, Andy Shevchenko wrote:
> On Mon, Nov 18, 2024 at 02:35:44PM +0000, Daniel Walker (danielwa) wrote:
>> On Mon, Nov 18, 2024 at 03:49:32PM +0200, Andy Shevchenko wrote:
>>> On Mon, Nov 18, 2024 at 01:32:55PM +0000, Daniel Walker (danielwa) wrote:
>>>> On Mon, Nov 18, 2024 at 03:24:20PM +0200, Andy Shevchenko wrote:
>>>>> On Mon, Nov 18, 2024 at 12:40:16PM +0000, Daniel Walker (danielwa) wrote:
> 
> ...
> 
>>>>> Are you referring to LPC GPIO?
>>>>  
>>>>  I don't know the hardware well enough to say for certain. It's whatever device 8086:19dd is.
>>>
>>> This is device which represents p2sb. It's not a GPIO device you are talking
>>> about for sure. You can send privately more detailed info in case this is shouldn't
>>> be on public to me to understand what would be the best approach to fix your issue.
>>
>> Here's a comment,
>>
>> /* INTEL Denverton GPIO registers are accessible using SBREG_BAR(bar 0) as base */
>>
>> We have gpio wired to an FPGA and I believe the gpio line is used to reset the
>> fpga.
>>
>> So the pci resources attached to 8086:19dd can be used to access gpio of some
>> type. 
>>
>> I'm not a pci expert but on the 19bb device bar 0 we use the below offset to manipulate
>> the gpio,
>>
>> #define INTEL_GPIO_REG_RESET_OFFSET          0xC50578
>>
>> The comments suggest this is gpio 6. I would seems your reaction would be that
>> there is no gpio on the 19dd device. Maybe our driver is access gpio thru p2sb
>> or something like that.
>>
>> Does the offset above make sense to you in the context of the p2sb ?
> 
> Yes, everything makes sense. Please, enable lpc_ich driver and forget about
> talking to the p2sb memory mapped IO.
> 
> /* Offset data for Denverton GPIO controllers */
> static const resource_size_t dnv_gpio_offsets[DNV_GPIO_NR_RESOURCES] = {
> 	[DNV_GPIO_NORTH] = 0xc20000,
> 	[DNV_GPIO_SOUTH] = 0xc50000,
> };
> 
> So, you are using a pin from the Community "South" of the on-die Denverton GPIO.
> 
> In EDS this called GPIO_6, while in the driver it's pin 88, i.e. SMB3_IE0_DATA.
> 
> You will need to
> - enable lpc_ich driver (CONFIG_LPC_ICH)
> - enable Intel Denverton pin control driver (CONFIG_PINCTRL_DENVERTON)
> - replace your custom approach to:
>   - GPIO lookup table
>   - proper GPIO APIs, as gpiod_get() or alike
> 
> See how it was done in the current kernel code:
> 
> 8241b55f1ded ("drm/i915/dsi: Replace poking of VLV GPIOs behind the driver's back")
> a6c80bec3c93 ("leds: simatic-ipc-leds-gpio: Add GPIO version of Siemens driver")
> 
> Hans, there will be no need to fix anything if they implement correct access
> to the GPIO, i.e. via driver and board code with GPIO lookup tables.

Agreed, still I'm not sure how I feel about us hiding the previously unhidden P2SB.

OTOH I guess it may have only been unhidden in the BIOS to make the hack they
are using possible in the first place.

Regards,

Hans

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Andy Shevchenko]

On Mon, Nov 18, 2024 at 05:00:52PM +0100, Hans de Goede wrote:
> On 18-Nov-24 4:55 PM, Andy Shevchenko wrote:

...

> Agreed, still I'm not sure how I feel about us hiding the previously unhidden P2SB.
> 
> OTOH I guess it may have only been unhidden in the BIOS to make the hack they
> are using possible in the first place.

Yeah, sounds like this, because the whole dance with this hiding-unhiding
is due to the Windows yellow bang, due to no driver for the thingy, and
scared users. Reference BIOS should hide it by default when giving a control
to the bootloader and OS.

-- 
With Best Regards,
Andy Shevchenko

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Daniel Walker (danielwa)]

On Mon, Nov 18, 2024 at 05:00:52PM +0100, Hans de Goede wrote:
> Hi,
> 
> On 18-Nov-24 4:55 PM, Andy Shevchenko wrote:
> > On Mon, Nov 18, 2024 at 02:35:44PM +0000, Daniel Walker (danielwa) wrote:
> >> On Mon, Nov 18, 2024 at 03:49:32PM +0200, Andy Shevchenko wrote:
> >>> On Mon, Nov 18, 2024 at 01:32:55PM +0000, Daniel Walker (danielwa) wrote:
> >>>> On Mon, Nov 18, 2024 at 03:24:20PM +0200, Andy Shevchenko wrote:
> >>>>> On Mon, Nov 18, 2024 at 12:40:16PM +0000, Daniel Walker (danielwa) wrote:
> > 
> > ...
> > 
> >>>>> Are you referring to LPC GPIO?
> >>>>  
> >>>>  I don't know the hardware well enough to say for certain. It's whatever device 8086:19dd is.
> >>>
> >>> This is device which represents p2sb. It's not a GPIO device you are talking
> >>> about for sure. You can send privately more detailed info in case this is shouldn't
> >>> be on public to me to understand what would be the best approach to fix your issue.
> >>
> >> Here's a comment,
> >>
> >> /* INTEL Denverton GPIO registers are accessible using SBREG_BAR(bar 0) as base */
> >>
> >> We have gpio wired to an FPGA and I believe the gpio line is used to reset the
> >> fpga.
> >>
> >> So the pci resources attached to 8086:19dd can be used to access gpio of some
> >> type. 
> >>
> >> I'm not a pci expert but on the 19bb device bar 0 we use the below offset to manipulate
> >> the gpio,
> >>
> >> #define INTEL_GPIO_REG_RESET_OFFSET          0xC50578
> >>
> >> The comments suggest this is gpio 6. I would seems your reaction would be that
> >> there is no gpio on the 19dd device. Maybe our driver is access gpio thru p2sb
> >> or something like that.
> >>
> >> Does the offset above make sense to you in the context of the p2sb ?
> > 
> > Yes, everything makes sense. Please, enable lpc_ich driver and forget about
> > talking to the p2sb memory mapped IO.
> > 
> > /* Offset data for Denverton GPIO controllers */
> > static const resource_size_t dnv_gpio_offsets[DNV_GPIO_NR_RESOURCES] = {
> > 	[DNV_GPIO_NORTH] = 0xc20000,
> > 	[DNV_GPIO_SOUTH] = 0xc50000,
> > };
> > 
> > So, you are using a pin from the Community "South" of the on-die Denverton GPIO.
> > 
> > In EDS this called GPIO_6, while in the driver it's pin 88, i.e. SMB3_IE0_DATA.
> > 
> > You will need to
> > - enable lpc_ich driver (CONFIG_LPC_ICH)
> > - enable Intel Denverton pin control driver (CONFIG_PINCTRL_DENVERTON)
> > - replace your custom approach to:
> >   - GPIO lookup table
> >   - proper GPIO APIs, as gpiod_get() or alike
> > 
> > See how it was done in the current kernel code:
> > 
> > 8241b55f1ded ("drm/i915/dsi: Replace poking of VLV GPIOs behind the driver's back")
> > a6c80bec3c93 ("leds: simatic-ipc-leds-gpio: Add GPIO version of Siemens driver")
> > 
> > Hans, there will be no need to fix anything if they implement correct access
> > to the GPIO, i.e. via driver and board code with GPIO lookup tables.
> 
> Agreed, still I'm not sure how I feel about us hiding the previously unhidden P2SB.
> 
> OTOH I guess it may have only been unhidden in the BIOS to make the hack they
> are using possible in the first place.

From a flexibility POV I would suggest if you can not hide it if it's not already
hidden by the BIOS that would be better since some company may have a good
reason to make a custom driver or to export the pci device to userspace thru
UIO. The current situation is you can't make a custom driver if p2sb is enable
with this additional patch even if you unhide the device inside the BIOS.

In our case it seems like we could use the already existing solution with
pinctrl, but others may not be able to do that or may not want to for different
reasons.

Daniel

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Shinichiro Kawasaki]

On Nov 18, 2024 / 17:15, Daniel Walker (danielwa) wrote:
> On Mon, Nov 18, 2024 at 05:00:52PM +0100, Hans de Goede wrote:
> > Hi,
> > 
> > On 18-Nov-24 4:55 PM, Andy Shevchenko wrote:
[...]
> > > Hans, there will be no need to fix anything if they implement correct access
> > > to the GPIO, i.e. via driver and board code with GPIO lookup tables.
> > 
> > Agreed, still I'm not sure how I feel about us hiding the previously unhidden P2SB.
> > 
> > OTOH I guess it may have only been unhidden in the BIOS to make the hack they
> > are using possible in the first place.
> 
> From a flexibility POV I would suggest if you can not hide it if it's not already
> hidden by the BIOS that would be better since some company may have a good
> reason to make a custom driver or to export the pci device to userspace thru
> UIO. The current situation is you can't make a custom driver if p2sb is enable
> with this additional patch even if you unhide the device inside the BIOS.
> 
> In our case it seems like we could use the already existing solution with
> pinctrl, but others may not be able to do that or may not want to for different
> reasons.

I don't have strong opinion about the choice, but I wonder how the p2sb code
will be if we keep the unhidden P2SB. I created a trial patch below. If the
device is not hidden, it does not call pci_scan_single_device() and
pci_stop_and_remove_bus_device(). Instead, it calls pci_get_slot() and
pci_dev_put(). I don't have the environment which unhides P2SB. Daniel, if you
have time to afford, please try it out.

diff --git a/drivers/platform/x86/p2sb.c b/drivers/platform/x86/p2sb.c
index 31f38309b389..dec3d43ce929 100644
--- a/drivers/platform/x86/p2sb.c
+++ b/drivers/platform/x86/p2sb.c
@@ -79,29 +79,49 @@ static void p2sb_read_bar0(struct pci_dev *pdev, struct resource *mem)
 	mem->desc = bar0->desc;
 }
 
-static void p2sb_scan_and_cache_devfn(struct pci_bus *bus, unsigned int devfn)
+static void p2sb_scan_and_cache_devfn(struct pci_bus *bus, unsigned int devfn,
+				      bool hidden)
 {
 	struct p2sb_res_cache *cache = &p2sb_resources[PCI_FUNC(devfn)];
-	struct pci_dev *pdev;
+	struct pci_dev *pdev = NULL;
+
+	if (!hidden)
+		pdev = pci_get_slot(bus, devfn);
+
+	pr_info("%s: hidden=%d pci_get_slot=%px\n", __func__, hidden, pdev);
+
+	if (!pdev) {
+		hidden = true;
+		pdev = pci_scan_single_device(bus, devfn);
+	}
 
-	pdev = pci_scan_single_device(bus, devfn);
 	if (!pdev)
 		return;
 
 	p2sb_read_bar0(pdev, &cache->res);
+
+	pr_info("%s: devfn=%x.%x\n", __func__,
+		PCI_SLOT(devfn), PCI_FUNC(devfn));
+	pr_info("%s: %llx-%llx: %lx\n", __func__,
+		cache->res.start, cache->res.end, cache->res.flags);
+
 	cache->bus_dev_id = bus->dev.id;
 
-	pci_stop_and_remove_bus_device(pdev);
+	if (hidden)
+		pci_stop_and_remove_bus_device(pdev);
+	else
+		pci_dev_put(pdev);
 }
 
-static int p2sb_scan_and_cache(struct pci_bus *bus, unsigned int devfn)
+static int p2sb_scan_and_cache(struct pci_bus *bus, unsigned int devfn,
+			       bool hidden)
 {
 	/* Scan the P2SB device and cache its BAR0 */
-	p2sb_scan_and_cache_devfn(bus, devfn);
+	p2sb_scan_and_cache_devfn(bus, devfn, hidden);
 
 	/* On Goldmont p2sb_bar() also gets called for the SPI controller */
 	if (devfn == P2SB_DEVFN_GOLDMONT)
-		p2sb_scan_and_cache_devfn(bus, SPI_DEVFN_GOLDMONT);
+		p2sb_scan_and_cache_devfn(bus, SPI_DEVFN_GOLDMONT, hidden);
 
 	if (!p2sb_valid_resource(&p2sb_resources[PCI_FUNC(devfn)].res))
 		return -ENOENT;
@@ -127,9 +147,12 @@ static int p2sb_cache_resources(void)
 	unsigned int devfn_p2sb;
 	u32 value = P2SBC_HIDE;
 	struct pci_bus *bus;
+	bool hidden;
 	u16 class;
 	int ret;
 
+	pr_info("%s\n", __func__);
+
 	/* Get devfn for P2SB device itself */
 	p2sb_get_devfn(&devfn_p2sb);
 
@@ -157,13 +180,15 @@ static int p2sb_cache_resources(void)
 	 * Unhide the P2SB device here, if needed.
 	 */
 	pci_bus_read_config_dword(bus, devfn_p2sb, P2SBC, &value);
-	if (value & P2SBC_HIDE)
+	hidden = value & P2SBC_HIDE;
+	pr_info("%s: P2SBC_HIDE=%u\n", __func__, value & hidden);
+	if (hidden)
 		pci_bus_write_config_dword(bus, devfn_p2sb, P2SBC, 0);
 
-	ret = p2sb_scan_and_cache(bus, devfn_p2sb);
+	ret = p2sb_scan_and_cache(bus, devfn_p2sb, hidden);
 
 	/* Hide the P2SB device, if it was hidden */
-	if (value & P2SBC_HIDE)
+	if (hidden)
 		pci_bus_write_config_dword(bus, devfn_p2sb, P2SBC, P2SBC_HIDE);
 
 	pci_unlock_rescan_remove();
@@ -189,6 +214,8 @@ int p2sb_bar(struct pci_bus *bus, unsigned int devfn, struct resource *mem)
 {
 	struct p2sb_res_cache *cache;
 
+	pr_info("%s\n", __func__);
+
 	bus = p2sb_get_bus(bus);
 	if (!bus)
 		return -ENODEV;
@@ -204,6 +231,12 @@ int p2sb_bar(struct pci_bus *bus, unsigned int devfn, struct resource *mem)
 		return -ENOENT;
 
 	memcpy(mem, &cache->res, sizeof(*mem));
+
+	pr_info("%s: devfn=%x.%x\n", __func__,
+		PCI_SLOT(devfn), PCI_FUNC(devfn));
+	pr_info("%s: %llx-%llx: %lx\n", __func__,
+		cache->res.start, cache->res.end, cache->res.flags);
+
 	return 0;
 }
 EXPORT_SYMBOL_GPL(p2sb_bar);

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Andy Shevchenko]

On Tue, Nov 19, 2024 at 02:20:48AM +0000, Shinichiro Kawasaki wrote:
> On Nov 18, 2024 / 17:15, Daniel Walker (danielwa) wrote:
> > On Mon, Nov 18, 2024 at 05:00:52PM +0100, Hans de Goede wrote:

[...]

> > > > Hans, there will be no need to fix anything if they implement correct access
> > > > to the GPIO, i.e. via driver and board code with GPIO lookup tables.
> > > 
> > > Agreed, still I'm not sure how I feel about us hiding the previously unhidden P2SB.
> > > 
> > > OTOH I guess it may have only been unhidden in the BIOS to make the hack they
> > > are using possible in the first place.
> > 
> > From a flexibility POV I would suggest if you can not hide it if it's not already
> > hidden by the BIOS that would be better since some company may have a good
> > reason to make a custom driver or to export the pci device to userspace thru
> > UIO. The current situation is you can't make a custom driver if p2sb is enable
> > with this additional patch even if you unhide the device inside the BIOS.
> > 
> > In our case it seems like we could use the already existing solution with
> > pinctrl, but others may not be able to do that or may not want to for different
> > reasons.
> 
> I don't have strong opinion about the choice, but I wonder how the p2sb code
> will be if we keep the unhidden P2SB. I created a trial patch below. If the
> device is not hidden, it does not call pci_scan_single_device() and
> pci_stop_and_remove_bus_device(). Instead, it calls pci_get_slot() and
> pci_dev_put(). I don't have the environment which unhides P2SB. Daniel, if you
> have time to afford, please try it out.

...

> -	ret = p2sb_scan_and_cache(bus, devfn_p2sb);
> +	ret = p2sb_scan_and_cache(bus, devfn_p2sb, hidden);


If you still want to implement this, rather make the two different calls here

	if (hidden)
		ret = p2sb_scan_and_cache(bus, devfn_p2sb);
	else
		ret = p2sb_read_and_cache(...);

otherwise the code looks much uglier... :-(

-- 
With Best Regards,
Andy Shevchenko

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Andy Shevchenko]

On Mon, Nov 18, 2024 at 05:15:17PM +0000, Daniel Walker (danielwa) wrote:
> On Mon, Nov 18, 2024 at 05:00:52PM +0100, Hans de Goede wrote:
> > On 18-Nov-24 4:55 PM, Andy Shevchenko wrote:
> > > On Mon, Nov 18, 2024 at 02:35:44PM +0000, Daniel Walker (danielwa) wrote:
> > >> On Mon, Nov 18, 2024 at 03:49:32PM +0200, Andy Shevchenko wrote:
> > >>> On Mon, Nov 18, 2024 at 01:32:55PM +0000, Daniel Walker (danielwa) wrote:
> > >>>> On Mon, Nov 18, 2024 at 03:24:20PM +0200, Andy Shevchenko wrote:
> > >>>>> On Mon, Nov 18, 2024 at 12:40:16PM +0000, Daniel Walker (danielwa) wrote:

...

> > >>>>> Are you referring to LPC GPIO?
> > >>>>  
> > >>>>  I don't know the hardware well enough to say for certain. It's whatever device 8086:19dd is.
> > >>>
> > >>> This is device which represents p2sb. It's not a GPIO device you are talking
> > >>> about for sure. You can send privately more detailed info in case this is shouldn't
> > >>> be on public to me to understand what would be the best approach to fix your issue.
> > >>
> > >> Here's a comment,
> > >>
> > >> /* INTEL Denverton GPIO registers are accessible using SBREG_BAR(bar 0) as base */
> > >>
> > >> We have gpio wired to an FPGA and I believe the gpio line is used to reset the
> > >> fpga.
> > >>
> > >> So the pci resources attached to 8086:19dd can be used to access gpio of some
> > >> type. 
> > >>
> > >> I'm not a pci expert but on the 19bb device bar 0 we use the below offset to manipulate
> > >> the gpio,
> > >>
> > >> #define INTEL_GPIO_REG_RESET_OFFSET          0xC50578
> > >>
> > >> The comments suggest this is gpio 6. I would seems your reaction would be that
> > >> there is no gpio on the 19dd device. Maybe our driver is access gpio thru p2sb
> > >> or something like that.
> > >>
> > >> Does the offset above make sense to you in the context of the p2sb ?
> > > 
> > > Yes, everything makes sense. Please, enable lpc_ich driver and forget about
> > > talking to the p2sb memory mapped IO.
> > > 
> > > /* Offset data for Denverton GPIO controllers */
> > > static const resource_size_t dnv_gpio_offsets[DNV_GPIO_NR_RESOURCES] = {
> > > 	[DNV_GPIO_NORTH] = 0xc20000,
> > > 	[DNV_GPIO_SOUTH] = 0xc50000,
> > > };
> > > 
> > > So, you are using a pin from the Community "South" of the on-die Denverton GPIO.
> > > 
> > > In EDS this called GPIO_6, while in the driver it's pin 88, i.e. SMB3_IE0_DATA.
> > > 
> > > You will need to
> > > - enable lpc_ich driver (CONFIG_LPC_ICH)
> > > - enable Intel Denverton pin control driver (CONFIG_PINCTRL_DENVERTON)
> > > - replace your custom approach to:
> > >   - GPIO lookup table
> > >   - proper GPIO APIs, as gpiod_get() or alike
> > > 
> > > See how it was done in the current kernel code:
> > > 
> > > 8241b55f1ded ("drm/i915/dsi: Replace poking of VLV GPIOs behind the driver's back")
> > > a6c80bec3c93 ("leds: simatic-ipc-leds-gpio: Add GPIO version of Siemens driver")
> > > 
> > > Hans, there will be no need to fix anything if they implement correct access
> > > to the GPIO, i.e. via driver and board code with GPIO lookup tables.
> > 
> > Agreed, still I'm not sure how I feel about us hiding the previously unhidden P2SB.
> > 
> > OTOH I guess it may have only been unhidden in the BIOS to make the hack they
> > are using possible in the first place.
> 
> From a flexibility POV I would suggest if you can not hide it if it's not already
> hidden by the BIOS that would be better since some company may have a good
> reason to make a custom driver or to export the pci device to userspace thru
> UIO.

The previous emails used wrong terminology, the hidden device is left hidden, and
all the opposite: unhidden is not touched in this sense.

The problem there that for the initially unhidden devices we call pci_stop_...
which seems incorrect and needs to be fixed, indeed.

> The current situation is you can't make a custom driver if p2sb is enable
> with this additional patch even if you unhide the device inside the BIOS.

Yeah, but I do not consider that is anyhow related to upstream.

> In our case it seems like we could use the already existing solution with
> pinctrl, but others may not be able to do that or may not want to for different
> reasons.

Please, please, go with the pin control approach, let's not increase
the surface of the untested fields. Feel free to ask for a help from me
(and possibly Hans) if you need a such.

-- 
With Best Regards,
Andy Shevchenko

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Daniel Walker (danielwa)]

On Tue, Nov 19, 2024 at 11:41:49AM +0200, Andy Shevchenko wrote:
> On Mon, Nov 18, 2024 at 05:15:17PM +0000, Daniel Walker (danielwa) wrote:
> > On Mon, Nov 18, 2024 at 05:00:52PM +0100, Hans de Goede wrote:
> > > On 18-Nov-24 4:55 PM, Andy Shevchenko wrote:
> > > > On Mon, Nov 18, 2024 at 02:35:44PM +0000, Daniel Walker (danielwa) wrote:
> > > >> On Mon, Nov 18, 2024 at 03:49:32PM +0200, Andy Shevchenko wrote:
> > > >>> On Mon, Nov 18, 2024 at 01:32:55PM +0000, Daniel Walker (danielwa) wrote:
> > > >>>> On Mon, Nov 18, 2024 at 03:24:20PM +0200, Andy Shevchenko wrote:
> > > >>>>> On Mon, Nov 18, 2024 at 12:40:16PM +0000, Daniel Walker (danielwa) wrote:
> 
> ...
> 
> > > >>>>> Are you referring to LPC GPIO?
> > > >>>>  
> > > >>>>  I don't know the hardware well enough to say for certain. It's whatever device 8086:19dd is.
> > > >>>
> > > >>> This is device which represents p2sb. It's not a GPIO device you are talking
> > > >>> about for sure. You can send privately more detailed info in case this is shouldn't
> > > >>> be on public to me to understand what would be the best approach to fix your issue.
> > > >>
> > > >> Here's a comment,
> > > >>
> > > >> /* INTEL Denverton GPIO registers are accessible using SBREG_BAR(bar 0) as base */
> > > >>
> > > >> We have gpio wired to an FPGA and I believe the gpio line is used to reset the
> > > >> fpga.
> > > >>
> > > >> So the pci resources attached to 8086:19dd can be used to access gpio of some
> > > >> type. 
> > > >>
> > > >> I'm not a pci expert but on the 19bb device bar 0 we use the below offset to manipulate
> > > >> the gpio,
> > > >>
> > > >> #define INTEL_GPIO_REG_RESET_OFFSET          0xC50578
> > > >>
> > > >> The comments suggest this is gpio 6. I would seems your reaction would be that
> > > >> there is no gpio on the 19dd device. Maybe our driver is access gpio thru p2sb
> > > >> or something like that.
> > > >>
> > > >> Does the offset above make sense to you in the context of the p2sb ?
> > > > 
> > > > Yes, everything makes sense. Please, enable lpc_ich driver and forget about
> > > > talking to the p2sb memory mapped IO.
> > > > 
> > > > /* Offset data for Denverton GPIO controllers */
> > > > static const resource_size_t dnv_gpio_offsets[DNV_GPIO_NR_RESOURCES] = {
> > > > 	[DNV_GPIO_NORTH] = 0xc20000,
> > > > 	[DNV_GPIO_SOUTH] = 0xc50000,
> > > > };
> > > > 
> > > > So, you are using a pin from the Community "South" of the on-die Denverton GPIO.
> > > > 
> > > > In EDS this called GPIO_6, while in the driver it's pin 88, i.e. SMB3_IE0_DATA.
> > > > 
> > > > You will need to
> > > > - enable lpc_ich driver (CONFIG_LPC_ICH)
> > > > - enable Intel Denverton pin control driver (CONFIG_PINCTRL_DENVERTON)
> > > > - replace your custom approach to:
> > > >   - GPIO lookup table
> > > >   - proper GPIO APIs, as gpiod_get() or alike
> > > > 
> > > > See how it was done in the current kernel code:
> > > > 
> > > > 8241b55f1ded ("drm/i915/dsi: Replace poking of VLV GPIOs behind the driver's back")
> > > > a6c80bec3c93 ("leds: simatic-ipc-leds-gpio: Add GPIO version of Siemens driver")
> > > > 
> > > > Hans, there will be no need to fix anything if they implement correct access
> > > > to the GPIO, i.e. via driver and board code with GPIO lookup tables.
> > > 
> > > Agreed, still I'm not sure how I feel about us hiding the previously unhidden P2SB.
> > > 
> > > OTOH I guess it may have only been unhidden in the BIOS to make the hack they
> > > are using possible in the first place.
> > 
> > From a flexibility POV I would suggest if you can not hide it if it's not already
> > hidden by the BIOS that would be better since some company may have a good
> > reason to make a custom driver or to export the pci device to userspace thru
> > UIO.
> 
> The previous emails used wrong terminology, the hidden device is left hidden, and
> all the opposite: unhidden is not touched in this sense.
> 
> The problem there that for the initially unhidden devices we call pci_stop_...
> which seems incorrect and needs to be fixed, indeed.
> 
> > The current situation is you can't make a custom driver if p2sb is enable
> > with this additional patch even if you unhide the device inside the BIOS.
> 
> Yeah, but I do not consider that is anyhow related to upstream.
> 

Not true. Have you used the UIO system ? You can make a custom userspace driver
for a pci device with zero kernel changes. We have a custom LKM , but we could
easily have done it with UIO. With the PCI device completely gone, there is no
way to use UIO to make a userspace driver.

Daniel

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Andy Shevchenko]

On Tue, Nov 19, 2024 at 02:47:02PM +0000, Daniel Walker (danielwa) wrote:
> On Tue, Nov 19, 2024 at 11:41:49AM +0200, Andy Shevchenko wrote:
> > On Mon, Nov 18, 2024 at 05:15:17PM +0000, Daniel Walker (danielwa) wrote:
> > > On Mon, Nov 18, 2024 at 05:00:52PM +0100, Hans de Goede wrote:
> > > > On 18-Nov-24 4:55 PM, Andy Shevchenko wrote:
> > > > > On Mon, Nov 18, 2024 at 02:35:44PM +0000, Daniel Walker (danielwa) wrote:
> > > > >> On Mon, Nov 18, 2024 at 03:49:32PM +0200, Andy Shevchenko wrote:
> > > > >>> On Mon, Nov 18, 2024 at 01:32:55PM +0000, Daniel Walker (danielwa) wrote:
> > > > >>>> On Mon, Nov 18, 2024 at 03:24:20PM +0200, Andy Shevchenko wrote:
> > > > >>>>> On Mon, Nov 18, 2024 at 12:40:16PM +0000, Daniel Walker (danielwa) wrote:

...

> > > > >>>>> Are you referring to LPC GPIO?
> > > > >>>>  
> > > > >>>>  I don't know the hardware well enough to say for certain. It's whatever device 8086:19dd is.
> > > > >>>
> > > > >>> This is device which represents p2sb. It's not a GPIO device you are talking
> > > > >>> about for sure. You can send privately more detailed info in case this is shouldn't
> > > > >>> be on public to me to understand what would be the best approach to fix your issue.
> > > > >>
> > > > >> Here's a comment,
> > > > >>
> > > > >> /* INTEL Denverton GPIO registers are accessible using SBREG_BAR(bar 0) as base */
> > > > >>
> > > > >> We have gpio wired to an FPGA and I believe the gpio line is used to reset the
> > > > >> fpga.
> > > > >>
> > > > >> So the pci resources attached to 8086:19dd can be used to access gpio of some
> > > > >> type. 
> > > > >>
> > > > >> I'm not a pci expert but on the 19bb device bar 0 we use the below offset to manipulate
> > > > >> the gpio,
> > > > >>
> > > > >> #define INTEL_GPIO_REG_RESET_OFFSET          0xC50578
> > > > >>
> > > > >> The comments suggest this is gpio 6. I would seems your reaction would be that
> > > > >> there is no gpio on the 19dd device. Maybe our driver is access gpio thru p2sb
> > > > >> or something like that.
> > > > >>
> > > > >> Does the offset above make sense to you in the context of the p2sb ?
> > > > > 
> > > > > Yes, everything makes sense. Please, enable lpc_ich driver and forget about
> > > > > talking to the p2sb memory mapped IO.
> > > > > 
> > > > > /* Offset data for Denverton GPIO controllers */
> > > > > static const resource_size_t dnv_gpio_offsets[DNV_GPIO_NR_RESOURCES] = {
> > > > > 	[DNV_GPIO_NORTH] = 0xc20000,
> > > > > 	[DNV_GPIO_SOUTH] = 0xc50000,
> > > > > };
> > > > > 
> > > > > So, you are using a pin from the Community "South" of the on-die Denverton GPIO.
> > > > > 
> > > > > In EDS this called GPIO_6, while in the driver it's pin 88, i.e. SMB3_IE0_DATA.
> > > > > 
> > > > > You will need to
> > > > > - enable lpc_ich driver (CONFIG_LPC_ICH)
> > > > > - enable Intel Denverton pin control driver (CONFIG_PINCTRL_DENVERTON)
> > > > > - replace your custom approach to:
> > > > >   - GPIO lookup table
> > > > >   - proper GPIO APIs, as gpiod_get() or alike
> > > > > 
> > > > > See how it was done in the current kernel code:
> > > > > 
> > > > > 8241b55f1ded ("drm/i915/dsi: Replace poking of VLV GPIOs behind the driver's back")
> > > > > a6c80bec3c93 ("leds: simatic-ipc-leds-gpio: Add GPIO version of Siemens driver")
> > > > > 
> > > > > Hans, there will be no need to fix anything if they implement correct access
> > > > > to the GPIO, i.e. via driver and board code with GPIO lookup tables.
> > > > 
> > > > Agreed, still I'm not sure how I feel about us hiding the previously unhidden P2SB.
> > > > 
> > > > OTOH I guess it may have only been unhidden in the BIOS to make the hack they
> > > > are using possible in the first place.
> > > 
> > > From a flexibility POV I would suggest if you can not hide it if it's not already
> > > hidden by the BIOS that would be better since some company may have a good
> > > reason to make a custom driver or to export the pci device to userspace thru
> > > UIO.
> > 
> > The previous emails used wrong terminology, the hidden device is left hidden, and
> > all the opposite: unhidden is not touched in this sense.
> > 
> > The problem there that for the initially unhidden devices we call pci_stop_...
> > which seems incorrect and needs to be fixed, indeed.
> > 
> > > The current situation is you can't make a custom driver if p2sb is enable
> > > with this additional patch even if you unhide the device inside the BIOS.
> > 
> > Yeah, but I do not consider that is anyhow related to upstream.
> 
> Not true. Have you used the UIO system ? You can make a custom userspace driver
> for a pci device with zero kernel changes. We have a custom LKM , but we could
> easily have done it with UIO. With the PCI device completely gone, there is no
> way to use UIO to make a userspace driver.

Is that what you are using with p2sb, or is it just a general remark?
Because if it's the former, it's already double slippery slope.
If you are talking in general, yes, that might be a case. And note,
I am not against fixing p2sb case here, however it sounds like people
want to seek for a problem...

-- 
With Best Regards,
Andy Shevchenko

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Hans de Goede]

Hi,

On 19-Nov-24 3:20 AM, Shinichiro Kawasaki wrote:
> On Nov 18, 2024 / 17:15, Daniel Walker (danielwa) wrote:
>> On Mon, Nov 18, 2024 at 05:00:52PM +0100, Hans de Goede wrote:
>>> Hi,
>>>
>>> On 18-Nov-24 4:55 PM, Andy Shevchenko wrote:
> [...]
>>>> Hans, there will be no need to fix anything if they implement correct access
>>>> to the GPIO, i.e. via driver and board code with GPIO lookup tables.
>>>
>>> Agreed, still I'm not sure how I feel about us hiding the previously unhidden P2SB.
>>>
>>> OTOH I guess it may have only been unhidden in the BIOS to make the hack they
>>> are using possible in the first place.
>>
>> From a flexibility POV I would suggest if you can not hide it if it's not already
>> hidden by the BIOS that would be better since some company may have a good
>> reason to make a custom driver or to export the pci device to userspace thru
>> UIO. The current situation is you can't make a custom driver if p2sb is enable
>> with this additional patch even if you unhide the device inside the BIOS.
>>
>> In our case it seems like we could use the already existing solution with
>> pinctrl, but others may not be able to do that or may not want to for different
>> reasons.
> 
> I don't have strong opinion about the choice, but I wonder how the p2sb code
> will be if we keep the unhidden P2SB. I created a trial patch below. If the
> device is not hidden, it does not call pci_scan_single_device() and
> pci_stop_and_remove_bus_device(). Instead, it calls pci_get_slot() and
> pci_dev_put(). I don't have the environment which unhides P2SB. Daniel, if you
> have time to afford, please try it out.

Thank you for looking into this.

Daniel can you give this a try? It should fix the regression you are seeing
without needing to rework your code (reworking your code to be cleaner
might still be a good idea though).

Shinichiro, can you turn this into a proper patch without all the debug
prints and maybe follow Andy's suggestion to just make this a separate
function.

Regards,

Hans




> 
> diff --git a/drivers/platform/x86/p2sb.c b/drivers/platform/x86/p2sb.c
> index 31f38309b389..dec3d43ce929 100644
> --- a/drivers/platform/x86/p2sb.c
> +++ b/drivers/platform/x86/p2sb.c
> @@ -79,29 +79,49 @@ static void p2sb_read_bar0(struct pci_dev *pdev, struct resource *mem)
>  	mem->desc = bar0->desc;
>  }
>  
> -static void p2sb_scan_and_cache_devfn(struct pci_bus *bus, unsigned int devfn)
> +static void p2sb_scan_and_cache_devfn(struct pci_bus *bus, unsigned int devfn,
> +				      bool hidden)
>  {
>  	struct p2sb_res_cache *cache = &p2sb_resources[PCI_FUNC(devfn)];
> -	struct pci_dev *pdev;
> +	struct pci_dev *pdev = NULL;
> +
> +	if (!hidden)
> +		pdev = pci_get_slot(bus, devfn);
> +
> +	pr_info("%s: hidden=%d pci_get_slot=%px\n", __func__, hidden, pdev);
> +
> +	if (!pdev) {
> +		hidden = true;
> +		pdev = pci_scan_single_device(bus, devfn);
> +	}
>  
> -	pdev = pci_scan_single_device(bus, devfn);
>  	if (!pdev)
>  		return;
>  
>  	p2sb_read_bar0(pdev, &cache->res);
> +
> +	pr_info("%s: devfn=%x.%x\n", __func__,
> +		PCI_SLOT(devfn), PCI_FUNC(devfn));
> +	pr_info("%s: %llx-%llx: %lx\n", __func__,
> +		cache->res.start, cache->res.end, cache->res.flags);
> +
>  	cache->bus_dev_id = bus->dev.id;
>  
> -	pci_stop_and_remove_bus_device(pdev);
> +	if (hidden)
> +		pci_stop_and_remove_bus_device(pdev);
> +	else
> +		pci_dev_put(pdev);
>  }
>  
> -static int p2sb_scan_and_cache(struct pci_bus *bus, unsigned int devfn)
> +static int p2sb_scan_and_cache(struct pci_bus *bus, unsigned int devfn,
> +			       bool hidden)
>  {
>  	/* Scan the P2SB device and cache its BAR0 */
> -	p2sb_scan_and_cache_devfn(bus, devfn);
> +	p2sb_scan_and_cache_devfn(bus, devfn, hidden);
>  
>  	/* On Goldmont p2sb_bar() also gets called for the SPI controller */
>  	if (devfn == P2SB_DEVFN_GOLDMONT)
> -		p2sb_scan_and_cache_devfn(bus, SPI_DEVFN_GOLDMONT);
> +		p2sb_scan_and_cache_devfn(bus, SPI_DEVFN_GOLDMONT, hidden);
>  
>  	if (!p2sb_valid_resource(&p2sb_resources[PCI_FUNC(devfn)].res))
>  		return -ENOENT;
> @@ -127,9 +147,12 @@ static int p2sb_cache_resources(void)
>  	unsigned int devfn_p2sb;
>  	u32 value = P2SBC_HIDE;
>  	struct pci_bus *bus;
> +	bool hidden;
>  	u16 class;
>  	int ret;
>  
> +	pr_info("%s\n", __func__);
> +
>  	/* Get devfn for P2SB device itself */
>  	p2sb_get_devfn(&devfn_p2sb);
>  
> @@ -157,13 +180,15 @@ static int p2sb_cache_resources(void)
>  	 * Unhide the P2SB device here, if needed.
>  	 */
>  	pci_bus_read_config_dword(bus, devfn_p2sb, P2SBC, &value);
> -	if (value & P2SBC_HIDE)
> +	hidden = value & P2SBC_HIDE;
> +	pr_info("%s: P2SBC_HIDE=%u\n", __func__, value & hidden);
> +	if (hidden)
>  		pci_bus_write_config_dword(bus, devfn_p2sb, P2SBC, 0);
>  
> -	ret = p2sb_scan_and_cache(bus, devfn_p2sb);
> +	ret = p2sb_scan_and_cache(bus, devfn_p2sb, hidden);
>  
>  	/* Hide the P2SB device, if it was hidden */
> -	if (value & P2SBC_HIDE)
> +	if (hidden)
>  		pci_bus_write_config_dword(bus, devfn_p2sb, P2SBC, P2SBC_HIDE);
>  
>  	pci_unlock_rescan_remove();
> @@ -189,6 +214,8 @@ int p2sb_bar(struct pci_bus *bus, unsigned int devfn, struct resource *mem)
>  {
>  	struct p2sb_res_cache *cache;
>  
> +	pr_info("%s\n", __func__);
> +
>  	bus = p2sb_get_bus(bus);
>  	if (!bus)
>  		return -ENODEV;
> @@ -204,6 +231,12 @@ int p2sb_bar(struct pci_bus *bus, unsigned int devfn, struct resource *mem)
>  		return -ENOENT;
>  
>  	memcpy(mem, &cache->res, sizeof(*mem));
> +
> +	pr_info("%s: devfn=%x.%x\n", __func__,
> +		PCI_SLOT(devfn), PCI_FUNC(devfn));
> +	pr_info("%s: %llx-%llx: %lx\n", __func__,
> +		cache->res.start, cache->res.end, cache->res.flags);
> +
>  	return 0;
>  }
>  EXPORT_SYMBOL_GPL(p2sb_bar);

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Daniel Walker (danielwa)]

On Tue, Nov 19, 2024 at 07:28:16PM +0100, Hans de Goede wrote:
> Hi,
> 
> On 19-Nov-24 3:20 AM, Shinichiro Kawasaki wrote:
> > On Nov 18, 2024 / 17:15, Daniel Walker (danielwa) wrote:
> >> On Mon, Nov 18, 2024 at 05:00:52PM +0100, Hans de Goede wrote:
> >>> Hi,
> >>>
> >>> On 18-Nov-24 4:55 PM, Andy Shevchenko wrote:
> > [...]
> >>>> Hans, there will be no need to fix anything if they implement correct access
> >>>> to the GPIO, i.e. via driver and board code with GPIO lookup tables.
> >>>
> >>> Agreed, still I'm not sure how I feel about us hiding the previously unhidden P2SB.
> >>>
> >>> OTOH I guess it may have only been unhidden in the BIOS to make the hack they
> >>> are using possible in the first place.
> >>
> >> From a flexibility POV I would suggest if you can not hide it if it's not already
> >> hidden by the BIOS that would be better since some company may have a good
> >> reason to make a custom driver or to export the pci device to userspace thru
> >> UIO. The current situation is you can't make a custom driver if p2sb is enable
> >> with this additional patch even if you unhide the device inside the BIOS.
> >>
> >> In our case it seems like we could use the already existing solution with
> >> pinctrl, but others may not be able to do that or may not want to for different
> >> reasons.
> > 
> > I don't have strong opinion about the choice, but I wonder how the p2sb code
> > will be if we keep the unhidden P2SB. I created a trial patch below. If the
> > device is not hidden, it does not call pci_scan_single_device() and
> > pci_stop_and_remove_bus_device(). Instead, it calls pci_get_slot() and
> > pci_dev_put(). I don't have the environment which unhides P2SB. Daniel, if you
> > have time to afford, please try it out.
> 
> Thank you for looking into this.
> 
> Daniel can you give this a try? It should fix the regression you are seeing
> without needing to rework your code (reworking your code to be cleaner
> might still be a good idea though).
> 

The patch restores the device 8086:19dd back to lspci which is the pre commit
5913320eb behavior. So I would say it's working on my system.

Daniel

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Shinichiro Kawasaki]

On Nov 19, 2024 / 11:37, Andy Shevchenko wrote:
> On Tue, Nov 19, 2024 at 02:20:48AM +0000, Shinichiro Kawasaki wrote:
> > On Nov 18, 2024 / 17:15, Daniel Walker (danielwa) wrote:
> > > On Mon, Nov 18, 2024 at 05:00:52PM +0100, Hans de Goede wrote:
> 
> [...]
> 
> > > > > Hans, there will be no need to fix anything if they implement correct access
> > > > > to the GPIO, i.e. via driver and board code with GPIO lookup tables.
> > > > 
> > > > Agreed, still I'm not sure how I feel about us hiding the previously unhidden P2SB.
> > > > 
> > > > OTOH I guess it may have only been unhidden in the BIOS to make the hack they
> > > > are using possible in the first place.
> > > 
> > > From a flexibility POV I would suggest if you can not hide it if it's not already
> > > hidden by the BIOS that would be better since some company may have a good
> > > reason to make a custom driver or to export the pci device to userspace thru
> > > UIO. The current situation is you can't make a custom driver if p2sb is enable
> > > with this additional patch even if you unhide the device inside the BIOS.
> > > 
> > > In our case it seems like we could use the already existing solution with
> > > pinctrl, but others may not be able to do that or may not want to for different
> > > reasons.
> > 
> > I don't have strong opinion about the choice, but I wonder how the p2sb code
> > will be if we keep the unhidden P2SB. I created a trial patch below. If the
> > device is not hidden, it does not call pci_scan_single_device() and
> > pci_stop_and_remove_bus_device(). Instead, it calls pci_get_slot() and
> > pci_dev_put(). I don't have the environment which unhides P2SB. Daniel, if you
> > have time to afford, please try it out.
> 
> ...
> 
> > -	ret = p2sb_scan_and_cache(bus, devfn_p2sb);
> > +	ret = p2sb_scan_and_cache(bus, devfn_p2sb, hidden);
> 
> 
> If you still want to implement this, rather make the two different calls here
> 
> 	if (hidden)
> 		ret = p2sb_scan_and_cache(bus, devfn_p2sb);
> 	else
> 		ret = p2sb_read_and_cache(...);
> 
> otherwise the code looks much uglier... :-(

Per the comment by Hans, I will create and post the formal version of the patch.
I will try to reflect your comment above. p2sb_scan_and_cache() and
p2sb_read_and_cache() will do almost same thing, then will need some tweak to
avoid code duplication. I will cook something for review.

Re: platform/x86: p2sb: Allow p2sb_bar() calls during PCI device probe

[Shinichiro Kawasaki]

On Nov 19, 2024 / 19:28, Hans de Goede wrote:
> Hi,
> 
> On 19-Nov-24 3:20 AM, Shinichiro Kawasaki wrote:
> > On Nov 18, 2024 / 17:15, Daniel Walker (danielwa) wrote:
> >> On Mon, Nov 18, 2024 at 05:00:52PM +0100, Hans de Goede wrote:
> >>> Hi,
> >>>
> >>> On 18-Nov-24 4:55 PM, Andy Shevchenko wrote:
> > [...]
> >>>> Hans, there will be no need to fix anything if they implement correct access
> >>>> to the GPIO, i.e. via driver and board code with GPIO lookup tables.
> >>>
> >>> Agreed, still I'm not sure how I feel about us hiding the previously unhidden P2SB.
> >>>
> >>> OTOH I guess it may have only been unhidden in the BIOS to make the hack they
> >>> are using possible in the first place.
> >>
> >> From a flexibility POV I would suggest if you can not hide it if it's not already
> >> hidden by the BIOS that would be better since some company may have a good
> >> reason to make a custom driver or to export the pci device to userspace thru
> >> UIO. The current situation is you can't make a custom driver if p2sb is enable
> >> with this additional patch even if you unhide the device inside the BIOS.
> >>
> >> In our case it seems like we could use the already existing solution with
> >> pinctrl, but others may not be able to do that or may not want to for different
> >> reasons.
> > 
> > I don't have strong opinion about the choice, but I wonder how the p2sb code
> > will be if we keep the unhidden P2SB. I created a trial patch below. If the
> > device is not hidden, it does not call pci_scan_single_device() and
> > pci_stop_and_remove_bus_device(). Instead, it calls pci_get_slot() and
> > pci_dev_put(). I don't have the environment which unhides P2SB. Daniel, if you
> > have time to afford, please try it out.
> 
> Thank you for looking into this.
> 
> Daniel can you give this a try? It should fix the regression you are seeing
> without needing to rework your code (reworking your code to be cleaner
> might still be a good idea though).
> 
> Shinichiro, can you turn this into a proper patch without all the debug
> prints and maybe follow Andy's suggestion to just make this a separate
> function.

Sure, I have posted it [*]. Review comments will be welcomed.

Daniel, thank you for testing the trial patch. Your confirmation on the formal
patch will be appreciated also.

[*] https://lore.kernel.org/platform-driver-x86/20241120064055.245969-1-shinichiro.kawasaki@wdc.com/