[PATCH v5 0/2] PCI: Fix isolated function probing and enable ARI for s390

[PATCH v5 0/2] PCI: Fix isolated function probing and enable ARI for s390

[Niklas Schnelle]

Hi Bjorn,

This series originally aimed to correctly detect ARI as being used on
s390. I had missed however that this, in a pretty obious manner, breaks
the isolated function probing resulting devices not getting probed. For
example if a partition/system only has a PF with devfn 1 passed-through.
Additionally the fix may (TBD) help with an issue that LoongArch has
encountered when using isolated function probing and tried to fix by
limiting it to bus 0 ([0]). If it does fix this it may make sense to
apply this separately from the second patch.

Besides the effect on s390 the second patch should also ensure that VFs
on a virtual bus created via virtfn_add_bus() correctly present ARI as
enabled. Sadly I don't have access to any device to test this.

Speaking of testing. For the first patch I can reproduce the AER
scenario described by only applying the second patch. The SR-IOV
scenario I encountered in the past before commit 25f39d3dcb48 ("s390/pci:
Ignore RID for isolated VFs") and tested the fix now with a partial
revert hack of that commit.

I also tested the series on x86_64 both on a Ryzen AI 340 based laptop
and a Xeon Sapphire Rapids based system (including with SR-IOV on
a ConnectX-6 DX).

Thanks,
Niklas

[0] https://lore.kernel.org/linux-pci/20251014074100.2149737-1-chenhuacai@loongson.cn/

Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
---
Changes in v5:
- Rename variables to match "fn" abbreviation
- Add Cc stable
- Rebase on v6.18-rc3
- Link to v4: https://lore.kernel.org/r/20251023-ari_no_bus_dev-v4-0-8482e2ed10bd@linux.ibm.com

Changes in v4:
- Add fix of isolated function probing with ARI enabled and in certain
  cases with SR-IOV devices.
- Link to v3: https://lore.kernel.org/r/20250417-ari_no_bus_dev-v3-1-ba48f349aa47@linux.ibm.com

Changes in v3:
- Move setting of ari_enabled on s390 to bus creation and clear it in
  pcibios_add_device() if the capability is not available.
- Rebase on v6.15-rc2
- Link to v2: https://lore.kernel.org/r/20240918-ari_no_bus_dev-v2-1-83cfa991082f@linux.ibm.com

Changes in v2:
- Rebased on v6.11
- Link to v1: https://lore.kernel.org/r/20240730-ari_no_bus_dev-v1-1-7de17676f9fe@linux.ibm.com

---
Niklas Schnelle (2):
      PCI: Fix isolated PCI function probing with ARI and SR-IOV
      PCI: s390: Handle ARI on bus without associated struct pci_dev

 arch/s390/pci/pci.c     |  7 +++++++
 arch/s390/pci/pci_bus.c | 10 ++++++++++
 drivers/pci/pci.c       |  4 ++--
 drivers/pci/probe.c     | 22 ++++++++++++++--------
 include/linux/pci.h     |  4 ++--
 5 files changed, 35 insertions(+), 12 deletions(-)
---
base-commit: dcb6fa37fd7bc9c3d2b066329b0d27dedf8becaa
change-id: 20240724-ari_no_bus_dev-52b2a27f3466

Best regards,
-- 
Niklas Schnelle

[PATCH v5 1/2] PCI: Fix isolated PCI function probing with ARI and SR-IOV

[Niklas Schnelle]

When the isolated PCI function probing mechanism is used in conjunction
with ARI or SR-IOV it may not find all available PCI functions. In the
case of ARI the problem is that next_ari_fn() always returns -ENODEV if
dev is NULL and thus if fn 0 is missing the scan stops.

For SR-IOV things are more complex. Here the problem is that the check
for multifunction may fail. One example where this can occur is if the
first passed-through function is a VF with devfn 8. Now in
pci_scan_slot() this means it is fn 0 and thus multifunction doesn't get
set. Since VFs don't get multifunction set via PCI_HEADER_TYPE_MFD it
remains unset and probing stops even if there is a devfn 9.

Now at the moment both of these issues are hidden on s390. The first one
because ARI is detected as disabled as struct pci_bus's self is NULL
even though firmware does enable and use ARI. The second issue is hidden
as a side effect of commit 25f39d3dcb48 ("s390/pci: Ignore RID for
isolated VFs"). This is because VFs are either put on their own virtual
bus if the parent PF is not passed-through to the same instance or VFs
are hotplugged once SR-IOV is enabled on the parent PF and then
pci_scan_single_device() is used.

Still especially the first issue prevents correct detection of ARI and
the second might be a problem for other users of isolated function
probing. Fix both issues by keeping things as simple as possible. If
isolated function probing is enabled simply scan every possible devfn.

Cc: stable@vger.kernel.org
Fixes: 189c6c33ff42 ("PCI: Extend isolated function probing to s390")
Link: https://lore.kernel.org/linux-pci/d3f11e8562f589ddb2c1c83e74161bd8948084c3.camel@linux.ibm.com/
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
---
 drivers/pci/probe.c | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)

diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 0ce98e18b5a876afe72af35a9f4a44d598e8d500..41dd1a339a994956a6bc7e1fea0fe0d55452a963 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -2808,16 +2808,19 @@ static int next_ari_fn(struct pci_bus *bus, struct pci_dev *dev, int fn)
 	return next_fn;
 }
 
-static int next_fn(struct pci_bus *bus, struct pci_dev *dev, int fn)
+static int next_fn(struct pci_bus *bus, struct pci_dev *dev, int fn,
+		   bool isolated_fns)
 {
-	if (pci_ari_enabled(bus))
-		return next_ari_fn(bus, dev, fn);
+	if (!isolated_fns) {
+		if (pci_ari_enabled(bus))
+			return next_ari_fn(bus, dev, fn);
 
+		/* only multifunction devices may have more functions */
+		if (dev && !dev->multifunction)
+			return -ENODEV;
+	}
 	if (fn >= 7)
 		return -ENODEV;
-	/* only multifunction devices may have more functions */
-	if (dev && !dev->multifunction)
-		return -ENODEV;
 
 	return fn + 1;
 }
@@ -2859,10 +2862,12 @@ int pci_scan_slot(struct pci_bus *bus, int devfn)
 {
 	struct pci_dev *dev;
 	int fn = 0, nr = 0;
+	bool isolated_fns;
 
 	if (only_one_child(bus) && (devfn > 0))
 		return 0; /* Already scanned the entire slot */
 
+	isolated_fns = hypervisor_isolated_pci_functions();
 	do {
 		dev = pci_scan_single_device(bus, devfn + fn);
 		if (dev) {
@@ -2876,10 +2881,10 @@ int pci_scan_slot(struct pci_bus *bus, int devfn)
 			 * a hypervisor that passes through individual PCI
 			 * functions.
 			 */
-			if (!hypervisor_isolated_pci_functions())
+			if (!isolated_fns)
 				break;
 		}
-		fn = next_fn(bus, dev, fn);
+		fn = next_fn(bus, dev, fn, isolated_fns);
 	} while (fn >= 0);
 
 	/* Only one slot has PCIe device */

-- 
2.48.1

Re: [PATCH v5 1/2] PCI: Fix isolated PCI function probing with ARI and SR-IOV

[Huacai Chen]

Hi, Niklas,

On Wed, Oct 29, 2025 at 5:42 PM Niklas Schnelle <schnelle@linux.ibm.com> wrote:
>
> When the isolated PCI function probing mechanism is used in conjunction
> with ARI or SR-IOV it may not find all available PCI functions. In the
> case of ARI the problem is that next_ari_fn() always returns -ENODEV if
> dev is NULL and thus if fn 0 is missing the scan stops.
>
> For SR-IOV things are more complex. Here the problem is that the check
> for multifunction may fail. One example where this can occur is if the
> first passed-through function is a VF with devfn 8. Now in
> pci_scan_slot() this means it is fn 0 and thus multifunction doesn't get
> set. Since VFs don't get multifunction set via PCI_HEADER_TYPE_MFD it
> remains unset and probing stops even if there is a devfn 9.
>
> Now at the moment both of these issues are hidden on s390. The first one
> because ARI is detected as disabled as struct pci_bus's self is NULL
> even though firmware does enable and use ARI. The second issue is hidden
> as a side effect of commit 25f39d3dcb48 ("s390/pci: Ignore RID for
> isolated VFs"). This is because VFs are either put on their own virtual
> bus if the parent PF is not passed-through to the same instance or VFs
> are hotplugged once SR-IOV is enabled on the parent PF and then
> pci_scan_single_device() is used.
>
> Still especially the first issue prevents correct detection of ARI and
> the second might be a problem for other users of isolated function
> probing. Fix both issues by keeping things as simple as possible. If
> isolated function probing is enabled simply scan every possible devfn.
I'm very sorry, but applying this patch on top of commit a02fd05661d7
("PCI: Extend isolated function probing to LoongArch") we fail to
boot.

Boot log:
[   10.365340] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16
00:01:03 EST 2006)
[   10.372628] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
[   10.379564] megasas: 07.734.00.00-rc1
[   10.383222] mpt3sas version 54.100.00.00 loaded
[   10.388304] nvme nvme0: pci function 0000:08:00.0
[   10.395088] Freeing initrd memory: 45632K
[   10.469822] ------------[ cut here ]------------
[   10.474409] WARNING: CPU: 0 PID: 247 at drivers/ata/libahci.c:233
ahci_enable_ahci+0x64/0xb8
[   10.482804] Modules linked in:
[   10.485838] CPU: 0 UID: 0 PID: 247 Comm: kworker/0:11 Not tainted
6.18.0-rc3 #1 PREEMPT(full)
[   10.494397] Hardware name: To be filled by O.E.M.To be fill To be
filled by O.E.M.To be fill/To be filled by O.E.M.To be fill, BIOS
Loongson-UDK2018-V4.0.
[   10.508139] Workqueue: events work_for_cpu_fn
[   10.512468] pc 900000000103be2c ra 900000000103be28 tp
900000010ae44000 sp 900000010ae47be0
[   10.520769] a0 0000000000000000 a1 00000000000000b0 a2
0000000000000001 a3 9000000001810e0c
[   10.529069] a4 9000000002343e20 a5 0000000000000001 a6
0000000000000010 a7 0000000000000000
[   10.537373] t0 d10951fa66920f31 t1 d10951fa66920f31 t2
0000000000001280 t3 000000000674c000
[   10.545673] t4 0000000000000000 t5 0000000000000000 t6
9000000008002480 t7 00000000000000b4
[   10.553972] t8 90000001055eab90 u0 900000010ae47b68 s9
9000000002221a50 s0 0000000000000000
[   10.562272] s1 ffff800032435800 s2 0000000000000000 s3
ffffffff80000000 s4 9000000002221570
[   10.570571] s5 0000000000000005 s6 9000000101ccf0b8 s7
90000000023dd000 s8 900000010ae47d08
[   10.578869]    ra: 900000000103be28 ahci_enable_ahci+0x60/0xb8
[   10.584665]   ERA: 900000000103be2c ahci_enable_ahci+0x64/0xb8
[   10.590461]  CRMD: 000000b0 (PLV0 -IE -DA +PG DACF=CC DACM=CC -WE)
[   10.596609]  PRMD: 00000004 (PPLV0 +PIE -PWE)
[   10.600937]  EUEN: 00000000 (-FPE -SXE -ASXE -BTE)
[   10.605698]  ECFG: 00071c1d (LIE=0,2-4,10-12 VS=7)
[   10.610458] ESTAT: 000c0000 [BRK] (IS= ECode=12 EsubCode=0)
[   10.615994]  PRID: 0014d010 (Loongson-64bit, Loongson-3C6000/S)
[   10.621875] CPU: 0 UID: 0 PID: 247 Comm: kworker/0:11 Not tainted
6.18.0-rc3 #1 PREEMPT(full)
[   10.621877] Hardware name: To be filled by O.E.M.To be fill To be
filled by O.E.M.To be fill/To be filled by O.E.M.To be fill, BIOS
Loongson-UDK2018-V4.0.
[   10.621878] Workqueue: events work_for_cpu_fn
[   10.621881] Stack : 900000010ae47848 0000000000000000
90000000002436bc 900000010ae44000
[   10.621884]         900000010ae47820 900000010ae47828
0000000000000000 900000010ae47968
[   10.621887]         900000010ae47960 900000010ae47960
900000010ae47630 0000000000000001
[   10.621890]         0000000000000001 900000010ae47828
d10951fa66920f31 9000000100414300
[   10.621893]         80000000ffffe34d fffffffffffffffe
000000000000034f 000000000000002f
[   10.621896]         0000000000000063 0000000000000001
000000000674c000 9000000002221a50
[   10.621899]         0000000000000000 0000000000000000
90000000020b6500 90000000023dd000
[   10.621902]         00000000000000e9 0000000000000009
0000000000000002 90000000023dd000
[   10.621905]         900000010ae47d08 0000000000000000
90000000002436d4 0000000000000000
[   10.621908]         00000000000000b0 0000000000000004
0000000000000000 0000000000071c1d
[   10.621910]         ...
[   10.621912] Call Trace:
[   10.621913] [<90000000002436d4>] show_stack+0x5c/0x180
[   10.621918] [<900000000023f328>] dump_stack_lvl+0x6c/0x9c
[   10.621923] [<9000000000266eb8>] __warn+0x80/0x108
[   10.621927] [<90000000017d1910>] report_bug+0x158/0x2a8
[   10.621932] [<900000000180b610>] do_bp+0x2d0/0x340
[   10.621938] [<9000000000241da0>] handle_bp+0x120/0x1c0
[   10.621940] [<900000000103be2c>] ahci_enable_ahci+0x64/0xb8
[   10.621943] [<900000000103beb8>] ahci_save_initial_config+0x38/0x4d8
[   10.621946] [<90000000010391b4>] ahci_init_one+0x354/0x1088
[   10.621950] [<9000000000d16cdc>] local_pci_probe+0x44/0xb8
[   10.621953] [<9000000000286f78>] work_for_cpu_fn+0x18/0x30
[   10.621956] [<900000000028a840>] process_one_work+0x160/0x330
[   10.621961] [<900000000028b208>] worker_thread+0x330/0x460
[   10.621964] [<9000000000295fdc>] kthread+0x11c/0x138
[   10.621968] [<900000000180b740>] ret_from_kernel_thread+0x28/0xa8
[   10.621971] [<9000000000241484>] ret_from_kernel_thread_asm+0xc/0x88
[   10.621973]

Huacai

>
> Cc: stable@vger.kernel.org
> Fixes: 189c6c33ff42 ("PCI: Extend isolated function probing to s390")
> Link: https://lore.kernel.org/linux-pci/d3f11e8562f589ddb2c1c83e74161bd8948084c3.camel@linux.ibm.com/
> Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
> ---
>  drivers/pci/probe.c | 21 +++++++++++++--------
>  1 file changed, 13 insertions(+), 8 deletions(-)
>
> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
> index 0ce98e18b5a876afe72af35a9f4a44d598e8d500..41dd1a339a994956a6bc7e1fea0fe0d55452a963 100644
> --- a/drivers/pci/probe.c
> +++ b/drivers/pci/probe.c
> @@ -2808,16 +2808,19 @@ static int next_ari_fn(struct pci_bus *bus, struct pci_dev *dev, int fn)
>         return next_fn;
>  }
>
> -static int next_fn(struct pci_bus *bus, struct pci_dev *dev, int fn)
> +static int next_fn(struct pci_bus *bus, struct pci_dev *dev, int fn,
> +                  bool isolated_fns)
>  {
> -       if (pci_ari_enabled(bus))
> -               return next_ari_fn(bus, dev, fn);
> +       if (!isolated_fns) {
> +               if (pci_ari_enabled(bus))
> +                       return next_ari_fn(bus, dev, fn);
>
> +               /* only multifunction devices may have more functions */
> +               if (dev && !dev->multifunction)
> +                       return -ENODEV;
> +       }
>         if (fn >= 7)
>                 return -ENODEV;
> -       /* only multifunction devices may have more functions */
> -       if (dev && !dev->multifunction)
> -               return -ENODEV;
>
>         return fn + 1;
>  }
> @@ -2859,10 +2862,12 @@ int pci_scan_slot(struct pci_bus *bus, int devfn)
>  {
>         struct pci_dev *dev;
>         int fn = 0, nr = 0;
> +       bool isolated_fns;
>
>         if (only_one_child(bus) && (devfn > 0))
>                 return 0; /* Already scanned the entire slot */
>
> +       isolated_fns = hypervisor_isolated_pci_functions();
>         do {
>                 dev = pci_scan_single_device(bus, devfn + fn);
>                 if (dev) {
> @@ -2876,10 +2881,10 @@ int pci_scan_slot(struct pci_bus *bus, int devfn)
>                          * a hypervisor that passes through individual PCI
>                          * functions.
>                          */
> -                       if (!hypervisor_isolated_pci_functions())
> +                       if (!isolated_fns)
>                                 break;
>                 }
> -               fn = next_fn(bus, dev, fn);
> +               fn = next_fn(bus, dev, fn, isolated_fns);
>         } while (fn >= 0);
>
>         /* Only one slot has PCIe device */
>
> --
> 2.48.1
>

Re: [PATCH v5 1/2] PCI: Fix isolated PCI function probing with ARI and SR-IOV

[Niklas Schnelle]

On Mon, 2025-11-03 at 17:50 +0800, Huacai Chen wrote:
> Hi, Niklas,
> 
> On Wed, Oct 29, 2025 at 5:42 PM Niklas Schnelle <schnelle@linux.ibm.com> wrote:
> > 
> > When the isolated PCI function probing mechanism is used in conjunction
> > with ARI or SR-IOV it may not find all available PCI functions. In the
> > case of ARI the problem is that next_ari_fn() always returns -ENODEV if
> > dev is NULL and thus if fn 0 is missing the scan stops.
> > 
> > For SR-IOV things are more complex. Here the problem is that the check
> > for multifunction may fail. One example where this can occur is if the
> > first passed-through function is a VF with devfn 8. Now in
> > pci_scan_slot() this means it is fn 0 and thus multifunction doesn't get
> > set. Since VFs don't get multifunction set via PCI_HEADER_TYPE_MFD it
> > remains unset and probing stops even if there is a devfn 9.
> > 
> > Now at the moment both of these issues are hidden on s390. The first one
> > because ARI is detected as disabled as struct pci_bus's self is NULL
> > even though firmware does enable and use ARI. The second issue is hidden
> > as a side effect of commit 25f39d3dcb48 ("s390/pci: Ignore RID for
> > isolated VFs"). This is because VFs are either put on their own virtual
> > bus if the parent PF is not passed-through to the same instance or VFs
> > are hotplugged once SR-IOV is enabled on the parent PF and then
> > pci_scan_single_device() is used.
> > 
> > Still especially the first issue prevents correct detection of ARI and
> > the second might be a problem for other users of isolated function
> > probing. Fix both issues by keeping things as simple as possible. If
> > isolated function probing is enabled simply scan every possible devfn.
> I'm very sorry, but applying this patch on top of commit a02fd05661d7
> ("PCI: Extend isolated function probing to LoongArch") we fail to
> boot.
> 
> Boot log:
> [   10.365340] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16
> 00:01:03 EST 2006)
> [   10.372628] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
> [   10.379564] megasas: 07.734.00.00-rc1
> [   10.383222] mpt3sas version 54.100.00.00 loaded
> [   10.388304] nvme nvme0: pci function 0000:08:00.0
> [   10.395088] Freeing initrd memory: 45632K
> [   10.469822] ------------[ cut here ]------------
> [   10.474409] WARNING: CPU: 0 PID: 247 at drivers/ata/libahci.c:233
> ahci_enable_ahci+0x64/0xb8
> [   10.482804] Modules linked in:
> [   10.485838] CPU: 0 UID: 0 PID: 247 Comm: kworker/0:11 Not tainted
> 6.18.0-rc3 #1 PREEMPT(full)
> [   10.494397] Hardware name: To be filled by O.E.M.To be fill To be
> filled by O.E.M.To be fill/To be filled by O.E.M.To be fill, BIOS
> Loongson-UDK2018-V4.0.
> [   10.508139] Workqueue: events work_for_cpu_fn
> [   10.512468] pc 900000000103be2c ra 900000000103be28 tp
> 900000010ae44000 sp 900000010ae47be0
> [   10.520769] a0 0000000000000000 a1 00000000000000b0 a2
> 0000000000000001 a3 9000000001810e0c
> [   10.529069] a4 9000000002343e20 a5 0000000000000001 a6
> 0000000000000010 a7 0000000000000000
> [   10.537373] t0 d10951fa66920f31 t1 d10951fa66920f31 t2
> 0000000000001280 t3 000000000674c000
> [   10.545673] t4 0000000000000000 t5 0000000000000000 t6
> 9000000008002480 t7 00000000000000b4
> [   10.553972] t8 90000001055eab90 u0 900000010ae47b68 s9
> 9000000002221a50 s0 0000000000000000
> [   10.562272] s1 ffff800032435800 s2 0000000000000000 s3
> ffffffff80000000 s4 9000000002221570
> [   10.570571] s5 0000000000000005 s6 9000000101ccf0b8 s7
> 90000000023dd000 s8 900000010ae47d08
> [   10.578869]    ra: 900000000103be28 ahci_enable_ahci+0x60/0xb8
> [   10.584665]   ERA: 900000000103be2c ahci_enable_ahci+0x64/0xb8
> [   10.590461]  CRMD: 000000b0 (PLV0 -IE -DA +PG DACF=CC DACM=CC -WE)
> [   10.596609]  PRMD: 00000004 (PPLV0 +PIE -PWE)
> [   10.600937]  EUEN: 00000000 (-FPE -SXE -ASXE -BTE)
> [   10.605698]  ECFG: 00071c1d (LIE=0,2-4,10-12 VS=7)
> [   10.610458] ESTAT: 000c0000 [BRK] (IS= ECode=12 EsubCode=0)
> [   10.615994]  PRID: 0014d010 (Loongson-64bit, Loongson-3C6000/S)
> [   10.621875] CPU: 0 UID: 0 PID: 247 Comm: kworker/0:11 Not tainted
> 6.18.0-rc3 #1 PREEMPT(full)
> [   10.621877] Hardware name: To be filled by O.E.M.To be fill To be
> filled by O.E.M.To be fill/To be filled by O.E.M.To be fill, BIOS
> Loongson-UDK2018-V4.0.
> [   10.621878] Workqueue: events work_for_cpu_fn
> [   10.621881] Stack : 900000010ae47848 0000000000000000
> 90000000002436bc 900000010ae44000
> [   10.621884]         900000010ae47820 900000010ae47828
> 0000000000000000 900000010ae47968
> [   10.621887]         900000010ae47960 900000010ae47960
> 900000010ae47630 0000000000000001
> [   10.621890]         0000000000000001 900000010ae47828
> d10951fa66920f31 9000000100414300
> [   10.621893]         80000000ffffe34d fffffffffffffffe
> 000000000000034f 000000000000002f
> [   10.621896]         0000000000000063 0000000000000001
> 000000000674c000 9000000002221a50
> [   10.621899]         0000000000000000 0000000000000000
> 90000000020b6500 90000000023dd000
> [   10.621902]         00000000000000e9 0000000000000009
> 0000000000000002 90000000023dd000
> [   10.621905]         900000010ae47d08 0000000000000000
> 90000000002436d4 0000000000000000
> [   10.621908]         00000000000000b0 0000000000000004
> 0000000000000000 0000000000071c1d
> [   10.621910]         ...
> [   10.621912] Call Trace:
> [   10.621913] [<90000000002436d4>] show_stack+0x5c/0x180
> [   10.621918] [<900000000023f328>] dump_stack_lvl+0x6c/0x9c
> [   10.621923] [<9000000000266eb8>] __warn+0x80/0x108
> [   10.621927] [<90000000017d1910>] report_bug+0x158/0x2a8
> [   10.621932] [<900000000180b610>] do_bp+0x2d0/0x340
> [   10.621938] [<9000000000241da0>] handle_bp+0x120/0x1c0
> [   10.621940] [<900000000103be2c>] ahci_enable_ahci+0x64/0xb8
> [   10.621943] [<900000000103beb8>] ahci_save_initial_config+0x38/0x4d8
> [   10.621946] [<90000000010391b4>] ahci_init_one+0x354/0x1088
> [   10.621950] [<9000000000d16cdc>] local_pci_probe+0x44/0xb8
> [   10.621953] [<9000000000286f78>] work_for_cpu_fn+0x18/0x30
> [   10.621956] [<900000000028a840>] process_one_work+0x160/0x330
> [   10.621961] [<900000000028b208>] worker_thread+0x330/0x460
> [   10.621964] [<9000000000295fdc>] kthread+0x11c/0x138
> [   10.621968] [<900000000180b740>] ret_from_kernel_thread+0x28/0xa8
> [   10.621971] [<9000000000241484>] ret_from_kernel_thread_asm+0xc/0x88
> [   10.621973]
> 
> 

This looks like a warning telling us that AHCI enable failed / timed
out. Do you have Panic on Warn on that this directly causes a boot
failure? The only relation I can see with my patch is that maybe this
AHCI device wasn't probed before and somehow isn't working?

Thanks,
Niklas

Re: [PATCH v5 1/2] PCI: Fix isolated PCI function probing with ARI and SR-IOV

[Huacai Chen]

On Mon, Nov 3, 2025 at 7:23 PM Niklas Schnelle <schnelle@linux.ibm.com> wrote:
>
> On Mon, 2025-11-03 at 17:50 +0800, Huacai Chen wrote:
> > Hi, Niklas,
> >
> > On Wed, Oct 29, 2025 at 5:42 PM Niklas Schnelle <schnelle@linux.ibm.com> wrote:
> > >
> > > When the isolated PCI function probing mechanism is used in conjunction
> > > with ARI or SR-IOV it may not find all available PCI functions. In the
> > > case of ARI the problem is that next_ari_fn() always returns -ENODEV if
> > > dev is NULL and thus if fn 0 is missing the scan stops.
> > >
> > > For SR-IOV things are more complex. Here the problem is that the check
> > > for multifunction may fail. One example where this can occur is if the
> > > first passed-through function is a VF with devfn 8. Now in
> > > pci_scan_slot() this means it is fn 0 and thus multifunction doesn't get
> > > set. Since VFs don't get multifunction set via PCI_HEADER_TYPE_MFD it
> > > remains unset and probing stops even if there is a devfn 9.
> > >
> > > Now at the moment both of these issues are hidden on s390. The first one
> > > because ARI is detected as disabled as struct pci_bus's self is NULL
> > > even though firmware does enable and use ARI. The second issue is hidden
> > > as a side effect of commit 25f39d3dcb48 ("s390/pci: Ignore RID for
> > > isolated VFs"). This is because VFs are either put on their own virtual
> > > bus if the parent PF is not passed-through to the same instance or VFs
> > > are hotplugged once SR-IOV is enabled on the parent PF and then
> > > pci_scan_single_device() is used.
> > >
> > > Still especially the first issue prevents correct detection of ARI and
> > > the second might be a problem for other users of isolated function
> > > probing. Fix both issues by keeping things as simple as possible. If
> > > isolated function probing is enabled simply scan every possible devfn.
> > I'm very sorry, but applying this patch on top of commit a02fd05661d7
> > ("PCI: Extend isolated function probing to LoongArch") we fail to
> > boot.
> >
> > Boot log:
> > [   10.365340] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16
> > 00:01:03 EST 2006)
> > [   10.372628] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
> > [   10.379564] megasas: 07.734.00.00-rc1
> > [   10.383222] mpt3sas version 54.100.00.00 loaded
> > [   10.388304] nvme nvme0: pci function 0000:08:00.0
> > [   10.395088] Freeing initrd memory: 45632K
> > [   10.469822] ------------[ cut here ]------------
> > [   10.474409] WARNING: CPU: 0 PID: 247 at drivers/ata/libahci.c:233
> > ahci_enable_ahci+0x64/0xb8
> > [   10.482804] Modules linked in:
> > [   10.485838] CPU: 0 UID: 0 PID: 247 Comm: kworker/0:11 Not tainted
> > 6.18.0-rc3 #1 PREEMPT(full)
> > [   10.494397] Hardware name: To be filled by O.E.M.To be fill To be
> > filled by O.E.M.To be fill/To be filled by O.E.M.To be fill, BIOS
> > Loongson-UDK2018-V4.0.
> > [   10.508139] Workqueue: events work_for_cpu_fn
> > [   10.512468] pc 900000000103be2c ra 900000000103be28 tp
> > 900000010ae44000 sp 900000010ae47be0
> > [   10.520769] a0 0000000000000000 a1 00000000000000b0 a2
> > 0000000000000001 a3 9000000001810e0c
> > [   10.529069] a4 9000000002343e20 a5 0000000000000001 a6
> > 0000000000000010 a7 0000000000000000
> > [   10.537373] t0 d10951fa66920f31 t1 d10951fa66920f31 t2
> > 0000000000001280 t3 000000000674c000
> > [   10.545673] t4 0000000000000000 t5 0000000000000000 t6
> > 9000000008002480 t7 00000000000000b4
> > [   10.553972] t8 90000001055eab90 u0 900000010ae47b68 s9
> > 9000000002221a50 s0 0000000000000000
> > [   10.562272] s1 ffff800032435800 s2 0000000000000000 s3
> > ffffffff80000000 s4 9000000002221570
> > [   10.570571] s5 0000000000000005 s6 9000000101ccf0b8 s7
> > 90000000023dd000 s8 900000010ae47d08
> > [   10.578869]    ra: 900000000103be28 ahci_enable_ahci+0x60/0xb8
> > [   10.584665]   ERA: 900000000103be2c ahci_enable_ahci+0x64/0xb8
> > [   10.590461]  CRMD: 000000b0 (PLV0 -IE -DA +PG DACF=CC DACM=CC -WE)
> > [   10.596609]  PRMD: 00000004 (PPLV0 +PIE -PWE)
> > [   10.600937]  EUEN: 00000000 (-FPE -SXE -ASXE -BTE)
> > [   10.605698]  ECFG: 00071c1d (LIE=0,2-4,10-12 VS=7)
> > [   10.610458] ESTAT: 000c0000 [BRK] (IS= ECode=12 EsubCode=0)
> > [   10.615994]  PRID: 0014d010 (Loongson-64bit, Loongson-3C6000/S)
> > [   10.621875] CPU: 0 UID: 0 PID: 247 Comm: kworker/0:11 Not tainted
> > 6.18.0-rc3 #1 PREEMPT(full)
> > [   10.621877] Hardware name: To be filled by O.E.M.To be fill To be
> > filled by O.E.M.To be fill/To be filled by O.E.M.To be fill, BIOS
> > Loongson-UDK2018-V4.0.
> > [   10.621878] Workqueue: events work_for_cpu_fn
> > [   10.621881] Stack : 900000010ae47848 0000000000000000
> > 90000000002436bc 900000010ae44000
> > [   10.621884]         900000010ae47820 900000010ae47828
> > 0000000000000000 900000010ae47968
> > [   10.621887]         900000010ae47960 900000010ae47960
> > 900000010ae47630 0000000000000001
> > [   10.621890]         0000000000000001 900000010ae47828
> > d10951fa66920f31 9000000100414300
> > [   10.621893]         80000000ffffe34d fffffffffffffffe
> > 000000000000034f 000000000000002f
> > [   10.621896]         0000000000000063 0000000000000001
> > 000000000674c000 9000000002221a50
> > [   10.621899]         0000000000000000 0000000000000000
> > 90000000020b6500 90000000023dd000
> > [   10.621902]         00000000000000e9 0000000000000009
> > 0000000000000002 90000000023dd000
> > [   10.621905]         900000010ae47d08 0000000000000000
> > 90000000002436d4 0000000000000000
> > [   10.621908]         00000000000000b0 0000000000000004
> > 0000000000000000 0000000000071c1d
> > [   10.621910]         ...
> > [   10.621912] Call Trace:
> > [   10.621913] [<90000000002436d4>] show_stack+0x5c/0x180
> > [   10.621918] [<900000000023f328>] dump_stack_lvl+0x6c/0x9c
> > [   10.621923] [<9000000000266eb8>] __warn+0x80/0x108
> > [   10.621927] [<90000000017d1910>] report_bug+0x158/0x2a8
> > [   10.621932] [<900000000180b610>] do_bp+0x2d0/0x340
> > [   10.621938] [<9000000000241da0>] handle_bp+0x120/0x1c0
> > [   10.621940] [<900000000103be2c>] ahci_enable_ahci+0x64/0xb8
> > [   10.621943] [<900000000103beb8>] ahci_save_initial_config+0x38/0x4d8
> > [   10.621946] [<90000000010391b4>] ahci_init_one+0x354/0x1088
> > [   10.621950] [<9000000000d16cdc>] local_pci_probe+0x44/0xb8
> > [   10.621953] [<9000000000286f78>] work_for_cpu_fn+0x18/0x30
> > [   10.621956] [<900000000028a840>] process_one_work+0x160/0x330
> > [   10.621961] [<900000000028b208>] worker_thread+0x330/0x460
> > [   10.621964] [<9000000000295fdc>] kthread+0x11c/0x138
> > [   10.621968] [<900000000180b740>] ret_from_kernel_thread+0x28/0xa8
> > [   10.621971] [<9000000000241484>] ret_from_kernel_thread_asm+0xc/0x88
> > [   10.621973]
> >
> >
>
> This looks like a warning telling us that AHCI enable failed / timed
> out. Do you have Panic on Warn on that this directly causes a boot
> failure? The only relation I can see with my patch is that maybe this
> AHCI device wasn't probed before and somehow isn't working?
The rootfs is on the AHCI controller, so AHCI failure causes the boot
failure, without this patch no boot problems.

Huacai

>
> Thanks,
> Niklas

Re: [PATCH v5 1/2] PCI: Fix isolated PCI function probing with ARI and SR-IOV

[Niklas Schnelle]

On Wed, 2025-11-05 at 09:01 +0800, Huacai Chen wrote:
> On Mon, Nov 3, 2025 at 7:23 PM Niklas Schnelle <schnelle@linux.ibm.com> wrote:
> > 
> > On Mon, 2025-11-03 at 17:50 +0800, Huacai Chen wrote:
> > > Hi, Niklas,
> > > 
> > > On Wed, Oct 29, 2025 at 5:42 PM Niklas Schnelle <schnelle@linux.ibm.com> wrote:
> > > > 
> > > > When the isolated PCI function probing mechanism is used in conjunction
> > > > with ARI or SR-IOV it may not find all available PCI functions. In the
> > > > case of ARI the problem is that next_ari_fn() always returns -ENODEV if
> > > > dev is NULL and thus if fn 0 is missing the scan stops.
> > > > 
> > > > For SR-IOV things are more complex. Here the problem is that the check
> > > > for multifunction may fail. One example where this can occur is if the
> > > > first passed-through function is a VF with devfn 8. Now in
> > > > pci_scan_slot() this means it is fn 0 and thus multifunction doesn't get
> > > > set. Since VFs don't get multifunction set via PCI_HEADER_TYPE_MFD it
> > > > remains unset and probing stops even if there is a devfn 9.
> > > > 
> > > > Now at the moment both of these issues are hidden on s390. The first one
> > > > because ARI is detected as disabled as struct pci_bus's self is NULL
> > > > even though firmware does enable and use ARI. The second issue is hidden
> > > > as a side effect of commit 25f39d3dcb48 ("s390/pci: Ignore RID for
> > > > isolated VFs"). This is because VFs are either put on their own virtual
> > > > bus if the parent PF is not passed-through to the same instance or VFs
> > > > are hotplugged once SR-IOV is enabled on the parent PF and then
> > > > pci_scan_single_device() is used.
> > > > 
> > > > Still especially the first issue prevents correct detection of ARI and
> > > > the second might be a problem for other users of isolated function
> > > > probing. Fix both issues by keeping things as simple as possible. If
> > > > isolated function probing is enabled simply scan every possible devfn.
> > > I'm very sorry, but applying this patch on top of commit a02fd05661d7
> > > ("PCI: Extend isolated function probing to LoongArch") we fail to
> > > boot.
> > > 
> > > Boot log:
> > > [   10.365340] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16
> > > 00:01:03 EST 2006)
> > > [   10.372628] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
> > > [   10.379564] megasas: 07.734.00.00-rc1
> > > [   10.383222] mpt3sas version 54.100.00.00 loaded
> > > [   10.388304] nvme nvme0: pci function 0000:08:00.0
> > > [   10.395088] Freeing initrd memory: 45632K
> > > [   10.469822] ------------[ cut here ]------------
> > > [   10.474409] WARNING: CPU: 0 PID: 247 at drivers/ata/libahci.c:233
> > > ahci_enable_ahci+0x64/0xb8
> > > [   10.482804] Modules linked in:
> > > [   10.485838] CPU: 0 UID: 0 PID: 247 Comm: kworker/0:11 Not tainted
> > > 6.18.0-rc3 #1 PREEMPT(full)
> > > [   10.494397] Hardware name: To be filled by O.E.M.To be fill To be
> > > filled by O.E.M.To be fill/To be filled by O.E.M.To be fill, BIOS
> > > Loongson-UDK2018-V4.0.
> > > [   10.508139] Workqueue: events work_for_cpu_fn
> > > [   10.512468] pc 900000000103be2c ra 900000000103be28 tp
> > > 900000010ae44000 sp 900000010ae47be0
> > > [   10.520769] a0 0000000000000000 a1 00000000000000b0 a2
> > > 0000000000000001 a3 9000000001810e0c
> > > [   10.529069] a4 9000000002343e20 a5 0000000000000001 a6
> > > 0000000000000010 a7 0000000000000000
> > > [   10.537373] t0 d10951fa66920f31 t1 d10951fa66920f31 t2
> > > 0000000000001280 t3 000000000674c000
> > > [   10.545673] t4 0000000000000000 t5 0000000000000000 t6
> > > 9000000008002480 t7 00000000000000b4
> > > [   10.553972] t8 90000001055eab90 u0 900000010ae47b68 s9
> > > 9000000002221a50 s0 0000000000000000
> > > [   10.562272] s1 ffff800032435800 s2 0000000000000000 s3
> > > ffffffff80000000 s4 9000000002221570
> > > [   10.570571] s5 0000000000000005 s6 9000000101ccf0b8 s7
> > > 90000000023dd000 s8 900000010ae47d08
> > > [   10.578869]    ra: 900000000103be28 ahci_enable_ahci+0x60/0xb8
> > > [   10.584665]   ERA: 900000000103be2c ahci_enable_ahci+0x64/0xb8
> > > [   10.590461]  CRMD: 000000b0 (PLV0 -IE -DA +PG DACF=CC DACM=CC -WE)
> > > [   10.596609]  PRMD: 00000004 (PPLV0 +PIE -PWE)
> > > [   10.600937]  EUEN: 00000000 (-FPE -SXE -ASXE -BTE)
> > > [   10.605698]  ECFG: 00071c1d (LIE=0,2-4,10-12 VS=7)
> > > [   10.610458] ESTAT: 000c0000 [BRK] (IS= ECode=12 EsubCode=0)
> > > [   10.615994]  PRID: 0014d010 (Loongson-64bit, Loongson-3C6000/S)
> > > [   10.621875] CPU: 0 UID: 0 PID: 247 Comm: kworker/0:11 Not tainted
> > > 6.18.0-rc3 #1 PREEMPT(full)
> > > [   10.621877] Hardware name: To be filled by O.E.M.To be fill To be
> > > filled by O.E.M.To be fill/To be filled by O.E.M.To be fill, BIOS
> > > Loongson-UDK2018-V4.0.
> > > [   10.621878] Workqueue: events work_for_cpu_fn
> > > [   10.621881] Stack : 900000010ae47848 0000000000000000
> > > 90000000002436bc 900000010ae44000
> > > [   10.621884]         900000010ae47820 900000010ae47828
> > > 0000000000000000 900000010ae47968
> > > [   10.621887]         900000010ae47960 900000010ae47960
> > > 900000010ae47630 0000000000000001
> > > [   10.621890]         0000000000000001 900000010ae47828
> > > d10951fa66920f31 9000000100414300
> > > [   10.621893]         80000000ffffe34d fffffffffffffffe
> > > 000000000000034f 000000000000002f
> > > [   10.621896]         0000000000000063 0000000000000001
> > > 000000000674c000 9000000002221a50
> > > [   10.621899]         0000000000000000 0000000000000000
> > > 90000000020b6500 90000000023dd000
> > > [   10.621902]         00000000000000e9 0000000000000009
> > > 0000000000000002 90000000023dd000
> > > [   10.621905]         900000010ae47d08 0000000000000000
> > > 90000000002436d4 0000000000000000
> > > [   10.621908]         00000000000000b0 0000000000000004
> > > 0000000000000000 0000000000071c1d
> > > [   10.621910]         ...
> > > [   10.621912] Call Trace:
> > > [   10.621913] [<90000000002436d4>] show_stack+0x5c/0x180
> > > [   10.621918] [<900000000023f328>] dump_stack_lvl+0x6c/0x9c
> > > [   10.621923] [<9000000000266eb8>] __warn+0x80/0x108
> > > [   10.621927] [<90000000017d1910>] report_bug+0x158/0x2a8
> > > [   10.621932] [<900000000180b610>] do_bp+0x2d0/0x340
> > > [   10.621938] [<9000000000241da0>] handle_bp+0x120/0x1c0
> > > [   10.621940] [<900000000103be2c>] ahci_enable_ahci+0x64/0xb8
> > > [   10.621943] [<900000000103beb8>] ahci_save_initial_config+0x38/0x4d8
> > > [   10.621946] [<90000000010391b4>] ahci_init_one+0x354/0x1088
> > > [   10.621950] [<9000000000d16cdc>] local_pci_probe+0x44/0xb8
> > > [   10.621953] [<9000000000286f78>] work_for_cpu_fn+0x18/0x30
> > > [   10.621956] [<900000000028a840>] process_one_work+0x160/0x330
> > > [   10.621961] [<900000000028b208>] worker_thread+0x330/0x460
> > > [   10.621964] [<9000000000295fdc>] kthread+0x11c/0x138
> > > [   10.621968] [<900000000180b740>] ret_from_kernel_thread+0x28/0xa8
> > > [   10.621971] [<9000000000241484>] ret_from_kernel_thread_asm+0xc/0x88
> > > [   10.621973]
> > > 
> > > 
> > 
> > This looks like a warning telling us that AHCI enable failed / timed
> > out. Do you have Panic on Warn on that this directly causes a boot
> > failure? The only relation I can see with my patch is that maybe this
> > AHCI device wasn't probed before and somehow isn't working?
> The rootfs is on the AHCI controller, so AHCI failure causes the boot
> failure, without this patch no boot problems.
> 
> Huacai
> 

Ok, I'm going to need more details to make sense of this. Can you tell
me if ARI is enabled for that bus? Did you test with both patches or
just this one? Could you provide lspci -vv from a good boot and can you
tell which AHCI device the root device is on? Also could you clarify
why you set hypervisor_isolated_pci_functions() in particular this
seems like a bare metal boot, right? When running in KVM do you pass-
through individual PCI functions with the guest seeing a devfn other
than 0 alone, i.e. a missing devfn 0? Or do you need this for bare
metal for some reason? If you don't need it for bare metal, does the
boot work if you return 0 from hypervisor_isolated_pci_functions() with
this patch?

Thanks,
Niklas

Re: [PATCH v5 1/2] PCI: Fix isolated PCI function probing with ARI and SR-IOV

[Huacai Chen]

[-- Attachment #1: Type: text/plain, Size: 8956 bytes --]

On Wed, Nov 5, 2025 at 5:46 PM Niklas Schnelle <schnelle@linux.ibm.com> wrote:
>
> On Wed, 2025-11-05 at 09:01 +0800, Huacai Chen wrote:
> > On Mon, Nov 3, 2025 at 7:23 PM Niklas Schnelle <schnelle@linux.ibm.com> wrote:
> > >
> > > On Mon, 2025-11-03 at 17:50 +0800, Huacai Chen wrote:
> > > > Hi, Niklas,
> > > >
> > > > On Wed, Oct 29, 2025 at 5:42 PM Niklas Schnelle <schnelle@linux.ibm.com> wrote:
> > > > >
> > > > > When the isolated PCI function probing mechanism is used in conjunction
> > > > > with ARI or SR-IOV it may not find all available PCI functions. In the
> > > > > case of ARI the problem is that next_ari_fn() always returns -ENODEV if
> > > > > dev is NULL and thus if fn 0 is missing the scan stops.
> > > > >
> > > > > For SR-IOV things are more complex. Here the problem is that the check
> > > > > for multifunction may fail. One example where this can occur is if the
> > > > > first passed-through function is a VF with devfn 8. Now in
> > > > > pci_scan_slot() this means it is fn 0 and thus multifunction doesn't get
> > > > > set. Since VFs don't get multifunction set via PCI_HEADER_TYPE_MFD it
> > > > > remains unset and probing stops even if there is a devfn 9.
> > > > >
> > > > > Now at the moment both of these issues are hidden on s390. The first one
> > > > > because ARI is detected as disabled as struct pci_bus's self is NULL
> > > > > even though firmware does enable and use ARI. The second issue is hidden
> > > > > as a side effect of commit 25f39d3dcb48 ("s390/pci: Ignore RID for
> > > > > isolated VFs"). This is because VFs are either put on their own virtual
> > > > > bus if the parent PF is not passed-through to the same instance or VFs
> > > > > are hotplugged once SR-IOV is enabled on the parent PF and then
> > > > > pci_scan_single_device() is used.
> > > > >
> > > > > Still especially the first issue prevents correct detection of ARI and
> > > > > the second might be a problem for other users of isolated function
> > > > > probing. Fix both issues by keeping things as simple as possible. If
> > > > > isolated function probing is enabled simply scan every possible devfn.
> > > > I'm very sorry, but applying this patch on top of commit a02fd05661d7
> > > > ("PCI: Extend isolated function probing to LoongArch") we fail to
> > > > boot.
> > > >
> > > > Boot log:
> > > > [   10.365340] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16
> > > > 00:01:03 EST 2006)
> > > > [   10.372628] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
> > > > [   10.379564] megasas: 07.734.00.00-rc1
> > > > [   10.383222] mpt3sas version 54.100.00.00 loaded
> > > > [   10.388304] nvme nvme0: pci function 0000:08:00.0
> > > > [   10.395088] Freeing initrd memory: 45632K
> > > > [   10.469822] ------------[ cut here ]------------
> > > > [   10.474409] WARNING: CPU: 0 PID: 247 at drivers/ata/libahci.c:233
> > > > ahci_enable_ahci+0x64/0xb8
> > > > [   10.482804] Modules linked in:
> > > > [   10.485838] CPU: 0 UID: 0 PID: 247 Comm: kworker/0:11 Not tainted
> > > > 6.18.0-rc3 #1 PREEMPT(full)
> > > > [   10.494397] Hardware name: To be filled by O.E.M.To be fill To be
> > > > filled by O.E.M.To be fill/To be filled by O.E.M.To be fill, BIOS
> > > > Loongson-UDK2018-V4.0.
> > > > [   10.508139] Workqueue: events work_for_cpu_fn
> > > > [   10.512468] pc 900000000103be2c ra 900000000103be28 tp
> > > > 900000010ae44000 sp 900000010ae47be0
> > > > [   10.520769] a0 0000000000000000 a1 00000000000000b0 a2
> > > > 0000000000000001 a3 9000000001810e0c
> > > > [   10.529069] a4 9000000002343e20 a5 0000000000000001 a6
> > > > 0000000000000010 a7 0000000000000000
> > > > [   10.537373] t0 d10951fa66920f31 t1 d10951fa66920f31 t2
> > > > 0000000000001280 t3 000000000674c000
> > > > [   10.545673] t4 0000000000000000 t5 0000000000000000 t6
> > > > 9000000008002480 t7 00000000000000b4
> > > > [   10.553972] t8 90000001055eab90 u0 900000010ae47b68 s9
> > > > 9000000002221a50 s0 0000000000000000
> > > > [   10.562272] s1 ffff800032435800 s2 0000000000000000 s3
> > > > ffffffff80000000 s4 9000000002221570
> > > > [   10.570571] s5 0000000000000005 s6 9000000101ccf0b8 s7
> > > > 90000000023dd000 s8 900000010ae47d08
> > > > [   10.578869]    ra: 900000000103be28 ahci_enable_ahci+0x60/0xb8
> > > > [   10.584665]   ERA: 900000000103be2c ahci_enable_ahci+0x64/0xb8
> > > > [   10.590461]  CRMD: 000000b0 (PLV0 -IE -DA +PG DACF=CC DACM=CC -WE)
> > > > [   10.596609]  PRMD: 00000004 (PPLV0 +PIE -PWE)
> > > > [   10.600937]  EUEN: 00000000 (-FPE -SXE -ASXE -BTE)
> > > > [   10.605698]  ECFG: 00071c1d (LIE=0,2-4,10-12 VS=7)
> > > > [   10.610458] ESTAT: 000c0000 [BRK] (IS= ECode=12 EsubCode=0)
> > > > [   10.615994]  PRID: 0014d010 (Loongson-64bit, Loongson-3C6000/S)
> > > > [   10.621875] CPU: 0 UID: 0 PID: 247 Comm: kworker/0:11 Not tainted
> > > > 6.18.0-rc3 #1 PREEMPT(full)
> > > > [   10.621877] Hardware name: To be filled by O.E.M.To be fill To be
> > > > filled by O.E.M.To be fill/To be filled by O.E.M.To be fill, BIOS
> > > > Loongson-UDK2018-V4.0.
> > > > [   10.621878] Workqueue: events work_for_cpu_fn
> > > > [   10.621881] Stack : 900000010ae47848 0000000000000000
> > > > 90000000002436bc 900000010ae44000
> > > > [   10.621884]         900000010ae47820 900000010ae47828
> > > > 0000000000000000 900000010ae47968
> > > > [   10.621887]         900000010ae47960 900000010ae47960
> > > > 900000010ae47630 0000000000000001
> > > > [   10.621890]         0000000000000001 900000010ae47828
> > > > d10951fa66920f31 9000000100414300
> > > > [   10.621893]         80000000ffffe34d fffffffffffffffe
> > > > 000000000000034f 000000000000002f
> > > > [   10.621896]         0000000000000063 0000000000000001
> > > > 000000000674c000 9000000002221a50
> > > > [   10.621899]         0000000000000000 0000000000000000
> > > > 90000000020b6500 90000000023dd000
> > > > [   10.621902]         00000000000000e9 0000000000000009
> > > > 0000000000000002 90000000023dd000
> > > > [   10.621905]         900000010ae47d08 0000000000000000
> > > > 90000000002436d4 0000000000000000
> > > > [   10.621908]         00000000000000b0 0000000000000004
> > > > 0000000000000000 0000000000071c1d
> > > > [   10.621910]         ...
> > > > [   10.621912] Call Trace:
> > > > [   10.621913] [<90000000002436d4>] show_stack+0x5c/0x180
> > > > [   10.621918] [<900000000023f328>] dump_stack_lvl+0x6c/0x9c
> > > > [   10.621923] [<9000000000266eb8>] __warn+0x80/0x108
> > > > [   10.621927] [<90000000017d1910>] report_bug+0x158/0x2a8
> > > > [   10.621932] [<900000000180b610>] do_bp+0x2d0/0x340
> > > > [   10.621938] [<9000000000241da0>] handle_bp+0x120/0x1c0
> > > > [   10.621940] [<900000000103be2c>] ahci_enable_ahci+0x64/0xb8
> > > > [   10.621943] [<900000000103beb8>] ahci_save_initial_config+0x38/0x4d8
> > > > [   10.621946] [<90000000010391b4>] ahci_init_one+0x354/0x1088
> > > > [   10.621950] [<9000000000d16cdc>] local_pci_probe+0x44/0xb8
> > > > [   10.621953] [<9000000000286f78>] work_for_cpu_fn+0x18/0x30
> > > > [   10.621956] [<900000000028a840>] process_one_work+0x160/0x330
> > > > [   10.621961] [<900000000028b208>] worker_thread+0x330/0x460
> > > > [   10.621964] [<9000000000295fdc>] kthread+0x11c/0x138
> > > > [   10.621968] [<900000000180b740>] ret_from_kernel_thread+0x28/0xa8
> > > > [   10.621971] [<9000000000241484>] ret_from_kernel_thread_asm+0xc/0x88
> > > > [   10.621973]
> > > >
> > > >
> > >
> > > This looks like a warning telling us that AHCI enable failed / timed
> > > out. Do you have Panic on Warn on that this directly causes a boot
> > > failure? The only relation I can see with my patch is that maybe this
> > > AHCI device wasn't probed before and somehow isn't working?
> > The rootfs is on the AHCI controller, so AHCI failure causes the boot
> > failure, without this patch no boot problems.
> >
> > Huacai
> >
>
> Ok, I'm going to need more details to make sense of this. Can you tell
> me if ARI is enabled for that bus? Did you test with both patches or
> just this one? Could you provide lspci -vv from a good boot and can you
> tell which AHCI device the root device is on? Also could you clarify
> why you set hypervisor_isolated_pci_functions() in particular this
> seems like a bare metal boot, right? When running in KVM do you pass-
> through individual PCI functions with the guest seeing a devfn other
> than 0 alone, i.e. a missing devfn 0? Or do you need this for bare
> metal for some reason? If you don't need it for bare metal, does the
> boot work if you return 0 from hypervisor_isolated_pci_functions() with
> this patch?
1. ARI isn't enabled.
2. Only test the first patch.
3. This is a bare metal boot.
4. If hypervisor_isolated_pci_functions() return 0 then boot is OK
5. PCI information please see the attachment.

Huacai

>
> Thanks,
> Niklas

[-- Attachment #2: pci log --]
[-- Type: application/octet-stream, Size: 89099 bytes --]

root@test:/sys/bus/pci/devices# 
root@test:/sys/bus/pci/devices# lspci
00:00.0 Host bridge: Loongson Technology LLC PCIe x8 Root Port
00:03.0 Ethernet controller: Loongson Technology LLC 7A2000 PCH Gigabit Ethernet Controller
00:04.0 USB controller: Loongson Technology LLC OHCI USB Controller (rev 02)
00:04.1 USB controller: Loongson Technology LLC EHCI USB Controller (rev 02)
00:05.0 USB controller: Loongson Technology LLC OHCI USB Controller (rev 02)
00:05.1 USB controller: Loongson Technology LLC EHCI USB Controller (rev 02)
00:07.0 Audio device: Loongson Technology LLC HDA (High Definition Audio) Controller
00:08.0 SATA controller: Loongson Technology LLC SATA 3 AHCI Controller
00:09.0 PCI bridge: Loongson Technology LLC PCIe x4 Root Port
00:0a.0 PCI bridge: Loongson Technology LLC PCIe x1 Root Port
00:0b.0 PCI bridge: Loongson Technology LLC PCIe x1 Root Port
00:0c.0 PCI bridge: Loongson Technology LLC PCIe x1 Root Port
00:0d.0 PCI bridge: Loongson Technology LLC PCIe x4 Root Port
00:0f.0 PCI bridge: Loongson Technology LLC PCIe x16 Root Port
00:10.0 PCI bridge: Loongson Technology LLC PCIe x8 Root Port
00:16.0 System peripheral: Loongson Technology LLC SPI Controller
00:19.0 USB controller: Loongson Technology LLC xHCI USB Controller
00:1c.0 PCI bridge: Loongson Technology LLC Device 3c09
00:1d.0 IOMMU: Loongson Technology LLC Device 3c0f
00:1e.0 PCI bridge: Loongson Technology LLC Device 3c09
05:00.0 Ethernet controller: Motorcomm Microelectronics. YT6801 Gigabit Ethernet Controller (rev 01)
08:00.0 Non-Volatile memory controller: Silicon Motion, Inc. SM2269XT (DRAM-less) NVMe SSD Controller (rev 03)
0b:00.0 PCI bridge: Loongson Technology LLC Device 3c19
0b:01.0 PCI bridge: Loongson Technology LLC Device 3c29
0b:02.0 PCI bridge: Loongson Technology LLC Device 3c29
0f:00.0 PCI bridge: Loongson Technology LLC Device 3c19
0f:01.0 PCI bridge: Loongson Technology LLC Device 3c19
0f:04.0 IOMMU: Loongson Technology LLC Device 3c0f
11:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Oland [Radeon HD 8570 / R5 430 OEM / R7 240/340 / Radeon 520 OEM] (rev 87)
11:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Oland/Hainan/Cape Verde/Pitcairn HDMI Audio [Radeon HD 7000 Series]
root@test:/sys/bus/pci/devices# 
root@test:/sys/bus/pci/devices# cat 0000\:11\:00.0/ari_enabled 
0
root@test:/sys/bus/pci/devices# cat 0000\:11\:00.1/ari_enabled 
0
root@test:/sys/bus/pci/devices# 
root@test:/sys/bus/pci/devices# lspci -vv
00:00.0 Host bridge: Loongson Technology LLC PCIe x8 Root Port
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 255
	NUMA node: 0
	Region 0: Memory at e8011141000 (64-bit, non-prefetchable) [size=4K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable+ Count=32/1 Maskable- 64bit+
		Address: 000000fdf8000000  Data: 0000
	Capabilities: [70] Express (v2) Endpoint, IntMsgNum 0
		DevCap:	MaxPayload 2048 bytes, PhantFunc 0, Latency L0s <512ns, L1 <1us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0W TEE-IO-
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 512 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ NonFatalErr+ FatalErr- UnsupReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <512ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x8
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP- LTR-
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- TPHComp- ExtTPHComp-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
			 AtomicOpsCtl: ReqEn-
			 IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
			 10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
		LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
			 EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO+ CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		CESta:	RxErr+ BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr- HeaderOF-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr+ HeaderOF-
		AERCap:	First Error Pointer: 0e, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [140 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0

00:03.0 Ethernet controller: Loongson Technology LLC 7A2000 PCH Gigabit Ethernet Controller
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop+ ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 82
	NUMA node: 0
	Region 0: Memory at e8011110000 (64-bit, non-prefetchable) [size=32K]
	Kernel driver in use: dwmac-loongson-pci
	Kernel modules: dwmac_loongson

00:04.0 USB controller: Loongson Technology LLC OHCI USB Controller (rev 02) (prog-if 10 [OHCI])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop+ ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 76
	NUMA node: 0
	Region 0: Memory at e8011119000 (64-bit, non-prefetchable) [size=28K]
	Kernel driver in use: ohci-pci

00:04.1 USB controller: Loongson Technology LLC EHCI USB Controller (rev 02) (prog-if 20 [EHCI])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin B routed to IRQ 77
	NUMA node: 0
	Region 0: Memory at e8011120000 (64-bit, non-prefetchable) [size=32K]
	Kernel driver in use: ehci-pci

00:05.0 USB controller: Loongson Technology LLC OHCI USB Controller (rev 02) (prog-if 10 [OHCI])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop+ ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 79
	NUMA node: 0
	Region 0: Memory at e8011129000 (64-bit, non-prefetchable) [size=28K]
	Kernel driver in use: ohci-pci

00:05.1 USB controller: Loongson Technology LLC EHCI USB Controller (rev 02) (prog-if 20 [EHCI])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin B routed to IRQ 78
	NUMA node: 0
	Region 0: Memory at e8011130000 (64-bit, non-prefetchable) [size=32K]
	Kernel driver in use: ehci-pci

00:07.0 Audio device: Loongson Technology LLC HDA (High Definition Audio) Controller
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop+ ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 89
	NUMA node: 0
	Region 0: Memory at e8011100000 (64-bit, non-prefetchable) [size=64K]
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel

00:08.0 SATA controller: Loongson Technology LLC SATA 3 AHCI Controller (prog-if 01 [AHCI 1.0])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop+ ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 58
	NUMA node: 0
	Region 5: Memory at e0030501800 (32-bit, non-prefetchable) [size=1K]
	Kernel driver in use: ahci

00:09.0 PCI bridge: Loongson Technology LLC PCIe x4 Root Port (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 35
	NUMA node: 0
	Region 0: Memory at e8011138000 (64-bit, non-prefetchable) [size=4K]
	Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
	I/O behind bridge: [disabled] [32-bit]
	Memory behind bridge: [disabled] [32-bit]
	Prefetchable memory behind bridge: [disabled] [64-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable+ DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 000000002fe00100  Data: 0000
	Capabilities: [70] Express (v2) Root Port (Slot-), IntMsgNum 0
		DevCap:	MaxPayload 2048 bytes, PhantFunc 0
			ExtTag- RBE+ TEE-IO-
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		RootCap: CRSVisible-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP+ LTR-
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd-
			 AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd-
			 AtomicOpsCtl: ReqEn- EgressBlck-
			 IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
			 10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
		LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CorrIntErr- HeaderOF-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr+ HeaderOF-
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
		RootCmd: CERptEn- NFERptEn- FERptEn-
		RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
			 FirstFatal- NonFatalMsg- FatalMsg- IntMsgNum 0
		ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
	Capabilities: [140 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Kernel driver in use: pcieport

00:0a.0 PCI bridge: Loongson Technology LLC PCIe x1 Root Port (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 37
	NUMA node: 0
	Region 0: Memory at e8011139000 (64-bit, non-prefetchable) [size=4K]
	Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
	I/O behind bridge: 4000-4fff [size=4K] [16-bit]
	Memory behind bridge: 30000000-300fffff [size=1M] [32-bit]
	Prefetchable memory behind bridge: [disabled] [64-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 000000002fe01100  Data: 0000
	Capabilities: [70] Express (v2) Root Port (Slot-), IntMsgNum 0
		DevCap:	MaxPayload 2048 bytes, PhantFunc 0
			ExtTag- RBE+ TEE-IO-
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <1us
			ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1
			TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
		RootCap: CRSVisible-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP+ LTR-
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd-
			 AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd-
			 AtomicOpsCtl: ReqEn- EgressBlck-
			 IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
			 10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
		LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CorrIntErr- HeaderOF-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr+ HeaderOF-
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
		RootCmd: CERptEn- NFERptEn- FERptEn-
		RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
			 FirstFatal- NonFatalMsg- FatalMsg- IntMsgNum 0
		ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
	Capabilities: [140 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Kernel driver in use: pcieport

00:0b.0 PCI bridge: Loongson Technology LLC PCIe x1 Root Port (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 39
	NUMA node: 0
	Region 0: Memory at e801113a000 (64-bit, non-prefetchable) [size=4K]
	Bus: primary=00, secondary=06, subordinate=06, sec-latency=0
	I/O behind bridge: [disabled] [32-bit]
	Memory behind bridge: [disabled] [32-bit]
	Prefetchable memory behind bridge: [disabled] [64-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable+ DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 000000002fe02100  Data: 0000
	Capabilities: [70] Express (v2) Root Port (Slot-), IntMsgNum 0
		DevCap:	MaxPayload 2048 bytes, PhantFunc 0
			ExtTag- RBE+ TEE-IO-
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <1us
			ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		RootCap: CRSVisible-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP+ LTR-
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd-
			 AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd-
			 AtomicOpsCtl: ReqEn- EgressBlck-
			 IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
			 10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
		LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CorrIntErr- HeaderOF-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr+ HeaderOF-
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
		RootCmd: CERptEn- NFERptEn- FERptEn-
		RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
			 FirstFatal- NonFatalMsg- FatalMsg- IntMsgNum 0
		ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
	Capabilities: [140 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Kernel driver in use: pcieport

00:0c.0 PCI bridge: Loongson Technology LLC PCIe x1 Root Port (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 41
	NUMA node: 0
	Region 0: Memory at e801113b000 (64-bit, non-prefetchable) [size=4K]
	Bus: primary=00, secondary=07, subordinate=07, sec-latency=0
	I/O behind bridge: [disabled] [32-bit]
	Memory behind bridge: [disabled] [32-bit]
	Prefetchable memory behind bridge: [disabled] [64-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable+ DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 000000002fe03100  Data: 0000
	Capabilities: [70] Express (v2) Root Port (Slot-), IntMsgNum 0
		DevCap:	MaxPayload 2048 bytes, PhantFunc 0
			ExtTag- RBE+ TEE-IO-
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <1us
			ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		RootCap: CRSVisible-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP+ LTR-
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd-
			 AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd-
			 AtomicOpsCtl: ReqEn- EgressBlck-
			 IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
			 10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
		LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CorrIntErr- HeaderOF-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr+ HeaderOF-
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
		RootCmd: CERptEn- NFERptEn- FERptEn-
		RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
			 FirstFatal- NonFatalMsg- FatalMsg- IntMsgNum 0
		ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
	Capabilities: [140 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Kernel driver in use: pcieport

00:0d.0 PCI bridge: Loongson Technology LLC PCIe x4 Root Port (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 43
	NUMA node: 0
	Region 0: Memory at e801113c000 (64-bit, non-prefetchable) [size=4K]
	Bus: primary=00, secondary=08, subordinate=08, sec-latency=0
	I/O behind bridge: [disabled] [32-bit]
	Memory behind bridge: 30100000-301fffff [size=1M] [32-bit]
	Prefetchable memory behind bridge: [disabled] [64-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 000000002fe04100  Data: 0000
	Capabilities: [70] Express (v2) Root Port (Slot-), IntMsgNum 0
		DevCap:	MaxPayload 2048 bytes, PhantFunc 0
			ExtTag- RBE+ TEE-IO-
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 256 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <1us
			ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x4
			TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt+
		RootCap: CRSVisible-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP+ LTR-
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd-
			 AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd-
			 AtomicOpsCtl: ReqEn- EgressBlck-
			 IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
			 10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
		LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
			 EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		CESta:	RxErr+ BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CorrIntErr- HeaderOF-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr+ HeaderOF-
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
		RootCmd: CERptEn- NFERptEn- FERptEn-
		RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
			 FirstFatal- NonFatalMsg- FatalMsg- IntMsgNum 0
		ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
	Capabilities: [140 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Kernel driver in use: pcieport

00:0f.0 PCI bridge: Loongson Technology LLC PCIe x16 Root Port (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 45
	NUMA node: 0
	Region 0: Memory at e801113d000 (64-bit, non-prefetchable) [size=4K]
	Bus: primary=00, secondary=09, subordinate=09, sec-latency=0
	I/O behind bridge: [disabled] [32-bit]
	Memory behind bridge: [disabled] [32-bit]
	Prefetchable memory behind bridge: [disabled] [64-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable+ DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 000000002fe05100  Data: 0000
	Capabilities: [70] Express (v2) Root Port (Slot-), IntMsgNum 0
		DevCap:	MaxPayload 2048 bytes, PhantFunc 0
			ExtTag- RBE+ TEE-IO-
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <1us
			ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		RootCap: CRSVisible-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP+ LTR-
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd-
			 AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd-
			 AtomicOpsCtl: ReqEn- EgressBlck-
			 IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
			 10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
		LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CorrIntErr- HeaderOF-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr+ HeaderOF-
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
		RootCmd: CERptEn- NFERptEn- FERptEn-
		RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
			 FirstFatal- NonFatalMsg- FatalMsg- IntMsgNum 0
		ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
	Capabilities: [140 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Kernel driver in use: pcieport

00:10.0 PCI bridge: Loongson Technology LLC PCIe x8 Root Port (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 47
	NUMA node: 0
	Region 0: Memory at e801113e000 (64-bit, non-prefetchable) [size=4K]
	Bus: primary=00, secondary=0a, subordinate=0a, sec-latency=0
	I/O behind bridge: [disabled] [32-bit]
	Memory behind bridge: [disabled] [32-bit]
	Prefetchable memory behind bridge: [disabled] [64-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable+ DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 000000002fe06100  Data: 0000
	Capabilities: [70] Express (v2) Root Port (Slot-), IntMsgNum 0
		DevCap:	MaxPayload 2048 bytes, PhantFunc 0
			ExtTag- RBE+ TEE-IO-
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <512ns, L1 <1us
			ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		RootCap: CRSVisible-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP+ LTR-
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd-
			 AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd-
			 AtomicOpsCtl: ReqEn- EgressBlck-
			 IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
			 10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
		LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CorrIntErr- HeaderOF-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr+ HeaderOF-
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
		RootCmd: CERptEn- NFERptEn- FERptEn-
		RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
			 FirstFatal- NonFatalMsg- FatalMsg- IntMsgNum 0
		ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
	Capabilities: [140 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Kernel driver in use: pcieport

00:16.0 System peripheral: Loongson Technology LLC SPI Controller
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop+ ParErr+ Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 255
	NUMA node: 0
	Region 0: Memory at e801113f000 (64-bit, non-prefetchable) [size=4K]
	Region 2: Memory at e8010000000 (64-bit, non-prefetchable) [size=16M]
	Kernel driver in use: loongson-spi-pci

00:19.0 USB controller: Loongson Technology LLC xHCI USB Controller (prog-if 30 [XHCI])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 80
	NUMA node: 0
	Region 0: Memory at e8011000000 (64-bit, non-prefetchable) [size=1M]
	Kernel driver in use: xhci_hcd

00:1c.0 PCI bridge: Loongson Technology LLC Device 3c09 (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop+ ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	NUMA node: 0
	Bus: primary=00, secondary=0b, subordinate=0e, sec-latency=0
	I/O behind bridge: [disabled] [32-bit]
	Memory behind bridge: 30200000-302fffff [size=1M] [32-bit]
	Prefetchable memory behind bridge: [disabled] [64-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	Expansion ROM at e0030500000 [disabled] [size=2K]
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 0
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit-
		Address: 00000000  Data: 0000
	Capabilities: [70] Express (v2) Upstream Port, IntMsgNum 0
		DevCap:	MaxPayload 2048 bytes, PhantFunc 0
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 0W TEE-IO-
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 2048 bytes, MaxReadReq 2048 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <1us
			ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; LnkDisable- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 16GT/s, Width x16
			TrErr+ Train+ SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR-
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-
			 AtomicOpsCap: Routing-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
			 AtomicOpsCtl: EgressBlck-
			 IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
			 10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Kernel driver in use: pcieport

00:1d.0 IOMMU: Loongson Technology LLC Device 3c0f
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop+ ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	NUMA node: 0
	Region 0: Memory at e8011140000 (64-bit, non-prefetchable) [size=1K]
	Expansion ROM at e0030500800 [disabled] [size=2K]
	Kernel driver in use: loongarch-iommu
	Kernel modules: loongarch_iommu

00:1e.0 PCI bridge: Loongson Technology LLC Device 3c09 (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop+ ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	NUMA node: 0
	Bus: primary=00, secondary=0f, subordinate=11, sec-latency=0
	I/O behind bridge: 5000-5fff [size=4K] [16-bit]
	Memory behind bridge: 30300000-304fffff [size=2M] [32-bit]
	Prefetchable memory behind bridge: e8000000000-e800fffffff [size=256M] [32-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	Expansion ROM at e0030501000 [disabled] [size=2K]
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 0
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME+
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit-
		Address: 00000000  Data: 0000
	Capabilities: [70] Express (v2) Upstream Port, IntMsgNum 0
		DevCap:	MaxPayload 2048 bytes, PhantFunc 0
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 0W TEE-IO-
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 2048 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <1us
			ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; LnkDisable- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 16GT/s, Width x16
			TrErr+ Train+ SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR-
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-
			 AtomicOpsCap: Routing-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
			 AtomicOpsCtl: EgressBlck-
			 IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
			 10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Kernel driver in use: pcieport

05:00.0 Ethernet controller: Motorcomm Microelectronics. YT6801 Gigabit Ethernet Controller (rev 01)
	Subsystem: Device 2066:9806
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 36
	NUMA node: 0
	Region 0: Memory at e0030000000 (64-bit, non-prefetchable) [size=16K]
	Region 4: I/O ports at 4000 [size=256]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2- AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable- Count=1/8 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [70] Express (v2) Endpoint, IntMsgNum 0
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0W TEE-IO-
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 4096 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L1 <64us
			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP- LTR+
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- TPHComp- ExtTPHComp-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
			 AtomicOpsCtl: ReqEn-
			 IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
			 10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
		LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [b0] MSI-X: Enable+ Count=6 Masked-
		Vector table: BAR=0 offset=00001200
		PBA: BAR=0 offset=00001300
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CorrIntErr- HeaderOF-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr+ HeaderOF+
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [148 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Capabilities: [150 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=0us LTR1.2_Threshold=0ns
		L1SubCtl2: T_PwrOn=10us
	Kernel driver in use: yt6801
	Kernel modules: yt6801

08:00.0 Non-Volatile memory controller: Silicon Motion, Inc. SM2269XT (DRAM-less) NVMe SSD Controller (rev 03) (prog-if 02 [NVM Express])
	Subsystem: Silicon Motion, Inc. SM2269XT (DRAM-less) NVMe SSD Controller
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 42
	NUMA node: 0
	Region 0: Memory at e0030100000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable- Count=1/16 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [70] Express (v2) Endpoint, IntMsgNum 0
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W TEE-IO-
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
			MaxPayload 256 bytes, MaxReadReq 1024 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s (downgraded), Width x4
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
			 10BitTagComp+ 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS+ TPHComp- ExtTPHComp-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
			 AtomicOpsCtl: ReqEn-
			 IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
			 10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
		LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS+
		LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
			 EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: Upstream Port
	Capabilities: [b0] MSI-X: Enable+ Count=17 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CorrIntErr- HeaderOF-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr+ HeaderOF+
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [148 v1] Power Budgeting <?>
	Capabilities: [158 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [168 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Capabilities: [188 v1] Physical Layer 16.0 GT/s <?>
	Capabilities: [1ac v1] Lane Margining at the Receiver
		PortCap: Uses Driver-
		PortSta: MargReady- MargSoftReady-
	Capabilities: [204 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Capabilities: [20c v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=10us PortTPowerOnTime=60us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=0us LTR1.2_Threshold=0ns
		L1SubCtl2: T_PwrOn=10us
	Capabilities: [390 v1] Data Link Feature <?>
	Kernel driver in use: nvme

0b:00.0 PCI bridge: Loongson Technology LLC Device 3c19 (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 49
	NUMA node: 0
	IOMMU group: 0
	Region 0: Memory at e0030200000 (64-bit, non-prefetchable) [size=4K]
	Bus: primary=0b, secondary=0c, subordinate=0c, sec-latency=0
	I/O behind bridge: [disabled] [32-bit]
	Memory behind bridge: [disabled] [32-bit]
	Prefetchable memory behind bridge: [disabled] [64-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold+)
		Status: D3 NoSoftRst+ PME-Enable+ DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 000000002fe07100  Data: 0000
	Capabilities: [70] Express (v2) Root Port (Slot-), IntMsgNum 0
		DevCap:	MaxPayload 2048 bytes, PhantFunc 0
			ExtTag- RBE+ TEE-IO-
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 2048 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <1us
			ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		RootCap: CRSVisible-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP+ LTR-
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd-
			 AtomicOpsCap: Routing+ 32bit+ 64bit+ 128bitCAS+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd-
			 AtomicOpsCtl: ReqEn- EgressBlck-
			 IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
			 10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
		LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CorrIntErr- HeaderOF-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr+ HeaderOF-
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
		RootCmd: CERptEn- NFERptEn- FERptEn-
		RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
			 FirstFatal- NonFatalMsg- FatalMsg- IntMsgNum 0
		ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
	Capabilities: [140 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Capabilities: [180 v1] Physical Layer 16.0 GT/s <?>
	Capabilities: [1b0 v1] Lane Margining at the Receiver
		PortCap: Uses Driver-
		PortSta: MargReady- MargSoftReady-
	Capabilities: [200 v1] Data Link Feature <?>
	Kernel driver in use: pcieport

0b:01.0 PCI bridge: Loongson Technology LLC Device 3c29 (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 51
	NUMA node: 0
	IOMMU group: 1
	Region 0: Memory at e0030201000 (64-bit, non-prefetchable) [size=4K]
	Bus: primary=0b, secondary=0d, subordinate=0d, sec-latency=0
	I/O behind bridge: [disabled] [32-bit]
	Memory behind bridge: [disabled] [32-bit]
	Prefetchable memory behind bridge: [disabled] [64-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold+)
		Status: D3 NoSoftRst+ PME-Enable+ DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 000000002fe08100  Data: 0000
	Capabilities: [70] Express (v2) Root Port (Slot-), IntMsgNum 0
		DevCap:	MaxPayload 2048 bytes, PhantFunc 0
			ExtTag- RBE+ TEE-IO-
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 2048 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 16GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <512ns, L1 <1us
			ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		RootCap: CRSVisible-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP+ LTR-
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd-
			 AtomicOpsCap: Routing+ 32bit+ 64bit+ 128bitCAS+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd-
			 AtomicOpsCtl: ReqEn- EgressBlck-
			 IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
			 10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
		LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CorrIntErr- HeaderOF-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr+ HeaderOF-
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
		RootCmd: CERptEn- NFERptEn- FERptEn-
		RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
			 FirstFatal- NonFatalMsg- FatalMsg- IntMsgNum 0
		ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
	Capabilities: [140 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Capabilities: [180 v1] Physical Layer 16.0 GT/s <?>
	Capabilities: [1b0 v1] Lane Margining at the Receiver
		PortCap: Uses Driver-
		PortSta: MargReady- MargSoftReady-
	Capabilities: [200 v1] Data Link Feature <?>
	Kernel driver in use: pcieport

0b:02.0 PCI bridge: Loongson Technology LLC Device 3c29 (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 53
	NUMA node: 0
	IOMMU group: 2
	Region 0: Memory at e0030202000 (64-bit, non-prefetchable) [size=4K]
	Bus: primary=0b, secondary=0e, subordinate=0e, sec-latency=0
	I/O behind bridge: [disabled] [32-bit]
	Memory behind bridge: [disabled] [32-bit]
	Prefetchable memory behind bridge: [disabled] [64-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold+)
		Status: D3 NoSoftRst+ PME-Enable+ DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 000000002fe09100  Data: 0000
	Capabilities: [70] Express (v2) Root Port (Slot-), IntMsgNum 0
		DevCap:	MaxPayload 2048 bytes, PhantFunc 0
			ExtTag- RBE+ TEE-IO-
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 2048 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 16GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <512ns, L1 <1us
			ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		RootCap: CRSVisible-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP+ LTR-
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd-
			 AtomicOpsCap: Routing+ 32bit+ 64bit+ 128bitCAS+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd-
			 AtomicOpsCtl: ReqEn- EgressBlck-
			 IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
			 10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
		LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CorrIntErr- HeaderOF-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr+ HeaderOF-
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
		RootCmd: CERptEn- NFERptEn- FERptEn-
		RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
			 FirstFatal- NonFatalMsg- FatalMsg- IntMsgNum 0
		ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
	Capabilities: [140 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Capabilities: [180 v1] Physical Layer 16.0 GT/s <?>
	Capabilities: [1b0 v1] Lane Margining at the Receiver
		PortCap: Uses Driver-
		PortSta: MargReady- MargSoftReady-
	Capabilities: [200 v1] Data Link Feature <?>
	Kernel driver in use: pcieport

0f:00.0 PCI bridge: Loongson Technology LLC Device 3c19 (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 55
	NUMA node: 0
	IOMMU group: 3
	Region 0: Memory at e0030400000 (64-bit, non-prefetchable) [size=4K]
	Bus: primary=0f, secondary=10, subordinate=10, sec-latency=0
	I/O behind bridge: [disabled] [32-bit]
	Memory behind bridge: [disabled] [32-bit]
	Prefetchable memory behind bridge: [disabled] [64-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold+)
		Status: D3 NoSoftRst+ PME-Enable+ DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 000000002fe0a100  Data: 0000
	Capabilities: [70] Express (v2) Root Port (Slot-), IntMsgNum 0
		DevCap:	MaxPayload 2048 bytes, PhantFunc 0
			ExtTag- RBE+ TEE-IO-
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 2048 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <1us
			ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		RootCap: CRSVisible-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP+ LTR-
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd-
			 AtomicOpsCap: Routing+ 32bit+ 64bit+ 128bitCAS+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd-
			 AtomicOpsCtl: ReqEn- EgressBlck-
			 IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
			 10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
		LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CorrIntErr- HeaderOF-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr+ HeaderOF-
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
		RootCmd: CERptEn- NFERptEn- FERptEn-
		RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
			 FirstFatal- NonFatalMsg- FatalMsg- IntMsgNum 0
		ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
	Capabilities: [140 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Capabilities: [180 v1] Physical Layer 16.0 GT/s <?>
	Capabilities: [1b0 v1] Lane Margining at the Receiver
		PortCap: Uses Driver-
		PortSta: MargReady- MargSoftReady-
	Capabilities: [200 v1] Data Link Feature <?>
	Kernel driver in use: pcieport

0f:01.0 PCI bridge: Loongson Technology LLC Device 3c19 (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 57
	NUMA node: 0
	IOMMU group: 4
	Region 0: Memory at e0030401000 (64-bit, non-prefetchable) [size=4K]
	Bus: primary=0f, secondary=11, subordinate=11, sec-latency=0
	I/O behind bridge: 5000-5fff [size=4K] [16-bit]
	Memory behind bridge: 30300000-303fffff [size=1M] [32-bit]
	Prefetchable memory behind bridge: e8000000000-e800fffffff [size=256M] [32-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 000000002fe0b100  Data: 0000
	Capabilities: [70] Express (v2) Root Port (Slot-), IntMsgNum 0
		DevCap:	MaxPayload 2048 bytes, PhantFunc 0
			ExtTag- RBE+ TEE-IO-
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <1us
			ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x8
			TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
		RootCap: CRSVisible-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP+ LTR-
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd-
			 AtomicOpsCap: Routing+ 32bit+ 64bit+ 128bitCAS+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd-
			 AtomicOpsCtl: ReqEn- EgressBlck-
			 IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
			 10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
		LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance+ SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
			 EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CorrIntErr- HeaderOF-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr+ HeaderOF-
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
		RootCmd: CERptEn- NFERptEn- FERptEn-
		RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
			 FirstFatal- NonFatalMsg- FatalMsg- IntMsgNum 0
		ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
	Capabilities: [140 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Capabilities: [180 v1] Physical Layer 16.0 GT/s <?>
	Capabilities: [1b0 v1] Lane Margining at the Receiver
		PortCap: Uses Driver-
		PortSta: MargReady- MargSoftReady-
	Capabilities: [200 v1] Data Link Feature <?>
	Kernel driver in use: pcieport

0f:04.0 IOMMU: Loongson Technology LLC Device 3c0f
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop+ ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	NUMA node: 0
	Region 0: Memory at e0030402800 (64-bit, non-prefetchable) [size=1K]
	Expansion ROM at e0030402000 [disabled] [size=2K]
	Kernel driver in use: loongarch-iommu
	Kernel modules: loongarch_iommu

11:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Oland [Radeon HD 8570 / R5 430 OEM / R7 240/340 / Radeon 520 OEM] (rev 87) (prog-if 00 [VGA controller])
	Subsystem: Bitland(ShenZhen) Information Technology Co., Ltd. Device 1601
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 81
	NUMA node: 0
	IOMMU group: 5
	Region 0: Memory at e8000000000 (64-bit, prefetchable) [size=256M]
	Region 2: Memory at e0030300000 (64-bit, non-prefetchable) [size=256K]
	Region 4: I/O ports at 5000 [size=256]
	Expansion ROM at e0030340000 [disabled] [size=128K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Express (v2) Legacy Endpoint, IntMsgNum 0
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- TEE-IO-
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
		LnkCap:	Port #4, Speed 8GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x8
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR-
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
			 AtomicOpsCtl: ReqEn-
			 IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
			 10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
		LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
			 EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 000000002fe0f100  Data: 0000
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout+ AdvNonFatalErr+ CorrIntErr- HeaderOF-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr- HeaderOF-
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [200 v1] Physical Resizable BAR
		BAR 0: current size: 256MB, supported: 256MB 512MB 1GB 2GB 4GB
	Capabilities: [270 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Kernel driver in use: radeon
	Kernel modules: radeon, amdgpu

11:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Oland/Hainan/Cape Verde/Pitcairn HDMI Audio [Radeon HD 7000 Series]
	Subsystem: Bitland(ShenZhen) Information Technology Co., Ltd. Device aab0
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin B routed to IRQ 91
	NUMA node: 0
	IOMMU group: 5
	Region 0: Memory at e0030360000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Express (v2) Legacy Endpoint, IntMsgNum 0
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- TEE-IO-
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
		LnkCap:	Port #4, Speed 8GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x8
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR-
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
			 AtomicOpsCtl: ReqEn-
			 IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
			 10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 000000002fe1c100  Data: 0000
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout+ AdvNonFatalErr+ CorrIntErr- HeaderOF-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr- HeaderOF-
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel

root@test:/sys/bus/pci/devices# 

[PATCH v5 2/2] PCI: s390: Handle ARI on bus without associated struct pci_dev

[Niklas Schnelle]

On s390 PCI busses are virtualized and the downstream ports are
invisible to the OS and self in struct pci_bus is NULL. This associated
struct pci_dev is however relied upon in pci_ari_enabled() to check
whether ARI is enabled for the bus. ARI is therefor always detected as
disabled. At the same time, firmware on s390 always enables and relies
upon ARI thus causing a mismatch.

Despite simply being a mismatch this causes problems as some PCI devices
present a different SR-IOV topology depending on PCI_SRIOV_CTRL_ARI.

A similar mismatch may occur with SR-IOV when virtfn_add_bus() creates new
busses with no associated struct pci_dev. Here too pci_ari_enabled()
on these busses returns false even if ARI is actually used.

Prevent both mismatches by moving the ari_enabled flag from struct
pci_dev to struct pci_bus making it independent from self in struct
pci_bus. Let the bus inherit the ari_enabled state from its parent bus
when there is no bridge device such that busses added by virtfn_add_bus()
match their parent. For s390 set ari_enabled when the device supports
ARI in the awareness that all PCIe ports on s390 systems are ARI
capable.

Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
---
 arch/s390/pci/pci.c     |  7 +++++++
 arch/s390/pci/pci_bus.c | 10 ++++++++++
 drivers/pci/pci.c       |  4 ++--
 drivers/pci/probe.c     |  1 +
 include/linux/pci.h     |  4 ++--
 5 files changed, 22 insertions(+), 4 deletions(-)

diff --git a/arch/s390/pci/pci.c b/arch/s390/pci/pci.c
index c82c577db2bcd2143476cb8189fd89b9a4dc9836..773c0cbfc313ea1a6419a44d6158397dd13f6e76 100644
--- a/arch/s390/pci/pci.c
+++ b/arch/s390/pci/pci.c
@@ -609,6 +609,13 @@ int pcibios_device_add(struct pci_dev *pdev)
 			continue;
 		pci_claim_resource(pdev, i);
 	}
+	/*
+	 * The below is the s390 equivalent of pci_configure_ari()
+	 * which we can't use directly because the bridge devices
+	 * are hidden in firmware.
+	 */
+	if (!pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_ARI))
+		zdev->zbus->bus->ari_enabled = 0;
 
 	return 0;
 }
diff --git a/arch/s390/pci/pci_bus.c b/arch/s390/pci/pci_bus.c
index 45a1c36c5a54e3a841e61cc365d3f36e9a94ba50..c887e61eb384ca98ff27d4f8af69e58c715b5002 100644
--- a/arch/s390/pci/pci_bus.c
+++ b/arch/s390/pci/pci_bus.c
@@ -207,6 +207,16 @@ static int zpci_bus_create_pci_bus(struct zpci_bus *zbus, struct zpci_dev *fr, s
 		return -EFAULT;
 	}
 
+	/*
+	 * On s390 PCI busses are virtualized and the bridge
+	 * devices are invisible to the OS. Furthermore busses
+	 * may exist without a devfn 0 function. Thus the normal
+	 * ARI detection does not work. At the same time fw/hw
+	 * has always enabled ARI when possible. Reflect the actual
+	 * state by setting ari_enabled whenever a device on the bus
+	 * supports it.
+	 */
+	bus->ari_enabled = 1;
 	zbus->bus = bus;
 
 	return 0;
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index b14dd064006cca80ec5275e45a35d6dc2b4d0bbc..8ef3c68280a629449e0a2176d938bf987c68dddf 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -3532,11 +3532,11 @@ void pci_configure_ari(struct pci_dev *dev)
 	if (pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ARI)) {
 		pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2,
 					 PCI_EXP_DEVCTL2_ARI);
-		bridge->ari_enabled = 1;
+		dev->bus->ari_enabled = 1;
 	} else {
 		pcie_capability_clear_word(bridge, PCI_EXP_DEVCTL2,
 					   PCI_EXP_DEVCTL2_ARI);
-		bridge->ari_enabled = 0;
+		dev->bus->ari_enabled = 0;
 	}
 }
 
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 41dd1a339a994956a6bc7e1fea0fe0d55452a963..6fc03328a62f54f33f47309f8300516e8f52b5eb 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1216,6 +1216,7 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus *parent,
 
 	if (!bridge) {
 		child->dev.parent = parent->bridge;
+		child->ari_enabled = parent->ari_enabled;
 		goto add_dev;
 	}
 
diff --git a/include/linux/pci.h b/include/linux/pci.h
index d1fdf81fbe1e427aecbc951fa3fdf65c20450b05..a9c3dbf17339e523362bd179ad3c7c8c91293cf0 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -445,7 +445,6 @@ struct pci_dev {
 	unsigned int	irq_reroute_variant:2;	/* Needs IRQ rerouting variant */
 	unsigned int	msi_enabled:1;
 	unsigned int	msix_enabled:1;
-	unsigned int	ari_enabled:1;		/* ARI forwarding */
 	unsigned int	ats_enabled:1;		/* Address Translation Svc */
 	unsigned int	pasid_enabled:1;	/* Process Address Space ID */
 	unsigned int	pri_enabled:1;		/* Page Request Interface */
@@ -691,6 +690,7 @@ struct pci_bus {
 	unsigned int		is_added:1;
 	unsigned int		unsafe_warn:1;	/* warned about RW1C config write */
 	unsigned int		flit_mode:1;	/* Link in Flit mode */
+	unsigned int		ari_enabled:1;	/* ARI forwarding enabled */
 };
 
 #define to_pci_bus(n)	container_of(n, struct pci_bus, dev)
@@ -2740,7 +2740,7 @@ static inline bool pci_is_dev_assigned(struct pci_dev *pdev)
  */
 static inline bool pci_ari_enabled(struct pci_bus *bus)
 {
-	return bus->self && bus->self->ari_enabled;
+	return bus->ari_enabled;
 }
 
 /**

-- 
2.48.1