[PATCH 0/7] disable bridge ari forwarding after connected ari device hot removed

[PATCH 0/7] disable bridge ari forwarding after connected ari device hot removed

[Yijing Wang]

pci_enable_ari will be called if an ARI pci device found, then set its bridge ARI Forwarding Enable
bit in Device Control 2 Register. But the bridge ARI Forwarding Enable bit will never be cleared
when an ARI device hot removed.

my steps:
1. Hot add an ARI pci device;
2. Hot remove the ARI pci device;
3. Hot add an non ARI pci device;

In this case, after setp 3, we could only find fun 0 of non ARI pci device because of its bridge ARI Forwarding Enable
bit set.

As PCIe Spec 2.0(6.13/441) recommends:
"Following a hot-plug event below a Downstream Port, it is strongly recommended that software
Clear the ARI Forwarding Enable bit in the Downstream Port until software determines that a
newly added component is in fact an ARI Device"

This series of patches fix this problem.

Yijing Wang (7):
  PCI: rework pci_enable_ari for support disable ari forwarding
  PCI, acpiphp: disable ARI forwarding for acpiphp
  PCI, pciehp: disable ARI forwarding for pciehp
  PCI, cpqphp: disable ARI forwarding for cpqphp
  PCI, shpchp: disable ARI forwarding for shpchp
  PCI, sgi: disable ARI forwarding for sgiphp
  PCI, ibmphp: disable ARI forwarding for ibmphp

 drivers/pci/hotplug/acpiphp_glue.c     |    1 +
 drivers/pci/hotplug/cpci_hotplug_pci.c |    1 +
 drivers/pci/hotplug/cpqphp_pci.c       |    1 +
 drivers/pci/hotplug/ibmphp_core.c      |    1 +
 drivers/pci/hotplug/pciehp_pci.c       |    1 +
 drivers/pci/hotplug/sgi_hotplug.c      |    1 +
 drivers/pci/hotplug/shpchp_pci.c       |    1 +
 drivers/pci/pci.c                      |   16 +++++++++++-----
 drivers/pci/pci.h                      |    2 +-
 drivers/pci/probe.c                    |    2 +-
 10 files changed, 20 insertions(+), 7 deletions(-)

[PATCH 1/7] PCI: rework pci_enable_ari for support disable ari forwarding

[Yijing Wang]

pci_enable_ari will be called if an ARI pci device found, set its bridge ARI Forwarding Enable
bit in Device Control 2 Register. But the bridge ARI Forwarding Enable bit will never be cleared
when an ARI device hot removed.

As PCIe Spec 2.0(6.13/441) recommends:
"Following a hot-plug event below a Downstream Port, it is strongly recommended that software
Clear the ARI Forwarding Enable bit in the Downstream Port until software determines that a
newly added component is in fact an ARI Device"

So, this patch rework pci_enable_ari for support disable ari forwarding when a pci device
was hot removed.

Signed-off-by: Yijing Wang <wangyijing@huawei.com>
---
 drivers/pci/pci.c   |   16 +++++++++++-----
 drivers/pci/pci.h   |    2 +-
 drivers/pci/probe.c |    2 +-
 3 files changed, 13 insertions(+), 7 deletions(-)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 5485883..27f57d7 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -2010,10 +2010,10 @@ void pci_free_cap_save_buffers(struct pci_dev *dev)
 }
 
 /**
- * pci_enable_ari - enable ARI forwarding if hardware support it
+ * pci_configure_ari - enable or disable ARI forwarding if hardware support it
  * @dev: the PCI device
  */
-void pci_enable_ari(struct pci_dev *dev)
+void pci_configure_ari(struct pci_dev *dev, bool enable)
 {
 	u32 cap;
 	struct pci_dev *bridge;
@@ -2031,10 +2031,16 @@ void pci_enable_ari(struct pci_dev *dev)
 	pcie_capability_read_dword(bridge, PCI_EXP_DEVCAP2, &cap);
 	if (!(cap & PCI_EXP_DEVCAP2_ARI))
 		return;
-
-	pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2, PCI_EXP_DEVCTL2_ARI);
-	bridge->ari_enabled = 1;
+	
+	if (enable) {
+		pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2, PCI_EXP_DEVCTL2_ARI);
+		bridge->ari_enabled = 1;
+	} else {
+		pcie_capability_clear_word(bridge, PCI_EXP_DEVCTL2, PCI_EXP_DEVCTL2_ARI);
+		bridge->ari_enabled = 0;
+	}
 }
+EXPORT_SYMBOL(pci_configure_ari);
 
 /**
  * pci_enable_ido - enable ID-based Ordering on a device
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index bacbcba..3eb05d4 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -211,7 +211,7 @@ extern int __pci_read_base(struct pci_dev *dev, enum pci_bar_type type,
 extern int pci_resource_bar(struct pci_dev *dev, int resno,
 			    enum pci_bar_type *type);
 extern int pci_bus_add_child(struct pci_bus *bus);
-extern void pci_enable_ari(struct pci_dev *dev);
+extern void pci_configure_ari(struct pci_dev *dev, bool enable);
 /**
  * pci_ari_enabled - query ARI forwarding status
  * @bus: the PCI bus
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index ec909af..9d6deb6 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1282,7 +1282,7 @@ static void pci_init_capabilities(struct pci_dev *dev)
 	pci_vpd_pci22_init(dev);
 
 	/* Alternative Routing-ID Forwarding */
-	pci_enable_ari(dev);
+	pci_configure_ari(dev, true);
 
 	/* Single Root I/O Virtualization */
 	pci_iov_init(dev);
-- 
1.7.1

[PATCH 2/7] PCI, acpiphp: disable ARI forwarding for acpiphp

[Yijing Wang]

Disable the port ARI forwarding after an ARI device hot removed.

Signed-off-by: Yijing Wang <wangyijing@huawei.com>
---
 drivers/pci/hotplug/acpiphp_glue.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/drivers/pci/hotplug/acpiphp_glue.c b/drivers/pci/hotplug/acpiphp_glue.c
index 3d6d4fd..6c2a1b6 100644
--- a/drivers/pci/hotplug/acpiphp_glue.c
+++ b/drivers/pci/hotplug/acpiphp_glue.c
@@ -944,6 +944,7 @@ static int disable_device(struct acpiphp_slot *slot)
 	 */
 	while ((pdev = dev_in_slot(slot))) {
 		pci_stop_and_remove_bus_device(pdev);
+		pci_configure_ari(pdev, false);
 		pci_dev_put(pdev);
 	}
 
-- 
1.7.1

[PATCH 3/7] PCI, pciehp: disable ARI forwarding for pciehp

[Yijing Wang]

Disable the port ARI forwarding after an ARI device hot removed.

Signed-off-by: Yijing Wang <wangyijing@huawei.com>
---
 drivers/pci/hotplug/pciehp_pci.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/drivers/pci/hotplug/pciehp_pci.c b/drivers/pci/hotplug/pciehp_pci.c
index 09cecaf..6d12d20 100644
--- a/drivers/pci/hotplug/pciehp_pci.c
+++ b/drivers/pci/hotplug/pciehp_pci.c
@@ -118,6 +118,7 @@ int pciehp_unconfigure_device(struct slot *p_slot)
 			}
 		}
 		pci_stop_and_remove_bus_device(temp);
+		pci_configure_ari(temp, false);
 		/*
 		 * Ensure that no new Requests will be generated from
 		 * the device.
-- 
1.7.1

[PATCH 4/7] PCI, cpqphp: disable ARI forwarding for cpqphp

[Yijing Wang]

Disable the port ARI forwarding after an ARI device hot removed.

Signed-off-by: Yijing Wang <wangyijing@huawei.com>
---
 drivers/pci/hotplug/cpci_hotplug_pci.c |    1 +
 drivers/pci/hotplug/cpqphp_pci.c       |    1 +
 2 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/drivers/pci/hotplug/cpci_hotplug_pci.c b/drivers/pci/hotplug/cpci_hotplug_pci.c
index dcc75c7..d59d228 100644
--- a/drivers/pci/hotplug/cpci_hotplug_pci.c
+++ b/drivers/pci/hotplug/cpci_hotplug_pci.c
@@ -319,6 +319,7 @@ int cpci_unconfigure_slot(struct slot* slot)
 				    PCI_DEVFN(PCI_SLOT(slot->devfn), i));
 		if (dev) {
 			pci_stop_and_remove_bus_device(dev);
+			pci_configure_ari(dev, false);
 			pci_dev_put(dev);
 		}
 	}
diff --git a/drivers/pci/hotplug/cpqphp_pci.c b/drivers/pci/hotplug/cpqphp_pci.c
index 09801c6..a818c1a 100644
--- a/drivers/pci/hotplug/cpqphp_pci.c
+++ b/drivers/pci/hotplug/cpqphp_pci.c
@@ -128,6 +128,7 @@ int cpqhp_unconfigure_device(struct pci_func* func)
 		if (temp) {
 			pci_dev_put(temp);
 			pci_stop_and_remove_bus_device(temp);
+			pci_configure_ari(temp, false);
 		}
 	}
 	return 0;
-- 
1.7.1

[PATCH 5/7] PCI, shpchp: disable ARI forwarding for shpchp

[Yijing Wang]

Disable the port ARI forwarding after an ARI device hot removed.

Signed-off-by: Yijing Wang <wangyijing@huawei.com>
---
 drivers/pci/hotplug/shpchp_pci.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/drivers/pci/hotplug/shpchp_pci.c b/drivers/pci/hotplug/shpchp_pci.c
index c627ed9..0f07577 100644
--- a/drivers/pci/hotplug/shpchp_pci.c
+++ b/drivers/pci/hotplug/shpchp_pci.c
@@ -110,6 +110,7 @@ int shpchp_unconfigure_device(struct slot *p_slot)
 			}
 		}
 		pci_stop_and_remove_bus_device(temp);
+		pci_configure_ari(temp, false);
 		pci_dev_put(temp);
 	}
 	return rc;
-- 
1.7.1

[PATCH 6/7] PCI, sgi: disable ARI forwarding for sgiphp

[Yijing Wang]

Disable the port ARI forwarding when a ARI device was hot removed.

Signed-off-by: Yijing Wang <wangyijing@huawei.com>
---
 drivers/pci/hotplug/sgi_hotplug.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/drivers/pci/hotplug/sgi_hotplug.c b/drivers/pci/hotplug/sgi_hotplug.c
index f64ca92..8d02367 100644
--- a/drivers/pci/hotplug/sgi_hotplug.c
+++ b/drivers/pci/hotplug/sgi_hotplug.c
@@ -553,6 +553,7 @@ static int disable_slot(struct hotplug_slot *bss_hotplug_slot)
 		if (dev) {
 			sn_bus_free_data(dev);
 			pci_stop_and_remove_bus_device(dev);
+			pci_configure_ari(dev, false);
 			pci_dev_put(dev);
 		}
 	}
-- 
1.7.1

[PATCH 7/7] PCI, ibmphp: disable ARI forwarding for ibmphp

[Yijing Wang]

Disable the port ARI forwarding when a ARI device was hot removed.

Signed-off-by: Yijing Wang <wangyijing@huawei.com>
---
 drivers/pci/hotplug/ibmphp_core.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/drivers/pci/hotplug/ibmphp_core.c b/drivers/pci/hotplug/ibmphp_core.c
index cbd72d8..c07afba 100644
--- a/drivers/pci/hotplug/ibmphp_core.c
+++ b/drivers/pci/hotplug/ibmphp_core.c
@@ -722,6 +722,7 @@ static void ibm_unconfigure_device(struct pci_func *func)
 		temp = pci_get_bus_and_slot(func->busno, (func->device << 3) | j);
 		if (temp) {
 			pci_stop_and_remove_bus_device(temp);
+			pci_configure_ari(temp, false);
 			pci_dev_put(temp);
 		}
 	}
-- 
1.7.1

Re: [PATCH 0/7] disable bridge ari forwarding after connected ari device hot removed

[Jiang Liu]

Seems no need to change cpci/cpqphp/ibmphp/sgi/shpcihp, they all deal with PCI devices,
so only need to change acpiphp and pciehp.

On 2012-10-9 11:03, Yijing Wang wrote:
> pci_enable_ari will be called if an ARI pci device found, then set its bridge ARI Forwarding Enable
> bit in Device Control 2 Register. But the bridge ARI Forwarding Enable bit will never be cleared
> when an ARI device hot removed.
> 
> my steps:
> 1. Hot add an ARI pci device;
> 2. Hot remove the ARI pci device;
> 3. Hot add an non ARI pci device;
> 
> In this case, after setp 3, we could only find fun 0 of non ARI pci device because of its bridge ARI Forwarding Enable
> bit set.
> 
> As PCIe Spec 2.0(6.13/441) recommends:
> "Following a hot-plug event below a Downstream Port, it is strongly recommended that software
> Clear the ARI Forwarding Enable bit in the Downstream Port until software determines that a
> newly added component is in fact an ARI Device"
> 
> This series of patches fix this problem.
> 
> Yijing Wang (7):
>   PCI: rework pci_enable_ari for support disable ari forwarding
>   PCI, acpiphp: disable ARI forwarding for acpiphp
>   PCI, pciehp: disable ARI forwarding for pciehp
>   PCI, cpqphp: disable ARI forwarding for cpqphp
>   PCI, shpchp: disable ARI forwarding for shpchp
>   PCI, sgi: disable ARI forwarding for sgiphp
>   PCI, ibmphp: disable ARI forwarding for ibmphp
> 
>  drivers/pci/hotplug/acpiphp_glue.c     |    1 +
>  drivers/pci/hotplug/cpci_hotplug_pci.c |    1 +
>  drivers/pci/hotplug/cpqphp_pci.c       |    1 +
>  drivers/pci/hotplug/ibmphp_core.c      |    1 +
>  drivers/pci/hotplug/pciehp_pci.c       |    1 +
>  drivers/pci/hotplug/sgi_hotplug.c      |    1 +
>  drivers/pci/hotplug/shpchp_pci.c       |    1 +
>  drivers/pci/pci.c                      |   16 +++++++++++-----
>  drivers/pci/pci.h                      |    2 +-
>  drivers/pci/probe.c                    |    2 +-
>  10 files changed, 20 insertions(+), 7 deletions(-)
> 
> 
> 
> .
> 

Re: [PATCH 0/7] disable bridge ari forwarding after connected ari device hot removed

[Yijing Wang]

On 2012/10/9 11:08, Jiang Liu wrote:
> Seems no need to change cpci/cpqphp/ibmphp/sgi/shpcihp, they all deal with PCI devices,
> so only need to change acpiphp and pciehp.
> 

ah, I will drop the patches for cpci/cpqphp/ibmphp/sgi/shpcihp.

Thanks!
Yijing

> On 2012-10-9 11:03, Yijing Wang wrote:
>> pci_enable_ari will be called if an ARI pci device found, then set its bridge ARI Forwarding Enable
>> bit in Device Control 2 Register. But the bridge ARI Forwarding Enable bit will never be cleared
>> when an ARI device hot removed.
>>
>> my steps:
>> 1. Hot add an ARI pci device;
>> 2. Hot remove the ARI pci device;
>> 3. Hot add an non ARI pci device;
>>
>> In this case, after setp 3, we could only find fun 0 of non ARI pci device because of its bridge ARI Forwarding Enable
>> bit set.
>>
>> As PCIe Spec 2.0(6.13/441) recommends:
>> "Following a hot-plug event below a Downstream Port, it is strongly recommended that software
>> Clear the ARI Forwarding Enable bit in the Downstream Port until software determines that a
>> newly added component is in fact an ARI Device"
>>
>> This series of patches fix this problem.
>>
>> Yijing Wang (7):
>>   PCI: rework pci_enable_ari for support disable ari forwarding
>>   PCI, acpiphp: disable ARI forwarding for acpiphp
>>   PCI, pciehp: disable ARI forwarding for pciehp
>>   PCI, cpqphp: disable ARI forwarding for cpqphp
>>   PCI, shpchp: disable ARI forwarding for shpchp
>>   PCI, sgi: disable ARI forwarding for sgiphp
>>   PCI, ibmphp: disable ARI forwarding for ibmphp
>>
>>  drivers/pci/hotplug/acpiphp_glue.c     |    1 +
>>  drivers/pci/hotplug/cpci_hotplug_pci.c |    1 +
>>  drivers/pci/hotplug/cpqphp_pci.c       |    1 +
>>  drivers/pci/hotplug/ibmphp_core.c      |    1 +
>>  drivers/pci/hotplug/pciehp_pci.c       |    1 +
>>  drivers/pci/hotplug/sgi_hotplug.c      |    1 +
>>  drivers/pci/hotplug/shpchp_pci.c       |    1 +
>>  drivers/pci/pci.c                      |   16 +++++++++++-----
>>  drivers/pci/pci.h                      |    2 +-
>>  drivers/pci/probe.c                    |    2 +-
>>  10 files changed, 20 insertions(+), 7 deletions(-)
>>
>>
>>
>> .
>>
> 
> 
> 
> .
> 


-- 
Thanks!
Yijing

Re: [PATCH 0/7] disable bridge ari forwarding after connected ari device hot removed

[Jiang Liu]

How about this patch, which disables ARI when adding new PCI devices?
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 5485883..c841aa6 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -2016,13 +2016,14 @@ void pci_free_cap_save_buffers(struct pci_dev *dev)
 void pci_enable_ari(struct pci_dev *dev)
 {
        u32 cap;
+       bool enable = true;
        struct pci_dev *bridge;

        if (pcie_ari_disabled || !pci_is_pcie(dev) || dev->devfn)
                return;

        if (!pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ARI))
-               return;
+               enable = false;

        bridge = dev->bus->self;
        if (!bridge)
@@ -2032,8 +2033,15 @@ void pci_enable_ari(struct pci_dev *dev)
        if (!(cap & PCI_EXP_DEVCAP2_ARI))
                return;

-       pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2, PCI_EXP_DEVCTL2_ARI);
-       bridge->ari_enabled = 1;
+       if (enable) {
+               pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2,
+                                        PCI_EXP_DEVCTL2_ARI);
+               bridge->ari_enabled = 1;
+       } else {
+               pcie_capability_clear_word(bridge, PCI_EXP_DEVCTL2,
+                                        PCI_EXP_DEVCTL2_ARI);
+               bridge->ari_enabled = 0;
+       }
 }

 /**

On 2012-10-9 11:03, Yijing Wang wrote:
> pci_enable_ari will be called if an ARI pci device found, then set its bridge ARI Forwarding Enable
> bit in Device Control 2 Register. But the bridge ARI Forwarding Enable bit will never be cleared
> when an ARI device hot removed.
> 
> my steps:
> 1. Hot add an ARI pci device;
> 2. Hot remove the ARI pci device;
> 3. Hot add an non ARI pci device;
> 
> In this case, after setp 3, we could only find fun 0 of non ARI pci device because of its bridge ARI Forwarding Enable
> bit set.
> 
> As PCIe Spec 2.0(6.13/441) recommends:
> "Following a hot-plug event below a Downstream Port, it is strongly recommended that software
> Clear the ARI Forwarding Enable bit in the Downstream Port until software determines that a
> newly added component is in fact an ARI Device"
> 
> This series of patches fix this problem.
> 
> Yijing Wang (7):
>   PCI: rework pci_enable_ari for support disable ari forwarding
>   PCI, acpiphp: disable ARI forwarding for acpiphp
>   PCI, pciehp: disable ARI forwarding for pciehp
>   PCI, cpqphp: disable ARI forwarding for cpqphp
>   PCI, shpchp: disable ARI forwarding for shpchp
>   PCI, sgi: disable ARI forwarding for sgiphp
>   PCI, ibmphp: disable ARI forwarding for ibmphp
> 
>  drivers/pci/hotplug/acpiphp_glue.c     |    1 +
>  drivers/pci/hotplug/cpci_hotplug_pci.c |    1 +
>  drivers/pci/hotplug/cpqphp_pci.c       |    1 +
>  drivers/pci/hotplug/ibmphp_core.c      |    1 +
>  drivers/pci/hotplug/pciehp_pci.c       |    1 +
>  drivers/pci/hotplug/sgi_hotplug.c      |    1 +
>  drivers/pci/hotplug/shpchp_pci.c       |    1 +
>  drivers/pci/pci.c                      |   16 +++++++++++-----
>  drivers/pci/pci.h                      |    2 +-
>  drivers/pci/probe.c                    |    2 +-
>  10 files changed, 20 insertions(+), 7 deletions(-)
> 
> 
> 
> .
> 

Re: [PATCH 0/7] disable bridge ari forwarding after connected ari device hot removed

[Yijing Wang]

On 2012/10/9 11:42, Jiang Liu wrote:
> How about this patch, which disables ARI when adding new PCI devices?
> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> index 5485883..c841aa6 100644
> --- a/drivers/pci/pci.c
> +++ b/drivers/pci/pci.c
> @@ -2016,13 +2016,14 @@ void pci_free_cap_save_buffers(struct pci_dev *dev)
>  void pci_enable_ari(struct pci_dev *dev)
>  {

I think it's ok. Delay to disable ARI until next hot add is safe.
Maybe rename as pci_configure_ari is better.

>         u32 cap;
> +       bool enable = true;
>         struct pci_dev *bridge;
> 
>         if (pcie_ari_disabled || !pci_is_pcie(dev) || dev->devfn)
>                 return;
> 
>         if (!pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ARI))
> -               return;
> +               enable = false;
> 
>         bridge = dev->bus->self;
>         if (!bridge)
> @@ -2032,8 +2033,15 @@ void pci_enable_ari(struct pci_dev *dev)
>         if (!(cap & PCI_EXP_DEVCAP2_ARI))
>                 return;
> 
> -       pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2, PCI_EXP_DEVCTL2_ARI);
> -       bridge->ari_enabled = 1;
> +       if (enable) {
> +               pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2,
> +                                        PCI_EXP_DEVCTL2_ARI);
> +               bridge->ari_enabled = 1;
> +       } else {
> +               pcie_capability_clear_word(bridge, PCI_EXP_DEVCTL2,
> +                                        PCI_EXP_DEVCTL2_ARI);
> +               bridge->ari_enabled = 0;
> +       }
>  }
> 
>  /**
>
> On 2012-10-9 11:03, Yijing Wang wrote:
>> pci_enable_ari will be called if an ARI pci device found, then set its bridge ARI Forwarding Enable
>> bit in Device Control 2 Register. But the bridge ARI Forwarding Enable bit will never be cleared
>> when an ARI device hot removed.
>>
>> my steps:
>> 1. Hot add an ARI pci device;
>> 2. Hot remove the ARI pci device;
>> 3. Hot add an non ARI pci device;
>>
>> In this case, after setp 3, we could only find fun 0 of non ARI pci device because of its bridge ARI Forwarding Enable
>> bit set.
>>
>> As PCIe Spec 2.0(6.13/441) recommends:
>> "Following a hot-plug event below a Downstream Port, it is strongly recommended that software
>> Clear the ARI Forwarding Enable bit in the Downstream Port until software determines that a
>> newly added component is in fact an ARI Device"
>>
>> This series of patches fix this problem.
>>
>> Yijing Wang (7):
>>   PCI: rework pci_enable_ari for support disable ari forwarding
>>   PCI, acpiphp: disable ARI forwarding for acpiphp
>>   PCI, pciehp: disable ARI forwarding for pciehp
>>   PCI, cpqphp: disable ARI forwarding for cpqphp
>>   PCI, shpchp: disable ARI forwarding for shpchp
>>   PCI, sgi: disable ARI forwarding for sgiphp
>>   PCI, ibmphp: disable ARI forwarding for ibmphp
>>
>>  drivers/pci/hotplug/acpiphp_glue.c     |    1 +
>>  drivers/pci/hotplug/cpci_hotplug_pci.c |    1 +
>>  drivers/pci/hotplug/cpqphp_pci.c       |    1 +
>>  drivers/pci/hotplug/ibmphp_core.c      |    1 +
>>  drivers/pci/hotplug/pciehp_pci.c       |    1 +
>>  drivers/pci/hotplug/sgi_hotplug.c      |    1 +
>>  drivers/pci/hotplug/shpchp_pci.c       |    1 +
>>  drivers/pci/pci.c                      |   16 +++++++++++-----
>>  drivers/pci/pci.h                      |    2 +-
>>  drivers/pci/probe.c                    |    2 +-
>>  10 files changed, 20 insertions(+), 7 deletions(-)
>>
>>
>>
>> .
>>
> 
> 
> 
> .
> 


-- 
Thanks!
Yijing

Re: [PATCH 0/7] disable bridge ari forwarding after connected ari device hot removed

[Jiang Liu]

When scanning PCI devices for hot-add, it guarantee function 0 is scanned
at first. And when destroying PCI devices for hot-removal, we also need to
destroy function 0 at last, but currently there's no such guarantees. 

On 2012-10-9 15:42, Yijing Wang wrote:
> On 2012/10/9 11:42, Jiang Liu wrote:
>> How about this patch, which disables ARI when adding new PCI devices?
>> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
>> index 5485883..c841aa6 100644
>> --- a/drivers/pci/pci.c
>> +++ b/drivers/pci/pci.c
>> @@ -2016,13 +2016,14 @@ void pci_free_cap_save_buffers(struct pci_dev *dev)
>>  void pci_enable_ari(struct pci_dev *dev)
>>  {
> 
> I think it's ok. Delay to disable ARI until next hot add is safe.
> Maybe rename as pci_configure_ari is better.
> 
>>         u32 cap;
>> +       bool enable = true;
>>         struct pci_dev *bridge;
>>
>>         if (pcie_ari_disabled || !pci_is_pcie(dev) || dev->devfn)
>>                 return;
>>
>>         if (!pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ARI))
>> -               return;
>> +               enable = false;
>>
>>         bridge = dev->bus->self;
>>         if (!bridge)
>> @@ -2032,8 +2033,15 @@ void pci_enable_ari(struct pci_dev *dev)
>>         if (!(cap & PCI_EXP_DEVCAP2_ARI))
>>                 return;
>>
>> -       pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2, PCI_EXP_DEVCTL2_ARI);
>> -       bridge->ari_enabled = 1;
>> +       if (enable) {
>> +               pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2,
>> +                                        PCI_EXP_DEVCTL2_ARI);
>> +               bridge->ari_enabled = 1;
>> +       } else {
>> +               pcie_capability_clear_word(bridge, PCI_EXP_DEVCTL2,
>> +                                        PCI_EXP_DEVCTL2_ARI);
>> +               bridge->ari_enabled = 0;
>> +       }
>>  }
>>
>>  /**
>>
>> On 2012-10-9 11:03, Yijing Wang wrote:
>>> pci_enable_ari will be called if an ARI pci device found, then set its bridge ARI Forwarding Enable
>>> bit in Device Control 2 Register. But the bridge ARI Forwarding Enable bit will never be cleared
>>> when an ARI device hot removed.
>>>
>>> my steps:
>>> 1. Hot add an ARI pci device;
>>> 2. Hot remove the ARI pci device;
>>> 3. Hot add an non ARI pci device;
>>>
>>> In this case, after setp 3, we could only find fun 0 of non ARI pci device because of its bridge ARI Forwarding Enable
>>> bit set.
>>>
>>> As PCIe Spec 2.0(6.13/441) recommends:
>>> "Following a hot-plug event below a Downstream Port, it is strongly recommended that software
>>> Clear the ARI Forwarding Enable bit in the Downstream Port until software determines that a
>>> newly added component is in fact an ARI Device"
>>>
>>> This series of patches fix this problem.
>>>
>>> Yijing Wang (7):
>>>   PCI: rework pci_enable_ari for support disable ari forwarding
>>>   PCI, acpiphp: disable ARI forwarding for acpiphp
>>>   PCI, pciehp: disable ARI forwarding for pciehp
>>>   PCI, cpqphp: disable ARI forwarding for cpqphp
>>>   PCI, shpchp: disable ARI forwarding for shpchp
>>>   PCI, sgi: disable ARI forwarding for sgiphp
>>>   PCI, ibmphp: disable ARI forwarding for ibmphp
>>>
>>>  drivers/pci/hotplug/acpiphp_glue.c     |    1 +
>>>  drivers/pci/hotplug/cpci_hotplug_pci.c |    1 +
>>>  drivers/pci/hotplug/cpqphp_pci.c       |    1 +
>>>  drivers/pci/hotplug/ibmphp_core.c      |    1 +
>>>  drivers/pci/hotplug/pciehp_pci.c       |    1 +
>>>  drivers/pci/hotplug/sgi_hotplug.c      |    1 +
>>>  drivers/pci/hotplug/shpchp_pci.c       |    1 +
>>>  drivers/pci/pci.c                      |   16 +++++++++++-----
>>>  drivers/pci/pci.h                      |    2 +-
>>>  drivers/pci/probe.c                    |    2 +-
>>>  10 files changed, 20 insertions(+), 7 deletions(-)
>>>
>>>
>>>
>>> .
>>>
>>
>>
>>
>> .
>>
> 
> 

Re: [PATCH 0/7] disable bridge ari forwarding after connected ari device hot removed

[Yijing Wang]

On 2012/10/9 15:51, Jiang Liu wrote:
> When scanning PCI devices for hot-add, it guarantee function 0 is scanned
> at first. And when destroying PCI devices for hot-removal, we also need to
> destroy function 0 at last, but currently there's no such guarantees. 
> 

This is an issue, btw in pciehp now only try to hot remove fun from 0 --> 8,
ARI device supports up to 256 function device.
for (j = 0; j < 8; j++) {
	struct pci_dev *temp = pci_get_slot(parent, PCI_DEVFN(0, j));
	if (!temp)
		continue;


> On 2012-10-9 15:42, Yijing Wang wrote:
>> On 2012/10/9 11:42, Jiang Liu wrote:
>>> How about this patch, which disables ARI when adding new PCI devices?
>>> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
>>> index 5485883..c841aa6 100644
>>> --- a/drivers/pci/pci.c
>>> +++ b/drivers/pci/pci.c
>>> @@ -2016,13 +2016,14 @@ void pci_free_cap_save_buffers(struct pci_dev *dev)
>>>  void pci_enable_ari(struct pci_dev *dev)
>>>  {
>>
>> I think it's ok. Delay to disable ARI until next hot add is safe.
>> Maybe rename as pci_configure_ari is better.
>>
>>>         u32 cap;
>>> +       bool enable = true;
>>>         struct pci_dev *bridge;
>>>
>>>         if (pcie_ari_disabled || !pci_is_pcie(dev) || dev->devfn)
>>>                 return;
>>>
>>>         if (!pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ARI))
>>> -               return;
>>> +               enable = false;
>>>
>>>         bridge = dev->bus->self;
>>>         if (!bridge)
>>> @@ -2032,8 +2033,15 @@ void pci_enable_ari(struct pci_dev *dev)
>>>         if (!(cap & PCI_EXP_DEVCAP2_ARI))
>>>                 return;
>>>
>>> -       pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2, PCI_EXP_DEVCTL2_ARI);
>>> -       bridge->ari_enabled = 1;
>>> +       if (enable) {
>>> +               pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2,
>>> +                                        PCI_EXP_DEVCTL2_ARI);
>>> +               bridge->ari_enabled = 1;
>>> +       } else {
>>> +               pcie_capability_clear_word(bridge, PCI_EXP_DEVCTL2,
>>> +                                        PCI_EXP_DEVCTL2_ARI);
>>> +               bridge->ari_enabled = 0;
>>> +       }
>>>  }
>>>
>>>  /**
>>>
>>> On 2012-10-9 11:03, Yijing Wang wrote:
>>>> pci_enable_ari will be called if an ARI pci device found, then set its bridge ARI Forwarding Enable
>>>> bit in Device Control 2 Register. But the bridge ARI Forwarding Enable bit will never be cleared
>>>> when an ARI device hot removed.
>>>>
>>>> my steps:
>>>> 1. Hot add an ARI pci device;
>>>> 2. Hot remove the ARI pci device;
>>>> 3. Hot add an non ARI pci device;
>>>>
>>>> In this case, after setp 3, we could only find fun 0 of non ARI pci device because of its bridge ARI Forwarding Enable
>>>> bit set.
>>>>
>>>> As PCIe Spec 2.0(6.13/441) recommends:
>>>> "Following a hot-plug event below a Downstream Port, it is strongly recommended that software
>>>> Clear the ARI Forwarding Enable bit in the Downstream Port until software determines that a
>>>> newly added component is in fact an ARI Device"
>>>>
>>>> This series of patches fix this problem.
>>>>
>>>> Yijing Wang (7):
>>>>   PCI: rework pci_enable_ari for support disable ari forwarding
>>>>   PCI, acpiphp: disable ARI forwarding for acpiphp
>>>>   PCI, pciehp: disable ARI forwarding for pciehp
>>>>   PCI, cpqphp: disable ARI forwarding for cpqphp
>>>>   PCI, shpchp: disable ARI forwarding for shpchp
>>>>   PCI, sgi: disable ARI forwarding for sgiphp
>>>>   PCI, ibmphp: disable ARI forwarding for ibmphp
>>>>
>>>>  drivers/pci/hotplug/acpiphp_glue.c     |    1 +
>>>>  drivers/pci/hotplug/cpci_hotplug_pci.c |    1 +
>>>>  drivers/pci/hotplug/cpqphp_pci.c       |    1 +
>>>>  drivers/pci/hotplug/ibmphp_core.c      |    1 +
>>>>  drivers/pci/hotplug/pciehp_pci.c       |    1 +
>>>>  drivers/pci/hotplug/sgi_hotplug.c      |    1 +
>>>>  drivers/pci/hotplug/shpchp_pci.c       |    1 +
>>>>  drivers/pci/pci.c                      |   16 +++++++++++-----
>>>>  drivers/pci/pci.h                      |    2 +-
>>>>  drivers/pci/probe.c                    |    2 +-
>>>>  10 files changed, 20 insertions(+), 7 deletions(-)
>>>>
>>>>
>>>>
>>>> .
>>>>
>>>
>>>
>>>
>>> .
>>>
>>
>>
> 
> 
> 
> .
> 


-- 
Thanks!
Yijing

Re: [PATCH 0/7] disable bridge ari forwarding after connected ari device hot removed

[Jiang Liu]

On 2012-10-9 16:14, Yijing Wang wrote:
> On 2012/10/9 15:51, Jiang Liu wrote:
>> When scanning PCI devices for hot-add, it guarantee function 0 is scanned
>> at first. And when destroying PCI devices for hot-removal, we also need to
>> destroy function 0 at last, but currently there's no such guarantees. 
>>
> 
> This is an issue, btw in pciehp now only try to hot remove fun from 0 --> 8,
> ARI device supports up to 256 function device.
> for (j = 0; j < 8; j++) {
> 	struct pci_dev *temp = pci_get_slot(parent, PCI_DEVFN(0, j));
> 	if (!temp)
> 		continue;
Another patch to fix.

Re: [PATCH 0/7] disable bridge ari forwarding after connected ari device hot removed

[Jiang Liu]

Please use this as your base
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 5485883..8a277d7 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -2018,22 +2018,25 @@ void pci_enable_ari(struct pci_dev *dev)
        u32 cap;
        struct pci_dev *bridge;

-       if (pcie_ari_disabled || !pci_is_pcie(dev) || dev->devfn)
-               return;
-
-       if (!pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ARI))
+       if (pcie_ari_disabled || !pci_is_pcie(dev))
                return;

        bridge = dev->bus->self;
-       if (!bridge)
+       if (!bridge || !list_empty(bridge->bus_list))
                return;
-
        pcie_capability_read_dword(bridge, PCI_EXP_DEVCAP2, &cap);
        if (!(cap & PCI_EXP_DEVCAP2_ARI))
                return;

-       pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2, PCI_EXP_DEVCTL2_ARI);
-       bridge->ari_enabled = 1;
+       if (pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ARI)) {
+               pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2,
+                                        PCI_EXP_DEVCTL2_ARI);
+               bridge->ari_enabled = 1;
+       } else {
+               pcie_capability_clear_word(bridge, PCI_EXP_DEVCTL2,
+                                          PCI_EXP_DEVCTL2_ARI);
+               bridge->ari_enabled = 0;
+       }
 }

 /**

On 2012-10-9 16:14, Yijing Wang wrote:
> On 2012/10/9 15:51, Jiang Liu wrote:
>> When scanning PCI devices for hot-add, it guarantee function 0 is scanned
>> at first. And when destroying PCI devices for hot-removal, we also need to
>> destroy function 0 at last, but currently there's no such guarantees. 
>>
> 
> This is an issue, btw in pciehp now only try to hot remove fun from 0 --> 8,
> ARI device supports up to 256 function device.
> for (j = 0; j < 8; j++) {
> 	struct pci_dev *temp = pci_get_slot(parent, PCI_DEVFN(0, j));
> 	if (!temp)
> 		continue;
> 
> 
>> On 2012-10-9 15:42, Yijing Wang wrote:
>>> On 2012/10/9 11:42, Jiang Liu wrote:
>>>> How about this patch, which disables ARI when adding new PCI devices?
>>>> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
>>>> index 5485883..c841aa6 100644
>>>> --- a/drivers/pci/pci.c
>>>> +++ b/drivers/pci/pci.c
>>>> @@ -2016,13 +2016,14 @@ void pci_free_cap_save_buffers(struct pci_dev *dev)
>>>>  void pci_enable_ari(struct pci_dev *dev)
>>>>  {
>>>
>>> I think it's ok. Delay to disable ARI until next hot add is safe.
>>> Maybe rename as pci_configure_ari is better.
>>>
>>>>         u32 cap;
>>>> +       bool enable = true;
>>>>         struct pci_dev *bridge;
>>>>
>>>>         if (pcie_ari_disabled || !pci_is_pcie(dev) || dev->devfn)
>>>>                 return;
>>>>
>>>>         if (!pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ARI))
>>>> -               return;
>>>> +               enable = false;
>>>>
>>>>         bridge = dev->bus->self;
>>>>         if (!bridge)
>>>> @@ -2032,8 +2033,15 @@ void pci_enable_ari(struct pci_dev *dev)
>>>>         if (!(cap & PCI_EXP_DEVCAP2_ARI))
>>>>                 return;
>>>>
>>>> -       pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2, PCI_EXP_DEVCTL2_ARI);
>>>> -       bridge->ari_enabled = 1;
>>>> +       if (enable) {
>>>> +               pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2,
>>>> +                                        PCI_EXP_DEVCTL2_ARI);
>>>> +               bridge->ari_enabled = 1;
>>>> +       } else {
>>>> +               pcie_capability_clear_word(bridge, PCI_EXP_DEVCTL2,
>>>> +                                        PCI_EXP_DEVCTL2_ARI);
>>>> +               bridge->ari_enabled = 0;
>>>> +       }
>>>>  }
>>>>
>>>>  /**
>>>>
>>>> On 2012-10-9 11:03, Yijing Wang wrote:
>>>>> pci_enable_ari will be called if an ARI pci device found, then set its bridge ARI Forwarding Enable
>>>>> bit in Device Control 2 Register. But the bridge ARI Forwarding Enable bit will never be cleared
>>>>> when an ARI device hot removed.
>>>>>
>>>>> my steps:
>>>>> 1. Hot add an ARI pci device;
>>>>> 2. Hot remove the ARI pci device;
>>>>> 3. Hot add an non ARI pci device;
>>>>>
>>>>> In this case, after setp 3, we could only find fun 0 of non ARI pci device because of its bridge ARI Forwarding Enable
>>>>> bit set.
>>>>>
>>>>> As PCIe Spec 2.0(6.13/441) recommends:
>>>>> "Following a hot-plug event below a Downstream Port, it is strongly recommended that software
>>>>> Clear the ARI Forwarding Enable bit in the Downstream Port until software determines that a
>>>>> newly added component is in fact an ARI Device"
>>>>>
>>>>> This series of patches fix this problem.
>>>>>
>>>>> Yijing Wang (7):
>>>>>   PCI: rework pci_enable_ari for support disable ari forwarding
>>>>>   PCI, acpiphp: disable ARI forwarding for acpiphp
>>>>>   PCI, pciehp: disable ARI forwarding for pciehp
>>>>>   PCI, cpqphp: disable ARI forwarding for cpqphp
>>>>>   PCI, shpchp: disable ARI forwarding for shpchp
>>>>>   PCI, sgi: disable ARI forwarding for sgiphp
>>>>>   PCI, ibmphp: disable ARI forwarding for ibmphp
>>>>>
>>>>>  drivers/pci/hotplug/acpiphp_glue.c     |    1 +
>>>>>  drivers/pci/hotplug/cpci_hotplug_pci.c |    1 +
>>>>>  drivers/pci/hotplug/cpqphp_pci.c       |    1 +
>>>>>  drivers/pci/hotplug/ibmphp_core.c      |    1 +
>>>>>  drivers/pci/hotplug/pciehp_pci.c       |    1 +
>>>>>  drivers/pci/hotplug/sgi_hotplug.c      |    1 +
>>>>>  drivers/pci/hotplug/shpchp_pci.c       |    1 +
>>>>>  drivers/pci/pci.c                      |   16 +++++++++++-----
>>>>>  drivers/pci/pci.h                      |    2 +-
>>>>>  drivers/pci/probe.c                    |    2 +-
>>>>>  10 files changed, 20 insertions(+), 7 deletions(-)
>>>>>
>>>>>
>>>>>
>>>>> .
>>>>>
>>>>
>>>>
>>>>
>>>> .
>>>>
>>>
>>>
>>
>>
>>
>> .
>>
> 
>