[PATCH -v2 1/2] PCI: decrease pci_dev->enable_cnt when no pcie capability found

[PATCH -v2 1/2] PCI: decrease pci_dev->enable_cnt when no pcie capability found

[Yijing Wang]

We should decrease dev->enable_cnt when no pcie port capability
found for balance.

Signed-off-by: Yijing Wang <wangyijing@huawei.com>
Cc: Jiang Liu <jiang.liu@huawei.com>
Cc: Kenji Kaneshige <kaneshige.kenji@jp.fujitsu.com>
Cc: Shengzhou Liu <Shengzhou.Liu@freescale.com>
---
 drivers/pci/pcie/portdrv_core.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/pcie/portdrv_core.c b/drivers/pci/pcie/portdrv_core.c
index 31063ac..aef3fac 100644
--- a/drivers/pci/pcie/portdrv_core.c
+++ b/drivers/pci/pcie/portdrv_core.c
@@ -369,8 +369,8 @@ int pcie_port_device_register(struct pci_dev *dev)
 
 	/* Get and check PCI Express port services */
 	capabilities = get_port_device_capability(dev);
-	if (!capabilities)
-		return 0;
+	if (!capabilities)
+		goto error_disable;
 
 	pci_set_master(dev);
 	/*
-- 
1.7.1

[PATCH -v2 2/2] PCI/IA64: fix pci_dev->enable_cnt balance when doing pci hotplug

[Yijing Wang]

v1-->v2: use pci_assign_unassigned_resources() in pcibios_init()
         intead of directly calling pci_enable_bridges() in pci_acpi_scan_root()
         to avoid double enable pci bridges when hot add pci host bridge in IA64.

Under IA64 platform, we don't call pci_enable_bridges()
when scan all pci buses during system boot up. But under
X86 we do it. Further, IA64 don't assign the unassigned pci device
resource in boot path. This patch fix the pcie port device unbalanced
enable_cnt and make IA64 support pci device resource reassignment.

pcibios_assign_resources()
   pci_assign_unassigned_resources()
       ...........
       pci_enable_bridges()
          pci_enable_device()  first increase enable_cnt here

pcie_portdrv_probe()
   pcie_port_device_register()
       pci_enable_device()   second increase enable_cnt

Then when we doing hot remove

acpiphp_disable_slot()
   pci_stop_and_remove_bus_device()
       pci_stop_bus_device()
          .............
          pcie_portdrv_remove()
			  pcie_port_device_remove()
                  pci_disable_device()   first decrease enable_cnt  here
			  pci_disable_device()     second decrease enable_cnt

Following Warning info found under IA64 when doing pci hotplug.

------------[ cut here ]------------
WARNING: at drivers/pci/pci.c:1397 pci_disable_device+0x1c0/0x220()
Hardware name: MH8900
Device pcieport
disabling already-disabled device
Modules linked in: acpiphp ipv6 ipmi_si(+) ipmi_devintf ipmi_msghandler fuse vfaa
t fat dm_mod iTCO_wdt iTCO_vendor_support lpc_ich i2c_i801 mfd_core i2c_core sg
sd_mod crc_t10dif ext3 mbcache jbd ata_piix

Call Trace:
 [<a000000100015c00>] show_stack+0x80/0xa0
                                sp=e000000fd629fc00 bsp=e000000fd62996e0
 [<a000000100a9ca20>] dump_stack+0x30/0x50
                                sp=e000000fd629fdd0 bsp=e000000fd62996c8
 [<a000000100061960>] warn_slowpath_common+0xc0/0x100
                                sp=e000000fd629fdd0 bsp=e000000fd6299688
 [<a000000100061b50>] warn_slowpath_fmt+0x90/0xc0
                                sp=e000000fd629fdd0 bsp=e000000fd6299628
 [<a000000100495be0>] pci_disable_device+0x1c0/0x220
                                sp=e000000fd629fe10 bsp=e000000fd62995e8
 [<a0000001004b3ba0>] pcie_portdrv_remove+0xc0/0xe0
                                sp=e000000fd629fe10 bsp=e000000fd62995c8
 [<a000000100499670>] pci_device_remove+0x90/0x1e0
                                sp=e000000fd629fe10 bsp=e000000fd6299598
 [<a000000100636490>] __device_release_driver+0x150/0x280
                                sp=e000000fd629fe10 bsp=e000000fd6299560
 [<a000000100636830>] device_release_driver+0x30/0x60
                                sp=e000000fd629fe10 bsp=e000000fd6299538
 [<a000000100634a40>] bus_remove_device+0x2c0/0x3c0
                                sp=e000000fd629fe10 bsp=e000000fd62994f0
 [<a0000001006306d0>] device_del+0x290/0x440
                                sp=e000000fd629fe10 bsp=e000000fd62994a8
 [<a00000010048d550>] pci_stop_bus_device+0x150/0x200
                                sp=e000000fd629fe10 bsp=e000000fd6299478
 [<a00000010048d470>] pci_stop_bus_device+0x70/0x200
                                sp=e000000fd629fe10 bsp=e000000fd6299448
 [<a00000010048d470>] pci_stop_bus_device+0x70/0x200
                                sp=e000000fd629fe10 bsp=e000000fd6299418
 [<a00000010048d9a0>] pci_stop_and_remove_bus_device+0x20/0x60
                                sp=e000000fd629fe10 bsp=e000000fd62993f0
 [<a0000002089aa100>] acpiphp_disable_slot+0x240/0x4e0 [acpiphp]
                                sp=e000000fd629fe10 bsp=e000000fd62993a0
 [<a0000002089a8a30>] disable_slot+0x50/0x160 [acpiphp]
                                sp=e000000fd629fe20 bsp=e000000fd6299378
 [<a0000001004ba080>] power_write_file+0x140/0x2a0
                                sp=e000000fd629fe20 bsp=e000000fd6299348
 [<a0000001004a6320>] pci_slot_attr_store+0x60/0xa0
                                sp=e000000fd629fe20 bsp=e000000fd6299310
 [<a00000010032a100>] sysfs_write_file+0x240/0x340
                                sp=e000000fd629fe20 bsp=e000000fd62992b8
 [<a00000010023c910>] vfs_write+0x1b0/0x3a0
                                sp=e000000fd629fe20 bsp=e000000fd6299270
 [<a00000010023cc90>] sys_write+0x90/0xe0
                                sp=e000000fd629fe20 bsp=e000000fd62991f0
 [<a00000010000bce0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e000000fd629fe30 bsp=e000000fd62991f0
 [<a000000000040720>] __kernel_syscall_via_break+0x0/0x20
                                sp=e000000fd62a0000 bsp=e000000fd62991f0
---[ end trace 34d87c78dbff78ce ]---
GSI 37 (level, low) -> CPU 15 (0x01e0) vector 68 unregistered
pcie_pme 0000:00:07.0:pcie01: unloading service driver pcie_pme
aer 0000:00:07.0:pcie02: unloading service driver aer

Signed-off-by: Yijing Wang <wangyijing@huawei.com>
Cc: Fenghua Yu <fenghua.yu@intel.com>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Thierry Reding <thierry.reding@avionic-design.de>
Cc: "Rafael J. Wysocki" <rafael.j.wysocki@intel.com>
---
 arch/ia64/pci/pci.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/arch/ia64/pci/pci.c b/arch/ia64/pci/pci.c
index 60532ab..900660f 100644
--- a/arch/ia64/pci/pci.c
+++ b/arch/ia64/pci/pci.c
@@ -724,6 +724,7 @@ EXPORT_SYMBOL_GPL(dma_get_required_mask);
 static int __init pcibios_init(void)
 {
 	set_pci_dfl_cacheline_size();
+	pci_assign_unassigned_resources();
 	return 0;
 }
 
-- 
1.7.1

Re: [PATCH -v2 1/2] PCI: decrease pci_dev->enable_cnt when no pcie capability found

[Bjorn Helgaas]

[+cc Rafael, Huang]

On Mon, Apr 15, 2013 at 12:58 AM, Yijing Wang <wangyijing@huawei.com> wrote:
> We should decrease dev->enable_cnt when no pcie port capability
> found for balance.
>
> Signed-off-by: Yijing Wang <wangyijing@huawei.com>
> Cc: Jiang Liu <jiang.liu@huawei.com>
> Cc: Kenji Kaneshige <kaneshige.kenji@jp.fujitsu.com>
> Cc: Shengzhou Liu <Shengzhou.Liu@freescale.com>
> ---
>  drivers/pci/pcie/portdrv_core.c |    4 ++--
>  1 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/pci/pcie/portdrv_core.c b/drivers/pci/pcie/portdrv_core.c
> index 31063ac..aef3fac 100644
> --- a/drivers/pci/pcie/portdrv_core.c
> +++ b/drivers/pci/pcie/portdrv_core.c
> @@ -369,8 +369,8 @@ int pcie_port_device_register(struct pci_dev *dev)
>
>         /* Get and check PCI Express port services */
>         capabilities = get_port_device_capability(dev);
> -       if (!capabilities)
> -               return 0;
> +       if (!capabilities)
> +               goto error_disable;
>
>         pci_set_master(dev);
>         /*

Does this fix a problem you observed?  If so, please refer to it in
your changelog.

I think this patch is incorrect because pcie_portdrv_probe() will
return 0 (success) with the device disabled.  When we call
pcie_portdrv_remove(), we will attempt to disable the device again,
even though it's already disabled.

I don't know whether it is desirable for pcie_portdrv_probe() to
succeed when no capabilities are available or not.  Maybe somebody
else has an opinion.

Bjorn

Re: [PATCH -v2 1/2] PCI: decrease pci_dev->enable_cnt when no pcie capability found

[huang ying]

On Mon, Apr 15, 2013 at 11:27 PM, Bjorn Helgaas <bhelgaas@google.com> wrote:
> [+cc Rafael, Huang]
>
> On Mon, Apr 15, 2013 at 12:58 AM, Yijing Wang <wangyijing@huawei.com> wrote:
>> We should decrease dev->enable_cnt when no pcie port capability
>> found for balance.
>>
>> Signed-off-by: Yijing Wang <wangyijing@huawei.com>
>> Cc: Jiang Liu <jiang.liu@huawei.com>
>> Cc: Kenji Kaneshige <kaneshige.kenji@jp.fujitsu.com>
>> Cc: Shengzhou Liu <Shengzhou.Liu@freescale.com>
>> ---
>>  drivers/pci/pcie/portdrv_core.c |    4 ++--
>>  1 files changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/drivers/pci/pcie/portdrv_core.c b/drivers/pci/pcie/portdrv_core.c
>> index 31063ac..aef3fac 100644
>> --- a/drivers/pci/pcie/portdrv_core.c
>> +++ b/drivers/pci/pcie/portdrv_core.c
>> @@ -369,8 +369,8 @@ int pcie_port_device_register(struct pci_dev *dev)
>>
>>         /* Get and check PCI Express port services */
>>         capabilities = get_port_device_capability(dev);
>> -       if (!capabilities)
>> -               return 0;
>> +       if (!capabilities)
>> +               goto error_disable;
>>
>>         pci_set_master(dev);
>>         /*
>
> Does this fix a problem you observed?  If so, please refer to it in
> your changelog.
>
> I think this patch is incorrect because pcie_portdrv_probe() will
> return 0 (success) with the device disabled.  When we call
> pcie_portdrv_remove(), we will attempt to disable the device again,
> even though it's already disabled.
>
> I don't know whether it is desirable for pcie_portdrv_probe() to
> succeed when no capabilities are available or not.  Maybe somebody
> else has an opinion.

I think this patchset is incorrect too.  Even if there is no PCIe
services for the PCIe port.  We still need the PCIe drivers, at least
for runtime power management.  Although runtime power management is
disabled for the PCIe port now, I think we will enable it again after
we fixed the corresponding issues.

Best Regards,
Huang Ying

Re: [PATCH -v2 1/2] PCI: decrease pci_dev->enable_cnt when no pcie capability found

[Yijing Wang]

[-- Attachment #1: Type: text/plain, Size: 4505 bytes --]

>> diff --git a/drivers/pci/pcie/portdrv_core.c b/drivers/pci/pcie/portdrv_core.c
>> index 31063ac..aef3fac 100644
>> --- a/drivers/pci/pcie/portdrv_core.c
>> +++ b/drivers/pci/pcie/portdrv_core.c
>> @@ -369,8 +369,8 @@ int pcie_port_device_register(struct pci_dev *dev)
>>
>>         /* Get and check PCI Express port services */
>>         capabilities = get_port_device_capability(dev);
>> -       if (!capabilities)
>> -               return 0;
>> +       if (!capabilities)
>> +               goto error_disable;
>>
>>         pci_set_master(dev);
>>         /*
> 
> Does this fix a problem you observed?  If so, please refer to it in
> your changelog.

Hi Bjorn,
   I found this problem when I try to fix the problem described in
[PATCH 2/2] PCI/IA64: fix pci_dev->enable_cnt balance when doing pci hotplug.

> 
> I think this patch is incorrect because pcie_portdrv_probe() will
> return 0 (success) with the device disabled.  When we call
> pcie_portdrv_remove(), we will attempt to disable the device again,
> even though it's already disabled.

Hmm, that's a problem, the driver will disable pcie port device twice regardless
any pcie capabilities found in the pcie port. There is another problem here.

enable pci bridge device:
1. first call pci_enable_bridges() after pci device resource assignment.
2. second call pci_enable_device() in pcie_port_device_register() in pcie port driver .probe.
above enable path, fist is in pci level, and second in pcie level.

disable pci bridge device:
1. first call pci_disable_device() in pcie_port_device_remove().
2. second call pci_disable_device() in pcie_portdrv_remove().

above disable path, first and second disable action are both in pcie level.

I think the enable and disable actions are not symmetric.

So it will cause another problem like this:
If we unbind a pcie port device driver, the pcie port device will be disabled by the pcie port driver.
the busMaster and irq.. will be disabled. So if there are some child devices under this port, this devices
maybe encounter problems during running, in my ia64, the child device network cannot transmit data anymore.

-+-[0000:40]-+-00.0-[0000:41]--
 |           +-01.0-[0000:42]--+-00.0  Intel Corporation 82576 Gigabit Network Connection
 |           |                 \-00.1  Intel Corporation 82576 Gigabit Network Connection
 |           +-03.0-[0000:43]----00.0  LSI Logic / Symbios Logic SAS1064ET PCI-Express Fusion-MPT SAS

linux-ha2:~ # lspci -vvv -s 0000:40:01.0 > before_unbind.log
linux-ha2:~ # cd /sys/bus/pci/devices/0000\:40\:01.0/driver/
linux-ha2:/sys/bus/pci/devices/0000:40:01.0/driver # ls
0000:00:01.0  0000:00:04.0  0000:00:07.0  0000:00:1c.1  0000:00:1c.3  0000:00:1c.5  0000:40:01.0  0000:40:04.0  0000:40:07.0  new_id     uevent
0000:00:03.0  0000:00:05.0  0000:00:1c.0  0000:00:1c.2  0000:00:1c.4  0000:40:00.0  0000:40:03.0  0000:40:05.0  bind          remove_id  unbind
linux-ha2:/sys/bus/pci/devices/0000:40:01.0/driver # echo "0000:40:01.0" > unbind
linux-ha2:/sys/bus/pci/devices/0000:40:01.0/driver # cd
linux-ha2:~ # lspci -vvv -s 0000:40:01.0 > after_unbind.log
linux-ha2:~ # diff before_unbind.log after_unbind.log
2c2
< 	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
---
> 	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
4d3
< 	Latency: 0, Cache Line Size: 64 bytes
13c12
< 	Capabilities: [60] Message Signalled Interrupts: Mask+ 64bit- Count=1/2 Enable+
---
> 	Capabilities: [60] Message Signalled Interrupts: Mask+ 64bit- Count=1/2 Enable-
15c14
< 		Masking: 00000002  Pending: 00000000
---
> 		Masking: 00000000  Pending: 00000000
19c18
< 		DevCtl:	Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
---
> 		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
34c33
< 		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
---
> 		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
57d55
< 	Kernel driver in use: pcieport

I prefer to move the second pci_disable_device() into driver/pci/remove.c. Disable pci bridge
during stopping the pci bridge. So we enable and disable the pcie port device symmetrically.

I tested the following attached patch, and result is ok.


Thanks!
Yijing.



> 
> I don't know whether it is desirable for pcie_portdrv_probe() to
> succeed when no capabilities are available or not.  Maybe somebody
> else has an opinion.
> 







-- 
Thanks!
Yijing

[-- Attachment #2: 0001-PCI-move-second-pci_disable_device-into-pci_stop_bus.patch --]
[-- Type: text/x-patch, Size: 1661 bytes --]

>From 44914e0e39dbe51e1a932492d6b4909d5967308e Mon Sep 17 00:00:00 2001
From: Yijing Wang <wangyijing@huawei.com>
Date: Tue, 16 Apr 2013 11:41:47 +0800
Subject: [PATCH] PCI: move second pci_disable_device into pci_stop_bus_device() for symmetry

Currently, we enable and disable pcie port device is not symmetrical. If
we unbind the pcie port driver for pcie port device, we will call pci_disable_device()
twice. Then the pcie port device is disabled, if there are some child devices
under it, the child device maybe cannot transmit data anymore. This patch move the
second pci_disable_device() int pci_stop_bus_device() to avoid this bug.

Signed-off-by: Yijing Wang <wangyijing@huawei.com>
---
 drivers/pci/pcie/portdrv_pci.c |    1 -
 drivers/pci/remove.c           |    1 +
 2 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/drivers/pci/pcie/portdrv_pci.c b/drivers/pci/pcie/portdrv_pci.c
index ed4d094..2ca1a0b 100644
--- a/drivers/pci/pcie/portdrv_pci.c
+++ b/drivers/pci/pcie/portdrv_pci.c
@@ -223,7 +223,6 @@ static int pcie_portdrv_probe(struct pci_dev *dev,
 static void pcie_portdrv_remove(struct pci_dev *dev)
 {
 	pcie_port_device_remove(dev);
-	pci_disable_device(dev);
 }
 
 static int error_detected_iter(struct device *device, void *data)
diff --git a/drivers/pci/remove.c b/drivers/pci/remove.c
index cc875e6..e8f7c3c 100644
--- a/drivers/pci/remove.c
+++ b/drivers/pci/remove.c
@@ -73,6 +73,7 @@ static void pci_stop_bus_device(struct pci_dev *dev)
 		list_for_each_entry_safe_reverse(child, tmp,
 						 &bus->devices, bus_list)
 			pci_stop_bus_device(child);
+			pci_disable_device(dev);
 	}
 
 	pci_stop_dev(dev);
-- 
1.7.1

Re: [PATCH -v2 1/2] PCI: decrease pci_dev->enable_cnt when no pcie capability found

[Yijing Wang]

On 2013/4/16 16:22, huang ying wrote:
> On Mon, Apr 15, 2013 at 11:27 PM, Bjorn Helgaas <bhelgaas@google.com> wrote:
>> [+cc Rafael, Huang]
>>
>> On Mon, Apr 15, 2013 at 12:58 AM, Yijing Wang <wangyijing@huawei.com> wrote:
>>> We should decrease dev->enable_cnt when no pcie port capability
>>> found for balance.
>>>
>>> Signed-off-by: Yijing Wang <wangyijing@huawei.com>
>>> Cc: Jiang Liu <jiang.liu@huawei.com>
>>> Cc: Kenji Kaneshige <kaneshige.kenji@jp.fujitsu.com>
>>> Cc: Shengzhou Liu <Shengzhou.Liu@freescale.com>
>>> ---
>>>  drivers/pci/pcie/portdrv_core.c |    4 ++--
>>>  1 files changed, 2 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/drivers/pci/pcie/portdrv_core.c b/drivers/pci/pcie/portdrv_core.c
>>> index 31063ac..aef3fac 100644
>>> --- a/drivers/pci/pcie/portdrv_core.c
>>> +++ b/drivers/pci/pcie/portdrv_core.c
>>> @@ -369,8 +369,8 @@ int pcie_port_device_register(struct pci_dev *dev)
>>>
>>>         /* Get and check PCI Express port services */
>>>         capabilities = get_port_device_capability(dev);
>>> -       if (!capabilities)
>>> -               return 0;
>>> +       if (!capabilities)
>>> +               goto error_disable;
>>>
>>>         pci_set_master(dev);
>>>         /*
>>
>> Does this fix a problem you observed?  If so, please refer to it in
>> your changelog.
>>
>> I think this patch is incorrect because pcie_portdrv_probe() will
>> return 0 (success) with the device disabled.  When we call
>> pcie_portdrv_remove(), we will attempt to disable the device again,
>> even though it's already disabled.
>>
>> I don't know whether it is desirable for pcie_portdrv_probe() to
>> succeed when no capabilities are available or not.  Maybe somebody
>> else has an opinion.
> 
> I think this patchset is incorrect too.  Even if there is no PCIe
> services for the PCIe port.  We still need the PCIe drivers, at least
> for runtime power management.  Although runtime power management is
> disabled for the PCIe port now, I think we will enable it again after
> we fixed the corresponding issues.

Hi Huang Ying,
   Thanks for your comments! I will drop this patch, because as you said pcie port
device need pcie port driver regardless any pcie capabilities found.

But there is still a problem about two pci_disable_device() called in pcie_portdrv_remove().
In this case, If we unbind pcie port driver, the pcie port device will be disabled. The child devices
under it cannot use anymore. I send this patch in another reply.

Thanks!
Yijing

> 
> Best Regards,
> Huang Ying
> 
> .
> 


-- 
Thanks!
Yijing