[PATCH RESEND] (revised) Calgary: increase max PHB number

[PATCH RESEND] (revised) Calgary: increase max PHB number

[Corinna Schultz]

There was a small difference in 35-rc3 that prevented this patch from  
applying, so I regenerated it.

===============================

Newer systems (x3950M2) can have 48 PHBs per chassis and 8 chassis, so  
bump the
limits up and provide an explanation of the requirements for each class.

Signed-off-by: Darrick J. Wong <djwong@us.ibm.com>
Acked-by: Muli Ben-Yehuda <muli@il.ibm.com>

--- a/arch/x86/kernel/pci_calgary_64.c     2010-06-21 10:57:15.000000000 -0700
+++ b/arch/x86/kernel/pci_calgary_64.c    2010-06-21 11:01:56.000000000 -0700
@@ -103,10 +103,15 @@
  #define PMR_SOFTSTOPFAULT      0x40000000
  #define PMR_HARDSTOP           0x20000000

-#define MAX_NUM_OF_PHBS                8 /* how many PHBs in total? */
-#define MAX_NUM_CHASSIS                8 /* max number of chassis */
-/* MAX_PHB_BUS_NUM is the maximal possible dev->bus->number */
-#define MAX_PHB_BUS_NUM                (MAX_NUM_OF_PHBS *  
MAX_NUM_CHASSIS * 2)
+/*
+ * The maximum PHB bus number.
+ * x3950M2 (rare): 8 chassis, 48 PHBs per chassis = 384
+ * x3950M2: 4 chassis, 48 PHBs per chassis        = 192
+ * x3950 (PCIE): 8 chassis, 32 PHBs per chassis   = 256
+ * x3950 (PCIX): 8 chassis, 16 PHBs per chassis   = 128
+ */
+#define MAX_PHB_BUS_NUM       384
+
  #define PHBS_PER_CALGARY       4

  /* register offsets in Calgary's internal register space */

Re: [PATCH RESEND] (revised) Calgary: increase max PHB number

[Darrick J. Wong]

Newer systems (x3950M2) can have 48 PHBs per chassis and 8 chassis, so bump the
limits up and provide an explanation of the requirements for each class.

Signed-off-by: Darrick J. Wong <djwong@us.ibm.com>
Acked-by: Muli Ben-Yehuda <muli@il.ibm.com>

diff --git a/arch/x86/kernel/pci-calgary_64.c b/arch/x86/kernel/pci-calgary_64.c
index fb99f7e..967c646 100644
--- a/arch/x86/kernel/pci-calgary_64.c
+++ b/arch/x86/kernel/pci-calgary_64.c
@@ -103,11 +103,14 @@ int use_calgary __read_mostly = 0;
 #define PMR_SOFTSTOPFAULT	0x40000000
 #define PMR_HARDSTOP		0x20000000
 
-#define MAX_NUM_OF_PHBS		8 /* how many PHBs in total? */
-#define MAX_NUM_CHASSIS		8 /* max number of chassis */
-/* MAX_PHB_BUS_NUM is the maximal possible dev->bus->number */
-#define MAX_PHB_BUS_NUM		(MAX_NUM_OF_PHBS * MAX_NUM_CHASSIS * 2)
-#define PHBS_PER_CALGARY	4
+/*
+ * The maximum PHB bus number.
+ * x3950M2 (rare): 8 chassis, 48 PHBs per chassis = 384
+ * x3950M2: 4 chassis, 48 PHBs per chassis        = 192
+ * x3950 (PCIE): 8 chassis, 32 PHBs per chassis   = 256
+ * x3950 (PCIX): 8 chassis, 16 PHBs per chassis   = 128
+ */
+#define MAX_PHB_BUS_NUM		384
 
 /* register offsets in Calgary's internal register space */
 static const unsigned long tar_offsets[] = {

Re: [PATCH RESEND] (revised) Calgary: increase max PHB number

[Ingo Molnar]

* Darrick J. Wong <djwong@us.ibm.com> wrote:

> Newer systems (x3950M2) can have 48 PHBs per chassis and 8 chassis, so bump the
> limits up and provide an explanation of the requirements for each class.
> 
> Signed-off-by: Darrick J. Wong <djwong@us.ibm.com>
> Acked-by: Muli Ben-Yehuda <muli@il.ibm.com>
> 
> diff --git a/arch/x86/kernel/pci-calgary_64.c b/arch/x86/kernel/pci-calgary_64.c
> index fb99f7e..967c646 100644
> --- a/arch/x86/kernel/pci-calgary_64.c
> +++ b/arch/x86/kernel/pci-calgary_64.c
> @@ -103,11 +103,14 @@ int use_calgary __read_mostly = 0;
>  #define PMR_SOFTSTOPFAULT	0x40000000
>  #define PMR_HARDSTOP		0x20000000
>  
> -#define MAX_NUM_OF_PHBS		8 /* how many PHBs in total? */
> -#define MAX_NUM_CHASSIS		8 /* max number of chassis */
> -/* MAX_PHB_BUS_NUM is the maximal possible dev->bus->number */
> -#define MAX_PHB_BUS_NUM		(MAX_NUM_OF_PHBS * MAX_NUM_CHASSIS * 2)
> -#define PHBS_PER_CALGARY	4
> +/*
> + * The maximum PHB bus number.
> + * x3950M2 (rare): 8 chassis, 48 PHBs per chassis = 384
> + * x3950M2: 4 chassis, 48 PHBs per chassis        = 192
> + * x3950 (PCIE): 8 chassis, 32 PHBs per chassis   = 256
> + * x3950 (PCIX): 8 chassis, 16 PHBs per chassis   = 128
> + */
> +#define MAX_PHB_BUS_NUM		384

Doesnt build:

arch/x86/kernel/pci-calgary_64.c: In function ?calgary_locate_bbars?:
arch/x86/kernel/pci-calgary_64.c:1107: error: ?PHBS_PER_CALGARY? undeclared (first use in this function)
arch/x86/kernel/pci-calgary_64.c:1107: error: (Each undeclared identifier is reported only once
arch/x86/kernel/pci-calgary_64.c:1107: error: for each function it appears in.)

i have added back the PHBS_PER_CALGARY line - that is still needed, right?

Thanks,

	Ingo

[tip:x86/urgent] x86, Calgary: Increase max PHB number

[tip-bot for Darrick J. Wong]

Commit-ID:  499a00e92dd9a75395081f595e681629eb1eebad
Gitweb:     http://git.kernel.org/tip/499a00e92dd9a75395081f595e681629eb1eebad
Author:     Darrick J. Wong <djwong@us.ibm.com>
AuthorDate: Thu, 24 Jun 2010 14:26:47 -0700
Committer:  Ingo Molnar <mingo@elte.hu>
CommitDate: Fri, 25 Jun 2010 16:14:58 +0200

x86, Calgary: Increase max PHB number

Newer systems (x3950M2) can have 48 PHBs per chassis and 8
chassis, so bump the limits up and provide an explanation
of the requirements for each class.

Signed-off-by: Darrick J. Wong <djwong@us.ibm.com>
Acked-by: Muli Ben-Yehuda <muli@il.ibm.com>
Cc: Corinna Schultz <cschultz@linux.vnet.ibm.com>
Cc: <stable@kernel.org>
LKML-Reference: <20100624212647.GI15515@tux1.beaverton.ibm.com>
[ v2: Fixed build bug, added back PHBS_PER_CALGARY == 4 ]
Signed-off-by: Ingo Molnar <mingo@elte.hu>
---
 arch/x86/kernel/pci-calgary_64.c |   15 ++++++++++-----
 1 files changed, 10 insertions(+), 5 deletions(-)

diff --git a/arch/x86/kernel/pci-calgary_64.c b/arch/x86/kernel/pci-calgary_64.c
index fb99f7e..0b96b55 100644
--- a/arch/x86/kernel/pci-calgary_64.c
+++ b/arch/x86/kernel/pci-calgary_64.c
@@ -103,11 +103,16 @@ int use_calgary __read_mostly = 0;
 #define PMR_SOFTSTOPFAULT	0x40000000
 #define PMR_HARDSTOP		0x20000000
 
-#define MAX_NUM_OF_PHBS		8 /* how many PHBs in total? */
-#define MAX_NUM_CHASSIS		8 /* max number of chassis */
-/* MAX_PHB_BUS_NUM is the maximal possible dev->bus->number */
-#define MAX_PHB_BUS_NUM		(MAX_NUM_OF_PHBS * MAX_NUM_CHASSIS * 2)
-#define PHBS_PER_CALGARY	4
+/*
+ * The maximum PHB bus number.
+ * x3950M2 (rare): 8 chassis, 48 PHBs per chassis = 384
+ * x3950M2: 4 chassis, 48 PHBs per chassis        = 192
+ * x3950 (PCIE): 8 chassis, 32 PHBs per chassis   = 256
+ * x3950 (PCIX): 8 chassis, 16 PHBs per chassis   = 128
+ */
+#define MAX_PHB_BUS_NUM		384
+
+#define PHBS_PER_CALGARY	  4
 
 /* register offsets in Calgary's internal register space */
 static const unsigned long tar_offsets[] = {

Re: [tip:x86/urgent] x86, Calgary: Increase max PHB number

[Andrew Morton]

On Fri, 25 Jun 2010 15:29:50 GMT
"tip-bot for Darrick J. Wong" <djwong@us.ibm.com> wrote:

> Commit-ID:  499a00e92dd9a75395081f595e681629eb1eebad
> Gitweb:     http://git.kernel.org/tip/499a00e92dd9a75395081f595e681629eb1eebad
> Author:     Darrick J. Wong <djwong@us.ibm.com>
> AuthorDate: Thu, 24 Jun 2010 14:26:47 -0700
> Committer:  Ingo Molnar <mingo@elte.hu>
> CommitDate: Fri, 25 Jun 2010 16:14:58 +0200
> 
> x86, Calgary: Increase max PHB number

arch/x86/kernel/pci-calgary_64.c: In function 'calgary_init_one':
arch/x86/kernel/pci-calgary_64.c:1059: warning: comparison is always false due to limited range of data type

from

        BUG_ON(dev->bus->number >= MAX_PHB_BUS_NUM);

with

http://userweb.kernel.org/~akpm/stuff/config-akpm2.txt

Re: [tip:x86/urgent] x86, Calgary: Increase max PHB number

[H. Peter Anvin]

On 06/29/2010 03:51 PM, Andrew Morton wrote:
> On Fri, 25 Jun 2010 15:29:50 GMT
> "tip-bot for Darrick J. Wong" <djwong@us.ibm.com> wrote:
> 
>> Commit-ID:  499a00e92dd9a75395081f595e681629eb1eebad
>> Gitweb:     http://git.kernel.org/tip/499a00e92dd9a75395081f595e681629eb1eebad
>> Author:     Darrick J. Wong <djwong@us.ibm.com>
>> AuthorDate: Thu, 24 Jun 2010 14:26:47 -0700
>> Committer:  Ingo Molnar <mingo@elte.hu>
>> CommitDate: Fri, 25 Jun 2010 16:14:58 +0200
>>
>> x86, Calgary: Increase max PHB number
> 
> arch/x86/kernel/pci-calgary_64.c: In function 'calgary_init_one':
> arch/x86/kernel/pci-calgary_64.c:1059: warning: comparison is always false due to limited range of data type
> 
> from
> 
>         BUG_ON(dev->bus->number >= MAX_PHB_BUS_NUM);
> 
> with
> 
> http://userweb.kernel.org/~akpm/stuff/config-akpm2.txt

This comes from:

/*
 * The maximum PHB bus number.
 * x3950M2 (rare): 8 chassis, 48 PHBs per chassis = 384
 * x3950M2: 4 chassis, 48 PHBs per chassis        = 192
 * x3950 (PCIE): 8 chassis, 32 PHBs per chassis   = 256
 * x3950 (PCIX): 8 chassis, 16 PHBs per chassis   = 128
 */
#define MAX_PHB_BUS_NUM		384

Clearly there can't be 384 busses with standard PCI numbering (bus
numbers are 8 bits).  That means either that the number 384 is just
wrong, or it means that there are multiple PCI domains involved, and
that the BUG_ON() should be something else.

Furthermore, in get_tce_space_from_tar() we have:

	for (bus = 0; bus < MAX_PHB_BUS_NUM; bus++) {
		struct calgary_bus_info *info = &bus_info[bus];
		unsigned short pci_device;
		u32 val;

		val = read_pci_config(bus, 0, 0, 0);
		pci_device = (val & 0xFFFF0000) >> 16;

... which assumes the bus is a PCI bus number, no domain involved.

Does this mean the limit should be 256 (in which case we can just drop
the BUG_ON()), or is there support for domains which should be in this
code but isn't?

	-hpa

Re: [tip:x86/urgent] x86, Calgary: Increase max PHB number

[Darrick J. Wong]

On Wed, Jun 30, 2010 at 02:31:44PM -0700, H. Peter Anvin wrote:
> On 06/29/2010 03:51 PM, Andrew Morton wrote:
> > On Fri, 25 Jun 2010 15:29:50 GMT
> > "tip-bot for Darrick J. Wong" <djwong@us.ibm.com> wrote:
> > 
> >> Commit-ID:  499a00e92dd9a75395081f595e681629eb1eebad
> >> Gitweb:     http://git.kernel.org/tip/499a00e92dd9a75395081f595e681629eb1eebad
> >> Author:     Darrick J. Wong <djwong@us.ibm.com>
> >> AuthorDate: Thu, 24 Jun 2010 14:26:47 -0700
> >> Committer:  Ingo Molnar <mingo@elte.hu>
> >> CommitDate: Fri, 25 Jun 2010 16:14:58 +0200
> >>
> >> x86, Calgary: Increase max PHB number
> > 
> > arch/x86/kernel/pci-calgary_64.c: In function 'calgary_init_one':
> > arch/x86/kernel/pci-calgary_64.c:1059: warning: comparison is always false due to limited range of data type
> > 
> > from
> > 
> >         BUG_ON(dev->bus->number >= MAX_PHB_BUS_NUM);
> > 
> > with
> > 
> > http://userweb.kernel.org/~akpm/stuff/config-akpm2.txt
> 
> This comes from:
> 
> /*
>  * The maximum PHB bus number.
>  * x3950M2 (rare): 8 chassis, 48 PHBs per chassis = 384
>  * x3950M2: 4 chassis, 48 PHBs per chassis        = 192
>  * x3950 (PCIE): 8 chassis, 32 PHBs per chassis   = 256
>  * x3950 (PCIX): 8 chassis, 16 PHBs per chassis   = 128
>  */
> #define MAX_PHB_BUS_NUM		384
> 
> Clearly there can't be 384 busses with standard PCI numbering (bus
> numbers are 8 bits).  That means either that the number 384 is just
> wrong, or it means that there are multiple PCI domains involved, and
> that the BUG_ON() should be something else.
> 
> Furthermore, in get_tce_space_from_tar() we have:
> 
> 	for (bus = 0; bus < MAX_PHB_BUS_NUM; bus++) {
> 		struct calgary_bus_info *info = &bus_info[bus];
> 		unsigned short pci_device;
> 		u32 val;
> 
> 		val = read_pci_config(bus, 0, 0, 0);
> 		pci_device = (val & 0xFFFF0000) >> 16;
> 
> ... which assumes the bus is a PCI bus number, no domain involved.
> 
> Does this mean the limit should be 256 (in which case we can just drop
> the BUG_ON()), or is there support for domains which should be in this
> code but isn't?

Yes, it should be 256.  Some time ago there was talk of building 8-node
x3950M2s with PCI domain support, but the final word now is that nobody will
ever have more than 4 nodes.  The BUG_ON can be dropped, too.  I'll send a
patch shortly that incorporates both updates.

(Honestly I sort of wonder if the Calgary code should be removed entirely,
but that's a whole different battle....)

--D

[PATCH] x86, Calgary: Increase max PHB number

[Darrick J. Wong]

Newer systems (x3950M2) can have 48 PHBs per chassis and 4 chassis, so bump the
limits up and provide an explanation of the requirements for each class.  Since
we can't have more than 256 PCI buses in these systems, we don't need the array
check.

Signed-off-by: Darrick J. Wong <djwong@us.ibm.com>
---

 arch/x86/kernel/pci-calgary_64.c |   14 ++++++++------
 1 files changed, 8 insertions(+), 6 deletions(-)


diff --git a/arch/x86/kernel/pci-calgary_64.c b/arch/x86/kernel/pci-calgary_64.c
index fb99f7e..fd4e27b 100644
--- a/arch/x86/kernel/pci-calgary_64.c
+++ b/arch/x86/kernel/pci-calgary_64.c
@@ -103,10 +103,14 @@ int use_calgary __read_mostly = 0;
 #define PMR_SOFTSTOPFAULT	0x40000000
 #define PMR_HARDSTOP		0x20000000
 
-#define MAX_NUM_OF_PHBS		8 /* how many PHBs in total? */
-#define MAX_NUM_CHASSIS		8 /* max number of chassis */
-/* MAX_PHB_BUS_NUM is the maximal possible dev->bus->number */
-#define MAX_PHB_BUS_NUM		(MAX_NUM_OF_PHBS * MAX_NUM_CHASSIS * 2)
+/*
+ * The maximum PHB bus number.
+ * x3950M2: 4 chassis, 48 PHBs per chassis        = 192
+ * x3950 (PCIE): 8 chassis, 32 PHBs per chassis   = 256
+ * x3950 (PCIX): 8 chassis, 16 PHBs per chassis   = 128
+ */
+#define MAX_PHB_BUS_NUM		256
+
 #define PHBS_PER_CALGARY	4
 
 /* register offsets in Calgary's internal register space */
@@ -1051,8 +1055,6 @@ static int __init calgary_init_one(struct pci_dev *dev)
 	struct iommu_table *tbl;
 	int ret;
 
-	BUG_ON(dev->bus->number >= MAX_PHB_BUS_NUM);
-
 	bbar = busno_to_bbar(dev->bus->number);
 	ret = calgary_setup_tar(dev, bbar);
 	if (ret)

Re: [PATCH] x86, Calgary: Increase max PHB number

[H. Peter Anvin]

On 06/30/2010 02:49 PM, Darrick J. Wong wrote:
> Newer systems (x3950M2) can have 48 PHBs per chassis and 4 chassis, so bump the
> limits up and provide an explanation of the requirements for each class.  Since
> we can't have more than 256 PCI buses in these systems, we don't need the array
> check.

The 384-entry patch is already upstream.  Can you send a patch relative
to current -linus?

	-hpa

[PATCH] x86, Calgary: Increase max PHB number

[Darrick J. Wong]

The x3950 family can have as many as 256 PCI buses in a single system, so
change the limits to the maximum.  Since there can only be 256 PCI buses in one
domain, we no longer need the BUG_ON check.

Signed-off-by: Darrick J. Wong <djwong@us.ibm.com>
---

 arch/x86/kernel/pci-calgary_64.c |    4 +---
 1 files changed, 1 insertions(+), 3 deletions(-)


diff --git a/arch/x86/kernel/pci-calgary_64.c b/arch/x86/kernel/pci-calgary_64.c
index 0b96b55..078d4ec 100644
--- a/arch/x86/kernel/pci-calgary_64.c
+++ b/arch/x86/kernel/pci-calgary_64.c
@@ -110,7 +110,7 @@ int use_calgary __read_mostly = 0;
  * x3950 (PCIE): 8 chassis, 32 PHBs per chassis   = 256
  * x3950 (PCIX): 8 chassis, 16 PHBs per chassis   = 128
  */
-#define MAX_PHB_BUS_NUM		384
+#define MAX_PHB_BUS_NUM		256
 
 #define PHBS_PER_CALGARY	  4
 
@@ -1056,8 +1056,6 @@ static int __init calgary_init_one(struct pci_dev *dev)
 	struct iommu_table *tbl;
 	int ret;
 
-	BUG_ON(dev->bus->number >= MAX_PHB_BUS_NUM);
-
 	bbar = busno_to_bbar(dev->bus->number);
 	ret = calgary_setup_tar(dev, bbar);
 	if (ret)

[tip:x86/urgent] x86, Calgary: Limit the max PHB number to 256

[tip-bot for Darrick J. Wong]

Commit-ID:  d596043d71ff0d7b3d0bead19b1d68c55f003093
Gitweb:     http://git.kernel.org/tip/d596043d71ff0d7b3d0bead19b1d68c55f003093
Author:     Darrick J. Wong <djwong@us.ibm.com>
AuthorDate: Wed, 30 Jun 2010 17:45:19 -0700
Committer:  H. Peter Anvin <hpa@zytor.com>
CommitDate: Wed, 30 Jun 2010 22:41:42 -0700

x86, Calgary: Limit the max PHB number to 256

The x3950 family can have as many as 256 PCI buses in a single system, so
change the limits to the maximum.  Since there can only be 256 PCI buses in one
domain, we no longer need the BUG_ON check.

Signed-off-by: Darrick J. Wong <djwong@us.ibm.com>
LKML-Reference: <20100701004519.GQ15515@tux1.beaverton.ibm.com>
Signed-off-by: H. Peter Anvin <hpa@zytor.com>
---
 arch/x86/kernel/pci-calgary_64.c |    4 +---
 1 files changed, 1 insertions(+), 3 deletions(-)

diff --git a/arch/x86/kernel/pci-calgary_64.c b/arch/x86/kernel/pci-calgary_64.c
index 0b96b55..078d4ec 100644
--- a/arch/x86/kernel/pci-calgary_64.c
+++ b/arch/x86/kernel/pci-calgary_64.c
@@ -110,7 +110,7 @@ int use_calgary __read_mostly = 0;
  * x3950 (PCIE): 8 chassis, 32 PHBs per chassis   = 256
  * x3950 (PCIX): 8 chassis, 16 PHBs per chassis   = 128
  */
-#define MAX_PHB_BUS_NUM		384
+#define MAX_PHB_BUS_NUM		256
 
 #define PHBS_PER_CALGARY	  4
 
@@ -1056,8 +1056,6 @@ static int __init calgary_init_one(struct pci_dev *dev)
 	struct iommu_table *tbl;
 	int ret;
 
-	BUG_ON(dev->bus->number >= MAX_PHB_BUS_NUM);
-
 	bbar = busno_to_bbar(dev->bus->number);
 	ret = calgary_setup_tar(dev, bbar);
 	if (ret)