[PATCH] powerpc: align DTL buffer to AMS boundary

[PATCH] powerpc: align DTL buffer to AMS boundary

[Nishanth Aravamudan]

PAPR specifies that DTL buffers can not cross AMS environments (aka CMO
in the PAPR) and can not cross a memory entitlement granule boundary
(4k). This is found in section 14.11.3.2 H_REGISTER_VPA of the PAPR.
kmalloc does not guarantee an alignment of the allocation, though,
beyond 8 bytes (at least in my understanding). Over-allocate and align
the resulting address. Tested both with and without AMS on a p7
partition.

Signed-off-by: Nishanth Aravamudan <nacc@us.ibm.com>

---
Note, I initially put this in a firmware check if-block, but we have
also seen some issues with alignment with non-AMS partitions. The wasted
memory is unfortunate, though.

Cc: Paul Mackerras <paulus@samba.org>
Cc: Anton Blanchard <anton@samba.org>
Cc: linuxppc-dev@ozlabs.org

 arch/powerpc/platforms/pseries/setup.c |    6 ++++--
 1 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/arch/powerpc/platforms/pseries/setup.c b/arch/powerpc/platforms/pseries/setup.c
index 0007241..7df5ddb 100644
--- a/arch/powerpc/platforms/pseries/setup.c
+++ b/arch/powerpc/platforms/pseries/setup.c
@@ -39,6 +39,7 @@
 #include <linux/irq.h>
 #include <linux/seq_file.h>
 #include <linux/root_dev.h>
+#include <linux/memblock.h>
 
 #include <asm/mmu.h>
 #include <asm/processor.h>
@@ -293,14 +294,15 @@ static int alloc_dispatch_logs(void)
 
 	for_each_possible_cpu(cpu) {
 		pp = &paca[cpu];
-		dtl = kmalloc_node(DISPATCH_LOG_BYTES, GFP_KERNEL,
-				   cpu_to_node(cpu));
+		dtl = kmalloc_node(DISPATCH_LOG_BYTES + DISPATCH_LOG_BYTES - 1,
+				   GFP_KERNEL, cpu_to_node(cpu));
 		if (!dtl) {
 			pr_warn("Failed to allocate dispatch trace log for cpu %d\n",
 				cpu);
 			pr_warn("Stolen time statistics will be unreliable\n");
 			break;
 		}
+		dtl = PTR_ALIGN(dtl, DISPATCH_LOG_BYTES);
 
 		pp->dtl_ridx = 0;
 		pp->dispatch_log = dtl;
-- 
1.7.4.1

[PATCH v2] powerpc: align DTL buffer to AMS boundary

[Nishanth Aravamudan]

PAPR specifies that DTL buffers can not cross AMS environments (aka CMO
in the PAPR) and can not cross a memory entitlement granule boundary
(4k). This is found in section 14.11.3.2 H_REGISTER_VPA of the PAPR.
kmalloc does not guarantee an alignment of the allocation, though,
beyond 8 bytes (at least in my understanding). Over-allocate and align
the resulting address. Tested both with and without AMS on a p7
partition.

---
Change from v1: removed extraneous #include.

Note, I initially put this in a firmware check if-block, but we have
also seen some issues with alignment with non-AMS partitions. The wasted
memory is unfortunate, though.

Cc: Paul Mackerras <paulus@samba.org>
Cc: Anton Blanchard <anton@samba.org>
Cc: linuxppc-dev@lists.ozlabs.org

 arch/powerpc/platforms/pseries/setup.c |    5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/arch/powerpc/platforms/pseries/setup.c b/arch/powerpc/platforms/pseries/setup.c
index 0007241..a67be48 100644
--- a/arch/powerpc/platforms/pseries/setup.c
+++ b/arch/powerpc/platforms/pseries/setup.c
@@ -293,14 +293,15 @@ static int alloc_dispatch_logs(void)
 
 	for_each_possible_cpu(cpu) {
 		pp = &paca[cpu];
-		dtl = kmalloc_node(DISPATCH_LOG_BYTES, GFP_KERNEL,
-				   cpu_to_node(cpu));
+		dtl = kmalloc_node(DISPATCH_LOG_BYTES + DISPATCH_LOG_BYTES - 1,
+				   GFP_KERNEL, cpu_to_node(cpu));
 		if (!dtl) {
 			pr_warn("Failed to allocate dispatch trace log for cpu %d\n",
 				cpu);
 			pr_warn("Stolen time statistics will be unreliable\n");
 			break;
 		}
+		dtl = PTR_ALIGN(dtl, DISPATCH_LOG_BYTES);
 
 		pp->dtl_ridx = 0;
 		pp->dispatch_log = dtl;
-- 
1.7.4.1

Re: [PATCH v2] powerpc: align DTL buffer to AMS boundary

[Michael Ellerman]

[-- Attachment #1: Type: text/plain, Size: 678 bytes --]

On Wed, 2011-04-13 at 15:33 -0700, Nishanth Aravamudan wrote:
> PAPR specifies that DTL buffers can not cross AMS environments (aka CMO
> in the PAPR) and can not cross a memory entitlement granule boundary
> (4k). This is found in section 14.11.3.2 H_REGISTER_VPA of the PAPR.
> kmalloc does not guarantee an alignment of the allocation, though,
> beyond 8 bytes (at least in my understanding). Over-allocate and align
> the resulting address.

You can specify alignment by creating your own kmem_cache, ie.
kmem_cache_create().

Obviously there will be some overhead to create the cache structure, but
I'd think it will be less than 4k * NR_CPUs.

cheers




[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[PATCH v3] powerpc: use a kmem cache for DTL buffers

[Nishanth Aravamudan]

PAPR specifies that DTL buffers can not cross AMS environments (aka CMO
in the PAPR) and can not cross a memory entitlement granule boundary
(4k). This is found in section 14.11.3.2 H_REGISTER_VPA of the PAPR.
kmalloc does not guarantee an alignment of the allocation, though,
beyond 8 bytes (at least in my understanding). Create a special kmem
cache for DTL buffers with the alignment requirement.

---
Change from v1: removed extraneous #include.
Change from v2: instead of overallocating and aligning, use a kmem cache

Note, I initially put this in a firmware check if-block, but we have
also seen some issues with alignment with non-AMS partitions. The wasted
memory is unfortunate, though.

Signed-off-by: Nishanth Aravamudan <nacc@us.ibm.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Ben Herrenschmidt <benh@kernel.crashing.org>
Cc: Anton Blanchard <anton@samba.org>
Cc: linuxppc-dev@lists.ozlabs.org

 arch/powerpc/platforms/pseries/setup.c |    7 +++++--
 1 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/arch/powerpc/platforms/pseries/setup.c b/arch/powerpc/platforms/pseries/setup.c
index 0007241..629b703 100644
--- a/arch/powerpc/platforms/pseries/setup.c
+++ b/arch/powerpc/platforms/pseries/setup.c
@@ -287,14 +287,17 @@ static int alloc_dispatch_logs(void)
 	int cpu, ret;
 	struct paca_struct *pp;
 	struct dtl_entry *dtl;
+	struct kmem_cache *dtl_cache;
 
 	if (!firmware_has_feature(FW_FEATURE_SPLPAR))
 		return 0;
 
+	dtl_cache = kmem_cache_create("dtl", DISPATCH_LOG_BYTES,
+						DISPATCH_LOG_BYTES, 0, NULL); 
+
 	for_each_possible_cpu(cpu) {
 		pp = &paca[cpu];
-		dtl = kmalloc_node(DISPATCH_LOG_BYTES, GFP_KERNEL,
-				   cpu_to_node(cpu));
+		dtl = kmem_cache_alloc(dtl_cache, GFP_KERNEL);
 		if (!dtl) {
 			pr_warn("Failed to allocate dispatch trace log for cpu %d\n",
 				cpu);
-- 
1.7.4.1

[PATCH v4] powerpc: use a kmem cache for DTL buffers

[Nishanth Aravamudan]

PAPR specifies that DTL buffers can not cross AMS environments (aka CMO
in the PAPR) and can not cross a memory entitlement granule boundary
(4k). This is found in section 14.11.3.2 H_REGISTER_VPA of the PAPR.
kmalloc does not guarantee an alignment of the allocation, though,
beyond 8 bytes (at least in my understanding). Create a special kmem
cache for DTL buffers with the alignment requirement.

---
Change from v1: removed extraneous #include.
Change from v2: instead of overallocating and aligning, use a kmem cache
Change from v3: handle cache creation failure

Note, I initially put this in a firmware check if-block, but we have
also seen some issues with alignment with non-AMS partitions. The wasted
memory is unfortunate, though.

Signed-off-by: Nishanth Aravamudan <nacc@us.ibm.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Ben Herrenschmidt <benh@kernel.crashing.org>
Cc: Anton Blanchard <anton@samba.org>
Cc: linuxppc-dev@lists.ozlabs.org

 arch/powerpc/platforms/pseries/setup.c |   12 ++++++++++--
 1 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/arch/powerpc/platforms/pseries/setup.c b/arch/powerpc/platforms/pseries/setup.c
index 0007241..9e20b83 100644
--- a/arch/powerpc/platforms/pseries/setup.c
+++ b/arch/powerpc/platforms/pseries/setup.c
@@ -287,14 +287,22 @@ static int alloc_dispatch_logs(void)
 	int cpu, ret;
 	struct paca_struct *pp;
 	struct dtl_entry *dtl;
+	struct kmem_cache *dtl_cache;
 
 	if (!firmware_has_feature(FW_FEATURE_SPLPAR))
 		return 0;
 
+	dtl_cache = kmem_cache_create("dtl", DISPATCH_LOG_BYTES,
+						DISPATCH_LOG_BYTES, 0, NULL); 
+	if (!dtl_cache) {
+		pr_warn("Failed to create dispatch trace log buffer cache\n");
+		pr_warn("Stolen time statistics will be unreliable\n");
+		return 0;
+	}
+
 	for_each_possible_cpu(cpu) {
 		pp = &paca[cpu];
-		dtl = kmalloc_node(DISPATCH_LOG_BYTES, GFP_KERNEL,
-				   cpu_to_node(cpu));
+		dtl = kmem_cache_alloc(dtl_cache, GFP_KERNEL);
 		if (!dtl) {
 			pr_warn("Failed to allocate dispatch trace log for cpu %d\n",
 				cpu);
-- 
1.7.4.1

Re: [PATCH] powerpc: align DTL buffer to AMS boundary

[David Laight]

> From:=20
> linuxppc-dev-bounces+david.laight=3Daculab.com@lists.ozlabs.org=20
> [mailto:linuxppc-dev-bounces+david.laight=3Daculab.com@lists.ozl
> abs.org] On Behalf Of Nishanth Aravamudan
> Sent: 13 April 2011 15:53
> To: Ben Herrenschmidt
> Cc: linuxppc-dev@ozlabs.org; Paul Mackerras; Anton Blanchard
> Subject: [PATCH] powerpc: align DTL buffer to AMS boundary
>=20
> PAPR specifies that DTL buffers can not cross AMS environments (aka
CMO
> in the PAPR) and can not cross a memory entitlement granule boundary
> (4k)....

How big is the buffer being allocated?
If it is much less than 4k then it might be worth allocating
a buffer of the correct size, and only if that crosses a 4k boundary
allocate the larger buffer.

Also, if the buffer is ever freed, the actual base address is needed
for the free.

	David

Re: [PATCH] powerpc: align DTL buffer to AMS boundary

[Nishanth Aravamudan]

On 13.04.2011 [15:59:44 +0100], David Laight wrote:
> > From: 
> > linuxppc-dev-bounces+david.laight=aculab.com@lists.ozlabs.org 
> > [mailto:linuxppc-dev-bounces+david.laight=aculab.com@lists.ozl
> > abs.org] On Behalf Of Nishanth Aravamudan
> > Sent: 13 April 2011 15:53
> > To: Ben Herrenschmidt
> > Cc: linuxppc-dev@ozlabs.org; Paul Mackerras; Anton Blanchard
> > Subject: [PATCH] powerpc: align DTL buffer to AMS boundary
> > 
> > PAPR specifies that DTL buffers can not cross AMS environments (aka
> CMO
> > in the PAPR) and can not cross a memory entitlement granule boundary
> > (4k)....
> 
> How big is the buffer being allocated?
> If it is much less than 4k then it might be worth allocating
> a buffer of the correct size, and only if that crosses a 4k boundary
> allocate the larger buffer.

Currently, DISPATCH_LOG_BYTES is 4k (which I assume was actually in
deference to this requirement).

> Also, if the buffer is ever freed, the actual base address is needed
> for the free.

I didn't see any free patch for this sequence of code -- which make
sense as this allocation occurs under a for_each_possible_cpu loop.

Thanks for the feedback,
Nish