[PATCH 1/3] IB/ehca: Replace vmalloc with kmalloc

[PATCH 1/3] IB/ehca: Replace vmalloc with kmalloc

[Stefan Roscher]

From: Anton Blanchard <antonb at au1.ibm.com>

To improve performance of driver ressource allocation,
replace the vmalloc() call with kmalloc().

Signed-off-by: Stefan Roscher <stefan.roscher at de.ibm.com>
---
 drivers/infiniband/hw/ehca/ipz_pt_fn.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/infiniband/hw/ehca/ipz_pt_fn.c b/drivers/infiniband/hw/ehca/ipz_pt_fn.c
index c3a3284..a260559 100644
--- a/drivers/infiniband/hw/ehca/ipz_pt_fn.c
+++ b/drivers/infiniband/hw/ehca/ipz_pt_fn.c
@@ -220,7 +220,7 @@ int ipz_queue_ctor(struct ehca_pd *pd, struct ipz_queue *queue,
 	queue->small_page = NULL;
 
 	/* allocate queue page pointers */
-	queue->queue_pages = vmalloc(nr_of_pages * sizeof(void *));
+	queue->queue_pages = kmalloc(nr_of_pages * sizeof(void *), GFP_KERNEL);
 	if (!queue->queue_pages) {
 		ehca_gen_err("Couldn't allocate queue page list");
 		return 0;
@@ -240,7 +240,7 @@ int ipz_queue_ctor(struct ehca_pd *pd, struct ipz_queue *queue,
 ipz_queue_ctor_exit0:
 	ehca_gen_err("Couldn't alloc pages queue=%p "
 		 "nr_of_pages=%x",  queue, nr_of_pages);
-	vfree(queue->queue_pages);
+	kfree(queue->queue_pages);
 
 	return 0;
 }
@@ -262,7 +262,7 @@ int ipz_queue_dtor(struct ehca_pd *pd, struct ipz_queue *queue)
 			free_page((unsigned long)queue->queue_pages[i]);
 	}
 
-	vfree(queue->queue_pages);
+	kfree(queue->queue_pages);
 
 	return 1;
 }
-- 
1.5.5

Re: [PATCH 1/3] IB/ehca: Replace vmalloc with kmalloc

[Roland Dreier]

 > +	queue->queue_pages = kmalloc(nr_of_pages * sizeof(void *), GFP_KERNEL);

How big might this buffer be?  Any chance of allocation failure due to
memory fragmentation?

 - R.

Re: [PATCH 1/3] IB/ehca: Replace vmalloc with kmalloc

[Stefan Roscher]

[-- Attachment #1: Type: text/plain, Size: 2137 bytes --]

Hi Roland, 
thanks for the quick review. I was hoping you could apply these changes 
for 2.6.30 because this will be the codebase for the next OFED release.
The patch is well tested in HPC environment and we haven't seen any 
issues.
Regarding Antons patch you are right. If a user allocates an 
unrealistically large queue pair it could happen that kmalloc() is not 
able to allocate the memory. In this case we will return ENOMEM to the 
user so the kernel will not be affected at all. We plan to add vmalloc() 
call in case kmalloc() fails for the next kernel release.
 
Mit freundlichen Grüßen / Kind regards
 
Stefan Roscher
 
eHCA/eHEA Linux Driver Development
IBM Systems &Technology Group, Systems Software Development / FW I/O 
Firmware Entwicklung 2
-------------------------------------------------------------------------------------------------------------------------------------------
IBM Deutschland
Schoenaicher Str. 220
71032 Boeblingen
Phone: +49-7031-16-2015
E-Mail: stefan.roscher@de.ibm.com
-------------------------------------------------------------------------------------------------------------------------------------------
IBM Deutschland Research & Development GmbH / Vorsitzender des 
Aufsichtsrats: Martin Jetter
Geschäftsführung: Herbert Kircher
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, 
HRB 243294



From:
Roland Dreier <rdreier@cisco.com>
To:
Stefan Roscher <ossrosch@linux.vnet.ibm.com>
Cc:
"LinuxPPC-Dev" <linuxppc-dev@ozlabs.org>, LKML 
<linux-kernel@vger.kernel.org>, "OF-EWG" <ewg@lists.openfabrics.org>, 
Roland Dreier <rolandd@cisco.com>, Joachim Fenkes/Germany/IBM@IBMDE, 
Christoph Raisch/Germany/IBM@IBMDE, Alexander Schmidt1/Germany/IBM@IBMDE, 
Stefan Roscher/Germany/IBM@IBMDE, Hoang-Nam Nguyen/Germany/IBM@IBMDE
Date:
21.04.2009 19:34
Subject:
Re: [PATCH 1/3] IB/ehca: Replace vmalloc with kmalloc



 > +             queue->queue_pages = kmalloc(nr_of_pages * sizeof(void 
*), GFP_KERNEL);

How big might this buffer be?  Any chance of allocation failure due to
memory fragmentation?

 - R.



[-- Attachment #2: Type: text/html, Size: 3277 bytes --]

Re: [PATCH 1/3] IB/ehca: Replace vmalloc with kmalloc

[Stefan Roscher]

In case of large queue pairs there is the possibillity of allocation failures 
due to memory fragmentationo with kmalloc().To ensure the memory is allocated even
if kmalloc() can not find chunks which are big enough, we try to allocate the memory
with vmalloc().

Signed-off-by: Stefan Roscher <stefan.roscher@de.ibm.com>
---

On Tuesday 21 April 2009 07:34:30 pm Roland Dreier wrote:
>  > +	queue->queue_pages = kmalloc(nr_of_pages * sizeof(void *), GFP_KERNEL);
> 
> How big might this buffer be?  Any chance of allocation failure due to
> memory fragmentation?
> 
>  - R.
Hey Roland, 
yes you are right and here is the patch to circumvent the described problem.
It will apply on top of the patchset.
regards Stefan


 
 drivers/infiniband/hw/ehca/ipz_pt_fn.c |   17 +++++++++++++----
 1 files changed, 13 insertions(+), 4 deletions(-)

diff --git a/drivers/infiniband/hw/ehca/ipz_pt_fn.c b/drivers/infiniband/hw/ehca/ipz_pt_fn.c
index a260559..1227c59 100644
--- a/drivers/infiniband/hw/ehca/ipz_pt_fn.c
+++ b/drivers/infiniband/hw/ehca/ipz_pt_fn.c
@@ -222,8 +222,11 @@ int ipz_queue_ctor(struct ehca_pd *pd, struct ipz_queue *queue,
 	/* allocate queue page pointers */
 	queue->queue_pages = kmalloc(nr_of_pages * sizeof(void *), GFP_KERNEL);
 	if (!queue->queue_pages) {
-		ehca_gen_err("Couldn't allocate queue page list");
-		return 0;
+		queue->queue_pages = vmalloc(nr_of_pages * sizeof(void *));
+		if (!queue->queue_pages) {
+			ehca_gen_err("Couldn't allocate queue page list");
+			return 0;
+		}
 	}
 	memset(queue->queue_pages, 0, nr_of_pages * sizeof(void *));
 
@@ -240,7 +243,10 @@ int ipz_queue_ctor(struct ehca_pd *pd, struct ipz_queue *queue,
 ipz_queue_ctor_exit0:
 	ehca_gen_err("Couldn't alloc pages queue=%p "
 		 "nr_of_pages=%x",  queue, nr_of_pages);
-	kfree(queue->queue_pages);
+	if (is_vmalloc_addr(queue->queue_pages))
+		vfree(queue->queue_pages);
+	else
+		kfree(queue->queue_pages);
 
 	return 0;
 }
@@ -262,7 +268,10 @@ int ipz_queue_dtor(struct ehca_pd *pd, struct ipz_queue *queue)
 			free_page((unsigned long)queue->queue_pages[i]);
 	}
 
-	kfree(queue->queue_pages);
+	if (is_vmalloc_addr(queue->queue_pages))
+		vfree(queue->queue_pages);
+	else
+		kfree(queue->queue_pages);
 
 	return 1;
 }
-- 
1.5.5

Re: [PATCH 1/3] IB/ehca: Replace vmalloc with kmalloc

[michael]

Hi,

Stefan Roscher wrote:
> In case of large queue pairs there is the possibillity of allocation failures 
> due to memory fragmentationo with kmalloc().To ensure the memory is allocated even
> if kmalloc() can not find chunks which are big enough, we try to allocate the memory
> with vmalloc().
>
> Signed-off-by: Stefan Roscher <stefan.roscher@de.ibm.com>
> ---
>
> On Tuesday 21 April 2009 07:34:30 pm Roland Dreier wrote:
>   
>>  > +	queue->queue_pages = kmalloc(nr_of_pages * sizeof(void *), GFP_KERNEL);
>>
>> How big might this buffer be?  Any chance of allocation failure due to
>> memory fragmentation?
>>
>>  - R.
>>     
> Hey Roland, 
> yes you are right and here is the patch to circumvent the described problem.
> It will apply on top of the patchset.
> regards Stefan
>
>   
I don't take the point, if it is not import use the vmalloc. Why you try 
with a kmalloc
alloc first? and why do not use kzalloc?
>  
>  drivers/infiniband/hw/ehca/ipz_pt_fn.c |   17 +++++++++++++----
>  1 files changed, 13 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/infiniband/hw/ehca/ipz_pt_fn.c b/drivers/infiniband/hw/ehca/ipz_pt_fn.c
> index a260559..1227c59 100644
> --- a/drivers/infiniband/hw/ehca/ipz_pt_fn.c
> +++ b/drivers/infiniband/hw/ehca/ipz_pt_fn.c
> @@ -222,8 +222,11 @@ int ipz_queue_ctor(struct ehca_pd *pd, struct ipz_queue *queue,
>  	/* allocate queue page pointers */
>  	queue->queue_pages = kmalloc(nr_of_pages * sizeof(void *), GFP_KERNEL);
>  	if (!queue->queue_pages) {
> -		ehca_gen_err("Couldn't allocate queue page list");
> -		return 0;
> +		queue->queue_pages = vmalloc(nr_of_pages * sizeof(void *));
> +		if (!queue->queue_pages) {
> +			ehca_gen_err("Couldn't allocate queue page list");
> +			return 0;
> +		}
>  	}
>  	memset(queue->queue_pages, 0, nr_of_pages * sizeof(void *));
>  
> @@ -240,7 +243,10 @@ int ipz_queue_ctor(struct ehca_pd *pd, struct ipz_queue *queue,
>  ipz_queue_ctor_exit0:
>  	ehca_gen_err("Couldn't alloc pages queue=%p "
>  		 "nr_of_pages=%x",  queue, nr_of_pages);
> -	kfree(queue->queue_pages);
> +	if (is_vmalloc_addr(queue->queue_pages))
> +		vfree(queue->queue_pages);
> +	else
> +		kfree(queue->queue_pages);
>  
>  	return 0;
>  }
> @@ -262,7 +268,10 @@ int ipz_queue_dtor(struct ehca_pd *pd, struct ipz_queue *queue)
>  			free_page((unsigned long)queue->queue_pages[i]);
>  	}
>  
> -	kfree(queue->queue_pages);
> +	if (is_vmalloc_addr(queue->queue_pages))
> +		vfree(queue->queue_pages);
> +	else
> +		kfree(queue->queue_pages);
>  
>  	return 1;
>  }
>   

Regards Michael

Re: [PATCH 1/3] IB/ehca: Replace vmalloc with kmalloc

[Stefan Roscher]

On Wednesday 22 April 2009 04:10:18 pm michael wrote:
> Hi,
> 

> I don't take the point, if it is not import use the vmalloc. Why you try 
> with a kmalloc
> alloc first? and why do not use kzalloc?

Because kmalloc() is faster than vmalloc() causing a huge performance win
when someone allocates a large number of queue pairs. We fall back to
vmalloc() only if kmalloc() can't deliver the memory chunk.
We don't need kzalloc because we fill the list right after the alloc.

regards Stefan

Re: [PATCH 1/3] IB/ehca: Replace vmalloc with kmalloc

[michael]

Hi,

Stefan Roscher wrote:
> On Wednesday 22 April 2009 04:10:18 pm michael wrote:
>   
>> Hi,
>>
>>     
>
>   
>> I don't take the point, if it is not import use the vmalloc. Why you try 
>> with a kmalloc
>> alloc first? and why do not use kzalloc?
>>     
>
> Because kmalloc() is faster than vmalloc() causing a huge performance win
> when someone allocates a large number of queue pairs. We fall back to
> vmalloc() only if kmalloc() can't deliver the memory chunk.
>   
Sorry I catch later the performace issue.
> We don't need kzalloc because we fill the list right after the alloc.
>
> regards Stefan
>   
Regards Michael
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-dev
>
>   

Re: [ewg] Re: [PATCH 1/3] IB/ehca: Replace vmalloc with kmalloc

[Alexander Schmidt]

Hi Roland,

did you have a chance to take a look at the patchset and will you apply it, or
are there any outstanding issues we need to address?

Regards,
Alex

On Wed, 22 Apr 2009 16:02:28 +0200
Stefan Roscher <ossrosch@linux.vnet.ibm.com> wrote:

> In case of large queue pairs there is the possibillity of allocation failures 
> due to memory fragmentationo with kmalloc().To ensure the memory is allocated even
> if kmalloc() can not find chunks which are big enough, we try to allocate the memory
> with vmalloc().
> 
> Signed-off-by: Stefan Roscher <stefan.roscher@de.ibm.com>
> ---
> 
> On Tuesday 21 April 2009 07:34:30 pm Roland Dreier wrote:
> >  > +	queue->queue_pages = kmalloc(nr_of_pages * sizeof(void *), GFP_KERNEL);
> > 
> > How big might this buffer be?  Any chance of allocation failure due to
> > memory fragmentation?
> > 
> >  - R.
> Hey Roland, 
> yes you are right and here is the patch to circumvent the described problem.
> It will apply on top of the patchset.
> regards Stefan
> 
> 
> 
>  drivers/infiniband/hw/ehca/ipz_pt_fn.c |   17 +++++++++++++----
>  1 files changed, 13 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/infiniband/hw/ehca/ipz_pt_fn.c b/drivers/infiniband/hw/ehca/ipz_pt_fn.c
> index a260559..1227c59 100644
> --- a/drivers/infiniband/hw/ehca/ipz_pt_fn.c
> +++ b/drivers/infiniband/hw/ehca/ipz_pt_fn.c
> @@ -222,8 +222,11 @@ int ipz_queue_ctor(struct ehca_pd *pd, struct ipz_queue *queue,
>  	/* allocate queue page pointers */
>  	queue->queue_pages = kmalloc(nr_of_pages * sizeof(void *), GFP_KERNEL);
>  	if (!queue->queue_pages) {
> -		ehca_gen_err("Couldn't allocate queue page list");
> -		return 0;
> +		queue->queue_pages = vmalloc(nr_of_pages * sizeof(void *));
> +		if (!queue->queue_pages) {
> +			ehca_gen_err("Couldn't allocate queue page list");
> +			return 0;
> +		}
>  	}
>  	memset(queue->queue_pages, 0, nr_of_pages * sizeof(void *));
> 
> @@ -240,7 +243,10 @@ int ipz_queue_ctor(struct ehca_pd *pd, struct ipz_queue *queue,
>  ipz_queue_ctor_exit0:
>  	ehca_gen_err("Couldn't alloc pages queue=%p "
>  		 "nr_of_pages=%x",  queue, nr_of_pages);
> -	kfree(queue->queue_pages);
> +	if (is_vmalloc_addr(queue->queue_pages))
> +		vfree(queue->queue_pages);
> +	else
> +		kfree(queue->queue_pages);
> 
>  	return 0;
>  }
> @@ -262,7 +268,10 @@ int ipz_queue_dtor(struct ehca_pd *pd, struct ipz_queue *queue)
>  			free_page((unsigned long)queue->queue_pages[i]);
>  	}
> 
> -	kfree(queue->queue_pages);
> +	if (is_vmalloc_addr(queue->queue_pages))
> +		vfree(queue->queue_pages);
> +	else
> +		kfree(queue->queue_pages);
> 
>  	return 1;
>  }

Re: [ewg] Re: [PATCH 1/3] IB/ehca: Replace vmalloc with kmalloc

[Roland Dreier]

 > did you have a chance to take a look at the patchset and will you apply it, or
 > are there any outstanding issues we need to address?

I guess it's OK, but definitely 2.6.31 material.  I guess I'll stick it
linux-next soon.

 - R.

Re: [ewg] Re: [PATCH 1/3] IB/ehca: Replace vmalloc with kmalloc

[Alexander Schmidt]

On Tue, 28 Apr 2009 07:01:32 -0700
Roland Dreier <rdreier@cisco.com> wrote:

>  > did you have a chance to take a look at the patchset and will you apply it, or
>  > are there any outstanding issues we need to address?
> 
> I guess it's OK, but definitely 2.6.31 material.  I guess I'll stick it
> linux-next soon.
> 
>  - R.

Okay with us, thank you very much!

Alex

Re: [PATCH 1/3] IB/ehca: Replace vmalloc with kmalloc

[Dave Hansen]

On Tue, 2009-04-21 at 17:16 +0200, Stefan Roscher wrote:
> From: Anton Blanchard <antonb at au1.ibm.com>
> 
> To improve performance of driver ressource allocation,
> replace the vmalloc() call with kmalloc().

Just curious, but how big are these allocations?  Why was vmalloc() even
ever used if we know they'll be small?

-- Dave

Re: [PATCH 1/3] IB/ehca: Replace vmalloc with kmalloc

[Stefan Roscher]

On Tuesday 28 April 2009 05:12:51 pm Dave Hansen wrote:
> On Tue, 2009-04-21 at 17:16 +0200, Stefan Roscher wrote:
> > From: Anton Blanchard <antonb at au1.ibm.com>
> > 
> > To improve performance of driver ressource allocation,
> > replace the vmalloc() call with kmalloc().
> 
> Just curious, but how big are these allocations?  Why was vmalloc() even
> ever used if we know they'll be small?
> 
> -- Dave
> 
> 

The theoretical maximum size can be 512k, but for common queue pairs 
less than 128k is used.Because of the theoretical maximum we implemented
vmalloc() first, but recognized a huge performance impact.

-- Stefan 

Re: [PATCH 1/3] IB/ehca: Replace vmalloc with kmalloc

[Roland Dreier]

thanks, applied.

 > From: Anton Blanchard <antonb at au1.ibm.com>
 > Signed-off-by: Stefan Roscher <stefan.roscher at de.ibm.com>

please use '@' signs so these are real email addresses.

 - R.