[PATCH] rtlwifi: Fix panic due to memory allocation failure

[PATCH] rtlwifi: Fix panic due to memory allocation failure

[Larry Finger]

The PCI routine of this driver is allocating receive buffers of order 2,
which causes an unnecessary fragmentation of memory. To make matters
worse, there are locations that fail to check for allocation failures,
or return success when the allocation actually failed. Kernel panics
result.

Signed-off-by: Larry Finger <Larry.Finger@lwfinger.net>
Cc: Stable <stable@vger.kernel.org>              [2.6.37 and 2.6.38]
---

John,

This is 2.6.39 material. I hope we make the cutoff.

Larry
---

Index: wireless-testing-new/drivers/net/wireless/rtlwifi/pci.c
===================================================================
--- wireless-testing-new.orig/drivers/net/wireless/rtlwifi/pci.c
+++ wireless-testing-new/drivers/net/wireless/rtlwifi/pci.c
@@ -732,10 +732,12 @@ static void _rtl_pci_rx_interrupt(struct
 					    false))) {
 						dev_kfree_skb_any(skb);
 					} else {
-						struct sk_buff *uskb = NULL;
+						struct sk_buff *uskb;
 						u8 *pdata;
 						uskb = dev_alloc_skb(skb->len
 								     + 128);
+						if (!uskb)
+							return;
 						memcpy(IEEE80211_SKB_RXCB(uskb),
 						       &rx_status,
 						       sizeof(rx_status));
@@ -996,7 +998,7 @@ static void _rtl_pci_init_trx_var(struct
 	 */
 	rtlpci->txringcount[BE_QUEUE] = RT_TXDESC_NUM_BE_QUEUE;
 
-	rtlpci->rxbuffersize = 9100;	/*2048/1024; */
+	rtlpci->rxbuffersize = 4096;
 	rtlpci->rxringcount = RTL_PCI_MAX_RX_COUNT;	/*64; */
 }
 
@@ -1118,7 +1120,7 @@ static int _rtl_pci_init_rx_ring(struct
 			    dev_alloc_skb(rtlpci->rxbuffersize);
 			u32 bufferaddress;
 			if (!skb)
-				return 0;
+				return -ENOMEM;
 			entry = &rtlpci->rx_ring[rx_queue_idx].desc[i];
 
 			/*skb->dev = dev; */

Re: [PATCH] rtlwifi: Fix panic due to memory allocation failure

[Christian Lamparter]

On Wednesday 11 May 2011 18:17:51 Larry Finger wrote:
> The PCI routine of this driver is allocating receive buffers of order 2,
> which causes an unnecessary fragmentation of memory. To make matters
> worse, there are locations that fail to check for allocation failures,
> or return success when the allocation actually failed. Kernel panics
> result.
> 
> Signed-off-by: Larry Finger <Larry.Finger@lwfinger.net>
> Cc: Stable <stable@vger.kernel.org>              [2.6.37 and 2.6.38]
> ---
> 
> John,
> 
> This is 2.6.39 material. I hope we make the cutoff.
> 
> Larry
> ---
> 
> Index: wireless-testing-new/drivers/net/wireless/rtlwifi/pci.c
> ===================================================================
> --- wireless-testing-new.orig/drivers/net/wireless/rtlwifi/pci.c
> +++ wireless-testing-new/drivers/net/wireless/rtlwifi/pci.c
> @@ -996,7 +998,7 @@ static void _rtl_pci_init_trx_var(struct
>  	 */
>  	rtlpci->txringcount[BE_QUEUE] = RT_TXDESC_NUM_BE_QUEUE;
>  
> -	rtlpci->rxbuffersize = 9100;	/*2048/1024; */
> +	rtlpci->rxbuffersize = 4096;
>  	rtlpci->rxringcount = RTL_PCI_MAX_RX_COUNT;	/*64; */
>  }
>
Are you sure this change won't break 8k AMSDU rx?
[or is there some magic that disables disable_amsdu_8k
for pci devices?]

Regards,
	Chr

Re: [PATCH] rtlwifi: Fix panic due to memory allocation failure

[Larry Finger]

On 05/11/2011 11:28 AM, Christian Lamparter wrote:
> On Wednesday 11 May 2011 18:17:51 Larry Finger wrote:
>> The PCI routine of this driver is allocating receive buffers of order 2,
>> which causes an unnecessary fragmentation of memory. To make matters
>> worse, there are locations that fail to check for allocation failures,
>> or return success when the allocation actually failed. Kernel panics
>> result.
>>
>> Signed-off-by: Larry Finger<Larry.Finger@lwfinger.net>
>> Cc: Stable<stable@vger.kernel.org>               [2.6.37 and 2.6.38]
>> ---
>>
>> John,
>>
>> This is 2.6.39 material. I hope we make the cutoff.
>>
>> Larry
>> ---
>>
>> Index: wireless-testing-new/drivers/net/wireless/rtlwifi/pci.c
>> ===================================================================
>> --- wireless-testing-new.orig/drivers/net/wireless/rtlwifi/pci.c
>> +++ wireless-testing-new/drivers/net/wireless/rtlwifi/pci.c
>> @@ -996,7 +998,7 @@ static void _rtl_pci_init_trx_var(struct
>>   	 */
>>   	rtlpci->txringcount[BE_QUEUE] = RT_TXDESC_NUM_BE_QUEUE;
>>
>> -	rtlpci->rxbuffersize = 9100;	/*2048/1024; */
>> +	rtlpci->rxbuffersize = 4096;
>>   	rtlpci->rxringcount = RTL_PCI_MAX_RX_COUNT;	/*64; */
>>   }
>>
> Are you sure this change won't break 8k AMSDU rx?
> [or is there some magic that disables disable_amsdu_8k
> for pci devices?]

It is disabled in the newest driver that is not yet submitted, but not in the 
two other PCI drivers. I need to rework this patch.

John - please drop it.

Larry