Re: [PATCH v3 00/21] NFS/RDMA client patches for 3.17

[Shirley Ma <shirley.ma-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>]

On 07/16/2014 10:57 AM, Chuck Lever wrote:
> 
>> On Jul 16, 2014, at 11:48 AM, Shirley Ma <shirley.ma-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> wrote:
>>
>> These two patches have been significantly reduced interrupt rate by around 4 times.
>>
>>     xprtrdma: Disable completions for FAST_REG_MR Work Requests
>>     xprtrdma: Disable completions for LOCAL_INV Work Requests 
> 
> Thanks Shirley!  This is result applies only to FRMR, correct? Also, i'd imagine the savings would be even greater for adapters that have short page list depth?

Yes, only tested FRMR with mlx4. I can hack the code to test short page page list depth to check the savings. When looking the difference between irq and softirq, it is much closer now.

> 
>>
>> Same NFS read/write workload, here are interrupts rate irq/per sec report based upon /proc/interrupts:
>>
>> w/o patches:
>> -----------
>> PCI-MSI-edge mlx4-ib (204):  105176
>> PCI-MSI-edge mlx4-ib (204):  123650
>> PCI-MSI-edge mlx4-ib (204):  123690
>> PCI-MSI-edge mlx4-ib (204):  116554
>> PCI-MSI-edge mlx4-ib (204):  122864
>>
>> And perf stat irq report: 
>> Performance counter stats for 'system wide':
>>
>>         2,131,870      irq:irq_handler_entry                                        [100.00%]
>>         2,131,870      irq:irq_handler_exit                                         [100.00%]
>>           635,587      irq:softirq_entry                                            [100.00%]
>>           635,597      irq:softirq_exit                                             [100.00%]
>>           636,155      irq:softirq_raise
>>
>>      25.422821792 seconds time elapsed
>>
>> w/i patches:
>> -----------
>> PCI-MSI-edge mlx4-ib (204):  31131
>> PCI-MSI-edge mlx4-ib (204):  32958
>> PCI-MSI-edge mlx4-ib (204):  31068
>> PCI-MSI-edge mlx4-ib (204):  30236
>> PCI-MSI-edge mlx4-ib (204):  33041
>>
>> And perf stat irq report:
>>
>> Performance counter stats for 'system wide':
>>
>>           653,548      irq:irq_handler_entry                                        [100.00%]
>>           653,548      irq:irq_handler_exit                                         [100.00%]
>>           568,138      irq:softirq_entry                                            [100.00%]
>>           568,148      irq:softirq_exit                                             [100.00%]
>>           568,690      irq:softirq_raise                                           
>>
>>      21.675597062 seconds time elapsed
>>
>> Shirley
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html