From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932483AbbIUQCf (ORCPT ); Mon, 21 Sep 2015 12:02:35 -0400 Received: from aserp1040.oracle.com ([141.146.126.69]:24784 "EHLO aserp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752372AbbIUQCd (ORCPT ); Mon, 21 Sep 2015 12:02:33 -0400 Subject: Re: [PATCH 00/15] RDS: connection scalability and performance improvements To: Sagi Grimberg , netdev@vger.kernel.org References: <1442703892-26692-1-git-send-email-santosh.shilimkar@oracle.com> <55FE7060.6010205@dev.mellanox.co.il> Cc: linux-kernel@vger.kernel.org, davem@davemloft.net, ssantosh@kernel.org, "linux-rdma@vger.kernel.org" From: santosh shilimkar Organization: Oracle Corporation Message-ID: <56002A03.5060305@oracle.com> Date: Mon, 21 Sep 2015 09:02:11 -0700 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <55FE7060.6010205@dev.mellanox.co.il> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Source-IP: aserv0021.oracle.com [141.146.126.233] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 9/20/2015 1:37 AM, Sagi Grimberg wrote: > On 9/20/2015 2:04 AM, Santosh Shilimkar wrote: >> This series addresses RDS connection bottlenecks on massive workloads and >> improve the RDMA performance almost by 3X. RDS TCP also gets a small gain >> of about 12%. >> >> RDS is being used in massive systems with high scalability where several >> hundred thousand end points and tens of thousands of local processes >> are operating in tens of thousand sockets. Being RC(reliable connection), >> socket bind and release happens very often and any inefficiencies in >> bind hash look ups hurts the overall system performance. RDS bin >> hash-table >> uses global spin-lock which is the biggest bottleneck. To make matter >> worst, >> it uses rcu inside global lock for hash buckets. >> This is being addressed by simply using per bucket rw lock which makes >> the >> locking simple and very efficient. The hash table size is also scaled up >> accordingly. >> >> For RDS RDMA improvement, the completion handling is revamped so that we >> can do batch completions. Both send and receive completion handlers are >> split logically to achieve the same. RDS 8K messages being one of the >> key usecase, mr pool is adapted to have the 8K mrs along with default 1M >> mrs. And while doing this, few fixes and couple of bottlenecks seen with >> rds_sendmsg() are addressed. > > Hi Santosh, > > I think that can get a more effective code review if you CC the > Linux-rdma mailing list. > I will do that from next time. Thanks Sagi !! Regards, Santosh