Re: [PATCH v4 00/14] RDS: connection scalability and performance improvements

[David Miller <davem@davemloft.net>]

From: Santosh Shilimkar <santosh.shilimkar@oracle.com>
Date: Wed,  7 Oct 2015 08:53:34 -0700

> [v4]
> Re-sending the same patches from v3 again since my repost of
> patch 05/14 from v3 was whitespace damaged.
> 
> [v3]
> Updated patch "[PATCH v2 05/14] RDS: defer the over_batch work to
> send worker" as per David Miller's comment [4] to avoid the magic
> value usage. Patch now makes use of already available but unused
> send_batch_count module parameter. Rest of the patches are same as
> earlier version v2 [3]
> 
> [v2]:
> Dropped "[PATCH 05/15] RDS: increase size of hash-table to 8K" from
> earlier version [1]. I plan to address the hash table scalability using
> re-sizable hash tables as suggested by David Laight and David Miller [2]
> 
> This series addresses RDS connection bottlenecks on massive workloads and
> improve the RDMA performance almost by 3X. RDS TCP also gets a small gain
> of about 12%.
> 
> RDS is being used in massive systems with high scalability where several
> hundred thousand end points and tens of thousands of local processes
> are operating in tens of thousand sockets. Being RC(reliable connection),
> socket bind and release happens very often and any inefficiencies in
> bind hash look ups hurts the overall system performance. RDS bin hash-table
> uses global spin-lock which is the biggest bottleneck. To make matter worst,
> it uses rcu inside global lock for hash buckets.
> This is being addressed by simply using per bucket rw lock which makes the
> locking simple and very efficient. The hash table size is still an issue and
> I plan to address it by using re-sizable hash tables as suggested on the list.
> 
> For RDS RDMA improvement, the completion handling is revamped so that we
> can do batch completions. Both send and receive completion handlers are
> split logically to achieve the same. RDS 8K messages being one of the
> key usecase, mr pool is adapted to have the 8K mrs along with default 1M
> mrs. And while doing this, few fixes and couple of bottlenecks seen with
> rds_sendmsg() are addressed.
> 
> Series applies against 4.3-rc1 as well net-next. Its tested on Oracle
> hardware with IB fabric for both bcopy as well as RDMA mode. RDS TCP is
> tested with iXGB NIC. Like last time, iWARP transport is untested with
> these changes. The patchset is also available at below git repo:
> 
> git://git.kernel.org/pub/scm/linux/kernel/git/ssantosh/linux.git net/rds/4.3-v3
> 
> As a side note, the IB HCA driver I used for testing misses at least 3
> important patches in upstream to see the full blown IB performance and
> am hoping to get that in mainline with help of them.

Pulled, thank you.