From: santosh shilimkar <santosh.shilimkar-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
To: Sagi Grimberg
<sagig-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>,
netdev-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Cc: linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org,
ssantosh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org,
"linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org"
<linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
Subject: Re: [PATCH 00/15] RDS: connection scalability and performance improvements
Date: Mon, 21 Sep 2015 09:02:11 -0700 [thread overview]
Message-ID: <56002A03.5060305@oracle.com> (raw)
In-Reply-To: <55FE7060.6010205-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
On 9/20/2015 1:37 AM, Sagi Grimberg wrote:
> On 9/20/2015 2:04 AM, Santosh Shilimkar wrote:
>> This series addresses RDS connection bottlenecks on massive workloads and
>> improve the RDMA performance almost by 3X. RDS TCP also gets a small gain
>> of about 12%.
>>
>> RDS is being used in massive systems with high scalability where several
>> hundred thousand end points and tens of thousands of local processes
>> are operating in tens of thousand sockets. Being RC(reliable connection),
>> socket bind and release happens very often and any inefficiencies in
>> bind hash look ups hurts the overall system performance. RDS bin
>> hash-table
>> uses global spin-lock which is the biggest bottleneck. To make matter
>> worst,
>> it uses rcu inside global lock for hash buckets.
>> This is being addressed by simply using per bucket rw lock which makes
>> the
>> locking simple and very efficient. The hash table size is also scaled up
>> accordingly.
>>
>> For RDS RDMA improvement, the completion handling is revamped so that we
>> can do batch completions. Both send and receive completion handlers are
>> split logically to achieve the same. RDS 8K messages being one of the
>> key usecase, mr pool is adapted to have the 8K mrs along with default 1M
>> mrs. And while doing this, few fixes and couple of bottlenecks seen with
>> rds_sendmsg() are addressed.
>
> Hi Santosh,
>
> I think that can get a more effective code review if you CC the
> Linux-rdma mailing list.
>
I will do that from next time. Thanks Sagi !!
Regards,
Santosh
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
WARNING: multiple messages have this Message-ID (diff)
From: santosh shilimkar <santosh.shilimkar@oracle.com>
To: Sagi Grimberg <sagig@dev.mellanox.co.il>, netdev@vger.kernel.org
Cc: linux-kernel@vger.kernel.org, davem@davemloft.net,
ssantosh@kernel.org,
"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>
Subject: Re: [PATCH 00/15] RDS: connection scalability and performance improvements
Date: Mon, 21 Sep 2015 09:02:11 -0700 [thread overview]
Message-ID: <56002A03.5060305@oracle.com> (raw)
In-Reply-To: <55FE7060.6010205@dev.mellanox.co.il>
On 9/20/2015 1:37 AM, Sagi Grimberg wrote:
> On 9/20/2015 2:04 AM, Santosh Shilimkar wrote:
>> This series addresses RDS connection bottlenecks on massive workloads and
>> improve the RDMA performance almost by 3X. RDS TCP also gets a small gain
>> of about 12%.
>>
>> RDS is being used in massive systems with high scalability where several
>> hundred thousand end points and tens of thousands of local processes
>> are operating in tens of thousand sockets. Being RC(reliable connection),
>> socket bind and release happens very often and any inefficiencies in
>> bind hash look ups hurts the overall system performance. RDS bin
>> hash-table
>> uses global spin-lock which is the biggest bottleneck. To make matter
>> worst,
>> it uses rcu inside global lock for hash buckets.
>> This is being addressed by simply using per bucket rw lock which makes
>> the
>> locking simple and very efficient. The hash table size is also scaled up
>> accordingly.
>>
>> For RDS RDMA improvement, the completion handling is revamped so that we
>> can do batch completions. Both send and receive completion handlers are
>> split logically to achieve the same. RDS 8K messages being one of the
>> key usecase, mr pool is adapted to have the 8K mrs along with default 1M
>> mrs. And while doing this, few fixes and couple of bottlenecks seen with
>> rds_sendmsg() are addressed.
>
> Hi Santosh,
>
> I think that can get a more effective code review if you CC the
> Linux-rdma mailing list.
>
I will do that from next time. Thanks Sagi !!
Regards,
Santosh
next prev parent reply other threads:[~2015-09-21 16:02 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-09-19 23:04 [PATCH 00/15] RDS: connection scalability and performance improvements Santosh Shilimkar
2015-09-19 23:04 ` [PATCH 01/15] RDS: use kfree_rcu in rds_ib_remove_ipaddr Santosh Shilimkar
2015-09-19 23:04 ` [PATCH 02/15] RDS: make socket bind/release locking scheme simple and more efficient Santosh Shilimkar
2015-09-19 23:04 ` [PATCH 03/15] RDS: fix rds_sock reference bug while doing bind Santosh Shilimkar
2015-09-19 23:04 ` [PATCH 04/15] RDS: Use per-bucket rw lock for bind hash-table Santosh Shilimkar
2015-09-19 23:04 ` [PATCH 05/15] RDS: increase size of hash-table to 8K Santosh Shilimkar
2015-09-21 8:31 ` David Laight
2015-09-21 15:52 ` santosh shilimkar
2015-09-21 23:05 ` David Miller
2015-09-21 23:55 ` santosh shilimkar
2015-09-23 18:06 ` santosh shilimkar
2015-09-19 23:04 ` [PATCH 06/15] RDS: defer the over_batch work to send worker Santosh Shilimkar
2015-09-19 23:04 ` [PATCH 07/15] RDS: use rds_send_xmit() state instead of RDS_LL_SEND_FULL Santosh Shilimkar
2015-09-19 23:04 ` [PATCH 08/15] RDS: ack more receive completions to improve performance Santosh Shilimkar
2015-09-19 23:04 ` [PATCH 09/15] RDS: split send completion handling and do batch ack Santosh Shilimkar
2015-09-19 23:04 ` [PATCH 10/15] RDS: handle rds_ibdev release case instead of crashing the kernel Santosh Shilimkar
2015-09-19 23:04 ` [PATCH 11/15] RDS: fix the rds_ib_fmr_wq kick call Santosh Shilimkar
2015-09-19 23:04 ` [PATCH 12/15] RDS: use already available pool handle from ibmr Santosh Shilimkar
2015-09-19 23:04 ` [PATCH 13/15] RDS: mark rds_ib_fmr_wq static Santosh Shilimkar
2015-09-19 23:04 ` [PATCH 14/15] RDS: use max_mr from HCA caps than max_fmr Santosh Shilimkar
2015-09-19 23:04 ` [PATCH 15/15] RDS: split mr pool to improve 8K messages performance Santosh Shilimkar
2015-09-20 8:37 ` [PATCH 00/15] RDS: connection scalability and performance improvements Sagi Grimberg
[not found] ` <55FE7060.6010205-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2015-09-21 16:02 ` santosh shilimkar [this message]
2015-09-21 16:02 ` santosh shilimkar
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=56002A03.5060305@oracle.com \
--to=santosh.shilimkar-qhclzuegtsvqt0dzr+alfa@public.gmane.org \
--cc=davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org \
--cc=linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
--cc=linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
--cc=netdev-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
--cc=sagig-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org \
--cc=ssantosh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.