On Tue, 2016-12-06 at 16:58 +0000, Walker, Benjamin wrote:
> On Tue, 2016-12-06 at 16:35 +0000, Harris, James R wrote:
> > 
> > > On Dec 6, 2016, at 5:49 AM, Kumaraparameshwaran Rathnavel <krath@
> > > cloudsimple
> > > .com> wrote:
> > > 
> > > The setup is the default setup of NVMf target with Direct
> > > attached
> > > Subsystem. The FIO plugin was at the target side and the IOPS
> > > from SPDK was
> > > found to be around 2.5K and the IOPS from kernel was found to be
> > > 100k.
> > 
> > 
> > Hi Param,
> > 
> > Can you check the AcceptorPollRate setting in your configuration
> > file?  There
> > was a recent fix to the default value for this parameter.  You
> > should set it
> > to at least 10000 (10ms).
> > 
> > https://github.com/spdk/spdk/commit/5f3761cf7ec50cf03183203dc50657a
> > 9f2a18908
> > 
> > Thanks,
> 
> Just to reinforce here - this is the solution. We saw this internally
> a few
> weeks ago. Increasing the AcceptorPollRate to at least 10ms (you can
> go as much
> as 1 second if you want, but the performance benefit decreases the
> larger you
> make it), or moving the accceptor to a different core using the
> AcceptorCore
> option will fix the problem.

To follow up on this - we've now root caused exactly why our acceptor
polling function was slowing down the system so much. Before, I had
suggested polling less often or doing the polling on a separate core.
After these two commits

https://github.com/spdk/spdk/commit/d2c0feac8a13944bc9af7c60ed047809cf1f5d9dhttps://github.com/spdk/spdk/commit/4a95a81e694c4f58031b181a6127583d6f55483e

that is no longer necessary. Any remotely reasonable value for AcceptorPollRate
and placing the Acceptor on any core will now yield excellent performance
results with SPDK.