Re: [SPDK] SPDK and CMB.

[Walker, Benjamin <benjamin.walker at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 5084 bytes --]

On Mon, 2019-04-01 at 19:51 +0000, Liang Yang wrote:
> Hi Jim,
> 
> Great to know that and thanks for your explanations in details! My Intel P4610
> U.2 NVMe does not come with CMB support so I need to look for another model to
> verify if CMB can help improve further latency(1-2us are good savings for QD 1
> workloads) 😊 
> 
> For the overhead of MMO writes tied to SQ doorbell you mentioned, do you
> think, if it makes sense to combine multiple doorbell rings on the submission
> for very high IOPS traffic? There is no mandatory requirement in NVMe spec for
> 1:1 submission:doorbell.  For kernel mode driver, we have seen no perf
> benefits to do that kind of doorbell combination on the submission path. With
> SPDK capable of handling much higher traffic, maybe it is worth re-considering 
> the SQ doorbell coalescing?

The patch series that Jim linked below[1] is doing exactly that. It yields very
large performance improvements with SPDK because everything else is so highly
optimized that the MMIO writes were actually the bottleneck.

[1] https://review.gerrithub.io/c/spdk/spdk/+/447239

> 
> Thoughts?
> 
> Liang
> 
> -----Original Message-----
> From: SPDK <spdk-bounces(a)lists.01.org> On Behalf Of Harris, James R
> Sent: Monday, April 1, 2019 12:30 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] SPDK and CMB.
> 
> Hi Liang,
> 
> SPDK does support using CMB for NVMe SQs - and in fact will do so by default
> if the SSD supports putting SQs in the CMB.
> 
> The CMB is mapped write combining, so writing the SQ entry is fairly efficient
> and saves the DMA as you mentioned.  (Host CPU never reads the SQ so we're
> only concerned about the writes.) Skipping the DMA is more of a latency rather
> than bandwidth advantage since the entries are so small.  You should probably
> expect somewhere around 1-2us in saved latency for QD=1 workloads but that's
> dependent on platform and SSD characteristics.
> 
> One potential downside is that writing the SQ entry is effectively adding an
> MMIO write.  At very high IOPs rates (>3-4M IOPs on a single Xeon core), we
> see SQ doorbell MMIO writes becoming a bottleneck even without the CMB.  You
> can see a recent patch set[1] from Ben Walker that's been merged recently to
> master that tries to reduce the SQ doorbell MMIO write rate.  This would be
> something to watch out for if enabling SQs in CMB with a large number of SSDs
> and measuring per core IOPs rates.
> 
> Note that the SPDK nvme/perf application provides the -D option to disable use
> of the CMB for SQs even if the SSD supports it.
> 
> -Jim
> 
> [1] 
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Freview.gerrithub.io%2Fc%2Fspdk%2Fspdk%2F%2B%2F448310%2F2&amp;data=02%7C01%7Cliangy%40microsoft.com%7C75a240b93fd047db93d408d6b6d87a11%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636897438264821210&amp;sdata=QJKOCfJM%2FFiwDas%2BWQIqI2Tv%2Bk5xzbV85mQ2L%2B7EHjw%3D&amp;reserved=0
> 
> 
> On 4/1/19, 11:31 AM, "SPDK on behalf of Liang Yang" <
> spdk-bounces(a)lists.01.org on behalf of liangy(a)microsoft.com> wrote:
> 
>     The only place we have seen mentioning CMB as part of SPDK doc is using
> CMB to enable fast memcpy:
>     
>     -------------------------------------
>     
>     Overview
>     
>     Peer-2-Peer (P2P) is the concept of DMAing data directly from one PCI End
> Point (EP) to another without using a system memory buffer. The most obvious
> example of this from an SPDK perspective is using a NVMe Controller Memory
> Buffer (CMB) to enable direct copies of data between two NVMe SSDs.
>     -------------------------------------
>     
>     My questions here are: does SPDK support using CMB as a place to host NVMe
> submission queue instead of using host memory? I can see pros(reduce NVMe
> command transfer traffic) and cons(slower access of CMB compared with host
> memory) of using CMB for that?
>     
>     Thanks,
>     Liang
>     
>     
>     _______________________________________________
>     SPDK mailing list
>     SPDK(a)lists.01.org
>     
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.01.org%2Fmailman%2Flistinfo%2Fspdk&amp;data=02%7C01%7Cliangy%40microsoft.com%7C75a240b93fd047db93d408d6b6d87a11%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636897438264821210&amp;sdata=sGGORlUZO2VN4%2BUrJm1nHe4g2WyyRLLZrpUwHjrSwcg%3D&amp;reserved=0
>     
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.01.org%2Fmailman%2Flistinfo%2Fspdk&amp;data=02%7C01%7Cliangy%40microsoft.com%7C75a240b93fd047db93d408d6b6d87a11%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636897438264821210&amp;sdata=sGGORlUZO2VN4%2BUrJm1nHe4g2WyyRLLZrpUwHjrSwcg%3D&amp;reserved=0
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk