Re: [SPDK] SPDK and CMB.

[Harris, James R <james.r.harris at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 6081 bytes --]

Hi Liang,

The patch set referenced below does try to coalesce multiple SQ entries into one doorbell write.  It does this completely transparently when the coalescing feature is enabled on an I/O qpair.  The SQ doorbell will not be written when an SQ entry is written - rather it will wait to write the SQ doorbell until the next time that qpair is checked for completions.

More aggressive coalescing would require some kind of time-based mechanism, or an explicit API that users must call (separate from checking for completions) to ring the SQ doorbell.  Something worth looking at, but both require some amount of extra work before they could be implemented.  I think the time-based mechanism could be interesting, provided we do it in a way that doesn't explode the number of rdtsc instructions executed (which can add up).  It could be used to coalesce CQ doorbells as well.

I'd be curious to know what kind of IOPs rates you were testing when trying coalescing the kernel mode driver.  We had to get north of 4M IOPs on a single Xeon core before we could see benefits from coalescing.  This is definitely in the realm of extreme performance.

-Jim





On 4/1/19, 12:51 PM, "SPDK on behalf of Liang Yang" <spdk-bounces(a)lists.01.org on behalf of liangy(a)microsoft.com> wrote:

    Hi Jim,
    
    Great to know that and thanks for your explanations in details! My Intel P4610 U.2 NVMe does not come with CMB support so I need to look for another model to verify if CMB can help improve further latency(1-2us are good savings for QD 1 workloads) 😊 
    
    For the overhead of MMO writes tied to SQ doorbell you mentioned, do you think, if it makes sense to combine multiple doorbell rings on the submission for very high IOPS traffic? There is no mandatory requirement in NVMe spec for 1:1 submission:doorbell.  For kernel mode driver, we have seen no perf benefits to do that kind of doorbell combination on the submission path. With SPDK capable of handling much higher traffic, maybe it is worth re-considering the SQ doorbell coalescing?
    
    Thoughts?
    
    Liang
    
    -----Original Message-----
    From: SPDK <spdk-bounces(a)lists.01.org> On Behalf Of Harris, James R
    Sent: Monday, April 1, 2019 12:30 PM
    To: Storage Performance Development Kit <spdk(a)lists.01.org>
    Subject: Re: [SPDK] SPDK and CMB.
    
    Hi Liang,
    
    SPDK does support using CMB for NVMe SQs - and in fact will do so by default if the SSD supports putting SQs in the CMB.
    
    The CMB is mapped write combining, so writing the SQ entry is fairly efficient and saves the DMA as you mentioned.  (Host CPU never reads the SQ so we're only concerned about the writes.) Skipping the DMA is more of a latency rather than bandwidth advantage since the entries are so small.  You should probably expect somewhere around 1-2us in saved latency for QD=1 workloads but that's dependent on platform and SSD characteristics.
    
    One potential downside is that writing the SQ entry is effectively adding an MMIO write.  At very high IOPs rates (>3-4M IOPs on a single Xeon core), we see SQ doorbell MMIO writes becoming a bottleneck even without the CMB.  You can see a recent patch set[1] from Ben Walker that's been merged recently to master that tries to reduce the SQ doorbell MMIO write rate.  This would be something to watch out for if enabling SQs in CMB with a large number of SSDs and measuring per core IOPs rates.
    
    Note that the SPDK nvme/perf application provides the -D option to disable use of the CMB for SQs even if the SSD supports it.
    
    -Jim
    
    [1] https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Freview.gerrithub.io%2Fc%2Fspdk%2Fspdk%2F%2B%2F448310%2F2&amp;data=02%7C01%7Cliangy%40microsoft.com%7C75a240b93fd047db93d408d6b6d87a11%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636897438264821210&amp;sdata=QJKOCfJM%2FFiwDas%2BWQIqI2Tv%2Bk5xzbV85mQ2L%2B7EHjw%3D&amp;reserved=0
    
    
    On 4/1/19, 11:31 AM, "SPDK on behalf of Liang Yang" <spdk-bounces(a)lists.01.org on behalf of liangy(a)microsoft.com> wrote:
    
        The only place we have seen mentioning CMB as part of SPDK doc is using CMB to enable fast memcpy:
        
        -------------------------------------
        
        Overview
        
        Peer-2-Peer (P2P) is the concept of DMAing data directly from one PCI End Point (EP) to another without using a system memory buffer. The most obvious example of this from an SPDK perspective is using a NVMe Controller Memory Buffer (CMB) to enable direct copies of data between two NVMe SSDs.
        -------------------------------------
        
        My questions here are: does SPDK support using CMB as a place to host NVMe submission queue instead of using host memory? I can see pros(reduce NVMe command transfer traffic) and cons(slower access of CMB compared with host memory) of using CMB for that?
        
        Thanks,
        Liang
        
        
        _______________________________________________
        SPDK mailing list
        SPDK(a)lists.01.org
        https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.01.org%2Fmailman%2Flistinfo%2Fspdk&amp;data=02%7C01%7Cliangy%40microsoft.com%7C75a240b93fd047db93d408d6b6d87a11%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636897438264821210&amp;sdata=sGGORlUZO2VN4%2BUrJm1nHe4g2WyyRLLZrpUwHjrSwcg%3D&amp;reserved=0
        
    
    _______________________________________________
    SPDK mailing list
    SPDK(a)lists.01.org
    https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.01.org%2Fmailman%2Flistinfo%2Fspdk&amp;data=02%7C01%7Cliangy%40microsoft.com%7C75a240b93fd047db93d408d6b6d87a11%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636897438264821210&amp;sdata=sGGORlUZO2VN4%2BUrJm1nHe4g2WyyRLLZrpUwHjrSwcg%3D&amp;reserved=0
    _______________________________________________
    SPDK mailing list
    SPDK(a)lists.01.org
    https://lists.01.org/mailman/listinfo/spdk