From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============0407513968799534534==" MIME-Version: 1.0 From: Harris, James R Subject: Re: [SPDK] SPDK and CMB. Date: Mon, 01 Apr 2019 20:03:10 +0000 Message-ID: In-Reply-To: MW2PR2101MB1113F52606E1E4755F674049D5550@MW2PR2101MB1113.namprd21.prod.outlook.com List-ID: To: spdk@lists.01.org --===============0407513968799534534== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi Liang, The patch set referenced below does try to coalesce multiple SQ entries int= o one doorbell write. It does this completely transparently when the coale= scing feature is enabled on an I/O qpair. The SQ doorbell will not be writ= ten when an SQ entry is written - rather it will wait to write the SQ doorb= ell until the next time that qpair is checked for completions. More aggressive coalescing would require some kind of time-based mechanism,= or an explicit API that users must call (separate from checking for comple= tions) to ring the SQ doorbell. Something worth looking at, but both requi= re some amount of extra work before they could be implemented. I think the= time-based mechanism could be interesting, provided we do it in a way that= doesn't explode the number of rdtsc instructions executed (which can add u= p). It could be used to coalesce CQ doorbells as well. I'd be curious to know what kind of IOPs rates you were testing when trying= coalescing the kernel mode driver. We had to get north of 4M IOPs on a si= ngle Xeon core before we could see benefits from coalescing. This is defin= itely in the realm of extreme performance. -Jim =EF=BB=BFOn 4/1/19, 12:51 PM, "SPDK on behalf of Liang Yang" wrote: Hi Jim, = Great to know that and thanks for your explanations in details! My Inte= l P4610 U.2 NVMe does not come with CMB support so I need to look for anoth= er model to verify if CMB can help improve further latency(1-2us are good s= avings for QD 1 workloads) =F0=9F=98=8A = = For the overhead of MMO writes tied to SQ doorbell you mentioned, do yo= u think, if it makes sense to combine multiple doorbell rings on the submis= sion for very high IOPS traffic? There is no mandatory requirement in NVMe = spec for 1:1 submission:doorbell. For kernel mode driver, we have seen no = perf benefits to do that kind of doorbell combination on the submission pat= h. With SPDK capable of handling much higher traffic, maybe it is worth re-= considering the SQ doorbell coalescing? = Thoughts? = Liang = -----Original Message----- From: SPDK On Behalf Of Harris, James R Sent: Monday, April 1, 2019 12:30 PM To: Storage Performance Development Kit Subject: Re: [SPDK] SPDK and CMB. = Hi Liang, = SPDK does support using CMB for NVMe SQs - and in fact will do so by de= fault if the SSD supports putting SQs in the CMB. = The CMB is mapped write combining, so writing the SQ entry is fairly ef= ficient and saves the DMA as you mentioned. (Host CPU never reads the SQ s= o we're only concerned about the writes.) Skipping the DMA is more of a lat= ency rather than bandwidth advantage since the entries are so small. You s= hould probably expect somewhere around 1-2us in saved latency for QD=3D1 wo= rkloads but that's dependent on platform and SSD characteristics. = One potential downside is that writing the SQ entry is effectively addi= ng an MMIO write. At very high IOPs rates (>3-4M IOPs on a single Xeon cor= e), we see SQ doorbell MMIO writes becoming a bottleneck even without the C= MB. You can see a recent patch set[1] from Ben Walker that's been merged r= ecently to master that tries to reduce the SQ doorbell MMIO write rate. Th= is would be something to watch out for if enabling SQs in CMB with a large = number of SSDs and measuring per core IOPs rates. = Note that the SPDK nvme/perf application provides the -D option to disa= ble use of the CMB for SQs even if the SSD supports it. = -Jim = [1] https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2= Freview.gerrithub.io%2Fc%2Fspdk%2Fspdk%2F%2B%2F448310%2F2&data=3D02%7C0= 1%7Cliangy%40microsoft.com%7C75a240b93fd047db93d408d6b6d87a11%7C72f988bf86f= 141af91ab2d7cd011db47%7C1%7C0%7C636897438264821210&sdata=3DQJKOCfJM%2FF= iwDas%2BWQIqI2Tv%2Bk5xzbV85mQ2L%2B7EHjw%3D&reserved=3D0 = = On 4/1/19, 11:31 AM, "SPDK on behalf of Liang Yang" wrote: = The only place we have seen mentioning CMB as part of SPDK doc is u= sing CMB to enable fast memcpy: = ------------------------------------- = Overview = Peer-2-Peer (P2P) is the concept of DMAing data directly from one P= CI End Point (EP) to another without using a system memory buffer. The most= obvious example of this from an SPDK perspective is using a NVMe Controlle= r Memory Buffer (CMB) to enable direct copies of data between two NVMe SSDs. ------------------------------------- = My questions here are: does SPDK support using CMB as a place to ho= st NVMe submission queue instead of using host memory? I can see pros(reduc= e NVMe command transfer traffic) and cons(slower access of CMB compared wit= h host memory) of using CMB for that? = Thanks, Liang = = _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2= Flists.01.org%2Fmailman%2Flistinfo%2Fspdk&data=3D02%7C01%7Cliangy%40mic= rosoft.com%7C75a240b93fd047db93d408d6b6d87a11%7C72f988bf86f141af91ab2d7cd01= 1db47%7C1%7C0%7C636897438264821210&sdata=3DsGGORlUZO2VN4%2BUrJm1nHe4g2W= yyRLLZrpUwHjrSwcg%3D&reserved=3D0 = = _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Flis= ts.01.org%2Fmailman%2Flistinfo%2Fspdk&data=3D02%7C01%7Cliangy%40microso= ft.com%7C75a240b93fd047db93d408d6b6d87a11%7C72f988bf86f141af91ab2d7cd011db4= 7%7C1%7C0%7C636897438264821210&sdata=3DsGGORlUZO2VN4%2BUrJm1nHe4g2WyyRL= LZrpUwHjrSwcg%3D&reserved=3D0 _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk = --===============0407513968799534534==--