From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============5439143416813523464==" MIME-Version: 1.0 From: Walker, Benjamin Subject: Re: [SPDK] SPDK and CMB. Date: Mon, 01 Apr 2019 20:02:41 +0000 Message-ID: In-Reply-To: MW2PR2101MB1113F52606E1E4755F674049D5550@MW2PR2101MB1113.namprd21.prod.outlook.com List-ID: To: spdk@lists.01.org --===============5439143416813523464== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Mon, 2019-04-01 at 19:51 +0000, Liang Yang wrote: > Hi Jim, > = > Great to know that and thanks for your explanations in details! My Intel = P4610 > U.2 NVMe does not come with CMB support so I need to look for another mod= el to > verify if CMB can help improve further latency(1-2us are good savings for= QD 1 > workloads) =F0=9F=98=8A = > = > For the overhead of MMO writes tied to SQ doorbell you mentioned, do you > think, if it makes sense to combine multiple doorbell rings on the submis= sion > for very high IOPS traffic? There is no mandatory requirement in NVMe spe= c for > 1:1 submission:doorbell. For kernel mode driver, we have seen no perf > benefits to do that kind of doorbell combination on the submission path. = With > SPDK capable of handling much higher traffic, maybe it is worth re-consid= ering = > the SQ doorbell coalescing? The patch series that Jim linked below[1] is doing exactly that. It yields = very large performance improvements with SPDK because everything else is so high= ly optimized that the MMIO writes were actually the bottleneck. [1] https://review.gerrithub.io/c/spdk/spdk/+/447239 > = > Thoughts? > = > Liang > = > -----Original Message----- > From: SPDK On Behalf Of Harris, James R > Sent: Monday, April 1, 2019 12:30 PM > To: Storage Performance Development Kit > Subject: Re: [SPDK] SPDK and CMB. > = > Hi Liang, > = > SPDK does support using CMB for NVMe SQs - and in fact will do so by defa= ult > if the SSD supports putting SQs in the CMB. > = > The CMB is mapped write combining, so writing the SQ entry is fairly effi= cient > and saves the DMA as you mentioned. (Host CPU never reads the SQ so we're > only concerned about the writes.) Skipping the DMA is more of a latency r= ather > than bandwidth advantage since the entries are so small. You should prob= ably > expect somewhere around 1-2us in saved latency for QD=3D1 workloads but t= hat's > dependent on platform and SSD characteristics. > = > One potential downside is that writing the SQ entry is effectively adding= an > MMIO write. At very high IOPs rates (>3-4M IOPs on a single Xeon core), = we > see SQ doorbell MMIO writes becoming a bottleneck even without the CMB. = You > can see a recent patch set[1] from Ben Walker that's been merged recently= to > master that tries to reduce the SQ doorbell MMIO write rate. This would = be > something to watch out for if enabling SQs in CMB with a large number of = SSDs > and measuring per core IOPs rates. > = > Note that the SPDK nvme/perf application provides the -D option to disabl= e use > of the CMB for SQs even if the SSD supports it. > = > -Jim > = > [1] = > https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Frevie= w.gerrithub.io%2Fc%2Fspdk%2Fspdk%2F%2B%2F448310%2F2&data=3D02%7C01%7Cli= angy%40microsoft.com%7C75a240b93fd047db93d408d6b6d87a11%7C72f988bf86f141af9= 1ab2d7cd011db47%7C1%7C0%7C636897438264821210&sdata=3DQJKOCfJM%2FFiwDas%= 2BWQIqI2Tv%2Bk5xzbV85mQ2L%2B7EHjw%3D&reserved=3D0 > = > = > =EF=BB=BFOn 4/1/19, 11:31 AM, "SPDK on behalf of Liang Yang" < > spdk-bounces(a)lists.01.org on behalf of liangy(a)microsoft.com> wrote: > = > The only place we have seen mentioning CMB as part of SPDK doc is usi= ng > CMB to enable fast memcpy: > = > ------------------------------------- > = > Overview > = > Peer-2-Peer (P2P) is the concept of DMAing data directly from one PCI= End > Point (EP) to another without using a system memory buffer. The most obvi= ous > example of this from an SPDK perspective is using a NVMe Controller Memory > Buffer (CMB) to enable direct copies of data between two NVMe SSDs. > ------------------------------------- > = > My questions here are: does SPDK support using CMB as a place to host= NVMe > submission queue instead of using host memory? I can see pros(reduce NVMe > command transfer traffic) and cons(slower access of CMB compared with host > memory) of using CMB for that? > = > Thanks, > Liang > = > = > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > = > https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Flists= .01.org%2Fmailman%2Flistinfo%2Fspdk&data=3D02%7C01%7Cliangy%40microsoft= .com%7C75a240b93fd047db93d408d6b6d87a11%7C72f988bf86f141af91ab2d7cd011db47%= 7C1%7C0%7C636897438264821210&sdata=3DsGGORlUZO2VN4%2BUrJm1nHe4g2WyyRLLZ= rpUwHjrSwcg%3D&reserved=3D0 > = > = > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Flists= .01.org%2Fmailman%2Flistinfo%2Fspdk&data=3D02%7C01%7Cliangy%40microsoft= .com%7C75a240b93fd047db93d408d6b6d87a11%7C72f988bf86f141af91ab2d7cd011db47%= 7C1%7C0%7C636897438264821210&sdata=3DsGGORlUZO2VN4%2BUrJm1nHe4g2WyyRLLZ= rpUwHjrSwcg%3D&reserved=3D0 > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk --===============5439143416813523464==--