From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============1245023145423199012==" MIME-Version: 1.0 From: Harris, James R Subject: Re: [SPDK] Problem with Blobstore when write 65MB continously Date: Wed, 10 Jan 2018 16:21:17 +0000 Message-ID: <6CE0D7EB-15CC-44A2-888B-C283A1C54050@intel.com> In-Reply-To: 82C9F782B054C94B9FC04A331649C77A9D4926BB@fmsmsx104.amr.corp.intel.com List-ID: To: spdk@lists.01.org --===============1245023145423199012== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi Paul and Zhengyu, The problem is that the app is not giving the block device a chance to comp= lete any I/O while submitting the 520 back-to-back requests. Blobstore is = passive here =E2=80=93 it does not do any polling on the block device =E2= =80=93 that is up to the application. Technically, with a malloc backend, there is really no polling required sin= ce it=E2=80=99s just a memcpy =E2=80=93 but the bdev layer defers immediate= completions as an event so that bdev API users are ensured they will never= get their completion callback invoked in the context of the bdev IO submis= sion. So in this test case, malloc ends up behaving similarly to an asynch= ronous block device backend like NVMe. For NVMe, just giving the app time to poll will not guarantee that completi= ons will occur fast enough to allow more submissions. The CPU will always = be able to submit I/O at a faster rate than the NVMe device can complete th= em (even for very small I/O). Increasing the number of channel reqs would work =E2=80=93 but at some poin= t these will still run out. So it really depends on your application =E2= =80=93 either increase the channel reqs to the absolutely maximum you will = ever need, or add ENOMEM handling. Note that using more channels will only work if those channels are each all= ocated on a separate thread. Multiple requests to allocate a Blobstore cha= nnel on the same thread will always return the same channel. Regards, = -Jim On 1/10/18, 9:03 AM, "SPDK on behalf of Luse, Paul E" wrote: Hi Zhang, = I'm not suggesting changing anything right now, thanks for the point on= the SSD though - I'd wait for Ben or someone else to jump in with a bit mo= re info on why this is happening. = Thanks!! Paul = -----Original Message----- From: Zhengyu Zhang [mailto:freeman.zhang1992(a)gmail.com] = Sent: Wednesday, January 10, 2018 9:00 AM To: Luse, Paul E Cc: Storage Performance Development Kit Subject: Re: [SPDK] Problem with Blobstore when write 65MB continously = Hi Paul = Thanks for your reply! = = On 1/10/18 11:18 PM, Luse, Paul E wrote: > So what=E2=80=99s happening here is internally within Blobstore when > _spdk_blob_request_submit_op() tries to get a channel->req via > spdk_bs_batch_open() for a write (bear the end) it doesn=E2=80=99t ha= ve any = > available so returns NULL which results in a callback error of =E2=80= =93ENOMEM = > to the hello_blob callback. The default number of channel reqs is 51= 2 = > and the hello_blob app doesn=E2=80=99t change that and uses a single = channel = > for submitting the 520 back to back write requests that you are = > issuing and this failure happens right there towards the end. > = = So you are suggesting me to tune the number of channel reqs or using mu= ltiple channels if I want to write more? = > = > Ben, if I crank up the max channel reqs this works OK. I=E2=80=99m t= hinking = > this shouldn=E2=80=99t be needed and wondering why we aren=E2=80=99t = placing completed = > channel reqs back on the list fast enough (if that=E2=80=99s the real= problem). > Something to do with this being a malloc backend maybe? Would try = > w/nvme but my dev system isn=E2=80=99t quite ready for prime time yet= after = > reblasting it=E2=80=A6. > = > = I tested on both malloc backend and an P3700 NVMe SSD. Their results ar= e the same. = Thanks! Zhengyu _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk = --===============1245023145423199012==--