From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============6354995031644946265==" MIME-Version: 1.0 From: Harris, James R Subject: Re: [SPDK] SPDK + user space appliance Date: Mon, 07 May 2018 18:18:20 +0000 Message-ID: <7DF9BCBB-C791-4737-8527-D02A080907B7@intel.com> In-Reply-To: 1525716175.46960.59.camel@intel.com List-ID: To: spdk@lists.01.org --===============6354995031644946265== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable There are also calls such as spdk_call_unaffinitized() and spdk_unaffinitiz= e_thread() which have been added to enable cases where a bdev module may ne= ed to spawn non-polling threads and don=E2=80=99t want those threads to inh= erit the affinity of the calling thread. The SPDK rbd module currently use= s these (see git commit fa5206c4) since rbd_open is a blocking call. (Note= that librbd does now support rbd_aio_open which is better suited for SPDK.) -Jim On 5/7/18, 11:02 AM, "SPDK on behalf of Walker, Benjamin" wrote: Hi Shahar, = Thank you for submitting the patch. I've looked through it in detail an= d I think I understand the purpose of this code, but I'm going to explain it back= to you so you can correct me where I'm wrong. = I think this code solves two distinct problems: = 1) You need to forward I/O out of the bdev layer to some custom backend= , and you want the code that does that to live outside of the SPDK repository. = 2) Your custom back-end library isn't suitable for use in a run-to-comp= letion model. By that I mean that you can't just call your library directly on= the thread that originally receives the spdk_bdev_io request because your l= ibrary either blocks or generally takes too long to return from the submission= call (maybe it is doing inline compression or something). Instead, you need = to shuttle those requests off to separate threads for handling. = As far as point #1, today the SPDK build system does not nicely accommo= date bdev modules whose code lives outside of SPDK. SPDK expects them to be in lib/bdev/. However, that's a fairly straightforward change= to the build system and it's one we've been intending to make for some time. = For point #2, this is likely the case for a large number of storage bac= k-ends, but I think the proper way to solve it is probably back-end specific an= d not general purpose. As a counter-point, SPDK already integrates with a num= ber of third-party storage back-ends today (Ceph RBD, libiscsi, libaio, etc.) = and none of those ended up needing to pass messages to other threads. They all s= upport asynchronous operations, though. I could imagine writing a bdev module = that ultimately makes POSIX preadv calls, for instance. That would need to be implemented with a thread pool and each bdev_io gets funneled off to a = thread in the pool to perform the blocking operation. = Ok - I explained what I think I'm understanding. Now tell me where I we= nt wrong :) = Thanks, Ben = On Sun, 2018-05-06 at 10:32 +0000, Shahar Salzman wrote: > Hi, > = > I pushed the code for review, thanks Daniel for the help. > = > In a nutshell: > - bdev_user - an API for a user appliance to use spdk as an iSCSI/NVM= eF target > - bdev_user_example - reference application > - The API relies on rings in order to submit/complete IOs > - User appliance registers callbacks for submit_io (should we have > read/write/other instead?) > - User appliance registers its devices so that they may be added to an > existing namespace (I am using RPC to do the management) > = > Thanks, > Shahar > = > = > From: SPDK on behalf of Verkamp, Daniel= rkamp(a)intel.com> > Sent: Thursday, May 3, 2018 8:50 PM > To: Storage Performance Development Kit > Subject: Re: [SPDK] SPDK + user space appliance > = > Hi Shahar, > = > The target branch for the push should be =E2=80=98refs/for/master=E2= =80=99, not =E2=80=98master=E2=80=99 =E2=80=93 if > you configured a remote as specified in http://www.spdk.io/developmen= t/ it > should look like: > = > [remote "review"] > url =3D https://review.gerrithub.io/spdk/spdk > push =3D HEAD:refs/for/master > = > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Shahar S= alzman > Sent: Thursday, May 3, 2018 1:00 AM > To: Storage Performance Development Kit > Subject: Re: [SPDK] SPDK + user space appliance > = > Hi Ben, > = > I have the code ready for review (spdk/master on dpdk/18.02), but I d= o not > have push rights for gerrithub: > shahar.salzman(a)shahars-vm:~/Kaminario/git/spdk$ git push spdk-review > HEAD:master > Password for 'https://ShaharSalzman-K(a)review.gerrithub.io': = > Counting objects: 109, done. > Compressing objects: 100% (22/22), done. > Writing objects: 100% (22/22), 8.70 KiB | 0 bytes/s, done. > Total 22 (delta 14), reused 0 (delta 0) > remote: Resolving deltas: 100% (14/14) > remote: Branch refs/heads/master: > remote: You are not allowed to perform this operation. > remote: To push into this reference you need 'Push' rights. > remote: User: ShaharSalzman-K > remote: Please read the documentation and contact an administrator > remote: if you feel the configuration is incorrect > remote: Processing changes: refs: 1, done = > To https://ShaharSalzman-K(a)review.gerrithub.io/a/spdk/spdk > ! [remote rejected] HEAD -> master (prohibited by Gerrit: ref update= access > denied) > error: failed to push some refs to 'https://ShaharSalzman-K(a)review.= gerrithub.i > o/a/spdk/spdk' > = > Am I doing something incorrect, or is this just a permission issue? > = > Thanks, > Shahar > From: SPDK on behalf of Shahar Salzman = zman(a)kaminario.com> > Sent: Wednesday, April 25, 2018 9:02:38 AM > To: Storage Performance Development Kit > Subject: Re: [SPDK] SPDK + user space appliance > = > Hi Ben, > = > The code is currently working on v17.07, we are planning on bumping t= he > version to one of the latest stable versions (18.01?) + master. > It will take me (hopefully) a few days to update the code and have our > internal CI start running on this version, not sure it would be usefu= l, but I > can get our working 17.07 code (+ reference application) for review m= uch > faster. > What is the best course of action? > = > Shahar > From: SPDK on behalf of Walker, Benjami= n .walker(a)intel.com> > Sent: Tuesday, April 24, 2018 7:19:12 PM > To: Storage Performance Development Kit > Subject: Re: [SPDK] SPDK + user space appliance > = > Hi Shahar, > = > Would you be willing to submit your bdev module as a patch on GerritH= ub? That > way everyone can take a look and provide feedback. If you don=E2=80= =99t want it to run > the tests, you can put [RFC] and the beginning of the commit message. > = > Thanks, > Ben > = > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Shahar S= alzman > Sent: Monday, April 23, 2018 8:45 AM > To: spdk(a)lists.01.org > Subject: Re: [SPDK] SPDK + user space appliance > = > Hi Ben, > = > Bumping this thread since I've been having some new thoughts on the i= ssue now > that we are starting integration with newer spdk versions. > Unfortunately the merge isn't as smooth as I'd like it to be since th= e bdev > module is pretty tightly integrated into spdk, perhaps we made some f= alse > assumptions writing the module, but it seems some of the newer spdk f= eatures > are complicating the integration. > My question is, if this passthrough module is useful, wouldn't it be = better to > maintain it as part of spdk so that we can catch issues as soon as th= ey show > up? > We would be happy to help with maintaining this module, the module wi= th is > currently part of our CI with our "frozen" spdk version, but once int= egrated > into the newer version we choose, I'll add it to the CI our CI as wel= l. > = > Shahar > From: SPDK on behalf of Walker, Benjami= n .walker(a)intel.com> > Sent: Friday, February 2, 2018 11:43:58 PM > To: spdk(a)lists.01.org > Subject: Re: [SPDK] SPDK + user space appliance > = > On Thu, 2018-02-01 at 08:29 +0000, Shahar Salzman wrote: > > Hi Ben, > > = > > Would you also like to take a look at the bdev_user module? > > It still needs some patching (as some of the stuff is still hard co= ded), but > I > > think we can get most of it cleaned up in a couple of days. > > = > > In any case, is it the intention that the user write his own bdev m= odule, or > > would this user appliance glue be a useful generic module? > = > For existing storage stacks that serve block I/O, like the internals = of a SAN, > the idea is that you write your own bdev module to forward I/O coming= out of > the > SPDK bdev layer. Then you can use the SPDK iSCSI/NVMe-oF/vhost target= s mostly > as-is. > = > In some cases, the actual iSCSI/NVMe-oF/vhost target applications won= 't > integrate nicely directly into an existing storage application becaus= e they > spawn their own threads and allocate their own memory. To support tha= t, the > libraries may be consumed directly instead of the applications (lib/i= scsi, > lib/scsi, lib/nvmf, etc.). The libraries don't spawn any of their own= threads, > but instead rely on SPDK's abstractions in include/spdk/io_channel.h.= See > = > http://www.spdk.io/doc/concurrency.html > = > We don't currently have a way to write a custom bdev module that resi= des > outside > of the SPDK source tree, but it's very possible to add support for th= at. But > beyond that inconvenience (just drop your module in lib/bdev for now)= , writing > a > bdev module is the recommended way of interacting with the bottom end= of the > SPDK bdev layer. I think that's what you really want to be doing in y= our code, > from what I can tell. > = > I hope that helps! > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > = > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk = --===============6354995031644946265==--