From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============5719869775532570137==" MIME-Version: 1.0 From: Harris, James R Subject: Re: [SPDK] BDEV-IO Lifecycle - Need your input. Date: Mon, 09 Jul 2018 15:30:23 +0000 Message-ID: <4F90FCAF-FDFA-4AD4-828B-A395698463BF@intel.com> In-Reply-To: BN7PR06MB4033F4C2F83CA65463A038A892440@BN7PR06MB4033.namprd06.prod.outlook.com List-ID: To: spdk@lists.01.org --===============5719869775532570137== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi Sriram: Ben has some later patches in this series with some examples: https://review.gerrithub.io/#/c/spdk/spdk/+/386167/ - implements this zero = copy API in the simple =E2=80=98malloc=E2=80=99 bdev module https://review.gerrithub.io/#/c/spdk/spdk/+/416579/ - uses the zero copy AP= I in the SPDK bdevperf utility -Jim On 7/9/18, 3:34 AM, "Popuri, Sriram" wrote: Sorry was not focusing on this change. Just give me a day or two to get= back. From a quick glance I didn't understand how zcopy start/end fits into a= read/write life cycle. Is there an example on how zcopy start/end is consu= med or someone can do give me a quick dump on how its envisioned to be used. = Regards, ~Sriram = -----Original Message----- From: Meneghini, John = Sent: Friday, July 6, 2018 10:16 PM To: Walker, Benjamin ; Verkamp, Daniel ; Harris, James R ; Ka= ligotla, Srikanth Cc: raju.gottumukkala(a)broadcom.com; spdk(a)lists.01.org; Rodriguez, E= dwin ; Pai, Madhu ;= NGC-john.barnard-broadcom.com ; Popuri, Srira= m Subject: Re: BDEV-IO Lifecycle - Need your input. = I'm adding Sriram to this thread. Sriram is another NetApp engineer wh= o is working on this stuff internally. = /John = On 7/6/18, 12:41 PM, "Walker, Benjamin" w= rote: = Hi Srikanth, = I wanted to check in to see if you had any additional feedback on t= he zero copy operations in the bdev layer here: = https://review.gerrithub.io/#/c/spdk/spdk/+/386166/ = This does not address extending the lifetime of the bdev_io in the = NVMe-oF target, but I think this is looking like the right mechanism for th= e bdev layer. = Thanks, Ben = On Thu, 2018-06-21 at 18:11 -0700, Harris, James R wrote: > Thanks Srikanth. Sounds like the spdk_bdev_io pool sizing along = with > spdk_bdev_queue_io_wait() meets your needs then regarding the spd= k_bdev_io > memory. > = > Regarding zcopy_start/zcopy_end =E2=80=93 it looks like you=E2=80= =99ve already added a bunch > of comments to Ben=E2=80=99s patch on GerritHub. For now I=E2=80= =99d say let=E2=80=99s continue our > discussion there. I=E2=80=99ve responded to a couple of similar = questions there and > I=E2=80=99m sure Ben will have more replies tomorrow. > = > -Jim > = > = > From: "Kaligotla, Srikanth" > Date: Thursday, June 21, 2018 at 5:01 PM > To: James Harris , "Walker, Benjamin"= er(a)intel.com>, Daniel Verkamp > Cc: "raju.gottumukkala(a)broadcom.com" , > "Meneghini, John" , "Rodriguez, Edwi= n" z(a)netapp.com>, "Pai, Madhu" , "NGC= -john.barnard- > broadcom.com" , "spdk(a)lists.01.org= " org> > Subject: RE: BDEV-IO Lifecycle - Need your input. > = > Hi Jim, > = > I wish I joined the community meeting, I also missed the > spdk_bdev_queue_io_wait(). > = > So, there are 2 issues I am intending to solve, > = > 1. We want to ensure that an instance of bdev-io is acquire= d prior to or > along with I/O data buffer. This allows for better error handling= when bdev_io > pool is exhausted. Yes, we can solve it via sizing the pool right= . The changes > queue-io-wait also addresses the problem. > 2. Most importantly, by extending the life of bdev-io (Same= life span as > nvmf_request structure), abort use case and other use cases that = involve > cleaning up of IO data buffer are accomplished effectively. Let m= e elaborate; > bdev-io is the fabric that connects the nvme command and operatio= n with the > backend. The driver context and data buffer context is stored in = bdev-io. The > freeing up of bdev_io resource is pushed to the end so the I/O cl= eanup can > happen after controller has transmitted the data to the host. In = absence of > bdev_io, we will have end up adding more and more void context in= request > structure. Hence the push to extend the life of bdev_io. > = > I see Ben=E2=80=99s recent patch =E2=80=9Czcopy_start=E2=80=9D an= d =E2=80=9Czcopy_end=E2=80=9D; We can make that work > as long as the bdev-io allocated/acquired stays till the end. One= of the > challenges I see with that patch is defining a callback for the I= /O > submission. For instance, zopy_start will allocate bdev_io and su= bmits the I/O > to the bdev device. Underlying implementation can be synchronous = or can be > asynchronous. The callback for this submission should check to se= e if it is a > BUFFER-READY or PENDING and accordingly relay it back to transpor= t. Next phase > is actual I/O submission. Let=E2=80=99s say, it reuses the bdev-i= o obtained in ZCOPY- > START, now the callback should determine if it is a success or fa= ilure. > Finally when ZCOPY-END is invoked the supplied bdev_io will have = all necessary > data to release the WRITE buffer or unlock the read buffer based = on the > operation performed. > = > I hope I=E2=80=99m making sense. I guess, my effort is to extend = the lifespan of bdev- > io and let it continue to host the driver context and buffer cont= ext populated > during the BEGIN phase. > = > Regards, > Srikanth > = > From: Harris, James R = > Sent: Thursday, June 21, 2018 2:21 PM > To: Kaligotla, Srikanth ; Walker= , Benjamin jamin.walker(a)intel.com>; Verkamp, Daniel > Cc: raju.gottumukkala(a)broadcom.com; Meneghini, John >; Rodriguez, Edwin ; Pai, Madhu pp.com>; NGC-john.barnard-broadcom.com ; spdk(a)lists > .01.org > Subject: Re: BDEV-IO Lifecycle - Need your input. > = > Hi Srikanth, > = > Following up on this thread and the discussion in yesterday=E2=80= =99s community > meeting. > = > The recent spdk_bdev_queue_io_wait() changes allow an SPDK applic= ation to work > better in general when the spdk_bdev_io buffer pool is exhausted.= We still > need to make changes to the NVMe-oF target to use this new API bu= t that should > be done soon-ish. > = > With that in place, the spdk_bdev_io buffer pool itself can be co= nfigured up > or down when the application starts. Currently the default is 64K > spdk_bdev_io buffers. sizeof(struct spdk_bdev_io) =3D=3D 216 plu= s the per-IO > context size allocated for the bdev module. This can be up to 19= 2 bytes > (virtio bdev module) but is likely much smaller for you depending= on the > context size for your ontap bdev module. > = > Let=E2=80=99s assume your per-IO context size is 64 bytes. 64K x= (192 + 64) =3D 16MB. > = > I=E2=80=99m not sure how many spdk_bdev_io you need in flight at = any given time. 64K > seems like a lot but I=E2=80=99d be curious to hear your thoughts= on this. If this is > the right number, then worst case, there would be about 16MB of D= RAM that > would sit unused if the NVMe-oF target in your system was not act= ive. Is that > too burdensome for your application? > = > Thanks, > = > -Jim > = > = > = > From: "Kaligotla, Srikanth" > Date: Wednesday, June 20, 2018 at 12:47 PM > To: "Walker, Benjamin" , James Harri= s is(a)intel.com>, Daniel Verkamp > Cc: "raju.gottumukkala(a)broadcom.com" , > "Meneghini, John" , "Rodriguez, Edwi= n" z(a)netapp.com>, "Pai, Madhu" , "NGC= -john.barnard- > broadcom.com" , "spdk(a)lists.01.org= " org> > Subject: RE: BDEV-IO Lifecycle - Need your input. > = > Hello, > = > First revision of changes to extend the lifecycle of bdev_io are = available for > review. I would like to solicit your input on the proposed API/Co= de flow > changes. > = > https://review.gerrithub.io/c/spdk/spdk/+/415860 > = > Thanks, > Srikanth > = > = > From: "Kaligotla, Srikanth" > Date: Friday, May 11, 2018 at 2:27 PM > To: "Walker, Benjamin" , "Harris, Ja= mes R" .harris(a)intel.com> > Cc: "raju.gottumukkala(a)broadcom.com" , > "Meneghini, John" , "Rodriguez, Edwi= n" z(a)netapp.com>, "Pai, Madhu" , "NGC= -john.barnard- > broadcom.com" , "spdk(a)lists.01.org= " org> > Subject: RE: BDEV-IO Lifecycle - Need your input. > = > CC: List > = > Hi Ben, > = > Your proposal to interface with the backend to acquire and releas= e buffers is > good. You have accurately stated that the challenge is in develop= ing intuitive > semantics. And that has been my struggle. To me, there are two pr= oblem > statements; > = > 1. It is expected that bdev_io pool is sized correctly so t= he call to > get bdev_io succeeds. Failure to acquire bdev_io will result in D= EVICE-ERROR. > The transport is already capable of handling temporary memory fai= lures by > moving the request to PENDING-Queue. Hence the proposal to change= the bdev_io > lifecycle and perhaps connect it with the spdk_nvmf_request objec= t. Thus all > buffer needs are addressed at the beginning of I/O request. > 2. I/O Data buffers are sourced and managed by the backend.= One of the > challenges I see with your proposed interface is the lack of deta= ils like if > resource is being acquired for a READ operation or WRITE operatio= n. The > handling is quite different in each case. Since bdev_io->type is = overloaded > the type of I/O operation is lost. I suppose one can cast the cb_= arg > (nvmf_request) and then proceed. Also, the bdev_io should be pres= ent to > RELEASE the buffer. Zero copy semantics warrants that data buffer= stays until > controller-to-host has occurred. In other words, bdev_io lives ti= ll REQUEST > come to COMPLETE state. > = > What are your thoughts in introducing spdk_bdev_init() and spdk_b= dev_fini() as > an alternative approach to extend the lifecyle of bdev_io and all= ow data > buffer management via bdev fn_table ? > = > I hope I=E2=80=99m making sense=E2=80=A6 > = > Thanks, > Srikanth > = > From: Walker, Benjamin = > Sent: Friday, May 11, 2018 12:28 PM > To: Harris, James R ; Kaligotla, Srik= anth Kaligotla(a)netapp.com> > Cc: raju.gottumukkala(a)broadcom.com; Meneghini, John >; Rodriguez, Edwin ; Pai, Madhu pp.com>; NGC-john.barnard-broadcom.com > Subject: Re: BDEV-IO Lifecycle - Need your input. > = > Hi Srikanth, > = > Yes - we'll need to introduce some way to acquire and release buf= fers from the > bdev layer earlier on in the state machine that processes an NVMe= -oF request. > I've had this patch out for review for several months as a propos= al for this > scenario: > = > https://review.gerrithub.io/#/c/spdk/spdk/+/386166/ > = > It doesn't pass the tests - it's just a proposal for the interfac= e. 90% of the > challenge here is in developing intuitive semantics. > = > Thanks, > Ben > = > P.S. This is the kind of discussion that would fit perfectly on t= he mailing > list. > = > On Wed, 2018-05-09 at 20:35 +0000, Kaligotla, Srikanth wrote: > > Hi Ben, Hi James, > > = > > I would like to solicit opinions on the lifecycle for bdev-io r= esource > > object. Attached is the image of RDMA state machine in its curr= ent > > implementation. When the REQUEST enters the NEED-BUFFER state, = the buffers > > necessary for carrying out I/O operation are allocated/acquired= from the > > memory pool. An instance of BDEV-IO comes into existence after = REQUEST > > reaches READY-TO-EXECUTE state. The BDEV-IO is teared down as s= oon the > > backend returns. From a BDEV perspective, BDEV-IO is simply a t= ranslation > > unit that facilitates I/O buffers from the backend. The driver = context > > embedded within bdev_io holds great deal of information pertain= ing to the > > I/O under execution. It assists in error handling, dereferencin= g the buffers > > upon I/O completion and in abort handling. In summary, the bdev= _io stays > > alive until request has come to COMPLETE state. I=E2=80=99d lik= e to hear peoples > > thoughts in introducing the plumbing to acquire BDEV-IO resourc= e in REQUEST- > > NEED-BUFFER state and release it in REQUEST-COMPLETE state. I = will shortly > > have patch available for review that introduces spdk_bdev_init = and > > spdk_bdev_fini which in turn invokes the corresponding bdev fn_= table to > > initialize/cleanup. = > > = > > I wanted to use this email to communicate our intent and solici= t your > > feedback. We have a working implementation of the above proposa= l and prior > > to pushing it upstream for review would like to hear your thoug= hts. These > > proposed changes to upstream are a result of FC transport work = in > > collaboration with the Broadcom team who are also copied to thi= s mail. > > Myself and John will be at SPDK Dev conference and if required = we can > > elaborate further on this proposal. > > = > > Thanks, > > Srikanth > = > = = = = --===============5719869775532570137==--