From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============3986139499050620238==" MIME-Version: 1.0 From: Walker, Benjamin Subject: Re: [SPDK] Integrate OpenStack/Cyborg into SPDK Architecture Date: Tue, 19 Jun 2018 16:42:54 +0000 Message-ID: In-Reply-To: CAHZqm+Xzd9zCVLx_EqfJ28kQRQx4O9hbGF5xzGgW2PWL084cmg@mail.gmail.com List-ID: To: spdk@lists.01.org --===============3986139499050620238== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable I've been mulling over this discussion for the past few days. I'm certainly always supportive of projects that integrate SPDK, and this situation is no different. However, I can't wrap my head around exactly how "NVMe" or "NVMe= -oF" can be thought of as an accelerator. Because of that, I'm not in a position= to know when to point people at the Cyborg project as opposed to Cinder if the= y're interested in using SPDK with OpenStack. So I'd like to gain a deeper understanding here, if you'll all help me do that. My understanding is that Cyborg is a framework for discovering available "accelerators". To me, an accelerator is some code library or specialized p= iece of hardware that performs some computation more quickly than a simple algor= ithm running on the CPU. This could range from FPGAs to GPUs to even specialized implementations of algorithms that still run on the CPU, but take advantage= of instructions or designs that the compilers are unlikely to emit from plain = old C code (like ISA-L). To me, traditional NVMe devices do not fall into this category. They don't accelerate some computation, but rather are simply a block storage device. Further, accelerators are stateless - they take some input and produce some output in a repeatable fashion. NVMe devices, in contrast, are stateful - t= heir entire purpose is to persistently store state. So what, specifically, is Cy= borg exposing about the device that Cinder cannot? Note that there are "NVMe" devices on the market that really are accelerato= rs - they contain no persistent storage and are only using NVMe as a convenient interface. They act as accelerators by allowing the user to write to a cert= ain range of blocks, then read the data back from that location and some operat= ion such as encryption or compression will have been performed on it. These dev= ices are indeed accelerators and should be exposed through Cyborg. If you're only talking specifically about these devices (which are a very niche market as = of this writing), then I understand fully what you're doing here and we can mo= ve forward. Thanks, Ben On Sun, 2018-06-17 at 08:20 +0800, Zhipeng Huang wrote: > Thanks Paul, > = > Will there is no extra features per se at the moment, but we will gradual= ly > rolls that out hopefully. The current spdk driver code could be found at = https > ://github.com/openstack/cyborg/blob/master/cyborg/accelerator/drivers/spd= k/nvm > f/nvmf.py = > = > Well the purpose of the discussion is actually quite simple, we suggest t= o add > Cyborg as another project like Cinder which supports spdk in the SPDK > architecture figure :) I think our work in Cyborg is good for SPDK's ecos= ystem > and would be great to acknowledge that :) > = > Sorry for the many lengthy emails :) > = > = > On Sat, Jun 16, 2018 at 11:15 PM Luse, Paul E w= rote: > > Thanks for the information! Are there currently any Cyborg drivers (is = that > > the right term?) that implement NVMeOF? It would be great to take a loo= k at > > one to help clear things up a bit =E2=80=93 what I=E2=80=99m still not = seeing is exactly > > what =E2=80=9Cextra capabilities=E2=80=9D you mention below that could = be exposed/exploited > > via Cyborg & SPDK vs Cinder & SPDK. > > = > > Also, assuming that nothing else is required from the SPDK > > codebase/community, is the purpose of this email chain just to > > inform/educate and help enlist the SPDK community in promoting its use = in > > Cyborg? > > = > > Note that I=E2=80=99m on holiday from now until July 2 and starting Mon= afternoon > > will not have access to email until I get back so when you don=E2=80=99= t get any > > kind of response from me, that=E2=80=99s why J Hopefully others will ca= rry the > > conversation forward as I know others are interested as well and fully > > appreciate all of Helloway=E2=80=99s interest in SPDK thus far. I=E2= =80=99ll definitely > > touch base when I return, thanks again! > > = > > -Paul > > = > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Zhipeng Hu= ang > > Sent: Friday, June 15, 2018 6:27 PM > > To: Storage Performance Development Kit > > Subject: Re: [SPDK] Integrate OpenStack/Cyborg into SPDK Architecture > > = > > Hi Maciek and Paul, > > = > > I'm the current project lead of Cyborg so let me try to answer your > > questions, maybe hellowway did not explain well before. > > = > > Cyborg[0] is a general management framework for accelerators (GPU. FPGA, > > ASIC, NVMe/NOF, ...). We actually worked with Moshe's team in the proje= ct's > > early days :) . The support of spdk driver has already landed in cyborg > > (although still very premature). > > = > > It is important to notice that Cyborg does not provide volume management > > which is the job of Cinder. Cyborg provide management on NVMe/NVMe Over > > Fabric SSDs, meaning leveraging information that could be attained via = the > > Bus (and the NVMe/NOF protocol). A similar comparison could be found in > > Cyborg management of FPGA/GPU. This could enable many great features > > developed in NVMe protocol could be exposed via Cyborg to support more > > granulized scheduling if the user want, instead of just being treated a= s a > > block device without difference to traditional SSDs. > > = > > Therefore Cyborg is working alongside of Nova and Cinder. An example of= how > > Cyborg work with Nova on NVMe devices could be found in [1] starting 27= :13, > > which is also proposed from Intel developers. In essence, Cyborg helps = Nova > > get knowledge of NVMe SSDs for its various extra capabilities compared = to > > normal SSD (i.e, as an accelerator), Nova scheduler then will select a = node > > with such capability if the user desires to spawn a VM, and then Cinder= will > > just do its job on volume management. > > = > > So on this note, Cyborg is working with Nova and Cinder, the relationsh= ip is > > complimentary. Cyborg will interact with Nova through Placement, there = is no > > need at the moment for Cyborg to interact with Cinder. It could still w= ork > > without Cyborg in the picture for sure, but NVMe SSD will be seen just = as a > > normal block device by Nova and Cinder, and no advanced scheduling coul= d be > > performed. > > = > > Re Paul's question: > > - Just a good Cyborg SPDK driver could make it work in OpenStack > > - No additional requirement on SPDK community itself > > - No additional tweaks in SPDK needed specifically for Cyborg to work w= ith > > it. > > = > > OpenSDS is just another option, since it supports capability report > > functionality, it could gather information from Cyborg to make more > > granulized scheduling, unlike Cinder which does not get input from Cybo= rg > > and just perform regular volume management. There is no good or bad to > > compare two solutions. It depends on user's requirement. > > = > > I hope this writing could clear things up :) > > = > > [0] https://wiki.openstack.org/wiki/Cyborg > > [1] https://www.youtube.com/watch?v=3Dq84Q-6FSXts = > > = > > = > > On Fri, Jun 15, 2018 at 10:28 PM Szwed, Maciej > > wrote: > > > Hi Helloway, > > > I=E2=80=99ve been working on Cinder drivers (volume and target) for S= PDK. I=E2=80=99m not > > > familiar with Cyborg. Does Cyborg have capabilities to manage volumes= ? How > > > does it interact with Nova =E2=80=93 how does it provide storage to c= ompute nodes? > > > = > > > Thanks, > > > Maciek > > > = > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Pa= ul E > > > Sent: Thursday, June 14, 2018 5:27 AM > > > To: Storage Performance Development Kit > > > Subject: Re: [SPDK] Integrate OpenStack/Cyborg into SPDK Architecture > > > = > > > Hi Helloway, > > > = > > > No problem. Cinder is the block storage provisioning project for > > > Openstack. A VM needing block storage interacts with Cinder to request > > > what it needs and storage providers that have written provider (aka > > > drivers) for Cinders are matched up based on requests. Here=E2=80=99s= the overview > > > from the project: https://docs.openstack.org/newton/install-guide-ubu= ntu/c > > > ommon/get-started-block-storage.html and this is by far the most comm= on > > > way that I=E2=80=99m aware of to enable a new block storage applicati= on in > > > OpenStack. > > > = > > > I haven=E2=80=99t worked in that world in a few years such that Cybor= g and OpenSDS > > > have been introduced since I was active but as far as I know Cinder is > > > still the best place to start introducing SPDK based block storage in= to > > > the Openstack cloud. I do have some OpenSDS contacts, and still some > > > friends who work on Cinder. Let me ask around a little, note that I = will > > > be out of the office for 2 weeks of =E2=80=9Cdisconnected=E2=80=9D va= cation after this > > > Friday but I=E2=80=99ll try and get a bit more info before then and g= et back to > > > ya. > > > = > > > Anyone else out there feel free to chime in if you have more info on > > > Cyborg+OpenSDS vs Cinder as it pertains to this discussion. Once we f= igure > > > that out, the questions below are still very relevant for next steps: > > > = > > > - what else is required to be pushed into OpenStack for this to work > > > - is anything required in the SPDK repo for this to work > > > - how will the necessary SPDK components be associated with the VM in > > > question and subsequently configured > > > = > > > Thanks! > > > Paul > > > = > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of helloway > > > Sent: Wednesday, June 13, 2018 7:47 PM > > > To: Storage Performance Development Kit > > > Cc: Storage Performance Development Kit > > > Subject: Re: [SPDK] Integrate OpenStack/Cyborg into SPDK Architecture > > > = > > > Hi Paul, > > > = > > > I am sorry for that I'm not an expert on Cinder, I=E2=80=98ll try to = tell you what > > > I think. If I was wrong, please correct me. From my perspective, Cind= er > > > cares more about the capacity of the pool whose pool capabilities > > > reporting interface is fixed, whereas, in addition to the capacity, C= yborg > > > also cares about the fine-grained accelerator capabilities (e.g. iops, > > > queue, etc.). These capabilities, reported from the Cyborg, can be > > > dynamic configured and handled through the OpenSDS=E2=80=99 profile. = For this > > > reason, it provides a more flexible and simple configuration which ca= n be > > > called conveniently. > > > = > > > Thx, > > > Helloway > > > = > > > On 06/13/2018 22:46=EF=BC=8CLuse, Paul E wro= te=EF=BC=9A > > > > Hi Helloway, > > > > = > > > > That=E2=80=99s a great start but I still have the same open questio= ns below, > > > > maybe you can try and address those directly? Also, below is the li= nk > > > > for adding SPDK based NVMeOF as a Cinder plug-in. In addition to t= he > > > > question below can you please explain for everyone how you see the > > > > approach of using Cyborg and OpenSDS compares with the seemingly si= mpler > > > > approach of providing a Cinder plug-in? > > > > = > > > > https://review.openstack.org/#/c/564229/ > > > > = > > > > Thanks!! > > > > Paul > > > > = > > > > = > > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of hellow= ay > > > > Sent: Tuesday, June 12, 2018 5:22 PM > > > > To: Storage Performance Development Kit > > > > Cc: Storage Performance Development Kit > > > > Subject: Re: [SPDK] Integrate OpenStack/Cyborg into SPDK Architectu= re > > > > = > > > > Hi Paul, > > > > Thank you for your response, I have submitted a trello [0] titled > > > > =E2=80=9CIntegrate OpenStack/Cyborg into SPDK Architecture=E2=80= =9D. I am trying to > > > > answer your questions from this trello, do I make sense? I really h= ope > > > > to receive your feedback. > > > > = > > > > [0]https://trello.com/c/QfSAkLSS/121-integrate-openstack-cyborg-int= o-spd > > > > k-architecture = > > > > = > > > > Thx, > > > > Helloway > > > > On 06/12/2018 08:16=EF=BC=8CLuse, Paul E w= rote=EF=BC=9A > > > > > Hi Helloway, > > > > > = > > > > > I was actually just wondering what had happened with this. Looki= ng at > > > > > the OpenStack patch it looks like it=E2=80=99s close to landing. = Somewhere out > > > > > there we have a Cinder driver that=E2=80=99s also getting fairly = close I > > > > > believe so for sure integration with OpenStack is interesting to = many > > > > > in the community. > > > > > = > > > > > Would you be able to summarize more specifically how your patch w= ould > > > > > work once it lands? I of course see your high level description b= elow > > > > > that some questions that I have, I=E2=80=99m assuming others as w= ell, include: > > > > > = > > > > > - what else is required to be pushed into OpenStack for this to w= ork > > > > > - is anything required in the SPDK repo for this to work > > > > > - how will the necessary SPDK components be associated with the V= M in > > > > > question and subsequently configured > > > > > = > > > > > Thanks for continuing to work on this! > > > > > = > > > > > -Paul > > > > > = > > > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of hell= oway > > > > > Sent: Monday, June 11, 2018 1:35 AM > > > > > To: Storage Performance Development Kit > > > > > Subject: [SPDK] Integrate OpenStack/Cyborg into SPDK Architecture > > > > > = > > > > > Hi Jim and all, > > > > > = > > > > > Do you know OpenStack/Cyborg? It is OpenStack acceleration service > > > > > which provides a management framework for accelerator devices (e.= g. > > > > > FPGA, GPU, NVMe SSD). There is a strong demand for OpenStack to > > > > > support hardware accelerated devices in a dynamic model(mentioned= in > > > > > OpenStack Summit Vancouver 2018 [0]). = > > > > > = > > > > > For this reason, we can use Cyborg to interactive with nvmf_tgt to > > > > > realize the management of the user space accelerator NVMe SSD dev= ice, > > > > > which can badly promote the efficiency. It is worth mentioning th= at > > > > > the Cyborg_SPDK_Driver I summitted has been merged into the OpenS= tack > > > > > version Q [1]. The driver can report the detailed information of = the > > > > > device to the Cyborg agent. When user requests a vm with a user s= pace > > > > > NVMe SSD, Cyborg agent will update the Nova/Placement inventory on > > > > > available NVMe devices. This is a complete process to describe the > > > > > connection of Cyborg and SPDK. > > > > > = > > > > > I wonder whether you guys are interested in integrating > > > > > OpenStack/Cyborg into SPDK architecture? Do I make sense? Please = let > > > > > me know what your thoughts. > > > > > = > > > > > [0]https://www.openstack.org/videos/vancouver-2018/optimized-hpca= i-clo > > > > > ud-with-openstack-acceleration-service-and-composable-hardware > > > > > [1]https://review.openstack.org/#/c/538164/ > > > > > = > > > > > = > > > > > Thx, > > > > > Helloway > > > = > > > _______________________________________________ > > > SPDK mailing list > > > SPDK(a)lists.01.org > > > https://lists.01.org/mailman/listinfo/spdk > > = > > = > > -- > > Zhipeng (Howard) Huang > > = > > Standard Engineer > > IT Standard & Patent/IT Product Line > > Huawei Technologies Co,. Ltd > > Email: huangzhipeng(a)huawei.com > > Office: Huawei Industrial Base, Longgang, Shenzhen > > = > > (Previous) > > Research Assistant > > Mobile Ad-Hoc Network Lab, Calit2 > > University of California, Irvine > > Email: zhipengh(a)uci.edu > > Office: Calit2 Building Room 2402 > > = > > OpenStack, OPNFV, OpenDaylight, OpenCompute Aficionado > > _______________________________________________ > > SPDK mailing list > > SPDK(a)lists.01.org > > https://lists.01.org/mailman/listinfo/spdk > = > = > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk --===============3986139499050620238==--