From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============0941592821696545155==" MIME-Version: 1.0 From: Walker, Benjamin Subject: Re: [SPDK] SPDK pooled volume module in bdev Date: Tue, 19 Jun 2018 16:51:33 +0000 Message-ID: In-Reply-To: CO2PR04MB2248024563A69C82959B268DBF700@CO2PR04MB2248.namprd04.prod.outlook.com List-ID: To: spdk@lists.01.org --===============0941592821696545155== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi all, SPDK's virtual block devices (vbdevs) are a framework for creating an I/O processing pipeline by stacking block devices (bdevs) on top of one another. Each block device has a type and has internal logic implemented by a bdev module. For example, there is an NVMe bdev, which routes I/O to NVMe device= s, and there is also a logical volume (lvol) bdev, which carves up an underlyi= ng bdev into many smaller bdevs dynamically. The (v)bdev layer does not contain any actual I/O processing logic in and of itself - it's just the framework. It delegates to bdev modules to implement= the actual logic. So the bdev layer today does not implement concatenation or anything of that nature. What Kunal is implementing here is a bdev module that implements RAID0. It stripes the RAID volume across N underlying bdevs and exposes a single bdev= on top. Thanks, Ben On Tue, 2018-06-19 at 03:23 +0000, Senthil Kumar Veluswamy wrote: > Hi Kunal, > >> I am assuming here you are asking about vbdev concatenation vs PV= OL > concatenation only. I don=E2=80=99t think vbdev is supporting concatenati= on. > = > I=E2=80=99m going thru the Paul=E2=80=99s presentation from last SPDK US = submit on =E2=80=9CVirtual > Bdevs: The secret to customizing SPDK=E2=80=9D and found below slides. > = > = > = > As you can see, from VBDEV explanation it looks very similar to PVOL . Th= is is > where the confusion is. If I understand it correctly, even VBDEV can > concatenate more than one BDEVs. > = > May be Paul can help here from VBDEV perspective.. > = > Hi Paul, > Can you please help here to understand the use-cases for VBDEV and P= VOLs? > & How they are architecturally different and how they can co-exists? > = > Thank you. > = > Regards, > Senthil Kumar V. > = > From: SPDK On Behalf Of Sablok, Kunal > Sent: Monday, June 18, 2018 10:12 PM > To: Senthil Kumar Veluswamy > Cc: Storage Performance Development Kit > Subject: Re: [SPDK] SPDK pooled volume module in bdev > = > Hi Senthil, > Please find inline. > = > Regards, > Kunal > = > From: Senthil Kumar Veluswamy [mailto:Senthil.Kumar.Veluswamy(a)wdc.com] = > Sent: Sunday, June 17, 2018 10:07 AM > To: Sablok, Kunal > Cc: Storage Performance Development Kit > Subject: RE: SPDK pooled volume module in bdev > = > Hi Kunal, > Thank you for the explanation. I=E2=80=99ve few more questions. My r= esponse > inline tagged with =E2=80=9C[Senthil-1]=E2=80=9D. > = > Regards, > Senthil Kumar V. > = > From: Sablok, Kunal = > Sent: Friday, June 15, 2018 10:40 PM > To: Senthil Kumar Veluswamy > Cc: Storage Performance Development Kit > Subject: RE: SPDK pooled volume module in bdev > = > Hi Senthil, > Please find inline. > = > Regards, > Kunal > = > From: Senthil Kumar Veluswamy [mailto:Senthil.Kumar.Veluswamy(a)wdc.com] = > Sent: Thursday, June 14, 2018 12:20 PM > To: Sablok, Kunal > Cc: Storage Performance Development Kit > Subject: RE: SPDK pooled volume module in bdev > = > Hi Kunal, > I=E2=80=99m trying understand the pVol architecture and have few que= stions. > = > 1. How this pvol model co-exists with vbdev that is present today? > [Kunal] In current SPDK, when any bdev is getting unregistered, > spdk_bdev_unregister() removes all the underlying base bdevs from vbdev b= ut > for PVOL scenario (since it is similar to Raid0 device), only PVOL bdev g= ets > unregistered and it still needs to keep the information of current base > devices which are still claimed. PVOL needs to keep many more information > apart from base_bdev reference per base_bdev level like base_bdev's > descriptor, some flags for hotplug scenarios etc[will upload in hotplug > review] which it is managing itself in pvol module. > = > During hotplug scenarios, when one of the base bdev of PVOL is removed and > PVOL gets its notification, it un-registers the PVOL from bdev layer sinc= e all > underlying base_bdevs are not present (but bdev layer implementation also > removes all the base bdevs in unregister call) but in this case also it s= till > needs to keep the base bdevs and other information of the base bdevs whic= h are > still present and claimed by PVOL. When the removed base_bdev is added ag= ain, > PVOL gets the notification via examine() and it will claim only that base= _bdev > now, all other base_bdevs were already present and so PVOL gets registered > again with bdev layer. > = > Base bdevs and its information can only be removed from PVOL bdev if below > layer like NVMe notifies the PVOL explicitly of base device removal. So, = idea > is to hold the base_bdevs in PVOL which are still present, claimed and not > removed for various reasons, so as of now PVOL is using spdk_bdev_registe= r() > to register itself with bdev layer, and other modules like LVS/LVOL can c= reate > logical volumes over PVOL [physical volume]. > = > 2. If I understand correctly, vbdev also allows to concatenate more = than > one base physical bdevs and used for creating logical volume. So shall we= say > pvol is an extension of vbdev? > [Kunal] We can say it is an extension of vbdev but it maintains more > information than just base_bdevs in its own module itself to have better > control as explained above. > = > [Senthil-1] Since Pvol maintains more info of base_bdevs than vbdev, can = we > say that PVol is a superset of vbdev? If so, can you give an example of t= heir > data structure mappings? > [Kunal] PVOL maintains the base bdev information in its module itself and > registers with bdev layer via spdk_bdev_register(). So there is no mapping > exists between PVOL and vbdev. However once PVOL is created with 1 or more > NVMe bdevs, LVS/LVOl can be created over PVOL itself. > = > [Senthil-1] Would it possible for you to explain the diff b/w PVol & vbde= v by > their real-time use-cases? Like when PVol will be used vs vbdev. I assumi= ng > both vbdev and PVol are going to coexist. > [Kunal] I am assuming here you are asking about vbdev concatenation vs PV= OL > concatenation only. I don=E2=80=99t think vbdev is supporting concatenati= on. For that > once code is pushed, PVOL will be used for striping across drives which w= ill > have IO split logic to split IOs and submit to underlying nvme bdevs. To > create PVOL across 1 or more base nvme bdevs, either RPC command can be u= sed > or pvol config file can be updated with the details to create PVOL during > bootup. Once PVOL bdev is created, other modules get examine calls for pv= ol > bdev also. LVS and LVOL can be create over PVOL and exposed to initiator = via > subsystems. > = > 3. It would be good if you could modify your =E2=80=9CPooled volume= =E2=80=9D PPT slides > to show how vbdev fits in there. > = > Regards, > Senthil Kumar V. > = > From: SPDK On Behalf Of Luse, Paul E > Sent: Friday, May 18, 2018 12:15 AM > To: Storage Performance Development Kit > Subject: Re: [SPDK] SPDK pooled volume module in bdev > = > Excellent, thanks Kunal! There=E2=80=99s a lot to look at here and with = Fenggang=E2=80=99s > aggregation patch as well. I encourage everyone to take a look at both (= and I > promise I will soon too after post-summit catch-up stuff) > = > J > = > = > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sablok, Kunal > Sent: Sunday, May 13, 2018 9:51 AM > To: spdk(a)lists.01.org > Subject: [SPDK] SPDK pooled volume module in bdev > = > Hi, > SPDK pooled volume (PVOL) module is a new bdev module which is responsibl= e for > striping various NVMe devices and expose the pooled volume to bdev layer = which > would enhance the performance and capacity. It can support theoretically = 255 > base devices which can be easily enhanced if required (currently it is be= ing > tested max upto 8 base devices). Multiple strip sizes like 32KB, 64KB, 12= 8KB, > 256KB, 512KB etc is supported. New RPC commands like "create pvol", "dest= roy > pvol" and "get pvols" are introduced to configure pooled volume dynamical= ly in > a running SPDK system. > Please find attached more information in a ppt with system end to end tes= ting > done on this module in SPDK stack in SPDK multi-core environment. > Please find below gerrithub review details: > https://review.gerrithub.io/#/c/spdk/spdk/+/410484/ > = > = > Regards, > Kunal > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk --===============0941592821696545155==--