Re: [SPDK] SPDK pooled volume module in bdev

[Walker, Benjamin <benjamin.walker at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 8712 bytes --]

Hi all,

SPDK's virtual block devices (vbdevs) are a framework for creating an I/O
processing pipeline by stacking block devices (bdevs) on top of one another.
Each block device has a type and has internal logic implemented by a bdev
module. For example, there is an NVMe bdev, which routes I/O to NVMe devices,
and there is also a logical volume (lvol) bdev, which carves up an underlying
bdev into many smaller bdevs dynamically.

The (v)bdev layer does not contain any actual I/O processing logic in and of
itself - it's just the framework. It delegates to bdev modules to implement the
actual logic. So the bdev layer today does not implement concatenation or
anything of that nature.

What Kunal is implementing here is a bdev module that implements RAID0. It
stripes the RAID volume across N underlying bdevs and exposes a single bdev on
top.

Thanks,
Ben

On Tue, 2018-06-19 at 03:23 +0000, Senthil Kumar Veluswamy wrote:
> Hi Kunal,
>      >> I am assuming here you are asking about vbdev concatenation vs PVOL
> concatenation only. I don’t think vbdev is supporting concatenation.
>  
> I’m going thru the Paul’s presentation from last SPDK US submit on “Virtual
> Bdevs: The secret to customizing SPDK” and found below slides.
>  
>      
>  
> As you can see, from VBDEV explanation it looks very similar to PVOL . This is
> where the confusion is. If I understand it correctly, even VBDEV can
> concatenate more than one BDEVs.
>  
> May be Paul can help here from VBDEV perspective..
>  
> Hi Paul,
>      Can you please help here to understand the use-cases for VBDEV and PVOLs?
> & How they are architecturally different and how they can co-exists?
>  
> Thank you.
>  
> Regards,
> Senthil Kumar V.
>  
> From: SPDK <spdk-bounces(a)lists.01.org> On Behalf Of Sablok, Kunal
> Sent: Monday, June 18, 2018 10:12 PM
> To: Senthil Kumar Veluswamy <Senthil.Kumar.Veluswamy(a)wdc.com>
> Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] SPDK pooled volume module in bdev
>  
> Hi Senthil,
> Please find inline.
>  
> Regards,
> Kunal
>  
> From: Senthil Kumar Veluswamy [mailto:Senthil.Kumar.Veluswamy(a)wdc.com] 
> Sent: Sunday, June 17, 2018 10:07 AM
> To: Sablok, Kunal <kunal.sablok(a)intel.com>
> Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: RE: SPDK pooled volume module in bdev
>  
> Hi Kunal,
>      Thank you for the explanation. I’ve few more questions. My response
> inline tagged with “[Senthil-1]”.
>  
> Regards,
> Senthil Kumar V.
>  
> From: Sablok, Kunal <kunal.sablok(a)intel.com> 
> Sent: Friday, June 15, 2018 10:40 PM
> To: Senthil Kumar Veluswamy <Senthil.Kumar.Veluswamy(a)wdc.com>
> Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: RE: SPDK pooled volume module in bdev
>  
> Hi Senthil,
> Please find inline.
>  
> Regards,
> Kunal
>  
> From: Senthil Kumar Veluswamy [mailto:Senthil.Kumar.Veluswamy(a)wdc.com] 
> Sent: Thursday, June 14, 2018 12:20 PM
> To: Sablok, Kunal <kunal.sablok(a)intel.com>
> Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: RE: SPDK pooled volume module in bdev
>  
> Hi Kunal,
>      I’m trying understand the pVol architecture and have few questions.
>  
> 1.      How this pvol model co-exists with vbdev that is present today?
> [Kunal] In current SPDK, when any bdev is getting unregistered,
> spdk_bdev_unregister() removes all the underlying base bdevs from vbdev but
> for PVOL scenario (since it is similar to Raid0 device), only PVOL bdev gets
> unregistered and it still needs to keep the information of current base
> devices which are still claimed. PVOL needs to keep many more information
> apart from base_bdev reference per base_bdev level like base_bdev's
> descriptor, some flags for hotplug scenarios etc[will upload in hotplug
> review] which it is managing itself in pvol module.
>  
> During hotplug scenarios, when one of the base bdev of PVOL is removed and
> PVOL gets its notification, it un-registers the PVOL from bdev layer since all
> underlying base_bdevs are not present (but bdev layer implementation also
> removes all the base bdevs in unregister call) but in this case also it still
> needs to keep the base bdevs and other information of the base bdevs which are
> still present and claimed by PVOL. When the removed base_bdev is added again,
> PVOL gets the notification via examine() and it will claim only that base_bdev
> now, all other base_bdevs were already present and so PVOL gets registered
> again with bdev layer.
>  
> Base bdevs and its information can only be removed from PVOL bdev if below
> layer like NVMe notifies the PVOL explicitly of base device removal. So, idea
> is to hold the base_bdevs in PVOL which are still present, claimed and not
> removed for various reasons, so as of now PVOL is using spdk_bdev_register()
> to register itself with bdev layer, and other modules like LVS/LVOL can create
> logical volumes over PVOL [physical volume].
>  
> 2.      If I understand correctly, vbdev also allows to concatenate more than
> one base physical bdevs and used for creating logical volume. So shall we say
> pvol is an extension of vbdev?
> [Kunal] We can say it is an extension of vbdev but it maintains more
> information than just base_bdevs in its own module itself to have better
> control as explained above.
>  
> [Senthil-1] Since Pvol maintains more info of base_bdevs than vbdev, can we
> say that PVol is a superset of vbdev? If so, can you give an example of their
> data structure mappings?
> [Kunal] PVOL maintains the base bdev information in its module itself and
> registers with bdev layer via spdk_bdev_register(). So there is no mapping
> exists between PVOL and vbdev. However once PVOL is created with 1 or more
> NVMe bdevs, LVS/LVOl can be created over PVOL itself.
>  
> [Senthil-1] Would it possible for you to explain the diff b/w PVol & vbdev by
> their real-time use-cases? Like when PVol will be used vs vbdev. I assuming
> both vbdev and PVol are going to coexist.
> [Kunal] I am assuming here you are asking about vbdev concatenation vs PVOL
> concatenation only. I don’t think vbdev is supporting concatenation. For that
> once code is pushed, PVOL will be used for striping across drives which will
> have IO split logic to split IOs and submit to underlying nvme bdevs. To
> create PVOL across 1 or more base nvme bdevs, either RPC command can be used
> or pvol config file can be updated with the details to create PVOL during
> bootup. Once PVOL bdev is created, other modules get examine calls for pvol
> bdev also. LVS and LVOL can be create over PVOL and exposed to initiator via
> subsystems.
>  
> 3.      It would be good if you could modify your “Pooled volume” PPT slides
> to show how vbdev fits in there.
>  
> Regards,
> Senthil Kumar V.
>  
> From: SPDK <spdk-bounces(a)lists.01.org> On Behalf Of Luse, Paul E
> Sent: Friday, May 18, 2018 12:15 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] SPDK pooled volume module in bdev
>  
> Excellent, thanks Kunal!  There’s a lot to look at here and with Fenggang’s
> aggregation patch as well.  I encourage everyone to take a look at both (and I
> promise I will soon too after post-summit catch-up stuff)
>  
> J
>  
>  
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sablok, Kunal
> Sent: Sunday, May 13, 2018 9:51 AM
> To: spdk(a)lists.01.org
> Subject: [SPDK] SPDK pooled volume module in bdev
>  
> Hi,
> SPDK pooled volume (PVOL) module is a new bdev module which is responsible for
> striping various NVMe devices and expose the pooled volume to bdev layer which
> would enhance the performance and capacity. It can support theoretically 255
> base devices which can be easily enhanced if required (currently it is being
> tested max upto 8 base devices). Multiple strip sizes like 32KB, 64KB, 128KB,
> 256KB, 512KB etc is supported. New RPC commands like "create pvol", "destroy
> pvol" and "get pvols" are introduced to configure pooled volume dynamically in
> a running SPDK system.
> Please find attached more information in a ppt with system end to end testing
> done on this module in SPDK stack in SPDK multi-core environment.
> Please find below gerrithub review details:
> https://review.gerrithub.io/#/c/spdk/spdk/+/410484/
>  
>  
> Regards,
> Kunal
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk