[SPDK] RAID1E POC Status

[SPDK] RAID1E POC Status

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 2078 bytes --]

Quick refresher lab, this implantation lays out strips as follow.  Dn for Data where n is the copy number for that strip.

2 disk R1E (1 copy of data, RAID0)
D1         D2
D3         D4
D5         D6

3 disk R1E (2 copies of data)
D1         D1         D2
D2         D3         D3
D4         D4         D5...

3 disk R1 (3 copeis)
D1         D1         D1
D2         D2         D2
D3         D3         D3

So some key points as this patch is just a POC:


*       Major benefit is, of course, shared code.  Although I added a function table for RAID related stuff, RAID1E doesn't need it. RAID0 is just a special case of RAID1E (num_copes =1).

o   It also rotates which data copy to read.

*       Because of code movement, experiencing a bit this patch is larger than it needs to be.  Just take a look at

o   raid0_1e_start_rw_request() which now shares the same completion handler as the base_bdev focused functions like reset, unmap, etc.

*       Some of the non-data payload functions are coded but they're likely incomplete and full of small bugs as I haven't tested them yet

*       r/w have been pretty thoroughly tested (for a POC) w/data integrity testing Multiple variants of:

o   num drives 3,4,5

o   num copies 2,3,4

o   strip sizes from 512 to 128

o   strip size & io_size combo (size < strip, size == strip, size > strip)

o   outstanding IO from 1-64

*       there's no UT or config stuff (but only one small change is needed for config), right the RAID1 config is hardcoded so to change recompile

*       this can be broken into 4 or 5 patches probably, if we pursue it I will do so of course


Still just a POC and a lot of work to do before we'd consider production worthy if we decide to continue with it (I'll work on putting a list together over the next few weeks).

Feedback is welcome but please don't review the patch as if it were being considered for merge, right now it's just a proof of concept to facilitate discussion.  Thanks!!!


https://review.gerrithub.io/c/spdk/spdk/+/468187

Re: [SPDK] RAID1E POC Status

[Liu, Xiaodong]

[-- Attachment #1: Type: text/plain, Size: 3227 bytes --]

Hi, Paul
Thank you for the quick refreshment on your ongoing RAID enhancement.
Your RAID1E implementation really simplifies the code and its logics.

From my understanding, since all RAID1E situations share a same formula, just with different arguments.
So just one idea: We can even break up the limitation that number of copies N must be less than or equal to number of disks M.
Even N > M might be meaningless, or even stupid, but who knows, maybe some user has that type of weird usage scenario.

How about also supporting N>M, like:
4 disk R1 (5 copeis)
D1        D1         D1        D1
D1        D2        D2        D2
D2        D2        D3        D3
D3        D3        D3        D4
D4        D4        D4        D4

 --Thanks
From Xiaodong

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Sunday, September 22, 2019 11:43 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] RAID1E POC Status

Quick refresher lab, this implantation lays out strips as follow.  Dn for Data where n is the copy number for that strip.

2 disk R1E (1 copy of data, RAID0)
D1         D2
D3         D4
D5         D6

3 disk R1E (2 copies of data)
D1         D1         D2
D2         D3         D3
D4         D4         D5...

3 disk R1 (3 copeis)
D1         D1         D1
D2         D2         D2
D3         D3         D3

So some key points as this patch is just a POC:


*       Major benefit is, of course, shared code.  Although I added a function table for RAID related stuff, RAID1E doesn't need it. RAID0 is just a special case of RAID1E (num_copes =1).

o   It also rotates which data copy to read.

*       Because of code movement, experiencing a bit this patch is larger than it needs to be.  Just take a look at

o   raid0_1e_start_rw_request() which now shares the same completion handler as the base_bdev focused functions like reset, unmap, etc.

*       Some of the non-data payload functions are coded but they're likely incomplete and full of small bugs as I haven't tested them yet

*       r/w have been pretty thoroughly tested (for a POC) w/data integrity testing Multiple variants of:

o   num drives 3,4,5

o   num copies 2,3,4

o   strip sizes from 512 to 128

o   strip size & io_size combo (size < strip, size == strip, size > strip)

o   outstanding IO from 1-64

*       there's no UT or config stuff (but only one small change is needed for config), right the RAID1 config is hardcoded so to change recompile

*       this can be broken into 4 or 5 patches probably, if we pursue it I will do so of course


Still just a POC and a lot of work to do before we'd consider production worthy if we decide to continue with it (I'll work on putting a list together over the next few weeks).

Feedback is welcome but please don't review the patch as if it were being considered for merge, right now it's just a proof of concept to facilitate discussion.  Thanks!!!


https://review.gerrithub.io/c/spdk/spdk/+/468187

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] RAID1E POC Status

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 4427 bytes --]

Hi Xiadong,

Thanks for taking a look!  Yes we can totally limit configurations if nothing else to simplify both UT and functional test code.  I'll probably work on breaking this patch a bit over the weekend so it's easier to see what's specific to RAID1E or not. Right now this is sort of a fun science project as it also would need some basic config on disk to store disk and volume state (at a minimum) over reboots and one could argue that at least a semi-intelligent rebuild scheme would be needed.

If you've got other ideas please feel free to share!!

-Paul

On 9/26/19, 8:24 AM, "SPDK on behalf of Liu, Xiaodong" <spdk-bounces(a)lists.01.org on behalf of xiaodong.liu(a)intel.com> wrote:

    Hi, Paul
    Thank you for the quick refreshment on your ongoing RAID enhancement.
    Your RAID1E implementation really simplifies the code and its logics.
    
    From my understanding, since all RAID1E situations share a same formula, just with different arguments.
    So just one idea: We can even break up the limitation that number of copies N must be less than or equal to number of disks M.
    Even N > M might be meaningless, or even stupid, but who knows, maybe some user has that type of weird usage scenario.
    
    How about also supporting N>M, like:
    4 disk R1 (5 copeis)
    D1        D1         D1        D1
    D1        D2        D2        D2
    D2        D2        D3        D3
    D3        D3        D3        D4
    D4        D4        D4        D4
    
     --Thanks
    From Xiaodong
    
    -----Original Message-----
    From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
    Sent: Sunday, September 22, 2019 11:43 PM
    To: Storage Performance Development Kit <spdk(a)lists.01.org>
    Subject: [SPDK] RAID1E POC Status
    
    Quick refresher lab, this implantation lays out strips as follow.  Dn for Data where n is the copy number for that strip.
    
    2 disk R1E (1 copy of data, RAID0)
    D1         D2
    D3         D4
    D5         D6
    
    3 disk R1E (2 copies of data)
    D1         D1         D2
    D2         D3         D3
    D4         D4         D5...
    
    3 disk R1 (3 copeis)
    D1         D1         D1
    D2         D2         D2
    D3         D3         D3
    
    So some key points as this patch is just a POC:
    
    
    *       Major benefit is, of course, shared code.  Although I added a function table for RAID related stuff, RAID1E doesn't need it. RAID0 is just a special case of RAID1E (num_copes =1).
    
    o   It also rotates which data copy to read.
    
    *       Because of code movement, experiencing a bit this patch is larger than it needs to be.  Just take a look at
    
    o   raid0_1e_start_rw_request() which now shares the same completion handler as the base_bdev focused functions like reset, unmap, etc.
    
    *       Some of the non-data payload functions are coded but they're likely incomplete and full of small bugs as I haven't tested them yet
    
    *       r/w have been pretty thoroughly tested (for a POC) w/data integrity testing Multiple variants of:
    
    o   num drives 3,4,5
    
    o   num copies 2,3,4
    
    o   strip sizes from 512 to 128
    
    o   strip size & io_size combo (size < strip, size == strip, size > strip)
    
    o   outstanding IO from 1-64
    
    *       there's no UT or config stuff (but only one small change is needed for config), right the RAID1 config is hardcoded so to change recompile
    
    *       this can be broken into 4 or 5 patches probably, if we pursue it I will do so of course
    
    
    Still just a POC and a lot of work to do before we'd consider production worthy if we decide to continue with it (I'll work on putting a list together over the next few weeks).
    
    Feedback is welcome but please don't review the patch as if it were being considered for merge, right now it's just a proof of concept to facilitate discussion.  Thanks!!!
    
    
    https://review.gerrithub.io/c/spdk/spdk/+/468187
    
    _______________________________________________
    SPDK mailing list
    SPDK(a)lists.01.org
    https://lists.01.org/mailman/listinfo/spdk
    _______________________________________________
    SPDK mailing list
    SPDK(a)lists.01.org
    https://lists.01.org/mailman/listinfo/spdk
    

Re: [SPDK] RAID1E POC Status

[David Butterfield]

[-- Attachment #1: Type: text/plain, Size: 1443 bytes --]

On 9/22/19 9:42 AM, Luse, Paul E wrote:
> Still just a POC and a lot of work to do before we'd consider production worthy> if we decide to continue with it (I'll work on putting a list together over the next few weeks).

Are any of the Linux kernel RAID implementations considered good?

I have DRBD 9.0 running in usermode under SPDK as shown in this diagram:

    https://raw.githubusercontent.com/DavidButterfield/spdk/tcmu-runner/spdk_drbd.pdf

One possibility would be to port a kernel RAID module into the spot occupied by DRBD in the diagram.

The "port" of DRBD to run in usermode changes fewer than a dozen lines of code from the original source in the LINBIT repository.  Rather than changing the source code from the DRBD kernel module, its expected environment is simulated around it.  (This is intended to make it easier to update the usermode port to newer versions of the application as they appear.)

I did the same thing with SCST a couple of years ago.  I would expect the same to be possible for a kernel RAID module.  It won't just "drop in", because the set of emulated kernel functions has to be expanded to include whatever the RAID module uses that isn't already covered by the existing ports of SCST and DRBD.  I estimate it would take me one to two months of full-time work to get a kernel RAID module up and running well enough to be tested and used for experimentation.

Regards,
David Butterfield