[SPDK] Re: SPDK RAID5 support

[David Butterfield <dab21774 at gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 12353 bytes --]

On 10/3/19 2:44 PM, Marushak, Nathan wrote:
> Do you happen to have any performance and efficiency details? While the port you did was done with minimal changes, great work by the way, we have typically seen that most existing SW architectures require real changes to provide the necessary performance and efficiency improvements required for today's NVM and Networking performance.

Diagram: https://raw.githubusercontent.com/DavidButterfield/spdk/tcmu-runner/spdk_drbd.pdf

Hi Nathan, Paul:

After your replies to my earlier message, I did some looking at the DRBD code and took some very
simple measurements; below is what I found.

Regarding the idea of importing a kernel RAID implementation rather than writing a new one:

I would consider it seriously -- that's an awful lot of logic that might not have to be reimple-
mented, and then matured a couple of years before it's ready for production use with critical
data.  And going forward, instead of two implementations to be maintained in parallel, a single
common implementation that behaves consistently whether it's running in the kernel or in a
usermode process.  That's a lot of potential value to be weighed along with other factors.

Clearly it is essential that it must be able to perform very well within an SPDK process and
operate smoothly within the datapath.  If it can't do that, then there's little choice but to
redevelop it despite the cost.  But it seems worth looking for ways to shorten what is likely to
be a fairly long grind.

I'm not familiar with the other kernel RAID modules, so I'll refer here in terms of DRBD, which
includes around 50,000 lines of kernel code.  But I see no reason to expect other kernel RAID
modules to be any harder to port to usermode than DRBD was (or SCST at ~80,000 lines of code).

What are the dimensions of architectural concern?  In writing interface shims between SPDK and
DRBD I noticed three main areas of mismatch (elaborated further below):

  (1) Mismatch between the SPDK bdev protocol and the bio protocol expected by DRBD
  (2) Network interface between DRBD replication peers
  (3) Mismatch between SPDK and kernel mechanisms for threading and work handoff

Are there additional areas I should watch out for that have caused trouble in past efforts?

Performance
===========
This is a home project, and I don't have equipment to carry out robust performance testing; but
I've started by taking a few very simple measurements to estimate the time DRBD takes to process
a Read operation, which is assumed to be the same as the increase in Read operation response-
time when DRBD is inserted into an SPDK bdev chain.

My SPDK test machine is a laptop with a 4-threaded Core i5-2520M @ 2.5GHz with 4GiB of DDR3 1333
(0.8 ns) RAM (barely enough RAM to run the SPDK iSCSI server).  The laptop is running Ubuntu
18.04.1 and kernel 5.0.0-29.  I ran the SPDK iSCSI server (compiled -O3 without DEBUG) with a
single reactor thread on CPU0.

I used dd(1) to do some simple 4KiB sequential Read tests from raw LUNs on the SPDK iSCSI
server, configured with LUNs 0, 4, and 6 (all backed by bdev_malloc) as shown in the diagram.
I used the same machine for the initiator and the server, connecting to the IP address of its
own (Intel 82579V) Ethernet interface to avoid the 1 Gb Ethernet bottleneck.

For these measurements the DRBD server was not connected to a peer, so replication over the
network was not active.  This should be the fastest path through DRBD to its backing storage,
without interference by replication or network considerations.  (Also, since this was a Read
test only, nothing should be going over the peer-to-peer network anyway)

Each LUN was given 16 trials (16 runs of dd), each time reading 130,000 4KiB blocks (~1/2 GB)
through the iSCSI server, ultimately from an instance of bdev_malloc (the SPDK ramdisk).  Shown
below for each LUN are the fastest trial out of its 16 trials, and also the average (mean) of
the top five fastest trials out of the 16.  From the MBPS reported by dd(1) I have also
calculated the microseconds per 4KiB Read operation:

                MBPS Reported by dd(1)                  Microseconds per OP
LUN Config      Best_of_16_trials   Avg(Top_5_trials)   Best    5Avg   Delta
--- ------      -----------------   -----------------   ----    ----   -----
 0  Malloc             1435                1412         2.85    2.90
 4  bio                1006                 989         4.07    4.14    1.24
 6  bio+DRBD            893                 887         4.59    4.62    0.48
   (DRBD without bio)                     (1212)               (3.38)

LUN0 is a standard SPDK bdev_malloc instance, to compare with the other measurements.

LUN4 adds bdev_bio and bio_spdk instances back-to-back ahead of the bdev_malloc instance,
translating each request to kernel bio protocol and back before it gets to bdev_malloc (see
diagram).  The timing difference between LUN4 and LUN0 therefore represents per-OP overhead
contributed by those two modules taken together.

Note that the overhead is not much in the bdev protocol translation; I assume it's mostly wakeup
latency.  The bdev_bio and bio_spdk modules each do handoffs of requests and responses from one
thread to another:  bdev_bio hands requests off from an SPDK thread to a DRBD/UMC thread through
a queue_work() call; and bio_spdk hands requests off from a DRBD/UMC thread to an SPDK thread
through a call to spdk_thread_send_msg().  And similarly in the reply direction.  So the per-OP
timing difference of 1.24 microseconds between LUN4 and LUN0 includes four thread handoffs, two
of them with wakeup latency.

LUN6 adds DRBD into the configuration of LUN4, between the bdev_bio and bio_spdk instances.
So the timing difference of 0.48 microseconds between LUN4 and LUN6 represents per-OP processing
contributed by DRBD.

(DRBD without bio) shows calculated hypothetical timing with DRBD alone, subtracting the
bdev_bio and bio_spdk translations and wakeup latencies.  This is the expected timing if DRBD is
modified to run on SPDK threads (discussed below), eliminating the thread context switches.
[Calculated as (4.62 - 4.14) + 2.90 = 3.38 and hypothetical MBPS back-calculated from that.]

From these measurements, DRBD appears to be adding about 480 nanoseconds of processing time per
Read operation -- about a 17% increase over the straight bdev_malloc device for a 4KiB Read.
For larger reads the 480 ns should represent smaller percentages.

Finally, I re-ran all of the above experiments twice more with substantially the same results.
All the "best" and "average(best_5)" results for each LUN were within 3% of each other across
re-runs of the experiments (all but one were within 2%).

[Because the test times were fairly short, random scheduling events with heavy impact but low
frequency can spoil any particular run with performance far below average.  Test time depends on
the length of the volume, which is difficult to enlarge because I'm backing with a bdev_malloc
instance on a machine with 4GiB RAM.  That is why I chose to average the top 5 out of 16 -- to
drop spoiled runs, yet not rely completely on one "best" run.  A large spread between the "best"
and the "average(best_5)" would indicate that the "best" time was unusually high.]

================================================================================================

(1) Mismatch between the SPDK bdev protocol and the bio protocol expected by DRBD

This one is pretty trivial.  Most usage in DRBD of the bio structure and protocol is
concentrated in six places in the code:  two near the "top" where client requests arrive, two
near the "bottom" where requests to backing storage are issued, and two near the "middle" where
peer-to-peer communication occurs.  These areas together total around 500 lines of code.  I
reckon a couple hundred lines of new code could be added under #ifdef to change these places to
understand the SPDK bdev structures and calls instead of kernel bio structures and calls.

(2) Network interface to DRBD replication peers

In the demo prototype the iSCSI network I/O is done using the SPDK/DPDK networking facility; but
DRBD continues to implement network I/O to replication peers using socket(7) calls.  This is
mainly because I didn't need to change that to get the prototype running.

The implementation of the peer transport service within DRBD is isolated behind a DRBD-internal
transport ops vector, so it's already designed to be easily replaced with other transport
implementations.  The implementation in drbd_transport_tcp.c would be replaced with one that is
nonblocking and issues SPDK networking calls instead of socket(7) calls.

Some changes are probably needed to the peer receive-side logic so that it can operate using
non-blocking network I/O only.  There is already a dispatcher "drbdd()" that calls service
functions based on the inter-peer command type in the incoming header; but those functions then
know how much additional data they want, and call for it synchronously.  They may have to have
their post-recv processing split out into callback functions to be called (on a reactor thread)
when the amount of data they want is available from the network.  This will be a bit of a chore,
because there are a good few of them; but it's straightforward and the rearranged code could
work for both kernel and usermode and still be clean without needing #ifdefs in each place.

(3) Mismatch between SPDK and kernel mechanisms for threading and work handoff

DRBD already issues backing store I/O operations for asynchronous completion from a small set of
threads (i.e. it does *not* use a large number of threads each doing a synchronous I/O call).

I'll use replicated Write for discussion because it is the more complicated case.  Consider a
set of DRBD servers acting as peers in a network serving some storage resource.  They maintain
network connections with each other while replication of the resource is active.  DRBD has some
service threads associated with each of these connections (which could go away under SPDK).

One of the peer-connection service threads is a "drbd_sender" thread.  An incoming Write request
on a resource is processed by
    (1a) queueing a copy of the request to the work queue of the drbd_sender thread for each of
         the connected peers for that resource;
    (1b) waking up those drbd_sender threads;
    (2a) attempting a "fast-track" submission of the I/O request to the local backing store;
    (2b) if (2a) fails, queueing the I/O request to a "drbd_submitter" thread.
         (I don't know the relative frequency of (2b) as compared with (2a))

The queueing at (2b) is currently done using the kernel queue_work() interface, which does the
wakeup and arrives (once for each call to queue_work) at the specified function on the work
queue's service thread.  So I think it is already in the right model and can be simply #ifdef'd
to use spdk_thread_send_msg() in place of queue_work(), and let a reactor thread do the I/O
submission to the backing store (eliminating the "drbd-submitter" work_queue service thread).

The queueing at (1a) is a little more involved, but I have sketched out the changes on paper and
they are straightforward, maybe 100 lines of added code under #ifdef.  The top-level sender
function with the loop that waits for work and then services it has to be split, with the
"service" part getting called by the SPDK reactor thread in response to a spdk_thread_send_msg()
call.  Steps (1a) and (1b) on the requesting thread are then replaced with a call to
spdk_thread_send_msg().

Or, the requesting thread may call the sender service function directly under some or all
conditions (to be analyzed).  Either way the drbd_sender threads would also be eliminated,
replaced by reactor threads running a nonblocking network transport implementation.

In any case I believe the code changes needed to fit the DRBD datapath smoothly into the SPDK
model, with commensurate performance, would be a tiny fraction of the 50,000 lines of relatively
mature code that would then be available for use in the SPDK environment.

Regards,
David Butterfield