From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============2638868249369670427==" MIME-Version: 1.0 From: David Butterfield Subject: [SPDK] Re: SPDK RAID5 support Date: Tue, 08 Oct 2019 12:25:57 -0600 Message-ID: <3d1b2f26-ed85-192f-b038-21078e46edf5@gmail.com> In-Reply-To: 47C784D99F4D124BB2973FE5C27BDCF17DD3B046@FMSMSX109.amr.corp.intel.com List-ID: To: spdk@lists.01.org --===============2638868249369670427== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On 10/3/19 2:44 PM, Marushak, Nathan wrote: > Do you happen to have any performance and efficiency details? While the p= ort you did was done with minimal changes, great work by the way, we have t= ypically seen that most existing SW architectures require real changes to p= rovide the necessary performance and efficiency improvements required for t= oday's NVM and Networking performance. Diagram: https://raw.githubusercontent.com/DavidButterfield/spdk/tcmu-runne= r/spdk_drbd.pdf Hi Nathan, Paul: After your replies to my earlier message, I did some looking at the DRBD co= de and took some very simple measurements; below is what I found. Regarding the idea of importing a kernel RAID implementation rather than wr= iting a new one: I would consider it seriously -- that's an awful lot of logic that might no= t have to be reimple- mented, and then matured a couple of years before it's ready for production= use with critical data. And going forward, instead of two implementations to be maintained i= n parallel, a single common implementation that behaves consistently whether it's running in the= kernel or in a usermode process. That's a lot of potential value to be weighed along with= other factors. Clearly it is essential that it must be able to perform very well within an= SPDK process and operate smoothly within the datapath. If it can't do that, then there's li= ttle choice but to redevelop it despite the cost. But it seems worth looking for ways to shor= ten what is likely to be a fairly long grind. I'm not familiar with the other kernel RAID modules, so I'll refer here in = terms of DRBD, which includes around 50,000 lines of kernel code. But I see no reason to expect= other kernel RAID modules to be any harder to port to usermode than DRBD was (or SCST at ~80,= 000 lines of code). What are the dimensions of architectural concern? In writing interface shi= ms between SPDK and DRBD I noticed three main areas of mismatch (elaborated further below): (1) Mismatch between the SPDK bdev protocol and the bio protocol expected= by DRBD (2) Network interface between DRBD replication peers (3) Mismatch between SPDK and kernel mechanisms for threading and work ha= ndoff Are there additional areas I should watch out for that have caused trouble = in past efforts? Performance =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D This is a home project, and I don't have equipment to carry out robust perf= ormance testing; but I've started by taking a few very simple measurements to estimate the time = DRBD takes to process a Read operation, which is assumed to be the same as the increase in Read o= peration response- time when DRBD is inserted into an SPDK bdev chain. My SPDK test machine is a laptop with a 4-threaded Core i5-2520M @ 2.5GHz w= ith 4GiB of DDR3 1333 (0.8 ns) RAM (barely enough RAM to run the SPDK iSCSI server). The laptop = is running Ubuntu 18.04.1 and kernel 5.0.0-29. I ran the SPDK iSCSI server (compiled -O3 wit= hout DEBUG) with a single reactor thread on CPU0. I used dd(1) to do some simple 4KiB sequential Read tests from raw LUNs on = the SPDK iSCSI server, configured with LUNs 0, 4, and 6 (all backed by bdev_malloc) as sho= wn in the diagram. I used the same machine for the initiator and the server, connecting to the= IP address of its own (Intel 82579V) Ethernet interface to avoid the 1 Gb Ethernet bottleneck. For these measurements the DRBD server was not connected to a peer, so repl= ication over the network was not active. This should be the fastest path through DRBD to it= s backing storage, without interference by replication or network considerations. (Also, sinc= e this was a Read test only, nothing should be going over the peer-to-peer network anyway) Each LUN was given 16 trials (16 runs of dd), each time reading 130,000 4Ki= B blocks (~1/2 GB) through the iSCSI server, ultimately from an instance of bdev_malloc (the S= PDK ramdisk). Shown below for each LUN are the fastest trial out of its 16 trials, and also the= average (mean) of the top five fastest trials out of the 16. From the MBPS reported by dd(1)= I have also calculated the microseconds per 4KiB Read operation: MBPS Reported by dd(1) Microseconds per OP LUN Config Best_of_16_trials Avg(Top_5_trials) Best 5Avg Delta --- ------ ----------------- ----------------- ---- ---- ----- 0 Malloc 1435 1412 2.85 2.90 4 bio 1006 989 4.07 4.14 1.24 6 bio+DRBD 893 887 4.59 4.62 0.48 (DRBD without bio) (1212) (3.38) LUN0 is a standard SPDK bdev_malloc instance, to compare with the other mea= surements. LUN4 adds bdev_bio and bio_spdk instances back-to-back ahead of the bdev_ma= lloc instance, translating each request to kernel bio protocol and back before it gets to = bdev_malloc (see diagram). The timing difference between LUN4 and LUN0 therefore represents= per-OP overhead contributed by those two modules taken together. Note that the overhead is not much in the bdev protocol translation; I assu= me it's mostly wakeup latency. The bdev_bio and bio_spdk modules each do handoffs of requests an= d responses from one thread to another: bdev_bio hands requests off from an SPDK thread to a DR= BD/UMC thread through a queue_work() call; and bio_spdk hands requests off from a DRBD/UMC thread= to an SPDK thread through a call to spdk_thread_send_msg(). And similarly in the reply direc= tion. So the per-OP timing difference of 1.24 microseconds between LUN4 and LUN0 includes four = thread handoffs, two of them with wakeup latency. LUN6 adds DRBD into the configuration of LUN4, between the bdev_bio and bio= _spdk instances. So the timing difference of 0.48 microseconds between LUN4 and LUN6 represe= nts per-OP processing contributed by DRBD. (DRBD without bio) shows calculated hypothetical timing with DRBD alone, su= btracting the bdev_bio and bio_spdk translations and wakeup latencies. This is the expec= ted timing if DRBD is modified to run on SPDK threads (discussed below), eliminating the thread c= ontext switches. [Calculated as (4.62 - 4.14) + 2.90 =3D 3.38 and hypothetical MBPS back-cal= culated from that.] >From these measurements, DRBD appears to be adding about 480 nanoseconds of= processing time per Read operation -- about a 17% increase over the straight bdev_malloc device= for a 4KiB Read. For larger reads the 480 ns should represent smaller percentages. Finally, I re-ran all of the above experiments twice more with substantiall= y the same results. All the "best" and "average(best_5)" results for each LUN were within 3% of= each other across re-runs of the experiments (all but one were within 2%). [Because the test times were fairly short, random scheduling events with he= avy impact but low frequency can spoil any particular run with performance far below average. = Test time depends on the length of the volume, which is difficult to enlarge because I'm backing= with a bdev_malloc instance on a machine with 4GiB RAM. That is why I chose to average the to= p 5 out of 16 -- to drop spoiled runs, yet not rely completely on one "best" run. A large spre= ad between the "best" and the "average(best_5)" would indicate that the "best" time was unusually= high.] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D (1) Mismatch between the SPDK bdev protocol and the bio protocol expected b= y DRBD This one is pretty trivial. Most usage in DRBD of the bio structure and pr= otocol is concentrated in six places in the code: two near the "top" where client re= quests arrive, two near the "bottom" where requests to backing storage are issued, and two nea= r the "middle" where peer-to-peer communication occurs. These areas together total around 500 l= ines of code. I reckon a couple hundred lines of new code could be added under #ifdef to ch= ange these places to understand the SPDK bdev structures and calls instead of kernel bio structu= res and calls. (2) Network interface to DRBD replication peers In the demo prototype the iSCSI network I/O is done using the SPDK/DPDK net= working facility; but DRBD continues to implement network I/O to replication peers using socket(7= ) calls. This is mainly because I didn't need to change that to get the prototype running. The implementation of the peer transport service within DRBD is isolated be= hind a DRBD-internal transport ops vector, so it's already designed to be easily replaced with o= ther transport implementations. The implementation in drbd_transport_tcp.c would be repla= ced with one that is nonblocking and issues SPDK networking calls instead of socket(7) calls. Some changes are probably needed to the peer receive-side logic so that it = can operate using non-blocking network I/O only. There is already a dispatcher "drbdd()" tha= t calls service functions based on the inter-peer command type in the incoming header; but = those functions then know how much additional data they want, and call for it synchronously. Th= ey may have to have their post-recv processing split out into callback functions to be called (= on a reactor thread) when the amount of data they want is available from the network. This will= be a bit of a chore, because there are a good few of them; but it's straightforward and the rear= ranged code could work for both kernel and usermode and still be clean without needing #ifdef= s in each place. (3) Mismatch between SPDK and kernel mechanisms for threading and work hand= off DRBD already issues backing store I/O operations for asynchronous completio= n from a small set of threads (i.e. it does *not* use a large number of threads each doing a sync= hronous I/O call). I'll use replicated Write for discussion because it is the more complicated= case. Consider a set of DRBD servers acting as peers in a network serving some storage resou= rce. They maintain network connections with each other while replication of the resource is ac= tive. DRBD has some service threads associated with each of these connections (which could go a= way under SPDK). One of the peer-connection service threads is a "drbd_sender" thread. An i= ncoming Write request on a resource is processed by (1a) queueing a copy of the request to the work queue of the drbd_sende= r thread for each of the connected peers for that resource; (1b) waking up those drbd_sender threads; (2a) attempting a "fast-track" submission of the I/O request to the loc= al backing store; (2b) if (2a) fails, queueing the I/O request to a "drbd_submitter" thre= ad. (I don't know the relative frequency of (2b) as compared with (2a)) The queueing at (2b) is currently done using the kernel queue_work() interf= ace, which does the wakeup and arrives (once for each call to queue_work) at the specified func= tion on the work queue's service thread. So I think it is already in the right model and ca= n be simply #ifdef'd to use spdk_thread_send_msg() in place of queue_work(), and let a reactor t= hread do the I/O submission to the backing store (eliminating the "drbd-submitter" work_queu= e service thread). The queueing at (1a) is a little more involved, but I have sketched out the= changes on paper and they are straightforward, maybe 100 lines of added code under #ifdef. The = top-level sender function with the loop that waits for work and then services it has to be s= plit, with the "service" part getting called by the SPDK reactor thread in response to a s= pdk_thread_send_msg() call. Steps (1a) and (1b) on the requesting thread are then replaced with = a call to spdk_thread_send_msg(). Or, the requesting thread may call the sender service function directly und= er some or all conditions (to be analyzed). Either way the drbd_sender threads would also= be eliminated, replaced by reactor threads running a nonblocking network transport impleme= ntation. In any case I believe the code changes needed to fit the DRBD datapath smoo= thly into the SPDK model, with commensurate performance, would be a tiny fraction of the 50,00= 0 lines of relatively mature code that would then be available for use in the SPDK environment. Regards, David Butterfield --===============2638868249369670427==--