From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============2847486152170253395==" MIME-Version: 1.0 From: David Butterfield Subject: [SPDK] SPDK + DRBD + tcmu-runner storage handlers Date: Wed, 18 Sep 2019 09:53:03 -0600 Message-ID: <806b8d90-272f-3e82-3fb6-9495e48ef20d@gmail.com> In-Reply-To: 82C9F782B054C94B9FC04A331649C77ABFEEC2A2@FMSMSX125.amr.corp.intel.com List-ID: To: spdk@lists.01.org --===============2847486152170253395== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable (Refer to diagram in spdk_drbd.pdf) tcmu-runner block storage handlers running under SPDK =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D A prototype of a new block device module "bdev_tcmur" running under the Sto= rage Performance Development Kit allows access to block storage using tcmu-runner handlers. = (tcmu-runner itself is not involved; only its loadable handlers are used here.) The bdev_tcmur= module is based on the bdev_aio module source. It enables the pathways for LUN 2 and LUN 3 sh= own in the diagram. Distributed Replicated Block Device (DRBD 9.0) running in usermode =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D A recent project ported DRBD from the kernel to run in usermode as a Linux = process, using support from emulated kernel functions and a multi-threaded engine based on= epoll_wait(). The DRBD source code itself is unmodified, with its expected environment simula= ted around it. It receives requests from clients through the kernel's block-I/O ("bio") proto= col, and also makes requests to its backing storage using that same protocol. Usermode DRBD ca= n be plumbed under Usermode SCST (not shown in this diagram), or under a FUSE interface (drbd1= in the diagram). DRBD running with SPDK =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D To bring usermode DRBD into an SPDK process, a new SPDK bdev module "bdev_b= io" implements translation of SPDK block device requests into the kernel's block-I/O ("bio= ") protocol, as expected by DRBD. This enables the pathways for LUN 4 and LUN 5 shown in t= he diagram. DRBD then makes bio requests to its backing storage, which at present must = be a tcmu-runner device. To support arbitrary SPDK devices (e.g. use Malloc0 to back a DRBD= device) requires a "bio_bdev" module to translate bio requests into SPDK bdev protocol. (TBD) The SPDK configuration file plus an external helper provide enough for SPDK= to configure DRBD with the devices needed by SPDK. Once the SPDK+DRBD server is up and runni= ng, the DRBD logic can be controlled using the native DRBD management commands (drbdsetup and = drbdadm). The emulated kernel functions (UMC - usermode compatibility) make use of se= rvices provided by a multithreaded event engine (MTE) implemented around epoll_wait(). The MTE = services are accessed by UMC through an ops vector backed by MTE services for memory, time, and t= hreads, as well as event polling of file descriptors, timers, and a FIFO of work to be done AS= AP. I anticipate an easy time converting the ops vector to point at a shim to SPDK services in = place of MTE calls. Limitations =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D The implementation is very new. So far I have mainly tested it using the S= PDK iSCSI server, exporting tcmu-runner backend devices as SCSI LUNs. That seems to work rel= iably. The drbd and tcmur devices can alternatively be mounted locally through the FUSE interfa= ce, which also works. I have only tried it with one reactor core. This prototype implementation is clearly in need of some cleaning up and in= terfaces straightened out. I've been studying SPDK for less then two weeks, and I guessed at a f= ew things that I need to go back over carefully. But it runs. The makefiles have optimizations turned off and debugs turned on. The UMC FUSE implementation is single-threaded and synchronous; thus it ope= rates at an effective queue depth of one. This matters most when using it to access replicated v= olumes with DRBD Protocol C, where performance will suffer significantly. Accessing volumes= with Protocol A configured to "pull-ahead" performs reasonably, as does accessing the same = data through an iSCSI LUN, which does not have the QD=3D1 limitation, NOTE: Only tcmu-runner modules handler_ram.so and handler_file.so have been= tried so far; the latter is significantly faster, so it is the one specified in the exa= mple configuration files. An *async* tcmu-runner handler (nr_threads =3D=3D 0) has yet = to be tried! Usermode DRBD Limitations =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Netlink multicast emulation not yet implemented, so anything like "drbdsetu= p wait*" hangs. The bio block device nodes are exposed through a mount of the server's UMC = fuse filesystem implementation. The fuse-tree node that represents a DRBD or TCMUR block d= evice appears as a regular file rather than as a block device (because otherwise fuse directs = I/O for that dev_t to the kernel instead of the fuse filesystem server). So when communicating w= ith a usermode server, the DRBD utilities are modified to omit the check that their device= is S_IFBLK() rather than S_IFREG(). Messages from the utilities and in the logs have not been modified, so will= still refer to "the kernel" etc when referring to code that has been ported from the kernel to = usermode. Resync may run noticeably slower when observing resync network traffic with= tcpdump. Something I expect NOT to work is running the server executable off of a di= sk it implements. I have only run the usermode server on machines without DRBD installed in t= he kernel. The build script and the config/run instructions assume that there are no DRBD module= s or utilities installed. (That would likely be very confusing, but might actually work i= f assigned separate ports) Bugs =3D=3D=3D=3D In general only the "happy path" has received any exercise -- expect bugs i= n untested error- handling logic. "Exclusive" opens aren't really exclusive, so be careful not to mount the s= ame storage twice; for example /UMCfuse/dev/file_c and /UMCfuse/dev/drbd1 are the same storage= in the example configuration. For another example, SPDK configuration [BIO] for bdev_bio = should never consume both drbd2 and ram_b concurrently. "Holders" and "claims" are not yet impl= emented. The "writable" bits in the mode permissions do not appear correctly in /UMC= fuse/dev. The server apparently can mount and write a replicated DRBD device on a sec= ondary node. fsync/flush is probably ineffective. 4096 is the only tested block size; possible bugs with others. Stacktrace is broken. Probably there are broken untested refcountings on things that usually only= get opened once. (E.g. two concurrent dd commands to the same device or things like that). Clean shutdown does not work at all. I always "make clean" before make, because my makefiles don't calculate dep= endencies right. The makefiles are hateworthy. SCST repository is unnecessarily tangled up with= the build. Sometimes DRBD resync doesn't start upon reconnect after restarting the ser= ver. If it doesn't start, disconnecting + reconnecting to the peer usually gets it going. I have seen a very weird problem using the tcmu-runner handler_file.so. Af= ter dlopen(), libtcmur.c looks up the symbol for the handler_init routine and calls it. = The handler calls back with the address of its ops vector. The function addresses in the ops= vector are properly relocated for the loaded module, and the main module calls functions throug= h the ops vector thousands of times... and then suddenly SIGSEGV, and examining the ops vect= or (under gdb) the function addresses are all back to their original UNRELOCATED relative valu= es! (And the faulting program counter address matches the unrelocated value in the membe= r of the ops vector it was trying to call through.) I have never seen this happen with handler= _ram. However, I have not seen the problem since I ensured adequate memory for th= e SPDK server. The SPDK test machine has "only" 4GiB RAM, and swap space used was increasing d= uring problem tests. Because handler_file runs significantly faster than handler_ram for mounted= filesystems, all the tcmu-runner handler devices in the example are now by default configured to= use handler_file (despite some names in /UMCfuse/dev and /tmp continuing to be called "ram" = rather than "file"). Building from Source Code =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D The source code to build SPDK with support for tcmu-runner handlers is in m= y forks of the SPDK and tcmu-runner repositories. Building-in DRBD support requires several add= itional repositories. Because building is presently a mess, I've included scripts that will downl= oad the repositories and build SPDK with support for tcmu-runner loadable handlers and/or DRBD. = To download and build the SPDK iSCSI server with support for BOTH, cd into an empty directo= ry and do: wget https://raw.githubusercontent.com/DavidButterfield/spdk/tcmu-runne= r/BUILD_spdk_drbd.sh chmod 755 BUILD_spdk_drbd.sh ./BUILD_spdk_drbd.sh To OMIT DRBD and only download/build SPDK with support for tcmu-runner hand= lers do this instead: wget https://raw.githubusercontent.com/DavidButterfield/spdk/tcmu-runne= r/BUILD_spdk_tcmur.sh chmod 755 BUILD_spdk_tcmur.sh ./BUILD_spdk_tcmur.sh The (former) DRBD script downloads and builds a superset of what the (latte= r) TCMUR script does, and after the DRBD download you can specify to build the more limited serve= r (to support TCMUR but not DRBD) by selection of configuration options: --with-tcmur # SPDK with tcmu-runner only --with-tcmur --with-drbd # SPDK with DRBD and tcmu-runner Comments in the download/build scripts document the process in case you wan= t to do some steps manually. (It asks for the sudo password to install, so you might want to = look at it first.) The SCRIPTS ASSUME you already have the tools and libraries installed such = that you can build the standard SPDK, DRBD, and tcmu-runner repositories. Some of the makefil= es require various build tools -- here are package names I added to a fresh installation of Ub= untu 18.04 LTS to complete the build: build-essential g++ gcc git make gdb valgrind cscope exuberant-= ctags libfuse-dev libaio-dev libglib2.0-dev libkmod-dev libnl-3-dev libn= l-genl-3-dev librbd-dev autoconf automake flex coccinelle cmake I always "make clean" before "make", because my makefiles don't calculate d= ependencies right. There should be no compile errors, but there will be some warnings in the D= RBD code. The build script documents a few that are expected and can be ignored for now. Configuring =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D The example config files in etc/drbd.d are from a node in my setup. They w= ill have to be modified to suit your network configuration, and put into /etc/drbd.d on yo= ur test system. There is also a nasty "helper" script /usr/sbin/drbdadm_up_primary which at= present can only bring up one specific SPDK/DRBD device in the example configuration. To su= pport a different configuration, that file probably needs updating (in addition to /etc/drbd.= d/* and the SPDK configuration file). Running =3D=3D=3D=3D=3D=3D=3D To run the DRBD management utilities so that they refer to the simulated /p= roc that talks to the usermode server process (rather than the real /proc that talks to the kerne= l): export UMC_FS_ROOT=3D/UMCfuse # *** SET ENVIRONMENT *** The utilities need the $UMC_FS_ROOT environment variable set to control the= usermode DRBD server instead of a kernel-based server. But they also need to run superuser. Ke= ep in mind that the sudo program does not pass your shell environment through to the program gi= ven on its command line, unless you specify "sudo -E". (Omitting the "-E" leads to bewilderin= g non-sequitur error messages because the utility is trying to parse an earlier version of the c= ommand language) Also the *server* needs the $UMC_FS_ROOT environment variable set, because = it invokes the utilities through a "usermode helper", and they inherit the variable from t= he server. The download/build script ends with a suggested server command-line, that d= epends on which script you used. The two scripts refer to different configuration files de= pending on whether DRBD support was selected or not. Troubleshooting =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D If you didn't read the sections "Configuring" and "Running" just above, rea= d those. The implementation and configuration of SPDK+DRBD is an order of magnitude = more complex than the relatively straightforward implementation of tcmu-runner handlers under SPD= K. You may wish to make sure the simpler case works before bringing in DRBD. Make sure your configuration files were suitably modified for your names, a= ddresses, etc. Make sure you are running the server and the utilities with environment var= iable set: export UMC_FS_ROOT=3D/UMCfuse sudo -E drbdadm ... # -E to pass the environment variable through sudo Missing the environment variable leads to bewildering non-sequitur error me= ssages because the utility is trying to parse an earlier version of the command language. The= se messages in the server log or output from a DRBD utility probably mean the environment vari= able is not set: Cannot determine minor device number of device Missing connection endpoint argument Parse error: 'disk | device | address | meta-disk | flexible-meta-disk'= expected, but got 'node-id' /proc and /sys/module entries for the DRBD usermode server can be observed = under /UMCfuse. After starting the server, a node should appear in /UMCfuse/dev for each bi= o or tcmu-runner device configured by SPDK. DRBD resource "nonspdk" (drbd1) is not configur= ed as an SPDK device. After the server is up the resource may be enabled using the native DRBD co= mmand, after which its node should appear under /UMCfuse/dev: drbdadm up nonspdk # assumes metadata previously created Multiple names can refer to the same underlying storage. Referring to the = diagram, LUN 5, bio1, /UMCfuse/dev/drbd2, and /UMCfuse/dev/ram_b all refer to the same underlying= storage in /tmp/tcmur_ram01. A filesystem can be mounted on an iSCSI initiator as LUN= 5, or the same filesystem can be mounted locally, e.g. sudo mount /UMCfuse/dev/drbd2 /mnt/x One bug is that exclusive open is not currently exclusive, so be careful no= t to use storage multiple ways at the same time! More Information =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D The DRBD kernel source code ported to usermode is (within a dozen lines of)= unmodified from the original code in the LINBIT repository, with its expected kernel environmen= t simulated around it. For more information about how that was done, see the README.md with d= iagrams at https://github.com/DavidButterfield/SCST-Usermode-Adaptation David Butterfield Tue 17 Sep 2019 09:43:35 PM MDT --===============2847486152170253395==--