* engine rbd broken after commit 2466096336bd0fbc1a94811aa338926af6baf42f @ 2016-01-08 11:15 Jens Rosenboom 2016-01-11 11:09 ` Jens Rosenboom 0 siblings, 1 reply; 5+ messages in thread From: Jens Rosenboom @ 2016-01-08 11:15 UTC (permalink / raw) To: fio When running fio-2.3 with the rbd engine, the tests themselves run fine, but instead of the results being printed, I get: fio: mutex.c:163: fio_mutex_down: Assertion `mutex->magic == 0x4d555445U' failed. The command I am using is fio --ioengine=rbd --clientname=admin --pool=test --rbdname=test1 --rw=randwrite --runtime=10 --ramp_time=None --numjobs=1 --direct=1 --bs=1048576B --iodepth=64 --name=test1 If I instead use another engine, everything is fine, e.g.: fio --ioengine=libaio --filename=/dev/sdb1 --rw=randwrite --runtime=10 --ramp_time=None --numjobs=1 --direct=1 --bs=1048576B --iodepth=64 --name=test1 Going back to fio-2.2.13 makes the error go away and running a git bisect leads to the commit mentioned in the subject as the culprit, the error turns into a Segmentation Fault at that point, though. ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: engine rbd broken after commit 2466096336bd0fbc1a94811aa338926af6baf42f 2016-01-08 11:15 engine rbd broken after commit 2466096336bd0fbc1a94811aa338926af6baf42f Jens Rosenboom @ 2016-01-11 11:09 ` Jens Rosenboom 2016-01-11 14:03 ` Jens Rosenboom 0 siblings, 1 reply; 5+ messages in thread From: Jens Rosenboom @ 2016-01-11 11:09 UTC (permalink / raw) To: fio 2016-01-08 12:15 GMT+01:00 Jens Rosenboom <j.rosenboom@x-ion.de>: > When running fio-2.3 with the rbd engine, the tests themselves run > fine, but instead of the results being printed, I get: > > fio: mutex.c:163: fio_mutex_down: Assertion `mutex->magic == > 0x4d555445U' failed. > > The command I am using is > > fio --ioengine=rbd --clientname=admin --pool=test --rbdname=test1 > --rw=randwrite --runtime=10 --ramp_time=None --numjobs=1 --direct=1 > --bs=1048576B --iodepth=64 --name=test1 > > If I instead use another engine, everything is fine, e.g.: > > fio --ioengine=libaio --filename=/dev/sdb1 --rw=randwrite --runtime=10 > --ramp_time=None --numjobs=1 --direct=1 --bs=1048576B --iodepth=64 > --name=test1 > > Going back to fio-2.2.13 makes the error go away and running a git > bisect leads to the commit mentioned in the subject as the culprit, > the error turns into a Segmentation Fault at that point, though. Accidentally I found out that I can also reproduce the bug with libaio if I set LD_PRELOAD=../jemalloc/lib/libjemalloc.so compiled from https://github.com/jemalloc/jemalloc.git (tested with 3.6.0 and 4.0.4) So it seems there is some use-after-free issue in this commit. I'll try to dig into this a bit more when I get the time, but I won't be sad if someone else is faster. ;) ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: engine rbd broken after commit 2466096336bd0fbc1a94811aa338926af6baf42f 2016-01-11 11:09 ` Jens Rosenboom @ 2016-01-11 14:03 ` Jens Rosenboom 2016-01-13 16:36 ` Jens Axboe 0 siblings, 1 reply; 5+ messages in thread From: Jens Rosenboom @ 2016-01-11 14:03 UTC (permalink / raw) To: fio [-- Attachment #1: Type: text/plain, Size: 1633 bytes --] 2016-01-11 12:09 GMT+01:00 Jens Rosenboom <j.rosenboom@x-ion.de>: > 2016-01-08 12:15 GMT+01:00 Jens Rosenboom <j.rosenboom@x-ion.de>: >> When running fio-2.3 with the rbd engine, the tests themselves run >> fine, but instead of the results being printed, I get: >> >> fio: mutex.c:163: fio_mutex_down: Assertion `mutex->magic == >> 0x4d555445U' failed. >> >> The command I am using is >> >> fio --ioengine=rbd --clientname=admin --pool=test --rbdname=test1 >> --rw=randwrite --runtime=10 --ramp_time=None --numjobs=1 --direct=1 >> --bs=1048576B --iodepth=64 --name=test1 >> >> If I instead use another engine, everything is fine, e.g.: >> >> fio --ioengine=libaio --filename=/dev/sdb1 --rw=randwrite --runtime=10 >> --ramp_time=None --numjobs=1 --direct=1 --bs=1048576B --iodepth=64 >> --name=test1 >> >> Going back to fio-2.2.13 makes the error go away and running a git >> bisect leads to the commit mentioned in the subject as the culprit, >> the error turns into a Segmentation Fault at that point, though. > > Accidentally I found out that I can also reproduce the bug with libaio if I set > > LD_PRELOAD=../jemalloc/lib/libjemalloc.so > > compiled from https://github.com/jemalloc/jemalloc.git (tested with > 3.6.0 and 4.0.4) > > So it seems there is some use-after-free issue in this commit. I'll > try to dig into this a bit more when I get the time, but I won't be > sad if someone else is faster. ;) In the standalone case, sk_out_key is used uninitialized, which may happen to work fine until some other library also uses a pthread_key, like it seems both librbd and libjemalloc do. Attached is a patch that fixes this. [-- Attachment #2: 0001-Don-t-use-uninitialized-pthread_key.patch --] [-- Type: text/x-diff, Size: 1836 bytes --] From fd4f2df3243b215622fc35ed7c175c9e6c0d1f62 Mon Sep 17 00:00:00 2001 From: Jens Rosenboom <j.rosenboom@x-ion.de> Date: Mon, 11 Jan 2016 14:41:41 +0100 Subject: [PATCH] Don't use uninitialized pthread_key When running standalone instead of as a server, sk_out_key doesn't get initialized, but still will be referenced lateron, which according to POSIX is unspecified. Signed-off-by: Jens Rosenboom <j.rosenboom@x-ion.de> --- fio.c | 6 ++++-- server.c | 14 ++++++++++---- server.h | 2 +- 3 files changed, 15 insertions(+), 7 deletions(-) diff --git a/fio.c b/fio.c index bd3e260..d59c567 100644 --- a/fio.c +++ b/fio.c @@ -56,8 +56,10 @@ int main(int argc, char *argv[], char *envp[]) if (fio_start_all_clients()) goto done; ret = fio_handle_clients(&fio_client_ops); - } else - ret = fio_backend(NULL); + } else { + if (!server_create_sk_out_key()) + ret = fio_backend(NULL); + } done: deinitialize_fio(); diff --git a/server.c b/server.c index eccd9d3..85174c2 100644 --- a/server.c +++ b/server.c @@ -2186,17 +2186,23 @@ static void set_sig_handlers(void) sigaction(SIGINT, &act, NULL); } -static int fio_server(void) -{ - int sk, ret; - +int server_create_sk_out_key(void) { if (pthread_key_create(&sk_out_key, NULL)) { log_err("fio: can't create sk_out backend key\n"); return -1; } pthread_setspecific(sk_out_key, NULL); + return 0; +} +static int fio_server(void) +{ + int sk, ret; + + + if (server_create_sk_out_key()) + return -1; dprint(FD_NET, "starting server\n"); if (fio_handle_server_arg()) diff --git a/server.h b/server.h index 9205ae6..3ae55d4 100644 --- a/server.h +++ b/server.h @@ -222,5 +222,5 @@ extern int fio_net_send_quit(int sk); extern int exit_backend; extern int fio_net_port; - +extern int server_create_sk_out_key(void); #endif -- 2.4.10 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: engine rbd broken after commit 2466096336bd0fbc1a94811aa338926af6baf42f 2016-01-11 14:03 ` Jens Rosenboom @ 2016-01-13 16:36 ` Jens Axboe 2016-01-14 8:54 ` Jens Rosenboom 0 siblings, 1 reply; 5+ messages in thread From: Jens Axboe @ 2016-01-13 16:36 UTC (permalink / raw) To: Jens Rosenboom, fio On 01/11/2016 07:03 AM, Jens Rosenboom wrote: > 2016-01-11 12:09 GMT+01:00 Jens Rosenboom <j.rosenboom@x-ion.de>: >> 2016-01-08 12:15 GMT+01:00 Jens Rosenboom <j.rosenboom@x-ion.de>: >>> When running fio-2.3 with the rbd engine, the tests themselves run >>> fine, but instead of the results being printed, I get: >>> >>> fio: mutex.c:163: fio_mutex_down: Assertion `mutex->magic == >>> 0x4d555445U' failed. >>> >>> The command I am using is >>> >>> fio --ioengine=rbd --clientname=admin --pool=test --rbdname=test1 >>> --rw=randwrite --runtime=10 --ramp_time=None --numjobs=1 --direct=1 >>> --bs=1048576B --iodepth=64 --name=test1 >>> >>> If I instead use another engine, everything is fine, e.g.: >>> >>> fio --ioengine=libaio --filename=/dev/sdb1 --rw=randwrite --runtime=10 >>> --ramp_time=None --numjobs=1 --direct=1 --bs=1048576B --iodepth=64 >>> --name=test1 >>> >>> Going back to fio-2.2.13 makes the error go away and running a git >>> bisect leads to the commit mentioned in the subject as the culprit, >>> the error turns into a Segmentation Fault at that point, though. >> >> Accidentally I found out that I can also reproduce the bug with libaio if I set >> >> LD_PRELOAD=../jemalloc/lib/libjemalloc.so >> >> compiled from https://github.com/jemalloc/jemalloc.git (tested with >> 3.6.0 and 4.0.4) >> >> So it seems there is some use-after-free issue in this commit. I'll >> try to dig into this a bit more when I get the time, but I won't be >> sad if someone else is faster. ;) > > In the standalone case, sk_out_key is used uninitialized, which may > happen to work fine until some other library also uses a pthread_key, > like it seems both librbd and libjemalloc do. > > Attached is a patch that fixes this. Thanks for reporting this! I merged a cleaner fix, we can just set up the key earlier and not have to worry about server vs non-server. Can you check if current -git works for you? -- Jens Axboe ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: engine rbd broken after commit 2466096336bd0fbc1a94811aa338926af6baf42f 2016-01-13 16:36 ` Jens Axboe @ 2016-01-14 8:54 ` Jens Rosenboom 0 siblings, 0 replies; 5+ messages in thread From: Jens Rosenboom @ 2016-01-14 8:54 UTC (permalink / raw) To: Jens Axboe; +Cc: fio 2016-01-13 17:36 GMT+01:00 Jens Axboe <axboe@kernel.dk>: ... > Thanks for reporting this! I merged a cleaner fix, we can just set up the > key earlier and not have to worry about server vs non-server. Can you check > if current -git works for you? Yep, working fine, thx. ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2016-01-14 8:54 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2016-01-08 11:15 engine rbd broken after commit 2466096336bd0fbc1a94811aa338926af6baf42f Jens Rosenboom 2016-01-11 11:09 ` Jens Rosenboom 2016-01-11 14:03 ` Jens Rosenboom 2016-01-13 16:36 ` Jens Axboe 2016-01-14 8:54 ` Jens Rosenboom
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.