All of lore.kernel.org
 help / color / mirror / Atom feed
From: Mark Kirkwood <mark.kirkwood-6STWZtX7tXAqAMOr+u8IRA@public.gmane.org>
To: Mark Nelson
	<mark.a.nelson-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>,
	Jens Axboe <axboe-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org>,
	fio-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Cc: "d.gollub-+tb+GG71Y8CELgA04lAiVw@public.gmane.org >> Daniel
	Gollub" <d.gollub-+tb+GG71Y8CELgA04lAiVw@public.gmane.org>,
	"xan.peng" <xanpeng-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>,
	"ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org"
	<ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
Subject: Re: fio rbd hang for block sizes > 1M
Date: Sat, 25 Oct 2014 11:45:17 +1300	[thread overview]
Message-ID: <544AD67D.4030603@catalyst.net.nz> (raw)
In-Reply-To: <544A5DA6.2010709-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>

Interestingly, I first encountered this on (what I think is) 0.86 
release (0.86-1precise). I wonder if you had a bigger rbd cache on the 
release cluster you tested?

As mentioned in the same named thread on -users, disabling the rbd cache 
stops the hang.

Regards

Mark

On 25/10/14 03:09, Mark Nelson wrote:
> More info:
>
> I went back and tested fio versions back to 2.1.10 and still encountered
> the issue.  I then went back and tested the v0.86 release versus giant
> and was able to get through a 4MB read test without error.  I suspect
> this is not an fio problem.  I'll try to narrow down the commit after
> 0.86 that is causing this.
>
> Mark
>
> On 10/24/2014 08:19 AM, Mark Nelson wrote:
>> FWIW we are seeing this at Redhat/Inktank with recent fio from master
>> and ceph giant branch as well.
>>
>> Mark
>>
>> On 10/24/2014 01:17 AM, Mark Kirkwood wrote:
>>> On 24/10/14 18:35, Jens Axboe wrote:
>>>> CC'ing relevant parties, leaving email intact.
>>>>
>>>
>>> Note that the 'Killed' is because I killed the run - it hangs and
>>> appears to be non interruptable. I missed that when pasting, sorry!
>>>
>>>>> $ fio read-test.fio     # attached
>>>>> rbd_thread: (g=0): rw=read, bs=2M-2M/2M-2M/2M-2M, ioengine=rbd,
>>>>> iodepth=32
>>>>> fio-2.1.13-88-gb2ee7
>>>>> Starting 1 process
>>>>> rbd engine: RBD version: 0.1.8
>>>>> Killed1 (f=1): [R(1)] [inf% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta
>>>>> 1158050441d:06h:59m:33s]
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe fio" in
>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>

WARNING: multiple messages have this Message-ID (diff)
From: Mark Kirkwood <mark.kirkwood@catalyst.net.nz>
To: Mark Nelson <mark.a.nelson@gmail.com>,
	Jens Axboe <axboe@kernel.dk>,
	fio@vger.kernel.org
Cc: "d.gollub@telekom.de >> Daniel Gollub" <d.gollub@telekom.de>,
	"xan.peng" <xanpeng@gmail.com>,
	"ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
Subject: Re: fio rbd hang for block sizes > 1M
Date: Sat, 25 Oct 2014 11:45:17 +1300	[thread overview]
Message-ID: <544AD67D.4030603@catalyst.net.nz> (raw)
In-Reply-To: <544A5DA6.2010709@gmail.com>

Interestingly, I first encountered this on (what I think is) 0.86 
release (0.86-1precise). I wonder if you had a bigger rbd cache on the 
release cluster you tested?

As mentioned in the same named thread on -users, disabling the rbd cache 
stops the hang.

Regards

Mark

On 25/10/14 03:09, Mark Nelson wrote:
> More info:
>
> I went back and tested fio versions back to 2.1.10 and still encountered
> the issue.  I then went back and tested the v0.86 release versus giant
> and was able to get through a 4MB read test without error.  I suspect
> this is not an fio problem.  I'll try to narrow down the commit after
> 0.86 that is causing this.
>
> Mark
>
> On 10/24/2014 08:19 AM, Mark Nelson wrote:
>> FWIW we are seeing this at Redhat/Inktank with recent fio from master
>> and ceph giant branch as well.
>>
>> Mark
>>
>> On 10/24/2014 01:17 AM, Mark Kirkwood wrote:
>>> On 24/10/14 18:35, Jens Axboe wrote:
>>>> CC'ing relevant parties, leaving email intact.
>>>>
>>>
>>> Note that the 'Killed' is because I killed the run - it hangs and
>>> appears to be non interruptable. I missed that when pasting, sorry!
>>>
>>>>> $ fio read-test.fio     # attached
>>>>> rbd_thread: (g=0): rw=read, bs=2M-2M/2M-2M/2M-2M, ioengine=rbd,
>>>>> iodepth=32
>>>>> fio-2.1.13-88-gb2ee7
>>>>> Starting 1 process
>>>>> rbd engine: RBD version: 0.1.8
>>>>> Killed1 (f=1): [R(1)] [inf% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta
>>>>> 1158050441d:06h:59m:33s]
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe fio" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>



  parent reply	other threads:[~2014-10-24 22:45 UTC|newest]

Thread overview: 77+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2014-10-24  2:38 fio rbd hang for block sizes > 1M Mark Kirkwood
2014-10-24  5:35 ` Jens Axboe
2014-10-24  6:17   ` Mark Kirkwood
2014-10-24 13:19     ` Mark Nelson
     [not found]       ` <544A51C7.40803-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2014-10-24 14:09         ` Mark Nelson
2014-10-24 14:09           ` Mark Nelson
     [not found]           ` <544A5DA6.2010709-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2014-10-24 14:30             ` Jens Axboe
2014-10-24 14:30               ` Jens Axboe
2014-10-24 22:45             ` Mark Kirkwood [this message]
2014-10-24 22:45               ` Mark Kirkwood
     [not found]               ` <544AD67D.4030603-6STWZtX7tXAqAMOr+u8IRA@public.gmane.org>
2014-10-25  0:12                 ` Mark Nelson
2014-10-25  0:12                   ` Mark Nelson
     [not found]                   ` <544AEAE7.6080603-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2014-10-25  0:37                     ` Mark Kirkwood
2014-10-25  0:37                       ` Mark Kirkwood
     [not found]                       ` <544AF0D2.1050405-6STWZtX7tXAqAMOr+u8IRA@public.gmane.org>
2014-10-25  2:35                         ` Mark Kirkwood
2014-10-25  2:35                           ` Mark Kirkwood
2014-10-25  3:47                           ` Jens Axboe
2014-10-25  4:50                             ` fio rbd completions (Was: fio rbd hang for block sizes > 1M) Mark Kirkwood
2014-10-25 19:20                               ` Jens Axboe
     [not found]                                 ` <544BF808.2090800-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org>
2014-10-25 22:25                                   ` Mark Kirkwood
2014-10-25 22:25                                     ` Mark Kirkwood
2014-10-27  9:27                                     ` Ketor D
     [not found]                                       ` <CAM9_UU_S7qhenZW34Lw3r=RHoVa1__610RRsFScgt0adi1dpFw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2014-10-27 10:25                                         ` Ketor D
2014-10-27 10:25                                           ` Ketor D
2014-10-27 14:19                                           ` Jens Axboe
2014-10-27 14:15                                       ` Jens Axboe
     [not found]                                     ` <544C2371.1020403-6STWZtX7tXAqAMOr+u8IRA@public.gmane.org>
2014-10-27 14:19                                       ` Jens Axboe
2014-10-27 14:19                                         ` Jens Axboe
     [not found]                                         ` <544E547C.30009-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org>
2014-10-27 15:12                                           ` Ketor D
2014-10-27 15:12                                             ` Ketor D
2014-10-27 15:22                                             ` Jens Axboe
2014-10-27 15:25                                               ` Jens Axboe
2014-10-27 15:29                                                 ` Ketor D
     [not found]                                                   ` <CAM9_UU-35i9uRu9EDQSM-b7CjmxrKYV2Gz8ocrykOYk+2q++hw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2014-10-27 15:36                                                     ` Jens Axboe
2014-10-27 15:36                                                       ` Jens Axboe
2014-10-27 15:45                                                       ` Ketor D
2014-10-27 15:53                                                         ` Jens Axboe
2014-10-27 16:20                                                           ` Ketor D
     [not found]                                                           ` <CAM9_UU8x2uZZUWaPPoy+LH mUhC_3sqKZ9GPsEqDwKUkprg4kdQ@mail.gmail.com>
     [not found]                                                             ` <CAM9_UU8x2uZZUWaPPoy+LHmUhC_3sqKZ9GPsEqDwKUkprg4kdQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2014-10-27 16:55                                                               ` Jens Axboe
2014-10-27 16:55                                                                 ` Jens Axboe
2014-10-27 21:59                                                               ` Mark Kirkwood
2014-10-27 21:59                                                                 ` Mark Kirkwood
2014-10-27 22:32                                                                 ` Jens Axboe
2014-10-27 22:32                                                                   ` Jens Axboe
     [not found]                                                                   ` <544EC7F1.6010900-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org>
2014-10-27 23:21                                                                     ` Mark Kirkwood
2014-10-27 23:21                                                                       ` Mark Kirkwood
     [not found]                                                                       ` <544ED37D.6060800-6STWZtX7tXAqAMOr+u8IRA@public.gmane.org>
2014-10-28  3:23                                                                         ` Ketor D
2014-10-28  3:23                                                                           ` Ketor D
     [not found]                                                                           ` <CAM9_UU8MHdj+mjAWBziETxPDnwTt0JBuHrQp2Fu9YtF=msae3w-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2014-10-28  4:01                                                                             ` Mark Kirkwood
2014-10-28  4:01                                                                               ` Mark Kirkwood
2014-10-28  4:05                                                                           ` Jens Axboe
2014-10-28  4:49                                                                             ` Ketor D
     [not found]                                                                               ` <CAM9_UU9G5vQ68UxMakte-Wb5B9_KBo24ov7=hNHpYqEtko2nQg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2014-10-28 15:14                                                                                 ` Jens Axboe
2014-10-28 15:14                                                                                   ` Jens Axboe
2014-10-28 15:49                                                                                   ` Ketor D
     [not found]                                                                                     ` <CAM9_UU_o8kS1wJnDKTvd8+qkm9=93yfW3THr_8ni8C+5=TH6tg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2014-10-28 15:53                                                                                       ` Jens Axboe
2014-10-28 15:53                                                                                         ` Jens Axboe
2014-10-28 17:09                                                                                       ` Jens Axboe
2014-10-28 17:09                                                                                         ` Jens Axboe
2014-10-28 18:43                                                                                         ` Ketor D
2014-10-29  7:15                                                                                           ` Ketor D
     [not found]                                                                                             ` <CAM9_UU8=MtUnQiembBfr8YQiDOD7TNey=mp8_H6gySenRVHy6A-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2014-10-29 14:31                                                                                               ` Jens Axboe
2014-10-29 14:31                                                                                                 ` Jens Axboe
     [not found]                                                                                                 ` <5450FA47.2030203-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org>
2014-10-30  2:50                                                                                                   ` Ketor D
2014-10-30  2:50                                                                                                     ` Ketor D
2014-10-30  2:55                                                                                                     ` Jens Axboe
2014-10-30  2:55                                                                                                       ` Jens Axboe
2014-10-30  5:29                                                                                                       ` Ketor D
2014-10-30  7:44                                                                                                 ` Mark Kirkwood
2014-10-30  7:44                                                                                                   ` Mark Kirkwood
2014-10-30  8:04                                                                                                   ` Ketor D
2014-10-31  8:54                                                                                                     ` Mark Kirkwood
2014-10-31  8:54                                                                                                       ` Mark Kirkwood
2014-10-24 22:30       ` fio rbd hang for block sizes > 1M Mark Kirkwood
2014-10-24 22:38         ` Mark Nelson
2014-10-24 14:11   ` Danny Al-Gaaf
2014-10-24 14:31     ` Jens Axboe

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=544AD67D.4030603@catalyst.net.nz \
    --to=mark.kirkwood-6stwztx7txaqamor+u8ira@public.gmane.org \
    --cc=axboe-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org \
    --cc=ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
    --cc=d.gollub-+tb+GG71Y8CELgA04lAiVw@public.gmane.org \
    --cc=fio-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
    --cc=mark.a.nelson-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org \
    --cc=xanpeng-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.