iozone test crashed on ceph

iozone test crashed on ceph

[udit agarwal]

Hi,
 I have set up ceph system with a client, mon and mds on one system which is
connected to 2 osds. I ran iozone test with a 10G file and it ran fine. But when
I ran iozone test with a 5G file, the process got killed and our ceph system
hanged. Can anyone please help me with this.

 Thanks in advance.

--Udit

Re: iozone test crashed on ceph

[Sam Just]

Hi,

Thanks for letting us know.  What version are you running?  Can you
post your ceph.conf to give us an idea of how your cluster is
configured?  Also, did any of the daemons crash?  If it's
reproducible, it would help to turn up osd and mds debugging to 20 and
post the logs.

Thanks
-Sam

On Thu, May 31, 2012 at 5:58 PM, udit agarwal <fzdudit@gmail.com> wrote:
> Hi,
>  I have set up ceph system with a client, mon and mds on one system which is
> connected to 2 osds. I ran iozone test with a 10G file and it ran fine. But when
> I ran iozone test with a 5G file, the process got killed and our ceph system
> hanged. Can anyone please help me with this.
>
>  Thanks in advance.
>
> --Udit
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iozone test crashed on ceph

[udit agarwal]

Hi,
 thanks for your reply.
 The output of 'modinfo ceph' is as follows:
 filename:       /lib/modules/3.1.10-1.9-desktop/kernel/fs/ceph/ceph.ko
license:        GPL
description:    Ceph filesystem for Linux
author:         Patience Warnick <patience@newdream.net>
author:         Yehuda Sadeh <yehuda@hq.newdream.net>
author:         Sage Weil <sage@newdream.net>
srcversion:     AFEFF779535E750AFD4072D
depends:       
vermagic:       3.1.10-1.9-desktop SMP preempt mod_unload modversions

And my ceph.conf file is as follows:

[global]
   pid file = /var/run/ceph/$name.pid
   logger dir = /var/log/ceph
   log dir = /var/log/ceph
   user = root
[mon]
   mon data = /var/local/data/mon$id
;   debug ms = 1
;   debug mon = 20
;   debug paxos = 20
[mon.0]
   host = hp1
   mon addr = 192.168.20.6:6789
;[mon.1]
;   host = hp2
;   mon addr = 192.168.20.7:6789
;[mon.2]
;   host = bb1
;   mon addr = 192.168.20.2:6789
[mds]
;   debug ms = 1            ; message traffic
;   debug mds = 1           ; mds
;   debug mds balancer = 20 ; load balancing
;   debug mds log = 20      ; mds journaling
;   debug mds_migrator = 20 ; metadata migration
;   debug monc = 20         ; monitor interaction, startup
[mds.0]
   host = hp1
;[mds.1]
;   host = hp2
[osd]
   osd journal = /var/local/data/osd$id/journal
   osd journal size = 10000
   filestore journal writeahead = true
   osd data = /var/local/data/osd$id
;   debug ms = 1            ; message traffic
;   debug osd = 20
;   debug filestore = 20    ; local object storage
;   debug journal = 20      ; local journaling
;   debug monc = 20         ; monitor interaction, startup
[osd.0]
   host = el1
   btrfs devs = /dev/sda3
[osd.1]
   host = el1
   btrfs devs = /dev/sdb
[osd.2]
   host = bb1
   btrfs devs = /dev/sda3

No, I don't think so if any of them crashed.
Thanks in advance and let me know if you need further info.

--Udit

Re: iozone test crashed on ceph

[Greg Farnum]

On Thursday, May 31, 2012 at 5:58 PM, udit agarwal wrote:
> Hi,
> I have set up ceph system with a client, mon and mds on one system which is
> connected to 2 osds. I ran iozone test with a 10G file and it ran fine. But when
> I ran iozone test with a 5G file, the process got killed and our ceph system
> hanged. Can anyone please help me with this.

What do you mean, "the process got killed"? It hung and some task watcher killed it? Or it got OOMed?
How did you determine that the "ceph system" hung? The cluster stopped responding to requests, or just the local mount point?
-Greg

Re: iozone test crashed on ceph

[udit agarwal]

I ran the 5g iozone test on my ceph system and I got the following output on the
terminal: 

hp1:/media/ceph # /opt/iozone/bin/iozone -Rab 20120519_1641_results_5g.xls -R -i
0 -i 2 -f /media/ceph/iozone.file -s 5g
	Iozone: Performance Test of File I/O
	        Version $Revision: 3.408 $
		Compiled for 64 bit mode.
		Build: linux-AMD64 

	Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
	             Al Slater, Scott Rhine, Mike Wisner, Ken Goss
	             Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
	             Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
	             Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
	             Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
	             Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer.
	             Ben England.

	Run began: Mon Jun  4 20:27:31 2012

	Excel chart generation enabled
	Auto Mode
	Excel chart generation enabled
	File size set to 5242880 KB
	Command line used: /opt/iozone/bin/iozone -Rab 20120519_1641_results_5g.xls -R
-i 0 -i 2 -f /media/ceph/iozone.file -s 5g
	Output is in Kbytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 Kbytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
                                                            random  random   
bkwd   record   stride                                   
              KB  reclen   write rewrite    read    reread    read   write   
read  rewrite     read   fwrite frewrite   fread  freread
         5242880       4   34256   32763                  Killed

Message from syslogd@hp1 at Jun  4 22:19:03 ...
 kernel:[ 7627.132065] Oops: 0000 [#1] PREEMPT SMP 

Message from syslogd@hp1 at Jun  4 22:19:03 ...
 kernel:[ 7627.132190] Stack:

Message from syslogd@hp1 at Jun  4 22:19:03 ...
 kernel:[ 7627.132213] Call Trace:

Message from syslogd@hp1 at Jun  4 22:19:03 ...
 kernel:[ 7627.132356] Code: ab c0 fe ff ff 0f 84 85 00 00 00 31 f6 48 89 df e8
03 d7 ff ff 4c 89 ef 48 89 44 24 28 e8 f6 91 ed e0 48 8b 44 24 28 48 8b 50 08 

Message from syslogd@hp1 at Jun  4 22:19:03 ...
 kernel:[ 7627.132426] CR2: 000000000000000a

Can anybody help me with this??

--Udit Agarwal

Re: iozone test crashed on ceph

[Tommi Virtanen]

On Mon, Jun 4, 2012 at 2:52 PM, udit agarwal <fzdudit@gmail.com> wrote:
> I ran the 5g iozone test on my ceph system and I got the following output on the
> terminal:
...
> Message from syslogd@hp1 at Jun  4 22:19:03 ...
>  kernel:[ 7627.132065] Oops: 0000 [#1] PREEMPT SMP
> Message from syslogd@hp1 at Jun  4 22:19:03 ...
>  kernel:[ 7627.132190] Stack:
> Message from syslogd@hp1 at Jun  4 22:19:03 ...
>  kernel:[ 7627.132213] Call Trace:
> Message from syslogd@hp1 at Jun  4 22:19:03 ...
>  kernel:[ 7627.132356] Code: ab c0 fe ff ff 0f 84 85 00 00 00 31 f6 48 89 df e8
> 03 d7 ff ff 4c 89 ef 48 89 44 24 28 e8 f6 91 ed e0 48 8b 44 24 28 48 8b 50 08

Can you share the whole message? That part is not very useful by itself.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iozone test crashed on ceph

[udit agarwal]

Hi,
 Thanks for your reply. 
 
 Please follow this link
"https://docs.google.com/document".
"d/1mYVyI75FGMYqPes5T5fkI0aUX8h2q6TFeWdoV9uJdQI/edit?pli=1"
to find the whole message. (plz concatenate both strings for the link as I
wasn't able to post it in whole i.e. link is https://........?pli=1 .

Hope you can help me in this matter.

--Udit Agarwal

Re: iozone test crashed on ceph

[udit agarwal]

Sorry ,the link is:

https://docs.google.com/document/d/
1mYVyI75FGMYqPes5T5fkI0aUX8h2q6TFeWdoV9uJdQI/edit?pli=1

hope you can help me in this matter.

--Udit Agarwal

Re: iozone test crashed on ceph

[Tommi Virtanen]

On Mon, Jun 4, 2012 at 4:43 PM, udit agarwal <fzdudit@gmail.com> wrote:
> Sorry ,the link is: [...]

If you run iozone again, does the bug happen again?

Comparing your iozone run with our test suite, we don't currently do
-i 2 (random read/write), and we only run a few specific record sizes
to save time (-r 16K,1M); your run is talking about record len "4"
(K?) when it crashes. That might be relevant for triggering this.

Summarizing for others:

Jun  4 22:19:03 hp1 kernel: [ 7627.132026] BUG: unable to handle
kernel NULL pointer dereference at 000000000000000a
Jun  4 22:19:03 hp1 kernel: [ 7627.132036] IP: [<ffffffffa01a1fd3>]
ceph_update_writeable_page+0xe3/0x590 [ceph]
Jun  4 22:19:03 hp1 kernel: [ 7627.132058] PGD 795e3067 PUD 37673067 PMD 0
Jun  4 22:19:03 hp1 kernel: [ 7627.132065] Oops: 0000 [#1] PREEMPT SMP
...
Jun  4 22:19:03 hp1 kernel: [ 7627.132213] Call Trace:
Jun  4 22:19:03 hp1 kernel: [ 7627.132247]  [<ffffffffa01a24ec>]
ceph_write_begin+0x6c/0x100 [ceph]
Jun  4 22:19:03 hp1 kernel: [ 7627.132267]  [<ffffffff810f6652>]
generic_perform_write+0xc2/0x200
Jun  4 22:19:03 hp1 kernel: [ 7627.132277]  [<ffffffff810f67ea>]
generic_file_buffered_write+0x5a/0x90
Jun  4 22:19:03 hp1 kernel: [ 7627.132284]  [<ffffffff810f7dc9>]
__generic_file_aio_write+0x219/0x410
Jun  4 22:19:03 hp1 kernel: [ 7627.132293]  [<ffffffff810f802f>]
generic_file_aio_write+0x6f/0xf0
Jun  4 22:19:03 hp1 kernel: [ 7627.132306]  [<ffffffffa019cfbf>]
ceph_aio_write+0x2cf/0x580 [ceph]
Jun  4 22:19:03 hp1 kernel: [ 7627.132323]  [<ffffffff811533d8>]
do_sync_write+0xb8/0xf0
Jun  4 22:19:03 hp1 kernel: [ 7627.132330]  [<ffffffff81153bce>]
vfs_write+0xae/0x180
Jun  4 22:19:03 hp1 kernel: [ 7627.132337]  [<ffffffff81153ef7>]
sys_write+0x47/0x90
Jun  4 22:19:03 hp1 kernel: [ 7627.132344]  [<ffffffff815a5d12>]
system_call_fastpath+0x16/0x1b

Any takers?

Now, while we're interested in resolving this bug, we are currently
focusing on RADOS, RBD, and radosgw, and that means that the Ceph
Distributed File System is not getting as much of our time. It may
take a while for us to come back. If none of the developers see an
easy fix for this, we'll need to file it in the bug tracker and come
back to it in a few months.

Re: iozone test crashed on ceph

[udit agarwal]

Hi,
 Thanks for your reply.
 After you hinted that this problem may have caused due to using record size of
length 4k. Then, I ran iozone test on a 5m file with record length 4k and to my
surprise, the ceph system again hanged up. Then, I tried same with record
lengths from 8k to 1m and everything worked fine for these cases with 5 MB file.
Now, I have already executed the iozone test with 5G file and with record sizes
from 8k to 16m and will see what happens!!

--Udit Agarwal

Re: iozone test crashed on ceph

[udit agarwal]

Hi,
 Thank you all for your support in resolving the issue. As I mentioned in my
previous post that I ran my iozone test again on a 5G file excluding the 4k
record size. And now, I have successfully completed this test with record sizes
of 8k and 16k and hopefully all others will also work fine. But I am still
confused with the failure of the iozone test on 5G & 5M file with record size
4k, while it worked happily for files of sizes 10M & 10G.

 Can anyone help me with this problem with suitable reasons??
 Thanks in advance.

--Udit Agarwal

Re: iozone test crashed on ceph

[Tommi Virtanen]

On Tue, Jun 5, 2012 at 9:15 PM, udit agarwal <fzdudit@gmail.com> wrote:
> Hi,
>  Thank you all for your support in resolving the issue. As I mentioned in my
> previous post that I ran my iozone test again on a 5G file excluding the 4k
> record size. And now, I have successfully completed this test with record sizes
> of 8k and 16k and hopefully all others will also work fine. But I am still
> confused with the failure of the iozone test on 5G & 5M file with record size
> 4k, while it worked happily for files of sizes 10M & 10G.
>
>  Can anyone help me with this problem with suitable reasons??
>  Thanks in advance.

Smaller records means more operations which means all kinds of locking
bugs and races are more likely to manifest.

I created ticket http://tracker.newdream.net/issues/2520 for us to add
iozone 4k random read/write into our test suite.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html