OOM's on the Ceph client machine

OOM's on the Ceph client machine

[Theodore Ts'o]

Hi there,

I've recently been playing with Ceph on an evaluation basis, and found
that I was able to fairly reliably induce an OOM kill on my the ceph
client machine by using FFSB with the following configuration file (see
attached, below).

I am using Ceph v0.21.3 plus a few commits that were on the testing
branch as of late September (commit ID 569d96b).  The Ceph cluster
contains 10 commodity servers with 5 disks configured for Ceph object
storage on each server (plus a separate spindle for the journal files),
so there are 5 instances of cosd on each OSD server.  The disks are
formatted using ext4 in no-journal mode.  I am using 3 servers for the
MDS and montioring daemons, with the MDS and monitoring daemons
colocated these 3 servers.  The machines all have gigabit ethernet
cards.

I've been running the client on a separate machine, and this is the
machine which has been dying with an OOM.

Any help, suggestions, or "hey stupid!  You screwed up XXXX in your
ceph.conf file" would be gratefully accepted.

Thanks,

	      	       	     	     - Ted

P.S.  In case people are curious, here are the results of the "boxacle"
(http://btrfs.boxacle.net) FFSB workloads that I ran.  The results are
fairly stable, except very often the 8 thread random_write workload is a
little hard to reproduce because it very often OOM's.  I've never gotten
a 32 thread random_write workload measurement, since it very reliably
OOM's on my client machine.  

Do these results look reasonable to you?  I confess I'm a little
disappointed with the sequential and random read numbers in particular.
And given 10 servers and fifty spindles, even the large_file_create
numbers seems surprising slow.

(Also, given the we are using gigabit ethernet in this evaluation
cluster, the 1GB/sec seems ridiculously high, which suggests to me that
the fsync request wasn't honored -- FFSB includes the fsync time when
calculating write bandwidth -- and it may explain why we are OOM'ing in
the random_write workload.)

                    1 thread           8 threads            32 threads 
large_file_create   101 MB/sec         102 MB/sec           101 MB/sec 
sequential_reads     35 MB/sec         113 MB/sec           114 MB/sec 
random_reads          1.48 MB/sec        5.44 MB/sec        11.7 MB/sec 
random_writes      923 MB/sec           1.09 GB/sec             (*) 

For comparison, here are the FFSB numbers on a single local ext4 disk
with no journal:

                    1 thread           8 threads            32 threads 
large_file_create   75.5 MB/sec        72.2 MB/sec	    74.2 MB/sec
sequential_reads    77.2 MB/sec	       69.2 MB/sec	    70.3 MB/sec
random_reads        734 K/sec	       537 K/sec	    537 K/sec
random_writes       44.5 MB/sec	       41.5 MB/sec	    41.6 MB/sec

It's very possible that I may have done something wrong, so I've
enclosed the ceph.conf file I used for doing this test run....  please
let me know if there's something I've screwed up.

---------------------------- random_write.32.ffsb
# Large file random writes.
# 1024 files, 100MB per file.

time=300  # 5 min
alignio=1

[filesystem0]
	location=/mnt/ffsb1
	num_files=1024
	min_filesize=104857600  # 100 MB
	max_filesize=104857600
	reuse=1
[end0]

[threadgroup0]
	num_threads=32

	write_random=1
	write_weight=1

	write_size=5242880  # 5 MB
	write_blocksize=4096

	[stats]
		enable_stats=1
		enable_range=1

		msec_range    0.00      0.01
		msec_range    0.01      0.02
		msec_range    0.02      0.05
		msec_range    0.05      0.10
		msec_range    0.10      0.20
		msec_range    0.20      0.50
		msec_range    0.50      1.00
		msec_range    1.00      2.00
		msec_range    2.00      5.00
		msec_range    5.00     10.00
		msec_range   10.00     20.00
		msec_range   20.00     50.00
		msec_range   50.00    100.00
		msec_range  100.00    200.00
		msec_range  200.00    500.00
		msec_range  500.00   1000.00
		msec_range 1000.00   2000.00
		msec_range 2000.00   5000.00
		msec_range 5000.00  10000.00
	[end]
[end0]
------------------------------------------------ My ceph.conf file

;
; This is the test ceph configuration file
;
; [tytso:20101007.0813EDT]
;
; This file defines cluster membership, the various locations
; that Ceph stores data, and any other runtime options.
;
; If a 'host' is defined for a daemon, the start/stop script will
; verify that it matches the hostname (or else ignore it).  If it is
; not defined, it is assumed that the daemon is intended to start on
; the current host (e.g., in a setup with a startup.conf on each
; node).

; global
[global]
	user = root
	pid file = /disk/sda3/tmp/ceph/$name.pid
	logger dir = /disk/sda3/tmp/ceph
	log dir = /disk/sda3/tmp/ceph
	chdir = /disk/sda3

; monitors
;  You need at least one.  You need at least three if you want to
;  tolerate any node failures.  Always create an odd number.
[mon]
	mon data = /disk/sda3/cephmon/data/mon$id

	; logging, for debugging monitor crashes, in order of
	; their likelihood of being helpful :)
	;debug ms = 1
	;debug mon = 20
	;debug paxos = 20
	;debug auth = 20

[mon0]
	host = mach1
	mon addr = 1.2.3.4:6789

[mon1]
	host = mach2
	mon addr = 1.2.3.5:6789

[mon1]
	host = mach3
	mon addr = 1.2.3.6:6789

; mds
;  You need at least one.  Define two to get a standby.
[mds]
	; where the mds keeps it's secret encryption keys
	keyring = /data/keyring.$name

	; mds logging to debug issues.
	;debug ms = 1
	;debug mds = 20

[mds.alpha]
	host = mach2

[mds.beta]
	host = mach3

[mds.gamma]
	host = mach1

; osd
;  You need at least one.  Two if you want data to be replicated.
;  Define as many as you like.
[osd]
	; osd logging to debug osd issues, in order of likelihood of being
	; helpful
	;debug ms = 1
	;debug osd = 20
	;debug filestore = 20
	;debug journal = 20

[osd0]
	host = mach10
	osd data = /disk/sdb3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdb3

[osd1]
	host = mach11
	osd data = /disk/sdb3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdb3

[osd2]
	host = mach12
	osd data = /disk/sdb3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdb3

[osd3]
	host = mach13
	osd data = /disk/sdb3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdb3

[osd4]
	host = mach14
	osd data = /disk/sdb3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdb3

[osd5]
	host = mach15
	osd data = /disk/sdb3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdb3

[osd6]
	host = mach16
	osd data = /disk/sdb3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdb3

[osd7]
	host = mach17
	osd data = /disk/sdb3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdb3

[osd8]
	host = mach18
	osd data = /disk/sdb3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdb3

[osd9]
	host = mach19
	osd data = /disk/sdb3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdb3

[osd10]
	host = mach10
	osd data = /disk/sdd3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdd3

[osd11]
	host = mach11
	osd data = /disk/sdd3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdd3

[osd12]
	host = mach12
	osd data = /disk/sdd3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdd3

[osd13]
	host = mach13
	osd data = /disk/sdd3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdd3

[osd14]
	host = mach14
	osd data = /disk/sdd3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdd3

[osd15]
	host = mach15
	osd data = /disk/sdd3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdd3

[osd16]
	host = mach16
	osd data = /disk/sdd3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdd3

[osd17]
	host = mach17
	osd data = /disk/sdd3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdd3

[osd18]
	host = mach18
	osd data = /disk/sdd3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdd3

[osd19]
	host = mach19
	osd data = /disk/sdd3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdd3

[osd20]
	host = mach10
	osd data = /disk/sde3/cephdata
	osd journal = /disk/sdc3/cephjnl.sde3

[osd21]
	host = mach11
	osd data = /disk/sde3/cephdata
	osd journal = /disk/sdc3/cephjnl.sde3

[osd22]
	host = mach12
	osd data = /disk/sde3/cephdata
	osd journal = /disk/sdc3/cephjnl.sde3

[osd23]
	host = mach13
	osd data = /disk/sde3/cephdata
	osd journal = /disk/sdc3/cephjnl.sde3

[osd24]
	host = mach14
	osd data = /disk/sde3/cephdata
	osd journal = /disk/sdc3/cephjnl.sde3

[osd25]
	host = mach15
	osd data = /disk/sde3/cephdata
	osd journal = /disk/sdc3/cephjnl.sde3

[osd26]
	host = mach16
	osd data = /disk/sde3/cephdata
	osd journal = /disk/sdc3/cephjnl.sde3

[osd27]
	host = mach17
	osd data = /disk/sde3/cephdata
	osd journal = /disk/sdc3/cephjnl.sde3

[osd28]
	host = mach18
	osd data = /disk/sde3/cephdata
	osd journal = /disk/sdc3/cephjnl.sde3

[osd29]
	host = mach19
	osd data = /disk/sde3/cephdata
	osd journal = /disk/sdc3/cephjnl.sde3

[osd30]
	host = mach10
	osd data = /disk/sdf3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdf3

[osd31]
	host = mach11
	osd data = /disk/sdf3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdf3

[osd32]
	host = mach12
	osd data = /disk/sdf3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdf3

[osd33]
	host = mach13
	osd data = /disk/sdf3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdf3

[osd34]
	host = mach14
	osd data = /disk/sdf3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdf3

[osd35]
	host = mach15
	osd data = /disk/sdf3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdf3

[osd36]
	host = mach16
	osd data = /disk/sdf3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdf3

[osd37]
	host = mach17
	osd data = /disk/sdf3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdf3

[osd38]
	host = mach18
	osd data = /disk/sdf3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdf3

[osd39]
	host = mach19
	osd data = /disk/sdf3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdf3

[osd40]
	host = mach10
	osd data = /disk/sdg3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdg3

[osd41]
	host = mach11
	osd data = /disk/sdg3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdg3

[osd42]
	host = mach12
	osd data = /disk/sdg3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdg3

[osd43]
	host = mach13
	osd data = /disk/sdg3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdg3

[osd44]
	host = mach14
	osd data = /disk/sdg3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdg3

[osd45]
	host = mach15
	osd data = /disk/sdg3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdg3

[osd46]
	host = mach16
	osd data = /disk/sdg3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdg3

[osd47]
	host = mach17
	osd data = /disk/sdg3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdg3

[osd48]
	host = mach18
	osd data = /disk/sdg3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdg3

[osd49]
	host = mach19
	osd data = /disk/sdg3/cephdata
	osd journal = /disk/sdc3/cephjnl.sdg3

Re: OOM's on the Ceph client machine

[Gregory Farnum]

On Tue, Oct 12, 2010 at 5:31 PM, Theodore Ts'o <tytso@mit.edu> wrote:
> Hi there,
>
> I've recently been playing with Ceph on an evaluation basis, and found
> that I was able to fairly reliably induce an OOM kill on my the ceph
> client machine by using FFSB with the following configuration file (see
> attached, below).
Does this mean you're using cfuse rather than the kernel client?
FUSE performance in general is fairly disappointing and our cfuse is
probably not as fast as the kernel client even so, though I don't
think it should be *that* unhappy in most environments.

> I am using Ceph v0.21.3 plus a few commits that were on the testing
> branch as of late September (commit ID 569d96b).  The Ceph cluster
> contains 10 commodity servers with 5 disks configured for Ceph object
> storage on each server (plus a separate spindle for the journal files),
> so there are 5 instances of cosd on each OSD server.  The disks are
> formatted using ext4 in no-journal mode.  I am using 3 servers for the
> MDS and montioring daemons, with the MDS and monitoring daemons
> colocated these 3 servers.  The machines all have gigabit ethernet
> cards.
So you have 5 journals running on one spindle? This could be the cause
of your slightly low sequential write performance; in the current
default configuration writes have to go to the journal before going to
the main disk and with multiple OSDs on one journal spindle they could
be getting in each other's way.
Also, how much memory do you have on these machines?

> P.S.  In case people are curious, here are the results of the "boxacle"
> (http://btrfs.boxacle.net) FFSB workloads that I ran.  The results are
> fairly stable, except very often the 8 thread random_write workload is a
> little hard to reproduce because it very often OOM's.  I've never gotten
> a 32 thread random_write workload measurement, since it very reliably
> OOM's on my client machine.
>
> Do these results look reasonable to you?  I confess I'm a little
> disappointed with the sequential and random read numbers in particular.
> And given 10 servers and fifty spindles, even the large_file_create
> numbers seems surprising slow.
I'm not familiar with FFSB and there doesn't seem to be any
easily-accessible documentation, can you tell us a little more about
how it works? For instance, how are the test files created (are they
written out for the reads and then tested? Does the random write
create the files as it goes, or are they pre-existing and then
overwritten)?
A few thoughts/wild guesses:
I'm not sure exactly what the limit is, but 114MB/s reads are close to
what you can get over a 1Gb link.
If single-threaded FFSB means there's only one request in-flight at a
time there may be a latency issue which is causing those 35MB/s reads.
The kernel client ought to be prefetching but maybe it's not doing so
properly, and I don't recall how much prefetching cfuse is actually
capable of. Sage can say more on this.

> (Also, given the we are using gigabit ethernet in this evaluation
> cluster, the 1GB/sec seems ridiculously high, which suggests to me that
> the fsync request wasn't honored -- FFSB includes the fsync time when
> calculating write bandwidth -- and it may explain why we are OOM'ing in
> the random_write workload.)
Err, yes. Extremely odd. In glancing over cfuse this looks like it's
working properly, but if you confirm that's what you're using I'll
trace it.
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OOM's on the Ceph client machine

[Ted Ts'o]

On Tue, Oct 12, 2010 at 07:30:48PM -0700, Gregory Farnum wrote:
> Does this mean you're using cfuse rather than the kernel client?
> FUSE performance in general is fairly disappointing and our cfuse is
> probably not as fast as the kernel client even so, though I don't
> think it should be *that* unhappy in most environments.

No, I'm using the kernel client (from 2.6.34).  Specifically, I'm
doing a "modprobe ceph; mount -t ceph 1.2.3.4:/ /mnt"

Sorry, I should have mentioned that.  I can use a more recent kernel
(i.e., 2.6.36-rc7) if that's likely to help.

> So you have 5 journals running on one spindle? This could be the cause
> of your slightly low sequential write performance; in the current
> default configuration writes have to go to the journal before going to
> the main disk and with multiple OSDs on one journal spindle they could
> be getting in each other's way.

Hmm, what do you recommend, then?  The problem is if the journal only
needs to be a few gigabytes (I used a 5GB file), using an entire 1T or
2T disk just so each of the journals can have their own spindle is
pretty wasteful.

> Also, how much memory do you have on these machines?

32GB

> I'm not familiar with FFSB and there doesn't seem to be any
> easily-accessible documentation, can you tell us a little more about
> how it works? For instance, how are the test files created (are they
> written out for the reads and then tested? Does the random write
> create the files as it goes, or are they pre-existing and then
> overwritten)?

There's a quicky explanation of these workloads at
http://btrfs.boxacle.net (I'm using the raid configuration FFSB
files), but essentially, the large file create test is creating 100MB
files as quickly as possible.  In the rest of the tests we create 1024
100MB files, and then try (a) reading from them sequentially as
quickly as possible, (b) picking a random file, and a random offset,
and read 5MB, and repeat, (c) picking a random file, and a random
offset, and write 5MB, and repeat.  The creation of the 1024 100MB
files (if necessary; the tests will reuse the previously created set
of 100MB files) is not counted in the benchmark time.  So in the last
three tests there is no block allocation; just the time it takes to
read or overwrite existing data blocks.

Note BTW that this is not intrinsic to FFSB; FFSB stands for the
"flexible filesystem benchmark" system.  All of this is configurable
using the ffsb config files.  I'm just reusing the "boxacle workloads"
just because they are convenient, and I'm familiar with how they work
on local disk filesystems.  They're used for example for benchmarking
ext4 here: http://free.linux.hp.com/~enw/ext4/2.6.35/, and for btrfs
here: http://btrfs.boxacle.net.  (And when the IBM folks have done
btrfs benchmarks, since they are so detailed and the
hardware/configurations are so well described, I've also used them to
help improve ext4's performance.)

> A few thoughts/wild guesses:
> I'm not sure exactly what the limit is, but 114MB/s reads are close to
> what you can get over a 1Gb link.
> If single-threaded FFSB means there's only one request in-flight at a
> time there may be a latency issue which is causing those 35MB/s reads.
> The kernel client ought to be prefetching but maybe it's not doing so
> properly, and I don't recall how much prefetching cfuse is actually
> capable of. Sage can say more on this.

I'm not using cfuse; I'm using the in-kernel Ceph module.  As far as
network latency is concerned, ping RTT time is under 0.25ms.

And sure maybe it's a prefetching issue --- but in that case I would
have expected 8 thread would have had better than 2x the 1 thread case.

     	      	       	     	      	     - Ted

Re: OOM's on the Ceph client machine

[Sage Weil]

Hi Ted,

On Tue, 12 Oct 2010, Ted Ts'o wrote:
> On Tue, Oct 12, 2010 at 07:30:48PM -0700, Gregory Farnum wrote:
> > Does this mean you're using cfuse rather than the kernel client?
> > FUSE performance in general is fairly disappointing and our cfuse is
> > probably not as fast as the kernel client even so, though I don't
> > think it should be *that* unhappy in most environments.
> 
> No, I'm using the kernel client (from 2.6.34).  Specifically, I'm
> doing a "modprobe ceph; mount -t ceph 1.2.3.4:/ /mnt"
> 
> Sorry, I should have mentioned that.  I can use a more recent kernel
> (i.e., 2.6.36-rc7) if that's likely to help.

There have been a number of memory leak fixes since then, at least one of 
which may be causing your problem (it was caused by an uninitialized 
variable and didn't usually trigger for us, but may in your environment).  
Can you retry with the latest mainline?  The benchmark completes without 
problems in my test environment.

> > So you have 5 journals running on one spindle? This could be the cause
> > of your slightly low sequential write performance; in the current
> > default configuration writes have to go to the journal before going to
> > the main disk and with multiple OSDs on one journal spindle they could
> > be getting in each other's way.
> 
> Hmm, what do you recommend, then?  The problem is if the journal only
> needs to be a few gigabytes (I used a 5GB file), using an entire 1T or
> 2T disk just so each of the journals can have their own spindle is
> pretty wasteful.

If fsync on a single file in journal-less ext4 doesn't do any extra work, 
I would just put the (preallocated) journal file together with the data on 
each disk.  Usually that's bad news because of the journal flushing, but 
you shouldn't have that problem.  Alternatively, you could use a small 
separate partition on the same spindle.  

sage

Re: OOM's on the Ceph client machine

[Ted Ts'o]

On Wed, Oct 13, 2010 at 10:29:43AM -0700, Sage Weil wrote:
> There have been a number of memory leak fixes since then, at least one of 
> which may be causing your problem (it was caused by an uninitialized 
> variable and didn't usually trigger for us, but may in your environment).  
> Can you retry with the latest mainline?  The benchmark completes without 
> problems in my test environment.

Sure.  This may have to wait until early next week for me to retry
with the latest mainline, but I'll definitely move to 2.6.36 in the
near future.

> If fsync on a single file in journal-less ext4 doesn't do any extra work, 
> I would just put the (preallocated) journal file together with the data on 
> each disk.  Usually that's bad news because of the journal flushing, but 
> you shouldn't have that problem.  Alternatively, you could use a small 
> separate partition on the same spindle.

I'm currently reformatting the Ceph cluster to put the journal for
/dev/sdX3 on /disk/sdX3/ceph.journal, so I'll try that test first, and
see what difference that makes.  That way I can make one change at a
time and see what difference each change in my cluster configuration
actually gives me.

BTW, this might be a good time to report a tiny little problem which I
found.  If the journal file doesn't exist, then when you run mkcephfs,
cosd will attempt to create the file for you.  But it creates it as a
4k file, and then it loops forever in FileJournal::wrap_read_bl() on
line 808, because get_top() and and header.max_size are both 4096, and
it results in it being an expensive while (1) loop.  This completely
stalls the mkcephfs operation, and it took me a while to debug.

It might be nice if cosd either (a) failed completely if the journal
file is missing, or too small, or (b) if cosd is started in mkfs mode,
and the journal file does not exist, perhaps it should create a
journal file with some suitable default size.

For stuff like this, I assume the right thing to do is to just open a
bug in tracker.newdream.net?  Is there any project-specific customs I
should be aware of?

Thanks,

						- Ted

Re: OOM's on the Ceph client machine

[Sage Weil]

On Wed, 13 Oct 2010, Ted Ts'o wrote:
> On Wed, Oct 13, 2010 at 10:29:43AM -0700, Sage Weil wrote:
> > There have been a number of memory leak fixes since then, at least one of 
> > which may be causing your problem (it was caused by an uninitialized 
> > variable and didn't usually trigger for us, but may in your environment).  
> > Can you retry with the latest mainline?  The benchmark completes without 
> > problems in my test environment.
> 
> Sure.  This may have to wait until early next week for me to retry
> with the latest mainline, but I'll definitely move to 2.6.36 in the
> near future.
> 
> > If fsync on a single file in journal-less ext4 doesn't do any extra work, 
> > I would just put the (preallocated) journal file together with the data on 
> > each disk.  Usually that's bad news because of the journal flushing, but 
> > you shouldn't have that problem.  Alternatively, you could use a small 
> > separate partition on the same spindle.
> 
> I'm currently reformatting the Ceph cluster to put the journal for
> /dev/sdX3 on /disk/sdX3/ceph.journal, so I'll try that test first, and
> see what difference that makes.  That way I can make one change at a
> time and see what difference each change in my cluster configuration
> actually gives me.

Sounds good!

> BTW, this might be a good time to report a tiny little problem which I
> found.  If the journal file doesn't exist, then when you run mkcephfs,
> cosd will attempt to create the file for you.  But it creates it as a
> 4k file, and then it loops forever in FileJournal::wrap_read_bl() on
> line 808, because get_top() and and header.max_size are both 4096, and
> it results in it being an expensive while (1) loop.  This completely
> stalls the mkcephfs operation, and it took me a while to debug.

That likely explains the hang Martin saw a few days back, and why we 
haven't hit it (our journal files usually already exist).  Thanks for 
tracking that down!
 
> It might be nice if cosd either (a) failed completely if the journal
> file is missing, or too small, or (b) if cosd is started in mkfs mode,
> and the journal file does not exist, perhaps it should create a
> journal file with some suitable default size.
> 
> For stuff like this, I assume the right thing to do is to just open a
> bug in tracker.newdream.net?  Is there any project-specific customs I
> should be aware of?

Yeah, entering it directly in the tracker is nice, although just reporting 
it here is fine as well.  I went ahead and added this one (#487).

Thanks!
sage

Re: OOM's on the Ceph client machine

[Ted Ts'o]

On Wed, Oct 13, 2010 at 08:03:06PM -0400, Ted Ts'o wrote:
> On Wed, Oct 13, 2010 at 10:29:43AM -0700, Sage Weil wrote:
> > There have been a number of memory leak fixes since then, at least one of 
> > which may be causing your problem (it was caused by an uninitialized 
> > variable and didn't usually trigger for us, but may in your environment).  
> > Can you retry with the latest mainline?  The benchmark completes without 
> > problems in my test environment.
> 
> Sure.  This may have to wait until early next week for me to retry
> with the latest mainline, but I'll definitely move to 2.6.36 in the
> near future.

Just to give you an update.  I've tried to use 2.6.34 with nearly all
of the commits that apply to fs/ceph between 2.6.34 and 2.6.36-rc7
both with the 0.21 version of Ceph servers, as well as 0.22 plus some
testing bug fixes (up to fd42c852).  In both cases, using newer Ceph
client causes the FFSB process to hang when it tries running the sync
command.  The dmesg is filled with lines like this:

[ 4756.662789] ceph: skipping osd40 192.168.11.8:6808 seq 2495, expected 2496
[ 4756.662832] ceph: skipping osd7 192.168.12.18:6800 seq 4274, expected 4275
[ 4756.662843] ceph: skipping osd14 192.168.12.15:6802 seq 4124, expected 4125
[ 4756.662853] ceph: skipping osd38 192.168.11.3:6806 seq 3289, expected 3290
[ 4756.663093] ceph: skipping osd7 192.168.12.18:6800 seq 4275, expected 4276
[ 4756.882336] ceph: skipping osd7 192.168.12.18:6800 seq 4276, expected 4277
[ 4757.996962] ceph: skipping osd40 192.168.11.8:6808 seq 2496, expected 2497
[ 4757.997267] ceph: skipping osd7 192.168.12.18:6800 seq 4277, expected 4278
[ 4758.000149] ceph: skipping osd38 192.168.11.3:6806 seq 3290, expected 3291
[ 4758.003755] ceph: skipping osd14 192.168.12.15:6802 seq 4125, expected 4126
[ 4758.018078] ceph: skipping osd14 192.168.12.15:6802 seq 4126, expected 4127
[ 4758.018787] ceph: skipping osd7 192.168.12.18:6800 seq 4278, expected 4279
[ 4758.020263] ceph: skipping osd40 192.168.11.8:6808 seq 2497, expected 2498
[ 4758.020370] ceph: skipping osd10 192.168.11.8:6802 seq 946, expected 947
[ 4761.670848] ceph:  tid 4422463 timed out on osd7, will reset osd
[ 4761.813068] ceph:  tid 4480042 timed out on osd40, will reset osd
[ 4761.956584] ceph:  tid 4487615 timed out on osd14, will reset osd
[ 4762.102343] ceph:  tid 4645028 timed out on osd38, will reset osd
[ 4762.249425] ceph: skipping osd10 192.168.11.8:6802 seq 947, expected 948
[ 4767.257944] ceph: skipping osd10 192.168.11.8:6802 seq 948, expected 949
[ 4768.047058] ceph: skipping osd10 192.168.11.8:6802 seq 949, expected 950
[ 4772.260309] ceph:  tid 4817033 timed out on osd10, will reset osd

It's very possible (likely, even) that this was caused by my backwards
porting of the various ceph patches to 2.6.34.  Hopefully later today
I'll be able to do an actual test run using 2.6.36, without needing to
use "git cherry-pick" on some 170 odd patches.  For a variety of
reasons it was easier for me to use 2.6.34 as a base (drivers, patches
that support dmesg dumps over the network after kernel panic/oops, and
other stuff needed for our environment) but I should be able to move
to 2.6.36 soon.

I also ran into strange problems (which I haven't tried to
characterize accurately enough for a bug report) when using the 2.6.34
client against the new 0.22 release.  Is this expected to work?  If
so, I can try to more accurately characterize what was going on.  

Also, It seems that there are issues moving back and forth between
0.21 and 0.22 without reformating the ceph client.  Is that accurate?
It looked like when I tried going back to 0.21, I needed to rerun
mkcephfs, or else the 0.21 cmon, cosd or cmds daemons would die with
various failures when they saw that 0.22 data files.  That's not
surprising, but it does make it a little harder for me to go back and
forth between 0.21 and 0.22 for the purpose of differential debugging.

If I can get something stable working with 0.22 against either the
2.6.34 or 2.6.36 Ceph client, I'll drop my efforts using 0.21.

Thanks, regards,

	      	      	       	    	    - Ted

Re: OOM's on the Ceph client machine

[Sage Weil]

On Thu, 21 Oct 2010, Ted Ts'o wrote:
> On Wed, Oct 13, 2010 at 08:03:06PM -0400, Ted Ts'o wrote:
> > On Wed, Oct 13, 2010 at 10:29:43AM -0700, Sage Weil wrote:
> > > There have been a number of memory leak fixes since then, at least one of 
> > > which may be causing your problem (it was caused by an uninitialized 
> > > variable and didn't usually trigger for us, but may in your environment).  
> > > Can you retry with the latest mainline?  The benchmark completes without 
> > > problems in my test environment.
> > 
> > Sure.  This may have to wait until early next week for me to retry
> > with the latest mainline, but I'll definitely move to 2.6.36 in the
> > near future.
> 
> Just to give you an update.  I've tried to use 2.6.34 with nearly all
> of the commits that apply to fs/ceph between 2.6.34 and 2.6.36-rc7
> both with the 0.21 version of Ceph servers, as well as 0.22 plus some
> testing bug fixes (up to fd42c852).  In both cases, using newer Ceph
> client causes the FFSB process to hang when it tries running the sync
> command.  The dmesg is filled with lines like this:
> 
> [ 4756.662789] ceph: skipping osd40 192.168.11.8:6808 seq 2495, expected 2496
> [ 4756.662832] ceph: skipping osd7 192.168.12.18:6800 seq 4274, expected 4275
> [ 4756.662843] ceph: skipping osd14 192.168.12.15:6802 seq 4124, expected 4125
> [ 4756.662853] ceph: skipping osd38 192.168.11.3:6806 seq 3289, expected 3290
> [ 4756.663093] ceph: skipping osd7 192.168.12.18:6800 seq 4275, expected 4276
> [ 4756.882336] ceph: skipping osd7 192.168.12.18:6800 seq 4276, expected 4277
> [ 4757.996962] ceph: skipping osd40 192.168.11.8:6808 seq 2496, expected 2497
> [ 4757.997267] ceph: skipping osd7 192.168.12.18:6800 seq 4277, expected 4278
> [ 4758.000149] ceph: skipping osd38 192.168.11.3:6806 seq 3290, expected 3291
> [ 4758.003755] ceph: skipping osd14 192.168.12.15:6802 seq 4125, expected 4126
> [ 4758.018078] ceph: skipping osd14 192.168.12.15:6802 seq 4126, expected 4127
> [ 4758.018787] ceph: skipping osd7 192.168.12.18:6800 seq 4278, expected 4279
> [ 4758.020263] ceph: skipping osd40 192.168.11.8:6808 seq 2497, expected 2498
> [ 4758.020370] ceph: skipping osd10 192.168.11.8:6802 seq 946, expected 947
> [ 4761.670848] ceph:  tid 4422463 timed out on osd7, will reset osd
> [ 4761.813068] ceph:  tid 4480042 timed out on osd40, will reset osd
> [ 4761.956584] ceph:  tid 4487615 timed out on osd14, will reset osd
> [ 4762.102343] ceph:  tid 4645028 timed out on osd38, will reset osd
> [ 4762.249425] ceph: skipping osd10 192.168.11.8:6802 seq 947, expected 948
> [ 4767.257944] ceph: skipping osd10 192.168.11.8:6802 seq 948, expected 949
> [ 4768.047058] ceph: skipping osd10 192.168.11.8:6802 seq 949, expected 950
> [ 4772.260309] ceph:  tid 4817033 timed out on osd10, will reset osd
> 
> It's very possible (likely, even) that this was caused by my backwards
> porting of the various ceph patches to 2.6.34.  Hopefully later today
> I'll be able to do an actual test run using 2.6.36, without needing to
> use "git cherry-pick" on some 170 odd patches.  For a variety of
> reasons it was easier for me to use 2.6.34 as a base (drivers, patches
> that support dmesg dumps over the network after kernel panic/oops, and
> other stuff needed for our environment) but I should be able to move
> to 2.6.36 soon.

There is a ceph-client-standalone.git that has just the module source, 
with backport #ifdefs through 2.6.27 (see the master-backport or 
unstable-backport branches).  It isn't well tested, but may be worth a 
shot if 2.6.36 is problematic for other reasons.

Unfortunately it's not obvious to me from dmesg where the problem is, 
other than that it looks like some of the osds aren't responding (but are 
apparently still up).  There is a known regression in v0.22 that can cause 
crashes in the osd cluster; we should have a fix pushed later today.  
That would look a bit different, though (you'd see osd down messages).  
I'll post an update (and probably v0.22.1) when that's been tested.

> I also ran into strange problems (which I haven't tried to
> characterize accurately enough for a bug report) when using the 2.6.34
> client against the new 0.22 release.  Is this expected to work?  If
> so, I can try to more accurately characterize what was going on.  

The vanilla 2.6.34 client you mean?  There have been a range of bugs fixed 
since then (enough for me to lose track of), so I wouldn't be surpised to 
see problems.  And it's not something we've been testing.  That said, the 
basics should work.

> Also, It seems that there are issues moving back and forth between
> 0.21 and 0.22 without reformating the ceph client.  Is that accurate?

Yeah, that isn't expected to work.  In general, rolling backward isn't 
supported.  In this case we forgot to add an incompat flag to generate a 
nice error message to that effect.

sage

Re: OOM's on the Ceph client machine

[Ted Ts'o]

On Thu, Oct 21, 2010 at 02:46:11PM -0700, Sage Weil wrote:
> 
> Unfortunately it's not obvious to me from dmesg where the problem is, 
> other than that it looks like some of the osds aren't responding (but are 
> apparently still up).  There is a known regression in v0.22 that can cause 
> crashes in the osd cluster; we should have a fix pushed later today.  
> That would look a bit different, though (you'd see osd down messages).  
> I'll post an update (and probably v0.22.1) when that's been tested.

I looked earlier in the logs, and I do see some "osd down", "osd up",
and "osd socket closed" messages.  So it looks like the v0.22
regression you mentioned.  I'll wait for the git update and try
rebuilding the server.  Thanks!!

> > Also, It seems that there are issues moving back and forth between
> > 0.21 and 0.22 without reformating the ceph client.  Is that accurate?
> 
> Yeah, that isn't expected to work.  In general, rolling backward isn't 
> supported.  In this case we forgot to add an incompat flag to generate a 
> nice error message to that effect.

Is rolling forward between 0.21 and 0.22 expected to work?  Or should
I just do a mkcephfs just to be safe?  It's not a data preservation
issue, but rather the time it takes to do a mkcephfs.  Random
question: how do you feel about using Python?  Trying to make a
version of mkcephfs that runs in parallel would probably be easier if
we could port the shell script to a python script.  I don't think
there are any Python dependencies in Ceph right now, though.

      	      	     		     	  - Ted

Re: OOM's on the Ceph client machine

[Sage Weil]

On Thu, 21 Oct 2010, Ted Ts'o wrote:
> On Thu, Oct 21, 2010 at 02:46:11PM -0700, Sage Weil wrote:
> > 
> > Unfortunately it's not obvious to me from dmesg where the problem is, 
> > other than that it looks like some of the osds aren't responding (but are 
> > apparently still up).  There is a known regression in v0.22 that can cause 
> > crashes in the osd cluster; we should have a fix pushed later today.  
> > That would look a bit different, though (you'd see osd down messages).  
> > I'll post an update (and probably v0.22.1) when that's been tested.
> 
> I looked earlier in the logs, and I do see some "osd down", "osd up",
> and "osd socket closed" messages.  So it looks like the v0.22
> regression you mentioned.  I'll wait for the git update and try
> rebuilding the server.  Thanks!!

Phew!  :)

> > > Also, It seems that there are issues moving back and forth between
> > > 0.21 and 0.22 without reformating the ceph client.  Is that accurate?
> > 
> > Yeah, that isn't expected to work.  In general, rolling backward isn't 
> > supported.  In this case we forgot to add an incompat flag to generate a 
> > nice error message to that effect.
> 
> Is rolling forward between 0.21 and 0.22 expected to work?  Or should
> I just do a mkcephfs just to be safe?  It's not a data preservation
> issue, but rather the time it takes to do a mkcephfs. 

Rolling forward is always supposed to work.  (And if we do end up changing 
things in a non-backward compatible way, we'll make some noise about it.)

> Random
> question: how do you feel about using Python?  Trying to make a
> version of mkcephfs that runs in parallel would probably be easier if
> we could port the shell script to a python script.  I don't think
> there are any Python dependencies in Ceph right now, though.

Python's fine.  There's an issue in the tracker relating to this, btw.  
The goal will be to create discrete steps that let you use whatever 
cluster-specific tools you have for launching parallel jobs. 
	http://tracker.newdream.net/issues/400

sage

Re: OOM's on the Ceph client machine

[DongJin Lee]

Hi Ted:

I'd like to follow your similar setup, too.
At this stage, I'm with the very recent version, I've tried btrfs but
ceph mount freezes as soon as I run any high benchmark or heavy iops.
I think it is to do with syncing, so I'm now trying ext4 without the
journal, hoping for some good news.

1. have you tried btrfs, with all configs the same?
2. did you use mkfs.ext4 -O ^has_journal to disable? (sorry, you know
the most since you are the ext4 man!)
3. did the change of journal size in ceph.conf changed any results?
e.g., 20MB to 1000MB?

Have you had any success in other benchmarks? I'd really like to know
if you could try 'fio' benchmark, if your time allows.
run ./fio example_file

and the example_file content is below, e.g.,

[global]
bs=4k
ioengine=libaio
iodepth=1
size=1g
direct=1
runtime=30
filename=/media/cephmount/afile

[seq-read]
  rw=read
  stonewall
[rand-read]
  rw=randread
  stonewall
[seq-write]
  rw=write
  stonewall
[rand-write]
  rw=randwrite
  stonewall

thanks a lot.

On Wed, Oct 13, 2010 at 1:31 PM, Theodore Ts'o <tytso@mit.edu> wrote:
> Hi there,
>
> I've recently been playing with Ceph on an evaluation basis, and found
> that I was able to fairly reliably induce an OOM kill on my the ceph
> client machine by using FFSB with the following configuration file (see
> attached, below).
>
> I am using Ceph v0.21.3 plus a few commits that were on the testing
> branch as of late September (commit ID 569d96b).  The Ceph cluster
> contains 10 commodity servers with 5 disks configured for Ceph object
> storage on each server (plus a separate spindle for the journal files),
> so there are 5 instances of cosd on each OSD server.  The disks are
> formatted using ext4 in no-journal mode.  I am using 3 servers for the
> MDS and montioring daemons, with the MDS and monitoring daemons
> colocated these 3 servers.  The machines all have gigabit ethernet
> cards.
>
> I've been running the client on a separate machine, and this is the
> machine which has been dying with an OOM.
>
> Any help, suggestions, or "hey stupid!  You screwed up XXXX in your
> ceph.conf file" would be gratefully accepted.
>
> Thanks,
>
>                                     - Ted
>
> P.S.  In case people are curious, here are the results of the "boxacle"
> (http://btrfs.boxacle.net) FFSB workloads that I ran.  The results are
> fairly stable, except very often the 8 thread random_write workload is a
> little hard to reproduce because it very often OOM's.  I've never gotten
> a 32 thread random_write workload measurement, since it very reliably
> OOM's on my client machine.
>
> Do these results look reasonable to you?  I confess I'm a little
> disappointed with the sequential and random read numbers in particular.
> And given 10 servers and fifty spindles, even the large_file_create
> numbers seems surprising slow.
>
> (Also, given the we are using gigabit ethernet in this evaluation
> cluster, the 1GB/sec seems ridiculously high, which suggests to me that
> the fsync request wasn't honored -- FFSB includes the fsync time when
> calculating write bandwidth -- and it may explain why we are OOM'ing in
> the random_write workload.)
>
>                    1 thread           8 threads            32 threads
> large_file_create   101 MB/sec         102 MB/sec           101 MB/sec
> sequential_reads     35 MB/sec         113 MB/sec           114 MB/sec
> random_reads          1.48 MB/sec        5.44 MB/sec        11.7 MB/sec
> random_writes      923 MB/sec           1.09 GB/sec             (*)
>
> For comparison, here are the FFSB numbers on a single local ext4 disk
> with no journal:
>
>                    1 thread           8 threads            32 threads
> large_file_create   75.5 MB/sec        72.2 MB/sec          74.2 MB/sec
> sequential_reads    77.2 MB/sec        69.2 MB/sec          70.3 MB/sec
> random_reads        734 K/sec          537 K/sec            537 K/sec
> random_writes       44.5 MB/sec        41.5 MB/sec          41.6 MB/sec
>
> It's very possible that I may have done something wrong, so I've
> enclosed the ceph.conf file I used for doing this test run....  please
> let me know if there's something I've screwed up.
>
> ---------------------------- random_write.32.ffsb
> # Large file random writes.
> # 1024 files, 100MB per file.
>
> time=300  # 5 min
> alignio=1
>
> [filesystem0]
>        location=/mnt/ffsb1
>        num_files=1024
>        min_filesize=104857600  # 100 MB
>        max_filesize=104857600
>        reuse=1
> [end0]
>
> [threadgroup0]
>        num_threads=32
>
>        write_random=1
>        write_weight=1
>
>        write_size=5242880  # 5 MB
>        write_blocksize=4096
>
>        [stats]
>                enable_stats=1
>                enable_range=1
>
>                msec_range    0.00      0.01
>                msec_range    0.01      0.02
>                msec_range    0.02      0.05
>                msec_range    0.05      0.10
>                msec_range    0.10      0.20
>                msec_range    0.20      0.50
>                msec_range    0.50      1.00
>                msec_range    1.00      2.00
>                msec_range    2.00      5.00
>                msec_range    5.00     10.00
>                msec_range   10.00     20.00
>                msec_range   20.00     50.00
>                msec_range   50.00    100.00
>                msec_range  100.00    200.00
>                msec_range  200.00    500.00
>                msec_range  500.00   1000.00
>                msec_range 1000.00   2000.00
>                msec_range 2000.00   5000.00
>                msec_range 5000.00  10000.00
>        [end]
> [end0]
> ------------------------------------------------ My ceph.conf file
>
> ;
> ; This is the test ceph configuration file
> ;
> ; [tytso:20101007.0813EDT]
> ;
> ; This file defines cluster membership, the various locations
> ; that Ceph stores data, and any other runtime options.
> ;
> ; If a 'host' is defined for a daemon, the start/stop script will
> ; verify that it matches the hostname (or else ignore it).  If it is
> ; not defined, it is assumed that the daemon is intended to start on
> ; the current host (e.g., in a setup with a startup.conf on each
> ; node).
>
> ; global
> [global]
>        user = root
>        pid file = /disk/sda3/tmp/ceph/$name.pid
>        logger dir = /disk/sda3/tmp/ceph
>        log dir = /disk/sda3/tmp/ceph
>        chdir = /disk/sda3
>
> ; monitors
> ;  You need at least one.  You need at least three if you want to
> ;  tolerate any node failures.  Always create an odd number.
> [mon]
>        mon data = /disk/sda3/cephmon/data/mon$id
>
>        ; logging, for debugging monitor crashes, in order of
>        ; their likelihood of being helpful :)
>        ;debug ms = 1
>        ;debug mon = 20
>        ;debug paxos = 20
>        ;debug auth = 20
>
> [mon0]
>        host = mach1
>        mon addr = 1.2.3.4:6789
>
> [mon1]
>        host = mach2
>        mon addr = 1.2.3.5:6789
>
> [mon1]
>        host = mach3
>        mon addr = 1.2.3.6:6789
>
> ; mds
> ;  You need at least one.  Define two to get a standby.
> [mds]
>        ; where the mds keeps it's secret encryption keys
>        keyring = /data/keyring.$name
>
>        ; mds logging to debug issues.
>        ;debug ms = 1
>        ;debug mds = 20
>
> [mds.alpha]
>        host = mach2
>
> [mds.beta]
>        host = mach3
>
> [mds.gamma]
>        host = mach1
>
> ; osd
> ;  You need at least one.  Two if you want data to be replicated.
> ;  Define as many as you like.
> [osd]
>        ; osd logging to debug osd issues, in order of likelihood of being
>        ; helpful
>        ;debug ms = 1
>        ;debug osd = 20
>        ;debug filestore = 20
>        ;debug journal = 20
>
> [osd0]
>        host = mach10
>        osd data = /disk/sdb3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdb3
>
> [osd1]
>        host = mach11
>        osd data = /disk/sdb3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdb3
>
> [osd2]
>        host = mach12
>        osd data = /disk/sdb3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdb3
>
> [osd3]
>        host = mach13
>        osd data = /disk/sdb3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdb3
>
> [osd4]
>        host = mach14
>        osd data = /disk/sdb3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdb3
>
> [osd5]
>        host = mach15
>        osd data = /disk/sdb3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdb3
>
> [osd6]
>        host = mach16
>        osd data = /disk/sdb3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdb3
>
> [osd7]
>        host = mach17
>        osd data = /disk/sdb3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdb3
>
> [osd8]
>        host = mach18
>        osd data = /disk/sdb3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdb3
>
> [osd9]
>        host = mach19
>        osd data = /disk/sdb3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdb3
>
> [osd10]
>        host = mach10
>        osd data = /disk/sdd3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdd3
>
> [osd11]
>        host = mach11
>        osd data = /disk/sdd3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdd3
>
> [osd12]
>        host = mach12
>        osd data = /disk/sdd3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdd3
>
> [osd13]
>        host = mach13
>        osd data = /disk/sdd3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdd3
>
> [osd14]
>        host = mach14
>        osd data = /disk/sdd3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdd3
>
> [osd15]
>        host = mach15
>        osd data = /disk/sdd3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdd3
>
> [osd16]
>        host = mach16
>        osd data = /disk/sdd3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdd3
>
> [osd17]
>        host = mach17
>        osd data = /disk/sdd3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdd3
>
> [osd18]
>        host = mach18
>        osd data = /disk/sdd3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdd3
>
> [osd19]
>        host = mach19
>        osd data = /disk/sdd3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdd3
>
> [osd20]
>        host = mach10
>        osd data = /disk/sde3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sde3
>
> [osd21]
>        host = mach11
>        osd data = /disk/sde3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sde3
>
> [osd22]
>        host = mach12
>        osd data = /disk/sde3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sde3
>
> [osd23]
>        host = mach13
>        osd data = /disk/sde3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sde3
>
> [osd24]
>        host = mach14
>        osd data = /disk/sde3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sde3
>
> [osd25]
>        host = mach15
>        osd data = /disk/sde3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sde3
>
> [osd26]
>        host = mach16
>        osd data = /disk/sde3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sde3
>
> [osd27]
>        host = mach17
>        osd data = /disk/sde3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sde3
>
> [osd28]
>        host = mach18
>        osd data = /disk/sde3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sde3
>
> [osd29]
>        host = mach19
>        osd data = /disk/sde3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sde3
>
> [osd30]
>        host = mach10
>        osd data = /disk/sdf3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdf3
>
> [osd31]
>        host = mach11
>        osd data = /disk/sdf3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdf3
>
> [osd32]
>        host = mach12
>        osd data = /disk/sdf3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdf3
>
> [osd33]
>        host = mach13
>        osd data = /disk/sdf3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdf3
>
> [osd34]
>        host = mach14
>        osd data = /disk/sdf3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdf3
>
> [osd35]
>        host = mach15
>        osd data = /disk/sdf3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdf3
>
> [osd36]
>        host = mach16
>        osd data = /disk/sdf3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdf3
>
> [osd37]
>        host = mach17
>        osd data = /disk/sdf3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdf3
>
> [osd38]
>        host = mach18
>        osd data = /disk/sdf3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdf3
>
> [osd39]
>        host = mach19
>        osd data = /disk/sdf3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdf3
>
> [osd40]
>        host = mach10
>        osd data = /disk/sdg3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdg3
>
> [osd41]
>        host = mach11
>        osd data = /disk/sdg3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdg3
>
> [osd42]
>        host = mach12
>        osd data = /disk/sdg3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdg3
>
> [osd43]
>        host = mach13
>        osd data = /disk/sdg3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdg3
>
> [osd44]
>        host = mach14
>        osd data = /disk/sdg3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdg3
>
> [osd45]
>        host = mach15
>        osd data = /disk/sdg3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdg3
>
> [osd46]
>        host = mach16
>        osd data = /disk/sdg3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdg3
>
> [osd47]
>        host = mach17
>        osd data = /disk/sdg3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdg3
>
> [osd48]
>        host = mach18
>        osd data = /disk/sdg3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdg3
>
> [osd49]
>        host = mach19
>        osd data = /disk/sdg3/cephdata
>        osd journal = /disk/sdc3/cephjnl.sdg3
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OOM's on the Ceph client machine

[Sage Weil]

Hi Ted,

On Tue, 12 Oct 2010, Theodore Ts'o wrote:
> P.S.  In case people are curious, here are the results of the "boxacle"
> (http://btrfs.boxacle.net) FFSB workloads that I ran.  The results are
> fairly stable, except very often the 8 thread random_write workload is a
> little hard to reproduce because it very often OOM's.  I've never gotten
> a 32 thread random_write workload measurement, since it very reliably
> OOM's on my client machine.  
> 
> Do these results look reasonable to you?  I confess I'm a little
> disappointed with the sequential and random read numbers in particular.
> And given 10 servers and fifty spindles, even the large_file_create
> numbers seems surprising slow.
> 
> (Also, given the we are using gigabit ethernet in this evaluation
> cluster, the 1GB/sec seems ridiculously high, which suggests to me that
> the fsync request wasn't honored -- FFSB includes the fsync time when
> calculating write bandwidth -- and it may explain why we are OOM'ing in
> the random_write workload.)
> 
>                     1 thread           8 threads            32 threads 
> large_file_create   101 MB/sec         102 MB/sec           101 MB/sec 

These may be a bit below the ceiling imposed by the gigabit ethernet 
because of the combined journaling disk; effectively all writes for the 
whole host were going to the same spindle.  Please try distributing the 
journals across the spindles.

> sequential_reads     35 MB/sec         113 MB/sec           114 MB/sec 

These are mostly reasonable.  The single thread performance is primarily 
governed by the MM readahead behavior.  There is a mount option tunable to 
adjust the max readahead on the BDI: rsize=<bytes> (the default is only 
512KB, IIRC).  Some users have reported improved read performance with a 
larger rsize, but it's not something we've had time to tune ourselves.

> random_reads          1.48 MB/sec        5.44 MB/sec        11.7 MB/sec 

This one looks way too slow.  I'm going to run this locally and see what 
is going on.

> random_writes      923 MB/sec           1.09 GB/sec             (*) 

And there is definitely something wrong here with the client.  :)  Let's 
see what happens with the latest mainline!

sage


> 
> For comparison, here are the FFSB numbers on a single local ext4 disk
> with no journal:
> 
>                     1 thread           8 threads            32 threads 
> large_file_create   75.5 MB/sec        72.2 MB/sec	    74.2 MB/sec
> sequential_reads    77.2 MB/sec	       69.2 MB/sec	    70.3 MB/sec
> random_reads        734 K/sec	       537 K/sec	    537 K/sec
> random_writes       44.5 MB/sec	       41.5 MB/sec	    41.6 MB/sec
> 
> It's very possible that I may have done something wrong, so I've
> enclosed the ceph.conf file I used for doing this test run....  please
> let me know if there's something I've screwed up.
> 
> ---------------------------- random_write.32.ffsb
> # Large file random writes.
> # 1024 files, 100MB per file.
> 
> time=300  # 5 min
> alignio=1
> 
> [filesystem0]
> 	location=/mnt/ffsb1
> 	num_files=1024
> 	min_filesize=104857600  # 100 MB
> 	max_filesize=104857600
> 	reuse=1
> [end0]
> 
> [threadgroup0]
> 	num_threads=32
> 
> 	write_random=1
> 	write_weight=1
> 
> 	write_size=5242880  # 5 MB
> 	write_blocksize=4096
> 
> 	[stats]
> 		enable_stats=1
> 		enable_range=1
> 
> 		msec_range    0.00      0.01
> 		msec_range    0.01      0.02
> 		msec_range    0.02      0.05
> 		msec_range    0.05      0.10
> 		msec_range    0.10      0.20
> 		msec_range    0.20      0.50
> 		msec_range    0.50      1.00
> 		msec_range    1.00      2.00
> 		msec_range    2.00      5.00
> 		msec_range    5.00     10.00
> 		msec_range   10.00     20.00
> 		msec_range   20.00     50.00
> 		msec_range   50.00    100.00
> 		msec_range  100.00    200.00
> 		msec_range  200.00    500.00
> 		msec_range  500.00   1000.00
> 		msec_range 1000.00   2000.00
> 		msec_range 2000.00   5000.00
> 		msec_range 5000.00  10000.00
> 	[end]
> [end0]
> ------------------------------------------------ My ceph.conf file
> 
> ;
> ; This is the test ceph configuration file
> ;
> ; [tytso:20101007.0813EDT]
> ;
> ; This file defines cluster membership, the various locations
> ; that Ceph stores data, and any other runtime options.
> ;
> ; If a 'host' is defined for a daemon, the start/stop script will
> ; verify that it matches the hostname (or else ignore it).  If it is
> ; not defined, it is assumed that the daemon is intended to start on
> ; the current host (e.g., in a setup with a startup.conf on each
> ; node).
> 
> ; global
> [global]
> 	user = root
> 	pid file = /disk/sda3/tmp/ceph/$name.pid
> 	logger dir = /disk/sda3/tmp/ceph
> 	log dir = /disk/sda3/tmp/ceph
> 	chdir = /disk/sda3
> 
> ; monitors
> ;  You need at least one.  You need at least three if you want to
> ;  tolerate any node failures.  Always create an odd number.
> [mon]
> 	mon data = /disk/sda3/cephmon/data/mon$id
> 
> 	; logging, for debugging monitor crashes, in order of
> 	; their likelihood of being helpful :)
> 	;debug ms = 1
> 	;debug mon = 20
> 	;debug paxos = 20
> 	;debug auth = 20
> 
> [mon0]
> 	host = mach1
> 	mon addr = 1.2.3.4:6789
> 
> [mon1]
> 	host = mach2
> 	mon addr = 1.2.3.5:6789
> 
> [mon1]
> 	host = mach3
> 	mon addr = 1.2.3.6:6789
> 
> ; mds
> ;  You need at least one.  Define two to get a standby.
> [mds]
> 	; where the mds keeps it's secret encryption keys
> 	keyring = /data/keyring.$name
> 
> 	; mds logging to debug issues.
> 	;debug ms = 1
> 	;debug mds = 20
> 
> [mds.alpha]
> 	host = mach2
> 
> [mds.beta]
> 	host = mach3
> 
> [mds.gamma]
> 	host = mach1
> 
> ; osd
> ;  You need at least one.  Two if you want data to be replicated.
> ;  Define as many as you like.
> [osd]
> 	; osd logging to debug osd issues, in order of likelihood of being
> 	; helpful
> 	;debug ms = 1
> 	;debug osd = 20
> 	;debug filestore = 20
> 	;debug journal = 20
> 
> [osd0]
> 	host = mach10
> 	osd data = /disk/sdb3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdb3
> 
> [osd1]
> 	host = mach11
> 	osd data = /disk/sdb3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdb3
> 
> [osd2]
> 	host = mach12
> 	osd data = /disk/sdb3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdb3
> 
> [osd3]
> 	host = mach13
> 	osd data = /disk/sdb3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdb3
> 
> [osd4]
> 	host = mach14
> 	osd data = /disk/sdb3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdb3
> 
> [osd5]
> 	host = mach15
> 	osd data = /disk/sdb3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdb3
> 
> [osd6]
> 	host = mach16
> 	osd data = /disk/sdb3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdb3
> 
> [osd7]
> 	host = mach17
> 	osd data = /disk/sdb3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdb3
> 
> [osd8]
> 	host = mach18
> 	osd data = /disk/sdb3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdb3
> 
> [osd9]
> 	host = mach19
> 	osd data = /disk/sdb3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdb3
> 
> [osd10]
> 	host = mach10
> 	osd data = /disk/sdd3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdd3
> 
> [osd11]
> 	host = mach11
> 	osd data = /disk/sdd3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdd3
> 
> [osd12]
> 	host = mach12
> 	osd data = /disk/sdd3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdd3
> 
> [osd13]
> 	host = mach13
> 	osd data = /disk/sdd3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdd3
> 
> [osd14]
> 	host = mach14
> 	osd data = /disk/sdd3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdd3
> 
> [osd15]
> 	host = mach15
> 	osd data = /disk/sdd3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdd3
> 
> [osd16]
> 	host = mach16
> 	osd data = /disk/sdd3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdd3
> 
> [osd17]
> 	host = mach17
> 	osd data = /disk/sdd3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdd3
> 
> [osd18]
> 	host = mach18
> 	osd data = /disk/sdd3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdd3
> 
> [osd19]
> 	host = mach19
> 	osd data = /disk/sdd3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdd3
> 
> [osd20]
> 	host = mach10
> 	osd data = /disk/sde3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sde3
> 
> [osd21]
> 	host = mach11
> 	osd data = /disk/sde3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sde3
> 
> [osd22]
> 	host = mach12
> 	osd data = /disk/sde3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sde3
> 
> [osd23]
> 	host = mach13
> 	osd data = /disk/sde3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sde3
> 
> [osd24]
> 	host = mach14
> 	osd data = /disk/sde3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sde3
> 
> [osd25]
> 	host = mach15
> 	osd data = /disk/sde3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sde3
> 
> [osd26]
> 	host = mach16
> 	osd data = /disk/sde3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sde3
> 
> [osd27]
> 	host = mach17
> 	osd data = /disk/sde3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sde3
> 
> [osd28]
> 	host = mach18
> 	osd data = /disk/sde3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sde3
> 
> [osd29]
> 	host = mach19
> 	osd data = /disk/sde3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sde3
> 
> [osd30]
> 	host = mach10
> 	osd data = /disk/sdf3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdf3
> 
> [osd31]
> 	host = mach11
> 	osd data = /disk/sdf3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdf3
> 
> [osd32]
> 	host = mach12
> 	osd data = /disk/sdf3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdf3
> 
> [osd33]
> 	host = mach13
> 	osd data = /disk/sdf3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdf3
> 
> [osd34]
> 	host = mach14
> 	osd data = /disk/sdf3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdf3
> 
> [osd35]
> 	host = mach15
> 	osd data = /disk/sdf3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdf3
> 
> [osd36]
> 	host = mach16
> 	osd data = /disk/sdf3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdf3
> 
> [osd37]
> 	host = mach17
> 	osd data = /disk/sdf3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdf3
> 
> [osd38]
> 	host = mach18
> 	osd data = /disk/sdf3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdf3
> 
> [osd39]
> 	host = mach19
> 	osd data = /disk/sdf3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdf3
> 
> [osd40]
> 	host = mach10
> 	osd data = /disk/sdg3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdg3
> 
> [osd41]
> 	host = mach11
> 	osd data = /disk/sdg3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdg3
> 
> [osd42]
> 	host = mach12
> 	osd data = /disk/sdg3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdg3
> 
> [osd43]
> 	host = mach13
> 	osd data = /disk/sdg3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdg3
> 
> [osd44]
> 	host = mach14
> 	osd data = /disk/sdg3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdg3
> 
> [osd45]
> 	host = mach15
> 	osd data = /disk/sdg3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdg3
> 
> [osd46]
> 	host = mach16
> 	osd data = /disk/sdg3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdg3
> 
> [osd47]
> 	host = mach17
> 	osd data = /disk/sdg3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdg3
> 
> [osd48]
> 	host = mach18
> 	osd data = /disk/sdg3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdg3
> 
> [osd49]
> 	host = mach19
> 	osd data = /disk/sdg3/cephdata
> 	osd journal = /disk/sdc3/cephjnl.sdg3
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: OOM's on the Ceph client machine

[Sage Weil]

On Wed, 13 Oct 2010, Sage Weil wrote:
> On Tue, 12 Oct 2010, Theodore Ts'o wrote:
> > random_reads          1.48 MB/sec        5.44 MB/sec        11.7 MB/sec 
> 
> This one looks way too slow.  I'm going to run this locally and see what 
> is going on.

I looked closer at this one, and it looks like what ffsb is doing is each 
thread picks a random 5 MB chunk and does 4KB reads from within that chunk 
at random offsets.  Because the reads are random, there's no readahead, 
and we have lots of little 4KB read requests going over the wire.  
Increasing the number of threads just means more small reads in parallel.

That being the case, the single thread number isn't so surprising. 
Performance is mainly bounded by the request latency.  What is a bit 
surprising is that it doesn't scale that well as threads increase, I 
assume because of some contention on the OSDs (balancing is pseudorandom).  
FWIW, in my environment (25 single spindle OSDs, btrfs) for random_reads 
and 1/8/32 threads I got

random_reads	4.1 MB/sec	7.41 MB/sec	15.3MB/sec

sage


> 
> > random_writes      923 MB/sec           1.09 GB/sec             (*) 
> 
> And there is definitely something wrong here with the client.  :)  Let's 
> see what happens with the latest mainline!
> 
> sage
> 
> 
> > 
> > For comparison, here are the FFSB numbers on a single local ext4 disk
> > with no journal:
> > 
> >                     1 thread           8 threads            32 threads 
> > large_file_create   75.5 MB/sec        72.2 MB/sec	    74.2 MB/sec
> > sequential_reads    77.2 MB/sec	       69.2 MB/sec	    70.3 MB/sec
> > random_reads        734 K/sec	       537 K/sec	    537 K/sec
> > random_writes       44.5 MB/sec	       41.5 MB/sec	    41.6 MB/sec
> > 
> > It's very possible that I may have done something wrong, so I've
> > enclosed the ceph.conf file I used for doing this test run....  please
> > let me know if there's something I've screwed up.
> > 
> > ---------------------------- random_write.32.ffsb
> > # Large file random writes.
> > # 1024 files, 100MB per file.
> > 
> > time=300  # 5 min
> > alignio=1
> > 
> > [filesystem0]
> > 	location=/mnt/ffsb1
> > 	num_files=1024
> > 	min_filesize=104857600  # 100 MB
> > 	max_filesize=104857600
> > 	reuse=1
> > [end0]
> > 
> > [threadgroup0]
> > 	num_threads=32
> > 
> > 	write_random=1
> > 	write_weight=1
> > 
> > 	write_size=5242880  # 5 MB
> > 	write_blocksize=4096
> > 
> > 	[stats]
> > 		enable_stats=1
> > 		enable_range=1
> > 
> > 		msec_range    0.00      0.01
> > 		msec_range    0.01      0.02
> > 		msec_range    0.02      0.05
> > 		msec_range    0.05      0.10
> > 		msec_range    0.10      0.20
> > 		msec_range    0.20      0.50
> > 		msec_range    0.50      1.00
> > 		msec_range    1.00      2.00
> > 		msec_range    2.00      5.00
> > 		msec_range    5.00     10.00
> > 		msec_range   10.00     20.00
> > 		msec_range   20.00     50.00
> > 		msec_range   50.00    100.00
> > 		msec_range  100.00    200.00
> > 		msec_range  200.00    500.00
> > 		msec_range  500.00   1000.00
> > 		msec_range 1000.00   2000.00
> > 		msec_range 2000.00   5000.00
> > 		msec_range 5000.00  10000.00
> > 	[end]
> > [end0]
> > ------------------------------------------------ My ceph.conf file
> > 
> > ;
> > ; This is the test ceph configuration file
> > ;
> > ; [tytso:20101007.0813EDT]
> > ;
> > ; This file defines cluster membership, the various locations
> > ; that Ceph stores data, and any other runtime options.
> > ;
> > ; If a 'host' is defined for a daemon, the start/stop script will
> > ; verify that it matches the hostname (or else ignore it).  If it is
> > ; not defined, it is assumed that the daemon is intended to start on
> > ; the current host (e.g., in a setup with a startup.conf on each
> > ; node).
> > 
> > ; global
> > [global]
> > 	user = root
> > 	pid file = /disk/sda3/tmp/ceph/$name.pid
> > 	logger dir = /disk/sda3/tmp/ceph
> > 	log dir = /disk/sda3/tmp/ceph
> > 	chdir = /disk/sda3
> > 
> > ; monitors
> > ;  You need at least one.  You need at least three if you want to
> > ;  tolerate any node failures.  Always create an odd number.
> > [mon]
> > 	mon data = /disk/sda3/cephmon/data/mon$id
> > 
> > 	; logging, for debugging monitor crashes, in order of
> > 	; their likelihood of being helpful :)
> > 	;debug ms = 1
> > 	;debug mon = 20
> > 	;debug paxos = 20
> > 	;debug auth = 20
> > 
> > [mon0]
> > 	host = mach1
> > 	mon addr = 1.2.3.4:6789
> > 
> > [mon1]
> > 	host = mach2
> > 	mon addr = 1.2.3.5:6789
> > 
> > [mon1]
> > 	host = mach3
> > 	mon addr = 1.2.3.6:6789
> > 
> > ; mds
> > ;  You need at least one.  Define two to get a standby.
> > [mds]
> > 	; where the mds keeps it's secret encryption keys
> > 	keyring = /data/keyring.$name
> > 
> > 	; mds logging to debug issues.
> > 	;debug ms = 1
> > 	;debug mds = 20
> > 
> > [mds.alpha]
> > 	host = mach2
> > 
> > [mds.beta]
> > 	host = mach3
> > 
> > [mds.gamma]
> > 	host = mach1
> > 
> > ; osd
> > ;  You need at least one.  Two if you want data to be replicated.
> > ;  Define as many as you like.
> > [osd]
> > 	; osd logging to debug osd issues, in order of likelihood of being
> > 	; helpful
> > 	;debug ms = 1
> > 	;debug osd = 20
> > 	;debug filestore = 20
> > 	;debug journal = 20
> > 
> > [osd0]
> > 	host = mach10
> > 	osd data = /disk/sdb3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdb3
> > 
> > [osd1]
> > 	host = mach11
> > 	osd data = /disk/sdb3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdb3
> > 
> > [osd2]
> > 	host = mach12
> > 	osd data = /disk/sdb3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdb3
> > 
> > [osd3]
> > 	host = mach13
> > 	osd data = /disk/sdb3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdb3
> > 
> > [osd4]
> > 	host = mach14
> > 	osd data = /disk/sdb3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdb3
> > 
> > [osd5]
> > 	host = mach15
> > 	osd data = /disk/sdb3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdb3
> > 
> > [osd6]
> > 	host = mach16
> > 	osd data = /disk/sdb3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdb3
> > 
> > [osd7]
> > 	host = mach17
> > 	osd data = /disk/sdb3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdb3
> > 
> > [osd8]
> > 	host = mach18
> > 	osd data = /disk/sdb3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdb3
> > 
> > [osd9]
> > 	host = mach19
> > 	osd data = /disk/sdb3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdb3
> > 
> > [osd10]
> > 	host = mach10
> > 	osd data = /disk/sdd3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdd3
> > 
> > [osd11]
> > 	host = mach11
> > 	osd data = /disk/sdd3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdd3
> > 
> > [osd12]
> > 	host = mach12
> > 	osd data = /disk/sdd3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdd3
> > 
> > [osd13]
> > 	host = mach13
> > 	osd data = /disk/sdd3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdd3
> > 
> > [osd14]
> > 	host = mach14
> > 	osd data = /disk/sdd3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdd3
> > 
> > [osd15]
> > 	host = mach15
> > 	osd data = /disk/sdd3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdd3
> > 
> > [osd16]
> > 	host = mach16
> > 	osd data = /disk/sdd3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdd3
> > 
> > [osd17]
> > 	host = mach17
> > 	osd data = /disk/sdd3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdd3
> > 
> > [osd18]
> > 	host = mach18
> > 	osd data = /disk/sdd3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdd3
> > 
> > [osd19]
> > 	host = mach19
> > 	osd data = /disk/sdd3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdd3
> > 
> > [osd20]
> > 	host = mach10
> > 	osd data = /disk/sde3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sde3
> > 
> > [osd21]
> > 	host = mach11
> > 	osd data = /disk/sde3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sde3
> > 
> > [osd22]
> > 	host = mach12
> > 	osd data = /disk/sde3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sde3
> > 
> > [osd23]
> > 	host = mach13
> > 	osd data = /disk/sde3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sde3
> > 
> > [osd24]
> > 	host = mach14
> > 	osd data = /disk/sde3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sde3
> > 
> > [osd25]
> > 	host = mach15
> > 	osd data = /disk/sde3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sde3
> > 
> > [osd26]
> > 	host = mach16
> > 	osd data = /disk/sde3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sde3
> > 
> > [osd27]
> > 	host = mach17
> > 	osd data = /disk/sde3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sde3
> > 
> > [osd28]
> > 	host = mach18
> > 	osd data = /disk/sde3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sde3
> > 
> > [osd29]
> > 	host = mach19
> > 	osd data = /disk/sde3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sde3
> > 
> > [osd30]
> > 	host = mach10
> > 	osd data = /disk/sdf3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdf3
> > 
> > [osd31]
> > 	host = mach11
> > 	osd data = /disk/sdf3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdf3
> > 
> > [osd32]
> > 	host = mach12
> > 	osd data = /disk/sdf3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdf3
> > 
> > [osd33]
> > 	host = mach13
> > 	osd data = /disk/sdf3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdf3
> > 
> > [osd34]
> > 	host = mach14
> > 	osd data = /disk/sdf3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdf3
> > 
> > [osd35]
> > 	host = mach15
> > 	osd data = /disk/sdf3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdf3
> > 
> > [osd36]
> > 	host = mach16
> > 	osd data = /disk/sdf3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdf3
> > 
> > [osd37]
> > 	host = mach17
> > 	osd data = /disk/sdf3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdf3
> > 
> > [osd38]
> > 	host = mach18
> > 	osd data = /disk/sdf3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdf3
> > 
> > [osd39]
> > 	host = mach19
> > 	osd data = /disk/sdf3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdf3
> > 
> > [osd40]
> > 	host = mach10
> > 	osd data = /disk/sdg3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdg3
> > 
> > [osd41]
> > 	host = mach11
> > 	osd data = /disk/sdg3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdg3
> > 
> > [osd42]
> > 	host = mach12
> > 	osd data = /disk/sdg3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdg3
> > 
> > [osd43]
> > 	host = mach13
> > 	osd data = /disk/sdg3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdg3
> > 
> > [osd44]
> > 	host = mach14
> > 	osd data = /disk/sdg3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdg3
> > 
> > [osd45]
> > 	host = mach15
> > 	osd data = /disk/sdg3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdg3
> > 
> > [osd46]
> > 	host = mach16
> > 	osd data = /disk/sdg3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdg3
> > 
> > [osd47]
> > 	host = mach17
> > 	osd data = /disk/sdg3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdg3
> > 
> > [osd48]
> > 	host = mach18
> > 	osd data = /disk/sdg3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdg3
> > 
> > [osd49]
> > 	host = mach19
> > 	osd data = /disk/sdg3/cephdata
> > 	osd journal = /disk/sdc3/cephjnl.sdg3
> > 
> > 
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> > 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
>