Creating the filesystem and tweaking configurations

Creating the filesystem and tweaking configurations

[Steena Monteiro]

1) I used apt-get to procure and install Ceph excluding the standalone 
client on
Lucid Lynx. I obtained built the  client using instructions 
on the Ceph wiki and
it worked fine after I reverted certain commits. However, mkcephfs (from: $
/usr/sbin/mkcephfs -c /etc/ceph/ceph.conf --allhosts -v) has been breaking:

2011-03-28 13:23:23.596948 b77b16d0 OSD::mkfs: 
couldn't mount FileStore: error -95
2011-03-28 13:23:23.597008 b77b16d0  ** ERROR: 
error creating empty object store
in /data/osd0: error 95: Operation not supported
failed: '/usr/bin/cosd -c /etc/ceph/ceph.conf --monmap
/tmp/mkcephfs.DZdhzxPY5k/monmap.24380 -i 0 --mkfs --osd-data /data/osd0'

Additionally, I checked and /data/osd0 does exist...

2) I want to be able to tune Ceph's configuration 
parameters (located in
config.cc within the Ceph source) and see how each 
configuration change impacts Ceph's performance.
However, I am not certain how changes in the ceph source
code will be reflected in the standalone-client in terms of 
Ceph's performance.
Is there a suggested way I could work to accomplish changing parameters in the
source in order to have the file system impacted by these changes?

I'd appreciate your help!

Thanks in advance,
Steena

Re: Creating the filesystem and tweaking configurations

[Ben De Luca]

Last I checked the lucid package was out of date, is that still the case?

> On Mon, Mar 28, 2011 at 8:54 PM, Steena Monteiro
> <steena.m@aggiemail.usu.edu> wrote:
>>
>> 1) I used apt-get to procure and install Ceph excluding the standalone
>> client on
>> Lucid Lynx. I obtained built the  client using instructions
>> on the Ceph wiki and
>> it worked fine after I reverted certain commits. However, mkcephfs (from:
>> $
>> /usr/sbin/mkcephfs -c /etc/ceph/ceph.conf --allhosts -v) has been
>> breaking:
>>
>> 2011-03-28 13:23:23.596948 b77b16d0 OSD::mkfs:
>> couldn't mount FileStore: error -95
>> 2011-03-28 13:23:23.597008 b77b16d0  ** ERROR:
>> error creating empty object store
>> in /data/osd0: error 95: Operation not supported
>> failed: '/usr/bin/cosd -c /etc/ceph/ceph.conf --monmap
>> /tmp/mkcephfs.DZdhzxPY5k/monmap.24380 -i 0 --mkfs --osd-data /data/osd0'
>>
>> Additionally, I checked and /data/osd0 does exist...
>>
>> 2) I want to be able to tune Ceph's configuration
>> parameters (located in
>> config.cc within the Ceph source) and see how each
>> configuration change impacts Ceph's performance.
>> However, I am not certain how changes in the ceph source
>> code will be reflected in the standalone-client in terms of
>> Ceph's performance.
>> Is there a suggested way I could work to accomplish changing parameters in
>> the
>> source in order to have the file system impacted by these changes?
>>
>> I'd appreciate your help!
>>
>> Thanks in advance,
>> Steena
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Creating the filesystem and tweaking configurations

[Tommi Virtanen]

On Mon, Mar 28, 2011 at 09:00:44PM +0100, Ben De Luca wrote:
> Last I checked the lucid package was out of date, is that still the case?

I see both

http://ceph.newdream.net/debian/dists/lucid/main/binary-i386/Packages

and

http://ceph.newdream.net/debian/dists/maverick/main/binary-i386/Packages

listing a 0.25.2-1foo version, so it doesn't seem like that's true
anymore.

-- 
:(){ :|:&};:

Re: Creating the filesystem and tweaking configurations

[Ben De Luca]

aah appeared on the 23rd great :)

On Mon, Mar 28, 2011 at 9:41 PM, Tommi Virtanen
<tommi.virtanen@dreamhost.com> wrote:
> On Mon, Mar 28, 2011 at 09:00:44PM +0100, Ben De Luca wrote:
>> Last I checked the lucid package was out of date, is that still the case?
>
> I see both
>
> http://ceph.newdream.net/debian/dists/lucid/main/binary-i386/Packages
>
> and
>
> http://ceph.newdream.net/debian/dists/maverick/main/binary-i386/Packages
>
> listing a 0.25.2-1foo version, so it doesn't seem like that's true
> anymore.
>
> --
> :(){ :|:&};:
>

Re: Creating the filesystem and tweaking configurations

[Tommi Virtanen]

On Mon, Mar 28, 2011 at 07:54:13PM +0000, Steena Monteiro wrote:
> 1) I used apt-get to procure and install Ceph excluding the standalone 
> client on
> Lucid Lynx. I obtained built the  client using instructions 
> on the Ceph wiki and
> it worked fine after I reverted certain commits. However, mkcephfs (from: $
> /usr/sbin/mkcephfs -c /etc/ceph/ceph.conf --allhosts -v) has been breaking:
> 
> 2011-03-28 13:23:23.596948 b77b16d0 OSD::mkfs: 
> couldn't mount FileStore: error -95
> 2011-03-28 13:23:23.597008 b77b16d0  ** ERROR: 
> error creating empty object store
> in /data/osd0: error 95: Operation not supported
> failed: '/usr/bin/cosd -c /etc/ceph/ceph.conf --monmap
> /tmp/mkcephfs.DZdhzxPY5k/monmap.24380 -i 0 --mkfs --osd-data /data/osd0'
> 
> Additionally, I checked and /data/osd0 does exist...

Just to eliminate possible causes, can you please list the kernel
version and the ceph version you are using?

Can you show us your config file, and any logs that were written?

> 2) I want to be able to tune Ceph's configuration 
> parameters (located in
> config.cc within the Ceph source) and see how each 
> configuration change impacts Ceph's performance.
> However, I am not certain how changes in the ceph source
> code will be reflected in the standalone-client in terms of 
> Ceph's performance.
> Is there a suggested way I could work to accomplish changing parameters in the
> source in order to have the file system impacted by these changes?

Anything you see in src/common/config.cc as OPTION(...) is editable
via the config file, you don't need to change the source.

If you change the config file (across all nodes), unmount the client,
restart the whole cluster (maybe even reboot for even more predictable
benchmark results), and then mount the client again, changes will have
very definitely been forced into effect; that should give you good,
benchmarkable, numbers.

-- 
:(){ :|:&};:

Re: Creating the filesystem and tweaking configurations

[Sage Weil]

On Mon, 28 Mar 2011, Steena Monteiro wrote:
> 1) I used apt-get to procure and install Ceph excluding the standalone 
> client on
> Lucid Lynx. I obtained built the  client using instructions 
> on the Ceph wiki and
> it worked fine after I reverted certain commits. However, mkcephfs (from: $
> /usr/sbin/mkcephfs -c /etc/ceph/ceph.conf --allhosts -v) has been breaking:
> 
> 2011-03-28 13:23:23.596948 b77b16d0 OSD::mkfs: 
> couldn't mount FileStore: error -95
> 2011-03-28 13:23:23.597008 b77b16d0  ** ERROR: 
> error creating empty object store
> in /data/osd0: error 95: Operation not supported
> failed: '/usr/bin/cosd -c /etc/ceph/ceph.conf --monmap
> /tmp/mkcephfs.DZdhzxPY5k/monmap.24380 -i 0 --mkfs --osd-data /data/osd0'
> 
> Additionally, I checked and /data/osd0 does exist...

That usually means that /data/osd0 is on ext3 and you didn't mount with 
user_xattr.  The osd log should have a detailed error.

> 2) I want to be able to tune Ceph's configuration 
> parameters (located in
> config.cc within the Ceph source) and see how each 
> configuration change impacts Ceph's performance.
> However, I am not certain how changes in the ceph source
> code will be reflected in the standalone-client in terms of 
> Ceph's performance.
> Is there a suggested way I could work to accomplish changing parameters in the
> source in order to have the file system impacted by these changes?

Everything in config.cc can be set via ceph.conf, so that should save you 
some compile time.

Which tunables affect what will depend pretty heavily on the workload 
you're testing.  The things I would look at first would probably be with 
the OSD thread counts,r kernel client readahead, or some of the cache 
tunables.  IIRC I sent an annotated list of the config options with 
comments on which would be good candidates for testing.  I suspect 
focusing on something specific (say, OSD) would help get the process 
sorted out without too many variables.

sage

Re: Creating the filesystem and tweaking configurations

[Tommi Virtanen]

On Mon, Mar 28, 2011 at 03:07:58PM -0600, Steena Monteiro wrote:
> This does seem to be the case. The osd log snippet:
...
> 2011-03-28 13:23:23.597008 b77b16d0  ** ERROR: error creating empty object
> store in /data/osd0: error 95: Operation not supported
> 
> Would *mount -t 127.0.0.1:/ /ceph, user_xattr* solve this?

This is about the local filesystem as accessed by the osd.

You need either btrfs, or ext3/4 with the user_xattr mount option, on
the filesystem that contains /data/osd0.

If it's ext3/4, as it seems to be, you can do this temporarily with:

   mount -o remount,user_xattr /data/osd0

If the actual filesystem root is /data or /, adjust accordingly.

To make that permanent, you need to edit /etc/fstab.

-- 
:(){ :|:&};:

Re: Creating the filesystem and tweaking configurations

[Tommi Virtanen]

On Mon, Mar 28, 2011 at 04:33:57PM -0600, Steena Monteiro wrote:
> Thanks, that worked. However, I'm getting a mount error now when I'm finally
> trying to mount the ceph file system.
> 
> sudo mount -t ceph 127.0.0.1:/ /ceph
> mount error 5 = Input/output error
> 
> Inspite of this, I can still see data move around with ceph -w ...

Input/output error is the generic "I couldn't talk to the server"
error message from Ceph mounts, there are multiple reasons for it.
Perhaps there is something useful in the kernel logs, see
/var/log/kern.log. You may have for example configured the server to
demand authentication, and the above mount line is without
authentication.

What you see with ceph -w is the clusters internal heartbeat etc
traffic.


Also, note that we don't recommend using the kernel client on the same
machine that is serving the Ceph cluster -- there are known deadlocks
there, under memory pressure, unrelated to Ceph (NFS would trigger the
same problem). Cfuse should be safe.

-- 
:(){ :|:&};: