ceph stability

ceph stability

[Roman Hlynovskiy]

Hello,

I have 2 issues with ceph stability and looking for help to resolve them.
My setup is pretty simple - 3 debian 32bit stable systems each running
osd, mon and mds.
the conf is the following:
--------------------
[global]
    auth cluster required = none
    auth service required = none
    auth client required = none

[osd]
    osd journal size = 1000
    filestore xattr use omap = true

[mon.a]
    host = ceph-node01
    mon addr = 192.168.7.11:6789

[mon.b]
    host = ceph-node02
    mon addr = 192.168.7.12:6789

[mon.c]
        host = ceph-node03
        mon addr = 192.168.7.13:6789

[mds.a]
        host = ceph-node01

[mds.b]
        host = ceph-node02

[mds.c]
        host = ceph-node03

[osd.0]
    host = ceph-node01

[osd.1]
    host = ceph-node02

[osd.2]
    host = ceph-node03
--------------------
ceph -s is:
   health HEALTH_OK
   monmap e4: 3 mons at
{a=192.168.7.11:6789/0,b=192.168.7.12:6789/0,c=192.168.7.13:6789/0},
election epoch 118, quorum 0,1,2 a,b,c
   osdmap e197: 3 osds: 3 up, 3 in
    pgmap v43305: 384 pgs: 384 active+clean; 72351 MB data, 144 GB
used, 105 GB / 249 GB avail
   mdsmap e4439: 1/1/1 up {0=a=up:active}, 2 up:standby
--------------------

My first problem - I am getting spurious mon's deaths, which usually
looks like this:

--- begin dump of recent events ---
     0> 2012-12-19 10:35:58.912119 b41eab70 -1 *** Caught signal (Aborted) **
 in thread b41eab70

 ceph version 0.55.1 (8e25c8d984f9258644389a18997ec6bdef8e056b)
 1: /usr/bin/ceph-mon() [0x8183a11]
 2: [0xb7714400]
 3: (gsignal()+0x47) [0xb7337577]
 4: (abort()+0x182) [0xb733a962]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x14f) [0xb755653f]
 6: (()+0xbd405) [0xb7554405]
 7: (()+0xbd442) [0xb7554442]
 8: (()+0xbd581) [0xb7554581]
 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x80f) [0x824cabf]
 10: /usr/bin/ceph-mon() [0x80e3c1d]
 11: (MDSMonitor::tick()+0x1e3b) [0x811ea0b]
 12: (MDSMonitor::on_active()+0x1d) [0x81188dd]
 13: (PaxosService::_active()+0x212) [0x80e4b02]
 14: (Context::complete(int)+0x19) [0x80c4cf9]
 15: (finish_contexts(CephContext*, std::list<Context*,
std::allocator<Context*> >&, int)+0x13f) [0x80d208f]
 16: (Monitor::recovered_leader(int)+0x3ac) [0x80ac5ac]
 17: (Paxos::handle_last(MMonPaxos*)+0xb02) [0x80e0572]
 18: (Paxos::dispatch(PaxosServiceMessage*)+0x2c4) [0x80e0e94]
 19: (Monitor::_ms_dispatch(Message*)+0x1181) [0x80c3b11]
 20: (Monitor::ms_dispatch(Message*)+0x31) [0x80d5021]
 21: (DispatchQueue::entry()+0x337) [0x82afa47]
 22: (DispatchQueue::DispatchThread::entry()+0x20) [0x823eec0]
 23: (Thread::_entry_func(void*)+0x11) [0x824be41]
 24: (()+0x57b0) [0xb75ef7b0]
 25: (clone()+0x5e) [0xb73d8cde]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   0/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   0/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/ 5 hadoop
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent    100000
  max_new         1000
  log_file /var/log/ceph/ceph-mon.a.log
--- end dump of recent events ---

the binaries are coming from ceph.com debian-testing repo.

My second problem - I have 2 systems which mount ceph. Whenever I
mount ceph on any other system it usually mounts but get stuck on
stat* operations (i.e. simple ls -al will hang with read( from the
ceph-mounted directory for ages). This kind of client stuck also
affects two working clients. they also start to stuck on the stat*
even after shutdown of the third client. so usually umount/mount or
even reboot for existing clients solves the issue)


--
...WBR, Roman Hlynovskiy

Re: ceph stability

[Joao Eduardo Luis]

On 12/19/2012 09:03 AM, Roman Hlynovskiy wrote:
> My first problem - I am getting spurious mon's deaths, which usually
> looks like this:
>
> --- begin dump of recent events ---
>       0> 2012-12-19 10:35:58.912119 b41eab70 -1 *** Caught signal (Aborted) **
>   in thread b41eab70
>
>   ceph version 0.55.1 (8e25c8d984f9258644389a18997ec6bdef8e056b)
>   1: /usr/bin/ceph-mon() [0x8183a11]
>   2: [0xb7714400]
>   3: (gsignal()+0x47) [0xb7337577]
>   4: (abort()+0x182) [0xb733a962]
>   5: (__gnu_cxx::__verbose_terminate_handler()+0x14f) [0xb755653f]
>   6: (()+0xbd405) [0xb7554405]
>   7: (()+0xbd442) [0xb7554442]
>   8: (()+0xbd581) [0xb7554581]
>   9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x80f) [0x824cabf]
>   10: /usr/bin/ceph-mon() [0x80e3c1d]
>   11: (MDSMonitor::tick()+0x1e3b) [0x811ea0b]
>   12: (MDSMonitor::on_active()+0x1d) [0x81188dd]
>   13: (PaxosService::_active()+0x212) [0x80e4b02]
>   14: (Context::complete(int)+0x19) [0x80c4cf9]
>   15: (finish_contexts(CephContext*, std::list<Context*,
> std::allocator<Context*> >&, int)+0x13f) [0x80d208f]
>   16: (Monitor::recovered_leader(int)+0x3ac) [0x80ac5ac]
>   17: (Paxos::handle_last(MMonPaxos*)+0xb02) [0x80e0572]
>   18: (Paxos::dispatch(PaxosServiceMessage*)+0x2c4) [0x80e0e94]
>   19: (Monitor::_ms_dispatch(Message*)+0x1181) [0x80c3b11]
>   20: (Monitor::ms_dispatch(Message*)+0x31) [0x80d5021]
>   21: (DispatchQueue::entry()+0x337) [0x82afa47]
>   22: (DispatchQueue::DispatchThread::entry()+0x20) [0x823eec0]
>   23: (Thread::_entry_func(void*)+0x11) [0x824be41]
>   24: (()+0x57b0) [0xb75ef7b0]
>   25: (clone()+0x5e) [0xb73d8cde]
>   NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> needed to interpret this.

http://tracker.newdream.net/issues/3495

Should be fixed in latest master, but the latest patch didn't make it to 
v0.55.1.

   -Joao

Re: ceph stability

[Roman Hlynovskiy]

Hello Joao,

 thanks for feedback. is this fix available on the svn? i can provide
heavy testing for it.

2012/12/19 Joao Eduardo Luis <joao.luis@inktank.com>:
> On 12/19/2012 09:03 AM, Roman Hlynovskiy wrote:
>>
>> My first problem - I am getting spurious mon's deaths, which usually
>> looks like this:
>>
>> --- begin dump of recent events ---
>>       0> 2012-12-19 10:35:58.912119 b41eab70 -1 *** Caught signal
>> (Aborted) **
>>   in thread b41eab70
>>
>>   ceph version 0.55.1 (8e25c8d984f9258644389a18997ec6bdef8e056b)
>>   1: /usr/bin/ceph-mon() [0x8183a11]
>>   2: [0xb7714400]
>>   3: (gsignal()+0x47) [0xb7337577]
>>   4: (abort()+0x182) [0xb733a962]
>>   5: (__gnu_cxx::__verbose_terminate_handler()+0x14f) [0xb755653f]
>>   6: (()+0xbd405) [0xb7554405]
>>   7: (()+0xbd442) [0xb7554442]
>>   8: (()+0xbd581) [0xb7554581]
>>   9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x80f) [0x824cabf]
>>   10: /usr/bin/ceph-mon() [0x80e3c1d]
>>   11: (MDSMonitor::tick()+0x1e3b) [0x811ea0b]
>>   12: (MDSMonitor::on_active()+0x1d) [0x81188dd]
>>   13: (PaxosService::_active()+0x212) [0x80e4b02]
>>   14: (Context::complete(int)+0x19) [0x80c4cf9]
>>   15: (finish_contexts(CephContext*, std::list<Context*,
>> std::allocator<Context*> >&, int)+0x13f) [0x80d208f]
>>   16: (Monitor::recovered_leader(int)+0x3ac) [0x80ac5ac]
>>   17: (Paxos::handle_last(MMonPaxos*)+0xb02) [0x80e0572]
>>   18: (Paxos::dispatch(PaxosServiceMessage*)+0x2c4) [0x80e0e94]
>>   19: (Monitor::_ms_dispatch(Message*)+0x1181) [0x80c3b11]
>>   20: (Monitor::ms_dispatch(Message*)+0x31) [0x80d5021]
>>   21: (DispatchQueue::entry()+0x337) [0x82afa47]
>>   22: (DispatchQueue::DispatchThread::entry()+0x20) [0x823eec0]
>>   23: (Thread::_entry_func(void*)+0x11) [0x824be41]
>>   24: (()+0x57b0) [0xb75ef7b0]
>>   25: (clone()+0x5e) [0xb73d8cde]
>>   NOTE: a copy of the executable, or `objdump -rdS <executable>` is
>> needed to interpret this.
>
>
> http://tracker.newdream.net/issues/3495
>
> Should be fixed in latest master, but the latest patch didn't make it to
> v0.55.1.
>
>   -Joao



-- 
...WBR, Roman Hlynovskiy

Re: ceph stability

[Joao Eduardo Luis]

On 12/19/2012 10:58 AM, Roman Hlynovskiy wrote:
> Hello Joao,
>
>   thanks for feedback. is this fix available on the svn? i can provide
> heavy testing for it.

Yes, the fix in on github's (not svn ;) master branch.

All testing is most welcome!

Thanks.

   -Joao

>
> 2012/12/19 Joao Eduardo Luis <joao.luis@inktank.com>:
>> On 12/19/2012 09:03 AM, Roman Hlynovskiy wrote:
>>>
>>> My first problem - I am getting spurious mon's deaths, which usually
>>> looks like this:
>>>
>>> --- begin dump of recent events ---
>>>        0> 2012-12-19 10:35:58.912119 b41eab70 -1 *** Caught signal
>>> (Aborted) **
>>>    in thread b41eab70
>>>
>>>    ceph version 0.55.1 (8e25c8d984f9258644389a18997ec6bdef8e056b)
>>>    1: /usr/bin/ceph-mon() [0x8183a11]
>>>    2: [0xb7714400]
>>>    3: (gsignal()+0x47) [0xb7337577]
>>>    4: (abort()+0x182) [0xb733a962]
>>>    5: (__gnu_cxx::__verbose_terminate_handler()+0x14f) [0xb755653f]
>>>    6: (()+0xbd405) [0xb7554405]
>>>    7: (()+0xbd442) [0xb7554442]
>>>    8: (()+0xbd581) [0xb7554581]
>>>    9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>>> const*)+0x80f) [0x824cabf]
>>>    10: /usr/bin/ceph-mon() [0x80e3c1d]
>>>    11: (MDSMonitor::tick()+0x1e3b) [0x811ea0b]
>>>    12: (MDSMonitor::on_active()+0x1d) [0x81188dd]
>>>    13: (PaxosService::_active()+0x212) [0x80e4b02]
>>>    14: (Context::complete(int)+0x19) [0x80c4cf9]
>>>    15: (finish_contexts(CephContext*, std::list<Context*,
>>> std::allocator<Context*> >&, int)+0x13f) [0x80d208f]
>>>    16: (Monitor::recovered_leader(int)+0x3ac) [0x80ac5ac]
>>>    17: (Paxos::handle_last(MMonPaxos*)+0xb02) [0x80e0572]
>>>    18: (Paxos::dispatch(PaxosServiceMessage*)+0x2c4) [0x80e0e94]
>>>    19: (Monitor::_ms_dispatch(Message*)+0x1181) [0x80c3b11]
>>>    20: (Monitor::ms_dispatch(Message*)+0x31) [0x80d5021]
>>>    21: (DispatchQueue::entry()+0x337) [0x82afa47]
>>>    22: (DispatchQueue::DispatchThread::entry()+0x20) [0x823eec0]
>>>    23: (Thread::_entry_func(void*)+0x11) [0x824be41]
>>>    24: (()+0x57b0) [0xb75ef7b0]
>>>    25: (clone()+0x5e) [0xb73d8cde]
>>>    NOTE: a copy of the executable, or `objdump -rdS <executable>` is
>>> needed to interpret this.
>>
>>
>> http://tracker.newdream.net/issues/3495
>>
>> Should be fixed in latest master, but the latest patch didn't make it to
>> v0.55.1.
>>
>>    -Joao
>
>
>

Re: ceph stability

[Mark Nelson]

On 12/19/2012 03:03 AM, Roman Hlynovskiy wrote:
> Hello,
>
> I have 2 issues with ceph stability and looking for help to resolve them.
> My setup is pretty simple - 3 debian 32bit stable systems each running
> osd, mon and mds.
> the conf is the following:
> --------------------
> [global]
>      auth cluster required = none
>      auth service required = none
>      auth client required = none
>
> [osd]
>      osd journal size = 1000
>      filestore xattr use omap = true
>
> [mon.a]
>      host = ceph-node01
>      mon addr = 192.168.7.11:6789
>
> [mon.b]
>      host = ceph-node02
>      mon addr = 192.168.7.12:6789
>
> [mon.c]
>          host = ceph-node03
>          mon addr = 192.168.7.13:6789
>
> [mds.a]
>          host = ceph-node01
>
> [mds.b]
>          host = ceph-node02
>
> [mds.c]
>          host = ceph-node03

A quicky side-node:  multi-mds solutions aren't being supported in 
production right now.  Not sure if your stat problems below are related, 
but you may want to try starting out with a single mds and see if the 
problem goes away.  If so, there may be some hints in the mds logs 
regarding what's going on.  Bug reports are welcome!

>
> [osd.0]
>      host = ceph-node01
>
> [osd.1]
>      host = ceph-node02
>
> [osd.2]
>      host = ceph-node03
> --------------------
> ceph -s is:
>     health HEALTH_OK
>     monmap e4: 3 mons at
> {a=192.168.7.11:6789/0,b=192.168.7.12:6789/0,c=192.168.7.13:6789/0},
> election epoch 118, quorum 0,1,2 a,b,c
>     osdmap e197: 3 osds: 3 up, 3 in
>      pgmap v43305: 384 pgs: 384 active+clean; 72351 MB data, 144 GB
> used, 105 GB / 249 GB avail
>     mdsmap e4439: 1/1/1 up {0=a=up:active}, 2 up:standby
> --------------------
>
> My first problem - I am getting spurious mon's deaths, which usually
> looks like this:
>
> --- begin dump of recent events ---
>       0> 2012-12-19 10:35:58.912119 b41eab70 -1 *** Caught signal (Aborted) **
>   in thread b41eab70
>
>   ceph version 0.55.1 (8e25c8d984f9258644389a18997ec6bdef8e056b)
>   1: /usr/bin/ceph-mon() [0x8183a11]
>   2: [0xb7714400]
>   3: (gsignal()+0x47) [0xb7337577]
>   4: (abort()+0x182) [0xb733a962]
>   5: (__gnu_cxx::__verbose_terminate_handler()+0x14f) [0xb755653f]
>   6: (()+0xbd405) [0xb7554405]
>   7: (()+0xbd442) [0xb7554442]
>   8: (()+0xbd581) [0xb7554581]
>   9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x80f) [0x824cabf]
>   10: /usr/bin/ceph-mon() [0x80e3c1d]
>   11: (MDSMonitor::tick()+0x1e3b) [0x811ea0b]
>   12: (MDSMonitor::on_active()+0x1d) [0x81188dd]
>   13: (PaxosService::_active()+0x212) [0x80e4b02]
>   14: (Context::complete(int)+0x19) [0x80c4cf9]
>   15: (finish_contexts(CephContext*, std::list<Context*,
> std::allocator<Context*> >&, int)+0x13f) [0x80d208f]
>   16: (Monitor::recovered_leader(int)+0x3ac) [0x80ac5ac]
>   17: (Paxos::handle_last(MMonPaxos*)+0xb02) [0x80e0572]
>   18: (Paxos::dispatch(PaxosServiceMessage*)+0x2c4) [0x80e0e94]
>   19: (Monitor::_ms_dispatch(Message*)+0x1181) [0x80c3b11]
>   20: (Monitor::ms_dispatch(Message*)+0x31) [0x80d5021]
>   21: (DispatchQueue::entry()+0x337) [0x82afa47]
>   22: (DispatchQueue::DispatchThread::entry()+0x20) [0x823eec0]
>   23: (Thread::_entry_func(void*)+0x11) [0x824be41]
>   24: (()+0x57b0) [0xb75ef7b0]
>   25: (clone()+0x5e) [0xb73d8cde]
>   NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> needed to interpret this.
>
> --- logging levels ---
>     0/ 5 none
>     0/ 1 lockdep
>     0/ 1 context
>     1/ 1 crush
>     1/ 5 mds
>     1/ 5 mds_balancer
>     1/ 5 mds_locker
>     1/ 5 mds_log
>     1/ 5 mds_log_expire
>     1/ 5 mds_migrator
>     0/ 1 buffer
>     0/ 1 timer
>     0/ 1 filer
>     0/ 1 striper
>     0/ 1 objecter
>     0/ 5 rados
>     0/ 5 rbd
>     0/ 5 journaler
>     0/ 5 objectcacher
>     0/ 5 client
>     0/ 5 osd
>     0/ 5 optracker
>     0/ 5 objclass
>     1/ 3 filestore
>     1/ 3 journal
>     0/ 5 ms
>     1/ 5 mon
>     0/10 monc
>     0/ 5 paxos
>     0/ 5 tp
>     1/ 5 auth
>     1/ 5 crypto
>     1/ 1 finisher
>     1/ 5 heartbeatmap
>     1/ 5 perfcounter
>     1/ 5 rgw
>     1/ 5 hadoop
>     1/ 5 javaclient
>     1/ 5 asok
>     1/ 1 throttle
>    -2/-2 (syslog threshold)
>    -1/-1 (stderr threshold)
>    max_recent    100000
>    max_new         1000
>    log_file /var/log/ceph/ceph-mon.a.log
> --- end dump of recent events ---
>
> the binaries are coming from ceph.com debian-testing repo.
>
> My second problem - I have 2 systems which mount ceph. Whenever I
> mount ceph on any other system it usually mounts but get stuck on
> stat* operations (i.e. simple ls -al will hang with read( from the
> ceph-mounted directory for ages). This kind of client stuck also
> affects two working clients. they also start to stuck on the stat*
> even after shutdown of the third client. so usually umount/mount or
> even reboot for existing clients solves the issue)
>
>
> --
> ...WBR, Roman Hlynovskiy
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: ceph stability

[Roman Hlynovskiy]

Hello Mark,

for multi-mds solutions do you refer to multi-active arch or 1 active
and many standby arch?

http://ceph.com/docs/master/architecture/ says:
-----
The Ceph filesystem service is provided by a daemon called ceph-mds.
It uses RADOS to store all the filesystem metadata (directories, file
ownership, access modes, etc), and directs clients to access RADOS
directly for the file contents. The Ceph filesystem aims for POSIX
compatibility. ceph-mds can run as a single process, or it can be
distributed out to multiple physical machines, either for high
availability or for scalability.

High Availability: The extra ceph-mds instances can be standby, ready
to take over the duties of any failed ceph-mds that was active. This
is easy because all the data, including the journal, is stored on
RADOS. The transition is triggered automatically by ceph-mon.
Scalability: Multiple ceph-mds instances can be active, and they will
split the directory tree into subtrees (and shards of a single busy
directory), effectively balancing the load amongst all active servers.

Combinations of standby and active etc are possible, for example
running 3 active ceph-mds instances for scaling, and one standby
intance for high availability.
-----

 I saw cases while standby mds took over traffic from the active one.
Looks like it's working. Would you please clarify.
 I tried to disable 2 standby mds and happily reproduced the problem )
so, it's something else. I will try playing with mds log level and
provide more accurate details.

thanks!


2012/12/19 Mark Nelson <mark.nelson@inktank.com>:

>
>
> A quicky side-node:  multi-mds solutions aren't being supported in
> production right now.  Not sure if your stat problems below are related, but
> you may want to try starting out with a single mds and see if the problem
> goes away.  If so, there may be some hints in the mds logs regarding what's
> going on.  Bug reports are welcome!
>
>>
>> [osd.0]
>>      host = ceph-node01
>>
>> [osd.1]
>>      host = ceph-node02
>>
>> [osd.2]
>>      host = ceph-node03




--
...WBR, Roman Hlynovskiy

Re: ceph stability

[Mark Nelson]

On 12/20/2012 01:08 AM, Roman Hlynovskiy wrote:
> Hello Mark,
>
> for multi-mds solutions do you refer to multi-active arch or 1 active
> and many standby arch?

That's a good question!  I know we don't really recommend multi-active 
right now for production use.  Not sure what our current recommendations 
are for multi-standby.  As far as I know it's considered to be more 
stable.  I'm sure Greg or Sage can chime in with a more accurate assessment.

>
> http://ceph.com/docs/master/architecture/ says:
> -----
> The Ceph filesystem service is provided by a daemon called ceph-mds.
> It uses RADOS to store all the filesystem metadata (directories, file
> ownership, access modes, etc), and directs clients to access RADOS
> directly for the file contents. The Ceph filesystem aims for POSIX
> compatibility. ceph-mds can run as a single process, or it can be
> distributed out to multiple physical machines, either for high
> availability or for scalability.
>
> High Availability: The extra ceph-mds instances can be standby, ready
> to take over the duties of any failed ceph-mds that was active. This
> is easy because all the data, including the journal, is stored on
> RADOS. The transition is triggered automatically by ceph-mon.
> Scalability: Multiple ceph-mds instances can be active, and they will
> split the directory tree into subtrees (and shards of a single busy
> directory), effectively balancing the load amongst all active servers.
>
> Combinations of standby and active etc are possible, for example
> running 3 active ceph-mds instances for scaling, and one standby
> intance for high availability.
> -----
>
>   I saw cases while standby mds took over traffic from the active one.
> Looks like it's working. Would you please clarify.

I don't think there is anything that would prevent even active-active 
MDS setups to work, it's just that it may not be stable.

>   I tried to disable 2 standby mds and happily reproduced the problem )
> so, it's something else. I will try playing with mds log level and
> provide more accurate details.

Good info!  Btw, regarding the bug tracker, if you make an account and 
go to the "fs" project, you should see a "new issue" link.

>
> thanks!
>
>
> 2012/12/19 Mark Nelson <mark.nelson@inktank.com>:
>
>>
>>
>> A quicky side-node:  multi-mds solutions aren't being supported in
>> production right now.  Not sure if your stat problems below are related, but
>> you may want to try starting out with a single mds and see if the problem
>> goes away.  If so, there may be some hints in the mds logs regarding what's
>> going on.  Bug reports are welcome!
>>
>>>
>>> [osd.0]
>>>       host = ceph-node01
>>>
>>> [osd.1]
>>>       host = ceph-node02
>>>
>>> [osd.2]
>>>       host = ceph-node03
>
>
>
>
> --
> ...WBR, Roman Hlynovskiy
>

Re: ceph stability

[Amon Ott]

Am 20.12.2012 15:31, schrieb Mark Nelson:
> On 12/20/2012 01:08 AM, Roman Hlynovskiy wrote:
>> Hello Mark,
>>
>> for multi-mds solutions do you refer to multi-active arch or 1 active
>> and many standby arch?
> 
> That's a good question!  I know we don't really recommend multi-active
> right now for production use.  Not sure what our current recommendations
> are for multi-standby.  As far as I know it's considered to be more
> stable.  I'm sure Greg or Sage can chime in with a more accurate
> assessment.

We have been testing a lot with multi-standby, because a single MDS does
not make a lot of sense in a cluster. Maybe the clue is to have only one
standby, making SPOF a DPOF?

Up to 0.55 with some of Yan's fixes, Ceph has been too instable whenever
it came to MDS failover. As long as the same MDS has been kept active,
it was quite stable.

We are really hoping for 0.56 with the additional MDS fixes and also
fixed kernel 3.8 code. We have been waiting to replace our existing
cluster fs with CephFS for at least one and a half years, but it has
never been stable enough in our setup with standby MDS and takeover, let
alone multi-active. We do not want to risk a complete cluster failure.

Amon Ott
-- 
Dr. Amon Ott
m-privacy GmbH           Tel: +49 30 24342334
Am Köllnischen Park 1    Fax: +49 30 99296856
10179 Berlin             http://www.m-privacy.de

Amtsgericht Charlottenburg, HRB 84946

Geschäftsführer:
 Dipl.-Kfm. Holger Maczkowsky,
 Roman Maczkowsky

GnuPG-Key-ID: 0x2DD3A649

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ceph stability

[Gregory Farnum]

On Fri, Dec 21, 2012 at 2:07 AM, Amon Ott <ao@m-privacy.de> wrote:
> Am 20.12.2012 15:31, schrieb Mark Nelson:
>> On 12/20/2012 01:08 AM, Roman Hlynovskiy wrote:
>>> Hello Mark,
>>>
>>> for multi-mds solutions do you refer to multi-active arch or 1 active
>>> and many standby arch?
>>
>> That's a good question!  I know we don't really recommend multi-active
>> right now for production use.  Not sure what our current recommendations
>> are for multi-standby.  As far as I know it's considered to be more
>> stable.  I'm sure Greg or Sage can chime in with a more accurate
>> assessment.
>
> We have been testing a lot with multi-standby, because a single MDS does
> not make a lot of sense in a cluster. Maybe the clue is to have only one
> standby, making SPOF a DPOF?

The number of standbys should not have any impact on stability — they
are read-only until the active MDS gets declared down, at which point
one of them becomes active. I suppose having a standby could reduce
stability if your active MDS is overloaded enough to miss check-ins
without actually going down — without a standby it would eventually
recover, but with a standby a transition happens. That's the only
difference, though.
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ceph stability

[Sage Weil]

On Wed, 19 Dec 2012, Roman Hlynovskiy wrote:
> My second problem - I have 2 systems which mount ceph. Whenever I
> mount ceph on any other system it usually mounts but get stuck on
> stat* operations (i.e. simple ls -al will hang with read( from the
> ceph-mounted directory for ages). This kind of client stuck also
> affects two working clients. they also start to stuck on the stat*
> even after shutdown of the third client. so usually umount/mount or
> even reboot for existing clients solves the issue)

If this is something you can easily reproduce, can you please run with

[mds]
	debug mds = 20
	debug ms = 1

in your ceph.conf and attach the resulting mds log to a bug in the 
tracker?  We've seen this sort of thing in the past, but not recently.

Is this the kernel client mount?  What version of the client (kernel) are 
you running?

Thanks!
sage

Re: ceph stability

[Roman Hlynovskiy]

Hello Sage,

Yes, I can easily reproduce it.

how can I open a bug on the tracker? I can't find the link for this.
log is ready to be uploaded.
this is pure kernel client mount. my kernel on the client is
3.2.0-0.bpo.3-686-pae

2012/12/19 Sage Weil <sage@inktank.com>:
> On Wed, 19 Dec 2012, Roman Hlynovskiy wrote:
>> My second problem - I have 2 systems which mount ceph. Whenever I
>> mount ceph on any other system it usually mounts but get stuck on
>> stat* operations (i.e. simple ls -al will hang with read( from the
>> ceph-mounted directory for ages). This kind of client stuck also
>> affects two working clients. they also start to stuck on the stat*
>> even after shutdown of the third client. so usually umount/mount or
>> even reboot for existing clients solves the issue)
>
> If this is something you can easily reproduce, can you please run with
>
> [mds]
>         debug mds = 20
>         debug ms = 1
>
> in your ceph.conf and attach the resulting mds log to a bug in the
> tracker?  We've seen this sort of thing in the past, but not recently.
>
> Is this the kernel client mount?  What version of the client (kernel) are
> you running?
>
> Thanks!
> sage
>
>
>



-- 
...WBR, Roman Hlynovskiy

Re: ceph stability

[Sam Lang]

On 12/19/2012 10:02 PM, Roman Hlynovskiy wrote:
> Hello Sage,
>
> Yes, I can easily reproduce it.
>
> how can I open a bug on the tracker? I can't find the link for this.
> log is ready to be uploaded.

Hi Roman,

The bug tracker is at:  http://tracker.newdream.net/

Once you've created an account, you can jump to the project 'fs' through 
the scroll down bar on the right, and then click on the 'New Issue' tab.

-sam

> this is pure kernel client mount. my kernel on the client is
> 3.2.0-0.bpo.3-686-pae
>
> 2012/12/19 Sage Weil <sage@inktank.com>:
>> On Wed, 19 Dec 2012, Roman Hlynovskiy wrote:
>>> My second problem - I have 2 systems which mount ceph. Whenever I
>>> mount ceph on any other system it usually mounts but get stuck on
>>> stat* operations (i.e. simple ls -al will hang with read( from the
>>> ceph-mounted directory for ages). This kind of client stuck also
>>> affects two working clients. they also start to stuck on the stat*
>>> even after shutdown of the third client. so usually umount/mount or
>>> even reboot for existing clients solves the issue)
>>
>> If this is something you can easily reproduce, can you please run with
>>
>> [mds]
>>          debug mds = 20
>>          debug ms = 1
>>
>> in your ceph.conf and attach the resulting mds log to a bug in the
>> tracker?  We've seen this sort of thing in the past, but not recently.
>>
>> Is this the kernel client mount?  What version of the client (kernel) are
>> you running?
>>
>> Thanks!
>> sage
>>
>>
>>
>
>
>

Re: ceph stability

[Roman Hlynovskiy]

Hello Sage,

Just opened the bug. here is the link
http://tracker.newdream.net/issues/3663

thanks!

2012/12/20 Sam Lang <sam.lang@inktank.com>:
> On 12/19/2012 10:02 PM, Roman Hlynovskiy wrote:
>>
>> Hello Sage,
>>
>> Yes, I can easily reproduce it.
>>
>> how can I open a bug on the tracker? I can't find the link for this.
>> log is ready to be uploaded.
>
>
> Hi Roman,
>
> The bug tracker is at:  http://tracker.newdream.net/
>
> Once you've created an account, you can jump to the project 'fs' through the
> scroll down bar on the right, and then click on the 'New Issue' tab.
>
> -sam
>
>
>> this is pure kernel client mount. my kernel on the client is
>> 3.2.0-0.bpo.3-686-pae
>>
>> 2012/12/19 Sage Weil <sage@inktank.com>:
>>>
>>> On Wed, 19 Dec 2012, Roman Hlynovskiy wrote:
>>>>
>>>> My second problem - I have 2 systems which mount ceph. Whenever I
>>>> mount ceph on any other system it usually mounts but get stuck on
>>>> stat* operations (i.e. simple ls -al will hang with read( from the
>>>> ceph-mounted directory for ages). This kind of client stuck also
>>>> affects two working clients. they also start to stuck on the stat*
>>>> even after shutdown of the third client. so usually umount/mount or
>>>> even reboot for existing clients solves the issue)
>>>
>>>
>>> If this is something you can easily reproduce, can you please run with
>>>
>>> [mds]
>>>          debug mds = 20
>>>          debug ms = 1
>>>
>>> in your ceph.conf and attach the resulting mds log to a bug in the
>>> tracker?  We've seen this sort of thing in the past, but not recently.
>>>
>>> Is this the kernel client mount?  What version of the client (kernel) are
>>> you running?
>>>
>>> Thanks!
>>> sage
>>>
>>>
>>>
>>
>>
>>
>



-- 
...WBR, Roman Hlynovskiy