How to reduce active mds number

How to reduce active mds number

[AnnyRen]

Hi,

I'm testing MDS expasion,
my original environment has 3 MON, 2 MDS (one active, one standby), 10
OSD which are built on 15 physical machines.

the following is my step to test add/remove active MDS:


Step1: check mds status

root@MON1:~# ceph mds dump -o -
4097:   192.168.200.184:6800/2095 '0' mds0.1 up:active seq 66
4107:   192.168.200.185:6800/1911 '1' mds-1.0 up:standby seq 2
There are two mds, one is active, another is standby

Step2: ceph mds set_max_mds 2

Step3: ceph -s
2011-04-22 08:33:03.836554   mds e18: 2/2/2 up {0=up:active,1=up:active}

or
root@MON1:~# ceph mds dump -o -
max_mds 2
in      0,1
up      {0=4097,1=4107}
failed
stopped
4097:   192.168.200.184:6800/2095 '0' mds0.1 up:active seq 66
4107:   192.168.200.185:6800/1911 '1' mds1.1 up:active seq 25635


Step4: I want to set one mds to standby, so I execute "ceph mds set_max_mds 1"

root@MON1:~# ceph mds set_max_mds 1
2011-04-22 08:35:29.964667 mon <- [mds,set_max_mds,1]
2011-04-22 08:35:30.133274 mon0 -> 'max_mds = 1' (0)


Step5: Then I check mds status

root@MON1:~# ceph mds stat
2011-04-22 08:39:04.160139 mon <- [mds,stat]
2011-04-22 08:39:04.160823 mon0 -> 'e19: 2/2/1 up {0=up:active,1=up:active}' (0)
                                                              ^^^^^^
What does the 2/2/1 means ?

I found that the number of active MDS still remains two...
Could you tell me how to reduce the active mds number from 2 to 1...?
and what the 2/2/1 means ?

Thanks a lot !  :)

Best Regards,
Annie

Re: How to reduce active mds number

[Sage Weil]

On Fri, 22 Apr 2011, AnnyRen wrote:
> Hi,
> 
> I'm testing MDS expasion,
> my original environment has 3 MON, 2 MDS (one active, one standby), 10
> OSD which are built on 15 physical machines.
> 
> the following is my step to test add/remove active MDS:
> 
> 
> Step1: check mds status
> 
> root@MON1:~# ceph mds dump -o -
> 4097:   192.168.200.184:6800/2095 '0' mds0.1 up:active seq 66
> 4107:   192.168.200.185:6800/1911 '1' mds-1.0 up:standby seq 2
> There are two mds, one is active, another is standby
> 
> Step2: ceph mds set_max_mds 2
> 
> Step3: ceph -s
> 2011-04-22 08:33:03.836554   mds e18: 2/2/2 up {0=up:active,1=up:active}
> 
> or
> root@MON1:~# ceph mds dump -o -
> max_mds 2
> in      0,1
> up      {0=4097,1=4107}
> failed
> stopped
> 4097:   192.168.200.184:6800/2095 '0' mds0.1 up:active seq 66
> 4107:   192.168.200.185:6800/1911 '1' mds1.1 up:active seq 25635
> 
> 
> Step4: I want to set one mds to standby, so I execute "ceph mds set_max_mds 1"
> 
> root@MON1:~# ceph mds set_max_mds 1
> 2011-04-22 08:35:29.964667 mon <- [mds,set_max_mds,1]
> 2011-04-22 08:35:30.133274 mon0 -> 'max_mds = 1' (0)
> 
> 
> Step5: Then I check mds status
> 
> root@MON1:~# ceph mds stat
> 2011-04-22 08:39:04.160139 mon <- [mds,stat]
> 2011-04-22 08:39:04.160823 mon0 -> 'e19: 2/2/1 up {0=up:active,1=up:active}' (0)
>                                                               ^^^^^^
> What does the 2/2/1 means ?
> 
> I found that the number of active MDS still remains two...
> Could you tell me how to reduce the active mds number from 2 to 1...?
> and what the 2/2/1 means ?

2 are "up", 2 are "in" (the cluster set includes 2), and 1 is the 
target/max.

You can do

$ ceph mds stop 1

to tell mds1 to export everything and shut down cleanly.

sage

Re: How to reduce active mds number

[doki74216]

Hi,
I am testing the same case.
My system is 1 MON , 2 MDS, 3OSD.
Here is my steps:

1) First, I set the max number of MDS to 2 , so there are two active MDSs.
192.168.200.184:6800/2095 '0' mds0.1 up:active seq 66
192.168.200.185:6800/1911 '1' mds1.1 up:active seq 25635
2) I want to set the active one to standby. I excute"ceph mds
set_max_mds 1" and "ceph mds stop 0".
    But here is why I confuse:
    mds0 becomes stopping not standby.??
    It shows:
192.138.200.185:6800/14465 ‘1’mds0.6 up:stopping seq 14
192.138.200.184:6800/15442 ‘0’mds1.1 up:active seq 210

3) When I excute"ceph mds set_max_mds 2", mds0 doesn't become active,
    I type"ceph mds dump -o -"
    But it shows:
192.138.200.185:6800/14465 ‘1’mds0.6 up:stopping seq 14 export_targets=1
192.138.200.184:6800/15442 ‘0’mds1.1 up:active seq 210
   There is no two active MDSes, why?

4) Therefore, I still hope that there are one active mds and one
standby mds(by default).
    I restart the system. I execute" /etc/init.d/ceph -a stop" and "
/etc/init.d/ceph -a start"
    I type"ceph -s"
    But it shows:
mds e52:  2/2/1 up {0=up:rejoin, 1=up:resolve}
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  what means?
    After awhile I execue”ceph mds stat” , it shows:
‘e61: 2/2/1 up {o=up:relay, 1=up:relay}’ (0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ what means?

5) I set the man active number to one because I still want the system
by default value(one active and one standby).
    I execute"ceph mds set_max_mds 1"
    It shows:
mds e80: 2/2/1 up {0=up:rejoin, 1=up:rejoin}, 1:up:standby
mds e80: 2/2/1 up {0=up:active, 1=up:rejoin}, 1:up:standby
mds e80: 2/2/1 up {0=up:active, 1=up:active}, 1:up:standby
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    I don't know what it means.
    Why there are three statuses? Aren't there just 2 MDSes in my system?
    I execue"ceph -s"
    It shows:
mds e94: 2/2/1 up {0=up:active,1=up:active}, 1 up:standby

Please help me to solve these questions..
And teach me how to set the MDS by the default(1 active & 1 standby)?
Thank you very much  ^^


Best regards

Stefanie Chen
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How to reduce active mds number

[Tommi Virtanen]

On Tue, May 03, 2011 at 10:07:01AM +0800, doki74216@gmail.com wrote:
> 2) I want to set the active one to standby. I excute"ceph mds
> set_max_mds 1" and "ceph mds stop 0".
>     But here is why I confuse:
>     mds0 becomes stopping not standby.??
>     It shows:
> 192.138.200.185:6800/14465 ‘1’mds0.6 up:stopping seq 14
> 192.138.200.184:6800/15442 ‘0’mds1.1 up:active seq 210

I think the confusion here is that "ceph mds stop 0" really tells it
to stop, not to go to standby. Once it finishes stopping (= has safely
exported all its data to the other mds), you can start it again, and
then it'll get to be standby.

This ASCII art from the source might explain. Ignore the extra detail:

 boot  --> standby, creating, or starting.


 dne  ---->   creating  ----->   active*
 ^ ^___________/                /  ^ ^
 |                             /  /  |
 destroying                   /  /   |
   ^                         /  /    |
   |                        /  /     |
 stopped <---- stopping* <-/  /      |
      \                      /       |
        ----- starting* ----/        |
                                     |
 failed                              |
    \                                |
     \--> replay*  --> reconnect* --> rejoin*

     * = can fail


Your mds is still in the state "stopping". To get to standby, it needs
to finish that, then get to "boot" again (by being restarted), and
then it can enter standby.

On the other hand, I can reproduce the problem of an MDS just being in
state "stopping" for along time. I'll see what others have to say
about that.


> 3) When I excute"ceph mds set_max_mds 2", mds0 doesn't become active,
>     I type"ceph mds dump -o -"
>     But it shows:
> 192.138.200.185:6800/14465 ‘1’mds0.6 up:stopping seq 14 export_targets=1
> 192.138.200.184:6800/15442 ‘0’mds1.1 up:active seq 210
>    There is no two active MDSes, why?

Your mds is still in the state "stopping". It needs to finish that
first. If you've set max_mds==2, it should get to active once it's
done.


> 4) Therefore, I still hope that there are one active mds and one
> standby mds(by default).
>     I restart the system. I execute" /etc/init.d/ceph -a stop" and "
> /etc/init.d/ceph -a start"
>     I type"ceph -s"
>     But it shows:
> mds e52:  2/2/1 up {0=up:rejoin, 1=up:resolve}
>                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  what means?
>     After awhile I execue”ceph mds stat” , it shows:
> ‘e61: 2/2/1 up {o=up:relay, 1=up:relay}’ (0)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ what means?

The word "relay" is never mentioned in the source tree. And seeing
that "o" in there makes me think you're typing these lines in. If
you're copying log lines to email, please use copy-paste and don't
type them in manually; variation from the exact message makes helping
you harder.

"replay" is when an MDS is starting up and reading its journal,
replaying the operations stored there against the final data storage.

"resolve" is when the MDSes go through their journals and figure out
how to handle operations that can cross MDS boundaries, such as
renames across directories.


> 5) I set the man active number to one because I still want the system
> by default value(one active and one standby).
>     I execute"ceph mds set_max_mds 1"
>     It shows:
> mds e80: 2/2/1 up {0=up:rejoin, 1=up:rejoin}, 1:up:standby
> mds e80: 2/2/1 up {0=up:active, 1=up:rejoin}, 1:up:standby
> mds e80: 2/2/1 up {0=up:active, 1=up:active}, 1:up:standby
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>     I don't know what it means.
>     Why there are three statuses? Aren't there just 2 MDSes in my system?
>     I execue"ceph -s"
>     It shows:
> mds e94: 2/2/1 up {0=up:active,1=up:active}, 1 up:standby
> 
> Please help me to solve these questions..
> And teach me how to set the MDS by the default(1 active & 1 standby)?
> Thank you very much  ^^

I don't have a good answer to this one. The /1 in the 2/2/1 means your
max_mds really is 1. It might be a question of there being no
automatic transition from active to standby.

I can reproduce this problem locally, and will try to figure it out.

-- 
:(){ :|:&};:
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How to reduce active mds number

[Tommi Virtanen]

On Tue, May 03, 2011 at 10:34:45AM -0700, Tommi Virtanen wrote:
> On the other hand, I can reproduce the problem of an MDS just being in
> state "stopping" for along time. I'll see what others have to say
> about that.

Filed as issue #1048.

http://tracker.newdream.net/issues/1048

-- 
:(){ :|:&};: