question about mdmon --takeover

question about mdmon --takeover

[Francis Moreau]

Hello list,

I thought that using "--takeover" would hint mdmon to replace existing
mdmon process, and therefore the old one would exit somehow.

However after several "mdmon --takeover" I can see this:
 $ ps aux | grep dmon
root       233  0.0  0.2  80388 10752 ?        SLsl 14:02   0:00 @dmon
--offroot md127
root      3326  0.0  0.2  14920 10820 ?        SLsl 15:16   0:00 mdmon
--takeover md127
root      3343  0.0  0.2  14920 10820 ?        SLsl 15:17   0:00 mdmon
--takeover md127

Is this expected ?

Thanks.
-- 
Francis

Re: question about mdmon --takeover

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1204 bytes --]

On Wed, 31 Jul 2013 16:30:34 +0200 Francis Moreau <francis.moro@gmail.com>
wrote:

> Hello list,
> 
> I thought that using "--takeover" would hint mdmon to replace existing
> mdmon process, and therefore the old one would exit somehow.
> 
> However after several "mdmon --takeover" I can see this:
>  $ ps aux | grep dmon
> root       233  0.0  0.2  80388 10752 ?        SLsl 14:02   0:00 @dmon
> --offroot md127
> root      3326  0.0  0.2  14920 10820 ?        SLsl 15:16   0:00 mdmon
> --takeover md127
> root      3343  0.0  0.2  14920 10820 ?        SLsl 15:17   0:00 mdmon
> --takeover md127
> 
> Is this expected ?
> 
> Thanks.

Nope.  That's not expected.

mdmon should send SIGTERM to the old mdmon and then wait for it to exit.

If the new and old mdmon were compiled different and look for the pid file in
different directories that might explain what you see.

If you compile mdadm from source it will use /run/mdadm.  However if your
distro doesn't have /run the the distro-provided mdadm will be compiled
differently.

It is safe to kill the old mdmons once a new one is running, but you should
probably find out what the incompatibility is...

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: question about mdmon --takeover

[Francis Moreau]

Hello Neil,


Sorry for the late reply.

On Mon, Aug 5, 2013 at 8:59 AM, NeilBrown <neilb@suse.de> wrote:
> On Wed, 31 Jul 2013 16:30:34 +0200 Francis Moreau <francis.moro@gmail.com>
> wrote:
>
>> Hello list,
>>
>> I thought that using "--takeover" would hint mdmon to replace existing
>> mdmon process, and therefore the old one would exit somehow.
>>
>> However after several "mdmon --takeover" I can see this:
>>  $ ps aux | grep dmon
>> root       233  0.0  0.2  80388 10752 ?        SLsl 14:02   0:00 @dmon
>> --offroot md127
>> root      3326  0.0  0.2  14920 10820 ?        SLsl 15:16   0:00 mdmon
>> --takeover md127
>> root      3343  0.0  0.2  14920 10820 ?        SLsl 15:17   0:00 mdmon
>> --takeover md127
>>
>> Is this expected ?
>>
>> Thanks.
>
> Nope.  That's not expected.
>
> mdmon should send SIGTERM to the old mdmon and then wait for it to exit.
>
> If the new and old mdmon were compiled different and look for the pid file in
> different directories that might explain what you see.
>
> If you compile mdadm from source it will use /run/mdadm.  However if your
> distro doesn't have /run the the distro-provided mdadm will be compiled
> differently.
>

It doesn't to be the case.

Actually sending SIGTERM manually to mdmon has no effects.


# mdadm --version
mdadm - v3.2.6 - 25th October 2012

# ps aux | grep dmon
root       235  0.1  1.0  80612 10976 ?        SLsl 19:08   0:00 @dmon
--offroot md127
root       339  0.0  1.0  15044 10944 ?        SLsl 19:08   0:00
/sbin/mdmon --takeover md127

# cat /run/mdadm/md127.pid
339

# kill -SIGTERM 339
# ps aux | grep dmon
root       235  0.0  1.0  80612 10976 ?        SLsl 19:08   0:00 @dmon
--offroot md127
root       339  0.0  1.0  15044 10944 ?        SLsl 19:08   0:00
/sbin/mdmon --takeover md127

# ps aux | grep dmon
root       235  0.0  1.0  80612 10976 ?        SLsl 19:08   0:00 @dmon
--offroot md127
root       339  0.0  1.0  15044 10944 ?        SLsl 19:08   0:00
/sbin/mdmon --takeover md127
root      2352  0.1  1.0  15076 10976 ?        SLsl 19:12   0:00 mdmon
--takeover /dev/md127

# cat /run/mdadm/md127.pid
2352

# pkill -SIGTERM mdmon
[root@localhost ~]# ps aux | grep dmon
root       235  0.0  1.0  80612 10976 ?        SLsl 19:08   0:00 @dmon
--offroot md127
root       339  0.0  1.0  80580 10944 ?        SLsl 19:08   0:00
/sbin/mdmon --takeover md127
root      2352  0.0  1.0  80612 10976 ?        SLsl 19:12   0:00 mdmon
--takeover /dev/md127

Can't you reproduce ?

Thanks
-- 
Francis

Re: question about mdmon --takeover

[Francis Moreau]

On Wed, Aug 28, 2013 at 7:14 PM, Francis Moreau <francis.moro@gmail.com> wrote:
> Hello Neil,
>
>
> Sorry for the late reply.
>
> On Mon, Aug 5, 2013 at 8:59 AM, NeilBrown <neilb@suse.de> wrote:
>> On Wed, 31 Jul 2013 16:30:34 +0200 Francis Moreau <francis.moro@gmail.com>
>> wrote:
>>
>>> Hello list,
>>>
>>> I thought that using "--takeover" would hint mdmon to replace existing
>>> mdmon process, and therefore the old one would exit somehow.
>>>
>>> However after several "mdmon --takeover" I can see this:
>>>  $ ps aux | grep dmon
>>> root       233  0.0  0.2  80388 10752 ?        SLsl 14:02   0:00 @dmon
>>> --offroot md127
>>> root      3326  0.0  0.2  14920 10820 ?        SLsl 15:16   0:00 mdmon
>>> --takeover md127
>>> root      3343  0.0  0.2  14920 10820 ?        SLsl 15:17   0:00 mdmon
>>> --takeover md127
>>>
>>> Is this expected ?
>>>
>>> Thanks.
>>
>> Nope.  That's not expected.
>>
>> mdmon should send SIGTERM to the old mdmon and then wait for it to exit.
>>
>> If the new and old mdmon were compiled different and look for the pid file in
>> different directories that might explain what you see.
>>
>> If you compile mdadm from source it will use /run/mdadm.  However if your
>> distro doesn't have /run the the distro-provided mdadm will be compiled
>> differently.
>>
>
> It doesn't to be the case.
>
> Actually sending SIGTERM manually to mdmon has no effects.
>
>
> # mdadm --version
> mdadm - v3.2.6 - 25th October 2012
>
> # ps aux | grep dmon
> root       235  0.1  1.0  80612 10976 ?        SLsl 19:08   0:00 @dmon
> --offroot md127
> root       339  0.0  1.0  15044 10944 ?        SLsl 19:08   0:00
> /sbin/mdmon --takeover md127
>
> # cat /run/mdadm/md127.pid
> 339
>
> # kill -SIGTERM 339
> # ps aux | grep dmon
> root       235  0.0  1.0  80612 10976 ?        SLsl 19:08   0:00 @dmon
> --offroot md127
> root       339  0.0  1.0  15044 10944 ?        SLsl 19:08   0:00
> /sbin/mdmon --takeover md127
>
> # ps aux | grep dmon
> root       235  0.0  1.0  80612 10976 ?        SLsl 19:08   0:00 @dmon
> --offroot md127
> root       339  0.0  1.0  15044 10944 ?        SLsl 19:08   0:00
> /sbin/mdmon --takeover md127
> root      2352  0.1  1.0  15076 10976 ?        SLsl 19:12   0:00 mdmon
> --takeover /dev/md127
>
> # cat /run/mdadm/md127.pid
> 2352
>
> # pkill -SIGTERM mdmon
> [root@localhost ~]# ps aux | grep dmon
> root       235  0.0  1.0  80612 10976 ?        SLsl 19:08   0:00 @dmon
> --offroot md127
> root       339  0.0  1.0  80580 10944 ?        SLsl 19:08   0:00
> /sbin/mdmon --takeover md127
> root      2352  0.0  1.0  80612 10976 ?        SLsl 19:12   0:00 mdmon
> --takeover /dev/md127
>
> Can't you reproduce ?
>

and running latest git head is even worse, mdmon can't be started at all:

# mdadm --version
mdadm - v3.3-rc2-58-g2cdd5ce - 28th August 2013

# ls /run/mdadm/
autorebuild.pid  map

# mdmon --takeover /dev/md127
# ps aux | grep dmon
root      2347  0.0  0.0   9680   848 ttyS0    S+   21:40   0:00 grep
--color dmon

# ls /run/mdadm/
autorebuild.pid  map


-- 
Francis

Re: question about mdmon --takeover

[Francis Moreau]

Hello Martin,

I'm adding you in the discussion because you might have an idea on
what's going on with my array which is using DDF metadata.

On Wed, Aug 28, 2013 at 7:14 PM, Francis Moreau <francis.moro@gmail.com> wrote:
> Hello Neil,
>
>
> Sorry for the late reply.
>
> On Mon, Aug 5, 2013 at 8:59 AM, NeilBrown <neilb@suse.de> wrote:
>> On Wed, 31 Jul 2013 16:30:34 +0200 Francis Moreau <francis.moro@gmail.com>
>> wrote:
>>
>>> Hello list,
>>>
>>> I thought that using "--takeover" would hint mdmon to replace existing
>>> mdmon process, and therefore the old one would exit somehow.
>>>
>>> However after several "mdmon --takeover" I can see this:
>>>  $ ps aux | grep dmon
>>> root       233  0.0  0.2  80388 10752 ?        SLsl 14:02   0:00 @dmon
>>> --offroot md127
>>> root      3326  0.0  0.2  14920 10820 ?        SLsl 15:16   0:00 mdmon
>>> --takeover md127
>>> root      3343  0.0  0.2  14920 10820 ?        SLsl 15:17   0:00 mdmon
>>> --takeover md127
>>>
>>> Is this expected ?
>>>
>>> Thanks.
>>
>> Nope.  That's not expected.
>>
>> mdmon should send SIGTERM to the old mdmon and then wait for it to exit.
>>
>> If the new and old mdmon were compiled different and look for the pid file in
>> different directories that might explain what you see.
>>
>> If you compile mdadm from source it will use /run/mdadm.  However if your
>> distro doesn't have /run the the distro-provided mdadm will be compiled
>> differently.
>>
>
> It doesn't to be the case.
>
> Actually sending SIGTERM manually to mdmon has no effects.
>
>
> # mdadm --version
> mdadm - v3.2.6 - 25th October 2012
>
> # ps aux | grep dmon
> root       235  0.1  1.0  80612 10976 ?        SLsl 19:08   0:00 @dmon
> --offroot md127
> root       339  0.0  1.0  15044 10944 ?        SLsl 19:08   0:00
> /sbin/mdmon --takeover md127
>
> # cat /run/mdadm/md127.pid
> 339
>
> # kill -SIGTERM 339
> # ps aux | grep dmon
> root       235  0.0  1.0  80612 10976 ?        SLsl 19:08   0:00 @dmon
> --offroot md127
> root       339  0.0  1.0  15044 10944 ?        SLsl 19:08   0:00
> /sbin/mdmon --takeover md127
>
> # ps aux | grep dmon
> root       235  0.0  1.0  80612 10976 ?        SLsl 19:08   0:00 @dmon
> --offroot md127
> root       339  0.0  1.0  15044 10944 ?        SLsl 19:08   0:00
> /sbin/mdmon --takeover md127
> root      2352  0.1  1.0  15076 10976 ?        SLsl 19:12   0:00 mdmon
> --takeover /dev/md127
>
> # cat /run/mdadm/md127.pid
> 2352
>
> # pkill -SIGTERM mdmon
> [root@localhost ~]# ps aux | grep dmon
> root       235  0.0  1.0  80612 10976 ?        SLsl 19:08   0:00 @dmon
> --offroot md127
> root       339  0.0  1.0  80580 10944 ?        SLsl 19:08   0:00
> /sbin/mdmon --takeover md127
> root      2352  0.0  1.0  80612 10976 ?        SLsl 19:12   0:00 mdmon
> --takeover /dev/md127
>
> Can't you reproduce ?
>

Here are some additionnal information:

# cat /proc/mdstat
Personalities : [raid1]
md126 : active raid1 sda[1] sdb[0]
      2064384 blocks super external:/md127/0 [2/2] [UU]

md127 : inactive sda[1](S) sdb[0](S)
      65536 blocks super external:ddf

unused devices: <none>


# mdadm -E /dev/md126
/dev/md126:
   MBR Magic : aa55
Partition[0] :       367447 sectors at         2048 (type 82)
Partition[1] :      3755997 sectors at       372708 (type 05)
[root@localhost ~]# mdadm -E /dev/md127
/dev/md127:
          Magic : de11de11
        Version : 01.02.00
Controller GUID : 4C696E75:782D4D44:20202020:2020206C:6F63616C:686F7374
                  (Linux-MD       localhost)
 Container GUID : 4C696E75:782D4D44:DEADBEEF:00000000:3F4FB732:8435623D
                  (Linux-MD 08/28/13 22:27:30)
            Seq : 00000001
  Redundant hdr : no
  Virtual Disks : 1

      VD GUID[0] : 4C696E75:782D4D44:DEADBEEF:00000000:3F4FB739:E0C8B16E
                  (Linux-MD 08/28/13 22:27:37)
         unit[0] : 126
        state[0] : Optimal, Not Consistent
   init state[0] : Fully Initialised
       access[0] : Read/Write
         Name[0] : array1
 Raid Devices[0] : 2 (0 1)
   Raid Level[0] : RAID1
  Device Size[0] : 2064384
   Array Size[0] : 2064384

 Physical Disks : 2
      Number    RefNo      Size       Device      Type/State
         0    2cf00056   2064384K /dev/sda        active/Online
         1    b342fbdc   2064384K /dev/sdb        active/Online

I used GDB to trace what's going on in the monitor (thread) when
sending a SIGTERM to the mdmon process:

Breakpoint 1, wait_and_act (container=0x141a2c0, nowait=0) at monitor.c:634
634        for (a = *aap; a ; a = a->next) {
(gdb) n
632        rv = 0;
(gdb)
633        dirty_arrays = 0;
(gdb)
634        for (a = *aap; a ; a = a->next) {
(gdb)
649                int ret = read_and_act(a);
(gdb) p *a
$2 = {
  info = {
    array = {
      major_version = 0,
      minor_version = 0,
      patch_version = 0,
      ctime = 0,
      level = 1,
      size = 0,
      nr_disks = 0,
      raid_disks = 2,
      md_minor = 0,
      not_persistent = 0,
      utime = 0,
      state = 0,
      active_disks = 0,
      working_disks = 0,
      failed_disks = 0,
      spare_disks = 0,
      layout = 0,
      chunk_size = 0
    },
    disk = {
      number = 0,
      major = 0,
      minor = 0,
      raid_disk = 0,
      state = 0
    },
    events = 0,
    uuid = {0, 0, 0, 0},
    name = '\000' <repeats 32 times>,
    data_offset = 0,
    component_size = 4128768,
    custom_array_size = 0,
    reshape_active = 0,
    reshape_progress = 0,
    recovery_blocked = 0,
    {
      resync_start = 18446744073709551615,
      recovery_start = 18446744073709551615
    },
    bitmap_offset = 0,
    safe_mode_delay = 0,
    new_level = 0,
    delta_disks = 0,
    new_layout = 0,
    new_chunk = 0,
    errors = 0,
    cache_size = 0,
    mismatch_cnt = 0,
    text_version = '\000' <repeats 49 times>,
    container_member = 0,
    container_enough = 0,
    sys_name = "md126", '\000' <repeats 14 times>,
    devs = 0x14213c0,
    next = 0x0,
    recovery_fd = 0,
    state_fd = 12,
    prev_state = 0,
    curr_state = 0,
    next_state = 0
  },
  container = 0x141a2c0,
  next = 0x0,
  replaces = 0x0,
  to_remove = 0,
  action_fd = 11,
  resync_start_fd = 13,
  metadata_fd = 14,
  sync_completed_fd = 15,
  last_checkpoint = 3872640,
  prev_state = active,
  curr_state = active,
  next_state = bad_word,
  prev_action = idle,
  curr_action = idle,
  next_action = bad_action,
  check_degraded = 0,
  check_reshape = 0,
  devnum = 126
}

(gdb) n
636            if (a->replaces && !discard_this) {
(gdb)
648            if (a->container && !a->to_remove) {
(gdb)
649                int ret = read_and_act(a);
(gdb)
656                if (sigterm && !(ret & ARRAY_DIRTY))
(gdb) p ret
$3 = 1

It seems that the array is dirty, and that's the reason why the
monitor never terminates.

I got the same behaviour if I'm starting a new mdmon process on this
array using --takeover switch. It seems weird since in my
understanding --takeover indicate mdmon to replace the current mdmon
process even if the array is dirty, no ?

Do you have any idea on this behaviour ?
-- 
Francis

Re: question about mdmon --takeover

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 468 bytes --]

On Sat, 31 Aug 2013 22:21:36 +0200 Francis Moreau <francis.moro@gmail.com>
wrote:

> Hello Martin,
> 
> I'm adding you in the discussion because you might have an idea on
> what's going on with my array which is using DDF metadata.

Hi Francis,
 I think this problem will be fixed by Martin's latest patch set.
The root cause seems to be that safe_mode_delay isn't being set properly
which is the same root cause as the other problem you had.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: question about mdmon --takeover

[Francis Moreau]

Hi Neil,

On Mon, Sep 2, 2013 at 4:04 AM, NeilBrown <neilb@suse.de> wrote:
> On Sat, 31 Aug 2013 22:21:36 +0200 Francis Moreau <francis.moro@gmail.com>
> wrote:
>
>> Hello Martin,
>>
>> I'm adding you in the discussion because you might have an idea on
>> what's going on with my array which is using DDF metadata.
>
> Hi Francis,
>  I think this problem will be fixed by Martin's latest patch set.
> The root cause seems to be that safe_mode_delay isn't being set properly
> which is the same root cause as the other problem you had.
>

That's correct, this patch fixes this issue as well.

May I ask you when the 3.3 release is planned ?

Thanks
-- 
Francis

Re: question about mdmon --takeover

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 954 bytes --]

On Mon, 2 Sep 2013 09:56:26 +0200 Francis Moreau <francis.moro@gmail.com>
wrote:

> Hi Neil,
> 
> On Mon, Sep 2, 2013 at 4:04 AM, NeilBrown <neilb@suse.de> wrote:
> > On Sat, 31 Aug 2013 22:21:36 +0200 Francis Moreau <francis.moro@gmail.com>
> > wrote:
> >
> >> Hello Martin,
> >>
> >> I'm adding you in the discussion because you might have an idea on
> >> what's going on with my array which is using DDF metadata.
> >
> > Hi Francis,
> >  I think this problem will be fixed by Martin's latest patch set.
> > The root cause seems to be that safe_mode_delay isn't being set properly
> > which is the same root cause as the other problem you had.
> >
> 
> That's correct, this patch fixes this issue as well.

Thanks.

> 
> May I ask you when the 3.3 release is planned ?

I was waiting for a confirmation from you and one other from Martin.
So you have just brought it closer.

Tuesday or Wednesday I expect.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: question about mdmon --takeover

[Francis Moreau]

On Mon, Sep 2, 2013 at 10:10 AM, NeilBrown <neilb@suse.de> wrote:
> On Mon, 2 Sep 2013 09:56:26 +0200 Francis Moreau <francis.moro@gmail.com>
> wrote:
>
>> Hi Neil,
>>
>> On Mon, Sep 2, 2013 at 4:04 AM, NeilBrown <neilb@suse.de> wrote:
>> > On Sat, 31 Aug 2013 22:21:36 +0200 Francis Moreau <francis.moro@gmail.com>
>> > wrote:
>> >
>> >> Hello Martin,
>> >>
>> >> I'm adding you in the discussion because you might have an idea on
>> >> what's going on with my array which is using DDF metadata.
>> >
>> > Hi Francis,
>> >  I think this problem will be fixed by Martin's latest patch set.
>> > The root cause seems to be that safe_mode_delay isn't being set properly
>> > which is the same root cause as the other problem you had.
>> >
>>
>> That's correct, this patch fixes this issue as well.
>
> Thanks.
>
>>
>> May I ask you when the 3.3 release is planned ?
>
> I was waiting for a confirmation from you and one other from Martin.
> So you have just brought it closer.
>
> Tuesday or Wednesday I expect.

Great news.

-- 
Francis