Re: question about mdmon --takeover

[Francis Moreau <francis.moro@gmail.com>]

Hello Martin,

I'm adding you in the discussion because you might have an idea on
what's going on with my array which is using DDF metadata.

On Wed, Aug 28, 2013 at 7:14 PM, Francis Moreau <francis.moro@gmail.com> wrote:
> Hello Neil,
>
>
> Sorry for the late reply.
>
> On Mon, Aug 5, 2013 at 8:59 AM, NeilBrown <neilb@suse.de> wrote:
>> On Wed, 31 Jul 2013 16:30:34 +0200 Francis Moreau <francis.moro@gmail.com>
>> wrote:
>>
>>> Hello list,
>>>
>>> I thought that using "--takeover" would hint mdmon to replace existing
>>> mdmon process, and therefore the old one would exit somehow.
>>>
>>> However after several "mdmon --takeover" I can see this:
>>>  $ ps aux | grep dmon
>>> root       233  0.0  0.2  80388 10752 ?        SLsl 14:02   0:00 @dmon
>>> --offroot md127
>>> root      3326  0.0  0.2  14920 10820 ?        SLsl 15:16   0:00 mdmon
>>> --takeover md127
>>> root      3343  0.0  0.2  14920 10820 ?        SLsl 15:17   0:00 mdmon
>>> --takeover md127
>>>
>>> Is this expected ?
>>>
>>> Thanks.
>>
>> Nope.  That's not expected.
>>
>> mdmon should send SIGTERM to the old mdmon and then wait for it to exit.
>>
>> If the new and old mdmon were compiled different and look for the pid file in
>> different directories that might explain what you see.
>>
>> If you compile mdadm from source it will use /run/mdadm.  However if your
>> distro doesn't have /run the the distro-provided mdadm will be compiled
>> differently.
>>
>
> It doesn't to be the case.
>
> Actually sending SIGTERM manually to mdmon has no effects.
>
>
> # mdadm --version
> mdadm - v3.2.6 - 25th October 2012
>
> # ps aux | grep dmon
> root       235  0.1  1.0  80612 10976 ?        SLsl 19:08   0:00 @dmon
> --offroot md127
> root       339  0.0  1.0  15044 10944 ?        SLsl 19:08   0:00
> /sbin/mdmon --takeover md127
>
> # cat /run/mdadm/md127.pid
> 339
>
> # kill -SIGTERM 339
> # ps aux | grep dmon
> root       235  0.0  1.0  80612 10976 ?        SLsl 19:08   0:00 @dmon
> --offroot md127
> root       339  0.0  1.0  15044 10944 ?        SLsl 19:08   0:00
> /sbin/mdmon --takeover md127
>
> # ps aux | grep dmon
> root       235  0.0  1.0  80612 10976 ?        SLsl 19:08   0:00 @dmon
> --offroot md127
> root       339  0.0  1.0  15044 10944 ?        SLsl 19:08   0:00
> /sbin/mdmon --takeover md127
> root      2352  0.1  1.0  15076 10976 ?        SLsl 19:12   0:00 mdmon
> --takeover /dev/md127
>
> # cat /run/mdadm/md127.pid
> 2352
>
> # pkill -SIGTERM mdmon
> [root@localhost ~]# ps aux | grep dmon
> root       235  0.0  1.0  80612 10976 ?        SLsl 19:08   0:00 @dmon
> --offroot md127
> root       339  0.0  1.0  80580 10944 ?        SLsl 19:08   0:00
> /sbin/mdmon --takeover md127
> root      2352  0.0  1.0  80612 10976 ?        SLsl 19:12   0:00 mdmon
> --takeover /dev/md127
>
> Can't you reproduce ?
>

Here are some additionnal information:

# cat /proc/mdstat
Personalities : [raid1]
md126 : active raid1 sda[1] sdb[0]
      2064384 blocks super external:/md127/0 [2/2] [UU]

md127 : inactive sda[1](S) sdb[0](S)
      65536 blocks super external:ddf

unused devices: <none>


# mdadm -E /dev/md126
/dev/md126:
   MBR Magic : aa55
Partition[0] :       367447 sectors at         2048 (type 82)
Partition[1] :      3755997 sectors at       372708 (type 05)
[root@localhost ~]# mdadm -E /dev/md127
/dev/md127:
          Magic : de11de11
        Version : 01.02.00
Controller GUID : 4C696E75:782D4D44:20202020:2020206C:6F63616C:686F7374
                  (Linux-MD       localhost)
 Container GUID : 4C696E75:782D4D44:DEADBEEF:00000000:3F4FB732:8435623D
                  (Linux-MD 08/28/13 22:27:30)
            Seq : 00000001
  Redundant hdr : no
  Virtual Disks : 1

      VD GUID[0] : 4C696E75:782D4D44:DEADBEEF:00000000:3F4FB739:E0C8B16E
                  (Linux-MD 08/28/13 22:27:37)
         unit[0] : 126
        state[0] : Optimal, Not Consistent
   init state[0] : Fully Initialised
       access[0] : Read/Write
         Name[0] : array1
 Raid Devices[0] : 2 (0 1)
   Raid Level[0] : RAID1
  Device Size[0] : 2064384
   Array Size[0] : 2064384

 Physical Disks : 2
      Number    RefNo      Size       Device      Type/State
         0    2cf00056   2064384K /dev/sda        active/Online
         1    b342fbdc   2064384K /dev/sdb        active/Online

I used GDB to trace what's going on in the monitor (thread) when
sending a SIGTERM to the mdmon process:

Breakpoint 1, wait_and_act (container=0x141a2c0, nowait=0) at monitor.c:634
634        for (a = *aap; a ; a = a->next) {
(gdb) n
632        rv = 0;
(gdb)
633        dirty_arrays = 0;
(gdb)
634        for (a = *aap; a ; a = a->next) {
(gdb)
649                int ret = read_and_act(a);
(gdb) p *a
$2 = {
  info = {
    array = {
      major_version = 0,
      minor_version = 0,
      patch_version = 0,
      ctime = 0,
      level = 1,
      size = 0,
      nr_disks = 0,
      raid_disks = 2,
      md_minor = 0,
      not_persistent = 0,
      utime = 0,
      state = 0,
      active_disks = 0,
      working_disks = 0,
      failed_disks = 0,
      spare_disks = 0,
      layout = 0,
      chunk_size = 0
    },
    disk = {
      number = 0,
      major = 0,
      minor = 0,
      raid_disk = 0,
      state = 0
    },
    events = 0,
    uuid = {0, 0, 0, 0},
    name = '\000' <repeats 32 times>,
    data_offset = 0,
    component_size = 4128768,
    custom_array_size = 0,
    reshape_active = 0,
    reshape_progress = 0,
    recovery_blocked = 0,
    {
      resync_start = 18446744073709551615,
      recovery_start = 18446744073709551615
    },
    bitmap_offset = 0,
    safe_mode_delay = 0,
    new_level = 0,
    delta_disks = 0,
    new_layout = 0,
    new_chunk = 0,
    errors = 0,
    cache_size = 0,
    mismatch_cnt = 0,
    text_version = '\000' <repeats 49 times>,
    container_member = 0,
    container_enough = 0,
    sys_name = "md126", '\000' <repeats 14 times>,
    devs = 0x14213c0,
    next = 0x0,
    recovery_fd = 0,
    state_fd = 12,
    prev_state = 0,
    curr_state = 0,
    next_state = 0
  },
  container = 0x141a2c0,
  next = 0x0,
  replaces = 0x0,
  to_remove = 0,
  action_fd = 11,
  resync_start_fd = 13,
  metadata_fd = 14,
  sync_completed_fd = 15,
  last_checkpoint = 3872640,
  prev_state = active,
  curr_state = active,
  next_state = bad_word,
  prev_action = idle,
  curr_action = idle,
  next_action = bad_action,
  check_degraded = 0,
  check_reshape = 0,
  devnum = 126
}

(gdb) n
636            if (a->replaces && !discard_this) {
(gdb)
648            if (a->container && !a->to_remove) {
(gdb)
649                int ret = read_and_act(a);
(gdb)
656                if (sigterm && !(ret & ARRAY_DIRTY))
(gdb) p ret
$3 = 1

It seems that the array is dirty, and that's the reason why the
monitor never terminates.

I got the same behaviour if I'm starting a new mdmon process on this
array using --takeover switch. It seems weird since in my
understanding --takeover indicate mdmon to replace the current mdmon
process even if the array is dirty, no ?

Do you have any idea on this behaviour ?
-- 
Francis