Re: raid5 reshape is stuck

[NeilBrown <neilb@suse.de>]

[-- Attachment #1: Type: text/plain, Size: 6364 bytes --]

On Thu, 28 May 2015 02:32:51 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:

> 
> 
> ----- Original Message -----
> > From: "NeilBrown" <neilb@suse.de>
> > To: "Xiao Ni" <xni@redhat.com>
> > Cc: linux-raid@vger.kernel.org
> > Sent: Thursday, May 28, 2015 6:59:58 AM
> > Subject: Re: raid5 reshape is stuck
> > 
> > On Wed, 27 May 2015 08:04:24 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
> > 
> > > 
> > > 
> > > ----- Original Message -----
> > > > From: "NeilBrown" <neilb@suse.de>
> > > > To: "Xiao Ni" <xni@redhat.com>
> > > > Cc: linux-raid@vger.kernel.org
> > > > Sent: Wednesday, May 27, 2015 7:34:49 PM
> > > > Subject: Re: raid5 reshape is stuck
> > > > 
> > > > On Wed, 27 May 2015 07:28:04 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
> > > > 
> > > > 
> > > > > [root@intel-waimeabay-hedt-01 mdadm]# cat
> > > > > /usr/lib/systemd/system/mdadm-grow-continue\@.service
> > > > > #  This file is part of mdadm.
> > > > > #
> > > > > #  mdadm is free software; you can redistribute it and/or modify it
> > > > > #  under the terms of the GNU General Public License as published by
> > > > > #  the Free Software Foundation; either version 2 of the License, or
> > > > > #  (at your option) any later version.
> > > > > 
> > > > > [Unit]
> > > > > Description=Manage MD Reshape on /dev/%I
> > > > > DefaultDependencies=no
> > > > > 
> > > > > [Service]
> > > > > ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I
> > > > > --backup-file=/root/tmp0
> > > > 
> > > > Please remove the ---backup-file=/root/tmp0 for further testing.  The
> > > > patch I
> > > > provided should make that unnecessary.
> > > > 
> > > > > StandardInput=null
> > > > > StandardOutput=null
> > > > > StandardError=null
> > > > 
> > > > Could you try removing these - that might allow error messages to appear.
> > > > I wonder why I included them - they shouldn't be needed.
> > > > 
> > > > Thanks,
> > > > NeilBrown
> > > > 
> > > > 
> > > 
> > > [root@intel-waimeabay-hedt-01 mdadm]# mdadm -CR /dev/md0 -l5 -n4
> > > /dev/loop[0-3] --assume-clean
> > > mdadm: /dev/loop0 appears to be part of a raid array:
> > >        level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
> > > mdadm: /dev/loop1 appears to be part of a raid array:
> > >        level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
> > > mdadm: /dev/loop2 appears to be part of a raid array:
> > >        level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
> > > mdadm: /dev/loop3 appears to be part of a raid array:
> > >        level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
> > > mdadm: Defaulting to version 1.2 metadata
> > > mdadm: array /dev/md0 started.
> > > [root@intel-waimeabay-hedt-01 mdadm]# mdadm /dev/md0 -a /dev/loop4
> > > mdadm: added /dev/loop4
> > > [root@intel-waimeabay-hedt-01 mdadm]# mdadm --grow /dev/md0
> > > --raid-devices=5
> > > mdadm: Need to backup 6144K of critical section..
> > > [root@intel-waimeabay-hedt-01 mdadm]# cat /proc/mdstat
> > > Personalities : [raid6] [raid5] [raid4]
> > > md0 : active raid5 loop4[4] loop3[3] loop2[2] loop1[1] loop0[0]
> > >       1532928 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/5]
> > >       [UUUUU]
> > >       [>....................]  reshape =  0.0% (0/510976) finish=532.2min
> > >       speed=0K/sec
> > >       
> > > unused devices: <none>
> > > [root@intel-waimeabay-hedt-01 mdadm]# cat
> > > /usr/lib/systemd/system/mdadm-grow-continue\@.service
> > > #  This file is part of mdadm.
> > > #
> > > #  mdadm is free software; you can redistribute it and/or modify it
> > > #  under the terms of the GNU General Public License as published by
> > > #  the Free Software Foundation; either version 2 of the License, or
> > > #  (at your option) any later version.
> > > 
> > > [Unit]
> > > Description=Manage MD Reshape on /dev/%I
> > > DefaultDependencies=no
> > > 
> > > [Service]
> > > ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I
> > > #StandardInput=null
> > > #StandardOutput=null
> > > #StandardError=null
> > > KillMode=none
> > > 
> > > 
> > > The problem still exist. And there are messages in /var/log/messages
> > > 
> > > May 27 08:03:29 intel-waimeabay-hedt-01 systemd:
> > > mdadm-grow-continue@md0.service: main process exited, code=exited,
> > > status=1/FAILURE
> > > May 27 08:03:29 intel-waimeabay-hedt-01 systemd: Unit
> > > mdadm-grow-continue@md0.service entered failed state.
> > > 
> > 
> > Does
> >   systemctl status -l mdadm-grow-continue@md0.service
> > 
> > report anything different.  That was the result I expected from removing the
> > Standard*=null lines.
> > 
> > I assume the new mdadm is installed in /usr/sbin/mdadm.
> > 
> > Thanks,
> > NeilBrown
> > 
> 
> Yes! There are some new messages:
> [root@intel-waimeabay-hedt-01 ~]# systemctl status -l mdadm-grow-continue@md0.service
> mdadm-grow-continue@md0.service - Manage MD Reshape on /dev/md0
>    Loaded: loaded (/usr/lib/systemd/system/mdadm-grow-continue@.service; static)
>    Active: failed (Result: exit-code) since Thu 2015-05-28 02:30:50 EDT; 2s ago
>   Process: 26618 ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I (code=exited, status=1/FAILURE)
>  Main PID: 26618 (code=exited, status=1/FAILURE)
> 
> May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Started Manage MD Reshape on /dev/md0.
> May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com mdadm[26618]: mdadm: Need to backup 6144K of critical section..
> May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com mdadm[26618]: mdadm: array: cannot open component /dev/vcs6
> May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: mdadm-grow-continue@md0.service: main process exited, code=exited, status=1/FAILURE
> May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Unit mdadm-grow-continue@md0.service entered failed state.

any idea why it cannot open it?

The message is probably coming from reshape_prepare_fdlist()
Could you get those "pr_err"s to print out errno as well?
The device really has to exist, because mdadm has managed to find that name
in /dev.  Could this be a 'selinux' related issue?  I can only think that it
might be a permission problem but root shouldn't have those.

Thanks,
NeilBrown

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]