* Re: Fwd: Installing Linux directly onto RAID6 Array...........
From: Mikael Abrahamsson @ 2015-05-27 4:59 UTC (permalink / raw)
To: Wols Lists; +Cc: Another Sillyname, linux-raid
In-Reply-To: <5564D373.30705@youngman.org.uk>
On Tue, 26 May 2015, Wols Lists wrote:
> And you're wasting a lot of disk space. What's important to you (that a
> live swap disk shouldn't fail) is not important to me and doesn't seem
> to be important to Another Sillyname.
I am not so sure. He seems to want "maximum resiliance", or at least
that's what he's saying.
> I don't know why Another Sillyname wants to chain his swap partitions,
> but it's his choice. Maybe like me, his system gets rebooted a couple of
> times a day, and failure to start is a far more real risk than failure
> in use. He's running Fedora - that seems likely then ...
Yes, it's his choice, but he should also make this choice understanding
what he's doing, so it's an informed and corrent choice. When I read his
postings, I am not so sure this is the case.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: [PATCH v4 06/11] md/raid5: get rid of bio_fits_rdev()
From: Christoph Hellwig @ 2015-05-27 8:15 UTC (permalink / raw)
To: NeilBrown
Cc: Ming Lin, Christoph Hellwig, lkml, Kent Overstreet, Jens Axboe,
Dongsu Park, linux-raid
In-Reply-To: <20150527090309.2cddf33b@notabene.brown>
On Wed, May 27, 2015 at 09:03:09AM +1000, NeilBrown wrote:
> But I don't think the sequence is right.
>
> You cannot remove ->merge_bvec_fn for *any* stacked device until *all* devices
> make use of blk_queue_split() (or otherwise handle arbitrarily large bios).
>
> I think it would be easiest to:
> - add blk_queue_split() and call it from common code before ->make_request_fn
> is called. The ensure all devices can accept arbitrarily large bios.
> - driver-by-driver remove merge_bvec_fn and make sure the the driver can cope
> with arbitrary bios themselve, calling blk_queue_split in the make_request
> function only if needed
> - finally remove the call to blk_queue_split from the common code.
>
> Does that make sense to others?
Ok, sorry for leading in the wrong direction. Because we stack
->merge_bvec_fn calls we do indeed need it until the end.
In that case I think it's better to just go back to something like the
original order and not split the patches up, everything else is just
getting too complicated unfrotunately.
^ permalink raw reply
* Re: raid5 reshape is stuck
From: Xiao Ni @ 2015-05-27 11:28 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150527111004.5f136f23@notabene.brown>
----- Original Message -----
> From: "NeilBrown" <neilb@suse.de>
> To: "Xiao Ni" <xni@redhat.com>
> Cc: linux-raid@vger.kernel.org
> Sent: Wednesday, May 27, 2015 9:10:04 AM
> Subject: Re: raid5 reshape is stuck
>
> On Wed, 27 May 2015 10:02:53 +1000 NeilBrown <neilb@suse.de> wrote:
>
> > On Tue, 26 May 2015 06:48:23 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
> >
> >
> > > > >
> > > > > In the function continue_via_systemd the parent find pid is bigger
> > > > > than
> > > > > 0 and
> > > > > status is 0. So it return 1. So it have no opportunity to call
> > > > > child_monitor.
> > > >
> > > > If continue_via_systemd succeeded, that implies that
> > > > systemctl start mdadm-grow-continue@mdXXX.service
> > > >
> > > > succeeded. So
> > > > mdadm --grow --continue /dev/mdXXX
> > > >
> > > > was run, so that mdadm should call 'child_monitor' and update sync_max
> > > > when
> > > > appropriate. Can you check if it does?
> > >
> > > The service is not running.
> > >
> > > [root@intel-waimeabay-hedt-01 create_assemble]# systemctl start
> > > mdadm-grow-continue@md0.service
> > > [root@intel-waimeabay-hedt-01 create_assemble]# echo $?
> > > 0
> > > [root@intel-waimeabay-hedt-01 create_assemble]# systemctl status
> > > mdadm-grow-continue@md0.service
> > > mdadm-grow-continue@md0.service - Manage MD Reshape on /dev/md0
> > > Loaded: loaded (/usr/lib/systemd/system/mdadm-grow-continue@.service;
> > > static)
> > > Active: failed (Result: exit-code) since Tue 2015-05-26 05:33:59 EDT;
> > > 21s ago
> > > Process: 5374 ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I
> > > (code=exited, status=1/FAILURE)
> > > Main PID: 5374 (code=exited, status=1/FAILURE)
> > >
> > > May 26 05:33:59 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com
> > > systemd[1]: Started Manage MD Reshape on /dev/md0.
> > > May 26 05:33:59 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com
> > > systemd[1]: mdadm-grow-continue@md0.service: main process exited, ...URE
> > > May 26 05:33:59 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com
> > > systemd[1]: Unit mdadm-grow-continue@md0.service entered failed state.
> > > Hint: Some lines were ellipsized, use -l to show in full.
> >
> > Hmm.. I wonder why systemctl isn't reporting the error message from mdadm.
I don't know the reason too. The return value $? is 0 after run systemctl start.
But the status is failed.
> >
> >
> > >
> > > [root@intel-waimeabay-hedt-01 create_assemble]# mdadm --grow --continue
> > > /dev/md0 --backup-file=tmp0
> > > mdadm: Need to backup 6144K of critical section..
> > >
> > > Now the reshape start.
> > >
> > > Try modify the service file :
> > > ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I
> > > --backup-file=/root/tmp0
> > >
> > > It doesn't work too.
> >
> > I tried that change and it make it work.
[root@intel-waimeabay-hedt-01 mdadm]# cat /usr/lib/systemd/system/mdadm-grow-continue\@.service
# This file is part of mdadm.
#
# mdadm is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
[Unit]
Description=Manage MD Reshape on /dev/%I
DefaultDependencies=no
[Service]
ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I --backup-file=/root/tmp0
StandardInput=null
StandardOutput=null
StandardError=null
KillMode=none
[root@intel-waimeabay-hedt-01 mdadm]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 loop4[4] loop3[3] loop2[2] loop1[1] loop0[0]
1532928 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/5] [UUUUU]
[>....................] reshape = 0.0% (0/510976) finish=532.2min speed=0K/sec
unused devices: <none>
[root@intel-waimeabay-hedt-01 mdadm]# systemctl start mdadm-grow-continue@md0.service
[root@intel-waimeabay-hedt-01 mdadm]# systemctl status mdadm-grow-continue@md0.service
mdadm-grow-continue@md0.service - Manage MD Reshape on /dev/md0
Loaded: loaded (/usr/lib/systemd/system/mdadm-grow-continue@.service; static)
Active: failed (Result: exit-code) since Wed 2015-05-27 02:45:40 EDT; 12s ago
Process: 24596 ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I --backup-file=/root/tmp0 (code=exited, status=1/FAILURE)
Main PID: 24596 (code=exited, status=1/FAILURE)
May 27 02:45:40 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Started Manage MD Reshape on /dev/md0.
May 27 02:45:40 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: mdadm-grow-continue@md0.service: main process exited, ...URE
May 27 02:45:40 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Unit mdadm-grow-continue@md0.service entered failed state.
Hint: Some lines were ellipsized, use -l to show in full.
It's still failed after changing the file.
> >
> > >
> > > [root@intel-waimeabay-hedt-01 ~]# systemctl daemon-reload
> > > [root@intel-waimeabay-hedt-01 ~]# systemctl start
> > > mdadm-grow-continue@md0.service
> > > [root@intel-waimeabay-hedt-01 ~]# systemctl status
> > > mdadm-grow-continue@md0.service
> > > mdadm-grow-continue@md0.service - Manage MD Reshape on /dev/md0
> > > Loaded: loaded (/usr/lib/systemd/system/mdadm-grow-continue@.service;
> > > static)
> > > Active: failed (Result: exit-code) since Tue 2015-05-26 05:50:22 EDT;
> > > 10s ago
> > > Process: 6475 ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I
> > > --backup-file=/root/tmp0 (code=exited, status=1/FAILURE)
> > > Main PID: 6475 (code=exited, status=1/FAILURE)
> > >
> > > May 26 05:50:22 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com
> > > systemd[1]: Started Manage MD Reshape on /dev/md0.
> > > May 26 05:50:22 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com
> > > systemd[1]: mdadm-grow-continue@md0.service: main process exited, ...URE
> > > May 26 05:50:22 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com
> > > systemd[1]: Unit mdadm-grow-continue@md0.service entered failed state.
> > > Hint: Some lines were ellipsized, use -l to show in full.
> > >
> > >
> > >
> > > >
> > > >
> > > > >
> > > > >
> > > > > And if it want to set sync_max to 0 until the backup has been
> > > > > taken. Why
> > > > > does not
> > > > > set sync_max to 0 directly, but use the value reshape_progress? There
> > > > > is a
> > > > > little confused.
> > > >
> > > > When reshaping an array to a different array of the same size, such as
> > > > a
> > > > 4-driver RAID5 to a 5-driver RAID6, then mdadm needs to backup, one
> > > > piece at
> > > > a time, the entire array (unless it can change data_offset, which is a
> > > > relatively new ability).
> > > >
> > > > If you stop an array when it is in the middle of such a reshape, and
> > > > then
> > > > reassemble the array, the backup process need to recommence where it
> > > > left
> > > > off.
> > > > So it tells the kernel that the reshape can progress as far as where it
> > > > was
> > > > up to before. So 'sync_max' is set based on the value of
> > > > 'reshape_progress'.
> > > > (This will happen almost instantly).
> > > >
> > > > Then the background mdadm (or the mdadm started by systemd) will backup
> > > > the
> > > > next few stripes, update sync_max, wait for those stripes to be
> > > > reshaped,
> > > > then
> > > > discard the old backup, create a new one of the few stripes after that,
> > > > and
> > > > continue.
> > > >
> > > > Does that make it a little clearer?
> > >
> > > This is a big dinner for me. I need digest this for a while. Thanks very
> > > much
> > > for this. What's the "backup process"?
> > >
> > > Could you explain backup in detail. I read the man about backup file.
> > >
> > > When relocating the first few stripes on a RAID5 or RAID6, it is not
> > > possible to keep the data on disk completely
> > > consistent and crash-proof. To provide the required safety, mdadm
> > > disables writes to the array while this "critical
> > > section" is reshaped, and takes a backup of the data that is in that
> > > section.
> > >
> > > What's the reason about data consistent when relocate data?
> >
> > If you are reshaping a RAID5 from 3 drives to 4 drives, then the first
> > stripe
> > will start out as:
> >
> > D0 D1 P -
> >
> > and you want to change it to
> >
> > D0 D1 D2 P
> >
> > If the system crashes while that is happening, you won't know if either or
> > both of D2 and P were written, but it is fairly safe just to assume they
> > weren't and recalculate the parity.
> > However the second stripe will initially be:
> >
> > P D2 D3
> >
> > and you want to change it to
> >
> > P D3 D4 D5
> >
> > If you crash in the middle of doing that you cannot know which block is D3
> > - if either. D4 might have been written, and D3 not yet written. So D3 is
> > lost.
> >
> > So mdadm takes a copy of a whole stripe, allows the kernel to reshape that
> > one stripe, updates the metadata to record that the stripe has been fully
> > reshaped, and then discards the backup.
> > So if you crash in the middle of reshaping the second stripe above, mdadm
> > will restore it from the backup.
> >
> > The backup can be stored in a separate file, or in a device which is being
> > added to the array.
> >
> >
> > The reason why "mdadm --grow --continue" doesn't work unless you add the
> > "--backup=...." is because it doesn't find the "device being added" - it
> > looks for a spare, but there aren't any spares any more. That should be
> > easy enough to fix.
:) I got this. Thanks for the details
>
> That wasn't too painful - I think this fixes the problem.
> Could you confirm?
>
> Thanks,
> NeilBrown
>
>
> diff --git a/Grow.c b/Grow.c
> index a20ff3e70142..85de1d27f03a 100644
> --- a/Grow.c
> +++ b/Grow.c
> @@ -850,7 +850,8 @@ int reshape_prepare_fdlist(char *devname,
> for (sd = sra->devs; sd; sd = sd->next) {
> if (sd->disk.state & (1<<MD_DISK_FAULTY))
> continue;
> - if (sd->disk.state & (1<<MD_DISK_SYNC)) {
> + if (sd->disk.state & (1<<MD_DISK_SYNC) &&
> + sd->disk.raid_disk < raid_disks) {
> char *dn = map_dev(sd->disk.major,
> sd->disk.minor, 1);
> fdlist[sd->disk.raid_disk]
> @@ -3184,7 +3185,7 @@ started:
> d = reshape_prepare_fdlist(devname, sra, odisks,
> nrdisks, blocks, backup_file,
> fdlist, offsets);
> - if (d < 0) {
> + if (d < odisks) {
> goto release;
> }
> if ((st->ss->manage_reshape == NULL) ||
> @@ -3196,7 +3197,7 @@ started:
> devname);
> pr_err(" Please provide one with \"--backup=...\"\n");
> goto release;
> - } else if (sra->array.spare_disks == 0) {
> + } else if (d == odisks) {
> pr_err("%s: Cannot grow - need a spare or backup-file to backup critical
> section\n", devname);
> goto release;
> }
>
>
I tried this, it doesn't work.
^ permalink raw reply
* Re: raid5 reshape is stuck
From: NeilBrown @ 2015-05-27 11:34 UTC (permalink / raw)
To: Xiao Ni; +Cc: linux-raid
In-Reply-To: <2129908770.5092770.1432726084717.JavaMail.zimbra@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 1004 bytes --]
On Wed, 27 May 2015 07:28:04 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
> [root@intel-waimeabay-hedt-01 mdadm]# cat /usr/lib/systemd/system/mdadm-grow-continue\@.service
> # This file is part of mdadm.
> #
> # mdadm is free software; you can redistribute it and/or modify it
> # under the terms of the GNU General Public License as published by
> # the Free Software Foundation; either version 2 of the License, or
> # (at your option) any later version.
>
> [Unit]
> Description=Manage MD Reshape on /dev/%I
> DefaultDependencies=no
>
> [Service]
> ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I --backup-file=/root/tmp0
Please remove the ---backup-file=/root/tmp0 for further testing. The patch I
provided should make that unnecessary.
> StandardInput=null
> StandardOutput=null
> StandardError=null
Could you try removing these - that might allow error messages to appear.
I wonder why I included them - they shouldn't be needed.
Thanks,
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Which physical device failed?
From: Michael Munger @ 2015-05-27 12:04 UTC (permalink / raw)
To: linux-raid
In a degraded array (4 disks: sdb,sdc,sdd,sde in RAID 5). One of my disks
failed (sdc).
Is there an easy way to figure out which PHYSICAL disk that is? I have
always just unplugged the sata cables one at a time to map it, but it
occurs to me that the drives may be enumerated in order (SATA port0 is sda,
SATA port 1 is sdb, etc...)
Or, does the OS have access to serial numbers, etc...?
I have to guide someone through a drive replacement on the phone, and it
would be great if I could tell them exactly which drive to swap out...
Thanks in advance,
Michael
^ permalink raw reply
* Re: raid5 reshape is stuck
From: Xiao Ni @ 2015-05-27 12:04 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150527213449.6e017deb@notabene.brown>
----- Original Message -----
> From: "NeilBrown" <neilb@suse.de>
> To: "Xiao Ni" <xni@redhat.com>
> Cc: linux-raid@vger.kernel.org
> Sent: Wednesday, May 27, 2015 7:34:49 PM
> Subject: Re: raid5 reshape is stuck
>
> On Wed, 27 May 2015 07:28:04 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
>
>
> > [root@intel-waimeabay-hedt-01 mdadm]# cat
> > /usr/lib/systemd/system/mdadm-grow-continue\@.service
> > # This file is part of mdadm.
> > #
> > # mdadm is free software; you can redistribute it and/or modify it
> > # under the terms of the GNU General Public License as published by
> > # the Free Software Foundation; either version 2 of the License, or
> > # (at your option) any later version.
> >
> > [Unit]
> > Description=Manage MD Reshape on /dev/%I
> > DefaultDependencies=no
> >
> > [Service]
> > ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I
> > --backup-file=/root/tmp0
>
> Please remove the ---backup-file=/root/tmp0 for further testing. The patch I
> provided should make that unnecessary.
>
> > StandardInput=null
> > StandardOutput=null
> > StandardError=null
>
> Could you try removing these - that might allow error messages to appear.
> I wonder why I included them - they shouldn't be needed.
>
> Thanks,
> NeilBrown
>
>
[root@intel-waimeabay-hedt-01 mdadm]# mdadm -CR /dev/md0 -l5 -n4 /dev/loop[0-3] --assume-clean
mdadm: /dev/loop0 appears to be part of a raid array:
level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
mdadm: /dev/loop1 appears to be part of a raid array:
level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
mdadm: /dev/loop2 appears to be part of a raid array:
level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
mdadm: /dev/loop3 appears to be part of a raid array:
level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
[root@intel-waimeabay-hedt-01 mdadm]# mdadm /dev/md0 -a /dev/loop4
mdadm: added /dev/loop4
[root@intel-waimeabay-hedt-01 mdadm]# mdadm --grow /dev/md0 --raid-devices=5
mdadm: Need to backup 6144K of critical section..
[root@intel-waimeabay-hedt-01 mdadm]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 loop4[4] loop3[3] loop2[2] loop1[1] loop0[0]
1532928 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/5] [UUUUU]
[>....................] reshape = 0.0% (0/510976) finish=532.2min speed=0K/sec
unused devices: <none>
[root@intel-waimeabay-hedt-01 mdadm]# cat /usr/lib/systemd/system/mdadm-grow-continue\@.service
# This file is part of mdadm.
#
# mdadm is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
[Unit]
Description=Manage MD Reshape on /dev/%I
DefaultDependencies=no
[Service]
ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I
#StandardInput=null
#StandardOutput=null
#StandardError=null
KillMode=none
The problem still exist. And there are messages in /var/log/messages
May 27 08:03:29 intel-waimeabay-hedt-01 systemd: mdadm-grow-continue@md0.service: main process exited, code=exited, status=1/FAILURE
May 27 08:03:29 intel-waimeabay-hedt-01 systemd: Unit mdadm-grow-continue@md0.service entered failed state.
Xiao
^ permalink raw reply
* Re: Which physical device failed?
From: Carsten Aulbert @ 2015-05-27 12:10 UTC (permalink / raw)
To: Michael Munger, linux-raid
In-Reply-To: <5565B2BB.5090701@gmail.com>
Hi
On 05/27/2015 02:04 PM, Michael Munger wrote:
> Or, does the OS have access to serial numbers, etc...?
>
> I have to guide someone through a drive replacement on the phone, and it
> would be great if I could tell them exactly which drive to swap out...
If you have direct knowledge, which serial number is where, you could
use hdparm -I /dev/sdX or smartctl -a /dev/sdX against the still
reachable drives.
Possibly the easiest is to *read* data from the still reachable disks
and check which LED does not blink anymore - if you have LEDs for each
of the drives (dd if=/dev/sdX of=/dev/null).
Otherwise someone needs to look at the "open" hardware to find the
serial numbers.
Does this help?
cheers
Carsten
^ permalink raw reply
* Re: Which physical device failed?
From: Roman Mamedov @ 2015-05-27 12:27 UTC (permalink / raw)
To: Carsten Aulbert; +Cc: Michael Munger, linux-raid
In-Reply-To: <5565B41B.8040104@aei.mpg.de>
[-- Attachment #1: Type: text/plain, Size: 926 bytes --]
On Wed, 27 May 2015 14:10:03 +0200
Carsten Aulbert <Carsten.Aulbert@aei.mpg.de> wrote:
> On 05/27/2015 02:04 PM, Michael Munger wrote:
> > Or, does the OS have access to serial numbers, etc...?
> >
> > I have to guide someone through a drive replacement on the phone, and it
> > would be great if I could tell them exactly which drive to swap out...
>
> If you have direct knowledge, which serial number is where, you could
> use hdparm -I /dev/sdX or smartctl -a /dev/sdX against the still
> reachable drives.
If /dev/sdc is still present in the system (even if not responding correctly to
hdparm or smartctl anymore), you should be able to find its serial number from
the udev symlink that was registered earlier, by running e.g.:
ls -la /dev/disk/by-id/ | grep sdc$
Serial number is typically the last piece of the ID, after the manufacturer
name and model number.
--
With respect,
Roman
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply
* Re: Which physical device failed?
From: Phil Turmel @ 2015-05-27 13:16 UTC (permalink / raw)
To: Roman Mamedov, Carsten Aulbert; +Cc: Michael Munger, linux-raid
In-Reply-To: <20150527172701.1090e525@natsu>
On 05/27/2015 08:27 AM, Roman Mamedov wrote:
> On Wed, 27 May 2015 14:10:03 +0200
> Carsten Aulbert <Carsten.Aulbert@aei.mpg.de> wrote:
>
>> On 05/27/2015 02:04 PM, Michael Munger wrote:
>>> Or, does the OS have access to serial numbers, etc...?
>>>
>>> I have to guide someone through a drive replacement on the phone, and it
>>> would be great if I could tell them exactly which drive to swap out...
>>
>> If you have direct knowledge, which serial number is where, you could
>> use hdparm -I /dev/sdX or smartctl -a /dev/sdX against the still
>> reachable drives.
>
> If /dev/sdc is still present in the system (even if not responding correctly to
> hdparm or smartctl anymore), you should be able to find its serial number from
> the udev symlink that was registered earlier, by running e.g.:
>
> ls -la /dev/disk/by-id/ | grep sdc$
>
> Serial number is typically the last piece of the ID, after the manufacturer
> name and model number.
>
This is one of the reasons I wrote lsdrv [1], especially after I noticed
that the port sequence it reports is stable for the various ports on
every mobo and sata expansion card I've handled. Per controller, at least.
I save of copy of an lsdrv report for each system I commission so that
there's no ambiguity later.
Phil
[1] https://github.com/pturmel/lsdrv
^ permalink raw reply
* Re: Which physical device failed?
From: Michael Munger @ 2015-05-27 14:24 UTC (permalink / raw)
To: Phil Turmel, Roman Mamedov, Carsten Aulbert; +Cc: linux-raid
In-Reply-To: <5565C3AA.7020809@turmel.org>
Phil!
lsdrv did the trick. Roman and Carsten were correct, and I was in the
middle of executing their suggestions when I got your email.
Running lsdrv showed me that the drive was, in fact, still there, but
just inactive. I removed it and re-added it to the array, and it is
rebuilding.
I have sent the output of lsdrv to the client with the note: "Keep this
for your records. At some point, a drive will fail, and we'll use this
to figure out which drive you need to replace."
Thank you all.
On 05/27/2015 09:16 AM, Phil Turmel wrote:
> On 05/27/2015 08:27 AM, Roman Mamedov wrote:
>> On Wed, 27 May 2015 14:10:03 +0200
>> Carsten Aulbert <Carsten.Aulbert@aei.mpg.de> wrote:
>>
>>> On 05/27/2015 02:04 PM, Michael Munger wrote:
>>>> Or, does the OS have access to serial numbers, etc...?
>>>>
>>>> I have to guide someone through a drive replacement on the phone, and it
>>>> would be great if I could tell them exactly which drive to swap out...
>>> If you have direct knowledge, which serial number is where, you could
>>> use hdparm -I /dev/sdX or smartctl -a /dev/sdX against the still
>>> reachable drives.
>> If /dev/sdc is still present in the system (even if not responding correctly to
>> hdparm or smartctl anymore), you should be able to find its serial number from
>> the udev symlink that was registered earlier, by running e.g.:
>>
>> ls -la /dev/disk/by-id/ | grep sdc$
>>
>> Serial number is typically the last piece of the ID, after the manufacturer
>> name and model number.
>>
> This is one of the reasons I wrote lsdrv [1], especially after I noticed
> that the port sequence it reports is stable for the various ports on
> every mobo and sata expansion card I've handled. Per controller, at least.
>
> I save of copy of an lsdrv report for each system I commission so that
> there's no ambiguity later.
>
> Phil
>
> [1] https://github.com/pturmel/lsdrv
>
^ permalink raw reply
* Re: Which physical device failed?
From: Wols Lists @ 2015-05-27 17:48 UTC (permalink / raw)
To: Roman Mamedov, Carsten Aulbert; +Cc: Michael Munger, linux-raid
In-Reply-To: <20150527172701.1090e525@natsu>
On 27/05/15 13:27, Roman Mamedov wrote:
> On Wed, 27 May 2015 14:10:03 +0200 Carsten Aulbert
> <Carsten.Aulbert@aei.mpg.de> wrote:
>
>> On 05/27/2015 02:04 PM, Michael Munger wrote:
>>> Or, does the OS have access to serial numbers, etc...?
>>>
>>> I have to guide someone through a drive replacement on the
>>> phone, and it would be great if I could tell them exactly which
>>> drive to swap out...
>>
>> If you have direct knowledge, which serial number is where, you
>> could use hdparm -I /dev/sdX or smartctl -a /dev/sdX against the
>> still reachable drives.
>
> If /dev/sdc is still present in the system (even if not responding
> correctly to hdparm or smartctl anymore), you should be able to
> find its serial number from the udev symlink that was registered
> earlier, by running e.g.:
>
> ls -la /dev/disk/by-id/ | grep sdc$
>
> Serial number is typically the last piece of the ID, after the
> manufacturer name and model number.
>
Just for info, I've done an ls -al on my by-id directory, and I have
no clue whatsoever as to what the serial number is. All my drives
appear twice (Seagate Barracudas), there is no manufacturer name that
I can see, and while the model number appears in one of the records,
there is nothing obvious to indicate whether what follows is the
serial number or whether the serial number is part of the other
directory entry.
Not helped, of course, by the fact I have no clue what the serial
number looks like ... :-)
ashdown by-id # ls -la
total 0
drwxr-xr-x 2 root root 620 May 27 08:10 .
drwxr-xr-x 7 root root 140 May 27 08:10 ..
lrwxrwxrwx 1 root root 9 May 27 08:10 ata-Optiarc_DVD_RW_AD-7241S ->
../../sr0
lrwxrwxrwx 1 root root 9 May 27 08:10
ata-ST3000DM001-1CH166_W1F4JWRP -> ../../sdb
lrwxrwxrwx 1 root root 10 May 27 08:10
ata-ST3000DM001-1CH166_W1F4JWRP-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 May 27 08:10
ata-ST3000DM001-1CH166_W1F4JWRP-part2 -> ../../sdb2
lrwxrwxrwx 1 root root 10 May 27 08:10
ata-ST3000DM001-1CH166_W1F4JWRP-part3 -> ../../sdb3
lrwxrwxrwx 1 root root 10 May 27 08:10
ata-ST3000DM001-1CH166_W1F4JWRP-part4 -> ../../sdb4
lrwxrwxrwx 1 root root 10 May 27 08:10
ata-ST3000DM001-1CH166_W1F4JWRP-part5 -> ../../sdb5
lrwxrwxrwx 1 root root 9 May 27 08:10
ata-ST3000DM001-1CH166_W1F50K0F -> ../../sda
lrwxrwxrwx 1 root root 10 May 27 08:10
ata-ST3000DM001-1CH166_W1F50K0F-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 May 27 08:10
ata-ST3000DM001-1CH166_W1F50K0F-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 May 27 08:10
ata-ST3000DM001-1CH166_W1F50K0F-part3 -> ../../sda3
lrwxrwxrwx 1 root root 10 May 27 08:10
ata-ST3000DM001-1CH166_W1F50K0F-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 May 27 08:10
ata-ST3000DM001-1CH166_W1F50K0F-part5 -> ../../sda5
lrwxrwxrwx 1 root root 11 May 27 08:10 md-name-ashdown:0 -> ../../md126
lrwxrwxrwx 1 root root 11 May 27 08:10
md-uuid-39b62a86:885bf50d:33f360cf:a409bd11 -> ../../md126
lrwxrwxrwx 1 root root 11 May 27 08:10
md-uuid-42514e8a:2d127c98:7c2f52fe:60835b32 -> ../../md127
lrwxrwxrwx 1 root root 11 May 27 08:10
md-uuid-69270eac:a840f6e7:0199064b:d5863c5d -> ../../md125
lrwxrwxrwx 1 root root 9 May 27 08:10 wwn-0x5000c50072af4400 ->
../../sdb
lrwxrwxrwx 1 root root 10 May 27 08:10 wwn-0x5000c50072af4400-part1
-> ../../sdb1
lrwxrwxrwx 1 root root 10 May 27 08:10 wwn-0x5000c50072af4400-part2
-> ../../sdb2
lrwxrwxrwx 1 root root 10 May 27 08:10 wwn-0x5000c50072af4400-part3
-> ../../sdb3
lrwxrwxrwx 1 root root 10 May 27 08:10 wwn-0x5000c50072af4400-part4
-> ../../sdb4
lrwxrwxrwx 1 root root 10 May 27 08:10 wwn-0x5000c50072af4400-part5
-> ../../sdb5
lrwxrwxrwx 1 root root 9 May 27 08:10 wwn-0x5000c500737a98a4 ->
../../sda
lrwxrwxrwx 1 root root 10 May 27 08:10 wwn-0x5000c500737a98a4-part1
-> ../../sda1
lrwxrwxrwx 1 root root 10 May 27 08:10 wwn-0x5000c500737a98a4-part2
-> ../../sda2
lrwxrwxrwx 1 root root 10 May 27 08:10 wwn-0x5000c500737a98a4-part3
-> ../../sda3
lrwxrwxrwx 1 root root 10 May 27 08:10 wwn-0x5000c500737a98a4-part4
-> ../../sda4
lrwxrwxrwx 1 root root 10 May 27 08:10 wwn-0x5000c500737a98a4-part5
-> ../../sda5
Cheers,
Wol
^ permalink raw reply
* Re: Which physical device failed?
From: Roman Mamedov @ 2015-05-27 18:02 UTC (permalink / raw)
To: Wols Lists; +Cc: Carsten Aulbert, Michael Munger, linux-raid
In-Reply-To: <55660373.3030700@youngman.org.uk>
[-- Attachment #1: Type: text/plain, Size: 996 bytes --]
On Wed, 27 May 2015 18:48:35 +0100
Wols Lists <antlists@youngman.org.uk> wrote:
> Just for info, I've done an ls -al on my by-id directory, and I have
> no clue whatsoever as to what the serial number is. All my drives
> appear twice (Seagate Barracudas), there is no manufacturer name that
> I can see, and while the model number appears in one of the records,
> there is nothing obvious to indicate whether what follows is the
> serial number or whether the serial number is part of the other
> directory entry.
>
> Not helped, of course, by the fact I have no clue what the serial
> number looks like ... :-)
>
> ata-ST3000DM001-1CH166_W1F4JWRP -> ../../sdb
> ata-ST3000DM001-1CH166_W1F50K0F -> ../../sda
From a picture of this model's top side
http://www.nix.ru/autocatalog/hdd_seagate/126689_2245_draft_large.jpg
the serial number on this model is 8 alphanumeric characters, so in your case
W1F4JWRP and W1F50K0F are the serial numbers.
--
With respect,
Roman
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply
* Re: Which physical device failed?
From: Wilson, Jonathan @ 2015-05-27 18:16 UTC (permalink / raw)
To: Phil Turmel; +Cc: Roman Mamedov, Carsten Aulbert, Michael Munger, linux-raid
In-Reply-To: <5565C3AA.7020809@turmel.org>
On Wed, 2015-05-27 at 09:16 -0400, Phil Turmel wrote:
> On 05/27/2015 08:27 AM, Roman Mamedov wrote:
> > On Wed, 27 May 2015 14:10:03 +0200
> > Carsten Aulbert <Carsten.Aulbert@aei.mpg.de> wrote:
> >
> >> On 05/27/2015 02:04 PM, Michael Munger wrote:
> >>> Or, does the OS have access to serial numbers, etc...?
> >>>
> >>> I have to guide someone through a drive replacement on the phone, and it
> >>> would be great if I could tell them exactly which drive to swap out...
> >>
> >> If you have direct knowledge, which serial number is where, you could
> >> use hdparm -I /dev/sdX or smartctl -a /dev/sdX against the still
> >> reachable drives.
> >
> > If /dev/sdc is still present in the system (even if not responding correctly to
> > hdparm or smartctl anymore), you should be able to find its serial number from
> > the udev symlink that was registered earlier, by running e.g.:
> >
> > ls -la /dev/disk/by-id/ | grep sdc$
> >
> > Serial number is typically the last piece of the ID, after the manufacturer
> > name and model number.
> >
>
> This is one of the reasons I wrote lsdrv [1], especially after I noticed
> that the port sequence it reports is stable for the various ports on
> every mobo and sata expansion card I've handled. Per controller, at least.
Interesting that you should say that as on my z97 board if I do a power
off, power on, the drives do indeed stay numbered to the sata ports...
however if I do a "restart" sometimes, very rarely, the drives are
listed with different sdX designations. It may be a quirk of either the
efi, linux, or the fact the drives are not, I believe, turned off during
a restart which may impact on designation. I didn't investigate the whys
as I just noticed that two drives had swapped in two arrays (sdb moved
from a raid10 into the raid6 and that sdc moved from the raid6 into the
raid10) which scared the heck out of me until I realised that it was
just the sdX that had changed not the drives so for one minute I was
expecting massive problems to ensue.
>
> I save of copy of an lsdrv report for each system I commission so that
> there's no ambiguity later.
>
> Phil
>
> [1] https://github.com/pturmel/lsdrv
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* Re: Which physical device failed?
From: Can Jeuleers @ 2015-05-27 18:19 UTC (permalink / raw)
To: Roman Mamedov, Wols Lists; +Cc: Carsten Aulbert, Michael Munger, linux-raid
In-Reply-To: <20150527230223.4ef48749@natsu>
On 27/05/15 20:02, Roman Mamedov wrote:
>> ata-ST3000DM001-1CH166_W1F4JWRP -> ../../sdb
>> ata-ST3000DM001-1CH166_W1F50K0F -> ../../sda
>
> From a picture of this model's top side
> http://www.nix.ru/autocatalog/hdd_seagate/126689_2245_draft_large.jpg
> the serial number on this model is 8 alphanumeric characters, so in your case
> W1F4JWRP and W1F50K0F are the serial numbers.
Indeed, and hdparm -i will also tell you.
^ permalink raw reply
* Re: Which physical device failed?
From: Phil Turmel @ 2015-05-27 18:38 UTC (permalink / raw)
To: Wilson, Jonathan
Cc: Roman Mamedov, Carsten Aulbert, Michael Munger, linux-raid
In-Reply-To: <BLU436-SMTP2403ED333F23D8DB23296C398CB0@phx.gbl>
Hi Jonathan,
On 05/27/2015 02:16 PM, Wilson, Jonathan wrote:
> On Wed, 2015-05-27 at 09:16 -0400, Phil Turmel wrote:
>> This is one of the reasons I wrote lsdrv [1], especially after I noticed
>> that the port sequence it reports is stable for the various ports on
>> every mobo and sata expansion card I've handled. Per controller, at least.
>
> Interesting that you should say that as on my z97 board if I do a power
> off, power on, the drives do indeed stay numbered to the sata ports...
> however if I do a "restart" sometimes, very rarely, the drives are
> listed with different sdX designations. It may be a quirk of either the
> efi, linux, or the fact the drives are not, I believe, turned off during
> a restart which may impact on designation. I didn't investigate the whys
> as I just noticed that two drives had swapped in two arrays (sdb moved
> from a raid10 into the raid6 and that sdc moved from the raid6 into the
> raid10) which scared the heck out of me until I realised that it was
> just the sdX that had changed not the drives so for one minute I was
> expecting massive problems to ensue.
I didn't say the names are consistent--in fact, your experience is
entirely normal with modern kernel's device discovery. The names come
out the same for many people by chance (timing, interrupts, whatever).
But a new kernel might have slight differences, and then the names change.
My comment was referring to the SCSI LUNs "N:P:Q:R" that appear under
each controller in lsdrv. These correspond to the hostN/targetP:Q:R
folders in sysfs. P:Q:R appears to reliably correspond to physical
ports. Sometimes with phantom ports, but reliably so. Which is why
lsdrv shows them in order, even if empty. For the controllers I've
played with so far, that is. Consider labeling your cables with the
mobo or adapter's silkscreened port ID and the corresponding P:Q:R string.
Anyways, MD uses the superblock metadata to make sure array members are
properly assembled regardless what name they have at any moment. LVM
does so as well. The mdadm --detail report that shows kernel names
cannot be trusted across boots or between kernel versions.
If you are using /dev/sdX names in fstab or mdadm.conf, you may be
surprised by a boot failure at some point.
Phil
^ permalink raw reply
* Re: Which physical device failed?
From: Wols Lists @ 2015-05-27 18:38 UTC (permalink / raw)
To: Can Jeuleers, Roman Mamedov; +Cc: Carsten Aulbert, Michael Munger, linux-raid
In-Reply-To: <55660AA0.4030308@gmail.com>
On 27/05/15 19:19, Can Jeuleers wrote:
> On 27/05/15 20:02, Roman Mamedov wrote:
>>> ata-ST3000DM001-1CH166_W1F4JWRP -> ../../sdb
>>> ata-ST3000DM001-1CH166_W1F50K0F -> ../../sda
>>
>> From a picture of this model's top side
>> http://www.nix.ru/autocatalog/hdd_seagate/126689_2245_draft_large.jpg
>> the serial number on this model is 8 alphanumeric characters, so in your case
>> W1F4JWRP and W1F50K0F are the serial numbers.
>
> Indeed, and hdparm -i will also tell you.
>
ashdown by-id # hdparm -i /dev/sda
bash: hdparm: command not found
:-)
Just emerged it and yes, lots of useful info thanks.
Cheers,
Wol
^ permalink raw reply
* Re: Which physical device failed?
From: Benjamin ESTRABAUD @ 2015-05-27 18:41 UTC (permalink / raw)
To: Can Jeuleers, Roman Mamedov, Wols Lists
Cc: Carsten Aulbert, Michael Munger, linux-raid
In-Reply-To: <55660AA0.4030308@gmail.com>
On 27/05/15 19:19, Can Jeuleers wrote:
> On 27/05/15 20:02, Roman Mamedov wrote:
>>> ata-ST3000DM001-1CH166_W1F4JWRP -> ../../sdb
>>> ata-ST3000DM001-1CH166_W1F50K0F -> ../../sda
>>
>> From a picture of this model's top side
>> http://www.nix.ru/autocatalog/hdd_seagate/126689_2245_draft_large.jpg
>> the serial number on this model is 8 alphanumeric characters, so in your case
>> W1F4JWRP and W1F50K0F are the serial numbers.
>
> Indeed, and hdparm -i will also tell you.
Or you could use the excellent "sg_inq" tool from the sg3_utils package
by Douglas Gilbert to get the serial number, model and more.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* Re: raid5 reshape is stuck
From: NeilBrown @ 2015-05-27 22:59 UTC (permalink / raw)
To: Xiao Ni; +Cc: linux-raid
In-Reply-To: <476656362.5105083.1432728264276.JavaMail.zimbra@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 3958 bytes --]
On Wed, 27 May 2015 08:04:24 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
>
>
> ----- Original Message -----
> > From: "NeilBrown" <neilb@suse.de>
> > To: "Xiao Ni" <xni@redhat.com>
> > Cc: linux-raid@vger.kernel.org
> > Sent: Wednesday, May 27, 2015 7:34:49 PM
> > Subject: Re: raid5 reshape is stuck
> >
> > On Wed, 27 May 2015 07:28:04 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
> >
> >
> > > [root@intel-waimeabay-hedt-01 mdadm]# cat
> > > /usr/lib/systemd/system/mdadm-grow-continue\@.service
> > > # This file is part of mdadm.
> > > #
> > > # mdadm is free software; you can redistribute it and/or modify it
> > > # under the terms of the GNU General Public License as published by
> > > # the Free Software Foundation; either version 2 of the License, or
> > > # (at your option) any later version.
> > >
> > > [Unit]
> > > Description=Manage MD Reshape on /dev/%I
> > > DefaultDependencies=no
> > >
> > > [Service]
> > > ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I
> > > --backup-file=/root/tmp0
> >
> > Please remove the ---backup-file=/root/tmp0 for further testing. The patch I
> > provided should make that unnecessary.
> >
> > > StandardInput=null
> > > StandardOutput=null
> > > StandardError=null
> >
> > Could you try removing these - that might allow error messages to appear.
> > I wonder why I included them - they shouldn't be needed.
> >
> > Thanks,
> > NeilBrown
> >
> >
>
> [root@intel-waimeabay-hedt-01 mdadm]# mdadm -CR /dev/md0 -l5 -n4 /dev/loop[0-3] --assume-clean
> mdadm: /dev/loop0 appears to be part of a raid array:
> level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
> mdadm: /dev/loop1 appears to be part of a raid array:
> level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
> mdadm: /dev/loop2 appears to be part of a raid array:
> level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
> mdadm: /dev/loop3 appears to be part of a raid array:
> level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
> mdadm: Defaulting to version 1.2 metadata
> mdadm: array /dev/md0 started.
> [root@intel-waimeabay-hedt-01 mdadm]# mdadm /dev/md0 -a /dev/loop4
> mdadm: added /dev/loop4
> [root@intel-waimeabay-hedt-01 mdadm]# mdadm --grow /dev/md0 --raid-devices=5
> mdadm: Need to backup 6144K of critical section..
> [root@intel-waimeabay-hedt-01 mdadm]# cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4]
> md0 : active raid5 loop4[4] loop3[3] loop2[2] loop1[1] loop0[0]
> 1532928 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/5] [UUUUU]
> [>....................] reshape = 0.0% (0/510976) finish=532.2min speed=0K/sec
>
> unused devices: <none>
> [root@intel-waimeabay-hedt-01 mdadm]# cat /usr/lib/systemd/system/mdadm-grow-continue\@.service
> # This file is part of mdadm.
> #
> # mdadm is free software; you can redistribute it and/or modify it
> # under the terms of the GNU General Public License as published by
> # the Free Software Foundation; either version 2 of the License, or
> # (at your option) any later version.
>
> [Unit]
> Description=Manage MD Reshape on /dev/%I
> DefaultDependencies=no
>
> [Service]
> ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I
> #StandardInput=null
> #StandardOutput=null
> #StandardError=null
> KillMode=none
>
>
> The problem still exist. And there are messages in /var/log/messages
>
> May 27 08:03:29 intel-waimeabay-hedt-01 systemd: mdadm-grow-continue@md0.service: main process exited, code=exited, status=1/FAILURE
> May 27 08:03:29 intel-waimeabay-hedt-01 systemd: Unit mdadm-grow-continue@md0.service entered failed state.
>
Does
systemctl status -l mdadm-grow-continue@md0.service
report anything different. That was the result I expected from removing the
Standard*=null lines.
I assume the new mdadm is installed in /usr/sbin/mdadm.
Thanks,
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: raid5 reshape is stuck
From: Xiao Ni @ 2015-05-28 6:32 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150528085958.0f95e323@notabene.brown>
----- Original Message -----
> From: "NeilBrown" <neilb@suse.de>
> To: "Xiao Ni" <xni@redhat.com>
> Cc: linux-raid@vger.kernel.org
> Sent: Thursday, May 28, 2015 6:59:58 AM
> Subject: Re: raid5 reshape is stuck
>
> On Wed, 27 May 2015 08:04:24 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
>
> >
> >
> > ----- Original Message -----
> > > From: "NeilBrown" <neilb@suse.de>
> > > To: "Xiao Ni" <xni@redhat.com>
> > > Cc: linux-raid@vger.kernel.org
> > > Sent: Wednesday, May 27, 2015 7:34:49 PM
> > > Subject: Re: raid5 reshape is stuck
> > >
> > > On Wed, 27 May 2015 07:28:04 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
> > >
> > >
> > > > [root@intel-waimeabay-hedt-01 mdadm]# cat
> > > > /usr/lib/systemd/system/mdadm-grow-continue\@.service
> > > > # This file is part of mdadm.
> > > > #
> > > > # mdadm is free software; you can redistribute it and/or modify it
> > > > # under the terms of the GNU General Public License as published by
> > > > # the Free Software Foundation; either version 2 of the License, or
> > > > # (at your option) any later version.
> > > >
> > > > [Unit]
> > > > Description=Manage MD Reshape on /dev/%I
> > > > DefaultDependencies=no
> > > >
> > > > [Service]
> > > > ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I
> > > > --backup-file=/root/tmp0
> > >
> > > Please remove the ---backup-file=/root/tmp0 for further testing. The
> > > patch I
> > > provided should make that unnecessary.
> > >
> > > > StandardInput=null
> > > > StandardOutput=null
> > > > StandardError=null
> > >
> > > Could you try removing these - that might allow error messages to appear.
> > > I wonder why I included them - they shouldn't be needed.
> > >
> > > Thanks,
> > > NeilBrown
> > >
> > >
> >
> > [root@intel-waimeabay-hedt-01 mdadm]# mdadm -CR /dev/md0 -l5 -n4
> > /dev/loop[0-3] --assume-clean
> > mdadm: /dev/loop0 appears to be part of a raid array:
> > level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
> > mdadm: /dev/loop1 appears to be part of a raid array:
> > level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
> > mdadm: /dev/loop2 appears to be part of a raid array:
> > level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
> > mdadm: /dev/loop3 appears to be part of a raid array:
> > level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
> > mdadm: Defaulting to version 1.2 metadata
> > mdadm: array /dev/md0 started.
> > [root@intel-waimeabay-hedt-01 mdadm]# mdadm /dev/md0 -a /dev/loop4
> > mdadm: added /dev/loop4
> > [root@intel-waimeabay-hedt-01 mdadm]# mdadm --grow /dev/md0
> > --raid-devices=5
> > mdadm: Need to backup 6144K of critical section..
> > [root@intel-waimeabay-hedt-01 mdadm]# cat /proc/mdstat
> > Personalities : [raid6] [raid5] [raid4]
> > md0 : active raid5 loop4[4] loop3[3] loop2[2] loop1[1] loop0[0]
> > 1532928 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/5]
> > [UUUUU]
> > [>....................] reshape = 0.0% (0/510976) finish=532.2min
> > speed=0K/sec
> >
> > unused devices: <none>
> > [root@intel-waimeabay-hedt-01 mdadm]# cat
> > /usr/lib/systemd/system/mdadm-grow-continue\@.service
> > # This file is part of mdadm.
> > #
> > # mdadm is free software; you can redistribute it and/or modify it
> > # under the terms of the GNU General Public License as published by
> > # the Free Software Foundation; either version 2 of the License, or
> > # (at your option) any later version.
> >
> > [Unit]
> > Description=Manage MD Reshape on /dev/%I
> > DefaultDependencies=no
> >
> > [Service]
> > ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I
> > #StandardInput=null
> > #StandardOutput=null
> > #StandardError=null
> > KillMode=none
> >
> >
> > The problem still exist. And there are messages in /var/log/messages
> >
> > May 27 08:03:29 intel-waimeabay-hedt-01 systemd:
> > mdadm-grow-continue@md0.service: main process exited, code=exited,
> > status=1/FAILURE
> > May 27 08:03:29 intel-waimeabay-hedt-01 systemd: Unit
> > mdadm-grow-continue@md0.service entered failed state.
> >
>
> Does
> systemctl status -l mdadm-grow-continue@md0.service
>
> report anything different. That was the result I expected from removing the
> Standard*=null lines.
>
> I assume the new mdadm is installed in /usr/sbin/mdadm.
>
> Thanks,
> NeilBrown
>
Yes! There are some new messages:
[root@intel-waimeabay-hedt-01 ~]# systemctl status -l mdadm-grow-continue@md0.service
mdadm-grow-continue@md0.service - Manage MD Reshape on /dev/md0
Loaded: loaded (/usr/lib/systemd/system/mdadm-grow-continue@.service; static)
Active: failed (Result: exit-code) since Thu 2015-05-28 02:30:50 EDT; 2s ago
Process: 26618 ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I (code=exited, status=1/FAILURE)
Main PID: 26618 (code=exited, status=1/FAILURE)
May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Started Manage MD Reshape on /dev/md0.
May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com mdadm[26618]: mdadm: Need to backup 6144K of critical section..
May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com mdadm[26618]: mdadm: array: cannot open component /dev/vcs6
May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: mdadm-grow-continue@md0.service: main process exited, code=exited, status=1/FAILURE
May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Unit mdadm-grow-continue@md0.service entered failed state.
Xiao
^ permalink raw reply
* Re: raid5 reshape is stuck
From: NeilBrown @ 2015-05-28 6:49 UTC (permalink / raw)
To: Xiao Ni; +Cc: linux-raid
In-Reply-To: <45685228.5717919.1432794771906.JavaMail.zimbra@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 6364 bytes --]
On Thu, 28 May 2015 02:32:51 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
>
>
> ----- Original Message -----
> > From: "NeilBrown" <neilb@suse.de>
> > To: "Xiao Ni" <xni@redhat.com>
> > Cc: linux-raid@vger.kernel.org
> > Sent: Thursday, May 28, 2015 6:59:58 AM
> > Subject: Re: raid5 reshape is stuck
> >
> > On Wed, 27 May 2015 08:04:24 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
> >
> > >
> > >
> > > ----- Original Message -----
> > > > From: "NeilBrown" <neilb@suse.de>
> > > > To: "Xiao Ni" <xni@redhat.com>
> > > > Cc: linux-raid@vger.kernel.org
> > > > Sent: Wednesday, May 27, 2015 7:34:49 PM
> > > > Subject: Re: raid5 reshape is stuck
> > > >
> > > > On Wed, 27 May 2015 07:28:04 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
> > > >
> > > >
> > > > > [root@intel-waimeabay-hedt-01 mdadm]# cat
> > > > > /usr/lib/systemd/system/mdadm-grow-continue\@.service
> > > > > # This file is part of mdadm.
> > > > > #
> > > > > # mdadm is free software; you can redistribute it and/or modify it
> > > > > # under the terms of the GNU General Public License as published by
> > > > > # the Free Software Foundation; either version 2 of the License, or
> > > > > # (at your option) any later version.
> > > > >
> > > > > [Unit]
> > > > > Description=Manage MD Reshape on /dev/%I
> > > > > DefaultDependencies=no
> > > > >
> > > > > [Service]
> > > > > ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I
> > > > > --backup-file=/root/tmp0
> > > >
> > > > Please remove the ---backup-file=/root/tmp0 for further testing. The
> > > > patch I
> > > > provided should make that unnecessary.
> > > >
> > > > > StandardInput=null
> > > > > StandardOutput=null
> > > > > StandardError=null
> > > >
> > > > Could you try removing these - that might allow error messages to appear.
> > > > I wonder why I included them - they shouldn't be needed.
> > > >
> > > > Thanks,
> > > > NeilBrown
> > > >
> > > >
> > >
> > > [root@intel-waimeabay-hedt-01 mdadm]# mdadm -CR /dev/md0 -l5 -n4
> > > /dev/loop[0-3] --assume-clean
> > > mdadm: /dev/loop0 appears to be part of a raid array:
> > > level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
> > > mdadm: /dev/loop1 appears to be part of a raid array:
> > > level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
> > > mdadm: /dev/loop2 appears to be part of a raid array:
> > > level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
> > > mdadm: /dev/loop3 appears to be part of a raid array:
> > > level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
> > > mdadm: Defaulting to version 1.2 metadata
> > > mdadm: array /dev/md0 started.
> > > [root@intel-waimeabay-hedt-01 mdadm]# mdadm /dev/md0 -a /dev/loop4
> > > mdadm: added /dev/loop4
> > > [root@intel-waimeabay-hedt-01 mdadm]# mdadm --grow /dev/md0
> > > --raid-devices=5
> > > mdadm: Need to backup 6144K of critical section..
> > > [root@intel-waimeabay-hedt-01 mdadm]# cat /proc/mdstat
> > > Personalities : [raid6] [raid5] [raid4]
> > > md0 : active raid5 loop4[4] loop3[3] loop2[2] loop1[1] loop0[0]
> > > 1532928 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/5]
> > > [UUUUU]
> > > [>....................] reshape = 0.0% (0/510976) finish=532.2min
> > > speed=0K/sec
> > >
> > > unused devices: <none>
> > > [root@intel-waimeabay-hedt-01 mdadm]# cat
> > > /usr/lib/systemd/system/mdadm-grow-continue\@.service
> > > # This file is part of mdadm.
> > > #
> > > # mdadm is free software; you can redistribute it and/or modify it
> > > # under the terms of the GNU General Public License as published by
> > > # the Free Software Foundation; either version 2 of the License, or
> > > # (at your option) any later version.
> > >
> > > [Unit]
> > > Description=Manage MD Reshape on /dev/%I
> > > DefaultDependencies=no
> > >
> > > [Service]
> > > ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I
> > > #StandardInput=null
> > > #StandardOutput=null
> > > #StandardError=null
> > > KillMode=none
> > >
> > >
> > > The problem still exist. And there are messages in /var/log/messages
> > >
> > > May 27 08:03:29 intel-waimeabay-hedt-01 systemd:
> > > mdadm-grow-continue@md0.service: main process exited, code=exited,
> > > status=1/FAILURE
> > > May 27 08:03:29 intel-waimeabay-hedt-01 systemd: Unit
> > > mdadm-grow-continue@md0.service entered failed state.
> > >
> >
> > Does
> > systemctl status -l mdadm-grow-continue@md0.service
> >
> > report anything different. That was the result I expected from removing the
> > Standard*=null lines.
> >
> > I assume the new mdadm is installed in /usr/sbin/mdadm.
> >
> > Thanks,
> > NeilBrown
> >
>
> Yes! There are some new messages:
> [root@intel-waimeabay-hedt-01 ~]# systemctl status -l mdadm-grow-continue@md0.service
> mdadm-grow-continue@md0.service - Manage MD Reshape on /dev/md0
> Loaded: loaded (/usr/lib/systemd/system/mdadm-grow-continue@.service; static)
> Active: failed (Result: exit-code) since Thu 2015-05-28 02:30:50 EDT; 2s ago
> Process: 26618 ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I (code=exited, status=1/FAILURE)
> Main PID: 26618 (code=exited, status=1/FAILURE)
>
> May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Started Manage MD Reshape on /dev/md0.
> May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com mdadm[26618]: mdadm: Need to backup 6144K of critical section..
> May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com mdadm[26618]: mdadm: array: cannot open component /dev/vcs6
> May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: mdadm-grow-continue@md0.service: main process exited, code=exited, status=1/FAILURE
> May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Unit mdadm-grow-continue@md0.service entered failed state.
any idea why it cannot open it?
The message is probably coming from reshape_prepare_fdlist()
Could you get those "pr_err"s to print out errno as well?
The device really has to exist, because mdadm has managed to find that name
in /dev. Could this be a 'selinux' related issue? I can only think that it
might be a permission problem but root shouldn't have those.
Thanks,
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Which physical device failed?
From: Wilson, Jonathan @ 2015-05-28 9:03 UTC (permalink / raw)
To: Phil Turmel; +Cc: Roman Mamedov, Carsten Aulbert, Michael Munger, linux-raid
In-Reply-To: <55660F37.8000402@turmel.org>
On Wed, 2015-05-27 at 14:38 -0400, Phil Turmel wrote:
> Hi Jonathan,
>
> On 05/27/2015 02:16 PM, Wilson, Jonathan wrote:
> > On Wed, 2015-05-27 at 09:16 -0400, Phil Turmel wrote:
> >> This is one of the reasons I wrote lsdrv [1], especially after I noticed
> >> that the port sequence it reports is stable for the various ports on
> >> every mobo and sata expansion card I've handled. Per controller, at least.
> >
> > Interesting that you should say that as on my z97 board if I do a power
> > off, power on, the drives do indeed stay numbered to the sata ports...
> > however if I do a "restart" sometimes, very rarely, the drives are
> > listed with different sdX designations. It may be a quirk of either the
> > efi, linux, or the fact the drives are not, I believe, turned off during
> > a restart which may impact on designation. I didn't investigate the whys
> > as I just noticed that two drives had swapped in two arrays (sdb moved
> > from a raid10 into the raid6 and that sdc moved from the raid6 into the
> > raid10) which scared the heck out of me until I realised that it was
> > just the sdX that had changed not the drives so for one minute I was
> > expecting massive problems to ensue.
>
> I didn't say the names are consistent--in fact, your experience is
> entirely normal with modern kernel's device discovery. The names come
> out the same for many people by chance (timing, interrupts, whatever).
> But a new kernel might have slight differences, and then the names change.
My mistake I misinterpreted what you said.
>
> My comment was referring to the SCSI LUNs "N:P:Q:R" that appear under
> each controller in lsdrv. These correspond to the hostN/targetP:Q:R
> folders in sysfs. P:Q:R appears to reliably correspond to physical
> ports. Sometimes with phantom ports, but reliably so. Which is why
> lsdrv shows them in order, even if empty. For the controllers I've
> played with so far, that is. Consider labeling your cables with the
> mobo or adapter's silkscreened port ID and the corresponding P:Q:R string.
I did something similar, "card/port" A-[1-6] (main cpu sata, port)
B-[1-2] (marvel on board, port) C-[1-4] (jbod marvel card, port).
The main reason was that unlike older boards for some strange reason
(circuit paths I guess) the ordering of the physical sata port sockets
bares no relation to the sequence.
>
> Anyways, MD uses the superblock metadata to make sure array members are
> properly assembled regardless what name they have at any moment. LVM
> does so as well. The mdadm --detail report that shows kernel names
> cannot be trusted across boots or between kernel versions.
>
> If you are using /dev/sdX names in fstab or mdadm.conf, you may be
> surprised by a boot failure at some point.
Luckily I have always used GUIDs, but after 4 OS upgrades and many years
of use had never once seen devices not follow chip/port when providing
sdX names so it came as quite a shock even though I knew sdX names can
not be trusted to remain consistent. The only time I had seen them
shift/swap was when say a usb device grabbed sda and the rest shifted 1
letter up.
>
> Phil
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* [PATCH 1/4] raid5: wakeup raid5d when R5_ALLOC_MORE is set
From: Shaohua Li @ 2015-05-29 0:33 UTC (permalink / raw)
To: linux-raid; +Cc: neilb
The run time stripe allocation is done at raid5d. When we set the
R5_ALLOC_MORE flag, we should notify raid5d to handle it
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/raid5.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 61e8e04..bfa2042 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -674,9 +674,11 @@ get_active_stripe(struct r5conf *conf, sector_t sector,
if (!test_bit(R5_INACTIVE_BLOCKED, &conf->cache_state)) {
sh = get_free_stripe(conf, hash);
if (!sh && llist_empty(&conf->released_stripes) &&
- !test_bit(R5_DID_ALLOC, &conf->cache_state))
+ !test_bit(R5_DID_ALLOC, &conf->cache_state)) {
set_bit(R5_ALLOC_MORE,
&conf->cache_state);
+ md_wakeup_thread(conf->mddev->thread);
+ }
}
if (noblock && sh == NULL)
break;
--
1.8.1
^ permalink raw reply related
* [PATCH 2/4] raid5: grown at least NR_STRIPE_HASH_LOCKS stripes
From: Shaohua Li @ 2015-05-29 0:33 UTC (permalink / raw)
To: linux-raid; +Cc: neilb
In-Reply-To: <a92a0c36813c740c5de65a95b75ef161c1f2ede1.1432859513.git.shli@fb.com>
stripes are in hash list. If we are waiting for a free stripe, we must
make sure there is free stripe in corresponding hash list. To do this,
we simpliy allocate at lease NR_STRIPE_HASH_LOCKS stripes at runtime
stripe allocation.
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/raid5.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index bfa2042..0cceb71 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -5867,7 +5867,9 @@ static void raid5d(struct md_thread *thread)
spin_unlock_irq(&conf->device_lock);
if (test_and_clear_bit(R5_ALLOC_MORE, &conf->cache_state)) {
- grow_one_stripe(conf, __GFP_NOWARN);
+ int i;
+ for (i = 0; i < NR_STRIPE_HASH_LOCKS; i++)
+ grow_one_stripe(conf, __GFP_NOWARN);
/* Set flag even if allocation failed. This helps
* slow down allocation requests when mem is short
*/
--
1.8.1
^ permalink raw reply related
* [PATCH 3/4] raid5: ignore released_stripes check
From: Shaohua Li @ 2015-05-29 0:33 UTC (permalink / raw)
To: linux-raid; +Cc: neilb
In-Reply-To: <a92a0c36813c740c5de65a95b75ef161c1f2ede1.1432859513.git.shli@fb.com>
conf->released_stripes list ins't always related if there is free
stripes pending. active stripes can be in the list too.
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/raid5.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 0cceb71..67626f3 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -673,8 +673,8 @@ get_active_stripe(struct r5conf *conf, sector_t sector,
if (!sh) {
if (!test_bit(R5_INACTIVE_BLOCKED, &conf->cache_state)) {
sh = get_free_stripe(conf, hash);
- if (!sh && llist_empty(&conf->released_stripes) &&
- !test_bit(R5_DID_ALLOC, &conf->cache_state)) {
+ if (!sh && !test_bit(R5_DID_ALLOC,
+ &conf->cache_state)) {
set_bit(R5_ALLOC_MORE,
&conf->cache_state);
md_wakeup_thread(conf->mddev->thread);
--
1.8.1
^ permalink raw reply related
* [PATCH 4/4] raid5: fix wakeup condition
From: Shaohua Li @ 2015-05-29 0:33 UTC (permalink / raw)
To: linux-raid; +Cc: neilb
In-Reply-To: <a92a0c36813c740c5de65a95b75ef161c1f2ede1.1432859513.git.shli@fb.com>
Since we have several stripe hash list, the conf->active_stripes doesn't
determine if there is free stripe in a specific hash list, so delete the
check. After this, the R5_INACTIVE_BLOCKED check is inappropriate. There
is no point not to wakeup a task if there is free stripe.
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/raid5.c | 6 +-----
1 file changed, 1 insertion(+), 5 deletions(-)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 67626f3..4b5a03c 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -687,11 +687,7 @@ get_active_stripe(struct r5conf *conf, sector_t sector,
&conf->cache_state);
wait_event_lock_irq(
conf->wait_for_stripe,
- !list_empty(conf->inactive_list + hash) &&
- (atomic_read(&conf->active_stripes)
- < (conf->max_nr_stripes * 3 / 4)
- || !test_bit(R5_INACTIVE_BLOCKED,
- &conf->cache_state)),
+ !list_empty(conf->inactive_list + hash),
*(conf->hash_locks + hash));
clear_bit(R5_INACTIVE_BLOCKED,
&conf->cache_state);
--
1.8.1
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox