* Re: RAID1 working correctly, error messages during boot
From: NeilBrown @ 2015-05-21 11:24 UTC (permalink / raw)
To: Hans Malissa; +Cc: linux-raid
In-Reply-To: <20150521212049.6f9cf650@notabene.brown>
[-- Attachment #1: Type: text/plain, Size: 256 bytes --]
On Thu, 21 May 2015 21:20:49 +1000 NeilBrown <neilb@suse.de> wrote:
> Adding the 'md' arg to the "mkinitrd" command might be sufficient, but I
> don't have a 12.2 install lying around that I can play with.
Make that "mkinitrd -f md".
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: RAID1 working correctly, error messages during boot
From: NeilBrown @ 2015-05-21 11:20 UTC (permalink / raw)
To: Hans Malissa; +Cc: linux-raid
In-Reply-To: <CE885044-F8E8-4286-9CAC-1D17560F1E79@me.com>
[-- Attachment #1: Type: text/plain, Size: 2348 bytes --]
On Mon, 18 May 2015 14:29:23 -0600 Hans Malissa <hmalissa@me.com> wrote:
> I have a software-RAID1 that seems to be working correctly:
>
> # cat /proc/mdstat
> Personalities : [raid1]
> md0 : active raid1 sdc1[1] sdb1[0]
> 976629568 blocks super 1.2 [2/2] [UU]
>
> unused devices: <none>
>
> # mdadm --detail /dev/md0
> /dev/md0:
> Version : 1.2
> Creation Time : Sun May 17 15:21:30 2015
> Raid Level : raid1
> Array Size : 976629568 (931.39 GiB 1000.07 GB)
> Used Dev Size : 976629568 (931.39 GiB 1000.07 GB)
> Raid Devices : 2
> Total Devices : 2
> Persistence : Superblock is persistent
>
> Update Time : Mon May 18 10:28:36 2015
> State : clean
> Active Devices : 2
> Working Devices : 2
> Failed Devices : 0
> Spare Devices : 0
>
> Name : eprb21:0 (local to host eprb21)
> UUID : 0901fe50:444a29b6:d3caff14:e45ef9cc
> Events : 19
>
> Number Major Minor RaidDevice State
> 0 8 17 0 active sync /dev/sdb1
> 1 8 33 1 active sync /dev/sdc1
>
> But, on the other hand, when the system boots, I briefly see the following messages:
>
> doing fast boot
> Creating device nodes with udev
> udevd[174]: failed to execute ‘/sbin/mdadm’ ‘/sbin/mdadm --incremental /dev/sdb1
>
> udevd[175]: failed to execute ‘/sbin/mdadm’ ‘/sbin/mdadm --incremental /dev/sdc1 --offroot’: No such file or directory
>
> But otherwise the system appears to run normally. After booting, /dev/md0 seems to be working correctly.
> What does it mean, and should I worry about it? What can I do about it? My system is openSUSE 12.2.
> Thanks a lot,
>
I suspect that some udev scripts on the initrd say to run mdadm, but mdadm
isn't installed on the initrd.
/dev/md0 doesn't hold the root filesystem does it? mdadm is only installed
on the initrd if it is needed for root, swap, or suspend-to-disk.
If you really wanted to get rid of the messages - which are definitely
harmless - you would need to recreate the initrd either without those udev
rules files, or with mdadm.
Adding the 'md' arg to the "mkinitrd" command might be sufficient, but I
don't have a 12.2 install lying around that I can play with.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* WARNING: Software Raid 0 on SSD's and discard corrupts data
From: Holger Kiehl @ 2015-05-21 8:24 UTC (permalink / raw)
To: linux-kernel, linux-raid; +Cc: NeilBrown
Hello,
all users using a Software Raid 0 on SSD's with discard should disable
discard, if they use any recent kernel since mid-April 2015. The bug
was introduced by commit 47d68979cc968535cb87f3e5f2e6a3533ea48fbd and
the fix is not yet in Linus tree. The fix can be found here:
http://git.neil.brown.name/?p=md.git;a=commitdiff;h=a81157768a00e8cf8a7b43b5ea5cac931262374f
Users should immediately remove the discard option from any mounted
software Raid 0 filesystems. Any delete or modification of files can
lead to random destruction on the filesystem. Use the remount option
of the mount command to remove the discard option. Do not do it via
editing /etc/fstab if your root filesystem is on a software raid 0.
Regards,
Holger
^ permalink raw reply
* Re: Recent drive errors
From: Mikael Abrahamsson @ 2015-05-21 7:58 UTC (permalink / raw)
To: Thomas Fjellstrom; +Cc: Phil Turmel, linux-raid@vger.kernel.org
In-Reply-To: <84264713.v03zHsT0Cj@balsa>
On Tue, 19 May 2015, Thomas Fjellstrom wrote:
> How many UREs are considered "ok"? Tens, hundreds, thousands, tens of
> thousands?
I will replace any drive that have developed UNC sectors a few times, so
I'd say "less than 10".
+1 on the "set kernel timeout to more than 120 seconds". I have this in
/etc/rc.local:
for x in /sys/block/sd[a-z] ; do
echo 180 > $x/device/timeout
done
echo 4096 > /sys/block/md0/md/stripe_cache_size
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: Filesystem corruption MD (imsm) Raid0 via 2 SSD's + discard
From: NeilBrown @ 2015-05-21 7:14 UTC (permalink / raw)
To: Holger Kiehl; +Cc: Roman Mamedov, linux-kernel, linux-raid
In-Reply-To: <alpine.LRH.2.11.1505210636230.9060@praktifix.dwd.de>
[-- Attachment #1: Type: text/plain, Size: 2400 bytes --]
On Thu, 21 May 2015 06:44:27 +0000 (UTC) Holger Kiehl <Holger.Kiehl@dwd.de>
wrote:
> On Thu, 21 May 2015, NeilBrown wrote:
>
> > On Thu, 21 May 2015 01:32:13 +0500 Roman Mamedov <rm@romanrm.net> wrote:
> >
> >> On Wed, 20 May 2015 20:12:31 +0000 (UTC)
> >> Holger Kiehl <Holger.Kiehl@dwd.de> wrote:
> >>
> >>> The kernel I was running when I discovered the
> >>> problem was 4.0.2 from kernel.org. However, after reinstalling from DVD
> >>> I updated to Fedora's lattest kernel, which was 3.19.? (I do not remember
> >>> the last numbers). So that kernel seems also effected, but I assume it
> >>> contains many 'fixes' from 4.0.x. As filesystem I use ext4, distribution
> >>> is Fedora 21 and hardware is: Xeon E3-1275, 16GB ECC Ram.
> >>>
> >>> My system seems to be now running stable for some days with kernel.org
> >>> kernel 4.0.3 and with discard DISABLED. But I am still unsure what could
> >>> be the real cause.
> >>
> >> It is a bug in the 4.0.2 kernel, fixed in 4.0.3.
> >>
> >> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=785672
> >> https://bbs.archlinux.org/viewtopic.php?id=197400
> >> https://kernel.googlesource.com/pub/scm/linux/kernel/git/stable/linux-stable/+/d2dc317d564a46dfc683978a2e5a4f91434e9711
> >>
> >>
> >
> > I suspect that is a different bug.
> > I think this one is
> > https://bugzilla.kernel.org/show_bug.cgi?id=98501
> >
> Should there not be a big fat warning going around telling users to disable
> discard on Raid 0 until this is fixed? This breaks the filesystem completely
> and I believe there is absolutly no way one can get back the data.
Probably. Would you like to do that?
>
> Is this fixed in 4.0.4? And which kernels are effected? There could be many
> people running systems that have not noticed this and don't know in what
> dangerous situation they are when they delete data.
The patch was only added to my tree today. I will send to Linus tomorrow so
it should appear in the next -rc.
Any -stable kernel released since mid-April probably has the bug. It was
caused by
commit 47d68979cc968535cb87f3e5f2e6a3533ea48fbd
Once the fix gets into Linus' tree, it should get into subsequent -stable releases.
The fix is here:
http://git.neil.brown.name/?p=md.git;a=commitdiff;h=a81157768a00e8cf8a7b43b5ea5cac931262374f
commit id should remain unchanged.
Thanks,
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Filesystem corruption MD (imsm) Raid0 via 2 SSD's + discard
From: Holger Kiehl @ 2015-05-21 6:44 UTC (permalink / raw)
To: NeilBrown; +Cc: Roman Mamedov, linux-kernel, linux-raid
In-Reply-To: <20150521090845.7e228e46@notabene.brown>
On Thu, 21 May 2015, NeilBrown wrote:
> On Thu, 21 May 2015 01:32:13 +0500 Roman Mamedov <rm@romanrm.net> wrote:
>
>> On Wed, 20 May 2015 20:12:31 +0000 (UTC)
>> Holger Kiehl <Holger.Kiehl@dwd.de> wrote:
>>
>>> The kernel I was running when I discovered the
>>> problem was 4.0.2 from kernel.org. However, after reinstalling from DVD
>>> I updated to Fedora's lattest kernel, which was 3.19.? (I do not remember
>>> the last numbers). So that kernel seems also effected, but I assume it
>>> contains many 'fixes' from 4.0.x. As filesystem I use ext4, distribution
>>> is Fedora 21 and hardware is: Xeon E3-1275, 16GB ECC Ram.
>>>
>>> My system seems to be now running stable for some days with kernel.org
>>> kernel 4.0.3 and with discard DISABLED. But I am still unsure what could
>>> be the real cause.
>>
>> It is a bug in the 4.0.2 kernel, fixed in 4.0.3.
>>
>> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=785672
>> https://bbs.archlinux.org/viewtopic.php?id=197400
>> https://kernel.googlesource.com/pub/scm/linux/kernel/git/stable/linux-stable/+/d2dc317d564a46dfc683978a2e5a4f91434e9711
>>
>>
>
> I suspect that is a different bug.
> I think this one is
> https://bugzilla.kernel.org/show_bug.cgi?id=98501
>
Should there not be a big fat warning going around telling users to disable
discard on Raid 0 until this is fixed? This breaks the filesystem completely
and I believe there is absolutly no way one can get back the data.
Is this fixed in 4.0.4? And which kernels are effected? There could be many
people running systems that have not noticed this and don't know in what
dangerous situation they are when they delete data.
Regards,
Holger
^ permalink raw reply
* Re: Data corruption after resizing partition, when using bitmaps
From: Jim Paris @ 2015-05-21 5:58 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150521102444.6346dc5d@notabene.brown>
NeilBrown wrote:
> On Wed, 20 May 2015 02:31:50 -0400 Jim Paris <jim@jtan.com> wrote:
>
> > NeilBrown wrote:
> > > On Tue, 19 May 2015 10:12:40 -0400 Jim Paris <jim@jtan.com> wrote:
> > >
> > > > I had a raid1 mirror consisting of big partitions on two disks.
> > > > The first disk was 2TB, partitioned like this:
> > > >
> > > > [--sda1(128M)--][-------sda2(~2T)--------------]
> > > >
> > > > The second disk was 3TB, partitioned like this:
> > > >
> > > > [--sdb1(128M)--][-------sdb2(~3T)------------------------------------]
> > > >
> > > > sda2 and sdb2 were part of the array, which was only ~2TB in size due
> > > > to the smaller disk.
> > > >
> > > > I realized that I needed to add a BIOS boot partition to the 3TB disk,
> > > > so I removed sdb2 from the array, and repartitioned sdb like this:
> > > >
> > > > [--sdb1(128M)--][--sdb2(1M)--][-------sdb3(~3T)----------------------]
> > > >
> > > > Then I added sdb3 to the array. And lost all my data. :(
> > > >
> > > > What happened was that the last sector of the big partition did not
> > > > change location. So the metadata (0.90) at the end was still present.
> > >
> > > This is one of the big reasons why 1.x was invented.
> > >
> > > > Adding sdb3 to the array was considered a "re-add" because the UUID
> > > > and array sizes still matched the array, even though the partition
> > > > itself shrank. And the resync was thus guided by an out-of-date
> > > > bitmap, which caused very little data to actually be written to sdb3,
> > > > so half the reads from the array started returning junk. Once the
> > > > filesystem got involved, the result was rapid corruption.
> > > >
> > > > If I had not been using write-intent bitmaps, everything would have
> > > > worked fine. I only recently started using bitmaps, and never had any
> > > > problems with adjusting partitions like this before that.
> > > >
> > > > Perhaps mdadm can be more careful here -- for example, maybe checking
> > > > the actual device size and not just the "used dev size" when
> > > > determining whether to trust the bitmap.
> > >
> > > It is perfectly acceptable to have the various devices in an array of
> > > different sizes. Unfortunately I don't think there is anything that mdadm
> > > can usefully do here.
> > >
> > > Thanks for the report anyway,
> > > NeilBrown
> >
> > Hi Neil,
> >
> > Can we add u64 device_size to bitmap_super_t, and ensure that it
> > matches the actual current device size before trusting the bitmap?
>
> Well .... we could, but the bitmap_super is currently the same on all
> devices. This would make it different.
> And if we a going to change the metadata, why not just convert from 0.90 to
> 1.0?
My thinking was that the extra field could be added to bitmap_super
automatically -- just start writing it now, but only use it to
determine bitmap validity if the current value is non-zero. No
explicit user-visible conversion.
I see your point; it's a bit strange to change outdated stuff. But I
also feel that if there's something mdadm could have done to prevent
my data loss, that's worth putting in there.
> mdadm --stop /dev/mdXX
> mdadm --assemble /dev/mdXX --update=metadata /dev/...list-of-devices....
>
> You might need to remove the bitmap first, and add it back afterwards.
Cool. Much simpler than what's currently listed in the wiki for that
conversion.
Thanks Neil.
xJim
>
> NeilBrown
^ permalink raw reply
* Re: raid5 reshape is stuck
From: Xiao Ni @ 2015-05-21 3:37 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150521094837.6a2d29c4@notabene.brown>
----- Original Message -----
> From: "NeilBrown" <neilb@suse.de>
> To: "Xiao Ni" <xni@redhat.com>
> Cc: linux-raid@vger.kernel.org
> Sent: Thursday, May 21, 2015 7:48:37 AM
> Subject: Re: raid5 reshape is stuck
>
> On Fri, 15 May 2015 03:00:24 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
>
> > Hi Neil
> >
> > I encounter the problem when I reshape a 4-disks raid5 to raid5. It just
> > can
> > appear with loop devices.
> >
> > The steps are:
> >
> > [root@dhcp-12-158 mdadm-3.3.2]# mdadm -CR /dev/md0 -l5 -n5 /dev/loop[0-4]
> > --assume-clean
> > mdadm: /dev/loop0 appears to be part of a raid array:
> > level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > mdadm: /dev/loop1 appears to be part of a raid array:
> > level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > mdadm: /dev/loop2 appears to be part of a raid array:
> > level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > mdadm: /dev/loop3 appears to be part of a raid array:
> > level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > mdadm: /dev/loop4 appears to be part of a raid array:
> > level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > mdadm: Defaulting to version 1.2 metadata
> > mdadm: array /dev/md0 started.
> > [root@dhcp-12-158 mdadm-3.3.2]# mdadm /dev/md0 -a /dev/loop5
> > mdadm: added /dev/loop5
> > [root@dhcp-12-158 mdadm-3.3.2]# mdadm --grow /dev/md0 --raid-devices 6
> > mdadm: Need to backup 10240K of critical section..
> > [root@dhcp-12-158 mdadm-3.3.2]# cat /proc/mdstat
> > Personalities : [raid6] [raid5] [raid4]
> > md0 : active raid5 loop5[5] loop4[4] loop3[3] loop2[2] loop1[1] loop0[0]
> > 8187904 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/6]
> > [UUUUUU]
> > [>....................] reshape = 0.0% (0/2046976) finish=6396.8min
> > speed=0K/sec
> >
> > unused devices: <none>
> >
> > It because the sync_max is set to 0 when run the command --grow
> >
> > [root@dhcp-12-158 mdadm-3.3.2]# cd /sys/block/md0/md/
> > [root@dhcp-12-158 md]# cat sync_max
> > 0
> >
> > I tried reproduce with normal sata devices. The progress of reshape is
> > no problem. Then
> > I checked the Grow.c. If I use sata devices, in function reshape_array, the
> > return value
> > of set_new_data_offset is 0. But if I used loop devices, it return 1. Then
> > it call the function
> > start_reshape.
>
> set_new_data_offset returns '0' if there is room on the devices to reduce the
> data offset so that the reshape starts writing to unused space on the array.
> This removes the need for a backup file, or the use of a spare device to
> store a temporary backup.
> It returns '1' if there was no room for relocating the data_offset.
>
> So on your sata devices (which are presumably larger than your loop devices)
> there was room. On your loop devices there was not.
>
>
> >
> > In the function start_reshape it set the sync_max to reshape_progress.
> > But in sysfs_read it
> > doesn't read reshape_progress. So it's 0 and the sync_max is set to 0. Why
> > it need to set the
> > sync_max at this? I'm not sure about this.
>
> sync_max is set to 0 so that the reshape does not start until the backup has
> been taken.
> Once the backup is taken, child_monitor() should set sync_max to "max".
>
> Can you check if that is happening?
>
> Thanks,
> NeilBrown
>
>
Thanks very much for the explaining. The problem maybe is fixed. I tried reproduce this with newest
kernel and newest mdadm. Now the problem don't exist. I'll do more tests and give the answer above later.
Best Regards
Xiao
^ permalink raw reply
* Re: [PATCH] need read reshape_groress from sys
From: Xiao Ni @ 2015-05-21 3:35 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150520152042.407313f9@notabene.brown>
----- Original Message -----
> From: "NeilBrown" <neilb@suse.de>
> To: "Xiao Ni" <xni@redhat.com>
> Cc: linux-raid@vger.kernel.org
> Sent: Wednesday, May 20, 2015 1:20:42 PM
> Subject: Re: [PATCH] need read reshape_groress from sys
>
> On Tue, 19 May 2015 22:35:35 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
>
> > Hi all
> >
> > Send the patch again. Because I don't see them at
> > http://www.spinics.net/lists/raid/
> >
> >
> > ----- Forwarded Message -----
> > From: "Xiao Ni" <xni@redhat.com>
> > To: linux-raid@vger.kernel.org
> > Cc: "Xiao Ni" <xni@redhat.com>
> > Sent: Friday, May 15, 2015 3:07:17 PM
> > Subject: [PATCH] need read reshape_groress from sys
> >
> >
>
>
> Please actually explain the purpose of the patch.
> Then re-read it to ensure it makes sense and remove obvious typos.
>
> and don't just remove unrelated blank lines.
> Only include in the patch things that need to be in the patch.
>
>
Hi Neil
I messed the mails, sorry for this. I have sent a mail which subject is "raid5 reshape is stuck" and
two patch mails several days before. But I can't find them so I sent them again yesterday.
And I try to reproduce this with newest kernel and newest mdadm. The problem is fixed already. But I
still have a question which is below the steps I did.
Now I paste the reason here:
I encounter the problem when I reshape a 4-disks raid5 to 5-disks raid5. It just can
appear with loop devices.
The steps are:
[root@dhcp-12-158 mdadm-3.3.2]# mdadm -CR /dev/md0 -l5 -n5 /dev/loop[0-4] --assume-clean
mdadm: /dev/loop0 appears to be part of a raid array:
level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
mdadm: /dev/loop1 appears to be part of a raid array:
level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
mdadm: /dev/loop2 appears to be part of a raid array:
level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
mdadm: /dev/loop3 appears to be part of a raid array:
level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
mdadm: /dev/loop4 appears to be part of a raid array:
level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
[root@dhcp-12-158 mdadm-3.3.2]# mdadm /dev/md0 -a /dev/loop5
mdadm: added /dev/loop5
[root@dhcp-12-158 mdadm-3.3.2]# mdadm --grow /dev/md0 --raid-devices 6
mdadm: Need to backup 10240K of critical section..
[root@dhcp-12-158 mdadm-3.3.2]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 loop5[5] loop4[4] loop3[3] loop2[2] loop1[1] loop0[0]
8187904 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/6] [UUUUUU]
[>....................] reshape = 0.0% (0/2046976) finish=6396.8min speed=0K/sec
unused devices: <none>
It because the sync_max is set to 0 when run the command --grow
[root@dhcp-12-158 mdadm-3.3.2]# cd /sys/block/md0/md/
[root@dhcp-12-158 md]# cat sync_max
0
I tried reproduce with normal sata devices. The progress of reshape is no problem. Then
I checked the Grow.c. If I use sata devices, in function reshape_array, the return value
of set_new_data_offset is 0. But if I used loop devices, it return 1. Then it call the function
start_reshape.
In the function start_reshape it set the sync_max to reshape_progress. But in sysfs_read it
doesn't read reshape_progress. So it's 0 and the sync_max is set to 0. Why it need to set the
sync_max at this? I'm not sure about this.
>
>
> > Signed-off-by: Xiao Ni <xni@redhat.com>
> > ---
> > Grow.c | 18 ++++++++++++------
> > mdadm.h | 1 +
> > sysfs.c | 15 +++++++++++++++
> > 3 files changed, 28 insertions(+), 6 deletions(-)
> >
> > diff --git a/Grow.c b/Grow.c
> > index 568e399..44ee8a7 100644
> > --- a/Grow.c
> > +++ b/Grow.c
> > @@ -710,11 +710,17 @@ int start_reshape(struct mdinfo *sra, int
> > already_running,
> > err = sysfs_set_num(sra, NULL, "suspend_hi", sra->reshape_progress);
> > err = err ?: sysfs_set_num(sra, NULL, "suspend_lo",
> > sra->reshape_progress);
> > - if (before_data_disks <= data_disks)
> > - sync_max_to_set = sra->reshape_progress / data_disks;
> > - else
> > - sync_max_to_set = (sra->component_size * data_disks
> > - - sra->reshape_progress) / data_disks;
> > +
> > + if (sra->reshape_progress == UINT64_MAX) {
> > + err = err ?: sysfs_set_str(sra, NULL, "sync_max", "max");
> > + } else {
> > + if (before_data_disks <= data_disks)
> > + sync_max_to_set = sra->reshape_progress / data_disks;
> > + else
> > + sync_max_to_set = (sra->component_size * 2 * data_disks
> > + - sra->reshape_progress) / data_disks;
> > + }
> > +
> > if (!already_running)
> > sysfs_set_num(sra, NULL, "sync_min", sync_max_to_set);
> > err = err ?: sysfs_set_num(sra, NULL, "sync_max", sync_max_to_set);
> > @@ -3075,7 +3081,7 @@ static int reshape_array(char *container, int fd,
> > char *devname,
> > }
> > sra = sysfs_read(fd, NULL,
> > GET_COMPONENT|GET_DEVS|GET_OFFSET|GET_STATE|GET_CHUNK|
> > - GET_CACHE);
> > + GET_CACHE|GET_RESHAPE_PROGRESS);
> > if (!sra) {
> > pr_err("%s: Cannot get array details from sysfs\n",
> > devname);
> > diff --git a/mdadm.h b/mdadm.h
> > index 141f963..6fb17e1 100644
> > --- a/mdadm.h
> > +++ b/mdadm.h
> > @@ -526,6 +526,7 @@ enum sysfs_read_flags {
> > GET_DEGRADED = (1 << 8),
> > GET_SAFEMODE = (1 << 9),
> > GET_BITMAP_LOCATION = (1 << 10),
> > + GET_RESHAPE_PROGRESS = (1 << 11),
> >
> > GET_DEVS = (1 << 20), /* gets role, major, minor */
> > GET_OFFSET = (1 << 21),
> > diff --git a/sysfs.c b/sysfs.c
> > index 18f3df9..09b0c93 100644
> > --- a/sysfs.c
> > +++ b/sysfs.c
> > @@ -26,6 +26,7 @@
> > #include "mdadm.h"
> > #include <dirent.h>
> > #include <ctype.h>
> > +#include <stdint.h>
> >
> > int load_sys(char *path, char *buf)
> > {
> > @@ -210,6 +211,19 @@ struct mdinfo *sysfs_read(int fd, char *devnm,
> > unsigned long options)
> > msec = (msec * 1000) / scale;
> > sra->safe_mode_delay = msec;
> > }
> > +
> > + if (options & GET_RESHAPE_PROGRESS) {
> > +
> > + strcpy(base, "reshape_progress");
> > + if (load_sys(fname, buf))
> > + goto abort;
> > +
> > + if (strncmp(buf, "max", 3) == 0)
> > + sra->reshape_progress = UINT64_MAX;
> > + else
> > + sra->reshape_progress = strtol(buf, NULL, 10);
> > + }
> > +
> > if (options & GET_BITMAP_LOCATION) {
> > strcpy(base, "bitmap/location");
> > if (load_sys(fname, buf))
> > @@ -224,6 +238,7 @@ struct mdinfo *sysfs_read(int fd, char *devnm, unsigned
> > long options)
> > goto abort;
> > }
> >
> > +
> > if (! (options & GET_DEVS))
> > return sra;
> >
>
>
^ permalink raw reply
* Re: Breaks LSI RAID on C600 chipset
From: NeilBrown @ 2015-05-21 0:29 UTC (permalink / raw)
To: David F.; +Cc: John Stoffel, linux-raid@vger.kernel.org
In-Reply-To: <CAGRSmLsRr6dHKjOuZB5oy+BALKatJ96P3PMGB9r+oVcft75Z4A@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2755 bytes --]
On Mon, 18 May 2015 10:58:19 -0700 "David F." <df7729@gmail.com> wrote:
> Do you think the latest version at
> http://git.neil.brown.name/?p=mdadm.git;a=summary is stable enough to
> use? It definitely fixes the major issue (losing the raid config)
> from the prior version.
It should be OK. I'll probably make a 3.3.3 in a little while to make it
official.
NeilBrown
>
> On Wed, May 13, 2015 at 7:26 AM, David F. <df7729@gmail.com> wrote:
> > Thanks.
> >
> > 1) The update did fix the broken RAID. I can reboot and the RAID
> > mirror still exists after reboots.
> >
> > 2) I do want to report that on the first test, the reboot was clean.
> > On the second reboot there was a warning message about the BIOS
> > detecting "Inconsistent Timestamps" on one of the drives and something
> > about which CONFIG the BIOS would use. I didn't have time to get it
> > all jotted down. Whatever it used was still okay since the RAID
> > mirror was still there after that message.
> >
> >
> > On Tue, May 12, 2015 at 5:35 PM, NeilBrown <neilb@suse.de> wrote:
> >> On Mon, 11 May 2015 09:31:26 -0700 "David F." <df7729@gmail.com> wrote:
> >>
> >>> any progress or more information needed on this?
> >>>
> >>> On Mon, Apr 20, 2015 at 10:39 PM, David F. <df7729@gmail.com> wrote:
> >>> > If you need access to the system, a network kvm is available - or if
> >>> > you have a C600 based system, you should see the same problem there.
> >>> >
> >>> > On Fri, Apr 10, 2015 at 1:07 PM, David F. <df7729@gmail.com> wrote:
> >>> >> Okay, and to confirm, this is happening on other C600 based systems.
> >>> >> Boot to Linux with MDADM raid support and the raid is gone after
> >>> >> reboot.
> >>> >>
> >>> >> On Thu, Apr 9, 2015 at 5:50 PM, NeilBrown <neilb@suse.de> wrote:
> >>> >>> On Thu, 9 Apr 2015 14:13:05 -0700 "David F." <df7729@gmail.com> wrote:
> >>> >>>
> >>> >>>> Hello,
> >>> >>>>
> >>> >>>> I built a new system, installed new drives (no partitioning, just raw
> >>> >>>> new drives), configured RAID 1, boot to linux, created reports
> >>> >>>> attached. Rebooted and the system doesn't see any configured raid
> >>> >>>> drives (just the two raw drives).
> >>> >>>
> >>> >>> Thanks. I might take a look, but I would greatly prefer it if you kept the
> >>> >>> linux-raid list on the Cc.....
> >>> >>>
> >>> >>> NeilBrown
> >>> >>>
> >>
> >> I must confess that I haven't even looked at it.
> >> But I just saw and email on linux-raid from Martin Wilck:
> >>
> >> Subject: [PATCH] DDF: _write_super_to_disk: fix anchor header type
> >>
> >> which very likely fixes your problem. I've applied it and pushed out to
> >> git://neil.brown.name/mdadm/
> >>
> >> NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Data corruption after resizing partition, when using bitmaps
From: NeilBrown @ 2015-05-21 0:24 UTC (permalink / raw)
To: Jim Paris; +Cc: linux-raid
In-Reply-To: <20150520063149.GA8346@psychosis.jim.sh>
[-- Attachment #1: Type: text/plain, Size: 2828 bytes --]
On Wed, 20 May 2015 02:31:50 -0400 Jim Paris <jim@jtan.com> wrote:
> NeilBrown wrote:
> > On Tue, 19 May 2015 10:12:40 -0400 Jim Paris <jim@jtan.com> wrote:
> >
> > > I had a raid1 mirror consisting of big partitions on two disks.
> > > The first disk was 2TB, partitioned like this:
> > >
> > > [--sda1(128M)--][-------sda2(~2T)--------------]
> > >
> > > The second disk was 3TB, partitioned like this:
> > >
> > > [--sdb1(128M)--][-------sdb2(~3T)------------------------------------]
> > >
> > > sda2 and sdb2 were part of the array, which was only ~2TB in size due
> > > to the smaller disk.
> > >
> > > I realized that I needed to add a BIOS boot partition to the 3TB disk,
> > > so I removed sdb2 from the array, and repartitioned sdb like this:
> > >
> > > [--sdb1(128M)--][--sdb2(1M)--][-------sdb3(~3T)----------------------]
> > >
> > > Then I added sdb3 to the array. And lost all my data. :(
> > >
> > > What happened was that the last sector of the big partition did not
> > > change location. So the metadata (0.90) at the end was still present.
> >
> > This is one of the big reasons why 1.x was invented.
> >
> > > Adding sdb3 to the array was considered a "re-add" because the UUID
> > > and array sizes still matched the array, even though the partition
> > > itself shrank. And the resync was thus guided by an out-of-date
> > > bitmap, which caused very little data to actually be written to sdb3,
> > > so half the reads from the array started returning junk. Once the
> > > filesystem got involved, the result was rapid corruption.
> > >
> > > If I had not been using write-intent bitmaps, everything would have
> > > worked fine. I only recently started using bitmaps, and never had any
> > > problems with adjusting partitions like this before that.
> > >
> > > Perhaps mdadm can be more careful here -- for example, maybe checking
> > > the actual device size and not just the "used dev size" when
> > > determining whether to trust the bitmap.
> >
> > It is perfectly acceptable to have the various devices in an array of
> > different sizes. Unfortunately I don't think there is anything that mdadm
> > can usefully do here.
> >
> > Thanks for the report anyway,
> > NeilBrown
>
> Hi Neil,
>
> Can we add u64 device_size to bitmap_super_t, and ensure that it
> matches the actual current device size before trusting the bitmap?
Well .... we could, but the bitmap_super is currently the same on all
devices. This would make it different.
And if we a going to change the metadata, why not just convert from 0.90 to
1.0?
mdadm --stop /dev/mdXX
mdadm --assemble /dev/mdXX --update=metadata /dev/...list-of-devices....
You might need to remove the bitmap first, and add it back afterwards.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: raid5 reshape is stuck
From: NeilBrown @ 2015-05-20 23:48 UTC (permalink / raw)
To: Xiao Ni; +Cc: linux-raid
In-Reply-To: <2043891461.15360424.1431673224036.JavaMail.zimbra@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 3391 bytes --]
On Fri, 15 May 2015 03:00:24 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
> Hi Neil
>
> I encounter the problem when I reshape a 4-disks raid5 to raid5. It just can
> appear with loop devices.
>
> The steps are:
>
> [root@dhcp-12-158 mdadm-3.3.2]# mdadm -CR /dev/md0 -l5 -n5 /dev/loop[0-4] --assume-clean
> mdadm: /dev/loop0 appears to be part of a raid array:
> level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> mdadm: /dev/loop1 appears to be part of a raid array:
> level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> mdadm: /dev/loop2 appears to be part of a raid array:
> level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> mdadm: /dev/loop3 appears to be part of a raid array:
> level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> mdadm: /dev/loop4 appears to be part of a raid array:
> level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> mdadm: Defaulting to version 1.2 metadata
> mdadm: array /dev/md0 started.
> [root@dhcp-12-158 mdadm-3.3.2]# mdadm /dev/md0 -a /dev/loop5
> mdadm: added /dev/loop5
> [root@dhcp-12-158 mdadm-3.3.2]# mdadm --grow /dev/md0 --raid-devices 6
> mdadm: Need to backup 10240K of critical section..
> [root@dhcp-12-158 mdadm-3.3.2]# cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4]
> md0 : active raid5 loop5[5] loop4[4] loop3[3] loop2[2] loop1[1] loop0[0]
> 8187904 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/6] [UUUUUU]
> [>....................] reshape = 0.0% (0/2046976) finish=6396.8min speed=0K/sec
>
> unused devices: <none>
>
> It because the sync_max is set to 0 when run the command --grow
>
> [root@dhcp-12-158 mdadm-3.3.2]# cd /sys/block/md0/md/
> [root@dhcp-12-158 md]# cat sync_max
> 0
>
> I tried reproduce with normal sata devices. The progress of reshape is no problem. Then
> I checked the Grow.c. If I use sata devices, in function reshape_array, the return value
> of set_new_data_offset is 0. But if I used loop devices, it return 1. Then it call the function
> start_reshape.
set_new_data_offset returns '0' if there is room on the devices to reduce the
data offset so that the reshape starts writing to unused space on the array.
This removes the need for a backup file, or the use of a spare device to
store a temporary backup.
It returns '1' if there was no room for relocating the data_offset.
So on your sata devices (which are presumably larger than your loop devices)
there was room. On your loop devices there was not.
>
> In the function start_reshape it set the sync_max to reshape_progress. But in sysfs_read it
> doesn't read reshape_progress. So it's 0 and the sync_max is set to 0. Why it need to set the
> sync_max at this? I'm not sure about this.
sync_max is set to 0 so that the reshape does not start until the backup has
been taken.
Once the backup is taken, child_monitor() should set sync_max to "max".
Can you check if that is happening?
Thanks,
NeilBrown
>
> I tried to fix this but I'm not sure whether it's the right way. I'll send the patches in
> other mails.
>
> Best Regards
> Xiao
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Filesystem corruption MD (imsm) Raid0 via 2 SSD's + discard
From: NeilBrown @ 2015-05-20 23:08 UTC (permalink / raw)
To: Roman Mamedov; +Cc: Holger Kiehl, linux-kernel, linux-raid
In-Reply-To: <20150521013213.536deed0@natsu>
[-- Attachment #1: Type: text/plain, Size: 1207 bytes --]
On Thu, 21 May 2015 01:32:13 +0500 Roman Mamedov <rm@romanrm.net> wrote:
> On Wed, 20 May 2015 20:12:31 +0000 (UTC)
> Holger Kiehl <Holger.Kiehl@dwd.de> wrote:
>
> > The kernel I was running when I discovered the
> > problem was 4.0.2 from kernel.org. However, after reinstalling from DVD
> > I updated to Fedora's lattest kernel, which was 3.19.? (I do not remember
> > the last numbers). So that kernel seems also effected, but I assume it
> > contains many 'fixes' from 4.0.x. As filesystem I use ext4, distribution
> > is Fedora 21 and hardware is: Xeon E3-1275, 16GB ECC Ram.
> >
> > My system seems to be now running stable for some days with kernel.org
> > kernel 4.0.3 and with discard DISABLED. But I am still unsure what could
> > be the real cause.
>
> It is a bug in the 4.0.2 kernel, fixed in 4.0.3.
>
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=785672
> https://bbs.archlinux.org/viewtopic.php?id=197400
> https://kernel.googlesource.com/pub/scm/linux/kernel/git/stable/linux-stable/+/d2dc317d564a46dfc683978a2e5a4f91434e9711
>
>
I suspect that is a different bug.
I think this one is
https://bugzilla.kernel.org/show_bug.cgi?id=98501
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Massive data corruption on replace + fail + rebuild
From: James J @ 2015-05-20 20:36 UTC (permalink / raw)
To: linux-raid
Hello all,
I wrote some days ago to the ML with subject "MD RAID hot-replace wants
to rewrite to the source! (and fails, and kicks)"
The problem went much worse than that and ended with massive data
corruption on the replacement drive sdm.
In this report I will use the same letters as in previous email: sdl for
the failing drive, sdm for the replacement drive.
This report is for raid5 on kernel 3.4.34 .
Bad blocks list is not enabled.
Bitmap is enabled
See previous post for details, mdstat and dmesg log.
It seems that the following series of events deeply corrupts the
replacement drive:
1) drive sdl is flaky, so the user initiates replacement process
(want_replacement) to the spare drive sdm
2) the disk sdl has read errors on some sectors. MD performs reconstruct
read and then rewrite for those sectors. Currently MD wants to rewrite
the source drive sdl, instead of just to the replacement drive sdm which
I would much prefer.
3) sdl unfortunately is too flaky to receive rewrites, so it fails on
the rewrites and is kicked by MD. The array is now degraded
4) At this point, MD apparently continues the rebuild process
transforming the replacement into a full rebuild, but continuing from
the point where sdl was kicked, and not from the start. This seems
smart, however I guess there is a bug in doing this, maybe an off-by-N
error. In the previous post you can see the dmesg line "[865031.586650]
md: resuming recovery of md54 from checkpoint."
At the end of the rebuild, when MD starts using sdm as a member drive,
an enormous amount of errors appear on the filesystems located on that
array.
In fact, I performed a check afterwards, and this was the mismatch_cnt:
root@server:/sys/block/md54/md# cat mismatch_cnt
5296438776
This is on a 5x3TB array so about 90.4% of it has mismatches, if my math
is correct.
The drive sdl was kicked at about 1% of the replacement process, so this
90.4% would not match, should have been 99%, but considering that many
stripes could be zeroes, an 8.5% could be parities on zeroes which match
just by chance.
So I suppose there is a problem in the handover between the replacement
and the rebuild. I would bet on an off-by-N problem, i.e. a shifting of
the data. Maybe the sources start a reconstruct-read from the beginning
of the array while the destination continues writing from the point of
the handover, or vice versa.
Currently I have "solved" the problem already, by artificially failing
the drive sdm, and introducing another disk as spare to perform a clean
rebuild from scratch. After failing sdm and dropping the caches, before
inserting the new spare, the filesystems were readable again, so I was
optimistic, and in fact at the end of the clean rebuild this appears to
have recovered our data and mismatch_cnt is now zero. However I was
lucky to have immediately understood what happened, otherwise we would
have probably lost all our data, so please look into this.
Thanks for your work
PS: I would appreciate if you can also make MD not rewrite to the source
during replacement :-)
JJ
^ permalink raw reply
* Re: Filesystem corruption MD (imsm) Raid0 via 2 SSD's + discard
From: Roman Mamedov @ 2015-05-20 20:32 UTC (permalink / raw)
To: Holger Kiehl; +Cc: linux-kernel, linux-raid
In-Reply-To: <alpine.LRH.2.11.1505201918570.4751@praktifix.dwd.de>
[-- Attachment #1: Type: text/plain, Size: 991 bytes --]
On Wed, 20 May 2015 20:12:31 +0000 (UTC)
Holger Kiehl <Holger.Kiehl@dwd.de> wrote:
> The kernel I was running when I discovered the
> problem was 4.0.2 from kernel.org. However, after reinstalling from DVD
> I updated to Fedora's lattest kernel, which was 3.19.? (I do not remember
> the last numbers). So that kernel seems also effected, but I assume it
> contains many 'fixes' from 4.0.x. As filesystem I use ext4, distribution
> is Fedora 21 and hardware is: Xeon E3-1275, 16GB ECC Ram.
>
> My system seems to be now running stable for some days with kernel.org
> kernel 4.0.3 and with discard DISABLED. But I am still unsure what could
> be the real cause.
It is a bug in the 4.0.2 kernel, fixed in 4.0.3.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=785672
https://bbs.archlinux.org/viewtopic.php?id=197400
https://kernel.googlesource.com/pub/scm/linux/kernel/git/stable/linux-stable/+/d2dc317d564a46dfc683978a2e5a4f91434e9711
--
With respect,
Roman
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply
* Filesystem corruption MD (imsm) Raid0 via 2 SSD's + discard
From: Holger Kiehl @ 2015-05-20 20:12 UTC (permalink / raw)
To: linux-kernel, linux-raid
Hello,
I had a terrible weekend recovering my home system. Always when files
where deleted some data got corrupted. At first I did not notice it,
but when I rebooted the system would not come up again, systemd crashed
with SIGSEGV and that was it. Booting from an USB stick I saw that
some glibc lib had a different size from that in the original RPM. So
all I did reinstalled that lib from USB stick and everything was fine
after rebooting from Raid 0. But I then wanted to make sure that
no other files where corrupted so I checked and found more. So again I
reinstalled those RPM's and rebooted. To my big surprise the system was
again broken and failed to boot. I again tried to recover my system
from USB stick, but this time did not manage to recover the system. So
decided to reinstall the system completely from DVD. Everything looked good
until that moment when I had activated the discard option in /etc/fstab.
After doing some more work (adding and removing things) I rebooted and
again the system failed to boot. Booting from the USB stick I saw that
the /etc/fstab was all filled with NULL's. This gave me the clue that
there must be some problem with discard (trim). My system is using
a software raid 0 IMSM (intel 'fake' raid) on two Samsung SSD 840 pro.
A window system on the same disks (that is why I am using IMSM raid)
was not effected by this problem. I have checked the ram with memtest86
and everything is ok. The kernel I was running when I discovered the
problem was 4.0.2 from kernel.org. However, after reinstalling from DVD
I updated to Fedora's lattest kernel, which was 3.19.? (I do not remember
the last numbers). So that kernel seems also effected, but I assume it
contains many 'fixes' from 4.0.x. As filesystem I use ext4, distribution
is Fedora 21 and hardware is: Xeon E3-1275, 16GB ECC Ram.
My system seems to be now running stable for some days with kernel.org
kernel 4.0.3 and with discard DISABLED. But I am still unsure what could
be the real cause.
Regards,
Holger
^ permalink raw reply
* Re: Replacing a failed disk "in advance"
From: Jan Kasprzak @ 2015-05-20 15:15 UTC (permalink / raw)
To: Caspar Smit; +Cc: linux-raid
In-Reply-To: <CADvHm=aB2MDFxdR5Q3JBL5_WbD7oc2d99kFe4Jjbun1Lw667MQ@mail.gmail.com>
Caspar Smit wrote:
: Op 20 mei 2015 16:51 schreef "Jan Kasprzak" <kas@fi.muni.cz>:
: >
: > OK, this seems to be a way to go.
: >
: > Unfortunately, the system in question is too old,
: > and its mdadm does not know about the --replace option, according to
: > mdadm --manage --help. So I will look for a newer mdadm and hope the old
: > kernel of that system supports the mdadm --replace.
: >
:
: You'll need kernel version 3.3+ to support advance replacement.
OK, I have grabbed the Fedora 22 mdadm package, rebuilt it,
and apparently mdadm --manage /dev/mdXX --replace /dev/sdOLD1 --with /dev/sdNEW1
works as expected. Thanks!
-Yenya.
--
| Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> |
| New GPG 4096R/A45477D5 -- see http://www.fi.muni.cz/~kas/pgp-rollover.txt |
| http://www.fi.muni.cz/~kas/ Journal: http://www.fi.muni.cz/~kas/blog/ |
Smart data structures and dumb code works a lot better
than the other way around. --Eric S. Raymond
^ permalink raw reply
* Re: Replacing a failed disk "in advance"
From: Caspar Smit @ 2015-05-20 15:12 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <20150520145126.GU24672@fi.muni.cz>
Op 20 mei 2015 16:51 schreef "Jan Kasprzak" <kas@fi.muni.cz>:
>
> Hi Ladislav,
>
> Ladislav Mate wrote:
> : > I have a RAID-5 volume of 8 physical disks. One of these disks failed the SMART
> : > self-test with an unreadable block error. Unfortunately I have discovered
> : > that there is _another_ bad block on another disk. It is a different block,
> : > so the RAID-5 volume as a whole is still working. But as a whole, the
> : > RAID-5 volume has at least two unreadable sectors on two different disks.
> : >
> : > What is the best way to replace these two failing disks one by one without
> : > the loss of data? I cannot mdadm --fail one of them, because the
> : > subsequent rebuild on a new disk would fail on reading the other bad block.
> : >
> : > I would like to add the ninth drive to the RAID-5 volume, and put a replica
> : > of one of the failing drives to it. Then remove the just-replicated drive,
> : > and do the same with the other failing drive.
> : When you take a look in mdadm man page and search for --replace you'll find what you are looking for.
> :
> : --replace
> : Mark listed devices as requiring replacement. As soon as a spare is available, it will be rebuilt and will replace
> : the marked device. This is similar to marking a device as faulty, but the device remains in service during the recov-
> : ery process to increase resilience against multiple failures. When the replacement process finishes, the replaced
> : device will be marked as faulty.
> :
> : --with This can follow a list of --replace devices. The devices listed after --with will be preferentially used to replace
> : the devices listed after --replace. These device must already be spare devices in the array.
>
> OK, this seems to be a way to go.
>
> Unfortunately, the system in question is too old,
> and its mdadm does not know about the --replace option, according to
> mdadm --manage --help. So I will look for a newer mdadm and hope the old
> kernel of that system supports the mdadm --replace.
>
You'll need kernel version 3.3+ to support advance replacement.
Kind regards,
Caspar
> Thanks!
>
> -Yenya
>
> --
> | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> |
> | New GPG 4096R/A45477D5 -- see http://www.fi.muni.cz/~kas/pgp-rollover.txt |
> | http://www.fi.muni.cz/~kas/ Journal: http://www.fi.muni.cz/~kas/blog/ |
> Smart data structures and dumb code works a lot better
> than the other way around. --Eric S. Raymond
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Replacing a failed disk "in advance"
From: Jan Kasprzak @ 2015-05-20 14:51 UTC (permalink / raw)
To: Ladislav Mate; +Cc: linux-raid
In-Reply-To: <20150520144001.GA2826@dev>
Hi Ladislav,
Ladislav Mate wrote:
: > I have a RAID-5 volume of 8 physical disks. One of these disks failed the SMART
: > self-test with an unreadable block error. Unfortunately I have discovered
: > that there is _another_ bad block on another disk. It is a different block,
: > so the RAID-5 volume as a whole is still working. But as a whole, the
: > RAID-5 volume has at least two unreadable sectors on two different disks.
: >
: > What is the best way to replace these two failing disks one by one without
: > the loss of data? I cannot mdadm --fail one of them, because the
: > subsequent rebuild on a new disk would fail on reading the other bad block.
: >
: > I would like to add the ninth drive to the RAID-5 volume, and put a replica
: > of one of the failing drives to it. Then remove the just-replicated drive,
: > and do the same with the other failing drive.
: When you take a look in mdadm man page and search for --replace you'll find what you are looking for.
:
: --replace
: Mark listed devices as requiring replacement. As soon as a spare is available, it will be rebuilt and will replace
: the marked device. This is similar to marking a device as faulty, but the device remains in service during the recov-
: ery process to increase resilience against multiple failures. When the replacement process finishes, the replaced
: device will be marked as faulty.
:
: --with This can follow a list of --replace devices. The devices listed after --with will be preferentially used to replace
: the devices listed after --replace. These device must already be spare devices in the array.
OK, this seems to be a way to go.
Unfortunately, the system in question is too old,
and its mdadm does not know about the --replace option, according to
mdadm --manage --help. So I will look for a newer mdadm and hope the old
kernel of that system supports the mdadm --replace.
Thanks!
-Yenya
--
| Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> |
| New GPG 4096R/A45477D5 -- see http://www.fi.muni.cz/~kas/pgp-rollover.txt |
| http://www.fi.muni.cz/~kas/ Journal: http://www.fi.muni.cz/~kas/blog/ |
Smart data structures and dumb code works a lot better
than the other way around. --Eric S. Raymond
^ permalink raw reply
* Re: Replacing a failed disk "in advance"
From: Ladislav Mate @ 2015-05-20 14:40 UTC (permalink / raw)
To: Jan Kasprzak; +Cc: linux-raid
In-Reply-To: <20150520142049.GT24672@fi.muni.cz>
On Wed, May 20, 2015 at 04:20:49PM +0200, Jan Kasprzak wrote:
> Hello,
Hi Jan,
>
> I have a RAID-5 volume of 8 physical disks. One of these disks failed the SMART
> self-test with an unreadable block error. Unfortunately I have discovered
> that there is _another_ bad block on another disk. It is a different block,
> so the RAID-5 volume as a whole is still working. But as a whole, the
> RAID-5 volume has at least two unreadable sectors on two different disks.
>
> What is the best way to replace these two failing disks one by one without
> the loss of data? I cannot mdadm --fail one of them, because the
> subsequent rebuild on a new disk would fail on reading the other bad block.
>
> I would like to add the ninth drive to the RAID-5 volume, and put a replica
> of one of the failing drives to it. Then remove the just-replicated drive,
> and do the same with the other failing drive.
When you take a look in mdadm man page and search for --replace you'll find what you are looking for.
--replace
Mark listed devices as requiring replacement. As soon as a spare is available, it will be rebuilt and will replace
the marked device. This is similar to marking a device as faulty, but the device remains in service during the recov-
ery process to increase resilience against multiple failures. When the replacement process finishes, the replaced
device will be marked as faulty.
--with This can follow a list of --replace devices. The devices listed after --with will be preferentially used to replace
the devices listed after --replace. These device must already be spare devices in the array.
HTH,
/lm
>
> Thanks,
>
> -Yenya
>
> --
> | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> |
> | New GPG 4096R/A45477D5 -- see http://www.fi.muni.cz/~kas/pgp-rollover.txt |
> | http://www.fi.muni.cz/~kas/ Journal: http://www.fi.muni.cz/~kas/blog/ |
> Smart data structures and dumb code works a lot better
> than the other way around. --Eric S. Raymond
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Replacing a failed disk "in advance"
From: Jan Kasprzak @ 2015-05-20 14:20 UTC (permalink / raw)
To: linux-raid
Hello,
I have a RAID-5 volume of 8 physical disks. One of these disks failed the SMART
self-test with an unreadable block error. Unfortunately I have discovered
that there is _another_ bad block on another disk. It is a different block,
so the RAID-5 volume as a whole is still working. But as a whole, the
RAID-5 volume has at least two unreadable sectors on two different disks.
What is the best way to replace these two failing disks one by one without
the loss of data? I cannot mdadm --fail one of them, because the
subsequent rebuild on a new disk would fail on reading the other bad block.
I would like to add the ninth drive to the RAID-5 volume, and put a replica
of one of the failing drives to it. Then remove the just-replicated drive,
and do the same with the other failing drive.
Thanks,
-Yenya
--
| Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> |
| New GPG 4096R/A45477D5 -- see http://www.fi.muni.cz/~kas/pgp-rollover.txt |
| http://www.fi.muni.cz/~kas/ Journal: http://www.fi.muni.cz/~kas/blog/ |
Smart data structures and dumb code works a lot better
than the other way around. --Eric S. Raymond
^ permalink raw reply
* Re: Data corruption after resizing partition, when using bitmaps
From: Jim Paris @ 2015-05-20 6:31 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150520153104.7ac99de1@notabene.brown>
NeilBrown wrote:
> On Tue, 19 May 2015 10:12:40 -0400 Jim Paris <jim@jtan.com> wrote:
>
> > I had a raid1 mirror consisting of big partitions on two disks.
> > The first disk was 2TB, partitioned like this:
> >
> > [--sda1(128M)--][-------sda2(~2T)--------------]
> >
> > The second disk was 3TB, partitioned like this:
> >
> > [--sdb1(128M)--][-------sdb2(~3T)------------------------------------]
> >
> > sda2 and sdb2 were part of the array, which was only ~2TB in size due
> > to the smaller disk.
> >
> > I realized that I needed to add a BIOS boot partition to the 3TB disk,
> > so I removed sdb2 from the array, and repartitioned sdb like this:
> >
> > [--sdb1(128M)--][--sdb2(1M)--][-------sdb3(~3T)----------------------]
> >
> > Then I added sdb3 to the array. And lost all my data. :(
> >
> > What happened was that the last sector of the big partition did not
> > change location. So the metadata (0.90) at the end was still present.
>
> This is one of the big reasons why 1.x was invented.
>
> > Adding sdb3 to the array was considered a "re-add" because the UUID
> > and array sizes still matched the array, even though the partition
> > itself shrank. And the resync was thus guided by an out-of-date
> > bitmap, which caused very little data to actually be written to sdb3,
> > so half the reads from the array started returning junk. Once the
> > filesystem got involved, the result was rapid corruption.
> >
> > If I had not been using write-intent bitmaps, everything would have
> > worked fine. I only recently started using bitmaps, and never had any
> > problems with adjusting partitions like this before that.
> >
> > Perhaps mdadm can be more careful here -- for example, maybe checking
> > the actual device size and not just the "used dev size" when
> > determining whether to trust the bitmap.
>
> It is perfectly acceptable to have the various devices in an array of
> different sizes. Unfortunately I don't think there is anything that mdadm
> can usefully do here.
>
> Thanks for the report anyway,
> NeilBrown
Hi Neil,
Can we add u64 device_size to bitmap_super_t, and ensure that it
matches the actual current device size before trusting the bitmap?
Jim
>
>
> >
> > I wrote a script (attached) to recreate what happened, using some loop
> > devices. It works fine if BITMAP=none, and fails with BITMAP=internal.
> >
> > Jim
>
^ permalink raw reply
* Re: [PATCH] raid5: fix broken async operation chain
From: NeilBrown @ 2015-05-20 6:14 UTC (permalink / raw)
To: Shaohua Li; +Cc: linux-raid, maxime.ripard
In-Reply-To: <abe8bcda859c8d2a11544a273c117c190fe736a4.1431464773.git.shli@kernel.org>
[-- Attachment #1: Type: text/plain, Size: 1202 bytes --]
On Wed, 13 May 2015 09:30:08 -0700 Shaohua Li <shli@kernel.org> wrote:
> ops_run_reconstruct6() doesn't correctly chain asyn operations. The tx returned
> by async_gen_syndrome should be added as the dependent tx of next stripe.
>
> The issue is introduced by commit 59fc630b8b5f9f21c8ce3ba153341c107dce1b0c
> RAID5: batch adjacent full stripe write
>
> Reported-and-tested-by: Maxime Ripard <maxime.ripard@free-electrons.com>
> Signed-off-by: Shaohua Li <shli@fb.com>
> ---
> drivers/md/raid5.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> index 1ba97fd..b9f2b9c 100644
> --- a/drivers/md/raid5.c
> +++ b/drivers/md/raid5.c
> @@ -1822,7 +1822,7 @@ ops_run_reconstruct6(struct stripe_head *sh, struct raid5_percpu *percpu,
> } else
> init_async_submit(&submit, 0, tx, NULL, NULL,
> to_addr_conv(sh, percpu, j));
> - async_gen_syndrome(blocks, 0, count+2, STRIPE_SIZE, &submit);
> + tx = async_gen_syndrome(blocks, 0, count+2, STRIPE_SIZE, &submit);
> if (!last_stripe) {
> j++;
> sh = list_first_entry(&sh->batch_list, struct stripe_head,
applied, thanks.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [PATCH 7/7] md/raid5: fix handling of degraded stripes in batches.
From: Shaohua Li @ 2015-05-20 5:56 UTC (permalink / raw)
To: NeilBrown
Cc: linux-raid-u79uwXL29TY76Z2rM5mHXA,
linux-nfs-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20150513105604.55ac700a-wvvUuzkyo1EYVZTmpyfIwg@public.gmane.org>
On Wed, May 13, 2015 at 10:56:04AM +1000, NeilBrown wrote:
> On Fri, 8 May 2015 12:12:23 -0700 Shaohua Li <shli-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>
> > On Fri, May 08, 2015 at 06:56:12PM +1000, NeilBrown wrote:
> > > There is no need for special handling of stripe-batches when the array
> > > is degraded.
> > >
> > > There may be if there is a failure in the batch, but STRIPE_DEGRADED
> > > does not imply an error.
> > >
> > > So don't set STRIPE_BATCH_ERR in ops_run_io just because the array is
> > > degraded.
> > > This actually causes a bug: the STRIPE_DEGRADED flag gets cleared in
> > > check_break_stripe_batch_list() and so the bitmap bit gets cleared
> > > when it shouldn't.
> > >
> > > So in check_break_stripe_batch_list(), split the batch up completely -
> > > again STRIPE_DEGRADED isn't meaningful.
> > >
> > > Also don't set STRIPE_BATCH_ERR when there is a write error to a
> > > replacement device. This simply removes the replacement device and
> > > requires no extra handling.
> > >
> > > Signed-off-by: NeilBrown <neilb-l3A5Bk7waGM@public.gmane.org>
> > > ---
> > > drivers/md/raid5.c | 17 +++--------------
> > > 1 file changed, 3 insertions(+), 14 deletions(-)
> > >
> > > diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> > > index 3873eaa6fa2e..1ba97fdc6df1 100644
> > > --- a/drivers/md/raid5.c
> > > +++ b/drivers/md/raid5.c
> > > @@ -1078,9 +1078,6 @@ again:
> > > pr_debug("skip op %ld on disc %d for sector %llu\n",
> > > bi->bi_rw, i, (unsigned long long)sh->sector);
> > > clear_bit(R5_LOCKED, &sh->dev[i].flags);
> > > - if (sh->batch_head)
> > > - set_bit(STRIPE_BATCH_ERR,
> > > - &sh->batch_head->state);
> > > set_bit(STRIPE_HANDLE, &sh->state);
> > > }
> >
> > Patches look good to me. I had a question here. Is it possible some stripes in
> > a batch become degraded here but some not? Seems possible, then the batch
> > should be splitted too.
>
> Why?
>
> I don't really understand the purpose of splitting up the batch.
> The only possible error handling on a full-stripe write is:
> - fail a device, or
> - record a bad-block.
>
> The first case affects all stripes in a batch equally so there is no need to
> split it up.
> The second case it is probably best to record the bad blocks while iterating
> through the batch in handle_stripe_clean_event().
>
> What exactly do you expect to happen after the stripes in a batch after they
> have been split up?
My original concern is a device failure can causes some stripes fail but some
not, eg, get rdev returns NULL in ops_run_io for some stripes but not all of a
batch. There is no any locking, so seems possible. But you are right, the
stripes without error in ops_run_io will get an IO error eventually, the whole
batch stripes are still in the same state. So my concern is invalid, but I
forgot to reply the email, sorry.
The batch split is to handle IO error, eg, record bad-block and so on. I'm not
confident to change existing code to handle the error case, so I feel spliting
it and handling the stripe in normal way is the best thing to do. There
certainly might be better way. Again the device failure case should be ignored,
which I didn't realize originally.
BTW, can you apply the fix reported by Maxime, which is introduced by the batch patch.
http://marc.info/?l=linux-raid&m=143153461415534&w=2
Thanks,
Shaohua
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Recent drive errors
From: Thomas Fjellstrom @ 2015-05-20 5:38 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <34123545.0U2U7Wo4pc@balsa>
On Tue 19 May 2015 10:07:49 AM Thomas Fjellstrom wrote:
> On Tue 19 May 2015 10:51:59 AM you wrote:
> > On 05/19/2015 10:32 AM, Thomas Fjellstrom wrote:
> > > On Tue 19 May 2015 09:23:20 AM Phil Turmel wrote:
> > >> Depends. In a properly functioning array that gets scrubbed
> > >> occasionally, or sufficiently heavy use to read the entire contents
> > >> occasionally, the UREs get rewritten by MD right away. Any UREs then
> > >> only show up once.
> > >
> > > I have made sure that it's doing regular scrubs, and regular SMART
> > > scans.
> > > This time...
> >
> > Yes, and this drive was kicked out. Because it wouldn't be listening
> > when MD tried to write over the error it found.
>
[snip]
>
> > I posted this link earlier, but it is particularly relevant:
> > http://marc.info/?l=linux-raid&m=133665797115876&w=2
> >
> > >> Interesting. I suspect that if you wipe that disk with noise, read it
> > >> all back, and wipe it again, you'll have a handful of relocations.
> > >
> > > It looks like each one of the blocks in that display is 128KiB. Which i
> > > think means those red blocks aren't very far apart. Maybe 80MiB apart?
> > > Would it reallocate all of those? That'd be a lot of reallocated
> > > sectors.
> >
> > Drives will only reallocate where a previous read failed (making it
> > pending), then write and follow-up verification fails. In general,
> > writes are unverified at the time of write (or your write performance
> > would be dramatically slower than read).
>
> Right. I was just thinking about how you mentioned that I'd get a handful of
> reallocations based on the latency shown in the image I posted. It's a lot
> of sectors that seem to be affected by the latency spikes, so I assumed
> (probably wrongly) that many of them may be reallocated afterwards.
>
> If this drive ends up not reallocating a single sector, or only a few, I may
> just keep it around as a hot spare, though i feel that's not the best idea,
> if it is degrading, then when it actually goes to use that disk it has a
> higher chance of failing.
Well here's something:
[78447.747221] sd 0:0:15:0: [sdf] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[78447.749092] sd 0:0:15:0: [sdf] Sense Key : Medium Error [current]
[78447.751034] sd 0:0:15:0: [sdf] Add. Sense: Unrecovered read error
[78447.752925] sd 0:0:15:0: [sdf] CDB: Read(16) 88 00 00 00 00 00 ef 7a 0f b0 00 00 00 08 00 00
[78447.754746] blk_update_request: critical medium error, dev sdf, sector 4017754032
[78447.756700] Buffer I/O error on dev sdf, logical block 502219254, async page read
<many many more of the above>
5 Reallocated_Sector_Ct PO--CK 087 087 036 - 17232
187 Reported_Uncorrect -O--CK 001 001 000 - 8236
197 Current_Pending_Sector -O--C- 024 024 000 - 12584
198 Offline_Uncorrectable ----C- 024 024 000 - 12584
Badblocks is showing a bunch of errors now, and the above is what's in dmesg and smartctl.
So I guess it was dead after all.
> > >> You have it backwards. If you have WD Reds, they are correct out of
> > >> the
> > >> box. It's when you *don't* have ERC support, or you only have desktop
> > >> ERC, that you need to take special action.
> > >
> > > I was under the impression you still had to enable ERC on boot. And I
> > > /thought/ I read that you still want to adjust the timeouts, though not
> > > the
> > > same as for consumer drives.
> >
> > Desktop / consumer drives that support ERC typically ship with it
> > disabled, so they behave just like drives that don't support it at all.
> >
> > So a boot script would enable ERC on drives where it can (and not
> >
> > already OK), and set long driver timeouts on the rest.
> >
> > Any drive that claims "raid" compatibility will have ERC enabled by
> > default. Typically 7.0 seconds. WD Reds do. Enterprise drives do, and
> > have better URE specs, too.
>
> Good to know.
>
> > >> If you have consumer grade drives in a raid array, and you don't have
> > >> boot scripts or udev rules to deal with timeout mismatch, your *ss is
> > >> hanging in the wind. The links in my last msg should help you out.
> > >
> > > There was some talk of ERC/TLER and md. I'll still have to find or write
> > > a
> > > script to properly set up timeouts and enable TLER on drives capable of
> > > it
> > > (that don't come with it enabled by default).
> >
> > Before I got everything onto proper drives, I just put what I needed
> > into rc.local.
>
[snip]
>
> > Chris Murphy posted some udev rules that will likely work for you. I
> > haven't tried them myself, though.
> >
> > https://www.marc.info/?l=linux-raid&m=142487508806844&w=3
>
> Thanks :)
>
> > Phil
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Thomas Fjellstrom
thomas@fjellstrom.ca
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox