* Re: [PATCH 0/1] RFC: Unreliable discard performance can cripple RAID1
From: Martin K. Petersen @ 2015-06-25 1:05 UTC (permalink / raw)
To: Jes Sorensen; +Cc: NeilBrown, linux-raid
In-Reply-To: <wrfj616dtlmb.fsf@redhat.com>
>>>>> "Jes" == Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>> Or is this some non-SATA/SCSI SSD that has a 'make_request_fn'
>> driver? I think I came across one of those before (NVMe). In that
>> case - the driver needs to be fixed.
Jes> Nope the block layer doesn't merge the requests at this point.
Then something broke it. Discard merging used to work.
--
Martin K. Petersen Oracle Linux Engineering
^ permalink raw reply
* Re: mismatch_cnt constantly goes up on ssd+hdd raid1
From: NeilBrown @ 2015-06-25 1:33 UTC (permalink / raw)
To: tlknv; +Cc: linux-raid
In-Reply-To: <3700381434301996@web21o.yandex.ru>
On Sun, 14 Jun 2015 20:13:16 +0300 tlknv <tlknv@yandex.ru> wrote:
> Hello,
> I have raid 1 which mirrors a root/boot partition on 1SSD and 2HDD
> (write-mostly). mismatch_cnt goes up even when there are very few
> writes to the partition as /var is mounted separatly. After I update
> several packages I typically see mismatch_cnt somewhere between
> 500,000 and 2,000,000. I have read a number of threads in this DL
> but could not find an explanation of what could cause mismatch_cnt
> to grow that much. I checked md5 sums using
> /var/lib/dpkg/info/*.md5sums, and didn't see many errors, even
> though there are few, mostly in text files which look ok to me. I
> guess when I check, all reads go to SSD (as both HDDs in this raid
> are write-mostly), and thus md5sum only shows no problem on
> SSD. Note, this partition is used as both boot and root and just in
> case here is some more info about my system:
This does surprise me.
I had another look at the code and there could be a bug that would let
'check' see the difference between when the first write completes and
when the write-behind writes complete, but you would need to run the
check while the install was happening for that to be noticed, and even
then you would need to be unlucky.
What you could try is:
- add a bitmap (mdadm --grow /dev/md0 --bitmap=internal) so that
recovery will be fast if you remove then re-add a device.
- fail and remove one of the HDDs
mdadm /dev/md0 --fail /dev/sda2
mdadm /dev/md0 --remove /dev/sda2
- Find the data offset and use losetup to access the data directly.
mdadm --examine /dev/sda2 | grep 'Data Offset'
Data Offset : 160 sectors.
convert that to 'K' and
losetup --read-only --offset=80K /dev/loop0 /dev/sda2
- perform some *read-only* examintion of loop0.
fsck -n /dev/loop0
mount -o ro /dev/loop0 /mnt
and see if there are any differences in files that have changed
recently.
- when finished, "umount /mnt", "losetup -d /dev/loop0" and
mdadm /dev/md0 --re-add /dev/sda2
> root@tbeh:~# sync; cmp -l /dev/sdc2 /dev/sda2|wc -l
> cmp: EOF on /dev/sdc2
> 1903215
>
> BTW, only first few hundren bytes (at most) have non-zero value on SSD, the rest of differences has 0 bytes on SSD.
> 4233 0 347
> 4234 70 65
> 4235 232 241
> 4257 0 1
Any bytes before the "Data Offset" identified above could easily be
different, or after "Data Offset" + "Used Dev Size".
What bytes are different within that range/
NeilBrown
^ permalink raw reply
* ONLY
From: Federico, L. @ 2015-06-25 5:13 UTC (permalink / raw)
My name is Gatan Magsino, I work with Mediterranean Bank in Malta. Can i trust you with a business worth 8.3 million USD? Please reply ONLY to my private email: magsinoga@yahoo.com.hk for more information.
^ permalink raw reply
* Re: mismatch_cnt constantly goes up on ssd+hdd raid1
From: Roman Mamedov @ 2015-06-25 5:19 UTC (permalink / raw)
To: NeilBrown; +Cc: tlknv, linux-raid
In-Reply-To: <20150625113335.7bf72b0b@noble>
[-- Attachment #1: Type: text/plain, Size: 1891 bytes --]
On Thu, 25 Jun 2015 11:33:35 +1000
NeilBrown <neilb@suse.com> wrote:
> On Sun, 14 Jun 2015 20:13:16 +0300 tlknv <tlknv@yandex.ru> wrote:
>
> > Hello,
>
> > I have raid 1 which mirrors a root/boot partition on 1SSD and 2HDD
> > (write-mostly). mismatch_cnt goes up even when there are very few
> > writes to the partition as /var is mounted separatly. After I update
> > several packages I typically see mismatch_cnt somewhere between
> > 500,000 and 2,000,000. I have read a number of threads in this DL
> > but could not find an explanation of what could cause mismatch_cnt
> > to grow that much. I checked md5 sums using
> > /var/lib/dpkg/info/*.md5sums, and didn't see many errors, even
> > though there are few, mostly in text files which look ok to me. I
> > guess when I check, all reads go to SSD (as both HDDs in this raid
> > are write-mostly), and thus md5sum only shows no problem on
> > SSD. Note, this partition is used as both boot and root and just in
> > case here is some more info about my system:
>
> This does surprise me.
>
> I had another look at the code and there could be a bug that would let
> 'check' see the difference between when the first write completes and
> when the write-behind writes complete, but you would need to run the
> check while the install was happening for that to be noticed, and even
> then you would need to be unlucky.
Couldn't this be simply the normal observed effect of using TRIM on SSD?
After deleting some files, the filesystem issues a discard request, it
does nothing to the HDDs, but the content of the discared areas on SSD is no
longer deterministic (or mostly zeroed, as mentioned in the original report).
So there is now a mismatch between the content of HDDs and SSD, but since it
is in the area of deleted files, it doesn't affect the system in any way.
--
With respect,
Roman
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply
* Re: mismatch_cnt constantly goes up on ssd+hdd raid1
From: Brad Campbell @ 2015-06-25 5:30 UTC (permalink / raw)
To: Roman Mamedov, NeilBrown; +Cc: tlknv, linux-raid
In-Reply-To: <20150625101959.23981c72@natsu>
On 25/06/15 13:19, Roman Mamedov wrote:
> Couldn't this be simply the normal observed effect of using TRIM on SSD?
> After deleting some files, the filesystem issues a discard request, it
> does nothing to the HDDs, but the content of the discared areas on SSD is no
> longer deterministic (or mostly zeroed, as mentioned in the original report).
> So there is now a mismatch between the content of HDDs and SSD, but since it
> is in the area of deleted files, it doesn't affect the system in any way.
>
I get this on a RAID10 with Intel & Samsung SSD's. One supports
deterministic after TRIM and the other doesn't. Mismatch count is always
through the roof as a result but there are no other negative effects.
^ permalink raw reply
* Re: NULL pointer dereference at 0000000000000038 IP: [<ffffffff815f514f>] bitmap_load+0x45f/0x610
From: NeilBrown @ 2015-06-25 7:11 UTC (permalink / raw)
To: Nate Clark; +Cc: linux-raid
In-Reply-To: <CAG2YfWODJEAV2i=WhzvJQw2AicKL0-c-EvAMA+Y-UVQRo+jgrQ@mail.gmail.com>
On Fri, 19 Jun 2015 17:18:45 -0400 Nate Clark <nate@neworld.us> wrote:
> Hi,
>
> I encountered a null pointer in md on kernel 4.0.4 and 4.0.5. I was running
> Fedora so I filed this bug with redhat,
> https://bugzilla.redhat.com/show_bug.cgi?id=1232492.
>
> It seems pretty easy to encounter.
> 1) Add PROGRAM line in mdadm.conf, which points to a script that just
> sleeps for 5 or 10 seconds
> 2) Create md device (I used raid 1 but I don't think that matters)
> 3) Stop that md device
> 4) Before the monitor program finishes execution assemble that md device.
>
> On my system this always cause an Oops.
Hi,
thanks for the report.
I managed to reproduce this, though it didn't seem quite as easy for me as for
you.
Anyway I found the bug and have a fix - see below.
should get into 4.2 soon and into stable releases in due course.
Thanks,
NeilBrown
From: NeilBrown <neilb@suse.de>
Date: Thu, 25 Jun 2015 17:01:40 +1000
Subject: [PATCH] md: clear mddev->private when it has been freed.
If ->private is set when ->run is called, it is assumed to be
a 'config' prepared as part of 'reshape'.
So it is important when we free that config, that we also clear ->private.
This is not often a problem as the mddev will normally be discarded
shortly after the config us freed.
However if an 'assemble' races with a final close, the assemble can use
the old mddev which has a stale ->private. This leads to any of
various sorts of crashes.
So clear ->private after calling ->free().
Reported-by: Nate Clark <nate@neworld.us>
Cc: stable@vger.kernel.org (v4.0+)
Fixes: afa0f557cb15 ("md: rename ->stop to ->free")
Signed-off-by: NeilBrown <neilb@suse.com>
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 5a6681ad9778..4b7b31b6f25c 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -5178,6 +5178,7 @@ int md_run(struct mddev *mddev)
mddev_detach(mddev);
if (mddev->private)
pers->free(mddev, mddev->private);
+ mddev->private = NULL;
module_put(pers->owner);
bitmap_destroy(mddev);
return err;
@@ -5313,6 +5314,7 @@ static void md_clean(struct mddev *mddev)
mddev->changed = 0;
mddev->degraded = 0;
mddev->safemode = 0;
+ mddev->private = NULL;
mddev->merge_check_needed = 0;
mddev->bitmap_info.offset = 0;
mddev->bitmap_info.default_offset = 0;
@@ -5385,6 +5387,7 @@ static void __md_stop(struct mddev *mddev)
mddev->pers = NULL;
spin_unlock(&mddev->lock);
pers->free(mddev, mddev->private);
+ mddev->private = NULL;
if (pers->sync_request && mddev->to_remove == NULL)
mddev->to_remove = &md_redundancy_group;
module_put(pers->owner);
^ permalink raw reply related
* Re: mismatch_cnt constantly goes up on ssd+hdd raid1
From: NeilBrown @ 2015-06-25 7:25 UTC (permalink / raw)
To: Roman Mamedov; +Cc: tlknv, linux-raid
In-Reply-To: <20150625101959.23981c72@natsu>
On Thu, 25 Jun 2015 10:19:59 +0500 Roman Mamedov <rm@romanrm.net> wrote:
> On Thu, 25 Jun 2015 11:33:35 +1000
> NeilBrown <neilb@suse.com> wrote:
>
> > On Sun, 14 Jun 2015 20:13:16 +0300 tlknv <tlknv@yandex.ru> wrote:
> >
> > > Hello,
> >
> > > I have raid 1 which mirrors a root/boot partition on 1SSD and 2HDD
> > > (write-mostly). mismatch_cnt goes up even when there are very few
> > > writes to the partition as /var is mounted separatly. After I update
> > > several packages I typically see mismatch_cnt somewhere between
> > > 500,000 and 2,000,000. I have read a number of threads in this DL
> > > but could not find an explanation of what could cause mismatch_cnt
> > > to grow that much. I checked md5 sums using
> > > /var/lib/dpkg/info/*.md5sums, and didn't see many errors, even
> > > though there are few, mostly in text files which look ok to me. I
> > > guess when I check, all reads go to SSD (as both HDDs in this raid
> > > are write-mostly), and thus md5sum only shows no problem on
> > > SSD. Note, this partition is used as both boot and root and just in
> > > case here is some more info about my system:
> >
> > This does surprise me.
> >
> > I had another look at the code and there could be a bug that would let
> > 'check' see the difference between when the first write completes and
> > when the write-behind writes complete, but you would need to run the
> > check while the install was happening for that to be noticed, and even
> > then you would need to be unlucky.
>
> Couldn't this be simply the normal observed effect of using TRIM on SSD?
Yes, of course it could. I try not to think about TRIM to much - makes me ill :-)
Thanks,
NeilBrown
> After deleting some files, the filesystem issues a discard request, it
> does nothing to the HDDs, but the content of the discared areas on SSD is no
> longer deterministic (or mostly zeroed, as mentioned in the original report).
> So there is now a mismatch between the content of HDDs and SSD, but since it
> is in the area of deleted files, it doesn't affect the system in any way.
>
^ permalink raw reply
* Re: RAID1 removing failed disk returns EBUSY
From: Xiao Ni @ 2015-06-25 9:42 UTC (permalink / raw)
To: Neil Brown; +Cc: Joe Lawrence, linux-raid, Bill Kuzeja
In-Reply-To: <20150617125151.372bb103@home.neil.brown.name>
----- Original Message -----
> From: "Neil Brown" <neilb@suse.de>
> To: "XiaoNi" <xni@redhat.com>
> Cc: "Joe Lawrence" <joe.lawrence@stratus.com>, linux-raid@vger.kernel.org, "Bill Kuzeja" <william.kuzeja@stratus.com>
> Sent: Wednesday, June 17, 2015 10:51:51 AM
> Subject: Re: RAID1 removing failed disk returns EBUSY
>
> On Wed, 10 Jun 2015 14:26:41 +0800
> XiaoNi <xni@redhat.com> wrote:
>
> >
> >
> > On 02/03/2015 04:10 PM, Xiao Ni wrote:
> > >
> > > ----- Original Message -----
> > >> From: "NeilBrown" <neilb@suse.de>
> > >> To: "Xiao Ni" <xni@redhat.com>
> > >> Cc: "Joe Lawrence" <joe.lawrence@stratus.com>,
> > >> linux-raid@vger.kernel.org, "Bill Kuzeja" <william.kuzeja@stratus.com>
> > >> Sent: Monday, February 2, 2015 2:36:01 PM
> > >> Subject: Re: RAID1 removing failed disk returns EBUSY
> > >>
> > >> On Thu, 29 Jan 2015 07:14:16 -0500 (EST) Xiao Ni <xni@redhat.com> wrote:
> > >>
> > >>>
> > >>> ----- Original Message -----
> > >>>> From: "NeilBrown" <neilb@suse.de>
> > >>>> To: "Xiao Ni" <xni@redhat.com>
> > >>>> Cc: "Joe Lawrence" <joe.lawrence@stratus.com>,
> > >>>> linux-raid@vger.kernel.org, "Bill Kuzeja" <william.kuzeja@stratus.com>
> > >>>> Sent: Thursday, January 29, 2015 11:52:17 AM
> > >>>> Subject: Re: RAID1 removing failed disk returns EBUSY
> > >>>>
> > >>>> On Sun, 18 Jan 2015 21:33:50 -0500 (EST) Xiao Ni <xni@redhat.com>
> > >>>> wrote:
> > >>>>
> > >>>>>
> > >>>>> ----- Original Message -----
> > >>>>>> From: "Joe Lawrence" <joe.lawrence@stratus.com>
> > >>>>>> To: "Xiao Ni" <xni@redhat.com>
> > >>>>>> Cc: "NeilBrown" <neilb@suse.de>, linux-raid@vger.kernel.org, "Bill
> > >>>>>> Kuzeja" <william.kuzeja@stratus.com>
> > >>>>>> Sent: Friday, January 16, 2015 11:10:31 PM
> > >>>>>> Subject: Re: RAID1 removing failed disk returns EBUSY
> > >>>>>>
> > >>>>>> On Fri, 16 Jan 2015 00:20:12 -0500
> > >>>>>> Xiao Ni <xni@redhat.com> wrote:
> > >>>>>>> Hi Joe
> > >>>>>>>
> > >>>>>>> Thanks for reminding me. I didn't do that. Now it can remove
> > >>>>>>> successfully after writing
> > >>>>>>> "idle" to sync_action.
> > >>>>>>>
> > >>>>>>> I thought wrongly that the patch referenced in this mail is
> > >>>>>>> fixed
> > >>>>>>> for
> > >>>>>>> the problem.
> > >>>>>> So it sounds like even with 3.18 and a new mdadm, this bug still
> > >>>>>> persists?
> > >>>>>>
> > >>>>>> -- Joe
> > >>>>>>
> > >>>>>> --
> > >>>>> Hi Joe
> > >>>>>
> > >>>>> I'm a little confused now. Does the patch
> > >>>>> 45eaf45dfa4850df16bc2e8e7903d89021137f40 from linux-stable
> > >>>>> resolve the problem?
> > >>>>>
> > >>>>> My environment is:
> > >>>>>
> > >>>>> [root@dhcp-12-133 mdadm]# mdadm --version
> > >>>>> mdadm - v3.3.2-18-g93d3bd3 - 18th December 2014 (this is the newest
> > >>>>> upstream)
> > >>>>> [root@dhcp-12-133 mdadm]# uname -r
> > >>>>> 3.18.2
> > >>>>>
> > >>>>>
> > >>>>> My steps are:
> > >>>>>
> > >>>>> [root@dhcp-12-133 mdadm]# lsblk
> > >>>>> sdb 8:16 0 931.5G 0 disk
> > >>>>> └─sdb1 8:17 0 5G 0 part
> > >>>>> sdc 8:32 0 186.3G 0 disk
> > >>>>> sdd 8:48 0 931.5G 0 disk
> > >>>>> └─sdd1 8:49 0 5G 0 part
> > >>>>> [root@dhcp-12-133 mdadm]# mdadm -CR /dev/md0 -l1 -n2 /dev/sdb1
> > >>>>> /dev/sdd1
> > >>>>> --assume-clean
> > >>>>> mdadm: Note: this array has metadata at the start and
> > >>>>> may not be suitable as a boot device. If you plan to
> > >>>>> store '/boot' on this device please ensure that
> > >>>>> your boot-loader understands md/v1.x metadata, or use
> > >>>>> --metadata=0.90
> > >>>>> mdadm: Defaulting to version 1.2 metadata
> > >>>>> mdadm: array /dev/md0 started.
> > >>>>>
> > >>>>> Then I unplug the disk.
> > >>>>>
> > >>>>> [root@dhcp-12-133 mdadm]# lsblk
> > >>>>> sdc 8:32 0 186.3G 0 disk
> > >>>>> sdd 8:48 0 931.5G 0 disk
> > >>>>> └─sdd1 8:49 0 5G 0 part
> > >>>>> └─md0 9:0 0 5G 0 raid1
> > >>>>> [root@dhcp-12-133 mdadm]# echo faulty >
> > >>>>> /sys/block/md0/md/dev-sdb1/state
> > >>>>> [root@dhcp-12-133 mdadm]# echo remove >
> > >>>>> /sys/block/md0/md/dev-sdb1/state
> > >>>>> -bash: echo: write error: Device or resource busy
> > >>>>> [root@dhcp-12-133 mdadm]# echo idle > /sys/block/md0/md/sync_action
> > >>>>> [root@dhcp-12-133 mdadm]# echo remove >
> > >>>>> /sys/block/md0/md/dev-sdb1/state
> > >>>>>
> > >>>> I cannot reproduce this - using linux 3.18.2. I'd be surprised if
> > >>>> mdadm
> > >>>> version affects things.
> > >>> Hi Neil
> > >>>
> > >>> I'm very curious, because it can reproduce in my machine 100%.
> > >>>
> > >>>> This error (Device or resoource busy) implies that rdev->raid_disk is
> > >>>> >=
> > >>>> 0
> > >>>> (tested in state_store()).
> > >>>>
> > >>>> ->raid_disk is set to -1 by remove_and_add_spares() providing:
> > >>>> 1/ it isn't Blocked (which is very unlikely)
> > >>>> 2/ hot_remove_disk succeeds, which it will if nr_pending is zero,
> > >>>> and
> > >>>> 3/ nr_pending is zero.
> > >>> I remember I have tired to check those reasons. But it's really is
> > >>> the
> > >>> reason 1
> > >>> which is very unlikely.
> > >>>
> > >>> I add some code in the function array_state_show
> > >>>
> > >>> array_state_show(struct mddev *mddev, char *page) {
> > >>> enum array_state st = inactive;
> > >>> struct md_rdev *rdev;
> > >>>
> > >>> rdev_for_each_rcu(rdev, mddev) {
> > >>> printk(KERN_ALERT "search for %s\n",
> > >>> rdev->bdev->bd_disk->disk_name);
> > >>> if (test_bit(Blocked, &rdev->flags))
> > >>> printk(KERN_ALERT "rdev is Blocked\n");
> > >>> else
> > >>> printk(KERN_ALERT "rdev is not Blocked\n");
> > >>> }
> > >>>
> > >>> When I echo 1 > /sys/block/sdc/device/delete, then I ran command:
> > >>>
> > >>> [root@dhcp-12-133 md]# cat /sys/block/md0/md/array_state
> > >>> read-auto
> > >> ^^^^^^^^^
> > >>
> > >> I think that is half the explanation.
> > >> You must have the md_mod.start_ro parameter set to '1'.
> > >>
> > >>
> > >>> [root@dhcp-12-133 md]# dmesg
> > >>> [ 2679.559185] search for sdc
> > >>> [ 2679.559189] rdev is Blocked
> > >>> [ 2679.559190] search for sdb
> > >>> [ 2679.559190] rdev is not Blocked
> > >>>
> > >>> So sdc is Blocked
> > >> and that is the other half - thanks.
> > >> (yes, I was wrong. Sometimes it is easier than being right, but still
> > >> yields results).
> > >>
> > >> When a device fails, it is Blocked until the metadata is updated to
> > >> record
> > >> the failure. This ensures that no writes succeed without writing to
> > >> that
> > >> device, until we a certain that no read will try reading from that
> > >> device,
> > >> even after a crash/restart.
> > >>
> > >> Blocked is cleared after the metadata is written, but read-auto (and
> > >> read-only) devices never write out their metadata. So blocked doesn't
> > >> get
> > >> cleared.
> > >>
> > >> When you "echo idle > .../sync_action" one of the side effects is to
> > >> with
> > >> from 'read-auto' to fully active. This allows the metadata to be
> > >> written,
> > >> Blocked to be cleared, and the device to be removed.
> > >>
> > >> If you
> > >> echo none > /sys/block/md0/md/dev-sdc/slot
> > >>
> > >> first, then the remove will work.
> > >>
> > >> We could possibly fix it with something like the following, but I'm not
> > >> sure
> > >> I like it. There is no guarantee that I can see which would ensure the
> > >> superblock got updated before the first write if the array switch to
> > >> read/write.
> > >>
> > >> NeilBrown
> > >>
> > >> diff --git a/drivers/md/md.c b/drivers/md/md.c
> > >> index 9233c71138f1..b3d1e8e5e067 100644
> > >> --- a/drivers/md/md.c
> > >> +++ b/drivers/md/md.c
> > >> @@ -7528,7 +7528,7 @@ static int remove_and_add_spares(struct mddev
> > >> *mddev,
> > >> rdev_for_each(rdev, mddev)
> > >> if ((this == NULL || rdev == this) &&
> > >> rdev->raid_disk >= 0 &&
> > >> - !test_bit(Blocked, &rdev->flags) &&
> > >> + (!test_bit(Blocked, &rdev->flags) || mddev->ro) &&
> > >> (test_bit(Faulty, &rdev->flags) ||
> > >> ! test_bit(In_sync, &rdev->flags)) &&
> > >> atomic_read(&rdev->nr_pending)==0) {
> > >>
> > >>
> > >>
> > > Hi Neil
> > >
> > > I have tried the patch and the problem can be fixed by it. But I'm
> > > sorry that I can't
> > > give more advices for better idea about this. I'm not familiar with the
> > > metadata part about
> > > the md. I'll try to get more time to read the code about md.
> > >
> > Hi Neil
> >
> > I don't see the patch in linux-stable, do you miss this?
>
> I don't believe this bug is sufficiently serious for the patch to go to
> -stable. However it doesn't need to be fixed - thanks for the reminder.
>
> I've just queued the following patch which I am happy with. If you
> could confirm that it works for you, I would appreciate that.
>
> Thanks,
> NeilBrown
>
>
> From: Neil Brown <neilb@suse.de>
> Date: Wed, 17 Jun 2015 12:31:46 +1000
> Subject: [PATCH] md: clear Blocked flag on failed devices when array is
> read-only.
>
> The Blocked flag indicates that a device has failed but that this
> fact hasn't been recorded in the metadata yet. Writes to such
> devices cannot be allowed until the metadata has been updated.
>
> On a read-only array, the Blocked flag will never be cleared.
> This prevents the device being removed from the array.
>
> If the metadata is being handled by the kernel
> (i.e. !mddev->external), then we can be sure that if the array is
> switch to writable, then a metadata update will happen and will
> record the failure. So we don't need the flag set.
>
> If metadata is externally managed, it is upto the external manager
> to clear the 'blocked' flag.
>
> Reported-by: XiaoNi <xni@redhat.com>
> Signed-off-by: NeilBrown <neilb@suse.de>
>
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index 3d339e2..5a6681a 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -8125,6 +8125,15 @@ void md_check_recovery(struct mddev *mddev)
> int spares = 0;
>
> if (mddev->ro) {
> + struct md_rdev *rdev;
> + if (!mddev->external && mddev->in_sync)
> + /* 'Blocked' flag not needed as failed devices
> + * will be recorded if array switched to read/write.
> + * Leaving it set will prevent the device
> + * from being removed.
> + */
> + rdev_for_each(rdev, mddev)
> + clear_bit(Blocked, &rdev->flags);
> /* On a read-only array we can:
> * - remove failed devices
> * - add already-in_sync devices if the array itself
>
>
Hi Neil
Sorry for late response for this.
I have tried the patch. When I unplug the disk(sdc1) which belongs to the raid1, the directory
/sys/block/md0/md/dev-sdc1 is deleted. I haven't read the code for unplug device. So is it what
you want?
Best Regards
Xiao
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: (R) in mdstat output, clean && degraded
From: Jared Mauch @ 2015-06-25 11:47 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150625085201.4b2c6995@noble>
> On Jun 24, 2015, at 6:52 PM, NeilBrown <neilb@suse.com> wrote:
>
> You should be able to turn this into a fully functional RAID1 array by:
>
> mdadm /dev/md127 --fail /dev/sdd1
> mdadm /dev/md127 --remove /dev/sdd1
> mdadm /dev/md127 --re-add /dev/sdd1
>
> When you fail sdd1, sdg1 will change from being a 'replacement' to being
> a regular member.
> When you --re-add /dev/sdd1 you benefit from the fact that raid1
> doesn't really care which device is in which slot (unlike raid5).
> So re-adding something marked for slot 0 into slot 1 is perfectly
> acceptable.
> As the bitmap is present and uptodate, the recovery will be very fast.
>
> I would recommend doing some basic checks for data consistency after
> removing sdd1 and before re-adding it. I might be wrong about
> something and sdg1 might contain complete garbage - it never hurts to
> check :-)
So this actually doesn’t work:
# mdadm /dev/md127 --fail /dev/sdd1
mdadm: set device faulty failed for /dev/sdd1: Device or resource busy
# mdadm /dev/md127 --fail /dev/sdg1
mdadm: set device faulty failed for /dev/sdg1: Device or resource busy
so I may have to physically remove sdd1 to recover this it seems, or stop
and reassemble is my guess.
- Jared
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: mismatch_cnt constantly goes up on ssd+hdd raid1
From: tlknv @ 2015-06-25 15:33 UTC (permalink / raw)
To: NeilBrown, Roman Mamedov; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <20150625172530.5c5338b6@noble>
Neil,
Thanks a lot for all the info and steps to identify the problem.
I have just discovered that I had 'discard' mount option even though I though it wasn't there :-(
After removing 'discard' and forcing 'repair' mismatch_cnt stays 0 even after a bunch of writes and deletes (the most importantly) to the partition. BTW, what are the units in mismatch_cnt? Is it 512 sectors or something else?
AFAIU md could potentially collect info on trimmed sectors/blocks and exclude them from mismatch checking. Could not it?
I'll look at the range of the sectors which are different even when mismatch_cnt is 0.
Thanks again,
Boris
25.06.2015, 10:25, "NeilBrown" <neilb@suse.com>:
> On Thu, 25 Jun 2015 10:19:59 +0500 Roman Mamedov <rm@romanrm.net> wrote:
>
>> On Thu, 25 Jun 2015 11:33:35 +1000
>> NeilBrown <neilb@suse.com> wrote:
>>
>> > On Sun, 14 Jun 2015 20:13:16 +0300 tlknv <tlknv@yandex.ru> wrote:
>> >
>> > > Hello,
>> >
>> > > I have raid 1 which mirrors a root/boot partition on 1SSD and 2HDD
>> > > (write-mostly). mismatch_cnt goes up even when there are very few
>> > > writes to the partition as /var is mounted separatly. After I update
>> > > several packages I typically see mismatch_cnt somewhere between
>> > > 500,000 and 2,000,000. I have read a number of threads in this DL
>> > > but could not find an explanation of what could cause mismatch_cnt
>> > > to grow that much. I checked md5 sums using
>> > > /var/lib/dpkg/info/*.md5sums, and didn't see many errors, even
>> > > though there are few, mostly in text files which look ok to me. I
>> > > guess when I check, all reads go to SSD (as both HDDs in this raid
>> > > are write-mostly), and thus md5sum only shows no problem on
>> > > SSD. Note, this partition is used as both boot and root and just in
>> > > case here is some more info about my system:
>> >
>> > This does surprise me.
>> >
>> > I had another look at the code and there could be a bug that would let
>> > 'check' see the difference between when the first write completes and
>> > when the write-behind writes complete, but you would need to run the
>> > check while the install was happening for that to be noticed, and even
>> > then you would need to be unlucky.
>>
>> Couldn't this be simply the normal observed effect of using TRIM on SSD?
>
> Yes, of course it could. I try not to think about TRIM to much - makes me ill :-)
>
> Thanks,
> NeilBrown
>
>> After deleting some files, the filesystem issues a discard request, it
>> does nothing to the HDDs, but the content of the discared areas on SSD is no
>> longer deterministic (or mostly zeroed, as mentioned in the original report).
>> So there is now a mismatch between the content of HDDs and SSD, but since it
>> is in the area of deleted files, it doesn't affect the system in any way.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: mismatch_cnt constantly goes up on ssd+hdd raid1
From: Roman Mamedov @ 2015-06-25 15:50 UTC (permalink / raw)
To: tlknv; +Cc: NeilBrown, linux-raid@vger.kernel.org
In-Reply-To: <2762371435246396@web9o.yandex.ru>
[-- Attachment #1: Type: text/plain, Size: 685 bytes --]
On Thu, 25 Jun 2015 18:33:16 +0300
tlknv <tlknv@yandex.ru> wrote:
> I have just discovered that I had 'discard' mount option even though I though it wasn't there :-(
> After removing 'discard' and forcing 'repair' mismatch_cnt stays 0 even after a bunch of writes and deletes (the most importantly) to the partition.
I wouldn't recommend disabling discard 'for good' though, from your experiment
we can probably conclude the mismatch_cnt numbers you had previously are indeed
harmless. If you had no other issues due to discard being active, then what you
did now is disable a working and useful feature to fix what's only a cosmetic
problem.
--
With respect,
Roman
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply
* Re: mismatch_cnt constantly goes up on ssd+hdd raid1
From: Boris T @ 2015-06-25 17:24 UTC (permalink / raw)
To: Roman Mamedov; +Cc: tlknv, NeilBrown, linux-raid@vger.kernel.org
In-Reply-To: <20150625205026.42f41a81@natsu>
Roman,
Thanks for the suggestion. I would probably consider keeping TRIM if I
had less free/unpartitioned space on my SSD. Currently all partitioned
space is just a fraction (about 1/3) of total SSD size. In this
situation I prefer to make sure that my raid is consistent than to
prolong the life of SSD a bit.
Thanks,
Boris
On Thu, Jun 25, 2015 at 8:50 AM, Roman Mamedov <rm@romanrm.net> wrote:
> On Thu, 25 Jun 2015 18:33:16 +0300
> tlknv <tlknv@yandex.ru> wrote:
>
>> I have just discovered that I had 'discard' mount option even though I though it wasn't there :-(
>> After removing 'discard' and forcing 'repair' mismatch_cnt stays 0 even after a bunch of writes and deletes (the most importantly) to the partition.
>
> I wouldn't recommend disabling discard 'for good' though, from your experiment
> we can probably conclude the mismatch_cnt numbers you had previously are indeed
> harmless. If you had no other issues due to discard being active, then what you
> did now is disable a working and useful feature to fix what's only a cosmetic
> problem.
>
> --
> With respect,
> Roman
^ permalink raw reply
* DM_CACHE_SMQ is experimental and enabled by default? (was: Re: dm cache: add stochastic-multi-queue (smq) policy)
From: Geert Uytterhoeven @ 2015-06-26 7:47 UTC (permalink / raw)
To: Joe Thornber, Mike Snitzer; +Cc: linux-raid, Linux Kernel Mailing List
On Fri, Jun 26, 2015 at 2:13 AM, Linux Kernel Mailing List
<linux-kernel@vger.kernel.org> wrote:
> Gitweb: http://git.kernel.org/linus/;a=commit;h=66a636356647a9be8885c2ce2948de126577698a
> Commit: 66a636356647a9be8885c2ce2948de126577698a
> Parent: 40775257b97e27305cf5c2425be7acaa6edee4ea
> Refname: refs/heads/master
> Author: Joe Thornber <ejt@redhat.com>
> AuthorDate: Fri May 15 15:33:34 2015 +0100
> Committer: Mike Snitzer <snitzer@redhat.com>
> CommitDate: Thu Jun 11 17:12:59 2015 -0400
>
> dm cache: add stochastic-multi-queue (smq) policy
> diff --git a/drivers/md/Kconfig b/drivers/md/Kconfig
> index edcf4ab..b597273 100644
> --- a/drivers/md/Kconfig
> +++ b/drivers/md/Kconfig
> @@ -304,6 +304,18 @@ config DM_CACHE_MQ
> This is meant to be a general purpose policy. It prioritises
> reads over writes.
>
> +config DM_CACHE_SMQ
> + tristate "Stochastic MQ Cache Policy (EXPERIMENTAL)"
> + depends on DM_CACHE
> + default y
A feature cannot be "EXPERIMENTAL", and be enabled by default.
Please drop (at least) one of them.
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply
* Re: DM_CACHE_SMQ is experimental and enabled by default? (was: Re: dm cache: add stochastic-multi-queue (smq) policy)
From: Mike Snitzer @ 2015-06-26 16:05 UTC (permalink / raw)
To: Geert Uytterhoeven
Cc: Joe Thornber, linux-raid, Linux Kernel Mailing List, dm-devel
In-Reply-To: <CAMuHMdVfK8x8xXqaq=yX41qZA0L3VP-sezKfDureYMa-QqfM6g@mail.gmail.com>
On Fri, Jun 26 2015 at 3:47am -0400,
Geert Uytterhoeven <geert@linux-m68k.org> wrote:
> On Fri, Jun 26, 2015 at 2:13 AM, Linux Kernel Mailing List
> <linux-kernel@vger.kernel.org> wrote:
> > Gitweb: http://git.kernel.org/linus/;a=commit;h=66a636356647a9be8885c2ce2948de126577698a
> > Commit: 66a636356647a9be8885c2ce2948de126577698a
> > Parent: 40775257b97e27305cf5c2425be7acaa6edee4ea
> > Refname: refs/heads/master
> > Author: Joe Thornber <ejt@redhat.com>
> > AuthorDate: Fri May 15 15:33:34 2015 +0100
> > Committer: Mike Snitzer <snitzer@redhat.com>
> > CommitDate: Thu Jun 11 17:12:59 2015 -0400
> >
> > dm cache: add stochastic-multi-queue (smq) policy
>
> > diff --git a/drivers/md/Kconfig b/drivers/md/Kconfig
> > index edcf4ab..b597273 100644
> > --- a/drivers/md/Kconfig
> > +++ b/drivers/md/Kconfig
> > @@ -304,6 +304,18 @@ config DM_CACHE_MQ
> > This is meant to be a general purpose policy. It prioritises
> > reads over writes.
> >
> > +config DM_CACHE_SMQ
> > + tristate "Stochastic MQ Cache Policy (EXPERIMENTAL)"
> > + depends on DM_CACHE
> > + default y
>
> A feature cannot be "EXPERIMENTAL", and be enabled by default.
> Please drop (at least) one of them.
Thanks for your concern, but: no.
If you look closer the entirety of DM cache is marked EXPERIMENTAL:
config DM_CACHE
tristate "Cache target (EXPERIMENTAL)"
...
config DM_CACHE_MQ
tristate "MQ Cache Policy (EXPERIMENTAL)"
...
config DM_CACHE_SMQ
tristate "Stochastic MQ Cache Policy (EXPERIMENTAL)"
...
config DM_CACHE_CLEANER
tristate "Cleaner Cache Policy (EXPERIMENTAL)"
^ permalink raw reply
* Re: [Bug 100491] New: Oops under bitmap_start_sync [md_mod] at boot
From: Sami Liedes @ 2015-06-28 20:53 UTC (permalink / raw)
To: Goldwyn Rodrigues, Neil Brown, bugzilla-daemon; +Cc: linux-raid, linux-kernel
In-Reply-To: <12832_1435266173_558C6C7D_12832_48_1_bug-100491-1606@https.bugzilla.kernel.org/>
On Thu, Jun 25, 2015 at 09:02:45PM +0000, bugzilla-daemon@bugzilla.kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=100491
>
> Bug ID: 100491
> Summary: Oops under bitmap_start_sync [md_mod] at boot
[...]
> Reading all physical valumes. This may take a while...
> Found volume group "rootvg" using metadata type lvm2
> device-mapper: raid: Device 0 specified for rebuild: Clearing superblock
> md/raid1:mdX: active with 1 out of 2 mirrors
> mdX: invalid bitmap file superblock: bad magic
> md-cluster module not found.
> mdX: Could not setup cluster service (256)
> BUG: unable to handle kernel NULL pointer dereference at 0000000000000100
> IP: [<ffffffff8159e4a9>] _raw_spin_lock_irq+0x29/0x70
> PGD 0
> Oops: 0002 [#1] PREEMPT SMP
[...]
I'm marking this as a regression in bugzilla, since this seems to
prevent booting on 4.1.0 at least in certain circumstances (namely
those which I have; I wonder if any raid1 recovery works?) while 4.0.6
boots correctly.
I bisected this down to one of four commits. Well, assuming that the
problem was caused by changes in drivers/md; a fair assumption, I
think. The commits are:
$ git bisect view --oneline
f9209a3 bitmap_create returns bitmap pointer
96ae923 Gather on-going resync information of other nodes
54519c5 Lock bitmap while joining the cluster
b97e9257 Use separate bitmaps for each nodes in the cluster
The crash happens whether or not CONFIG_MD_CLUSTER is enabled.
Here's the versions I tested:
git bisect start '--' 'drivers/md'
# bad: [b953c0d234bc72e8489d3bf51a276c5c4ec85345] Linux 4.1
# good: [39a8804455fb23f09157341d3ba7db6d7ae6ee76] Linux 4.0
# bad: [9ffc8f7cb9647b13dfe4d1ad0d5e1427bb8b46d6] md/raid5: don't do chunk aligned read on degraded array.
# bad: [6dc69c9c460b0cf05b5b3f323a8b944a2e52e76d] md: recover_bitmaps() can be static
# bad: [4b26a08af92c0d9c0bce07612b56ff326112321a] Perform resync for cluster node failure
# good: [cf921cc19cf7c1e99f730a2faa02d80817d684a2] Add node recovery callbacks
# skip: [96ae923ab659e37dd5fc1e05ecbf654e2f94bcbe] Gather on-going resync information of other nodes
# bad: [f9209a323547f054c7439a3bf67c45e64a054bdd] bitmap_create returns bitmap pointer
# skip: [54519c5f4b398bcfe599f652b4ef4004d5fa63ff] Lock bitmap while joining the cluster
Sami
^ permalink raw reply
* [GIT PULL REQUEST] md updates for 4.2
From: NeilBrown @ 2015-06-29 1:19 UTC (permalink / raw)
To: Linus Torvalds
Cc: lkml, linux-raid, XiaoNi, Nate Clark, Firo Yang, Shaohua Li,
Yuanhan Liu, Alexey Dobriyan, Kent Overstreet, Christoph Hellwig,
Dongsu Park, Ming Lin, Peter Zijlstra (Intel)
The following changes since commit
0f57d86787d8b1076ea8f9cbdddda2a46d534a27:
Linux 4.1-rc8 (2015-06-14 15:51:10 -1000)
are available in the git repository at:
git://neil.brown.name/md/ tags/md/4.2
for you to fetch changes up to ab16bfc732c436658d13455f28b0b4a2608a7476:
md: clear Blocked flag on failed devices when array is read-only.
(2015-06-25 17:16:49 +1000)
----------------------------------------------------------------
md updates for 4.2
A mixed bag
- a few bug fixes
- some performance improvement that decrease lock contention
- some clean-up
Nothing major.
----------------------------------------------------------------
Alexey Dobriyan (1):
md: convert to kstrto*()
Firo Yang (1):
md: fix a build warning
Kent Overstreet (1):
md/raid10: make sync_request_write() call bio_copy_data()
Neil Brown (1):
md: clear Blocked flag on failed devices when array is read-only.
NeilBrown (2):
md: clear mddev->private when it has been freed.
md: unlock mddev_lock on an error path.
Shaohua Li (1):
md/raid5: ignore released_stripes check
Yuanhan Liu (3):
wait: introduce wait_event_exclusive_cmd
md/raid5: split wait_for_stripe and introduce wait_for_quiescent
md/raid5: per hash value and exclusive wait_for_stripe
drivers/md/md.c | 167 ++++++++++++++++++++++++++++++---------------------
drivers/md/raid10.c | 18 ++----
drivers/md/raid5.c | 46 +++++++++-----
drivers/md/raid5.h | 3 +-
include/linux/wait.h | 13 ++++
5 files changed, 146 insertions(+), 101 deletions(-)
^ permalink raw reply
* Re: [Bug 100491] New: Oops under bitmap_start_sync [md_mod] at boot
From: Austin S Hemmelgarn @ 2015-06-29 12:28 UTC (permalink / raw)
To: Sami Liedes, Goldwyn Rodrigues, Neil Brown, bugzilla-daemon
Cc: linux-raid, linux-kernel
In-Reply-To: <20150628205347.GA18403@sli.dy.fi>
[-- Attachment #1: Type: text/plain, Size: 1831 bytes --]
On 2015-06-28 16:53, Sami Liedes wrote:
> On Thu, Jun 25, 2015 at 09:02:45PM +0000, bugzilla-daemon@bugzilla.kernel.org wrote:
>> https://bugzilla.kernel.org/show_bug.cgi?id=100491
>>
>> Bug ID: 100491
>> Summary: Oops under bitmap_start_sync [md_mod] at boot
> [...]
>> Reading all physical valumes. This may take a while...
>> Found volume group "rootvg" using metadata type lvm2
>> device-mapper: raid: Device 0 specified for rebuild: Clearing superblock
>> md/raid1:mdX: active with 1 out of 2 mirrors
>> mdX: invalid bitmap file superblock: bad magic
>> md-cluster module not found.
>> mdX: Could not setup cluster service (256)
>> BUG: unable to handle kernel NULL pointer dereference at 0000000000000100
>> IP: [<ffffffff8159e4a9>] _raw_spin_lock_irq+0x29/0x70
>> PGD 0
>> Oops: 0002 [#1] PREEMPT SMP
> [...]
>
> I'm marking this as a regression in bugzilla, since this seems to
> prevent booting on 4.1.0 at least in certain circumstances (namely
> those which I have; I wonder if any raid1 recovery works?) while 4.0.6
> boots correctly.
I can confirm having the same issue with the MD code being used through
dm-raid.
>
> I bisected this down to one of four commits. Well, assuming that the
> problem was caused by changes in drivers/md; a fair assumption, I
> think. The commits are:
>
> $ git bisect view --oneline
> f9209a3 bitmap_create returns bitmap pointer
> 96ae923 Gather on-going resync information of other nodes
> 54519c5 Lock bitmap while joining the cluster
> b97e9257 Use separate bitmaps for each nodes in the cluster
My own bisect turned up the same set of commits, although I wouldn't
have the time to go any further with it until next weekend.
>
> The crash happens whether or not CONFIG_MD_CLUSTER is enabled.
Again, same here.
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2967 bytes --]
^ permalink raw reply
* Raid Recovery
From: Carter J. Castor @ 2015-06-29 21:05 UTC (permalink / raw)
To: linux-raid
I have an ubuntu RAID 10 server using mdadm. It has six drives and a
hot spare. The eSATA cables to the box holding the harddrives got
disconnected from the server. When I plugged them back, it immediately
saw only three of the drives and my hotspare and started rebuilding
the array. Not wanting to screw it up, I let it finish, and then I
rebooted. On start up, the array was inactive. When I tried to
assemble it, it told me that there was only one disk(!?) so I tried to
assemble it with the --force command. It then assembled, but the
logical volume that I use to mount the RAID partition was gone. I have
backups, but I'd like to save the data if at all possible. What's my
best course of action from here?
Here's the /proc/mdstat after rebooting:
md127 : active (auto-read-only) raid10 sdg1[5] sdd1[1] sdh1[6] sde1[7]
sdf1[4] sdc1[2]
2929890816 blocks super 1.2 256K chunks 2 far-copies [6/6] [UUUUUU]
Blkid output:
/dev/sda1: LABEL="KINGSTON" UUID="C3D4-FB34" TYPE="vfat"
/dev/sdb1: UUID="dedc8d6c-8155-f88d-b035-c8cf66d4e561"
UUID_SUB="f858b034-29ab-4d59-02d3-775b9b2139cd" LABEL="Pangolin:0"
TYPE="linux_raid_member"
/dev/sdc1: UUID="dedc8d6c-8155-f88d-b035-c8cf66d4e561"
UUID_SUB="90675a7f-bf95-1514-4e54-e38ef61d5943" LABEL="Pangolin:0"
TYPE="linux_raid_member"
/dev/sdd1: UUID="dedc8d6c-8155-f88d-b035-c8cf66d4e561"
UUID_SUB="eaa50f1c-e1cd-d1a2-b10b-da77f2c0d5d7" LABEL="Pangolin:0"
TYPE="linux_raid_member"
/dev/sde1: UUID="dedc8d6c-8155-f88d-b035-c8cf66d4e561"
UUID_SUB="0b8210f9-cd73-a70d-bba7-01231bf9c578" LABEL="Pangolin:0"
TYPE="linux_raid_member"
/dev/sdf1: UUID="dedc8d6c-8155-f88d-b035-c8cf66d4e561"
UUID_SUB="fe58a109-7ec1-37d6-ee10-76197ffb4c67" LABEL="Pangolin:0"
TYPE="linux_raid_member"
/dev/sdg1: UUID="dedc8d6c-8155-f88d-b035-c8cf66d4e561"
UUID_SUB="726e7104-835c-adcc-32c4-952699b3b030" LABEL="Pangolin:0"
TYPE="linux_raid_member"
/dev/sdh1: UUID="dedc8d6c-8155-f88d-b035-c8cf66d4e561"
UUID_SUB="91095954-1368-ba5e-592a-b3d245904cb8" LABEL="Pangolin:0"
TYPE="linux_raid_member"
/dev/sdi1: UUID="d24a5068-14fb-4cec-831a-4b6d3bdba44d" TYPE="ext4"
/dev/sdi5: UUID="ab5617d1-58e1-4013-9690-64679982b2cf" TYPE="swap"
/dev/md127: UUID="S2atdF-aFUn-hmKm-yzsr-iqYW-DX3W-ioWjnl" TYPE="LVM2_member"
Carter J. Castor
^ permalink raw reply
* I/O error reading from raid 1 device but not slave devices
From: Nate Clark @ 2015-06-29 21:35 UTC (permalink / raw)
To: linux-raid
Hello,
I have encountered a strange error while reading from a raid 1 device.
If I read from the md device I encounter an I/O error, however if I
read from the underlying devices there is no issue.
-bash-4.3# dd if=/dev/md5 of=/dev/null bs=256K
dd: error reading ‘/dev/md5’: Input/output error
1007+1 records in
1007+1 records out
264134656 bytes (264 MB) copied, 1.86707 s, 141 MB/s
-bash-4.3# mdadm --detail -v /dev/md5
/dev/md5:
Version : 1.2
Creation Time : Mon Jun 29 22:34:37 2015
Raid Level : raid1
Array Size : 5238784 (5.00 GiB 5.36 GB)
Used Dev Size : 5238784 (5.00 GiB 5.36 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Tue Jun 30 05:05:43 2015
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Name : (none):5
UUID : 8933a60c:c34da7e7:f47bebfb:8b0ba6f6
Events : 864
Number Major Minor RaidDevice State
2 8 21 0 active sync /dev/sdb5
1 8 5 1 active sync /dev/sda5
-bash-4.3# dd if=/dev/sda5 of=/dev/null bs=256K
20480+0 records in
20480+0 records out
5368709120 bytes (5.4 GB) copied, 41.991 s, 128 MB/s
-bash-4.3# dd if=/dev/sdb5 of=/dev/null bs=256K
20480+0 records in
20480+0 records out
5368709120 bytes (5.4 GB) copied, 30.2417 s, 178 MB/s
I did perform a raid resync which completed successfully:
[20814.187596] md: requested-resync of RAID array md5
[20814.187602] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
[20814.187605] md: using maximum available idle IO bandwidth (but not
more than 200000 KB/sec) for requested-resync.
[20814.187612] md: using 128k window, over a total of 5238784k.
[20873.631782] md: md5: requested-resync done.
The only errors I can find in dmesg that seem to correlate to this are:
[22336.558454] Buffer I/O error on dev md5, logical block 64486, async page read
This system is running Fedora's 4.0.5-300 kernel with Neil's patch
"md: clear mddev->private when it has been freed."
I have not tried to reboot the system to see if that fixes the issue
since I didn't know how useful it would be for me to keep it in this
state if I can help with debugging. Another systems running the same
kernel with similar configuration does not seem to have any issue.
Also, the system with the issue only has it on one md device, which
happens to be /, but the other 7 raids work fine. This issue to be
some what rather rare.
Sorry, this email doesn't have too much useful data but I was not sure
what information was needed. Let me know if there is anything I can do
to provide more or better information.
Thanks,
-nate
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: I/O error reading from raid 1 device but not slave devices
From: Nate Clark @ 2015-06-30 14:46 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <CAG2YfWMfwtNkLGoswsZVSxpTm+BxdQD8e1ZB+Mc2+PxWrWSAYw@mail.gmail.com>
On Mon, Jun 29, 2015 at 5:35 PM, Nate Clark <nate@neworld.us> wrote:
> Hello,
>
> I have encountered a strange error while reading from a raid 1 device.
> If I read from the md device I encounter an I/O error, however if I
> read from the underlying devices there is no issue.
It appears both drives in the array have identical bad blocks list. I
am not sure why any blocks were marked bad since I don't see any drive
I/O errors in the logs and the smart output from each drive shows they
are healthy.
Thanks,
-nate
^ permalink raw reply
* Re: I/O error reading from raid 1 device but not slave devices
From: NeilBrown @ 2015-07-01 0:53 UTC (permalink / raw)
To: Nate Clark; +Cc: linux-raid
In-Reply-To: <CAG2YfWOn5_9KAK+qFY3XBBrh2L2HuxDCoc_6kzEEV8diyeG3Wg@mail.gmail.com>
On Tue, 30 Jun 2015 10:46:38 -0400 Nate Clark <nate@neworld.us> wrote:
> On Mon, Jun 29, 2015 at 5:35 PM, Nate Clark <nate@neworld.us> wrote:
> > Hello,
> >
> > I have encountered a strange error while reading from a raid 1 device.
> > If I read from the md device I encounter an I/O error, however if I
> > read from the underlying devices there is no issue.
>
> It appears both drives in the array have identical bad blocks list. I
> am not sure why any blocks were marked bad since I don't see any drive
> I/O errors in the logs and the smart output from each drive shows they
> are healthy.
That was my guess, but you confirmed before I got around to posting :-)
One way you could get bad blocks on perfectly healthy drives is if you
previously had an unhealthy drive.
Imagine a degraded RAID1 with a drive that has a couple of bad blocks.
You add a spare, it recovers but as it cannot read the bad blocks, it
adds them to the badblock list on the new device.
Then you remove the sick device, add a brand new one, and rebuild it -
from the first spare you added.
It will get the bad blocks "copied" onto it as well.
The blocks will stay 'bad' until something is written to them. Then
they will become good.
NeilBrown
^ permalink raw reply
* Re: Raid Recovery
From: NeilBrown @ 2015-07-01 1:01 UTC (permalink / raw)
To: Carter J. Castor; +Cc: linux-raid
In-Reply-To: <CADaa48=k9OE_MOmFgkdKZhkrUOfrRnJjEWeEVFfx6T7kQkRQ9w@mail.gmail.com>
On Mon, 29 Jun 2015 17:05:17 -0400 "Carter J. Castor"
<cjcastor@gmail.com> wrote:
> I have an ubuntu RAID 10 server using mdadm. It has six drives and a
> hot spare. The eSATA cables to the box holding the harddrives got
> disconnected from the server. When I plugged them back, it immediately
> saw only three of the drives and my hotspare and started rebuilding
> the array. Not wanting to screw it up, I let it finish, and then I
> rebooted. On start up, the array was inactive. When I tried to
> assemble it, it told me that there was only one disk(!?) so I tried to
> assemble it with the --force command. It then assembled, but the
> logical volume that I use to mount the RAID partition was gone. I have
> backups, but I'd like to save the data if at all possible. What's my
> best course of action from here?
>
> Here's the /proc/mdstat after rebooting:
> md127 : active (auto-read-only) raid10 sdg1[5] sdd1[1] sdh1[6] sde1[7]
> sdf1[4] sdc1[2]
> 2929890816 blocks super 1.2 256K chunks 2 far-copies [6/6] [UUUUUU]
>
>
> Blkid output:
>
> /dev/sda1: LABEL="KINGSTON" UUID="C3D4-FB34" TYPE="vfat"
> /dev/sdb1: UUID="dedc8d6c-8155-f88d-b035-c8cf66d4e561"
> UUID_SUB="f858b034-29ab-4d59-02d3-775b9b2139cd" LABEL="Pangolin:0"
> TYPE="linux_raid_member"
> /dev/sdc1: UUID="dedc8d6c-8155-f88d-b035-c8cf66d4e561"
> UUID_SUB="90675a7f-bf95-1514-4e54-e38ef61d5943" LABEL="Pangolin:0"
> TYPE="linux_raid_member"
> /dev/sdd1: UUID="dedc8d6c-8155-f88d-b035-c8cf66d4e561"
> UUID_SUB="eaa50f1c-e1cd-d1a2-b10b-da77f2c0d5d7" LABEL="Pangolin:0"
> TYPE="linux_raid_member"
> /dev/sde1: UUID="dedc8d6c-8155-f88d-b035-c8cf66d4e561"
> UUID_SUB="0b8210f9-cd73-a70d-bba7-01231bf9c578" LABEL="Pangolin:0"
> TYPE="linux_raid_member"
> /dev/sdf1: UUID="dedc8d6c-8155-f88d-b035-c8cf66d4e561"
> UUID_SUB="fe58a109-7ec1-37d6-ee10-76197ffb4c67" LABEL="Pangolin:0"
> TYPE="linux_raid_member"
> /dev/sdg1: UUID="dedc8d6c-8155-f88d-b035-c8cf66d4e561"
> UUID_SUB="726e7104-835c-adcc-32c4-952699b3b030" LABEL="Pangolin:0"
> TYPE="linux_raid_member"
> /dev/sdh1: UUID="dedc8d6c-8155-f88d-b035-c8cf66d4e561"
> UUID_SUB="91095954-1368-ba5e-592a-b3d245904cb8" LABEL="Pangolin:0"
> TYPE="linux_raid_member"
> /dev/sdi1: UUID="d24a5068-14fb-4cec-831a-4b6d3bdba44d" TYPE="ext4"
> /dev/sdi5: UUID="ab5617d1-58e1-4013-9690-64679982b2cf" TYPE="swap"
> /dev/md127: UUID="S2atdF-aFUn-hmKm-yzsr-iqYW-DX3W-ioWjnl" TYPE="LVM2_member"
>
"mdadm --examine" output of each drive is usually a good idea.
Also kernel logs from the time of the failure can help.
However blkid is reporting that "md127" is an "LVM2_member", which is
encouraging. Why do you think that the logical volume is gone?
What does
pvdisplay /dev/md127
report?
What about
pvck /dev/md127
??
NeilBrown
^ permalink raw reply
* Re: [PATCH] Fix read-balancing during node failure
From: NeilBrown @ 2015-07-01 2:24 UTC (permalink / raw)
To: Goldwyn Rodrigues; +Cc: teigland, gqJiang, linux-raid
In-Reply-To: <1435156232-8711-1-git-send-email-rgoldwyn@suse.com>
On Wed, 24 Jun 2015 09:30:32 -0500 Goldwyn Rodrigues
<rgoldwyn@suse.com> wrote:
> During a node failure, We need to suspend read balancing so that the
> reads are directed to the first device and stale data is not read.
> Suspending writes is not required because these would be recorded and
> synced eventually.
>
> A new flag MD_CLUSTER_SUSPEND_READ_BALANCING is set in recover_prep().
> area_resyncing() will respond true for the entire devices if this
> flag is set and the request type is READ. The flag is cleared
> in recover_done().
>
> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
> Reported-By: David Teigland <teigland@redhat.com>
> ---
> drivers/md/md-cluster.c | 12 +++++++++++-
> drivers/md/md-cluster.h | 2 +-
> drivers/md/raid1.c | 7 ++++---
> 3 files changed, 16 insertions(+), 5 deletions(-)
>
> diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
> index fcfc4b9..0072190 100644
> --- a/drivers/md/md-cluster.c
> +++ b/drivers/md/md-cluster.c
> @@ -44,6 +44,7 @@ struct resync_info {
>
> /* md_cluster_info flags */
> #define MD_CLUSTER_WAITING_FOR_NEWDISK 1
> +#define MD_CLUSTER_SUSPEND_READ_BALANCING 2
>
>
> struct md_cluster_info {
> @@ -275,6 +276,9 @@ clear_bit:
>
> static void recover_prep(void *arg)
> {
> + struct mddev *mddev = arg;
> + struct md_cluster_info *cinfo = mddev->cluster_info;
> + set_bit(MD_CLUSTER_SUSPEND_READ_BALANCING, &cinfo->state);
> }
>
> static void recover_slot(void *arg, struct dlm_slot *slot)
> @@ -307,6 +311,7 @@ static void recover_done(void *arg, struct dlm_slot *slots,
>
> cinfo->slot_number = our_slot;
> complete(&cinfo->completion);
> + clear_bit(MD_CLUSTER_SUSPEND_READ_BALANCING, &cinfo->state);
> }
>
> static const struct dlm_lockspace_ops md_ls_ops = {
> @@ -816,12 +821,17 @@ static void resync_finish(struct mddev *mddev)
> resync_send(mddev, RESYNCING, 0, 0);
> }
>
> -static int area_resyncing(struct mddev *mddev, sector_t lo, sector_t hi)
> +static int area_resyncing(struct mddev *mddev, int direction,
> + sector_t lo, sector_t hi)
> {
> struct md_cluster_info *cinfo = mddev->cluster_info;
> int ret = 0;
> struct suspend_info *s;
>
> + if ((direction == READ) &&
> + test_bit(MD_CLUSTER_SUSPEND_READ_BALANCING, &cinfo->state))
> + return 1;
> +
> spin_lock_irq(&cinfo->suspend_lock);
> if (list_empty(&cinfo->suspend_list))
> goto out;
> diff --git a/drivers/md/md-cluster.h b/drivers/md/md-cluster.h
> index 6817ee0..00defe2 100644
> --- a/drivers/md/md-cluster.h
> +++ b/drivers/md/md-cluster.h
> @@ -18,7 +18,7 @@ struct md_cluster_operations {
> int (*metadata_update_start)(struct mddev *mddev);
> int (*metadata_update_finish)(struct mddev *mddev);
> int (*metadata_update_cancel)(struct mddev *mddev);
> - int (*area_resyncing)(struct mddev *mddev, sector_t lo, sector_t hi);
> + int (*area_resyncing)(struct mddev *mddev, int direction, sector_t lo, sector_t hi);
> int (*add_new_disk_start)(struct mddev *mddev, struct md_rdev *rdev);
> int (*add_new_disk_finish)(struct mddev *mddev);
> int (*new_disk_ack)(struct mddev *mddev, bool ack);
> diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
> index 9157a29..b5626d2 100644
> --- a/drivers/md/raid1.c
> +++ b/drivers/md/raid1.c
> @@ -541,7 +541,7 @@ static int read_balance(struct r1conf *conf, struct r1bio *r1_bio, int *max_sect
>
> if ((conf->mddev->recovery_cp < this_sector + sectors) ||
> (mddev_is_clustered(conf->mddev) &&
> - md_cluster_ops->area_resyncing(conf->mddev, this_sector,
> + md_cluster_ops->area_resyncing(conf->mddev, READ, this_sector,
> this_sector + sectors)))
> choose_first = 1;
> else
> @@ -1111,7 +1111,8 @@ static void make_request(struct mddev *mddev, struct bio * bio)
> ((bio_end_sector(bio) > mddev->suspend_lo &&
> bio->bi_iter.bi_sector < mddev->suspend_hi) ||
> (mddev_is_clustered(mddev) &&
> - md_cluster_ops->area_resyncing(mddev, bio->bi_iter.bi_sector, bio_end_sector(bio))))) {
> + md_cluster_ops->area_resyncing(mddev, WRITE,
> + bio->bi_iter.bi_sector, bio_end_sector(bio))))) {
> /* As the suspend_* range is controlled by
> * userspace, we want an interruptible
> * wait.
> @@ -1124,7 +1125,7 @@ static void make_request(struct mddev *mddev, struct bio * bio)
> if (bio_end_sector(bio) <= mddev->suspend_lo ||
> bio->bi_iter.bi_sector >= mddev->suspend_hi ||
> (mddev_is_clustered(mddev) &&
> - !md_cluster_ops->area_resyncing(mddev,
> + !md_cluster_ops->area_resyncing(mddev, WRITE,
> bio->bi_iter.bi_sector, bio_end_sector(bio))))
> break;
> schedule();
Applied, thanks.
NeilBrown
^ permalink raw reply
* Re: DM_CACHE_SMQ is experimental and enabled by default? (was: Re: dm cache: add stochastic-multi-queue (smq) policy)
From: Geert Uytterhoeven @ 2015-07-01 7:25 UTC (permalink / raw)
To: Mike Snitzer
Cc: Joe Thornber, linux-raid, Linux Kernel Mailing List, dm-devel
In-Reply-To: <20150626160549.GA15434@redhat.com>
Hi Mike,
On Fri, Jun 26, 2015 at 6:05 PM, Mike Snitzer <snitzer@redhat.com> wrote:
> On Fri, Jun 26 2015 at 3:47am -0400,
> Geert Uytterhoeven <geert@linux-m68k.org> wrote:
>
>> On Fri, Jun 26, 2015 at 2:13 AM, Linux Kernel Mailing List
>> <linux-kernel@vger.kernel.org> wrote:
>> > Gitweb: http://git.kernel.org/linus/;a=commit;h=66a636356647a9be8885c2ce2948de126577698a
>> > Commit: 66a636356647a9be8885c2ce2948de126577698a
>> >
>> > dm cache: add stochastic-multi-queue (smq) policy
>>
>> > diff --git a/drivers/md/Kconfig b/drivers/md/Kconfig
>> > index edcf4ab..b597273 100644
>> > --- a/drivers/md/Kconfig
>> > +++ b/drivers/md/Kconfig
>> > @@ -304,6 +304,18 @@ config DM_CACHE_MQ
>> > This is meant to be a general purpose policy. It prioritises
>> > reads over writes.
>> >
>> > +config DM_CACHE_SMQ
>> > + tristate "Stochastic MQ Cache Policy (EXPERIMENTAL)"
>> > + depends on DM_CACHE
>> > + default y
>>
>> A feature cannot be "EXPERIMENTAL", and be enabled by default.
>> Please drop (at least) one of them.
>
> Thanks for your concern, but: no.
>
> If you look closer the entirety of DM cache is marked EXPERIMENTAL:
>
> config DM_CACHE
> tristate "Cache target (EXPERIMENTAL)"
I stand corrected. Sorry for the fuss.
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply
* (unknown),
From: Sasnett_Karen @ 2015-07-01 11:53 UTC (permalink / raw)
Haben Sie einen Investor brauchen?
Haben Sie geschäftliche oder persönliche Darlehen benötigen?
Wir geben Darlehen an eine natürliche Person und Unternehmen bei 3% Zinsen jährlich. Weitere Informationen Kontaktieren Sie uns per E-Mail: omfcreditspa@hotmail.com<mailto:omfcreditspa@hotmail.com>
HINWEIS: Leiten Sie Ihre Antwort nur an diese E-Mail: omfcreditspa@hotmail.com<mailto:omfcreditspa@hotmail.com>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox