* Re: [PATCH] [md] raid5: check faulty flag for array status during recovery.
From: NeilBrown @ 2015-02-19 21:51 UTC (permalink / raw)
To: Eric Mei; +Cc: linux-raid, eric.mei
In-Reply-To: <94BC57C5-6223-435B-96FD-7DA2F6B4E561@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 3665 bytes --]
On Tue, 6 Jan 2015 15:24:24 -0700 Eric Mei <meijia@gmail.com> wrote:
> Hi Neil,
>
> In a MDRAID derived work we found and fixed a data corruption bug. We think this also affect vanilla MDRAID, but we didn’t directly prove that by constructing a test to show the corruption. Following is the theoretical analysis, please kindly review and see if I missed something.
>
> To rebuild a stripe, MD checks whether array will be optimal after rebuild complete, if that’s true, we’ll mark the WIB bit to be cleared, the purpose is to enable “incremental rebuild”. The code section is like this:
>
> /* Need to check if array will still be degraded after recovery/resync
> * We don't need to check the 'failed' flag as when that gets set,
> * recovery aborts.
> */
> for (i = 0; i < conf->raid_disks; i++)
> if (conf->disks[i].rdev == NULL)
> still_degraded = 1;
>
> The problem is that only checking rdev == NULL might not be enough. Suppose both 2 drives D0 and D1 failed and marked as Faulty; We immediately removed D0 from array, but because some lingering IO on D1, it remains in array with Faulty flags on. A new drive pulled in, rebuild against D0 starts. Now because no rdev is NULL, MD thinks array will be optimal. If some writes happened before rebuild reaches the region, their dirty bits in WIB will be cleared. When later add D1 back into array, we’ll skip rebuilding those stripes, thus data corruption.
>
> The attached patch (against 3.18.0-rc6) is supposed to fix this issue.
>
> Thanks
> Eric
>
Hi Eric,
sorry for the delay, and thanks for the reminder...
The issue you described could only affect RAID6 as it requires the array to
continue with two failed drives.
However in the RAID6 case I think you are correct - there is a chance of
corruption if there is a double failure and a delay in removing one device.
Your patch isn't quite safe as conf->disks[i].rdev can become NULL at any
moment, so it could become NULL between testing and de-referencing.
So I've modified it as follows.
Thanks,
NeilBrown
Author: Eric Mei <eric.mei@seagate.com>
Date: Tue Jan 6 09:35:02 2015 -0800
raid5: check faulty flag for array status during recovery.
When we have more than 1 drive failure, it's possible we start
rebuild one drive while leaving another faulty drive in array.
To determine whether array will be optimal after building, current
code only check whether a drive is missing, which could potentially
lead to data corruption. This patch is to add checking Faulty flag.
Signed-off-by: NeilBrown <neilb@suse.de>
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index bc6d7595ad76..022a0d99e110 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -5120,12 +5120,17 @@ static inline sector_t sync_request(struct mddev *mddev, sector_t sector_nr, int
schedule_timeout_uninterruptible(1);
}
/* Need to check if array will still be degraded after recovery/resync
- * We don't need to check the 'failed' flag as when that gets set,
- * recovery aborts.
+ * Note in case of > 1 drive failures it's possible we're rebuilding
+ * one drive while leaving another faulty drive in array.
*/
- for (i = 0; i < conf->raid_disks; i++)
- if (conf->disks[i].rdev == NULL)
+ rcu_read_lock();
+ for (i = 0; i < conf->raid_disks; i++) {
+ struct md_rdev *rdev = ACCESS_ONCE(conf->disks[i].rdev);
+
+ if (rdev == NULL || test_bit(Faulty, &rdev->flags))
still_degraded = 1;
+ }
+ rcu_read_unlock();
bitmap_start_sync(mddev->bitmap, sector_nr, &sync_blocks, still_degraded);
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply related
* Re: mdadm raid 5 one disk overwritten file system failed
From: Wols Lists @ 2015-02-19 22:15 UTC (permalink / raw)
To: Adam Goryachev, John Andre Taule; +Cc: linux-raid
In-Reply-To: <54E5F5A7.3090609@websitemanagers.com.au>
On 19/02/15 14:39, Adam Goryachev wrote:
> From memory, there are filesystems which will do what you are asking
> (check that the data received from disk is correct, use multiple 'disks'
> and ensure protection from x failed drives, etc. I am certain zfs and
> btrfs both support this. (I've never used either due to stability
> concerns, but I read about them every now and then....)
When I used Pr1mes, I don't remember whether it was hardware or
software, but I believe their drives implemented some form of parity
check and recovery.
Basically, every eight-bit byte you wrote went to disk as sixteen bits -
a data byte and a parity byte. I don't know how it worked but (1) you
could reconstruct either byte from the other, and (2) for any 1-bit
error you could tell which of the data or parity bytes was corrupt. For
any 2-bit error I think you had a 90% chance of telling which byte was
corrupt - something like that anyway.
Of course, that's no use if your hacker feeds their corrupt stream
through your parity mechanism, or if 0x00000000 is valid when read from
disk.
Cheers,
Wol
^ permalink raw reply
* Re: [PATCH] [md] raid5: check faulty flag for array status during recovery.
From: Eric Mei @ 2015-02-19 22:49 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid, eric.mei
In-Reply-To: <20150220085147.03bb2247@notabene.brown>
Hi Neil, You are absolutely right we need RCU lock for this. Thank you
so much!
Eric
On 2015-02-19 2:51 PM, NeilBrown wrote:
> On Tue, 6 Jan 2015 15:24:24 -0700 Eric Mei <meijia@gmail.com> wrote:
>
>> Hi Neil,
>>
>> In a MDRAID derived work we found and fixed a data corruption bug. We think this also affect vanilla MDRAID, but we didn’t directly prove that by constructing a test to show the corruption. Following is the theoretical analysis, please kindly review and see if I missed something.
>>
>> To rebuild a stripe, MD checks whether array will be optimal after rebuild complete, if that’s true, we’ll mark the WIB bit to be cleared, the purpose is to enable “incremental rebuild”. The code section is like this:
>>
>> /* Need to check if array will still be degraded after recovery/resync
>> * We don't need to check the 'failed' flag as when that gets set,
>> * recovery aborts.
>> */
>> for (i = 0; i < conf->raid_disks; i++)
>> if (conf->disks[i].rdev == NULL)
>> still_degraded = 1;
>>
>> The problem is that only checking rdev == NULL might not be enough. Suppose both 2 drives D0 and D1 failed and marked as Faulty; We immediately removed D0 from array, but because some lingering IO on D1, it remains in array with Faulty flags on. A new drive pulled in, rebuild against D0 starts. Now because no rdev is NULL, MD thinks array will be optimal. If some writes happened before rebuild reaches the region, their dirty bits in WIB will be cleared. When later add D1 back into array, we’ll skip rebuilding those stripes, thus data corruption.
>>
>> The attached patch (against 3.18.0-rc6) is supposed to fix this issue.
>>
>> Thanks
>> Eric
>>
> Hi Eric,
> sorry for the delay, and thanks for the reminder...
>
> The issue you described could only affect RAID6 as it requires the array to
> continue with two failed drives.
>
> However in the RAID6 case I think you are correct - there is a chance of
> corruption if there is a double failure and a delay in removing one device.
>
> Your patch isn't quite safe as conf->disks[i].rdev can become NULL at any
> moment, so it could become NULL between testing and de-referencing.
> So I've modified it as follows.
>
> Thanks,
> NeilBrown
>
>
>
> Author: Eric Mei <eric.mei@seagate.com>
> Date: Tue Jan 6 09:35:02 2015 -0800
>
> raid5: check faulty flag for array status during recovery.
>
> When we have more than 1 drive failure, it's possible we start
> rebuild one drive while leaving another faulty drive in array.
> To determine whether array will be optimal after building, current
> code only check whether a drive is missing, which could potentially
> lead to data corruption. This patch is to add checking Faulty flag.
>
> Signed-off-by: NeilBrown <neilb@suse.de>
>
> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> index bc6d7595ad76..022a0d99e110 100644
> --- a/drivers/md/raid5.c
> +++ b/drivers/md/raid5.c
> @@ -5120,12 +5120,17 @@ static inline sector_t sync_request(struct mddev *mddev, sector_t sector_nr, int
> schedule_timeout_uninterruptible(1);
> }
> /* Need to check if array will still be degraded after recovery/resync
> - * We don't need to check the 'failed' flag as when that gets set,
> - * recovery aborts.
> + * Note in case of > 1 drive failures it's possible we're rebuilding
> + * one drive while leaving another faulty drive in array.
> */
> - for (i = 0; i < conf->raid_disks; i++)
> - if (conf->disks[i].rdev == NULL)
> + rcu_read_lock();
> + for (i = 0; i < conf->raid_disks; i++) {
> + struct md_rdev *rdev = ACCESS_ONCE(conf->disks[i].rdev);
> +
> + if (rdev == NULL || test_bit(Faulty, &rdev->flags))
> still_degraded = 1;
> + }
> + rcu_read_unlock();
>
> bitmap_start_sync(mddev->bitmap, sector_nr, &sync_blocks, still_degraded);
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: What are mdadm maintainers to do? (error recovery redundancy/data loss)
From: Roger Heflin @ 2015-02-20 5:12 UTC (permalink / raw)
To: Chris Murphy; +Cc: Linux RAID
In-Reply-To: <CAJCQCtQWBYRu7DFrdoKzj7G1-UKDJKBqyVn=6p91oagLfqd_Kw@mail.gmail.com>
On Thu, Feb 19, 2015 at 12:12 AM, Chris Murphy <lists@colorremedies.com> wrote:
> On Wed, Feb 18, 2015 at 4:04 AM, Chris <email.bug@arcor.de> wrote:
>>>
>>
>> Hello all,
>>
>
> On a single randomly selective drive, I disagree. In aggregate, that's
> true, eventually it will happen, you just won't know which drive or
> when it'll happen. I have a number of 5+ year old drives that have
> never reported a URE. Meanwhile another drive has so many bad sectors
> I only keep it around for abusive purposes.
And I have seen the same. Not all will fail even of a given type.
It also appears if one was really worried, running smartctl -t long often
(daily or weekly) can result in the disk finding and re-writing or moving
the bad sector. I have a disk that started given me trouble and the bad
block count has risen a few times without an os level error during the
-t long test.
>
>
>
>
> To get to one size fits all, where SCT ERC is disabled (consumer
> drive), and the kernel command timer is increased accordingly, we
> still need the delay reportable to user space. You can't have a by
> default 2-3 minute showstopper without an explanation so that the user
> can tune this back to 30 seconds or get rid of the drive or some other
> mitigation. Otherwise this is a 2-3 minute silent failure. I know a
> huge number of users who would assume this is a crash and force power
> off the system.
>
> The option where SCT ERC is configurable, you could also do this one
> size fits all by setting this to say 50-70 deciseconds, and for read
> failures to cause recovery if raid1+ is used, or cause a read retry
> if it's single, raid0, or linear. In other words, control the retries
> in software for these drives.
This gets more interesting. From what I can tell with my drivers (reds
and seagate video driver) they some allow erc to be set only 7 or higher,
and some allow things to be set lower. I have been setting mine lower
when it allows since I have raid 6 and expect to be able to get the data
from the other disks. This min 7 vs min of lower may be a further
distinction between the green(none), red 7, seagate VX (1.0 allowed).
My has video recordings...when the video pauses I counting how long.
I almost always appear to see the full 7 seconds, so I suspect that if
it does not recover in a short time it appears to be unlikely to recover it
all all. Given the data corruption issue without raid the vendors may
have the though that they cannot really do anything else but retry in the
no raid case.
>
>
>
> I can't agree at all, lacking facts, that this change is marginal for
> non-redundant configurations. I've seen no data how common long
> recovery incidents are, or how much more common data loss would be if
> long recovery were prevented.
>
> The mere fact they exist suggests they're necessary. It may very well
> be that the ECC code or hardware used is so slow that it really does
> take so unbelievably long (really 30 seconds is an eternity, and a
> minute seems outrageous, and 2-3 minutes seems wholly ridiculous as in
> worthy of brutal unrelenting ridicule); but that doesn't even matter
> even if it is true, that's the behavior of the ECC whether we like it
> or not, we can't just willy nilly turn these things off without
> understanding the consequences. Just saying it's marginal doesn't make
> it true.
>
> So if SCT ERC is short, now you have to have a mitigation for the
> possibly higher number of URE's this will result in, in the form of
> kernel instigated read retries on read fail. And in fact, this may be
> false. The retries the drive does internally might be completely
> different than the kernel doing another read. The way data is encoded
> on the drive these days bears no resemblance to discreet 1's and 0's.
Given the drive likely has some ability to adjust the levels of the 0 and 1,
I can see the disk retries possibly playing some games like that trying to get
a better answer. It is worth nothing that 7 seconds does mean around 70
retries of the read (data comes under the head 70 times). I doubt the ECC is
so slow it takes more than 10-20 ms to calculate more extreme failures. So
I am betting on the retries being what is recovering the data.
^ permalink raw reply
* rules and scripts (erc timeout fix)
From: email.bug @ 2015-02-20 16:54 UTC (permalink / raw)
To: linux-raid
Hello all,
enjoy, I tested the scripts set timeouts ok here, but I only have
drives that support erc timeouts (even if some have it disabled by default) none that
would really require setting a long controller timeout.
Cheers,
Chris
smartctl-timeouts README
The smartctl-timeouts scripts adjust the disk timeouts according to use-cases,
fixing common mismatching defaults that have often lead to data loss.
The scripts are to be called by udev rules during device initialization.
Every redundancy providing block device module may ship with proper udev rules
that initialize the timeouts for their possibly redundant devices.
The module may further adjust the actual status according to run-time changes.
NOTE: Correct execution during boot requires that distro package managers
hook smartctl and the smartctl-timeouts scripts into the initramfs.
RATIONALE
The error recovery (ERC) timout *must* be shorter than the controller timeout.
Otherwise read errors will cause controller resets, leading to direct data loss
or, if it is a redundant disk, loss of redundancy and a very high probability
of another read error and data loss when re-establishing the redundancy.
If a drive does not support adjusting its ERC timout, the controller timeout
must be increased above the drive's 'maximal error recovery time.
If you don't want that kind of long device timeout, you should look for a drive
with SCT ERC timout support. (smartctl -l scterc /dev/...)
IMPACT
If possible, the ERC timeout is adjusted to the controller timeout minus 5 seconds,
for all disks that contain possibly redundant data.
The controller timeout is only changed (raising it to LONG_CTRL_WAIT_SECONDS)
for drives without SCTERC support and entirely non-redundant-disks, to allow these
drives to properly finish their error recovery before a reset is triggerd.
Because controller timeouts are only increased selectively (only drives without SCTERC
support and surely non-redundant disks), the scripts won't change any timeouts in
professional, dedicated, redundant setups (e.g. storage servers etc.), except if
LONG_WAIT_ALL_NONREDUND_DISKS is configured to be true.
TODO
* non-redundant-partitions: conditional udev triggering, or a test in the script could
determine if all partions of the disk have been detected already and are all
non-redundant, to call non-redundant-disk in this case.
* parser to read ERC timout values?
- redundant-disk: a previously set "controller timeout - 5 seconds" ERC timeout
(possibly-redundant), could also be reset to 7 seconds, not just a "Disabled" value.
* If a redundancy controlling kernel module is to make dynamic adjustments,
"redundant-partition" needs implementation.
^ permalink raw reply
* Re: udev rules and scripts (erc timeout fix)
From: Chris @ 2015-02-22 10:23 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <B639B66A-F606-43CD-8FCC-D1A7810762D1@gmx.de>
[-- Attachment #1: Type: text/plain, Size: 24224 bytes --]
Hello all,
thanks for giving notice that the attachment didn't come through the
mailinglist.
The following is an improved version with its README, inserted inline.
It has changes to support configuring specific timeouts, and switching
between them.
Cheers,
Chris
--------
# smartctl-timeouts_defaults
# Defaults used by smartctl-timeouts scripts:
NONREDUNDANT_UNSURE_CONTROLLER_RESET_SECONDS="183"
# Should always be set. (e.g. 183)
# Used for disks without SCTERC support, to prevent too early
# resets. This long controller timeout value should be above
# the usual error recovery time of the harddrives without
# SCTERC support. Unfortunately, these values don't seem
# to be readable from the drives nor published.
NONREDUNDANT_UNSURE_RESET_ALL_DISKS=""
# If "true", ERC timout gets disabled for non-redundant disks
# an the value is used as the controller timeout.
# Can be set to "true" to try letting non-redundant disks fully
# complete their error recovery attempt.
# The configuration options below can be left blank or commmented
# out. This results in working with the hardware, kernel, or
# distribution defaults, and doing only necessary adaptions when
# initializing.
# But without configuring specific values, switching between
# the redundancy modes may not work well.
#NONREDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS="63"
# May be set to allow ample ERC time (e.g. 63).
# If blank the current timeout will not be changed, if possible.
# Note that the max. ERC timout is 99 seconds, so an exceeding
# controller timeout won't result in longer error correction
# attempts. Possibly use NONREDUNDANT_UNSURE_RESET_ALL_DISKS if your
# disk will do longer error correction attempts, if the ERC
# timeout is disabled.
#POSSIBLY_REDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS="48"
# May be set to allow some (-5s ) ERC timeout, yet not blocking
# redundant disks for too long.
# If blank the current setting will not be changed, if possible.
#REDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS="29"
# May be set to quickly reset blocking disks.
# If blank the current setting will not be changed, if possible.
TIMING_CMD="/usr/sbin/smartctl -l scterc"
set -o nounset -o errexit
-------
# do not edit this file, it will be overwritten on update
# Don't process any events if anaconda is running as anaconda brings up
# raid devices manually
ENV{ANACONDA}=="?*", GOTO="md_inc_end"
# assemble md arrays
SUBSYSTEM!="block", GOTO="md_inc_end"
# handle potential components of arrays (the ones supported by md)
ENV{ID_FS_TYPE}=="linux_raid_member", GOTO="md_inc"
# "noiswmd" on kernel command line stops mdadm from handling
# "isw" (aka IMSM - Intel RAID).
# "nodmraid" on kernel command line stops mdadm from handling
# "isw" or "ddf".
IMPORT{cmdline}="noiswmd"
IMPORT{cmdline}="nodmraid"
ENV{nodmraid}=="?*", GOTO="md_inc_end"
ENV{ID_FS_TYPE}=="ddf_raid_member", GOTO="md_inc"
ENV{noiswmd}=="?*", GOTO="md_inc_end"
ENV{ID_FS_TYPE}=="isw_raid_member", GOTO="md_inc"
GOTO="md_inc_end"
LABEL="md_inc"
# initialize redundancy possibility status
# (only the kernel module could set actual run-time state, and may in the future
# set a dynamic FASTFAIL kernel device property instead of calling smartctl-timeout scripts)
IMPORT{program}="BINDIR/mdadm --examine --export $tempnode"
ENV{MD_LEVEL}=="raid[1-9]*", ENV{REDUNDANT_DEV}="possibly"
ENV{MD_LEVEL}=="raid0", ENV{REDUNDANT_DEV}="false"
# remember you can limit what gets auto/incrementally assembled by
# mdadm.conf(5)'s 'AUTO' and selectively whitelist using 'ARRAY'
ACTION=="add|change", IMPORT{program}="BINDIR/mdadm --incremental --export $tempnode --offroot ${DEVLINKS}"
ACTION=="add|change", ENV{MD_STARTED}=="*unsafe*", ENV{MD_FOREIGN}=="no", ENV{SYSTEMD_WANTS}+="mdadm-last-resort@$env{MD_DEVICE}.timer"
ACTION=="remove", ENV{ID_PATH}=="?*", RUN+="BINDIR/mdadm -If $name --path $env{ID_PATH}"
ACTION=="remove", ENV{ID_PATH}!="?*", RUN+="BINDIR/mdadm -If $name"
LABEL="md_inc_end"
# initialize redundancy status for all surely non-redundant devices
# (The mdadm, btrfs, zfs, lvm, ... devices need too be adjusted by their own packages)
ENV{ID_FS_TYPE}!="linux_raid*|ddf_raid*|isw_raid*|lvm_*|LVM*|btrfs*|zfs*", ENV{REDUNDANT_DEV}="false"
# call initial HDD error correction timeouts adjustment
ENV{DEVTYPE}=="partition", ENV{REDUNDANT_DEV}=="possibly", TEST="/usr/sbin/smartctl", RUN+="BINDIR/smartctl-timeouts_possibly-redundant-partition.sh $parent"
ENV{DEVTYPE}=="partition", ENV{REDUNDANT_DEV}=="false", TEST="/usr/sbin/smartctl", RUN+="BINDIR/smartctl-timeouts_non-redundant-partition.sh $parent"
ENV{DEVTYPE}=="disk", ENV{REDUNDANT_DEV}=="possibly", TEST="/usr/sbin/smartctl", RUN+="BINDIR/smartctl-timeouts_posibly-redundant-disk.sh $devnode"
ENV{DEVTYPE}=="disk", ENV{REDUNDANT_DEV}=="false", TEST="/usr/sbin/smartctl", RUN+="BINDIR/smartctl-timeouts_non-redundant-disk.sh $devnode"
------
#!/bin/sh
# smartctl-timeouts_possibly-redundant-disk.sh
SCRIPT_DIR="$(dirname "$(readlink -f "$0")")"
. $SCRIPT_DIR/smartctl-timeouts_defaults
HDD_DEV="$1"
echo "Adjusting $HDD_DEV timeouts:"
if ! ${TIMING_CMD} /dev/${HDD_DEV} | grep -q Disabled \
&& ! ${TIMING_CMD} /dev/${HDD_DEV} | grep -q seconds
then
# ERC timeout is not supported (not disabled and not set):
# * Set the controller timeout to be considerably loooooong.
# - To allow the drive to give up its ERC attempts by itself.
# - Let the drive return a proper read error, so that the redundancy
# provider (md, lvm, btrfs, ...) can re-write the bad block.
# - Disk read errors thus result in long i/o blocking periods with
# no error messages that may not be watched by or reported to the user,
# - but waiting this long should prevent unecessary controller resets of the
# entire drive and the corresponding loss of redundancy/data.
echo "Drive without ERC timeout support, setting NONREDUNDANT_UNSURE_CONTROLLER_RESET_SECONDS (${NONREDUNDANT_UNSURE_CONTROLLER_RESET_SECONDS}s)"
echo ${NONREDUNDANT_UNSURE_CONTROLLER_RESET_SECONDS} >/sys/block/${HDD_DEV}/device/timeout
else
SWITCH_FROM_OTHER_CONFIGURED_SMARTCTL_TIMEOUT="false"
# reset controller timeout, if a configured value was previously set
if [ `cat /sys/block/${HDD_DEV}/device/timeout` = ${NONREDUNDANT_UNSURE_CONTROLLER_RESET_SECONDS:--1} ] \
|| [ `cat /sys/block/${HDD_DEV}/device/timeout` = ${NONREDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS:--1} ] \
|| [ `cat /sys/block/${HDD_DEV}/device/timeout` = ${REDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS:--1} ]
then
SWITCH_FROM_OTHER_CONFIGURED_SMARTCTL_TIMEOUT="true"
echo "resetting controller from another configured value (`cat /sys/block/${HDD_DEV}/device/timeout`s) to ${POSSIBLY_REDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS:-30}s"
echo ${POSSIBLY_REDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS:-30} >/sys/block/${HDD_DEV}/device/timeout
else
# set possibly-redundant timeout anyway, if configured
if [ ${POSSIBLY_REDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS:-undefined} != "undefined" ] ; then
echo "setting controller timeout to POSSIBLY_REDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS (${POSSIBLY_REDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS}s)"
echo ${POSSIBLY_REDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS} >/sys/block/${HDD_DEV}/device/timeout
fi
fi
if ${TIMING_CMD} /dev/${HDD_DEV} | grep -q Disabled \
|| [ $SWITCH_FROM_OTHER_CONFIGURED_SMARTCTL_TIMEOUT = "true" ] \
|| [ ${POSSIBLY_REDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS:-undefined} != "undefined" ]
then
# ERC timeout is disabled or configured:
# * set it to controller timeout -5 seconds
# - Allows redundancy provider to read data from another disk and re-write the bad block
# before the controller resets the entire drive and the raid looses redundancy/data completely.
# - Longer than the usual 7s default of dedicated raid drives, to allow as much
# ERC time as possible (good if degraded and for non-redundant partitions on same drive).
ERC_TENTHS=$(expr `cat /sys/block/${HDD_DEV}/device/timeout` \* 10 - 50)
# prevent exceeding max. scterc value
if [ $ERC_TENTHS -gt 999 ] ; then
ERC_TENTHS="999"
fi
echo "maximizing ERC timeout to controller timeout -5 seconds (`expr $ERC_TENTHS / 10`s)"
${TIMING_CMD},$ERC_TENTHS,$ERC_TENTHS /dev/${HDD_DEV} > /dev/null
fi
fi
----------
#! /bin/sh
# smartctl-timeouts_possibly-redundant-partition.sh
# This script sets the timeouts for "mixed drives" that contain redundant
# and non-redundant partitions.
# A single, possibly-redundant partition is enough to set the entire drive's
# timeouts to possibly-redundant settings (with a determined ERC timeout slightly
# below the default or configured controller timout, if possible).
#
# This avoids to risk unknown disk recovery times and needing a very long
# controller timeouts. Where configuring such a ERC timout is possible,
# this means the disk recovery may be terminated quicker than the drive
# would without the timout set, but it ensures that there will be no resets
# leading to data loss and redundancy loss.
SCRIPT_DIR="$(dirname "$(readlink -f "$0")")"
$SCRIPT_DIR/smartctl-timeouts_possibly-redundant-disk.sh $1
--------------
#!/bin/sh
# smartctl-timeouts_non-redundant-disk.sh
SCRIPT_DIR="$(dirname "$(readlink -f "$0")")"
. $SCRIPT_DIR/smartctl-timeouts_defaults
HDD_DEV="$1"
echo "Adjusting $HDD_DEV timeouts:"
if [ ${NONREDUNDANT_UNSURE_RESET_ALL_DISKS:-false} = "true" ] ; then
# * disable any ERC timeout
# - Allows the drive to do ERC without imposing a timeout.
${TIMING_CMD},0,0 /dev/${HDD_DEV} > /dev/null
# * Set the controller timeout to be considerably loooooong.
# - To allow the drive to give up its ERC attempts by itself.
# - Let the drive return a proper read error, so that the redundancy
# provider (md, lvm, btrfs, ...) can re-write the bad block.
# - Disk read errors thus result in long i/o blocking periods with
# no error messages that may not be watched by or reported to the user,
# - but waiting this long should prevent unecessary controller resets of the
# entire drive and the corresponding loss of redundancy/data.
echo "NONREDUNDANT_UNSURE_RESET_ALL_DISKS is true, setting NONREDUNDANT_UNSURE_CONTROLLER_RESET_SECONDS (${NONREDUNDANT_UNSURE_CONTROLLER_RESET_SECONDS}s)"
echo ${NONREDUNDANT_UNSURE_CONTROLLER_RESET_SECONDS} >/sys/block/${HDD_DEV}/device/timeout
else
if ! ${TIMING_CMD} /dev/${HDD_DEV} | grep -q Disabled \
&& ! ${TIMING_CMD} /dev/${HDD_DEV} | grep -q seconds
then
# ERC timeout is not supported (not disabled and not set)
# * Set the controller timeout to be considerably loooooong.
echo "Drive without ERC timeout support, setting NONREDUNDANT_UNSURE_CONTROLLER_RESET_SECONDS (${NONREDUNDANT_UNSURE_CONTROLLER_RESET_SECONDS}s)"
echo ${NONREDUNDANT_UNSURE_CONTROLLER_RESET_SECONDS} >/sys/block/${HDD_DEV}/device/timeout
else
if ${TIMING_CMD} /dev/${HDD_DEV} | grep -q seconds \
|| [ ${NONREDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS:-undefined} != "undefined" ]
then
# reset controller timeout, if a configured value was previously set
if [ `cat /sys/block/${HDD_DEV}/device/timeout` = ${NONREDUNDANT_UNSURE_CONTROLLER_RESET_SECONDS:--1} ] \
|| [ `cat /sys/block/${HDD_DEV}/device/timeout` = ${POSSIBLY_REDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS:--1} ] \
|| [ `cat /sys/block/${HDD_DEV}/device/timeout` = ${REDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS:--1} ]
then
echo "resetting controller from another configured value (`cat /sys/block/${HDD_DEV}/device/timeout`s) to ${NONREDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS:-60}s"
echo ${NONREDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS:-60} >/sys/block/${HDD_DEV}/device/timeout
else
# set non-redundant timeout anyway, if configured
if [ ${NONREDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS:-undefined} != "undefined" ] ; then
echo "setting configured NONREDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS (${NONREDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS}s)"
echo ${NONREDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS} >/sys/block/${HDD_DEV}/device/timeout
fi
fi
# An ERC timeout is set or configured:
# * change ERC timout to controller timeout -5 seconds
# - Longer than the usual 7s default of dedicated raid drives, to allow as much
# ERC time as possible.
ERC_TENTHS=$(expr `cat /sys/block/${HDD_DEV}/device/timeout` \* 10 - 50)
# prevent exceeding max. scterc value
if [ $ERC_TENTHS -gt 999 ] ; then
ERC_TENTHS="999"
fi
echo "maximizing ERC timeout to controller timeout -5 seconds (`expr $ERC_TENTHS / 10`s)"
${TIMING_CMD},$ERC_TENTHS,$ERC_TENTHS /dev/${HDD_DEV} > /dev/null
else # ERC timeout disabled
echo "found ERC timeout disabled, setting NONREDUNDANT_UNSURE_CONTROLLER_RESET_SECONDS (${NONREDUNDANT_UNSURE_CONTROLLER_RESET_SECONDS}s)"
echo ${NONREDUNDANT_UNSURE_CONTROLLER_RESET_SECONDS} >/sys/block/${HDD_DEV}/device/timeout
fi
fi
fi
--------------
#!/bin/sh
# smartctl-timeouts_non-redundant-partition.sh
# Because there may also be redundant partitions on this disk we must not
# unconditionally alter the timeouts.
SCRIPT_DIR="$(dirname "$(readlink -f "$0")")"
. $SCRIPT_DIR/smartctl-timeouts_defaults
HDD_DEV="$1"
REDUNDANT_DISK="unchecked"
# TODO
#for dev in `cd /sys/block/${HDD_DEV} ; ls -d ${HDD_DEV}*` ; do
# if equvalent to udev's ENV{REDUNDANT_DEV}=="yes|possibly"; then
# $REDUNDANT_DISK="possibly"
# fi
#done
#if [ $REDUNDANT_DISK="unchecked" ] ; then
# REDUNDANT_DISK="false"
#fi
if [ $REDUNDANT_DISK = "false" ] \
# TODO && all partitions have been detected by udev already
then
$SCRIPT_DIR/smartctl-timeouts_non-redundant-disk.sh $1
else
$SCRIPT_DIR/smartctl-timeouts_possibly-redundant-disk.sh $1
fi
-------------
#!/bin/sh
# smartctl-timeouts_redundant-disk.sh
# Redundant timouts are NEVER to be triggerd by udev rules!
# Because only the redundancy providing kernel module knows the actual run-time
# redundancy status, can adjust it and call this script dynamically.
# Udev rules can only determine "possibly redundant" devices.
SCRIPT_DIR="$(dirname "$(readlink -f "$0")")"
. $SCRIPT_DIR/smartctl-timeouts_defaults
HDD_DEV="$1"
echo "Adjusting $HDD_DEV timeouts:"
if ! ${TIMING_CMD} /dev/${HDD_DEV} | grep -q Disabled \
&& ! ${TIMING_CMD} /dev/${HDD_DEV} | grep -q seconds
then
# ERC timeout is not supported (not disabled and not set):
# * Set the controller timeout to be considerably loooooong.
# - To allow the drive to give up its ERC attempts by itself.
# - Let the drive return a proper read error, so that the redundancy
# provider (md, lvm, btrfs, ...) can re-write the bad block.
# - Disk read errors thus result in long i/o blocking periods with
# no error messages that may not be watched by or reported to the user,
# - but waiting this long should prevent unecessary controller resets of the
# entire drive and the corresponding loss of redundancy/data.
echo "Drive without ERC timeout support, setting NONREDUNDANT_UNSURE_CONTROLLER_RESET_SECONDS (${NONREDUNDANT_UNSURE_CONTROLLER_RESET_SECONDS}s)"
echo ${NONREDUNDANT_UNSURE_CONTROLLER_RESET_SECONDS} >/sys/block/${HDD_DEV}/device/timeout
else
SWITCH_FROM_OTHER_CONFIGURED_SMARTCTL_TIMEOUT="false"
# reset controller timeout, if a configured value was previously set
if [ `cat /sys/block/${HDD_DEV}/device/timeout` = ${NONREDUNDANT_UNSURE_CONTROLLER_RESET_SECONDS:--1} ] \
|| [ `cat /sys/block/${HDD_DEV}/device/timeout` = ${NONREDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS:--1} ] \
|| [ `cat /sys/block/${HDD_DEV}/device/timeout` = ${POSSIBLY_REDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS:--1} ]
then
SWITCH_FROM_OTHER_CONFIGURED_SMARTCTL_TIMEOUT="true"
echo "resetting controller from another configured value (`cat /sys/block/${HDD_DEV}/device/timeout`s) to ${REDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS:-30}s"
echo ${REDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS:-30} >/sys/block/${HDD_DEV}/device/timeout
else
# set possibly-redundant timeout anyway, if configured
if [ ${REDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS:-undefined} != "undefined" ] ; then
echo "setting controller timeout to REDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS (${REDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS}s)"
echo ${REDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS} >/sys/block/${HDD_DEV}/device/timeout
else
if [ `cat /sys/block/${HDD_DEV}/device/timeout` -gt 30 ] ; then
echo "reducing controller timout to 30 seconds"
echo 30 >/sys/block/${HDD_DEV}/device/timeout
fi
fi
fi
if ${TIMING_CMD} /dev/${HDD_DEV} | grep -q Disabled \
|| [ $SWITCH_FROM_OTHER_CONFIGURED_SMARTCTL_TIMEOUT = "true" ] \
|| [ ${REDUNDANT_DISK_CONTROLLER_TIMEOUT_SECONDS:-undefined} != "undefined" ] \
# TODO: || [ $(expr `cat /sys/block/${HDD_DEV}/device/timeout` \* 10 - 50) = read of current ERC timeout value ]
then
# ERC timeout is disabled, is configured, or has been "maximized" to the controller timeout -5 seconds:
# * set it to 7 seconds
# - The usual quick 7s default of dedicated raid drives.
# - Allows redundancy provider to quickly read data from another disk and re-write the bad block
# before the controller resets the entire drive and the raid looses redundancy/data completely.
echo "setting ERC timeout to 7 seconds"
${TIMING_CMD},70,70 /dev/${HDD_DEV} > /dev/null
fi
fi
------
#!/bin/sh
# smartctl-timeouts_redundant-partition.sh
# Redundant timouts are NEVER to be triggerd by udev rules!
# Because only the redundancy providing kernel module knows the actual run-time
# redundancy status, can adjust it and call this script dynamically.
# Udev rules can only determine "possibly redundant" devices.
SCRIPT_DIR="$(dirname "$(readlink -f "$0")")"
. $SCRIPT_DIR/smartctl-timeouts_defaults
HDD_DEV="$1"
REDUNDANT_DISK="unchecked"
# TODO
#for dev in `cd /sys/block/${HDD_DEV} ; ls -d ${HDD_DEV}*` ; do
# if equvalent to udev's ENV{REDUNDANT_DEV}=="false"; then
# $REDUNDANT_DISK="false"
# fi
#done
#if [ $REDUNDANT_DISK="unchecked" ] ; then
# REDUNDANT_DISK="true"
#fi
if [ $REDUNDANT_DISK = "true" ] \
# TODO: && all partitions have been detected by udev already
then
$SCRIPT_DIR/smartctl-timeouts_redundant-disk.sh $1
else
$SCRIPT_DIR/smartctl-timeouts_possibly-redundant-disk.sh $1
fi
---------
smartctl-timeouts README
The smartctl-timeouts scripts adjust controller and disk timeouts according
to redundancy status, and fix commonly mismatching defaults with drives that
have no error recovery timeout configured, which has often lead to data loss.
The scripts are to be called by udev rules during device initialization,
and by kernel modules acording to the run-time redundancy status changes.
Every redundancy providing block device module may ship with proper udev rules
that initialize the timeouts for their possibly redundant devices.
An alternative to these scripts may be to investigate the FASTFAIL
feature in the kernel.
NOTE: Correct execution during boot requires that distro package managers
hook smartctl and the smartctl-timeouts scripts into the initramfs.
RATIONALE
The error recovery (ERC) timeout *must* be shorter than the controller timeout.
Otherwise read errors will cause controller resets, leading to direct data loss
or, if it is a redundant disk, loss of redundancy and a very high probability
of another read error and data loss when re-establishing the redundancy.
If a drive does not support adjusting its ERC timeout, the controller timeout
must be increased above the drive's maximal error recovery time.
If you don't want that kind of long device timeout, you should look for a drive
with SCT ERC timeout support. (smartctl -l scterc /dev/...)
IMPACT (without having specific timeouts configured)
For possibly redundant disks: If supported but simply disabled in the drive,
the ERC timeout is adjusted to the current controller timeout minus 5 seconds.
The controller timeout is only raised (to NONREDUNDANT_UNSURE_CONTROLLER_RESET_SECONDS)
for drives without SCTERC support. As well as for entirely non-redundant-disks,
in an attempt to allow these drives to finish their error recovery regularily
before a reset is triggerd.
As controller timeouts are only increased selectively (only drives without SCTERC
support and surely non-redundant disks), the scripts only adapt mismatching
timeouts, by default. Existing manufacturer or custom ERC timeout settings (as in
professional, dedicated, redundant setups, e.g. storage servers etc.) won't be
changed, except with specific configuration options.
TODO
* non-redundant-partitions: conditional udev triggering, or a test in the script could
determine if all partions of the disk have been detected already and are all
non-redundant, to call non-redundant-disk in this case.
* parser to read ERC timeout values?
- redundant-disk: a previously set "controller timeout - 5 seconds" ERC timeout
(possibly-redundant), could also be reset to 7 seconds, not just a "Disabled" value.
* If a redundancy controlling kernel module is to make dynamic adjustments,
"redundant-partition" needs implementation.
smartctl-timeouts README
The smartctl-timeouts scripts adjust controller and disk timeouts according
to redundancy status, and fix commonly mismatching defaults with drives that
have no error recovery timeout configured, which has often lead to data loss.
The scripts are to be called by udev rules during device initialization,
and by kernel modules acording to the run-time redundancy status changes.
Every redundancy providing block device module may ship with proper udev rules
that initialize the timeouts for their possibly redundant devices.
An alternative to these scripts may be to investigate the FASTFAIL
feature in the kernel.
NOTE: Correct execution during boot requires that distro package managers
hook smartctl and the smartctl-timeouts scripts into the initramfs.
RATIONALE
The error recovery (ERC) timeout *must* be shorter than the controller timeout.
Otherwise read errors will cause controller resets, leading to direct data loss
or, if it is a redundant disk, loss of redundancy and a very high probability
of another read error and data loss when re-establishing the redundancy.
If a drive does not support adjusting its ERC timeout, the controller timeout
must be increased above the drive's maximal error recovery time.
If you don't want that kind of long device timeout, you should look for a drive
with SCT ERC timeout support. (smartctl -l scterc /dev/...)
IMPACT (without having specific timeouts configured)
For possibly redundant disks: If supported but simply disabled in the drive,
the ERC timeout is adjusted to the current controller timeout minus 5 seconds.
The controller timeout is only raised (to NONREDUNDANT_UNSURE_CONTROLLER_RESET_SECONDS)
for drives without SCTERC support. As well as for entirely non-redundant-disks,
in an attempt to allow these drives to finish their error recovery regularily
before a reset is triggerd.
As controller timeouts are only increased selectively (only drives without SCTERC
support and surely non-redundant disks), the scripts only adapt mismatching
timeouts, by default. Existing manufacturer or custom ERC timeout settings (as in
professional, dedicated, redundant setups, e.g. storage servers etc.) won't be
changed, except with specific configuration options.
TODO
* non-redundant-partitions: conditional udev triggering, or a test in the script could
determine if all partions of the disk have been detected already and are all
non-redundant, to call non-redundant-disk in this case.
* parser to read ERC timeout values?
- redundant-disk: a previously set "controller timeout - 5 seconds" ERC timeout
(possibly-redundant), could also be reset to 7 seconds, not just a "Disabled" value.
* If a redundancy controlling kernel module is to make dynamic adjustments,
"redundant-partition" needs implementation.
[-- Attachment #2: smartctl-timeouts_email2.zip --]
[-- Type: application/zip, Size: 9916 bytes --]
^ permalink raw reply
* Optimal chunk size for RAID5?
From: Christer Solskogen @ 2015-02-22 11:31 UTC (permalink / raw)
To: linux-raid
Hi!
I'm about to create a RAID5 with three 4TB disks (ST4000VN000
from Seagate) and I wonder what the optimal chunk size is/should be.
There are so many different views on the internet that I've almost lost
faith in the internets.
Is there even such a thing as optimal chunk size?
Are there any other stuff I should think about?
--
chs
^ permalink raw reply
* Re: Optimal chunk size for RAID5?
From: Roman Mamedov @ 2015-02-22 12:30 UTC (permalink / raw)
To: Christer Solskogen; +Cc: linux-raid
In-Reply-To: <mccemd$8pi$1@ger.gmane.org>
On Sun, 22 Feb 2015 12:31:23 +0100
Christer Solskogen <christer.solskogen@gmail.com> wrote:
> There are so many different views on the internet
...and yet you're asking for some more? :)
> Is there even such a thing as optimal chunk size?
64K should be fine:
http://louwrentius.com/linux-raid-level-and-chunk-size-the-benchmarks.html
--
With respect,
Roman
^ permalink raw reply
* Re: Optimal chunk size for RAID5?
From: Christer Solskogen @ 2015-02-22 12:46 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <20150222173047.1ea65d67@natsu>
On 22.02.2015 13:30, Roman Mamedov wrote:
> On Sun, 22 Feb 2015 12:31:23 +0100
> Christer Solskogen <christer.solskogen@gmail.com> wrote:
>
>> There are so many different views on the internet
>
> ...and yet you're asking for some more? :)
>
Heh, yes. I see the irony in that :-) But this is pretty much as the
source as you can get.
>> Is there even such a thing as optimal chunk size?
>
> 64K should be fine:
> http://louwrentius.com/linux-raid-level-and-chunk-size-the-benchmarks.html
>
Okay, even if this is almost 5 years old, it is still applicable?
The disks I have are 4k sectors. Is there anything special I need to
think about or is this handled automagicly these days? (I run a pretty
fresh distro)
--
chs
^ permalink raw reply
* Re: re-add POLICY
From: Chris @ 2015-02-22 13:23 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <loom.20150217T152416-500@post.gmane.org>
Hello,
I just noticed that I somehow overlooked that md3 and md7 on that old ubuntu
system *did* have a write-intent bitmap.
So in my tests action="spare" does not seem to allow automatic re-sync of
arrays without a bitmap.
To quote the man page again on "spare":
"if the device is bare it can become a spare if there is any array that it
is a candidate for based on domains and metadata."
I am fankly not sure I fully understand that. A bare device has no
superblock, so does mdadm only look which array fits onto the device?
Since the partitions on the removed disk contain superblocks they are not
bare, may that by why action=spare does not apply, and an automatic re-sync
may either require a new action="re-sync" or be done by "re-add" as well?
Regards,
Chris
^ permalink raw reply
* Re: Optimal chunk size for RAID5?
From: Alireza Haghdoost @ 2015-02-22 14:33 UTC (permalink / raw)
To: Roman Mamedov; +Cc: Christer Solskogen, Linux RAID
In-Reply-To: <20150222173047.1ea65d67@natsu>
On Sun, Feb 22, 2015 at 6:30 AM, Roman Mamedov <rm@romanrm.net> wrote:
> On Sun, 22 Feb 2015 12:31:23 +0100
> Christer Solskogen <christer.solskogen@gmail.com> wrote:
>
>> There are so many different views on the internet
>
> ...and yet you're asking for some more? :)
>
>> Is there even such a thing as optimal chunk size?
>
> 64K should be fine:
> http://louwrentius.com/linux-raid-level-and-chunk-size-the-benchmarks.html
>
I have seen that people report 64K chunk size results better
performance. However, I was not able to find why mdadm maintainers
decided to switch into 512K default chunk size a few years ago ? Was
that decision related to the write-intent bitmap overhead ?
^ permalink raw reply
* Metadata > 0.90 and auto-assemble
From: Joshua Kinard @ 2015-02-22 18:54 UTC (permalink / raw)
To: linux-raid
Hi,
I tried a while back to use the newer metadata formats on my mdadm RAID5 on a
few machines, and discovered that the kernel auto-assembly will only work with
v0.90 metadata, not 1.0 or greater. Is there a solid reason for this? Based
one what I can find regarding the differences in the metadata formats, and
looking at the existing md code, it seems this is largely just because no one
has had the time or motivation to change the code to support auto-assembly on
the newer metdata formats.
I am told that the "correct" solution is to embed a small initramfs to bring
the RAID arrays online instead, before the real rootfs is loaded. I'd like to
avoid this if possible, as I haven't had to use an initramfs for normal booting
in the past, as long as I stay on metadata 0.90. So I thought I'd ask what the
official stance is on this.
Thanks!,
--
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
4096R/D25D95E3 2011-03-28
"The past tempts us, the present confuses us, the future frightens us. And our
lives slip away, moment by moment, lost in that vast, terrible in-between."
--Emperor Turhan, Centauri Republic
^ permalink raw reply
* Re: Metadata > 0.90 and auto-assemble
From: Chris Murphy @ 2015-02-22 21:29 UTC (permalink / raw)
Cc: linux-raid
In-Reply-To: <54EA25D3.8020300@gentoo.org>
On Sun, Feb 22, 2015 at 11:54 AM, Joshua Kinard <kumba@gentoo.org> wrote:
> I'd like to
> avoid this if possible, as I haven't had to use an initramfs for normal booting
> in the past, as long as I stay on metadata 0.90. So I thought I'd ask what the
> official stance is on this.
https://raid.wiki.kernel.org/index.php/Autodetect
Official stance is that it's deprecated, but people still use it.
--
Chris Murphy
^ permalink raw reply
* Re: Optimal chunk size for RAID5?
From: NeilBrown @ 2015-02-22 21:53 UTC (permalink / raw)
To: Alireza Haghdoost; +Cc: Roman Mamedov, Christer Solskogen, Linux RAID
In-Reply-To: <CAB-428=7y-KSbCeXo6y6o5Jfzgf-5h6pfYxY+4Z74bqYp=oMUg@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1266 bytes --]
On Sun, 22 Feb 2015 08:33:02 -0600 Alireza Haghdoost <alireza@cs.umn.edu>
wrote:
> On Sun, Feb 22, 2015 at 6:30 AM, Roman Mamedov <rm@romanrm.net> wrote:
> > On Sun, 22 Feb 2015 12:31:23 +0100
> > Christer Solskogen <christer.solskogen@gmail.com> wrote:
> >
> >> There are so many different views on the internet
> >
> > ...and yet you're asking for some more? :)
> >
> >> Is there even such a thing as optimal chunk size?
> >
> > 64K should be fine:
> > http://louwrentius.com/linux-raid-level-and-chunk-size-the-benchmarks.html
> >
>
> I have seen that people report 64K chunk size results better
> performance. However, I was not able to find why mdadm maintainers
> decided to switch into 512K default chunk size a few years ago ? Was
> that decision related to the write-intent bitmap overhead ?
No, write-intent-bitmap sizing is completely independent from chunk sizes.
I don't remember the detail for the change, but some measurement must have
gone faster with larger chunk size.
single threaded loads tend to prefer large chunk sizes.
multi-threaded small-request random IO tends to prefer smaller chunk sizes.
There is no "Optimal" without reference to a particular work load. Or
particular hardware.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Metadata > 0.90 and auto-assemble
From: Joshua Kinard @ 2015-02-22 21:54 UTC (permalink / raw)
To: Chris Murphy; +Cc: linux-raid
In-Reply-To: <CAJCQCtSPPm3ipb49SCeNFvgQ21y4HseqovBPg28LKXzvP5PY8Q@mail.gmail.com>
On 02/22/2015 16:29, Chris Murphy wrote:
> On Sun, Feb 22, 2015 at 11:54 AM, Joshua Kinard <kumba@gentoo.org> wrote:
>> I'd like to
>> avoid this if possible, as I haven't had to use an initramfs for normal booting
>> in the past, as long as I stay on metadata 0.90. So I thought I'd ask what the
>> official stance is on this.
>
> https://raid.wiki.kernel.org/index.php/Autodetect
>
> Official stance is that it's deprecated, but people still use it.
Yeah, but it's a pretty useful feature. I can't see why autodetect for simple
setups (several disks or partitions and building a basic array out of them) is
maintained, while userspace autodetect is required for the more complex setups.
But I suppose this has been discussed before in detail, though I cannot find
said discussion. The RAID Boot page has this one example only:
"This approach can cause problems in several situations (imagine moving part of
an old array onto another machine before wiping and repurposing it: reboot and
watch in horror as the piece of dead array gets assembled as part of the
running RAID array, ruining it); kernel autodetect is correspondingly deprecated."
Which I find to be rather unconvincing. The cited example is a fault of the
user not torching the superblock before moving the disks or trying to use
them...and I've done this to myself on several occasions. mdadm --misc
--zero-superblock and 'dd' saved the day in less than ~30s.
Are there any other discussions that might be more convincing, or offer up
other points of view? Perhaps there's a point I've yet to consider that might
be enlightening.
Thanks!,
--
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
4096R/D25D95E3 2011-03-28
"The past tempts us, the present confuses us, the future frightens us. And our
lives slip away, moment by moment, lost in that vast, terrible in-between."
--Emperor Turhan, Centauri Republic
^ permalink raw reply
* Re: Metadata > 0.90 and auto-assemble
From: Mark Knecht @ 2015-02-22 22:19 UTC (permalink / raw)
To: Joshua Kinard; +Cc: Linux-RAID
In-Reply-To: <54EA25D3.8020300@gentoo.org>
On Sun, Feb 22, 2015 at 10:54 AM, Joshua Kinard <kumba@gentoo.org> wrote:
> Hi,
>
> I tried a while back to use the newer metadata formats on my mdadm RAID5 on a
> few machines, and discovered that the kernel auto-assembly will only work with
> v0.90 metadata, not 1.0 or greater. Is there a solid reason for this? Based
> one what I can find regarding the differences in the metadata formats, and
> looking at the existing md code, it seems this is largely just because no one
> has had the time or motivation to change the code to support auto-assembly on
> the newer metdata formats.
>
> I am told that the "correct" solution is to embed a small initramfs to bring
> the RAID arrays online instead, before the real rootfs is loaded. I'd like to
> avoid this if possible, as I haven't had to use an initramfs for normal booting
> in the past, as long as I stay on metadata 0.90. So I thought I'd ask what the
> official stance is on this.
>
> Thanks!,
>
> --
> Joshua Kinard
> Gentoo/MIPS
> kumba@gentoo.org
> 4096R/D25D95E3 2011-03-28
>
I cannot speak to any of the reasons to support it or not but I'm a Gentoo
guy since late 2002 who avoided the initramfs for the longest time. I
finally bit the bullet and learned how to build it into my kernels so there are
no extra files and except for a recent problem with Gentoo devs making
changes to busybox defaults it's worked very well. The nice thing about
building it into the kernel is that old kernels continue to work perfectly as
best I can tell.
Anyway, from my perspective it was worth learning. I'm just a user type,
not a dev of any type.
Cheers,
Mark
^ permalink raw reply
* Re: An old "write-mostly" read balance issue
From: NeilBrown @ 2015-02-23 0:03 UTC (permalink / raw)
To: Dark Penguin; +Cc: linux-raid, tomas.hodek
In-Reply-To: <54D78357.7020808@yandex.ru>
[-- Attachment #1: Type: text/plain, Size: 5403 bytes --]
On Sun, 08 Feb 2015 18:40:07 +0300 Dark Penguin <darkpenguin@yandex.ru> wrote:
> There is an old issue about RAID1 read-balancing when "write-mostly"
> disks are present.
>
> The problem is, according to the manual, "md driver will avoid reading
> from these devices if at all possible".
>
> One way to understand this statement is that these drives will never be
> read from, except when the main drive can not be read from. There are A
> LOT of situations when this is the expected and desired behaviour:
> - People mirroring an SSD with an HDD and suffering a performance loss;
> - People mirroring a fast HDD with a slow HDD for reliability, for
> example, mirroring a 300Gb WD Raptor to a 300Gb partition on a 3Tb 5900
> "green" drive for backup; since the larger drive may be used for
> something other than this RAID, many would prefer it to be spared the
> workload.
> - In my case, I have a home RAID1 storage, which is idle 95% of the
> time, and 95% of the remaining 5% I only read from it. So I want one of
> the drives to spin down and never turn on, in order to avoid wearing
> down the mechanics. They say, "The best way to keep a device from
> breaking is to turn it off and not use it". :) But even if I simply
> retrieve the contents of my volume, that request is apparently enough to
> load the first drive to 100% for a split second, which causes the second
> drive to spin up, which is extremely undesirable.
>
> I've spent a lot of time looking for the answer "why does it spin up",
> and "normal forum users" couldn not even help me, but then I found out
> that there is another way to read that statement: apparently, there are
> other people who would like to see whatever little benefit reading from
> the second drive could give them. I can not say which side is a
> majority, but I respect their wishes as well, and personally I'm fine
> with any default behaviour as long as I have what I need.
>
> I've found a patch for that:
> http://marc.info/?l=linux-raid&m=135982797322422
> Apparently, it can be used with any kernel, but I'm not good enough to
> make sure nothing's broken everytime I upgrade the kernel, and frankly,
> I think there are A LOT of people who wish to see the behaviour I would
> expect. So my plea is for the developers to accept this patch and make
> this behaviour optional, if not default. At least give us a compile
> option to build the kernel this way! There are people out there who use
> RAID1 at home and not in production, and therefore care less about
> performance than home storage idling, and who understand the words "if
> at all possible" in the more obvious way! I think that's the whole
> reason why the "write-mostly" option is there in the first place, but if
> there are people who don't agree with me - I'm not going to argue, they
> can have it their way, just give us the option to do what we want, too!
>
>
Hi,
thanks for reporting this. It is definitely a bug. It was introduced by
commit 9dedf60313fa4dddfd5b9b226a0ef12a512bf9dc
md/raid1: read balance chooses idlest disk for SSD
I don't recall seeing the patch from Tomas Hodek which you provided a link
for - sorry Tomas.
I prefer the second of the two patches. I will submit the following to Linus
some time this week.
Thanks for pursuing this Dark Penguin.
NeilBrown
From: Tomas Hodek <tomas.hodek@volny.cz>
Date: Mon, 23 Feb 2015 11:00:38 +1100
Subject: [PATCH] Subject: md/raid1: fix read balance when a drive is
write-mostly.
When a drive is marked write-mostly it should only be the
target of reads if there is no other option.
This behaviour was broken by
commit 9dedf60313fa4dddfd5b9b226a0ef12a512bf9dc
md/raid1: read balance chooses idlest disk for SSD
which causes a write-mostly device to be *preferred* is some cases.
Restore correct behaviour by checking and setting
best_dist_disk and best_pending_disk rather than best_disk.
We only need to test one of these as they are both changed
from -1 or >=0 at the same time.
As we leave min_pending and best_dist unchanged, any non-write-mostly
device will appear better than the write-mostly device.
Reported-by: tomas.hodek@volny.cz
Reported-by: Dark Penguin <darkpenguin@yandex.ru>
Signed-off-by: NeilBrown <neilb@suse.de>
Link: http://marc.info/?l=linux-raid&m=135982797322422
Fixes: 9dedf60313fa4dddfd5b9b226a0ef12a512bf9dc
Cc: stable@vger.kernel.org (3.6+)
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index 0b6349f9c5c5..7742e0999bf2 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -560,7 +560,7 @@ static int read_balance(struct r1conf *conf, struct r1bio *r1_bio, int *max_sect
if (test_bit(WriteMostly, &rdev->flags)) {
/* Don't balance among write-mostly, just
* use the first as a last resort */
- if (best_disk < 0) {
+ if (best_dist_disk < 0) {
if (is_badblock(rdev, this_sector, sectors,
&first_bad, &bad_sectors)) {
if (first_bad < this_sector)
@@ -569,7 +569,8 @@ static int read_balance(struct r1conf *conf, struct r1bio *r1_bio, int *max_sect
best_good_sectors = first_bad - this_sector;
} else
best_good_sectors = sectors;
- best_disk = disk;
+ best_dist_disk = disk;
+ best_pending_disk = disk;
}
continue;
}
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply related
* Re: Metadata > 0.90 and auto-assemble
From: NeilBrown @ 2015-02-23 0:17 UTC (permalink / raw)
To: Joshua Kinard; +Cc: Chris Murphy, linux-raid
In-Reply-To: <54EA5004.3090104@gentoo.org>
[-- Attachment #1: Type: text/plain, Size: 2567 bytes --]
On Sun, 22 Feb 2015 16:54:12 -0500 Joshua Kinard <kumba@gentoo.org> wrote:
> On 02/22/2015 16:29, Chris Murphy wrote:
> > On Sun, Feb 22, 2015 at 11:54 AM, Joshua Kinard <kumba@gentoo.org> wrote:
> >> I'd like to
> >> avoid this if possible, as I haven't had to use an initramfs for normal booting
> >> in the past, as long as I stay on metadata 0.90. So I thought I'd ask what the
> >> official stance is on this.
> >
> > https://raid.wiki.kernel.org/index.php/Autodetect
> >
> > Official stance is that it's deprecated, but people still use it.
>
> Yeah, but it's a pretty useful feature. I can't see why autodetect for simple
> setups (several disks or partitions and building a basic array out of them) is
> maintained, while userspace autodetect is required for the more complex setups.
The in-kernel autodetect is only maintained because tearing it out and
throwing it away (my preferred option) would be a user-visible regression,
and those are not permitted.
The user-space version is more general and more flexible. If the
kernel-space version works for you, you can keep using it. But if you want
features added to it, you are out of luck.
As others have said, creating a simple initrd is really not that hard. Once
you spend the time to make it work, you will find that it "just works" and
wonder why you ever cared before.
There is even a README.initramfs in the mdadm source. It was written 10
years ago so I cannot promise it is 100% correct, but it is a reasonably good
and very simple starting point.
NeilBrown
>
> But I suppose this has been discussed before in detail, though I cannot find
> said discussion. The RAID Boot page has this one example only:
>
> "This approach can cause problems in several situations (imagine moving part of
> an old array onto another machine before wiping and repurposing it: reboot and
> watch in horror as the piece of dead array gets assembled as part of the
> running RAID array, ruining it); kernel autodetect is correspondingly deprecated."
>
> Which I find to be rather unconvincing. The cited example is a fault of the
> user not torching the superblock before moving the disks or trying to use
> them...and I've done this to myself on several occasions. mdadm --misc
> --zero-superblock and 'dd' saved the day in less than ~30s.
>
> Are there any other discussions that might be more convincing, or offer up
> other points of view? Perhaps there's a point I've yet to consider that might
> be enlightening.
>
> Thanks!,
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [dm-devel] [PATCH 0/3] md raid: enhancements to support the device mapper dm-raid target
From: NeilBrown @ 2015-02-23 1:07 UTC (permalink / raw)
To: Heinz Mauelshagen
Cc: device-mapper development, jbras >> Brassow Jonathan,
linux RAID
In-Reply-To: <54E47C88.1080203@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 6379 bytes --]
On Wed, 18 Feb 2015 12:50:32 +0100 Heinz Mauelshagen <heinzm@redhat.com>
wrote:
> On 02/18/2015 03:03 AM, NeilBrown wrote:
> > On Fri, 13 Feb 2015 19:47:59 +0100 heinzm@redhat.com wrote:
> >
> >> From: Heinz Mauelshagen <heinzm@redhat.com>
> >>
> >> I'm enhancing the device mapper raid target (dm-raid) to take
> >> advantage of so far unused md raid kernel funtionality:
> >> takeover, reshape, resize, addition and removal of devices to/from raid sets.
> >>
> >> This series of patches remove constraints doing so.
> >>
> >>
> >> Patch #1:
> >> add 2 API functions to allow dm-raid to access the raid takeover
> >> and resize functionality (namely md_takeover() and md_resize());
> >> reshape APIs are not needed in lieu of the existing personalilty ones
> >>
> >> Patch #2:
> >> because device mapper core manages a request queue per mapped device
> >> utilizing the md make_request API to pass on bios via the dm-raid target,
> >> no md instance underneath it needs to manage a request queue of its own.
> >> Thus dm-raid can't use the md raid0 personality as is, because the latter
> >> accesses the request queue unconditionally in 3 places via mddev->queue
> >> which this patch addresses.
> >>
> >> Patch #3:
> >> when dm-raid processes a down takeover to raid0, it needs to destroy
> >> any existing bitmap, because raid0 does not require one. The patch
> >> exports the bitmap_destroy() API to allow dm-raid to remove bitmaps.
> >>
> >>
> >> Heinz Mauelshagen (3):
> >> md core: add 2 API functions for takeover and resize to support dm-raid
> >> md raid0: access mddev->queue (request queue member) conditionally
> >> because it is not set when accessed from dm-raid
> >> md bitmap: export bitmap_destroy() to support dm-raid down takover to raid0
> >>
> >> drivers/md/bitmap.c | 1 +
> >> drivers/md/md.c | 39 ++++++++++++++++++++++++++++++---------
> >> drivers/md/md.h | 3 +++
> >> drivers/md/raid0.c | 48 +++++++++++++++++++++++++++---------------------
> >> 4 files changed, 61 insertions(+), 30 deletions(-)
> >>
> > Hi Heinz,
> > I don't object to these patches if you will find the exported functionality
> > useful, but I am a little surprised by them.
>
> Hi Neil,
>
> I find them useful to allow for atomic takeover using the already given
> md raid
> code rather than duplicating ACID takeover in dm-raid/lvm. If I'd not
> use md for this,
> I'd have to keep copies of the given md superblocks and restore them in case
> the assembly of the array failed and superblocks have been updated.
This argument doesn't make much sense to me.
There is no reason that the assembling the array in a new configuration would
fail, except possible malloc error or similar which would make putting it
back into the original configuration fail as well.
There is no need to synchronise updating the metadata with a take-over.
In every case, the "Before" and "After" configurations are functionally
identical.
A 2-drive RAID1 behaves identically to a 2-drive RAID5, for example.
So it doesn't really matter whether or not the metadata match how the kernel
is configured. Once you start a reshape (e.g. 2-drive RAID5 to 3-drive
RAID5) or add a spare, then you need the metadata to be correct, but that is
just a sequencing issue:
- start: metadata says "raid1".
- suspend array, reconfigure as RAID5 with 2 drives, resume.
- if everything went well, update metadata to "raid5".
- now update metadata to "0 block of progress into reshape from 2-drives to
3-drives".
- now start the reshape, which will further update the metadata as it
proceeds.
There really are no atomicity requirements, only sequencing.
>
> >
> > I would expect that dm-raid wouldn't ask md to 'takeover' from one level to
> > another, but instead would
> > - suspend the dm device
> > - dismantle the array using the old level
> > - assemble the array using the new level
> > - resume the dm device
>
> That scenario is on my TODO, because it is for instance paritcularly
> useful to
> convert a "striped" array (or a "raid0" array without metadata for that
> purpose)
> directly into a raid6_n_6 one (i.e. dedicated xor and syndrome devices)
> thus avoding any interim levels.
> In these cases, I'd only need to drop the metadata devs allocations if
> the array does not start up properly and restart the previous mapping.
>
Given that you plan to do this, I really think the dm and LVM code would be
simpler if all reconfigurations use this same approach.
>
> >
> > The reason md needs 'takeover' is because it doesn't have the same
> > device/target separation that dm does.
>
> Correct.
> Nonetheless, I found accessing md's takeover functionality still useful
> for the atomic updates to be simpler in dm/lvm.
>
> >
> > I was particularly surprised that you wanted to use md/raid0.c It is no
> > better than dm/dm-stripe.c and managing two different stripe engines under
> > LVM doesn't see like a good idea.
>
> I actually see differences in performance which I have not explained yet.
>
> In some cases, dm-stripe performs better, in others md raid0 does for
> the same mappings
> and load; exact same mappings are possible, because I've got patches to
> lvconvert back
> and forth between "striped" and "raid0", hence accesing exactly the same
> physical extents.
That is surprising. I would be great if we could characterise what sort of
workloads work better with one or the other...
>
> So supporting "raid0" in dm-raid is senseful for 3 reasons:
> - replace dm-stripe with md raid0
> - atomic md takeover from "raid0" -> "raid5"
> - potential performance implications
>
> >
> > Is there some reason that I have missed which makes it easier to use
> > 'takeover' rather than suspend/resume?
>
> Use md takover for atomic updates as mentioned above.
>
> You don't have issues with md_resize() which I use to shrink existing
> arrays?
>
I have exactly the same issue with md_resize() as with md_takeover(), and for
the same reasons.
How about we wait until you do implement the
suspend/dismantle/reassemble/resume
approach, and see if you still want md_resize/md_takeover after that?
Thanks,
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Optimal chunk size for RAID5?
From: Christer Solskogen @ 2015-02-23 1:36 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <20150223085358.302830d1@notabene.brown>
On 22.02.2015 22:53, NeilBrown wrote:
> There is no "Optimal" without reference to a particular work load. Or
> particular hardware.
>
Do you know of such a reference? I mean, some stats that show type of
workload / chunk size. The only one I've found is the 5 year old
benchmark that was done (
http://louwrentius.com/linux-raid-level-and-chunk-size-the-benchmarks.html)
- which shows that under benchmarking with dd that 64 is preferred.
--
chs
^ permalink raw reply
* Re: [PATCH 02/24] Add number of nodes to bitmap structure for clustering
From: NeilBrown @ 2015-02-23 1:38 UTC (permalink / raw)
To: Goldwyn Rodrigues; +Cc: lzhong, linux-raid
In-Reply-To: <20141218161523.GA29576@shrek.lan>
[-- Attachment #1: Type: text/plain, Size: 1510 bytes --]
On Thu, 18 Dec 2014 10:15:23 -0600 Goldwyn Rodrigues <rgoldwyn@suse.de> wrote:
> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
> ---
> drivers/md/bitmap.h | 5 +++--
> 1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/md/bitmap.h b/drivers/md/bitmap.h
> index 30210b9..6872945 100644
> --- a/drivers/md/bitmap.h
> +++ b/drivers/md/bitmap.h
> @@ -128,10 +128,11 @@ typedef struct bitmap_super_s {
> __le32 chunksize; /* 52 the bitmap chunk size in bytes */
> __le32 daemon_sleep; /* 56 seconds between disk flushes */
> __le32 write_behind; /* 60 number of outstanding write-behind writes */
> - __le32 sectors_reserved; /* 64 number of 512-byte sectors that are
> + __le32 nodes; /* 64 the maximum number of nodes in cluster. */
> + __le32 sectors_reserved; /* 68 number of 512-byte sectors that are
> * reserved for the bitmap. */
>
> - __u8 pad[256 - 68]; /* set to zero */
> + __u8 pad[256 - 72]; /* set to zero */
> } bitmap_super_t;
>
> /* notes:
Hi Goldwyn,
I was reviewing you latest series to make sure it wouldn't affect the
non-clustered use case at all, and I found this. I really should have
noticed it earlier....
You are changing the location of 'sectors_reserved' in the bitmap superblock.
That obviously cannot be allowed - new fields must always be added to the
end.
Can you update the series in git to fix that please? Then I will pull it in
for -next.
Thanks,
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Optimal chunk size for RAID5?
From: NeilBrown @ 2015-02-23 3:28 UTC (permalink / raw)
To: Christer Solskogen; +Cc: linux-raid
In-Reply-To: <mce06j$i2h$1@ger.gmane.org>
[-- Attachment #1: Type: text/plain, Size: 1632 bytes --]
On Mon, 23 Feb 2015 02:36:18 +0100 Christer Solskogen
<christer.solskogen@gmail.com> wrote:
> On 22.02.2015 22:53, NeilBrown wrote:
>
> > There is no "Optimal" without reference to a particular work load. Or
> > particular hardware.
> >
>
> Do you know of such a reference? I mean, some stats that show type of
> workload / chunk size. The only one I've found is the 5 year old
> benchmark that was done (
> http://louwrentius.com/linux-raid-level-and-chunk-size-the-benchmarks.html)
> - which shows that under benchmarking with dd that 64 is preferred.
>
Interesting graphs ... but when you see a big jump like they show between 64
and 128K chunk sizes for RAID5/6, that doesn't mean "64K is better" but
"something strange is happening here". My guess is that read-ahead is
working very well for some reason.
If your actually workload is writing 10GB files with 'dd', then the graphs
might be useful. For other workloads ... it's hard to tell.
Nothing beats performing your own tests on your own hardware with your own
choice of filesystem and getting your own results.
I did some tests myself recently (which I really want to automate and turn
into web pages etc ... one day).
For RAID5 on 4 drives I used chunk sizes of 4, 16, 64, 256, 1024 and applied
a variety of fio loads use XFS.
The only load that showed significant variation of chunk sizes was sequential
read which gets generally faster with larger chunk sizes, though for some
layouts (I tried la, ls, ra, rs) 1024k chunks were worse than 256k.
So any reference you find will probably lead you astray.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [dm-devel] [PATCH 0/3] md raid: enhancements to support the device mapper dm-raid target
From: Heinz Mauelshagen @ 2015-02-23 11:49 UTC (permalink / raw)
To: NeilBrown
Cc: device-mapper development, jbras >> Brassow Jonathan,
linux RAID
In-Reply-To: <20150223120718.03806a87@notabene.brown>
On 02/23/2015 02:07 AM, NeilBrown wrote:
> On Wed, 18 Feb 2015 12:50:32 +0100 Heinz Mauelshagen <heinzm@redhat.com>
> wrote:
>
>> On 02/18/2015 03:03 AM, NeilBrown wrote:
>>> On Fri, 13 Feb 2015 19:47:59 +0100 heinzm@redhat.com wrote:
>>>
>>>> From: Heinz Mauelshagen <heinzm@redhat.com>
>>>>
>>>> I'm enhancing the device mapper raid target (dm-raid) to take
>>>> advantage of so far unused md raid kernel funtionality:
>>>> takeover, reshape, resize, addition and removal of devices to/from raid sets.
>>>>
>>>> This series of patches remove constraints doing so.
>>>>
>>>>
>>>> Patch #1:
>>>> add 2 API functions to allow dm-raid to access the raid takeover
>>>> and resize functionality (namely md_takeover() and md_resize());
>>>> reshape APIs are not needed in lieu of the existing personalilty ones
>>>>
>>>> Patch #2:
>>>> because device mapper core manages a request queue per mapped device
>>>> utilizing the md make_request API to pass on bios via the dm-raid target,
>>>> no md instance underneath it needs to manage a request queue of its own.
>>>> Thus dm-raid can't use the md raid0 personality as is, because the latter
>>>> accesses the request queue unconditionally in 3 places via mddev->queue
>>>> which this patch addresses.
>>>>
>>>> Patch #3:
>>>> when dm-raid processes a down takeover to raid0, it needs to destroy
>>>> any existing bitmap, because raid0 does not require one. The patch
>>>> exports the bitmap_destroy() API to allow dm-raid to remove bitmaps.
>>>>
>>>>
>>>> Heinz Mauelshagen (3):
>>>> md core: add 2 API functions for takeover and resize to support dm-raid
>>>> md raid0: access mddev->queue (request queue member) conditionally
>>>> because it is not set when accessed from dm-raid
>>>> md bitmap: export bitmap_destroy() to support dm-raid down takover to raid0
>>>>
>>>> drivers/md/bitmap.c | 1 +
>>>> drivers/md/md.c | 39 ++++++++++++++++++++++++++++++---------
>>>> drivers/md/md.h | 3 +++
>>>> drivers/md/raid0.c | 48 +++++++++++++++++++++++++++---------------------
>>>> 4 files changed, 61 insertions(+), 30 deletions(-)
>>>>
>>> Hi Heinz,
>>> I don't object to these patches if you will find the exported functionality
>>> useful, but I am a little surprised by them.
>> Hi Neil,
>>
>> I find them useful to allow for atomic takeover using the already given
>> md raid
>> code rather than duplicating ACID takeover in dm-raid/lvm. If I'd not
>> use md for this,
>> I'd have to keep copies of the given md superblocks and restore them in case
>> the assembly of the array failed and superblocks have been updated.
> This argument doesn't make much sense to me.
>
> There is no reason that the assembling the array in a new configuration would
> fail, except possible malloc error or similar which would make putting it
> back into the original configuration fail as well.
>
> There is no need to synchronise updating the metadata with a take-over.
> In every case, the "Before" and "After" configurations are functionally
> identical.
> A 2-drive RAID1 behaves identically to a 2-drive RAID5, for example.
> So it doesn't really matter whether or not the metadata match how the kernel
> is configured. Once you start a reshape (e.g. 2-drive RAID5 to 3-drive
> RAID5) or add a spare, then you need the metadata to be correct, but that is
> just a sequencing issue:
>
> - start: metadata says "raid1".
> - suspend array, reconfigure as RAID5 with 2 drives, resume.
> - if everything went well, update metadata to "raid5".
> - now update metadata to "0 block of progress into reshape from 2-drives to
> 3-drives".
> - now start the reshape, which will further update the metadata as it
> proceeds.
>
> There really are no atomicity requirements, only sequencing.
Thanks for clarifying these conversions, I was presuming there were
atomicity issues in the md kernel code to conform to.
Canges to run those sequences look straightforward in the dm-raid target.
I'll implement them and test.
>
>
>>> I would expect that dm-raid wouldn't ask md to 'takeover' from one level to
>>> another, but instead would
>>> - suspend the dm device
>>> - dismantle the array using the old level
>>> - assemble the array using the new level
>>> - resume the dm device
>> That scenario is on my TODO, because it is for instance paritcularly
>> useful to
>> convert a "striped" array (or a "raid0" array without metadata for that
>> purpose)
>> directly into a raid6_n_6 one (i.e. dedicated xor and syndrome devices)
>> thus avoding any interim levels.
>> In these cases, I'd only need to drop the metadata devs allocations if
>> the array does not start up properly and restart the previous mapping.
>>
> Given that you plan to do this, I really think the dm and LVM code would be
> simpler if all reconfigurations use this same approach.
You got a point with regards to the dm-raid target:
if an MD takeover API is actually superfluous in the end, the target
won't have 2 code paths for
a) going from a non-metadata config to a metadata one (e.g. striped ->
raid5)
and
b) a metadata -> metadata one (e.g. raid6 -> raid5)
In lvm2/dm userspace there will be no difference, because it has to
update the userspace metadata and the kernel metadata comiting it
in the proper sequence and does not call any takeover api in userspace
at all which could be avoided as in the kernel.
>
>>> The reason md needs 'takeover' is because it doesn't have the same
>>> device/target separation that dm does.
>> Correct.
>> Nonetheless, I found accessing md's takeover functionality still useful
>> for the atomic updates to be simpler in dm/lvm.
>>
>>> I was particularly surprised that you wanted to use md/raid0.c It is no
>>> better than dm/dm-stripe.c and managing two different stripe engines under
>>> LVM doesn't see like a good idea.
>> I actually see differences in performance which I have not explained yet.
>>
>> In some cases, dm-stripe performs better, in others md raid0 does for
>> the same mappings
>> and load; exact same mappings are possible, because I've got patches to
>> lvconvert back
>> and forth between "striped" and "raid0", hence accesing exactly the same
>> physical extents.
> That is surprising. I would be great if we could characterise what sort of
> workloads work better with one or the other...
Agreed, we need more facts.
I've seen indications from "dd oflag=direct iflag=fullblock bs=1G
count=1 if=/dev/zero of=$LV
converting back and forth to/from raid0/striped mappings on an otherwise
idle system.
>> So supporting "raid0" in dm-raid is senseful for 3 reasons:
>> - replace dm-stripe with md raid0
>> - atomic md takeover from "raid0" -> "raid5"
>> - potential performance implications
>>
>>> Is there some reason that I have missed which makes it easier to use
>>> 'takeover' rather than suspend/resume?
>> Use md takover for atomic updates as mentioned above.
>>
>> You don't have issues with md_resize() which I use to shrink existing
>> arrays?
>>
> I have exactly the same issue with md_resize() as with md_takeover(), and for
> the same reasons.
Ok, let me do avoiding patches based on your clarifications
which'll take till next week including testing.
> How about we wait until you do implement the
> suspend/dismantle/reassemble/resume
> approach, and see if you still want md_resize/md_takeover after that?
Sure.
I'd like to see the raid0 conditonal request queue patch though.
Thanks,
Heinz
>
> Thanks,
> NeilBrown
>
^ permalink raw reply
* Re: [PATCH 02/24] Add number of nodes to bitmap structure for clustering
From: Goldwyn Rodrigues @ 2015-02-23 18:13 UTC (permalink / raw)
To: NeilBrown; +Cc: lzhong, linux-raid
In-Reply-To: <20150223123848.01416b69@notabene.brown>
Hi Neil,
On 02/22/2015 07:38 PM, NeilBrown wrote:
> On Thu, 18 Dec 2014 10:15:23 -0600 Goldwyn Rodrigues <rgoldwyn@suse.de> wrote:
>
>> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
>> ---
>> drivers/md/bitmap.h | 5 +++--
>> 1 file changed, 3 insertions(+), 2 deletions(-)
>>
>> diff --git a/drivers/md/bitmap.h b/drivers/md/bitmap.h
>> index 30210b9..6872945 100644
>> --- a/drivers/md/bitmap.h
>> +++ b/drivers/md/bitmap.h
>> @@ -128,10 +128,11 @@ typedef struct bitmap_super_s {
>> __le32 chunksize; /* 52 the bitmap chunk size in bytes */
>> __le32 daemon_sleep; /* 56 seconds between disk flushes */
>> __le32 write_behind; /* 60 number of outstanding write-behind writes */
>> - __le32 sectors_reserved; /* 64 number of 512-byte sectors that are
>> + __le32 nodes; /* 64 the maximum number of nodes in cluster. */
>> + __le32 sectors_reserved; /* 68 number of 512-byte sectors that are
>> * reserved for the bitmap. */
>>
>> - __u8 pad[256 - 68]; /* set to zero */
>> + __u8 pad[256 - 72]; /* set to zero */
>> } bitmap_super_t;
>>
>> /* notes:
>
> Hi Goldwyn,
> I was reviewing you latest series to make sure it wouldn't affect the
> non-clustered use case at all, and I found this. I really should have
> noticed it earlier....
>
> You are changing the location of 'sectors_reserved' in the bitmap superblock.
> That obviously cannot be allowed - new fields must always be added to the
> end.
Oops. Sorry.
>
> Can you update the series in git to fix that please? Then I will pull it in
> for -next.
I have done this and re-tested. However, I also rebased against
upstream. There were some conflicts with respect to md_personality. I
have re-done those patches, but it would be worth a second look. The one
which may need special review attention is "bitmap_create returns bitmap
pointer"
Regards,
--
Goldwyn
^ permalink raw reply
* Inject I/O latency for RAID5/6 read and writes
From: Alireza Haghdoost @ 2015-02-23 18:32 UTC (permalink / raw)
To: Linux RAID; +Cc: Neil Brown
I needed to inject I/O completion latency in the RAID5/6 codes for
test purpose. I was wondering where would be the good place in
md/raid5.c code to add delay ?
So far I have tried adding mdelay/udelay in raid5_end_write_request()
and it seems it works to increase I/O completion of writes. However,
adding delay in raid5_end_read_request() does not really change the
read I/O latency. Any idea ?
--Alireza
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox