* Re: [PATCH 5/6] Check write journal in incremental
From: Dan Williams @ 2015-10-06 20:17 UTC (permalink / raw)
To: Song Liu; +Cc: linux-raid, Shaohua Li, Neil Brown, Christoph Hellwig
In-Reply-To: <1440804426-1461372-6-git-send-email-songliubraving@fb.com>
On Fri, Aug 28, 2015 at 4:27 PM, Song Liu <songliubraving@fb.com> wrote:
> If journal device is missing, do not start the array, and shows:
>
> ./mdadm -I /dev/sdf
> mdadm: journal device is missing, not safe to start yet.
>
> The array will be started when the journal device is attached with -I
>
> ./mdadm -I /dev/sdb1
> mdadm: /dev/sdb1 attached to /dev/md/0_0, which has been started.
>
> To force start without journal device:
>
> ./mdadm -I /dev/sdf --run
> mdadm: Trying to run with missing journal device
> mdadm: /dev/sdf attached to /dev/md/0_0, which has been started.
>
> Signed-off-by: Shaohua Li <shli@fb.com>
> Signed-off-by: Song Liu <songliubraving@fb.com>
> ---
> Incremental.c | 31 +++++++++++++++++++++++++++----
> 1 file changed, 27 insertions(+), 4 deletions(-)
>
> diff --git a/Incremental.c b/Incremental.c
> index 304cc6d..74905e3 100644
> --- a/Incremental.c
> +++ b/Incremental.c
> @@ -35,7 +35,7 @@
>
> static int count_active(struct supertype *st, struct mdinfo *sra,
> int mdfd, char **availp,
> - struct mdinfo *info);
> + struct mdinfo *info, int *journal_device_missing);
> static void find_reject(int mdfd, struct supertype *st, struct mdinfo *sra,
> int number, __u64 events, int verbose,
> char *array_name);
> @@ -104,6 +104,7 @@ int Incremental(struct mddev_dev *devlist, struct context *c,
> struct map_ent target_array;
> int have_target;
> char *devname = devlist->devname;
> + int journal_device_missing = 0;
>
> struct createinfo *ci = conf_get_create_info();
>
> @@ -518,7 +519,7 @@ int Incremental(struct mddev_dev *devlist, struct context *c,
> sysfs_free(sra);
> sra = sysfs_read(mdfd, NULL, (GET_DEVS | GET_STATE |
> GET_OFFSET | GET_SIZE));
> - active_disks = count_active(st, sra, mdfd, &avail, &info);
> + active_disks = count_active(st, sra, mdfd, &avail, &info, &journal_device_missing);
> if (enough(info.array.level, info.array.raid_disks,
> info.array.layout, info.array.state & 1,
> avail) == 0) {
> @@ -548,10 +549,12 @@ int Incremental(struct mddev_dev *devlist, struct context *c,
> }
>
> map_unlock(&map);
> - if (c->runstop > 0 || active_disks >= info.array.working_disks) {
> + if (c->runstop > 0 || (!journal_device_missing && active_disks >= info.array.working_disks)) {
A minor comment, and I'd defer to Neil's opinion, but I think this is
asking for mdu_array_info_t to grow a "journal_disks" attribute.
^ permalink raw reply
* Re: [PATCH 4/6] Assemble array with write journal
From: Dan Williams @ 2015-10-06 20:11 UTC (permalink / raw)
To: Song Liu; +Cc: linux-raid, Shaohua Li, Neil Brown, Christoph Hellwig
In-Reply-To: <1440804426-1461372-5-git-send-email-songliubraving@fb.com>
On Fri, Aug 28, 2015 at 4:27 PM, Song Liu <songliubraving@fb.com> wrote:
> Example output:
>
> ./mdadm --assemble /dev/md0 /dev/sd[c-f] /dev/sdb1
> mdadm: /dev/md0 has been started with 4 drives and 1 journal.
>
> mdadm checks superblock for journal devices. If the
> array appears to have a journal device, but it is not given,
> it will complain as
>
> ./mdadm --assemble /dev/md0 /dev/sd[c-f]
> mdadm: Not safe to assemble with journal device missing, consider --force.
>
> This can be overwritten with --force
>
> ./mdadm --assemble /dev/md0 /dev/sd[c-f] --force
> mdadm: Force start with missing journal device...
> mdadm: /dev/md0 has been started with 4 drives.
>
> Signed-off-by: Shaohua Li <shli@fb.com>
> Signed-off-by: Song Liu <songliubraving@fb.com>
[..]
> diff --git a/mdadm.h b/mdadm.h
> index 3cc1532..bc6680f 100644
> --- a/mdadm.h
> +++ b/mdadm.h
> @@ -969,6 +969,9 @@ extern struct superswitch {
> /* validate container after assemble */
> int (*validate_container)(struct mdinfo *info);
>
> + /* whether the array require a journal device */
> + int (*require_journal)(struct supertype *st);
> +
> int swapuuid; /* true if uuid is bigending rather than hostendian */
> int external;
> const char *name; /* canonical metadata name */
I'm wondering why is this a function pointer rather than a flag? Is
it because "roles" are only an internal detail of v1 superblocks?
Might be time to make them a first class citizen. Again, I would use
that generic "roles" enabling for a new rev of the SRT patches.
^ permalink raw reply
* Re: [PATCH 3/6] Enable create array with write journal (--write-journal DEVICE).
From: Dan Williams @ 2015-10-06 18:32 UTC (permalink / raw)
To: Song Liu; +Cc: linux-raid, Shaohua Li, Neil Brown, Christoph Hellwig
In-Reply-To: <1440804426-1461372-4-git-send-email-songliubraving@fb.com>
On Fri, Aug 28, 2015 at 4:27 PM, Song Liu <songliubraving@fb.com> wrote:
> Specify the write journal device with --write-journal DEVICE
>
> ./mdadm --create -f /dev/md0 --assume-clean -c 32 --raid-devices=4 --level=5 /dev/sd[c-f] --write-journal /dev/sdb1
> mdadm: Defaulting to version 1.2 metadata
> mdadm: array /dev/md0 started.
>
> Only one journal device is allowed. If multiple --write-journal
> are given, mdadm will use the first and ignore others
>
> ./mdadm --create -f /dev/md0 --assume-clean -c 32 --raid-devices=4 --level=5 /dev/sd[c-f] --write-journal /dev/sdb1 --write-journal /dev/sdx
> mdadm: Please specify only one journal device for the array.
> mdadm: Ignoring --write-journal /dev/sdx...
> mdadm: Defaulting to version 1.2 metadata
> mdadm: array /dev/md0 started.
Hmm, perhaps this should instead follow the same command line
mechanics as "--write-mostly" whereby all the devices listed after
that flag take on that role. Also, if we anticipate more roles I
wonder if the option should be "--role=" that assigns the given role
to the device(s) that follow.
Thoughts?
I'm looking at this from the perspective of reviving the SRT patches
[1] where a role for a read/write cache device is needed. Where a
read/write-cache is different than a journal.
[1]: https://lwn.net/Articles/596614/
^ permalink raw reply
* Re: raid5-cache I/O path improvements V3
From: Shaohua Li @ 2015-10-06 18:14 UTC (permalink / raw)
To: Christoph Hellwig; +Cc: neilb, dan.j.williams, Kernel-team, linux-raid
In-Reply-To: <1444030276-30850-1-git-send-email-hch@lst.de>
On Mon, Oct 05, 2015 at 09:31:05AM +0200, Christoph Hellwig wrote:
> Hi Shaohua, hi Neil,
>
> this series contains a few updates to the raid5-cache feature, and goes
> on top of dm/devel + the series Shaohua sent yesterday.
>
> Besides the rebase the only changes compared to the previous version
> is that we only bypasss the batch and flush state for devices that
> do not have a volatile cache at all like persistent memory or high
> performance flash devices.
Thanks Christoph, patches look great.
Reviewed-by: Shaohua Li <shli@fb.com>
^ permalink raw reply
* [PATCH 5/5] Fix adding of new disk with new reload code
From: rgoldwyn @ 2015-10-05 15:09 UTC (permalink / raw)
To: linux-raid, neilb; +Cc: Goldwyn Rodrigues
In-Reply-To: <1444057794-17256-1-git-send-email-rgoldwyn@suse.de>
From: Goldwyn Rodrigues <rgoldwyn@suse.com>
Adding the disk worked incorrectly with the new reload code. Fix it:
- No operation should be performed on rdev marked as Candidate
- After a metadata update operation, kick disk if role is 0xfffe
else clear Candidate bit and continue with the regular change check.
- On the initiating node, cancel the metadata_update (unlock token),
in case of an error while adding
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
---
drivers/md/md.c | 31 +++++++++++++++++++------------
1 file changed, 19 insertions(+), 12 deletions(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 75d14e3..e324544 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -3246,14 +3246,6 @@ static void analyze_sbs(struct mddev *mddev)
md_kick_rdev_from_array(rdev);
continue;
}
- /* No device should have a Candidate flag
- * when reading devices
- */
- if (test_bit(Candidate, &rdev->flags)) {
- pr_info("md: kicking Cluster Candidate %s from array!\n",
- bdevname(rdev->bdev, b));
- md_kick_rdev_from_array(rdev);
- }
}
if (mddev->level == LEVEL_MULTIPATH) {
rdev->desc_nr = i++;
@@ -5962,7 +5954,7 @@ static int add_new_disk(struct mddev *mddev, mdu_disk_info_t *info)
/* --add initiated by this node */
err = md_cluster_ops->add_new_disk_start(mddev, rdev);
if (err) {
- md_cluster_ops->add_new_disk_finish(mddev);
+ md_cluster_ops->metadata_update_cancel(mddev);
export_rdev(rdev);
return err;
}
@@ -5973,11 +5965,11 @@ static int add_new_disk(struct mddev *mddev, mdu_disk_info_t *info)
err = bind_rdev_to_array(rdev, mddev);
if (err)
export_rdev(rdev);
- else
+ else if (!(info->state & (1 << MD_DISK_CANDIDATE)))
err = add_bound_rdev(rdev);
- if (mddev_is_clustered(mddev) &&
+ if (err && mddev_is_clustered(mddev) &&
(info->state & (1 << MD_DISK_CLUSTER_ADD)))
- md_cluster_ops->add_new_disk_finish(mddev);
+ md_cluster_ops->metadata_update_cancel(mddev);
return err;
}
@@ -8038,6 +8030,8 @@ static int remove_and_add_spares(struct mddev *mddev,
rdev_for_each(rdev, mddev) {
if (this && this != rdev)
continue;
+ if (test_bit(Candidate, &rdev->flags))
+ continue;
if (rdev->raid_disk >= 0 &&
!test_bit(In_sync, &rdev->flags) &&
!test_bit(Faulty, &rdev->flags))
@@ -8955,6 +8949,19 @@ static void check_sb_changes(struct mddev *mddev, struct md_rdev *rdev)
/* Check if the roles changed */
role = le16_to_cpu(sb->dev_roles[rdev2->desc_nr]);
+
+ if (test_bit(Candidate, &rdev2->flags)) {
+ if (role == 0xfffe) {
+ pr_info("md: Removing Candidate device %s because add failed\n", bdevname(rdev2->bdev,b));
+ md_kick_rdev_from_array(rdev2);
+ continue;
+ }
+ else {
+ clear_bit(Candidate, &rdev2->flags);
+ rdev2->raid_disk = -1;
+ }
+ }
+
if (role != rdev2->raid_disk) {
/* got activated */
if (rdev2->raid_disk == -1 && role != 0xffff) {
--
1.8.5.6
^ permalink raw reply related
* [PATCH 4/5] Perform resync/recovery under a DLM lock
From: rgoldwyn @ 2015-10-05 15:09 UTC (permalink / raw)
To: linux-raid, neilb; +Cc: Goldwyn Rodrigues
In-Reply-To: <1444057794-17256-1-git-send-email-rgoldwyn@suse.de>
From: Goldwyn Rodrigues <rgoldwyn@suse.com>
Resync or recovery must be performed by only one node at a time.
A DLM lock resource, resync_lockres provides the mutual exclusion
so that only one node performs the recovery/resync at a time.
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
---
drivers/md/md-cluster.c | 29 +++++++++++++++++++++++++++++
drivers/md/md-cluster.h | 1 +
drivers/md/md.c | 17 ++++++++++++++---
3 files changed, 44 insertions(+), 3 deletions(-)
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index 4645a93..17f24bf 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -46,6 +46,7 @@ struct resync_info {
#define MD_CLUSTER_WAITING_FOR_NEWDISK 1
#define MD_CLUSTER_SUSPEND_READ_BALANCING 2
#define MD_CLUSTER_BEGIN_JOIN_CLUSTER 3
+#define MD_CLUSTER_RECOVERY_NEEDED 4
struct md_cluster_info {
@@ -55,6 +56,7 @@ struct md_cluster_info {
struct completion completion;
struct mutex sb_mutex;
struct dlm_lock_resource *bitmap_lockres;
+ struct dlm_lock_resource *resync_lockres;
struct list_head suspend_list;
spinlock_t suspend_lock;
struct md_thread *recovery_thread;
@@ -382,6 +384,8 @@ static void process_suspend_info(struct mddev *mddev,
if (!hi) {
remove_suspend_info(cinfo, slot);
+ if (test_and_clear_bit(MD_CLUSTER_RECOVERY_NEEDED, &cinfo->state))
+ set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
return;
}
s = kzalloc(sizeof(struct suspend_info), GFP_KERNEL);
@@ -755,6 +759,10 @@ static int join(struct mddev *mddev, int nodes)
goto err;
}
+ cinfo->resync_lockres = lockres_init(mddev, "resync", NULL, 0);
+ if (!cinfo->resync_lockres)
+ goto err;
+
ret = gather_all_resync_info(mddev, nodes);
if (ret)
goto err;
@@ -765,6 +773,7 @@ err:
lockres_free(cinfo->token_lockres);
lockres_free(cinfo->ack_lockres);
lockres_free(cinfo->no_new_dev_lockres);
+ lockres_free(cinfo->resync_lockres);
lockres_free(cinfo->bitmap_lockres);
if (cinfo->lockspace)
dlm_release_lockspace(cinfo->lockspace, 2);
@@ -838,6 +847,21 @@ static int metadata_update_cancel(struct mddev *mddev)
return dlm_unlock_sync(cinfo->token_lockres);
}
+static int resync_start(struct mddev *mddev)
+{
+ struct md_cluster_info *cinfo = mddev->cluster_info;
+ int ret = 0;
+ cinfo->resync_lockres->flags |= DLM_LKF_NOQUEUE;
+ ret = dlm_lock_sync(cinfo->resync_lockres, DLM_LOCK_EX);
+ if (ret) {
+ if (!test_bit(MD_CLUSTER_RECOVERY_NEEDED, &cinfo->state))
+ pr_info("md-cluster: Postponing recovery until other nodes finish recovery\n");
+ set_bit(MD_CLUSTER_RECOVERY_NEEDED, &cinfo->state);
+ } else
+ clear_bit(MD_CLUSTER_RECOVERY_NEEDED, &cinfo->state);
+ return ret;
+}
+
static int resync_info_update(struct mddev *mddev, sector_t lo, sector_t hi)
{
struct md_cluster_info *cinfo = mddev->cluster_info;
@@ -854,6 +878,10 @@ static int resync_info_update(struct mddev *mddev, sector_t lo, sector_t hi)
cmsg.slot = cpu_to_le32(slot);
cmsg.low = cpu_to_le64(lo);
cmsg.high = cpu_to_le64(hi);
+
+ /* Resync is finished */
+ if (hi == 0)
+ dlm_unlock_sync(cinfo->resync_lockres);
return sendmsg(cinfo, &cmsg);
}
@@ -972,6 +1000,7 @@ static struct md_cluster_operations cluster_ops = {
.join = join,
.leave = leave,
.slot_number = slot_number,
+ .resync_start = resync_start,
.resync_info_update = resync_info_update,
.metadata_update_start = metadata_update_start,
.metadata_update_finish = metadata_update_finish,
diff --git a/drivers/md/md-cluster.h b/drivers/md/md-cluster.h
index f5bdc0c..7061020 100644
--- a/drivers/md/md-cluster.h
+++ b/drivers/md/md-cluster.h
@@ -16,6 +16,7 @@ struct md_cluster_operations {
int (*metadata_update_start)(struct mddev *mddev);
int (*metadata_update_finish)(struct mddev *mddev);
int (*metadata_update_cancel)(struct mddev *mddev);
+ int (*resync_start)(struct mddev *mddev);
int (*area_resyncing)(struct mddev *mddev, int direction, sector_t lo, sector_t hi);
int (*add_new_disk_start)(struct mddev *mddev, struct md_rdev *rdev);
int (*add_new_disk_finish)(struct mddev *mddev);
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 9abd070..75d14e3 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -8071,14 +8071,25 @@ no_add:
static void md_start_sync(struct work_struct *ws)
{
struct mddev *mddev = container_of(ws, struct mddev, del_work);
+ int ret = 0;
+
+ if (mddev_is_clustered(mddev)) {
+ ret = md_cluster_ops->resync_start(mddev);
+ if (ret) {
+ mddev->sync_thread = NULL;
+ goto out;
+ }
+ }
mddev->sync_thread = md_register_thread(md_do_sync,
mddev,
"resync");
+out:
if (!mddev->sync_thread) {
- printk(KERN_ERR "%s: could not start resync"
- " thread...\n",
- mdname(mddev));
+ if (!mddev_is_clustered(mddev))
+ printk(KERN_ERR "%s: could not start resync"
+ " thread...\n",
+ mdname(mddev));
/* leave the spares where they are, it shouldn't hurt */
clear_bit(MD_RECOVERY_SYNC, &mddev->recovery);
clear_bit(MD_RECOVERY_RESHAPE, &mddev->recovery);
--
1.8.5.6
^ permalink raw reply related
* [PATCH 3/5] md-cluster: Perform a lazy update
From: rgoldwyn @ 2015-10-05 15:09 UTC (permalink / raw)
To: linux-raid, neilb; +Cc: Goldwyn Rodrigues
In-Reply-To: <1444057794-17256-1-git-send-email-rgoldwyn@suse.de>
From: Goldwyn Rodrigues <rgoldwyn@suse.com>
In a clustered environment, a change such as marking a device faulty,
can be recorded by any of the nodes. This is communicated to all the
nodes and re-recording such a change is unnecessary, and quite often
pretty disruptive.
With this patch, just before the update, we detect for the changes
and if the changes are already in superblock, we abort the update
after clearing all the flags
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
---
drivers/md/md.c | 101 ++++++++++++++++++++++++++++++++------------------------
1 file changed, 57 insertions(+), 44 deletions(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 36e751e..9abd070 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -2199,6 +2199,46 @@ static void sync_sbs(struct mddev *mddev, int nospares)
}
}
+static bool does_sb_need_changing(struct mddev *mddev)
+{
+ struct md_rdev *rdev;
+ struct mdp_superblock_1 *sb;
+ int role;
+
+ /* Find a good rdev */
+ rdev_for_each_rcu(rdev, mddev)
+ if ((rdev->raid_disk >= 0) && !test_bit(Faulty, &rdev->flags))
+ break;
+
+ /* No good device found. */
+ if (!rdev)
+ return false;
+
+ sb = page_address(rdev->sb_page);
+ /* Check if a device has become faulty or a spare become active */
+ rdev_for_each(rdev, mddev) {
+ role = le16_to_cpu(sb->dev_roles[rdev->desc_nr]);
+ /* Device activated? */
+ if (role == 0xffff && rdev->raid_disk >=0 && !test_bit(Faulty, &rdev->flags))
+ return true;
+ /* Device turned faulty? */
+ if (test_bit(Faulty, &rdev->flags) && (role < 0xfffd))
+ return true;
+ }
+
+ /* Check if any mddev parameters have changed */
+ if ((mddev->dev_sectors != le64_to_cpu(sb->size)) ||
+ (mddev->reshape_position != le64_to_cpu(sb->reshape_position)) ||
+ (mddev->recovery_cp != le64_to_cpu(sb->resync_offset)) ||
+ (mddev->layout != le64_to_cpu(sb->layout)) ||
+ (mddev->raid_disks != le32_to_cpu(sb->raid_disks)) ||
+ (mddev->chunk_sectors != le32_to_cpu(sb->chunksize))
+ )
+ return true;
+
+ return false;
+}
+
void md_update_sb(struct mddev *mddev, int force_change)
{
struct md_rdev *rdev;
@@ -2211,6 +2251,18 @@ void md_update_sb(struct mddev *mddev, int force_change)
set_bit(MD_CHANGE_DEVS, &mddev->flags);
return;
}
+
+ if (mddev_is_clustered(mddev)) {
+ if (test_and_clear_bit(MD_CHANGE_DEVS, &mddev->flags))
+ force_change = 1;
+ md_cluster_ops->metadata_update_start(mddev);
+ /* Has someone else has updated the sb */
+ if (!does_sb_need_changing(mddev)) {
+ md_cluster_ops->metadata_update_cancel(mddev);
+ clear_bit(MD_CHANGE_PENDING, &mddev->flags);
+ return;
+ }
+ }
repeat:
/* First make sure individual recovery_offsets are correct */
rdev_for_each(rdev, mddev) {
@@ -2359,6 +2411,9 @@ repeat:
clear_bit(BlockedBadBlocks, &rdev->flags);
wake_up(&rdev->blocked_wait);
}
+
+ if (mddev_is_clustered(mddev))
+ md_cluster_ops->metadata_update_finish(mddev);
}
EXPORT_SYMBOL(md_update_sb);
@@ -2496,13 +2551,9 @@ state_store(struct md_rdev *rdev, const char *buf, size_t len)
if (mddev_is_clustered(mddev))
md_cluster_ops->remove_disk(mddev, rdev);
md_kick_rdev_from_array(rdev);
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_start(mddev);
if (mddev->pers)
md_update_sb(mddev, 1);
md_new_event(mddev);
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_finish(mddev);
err = 0;
}
} else if (cmd_match(buf, "writemostly")) {
@@ -4063,12 +4114,8 @@ size_store(struct mddev *mddev, const char *buf, size_t len)
if (err)
return err;
if (mddev->pers) {
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_start(mddev);
err = update_size(mddev, sectors);
md_update_sb(mddev, 1);
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_finish(mddev);
} else {
if (mddev->dev_sectors == 0 ||
mddev->dev_sectors > sectors)
@@ -5306,8 +5353,6 @@ static void md_clean(struct mddev *mddev)
static void __md_stop_writes(struct mddev *mddev)
{
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_start(mddev);
set_bit(MD_RECOVERY_FROZEN, &mddev->recovery);
flush_workqueue(md_misc_wq);
if (mddev->sync_thread) {
@@ -5326,8 +5371,6 @@ static void __md_stop_writes(struct mddev *mddev)
mddev->in_sync = 1;
md_update_sb(mddev, 1);
}
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_finish(mddev);
}
void md_stop_writes(struct mddev *mddev)
@@ -6011,9 +6054,6 @@ static int hot_remove_disk(struct mddev *mddev, dev_t dev)
md_update_sb(mddev, 1);
md_new_event(mddev);
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_finish(mddev);
-
return 0;
busy:
printk(KERN_WARNING "md: cannot remove active disk %s from %s ...\n",
@@ -6066,14 +6106,12 @@ static int hot_add_disk(struct mddev *mddev, dev_t dev)
goto abort_export;
}
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_start(mddev);
clear_bit(In_sync, &rdev->flags);
rdev->desc_nr = -1;
rdev->saved_raid_disk = -1;
err = bind_rdev_to_array(rdev, mddev);
if (err)
- goto abort_clustered;
+ goto abort_export;
/*
* The rest should better be atomic, we can have disk failures
@@ -6083,9 +6121,6 @@ static int hot_add_disk(struct mddev *mddev, dev_t dev)
rdev->raid_disk = -1;
md_update_sb(mddev, 1);
-
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_finish(mddev);
/*
* Kick recovery, maybe this spare has to be added to the
* array immediately.
@@ -6095,9 +6130,6 @@ static int hot_add_disk(struct mddev *mddev, dev_t dev)
md_new_event(mddev);
return 0;
-abort_clustered:
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_cancel(mddev);
abort_export:
export_rdev(rdev);
return err;
@@ -6415,8 +6447,6 @@ static int update_array_info(struct mddev *mddev, mdu_array_info_t *info)
return rv;
}
}
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_start(mddev);
if (info->size >= 0 && mddev->dev_sectors / 2 != info->size)
rv = update_size(mddev, (sector_t)info->size * 2);
@@ -6474,12 +6504,8 @@ static int update_array_info(struct mddev *mddev, mdu_array_info_t *info)
}
}
md_update_sb(mddev, 1);
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_finish(mddev);
return rv;
err:
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_cancel(mddev);
return rv;
}
@@ -7592,11 +7618,7 @@ int md_allow_write(struct mddev *mddev)
mddev->safemode == 0)
mddev->safemode = 1;
spin_unlock(&mddev->lock);
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_start(mddev);
md_update_sb(mddev, 0);
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_finish(mddev);
sysfs_notify_dirent_safe(mddev->sysfs_state);
} else
spin_unlock(&mddev->lock);
@@ -8175,13 +8197,8 @@ void md_check_recovery(struct mddev *mddev)
sysfs_notify_dirent_safe(mddev->sysfs_state);
}
- if (mddev->flags & MD_UPDATE_SB_FLAGS) {
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_start(mddev);
+ if (mddev->flags & MD_UPDATE_SB_FLAGS)
md_update_sb(mddev, 0);
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_finish(mddev);
- }
if (test_bit(MD_RECOVERY_RUNNING, &mddev->recovery) &&
!test_bit(MD_RECOVERY_DONE, &mddev->recovery)) {
@@ -8279,8 +8296,6 @@ void md_reap_sync_thread(struct mddev *mddev)
set_bit(MD_CHANGE_DEVS, &mddev->flags);
}
}
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_start(mddev);
if (test_bit(MD_RECOVERY_RESHAPE, &mddev->recovery) &&
mddev->pers->finish_reshape)
mddev->pers->finish_reshape(mddev);
@@ -8293,8 +8308,6 @@ void md_reap_sync_thread(struct mddev *mddev)
rdev->saved_raid_disk = -1;
md_update_sb(mddev, 1);
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_finish(mddev);
clear_bit(MD_RECOVERY_RUNNING, &mddev->recovery);
clear_bit(MD_RECOVERY_DONE, &mddev->recovery);
clear_bit(MD_RECOVERY_SYNC, &mddev->recovery);
--
1.8.5.6
^ permalink raw reply related
* [PATCH 2/5] md-cluster: Improve md_reload_sb to be less error prone
From: rgoldwyn @ 2015-10-05 15:09 UTC (permalink / raw)
To: linux-raid, neilb; +Cc: Goldwyn Rodrigues
In-Reply-To: <1444057794-17256-1-git-send-email-rgoldwyn@suse.de>
From: Goldwyn Rodrigues <rgoldwyn@suse.com>
md_reload_sb is too simplistic and it explicitly needs to determine
the changes made by the writing node. However, there are multiple areas
where a simple reload could fail.
Instead, read the superblock of one of the "good" rdevs and update
the necessary information:
- read the superblock into a newly allocated page, by temporarily
swapping out rdev->sb_page and calling ->load_super.
- if that fails return
- if it succeeds, call check_sb_changes
1. iterates over list of active devices and checks the matching
dev_roles[] value.
If that is 'faulty', the device must be marked as faulty
- call md_error to mark the device as faulty. Make sure
not to set CHANGE_DEVS and wakeup mddev->thread or else
it would initiate a resync process, which is the responsibility
of the "primary" node.
- clear the Blocked bit
- Call remove_and_add_spares() to hot remove the device.
If the device is 'spare':
- call remove_and_add_spares() to get the number of spares
added in this operation.
- Reduce mddev->degraded to mark the array as not degraded.
2. reset recovery_cp
- read the rest of the rdevs to update recovery_offset
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
---
drivers/md/md-cluster.c | 27 ++++++-----
drivers/md/md.c | 120 ++++++++++++++++++++++++++++++++++++++++++------
drivers/md/md.h | 2 +-
drivers/md/raid1.c | 2 +
4 files changed, 124 insertions(+), 27 deletions(-)
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index b94a2e6..4645a93 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -424,8 +424,7 @@ static void process_add_new_disk(struct mddev *mddev, struct cluster_msg *cmsg)
static void process_metadata_update(struct mddev *mddev, struct cluster_msg *msg)
{
struct md_cluster_info *cinfo = mddev->cluster_info;
-
- md_reload_sb(mddev);
+ md_reload_sb(mddev, le32_to_cpu(msg->raid_slot));
dlm_lock_sync(cinfo->no_new_dev_lockres, DLM_LOCK_CR);
}
@@ -811,11 +810,23 @@ static int metadata_update_finish(struct mddev *mddev)
{
struct md_cluster_info *cinfo = mddev->cluster_info;
struct cluster_msg cmsg;
- int ret;
+ struct md_rdev *rdev;
+ int ret = 0;
memset(&cmsg, 0, sizeof(cmsg));
cmsg.type = cpu_to_le32(METADATA_UPDATED);
- ret = __sendmsg(cinfo, &cmsg);
+ cmsg.raid_slot = -1;
+ /* Pick up a good active device number to send.
+ */
+ rdev_for_each(rdev, mddev)
+ if (rdev->raid_disk > -1 && !test_bit(Faulty, &rdev->flags)) {
+ cmsg.raid_slot = cpu_to_le32(rdev->desc_nr);
+ break;
+ }
+ if (cmsg.raid_slot >= 0)
+ ret = __sendmsg(cinfo, &cmsg);
+ else
+ pr_warn("md-cluster: No good device id found to send\n");
unlock_comm(cinfo);
return ret;
}
@@ -899,15 +910,9 @@ static int add_new_disk_start(struct mddev *mddev, struct md_rdev *rdev)
static int add_new_disk_finish(struct mddev *mddev)
{
- struct cluster_msg cmsg;
- struct md_cluster_info *cinfo = mddev->cluster_info;
- int ret;
/* Write sb and inform others */
md_update_sb(mddev, 1);
- cmsg.type = METADATA_UPDATED;
- ret = __sendmsg(cinfo, &cmsg);
- unlock_comm(cinfo);
- return ret;
+ return metadata_update_finish(mddev);
}
static int new_disk_ack(struct mddev *mddev, bool ack)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index fcdf47b..36e751e 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -7987,7 +7987,7 @@ void md_do_sync(struct md_thread *thread)
EXPORT_SYMBOL_GPL(md_do_sync);
static int remove_and_add_spares(struct mddev *mddev,
- struct md_rdev *this)
+ struct md_rdev *this)
{
struct md_rdev *rdev;
int spares = 0;
@@ -8917,25 +8917,115 @@ err_wq:
return ret;
}
-void md_reload_sb(struct mddev *mddev)
+static void check_sb_changes(struct mddev *mddev, struct md_rdev *rdev)
{
- struct md_rdev *rdev, *tmp;
+ struct mdp_superblock_1 *sb = page_address(rdev->sb_page);
+ struct md_rdev *rdev2;
+ int role, ret;
+ char b[BDEVNAME_SIZE];
- rdev_for_each_safe(rdev, tmp, mddev) {
- rdev->sb_loaded = 0;
- ClearPageUptodate(rdev->sb_page);
+ /* Check for change of roles in the active devices */
+ rdev_for_each(rdev2, mddev) {
+ if (test_bit(Faulty, &rdev2->flags))
+ continue;
+
+ /* Check if the roles changed */
+ role = le16_to_cpu(sb->dev_roles[rdev2->desc_nr]);
+ if (role != rdev2->raid_disk) {
+ /* got activated */
+ if (rdev2->raid_disk == -1 && role != 0xffff) {
+ rdev2->saved_raid_disk = role;
+ ret = remove_and_add_spares(mddev, rdev2);
+ pr_info("Activated spare: %s\n",
+ bdevname(rdev2->bdev,b));
+ continue;
+ }
+ /* device faulty
+ * We just want to do the minimum to mark the disk
+ * as faulty. The recovery is performed by the
+ * one who initiated the error.
+ */
+ if ((role == 0xfffe) || (role == 0xfffd)) {
+ md_error(mddev, rdev2);
+ clear_bit(Blocked, &rdev2->flags);
+ remove_and_add_spares(mddev, rdev2);
+ }
+ }
}
- mddev->raid_disks = 0;
- analyze_sbs(mddev);
- rdev_for_each_safe(rdev, tmp, mddev) {
- struct mdp_superblock_1 *sb = page_address(rdev->sb_page);
- /* since we don't write to faulty devices, we figure out if the
- * disk is faulty by comparing events
- */
- if (mddev->events > sb->events)
- set_bit(Faulty, &rdev->flags);
+
+ /* recovery_cp changed */
+ if (le64_to_cpu(sb->resync_offset) != mddev->recovery_cp) {
+ pr_info("%s:%d recovery_cp changed from %lu to %lu\n", __func__,
+ __LINE__, mddev->recovery_cp,
+ (unsigned long) le64_to_cpu(sb->resync_offset));
+ mddev->recovery_cp = le64_to_cpu(sb->resync_offset);
}
+ /* Finally set the event to be up to date */
+ mddev->events = le64_to_cpu(sb->events);
+}
+
+static int read_rdev(struct mddev *mddev, struct md_rdev *rdev)
+{
+ int err;
+ struct page *swapout = rdev->sb_page;
+ struct mdp_superblock_1 *sb;
+
+ /* Store the sb page of the rdev in the swapout temporary
+ * variable in case we err in the future
+ */
+ rdev->sb_page = NULL;
+ alloc_disk_sb(rdev);
+ ClearPageUptodate(rdev->sb_page);
+ rdev->sb_loaded = 0;
+ err = super_types[mddev->major_version].load_super(rdev, NULL, mddev->minor_version);
+
+ if (err < 0) {
+ pr_warn("%s: %d Could not reload rdev(%d) err: %d. Restoring old values\n",
+ __func__, __LINE__, rdev->desc_nr, err);
+ put_page(rdev->sb_page);
+ rdev->sb_page = swapout;
+ rdev->sb_loaded = 1;
+ return err;
+ }
+
+ sb = page_address(rdev->sb_page);
+ rdev->recovery_offset = le64_to_cpu(sb->recovery_offset);
+
+ put_page(swapout);
+ return 0;
+}
+
+void md_reload_sb(struct mddev *mddev, int nr)
+{
+ struct md_rdev *rdev;
+ int err;
+
+ /* Find the rdev */
+ rdev_for_each_rcu(rdev, mddev) {
+ if (rdev->desc_nr == nr)
+ break;
+ }
+
+ if (!rdev || rdev->desc_nr != nr) {
+ pr_warn("%s: %d Could not find rdev with nr %d\n", __func__, __LINE__, nr);
+ return;
+ }
+
+ err = read_rdev(mddev, rdev);
+ if (err < 0)
+ return;
+
+ check_sb_changes(mddev, rdev);
+
+ /* Read the rest of the rdev's to update recovery_offset */
+ rdev_for_each_rcu(rdev, mddev) {
+ /* We have already read this one */
+ if (rdev->desc_nr == nr)
+ continue;
+
+ read_rdev(mddev, rdev);
+ }
}
EXPORT_SYMBOL(md_reload_sb);
diff --git a/drivers/md/md.h b/drivers/md/md.h
index ab33957..2ea0035 100644
--- a/drivers/md/md.h
+++ b/drivers/md/md.h
@@ -658,7 +658,7 @@ extern struct bio *bio_alloc_mddev(gfp_t gfp_mask, int nr_iovecs,
struct mddev *mddev);
extern void md_unplug(struct blk_plug_cb *cb, bool from_schedule);
-extern void md_reload_sb(struct mddev *mddev);
+extern void md_reload_sb(struct mddev *mddev, int raid_disk);
extern void md_update_sb(struct mddev *mddev, int force);
extern void md_kick_rdev_from_array(struct md_rdev * rdev);
struct md_rdev *md_find_rdev_nr_rcu(struct mddev *mddev, int nr);
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index 1dd13bb..03752bd 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -1608,6 +1608,8 @@ static int raid1_add_disk(struct mddev *mddev, struct md_rdev *rdev)
*/
if (rdev->saved_raid_disk < 0)
conf->fullsync = 1;
+ else if (mddev_is_clustered(mddev))
+ rdev->raid_disk = rdev->saved_raid_disk;
rcu_assign_pointer(p->rdev, rdev);
break;
}
--
1.8.5.6
^ permalink raw reply related
* [PATCH 1/5] remove_and_add_spares() to activate specific rdev
From: rgoldwyn @ 2015-10-05 15:09 UTC (permalink / raw)
To: linux-raid, neilb; +Cc: Goldwyn Rodrigues
In-Reply-To: <1444057794-17256-1-git-send-email-rgoldwyn@suse.de>
From: Goldwyn Rodrigues <rgoldwyn@suse.com>
remove_and_add_spares() checks for all devices to activate spare.
Change it to activate a specific device if a non-null rdev
argument is passed.
remove_and_add_spares() can be used to activate spares in
slot_store() as well.
For hot_remove_disk(), check if rdev->raid_disk == -1 before
calling remove_and_add_spares()
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
---
drivers/md/md.c | 24 +++++++++---------------
1 file changed, 9 insertions(+), 15 deletions(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 9798a99..fcdf47b 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -2691,15 +2691,9 @@ slot_store(struct md_rdev *rdev, const char *buf, size_t len)
rdev->saved_raid_disk = -1;
clear_bit(In_sync, &rdev->flags);
clear_bit(Bitmap_sync, &rdev->flags);
- err = rdev->mddev->pers->
- hot_add_disk(rdev->mddev, rdev);
- if (err) {
- rdev->raid_disk = -1;
- return err;
- } else
- sysfs_notify_dirent_safe(rdev->sysfs_state);
- if (sysfs_link_rdev(rdev->mddev, rdev))
- /* failure here is OK */;
+ remove_and_add_spares(rdev->mddev, rdev);
+ if (rdev->raid_disk == -1)
+ return -EBUSY;
/* don't wakeup anyone, leave that to userspace. */
} else {
if (slot >= rdev->mddev->raid_disks &&
@@ -6001,15 +5995,15 @@ static int hot_remove_disk(struct mddev *mddev, dev_t dev)
if (!rdev)
return -ENXIO;
+ if (rdev->raid_disk >= 0)
+ goto busy;
+
if (mddev_is_clustered(mddev))
md_cluster_ops->metadata_update_start(mddev);
clear_bit(Blocked, &rdev->flags);
remove_and_add_spares(mddev, rdev);
- if (rdev->raid_disk >= 0)
- goto busy;
-
if (mddev_is_clustered(mddev))
md_cluster_ops->remove_disk(mddev, rdev);
@@ -6022,8 +6016,6 @@ static int hot_remove_disk(struct mddev *mddev, dev_t dev)
return 0;
busy:
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_cancel(mddev);
printk(KERN_WARNING "md: cannot remove active disk %s from %s ...\n",
bdevname(rdev->bdev,b), mdname(mddev));
return -EBUSY;
@@ -8018,10 +8010,12 @@ static int remove_and_add_spares(struct mddev *mddev,
if (removed && mddev->kobj.sd)
sysfs_notify(&mddev->kobj, NULL, "degraded");
- if (this)
+ if (this && removed)
goto no_add;
rdev_for_each(rdev, mddev) {
+ if (this && this != rdev)
+ continue;
if (rdev->raid_disk >= 0 &&
!test_bit(In_sync, &rdev->flags) &&
!test_bit(Faulty, &rdev->flags))
--
1.8.5.6
^ permalink raw reply related
* [PATCH 0/5 v2] md-cluster: A better way for METADATA_UPDATED processing
From: rgoldwyn @ 2015-10-05 15:09 UTC (permalink / raw)
To: linux-raid, neilb
The processing of METADATA_UPDATED message is too simple and prone to
errors. Besides, it would not update the internal data structures as
required.
This set of patches reads the superblock from one of the device of the MD
and checks for changes in the in-memory data structures. If there is a change,
it performs the necessary actions to keep the internal data structures
as it would be in the primary node.
An example is if a devices turns faulty. The algorithm is:
1. The initiator node marks the device as faulty and updates the superblock
2. The initiator node sends METADATA_UPDATED with an advisory device number to the rest of the nodes.
3. The receiving node on receiving the METADATA_UPDATED message
3.1 Reads the superblock
3.2 Detects a device has failed by comparing with memory structure
3.3 Calls the necessary functions to record the failure and get the device out of the active array.
3.4 Acknowledges the message.
The patch series also fixes adding the disk which was impacted because of
the changes.
Patches can also be found at
https://github.com/goldwynr/linux branch md-next
md-cluster.c | 56 +++++++++--
md-cluster.h | 1
md.c | 293 +++++++++++++++++++++++++++++++++++++++++------------------
md.h | 2
raid1.c | 2
5 files changed, 253 insertions(+), 101 deletions(-)
^ permalink raw reply
* Re: [PATCH 1/5] fs: Verify access of user towards block device file when mounting
From: Seth Forshee @ 2015-10-05 14:26 UTC (permalink / raw)
To: Jan Kara
Cc: Eric W. Biederman, Mike Snitzer, Kent Overstreet, Alasdair Kergon,
dm-devel, Neil Brown, David Woodhouse, Brian Norris,
Alexander Viro, Jan Kara, Jeff Layton, J. Bruce Fields,
Serge Hallyn, Andy Lutomirski, linux-fsdevel,
linux-security-module, selinux, linux-kernel, linux-mtd,
linux-bcache, linux-raid
In-Reply-To: <20151001230700.GA10087@quack.suse.cz>
On Fri, Oct 02, 2015 at 01:07:00AM +0200, Jan Kara wrote:
> On Thu 01-10-15 10:55:50, Eric W. Biederman wrote:
> > The goal if possible is to run things like docker without needed to be
> > root or even more fun to run docker in a container, and in general
> > enable nested containers.
>
> Frankly at the filesystem side we are rather far from being able to safely
> mount untrusted device and I don't think we'll ever be robust enough to
> tolerate e.g. user changing the disk while fs is using it. So would this be
> FUSE-only thing or is someone still hoping that general purpose filesystems
> will be robust enough in future?
FUSE will almost certainly be first. I've also been working with ext4,
and I would like to see that eventually supported to some degree.
Seth
^ permalink raw reply
* [PATCH RESEND] imsm: don't call abort_reshape() in imsm_manage_reshape()
From: Artur Paszkiewicz @ 2015-10-05 13:18 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, Artur Paszkiewicz, Konrad Dabrowski
Calling abort_reshape() in imsm_manage_reshape() is unnecessary in case
of an error because it is handled by reshape_array(). Calling it when
reshape completes successfully is also unnecessary and leads to a race
condition:
- reshape ends
- mdadm calls abort_reshape() -> sets sync_action to idle
- MD_RECOVERY_INTR is set and md_reap_sync_thread() does not finish the
reshape
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Signed-off-by: Konrad Dabrowski <konrad.dabrowski@intel.com>
---
super-intel.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/super-intel.c b/super-intel.c
index 95a72b6..e609e0c 100644
--- a/super-intel.c
+++ b/super-intel.c
@@ -10601,7 +10601,6 @@ static int imsm_manage_reshape(
ret_val = 1;
abort:
free(buf);
- abort_reshape(sra);
return ret_val;
}
--
2.1.4
^ permalink raw reply related
* [PATCH] bcache: Really show state of work pending bit
From: Petr Mladek @ 2015-10-05 12:39 UTC (permalink / raw)
To: Kent Overstreet
Cc: Tejun Heo, Neil Brown, Jiri Kosina, linux-bcache, linux-raid,
linux-kernel, Petr Mladek
WORK_STRUCT_PENDING is a mask for testing the pending bit.
test_bit() expects the number of the bit and we need to
use WORK_STRUCT_PENDING_BIT there.
Also work_data_bits() is defined in workqueues.h now.
I have noticed this just by chance when looking how
WORK_STRUCT_PENDING_BIT is used. The change is compile
tested.
Signed-off-by: Petr Mladek <pmladek@suse.com>
---
drivers/md/bcache/closure.c | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/drivers/md/bcache/closure.c b/drivers/md/bcache/closure.c
index 7a228de95fd7..9eaf1d6e8302 100644
--- a/drivers/md/bcache/closure.c
+++ b/drivers/md/bcache/closure.c
@@ -167,8 +167,6 @@ EXPORT_SYMBOL(closure_debug_destroy);
static struct dentry *debug;
-#define work_data_bits(work) ((unsigned long *)(&(work)->data))
-
static int debug_seq_show(struct seq_file *f, void *data)
{
struct closure *cl;
@@ -182,7 +180,7 @@ static int debug_seq_show(struct seq_file *f, void *data)
r & CLOSURE_REMAINING_MASK);
seq_printf(f, "%s%s%s%s\n",
- test_bit(WORK_STRUCT_PENDING,
+ test_bit(WORK_STRUCT_PENDING_BIT,
work_data_bits(&cl->work)) ? "Q" : "",
r & CLOSURE_RUNNING ? "R" : "",
r & CLOSURE_STACK ? "S" : "",
--
1.8.5.6
^ permalink raw reply related
* Problem w/ commit ac8fa4196d20 on older, slower hardware
From: Joshua Kinard @ 2015-10-05 7:41 UTC (permalink / raw)
To: linux-raid
Per commit ac8fa4196d20:
> md: allow resync to go faster when there is competing IO.
>
> When md notices non-sync IO happening while it is trying to resync (or
> reshape or recover) it slows down to the set minimum.
>
> The default minimum might have made sense many years ago but the drives have
> become faster. Changing the default to match the times isn't really a long
> term solution.
This holds true for modern hardware, but this commit is causing problems on
older hardware, like SGI MIPS platforms, that use mdraid. Namely, while trying
to chase down an unrelated hardlock bug on an Onyx2, one of the arrays got out
of sync, so on the next reboot, mdraid's attempt to resync at full speed
absolutely murdered interactivity. It took close to 30mins for the system to
finally reach the login prompt.
Revert this patch was working to mitigate the problem at first, but it appears
that in recent kernels, this is no longer the case, and reverting this commit
has no noticeable effect anymore. I assume I'd have to hunt down newer commits
to revert, but it's probably saner to just highlight the problem and test any
proposed solutions.
Is there some way to resolve this in such a way that old hardware maintains
some level of interactivity during a resync, but that won't inconvenience the
more modern systems?
http://git.linux-mips.org/cgit/ralf/linux.git/commit/?id=ac8fa4196d20
Thanks!,
--J
^ permalink raw reply
* [PATCH 11/11] raid5-cache: use bio chaining
From: Christoph Hellwig @ 2015-10-05 7:31 UTC (permalink / raw)
To: Shaohua Li, neilb; +Cc: dan.j.williams, Kernel-team, linux-raid
In-Reply-To: <1444030276-30850-1-git-send-email-hch@lst.de>
Simplify the bio completion handler by using bio chaining and submitting
bios as soon as they are full.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
drivers/md/raid5-cache.c | 38 ++++++++++++++++----------------------
1 file changed, 16 insertions(+), 22 deletions(-)
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index e64f94e..040405e 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -100,8 +100,6 @@ struct r5l_io_unit {
struct page *meta_page; /* store meta block */
int meta_offset; /* current offset in meta_page */
- struct bio_list bios;
- atomic_t pending_io; /* pending bios not written to log yet */
struct bio *current_bio;/* current_bio accepting new data */
atomic_t pending_stripe;/* how many stripes not flushed to raid */
@@ -112,6 +110,7 @@ struct r5l_io_unit {
struct list_head stripe_list; /* stripes added to the io_unit */
int state;
+ bool need_split_bio;
};
/* r5l_io_unit state */
@@ -217,9 +216,6 @@ static void r5l_log_endio(struct bio *bio)
bio_put(bio);
- if (!atomic_dec_and_test(&io->pending_io))
- return;
-
spin_lock_irqsave(&log->io_list_lock, flags);
__r5l_set_io_unit_state(io, IO_UNIT_IO_END);
if (log->need_cache_flush)
@@ -237,7 +233,6 @@ static void r5l_submit_current_io(struct r5l_log *log)
{
struct r5l_io_unit *io = log->current_io;
struct r5l_meta_block *block;
- struct bio *bio;
unsigned long flags;
u32 crc;
@@ -254,22 +249,17 @@ static void r5l_submit_current_io(struct r5l_log *log)
__r5l_set_io_unit_state(io, IO_UNIT_IO_START);
spin_unlock_irqrestore(&log->io_list_lock, flags);
- while ((bio = bio_list_pop(&io->bios)))
- submit_bio(WRITE, bio);
+ submit_bio(WRITE, io->current_bio);
}
-static struct bio *r5l_bio_alloc(struct r5l_log *log, struct r5l_io_unit *io)
+static struct bio *r5l_bio_alloc(struct r5l_log *log)
{
struct bio *bio = bio_kmalloc(GFP_NOIO | __GFP_NOFAIL, BIO_MAX_PAGES);
bio->bi_rw = WRITE;
bio->bi_bdev = log->rdev->bdev;
bio->bi_iter.bi_sector = log->rdev->data_offset + log->log_start;
- bio->bi_end_io = r5l_log_endio;
- bio->bi_private = io;
- bio_list_add(&io->bios, bio);
- atomic_inc(&io->pending_io);
return bio;
}
@@ -285,7 +275,7 @@ static void r5_reserve_log_entry(struct r5l_log *log, struct r5l_io_unit *io)
* of BLOCK_SECTORS.
*/
if (log->log_start == 0)
- io->current_bio = NULL;
+ io->need_split_bio = true;
io->log_end = log->log_start;
}
@@ -298,7 +288,6 @@ static struct r5l_io_unit *r5l_new_meta(struct r5l_log *log)
/* We can't handle memory allocate failure so far */
io = kmem_cache_zalloc(log->io_kc, GFP_NOIO | __GFP_NOFAIL);
io->log = log;
- bio_list_init(&io->bios);
INIT_LIST_HEAD(&io->log_sibling);
INIT_LIST_HEAD(&io->stripe_list);
io->state = IO_UNIT_RUNNING;
@@ -314,7 +303,9 @@ static struct r5l_io_unit *r5l_new_meta(struct r5l_log *log)
io->meta_offset = sizeof(struct r5l_meta_block);
io->seq = log->seq++;
- io->current_bio = r5l_bio_alloc(log, io);
+ io->current_bio = r5l_bio_alloc(log);
+ io->current_bio->bi_end_io = r5l_log_endio;
+ io->current_bio->bi_private = io;
bio_add_page(io->current_bio, io->meta_page, PAGE_SIZE, 0);
r5_reserve_log_entry(log, io);
@@ -363,15 +354,18 @@ static void r5l_append_payload_page(struct r5l_log *log, struct page *page)
{
struct r5l_io_unit *io = log->current_io;
-alloc_bio:
- if (!io->current_bio)
- io->current_bio = r5l_bio_alloc(log, io);
+ if (io->need_split_bio) {
+ struct bio *prev = io->current_bio;
- if (!bio_add_page(io->current_bio, page, PAGE_SIZE, 0)) {
- io->current_bio = NULL;
- goto alloc_bio;
+ io->current_bio = r5l_bio_alloc(log);
+ bio_chain(io->current_bio, prev);
+
+ submit_bio(WRITE, prev);
}
+ if (!bio_add_page(io->current_bio, page, PAGE_SIZE, 0))
+ BUG();
+
r5_reserve_log_entry(log, io);
}
--
1.9.1
^ permalink raw reply related
* [PATCH 10/11] raid5-cache: small log->seq cleanup
From: Christoph Hellwig @ 2015-10-05 7:31 UTC (permalink / raw)
To: Shaohua Li, neilb; +Cc: dan.j.williams, Kernel-team, linux-raid
In-Reply-To: <1444030276-30850-1-git-send-email-hch@lst.de>
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
drivers/md/raid5-cache.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index 87fa3b5..e64f94e 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -312,12 +312,11 @@ static struct r5l_io_unit *r5l_new_meta(struct r5l_log *log)
io->log_start = log->log_start;
io->meta_offset = sizeof(struct r5l_meta_block);
- io->seq = log->seq;
+ io->seq = log->seq++;
io->current_bio = r5l_bio_alloc(log, io);
bio_add_page(io->current_bio, io->meta_page, PAGE_SIZE, 0);
- log->seq++;
r5_reserve_log_entry(log, io);
spin_lock_irq(&log->io_list_lock);
--
1.9.1
^ permalink raw reply related
* [PATCH 09/11] raid5-cache: new helper: r5_reserve_log_entry
From: Christoph Hellwig @ 2015-10-05 7:31 UTC (permalink / raw)
To: Shaohua Li, neilb; +Cc: dan.j.williams, Kernel-team, linux-raid
In-Reply-To: <1444030276-30850-1-git-send-email-hch@lst.de>
Factor out code to reserve log space.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
drivers/md/raid5-cache.c | 30 +++++++++++++++++++-----------
1 file changed, 19 insertions(+), 11 deletions(-)
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index b716c41..87fa3b5 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -273,6 +273,23 @@ static struct bio *r5l_bio_alloc(struct r5l_log *log, struct r5l_io_unit *io)
return bio;
}
+static void r5_reserve_log_entry(struct r5l_log *log, struct r5l_io_unit *io)
+{
+ log->log_start = r5l_ring_add(log, log->log_start, BLOCK_SECTORS);
+
+ /*
+ * If we filled up the log device start from the beginning again,
+ * which will require a new bio.
+ *
+ * Note: for this to work properly the log size needs to me a multiple
+ * of BLOCK_SECTORS.
+ */
+ if (log->log_start == 0)
+ io->current_bio = NULL;
+
+ io->log_end = log->log_start;
+}
+
static struct r5l_io_unit *r5l_new_meta(struct r5l_log *log)
{
struct r5l_io_unit *io;
@@ -301,11 +318,7 @@ static struct r5l_io_unit *r5l_new_meta(struct r5l_log *log)
bio_add_page(io->current_bio, io->meta_page, PAGE_SIZE, 0);
log->seq++;
- log->log_start = r5l_ring_add(log, log->log_start, BLOCK_SECTORS);
- io->log_end = log->log_start;
- /* current bio hit disk end */
- if (log->log_start == 0)
- io->current_bio = NULL;
+ r5_reserve_log_entry(log, io);
spin_lock_irq(&log->io_list_lock);
list_add_tail(&io->log_sibling, &log->running_ios);
@@ -359,13 +372,8 @@ alloc_bio:
io->current_bio = NULL;
goto alloc_bio;
}
- log->log_start = r5l_ring_add(log, log->log_start,
- BLOCK_SECTORS);
- /* current bio hit disk end */
- if (log->log_start == 0)
- io->current_bio = NULL;
- io->log_end = log->log_start;
+ r5_reserve_log_entry(log, io);
}
static void r5l_log_stripe(struct r5l_log *log, struct stripe_head *sh,
--
1.9.1
^ permalink raw reply related
* [PATCH 08/11] raid5-cache: inline r5l_alloc_io_unit into r5l_new_meta
From: Christoph Hellwig @ 2015-10-05 7:31 UTC (permalink / raw)
To: Shaohua Li, neilb; +Cc: dan.j.williams, Kernel-team, linux-raid
In-Reply-To: <1444030276-30850-1-git-send-email-hch@lst.de>
This is the only user, and keeping all code initializing the io_unit
structure together improves readbility.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
drivers/md/raid5-cache.c | 26 ++++++++------------------
1 file changed, 8 insertions(+), 18 deletions(-)
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index f3958e5..b716c41 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -150,23 +150,6 @@ static bool r5l_has_free_space(struct r5l_log *log, sector_t size)
return log->device_size > used_size + size;
}
-static struct r5l_io_unit *r5l_alloc_io_unit(struct r5l_log *log)
-{
- struct r5l_io_unit *io;
- /* We can't handle memory allocate failure so far */
- gfp_t gfp = GFP_NOIO | __GFP_NOFAIL;
-
- io = kmem_cache_zalloc(log->io_kc, gfp);
- io->log = log;
- io->meta_page = alloc_page(gfp | __GFP_ZERO);
-
- bio_list_init(&io->bios);
- INIT_LIST_HEAD(&io->log_sibling);
- INIT_LIST_HEAD(&io->stripe_list);
- io->state = IO_UNIT_RUNNING;
- return io;
-}
-
static void r5l_free_io_unit(struct r5l_log *log, struct r5l_io_unit *io)
{
__free_page(io->meta_page);
@@ -295,8 +278,15 @@ static struct r5l_io_unit *r5l_new_meta(struct r5l_log *log)
struct r5l_io_unit *io;
struct r5l_meta_block *block;
- io = r5l_alloc_io_unit(log);
+ /* We can't handle memory allocate failure so far */
+ io = kmem_cache_zalloc(log->io_kc, GFP_NOIO | __GFP_NOFAIL);
+ io->log = log;
+ bio_list_init(&io->bios);
+ INIT_LIST_HEAD(&io->log_sibling);
+ INIT_LIST_HEAD(&io->stripe_list);
+ io->state = IO_UNIT_RUNNING;
+ io->meta_page = alloc_page(GFP_NOIO | __GFP_NOFAIL | __GFP_ZERO);
block = page_address(io->meta_page);
block->magic = cpu_to_le32(R5LOG_MAGIC);
block->version = R5LOG_VERSION;
--
1.9.1
^ permalink raw reply related
* [PATCH 07/11] raid5-cache: take rdev->data_offset into account early on
From: Christoph Hellwig @ 2015-10-05 7:31 UTC (permalink / raw)
To: Shaohua Li, neilb; +Cc: dan.j.williams, Kernel-team, linux-raid
In-Reply-To: <1444030276-30850-1-git-send-email-hch@lst.de>
Set up bi_sector properly when we allocate an bio instead of updating it
at submission time.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
drivers/md/raid5-cache.c | 7 ++-----
1 file changed, 2 insertions(+), 5 deletions(-)
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index 4d4dc4d..f3958e5 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -271,11 +271,8 @@ static void r5l_submit_current_io(struct r5l_log *log)
__r5l_set_io_unit_state(io, IO_UNIT_IO_START);
spin_unlock_irqrestore(&log->io_list_lock, flags);
- while ((bio = bio_list_pop(&io->bios))) {
- /* all IO must start from rdev->data_offset */
- bio->bi_iter.bi_sector += log->rdev->data_offset;
+ while ((bio = bio_list_pop(&io->bios)))
submit_bio(WRITE, bio);
- }
}
static struct bio *r5l_bio_alloc(struct r5l_log *log, struct r5l_io_unit *io)
@@ -284,7 +281,7 @@ static struct bio *r5l_bio_alloc(struct r5l_log *log, struct r5l_io_unit *io)
bio->bi_rw = WRITE;
bio->bi_bdev = log->rdev->bdev;
- bio->bi_iter.bi_sector = log->log_start;
+ bio->bi_iter.bi_sector = log->rdev->data_offset + log->log_start;
bio->bi_end_io = r5l_log_endio;
bio->bi_private = io;
--
1.9.1
^ permalink raw reply related
* [PATCH 06/11] raid5-cache: refactor bio allocation
From: Christoph Hellwig @ 2015-10-05 7:31 UTC (permalink / raw)
To: Shaohua Li, neilb; +Cc: dan.j.williams, Kernel-team, linux-raid
In-Reply-To: <1444030276-30850-1-git-send-email-hch@lst.de>
Split out a helper to allocate a bio for log writes.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
drivers/md/raid5-cache.c | 45 ++++++++++++++++++++-------------------------
1 file changed, 20 insertions(+), 25 deletions(-)
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index 05126151..4d4dc4d 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -278,11 +278,25 @@ static void r5l_submit_current_io(struct r5l_log *log)
}
}
+static struct bio *r5l_bio_alloc(struct r5l_log *log, struct r5l_io_unit *io)
+{
+ struct bio *bio = bio_kmalloc(GFP_NOIO | __GFP_NOFAIL, BIO_MAX_PAGES);
+
+ bio->bi_rw = WRITE;
+ bio->bi_bdev = log->rdev->bdev;
+ bio->bi_iter.bi_sector = log->log_start;
+ bio->bi_end_io = r5l_log_endio;
+ bio->bi_private = io;
+
+ bio_list_add(&io->bios, bio);
+ atomic_inc(&io->pending_io);
+ return bio;
+}
+
static struct r5l_io_unit *r5l_new_meta(struct r5l_log *log)
{
struct r5l_io_unit *io;
struct r5l_meta_block *block;
- struct bio *bio;
io = r5l_alloc_io_unit(log);
@@ -296,17 +310,8 @@ static struct r5l_io_unit *r5l_new_meta(struct r5l_log *log)
io->meta_offset = sizeof(struct r5l_meta_block);
io->seq = log->seq;
- bio = bio_kmalloc(GFP_NOIO | __GFP_NOFAIL, BIO_MAX_PAGES);
- io->current_bio = bio;
- bio->bi_rw = WRITE;
- bio->bi_bdev = log->rdev->bdev;
- bio->bi_iter.bi_sector = log->log_start;
- bio_add_page(bio, io->meta_page, PAGE_SIZE, 0);
- bio->bi_end_io = r5l_log_endio;
- bio->bi_private = io;
-
- bio_list_add(&io->bios, bio);
- atomic_inc(&io->pending_io);
+ io->current_bio = r5l_bio_alloc(log, io);
+ bio_add_page(io->current_bio, io->meta_page, PAGE_SIZE, 0);
log->seq++;
log->log_start = r5l_ring_add(log, log->log_start, BLOCK_SECTORS);
@@ -360,19 +365,9 @@ static void r5l_append_payload_page(struct r5l_log *log, struct page *page)
struct r5l_io_unit *io = log->current_io;
alloc_bio:
- if (!io->current_bio) {
- struct bio *bio;
-
- bio = bio_kmalloc(GFP_NOIO | __GFP_NOFAIL, BIO_MAX_PAGES);
- bio->bi_rw = WRITE;
- bio->bi_bdev = log->rdev->bdev;
- bio->bi_iter.bi_sector = log->log_start;
- bio->bi_end_io = r5l_log_endio;
- bio->bi_private = io;
- bio_list_add(&io->bios, bio);
- atomic_inc(&io->pending_io);
- io->current_bio = bio;
- }
+ if (!io->current_bio)
+ io->current_bio = r5l_bio_alloc(log, io);
+
if (!bio_add_page(io->current_bio, page, PAGE_SIZE, 0)) {
io->current_bio = NULL;
goto alloc_bio;
--
1.9.1
^ permalink raw reply related
* [PATCH 05/11] raid5-cache: clean up r5l_get_meta
From: Christoph Hellwig @ 2015-10-05 7:31 UTC (permalink / raw)
To: Shaohua Li, neilb; +Cc: dan.j.williams, Kernel-team, linux-raid
In-Reply-To: <1444030276-30850-1-git-send-email-hch@lst.de>
Remove the only partially used local 'io' variable to simplify the code
flow.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
drivers/md/raid5-cache.c | 12 ++++--------
1 file changed, 4 insertions(+), 8 deletions(-)
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index 0d18ed7..05126151 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -324,16 +324,12 @@ static struct r5l_io_unit *r5l_new_meta(struct r5l_log *log)
static int r5l_get_meta(struct r5l_log *log, unsigned int payload_size)
{
- struct r5l_io_unit *io;
-
- io = log->current_io;
- if (io && io->meta_offset + payload_size > PAGE_SIZE)
+ if (log->current_io &&
+ log->current_io->meta_offset + payload_size > PAGE_SIZE)
r5l_submit_current_io(log);
- io = log->current_io;
- if (io)
- return 0;
- log->current_io = r5l_new_meta(log);
+ if (!log->current_io)
+ log->current_io = r5l_new_meta(log);
return 0;
}
--
1.9.1
^ permalink raw reply related
* [PATCH 04/11] raid5-cache: simplify state machine when caches flushes are not needed
From: Christoph Hellwig @ 2015-10-05 7:31 UTC (permalink / raw)
To: Shaohua Li, neilb; +Cc: dan.j.williams, Kernel-team, linux-raid
In-Reply-To: <1444030276-30850-1-git-send-email-hch@lst.de>
For devices without a volatile write cache we don't need to send a FLUSH
command to ensure writes are stable on disk, and thus can avoid the whole
step of batching up bios for processing by the MD thread.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
drivers/md/raid5-cache.c | 33 +++++++++++++++++++++++++++++----
1 file changed, 29 insertions(+), 4 deletions(-)
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index b04e908..0d18ed7 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -83,6 +83,8 @@ struct r5l_log {
struct list_head no_space_stripes; /* pending stripes, log has no space */
spinlock_t no_space_stripes_lock;
+
+ bool need_cache_flush;
};
/*
@@ -205,6 +207,22 @@ static void r5l_io_run_stripes(struct r5l_io_unit *io)
}
}
+static void r5l_log_run_stripes(struct r5l_log *log)
+{
+ struct r5l_io_unit *io, *next;
+
+ assert_spin_locked(&log->io_list_lock);
+
+ list_for_each_entry_safe(io, next, &log->running_ios, log_sibling) {
+ /* don't change list order */
+ if (io->state < IO_UNIT_IO_END)
+ break;
+
+ list_move_tail(&io->log_sibling, &log->finished_ios);
+ r5l_io_run_stripes(io);
+ }
+}
+
static void r5l_log_endio(struct bio *bio)
{
struct r5l_io_unit *io = bio->bi_private;
@@ -221,11 +239,15 @@ static void r5l_log_endio(struct bio *bio)
spin_lock_irqsave(&log->io_list_lock, flags);
__r5l_set_io_unit_state(io, IO_UNIT_IO_END);
- r5l_move_io_unit_list(&log->running_ios, &log->io_end_ios,
- IO_UNIT_IO_END);
+ if (log->need_cache_flush)
+ r5l_move_io_unit_list(&log->running_ios, &log->io_end_ios,
+ IO_UNIT_IO_END);
+ else
+ r5l_log_run_stripes(log);
spin_unlock_irqrestore(&log->io_list_lock, flags);
- md_wakeup_thread(log->rdev->mddev->thread);
+ if (log->need_cache_flush)
+ md_wakeup_thread(log->rdev->mddev->thread);
}
static void r5l_submit_current_io(struct r5l_log *log)
@@ -626,7 +648,8 @@ static void r5l_log_flush_endio(struct bio *bio)
void r5l_flush_stripe_to_raid(struct r5l_log *log)
{
bool do_flush;
- if (!log)
+
+ if (!log || !log->need_cache_flush)
return;
spin_lock_irq(&log->io_list_lock);
@@ -1115,6 +1138,8 @@ int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev)
return -ENOMEM;
log->rdev = rdev;
+ log->need_cache_flush = (rdev->bdev->bd_disk->queue->flush_flags != 0);
+
log->uuid_checksum = crc32_le(~0, (void *)rdev->mddev->uuid,
sizeof(rdev->mddev->uuid));
--
1.9.1
^ permalink raw reply related
* [PATCH 03/11] raid5-cache: factor out a helper to run all stripes for an I/O unit
From: Christoph Hellwig @ 2015-10-05 7:31 UTC (permalink / raw)
To: Shaohua Li, neilb; +Cc: dan.j.williams, Kernel-team, linux-raid
In-Reply-To: <1444030276-30850-1-git-send-email-hch@lst.de>
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
drivers/md/raid5-cache.c | 23 +++++++++++++----------
1 file changed, 13 insertions(+), 10 deletions(-)
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index 9f5a48e..b04e908 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -194,6 +194,17 @@ static void __r5l_set_io_unit_state(struct r5l_io_unit *io,
io->state = state;
}
+static void r5l_io_run_stripes(struct r5l_io_unit *io)
+{
+ struct stripe_head *sh, *next;
+
+ list_for_each_entry_safe(sh, next, &io->stripe_list, log_list) {
+ list_del_init(&sh->log_list);
+ set_bit(STRIPE_HANDLE, &sh->state);
+ raid5_release_stripe(sh);
+ }
+}
+
static void r5l_log_endio(struct bio *bio)
{
struct r5l_io_unit *io = bio->bi_private;
@@ -587,21 +598,13 @@ static void r5l_log_flush_endio(struct bio *bio)
flush_bio);
unsigned long flags;
struct r5l_io_unit *io;
- struct stripe_head *sh;
if (bio->bi_error)
md_error(log->rdev->mddev, log->rdev);
spin_lock_irqsave(&log->io_list_lock, flags);
- list_for_each_entry(io, &log->flushing_ios, log_sibling) {
- while (!list_empty(&io->stripe_list)) {
- sh = list_first_entry(&io->stripe_list,
- struct stripe_head, log_list);
- list_del_init(&sh->log_list);
- set_bit(STRIPE_HANDLE, &sh->state);
- raid5_release_stripe(sh);
- }
- }
+ list_for_each_entry(io, &log->flushing_ios, log_sibling)
+ r5l_io_run_stripes(io);
list_splice_tail_init(&log->flushing_ios, &log->finished_ios);
spin_unlock_irqrestore(&log->io_list_lock, flags);
}
--
1.9.1
^ permalink raw reply related
* [PATCH 02/11] raid5-cache: rename flushed_ios to finished_ios
From: Christoph Hellwig @ 2015-10-05 7:31 UTC (permalink / raw)
To: Shaohua Li, neilb; +Cc: dan.j.williams, Kernel-team, linux-raid
In-Reply-To: <1444030276-30850-1-git-send-email-hch@lst.de>
After this series we won't nessecarily have flushed the cache for these
I/Os, so give the list a more neutral name.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
drivers/md/raid5-cache.c | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index 5b5f346..9f5a48e 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -66,7 +66,7 @@ struct r5l_log {
* to the RAID */
struct list_head flushing_ios; /* io_units which are waiting for log
* cache flush */
- struct list_head flushed_ios; /* io_units which settle down in log disk */
+ struct list_head finished_ios; /* io_units which settle down in log disk */
struct bio flush_bio;
struct kmem_cache *io_kc;
@@ -525,14 +525,14 @@ static sector_t r5l_reclaimable_space(struct r5l_log *log)
log->next_checkpoint);
}
-static bool r5l_complete_flushed_ios(struct r5l_log *log)
+static bool r5l_complete_finished_ios(struct r5l_log *log)
{
struct r5l_io_unit *io, *next;
bool found = false;
assert_spin_locked(&log->io_list_lock);
- list_for_each_entry_safe(io, next, &log->flushed_ios, log_sibling) {
+ list_for_each_entry_safe(io, next, &log->finished_ios, log_sibling) {
/* don't change list order */
if (io->state < IO_UNIT_STRIPE_END)
break;
@@ -558,7 +558,7 @@ static void __r5l_stripe_write_finished(struct r5l_io_unit *io)
spin_lock_irqsave(&log->io_list_lock, flags);
__r5l_set_io_unit_state(io, IO_UNIT_STRIPE_END);
- if (!r5l_complete_flushed_ios(log)) {
+ if (!r5l_complete_finished_ios(log)) {
spin_unlock_irqrestore(&log->io_list_lock, flags);
return;
}
@@ -602,7 +602,7 @@ static void r5l_log_flush_endio(struct bio *bio)
raid5_release_stripe(sh);
}
}
- list_splice_tail_init(&log->flushing_ios, &log->flushed_ios);
+ list_splice_tail_init(&log->flushing_ios, &log->finished_ios);
spin_unlock_irqrestore(&log->io_list_lock, flags);
}
@@ -701,7 +701,7 @@ static void r5l_do_reclaim(struct r5l_log *log)
(list_empty(&log->running_ios) &&
list_empty(&log->io_end_ios) &&
list_empty(&log->flushing_ios) &&
- list_empty(&log->flushed_ios)))
+ list_empty(&log->finished_ios)))
break;
md_wakeup_thread(log->rdev->mddev->thread);
@@ -1121,7 +1121,7 @@ int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev)
INIT_LIST_HEAD(&log->running_ios);
INIT_LIST_HEAD(&log->io_end_ios);
INIT_LIST_HEAD(&log->flushing_ios);
- INIT_LIST_HEAD(&log->flushed_ios);
+ INIT_LIST_HEAD(&log->finished_ios);
bio_init(&log->flush_bio);
log->io_kc = KMEM_CACHE(r5l_io_unit, 0);
--
1.9.1
^ permalink raw reply related
* [PATCH 01/11] raid5-cache: free I/O units earlier
From: Christoph Hellwig @ 2015-10-05 7:31 UTC (permalink / raw)
To: Shaohua Li, neilb; +Cc: dan.j.williams, Kernel-team, linux-raid
In-Reply-To: <1444030276-30850-1-git-send-email-hch@lst.de>
There is no good reason to keep the I/O unit structures around after the
stripe has been written back to the RAID array. The only information
we need is the log sequence number, and the checkpoint offset of the
highest successfull writeback. Store those in the log structure, and
free the IO units from __r5l_stripe_write_finished.
Besides simplifying the code this also avoid having to keep the allocation
for the I/O unit around for a potentially long time as superblock updates
that checkpoint the log do not happen very often.
This also fixes the previously incorrect calculation of 'free' in
r5l_do_reclaim as a side effect: previous if took the last unit which
isn't checkpointed into account.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
drivers/md/raid5-cache.c | 144 ++++++++++++++++++-----------------------------
1 file changed, 55 insertions(+), 89 deletions(-)
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index e917e53..5b5f346 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -51,6 +51,9 @@ struct r5l_log {
sector_t log_start; /* log head. where new data appends */
u64 seq; /* log head sequence */
+ sector_t next_checkpoint;
+ u64 next_cp_seq;
+
struct mutex io_mutex;
struct r5l_io_unit *current_io; /* current io_unit accepting new data */
@@ -65,9 +68,6 @@ struct r5l_log {
* cache flush */
struct list_head flushed_ios; /* io_units which settle down in log disk */
struct bio flush_bio;
- struct list_head stripe_end_ios;/* io_units which have been completely
- * written to the RAID but have not yet
- * been considered for updating super */
struct kmem_cache *io_kc;
@@ -186,35 +186,6 @@ static void r5l_move_io_unit_list(struct list_head *from, struct list_head *to,
}
}
-/*
- * We don't want too many io_units reside in stripe_end_ios list, which will
- * waste a lot of memory. So we try to remove some. But we must keep at least 2
- * io_units. The superblock must point to a valid meta, if it's the last meta,
- * recovery can scan less
- */
-static void r5l_compress_stripe_end_list(struct r5l_log *log)
-{
- struct r5l_io_unit *first, *last, *io;
-
- first = list_first_entry(&log->stripe_end_ios,
- struct r5l_io_unit, log_sibling);
- last = list_last_entry(&log->stripe_end_ios,
- struct r5l_io_unit, log_sibling);
- if (first == last)
- return;
- list_del(&first->log_sibling);
- list_del(&last->log_sibling);
- while (!list_empty(&log->stripe_end_ios)) {
- io = list_first_entry(&log->stripe_end_ios,
- struct r5l_io_unit, log_sibling);
- list_del(&io->log_sibling);
- first->log_end = io->log_end;
- r5l_free_io_unit(log, io);
- }
- list_add_tail(&first->log_sibling, &log->stripe_end_ios);
- list_add_tail(&last->log_sibling, &log->stripe_end_ios);
-}
-
static void __r5l_set_io_unit_state(struct r5l_io_unit *io,
enum r5l_io_unit_state state)
{
@@ -548,31 +519,53 @@ static void r5l_run_no_space_stripes(struct r5l_log *log)
spin_unlock(&log->no_space_stripes_lock);
}
+static sector_t r5l_reclaimable_space(struct r5l_log *log)
+{
+ return r5l_ring_distance(log, log->last_checkpoint,
+ log->next_checkpoint);
+}
+
+static bool r5l_complete_flushed_ios(struct r5l_log *log)
+{
+ struct r5l_io_unit *io, *next;
+ bool found = false;
+
+ assert_spin_locked(&log->io_list_lock);
+
+ list_for_each_entry_safe(io, next, &log->flushed_ios, log_sibling) {
+ /* don't change list order */
+ if (io->state < IO_UNIT_STRIPE_END)
+ break;
+
+ log->next_checkpoint = io->log_start;
+ log->next_cp_seq = io->seq;
+
+ list_del(&io->log_sibling);
+ r5l_free_io_unit(log, io);
+
+ found = true;
+ }
+
+
+ return found;
+}
+
static void __r5l_stripe_write_finished(struct r5l_io_unit *io)
{
struct r5l_log *log = io->log;
- struct r5l_io_unit *last;
- sector_t reclaimable_space;
unsigned long flags;
spin_lock_irqsave(&log->io_list_lock, flags);
__r5l_set_io_unit_state(io, IO_UNIT_STRIPE_END);
- /* might move 0 entry */
- r5l_move_io_unit_list(&log->flushed_ios, &log->stripe_end_ios,
- IO_UNIT_STRIPE_END);
- if (list_empty(&log->stripe_end_ios)) {
+
+ if (!r5l_complete_flushed_ios(log)) {
spin_unlock_irqrestore(&log->io_list_lock, flags);
return;
}
- last = list_last_entry(&log->stripe_end_ios,
- struct r5l_io_unit, log_sibling);
- reclaimable_space = r5l_ring_distance(log, log->last_checkpoint,
- last->log_end);
- if (reclaimable_space >= log->max_free_space)
+ if (r5l_reclaimable_space(log) > log->max_free_space)
r5l_wake_reclaim(log, 0);
- r5l_compress_stripe_end_list(log);
spin_unlock_irqrestore(&log->io_list_lock, flags);
wake_up(&log->iounit_wait);
}
@@ -651,13 +644,6 @@ void r5l_flush_stripe_to_raid(struct r5l_log *log)
submit_bio(WRITE_FLUSH, &log->flush_bio);
}
-static void r5l_kick_io_unit(struct r5l_log *log)
-{
- md_wakeup_thread(log->rdev->mddev->thread);
- wait_event_lock_irq(log->iounit_wait, !list_empty(&log->stripe_end_ios),
- log->io_list_lock);
-}
-
static void r5l_write_super(struct r5l_log *log, sector_t cp);
static void r5l_write_super_and_discard_space(struct r5l_log *log,
sector_t end)
@@ -698,10 +684,10 @@ static void r5l_write_super_and_discard_space(struct r5l_log *log,
static void r5l_do_reclaim(struct r5l_log *log)
{
- struct r5l_io_unit *io, *last;
- LIST_HEAD(list);
- sector_t free = 0;
sector_t reclaim_target = xchg(&log->reclaim_target, 0);
+ sector_t reclaimable;
+ sector_t next_checkpoint;
+ u64 next_cp_seq;
spin_lock_irq(&log->io_list_lock);
/*
@@ -710,60 +696,41 @@ static void r5l_do_reclaim(struct r5l_log *log)
* shouldn't reuse space of an unreclaimable io_unit
*/
while (1) {
- struct list_head *target_list = NULL;
-
- while (!list_empty(&log->stripe_end_ios)) {
- io = list_first_entry(&log->stripe_end_ios,
- struct r5l_io_unit, log_sibling);
- list_move_tail(&io->log_sibling, &list);
- free += r5l_ring_distance(log, io->log_start,
- io->log_end);
- }
-
- if (free >= reclaim_target ||
+ reclaimable = r5l_reclaimable_space(log);
+ if (reclaimable >= reclaim_target ||
(list_empty(&log->running_ios) &&
list_empty(&log->io_end_ios) &&
list_empty(&log->flushing_ios) &&
list_empty(&log->flushed_ios)))
break;
- /* Below waiting mostly happens when we shutdown the raid */
- if (!list_empty(&log->flushed_ios))
- target_list = &log->flushed_ios;
- else if (!list_empty(&log->flushing_ios))
- target_list = &log->flushing_ios;
- else if (!list_empty(&log->io_end_ios))
- target_list = &log->io_end_ios;
- else if (!list_empty(&log->running_ios))
- target_list = &log->running_ios;
-
- r5l_kick_io_unit(log);
+ md_wakeup_thread(log->rdev->mddev->thread);
+ wait_event_lock_irq(log->iounit_wait,
+ r5l_reclaimable_space(log) > reclaimable,
+ log->io_list_lock);
}
+
+ next_checkpoint = log->next_checkpoint;
+ next_cp_seq = log->next_cp_seq;
spin_unlock_irq(&log->io_list_lock);
- if (list_empty(&list))
+ BUG_ON(reclaimable < 0);
+ if (reclaimable == 0)
return;
- /* super always point to last valid meta */
- last = list_last_entry(&list, struct r5l_io_unit, log_sibling);
/*
* write_super will flush cache of each raid disk. We must write super
* here, because the log area might be reused soon and we don't want to
* confuse recovery
*/
- r5l_write_super_and_discard_space(log, last->log_start);
+ r5l_write_super_and_discard_space(log, next_checkpoint);
mutex_lock(&log->io_mutex);
- log->last_checkpoint = last->log_start;
- log->last_cp_seq = last->seq;
+ log->last_checkpoint = next_checkpoint;
+ log->last_cp_seq = next_cp_seq;
mutex_unlock(&log->io_mutex);
- r5l_run_no_space_stripes(log);
- while (!list_empty(&list)) {
- io = list_first_entry(&list, struct r5l_io_unit, log_sibling);
- list_del(&io->log_sibling);
- r5l_free_io_unit(log, io);
- }
+ r5l_run_no_space_stripes(log);
}
static void r5l_reclaim_thread(struct md_thread *thread)
@@ -1153,7 +1120,6 @@ int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev)
spin_lock_init(&log->io_list_lock);
INIT_LIST_HEAD(&log->running_ios);
INIT_LIST_HEAD(&log->io_end_ios);
- INIT_LIST_HEAD(&log->stripe_end_ios);
INIT_LIST_HEAD(&log->flushing_ios);
INIT_LIST_HEAD(&log->flushed_ios);
bio_init(&log->flush_bio);
--
1.9.1
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox