* Re: [PATCH 2/2][v2] blk-plug: don't flush nested plug lists
From: Dave Chinner @ 2015-04-08 23:02 UTC (permalink / raw)
To: Jeff Moyer
Cc: Jens Axboe, Ming Lei, Konrad Rzeszutek Wilk, Roger Pau Monn??,
Alasdair Kergon, Mike Snitzer, Neil Brown, Nicholas A. Bellinger,
Alexander Viro, Chris Mason, Josef Bacik, David Sterba,
Theodore Ts'o, Andreas Dilger, Jaegeuk Kim, Changman Lee,
Steven Whitehouse, Mikulas Patocka, Andrew Morton, Rik van Riel,
Johannes Weiner, Mel Gorman, Trond Myklebust
In-Reply-To: <x49wq1nrcoe.fsf_-_@segfault.boston.devel.redhat.com>
On Tue, Apr 07, 2015 at 02:55:13PM -0400, Jeff Moyer wrote:
> The way the on-stack plugging currently works, each nesting level
> flushes its own list of I/Os. This can be less than optimal (read
> awful) for certain workloads. For example, consider an application
> that issues asynchronous O_DIRECT I/Os. It can send down a bunch of
> I/Os together in a single io_submit call, only to have each of them
> dispatched individually down in the bowels of the dirct I/O code.
> The reason is that there are blk_plug-s instantiated both at the upper
> call site in do_io_submit and down in do_direct_IO. The latter will
> submit as little as 1 I/O at a time (if you have a small enough I/O
> size) instead of performing the batching that the plugging
> infrastructure is supposed to provide.
I'm wondering what impact this will have on filesystem metadata IO
that needs to be issued immediately. e.g. we are doing writeback, so
there is a high level plug in place and we need to page in btree
blocks to do extent allocation. We do readahead at this point,
but it looks like this change will prevent the readahead from being
issued by the unplug in xfs_buf_iosubmit().
So while I can see how this can make your single microbenchmark
better (because it's only doing concurrent direct IO to the block
device and hence there are no dependencies between individual IOs),
I have significant reservations that it's actually a win for
filesystem-based workloads where we need direct control of flushing
to minimise IO latency due to IO dependencies...
Patches like this one:
https://lkml.org/lkml/2015/3/20/442
show similar real-world workload improvements to your patchset by
being smarter about using high level plugging to enable cross-file
merging of IO, but it still relies on the lower layers of plugging
to resolve latency bubbles caused by IO dependencies in the
filesystems.
> NOTE TO SUBSYSTEM MAINTAINERS: Before this patch, blk_finish_plug
> would always flush the plug list. After this patch, this is only the
> case for the outer-most plug. If you require the plug list to be
> flushed, you should be calling blk_flush_plug(current). Btrfs and dm
> maintainers should take a close look at this patch and ensure they get
> the right behavior in the end.
IOWs, you are saying we need to change all our current unplugs to
blk_flush_plug(current) to maintain the same behaviour as we
currently have?
If that is the case, shouldn't you actually be trying to fix the
specific plugging problem you've identified (i.e. do_direct_IO() is
flushing far too frequently) rather than making a sweeping
generalisation that the IO stack plugging infrastructure
needs to be fundamentally changed?
Cheers,
Dave.
>
> ---
> Changelog:
> v1->v2: Keep the blk_start_plug interface the same, suggested by Ming Lei.
>
> Test results
> ------------
> Virtio-blk:
>
> unpatched:
>
> job1: (groupid=0, jobs=1): err= 0: pid=8032: Tue Apr 7 13:33:53 2015
> read : io=2736.1MB, bw=280262KB/s, iops=70065, runt= 10000msec
> slat (usec): min=40, max=10472, avg=207.82, stdev=364.02
> clat (usec): min=211, max=35883, avg=14379.83, stdev=2213.95
> lat (usec): min=862, max=36000, avg=14587.72, stdev=2223.80
> clat percentiles (usec):
> | 1.00th=[11328], 5.00th=[12096], 10.00th=[12480], 20.00th=[12992],
> | 30.00th=[13376], 40.00th=[13760], 50.00th=[14144], 60.00th=[14400],
> | 70.00th=[14784], 80.00th=[15168], 90.00th=[15936], 95.00th=[16768],
> | 99.00th=[24448], 99.50th=[25216], 99.90th=[28544], 99.95th=[35072],
> | 99.99th=[36096]
> bw (KB /s): min=265984, max=302720, per=100.00%, avg=280549.84, stdev=10264.36
> lat (usec) : 250=0.01%, 1000=0.01%
> lat (msec) : 2=0.02%, 4=0.02%, 10=0.05%, 20=96.57%, 50=3.34%
> cpu : usr=7.56%, sys=55.57%, ctx=6174, majf=0, minf=523
> IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.1%, 32=0.1%, >=64=100.0%
> submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.1%
> issued : total=r=700656/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
> latency : target=0, window=0, percentile=100.00%, depth=1024
>
> Run status group 0 (all jobs):
> READ: io=2736.1MB, aggrb=280262KB/s, minb=280262KB/s, maxb=280262KB/s, mint=10000msec, maxt=10000msec
>
> Disk stats (read/write):
> vdd: ios=695490/0, merge=0/0, ticks=785741/0, in_queue=785442, util=90.69%
>
>
> patched:
> job1: (groupid=0, jobs=1): err= 0: pid=7743: Tue Apr 7 13:19:07 2015
> read : io=8126.6MB, bw=832158KB/s, iops=208039, runt= 10000msec
> slat (usec): min=20, max=14351, avg=55.08, stdev=143.47
> clat (usec): min=283, max=20003, avg=4846.77, stdev=1355.35
> lat (usec): min=609, max=20074, avg=4901.95, stdev=1362.40
> clat percentiles (usec):
> | 1.00th=[ 4016], 5.00th=[ 4048], 10.00th=[ 4080], 20.00th=[ 4128],
> | 30.00th=[ 4192], 40.00th=[ 4192], 50.00th=[ 4256], 60.00th=[ 4512],
> | 70.00th=[ 4896], 80.00th=[ 5664], 90.00th=[ 5920], 95.00th=[ 6752],
> | 99.00th=[11968], 99.50th=[13632], 99.90th=[15552], 99.95th=[17024],
> | 99.99th=[19840]
> bw (KB /s): min=740992, max=896640, per=100.00%, avg=836978.95, stdev=51034.87
> lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01%
> lat (msec) : 4=0.50%, 10=97.79%, 20=1.70%, 50=0.01%
> cpu : usr=20.28%, sys=69.11%, ctx=879, majf=0, minf=522
> IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.1%, 32=0.1%, >=64=100.0%
> submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.1%
> issued : total=r=2080396/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
> latency : target=0, window=0, percentile=100.00%, depth=1024
>
> Run status group 0 (all jobs):
> READ: io=8126.6MB, aggrb=832158KB/s, minb=832158KB/s, maxb=832158KB/s, mint=10000msec, maxt=10000msec
>
> Disk stats (read/write):
> vdd: ios=127877/0, merge=1918166/0, ticks=23118/0, in_queue=23047, util=94.08%
>
> micron p320h:
>
> unpatched:
>
> job1: (groupid=0, jobs=1): err= 0: pid=3244: Tue Apr 7 13:29:14 2015
> read : io=6728.9MB, bw=688968KB/s, iops=172241, runt= 10001msec
> slat (usec): min=43, max=6273, avg=81.79, stdev=125.96
> clat (usec): min=78, max=12485, avg=5852.06, stdev=1154.76
> lat (usec): min=146, max=12572, avg=5933.92, stdev=1163.75
> clat percentiles (usec):
> | 1.00th=[ 4192], 5.00th=[ 4384], 10.00th=[ 4576], 20.00th=[ 5600],
> | 30.00th=[ 5664], 40.00th=[ 5728], 50.00th=[ 5792], 60.00th=[ 5856],
> | 70.00th=[ 6112], 80.00th=[ 6176], 90.00th=[ 6240], 95.00th=[ 6368],
> | 99.00th=[11840], 99.50th=[11968], 99.90th=[12096], 99.95th=[12096],
> | 99.99th=[12224]
> bw (KB /s): min=648328, max=859264, per=98.80%, avg=680711.16, stdev=62016.70
> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> lat (msec) : 2=0.01%, 4=0.04%, 10=97.07%, 20=2.87%
> cpu : usr=10.28%, sys=73.61%, ctx=104436, majf=0, minf=6217
> IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.1%, 32=0.1%, >=64=100.0%
> submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.1%
> issued : total=r=1722592/w=0/d=0, short=r=0/w=0/d=0
> latency : target=0, window=0, percentile=100.00%, depth=1024
>
> Run status group 0 (all jobs):
> READ: io=6728.9MB, aggrb=688967KB/s, minb=688967KB/s, maxb=688967KB/s, mint=10001msec, maxt=10001msec
>
> Disk stats (read/write):
> rssda: ios=1688772/0, merge=0/0, ticks=188820/0, in_queue=188678, util=96.61%
>
> patched:
>
> job1: (groupid=0, jobs=1): err= 0: pid=9531: Tue Apr 7 13:22:28 2015
> read : io=11607MB, bw=1160.6MB/s, iops=297104, runt= 10001msec
> slat (usec): min=21, max=6376, avg=43.05, stdev=81.82
> clat (usec): min=116, max=9844, avg=3393.90, stdev=752.57
> lat (usec): min=167, max=9889, avg=3437.01, stdev=757.02
> clat percentiles (usec):
> | 1.00th=[ 2832], 5.00th=[ 2992], 10.00th=[ 3056], 20.00th=[ 3120],
> | 30.00th=[ 3152], 40.00th=[ 3248], 50.00th=[ 3280], 60.00th=[ 3344],
> | 70.00th=[ 3376], 80.00th=[ 3504], 90.00th=[ 3728], 95.00th=[ 3824],
> | 99.00th=[ 9152], 99.50th=[ 9408], 99.90th=[ 9664], 99.95th=[ 9664],
> | 99.99th=[ 9792]
> bw (MB /s): min= 1139, max= 1183, per=100.00%, avg=1161.07, stdev=10.58
> lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
> lat (msec) : 2=0.01%, 4=98.31%, 10=1.67%
> cpu : usr=18.59%, sys=66.65%, ctx=55655, majf=0, minf=6218
> IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.1%, 32=0.1%, >=64=100.0%
> submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.1%
> issued : total=r=2971338/w=0/d=0, short=r=0/w=0/d=0
> latency : target=0, window=0, percentile=100.00%, depth=1024
>
> Run status group 0 (all jobs):
> READ: io=11607MB, aggrb=1160.6MB/s, minb=1160.6MB/s, maxb=1160.6MB/s, mint=10001msec, maxt=10001msec
>
> Disk stats (read/write):
> rssda: ios=183005/0, merge=2745105/0, ticks=31972/0, in_queue=31948, util=97.63%
> ---
> block/blk-core.c | 29 ++++++++++++++++-------------
> block/blk-lib.c | 2 +-
> block/blk-throttle.c | 2 +-
> drivers/block/xen-blkback/blkback.c | 2 +-
> drivers/md/dm-bufio.c | 6 +++---
> drivers/md/dm-crypt.c | 2 +-
> drivers/md/dm-kcopyd.c | 2 +-
> drivers/md/dm-thin.c | 2 +-
> drivers/md/md.c | 2 +-
> drivers/md/raid1.c | 2 +-
> drivers/md/raid10.c | 2 +-
> drivers/md/raid5.c | 4 ++--
> drivers/target/target_core_iblock.c | 2 +-
> fs/aio.c | 2 +-
> fs/block_dev.c | 2 +-
> fs/btrfs/scrub.c | 2 +-
> fs/btrfs/transaction.c | 2 +-
> fs/btrfs/tree-log.c | 12 ++++++------
> fs/btrfs/volumes.c | 6 +++---
> fs/buffer.c | 2 +-
> fs/direct-io.c | 2 +-
> fs/ext4/file.c | 2 +-
> fs/ext4/inode.c | 4 ++--
> fs/f2fs/checkpoint.c | 2 +-
> fs/f2fs/gc.c | 2 +-
> fs/f2fs/node.c | 2 +-
> fs/gfs2/log.c | 2 +-
> fs/hpfs/buffer.c | 2 +-
> fs/jbd/checkpoint.c | 2 +-
> fs/jbd/commit.c | 4 ++--
> fs/jbd2/checkpoint.c | 2 +-
> fs/jbd2/commit.c | 2 +-
> fs/mpage.c | 2 +-
> fs/nfs/blocklayout/blocklayout.c | 4 ++--
> fs/xfs/xfs_buf.c | 4 ++--
> fs/xfs/xfs_dir2_readdir.c | 2 +-
> fs/xfs/xfs_itable.c | 2 +-
> include/linux/blkdev.h | 5 +++--
> mm/madvise.c | 2 +-
> mm/page-writeback.c | 2 +-
> mm/readahead.c | 2 +-
> mm/swap_state.c | 2 +-
> mm/vmscan.c | 2 +-
> 43 files changed, 74 insertions(+), 70 deletions(-)
>
> diff --git a/block/blk-core.c b/block/blk-core.c
> index 794c3e7..fcd9c2f 100644
> --- a/block/blk-core.c
> +++ b/block/blk-core.c
> @@ -3018,21 +3018,21 @@ void blk_start_plug(struct blk_plug *plug)
> {
> struct task_struct *tsk = current;
>
> + if (tsk->plug) {
> + tsk->plug->depth++;
> + return;
> + }
> +
> + plug->depth = 1;
> INIT_LIST_HEAD(&plug->list);
> INIT_LIST_HEAD(&plug->mq_list);
> INIT_LIST_HEAD(&plug->cb_list);
>
> /*
> - * If this is a nested plug, don't actually assign it. It will be
> - * flushed on its own.
> + * Store ordering should not be needed here, since a potential
> + * preempt will imply a full memory barrier
> */
> - if (!tsk->plug) {
> - /*
> - * Store ordering should not be needed here, since a potential
> - * preempt will imply a full memory barrier
> - */
> - tsk->plug = plug;
> - }
> + tsk->plug = plug;
> }
> EXPORT_SYMBOL(blk_start_plug);
>
> @@ -3177,12 +3177,15 @@ void blk_flush_plug_list(struct blk_plug *plug, bool from_schedule)
> local_irq_restore(flags);
> }
>
> -void blk_finish_plug(struct blk_plug *plug)
> +void blk_finish_plug(void)
> {
> - blk_flush_plug_list(plug, false);
> + struct blk_plug *plug = current->plug;
>
> - if (plug == current->plug)
> - current->plug = NULL;
> + if (--plug->depth > 0)
> + return;
> +
> + blk_flush_plug_list(plug, false);
> + current->plug = NULL;
> }
> EXPORT_SYMBOL(blk_finish_plug);
>
> diff --git a/block/blk-lib.c b/block/blk-lib.c
> index 7688ee3..ac347d3 100644
> --- a/block/blk-lib.c
> +++ b/block/blk-lib.c
> @@ -128,7 +128,7 @@ int blkdev_issue_discard(struct block_device *bdev, sector_t sector,
> */
> cond_resched();
> }
> - blk_finish_plug(&plug);
> + blk_finish_plug();
>
> /* Wait for bios in-flight */
> if (!atomic_dec_and_test(&bb.done))
> diff --git a/block/blk-throttle.c b/block/blk-throttle.c
> index 5b9c6d5..222a77a 100644
> --- a/block/blk-throttle.c
> +++ b/block/blk-throttle.c
> @@ -1281,7 +1281,7 @@ static void blk_throtl_dispatch_work_fn(struct work_struct *work)
> blk_start_plug(&plug);
> while((bio = bio_list_pop(&bio_list_on_stack)))
> generic_make_request(bio);
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> }
> }
>
> diff --git a/drivers/block/xen-blkback/blkback.c b/drivers/block/xen-blkback/blkback.c
> index 2a04d34..74bea21 100644
> --- a/drivers/block/xen-blkback/blkback.c
> +++ b/drivers/block/xen-blkback/blkback.c
> @@ -1374,7 +1374,7 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif,
> submit_bio(operation, biolist[i]);
>
> /* Let the I/Os go.. */
> - blk_finish_plug(&plug);
> + blk_finish_plug();
>
> if (operation == READ)
> blkif->st_rd_sect += preq.nr_sects;
> diff --git a/drivers/md/dm-bufio.c b/drivers/md/dm-bufio.c
> index 86dbbc7..502c63b 100644
> --- a/drivers/md/dm-bufio.c
> +++ b/drivers/md/dm-bufio.c
> @@ -715,7 +715,7 @@ static void __flush_write_list(struct list_head *write_list)
> submit_io(b, WRITE, b->block, write_endio);
> dm_bufio_cond_resched();
> }
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> }
>
> /*
> @@ -1126,7 +1126,7 @@ void dm_bufio_prefetch(struct dm_bufio_client *c,
> &write_list);
> if (unlikely(!list_empty(&write_list))) {
> dm_bufio_unlock(c);
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> __flush_write_list(&write_list);
> blk_start_plug(&plug);
> dm_bufio_lock(c);
> @@ -1149,7 +1149,7 @@ void dm_bufio_prefetch(struct dm_bufio_client *c,
> dm_bufio_unlock(c);
>
> flush_plug:
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> }
> EXPORT_SYMBOL_GPL(dm_bufio_prefetch);
>
> diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c
> index 713a962..65d7b72 100644
> --- a/drivers/md/dm-crypt.c
> +++ b/drivers/md/dm-crypt.c
> @@ -1224,7 +1224,7 @@ pop_from_list:
> rb_erase(&io->rb_node, &write_tree);
> kcryptd_io_write(io);
> } while (!RB_EMPTY_ROOT(&write_tree));
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> }
> return 0;
> }
> diff --git a/drivers/md/dm-kcopyd.c b/drivers/md/dm-kcopyd.c
> index 3a7cade..4a76e42 100644
> --- a/drivers/md/dm-kcopyd.c
> +++ b/drivers/md/dm-kcopyd.c
> @@ -593,7 +593,7 @@ static void do_work(struct work_struct *work)
> process_jobs(&kc->complete_jobs, kc, run_complete_job);
> process_jobs(&kc->pages_jobs, kc, run_pages_job);
> process_jobs(&kc->io_jobs, kc, run_io_job);
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> }
>
> /*
> diff --git a/drivers/md/dm-thin.c b/drivers/md/dm-thin.c
> index 921aafd..be42bf5 100644
> --- a/drivers/md/dm-thin.c
> +++ b/drivers/md/dm-thin.c
> @@ -1824,7 +1824,7 @@ static void process_thin_deferred_bios(struct thin_c *tc)
> dm_pool_issue_prefetches(pool->pmd);
> }
> }
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> }
>
> static int cmp_cells(const void *lhs, const void *rhs)
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index 717daad..c4ec179 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -7686,7 +7686,7 @@ void md_do_sync(struct md_thread *thread)
> /*
> * this also signals 'finished resyncing' to md_stop
> */
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> wait_event(mddev->recovery_wait, !atomic_read(&mddev->recovery_active));
>
> /* tell personality that we are finished */
> diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
> index d34e238..4f8fad4 100644
> --- a/drivers/md/raid1.c
> +++ b/drivers/md/raid1.c
> @@ -2441,7 +2441,7 @@ static void raid1d(struct md_thread *thread)
> if (mddev->flags & ~(1<<MD_CHANGE_PENDING))
> md_check_recovery(mddev);
> }
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> }
>
> static int init_resync(struct r1conf *conf)
> diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
> index a7196c4..92bb5dd 100644
> --- a/drivers/md/raid10.c
> +++ b/drivers/md/raid10.c
> @@ -2835,7 +2835,7 @@ static void raid10d(struct md_thread *thread)
> if (mddev->flags & ~(1<<MD_CHANGE_PENDING))
> md_check_recovery(mddev);
> }
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> }
>
> static int init_resync(struct r10conf *conf)
> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> index cd2f96b..695bf0f 100644
> --- a/drivers/md/raid5.c
> +++ b/drivers/md/raid5.c
> @@ -5281,7 +5281,7 @@ static void raid5_do_work(struct work_struct *work)
> pr_debug("%d stripes handled\n", handled);
>
> spin_unlock_irq(&conf->device_lock);
> - blk_finish_plug(&plug);
> + blk_finish_plug();
>
> pr_debug("--- raid5worker inactive\n");
> }
> @@ -5352,7 +5352,7 @@ static void raid5d(struct md_thread *thread)
> spin_unlock_irq(&conf->device_lock);
>
> async_tx_issue_pending_all();
> - blk_finish_plug(&plug);
> + blk_finish_plug();
>
> pr_debug("--- raid5d inactive\n");
> }
> diff --git a/drivers/target/target_core_iblock.c b/drivers/target/target_core_iblock.c
> index d4a4b0f..17d8730 100644
> --- a/drivers/target/target_core_iblock.c
> +++ b/drivers/target/target_core_iblock.c
> @@ -367,7 +367,7 @@ static void iblock_submit_bios(struct bio_list *list, int rw)
> blk_start_plug(&plug);
> while ((bio = bio_list_pop(list)))
> submit_bio(rw, bio);
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> }
>
> static void iblock_end_io_flush(struct bio *bio, int err)
> diff --git a/fs/aio.c b/fs/aio.c
> index f8e52a1..b873698 100644
> --- a/fs/aio.c
> +++ b/fs/aio.c
> @@ -1616,7 +1616,7 @@ long do_io_submit(aio_context_t ctx_id, long nr,
> if (ret)
> break;
> }
> - blk_finish_plug(&plug);
> + blk_finish_plug();
>
> percpu_ref_put(&ctx->users);
> return i ? i : ret;
> diff --git a/fs/block_dev.c b/fs/block_dev.c
> index 975266b..f5848de 100644
> --- a/fs/block_dev.c
> +++ b/fs/block_dev.c
> @@ -1609,7 +1609,7 @@ ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from)
> if (err < 0)
> ret = err;
> }
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> return ret;
> }
> EXPORT_SYMBOL_GPL(blkdev_write_iter);
> diff --git a/fs/btrfs/scrub.c b/fs/btrfs/scrub.c
> index ec57687..f314cfb8 100644
> --- a/fs/btrfs/scrub.c
> +++ b/fs/btrfs/scrub.c
> @@ -3316,7 +3316,7 @@ out:
> scrub_wr_submit(sctx);
> mutex_unlock(&sctx->wr_ctx.wr_lock);
>
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> btrfs_free_path(path);
> btrfs_free_path(ppath);
> return ret < 0 ? ret : 0;
> diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c
> index 8be4278..fee10af 100644
> --- a/fs/btrfs/transaction.c
> +++ b/fs/btrfs/transaction.c
> @@ -983,7 +983,7 @@ static int btrfs_write_and_wait_marked_extents(struct btrfs_root *root,
>
> blk_start_plug(&plug);
> ret = btrfs_write_marked_extents(root, dirty_pages, mark);
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> ret2 = btrfs_wait_marked_extents(root, dirty_pages, mark);
>
> if (ret)
> diff --git a/fs/btrfs/tree-log.c b/fs/btrfs/tree-log.c
> index c5b8ba3..879c7fd 100644
> --- a/fs/btrfs/tree-log.c
> +++ b/fs/btrfs/tree-log.c
> @@ -2574,7 +2574,7 @@ int btrfs_sync_log(struct btrfs_trans_handle *trans,
> blk_start_plug(&plug);
> ret = btrfs_write_marked_extents(log, &log->dirty_log_pages, mark);
> if (ret) {
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> btrfs_abort_transaction(trans, root, ret);
> btrfs_free_logged_extents(log, log_transid);
> btrfs_set_log_full_commit(root->fs_info, trans);
> @@ -2619,7 +2619,7 @@ int btrfs_sync_log(struct btrfs_trans_handle *trans,
> if (!list_empty(&root_log_ctx.list))
> list_del_init(&root_log_ctx.list);
>
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> btrfs_set_log_full_commit(root->fs_info, trans);
>
> if (ret != -ENOSPC) {
> @@ -2635,7 +2635,7 @@ int btrfs_sync_log(struct btrfs_trans_handle *trans,
> }
>
> if (log_root_tree->log_transid_committed >= root_log_ctx.log_transid) {
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> mutex_unlock(&log_root_tree->log_mutex);
> ret = root_log_ctx.log_ret;
> goto out;
> @@ -2643,7 +2643,7 @@ int btrfs_sync_log(struct btrfs_trans_handle *trans,
>
> index2 = root_log_ctx.log_transid % 2;
> if (atomic_read(&log_root_tree->log_commit[index2])) {
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> ret = btrfs_wait_marked_extents(log, &log->dirty_log_pages,
> mark);
> btrfs_wait_logged_extents(trans, log, log_transid);
> @@ -2669,7 +2669,7 @@ int btrfs_sync_log(struct btrfs_trans_handle *trans,
> * check the full commit flag again
> */
> if (btrfs_need_log_full_commit(root->fs_info, trans)) {
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> btrfs_wait_marked_extents(log, &log->dirty_log_pages, mark);
> btrfs_free_logged_extents(log, log_transid);
> mutex_unlock(&log_root_tree->log_mutex);
> @@ -2680,7 +2680,7 @@ int btrfs_sync_log(struct btrfs_trans_handle *trans,
> ret = btrfs_write_marked_extents(log_root_tree,
> &log_root_tree->dirty_log_pages,
> EXTENT_DIRTY | EXTENT_NEW);
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> if (ret) {
> btrfs_set_log_full_commit(root->fs_info, trans);
> btrfs_abort_transaction(trans, root, ret);
> diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
> index 8222f6f..16db068 100644
> --- a/fs/btrfs/volumes.c
> +++ b/fs/btrfs/volumes.c
> @@ -358,7 +358,7 @@ loop_lock:
> if (pending_bios == &device->pending_sync_bios) {
> sync_pending = 1;
> } else if (sync_pending) {
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> blk_start_plug(&plug);
> sync_pending = 0;
> }
> @@ -415,7 +415,7 @@ loop_lock:
> }
> /* unplug every 64 requests just for good measure */
> if (batch_run % 64 == 0) {
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> blk_start_plug(&plug);
> sync_pending = 0;
> }
> @@ -431,7 +431,7 @@ loop_lock:
> spin_unlock(&device->io_lock);
>
> done:
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> }
>
> static void pending_bios_fn(struct btrfs_work *work)
> diff --git a/fs/buffer.c b/fs/buffer.c
> index 20805db..8181c44 100644
> --- a/fs/buffer.c
> +++ b/fs/buffer.c
> @@ -758,7 +758,7 @@ static int fsync_buffers_list(spinlock_t *lock, struct list_head *list)
> }
>
> spin_unlock(lock);
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> spin_lock(lock);
>
> while (!list_empty(&tmp)) {
> diff --git a/fs/direct-io.c b/fs/direct-io.c
> index e181b6b..16f16ed 100644
> --- a/fs/direct-io.c
> +++ b/fs/direct-io.c
> @@ -1262,7 +1262,7 @@ do_blockdev_direct_IO(int rw, struct kiocb *iocb, struct inode *inode,
> if (sdio.bio)
> dio_bio_submit(dio, &sdio);
>
> - blk_finish_plug(&plug);
> + blk_finish_plug();
>
> /*
> * It is possible that, we return short IO due to end of file.
> diff --git a/fs/ext4/file.c b/fs/ext4/file.c
> index 33a09da..3a293eb 100644
> --- a/fs/ext4/file.c
> +++ b/fs/ext4/file.c
> @@ -183,7 +183,7 @@ ext4_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
> ret = err;
> }
> if (o_direct)
> - blk_finish_plug(&plug);
> + blk_finish_plug();
>
> errout:
> if (aio_mutex)
> diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
> index 5cb9a21..90ce0cb 100644
> --- a/fs/ext4/inode.c
> +++ b/fs/ext4/inode.c
> @@ -2302,7 +2302,7 @@ static int ext4_writepages(struct address_space *mapping,
>
> blk_start_plug(&plug);
> ret = write_cache_pages(mapping, wbc, __writepage, mapping);
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> goto out_writepages;
> }
>
> @@ -2438,7 +2438,7 @@ retry:
> if (ret)
> break;
> }
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> if (!ret && !cycled && wbc->nr_to_write > 0) {
> cycled = 1;
> mpd.last_page = writeback_index - 1;
> diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c
> index 7f794b7..86ba453 100644
> --- a/fs/f2fs/checkpoint.c
> +++ b/fs/f2fs/checkpoint.c
> @@ -846,7 +846,7 @@ retry_flush_nodes:
> goto retry_flush_nodes;
> }
> out:
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> return err;
> }
>
> diff --git a/fs/f2fs/gc.c b/fs/f2fs/gc.c
> index 76adbc3..abeef77 100644
> --- a/fs/f2fs/gc.c
> +++ b/fs/f2fs/gc.c
> @@ -678,7 +678,7 @@ static void do_garbage_collect(struct f2fs_sb_info *sbi, unsigned int segno,
> gc_data_segment(sbi, sum->entries, gc_list, segno, gc_type);
> break;
> }
> - blk_finish_plug(&plug);
> + blk_finish_plug();
>
> stat_inc_seg_count(sbi, GET_SUM_TYPE((&sum->footer)));
> stat_inc_call_count(sbi->stat_info);
> diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c
> index 97bd9d3..c4aa9e2 100644
> --- a/fs/f2fs/node.c
> +++ b/fs/f2fs/node.c
> @@ -1098,7 +1098,7 @@ repeat:
> ra_node_page(sbi, nid);
> }
>
> - blk_finish_plug(&plug);
> + blk_finish_plug();
>
> lock_page(page);
> if (unlikely(page->mapping != NODE_MAPPING(sbi))) {
> diff --git a/fs/gfs2/log.c b/fs/gfs2/log.c
> index 536e7a6..06f25d17 100644
> --- a/fs/gfs2/log.c
> +++ b/fs/gfs2/log.c
> @@ -159,7 +159,7 @@ restart:
> goto restart;
> }
> spin_unlock(&sdp->sd_ail_lock);
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> trace_gfs2_ail_flush(sdp, wbc, 0);
> }
>
> diff --git a/fs/hpfs/buffer.c b/fs/hpfs/buffer.c
> index 8057fe4..138462d 100644
> --- a/fs/hpfs/buffer.c
> +++ b/fs/hpfs/buffer.c
> @@ -35,7 +35,7 @@ void hpfs_prefetch_sectors(struct super_block *s, unsigned secno, int n)
> secno++;
> n--;
> }
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> }
>
> /* Map a sector into a buffer and return pointers to it and to the buffer. */
> diff --git a/fs/jbd/checkpoint.c b/fs/jbd/checkpoint.c
> index 08c0304..cd6b09f 100644
> --- a/fs/jbd/checkpoint.c
> +++ b/fs/jbd/checkpoint.c
> @@ -263,7 +263,7 @@ __flush_batch(journal_t *journal, struct buffer_head **bhs, int *batch_count)
> blk_start_plug(&plug);
> for (i = 0; i < *batch_count; i++)
> write_dirty_buffer(bhs[i], WRITE_SYNC);
> - blk_finish_plug(&plug);
> + blk_finish_plug();
>
> for (i = 0; i < *batch_count; i++) {
> struct buffer_head *bh = bhs[i];
> diff --git a/fs/jbd/commit.c b/fs/jbd/commit.c
> index bb217dc..e1046c3 100644
> --- a/fs/jbd/commit.c
> +++ b/fs/jbd/commit.c
> @@ -447,7 +447,7 @@ void journal_commit_transaction(journal_t *journal)
> blk_start_plug(&plug);
> err = journal_submit_data_buffers(journal, commit_transaction,
> write_op);
> - blk_finish_plug(&plug);
> + blk_finish_plug();
>
> /*
> * Wait for all previously submitted IO to complete.
> @@ -697,7 +697,7 @@ start_journal_io:
> }
> }
>
> - blk_finish_plug(&plug);
> + blk_finish_plug();
>
> /* Lo and behold: we have just managed to send a transaction to
> the log. Before we can commit it, wait for the IO so far to
> diff --git a/fs/jbd2/checkpoint.c b/fs/jbd2/checkpoint.c
> index 988b32e..6aa0039 100644
> --- a/fs/jbd2/checkpoint.c
> +++ b/fs/jbd2/checkpoint.c
> @@ -187,7 +187,7 @@ __flush_batch(journal_t *journal, int *batch_count)
> blk_start_plug(&plug);
> for (i = 0; i < *batch_count; i++)
> write_dirty_buffer(journal->j_chkpt_bhs[i], WRITE_SYNC);
> - blk_finish_plug(&plug);
> + blk_finish_plug();
>
> for (i = 0; i < *batch_count; i++) {
> struct buffer_head *bh = journal->j_chkpt_bhs[i];
> diff --git a/fs/jbd2/commit.c b/fs/jbd2/commit.c
> index b73e021..8f532c8 100644
> --- a/fs/jbd2/commit.c
> +++ b/fs/jbd2/commit.c
> @@ -805,7 +805,7 @@ start_journal_io:
> __jbd2_journal_abort_hard(journal);
> }
>
> - blk_finish_plug(&plug);
> + blk_finish_plug();
>
> /* Lo and behold: we have just managed to send a transaction to
> the log. Before we can commit it, wait for the IO so far to
> diff --git a/fs/mpage.c b/fs/mpage.c
> index 3e79220..bf7d6c3 100644
> --- a/fs/mpage.c
> +++ b/fs/mpage.c
> @@ -695,7 +695,7 @@ mpage_writepages(struct address_space *mapping,
> if (mpd.bio)
> mpage_bio_submit(WRITE, mpd.bio);
> }
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> return ret;
> }
> EXPORT_SYMBOL(mpage_writepages);
> diff --git a/fs/nfs/blocklayout/blocklayout.c b/fs/nfs/blocklayout/blocklayout.c
> index 1cac3c1..e93b6a8 100644
> --- a/fs/nfs/blocklayout/blocklayout.c
> +++ b/fs/nfs/blocklayout/blocklayout.c
> @@ -311,7 +311,7 @@ bl_read_pagelist(struct nfs_pgio_header *header)
> }
> out:
> bl_submit_bio(READ, bio);
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> put_parallel(par);
> return PNFS_ATTEMPTED;
> }
> @@ -433,7 +433,7 @@ bl_write_pagelist(struct nfs_pgio_header *header, int sync)
> header->res.count = header->args.count;
> out:
> bl_submit_bio(WRITE, bio);
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> put_parallel(par);
> return PNFS_ATTEMPTED;
> }
> diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c
> index 1790b00..2f89ca2 100644
> --- a/fs/xfs/xfs_buf.c
> +++ b/fs/xfs/xfs_buf.c
> @@ -1289,7 +1289,7 @@ _xfs_buf_ioapply(
> if (size <= 0)
> break; /* all done */
> }
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> }
>
> /*
> @@ -1823,7 +1823,7 @@ __xfs_buf_delwri_submit(
>
> xfs_buf_submit(bp);
> }
> - blk_finish_plug(&plug);
> + blk_finish_plug();
>
> return pinned;
> }
> diff --git a/fs/xfs/xfs_dir2_readdir.c b/fs/xfs/xfs_dir2_readdir.c
> index 098cd78..7e8fa3f 100644
> --- a/fs/xfs/xfs_dir2_readdir.c
> +++ b/fs/xfs/xfs_dir2_readdir.c
> @@ -455,7 +455,7 @@ xfs_dir2_leaf_readbuf(
> }
> }
> }
> - blk_finish_plug(&plug);
> + blk_finish_plug();
>
> out:
> *bpp = bp;
> diff --git a/fs/xfs/xfs_itable.c b/fs/xfs/xfs_itable.c
> index 82e3142..c3ac5ec 100644
> --- a/fs/xfs/xfs_itable.c
> +++ b/fs/xfs/xfs_itable.c
> @@ -196,7 +196,7 @@ xfs_bulkstat_ichunk_ra(
> &xfs_inode_buf_ops);
> }
> }
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> }
>
> /*
> diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
> index 7f9a516..188133f 100644
> --- a/include/linux/blkdev.h
> +++ b/include/linux/blkdev.h
> @@ -1091,6 +1091,7 @@ static inline void blk_post_runtime_resume(struct request_queue *q, int err) {}
> * schedule() where blk_schedule_flush_plug() is called.
> */
> struct blk_plug {
> + int depth; /* number of nested plugs */
> struct list_head list; /* requests */
> struct list_head mq_list; /* blk-mq requests */
> struct list_head cb_list; /* md requires an unplug callback */
> @@ -1107,7 +1108,7 @@ struct blk_plug_cb {
> extern struct blk_plug_cb *blk_check_plugged(blk_plug_cb_fn unplug,
> void *data, int size);
> extern void blk_start_plug(struct blk_plug *);
> -extern void blk_finish_plug(struct blk_plug *);
> +extern void blk_finish_plug(void);
> extern void blk_flush_plug_list(struct blk_plug *, bool);
>
> static inline void blk_flush_plug(struct task_struct *tsk)
> @@ -1646,7 +1647,7 @@ static inline void blk_start_plug(struct blk_plug *plug)
> {
> }
>
> -static inline void blk_finish_plug(struct blk_plug *plug)
> +static inline void blk_finish_plug(void)
> {
> }
>
> diff --git a/mm/madvise.c b/mm/madvise.c
> index d551475..18a34ee 100644
> --- a/mm/madvise.c
> +++ b/mm/madvise.c
> @@ -539,7 +539,7 @@ SYSCALL_DEFINE3(madvise, unsigned long, start, size_t, len_in, int, behavior)
> vma = find_vma(current->mm, start);
> }
> out:
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> if (write)
> up_write(¤t->mm->mmap_sem);
> else
> diff --git a/mm/page-writeback.c b/mm/page-writeback.c
> index 644bcb6..4570f6e 100644
> --- a/mm/page-writeback.c
> +++ b/mm/page-writeback.c
> @@ -2020,7 +2020,7 @@ int generic_writepages(struct address_space *mapping,
>
> blk_start_plug(&plug);
> ret = write_cache_pages(mapping, wbc, __writepage, mapping);
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> return ret;
> }
>
> diff --git a/mm/readahead.c b/mm/readahead.c
> index 9356758..64182a2 100644
> --- a/mm/readahead.c
> +++ b/mm/readahead.c
> @@ -136,7 +136,7 @@ static int read_pages(struct address_space *mapping, struct file *filp,
> ret = 0;
>
> out:
> - blk_finish_plug(&plug);
> + blk_finish_plug();
>
> return ret;
> }
> diff --git a/mm/swap_state.c b/mm/swap_state.c
> index 405923f..5721f64 100644
> --- a/mm/swap_state.c
> +++ b/mm/swap_state.c
> @@ -478,7 +478,7 @@ struct page *swapin_readahead(swp_entry_t entry, gfp_t gfp_mask,
> SetPageReadahead(page);
> page_cache_release(page);
> }
> - blk_finish_plug(&plug);
> + blk_finish_plug();
>
> lru_add_drain(); /* Push any new pages onto the LRU now */
> skip:
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 5e8eadd..56bb274 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -2222,7 +2222,7 @@ static void shrink_lruvec(struct lruvec *lruvec, int swappiness,
>
> scan_adjusted = true;
> }
> - blk_finish_plug(&plug);
> + blk_finish_plug();
> sc->nr_reclaimed += nr_reclaimed;
>
> /*
> --
> 1.8.3.1
>
>
--
Dave Chinner
david@fromorbit.com
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* [PATCH 4/4] md-cluster: re-add
From: Goldwyn Rodrigues @ 2015-04-08 19:24 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, GQJiang
This extends the capabilites of re-adding a failed device
to the clustering environment.
A new function gather_bitmaps gathers set bits from bitmaps of
all nodes, sends a message to all nodes to readd the disk
and then initiates the recovery process.
Question: Do you see a race in sending a READD and then performing
the bitmap resync/recovery? Should the initiating node perform the
recovery before sending the READD message? The recovery will send a
METADATA_UPDATE anyways.
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
---
drivers/md/bitmap.c | 20 +++++++++++---------
drivers/md/bitmap.h | 2 +-
drivers/md/md-cluster.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++++-
drivers/md/md-cluster.h | 1 +
drivers/md/md.c | 2 ++
5 files changed, 64 insertions(+), 11 deletions(-)
diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
index eccfa27..3e69583 100644
--- a/drivers/md/bitmap.c
+++ b/drivers/md/bitmap.c
@@ -1869,7 +1869,7 @@ EXPORT_SYMBOL_GPL(bitmap_load);
* to our bitmap
*/
int bitmap_copy_from_slot(struct mddev *mddev, int slot,
- sector_t *low, sector_t *high)
+ sector_t *low, sector_t *high, bool clear_bits)
{
int rv = 0, i, j;
sector_t block, lo = 0, hi = 0;
@@ -1896,14 +1896,16 @@ int bitmap_copy_from_slot(struct mddev *mddev, int slot,
}
}
- bitmap_update_sb(bitmap);
- /* Setting this for the ev_page should be enough.
- * And we do not require both write_all and PAGE_DIRT either
- */
- for (i = 0; i < bitmap->storage.file_pages; i++)
- set_page_attr(bitmap, i, BITMAP_PAGE_DIRTY);
- bitmap_write_all(bitmap);
- bitmap_unplug(bitmap);
+ if (clear_bits) {
+ bitmap_update_sb(bitmap);
+ /* Setting this for the ev_page should be enough.
+ * And we do not require both write_all and PAGE_DIRT either
+ */
+ for (i = 0; i < bitmap->storage.file_pages; i++)
+ set_page_attr(bitmap, i, BITMAP_PAGE_DIRTY);
+ bitmap_write_all(bitmap);
+ bitmap_unplug(bitmap);
+ }
*low = lo;
*high = hi;
err:
diff --git a/drivers/md/bitmap.h b/drivers/md/bitmap.h
index e838ea7..74bffc7 100644
--- a/drivers/md/bitmap.h
+++ b/drivers/md/bitmap.h
@@ -263,7 +263,7 @@ void bitmap_daemon_work(struct mddev *mddev);
int bitmap_resize(struct bitmap *bitmap, sector_t blocks,
int chunksize, int init);
int bitmap_copy_from_slot(struct mddev *mddev, int slot,
- sector_t *lo, sector_t *hi);
+ sector_t *lo, sector_t *hi, bool clear_bits);
#endif
#endif
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index d036c83..afffbee 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -50,6 +50,7 @@ struct md_cluster_info {
/* dlm lock space and resources for clustered raid. */
dlm_lockspace_t *lockspace;
int slot_number;
+ int total_slots;
struct completion completion;
struct dlm_lock_resource *sb_lock;
struct mutex sb_mutex;
@@ -73,6 +74,7 @@ enum msg_type {
RESYNCING,
NEWDISK,
REMOVE,
+ READD,
};
struct cluster_msg {
@@ -267,7 +269,7 @@ void recover_bitmaps(struct md_thread *thread)
str, ret);
goto clear_bit;
}
- ret = bitmap_copy_from_slot(mddev, slot, &lo, &hi);
+ ret = bitmap_copy_from_slot(mddev, slot, &lo, &hi, true);
if (ret) {
pr_err("md-cluster: Could not copy data from bitmap %d\n", slot);
goto dlm_unlock;
@@ -427,6 +429,17 @@ static void process_remove_disk(struct mddev *mddev, struct cluster_msg *msg)
pr_warn("%s: %d Could not find disk with uuid: %s", __func__, __LINE__, pretty_uuid(uuid, msg->uuid));
}
+static void process_readd_disk(struct mddev *mddev, struct cluster_msg *msg)
+{
+ struct md_rdev *rdev = find_rdev_uuid(mddev, msg->uuid);
+ char uuid[32];
+
+ if (rdev)
+ clear_bit(Faulty, &rdev->flags);
+ else
+ pr_warn("%s: %d Could not find disk with uuid: %s", __func__, __LINE__, pretty_uuid(uuid, msg->uuid));
+}
+
static void process_recvd_msg(struct mddev *mddev, struct cluster_msg *msg)
{
switch (msg->type) {
@@ -451,6 +464,11 @@ static void process_recvd_msg(struct mddev *mddev, struct cluster_msg *msg)
__func__, __LINE__, msg->slot);
process_remove_disk(mddev, msg);
break;
+ case READD:
+ pr_info("%s: %d Received READD from %d\n",
+ __func__, __LINE__, msg->slot);
+ process_readd_disk(mddev, msg);
+ break;
default:
pr_warn("%s:%d Received unknown message from %d\n",
__func__, __LINE__, msg->slot);
@@ -653,6 +671,7 @@ static int join(struct mddev *mddev, int nodes)
ret = -ERANGE;
goto err;
}
+ cinfo->total_slots = nodes;
cinfo->sb_lock = lockres_init(mddev, "cmd-super",
NULL, 0);
if (!cinfo->sb_lock) {
@@ -900,6 +919,34 @@ static int remove_disk(struct mddev *mddev, struct md_rdev *rdev)
return __sendmsg(cinfo, &cmsg);
}
+static int gather_bitmaps(struct md_rdev *rdev)
+{
+ int sn, err;
+ sector_t lo, hi;
+ struct cluster_msg cmsg;
+ struct mddev *mddev = rdev->mddev;
+ struct md_cluster_info *cinfo = mddev->cluster_info;
+ struct mdp_superblock_1 *sb = page_address(rdev->sb_page);
+ char *uuid = sb->device_uuid;
+
+ for (sn = 0; sn < cinfo->total_slots; sn++) {
+ if (sn == (cinfo->slot_number - 1))
+ continue;
+ err = bitmap_copy_from_slot(mddev, sn, &lo, &hi, false);
+ if (err) {
+ pr_warn("md-cluster: Could not gather bitmaps from slot %d", sn);
+ goto out;
+ }
+ if ((hi > 0) && (lo < mddev->recovery_cp))
+ mddev->recovery_cp = lo;
+ }
+ cmsg.type = READD;
+ memcpy(cmsg.uuid, uuid, 16);
+ err = sendmsg(cinfo, &cmsg);
+out:
+ return err;
+}
+
static struct md_cluster_operations cluster_ops = {
.join = join,
.leave = leave,
@@ -915,6 +962,7 @@ static struct md_cluster_operations cluster_ops = {
.add_new_disk_finish = add_new_disk_finish,
.new_disk_ack = new_disk_ack,
.remove_disk = remove_disk,
+ .gather_bitmaps = gather_bitmaps,
};
static int __init cluster_init(void)
diff --git a/drivers/md/md-cluster.h b/drivers/md/md-cluster.h
index 71e5143..6817ee0 100644
--- a/drivers/md/md-cluster.h
+++ b/drivers/md/md-cluster.h
@@ -23,6 +23,7 @@ struct md_cluster_operations {
int (*add_new_disk_finish)(struct mddev *mddev);
int (*new_disk_ack)(struct mddev *mddev, bool ack);
int (*remove_disk)(struct mddev *mddev, struct md_rdev *rdev);
+ int (*gather_bitmaps)(struct md_rdev *rdev);
};
#endif /* _MD_CLUSTER_H */
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 83a8e91..a233c09 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -2847,6 +2847,8 @@ state_store(struct md_rdev *rdev, const char *buf, size_t len)
err = 0;
}
} else if (cmd_match(buf, "re-add")) {
+ if (mddev_is_clustered(rdev->mddev))
+ md_cluster_ops->gather_bitmaps(rdev);
clear_bit(Faulty, &rdev->flags);
err = add_bound_rdev(rdev);
}
--
2.1.4
^ permalink raw reply related
* [PATCH 3/4] re-add a failed disk
From: Goldwyn Rodrigues @ 2015-04-08 19:23 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, GQJiang
This adds the capability of re-adding a failed disk by
writing "re-add" to /sys/block/mdXX/md/dev-YYY/state.
This works for generic md, and is not (completely)
related to clustering.
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
---
drivers/md/md.c | 56 ++++++++++++++++++++++++++++++++++++--------------------
1 file changed, 36 insertions(+), 20 deletions(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 0c65e51..83a8e91 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -79,6 +79,7 @@ static struct workqueue_struct *md_misc_wq;
static int remove_and_add_spares(struct mddev *mddev,
struct md_rdev *this);
+static int add_bound_rdev(struct md_rdev *);
#define MD_BUG(x...) { printk("md: bug in file %s, line %d\n", __FILE__, __LINE__); md_print_devices(); }
@@ -2845,6 +2846,9 @@ state_store(struct md_rdev *rdev, const char *buf, size_t len)
clear_bit(Replacement, &rdev->flags);
err = 0;
}
+ } else if (cmd_match(buf, "re-add")) {
+ clear_bit(Faulty, &rdev->flags);
+ err = add_bound_rdev(rdev);
}
if (!err)
sysfs_notify_dirent_safe(rdev->sysfs_state);
@@ -5861,6 +5865,37 @@ static int get_disk_info(struct mddev * mddev, void __user * arg)
return 0;
}
+static int add_bound_rdev(struct md_rdev *rdev)
+{
+ struct mddev *mddev = rdev->mddev;
+ int err = 0;
+
+ if (!mddev->pers->hot_remove_disk) {
+ /* If there is hot_add_disk but no hot_remove_disk
+ * then added disks for geometry changes,
+ * and should be added immediately.
+ */
+ super_types[mddev->major_version].
+ validate_super(mddev, rdev);
+ err = mddev->pers->hot_add_disk(mddev, rdev);
+ if (err)
+ unbind_rdev_from_array(rdev);
+ }
+ if (err)
+ export_rdev(rdev);
+ else
+ sysfs_notify_dirent_safe(rdev->sysfs_state);
+
+ set_bit(MD_CHANGE_DEVS, &mddev->flags);
+ if (mddev->degraded)
+ set_bit(MD_RECOVERY_RECOVER, &mddev->recovery);
+ set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
+ if (!err)
+ md_new_event(mddev);
+ md_wakeup_thread(mddev->thread);
+ return err;
+}
+
static int add_new_disk(struct mddev * mddev, mdu_disk_info_t *info)
{
char b[BDEVNAME_SIZE], b2[BDEVNAME_SIZE];
@@ -5989,29 +6024,10 @@ static int add_new_disk(struct mddev * mddev, mdu_disk_info_t *info)
rdev->raid_disk = -1;
err = bind_rdev_to_array(rdev, mddev);
- if (!err && !mddev->pers->hot_remove_disk) {
- /* If there is hot_add_disk but no hot_remove_disk
- * then added disks for geometry changes,
- * and should be added immediately.
- */
- super_types[mddev->major_version].
- validate_super(mddev, rdev);
- err = mddev->pers->hot_add_disk(mddev, rdev);
- if (err)
- unbind_rdev_from_array(rdev);
- }
if (err)
export_rdev(rdev);
else
- sysfs_notify_dirent_safe(rdev->sysfs_state);
-
- set_bit(MD_CHANGE_DEVS, &mddev->flags);
- if (mddev->degraded)
- set_bit(MD_RECOVERY_RECOVER, &mddev->recovery);
- set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
- if (!err)
- md_new_event(mddev);
- md_wakeup_thread(mddev->thread);
+ err = add_bound_rdev(rdev);
if (mddev_is_clustered(mddev) &&
(info->state & (1 << MD_DISK_CLUSTER_ADD)))
md_cluster_ops->add_new_disk_finish(mddev);
--
2.1.4
^ permalink raw reply related
* [PATCH 2/4] md-cluster: remove capabilities
From: Goldwyn Rodrigues @ 2015-04-08 19:22 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, GQJiang
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
---
drivers/md/md-cluster.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++
drivers/md/md-cluster.h | 1 +
drivers/md/md.c | 24 +++++++++++++++---------
drivers/md/md.h | 1 +
4 files changed, 64 insertions(+), 9 deletions(-)
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index 96679b2..d036c83 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -72,6 +72,7 @@ enum msg_type {
METADATA_UPDATED = 0,
RESYNCING,
NEWDISK,
+ REMOVE,
};
struct cluster_msg {
@@ -186,6 +187,20 @@ static char *pretty_uuid(char *dest, char *src)
return dest;
}
+static struct md_rdev *find_rdev_uuid(struct mddev *mddev, char *uuid)
+{
+ struct md_rdev *rdev;
+ struct mdp_superblock_1 *sb;
+
+ rdev_for_each_rcu(rdev, mddev) {
+ sb = page_address(rdev->sb_page);
+ if (!strncmp(uuid, sb->device_uuid, 16)) {
+ return rdev;
+ }
+ }
+ return NULL;
+}
+
static void add_resync_info(struct mddev *mddev, struct dlm_lock_resource *lockres,
sector_t lo, sector_t hi)
{
@@ -401,6 +416,17 @@ static void process_metadata_update(struct mddev *mddev, struct cluster_msg *msg
dlm_lock_sync(cinfo->no_new_dev_lockres, DLM_LOCK_CR);
}
+static void process_remove_disk(struct mddev *mddev, struct cluster_msg *msg)
+{
+ struct md_rdev *rdev = find_rdev_uuid(mddev, msg->uuid);
+ char uuid[32];
+
+ if (rdev)
+ md_kick_rdev_from_array(rdev);
+ else
+ pr_warn("%s: %d Could not find disk with uuid: %s", __func__, __LINE__, pretty_uuid(uuid, msg->uuid));
+}
+
static void process_recvd_msg(struct mddev *mddev, struct cluster_msg *msg)
{
switch (msg->type) {
@@ -419,6 +445,15 @@ static void process_recvd_msg(struct mddev *mddev, struct cluster_msg *msg)
pr_info("%s: %d Received message: NEWDISK from %d\n",
__func__, __LINE__, msg->slot);
process_add_new_disk(mddev, msg);
+ break;
+ case REMOVE:
+ pr_info("%s: %d Received REMOVE from %d\n",
+ __func__, __LINE__, msg->slot);
+ process_remove_disk(mddev, msg);
+ break;
+ default:
+ pr_warn("%s:%d Received unknown message from %d\n",
+ __func__, __LINE__, msg->slot);
};
}
@@ -854,6 +889,17 @@ static int new_disk_ack(struct mddev *mddev, bool ack)
return 0;
}
+static int remove_disk(struct mddev *mddev, struct md_rdev *rdev)
+{
+ struct cluster_msg cmsg;
+ struct md_cluster_info *cinfo = mddev->cluster_info;
+ struct mdp_superblock_1 *sb = page_address(rdev->sb_page);
+ char *uuid = sb->device_uuid;
+ cmsg.type = REMOVE;
+ memcpy(cmsg.uuid, uuid, 16);
+ return __sendmsg(cinfo, &cmsg);
+}
+
static struct md_cluster_operations cluster_ops = {
.join = join,
.leave = leave,
@@ -868,6 +914,7 @@ static struct md_cluster_operations cluster_ops = {
.add_new_disk_start = add_new_disk_start,
.add_new_disk_finish = add_new_disk_finish,
.new_disk_ack = new_disk_ack,
+ .remove_disk = remove_disk,
};
static int __init cluster_init(void)
diff --git a/drivers/md/md-cluster.h b/drivers/md/md-cluster.h
index 7417133..71e5143 100644
--- a/drivers/md/md-cluster.h
+++ b/drivers/md/md-cluster.h
@@ -22,6 +22,7 @@ struct md_cluster_operations {
int (*add_new_disk_start)(struct mddev *mddev, struct md_rdev *rdev);
int (*add_new_disk_finish)(struct mddev *mddev);
int (*new_disk_ack)(struct mddev *mddev, bool ack);
+ int (*remove_disk)(struct mddev *mddev, struct md_rdev *rdev);
};
#endif /* _MD_CLUSTER_H */
diff --git a/drivers/md/md.c b/drivers/md/md.c
index bc11551..0c65e51 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -2291,11 +2291,12 @@ static void export_rdev(struct md_rdev * rdev)
kobject_put(&rdev->kobj);
}
-static void kick_rdev_from_array(struct md_rdev * rdev)
+void md_kick_rdev_from_array(struct md_rdev * rdev)
{
unbind_rdev_from_array(rdev);
export_rdev(rdev);
}
+EXPORT_SYMBOL_GPL(md_kick_rdev_from_array);
static void export_array(struct mddev *mddev)
{
@@ -2306,7 +2307,7 @@ static void export_array(struct mddev *mddev)
MD_BUG();
continue;
}
- kick_rdev_from_array(rdev);
+ md_kick_rdev_from_array(rdev);
}
if (!list_empty(&mddev->disks))
MD_BUG();
@@ -2750,9 +2751,11 @@ state_store(struct md_rdev *rdev, const char *buf, size_t len)
err = -EBUSY;
else {
struct mddev *mddev = rdev->mddev;
- if (mddev_is_clustered(mddev))
+ if (mddev_is_clustered(mddev)) {
md_cluster_ops->metadata_update_start(mddev);
- kick_rdev_from_array(rdev);
+ md_cluster_ops->remove_disk(mddev, rdev);
+ }
+ md_kick_rdev_from_array(rdev);
if (mddev->pers) {
set_bit(MD_CHANGE_DEVS, &mddev->flags);
md_wakeup_thread(mddev->thread);
@@ -3424,7 +3427,7 @@ static void analyze_sbs(struct mddev * mddev)
"md: fatal superblock inconsistency in %s"
" -- removing from array\n",
bdevname(rdev->bdev,b));
- kick_rdev_from_array(rdev);
+ md_kick_rdev_from_array(rdev);
}
@@ -3440,7 +3443,7 @@ static void analyze_sbs(struct mddev * mddev)
"md: %s: %s: only %d devices permitted\n",
mdname(mddev), bdevname(rdev->bdev, b),
mddev->max_disks);
- kick_rdev_from_array(rdev);
+ md_kick_rdev_from_array(rdev);
continue;
}
if (rdev != freshest) {
@@ -3449,7 +3452,7 @@ static void analyze_sbs(struct mddev * mddev)
printk(KERN_WARNING "md: kicking non-fresh %s"
" from array!\n",
bdevname(rdev->bdev,b));
- kick_rdev_from_array(rdev);
+ md_kick_rdev_from_array(rdev);
continue;
}
/* No device should have a Candidate flag
@@ -3458,7 +3461,7 @@ static void analyze_sbs(struct mddev * mddev)
if (test_bit(Candidate, &rdev->flags)) {
pr_info("md: kicking Cluster Candidate %s from array!\n",
bdevname(rdev->bdev, b));
- kick_rdev_from_array(rdev);
+ md_kick_rdev_from_array(rdev);
}
}
if (mddev->level == LEVEL_MULTIPATH) {
@@ -6083,7 +6086,10 @@ static int hot_remove_disk(struct mddev * mddev, dev_t dev)
if (rdev->raid_disk >= 0)
goto busy;
- kick_rdev_from_array(rdev);
+ if (mddev_is_clustered(mddev))
+ md_cluster_ops->remove_disk(mddev, rdev);
+
+ md_kick_rdev_from_array(rdev);
set_bit(MD_CHANGE_DEVS, &mddev->flags);
if (mddev->thread)
md_wakeup_thread(mddev->thread);
diff --git a/drivers/md/md.h b/drivers/md/md.h
index 29ab989..c6c0846 100644
--- a/drivers/md/md.h
+++ b/drivers/md/md.h
@@ -651,6 +651,7 @@ extern void md_trim_bio(struct bio *bio, int offset, int size);
extern void md_unplug(struct blk_plug_cb *cb, bool from_schedule);
extern void md_reload_sb(struct mddev *mddev);
extern void md_update_sb(struct mddev *mddev, int force);
+extern void md_kick_rdev_from_array(struct md_rdev *rdev);
static inline void rdev_dec_pending(struct md_rdev *rdev, struct mddev *mddev)
{
--
2.1.4
^ permalink raw reply related
* [PATCH 1/4] md-cluster: fix bitmap sub-offset in bitmap_read_sb
From: Goldwyn Rodrigues @ 2015-04-08 19:22 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, GQJiang
bitmap_read_sb is modifying mddev->bitmap_info.offset. Thi works for
the first bitmap read. However, when multiple bitmaps need to be opened
by the same node, it ends up corrupting the offset. Fix it by using a
local variable.
Also, bitmap_read_sb is not required in bitmap_copy_from_slot since
it is called in bitmap_create. Remove bitmap_read_sb().
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
---
drivers/md/bitmap.c | 11 ++++-------
1 file changed, 4 insertions(+), 7 deletions(-)
diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
index e495604..eccfa27 100644
--- a/drivers/md/bitmap.c
+++ b/drivers/md/bitmap.c
@@ -547,6 +547,7 @@ static int bitmap_read_sb(struct bitmap *bitmap)
unsigned long sectors_reserved = 0;
int err = -EINVAL;
struct page *sb_page;
+ loff_t offset = bitmap->mddev->bitmap_info.offset;
if (!bitmap->storage.file && !bitmap->mddev->bitmap_info.offset) {
chunksize = 128 * 1024 * 1024;
@@ -573,9 +574,9 @@ re_read:
bm_blocks = ((bm_blocks+7) >> 3) + sizeof(bitmap_super_t);
/* to 4k blocks */
bm_blocks = DIV_ROUND_UP_SECTOR_T(bm_blocks, 4096);
- bitmap->mddev->bitmap_info.offset += bitmap->cluster_slot * (bm_blocks << 3);
+ offset = bitmap->mddev->bitmap_info.offset + (bitmap->cluster_slot * (bm_blocks << 3));
pr_info("%s:%d bm slot: %d offset: %llu\n", __func__, __LINE__,
- bitmap->cluster_slot, (unsigned long long)bitmap->mddev->bitmap_info.offset);
+ bitmap->cluster_slot, offset);
}
if (bitmap->storage.file) {
@@ -586,7 +587,7 @@ re_read:
bitmap, bytes, sb_page);
} else {
err = read_sb_page(bitmap->mddev,
- bitmap->mddev->bitmap_info.offset,
+ offset,
sb_page,
0, sizeof(bitmap_super_t));
}
@@ -1878,10 +1879,6 @@ int bitmap_copy_from_slot(struct mddev *mddev, int slot,
if (IS_ERR(bitmap))
return PTR_ERR(bitmap);
- rv = bitmap_read_sb(bitmap);
- if (rv)
- goto err;
-
rv = bitmap_init_from_disk(bitmap, 0);
if (rv)
goto err;
--
2.1.4
^ permalink raw reply related
* Re: Help recovering an interrupted raid0 reshape
From: Jonathan Harker (Jesusaurus) @ 2015-04-08 17:00 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150408120955.692b2564@notabene.brown>
On Tue, Apr 7, 2015 at 7:09 PM, NeilBrown <neilb@suse.de> wrote:
> On Tue, 7 Apr 2015 16:24:35 -0700 "Jonathan Harker (Jesusaurus)"
> <jesusaurus@gentlydownthe.net> wrote:
>
>> On Tue, Apr 7, 2015 at 3:56 PM, NeilBrown <neilb@suse.de> wrote:
>> >
>> >
>> > Hmm... I think I see the bug. It should be easy enough to fix, but I'd like
>> > to be able to test it.
>> > Could you please:
>> >
>> > mkdir /tmp/md.metadata
>> > mdadm --dump /tmp/md.metadata /dev/md/alpha /dev/md/beta /dev/md/gamma
>> > tar czSf /tmp/md.tgz /tmp/md.metadata
>> >
>> > and then send me /tmp/md.tgz, which should be tiny and contain just the
>> > metadata from the array.
>>
>> I'll send that to you once tar finishes.
>
> Thanks. And just as well I tested as there was another problem.
> I think it is all fixed now.
> If you:
>
> cd /tmp
> git clone git://neil.brown.name/mdadm
> cd mdadm
> make
> ./mdadm -A /dev/md124 --force -vvvv /dev/md/{alpha,beta,gamma}
>
> it should assemble the array and continue the reshape.
>
> When it finishes it will leave the array as degraded RAID4.
> Simply:
> mdadm /dev/md125 --grow --level=0
>
> and it will instantly become RAID0.
>
> NeilBrown
>
>>
>> >
>> > [[the patch which introduced the problem has a description which starts
>> > "This is a bit of a hack and ..."
>> > Never accept hacks!
>> > ]]
>>
>> I've reviewed and accepted a few hacks in my day as well. Hopefully
>> the cleanup isn't too ugly.
>> Thanks again for looking into this.
>>
>
Awesome, the interrupted reshape has restarted! And in about 900 short
minutes I should be able to reshape back to a raid0.
Thanks again for all your help.
--
Jonathan
^ permalink raw reply
* Re: [PATCH 2/2][v2] blk-plug: don't flush nested plug lists
From: Christoph Hellwig @ 2015-04-08 16:13 UTC (permalink / raw)
To: Jeff Moyer
Cc: Jens Axboe, Ming Lei, Konrad Rzeszutek Wilk, Roger Pau Monn??,
Alasdair Kergon, Mike Snitzer, Neil Brown, Nicholas A. Bellinger,
Alexander Viro, Chris Mason, Josef Bacik, David Sterba,
Theodore Ts'o, Andreas Dilger, Jaegeuk Kim, Changman Lee,
Steven Whitehouse, Mikulas Patocka, Andrew Morton, Rik van Riel,
Johannes Weiner, Mel Gorman, Trond Myklebust
In-Reply-To: <x49wq1nrcoe.fsf_-_@segfault.boston.devel.redhat.com>
This looks good, but without the blk_finish_plug argument we're bound
to grow programming mistakes where people forget it. Any chance we
could have annotations similar to say rcu_read_lock/rcu_read_unlock
or the spinlocks so that sparse warns about it?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* [PATCH] IMSM: Count arrays per orom
From: Pawel Baldysiak @ 2015-04-08 9:42 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, pawel.baldysiak, artur.paszkiewicz
Active arrays with IMSM metadata are counted per hba so far.
This is bad due to new functionality of orom shared between multiple
controllers i.e. more arrays can be created than is supported by orom.
This patch changes the way of counting arrays, so the result will be
sum of arrays under every hba supported by specific orom.
Signed-off-by: Pawel Baldysiak <pawel.baldysiak@intel.com>
---
platform-intel.c | 14 ++++++++-
platform-intel.h | 1 +
super-intel.c | 85 +++++++++++++++++++++++++++++++++++++-----------------
3 files changed, 71 insertions(+), 29 deletions(-)
diff --git a/platform-intel.c b/platform-intel.c
index 1e9ddcd..edb8679 100644
--- a/platform-intel.c
+++ b/platform-intel.c
@@ -233,7 +233,7 @@ struct pciExpDataStructFormat {
struct orom_entry *orom_entries;
-const struct imsm_orom *get_orom_by_device_id(__u16 dev_id)
+const struct orom_entry *get_orom_entry_by_device_id(__u16 dev_id)
{
struct orom_entry *entry;
struct devid_list *devid;
@@ -241,13 +241,23 @@ const struct imsm_orom *get_orom_by_device_id(__u16 dev_id)
for (entry = orom_entries; entry; entry = entry->next) {
for (devid = entry->devid_list; devid; devid = devid->next) {
if (devid->devid == dev_id)
- return &entry->orom;
+ return entry;
}
}
return NULL;
}
+const struct imsm_orom *get_orom_by_device_id(__u16 dev_id)
+{
+ const struct orom_entry *entry = get_orom_entry_by_device_id(dev_id);
+
+ if (entry)
+ return &entry->orom;
+
+ return NULL;
+}
+
static struct orom_entry *add_orom(const struct imsm_orom *orom)
{
struct orom_entry *list;
diff --git a/platform-intel.h b/platform-intel.h
index 631fa76..695d6c6 100644
--- a/platform-intel.h
+++ b/platform-intel.h
@@ -238,5 +238,6 @@ int devt_attached_to_hba(dev_t dev, const char *hba_path);
char *devt_to_devpath(dev_t dev);
int path_attached_to_hba(const char *disk_path, const char *hba_path);
const char *get_sys_dev_type(enum sys_dev_type);
+const struct orom_entry *get_orom_entry_by_device_id(__u16 dev_id);
const struct imsm_orom *get_orom_by_device_id(__u16 device_id);
struct sys_dev *device_by_id(__u16 device_id);
diff --git a/super-intel.c b/super-intel.c
index 77df8db..66cc8e5 100644
--- a/super-intel.c
+++ b/super-intel.c
@@ -5853,37 +5853,68 @@ count_volumes_list(struct md_list *devlist, char *homehost,
}
static int
-count_volumes(char *hba, int dpa, int verbose)
+count_volumes(struct intel_hba *hba, int dpa, int verbose)
{
- struct md_list *devlist = NULL;
+ struct sys_dev *idev, *intel_devices = find_intel_devices();
int count = 0;
- int found = 0;;
+ const struct orom_entry *entry;
+ struct devid_list *dv, *devid_list;
- devlist = get_devices(hba);
- /* if no intel devices return zero volumes */
- if (devlist == NULL)
+ if (!hba || !hba->path)
return 0;
- count = active_arrays_by_format("imsm", hba, &devlist, dpa, verbose);
- dprintf("path: %s active arrays: %d\n", hba, count);
- if (devlist == NULL)
+ for (idev = intel_devices; idev; idev = idev->next) {
+ if (strstr(idev->path, hba->path))
+ break;
+ }
+
+ if (!idev || !idev->dev_id)
return 0;
- do {
- found = 0;
- count += count_volumes_list(devlist,
- NULL,
- verbose,
- &found);
- dprintf("found %d count: %d\n", found, count);
- } while (found);
-
- dprintf("path: %s total number of volumes: %d\n", hba, count);
-
- while(devlist) {
- struct md_list *dv = devlist;
- devlist = devlist->next;
- free(dv->devname);
- free(dv);
+
+ entry = get_orom_entry_by_device_id(idev->dev_id);
+
+ if (!entry || !entry->devid_list)
+ return 0;
+
+ devid_list = entry->devid_list;
+ for (dv = devid_list; dv; dv = dv->next) {
+
+ struct md_list *devlist = NULL;
+ struct sys_dev *device = device_by_id(dv->devid);
+ char *hba_path;
+ int found = 0;
+
+ if (device)
+ hba_path = device->path;
+ else
+ return 0;
+
+ devlist = get_devices(hba_path);
+ /* if no intel devices return zero volumes */
+ if (devlist == NULL)
+ return 0;
+
+ count += active_arrays_by_format("imsm", hba_path, &devlist, dpa, verbose);
+ dprintf("path: %s active arrays: %d\n", hba_path, count);
+ if (devlist == NULL)
+ return 0;
+ do {
+ found = 0;
+ count += count_volumes_list(devlist,
+ NULL,
+ verbose,
+ &found);
+ dprintf("found %d count: %d\n", found, count);
+ } while (found);
+
+ dprintf("path: %s total number of volumes: %d\n", hba_path, count);
+
+ while (devlist) {
+ struct md_list *dv = devlist;
+ devlist = devlist->next;
+ free(dv->devname);
+ free(dv);
+ }
}
return count;
}
@@ -6105,7 +6136,7 @@ static int validate_geometry_imsm_volume(struct supertype *st, int level,
*freesize = maxsize;
if (super->orom) {
- int count = count_volumes(super->hba->path,
+ int count = count_volumes(super->hba,
super->orom->dpa, verbose);
if (super->orom->vphba <= count) {
pr_vrb(": platform does not support more than %d raid volumes.\n",
@@ -6261,7 +6292,7 @@ static int validate_geometry_imsm(struct supertype *st, int level, int layout,
created */
if (super->orom && freesize) {
int count;
- count = count_volumes(super->hba->path,
+ count = count_volumes(super->hba,
super->orom->dpa, verbose);
if (super->orom->vphba <= count) {
pr_vrb(": platform does not support more than %d raid volumes.\n",
^ permalink raw reply related
* Re: [PATCH] md: fix md io stats accounting broken
From: NeilBrown @ 2015-04-08 3:11 UTC (permalink / raw)
To: Gu Zheng; +Cc: sim, linux-raid, linux-kernel, stable
In-Reply-To: <1428021887-18128-1-git-send-email-guz.fnst@cn.fujitsu.com>
[-- Attachment #1: Type: text/plain, Size: 2752 bytes --]
On Fri, 3 Apr 2015 08:44:47 +0800 Gu Zheng <guz.fnst@cn.fujitsu.com> wrote:
> Simon reported the md io stats accounting issue:
> "
> I'm seeing "iostat -x -k 1" print this after a RAID1 rebuild on 4.0-rc5.
> It's not abnormal other than it's 3-disk, with one being SSD (sdc) and
> the other two being write-mostly:
>
> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
> sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 345.00 0.00 0.00 0.00 0.00 100.00
> md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 58779.00 0.00 0.00 0.00 0.00 100.00
> md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12.00 0.00 0.00 0.00 0.00 100.00
> "
> The cause is commit "18c0b223cf9901727ef3b02da6711ac930b4e5d4" uses the
> generic_start_io_acct to account the disk stats rather than the open code,
> but it also introduced the increase to .in_flight[rw] which is needless to
> md. So we re-use the open code here to fix it.
>
> Reported-by: Simon Kirby <sim@hostway.ca>
> Cc: <stable@vger.kernel.org> 3.19
> Signed-off-by: Gu Zheng <guz.fnst@cn.fujitsu.com>
> ---
> drivers/md/md.c | 6 +++++-
> 1 files changed, 5 insertions(+), 1 deletions(-)
>
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index 717daad..e617878 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -249,6 +249,7 @@ static void md_make_request(struct request_queue *q, struct bio *bio)
> const int rw = bio_data_dir(bio);
> struct mddev *mddev = q->queuedata;
> unsigned int sectors;
> + int cpu;
>
> if (mddev == NULL || mddev->pers == NULL
> || !mddev->ready) {
> @@ -284,7 +285,10 @@ static void md_make_request(struct request_queue *q, struct bio *bio)
> sectors = bio_sectors(bio);
> mddev->pers->make_request(mddev, bio);
>
> - generic_start_io_acct(rw, sectors, &mddev->gendisk->part0);
> + cpu = part_stat_lock();
> + part_stat_inc(cpu, &mddev->gendisk->part0, ios[rw]);
> + part_stat_add(cpu, &mddev->gendisk->part0, sectors[rw], sectors);
> + part_stat_unlock();
>
> if (atomic_dec_and_test(&mddev->active_io) && mddev->suspended)
> wake_up(&mddev->sb_wait);
Applied, thanks.
Will push to Linus in a day or 2.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Help recovering an interrupted raid0 reshape
From: NeilBrown @ 2015-04-08 2:09 UTC (permalink / raw)
To: Jonathan Harker (Jesusaurus); +Cc: linux-raid
In-Reply-To: <CAC_83AF0EHVVDF7Gwdfw9WB34v+nHCFmpjLR7AQ5qOz5QATgdQ@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1411 bytes --]
On Tue, 7 Apr 2015 16:24:35 -0700 "Jonathan Harker (Jesusaurus)"
<jesusaurus@gentlydownthe.net> wrote:
> On Tue, Apr 7, 2015 at 3:56 PM, NeilBrown <neilb@suse.de> wrote:
> >
> >
> > Hmm... I think I see the bug. It should be easy enough to fix, but I'd like
> > to be able to test it.
> > Could you please:
> >
> > mkdir /tmp/md.metadata
> > mdadm --dump /tmp/md.metadata /dev/md/alpha /dev/md/beta /dev/md/gamma
> > tar czSf /tmp/md.tgz /tmp/md.metadata
> >
> > and then send me /tmp/md.tgz, which should be tiny and contain just the
> > metadata from the array.
>
> I'll send that to you once tar finishes.
Thanks. And just as well I tested as there was another problem.
I think it is all fixed now.
If you:
cd /tmp
git clone git://neil.brown.name/mdadm
cd mdadm
make
./mdadm -A /dev/md124 --force -vvvv /dev/md/{alpha,beta,gamma}
it should assemble the array and continue the reshape.
When it finishes it will leave the array as degraded RAID4.
Simply:
mdadm /dev/md125 --grow --level=0
and it will instantly become RAID0.
NeilBrown
>
> >
> > [[the patch which introduced the problem has a description which starts
> > "This is a bit of a hack and ..."
> > Never accept hacks!
> > ]]
>
> I've reviewed and accepted a few hacks in my day as well. Hopefully
> the cleanup isn't too ugly.
> Thanks again for looking into this.
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Help recovering an interrupted raid0 reshape
From: Jonathan Harker (Jesusaurus) @ 2015-04-07 23:24 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150408085635.64fa7101@notabene.brown>
On Tue, Apr 7, 2015 at 3:56 PM, NeilBrown <neilb@suse.de> wrote:
>
>
> Hmm... I think I see the bug. It should be easy enough to fix, but I'd like
> to be able to test it.
> Could you please:
>
> mkdir /tmp/md.metadata
> mdadm --dump /tmp/md.metadata /dev/md/alpha /dev/md/beta /dev/md/gamma
> tar czSf /tmp/md.tgz /tmp/md.metadata
>
> and then send me /tmp/md.tgz, which should be tiny and contain just the
> metadata from the array.
I'll send that to you once tar finishes.
>
> [[the patch which introduced the problem has a description which starts
> "This is a bit of a hack and ..."
> Never accept hacks!
> ]]
I've reviewed and accepted a few hacks in my day as well. Hopefully
the cleanup isn't too ugly.
Thanks again for looking into this.
--
Jonathan
^ permalink raw reply
* Re: Help recovering an interrupted raid0 reshape
From: NeilBrown @ 2015-04-07 22:56 UTC (permalink / raw)
To: Jonathan Harker (Jesusaurus); +Cc: linux-raid
In-Reply-To: <CAC_83AGk2MK8=qy2CL-WN-ewfVzgA2D-WRkefpriXrdHSEjU-Q@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 12018 bytes --]
On Tue, 7 Apr 2015 15:31:32 -0700 "Jonathan Harker (Jesusaurus)"
<jesusaurus@gentlydownthe.net> wrote:
> On Tue, Apr 7, 2015 at 2:13 PM, NeilBrown <neilb@suse.de> wrote:
> > On Tue, 7 Apr 2015 10:02:13 -0700 "Jonathan Harker (Jesusaurus)"
> > <jesusaurus@gentlydownthe.net> wrote:
> >
> >> On Mon, Apr 6, 2015 at 11:30 PM, NeilBrown <neilb@suse.de> wrote:
> >> >
> >> > Try:
> >> > mdadm -S /dev/md124
> >> > mdadm -A /dev/md124 --update=revert-reshape /dev/md/alpha /dev/md/beta
> >> > mdadm -S /dev/md124
> >> > mdadm -A /dev/md124 -vvv /dev/md/alpha /dev/md/beta /dev/md/gamma
> >> >
> >> > What does that report?
> >> >
> >> > NeilBrown
> >> >
> >>
> >> # mdadm --stop /dev/md124
> >> mdadm: stopped /dev/md124
> >> # mdadm -A /dev/md124 --update=revert-reshape /dev/md/alpha /dev/md/beta
> >> mdadm: /dev/md124 assembled from 2 drives - not enough to start the array.
> >> # cat /proc/mdstat
> >> Personalities : [raid6] [raid5] [raid4] [raid1] [raid10] [raid0]
> >> [linear] [multipath]
> >> md124 : inactive md126[0](S) md127[1](S)
> >> 3907022200 blocks super 1.2
> >>
> >> md0 : active raid1 sda5[0] sdb2[1]
> >> 107652416 blocks [2/2] [UU]
> >> bitmap: 1/1 pages [4KB], 65536KB chunk
> >>
> >> md125 : active raid1 sdh1[0] sdg1[1]
> >> 2930134016 blocks super 1.2 [2/2] [UU]
> >> bitmap: 0/22 pages [0KB], 65536KB chunk
> >>
> >> md126 : active raid1 sdc1[0] sdd1[1]
> >> 1953512312 blocks super 1.2 [2/2] [UU]
> >>
> >> md127 : active raid1 sde1[2] sdf1[1]
> >> 1953512312 blocks super 1.2 [2/2] [UU]
> >>
> >> unused devices: <none>
> >> # mdadm --stop /dev/md124
> >> mdadm: stopped /dev/md124
> >> # mdadm -A /dev/md124 -vvv /dev/md/alpha /dev/md/beta /dev/md/gamma
> >> mdadm: looking for devices for /dev/md124
> >> mdadm: UUID differs from /dev/md0.
> >> mdadm: UUID differs from /dev/md/alpha.
> >> mdadm: UUID differs from /dev/md/beta.
> >> mdadm: UUID differs from /dev/md/gamma.
> >> mdadm: UUID differs from /dev/md0.
> >> mdadm: UUID differs from /dev/md/alpha.
> >> mdadm: UUID differs from /dev/md/beta.
> >> mdadm: UUID differs from /dev/md/gamma.
> >> mdadm: UUID differs from /dev/md0.
> >> mdadm: UUID differs from /dev/md/alpha.
> >> mdadm: UUID differs from /dev/md/beta.
> >> mdadm: UUID differs from /dev/md/gamma.
> >> mdadm: /dev/md/alpha is identified as a member of /dev/md124, slot 1.
> >> mdadm: /dev/md/beta is identified as a member of /dev/md124, slot 0.
> >> mdadm: /dev/md/gamma is identified as a member of /dev/md124, slot 2.
> >> mdadm: :/dev/md124 has an active reshape - checking if critical
> >> section needs to be restored
> >> mdadm: added /dev/md/alpha to /dev/md124 as 1
> >> mdadm: added /dev/md/gamma to /dev/md124 as 2 (possibly out of date)
> >> mdadm: no uptodate device for slot 6 of /dev/md124
> >> mdadm: added /dev/md/beta to /dev/md124 as 0
> >> mdadm: /dev/md124 assembled from 2 drives - not enough to start the array.
> >> # cat /proc/mdstat
> >> Personalities : [raid6] [raid5] [raid4] [raid1] [raid10] [raid0]
> >> [linear] [multipath]
> >> md124 : inactive md125[3](S) md127[1](S) md126[0](S)
> >> 6837155192 blocks super 1.2
> >>
> >> md0 : active raid1 sda5[0] sdb2[1]
> >> 107652416 blocks [2/2] [UU]
> >> bitmap: 0/1 pages [0KB], 65536KB chunk
> >>
> >> md125 : active raid1 sdh1[0] sdg1[1]
> >> 2930134016 blocks super 1.2 [2/2] [UU]
> >> bitmap: 0/22 pages [0KB], 65536KB chunk
> >>
> >> md126 : active raid1 sdc1[0] sdd1[1]
> >> 1953512312 blocks super 1.2 [2/2] [UU]
> >>
> >> md127 : active raid1 sde1[2] sdf1[1]
> >> 1953512312 blocks super 1.2 [2/2] [UU]
> >>
> >> unused devices: <none>
> >>
> >> # mdadm --examine /dev/md/alpha
> >> /dev/md/alpha:
> >> Magic : a92b4efc
> >> Version : 1.2
> >> Feature Map : 0x4
> >> Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
> >> Name : hordern:hordern1 (local to host hordern)
> >> Creation Time : Fri Jan 2 09:59:40 2009
> >> Raid Level : raid4
> >> Raid Devices : 4
> >>
> >> Avail Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
> >> Array Size : 5860532736 (5589.04 GiB 6001.19 GB)
> >> Data Offset : 2048 sectors
> >> Super Offset : 8 sectors
> >> Unused Space : before=1968 sectors, after=752 sectors
> >> State : active
> >> Device UUID : 63aaa2e4:2a09f495:8372c7f9:eb2f2773
> >>
> >> Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
> >> Delta Devices : 1 (3->4)
> >>
> >> Update Time : Sun Mar 29 15:11:35 2015
> >> Checksum : 8be5e0e8 - correct
> >> Events : 14013
> >>
> >> Chunk Size : 512K
> >>
> >> Device Role : Active device 1
> >> Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
> >>
> >> # mdadm --examine /dev/md/beta
> >> /dev/md/beta:
> >> Magic : a92b4efc
> >> Version : 1.2
> >> Feature Map : 0x4
> >> Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
> >> Name : hordern:hordern1 (local to host hordern)
> >> Creation Time : Fri Jan 2 09:59:40 2009
> >> Raid Level : raid4
> >> Raid Devices : 4
> >>
> >> Avail Dev Size : 3907022576 (1863.01 GiB 2000.40 GB)
> >> Array Size : 5860532736 (5589.04 GiB 6001.19 GB)
> >> Used Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
> >> Data Offset : 2048 sectors
> >> Super Offset : 8 sectors
> >> Unused Space : before=1968 sectors, after=752 sectors
> >> State : clean
> >> Device UUID : 6e6dce14:3ebb2bb5:187aa292:403a55f6
> >>
> >> Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
> >> Delta Devices : 1 (3->4)
> >>
> >> Update Time : Sun Mar 29 15:11:35 2015
> >> Checksum : f7526adf - correct
> >> Events : 14013
> >>
> >> Chunk Size : 512K
> >>
> >> Device Role : Active device 0
> >> Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
> >>
> >> # mdadm --examine /dev/md/gamma
> >> /dev/md/gamma:
> >> Magic : a92b4efc
> >> Version : 1.2
> >> Feature Map : 0x6
> >> Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
> >> Name : hordern:hordern1 (local to host hordern)
> >> Creation Time : Fri Jan 2 09:59:40 2009
> >> Raid Level : raid4
> >> Raid Devices : 4
> >>
> >> Avail Dev Size : 5860265984 (2794.39 GiB 3000.46 GB)
> >> Array Size : 5860532736 (5589.04 GiB 6001.19 GB)
> >> Used Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
> >> Data Offset : 2048 sectors
> >> Super Offset : 8 sectors
> >> Recovery Offset : 86403072 sectors
> >> Unused Space : before=1960 sectors, after=1953244160 sectors
> >> State : active
> >> Device UUID : 782873ea:e265ecd4:5cc80ddf:035ba2b4
> >>
> >> Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
> >> Delta Devices : 1 (3->4)
> >>
> >> Update Time : Sun Mar 29 00:05:29 2015
> >> Bad Block Log : 512 entries available at offset 72 sectors
> >> Checksum : 710dc078 - correct
> >> Events : 673
> >>
> >> Chunk Size : 512K
> >>
> >> Device Role : Active device 2
> >> Array State : AAA. ('A' == active, '.' == missing, 'R' == replacing)
> >>
> >> # mdadm --detail /dev/md124
> >> /dev/md124:
> >> Version : 1.2
> >> Raid Level : raid0
> >> Total Devices : 3
> >> Persistence : Superblock is persistent
> >>
> >> State : inactive
> >>
> >> Delta Devices : 1, (-1->0)
> >> New Level : raid4
> >> New Chunksize : 512K
> >>
> >> Name : hordern:hordern1 (local to host hordern)
> >> UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
> >> Events : 673
> >>
> >> Number Major Minor RaidDevice
> >>
> >> - 9 125 - /dev/md/gamma
> >> - 9 126 - /dev/md/beta
> >> - 9 127 - /dev/md/alpha
> >>
> >> So it looks like all three component devices have consistent
> >> superblocks now, awesome! But the raid0 array is still inactive with
> >> all three components listed as spares. It looks like /dev/md/gamma has
> >> a much lower event count, I'm guessing that is what causes the disk to
> >> be marked as possibly out of date.
> >>
> >> Is an "uptodate device" a specific thing, or does that simply mean
> >> that some component devices are out of date? The lack of spaces makes
> >> me think that uptodate is some keyword I'm not recognizing.
> >>
> >
> > Looks good. Nearly there.
> >
> > The difference in event counts is probably due to you trying lots of things
> > out, and them only affecting two devices.
> >
> > If you
> > # mdadm --stop /dev/md124
> > # mdadm -A --force /dev/md124 -vvv /dev/md/alpha /dev/md/beta /dev/md/gamma
> >
> > i.e. just add --force, it should ignored the difference in event count and
> > assemble the array.
> > For RAID0, the event count isn't really relevant to the data as there is no
> > possibility for inconsistency between data and parity on different devices.
> > As the reshape position is the same on all devices, I don't think there is
> > any risk at all in just using --force.
> > Of course, perform an fsck afterwards just to build confidence.
> >
> > NeilBrown
> >
>
> Unfortunately, adding --force didn't seem to make any difference:
>
> # mdadm --stop /dev/md124
> mdadm: stopped /dev/md124
> # mdadm -A --force /dev/md124 -vvv /dev/md/alpha /dev/md/beta /dev/md/gamma
> mdadm: looking for devices for /dev/md124
> mdadm: UUID differs from /dev/md0.
> mdadm: UUID differs from /dev/md/alpha.
> mdadm: UUID differs from /dev/md/beta.
> mdadm: UUID differs from /dev/md/gamma.
> mdadm: UUID differs from /dev/md0.
> mdadm: UUID differs from /dev/md/alpha.
> mdadm: UUID differs from /dev/md/beta.
> mdadm: UUID differs from /dev/md/gamma.
> mdadm: UUID differs from /dev/md0.
> mdadm: UUID differs from /dev/md/alpha.
> mdadm: UUID differs from /dev/md/beta.
> mdadm: UUID differs from /dev/md/gamma.
> mdadm: /dev/md/alpha is identified as a member of /dev/md124, slot 1.
> mdadm: /dev/md/beta is identified as a member of /dev/md124, slot 0.
> mdadm: /dev/md/gamma is identified as a member of /dev/md124, slot 2.
> mdadm: :/dev/md124 has an active reshape - checking if critical
> section needs to be restored
> mdadm: added /dev/md/alpha to /dev/md124 as 1
> mdadm: added /dev/md/gamma to /dev/md124 as 2 (possibly out of date)
> mdadm: no uptodate device for slot 6 of /dev/md124
> mdadm: added /dev/md/beta to /dev/md124 as 0
> mdadm: /dev/md124 assembled from 2 drives - not enough to start the array.
> # cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4] [raid1] [raid10] [raid0]
> [linear] [multipath]
> md124 : inactive md125[3](S) md127[1](S) md126[0](S)
> 6837155192 blocks super 1.2
>
> md0 : active raid1 sda5[0] sdb2[1]
> 107652416 blocks [2/2] [UU]
> bitmap: 0/1 pages [0KB], 65536KB chunk
>
> md125 : active raid1 sdh1[0] sdg1[1]
> 2930134016 blocks super 1.2 [2/2] [UU]
> bitmap: 0/22 pages [0KB], 65536KB chunk
>
> md126 : active raid1 sdc1[0] sdd1[1]
> 1953512312 blocks super 1.2 [2/2] [UU]
>
> md127 : active raid1 sde1[2] sdf1[1]
> 1953512312 blocks super 1.2 [2/2] [UU]
>
> unused devices: <none>
Hmm... I think I see the bug. It should be easy enough to fix, but I'd like
to be able to test it.
Could you please:
mkdir /tmp/md.metadata
mdadm --dump /tmp/md.metadata /dev/md/alpha /dev/md/beta /dev/md/gamma
tar czSf /tmp/md.tgz /tmp/md.metadata
and then send me /tmp/md.tgz, which should be tiny and contain just the
metadata from the array.
[[the patch which introduced the problem has a description which starts
"This is a bit of a hack and ..."
Never accept hacks!
]]
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Help recovering an interrupted raid0 reshape
From: Jonathan Harker (Jesusaurus) @ 2015-04-07 22:31 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150408071339.3295567b@notabene.brown>
On Tue, Apr 7, 2015 at 2:13 PM, NeilBrown <neilb@suse.de> wrote:
> On Tue, 7 Apr 2015 10:02:13 -0700 "Jonathan Harker (Jesusaurus)"
> <jesusaurus@gentlydownthe.net> wrote:
>
>> On Mon, Apr 6, 2015 at 11:30 PM, NeilBrown <neilb@suse.de> wrote:
>> >
>> > Try:
>> > mdadm -S /dev/md124
>> > mdadm -A /dev/md124 --update=revert-reshape /dev/md/alpha /dev/md/beta
>> > mdadm -S /dev/md124
>> > mdadm -A /dev/md124 -vvv /dev/md/alpha /dev/md/beta /dev/md/gamma
>> >
>> > What does that report?
>> >
>> > NeilBrown
>> >
>>
>> # mdadm --stop /dev/md124
>> mdadm: stopped /dev/md124
>> # mdadm -A /dev/md124 --update=revert-reshape /dev/md/alpha /dev/md/beta
>> mdadm: /dev/md124 assembled from 2 drives - not enough to start the array.
>> # cat /proc/mdstat
>> Personalities : [raid6] [raid5] [raid4] [raid1] [raid10] [raid0]
>> [linear] [multipath]
>> md124 : inactive md126[0](S) md127[1](S)
>> 3907022200 blocks super 1.2
>>
>> md0 : active raid1 sda5[0] sdb2[1]
>> 107652416 blocks [2/2] [UU]
>> bitmap: 1/1 pages [4KB], 65536KB chunk
>>
>> md125 : active raid1 sdh1[0] sdg1[1]
>> 2930134016 blocks super 1.2 [2/2] [UU]
>> bitmap: 0/22 pages [0KB], 65536KB chunk
>>
>> md126 : active raid1 sdc1[0] sdd1[1]
>> 1953512312 blocks super 1.2 [2/2] [UU]
>>
>> md127 : active raid1 sde1[2] sdf1[1]
>> 1953512312 blocks super 1.2 [2/2] [UU]
>>
>> unused devices: <none>
>> # mdadm --stop /dev/md124
>> mdadm: stopped /dev/md124
>> # mdadm -A /dev/md124 -vvv /dev/md/alpha /dev/md/beta /dev/md/gamma
>> mdadm: looking for devices for /dev/md124
>> mdadm: UUID differs from /dev/md0.
>> mdadm: UUID differs from /dev/md/alpha.
>> mdadm: UUID differs from /dev/md/beta.
>> mdadm: UUID differs from /dev/md/gamma.
>> mdadm: UUID differs from /dev/md0.
>> mdadm: UUID differs from /dev/md/alpha.
>> mdadm: UUID differs from /dev/md/beta.
>> mdadm: UUID differs from /dev/md/gamma.
>> mdadm: UUID differs from /dev/md0.
>> mdadm: UUID differs from /dev/md/alpha.
>> mdadm: UUID differs from /dev/md/beta.
>> mdadm: UUID differs from /dev/md/gamma.
>> mdadm: /dev/md/alpha is identified as a member of /dev/md124, slot 1.
>> mdadm: /dev/md/beta is identified as a member of /dev/md124, slot 0.
>> mdadm: /dev/md/gamma is identified as a member of /dev/md124, slot 2.
>> mdadm: :/dev/md124 has an active reshape - checking if critical
>> section needs to be restored
>> mdadm: added /dev/md/alpha to /dev/md124 as 1
>> mdadm: added /dev/md/gamma to /dev/md124 as 2 (possibly out of date)
>> mdadm: no uptodate device for slot 6 of /dev/md124
>> mdadm: added /dev/md/beta to /dev/md124 as 0
>> mdadm: /dev/md124 assembled from 2 drives - not enough to start the array.
>> # cat /proc/mdstat
>> Personalities : [raid6] [raid5] [raid4] [raid1] [raid10] [raid0]
>> [linear] [multipath]
>> md124 : inactive md125[3](S) md127[1](S) md126[0](S)
>> 6837155192 blocks super 1.2
>>
>> md0 : active raid1 sda5[0] sdb2[1]
>> 107652416 blocks [2/2] [UU]
>> bitmap: 0/1 pages [0KB], 65536KB chunk
>>
>> md125 : active raid1 sdh1[0] sdg1[1]
>> 2930134016 blocks super 1.2 [2/2] [UU]
>> bitmap: 0/22 pages [0KB], 65536KB chunk
>>
>> md126 : active raid1 sdc1[0] sdd1[1]
>> 1953512312 blocks super 1.2 [2/2] [UU]
>>
>> md127 : active raid1 sde1[2] sdf1[1]
>> 1953512312 blocks super 1.2 [2/2] [UU]
>>
>> unused devices: <none>
>>
>> # mdadm --examine /dev/md/alpha
>> /dev/md/alpha:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x4
>> Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
>> Name : hordern:hordern1 (local to host hordern)
>> Creation Time : Fri Jan 2 09:59:40 2009
>> Raid Level : raid4
>> Raid Devices : 4
>>
>> Avail Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
>> Array Size : 5860532736 (5589.04 GiB 6001.19 GB)
>> Data Offset : 2048 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=1968 sectors, after=752 sectors
>> State : active
>> Device UUID : 63aaa2e4:2a09f495:8372c7f9:eb2f2773
>>
>> Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
>> Delta Devices : 1 (3->4)
>>
>> Update Time : Sun Mar 29 15:11:35 2015
>> Checksum : 8be5e0e8 - correct
>> Events : 14013
>>
>> Chunk Size : 512K
>>
>> Device Role : Active device 1
>> Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
>>
>> # mdadm --examine /dev/md/beta
>> /dev/md/beta:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x4
>> Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
>> Name : hordern:hordern1 (local to host hordern)
>> Creation Time : Fri Jan 2 09:59:40 2009
>> Raid Level : raid4
>> Raid Devices : 4
>>
>> Avail Dev Size : 3907022576 (1863.01 GiB 2000.40 GB)
>> Array Size : 5860532736 (5589.04 GiB 6001.19 GB)
>> Used Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
>> Data Offset : 2048 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=1968 sectors, after=752 sectors
>> State : clean
>> Device UUID : 6e6dce14:3ebb2bb5:187aa292:403a55f6
>>
>> Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
>> Delta Devices : 1 (3->4)
>>
>> Update Time : Sun Mar 29 15:11:35 2015
>> Checksum : f7526adf - correct
>> Events : 14013
>>
>> Chunk Size : 512K
>>
>> Device Role : Active device 0
>> Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
>>
>> # mdadm --examine /dev/md/gamma
>> /dev/md/gamma:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x6
>> Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
>> Name : hordern:hordern1 (local to host hordern)
>> Creation Time : Fri Jan 2 09:59:40 2009
>> Raid Level : raid4
>> Raid Devices : 4
>>
>> Avail Dev Size : 5860265984 (2794.39 GiB 3000.46 GB)
>> Array Size : 5860532736 (5589.04 GiB 6001.19 GB)
>> Used Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
>> Data Offset : 2048 sectors
>> Super Offset : 8 sectors
>> Recovery Offset : 86403072 sectors
>> Unused Space : before=1960 sectors, after=1953244160 sectors
>> State : active
>> Device UUID : 782873ea:e265ecd4:5cc80ddf:035ba2b4
>>
>> Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
>> Delta Devices : 1 (3->4)
>>
>> Update Time : Sun Mar 29 00:05:29 2015
>> Bad Block Log : 512 entries available at offset 72 sectors
>> Checksum : 710dc078 - correct
>> Events : 673
>>
>> Chunk Size : 512K
>>
>> Device Role : Active device 2
>> Array State : AAA. ('A' == active, '.' == missing, 'R' == replacing)
>>
>> # mdadm --detail /dev/md124
>> /dev/md124:
>> Version : 1.2
>> Raid Level : raid0
>> Total Devices : 3
>> Persistence : Superblock is persistent
>>
>> State : inactive
>>
>> Delta Devices : 1, (-1->0)
>> New Level : raid4
>> New Chunksize : 512K
>>
>> Name : hordern:hordern1 (local to host hordern)
>> UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
>> Events : 673
>>
>> Number Major Minor RaidDevice
>>
>> - 9 125 - /dev/md/gamma
>> - 9 126 - /dev/md/beta
>> - 9 127 - /dev/md/alpha
>>
>> So it looks like all three component devices have consistent
>> superblocks now, awesome! But the raid0 array is still inactive with
>> all three components listed as spares. It looks like /dev/md/gamma has
>> a much lower event count, I'm guessing that is what causes the disk to
>> be marked as possibly out of date.
>>
>> Is an "uptodate device" a specific thing, or does that simply mean
>> that some component devices are out of date? The lack of spaces makes
>> me think that uptodate is some keyword I'm not recognizing.
>>
>
> Looks good. Nearly there.
>
> The difference in event counts is probably due to you trying lots of things
> out, and them only affecting two devices.
>
> If you
> # mdadm --stop /dev/md124
> # mdadm -A --force /dev/md124 -vvv /dev/md/alpha /dev/md/beta /dev/md/gamma
>
> i.e. just add --force, it should ignored the difference in event count and
> assemble the array.
> For RAID0, the event count isn't really relevant to the data as there is no
> possibility for inconsistency between data and parity on different devices.
> As the reshape position is the same on all devices, I don't think there is
> any risk at all in just using --force.
> Of course, perform an fsck afterwards just to build confidence.
>
> NeilBrown
>
Unfortunately, adding --force didn't seem to make any difference:
# mdadm --stop /dev/md124
mdadm: stopped /dev/md124
# mdadm -A --force /dev/md124 -vvv /dev/md/alpha /dev/md/beta /dev/md/gamma
mdadm: looking for devices for /dev/md124
mdadm: UUID differs from /dev/md0.
mdadm: UUID differs from /dev/md/alpha.
mdadm: UUID differs from /dev/md/beta.
mdadm: UUID differs from /dev/md/gamma.
mdadm: UUID differs from /dev/md0.
mdadm: UUID differs from /dev/md/alpha.
mdadm: UUID differs from /dev/md/beta.
mdadm: UUID differs from /dev/md/gamma.
mdadm: UUID differs from /dev/md0.
mdadm: UUID differs from /dev/md/alpha.
mdadm: UUID differs from /dev/md/beta.
mdadm: UUID differs from /dev/md/gamma.
mdadm: /dev/md/alpha is identified as a member of /dev/md124, slot 1.
mdadm: /dev/md/beta is identified as a member of /dev/md124, slot 0.
mdadm: /dev/md/gamma is identified as a member of /dev/md124, slot 2.
mdadm: :/dev/md124 has an active reshape - checking if critical
section needs to be restored
mdadm: added /dev/md/alpha to /dev/md124 as 1
mdadm: added /dev/md/gamma to /dev/md124 as 2 (possibly out of date)
mdadm: no uptodate device for slot 6 of /dev/md124
mdadm: added /dev/md/beta to /dev/md124 as 0
mdadm: /dev/md124 assembled from 2 drives - not enough to start the array.
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1] [raid10] [raid0]
[linear] [multipath]
md124 : inactive md125[3](S) md127[1](S) md126[0](S)
6837155192 blocks super 1.2
md0 : active raid1 sda5[0] sdb2[1]
107652416 blocks [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk
md125 : active raid1 sdh1[0] sdg1[1]
2930134016 blocks super 1.2 [2/2] [UU]
bitmap: 0/22 pages [0KB], 65536KB chunk
md126 : active raid1 sdc1[0] sdd1[1]
1953512312 blocks super 1.2 [2/2] [UU]
md127 : active raid1 sde1[2] sdf1[1]
1953512312 blocks super 1.2 [2/2] [UU]
unused devices: <none>
^ permalink raw reply
* Re: Creating volume in IMSM container
From: NeilBrown @ 2015-04-07 21:28 UTC (permalink / raw)
To: Ian Pilcher; +Cc: linux-raid
In-Reply-To: <mg0t3u$ol$1@ger.gmane.org>
[-- Attachment #1: Type: text/plain, Size: 1659 bytes --]
On Tue, 07 Apr 2015 10:29:02 -0500 Ian Pilcher <arequipeno@gmail.com> wrote:
> I am trying to create an IMSM container and volume in a KVM virtual
> machine (so that I can create a reproducer for a Fedora problem that
> seems to only/mostly affect IMSM RAID). I've managed to create a
> container with the following command:
>
> IMSM_NO_PLATFORM=1 mdadm --create --verbose /dev/md/imsm \
> /dev/vd{a,b} --raid-devices 2 --metadata=imsm
>
> /proc/mdstat now contains:
>
> Personalities: [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] ...
> md127 : inactive vdb[1](S) vda[0](S)
> 2210 blocks super external:imsm
2210 blocks is tiny.
I think each device needs at least 6306 sectors for IMSM.
I suggest going for at least 20,000.
>
> I haven't, however, been able to create a volume within the container:
>
> # IMSM_NO_PLATFORM=1 mdadm --create --verbose /dev/md/vol0 \
> /dev/md/imsm --raid-devices 2 --level 1 --metadata=imsm
> mdmon: not enough devices with space to create array.
This should work once you have devices with enough space.
>
> # IMSM_NO_PLATFORM=1 mdadm --create --verbose /dev/md/vol0 \
> /dev/vd{a,b} --raid-devices 2 --level 1 --metadata=imsm
> mdadm: /dev/vda appears to be part of a raid array:
> level=container devices=0 ctime=Thu Jan 1 00:00:00 1970
> mdmon: /dev/vdb is not in the same imsm set
> mdadm: /dev/vdb is not suitable for this array.
> mdadm: create aborted
>
This looks like a bug. What version of mdadm.
NeilBrown
> Am I doing something wrong, or is this simply not possible?
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Help recovering an interrupted raid0 reshape
From: NeilBrown @ 2015-04-07 21:13 UTC (permalink / raw)
To: Jonathan Harker (Jesusaurus); +Cc: linux-raid
In-Reply-To: <CAC_83AEtgtiN-WD-U8OG_nzsoRWQmrn6mxKMhKk6cxK4ZD5yaQ@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 8564 bytes --]
On Tue, 7 Apr 2015 10:02:13 -0700 "Jonathan Harker (Jesusaurus)"
<jesusaurus@gentlydownthe.net> wrote:
> On Mon, Apr 6, 2015 at 11:30 PM, NeilBrown <neilb@suse.de> wrote:
> >
> > Try:
> > mdadm -S /dev/md124
> > mdadm -A /dev/md124 --update=revert-reshape /dev/md/alpha /dev/md/beta
> > mdadm -S /dev/md124
> > mdadm -A /dev/md124 -vvv /dev/md/alpha /dev/md/beta /dev/md/gamma
> >
> > What does that report?
> >
> > NeilBrown
> >
>
> # mdadm --stop /dev/md124
> mdadm: stopped /dev/md124
> # mdadm -A /dev/md124 --update=revert-reshape /dev/md/alpha /dev/md/beta
> mdadm: /dev/md124 assembled from 2 drives - not enough to start the array.
> # cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4] [raid1] [raid10] [raid0]
> [linear] [multipath]
> md124 : inactive md126[0](S) md127[1](S)
> 3907022200 blocks super 1.2
>
> md0 : active raid1 sda5[0] sdb2[1]
> 107652416 blocks [2/2] [UU]
> bitmap: 1/1 pages [4KB], 65536KB chunk
>
> md125 : active raid1 sdh1[0] sdg1[1]
> 2930134016 blocks super 1.2 [2/2] [UU]
> bitmap: 0/22 pages [0KB], 65536KB chunk
>
> md126 : active raid1 sdc1[0] sdd1[1]
> 1953512312 blocks super 1.2 [2/2] [UU]
>
> md127 : active raid1 sde1[2] sdf1[1]
> 1953512312 blocks super 1.2 [2/2] [UU]
>
> unused devices: <none>
> # mdadm --stop /dev/md124
> mdadm: stopped /dev/md124
> # mdadm -A /dev/md124 -vvv /dev/md/alpha /dev/md/beta /dev/md/gamma
> mdadm: looking for devices for /dev/md124
> mdadm: UUID differs from /dev/md0.
> mdadm: UUID differs from /dev/md/alpha.
> mdadm: UUID differs from /dev/md/beta.
> mdadm: UUID differs from /dev/md/gamma.
> mdadm: UUID differs from /dev/md0.
> mdadm: UUID differs from /dev/md/alpha.
> mdadm: UUID differs from /dev/md/beta.
> mdadm: UUID differs from /dev/md/gamma.
> mdadm: UUID differs from /dev/md0.
> mdadm: UUID differs from /dev/md/alpha.
> mdadm: UUID differs from /dev/md/beta.
> mdadm: UUID differs from /dev/md/gamma.
> mdadm: /dev/md/alpha is identified as a member of /dev/md124, slot 1.
> mdadm: /dev/md/beta is identified as a member of /dev/md124, slot 0.
> mdadm: /dev/md/gamma is identified as a member of /dev/md124, slot 2.
> mdadm: :/dev/md124 has an active reshape - checking if critical
> section needs to be restored
> mdadm: added /dev/md/alpha to /dev/md124 as 1
> mdadm: added /dev/md/gamma to /dev/md124 as 2 (possibly out of date)
> mdadm: no uptodate device for slot 6 of /dev/md124
> mdadm: added /dev/md/beta to /dev/md124 as 0
> mdadm: /dev/md124 assembled from 2 drives - not enough to start the array.
> # cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4] [raid1] [raid10] [raid0]
> [linear] [multipath]
> md124 : inactive md125[3](S) md127[1](S) md126[0](S)
> 6837155192 blocks super 1.2
>
> md0 : active raid1 sda5[0] sdb2[1]
> 107652416 blocks [2/2] [UU]
> bitmap: 0/1 pages [0KB], 65536KB chunk
>
> md125 : active raid1 sdh1[0] sdg1[1]
> 2930134016 blocks super 1.2 [2/2] [UU]
> bitmap: 0/22 pages [0KB], 65536KB chunk
>
> md126 : active raid1 sdc1[0] sdd1[1]
> 1953512312 blocks super 1.2 [2/2] [UU]
>
> md127 : active raid1 sde1[2] sdf1[1]
> 1953512312 blocks super 1.2 [2/2] [UU]
>
> unused devices: <none>
>
> # mdadm --examine /dev/md/alpha
> /dev/md/alpha:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x4
> Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
> Name : hordern:hordern1 (local to host hordern)
> Creation Time : Fri Jan 2 09:59:40 2009
> Raid Level : raid4
> Raid Devices : 4
>
> Avail Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
> Array Size : 5860532736 (5589.04 GiB 6001.19 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> Unused Space : before=1968 sectors, after=752 sectors
> State : active
> Device UUID : 63aaa2e4:2a09f495:8372c7f9:eb2f2773
>
> Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
> Delta Devices : 1 (3->4)
>
> Update Time : Sun Mar 29 15:11:35 2015
> Checksum : 8be5e0e8 - correct
> Events : 14013
>
> Chunk Size : 512K
>
> Device Role : Active device 1
> Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
>
> # mdadm --examine /dev/md/beta
> /dev/md/beta:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x4
> Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
> Name : hordern:hordern1 (local to host hordern)
> Creation Time : Fri Jan 2 09:59:40 2009
> Raid Level : raid4
> Raid Devices : 4
>
> Avail Dev Size : 3907022576 (1863.01 GiB 2000.40 GB)
> Array Size : 5860532736 (5589.04 GiB 6001.19 GB)
> Used Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> Unused Space : before=1968 sectors, after=752 sectors
> State : clean
> Device UUID : 6e6dce14:3ebb2bb5:187aa292:403a55f6
>
> Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
> Delta Devices : 1 (3->4)
>
> Update Time : Sun Mar 29 15:11:35 2015
> Checksum : f7526adf - correct
> Events : 14013
>
> Chunk Size : 512K
>
> Device Role : Active device 0
> Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
>
> # mdadm --examine /dev/md/gamma
> /dev/md/gamma:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x6
> Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
> Name : hordern:hordern1 (local to host hordern)
> Creation Time : Fri Jan 2 09:59:40 2009
> Raid Level : raid4
> Raid Devices : 4
>
> Avail Dev Size : 5860265984 (2794.39 GiB 3000.46 GB)
> Array Size : 5860532736 (5589.04 GiB 6001.19 GB)
> Used Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> Recovery Offset : 86403072 sectors
> Unused Space : before=1960 sectors, after=1953244160 sectors
> State : active
> Device UUID : 782873ea:e265ecd4:5cc80ddf:035ba2b4
>
> Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
> Delta Devices : 1 (3->4)
>
> Update Time : Sun Mar 29 00:05:29 2015
> Bad Block Log : 512 entries available at offset 72 sectors
> Checksum : 710dc078 - correct
> Events : 673
>
> Chunk Size : 512K
>
> Device Role : Active device 2
> Array State : AAA. ('A' == active, '.' == missing, 'R' == replacing)
>
> # mdadm --detail /dev/md124
> /dev/md124:
> Version : 1.2
> Raid Level : raid0
> Total Devices : 3
> Persistence : Superblock is persistent
>
> State : inactive
>
> Delta Devices : 1, (-1->0)
> New Level : raid4
> New Chunksize : 512K
>
> Name : hordern:hordern1 (local to host hordern)
> UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
> Events : 673
>
> Number Major Minor RaidDevice
>
> - 9 125 - /dev/md/gamma
> - 9 126 - /dev/md/beta
> - 9 127 - /dev/md/alpha
>
> So it looks like all three component devices have consistent
> superblocks now, awesome! But the raid0 array is still inactive with
> all three components listed as spares. It looks like /dev/md/gamma has
> a much lower event count, I'm guessing that is what causes the disk to
> be marked as possibly out of date.
>
> Is an "uptodate device" a specific thing, or does that simply mean
> that some component devices are out of date? The lack of spaces makes
> me think that uptodate is some keyword I'm not recognizing.
>
Looks good. Nearly there.
The difference in event counts is probably due to you trying lots of things
out, and them only affecting two devices.
If you
# mdadm --stop /dev/md124
# mdadm -A --force /dev/md124 -vvv /dev/md/alpha /dev/md/beta /dev/md/gamma
i.e. just add --force, it should ignored the difference in event count and
assemble the array.
For RAID0, the event count isn't really relevant to the data as there is no
possibility for inconsistency between data and parity on different devices.
As the reshape position is the same on all devices, I don't think there is
any risk at all in just using --force.
Of course, perform an fsck afterwards just to build confidence.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* [PATCH 2/2][v2] blk-plug: don't flush nested plug lists
From: Jeff Moyer @ 2015-04-07 18:55 UTC (permalink / raw)
To: Jens Axboe, Ming Lei
Cc: Konrad Rzeszutek Wilk, Roger Pau Monn??, Alasdair Kergon,
Mike Snitzer, Neil Brown, Nicholas A. Bellinger, Alexander Viro,
Chris Mason, Josef Bacik, David Sterba, Theodore Ts'o,
Andreas Dilger, Jaegeuk Kim, Changman Lee, Steven Whitehouse,
Mikulas Patocka, Andrew Morton, Rik van Riel, Johannes Weiner,
Mel Gorman, Trond Myklebust <trond.myk>
In-Reply-To: <1428347694-17704-2-git-send-email-jmoyer@redhat.com>
The way the on-stack plugging currently works, each nesting level
flushes its own list of I/Os. This can be less than optimal (read
awful) for certain workloads. For example, consider an application
that issues asynchronous O_DIRECT I/Os. It can send down a bunch of
I/Os together in a single io_submit call, only to have each of them
dispatched individually down in the bowels of the dirct I/O code.
The reason is that there are blk_plug-s instantiated both at the upper
call site in do_io_submit and down in do_direct_IO. The latter will
submit as little as 1 I/O at a time (if you have a small enough I/O
size) instead of performing the batching that the plugging
infrastructure is supposed to provide.
Now, for the case where there is an elevator involved, this doesn't
really matter too much. The elevator will keep the I/O around long
enough for it to be merged. However, in cases where there is no
elevator (like blk-mq), I/Os are simply dispatched immediately.
Try this, for example:
fio --rw=read --bs=4k --iodepth=128 --iodepth_batch=16 --iodepth_batch_complete=16 --runtime=10s --direct=1 --filename=/dev/vdd --name=job1 --ioengine=libaio --time_based
If you run that on a current kernel, you will get zero merges. Zero!
After this patch, you will get many merges (the actual number depends
on how fast your storage is, obviously), and much better throughput.
Here are results from my test systems:
First, I tested in a VM using a virtio-blk device:
Unpatched kernel:
Throughput: 280,262 KB/s
avg latency: 14,587.72 usec
Patched kernel:
throughput: 832,158 KB/s
avg latency: 4,901.95 usec
Next, I tesetd using a micron p320h on bare metal:
Unpatched kernel:
Throughput: 688,967 KB/s
avg latency: 5,933.92 usec
Patched kernel:
Throughput: 1,160.6 MB/s
avg latency: 3,437.01 usec
As you can see, both throughput and latency improved dramatically.
I've included the full fio output below, so you can also see the
marked improvement in standard deviation as well.
I considered several approaches to solving the problem:
1) get rid of the inner-most plugs
2) handle nesting by using only one on-stack plug
2a) #2, except use a per-cpu blk_plug struct, which may clean up the
code a bit at the expense of memory footprint
Option 1 will be tricky or impossible to do, since inner most plug
lists are sometimes the only plug lists, depending on the call path.
Option 2 is what this patch implements. Option 2a may add unneeded
complexity.
Much of the patch involves modifying call sites to blk_finish_plug,
since its signature is changed. The meat of the patch is actually
pretty simple and constrained to block/blk-core.c and
include/linux/blkdev.h. The only tricky bits were places where plugs
were finished and then restarted to flush out I/O. There, I left things
as-is. So long as they are the outer-most plugs, they should continue
to function as before.
NOTE TO SUBSYSTEM MAINTAINERS: Before this patch, blk_finish_plug
would always flush the plug list. After this patch, this is only the
case for the outer-most plug. If you require the plug list to be
flushed, you should be calling blk_flush_plug(current). Btrfs and dm
maintainers should take a close look at this patch and ensure they get
the right behavior in the end.
Signed-off-by: Jeff Moyer <jmoyer@redhat.com>
---
Changelog:
v1->v2: Keep the blk_start_plug interface the same, suggested by Ming Lei.
Test results
------------
Virtio-blk:
unpatched:
job1: (groupid=0, jobs=1): err= 0: pid=8032: Tue Apr 7 13:33:53 2015
read : io=2736.1MB, bw=280262KB/s, iops=70065, runt= 10000msec
slat (usec): min=40, max=10472, avg=207.82, stdev=364.02
clat (usec): min=211, max=35883, avg=14379.83, stdev=2213.95
lat (usec): min=862, max=36000, avg=14587.72, stdev=2223.80
clat percentiles (usec):
| 1.00th=[11328], 5.00th=[12096], 10.00th=[12480], 20.00th=[12992],
| 30.00th=[13376], 40.00th=[13760], 50.00th=[14144], 60.00th=[14400],
| 70.00th=[14784], 80.00th=[15168], 90.00th=[15936], 95.00th=[16768],
| 99.00th=[24448], 99.50th=[25216], 99.90th=[28544], 99.95th=[35072],
| 99.99th=[36096]
bw (KB /s): min=265984, max=302720, per=100.00%, avg=280549.84, stdev=10264.36
lat (usec) : 250=0.01%, 1000=0.01%
lat (msec) : 2=0.02%, 4=0.02%, 10=0.05%, 20=96.57%, 50=3.34%
cpu : usr=7.56%, sys=55.57%, ctx=6174, majf=0, minf=523
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued : total=r=700656/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1024
Run status group 0 (all jobs):
READ: io=2736.1MB, aggrb=280262KB/s, minb=280262KB/s, maxb=280262KB/s, mint=10000msec, maxt=10000msec
Disk stats (read/write):
vdd: ios=695490/0, merge=0/0, ticks=785741/0, in_queue=785442, util=90.69%
patched:
job1: (groupid=0, jobs=1): err= 0: pid=7743: Tue Apr 7 13:19:07 2015
read : io=8126.6MB, bw=832158KB/s, iops=208039, runt= 10000msec
slat (usec): min=20, max=14351, avg=55.08, stdev=143.47
clat (usec): min=283, max=20003, avg=4846.77, stdev=1355.35
lat (usec): min=609, max=20074, avg=4901.95, stdev=1362.40
clat percentiles (usec):
| 1.00th=[ 4016], 5.00th=[ 4048], 10.00th=[ 4080], 20.00th=[ 4128],
| 30.00th=[ 4192], 40.00th=[ 4192], 50.00th=[ 4256], 60.00th=[ 4512],
| 70.00th=[ 4896], 80.00th=[ 5664], 90.00th=[ 5920], 95.00th=[ 6752],
| 99.00th=[11968], 99.50th=[13632], 99.90th=[15552], 99.95th=[17024],
| 99.99th=[19840]
bw (KB /s): min=740992, max=896640, per=100.00%, avg=836978.95, stdev=51034.87
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 4=0.50%, 10=97.79%, 20=1.70%, 50=0.01%
cpu : usr=20.28%, sys=69.11%, ctx=879, majf=0, minf=522
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued : total=r=2080396/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1024
Run status group 0 (all jobs):
READ: io=8126.6MB, aggrb=832158KB/s, minb=832158KB/s, maxb=832158KB/s, mint=10000msec, maxt=10000msec
Disk stats (read/write):
vdd: ios=127877/0, merge=1918166/0, ticks=23118/0, in_queue=23047, util=94.08%
micron p320h:
unpatched:
job1: (groupid=0, jobs=1): err= 0: pid=3244: Tue Apr 7 13:29:14 2015
read : io=6728.9MB, bw=688968KB/s, iops=172241, runt= 10001msec
slat (usec): min=43, max=6273, avg=81.79, stdev=125.96
clat (usec): min=78, max=12485, avg=5852.06, stdev=1154.76
lat (usec): min=146, max=12572, avg=5933.92, stdev=1163.75
clat percentiles (usec):
| 1.00th=[ 4192], 5.00th=[ 4384], 10.00th=[ 4576], 20.00th=[ 5600],
| 30.00th=[ 5664], 40.00th=[ 5728], 50.00th=[ 5792], 60.00th=[ 5856],
| 70.00th=[ 6112], 80.00th=[ 6176], 90.00th=[ 6240], 95.00th=[ 6368],
| 99.00th=[11840], 99.50th=[11968], 99.90th=[12096], 99.95th=[12096],
| 99.99th=[12224]
bw (KB /s): min=648328, max=859264, per=98.80%, avg=680711.16, stdev=62016.70
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.04%, 10=97.07%, 20=2.87%
cpu : usr=10.28%, sys=73.61%, ctx=104436, majf=0, minf=6217
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued : total=r=1722592/w=0/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1024
Run status group 0 (all jobs):
READ: io=6728.9MB, aggrb=688967KB/s, minb=688967KB/s, maxb=688967KB/s, mint=10001msec, maxt=10001msec
Disk stats (read/write):
rssda: ios=1688772/0, merge=0/0, ticks=188820/0, in_queue=188678, util=96.61%
patched:
job1: (groupid=0, jobs=1): err= 0: pid=9531: Tue Apr 7 13:22:28 2015
read : io=11607MB, bw=1160.6MB/s, iops=297104, runt= 10001msec
slat (usec): min=21, max=6376, avg=43.05, stdev=81.82
clat (usec): min=116, max=9844, avg=3393.90, stdev=752.57
lat (usec): min=167, max=9889, avg=3437.01, stdev=757.02
clat percentiles (usec):
| 1.00th=[ 2832], 5.00th=[ 2992], 10.00th=[ 3056], 20.00th=[ 3120],
| 30.00th=[ 3152], 40.00th=[ 3248], 50.00th=[ 3280], 60.00th=[ 3344],
| 70.00th=[ 3376], 80.00th=[ 3504], 90.00th=[ 3728], 95.00th=[ 3824],
| 99.00th=[ 9152], 99.50th=[ 9408], 99.90th=[ 9664], 99.95th=[ 9664],
| 99.99th=[ 9792]
bw (MB /s): min= 1139, max= 1183, per=100.00%, avg=1161.07, stdev=10.58
lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=98.31%, 10=1.67%
cpu : usr=18.59%, sys=66.65%, ctx=55655, majf=0, minf=6218
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued : total=r=2971338/w=0/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1024
Run status group 0 (all jobs):
READ: io=11607MB, aggrb=1160.6MB/s, minb=1160.6MB/s, maxb=1160.6MB/s, mint=10001msec, maxt=10001msec
Disk stats (read/write):
rssda: ios=183005/0, merge=2745105/0, ticks=31972/0, in_queue=31948, util=97.63%
---
block/blk-core.c | 29 ++++++++++++++++-------------
block/blk-lib.c | 2 +-
block/blk-throttle.c | 2 +-
drivers/block/xen-blkback/blkback.c | 2 +-
drivers/md/dm-bufio.c | 6 +++---
drivers/md/dm-crypt.c | 2 +-
drivers/md/dm-kcopyd.c | 2 +-
drivers/md/dm-thin.c | 2 +-
drivers/md/md.c | 2 +-
drivers/md/raid1.c | 2 +-
drivers/md/raid10.c | 2 +-
drivers/md/raid5.c | 4 ++--
drivers/target/target_core_iblock.c | 2 +-
fs/aio.c | 2 +-
fs/block_dev.c | 2 +-
fs/btrfs/scrub.c | 2 +-
fs/btrfs/transaction.c | 2 +-
fs/btrfs/tree-log.c | 12 ++++++------
fs/btrfs/volumes.c | 6 +++---
fs/buffer.c | 2 +-
fs/direct-io.c | 2 +-
fs/ext4/file.c | 2 +-
fs/ext4/inode.c | 4 ++--
fs/f2fs/checkpoint.c | 2 +-
fs/f2fs/gc.c | 2 +-
fs/f2fs/node.c | 2 +-
fs/gfs2/log.c | 2 +-
fs/hpfs/buffer.c | 2 +-
fs/jbd/checkpoint.c | 2 +-
fs/jbd/commit.c | 4 ++--
fs/jbd2/checkpoint.c | 2 +-
fs/jbd2/commit.c | 2 +-
fs/mpage.c | 2 +-
fs/nfs/blocklayout/blocklayout.c | 4 ++--
fs/xfs/xfs_buf.c | 4 ++--
fs/xfs/xfs_dir2_readdir.c | 2 +-
fs/xfs/xfs_itable.c | 2 +-
include/linux/blkdev.h | 5 +++--
mm/madvise.c | 2 +-
mm/page-writeback.c | 2 +-
mm/readahead.c | 2 +-
mm/swap_state.c | 2 +-
mm/vmscan.c | 2 +-
43 files changed, 74 insertions(+), 70 deletions(-)
diff --git a/block/blk-core.c b/block/blk-core.c
index 794c3e7..fcd9c2f 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -3018,21 +3018,21 @@ void blk_start_plug(struct blk_plug *plug)
{
struct task_struct *tsk = current;
+ if (tsk->plug) {
+ tsk->plug->depth++;
+ return;
+ }
+
+ plug->depth = 1;
INIT_LIST_HEAD(&plug->list);
INIT_LIST_HEAD(&plug->mq_list);
INIT_LIST_HEAD(&plug->cb_list);
/*
- * If this is a nested plug, don't actually assign it. It will be
- * flushed on its own.
+ * Store ordering should not be needed here, since a potential
+ * preempt will imply a full memory barrier
*/
- if (!tsk->plug) {
- /*
- * Store ordering should not be needed here, since a potential
- * preempt will imply a full memory barrier
- */
- tsk->plug = plug;
- }
+ tsk->plug = plug;
}
EXPORT_SYMBOL(blk_start_plug);
@@ -3177,12 +3177,15 @@ void blk_flush_plug_list(struct blk_plug *plug, bool from_schedule)
local_irq_restore(flags);
}
-void blk_finish_plug(struct blk_plug *plug)
+void blk_finish_plug(void)
{
- blk_flush_plug_list(plug, false);
+ struct blk_plug *plug = current->plug;
- if (plug == current->plug)
- current->plug = NULL;
+ if (--plug->depth > 0)
+ return;
+
+ blk_flush_plug_list(plug, false);
+ current->plug = NULL;
}
EXPORT_SYMBOL(blk_finish_plug);
diff --git a/block/blk-lib.c b/block/blk-lib.c
index 7688ee3..ac347d3 100644
--- a/block/blk-lib.c
+++ b/block/blk-lib.c
@@ -128,7 +128,7 @@ int blkdev_issue_discard(struct block_device *bdev, sector_t sector,
*/
cond_resched();
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
/* Wait for bios in-flight */
if (!atomic_dec_and_test(&bb.done))
diff --git a/block/blk-throttle.c b/block/blk-throttle.c
index 5b9c6d5..222a77a 100644
--- a/block/blk-throttle.c
+++ b/block/blk-throttle.c
@@ -1281,7 +1281,7 @@ static void blk_throtl_dispatch_work_fn(struct work_struct *work)
blk_start_plug(&plug);
while((bio = bio_list_pop(&bio_list_on_stack)))
generic_make_request(bio);
- blk_finish_plug(&plug);
+ blk_finish_plug();
}
}
diff --git a/drivers/block/xen-blkback/blkback.c b/drivers/block/xen-blkback/blkback.c
index 2a04d34..74bea21 100644
--- a/drivers/block/xen-blkback/blkback.c
+++ b/drivers/block/xen-blkback/blkback.c
@@ -1374,7 +1374,7 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif,
submit_bio(operation, biolist[i]);
/* Let the I/Os go.. */
- blk_finish_plug(&plug);
+ blk_finish_plug();
if (operation == READ)
blkif->st_rd_sect += preq.nr_sects;
diff --git a/drivers/md/dm-bufio.c b/drivers/md/dm-bufio.c
index 86dbbc7..502c63b 100644
--- a/drivers/md/dm-bufio.c
+++ b/drivers/md/dm-bufio.c
@@ -715,7 +715,7 @@ static void __flush_write_list(struct list_head *write_list)
submit_io(b, WRITE, b->block, write_endio);
dm_bufio_cond_resched();
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
}
/*
@@ -1126,7 +1126,7 @@ void dm_bufio_prefetch(struct dm_bufio_client *c,
&write_list);
if (unlikely(!list_empty(&write_list))) {
dm_bufio_unlock(c);
- blk_finish_plug(&plug);
+ blk_finish_plug();
__flush_write_list(&write_list);
blk_start_plug(&plug);
dm_bufio_lock(c);
@@ -1149,7 +1149,7 @@ void dm_bufio_prefetch(struct dm_bufio_client *c,
dm_bufio_unlock(c);
flush_plug:
- blk_finish_plug(&plug);
+ blk_finish_plug();
}
EXPORT_SYMBOL_GPL(dm_bufio_prefetch);
diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c
index 713a962..65d7b72 100644
--- a/drivers/md/dm-crypt.c
+++ b/drivers/md/dm-crypt.c
@@ -1224,7 +1224,7 @@ pop_from_list:
rb_erase(&io->rb_node, &write_tree);
kcryptd_io_write(io);
} while (!RB_EMPTY_ROOT(&write_tree));
- blk_finish_plug(&plug);
+ blk_finish_plug();
}
return 0;
}
diff --git a/drivers/md/dm-kcopyd.c b/drivers/md/dm-kcopyd.c
index 3a7cade..4a76e42 100644
--- a/drivers/md/dm-kcopyd.c
+++ b/drivers/md/dm-kcopyd.c
@@ -593,7 +593,7 @@ static void do_work(struct work_struct *work)
process_jobs(&kc->complete_jobs, kc, run_complete_job);
process_jobs(&kc->pages_jobs, kc, run_pages_job);
process_jobs(&kc->io_jobs, kc, run_io_job);
- blk_finish_plug(&plug);
+ blk_finish_plug();
}
/*
diff --git a/drivers/md/dm-thin.c b/drivers/md/dm-thin.c
index 921aafd..be42bf5 100644
--- a/drivers/md/dm-thin.c
+++ b/drivers/md/dm-thin.c
@@ -1824,7 +1824,7 @@ static void process_thin_deferred_bios(struct thin_c *tc)
dm_pool_issue_prefetches(pool->pmd);
}
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
}
static int cmp_cells(const void *lhs, const void *rhs)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 717daad..c4ec179 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -7686,7 +7686,7 @@ void md_do_sync(struct md_thread *thread)
/*
* this also signals 'finished resyncing' to md_stop
*/
- blk_finish_plug(&plug);
+ blk_finish_plug();
wait_event(mddev->recovery_wait, !atomic_read(&mddev->recovery_active));
/* tell personality that we are finished */
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index d34e238..4f8fad4 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -2441,7 +2441,7 @@ static void raid1d(struct md_thread *thread)
if (mddev->flags & ~(1<<MD_CHANGE_PENDING))
md_check_recovery(mddev);
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
}
static int init_resync(struct r1conf *conf)
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index a7196c4..92bb5dd 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -2835,7 +2835,7 @@ static void raid10d(struct md_thread *thread)
if (mddev->flags & ~(1<<MD_CHANGE_PENDING))
md_check_recovery(mddev);
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
}
static int init_resync(struct r10conf *conf)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index cd2f96b..695bf0f 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -5281,7 +5281,7 @@ static void raid5_do_work(struct work_struct *work)
pr_debug("%d stripes handled\n", handled);
spin_unlock_irq(&conf->device_lock);
- blk_finish_plug(&plug);
+ blk_finish_plug();
pr_debug("--- raid5worker inactive\n");
}
@@ -5352,7 +5352,7 @@ static void raid5d(struct md_thread *thread)
spin_unlock_irq(&conf->device_lock);
async_tx_issue_pending_all();
- blk_finish_plug(&plug);
+ blk_finish_plug();
pr_debug("--- raid5d inactive\n");
}
diff --git a/drivers/target/target_core_iblock.c b/drivers/target/target_core_iblock.c
index d4a4b0f..17d8730 100644
--- a/drivers/target/target_core_iblock.c
+++ b/drivers/target/target_core_iblock.c
@@ -367,7 +367,7 @@ static void iblock_submit_bios(struct bio_list *list, int rw)
blk_start_plug(&plug);
while ((bio = bio_list_pop(list)))
submit_bio(rw, bio);
- blk_finish_plug(&plug);
+ blk_finish_plug();
}
static void iblock_end_io_flush(struct bio *bio, int err)
diff --git a/fs/aio.c b/fs/aio.c
index f8e52a1..b873698 100644
--- a/fs/aio.c
+++ b/fs/aio.c
@@ -1616,7 +1616,7 @@ long do_io_submit(aio_context_t ctx_id, long nr,
if (ret)
break;
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
percpu_ref_put(&ctx->users);
return i ? i : ret;
diff --git a/fs/block_dev.c b/fs/block_dev.c
index 975266b..f5848de 100644
--- a/fs/block_dev.c
+++ b/fs/block_dev.c
@@ -1609,7 +1609,7 @@ ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from)
if (err < 0)
ret = err;
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
return ret;
}
EXPORT_SYMBOL_GPL(blkdev_write_iter);
diff --git a/fs/btrfs/scrub.c b/fs/btrfs/scrub.c
index ec57687..f314cfb8 100644
--- a/fs/btrfs/scrub.c
+++ b/fs/btrfs/scrub.c
@@ -3316,7 +3316,7 @@ out:
scrub_wr_submit(sctx);
mutex_unlock(&sctx->wr_ctx.wr_lock);
- blk_finish_plug(&plug);
+ blk_finish_plug();
btrfs_free_path(path);
btrfs_free_path(ppath);
return ret < 0 ? ret : 0;
diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c
index 8be4278..fee10af 100644
--- a/fs/btrfs/transaction.c
+++ b/fs/btrfs/transaction.c
@@ -983,7 +983,7 @@ static int btrfs_write_and_wait_marked_extents(struct btrfs_root *root,
blk_start_plug(&plug);
ret = btrfs_write_marked_extents(root, dirty_pages, mark);
- blk_finish_plug(&plug);
+ blk_finish_plug();
ret2 = btrfs_wait_marked_extents(root, dirty_pages, mark);
if (ret)
diff --git a/fs/btrfs/tree-log.c b/fs/btrfs/tree-log.c
index c5b8ba3..879c7fd 100644
--- a/fs/btrfs/tree-log.c
+++ b/fs/btrfs/tree-log.c
@@ -2574,7 +2574,7 @@ int btrfs_sync_log(struct btrfs_trans_handle *trans,
blk_start_plug(&plug);
ret = btrfs_write_marked_extents(log, &log->dirty_log_pages, mark);
if (ret) {
- blk_finish_plug(&plug);
+ blk_finish_plug();
btrfs_abort_transaction(trans, root, ret);
btrfs_free_logged_extents(log, log_transid);
btrfs_set_log_full_commit(root->fs_info, trans);
@@ -2619,7 +2619,7 @@ int btrfs_sync_log(struct btrfs_trans_handle *trans,
if (!list_empty(&root_log_ctx.list))
list_del_init(&root_log_ctx.list);
- blk_finish_plug(&plug);
+ blk_finish_plug();
btrfs_set_log_full_commit(root->fs_info, trans);
if (ret != -ENOSPC) {
@@ -2635,7 +2635,7 @@ int btrfs_sync_log(struct btrfs_trans_handle *trans,
}
if (log_root_tree->log_transid_committed >= root_log_ctx.log_transid) {
- blk_finish_plug(&plug);
+ blk_finish_plug();
mutex_unlock(&log_root_tree->log_mutex);
ret = root_log_ctx.log_ret;
goto out;
@@ -2643,7 +2643,7 @@ int btrfs_sync_log(struct btrfs_trans_handle *trans,
index2 = root_log_ctx.log_transid % 2;
if (atomic_read(&log_root_tree->log_commit[index2])) {
- blk_finish_plug(&plug);
+ blk_finish_plug();
ret = btrfs_wait_marked_extents(log, &log->dirty_log_pages,
mark);
btrfs_wait_logged_extents(trans, log, log_transid);
@@ -2669,7 +2669,7 @@ int btrfs_sync_log(struct btrfs_trans_handle *trans,
* check the full commit flag again
*/
if (btrfs_need_log_full_commit(root->fs_info, trans)) {
- blk_finish_plug(&plug);
+ blk_finish_plug();
btrfs_wait_marked_extents(log, &log->dirty_log_pages, mark);
btrfs_free_logged_extents(log, log_transid);
mutex_unlock(&log_root_tree->log_mutex);
@@ -2680,7 +2680,7 @@ int btrfs_sync_log(struct btrfs_trans_handle *trans,
ret = btrfs_write_marked_extents(log_root_tree,
&log_root_tree->dirty_log_pages,
EXTENT_DIRTY | EXTENT_NEW);
- blk_finish_plug(&plug);
+ blk_finish_plug();
if (ret) {
btrfs_set_log_full_commit(root->fs_info, trans);
btrfs_abort_transaction(trans, root, ret);
diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
index 8222f6f..16db068 100644
--- a/fs/btrfs/volumes.c
+++ b/fs/btrfs/volumes.c
@@ -358,7 +358,7 @@ loop_lock:
if (pending_bios == &device->pending_sync_bios) {
sync_pending = 1;
} else if (sync_pending) {
- blk_finish_plug(&plug);
+ blk_finish_plug();
blk_start_plug(&plug);
sync_pending = 0;
}
@@ -415,7 +415,7 @@ loop_lock:
}
/* unplug every 64 requests just for good measure */
if (batch_run % 64 == 0) {
- blk_finish_plug(&plug);
+ blk_finish_plug();
blk_start_plug(&plug);
sync_pending = 0;
}
@@ -431,7 +431,7 @@ loop_lock:
spin_unlock(&device->io_lock);
done:
- blk_finish_plug(&plug);
+ blk_finish_plug();
}
static void pending_bios_fn(struct btrfs_work *work)
diff --git a/fs/buffer.c b/fs/buffer.c
index 20805db..8181c44 100644
--- a/fs/buffer.c
+++ b/fs/buffer.c
@@ -758,7 +758,7 @@ static int fsync_buffers_list(spinlock_t *lock, struct list_head *list)
}
spin_unlock(lock);
- blk_finish_plug(&plug);
+ blk_finish_plug();
spin_lock(lock);
while (!list_empty(&tmp)) {
diff --git a/fs/direct-io.c b/fs/direct-io.c
index e181b6b..16f16ed 100644
--- a/fs/direct-io.c
+++ b/fs/direct-io.c
@@ -1262,7 +1262,7 @@ do_blockdev_direct_IO(int rw, struct kiocb *iocb, struct inode *inode,
if (sdio.bio)
dio_bio_submit(dio, &sdio);
- blk_finish_plug(&plug);
+ blk_finish_plug();
/*
* It is possible that, we return short IO due to end of file.
diff --git a/fs/ext4/file.c b/fs/ext4/file.c
index 33a09da..3a293eb 100644
--- a/fs/ext4/file.c
+++ b/fs/ext4/file.c
@@ -183,7 +183,7 @@ ext4_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
ret = err;
}
if (o_direct)
- blk_finish_plug(&plug);
+ blk_finish_plug();
errout:
if (aio_mutex)
diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index 5cb9a21..90ce0cb 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -2302,7 +2302,7 @@ static int ext4_writepages(struct address_space *mapping,
blk_start_plug(&plug);
ret = write_cache_pages(mapping, wbc, __writepage, mapping);
- blk_finish_plug(&plug);
+ blk_finish_plug();
goto out_writepages;
}
@@ -2438,7 +2438,7 @@ retry:
if (ret)
break;
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
if (!ret && !cycled && wbc->nr_to_write > 0) {
cycled = 1;
mpd.last_page = writeback_index - 1;
diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c
index 7f794b7..86ba453 100644
--- a/fs/f2fs/checkpoint.c
+++ b/fs/f2fs/checkpoint.c
@@ -846,7 +846,7 @@ retry_flush_nodes:
goto retry_flush_nodes;
}
out:
- blk_finish_plug(&plug);
+ blk_finish_plug();
return err;
}
diff --git a/fs/f2fs/gc.c b/fs/f2fs/gc.c
index 76adbc3..abeef77 100644
--- a/fs/f2fs/gc.c
+++ b/fs/f2fs/gc.c
@@ -678,7 +678,7 @@ static void do_garbage_collect(struct f2fs_sb_info *sbi, unsigned int segno,
gc_data_segment(sbi, sum->entries, gc_list, segno, gc_type);
break;
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
stat_inc_seg_count(sbi, GET_SUM_TYPE((&sum->footer)));
stat_inc_call_count(sbi->stat_info);
diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c
index 97bd9d3..c4aa9e2 100644
--- a/fs/f2fs/node.c
+++ b/fs/f2fs/node.c
@@ -1098,7 +1098,7 @@ repeat:
ra_node_page(sbi, nid);
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
lock_page(page);
if (unlikely(page->mapping != NODE_MAPPING(sbi))) {
diff --git a/fs/gfs2/log.c b/fs/gfs2/log.c
index 536e7a6..06f25d17 100644
--- a/fs/gfs2/log.c
+++ b/fs/gfs2/log.c
@@ -159,7 +159,7 @@ restart:
goto restart;
}
spin_unlock(&sdp->sd_ail_lock);
- blk_finish_plug(&plug);
+ blk_finish_plug();
trace_gfs2_ail_flush(sdp, wbc, 0);
}
diff --git a/fs/hpfs/buffer.c b/fs/hpfs/buffer.c
index 8057fe4..138462d 100644
--- a/fs/hpfs/buffer.c
+++ b/fs/hpfs/buffer.c
@@ -35,7 +35,7 @@ void hpfs_prefetch_sectors(struct super_block *s, unsigned secno, int n)
secno++;
n--;
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
}
/* Map a sector into a buffer and return pointers to it and to the buffer. */
diff --git a/fs/jbd/checkpoint.c b/fs/jbd/checkpoint.c
index 08c0304..cd6b09f 100644
--- a/fs/jbd/checkpoint.c
+++ b/fs/jbd/checkpoint.c
@@ -263,7 +263,7 @@ __flush_batch(journal_t *journal, struct buffer_head **bhs, int *batch_count)
blk_start_plug(&plug);
for (i = 0; i < *batch_count; i++)
write_dirty_buffer(bhs[i], WRITE_SYNC);
- blk_finish_plug(&plug);
+ blk_finish_plug();
for (i = 0; i < *batch_count; i++) {
struct buffer_head *bh = bhs[i];
diff --git a/fs/jbd/commit.c b/fs/jbd/commit.c
index bb217dc..e1046c3 100644
--- a/fs/jbd/commit.c
+++ b/fs/jbd/commit.c
@@ -447,7 +447,7 @@ void journal_commit_transaction(journal_t *journal)
blk_start_plug(&plug);
err = journal_submit_data_buffers(journal, commit_transaction,
write_op);
- blk_finish_plug(&plug);
+ blk_finish_plug();
/*
* Wait for all previously submitted IO to complete.
@@ -697,7 +697,7 @@ start_journal_io:
}
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
/* Lo and behold: we have just managed to send a transaction to
the log. Before we can commit it, wait for the IO so far to
diff --git a/fs/jbd2/checkpoint.c b/fs/jbd2/checkpoint.c
index 988b32e..6aa0039 100644
--- a/fs/jbd2/checkpoint.c
+++ b/fs/jbd2/checkpoint.c
@@ -187,7 +187,7 @@ __flush_batch(journal_t *journal, int *batch_count)
blk_start_plug(&plug);
for (i = 0; i < *batch_count; i++)
write_dirty_buffer(journal->j_chkpt_bhs[i], WRITE_SYNC);
- blk_finish_plug(&plug);
+ blk_finish_plug();
for (i = 0; i < *batch_count; i++) {
struct buffer_head *bh = journal->j_chkpt_bhs[i];
diff --git a/fs/jbd2/commit.c b/fs/jbd2/commit.c
index b73e021..8f532c8 100644
--- a/fs/jbd2/commit.c
+++ b/fs/jbd2/commit.c
@@ -805,7 +805,7 @@ start_journal_io:
__jbd2_journal_abort_hard(journal);
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
/* Lo and behold: we have just managed to send a transaction to
the log. Before we can commit it, wait for the IO so far to
diff --git a/fs/mpage.c b/fs/mpage.c
index 3e79220..bf7d6c3 100644
--- a/fs/mpage.c
+++ b/fs/mpage.c
@@ -695,7 +695,7 @@ mpage_writepages(struct address_space *mapping,
if (mpd.bio)
mpage_bio_submit(WRITE, mpd.bio);
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
return ret;
}
EXPORT_SYMBOL(mpage_writepages);
diff --git a/fs/nfs/blocklayout/blocklayout.c b/fs/nfs/blocklayout/blocklayout.c
index 1cac3c1..e93b6a8 100644
--- a/fs/nfs/blocklayout/blocklayout.c
+++ b/fs/nfs/blocklayout/blocklayout.c
@@ -311,7 +311,7 @@ bl_read_pagelist(struct nfs_pgio_header *header)
}
out:
bl_submit_bio(READ, bio);
- blk_finish_plug(&plug);
+ blk_finish_plug();
put_parallel(par);
return PNFS_ATTEMPTED;
}
@@ -433,7 +433,7 @@ bl_write_pagelist(struct nfs_pgio_header *header, int sync)
header->res.count = header->args.count;
out:
bl_submit_bio(WRITE, bio);
- blk_finish_plug(&plug);
+ blk_finish_plug();
put_parallel(par);
return PNFS_ATTEMPTED;
}
diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c
index 1790b00..2f89ca2 100644
--- a/fs/xfs/xfs_buf.c
+++ b/fs/xfs/xfs_buf.c
@@ -1289,7 +1289,7 @@ _xfs_buf_ioapply(
if (size <= 0)
break; /* all done */
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
}
/*
@@ -1823,7 +1823,7 @@ __xfs_buf_delwri_submit(
xfs_buf_submit(bp);
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
return pinned;
}
diff --git a/fs/xfs/xfs_dir2_readdir.c b/fs/xfs/xfs_dir2_readdir.c
index 098cd78..7e8fa3f 100644
--- a/fs/xfs/xfs_dir2_readdir.c
+++ b/fs/xfs/xfs_dir2_readdir.c
@@ -455,7 +455,7 @@ xfs_dir2_leaf_readbuf(
}
}
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
out:
*bpp = bp;
diff --git a/fs/xfs/xfs_itable.c b/fs/xfs/xfs_itable.c
index 82e3142..c3ac5ec 100644
--- a/fs/xfs/xfs_itable.c
+++ b/fs/xfs/xfs_itable.c
@@ -196,7 +196,7 @@ xfs_bulkstat_ichunk_ra(
&xfs_inode_buf_ops);
}
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
}
/*
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 7f9a516..188133f 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -1091,6 +1091,7 @@ static inline void blk_post_runtime_resume(struct request_queue *q, int err) {}
* schedule() where blk_schedule_flush_plug() is called.
*/
struct blk_plug {
+ int depth; /* number of nested plugs */
struct list_head list; /* requests */
struct list_head mq_list; /* blk-mq requests */
struct list_head cb_list; /* md requires an unplug callback */
@@ -1107,7 +1108,7 @@ struct blk_plug_cb {
extern struct blk_plug_cb *blk_check_plugged(blk_plug_cb_fn unplug,
void *data, int size);
extern void blk_start_plug(struct blk_plug *);
-extern void blk_finish_plug(struct blk_plug *);
+extern void blk_finish_plug(void);
extern void blk_flush_plug_list(struct blk_plug *, bool);
static inline void blk_flush_plug(struct task_struct *tsk)
@@ -1646,7 +1647,7 @@ static inline void blk_start_plug(struct blk_plug *plug)
{
}
-static inline void blk_finish_plug(struct blk_plug *plug)
+static inline void blk_finish_plug(void)
{
}
diff --git a/mm/madvise.c b/mm/madvise.c
index d551475..18a34ee 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -539,7 +539,7 @@ SYSCALL_DEFINE3(madvise, unsigned long, start, size_t, len_in, int, behavior)
vma = find_vma(current->mm, start);
}
out:
- blk_finish_plug(&plug);
+ blk_finish_plug();
if (write)
up_write(¤t->mm->mmap_sem);
else
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 644bcb6..4570f6e 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -2020,7 +2020,7 @@ int generic_writepages(struct address_space *mapping,
blk_start_plug(&plug);
ret = write_cache_pages(mapping, wbc, __writepage, mapping);
- blk_finish_plug(&plug);
+ blk_finish_plug();
return ret;
}
diff --git a/mm/readahead.c b/mm/readahead.c
index 9356758..64182a2 100644
--- a/mm/readahead.c
+++ b/mm/readahead.c
@@ -136,7 +136,7 @@ static int read_pages(struct address_space *mapping, struct file *filp,
ret = 0;
out:
- blk_finish_plug(&plug);
+ blk_finish_plug();
return ret;
}
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 405923f..5721f64 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -478,7 +478,7 @@ struct page *swapin_readahead(swp_entry_t entry, gfp_t gfp_mask,
SetPageReadahead(page);
page_cache_release(page);
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
lru_add_drain(); /* Push any new pages onto the LRU now */
skip:
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 5e8eadd..56bb274 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2222,7 +2222,7 @@ static void shrink_lruvec(struct lruvec *lruvec, int swappiness,
scan_adjusted = true;
}
- blk_finish_plug(&plug);
+ blk_finish_plug();
sc->nr_reclaimed += nr_reclaimed;
/*
--
1.8.3.1
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>
^ permalink raw reply related
* Re: Help recovering an interrupted raid0 reshape
From: Jonathan Harker (Jesusaurus) @ 2015-04-07 17:02 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150407163004.7550da77@notabene.brown>
On Mon, Apr 6, 2015 at 11:30 PM, NeilBrown <neilb@suse.de> wrote:
>
> Try:
> mdadm -S /dev/md124
> mdadm -A /dev/md124 --update=revert-reshape /dev/md/alpha /dev/md/beta
> mdadm -S /dev/md124
> mdadm -A /dev/md124 -vvv /dev/md/alpha /dev/md/beta /dev/md/gamma
>
> What does that report?
>
> NeilBrown
>
# mdadm --stop /dev/md124
mdadm: stopped /dev/md124
# mdadm -A /dev/md124 --update=revert-reshape /dev/md/alpha /dev/md/beta
mdadm: /dev/md124 assembled from 2 drives - not enough to start the array.
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1] [raid10] [raid0]
[linear] [multipath]
md124 : inactive md126[0](S) md127[1](S)
3907022200 blocks super 1.2
md0 : active raid1 sda5[0] sdb2[1]
107652416 blocks [2/2] [UU]
bitmap: 1/1 pages [4KB], 65536KB chunk
md125 : active raid1 sdh1[0] sdg1[1]
2930134016 blocks super 1.2 [2/2] [UU]
bitmap: 0/22 pages [0KB], 65536KB chunk
md126 : active raid1 sdc1[0] sdd1[1]
1953512312 blocks super 1.2 [2/2] [UU]
md127 : active raid1 sde1[2] sdf1[1]
1953512312 blocks super 1.2 [2/2] [UU]
unused devices: <none>
# mdadm --stop /dev/md124
mdadm: stopped /dev/md124
# mdadm -A /dev/md124 -vvv /dev/md/alpha /dev/md/beta /dev/md/gamma
mdadm: looking for devices for /dev/md124
mdadm: UUID differs from /dev/md0.
mdadm: UUID differs from /dev/md/alpha.
mdadm: UUID differs from /dev/md/beta.
mdadm: UUID differs from /dev/md/gamma.
mdadm: UUID differs from /dev/md0.
mdadm: UUID differs from /dev/md/alpha.
mdadm: UUID differs from /dev/md/beta.
mdadm: UUID differs from /dev/md/gamma.
mdadm: UUID differs from /dev/md0.
mdadm: UUID differs from /dev/md/alpha.
mdadm: UUID differs from /dev/md/beta.
mdadm: UUID differs from /dev/md/gamma.
mdadm: /dev/md/alpha is identified as a member of /dev/md124, slot 1.
mdadm: /dev/md/beta is identified as a member of /dev/md124, slot 0.
mdadm: /dev/md/gamma is identified as a member of /dev/md124, slot 2.
mdadm: :/dev/md124 has an active reshape - checking if critical
section needs to be restored
mdadm: added /dev/md/alpha to /dev/md124 as 1
mdadm: added /dev/md/gamma to /dev/md124 as 2 (possibly out of date)
mdadm: no uptodate device for slot 6 of /dev/md124
mdadm: added /dev/md/beta to /dev/md124 as 0
mdadm: /dev/md124 assembled from 2 drives - not enough to start the array.
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1] [raid10] [raid0]
[linear] [multipath]
md124 : inactive md125[3](S) md127[1](S) md126[0](S)
6837155192 blocks super 1.2
md0 : active raid1 sda5[0] sdb2[1]
107652416 blocks [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk
md125 : active raid1 sdh1[0] sdg1[1]
2930134016 blocks super 1.2 [2/2] [UU]
bitmap: 0/22 pages [0KB], 65536KB chunk
md126 : active raid1 sdc1[0] sdd1[1]
1953512312 blocks super 1.2 [2/2] [UU]
md127 : active raid1 sde1[2] sdf1[1]
1953512312 blocks super 1.2 [2/2] [UU]
unused devices: <none>
# mdadm --examine /dev/md/alpha
/dev/md/alpha:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
Name : hordern:hordern1 (local to host hordern)
Creation Time : Fri Jan 2 09:59:40 2009
Raid Level : raid4
Raid Devices : 4
Avail Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
Array Size : 5860532736 (5589.04 GiB 6001.19 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=752 sectors
State : active
Device UUID : 63aaa2e4:2a09f495:8372c7f9:eb2f2773
Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
Delta Devices : 1 (3->4)
Update Time : Sun Mar 29 15:11:35 2015
Checksum : 8be5e0e8 - correct
Events : 14013
Chunk Size : 512K
Device Role : Active device 1
Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
# mdadm --examine /dev/md/beta
/dev/md/beta:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
Name : hordern:hordern1 (local to host hordern)
Creation Time : Fri Jan 2 09:59:40 2009
Raid Level : raid4
Raid Devices : 4
Avail Dev Size : 3907022576 (1863.01 GiB 2000.40 GB)
Array Size : 5860532736 (5589.04 GiB 6001.19 GB)
Used Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=752 sectors
State : clean
Device UUID : 6e6dce14:3ebb2bb5:187aa292:403a55f6
Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
Delta Devices : 1 (3->4)
Update Time : Sun Mar 29 15:11:35 2015
Checksum : f7526adf - correct
Events : 14013
Chunk Size : 512K
Device Role : Active device 0
Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
# mdadm --examine /dev/md/gamma
/dev/md/gamma:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x6
Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
Name : hordern:hordern1 (local to host hordern)
Creation Time : Fri Jan 2 09:59:40 2009
Raid Level : raid4
Raid Devices : 4
Avail Dev Size : 5860265984 (2794.39 GiB 3000.46 GB)
Array Size : 5860532736 (5589.04 GiB 6001.19 GB)
Used Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Recovery Offset : 86403072 sectors
Unused Space : before=1960 sectors, after=1953244160 sectors
State : active
Device UUID : 782873ea:e265ecd4:5cc80ddf:035ba2b4
Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
Delta Devices : 1 (3->4)
Update Time : Sun Mar 29 00:05:29 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 710dc078 - correct
Events : 673
Chunk Size : 512K
Device Role : Active device 2
Array State : AAA. ('A' == active, '.' == missing, 'R' == replacing)
# mdadm --detail /dev/md124
/dev/md124:
Version : 1.2
Raid Level : raid0
Total Devices : 3
Persistence : Superblock is persistent
State : inactive
Delta Devices : 1, (-1->0)
New Level : raid4
New Chunksize : 512K
Name : hordern:hordern1 (local to host hordern)
UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
Events : 673
Number Major Minor RaidDevice
- 9 125 - /dev/md/gamma
- 9 126 - /dev/md/beta
- 9 127 - /dev/md/alpha
So it looks like all three component devices have consistent
superblocks now, awesome! But the raid0 array is still inactive with
all three components listed as spares. It looks like /dev/md/gamma has
a much lower event count, I'm guessing that is what causes the disk to
be marked as possibly out of date.
Is an "uptodate device" a specific thing, or does that simply mean
that some component devices are out of date? The lack of spaces makes
me think that uptodate is some keyword I'm not recognizing.
--
Jonathan
^ permalink raw reply
* Creating volume in IMSM container
From: Ian Pilcher @ 2015-04-07 15:29 UTC (permalink / raw)
To: linux-raid
I am trying to create an IMSM container and volume in a KVM virtual
machine (so that I can create a reproducer for a Fedora problem that
seems to only/mostly affect IMSM RAID). I've managed to create a
container with the following command:
IMSM_NO_PLATFORM=1 mdadm --create --verbose /dev/md/imsm \
/dev/vd{a,b} --raid-devices 2 --metadata=imsm
/proc/mdstat now contains:
Personalities: [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] ...
md127 : inactive vdb[1](S) vda[0](S)
2210 blocks super external:imsm
I haven't, however, been able to create a volume within the container:
# IMSM_NO_PLATFORM=1 mdadm --create --verbose /dev/md/vol0 \
/dev/md/imsm --raid-devices 2 --level 1 --metadata=imsm
mdmon: not enough devices with space to create array.
# IMSM_NO_PLATFORM=1 mdadm --create --verbose /dev/md/vol0 \
/dev/vd{a,b} --raid-devices 2 --level 1 --metadata=imsm
mdadm: /dev/vda appears to be part of a raid array:
level=container devices=0 ctime=Thu Jan 1 00:00:00 1970
mdmon: /dev/vdb is not in the same imsm set
mdadm: /dev/vdb is not suitable for this array.
mdadm: create aborted
Am I doing something wrong, or is this simply not possible?
--
========================================================================
Ian Pilcher arequipeno@gmail.com
-------- "I grew up before Mark Zuckerberg invented friendship" --------
========================================================================
^ permalink raw reply
* Re: [dm-devel] [PATCH 3/3] dm-crypt: Adds support for wiping key when doing suspend/hibernation
From: Alasdair G Kergon @ 2015-04-07 13:55 UTC (permalink / raw)
To: Pali Rohár
Cc: Alasdair Kergon, Mike Snitzer, Neil Brown, Rafael J. Wysocki,
Len Brown, Pavel Machek, linux-raid, dm-devel, linux-kernel,
linux-pm
In-Reply-To: <1428254419-7334-4-git-send-email-pali.rohar@gmail.com>
On Sun, Apr 05, 2015 at 07:20:19PM +0200, Pali Rohár wrote:
> This patch adds dm message commands and option strings to optionally wipe key
> from dm-crypt device before entering suspend or hibernate state.
Try to avoid 0/1 - use descriptive options instead.
E.g. key wipe_on_hibernation / key retain_on_hibernation (message)
wipe_key_on_hiberation ('dmsetup table' - don't forget the reporting interface!)
Have you tested against every state the driver might be in at the time of
suspend/hibernation?
> +static void crypt_suspend_and_wipe_key(struct crypt_config *cc)
> +{
> + dm_suspend_md(dm_table_get_md(cc->ti->table));
I'm not particularly keen on this - silently ignoring expected error states
like -EINVAL rather than checking first and not calling the function at all
when it's known not to be needed.
Alasdair
^ permalink raw reply
* Re: Help recovering an interrupted raid0 reshape
From: NeilBrown @ 2015-04-07 6:30 UTC (permalink / raw)
To: Jonathan Harker (Jesusaurus); +Cc: linux-raid
In-Reply-To: <CAC_83AFo47Zfe7rMqro7_k8dR-xpMcfiwQA7a9DTKpKPuHr=fA@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1915 bytes --]
On Mon, 6 Apr 2015 22:14:56 -0700 "Jonathan Harker (Jesusaurus)"
<jesusaurus@gentlydownthe.net> wrote:
> On Mon, Apr 6, 2015 at 6:50 PM, NeilBrown <neilb@suse.de> wrote:
> > On Tue, 7 Apr 2015 09:46:08 +1000 NeilBrown <neilb@suse.de> wrote:
> >
> >> On Fri, 3 Apr 2015 19:22:15 -0700 "Jonathan Harker (Jesusaurus)"
> >> <jesusaurus@gentlydownthe.net> wrote:
> >>
> >> >
> >> > Third, is it possible to resume the reshape? If not, can it be reverted?
> >> >
> >>
> >> I'll have to get back to you on that. Hopefully you can be patient a little
> >> longer.
> >>
> >> NeilBrown
> >>
>
> Thanks for taking the time to look into this. I've managed to be
> patient with the problem for about a week now (I was cautiously poking
> and prodding the array for a few days before reaching out to this
> list) so I think I can manage a little more patience.
>
> >>
> >
> > Hi again Jonathan,
> > please report what version of mdadm and of Linux that you are running.
>
> My mdadm version is "mdadm - v3.3.1 - 5th June 2014" and my linux
> version (according to `uname -r`) is 3.16.2-3.
>
> >
> > Also is there any chance that you tried
> > mdadm --assemble .... --update=revert-reshape
> > ??
> > And if you did, why didn't you say so :-)
>
> During those first couple days I tried a couple assemble and
> incremental commands to try to get the reshape to restart, though I
> don't remember exactly what they were. It's possible I tried something
> as scary and irreversible as --update=revert-reshape in some desperate
> haze, so let's assume I did, but I was trying hard to avoid any advice
> that included --force in the command.
Try:
mdadm -S /dev/md124
mdadm -A /dev/md124 --update=revert-reshape /dev/md/alpha /dev/md/beta
mdadm -S /dev/md124
mdadm -A /dev/md124 -vvv /dev/md/alpha /dev/md/beta /dev/md/gamma
What does that report?
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Help recovering an interrupted raid0 reshape
From: Jonathan Harker (Jesusaurus) @ 2015-04-07 5:14 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150407115033.1d63b65c@notabene.brown>
On Mon, Apr 6, 2015 at 6:50 PM, NeilBrown <neilb@suse.de> wrote:
> On Tue, 7 Apr 2015 09:46:08 +1000 NeilBrown <neilb@suse.de> wrote:
>
>> On Fri, 3 Apr 2015 19:22:15 -0700 "Jonathan Harker (Jesusaurus)"
>> <jesusaurus@gentlydownthe.net> wrote:
>>
>> >
>> > Third, is it possible to resume the reshape? If not, can it be reverted?
>> >
>>
>> I'll have to get back to you on that. Hopefully you can be patient a little
>> longer.
>>
>> NeilBrown
>>
Thanks for taking the time to look into this. I've managed to be
patient with the problem for about a week now (I was cautiously poking
and prodding the array for a few days before reaching out to this
list) so I think I can manage a little more patience.
>>
>
> Hi again Jonathan,
> please report what version of mdadm and of Linux that you are running.
My mdadm version is "mdadm - v3.3.1 - 5th June 2014" and my linux
version (according to `uname -r`) is 3.16.2-3.
>
> Also is there any chance that you tried
> mdadm --assemble .... --update=revert-reshape
> ??
> And if you did, why didn't you say so :-)
During those first couple days I tried a couple assemble and
incremental commands to try to get the reshape to restart, though I
don't remember exactly what they were. It's possible I tried something
as scary and irreversible as --update=revert-reshape in some desperate
haze, so let's assume I did, but I was trying hard to avoid any advice
that included --force in the command.
>
> NeilBrown
Thanks again for looking into this, and let me know if there is any
other useful information I can provide.
--
Jonathan
^ permalink raw reply
* Re: multi-threaded raid5
From: Thomas Fjellstrom @ 2015-04-07 4:34 UTC (permalink / raw)
To: NeilBrown, linux-raid
In-Reply-To: <20150407111157.2200fb0b@notabene.brown>
On Tue 07 Apr 2015 11:11:57 AM NeilBrown wrote:
> On Tue, 31 Mar 2015 19:17:34 +0200 stkoupr <p.kouril@cra.cz> wrote:
> > Hi,
> >
> > We are testing write performance of md raid5 with SSD array and we are
> > hitting limit which I understand is caused by writes being handled in
> > single thread (md0_raid5 process is eating 100% of 1 CPU core when the
> > test is running). I understood that there is some work in progress which
> > should make write handling multi-threaded. I'd like to ask about the
> > status of this work, is it still in progress or planned?
> >
> > Regards,
> > Prema
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> See /sys/block/mdXXX/md/group_thread_cnt
>
> Set it to a bigger number, get more threads.
>
> Requires 3.12 or later.
What does it mean when that property is set to 0?
> NeilBrown
--
Thomas Fjellstrom
thomas@fjellstrom.ca
^ permalink raw reply
* Re: Can't mount partitions after "mdadm --zero-superblock"
From: Eduard Rozenberg @ 2015-04-07 3:31 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150407105230.4ff9a97a@notabene.brown>
Hello Neil,
Success! This is stronger voodoo magic than I’ve ever
had to perform so really didn’t have the faith to continue
without the extra encouragement :). My initial mistake
had been to use the octal value with losetup instead of
getting the decimal value.
Documenting my steps below for anybody else who
might come here later.
In the examples below, we are using partition "/dev/sdac2"
Replace this with the appropriate partition you’re recovering.
Step 1: find the decimal value for the start of the partition
----------------------------------------------------------------------------
Note:
ext4 partitions have a “magic" octal value of “ef53"
to indicate the start of the partition. Note that "ef53"
may show more than once as you read further into
the partition. We are interested in the location the
very first occurrence of "ef53". Other types of
partitions (ext2, ext3, etc) probably have other magic
values to look for so this “ef53" may not apply there.
The "od" hex viewer command to search for “ef53":
od -x /dev/sdac2 | awk '$6 == "ef53"'
The results will look something like:
4002060 f3fd 5521 0004 0025 ef53 0001 0001 0000
1004000060 64be 4ec9 0000 0025 ef53 0000 0001 0000
1042630400 17f8 a7dd bb6e ee40 ef53 000d 3cfb 9e22
We are only interested in the first line. So we now have
the octal address of the ef53 magic value: it’s the first
long number on the line: "4002060" (octal value!)
According to Neil’s instructions then we have to subtract
the octal value “0002060" from this number we found.
We then have to convert the octal result into decimal.
Luckily an online calculator makes this easy:
http://www.csgnetwork.com/octaddsubcalc.html
"Enter a octal value" - Enter “4002060" here
"Enter Second Octal Value" - Enter “0002060" here
Then take the value from the line:
"Calculated Decimal Subtraction" - 1048576
This is the decimal value for the start of our partition.
Step 2: use the decimal start value to mount partition
———————————————————————————————————
First create a loop device loop0 as a handle to the
partition. We tell losetup where the start of the partition is:
losetup -o 1048576 /dev/loop0 /dev/sdac2
Next, try to mount loop0 read-only (hopefully it will work!)
mount -o ro /dev/loop0 /mnt
If the partition is unclean and needs to be fscked:
fsck.ext4 /dev/loop0
Thanks again Neil! Maybe a few years from now I’ll
understand why this worked when nothing else did
( linux tools still have some ways to go to being
intelligent enough to do this kind of recovery).
Regards,
—Ed
> On Apr 6, 2015, at 17:52, NeilBrown <neilb@suse.de> wrote:
>
> On Mon, 6 Apr 2015 16:45:58 -0700 Eduard Rozenberg <eduardr@pobox.com> wrote:
>
>> Hello folks,
>>
>> I previously had the following setup:
>>
>> sda & sdb partitioned w/ GPT, 7 partitions each (usr, opt, var etc...)
>> 7 raid1’s with 2 devices for each pair of partitions (/dev/sda1 & /dev/sdab1, etc)
>> They’d been created under Slackware 13.37.
>>
>> I was trying to clean out mdadm from those partitions but keep the data so I ran
>> "mdadm --zero-superblock” on each of those previously RAID1 mdadm 1.2 ext4
>> partitions.
>
> The "1.2" metadata is stored 4k from the start of the device. The actual
> data is some megabytes further in. I don't suppose you still have the output
> of "mdadm --examine" from before you destroyed the superblocks??
>
>>
>> As a result I am now currently unable to mount any partition after the first one on either
>> disk. The first partition does mount. The partition table is visible and looks fine in gdisk.
>>
>> mount -t ext4 /dev/sdac2 /mnt
>> mount: wrong fs type, bad option, bad superblock on /dev/sdac2,
>> missing codepage or helper program, or other error
>> In some cases useful info is found in syslog - try
>> dmesg | tail or so
>>
>> I did try superblock recovery with each backup superblock that ext4 normally creates,
>> but none of the superblock locations worked.
>>
>> For example:
>>
>> fsck.ext4 -b 4096000 /dev/sdac2
>> e2fsck 1.42.8 (20-Jun-2013)
>> /sbin/e2fsck: Invalid argument while trying to open /dev/sdac2
>>
>> The superblock could not be read or does not describe a correct ext2
>> filesystem. If the device is valid and it really contains an ext2
>> filesystem (and not swap or ufs or something else), then the superblock
>> is corrupt, and you might try running e2fsck with an alternate superblock:
>> e2fsck -b 8193 <device>
>>
>>
>> Would be grateful for any advice on anything else I can try.
>
> You need to find where the filesystem actually starts, then you need to
> create some way to access it as a block device, then it should "just work".
>
> An ext4 filesystem superblock has 0xef53 at an offset of 0x38, and the
> superblock is typically 1K from the start of the partition.
>
> So you could:
> od -x /dev/sdac2 | awk '$6 == "ef53"'
>
> Then subtrace 0002060 (octal) from the leading number, and that might be the
> start of the partition.
>
> Then
> losetup -o "start in decimal" /dev/loop0 /dev/sdac2
>
> and try 'fsck' on /dev/loop0
>
> Good luck.
>
> NeilBrown
>
>
>
>>
>> Regards,
>> —Ed--
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Can't mount partitions after "mdadm --zero-superblock"
From: NeilBrown @ 2015-04-07 1:52 UTC (permalink / raw)
To: Eduard Rozenberg; +Cc: linux-raid
In-Reply-To: <CE6323D1-485D-4951-9E99-CBE5A09AE7BD@pobox.com>
[-- Attachment #1: Type: text/plain, Size: 621 bytes --]
On Mon, 6 Apr 2015 18:33:36 -0700 Eduard Rozenberg <eduardr@pobox.com> wrote:
> Thanks very much Neil!
>
> I never do anything forward thinking like mdadm —examine before
> doing destructive things :). Unfortunately.
>
> I couldn’t figure out how to make the offset work so I used
>
> losetup -f --show -P /dev/sdac
Sounds like you are saying "I couldn't make the drill work, so I tried using a
hammer"... and then were surprised that you couldn't make any holes.
Try harder to make the offset work. if/when it doesn't, report exactly what
you did and exactly what the result was.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Help recovering an interrupted raid0 reshape
From: NeilBrown @ 2015-04-07 1:50 UTC (permalink / raw)
To: Jonathan Harker (Jesusaurus); +Cc: linux-raid
In-Reply-To: <20150407094608.4a9dd142@notabene.brown>
[-- Attachment #1: Type: text/plain, Size: 642 bytes --]
On Tue, 7 Apr 2015 09:46:08 +1000 NeilBrown <neilb@suse.de> wrote:
> On Fri, 3 Apr 2015 19:22:15 -0700 "Jonathan Harker (Jesusaurus)"
> <jesusaurus@gentlydownthe.net> wrote:
>
> >
> > Third, is it possible to resume the reshape? If not, can it be reverted?
> >
>
> I'll have to get back to you on that. Hopefully you can be patient a little
> longer.
>
> NeilBrown
>
>
Hi again Jonathan,
please report what version of mdadm and of Linux that you are running.
Also is there any chance that you tried
mdadm --assemble .... --update=revert-reshape
??
And if you did, why didn't you say so :-)
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox