* Re: [RFC 5/5] r5cache: naive reclaim approach
From: NeilBrown @ 2016-06-01 3:16 UTC (permalink / raw)
To: linux-raid; +Cc: shli, dan.j.williams, hch, kernel-team, Song Liu
In-Reply-To: <1464326983-3798454-6-git-send-email-songliubraving@fb.com>
[-- Attachment #1: Type: text/plain, Size: 209 bytes --]
On Fri, May 27 2016, Song Liu wrote:
> + sector_t end = -1L;
Please use "MaxSector", not "-1L".
sector_t might be "long long" so -1L might be wrong.
(I haven't looked at much else in this patch)
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [RFC 4/5] r5cache: write part of r5cache
From: NeilBrown @ 2016-06-01 3:12 UTC (permalink / raw)
To: linux-raid; +Cc: shli, dan.j.williams, hch, kernel-team, Song Liu
In-Reply-To: <1464326983-3798454-5-git-send-email-songliubraving@fb.com>
[-- Attachment #1: Type: text/plain, Size: 4989 bytes --]
On Fri, May 27 2016, Song Liu wrote:
> This is the write part of r5cache. The cache is integrated with
> stripe cache of raid456. It leverages code of r5l_log to write
> data to journal device.
>
> r5cache split current write path into 2 parts: the write path
> and the reclaim path. The write path is as following:
> 1. write data to journal
> 2. call bio_endio
>
> Then the reclaim path is as:
> 1. Freeze the stripe (no more writes coming in)
> 2. Calcualte parity (reconstruct or RMW)
> 3. Write parity to journal device (data is already written to it)
> 4. Write data and parity to RAID disks
>
> With r5cache, write operation does not wait for parity calculation
> and write out, so the write latency is lower (1 write to journal
> device vs. read and then write to raid disks). Also, r5cache will
> reduce RAID overhead (multipile IO due to read-modify-write of
> parity) and provide more opportunities of full stripe writes.
>
> r5cache adds a new state of each stripe: enum r5c_states. The write
> path runs in state CLEAN and RUNNING (data in cache). Cache writes
> start from r5c_handle_stripe_dirtying(), where bit R5_Wantcache is
> set for devices with bio in towrite. Then, the data is written to
> the journal through r5l_log implementation. Once the data is in the
> journal, we set bit R5_InCache, and presue bio_endio for these
> writes.
I don't think this new state field is a good idea. We already have
plenty of state information and it would be best to fit in with that.
When handle_stripe() runs it assesses the state of the stripe and
decides what to do next. The sh->count is incremented and it doesn't
return to zero until all the actions that were found to be necessary
have completely. When it does return to zero, handle_stripe() runs
again to see what else is needed.
You may well need to add sh->state or dev->flags flags to record new
aspects of the state, such as whether data is safe in the log or safe in
the RAID, but I don't think a new state variable will really help.
>
> The reclaim path starts by freezing the stripe (no more writes).
> This stripes back to raid5 state machine, where
> handle_stripe_dirtying will evaluate the stripe for reconstruct
> writes or RMW writes (read data and calculate parity).
>
> For RMW, the code allocates extra page for prexor. Specifically,
> a new page is allocated for r5dev->page to do prexor; while
> r5dev->orig_page keeps the cached data. The extra page is freed
> after prexor.
It isn't at all clear to me why you need the extra page. Could you
explain when and why the ->page and the ->orig_page would contain
different content and both of the content would be needed?
>
> r5cache naturally excludes SkipCopy. With R5_Wantcache bit set,
> async_copy_data will not skip copy.
>
> Before writing data to RAID disks, the r5l_log logic stores
> parity (and non-overwrite data) to the journal.
>
> Instead of inactive_list, stripes with cached data are tracked in
> r5conf->r5c_cached_list. r5conf->r5c_cached_stripes tracks how
> many stripes has dirty data in the cache.
>
> Two sysfs entries are provided for the write cache:
> 1. r5c_cached_stripes shows how many stripes have cached data.
> Writing anything to r5c_cached_stripes will flush all data
> to RAID disks.
> 2. r5c_cache_mode provides knob to switch the system between
> write-back or write-through (only write-log).
>
> There are some known limitations of the cache implementation:
>
> 1. Write cache only covers full page writes (R5_OVERWRITE). Writes
> of smaller granularity are write through.
> 2. Only one log io (sh->log_io) for each stripe at anytime. Later
> writes for the same stripe have to wait. This can be improved by
> moving log_io to r5dev.
>
> Signed-off-by: Song Liu <songliubraving@fb.com>
> Signed-off-by: Shaohua Li <shli@fb.com>
> ---
> drivers/md/raid5-cache.c | 399 +++++++++++++++++++++++++++++++++++++++++++++--
> drivers/md/raid5.c | 172 +++++++++++++++++---
> drivers/md/raid5.h | 38 ++++-
> 3 files changed, 577 insertions(+), 32 deletions(-)
>
> diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
> index 5f0d96f..66a3cd5 100644
> --- a/drivers/md/raid5-cache.c
> +++ b/drivers/md/raid5-cache.c
> @@ -40,7 +40,19 @@
> */
> #define R5L_POOL_SIZE 4
>
> +enum r5c_cache_mode {
> + R5C_MODE_NO_CACHE = 0,
> + R5C_MODE_WRITE_THROUGH = 1,
> + R5C_MODE_WRITE_BACK = 2,
> +};
> +
> +static char *r5c_cache_mode_str[] = {"no-cache", "write-through", "write-back"};
> +
> struct r5c_cache {
> + int flush_threshold; /* flush the stripe when flush_threshold buffers are dirty */
> + int mode; /* enum r5c_cache_mode */
why isn't this just "enum r5c_cache_mode mode;" ??
Clearly this structure is more than just statistics. But now I wonder
why these fields aren't simply added to struct r5conf. Or maybe in
r5l_log.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [RFC 3/5] r5cache: look up stripe cache for chunk_aligned_read
From: NeilBrown @ 2016-06-01 2:52 UTC (permalink / raw)
To: linux-raid; +Cc: shli, dan.j.williams, hch, kernel-team, Song Liu
In-Reply-To: <1464326983-3798454-4-git-send-email-songliubraving@fb.com>
[-- Attachment #1: Type: text/plain, Size: 14643 bytes --]
On Fri, May 27 2016, Song Liu wrote:
> This is the read part of raid5 cache (r5cache).
>
> In raid456, when the array is in-sync, the md layer bypasses stripe
> cache for chunk_aligned_read(). However, with write back cache,
> data in the RAID disks may not be uptodate. Therefore, it is necessary
> to search the stripe cache latest data.
>
> With this patch, raid5_read_one_chunk() looks up data in stripe cache.
> The outcome of this lookup could be read_full_hit (all data of the
> chunk are in stripe cache), read_partial_hit (only part of the chunk
> is in stripe cache), or read_miss (no data of the chunk in stripe cache).
>
> For read_full_hit, raid5_read_one_chunk returns data directly from
> stripe cache; for read_miss, raid5_read_one_chunk reads all data from
> the disk; for read_partial_hit, raid5_read_one_chunk reads data from
> disk, and amends the data with data in stripe cache in endio
> (r5c_chunk_aligned_read_endio).
I wonder if all this complexity is really justified.
If we cannot bypass the cache, then we can just perform the read using
the cache with the existing code. That will be slower than the direct
bypass, but is it so much slower that the complexity is needed.
Alternately.... is a full hit at all likely? If data is in the stripe
cache, then there is a good chance it is in the page cache too, so it
won't be read.
If we assume that all reads will either be partial-hits or misses, then
your approach will always read from the disk, then add updates (possibly
none) from the cache. In that case it might be simpler to do exactly
that.
i.e. schedule the read and then when that finishes, find all the stripes
which overlap and if they contain unwritten data, attach the bio and
queue for normal handling.
I suspect the code would end up being much simpler as it would reuse
more existing code.
>
> Sysfs entry is added to show statistics of read_full_hits,
> read_partial_hits, and read_misses.
I'm not sure putting statistics like this in /sys is a good idea. If
they are just for debugging, then put them in debugfs.
>
> Signed-off-by: Song Liu <songliubraving@fb.com>
> Signed-off-by: Shaohua Li <shli@fb.com>
> ---
> drivers/md/raid5-cache.c | 238 +++++++++++++++++++++++++++++++++++++++++++++++
> drivers/md/raid5.c | 23 ++++-
> drivers/md/raid5.h | 6 ++
> 3 files changed, 265 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
> index e889e2d..5f0d96f 100644
> --- a/drivers/md/raid5-cache.c
> +++ b/drivers/md/raid5-cache.c
> @@ -40,8 +40,15 @@
> */
> #define R5L_POOL_SIZE 4
>
> +struct r5c_cache {
> + atomic64_t read_full_hits; /* the whole chunk in cache */
> + atomic64_t read_partial_hits; /* some pages of the chunk in cache */
> + atomic64_t read_misses; /* the whold chunk is not in cache */
> +};
If this structure just holds statistics, then the name of the structure
should indicate that. "r5c_cache" makes it seem like it is a cache...
> +
> struct r5l_log {
> struct md_rdev *rdev;
> + struct r5c_cache cache;
>
> u32 uuid_checksum;
>
> @@ -134,6 +141,21 @@ enum r5l_io_unit_state {
> IO_UNIT_STRIPE_END = 3, /* stripes data finished writing to raid */
> };
>
> +struct r5c_chunk_map {
> + int sh_count;
> + struct r5conf *conf;
> + struct bio *parent_bi;
> + int dd_idx;
> + struct stripe_head *sh_array[0];
> +};
> +
> +static void init_r5c_cache(struct r5conf *conf, struct r5c_cache *cache)
> +{
> + atomic64_set(&cache->read_full_hits, 0);
> + atomic64_set(&cache->read_partial_hits, 0);
> + atomic64_set(&cache->read_misses, 0);
> +}
> +
> static sector_t r5l_ring_add(struct r5l_log *log, sector_t start, sector_t inc)
> {
> start += inc;
> @@ -1120,6 +1142,220 @@ static void r5l_write_super(struct r5l_log *log, sector_t cp)
> set_bit(MD_CHANGE_DEVS, &mddev->flags);
> }
>
> +/* TODO: use async copy */
> +static void r5c_copy_data_to_bvec(struct r5dev *rdev, int sh_offset,
> + struct bio_vec *bvec, int bvec_offset, int len)
> +{
> + /* We always copy data from orig_page. This is because in R-M-W, we use
> + * page to do prexor of parity */
> + void *src_p = kmap_atomic(rdev->orig_page);
> + void *dst_p = kmap_atomic(bvec->bv_page);
> + memcpy(dst_p + bvec_offset, src_p + sh_offset, len);
> + kunmap_atomic(dst_p);
> + kunmap_atomic(src_p);
> +}
> +
> +/*
> + * copy data from a chunk_map to a bio
> + */
> +static void r5c_copy_chunk_map_to_bio(struct r5c_chunk_map *chunk_map,
> + struct bio *bio)
> +{
> + struct bvec_iter iter;
> + struct bio_vec bvec;
> + int sh_idx;
> + unsigned sh_offset;
> +
> + sh_idx = 0;
> + sh_offset = (bio->bi_iter.bi_sector & ((sector_t)STRIPE_SECTORS-1)) << 9;
> +
> + /*
> + * If bio is not page aligned, the chunk_map will have 1 more sh than bvecs
> + * in the bio. Chunk_map may also have NULL-sh. To copy the right data, we
> + * need to walk through the chunk_map carefully. In this implementation,
> + * bvec/bvec_offset always matches with sh_array[sh_idx]/sh_offset.
> + *
> + * In the following example, the nested loop will run 4 times; and
> + * r5c_copy_data_to_bvec will be called for the first and last iteration.
> + *
> + * --------------------------------
> + * chunk_map | valid sh | NULL | valid sh |
> + * --------------------------------
> + * ---------------------
> + * bio | | |
> + * ---------------------
> + *
> + * | | | | |
> + * copy_data | Y | N | N | Y |
> + */
> + bio_for_each_segment(bvec, bio, iter) {
> + int len;
> + unsigned bvec_offset = bvec.bv_offset;
> + while (bvec_offset < PAGE_SIZE) {
> + len = min_t(unsigned, PAGE_SIZE - bvec_offset, PAGE_SIZE - sh_offset);
> + if (chunk_map->sh_array[sh_idx])
> + r5c_copy_data_to_bvec(&chunk_map->sh_array[sh_idx]->dev[chunk_map->dd_idx], sh_offset,
> + &bvec, bvec_offset, len);
> + bvec_offset += len;
> + sh_offset += len;
> + if (sh_offset == PAGE_SIZE) {
> + sh_idx += 1;
> + sh_offset = 0;
> + }
> + }
> + }
> + return;
> +}
> +
> +/*
> + * release stripes in chunk_map and free the chunk_map
> + */
> +static void free_r5c_chunk_map(struct r5c_chunk_map *chunk_map)
> +{
> + unsigned sh_idx;
> + struct stripe_head *sh;
> +
> + for (sh_idx = 0; sh_idx < chunk_map->sh_count; ++sh_idx) {
> + sh = chunk_map->sh_array[sh_idx];
> + if (sh) {
> + set_bit(STRIPE_HANDLE, &sh->state);
> + raid5_release_stripe(sh);
> + }
> + }
> + kfree(chunk_map);
> +}
> +
> +static void r5c_chunk_aligned_read_endio(struct bio *bio)
> +{
> + struct r5c_chunk_map *chunk_map = (struct r5c_chunk_map *) bio->bi_private;
> + struct bio *parent_bi = chunk_map->parent_bi;
> +
> + r5c_copy_chunk_map_to_bio(chunk_map, bio);
> + free_r5c_chunk_map(chunk_map);
> + bio_put(bio);
> + bio_endio(parent_bi);
> +}
> +
> +/*
> + * look up bio in stripe cache
> + * return raid_bio -> no data in cache, read the chunk from disk
> + * return new r5c_bio -> partial data in cache, read from disk, and amend in r5c_align_endio
> + * return NULL -> all data in cache, no need to read disk
> + */
> +struct bio *r5c_lookup_chunk(struct r5l_log *log, struct bio *raid_bio)
> +{
> + struct r5conf *conf;
> + sector_t logical_sector;
> + sector_t first_stripe, last_stripe; /* first (inclusive) stripe and last (exclusive) */
> + int dd_idx;
> + struct stripe_head *sh;
> + unsigned sh_count, sh_idx, sh_cached;
> + struct r5c_chunk_map *chunk_map;
> + struct bio *r5c_bio;
> + int hash;
> + unsigned long flags;
> +
> + if (!log)
> + return raid_bio;
> +
> + conf = log->rdev->mddev->private;
> +
> + logical_sector = raid_bio->bi_iter.bi_sector &
> + ~((sector_t)STRIPE_SECTORS-1);
> + sh_count = DIV_ROUND_UP_SECTOR_T(bio_end_sector(raid_bio) - logical_sector, STRIPE_SECTORS);
> +
> + first_stripe = raid5_compute_sector(conf, logical_sector, 0, &dd_idx, NULL);
> + last_stripe = first_stripe + STRIPE_SECTORS * sh_count;
> +
> + chunk_map = kzalloc(sizeof(struct r5c_chunk_map) + sh_count * sizeof(struct stripe_head*), GFP_NOIO);
> + sh_cached = 0;
> +
> + for (sh_idx = 0; sh_idx < sh_count; ++sh_idx) {
> + hash = stripe_hash_locks_hash(first_stripe + sh_idx * STRIPE_SECTORS);
> + spin_lock_irqsave(conf->hash_locks + hash, flags);
> + sh = __find_stripe(conf, first_stripe + sh_idx * STRIPE_SECTORS, conf->generation);
> + if (sh &&
> + test_bit(R5_UPTODATE, &sh->dev[dd_idx].flags)) {
> + if (!atomic_inc_not_zero(&sh->count)) {
> + spin_lock(&conf->device_lock);
> + if (!atomic_read(&sh->count)) {
> + if (!test_bit(STRIPE_HANDLE, &sh->state))
> + atomic_inc(&conf->active_stripes);
> + BUG_ON(list_empty(&sh->lru) &&
> + !test_bit(STRIPE_EXPANDING, &sh->state));
> + list_del_init(&sh->lru);
> + if (sh->group) {
> + sh->group->stripes_cnt--;
> + sh->group = NULL;
> + }
> + }
> + atomic_inc(&sh->count);
> + spin_unlock(&conf->device_lock);
> + }
> + chunk_map->sh_array[sh_idx] = sh;
> + ++sh_cached;
> + }
> + spin_unlock_irqrestore(conf->hash_locks + hash, flags);
> + }
> +
> + if (sh_cached == 0) {
> + atomic64_inc(&log->cache.read_misses);
> + kfree(chunk_map);
> + return raid_bio;
> + }
> +
> + chunk_map->sh_count = sh_count;
> + chunk_map->dd_idx = dd_idx;
> +
> + if (sh_cached == sh_count) {
> + atomic64_inc(&log->cache.read_full_hits);
> + r5c_copy_chunk_map_to_bio(chunk_map, raid_bio);
> + free_r5c_chunk_map(chunk_map);
> + bio_endio(raid_bio);
> + return NULL;
> + }
> +
> + chunk_map->parent_bi = raid_bio;
> + chunk_map->conf = conf;
> +
> + atomic64_inc(&log->cache.read_partial_hits);
> +
> + /* TODO: handle bio_clone failure? */
> + r5c_bio = bio_clone_mddev(raid_bio, GFP_NOIO, log->rdev->mddev);
> +
> + r5c_bio->bi_private = chunk_map;
> + r5c_bio->bi_end_io = r5c_chunk_aligned_read_endio;
> +
> + return r5c_bio;
> +}
> +
> +ssize_t
> +r5c_stat_show(struct mddev *mddev, char* page)
> +{
> + struct r5conf *conf = mddev->private;
> + struct r5l_log *log;
> + int ret = 0;
> +
> + if (!conf)
> + return 0;
> +
> + log = conf->log;
> +
> + if (!log)
> + return 0;
> +
> + ret += snprintf(page + ret, PAGE_SIZE - ret, "r5c_read_full_hits: %llu\n",
> + (unsigned long long) atomic64_read(&log->cache.read_full_hits));
> +
> + ret += snprintf(page + ret, PAGE_SIZE - ret, "r5c_read_partial_hits: %llu\n",
> + (unsigned long long) atomic64_read(&log->cache.read_partial_hits));
> +
> + ret += snprintf(page + ret, PAGE_SIZE - ret, "r5c_read_misses: %llu\n",
> + (unsigned long long) atomic64_read(&log->cache.read_misses));
> +
> + return ret;
> +}
> +
> static int r5l_load_log(struct r5l_log *log)
> {
> struct md_rdev *rdev = log->rdev;
> @@ -1239,6 +1475,8 @@ int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev)
> INIT_LIST_HEAD(&log->no_space_stripes);
> spin_lock_init(&log->no_space_stripes_lock);
>
> + init_r5c_cache(conf, &log->cache);
> +
> if (r5l_load_log(log))
> goto error;
>
> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> index dc24b664..cdd9c4b 100644
> --- a/drivers/md/raid5.c
> +++ b/drivers/md/raid5.c
> @@ -503,7 +503,7 @@ retry:
> set_bit(STRIPE_BATCH_READY, &sh->state);
> }
>
> -static struct stripe_head *__find_stripe(struct r5conf *conf, sector_t sector,
> +struct stripe_head *__find_stripe(struct r5conf *conf, sector_t sector,
> short generation)
> {
> struct stripe_head *sh;
> @@ -515,6 +515,7 @@ static struct stripe_head *__find_stripe(struct r5conf *conf, sector_t sector,
> pr_debug("__stripe %llu not in cache\n", (unsigned long long)sector);
> return NULL;
> }
> +EXPORT_SYMBOL(__find_stripe);
Why are you exporting this? raid5-cache is not a separate module.
If you were going the export it the name would need to be changed to
include "md_".
NeilBrown
>
> /*
> * Need to check if array has failed when deciding whether to:
> @@ -4726,7 +4727,8 @@ static int raid5_read_one_chunk(struct mddev *mddev, struct bio *raid_bio)
> {
> struct r5conf *conf = mddev->private;
> int dd_idx;
> - struct bio* align_bi;
> + struct bio *align_bi;
> + struct bio *r5c_bio;
> struct md_rdev *rdev;
> sector_t end_sector;
>
> @@ -4734,6 +4736,18 @@ static int raid5_read_one_chunk(struct mddev *mddev, struct bio *raid_bio)
> pr_debug("%s: non aligned\n", __func__);
> return 0;
> }
> +
> + r5c_bio = r5c_lookup_chunk(conf->log, raid_bio);
> +
> + if (r5c_bio == NULL) {
> + pr_debug("Read all data from stripe cache\n");
> + return 1;
> + } else if (r5c_bio == raid_bio)
> + pr_debug("No data in stripe cache, read all from disk\n");
> + else {
> + pr_debug("Partial data in stripe cache, read and amend\n");
> + raid_bio = r5c_bio;
> + }
> /*
> * use bio_clone_mddev to make a copy of the bio
> */
> @@ -6157,6 +6171,10 @@ raid5_group_thread_cnt = __ATTR(group_thread_cnt, S_IRUGO | S_IWUSR,
> raid5_show_group_thread_cnt,
> raid5_store_group_thread_cnt);
>
> +static struct md_sysfs_entry
> +r5c_cache_stats = __ATTR(r5c_cache_stats, S_IRUGO,
> + r5c_stat_show, NULL);
> +
> static struct attribute *raid5_attrs[] = {
> &raid5_stripecache_size.attr,
> &raid5_stripecache_active.attr,
> @@ -6164,6 +6182,7 @@ static struct attribute *raid5_attrs[] = {
> &raid5_group_thread_cnt.attr,
> &raid5_skip_copy.attr,
> &raid5_rmw_level.attr,
> + &r5c_cache_stats.attr,
> NULL,
> };
> static struct attribute_group raid5_attrs_group = {
> diff --git a/drivers/md/raid5.h b/drivers/md/raid5.h
> index 3b68d4f..de11514 100644
> --- a/drivers/md/raid5.h
> +++ b/drivers/md/raid5.h
> @@ -690,4 +690,10 @@ extern void r5l_stripe_write_finished(struct stripe_head *sh);
> extern int r5l_handle_flush_request(struct r5l_log *log, struct bio *bio);
> extern void r5l_quiesce(struct r5l_log *log, int state);
> extern bool r5l_log_disk_error(struct r5conf *conf);
> +
> +extern struct bio *r5c_lookup_chunk(struct r5l_log *log, struct bio *raid_bio);
> +extern struct stripe_head *__find_stripe(struct r5conf *conf, sector_t sector,
> + short generation);
> +
> +extern ssize_t r5c_stat_show(struct mddev *mddev, char* page);
> #endif
> --
> 2.8.0.rc2
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: raid 5 crashed
From: Brad Campbell @ 2016-06-01 1:48 UTC (permalink / raw)
To: Wols Lists, Phil Turmel, bobzer; +Cc: linux-raid, Mikael Abrahamsson
In-Reply-To: <574DDCBD.40801@youngman.org.uk>
On 01/06/16 02:49, Wols Lists wrote:
> When the new drives arrive, copy each old drive in turn onto a new drive...
>
> dd if=/dev/sda of=/dev/sde
>
> etc etc. Expect it to take a little while ... (others may chime in and
> tell you to use ddrescue rather than dd - I don't know which one is
> best, both should work).
Do NOT use dd on a drive with bad sectors.... ever....
Using dd like you prescribe above will simply abort when it hits the
first read error. Using dd with the 'noerror' option will appear to work
and make you feel all warm and fuzzy, until you eventually realize that
when it encounters a read error, it skips that input block but does
*not* pad the output appropriately. So you wind up with everything after
the first read error in the wrong place on the disk. That will never end
well.
Use one of the ddrescue style of applications to guarantee everything
comes out where it needs to be regardless of input read errors.
Now, having said that :
Much better to try and get the array running in a read-only state with
all disks in place and clone the data from the array rather than the
disks after they've been ddrescued. In the case of a running array, a
read error on one of the array members will see the RAID attempt to get
the data from elsewhere (a reconstruction), whereas a read from a disc
cloned with ddrescue will happily just report what was a faulty sector
as a big pile of zeros, and *poof* your data is gone.
Set the timeouts appropriately (and conservatively) to give the disks
time to actually report they can't read the sector. This will allow md
to try and get it elsewhere rather than kicking the disc out because the
storage stack timed it out as faulty.
Brad.
^ permalink raw reply
* Re: RAID6 recovery with 6/9 drives out-of-sync
From: Phil Turmel @ 2016-05-31 19:19 UTC (permalink / raw)
To: Peckins, Steven E, linux-raid@vger.kernel.org
In-Reply-To: <C50C72C6-FC75-4863-B743-F63F09AF93B0@illinois.edu>
On 05/30/2016 10:43 PM, Peckins, Steven E wrote:
>
> I have a system with a 9+1 disk RAID6 array that has "3 drives and 1 spare - not enough to start the array." The metadata version is 1.1; mdadm version is v3.3.
>
> The component devices in the array are supposed to be multipath devices (dm-multipath), but for some reason, when the server was restarted, md grabbed both dm-* components and raw devices. I *think* that this is what caused the problem.
Quite possible. You probably need a DEVICES clause in your mdadm.conf
to exclude the raw devices from the arrays.
> I'm seeking advice on how to proceed at this point. If more information is required, please ask.
Hmmm. The partial success on mdadm --force suggests trying that again.
Possible with --force twice on the command line.
Forced assembly is precisely what you need -- don't despair and attempt
anything else.
Do review /proc/mdstat before each assembly attempt to make sure nothing
is partially assembled with those devices or the underlying raw devices.
Phil
^ permalink raw reply
* Re: raid 5 crashed
From: Wols Lists @ 2016-05-31 18:49 UTC (permalink / raw)
To: Phil Turmel, bobzer; +Cc: linux-raid, Mikael Abrahamsson
In-Reply-To: <574D958F.8060209@turmel.org>
On 31/05/16 14:45, Phil Turmel wrote:
> On 05/30/2016 06:00 PM, bobzer wrote:
>> I did follow his advice for the smartctl i can't right now but i will
>> post the output
>> my drives are desktop grade normal seagate, not NAS nor entreprise so
>> i will retry but i will limit the speed of reconstruction with :
>> echo 40000 > /proc/sys/dev/raid/speed_limit_max
>> echo 1000 > /proc/sys/dev/raid/speed_limit_min
>> I hope that it will permit to finish the rebuilding
>
> No, that is unlikely to help. You need to read about "timeout mismatch"
> as you clearly have a bad case of it. This is not a new issue, as you
> can see from the dates of the following:
>
> http://marc.info/?l=linux-raid&m=139050322510249&w=2
> http://marc.info/?l=linux-raid&m=135863964624202&w=2
> http://marc.info/?l=linux-raid&m=135811522817345&w=1
> http://marc.info/?l=linux-raid&m=133761065622164&w=2
> http://marc.info/?l=linux-raid&m=132477199207506
> http://marc.info/?l=linux-raid&m=133665797115876&w=2
> http://marc.info/?l=linux-raid&m=142487508806844&w=3
> http://marc.info/?l=linux-raid&m=144535576302583&w=2
>
And you need to follow Mikael's advice EVERY boot. It sounds like this
is your problem. So once you've managed to get your array reconstructed,
you need to replace your drives FAST.
In fact, with your setup, I'd be inclined to GIVE UP RIGHT NOW trying to
reconstruct the raid as-is. Get four replacement, NAS drives (they can
be 3TB drives if you want to increase the space, this technique should
work regardless...).
When the new drives arrive, copy each old drive in turn onto a new drive...
dd if=/dev/sda of=/dev/sde
etc etc. Expect it to take a little while ... (others may chime in and
tell you to use ddrescue rather than dd - I don't know which one is
best, both should work).
Once you've copied and replaced all four drives, your system should boot
and recover without difficulty. And if there IS a problem, at least you
still have the original drives UNTOUCHED.
Once you've got your system back, you should be able to claim the new
space if you did buy bigger drives. The old drives are probably fine -
you've just gone over the size limit at which fatal errors become a
probability for non-NAS drives.
I've got two 3TB Seagate Barracudas in a mirror. I can get away with a
mirror, I hope, but there's no way I'd go to raid 5 without proper NAS
drives (I'm a bit gutted - I originally bought the Barracudas intending
to do just that, but 3x3TB is just asking for trouble!)
Cheers,
Wol
^ permalink raw reply
* Re: [PATCH 0/2] Fixes for lots of arrays
From: Mike Lovell @ 2016-05-31 17:33 UTC (permalink / raw)
To: linux-raid, Jes Sorensen
In-Reply-To: <1463595794-17045-1-git-send-email-mlovell@bluehost.com>
ping. its been about 2 week since i posted these. just following up on it.
mike
On Wed, May 18, 2016 at 12:23 PM, Mike Lovell <mlovell@bluehost.com> wrote:
> This patch series fixes two issues around having more than 127 arrays on a
> system. The first one fixes an issue with using a dev_t as int and the
> number going negative when the array number would be larger than 2<<19. This
> would happen when more than 128 arrays were created on the system without
> creating the arrays by name. Manually specifying the large number would also
> fail.
>
> The second patch changes find_free_devnm in mdopen.c to use go to (2<<9)-1
> after 128 arrays have been created. Newer versions of the kernel don't allow
> the user to specify an array number than 511 so mdadm shouldn't automatically
> choose a bigger number. There was discussion about checking for new_array
> in /sys/module/md_mod/parameters on the list but that parameter has been in
> the kernel since 2.6.29. Any kernel from the last 7 years would still be
> limited by the check so it probably isn't worth a special case.
>
> Mike Lovell (2):
> Use dev_t for devnm2devid and devid2devnm
> Change behavior in find_free_devnm when wrapping around.
>
> Detail.c | 4 ++--
> Grow.c | 2 +-
> lib.c | 2 +-
> mapfile.c | 2 +-
> mdadm.h | 4 ++--
> mdopen.c | 6 +++---
> util.c | 6 +++---
> 7 files changed, 13 insertions(+), 13 deletions(-)
>
> --
> 1.9.1
>
^ permalink raw reply
* Re: raid 5 crashed
From: Phil Turmel @ 2016-05-31 13:45 UTC (permalink / raw)
To: bobzer, Anthonys Lists; +Cc: linux-raid, Mikael Abrahamsson
In-Reply-To: <CADzS=arbwpZg=VRcaiLaMJrLzr8pOjFTC-EwUtSpC1A3SmaTcQ@mail.gmail.com>
On 05/30/2016 06:00 PM, bobzer wrote:
> I did follow his advice for the smartctl i can't right now but i will
> post the output
> my drives are desktop grade normal seagate, not NAS nor entreprise so
> i will retry but i will limit the speed of reconstruction with :
> echo 40000 > /proc/sys/dev/raid/speed_limit_max
> echo 1000 > /proc/sys/dev/raid/speed_limit_min
> I hope that it will permit to finish the rebuilding
No, that is unlikely to help. You need to read about "timeout mismatch"
as you clearly have a bad case of it. This is not a new issue, as you
can see from the dates of the following:
http://marc.info/?l=linux-raid&m=139050322510249&w=2
http://marc.info/?l=linux-raid&m=135863964624202&w=2
http://marc.info/?l=linux-raid&m=135811522817345&w=1
http://marc.info/?l=linux-raid&m=133761065622164&w=2
http://marc.info/?l=linux-raid&m=132477199207506
http://marc.info/?l=linux-raid&m=133665797115876&w=2
http://marc.info/?l=linux-raid&m=142487508806844&w=3
http://marc.info/?l=linux-raid&m=144535576302583&w=2
^ permalink raw reply
* Re: [RFC 1/5] add bio_split_mddev
From: Guoqing Jiang @ 2016-05-31 9:13 UTC (permalink / raw)
To: Song Liu, linux-raid; +Cc: shli, nfbrown, dan.j.williams, hch, kernel-team
In-Reply-To: <1464326983-3798454-2-git-send-email-songliubraving@fb.com>
On 05/27/2016 01:29 AM, Song Liu wrote:
> similar to bio_clone_mddev, bio_alloc_mddev, this patch added
> bio_split_mddev, which uses a local bio set.
>
> Signed-off-by: Song Liu <songliubraving@fb.com>
> Signed-off-by: Shaohua Li <shli@fb.com>
> ---
> drivers/md/md.c | 14 +++++++++++---
> drivers/md/md.h | 2 ++
> drivers/md/raid5.c | 2 +-
> 3 files changed, 14 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index 866825f..f42f8d0 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -158,10 +158,9 @@ static const struct block_device_operations md_fops;
>
> static int start_readonly;
>
> -/* bio_clone_mddev
> - * like bio_clone, but with a local bio set
> +/* bio_alloc_mddev, bio_clone_mddev, bio_split_mddev
> + * like bio_alloc, bio_clone, bio_split, but with a local bio set
> */
> -
> struct bio *bio_alloc_mddev(gfp_t gfp_mask, int nr_iovecs,
> struct mddev *mddev)
> {
> @@ -187,6 +186,15 @@ struct bio *bio_clone_mddev(struct bio *bio, gfp_t gfp_mask,
> }
> EXPORT_SYMBOL_GPL(bio_clone_mddev);
>
> +struct bio *bio_split_mddev(struct bio *bio, int sectors,
> + gfp_t gfp, struct mddev *mddev)
> +{
> + if (!mddev || !mddev->bio_set)
> + return bio_split(bio, sectors, gfp, NULL);
> + return bio_split(bio, sectors, gfp, mddev->bio_set);
> +}
> +EXPORT_SYMBOL_GPL(bio_split_mddev);
> +
Compared with bio_alloc_mddev and bio_clone_mddev, there is no
bio_split_bioset
func, I think use bio_split directly is enough. Also why the last
parameter of the
first bio_split is NULL instead of fs_bio_set?
> /*
> * We have a system wide 'event count' that is incremented
> * on any 'interesting' event, and readers of /proc/mdstat
> diff --git a/drivers/md/md.h b/drivers/md/md.h
> index b5c4be7..9e1d4bf 100644
> --- a/drivers/md/md.h
> +++ b/drivers/md/md.h
> @@ -642,6 +642,8 @@ extern struct bio *bio_clone_mddev(struct bio *bio, gfp_t gfp_mask,
> struct mddev *mddev);
> extern struct bio *bio_alloc_mddev(gfp_t gfp_mask, int nr_iovecs,
> struct mddev *mddev);
> +extern struct bio *bio_split_mddev(struct bio *bio, int sectors,
> + gfp_t gfp, struct mddev *mddev);
>
> extern void md_unplug(struct blk_plug_cb *cb, bool from_schedule);
> extern void md_reload_sb(struct mddev *mddev, int raid_disk);
> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> index ad9e15a..8e25e67 100644
> --- a/drivers/md/raid5.c
> +++ b/drivers/md/raid5.c
> @@ -4871,7 +4871,7 @@ static struct bio *chunk_aligned_read(struct mddev *mddev, struct bio *raid_bio)
> unsigned sectors = chunk_sects - (sector & (chunk_sects-1));
>
> if (sectors < bio_sectors(raid_bio)) {
> - split = bio_split(raid_bio, sectors, GFP_NOIO, fs_bio_set);
The original place use fs_bio_set here.
> + split = bio_split_mddev(raid_bio, sectors, GFP_NOIO, mddev);
> bio_chain(split, raid_bio);
> } else
> split = raid_bio;
Thanks,
Guoqing
^ permalink raw reply
* Re: [RFC 4/5] r5cache: write part of r5cache
From: Guoqing Jiang @ 2016-05-31 9:00 UTC (permalink / raw)
To: Song Liu, linux-raid; +Cc: shli, nfbrown, dan.j.williams, hch, kernel-team
In-Reply-To: <1464326983-3798454-5-git-send-email-songliubraving@fb.com>
Hi,
I don't know a lot about raid5-cache, just a quick glance about syntax.
On 05/27/2016 01:29 AM, Song Liu wrote:
> This is the write part of r5cache. The cache is integrated with
> stripe cache of raid456. It leverages code of r5l_log to write
> data to journal device.
>
> r5cache split current write path into 2 parts: the write path
> and the reclaim path. The write path is as following:
> 1. write data to journal
> 2. call bio_endio
>
> Then the reclaim path is as:
> 1. Freeze the stripe (no more writes coming in)
> 2. Calcualte parity (reconstruct or RMW)
> 3. Write parity to journal device (data is already written to it)
> 4. Write data and parity to RAID disks
>
> With r5cache, write operation does not wait for parity calculation
> and write out, so the write latency is lower (1 write to journal
> device vs. read and then write to raid disks). Also, r5cache will
> reduce RAID overhead (multipile IO due to read-modify-write of
> parity) and provide more opportunities of full stripe writes.
>
> r5cache adds a new state of each stripe: enum r5c_states. The write
> path runs in state CLEAN and RUNNING (data in cache). Cache writes
> start from r5c_handle_stripe_dirtying(), where bit R5_Wantcache is
> set for devices with bio in towrite. Then, the data is written to
> the journal through r5l_log implementation. Once the data is in the
> journal, we set bit R5_InCache, and presue bio_endio for these
> writes.
>
> The reclaim path starts by freezing the stripe (no more writes).
> This stripes back to raid5 state machine, where
> handle_stripe_dirtying will evaluate the stripe for reconstruct
> writes or RMW writes (read data and calculate parity).
>
> For RMW, the code allocates extra page for prexor. Specifically,
> a new page is allocated for r5dev->page to do prexor; while
> r5dev->orig_page keeps the cached data. The extra page is freed
> after prexor.
>
> r5cache naturally excludes SkipCopy. With R5_Wantcache bit set,
> async_copy_data will not skip copy.
>
> Before writing data to RAID disks, the r5l_log logic stores
> parity (and non-overwrite data) to the journal.
>
> Instead of inactive_list, stripes with cached data are tracked in
> r5conf->r5c_cached_list. r5conf->r5c_cached_stripes tracks how
> many stripes has dirty data in the cache.
>
> Two sysfs entries are provided for the write cache:
> 1. r5c_cached_stripes shows how many stripes have cached data.
> Writing anything to r5c_cached_stripes will flush all data
> to RAID disks.
> 2. r5c_cache_mode provides knob to switch the system between
> write-back or write-through (only write-log).
>
> There are some known limitations of the cache implementation:
>
> 1. Write cache only covers full page writes (R5_OVERWRITE). Writes
> of smaller granularity are write through.
> 2. Only one log io (sh->log_io) for each stripe at anytime. Later
> writes for the same stripe have to wait. This can be improved by
> moving log_io to r5dev.
>
> Signed-off-by: Song Liu <songliubraving@fb.com>
> Signed-off-by: Shaohua Li <shli@fb.com>
> ---
> drivers/md/raid5-cache.c | 399 +++++++++++++++++++++++++++++++++++++++++++++--
> drivers/md/raid5.c | 172 +++++++++++++++++---
> drivers/md/raid5.h | 38 ++++-
> 3 files changed, 577 insertions(+), 32 deletions(-)
>
[snip]
> +
> +/*
> + * this journal write must contain full parity,
> + * it may also contain data of none-overwrites
> + */
> +static void r5c_handle_parity_cached(struct stripe_head *sh)
> +{
> + int i;
> +
> + for (i = sh->disks; i--; )
> + if (test_bit(R5_InCache, &sh->dev[i].flags))
> + set_bit(R5_Wantwrite, &sh->dev[i].flags);
> + r5c_set_state(sh, R5C_STATE_PARITY_DONE);
> +}
> +
> +static void r5c_finish_cache_stripe(struct stripe_head *sh)
> +{
> + switch (sh->r5c_state) {
> + case R5C_STATE_PARITY_RUN:
> + r5c_handle_parity_cached(sh);
> + break;
> + case R5C_STATE_CLEAN:
> + r5c_set_state(sh, R5C_STATE_RUNNING);
Maybe you missed break here?
> + case R5C_STATE_RUNNING:
> + r5c_handle_data_cached(sh);
> + break;
> + default:
> + BUG();
> + }
> +}
> +
[snip]
> meta_size =
> ((sizeof(struct r5l_payload_data_parity) + sizeof(__le32))
> * data_pages) +
> @@ -766,7 +865,6 @@ static void r5l_write_super_and_discard_space(struct r5l_log *log,
> }
> }
>
> -
> static void r5l_do_reclaim(struct r5l_log *log)
> {
> sector_t reclaim_target = xchg(&log->reclaim_target, 0);
> @@ -826,10 +924,15 @@ static void r5l_reclaim_thread(struct md_thread *thread)
>
> if (!log)
> return;
> +/*
> + spin_lock_irq(&conf->device_lock);
> + r5c_do_reclaim(conf);
> + spin_unlock_irq(&conf->device_lock);
> +*/
The above part doesn't make sense.
BTW: there are lots of issues reported by checkpatch.
Thanks,
Guoqing
^ permalink raw reply
* RAID6 recovery with 6/9 drives out-of-sync
From: Peckins, Steven E @ 2016-05-31 2:43 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
I have a system with a 9+1 disk RAID6 array that has "3 drives and 1 spare - not enough to start the array." The metadata version is 1.1; mdadm version is v3.3.
The component devices in the array are supposed to be multipath devices (dm-multipath), but for some reason, when the server was restarted, md grabbed both dm-* components and raw devices. I *think* that this is what caused the problem.
The output from "mdadm --examine" shows that the drives in this array have either 44 events (4 drives, including the spare) or 35 events (6 drives) and have an earlier "Updated Time." All components have a "clean" State, but the drives with later timestamps regard the other drives as missing (AAA......).
Six drives report this:
Update Time : Thu May 26 12:10:15 2016
Events : 35
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
Four drives report this:
Update Time : Thu May 26 15:44:23 2016
Events : 44
Array State : AAA...... ('A' == active, '.' == missing, 'R' == replacing)
I've used dd to duplicate all but one of the "missing" drives to other spares in the system prior to running any "forceful" mdadm commands on this array. One of the drives (dm-15) errored-out early in the process with what looks like a bad sector, but the others completed fine.
After making those copies, I ran mdadm --assemble --force, the best it could do was five drives: "/dev/md10 assembled from 5 drives and 1 spare - not enough to start the array."
Interestingly it says it cleared the "FAULTY" flag from two devices. But output from --examine showed all components as clean.
(There are five other 9+1 RAID6 arrays in this system, and they all came up without issue.)
I'm seeking advice on how to proceed at this point. If more information is required, please ask.
Output from --examine: http://pastebin.com/khvPWrba
Output from --assemble: http://pastebin.com/s2GkHkah
^ permalink raw reply
* Re: raid 5 crashed
From: bobzer @ 2016-05-30 22:00 UTC (permalink / raw)
To: Anthonys Lists; +Cc: linux-raid, Mikael Abrahamsson
In-Reply-To: <574C8EB9.3070706@youngman.org.uk>
I did follow his advice for the smartctl i can't right now but i will
post the output
my drives are desktop grade normal seagate, not NAS nor entreprise so
i will retry but i will limit the speed of reconstruction with :
echo 40000 > /proc/sys/dev/raid/speed_limit_max
echo 1000 > /proc/sys/dev/raid/speed_limit_min
I hope that it will permit to finish the rebuilding
On Mon, May 30, 2016 at 3:04 PM, Anthonys Lists
<antlists@youngman.org.uk> wrote:
> On 30/05/2016 16:01, bobzer wrote:
>>
>> HI,
>>
>> i did a dd_rescue which recovered a lot but not everything :
>> # Rescue Logfile. Created by GNU ddrescue version 1.19
>> # Command line: ddrescue -d -f -r3 /dev/sdf1
>> wdc/9bececcb-d520-ca38-fd88-d9565718e361.dd
>> wdc/9bececcb-d520-ca38-fd88-d9565718e361.mapfile
>> # Start time: 2016-05-29 22:32:38
>> # Current time: 2016-05-29 23:04:49
>> # Finished
>> # current_pos current_status
>> 0x129F7DB6000 +
>> # pos size status
>> 0x00000000 0x1118E2AB200 +
>> 0x1118E2AB200 0x00001000 -
>> 0x1118E2AC200 0x00090000 +
>> 0x1118E33C200 0x00001000 -
>> 0x1118E33D200 0x00817000 +
>> 0x1118EB54200 0x00001000 -
>> 0x1118EB55200 0x0016B000 +
>> 0x1118ECC0200 0x00001000 -
>> 0x1118ECC1200 0x641976000 +
>> 0x117D0637200 0x00001000 -
>> 0x117D0638200 0x000FE000 +
>> 0x117D0736200 0x00001000 -
>> 0x117D0737200 0x000EC000 +
>> 0x117D0823200 0x00001000 -
>> 0x117D0824200 0x0010C000 +
>> 0x117D0930200 0x00001000 -
>> 0x117D0931200 0x0010C000 +
>> 0x117D0A3D200 0x00001000 -
>> 0x117D0A3E200 0x00375000 +
>> 0x117D0DB3200 0x00001000 -
>> 0x117D0DB4200 0x0010B000 +
>> 0x117D0EBF200 0x00002000 -
>> 0x117D0EC1200 0x0010C000 +
>> 0x117D0FCD200 0x00001000 -
>> 0x117D0FCE200 0x001EB000 +
>> 0x117D11B9200 0x00001000 -
>> 0x117D11BA200 0x00112000 +
>> 0x117D12CC200 0x00001000 -
>> 0x117D12CD200 0x00077000 +
>> 0x117D1344200 0x00001000 -
>> 0x117D1345200 0x000EB000 +
>> 0x117D1430200 0x00001000 -
>> 0x117D1431200 0x000FD000 +
>> 0x117D152E200 0x00001000 -
>> 0x117D152F200 0x0010C000 +
>> 0x117D163B200 0x00001000 -
>> 0x117D163C200 0x00251000 +
>> 0x117D188D200 0x00002000 -
>> 0x117D188F200 0x12264B3000 +
>> 0x129F7D42200 0x00001000 -
>> 0x129F7D43200 0x0004F000 +
>> 0x129F7D92200 0x00001000 -
>> 0x129F7D93200 0x00022000 +
>> 0x129F7DB5200 0x00001000 -
>> 0x129F7DB6200 0xA7C90DA200 +
>>
>> i don't know if i can recover more but i tried to reassemble the raid
>> and it work but during the rebuilding the sdb1 failed again
>> so my data are there but i'm not sure to know what to do to get them all.
>>
>> thanks
>>
>>
> Did you follow Mikael's advice? Can you do smartctl on the drive (I think
> the option you want is "smartctl -x" - display "extended all") and post the
> output?
>
> We're assuming you're using proper raid-capable drives, but if you aren't
> then the drives could be fine but giving up under load and causing your raid
> problems. On the other hand, if your drives are proper raid drives, then
> they're probably not fit for anything more than the scrapheap.
>
> Cheers,
> Wol
^ permalink raw reply
* Re: raid 5 crashed
From: Anthonys Lists @ 2016-05-30 19:04 UTC (permalink / raw)
To: bobzer, linux-raid, Mikael Abrahamsson
In-Reply-To: <CADzS=are4kkK5Lpm2WTAnqhE1D3XuMFwTKbO8_FzDbGtmqu2Dw@mail.gmail.com>
On 30/05/2016 16:01, bobzer wrote:
> HI,
>
> i did a dd_rescue which recovered a lot but not everything :
> # Rescue Logfile. Created by GNU ddrescue version 1.19
> # Command line: ddrescue -d -f -r3 /dev/sdf1
> wdc/9bececcb-d520-ca38-fd88-d9565718e361.dd
> wdc/9bececcb-d520-ca38-fd88-d9565718e361.mapfile
> # Start time: 2016-05-29 22:32:38
> # Current time: 2016-05-29 23:04:49
> # Finished
> # current_pos current_status
> 0x129F7DB6000 +
> # pos size status
> 0x00000000 0x1118E2AB200 +
> 0x1118E2AB200 0x00001000 -
> 0x1118E2AC200 0x00090000 +
> 0x1118E33C200 0x00001000 -
> 0x1118E33D200 0x00817000 +
> 0x1118EB54200 0x00001000 -
> 0x1118EB55200 0x0016B000 +
> 0x1118ECC0200 0x00001000 -
> 0x1118ECC1200 0x641976000 +
> 0x117D0637200 0x00001000 -
> 0x117D0638200 0x000FE000 +
> 0x117D0736200 0x00001000 -
> 0x117D0737200 0x000EC000 +
> 0x117D0823200 0x00001000 -
> 0x117D0824200 0x0010C000 +
> 0x117D0930200 0x00001000 -
> 0x117D0931200 0x0010C000 +
> 0x117D0A3D200 0x00001000 -
> 0x117D0A3E200 0x00375000 +
> 0x117D0DB3200 0x00001000 -
> 0x117D0DB4200 0x0010B000 +
> 0x117D0EBF200 0x00002000 -
> 0x117D0EC1200 0x0010C000 +
> 0x117D0FCD200 0x00001000 -
> 0x117D0FCE200 0x001EB000 +
> 0x117D11B9200 0x00001000 -
> 0x117D11BA200 0x00112000 +
> 0x117D12CC200 0x00001000 -
> 0x117D12CD200 0x00077000 +
> 0x117D1344200 0x00001000 -
> 0x117D1345200 0x000EB000 +
> 0x117D1430200 0x00001000 -
> 0x117D1431200 0x000FD000 +
> 0x117D152E200 0x00001000 -
> 0x117D152F200 0x0010C000 +
> 0x117D163B200 0x00001000 -
> 0x117D163C200 0x00251000 +
> 0x117D188D200 0x00002000 -
> 0x117D188F200 0x12264B3000 +
> 0x129F7D42200 0x00001000 -
> 0x129F7D43200 0x0004F000 +
> 0x129F7D92200 0x00001000 -
> 0x129F7D93200 0x00022000 +
> 0x129F7DB5200 0x00001000 -
> 0x129F7DB6200 0xA7C90DA200 +
>
> i don't know if i can recover more but i tried to reassemble the raid
> and it work but during the rebuilding the sdb1 failed again
> so my data are there but i'm not sure to know what to do to get them all.
>
> thanks
>
>
Did you follow Mikael's advice? Can you do smartctl on the drive (I
think the option you want is "smartctl -x" - display "extended all") and
post the output?
We're assuming you're using proper raid-capable drives, but if you
aren't then the drives could be fine but giving up under load and
causing your raid problems. On the other hand, if your drives are proper
raid drives, then they're probably not fit for anything more than the
scrapheap.
Cheers,
Wol
^ permalink raw reply
* [PATCH 1/1] md/bitmap.c:bitmap_status(): Fix filename escaping
From: Nominal Animal @ 2016-05-30 18:00 UTC (permalink / raw)
To: Shaohua Li, linux-raid, linux-kernel
The call to seq_file_path() does not include backslash itself
in the set of file names escaped. This results in files named
"foo\t-\nbar" and "foo\\011-\\012bar" being both shown
as "foo\\011-\\012bar". Fix this by including backslash
in the escaped set.
diff -Nabpur linux-4.6/drivers/md/bitmap.c linux-new/drivers/md/bitmap.c
--- linux-4.6/drivers/md/bitmap.c 2016-05-16 01:43:13.000000000 +0300
+++ linux-new/drivers/md/bitmap.c 2016-05-30 19:56:43.791298226 +0300
@@ -1928,7 +1928,7 @@ void bitmap_status(struct seq_file *seq,
chunk_kb ? "KB" : "B");
if (bitmap->storage.file) {
seq_printf(seq, ", file: ");
- seq_file_path(seq, bitmap->storage.file, " \t\n");
+ seq_file_path(seq, bitmap->storage.file, " \t\n\\");
}
seq_printf(seq, "\n");
Regards,
Nominal Animal
^ permalink raw reply
* Re: raid 5 crashed
From: bobzer @ 2016-05-30 15:01 UTC (permalink / raw)
To: linux-raid, Robin Hill, Mikael Abrahamsson
In-Reply-To: <CADzS=apUB=XZRmCWjpOHuRgvk7EhqrbpRQhHQ+MHf8NHyGCU0w@mail.gmail.com>
HI,
i did a dd_rescue which recovered a lot but not everything :
# Rescue Logfile. Created by GNU ddrescue version 1.19
# Command line: ddrescue -d -f -r3 /dev/sdf1
wdc/9bececcb-d520-ca38-fd88-d9565718e361.dd
wdc/9bececcb-d520-ca38-fd88-d9565718e361.mapfile
# Start time: 2016-05-29 22:32:38
# Current time: 2016-05-29 23:04:49
# Finished
# current_pos current_status
0x129F7DB6000 +
# pos size status
0x00000000 0x1118E2AB200 +
0x1118E2AB200 0x00001000 -
0x1118E2AC200 0x00090000 +
0x1118E33C200 0x00001000 -
0x1118E33D200 0x00817000 +
0x1118EB54200 0x00001000 -
0x1118EB55200 0x0016B000 +
0x1118ECC0200 0x00001000 -
0x1118ECC1200 0x641976000 +
0x117D0637200 0x00001000 -
0x117D0638200 0x000FE000 +
0x117D0736200 0x00001000 -
0x117D0737200 0x000EC000 +
0x117D0823200 0x00001000 -
0x117D0824200 0x0010C000 +
0x117D0930200 0x00001000 -
0x117D0931200 0x0010C000 +
0x117D0A3D200 0x00001000 -
0x117D0A3E200 0x00375000 +
0x117D0DB3200 0x00001000 -
0x117D0DB4200 0x0010B000 +
0x117D0EBF200 0x00002000 -
0x117D0EC1200 0x0010C000 +
0x117D0FCD200 0x00001000 -
0x117D0FCE200 0x001EB000 +
0x117D11B9200 0x00001000 -
0x117D11BA200 0x00112000 +
0x117D12CC200 0x00001000 -
0x117D12CD200 0x00077000 +
0x117D1344200 0x00001000 -
0x117D1345200 0x000EB000 +
0x117D1430200 0x00001000 -
0x117D1431200 0x000FD000 +
0x117D152E200 0x00001000 -
0x117D152F200 0x0010C000 +
0x117D163B200 0x00001000 -
0x117D163C200 0x00251000 +
0x117D188D200 0x00002000 -
0x117D188F200 0x12264B3000 +
0x129F7D42200 0x00001000 -
0x129F7D43200 0x0004F000 +
0x129F7D92200 0x00001000 -
0x129F7D93200 0x00022000 +
0x129F7DB5200 0x00001000 -
0x129F7DB6200 0xA7C90DA200 +
i don't know if i can recover more but i tried to reassemble the raid
and it work but during the rebuilding the sdb1 failed again
so my data are there but i'm not sure to know what to do to get them all.
thanks
On Fri, May 27, 2016 at 3:19 PM, bobzer <bobzer@gmail.com> wrote:
> hi,
>
> I'm afraid to make the problem worst but i received a new HD to do a
> dd_rescue :-)
> I'm ready to buy another HD but the problem is that i don't know
> what's the best to recover my data
>
> My question is : There is a way to test if the data/raid is ok without
> take the risk of losting anything more ?
>
> help me please :-(
>
> best regards
> Mathieu
>
> On Wed, May 25, 2016 at 11:06 PM, bobzer <bobzer@gmail.com> wrote:
>> thanks for your help.
>> i took time to answer because unlucky me, the power supply of my
>> laptop fried so no laptop and no way to work on my raid :-(
>> any way i got a new one :-)
>>
>> for the dmesg i paste it here : http://pastebin.com/whUHs256
>>
>> root@serveur:~# uname -a
>> Linux serveur 3.2.0-4-amd64 #1 SMP Debian 3.2.63-2+deb7u1 x86_64 GNU/Linux
>> root@serveur:~# mdadm -V
>> mdadm - v3.3-78-gf43f5b3 - 02nd avril 2014
>>
>> about zero the superblock on the wrong device, i hope i didn't do
>> that, but also i really don't think i did that, because i took care
>> and at that time the raid was working
>>
>> i don't know what to do, if i use dd_rescue and can't get back 100% of
>> the data could i be able to start the raid anyway ?
>> what are my risk if i try something like :
>> mdadm --assemble --force /dev/md0 /dev/sd[bcde]1
>>
>> thank you very much for your time
>> Mathieu
>>
>>
>> On Wed, May 11, 2016 at 3:15 PM, Robin Hill <robin@robinhill.me.uk> wrote:
>>> On Tue May 10, 2016 at 11:28:31PM +0200, bobzer wrote:
>>>
>>>> hi everyone,
>>>>
>>>> I'm in panic mode :-( because i got a raid 5 with 4 disk but 2 removed
>>>> yesterday i got a power outage which removed one disk. the disks
>>>> sd[bcd]1 was ok and saying that sde1 is removed but sde1 said that
>>>> everything is fine.
>>>> so i stop the raid, zero the superblock of sde1, start the raid and
>>>> add sde1 to the raid. then it start to reconstruct, i think it had
>>>> time to finish before this problem (i'm not 100% sure that it finish
>>>> but i think so)
>>>> the data was accessible so i went to sleep
>>>> today i discovered the raid in this state :
>>>>
>>>> root@serveur:/home/math# mdadm -D /dev/md0
>>>> /dev/md0:
>>>> Version : 1.2
>>>> Creation Time : Sun Mar 4 22:49:14 2012
>>>> Raid Level : raid5
>>>> Array Size : 5860532352 (5589.04 GiB 6001.19 GB)
>>>> Used Dev Size : 1953510784 (1863.01 GiB 2000.40 GB)
>>>> Raid Devices : 4
>>>> Total Devices : 4
>>>> Persistence : Superblock is persistent
>>>>
>>>> Update Time : Fri May 6 17:44:02 2016
>>>> State : clean, FAILED
>>>> Active Devices : 2
>>>> Working Devices : 3
>>>> Failed Devices : 1
>>>> Spare Devices : 1
>>>>
>>>> Layout : left-symmetric
>>>> Chunk Size : 128K
>>>>
>>>> Name : debian:0
>>>> UUID : bf3c605b:9699aa55:d45119a2:7ba58d56
>>>> Events : 892482
>>>>
>>>> Number Major Minor RaidDevice State
>>>> 3 8 33 0 active sync /dev/sdc1
>>>> 1 8 49 1 active sync /dev/sdd1
>>>> 4 0 0 4 removed
>>>> 6 0 0 6 removed
>>>>
>>>> 4 8 17 - faulty /dev/sdb1
>>>> 5 8 65 - spare /dev/sde1
>>>>
>>> So this reports /dev/sdb1 faulty and /dev/sde1 spare. That would
>>> indicate that the rebuild hadn't finished.
>>>
>>>> root@serveur:/home/math# mdadm --examine /dev/sdb1
>>>> /dev/sdb1:
>>>> Magic : a92b4efc
>>>> Version : 1.2
>>>> Feature Map : 0x0
>>>> Array UUID : bf3c605b:9699aa55:d45119a2:7ba58d56
>>>> Name : debian:0
>>>> Creation Time : Sun Mar 4 22:49:14 2012
>>>> Raid Level : raid5
>>>> Raid Devices : 4
>>>>
>>>> Avail Dev Size : 3907021954 (1863.01 GiB 2000.40 GB)
>>>> Array Size : 5860532352 (5589.04 GiB 6001.19 GB)
>>>> Used Dev Size : 3907021568 (1863.01 GiB 2000.40 GB)
>>>> Data Offset : 2048 sectors
>>>> Super Offset : 8 sectors
>>>> Unused Space : before=1960 sectors, after=386 sectors
>>>> State : clean
>>>> Device UUID : 9bececcb:d520ca38:fd88d956:5718e361
>>>>
>>>> Update Time : Fri May 6 02:07:00 2016
>>>> Bad Block Log : 512 entries available at offset 72 sectors
>>>> Checksum : dc2a133a - correct
>>>> Events : 892215
>>>>
>>>> Layout : left-symmetric
>>>> Chunk Size : 128K
>>>>
>>>> Device Role : Active device 2
>>>> Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
>>>>
>>> We can see /dev/sdb1 has a lower event count than the others and also
>>> that it indicates all the drives in the array were active when it was
>>> last running. That would strongly suggest that it was not in the array
>>> when /dev/sde1 was added to rebuild. The update time is also nearly 16
>>> hours earlier than that of the other drives.
>>>
>>>> root@serveur:/home/math# mdadm --examine /dev/sdc1
>>>> /dev/sdc1:
>>>> Magic : a92b4efc
>>>> Version : 1.2
>>>> Feature Map : 0x0
>>>> Array UUID : bf3c605b:9699aa55:d45119a2:7ba58d56
>>>> Name : debian:0
>>>> Creation Time : Sun Mar 4 22:49:14 2012
>>>> Raid Level : raid5
>>>> Raid Devices : 4
>>>>
>>>> Avail Dev Size : 3907021954 (1863.01 GiB 2000.40 GB)
>>>> Array Size : 5860532352 (5589.04 GiB 6001.19 GB)
>>>> Used Dev Size : 3907021568 (1863.01 GiB 2000.40 GB)
>>>> Data Offset : 2048 sectors
>>>> Super Offset : 8 sectors
>>>> Unused Space : before=1960 sectors, after=386 sectors
>>>> State : clean
>>>> Device UUID : 1ecaf51c:3289a902:7bb71a93:237c68e8
>>>>
>>>> Update Time : Fri May 6 17:58:27 2016
>>>> Bad Block Log : 512 entries available at offset 72 sectors
>>>> Checksum : b9d6aa84 - correct
>>>> Events : 892484
>>>>
>>>> Layout : left-symmetric
>>>> Chunk Size : 128K
>>>>
>>>> Device Role : Active device 0
>>>> Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
>>>>
>>>> root@serveur:/home/math# mdadm --examine /dev/sdd1
>>>> /dev/sdd1:
>>>> Magic : a92b4efc
>>>> Version : 1.2
>>>> Feature Map : 0x0
>>>> Array UUID : bf3c605b:9699aa55:d45119a2:7ba58d56
>>>> Name : debian:0
>>>> Creation Time : Sun Mar 4 22:49:14 2012
>>>> Raid Level : raid5
>>>> Raid Devices : 4
>>>>
>>>> Avail Dev Size : 3907021954 (1863.01 GiB 2000.40 GB)
>>>> Array Size : 5860532352 (5589.04 GiB 6001.19 GB)
>>>> Used Dev Size : 3907021568 (1863.01 GiB 2000.40 GB)
>>>> Data Offset : 2048 sectors
>>>> Super Offset : 8 sectors
>>>> Unused Space : before=0 sectors, after=386 sectors
>>>> State : clean
>>>> Device UUID : 406c4cb5:c188e4a9:7ed8be9f:14a49b16
>>>>
>>>> Update Time : Fri May 6 17:58:27 2016
>>>> Bad Block Log : 512 entries available at offset 2032 sectors
>>>> Checksum : 343f9cd0 - correct
>>>> Events : 892484
>>>>
>>>> Layout : left-symmetric
>>>> Chunk Size : 128K
>>>>
>>>> Device Role : Active device 1
>>>> Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
>>>>
>>> These two drives contain the same information. They indicate that they
>>> were the only 2 running members in the array when they were last updated.
>>>
>>>> root@serveur:/home/math# mdadm --examine /dev/sde1
>>>> /dev/sde1:
>>>> Magic : a92b4efc
>>>> Version : 1.2
>>>> Feature Map : 0x8
>>>> Array UUID : bf3c605b:9699aa55:d45119a2:7ba58d56
>>>> Name : debian:0
>>>> Creation Time : Sun Mar 4 22:49:14 2012
>>>> Raid Level : raid5
>>>> Raid Devices : 4
>>>>
>>>> Avail Dev Size : 3907025072 (1863.01 GiB 2000.40 GB)
>>>> Array Size : 5860532352 (5589.04 GiB 6001.19 GB)
>>>> Used Dev Size : 3907021568 (1863.01 GiB 2000.40 GB)
>>>> Data Offset : 2048 sectors
>>>> Super Offset : 8 sectors
>>>> Unused Space : before=1960 sectors, after=3504 sectors
>>>> State : clean
>>>> Device UUID : f2e9c1ec:2852cf21:1a588581:b9f49a8b
>>>>
>>>> Update Time : Fri May 6 17:58:27 2016
>>>> Bad Block Log : 512 entries available at offset 72 sectors - bad
>>>> blocks present.
>>>> Checksum : 3a65b8bc - correct
>>>> Events : 892484
>>>>
>>>> Layout : left-symmetric
>>>> Chunk Size : 128K
>>>>
>>>> Device Role : spare
>>>> Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
>>>>
>>> And finally /dev/sde1 shows as a spare, with the rest of the data
>>> matching /dev/sdc1 and /dev/sde1.
>>>
>>>> PLEASE help me :-) i don't know what to do so i did nothing to not do
>>>> any stupid things
>>>> 1000 thank you
>>>>
>>>> ps i just saw this, i hope it not mak y case worst
>>>> root@serveur:/home/math# cat /etc/mdadm/mdadm.conf
>>>> DEVICE /dev/sd[bcd]1
>>>> ARRAY /dev/md0 metadata=1.2 name=debian:0
>>>> UUID=bf3c605b:9699aa55:d45119a2:7ba58d56
>>>>
>>>
>>> From the data here, if looks to me as though /dev/sdb1 failed originally
>>> (hence it thinks the array was complete). Either then /dev/sde1 also
>>> failed, or you've proceeded to zero the superblock on the wrong drive.
>>> You really need to look through the system logs and verify what happened
>>> when and to what disk (if you rebooted at any point, the drive ordering
>>> may have changed, so don't take for granted that the drive names are
>>> consistent throughout).
>>>
>>> Cheers,
>>> Robin
>>> --
>>> ___
>>> ( ' } | Robin Hill <robin@robinhill.me.uk> |
>>> / / ) | Little Jim says .... |
>>> // !! | "He fallen in de water !!" |
On Fri, May 27, 2016 at 9:19 PM, bobzer <bobzer@gmail.com> wrote:
> hi,
>
> I'm afraid to make the problem worst but i received a new HD to do a
> dd_rescue :-)
> I'm ready to buy another HD but the problem is that i don't know
> what's the best to recover my data
>
> My question is : There is a way to test if the data/raid is ok without
> take the risk of losting anything more ?
>
> help me please :-(
>
> best regards
> Mathieu
>
> On Wed, May 25, 2016 at 11:06 PM, bobzer <bobzer@gmail.com> wrote:
>> thanks for your help.
>> i took time to answer because unlucky me, the power supply of my
>> laptop fried so no laptop and no way to work on my raid :-(
>> any way i got a new one :-)
>>
>> for the dmesg i paste it here : http://pastebin.com/whUHs256
>>
>> root@serveur:~# uname -a
>> Linux serveur 3.2.0-4-amd64 #1 SMP Debian 3.2.63-2+deb7u1 x86_64 GNU/Linux
>> root@serveur:~# mdadm -V
>> mdadm - v3.3-78-gf43f5b3 - 02nd avril 2014
>>
>> about zero the superblock on the wrong device, i hope i didn't do
>> that, but also i really don't think i did that, because i took care
>> and at that time the raid was working
>>
>> i don't know what to do, if i use dd_rescue and can't get back 100% of
>> the data could i be able to start the raid anyway ?
>> what are my risk if i try something like :
>> mdadm --assemble --force /dev/md0 /dev/sd[bcde]1
>>
>> thank you very much for your time
>> Mathieu
>>
>>
>> On Wed, May 11, 2016 at 3:15 PM, Robin Hill <robin@robinhill.me.uk> wrote:
>>> On Tue May 10, 2016 at 11:28:31PM +0200, bobzer wrote:
>>>
>>>> hi everyone,
>>>>
>>>> I'm in panic mode :-( because i got a raid 5 with 4 disk but 2 removed
>>>> yesterday i got a power outage which removed one disk. the disks
>>>> sd[bcd]1 was ok and saying that sde1 is removed but sde1 said that
>>>> everything is fine.
>>>> so i stop the raid, zero the superblock of sde1, start the raid and
>>>> add sde1 to the raid. then it start to reconstruct, i think it had
>>>> time to finish before this problem (i'm not 100% sure that it finish
>>>> but i think so)
>>>> the data was accessible so i went to sleep
>>>> today i discovered the raid in this state :
>>>>
>>>> root@serveur:/home/math# mdadm -D /dev/md0
>>>> /dev/md0:
>>>> Version : 1.2
>>>> Creation Time : Sun Mar 4 22:49:14 2012
>>>> Raid Level : raid5
>>>> Array Size : 5860532352 (5589.04 GiB 6001.19 GB)
>>>> Used Dev Size : 1953510784 (1863.01 GiB 2000.40 GB)
>>>> Raid Devices : 4
>>>> Total Devices : 4
>>>> Persistence : Superblock is persistent
>>>>
>>>> Update Time : Fri May 6 17:44:02 2016
>>>> State : clean, FAILED
>>>> Active Devices : 2
>>>> Working Devices : 3
>>>> Failed Devices : 1
>>>> Spare Devices : 1
>>>>
>>>> Layout : left-symmetric
>>>> Chunk Size : 128K
>>>>
>>>> Name : debian:0
>>>> UUID : bf3c605b:9699aa55:d45119a2:7ba58d56
>>>> Events : 892482
>>>>
>>>> Number Major Minor RaidDevice State
>>>> 3 8 33 0 active sync /dev/sdc1
>>>> 1 8 49 1 active sync /dev/sdd1
>>>> 4 0 0 4 removed
>>>> 6 0 0 6 removed
>>>>
>>>> 4 8 17 - faulty /dev/sdb1
>>>> 5 8 65 - spare /dev/sde1
>>>>
>>> So this reports /dev/sdb1 faulty and /dev/sde1 spare. That would
>>> indicate that the rebuild hadn't finished.
>>>
>>>> root@serveur:/home/math# mdadm --examine /dev/sdb1
>>>> /dev/sdb1:
>>>> Magic : a92b4efc
>>>> Version : 1.2
>>>> Feature Map : 0x0
>>>> Array UUID : bf3c605b:9699aa55:d45119a2:7ba58d56
>>>> Name : debian:0
>>>> Creation Time : Sun Mar 4 22:49:14 2012
>>>> Raid Level : raid5
>>>> Raid Devices : 4
>>>>
>>>> Avail Dev Size : 3907021954 (1863.01 GiB 2000.40 GB)
>>>> Array Size : 5860532352 (5589.04 GiB 6001.19 GB)
>>>> Used Dev Size : 3907021568 (1863.01 GiB 2000.40 GB)
>>>> Data Offset : 2048 sectors
>>>> Super Offset : 8 sectors
>>>> Unused Space : before=1960 sectors, after=386 sectors
>>>> State : clean
>>>> Device UUID : 9bececcb:d520ca38:fd88d956:5718e361
>>>>
>>>> Update Time : Fri May 6 02:07:00 2016
>>>> Bad Block Log : 512 entries available at offset 72 sectors
>>>> Checksum : dc2a133a - correct
>>>> Events : 892215
>>>>
>>>> Layout : left-symmetric
>>>> Chunk Size : 128K
>>>>
>>>> Device Role : Active device 2
>>>> Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
>>>>
>>> We can see /dev/sdb1 has a lower event count than the others and also
>>> that it indicates all the drives in the array were active when it was
>>> last running. That would strongly suggest that it was not in the array
>>> when /dev/sde1 was added to rebuild. The update time is also nearly 16
>>> hours earlier than that of the other drives.
>>>
>>>> root@serveur:/home/math# mdadm --examine /dev/sdc1
>>>> /dev/sdc1:
>>>> Magic : a92b4efc
>>>> Version : 1.2
>>>> Feature Map : 0x0
>>>> Array UUID : bf3c605b:9699aa55:d45119a2:7ba58d56
>>>> Name : debian:0
>>>> Creation Time : Sun Mar 4 22:49:14 2012
>>>> Raid Level : raid5
>>>> Raid Devices : 4
>>>>
>>>> Avail Dev Size : 3907021954 (1863.01 GiB 2000.40 GB)
>>>> Array Size : 5860532352 (5589.04 GiB 6001.19 GB)
>>>> Used Dev Size : 3907021568 (1863.01 GiB 2000.40 GB)
>>>> Data Offset : 2048 sectors
>>>> Super Offset : 8 sectors
>>>> Unused Space : before=1960 sectors, after=386 sectors
>>>> State : clean
>>>> Device UUID : 1ecaf51c:3289a902:7bb71a93:237c68e8
>>>>
>>>> Update Time : Fri May 6 17:58:27 2016
>>>> Bad Block Log : 512 entries available at offset 72 sectors
>>>> Checksum : b9d6aa84 - correct
>>>> Events : 892484
>>>>
>>>> Layout : left-symmetric
>>>> Chunk Size : 128K
>>>>
>>>> Device Role : Active device 0
>>>> Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
>>>>
>>>> root@serveur:/home/math# mdadm --examine /dev/sdd1
>>>> /dev/sdd1:
>>>> Magic : a92b4efc
>>>> Version : 1.2
>>>> Feature Map : 0x0
>>>> Array UUID : bf3c605b:9699aa55:d45119a2:7ba58d56
>>>> Name : debian:0
>>>> Creation Time : Sun Mar 4 22:49:14 2012
>>>> Raid Level : raid5
>>>> Raid Devices : 4
>>>>
>>>> Avail Dev Size : 3907021954 (1863.01 GiB 2000.40 GB)
>>>> Array Size : 5860532352 (5589.04 GiB 6001.19 GB)
>>>> Used Dev Size : 3907021568 (1863.01 GiB 2000.40 GB)
>>>> Data Offset : 2048 sectors
>>>> Super Offset : 8 sectors
>>>> Unused Space : before=0 sectors, after=386 sectors
>>>> State : clean
>>>> Device UUID : 406c4cb5:c188e4a9:7ed8be9f:14a49b16
>>>>
>>>> Update Time : Fri May 6 17:58:27 2016
>>>> Bad Block Log : 512 entries available at offset 2032 sectors
>>>> Checksum : 343f9cd0 - correct
>>>> Events : 892484
>>>>
>>>> Layout : left-symmetric
>>>> Chunk Size : 128K
>>>>
>>>> Device Role : Active device 1
>>>> Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
>>>>
>>> These two drives contain the same information. They indicate that they
>>> were the only 2 running members in the array when they were last updated.
>>>
>>>> root@serveur:/home/math# mdadm --examine /dev/sde1
>>>> /dev/sde1:
>>>> Magic : a92b4efc
>>>> Version : 1.2
>>>> Feature Map : 0x8
>>>> Array UUID : bf3c605b:9699aa55:d45119a2:7ba58d56
>>>> Name : debian:0
>>>> Creation Time : Sun Mar 4 22:49:14 2012
>>>> Raid Level : raid5
>>>> Raid Devices : 4
>>>>
>>>> Avail Dev Size : 3907025072 (1863.01 GiB 2000.40 GB)
>>>> Array Size : 5860532352 (5589.04 GiB 6001.19 GB)
>>>> Used Dev Size : 3907021568 (1863.01 GiB 2000.40 GB)
>>>> Data Offset : 2048 sectors
>>>> Super Offset : 8 sectors
>>>> Unused Space : before=1960 sectors, after=3504 sectors
>>>> State : clean
>>>> Device UUID : f2e9c1ec:2852cf21:1a588581:b9f49a8b
>>>>
>>>> Update Time : Fri May 6 17:58:27 2016
>>>> Bad Block Log : 512 entries available at offset 72 sectors - bad
>>>> blocks present.
>>>> Checksum : 3a65b8bc - correct
>>>> Events : 892484
>>>>
>>>> Layout : left-symmetric
>>>> Chunk Size : 128K
>>>>
>>>> Device Role : spare
>>>> Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
>>>>
>>> And finally /dev/sde1 shows as a spare, with the rest of the data
>>> matching /dev/sdc1 and /dev/sde1.
>>>
>>>> PLEASE help me :-) i don't know what to do so i did nothing to not do
>>>> any stupid things
>>>> 1000 thank you
>>>>
>>>> ps i just saw this, i hope it not mak y case worst
>>>> root@serveur:/home/math# cat /etc/mdadm/mdadm.conf
>>>> DEVICE /dev/sd[bcd]1
>>>> ARRAY /dev/md0 metadata=1.2 name=debian:0
>>>> UUID=bf3c605b:9699aa55:d45119a2:7ba58d56
>>>>
>>>
>>> From the data here, if looks to me as though /dev/sdb1 failed originally
>>> (hence it thinks the array was complete). Either then /dev/sde1 also
>>> failed, or you've proceeded to zero the superblock on the wrong drive.
>>> You really need to look through the system logs and verify what happened
>>> when and to what disk (if you rebooted at any point, the drive ordering
>>> may have changed, so don't take for granted that the drive names are
>>> consistent throughout).
>>>
>>> Cheers,
>>> Robin
>>> --
>>> ___
>>> ( ' } | Robin Hill <robin@robinhill.me.uk> |
>>> / / ) | Little Jim says .... |
>>> // !! | "He fallen in de water !!" |
^ permalink raw reply
* Why does raid0 set max_hw_sectors as chunk size but the other raid types doesn't?
From: Joey Liao @ 2016-05-30 11:55 UTC (permalink / raw)
To: linux-raid
Hi,
I have no idea why does raid0_run() in raid0.c use
blk_queue_max_hw_sectors() to set max_hw_sectors as the chunk size,
but the other raid types doesn't?
What's the purpose to limit the max_hw_sectors in raid0?
Is it related to the source code logic issue or the performance issue?
Besides, I have an interesting observation. If I remove all the
following queue limitation codes in raid0_run() of raid0.c, the block
size in iostat is still the same as chunk size even the input block
size is much larger than the chunk size. Why???
- blk_queue_max_hw_sectors(mddev->queue, mddev->chunk_sectors);
- blk_queue_max_write_same_sectors(mddev->queue, mddev->chunk_sectors);
+ //blk_queue_max_hw_sectors(mddev->queue, mddev->chunk_sectors);
+ //blk_queue_max_write_same_sectors(mddev->queue, mddev->chunk_sectors);
- blk_queue_io_min(mddev->queue, mddev->chunk_sectors << 9);
- blk_queue_io_opt(mddev->queue,
- (mddev->chunk_sectors << 9) * mddev->raid_disks);
+ //blk_queue_io_min(mddev->queue, mddev->chunk_sectors << 9);
+ /*blk_queue_io_opt(mddev->queue,
+ (mddev->chunk_sectors << 9) * mddev->raid_disks);*/
^ permalink raw reply
* ﹢﹣Hello
From: hi @ 2016-05-29 8:42 UTC (permalink / raw)
To: mariaferrazzi
Hello
TV, camera, laptop, bike, dj, iphone, samsung product ..... 50% discount for all our products
www. slooone .com
^ permalink raw reply
* Hello
From: program12 @ 2016-05-29 0:45 UTC (permalink / raw)
To: Recipients
Hi,
I am Kristi and i saw your contact through google page of your country which impress me and i feel we can be friends to share ideals and reason together for good,if not bad then let us know more about each other and i hope there will be no problem for us to be friend with no bad intention.You can also reply me through email(cplkristibakar@outlook.com) so that i can share details about myself, i will be waiting to hear from you soon.
^ permalink raw reply
* Re: raid 5 crashed
From: bobzer @ 2016-05-27 19:19 UTC (permalink / raw)
To: linux-raid, Robin Hill, Mikael Abrahamsson
In-Reply-To: <CADzS=aoKqrG0QV1-Fe5g_b=n11gmbnerTyZ+eY4rBtJieNz_2w@mail.gmail.com>
hi,
I'm afraid to make the problem worst but i received a new HD to do a
dd_rescue :-)
I'm ready to buy another HD but the problem is that i don't know
what's the best to recover my data
My question is : There is a way to test if the data/raid is ok without
take the risk of losting anything more ?
help me please :-(
best regards
Mathieu
On Wed, May 25, 2016 at 11:06 PM, bobzer <bobzer@gmail.com> wrote:
> thanks for your help.
> i took time to answer because unlucky me, the power supply of my
> laptop fried so no laptop and no way to work on my raid :-(
> any way i got a new one :-)
>
> for the dmesg i paste it here : http://pastebin.com/whUHs256
>
> root@serveur:~# uname -a
> Linux serveur 3.2.0-4-amd64 #1 SMP Debian 3.2.63-2+deb7u1 x86_64 GNU/Linux
> root@serveur:~# mdadm -V
> mdadm - v3.3-78-gf43f5b3 - 02nd avril 2014
>
> about zero the superblock on the wrong device, i hope i didn't do
> that, but also i really don't think i did that, because i took care
> and at that time the raid was working
>
> i don't know what to do, if i use dd_rescue and can't get back 100% of
> the data could i be able to start the raid anyway ?
> what are my risk if i try something like :
> mdadm --assemble --force /dev/md0 /dev/sd[bcde]1
>
> thank you very much for your time
> Mathieu
>
>
> On Wed, May 11, 2016 at 3:15 PM, Robin Hill <robin@robinhill.me.uk> wrote:
>> On Tue May 10, 2016 at 11:28:31PM +0200, bobzer wrote:
>>
>>> hi everyone,
>>>
>>> I'm in panic mode :-( because i got a raid 5 with 4 disk but 2 removed
>>> yesterday i got a power outage which removed one disk. the disks
>>> sd[bcd]1 was ok and saying that sde1 is removed but sde1 said that
>>> everything is fine.
>>> so i stop the raid, zero the superblock of sde1, start the raid and
>>> add sde1 to the raid. then it start to reconstruct, i think it had
>>> time to finish before this problem (i'm not 100% sure that it finish
>>> but i think so)
>>> the data was accessible so i went to sleep
>>> today i discovered the raid in this state :
>>>
>>> root@serveur:/home/math# mdadm -D /dev/md0
>>> /dev/md0:
>>> Version : 1.2
>>> Creation Time : Sun Mar 4 22:49:14 2012
>>> Raid Level : raid5
>>> Array Size : 5860532352 (5589.04 GiB 6001.19 GB)
>>> Used Dev Size : 1953510784 (1863.01 GiB 2000.40 GB)
>>> Raid Devices : 4
>>> Total Devices : 4
>>> Persistence : Superblock is persistent
>>>
>>> Update Time : Fri May 6 17:44:02 2016
>>> State : clean, FAILED
>>> Active Devices : 2
>>> Working Devices : 3
>>> Failed Devices : 1
>>> Spare Devices : 1
>>>
>>> Layout : left-symmetric
>>> Chunk Size : 128K
>>>
>>> Name : debian:0
>>> UUID : bf3c605b:9699aa55:d45119a2:7ba58d56
>>> Events : 892482
>>>
>>> Number Major Minor RaidDevice State
>>> 3 8 33 0 active sync /dev/sdc1
>>> 1 8 49 1 active sync /dev/sdd1
>>> 4 0 0 4 removed
>>> 6 0 0 6 removed
>>>
>>> 4 8 17 - faulty /dev/sdb1
>>> 5 8 65 - spare /dev/sde1
>>>
>> So this reports /dev/sdb1 faulty and /dev/sde1 spare. That would
>> indicate that the rebuild hadn't finished.
>>
>>> root@serveur:/home/math# mdadm --examine /dev/sdb1
>>> /dev/sdb1:
>>> Magic : a92b4efc
>>> Version : 1.2
>>> Feature Map : 0x0
>>> Array UUID : bf3c605b:9699aa55:d45119a2:7ba58d56
>>> Name : debian:0
>>> Creation Time : Sun Mar 4 22:49:14 2012
>>> Raid Level : raid5
>>> Raid Devices : 4
>>>
>>> Avail Dev Size : 3907021954 (1863.01 GiB 2000.40 GB)
>>> Array Size : 5860532352 (5589.04 GiB 6001.19 GB)
>>> Used Dev Size : 3907021568 (1863.01 GiB 2000.40 GB)
>>> Data Offset : 2048 sectors
>>> Super Offset : 8 sectors
>>> Unused Space : before=1960 sectors, after=386 sectors
>>> State : clean
>>> Device UUID : 9bececcb:d520ca38:fd88d956:5718e361
>>>
>>> Update Time : Fri May 6 02:07:00 2016
>>> Bad Block Log : 512 entries available at offset 72 sectors
>>> Checksum : dc2a133a - correct
>>> Events : 892215
>>>
>>> Layout : left-symmetric
>>> Chunk Size : 128K
>>>
>>> Device Role : Active device 2
>>> Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
>>>
>> We can see /dev/sdb1 has a lower event count than the others and also
>> that it indicates all the drives in the array were active when it was
>> last running. That would strongly suggest that it was not in the array
>> when /dev/sde1 was added to rebuild. The update time is also nearly 16
>> hours earlier than that of the other drives.
>>
>>> root@serveur:/home/math# mdadm --examine /dev/sdc1
>>> /dev/sdc1:
>>> Magic : a92b4efc
>>> Version : 1.2
>>> Feature Map : 0x0
>>> Array UUID : bf3c605b:9699aa55:d45119a2:7ba58d56
>>> Name : debian:0
>>> Creation Time : Sun Mar 4 22:49:14 2012
>>> Raid Level : raid5
>>> Raid Devices : 4
>>>
>>> Avail Dev Size : 3907021954 (1863.01 GiB 2000.40 GB)
>>> Array Size : 5860532352 (5589.04 GiB 6001.19 GB)
>>> Used Dev Size : 3907021568 (1863.01 GiB 2000.40 GB)
>>> Data Offset : 2048 sectors
>>> Super Offset : 8 sectors
>>> Unused Space : before=1960 sectors, after=386 sectors
>>> State : clean
>>> Device UUID : 1ecaf51c:3289a902:7bb71a93:237c68e8
>>>
>>> Update Time : Fri May 6 17:58:27 2016
>>> Bad Block Log : 512 entries available at offset 72 sectors
>>> Checksum : b9d6aa84 - correct
>>> Events : 892484
>>>
>>> Layout : left-symmetric
>>> Chunk Size : 128K
>>>
>>> Device Role : Active device 0
>>> Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
>>>
>>> root@serveur:/home/math# mdadm --examine /dev/sdd1
>>> /dev/sdd1:
>>> Magic : a92b4efc
>>> Version : 1.2
>>> Feature Map : 0x0
>>> Array UUID : bf3c605b:9699aa55:d45119a2:7ba58d56
>>> Name : debian:0
>>> Creation Time : Sun Mar 4 22:49:14 2012
>>> Raid Level : raid5
>>> Raid Devices : 4
>>>
>>> Avail Dev Size : 3907021954 (1863.01 GiB 2000.40 GB)
>>> Array Size : 5860532352 (5589.04 GiB 6001.19 GB)
>>> Used Dev Size : 3907021568 (1863.01 GiB 2000.40 GB)
>>> Data Offset : 2048 sectors
>>> Super Offset : 8 sectors
>>> Unused Space : before=0 sectors, after=386 sectors
>>> State : clean
>>> Device UUID : 406c4cb5:c188e4a9:7ed8be9f:14a49b16
>>>
>>> Update Time : Fri May 6 17:58:27 2016
>>> Bad Block Log : 512 entries available at offset 2032 sectors
>>> Checksum : 343f9cd0 - correct
>>> Events : 892484
>>>
>>> Layout : left-symmetric
>>> Chunk Size : 128K
>>>
>>> Device Role : Active device 1
>>> Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
>>>
>> These two drives contain the same information. They indicate that they
>> were the only 2 running members in the array when they were last updated.
>>
>>> root@serveur:/home/math# mdadm --examine /dev/sde1
>>> /dev/sde1:
>>> Magic : a92b4efc
>>> Version : 1.2
>>> Feature Map : 0x8
>>> Array UUID : bf3c605b:9699aa55:d45119a2:7ba58d56
>>> Name : debian:0
>>> Creation Time : Sun Mar 4 22:49:14 2012
>>> Raid Level : raid5
>>> Raid Devices : 4
>>>
>>> Avail Dev Size : 3907025072 (1863.01 GiB 2000.40 GB)
>>> Array Size : 5860532352 (5589.04 GiB 6001.19 GB)
>>> Used Dev Size : 3907021568 (1863.01 GiB 2000.40 GB)
>>> Data Offset : 2048 sectors
>>> Super Offset : 8 sectors
>>> Unused Space : before=1960 sectors, after=3504 sectors
>>> State : clean
>>> Device UUID : f2e9c1ec:2852cf21:1a588581:b9f49a8b
>>>
>>> Update Time : Fri May 6 17:58:27 2016
>>> Bad Block Log : 512 entries available at offset 72 sectors - bad
>>> blocks present.
>>> Checksum : 3a65b8bc - correct
>>> Events : 892484
>>>
>>> Layout : left-symmetric
>>> Chunk Size : 128K
>>>
>>> Device Role : spare
>>> Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
>>>
>> And finally /dev/sde1 shows as a spare, with the rest of the data
>> matching /dev/sdc1 and /dev/sde1.
>>
>>> PLEASE help me :-) i don't know what to do so i did nothing to not do
>>> any stupid things
>>> 1000 thank you
>>>
>>> ps i just saw this, i hope it not mak y case worst
>>> root@serveur:/home/math# cat /etc/mdadm/mdadm.conf
>>> DEVICE /dev/sd[bcd]1
>>> ARRAY /dev/md0 metadata=1.2 name=debian:0
>>> UUID=bf3c605b:9699aa55:d45119a2:7ba58d56
>>>
>>
>> From the data here, if looks to me as though /dev/sdb1 failed originally
>> (hence it thinks the array was complete). Either then /dev/sde1 also
>> failed, or you've proceeded to zero the superblock on the wrong drive.
>> You really need to look through the system logs and verify what happened
>> when and to what disk (if you rebooted at any point, the drive ordering
>> may have changed, so don't take for granted that the drive names are
>> consistent throughout).
>>
>> Cheers,
>> Robin
>> --
>> ___
>> ( ' } | Robin Hill <robin@robinhill.me.uk> |
>> / / ) | Little Jim says .... |
>> // !! | "He fallen in de water !!" |
^ permalink raw reply
* Re: [PATCH] dm-log-writes: fix bug with too large bios
From: Josef Bacik @ 2016-05-27 15:36 UTC (permalink / raw)
To: Mikulas Patocka, Alasdair G. Kergon, Mike Snitzer
Cc: dm-devel, James Johnston, 'Eric Wheeler',
'Tim Small', 'Kent Overstreet', linux-bcache,
dm-crypt, 'Neil Brown', linux-raid
In-Reply-To: <alpine.LRH.2.02.1605271047280.2109@file01.intranet.prod.int.rdu2.redhat.com>
On 05/27/2016 10:51 AM, Mikulas Patocka wrote:
> bio_alloc can allocate a bio with at most BIO_MAX_PAGES (256) vector
> entries. However, the incoming bio may have more vector entries if it was
> allocated by other means. For example, bcache submits bios with more than
> BIO_MAX_PAGES entries. This results in bio_alloc failure.
>
> To avoid the failure, change the code so that it allocates bio with at
> most BIO_MAX_PAGES entries. If the incoming bio has more entries,
> bio_add_page will fail and a new bio will be allocated - the code that
> handles bio_add_page failure already exists in the dm-log-writes target.
>
> Also, move atomic_inc(&lc->io_blocks) before bio_alloc to fix a bug that
> the target hangs if bio_alloc fails. The error path does put_io_block(lc),
> so we must do atomic_inc(&lc->io_blocks) before invoking the error path to
> avoid underflow of lc->io_blocks.
>
> Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
> Cc: stable@vger.kernel.org # v4.1+
Reviewed-by: Josef Bacik <jbacik@fb.com>
Thanks,
Josef
^ permalink raw reply
* [PATCH] dm-log-writes: fix bug with too large bios
From: Mikulas Patocka @ 2016-05-27 14:51 UTC (permalink / raw)
To: Alasdair G. Kergon, Mike Snitzer, Josef Bacik
Cc: dm-devel, James Johnston, 'Eric Wheeler',
'Tim Small', 'Kent Overstreet', linux-bcache,
dm-crypt, 'Neil Brown', linux-raid
bio_alloc can allocate a bio with at most BIO_MAX_PAGES (256) vector
entries. However, the incoming bio may have more vector entries if it was
allocated by other means. For example, bcache submits bios with more than
BIO_MAX_PAGES entries. This results in bio_alloc failure.
To avoid the failure, change the code so that it allocates bio with at
most BIO_MAX_PAGES entries. If the incoming bio has more entries,
bio_add_page will fail and a new bio will be allocated - the code that
handles bio_add_page failure already exists in the dm-log-writes target.
Also, move atomic_inc(&lc->io_blocks) before bio_alloc to fix a bug that
the target hangs if bio_alloc fails. The error path does put_io_block(lc),
so we must do atomic_inc(&lc->io_blocks) before invoking the error path to
avoid underflow of lc->io_blocks.
Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
Cc: stable@vger.kernel.org # v4.1+
Index: linux-4.6/drivers/md/dm-log-writes.c
===================================================================
--- linux-4.6.orig/drivers/md/dm-log-writes.c
+++ linux-4.6/drivers/md/dm-log-writes.c
@@ -258,12 +258,12 @@ static int log_one_block(struct log_writ
goto out;
sector++;
- bio = bio_alloc(GFP_KERNEL, block->vec_cnt);
+ atomic_inc(&lc->io_blocks);
+ bio = bio_alloc(GFP_KERNEL, min(block->vec_cnt, BIO_MAX_PAGES));
if (!bio) {
DMERR("Couldn't alloc log bio");
goto error;
}
- atomic_inc(&lc->io_blocks);
bio->bi_iter.bi_size = 0;
bio->bi_iter.bi_sector = sector;
bio->bi_bdev = lc->logdev->bdev;
@@ -280,7 +280,7 @@ static int log_one_block(struct log_writ
if (ret != block->vecs[i].bv_len) {
atomic_inc(&lc->io_blocks);
submit_bio(WRITE, bio);
- bio = bio_alloc(GFP_KERNEL, block->vec_cnt - i);
+ bio = bio_alloc(GFP_KERNEL, min(block->vec_cnt - i, BIO_MAX_PAGES));
if (!bio) {
DMERR("Couldn't alloc log bio");
goto error;
^ permalink raw reply
* [PATCH] dm-crypt: Fix error with too large bios (was: bcache gets stuck flushing writeback cache when used in combination with LUKS/dm-crypt and non-default bucket size)
From: Mikulas Patocka @ 2016-05-27 14:47 UTC (permalink / raw)
To: James Johnston
Cc: 'Eric Wheeler', 'Tim Small',
'Kent Overstreet', 'Alasdair Kergon',
'Mike Snitzer', linux-bcache, dm-devel, dm-crypt,
'Neil Brown', linux-raid
In-Reply-To: <034e01d1b3e2$1de2f910$59a8eb30$@codenest.com>
Hi
Here I'm sending a patch for this bug.
BTW. I found several other bugs in bcache when testing this.
1) make-bcache and the other tools do not perform endian conversion -
consequently bcache doesn't work on big-endian machines.
2) bcache cannot be compiled on newer gcc because of inline keyword. Note
that in GNU C, the inline keyword is just a hint that doesn't change
correntness or behavior of a program. However, according to ANSI C, the
inline keywork changes meaning of a program - GCC recently switched to
ANSI C by default and so the code doesn't compile. This is a patch:
--- bcache-tools.orig/bcache.c
+++ bcache-tools/bcache.c
@@ -115,7 +115,7 @@ static const uint64_t crc_table[256] = {
0x9AFCE626CE85B507ULL
};
-inline uint64_t crc64(const void *_data, size_t len)
+uint64_t crc64(const void *_data, size_t len)
{
uint64_t crc = 0xFFFFFFFFFFFFFFFFULL;
const unsigned char *data = _data;
3) dm-crypt returns large bios with -EIO and bcache responds by attempting
to submit the bios again and again (which results in the reported loop).
The patch below fixes dm-crypt to not return errors, however you should
also fix bcache to handle errors gracefully (i.e. stop using the device on
I/O error, and don't submit the bios over and over again).
Mikulas
On Sun, 22 May 2016, James Johnston wrote:
> > On Fri, 20 May 2016, James Johnston wrote:
> >
> > > > On Mon, 16 May 2016, Tim Small wrote:
> > > >
> > > > > On 08/05/16 19:39, James Johnston wrote:
> > > > > > I've run into a problem where the bcache writeback cache can't be flushed to
> > > > > > disk when the backing device is a LUKS / dm-crypt device and the cache set has
> > > > > > a non-default bucket size. Basically, only a few megabytes will be flushed to
> > > > > > disk, and then it gets stuck. Stuck means that the bcache writeback task
> > > > > > thrashes the disk by constantly reading hundreds of MB/second from the cache set
> > > > > > in an infinite loop, while not actually progressing (dirty_data never decreases
> > > > > > beyond a certain point).
> > > > >
> > > > > > [...]
> > > > >
> > > > > > The situation is basically unrecoverable as far as I can tell: if you attempt
> > > > > > to detach the cache set then the cache set disk gets thrashed extra-hard
> > > > > > forever, and it's impossible to actually get the cache set detached. The only
> > > > > > solution seems to be to back up the data and destroy the volume...
> > > > >
> > > > > You can boot an older kernel to flush the device without destroying it
> > > > > (I'm guessing that's because older kernels split down the big requests
> > > > > which are failing on the 4.4 kernel). Once flushed you could put the
> > > > > cache into writethrough mode, or use a smaller bucket size.
> > > >
> > > > Indeed, can someone test 4.1.y and see if the problem persists with a 2M
> > > > bucket size? (If someone has already tested 4.1, then appologies as I've
> > > > not yet seen that report.)
> > > >
> > > > If 4.1 works, then I think a bisect is in order. Such a bisect would at
> > > > least highlight the problem and might indicate a (hopefully trivial) fix.
> > >
> > > To help narrow this down, I tested the following generic pre-compiled mainline kernels
> > > on Ubuntu 15.10:
> > >
> > > * WORKS: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.3.6-wily/
> > > * DOES NOT WORK: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.4-rc1+cod1-wily/
> > >
> > > I also tried the default & latest distribution-provided 4.2 kernel. It worked.
> > > This one also worked:
> > >
> > > * WORKS: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.2.8-wily/
> > >
> > > So it seems to me that it is a regression from 4.3.6 kernel to any 4.4 kernel. That
> > > should help save time with bisection...
> >
> > Below is the patchlist for md and block that might help with a place to
> > start. Are there any other places in the Linux tree where we should watch
> > for changes?
> >
> > I'm wondering if it might be in dm-4.4-changes since this is dm-crypt
> > related, but it could be ac322de which was quite large.
> >
> > James or Tim,
> >
> > Can you try building ac322de? If that produces the problem, then there
> > are only 3 more to try (unless this was actually a problem in 4.3 which
> > was fixed in 4.3.y, but hopefully that isn't so).
> >
> > ccf21b6 is probably the next to test to rule out neil's big md patch,
> > which Linus abreviated in the commit log so it must be quite long. OTOH,
> > if dm-4.4-changes works, then I'm not sure what commit might produce the
> > problem because the rest are not obviously relevant to the issue that are
> > more recent.
>
> So I decided to go ahead and bisect it today. Looks like the bad commit is
> this one. The commit prior flushed the bcache writeback cache without
> incident; this one does not and I guess caused this bcache regression.
> (FWIW ac322de came up during bisection, and tested good.)
>
> johnstonj@kernel-build:~/linux$ git bisect bad
> dbba42d8a9ebddcc1c1412e8457f79f3cb6ef6e7 is the first bad commit
> commit dbba42d8a9ebddcc1c1412e8457f79f3cb6ef6e7
> Author: Mikulas Patocka <mpatocka@redhat.com>
> Date: Wed Oct 21 16:34:20 2015 -0400
>
> dm: eliminate unused "bioset" process for each bio-based DM device
>
> Commit 54efd50bfd873e2dbf784e0b21a8027ba4299a3e ("block: make
> generic_make_request handle arbitrarily sized bios") makes it possible
> for block devices to process large bios. In doing so that commit
> allocates a new queue->bio_split bioset for each block device, this
> bioset is used for allocating bios when the driver needs to split large
> bios.
>
> Each bioset allocates a workqueue process, thus the above commit
> increases the number of processes allocated per block device.
>
> DM doesn't need the queue->bio_split bioset, thus we can deallocate it.
> This reduces the number of allocated processes per bio-based DM device
> from 3 to 2. Also remove the call to blk_queue_split(), it is not
> needed because DM does its own splitting.
>
> Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
> Signed-off-by: Mike Snitzer <snitzer@redhat.com>
>
> The patch for this commit is very brief; reproduced here:
>
> diff --git a/drivers/md/dm.c b/drivers/md/dm.c
> index 9555843..64b50b7 100644
> --- a/drivers/md/dm.c
> +++ b/drivers/md/dm.c
> @@ -1763,8 +1763,6 @@ static void dm_make_request(struct request_queue *q, struct bio *bio)
>
> map = dm_get_live_table(md, &srcu_idx);
>
> - blk_queue_split(q, &bio, q->bio_split);
> -
> generic_start_io_acct(rw, bio_sectors(bio), &dm_disk(md)->part0);
>
> /* if we're suspended, we have to queue this io for later */
> @@ -2792,6 +2790,12 @@ int dm_setup_md_queue(struct mapped_device *md)
> case DM_TYPE_BIO_BASED:
> dm_init_old_md_queue(md);
> blk_queue_make_request(md->queue, dm_make_request);
> + /*
> + * DM handles splitting bios as needed. Free the bio_split bioset
> + * since it won't be used (saves 1 process per bio-based DM device).
> + */
> + bioset_free(md->queue->bio_split);
> + md->queue->bio_split = NULL;
> break;
> }
>
> Here is the bisect log:
>
> johnstonj@kernel-build:~/linux$ git bisect log
> git bisect start
> # good: [6a13feb9c82803e2b815eca72fa7a9f5561d7861] Linux 4.3
> git bisect good 6a13feb9c82803e2b815eca72fa7a9f5561d7861
> # bad: [8005c49d9aea74d382f474ce11afbbc7d7130bec] Linux 4.4-rc1
> git bisect bad 8005c49d9aea74d382f474ce11afbbc7d7130bec
> # bad: [118c216e16c5ccb028cd03a0dcd56d17a07ff8d7] Merge tag 'staging-4.4-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging
> git bisect bad 118c216e16c5ccb028cd03a0dcd56d17a07ff8d7
> # good: [e627078a0cbdc0c391efeb5a2c4eb287328fd633] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/s390/linux
> git bisect good e627078a0cbdc0c391efeb5a2c4eb287328fd633
> # good: [c17c6da659571a115c7b4983da6c6ac464317c34] staging: wilc1000: rename pfScanResult of struct scan_attr
> git bisect good c17c6da659571a115c7b4983da6c6ac464317c34
> # good: [7bdb7d554e0e433b92b63f3472523cc3067f8ab4] Staging: rtl8192u: ieee80211: corrected indent
> git bisect good 7bdb7d554e0e433b92b63f3472523cc3067f8ab4
> # good: [ac322de6bf5416cb145b58599297b8be73cd86ac] Merge tag 'md/4.4' of git://neil.brown.name/md
> git bisect good ac322de6bf5416cb145b58599297b8be73cd86ac
> # good: [a4d8e93c3182a54d8d21a4d1cec6538ae1be9e16] Merge tag 'usb-for-v4.4' of git://git.kernel.org/pub/scm/linux/kernel/git/balbi/usb into usb-next
> git bisect good a4d8e93c3182a54d8d21a4d1cec6538ae1be9e16
> # good: [4f56f3fdca43c9a18339b6e0c3b1aa2f57f6d0b0] serial: 8250: Tolerate clock variance for max baud rate
> git bisect good 4f56f3fdca43c9a18339b6e0c3b1aa2f57f6d0b0
> # good: [e052c6d15c61cc4caff2f06cbca72b183da9f15e] tty: Use unbound workqueue for all input workers
> git bisect good e052c6d15c61cc4caff2f06cbca72b183da9f15e
> # good: [b9ca0c948c921e960006aaf319a29c004917cdf6] uwb: neh: Use setup_timer
> git bisect good b9ca0c948c921e960006aaf319a29c004917cdf6
> # bad: [aad9ae4550755edc020b5c511a8b54f0104b2f47] dm switch: simplify conditional in alloc_region_table()
> git bisect bad aad9ae4550755edc020b5c511a8b54f0104b2f47
> # good: [a3d939ae7b5f82688a6d3450f95286eaea338328] dm: convert ffs to __ffs
> git bisect good a3d939ae7b5f82688a6d3450f95286eaea338328
> # bad: [00272c854ee17b804ce81ef706f611dac17f4f89] dm linear: remove redundant target name from error messages
> git bisect bad 00272c854ee17b804ce81ef706f611dac17f4f89
> # bad: [4c7da06f5a780bbf44ebd7547789e48536d0a823] dm persistent data: eliminate unnecessary return values
> git bisect bad 4c7da06f5a780bbf44ebd7547789e48536d0a823
> # bad: [dbba42d8a9ebddcc1c1412e8457f79f3cb6ef6e7] dm: eliminate unused "bioset" process for each bio-based DM device
> git bisect bad dbba42d8a9ebddcc1c1412e8457f79f3cb6ef6e7
> # first bad commit: [dbba42d8a9ebddcc1c1412e8457f79f3cb6ef6e7] dm: eliminate unused "bioset" process for each bio-based DM device
>
> Commands used for testing:
>
> # Make cache set
> make-bcache --bucket 2M -C /dev/sdb
> # Set up backing device crypto
> cryptsetup luksFormat /dev/sdc
> cryptsetup open --type luks /dev/sdc backCrypt
> # Make backing device & enable writeback
> make-bcache -B /dev/mapper/backCrypt
> bcache-super-show /dev/sdb | grep cset.uuid | cut -f 3 > /sys/block/bcache0/bcache/attach
> echo writeback > /sys/block/bcache0/bcache/cache_mode
>
> # KILL SEQUENCE
>
> cd /sys/block/bcache0/bcache
> echo 0 > sequential_cutoff
> # Verify that the cache is attached (i.e. does not say "no cache")
> cat state
> dd if=/dev/urandom of=/dev/bcache0 bs=1M count=250
> cat dirty_data
> cat state
> # Next line causes severe disk thrashing and failure to flush writeback cache
> # on bad commits.
> echo 1 > detach
> cat dirty_data
> cat state
>
> Hope this provides some insight into the problem...
>
> James
dm-crypt: Fix error with too large bios
When dm-crypt processes writes, it allocates a new bio in the function
crypt_alloc_buffer. The bio is allocated from a bio set and it can have at
most BIO_MAX_PAGES vector entries, however the incoming bio can be larger
if it was allocated by other means. For example, bcache creates bios
larger than BIO_MAX_PAGES. If the incoming bio is larger, bio_alloc_bioset
fails and error is returned.
To avoid the error, we test for too large bio in the function crypt_map
and dm_accept_partial_bio to split the bio. dm_accept_partial_bio trims
the current bio to the desired size and requests that the device mapper
core sends another bio with the rest of the data.
Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
Cc: stable@vger.kernel.org # v3.16+
Index: linux-4.6/drivers/md/dm-crypt.c
===================================================================
--- linux-4.6.orig/drivers/md/dm-crypt.c
+++ linux-4.6/drivers/md/dm-crypt.c
@@ -2137,6 +2137,10 @@ static int crypt_map(struct dm_target *t
struct dm_crypt_io *io;
struct crypt_config *cc = ti->private;
+ if (unlikely(bio->bi_iter.bi_size > BIO_MAX_SIZE) &&
+ (bio->bi_rw & (REQ_FLUSH | REQ_DISCARD | REQ_WRITE)) == REQ_WRITE)
+ dm_accept_partial_bio(bio, BIO_MAX_SIZE >> SECTOR_SHIFT);
+
/*
* If bio is REQ_FLUSH or REQ_DISCARD, just bypass crypt queues.
* - for REQ_FLUSH device-mapper core ensures that no IO is in-flight
^ permalink raw reply
* [RFC v2 3/3] md: dm-crypt: Introduce the bulk mode method when sending request
From: Baolin Wang @ 2016-05-27 11:11 UTC (permalink / raw)
To: axboe, agk, snitzer, dm-devel, herbert, davem
Cc: ebiggers3, js1304, tadeusz.struk, smueller, standby24x7, shli,
dan.j.williams, martin.petersen, sagig, kent.overstreet,
keith.busch, tj, ming.lei, broonie, arnd, linux-crypto,
linux-block, linux-raid, linux-kernel, baolin.wang
In-Reply-To: <cover.1464346333.git.baolin.wang@linaro.org>
In now dm-crypt code, it is ineffective to map one segment (always one
sector) of one bio with just only one scatterlist at one time for hardware
crypto engine. Especially for some encryption mode (like ecb or xts mode)
cooperating with the crypto engine, they just need one initial IV or null
IV instead of different IV for each sector. In this situation We can consider
to use multiple scatterlists to map the whole bio and send all scatterlists
of one bio to crypto engine to encrypt or decrypt, which can improve the
hardware engine's efficiency.
With this optimization, On my test setup (beaglebone black board) using 64KB
I/Os on an eMMC storage device I saw about 60% improvement in throughput for
encrypted writes, and about 100% improvement for encrypted reads. But this
is not fit for other modes which need different IV for each sector.
Signed-off-by: Baolin Wang <baolin.wang@linaro.org>
---
drivers/md/dm-crypt.c | 145 ++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 144 insertions(+), 1 deletion(-)
diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c
index 4f3cb35..2101f35 100644
--- a/drivers/md/dm-crypt.c
+++ b/drivers/md/dm-crypt.c
@@ -33,6 +33,7 @@
#include <linux/device-mapper.h>
#define DM_MSG_PREFIX "crypt"
+#define DM_MAX_SG_LIST 1024
/*
* context holding the current state of a multi-part conversion
@@ -142,6 +143,9 @@ struct crypt_config {
char *cipher;
char *cipher_string;
+ struct sg_table sgt_in;
+ struct sg_table sgt_out;
+
struct crypt_iv_operations *iv_gen_ops;
union {
struct iv_essiv_private essiv;
@@ -837,6 +841,129 @@ static u8 *iv_of_dmreq(struct crypt_config *cc,
crypto_skcipher_alignmask(any_tfm(cc)) + 1);
}
+static void crypt_init_sg_table(struct scatterlist *sgl)
+{
+ struct scatterlist *sg;
+ int i;
+
+ for_each_sg(sgl, sg, DM_MAX_SG_LIST, i) {
+ if (i < DM_MAX_SG_LIST - 1 && sg_is_last(sg))
+ sg_unmark_end(sg);
+ else if (i == DM_MAX_SG_LIST - 1)
+ sg_mark_end(sg);
+ }
+
+ for_each_sg(sgl, sg, DM_MAX_SG_LIST, i) {
+ memset(sg, 0, sizeof(struct scatterlist));
+
+ if (i == DM_MAX_SG_LIST - 1)
+ sg_mark_end(sg);
+ }
+}
+
+static void crypt_reinit_sg_table(struct crypt_config *cc)
+{
+ if (!cc->sgt_in.orig_nents || !cc->sgt_out.orig_nents)
+ return;
+
+ crypt_init_sg_table(cc->sgt_in.sgl);
+ crypt_init_sg_table(cc->sgt_out.sgl);
+}
+
+static int crypt_alloc_sg_table(struct crypt_config *cc)
+{
+ unsigned int bulk_mode = skcipher_is_bulk_mode(any_tfm(cc));
+ int ret = 0;
+
+ if (!bulk_mode)
+ goto out_skip_alloc;
+
+ ret = sg_alloc_table(&cc->sgt_in, DM_MAX_SG_LIST, GFP_KERNEL);
+ if (ret)
+ goto out_skip_alloc;
+
+ ret = sg_alloc_table(&cc->sgt_out, DM_MAX_SG_LIST, GFP_KERNEL);
+ if (ret)
+ goto out_free_table;
+
+ return 0;
+
+out_free_table:
+ sg_free_table(&cc->sgt_in);
+out_skip_alloc:
+ cc->sgt_in.orig_nents = 0;
+ cc->sgt_out.orig_nents = 0;
+
+ return ret;
+}
+
+static int crypt_convert_bulk_block(struct crypt_config *cc,
+ struct convert_context *ctx,
+ struct skcipher_request *req)
+{
+ struct bio *bio_in = ctx->bio_in;
+ struct bio *bio_out = ctx->bio_out;
+ unsigned int total_bytes = bio_in->bi_iter.bi_size;
+ unsigned int total_sg_in, total_sg_out;
+ struct scatterlist *sg_in, *sg_out;
+ struct dm_crypt_request *dmreq;
+ u8 *iv;
+ int r;
+
+ if (!cc->sgt_in.orig_nents || !cc->sgt_out.orig_nents)
+ return -EINVAL;
+
+ dmreq = dmreq_of_req(cc, req);
+ iv = iv_of_dmreq(cc, dmreq);
+ dmreq->iv_sector = ctx->cc_sector;
+ dmreq->ctx = ctx;
+
+ total_sg_in = blk_bio_map_sg(bdev_get_queue(bio_in->bi_bdev),
+ bio_in, cc->sgt_in.sgl);
+ if ((total_sg_in <= 0) || (total_sg_in > DM_MAX_SG_LIST)) {
+ DMERR("%s in sg map error %d, sg table nents[%d]\n",
+ __func__, total_sg_in, cc->sgt_in.orig_nents);
+ return -EINVAL;
+ }
+
+ ctx->iter_in.bi_size -= total_bytes;
+ sg_in = cc->sgt_in.sgl;
+ sg_out = cc->sgt_in.sgl;
+
+ if (bio_data_dir(bio_in) == READ)
+ goto set_crypt;
+
+ total_sg_out = blk_bio_map_sg(bdev_get_queue(bio_out->bi_bdev),
+ bio_out, cc->sgt_out.sgl);
+ if ((total_sg_out <= 0) || (total_sg_out > DM_MAX_SG_LIST)) {
+ DMERR("%s out sg map error %d, sg table nents[%d]\n",
+ __func__, total_sg_out, cc->sgt_out.orig_nents);
+ return -EINVAL;
+ }
+
+ ctx->iter_out.bi_size -= total_bytes;
+ sg_out = cc->sgt_out.sgl;
+
+set_crypt:
+ if (cc->iv_gen_ops) {
+ r = cc->iv_gen_ops->generator(cc, iv, dmreq);
+ if (r < 0)
+ return r;
+ }
+
+ skcipher_request_set_crypt(req, sg_in, sg_out, total_bytes, iv);
+
+ if (bio_data_dir(ctx->bio_in) == WRITE)
+ r = crypto_skcipher_encrypt(req);
+ else
+ r = crypto_skcipher_decrypt(req);
+
+ if (!r && cc->iv_gen_ops && cc->iv_gen_ops->post)
+ r = cc->iv_gen_ops->post(cc, iv, dmreq);
+
+ return r;
+}
+
static int crypt_convert_block(struct crypt_config *cc,
struct convert_context *ctx,
struct skcipher_request *req)
@@ -920,6 +1047,7 @@ static void crypt_free_req(struct crypt_config *cc,
static int crypt_convert(struct crypt_config *cc,
struct convert_context *ctx)
{
+ unsigned int bulk_mode;
int r;
atomic_set(&ctx->cc_pending, 1);
@@ -930,7 +1058,14 @@ static int crypt_convert(struct crypt_config *cc,
atomic_inc(&ctx->cc_pending);
- r = crypt_convert_block(cc, ctx, ctx->req);
+ bulk_mode = skcipher_is_bulk_mode(any_tfm(cc));
+ if (!bulk_mode) {
+ r = crypt_convert_block(cc, ctx, ctx->req);
+ } else {
+ r = crypt_convert_bulk_block(cc, ctx, ctx->req);
+ if (r == -EINVAL)
+ r = crypt_convert_block(cc, ctx, ctx->req);
+ }
switch (r) {
/*
@@ -1081,6 +1216,7 @@ static void crypt_dec_pending(struct dm_crypt_io *io)
if (io->ctx.req)
crypt_free_req(cc, io->ctx.req, base_bio);
+ crypt_reinit_sg_table(cc);
base_bio->bi_error = error;
bio_endio(base_bio);
}
@@ -1563,6 +1699,9 @@ static void crypt_dtr(struct dm_target *ti)
kzfree(cc->cipher);
kzfree(cc->cipher_string);
+ sg_free_table(&cc->sgt_in);
+ sg_free_table(&cc->sgt_out);
+
/* Must zero key material before freeing */
kzfree(cc);
}
@@ -1718,6 +1857,10 @@ static int crypt_ctr_cipher(struct dm_target *ti,
}
}
+ ret = crypt_alloc_sg_table(cc);
+ if (ret)
+ DMWARN("Allocate sg table for bulk mode failed");
+
ret = 0;
bad:
kfree(cipher_api);
--
1.7.9.5
^ permalink raw reply related
* [RFC v2 2/3] crypto: Introduce CRYPTO_ALG_BULK flag
From: Baolin Wang @ 2016-05-27 11:11 UTC (permalink / raw)
To: axboe, agk, snitzer, dm-devel, herbert, davem
Cc: ebiggers3, js1304, tadeusz.struk, smueller, standby24x7, shli,
dan.j.williams, martin.petersen, sagig, kent.overstreet,
keith.busch, tj, ming.lei, broonie, arnd, linux-crypto,
linux-block, linux-raid, linux-kernel, baolin.wang
In-Reply-To: <cover.1464346333.git.baolin.wang@linaro.org>
Now some cipher hardware engines prefer to handle bulk block rather than one
sector (512 bytes) created by dm-crypt, cause these cipher engines can handle
the intermediate values (IV) by themselves in one bulk block. This means we
can increase the size of the request by merging request rather than always 512
bytes and thus increase the hardware engine processing speed.
So introduce 'CRYPTO_ALG_BULK' flag to indicate this cipher can support bulk
mode.
Signed-off-by: Baolin Wang <baolin.wang@linaro.org>
---
include/crypto/skcipher.h | 7 +++++++
include/linux/crypto.h | 6 ++++++
2 files changed, 13 insertions(+)
diff --git a/include/crypto/skcipher.h b/include/crypto/skcipher.h
index 0f987f5..d89d29a 100644
--- a/include/crypto/skcipher.h
+++ b/include/crypto/skcipher.h
@@ -519,5 +519,12 @@ static inline void skcipher_request_set_crypt(
req->iv = iv;
}
+static inline unsigned int skcipher_is_bulk_mode(struct crypto_skcipher *sk_tfm)
+{
+ struct crypto_tfm *tfm = crypto_skcipher_tfm(sk_tfm);
+
+ return crypto_tfm_alg_bulk(tfm);
+}
+
#endif /* _CRYPTO_SKCIPHER_H */
diff --git a/include/linux/crypto.h b/include/linux/crypto.h
index 6e28c89..a315487 100644
--- a/include/linux/crypto.h
+++ b/include/linux/crypto.h
@@ -63,6 +63,7 @@
#define CRYPTO_ALG_DEAD 0x00000020
#define CRYPTO_ALG_DYING 0x00000040
#define CRYPTO_ALG_ASYNC 0x00000080
+#define CRYPTO_ALG_BULK 0x00000100
/*
* Set this bit if and only if the algorithm requires another algorithm of
@@ -623,6 +624,11 @@ static inline u32 crypto_tfm_alg_type(struct crypto_tfm *tfm)
return tfm->__crt_alg->cra_flags & CRYPTO_ALG_TYPE_MASK;
}
+static inline unsigned int crypto_tfm_alg_bulk(struct crypto_tfm *tfm)
+{
+ return tfm->__crt_alg->cra_flags & CRYPTO_ALG_BULK;
+}
+
static inline unsigned int crypto_tfm_alg_blocksize(struct crypto_tfm *tfm)
{
return tfm->__crt_alg->cra_blocksize;
--
1.7.9.5
^ permalink raw reply related
* [RFC v2 1/3] block: Introduce blk_bio_map_sg() to map one bio
From: Baolin Wang @ 2016-05-27 11:11 UTC (permalink / raw)
To: axboe, agk, snitzer, dm-devel, herbert, davem
Cc: ebiggers3, js1304, tadeusz.struk, smueller, standby24x7, shli,
dan.j.williams, martin.petersen, sagig, kent.overstreet,
keith.busch, tj, ming.lei, broonie, arnd, linux-crypto,
linux-block, linux-raid, linux-kernel, baolin.wang
In-Reply-To: <cover.1464346333.git.baolin.wang@linaro.org>
In dm-crypt, it need to map one bio to scatterlist for improving the
hardware engine encryption efficiency. Thus this patch introduces the
blk_bio_map_sg() function to map one bio with scatterlists.
For avoiding the duplicated code in __blk_bios_map_sg() function, add
one parameter to distinguish bio map or request map.
Signed-off-by: Baolin Wang <baolin.wang@linaro.org>
---
block/blk-merge.c | 36 +++++++++++++++++++++++++++++++-----
include/linux/blkdev.h | 2 ++
2 files changed, 33 insertions(+), 5 deletions(-)
diff --git a/block/blk-merge.c b/block/blk-merge.c
index 2613531..badae44 100644
--- a/block/blk-merge.c
+++ b/block/blk-merge.c
@@ -376,7 +376,7 @@ new_segment:
static int __blk_bios_map_sg(struct request_queue *q, struct bio *bio,
struct scatterlist *sglist,
- struct scatterlist **sg)
+ struct scatterlist **sg, bool single_bio)
{
struct bio_vec bvec, bvprv = { NULL };
struct bvec_iter iter;
@@ -408,13 +408,39 @@ single_segment:
return 1;
}
- for_each_bio(bio)
+ if (!single_bio) {
+ for_each_bio(bio)
+ bio_for_each_segment(bvec, bio, iter)
+ __blk_segment_map_sg(q, &bvec, sglist, &bvprv,
+ sg, &nsegs, &cluster);
+ } else {
bio_for_each_segment(bvec, bio, iter)
- __blk_segment_map_sg(q, &bvec, sglist, &bvprv, sg,
- &nsegs, &cluster);
+ __blk_segment_map_sg(q, &bvec, sglist, &bvprv,
+ sg, &nsegs, &cluster);
+ }
+
+ return nsegs;
+}
+
+/*
+ * Map a bio to scatterlist, return number of sg entries setup. Caller must
+ * make sure sg can hold bio segments entries.
+ */
+int blk_bio_map_sg(struct request_queue *q, struct bio *bio,
+ struct scatterlist *sglist)
+{
+ struct scatterlist *sg = NULL;
+ int nsegs = 0;
+
+ if (bio)
+ nsegs = __blk_bios_map_sg(q, bio, sglist, &sg, true);
+
+ if (sg)
+ sg_mark_end(sg);
return nsegs;
}
+EXPORT_SYMBOL(blk_bio_map_sg);
/*
* map a request to scatterlist, return number of sg entries setup. Caller
@@ -427,7 +453,7 @@ int blk_rq_map_sg(struct request_queue *q, struct request *rq,
int nsegs = 0;
if (rq->bio)
- nsegs = __blk_bios_map_sg(q, rq->bio, sglist, &sg);
+ nsegs = __blk_bios_map_sg(q, rq->bio, sglist, &sg, false);
if (unlikely(rq->cmd_flags & REQ_COPY_USER) &&
(blk_rq_bytes(rq) & q->dma_pad_mask)) {
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 1fd8fdf..5868062 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -1013,6 +1013,8 @@ extern void blk_queue_write_cache(struct request_queue *q, bool enabled, bool fu
extern struct backing_dev_info *blk_get_backing_dev_info(struct block_device *bdev);
extern int blk_rq_map_sg(struct request_queue *, struct request *, struct scatterlist *);
+extern int blk_bio_map_sg(struct request_queue *q, struct bio *bio,
+ struct scatterlist *sglist);
extern void blk_dump_rq_flags(struct request *, char *);
extern long nr_blockdev_pages(void);
--
1.7.9.5
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox