* Linux raid wiki - setting up a system - advice wanted :-)
From: Wols Lists @ 2016-09-25 21:16 UTC (permalink / raw)
To: linux-raid
This is a great way for learning lots about raid :-)
I'm planning a section on setting up a new system, and I need to know
what will happen if you give entire drives to mdadm.
Does it leave the first 2 megs empty? Basically, what I'm asking is if I do
mdadm --create /dev/md/bigarray -add /dev/sda /dev/sdb /dev/sdc
(note I am passing the entire drive, not the first partition) and then I
install grub on those drives, will I trash the array?
Cheers,
Wol
^ permalink raw reply
* Re: [PATCH 1/5] r5cache: write part of r5cache
From: Song Liu @ 2016-09-25 7:02 UTC (permalink / raw)
To: Zhengyuan Liu
Cc: linux-raid, neilb@suse.com, Shaohua Li, Kernel Team,
dan.j.williams@intel.com, hch@infradead.org
In-Reply-To: <CAOOPZo5BvMkpnBxRBNS+pmn3ELJr4+82kPMMLZcnUOCnVwXHGQ@mail.gmail.com>
Hi Zhengyuan,
Thanks for your input. I will add your patch (as a separate patch) to my v2 set (coming
soon).
For your question, I think updating superblock every 5 second is OK. We sure can fix that
if it becomes a concern.
Thanks,
Song
>> On 9/24/16, 7:51 AM, "Zhengyuan Liu" <liuzhengyuang521@gmail.com> wrote:
100c221268fd893bfb35273ef9ba14994f38bc6c
^ permalink raw reply
* Re: [PATCH 1/5] r5cache: write part of r5cache
From: Zhengyuan Liu @ 2016-09-24 14:51 UTC (permalink / raw)
To: Song Liu; +Cc: linux-raid, neilb, shli, kernel-team, dan.j.williams, hch
In-Reply-To: <1472681902-1172317-2-git-send-email-songliubraving@fb.com>
Hi, Song Liu
I think there is a bug about the counter of pending_full_writes, I try
to fix it with bellow patch. There is another question around me, the
reclaim thread is wake up every 5 seconds which would update the
super block no matter there is IO activity or not. If it should have some
checks to update the super block?
commit 100c221268fd893bfb35273ef9ba14994f38bc6c
Author: ZhengYuan Liu <liuzhengyuan@kylinos.cn>
Date: Sat Sep 24 21:59:01 2016 +0800
md/r5cache: decrease the counter after full-write stripe was reclaimed
Once the data was written to cache device,r5c_handle_cached_data_endio would
be called to set dev->written with null and return the bio to up layer.
As a result,handle_stripe_clean_event has no chance to be called to decrease
the counter of conf->pending_full_writes when the stripe was written to raid
disks at reclaim stage. It should test the STRIPE_FULL_WRITE state
and decrease
the counter at somewhere when the full-write stripe was reclaimed,
r5c_handle_stripe_written may be the right place to do that.
Signed-off-by: ZhengYuan Liu <liuzhengyuan@kylinos.cn>
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index 3446bcc..326457d 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -1955,6 +1955,10 @@ void r5c_handle_stripe_written(struct r5conf *conf,
list_del_init(&sh->r5c);
spin_unlock_irqrestore(&conf->log->stripe_in_cache_lock, flags);
sh->log_start = MaxSector;
+
+ if (test_and_clear_bit(STRIPE_FULL_WRITE, &sh->state))
+ if (atomic_dec_and_test(&conf->pending_full_writes))
+ md_wakeup_thread(conf->mddev->thread);
}
if (do_wakeup)
2016-09-01 6:18 GMT+08:00 Song Liu <songliubraving@fb.com>:
> This is the write part of r5cache. The cache is integrated with
> stripe cache of raid456. It leverages code of r5l_log to write
> data to journal device.
>
> r5cache split current write path into 2 parts: the write path
> and the reclaim path. The write path is as following:
> 1. write data to journal
> (r5c_handle_stripe_dirtying, r5c_cache_data)
> 2. call bio_endio
> (r5c_handle_data_cached, r5c_return_dev_pending_writes).
>
> Then the reclaim path is as:
> 1. Freeze the stripe (r5c_freeze_stripe_for_reclaim)
> 2. Calcualte parity (reconstruct or RMW)
> 3. Write parity (and maybe some other data) to journal device
> 4. Write data and parity to RAID disks
>
> Step 3 and 4 of reclaim path is very similar to write path of
> raid5 journal.
>
> With r5cache, write operation does not wait for parity calculation
> and write out, so the write latency is lower (1 write to journal
> device vs. read and then write to raid disks). Also, r5cache will
> reduce RAID overhead (multipile IO due to read-modify-write of
> parity) and provide more opportunities of full stripe writes.
>
> r5cache adds 2 flags to stripe_head.state: STRIPE_R5C_FROZEN and
> STRIPE_R5C_WRITTEN. The write path runs w/ STRIPE_R5C_FROZEN == 0.
> Cache writes start from r5c_handle_stripe_dirtying(), where bit
> R5_Wantcache is set for devices with bio in towrite. Then, the
> data is written to the journal through r5l_log implementation.
> Once the data is in the journal, we set bit R5_InCache, and
> presue bio_endio for these writes.
>
> The reclaim path starts by setting STRIPE_R5C_FROZEN. This makes
> the stripe into reclaim. If some write operation arrives at this
> time, it will be handled as raid5 journal (calculate parity,
> write to jorunal, write to disks, bio_endio).
>
> Once frozen, the stripe is sent back to raid5 state machine,
> where handle_stripe_dirtying will evaluate the stripe for
> reconstruct writes or RMW writes (read data and calculate parity).
>
> For RMW, the code allocates an extra page for each data block
> being updated. This is stored in r5dev->page and the old data
> is read into it. Then the prexor calculation subtracts ->page
> from the parity block, and the reconstruct calculation adds the
> ->orig_page data back into the parity block.
>
> r5cache naturally excludes SkipCopy. With R5_Wantcache bit set,
> async_copy_data will not skip copy.
>
> Before writing data to RAID disks, the r5l_log logic stores
> parity (and non-overwrite data) to the journal.
>
> Instead of inactive_list, stripes with cached data are tracked in
> r5conf->r5c_cached_list. r5conf->r5c_cached_stripes tracks how
> many stripes has dirty data in the cache.
>
> There are some known limitations of the cache implementation:
>
> 1. Write cache only covers full page writes (R5_OVERWRITE). Writes
> of smaller granularity are write through.
> 2. Only one log io (sh->log_io) for each stripe at anytime. Later
> writes for the same stripe have to wait. This can be improved by
> moving log_io to r5dev.
> 3. When use with bitmap, there is a deadlock in add_stripe_bio =>
> bitmap_startwrite.
>
> Signed-off-by: Song Liu <songliubraving@fb.com>
> Signed-off-by: Shaohua Li <shli@fb.com>
> ---
> drivers/md/raid5-cache.c | 271 +++++++++++++++++++++++++++++++++++++++++++++--
> drivers/md/raid5.c | 164 ++++++++++++++++++++++++----
> drivers/md/raid5.h | 25 ++++-
> 3 files changed, 429 insertions(+), 31 deletions(-)
>
> diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
> index 1b1ab4a..cc7b80d 100644
> --- a/drivers/md/raid5-cache.c
> +++ b/drivers/md/raid5-cache.c
> @@ -40,6 +40,13 @@
> */
> #define R5L_POOL_SIZE 4
>
> +enum r5c_state {
> + R5C_STATE_NO_CACHE = 0,
> + R5C_STATE_WRITE_THROUGH = 1,
> + R5C_STATE_WRITE_BACK = 2,
> + R5C_STATE_CACHE_BROKEN = 3,
> +};
> +
> struct r5l_log {
> struct md_rdev *rdev;
>
> @@ -96,6 +103,9 @@ struct r5l_log {
> spinlock_t no_space_stripes_lock;
>
> bool need_cache_flush;
> +
> + /* for r5c_cache */
> + enum r5c_state r5c_state;
> };
>
> /*
> @@ -168,12 +178,73 @@ static void __r5l_set_io_unit_state(struct r5l_io_unit *io,
> io->state = state;
> }
>
> +/*
> + * Freeze the stripe, thus send the stripe into reclaim path.
> + *
> + * This function should only be called from raid5d that handling this stripe,
> + * or when the stripe is on r5c_cached_list (hold conf->device_lock)
> + */
> +void r5c_freeze_stripe_for_reclaim(struct stripe_head *sh)
> +{
> + struct r5conf *conf = sh->raid_conf;
> +
> + if (!conf->log)
> + return;
> +
> + WARN_ON(test_bit(STRIPE_R5C_FROZEN, &sh->state));
> + set_bit(STRIPE_R5C_FROZEN, &sh->state);
> +
> + if (!test_and_set_bit(STRIPE_PREREAD_ACTIVE, &sh->state))
> + atomic_inc(&conf->preread_active_stripes);
> + if (test_and_clear_bit(STRIPE_IN_R5C_CACHE, &sh->state)) {
> + BUG_ON(atomic_read(&conf->r5c_cached_stripes) == 0);
> + atomic_dec(&conf->r5c_cached_stripes);
> + }
> +}
> +
> +static void r5c_handle_data_cached(struct stripe_head *sh)
> +{
> + int i;
> +
> + for (i = sh->disks; i--; )
> + if (test_and_clear_bit(R5_Wantcache, &sh->dev[i].flags)) {
> + set_bit(R5_InCache, &sh->dev[i].flags);
> + clear_bit(R5_LOCKED, &sh->dev[i].flags);
> + atomic_inc(&sh->dev_in_cache);
> + }
> +}
> +
> +/*
> + * this journal write must contain full parity,
> + * it may also contain some data pages
> + */
> +static void r5c_handle_parity_cached(struct stripe_head *sh)
> +{
> + int i;
> +
> + for (i = sh->disks; i--; )
> + if (test_bit(R5_InCache, &sh->dev[i].flags))
> + set_bit(R5_Wantwrite, &sh->dev[i].flags);
> + set_bit(STRIPE_R5C_WRITTEN, &sh->state);
> +}
> +
> +static void r5c_finish_cache_stripe(struct stripe_head *sh)
> +{
> + if (test_bit(STRIPE_R5C_FROZEN, &sh->state))
> + r5c_handle_parity_cached(sh);
> + else
> + r5c_handle_data_cached(sh);
> +}
> +
> static void r5l_io_run_stripes(struct r5l_io_unit *io)
> {
> struct stripe_head *sh, *next;
>
> list_for_each_entry_safe(sh, next, &io->stripe_list, log_list) {
> list_del_init(&sh->log_list);
> +
> + r5c_finish_cache_stripe(sh);
> +
> set_bit(STRIPE_HANDLE, &sh->state);
> raid5_release_stripe(sh);
> }
> @@ -402,7 +473,8 @@ static int r5l_log_stripe(struct r5l_log *log, struct stripe_head *sh,
> io = log->current_io;
>
> for (i = 0; i < sh->disks; i++) {
> - if (!test_bit(R5_Wantwrite, &sh->dev[i].flags))
> + if (!test_bit(R5_Wantwrite, &sh->dev[i].flags) &&
> + !test_bit(R5_Wantcache, &sh->dev[i].flags))
> continue;
> if (i == sh->pd_idx || i == sh->qd_idx)
> continue;
> @@ -412,18 +484,19 @@ static int r5l_log_stripe(struct r5l_log *log, struct stripe_head *sh,
> r5l_append_payload_page(log, sh->dev[i].page);
> }
>
> - if (sh->qd_idx >= 0) {
> + if (parity_pages == 2) {
> r5l_append_payload_meta(log, R5LOG_PAYLOAD_PARITY,
> sh->sector, sh->dev[sh->pd_idx].log_checksum,
> sh->dev[sh->qd_idx].log_checksum, true);
> r5l_append_payload_page(log, sh->dev[sh->pd_idx].page);
> r5l_append_payload_page(log, sh->dev[sh->qd_idx].page);
> - } else {
> + } else if (parity_pages == 1) {
> r5l_append_payload_meta(log, R5LOG_PAYLOAD_PARITY,
> sh->sector, sh->dev[sh->pd_idx].log_checksum,
> 0, false);
> r5l_append_payload_page(log, sh->dev[sh->pd_idx].page);
> - }
> + } else
> + BUG_ON(parity_pages != 0);
>
> list_add_tail(&sh->log_list, &io->stripe_list);
> atomic_inc(&io->pending_stripe);
> @@ -432,7 +505,6 @@ static int r5l_log_stripe(struct r5l_log *log, struct stripe_head *sh,
> return 0;
> }
>
> -static void r5l_wake_reclaim(struct r5l_log *log, sector_t space);
> /*
> * running in raid5d, where reclaim could wait for raid5d too (when it flushes
> * data from log to raid disks), so we shouldn't wait for reclaim here
> @@ -456,11 +528,17 @@ int r5l_write_stripe(struct r5l_log *log, struct stripe_head *sh)
> return -EAGAIN;
> }
>
> + WARN_ON(!test_bit(STRIPE_R5C_FROZEN, &sh->state));
> +
> for (i = 0; i < sh->disks; i++) {
> void *addr;
>
> if (!test_bit(R5_Wantwrite, &sh->dev[i].flags))
> continue;
> +
> + if (test_bit(R5_InCache, &sh->dev[i].flags))
> + continue;
> +
> write_disks++;
> /* checksum is already calculated in last run */
> if (test_bit(STRIPE_LOG_TRAPPED, &sh->state))
> @@ -473,6 +551,9 @@ int r5l_write_stripe(struct r5l_log *log, struct stripe_head *sh)
> parity_pages = 1 + !!(sh->qd_idx >= 0);
> data_pages = write_disks - parity_pages;
>
> + pr_debug("%s: write %d data_pages and %d parity_pages\n",
> + __func__, data_pages, parity_pages);
> +
> meta_size =
> ((sizeof(struct r5l_payload_data_parity) + sizeof(__le32))
> * data_pages) +
> @@ -735,7 +816,6 @@ static void r5l_write_super_and_discard_space(struct r5l_log *log,
> }
> }
>
> -
> static void r5l_do_reclaim(struct r5l_log *log)
> {
> sector_t reclaim_target = xchg(&log->reclaim_target, 0);
> @@ -798,7 +878,7 @@ static void r5l_reclaim_thread(struct md_thread *thread)
> r5l_do_reclaim(log);
> }
>
> -static void r5l_wake_reclaim(struct r5l_log *log, sector_t space)
> +void r5l_wake_reclaim(struct r5l_log *log, sector_t space)
> {
> unsigned long target;
> unsigned long new = (unsigned long)space; /* overflow in theory */
> @@ -1111,6 +1191,180 @@ static void r5l_write_super(struct r5l_log *log, sector_t cp)
> set_bit(MD_CHANGE_DEVS, &mddev->flags);
> }
>
> +static void r5c_flush_stripe(struct r5conf *conf, struct stripe_head *sh)
> +{
> + list_del_init(&sh->lru);
> + r5c_freeze_stripe_for_reclaim(sh);
> + atomic_inc(&conf->active_stripes);
> + atomic_inc(&sh->count);
> + set_bit(STRIPE_HANDLE, &sh->state);
> + raid5_release_stripe(sh);
> +}
> +
> +int r5c_flush_cache(struct r5conf *conf)
> +{
> + int count = 0;
> +
> + assert_spin_locked(&conf->device_lock);
> + if (!conf->log)
> + return 0;
> + while (!list_empty(&conf->r5c_cached_list)) {
> + struct list_head *l = conf->r5c_cached_list.next;
> + struct stripe_head *sh;
> +
> + sh = list_entry(l, struct stripe_head, lru);
> + r5c_flush_stripe(conf, sh);
> + ++count;
> + }
> + return count;
> +}
> +
> +int r5c_handle_stripe_dirtying(struct r5conf *conf,
> + struct stripe_head *sh,
> + struct stripe_head_state *s,
> + int disks) {
> + struct r5l_log *log = conf->log;
> + int i;
> + struct r5dev *dev;
> +
> + if (!log || test_bit(STRIPE_R5C_FROZEN, &sh->state))
> + return -EAGAIN;
> +
> + if (conf->log->r5c_state == R5C_STATE_WRITE_THROUGH ||
> + conf->quiesce != 0 || conf->mddev->degraded != 0) {
> + /* write through mode */
> + r5c_freeze_stripe_for_reclaim(sh);
> + return -EAGAIN;
> + }
> +
> + s->to_cache = 0;
> +
> + for (i = disks; i--; ) {
> + dev = &sh->dev[i];
> + /* if none-overwrite, use the reclaim path (write through) */
> + if (dev->towrite && !test_bit(R5_OVERWRITE, &dev->flags) &&
> + !test_bit(R5_InCache, &dev->flags)) {
> + r5c_freeze_stripe_for_reclaim(sh);
> + return -EAGAIN;
> + }
> + }
> +
> + for (i = disks; i--; ) {
> + dev = &sh->dev[i];
> + if (dev->towrite) {
> + set_bit(R5_Wantcache, &dev->flags);
> + set_bit(R5_Wantdrain, &dev->flags);
> + set_bit(R5_LOCKED, &dev->flags);
> + s->to_cache++;
> + }
> + }
> +
> + if (s->to_cache)
> + set_bit(STRIPE_OP_BIODRAIN, &s->ops_request);
> +
> + return 0;
> +}
> +
> +void r5c_handle_stripe_written(struct r5conf *conf,
> + struct stripe_head *sh) {
> + int i;
> + int do_wakeup = 0;
> +
> + if (test_and_clear_bit(STRIPE_R5C_WRITTEN, &sh->state)) {
> + WARN_ON(!test_bit(STRIPE_R5C_FROZEN, &sh->state));
> + clear_bit(STRIPE_R5C_FROZEN, &sh->state);
> +
> + for (i = sh->disks; i--; ) {
> + if (test_and_clear_bit(R5_InCache, &sh->dev[i].flags))
> + atomic_dec(&sh->dev_in_cache);
> + clear_bit(R5_UPTODATE, &sh->dev[i].flags);
> + if (test_and_clear_bit(R5_Overlap, &sh->dev[i].flags))
> + do_wakeup = 1;
> + }
> + }
> +
> + if (do_wakeup)
> + wake_up(&conf->wait_for_overlap);
> +}
> +
> +int
> +r5c_cache_data(struct r5l_log *log, struct stripe_head *sh,
> + struct stripe_head_state *s)
> +{
> + int pages;
> + int meta_size;
> + int reserve;
> + int i;
> + int ret = 0;
> + int page_count = 0;
> +
> + BUG_ON(!log);
> + BUG_ON(s->to_cache == 0);
> +
> + for (i = 0; i < sh->disks; i++) {
> + void *addr;
> +
> + if (!test_bit(R5_Wantcache, &sh->dev[i].flags))
> + continue;
> + addr = kmap_atomic(sh->dev[i].page);
> + sh->dev[i].log_checksum = crc32c_le(log->uuid_checksum,
> + addr, PAGE_SIZE);
> + kunmap_atomic(addr);
> + page_count++;
> + }
> + WARN_ON(page_count != s->to_cache);
> +
> + pages = s->to_cache;
> +
> + meta_size =
> + ((sizeof(struct r5l_payload_data_parity) + sizeof(__le32))
> + * pages);
> + /* Doesn't work with very big raid array */
> + if (meta_size + sizeof(struct r5l_meta_block) > PAGE_SIZE)
> + return -EINVAL;
> +
> + /*
> + * The stripe must enter state machine again to call endio, so
> + * don't delay.
> + */
> + clear_bit(STRIPE_DELAYED, &sh->state);
> + atomic_inc(&sh->count);
> +
> + mutex_lock(&log->io_mutex);
> + /* meta + data */
> + reserve = (1 + pages) << (PAGE_SHIFT - 9);
> + if (!r5l_has_free_space(log, reserve)) {
> + spin_lock(&log->no_space_stripes_lock);
> + list_add_tail(&sh->log_list, &log->no_space_stripes);
> + spin_unlock(&log->no_space_stripes_lock);
> +
> + r5l_wake_reclaim(log, reserve);
> + } else {
> + ret = r5l_log_stripe(log, sh, pages, 0);
> + if (ret) {
> + spin_lock_irq(&log->io_list_lock);
> + list_add_tail(&sh->log_list, &log->no_mem_stripes);
> + spin_unlock_irq(&log->io_list_lock);
> + }
> + }
> +
> + mutex_unlock(&log->io_mutex);
> + return 0;
> +}
> +
> +void r5c_do_reclaim(struct r5conf *conf)
> +{
> + struct stripe_head *sh, *next;
> + struct r5l_log *log = conf->log;
> +
> + assert_spin_locked(&conf->device_lock);
> +
> + if (!log)
> + return;
> + list_for_each_entry_safe(sh, next, &conf->r5c_cached_list, lru)
> + r5c_flush_stripe(conf, sh);
> +}
> +
> static int r5l_load_log(struct r5l_log *log)
> {
> struct md_rdev *rdev = log->rdev;
> @@ -1230,6 +1484,9 @@ int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev)
> INIT_LIST_HEAD(&log->no_space_stripes);
> spin_lock_init(&log->no_space_stripes_lock);
>
> + /* flush full stripe */
> + log->r5c_state = R5C_STATE_WRITE_BACK;
> +
> if (r5l_load_log(log))
> goto error;
>
> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> index b95c54c..ed6efd0 100644
> --- a/drivers/md/raid5.c
> +++ b/drivers/md/raid5.c
> @@ -316,8 +316,16 @@ static void do_release_stripe(struct r5conf *conf, struct stripe_head *sh,
> < IO_THRESHOLD)
> md_wakeup_thread(conf->mddev->thread);
> atomic_dec(&conf->active_stripes);
> - if (!test_bit(STRIPE_EXPANDING, &sh->state))
> - list_add_tail(&sh->lru, temp_inactive_list);
> + if (!test_bit(STRIPE_EXPANDING, &sh->state)) {
> + if (atomic_read(&sh->dev_in_cache) == 0) {
> + list_add_tail(&sh->lru, temp_inactive_list);
> + } else {
> + if (!test_and_set_bit(STRIPE_IN_R5C_CACHE,
> + &sh->state))
> + atomic_inc(&conf->r5c_cached_stripes);
> + list_add_tail(&sh->lru, &conf->r5c_cached_list);
> + }
> + }
> }
> }
>
> @@ -901,6 +909,13 @@ static void ops_run_io(struct stripe_head *sh, struct stripe_head_state *s)
>
> might_sleep();
>
> + if (s->to_cache) {
> + if (r5c_cache_data(conf->log, sh, s) == 0)
> + return;
> + /* array is too big that meta data size > PAGE_SIZE */
> + r5c_freeze_stripe_for_reclaim(sh);
> + }
> +
> if (r5l_write_stripe(conf->log, sh) == 0)
> return;
> for (i = disks; i--; ) {
> @@ -1029,6 +1044,7 @@ again:
>
> if (test_bit(R5_SkipCopy, &sh->dev[i].flags))
> WARN_ON(test_bit(R5_UPTODATE, &sh->dev[i].flags));
> +
> sh->dev[i].vec.bv_page = sh->dev[i].page;
> bi->bi_vcnt = 1;
> bi->bi_io_vec[0].bv_len = STRIPE_SIZE;
> @@ -1115,7 +1131,7 @@ again:
> static struct dma_async_tx_descriptor *
> async_copy_data(int frombio, struct bio *bio, struct page **page,
> sector_t sector, struct dma_async_tx_descriptor *tx,
> - struct stripe_head *sh)
> + struct stripe_head *sh, int no_skipcopy)
> {
> struct bio_vec bvl;
> struct bvec_iter iter;
> @@ -1155,7 +1171,8 @@ async_copy_data(int frombio, struct bio *bio, struct page **page,
> if (frombio) {
> if (sh->raid_conf->skip_copy &&
> b_offset == 0 && page_offset == 0 &&
> - clen == STRIPE_SIZE)
> + clen == STRIPE_SIZE &&
> + !no_skipcopy)
> *page = bio_page;
> else
> tx = async_memcpy(*page, bio_page, page_offset,
> @@ -1237,7 +1254,7 @@ static void ops_run_biofill(struct stripe_head *sh)
> while (rbi && rbi->bi_iter.bi_sector <
> dev->sector + STRIPE_SECTORS) {
> tx = async_copy_data(0, rbi, &dev->page,
> - dev->sector, tx, sh);
> + dev->sector, tx, sh, 0);
> rbi = r5_next_bio(rbi, dev->sector);
> }
> }
> @@ -1364,7 +1381,8 @@ static int set_syndrome_sources(struct page **srcs,
> if (i == sh->qd_idx || i == sh->pd_idx ||
> (srctype == SYNDROME_SRC_ALL) ||
> (srctype == SYNDROME_SRC_WANT_DRAIN &&
> - test_bit(R5_Wantdrain, &dev->flags)) ||
> + (test_bit(R5_Wantdrain, &dev->flags) ||
> + test_bit(R5_InCache, &dev->flags))) ||
> (srctype == SYNDROME_SRC_WRITTEN &&
> dev->written))
> srcs[slot] = sh->dev[i].page;
> @@ -1543,9 +1561,18 @@ ops_run_compute6_2(struct stripe_head *sh, struct raid5_percpu *percpu)
> static void ops_complete_prexor(void *stripe_head_ref)
> {
> struct stripe_head *sh = stripe_head_ref;
> + int i;
>
> pr_debug("%s: stripe %llu\n", __func__,
> (unsigned long long)sh->sector);
> +
> + for (i = sh->disks; i--; )
> + if (sh->dev[i].page != sh->dev[i].orig_page) {
> + struct page *p = sh->dev[i].page;
> +
> + sh->dev[i].page = sh->dev[i].orig_page;
> + put_page(p);
> + }
> }
>
> static struct dma_async_tx_descriptor *
> @@ -1567,7 +1594,8 @@ ops_run_prexor5(struct stripe_head *sh, struct raid5_percpu *percpu,
> for (i = disks; i--; ) {
> struct r5dev *dev = &sh->dev[i];
> /* Only process blocks that are known to be uptodate */
> - if (test_bit(R5_Wantdrain, &dev->flags))
> + if (test_bit(R5_Wantdrain, &dev->flags) ||
> + test_bit(R5_InCache, &dev->flags))
> xor_srcs[count++] = dev->page;
> }
>
> @@ -1618,6 +1646,10 @@ ops_run_biodrain(struct stripe_head *sh, struct dma_async_tx_descriptor *tx)
>
> again:
> dev = &sh->dev[i];
> + if (test_and_clear_bit(R5_InCache, &dev->flags)) {
> + BUG_ON(atomic_read(&sh->dev_in_cache) == 0);
> + atomic_dec(&sh->dev_in_cache);
> + }
> spin_lock_irq(&sh->stripe_lock);
> chosen = dev->towrite;
> dev->towrite = NULL;
> @@ -1625,7 +1657,8 @@ again:
> BUG_ON(dev->written);
> wbi = dev->written = chosen;
> spin_unlock_irq(&sh->stripe_lock);
> - WARN_ON(dev->page != dev->orig_page);
> + if (!test_bit(R5_Wantcache, &dev->flags))
> + WARN_ON(dev->page != dev->orig_page);
>
> while (wbi && wbi->bi_iter.bi_sector <
> dev->sector + STRIPE_SECTORS) {
> @@ -1637,8 +1670,10 @@ again:
> set_bit(R5_Discard, &dev->flags);
> else {
> tx = async_copy_data(1, wbi, &dev->page,
> - dev->sector, tx, sh);
> - if (dev->page != dev->orig_page) {
> + dev->sector, tx, sh,
> + test_bit(R5_Wantcache, &dev->flags));
> + if (dev->page != dev->orig_page &&
> + !test_bit(R5_Wantcache, &dev->flags)) {
> set_bit(R5_SkipCopy, &dev->flags);
> clear_bit(R5_UPTODATE, &dev->flags);
> clear_bit(R5_OVERWRITE, &dev->flags);
> @@ -1746,7 +1781,8 @@ again:
> xor_dest = xor_srcs[count++] = sh->dev[pd_idx].page;
> for (i = disks; i--; ) {
> struct r5dev *dev = &sh->dev[i];
> - if (head_sh->dev[i].written)
> + if (head_sh->dev[i].written ||
> + test_bit(R5_InCache, &head_sh->dev[i].flags))
> xor_srcs[count++] = dev->page;
> }
> } else {
> @@ -2001,6 +2037,7 @@ static struct stripe_head *alloc_stripe(struct kmem_cache *sc, gfp_t gfp,
> INIT_LIST_HEAD(&sh->batch_list);
> INIT_LIST_HEAD(&sh->lru);
> atomic_set(&sh->count, 1);
> + atomic_set(&sh->dev_in_cache, 0);
> for (i = 0; i < disks; i++) {
> struct r5dev *dev = &sh->dev[i];
>
> @@ -2887,6 +2924,9 @@ schedule_reconstruction(struct stripe_head *sh, struct stripe_head_state *s,
> if (!expand)
> clear_bit(R5_UPTODATE, &dev->flags);
> s->locked++;
> + } else if (test_bit(R5_InCache, &dev->flags)) {
> + set_bit(R5_LOCKED, &dev->flags);
> + s->locked++;
> }
> }
> /* if we are not expanding this is a proper write request, and
> @@ -2926,6 +2966,9 @@ schedule_reconstruction(struct stripe_head *sh, struct stripe_head_state *s,
> set_bit(R5_LOCKED, &dev->flags);
> clear_bit(R5_UPTODATE, &dev->flags);
> s->locked++;
> + } else if (test_bit(R5_InCache, &dev->flags)) {
> + set_bit(R5_LOCKED, &dev->flags);
> + s->locked++;
> }
> }
> if (!s->locked)
> @@ -3577,6 +3620,9 @@ static void handle_stripe_dirtying(struct r5conf *conf,
> int rmw = 0, rcw = 0, i;
> sector_t recovery_cp = conf->mddev->recovery_cp;
>
> + if (r5c_handle_stripe_dirtying(conf, sh, s, disks) == 0)
> + return;
> +
> /* Check whether resync is now happening or should start.
> * If yes, then the array is dirty (after unclean shutdown or
> * initial creation), so parity in some stripes might be inconsistent.
> @@ -3597,9 +3643,12 @@ static void handle_stripe_dirtying(struct r5conf *conf,
> } else for (i = disks; i--; ) {
> /* would I have to read this buffer for read_modify_write */
> struct r5dev *dev = &sh->dev[i];
> - if ((dev->towrite || i == sh->pd_idx || i == sh->qd_idx) &&
> + if ((dev->towrite || i == sh->pd_idx || i == sh->qd_idx ||
> + test_bit(R5_InCache, &dev->flags)) &&
> !test_bit(R5_LOCKED, &dev->flags) &&
> - !(test_bit(R5_UPTODATE, &dev->flags) ||
> + !((test_bit(R5_UPTODATE, &dev->flags) &&
> + (!test_bit(R5_InCache, &dev->flags) ||
> + dev->page != dev->orig_page)) ||
> test_bit(R5_Wantcompute, &dev->flags))) {
> if (test_bit(R5_Insync, &dev->flags))
> rmw++;
> @@ -3611,13 +3660,15 @@ static void handle_stripe_dirtying(struct r5conf *conf,
> i != sh->pd_idx && i != sh->qd_idx &&
> !test_bit(R5_LOCKED, &dev->flags) &&
> !(test_bit(R5_UPTODATE, &dev->flags) ||
> - test_bit(R5_Wantcompute, &dev->flags))) {
> + test_bit(R5_InCache, &dev->flags) ||
> + test_bit(R5_Wantcompute, &dev->flags))) {
> if (test_bit(R5_Insync, &dev->flags))
> rcw++;
> else
> rcw += 2*disks;
> }
> }
> +
> pr_debug("for sector %llu, rmw=%d rcw=%d\n",
> (unsigned long long)sh->sector, rmw, rcw);
> set_bit(STRIPE_HANDLE, &sh->state);
> @@ -3629,10 +3680,18 @@ static void handle_stripe_dirtying(struct r5conf *conf,
> (unsigned long long)sh->sector, rmw);
> for (i = disks; i--; ) {
> struct r5dev *dev = &sh->dev[i];
> - if ((dev->towrite || i == sh->pd_idx || i == sh->qd_idx) &&
> + if (test_bit(R5_InCache, &dev->flags) &&
> + dev->page == dev->orig_page)
> + dev->page = alloc_page(GFP_NOIO); /* prexor */
> +
> + if ((dev->towrite ||
> + i == sh->pd_idx || i == sh->qd_idx ||
> + test_bit(R5_InCache, &dev->flags)) &&
> !test_bit(R5_LOCKED, &dev->flags) &&
> - !(test_bit(R5_UPTODATE, &dev->flags) ||
> - test_bit(R5_Wantcompute, &dev->flags)) &&
> + !((test_bit(R5_UPTODATE, &dev->flags) &&
> + (!test_bit(R5_InCache, &dev->flags) ||
> + dev->page != dev->orig_page)) ||
> + test_bit(R5_Wantcompute, &dev->flags)) &&
> test_bit(R5_Insync, &dev->flags)) {
> if (test_bit(STRIPE_PREREAD_ACTIVE,
> &sh->state)) {
> @@ -3658,6 +3717,7 @@ static void handle_stripe_dirtying(struct r5conf *conf,
> i != sh->pd_idx && i != sh->qd_idx &&
> !test_bit(R5_LOCKED, &dev->flags) &&
> !(test_bit(R5_UPTODATE, &dev->flags) ||
> + test_bit(R5_InCache, &dev->flags) ||
> test_bit(R5_Wantcompute, &dev->flags))) {
> rcw++;
> if (test_bit(R5_Insync, &dev->flags) &&
> @@ -3697,7 +3757,7 @@ static void handle_stripe_dirtying(struct r5conf *conf,
> */
> if ((s->req_compute || !test_bit(STRIPE_COMPUTE_RUN, &sh->state)) &&
> (s->locked == 0 && (rcw == 0 || rmw == 0) &&
> - !test_bit(STRIPE_BIT_DELAY, &sh->state)))
> + !test_bit(STRIPE_BIT_DELAY, &sh->state)))
> schedule_reconstruction(sh, s, rcw == 0, 0);
> }
>
> @@ -4010,6 +4070,46 @@ static void handle_stripe_expansion(struct r5conf *conf, struct stripe_head *sh)
> async_tx_quiesce(&tx);
> }
>
> +static void
> +r5c_return_dev_pending_writes(struct r5conf *conf, struct r5dev *dev,
> + struct bio_list *return_bi)
> +{
> + struct bio *wbi, *wbi2;
> +
> + wbi = dev->written;
> + dev->written = NULL;
> + while (wbi && wbi->bi_iter.bi_sector <
> + dev->sector + STRIPE_SECTORS) {
> + wbi2 = r5_next_bio(wbi, dev->sector);
> + if (!raid5_dec_bi_active_stripes(wbi)) {
> + md_write_end(conf->mddev);
> + bio_list_add(return_bi, wbi);
> + }
> + wbi = wbi2;
> + }
> +}
> +
> +static void r5c_handle_cached_data_endio(struct r5conf *conf,
> + struct stripe_head *sh, int disks, struct bio_list *return_bi)
> +{
> + int i;
> +
> + for (i = sh->disks; i--; ) {
> + if (test_bit(R5_InCache, &sh->dev[i].flags) &&
> + sh->dev[i].written) {
> + set_bit(R5_UPTODATE, &sh->dev[i].flags);
> + r5c_return_dev_pending_writes(conf, &sh->dev[i],
> + return_bi);
> + }
> + }
> + r5l_stripe_write_finished(sh);
> +
> + bitmap_endwrite(conf->mddev->bitmap, sh->sector,
> + STRIPE_SECTORS,
> + !test_bit(STRIPE_DEGRADED, &sh->state),
> + 0);
> +}
> +
> /*
> * handle_stripe - do things to a stripe.
> *
> @@ -4188,6 +4288,8 @@ static void analyse_stripe(struct stripe_head *sh, struct stripe_head_state *s)
> if (rdev && !test_bit(Faulty, &rdev->flags))
> do_recovery = 1;
> }
> + if (test_bit(R5_InCache, &dev->flags) && dev->written)
> + s->just_cached++;
> }
> if (test_bit(STRIPE_SYNCING, &sh->state)) {
> /* If there is a failed device being replaced,
> @@ -4416,7 +4518,7 @@ static void handle_stripe(struct stripe_head *sh)
> struct r5dev *dev = &sh->dev[i];
> if (test_bit(R5_LOCKED, &dev->flags) &&
> (i == sh->pd_idx || i == sh->qd_idx ||
> - dev->written)) {
> + dev->written || test_bit(R5_InCache, &dev->flags))) {
> pr_debug("Writing block %d\n", i);
> set_bit(R5_Wantwrite, &dev->flags);
> if (prexor)
> @@ -4456,6 +4558,12 @@ static void handle_stripe(struct stripe_head *sh)
> test_bit(R5_Discard, &qdev->flags))))))
> handle_stripe_clean_event(conf, sh, disks, &s.return_bi);
>
> + if (s.just_cached)
> + r5c_handle_cached_data_endio(conf, sh, disks, &s.return_bi);
> +
> + if (test_bit(STRIPE_R5C_FROZEN, &sh->state))
> + r5l_stripe_write_finished(sh);
> +
> /* Now we might consider reading some blocks, either to check/generate
> * parity, or to satisfy requests
> * or to load a block that is being partially written.
> @@ -4467,13 +4575,17 @@ static void handle_stripe(struct stripe_head *sh)
> || s.expanding)
> handle_stripe_fill(sh, &s, disks);
>
> - /* Now to consider new write requests and what else, if anything
> - * should be read. We do not handle new writes when:
> + r5c_handle_stripe_written(conf, sh);
> +
> + /* Now to consider new write requests, cache write back and what else,
> + * if anything should be read. We do not handle new writes when:
> * 1/ A 'write' operation (copy+xor) is already in flight.
> * 2/ A 'check' operation is in flight, as it may clobber the parity
> * block.
> + * 3/ A r5c cache log write is in flight.
> */
> - if (s.to_write && !sh->reconstruct_state && !sh->check_state)
> + if ((s.to_write || test_bit(STRIPE_R5C_FROZEN, &sh->state)) &&
> + !sh->reconstruct_state && !sh->check_state && !sh->log_io)
> handle_stripe_dirtying(conf, sh, &s, disks);
>
> /* maybe we need to check and possibly fix the parity for this stripe
> @@ -5192,7 +5304,7 @@ static void raid5_make_request(struct mddev *mddev, struct bio * bi)
> * later we might have to read it again in order to reconstruct
> * data on failed drives.
> */
> - if (rw == READ && mddev->degraded == 0 &&
> + if (rw == READ && mddev->degraded == 0 && conf->log == NULL &&
> mddev->reshape_position == MaxSector) {
> bi = chunk_aligned_read(mddev, bi);
> if (!bi)
> @@ -5917,6 +6029,7 @@ static void raid5d(struct md_thread *thread)
> md_check_recovery(mddev);
> spin_lock_irq(&conf->device_lock);
> }
> + r5c_do_reclaim(conf);
> }
> pr_debug("%d stripes handled\n", handled);
>
> @@ -6583,6 +6696,9 @@ static struct r5conf *setup_conf(struct mddev *mddev)
> for (i = 0; i < NR_STRIPE_HASH_LOCKS; i++)
> INIT_LIST_HEAD(conf->temp_inactive_list + i);
>
> + atomic_set(&conf->r5c_cached_stripes, 0);
> + INIT_LIST_HEAD(&conf->r5c_cached_list);
> +
> conf->level = mddev->new_level;
> conf->chunk_sectors = mddev->new_chunk_sectors;
> if (raid5_alloc_percpu(conf) != 0)
> @@ -7655,8 +7771,10 @@ static void raid5_quiesce(struct mddev *mddev, int state)
> /* '2' tells resync/reshape to pause so that all
> * active stripes can drain
> */
> + r5c_flush_cache(conf);
> conf->quiesce = 2;
> wait_event_cmd(conf->wait_for_quiescent,
> + atomic_read(&conf->r5c_cached_stripes) == 0 &&
> atomic_read(&conf->active_stripes) == 0 &&
> atomic_read(&conf->active_aligned_reads) == 0,
> unlock_all_device_hash_locks_irq(conf),
> diff --git a/drivers/md/raid5.h b/drivers/md/raid5.h
> index 517d4b6..e301d0e 100644
> --- a/drivers/md/raid5.h
> +++ b/drivers/md/raid5.h
> @@ -226,6 +226,7 @@ struct stripe_head {
>
> struct r5l_io_unit *log_io;
> struct list_head log_list;
> + atomic_t dev_in_cache;
> /**
> * struct stripe_operations
> * @target - STRIPE_OP_COMPUTE_BLK target
> @@ -263,6 +264,7 @@ struct stripe_head_state {
> */
> int syncing, expanding, expanded, replacing;
> int locked, uptodate, to_read, to_write, failed, written;
> + int to_cache, just_cached;
> int to_fill, compute, req_compute, non_overwrite;
> int failed_num[2];
> int p_failed, q_failed;
> @@ -313,6 +315,8 @@ enum r5dev_flags {
> */
> R5_Discard, /* Discard the stripe */
> R5_SkipCopy, /* Don't copy data from bio to stripe cache */
> + R5_Wantcache, /* Want write data to write cache */
> + R5_InCache, /* Data in cache */
> };
>
> /*
> @@ -345,7 +349,11 @@ enum {
> STRIPE_BITMAP_PENDING, /* Being added to bitmap, don't add
> * to batch yet.
> */
> - STRIPE_LOG_TRAPPED, /* trapped into log */
> + STRIPE_LOG_TRAPPED, /* trapped into log */
> + STRIPE_IN_R5C_CACHE, /* in r5c cache (to-be/being handled or
> + * in conf->r5c_cached_list) */
> + STRIPE_R5C_FROZEN, /* r5c_cache frozen and being written out */
> + STRIPE_R5C_WRITTEN, /* ready for r5c_handle_stripe_written() */
> };
>
> #define STRIPE_EXPAND_SYNC_FLAGS \
> @@ -521,6 +529,8 @@ struct r5conf {
> */
> atomic_t active_stripes;
> struct list_head inactive_list[NR_STRIPE_HASH_LOCKS];
> + atomic_t r5c_cached_stripes;
> + struct list_head r5c_cached_list;
> atomic_t empty_inactive_list_nr;
> struct llist_head released_stripes;
> wait_queue_head_t wait_for_quiescent;
> @@ -635,4 +645,17 @@ extern void r5l_stripe_write_finished(struct stripe_head *sh);
> extern int r5l_handle_flush_request(struct r5l_log *log, struct bio *bio);
> extern void r5l_quiesce(struct r5l_log *log, int state);
> extern bool r5l_log_disk_error(struct r5conf *conf);
> +extern void r5l_wake_reclaim(struct r5l_log *log, sector_t space);
> +extern int
> +r5c_handle_stripe_dirtying(struct r5conf *conf, struct stripe_head *sh,
> + struct stripe_head_state *s, int disks);
> +extern int r5c_cache_data(struct r5l_log *log, struct stripe_head *sh,
> + struct stripe_head_state *s);
> +extern int r5c_cache_parity(struct r5l_log *log, struct stripe_head *sh,
> + struct stripe_head_state *s);
> +extern void
> +r5c_handle_stripe_written(struct r5conf *conf, struct stripe_head *sh);
> +extern void r5c_freeze_stripe_for_reclaim(struct stripe_head *sh);
> +extern void r5c_do_reclaim(struct r5conf *conf);
> +extern int r5c_flush_cache(struct r5conf *conf);
> #endif
> --
> 2.8.0.rc2
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* Re: Linux raid wiki
From: Wols Lists @ 2016-09-24 13:18 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <57E469E0.4020005@youngman.org.uk>
On 23/09/16 00:31, Wols Lists wrote:
> I've added the "When Things Go Wrogn" section, but so far only the first
> two pages - "Asking for help" and "Timeout Mismatch" - are all my work.
> The other three pages were already there, but I moved them here because
> I felt they belonged here.
>
> Please feel free to criticize it (or offer bouquets :-), and give advice
> as how to improve things, either in private email or on the list.
Replying to myself, but I'm reasonably happy with the first three
sections in "When Things Go Wrogn". But it's important that they're
correct! Would a couple of experts mind looking them over and sending a
critique to the list? Just a simple "Looks good" would be great and set
my mind at rest that I have understood things properly and I'm not
giving out bad advice.
Note that the next section is going to be along the lines of "My array
won't assemble / run"
https://raid.wiki.kernel.org/index.php/Asking_for_help
https://raid.wiki.kernel.org/index.php/Timeout_Mismatch
https://raid.wiki.kernel.org/index.php/Replacing_a_failed_drive
Cheers,
Wol
^ permalink raw reply
* Re: Linux raid wiki - force assembling an array where one drive has a different event count - advice needed
From: Phil Turmel @ 2016-09-24 3:43 UTC (permalink / raw)
To: Adam Goryachev, Wols Lists, linux-raid
In-Reply-To: <f9e2b742-1a26-c6f1-2b35-1c3cbd3099e4@websitemanagers.com.au>
On 09/23/2016 07:46 PM, Adam Goryachev wrote:
>
>
> On 24/09/2016 09:15, Wols Lists wrote:
>> As I understand it, the event count on all devices in an array should be
>> the same. If they're a little bit different it doesn't matter too much.
>> My question is how much does it matter?
>>
>> Let's say I've got a raid-5 and suddenly realise that one of the drives
>> has failed and been kicked from the array. What happens if I force a
>> reassemble? Or do a --re-add?
>>
>> I don't actually have a clue, and if I'm updating the wiki I need to
>> know. What I would HOPE happens, is that the raid code fires off an
>> integrity scan, reading each stripe, and updating the re-added drive if
>> it's out-of-date. Is this what the bitmap enables? So the raid code can
>> work out what changes have been made since the drive has been booted?
> If you have a bitmap, and you re-add a drive to the array, then it will
> check the bitmap to find out what is out of date, and then re-sync those
> parts of the drive.
> If there is no bitmap, and you re-add a drive, then the entire drive
> will be re-written/synced.
>> Or does forced re-adding risk damaging the data because the raid code
>> can't tell what is out-of-date and what is current on the re-added drive?
> There is no need to "force" re-adding a drive if there is still all the
> data in the array. The only reason you would force an array to assemble
> is in the case one drive has failed (totally dead/unusable, or failed a
> long time ago) and a second drive has failed with only a few bad
> sectors, or a timeout mismatch. Then you forget the oldest drive, force
> assemble with the good drives plus the recently failed drive, and then
> recover as much data as possible.
> The other useful scenario is a small number of bad sectors on two
> drives, but not in the same location. You won't survive a full re-sync
> on either drive, but forcing assembly might allow you to read all the data.
>> Basically, what I'm trying to get at, is that if there's one disk
>> missing in a raid5, is a user better off just adding a new drive and
>> rebuilding the array (risking a failure in another drive), or are they
>> better off trying to add the failed drive back in, and then doing a
>> --replace.
> It will depend on why the drive failed in the first place. If it failed
> due to a timeout mismatch, or user error (pulled the drive by accident,
> etc and you have a bitmap enabled, then re-adding is the best option
> (because you will only re-sync a small portion of the drive. If you do
> not have a bitmap, and you suspect one or more of your drives are having
> problems / likely to have read failures during the re-sync (this is
> usually when people come to the list/wiki) then it could be helpful to
> force the assemble (ie, ignore the event count). This will allow you to
> get your data off the array, or at least get to a point of full
> redundancy, you could then either add a drive to move to RAID6, or
> replace each drive (one by one) to get rid of the unreliable drives.
>> And I guess the same logic applies with raid6.
> Again, usually you would only do this if you lost 3 drives on the RAID6,
> 2 on RAID5.... Otherwise, you should re-build the drive in full (or
> suffer random data corruption). The bitmap is only useful after you
> temporarily lose one or more drives <= to the number of redundant drives
> (ie, this applies to RAID1 and RAID10 as well).
>
> So, IMHO, in the general scenario, you should not force assemble if you
> want to be sure you recover all your data. It is something that is done
> to reduce data loss from 100% to some unknown value depending on the
> event count difference, but usually it is very small.
>
> Generally, you should do a verify/repair on the array afterwards (even
> if the data is lost, at least they will be consistent about what it
> returns), and a fsck.
>
> Don't consider the above as gospel, but it matches the various scenarios
> I've seen on this list....
This is a very good summary. Especially the bit about forced assembly
being the point most people are at when they come to this list.
Phil
^ permalink raw reply
* Re: Linux raid wiki
From: Phil Turmel @ 2016-09-24 3:38 UTC (permalink / raw)
To: Wols Lists, WNSDEV, linux-raid
In-Reply-To: <57E577BE.8080101@youngman.org.uk>
On 09/23/2016 02:43 PM, Wols Lists wrote:
> Thanks.
>
> My understanding of role numbers is that they are the order of the disks
> in which stripes are written, so role 0 has the first stripe, role 1 the
> second, etc etc.
Yes, this is correct. But the numbers in brackets in mdstat aren't the
role number, but the slot number in the superblock member list. Which
only matches role number for the active devices at first creation.
Spares occupy additional slots, and keep their slot number for their
life in the array, including after assuming an active role.
> This is *normally* irrelevant, but should the array ever get trashed,
> the disks need to be listed *in* *that* *order* in a new --create statement.
You must use the "this device" role numbers in an mdadm -E report to be
sure, or use lsdrv.
mdstat is only useful for human review of array status.
Phil
^ permalink raw reply
* Re: Linux raid wiki - force assembling an array where one drive has a different event count - advice needed
From: Adam Goryachev @ 2016-09-23 23:46 UTC (permalink / raw)
To: Wols Lists, linux-raid
In-Reply-To: <57E5B77D.5080608@youngman.org.uk>
On 24/09/2016 09:15, Wols Lists wrote:
> As I understand it, the event count on all devices in an array should be
> the same. If they're a little bit different it doesn't matter too much.
> My question is how much does it matter?
>
> Let's say I've got a raid-5 and suddenly realise that one of the drives
> has failed and been kicked from the array. What happens if I force a
> reassemble? Or do a --re-add?
>
> I don't actually have a clue, and if I'm updating the wiki I need to
> know. What I would HOPE happens, is that the raid code fires off an
> integrity scan, reading each stripe, and updating the re-added drive if
> it's out-of-date. Is this what the bitmap enables? So the raid code can
> work out what changes have been made since the drive has been booted?
If you have a bitmap, and you re-add a drive to the array, then it will
check the bitmap to find out what is out of date, and then re-sync those
parts of the drive.
If there is no bitmap, and you re-add a drive, then the entire drive
will be re-written/synced.
> Or does forced re-adding risk damaging the data because the raid code
> can't tell what is out-of-date and what is current on the re-added drive?
There is no need to "force" re-adding a drive if there is still all the
data in the array. The only reason you would force an array to assemble
is in the case one drive has failed (totally dead/unusable, or failed a
long time ago) and a second drive has failed with only a few bad
sectors, or a timeout mismatch. Then you forget the oldest drive, force
assemble with the good drives plus the recently failed drive, and then
recover as much data as possible.
The other useful scenario is a small number of bad sectors on two
drives, but not in the same location. You won't survive a full re-sync
on either drive, but forcing assembly might allow you to read all the data.
> Basically, what I'm trying to get at, is that if there's one disk
> missing in a raid5, is a user better off just adding a new drive and
> rebuilding the array (risking a failure in another drive), or are they
> better off trying to add the failed drive back in, and then doing a
> --replace.
It will depend on why the drive failed in the first place. If it failed
due to a timeout mismatch, or user error (pulled the drive by accident,
etc and you have a bitmap enabled, then re-adding is the best option
(because you will only re-sync a small portion of the drive. If you do
not have a bitmap, and you suspect one or more of your drives are having
problems / likely to have read failures during the re-sync (this is
usually when people come to the list/wiki) then it could be helpful to
force the assemble (ie, ignore the event count). This will allow you to
get your data off the array, or at least get to a point of full
redundancy, you could then either add a drive to move to RAID6, or
replace each drive (one by one) to get rid of the unreliable drives.
> And I guess the same logic applies with raid6.
Again, usually you would only do this if you lost 3 drives on the RAID6,
2 on RAID5.... Otherwise, you should re-build the drive in full (or
suffer random data corruption). The bitmap is only useful after you
temporarily lose one or more drives <= to the number of redundant drives
(ie, this applies to RAID1 and RAID10 as well).
So, IMHO, in the general scenario, you should not force assemble if you
want to be sure you recover all your data. It is something that is done
to reduce data loss from 100% to some unknown value depending on the
event count difference, but usually it is very small.
Generally, you should do a verify/repair on the array afterwards (even
if the data is lost, at least they will be consistent about what it
returns), and a fsck.
Don't consider the above as gospel, but it matches the various scenarios
I've seen on this list....
Regards,
Adam
^ permalink raw reply
* Linux raid wiki - force assembling an array where one drive has a different event count - advice needed
From: Wols Lists @ 2016-09-23 23:15 UTC (permalink / raw)
To: linux-raid
As I understand it, the event count on all devices in an array should be
the same. If they're a little bit different it doesn't matter too much.
My question is how much does it matter?
Let's say I've got a raid-5 and suddenly realise that one of the drives
has failed and been kicked from the array. What happens if I force a
reassemble? Or do a --re-add?
I don't actually have a clue, and if I'm updating the wiki I need to
know. What I would HOPE happens, is that the raid code fires off an
integrity scan, reading each stripe, and updating the re-added drive if
it's out-of-date. Is this what the bitmap enables? So the raid code can
work out what changes have been made since the drive has been booted?
Or does forced re-adding risk damaging the data because the raid code
can't tell what is out-of-date and what is current on the re-added drive?
Basically, what I'm trying to get at, is that if there's one disk
missing in a raid5, is a user better off just adding a new drive and
rebuilding the array (risking a failure in another drive), or are they
better off trying to add the failed drive back in, and then doing a
--replace.
And I guess the same logic applies with raid6.
Cheers,
Wol
^ permalink raw reply
* RE: Linux raid wiki
From: Peter Sangas @ 2016-09-23 19:02 UTC (permalink / raw)
To: 'Wols Lists', linux-raid
In-Reply-To: <57E577BE.8080101@youngman.org.uk>
Interesting: What about for a RAID1 array?. My configuration is a 2 disk, 3 partition RAID1. After installation of Ubuntu 16.04 the role numbers are 0 and 1.
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4][raid10]
md2 : active raid1 sda3[0] sdb3[1]
634795008 blocks super 1.2 [2/2] [UU]
bitmap: 0/5 pages [0KB], 65536KB chunk
md1 : active raid1 sda2[0] sdb2[1]
126887936 blocks super 1.2 [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk
md0 : active raid1 sda1[0] sdb1[1]
19514368 blocks super 1.2 [2/2] [UU]
But after I replace one of the drives (sdb) with a new drive and sync it the role number of md0,sdb1 changes from 1 to 2? Is that supposed to happen?
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4][raid10]
md2 : active raid1 sdb3[1] sda3[0]
634795008 blocks super 1.2 [2/2] [UU]
bitmap: 0/5 pages [0KB], 65536KB chunk
md1 : active raid1 sdb2[1] sda2[0]
126887936 blocks super 1.2 [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk
md0 : active raid1 sdb1[2] sda1[0]
19514368 blocks super 1.2 [2/2] [UU]
Thank you.
Pete
-----Original Message-----
From: Wols Lists [mailto:antlists@youngman.org.uk]
Sent: Friday, September 23, 2016 11:43 AM
To: WNSDEV; linux-raid@vger.kernel.org
Subject: Re: Linux raid wiki
Thanks.
My understanding of role numbers is that they are the order of the disks in which stripes are written, so role 0 has the first stripe, role 1 the second, etc etc.
This is *normally* irrelevant, but should the array ever get trashed, the disks need to be listed *in* *that* *order* in a new --create statement.
Can someone - Neil? Phil? - please confirm I've understood that correctly before I update anything ...
(and I will be posting updates to the list - anything I'm not sure of I will be asking here to check I get it right :-)
Cheers,
Wol
On 23/09/16 18:35, WNSDEV wrote:
> Hi Wol,
>
> Thanks for your work on the wiki... I'm sending a virtual bouquet. I've been tracking down an issue regarding role numbers which I've posted to the list http://marc.info/?l=linux-raid&m=147439325405842&w=2. In that process I've come across other postings that say information in the wiki is incorrect so maybe this is an opportunity to update this part of the wiki.
>
> According to https://raid.wiki.kernel.org/index.php/Mdstat we find:
>
> "The raid role numbers [#] following each device indicate its role, or function, within the raid set. Any device with "n" or higher are spare disks. 0,1,..,n-1 are for the working array. Notice that there is no device 3."
>
> But according to the following two postings,
> http://www.spinics.net/lists/raid/msg44491.html and
> http://www.spinics.net/lists/raid/msg49766.html
> the information in the wiki about the numbers in brackets (role numbers ) is wrong.
>
> Thanks,
> Peter Sangas
>
^ permalink raw reply
* Re: Linux raid wiki
From: Wols Lists @ 2016-09-23 18:43 UTC (permalink / raw)
To: WNSDEV, linux-raid
In-Reply-To: <002a01d215c0$e3f20980$abd61c80$@wnsdev.com>
Thanks.
My understanding of role numbers is that they are the order of the disks
in which stripes are written, so role 0 has the first stripe, role 1 the
second, etc etc.
This is *normally* irrelevant, but should the array ever get trashed,
the disks need to be listed *in* *that* *order* in a new --create statement.
Can someone - Neil? Phil? - please confirm I've understood that
correctly before I update anything ...
(and I will be posting updates to the list - anything I'm not sure of I
will be asking here to check I get it right :-)
Cheers,
Wol
On 23/09/16 18:35, WNSDEV wrote:
> Hi Wol,
>
> Thanks for your work on the wiki... I'm sending a virtual bouquet. I've been tracking down an issue regarding role numbers which I've posted to the list http://marc.info/?l=linux-raid&m=147439325405842&w=2. In that process I've come across other postings that say information in the wiki is incorrect so maybe this is an opportunity to update this part of the wiki.
>
> According to https://raid.wiki.kernel.org/index.php/Mdstat we find:
>
> "The raid role numbers [#] following each device indicate its role, or function, within the raid set. Any device with "n" or higher are spare disks. 0,1,..,n-1 are for the working array. Notice that there is no device 3."
>
> But according to the following two postings, http://www.spinics.net/lists/raid/msg44491.html and http://www.spinics.net/lists/raid/msg49766.html
> the information in the wiki about the numbers in brackets (role numbers ) is wrong.
>
> Thanks,
> Peter Sangas
>
^ permalink raw reply
* RE: Linux raid wiki
From: WNSDEV @ 2016-09-23 17:35 UTC (permalink / raw)
To: 'Wols Lists', linux-raid
In-Reply-To: <57E469E0.4020005@youngman.org.uk>
Hi Wol,
Thanks for your work on the wiki... I'm sending a virtual bouquet. I've been tracking down an issue regarding role numbers which I've posted to the list http://marc.info/?l=linux-raid&m=147439325405842&w=2. In that process I've come across other postings that say information in the wiki is incorrect so maybe this is an opportunity to update this part of the wiki.
According to https://raid.wiki.kernel.org/index.php/Mdstat we find:
"The raid role numbers [#] following each device indicate its role, or function, within the raid set. Any device with "n" or higher are spare disks. 0,1,..,n-1 are for the working array. Notice that there is no device 3."
But according to the following two postings, http://www.spinics.net/lists/raid/msg44491.html and http://www.spinics.net/lists/raid/msg49766.html
the information in the wiki about the numbers in brackets (role numbers ) is wrong.
Thanks,
Peter Sangas
-----Original Message-----
From: Wols Lists [mailto:antlists@youngman.org.uk]
Sent: Thursday, September 22, 2016 4:32 PM
To: linux-raid@vger.kernel.org
Subject: Linux raid wiki
I've finally got myself access to the wiki :-) and have been working on bringing it a bit more up-to-date.
I've somewhat updated the overview section, so by all means be critical, but it's not mostly my work.
I've added the "When Things Go Wrogn" section, but so far only the first two pages - "Asking for help" and "Timeout Mismatch" - are all my work.
The other three pages were already there, but I moved them here because I felt they belonged here.
Please feel free to criticize it (or offer bouquets :-), and give advice as how to improve things, either in private email or on the list.
I know we're a friendly bunch, but so many people seem to keep coming for the same problems over and over again, and I want this to be an up-to-date resource we can point people at to try and help them help themselves.
(And if you think any stuff is out-of-date and misleading, tell me. I'm planning to create a "softtware archaeology" section for all the stuff that is no longer relevant, but might be wanted by people looking after old systems.)
Cheers,
Wol
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: RAID5 grow interrupted.
From: Axel Spallek @ 2016-09-23 5:23 UTC (permalink / raw)
To: Andreas Klauer, linux-raid
In-Reply-To: <20160922194517.GA6019@metamorpher.de>
This one did it, but not the first time I issued this command. Maybe
--assemble --scan was needed first.
Thanks to all!
root@s10:~# mdadm --assemble /dev/md1 /dev/sd[a-h]1
--backup-file=/var/backups/mdadm.backup
mdadm: Need to backup 21504K of critical section..
mdadm: /dev/md1 has been started with 8 drives.
root@s10:~# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md1 : active raid5 sdh1[0] sda1[5] sdb1[6] sdc1[7] sdd1[8] sdf1[4]
sde1[2] sdg1[1]
23441292288 blocks super 1.2 level 5, 512k chunk, algorithm 2
[7/7] [UUUUUUU]
[>....................] reshape = 4.0% (159293440/3906882048)
finish=50748595.7min speed=0K/sec
unused devices: <none>
Aksels
Am 22.09.2016 um 21:45 schrieb Andreas Klauer:
> On Thu, Sep 22, 2016 at 08:41:59PM +0200, Axel Spallek wrote:
>> root@s10:~# mdadm --assemble /dev/md1 /dev/sd[a-h]1
>> mdadm: /dev/md1: Need a backup file to complete reshape of this array.
>> mdadm: Please provided one with "--backup-file=..."
> I assume it was assembled anyway at this point so --stop again...
>
>> root@s10:~# mdadm --assemble /dev/md1 /dev/sd[a-h]1
>> --backup-file=/var/backups/mdadm.backup
>> mdadm: /dev/sda1 is busy - skipping
>> mdadm: /dev/sdb1 is busy - skipping
> regards
> Andreas Klauer
--
Mit freundlichem Gruß,
Axel Spallek
Dipl.-Ing. FH IE
Hochdorfer Straße 34
88477 Schönebürg
Mobil: 01577 7929886
E-Mail: axel@spallek.org
Bankverbindung:
Volksbank Schwendi
IBAN: DE20654913200095195009
BIC: GENODES1VBL
Steuernummer: 5438000091
UST.-ID.: DE290536647
^ permalink raw reply
* RE: [PATCH v2] raid6/test/test.c: bug fix: Specify aligned(alignment) attributes to the char arrays
From: Kammela, Gayatri @ 2016-09-23 0:18 UTC (permalink / raw)
To: linux-raid@vger.kernel.org, linux-kernel@vger.kernel.org
Cc: shli@kernel.org, Anvin, H Peter, Shankar, Ravi V, Yu, Fenghua,
H . Peter Anvin, Yu, Yu-cheng
In-Reply-To: <1474589275-12045-1-git-send-email-gayatri.kammela@intel.com>
Hi all,
Sorry for the noise! I didn't mean to send the version2.
-----Original Message-----
From: Kammela, Gayatri
Sent: Thursday, September 22, 2016 5:08 PM
To: linux-raid@vger.kernel.org; linux-kernel@vger.kernel.org
Cc: shli@kernel.org; Anvin, H Peter <h.peter.anvin@intel.com>; Shankar, Ravi V <ravi.v.shankar@intel.com>; Yu, Fenghua <fenghua.yu@intel.com>; Kammela, Gayatri <gayatri.kammela@intel.com>; H . Peter Anvin <hpa@zytor.com>; Yu, Yu-cheng <yu-cheng.yu@intel.com>
Subject: [PATCH v2] raid6/test/test.c: bug fix: Specify aligned(alignment) attributes to the char arrays
Specifying the aligned attributes to the char recovi[PAGE_SIZE] and char recovi[PAGE_SIZE] arrays, so that all malloc memory is page boundary aligned.
Without these alignment attributes, the test causes a segfault in userspace when the NDISKS are changed to 4 from 16.
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Yu-cheng Yu <yu-cheng.yu@intel.com>
Signed-off-by: Gayatri Kammela <gayatri.kammela@intel.com>
---
lib/raid6/test/test.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/lib/raid6/test/test.c b/lib/raid6/test/test.c index 3bebbabdb510..32a00f11ac50 100644
--- a/lib/raid6/test/test.c
+++ b/lib/raid6/test/test.c
@@ -21,12 +21,13 @@
#define NDISKS 16 /* Including P and Q */
-const char raid6_empty_zero_page[PAGE_SIZE] __attribute__((aligned(256)));
+const char raid6_empty_zero_page[PAGE_SIZE]
+__attribute__((aligned(PAGE_SIZE)));
struct raid6_calls raid6_call;
char *dataptrs[NDISKS];
char data[NDISKS][PAGE_SIZE];
-char recovi[PAGE_SIZE], recovj[PAGE_SIZE];
+char recovi[PAGE_SIZE] __attribute__((aligned(PAGE_SIZE)));
+char recovj[PAGE_SIZE] __attribute__((aligned(PAGE_SIZE)));
static void makedata(int start, int stop) {
--
2.7.4
^ permalink raw reply
* [PATCH] raid6/test/test.c: bug fix: Specify aligned(alignment) attributes to the char arrays
From: Gayatri Kammela @ 2016-09-23 0:14 UTC (permalink / raw)
To: linux-raid, linux-kernel
Cc: shli, h.peter.anvin, ravi.v.shankar, fenghua.yu, Gayatri Kammela,
H . Peter Anvin, Yu-cheng Yu
Specifying the aligned attributes to the char recovi[PAGE_SIZE]
and char recovi[PAGE_SIZE] arrays, so that all malloc memory is page
boundary aligned.
Without these alignment attributes, the test causes a segfault in
userspace when the NDISKS are changed to 4 from 16.
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Yu-cheng Yu <yu-cheng.yu@intel.com>
Signed-off-by: Gayatri Kammela <gayatri.kammela@intel.com>
Reviewed-by: H. Peter Anvin <hpa@linux.intel.com>
---
lib/raid6/test/test.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/lib/raid6/test/test.c b/lib/raid6/test/test.c
index 3bebbabdb510..32a00f11ac50 100644
--- a/lib/raid6/test/test.c
+++ b/lib/raid6/test/test.c
@@ -21,12 +21,13 @@
#define NDISKS 16 /* Including P and Q */
-const char raid6_empty_zero_page[PAGE_SIZE] __attribute__((aligned(256)));
+const char raid6_empty_zero_page[PAGE_SIZE] __attribute__((aligned(PAGE_SIZE)));
struct raid6_calls raid6_call;
char *dataptrs[NDISKS];
char data[NDISKS][PAGE_SIZE];
-char recovi[PAGE_SIZE], recovj[PAGE_SIZE];
+char recovi[PAGE_SIZE] __attribute__((aligned(PAGE_SIZE)));
+char recovj[PAGE_SIZE] __attribute__((aligned(PAGE_SIZE)));
static void makedata(int start, int stop)
{
--
2.7.4
^ permalink raw reply related
* [PATCH v2] raid6/test/test.c: bug fix: Specify aligned(alignment) attributes to the char arrays
From: Gayatri Kammela @ 2016-09-23 0:07 UTC (permalink / raw)
To: linux-raid, linux-kernel
Cc: shli, h.peter.anvin, ravi.v.shankar, fenghua.yu, Gayatri Kammela,
H . Peter Anvin, Yu-cheng Yu
Specifying the aligned attributes to the char recovi[PAGE_SIZE]
and char recovi[PAGE_SIZE] arrays, so that all malloc memory is page
boundary aligned.
Without these alignment attributes, the test causes a segfault in
userspace when the NDISKS are changed to 4 from 16.
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Yu-cheng Yu <yu-cheng.yu@intel.com>
Signed-off-by: Gayatri Kammela <gayatri.kammela@intel.com>
---
lib/raid6/test/test.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/lib/raid6/test/test.c b/lib/raid6/test/test.c
index 3bebbabdb510..32a00f11ac50 100644
--- a/lib/raid6/test/test.c
+++ b/lib/raid6/test/test.c
@@ -21,12 +21,13 @@
#define NDISKS 16 /* Including P and Q */
-const char raid6_empty_zero_page[PAGE_SIZE] __attribute__((aligned(256)));
+const char raid6_empty_zero_page[PAGE_SIZE] __attribute__((aligned(PAGE_SIZE)));
struct raid6_calls raid6_call;
char *dataptrs[NDISKS];
char data[NDISKS][PAGE_SIZE];
-char recovi[PAGE_SIZE], recovj[PAGE_SIZE];
+char recovi[PAGE_SIZE] __attribute__((aligned(PAGE_SIZE)));
+char recovj[PAGE_SIZE] __attribute__((aligned(PAGE_SIZE)));
static void makedata(int start, int stop)
{
--
2.7.4
^ permalink raw reply related
* Linux raid wiki
From: Wols Lists @ 2016-09-22 23:31 UTC (permalink / raw)
To: linux-raid
I've finally got myself access to the wiki :-) and have been working on
bringing it a bit more up-to-date.
I've somewhat updated the overview section, so by all means be critical,
but it's not mostly my work.
I've added the "When Things Go Wrogn" section, but so far only the first
two pages - "Asking for help" and "Timeout Mismatch" - are all my work.
The other three pages were already there, but I moved them here because
I felt they belonged here.
Please feel free to criticize it (or offer bouquets :-), and give advice
as how to improve things, either in private email or on the list.
I know we're a friendly bunch, but so many people seem to keep coming
for the same problems over and over again, and I want this to be an
up-to-date resource we can point people at to try and help them help
themselves.
(And if you think any stuff is out-of-date and misleading, tell me. I'm
planning to create a "softtware archaeology" section for all the stuff
that is no longer relevant, but might be wanted by people looking after
old systems.)
Cheers,
Wol
^ permalink raw reply
* Re: RAID5 grow interrupted.
From: Andreas Klauer @ 2016-09-22 19:45 UTC (permalink / raw)
To: Axel Spallek; +Cc: Adam Goryachev, linux-raid
In-Reply-To: <af2cbd12-ba6c-0e39-ba1d-651feda5d07c@spallek.org>
On Thu, Sep 22, 2016 at 08:41:59PM +0200, Axel Spallek wrote:
> root@s10:~# mdadm --assemble /dev/md1 /dev/sd[a-h]1
> mdadm: /dev/md1: Need a backup file to complete reshape of this array.
> mdadm: Please provided one with "--backup-file=..."
I assume it was assembled anyway at this point so --stop again...
> root@s10:~# mdadm --assemble /dev/md1 /dev/sd[a-h]1
> --backup-file=/var/backups/mdadm.backup
> mdadm: /dev/sda1 is busy - skipping
> mdadm: /dev/sdb1 is busy - skipping
regards
Andreas Klauer
^ permalink raw reply
* Re: RAID5 grow interrupted.
From: Axel Spallek @ 2016-09-22 18:59 UTC (permalink / raw)
To: Andreas Klauer, linux-raid
In-Reply-To: <20160922171915.GA5392@metamorpher.de>
Am 22.09.2016 um 19:19 schrieb Andreas Klauer:
> On Thu, Sep 22, 2016 at 04:11:28PM +0200, Axel Spallek wrote:
>> I tried to get a disk out of a RAID5 with 8 hdds (4TB) to get a Hotspare
>> to change to RAID6 afterwards.
> I'm a bit confused here.
>
> You started out with 8 disk RAID-5? And final result is to be what?
> RAID-6 with 8 disks?
Correct.
>
>> Therefore I issued the following commands:
>>
>> mdadm --grow -n7 /dev/md1 <-- just to get the size for the next command.
>>
>> mdadm --grow /dev/md1 --array-size 23441292288
>>
>> mdadm --grow -n7 /dev/md1 --backup-file /var/backups/mdadm.backup
> I'm not sure, but you might not have needed any of these intermediate steps.
Hmmm. Had RAID5 with 8 disks. To convert to RAID6 you need a Hotspare,
but all my drive-cage is full.
> But now you started it you might have to finish them.
> That said there is a reshape revert (which is not yet in the manpage?)
> which might help if the regular reshape just won't resume.
Yes. That info would help.
>
>> root@s10:~# mdadm -A --scan --verbose
>> mdadm: looking for devices for further assembly
>> mdadm: /dev/sdg1 is busy - skipping
> It says busy because already assembled (or in use by something else).
> So you have to stop first before assembeling again.
>
> I didn't see anything out of place in your --examine.
> It seems to be in sync, event count identical, ....
>
> Good luck
> Andreas Klauer
--
Mit freundlichem Gruß,
Axel Spallek
Dipl.-Ing. FH IE
Hochdorfer Straße 34
88477 Schönebürg
Mobil: 01577 7929886
E-Mail: axel@spallek.org
Bankverbindung:
Volksbank Schwendi
IBAN: DE20654913200095195009
BIC: GENODES1VBL
Steuernummer: 5438000091
UST.-ID.: DE290536647
^ permalink raw reply
* Re: RAID5 grow interrupted.
From: Axel Spallek @ 2016-09-22 18:44 UTC (permalink / raw)
To: Wols Lists, linux-raid
In-Reply-To: <57E3F62E.10706@youngman.org.uk>
root@s10:~# mdadm --stop /dev/md1
mdadm: stopped /dev/md1
root@s10:~#
root@s10:~# mdadm --assemble --scan
mdadm: /dev/md/1: Need a backup file to complete reshape of this array.
mdadm: Please provided one with "--backup-file=..."
mdadm: No arrays found in config file or automatically
root@s10:~# mdadm --assemble --scan --backup-file=/var/backups/mdadm.backup
mdadm: --backup_file not meaningful with a --scan assembly.
Am 22.09.2016 um 17:18 schrieb Wols Lists:
> mdadm --assemble --scan
^ permalink raw reply
* Re: RAID5 grow interrupted.
From: Axel Spallek @ 2016-09-22 18:41 UTC (permalink / raw)
To: Adam Goryachev, linux-raid
In-Reply-To: <9c618b69-1e84-60a8-d25c-55171058c855@websitemanagers.com.au>
Here the output:
root@s10:~# mdadm --stop /dev/md1
mdadm: stopped /dev/md1
root@s10:~# cat /proc/mdstat
Personalities :
unused devices: <none>
root@s10:~# mdadm --assemble /dev/md1 /dev/sd[a-h]1
mdadm: /dev/md1: Need a backup file to complete reshape of this array.
mdadm: Please provided one with "--backup-file=..."
root@s10:~# mdadm --assemble /dev/md1 /dev/sd[a-h]1
--backup-file=/var/backups/mdadm.backup
mdadm: /dev/sda1 is busy - skipping
mdadm: /dev/sdb1 is busy - skipping
mdadm: /dev/sdc1 is busy - skipping
mdadm: /dev/sdd1 is busy - skipping
mdadm: /dev/sde1 is busy - skipping
mdadm: /dev/sdf1 is busy - skipping
mdadm: /dev/sdg1 is busy - skipping
mdadm: /dev/sdh1 is busy - skipping
root@s10:~# cat /proc/mdstat
Personalities :
md1 : inactive sdh1[0](S) sda1[5](S) sdb1[6](S) sdc1[7](S) sdd1[8](S)
sdf1[4](S) sde1[2](S) sdg1[1](S)
31255059140 blocks super 1.2
unused devices: <none>
Am 22.09.2016 um 16:27 schrieb Adam Goryachev:
>
>
> On 23/09/2016 00:11, Axel Spallek wrote:
>> Hello there.
>>
>> I did something wrong.
>>
>> I tried to get a disk out of a RAID5 with 8 hdds (4TB) to get a
>> Hotspare to change to RAID6 afterwards.
>>
>> The RAID was clean and not rebuilding before I started.
>>
>> The partition was only 11TB big. Not yet resized, because not created
>> with 64Bit, which I wanted to do afterwards.
>>
>> Therefore I issued the following commands:
>>
>> mdadm --grow -n7 /dev/md1 <-- just to get the size for the next
>> command.
>>
>> mdadm --grow /dev/md1 --array-size 23441292288
>>
>> mdadm --grow -n7 /dev/md1 --backup-file /var/backups/mdadm.backup
>> UST.-ID.: DE290536647
^ permalink raw reply
* Re: RAID5 grow interrupted.
From: Andreas Klauer @ 2016-09-22 17:19 UTC (permalink / raw)
To: Axel Spallek; +Cc: linux-raid
In-Reply-To: <36947ef9-64e2-f084-4949-476107e2771e@spallek.org>
On Thu, Sep 22, 2016 at 04:11:28PM +0200, Axel Spallek wrote:
> I tried to get a disk out of a RAID5 with 8 hdds (4TB) to get a Hotspare
> to change to RAID6 afterwards.
I'm a bit confused here.
You started out with 8 disk RAID-5? And final result is to be what?
RAID-6 with 8 disks?
> Therefore I issued the following commands:
>
> mdadm --grow -n7 /dev/md1 <-- just to get the size for the next command.
>
> mdadm --grow /dev/md1 --array-size 23441292288
>
> mdadm --grow -n7 /dev/md1 --backup-file /var/backups/mdadm.backup
I'm not sure, but you might not have needed any of these intermediate steps.
But now you started it you might have to finish them.
That said there is a reshape revert (which is not yet in the manpage?)
which might help if the regular reshape just won't resume.
> root@s10:~# mdadm -A --scan --verbose
> mdadm: looking for devices for further assembly
> mdadm: /dev/sdg1 is busy - skipping
It says busy because already assembled (or in use by something else).
So you have to stop first before assembeling again.
I didn't see anything out of place in your --examine.
It seems to be in sync, event count identical, ....
Good luck
Andreas Klauer
^ permalink raw reply
* Re: 95a05b3 broke mdadm --add on my superblock 1.0 array
From: Anthony DeRobertis @ 2016-09-22 16:53 UTC (permalink / raw)
To: Guoqing Jiang; +Cc: linux-raid, 837964
In-Reply-To: <57E34496.8020108@suse.com>
On Wed, Sep 21, 2016 at 10:40:22PM -0400, Guoqing Jiang wrote:
>
> Does the below change work? Thanks.
Yes, this patch (after un-email-mangling it) fixes it.
>
> diff --git a/super1.c b/super1.c
> index 9f62d23..6a0b075 100644
> --- a/super1.c
> +++ b/super1.c
> @@ -2433,7 +2433,10 @@ static int write_bitmap1(struct supertype *st, int
> fd, enum bitmap_update update
> memset(buf, 0xff, 4096);
> memcpy(buf, (char *)bms, sizeof(bitmap_super_t));
>
> - towrite = calc_bitmap_size(bms, 4096);
> + if (__le32_to_cpu(bms->nodes) == 0)
> + towrite = calc_bitmap_size(bms, 512);
> + else
> + towrite = calc_bitmap_size(bms, 4096);
> while (towrite > 0) {
>
> Regards,
> Guoqing
^ permalink raw reply
* Re: [PATCH v2] Fix RAID metadata check
From: Jes Sorensen @ 2016-09-22 15:35 UTC (permalink / raw)
To: Mariusz Dabrowski
Cc: linux-raid, tomasz.majchrzak, aleksey.obitotskiy, pawel.baldysiak,
artur.paszkiewicz, maksymilian.kunt
In-Reply-To: <1474527731-29542-1-git-send-email-mariusz.dabrowski@intel.com>
Mariusz Dabrowski <mariusz.dabrowski@intel.com> writes:
> mdadm recognizes devices with partition table as part of an RAID array
> and invalid warning message is displayed. After this fix proper warning
> messages are being displayed for MBR/GPT disks and devices with RAID
> metadata.
>
> Signed-off-by: Mariusz Dabrowski <mariusz.dabrowski@intel.com>
> ---
> util.c | 27 ++++++++++++++++-----------
> 1 file changed, 16 insertions(+), 11 deletions(-)
Looks good, applied!
Thanks,
Jes
> diff --git a/util.c b/util.c
> index 8b52242..a238a21 100644
> --- a/util.c
> +++ b/util.c
> @@ -710,17 +710,22 @@ int check_raid(int fd, char *name)
>
> if (!st)
> return 0;
> - st->ss->load_super(st, fd, name);
> - /* Looks like a raid array .. */
> - pr_err("%s appears to be part of a raid array:\n",
> - name);
> - st->ss->getinfo_super(st, &info, NULL);
> - st->ss->free_super(st);
> - crtime = info.array.ctime;
> - level = map_num(pers, info.array.level);
> - if (!level) level = "-unknown-";
> - cont_err("level=%s devices=%d ctime=%s",
> - level, info.array.raid_disks, ctime(&crtime));
> + if (st->ss->add_to_super != NULL) {
> + st->ss->load_super(st, fd, name);
> + /* Looks like a raid array .. */
> + pr_err("%s appears to be part of a raid array:\n", name);
> + st->ss->getinfo_super(st, &info, NULL);
> + st->ss->free_super(st);
> + crtime = info.array.ctime;
> + level = map_num(pers, info.array.level);
> + if (!level)
> + level = "-unknown-";
> + cont_err("level=%s devices=%d ctime=%s",
> + level, info.array.raid_disks, ctime(&crtime));
> + } else {
> + /* Looks like GPT or MBR */
> + pr_err("partition table exists on %s\n", name);
> + }
> return 1;
> }
^ permalink raw reply
* Re: RAID5 grow interrupted.
From: Wols Lists @ 2016-09-22 15:18 UTC (permalink / raw)
To: Axel Spallek, linux-raid
In-Reply-To: <36947ef9-64e2-f084-4949-476107e2771e@spallek.org>
On 22/09/16 15:11, Axel Spallek wrote:
> The server came up again, but without /dev/md1.
>
> I had made a Backup, which is 2 days old. Not so bad, because I have the
> seafile data on several computers. But to get the RAID back to work
> would be better.
>
> How do I restart the rebuild process with the backup-file?
>
> This is what I get in the console:
>
> root@s10:~# cat /proc/mdstat
> Personalities :
> md1 : inactive sdh1[0](S) sda1[5](S) sdb1[6](S) sdc1[7](S) sdd1[8](S)
> sdf1[4](S) sde1[2](S) sdg1[1](S)
> 31255059140 blocks super 1.2
>
> unused devices: <none>
Okay, quick response here. Won't do any damage, might work.
Stop and reassemble the array ...
mdadm --stop /dev/md1
mdadm --assemble --scan
What you describe sounds like something tried to start the array while
it was half-assembled. There's a bunch of interactions between udev,
systemd, and mdadm, which seem to get badly off-kilter if something
hiccups anywhere.
My suggestion won't do any harm - it might fix the problem, and it'll
give the experts the chance to chime in.
Cheers,
Wol
^ permalink raw reply
* RAID5 grow interrupted.
From: Axel Spallek @ 2016-09-22 14:11 UTC (permalink / raw)
To: linux-raid
Hello there.
I did something wrong.
I tried to get a disk out of a RAID5 with 8 hdds (4TB) to get a Hotspare
to change to RAID6 afterwards.
The RAID was clean and not rebuilding before I started.
The partition was only 11TB big. Not yet resized, because not created
with 64Bit, which I wanted to do afterwards.
Therefore I issued the following commands:
mdadm --grow -n7 /dev/md1 <-- just to get the size for the next command.
mdadm --grow /dev/md1 --array-size 23441292288
mdadm --grow -n7 /dev/md1 --backup-file /var/backups/mdadm.backup
The RAID /dev/md1 is mounted on /srv, so the backupfile is safe.
After a some time, someone told me, that seafile does not work. Since I
was in a hurry, I just rebootet the server and forgot the RAID rebuild.
The server came up again, but without /dev/md1.
I had made a Backup, which is 2 days old. Not so bad, because I have the
seafile data on several computers. But to get the RAID back to work
would be better.
How do I restart the rebuild process with the backup-file?
This is what I get in the console:
root@s10:~# cat /proc/mdstat
Personalities :
md1 : inactive sdh1[0](S) sda1[5](S) sdb1[6](S) sdc1[7](S) sdd1[8](S)
sdf1[4](S) sde1[2](S) sdg1[1](S)
31255059140 blocks super 1.2
unused devices: <none>
root@s10:~# mdadm -A --scan --verbose
mdadm: looking for devices for further assembly
mdadm: /dev/sdg1 is busy - skipping
mdadm: Cannot assemble mbr metadata on /dev/sdg
mdadm: /dev/sdd1 is busy - skipping
mdadm: Cannot assemble mbr metadata on /dev/sdd
mdadm: /dev/sdc1 is busy - skipping
mdadm: Cannot assemble mbr metadata on /dev/sdc
mdadm: /dev/sda1 is busy - skipping
mdadm: Cannot assemble mbr metadata on /dev/sda
mdadm: /dev/sdb1 is busy - skipping
mdadm: Cannot assemble mbr metadata on /dev/sdb
mdadm: /dev/sdf1 is busy - skipping
mdadm: Cannot assemble mbr metadata on /dev/sdf
mdadm: /dev/sdh1 is busy - skipping
mdadm: Cannot assemble mbr metadata on /dev/sdh
mdadm: /dev/sde1 is busy - skipping
mdadm: Cannot assemble mbr metadata on /dev/sde
mdadm: no recogniseable superblock on /dev/sdj5
mdadm: Cannot assemble mbr metadata on /dev/sdj2
mdadm: no recogniseable superblock on /dev/sdj1
mdadm: Cannot assemble mbr metadata on /dev/sdj
mdadm: no recogniseable superblock on /dev/sdi1
mdadm: Cannot assemble mbr metadata on /dev/sdi
mdadm: No arrays found in config file or automatically
root@s10:~# mdadm --examine --scan
ARRAY /dev/md/1 metadata=1.2 UUID=48f60e15:900f47cc:6c5f42b1:82f01530
name=s10:1
root@s10:~# mdadm --examine /dev/sd*
/dev/sda:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sda1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : 48f60e15:900f47cc:6c5f42b1:82f01530
Name : s10:1 (local to host s10)
Creation Time : Fri Sep 16 06:59:32 2016
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7813764785 (3725.89 GiB 4000.65 GB)
Array Size : 23441292288 (22355.36 GiB 24003.88 GB)
Used Dev Size : 7813764096 (3725.89 GiB 4000.65 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=689 sectors
State : clean
Device UUID : 9fe5980b:be2beb6b:59537ad1:90091564
Reshape pos'n : 22485531648 (21443.87 GiB 23025.18 GB)
Delta Devices : -1 (8->7)
Update Time : Thu Sep 22 14:23:54 2016
Checksum : 1ea679f3 - correct
Events : 180344
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 7
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : 48f60e15:900f47cc:6c5f42b1:82f01530
Name : s10:1 (local to host s10)
Creation Time : Fri Sep 16 06:59:32 2016
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7813764785 (3725.89 GiB 4000.65 GB)
Array Size : 23441292288 (22355.36 GiB 24003.88 GB)
Used Dev Size : 7813764096 (3725.89 GiB 4000.65 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=689 sectors
State : clean
Device UUID : 370c5540:2d3bdd3e:40a36449:b82309a8
Reshape pos'n : 22485531648 (21443.87 GiB 23025.18 GB)
Delta Devices : -1 (8->7)
Update Time : Thu Sep 22 14:23:54 2016
Checksum : e2331a09 - correct
Events : 180344
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 6
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : 48f60e15:900f47cc:6c5f42b1:82f01530
Name : s10:1 (local to host s10)
Creation Time : Fri Sep 16 06:59:32 2016
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7813764785 (3725.89 GiB 4000.65 GB)
Array Size : 23441292288 (22355.36 GiB 24003.88 GB)
Used Dev Size : 7813764096 (3725.89 GiB 4000.65 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=689 sectors
State : clean
Device UUID : a9e8caf9:4c70d937:50e55bdf:b736cb97
Reshape pos'n : 22485531648 (21443.87 GiB 23025.18 GB)
Delta Devices : -1 (8->7)
Update Time : Thu Sep 22 14:23:54 2016
Checksum : 1a5e80ac - correct
Events : 180344
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 5
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : 48f60e15:900f47cc:6c5f42b1:82f01530
Name : s10:1 (local to host s10)
Creation Time : Fri Sep 16 06:59:32 2016
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7813764785 (3725.89 GiB 4000.65 GB)
Array Size : 23441292288 (22355.36 GiB 24003.88 GB)
Used Dev Size : 7813764096 (3725.89 GiB 4000.65 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=689 sectors
State : clean
Device UUID : 50a6466d:057b9171:22865989:cabb59ce
Reshape pos'n : 22485531648 (21443.87 GiB 23025.18 GB)
Delta Devices : -1 (8->7)
Update Time : Thu Sep 22 14:23:54 2016
Checksum : a81e6ef1 - correct
Events : 180344
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : 48f60e15:900f47cc:6c5f42b1:82f01530
Name : s10:1 (local to host s10)
Creation Time : Fri Sep 16 06:59:32 2016
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7813764785 (3725.89 GiB 4000.65 GB)
Array Size : 23441292288 (22355.36 GiB 24003.88 GB)
Used Dev Size : 7813764096 (3725.89 GiB 4000.65 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=689 sectors
State : clean
Device UUID : 1fd7fdde:3220d053:1cad772f:508ed8a7
Reshape pos'n : 22485531648 (21443.87 GiB 23025.18 GB)
Delta Devices : -1 (8->7)
Update Time : Thu Sep 22 14:23:54 2016
Checksum : 7bb13e67 - correct
Events : 180344
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : 48f60e15:900f47cc:6c5f42b1:82f01530
Name : s10:1 (local to host s10)
Creation Time : Fri Sep 16 06:59:32 2016
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7813764785 (3725.89 GiB 4000.65 GB)
Array Size : 23441292288 (22355.36 GiB 24003.88 GB)
Used Dev Size : 7813764096 (3725.89 GiB 4000.65 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=689 sectors
State : clean
Device UUID : b2788315:a75fec1f:1d2681ee:2bba1be7
Reshape pos'n : 22485531648 (21443.87 GiB 23025.18 GB)
Delta Devices : -1 (8->7)
Update Time : Thu Sep 22 14:23:54 2016
Checksum : 7c9fc44d - correct
Events : 180344
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdg1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : 48f60e15:900f47cc:6c5f42b1:82f01530
Name : s10:1 (local to host s10)
Creation Time : Fri Sep 16 06:59:32 2016
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7813764785 (3725.89 GiB 4000.65 GB)
Array Size : 23441292288 (22355.36 GiB 24003.88 GB)
Used Dev Size : 7813764096 (3725.89 GiB 4000.65 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=689 sectors
State : clean
Device UUID : 4e7223d6:4416983d:7812788b:a2114dec
Reshape pos'n : 22485531648 (21443.87 GiB 23025.18 GB)
Delta Devices : -1 (8->7)
Update Time : Thu Sep 22 14:23:54 2016
Checksum : fd13b855 - correct
Events : 180344
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdh:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdh1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : 48f60e15:900f47cc:6c5f42b1:82f01530
Name : s10:1 (local to host s10)
Creation Time : Fri Sep 16 06:59:32 2016
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7813764785 (3725.89 GiB 4000.65 GB)
Array Size : 23441292288 (22355.36 GiB 24003.88 GB)
Used Dev Size : 7813764096 (3725.89 GiB 4000.65 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=689 sectors
State : clean
Device UUID : c1226e5c:4145cbcf:0bbe6160:4e0da07b
Reshape pos'n : 22485531648 (21443.87 GiB 23025.18 GB)
Delta Devices : -1 (8->7)
Update Time : Thu Sep 22 14:23:54 2016
Checksum : 79ce3f03 - correct
Events : 180344
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdi:
MBR Magic : aa55
Partition[0] : 500109312 sectors at 2048 (type 83)
mdadm: No md superblock detected on /dev/sdi1.
/dev/sdj:
MBR Magic : aa55
Partition[0] : 56143872 sectors at 2048 (type 83)
Partition[1] : 2478082 sectors at 56147966 (type 05)
mdadm: No md superblock detected on /dev/sdj1.
/dev/sdj2:
MBR Magic : aa55
Partition[0] : 2478080 sectors at 2 (type 82)
mdadm: No md superblock detected on /dev/sdj5.
root@s10:~# mdadm --detail /dev/md1
/dev/md1:
Version : 1.2
Raid Level : raid0
Total Devices : 8
Persistence : Superblock is persistent
State : inactive
Delta Devices : -1, (1->0)
New Level : raid5
New Layout : left-symmetric
New Chunksize : 512K
Name : s10:1 (local to host s10)
UUID : 48f60e15:900f47cc:6c5f42b1:82f01530
Events : 180344
Number Major Minor RaidDevice
- 8 1 - /dev/sda1
- 8 17 - /dev/sdb1
- 8 33 - /dev/sdc1
- 8 49 - /dev/sdd1
- 8 65 - /dev/sde1
- 8 81 - /dev/sdf1
- 8 97 - /dev/sdg1
- 8 113 - /dev/sdh1
--
Mit freundlichem Gruß,
Axel Spallek
Dipl.-Ing. FH IE
Hochdorfer Straße 34
88477 Schönebürg
Mobil: 01577 7929886
E-Mail: axel@spallek.org
Bankverbindung:
Volksbank Schwendi
IBAN: DE20654913200095195009
BIC: GENODES1VBL
Steuernummer: 5438000091
UST.-ID.: DE290536647
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox