* Re: [PATCH v3 5/8] md: don't allow resize/reshape with cache support
From: Neil Brown @ 2015-06-18 1:16 UTC (permalink / raw)
To: Shaohua Li; +Cc: linux-raid, Kernel-team, songliubraving, hch, dan.j.williams
In-Reply-To: <818a40768c716d0d553d55a7cc7b382ebcd7d036.1433356864.git.shli@fb.com>
On Wed, 3 Jun 2015 15:48:40 -0700
Shaohua Li <shli@fb.com> wrote:
> If cache support is enabled, don't allow resize/reshape in current
> stage. In the future, we can flush all data from cache to raid before
> resize/reshape and then allow resize/reshape.
>
> Signed-off-by: Shaohua Li <shli@fb.com>
> ---
> drivers/md/raid5.c | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> index 26561d8..29f49c7 100644
> --- a/drivers/md/raid5.c
> +++ b/drivers/md/raid5.c
> @@ -7207,6 +7207,10 @@ static int raid5_resize(struct mddev *mddev, sector_t sectors)
> * worth it.
> */
> sector_t newsize;
> + struct r5conf *conf = mddev->private;
> +
> + if (conf->cache)
> + return -EINVAL;
> sectors &= ~((sector_t)mddev->chunk_sectors - 1);
> newsize = raid5_size(mddev, sectors, mddev->raid_disks);
> if (mddev->external_size &&
> @@ -7258,6 +7262,8 @@ static int check_reshape(struct mddev *mddev)
> {
> struct r5conf *conf = mddev->private;
>
> + if (conf->cache)
> + return -EINVAL;
> if (mddev->delta_disks == 0 &&
> mddev->new_layout == mddev->layout &&
> mddev->new_chunk_sectors == mddev->chunk_sectors)
This patch should come before that patch which enables caches - the
could should be correct at each point in the series.
NeilBrown
^ permalink raw reply
* Re: [PATCH v3 3/8] raid5: A caching layer for RAID5/6
From: Neil Brown @ 2015-06-18 1:00 UTC (permalink / raw)
To: Shaohua Li; +Cc: linux-raid, Kernel-team, songliubraving, hch, dan.j.williams
In-Reply-To: <c6df8779f11a4dc3362a04e7cee0be2aec213ebe.1433356864.git.shli@fb.com>
On Wed, 3 Jun 2015 15:48:38 -0700 Shaohua Li <shli@fb.com> wrote:
Hi,
sorry for the delay in getting to this.
3000 lines of code is rather hard to review - especially with so few
comments :-)
There seem to be a number of fairly well defined modules in the code,
such as managing the various data structures. Maybe if these arrived
one per patch it would be easier to review them individually.
> Main goal of the caching layer is to aggregate IO write to hopefully
> make full stripe IO write and fix write hole issue. This might speed up
> read too, but it's not optimized for read, eg, we don't proactively
> cache data for read. The aggregation makes a lot of sense for workloads
> which sequentially write to several files with/without fsync. Such
> workloads are popular in today's datacenter.
>
> Write IO data will write to a cache disk (SSD) first, then later the
> data will be flushed to raid disks.
>
> The cache disk will be organized as a simple ring buffer log. For IO
> data, a tuple (raid_sector, io length, checksum, data) will be appended
> to the log; for raid 5/6 parity, a tuple (stripe_sector, parity length,
> checksum, parity data) will be appended to the log. We don't have
> on-disk index for the data appended to the log. So either we can rebuild
> an in-memory index at startup with scanning the whole log, or we can
> flush all data from cache disk to raid disks at shutdown so cache disk
> has no valid data. Current code chooses the second option, but this can
> be easily changed.
>
> We have a simple meta data for above tuples. It's essentially a
> tuple (sequence, metadata length).
How are these tuples store in the log? One tuples per block? A block
with lots of tuples followed by all the described data? Or does the
block of tuples follow the data?
> Crash recovery or startup will scan
> the log to read the metadata and rebuild in-memory index. If metadata is
> broken at the head of the log, even metadata afterward is ok, the
> scanning will not work well. So we take some steps to mitigate the
> issue:
> -A flush request is sent to cache disk every second to relieve the issue
I don't feel at all comfortable about the every-second flush.
I can certainly understand a flush after 50% of the log space is
written, or even 25%. but time-based flushes should be coming from the
filesystem.
> -FLUSH/FUA request is carefully handled. FLUSH/FUA will only be
> dispatched after all previous requests finish.
This certainly makes sense.
>
> The in-memory index is a simple list of io_range (sequence, metadata
> sector, data sector, length). The list is orded by sequence. The first
> io_range entry's metadata sector is the tail of the log. There is also a
> struct to track io ranges within a stripe. All stripes will be organized
> as a radix tree.
It would be useful here to briefly say how these data structures are
used.
I assumed the order-by-sequence list is to flush data to RAID when the
log is getting full?
The radix-tree tracks where in the log each stripe is - is that correct?
So it is easy to find the stripe at the tail of the list to flush it
out.
>
> All IO data will be copied to a memory pool for caching too until the
> data is flushed to raid disks. This is just to simplify the
> implementation, it's not mandated. In the future flush can do a data
> copy from cache disk to raid disks, so the memory pool can be discarded.
> If a stripe is flushed to raid disks, memory of the stripe can be
> reused.
Again, why a separate memory pool rather than just leaving the data in
the stripe cache until it is safe in the RAID?
>
> We have two limited resources here, memory pool and cache disk space. If
> resource is tight, we will do reclaim. In either case, we will flush
> some data from cache disk to raid disks. However, we implement different
> strategies. For memory reclaim, we prefer reclaiming full stripe. For
> cache disk space reclaim, we prefer reclaiming io_range entry at the
> head of index list.
Wouldn't full stripes be scheduled to the RAID immediately they become
full (or immediately after the parity is safe in the log)?
So for memory reclaim it would make sense to prefer stripes that have
been idle for a long time - so an LRU list.
Certainly when the log approaches full you need to start flushing
things at the start of the log.
>
> If cache disk has IO error, since all data are in memory pool, we will
> flush all data to raid disks and fallback to no-cache-disk mode. IO
> error of cache disk doesn't corrupt any data. After some time, we will
> try to use cache disk again if the disk is ok. The retry will stop after
> several rounds of failure.
The normal data flow involves lots of writing to the cache device and
very little if any reading. So we will probably need some sort of
"scrub" process to read and verify the log occasionally, just to be
sure that reads still work. That could be largely done in user-space.
>
> We always do reclaim in stripe unit. Reclaim could create holes in the
> log, eg, some io_range in the middle is reclaimed, but io_range at the
> head remains. So the index list entries don't always have continuous
> sequence. But this doesn't matter, the first io_range is always the log
> tail. Superblock has a field pointing to the position of log tail. The
> hole can waste a lot of disk space though. In the future, we can
> implement a garbage collection to mitigate the issue, eg, copy data
> from the index tail to head.
I doubt there would be value in garbage collection. Just flush the old
stripes to the RAID. Maybe I'm wrong, but I certainly agree that it
isn't a priority.
>
> In the process reclaim flush data to raid disk, stripe parity will be
> append to cache log. Parity is always appended after its corresponding
> data. After parity is in cache disk, a flush_start block is appended to
> log, which indicates stripe is flushing to raid disks. Data writing to
> raid disks only happens after all data and parity are already in cache
> disk. This will fix the write whole issue. After a stripe is flushed to
> raid disks, a flush_end block is appended to log, which indicates a
> stripe is settled down in raid disks.
I'm not sure that a separate "flush start" block is needed. Once all
the parity blocks have been written we can assume that the flush has
started.
A "flush-end" block does make sense, though I would think of it as an
'invalidate' block in that it invalidates specific previous data blocks.
Same concept though.
>
> Recovery relies on the flush_start and flush_end block. If recovery
> finds data and parity of a stripe, the flush start/end block will be
> used to determine which stage the stripe is in flush. If the stripe is
> listed in flush end block, the stripe is in raid disks, all data and
> parity of the stripe can be discarded. If the stripe isn't listed in
> flush start block, the stripe hasn't started flush to raid yet, its
> parity can be ignored. Otherwise, the stripe is in the middle of
> flushing to raid disks. Since we have both data and parity, the
> recovery just rewrite them to raid disks.
>
> IO write code path:
> 1. copy bio data to stripe memory pages
> 2. append metadata and data to cache log
> 3. IO write endio
>
> reclaim code path:
> 1. select stripe to reclaim
> 2. write all stripe data to raid disk
> 3. in raid5 ops_run_io, append metadata and parity data to cache log.
> ops_run_io doesn't write data/parity to raid disks at this time
> 4. flush cache disk and write flush_start block
> 5. ops_run_io continues. data/parity will be written to raid disks
> 6. flush all raid disks cache
> 7. write flush_end block
> 8. delete in-memory index of the stripe, and advance superblock log checkpoint
I find this a bit confusing. Step 2 writes to the raid disk, but step
3 says it doesn't write to the raid disk yet. It also doesn't tell me
when parity is calculated.
I imagine:
1. select stripe to reclaim
2. read missing data or parity block so new parity can be calculated.
3. calculate parity and write to log - this records that the flush has
started.
4. flush cache device - probably multiple stripes will get up to the
step, and then a single flush will be performed for all of them.
5. schedule writes to the RAID disks, both data an parity.
6. flush RAID disks - again, wait for lots of stripes to reach this
point, then perform a single flush
7. write flush_end / invalidate block for all of the flushed stripes
8. delete in-memory index and advance superblock log checkpoint.
flush the invalidate blocks before writing the superblock.
If you get '4' for one set of stripes to line up with '8' for the
previous set of stripes, then you can combine the two 'flush'
operations to the cache devices.
So the cache device see:
FLUSH superblock update, new parity writes, flush_end of older writes FLUSH
which data writes sprinkled through wherever needed.
>
> Recovery:
> Crash in IO write code path doesn't need recovery. If data and checksum
> don't match, the data will be ignored so read will return old data. In
> reclaim code path, crash before step 4 doesn't need recovery as
> data/parity don't touch raid disk yet. Parity can be ignored too. crash
> after 7 doesn't need recovery too, as the stripe is fully flushed to
> raid disks. Crash between 4 and 7 need recovery. Data and parity in the
> log will be written to raid disks.
So recovery involves reading everything in the log, adding data and
parity to the stripe cache as it is found, invalidating any entries
when an 'invalidate' block is found.
Then any stripe that has all required parity is written to the RAID,
and any other stripe is discarded.
Then the log is emptied and we start from scratch.
>
> The performance of the raid will largely be determined by reclaim speed
> at run time. Several stages of the reclaim process involves IO wait or
> disk cache flush, which significantly impact the raid disk utilization
> and performance. The flush_start and flush_end block make running
> multiple reclaim possible. Each reclaim only records stripes in the
> flush start/end block which it is reclaiming. Recovery can use the
> information to correctly determine stripe's flush stage. An example of 2
> reclaimer:
>
> xxx1 belongs to stripe1 and the same for stripe2
>
> reclaim 1: |data1|...|parity1|...|flush_start1|...|flush_end1|
> reclaim 2: |data2|..|parity2|...............|flush_start2|...|flush_end2|
>
> the reclaims only record its own stripes in flush block. If, for
> exmaple, recovery finds flush_start1, it knows stripe1 is flushing to
> raid disk. Recovery will ignore stripe2, since stripe2 isn't in
> flush_start1.
>
> Multiple reclaim will efficiently solve the performance issue. Current
> code hasn't add multiple reclaim yet, but certainly will be added soon.
Maybe this is what you have in mind, but I would use a state-machine
approach for the reclaim
i.e. schedule a collection of stripes for parity calculation.
as they complete, schedule the writes to cache
When there are no pending writes to cache, schedule a flush
etc.
>
> V3:
> -fix a use after free bug. fix bio to cache disk crosses disk boundary
> -adjust memory watermark
> V2:
> -fix bugs and code clean up
> -log meta data write isn't FUA any more
> -discard only runs when the discard area is big enough
>
> Signed-off-by: Shaohua Li <shli@fb.com>
> ---
> drivers/md/Makefile | 2 +-
> drivers/md/raid5-cache.c | 3246 ++++++++++++++++++++++++++++++++++++++++
> drivers/md/raid5.c | 69 +-
> drivers/md/raid5.h | 15 +-
> include/uapi/linux/raid/md_p.h | 72 +
> 5 files changed, 3392 insertions(+), 12 deletions(-)
> create mode 100644 drivers/md/raid5-cache.c
>
> diff --git a/drivers/md/Makefile b/drivers/md/Makefile
> index dba4db5..aeb4330 100644
> --- a/drivers/md/Makefile
> +++ b/drivers/md/Makefile
> @@ -16,7 +16,7 @@ dm-cache-mq-y += dm-cache-policy-mq.o
> dm-cache-cleaner-y += dm-cache-policy-cleaner.o
> dm-era-y += dm-era-target.o
> md-mod-y += md.o bitmap.o
> -raid456-y += raid5.o
> +raid456-y += raid5.o raid5-cache.o
>
> # Note: link order is important. All raid personalities
> # and must come before md.o, as they each initialise
> diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
> new file mode 100644
> index 0000000..c21d2f2
> --- /dev/null
> +++ b/drivers/md/raid5-cache.c
> @@ -0,0 +1,3246 @@
> +/*
> + * A caching layer for raid 4/5/6
> + *
> + * Copyright (C) 2015 Shaohua Li <shli@fb.com>
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms and conditions of the GNU General Public License,
> + * version 2, as published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope it will be useful, but WITHOUT
> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
> + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
> + * more details.
> + *
> + * You should have received a copy of the GNU General Public License along with
> + * this program; if not, write to the Free Software Foundation, Inc.,
> + * 51 Franklin St - Fifth Floor, Boston, MA 02110-1301 USA.
> + */
> +#include <linux/kernel.h>
> +#include <linux/wait.h>
> +#include <linux/blkdev.h>
> +#include <linux/slab.h>
> +#include <linux/raid/md_p.h>
> +#include <linux/crc32.h>
> +#include <linux/list_sort.h>
> +#include "md.h"
> +#include "raid5.h"
> +
> +/* the log disk cache will be flushed forcely every second */
> +#define LOG_FLUSH_TIME HZ
> +#define LOG_SUPER_WRITE_TIME (5 * HZ)
> +
> +#define MAX_MEM (256 * 1024 * 1024)
> +#define RECLAIM_BATCH 16
> +#define CHECKPOINT_TIMEOUT (5 * HZ)
> +
> +#define MAX_RETRY 10
> +#define IOERR_RETRY_TIME (5 * 60 * HZ)
> +#define MEMERR_RETRY_TIME (60 * HZ)
> +
> +typedef u64 r5blk_t; /* log blocks, could be 512B - 4k */
Why isn't this just "sector_t" ??
> +
> +struct r5l_log {
> + struct r5c_cache *cache;
It isn't immediately clear what the "log" is separate from the "cache".
If there is a good reason, then a comment with that reason here would
be very helpful.
> + struct block_device *bdev;
> + struct md_rdev *rdev;
Is there a reason for storing both of these instead of just 'rdev' and
using 'rdev->bdev' when that is needed?
> +
> + struct page *super_page;
> + u8 uuid[16];
> + u32 uuid_checksum_data;
> + u32 uuid_checksum_meta;
> +
> + unsigned int block_size; /* bytes */
> + unsigned int block_sector_shift;
> + unsigned int page_block_shift;
> + unsigned int stripe_data_size; /* sector */
> + unsigned int chunk_size; /* sector */
> + unsigned int stripe_size; /* sector */
> + unsigned int parity_disks;
Some of these are obvious, some are not.
Some more specific comments would help.
> +
> + r5blk_t total_blocks;
> + r5blk_t first_block;
> + r5blk_t last_block;
> +
> + r5blk_t low_watermark; /* For disk space */
> + r5blk_t high_watermark;
> +
> + r5blk_t last_checkpoint;
> + r5blk_t last_freepoint;
> + r5blk_t super_point;
> + u64 last_cp_seq;
> + u64 last_freeseq;
> + unsigned long last_cp_time;
> + struct work_struct discard_work;
> +
> + int do_discard;
> +
> + u64 seq; /* get after read log */
> + r5blk_t log_start; /* get after read log */
> +
> + u8 data_checksum_type;
> + u8 meta_checksum_type;
> +
> + unsigned int reserved_blocks;
> + wait_queue_head_t space_waitq;
> +
> + struct mutex io_mutex;
> + struct r5l_io_unit *current_io;
> +
> + spinlock_t io_list_lock;
> + struct list_head running_ios; /* order is important */
> +
> + struct list_head task_list;
> + struct list_head parity_task_list;
> + spinlock_t task_lock;
> +
> + struct kmem_cache *io_kc;
> + mempool_t *io_pool;
> + struct bio_set *bio_set;
> +
> + unsigned long next_flush_time;
> + struct work_struct flush_work;
> +};
> +
> +struct r5l_task;
> +/* end function must free task */
> +typedef void (r5l_task_end_fn)(struct r5l_task *task, int error);
> +struct r5l_task {
A comment here telling me what these tasks do would help me understand
the data structure.
> + struct list_head list;
> + int type;
> + struct bio *bio;
> + union {
> + struct bio *orig_bio;
> + struct {
> + sector_t stripe_sector;
> + struct page *page_p;
> + struct page *page_q;
> + };
> + };
> + /* tasks in a single r5l_io_unit will have the same seq and meta_start */
> + sector_t meta_start;
> + sector_t data_start;
> + u64 seq;
> + r5l_task_end_fn *fn;
> + void *private;
> + unsigned int reserved_blocks;
> + u32 checksum[];
> +};
> +
> +/*
> + * Data and metadata in log is written with normal IO write. A power failure
> + * can cause data loss. To relieve the issue, log disk cache will be flushed
> + * forcely every second.
> + *
> + * Note IO is finished out of order. If metadata corrupts in the middle,
> + * recovery can't work well even metadata/data at the tail is good. IO in log
> + * tail could finish earlier than IO ahead, so we must be very careful to
> + * handle FLUSH/FUA bio.
> + *
> + * FLUSH bio: the bio will be dispatched after all previous IO finish. FLUSH
> + * syntax doesn't require pending IO finish, but pending IO might be a metadata
> + * in the middle of log, we must force this order.
> + *
> + * FUA bio: same like FLUSH bio, previous meta IO must be finished before
> + * dispatching this bio. To simplify implementation, we wait all previous IO.
> + * And we must add a FLUSH to make sure previous IO hit disk media. metadata
> + * and the bio itself must be written with FUA.
> + * */
> +struct r5l_io_unit {
> + struct r5l_log *log;
> + struct list_head log_sibling;
> +
> + struct page *meta_page;
> + sector_t meta_sector;
> + int meta_offset;
> + u64 seq;
> + struct bio *meta_bio;
> +
> + struct list_head tasks;
> + struct bio *current_bio;
> + atomic_t refcnt;
> +
> + unsigned int has_flush:1; /* include flush request */
> + unsigned int has_fua:1; /* include fua request */
> + unsigned int has_null_flush:1; /* include empty flush request */
> + /*
> + * io isn't sent yet, flush/fua request can only be submitted till it's
> + * the first IO in running_ios list
> + * */
> + unsigned int io_deferred:1;
> + int error;
> +};
> +
> +struct r5c_io_range {
> + struct list_head log_sibling;
> + struct list_head stripe_sibling;
> +
> + u64 seq;
> +
> + sector_t meta_start; /* cache position */
> + sector_t data_start; /* cache position */
> + sector_t raid_start;
> + unsigned int data_sectors;
> +
> + struct r5c_stripe *stripe;
> + union {
> + struct bio *bio;
> + u32 *checksum; /* only for recovery */
> + };
> +};
> +
> +struct r5c_stripe {
> + u64 raid_index;
> + struct r5c_cache *cache;
> + atomic_t ref;
> + int state;
> + int recovery_state; /* just for recovery */
> +
> + struct list_head io_ranges; /* order list */
> + union {
> + struct list_head stripes;
> + struct list_head parity_list; /* just for recovery */
> + };
> +
> + struct list_head lru;
> +
> + int existing_pages;
> + atomic_t dirty_stripes;
> + atomic_t pending_bios;
> + struct page **parity_pages; /* just for recovery */
> + struct page *data_pages[];
> +};
> +
> +enum {
> + STRIPE_RUNNING = 0,
> + STRIPE_FROZEN = 1, /* Doesn't accept new IO */
> + STRIPE_PARITY_DONE = 2,
> + STRIPE_INRAID = 3,
> + STRIPE_DEAD = 4,
> +
> + RECOVERY_NO_FLUSH = 0,
> + RECOVERY_FLUSH_START = 1, /* stripe in a start flush block */
> + RECOVERY_FLUSH_END = 2,
> +};
I wouldn't be surprised if some compilers rejected that. Two separate
enums would be better.
> +
> +#define STRIPE_LOCK_BITS 8
> +struct r5c_cache {
> + struct mddev *mddev;
> + struct md_rdev *rdev;
> +
> + struct r5l_log log;
> +
Ahh - the log is embedded in the cache.
That probably makes sense. But you don't need a pointer from the log
to the cache, just use container_of().
> + spinlock_t tree_lock; /* protect stripe_tree, log_list, full_stripes */
> + struct radix_tree_root stripe_tree;
> + struct list_head log_list; /* sorted list of io_range */
> + struct list_head full_stripes;
> + int full_stripe_cnt;
> +
> + struct list_head page_pool;
> + spinlock_t pool_lock;
> + u64 free_pages;
> + u64 total_pages;
> + u64 max_pages;
> + u64 low_watermark; /* for memory, pages */
> + u64 high_watermark;
> +
> + unsigned int stripe_data_size; /* stripe size excluding parity, sector */
> + unsigned int chunk_size; /* one disk chunk size including parity, sector */
> + unsigned int stripe_size; /* stripe size including parity, sector */
> + unsigned int parity_disks;
> + unsigned int stripe_data_pages;
> + unsigned int stripe_parity_pages;
> +
> + unsigned int reclaim_batch;
> +
> + unsigned int reserved_space; /* log reserved size, sector */
> +
> + unsigned long reclaim_reason;
> + wait_queue_head_t reclaim_wait;
> + struct md_thread *reclaim_thread;
> + __le64 *stripe_flush_data;
> + int quiesce_state;
> +
> + int in_recovery;
> +
> + struct work_struct pending_io_work;
> +
> + spinlock_t stripe_locks[1 << STRIPE_LOCK_BITS];
> + wait_queue_head_t stripe_waitq[1 << STRIPE_LOCK_BITS];
> +
> + int error_state;
> + int error_type;
> + int retry_cnt;
> + unsigned long next_retry_time;
> + struct bio_list retry_bio_list;
> + wait_queue_head_t error_wait;
> +
> + struct kmem_cache *io_range_kc;
> + struct kmem_cache *stripe_kc;
> + struct bio_set *bio_set;
> +};
> +
> +enum {
> + RECLAIM_MEM = 0, /* work hard to reclaim memory */
> + RECLAIM_MEM_BACKGROUND = 1, /* try to reclaim memory */
> + RECLAIM_MEM_FULL = 2, /* only reclaim full stripe */
> + RECLAIM_DISK = 8, /* work hard to reclaim disk */
> + RECLAIM_DISK_BACKGROUND = 9, /* try to reclaim disk */
> + RECLAIM_FLUSH_ALL = 16, /* flush all data to raid */
> +
> + QUIESCE_NONE = 0,
> + QUIESCE_START = 1,
> + QUIESCE_END = 2,
> +
> + ERROR_NOERROR = 0,
> + ERROR_PREPARE = 1, /* Had an error, flushing cache to raid */
> + ERROR_FINISH = 2, /* Had an error, cache has no data */
> +};
Three enums here :-)
> +
> +#define PAGE_SECTOR_SHIFT (PAGE_SHIFT - 9)
Maybe this should go in md.h - raid1.c raid10.c and raid5.c all use
something like it.
... and that's as far as I got.
I really need this a bit more like md-cluster.c: add modules one at a
time which I can understand and review and units.
Thanks,
NeilBrown
^ permalink raw reply
* Re: [PATCH v3 1/8] MD: add a new disk role to present cache device
From: Neil Brown @ 2015-06-17 23:32 UTC (permalink / raw)
To: Shaohua Li; +Cc: linux-raid, Kernel-team, songliubraving, hch, dan.j.williams
In-Reply-To: <d93a4bc7511c04c39914eebe70816ca671a362ea.1433356864.git.shli@fb.com>
On Wed, 3 Jun 2015 15:48:36 -0700
Shaohua Li <shli@fb.com> wrote:
> From: Song Liu <songliubraving@fb.com>
>
> Next patches will use a disk as raid5/6 caching. We need a new disk role
> to present the cache device
>
> Not sure if we should bump up the MD superblock version for the disk
> role.
No need to increase the superblock version, but you would need to add a
feature flag (for feature_map) which was set whenever the array had a
caching device.
NeilBrown
>
> Signed-off-by: Song Liu <songliubraving@fb.com>
> Signed-off-by: Shaohua Li <shli@fb.com>
> ---
> drivers/md/md.c | 14 +++++++++++++-
> drivers/md/md.h | 4 ++++
> include/uapi/linux/raid/md_p.h | 1 +
> 3 files changed, 18 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index 2750630..6297087 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -1656,6 +1656,9 @@ static int super_1_validate(struct mddev *mddev, struct md_rdev *rdev)
> case 0xfffe: /* faulty */
> set_bit(Faulty, &rdev->flags);
> break;
> + case 0xfffd: /* cache device */
> + set_bit(WriteCache, &rdev->flags);
> + break;
> default:
> rdev->saved_raid_disk = role;
> if ((le32_to_cpu(sb->feature_map) &
> @@ -1811,6 +1814,8 @@ static void super_1_sync(struct mddev *mddev, struct md_rdev *rdev)
> sb->dev_roles[i] = cpu_to_le16(0xfffe);
> else if (test_bit(In_sync, &rdev2->flags))
> sb->dev_roles[i] = cpu_to_le16(rdev2->raid_disk);
> + else if (test_bit(WriteCache, &rdev2->flags))
> + sb->dev_roles[i] = cpu_to_le16(0xfffd);
> else if (rdev2->raid_disk >= 0)
> sb->dev_roles[i] = cpu_to_le16(rdev2->raid_disk);
> else
> @@ -5780,7 +5785,8 @@ static int get_disk_info(struct mddev *mddev, void __user * arg)
> else if (test_bit(In_sync, &rdev->flags)) {
> info.state |= (1<<MD_DISK_ACTIVE);
> info.state |= (1<<MD_DISK_SYNC);
> - }
> + } else if (test_bit(WriteCache, &rdev->flags))
> + info.state |= (1<<MD_DISK_WRITECACHE);
> if (test_bit(WriteMostly, &rdev->flags))
> info.state |= (1<<MD_DISK_WRITEMOSTLY);
> } else {
> @@ -5895,6 +5901,8 @@ static int add_new_disk(struct mddev *mddev, mdu_disk_info_t *info)
> else
> clear_bit(WriteMostly, &rdev->flags);
>
> + if (info->state & (1<<MD_DISK_WRITECACHE))
> + set_bit(WriteCache, &rdev->flags);
> /*
> * check whether the device shows up in other nodes
> */
> @@ -7263,6 +7271,10 @@ static int md_seq_show(struct seq_file *seq, void *v)
> seq_printf(seq, "(F)");
> continue;
> }
> + if (test_bit(WriteCache, &rdev->flags)) {
> + seq_printf(seq, "(C)");
> + continue;
> + }
> if (rdev->raid_disk < 0)
> seq_printf(seq, "(S)"); /* spare */
> if (test_bit(Replacement, &rdev->flags))
> diff --git a/drivers/md/md.h b/drivers/md/md.h
> index 4046a6c..6857592 100644
> --- a/drivers/md/md.h
> +++ b/drivers/md/md.h
> @@ -175,6 +175,10 @@ enum flag_bits {
> * This device is seen locally but not
> * by the whole cluster
> */
> + WriteCache, /* This device is used as write cache.
> + * Usually, this device should be faster
> + * than other devices in the array
> + */
> };
>
> #define BB_LEN_MASK (0x00000000000001FFULL)
> diff --git a/include/uapi/linux/raid/md_p.h b/include/uapi/linux/raid/md_p.h
> index 2ae6131..9d36b91 100644
> --- a/include/uapi/linux/raid/md_p.h
> +++ b/include/uapi/linux/raid/md_p.h
> @@ -89,6 +89,7 @@
> * read requests will only be sent here in
> * dire need
> */
> +#define MD_DISK_WRITECACHE 18 /* disk is used as the write cache in RAID-5/6 */
>
> typedef struct mdp_device_descriptor_s {
> __u32 number; /* 0 Device number in the entire set */
^ permalink raw reply
* Re: RAID1 removing failed disk returns EBUSY
From: Neil Brown @ 2015-06-17 2:51 UTC (permalink / raw)
To: XiaoNi; +Cc: Joe Lawrence, linux-raid, Bill Kuzeja
In-Reply-To: <5577D8A1.9060605@redhat.com>
On Wed, 10 Jun 2015 14:26:41 +0800
XiaoNi <xni@redhat.com> wrote:
>
>
> On 02/03/2015 04:10 PM, Xiao Ni wrote:
> >
> > ----- Original Message -----
> >> From: "NeilBrown" <neilb@suse.de>
> >> To: "Xiao Ni" <xni@redhat.com>
> >> Cc: "Joe Lawrence" <joe.lawrence@stratus.com>, linux-raid@vger.kernel.org, "Bill Kuzeja" <william.kuzeja@stratus.com>
> >> Sent: Monday, February 2, 2015 2:36:01 PM
> >> Subject: Re: RAID1 removing failed disk returns EBUSY
> >>
> >> On Thu, 29 Jan 2015 07:14:16 -0500 (EST) Xiao Ni <xni@redhat.com> wrote:
> >>
> >>>
> >>> ----- Original Message -----
> >>>> From: "NeilBrown" <neilb@suse.de>
> >>>> To: "Xiao Ni" <xni@redhat.com>
> >>>> Cc: "Joe Lawrence" <joe.lawrence@stratus.com>,
> >>>> linux-raid@vger.kernel.org, "Bill Kuzeja" <william.kuzeja@stratus.com>
> >>>> Sent: Thursday, January 29, 2015 11:52:17 AM
> >>>> Subject: Re: RAID1 removing failed disk returns EBUSY
> >>>>
> >>>> On Sun, 18 Jan 2015 21:33:50 -0500 (EST) Xiao Ni <xni@redhat.com> wrote:
> >>>>
> >>>>>
> >>>>> ----- Original Message -----
> >>>>>> From: "Joe Lawrence" <joe.lawrence@stratus.com>
> >>>>>> To: "Xiao Ni" <xni@redhat.com>
> >>>>>> Cc: "NeilBrown" <neilb@suse.de>, linux-raid@vger.kernel.org, "Bill
> >>>>>> Kuzeja" <william.kuzeja@stratus.com>
> >>>>>> Sent: Friday, January 16, 2015 11:10:31 PM
> >>>>>> Subject: Re: RAID1 removing failed disk returns EBUSY
> >>>>>>
> >>>>>> On Fri, 16 Jan 2015 00:20:12 -0500
> >>>>>> Xiao Ni <xni@redhat.com> wrote:
> >>>>>>> Hi Joe
> >>>>>>>
> >>>>>>> Thanks for reminding me. I didn't do that. Now it can remove
> >>>>>>> successfully after writing
> >>>>>>> "idle" to sync_action.
> >>>>>>>
> >>>>>>> I thought wrongly that the patch referenced in this mail is
> >>>>>>> fixed
> >>>>>>> for
> >>>>>>> the problem.
> >>>>>> So it sounds like even with 3.18 and a new mdadm, this bug still
> >>>>>> persists?
> >>>>>>
> >>>>>> -- Joe
> >>>>>>
> >>>>>> --
> >>>>> Hi Joe
> >>>>>
> >>>>> I'm a little confused now. Does the patch
> >>>>> 45eaf45dfa4850df16bc2e8e7903d89021137f40 from linux-stable
> >>>>> resolve the problem?
> >>>>>
> >>>>> My environment is:
> >>>>>
> >>>>> [root@dhcp-12-133 mdadm]# mdadm --version
> >>>>> mdadm - v3.3.2-18-g93d3bd3 - 18th December 2014 (this is the newest
> >>>>> upstream)
> >>>>> [root@dhcp-12-133 mdadm]# uname -r
> >>>>> 3.18.2
> >>>>>
> >>>>>
> >>>>> My steps are:
> >>>>>
> >>>>> [root@dhcp-12-133 mdadm]# lsblk
> >>>>> sdb 8:16 0 931.5G 0 disk
> >>>>> └─sdb1 8:17 0 5G 0 part
> >>>>> sdc 8:32 0 186.3G 0 disk
> >>>>> sdd 8:48 0 931.5G 0 disk
> >>>>> └─sdd1 8:49 0 5G 0 part
> >>>>> [root@dhcp-12-133 mdadm]# mdadm -CR /dev/md0 -l1 -n2 /dev/sdb1
> >>>>> /dev/sdd1
> >>>>> --assume-clean
> >>>>> mdadm: Note: this array has metadata at the start and
> >>>>> may not be suitable as a boot device. If you plan to
> >>>>> store '/boot' on this device please ensure that
> >>>>> your boot-loader understands md/v1.x metadata, or use
> >>>>> --metadata=0.90
> >>>>> mdadm: Defaulting to version 1.2 metadata
> >>>>> mdadm: array /dev/md0 started.
> >>>>>
> >>>>> Then I unplug the disk.
> >>>>>
> >>>>> [root@dhcp-12-133 mdadm]# lsblk
> >>>>> sdc 8:32 0 186.3G 0 disk
> >>>>> sdd 8:48 0 931.5G 0 disk
> >>>>> └─sdd1 8:49 0 5G 0 part
> >>>>> └─md0 9:0 0 5G 0 raid1
> >>>>> [root@dhcp-12-133 mdadm]# echo faulty >
> >>>>> /sys/block/md0/md/dev-sdb1/state
> >>>>> [root@dhcp-12-133 mdadm]# echo remove >
> >>>>> /sys/block/md0/md/dev-sdb1/state
> >>>>> -bash: echo: write error: Device or resource busy
> >>>>> [root@dhcp-12-133 mdadm]# echo idle > /sys/block/md0/md/sync_action
> >>>>> [root@dhcp-12-133 mdadm]# echo remove >
> >>>>> /sys/block/md0/md/dev-sdb1/state
> >>>>>
> >>>> I cannot reproduce this - using linux 3.18.2. I'd be surprised if mdadm
> >>>> version affects things.
> >>> Hi Neil
> >>>
> >>> I'm very curious, because it can reproduce in my machine 100%.
> >>>
> >>>> This error (Device or resoource busy) implies that rdev->raid_disk is >=
> >>>> 0
> >>>> (tested in state_store()).
> >>>>
> >>>> ->raid_disk is set to -1 by remove_and_add_spares() providing:
> >>>> 1/ it isn't Blocked (which is very unlikely)
> >>>> 2/ hot_remove_disk succeeds, which it will if nr_pending is zero, and
> >>>> 3/ nr_pending is zero.
> >>> I remember I have tired to check those reasons. But it's really is the
> >>> reason 1
> >>> which is very unlikely.
> >>>
> >>> I add some code in the function array_state_show
> >>>
> >>> array_state_show(struct mddev *mddev, char *page) {
> >>> enum array_state st = inactive;
> >>> struct md_rdev *rdev;
> >>>
> >>> rdev_for_each_rcu(rdev, mddev) {
> >>> printk(KERN_ALERT "search for %s\n",
> >>> rdev->bdev->bd_disk->disk_name);
> >>> if (test_bit(Blocked, &rdev->flags))
> >>> printk(KERN_ALERT "rdev is Blocked\n");
> >>> else
> >>> printk(KERN_ALERT "rdev is not Blocked\n");
> >>> }
> >>>
> >>> When I echo 1 > /sys/block/sdc/device/delete, then I ran command:
> >>>
> >>> [root@dhcp-12-133 md]# cat /sys/block/md0/md/array_state
> >>> read-auto
> >> ^^^^^^^^^
> >>
> >> I think that is half the explanation.
> >> You must have the md_mod.start_ro parameter set to '1'.
> >>
> >>
> >>> [root@dhcp-12-133 md]# dmesg
> >>> [ 2679.559185] search for sdc
> >>> [ 2679.559189] rdev is Blocked
> >>> [ 2679.559190] search for sdb
> >>> [ 2679.559190] rdev is not Blocked
> >>>
> >>> So sdc is Blocked
> >> and that is the other half - thanks.
> >> (yes, I was wrong. Sometimes it is easier than being right, but still
> >> yields results).
> >>
> >> When a device fails, it is Blocked until the metadata is updated to record
> >> the failure. This ensures that no writes succeed without writing to that
> >> device, until we a certain that no read will try reading from that device,
> >> even after a crash/restart.
> >>
> >> Blocked is cleared after the metadata is written, but read-auto (and
> >> read-only) devices never write out their metadata. So blocked doesn't get
> >> cleared.
> >>
> >> When you "echo idle > .../sync_action" one of the side effects is to with
> >> from 'read-auto' to fully active. This allows the metadata to be written,
> >> Blocked to be cleared, and the device to be removed.
> >>
> >> If you
> >> echo none > /sys/block/md0/md/dev-sdc/slot
> >>
> >> first, then the remove will work.
> >>
> >> We could possibly fix it with something like the following, but I'm not sure
> >> I like it. There is no guarantee that I can see which would ensure the
> >> superblock got updated before the first write if the array switch to
> >> read/write.
> >>
> >> NeilBrown
> >>
> >> diff --git a/drivers/md/md.c b/drivers/md/md.c
> >> index 9233c71138f1..b3d1e8e5e067 100644
> >> --- a/drivers/md/md.c
> >> +++ b/drivers/md/md.c
> >> @@ -7528,7 +7528,7 @@ static int remove_and_add_spares(struct mddev *mddev,
> >> rdev_for_each(rdev, mddev)
> >> if ((this == NULL || rdev == this) &&
> >> rdev->raid_disk >= 0 &&
> >> - !test_bit(Blocked, &rdev->flags) &&
> >> + (!test_bit(Blocked, &rdev->flags) || mddev->ro) &&
> >> (test_bit(Faulty, &rdev->flags) ||
> >> ! test_bit(In_sync, &rdev->flags)) &&
> >> atomic_read(&rdev->nr_pending)==0) {
> >>
> >>
> >>
> > Hi Neil
> >
> > I have tried the patch and the problem can be fixed by it. But I'm sorry that I can't
> > give more advices for better idea about this. I'm not familiar with the metadata part about
> > the md. I'll try to get more time to read the code about md.
> >
> Hi Neil
>
> I don't see the patch in linux-stable, do you miss this?
I don't believe this bug is sufficiently serious for the patch to go to
-stable. However it doesn't need to be fixed - thanks for the reminder.
I've just queued the following patch which I am happy with. If you
could confirm that it works for you, I would appreciate that.
Thanks,
NeilBrown
From: Neil Brown <neilb@suse.de>
Date: Wed, 17 Jun 2015 12:31:46 +1000
Subject: [PATCH] md: clear Blocked flag on failed devices when array is
read-only.
The Blocked flag indicates that a device has failed but that this
fact hasn't been recorded in the metadata yet. Writes to such
devices cannot be allowed until the metadata has been updated.
On a read-only array, the Blocked flag will never be cleared.
This prevents the device being removed from the array.
If the metadata is being handled by the kernel
(i.e. !mddev->external), then we can be sure that if the array is
switch to writable, then a metadata update will happen and will
record the failure. So we don't need the flag set.
If metadata is externally managed, it is upto the external manager
to clear the 'blocked' flag.
Reported-by: XiaoNi <xni@redhat.com>
Signed-off-by: NeilBrown <neilb@suse.de>
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 3d339e2..5a6681a 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -8125,6 +8125,15 @@ void md_check_recovery(struct mddev *mddev)
int spares = 0;
if (mddev->ro) {
+ struct md_rdev *rdev;
+ if (!mddev->external && mddev->in_sync)
+ /* 'Blocked' flag not needed as failed devices
+ * will be recorded if array switched to read/write.
+ * Leaving it set will prevent the device
+ * from being removed.
+ */
+ rdev_for_each(rdev, mddev)
+ clear_bit(Blocked, &rdev->flags);
/* On a read-only array we can:
* - remove failed devices
* - add already-in_sync devices if the array itself
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* Re: [PATCH 11/11] Reuse the write_bitmap for update uuid
From: Guoqing Jiang @ 2015-06-17 1:53 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid, rgoldwyn
In-Reply-To: <20150617095407.14d4ed4e@home.neil.brown.name>
Hi Neil,
>> To handle different situations, it can support updating the uuid of
>> the bitmap. This patch also removes the redundant bitmap_update_uuid.
>>
>> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
>>
>
> Hi,
> I have applied all patches except this one to a new branch called
> 'cluster' in git://neil.brown.name/mdadm/
> I'll merge it with the master branch after the next point release.
>
> I made a few changes on the way - mostly fixing indenting. One more
> significant change is that I made it compile if /usr/include/corosync
> doesn't exist.
>
Thanks a lot for that!
> I didn't apply this change because it doesn't make sense to me. The
> format of the bitmap is independent of the metadata used, so using a
> ->ss method to update t he bitmap doesn't make sense.
>
> If you can explain what problem you are trying to fix, I'm sure we can
> find a solution that we are both happy with.
>
>
There is no problem with previous code, this change just want to
reuse existed function, and it is ok to drop the patch.
Thanks,
Guoqing
^ permalink raw reply
* Re: [PATCH 11/11] Reuse the write_bitmap for update uuid
From: Neil Brown @ 2015-06-16 23:54 UTC (permalink / raw)
To: Guoqing Jiang; +Cc: linux-raid, rgoldwyn
In-Reply-To: <1433914934-21195-12-git-send-email-gqjiang@suse.com>
On Wed, 10 Jun 2015 13:42:14 +0800
Guoqing Jiang <gqjiang@suse.com> wrote:
> To handle different situations, it can support updating the uuid of
> the bitmap. This patch also removes the redundant bitmap_update_uuid.
>
> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
Hi,
I have applied all patches except this one to a new branch called
'cluster' in git://neil.brown.name/mdadm/
I'll merge it with the master branch after the next point release.
I made a few changes on the way - mostly fixing indenting. One more
significant change is that I made it compile if /usr/include/corosync
doesn't exist.
I didn't apply this change because it doesn't make sense to me. The
format of the bitmap is independent of the metadata used, so using a
->ss method to update t he bitmap doesn't make sense.
If you can explain what problem you are trying to fix, I'm sure we can
find a solution that we are both happy with.
Thanks,
NeilBrown
> ---
> Assemble.c | 5 ++---
> bitmap.c | 20 --------------------
> mdadm.h | 2 +-
> super0.c | 11 +++++++++++
> super1.c | 4 ++++
> 5 files changed, 18 insertions(+), 24 deletions(-)
>
> diff --git a/Assemble.c b/Assemble.c
> index d163eaa..2662261 100644
> --- a/Assemble.c
> +++ b/Assemble.c
> @@ -662,9 +662,8 @@ static int load_devices(struct devs *devices, char *devmap,
>
> if (strcmp(c->update, "uuid")==0 &&
> ident->bitmap_fd >= 0 && !bitmap_done) {
> - if (bitmap_update_uuid(ident->bitmap_fd,
> - content->uuid,
> - tst->ss->swapuuid) != 0)
> + copy_uuid(tst->devs->uuid, content->uuid, tst->ss->swapuuid);
> + if (tst->ss->write_bitmap(tst, dfd, UUIDUpdate))
> pr_err("Could not update uuid on external bitmap.\n");
> else
> bitmap_done = 1;
> diff --git a/bitmap.c b/bitmap.c
> index d21e5cc..575fcb9 100644
> --- a/bitmap.c
> +++ b/bitmap.c
> @@ -457,23 +457,3 @@ out:
> unlink(filename); /* possibly corrupted, better get rid of it */
> return rv;
> }
> -
> -int bitmap_update_uuid(int fd, int *uuid, int swap)
> -{
> - struct bitmap_super_s bm;
> - if (lseek(fd, 0, 0) != 0)
> - return 1;
> - if (read(fd, &bm, sizeof(bm)) != sizeof(bm))
> - return 1;
> - if (bm.magic != __cpu_to_le32(BITMAP_MAGIC))
> - return 1;
> - copy_uuid(bm.uuid, uuid, swap);
> - if (lseek(fd, 0, 0) != 0)
> - return 2;
> - if (write(fd, &bm, sizeof(bm)) != sizeof(bm)) {
> - lseek(fd, 0, 0);
> - return 2;
> - }
> - lseek(fd, 0, 0);
> - return 0;
> -}
> diff --git a/mdadm.h b/mdadm.h
> index 97892e6..7b9bb28 100644
> --- a/mdadm.h
> +++ b/mdadm.h
> @@ -358,6 +358,7 @@ enum bitmap_update {
> NoUpdate,
> NameUpdate,
> NodeNumUpdate,
> + UUIDUpdate,
> };
>
> /* structures read from config file */
> @@ -1273,7 +1274,6 @@ extern int CreateBitmap(char *filename, int force, char uuid[16],
> int major);
> extern int ExamineBitmap(char *filename, int brief, struct supertype *st);
> extern int Write_rules(char *rule_name);
> -extern int bitmap_update_uuid(int fd, int *uuid, int swap);
> extern unsigned long bitmap_sectors(struct bitmap_super_s *bsb);
> extern int Dump_metadata(char *dev, char *dir, struct context *c,
> struct supertype *st);
> diff --git a/super0.c b/super0.c
> index 6ad9d39..49267d1 100644
> --- a/super0.c
> +++ b/super0.c
> @@ -1180,12 +1180,23 @@ static int write_bitmap0(struct supertype *st, int fd, enum bitmap_update update
> unsigned long long dsize;
> unsigned long long offset;
> mdp_super_t *sb = st->sb;
> + bitmap_super_t *bms = (bitmap_super_t*)(((char*)sb) + MD_SB_BYTES);
>
> int rv = 0;
>
> int towrite, n;
> void *buf;
>
> + switch (update) {
> + case UUIDUpdate:
> + memset((char *)bms->uuid, 0, sizeof(bms->uuid));
> + strncpy((char *)bms->uuid, (char *)st->devs->uuid, sizeof(bms->uuid));
> + break;
> + case NoUpdate:
> + default:
> + break;
> + }
> +
> if (!get_dev_size(fd, NULL, &dsize))
> return 1;
>
> diff --git a/super1.c b/super1.c
> index 8128750..4ce7773 100644
> --- a/super1.c
> +++ b/super1.c
> @@ -2232,6 +2232,10 @@ static int write_bitmap1(struct supertype *st, int fd, enum bitmap_update update
>
> bms->nodes = __cpu_to_le32(st->nodes);
> break;
> + case UUIDUpdate:
> + memset((char *)bms->uuid, 0, sizeof(bms->uuid));
> + strncpy((char *)bms->uuid, (char *)st->devs->uuid, sizeof(bms->uuid));
> + break;
> case NoUpdate:
> default:
> break;
^ permalink raw reply
* Re: seagate-"archive"-disks with raid6?
From: wiebittewas @ 2015-06-16 2:58 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <alpine.DEB.2.02.1506132207180.9487@uplift.swm.pp.se>
Am 13.06.2015 22:11 schrieb "Mikael Abrahamsson" bzgl. "Re: seagate-"archive"-disks with raid6?":
> http://www.spinics.net/lists/linux-ide/msg50641.html
thanks for this link, which shows us, that there might be better solutions in near future.
> These drives are different beasts than regular HDDs, and we seem to be seeing problems with them just the way SSDs were problematic in the beginning.
if "different" means that they use SMR - yes, we know this. Our current use is mostly write-once/read-many, so the described additional timeouts might not happen here.
nevertheless, the cooperation beetween filesystem and disk, which is mentioned at the given link, sounds better, so we'll try to wait some weeks or might find a temporary solution.
(perhaps this new series will have ERC/TLER enabled, when they let the kernel do the SMR-control...)
regards
w.
^ permalink raw reply
* Re: Migrating a RAID 5 from 4x2TB to 3x6TB ?
From: Wols Lists @ 2015-06-15 21:45 UTC (permalink / raw)
To: Wilson, Jonathan, Pierre Wieser; +Cc: linux-raid
In-Reply-To: <BLU436-SMTP134979C5D563F30AFDBD89898B80@phx.gbl>
On 15/06/15 11:46, Wilson, Jonathan wrote:
> On the 4 disks, create 17G partitions then create a 4 disk raid10 far2
> array with 64K chunk. This will give you a swap file of 34G in size
> (well over provisioned, but doesn't hurt or impact performance). As its
> likely swap access will be in small random amounts this means the disk
> write size is not overly large, no point in writing/reading 512K chunks
> (the current default) for a 4K page swap/memory access; raid10 is fast;
> far2 from what I've read also improves the speed of read/writes in some
> tests (I don't know why or if the tests I've seen mentioned on the web
> are accurate for the type of access swap will cause but on my setup I
> can get a dd speed of 582M read and 215M write from drives with a single
> device speed of about 80-100M as a rough and ready speed test).
Bear in mind that linux will all by itself do a raid-0 on your swap
partitions if you ask it to.
I *always* size my swap partitions at twice mobo max ram. If you read
the release notes for linux 2.4.early, you'll notice Linus says "if you
have swap, it MUST be twice ram or more" - that was a kernel panic if
you ignored it ! Given that people have been saying that rule was
obsolete since before linux was born, and that while a lot of water has
passed under the bridge since then I've seen and heard nothing to tell
me that the fundamentals have changed ... so because my mobo maxes out
at 16MB ram, all my disks have a 32GB swap partition each.
If you want to raid10 that lot, fine, I just set equal priority on all
my swap partitions, and linux will raid-0 it for me.
The other thing to bear in mind, it's all very well doing speed tests on
your drive, but if you're hammering swap and backing store at the same
time, your speeds are going to plummet as your drive starts seeking all
over the shop ... mind you, with a decent amount of ram you probably
won't need swap at all.
Cheers,
Wol
Cheers,
Wol
^ permalink raw reply
* Re: Fw: Problems with bdev_write_page().
From: Charles Bertsch @ 2015-06-15 14:19 UTC (permalink / raw)
To: Matthew Wilcox
Cc: Neil Brown, linux-fsdevel, linux-raid, linux-kernel,
BertschC@acm.org
In-Reply-To: <fSoD1q00Z1Qq7Ka01SoEyb>
On 06/12/2015 07:48 AM, Matthew Wilcox wrote:
...
>>
>> Can you propose a fix for Charles, who can trigger this bug and nicely
>> bisected it for us - thanks Charles!!!
>
....
>
> (this patch probably doesn't apply to the current tree; it's done against
> a bit of a mishmash tree in my current working directory. it's not for
> applynig, but for commentary).
>
Matthhew,
I tested again with v4.1-rc7, bug still present.
I was able to apply the patch (at offsets -10, -1, -8 for
fs/block_dev.c, fs/mpage.c, and mm/page_io.c). That test has been
running without error for 14 hours. The bug would occur with this setup
within 10 minutes.
Thanks.
Charles Bertsch
^ permalink raw reply
* 4.0.5: WARNING: CPU: 3 PID: 249 at /home/kernel/COD/linux/mm/backing-dev.c:372 bdi_unregister+0x36/0x40()
From: Tomasz Chmielewski @ 2015-06-15 12:39 UTC (permalink / raw)
To: linux-kernel; +Cc: linux-raid
Got this after stopping a RAID-1 array:
[ 626.694737] md: md3 still in use.
[ 626.694946] md: delaying resync of md3 until md2 has finished (they
share one or more physical units)
[ 628.256210] md3: detected capacity change from 3888444473344 to 0
[ 628.256372] md: md3 stopped.
[ 628.256383] md: unbind<sdb4>
[ 628.274852] md: export_rdev(sdb4)
[ 628.274909] md: unbind<sda4>
[ 628.282856] md: export_rdev(sda4)
[ 628.283246] ------------[ cut here ]------------
[ 628.283258] WARNING: CPU: 3 PID: 249 at
/home/kernel/COD/linux/mm/backing-dev.c:372 bdi_unregister+0x36/0x40()
[ 628.283261] Modules linked in: intel_rapl iosf_mbi
x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64
lrw gf128mul glue_helper ablk_helper eeepc_wmi asus_wmi ppdev
sparse_keymap parport_pc cryptd video shpchp 8250_fintek lpc_ich lp
tpm_infineon wmi mac_hid parport serio_raw btrfs pata_acpi raid10
raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor
raid6_pq raid1 ahci r8169 libahci raid0 mii pata_via multipath linear
[ 628.283320] CPU: 3 PID: 249 Comm: kworker/3:1 Not tainted
4.0.5-040005-generic #201506061639
[ 628.283322] Hardware name: System manufacturer System Product
Name/P8H67-M PRO, BIOS 3904 04/27/2013
[ 628.283328] Workqueue: md_misc mddev_delayed_delete
[ 628.283341] 0000000000000174 ffff88040867bca8 ffffffff817e4a5d
0000000000000007
[ 628.283347] 0000000000000000 ffff88040867bce8 ffffffff81079227
ffffffff81adc895
[ 628.283352] ffff880409197c00 0000000000000000 0000000000000000
ffff88041f2da900
[ 628.283365] Call Trace:
[ 628.283374] [<ffffffff817e4a5d>] dump_stack+0x45/0x57
[ 628.283381] [<ffffffff81079227>] warn_slowpath_common+0x97/0xe0
[ 628.283386] [<ffffffff8107928a>] warn_slowpath_null+0x1a/0x20
[ 628.283390] [<ffffffff811a3f16>] bdi_unregister+0x36/0x40
[ 628.283397] [<ffffffff813955f8>] del_gendisk+0x108/0x260
[ 628.283402] [<ffffffff81648eec>] md_free+0x4c/0x70
[ 628.283408] [<ffffffff813b6b62>] kobject_cleanup+0x82/0x1c0
[ 628.283413] [<ffffffff813b69f0>] kobject_put+0x30/0x70
[ 628.283417] [<ffffffff81649c44>] mddev_delayed_delete+0x34/0x40
[ 628.283422] [<ffffffff81092204>] process_one_work+0x144/0x490
[ 628.283426] [<ffffffff81092c6e>] worker_thread+0x11e/0x450
[ 628.283431] [<ffffffff81092b50>] ? create_worker+0x1f0/0x1f0
[ 628.283436] [<ffffffff81098999>] kthread+0xc9/0xe0
[ 628.283442] [<ffffffff810988d0>] ? flush_kthread_worker+0x90/0x90
[ 628.283448] [<ffffffff817f1118>] ret_from_fork+0x58/0x90
[ 628.283453] [<ffffffff810988d0>] ? flush_kthread_worker+0x90/0x90
[ 628.283457] ---[ end trace 2c187f15cc11aca4 ]---
--
Tomasz Chmielewski
http://wpkg.org
^ permalink raw reply
* Re: Migrating a RAID 5 from 4x2TB to 3x6TB ?
From: Wilson, Jonathan @ 2015-06-15 11:31 UTC (permalink / raw)
To: Wols Lists; +Cc: Can Jeuleers, Pierre Wieser, linux-raid
In-Reply-To: <55773493.3050605@youngman.org.uk>
On Tue, 2015-06-09 at 19:46 +0100, Wols Lists wrote:
> On 09/06/15 06:23, Can Jeuleers wrote:
> > On 08/06/15 21:28, Pierre Wieser wrote:
> >> Hi all,
> >>
> >> I currently have an almost full RAID 5 built with 4 x 2 TB disks.
> >> I wonder if it would be possible to migrate it to a bigger RAID 5
> >> with 3 x 6TB new disks.
> >
> > I'd recommend against it:
> >
> > https://en.wikipedia.org/wiki/RAID#Unrecoverable_read_errors_during_rebuild
> >
> > Jan
> >
> Please expand! Having read the article, it doesn't seem to say anything
> more than what is repeated time and time on this list - MAKE SURE YOUR
> DRIVES ARE DECENT RAID DRIVES.
>
> If you have ERC, then the odd "soft" read error doesn't matter. If you
> don't have ERC, then your data is at risk when you replace a drive, and
> it doesn't matter how big your drives are, it's the array size that matters.
TLER doesn't actually affect the raid or its integrity compared to
non-tler drives (well strictly it _might_ as drives with tler might have
better lifespans, might have longer warranties (which suggests better
life), might have better URE rates, etc.) but the difference to how
mdadm handles things is actually down to the way the block device layer
handles things.
From what I can tell, with TLER the disk just gives up and reports an
error very quickly, this is then passed up the stack to the raid layer
which then tries to resolve the problem using various methods... a TLER
"error" does not mean the device is kicked, only if mdadm can't resolve
the problem does the device get booted. (I think it tries to recover the
data then tries to write recovered data back to the device, only if this
fails does the disk get booted)
Without TLER the disk tries to sort its own problems out instead of
reporting an error, this might take a long time, it might try to resolve
the problem forever in one long endless loop. The block layer (sdX)
knows it asked for something to happen, it gets bored and decides its
taken to long for the disk to return data so it decides that the disk no
longer exists, it (the device block layer as far as I can tell) kicks
the disk then passes on a message to mdadm that the disk is down for the
count and has been booted from the system.
I don't know who sets the block layer time out, or if it varies
depending on if the disk is a "file system" or is "a raid member" but
someone decided that after a few seconds the device should disappear/be
marked as bad within the system to prevent the raid from stalling, or as
a "normal" disk/file system various types of errors up to and including
a complete crash.
By setting the time out in the block layer /sys/block/sdX/device/timeout
to a high(ish) value the raid will stall (not a problem for most end
users, big no-no for a high end data server with 100's of users relying
on quick responses) or "hang" on a "normal" disk producing a frozen
screen or what not to the end user... while a pain, better than a disk
fail especially if eventually the disk internally manages to sort the
problem and give valid data back instead of the system crashing.
I set the block time out to 180 seconds on all disks (3 mins) for disks
with TLER enabled they will still give up and send an error message up
the stack in less than 7 seconds, for my other "green" drives with no
TLER they will try their best to recover and if not will eventually pass
the error up to the block layer or after 3 mins the block layer will
report they timed out to either mdadm or to the file system.
Unlike with mdadm and the block device which can be tuned, a hardware
raid will give up on the drive after 7 seconds and kick it (which is why
you should only use raid/TLER drives in a HW raid); at least with mdadm,
specifically the block device layer, depending on the type of drive and
how much internal (to the disk) error recovery is performed and how
important response times are you can use any old disk with mdadm raid
with no problems.
It should also be noted that the same issue would happen without raid, a
pause/hang or a drive marked as failed and/or the system crashing if the
block layer gives up or an error message passed up to the file system if
the disk has TLER and is used in a non raid way... how the file system
handles it is up to the file system.
>
> Cheers,
> Wol
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* Re: Migrating a RAID 5 from 4x2TB to 3x6TB ?
From: Wilson, Jonathan @ 2015-06-15 10:46 UTC (permalink / raw)
To: Pierre Wieser; +Cc: linux-raid
In-Reply-To: <1056149272.1412.1433965061737.JavaMail.zimbra@wieser.fr>
On Wed, 2015-06-10 at 21:37 +0200, Pierre Wieser wrote:
> Hi all,
>
> > I currently have an almost full RAID 5 built with 4 x 2 TB disks.
> > I wonder if it would be possible to migrate it to a bigger RAID 5
> > with 3 x 6TB new disks.
>
> Due to all suggestions, and I'd wish another time thank all contributions,
> I've spent some hours reading the mailing list archives, surfing the web
> with more appropiate keywords, and so on..
>
> So, here what I now plan to do:
>
> First, I hace cancelled my order for the new 6TB desktop-grade disks,
> replacing it with 4TB WD RedPro, and one 6TB desktop-grade (see below its use)
>
> As the full RAID5 array I planned to migrate is already my backup system,
> I cannot rely on a restore :(. So the first thing is to rsync the current
> array to the directly attached 6TB disk. I don't thing I have a free SATA
> port on my motherboard, but at worst I will be able to use the one currently
> used for the DVD drive.
>
> I've chosen to build new RAID10 arrays.
> I've moved away the RAID6 suggestion due to its known bad write performance,
> and also because I'm willing/able to put a bit more money to get better perfs.
First I am not an expert, the following is based on multiple web sites
so is kind of cobbled together and using your setup but based on my
system as it stands now.
I'm going to make some assumptions here...
1) the motherboard can see and boot from 4TB drives instead of only
seeing 750G (approx) in the bios, if not you will need a smaller disk
for the boot/os.
2) this will be a "bios" install/boot or UEFI with CSM to simulate a
bios install/boot
3) these will be the only disks in the system.
>
> The 4x4TB new disks will be partitioned as:
As this will be a clean install, make a 1M partition flagged as "bios
boot" (EF02 in gdisk) this will allow grub2 to install into the member
(as normal) and its larger next stage loader & raid "drivers/ability" to
be installed into the "bios boot" partition, do this for all 4 disks.
(see #a later)
> - 512MB to be a RAID1 array mounted as /boot
Of the 4 drives 512M partitions, create a 4 way raid1 for /boot
(grub2/config and the kernels & initiramfs will live in here) (see #b
later)
> - 8GB to be a RAID10 array used as swap
On the 4 disks, create 17G partitions then create a 4 disk raid10 far2
array with 64K chunk. This will give you a swap file of 34G in size
(well over provisioned, but doesn't hurt or impact performance). As its
likely swap access will be in small random amounts this means the disk
write size is not overly large, no point in writing/reading 512K chunks
(the current default) for a 4K page swap/memory access; raid10 is fast;
far2 from what I've read also improves the speed of read/writes in some
tests (I don't know why or if the tests I've seen mentioned on the web
are accurate for the type of access swap will cause but on my setup I
can get a dd speed of 582M read and 215M write from drives with a single
device speed of about 80-100M as a rough and ready speed test).
> - two 25 GB parts to be two RAID10 arrays used as root filesystem
> (plus place for an alternate when upgrading the OS)
25G partition on all 4 disks in to a single raid10 far2 (default 512K
chunk) = 50G for "root"
Duplicate above for a second "root/install" (this might be useful for #b
later also)
> - the rest of the disk will be splitted in four equal parts (about 930 MB
> I think), each of which being member of a separate data RAID10 array.
I would not bother creating 4 smaller partitions on each disk, nothing
will be gained except more complexity and may even reduce speeds due to
increasing seeks when data doesn't reside exactly on one raid group. LVM
can still sit on the top for flexibility later. You could also go for a
4 disk raid6 (which I have) which would give you the same amount of
storage space on creation but would then mean 1 extra disk=1 extra disks
worth of space, not half,as you add more. (I'm not sure about R/W
speeds, also while I think it can - I'm not sure if mdadm --grow works
on raid10)
>
> I am conscious that this seems as a waste of space, and especially for the
> /boot partition. But this scheme will let me:
> a) have banalized disks: all disks have same rules, are partitioned identically
I have found with GPT/raid etc. that as time has gone on I have created
partitions with the same "number" as the md/X numbering, while not
needed it does mean I know "/dev/md/3" is made up of /dev/sd[a-d]3 so if
at some future point I add more disks and create a new array I do it by
creating partition number(s) "4" and array /dev/md/4 instead of having a
bunch of partition "1"s with a multitude of differing number mdadm
arrays which gives my brain a kick to remind me that "no you can't
delete that partition because it doesn't match the array number you are
doing stuff with".
> b) replace my system disk which is not part of any RAID system as of today,
> thus gaining actually both a SATA port for the RAID systems and more security
> for the boot and root filesystems
See my assumption 1, on my old P45DE core2/quad system linux can happily
see big drives (over 2TB I think is the limit) and use all the space as
one large partition or further divided, but the bios could only see a
smaller 750G amount so could not boot from my 3TB drives so while I did
all the partitioning mentioned in my replies (ready for when I upgraded
to newer hardware, which I have done) I needed a 1TB disk to hold the
"bios boot," "/boot," and "root" to be able to then see the larger
drives. (actually strictly speaking you could probably get away with
just "bios boot" and "/boot" on the smaller disk, and have /root on the
larger ones as grub2 loads the kernel file and initramfs from /boot...
I'm not 100% sure but I think grub2 can also see and understand larger
disks, so you might be able to install grub2 to the small disk (or flash
drive), which the bios can then boot from, which can then load the
kernel from the large disk's /boot raid.)
> c) also because I use to use LVM on top of RAID to get advantages of its
> flexibility (so several PVs which may or may not be aggregated later)
> Other suggestions include the use of smartctl tool. I've checked that the
> daemon was already running. But I didn't use the '-x' option that I understand
> is hardly an option !
>
> I plan to build these RAID devices out of CentOS 7 standard install process
> (I'm currently downloading a CentOS Live iso), thus presenting to the install
> some predefined partitions.
>
> I expect about 5-10 days to get these orders delivered. So more news at this time :)
>
> Thank you all for your help. I keep reading the list that I discovered for
> the occasion...
(#a) After installing to sda and booting etc, you than then install grub
on to sd[b,c,d]. This means that should you lose sda, you can boot from
any of the remaining disks without having to worry about getting a
"live" cd or some such method of recovering the system.
(#b) Should you upgrade to a UEFI motherboard and/or disable CSM remove
the array on the 4 disks (512M partitions), mark them as EF00 (EFI
System) in gdisk, format them all as fat32, install the boot loader
& /boot to disk sda2 to get a working system, replicate sda2 into
sd[b,c,d]2, to allow recovery should sda fail, and use efibootmgr to add
boot entries to NVRAM for disks b2,c2,d2. (I think grub install to each
disk under uefi should also add boot entries to the uefi NVRAM but UEFI
is much more of a pain than "bios" with stupid things such as forgetting
its entries if a disk is removed/replaced, so efibootmgr is a tool to
get used to)
>
> Regards
> Pierre
Jon.
(all the above is based on my experience/hassles as an "end user/self
learner" and various web searches and posts on this list, so may be
totally different advice from what a systems administrator would give
for a work server set up with way more experience and knowledge of just
what works best, and why, especially system/raid performance which is an
art that an end user doesn't really have to worry about as "its fast
enough/it works ok" usually suffices.)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* Re: [PATCH v2] md: fix a build warning
From: Neil Brown @ 2015-06-15 5:58 UTC (permalink / raw)
To: Firo Yang; +Cc: linux-raid, kernel-janitors
In-Reply-To: <1433986870-7014-1-git-send-email-firogm@gmail.com>
On Thu, 11 Jun 2015 09:41:10 +0800
Firo Yang <firogm@gmail.com> wrote:
> Warning like this:
>
> drivers/md/md.c: In function ‘update_array_info’:
> drivers/md/md.c:6394:26: warning: logical not is only applied
> to the left hand side of comparison [-Wlogical-not-parentheses]
> !mddev->persistent != info->not_persistent||
>
> Fix it as Neil Brown said:
> mddev->persistent != !info->not_persistent ||
>
> Signed-off-by: Firo Yang <firogm@gmail.com>
> ---
> drivers/md/md.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index dd85be9..a3528b9 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -6391,7 +6391,7 @@ static int update_array_info(struct mddev *mddev, mdu_array_info_t *info)
> mddev->ctime != info->ctime ||
> mddev->level != info->level ||
> /* mddev->layout != info->layout || */
> - !mddev->persistent != info->not_persistent||
> + mddev->persistent != !info->not_persistent ||
> mddev->chunk_sectors != info->chunk_size >> 9 ||
> /* ignore bottom 8 bits of state, and allow SB_BITMAP_PRESENT to change */
> ((state^info->state) & 0xfffffe00)
applied, thanks.
NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* mismatch_cnt constantly goes up on ssd+hdd raid1
From: tlknv @ 2015-06-14 17:13 UTC (permalink / raw)
To: linux-raid
Hello,
I have raid 1 which mirrors a root/boot partition on 1SSD and 2HDD (write-mostly). mismatch_cnt goes up even when there are very few writes to the partition as /var is mounted separatly. After I update several packages I typically see mismatch_cnt somewhere between 500,000 and 2,000,000. I have read a number of threads in this DL but could not find an explanation of what could cause mismatch_cnt to grow that much. I checked md5 sums using /var/lib/dpkg/info/*.md5sums, and didn't see many errors, even though there are few, mostly in text files which look ok to me. I guess when I check, all reads go to SSD (as both HDDs in this raid are write-mostly), and thus md5sum only shows no problem on SSD. Note, this partition is used as both boot and root and just in case here is some more info about
my system:
root@tbeh:~# uname -a
Linux tbeh 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1 (2015-05-24) x86_64 GNU/Linux
root@tbeh:~# mdadm -D /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Sun Jun 7 18:38:51 2015
Raid Level : raid1
Array Size : 13442048 (12.82 GiB 13.76 GB)
Used Dev Size : 13442048 (12.82 GiB 13.76 GB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent
Update Time : Sun Jun 14 08:12:28 2015
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Name : tbeh:0 (local to host tbeh)
UUID : c50d3fbf:5da849fc:9a6872ae:6905e381
Events : 213
Number Major Minor RaidDevice State
0 8 34 0 active sync /dev/sdc2
2 8 18 1 active sync writemostly /dev/sdb2
1 8 2 2 active sync writemostly /dev/sda2
root@tbeh:~# fdisk -l /dev/sdc
Disk /dev/sdc: 111.8 GiB, 120034123776 bytes, 234441648 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x858bea60
Device Boot Start End Sectors Size Id Type
/dev/sdc1 * 2048 50333695 50331648 24G 83 Linux
/dev/sdc2 50333696 77234175 26900480 12.8G da Non-FS data
root@tbeh:~# fdisk -l /dev/sda
Disk /dev/sda: 596.2 GiB, 640135028736 bytes, 1250263728 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0f06bf61
Device Boot Start End Sectors Size Id Type
/dev/sda1 63 498014 497952 243.1M da Non-FS data
/dev/sda2 498015 27856709 27358695 13G da Non-FS data
/dev/sda3 27856710 35889209 8032500 3.9G da Non-FS data
/dev/sda4 35889210 1250258624 1214369415 579.1G 5 Extended
/dev/sda5 35889273 82782944 46893672 22.4G da Non-FS data
/dev/sda6 82783008 976768064 893985057 426.3G da Non-FS data
/dev/sda7 976768128 1250258624 273490497 130.4G 83 Linux
root@tbeh:~# fdisk -l /dev/sdb
Disk /dev/sdb: 465.8 GiB, 500107862016 bytes, 976773168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x99a9f6d9
Device Boot Start End Sectors Size Id Type
/dev/sdb1 63 498014 497952 243.1M da Non-FS data
/dev/sdb2 498015 27856709 27358695 13G da Non-FS data
/dev/sdb3 27856710 35889209 8032500 3.9G da Non-FS data
/dev/sdb4 35889210 976768064 940878855 448.7G 5 Extended
/dev/sdb5 35889273 82782944 46893672 22.4G da Non-FS data
/dev/sdb6 82783008 976768064 893985057 426.3G da Non-FS data
Just to minize the damage, now I mount / as read only, and remount as rw as necessary. Unfortunatelly right now I don't have anything to update but AFAIR right after package update (and running echo check > /sys/block/md0/md/sync_action) mismatch_cnt wasn't too high, but it went up after I reboot the system (and ran echo check > /sys/block/md0/md/sync_action).
The following may have nothing to do with mismatch_cnt as it's observed even when mismatch_cnt is 0 (after checking of ro partition) but I want to undestand how it's possible.
Cmp of SSD and HDD partitions shows lots of differences
root@tbeh:~# sync; cmp -l /dev/sdc2 /dev/sda2|wc -l
cmp: EOF on /dev/sdc2
1903215
BTW, only first few hundren bytes (at most) have non-zero value on SSD, the rest of differences has 0 bytes on SSD.
4233 0 347
4234 70 65
4235 232 241
4257 0 1
4265 51 264
4266 271 260
4267 14 301
4268 116 317
4269 353 326
4270 21 221
4271 360 176
4272 133 265
4273 154 262
4274 56 120
4275 116 370
4276 304 72
4277 233 62
4278 241 4
4279 161 243
4280 363 353
4281 0 1
4313 31 125
4314 201 173
4315 34 102
4316 15 127
4609 0 376
4610 0 377
4611 0 376
4612 0 377
4613 0 376
4614 0 377
4615 0 376
4616 0 377
4617 0 376
4618 0 377
4619 0 376
4620 0 377
4621 0 376
4622 0 377
4623 0 376
4624 0 377
4625 0 376
4626 0 377
4627 0 376
4628 0 377
4629 0 376
...
I don't see any differences between 2 HDD partitions though.
Does anyone have any idea what could be wrong with my system or what could I try to localize the problem?
Thanks,
Boris
^ permalink raw reply
* Re: seagate-"archive"-disks with raid6?
From: Wols Lists @ 2015-06-13 20:47 UTC (permalink / raw)
To: Mikael Abrahamsson, wiebittewas; +Cc: linux-raid
In-Reply-To: <alpine.DEB.2.02.1506132207180.9487@uplift.swm.pp.se>
On 13/06/15 21:11, Mikael Abrahamsson wrote:
> On Sat, 13 Jun 2015, wiebittewas wrote:
>
>> for a larger archive, we're thinking about buying six
>> 8TB-Archive-Disks from Seagate to build a 4+2 Raid6-Array.
>>
>> now we've seen, that these disks are told not-recommended for raid,
>> because they lack ERC/TLER (like many desktop-disks)
>
> Not only that.
>
> You might want to read:
>
> http://www.spinics.net/lists/linux-ide/msg50641.html
>
> These drives are different beasts than regular HDDs, and we seem to be
> seeing problems with them just the way SSDs were problematic in the
> beginning.
>
> So you might want to reconsider using SMR drives for RAID use. I was
> considering them until I read up on them, and then I decided it was too
> early in the deployment cycle to use them for RAID use.
>
I didn't know these were shingled drives ... does the OP know what a
shingled drive is?
Basically, to get decent performance out of these drives, you have to
stream data at them - they're more like a tape-drive than a
random-access disk. So. They're great for backups, much less so for
normal use.
Cheers,
Wol
^ permalink raw reply
* Re: doubts about sdd raid1 and cfs
From: Roberto Spadim @ 2015-06-13 20:33 UTC (permalink / raw)
To: Mikael Abrahamsson; +Cc: Linux-RAID
In-Reply-To: <alpine.DEB.2.02.1506132223340.9487@uplift.swm.pp.se>
nice :)
about the space, that was my next doubt ehhe thanks ! leave 40gb to
ssd "bad blocks"
about the writemostly, any comment?
2015-06-13 17:32 GMT-03:00 Mikael Abrahamsson <swmike@swm.pp.se>:
> On Sat, 13 Jun 2015, Roberto Spadim wrote:
>
>> hum, but what about the 850 version instead of 840?
>
>
> SSDs have been around for more than 5 years (intel released the X25-M in
> 2008 according to wikipedia), TRIM has been around for 4-5 years.
>
> I have seen so many problems related to TRIM, that I will not use it unless
> it comes default on from a manufacturer that has tested the entire chain,
> including hardware and software (my Apple laptop for instance).
>
> We have seen TRIM not being NCQ enabled and stalling performance when doing
> TRIM, we have seen firmware bugs that cause drives to lose data when doing
> TRIM, we have seen Linux kernel bugs that also caused loss of data.
>
> First of all, ask yourself why you want TRIM, understand how it works and if
> you will benefit, do your research properly, and then enable it.
>
> Personally, I overprovision my SSDs instead:
>
> http://www.samsung.com/global/business/semiconductor/minisite/SSD/global/html/whitepaper/whitepaper05.html
> https://en.wikipedia.org/wiki/Write_amplification#Over-provisioning
> http://www.seagate.com/gb/en/tech-insights/ssd-over-provisioning-benefits-master-ti/
> http://www.edn.com/design/systems-design/4404566/Understanding-SSD-over-provisioning
> http://www.kingston.com/en/ssd/overprovisioning
>
> So I basically leave space on the drive that I don't use. In your case I
> would only partition 200GB (or even less) of that 240GB drive, and I would
> run it without TRIM.
>
>
> --
> Mikael Abrahamsson email: swmike@swm.pp.se
--
Roberto Spadim
SPAEmpresarial - Software ERP
Eng. Automação e Controle
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: doubts about sdd raid1 and cfs
From: Mikael Abrahamsson @ 2015-06-13 20:32 UTC (permalink / raw)
To: Roberto Spadim; +Cc: Linux-RAID
In-Reply-To: <CAH3kUhGHPD5PhBcMzjH1iQBJgviuUa76KtAp-UVSqhwFnBpY1g@mail.gmail.com>
On Sat, 13 Jun 2015, Roberto Spadim wrote:
> hum, but what about the 850 version instead of 840?
SSDs have been around for more than 5 years (intel released the X25-M
in 2008 according to wikipedia), TRIM has been around for 4-5 years.
I have seen so many problems related to TRIM, that I will not use it
unless it comes default on from a manufacturer that has tested the entire
chain, including hardware and software (my Apple laptop for instance).
We have seen TRIM not being NCQ enabled and stalling performance when
doing TRIM, we have seen firmware bugs that cause drives to lose data when
doing TRIM, we have seen Linux kernel bugs that also caused loss of data.
First of all, ask yourself why you want TRIM, understand how it works and
if you will benefit, do your research properly, and then enable it.
Personally, I overprovision my SSDs instead:
http://www.samsung.com/global/business/semiconductor/minisite/SSD/global/html/whitepaper/whitepaper05.html
https://en.wikipedia.org/wiki/Write_amplification#Over-provisioning
http://www.seagate.com/gb/en/tech-insights/ssd-over-provisioning-benefits-master-ti/
http://www.edn.com/design/systems-design/4404566/Understanding-SSD-over-provisioning
http://www.kingston.com/en/ssd/overprovisioning
So I basically leave space on the drive that I don't use. In your case I
would only partition 200GB (or even less) of that 240GB drive, and I would
run it without TRIM.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: doubts about sdd raid1 and cfs
From: Roberto Spadim @ 2015-06-13 20:18 UTC (permalink / raw)
To: Mikael Abrahamsson; +Cc: Linux-RAID
In-Reply-To: <alpine.DEB.2.02.1506132211560.9487@uplift.swm.pp.se>
hum, but what about the 850 version instead of 840?
2015-06-13 17:14 GMT-03:00 Mikael Abrahamsson <swmike@swm.pp.se>:
> On Sat, 13 Jun 2015, Roberto Spadim wrote:
>
>> hi guys, i`m setting up a new server, it have
>> 1) 2x 256gb sdd
>> Samsung SSD 850 PRO 256GB, EXM02B6Q
>> 500118192 sectors, multi 1: LBA48 NCQ (depth 31/32)
>> 2)1tb hdd
>> Hitachi HUA722010CLA330, JP4OA3EA
>> 1953525168 sectors, multi 16: LBA48 NCQ (depth 31/32)
>>
>> i`m plannig a raid1 with ssd and hdd, hdd being a "write-mostly", and
>> 750gb of hdd i will use to backup or other useless files
>>
>> i have somedoubts about the setup..
>> i use xfs at filesystem, should i consider TRIM command at filesystem?
>> the problem is hdd being part of raid1 setup with others ssd, could
>> the ssd receive the TRIM and the hdd too?
>> any idea is wellcome
>
>
> Frankly, reading about the firmware problems on Samsung SSDs in combination
> with TRIM, I would recommend against using TRIM on samsung drives.
>
> https://bugs.launchpad.net/ubuntu/+source/fstrim/+bug/1449005
>
> --
> Mikael Abrahamsson email: swmike@swm.pp.se
--
Roberto Spadim
SPAEmpresarial - Software ERP
Eng. Automação e Controle
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: doubts about sdd raid1 and cfs
From: Mikael Abrahamsson @ 2015-06-13 20:14 UTC (permalink / raw)
To: Roberto Spadim; +Cc: Linux-RAID
In-Reply-To: <CAH3kUhFLHdJ6T_RHaL2AyFHuH3Ta90DLhj8TKM5jZ_31mY2OCw@mail.gmail.com>
On Sat, 13 Jun 2015, Roberto Spadim wrote:
> hi guys, i`m setting up a new server, it have
> 1) 2x 256gb sdd
> Samsung SSD 850 PRO 256GB, EXM02B6Q
> 500118192 sectors, multi 1: LBA48 NCQ (depth 31/32)
> 2)1tb hdd
> Hitachi HUA722010CLA330, JP4OA3EA
> 1953525168 sectors, multi 16: LBA48 NCQ (depth 31/32)
>
> i`m plannig a raid1 with ssd and hdd, hdd being a "write-mostly", and
> 750gb of hdd i will use to backup or other useless files
>
> i have somedoubts about the setup..
> i use xfs at filesystem, should i consider TRIM command at filesystem?
> the problem is hdd being part of raid1 setup with others ssd, could
> the ssd receive the TRIM and the hdd too?
> any idea is wellcome
Frankly, reading about the firmware problems on Samsung SSDs in
combination with TRIM, I would recommend against using TRIM on samsung
drives.
https://bugs.launchpad.net/ubuntu/+source/fstrim/+bug/1449005
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: seagate-"archive"-disks with raid6?
From: Mikael Abrahamsson @ 2015-06-13 20:11 UTC (permalink / raw)
To: wiebittewas; +Cc: linux-raid
In-Reply-To: <557C5330.1090605@gmail.com>
On Sat, 13 Jun 2015, wiebittewas wrote:
> for a larger archive, we're thinking about buying six 8TB-Archive-Disks from Seagate to build a 4+2 Raid6-Array.
>
> now we've seen, that these disks are told not-recommended for raid, because they lack ERC/TLER (like many desktop-disks)
Not only that.
You might want to read:
http://www.spinics.net/lists/linux-ide/msg50641.html
These drives are different beasts than regular HDDs, and we seem to be
seeing problems with them just the way SSDs were problematic in the
beginning.
So you might want to reconsider using SMR drives for RAID use. I was
considering them until I read up on them, and then I decided it was too
early in the deployment cycle to use them for RAID use.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* doubts about sdd raid1 and cfs
From: Roberto Spadim @ 2015-06-13 19:42 UTC (permalink / raw)
To: Linux-RAID
hi guys, i`m setting up a new server, it have
1) 2x 256gb sdd
Samsung SSD 850 PRO 256GB, EXM02B6Q
500118192 sectors, multi 1: LBA48 NCQ (depth 31/32)
2)1tb hdd
Hitachi HUA722010CLA330, JP4OA3EA
1953525168 sectors, multi 16: LBA48 NCQ (depth 31/32)
i`m plannig a raid1 with ssd and hdd, hdd being a "write-mostly", and
750gb of hdd i will use to backup or other useless files
i have somedoubts about the setup..
i use xfs at filesystem, should i consider TRIM command at filesystem?
the problem is hdd being part of raid1 setup with others ssd, could
the ssd receive the TRIM and the hdd too?
any idea is wellcome
--
Roberto Spadim
^ permalink raw reply
* Re: seagate-"archive"-disks with raid6?
From: Wols Lists @ 2015-06-13 16:20 UTC (permalink / raw)
To: wiebittewas, linux-raid
In-Reply-To: <557C5330.1090605@gmail.com>
On 13/06/15 16:58, wiebittewas wrote:
> hi.
>
> for a larger archive, we're thinking about buying six 8TB-Archive-Disks from Seagate to build a 4+2 Raid6-Array.
>
> now we've seen, that these disks are told not-recommended for raid, because they lack ERC/TLER (like many desktop-disks)
>
> unfortunately we didn't found any real actual information about behaviour of linux-raid with such disks.
>
> so our first questions is, if anyone here already has disks (without ERC/TLER) with linux-raid running and can say something about?
>
I'm running two Seagate Barracuda disks in Raid-1. That, I believe is
okay. I've never had any problem, but the disks are pretty new...
>
> other item is, - as we understand - ERC leads to a limited timeout on read-errors, so the wanted data of that read-access may recovered by the redundant data and marks that disk for soon replace (or a sync to an existant spare is started).
>
> but this is also done, if that disk does not timeout and will be dropped by the controller, or is this wrong?
>
This is right. 6 by 8TB is 48TB of disk. If the disk spec is standard,
it says "expect one soft error per 10TB read", so a rebuild will get
FIVE errors.
If you don't do anything special, each error will kick a disk out of the
array ... OOPS !!!
> so the only benefit seems to be, that with ERC the disk is not dropped immediately, so errors on two further disks won't lead to a data-loss before sync (as long the additional errors are not corresponding raid-sectors)
>
> but: how likely is it, that three disks have permamnent errors during the sync? (we already had two disks in the past, but never three at a time...)
>
> how is your experience with multiple disk-errors?
>
No experience here, but I'm planning to upgrade to raid 5 or 6, so I've
been mugging up on it.
You need to increase the raid timeout, so the disk times out before the
raid does. That's why you really need ERC, so you can make the disk time
out before the raid does.
Cheers,
Wol
^ permalink raw reply
* seagate-"archive"-disks with raid6?
From: wiebittewas @ 2015-06-13 15:58 UTC (permalink / raw)
To: linux-raid
hi.
for a larger archive, we're thinking about buying six 8TB-Archive-Disks from Seagate to build a 4+2 Raid6-Array.
now we've seen, that these disks are told not-recommended for raid, because they lack ERC/TLER (like many desktop-disks)
unfortunately we didn't found any real actual information about behaviour of linux-raid with such disks.
so our first questions is, if anyone here already has disks (without ERC/TLER) with linux-raid running and can say something about?
other item is, - as we understand - ERC leads to a limited timeout on read-errors, so the wanted data of that read-access may recovered by the redundant data and marks that disk for soon replace (or a sync to an existant spare is started).
but this is also done, if that disk does not timeout and will be dropped by the controller, or is this wrong?
so the only benefit seems to be, that with ERC the disk is not dropped immediately, so errors on two further disks won't lead to a data-loss before sync (as long the additional errors are not corresponding raid-sectors)
but: how likely is it, that three disks have permamnent errors during the sync? (we already had two disks in the past, but never three at a time...)
how is your experience with multiple disk-errors?
regards
w.
^ permalink raw reply
* Lieber Freund!!!
From: Herr Martins D Weber @ 2015-06-13 12:14 UTC (permalink / raw)
Lieber Freund!!!
Ich vermute das diese E-Mail eine ?berraschung f?r Sie sein wird, aber es ist wahr.Ich bin bei einer routinen ?berpr?fung in meiner Bank (StandardBank PLC von S?d Afrika) wo ich arbeite, auf einem Konto gesto?en, was nicht in anspruch genommen worden ist, wo derzeit USD$18.5M (Achtzehn Million, F?nf Hundert Tausend, US Dollar) gutgeschrieben sind.Dieses Konto geh?rte Herrn Manfred Becker, der ein Kunde in unsere Bank war, der leider verstorben ist. Herrn Manfred Becker war ein geb?rtiger Deutscher.
Damit es mir m?glich ist dieses Geld $18,500,000 inanspruch zunehmen,ben?tige ich die zusammenarbeit eines Ausl?ndischen Partners wie Sie,den ich als Verwandter und Erbe des verstorbenen Herrn Manfred Becker vorstellen kann,damit wir das Geld inanspruch nehmen k?nnen. F?r diese Unterst?tzung erhalten Sie 30% der Erbschaftsumme und die restlichen 70% teile ich mirmit meinen zwei Arbeitskollegen, die mich bei dieser Transaktion ebenfalls unterst?tzen.
Wenn Sie interessiert sind, k?nnen Sie mir bitte eine E-Mail schicken, damit ich Ihnen mehr Details zukommen lassen kann. N.B.BITTE SENDEN SIE MIR Martins D Weber ANTWORT ZU durch mein E-mail: ( martinsdweber@aim.com ) f?R VERTRAULICHEN GRUND. Schicken Sie keine POST ZU MEINEM B?RO-E-MAIL.(martinsdweber@aim.com )
If you understand english,please kindly reply with english ( martinsdweber@aim.com )
Mit freundlichen GrьЯen
Herr Martins D Weber
---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: 4.1-rc6 radi5 OOPS
From: Neil Brown @ 2015-06-13 4:26 UTC (permalink / raw)
To: Jes Sorensen; +Cc: linux-raid, Xiao Ni
In-Reply-To: <wrfj1thg4mrp.fsf@redhat.com>
On Fri, 12 Jun 2015 17:52:58 -0400
Jes Sorensen <Jes.Sorensen@redhat.com> wrote:
> Neil Brown <neilb@suse.de> writes:
> > On Wed, 10 Jun 2015 12:27:35 -0400
> > Jes Sorensen <Jes.Sorensen@redhat.com> wrote:
> >
> >> Neil Brown <neilb@suse.de> writes:
> >> > On Wed, 10 Jun 2015 10:19:42 +1000 Neil Brown <neilb@suse.de> wrote:
> >> >
> >> >> So it looks like some sort of race. I have other evidence of a race
> >> >> with the resync/reshape thread starting/stopping. If I track that
> >> >> down it'll probably fix this issue too.
> >> >
> >> > I think I have found just such a race. If you request a reshape just
> >> > as a recovery completes, you can end up with two reshapes running.
> >> > This causes confusion :-)
> >> >
> >> > Can you try this patch? If I can remember how to reproduce my race
> >> > I'll test it on that too.
> >> >
> >> > Thanks,
> >> > NeilBrown
> >>
> >> Hi Neil,
> >>
> >> Thanks for the patch - I tried with this applied, but it still crashed
> >> for me :( I had to mangle it manually, somehow it got modified in the
> >> email.
> >
> > Very :-(
> >
> > I had high hopes for that patch. I cannot find anything else that could lead
> > to what you are seeing. I wish I could reproduce it but it is probably highly
> > sensitive to timing so some hardware shows it and others don't.
> >
> > It looks very much like two 'resync' threads are running at the same time.
> > When one finishes, it sets ->reshape_progress to -1 (MaxSector), which trips up
> > the other one.
> >
> > In the hang that I very rarely see, one thread (presumably) finishes and sets
> > MD_RECOVERY_DONE, so the raid5d threads waits for the resync thread to
> > complete, and that thread is waiting for the raid5d to retire some stripe_heads.
> >
> > ... though the 'resync' thread is probably actually doing a 'reshape'...
>
> Neil
>
> Good news - albeit not guaranteed yet. I tried with the full patch that
> you sent to Linus, and with that I haven't been able to reproduce the
> problem so far. I'll try and do some more testing over the weekend.
>
> The patch I manually applied only had two hunks in it, the one you
> pushed to Linus looks a lot more complete :)
Thanks for testing. I'm fairly sure you issue is fixed now, but it is very
nice to have it confirmed.
>
> > Did you get a chance to bisect it? I must admit that I doubt that would be
> > useful. It probably starts when "md_start_sync" was introduced and maybe made
> > worse when some locking with mddev_lock was relaxed.
> >
> > The only way I can see a race is if MD_RECOVERY_DONE gets left set. When a new
> > thread is started. md_check_recovery always clears it before starting a thread,
> > but raid5_start_reshape doesn't - or didn't before the patch I gave you.
> >
> > It might make more sense to clear the bit in md_reap_sync_thread as below,
> > but if the first patch didn't work, this one is unlikely to.
> >
> > Would you be able to test with the following patch? There is a chance it might
> > confirm whether two sync threads are running at the same time.
>
> I can try with this patch on too, but I won't get to it before next
> week. It's been a week of non related MD issues.
Don't bother - that one is just an early version of one that went to Linus, so
you have tested the important bit.
Thanks,
NeilBrown
>
> Thanks a lot!
>
> Cheers,
> Jes
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox