* Re: [PATCH V3 1/2] RAID1: a new I/O barrier implementation to remove resync window
From: Coly Li @ 2017-02-21 11:30 UTC (permalink / raw)
To: Wols Lists
Cc: Shaohua Li, NeilBrown, NeilBrown, linux-raid, Shaohua Li,
Johannes Thumshirn, Guoqing Jiang
In-Reply-To: <58AB320B.1060707@youngman.org.uk>
On 2017/2/21 上午2:14, Wols Lists wrote:
> On 20/02/17 08:07, Coly Li wrote:
>> For the function pointer asignment, it is because I see a brach happens in a loop. If I use a function pointer, I can avoid redundant brach inside the loop. raid1_read_request() and raid1_write_request() are not simple functions, I don't know whether gcc may make them inline or not, so I am on the way to check the disassembled code..
>
> Can you force gcc to inline or compile a function? Isn't it dangerous to
> rely on default behaviour and assume it won't change when the compiler
> is upgraded?
I choose to trust compiler, and trust the people behind gcc.
Coly
^ permalink raw reply
* Re: Reshape stalled at first badblock location (was: RAID 5 --assemble doesn't recognize all overlays as component devices)
From: Tomasz Majchrzak @ 2017-02-21 9:51 UTC (permalink / raw)
To: George Rapp; +Cc: Linux-RAID, Matthew Krumwiede
In-Reply-To: <CAF-KpgY0ySvCN9ftbDmW_P6wDiyfN2yWE6=NECVru4=vCe+pbQ@mail.gmail.com>
On Mon, Feb 20, 2017 at 05:18:46PM -0500, George Rapp wrote:
> On Sat, Feb 11, 2017 at 7:32 PM, George Rapp <george.rapp@gmail.com> wrote:
> > Previous thread: http://marc.info/?l=linux-raid&m=148564798430138&w=2
> > -- to summarize, while adding two drives to a RAID 5 array, one of the
> > existing RAID 5 component drives failed, causing the reshape progress
> > to stall at 77.5%. I removed the previous thread from this message to
> > conserve space -- before resolving that situation, another problem has
> > arisen.
> >
> > We have cloned and replaced the failed /dev/sdg with "ddrescue --force
> > -r3 -n /dev/sdh /dev/sde c/sdh-sde-recovery.log"; copied in below, or
> > viewable via https://app.box.com/v/sdh-sde-recovery . The failing
> > device was removed from the server, and the RAID component partition
> > on the cloned drive is now /dev/sdg4.
>
> [previous thread snipped - after stepping through the code under gdb,
> I realized that "mdadm --assemble --force" was needed.]
>
> # uname -a
> Linux localhost 4.3.4-200.fc22.x86_64 #1 SMP Mon Jan 25 13:37:15 UTC
> 2016 x86_64 x86_64 x86_64 GNU/Linux
> # mdadm --version
> mdadm - v3.3.4 - 3rd August 2015
>
> As previously mentioned, the device that originally failed was cloned
> to a new drive. This copy included the bad blocks list from the md
> metadata, because I'm showing 23 bad blocks on the clone target drive,
> /dev/sdg4:
>
> # mdadm --examine-badblocks /dev/sdg4
> Bad-blocks on /dev/sdg4:
> 3802454640 for 512 sectors
> 3802455664 for 512 sectors
> 3802456176 for 512 sectors
> 3802456688 for 512 sectors
> 3802457200 for 512 sectors
> 3802457712 for 512 sectors
> 3802458224 for 512 sectors
> 3802458736 for 512 sectors
> 3802459248 for 512 sectors
> 3802459760 for 512 sectors
> 3802460272 for 512 sectors
> 3802460784 for 512 sectors
> 3802461296 for 512 sectors
> 3802461808 for 512 sectors
> 3802462320 for 512 sectors
> 3802462832 for 512 sectors
> 3802463344 for 512 sectors
> 3802463856 for 512 sectors
> 3802464368 for 512 sectors
> 3802464880 for 512 sectors
> 3802465392 for 512 sectors
> 3802465904 for 512 sectors
> 3802466416 for 512 sectors
>
> However, when I run the following command to attempt to read each of
> the bad blocks, no I/O errors pop up either on the command line or in
> /var/log messages:
>
> # for i in $(mdadm --examine-badblocks /dev/sdg4 | grep "512 sectors"
> | cut -c11-20) ; do dd bs=512 if=/dev/sdg4 skip=$i count=512 | wc -c;
> done
>
> I've truncated the output, but in each case it is similar to this:
>
> 512+0 records in
> 512+0 records out
> 262144
> 262144 bytes (262 kB) copied, 0.636762 s, 412 kB/s
>
> Thus, the bad blocks on the failed hard drive are apparently now
> readable on the cloned drive.
>
> When I try to assemble the RAID 5 array, though, the process gets
> stuck at the location of the first bad block. The assemble command is:
>
> # mdadm --assemble --force /dev/md4
> --backup-file=/home/gwr/2017/2017-01/md4_backup__2017-01-25 /dev/sde4
> /dev/sdf4 /dev/sdh4 /dev/sdl4 /dev/sdg4 /dev/sdk4 /dev/sdi4 /dev/sdj4
> /dev/sdb4 /dev/sdd4
> mdadm: accepting backup with timestamp 1485366772 for array with
> timestamp 1487624068
> mdadm: /dev/md4 has been started with 9 drives (out of 10).
>
> The md4_raid5 process immediately spikes to 100% CPU utilization, and
> the reshape stops at 1901225472 KiB (which is exactly half of the
> first bad sector value, 3802454640):
>
> # cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4]
> md4 : active raid5 sde4[0] sdb4[12] sdj4[7] sdi4[8] sdk4[11] sdg4[10]
> sdl4[9] sdh4[2] sdf4[1]
> 13454923776 blocks super 1.1 level 5, 512k chunk, algorithm 2
> [10/9] [UUUUUUUUU_]
> [===================>.] reshape = 98.9% (1901225472/1922131968)
> finish=2780.9min speed=125K/sec
>
> unused devices: <none>
>
> Googling around, I get the impression that resetting the badblocks
> list is (a) not supported by the mdadm command; and (b) considered
> harmful. However, if the blocks aren't really bad any more, as they
> are now readable, does that risk still hold? How can I get this
> reshape to proceed?
Indeed, it is not possible to reset badblocks list. This list is the
indication for a driver which blocks it is not allowed to read - the
operation might succeed but the data can be out-of-date. But it still
attempts to write those blocks. If write operation is successful, the block
is removed from bad block list. I think it is the only way to clear the
list.
I guess your reshape is blocked because it cannot read the data from the
disk to restore the data on the other drive. On the other hand it cannot
clear bad block as it's not possible to execute a write on a degraded array
with bad block.
As long as you're sure the data on the disk is valid, I believe clearing
bad block list manually in metadata (no easy way to do it) would allow
reshape to complete.
Tomek
^ permalink raw reply
* Re: [PATCH V3 1/2] RAID1: a new I/O barrier implementation to remove resync window
From: Coly Li @ 2017-02-21 9:45 UTC (permalink / raw)
To: NeilBrown
Cc: Shaohua Li, linux-raid, Shaohua Li, Johannes Thumshirn,
Guoqing Jiang
In-Reply-To: <878tp0s7w8.fsf@notabene.neil.brown.name>
On 2017/2/21 上午8:29, NeilBrown wrote:
> On Mon, Feb 20 2017, Coly Li wrote:
>
>>> 在 2017年2月20日，下午3:04，Shaohua Li <shli@kernel.org> 写道：
>>>
>>>> On Mon, Feb 20, 2017 at 01:51:22PM +1100, Neil Brown wrote:
>>>>> On Mon, Feb 20 2017, NeilBrown wrote:
>>>>>
>>>>>> On Fri, Feb 17 2017, Coly Li wrote:
>>>>>>
>>>>>>> On 2017/2/16 下午3:04, NeilBrown wrote: I know you are
>>>>>>> going to change this as Shaohua wantsthe spitting to
>>>>>>> happen in a separate function, which I agree with, but
>>>>>>> there is something else wrong here. Calling
>>>>>>> bio_split/bio_chain repeatedly in a loop is dangerous.
>>>>>>> It is OK for simple devices, but when one request can
>>>>>>> wait for another request to the same device it can
>>>>>>> deadlock. This can happen with raid1. If a resync
>>>>>>> request calls raise_barrier() between one request and
>>>>>>> the next, then the next has to wait for the resync
>>>>>>> request, which has to wait for the first request. As
>>>>>>> the first request will be stuck in the queue in
>>>>>>> generic_make_request(), you get a deadlock.
>>>>>>
>>>>>> For md raid1, queue in generic_make_request(), can I
>>>>>> understand it as bio_list_on_stack in this function? And
>>>>>> queue in underlying device, can I understand it as the
>>>>>> data structures like plug->pending and
>>>>>> conf->pending_bio_list ?
>>>>>
>>>>> Yes, the queue in generic_make_request() is the
>>>>> bio_list_on_stack. That is the only queue I am talking
>>>>> about. I'm not referring to plug->pending or
>>>>> conf->pending_bio_list at all.
>>>>>
>>>>>>
>>>>>> I still don't get the point of deadlock, let me try to
>>>>>> explain why I don't see the possible deadlock. If a bio
>>>>>> is split, and the first part is processed by
>>>>>> make_request_fn(), and then a resync comes and it will
>>>>>> raise a barrier, there are 3 possible conditions, - the
>>>>>> resync I/O tries to raise barrier on same bucket of the
>>>>>> first regular bio. Then the resync task has to wait to
>>>>>> the first bio drops its conf->nr_pending[idx]
>>>>>
>>>>> Not quite. First, the resync task (in raise_barrier()) will
>>>>> wait for ->nr_waiting[idx] to be zero. We can assume this
>>>>> happens immediately. Then the resync_task will increment
>>>>> ->barrier[idx]. Only then will it wait for the first bio to
>>>>> drop ->nr_pending[idx]. The processing of that first bio
>>>>> will have submitted bios to the underlying device, and they
>>>>> will be in the bio_list_on_stack queue, and will not be
>>>>> processed until raid1_make_request() completes.
>>>>>
>>>>> The loop in raid1_make_request() will then call
>>>>> make_request_fn() which will call wait_barrier(), which
>>>>> will wait for ->barrier[idx] to be zero.
>>>>
>>>> Thinking more carefully about this.. the 'idx' that the
>>>> second bio will wait for will normally be different, so there
>>>> won't be a deadlock after all.
>>>>
>>>> However it is possible for hash_long() to produce the same
>>>> idx for two consecutive barrier_units so there is still the
>>>> possibility of a deadlock, though it isn't as likely as I
>>>> thought at first.
>>>
>>> Wrapped the function pointer issue Neil pointed out into Coly's
>>> original patch. Also fix a 'use-after-free' bug. For the
>>> deadlock issue, I'll add below patch, please check.
>>>
>>> Thanks, Shaohua
>>>
>>
Neil,
Thanks for your patient explanation, I feel I come to follow up what
you mean. Let me try to re-tell what I understand, correct me if I am
wrong.
>> Hmm, please hold, I am still thinking of it. With barrier bucket
>> and hash_long(), I don't see dead lock yet. For raid10 it might
>> happen, but once we have barrier bucket on it , there will no
>> deadlock.
>>
>> My question is, this deadlock only happens when a big bio is
>> split, and the split small bios are continuous, and the resync io
>> visiting barrier buckets in sequntial order too. In the case if
>> adjacent split regular bios or resync bios hit same barrier
>> bucket, it will be a very big failure of hash design, and should
>> have been found already. But no one complain it, so I don't
>> convince myself tje deadlock is real with io barrier buckets
>> (this is what Neil concerns).
>
> I think you are wrong about the design goal of a hash function.
> When feed a sequence of inputs, with any stride (i.e. with any
> constant difference between consecutive inputs), the output of the
> hash function should appear to be random. A random sequence can
> produce the same number twice in a row. If the hash function
> produces a number from 0 to N-1, you would expect two consecutive
> outputs to be the same about once every N inputs.
>
Yes, you are right. But when I mentioned hash conflict, I limit the
integers in range [0, 1<<38]. 38 is (64-17-9), when a 64bit LBA
address divided by 64MB I/O barrier unit size, its value range is
reduced to [0, 1<<38].
Maximum size of normal bio is 1MB, it could be split into 2 bios at most.
For DISCARD bio, its maximum size is 4GB, it could be split into 65
bios at most.
Then in this patch, the hash question is degraded to: for any
consecutive 65 integers in range [0, 1<<38], use hash_long() to hash
these 65 integers into range [0, 1023], will any hash conflict happen
among these integers ?
I tried a half range [0, 1<<37] to check hash conflict, by writing a
simple code to emulate hash calculation in the new I/O barrier patch,
to iterate all consecutive {2, 65, 128, 512} integers in range [0,
1<<37] for hash conflict.
On a 20 core CPU each run spent 7+ hours, finally I find no hash
conflict detected up to 512 consecutive integers in above limited
condition. For 1024, there are a lot hash conflict detected.
[0, 1<<37] range back to [0, 63] LBA range, this is large enough for
almost all existing md raid configuration. So for current kernel
implementation and real world device, for a single bio, there is no
possible hash conflict the new I/O barrier patch.
If bi_iter.bi_size changes from unsigned int to unsigned long in
future, the above assumption will be wrong. There will be hash
conflict, and potential dead lock, which is quite implicit. Yes, I
agree with you. No, bio split inside loop is not perfect.
> Even if there was no possibility of a deadlock from a resync
> request happening between two bios, there are other possibilities.
>
The bellowed text makes me know more about raid1 code, but confuses me
more as well. Here comes my questions,
> It is not, in general, safe to call mempool_alloc() twice in a
> row, without first ensuring that the first allocation will get
> freed by some other thread. raid1_write_request() allocates from
> r1bio_pool, and then submits bios to the underlying device, which
> get queued on bio_list_on_stack. They will not be processed until
> after raid1_make_request() completes, so when raid1_make_request
> loops around and calls raid1_write_request() again, it will try to
> allocate another r1bio from r1bio_pool, and this might end up
> waiting for the r1bio which is trapped and cannot complete.
>
Can I say that it is because blk_finish_plug() won't be called before
raid1_make_request() returns ? Then in raid1_write_request(), mbio
will be added into plug->pending, but before blk_finish_plug() is
called, they won't be handled.
> As r1bio_pool preallocates 256 entries, this is unlikely but not
> impossible. If 256 threads all attempt a write (or read) that
> crosses a boundary, then they will consume all 256 preallocated
> entries, and want more. If there is no free memory, they will block
> indefinitely.
>
If raid1_make_request() is modified into this way,
+ if (bio_data_dir(split) == READ)
+ raid1_read_request(mddev, split);
+ else
+ raid1_write_request(mddev, split);
+ if (split != bio)
+ generic_make_request(bio);
Then the original bio will be added into the bio_list_on_stack of top
level generic_make_request(), current->bio_list is initialized, when
generic_make_request() is called nested in raid1_make_request(), the
split bio will be added into current->bio_list and nothing else happens.
After the nested generic_make_request() returns, the code back to next
code of generic_make_request(),
2022 ret = q->make_request_fn(q, bio);
2023
2024 blk_queue_exit(q);
2025
2026 bio = bio_list_pop(current->bio_list);
bio_list_pop() will return the second half of the split bio, and it is
continuously handled in raid1_make_request() by calling,
2022 ret = q->make_request_fn(q, bio);
Then there is no dead lock, at all.
> bio_alloc_bioset() has punt_bios_to_rescuer() to attempt to work
> around a deadlock very similar to this, but it is a very specific
> solution, wouldn't help raid1, and is much more complex than just
> rearranging the code.
>
Yes, it is, definitely.
>
>>
>> For the function pointer asignment, it is because I see a brach
>> happens in a loop. If I use a function pointer, I can avoid
>> redundant brach inside the loop. raid1_read_request() and
>> raid1_write_request() are not simple functions, I don't know
>> whether gcc may make them inline or not, so I am on the way to
>> check the disassembled code..
>
> It is a long time since I studied how CPUs handle different sorts
> of machine code, but I'm fairly sure that and indirect branch (i.e.
> a branch through a function pointer) is harder to optimize than a
> conditional branch.
>
It makes sense to me, yes, you are right.
> But I think the readability of the code is more important, and
> having an if-then-else in a loop is more familiar to most readers
> than using a function pointer like this.
>
Copied, I agree with you.
>>
>> The loop in raid1_make_request() is quite high level, I am not
>> sure whether CPU brach pridiction may work correctly, especially
>> when it is a big DISCARD bio, using function pointer may drop a
>> possible brach.
>>
>> So I need to check what we get and lose when use function pointer
>> or not. If it is not urgent, please hold this patch for a while.
>>
>>
>> The only thing I worry in the bellowed patch is, if a very big
>> DISCARD bio comes, will the kernel space stack trend to be
>> overflow？
>
> How would a large request cause extra stack space to be used?
If my understand is correct, there is no worry here.
Please correct me.
Coly
^ permalink raw reply
* Re: [md PATCH 10/14] md/raid1: stop using bi_phys_segment
From: Ming Lei @ 2017-02-21 7:41 UTC (permalink / raw)
To: NeilBrown
Cc: Shaohua Li, open list:SOFTWARE RAID (Multiple Disks) SUPPORT,
Christoph Hellwig
In-Reply-To: <87bmtws90u.fsf@notabene.neil.brown.name>
On Tue, Feb 21, 2017 at 8:05 AM, NeilBrown <neilb@suse.com> wrote:
> On Mon, Feb 20 2017, Ming Lei wrote:
>
>> On Thu, Feb 16, 2017 at 12:39 PM, NeilBrown <neilb@suse.com> wrote:
>>>
>>> +static void inc_pending(struct r1conf *conf, sector_t start_next_window,
>>> + sector_t bi_sector)
>>> +{
>>> + /* The current request requires multiple r1_bio, so
>>> + * we need to increment the pending count, and the corresponding
>>> + * window count.
>>> + */
>>> + spin_lock(&conf->resync_lock);
>>> + conf->nr_pending++;
>>
>> Just be curious, in current code 'nr_pending' is increased in wait_barrier(),
>> and looks this patch introduces inc_pending() to do that on each r10_bio, but
>> not see any change in wait_barrier(), so that means there might be issue in
>> current implementation about operating on this counter?
>
> Did you read the more detailed description in the previous raid10.c
> patch?
> This patch follows the same logic as that patch.
OK, I see the point now:
- for the 1st r1_bio, conf->nr_pending is increased in wait_barrier()
- for the others, conf->nr_pending is increased in inc_pending().
Also I have another question:
- before this patch, both number of requests in windows
are increased only for WRITE I/O(see wait_barrier()), and decreased
for both READ/WRITE in complete path(see allow_barrier())
- after this patch, except for the 1st r1_bio, number of requests in
windows are increased for both WRITE/READ I/O, and decreased
for both READ/WRITE too.
Could you explain a bit about this change?
Thanks,
Ming Lei
^ permalink raw reply
* Re: assistance recovering failed raid6 array
From: Brad Campbell @ 2017-02-21 2:03 UTC (permalink / raw)
To: Phil Turmel, Wols Lists, Martin Bosner; +Cc: linux-raid
In-Reply-To: <6286c879-2c58-ee43-86fb-22e83c1964c4@turmel.org>
On 21/02/17 05:21, Phil Turmel wrote:
> On 02/20/2017 03:45 PM, Wols Lists wrote:
>
>> But there's a request on the linux wiki program for someone to write a
>> utility program that takes a ddrescue log and flags the duff sectors as
>> "soft unreadable". That would mean that if you can recover 35 drives,
>> provided no stripe has lost two sectors across two drives, you wouldn't
>> lose any data.
>>
>> If you want to try and write that utility? Or if you want to email me a
>> ddrescue log with a bunch of failed sectors, I'll have a go at writing
>> it myself :-)
>
> Check out hdparm --make-bad-sector. You can get what you are describing
> by scripting that. It's marked very dangerous, but I guess if one has
> nothing to lose....
>
Wol and I have tic-tacced on that a couple of times. He suggested the
idea and I proved the viability of it by testing hdparm in a RAID to do
exactly that. Neither of us has had a chance to stitch it all together,
but the preliminary tests indicate that would do *exactly* what was
required.
Given the hdparm commands are completely reversible and non-permanent,
*and* they are being executed on a destination of a ddrescue, there is
pretty much no risk (as long as the underlying glue code to be written
gets the sector numbers and offsets right).
Regards,
Brad
^ permalink raw reply
* Re: [PATCH V3 1/2] RAID1: a new I/O barrier implementation to remove resync window
From: NeilBrown @ 2017-02-21 0:29 UTC (permalink / raw)
To: Coly Li, Shaohua Li
Cc: NeilBrown, linux-raid, Shaohua Li, Johannes Thumshirn,
Guoqing Jiang
In-Reply-To: <7178ADEA-6263-4FB0-95A5-96E5F71A9740@suse.de>
[-- Attachment #1: Type: text/plain, Size: 7052 bytes --]
On Mon, Feb 20 2017, Coly Li wrote:
> 发自我的 iPhone
>> 在 2017年2月20日，下午3:04，Shaohua Li <shli@kernel.org> 写道：
>>
>>> On Mon, Feb 20, 2017 at 01:51:22PM +1100, Neil Brown wrote:
>>>> On Mon, Feb 20 2017, NeilBrown wrote:
>>>>
>>>>> On Fri, Feb 17 2017, Coly Li wrote:
>>>>>
>>>>>> On 2017/2/16 下午3:04, NeilBrown wrote:
>>>>>> I know you are going to change this as Shaohua wantsthe spitting to
>>>>>> happen in a separate function, which I agree with, but there is
>>>>>> something else wrong here. Calling bio_split/bio_chain repeatedly
>>>>>> in a loop is dangerous. It is OK for simple devices, but when one
>>>>>> request can wait for another request to the same device it can
>>>>>> deadlock. This can happen with raid1. If a resync request calls
>>>>>> raise_barrier() between one request and the next, then the next has
>>>>>> to wait for the resync request, which has to wait for the first
>>>>>> request. As the first request will be stuck in the queue in
>>>>>> generic_make_request(), you get a deadlock.
>>>>>
>>>>> For md raid1, queue in generic_make_request(), can I understand it as
>>>>> bio_list_on_stack in this function? And queue in underlying device,
>>>>> can I understand it as the data structures like plug->pending and
>>>>> conf->pending_bio_list ?
>>>>
>>>> Yes, the queue in generic_make_request() is the bio_list_on_stack. That
>>>> is the only queue I am talking about. I'm not referring to
>>>> plug->pending or conf->pending_bio_list at all.
>>>>
>>>>>
>>>>> I still don't get the point of deadlock, let me try to explain why I
>>>>> don't see the possible deadlock. If a bio is split, and the first part
>>>>> is processed by make_request_fn(), and then a resync comes and it will
>>>>> raise a barrier, there are 3 possible conditions,
>>>>> - the resync I/O tries to raise barrier on same bucket of the first
>>>>> regular bio. Then the resync task has to wait to the first bio drops
>>>>> its conf->nr_pending[idx]
>>>>
>>>> Not quite.
>>>> First, the resync task (in raise_barrier()) will wait for ->nr_waiting[idx]
>>>> to be zero. We can assume this happens immediately.
>>>> Then the resync_task will increment ->barrier[idx].
>>>> Only then will it wait for the first bio to drop ->nr_pending[idx].
>>>> The processing of that first bio will have submitted bios to the
>>>> underlying device, and they will be in the bio_list_on_stack queue, and
>>>> will not be processed until raid1_make_request() completes.
>>>>
>>>> The loop in raid1_make_request() will then call make_request_fn() which
>>>> will call wait_barrier(), which will wait for ->barrier[idx] to be
>>>> zero.
>>>
>>> Thinking more carefully about this.. the 'idx' that the second bio will
>>> wait for will normally be different, so there won't be a deadlock after
>>> all.
>>>
>>> However it is possible for hash_long() to produce the same idx for two
>>> consecutive barrier_units so there is still the possibility of a
>>> deadlock, though it isn't as likely as I thought at first.
>>
>> Wrapped the function pointer issue Neil pointed out into Coly's original patch.
>> Also fix a 'use-after-free' bug. For the deadlock issue, I'll add below patch,
>> please check.
>>
>> Thanks,
>> Shaohua
>>
>
> Hmm, please hold, I am still thinking of it. With barrier bucket and
> hash_long(), I don't see dead lock yet. For raid10 it might happen,
> but once we have barrier bucket on it , there will no deadlock.
>
> My question is, this deadlock only happens when a big bio is split,
> and the split small bios are continuous, and the resync io visiting
> barrier buckets in sequntial order too. In the case if adjacent split
> regular bios or resync bios hit same barrier bucket, it will be a very
> big failure of hash design, and should have been found already. But no
> one complain it, so I don't convince myself tje deadlock is real with
> io barrier buckets (this is what Neil concerns).
I think you are wrong about the design goal of a hash function.
When feed a sequence of inputs, with any stride (i.e. with any constant
difference between consecutive inputs), the output of the hash function
should appear to be random.
A random sequence can produce the same number twice in a row.
If the hash function produces a number from 0 to N-1, you would expect
two consecutive outputs to be the same about once every N inputs.
Even if there was no possibility of a deadlock from a resync request
happening between two bios, there are other possibilities.
It is not, in general, safe to call mempool_alloc() twice in a row,
without first ensuring that the first allocation will get freed by some
other thread. raid1_write_request() allocates from r1bio_pool, and then
submits bios to the underlying device, which get queued on
bio_list_on_stack. They will not be processed until after
raid1_make_request() completes, so when raid1_make_request loops around
and calls raid1_write_request() again, it will try to allocate another
r1bio from r1bio_pool, and this might end up waiting for the r1bio which
is trapped and cannot complete.
As r1bio_pool preallocates 256 entries, this is unlikely but not
impossible. If 256 threads all attempt a write (or read) that crosses a
boundary, then they will consume all 256 preallocated entries, and want
more. If there is no free memory, they will block indefinitely.
bio_alloc_bioset() has punt_bios_to_rescuer() to attempt to work around
a deadlock very similar to this, but it is a very specific solution,
wouldn't help raid1, and is much more complex than just rearranging the
code.
>
> For the function pointer asignment, it is because I see a brach
> happens in a loop. If I use a function pointer, I can avoid redundant
> brach inside the loop. raid1_read_request() and raid1_write_request()
> are not simple functions, I don't know whether gcc may make them
> inline or not, so I am on the way to check the disassembled code..
It is a long time since I studied how CPUs handle different sorts of
machine code, but I'm fairly sure that and indirect branch (i.e. a
branch through a function pointer) is harder to optimize than a
conditional branch.
But I think the readability of the code is more important, and having an
if-then-else in a loop is more familiar to most readers than using a
function pointer like this.
>
> The loop in raid1_make_request() is quite high level, I am not sure
> whether CPU brach pridiction may work correctly, especially when it is
> a big DISCARD bio, using function pointer may drop a possible brach.
>
> So I need to check what we get and lose when use function pointer or
> not. If it is not urgent, please hold this patch for a while.
>
> The only thing I worry in the bellowed patch is, if a very big DISCARD
> bio comes, will the kernel space stack trend to be overflow？
How would a large request cause extra stack space to be used?
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply
* Re: [md PATCH 10/14] md/raid1: stop using bi_phys_segment
From: NeilBrown @ 2017-02-21 0:05 UTC (permalink / raw)
To: Ming Lei
Cc: Shaohua Li, open list:SOFTWARE RAID (Multiple Disks) SUPPORT,
Christoph Hellwig
In-Reply-To: <CACVXFVMZqivgG39HBAjPKE+CJ3=nXZ3fD0XfujeUbe+_3J9HLg@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 930 bytes --]
On Mon, Feb 20 2017, Ming Lei wrote:
> On Thu, Feb 16, 2017 at 12:39 PM, NeilBrown <neilb@suse.com> wrote:
>>
>> +static void inc_pending(struct r1conf *conf, sector_t start_next_window,
>> + sector_t bi_sector)
>> +{
>> + /* The current request requires multiple r1_bio, so
>> + * we need to increment the pending count, and the corresponding
>> + * window count.
>> + */
>> + spin_lock(&conf->resync_lock);
>> + conf->nr_pending++;
>
> Just be curious, in current code 'nr_pending' is increased in wait_barrier(),
> and looks this patch introduces inc_pending() to do that on each r10_bio, but
> not see any change in wait_barrier(), so that means there might be issue in
> current implementation about operating on this counter?
Did you read the more detailed description in the previous raid10.c
patch?
This patch follows the same logic as that patch.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply
* Reshape stalled at first badblock location (was: RAID 5 --assemble doesn't recognize all overlays as component devices)
From: George Rapp @ 2017-02-20 22:18 UTC (permalink / raw)
To: Linux-RAID; +Cc: Matthew Krumwiede
On Sat, Feb 11, 2017 at 7:32 PM, George Rapp <george.rapp@gmail.com> wrote:
> Previous thread: http://marc.info/?l=linux-raid&m=148564798430138&w=2
> -- to summarize, while adding two drives to a RAID 5 array, one of the
> existing RAID 5 component drives failed, causing the reshape progress
> to stall at 77.5%. I removed the previous thread from this message to
> conserve space -- before resolving that situation, another problem has
> arisen.
>
> We have cloned and replaced the failed /dev/sdg with "ddrescue --force
> -r3 -n /dev/sdh /dev/sde c/sdh-sde-recovery.log"; copied in below, or
> viewable via https://app.box.com/v/sdh-sde-recovery . The failing
> device was removed from the server, and the RAID component partition
> on the cloned drive is now /dev/sdg4.
[previous thread snipped - after stepping through the code under gdb,
I realized that "mdadm --assemble --force" was needed.]
# uname -a
Linux localhost 4.3.4-200.fc22.x86_64 #1 SMP Mon Jan 25 13:37:15 UTC
2016 x86_64 x86_64 x86_64 GNU/Linux
# mdadm --version
mdadm - v3.3.4 - 3rd August 2015
As previously mentioned, the device that originally failed was cloned
to a new drive. This copy included the bad blocks list from the md
metadata, because I'm showing 23 bad blocks on the clone target drive,
/dev/sdg4:
# mdadm --examine-badblocks /dev/sdg4
Bad-blocks on /dev/sdg4:
3802454640 for 512 sectors
3802455664 for 512 sectors
3802456176 for 512 sectors
3802456688 for 512 sectors
3802457200 for 512 sectors
3802457712 for 512 sectors
3802458224 for 512 sectors
3802458736 for 512 sectors
3802459248 for 512 sectors
3802459760 for 512 sectors
3802460272 for 512 sectors
3802460784 for 512 sectors
3802461296 for 512 sectors
3802461808 for 512 sectors
3802462320 for 512 sectors
3802462832 for 512 sectors
3802463344 for 512 sectors
3802463856 for 512 sectors
3802464368 for 512 sectors
3802464880 for 512 sectors
3802465392 for 512 sectors
3802465904 for 512 sectors
3802466416 for 512 sectors
However, when I run the following command to attempt to read each of
the bad blocks, no I/O errors pop up either on the command line or in
/var/log messages:
# for i in $(mdadm --examine-badblocks /dev/sdg4 | grep "512 sectors"
| cut -c11-20) ; do dd bs=512 if=/dev/sdg4 skip=$i count=512 | wc -c;
done
I've truncated the output, but in each case it is similar to this:
512+0 records in
512+0 records out
262144
262144 bytes (262 kB) copied, 0.636762 s, 412 kB/s
Thus, the bad blocks on the failed hard drive are apparently now
readable on the cloned drive.
When I try to assemble the RAID 5 array, though, the process gets
stuck at the location of the first bad block. The assemble command is:
# mdadm --assemble --force /dev/md4
--backup-file=/home/gwr/2017/2017-01/md4_backup__2017-01-25 /dev/sde4
/dev/sdf4 /dev/sdh4 /dev/sdl4 /dev/sdg4 /dev/sdk4 /dev/sdi4 /dev/sdj4
/dev/sdb4 /dev/sdd4
mdadm: accepting backup with timestamp 1485366772 for array with
timestamp 1487624068
mdadm: /dev/md4 has been started with 9 drives (out of 10).
The md4_raid5 process immediately spikes to 100% CPU utilization, and
the reshape stops at 1901225472 KiB (which is exactly half of the
first bad sector value, 3802454640):
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md4 : active raid5 sde4[0] sdb4[12] sdj4[7] sdi4[8] sdk4[11] sdg4[10]
sdl4[9] sdh4[2] sdf4[1]
13454923776 blocks super 1.1 level 5, 512k chunk, algorithm 2
[10/9] [UUUUUUUUU_]
[===================>.] reshape = 98.9% (1901225472/1922131968)
finish=2780.9min speed=125K/sec
unused devices: <none>
Googling around, I get the impression that resetting the badblocks
list is (a) not supported by the mdadm command; and (b) considered
harmful. However, if the blocks aren't really bad any more, as they
are now readable, does that risk still hold? How can I get this
reshape to proceed?
Updated mdadm --examine output is at
https://app.box.com/v/raid-status-2017-02-20
--
George Rapp (Pataskala, OH) Home: george.rapp -- at -- gmail.com
LinkedIn profile: https://www.linkedin.com/in/georgerapp
Phone: +1 740 936 RAPP (740 936 7277)
^ permalink raw reply
* Trouble reassembling RAID10
From: Roger Roglans @ 2017-02-20 21:42 UTC (permalink / raw)
To: linux-raid
Hey new to the mailing list and fairly new to RAIDs in general. I ran
into an issue and was hoping someone could help.
Our server that runs a 14 drive RAID10 through a rocketraid 2470
controller refused to assemble. Our goal is not necessarily to recover
a working RAID, but to get as much data back as possible.
Maybe as a consequence of the assembly failure, upon shutting down the
server, it would get stuck in boot loops. So I'm currently running
Ubuntu 16.04.1 from a USB. I've determined that 2 of 14 disks are
faulty and have determined which ones they are.
Here is the output of a mdadm --examine call.
ubuntu@ubuntu:~$ sudo mdadm --examine /dev/sd[c-p]1 | egrep
'Events | /dev/sd'
Events : 21988
Events : 21988
Events : 21988
Events : 21988
Events : 21988
Events : 21988
Events : 21988
Events : 21988
Events : 21988
Events : 21988
Events : 21988
Events : 560
Events : 21944
Events : 560
However, I keep running into an error:
ubuntu@ubuntu:~$ sudo mdadm --assemble --verbose --force /dev/md0
/dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1 /dev/sdi1
/dev/sdj1 /dev/sdk1 /dev/sdl1 /dev/sdm1 /dev/sdn1 /dev/sdo1 /dev/sdp1
mdadm: looking for devices for /dev/md0
mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdg1 is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sdh1 is identified as a member of /dev/md0, slot 5.
mdadm: /dev/sdi1 is identified as a member of /dev/md0, slot 6.
mdadm: /dev/sdj1 is identified as a member of /dev/md0, slot 7.
mdadm: /dev/sdk1 is identified as a member of /dev/md0, slot 8.
mdadm: /dev/sdl1 is identified as a member of /dev/md0, slot 9.
mdadm: /dev/sdm1 is identified as a member of /dev/md0, slot 10.
mdadm: /dev/sdn1 is identified as a member of /dev/md0, slot 11.
mdadm: /dev/sdo1 is identified as a member of /dev/md0, slot 12.
mdadm: /dev/sdp1 is identified as a member of /dev/md0, slot 13.
mdadm: added /dev/sdd1 to /dev/md0 as 1
mdadm: added /dev/sde1 to /dev/md0 as 2
mdadm: added /dev/sdf1 to /dev/md0 as 3
mdadm: added /dev/sdg1 to /dev/md0 as 4
mdadm: added /dev/sdh1 to /dev/md0 as 5
mdadm: added /dev/sdi1 to /dev/md0 as 6
mdadm: added /dev/sdj1 to /dev/md0 as 7
mdadm: added /dev/sdk1 to /dev/md0 as 8
mdadm: added /dev/sdl1 to /dev/md0 as 9
mdadm: added /dev/sdm1 to /dev/md0 as 10
mdadm: added /dev/sdn1 to /dev/md0 as 11 (possibly out of date)
mdadm: added /dev/sdo1 to /dev/md0 as 12 (possibly out of date)
mdadm: added /dev/sdp1 to /dev/md0 as 13 (possibly out of date)
mdadm: added /dev/sdc1 to /dev/md0 as 0
mdadm: /dev/md0 assembled from 11 drives - not enough to start the array.
and trying to add --run gives this error:
ubuntu@ubuntu:~$ sudo mdadm --assemble --verbose --run --force
/dev/md1 /dev/sdc1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1 /dev/sdi1
/dev/sdj1 /dev/sdk1 /dev/sdl1 /dev/sdm1 /dev/sdo1
mdadm: looking for devices for /dev/md1
mdadm: /dev/sdc1 is identified as a member of /dev/md1, slot 0.
mdadm: /dev/sde1 is identified as a member of /dev/md1, slot 2.
mdadm: /dev/sdf1 is identified as a member of /dev/md1, slot 3.
mdadm: /dev/sdg1 is identified as a member of /dev/md1, slot 4.
mdadm: /dev/sdh1 is identified as a member of /dev/md1, slot 5.
mdadm: /dev/sdi1 is identified as a member of /dev/md1, slot 6.
mdadm: /dev/sdj1 is identified as a member of /dev/md1, slot 7.
mdadm: /dev/sdk1 is identified as a member of /dev/md1, slot 8.
mdadm: /dev/sdl1 is identified as a member of /dev/md1, slot 9.
mdadm: /dev/sdm1 is identified as a member of /dev/md1, slot 10.
mdadm: /dev/sdo1 is identified as a member of /dev/md1, slot 12.
mdadm: no uptodate device for slot 2 of /dev/md1
mdadm: added /dev/sde1 to /dev/md1 as 2
mdadm: added /dev/sdf1 to /dev/md1 as 3
mdadm: added /dev/sdg1 to /dev/md1 as 4
mdadm: added /dev/sdh1 to /dev/md1 as 5
mdadm: added /dev/sdi1 to /dev/md1 as 6
mdadm: added /dev/sdj1 to /dev/md1 as 7
mdadm: added /dev/sdk1 to /dev/md1 as 8
mdadm: added /dev/sdl1 to /dev/md1 as 9
mdadm: added /dev/sdm1 to /dev/md1 as 10
mdadm: no uptodate device for slot 22 of /dev/md1
mdadm: added /dev/sdo1 to /dev/md1 as 12 (possibly out of date)
mdadm: no uptodate device for slot 26 of /dev/md1
mdadm: added /dev/sdc1 to /dev/md1 as 0
mdadm: failed to RUN_ARRAY /dev/md1: Input/output error
mdadm: Not enough devices to start the array.
So it's clear that the last three drives are out of date. It's
possible that drives 11 and 13 were never really active, but since
they were only partners in a raid 1, the array was unaffected until
now. I'm hoping that if I can reassemble with the 12th drive, then I
will be able to recover most of the data. Does anyone know what I can
do about that? I've also tried without the "inactive" drives but it
still isn't assembling drive 12. I know that I can try using --run,
but I'm not sure if I'll lose data that way. I'm also hesitant to zero
the superblock because I've always heard it was a last resort option.
Note that because I'm running this off a USB the usual `cat
/proc/mdstat` doesn't return the previous array. Also, I don't know
the exact structure of the array (if I did this would be a lot
easier).
Thanks in advance for the help,
Roger
^ permalink raw reply
* Re: assistance recovering failed raid6 array
From: Phil Turmel @ 2017-02-20 21:30 UTC (permalink / raw)
To: Martin Bosner; +Cc: linux-raid
In-Reply-To: <30DDAE62-6D76-4999-A69A-A6C16D082DA9@bosner.de>
On 02/20/2017 02:31 PM, Martin Bosner wrote:
> Is there a way tell mdadm to use sdk1 as a “active” device? And how
> can i tell the array that it should not try to recover sdm1 but set
> it active ? Is there any magic to force the state ? It might not be
> healthy for the normal use case but might be helpful for me.
No. There's no way to do that. Short of reading the kernel code to do
hex editing on the superblock. And since the spare status is there due
to an --add action, you can't trust that anything else there would be
safe for --create --assume-clean. You would scramble these eggs even
further.
Phil
^ permalink raw reply
* Re: assistance recovering failed raid6 array
From: Phil Turmel @ 2017-02-20 21:21 UTC (permalink / raw)
To: Wols Lists, Martin Bosner; +Cc: linux-raid
In-Reply-To: <58AB5555.5070504@youngman.org.uk>
On 02/20/2017 03:45 PM, Wols Lists wrote:
> But there's a request on the linux wiki program for someone to write a
> utility program that takes a ddrescue log and flags the duff sectors as
> "soft unreadable". That would mean that if you can recover 35 drives,
> provided no stripe has lost two sectors across two drives, you wouldn't
> lose any data.
>
> If you want to try and write that utility? Or if you want to email me a
> ddrescue log with a bunch of failed sectors, I'll have a go at writing
> it myself :-)
Check out hdparm --make-bad-sector. You can get what you are describing
by scripting that. It's marked very dangerous, but I guess if one has
nothing to lose....
Phil
^ permalink raw reply
* Re: assistance recovering failed raid6 array
From: Wols Lists @ 2017-02-20 20:45 UTC (permalink / raw)
To: Phil Turmel, Martin Bosner; +Cc: linux-raid
In-Reply-To: <fcf34399-2f7a-f444-bff7-512ae29d2e7b@turmel.org>
On 20/02/17 19:16, Phil Turmel wrote:
> For every stripe in the array, you need 34 devices of 36 to be
> readable. Any time you fall back on ddrescue to make one of those
> 34, you are ensuring that some data is lost. But yes, that would
> otherwise work. The 2/3 recovered disk is only useful in this (use
> ddrescue to get as much of the missing disk as possible).
I keep on asking :-)
But there's a request on the linux wiki program for someone to write a
utility program that takes a ddrescue log and flags the duff sectors as
"soft unreadable". That would mean that if you can recover 35 drives,
provided no stripe has lost two sectors across two drives, you wouldn't
lose any data.
If you want to try and write that utility? Or if you want to email me a
ddrescue log with a bunch of failed sectors, I'll have a go at writing
it myself :-)
Cheers,
Wol
^ permalink raw reply
* Re: assistance recovering failed raid6 array
From: Martin Bosner @ 2017-02-20 19:31 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
In-Reply-To: <fcf34399-2f7a-f444-bff7-512ae29d2e7b@turmel.org>
> On 20 Feb 2017, at 20:16, Phil Turmel <philip@turmel.org> wrote:
>
> If you can get to 35 of 36 original disks, even with scattered errors,
> you could complete a check scrub to make 35 good disks. With timeout
> mismatch, you'll have to override the kernel timeouts for all devices,
> so such a scrub would take a very long time, but would recover everything.
Is there a way tell mdadm to use sdk1 as a “active” device? And how can i tell the array that it should not try to recover sdm1 but set it active ? Is there any magic to force the state ? It might not be healthy for the normal use case but might be helpful for me.
Martin
^ permalink raw reply
* Re: assistance recovering failed raid6 array
From: Phil Turmel @ 2017-02-20 19:16 UTC (permalink / raw)
To: Martin Bosner; +Cc: linux-raid
In-Reply-To: <54F6D166-0D54-49EF-B967-124DC582B299@bosner.de>
On 02/20/2017 01:27 PM, Martin Bosner wrote:
>> If you have any access to the two "dead" drives, there might be a
>> slight chance. Since they were likely kicked out due to timeout
>> mismatch, not a complete failure, this could be possible.
>>
>> Otherwise, you are utterly screwed. Sorry.
>
> The disks are dead. I already tried different boards but that did
> not help.
Oh, well.
> What would happen if i recreate the array with —assume-clean ? Would
> i be able to start the array? Can I mark disks as clean? I actually
> have one failed disks, one nearly recovered disk and one that has
> been copied by 2/3 ...
For every stripe in the array, you need 34 devices of 36 to be
readable. Any time you fall back on ddrescue to make one of those
34, you are ensuring that some data is lost. But yes, that would
otherwise work. The 2/3 recovered disk is only useful in this (use
ddrescue to get as much of the missing disk as possible).
If you can get to 35 of 36 original disks, even with scattered errors,
you could complete a check scrub to make 35 good disks. With timeout
mismatch, you'll have to override the kernel timeouts for all devices,
so such a scrub would take a very long time, but would recover everything.
> Martin
Phil
^ permalink raw reply
* Re: assistance recovering failed raid6 array
From: Martin Bosner @ 2017-02-20 19:11 UTC (permalink / raw)
To: Wols Lists; +Cc: Phil Turmel, linux-raid
In-Reply-To: <58AB3D0F.50602@youngman.org.uk>
> On 20 Feb 2017, at 20:01, Wols Lists <antlists@youngman.org.uk> wrote:
>>
> You can try "--assemble --force". It sounds like you might well get away
> with it.
Would it be possible to start the array by adding sdk1 (setting state as active) and resetting the state of sdm1? The array failed while i was copying stuff to another place ...
With —assemble —force i get this:
mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Used Dev Size : -1
Raid Devices : 36
Total Devices : 35
Persistence : Superblock is persistent
Update Time : Wed Feb 15 14:08:28 2017
State : active, FAILED, Not Started
Active Devices : 33
Working Devices : 35
Failed Devices : 0
Spare Devices : 2
Layout : left-symmetric
Chunk Size : 512K
Name : media-storage:0 (local to host media-storage)
UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Events : 140559
Number Major Minor RaidDevice State
0 8 1 0 active sync /dev/sda1
1 8 17 1 active sync /dev/sdb1
2 8 33 2 active sync /dev/sdc1
3 8 49 3 active sync /dev/sdd1
4 8 65 4 active sync /dev/sde1
5 8 81 5 active sync /dev/sdf1
6 8 97 6 active sync /dev/sdg1
14 0 0 14 removed
8 8 129 8 active sync /dev/sdi1
9 8 145 9 active sync /dev/sdj1
20 0 0 20 removed
39 8 177 11 active sync /dev/sdl1
12 8 193 12 spare rebuilding /dev/sdm1
13 8 209 13 active sync /dev/sdn1
14 8 225 14 active sync /dev/sdo1
40 8 241 15 active sync /dev/sdp1
16 65 1 16 active sync /dev/sdq1
17 65 17 17 active sync /dev/sdr1
18 65 33 18 active sync /dev/sds1
19 65 49 19 active sync /dev/sdt1
20 65 65 20 active sync /dev/sdu1
21 65 81 21 active sync /dev/sdv1
22 65 97 22 active sync /dev/sdw1
43 65 113 23 active sync /dev/sdx1
36 65 129 24 active sync /dev/sdy1
25 65 145 25 active sync /dev/sdz1
41 65 161 26 active sync /dev/sdaa1
27 65 177 27 active sync /dev/sdab1
28 65 193 28 active sync /dev/sdac1
37 65 209 29 active sync /dev/sdad1
38 65 225 30 active sync /dev/sdae1
42 65 241 31 active sync /dev/sdaf1
32 66 1 32 active sync /dev/sdag1
33 66 17 33 active sync /dev/sdah1
34 66 33 34 active sync /dev/sdai1
35 66 49 35 active sync /dev/sdaj1
44 8 161 - spare /dev/sdk1
Cheers
Martin
^ permalink raw reply
* Re: assistance recovering failed raid6 array
From: Wols Lists @ 2017-02-20 19:01 UTC (permalink / raw)
To: Martin Bosner, Phil Turmel; +Cc: linux-raid
In-Reply-To: <54F6D166-0D54-49EF-B967-124DC582B299@bosner.de>
On 20/02/17 18:27, Martin Bosner wrote:
>
>> On 20 Feb 2017, at 19:11, Phil Turmel <philip@turmel.org> wrote:
>>
>> Of the 36 original disks, you have 34. You have one incomplete
>> rebuild, meaning it is still technically a spare. One of the still
>> active 34 is also showing pending relocations, meaning that disk will
>> not be able to supply all sectors to complete any recovery.
>> { /dev/sdah, serial # S1F0FPYR }
>>
>> If you have any access to the two "dead" drives, there might be
>> a slight chance. Since they were likely kicked out due to timeout
>> mismatch, not a complete failure, this could be possible.
>>
>> Otherwise, you are utterly screwed. Sorry.
>
> The disks are dead. I already tried different boards but that did not help.
If it had been the timeout problem, you would probably have been able to
recover the array. As it isn't :-(
>
> What would happen if i recreate the array with —assume-clean ? Would i be able to start the array? Can I mark disks as clean? I actually have one failed disks, one nearly recovered disk and one that has been copied by 2/3 ...
>
You can try "--assemble --force". It sounds like you might well get away
with it.
BUT! DO NOT ATTEMPT TO USE THE ARRAY IF IT LOOKS LIKE IT'S OKAY. Are all
the disks the same age? In which case, all the other drives are on the
verge of failure, too!
I don't know whether to suggest you use smartctl to see what state the
drives are in (I've seen too many reports of allegedly healthy drives
failing, so I wouldn't trust it, especially with this particular drive.)
ddrescue your remaining drives *now*, and hope you're okay. You say you
have 35GB, across 36 drives, so you should only be using the first 1GB
of each drive. We hope ...
> Martin
>
Cheers,
Wol
^ permalink raw reply
* Re: assistance recovering failed raid6 array
From: Martin Bosner @ 2017-02-20 18:27 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
In-Reply-To: <676bd1fa-4b97-2c6a-05b8-bd23290fb9a6@turmel.org>
> On 20 Feb 2017, at 19:11, Phil Turmel <philip@turmel.org> wrote:
>
> Of the 36 original disks, you have 34. You have one incomplete
> rebuild, meaning it is still technically a spare. One of the still
> active 34 is also showing pending relocations, meaning that disk will
> not be able to supply all sectors to complete any recovery.
> { /dev/sdah, serial # S1F0FPYR }
>
> If you have any access to the two "dead" drives, there might be
> a slight chance. Since they were likely kicked out due to timeout
> mismatch, not a complete failure, this could be possible.
>
> Otherwise, you are utterly screwed. Sorry.
The disks are dead. I already tried different boards but that did not help.
What would happen if i recreate the array with —assume-clean ? Would i be able to start the array? Can I mark disks as clean? I actually have one failed disks, one nearly recovered disk and one that has been copied by 2/3 ...
Martin
^ permalink raw reply
* Re: [PATCH V3 1/2] RAID1: a new I/O barrier implementation to remove resync window
From: Wols Lists @ 2017-02-20 18:14 UTC (permalink / raw)
To: Coly Li, Shaohua Li
Cc: NeilBrown, NeilBrown, linux-raid, Shaohua Li, Johannes Thumshirn,
Guoqing Jiang
In-Reply-To: <7178ADEA-6263-4FB0-95A5-96E5F71A9740@suse.de>
On 20/02/17 08:07, Coly Li wrote:
> For the function pointer asignment, it is because I see a brach happens in a loop. If I use a function pointer, I can avoid redundant brach inside the loop. raid1_read_request() and raid1_write_request() are not simple functions, I don't know whether gcc may make them inline or not, so I am on the way to check the disassembled code..
Can you force gcc to inline or compile a function? Isn't it dangerous to
rely on default behaviour and assume it won't change when the compiler
is upgraded?
Cheers,
Wol
^ permalink raw reply
* Re: assistance recovering failed raid6 array
From: Martin Bosner @ 2017-02-20 18:13 UTC (permalink / raw)
To: Roman Mamedov; +Cc: Phil Turmel, linux-raid
In-Reply-To: <20170220225028.59824758@natsu>
> So you have the most terrible hard drive possible [1][2](they WILL ALL fail),
That is very true.
^ permalink raw reply
* Re: assistance recovering failed raid6 array
From: Phil Turmel @ 2017-02-20 18:11 UTC (permalink / raw)
To: Martin Bosner; +Cc: linux-raid
In-Reply-To: <231629B1-0888-4B3D-BD81-F641937AC045@bosner.de>
Hi Martin,
On 02/20/2017 12:48 PM, Martin Bosner wrote:
> Hi Phil,
>
>> { Note: convention on kernel.org is reply-to-all, bottom post or
>> interleave, and trim unneeded material. }
> Sorry, hope this one is better.
>
>
>> for x in /dev/sd[a-z] /dev/sda[a-j] ; do mdadm -E ${x}1 ; done
>
> See below for full output.
Not good.
>> Eewwww! You have desktop drives. Which means your array has
>> blown up due to timeout mismatch. You have some reading to do.[1]
Of the 36 original disks, you have 34. You have one incomplete
rebuild, meaning it is still technically a spare. One of the still
active 34 is also showing pending relocations, meaning that disk will
not be able to supply all sectors to complete any recovery.
{ /dev/sdah, serial # S1F0FPYR }
If you have any access to the two "dead" drives, there might be
a slight chance. Since they were likely kicked out due to timeout
mismatch, not a complete failure, this could be possible.
Otherwise, you are utterly screwed. Sorry.
> I will be using so called “NAS” or “enterprise disks in the next
> cluster … especially these seagate disks were a bad decision.
Yes.
Phil
^ permalink raw reply
* Re: assistance recovering failed raid6 array
From: Roman Mamedov @ 2017-02-20 17:50 UTC (permalink / raw)
To: Martin Bosner; +Cc: Phil Turmel, linux-raid
In-Reply-To: <5cc1566c-1b4c-c663-56a1-2040b93b46d7@turmel.org>
> On 02/20/2017 12:05 PM, Martin Bosner wrote:
> > === START OF INFORMATION SECTION ===
> > Model Family: Seagate Barracuda 7200.14 (AF)
> > Device Model: ST3000DM001-1CH166
So you have the most terrible hard drive possible [1][2](they WILL ALL fail),
ran in about the most terrible RAID setup possible (only a single RAID5 would
have been worse). Now you realize why the latter was a bad idea: with such a
great number of disks, should have picked a 3x12-member RAID6 or similar. Just
let this be a lesson in component choice and risk assessment, restore from
your backups and move on.
[1] https://en.wikipedia.org/wiki/ST3000DM001
[2] https://www.quora.com/What-are-some-of-the-worst-hard-drive-designs-ever
--
With respect,
Roman
^ permalink raw reply
* Re: assistance recovering failed raid6 array
From: Martin Bosner @ 2017-02-20 17:48 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
In-Reply-To: <5cc1566c-1b4c-c663-56a1-2040b93b46d7@turmel.org>
Hi Phil,
> { Note: convention on kernel.org is reply-to-all, bottom post or
> interleave, and trim unneeded material. }
Sorry, hope this one is better.
> for x in /dev/sd[a-z] /dev/sda[a-j] ; do mdadm -E ${x}1 ; done
See below for full output.
> Eewwww! You have desktop drives. Which means your array has
> blown up due to timeout mismatch. You have some reading to
> do.[1]
I will be using so called “NAS” or “enterprise disks in the next cluster … especially these seagate disks were a bad decision.
> http://marc.info/?l=linux-raid&m=142487508806844&w=3
> http://marc.info/?l=linux-raid&m=144535576302583&w=2
I will read through it.
Martin
#########
for x in /dev/sd[a-z] /dev/sda[a-j] ; do mdadm -E ${x}1 ; done
/dev/sda1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : f90e9c41:5aa3c3b2:d715781b:1abbb439
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : b0b57ef2 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : bf3081d1:5afead3a:839df956:098403b9
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 83554c55 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 6e592519:9dafe1fd:6616a6b4:6de9ab52
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 8582048f - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 465ca8a2:45bd05a3:94b4d0cc:0bf9ee5d
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : d796c2dc - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 79f7fb50:15f0887f:4d7adf0c:632e9743
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 88248bf0 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 5a1097b1:7da38314:8633c3f9:d3b843ab
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : d24a9be4 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 5
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 31076c1e:97f2be2f:1ae76487:48667fc7
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 43842df - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 6
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
mdadm: No md superblock detected on /dev/sdh1.
/dev/sdi1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 34ecbf22:16e15f2f:a9075b03:6f8f2de3
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 9fd16018 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 8
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdj1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 390cbe6a:bceed865:5d88091f:86c7228b
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : e1ec468f - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 9
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdk1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x9
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 79b27fa7:954302fe:4f669a20:1ddf9a15
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
Checksum : 42e0375d - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdl1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x9
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : c7215fad:1c6ecbbf:2c2e0feb:aabbb208
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
Checksum : c8157597 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 11
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdm1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x8b
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Recovery Offset : 296160 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 1366f174:d65cf0f7:b20f5b4f:c2263bbe
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
Checksum : e1a5d782 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 12
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdn1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : c9b9b4ad:74b4989c:1c0003cd:dd402919
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 97a2aaf2 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 13
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdo1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : d5a5ec07:91695b21:087ebb12:f3e6bf3f
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : e2ec701c - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 14
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdp1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x9
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 7f4f1b78:7a0ae004:664b2208:da13eed6
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
Checksum : c334b517 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 15
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdq1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : a2158ac6:82ac5c70:31d9b49b:66077e36
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 70429e7a - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 16
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdr1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : af23e4bb:04bae1e3:6b79aa24:ebf1ccb3
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : df6644c9 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 17
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sds1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 0c1675d5:0e2e9b88:3ab7545c:f5f2cd99
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : bb5bea0a - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 18
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdt1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : f2920d03:7784140f:0108652e:ef335243
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : eb024f19 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 19
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdu1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 00b49bc7:54fa2eb2:38be88f4:177812ad
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 828ee067 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 20
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdv1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : a04149c8:d99b98d4:6ad76d27:b6930004
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 2f79445d - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 21
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdw1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 3609155b:7272da25:a6ccf32b:707e27e1
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : f533c282 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 22
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdx1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x9
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : c5462d71:6f69ab33:a6ddeb6a:28210c6e
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
Checksum : e4f9aae3 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 23
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdy1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 44823783:34d0edf7:03d4ae6f:522befc6
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 18ec4da1 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 24
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdz1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 063185d5:e0d3dbce:ab39c4fa:ae855b0c
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 12a9c008 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 25
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdaa1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x9
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : da6a846c:563d61bf:ee1e786a:c9e5280b
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
Checksum : 8afa8c7 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 26
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdab1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 0ea3c624:8e2e59ab:ed195520:23917089
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : e10e7875 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 27
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdac1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : e6c66c40:e0c49813:e87dc481:7b57ef8b
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : c8e25cf3 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 28
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdad1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x9
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : af24f1fb:a55ddb8b:3ecd2971:6f9dc92d
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
Checksum : 8de8e8dd - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 29
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdae1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x9
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : d4e87abe:00003a9d:e950d72c:f5b93939
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
Checksum : 28eeef8f - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 30
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdaf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x9
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 2ea54072:67978e8b:c6b99895:76f4157f
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
Checksum : 79a6e6b2 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 31
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdag1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 52ee4e92:9a60fdd5:b3393a70:f0e8e53f
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 7f956d5e - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 32
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdah1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x9
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 80818d5a:85638be1:f9974e08:081404d2
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
Checksum : 7d948cde - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 33
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdai1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 6f014106:2eea458c:4a6dcf1a:a2666db0
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : c4ecbb59 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 34
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdaj1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage)
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 0fdc0ec2:42d30dcb:f00d2c63:26171824
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 7b89d039 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 35
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
^ permalink raw reply
* Re: assistance recovering failed raid6 array
From: Phil Turmel @ 2017-02-20 17:36 UTC (permalink / raw)
To: Martin Bosner; +Cc: linux-raid
In-Reply-To: <E18A7C79-09E0-4361-9F89-68AE1E6FCBF6@bosner.de>
Hi Martin,
{ Note: convention on kernel.org is reply-to-all, bottom post or
interleave, and trim unneeded material. }
On 02/20/2017 12:05 PM, Martin Bosner wrote:
> for x in /dev/sd[a-z] /dev/sda[a-j] ; do echo mdadm -E ${x}1 ; smartctl -iA -l scterc $x ; done
Darn. I didn't mean to leave 'echo' there. Please run this
part over again:
for x in /dev/sd[a-z] /dev/sda[a-j] ; do mdadm -E ${x}1 ; done
> smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-62-generic] (local build)
> Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
>
> === START OF INFORMATION SECTION ===
> Model Family: Seagate Barracuda 7200.14 (AF)
> Device Model: ST3000DM001-1CH166
> Serial Number: Z1F4RN82
> LU WWN Device Id: 5 000c50 0662080a6
> Firmware Version: CC27
> User Capacity: 3,000,592,982,016 bytes [3.00 TB]
> Sector Sizes: 512 bytes logical, 4096 bytes physical
> Rotation Rate: 7200 rpm
> Form Factor: 3.5 inches
> Device is: In smartctl database [for details use: -P show]
> ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
> SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
> Local Time is: Mon Feb 20 17:04:09 2017 CET
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
>
> === START OF READ SMART DATA SECTION ===
> SMART Attributes Data Structure revision number: 10
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
> 1 Raw_Read_Error_Rate 0x000f 111 099 006 Pre-fail Always - 32613624
> 3 Spin_Up_Time 0x0003 098 098 000 Pre-fail Always - 0
> 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 4
> 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
> 7 Seek_Error_Rate 0x000f 067 060 030 Pre-fail Always - 5379264
> 9 Power_On_Hours 0x0032 096 096 000 Old_age Always - 4222
> 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
> 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 4
> 183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
> 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
> 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
> 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 0 0
> 189 High_Fly_Writes 0x003a 099 099 000 Old_age Always - 1
> 190 Airflow_Temperature_Cel 0x0022 072 066 045 Old_age Always - 28 (Min/Max 26/28)
> 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
> 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 4
> 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 4
> 194 Temperature_Celsius 0x0022 028 040 000 Old_age Always - 28 (0 25 0 0 0)
> 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
> 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
> 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
> 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 4222h+01m+40.011s
> 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 6286190299
> 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 103173878942
>
> SCT Error Recovery Control command not supported
^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^
Eewwww! You have desktop drives. Which means your array has
blown up due to timeout mismatch. You have some reading to
do.[1]
Phil
[1] Recommendations from the archives (whole threads):
http://marc.info/?l=linux-raid&m=139050322510249&w=2
http://marc.info/?l=linux-raid&m=135863964624202&w=2
http://marc.info/?l=linux-raid&m=135811522817345&w=1
http://marc.info/?l=linux-raid&m=133761065622164&w=2
http://marc.info/?l=linux-raid&m=132477199207506
http://marc.info/?l=linux-raid&m=133665797115876&w=2
http://marc.info/?l=linux-raid&m=142487508806844&w=3
http://marc.info/?l=linux-raid&m=144535576302583&w=2
^ permalink raw reply
* Device size for linux raid5 journal?
From: Christian Samsel @ 2017-02-20 16:50 UTC (permalink / raw)
To: linux-raid
Hello raid team,
First of all, thanks for your work.
So i recently read about Linux raid5/raid6 write-cache and journaling and thought about
giving it a try. I'm mainly interested in the additional safety provided by the journal but might
want be future proof to use the write cache as well.
I read how to create an array using a journal but i havent found the slightest indication of how
large the respective device/partition should be. I went through the lwn article [1], the slides [2] of the
respective engineers at facebook and a few commit messages.
So my question is, let's assume i have a 6TB raid5 array (3x3TB), what would a good journal device size be?
I'd probably went with 4GB, as this is kinda the upper bound of what hardware raid controller offer.
Christian
[1] https://lwn.net/Articles/665299/
[2] http://events.linuxfoundation.org/sites/events/files/slides/2016_vault_write_journal_cache_v2.pdf
^ permalink raw reply
* Re: assistance recovering failed raid6 array
From: Phil Turmel @ 2017-02-20 15:39 UTC (permalink / raw)
To: Martin Bosner, linux-raid
In-Reply-To: <58AA4B1E.1030809@bosner.de>
On 02/19/2017 08:49 PM, Martin Bosner wrote:
> I am running a software raid6 with 36 x 3TB disks (sda to sdaj). All
> disks have one partition (gpt, 100%, primary, raid on) and i am using
> btrfs on top of the raid.
>
> Last week one of the disks failed and was unrecoverable. I replaced the
> disk (sdk) with a new one and the resync process started.
> At around 80% recovery two further disks failed and the recovery process
> was stopped. That failed disks are sdm and sdh.
>
> All other disks seem to be fine and I was about the use the "mdadm
> --create" command when i remembered the lines
> "You have been warned! It's better to send an email to the linux-raid
> mailing list with detailed information"
>
> So here i am for an advice how to continue.
More information, please. Paste inline, untrimmed, in your reply with
line wrapping disabled. Plain text only. Use multiple mails if needed.
List limit is ~100k IIRC.
# dmesg
# for x in /dev/sd[a-z] /dev/sda[a-j] ; do echo mdadm -E ${x}1 ; smartctl -iA -l scterc $x ; done
Phil
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox