* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Alireza Haghdoost @ 2015-04-01 20:04 UTC (permalink / raw)
To: Wols Lists
Cc: Piergiorgio Sartor, Dan Williams, Shaohua Li, Neil Brown,
linux-raid, Song Liu, Kernel-team
In-Reply-To: <551C4DAA.4010701@youngman.org.uk>
On Wed, Apr 1, 2015 at 2:57 PM, Wols Lists <antlists@youngman.org.uk> wrote:
> On 01/04/15 19:46, Alireza Haghdoost wrote:
>>> Now, how can be assured, in that case, that the "cache"
>>> > device is safe after the power is restored?
>> You do sync write-ahead logging on the Flash cache. If it return
>> successful, you do fire the writes to the RAID. If system crash/fails
>> during the RAID writes (Write-hole), you just recover data by scanning
>> write-ahead log in the flash cache and replay the logs into the RAID
>> drives.
>>
> Just to throw something nasty into the mix, I'm not sure whether it's
> SSDs or SD-cards, but there certainly *was* a spate of corrupted
> *controllers*.
>
> In other words, a power failure would RELIABLY TRASH the device, if it
> happened at the wrong moment. Hopefully that's been fixed ...
>
That is certainly true. As Dan mentioned, the cache device it-self
should be safe against power failure. I agree this is not the case for
all SSD cards in the market but might be the case for Facebook. I hate
to say this but It seems these efforts are useful dependent to what
kind of hardware is deployed for cache device.
--Alireza
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Wols Lists @ 2015-04-01 19:57 UTC (permalink / raw)
To: Alireza Haghdoost, Piergiorgio Sartor
Cc: Dan Williams, Shaohua Li, Neil Brown, linux-raid, Song Liu,
Kernel-team
In-Reply-To: <CAB-428nVCN=5au0jONamXORNZ656UZ1Ecgrk-paArruX9tpHJQ@mail.gmail.com>
On 01/04/15 19:46, Alireza Haghdoost wrote:
>> Now, how can be assured, in that case, that the "cache"
>> > device is safe after the power is restored?
> You do sync write-ahead logging on the Flash cache. If it return
> successful, you do fire the writes to the RAID. If system crash/fails
> during the RAID writes (Write-hole), you just recover data by scanning
> write-ahead log in the flash cache and replay the logs into the RAID
> drives.
>
Just to throw something nasty into the mix, I'm not sure whether it's
SSDs or SD-cards, but there certainly *was* a spate of corrupted
*controllers*.
In other words, a power failure would RELIABLY TRASH the device, if it
happened at the wrong moment. Hopefully that's been fixed ...
Cheers.
Wol
^ permalink raw reply
* Re: Two system lockups, Magic SysKeys works, RAID6 resyncs on reboot
From: Mark Knecht @ 2015-04-01 19:07 UTC (permalink / raw)
To: Roman Mamedov; +Cc: Linux-RAID
In-Reply-To: <CAK2H+ec5Lo0=QaEXa=RrbXCzG5m6nRc7BynSPYEqvCc5Lb4z3g@mail.gmail.com>
On Tue, Mar 31, 2015 at 3:08 PM, Mark Knecht <markknecht@gmail.com> wrote:
> On Tue, Mar 31, 2015 at 1:30 PM, Roman Mamedov <rm@romanrm.net> wrote:
>> On Tue, 31 Mar 2015 08:00:27 -0700
>> Mark Knecht <markknecht@gmail.com> wrote:
>>
>>> md3 : active raid6 sdc3[9] sdg3[5] sdf3[6] sde3[7] sdd3[8]
>>> 1452264480 blocks super 1.2 level 6, 16k chunk, algorithm 2 [5/5] [UUUUU]
>>> [====>................] resync = 22.8% (110608304/484088160)
>>> finish=78.5min speed=79206K/sec
>>
>> Looks like you are running this without a write intent bitmap, why? It would
>> at least save you from a full resync each time.
>>
>> --
>> With respect,
>> Roman
>
> Honestly, I don't know. The machine was the first I ever built using
> RAID for the system drive and I probably didn't find out about it, or
> was just trying to keep it simple? Not at all sure.
>
> I'll check it out.
>
> Thanks,
> Mark
At this point, having put everything back together, I cannot find any
specific clues about what caused the problem. Possibly the system
might be showing its age. Maybe it's a corner condition in the 3.18
series kernel vs the 3.14 I was running before. No idea. The mdadm
RAID seems to have done it's job wonderfully. At least at the Linux
level there's no evidence at this time that anything was lost. Within
my Windows VMs that were running on the mdadm RAID6 at the time of the
Magic Sys Key reboot there seem to be minimal issues. One Windows
program (Schwab's StreetSmart Edge) lost some of my trading
environment configuration and I needed to recall only that file off
the web server to get the platform configured correctly again.
TradeStation was running in a second VM and showed no issues. Both
trading platforms were live at the time of the lockup so I came out of
this really well, and as I have good backups nothing would have been
lost anyway.
As per Roman's question/suggestion I've added the write intent bitmaps
to all three RAIDs. Nominally only md3 is mounted most of the time
anyway, but it's there to help if this problem continues.
If anyone sees anything else in the data I posted yesterday let me
know but everything mdadm specific seems good at this point.
Cheers,
Mark
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Alireza Haghdoost @ 2015-04-01 18:46 UTC (permalink / raw)
To: Piergiorgio Sartor
Cc: Dan Williams, Shaohua Li, Neil Brown, linux-raid, Song Liu,
Kernel-team
In-Reply-To: <20150401183630.GA3103@lazy.lzy>
On Wed, Apr 1, 2015 at 1:36 PM, Piergiorgio Sartor
<piergiorgio.sartor@nexgo.de> wrote:
> On Tue, Mar 31, 2015 at 08:47:04PM -0700, Dan Williams wrote:
>> On Mon, Mar 30, 2015 at 3:25 PM, Shaohua Li <shli@fb.com> wrote:
>> > This is my attempt to fix raid5/6 write hole issue, it's not for merge
>> > yet, I post it out for comments. Any comments and suggestions are
>> > welcome!
>> >
>> > Thanks,
>> > Shaohua
>> >
>> > We expect a completed raid5/6 stack with reliability and high
>> > performance. Currently raid5/6 has 2 issues:
>> >
>> > 1. read-modify-write for small size IO. To fix this issue, a cache layer
>> > above raid5/6 can be used to aggregate write to full stripe write.
>> > 2. write hole issue. A write log below raid5/6 can fix the issue.
>> >
>> > We plan to use a SSD to fix the two issues. Here we just fix the write
>> > hole issue.
>> >
>> > 1. We don't try to fix the issues together. A cache layer will do write
>> > acceleration. A log layer will fix write hole. The seperation will
>> > simplify things a lot.
>> >
>> > 2. Current assumption is flashcache/bcache will be used as the cache
>> > layer. If they don't work well, we can fix them or add a simple cache
>> > layer for raid write aggregation later. We also assume cache layer will
>> > absorb write, so log doesn't worry about write latency.
>>
>> It seems neither bcache nor dm-cache are tackling the write-buffering
>> problem head on... they still seem to be concerned with some amount of
>> read caching which I can see as useful for file servers and
>> workstations, but not necessarily scale out storage.
>>
>> I'll try to set aside time to take a look at the patch this week.
>
> There is one thing I do not really get.
>
> The target is to avoid the "write hole", which happens,
> for example, when there is a sudden power failure.
>
> Now, how can be assured, in that case, that the "cache"
> device is safe after the power is restored?
You do sync write-ahead logging on the Flash cache. If it return
successful, you do fire the writes to the RAID. If system crash/fails
during the RAID writes (Write-hole), you just recover data by scanning
write-ahead log in the flash cache and replay the logs into the RAID
drives.
> Doesn't this solution just shifts the problem from
> the array to a different device (SSD, for example)?
I don't see such a shift. Enterprise hardware RAIDs also used similar
technique to fix the write-hole issue.
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Dan Williams @ 2015-04-01 18:46 UTC (permalink / raw)
To: Piergiorgio Sartor
Cc: Shaohua Li, Neil Brown, linux-raid, Song Liu, Kernel-team
In-Reply-To: <20150401183630.GA3103@lazy.lzy>
On Wed, Apr 1, 2015 at 11:36 AM, Piergiorgio Sartor
<piergiorgio.sartor@nexgo.de> wrote:
> On Tue, Mar 31, 2015 at 08:47:04PM -0700, Dan Williams wrote:
>> On Mon, Mar 30, 2015 at 3:25 PM, Shaohua Li <shli@fb.com> wrote:
>> > This is my attempt to fix raid5/6 write hole issue, it's not for merge
>> > yet, I post it out for comments. Any comments and suggestions are
>> > welcome!
>> >
>> > Thanks,
>> > Shaohua
>> >
>> > We expect a completed raid5/6 stack with reliability and high
>> > performance. Currently raid5/6 has 2 issues:
>> >
>> > 1. read-modify-write for small size IO. To fix this issue, a cache layer
>> > above raid5/6 can be used to aggregate write to full stripe write.
>> > 2. write hole issue. A write log below raid5/6 can fix the issue.
>> >
>> > We plan to use a SSD to fix the two issues. Here we just fix the write
>> > hole issue.
>> >
>> > 1. We don't try to fix the issues together. A cache layer will do write
>> > acceleration. A log layer will fix write hole. The seperation will
>> > simplify things a lot.
>> >
>> > 2. Current assumption is flashcache/bcache will be used as the cache
>> > layer. If they don't work well, we can fix them or add a simple cache
>> > layer for raid write aggregation later. We also assume cache layer will
>> > absorb write, so log doesn't worry about write latency.
>>
>> It seems neither bcache nor dm-cache are tackling the write-buffering
>> problem head on... they still seem to be concerned with some amount of
>> read caching which I can see as useful for file servers and
>> workstations, but not necessarily scale out storage.
>>
>> I'll try to set aside time to take a look at the patch this week.
>
> There is one thing I do not really get.
>
> The target is to avoid the "write hole", which happens,
> for example, when there is a sudden power failure.
>
> Now, how can be assured, in that case, that the "cache"
> device is safe after the power is restored?
If you lose the cache the data-loss damage is greater, but this has
always been the case with hardware-raid adapters.
> Doesn't this solution just shifts the problem from
> the array to a different device (SSD, for example)?
>
> Speaking of SSD, these are quite "power failure"
> sensitive, it seems...
Simple, if a cache-device is not itself power-failure safe then it
should not be used for power-failure protection.
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Piergiorgio Sartor @ 2015-04-01 18:36 UTC (permalink / raw)
To: Dan Williams; +Cc: Shaohua Li, Neil Brown, linux-raid, Song Liu, Kernel-team
In-Reply-To: <CAPcyv4jqb+OphP-1sy0dqMYG-uNK-CwtNDmOM4YeQU=MXH72Hw@mail.gmail.com>
On Tue, Mar 31, 2015 at 08:47:04PM -0700, Dan Williams wrote:
> On Mon, Mar 30, 2015 at 3:25 PM, Shaohua Li <shli@fb.com> wrote:
> > This is my attempt to fix raid5/6 write hole issue, it's not for merge
> > yet, I post it out for comments. Any comments and suggestions are
> > welcome!
> >
> > Thanks,
> > Shaohua
> >
> > We expect a completed raid5/6 stack with reliability and high
> > performance. Currently raid5/6 has 2 issues:
> >
> > 1. read-modify-write for small size IO. To fix this issue, a cache layer
> > above raid5/6 can be used to aggregate write to full stripe write.
> > 2. write hole issue. A write log below raid5/6 can fix the issue.
> >
> > We plan to use a SSD to fix the two issues. Here we just fix the write
> > hole issue.
> >
> > 1. We don't try to fix the issues together. A cache layer will do write
> > acceleration. A log layer will fix write hole. The seperation will
> > simplify things a lot.
> >
> > 2. Current assumption is flashcache/bcache will be used as the cache
> > layer. If they don't work well, we can fix them or add a simple cache
> > layer for raid write aggregation later. We also assume cache layer will
> > absorb write, so log doesn't worry about write latency.
>
> It seems neither bcache nor dm-cache are tackling the write-buffering
> problem head on... they still seem to be concerned with some amount of
> read caching which I can see as useful for file servers and
> workstations, but not necessarily scale out storage.
>
> I'll try to set aside time to take a look at the patch this week.
There is one thing I do not really get.
The target is to avoid the "write hole", which happens,
for example, when there is a sudden power failure.
Now, how can be assured, in that case, that the "cache"
device is safe after the power is restored?
Doesn't this solution just shifts the problem from
the array to a different device (SSD, for example)?
Speaking of SSD, these are quite "power failure"
sensitive, it seems...
Thanks,
bye,
pg
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
piergiorgio
^ permalink raw reply
* write barrier in RAID5
From: Eric Mei @ 2015-04-01 18:24 UTC (permalink / raw)
To: linux-raid
I can see how FUA is handled, but REQ_FLUSH seems problematic. My
understanding of REQ_FLUSH semantics: all requests before the flush
request must *complete* before the flush request complete. For IO remap
layer like MD, we need to guarantee all previous requests are
*submitted* to component devices before submitting the flush request. Is
that correct?
If so why I don't see the code which make sure the pending write
requests in stripe cache are processed & submitted before
submit_flushes() get called? Thanks
Eric
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Shaohua Li @ 2015-04-01 17:14 UTC (permalink / raw)
To: NeilBrown; +Cc: Dan Williams, linux-raid, Song Liu, Kernel-team
In-Reply-To: <20150401170256.5efebaae@notabene.brown>
On Wed, Apr 01, 2015 at 05:02:56PM +1100, NeilBrown wrote:
> On Tue, 31 Mar 2015 22:53:21 -0700 Shaohua Li <shli@fb.com> wrote:
>
> > On Tue, Mar 31, 2015 at 08:47:04PM -0700, Dan Williams wrote:
> > > On Mon, Mar 30, 2015 at 3:25 PM, Shaohua Li <shli@fb.com> wrote:
> > > > This is my attempt to fix raid5/6 write hole issue, it's not for merge
> > > > yet, I post it out for comments. Any comments and suggestions are
> > > > welcome!
> > > >
> > > > Thanks,
> > > > Shaohua
> > > >
> > > > We expect a completed raid5/6 stack with reliability and high
> > > > performance. Currently raid5/6 has 2 issues:
> > > >
> > > > 1. read-modify-write for small size IO. To fix this issue, a cache layer
> > > > above raid5/6 can be used to aggregate write to full stripe write.
> > > > 2. write hole issue. A write log below raid5/6 can fix the issue.
> > > >
> > > > We plan to use a SSD to fix the two issues. Here we just fix the write
> > > > hole issue.
> > > >
> > > > 1. We don't try to fix the issues together. A cache layer will do write
> > > > acceleration. A log layer will fix write hole. The seperation will
> > > > simplify things a lot.
> > > >
> > > > 2. Current assumption is flashcache/bcache will be used as the cache
> > > > layer. If they don't work well, we can fix them or add a simple cache
> > > > layer for raid write aggregation later. We also assume cache layer will
> > > > absorb write, so log doesn't worry about write latency.
> > >
> > > It seems neither bcache nor dm-cache are tackling the write-buffering
> > > problem head on... they still seem to be concerned with some amount of
> > > read caching which I can see as useful for file servers and
> > > workstations, but not necessarily scale out storage.
> > >
> > > I'll try to set aside time to take a look at the patch this week.
> >
> > Thanks! The cache layer is definitely what I'll focus on next. bcache
> > supports writeback, I guess we can add an option to skip read data from
> > backing disks for read caching if it's possible. Another option is
> > writting a simple caching just for raid 5/6 write aggregation. We can
> > append all data to a log, and maintain an index in memory. At raid
> > shutdown, we can flush all data to raid disks, the index doesn't need
> > presistent in disk, which makes the caching fairly simple.
>
> Surely if the index doesn't need to persist in disk, then the data doesn't
> either, as without the index you cannot find the data...
I mean not just pure data. We can store tuple (disk offset, length,
data) to disk. index will be used to speed up search. If there is a
crash, we can rebuild the index using the tuple.
Thanks,
Shaohua
^ permalink raw reply
* Re: [PATCH] Fix bitmap offset calculations
From: Goldwyn Rodrigues @ 2015-04-01 14:14 UTC (permalink / raw)
To: Alireza Haghdoost, NeilBrown; +Cc: Linux RAID
In-Reply-To: <CAB-428mbVFcEmT_OryyGxEAhr56Uy0CODrEVEoPUPk34C45_Uw@mail.gmail.com>
Hi Alireza,
On 04/01/2015 08:58 AM, Alireza Haghdoost wrote:
> Does it means if some one use write-intent bitmap without this patch,
> he may end-up with some unsync stripes after system crash or power
> failure and RAID resynchronization ? It seems the bitmaps does not
> record correct address of unsynced blocks due to this bug.
>
> Would you please verify this.
This is for clustered md effort only (which is in Neil's md/for-next
tree). The regular md is unaffected.
In a clustered environment, different nodes use different bitmaps. While
it worked for bitmaps smaller than a page (which is again a
co-incidence), it did not work well for bitmaps which spanned multiple
pages. Each node in the cluster has different start offsets, and the
earlier calculation was incorrect because of conversion from bits to
bytes was inverted.
If you were able to assemble on different nodes, you should be fine with
respect to synchronization of unsynced blocks after a failure.
HTH,
>
> On Tue, Mar 24, 2015 at 9:15 PM, NeilBrown <neilb@suse.de> wrote:
>> On Tue, 24 Mar 2015 11:29:05 -0500 Goldwyn Rodrigues <rgoldwyn@suse.de> wrote:
>>
>>> The calculations of bitmap offset is incorrect with respect to bits to bytes
>>> conversion.
>>>
>>> Also, remove an irrelevant duplicate message.
>>>
>>> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
>>> ---
>>> diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
>>> index ac79fef..e98db04 100644
>>> --- a/drivers/md/bitmap.c
>>> +++ b/drivers/md/bitmap.c
>>> @@ -575,7 +575,9 @@ re_read:
>>>
>>> sector_div(bm_blocks,
>>> bitmap->mddev->bitmap_info.chunksize >> 9);
>>> - bm_blocks = bm_blocks << 3;
>>> + /* bits to bytes */
>>> + bm_blocks = ((bm_blocks+7) >> 3) + sizeof(bitmap_super_t);
>>> + /* to 4k blocks */
>>> bm_blocks = DIV_ROUND_UP_SECTOR_T(bm_blocks, 4096);
>>> bitmap->mddev->bitmap_info.offset += bitmap->cluster_slot * (bm_blocks << 3);
>>> pr_info("%s:%d bm slot: %d offset: %llu\n", __func__, __LINE__,
>>> @@ -672,9 +674,6 @@ out:
>>> goto out_no_sb;
>>> }
>>> bitmap->cluster_slot = md_cluster_ops->slot_number(bitmap->mddev);
>>> - pr_info("%s:%d bm slot: %d offset: %llu\n", __func__, __LINE__,
>>> - bitmap->cluster_slot,
>>> - (unsigned long long)bitmap->mddev->bitmap_info.offset);
>>> goto re_read;
>>> }
>>>
>>
>> Applied, thanks.
>>
>> NeilBrown
--
Goldwyn
^ permalink raw reply
* Re: [PATCH] Fix bitmap offset calculations
From: Alireza Haghdoost @ 2015-04-01 13:58 UTC (permalink / raw)
To: NeilBrown; +Cc: Goldwyn Rodrigues, Linux RAID
In-Reply-To: <20150325131502.1a947967@notabene.brown>
Does it means if some one use write-intent bitmap without this patch,
he may end-up with some unsync stripes after system crash or power
failure and RAID resynchronization ? It seems the bitmaps does not
record correct address of unsynced blocks due to this bug.
Would you please verify this.
On Tue, Mar 24, 2015 at 9:15 PM, NeilBrown <neilb@suse.de> wrote:
> On Tue, 24 Mar 2015 11:29:05 -0500 Goldwyn Rodrigues <rgoldwyn@suse.de> wrote:
>
>> The calculations of bitmap offset is incorrect with respect to bits to bytes
>> conversion.
>>
>> Also, remove an irrelevant duplicate message.
>>
>> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
>> ---
>> diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
>> index ac79fef..e98db04 100644
>> --- a/drivers/md/bitmap.c
>> +++ b/drivers/md/bitmap.c
>> @@ -575,7 +575,9 @@ re_read:
>>
>> sector_div(bm_blocks,
>> bitmap->mddev->bitmap_info.chunksize >> 9);
>> - bm_blocks = bm_blocks << 3;
>> + /* bits to bytes */
>> + bm_blocks = ((bm_blocks+7) >> 3) + sizeof(bitmap_super_t);
>> + /* to 4k blocks */
>> bm_blocks = DIV_ROUND_UP_SECTOR_T(bm_blocks, 4096);
>> bitmap->mddev->bitmap_info.offset += bitmap->cluster_slot * (bm_blocks << 3);
>> pr_info("%s:%d bm slot: %d offset: %llu\n", __func__, __LINE__,
>> @@ -672,9 +674,6 @@ out:
>> goto out_no_sb;
>> }
>> bitmap->cluster_slot = md_cluster_ops->slot_number(bitmap->mddev);
>> - pr_info("%s:%d bm slot: %d offset: %llu\n", __func__, __LINE__,
>> - bitmap->cluster_slot,
>> - (unsigned long long)bitmap->mddev->bitmap_info.offset);
>> goto re_read;
>> }
>>
>
> Applied, thanks.
>
> NeilBrown
^ permalink raw reply
* Re: [neilb@suse.de: Re: MD request counter broken on 4.0-rc5 RAID1]
From: NeilBrown @ 2015-04-01 11:07 UTC (permalink / raw)
To: Gu Zheng; +Cc: Simon Kirby, linux-raid
In-Reply-To: <551BA0C3.7090004@cn.fujitsu.com>
[-- Attachment #1: Type: text/plain, Size: 3298 bytes --]
On Wed, 1 Apr 2015 15:39:47 +0800 Gu Zheng <guz.fnst@cn.fujitsu.com> wrote:
> Hi Neil,
> One question about md io stats account:
> I find that we just accounting disk_stats.sectors[rw] and disk_stats.ios[rw] at
> md_make_request, but I did not find accounting disk_stats.ticks[rw], do not we
> need it? or I missed something?
No, I don't think you are missing anything. md doesn't track 'ticks' - and
no-one has ever complained.
NeilBrown
>
> Regards,
> Gu
>
> On 03/26/2015 01:33 AM, Simon Kirby wrote:
>
> > Hello!
> >
> > Please note the following.
> >
> > Thanks,
> >
> > Simon-
> >
> > ----- Forwarded message from NeilBrown <neilb@suse.de> -----
> >
> > Date: Wed, 25 Mar 2015 11:46:18 +1100
> > From: NeilBrown <neilb@suse.de>
> > To: Simon Kirby <sim@hostway.ca>
> > Cc: linux-raid@vger.kernel.org
> > Subject: Re: MD request counter broken on 4.0-rc5 RAID1
> > X-Mailer: Claws Mail 3.10.1-162-g4d0ed6 (GTK+ 2.24.25; x86_64-suse-linux-gnu)
> >
> > On Tue, 24 Mar 2015 17:04:03 -0700 Simon Kirby <sim@hostway.ca> wrote:
> >
> >> Hello :)
> >>
> >> I'm seeing "iostat -x -k 1" print this after a RAID1 rebuild on 4.0-rc5.
> >> It's not abnormal other than it's 3-disk, with one being SSD (sdc) and
> >> the other two being write-mostly:
> >>
> >> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
> >> sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >> sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >> md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 345.00 0.00 0.00 0.00 0.00 100.00
> >> md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 58779.00 0.00 0.00 0.00 0.00 100.00
> >> md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12.00 0.00 0.00 0.00 0.00 100.00
> >>
> >> This was working on 3.18. Shall I bisect, or is it obvious?
> >
> > Almost certainly 18c0b223cf9901727ef3b02da6711ac930b4e5d4 in 3.19.
> >
> > It adds 'part_inc_in_flight' and never decrements it.
> >
> > Complain to the author.
> >
> > Thanks,
> > NeilBrown
> >
> >>
> >> It seems to just be leaking as the counters were going up gradually while
> >> it was rebuilding.
> >>
> >> Personalities : [linear] [raid0] [raid1] [raid10]
> >> md1 : active raid1 sdb2[1](W) sda2[0](W) sdc2[2]
> >> 530048 blocks [3/3] [UUU]
> >>
> >> md2 : active raid1 sdb3[0](W) sdc3[2] sda3[1](W)
> >> 153613440 blocks [3/3] [UUU]
> >>
> >> md0 : active raid1 sdb1[0](W) sdc1[2] sda1[1](W)
> >> 2104384 blocks [3/3] [UUU]
> >>
> >> unused devices: <none>
> >>
> >> Simo-
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
> > ----- End forwarded message -----
> > .
> >
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Please let me know if you need to print color box, display box and labels
From: Jinghao Printing - CHINA @ 2015-04-01 8:48 UTC (permalink / raw)
Hi, this is David Wu from Shanghai, China.
Please let me know if you need color box, display box, corrugated box,
label, hang tag etc.
I will send you the website.
Best regards,
David Wu
^ permalink raw reply
* Re: [neilb@suse.de: Re: MD request counter broken on 4.0-rc5 RAID1]
From: Gu Zheng @ 2015-04-01 7:39 UTC (permalink / raw)
To: neilb; +Cc: Simon Kirby, linux-raid
In-Reply-To: <20150325173346.GF8519@hostway.ca>
Hi Neil,
One question about md io stats account:
I find that we just accounting disk_stats.sectors[rw] and disk_stats.ios[rw] at
md_make_request, but I did not find accounting disk_stats.ticks[rw], do not we
need it? or I missed something?
Regards,
Gu
On 03/26/2015 01:33 AM, Simon Kirby wrote:
> Hello!
>
> Please note the following.
>
> Thanks,
>
> Simon-
>
> ----- Forwarded message from NeilBrown <neilb@suse.de> -----
>
> Date: Wed, 25 Mar 2015 11:46:18 +1100
> From: NeilBrown <neilb@suse.de>
> To: Simon Kirby <sim@hostway.ca>
> Cc: linux-raid@vger.kernel.org
> Subject: Re: MD request counter broken on 4.0-rc5 RAID1
> X-Mailer: Claws Mail 3.10.1-162-g4d0ed6 (GTK+ 2.24.25; x86_64-suse-linux-gnu)
>
> On Tue, 24 Mar 2015 17:04:03 -0700 Simon Kirby <sim@hostway.ca> wrote:
>
>> Hello :)
>>
>> I'm seeing "iostat -x -k 1" print this after a RAID1 rebuild on 4.0-rc5.
>> It's not abnormal other than it's 3-disk, with one being SSD (sdc) and
>> the other two being write-mostly:
>>
>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
>> sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 345.00 0.00 0.00 0.00 0.00 100.00
>> md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 58779.00 0.00 0.00 0.00 0.00 100.00
>> md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12.00 0.00 0.00 0.00 0.00 100.00
>>
>> This was working on 3.18. Shall I bisect, or is it obvious?
>
> Almost certainly 18c0b223cf9901727ef3b02da6711ac930b4e5d4 in 3.19.
>
> It adds 'part_inc_in_flight' and never decrements it.
>
> Complain to the author.
>
> Thanks,
> NeilBrown
>
>>
>> It seems to just be leaking as the counters were going up gradually while
>> it was rebuilding.
>>
>> Personalities : [linear] [raid0] [raid1] [raid10]
>> md1 : active raid1 sdb2[1](W) sda2[0](W) sdc2[2]
>> 530048 blocks [3/3] [UUU]
>>
>> md2 : active raid1 sdb3[0](W) sdc3[2] sda3[1](W)
>> 153613440 blocks [3/3] [UUU]
>>
>> md0 : active raid1 sdb1[0](W) sdc1[2] sda1[1](W)
>> 2104384 blocks [3/3] [UUU]
>>
>> unused devices: <none>
>>
>> Simo-
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> ----- End forwarded message -----
> .
>
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: NeilBrown @ 2015-04-01 6:02 UTC (permalink / raw)
To: Shaohua Li; +Cc: Dan Williams, linux-raid, Song Liu, Kernel-team
In-Reply-To: <20150401055309.GA726662@devbig257.prn2.facebook.com>
[-- Attachment #1: Type: text/plain, Size: 2373 bytes --]
On Tue, 31 Mar 2015 22:53:21 -0700 Shaohua Li <shli@fb.com> wrote:
> On Tue, Mar 31, 2015 at 08:47:04PM -0700, Dan Williams wrote:
> > On Mon, Mar 30, 2015 at 3:25 PM, Shaohua Li <shli@fb.com> wrote:
> > > This is my attempt to fix raid5/6 write hole issue, it's not for merge
> > > yet, I post it out for comments. Any comments and suggestions are
> > > welcome!
> > >
> > > Thanks,
> > > Shaohua
> > >
> > > We expect a completed raid5/6 stack with reliability and high
> > > performance. Currently raid5/6 has 2 issues:
> > >
> > > 1. read-modify-write for small size IO. To fix this issue, a cache layer
> > > above raid5/6 can be used to aggregate write to full stripe write.
> > > 2. write hole issue. A write log below raid5/6 can fix the issue.
> > >
> > > We plan to use a SSD to fix the two issues. Here we just fix the write
> > > hole issue.
> > >
> > > 1. We don't try to fix the issues together. A cache layer will do write
> > > acceleration. A log layer will fix write hole. The seperation will
> > > simplify things a lot.
> > >
> > > 2. Current assumption is flashcache/bcache will be used as the cache
> > > layer. If they don't work well, we can fix them or add a simple cache
> > > layer for raid write aggregation later. We also assume cache layer will
> > > absorb write, so log doesn't worry about write latency.
> >
> > It seems neither bcache nor dm-cache are tackling the write-buffering
> > problem head on... they still seem to be concerned with some amount of
> > read caching which I can see as useful for file servers and
> > workstations, but not necessarily scale out storage.
> >
> > I'll try to set aside time to take a look at the patch this week.
>
> Thanks! The cache layer is definitely what I'll focus on next. bcache
> supports writeback, I guess we can add an option to skip read data from
> backing disks for read caching if it's possible. Another option is
> writting a simple caching just for raid 5/6 write aggregation. We can
> append all data to a log, and maintain an index in memory. At raid
> shutdown, we can flush all data to raid disks, the index doesn't need
> presistent in disk, which makes the caching fairly simple.
Surely if the index doesn't need to persist in disk, then the data doesn't
either, as without the index you cannot find the data...
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Shaohua Li @ 2015-04-01 5:53 UTC (permalink / raw)
To: Dan Williams; +Cc: Neil Brown, linux-raid, Song Liu, Kernel-team
In-Reply-To: <CAPcyv4jqb+OphP-1sy0dqMYG-uNK-CwtNDmOM4YeQU=MXH72Hw@mail.gmail.com>
On Tue, Mar 31, 2015 at 08:47:04PM -0700, Dan Williams wrote:
> On Mon, Mar 30, 2015 at 3:25 PM, Shaohua Li <shli@fb.com> wrote:
> > This is my attempt to fix raid5/6 write hole issue, it's not for merge
> > yet, I post it out for comments. Any comments and suggestions are
> > welcome!
> >
> > Thanks,
> > Shaohua
> >
> > We expect a completed raid5/6 stack with reliability and high
> > performance. Currently raid5/6 has 2 issues:
> >
> > 1. read-modify-write for small size IO. To fix this issue, a cache layer
> > above raid5/6 can be used to aggregate write to full stripe write.
> > 2. write hole issue. A write log below raid5/6 can fix the issue.
> >
> > We plan to use a SSD to fix the two issues. Here we just fix the write
> > hole issue.
> >
> > 1. We don't try to fix the issues together. A cache layer will do write
> > acceleration. A log layer will fix write hole. The seperation will
> > simplify things a lot.
> >
> > 2. Current assumption is flashcache/bcache will be used as the cache
> > layer. If they don't work well, we can fix them or add a simple cache
> > layer for raid write aggregation later. We also assume cache layer will
> > absorb write, so log doesn't worry about write latency.
>
> It seems neither bcache nor dm-cache are tackling the write-buffering
> problem head on... they still seem to be concerned with some amount of
> read caching which I can see as useful for file servers and
> workstations, but not necessarily scale out storage.
>
> I'll try to set aside time to take a look at the patch this week.
Thanks! The cache layer is definitely what I'll focus on next. bcache
supports writeback, I guess we can add an option to skip read data from
backing disks for read caching if it's possible. Another option is
writting a simple caching just for raid 5/6 write aggregation. We can
append all data to a log, and maintain an index in memory. At raid
shutdown, we can flush all data to raid disks, the index doesn't need
presistent in disk, which makes the caching fairly simple.
Thanks,
Shaohua
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Dan Williams @ 2015-04-01 3:47 UTC (permalink / raw)
To: Shaohua Li; +Cc: Neil Brown, linux-raid, Song Liu, Kernel-team
In-Reply-To: <20150330222459.GA575371@devbig257.prn2.facebook.com>
On Mon, Mar 30, 2015 at 3:25 PM, Shaohua Li <shli@fb.com> wrote:
> This is my attempt to fix raid5/6 write hole issue, it's not for merge
> yet, I post it out for comments. Any comments and suggestions are
> welcome!
>
> Thanks,
> Shaohua
>
> We expect a completed raid5/6 stack with reliability and high
> performance. Currently raid5/6 has 2 issues:
>
> 1. read-modify-write for small size IO. To fix this issue, a cache layer
> above raid5/6 can be used to aggregate write to full stripe write.
> 2. write hole issue. A write log below raid5/6 can fix the issue.
>
> We plan to use a SSD to fix the two issues. Here we just fix the write
> hole issue.
>
> 1. We don't try to fix the issues together. A cache layer will do write
> acceleration. A log layer will fix write hole. The seperation will
> simplify things a lot.
>
> 2. Current assumption is flashcache/bcache will be used as the cache
> layer. If they don't work well, we can fix them or add a simple cache
> layer for raid write aggregation later. We also assume cache layer will
> absorb write, so log doesn't worry about write latency.
It seems neither bcache nor dm-cache are tackling the write-buffering
problem head on... they still seem to be concerned with some amount of
read caching which I can see as useful for file servers and
workstations, but not necessarily scale out storage.
I'll try to set aside time to take a look at the patch this week.
^ permalink raw reply
* Re: /dev/md0 can't be created
From: NeilBrown @ 2015-04-01 2:53 UTC (permalink / raw)
To: Xiao Ni; +Cc: linux-raid
In-Reply-To: <160793753.4322898.1427436381735.JavaMail.zimbra@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 5310 bytes --]
On Fri, 27 Mar 2015 02:06:21 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
>
> I'm lucky, I reproduced it again and I ran the command udevadm monitor > monitor before test.
> I'll put it in as an attachment.
>
> I found there is a not whole message in the last line.
>
> Xiao
The line:
UDEV [7083.531441] add /devices/virtual/block/md0 (block)
confirms that udev saw an 'add' event for /dev/md0.
That should be enough for udev to create the device node.
If it doesn't, you need to look into udev and find out why it doesn't.
I can't really help you there - I don't find udev very easy to debug.
NeilBrown
>
> ----- Original Message -----
> > From: "Xiao Ni" <xni@redhat.com>
> > To: "NeilBrown" <neilb@suse.de>
> > Cc: linux-raid@vger.kernel.org
> > Sent: Thursday, March 26, 2015 3:36:35 PM
> > Subject: Re: /dev/md0 can't be created
> >
> > Hi Neil
> >
> > I can reproduce it now. Do you want me to run udevadm monitor before the
> > test?
> > And I checked the /var/log/messages, it show the information:
> >
> > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-2>
> > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-3>
> > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-4>
> > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-5>
> > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-6>
> > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-7>
> > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-9>
> > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-8>
> > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-7 operational as
> > raid disk 5
> > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-6 operational as
> > raid disk 4
> > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-5 operational as
> > raid disk 3
> > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-4 operational as
> > raid disk 2
> > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-3 operational as
> > raid disk 1
> > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-2 operational as
> > raid disk 0
> > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: allocated 0kB
> > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: raid level 5 active with 6
> > out of 7 devices, algorithm 2
> > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid456: discard support disabled due
> > to uncertainty.
> > Oct 20 19:59:59 ibm-z10-25 kernel: Set
> > raid456.devices_handle_discard_safely=Y to override.
> > Oct 20 19:59:59 ibm-z10-25 kernel: md0: detected capacity change from 0 to
> > 1881145344
> > Oct 20 19:59:59 ibm-z10-25 kernel: md: recovery of RAID array md0
> > Oct 20 19:59:59 ibm-z10-25 kernel: md: minimum _guaranteed_ speed: 1000
> > KB/sec/disk.
> > Oct 20 19:59:59 ibm-z10-25 kernel: md: using maximum available idle IO
> > bandwidth (but not more than 200000 KB/sec) for recovery.
> > Oct 20 19:59:59 ibm-z10-25 kernel: md: using 128k window, over a total of
> > 306176k.
> > Oct 20 19:59:59 ibm-z10-25 systemd-udevd: inotify_add_watch(7, /dev/md0, 10)
> > failed: No such file or directory
> >
> > Xiao
> >
> > ----- Original Message -----
> > > From: "NeilBrown" <neilb@suse.de>
> > > To: "Xiao Ni" <xni@redhat.com>
> > > Cc: linux-raid@vger.kernel.org
> > > Sent: Wednesday, March 25, 2015 2:35:29 PM
> > > Subject: Re: /dev/md0 can't be created
> > >
> > > On Wed, 25 Mar 2015 02:15:34 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
> > >
> > > > Hi all
> > > >
> > > > I have encountered so many times, the raid device is created
> > > > successfully, but the directory
> > > > /dev/md0 can't be created. It can't reproduce 100%.
> > > >
> > > > [root@intel-sugarbay-do-01 create_assemble]# cat /proc/mdstat
> > > > Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
> > > > md0 : active raid10 loop7[7](S) loop6[6] loop5[5] loop4[4] loop3[3]
> > > > loop2[2] loop1[1] loop0[0]
> > > > 1788416 blocks super 1.2 512K chunks 2 near-copies [7/7] [UUUUUUU]
> > > > bitmap: 0/1 pages [0KB], 65536KB chunk
> > > >
> > > > unused devices: <none>
> > > > [root@intel-sugarbay-do-01 create_assemble]# ls /dev/md0
> > > > ls: cannot access /dev/md0: No such file or directory
> > > >
> > > > The underline devices are loop devices which are created with big
> > > > file.
> > > >
> > > > The kernel I used is RHEL7 (3.10.0-234.el7.x86_64.debug, mdadm -
> > > > v3.3.2
> > > > - 21st August 2014)
> > > > I'll try to reproduce this with upstream kernel and mdadm. But I think it
> > > > shouldn't be the problem about kernel.
> > > >
> > > > What do you think I should check for this? And which tool is
> > > > responsible for creating the directory? Maybe
> > > > I can add some log to it to find the reason.
> > > >
> > >
> > > /dev/md0 is created by udev.
> > > Run
> > > udevadm monitor
> > >
> > > to see the events that udev is processing. When and ADD event for "md0" is
> > > processed, /dev/md0 should get created.
> > >
> > > NeilBrown
> > >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* openSUSE 13.2: md - Need advice which raid1 super format (1.0/1.2) is preferred
From: Dieter Nützel @ 2015-03-31 23:55 UTC (permalink / raw)
To: linux-raid
Hello List,
I need advice from the 'gurus' which raid1 md super format (1.0 or 1.2)
is preferred for SSD/'old' SATA disks.
Currently I have this on my gfx devel machine (Xeon X3430):
cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sdc1[0] sdd1[1]
156288896 blocks super 1.0 [2/2] [UU]
bitmap: 0/2 pages [0KB], 65536KB chunk
md1 : active raid1 sdb2[1] sda2[0]
60391744 blocks super 1.2 [2/2] [UU]
md0 : active raid1 sdb1[1] sda1[0]
2095104 blocks super 1.2 [2/2] [UU]
unused devices: <none>
md0 and md1
are on my SSD's which I've created by 'hand' (with mdadm) under openSUSE
13.2.
md2
is on my SATAs which I've created 'automatically' under yast2.
My question to you is:
Looks this sane (chunk only for super 1.0 on real SATAs and super 1.2 on
SSDs) or should I change anything, here.
I'll nuke md0 (swap) and merge it with md1 (root) 'cause I've upgraded
my RAM from 4 GB to 16 GB (soon to 24/32 GB max) and do not need swap
anylonger.
Any hints?
-Dieter
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: no good deed goes unpunished...
From: Dave Stevens @ 2015-03-31 23:31 UTC (permalink / raw)
To: Phil Turmel; +Cc: Brad Campbell, Roger Heflin, linux-raid
In-Reply-To: <551B179A.6020008@turmel.org>
Quoting Phil Turmel <philip@turmel.org>:
snip!
> On 03/31/2015 05:52 PM, Phil Turmel wrote:
> If you did this with those three disks, you are likely to have random
>> data corruption from the older mirror. If so, stop and re-assemble with
>> just the last good disks.
>
> Let me clarify: you only had *two* working disks for a month or so
> before the array finally died. Only assemble those two.
yes that's clear. I've done that, found the LVM volgroup and can now
see the Xen domains, working on getting data off.
many thanks!
D
>
> Phil
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
"As long as politics is the shadow cast on society by big business,
the attenuation of the shadow will not change the substance."
-- John Dewey
^ permalink raw reply
* Re: Two system lockups, Magic SysKeys works, RAID6 resyncs on reboot
From: Mark Knecht @ 2015-03-31 22:08 UTC (permalink / raw)
To: Roman Mamedov; +Cc: Linux-RAID
In-Reply-To: <20150401013032.5c908e85@natsu>
On Tue, Mar 31, 2015 at 1:30 PM, Roman Mamedov <rm@romanrm.net> wrote:
> On Tue, 31 Mar 2015 08:00:27 -0700
> Mark Knecht <markknecht@gmail.com> wrote:
>
>> md3 : active raid6 sdc3[9] sdg3[5] sdf3[6] sde3[7] sdd3[8]
>> 1452264480 blocks super 1.2 level 6, 16k chunk, algorithm 2 [5/5] [UUUUU]
>> [====>................] resync = 22.8% (110608304/484088160)
>> finish=78.5min speed=79206K/sec
>
> Looks like you are running this without a write intent bitmap, why? It would
> at least save you from a full resync each time.
>
> --
> With respect,
> Roman
Honestly, I don't know. The machine was the first I ever built using
RAID for the system drive and I probably didn't find out about it, or
was just trying to keep it simple? Not at all sure.
I'll check it out.
Thanks,
Mark
^ permalink raw reply
* Re: no good deed goes unpunished...
From: Dave Stevens @ 2015-03-31 21:55 UTC (permalink / raw)
To: Phil Turmel; +Cc: Brad Campbell, Roger Heflin, linux-raid
In-Reply-To: <551B1717.2090809@turmel.org>
Quoting Phil Turmel <philip@turmel.org>:
> On 03/31/2015 05:41 PM, Dave Stevens wrote:
>> Quoting Brad Campbell <brad@fnarfbargle.com>:
>>
>>> md12 has claimed sda2 and md10 has sdb&c. Try this
>>>
>>> mdadm --stop /dev/md12
>>> mdadm --stop /dev/md10
>>> mdadm --assemble --force /dev/md13 /dev/sd[abc]2
>>
>> I owe you a beer, Brad! Now I only have to mount the LVM partition,
>> retrieve the Xen domains and rejig the management tools so I can do a
>> meaningful backup and restore but this is the bit I couldn't get my head
>> around.
>
> If you did this with those three disks, you are likely to have random
> data corruption from the older mirror. If so, stop and re-assemble with
> just the last good disks.
>
> Phil
>
OK, thanks!
Dave
--
"As long as politics is the shadow cast on society by big business,
the attenuation of the shadow will not change the substance."
-- John Dewey
^ permalink raw reply
* Re: no good deed goes unpunished...
From: Phil Turmel @ 2015-03-31 21:54 UTC (permalink / raw)
To: Dave Stevens, Brad Campbell; +Cc: Roger Heflin, linux-raid
In-Reply-To: <551B1717.2090809@turmel.org>
On 03/31/2015 05:52 PM, Phil Turmel wrote:
> On 03/31/2015 05:41 PM, Dave Stevens wrote:
>> Quoting Brad Campbell <brad@fnarfbargle.com>:
>>
>>> md12 has claimed sda2 and md10 has sdb&c. Try this
>>>
>>> mdadm --stop /dev/md12
>>> mdadm --stop /dev/md10
>>> mdadm --assemble --force /dev/md13 /dev/sd[abc]2
>>
>> I owe you a beer, Brad! Now I only have to mount the LVM partition,
>> retrieve the Xen domains and rejig the management tools so I can do a
>> meaningful backup and restore but this is the bit I couldn't get my head
>> around.
>
> If you did this with those three disks, you are likely to have random
> data corruption from the older mirror. If so, stop and re-assemble with
> just the last good disks.
Let me clarify: you only had *two* working disks for a month or so
before the array finally died. Only assemble those two.
Phil
^ permalink raw reply
* Re: no good deed goes unpunished...
From: Phil Turmel @ 2015-03-31 21:52 UTC (permalink / raw)
To: Dave Stevens, Brad Campbell; +Cc: Roger Heflin, linux-raid
In-Reply-To: <20150331144155.14405dy3bqxotw9v@webmail.uniserve.com>
On 03/31/2015 05:41 PM, Dave Stevens wrote:
> Quoting Brad Campbell <brad@fnarfbargle.com>:
>
>> md12 has claimed sda2 and md10 has sdb&c. Try this
>>
>> mdadm --stop /dev/md12
>> mdadm --stop /dev/md10
>> mdadm --assemble --force /dev/md13 /dev/sd[abc]2
>
> I owe you a beer, Brad! Now I only have to mount the LVM partition,
> retrieve the Xen domains and rejig the management tools so I can do a
> meaningful backup and restore but this is the bit I couldn't get my head
> around.
If you did this with those three disks, you are likely to have random
data corruption from the older mirror. If so, stop and re-assemble with
just the last good disks.
Phil
^ permalink raw reply
* Re: no good deed goes unpunished...
From: Dave Stevens @ 2015-03-31 21:41 UTC (permalink / raw)
To: Brad Campbell; +Cc: Roger Heflin, linux-raid
In-Reply-To: <5515FC86.7030402@fnarfbargle.com>
Quoting Brad Campbell <brad@fnarfbargle.com>:
> md12 has claimed sda2 and md10 has sdb&c. Try this
>
> mdadm --stop /dev/md12
> mdadm --stop /dev/md10
> mdadm --assemble --force /dev/md13 /dev/sd[abc]2
I owe you a beer, Brad! Now I only have to mount the LVM partition,
retrieve the Xen domains and rejig the management tools so I can do a
meaningful backup and restore but this is the bit I couldn't get my
head around.
Thanks to all who helped. Does anyone care to provide a reference to a
good intro to software RAID? This experience was educational all
right, but I would on the whole prefer to be more orderly and have a
good reference.
Dave
>
>
>
> On 28/03/15 08:37, Dave Stevens wrote:
>> Quoting Roger Heflin <rogerheflin@gmail.com>:
>>
>>> I don't know what to make of it.
>>>
>>> Everyone will ask you for this info:
>>> mdadm --examine /dev/sdc2 and the same output against
>>> /dev/sdb2
>>
>> ok, like this:
>>
>> # cat /proc/mdstat
>>
>> Personalities :
>>
>> md12 : inactive sda2[0](S)
>> 732467520 blocks
>>
>> md10 : inactive sdc2[2](S) sdb2[1](S)
>> 1464935040 blocks
>>
>> unused devices: <none>
>>
>>
>> ------------------------------ reboot ----------------------
>>
>> # mdadm -A /dev/md13 /dev/sdb2 /dev/sdc2
>>
>> mdadm : /dev/md13 assembles from 1 drive - not enough to start the array
>>
>> # cat /proc/mdstat
>>
>> Personalities :
>>
>> md13 : inactive sdc2[2](S) sdb2[1](S)
>>
>> 1464935040 blocks
>>
>> unused devices : <none>
>>
>> # mkdir aa
>>
>> # mount /dev/md13 aa
>>
>> mount: /dev/md13: can't read superblock
>>
>>
>> Dave
>>
>>
>>
>>
>>>
>>> On Fri, Mar 27, 2015 at 12:33 PM, Dave Stevens <geek@uniserve.com> wrote:
>>>> From: Roger Heflin <rogerheflin@gmail.com>
>>>> To: Dave Stevens <geek@uniserve.com>
>>>> Cc: Linux RAID <linux-raid@vger.kernel.org>
>>>> Subject: Re: no good deed goes unpunished...
>>>> Date: Thu, 26 Mar 2015 19:04:45 -0500
>>>> Sender: linux-raid-owner@vger.kernel.org
>>>>
>>>> you should probably do cat /proc/mdstat
>>>>
>>>> It is likely that the livecd may have already tried to assemble it,
>>>> and that the busy is because it is already in use.
>>>>
>>>> If it is already assembled and you want to redo it first you will need
>>>> to stop the assembled array to be able to redo it.
>>>>
>>>> so this is what I did while running the live distro:
>>>>
>>>> # cat /proc/mdstat
>>>>
>>>> Personalities :
>>>>
>>>> md12 : inactive sda2[0](S)
>>>> 732467520 blocks
>>>>
>>>> md10 : inactive sdc2[2](S) sdb2[1](S)
>>>> 1464935040 blocks
>>>>
>>>> unused devices: <none>
>>>>
>>>>
>>>> It seemed to me that matters were needlessly complicated by possible
>>>> residual issues from my earlier unsuccessful attempts so I rebooted.
>>>>
>>>>
>>>> ------------------------------ reboot ----------------------
>>>>
>>>> # mdadm -A /dev/md13 /dev/sdb2 /dev/sdc2
>>>>
>>>> mdadm : /dev/md13 assembles from 1 drive - not enough to start the array
>>>>
>>>> # cat /proc/mdstat
>>>>
>>>> Personalities :
>>>>
>>>> md13 : inactive sdc2[2](S) sdb2[1](S)
>>>>
>>>> 1464935040 blocks
>>>>
>>>> unused devices : <none>
>>>>
>>>> # mkdir aa
>>>>
>>>> # mount /dev/md13 aa
>>>>
>>>> mount: /dev/md13: can't read superblock
>>>>
>>>> So I still don't see what to do. Advice welcome.
>>>>
>>>> Dave
>>>>
>>>>
>>>>
>>>> --
>>>> "As long as politics is the shadow cast on society by big business,
>>>> the attenuation of the shadow will not change the substance."
>>>>
>>>> -- John Dewey
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>
>>
>>
>
>
> --
> Dolphins are so intelligent that within a few weeks they can
> train Americans to stand at the edge of the pool and throw them
> fish.
>
--
"As long as politics is the shadow cast on society by big business,
the attenuation of the shadow will not change the substance."
-- John Dewey
^ permalink raw reply
* Re: Two system lockups, Magic SysKeys works, RAID6 resyncs on reboot
From: Roman Mamedov @ 2015-03-31 20:30 UTC (permalink / raw)
To: Mark Knecht; +Cc: Linux-RAID
In-Reply-To: <CAK2H+efDuxdjmWZiOiJw+EK3wohY6FxRWgsm8rJc1j1+fniWig@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 490 bytes --]
On Tue, 31 Mar 2015 08:00:27 -0700
Mark Knecht <markknecht@gmail.com> wrote:
> md3 : active raid6 sdc3[9] sdg3[5] sdf3[6] sde3[7] sdd3[8]
> 1452264480 blocks super 1.2 level 6, 16k chunk, algorithm 2 [5/5] [UUUUU]
> [====>................] resync = 22.8% (110608304/484088160)
> finish=78.5min speed=79206K/sec
Looks like you are running this without a write intent bitmap, why? It would
at least save you from a full resync each time.
--
With respect,
Roman
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox