* Re: Split RAID: Proposal for archival RAID using incremental batch checksum
From: Piergiorgio Sartor @ 2014-10-30 17:25 UTC (permalink / raw)
To: Anshuman Aggarwal; +Cc: Ethan Wilson, linux-raid
In-Reply-To: <CAK-d5dZgsoy+PF2urM=N1F67w7nLGyc83muyNoT2h+gQt5ODnQ@mail.gmail.com>
On Thu, Oct 30, 2014 at 08:27:27PM +0530, Anshuman Aggarwal wrote:
> What you are suggesting will work for delaying writing the checksum
> (but still making 2 disks work non stop and lead to failure, cost
> etc).
Hi Anshuman,
I'm a bit missing the point here.
In my experience, with my storage systems, I change
disks because they're too small, way long before they
are too old (way long before they fail).
That's why I end up with a collection of small HDDs.
which, in turn, I recycled in some custom storage
system (using disks of different size, like explained
in one of the links posted before).
Honestly, the only reason to spin down the disks, still
in my experience, is for reducing power consumption.
And this can be done with a RAID-6 without problems
and in a extremely flexible way.
So, the bottom line, still in my experience, is that
this you're describing seems quite a nice situation.
Or, I did not understood what you're proposing.
Thanks,
bye,
pg
> I am proposing N independent disks which are rarely accessed. When
> parity has to be written to the remaining 1,2 ...X disks ...it is
> batched up (bcache is feasible) and written out once in a while
> depending on how much write is happening. N-1 disks stay spun down and
> only X disks wake up periodically to get checksum written to (this
> would be tweaked by the user based on how up to date he needs the
> parity to be (tolerance of rebuilding parity in case of crash) and vs
> disk access for each parity write)
>
> It can't be done using any RAID6 because RAID5/6 will stripe all the
> data across the devices making any read access wake up all the
> devices. Ditto for writing to parity on every write to a single disk.
>
> The architecture being proposed is a lazy write to manage parity for
> individual disks which won't suffer from RAID catastrophic data loss
> and concurrent disk.
>
>
>
>
> On 30 October 2014 00:57, Ethan Wilson <ethan.wilson@shiftmail.org> wrote:
> > On 29/10/2014 10:25, Anshuman Aggarwal wrote:
> >>
> >> Right on most counts but please see comments below.
> >>
> >> On 29 October 2014 14:35, NeilBrown <neilb@suse.de> wrote:
> >>>
> >>> Just to be sure I understand, you would have N + X devices. Each of the
> >>> N
> >>> devices contains an independent filesystem and could be accessed directly
> >>> if
> >>> needed. Each of the X devices contains some codes so that if at most X
> >>> devices in total died, you would still be able to recover all of the
> >>> data.
> >>> If more than X devices failed, you would still get complete data from the
> >>> working devices.
> >>>
> >>> Every update would only write to the particular N device on which it is
> >>> relevant, and all of the X devices. So N needs to be quite a bit bigger
> >>> than X for the spin-down to be really worth it.
> >>>
> >>> Am I right so far?
> >>
> >> Perfectly right so far. I typically have a N to X ratio of 4 (4
> >> devices to 1 data) so spin down is totally worth it for data
> >> protection but more on that below.
> >>
> >>> For some reason the writes to X are delayed... I don't really understand
> >>> that part.
> >>
> >> This delay is basically designed around archival devices which are
> >> rarely read from and even more rarely written to. By delaying writes
> >> on 2 criteria ( designated cache buffer filling up or preset time
> >> duration from last write expiring) we can significantly reduce the
> >> writes on the parity device. This assumes that we are ok to lose a
> >> movie or two in case the parity disk is not totally up to date but are
> >> more interested in device longevity.
> >>
> >>> Sounds like multi-parity RAID6 with no parity rotation and
> >>> chunksize == devicesize
> >>
> >> RAID6 would present us with a joint device and currently only allows
> >> writes to that directly, yes? Any writes will be striped.
> >
> >
> > I am not totally sure I understand your design, but it seems to me that the
> > following solution could work for you:
> >
> > MD raid-6, maybe multi-parity (multi-parity not implemented yet in MD yet,
> > but just do a periodic scrub and 2 parities can be fine. Wake-up is not so
> > expensive that you can't scrub)
> >
> > Over that you put a raid1 of 2 x 4TB disks as a bcache cache device (those
> > two will never spin-down) in writeback mode with writeback_running=off .
> > This will prevent writes to backend and leave the backend array spun down.
> > When bcache is almost full (poll dirty_data), switch to writeback_running=on
> > and writethrough: it will wake up the backend raid6 array and flush all
> > dirty data. You can then then revert to writeback and writeback_running=off.
> > After this you can spin-down the backend array again.
> >
> > You also get read caching for free, which helps the backend array to stay
> > spun down as much as possible.
> >
> > Maybe you can modify bcache slightly so to implement an automatic switching
> > between the modes as described above, instead of polling the state from
> > outside.
> >
> > Would that work, or you are asking something different?
> >
> > EW
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
piergiorgio
^ permalink raw reply
* Re: Fwd: I will pay money for the correct RAID recovery instructions
From: Ian Young @ 2014-10-30 23:03 UTC (permalink / raw)
To: John Stoffel; +Cc: Another Sillyname, linux-raid
In-Reply-To: <21569.16459.69850.406372@quad.stoffel.home>
Robin's solution worked. I used ddrescue to copy sde to a new disk,
then put the new disk in place of sde, and started the array in
degraded mode. Now I just need to get a replacement disk for sdf and
the array should be all set.
I believe I owe Robin a pint. Got Paypal or Google Wallet?
On Fri, Oct 17, 2014 at 9:14 AM, John Stoffel <john@stoffel.org> wrote:
>
> Another> Sorry to state the obvious but...... To restore the degraded
> Another> array would (based on the info you've posted) likely take
> Another> longer then temporarily moving the data to a different set of
> Another> drives.
>
> How do you figure this? He hasn't posted any useful data on his array
> at all. We know it's six 3Tb disks in RAID10, so that's at most 9Tb
> of data. Moving 3Tb (bad disk ddrescue'd onto new disk) is only 3Tb
> to move. Then the resync will of course take time.
>
> But if he doesn't have the spare disk space, it's probably the only way.
>
> Another> As time seems to be a major consideration here (likely/possible
> Another> failure of sde) then surely the optimal strategy has to be to get the
> Another> data off first, then look at the rebuilding the degraded array?
>
> Another> Just my 2c
>
> Another> On 17 October 2014 15:05, John Stoffel <john@stoffel.org> wrote:
>>>
>>>
>>> Ian,
>>>
>>> It would also help if you posted the details of your setup using:
>>>
>>> cat /proc/partitions
>>> cat /proc/mdstat
>>>
>>> mdadm -D /dev/md#
>>> - for each of the devices above.
>>>
>>> mdadm -E /dev/sd<drive><#>
>>> - for each disk or partition in the array from above.
>>>
>>>
>>>
>>> But the suggestions to ddrescue the going bad drive onto a new disk is
>>> a good one. On my debian system, I would do the following:
>>>
>>> sudo apt-get install gddrescue
>>> ddrescue /dev/sde /dev/sdf /var/tmp/ddrecue-sde.log
>>>
>>> and see how that goes.
>>>
>>> Good luck,
>>> John
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Another> --
> Another> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> Another> the body of a message to majordomo@vger.kernel.org
> Another> More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH] mdadm: fix --detail for cases where device count > max_devices/2
From: Or Sagi @ 2014-10-31 9:10 UTC (permalink / raw)
To: linux-raid
mdadm misreports (less devices then there are, and as a result decides the array is degraded) in cases where there are > max_devices/2 devices.
This seems to fix it.
---
Detail.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/Detail.c b/Detail.c
index c4fcad9..62e5867 100644
--- a/Detail.c
+++ b/Detail.c
@@ -72,7 +72,7 @@ int Detail(char *dev, struct context *c)
int failed = 0;
struct supertype *st;
char *subarray = NULL;
- int max_disks = MD_SB_DISKS; /* just a default */
+ int max_disks = MD_SB_DISKS * 2; /* just a default */
struct mdinfo *info = NULL;
struct mdinfo *sra;
struct mdinfo *subdev;
@@ -124,7 +124,7 @@ int Detail(char *dev, struct context *c)
rv = 0;
if (st)
- max_disks = st->max_devs;
+ max_disks = st->max_devs * 2;
if (subarray) {
/* This is a subarray of some container.
--
1.7.9.5
--
Or Sagi | Chief System Architect
Reduxio Systems
^ permalink raw reply related
* Re: Split RAID: Proposal for archival RAID using incremental batch checksum
From: Anshuman Aggarwal @ 2014-10-31 11:05 UTC (permalink / raw)
To: Piergiorgio Sartor; +Cc: Ethan Wilson, linux-raid
In-Reply-To: <20141030172500.GA2591@lazy.lzy>
Hi pg,
With MD raid striping all the writes not only does it keep ALL disks
spinning to read/write the current content, it also leads to
catastrophic data loss in case the rebuild/disk failure exceeds the
number of parity disks.
But more importantly, I find myself setting up multiple RAID levels
(at least RAID6 and now thinking of more) just to make sure that MD
raid will recover my data and not lose the whole cluster if an
additional disk fails above the number of parity!!! The biggest
advantage of the scheme that I have outlined is that with a single
check sum I am mostly assure of a failed disk restoration and worst
case only the media (movies/music) on the failing disk are lost not on
the whole cluster.
Also in my experience about disks and usage, while what you are saying
was true a while ago when storage capacity had not hit multiple TBs.
Now if I am buying 3-4 TB disks they are likely to last a while
especially since the incremental % growth in sizes seem to be slowing
down.
Regards,
Anshuman
On 30 October 2014 22:55, Piergiorgio Sartor
<piergiorgio.sartor@nexgo.de> wrote:
> On Thu, Oct 30, 2014 at 08:27:27PM +0530, Anshuman Aggarwal wrote:
>> What you are suggesting will work for delaying writing the checksum
>> (but still making 2 disks work non stop and lead to failure, cost
>> etc).
>
> Hi Anshuman,
>
> I'm a bit missing the point here.
>
> In my experience, with my storage systems, I change
> disks because they're too small, way long before they
> are too old (way long before they fail).
> That's why I end up with a collection of small HDDs.
> which, in turn, I recycled in some custom storage
> system (using disks of different size, like explained
> in one of the links posted before).
>
> Honestly, the only reason to spin down the disks, still
> in my experience, is for reducing power consumption.
> And this can be done with a RAID-6 without problems
> and in a extremely flexible way.
>
> So, the bottom line, still in my experience, is that
> this you're describing seems quite a nice situation.
>
> Or, I did not understood what you're proposing.
>
> Thanks,
>
> bye,
>
> pg
>
>> I am proposing N independent disks which are rarely accessed. When
>> parity has to be written to the remaining 1,2 ...X disks ...it is
>> batched up (bcache is feasible) and written out once in a while
>> depending on how much write is happening. N-1 disks stay spun down and
>> only X disks wake up periodically to get checksum written to (this
>> would be tweaked by the user based on how up to date he needs the
>> parity to be (tolerance of rebuilding parity in case of crash) and vs
>> disk access for each parity write)
>>
>> It can't be done using any RAID6 because RAID5/6 will stripe all the
>> data across the devices making any read access wake up all the
>> devices. Ditto for writing to parity on every write to a single disk.
>>
>> The architecture being proposed is a lazy write to manage parity for
>> individual disks which won't suffer from RAID catastrophic data loss
>> and concurrent disk.
>>
>>
>>
>>
>> On 30 October 2014 00:57, Ethan Wilson <ethan.wilson@shiftmail.org> wrote:
>> > On 29/10/2014 10:25, Anshuman Aggarwal wrote:
>> >>
>> >> Right on most counts but please see comments below.
>> >>
>> >> On 29 October 2014 14:35, NeilBrown <neilb@suse.de> wrote:
>> >>>
>> >>> Just to be sure I understand, you would have N + X devices. Each of the
>> >>> N
>> >>> devices contains an independent filesystem and could be accessed directly
>> >>> if
>> >>> needed. Each of the X devices contains some codes so that if at most X
>> >>> devices in total died, you would still be able to recover all of the
>> >>> data.
>> >>> If more than X devices failed, you would still get complete data from the
>> >>> working devices.
>> >>>
>> >>> Every update would only write to the particular N device on which it is
>> >>> relevant, and all of the X devices. So N needs to be quite a bit bigger
>> >>> than X for the spin-down to be really worth it.
>> >>>
>> >>> Am I right so far?
>> >>
>> >> Perfectly right so far. I typically have a N to X ratio of 4 (4
>> >> devices to 1 data) so spin down is totally worth it for data
>> >> protection but more on that below.
>> >>
>> >>> For some reason the writes to X are delayed... I don't really understand
>> >>> that part.
>> >>
>> >> This delay is basically designed around archival devices which are
>> >> rarely read from and even more rarely written to. By delaying writes
>> >> on 2 criteria ( designated cache buffer filling up or preset time
>> >> duration from last write expiring) we can significantly reduce the
>> >> writes on the parity device. This assumes that we are ok to lose a
>> >> movie or two in case the parity disk is not totally up to date but are
>> >> more interested in device longevity.
>> >>
>> >>> Sounds like multi-parity RAID6 with no parity rotation and
>> >>> chunksize == devicesize
>> >>
>> >> RAID6 would present us with a joint device and currently only allows
>> >> writes to that directly, yes? Any writes will be striped.
>> >
>> >
>> > I am not totally sure I understand your design, but it seems to me that the
>> > following solution could work for you:
>> >
>> > MD raid-6, maybe multi-parity (multi-parity not implemented yet in MD yet,
>> > but just do a periodic scrub and 2 parities can be fine. Wake-up is not so
>> > expensive that you can't scrub)
>> >
>> > Over that you put a raid1 of 2 x 4TB disks as a bcache cache device (those
>> > two will never spin-down) in writeback mode with writeback_running=off .
>> > This will prevent writes to backend and leave the backend array spun down.
>> > When bcache is almost full (poll dirty_data), switch to writeback_running=on
>> > and writethrough: it will wake up the backend raid6 array and flush all
>> > dirty data. You can then then revert to writeback and writeback_running=off.
>> > After this you can spin-down the backend array again.
>> >
>> > You also get read caching for free, which helps the backend array to stay
>> > spun down as much as possible.
>> >
>> > Maybe you can modify bcache slightly so to implement an automatic switching
>> > between the modes as described above, instead of polling the state from
>> > outside.
>> >
>> > Would that work, or you are asking something different?
>> >
>> > EW
>> >
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> > the body of a message to majordomo@vger.kernel.org
>> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> --
>
> piergiorgio
^ permalink raw reply
* Raid5 drive fail during grow and no backup
From: Vince @ 2014-10-31 13:34 UTC (permalink / raw)
To: linux-raid
Hi,
got a drive failure (bad block) during Raid5 grow (4x3TB -> 5x3TB).
Well... i don't have a backup file :/
Mdadm shows 1 drive as removed.
All 4 'good' drives are in the same reshape pos'n.
Any idea how to finish the reshape process? Or get the array back?
mdadm --examine /dev/sdb
/dev/sdb:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : 14e9502c:4d51fb5c:a4f2e4d1:2b6a157e
Name : MyRaid:0
Creation Time : Mon Mar 18 12:52:00 2013
Raid Level : raid5
Raid Devices : 5
Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
Array Size : 11720540160 (11177.58 GiB 12001.83 GB)
Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : active
Device UUID : 705995da:442a6d8d:783abc2f:9d88e715
Reshape pos'n : 9243070464 (8814.88 GiB 9464.90 GB)
Delta Devices : 1 (4->5)
Update Time : Fri Oct 31 13:21:48 2014
Checksum : 82973929 - correct
Events : 18837
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : .AAAA ('A' == active, '.' == missing)
mdadm -detail:
/dev/md0:
Version : 1.2
Creation Time : Mon Mar 18 12:52:00 2013
Raid Level : raid5
Used Dev Size : -1
Raid Devices : 5
Total Devices : 4
Persistence : Superblock is persistent
Update Time : Fri Oct 31 13:21:48 2014
State : active, degraded, Not Started
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Delta Devices : 1, (4->5)
Name : MyRaid:0
UUID : 14e9502c:4d51fb5c:a4f2e4d1:2b6a157e
Events : 18837
Number Major Minor RaidDevice State
0 0 0 0 removed
5 8 48 1 active sync /dev/sdd
3 8 32 2 active sync /dev/sdc
4 8 80 3 active sync /dev/sdf
6 8 16 4 active sync /dev/sdb
mdadm -A scan -v:
mdadm: looking for devices for /dev/md/0
mdadm: /dev/sdf is identified as a member of /dev/md/0, slot 3.
mdadm: /dev/sdd is identified as a member of /dev/md/0, slot 1.
mdadm: /dev/sdc is identified as a member of /dev/md/0, slot 2.
mdadm: /dev/sdb is identified as a member of /dev/md/0, slot 4.
mdadm:/dev/md/0 has an active reshape - checking if critical section needs
to be restored
mdadm: too-old timestamp on backup-metadata on device-4
mdadm: no uptodate device for slot 0 of /dev/md/0
mdadm: added /dev/sdc to /dev/md/0 as 2
mdadm: added /dev/sdf to /dev/md/0 as 3
mdadm: added /dev/sdb to /dev/md/0 as 4
mdadm: added /dev/sdd to /dev/md/0 as 1
mdadm: /dev/md/0 assembled from 4 drives - not enough to start the array
while not clean - consider --force.
^ permalink raw reply
* Re: Split RAID: Proposal for archival RAID using incremental batch checksum
From: Matt Garman @ 2014-10-31 14:25 UTC (permalink / raw)
To: Anshuman Aggarwal; +Cc: Piergiorgio Sartor, Ethan Wilson, Mdadm
In-Reply-To: <CAK-d5dbL0wwC0jWgHG5KKLHgJfJ3t7=8Y7cEW=Qo1WHYA+jZbA@mail.gmail.com>
(Re-posting as I forgot to change to plaintext mode for the mailing
list, sorry for any dups.)
In a later post, you said you had a 4-to-1 scheme, but it wasn't clear
to me if that was 1 drive worth of data, and 4 drives worth of
checksum/backup, or the other way around.
In your proposed scheme, I assume you want your actual data drives to
be spinning all the time? Otherwise, when you go to read data (play
music/videos), you have the multi-second spinup delay... or is that OK
with you?
Some other considerations: modern 5400 RPM drives generally consume
less than five watts in idle state[1]. Actual AC draw will be higher
due to power supply inefficiency, so we'll err on the conservative
side and say each drive requires 10 AC watts of power. My electrical
rates in Chicago are about average for the USA (11 or 12 cents/kWH),
and conveniently it roughly works out such that one always-on watt
costs about $1/year. So, each always-running hard drive will cost
about $10/year to run, less with a more efficient power supply. I
know electricity is substantially more expensive in many parts of the
world; or maybe you're running off-the-grid (e.g. solar) and have a
very small power budget?
On Wed, Oct 29, 2014 at 2:15 AM, Anshuman Aggarwal
<anshuman.aggarwal@gmail.com> wrote:
>
> - SnapRAID (http://snapraid.sourceforge.net/) which is a snapshot
> based scheme (Its advantages are that its in user space and has cross
> platform support but has the huge disadvantage of every checksum being
> done from scratch slowing the system, causing immense wear and tear on
> every snapshot and also losing any information updates upto the
> snapshot point etc)
Last time I looked at SnapRAID, it seemed like yours was its target
use case. The "huge disadvantage of every checksum being done from
scratch" sounds like a SnapRAID feature enhancement that might be
simpler/easier/faster-to-get done than a major enhancement to the
Linux kernel (just speculating though).
But, on the other hand, by your use case description, writes are very
infrequent, and you're willing to buffer checksum updates for quite a
while... so what if you had a *monthly* cron job to do parity syncs?
Schedule it for a time when the system is unlikely to be used to
offset the increased load. That's only 12 "hard" tasks for the drive
per year. I'm not an expert, but that doesn't "feel" like a lot of
wear and tear.
On the issue of wear and tear, I've mostly given up trying to
understand what's best for my drives. One school of thought says many
spinup-spindown cycles are actually harder on the drive than running
24/7. But maybe consumer drives actually aren't designed for 24/7
operation, so they're better off being cycled up and down. Or
consumer drives can't handle the vibrations of being in a case with
other 24/7 drives. But failure to"exercise" the entire drive
regularly enough might result in a situation where an error has
developed but you don't know until it's too late or your warranty
period has expired.
[1] http://www.silentpcreview.com/article29-page2.html
On Fri, Oct 31, 2014 at 6:05 AM, Anshuman Aggarwal
<anshuman.aggarwal@gmail.com> wrote:
> Hi pg,
> With MD raid striping all the writes not only does it keep ALL disks
> spinning to read/write the current content, it also leads to
> catastrophic data loss in case the rebuild/disk failure exceeds the
> number of parity disks.
>
> But more importantly, I find myself setting up multiple RAID levels
> (at least RAID6 and now thinking of more) just to make sure that MD
> raid will recover my data and not lose the whole cluster if an
> additional disk fails above the number of parity!!! The biggest
> advantage of the scheme that I have outlined is that with a single
> check sum I am mostly assure of a failed disk restoration and worst
> case only the media (movies/music) on the failing disk are lost not on
> the whole cluster.
>
> Also in my experience about disks and usage, while what you are saying
> was true a while ago when storage capacity had not hit multiple TBs.
> Now if I am buying 3-4 TB disks they are likely to last a while
> especially since the incremental % growth in sizes seem to be slowing
> down.
>
> Regards,
> Anshuman
>
> On 30 October 2014 22:55, Piergiorgio Sartor
> <piergiorgio.sartor@nexgo.de> wrote:
>> On Thu, Oct 30, 2014 at 08:27:27PM +0530, Anshuman Aggarwal wrote:
>>> What you are suggesting will work for delaying writing the checksum
>>> (but still making 2 disks work non stop and lead to failure, cost
>>> etc).
>>
>> Hi Anshuman,
>>
>> I'm a bit missing the point here.
>>
>> In my experience, with my storage systems, I change
>> disks because they're too small, way long before they
>> are too old (way long before they fail).
>> That's why I end up with a collection of small HDDs.
>> which, in turn, I recycled in some custom storage
>> system (using disks of different size, like explained
>> in one of the links posted before).
>>
>> Honestly, the only reason to spin down the disks, still
>> in my experience, is for reducing power consumption.
>> And this can be done with a RAID-6 without problems
>> and in a extremely flexible way.
>>
>> So, the bottom line, still in my experience, is that
>> this you're describing seems quite a nice situation.
>>
>> Or, I did not understood what you're proposing.
>>
>> Thanks,
>>
>> bye,
>>
>> pg
>>
>>> I am proposing N independent disks which are rarely accessed. When
>>> parity has to be written to the remaining 1,2 ...X disks ...it is
>>> batched up (bcache is feasible) and written out once in a while
>>> depending on how much write is happening. N-1 disks stay spun down and
>>> only X disks wake up periodically to get checksum written to (this
>>> would be tweaked by the user based on how up to date he needs the
>>> parity to be (tolerance of rebuilding parity in case of crash) and vs
>>> disk access for each parity write)
>>>
>>> It can't be done using any RAID6 because RAID5/6 will stripe all the
>>> data across the devices making any read access wake up all the
>>> devices. Ditto for writing to parity on every write to a single disk.
>>>
>>> The architecture being proposed is a lazy write to manage parity for
>>> individual disks which won't suffer from RAID catastrophic data loss
>>> and concurrent disk.
>>>
>>>
>>>
>>>
>>> On 30 October 2014 00:57, Ethan Wilson <ethan.wilson@shiftmail.org> wrote:
>>> > On 29/10/2014 10:25, Anshuman Aggarwal wrote:
>>> >>
>>> >> Right on most counts but please see comments below.
>>> >>
>>> >> On 29 October 2014 14:35, NeilBrown <neilb@suse.de> wrote:
>>> >>>
>>> >>> Just to be sure I understand, you would have N + X devices. Each of the
>>> >>> N
>>> >>> devices contains an independent filesystem and could be accessed directly
>>> >>> if
>>> >>> needed. Each of the X devices contains some codes so that if at most X
>>> >>> devices in total died, you would still be able to recover all of the
>>> >>> data.
>>> >>> If more than X devices failed, you would still get complete data from the
>>> >>> working devices.
>>> >>>
>>> >>> Every update would only write to the particular N device on which it is
>>> >>> relevant, and all of the X devices. So N needs to be quite a bit bigger
>>> >>> than X for the spin-down to be really worth it.
>>> >>>
>>> >>> Am I right so far?
>>> >>
>>> >> Perfectly right so far. I typically have a N to X ratio of 4 (4
>>> >> devices to 1 data) so spin down is totally worth it for data
>>> >> protection but more on that below.
>>> >>
>>> >>> For some reason the writes to X are delayed... I don't really understand
>>> >>> that part.
>>> >>
>>> >> This delay is basically designed around archival devices which are
>>> >> rarely read from and even more rarely written to. By delaying writes
>>> >> on 2 criteria ( designated cache buffer filling up or preset time
>>> >> duration from last write expiring) we can significantly reduce the
>>> >> writes on the parity device. This assumes that we are ok to lose a
>>> >> movie or two in case the parity disk is not totally up to date but are
>>> >> more interested in device longevity.
>>> >>
>>> >>> Sounds like multi-parity RAID6 with no parity rotation and
>>> >>> chunksize == devicesize
>>> >>
>>> >> RAID6 would present us with a joint device and currently only allows
>>> >> writes to that directly, yes? Any writes will be striped.
>>> >
>>> >
>>> > I am not totally sure I understand your design, but it seems to me that the
>>> > following solution could work for you:
>>> >
>>> > MD raid-6, maybe multi-parity (multi-parity not implemented yet in MD yet,
>>> > but just do a periodic scrub and 2 parities can be fine. Wake-up is not so
>>> > expensive that you can't scrub)
>>> >
>>> > Over that you put a raid1 of 2 x 4TB disks as a bcache cache device (those
>>> > two will never spin-down) in writeback mode with writeback_running=off .
>>> > This will prevent writes to backend and leave the backend array spun down.
>>> > When bcache is almost full (poll dirty_data), switch to writeback_running=on
>>> > and writethrough: it will wake up the backend raid6 array and flush all
>>> > dirty data. You can then then revert to writeback and writeback_running=off.
>>> > After this you can spin-down the backend array again.
>>> >
>>> > You also get read caching for free, which helps the backend array to stay
>>> > spun down as much as possible.
>>> >
>>> > Maybe you can modify bcache slightly so to implement an automatic switching
>>> > between the modes as described above, instead of polling the state from
>>> > outside.
>>> >
>>> > Would that work, or you are asking something different?
>>> >
>>> > EW
>>> >
>>> > --
>>> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> > the body of a message to majordomo@vger.kernel.org
>>> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>> --
>>
>> piergiorgio
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Trying to get POLICY working
From: Caspar Smit @ 2014-10-31 15:19 UTC (permalink / raw)
To: linux-raid
Hi all,
I'm trying to get the POLICY framework of mdadm working but I can't seem to.
As i understand in the man page of mdadm the Incremental and POLICY
directives could allow adding a new disk without MD superblock as
spare to an already active array:
"Note that mdadm will normally only add devices to an array which were
previously working (active or spare) parts of that array. The support
for automatic inclusion of a new drive as a spare in some array
requires a configuration through POLICY in config file."
Furthermore:
"If no md metadata is found, the device may be still added to an array
as a spare if POLICY allows."
To get the basics working I created a system with 3 disks /dev/sdb,
/dev/sdc and /dev/sdd
Created a RAID5 with one missing disk:
mdadm -C /dev/md0 -l 5 -n 3 /dev/sd[b-c] missing
I set the POLICY in mdadm.conf to:
POLICY action=force-spare
This should add any device (passed through mdadm --incremental) as
spare no matter what (Am i correct?)
Now when I do:
#mdadm --incremental /dev/sdd
mdadm: no RAID superblock on /dev/sdd.
Well, i know there is no MD superblock on /dev/sdd but shouldn't the
policy setting kick in here and add /dev/sdd as spare (and hence start
rebuilding) to /dev/md0?
mdadm version: 3.2.5-5 (latest debian wheezy stable)
kernel version: 3.2.63-2 (latest debian wheezy stable)
Kind regards,
Caspar Smit
^ permalink raw reply
* Re: Trying to get POLICY working
From: Robin Hill @ 2014-10-31 15:34 UTC (permalink / raw)
To: Caspar Smit; +Cc: linux-raid
In-Reply-To: <CADvHm=ZLp08Ydp+P=gn+bu1=GgVORx0TSJoRU33z3UNubwQryQ@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2512 bytes --]
On Fri Oct 31, 2014 at 04:19:04PM +0100, Caspar Smit wrote:
> Hi all,
>
> I'm trying to get the POLICY framework of mdadm working but I can't seem to.
>
> As i understand in the man page of mdadm the Incremental and POLICY
> directives could allow adding a new disk without MD superblock as
> spare to an already active array:
>
> "Note that mdadm will normally only add devices to an array which were
> previously working (active or spare) parts of that array. The support
> for automatic inclusion of a new drive as a spare in some array
> requires a configuration through POLICY in config file."
>
> Furthermore:
>
> "If no md metadata is found, the device may be still added to an array
> as a spare if POLICY allows."
>
>
> To get the basics working I created a system with 3 disks /dev/sdb,
> /dev/sdc and /dev/sdd
>
> Created a RAID5 with one missing disk:
>
> mdadm -C /dev/md0 -l 5 -n 3 /dev/sd[b-c] missing
>
> I set the POLICY in mdadm.conf to:
>
> POLICY action=force-spare
>
> This should add any device (passed through mdadm --incremental) as
> spare no matter what (Am i correct?)
>
> Now when I do:
>
> #mdadm --incremental /dev/sdd
> mdadm: no RAID superblock on /dev/sdd.
>
> Well, i know there is no MD superblock on /dev/sdd but shouldn't the
> policy setting kick in here and add /dev/sdd as spare (and hence start
> rebuilding) to /dev/md0?
>
> mdadm version: 3.2.5-5 (latest debian wheezy stable)
> kernel version: 3.2.63-2 (latest debian wheezy stable)
>
According to the mdadm.conf manual page on my machine:
The action item determines the automatic behavior allowed for
devices matching the path and type in the same line. If a
device matches several lines with different actions then the
most permissive will apply. The ordering of policy lines is
irrelevant to the end result.
With the examples given being:
POLICY domain=domain1 metadata=imsm path=pci-0000:00:1f.2-scsi-* action=spare
POLICY domain=domain1 metadata=imsm path=pci-0000:04:00.0-scsi-[01]* action=include
So I'd guess that the path= entry is required (though the type value
would look to be optional, which is not clear from the text).
HTH,
Robin
--
___
( ' } | Robin Hill <robin@robinhill.me.uk> |
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply
* Re: question about MD raid rebuild performance degradation even with speed_limit_min/speed_limit_max set.
From: Peter Grandi @ 2014-10-31 19:44 UTC (permalink / raw)
To: Linux RAID
In-Reply-To: <54515493.4070402@cse.yorku.ca>
>>>>>> If, for example, I set speed_limit_min AND speed_limit_max to
>>>>>> 80000 then fail a disk when there is no other disk activity, then
>>>>>> I do get a rebuild rate of around 80 MB/s. However, if I then
>>>>>> start up a write intensive operation on the MD array (eg. a dd,
>>>>>> or a mkfs on an LVM logical volume that is created on that MD),
>>>>>> then, my write operation seems to get "full power", and my
>>>>>> rebuild drops to around 25 MB/s.
Linux MD RAID is fundamentally an IO address remapper, and actual IO is
scheduled and executed by the Linux block (page) IO subsystem. This
separation is beneficial in many ways.
Also the bandwidth delivered by a storage device is not a single
number. Disk drives transfer rate depend a lot on degree of randomness
of access, and outer vs. inner regions. Common disks can therefore
deliver bandwidth between 150MB/s and 0.5MB/s depending on the overall
traffic investing them.
Therefore in order to deliver a consistent transfer rate to the MD
resync kernel process the Linux block (page) IO subsystem would have to
be quite clever in controlling the rates of usage of all the processes
using a disk.
>>> I'm coming from using a 3Ware hardware RAID cotntroller where I
>>> could configure how much of the disk bandwidth is to be used for a
>>> rebuild versus I/O.
That (usually) works because the disk is completely dedicated to the
RAID card and the RAID card can schedule all IO to it.
>>> From what I understand, you're saying that MD [ ... ] and other
>>> system I/O needs that disk bandwidth, then there's nothing it can do
>>> about it. I guess I just don't understand why. Why can't md be given
>>> a priority in the kernel to allow the admin to decide how much
>>> bandwidth goes to system I/O versus rebuild I/O.
There is something you can do about it: rewrite the block IO subsystem
in the Linux kernel so that it can be configured to allocate IOPS and/or
bandwidth quotas to different processes, among them the MD resync kernel
process (extra awesome if that is also isochronous).
Because as well summarized below that process is just one of many
possible users of a given disk in a Linux based system:
>> There are difficulties in guaranteeing a minimum when the array uses
>> partitions from devices on which other partitions are used for other
>> things.
Put another way, designing MD RAID as fundamentally an IO address
remapper, and letting the MD resync kernel process run as "just another
process", has some big advantages and gives a lot of flexibility, but
means relying on the kernel block subsystem to do actual IO, and
accepting its current limitations. That is a free choice.
> The truth is, as people start to combine larger and larger
> disks, and rebuild times go up and up and up, this type of
> request will become more common....
Request of the type "I decided to use a physical storage design that
behaves in a way that I don't like so MD should do magic and work around
my decision" are already common. :-).
Using «larger and larger disks» is a *choice people make* and if they
don't like the obvious consequences they should then not make that
choice; they can instead choose to use smaller disks, or just the outer
part of larger disks (which can` be cheaper).
A critical metric for physical storage is IOPS/GB ratios (and their
variability dependent on workload) and ooking at those ratios I
personally think that common disks larger than 1TB are not suitable for
many cases of "typical" live data usage, and in the day job we sometimes
build MD RAID sets made of 146GB 15k disks, because fortunately my
colleagues understand the relevant tradeoffs too.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Trying to get POLICY working
From: NeilBrown @ 2014-11-01 0:20 UTC (permalink / raw)
To: Caspar Smit; +Cc: linux-raid
In-Reply-To: <CADvHm=ZLp08Ydp+P=gn+bu1=GgVORx0TSJoRU33z3UNubwQryQ@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2937 bytes --]
On Fri, 31 Oct 2014 16:19:04 +0100 Caspar Smit <c.smit@truebit.nl> wrote:
> Hi all,
>
> I'm trying to get the POLICY framework of mdadm working but I can't seem to.
>
> As i understand in the man page of mdadm the Incremental and POLICY
> directives could allow adding a new disk without MD superblock as
> spare to an already active array:
>
> "Note that mdadm will normally only add devices to an array which were
> previously working (active or spare) parts of that array. The support
> for automatic inclusion of a new drive as a spare in some array
> requires a configuration through POLICY in config file."
>
> Furthermore:
>
> "If no md metadata is found, the device may be still added to an array
> as a spare if POLICY allows."
>
>
> To get the basics working I created a system with 3 disks /dev/sdb,
> /dev/sdc and /dev/sdd
>
> Created a RAID5 with one missing disk:
>
> mdadm -C /dev/md0 -l 5 -n 3 /dev/sd[b-c] missing
>
> I set the POLICY in mdadm.conf to:
>
> POLICY action=force-spare
>
> This should add any device (passed through mdadm --incremental) as
> spare no matter what (Am i correct?)
That is the theory, yes.
>
> Now when I do:
>
> #mdadm --incremental /dev/sdd
> mdadm: no RAID superblock on /dev/sdd.
The message suggests that 'guess_super' found something on the device, but
it didn't turn out to be something useful.... not very helpful I know.
What does "mdadm --examine /dev/sdd" report?
I suspect there is a partition table and that is causing the confusion.
Try removing the partition table (dd /dev/zero to the device for a few K).
Then try again.
Probably need a fix like:
diff --git a/Incremental.c b/Incremental.c
index c9372587f518..3156190c4603 100644
--- a/Incremental.c
+++ b/Incremental.c
@@ -196,7 +196,7 @@ int Incremental(struct mddev_dev *devlist, struct context *c,
policy = disk_policy(&dinfo);
have_target = policy_check_path(&dinfo, &target_array);
- if (st == NULL && (st = guess_super(dfd)) == NULL) {
+ if (st == NULL && (st = guess_super_type(dfd, guess_array)) == NULL) {
if (c->verbose >= 0)
pr_err("no recognisable superblock on %s.\n",
devname);
and probably should improve the error messages...
Thanks for the report. Please let me know if that works, and what other
difficulties you hit.
Thanks,
NeilBrown
>
> Well, i know there is no MD superblock on /dev/sdd but shouldn't the
> policy setting kick in here and add /dev/sdd as spare (and hence start
> rebuilding) to /dev/md0?
>
> mdadm version: 3.2.5-5 (latest debian wheezy stable)
> kernel version: 3.2.63-2 (latest debian wheezy stable)
>
> Kind regards,
> Caspar Smit
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply related
* Re: Split RAID: Proposal for archival RAID using incremental batch checksum
From: Anshuman Aggarwal @ 2014-11-01 5:36 UTC (permalink / raw)
To: Matt Garman; +Cc: Mdadm
In-Reply-To: <CAJvUf-BktH_E6jb5d94VuMVEBf_Be4i_8u_kBYU52Df1cu0gmg@mail.gmail.com>
On 31 October 2014 19:53, Matt Garman <matthew.garman@gmail.com> wrote:
> In a later post, you said you had a 4-to-1 scheme, but it wasn't clear to me
> if that was 1 drive worth of data, and 4 drives worth of checksum/backup, or
> the other way around.
I was wondering if anybody would catch that slip. I meant 4 data to 1
parity seems about the right mix to me so far based on the my read and
feel of probability of drive failure.
>
> In your proposed scheme, I assume you want your actual data drives to be
> spinning all the time? Otherwise, when you go to read data (play
> music/videos), you have the multi-second spinup delay... or is that OK with
> you?
Well, actually in my experience with 6-8, 2-4TB drives there is a lot
of music/video content that I dont' end up playing that often. Those
drives can easily be spun down (maybe for days on end and at least all
night) and a small initial (one time) delay before playing a file who
drive hasn't been accessed easily seems like a good trade off ( both
for power and drive life )
>
> Some other considerations: modern 5400 RPM drives generally consume less
> than five watts in idle state[1]. Actual AC draw will be higher due to
> power supply inefficiency, so we'll err on the conservative side and say
> each drive requires 10 AC watts of power. My electrical rates in Chicago
> are about average for the USA (11 or 12 cents/kWH), and conveniently it
> roughly works out such that one always-on watt costs about $1/year. So,
> each always-running hard drive will cost about $10/year to run, less with a
> more efficient power supply. I know electricity is substantially more
> expensive in many parts of the world; or maybe you're running off-the-grid
> (e.g. solar) and have a very small power budget?
Besides the cost, there is an environmental aspect. If something has
superior efficiency and increases life of the product isn't it a good
thing wherever we live on the planet. BTW great calculation but I
moved back (to India) from San Francisco some time ago :) and the
electricity cost is quite high (and availability of supply is not 100%
yet). I'd like to maximize my backups and spinning disks that are not
being used for hours on end sounds bad.
Just to add, internet is metered per GB in many parts (and in mine
sadly :( for high speed access (meaning 4-8 MBps) so I have to store
content locally (before cloud suggestions are thrown around)
>
> On Wed, Oct 29, 2014 at 2:15 AM, Anshuman Aggarwal
> <anshuman.aggarwal@gmail.com> wrote:
>>
>> - SnapRAID (http://snapraid.sourceforge.net/) which is a snapshot
>> based scheme (Its advantages are that its in user space and has cross
>> platform support but has the huge disadvantage of every checksum being
>> done from scratch slowing the system, causing immense wear and tear on
>> every snapshot and also losing any information updates upto the
>> snapshot point etc)
>
>
> Last time I looked at SnapRAID, it seemed like yours was its target use
> case. The "huge disadvantage of every checksum being done from scratch"
> sounds like a SnapRAID feature enhancement that might be
> simpler/easier/faster-to-get done than a major enhancement to the Linux
> kernel (just speculating though).
SnapRAID can't be enhanced without involving the kernel because the
delta checksum will require knowing which blocks were written to and
only a kernel level driver can know that. This is a hard reality, no
way around it and that was my reason to propose this.
>
> But, on the other hand, by your use case description, writes are very
> infrequent, and you're willing to buffer checksum updates for quite a
> while... so what if you had a *monthly* cron job to do parity syncs?
> Schedule it for a time when the system is unlikely to be used to offset the
> increased load. That's only 12 "hard" tasks for the drive per year. I'm
> not an expert, but that doesn't "feel" like a lot of wear and tear.
Well, again, between infrequent updates down to weekly or monthly
crons sounds like a bad compromise either way when a better
incremental update could store the checksum in a buffer and write them
out eventually (2-3 times a day). Almost always the buffer will get
written out giving us an updated parity with little to none "extra"
wear and tear.
>
> On the issue of wear and tear, I've mostly given up trying to understand
> what's best for my drives. One school of thought says many spinup-spindown
> cycles are actually harder on the drive than running 24/7. But maybe
> consumer drives actually aren't designed for 24/7 operation, so they're
> better off being cycled up and down. Or consumer drives can't handle the
> vibrations of being in a case with other 24/7 drives. But failure
> to"exercise" the entire drive regularly enough might result in a situation
> where an error has developed but you don't know until it's too late or your
> warranty period has expired.
You are right about consumer drives where spin downs are good ...with
a time of an hour or so should reduce unnecessary spin up/downs. Once
spun down, most may stay that way for days which is better for all of
us (energy, wastage of drives etc). Spin down technology is
progressing faster than block failure (also because block density is
going up causing media failure and not the head failure to be the
primary cause of drive outage)
The drive can be tested periodically (by non destructive bad blocks
etc) as a pure testing exercise to find errors being developed. There
is no need to needlessly stress the drives out by reading/writing to
all parts continuously. Also RAID speeds are often no longer required
due to the higher R/W coming from the drives.
Thanks for reading and writing such a thorough reply.
Neil, would you be willing to assist/guide in helping design or with
the best approach to the same? I would like to avoid the obvious
pitfalls that any new kernel block level device writer is bound to
face.
Regards,
Anshuman
>
>
> [1] http://www.silentpcreview.com/article29-page2.html
>
>
^ permalink raw reply
* Re: Split RAID: Proposal for archival RAID using incremental batch checksum
From: Piergiorgio Sartor @ 2014-11-01 12:55 UTC (permalink / raw)
To: Anshuman Aggarwal; +Cc: Piergiorgio Sartor, Ethan Wilson, linux-raid
In-Reply-To: <CAK-d5dbL0wwC0jWgHG5KKLHgJfJ3t7=8Y7cEW=Qo1WHYA+jZbA@mail.gmail.com>
On Fri, Oct 31, 2014 at 04:35:11PM +0530, Anshuman Aggarwal wrote:
> Hi pg,
> With MD raid striping all the writes not only does it keep ALL disks
> spinning to read/write the current content, it also leads to
> catastrophic data loss in case the rebuild/disk failure exceeds the
> number of parity disks.
Hi Anshuman,
yes but do you have hard evidence that
this is a common RAID-6 problem?
Considering that we have now bad block list,
write intent bitmap and proactive replacement,
it does not seem to me really the main issue,
having a triple fail in RAID-6.
Considering that there are available libraries
for more that 2 parities, I think the multiple
failure case is quite a rarity.
Furthermore, I suspect there are other type
of catastrophic situation (lighting, for example)
that can destroy an array completely.
> But more importantly, I find myself setting up multiple RAID levels
> (at least RAID6 and now thinking of more) just to make sure that MD
> raid will recover my data and not lose the whole cluster if an
> additional disk fails above the number of parity!!! The biggest
> advantage of the scheme that I have outlined is that with a single
> check sum I am mostly assure of a failed disk restoration and worst
> case only the media (movies/music) on the failing disk are lost not on
> the whole cluster.
Each disk will have its own filesystem?
If this is not the case, you cannot say
if a single disk failure will lose only
some files.
> Also in my experience about disks and usage, while what you are saying
> was true a while ago when storage capacity had not hit multiple TBs.
> Now if I am buying 3-4 TB disks they are likely to last a while
> especially since the incremental % growth in sizes seem to be slowing
> down.
As wrote above, you can safely replace
disks before they fail, without compromising
the array.
bye,
pg
> Regards,
> Anshuman
>
> On 30 October 2014 22:55, Piergiorgio Sartor
> <piergiorgio.sartor@nexgo.de> wrote:
> > On Thu, Oct 30, 2014 at 08:27:27PM +0530, Anshuman Aggarwal wrote:
> >> What you are suggesting will work for delaying writing the checksum
> >> (but still making 2 disks work non stop and lead to failure, cost
> >> etc).
> >
> > Hi Anshuman,
> >
> > I'm a bit missing the point here.
> >
> > In my experience, with my storage systems, I change
> > disks because they're too small, way long before they
> > are too old (way long before they fail).
> > That's why I end up with a collection of small HDDs.
> > which, in turn, I recycled in some custom storage
> > system (using disks of different size, like explained
> > in one of the links posted before).
> >
> > Honestly, the only reason to spin down the disks, still
> > in my experience, is for reducing power consumption.
> > And this can be done with a RAID-6 without problems
> > and in a extremely flexible way.
> >
> > So, the bottom line, still in my experience, is that
> > this you're describing seems quite a nice situation.
> >
> > Or, I did not understood what you're proposing.
> >
> > Thanks,
> >
> > bye,
> >
> > pg
> >
> >> I am proposing N independent disks which are rarely accessed. When
> >> parity has to be written to the remaining 1,2 ...X disks ...it is
> >> batched up (bcache is feasible) and written out once in a while
> >> depending on how much write is happening. N-1 disks stay spun down and
> >> only X disks wake up periodically to get checksum written to (this
> >> would be tweaked by the user based on how up to date he needs the
> >> parity to be (tolerance of rebuilding parity in case of crash) and vs
> >> disk access for each parity write)
> >>
> >> It can't be done using any RAID6 because RAID5/6 will stripe all the
> >> data across the devices making any read access wake up all the
> >> devices. Ditto for writing to parity on every write to a single disk.
> >>
> >> The architecture being proposed is a lazy write to manage parity for
> >> individual disks which won't suffer from RAID catastrophic data loss
> >> and concurrent disk.
> >>
> >>
> >>
> >>
> >> On 30 October 2014 00:57, Ethan Wilson <ethan.wilson@shiftmail.org> wrote:
> >> > On 29/10/2014 10:25, Anshuman Aggarwal wrote:
> >> >>
> >> >> Right on most counts but please see comments below.
> >> >>
> >> >> On 29 October 2014 14:35, NeilBrown <neilb@suse.de> wrote:
> >> >>>
> >> >>> Just to be sure I understand, you would have N + X devices. Each of the
> >> >>> N
> >> >>> devices contains an independent filesystem and could be accessed directly
> >> >>> if
> >> >>> needed. Each of the X devices contains some codes so that if at most X
> >> >>> devices in total died, you would still be able to recover all of the
> >> >>> data.
> >> >>> If more than X devices failed, you would still get complete data from the
> >> >>> working devices.
> >> >>>
> >> >>> Every update would only write to the particular N device on which it is
> >> >>> relevant, and all of the X devices. So N needs to be quite a bit bigger
> >> >>> than X for the spin-down to be really worth it.
> >> >>>
> >> >>> Am I right so far?
> >> >>
> >> >> Perfectly right so far. I typically have a N to X ratio of 4 (4
> >> >> devices to 1 data) so spin down is totally worth it for data
> >> >> protection but more on that below.
> >> >>
> >> >>> For some reason the writes to X are delayed... I don't really understand
> >> >>> that part.
> >> >>
> >> >> This delay is basically designed around archival devices which are
> >> >> rarely read from and even more rarely written to. By delaying writes
> >> >> on 2 criteria ( designated cache buffer filling up or preset time
> >> >> duration from last write expiring) we can significantly reduce the
> >> >> writes on the parity device. This assumes that we are ok to lose a
> >> >> movie or two in case the parity disk is not totally up to date but are
> >> >> more interested in device longevity.
> >> >>
> >> >>> Sounds like multi-parity RAID6 with no parity rotation and
> >> >>> chunksize == devicesize
> >> >>
> >> >> RAID6 would present us with a joint device and currently only allows
> >> >> writes to that directly, yes? Any writes will be striped.
> >> >
> >> >
> >> > I am not totally sure I understand your design, but it seems to me that the
> >> > following solution could work for you:
> >> >
> >> > MD raid-6, maybe multi-parity (multi-parity not implemented yet in MD yet,
> >> > but just do a periodic scrub and 2 parities can be fine. Wake-up is not so
> >> > expensive that you can't scrub)
> >> >
> >> > Over that you put a raid1 of 2 x 4TB disks as a bcache cache device (those
> >> > two will never spin-down) in writeback mode with writeback_running=off .
> >> > This will prevent writes to backend and leave the backend array spun down.
> >> > When bcache is almost full (poll dirty_data), switch to writeback_running=on
> >> > and writethrough: it will wake up the backend raid6 array and flush all
> >> > dirty data. You can then then revert to writeback and writeback_running=off.
> >> > After this you can spin-down the backend array again.
> >> >
> >> > You also get read caching for free, which helps the backend array to stay
> >> > spun down as much as possible.
> >> >
> >> > Maybe you can modify bcache slightly so to implement an automatic switching
> >> > between the modes as described above, instead of polling the state from
> >> > outside.
> >> >
> >> > Would that work, or you are asking something different?
> >> >
> >> > EW
> >> >
> >> > --
> >> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> >> > the body of a message to majordomo@vger.kernel.org
> >> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
> > --
> >
> > piergiorgio
--
piergiorgio
^ permalink raw reply
* mdadm named array (metadata 1.2) and udev when assembling an array in initramfs
From: Stephen Kent @ 2014-11-01 18:53 UTC (permalink / raw)
To: linux-raid
I have set up a machine to use an md array as the root device. I have
created the array with metadata type 1.2, name "root," on host
"optimus" (so the full metadata name is "optimus:root").
I'm using Gentoo Linux (kernel 3.16.5-gentoo), and previously I had no
initramfs so I wrote a short one to assemble the array. My initramfs
sets the hostname and runs "mdadm -As," which correctly auto-assembles
the root device array on /dev/md/root (even without an
/etc/mdadm.conf).
Once the initramfs script is finished, init starts which then starts
udev. Now, at this point the array is already assembled, but the
/dev/md/* name is lost (and so is /run/mdadm/map) since those files
were on the initramfs' tempfs root. udev creates a /dev/mdXXX node
(ex. /dev/md127) and then runs a rule to try to figure out the
/dev/md/* name, which should be /dev/md/root. The default udev rule
calls mdadm --detail --export /dev/md127, which should print
MD_DEVNAME=root for udev, but instead it prints
MD_DEVNAME=optimus:root, and the symlink in /dev/md gets created as
/dev/md/optimus:root.
I think this is a bug in mdadm, but I'm not sure how to dig into it. I
have made sure that my main OS init sets the hostname before starting
udev, and I have tried with and without "HOMEHOST optimus" in
/etc/mdadm.conf.
The workaround I am using right now is to define the array name
manually in /etc/mdadm.conf like this:
ARRAY /dev/md/root UUID=782a671c:e646f2d5:848baad0:5668ee84
This works by avoiding automatic name detection and using the defined
name (/dev/md/root), but it really seems that mdadm should know how to
determine MD_DEVNAME correctly for an already-started array if the
hostname (or homehost) is correctly set. Is this a bug in mdadm? If
so, is this the correct place to report it or is there a bug tracking
system I should use? I am using mdadm v3.3.1.
(Please cc me on replies as I am not currently subscribed to the
linux-raid list.)
Thanks,
Stephen
^ permalink raw reply
* Re: Raid5 drive fail during grow and no backup
From: Phil Turmel @ 2014-11-02 3:22 UTC (permalink / raw)
To: Vince, linux-raid
In-Reply-To: <loom.20141031T141939-473@post.gmane.org>
On 10/31/2014 09:34 AM, Vince wrote:
> Hi,
>
> got a drive failure (bad block) during Raid5 grow (4x3TB -> 5x3TB).
> Well... i don't have a backup file :/
> Mdadm shows 1 drive as removed.
>
> All 4 'good' drives are in the same reshape pos'n.
>
> Any idea how to finish the reshape process? Or get the array back?
mdadm --stop /dev/md0
mdadm --assemble --force --verbose /dev/md0 /dev/sd[bcdf]
If that doesn't work, please show us the output.
You haven't (yet) lost your array. It's just degraded. You should
investigate why the one drive was kicked out of the array instead of
being rewritten properly (green drives?). In the meantime, assembly
with --force should give you access to the data to grab anything
critically important.
If you share the output of "smartctl -x /dev/sdX" for at least the
kicked drive, we can offer further advice.
Regards,
Phil
^ permalink raw reply
* Re: Raid1 element stuck in (S) state
From: micah @ 2014-11-02 15:45 UTC (permalink / raw)
To: NeilBrown; +Cc: micah anderson, linux-raid
In-Reply-To: <20141030094734.2451cc24@notabene.brown>
NeilBrown <neilb@suse.de> writes:
>> >> > Are there any errors in the kernel logs when you --add the device?
>> >
>> > You didn't answer this question either. Are there any messages in the
>> > kernel log: /var/log/kern.log on debian.
>> > Or in the output of "dmesg".
>>
>> The only thing I see in the log is:
>>
>> [307932.328420] mdadm: sending ioctl 1261 to a partition!
>> [307932.328425] mdadm: sending ioctl 1261 to a partition!
>> [307932.346642] mdadm: sending ioctl 1261 to a partition!
>> [307932.346648] mdadm: sending ioctl 1261 to a partition!
>> [307932.352466] mdadm: sending ioctl 1261 to a partition!
>> [307932.352468] mdadm: sending ioctl 1261 to a partition!
>> [307932.376821] mdadm: sending ioctl 1261 to a partition!
>> [307932.376824] mdadm: sending ioctl 1261 to a partition!
>> [307932.377623] mdadm: sending ioctl 1261 to a partition!
>> [307932.377630] mdadm: sending ioctl 1261 to a partition!
>> [307932.467292] md: bind<sdc1>
>> [307932.588154] RAID1 conf printout:
>> [307932.588159] --- wd:1 rd:2
>> [307932.588164] disk 0, wo:1, o:1, dev:sdc1
>> [307932.588167] disk 1, wo:0, o:1, dev:sdd1
>> [307932.588248] md: recovery of RAID array md3
>> [307932.588251] md: minimum _guaranteed_ speed: 50000 KB/sec/disk.
>> [307932.588254] md: using maximum available idle IO bandwidth (but not more than 2000000 KB/sec) for recovery.
>> [307932.588260] md: using 128k window, over a total of 976759672 blocks.
>>
>> but this is just when the device is added, after that it appears that
>> logrotation failed and I have a zero byte kern.log, and firewall spew
>> has filled up my dmesg ring.
I fixed my logging and re-added the device, and found there was a
hardware error preventing things from syncing properly. I've resolved
that error and now things are fine. Thanks for the push to look closer
there!
micah
^ permalink raw reply
* (unknown)
From: MRS GRACE MANDA @ 2014-11-02 19:54 UTC (permalink / raw)
In-Reply-To: <1480763910.146593.1414958012342.JavaMail.yahoo@jws10033.mail.ne1.yahoo.com>
[-- Attachment #1: Type: text/plain, Size: 71 bytes --]
This is Mrs Grace Manda ( Please I need your Help is Urgent).
[-- Attachment #2: Mrs Grace Manda.rtf --]
[-- Type: application/rtf, Size: 35796 bytes --]
^ permalink raw reply
* Re: mdadm named array (metadata 1.2) and udev when assembling an array in initramfs
From: NeilBrown @ 2014-11-03 1:50 UTC (permalink / raw)
To: Stephen Kent; +Cc: linux RAID
[-- Attachment #1: Type: text/plain, Size: 1343 bytes --]
Hi Stephen,
thanks for the bug report.
This patch should fix it.
NeilBrown
From 628cdf19ea35daad22e409e51c0abc7ffb19d6aa Mon Sep 17 00:00:00 2001
From: NeilBrown <neilb@suse.de>
Date: Mon, 3 Nov 2014 12:49:05 +1100
Subject: [PATCH] Rebuildmap: strip local host name from device name.
When /run/mdadm/map is being rebuilt, e.g. by "mdadm -Ir",
if the device doesn't exist in /dev, we have to choose
a name.
Currently we don't strip the hostname which is wrong if
it is the local host.
Reported-by: Stephen Kent <smkent@smkent.net>
Signed-off-by: NeilBrown <neilb@suse.de>
diff --git a/mapfile.c b/mapfile.c
index 4e7f24214b8a..41599df05f97 100644
--- a/mapfile.c
+++ b/mapfile.c
@@ -455,12 +455,19 @@ void RebuildMap(void)
sep = "";
}
}
- if (strchr(name, ':'))
- /* probably a uniquifying
+ if (strchr(name, ':')) {
+ /* Probably a uniquifying
* hostname prefix. Allow
- * without a suffix
+ * without a suffix, and strip
+ * hostname if it is us.
*/
+ if (homehost && unum == -1 &&
+ strncmp(name, homehost,
+ strlen(homehost)) == 0 &&
+ name[strlen(homehost)] == ':')
+ name += strlen(homehost)+1;
unum = -1;
+ }
while (conflict) {
if (unum >= 0)
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply related
* Re: Trying to get POLICY working
From: NeilBrown @ 2014-11-03 1:54 UTC (permalink / raw)
To: Caspar Smit; +Cc: linux-raid
In-Reply-To: <20141101112001.344e5f20@notabene.brown>
[-- Attachment #1: Type: text/plain, Size: 3106 bytes --]
On Sat, 1 Nov 2014 11:20:01 +1100 NeilBrown <neilb@suse.de> wrote:
> On Fri, 31 Oct 2014 16:19:04 +0100 Caspar Smit <c.smit@truebit.nl> wrote:
>
> > Hi all,
> >
> > I'm trying to get the POLICY framework of mdadm working but I can't seem to.
> >
> > As i understand in the man page of mdadm the Incremental and POLICY
> > directives could allow adding a new disk without MD superblock as
> > spare to an already active array:
> >
> > "Note that mdadm will normally only add devices to an array which were
> > previously working (active or spare) parts of that array. The support
> > for automatic inclusion of a new drive as a spare in some array
> > requires a configuration through POLICY in config file."
> >
> > Furthermore:
> >
> > "If no md metadata is found, the device may be still added to an array
> > as a spare if POLICY allows."
> >
> >
> > To get the basics working I created a system with 3 disks /dev/sdb,
> > /dev/sdc and /dev/sdd
> >
> > Created a RAID5 with one missing disk:
> >
> > mdadm -C /dev/md0 -l 5 -n 3 /dev/sd[b-c] missing
> >
> > I set the POLICY in mdadm.conf to:
> >
> > POLICY action=force-spare
> >
> > This should add any device (passed through mdadm --incremental) as
> > spare no matter what (Am i correct?)
>
> That is the theory, yes.
>
> >
> > Now when I do:
> >
> > #mdadm --incremental /dev/sdd
> > mdadm: no RAID superblock on /dev/sdd.
>
> The message suggests that 'guess_super' found something on the device, but
> it didn't turn out to be something useful.... not very helpful I know.
>
> What does "mdadm --examine /dev/sdd" report?
> I suspect there is a partition table and that is causing the confusion.
> Try removing the partition table (dd /dev/zero to the device for a few K).
> Then try again.
>
> Probably need a fix like:
>
> diff --git a/Incremental.c b/Incremental.c
> index c9372587f518..3156190c4603 100644
> --- a/Incremental.c
> +++ b/Incremental.c
> @@ -196,7 +196,7 @@ int Incremental(struct mddev_dev *devlist, struct context *c,
> policy = disk_policy(&dinfo);
> have_target = policy_check_path(&dinfo, &target_array);
>
> - if (st == NULL && (st = guess_super(dfd)) == NULL) {
> + if (st == NULL && (st = guess_super_type(dfd, guess_array)) == NULL) {
> if (c->verbose >= 0)
> pr_err("no recognisable superblock on %s.\n",
> devname);
>
>
> and probably should improve the error messages...
>
> Thanks for the report. Please let me know if that works, and what other
> difficulties you hit.
Actually, don't bother. I must have been asleep.
Your problem is that you haven't defined a 'domain'.
A new spare needs to be assigned to a 'domain', and it will be attached to
any array in the same domain, as needed.
You can give all devices the domain "default" with
POLICY domain=default
The domain of an array is inherited from the member devices, or can be set
with "spare-group=" in mdadm.conf.
So
POLICY domain=default action=force-spare
should make it work for you.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* Re: mdadm named array (metadata 1.2) and udev when assembling an array in initramfs
From: Stephen Kent @ 2014-11-03 4:16 UTC (permalink / raw)
To: NeilBrown; +Cc: linux RAID
In-Reply-To: <20141103125020.19308fd5@notabene.brown>
On Sun, Nov 2, 2014 at 5:50 PM, NeilBrown <neilb@suse.de> wrote:
>
> Hi Stephen,
> thanks for the bug report.
> This patch should fix it.
>
> NeilBrown
>
> From 628cdf19ea35daad22e409e51c0abc7ffb19d6aa Mon Sep 17 00:00:00 2001
> From: NeilBrown <neilb@suse.de>
> Date: Mon, 3 Nov 2014 12:49:05 +1100
> Subject: [PATCH] Rebuildmap: strip local host name from device name.
>
> When /run/mdadm/map is being rebuilt, e.g. by "mdadm -Ir",
> if the device doesn't exist in /dev, we have to choose
> a name.
> Currently we don't strip the hostname which is wrong if
> it is the local host.
>
> Reported-by: Stephen Kent <smkent@smkent.net>
> Signed-off-by: NeilBrown <neilb@suse.de>
>
> diff --git a/mapfile.c b/mapfile.c
> index 4e7f24214b8a..41599df05f97 100644
> --- a/mapfile.c
> +++ b/mapfile.c
> @@ -455,12 +455,19 @@ void RebuildMap(void)
> sep = "";
> }
> }
> - if (strchr(name, ':'))
> - /* probably a uniquifying
> + if (strchr(name, ':')) {
> + /* Probably a uniquifying
> * hostname prefix. Allow
> - * without a suffix
> + * without a suffix, and strip
> + * hostname if it is us.
> */
> + if (homehost && unum == -1 &&
> + strncmp(name, homehost,
> + strlen(homehost)) == 0 &&
> + name[strlen(homehost)] == ':')
> + name += strlen(homehost)+1;
> unum = -1;
> + }
>
> while (conflict) {
> if (unum >= 0)
Your patch works for me! Thanks so much for taking a look.
Stephen
^ permalink raw reply
* Re: mdadm named array (metadata 1.2) and udev when assembling an array in initramfs
From: NeilBrown @ 2014-11-03 5:43 UTC (permalink / raw)
To: Stephen Kent; +Cc: linux RAID
In-Reply-To: <CAC45b7=H2kFg2VdFq2xdaQ=z4MebV3QC-aA1X5TO0ra=749puA@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2681 bytes --]
On Sun, 2 Nov 2014 20:16:34 -0800 Stephen Kent <smkent@smkent.net> wrote:
> On Sun, Nov 2, 2014 at 5:50 PM, NeilBrown <neilb@suse.de> wrote:
> >
> > Hi Stephen,
> > thanks for the bug report.
> > This patch should fix it.
> >
> > NeilBrown
> >
> > From 628cdf19ea35daad22e409e51c0abc7ffb19d6aa Mon Sep 17 00:00:00 2001
> > From: NeilBrown <neilb@suse.de>
> > Date: Mon, 3 Nov 2014 12:49:05 +1100
> > Subject: [PATCH] Rebuildmap: strip local host name from device name.
> >
> > When /run/mdadm/map is being rebuilt, e.g. by "mdadm -Ir",
> > if the device doesn't exist in /dev, we have to choose
> > a name.
> > Currently we don't strip the hostname which is wrong if
> > it is the local host.
> >
> > Reported-by: Stephen Kent <smkent@smkent.net>
> > Signed-off-by: NeilBrown <neilb@suse.de>
> >
> > diff --git a/mapfile.c b/mapfile.c
> > index 4e7f24214b8a..41599df05f97 100644
> > --- a/mapfile.c
> > +++ b/mapfile.c
> > @@ -455,12 +455,19 @@ void RebuildMap(void)
> > sep = "";
> > }
> > }
> > - if (strchr(name, ':'))
> > - /* probably a uniquifying
> > + if (strchr(name, ':')) {
> > + /* Probably a uniquifying
> > * hostname prefix. Allow
> > - * without a suffix
> > + * without a suffix, and strip
> > + * hostname if it is us.
> > */
> > + if (homehost && unum == -1 &&
> > + strncmp(name, homehost,
> > + strlen(homehost)) == 0 &&
> > + name[strlen(homehost)] == ':')
> > + name += strlen(homehost)+1;
> > unum = -1;
> > + }
> >
> > while (conflict) {
> > if (unum >= 0)
>
>
> Your patch works for me! Thanks so much for taking a look.
>
Great - thanks for the confirmation. It will be in the next release.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* Re: Split RAID: Proposal for archival RAID using incremental batch checksum
From: NeilBrown @ 2014-11-03 5:52 UTC (permalink / raw)
To: Anshuman Aggarwal; +Cc: linux-raid
In-Reply-To: <CAK-d5dYm7tEY855w1CdkvPz22ukkg1DwGz_mQ_9-JRV0M=O6Rw@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 4572 bytes --]
On Thu, 30 Oct 2014 20:30:40 +0530 Anshuman Aggarwal
<anshuman.aggarwal@gmail.com> wrote:
> Would chunksize==disksize work? Wouldn't that lead to the entire
> parity be invalidated for any write to any of the disks (assuming md
> operates at a chunk level)...also please see my reply below
Operating at a chunk level would be a very poor design choice. md/raid5
operates in units of 1 page (4K).
>
> On 29 October 2014 14:55, Anshuman Aggarwal <anshuman.aggarwal@gmail.com> wrote:
> > Right on most counts but please see comments below.
> >
> > On 29 October 2014 14:35, NeilBrown <neilb@suse.de> wrote:
> >> Just to be sure I understand, you would have N + X devices. Each of the N
> >> devices contains an independent filesystem and could be accessed directly if
> >> needed. Each of the X devices contains some codes so that if at most X
> >> devices in total died, you would still be able to recover all of the data.
> >> If more than X devices failed, you would still get complete data from the
> >> working devices.
> >>
> >> Every update would only write to the particular N device on which it is
> >> relevant, and all of the X devices. So N needs to be quite a bit bigger
> >> than X for the spin-down to be really worth it.
> >>
> >> Am I right so far?
> >
> > Perfectly right so far. I typically have a N to X ratio of 4 (4
> > devices to 1 data) so spin down is totally worth it for data
> > protection but more on that below.
> >
> >>
> >> For some reason the writes to X are delayed... I don't really understand
> >> that part.
> >
> > This delay is basically designed around archival devices which are
> > rarely read from and even more rarely written to. By delaying writes
> > on 2 criteria ( designated cache buffer filling up or preset time
> > duration from last write expiring) we can significantly reduce the
> > writes on the parity device. This assumes that we are ok to lose a
> > movie or two in case the parity disk is not totally up to date but are
> > more interested in device longevity.
> >
> >>
> >> Sounds like multi-parity RAID6 with no parity rotation and
> >> chunksize == devicesize
> > RAID6 would present us with a joint device and currently only allows
> > writes to that directly, yes? Any writes will be striped.
If the chunksize equals the device size, then you need a very large write for
it to be striped.
> > In any case would md raid allow the underlying device to be written to
> > directly? Also how would it know that the device has been written to
> > and hence parity has to be updated? What about the superblock which
> > the FS would not know about?
No, you wouldn't write to the underlying device. You would carefully
partition the RAID5 so each partition aligns exactly with an underlying
device. Then write to the partition.
> >
> > Also except for the delayed checksum writing part which would be
> > significant if one of the objectives is to reduce the amount of
> > writes. Can we delay that in the code currently for RAID6? I
> > understand the objective of RAID6 is to ensure data recovery and we
> > are looking at a compromise in this case.
"simple matter of programming"
Of course there would be a limit to how much data can be buffered in memory
before it has to be flushed out.
If you are mostly storing movies, then they are probably too large to
buffer. Why not just write them out straight away?
NeilBrown
> >
> > If feasible, this can be an enhancement to MD RAID as well where N
> > devices are presented instead of a single joint device in case of
> > raid6 (maybe the multi part device can be individual disks?)
> >
> > It will certainly solve my problem of where to store the metadata. I
> > was currently hoping to just store it as a configuration file to be
> > read by the initramfs since in this case worst case scenario the
> > checksum goes out of sync and is rebuilt from scratch.
> >
> >>
> >> I wouldn't use device-mapper myself, but you are unlikely to get an entirely
> >> impartial opinion from me on that topic.
> >
> > I haven't hacked around the kernel internals much so far so will have
> > to dig out that history. I will welcome any particular links/mail
> > threads I should look at for guidance (with both yours and opposing
> > points of view)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* Re: Trying to get POLICY working
From: Caspar Smit @ 2014-11-03 9:43 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20141103125416.629a4810@notabene.brown>
Hi Neil,
Actually BOTH your answers were correct, thank you for that.
1) Your hunge was correct as my disk contained a partition table (in
my case an msdos label) and was not added with the error in my first
mail:
mdadm: no RAID superblock on /dev/sdd.
mdadm -E /dev/sdd shows:
/dev/sdd:
MBR Magic : aa55
So it finds 'something' but clearly unusable to mdadm.
Wiping the partition table and trying again resulted in a different
error message:
mdadm: no recognisable superblock on /dev/sdd.
Which is better but still the disk was not added to the array.
2) To make it work i also needed the domain=default in the POLICY setting.
It still gave me the:
mdadm: no recognisable superblock on /dev/sdd.
But now the disk got added to the array and started rebuilding.
Note: ONLY setting the domain=default in POLICY without clearing the
partition table results in:
mdadm: no RAID superblock on /dev/sdd. and the disk will not be added
so BOTH measures were needed.
Note2: I didn't need the spare-group directive so I think
domain=default is a special case were all disks and arrays are placed
in the same domain.
Furthermore i found out something which i think should not happen
(bug?) or maybe i am wrong:
With a working clean array:
# more /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd[3] sdc[1] sdb[0]
203776 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
# mdadm --fail /dev/md0 /dev/sdd
mdadm: set /dev/sdd faulty in /dev/md0
# mdadm --remove /dev/md0 /dev/sdd
mdadm: hot removed /dev/sdd from /dev/md0
# mdadm --incremental /dev/sdd
mdadm: failed to add /dev/sdd to /dev/md/0: Invalid argument.
So when it actually finds a device with an MD superblock it doesn't
add it, is this expected behavior as the disk was failed (so probably
not a good idea to add it back) or is this a bug?
Kind regards,
Caspar
2014-11-03 2:54 GMT+01:00 NeilBrown <neilb@suse.de>:
> On Sat, 1 Nov 2014 11:20:01 +1100 NeilBrown <neilb@suse.de> wrote:
>
>> On Fri, 31 Oct 2014 16:19:04 +0100 Caspar Smit <c.smit@truebit.nl> wrote:
>>
>> > Hi all,
>> >
>> > I'm trying to get the POLICY framework of mdadm working but I can't seem to.
>> >
>> > As i understand in the man page of mdadm the Incremental and POLICY
>> > directives could allow adding a new disk without MD superblock as
>> > spare to an already active array:
>> >
>> > "Note that mdadm will normally only add devices to an array which were
>> > previously working (active or spare) parts of that array. The support
>> > for automatic inclusion of a new drive as a spare in some array
>> > requires a configuration through POLICY in config file."
>> >
>> > Furthermore:
>> >
>> > "If no md metadata is found, the device may be still added to an array
>> > as a spare if POLICY allows."
>> >
>> >
>> > To get the basics working I created a system with 3 disks /dev/sdb,
>> > /dev/sdc and /dev/sdd
>> >
>> > Created a RAID5 with one missing disk:
>> >
>> > mdadm -C /dev/md0 -l 5 -n 3 /dev/sd[b-c] missing
>> >
>> > I set the POLICY in mdadm.conf to:
>> >
>> > POLICY action=force-spare
>> >
>> > This should add any device (passed through mdadm --incremental) as
>> > spare no matter what (Am i correct?)
>>
>> That is the theory, yes.
>>
>> >
>> > Now when I do:
>> >
>> > #mdadm --incremental /dev/sdd
>> > mdadm: no RAID superblock on /dev/sdd.
>>
>> The message suggests that 'guess_super' found something on the device, but
>> it didn't turn out to be something useful.... not very helpful I know.
>>
>> What does "mdadm --examine /dev/sdd" report?
>> I suspect there is a partition table and that is causing the confusion.
>> Try removing the partition table (dd /dev/zero to the device for a few K).
>> Then try again.
>>
>> Probably need a fix like:
>>
>> diff --git a/Incremental.c b/Incremental.c
>> index c9372587f518..3156190c4603 100644
>> --- a/Incremental.c
>> +++ b/Incremental.c
>> @@ -196,7 +196,7 @@ int Incremental(struct mddev_dev *devlist, struct context *c,
>> policy = disk_policy(&dinfo);
>> have_target = policy_check_path(&dinfo, &target_array);
>>
>> - if (st == NULL && (st = guess_super(dfd)) == NULL) {
>> + if (st == NULL && (st = guess_super_type(dfd, guess_array)) == NULL) {
>> if (c->verbose >= 0)
>> pr_err("no recognisable superblock on %s.\n",
>> devname);
>>
>>
>> and probably should improve the error messages...
>>
>> Thanks for the report. Please let me know if that works, and what other
>> difficulties you hit.
>
> Actually, don't bother. I must have been asleep.
>
> Your problem is that you haven't defined a 'domain'.
> A new spare needs to be assigned to a 'domain', and it will be attached to
> any array in the same domain, as needed.
>
> You can give all devices the domain "default" with
>
> POLICY domain=default
>
> The domain of an array is inherited from the member devices, or can be set
> with "spare-group=" in mdadm.conf.
>
> So
>
> POLICY domain=default action=force-spare
>
> should make it work for you.
>
> NeilBrown
^ permalink raw reply
* Re: Raid5 drive fail during grow and no backup
From: Vince @ 2014-11-03 14:45 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <5455A35C.2060000@turmel.org>
Phil Turmel <philip <at> turmel.org> writes:
>
> On 10/31/2014 09:34 AM, Vince wrote:
> > Hi,
> >
> > got a drive failure (bad block) during Raid5 grow (4x3TB -> 5x3TB).
> > Well... i don't have a backup file :/
> > Mdadm shows 1 drive as removed.
> >
> > All 4 'good' drives are in the same reshape pos'n.
> >
> > Any idea how to finish the reshape process? Or get the array back?
>
> mdadm --stop /dev/md0
> mdadm --assemble --force --verbose /dev/md0 /dev/sd[bcdf]
>
> If that doesn't work, please show us the output.
>
> You haven't (yet) lost your array. It's just degraded. You should
> investigate why the one drive was kicked out of the array instead of
> being rewritten properly (green drives?). In the meantime, assembly
> with --force should give you access to the data to grab anything
> critically important.
>
> If you share the output of "smartctl -x /dev/sdX" for at least the
> kicked drive, we can offer further advice.
>
> Regards,
>
> Phil
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo <at> vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
Hi Phil,
thx for your reply.
Already have the raid clean and up.
My drive was kicked due to read errors (bad sectors).
I fixed the bad sectors with hdparm --write-sector $bad_sector /dev/sdx
After some tries
mdadm --assemble --force --verbose /dev/md0 /dev/sd[bcdf]
works so far.
I was able to encrypt the drive and all logical volumes had beed detected
correct.
But i was unable to mount any lv (i guess due to an filesystem problem, but
i won't run es2fscheck during broken reshape)
So i did a backup and removed the superblock on the broken disk and added it
as spare to /dev/md0 (mdadm --add)
Now i had 4 Disk in sync, 1 removed and 1 spare
To restart the reshape i did mdadm --mdadm --readwrite /dev/md0
Well.. i had a backup of my most important files and in that situation i was
like... ok if all is lost now... i'll change a lot in future :)
Reshape restarted at ~80% (cat /proc/mdstat) but the funny thing was, that
all 4 drives only did write actions. No reading on the drives... I don't
know what happens there, but i let it go
After reshape was done, mdadm grabs the spare drive and started resync.
After resync was done, i did e2fsck -f on logical volumes
Finally i was able to mount all lv's without any data lost.
^ permalink raw reply
* Fake RAID 10 on ASRock Z97 Extreme3 with Slackware 14.1
From: gornea @ 2014-11-03 16:47 UTC (permalink / raw)
To: linux-raid
Hello,
I hope I'm on the right email list. I'm facing a problem during Slackware 14.1 installation. I have a ASRock motherboard (Z97 Extreme3) with Intel Rapid Storage Technology 13. I have 4 identical new 4-TBy WD hard drives. I configured the option ROM for RAID 10. When booting with Slackware 14.1 CD md finds the array but it always mounts it read only. I tried to work on it to make writable but I was not successful. I have tried to boot with the latest Ubuntu Server CD and it works well. I have also tried to delete the RAID 10 in the option ROM and then create it with mdadm after booting with Slackware 14.1 and I always produces a read only volume. I get no errors or warnings and in general everything seems fine except that I can not write on it (for example I can not create partitions). The
md version with Slackware 14.1 boot disk is 3.2.6 and the kernel is 3.10.17. So I'm starting to wonder if actually there is a know problem with that md/kernel? Or maybe there is a problem with the way I use it, but frankly I run out of ideas how to proceed further. This is the output of /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md126 : active (read-only) raid10 sda[3] sdb[2] sdc[1] sdd[0]
7814032384 blocks super external:/md127/0 64K chunks 2 near-copies [4/4] [UUUU]
md127 : inactive sda[3](S) sdb[2](S) sdc[1](S) sdd[0](S)
9040 blocks super external:imsm
unused devices: <none>
Thanks a lot for any suggestion about how to proceed! I would like to install the fake RAID if it is not impossible with Slackware 14.1. I know that eventually it is possible to install a software RAID but I would like to understand why the Intel fake RAID is not working with Slackware 14.1.
Best regards,
Razvan Gornea
^ permalink raw reply
* Re: Split RAID: Proposal for archival RAID using incremental batch checksum
From: Piergiorgio Sartor @ 2014-11-03 18:04 UTC (permalink / raw)
To: NeilBrown; +Cc: Anshuman Aggarwal, linux-raid
In-Reply-To: <20141103165217.3bfd3d3e@notabene.brown>
On Mon, Nov 03, 2014 at 04:52:17PM +1100, NeilBrown wrote:
[...]
> "simple matter of programming"
> Of course there would be a limit to how much data can be buffered in memory
> before it has to be flushed out.
> If you are mostly storing movies, then they are probably too large to
> buffer. Why not just write them out straight away?
One scenario I can envision is the following.
You've a bunch of HDDs in RAID-5/6, which are
almost always in standby (spin down).
Together, you've 2 SSDs in RAID-10.
All the write (and read, if possible) operations
are done towards the SSDs.
When the SSD RAID is X% full, the RAID-5/6 is
activated and the data *moved* (maybe copied, with
proper cache policy) there.
In case of reading (a large file), the RAID-5/6 is
activated, the file copied to the SSD RAID, and,
when finished, the HDDs put in standby again.
Of course, this is *not* a block device protocol,
it is a filesystem one.
It is the FS that must handle the caching, because
only the FS can know the file size, for example.
bye,
--
piergiorgio
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox