Re: Please help me save my data

Re: Please help me save my data

[martin.kihlgren]

> On 9/8/06, martin.kihlgren@adocca.com <martin.kihlgren@adocca.com> wrote:
>> So, what I want to do is:
>>
>>  * Mark the synced spare drive as working and in position 1
>>  * Assemble the array without the unsynced spare and check if this
>>    provides consistent data
>>  * If it didnt, I want to mark the synced spare as working and in
>>    position 3, and try the same thing again
>>  * When I have it working, I just want to add the unsynced spare and
>>    let it sync normally
>>  * Then I will create a write-intent bitmap to avoid the dangerously
>>    long sync times, and also buy a new USB controller hoping that it
will solve my problems
>
> You can recreate the raid array with 1 missing disk, like this:
>
> mdadm -C /dev/md1 /dev/sdn1 /dev/sdX1 /dev/sdn1 /dev/sdn1 missing
>
> The ordering is relevant, raid-disks 0,1,2,3,4 or so. beware, you have
to have block size and symmetry correct, so better backup mdadm
> --examine and --detail output beforehand.
>
> This create op causes no sync (no danger data overwrites), as there is
still the one drive missing, but raid-superblocks are rewritten.
>
> (On a sidenote, i'm uncertain if a bitmap helps in the case of
> single-device remove-add cycle? I thought it was only for crashes, at
least for now..)
>

Thanks for your help! Your advice is good, and I will use it next time.

This time I found an old USB memory stick to experiment with, and managed
to do pretty much the same thing with:

mdadm -C -l 5 -n 5 -f -e 1.2 --assume-clean /dev/md1 /dev....
mdadm -f /dev/md1 /dev/borken_device

And yes, the ordering was very relevant. An xfs_check showed me which
ordering was correct however. But I still have a problem with not easily
knowing what physical drive is what raid device, since USB devices get
ordered in some random way.

And no, the bitmap didnt help in this case (it has happened again but with
only one disk)... I wish my USB worked better, but I guess its a question
of time and kernel development.

Thanks anyhow!
regards,
//Martin

Re: Please help me save my data

[Patrick Hoover]

On 9/11/06, martin.kihlgren@adocca.com <martin.kihlgren@adocca.com> wrote:
> > On 9/8/06, martin.kihlgren@adocca.com <martin.kihlgren@adocca.com> wrote:
> >> So, what I want to do is:
> >>
> >>  * Mark the synced spare drive as working and in position 1
> >>  * Assemble the array without the unsynced spare and check if this
> >>    provides consistent data
> >>  * If it didnt, I want to mark the synced spare as working and in
> >>    position 3, and try the same thing again
> >>  * When I have it working, I just want to add the unsynced spare and
> >>    let it sync normally
> >>  * Then I will create a write-intent bitmap to avoid the dangerously
> >>    long sync times, and also buy a new USB controller hoping that it
> will solve my problems
> >
> > You can recreate the raid array with 1 missing disk, like this:
> >
> > mdadm -C /dev/md1 /dev/sdn1 /dev/sdX1 /dev/sdn1 /dev/sdn1 missing
> >
> > The ordering is relevant, raid-disks 0,1,2,3,4 or so. beware, you have
> to have block size and symmetry correct, so better backup mdadm
> > --examine and --detail output beforehand.
> >
> > This create op causes no sync (no danger data overwrites), as there is
> still the one drive missing, but raid-superblocks are rewritten.
> >
> > (On a sidenote, i'm uncertain if a bitmap helps in the case of
> > single-device remove-add cycle? I thought it was only for crashes, at
> least for now..)
> >
>
> Thanks for your help! Your advice is good, and I will use it next time.
>
> This time I found an old USB memory stick to experiment with, and managed
> to do pretty much the same thing with:
>
> mdadm -C -l 5 -n 5 -f -e 1.2 --assume-clean /dev/md1 /dev....
> mdadm -f /dev/md1 /dev/borken_device
>
> And yes, the ordering was very relevant. An xfs_check showed me which
> ordering was correct however. But I still have a problem with not easily
> knowing what physical drive is what raid device, since USB devices get
> ordered in some random way.
>
> And no, the bitmap didnt help in this case (it has happened again but with
> only one disk)... I wish my USB worked better, but I guess its a question
> of time and kernel development.
>
> Thanks anyhow!
> regards,
> //Martin
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>


Is anyone else having issues with USB interfaced disks to implement
RAID? Any thoughts on Pros / Cons for doing this?

Re: Please help me save my data

[Molle Bestefich]

Patrick Hoover wrote:
> Is anyone else having issues with USB interfaced disks to implement
> RAID? Any thoughts on Pros / Cons for doing this?

Sounds like a very good stress test for MD.

I often find servers completely hung when a disk fails, this usually
happens in the IDE layer.
If using USB disks circumvents the IDE layer enough, using USB disks
might get rid of these hangs.  Would be nice at least.  Maybe I'm just
dreaming.

For end users, USB might remove the need to take special care of
cooling in your cabinet.
OTOH, most USB disk enclosures have horrible thermal properties.

USB would make it a lot easier to add new disks (beyond your cabinet's
capacity) and to remove old disks when/if they're no longer needed.
Users might run into a bandwidth issue at some point..

Re: Please help me save my data

[martin.kihlgren]

> Patrick Hoover wrote:
>> Is anyone else having issues with USB interfaced disks to implement
>> RAID? Any thoughts on Pros / Cons for doing this?
>
> Sounds like a very good stress test for MD.
>
> I often find servers completely hung when a disk fails, this usually
> happens in the IDE layer.
> If using USB disks circumvents the IDE layer enough, using USB disks
> might get rid of these hangs.  Would be nice at least.  Maybe I'm just
> dreaming.
>
> For end users, USB might remove the need to take special care of
> cooling in your cabinet.
> OTOH, most USB disk enclosures have horrible thermal properties.
>
> USB would make it a lot easier to add new disks (beyond your cabinet's
> capacity) and to remove old disks when/if they're no longer needed.
> Users might run into a bandwidth issue at some point..

After I got rid of a crappy USB hub the catastrophic resets stopped. And
after I bought a separate PCI USB card the non-catastrophic resets have
almost stopped as well.

So now the system works as well as I hoped when I planned it!

And no, nothing hangs except the disk access to the device in question
when a disk fails.

My Seagate disks DO generate too much heat if I stack them on top of each
other, which their form factor suggests they would accept. If I put them a
bit more spacy though it works perfectly. And there ARE enclosures with
separate fans.

I have 10 external USB disks now - I got rid of my internal ones which
were too old and failing, and I plan on continuing to add on to my
external array. My RAID5 + LVM + dm_crypt + XFS setup allows for a very
extendable system.

And as long as I treat the entire disk set as one device, the bandwidth
will not be an issue since I will never demand more bandwidth from the
entire array than from a single USB drive anyway.

//Martin

USB disks for RAID storage (was Re: Please help me save my data)

[Daniel Pittman]

"Patrick Hoover" <phoover.eml@gmail.com> writes:

> Is anyone else having issues with USB interfaced disks to implement
> RAID? Any thoughts on Pros / Cons for doing this?  

I have done, as a trial.  USB disk support in recent 2.6 based
distributions was quite stable and reliable, and I had no significant
problems with RAID arrays -- including constructing arrays at boot time.

However, USB disk performance was not great; I wouldn't want to run more
than one, or possibly two, disks on a single USB controller.  Anything
more than that and you really see performance drop off.

Also, the USB transaction latency seems a bit greater than IDE or SATA,
so there are longer delays in data getting in or out of the system --
which add up, and make workloads with lots of fsync and small files
painful.


For bulk storage of large data with low performance requirements
chaining a whole bunch of USB disks off a single controller would be
acceptable, but otherwise I wouldn't bother.


I didn't investigate IEEE-1394 connected disks, but I anticipate them to
have similar issues with controller latency and bandwidth.  Tests with
two disks suggest a similar set of issues.[1]

Regards,
        Daniel

Footnotes: 
[1]  Also, Linux optimizes the 1394 bus less well than it could,
     so performance is hurt by that a bit.  

-- 
Digital Infrastructure Solutions -- making IT simple, stable and secure
Phone: 0401 155 707        email: contact@digital-infrastructure.com.au
                 http://digital-infrastructure.com.au/

Re: USB disks for RAID storage (was Re: Please help me save my data)

[Molle Bestefich]

Martin Kihlgren writes:
> And no, nothing hangs except the disk access to the device
> in question when a disk fails.

Sounds good!  +1 for USB...

> My Seagate disks DO generate too much heat if I stack them on top
> of each other, which their form factor suggests they would accept.

Starts to take up a lot of space if you need to lay them out like that.

(Just for reference, I've had external USB thingys fail with just two
 drives stacked.  They both failed.)

> My RAID5 + LVM + dm_crypt + XFS setup allows for a very extendable
> system.

That does give you a cool feature set :-).

With LVM and dm_crypt in there it does sound like you're running with
beta quality software to my ears, however.  If it works, great.

> And as long as I treat the entire disk set as one device, the
> bandwidth will not be an issue since I will never demand more
> bandwidth from the entire array than from a single USB drive
> anyway.

Fair enough.
It would be cool to get the extra bandwidth though.

One solution is to use external SATA ("eSATA") enclosures
instead of USB enclosures.  That would both raise bandwidth, and
fix the transaction latency issue mentioned by Daniel Pittman.

There's a couple of single-disk enclosures out there that allows
to connect disks via either eSATA or USB2.  None of them seems to
come with cooling, though :-/.

Re: USB disks for RAID storage (was Re: Please help me save my data)

[Patrick Hoover]

With USB interfaced disks, it appears that you lose access to the
SMART capabilities built into the disks. My understanding is that
there is no way to map the SMART transactions onto USB - it would be
great if I were mistaken here. There was another thread here on
testing arrays a short while ago. My take on that was using SMART
really was the way to go. However, in this case, where SMART doesn't
appear to work, what are the best options for monitoring disk
integrity / degradation?



-Pat


On 9/18/06, Molle Bestefich <molle.bestefich@gmail.com> wrote:
> Martin Kihlgren writes:
> > And no, nothing hangs except the disk access to the device
> > in question when a disk fails.
>
> Sounds good!  +1 for USB...
>
> > My Seagate disks DO generate too much heat if I stack them on top
> > of each other, which their form factor suggests they would accept.
>
> Starts to take up a lot of space if you need to lay them out like that.
>
> (Just for reference, I've had external USB thingys fail with just two
>  drives stacked.  They both failed.)
>
> > My RAID5 + LVM + dm_crypt + XFS setup allows for a very extendable
> > system.
>
> That does give you a cool feature set :-).
>
> With LVM and dm_crypt in there it does sound like you're running with
> beta quality software to my ears, however.  If it works, great.
>
> > And as long as I treat the entire disk set as one device, the
> > bandwidth will not be an issue since I will never demand more
> > bandwidth from the entire array than from a single USB drive
> > anyway.
>
> Fair enough.
> It would be cool to get the extra bandwidth though.
>
> One solution is to use external SATA ("eSATA") enclosures
> instead of USB enclosures.  That would both raise bandwidth, and
> fix the transaction latency issue mentioned by Daniel Pittman.
>
> There's a couple of single-disk enclosures out there that allows
> to connect disks via either eSATA or USB2.  None of them seems to
> come with cooling, though :-/.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: USB disks for RAID storage

[Daniel Pittman]

"Patrick Hoover" <phoover.eml@gmail.com> writes:

Top posting makes it hard to keep meaningful context in the discussion.
It would be nice if you would avoid that in future.

[... RAID on USB attached disks ...]

> With USB interfaced disks, it appears that you lose access to the
> SMART capabilities built into the disks. My understanding is that
> there is no way to map the SMART transactions onto USB - it would be
> great if I were mistaken here. 

You are mistaken: it is perfectly possible for a USB disk enclosure to
translate SMART requests to the disk.  

However you are not very much mistaken: none of them actually /do/ that
translation.  So, in practice you do give up on SMART notification of
pending issues.

> There was another thread here on testing arrays a short while ago. My
> take on that was using SMART really was the way to go. However, in
> this case, where SMART doesn't appear to work, what are the best
> options for monitoring disk integrity / degradation?

The case of "where SMART doesn't appear to work" is "if (1)" -- SMART
does not always alert you to a disk failure before it shows up.  I say
this from the experience of servers with extensive SMART monitoring that
have failed disks without warning.

So, don't assume that SMART is going to catch every fault for you, or
that you really lose that much without it.[1]

Regards,
        Daniel

Footnotes: 
[1]  I have had one disk failure noted by SMART first, dozens of
     failures noted by MD kicking out a disk, and one case where I
     swapped out the disk as SMART showed increasing error counts over
     the last couple of years.

-- 
Digital Infrastructure Solutions -- making IT simple, stable and secure
Phone: 0401 155 707        email: contact@digital-infrastructure.com.au
                 http://digital-infrastructure.com.au/

Re: USB disks for RAID storage (was Re: Please help me save my data)

[Molle Bestefich]

Patrick Hoover wrote:
> However, in this case, where SMART doesn't appear to work,
> what are the best options for monitoring disk integrity / degradation?

echo check > /sys/block/mdX/md/sync_action

AFAIK, using the above, MD will "repair" bad blocks and kick disks if it fails.
A failed repair means the spare area is full and the disk desperately
needs a replacement.

Having a spare disk in the array is probably a good idea too, to
minimize downtime, especially if you're not able to get to the machine
all the time.  I don't know if MD checks spares for read errors,
though that would definitely be useful.

Monitoring for kicked disks can be done by running mdadm in daemon
mode with --monitor, having it send you an email when such an event
occurs.

Something that I've found useful is to "dd if=/dev/hdX of=/dev/null
bs=1M count=512".  If there is a problem in the IDE driver, or the
cable is loose, or there is a controller problem, the disk might
respond, but will fail as soon as you read a large amount of data.
I've also seen disks succeed to read some amount of data, but at a
significantly lower rate than it should.  Monitoring the read rates of
each disk can be helpful (at least it is to me) to diagnose such
problems.