USB-to-SATA and RAID

USB-to-SATA and RAID

[Linus Lüssing]

Hi,

By chance I found this statement in the Linux RAID Wiki:

"Do NOT use [USB devices] as part of your array. They have
 a timeout/disconnect mechanism which interacts very
 badly with the raid code." [0]

Could someone elaborate further on this? Does "interacts very
badly" mean that there will be potential data loss / data
corruption? Or does "very badly" just mean short, temporary
performance issues / lags upon a disk failure?

Regards, Linus

[0]: https://raid.wiki.kernel.org/index.php/Choosing_your_hardware,_and_what_is_a_device%3F#USB_devices

Re: USB-to-SATA and RAID

[Wol's lists]

On 23/01/18 20:16, Linus Lüssing wrote:
> Hi,
> 
> By chance I found this statement in the Linux RAID Wiki:
> 
> "Do NOT use [USB devices] as part of your array. They have
>   a timeout/disconnect mechanism which interacts very
>   badly with the raid code." [0]

That was me that wrote that on the wiki. However, I'm just editing stuff 
so I don't actually have any experience of it.
> 
> Could someone elaborate further on this? Does "interacts very
> badly" mean that there will be potential data loss / data
> corruption? Or does "very badly" just mean short, temporary
> performance issues / lags upon a disk failure?
> 
The wiki also mentions the timeout problem. As far as I can make out - 
and bear in mind as I said above I don't have personal experience - it 
seems to me from what I've picked up that USB suffers this problem in 
spades.

In other words, what I *think* happens, from what I've picked up, is 
that the USB bus times out. And when the raid comes along to use the 
disk, the USB bus is no longer responding ... BOOM!

> Regards, Linus
> 
I hope that's helpful, and maybe people who know more will chime in, but 
that's it as I understand it.

Cheers,
Wol

Re: USB-to-SATA and RAID

[Vojtěch Kletečka]

Hi,

I have only a little technical insight into this topic, however, I
have some (terrible) personal experience with USB raid and I would
point out another aspect.
Although USBs are good for occasional connections of simple drives
they have quite a strong tendency to wear out. This becomes a problem
especially if you plan to connect raid often and means that wires
might not connect or lose connection during the transition of data.
I happened to have raid connected to older USB ports and connection
was pretty unstable. Multiple times unexpected disconnection due to
wires losing contact caused a problem and the disconnected disk
(although almost perfectly valid) had to be completely rebuilt. I
thought it was not that bad but later it caused sort of desync and I
was unable to connect any two of three disks so the data was lost. My
only luck was that I had all the data (at least important data) backed
up.

So if you're considering raid over USB I suggest you don't do that. It
will probably save you a lot of troubles.

With kind regards,

2018-01-23 21:46 GMT+01:00 Wol's lists <antlists@youngman.org.uk>:
> On 23/01/18 20:16, Linus Lüssing wrote:
>>
>> Hi,
>>
>> By chance I found this statement in the Linux RAID Wiki:
>>
>> "Do NOT use [USB devices] as part of your array. They have
>>   a timeout/disconnect mechanism which interacts very
>>   badly with the raid code." [0]
>
>
> That was me that wrote that on the wiki. However, I'm just editing stuff so
> I don't actually have any experience of it.
>>
>>
>> Could someone elaborate further on this? Does "interacts very
>> badly" mean that there will be potential data loss / data
>> corruption? Or does "very badly" just mean short, temporary
>> performance issues / lags upon a disk failure?
>>
> The wiki also mentions the timeout problem. As far as I can make out - and
> bear in mind as I said above I don't have personal experience - it seems to
> me from what I've picked up that USB suffers this problem in spades.
>
> In other words, what I *think* happens, from what I've picked up, is that
> the USB bus times out. And when the raid comes along to use the disk, the
> USB bus is no longer responding ... BOOM!
>
>> Regards, Linus
>>
> I hope that's helpful, and maybe people who know more will chime in, but
> that's it as I understand it.
>
> Cheers,
> Wol
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Pokud tento mail obsahuje tautologii, pak zřejmě tautologii skutečně obsahuje.

Vojtěch Kletečka

Re: USB-to-SATA and RAID

[Stan Hoeppner]

On 01/23/2018 02:16 PM, Linus Lüssing wrote:
> Hi,
>
> By chance I found this statement in the Linux RAID Wiki:
>
> "Do NOT use [USB devices] as part of your array. They have
>   a timeout/disconnect mechanism which interacts very
>   badly with the raid code." [0]
>
> Could someone elaborate further on this? Does "interacts very
> badly" mean that there will be potential data loss / data
> corruption? Or does "very badly" just mean short, temporary
> performance issues / lags upon a disk failure?

Potential data loss / data corruption.  Don't use USB.  Use eSATA 
instead.  The list archives of linux-raid and linux-xfs are dotted with 
USB disk horror stories.

> Regards, Linus
>
> [0]: https://raid.wiki.kernel.org/index.php/Choosing_your_hardware,_and_what_is_a_device%3F#USB_devices
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: USB-to-SATA and RAID

[Wols Lists]

On 23/01/18 22:23, Vojtěch Kletečka wrote:
> Although USBs are good for occasional connections of simple drives
> they have quite a strong tendency to wear out. This becomes a problem
> especially if you plan to connect raid often and means that wires
> might not connect or lose connection during the transition of data.

Something else I came across recently. Cooling in most enclosures ranges
from poor to non-existent. This leads to overheating drives as a matter
of course, and from that to failure.

(Personally, I have two enclosures that appear to have failed, and I
need to investigate to see if the drives themselves are dead. Two out of
two in the space of a few months is pretty bad going.)

Cheers,
Wol

Re: USB-to-SATA and RAID

[Linus Lüssing]

Hi,

Many thanks for all the quick and helpful responses so far!


On Tue, Jan 23, 2018 at 06:40:14PM -0600, Stan Hoeppner wrote:
> > Could someone elaborate further on this? Does "interacts very
> > badly" mean that there will be potential data loss / data
> > corruption? Or does "very badly" just mean short, temporary
> > performance issues / lags upon a disk failure?
> 
> Potential data loss / data corruption.  Don't use USB.  Use eSATA instead. 
> The list archives of linux-raid and linux-xfs are dotted with USB disk
> horror stories.

Interesting. Do you know what they could be attributed to
technically? Were they usually similar to what Vojtěch had
described, so connectivity issues due to the poor connectors and
thin cables which can often be found with USB? And things then
fell over when for a brief moment connectivity issues occured for
more disks than redundant disks were configured? Or do the extra
latencies with USB create an extended exposure to the write loop
hole in RAID5? Or something else?

Would you guys say that ARM boards with a SATA connector
internally weird to USB (for instance the Ordoid HC2[*]), and then
maybe some SATA Port Multiplier are similarly affected? Or would say
that those are mainly unaffected of the common issues with RAIDs
via SATA over USB?

Regards, Linus

[*]: http://www.hardkernel.com/main/products/prdt_info.php?g_code=G151505170472&tab_idx=2

Re: USB-to-SATA and RAID

[Stan Hoeppner]

On 01/24/2018 06:36 PM, Linus Lüssing wrote:
> Hi,
>
> Many thanks for all the quick and helpful responses so far!
>
>
> On Tue, Jan 23, 2018 at 06:40:14PM -0600, Stan Hoeppner wrote:
>>> Could someone elaborate further on this? Does "interacts very
>>> badly" mean that there will be potential data loss / data
>>> corruption? Or does "very badly" just mean short, temporary
>>> performance issues / lags upon a disk failure?
>> Potential data loss / data corruption.  Don't use USB.  Use eSATA instead.
>> The list archives of linux-raid and linux-xfs are dotted with USB disk
>> horror stories.
> Interesting. Do you know what they could be attributed to
> technically? Were they usually similar to what Vojtěch had
> described, so connectivity issues due to the poor connectors and
> thin cables which can often be found with USB? And things then
> fell over when for a brief moment connectivity issues occured for
> more disks than redundant disks were configured? Or do the extra
> latencies with USB create an extended exposure to the write loop
> hole in RAID5? Or something else?
Here is an example of what you may see with a USB connected drive:

http://forums.debian.net/viewtopic.php?f=10&t=48056

You'll have to read up on the USB protocol to see why devices are 
reset.  As with the user in this post, your USB connected drive may 
function fine for a while, but on occasion you'll hit one of these reset 
conditions.  When that occurs your filesystem may survive, or you may 
suffer corruption, or data loss.  I for one would not rely on storage 
that is reliable only 99% of the time. Murphy's law dictates it will 
fail at the worst possible time.

> Would you guys say that ARM boards with a SATA connector
> internally weird to USB (for instance the Ordoid HC2[*]), and then
> maybe some SATA Port Multiplier are similarly affected? Or would say
> that those are mainly unaffected of the common issues with RAIDs
> via SATA over USB?
>
> Regards, Linus
>
> [*]: http://www.hardkernel.com/main/products/prdt_info.php?g_code=G151505170472&tab_idx=2
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: USB-to-SATA and RAID

[John Stoffel]

Just to chime in here, I've got a five slot USB3 external enclosure
running on my Debian Jessie x86_64 main file/backup server for home.
The data disks arent on here, they're mirrored on SATA controllers.
But I do use the USB3 disks for storing backups taken with Burp.  So
far, it's been really reliable for me.

But I never touch the cable between the systems, and neither system
moves or gets moved around.  So you *might* have better luck with USB3
than the older, slower USB2 external drives and chassis.

John

Re: USB-to-SATA and RAID

[David Brown]

On 23/01/18 21:16, Linus Lüssing wrote:
> Hi,
> 
> By chance I found this statement in the Linux RAID Wiki:
> 
> "Do NOT use [USB devices] as part of your array. They have
>  a timeout/disconnect mechanism which interacts very
>  badly with the raid code." [0]
> 
> Could someone elaborate further on this? Does "interacts very
> badly" mean that there will be potential data loss / data
> corruption? Or does "very badly" just mean short, temporary
> performance issues / lags upon a disk failure?

I would expect that the majority of USB disk issues can be avoided by
due care and attention.

USB buses are rarely reset or re-initialised without provocation.  The
cause is generally flaky hardware on the bus, plugging or unplugging
devices, devices that sleep waking up, sudden changes in power
requirements of devices, and other physical events.

When using USB disks, I'd give the following advice:

1. Connect directly to a USB port on the computer, not via a hub.  That
way you don't have other devices on the same bus that could cause trouble.

2. Avoid plugging other hardware in and out of the other USB buses, and
certainly avoid it on the bus with the disk.  Keep the bus physically
stable.

3. Avoid having high power devices on USB, like DVD burners.  If the
disk has an optional external power supply, use it.

4. Ensure your USB disks are cooled appropriately.  Some USB disk
enclosures are very tight around the disk and can overheat.
Freestanding USB-to-SATA converters where you plug the bare disk in
directly can be good (but beware of dust if you are leaving them for a
while).

5. Use good cables and avoid plugging and unplugging things too much, to
limit the risk of broken wires and worn contacts.

6. Don't rely on the USB drive - use it as an extra.  They are great for
taking extra backups, and to provide additional redundancy during risky
operations (such as to add an asymmetric second parity drive to a raid5
set before replacing the raid5 disks with bigger devices).

7. Don't use "memory stick" USB drives for raid - these have very
limited write lifetimes and are easily corrupted.

8. And for the love of $DEITY, /don't/ have USB drives connected by
cables where you will trip over them and throw the drive across the
room.  (As Yoda says, good judgement comes from experience.  Experience
comes from bad judgement!)



> 
> Regards, Linus
> 
> [0]: https://raid.wiki.kernel.org/index.php/Choosing_your_hardware,_and_what_is_a_device%3F#USB_devices

Re: USB-to-SATA and RAID

[Roger Heflin]

Based on my experience most motherboards only have 2 or 3 actual usb
ports, and all of the others ports are off a hub connected to the real
ports.   And if you put more than one high bandwidth device on the
same port then things get really ugly if both devices are being
heavily used.  Most add-on cards also only have 1 port, if you look
hard enough you can find cards with 2-4 real ports.   I have one that
has 4 actual real ports, and worked reasonably well for several
webcams, but I believe this one was the only one on the market that
had 4 ports and was reasonably cheap.   In general the sharing apears
to be the major issue so it is probably kind of related to the SATA
port multiplier issue since most usb ports are shared resources and
really a similar design, with similar issues.  I am guessing if you
put one usb disk per real usb port and did not plug anything else into
it it would probably work reasonably well, but to built out a setup
like that you might was well just build out an esata setup.

On Wed, Jan 24, 2018 at 6:36 PM, Linus Lüssing <linus.luessing@c0d3.blue> wrote:
> Hi,
>
> Many thanks for all the quick and helpful responses so far!
>
>
> On Tue, Jan 23, 2018 at 06:40:14PM -0600, Stan Hoeppner wrote:
>> > Could someone elaborate further on this? Does "interacts very
>> > badly" mean that there will be potential data loss / data
>> > corruption? Or does "very badly" just mean short, temporary
>> > performance issues / lags upon a disk failure?
>>
>> Potential data loss / data corruption.  Don't use USB.  Use eSATA instead.
>> The list archives of linux-raid and linux-xfs are dotted with USB disk
>> horror stories.
>
> Interesting. Do you know what they could be attributed to
> technically? Were they usually similar to what Vojtěch had
> described, so connectivity issues due to the poor connectors and
> thin cables which can often be found with USB? And things then
> fell over when for a brief moment connectivity issues occured for
> more disks than redundant disks were configured? Or do the extra
> latencies with USB create an extended exposure to the write loop
> hole in RAID5? Or something else?
>
> Would you guys say that ARM boards with a SATA connector
> internally weird to USB (for instance the Ordoid HC2[*]), and then
> maybe some SATA Port Multiplier are similarly affected? Or would say
> that those are mainly unaffected of the common issues with RAIDs
> via SATA over USB?
>
> Regards, Linus
>
> [*]: http://www.hardkernel.com/main/products/prdt_info.php?g_code=G151505170472&tab_idx=2
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: USB-to-SATA and RAID

[Wols Lists]

On 25/01/18 00:36, Linus Lüssing wrote:
> Hi,
> 
> Many thanks for all the quick and helpful responses so far!
> 
> 
Okay, I've written this up and summarised it in a short page on the wiki.

https://raid.wiki.kernel.org/index.php/What's_all_this_with_USB%3F

Does that seem a fair enough summary?

Cheers,
Wol

Re: USB-to-SATA and RAID

[Linus Lüssing]

Hi,

Thanks to everyone again! By the way, why I am considering to use
USB at all is that the storage system is for a solar powered site.
And therefore I have to limit power usage to roughly 10-15W...

So my thinking was to use some ARM board and low-power 2.5" disks.

Unfortunately, many of the low-power, ARM based boards seem to
have no SATA. Or just a single SATA port, often internally wired
to USB...

And yeah, I basically bought a Pi3 and some SATA-to-USB dongles
before stumbling over this entry in the Wiki... my fault...


But I'm glad to hear that there does not seem to be an issue with
USB itself, but more the circumstances.

Regarding cabling and temperature, I'm not too worried. I already
3D-printed some stackable enclosures, even with snap-in mechanisms
and holes for srews, and with hopefully enough airflow capabilities
(people had uploaded some nice designs at thingiverse.com).

But I'll keep an eye on what other ARM based boards with on-board
SATA will come out in the future. And will keep in mind to swap
the board later.

Again, many thanks!

Regards, Linus

PS: And sharing one nightmare story with USB, too: Had a disk
from Toshiba connected to a SATA-to-USB adapter with no extra
power supply and that was connected to an Odroid C1+ (an ARM board
from Hardkernel). The Odroid was not capable of providing enough
power and turned off the USB port for half a second upon noticing.
So I had this nice sound of a disk trying to spin up about twice
a second. Ten seconds later I caught up with what was going on and
disconnected the disk. Nevertheless too late, the disk was broken
then... unrecoverable bad sectors even after trying to wipe the
disk. So in that case, an external USB Hub with its own power supply
would have actually saved that disk, I guess.