RAID for USB flash drives

RAID for USB flash drives

[Hank Barta]

Hi folks,
This is going to be one of those corner cases and I'm sure some of you
won't even think it a good idea. But I won;t let that stop me from
asking your opinion.

I'm building a NAS based on a micro-ATX Atom based board (D525MW.) The
board has two SATA ports to which I will attach two mirrored 2TB
drives form another system. There are 8 USB ports but no other ports
to which to attach a boot drive and I had not partitioned the 2 TB
drives for another partition. Rather than muck with them, I sought USB
boot devices and found prior art on RAIDing USB devices. I purchased
4x 4GB USB flash drives for a grand total of $27 (US)

I plan to operate them as a stripe set. I'm not too concerned about
reliability, thinking that the USB drives should be pretty reliable if
thery're not subject to going through the wash in my pocket.

I've done some initial testing (on another PC) and found that striping
provides increasing performance even if all of the sticks are on the
same USB bus. That surprised me a bit because I expected the USB bus
to be a bottleneck, but that does not happen until there are several
USB devices active.

The particular items that I'd appreciate opinions on are the
parameters used to construct the RAID and file system. The guidance
I've used so far is found in the following two articles:
http://www.analogbit.com/node/4 http://www.analogbit.com/node/13 (I
would feel a lot better if the folks here agreed that their selections
made sense.)

First is partitioning. I used fdisk and default partitioning of one
primary partition:
===============================================================
root@cypress:~# fdisk -l -c -u /dev/sdb

Disk /dev/sdb: 4009 MB, 4009754624 bytes
8 heads, 4 sectors/track, 244736 cylinders, total 7831552 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xf924b0d3

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048     7831551     3914752   fd  Linux raid autodetect
===============================================================

Next will be to create the RAID using:
# mdadm --create /dev/md0 -e 0.9 -n 6 --chunk=64 --level=raid0 /dev/sd[abcd]1


For the file system I saw arguments for and against journaling
filesystems for SSD drives. I decided to go with EXT4

Recalculating from the formula in the second article:
'stride' size is numDevice*chunkSize/blockSize => 4*64K/4k => 64

# mkfs.ext4 /dev/md0 -b 4096 -E stride=64

I plan to add noatime to the mount options. (Would relatime be
better?) I also plan to put active directories like /tmp and /var/log
into RAM via tmpfs. I'll use an OS w/out graphics and have purchased a
2GB RAM stick. I don't plan to use any swap.

Alternatives? I could probably boot off the network but that would be
less interesting. ;) I could also use RAID5 over the 4 USB drives to
guard against a single drive failure. I did try that nut it seemed to
sacrifice some throughput.

OS will be Ubuntu Server 10.04 (LTS) 32 bit.

So... Anything you would do differently?

Thanks for your comments.

best,
hank

-- 
'03 BMW F650CS - hers
'98 Dakar K12RS - "BABY K" grew up.
'93 R100R w/ Velorex 700 (MBD starts...)
'95 Miata - "OUR LC"
polish visor: apply squashed bugs, rinse, repeat
Beautiful Sunny Winfield, Illinois

Re: RAID for USB flash drives

[John Robinson]

On 04/01/2011 19:40, Hank Barta wrote:
> Hi folks,
> This is going to be one of those corner cases and I'm sure some of you
> won't even think it a good idea. But I won;t let that stop me from
> asking your opinion.
[...]

Go ahead :-)

> So... Anything you would do differently?

First up, booting. You'll have to boot off a bare device which may in 
fact be a mirror over 2 or more sticks. Make that your first partition. 
I think it's simpler to make it over all 4. Look up booting from RAID to 
find out how.

Then, as far as I know USB flash sticks have lousy or no wear-levelling 
algorithms and use cheap flash which may not have very many write 
cycles, so given that, I'd either:

1. Use RAID-1, 5 or 10 for the rest of the system with ext4 as you suggest,

or 2. Use btrfs for the rest of the system - btrfs is included in Ubuntu 
10.10 but you may be able to get it for 10.04.

And in either case, make sure there are regular backups of the whole lot 
to your main disc-based array, because those $7 USB sticks could still 
give you problems.

Or 3. I might just decide to use just one stick to boot the system from 
(as a /boot) and put the rest of the system in an LVM LV, or even a 
file, on the main array, and make sure I had a second stick to start the 
system from if the first one went bad.

In any case, it's an interesting project, so good luck!

Cheers,

John.

Re: RAID for USB flash drives

[Steven Haigh]

On 5/01/2011 7:09 AM, John Robinson wrote:
> Or 3. I might just decide to use just one stick to boot the system from
> (as a /boot) and put the rest of the system in an LVM LV, or even a
> file, on the main array, and make sure I had a second stick to start the
> system from if the first one went bad.

That might even be overkill... Yes its good for shits n giggles, however 
with 4Gb you can have the entire OS on a single stick... Even RAID might 
be overkill! Boot from a single stick and rsync it to somewhere else 
incase the thing dies.

> In any case, it's an interesting project, so good luck!

That it is.

-- 
Steven Haigh

Email: netwiz@crc.id.au
Web: http://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897
Fax: (03) 8338 0299

Re: RAID for USB flash drives

[Hank Barta]

On Tue, Jan 4, 2011 at 2:09 PM, John Robinson
<john.robinson@anonymous.org.uk> wrote:

>
> First up, booting. You'll have to boot off a bare device which may in fact
> be a mirror over 2 or more sticks. Make that your first partition. I think
> it's simpler to make it over all 4. Look up booting from RAID to find out
> how.

I should have read this more carefully. I performed a full install to
a raid0 only to find it incapable of setting up grub. It simply told
me there was a fatal error. :-/ It took me a bit more research before
the reality of your comment sunk in. So I proceeded again and tried to
use the first 100MB of each stick to create a 4 device raid1. For some
reason, the partitioning S/W (Ubuntu 10.04.1 server) didn't like that
and just hung when it tried to discover the disks. Repeatedly. I
finally worked around that by deleting the 100 MB partitions on three
of the four drives and just using the first 100 MB on one of the
drives for a /boot partition. I completed the installation but the
system hung on boot. It just provides a blank screen. Experimenting
with the installation USB drive, it seemed to me that I couldn't even
boot that with all five thumb drives installed. I scaled back to
installation three USB drives, install media on a fourth. I also
figured out which slot I needed to put the install media in so as to
boot. The BIOS - or perhaps some stage in the boot process - gets
confused otherwise.

I've booted a live CD and tried running grub-install from a chroot
with the /boot and / file systems and the command concluded w/out any
errors or warnings. But the system still will not boot. I'm not sure
if this is a grub or a RAID problem or some combination.

Is there something I need to include with grub when the root file
system is on a RAID0?

> Then, as far as I know USB flash sticks have lousy or no wear-levelling
> algorithms and use cheap flash which may not have very many write cycles, so
> given that, I'd either:

Time will tell. I plan to move active directories like /var/log and
/tmp to tmpfs to reduce writes to the USB sticks. The ones I got,
though inexpensive, are from a reputable manufacturer (Mushkin.)
Hopefully they have decent wear leveling. Certainly my benchmarks
produced irregular results which I suspect were a result of wear
leveling delaying writes.


>
> 1. Use RAID-1, 5 or 10 for the rest of the system with ext4 as you suggest,
>
> or 2. Use btrfs for the rest of the system - btrfs is included in Ubuntu
> 10.10 but you may be able to get it for 10.04.
>
> And in either case, make sure there are regular backups of the whole lot to
> your main disc-based array, because those $7 USB sticks could still give you
> problems.

Well... As long as I have the install media and capture any
customization to backup I should be OK.

>
> Or 3. I might just decide to use just one stick to boot the system from (as
> a /boot) and put the rest of the system in an LVM LV, or even a file, on the
> main array, and make sure I had a second stick to start the system from if
> the first one went bad.

I could do something like that. However I don't care to mess with the
main array (the two spinning drives) as recovery of those takes a
while.

I could just install to a single USB (no RAID) and try to migrate the
/ to a RAID once I'm off the install media. I could run from the
single stick (as Steven Haigh) suggested, but I'd like to take
advantage of the speed gains from a RAID0.

I could also use 'debirf' as suggested by Iordan Iordanov, but I
prefer to retain the full (disk based) environment with ease of
installing and updating S/W. I haven't studied debirf but I suspect I
would have to create and install a new image any time I want to update
or install new packages.

Suggestions on how to solve the boot problem would be welcome!

thanks,
hank

-- 
'03 BMW F650CS - hers
'98 Dakar K12RS - "BABY K" grew up.
'93 R100R w/ Velorex 700 (MBD starts...)
'95 Miata - "OUR LC"
polish visor: apply squashed bugs, rinse, repeat
Beautiful Sunny Winfield, Illinois

Re: RAID for USB flash drives

[John Robinson]

On 14/01/2011 23:25, Hank Barta wrote:
> On Tue, Jan 4, 2011 at 2:09 PM, John Robinson
> <john.robinson@anonymous.org.uk>  wrote:
>
>>
>> First up, booting. You'll have to boot off a bare device which may in fact
>> be a mirror over 2 or more sticks. Make that your first partition. I think
>> it's simpler to make it over all 4. Look up booting from RAID to find out
>> how.
>
> I should have read this more carefully. I performed a full install to
> a raid0 only to find it incapable of setting up grub. It simply told
> me there was a fatal error. :-/ It took me a bit more research before
> the reality of your comment sunk in. So I proceeded again and tried to
> use the first 100MB of each stick to create a 4 device raid1.

I'm sorry I wasn't clearer. Yes, I meant you needed a RAID-1 on small 
partitions across all 4 sticks for your /boot partition, just as you 
would need a RAID-1 across both your hard drives if you were installing 
on those. (I believe new grub can cope with other RAID levels but I 
don't have any experience with it.) You also need to have the partitions 
starting no earlier than sector 63, to give grub somewhere to put its 
filesystem-aware code.

> For some
> reason, the partitioning S/W (Ubuntu 10.04.1 server) didn't like that
> and just hung when it tried to discover the disks. Repeatedly. I
> finally worked around that by deleting the 100 MB partitions on three
> of the four drives and just using the first 100 MB on one of the
> drives for a /boot partition. I completed the installation but the
> system hung on boot. It just provides a blank screen. Experimenting
> with the installation USB drive, it seemed to me that I couldn't even
> boot that with all five thumb drives installed. I scaled back to
> installation three USB drives, install media on a fourth. I also
> figured out which slot I needed to put the install media in so as to
> boot. The BIOS - or perhaps some stage in the boot process - gets
> confused otherwise.
>
> I've booted a live CD and tried running grub-install from a chroot
> with the /boot and / file systems and the command concluded w/out any
> errors or warnings. But the system still will not boot. I'm not sure
> if this is a grub or a RAID problem or some combination.
>
> Is there something I need to include with grub when the root file
> system is on a RAID0?

I think you need your final boot media in the right place while you're 
running the installer. Booting off one USB stick (your installation 
drive) while trying to make another bootable is liable to go wrong, or 
at least give you a slightly incorrect mdadm.conf or make grub-install 
do the wrong thing. If you can, just plug in the 4 USB sticks you're 
installing on, and hook up a DVD-ROM drive via IDE or SATA to install from.

Even then, you could have trouble unless you make very sure grub-install 
installs on ALL FOUR of your destination USB sticks, because it's 
possible your BIOS will detect the USB sticks in a different order than 
the Linux kernel does. (I don't know that for sure but I wouldn't be 
surprised, I've seen it happen with hard drive interfaces.) grub-install 
should get it right as long as your RAID-1 /boot is spread across all 4 
USB sticks, as long as you're not using USB install media (as per 
previous para).

You will also need to make sure your initrd or initramfs includes the 
USB drivers if it doesn't do that automatically, as well as the 
usb-storage, scsi and md modules for your root filesystem, and just to 
be sure I'd include raid1.ko so it can potentially start /boot as well.

I may be being over-cautious with some of what I say above, but 
hopefully I've covered all the possibilities even if some are overkill, 
and your installer is likely getting more right than I'm giving it 
credit for. Anyway, once again, good luck!

Cheers,

John.

Re: RAID for USB flash drives

[Hank Barta]

On Tue, Jan 18, 2011 at 6:16 AM, John Robinson
<john.robinson@anonymous.org.uk> wrote:
> Did you mean to copy your message to the list? If so please go ahead and
> I'll also copy this reply to the list.

Yes, I meant to - thanks!

> Anyway: I only have experience of grub 0.9x, which has no knowledge of md
> arrays. To boot from a md array using grub 0.9x, you have to make /boot a
> RAID-1 with metadata at the end of the array (so mdadm 0.90 or 1.0
> superblock), so that grub can read the bare partition as a filesystem
> without understanding that it's part of a md array. grub-install does know
> about these arrays though, and will write its stage1 boot code into the MBR
> and the stage1_5 filesystem code into the next few sectors of all the drives
> participating in the array, so that whichever one the BIOS boots from, grub
> can read that mirror of the /boot filesystem and load the chosen kernel and
> initrd.

I'm not so sure that grub2 or the corresponding grub-install is any
further ahead on that account. I'm using Ubuntu which moved to grub2 a
couple releases ago. I'm using it for this attempt and it seemed not
to be able to install grub on a RAID0 drive. I tried both during the
initial install and subsequently booting from a live CD (on USB.)

IAC, I'm working with a non-RAID /boot partition at the moment. If I
get this up and running, it's not so hard to mirror that partition to
other drives.

>
> This is of course a fudge, because the system will fail to boot if there are
> bad sectors on the first drive where anything necessary to boot lives, even
> though there are good copies on another drive. (I've actually had this
> happen to me. I just told the BIOS to boot from the second drive until I
> could swap the faulty first drive.) This fudge is part of the reason for
> software RAID BIOSes and new grub.

I pretty much accept that. I think the only situation presently where
the raid would boot from a (second) RAID1 drive w/out intervention is
if the first drive failed in a way that it completely disappeared form
the system. I doubt that's a common situation.

>
> I understand that new grub does know how to read from md arrays, so
> potentially allowing booting from filesystems striped over several devices
> e.g. RAID-0 and RAID-5 as well as handling RAID-1 mirrors rather better, and
> it moves the fudge factor to just the few sectors grub itself needs and
> means it can cope with the kernel and initrd images having bad sectors in
> them (for RAID-1 and RAID-5). Software RAID BIOSes and hardware RAID cards
> avoid the fudge factor altogether.

Progress is good! :)

Wish I had made more. :(

>
> It looks like you're using new grub, which I don't have any experience of,
> so I'm afraid can't help with it, though I don't suppose it's all that
> different except that it must have another stage between the MBR boot code
> and the filesystem code.

Yes. And I may be battling a couple different problems. The testing I
described in the previous email was actually on another system (Eee PC
901) as it was simply more convenient. After I sent the email out I
tried the same USB sticks in the target system and found it fell far
short of the limited success I had with the Eee. I'm not sure it's
even loading the bootloader from the MBR. (Does that put anything on
the screen?) I can boot the Ubuntu install USB on that system so I
know it's capable of booting form a USB flash drive. But the install
USB sticks may be using some sort of emulation of the El Torrito
mechanism used to boot from CD.

Thanks again for your continuing help.

(Ordinarily I would trim quoted material from previous messages, but
I'll leave it intact since I forgot to CC the list with that.)

best,
hank

>
> On 17/01/2011 14:38, Hank Barta wrote:
>>
>> On Sat, Jan 15, 2011 at 2:54 PM, John Robinson
>> <john.robinson@anonymous.org.uk>  wrote:
>>>
>>> I'm sorry I wasn't clearer.
>>
>> No worries - you were clear enough. Lack of comprehension was my fault.
>>
>>>
>>> You will also need to make sure your initrd or initramfs includes the USB
>>> drivers if it doesn't do that automatically, as well as the usb-storage,
>>> scsi and md modules for your root filesystem, and just to be sure I'd
>>> include raid1.ko so it can potentially start /boot as well.
>>
>> I'm working through what I believe may be grub issues on this. At
>> present I'm working with three USB sticks. I've mounted them on my
>> main laptop, started the RAID and then tweaked the GRUB configurations
>> to put the menu up and add some boot options. I wanted to see the menu
>> to try to determine just how far GRUB was getting. The boot options
>> are from one of the pages I referenced earlier:
>> kopt=usb-storage.delay_use=1 rootfstype=ext4 rootflags=data=writeback
>>
>> (I have also tried it w/out "rootflags=data=writeback")
>>
>> The result is some progress during boot. Grub comes up and I get the
>> menu. When I select the preferred boot option, I can see it access the
>> boot stick for a couple seconds. Then there is no visible activity
>> followed by access of all three sticks. After that, there is an error
>> message:
>>
>> mount: mounting /dev/md1 on /root failed: invalid argument
>>
>> This is followed by three errors indicating that /dev, /sys and /proc
>> cannot be mounted due to "No such file or directory." After that an
>> error that init cannot be found (of course since root is not mounted.)
>>
>> At this point the console is dumped into BusyBox with the prompt
>> "(initramfs)" so it seems to me the problem is inability to mount the
>> raid.
>>
>> At this point I can 'cat /dev/mdstat' and the raid appears active. But
>> attempts to mount it fail with the same "Invalid argument" error.
>>
>> The 'blkid' command identifies the USB boot partition and the three
>> RAID partitions but does not find /dev/md1.
>>
>> there are no logs, but dmesg ends with the md messages:
>> [...] md1: detected capacity change from 0 to 12027691008
>> [...]  md1: p1 p2
>> (There are a bunch of normal looking raid0 messages preceding this.)
>>
>> The menu entry I'm trying to boot from /boot/grub.cfg is:
>> menuentry 'Ubuntu, with Linux 2.6.32-24-generic-pae' --class ubuntu
>> --class gnu-linux --class gnu --class os {
>>        recordfail
>>        insmod ext2
>>        set root='(hd1,1)'
>>        search --no-floppy --fs-uuid --set
>> 8db30d89-6a2b-4d41-842d-3bb8d24e14b8
>>        linux   /vmlinuz-2.6.32-24-generic-pae root=/dev/md1 ro
>> kopt=usb-storage.delay_use=1 rootfstype=ext4  quiet
>>        initrd  /initrd.img-2.6.32-24-generic-pae
>> }
>>
>> So it seems that the initramfs loads and the raid gets started, but it
>> cannot mount the root fs on /dev/md1.
>>
>>>
>>> I may be being over-cautious with some of what I say above, but hopefully
>>> I've covered all the possibilities even if some are overkill, and your
>>> installer is likely getting more right than I'm giving it credit for.
>>
>> I did find some reports of nasty bugs regarding the installer getting
>> things very wrong, but I think those issues have been fixed.
>>
>>> Anyway, once again, good luck!


-- 
'03 BMW F650CS - hers
'98 Dakar K12RS - "BABY K" grew up.
'93 R100R w/ Velorex 700 (MBD starts...)
'95 Miata - "OUR LC"
polish visor: apply squashed bugs, rinse, repeat
Beautiful Sunny Winfield, Illinois
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID for USB flash drives

[Iordan Iordanov]

Hi Hank,

On 01/14/11 18:25, Hank Barta wrote:
> I could also use 'debirf' as suggested by Iordan Iordanov, but I
> prefer to retain the full (disk based) environment with ease of
> installing and updating S/W. I haven't studied debirf but I suspect I
> would have to create and install a new image any time I want to update
> or install new packages.

Just to help clarify the debirf possibility. Debirf is Debian inside 
initrd, so yes, in order to make a change persistent, one would have to 
copy a new initrd over to the USB stick. However, since Debian is fully 
functional, running in a RAM disk, one is also able to install and 
update software on the fly using apt/aptitude. It's just that changes 
made to the ramdisk do not make it automatically into the initrd on the 
USB stick. You have to run one make command and copy the new image to 
the stick yourself. You can easily cron the building of the image, 
mounting of the USB stick, copying over, and unmounting once a day or 
week to keep the image up-to-date.

Also, all of this may be possible with Ubuntu, I just haven't had to do 
the research.

If you don't do debirf, don't forget to NOT put swap on the stick, and I 
would also make a ramdisk for /var/log and copy the logs to the stick 
upon shutdown (to prevent unnecessary flash wear as logs are gradually 
incremented).

Cheers and good luck!
Iordan

Re: RAID for USB flash drives

[Iordan Iordanov]

Debirf is a project which allows one to create an initrd image which 
contains a complete Debian system.

We use debirf to create an image that we then boot from a single USB 
stick. We use this to boot a fleet of machines which export iscsi 
targets, so we don't have to waste a drive bay for the OS.

http://cmrg.fifthhorseman.net/wiki/debirf

For us, this solution works great, since there is absolutely NO wear on 
the stick - the entire OS runs in memory. However, this does mean that 
you would have to consider Debian rather than Ubuntu, or find a way to 
make debirf work under Ubuntu.

As far as constructing the RAID sets, I would look into using larger 
chunks. Our testing showed that in our environment where our requests 
are small (of size around 1 - 2kb), large 1MB chunks gave the best 
performance. Chunks of size 512kb were a close second in performance for us.

Booting over the network can create certain chicken-and-egg scenarios 
which again depend on your environment :).

Cheers,
Iordan