File server structure suggestion

File server structure suggestion

[Andrew Flerchinger]

I am going to be building a new file server in a couple weeks and am
deciding on how to structure it. Previously, I ran RAID5 through mdadm
and XFS on Debian, but I had a silently bad drive that corrupted data
without dropping from the array. I suspected this was happening, and
changed over to ZFS on FreeNAS and the monthly scrub told me exactly
what was going on. That sold me on the idea of data checksums, but I'd
rather stay in linux than BSD, and I previously made use of online
capacity expansion as needed, which ZFS doesn't support.

Enter btrfs. Unfortunately, it's newer than ZFS and isn't as robust,
but it does support online capacity expansion, and the on disk format
is expect to be stable. It has data checksums and COW, which are the
primary things I'm after. RAID10 seems pretty stable, but RAID56
isn't.

So I'm looking for a suggestion. My end goal is RAID6 and expand it a
drive at a time as needed. For right now, I can either:

1) Run RAID6, but be aware of its limitations. I can manually remove
and add drives in separate steps if needed. Keep the server on a UPS
to limit unexpected shutdowns and any corruption there. The whole
array can't be scrubbed, but if there is a chechsum problem when
reading individual data, will that still be corrected and/or logged?
This will be a temporary situation, as over time, more features will
be built out, and the existing file system will be better supported.

2) Run RAID10, and convert the file system to RAID6 later once it is
stable. Since RAID10 is far more stable and feature complete than
RAID56 right now, all features will work okay, I'm just buying more
drives/running at lower capacity for the moment. If I have to grow the
array, I'd have to buy two drives. In the future, once RAID6 is better
supported, I can convert in-place to RAID6.

What do you think?

Thanks,
Andrew

Re: File server structure suggestion

[Tamas Papp]

On 07/10/2014 04:41 PM, Andrew Flerchinger wrote:
> what was going on. That sold me on the idea of data checksums, but I'd
> rather stay in linux than BSD, and I previously made use of online
> capacity expansion as needed, which ZFS doesn't support.

What do you mean by that?
What zfs doesn't support is reducing a pool.


tamas

Re: File server structure suggestion

[Andrew Flerchinger]

I want to increase the size of the vdev, not just the zpool. I want to
make a 3-drive array into a 4-drive array by adding a single drive
while still having one parity stripe across all data. Adding more
vdevs to a zpool isn't quite the same thing as online capacity
expansion. It's not something most businesses would do, which is why
the feature never made it into ZFS, but consumers are cheap and mdadm
supported it. From what I've read, btrfs supports it, too.

On Thu, Jul 10, 2014 at 10:52 AM, Tamas Papp <tompos@martos.bme.hu> wrote:
>
> On 07/10/2014 04:41 PM, Andrew Flerchinger wrote:
>>
>> what was going on. That sold me on the idea of data checksums, but I'd
>> rather stay in linux than BSD, and I previously made use of online
>> capacity expansion as needed, which ZFS doesn't support.
>
>
> What do you mean by that?
> What zfs doesn't support is reducing a pool.
>
>
> tamas

Re: File server structure suggestion

[Duncan]

Andrew Flerchinger posted on Thu, 10 Jul 2014 10:41:02 -0400 as excerpted:

> Enter btrfs. Unfortunately, it's newer than ZFS and isn't as robust, but
> it does support online capacity expansion, and the on disk format is
> expect to be stable. It has data checksums and COW, which are the
> primary things I'm after. RAID10 seems pretty stable, but RAID56 isn't.
> 
> So I'm looking for a suggestion. My end goal is RAID6 and expand it a
> drive at a time as needed. For right now, I can either:
> 
> 1) Run RAID6, but be aware of its limitations. I can manually remove and
> add drives in separate steps if needed. Keep the server on a UPS to
> limit unexpected shutdowns and any corruption there. The whole array
> can't be scrubbed, but if there is a chechsum problem when reading
> individual data, will that still be corrected and/or logged? This will
> be a temporary situation, as over time, more features will be built out,
> and the existing file system will be better supported.
> 
> 2) Run RAID10, and convert the file system to RAID6 later once it is
> stable. Since RAID10 is far more stable and feature complete than RAID56
> right now, all features will work okay, I'm just buying more
> drives/running at lower capacity for the moment. If I have to grow the
> array, I'd have to buy two drives. In the future, once RAID6 is better
> supported, I can convert in-place to RAID6.

I'd personally consider btrfs raid5/6 to be in practice a slow and lower 
capacity raid0, at this point, except that you'll get raid5/6 for "free" 
when that's fully supported, since it has been doing the writing for that 
all along, it just couldn't properly restore.  IOW, I wouldn't consider 
it trustworthy at all against loss of a device, which based on your 
suggestion, isn't appropriate for your usage.

That leaves either raid10 or raid1.  It's worth noting that btrfs raid1 
is at this point paired mirrors only, so no matter how many devices, you 
still have exactly two mirrors of all (meta)data.  N-way-mirroring is 
planned for after raid5/6 completion.  Which could put raid1 in the 
running for you, and as the simplest redundant raid, it might be easier 
to convert to raid5/6 later.

Then there's raid10, which takes more drives and is faster, but is still 
limited to two mirrors.  But while I haven't actually used raid10 myself, 
I do /not/ believe it's limited to pair-at-a-time additions.  I believe 
it'll take, for instance five devices, just fine, staggering chunk 
allocation as necessary to fill all at about the same rate.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

RE: File server structure suggestion

[Kyle Gates]

>
> Then there's raid10, which takes more drives and is faster, but is still
> limited to two mirrors. But while I haven't actually used raid10 myself,
> I do /not/ believe it's limited to pair-at-a-time additions. I believe
> it'll take, for instance five devices, just fine, staggering chunk
> allocation as necessary to fill all at about the same rate.
>
I am running just that: 3 separate raid10 btrfs filesystems (root, home, media/backups) on 5 drives and they are unequal sizes too!
My newer drives are bigger and have higher transfer rates which means they get more chunks but overall performance doesn't suffer.