Re: raid5 + HDFS

[Martin Tippmann <martin.tippmann@gmail.com>]

2015-10-03 0:07 GMT+02:00 Jim Dowling <jdowling@sics.se>:
> Hi

Hi, I'm not a btrfs developer but we run HDFS on top of btrfs (mainly
due to other use-cases that profit from checksumming data)

> I am interested in combining BtrFS RAID-5 with erasure-coded replication for
> HDFS. We have an implementation of Reed-Solomon replication for our HDFS
> distribution called HopsFS (www.hops.io).

I only know HDFS as being filesystem agnostic and JBOD is usually the way to go.

You plan to use btrfs directly for storing blocks and want to exploit
the RAID5 functionality for erasure coding? I'm not familiar with
hops.io and existing approaches to erasure coding on btrfs but a few
questions come to my mind:

How do you deal with HDFS checksums? AFAIK these are calculated in a
different way than btrfs does it?
Do you throw away the whole Java DataNode Idea and replace it with
somethings that talks directly to the filesystem?

> Some of the nice features of HDFS that make it suitable are:
> * not many small files
> * not excessive snapshotting
> * can probably manage disks being close to capacity, as its globally visible
> and blocks can be re-balanced in HDFS

HDFS Blocks are mostly 64MB to 512MB - I don't see the connection to
btrfs that checksums much smaller blocks?

> What BtrFS could give us:
> * stripping within a DataNode using RAID-5

At least for MapReduce Jobs JBOD exploits the fact that you can use a
disk for a job. Wouldn't RAID5 kind of destroy MapReduce (or random
access) performance?

> * higher throughput read/write for HDFS clients over 10 GbE without losing
> data locality (others are looking at stripping blocks over many different
> nodes).

We've got machines with 4x4TB disks and I'm seeing that they can reach
up to 500Mbyte/s on 10GbE in JBOD during shuffle. Would be great if
you give more details or some hints to read up on it why exactly doing
RAID5 is better than JBOD.

> * true snapshotting for HDFS by providing rollback of HDFS blocks

This would be great, but how do you deal with lost disks? The historic
data will be local to the node containing the btrfs file system?

> I am concerned (actually, excited!) that RAID-5 is not production ready. Any
> guidelines on how mature it is, since the big PR in Feb/March 2015?
> What about scrubbing for RAID-5?

As I've said I'm not a dev but from reading the list RAID5 scrubbing
should work with a recent kernel and recent btrfs-tools (4.1+). There
where some bugs but AFAIK these are resolved with Linux 4.2+

> Is there anything else I should know?
>
> Btw, here are some links i found about RAID-5:
>
>
> UREs are not as common on commodity disks - RAID-5 is safer than assumed:
> *<https://www.high-rely.com/blog/using-raid5-means-the-sky-is-falling/>https://www.high-rely.com/blog/using-raid5-means-the-sky-is-falling/*

If you have cluster nodes with 12 disks and 2 disks die at the same
time... can you deal with that? RAID5 means loosing all data on all 12
disks?

> Btrfs Linux 4.1 with 5 10K RPM spinning disks:
>
> <http://www.phoronix.com/scan.php?page=article&item=btrfs-raid015610-linux41&num=1>http://www.phoronix.com/scan.php?page=article&item=btrfs-raid015610-linux41&num=1
>
> *Raid*-*5 *Results: ~250 MB/s for sequential reads. 559 MB/s for sequential
> writes.

The ZFS guys try to move the parity calculation to SSE and AVX with
some good looking performance improvements. See
https://github.com/zfsonlinux/zfs/pull/3374 - so it's probably
possible to have a fast enough RAID5 implementation in btrfs - but I'm
lacking the bigger picture and the explicit use case for btrfs in this
setup?

hops.io looks very interesting through. Would be great if you could
clarify your ideas a little bit.

HTH & regards
Martin