Re: Status of FST and mount times

["Ellis H. Wilson III" <ellisw@panasas.com>]

On 02/15/2018 01:14 AM, Chris Murphy wrote:
> On Wed, Feb 14, 2018 at 9:00 AM, Ellis H. Wilson III <ellisw@panasas.com> wrote:
> 
>> Frame-of-reference here: RAID0.  Around 70TB raw capacity.  No compression.
>> No quotas enabled.  Many (potentially tens to hundreds) of subvolumes, each
>> with tens of snapshots.
> 
> Even if non-catastrophic to lose such a file system, it's big enough
> to be tedious and take time to set it up again. I think it's worth
> considering one of two things as alternatives:
> 
> a. metadata raid1, data single: you lose the striping performance of
> raid0, and if it's not randomly filled you'll end up with some disk
> contention for reads and writes *but* if you lose a drive you will not
> lose the file system. Any missing files on the dead drive will result
> in EIO (and I think also a kernel message with path to file), and so
> you could just run a script to delete those files and replace them
> with backup copies.

This option is on our roadmap for future releases of our parallel file 
system, but unfortunately we do not presently have the time to implement 
the functionality to report from the manager of that btrfs filesystem to 
the pfs manager that said files have gone missing.  We will absolutely 
be revisiting that as an option in early 2019, as replacing just one 
disk instead of N is highly attractive.  Waiting for EIO as you suggest 
in b is a non-starter for us, as we're working at scales sufficiently 
large that we don't want to wait for someone to stumble over a partially 
degraded file.  Pro-active reporting is what's needed, and we'll 
implement that Real Soon Now.

> b. Variation on the above would be to put it behind glusterfs
> replicated volume. Gluster getting EIO from a brick should cause it to
> get a copy from another brick and then fix up the bad one
> automatically. Or in your raid0 case, the whole volume is lost, and
> glusterfs helps do the full rebuild over 3-7 days while you're still
> able to access those 70TB of data normally. Of course, this option
> requires having two 70TB storage bricks available.

See my email address, which may help understand why GlusterFS is a 
non-starter.  Nevertheless, the idea is a fine one and we'll have 
something similar going on, but at higher raid levels and across 
typically a dozen or more of such bricks.

Best,

ellis