BTRFS on production: NVR 16+ IP Cameras

BTRFS on production: NVR 16+ IP Cameras

[Juan Alberto Cirez]

Is BTRFS mature enough to be deployed on a production system to underpin 
the storage layer of a 16+ ipcameras-based NVR (or VMS if you prefer)?

Based on our limited experience with BTRFS (1+ year) under the above 
scenario the answer seems to be no; but I wanted you ask the community 
at large for their experience before making a final decision to hold off 
on deploying BTRFS on production systems.

Let us be clear: We think BTRFS has great potential, and as it matures 
we will continue to watch its progress, so that at some future point we 
can return to using it.

The issue has been the myriad of problems we have encountered when 
deploying BTRFS as the storage fs for the NVR/VMS in cases were the 
camera count exceeds 10: Corrupted file systems, sudden read-only file 
system, re-balance kernel panics, broken partitions, etc.

One specific case saw the btrfs drive pool mounted under the /var 
partition so that upon installation the btrfs pool contained all files 
under /var; /lib/mysql as well as the video storage. Needless to say 
this was a catastrophe...

At the other end of the spectrum is a use case where ONLY the video 
storage was on the btrfs pool; but in that case, the btrfs pool became 
read-only suddenly and would not re-mount as rw despite all the recovery 
trick btrfs-tools could throw at it. This, of course prevented the 
NVR/VMS from recording any footage.

So, again, the question is: is BTRFS mature enough to be used in such 
use case and if so, what approach can be used to mitigate such issues.


Thank you all for your assistance

-Ryan-
-- 
Ryan Burt, B. Sc. for
Juan Alberto Cirez, CFPT

Re: BTRFS on production: NVR 16+ IP Cameras

[Nikolay Borisov]

On 15.11.18 г. 20:39 ч., Juan Alberto Cirez wrote:
> Is BTRFS mature enough to be deployed on a production system to underpin
> the storage layer of a 16+ ipcameras-based NVR (or VMS if you prefer)?
> 
> Based on our limited experience with BTRFS (1+ year) under the above
> scenario the answer seems to be no; but I wanted you ask the community
> at large for their experience before making a final decision to hold off
> on deploying BTRFS on production systems.
> 
> Let us be clear: We think BTRFS has great potential, and as it matures
> we will continue to watch its progress, so that at some future point we
> can return to using it.
> 
> The issue has been the myriad of problems we have encountered when
> deploying BTRFS as the storage fs for the NVR/VMS in cases were the
> camera count exceeds 10: Corrupted file systems, sudden read-only file
> system, re-balance kernel panics, broken partitions, etc.

What version of the file system, how many of those issues (if any) have
you reported to the upstream mailing list? I don't remember seeing any
reports from you.

> 
> One specific case saw the btrfs drive pool mounted under the /var
> partition so that upon installation the btrfs pool contained all files
> under /var; /lib/mysql as well as the video storage. Needless to say
> this was a catastrophe...

BTRFS is not suitable for random workloads, such as databases for
example. In those cases it's recommended, at the very least, to disable
COW on the database files. Note, disabling CoW doesn't preclude you from
creating snapshots, it just means that not every 4k write (for example)
will result in a CoW operation.

> 
> At the other end of the spectrum is a use case where ONLY the video
> storage was on the btrfs pool; but in that case, the btrfs pool became
> read-only suddenly and would not re-mount as rw despite all the recovery
> trick btrfs-tools could throw at it. This, of course prevented the
> NVR/VMS from recording any footage.

Again, you give absolutely no information about how you have configured
the filesystem or versions of the software used.

> 
> So, again, the question is: is BTRFS mature enough to be used in such
> use case and if so, what approach can be used to mitigate such issues.
> 
> 
> Thank you all for your assistance
> 
> -Ryan-

Re: BTRFS on production: NVR 16+ IP Cameras

[Anand Jain]

On 11/16/2018 03:51 AM, Nikolay Borisov wrote:
> 
> 
> On 15.11.18 г. 20:39 ч., Juan Alberto Cirez wrote:
>> Is BTRFS mature enough to be deployed on a production system to underpin
>> the storage layer of a 16+ ipcameras-based NVR (or VMS if you prefer)?
>>
>> Based on our limited experience with BTRFS (1+ year) under the above
>> scenario the answer seems to be no; but I wanted you ask the community
>> at large for their experience before making a final decision to hold off
>> on deploying BTRFS on production systems.
>>
>> Let us be clear: We think BTRFS has great potential, and as it matures
>> we will continue to watch its progress, so that at some future point we
>> can return to using it.
>>
>> The issue has been the myriad of problems we have encountered when
>> deploying BTRFS as the storage fs for the NVR/VMS in cases were the
>> camera count exceeds 10: Corrupted file systems, sudden read-only file
>> system, re-balance kernel panics, broken partitions, etc.
> 
> What version of the file system, how many of those issues (if any) have
> you reported to the upstream mailing list? I don't remember seeing any
> reports from you.
> 
>>
>> One specific case saw the btrfs drive pool mounted under the /var
>> partition so that upon installation the btrfs pool contained all files
>> under /var; /lib/mysql as well as the video storage. Needless to say
>> this was a catastrophe...
> 
> BTRFS is not suitable for random workloads, such as databases for
> example. In those cases it's recommended, at the very least, to disable
> COW on the database files. Note, disabling CoW doesn't preclude you from
> creating snapshots, it just means that not every 4k write (for example)
> will result in a CoW operation.
> 
>>
>> At the other end of the spectrum is a use case where ONLY the video
>> storage was on the btrfs pool; but in that case, the btrfs pool became
>> read-only suddenly and would not re-mount as rw despite all the recovery
>> trick btrfs-tools could throw at it. This, of course prevented the
>> NVR/VMS from recording any footage.
> 
> Again, you give absolutely no information about how you have configured
> the filesystem or versions of the software used.
> 
>>
>> So, again, the question is: is BTRFS mature enough to be used in such
>> use case and if so, what approach can be used to mitigate such issues.

  And..

   What is the blocksize used by the application to write into btrfs?
   And what type of write IO? async, directio or fsync?

Thanks, Anand


>>
>> Thank you all for your assistance
>>
>> -Ryan-

Re: BTRFS on production: NVR 16+ IP Cameras

[Roman Mamedov]

On Thu, 15 Nov 2018 11:39:58 -0700
Juan Alberto Cirez <jacirez@rdcsafety.com> wrote:

> Is BTRFS mature enough to be deployed on a production system to underpin 
> the storage layer of a 16+ ipcameras-based NVR (or VMS if you prefer)?

What are you looking to gain from using Btrfs on an NVR system? It doesn't
sound like any of its prime-time features -- such as snapshots, checksumming,
compression or reflink copying -- are a must for a bulk video recording
scenario.

And even if you don't need or use those features, you still pay the price for
having them, Btrfs can be 2x to 10x slower than its simpler competitors:
https://www.phoronix.com/scan.php?page=article&item=linux418-nvme-raid&num=2

If you just meant to use its multi-device support instead of a separate RAID
layer, IMO that can't be considered prime-time or depended upon in production
(yes, not even RAID 1 or 10).

-- 
With respect,
Roman

Re: BTRFS on production: NVR 16+ IP Cameras

[Austin S. Hemmelgarn]

On 2018-11-15 13:39, Juan Alberto Cirez wrote:
> Is BTRFS mature enough to be deployed on a production system to underpin 
> the storage layer of a 16+ ipcameras-based NVR (or VMS if you prefer)?
For NVR, I'd say no.  BTRFS does pretty horribly with append-only 
workloads, even if they are WORM style.  It also does a really bad job 
with most relational database systems that you would likely use for 
indexing.

If you can suggest your reasoning for wanting to use BTRFS though, I can 
probably point you at alternatives that would work more reliably for 
your use case.

Re: BTRFS on production: NVR 16+ IP Cameras

[Chris Murphy]

On Thu, Nov 15, 2018 at 10:39 AM Juan Alberto Cirez
<jacirez@rdcsafety.com> wrote:
>
> Is BTRFS mature enough to be deployed on a production system to underpin
> the storage layer of a 16+ ipcameras-based NVR (or VMS if you prefer)?
>
> Based on our limited experience with BTRFS (1+ year) under the above
> scenario the answer seems to be no; but I wanted you ask the community
> at large for their experience before making a final decision to hold off
> on deploying BTRFS on production systems.
>
> Let us be clear: We think BTRFS has great potential, and as it matures
> we will continue to watch its progress, so that at some future point we
> can return to using it.
>
> The issue has been the myriad of problems we have encountered when
> deploying BTRFS as the storage fs for the NVR/VMS in cases were the
> camera count exceeds 10: Corrupted file systems, sudden read-only file
> system, re-balance kernel panics, broken partitions, etc.

Performance problems are separate from reliability problems. No matter
what, there shouldn't be corruptions or failures when your process is
writing through the Btrfs kernel driver. Period. So you've either got
significant hardware/firmware problems as the root cause, or your use
case is exposing Btrfs bugs.

But you're burdened with providing sufficient details about the
hardware and storage stack configuration including kernel, btrfs-progs
versions and mkfs options and mount options being used. Without a way
for a developer to reproduce your problem it's unlikely the source of
the problem can be discovered and fixed.


> So, again, the question is: is BTRFS mature enough to be used in such
> use case and if so, what approach can be used to mitigate such issues.

What format are the cameras writing out in? It matters if this is a
continuous appending format, or if it's writing them out as individual
JPEG files, one per frame, or whatever. What rate, what size, and any
other concurrent operations, etc.

-- 
Chris Murphy