Recommendation on new system Arrays

Recommendation on new system Arrays

[Benjammin2068]

Hey guys,


 I have a question for you now that I'm diving into a machine with more drives on a single system than I've ever worked with before.


Hardware:

    Supermicro Rack PC with Avago MegaRAID SAS 9380-8e HBA. Motherboard has Dual Xeon E5-2640 V4 cpus (10 HT cores each)

    Supermicro SC847 JBOD housing with room for 44 disk but currently populated with 22 drives (rear backplane)

    (22) HGST 8TB drives - Model HUH728080AL5200 - https://www.hgst.com/products/hard-drives/ultrastar-he8

    OS is CentOS 7.x (I think 7.2. I just downloaded and installed the other night)

Of course, I'd like to avoid HW raid.

but.. for this array, I was going to split into a couple of RAID6 arrays with a couple hot spares in the rack.

    * one for owners data

    * the other as a data drive for use with surveillance software (from Milestone probably) running in a VM as a secondary storage vault for company cameras/NVRs that are remote. (like a backup system)

But anything else I should look out for?

Thanks,

 -Ben

Re: Recommendation on new system Arrays

[Wols Lists]

On 06/01/17 16:14, Benjammin2068 wrote:
>     * the other as a data drive for use with surveillance software (from Milestone probably) running in a VM as a secondary storage vault for company cameras/NVRs that are remote. (like a backup system)

If you were planning on a raid-6, how about a mirror for this drive? I
believe it is not a good choice for a general raid, but if you're only
writing *NEW* stuff for the most part, and streaming big files to disk,
how about these new WD shingled disks (Purples, I believe).

This sounds like one of the few use cases where they might be a good
idea. Worth considering, at least. Might even work well in your planned
raid-6 config.

Cheers,
Wol

Re: Recommendation on new system Arrays

[Benjammin2068]

On 01/07/2017 04:04 AM, Wols Lists wrote:
> On 06/01/17 16:14, Benjammin2068 wrote:
>>     * the other as a data drive for use with surveillance software (from Milestone probably) running in a VM as a secondary storage vault for company cameras/NVRs that are remote. (like a backup system)
> If you were planning on a raid-6, how about a mirror for this drive? I
> believe it is not a good choice for a general raid, but if you're only
> writing *NEW* stuff for the most part, and streaming big files to disk,
> how about these new WD shingled disks (Purples, I believe).
>
> This sounds like one of the few use cases where they might be a good
> idea. Worth considering, at least. Might even work well in your planned
> raid-6 config.
>

Yea, I thought about that too. So you're suggesting a RAID1 consisting of (2) RAID6 arrays.

Will mdadm do that?

Also in other news.... (maybe someone from this list can help since they've run into it before)

The motherboard of the computer has its own SATA controller as usual... as well as this Avago SAS controller.

When CentOS 7 boots up, it enumerates the external SAS drives starting at /dev/sda instead of the motherboard's drives.

The OCD in me wants the MB's SATA drives as /dev/sda-sdd.

Where does one even control that enumeration order? udev? Eeks.

(I'm not finding a query that google makes sense out of to give me a decent set of suggested links.)

 -Ben

Re: Recommendation on new system Arrays

[Stan Hoeppner]

On 01/07/2017 10:04 PM, Benjammin2068 wrote:
> On 01/07/2017 04:04 AM, Wols Lists wrote:
>> On 06/01/17 16:14, Benjammin2068 wrote:
>>>      * the other as a data drive for use with surveillance software (from Milestone probably) running in a VM as a secondary storage vault for company cameras/NVRs that are remote. (like a backup system)
>> If you were planning on a raid-6, how about a mirror for this drive? I
>> believe it is not a good choice for a general raid, but if you're only
>> writing *NEW* stuff for the most part, and streaming big files to disk,
>> how about these new WD shingled disks (Purples, I believe).
>>
>> This sounds like one of the few use cases where they might be a good
>> idea. Worth considering, at least. Might even work well in your planned
>> raid-6 config.
>>
> Yea, I thought about that too. So you're suggesting a RAID1 consisting of (2) RAID6 arrays.
That's not sane.  RAID 10, or more precisely, RAID 0 over many RAID 1 
pairs, will yield more usable capacity and without the parity penalty or 
RMW cycles.

>
> Will mdadm do that?
>
> Also in other news.... (maybe someone from this list can help since they've run into it before)
>
> The motherboard of the computer has its own SATA controller as usual... as well as this Avago SAS controller.
>
> When CentOS 7 boots up, it enumerates the external SAS drives starting at /dev/sda instead of the motherboard's drives.
>
> The OCD in me wants the MB's SATA drives as /dev/sda-sdd.
>
> Where does one even control that enumeration order? udev? Eeks.
>
> (I'm not finding a query that google makes sense out of to give me a decent set of suggested links.)
>
>   -Ben
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Recommendation on new system Arrays

[Wols Lists]

On 08/01/17 04:13, Stan Hoeppner wrote:
>>>
>>> This sounds like one of the few use cases where they might be a good
>>> idea. Worth considering, at least. Might even work well in your planned
>>> raid-6 config.
>>>

>> Yea, I thought about that too. So you're suggesting a RAID1 consisting
>> of (2) RAID6 arrays.

> That's not sane.  RAID 10, or more precisely, RAID 0 over many RAID 1
> pairs, will yield more usable capacity and without the parity penalty or
> RMW cycles.

I was actually thinking pure raid-6. Which was Benjammin's original
idea. The thing to bear in mind is that for these drives, rewrite
performance is going to be abysmal, but in this particular scenario
rewriting is going to be unusual, anyway. What we *don't* want with
these drives is RMW.

It's just struck me, this might be where it's worth sticking a small SSD
in front of the array as a journal, and flagging these drives as "write
mostly". But I'm not sure whether this functionality is yet standard or
are the devs still working on it?

Cheers,
Wol

Re: Recommendation on new system Arrays

[Phil Turmel]

Hi Ben,

{Convention on kernel.org is reply-to-all -- please do.}

On 01/07/2017 11:04 PM, Benjammin2068 wrote:

> Also in other news.... (maybe someone from this list can help since they've run into it before)
> 
> The motherboard of the computer has its own SATA controller as usual... as well as this Avago SAS controller.
> 
> When CentOS 7 boots up, it enumerates the external SAS drives starting at /dev/sda instead of the motherboard's drives.
> 
> The OCD in me wants the MB's SATA drives as /dev/sda-sdd.
> 
> Where does one even control that enumeration order? udev? Eeks.

You cannot control the order.  Period.  It is pseudo-random based on
hardware responses to driver loads.  New kernels can have entirely
different orders of devices, and if parallel loads are allowed, you can
have one controller get sd[aceg] while another gets sd[bdfh]. The entire
system of LABEL= and UUID= support via initramfs and blkid is intended
make fstab and other utilities deterministic in spite of varying names.

mdadm has always been resistant to naming problems thanks to its device
#s and UUIDs in the superblocks.

Phil

Re: Recommendation on new system Arrays

[Benjammin2068]

On 01/12/2017 09:39 AM, Phil Turmel wrote:
> Hi Ben,
>
> {Convention on kernel.org is reply-to-all -- please do.}
>
> On 01/07/2017 11:04 PM, Benjammin2068 wrote:
>
>> Also in other news.... (maybe someone from this list can help since they've run into it before)
>>
>> The motherboard of the computer has its own SATA controller as usual... as well as this Avago SAS controller.
>>
>> When CentOS 7 boots up, it enumerates the external SAS drives starting at /dev/sda instead of the motherboard's drives.
>>
>> The OCD in me wants the MB's SATA drives as /dev/sda-sdd.
>>
>> Where does one even control that enumeration order? udev? Eeks.
> You cannot control the order.  Period.  It is pseudo-random based on
> hardware responses to driver loads.  New kernels can have entirely
> different orders of devices, and if parallel loads are allowed, you can
> have one controller get sd[aceg] while another gets sd[bdfh]. The entire
> system of LABEL= and UUID= support via initramfs and blkid is intended
> make fstab and other utilities deterministic in spite of varying names.
>
> mdadm has always been resistant to naming problems thanks to its device
> #s and UUIDs in the superblocks.

Yea, I got that and understand... just the OCD in me. (sigh)

As for Reply/Reply-All -- most of the lists I happen to subscribe to "reply-to-list" (through one mechanism or another) as the default for "reply"... :(

The habit is hard to break.

Thanks!

 -Ben

Re: Recommendation on new system Arrays

[Benjammin2068]

On 01/08/2017 10:34 PM, Brad Campbell wrote:
>
> This can be a real trap.
>
> In general CCTV software works great for about the first 2 complete database writes. As the database ages you will find increasing levels of write fragmentation that kill your write latency. It also makes each write smaller as you are no longer streaming bulk data to disk but filling the holes left by expired footage. This destroys your array write latency and makes replay and searching for footage frustratingly slow.
>
> Make sure your CCTV storage RAID is not used for anything else, select a small chunk size to minimise stripe size and be prepared for fairly high search and playback latency.
>
> Most of the major CCTV systems suffer in the same way, and they generally get around it by using "hardware" RAID cards with large RAM buffers (until the BBU wears out and you start getting massive footage loss because they can't get the streams to disk fast enough).
>
> If performance is not an issue then a RAID6 being written from a Windows VM on a Linux host works just fine. Pass the md straight through to the Windows VM and let it manage the raw block device.
>
> It has bee a couple of years since I tested Milestone, but I've just finished a test with Indigo Vision, Aimetis, Genetec, Geutebruck, Bosch, Avigilon & March Networks. They all suffer to a certain degree and all use different storage architectures depending on their legacy. March works really well over SMB shares though. I'm just not fond of the product.
>
> Milestone sent me a new test license before Christmas but I've not got around to spooling it up yet.
>
> All of this stuff gets tested on KVM VM's writing to a couple of small RAID-6 arrays (10-14 drives).

Excellent -- the kind of info I was looking for (and wondering about.)

Thanks for the tip!

 -Ben

Re: Recommendation on new system Arrays

[Wols Lists]

On 10/01/17 08:48, Benjammin2068 wrote:
> Most of the major CCTV systems suffer in the same way, and they generally get around it by using "hardware" RAID cards with large RAM buffers (until the BBU wears out and you start getting massive footage loss because they can't get the streams to disk fast enough).
>>
>> If performance is not an issue then a RAID6 being written from a Windows VM on a Linux host works just fine. Pass the md straight through to the Windows VM and let it manage the raw block device.

This made me think. Bear in mind ext tends to over-allocate space to try
and avoid fragmentation. I don't know to what extent it happens
automatically, but this sounds similar to what Brad is recommending. If
you can match your file system to your raid array, such that the
file-system's default allocation unit is one stride of the raid, this
will help avoid RMW thrashing. Pretty obvious, in hindsight, this will
mean strides (mostly) never get split across files so are only ever
allocated and freed as whole units.

Cheers,
Wol