Disabling barriers on NVC-backed HDD

Disabling barriers on NVC-backed HDD

[Gionatan Danti]

Hi list,
I'm not sure this is the correct list to post my question; if so, feel 
free to ignore this message.

On XFS FAQs I (rightfully) read that barrier should be *always* enabled 
(and they are, by default), unless a write-loss protected writeback 
cache is used at the RAID card level. So far, so good.

Newer HDDs has themselves have a non-volatile cache (NVC) of some sort, 
basically:
- DRAM + eMCL: in a powerloss event, the DRAM cache is immediately 
dumped on the eMLC flash;
- DRAM + NOR flash: NOR mirrors a (small) portion of the DRAM cache, 
used for write acceleration purpose;
- dedicated "dump" areas on the disks: they effectively mirror a portion 
of the DRAM cache.

My question is: do you think it is safe to disable barriers, both for 
XFS and in general terms, on these disks? Or they should be considered 
as the same "dumb" unprotected DRAM caches found on classical HDD?

 From a side, these *are* powerloss-protected caches. Problem is that all 
these power-protection schemes are considered "secret sauce / trade 
secret" by HDD vendors and, for this reason, there are very little (if 
any) informations on their inner working.

Regards.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: Disabling barriers on NVC-backed HDD

[Darrick J. Wong]

On Wed, Nov 15, 2017 at 06:01:17PM +0100, Gionatan Danti wrote:
> Hi list,
> I'm not sure this is the correct list to post my question; if so, feel free
> to ignore this message.
> 
> On XFS FAQs I (rightfully) read that barrier should be *always* enabled (and
> they are, by default), unless a write-loss protected writeback cache is used
> at the RAID card level. So far, so good.
> 
> Newer HDDs has themselves have a non-volatile cache (NVC) of some sort,
> basically:
> - DRAM + eMCL: in a powerloss event, the DRAM cache is immediately dumped on
> the eMLC flash;
> - DRAM + NOR flash: NOR mirrors a (small) portion of the DRAM cache, used
> for write acceleration purpose;
> - dedicated "dump" areas on the disks: they effectively mirror a portion of
> the DRAM cache.
> 
> My question is: do you think it is safe to disable barriers, both for XFS
> and in general terms, on these disks? Or they should be considered as the
> same "dumb" unprotected DRAM caches found on classical HDD?
> 
> From a side, these *are* powerloss-protected caches. Problem is that all
> these power-protection schemes are considered "secret sauce / trade secret"
> by HDD vendors and, for this reason, there are very little (if any)
> informations on their inner working.

I would evaluate these drives to find out if you really /can/ yank the
power without losing anything.

That said, if the manufacturers aren't willing to tell you how that
feature works, I'd just as soon pretend the feature didn't exist and
continue sending flushes to the drive.

FWIW if the drive really /does/ have a non-volatile WC then a flush
should have nearly zero overhead.  (Or so you'd think...)

--D

> Regards.
> 
> -- 
> Danti Gionatan
> Supporto Tecnico
> Assyoma S.r.l. - www.assyoma.it
> email: g.danti@assyoma.it - info@assyoma.it
> GPG public key ID: FF5F32A8
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Disabling barriers on NVC-backed HDD

[Gionatan Danti]

Il 15-11-2017 18:31 Darrick J. Wong ha scritto:
> 
> I would evaluate these drives to find out if you really /can/ yank the
> power without losing anything.

If I can get some hard drives for testing, I'll surely do that. However, 
any in-house testing inevitably has limited scope and it can miss some 
important failure modes, so I am interested in other users' first-hand 
experience.

> That said, if the manufacturers aren't willing to tell you how that
> feature works, I'd just as soon pretend the feature didn't exist and
> continue sending flushes to the drive.

 From a point, I agree: I do not like black boxes. On the other hand, 
hardware RAID controllers *are* black boxes, and we all use them...

> FWIW if the drive really /does/ have a non-volatile WC then a flush
> should have nearly zero overhead.  (Or so you'd think...)

 From my understanding (and I can be wrong), these drives honour cache 
flushes, with the associated performance drop.

Ok, too much conjectures from my side, I think. I should *really* get my 
hand on some of these drives...

Regards.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: Disabling barriers on NVC-backed HDD

[Christoph Hellwig]

On Wed, Nov 15, 2017 at 09:31:00AM -0800, Darrick J. Wong wrote:
> I would evaluate these drives to find out if you really /can/ yank the
> power without losing anything.
> 
> That said, if the manufacturers aren't willing to tell you how that
> feature works, I'd just as soon pretend the feature didn't exist and
> continue sending flushes to the drive.
> 
> FWIW if the drive really /does/ have a non-volatile WC then a flush
> should have nearly zero overhead.  (Or so you'd think...)

And in that case it will report WCE=0 and Linux won't even flush.
As is the case for typical enterprise disks.

Re: Disabling barriers on NVC-backed HDD

[Gionatan Danti]

Il 15-11-2017 20:47 Christoph Hellwig ha scritto:
> And in that case it will report WCE=0 and Linux won't even flush.
> As is the case for typical enterprise disks.

Good point. Based on what I read here[1], page n.4, it *seems* that 
Seagate AWC (advanced write cache) enabled drives report WCD (write 
cache disabled) to the host OS.

I think this is the key parameter to watch: if the HDD vendor is 
confident enough to report WCE=0/WCD, than it should be safe running 
without barriers (after all, in this case Linux *will* disable barrier 
by default).

Thank you Christoph for pointing that.
Regards.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: Disabling barriers on NVC-backed HDD

[Gionatan Danti]

Il 15-11-2017 21:17 Gionatan Danti ha scritto:
> Il 15-11-2017 20:47 Christoph Hellwig ha scritto:
>> And in that case it will report WCE=0 and Linux won't even flush.
>> As is the case for typical enterprise disks.
> 
> Good point. Based on what I read here[1], page n.4, it *seems* that
> Seagate AWC (advanced write cache) enabled drives report WCD (write
> cache disabled) to the host OS.
> 
> I think this is the key parameter to watch: if the HDD vendor is
> confident enough to report WCE=0/WCD, than it should be safe running
> without barriers (after all, in this case Linux *will* disable barrier
> by default).
> 
> Thank you Christoph for pointing that.
> Regards.

Sorry, I forgot the link: 
https://www.seagate.com/files/www-content/product-content/enterprise-performance-savvio-fam/enterprise-performance-15k-hdd/_cross-product/_shared/doc/enchanced-cache-advantage-tp691.1-1610us.pdf

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: Disabling barriers on NVC-backed HDD

[Gionatan Danti]

Il 15-11-2017 21:18 Gionatan Danti ha scritto:
> Sorry, I forgot the link:
> https://www.seagate.com/files/www-content/product-content/enterprise-performance-savvio-fam/enterprise-performance-15k-hdd/_cross-product/_shared/doc/enchanced-cache-advantage-tp691.1-1610us.pdf

Ok, it seems that for Seagate drives the real document to read is that 
one[1]. Quoting:

"When WCE=0, Advanced provides NVC-protected write caching over the 
portion of the DRAM used to coalesce writes. Write data only goes into 
NVC when there is an unexpected power loss to the drive. The NVC has 
90-day data retention. When WCE=1, a Advanced Caching drive will operate 
on writes like a standard drive–writes in cache are not protected by NVC 
and may be lost with power loss."

In short:
- WCE=1 -> entire cache enabled -> powerloss = dataloss;
- WCE=0 -> small write cache segment enabled -> powerloss = nodataloss.

Unfortunately, I can not find anything on that regard about HGST's 
MediaCache...

Regards.

[1] 
https://www.seagate.com/www-content/product-content/enterprise-performance-savvio-fam/ent-perf-10k-hdd-v9-skybolt/en-us/docs/100818015c.pdf

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8