XFS reflinks

XFS reflinks

[Chris Cottam]

[-- Attachment #1.1: Type: text/plain, Size: 282 bytes --]

I've got a project where being able to take COW copies of a VMs disc
would be really useful (disc snapshots to be then archived off elsewhere)

How stable is the the experimental reflinks feature and do you have any
idea when it might be declared stable?

Regards

Chris


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: XFS reflinks

[Christoph Hellwig]

On Tue, Aug 29, 2017 at 12:23:13PM +0100, Chris Cottam wrote:
> I've got a project where being able to take COW copies of a VMs disc
> would be really useful (disc snapshots to be then archived off elsewhere)
> 
> How stable is the the experimental reflinks feature and do you have any
> idea when it might be declared stable?

I've got a customer that heavily uses (and tests) reflinks.  They
initially found a few issues, but these days the QA teams finds
bugs in decade old XFS code but not related to reflinks..

You're mileage my vary, but I think we should be able to drop the
experimental tag now.

Re: XFS reflinks

[Chris Cottam]

[-- Attachment #1.1: Type: text/plain, Size: 1194 bytes --]

Could I enquire about the severity of a few issues?

Regard

Chris

On 29/08/17 14:00, Christoph Hellwig wrote:
> On Tue, Aug 29, 2017 at 12:23:13PM +0100, Chris Cottam wrote:
>> I've got a project where being able to take COW copies of a VMs disc
>> would be really useful (disc snapshots to be then archived off elsewhere)
>>
>> How stable is the the experimental reflinks feature and do you have any
>> idea when it might be declared stable?
> 
> I've got a customer that heavily uses (and tests) reflinks.  They
> initially found a few issues, but these days the QA teams finds
> bugs in decade old XFS code but not related to reflinks..
> 
> You're mileage my vary, but I think we should be able to drop the
> experimental tag now.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
Chris Cottam                                   Bytemark Hosting Support
                                             http://www.bytemark.co.uk/
                                              tel: +44 (0) 1904 890 890


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: XFS reflinks

[Christoph Hellwig]

On Tue, Aug 29, 2017 at 02:39:56PM +0100, Chris Cottam wrote:
> Could I enquire about the severity of a few issues?

The most recent ones were:

xfs: try to avoid blowing out the transaction reservation when bunmaping a shared extent

which is a case wher we blew a transaction under extreme unlink
loads, leading to a fs shutdown in those cases.

xfs: try any AG when allocating the first btree block when reflinking

again could lead to a shutdown

xfs: fix COW writeback race

this could lead to a kernel crash

xfs: only reclaim unwritten COW extents periodically

could lead to a kernel crash

Re: XFS reflinks

[Darrick J. Wong]

On Tue, Aug 29, 2017 at 06:00:46AM -0700, Christoph Hellwig wrote:
> On Tue, Aug 29, 2017 at 12:23:13PM +0100, Chris Cottam wrote:
> > I've got a project where being able to take COW copies of a VMs disc
> > would be really useful (disc snapshots to be then archived off elsewhere)
> > 
> > How stable is the the experimental reflinks feature and do you have any
> > idea when it might be declared stable?
> 
> I've got a customer that heavily uses (and tests) reflinks.  They
> initially found a few issues, but these days the QA teams finds
> bugs in decade old XFS code but not related to reflinks..
> 
> You're mileage my vary, but I think we should be able to drop the
> experimental tag now.

I still have (at least) two unresolved questions before we drop the
experimental tag on reflink --

First, should we land the incore extent map rework (not that I've seen a
patchset yet) so that the increased extent map fragmentation resulting
from cow/dedupe don't overwhelm the memory allocator with high order
allocations?  Incore extent map memory usage hasn't been an issue here...

Second, regarding realtime + reflink: Right now we don't forbid mounting
a filesystem with reflink enabled and a realtime device.  You can't have
an inode with both realtime and reflink flags set, so if future xfs
supports it, old kernels won't totally stumble over those inodes.
(Though, EFSCORRUPTED is a little rude.)  Is that ok? or should we (a)
make reflink work on realtime devices, (b) complain at mount time, or
(c) leave everything the way it is?

(The "root a metadata btree in an inode" series, which is the first half
of realtime rmapbt, will enable us to create a refcount btree for the
realtime device, though any reflink-capable realtime device also has a
requirement that sb_rextsize <= page size unless we devise a mechanism
to cow larger blocks, which is also problematic.  I used to have a
realtime reflink patch set but it died when we went to iomap and I
haven't resurrected it.)

--D

> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: XFS reflinks

[Christoph Hellwig]

On Tue, Aug 29, 2017 at 09:03:08AM -0700, Darrick J. Wong wrote:
> First, should we land the incore extent map rework (not that I've seen a
> patchset yet) so that the increased extent map fragmentation resulting
> from cow/dedupe don't overwhelm the memory allocator with high order
> allocations?  Incore extent map memory usage hasn't been an issue here...

Working on this now, but so far this hasn't been a major issue.
The main workload where the extent list currently hurts and that
prompted my work in this area doesn't even involve reflinks (sparse
VM image).

> Second, regarding realtime + reflink: Right now we don't forbid mounting
> a filesystem with reflink enabled and a realtime device.  You can't have
> an inode with both realtime and reflink flags set, so if future xfs
> supports it, old kernels won't totally stumble over those inodes.
> (Though, EFSCORRUPTED is a little rude.)  Is that ok? or should we (a)
> make reflink work on realtime devices, (b) complain at mount time, or
> (c) leave everything the way it is?

I think the current situation is fine - if we add reflinks on the RT
device we can just add a ROCOMPAT flag.

Re: XFS reflinks

[Dave Chinner]

On Tue, Aug 29, 2017 at 09:23:10AM -0700, Christoph Hellwig wrote:
> On Tue, Aug 29, 2017 at 09:03:08AM -0700, Darrick J. Wong wrote:
> > First, should we land the incore extent map rework (not that I've seen a
> > patchset yet) so that the increased extent map fragmentation resulting
> > from cow/dedupe don't overwhelm the memory allocator with high order
> > allocations?  Incore extent map memory usage hasn't been an issue here...
> 
> Working on this now, but so far this hasn't been a major issue.
> The main workload where the extent list currently hurts and that
> prompted my work in this area doesn't even involve reflinks (sparse
> VM image).

Yup, that's a problem, and it's going to be even more of a problem
once people start using reflink to snapshot sparse VM images. I
think that the extent rework is probably the most important thing we
need to solve for COW to be generally usable.

Christoph, ignoring the CoW consideration, this is top of my list of
"things we've been ignoring for too long that need to be fixed
ASAP". I've got the time to work on this right now, so if you don't
have time to get it done in the next merge cycle I can probably make
that happen....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: XFS reflinks

[Christoph Hellwig]

On Wed, Aug 30, 2017 at 08:03:16AM +1000, Dave Chinner wrote:
> Christoph, ignoring the CoW consideration, this is top of my list of
> "things we've been ignoring for too long that need to be fixed
> ASAP". I've got the time to work on this right now, so if you don't
> have time to get it done in the next merge cycle I can probably make
> that happen....

I've been working on it for a while, and it's not just spare time
but customer work.  I've finally found a data structure that fits,
and I've already posted various cleanups to isolate the extent list.

I'll have something ready to post in the next few weeks.

Re: XFS reflinks

[Chris Cottam]

On 29/08/17 23:03, Dave Chinner wrote:
> On Tue, Aug 29, 2017 at 09:23:10AM -0700, Christoph Hellwig wrote:
>> On Tue, Aug 29, 2017 at 09:03:08AM -0700, Darrick J. Wong wrote:
>>> First, should we land the incore extent map rework (not that I've seen a
>>> patchset yet) so that the increased extent map fragmentation resulting
>>> from cow/dedupe don't overwhelm the memory allocator with high order
>>> allocations?  Incore extent map memory usage hasn't been an issue here...
>>
>> Working on this now, but so far this hasn't been a major issue.
>> The main workload where the extent list currently hurts and that
>> prompted my work in this area doesn't even involve reflinks (sparse
>> VM image).
> 
> Yup, that's a problem, and it's going to be even more of a problem
> once people start using reflink to snapshot sparse VM images. I
> think that the extent rework is probably the most important thing we
> need to solve for COW to be generally usable.

Assuming I was feeling brave, what should I be doing to limit the
chances of hitting this issue? Would smaller filesystems or lots of
memory on the host help etc?
Also what should I be monitoring to warn me before anything actually breaks?

Thanks for the help so far.

Chris

Re: XFS reflinks

[Christoph Hellwig]

On Thu, Aug 31, 2017 at 11:33:44AM +0100, Chris Cottam wrote:
> > Yup, that's a problem, and it's going to be even more of a problem
> > once people start using reflink to snapshot sparse VM images. I
> > think that the extent rework is probably the most important thing we
> > need to solve for COW to be generally usable.
> 
> Assuming I was feeling brave, what should I be doing to limit the
> chances of hitting this issue? Would smaller filesystems or lots of
> memory on the host help etc?
> Also what should I be monitoring to warn me before anything actually breaks?

Make sure you have memory compaction enabled is the best bet,
don't run under crazy memory pressure.  I've not actually seen an
actual deadlock due to it, just the possible memory allocation
deadlock warnings from kmem_alloc (and lots of those).