[RFCv4 0/2] man-pages: document reflink/dedupe ioctls

[RFCv4 0/2] man-pages: document reflink/dedupe ioctls

[Darrick J. Wong]

Hi all,

This patch set goes along with the fourth revision of an RFC adding to
XFS kernel support for tracking reverse-mappings of physical blocks to
file and metadata; and support for mapping multiple file logical
blocks to the same physical block, more commonly known as reflinking.

The two patches in this series document the clone, clone_range, and
dedupe ioctls in the form of man pages.

The patch set is based on a whole pile of proposed patches against the
current (4.4-rc5) upstream kernel; the ioctls are simply hoisted
version of the private btrfs ioctls.

If you're going to start using this mess, you probably ought to just
pull from my github trees for kernel[1], xfsprogs[2], and xfstests[3].
See also the xfs-docs[4] and manpage[5] updates.

This is an extraordinary way to eat your data.  Enjoy!

Comments and questions are, as always, welcome.

--D

[1] https://github.com/djwong/linux/tree/for-dave
[2] https://github.com/djwong/xfsprogs/tree/for-dave
[3] https://github.com/djwong/xfstests/tree/for-dave
[4] https://github.com/djwong/xfs-documentation/tree/for-dave
[5] https://github.com/djwong/man-pages/commits/for-mtk

[PATCH 1/2] man2: document FICLONE and FICLONERANGE

[Darrick J. Wong]

Document the FICLONE and FICLONERANGE ioctls, formerly known as the
BTRFS_IOC_CLONE and BTRFS_IOC_CLONE_RANGE ioctls.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 man2/ioctl_ficlone.2      |   89 +++++++++++++++++++++++++++++++++++
 man2/ioctl_ficlonerange.2 |  116 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 205 insertions(+)
 create mode 100644 man2/ioctl_ficlone.2
 create mode 100644 man2/ioctl_ficlonerange.2


diff --git a/man2/ioctl_ficlone.2 b/man2/ioctl_ficlone.2
new file mode 100644
index 0000000..ba851da
--- /dev/null
+++ b/man2/ioctl_ficlone.2
@@ -0,0 +1,89 @@
+.\" Copyright (C) 2015 Oracle.  All rights reserved.
+.\"
+.\" %%%LICENSE_START(VERBATIM)
+.\" This program is free software; you can redistribute it and/or
+.\" modify it under the terms of the GNU General Public License as
+.\" published by the Free Software Foundation.
+.\"
+.\" This program is distributed in the hope that it would be useful,
+.\" but WITHOUT ANY WARRANTY; without even the implied warranty of
+.\" MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+.\" GNU General Public License for more details.
+.\"
+.\" You should have received a copy of the GNU General Public License
+.\" along with this program; if not, write the Free Software Foundation,
+.\" Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+.\" %%%LICENSE_END
+.TH IOCTL-FICLONE 2 2015-12-15 "Linux" "Linux Programmer's Manual"
+.SH NAME
+ioctl_ficlone \- share all the data of one file with another file
+.SH SYNOPSIS
+.br
+.B #include <sys/ioctl.h>
+.sp
+.BI "int ioctl(int " dest_fd ", FICLONE, int " src_fd );
+.SH DESCRIPTION
+If a filesystem supports files sharing physical storage between multiple
+files ("reflink"), this
+.BR ioctl (2)
+system call can be used to make all the data in the
+.B src_fd
+file appear in the
+.B dest_fd
+file by sharing the underlying storage, which is faster than making a separate
+physical copy of the data.  If a file write should occur to a shared region,
+the filesystem must ensure that the changes remain private to the file being
+written.  This behavior is commonly referred to as "copy on write".
+.SH RETURN VALUE
+On error, \-1 is returned, and
+.I errno
+is set to indicate the error.
+.PP
+.SH ERRORS
+Error codes can be one of, but are not limited to, the following:
+.TP
+.B EXDEV
+.IR dest_fd " and " src_fd
+are not on the same mounted filesystem.
+.TP
+.B EISDIR
+One of the files is a directory and the filesystem does not support shared
+regions in directories.
+.TP
+.B EINVAL
+The filesystem does not support reflinking the ranges of the given files.  This
+error can also appear if either file descriptor represents a device, fifo, or
+socket.
+.TP
+.B EBADF
+.IR src_fd
+is not open for reading;
+.IR dest_fd
+is not open for writing or is open for append-only writes; or the filesystem
+which
+.IR src_fd
+resides on does not support reflink.
+.TP
+.B EPERM
+.IR dest_fd
+is immutable.
+.TP
+.B ETXTBSY
+One of the files is a swap file.  Swap files cannot share storage.
+.TP
+.B EOPNOTSUPP
+This can appear if the filesystem does not support reflinking either file
+descriptor.
+.SH NOTES
+Because a copy on write operation requires the allocation of new storage, the
+.B fallocate (2)
+operation may un-share shared blocks to guarantee that subsequent writes will
+not fail because of lack of disk space.
+.SH CONFORMING TO
+This API is Linux-specific.  This ioctl was previously known as
+.B BTRFS_IOC_CLONE
+and was private to btrfs.
+.fi
+.in
+.SH SEE ALSO
+.BR ioctl (2)
diff --git a/man2/ioctl_ficlonerange.2 b/man2/ioctl_ficlonerange.2
new file mode 100644
index 0000000..80263c5
--- /dev/null
+++ b/man2/ioctl_ficlonerange.2
@@ -0,0 +1,116 @@
+.\" Copyright (C) 2015 Oracle.  All rights reserved.
+.\"
+.\" %%%LICENSE_START(VERBATIM)
+.\" This program is free software; you can redistribute it and/or
+.\" modify it under the terms of the GNU General Public License as
+.\" published by the Free Software Foundation.
+.\"
+.\" This program is distributed in the hope that it would be useful,
+.\" but WITHOUT ANY WARRANTY; without even the implied warranty of
+.\" MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+.\" GNU General Public License for more details.
+.\"
+.\" You should have received a copy of the GNU General Public License
+.\" along with this program; if not, write the Free Software Foundation,
+.\" Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+.\" %%%LICENSE_END
+.TH IOCTL-FICLONERANGE 2 2015-12-15 "Linux" "Linux Programmer's Manual"
+.SH NAME
+ioctl_ficlonerange \- share some the data of one file with another file
+.SH SYNOPSIS
+.br
+.B #include <sys/ioctl.h>
+.sp
+.BI "int ioctl(int " dest_fd ", FICLONERANGE, struct file_clone_range * " arg );
+.SH DESCRIPTION
+If a filesystem supports files sharing physical storage between multiple
+files ("reflink"), this
+.BR ioctl (2)
+system call can be used to make some of the data in the
+.B src_fd
+file appear in the
+.B dest_fd
+file by sharing the underlying storage, which is faster than making a separate
+physical copy of the data.  If a file write should occur to a shared region,
+the filesystem must ensure that the changes remain private to the file being
+written.  This behavior is commonly referred to as "copy on write".
+
+This ioctl reflinks up to
+.IR src_length
+bytes from file descriptor
+.IR src_fd
+at offset
+.IR src_offset
+into the file
+.IR dest_fd
+at offset
+.IR dest_offset ",
+provided that both are files.  This information is conveyed in a structure of
+the following form:
+.in +4n
+.nf
+
+struct file_clone_range {
+        __s64 src_fd;
+        __u64 src_offset;
+        __u64 src_length;
+        __u64 dest_offset;
+};
+
+.fi
+.in
+.SH RETURN VALUE
+On error, \-1 is returned, and
+.I errno
+is set to indicate the error.
+.PP
+.SH ERRORS
+Error codes can be one of, but are not limited to, the following:
+.TP
+.B EXDEV
+.IR dest_fd " and " src_fd
+are not on the same mounted filesystem.
+.TP
+.B EISDIR
+One of the files is a directory and the filesystem does not support shared
+regions in directories.
+.TP
+.B EINVAL
+The filesystem does not support reflinking the ranges of the given files.  This
+error can also appear if either file descriptor represents a device, fifo, or
+socket.  Disk filesystems generally require the offset and length arguments
+to be aligned to the fundamental block size.  Not all filesystems support
+overlapping reflink ranges in the same file.
+.TP
+.B EBADF
+.IR src_fd
+is not open for reading;
+.IR dest_fd
+is not open for writing or is open for append-only writes; or the filesystem
+which
+.IR src_fd
+resides on does not support reflink.
+.TP
+.B EPERM
+.IR dest_fd
+is immutable.
+.TP
+.B ETXTBSY
+One of the files is a swap file.  Swap files cannot share storage.
+.TP
+.B EOPNOTSUPP
+This can appear if the filesystem does not support reflinking either file
+descriptor.
+.SH NOTES
+Because a copy on write operation requires the allocation of new storage, the
+.B fallocate (2)
+operation may un-share shared blocks to guarantee that subsequent writes will
+not fail because of lack of disk space.
+.SH CONFORMING TO
+This API is Linux-specific.  This ioctl was previously known as
+.B BTRFS_IOC_CLONE_RANGE
+and was private to btrfs.
+.fi
+.in
+.SH SEE ALSO
+.BR ioctl (2)

Re: [PATCH 1/2] man2: document FICLONE and FICLONERANGE

[Christoph Hellwig]

Now that we have this in 4.5-rc1 let's get this into man-pages as well.

> --- /dev/null
> +++ b/man2/ioctl_ficlone.2

think this should be a link to the clone range man page, and the
both ioctls should be documented together.  Basically just document
FICLONERANGE and then mentiond that FICLONE does a whole file clone.

> +.B #include <sys/ioctl.h>

Don't we need uapi/linux/fs.h as well?

> Not all filesystems support
> +overlapping reflink ranges in the same file.

Which one doesn't?


Also there should be a sentence that the clones are atomic vs
concurrent writes, so no locks need to be taken for a consistent
snapshot.

Re: [PATCH 1/2] man2: document FICLONE and FICLONERANGE

[Darrick J. Wong]

On Thu, Jan 28, 2016 at 05:33:29AM -0800, Christoph Hellwig wrote:
> Now that we have this in 4.5-rc1 let's get this into man-pages as well.
> 
> > --- /dev/null
> > +++ b/man2/ioctl_ficlone.2
> 
> think this should be a link to the clone range man page, and the
> both ioctls should be documented together.  Basically just document
> FICLONERANGE and then mentiond that FICLONE does a whole file clone.

Ok, that's an easy switch to make.

> > +.B #include <sys/ioctl.h>
> 
> Don't we need uapi/linux/fs.h as well?

Err... yes. :)

> > Not all filesystems support
> > +overlapping reflink ranges in the same file.
> 
> Which one doesn't?

At the moment, neither btrfs nor XFS support it.

I don't think it'd be difficult for XFS (make a copy of the extent mappings in
the source area, increase the refcounts of those blocks, do the usual
punch-and-remap, then delete & decrement that copy of the mappings) but that
can come later.

> Also there should be a sentence that the clones are atomic vs
> concurrent writes, so no locks need to be taken for a consistent
> snapshot.

<nod>

I've updated the manpages and will repush the patchset ... optimistically,
when XFS reflink RFCv5 comes out, but sooner if anyone requests it. :)

Thank you for the review!

--D

> --
> To unsubscribe from this list: send the line "unsubscribe linux-api" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/2] man2: document FICLONE and FICLONERANGE

[Christoph Hellwig]

Now that 4.5 has been released with the generic clone ioctls we should
get the man page included.  Do you want me to update it?

Re: [PATCH 1/2] man2: document FICLONE and FICLONERANGE

[Darrick J. Wong]

On Tue, Mar 15, 2016 at 01:27:37AM -0700, Christoph Hellwig wrote:
> Now that 4.5 has been released with the generic clone ioctls we should
> get the man page included.  Do you want me to update it?

I'll push out an updated patchset shortly.

--D

> --
> To unsubscribe from this list: send the line "unsubscribe linux-api" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 2/2] man2: document the FIDEDUPERANGE ioctl

[Darrick J. Wong]

Document the FIDEDUPERANGE ioctl, formerly known as BTRFS_IOC_EXTENT_SAME.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 man2/ioctl_fideduperange.2 |  164 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 164 insertions(+)
 create mode 100644 man2/ioctl_fideduperange.2


diff --git a/man2/ioctl_fideduperange.2 b/man2/ioctl_fideduperange.2
new file mode 100644
index 0000000..a8a19d0
--- /dev/null
+++ b/man2/ioctl_fideduperange.2
@@ -0,0 +1,164 @@
+.\" Copyright (C) 2015 Oracle.  All rights reserved.
+.\"
+.\" %%%LICENSE_START(VERBATIM)
+.\" This program is free software; you can redistribute it and/or
+.\" modify it under the terms of the GNU General Public License as
+.\" published by the Free Software Foundation.
+.\"
+.\" This program is distributed in the hope that it would be useful,
+.\" but WITHOUT ANY WARRANTY; without even the implied warranty of
+.\" MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+.\" GNU General Public License for more details.
+.\"
+.\" You should have received a copy of the GNU General Public License
+.\" along with this program; if not, write the Free Software Foundation,
+.\" Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+.\" %%%LICENSE_END
+.TH IOCTL-FIDEDUPERANGE 2 2015-12-15 "Linux" "Linux Programmer's Manual"
+.SH NAME
+ioctl_fideduperange \- share some the data of one file with another file
+.SH SYNOPSIS
+.br
+.B #include <sys/ioctl.h>
+.sp
+.BI "int ioctl(int " src_fd ", FIDEDUPERANGE, struct file_dedupe_range * " arg );
+.SH DESCRIPTION
+If a filesystem supports files sharing physical storage between multiple
+files, this
+.BR ioctl (2)
+system call can be used to make some of the data in the
+.B src_fd
+file appear in the
+.B dest_fd
+file by sharing the underlying storage if the file data is identical
+("deduplication").  This reduces storage consumption by allowing the filesystem
+to store one shared copy of the data.  If a file write should occur to a shared
+region, the filesystem must ensure that the changes remain private to the file
+being written.  This behavior is commonly referred to as "copy on write".
+
+This ioctl performs the "compare and share if identical" operation on up to
+.IR src_length
+bytes from file descriptor
+.IR src_fd
+at offset
+.IR src_offset ".
+This information is conveyed in a structure of the following form:
+.in +4n
+.nf
+
+struct file_dedupe_range {
+        __u64 src_offset;
+        __u64 src_length;
+        __u16 dest_count;
+        __u16 reserved1;
+        __u32 reserved2;
+        struct file_dedupe_range_info info[0];
+};
+.fi
+.in
+
+The fields 
+.IR reserved1 " and " reserved2
+must be zero.
+
+Destinations for the deduplication operation are conveyed in the array at the
+end of the structure.  The number of destinations is given in
+.IR dest_count ",
+and the destination information is conveyed in the following form:
+
+.in +4n
+.nf
+struct file_dedupe_range_info {
+        __s64 dest_fd;
+        __u64 dest_offset;
+        __u64 bytes_deduped;
+        __s32 status;
+        __u32 reserved;
+};
+
+.fi
+.in
+
+Each deduplication operation targets
+.IR length
+bytes in file descriptor
+.IR dest_fd
+at offset
+.IR logical_offset ".
+The field
+.IR reserved
+must be zero.
+
+Upon successful completion of this ioctl, the number of bytes successfully
+deduplicated is returned in
+.IR bytes_deduped
+and a status code for the deduplication operation is returned in
+.IR status ".
+
+The
+.IR status
+code is set to
+.B 0
+for success, a negative error code in case of error, or
+.B FILE_DEDUPE_RANGE_DIFFERS
+if the data did not match.
+
+.SH RETURN VALUE
+On error, \-1 is returned, and
+.I errno
+is set to indicate the error.
+.PP
+.SH ERRORS
+Error codes can be one of, but are not limited to, the following:
+.TP
+.B EXDEV
+.IR dest_fd " and " src_fd
+are not on the same mounted filesystem.
+.TP
+.B EISDIR
+One of the files is a directory and the filesystem does not support shared
+regions in directories.
+.TP
+.B EINVAL
+The filesystem does not support deduplicating the ranges of the given files.
+This error can also appear if either file descriptor represents a device, fifo,
+or socket.  Disk filesystems generally require the offset and length arguments
+to be aligned to the fundamental block size.  Not all filesystems support
+overlapping deduplication ranges in the same file.
+.TP
+.B EBADF
+.IR src_fd
+is not open for reading;
+.IR dest_fd
+is not open for writing or is open for append-only writes; or the filesystem
+which
+.IR src_fd
+resides on does not support deduplication.
+.TP
+.B EPERM
+.IR dest_fd
+is immutable.
+.TP
+.B ETXTBSY
+One of the files is a swap file.  Swap files cannot share storage.
+.TP
+.B EOPNOTSUPP
+This can appear if the filesystem does not support deduplicating either file
+descriptor.
+.SH NOTES
+Because a copy on write operation requires the allocation of new storage, the
+.B fallocate (2)
+operation may un-share shared blocks to guarantee that subsequent writes will
+not fail because of lack of disk space.
+
+Some filesystems may limit the amount of data that can be deduplicated in a
+single call.
+
+.SH CONFORMING TO
+This API is Linux-specific.  This ioctl was previously known as
+.B BTRFS_IOC_FILE_EXTENT_SAME
+and was private to btrfs.
+.fi
+.in
+.SH SEE ALSO
+.BR ioctl (2)