Re: [PATCH v3] btrfs: add BTRFS_IOC_GET_CSUMS ioctl

Mark Harmstone
2026-04-15T13:56:58Z

Thanks David.

On 14/04/2026 3.07 am, David Sterba wrote:
> On Mon, Apr 13, 2026 at 06:14:16PM +0100, Mark Harmstone wrote:
>> Add a new unprivileged BTRFS_IOC_GET_CSUMS ioctl, which can be used to
>> query the on-disk csums for a file.
>>
>> The ioctl is deliberately per-file rather than exposing raw csum tree
>> lookups, to avoid leaking information to users about files they may not
>> have access to.
>>
>> This is done by userspace passing a struct btrfs_ioctl_get_csums_args to
>> the kernel, which details the offset and length we're interested in, and
>> a buffer for the kernel to write its results into. The kernel writes a
>> struct btrfs_ioctl_get_csums_entry into the buffer, followed by the
>> csums if available.
>>
>> If the extent is an uncompressed, non-nodatasum extent, the kernel sets
>> the entry type to BTRFS_GET_CSUMS_HAS_CSUMS and follows it with the
>> csums. If it is sparse, preallocated, or beyond the EOF, it sets the
>> type to BTRFS_GET_CSUMS_ZEROED - this is so userspace knows it can use
>> the precomputed hash of the zero sector. Otherwise, it sets the type to
>> BTRFS_GET_CSUMS_NODATASUM, BTRFS_GET_CSUMS_COMPRESSED,
>> BTRFS_GET_CSUM_ENCRYPTED, or BTRFS_GET_CSUM_INLINE.
>>
>> For example, a file with a [0, 4K) hole and [4K, 12K) data extent would
>> produce the following output buffer:
>>
>> | [0, 4K) ZEROED | [4K, 12K) HAS_CSUMS | csum data |
>>
>> We do store the csums of compressed extents, but we deliberately don't
>> return them here: they're hashed over the compressed data, not the
>> uncompressed data that's returned to userspace. Similarly for encrypted
>> data, once encryption is supported, in which the csums will be on the
>> ciphertext.
>>
>> The main use case for this is for speeding up mkfs.btrfs --rootdir.
>
> In the past reading the checksums was desired for deduplication, and we
> had some proposed ioctls but I think it was not extensive to also cover
> the holes, compression. Basically a wrapper for the search tree ioctl on
> the checksum tree.
>
> Handling all the types in this patch seems to be complete.
>
>> For
>> the case when the source FS is btrfs and using the same csum algorithm,
>> we can avoid having to recalculate the csums - in my synthetic
>> benchmarks (16GB file on a spinning-rust drive), this resulted in a ~11%
>> speed-up (218s to 196s).
>>
>> When using the --reflink option added in btrfs-progs v6.16.1, we can forgo
>> reading the data entirely, resulting a ~2200% speed-up on the same test
>> (128s to 6s).
>
> Repeated mkfs is a specific use case, normaly it happens just once but I
> understand that for preparing various images the time savings are
> significiant.
>
> Once this ioctl is available I think the deduplication tools will make
> use of it as an initial filter before actually doing the real
> deduplication of extents.

IIRC the mathematics of the birthday paradox is that for an n-bit hash,
you need 2^(n/2) sectors to have a 50% chance of a collision.

For the 32-bit hashes, that means a 50% chance of a collision every 256
MB. For the 256-bit hashes, that's 2^140 bytes... whatever that works
out as. So the dedupe logic would no doubt be that for larger hash sizes
GET_CSUMS itself will be sufficient.

> As this is a unprivileged ioctl and touching structures and memory we'll
> need a security pass on the final version. So far I haven't see any
> issues known to me.
>
>> --- a/fs/btrfs/ioctl.c
>> +++ b/fs/btrfs/ioctl.c
>> @@ -56,6 +56,7 @@
>> #include "uuid-tree.h"
>> #include "ioctl.h"
>> #include "file.h"
>> +#include "file-item.h"
>> #include "scrub.h"
>> #include "super.h"
>>
>> @@ -5139,6 +5140,340 @@ static int btrfs_ioctl_shutdown(struct btrfs_fs_info *fs_info, unsigned long arg
>> }
>> #endif
>>
>> +#define GET_CSUMS_BUF_MAX SZ_16M
>> +
>> +static int copy_csums_to_user(struct btrfs_fs_info *fs_info, u64 disk_bytenr,
>> + u64 len, u8 __user *buf)
>> +{
>> + struct btrfs_root *csum_root;
>> + struct btrfs_ordered_sum *sums;
>> + LIST_HEAD(list);
>> + const u32 csum_size = fs_info->csum_size;
>> + int ret;
>> +
>> + csum_root = btrfs_csum_root(fs_info, disk_bytenr);
>> +
>> + ret = btrfs_lookup_csums_list(csum_root, disk_bytenr,
>> + disk_bytenr + len - 1, &list, false);
>> + if (ret < 0)
>> + return ret;
>> +
>> + /* Clear the output buffer to handle potential gaps in csum coverage. */
>> + if (clear_user(buf, (len >> fs_info->sectorsize_bits) * csum_size)) {
>
> Clearning the partial range here is good but I think it should be done
> globally right at the start of the ioctl, regardless of how many data we
> return.
>
> OTOH this can have a peformance hit in case a large buffer is provided
> by default but for various ranges and possibly short ones, not matching
> the entire buffer. So repeated calls will have to clear 16M but return
> a few kilobytes.

That's fine - this isn't an interface change, so we can always change it
later if it turns out to be a bottleneck. At any rate it's going to be
faster than userspace calculating the csums manually.

>> + ret = -EFAULT;
>> + goto out;
>> + }
>> +
>> + ret = 0;
>> + while (!list_empty(&list)) {
>> + u64 offset;
>> + size_t copy_size;
>> +
>> + sums = list_first_entry(&list, struct btrfs_ordered_sum, list);
>> + list_del(&sums->list);
>> +
>> + offset = ((sums->logical - disk_bytenr) >> fs_info->sectorsize_bits) * csum_size;
>> + copy_size = (sums->len >> fs_info->sectorsize_bits) * csum_size;
>> +
>> + if (copy_to_user(buf + offset, sums->sums, copy_size)) {
>> + kfree(sums);
>> + ret = -EFAULT;
>> + goto out;
>> + }
>> +
>> + kfree(sums);
>> + }
>> +
>> +out:
>> + while (!list_empty(&list)) {
>> + sums = list_first_entry(&list, struct btrfs_ordered_sum, list);
>> + list_del(&sums->list);
>> + kfree(sums);
>> + }
>> + return ret;
>> +}
>> +
>> +static int btrfs_ioctl_get_csums(struct file *file, void __user *argp)
>> +{
>> + struct inode *inode = file_inode(file);
>
> Please use vfs_inode
>
>> + struct btrfs_inode *bi = BTRFS_I(inode);
>
> and plain inode
>
>> + struct btrfs_fs_info *fs_info = bi->root->fs_info;
>> + struct btrfs_root *root = bi->root;
>> + struct btrfs_ioctl_get_csums_args args;
>> + BTRFS_PATH_AUTO_FREE(path);
>> + const u64 ino = btrfs_ino(bi);
>> + const u32 sectorsize = fs_info->sectorsize;
>> + const u32 csum_size = fs_info->csum_size;
>> + u8 __user *ubuf;
>> + u64 buf_limit;
>> + u64 buf_used = 0;
>> + u64 cur_offset;
>> + u64 end_offset;
>> + u64 prev_extent_end;
>> + struct btrfs_key key;
>> + int ret;
>> +
>> + if (!(file->f_mode & FMODE_READ))
>> + return -EBADF;
>> +
>> + if (!S_ISREG(inode->i_mode))
>> + return -EINVAL;
>> +
>> + if (copy_from_user(&args, argp, sizeof(args)))
>> + return -EFAULT;
>> +
>> + if (!IS_ALIGNED(args.offset, sectorsize) ||
>> + !IS_ALIGNED(args.length, sectorsize))
>> + return -EINVAL;
>> + if (args.length == 0)
>> + return -EINVAL;
>> + if (args.offset + args.length < args.offset)
>> + return -EOVERFLOW;
>> + if (args.flags != 0)
>> + return -EINVAL;
>> + if (args.buf_size < sizeof(struct btrfs_ioctl_get_csums_entry))
>> + return -EINVAL;
>> +
>> + buf_limit = min_t(u64, args.buf_size, GET_CSUMS_BUF_MAX);
>> + ubuf = (u8 __user *)(argp + offsetof(struct btrfs_ioctl_get_csums_args, buf));
>> + cur_offset = args.offset;
>> + end_offset = args.offset + args.length;
>> +
>> + path = btrfs_alloc_path();
>> + if (!path)
>> + return -ENOMEM;
>> +
>> + ret = btrfs_wait_ordered_range(bi, cur_offset, args.length);
>> + if (ret)
>> + return ret;
>> +
>> + btrfs_inode_lock(bi, BTRFS_ILOCK_SHARED);
>
> This needs to be interruptible, all other points are. There's no inode
> lock helper for that so you'd have to add it or call
> down_rwsem_interruptible().
>
>> +
>> + ret = btrfs_wait_ordered_range(bi, cur_offset, args.length);
>> + if (ret)
>> + goto out_unlock;
>> +
>> + /* NODATASUM early exit. */
>> + if (bi->flags & BTRFS_INODE_NODATASUM) {
>> + struct btrfs_ioctl_get_csums_entry entry = {
>> + .offset = cur_offset,
>> + .length = end_offset - cur_offset,
>> + .type = BTRFS_GET_CSUMS_NODATASUM,
>> + };
>> +
>> + if (copy_to_user(ubuf, &entry, sizeof(entry))) {
>> + ret = -EFAULT;
>> + goto out_unlock;
>> + }
>> +
>> + buf_used = sizeof(entry);
>> + cur_offset = end_offset;
>> + goto done;
>> + }
>> +
>> + prev_extent_end = cur_offset;
>> +
>> + while (cur_offset < end_offset) {
>> + struct btrfs_file_extent_item *ei;
>> + struct extent_buffer *leaf;
>> + struct btrfs_ioctl_get_csums_entry entry;
>
> Please initialize that right away with = { 0 }, it's done later by
> memset() but for something filled partially it's better to do it at the
> beginning than forgetting to do it later.
>
> Overall comment, please check the final patch that there are not
> unnecessary line breaks that overflow 80 chars only slightly. I've seen
> a few and would have fixed it but I think I've been pointing that out
> enough in the past. I'll fix anything that's left if needed.