Concurrent FIEMAP on reflinked files serialises on the AGF buffer lock

Concurrent FIEMAP on reflinked files serialises on the AGF buffer lock

[Lauri Gates]

Hi,

FIEMAP on files with XFS_DIFLAG2_REFLINK set does not scale across
threads. For every written extent, xfs_read_iomap_begin() calls
xfs_reflink_trim_around_shared() (IOMAP_REPORT), which goes to
xfs_reflink_find_shared() and reads the AGF of the extent's AG with
xfs_alloc_read_agf(pag, NULL, 0, &agbp). Concurrent callers queue on
that buffer's b_sema. A parallel du-style scan of a reflinked tree
stops scaling at roughly the AG count, and threads working in the same
AG run slower than one thread alone.

My question, in more detail at the end: does reporting the shared flag
for FIEMAP need the AGF buffer locked?

I ran into this adding a shared-block mode to dust, a du clone, which
FIEMAPs every file and counts FIEMAP_EXTENT_SHARED extents once.
Inodes without the reflink flag return early and are unaffected.

Wait stack, kernel 7.0.0-1014-gcp: bpftrace counted the scanner's
sched:sched_switch events into TASK_UNINTERRUPTIBLE, keyed by kernel
stack, for about 6 seconds of repeated scans. 730,556 switches had this
stack; the next most common had 47:

  down+88
  xfs_buf_lock+50
  xfs_buf_find_lock+106
  xfs_buf_lookup.constprop.0+318
  xfs_buf_get_map+249
  xfs_buf_read_map+83
  xfs_trans_read_buf_map+313
  xfs_read_agf+183
  xfs_alloc_read_agf+91
  xfs_reflink_find_shared+154
  xfs_reflink_trim_around_shared+235
  xfs_read_iomap_begin+356
  iomap_iter+304
  iomap_fiemap+409
  xfs_vn_fiemap+92
  do_vfs_ioctl+758
  __x64_sys_ioctl+131

7.2 on arm64 shows the same chain. The functions involved are
unchanged in v7.4-rc1 and xfs for-next; since v7.3,
xfs_buf_read_map() calls xfs_buf_find_lock() directly, which takes the
same lock.

Reproducer
----------

fiemap-mt.c and run.sh are below. For each AG count, run.sh makes an
8 GiB loop image (mkfs.xfs -d agcount=N -m reflink=1) and writes
100,000 files of 4-32 KiB into each of plain/ and src/. It then makes
a reflink copy of src/ with cp -a --reflink=always. fiemap-mt lists
the files first, then times open + FS_IOC_FIEMAP + close of each one
from N threads; the figures are medians of 5 warm-cache runs. Order
"seq" hands concurrent threads neighbouring files in one directory, so
they share an AG; "shuffle" uses a fixed-seed random order, so they
mostly hit different AGs.

Wall time, with voluntary context switches in brackets, reflinked src/:

GCP n2-standard-8 (8 vCPUs, x86_64), Ubuntu 24.04, 7.0.0-1014-gcp:

  order    AGs  1 thread        4 threads           8 threads
  shuffle    4  313.6 ms (1)    153.6 ms (19,859)   209.5 ms (38,629)
  shuffle   32  308.7 ms (1)     96.8 ms (1,670)     78.7 ms (3,751)
  seq        4  272.7 ms (1)    587.5 ms (98,954)   922.1 ms (98,868)
  seq       32  268.8 ms (1)    621.4 ms (99,220)   892.3 ms (99,022)

On this machine, with threads spread over AGs, 8 threads take 2.7x as
long on 4 AGs as on 32. With threads in one AG, 8 threads take 3.3x
to 3.4x as long as one thread, whatever the AG count. bpftrace during
these runs shows the same frames and offsets as above, from down+88 to
xfs_read_iomap_begin+356, where its 20-frame capture ends.

Same run on an arm64 VM (7.2.12, 13 vCPUs):

  order    AGs  1 thread        4 threads           13 threads
  shuffle    4  190.8 ms (1)    184.6 ms (15,761)   237.1 ms (28,167)
  shuffle   32  194.7 ms (1)     69.7 ms (1,389)    120.9 ms (6,291)
  seq        4   95.0 ms (1)    745.0 ms (74,991)   144.4 ms (32,586)
  seq       32  100.9 ms (1)    859.2 ms (80,052)   160.7 ms (33,328)

Here seq is worst at 4 threads (7.8x one thread) and recovers somewhat
at 13. Times are not monotonic in thread count on either machine; I
have not looked into why.

On plain/ (never reflinked) no configuration on either machine
exceeds 10 voluntary context switches.

The original workload, for scale: dust scanning the Linux 6.12 source
tree (86,618 files) plus a reflink copy of it, on the same GCP machine
type, at dust's default thread count, mean of 10 runs:

  kernel  4 AGs     32 AGs    dust's normal mode, no FIEMAP (either)
  7.0     506.3 ms  274.8 ms  105-106 ms
  5.15    476.6 ms  277.7 ms  104-106 ms

On 7.0, voluntary context switches fell from 67,089 with 4 AGs to
4,230 with 32. With 4 AGs, dust -T 8 ran slower than -T 4 (513 vs
419 ms). On ext4 the FIEMAP pass added nothing measurable (59.5 vs
60.6 ms, tree without a copy); on btrfs it cost 3.3x but with 131-300
voluntary context switches, so no lock queueing.

Question
--------

Does reporting the shared flag for FIEMAP need the AGF buffer locked?
On this path the refcount btree lookup runs with tp == NULL, under the
inode lock from xfs_ilock_data_map_shared(), and the FIEMAP result is
a snapshot that can be stale by the time userspace reads it.

I am not suggesting dropping the flag. Commit 5f9268ca53ac ("xfs:
don't bother looking at the refcount tree for reads") kept
IOMAP_REPORT on the refcount lookup deliberately, and Dave described
the shared flag as a field diagnostic for reflinked files:
https://lore.kernel.org/all/20210124214346.GC4626@dread.disaster.area/

What I would like to know is whether an unlocked or cached lookup is
acceptable here, for example building on 2aedf844be1d ("xfs: move
buffer locking out of xfs_find_get_buf", in v7.3), which prepares
lookups that do not lock the buffer. If it is, I can test patches,
or try writing one with guidance.

Thanks,
Lauri Gates

---- fiemap-mt.c ----
// fiemap-mt: dust-free reproducer for XFS FIEMAP contention on reflinked files.
//
//   fiemap-mt populate DIR NFILES      create NFILES small files (4-32 KiB, deterministic
//                                      content) in subdirectories of 1000
//   fiemap-mt run DIR THREADS [ORDER]  list regular files under DIR, then open + FIEMAP
//                                      each one from THREADS threads; print wall time,
//                                      voluntary/involuntary context switches, and
//                                      extent counts
//
// ORDER is `seq` (default: directory order, so concurrent threads work on
// neighbouring files, which XFS keeps in the same AG) or `shuffle` (a fixed-seed
// random order, so concurrent threads usually hit different AGs, closer to a
// parallel tree walker like dust).
//
// The file list is built before timing starts, so the timed section is only
// open/FS_IOC_FIEMAP/close, the calls dust --shared-blocks makes per file.
#define _GNU_SOURCE
#include <dirent.h>
#include <errno.h>
#include <fcntl.h>
#include <linux/fiemap.h>
#include <linux/fs.h>
#include <pthread.h>
#include <stdatomic.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/ioctl.h>
#include <sys/resource.h>
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>

#define NEXT 64

static char **paths;
static size_t npaths, cap;
static atomic_size_t next_idx;
static atomic_long n_ext, n_shared_ext, n_shared_files, n_err;

static void die(const char *what) { perror(what); exit(1); }

static void add_path(const char *p) {
  if (npaths == cap) {
    cap = cap ? cap * 2 : 4096;
    paths = realloc(paths, cap * sizeof(*paths));
    if (!paths) die("realloc");
  }
  paths[npaths++] = strdup(p);
}

static void list_files(const char *dir) {
  DIR *d = opendir(dir);
  if (!d) die(dir);
  struct dirent *de;
  char p[4096];
  while ((de = readdir(d))) {
    if (!strcmp(de->d_name, ".") || !strcmp(de->d_name, "..")) continue;
    snprintf(p, sizeof(p), "%s/%s", dir, de->d_name);
    struct stat st;
    if (lstat(p, &st)) continue;
    if (S_ISDIR(st.st_mode)) list_files(p);
    else if (S_ISREG(st.st_mode) && st.st_blocks > 0) add_path(p);
  }
  closedir(d);
}

static void fiemap_file(const char *p) {
  int fd = open(p, O_RDONLY | O_NOFOLLOW | O_CLOEXEC | O_NOATIME);
  if (fd < 0) fd = open(p, O_RDONLY | O_NOFOLLOW | O_CLOEXEC);
  if (fd < 0) { atomic_fetch_add(&n_err, 1); return; }
  char buf[sizeof(struct fiemap) + NEXT * sizeof(struct fiemap_extent)];
  struct fiemap *fm = (struct fiemap *)buf;
  uint64_t start = 0;
  long ext = 0, shared = 0;
  for (;;) {
    memset(fm, 0, sizeof(*fm));
    fm->fm_start = start;
    fm->fm_length = FIEMAP_MAX_OFFSET - start;
    fm->fm_extent_count = NEXT;
    if (ioctl(fd, FS_IOC_FIEMAP, fm) < 0) { atomic_fetch_add(&n_err, 1); break; }
    if (fm->fm_mapped_extents == 0) break;
    int last = 0;
    for (unsigned i = 0; i < fm->fm_mapped_extents; i++) {
      struct fiemap_extent *e = &fm->fm_extents[i];
      ext++;
      if (e->fe_flags & FIEMAP_EXTENT_SHARED) shared++;
      if (e->fe_flags & FIEMAP_EXTENT_LAST) last = 1;
      start = e->fe_logical + e->fe_length;
    }
    if (last || fm->fm_mapped_extents < NEXT) break;
  }
  close(fd);
  atomic_fetch_add(&n_ext, ext);
  atomic_fetch_add(&n_shared_ext, shared);
  if (shared) atomic_fetch_add(&n_shared_files, 1);
}

static void *worker(void *arg) {
  (void)arg;
  for (;;) {
    size_t i = atomic_fetch_add(&next_idx, 1);
    if (i >= npaths) return NULL;
    fiemap_file(paths[i]);
  }
}

static double now_ms(void) {
  struct timespec t;
  clock_gettime(CLOCK_MONOTONIC, &t);
  return t.tv_sec * 1e3 + t.tv_nsec / 1e6;
}

static void shuffle_paths(void) {
  uint64_t x = 0x2545F4914F6CDD1Dull;
  for (size_t i = npaths; i > 1; i--) {
    x ^= x << 13; x ^= x >> 7; x ^= x << 17;
    size_t j = x % i;
    char *t = paths[i - 1]; paths[i - 1] = paths[j]; paths[j] = t;
  }
}

static int cmd_run(const char *dir, int threads, const char *order) {
  list_files(dir);
  if (!strcmp(order, "shuffle")) shuffle_paths();
  else if (strcmp(order, "seq")) { fprintf(stderr, "ORDER must be seq or shuffle\n"); return 2; }
  pthread_t *tids = calloc(threads, sizeof(*tids));
  struct rusage r0, r1;
  getrusage(RUSAGE_SELF, &r0);
  double t0 = now_ms();
  for (int i = 0; i < threads; i++)
    if (pthread_create(&tids[i], NULL, worker, NULL)) die("pthread_create");
  for (int i = 0; i < threads; i++) pthread_join(tids[i], NULL);
  double t1 = now_ms();
  getrusage(RUSAGE_SELF, &r1);
  double sys = (r1.ru_stime.tv_sec - r0.ru_stime.tv_sec) * 1e3 +
               (r1.ru_stime.tv_usec - r0.ru_stime.tv_usec) / 1e3;
  printf("order=%s threads=%d files=%zu extents=%ld shared_ext=%ld shared_files=%ld errors=%ld "
         "wall_ms=%.1f sys_ms=%.1f vcsw=%ld ivcsw=%ld\n",
         order, threads, npaths, atomic_load(&n_ext), atomic_load(&n_shared_ext),
         atomic_load(&n_shared_files), atomic_load(&n_err), t1 - t0, sys,
         r1.ru_nvcsw - r0.ru_nvcsw, r1.ru_nivcsw - r0.ru_nivcsw);
  return atomic_load(&n_err) ? 1 : 0;
}

static int cmd_populate(const char *dir, long n) {
  static unsigned char buf[32768];
  uint64_t x = 0x9E3779B97F4A7C15ull;
  char p[4096];
  for (long i = 0; i < n; i++) {
    if (i % 1000 == 0) {
      snprintf(p, sizeof(p), "%s/d%04ld", dir, i / 1000);
      if (mkdir(p, 0755) && errno != EEXIST) die(p);
    }
    x ^= x << 13; x ^= x >> 7; x ^= x << 17;
    size_t len = 4096 * (1 + x % 8);  // 4-32 KiB
    for (size_t j = 0; j < len; j += 8) {
      x ^= x << 13; x ^= x >> 7; x ^= x << 17;
      memcpy(buf + j, &x, 8);
    }
    snprintf(p, sizeof(p), "%s/d%04ld/f%07ld", dir, i / 1000, i);
    int fd = open(p, O_WRONLY | O_CREAT | O_TRUNC | O_CLOEXEC, 0644);
    if (fd < 0) die(p);
    if (write(fd, buf, len) != (ssize_t)len) die("write");
    close(fd);
  }
  return 0;
}

int main(int argc, char **argv) {
  if (argc == 4 && !strcmp(argv[1], "populate")) return cmd_populate(argv[2], atol(argv[3]));
  if ((argc == 4 || argc == 5) && !strcmp(argv[1], "run"))
    return cmd_run(argv[2], atoi(argv[3]), argc == 5 ? argv[4] : "seq");
  fprintf(stderr, "usage: %s populate DIR NFILES | run DIR THREADS [seq|shuffle]\n", argv[0]);
  return 2;
}

---- run.sh ----
#!/usr/bin/env bash
# Dust-free reproducer for XFS FIEMAP contention on reflinked files.
# Run as root on any Linux host with xfsprogs, gcc and loop-device support
# (OrbStack machine, GCP VM, bare metal). Builds an XFS image per AG count,
# fills it with a never-reflinked tree and a tree that has a reflink copy, and
# times multithreaded open+FIEMAP over each.
#
#   sudo repro/run.sh                      # defaults below
#   sudo NFILES=50000 RUNS=3 AGCOUNTS="4 32" THREADS="1 4 8" ORDERS=shuffle repro/run.sh
set -euo pipefail

NFILES=${NFILES:-100000}
RUNS=${RUNS:-5}
AGCOUNTS=${AGCOUNTS:-"4 32"}
THREADS=${THREADS:-$(printf '%s\n' 1 4 8 "$(nproc)" | sort -nu | tr '\n' ' ')}
ORDERS=${ORDERS:-"seq shuffle"}
WORK=${WORK:-/var/tmp/xfs-fiemap-repro}
HERE=$(cd "$(dirname "$0")" && pwd)

[ "$(id -u)" = 0 ] || { echo "run as root" >&2; exit 1; }
mkdir -p "$WORK/mnt"
gcc -O2 -pthread -o "$WORK/fiemap-mt" "$HERE/fiemap-mt.c"
BIN=$WORK/fiemap-mt

cleanup() { mountpoint -q "$WORK/mnt" && umount "$WORK/mnt"; rm -f "$WORK/xfs.img"; }
trap cleanup EXIT

echo "kernel=$(uname -r) arch=$(uname -m) nproc=$(nproc) $(mkfs.xfs -V) nfiles=$NFILES runs=$RUNS"

# Median of RUNS runs by wall_ms; prints the whole median line.
median_run() {
  local dir=$1 t=$2 order=$3
  for _ in $(seq "$RUNS"); do "$BIN" run "$dir" "$t" "$order"; done |
    awk '{for (i = 1; i <= NF; i++) if ($i ~ /^wall_ms=/) {split($i, a, "="); print a[2], $0}}' |
    sort -n | awk -v m=$(( (RUNS + 1) / 2 )) 'NR == m {$1 = ""; sub(/^ /, ""); print}'
}

for ag in $AGCOUNTS; do
  cleanup
  truncate -s 8G "$WORK/xfs.img"
  mkfs.xfs -q -f -d agcount="$ag" -m reflink=1 "$WORK/xfs.img"
  mount -o loop "$WORK/xfs.img" "$WORK/mnt"
  echo "=== $(xfs_info "$WORK/mnt" | grep -o 'agcount=[0-9]*'), $(xfs_info "$WORK/mnt" | grep -o 'reflink=[0-9]')"
  M=$WORK/mnt
  mkdir "$M/plain" "$M/src"
  "$BIN" populate "$M/plain" "$NFILES"
  "$BIN" populate "$M/src" "$NFILES"
  sync
  cp -a --reflink=always "$M/src" "$M/copy"
  sync
  for tree in plain src; do
    "$BIN" run "$M/$tree" "$(nproc)" >/dev/null   # warm the inode, extent and buffer caches
    for order in $ORDERS; do
      for t in $THREADS; do
        echo "ag=$ag tree=$tree $(median_run "$M/$tree" "$t" "$order")"
      done
    done
  done
done

Re: Concurrent FIEMAP on reflinked files serialises on the AGF buffer lock

[Christoph Hellwig]

Hi Lauri,

On Mon, Nov 02, 2026 at 09:32:45PM +0200, Lauri Gates wrote:
> Hi,
> 
> FIEMAP on files with XFS_DIFLAG2_REFLINK set does not scale across
> threads. For every written extent, xfs_read_iomap_begin() calls
> xfs_reflink_trim_around_shared() (IOMAP_REPORT), which goes to
> xfs_reflink_find_shared() and reads the AGF of the extent's AG with
> xfs_alloc_read_agf(pag, NULL, 0, &agbp). Concurrent callers queue on
> that buffer's b_sema. A parallel du-style scan of a reflinked tree
> stops scaling at roughly the AG count, and threads working in the same
> AG run slower than one thread alone.

FIEMAP is purely a debug interface for file system developers.  So
running it in parallel for lots of files is highly unexpected and
suggests a misuse of the interface.

> My question, in more detail at the end: does reporting the shared flag
> for FIEMAP need the AGF buffer locked?

It needs to protect against agf_refcount_level changing, so yes.
In theory it would be fine with a shared lock on the AGF buffer if
we had a shared/exclusive lock on the buffers (which is something
I've wondered about a few times before), or some other high-level
synchronization against adding more reflinks.

> I ran into this adding a shared-block mode to dust, a du clone, which
> FIEMAPs every file and counts FIEMAP_EXTENT_SHARED extents once.
> Inodes without the reflink flag return early and are unaffected.

Ok, so this is the use case.  The more efficient way to do this
would be to subtract the multiple mapped extents based on the
refcount btree, i.e.

	for each AG (and RTG with rtreflink):
		for each entry in the (rt)refcountbt:
			subtract rc_blockcount * (rc_refcount - 1);

We'd need a new ioctl for that, but it should be pretty easy.
The major downside is that this only works for whole-device du,
but then also any other version would have a really hard time figuring
out what to do with reflinks the span the reported/not reported
boundary.

> I am not suggesting dropping the flag. Commit 5f9268ca53ac ("xfs:
> don't bother looking at the refcount tree for reads") kept
> IOMAP_REPORT on the refcount lookup deliberately, and Dave described
> the shared flag as a field diagnostic for reflinked files:
> https://lore.kernel.org/all/20210124214346.GC4626@dread.disaster.area/

IOMAP_REPORT/FIEMAP reporting the shared flag is probably exactly what
your tool relies on...

Re: Concurrent FIEMAP on reflinked files serialises on the AGF buffer lock

[Lauri Gates]

On Mon, Nov 02, 2026 at 09:25:00PM -0800, Christoph Hellwig wrote:
> FIEMAP is purely a debug interface for file system developers.  So
> running it in parallel for lots of files is highly unexpected and
> suggests a misuse of the interface.

Understood, and that is why I am looking at the alternatives below.

> It needs to protect against agf_refcount_level changing, so yes.
> In theory it would be fine with a shared lock on the AGF buffer if
> we had a shared/exclusive lock on the buffers (which is something
> I've wondered about a few times before), or some other high-level
> synchronization against adding more reflinks.

Thanks, that answers it. If shared buffer locks are ever tried, I can
run the reproducer against the patches.

> The more efficient way to do this
> would be to subtract the multiple mapped extents based on the
> refcount btree, i.e.
>
>	for each AG (and RTG with rtreflink):
>		for each entry in the (rt)refcountbt:
>			subtract rc_blockcount * (rc_refcount - 1);
>
> We'd need a new ioctl for that, but it should be pretty easy.
> The major downside is that this only works for whole-device du,
> but then also any other version would have a really hard time figuring
> out what to do with reflinks the span the reported/not reported
> boundary.

A whole-device total would be useful on its own, but du tools mostly
report per directory. dust handles the boundary by counting each
shared extent once within the scanned tree, so a block shared with a
file outside the scan is charged to the scan in full. That is a policy
choice, and it needs to know which files own which shared blocks.

GETFSMAP already returns that: with rmapbt, each shared range comes
back once per owning inode. On a test image with 100,000 reflinked
pairs (7.2.12, arm64 VM, xfsprogs 6.6.0 defaults, which enable rmapbt),
a whole-device fsmap walk took about 120 ms, against 387 ms for a
single-threaded FIEMAP walk of the same 200,000 files. But it returns
owners only with CAP_SYS_ADMIN, as ioctl_fsmap(2) documents, so it
covers a du tool run as root and not one run as an ordinary user.

Would the refcount-sum ioctl you describe be callable without
CAP_SYS_ADMIN? A device-wide total reveals much less than per-file
owners.

On the dust side, a per-AG mutex in userspace removes the worst case,
threads in one AG running slower than a single thread, but recovers
little parallelism when threads are already spread over AGs.

Thanks,
Lauri

Re: Concurrent FIEMAP on reflinked files serialises on the AGF buffer lock

[Dave Chinner]

On Mon, Nov 02, 2026 at 09:25:00PM -0800, Christoph Hellwig wrote:
> Hi Lauri,
> 
> On Mon, Nov 02, 2026 at 09:32:45PM +0200, Lauri Gates wrote:
> > Hi,
> > 
> > FIEMAP on files with XFS_DIFLAG2_REFLINK set does not scale across
> > threads. For every written extent, xfs_read_iomap_begin() calls
> > xfs_reflink_trim_around_shared() (IOMAP_REPORT), which goes to
> > xfs_reflink_find_shared() and reads the AGF of the extent's AG with
> > xfs_alloc_read_agf(pag, NULL, 0, &agbp). Concurrent callers queue on
> > that buffer's b_sema. A parallel du-style scan of a reflinked tree
> > stops scaling at roughly the AG count, and threads working in the same
> > AG run slower than one thread alone.
> 
> FIEMAP is purely a debug interface for file system developers.  So
> running it in parallel for lots of files is highly unexpected and
> suggests a misuse of the interface.
> 
> > My question, in more detail at the end: does reporting the shared flag
> > for FIEMAP need the AGF buffer locked?

The problem is that we don't have a 'this inode has shared extents'
flag in the file_getattr() flag set, hence there is no easy way to
query if the inode may or may not have shared extents in it. That
would be trivial to add, and would avoid the need to FIEMAP every
single file in the fs to check for shared extents...

> It needs to protect against agf_refcount_level changing, so yes.
> In theory it would be fine with a shared lock on the AGF buffer if
> we had a shared/exclusive lock on the buffers (which is something
> I've wondered about a few times before), or some other high-level
> synchronization against adding more reflinks.
> 
> > I ran into this adding a shared-block mode to dust, a du clone, which
> > FIEMAPs every file and counts FIEMAP_EXTENT_SHARED extents once.
> > Inodes without the reflink flag return early and are unaffected.
> 
> Ok, so this is the use case.  The more efficient way to do this
> would be to subtract the multiple mapped extents based on the
> refcount btree, i.e.
> 
> 	for each AG (and RTG with rtreflink):
> 		for each entry in the (rt)refcountbt:
> 			subtract rc_blockcount * (rc_refcount - 1);
> 
> We'd need a new ioctl for that, but it should be pretty easy.
> The major downside is that this only works for whole-device du,
> but then also any other version would have a really hard time figuring
> out what to do with reflinks the span the reported/not reported
> boundary.

Efficient ad hoc physical extent status querying is what the
GETFSMAP ioctl is supposed to provide, right?

The owner of an physical extent is part of the input key filters, so
it extending the implementation to filter extents by owner field
would allow arbitrary per-file extent map lookups. e.g.
functionality like "report only shared extents for a given inode
number range" is well within the functional scope of this interface.

That allows dust to fstat a file to get the total block count and
inode number, call file_getattr() to see if the file has shared
extents, and then use fsmap to efficiently query the exact shared
extent list for that inode and so calculate the shared block count
for that inode number....

Cheers,

Dave.
-- 
Dave Chinner
dgc@kernel.org

Re: Concurrent FIEMAP on reflinked files serialises on the AGF buffer lock

[Dave Chinner]

On Wed, Nov 04, 2026 at 02:41:16PM +0200, Lauri Gates wrote:
> On Mon, Nov 02, 2026 at 09:25:00PM -0800, Christoph Hellwig wrote:
> > FIEMAP is purely a debug interface for file system developers.  So
> > running it in parallel for lots of files is highly unexpected and
> > suggests a misuse of the interface.
> 
> Understood, and that is why I am looking at the alternatives below.
> 
> > It needs to protect against agf_refcount_level changing, so yes.
> > In theory it would be fine with a shared lock on the AGF buffer if
> > we had a shared/exclusive lock on the buffers (which is something
> > I've wondered about a few times before), or some other high-level
> > synchronization against adding more reflinks.
> 
> Thanks, that answers it. If shared buffer locks are ever tried, I can
> run the reproducer against the patches.
> 
> > The more efficient way to do this
> > would be to subtract the multiple mapped extents based on the
> > refcount btree, i.e.
> >
> >	for each AG (and RTG with rtreflink):
> >		for each entry in the (rt)refcountbt:
> >			subtract rc_blockcount * (rc_refcount - 1);
> >
> > We'd need a new ioctl for that, but it should be pretty easy.
> > The major downside is that this only works for whole-device du,
> > but then also any other version would have a really hard time figuring
> > out what to do with reflinks the span the reported/not reported
> > boundary.
> 
> A whole-device total would be useful on its own, but du tools mostly
> report per directory. dust handles the boundary by counting each
> shared extent once within the scanned tree, so a block shared with a
> file outside the scan is charged to the scan in full. That is a policy
> choice, and it needs to know which files own which shared blocks.
> 
> GETFSMAP already returns that: with rmapbt, each shared range comes
> back once per owning inode. On a test image with 100,000 reflinked
> pairs (7.2.12, arm64 VM, xfsprogs 6.6.0 defaults, which enable rmapbt),
> a whole-device fsmap walk took about 120 ms, against 387 ms for a
> single-threaded FIEMAP walk of the same 200,000 files. But it returns
> owners only with CAP_SYS_ADMIN, as ioctl_fsmap(2) documents, so it
> covers a du tool run as root and not one run as an ordinary user.

fsmap() takes an arbitrary open fd on the filesystem being queried.
We could allow owner-based queries if the owner in the filter key
matches the inode backing the fd that is passed into the query. If
the task already has a file open for that inode, then it's already
been given permission to access/read metadata for that file (same as
FIEMAP).

Cheers,

Dave.
-- 
Dave Chinner
dgc@kernel.org

Re: Concurrent FIEMAP on reflinked files serialises on the AGF buffer lock

[Lauri Gates]

On Thu, Nov 05, 2026 at 08:04:15AM +1100, Dave Chinner wrote:
> The problem is that we don't have a 'this inode has shared extents'
> flag in the file_getattr() flag set, hence there is no easy way to
> query if the inode may or may not have shared extents in it. That
> would be trivial to add, and would avoid the need to FIEMAP every
> single file in the fs to check for shared extents...

That would help most trees: for a file without the flag, a du tool
could skip open + FIEMAP + close and make one file_getattr() call by
path. On 100,000 never-reflinked files, warm cache, with open + FIEMAP
+ close run in one process per worker so the shared fd table stays out
of the comparison (file_getattr() opens nothing):

                                open+FIEMAP+close  file_getattr()
  Ryzen 9 3900X, 6.17, 24 workers        27.5 ms         11.3 ms
  GCP n2-standard-8, 7.0, 8 workers      49.3 ms         21.4 ms

Mirroring XFS_DIFLAG2_REFLINK, the flag would mean "may have shared
extents", which is all a du tool needs to decide whether to look
closer. If it helps, I can write the patch.

> Efficient ad hoc physical extent status querying is what the
> GETFSMAP ioctl is supposed to provide, right?
>
> The owner of an physical extent is part of the input key filters, so
> it extending the implementation to filter extents by owner field
> would allow arbitrary per-file extent map lookups. e.g.
> functionality like "report only shared extents for a given inode
> number range" is well within the functional scope of this interface.

I tried the nearest thing that works today: per file, a GETFSMAP query
limited to that file's physical range, taken from an untimed FIEMAP
beforehand, so the best case for an owner-filtered query. As root on
the same GCP VM, same 100,000 files, one process per worker:

                       1 worker   8 workers
  open+FIEMAP+close      200 ms      49 ms
  per-file GETFSMAP      560 ms     564 ms

Each query reads the AGF in __xfs_getfsmap_datadev() and holds it for
that AG's walk, so concurrent queries queue on the AGF buffer lock,
also for never-reflinked files, where FIEMAP returns before the
refcount lookup. In a traced run with 4 threads, bpftrace put every
D-state switch in xfs_buf_lock() under xfs_alloc_read_agf() and
xfs_getfsmap(). And since the rmapbt key orders by physical block
before owner, an owner filter would need the inode's extents mapped
first to avoid walking the whole rmapbt.

So for a scan that touches every file, I expect it to stay slower
than FIEMAP unless the AGF can be held shared. Am I missing a way to
use it here? The permission model from your other reply, the owner
matching the inode behind the fd, makes sense to me; the cost per
query is what I am unsure about.

Thanks,
Lauri

Re: Concurrent FIEMAP on reflinked files serialises on the AGF buffer lock

[Christoph Hellwig]

On Fri, Nov 06, 2026 at 02:23:55PM +0200, Lauri Gates wrote:
> Mirroring XFS_DIFLAG2_REFLINK, the flag would mean "may have shared
> extents", which is all a du tool needs to decide whether to look
> closer. If it helps, I can write the patch.

Please do.  The main question are:

 - do we want it in fileattr_get or as a STATX_ATTR_ in statx.
 - how do we make the query reliable, and do we even need to?

Besides xfs, btrfs and ocfs2 also support reflinks, and nfs (and maybe
cifs?) can export the semanics remotely.  So we really need a way
to indicate if the file system could support reflinks to start with,
and then one that the file could, so we probably need two bits

Re: Concurrent FIEMAP on reflinked files serialises on the AGF buffer lock

[Christoph Hellwig]

On Wed, Nov 04, 2026 at 02:41:16PM +0200, Lauri Gates wrote:
> Would the refcount-sum ioctl you describe be callable without
> CAP_SYS_ADMIN? A device-wide total reveals much less than per-file
> owners.

I was pointed that patches for refcount reporting already exist:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/commit/?h=report-refcounts&id=9f3eca4098ff8226911f84af7b21d79670413876

and that does not require privileges, although in this form it might
not be very useful for you.

Re: Concurrent FIEMAP on reflinked files serialises on the AGF buffer lock

[Lauri Gates]

On Fri, Nov 06, 2026 at 07:20:04AM -0800, Christoph Hellwig wrote:
> Please do.  The main question are:
>
>  - do we want it in fileattr_get or as a STATX_ATTR_ in statx.
>  - how do we make the query reliable, and do we even need to?

Thanks. For the newly Cc'd lists: the thread [1] started from du-like
tools that call FIEMAP on every file to count shared extents, which on
XFS serialises on the AGF buffer lock. Dave suggested a per-file "may
have shared extents" flag so such tools can skip FIEMAP on files that
cannot share, and Christoph asked for a patch. Before writing one I'd
like to settle the two questions above and the "two bits" question
below.

To see what the flag would buy, I emulated it: a walker that calls
statx() on every file and runs open + FIEMAP + close only on files
whose XFS_DIFLAG2_REFLINK was set, read beforehand with xfs_db -r. In
all 80 configurations (trees 0-100% reflinked, 1-24 threads, two
machines) it found the same shared files and extents as FIEMAP on
every file.

On placement, a STATX_ATTR_ bit (say STATX_ATTR_MAY_SHARE) looks like
the better fit to me, for two reasons, though I may well be missing
history here.

A du walk needs statx() for stx_blocks anyway, so a bit in
stx_attributes would cost it no extra syscall, while the path-based
file_getattr(2) adds a second call and lookup per file. Median of 5
runs, 100,000 files, warm cache, Ryzen 9 3900X on 6.17:

                               1 thread   24 threads
    statx()                      195 ms      11.4 ms
    statx() + file_getattr()     352 ms      19.9 ms

(An arm64 VM on 7.2 shows a smaller gap: statx() alone vs with
file_getattr() is 79 vs 97 ms on 1 thread, 8.9 vs 11.0 ms on 13.)

The other is that for a "may share" bit, clear is the unsafe answer,
so a filesystem that cannot answer should not look like one that
answers "not shared". As I understand it, stx_attributes_mask makes
that distinction, while struct file_attr has no validity mask today
(it is size-versioned, so one could be added), so through
file_getattr() btrfs would currently read as never shared. A spare
bit in stx_attributes adds no field to struct statx, which I hope
meets the concern Christian raised about spending statx space on the
case-folding RFC [2]. If fileattr is preferred, the bit could instead
mean "no shared extents", so that clear is the safe default, as it is
for FS_XFLAG_CASEFOLD.

> Besides xfs, btrfs and ocfs2 also support reflinks, and nfs (and maybe
> cifs?) can export the semanics remotely.  So we really need a way
> to indicate if the file system could support reflinks to start with,
> and then one that the file could, so we probably need two bits

For du, I think the mask could stand in for the "could support
reflinks" bit if it means "this filesystem can answer for this file":

  mask clear            not reported, fall back to FIEMAP
  mask set, bit clear   no shared extents
  mask set, bit set     may have shared extents

A filesystem that cannot share extents at all would set the mask and
leave the bit clear. If I read v7.4-rc1 right, only xfs, btrfs and
ocfs2 report shared extents, and all three implement
->remap_file_range, so the VFS might set the mask itself for files
whose filesystem lacks ->remap_file_range. What I could find per
filesystem:

 - xfs: XFS_DIFLAG2_REFLINK is set on both inodes in
   xfs_reflink_remap_prep() before any extent is remapped, so clear
   means no shared extents.
 - ocfs2: OCFS2_HAS_REFCOUNT_FL is set on both inodes before the
   remap and cleared only at zero clusters, which looks like the same
   meaning.
 - btrfs: I could not find a persisted per-inode reflink record
   (last_reflink_trans is in memory only and reset to last_trans when
   the inode is loaded), so btrfs would leave the bit out of the mask
   for now. btrfs folks, please correct me if I missed something.
 - nfs, cifs: I could not find a per-file attribute in NFSv4.2 or
   MS-FSCC that could carry it (space_freed comes closest, and nfsd
   does not implement it), so no mask bit there.

A filesystem-level "could reflink" bit might be a separate need:
nfsd encodes clone_blksize for every export and ksmbd advertises
FILE_SUPPORTS_BLOCK_REFCOUNTING unconditionally, I assume because
neither can ask the exported filesystem. Did you have nfsd and ksmbd
in mind for that bit?

On reliability, my tentative answer is that the bit does not need to
be exact for this use. A clear bit must never be wrong, and it never
was in my tests, including a survey of my desktop's XFS root (2.0M
files, flags read with xfs_db -r and matched by inode number). A
stale set bit costs one FIEMAP, so I would document it as "may have
shared extents"; since the contract is only "may", clearing it more
eagerly later would not change the uapi.

Stale bits do limit what it saves, though. The flag stays set when
the other copy is deleted or the file is overwritten; only truncation
to zero and FALLOC_FL_UNSHARE_RANGE (see the caveat below) clear it.
Ordinary copies set it: coreutils 9.4 cp and install, cat (which
copies with copy_file_range), Rust's std::fs::copy, and overlayfs
copy-up all reflink on XFS. On my desktop, leaving out benchmark
trees, 18% of inodes have the flag and 41% of those share nothing.
That leaves out ~/.rustup too: an earlier test reflink-copied it, so
all of its 132k inodes are flagged and none share; counting it, the
figures are 25% and 62%.

On real trees (24 threads, median of 5), statx() on every file, then
open + FIEMAP + close on every file, on flagged files, or on files
with shared extents only, the lower bound for a flag cleared when
sharing ends (/usr has no shared files):

                       statx only  every file  flagged  shared only
  /usr      224k files      28 ms      264 ms    29 ms       -
  ~/.local  372k files      63 ms      506 ms   487 ms    322 ms
  ~/.rustup 132k files      20 ms      319 ms   318 ms     21 ms

Would there be a reasonable point for XFS to clear the flag once
sharing ends? I don't know the code well enough to tell what that
would cost.

The caveat, which may be unintended or may be my misreading:
FALLOC_FL_UNSHARE_RANGE over a whole shared file does not clear the
flag on the first call. bpftrace on 7.2 shows
xfs_reflink_clear_inode_flag() entered and
xfs_reflink_inode_has_shared_extents() not, so it looks like
xfs_can_free_cowblocks() returned false, perhaps on I_DIRTY_PAGES
left by the unshare. A second call after sync clears the flag.

On Darrick's XFS_IOC_GETFSREFCOUNTS patch from your other mail:
thanks for the pointer. It gives a device-wide total without
privileges, which a du tool could show for the whole filesystem,
though not per directory.

If the statx direction sounds reasonable, I would be glad to put
together an RFC with the uapi bit, xfs and ocfs2 support, a man-pages
patch and an xfstest. This would be my first kernel patch, so
corrections on any of the above are very welcome.

[1] https://lore.kernel.org/linux-xfs/20261102193533.93690-1-lauri.gates@gmail.com/
[2] https://lore.kernel.org/all/20251006-zypressen-paarmal-4167375db973@brauner/

Thanks,
Lauri