[PATCH v11 0/1] mkfs: add ability to populate filesystem from directory

[PATCH v11 0/1] mkfs: add ability to populate filesystem from directory

[Luca Di Maio]

Currently the only way to pre populate an XFS partition is via the
prototype file. While it works it has some limitations like:
  - not allowed spaces in file names
  - not preserving timestamps of original inodes

This series adds a new -P option to mkfs.xfs that allows users to
populate a newly created filesystem directly from an existing directory.
While similar to the prototype functionality, this doesn't require
writing a prototype file.
The implementation preserves file and directory attributes (ownership,
permissions, timestamps) from the source directory when copying content
to the new filesystem.

[v1] -> [v2]
  remove changes to protofile spec
  ensure backward compatibility
[v2] -> [v3]
  use inode_set_[acm]time() as suggested
  avoid copying atime and ctime
  they are often problematic for reproducibility, and
  mtime is the important information to preserve anyway
[v3] -> [v4]
  rewrite functionality to populate directly from an input directory
  this is similar to mkfs.ext4 option.
[v4] -> [v5]
  reorder patch to make it easier to review
  reflow to keep code below 80 chars
  use _() macro in prints
add SPDX headers to new files
  fix comment styling
  move from typedef to structs
  move direntry handling to own function
[v5] -> [v6]
  rebase on 6.14
[v6] -> [v7]
  move functionality to common -p flag
  add noatime flag to skip atime copy and set to current time
  set ctime/crtime to current time
  preserve hardlinks
  preserve extended attributes for all file/dir types
  add fsxattr to copied files/dirs
[v7] -> [v8]
  changed directory source validation to use stat() instead of open()
  changed hardlink tracker to store inode numbers instead of inode pointers
  fixed path buffer handling for directory traversal
  handle blocking FIFOs filetypes
  handle hardlinks of symlinks
  improve setup_proto and parse proto using structured xfs_proto_source type
  renamed noatime to preserve_atime with inverted logic
  specify EBADF fgetxattr() and flistxattr() fallback for O_PATH fds
  switch to calloc() to initialize hardlinks_tracker
  switch to reallocarray() for hardlinks_tracker resize
[v8] -> [v9]
  squash commits in one
  clarify the linear search implications in the commit log
  fixed indentation and styling
  fail early for invalid inputs
  pass dir fd along to reduce amount of open()
  switch from ino_t to xfs_ino_t
  add support for socket files
  copy fsxattr also for cases where we don't have an fd
[v9] -> [v10]
  fix indentation, alignment and style
  improve comments
  avoid unneeded casts from ino_t to xfs_ino_t for source inodes
  avoid global hardlink_tracker as pointer
  avoid mixing code and declarations
  symlink target string now respects XFS_SYMLINK_MAXLEN
  set root directory ownership/mode/attrs/fsxattrs from source dir
[v10] -> [v11]
  fix indentation and comment style
  avoid TOCTOU races by first opening file, then using fstat on it
  use openat() to avoid string manipulation
  properly cleanup hardlink tracker
  cleanup the henalde_direntry() switch statement
  unify all file types except directories in a single create_inode()
  function
  create a separate create_directory_inode() function

Luca Di Maio (1):
  proto: add ability to populate a filesystem from a directory

 man/man8/mkfs.xfs.8.in |  41 ++-
 mkfs/proto.c           | 748 ++++++++++++++++++++++++++++++++++++++++-
 mkfs/proto.h           |  18 +-
 mkfs/xfs_mkfs.c        |  23 +-
 4 files changed, 804 insertions(+), 26 deletions(-)

--
2.50.0

[PATCH v11 1/1] proto: add ability to populate a filesystem from a directory

[Luca Di Maio]

This patch implements the functionality to populate a newly created XFS
filesystem directly from an existing directory structure.

It resuses existing protofile logic, it branches if input is a
directory.

The population process steps are as follows:
  - create the root inode before populating content
  - recursively process nested directories
  - handle regular files, directories, symlinks, char devices, block
    devices, sockets, fifos
  - preserve attributes (ownership, permissions)
  - preserve mtime timestamps from source files to maintain file history
    - use current time for atime/ctime/crtime
    - possible to specify atime=1 to preserve atime timestamps from
      source files
  - preserve extended attributes and fsxattrs for all file types
  - preserve hardlinks

At the moment, the implementation for the hardlink tracking is very
simple, as it involves a linear search.
from my local testing using larger source directories
(1.3mln inodes, ~400k hardlinks) the difference was actually
just a few seconds (given that most of the time is doing i/o).
We might want to revisit that in the future if this becomes a
bottleneck.

This functionality makes it easier to create populated filesystems
without having to mount them, it's particularly useful for
reproducible builds.

Signed-off-by: Luca Di Maio <luca.dimaio1@gmail.com>
---
 man/man8/mkfs.xfs.8.in |  41 ++-
 mkfs/proto.c           | 748 ++++++++++++++++++++++++++++++++++++++++-
 mkfs/proto.h           |  18 +-
 mkfs/xfs_mkfs.c        |  23 +-
 4 files changed, 804 insertions(+), 26 deletions(-)

diff --git a/man/man8/mkfs.xfs.8.in b/man/man8/mkfs.xfs.8.in
index bc804931..8c393ba2 100644
--- a/man/man8/mkfs.xfs.8.in
+++ b/man/man8/mkfs.xfs.8.in
@@ -28,7 +28,7 @@ mkfs.xfs \- construct an XFS filesystem
 .I naming_options
 ] [
 .B \-p
-.I protofile_options
+.I prototype_options
 ] [
 .B \-q
 ] [
@@ -977,30 +977,39 @@ option set.
 .PP
 .PD 0
 .TP
-.BI \-p " protofile_options"
+.BI \-p " prototype_options"
 .TP
 .BI "Section Name: " [proto]
 .PD
-These options specify the protofile parameters for populating the filesystem.
+These options specify the prototype parameters for populating the filesystem.
 The valid
-.I protofile_options
+.I prototype_options
 are:
 .RS 1.2i
 .TP
-.BI [file=] protofile
+.BI [file=]
 The
 .B file=
 prefix is not required for this CLI argument for legacy reasons.
 If specified as a config file directive, the prefix is required.
-
+.TP
+.BI [file=] directory
 If the optional
 .PD
-.I protofile
-argument is given,
+.I prototype
+argument is given, and it's a directory,
 .B mkfs.xfs
-uses
-.I protofile
-as a prototype file and takes its directions from that file.
+will populate the root file system with the contents of the given directory.
+Content, timestamps (atime, mtime), attributes and extended attributes
+are preserved for all file types.
+.TP
+.BI [file=] protofile
+If the optional
+.PD
+.I prototype
+argument is given, and points to a regular file,
+.B mkfs.xfs
+uses it as a prototype file and takes its directions from that file.
 The blocks and inodes specifiers in the
 .I protofile
 are provided for backwards compatibility, but are otherwise unused.
@@ -1136,8 +1145,16 @@ always terminated with the dollar (
 .B $
 ) token.
 .TP
+.BI atime= value
+If set to 1, when we're populating the root filesystem from a directory (
+.B file=directory
+option)
+access times are going to be preserved and are copied from the source files.
+Set to 0 to set access times to the current time instead.
+By default, this is set to 0.
+.TP
 .BI slashes_are_spaces= value
-If set to 1, slashes ("/") in the first token of each line of the protofile
+If set to 1, slashes ("/") in the first token of each line of the prototype file
 are converted to spaces.
 This enables the creation of a filesystem containing filenames with spaces.
 By default, this is set to 0.
diff --git a/mkfs/proto.c b/mkfs/proto.c
index 7f80bef8..31626211 100644
--- a/mkfs/proto.c
+++ b/mkfs/proto.c
@@ -5,6 +5,8 @@
  */

 #include "libxfs.h"
+#include <dirent.h>
+#include <sys/resource.h>
 #include <sys/stat.h>
 #include <sys/xattr.h>
 #include <linux/xattr.h>
@@ -21,6 +23,11 @@ static void rsvfile(xfs_mount_t *mp, xfs_inode_t *ip, long long len);
 static int newregfile(char **pp, char **fname);
 static void rtinit(xfs_mount_t *mp);
 static off_t filesize(int fd);
+static void populate_from_dir(struct xfs_mount *mp, struct fsxattr *fsxp,
+		char *source_dir);
+static void walk_dir(struct xfs_mount *mp, struct xfs_inode *pip,
+		struct fsxattr *fsxp, char *path_buf);
+static int preserve_atime;
 static int slashes_are_spaces;

 /*
@@ -54,7 +61,7 @@ getnum(
 	return i;
 }

-char *
+struct proto_source
 setup_proto(
 	char	*fname)
 {
@@ -63,14 +70,40 @@ setup_proto(
 	int		fd;
 	long		size;

-	if (!fname)
-		return dflt;
+	struct proto_source	result = {};
+	struct stat	statbuf;
+
+	/*
+	 * If no prototype path is supplied, use the default protofile which
+	 * creates only a root directory.
+	 */
+	if (!fname) {
+		result.type = PROTO_SRC_PROTOFILE;
+		result.data = dflt;
+		return result;
+	}
+
 	if ((fd = open(fname, O_RDONLY)) < 0 || (size = filesize(fd)) < 0) {
 		fprintf(stderr, _("%s: failed to open %s: %s\n"),
 			progname, fname, strerror(errno));
 		goto out_fail;
 	}

+	if (fstat(fd, &statbuf) < 0)
+		fail(_("invalid or unreadable source path"), errno);
+
+	/*
+	 * Handle directory inputs.
+	 */
+	if (S_ISDIR(statbuf.st_mode)) {
+		result.type = PROTO_SRC_DIR;
+		result.data = fname;
+		return result;
+	}
+
+	/*
+	 * Else this is a protofile, let's handle traditionally.
+	 */
 	buf = malloc(size + 1);
 	if (read(fd, buf, size) < size) {
 		fprintf(stderr, _("%s: read failed on %s: %s\n"),
@@ -90,7 +123,10 @@ setup_proto(
 	(void)getnum(getstr(&buf), 0, 0, false);	/* block count */
 	(void)getnum(getstr(&buf), 0, 0, false);	/* inode count */
 	close(fd);
-	return buf;
+
+	result.type = PROTO_SRC_PROTOFILE;
+	result.data = buf;
+	return result;

 out_fail:
 	if (fd >= 0)
@@ -379,6 +415,13 @@ writeattr(
 	int			error;

 	ret = fgetxattr(fd, attrname, valuebuf, valuelen);
+	/*
+	 * In case of filedescriptors with O_PATH, fgetxattr() will fail with
+	 * EBADF.
+	 * Let's try to fallback to lgetxattr() using input path.
+	 */
+	if (ret < 0 && errno == EBADF)
+		ret = lgetxattr(fname, attrname, valuebuf, valuelen);
 	if (ret < 0) {
 		if (errno == EOPNOTSUPP)
 			return;
@@ -425,6 +468,13 @@ writeattrs(
 		fail(_("error allocating xattr name buffer"), errno);

 	ret = flistxattr(fd, namebuf, XATTR_LIST_MAX);
+	/*
+	 * In case of filedescriptors with O_PATH, flistxattr() will fail with
+	 * EBADF.
+	 * Let's try to fallback to llistxattr() using input path.
+	 */
+	if (ret < 0 && errno == EBADF)
+		ret = llistxattr(fname, namebuf, XATTR_LIST_MAX);
 	if (ret < 0) {
 		if (errno == EOPNOTSUPP)
 			goto out_namebuf;
@@ -933,11 +983,27 @@ void
 parse_proto(
 	xfs_mount_t	*mp,
 	struct fsxattr	*fsx,
-	char		**pp,
-	int		proto_slashes_are_spaces)
+	struct proto_source	*protosource,
+	int		proto_slashes_are_spaces,
+	int		proto_preserve_atime)
 {
 	slashes_are_spaces = proto_slashes_are_spaces;
-	parseproto(mp, NULL, fsx, pp, NULL);
+	preserve_atime = proto_preserve_atime;
+
+	/*
+	 * In case of a file input, we will use the prototype file logic else
+	 * we will fallback to populate from dir.
+	 */
+	switch(protosource->type) {
+	case PROTO_SRC_PROTOFILE:
+		parseproto(mp, NULL, fsx, &protosource->data, NULL);
+		break;
+	case PROTO_SRC_DIR:
+		populate_from_dir(mp, fsx, protosource->data);
+		break;
+	case PROTO_SRC_NONE:
+		fail(_("invalid or unreadable source path"), ENOENT);
+	}
 }

 /* Create a sb-rooted metadata file. */
@@ -1172,3 +1238,671 @@ filesize(
 		return -1;
 	return stb.st_size;
 }
+
+/* Try to allow as many open directories as possible. */
+static void
+bump_max_fds(void)
+{
+	struct rlimit	rlim = {};
+	int		ret;
+
+	ret = getrlimit(RLIMIT_NOFILE, &rlim);
+	if (ret)
+		return;
+
+	rlim.rlim_cur = rlim.rlim_max;
+	ret = setrlimit(RLIMIT_NOFILE, &rlim);
+	if (ret < 0)
+		fprintf(stderr, _("%s: could not bump fd limit: [ %d - %s]\n"),
+			progname, errno, strerror(errno));
+}
+
+static void
+writefsxattrs(
+	struct xfs_inode	*ip,
+	struct fsxattr	*fsxp)
+{
+	ip->i_projid = fsxp->fsx_projid;
+	ip->i_extsize = fsxp->fsx_extsize;
+	ip->i_diflags = xfs_flags2diflags(ip, fsxp->fsx_xflags);
+	if (xfs_has_v3inodes(ip->i_mount)) {
+		ip->i_diflags2 = xfs_flags2diflags2(ip, fsxp->fsx_xflags);
+		ip->i_cowextsize = fsxp->fsx_cowextsize;
+	}
+}
+
+static void
+writetimestamps(
+	struct xfs_inode	*ip,
+	struct stat	*statbuf)
+{
+	struct timespec64	ts;
+
+	/*
+	 * Copy timestamps from source file to destination inode.
+	 * Usually reproducible archives will delete or not register
+	 * atime and ctime, for example:
+	 *    https://www.gnu.org/software/tar/manual/html_section/Reproducibility.html
+	 * hence we will only copy mtime, and let ctime/crtime be set to
+	 * current time.
+	 * atime will be copied over if atime is true.
+	 */
+	ts.tv_sec = statbuf->st_mtim.tv_sec;
+	ts.tv_nsec = statbuf->st_mtim.tv_nsec;
+	inode_set_mtime_to_ts(VFS_I(ip), ts);
+
+	/*
+	 * In case of atime option, we will copy the atime  timestamp
+	 * from source.
+	 */
+	if (preserve_atime) {
+		ts.tv_sec = statbuf->st_atim.tv_sec;
+		ts.tv_nsec = statbuf->st_atim.tv_nsec;
+		inode_set_atime_to_ts(VFS_I(ip), ts);
+	}
+}
+
+struct hardlink {
+	ino_t	src_ino;
+	xfs_ino_t	dst_ino;
+};
+
+struct hardlinks {
+	size_t		count;
+	size_t		size;
+	struct hardlink	*entries;
+};
+
+/* Growth strategy for hardlink tracking array */
+/* Double size for small arrays */
+#define HARDLINK_DEFAULT_GROWTH_FACTOR	2
+/* Grow by 25% for large arrays */
+#define HARDLINK_LARGE_GROWTH_FACTOR	0.25
+/* Threshold to switch growth strategies */
+#define HARDLINK_THRESHOLD		1024
+/* Initial allocation size */
+#define HARDLINK_TRACKER_INITIAL_SIZE	4096
+
+/*
+ * Keep track of source inodes that are from hardlinks so we can retrieve them
+ * when needed to setup in destination.
+ */
+static struct hardlinks hardlink_tracker = { 0 };
+
+static void
+init_hardlink_tracker(void)
+{
+	hardlink_tracker.size = HARDLINK_TRACKER_INITIAL_SIZE;
+	hardlink_tracker.entries = calloc(
+			hardlink_tracker.size,
+			sizeof(struct hardlink));
+	if (!hardlink_tracker.entries)
+		fail(_("error allocating hardlinks tracking array"), errno);
+}
+
+static void
+cleanup_hardlink_tracker(void)
+{
+	free(hardlink_tracker.entries);
+	hardlink_tracker.entries = NULL;
+	hardlink_tracker.count = 0;
+	hardlink_tracker.size = 0;
+}
+
+static xfs_ino_t
+get_hardlink_dst_inode(
+	xfs_ino_t	i_ino)
+{
+	for (size_t i = 0; i < hardlink_tracker.count; i++) {
+		if (hardlink_tracker.entries[i].src_ino == i_ino)
+			return hardlink_tracker.entries[i].dst_ino;
+	}
+	return 0;
+}
+
+static void
+track_hardlink_inode(
+	ino_t	src_ino,
+	xfs_ino_t	dst_ino)
+{
+	if (hardlink_tracker.count >= hardlink_tracker.size) {
+		/*
+		 * double for smaller capacity.
+		 * instead grow by 25% steps for larger capacities.
+		 */
+		const size_t old_size = hardlink_tracker.size;
+		size_t new_size = old_size * HARDLINK_DEFAULT_GROWTH_FACTOR;
+		if (old_size > HARDLINK_THRESHOLD)
+			new_size = old_size + (old_size * HARDLINK_LARGE_GROWTH_FACTOR);
+
+		struct hardlink *resized_array = reallocarray(
+			hardlink_tracker.entries,
+			new_size,
+			sizeof(struct hardlink));
+		if (!resized_array)
+			fail(_("error enlarging hardlinks tracking array"), errno);
+
+		memset(&resized_array[old_size], 0,
+				(new_size - old_size) * sizeof(struct hardlink));
+
+		hardlink_tracker.entries = resized_array;
+		hardlink_tracker.size = new_size;
+	}
+	hardlink_tracker.entries[hardlink_tracker.count].src_ino = src_ino;
+	hardlink_tracker.entries[hardlink_tracker.count].dst_ino = dst_ino;
+	hardlink_tracker.count++;
+}
+
+/*
+ * This function will first check in our tracker if the input hardlink has
+ * already been stored, if not report false so create_inode() can continue
+ * handling the inode as regularly, and later save the source inode in our
+ * buffer for future consumption.
+ */
+static bool
+handle_hardlink(
+	struct xfs_mount	*mp,
+	struct xfs_inode	*pip,
+	struct xfs_name	xname,
+	struct stat	file_stat)
+{
+	int		error;
+	xfs_ino_t		dst_ino;
+	struct xfs_inode	*ip;
+	struct xfs_trans	*tp;
+	struct xfs_parent_args *ppargs = NULL;
+
+	tp = getres(mp, 0);
+	ppargs = newpptr(mp);
+	dst_ino = get_hardlink_dst_inode(file_stat.st_ino);
+
+	/*
+	 * We didn't find the hardlink inode, this means it's the first time
+	 * we see it, report error so create_inode() can continue handling the
+	 * inode as a regular file type, and later save the source inode in our
+	 * buffer for future consumption.
+	 */
+	if (dst_ino == 0)
+		return false;
+
+	error = libxfs_iget(mp, NULL, dst_ino, 0, &ip);
+	if (error)
+		fail(_("failed to get inode"), error);
+
+	/*
+	 * In case the inode was already in our tracker we need to setup the
+	 * hardlink and skip file copy.
+	 */
+	libxfs_trans_ijoin(tp, pip, 0);
+	libxfs_trans_ijoin(tp, ip, 0);
+	newdirent(mp, tp, pip, &xname, ip, ppargs);
+
+	/*
+	 * Increment the link count
+	 */
+	libxfs_bumplink(tp, ip);
+
+	libxfs_trans_log_inode(tp, ip, XFS_ILOG_CORE);
+
+	error = -libxfs_trans_commit(tp);
+	if (error)
+		fail(_("Error encountered creating file from prototype file"), error);
+
+	libxfs_parent_finish(mp, ppargs);
+	libxfs_irele(ip);
+
+	return true;
+}
+
+static void
+create_directory_inode(
+	struct xfs_mount	*mp,
+	struct xfs_inode	*pip,
+	struct fsxattr  	*fsxp,
+	int		mode,
+	struct cred	creds,
+	struct xfs_name	xname,
+	int		flags,
+	struct stat	file_stat,
+	int 		fd,
+	char		*entryname,
+	char		*path)
+{
+
+	int		error;
+	struct xfs_inode	*ip;
+	struct xfs_trans	*tp;
+	struct xfs_parent_args *ppargs = NULL;
+
+	tp = getres(mp, 0);
+	ppargs = newpptr(mp);
+
+	error = creatproto(&tp, pip, mode, 0, &creds, fsxp, &ip);
+	if (error)
+		fail(_("Inode allocation failed"), error);
+
+	libxfs_trans_ijoin(tp, pip, 0);
+
+	newdirent(mp, tp, pip, &xname, ip, ppargs);
+
+	libxfs_bumplink(tp, pip);
+	libxfs_trans_log_inode(tp, pip, XFS_ILOG_CORE);
+	newdirectory(mp, tp, ip, pip);
+
+	/*
+	 * Copy over timestamps.
+	 */
+	writetimestamps(ip, &file_stat);
+
+	libxfs_trans_log_inode(tp, ip, flags);
+
+	error = -libxfs_trans_commit(tp);
+	if (error)
+		fail(_("Directory inode allocation failed."), error);
+
+	libxfs_parent_finish(mp, ppargs);
+	tp = NULL;
+
+	/*
+	 * Copy over attributes.
+	 */
+	writeattrs(ip, entryname, fd);
+	writefsxattrs(ip, fsxp);
+	close(fd);
+
+	walk_dir(mp, ip, fsxp, path);
+
+	libxfs_irele(ip);
+}
+
+static void
+create_inode(
+	struct xfs_mount	*mp,
+	struct xfs_inode	*pip,
+	struct fsxattr  	*fsxp,
+	int		mode,
+	struct cred	creds,
+	struct xfs_name	xname,
+	int		flags,
+	struct stat	file_stat,
+	xfs_dev_t	rdev,
+	int 		fd,
+	char		*fname)
+{
+
+	char		link_target[XFS_SYMLINK_MAXLEN];
+	ssize_t		link_len;
+	int		error;
+	struct xfs_inode	*ip;
+	struct xfs_trans	*tp;
+	struct xfs_parent_args *ppargs = NULL;
+
+	/*
+	 * If handle_hardlink() returns true it means the hardlink has been
+	 * correctly found and set, so we don't need to do anything else.
+	 */
+	if (file_stat.st_nlink > 1 && handle_hardlink(mp, pip, xname, file_stat)) {
+		close(fd);
+		return;
+	}
+	/*
+	 * If instead we have an error it means the hardlink was not registered,
+	 * so we proceed to treat it like a regular file, and save it to our
+	 * tracker later.
+	 */
+	tp = getres(mp, 0);
+	/*
+	 * In case of symlinks, we need to handle things a little differently.
+	 * We need to read out our link target and act accordingly.
+	 */
+	if (xname.type == XFS_DIR3_FT_SYMLINK) {
+		link_len = readlink(fname, link_target, XFS_SYMLINK_MAXLEN);
+		if (link_len < 0)
+			fail(_("could not resolve symlink"), errno);
+		if (link_len >= PATH_MAX)
+			fail(_("symlink target too long"), ENAMETOOLONG);
+		tp = getres(mp, XFS_B_TO_FSB(mp, link_len));
+	}
+	ppargs = newpptr(mp);
+
+	error = creatproto(&tp, pip, mode, rdev, &creds, fsxp, &ip);
+	if (error)
+		fail(_("Inode allocation failed"), error);
+
+	/*
+	 * In case of symlinks, we now write it down, for other file types
+	 * this is handled later before cleanup.
+	 */
+	if (xname.type == XFS_DIR3_FT_SYMLINK)
+		writesymlink(tp, ip, link_target, link_len);
+
+	libxfs_trans_ijoin(tp, pip, 0);
+	newdirent(mp, tp, pip, &xname, ip, ppargs);
+
+	/*
+	 * Copy over timestamps.
+	 */
+	writetimestamps(ip, &file_stat);
+
+	libxfs_trans_log_inode(tp, ip, flags);
+
+	error = -libxfs_trans_commit(tp);
+	if (error)
+		fail(_("Error encountered creating file from prototype file"), error);
+
+	libxfs_parent_finish(mp, ppargs);
+
+	/*
+	 * Copy over file content, attributes, extended attributes and
+	 * timestamps.
+	 */
+	if (fd >= 0) {
+		/* We need to writefile only when not dealing with a symlink. */
+		if (xname.type != XFS_DIR3_FT_SYMLINK)
+			writefile(ip, fname, fd);
+		writeattrs(ip, fname, fd);
+		close(fd);
+	}
+	/*
+	 * We do fsxattr also for file types where we don't have an fd,
+	 * for example FIFOs.
+	 */
+	writefsxattrs(ip, fsxp);
+
+	/*
+	 * If we're here it means this is the first time we're encountering an
+	 * hardlink, so we need to store it.
+	 */
+	if (file_stat.st_nlink > 1)
+		track_hardlink_inode(file_stat.st_ino, ip->i_ino);
+
+	libxfs_irele(ip);
+}
+
+static void
+handle_direntry(
+	struct xfs_mount	*mp,
+	struct xfs_inode	*pip,
+	struct fsxattr		*fsxp,
+	char			*path_buf,
+	struct dirent		*entry)
+{
+	char		cur_path_buf[PATH_MAX];
+	char		*fname = "";
+	int 		pathfd,fd = -1;
+	int 		flags;
+	int 		majdev;
+	int 		mindev;
+	int 		mode;
+	int 		rdev = 0;
+	struct stat	file_stat;
+	struct xfs_name	xname;
+
+	/* Ensure we're within the limits of PATH_MAX. */
+	if (strlen(path_buf) + strlen(entry->d_name) >= PATH_MAX)
+		fail(_("path name too long"), ENAMETOOLONG);
+	/*
+	 * Keep a reference to our current file path it will be useful
+	 * afterwards for fds open with O_PATH where we will have to fallback
+	 * to path based attribute handling.
+	 */
+	if (snprintf(cur_path_buf, PATH_MAX, "%s/%s",
+		     path_buf, entry->d_name) < 0)
+		fail(_("path encoding error"), -1);
+
+	pathfd = open(path_buf, O_NOFOLLOW | O_PATH);
+	if (pathfd < 0){
+		fprintf(stderr, _("%s: cannot open %s: %s\n"), progname,
+			cur_path_buf, strerror(errno));
+		exit(1);
+	}
+
+	/*
+	 * Symlinks and sockets will need to be opened with O_PATH to work, so
+	 * we handle this special case.
+	 */
+	fd = openat(pathfd, entry->d_name, O_NOFOLLOW | O_PATH);
+	if(fd < 0) {
+		fprintf(stderr, _("%s: cannot open %s: %s\n"), progname,
+			cur_path_buf, strerror(errno));
+		exit(1);
+	}
+
+	if (fstat(fd, &file_stat) < 0) {
+		fprintf(stderr, _("%s: cannot stat '%s': %s (errno=%d)\n"),
+				progname, cur_path_buf, strerror(errno), errno);
+		exit(1);
+	}
+
+	/*
+	 * Regular files instead need to be reopened with broader flags so we
+	 * check if that's the case and reopen those.
+	 */
+	if (!S_ISSOCK(file_stat.st_mode) &&
+	    !S_ISLNK(file_stat.st_mode)  &&
+	    !S_ISFIFO(file_stat.st_mode)) {
+		close(fd);
+		fd = openat(pathfd, entry->d_name,
+			    O_NOFOLLOW | O_RDONLY | O_NOATIME);
+		if(fd < 0) {
+			fprintf(stderr, _("%s: cannot open %s: %s\n"), progname,
+				cur_path_buf, strerror(errno));
+			exit(1);
+		}
+	}
+
+	struct cred creds = {
+		.cr_uid = file_stat.st_uid,
+		.cr_gid = file_stat.st_gid,
+	};
+
+	xname.name = (unsigned char *)entry->d_name;
+	xname.len = strlen(entry->d_name);
+	xname.type = 0;
+	mode = file_stat.st_mode;
+	flags = XFS_ILOG_CORE;
+
+	switch (file_stat.st_mode & S_IFMT) {
+	case S_IFDIR:
+		xname.type = XFS_DIR3_FT_DIR;
+		create_directory_inode(mp, pip, fsxp, mode, creds, xname, flags,
+				       file_stat, fd, entry->d_name, cur_path_buf);
+		return;
+	case S_IFREG:
+		xname.type = XFS_DIR3_FT_REG_FILE;
+		fname = entry->d_name;
+		break;
+	case S_IFCHR:
+		flags |= XFS_ILOG_DEV;
+		xname.type = XFS_DIR3_FT_CHRDEV;
+		majdev = major(file_stat.st_rdev);
+		mindev = minor(file_stat.st_rdev);
+		rdev = IRIX_MKDEV(majdev, mindev);
+		fname = entry->d_name;
+		break;
+	case S_IFBLK:
+		flags |= XFS_ILOG_DEV;
+		xname.type = XFS_DIR3_FT_BLKDEV;
+		majdev = major(file_stat.st_rdev);
+		mindev = minor(file_stat.st_rdev);
+		rdev = IRIX_MKDEV(majdev, mindev);
+		fname = entry->d_name;
+		break;
+	case S_IFLNK:
+		/*
+		 * Being a symlink we opened the filedescriptor with O_PATH
+		 * this will make flistxattr() and fgetxattr() fail wil EBADF,
+		 * so we  will need to fallback to llistxattr() and lgetxattr(),
+		 * this will need the full path to the original file, not just
+		 * the entry name.
+		 */
+		xname.type = XFS_DIR3_FT_SYMLINK;
+		fname = cur_path_buf;
+		break;
+	case S_IFIFO:
+		/*
+		 * Being a fifo we opened the filedescriptor with O_PATH
+		 * this will make flistxattr() and fgetxattr() fail wil EBADF,
+		 * so we  will need to fallback to llistxattr() and lgetxattr(),
+		 * this will need the full path to the original file, not just
+		 * the entry name.
+		 */
+		xname.type = XFS_DIR3_FT_FIFO;
+		fname = cur_path_buf;
+		break;
+	case S_IFSOCK:
+		/*
+		 * Being a socket we opened the filedescriptor with O_PATH
+		 * this will make flistxattr() and fgetxattr() fail wil EBADF,
+		 * so we  will need to fallback to llistxattr() and lgetxattr(),
+		 * this will need the full path to the original file, not just
+		 * the entry name.
+		 */
+		xname.type = XFS_DIR3_FT_SOCK;
+		fname = cur_path_buf;
+		break;
+	default:
+		break;
+	}
+
+	create_inode(mp, pip, fsxp, mode, creds, xname, flags, file_stat,
+		     rdev, fd, fname);
+}
+
+/*
+ * Walk_dir will recursively list files and directories and populate the
+ * mountpoint *mp with them using handle_direntry().
+ */
+static void
+walk_dir(
+	struct xfs_mount	*mp,
+	struct xfs_inode	*pip,
+	struct fsxattr		*fsxp,
+	char			*path_buf)
+{
+	DIR 			*dir;
+	struct dirent		*entry;
+
+	/*
+	 * Open input directory and iterate over all entries in it.
+	 * when another directory is found, we will recursively call walk_dir.
+	 */
+	if ((dir = opendir(path_buf)) == NULL) {
+		fprintf(stderr, _("%s: cannot open input dir: %s [%d - %s]\n"),
+				progname, path_buf, errno, strerror(errno));
+		exit(1);
+	}
+	while ((entry = readdir(dir)) != NULL) {
+		if (strcmp(entry->d_name, ".") == 0 ||
+		    strcmp(entry->d_name, "..") == 0)
+			continue;
+
+		handle_direntry(mp, pip, fsxp, path_buf, entry);
+	}
+	closedir(dir);
+}
+
+static void
+populate_from_dir(
+	struct xfs_mount	*mp,
+	struct fsxattr		*fsxp,
+	char			*cur_path)
+{
+	int 			error;
+	int 			mode;
+	int 			fd = -1;
+	char			path_buf[PATH_MAX];
+	struct stat		file_stat;
+	struct xfs_inode	*ip;
+	struct xfs_trans	*tp;
+
+	/*
+	 * Initialize path_buf cur_path, strip trailing slashes they're
+	 * automatically added when walking the dir.
+	 */
+	if (strlen(cur_path) > 1 && cur_path[strlen(cur_path)-1] == '/')
+		cur_path[strlen(cur_path)-1] = '\0';
+	if (snprintf(path_buf, PATH_MAX, "%s", cur_path) >= PATH_MAX)
+		fail(_("path name too long"), ENAMETOOLONG);
+
+	if (lstat(path_buf, &file_stat) < 0) {
+		fprintf(stderr, _("%s: cannot stat '%s': %s (errno=%d)\n"),
+			progname, path_buf, strerror(errno), errno);
+		exit(1);
+	}
+	fd = open(path_buf, O_NOFOLLOW | O_RDONLY | O_NOATIME);
+	if (fd < 0) {
+		fprintf(stderr, _("%s: cannot open %s: %s\n"),
+			progname, path_buf, strerror(errno));
+		exit(1);
+	}
+
+	/*
+	 * We first ensure we have the root inode.
+	 */
+	struct cred creds = {
+		.cr_uid = file_stat.st_uid,
+		.cr_gid = file_stat.st_gid,
+	};
+	mode = file_stat.st_mode;
+
+	tp = getres(mp, 0);
+
+	error = creatproto(&tp, NULL, mode | S_IFDIR, 0, &creds, fsxp, &ip);
+	if (error)
+		fail(_("Inode allocation failed"), error);
+
+	mp->m_sb.sb_rootino = ip->i_ino;
+	libxfs_log_sb(tp);
+	newdirectory(mp, tp, ip, ip);
+	libxfs_trans_log_inode(tp, ip, XFS_ILOG_CORE);
+
+	error = -libxfs_trans_commit(tp);
+	if (error)
+		fail(_("Inode allocation failed"), error);
+
+	libxfs_parent_finish(mp, NULL);
+
+	/*
+	 * Copy over attributes.
+	 */
+	writeattrs(ip, path_buf, fd);
+	writefsxattrs(ip, fsxp);
+
+	/*
+	 * RT initialization. Do this here to ensure that the RT inodes get
+	 * placed after the root inode.
+	 */
+	error = create_metadir(mp);
+	if (error)
+		fail(_("Creation of the metadata directory inode failed"), error);
+
+	rtinit(mp);
+
+	/*
+	 * By nature of walk_dir() we could be opening a great number of fds
+	 * for deeply nested directory trees. try to bump max fds limit.
+	 */
+	bump_max_fds();
+
+	/*
+	 * Initialize the hardlinks tracker.
+	 */
+	init_hardlink_tracker();
+	/*
+	 * Now that we have a root inode, let's walk the input dir and populate
+	 * the partition.
+	 */
+	walk_dir(mp, ip, fsxp, path_buf);
+
+	/*
+	 * Cleanup hardlinks tracker.
+	 */
+	cleanup_hardlink_tracker();
+
+	/*
+	 * We free up our root inode only when we finished populating the root
+	 * filesystem.
+	 */
+	libxfs_irele(ip);
+}
diff --git a/mkfs/proto.h b/mkfs/proto.h
index be1ceb45..476f7851 100644
--- a/mkfs/proto.h
+++ b/mkfs/proto.h
@@ -6,9 +6,21 @@
 #ifndef MKFS_PROTO_H_
 #define MKFS_PROTO_H_

-char *setup_proto(char *fname);
-void parse_proto(struct xfs_mount *mp, struct fsxattr *fsx, char **pp,
-		int proto_slashes_are_spaces);
+enum proto_source_type {
+	PROTO_SRC_NONE = 0,
+	PROTO_SRC_PROTOFILE,
+	PROTO_SRC_DIR
+};
+struct proto_source {
+	enum	proto_source_type type;
+	char	*data;
+};
+
+struct proto_source setup_proto(char *fname);
+void parse_proto(struct xfs_mount *mp, struct fsxattr *fsx,
+		 struct proto_source *protosource,
+		 int proto_slashes_are_spaces,
+		 int proto_preserve_atime);
 void res_failed(int err);

 #endif /* MKFS_PROTO_H_ */
diff --git a/mkfs/xfs_mkfs.c b/mkfs/xfs_mkfs.c
index 812241c4..885377f1 100644
--- a/mkfs/xfs_mkfs.c
+++ b/mkfs/xfs_mkfs.c
@@ -123,6 +123,7 @@ enum {

 enum {
 	P_FILE = 0,
+	P_ATIME,
 	P_SLASHES,
 	P_MAX_OPTS,
 };
@@ -714,6 +715,7 @@ static struct opt_params popts = {
 	.ini_section = "proto",
 	.subopts = {
 		[P_FILE] = "file",
+		[P_ATIME] = "atime",
 		[P_SLASHES] = "slashes_are_spaces",
 		[P_MAX_OPTS] = NULL,
 	},
@@ -722,6 +724,12 @@ static struct opt_params popts = {
 		  .conflicts = { { NULL, LAST_CONFLICT } },
 		  .defaultval = SUBOPT_NEEDS_VAL,
 		},
+		{ .index = P_ATIME,
+		  .conflicts = { { NULL, LAST_CONFLICT } },
+		  .minval = 0,
+		  .maxval = 1,
+		  .defaultval = 1,
+		},
 		{ .index = P_SLASHES,
 		  .conflicts = { { NULL, LAST_CONFLICT } },
 		  .minval = 0,
@@ -1079,6 +1087,7 @@ struct cli_params {
 	int	lsunit;
 	int	is_supported;
 	int	proto_slashes_are_spaces;
+	int	proto_atime;
 	int	data_concurrency;
 	int	log_concurrency;
 	int	rtvol_concurrency;
@@ -1206,6 +1215,7 @@ usage( void )
 /* naming */		[-n size=num,version=2|ci,ftype=0|1,parent=0|1]]\n\
 /* no-op info only */	[-N]\n\
 /* prototype file */	[-p fname]\n\
+/* populate from directory */	[-p dirname,atime=0|1]\n\
 /* quiet */		[-q]\n\
 /* realtime subvol */	[-r extsize=num,size=num,rtdev=xxx,rgcount=n,rgsize=n,\n\
 			    concurrency=num,zoned=0|1,start=n,reserved=n]\n\
@@ -2131,6 +2141,9 @@ proto_opts_parser(
 	case P_SLASHES:
 		cli->proto_slashes_are_spaces = getnum(value, opts, subopt);
 		break;
+	case P_ATIME:
+		cli->proto_atime = getnum(value, opts, subopt);
+		break;
 	case P_FILE:
 		fallthrough;
 	default:
@@ -5682,7 +5695,7 @@ main(
 	int			discard = 1;
 	int			force_overwrite = 0;
 	int			quiet = 0;
-	char			*protostring = NULL;
+	struct proto_source	protosource;
 	int			worst_freelist = 0;

 	struct libxfs_init	xi = {
@@ -5832,8 +5845,6 @@ main(
 	 */
 	cfgfile_parse(&cli);

-	protostring = setup_proto(cli.protofile);
-
 	/*
 	 * Extract as much of the valid config as we can from the CLI input
 	 * before opening the libxfs devices.
@@ -6010,7 +6021,11 @@ main(
 	/*
 	 * Allocate the root inode and anything else in the proto file.
 	 */
-	parse_proto(mp, &cli.fsx, &protostring, cli.proto_slashes_are_spaces);
+	protosource = setup_proto(cli.protofile);
+	parse_proto(mp, &cli.fsx,
+			&protosource,
+			cli.proto_slashes_are_spaces,
+			cli.proto_atime);

 	/*
 	 * Protect ourselves against possible stupidity
--
2.50.0

Re: [PATCH v11 1/1] proto: add ability to populate a filesystem from a directory

[Darrick J. Wong]

On Mon, Jul 28, 2025 at 05:29:21PM +0200, Luca Di Maio wrote:
> This patch implements the functionality to populate a newly created XFS
> filesystem directly from an existing directory structure.
> 
> It resuses existing protofile logic, it branches if input is a
> directory.
> 
> The population process steps are as follows:
>   - create the root inode before populating content
>   - recursively process nested directories
>   - handle regular files, directories, symlinks, char devices, block
>     devices, sockets, fifos
>   - preserve attributes (ownership, permissions)
>   - preserve mtime timestamps from source files to maintain file history
>     - use current time for atime/ctime/crtime
>     - possible to specify atime=1 to preserve atime timestamps from
>       source files
>   - preserve extended attributes and fsxattrs for all file types
>   - preserve hardlinks
> 
> At the moment, the implementation for the hardlink tracking is very
> simple, as it involves a linear search.
> from my local testing using larger source directories
> (1.3mln inodes, ~400k hardlinks) the difference was actually
> just a few seconds (given that most of the time is doing i/o).
> We might want to revisit that in the future if this becomes a
> bottleneck.
> 
> This functionality makes it easier to create populated filesystems
> without having to mount them, it's particularly useful for
> reproducible builds.
> 
> Signed-off-by: Luca Di Maio <luca.dimaio1@gmail.com>
> ---
>  man/man8/mkfs.xfs.8.in |  41 ++-
>  mkfs/proto.c           | 748 ++++++++++++++++++++++++++++++++++++++++-
>  mkfs/proto.h           |  18 +-
>  mkfs/xfs_mkfs.c        |  23 +-
>  4 files changed, 804 insertions(+), 26 deletions(-)
> 
> diff --git a/man/man8/mkfs.xfs.8.in b/man/man8/mkfs.xfs.8.in
> index bc804931..8c393ba2 100644
> --- a/man/man8/mkfs.xfs.8.in
> +++ b/man/man8/mkfs.xfs.8.in
> @@ -28,7 +28,7 @@ mkfs.xfs \- construct an XFS filesystem
>  .I naming_options
>  ] [
>  .B \-p
> -.I protofile_options
> +.I prototype_options
>  ] [
>  .B \-q
>  ] [
> @@ -977,30 +977,39 @@ option set.
>  .PP
>  .PD 0
>  .TP
> -.BI \-p " protofile_options"
> +.BI \-p " prototype_options"
>  .TP
>  .BI "Section Name: " [proto]
>  .PD
> -These options specify the protofile parameters for populating the filesystem.
> +These options specify the prototype parameters for populating the filesystem.
>  The valid
> -.I protofile_options
> +.I prototype_options
>  are:
>  .RS 1.2i
>  .TP
> -.BI [file=] protofile
> +.BI [file=]
>  The
>  .B file=
>  prefix is not required for this CLI argument for legacy reasons.
>  If specified as a config file directive, the prefix is required.
> -
> +.TP
> +.BI [file=] directory
>  If the optional
>  .PD
> -.I protofile
> -argument is given,
> +.I prototype
> +argument is given, and it's a directory,
>  .B mkfs.xfs
> -uses
> -.I protofile
> -as a prototype file and takes its directions from that file.
> +will populate the root file system with the contents of the given directory.

"...of the given directory tree."

(It's a subtle hint that it recursively imports the whole tree, not
one single directory)

> +Content, timestamps (atime, mtime), attributes and extended attributes
> +are preserved for all file types.
> +.TP
> +.BI [file=] protofile
> +If the optional
> +.PD
> +.I prototype
> +argument is given, and points to a regular file,
> +.B mkfs.xfs
> +uses it as a prototype file and takes its directions from that file.
>  The blocks and inodes specifiers in the
>  .I protofile
>  are provided for backwards compatibility, but are otherwise unused.
> @@ -1136,8 +1145,16 @@ always terminated with the dollar (
>  .B $
>  ) token.
>  .TP
> +.BI atime= value
> +If set to 1, when we're populating the root filesystem from a directory (

Who is "we"?

"If set to 1, mkfs will copy in access timestamps from the source
files.
Otherwise, access timestamps will be set to the current time."

> +.B file=directory
> +option)
> +access times are going to be preserved and are copied from the source files.
> +Set to 0 to set access times to the current time instead.
> +By default, this is set to 0.
> +.TP
>  .BI slashes_are_spaces= value
> -If set to 1, slashes ("/") in the first token of each line of the protofile
> +If set to 1, slashes ("/") in the first token of each line of the prototype file
>  are converted to spaces.
>  This enables the creation of a filesystem containing filenames with spaces.
>  By default, this is set to 0.
> diff --git a/mkfs/proto.c b/mkfs/proto.c
> index 7f80bef8..31626211 100644
> --- a/mkfs/proto.c
> +++ b/mkfs/proto.c
> @@ -5,6 +5,8 @@
>   */
> 
>  #include "libxfs.h"
> +#include <dirent.h>
> +#include <sys/resource.h>
>  #include <sys/stat.h>
>  #include <sys/xattr.h>
>  #include <linux/xattr.h>
> @@ -21,6 +23,11 @@ static void rsvfile(xfs_mount_t *mp, xfs_inode_t *ip, long long len);
>  static int newregfile(char **pp, char **fname);
>  static void rtinit(xfs_mount_t *mp);
>  static off_t filesize(int fd);
> +static void populate_from_dir(struct xfs_mount *mp, struct fsxattr *fsxp,
> +		char *source_dir);
> +static void walk_dir(struct xfs_mount *mp, struct xfs_inode *pip,
> +		struct fsxattr *fsxp, char *path_buf);
> +static int preserve_atime;
>  static int slashes_are_spaces;
> 
>  /*
> @@ -54,7 +61,7 @@ getnum(
>  	return i;
>  }
> 
> -char *
> +struct proto_source
>  setup_proto(
>  	char	*fname)
>  {
> @@ -63,14 +70,40 @@ setup_proto(
>  	int		fd;
>  	long		size;
> 
> -	if (!fname)
> -		return dflt;
> +	struct proto_source	result = {};
> +	struct stat	statbuf;
> +
> +	/*
> +	 * If no prototype path is supplied, use the default protofile which
> +	 * creates only a root directory.
> +	 */
> +	if (!fname) {
> +		result.type = PROTO_SRC_PROTOFILE;
> +		result.data = dflt;
> +		return result;
> +	}
> +
>  	if ((fd = open(fname, O_RDONLY)) < 0 || (size = filesize(fd)) < 0) {
>  		fprintf(stderr, _("%s: failed to open %s: %s\n"),
>  			progname, fname, strerror(errno));
>  		goto out_fail;
>  	}
> 
> +	if (fstat(fd, &statbuf) < 0)
> +		fail(_("invalid or unreadable source path"), errno);
> +
> +	/*
> +	 * Handle directory inputs.
> +	 */
> +	if (S_ISDIR(statbuf.st_mode)) {
> +		result.type = PROTO_SRC_DIR;
> +		result.data = fname;
> +		return result;

Er... this leaks the fd that you opened above.

> +	}
> +
> +	/*
> +	 * Else this is a protofile, let's handle traditionally.
> +	 */
>  	buf = malloc(size + 1);
>  	if (read(fd, buf, size) < size) {
>  		fprintf(stderr, _("%s: read failed on %s: %s\n"),
> @@ -90,7 +123,10 @@ setup_proto(
>  	(void)getnum(getstr(&buf), 0, 0, false);	/* block count */
>  	(void)getnum(getstr(&buf), 0, 0, false);	/* inode count */
>  	close(fd);
> -	return buf;
> +
> +	result.type = PROTO_SRC_PROTOFILE;
> +	result.data = buf;
> +	return result;
> 
>  out_fail:
>  	if (fd >= 0)
> @@ -379,6 +415,13 @@ writeattr(
>  	int			error;
> 
>  	ret = fgetxattr(fd, attrname, valuebuf, valuelen);
> +	/*
> +	 * In case of filedescriptors with O_PATH, fgetxattr() will fail with
> +	 * EBADF.
> +	 * Let's try to fallback to lgetxattr() using input path.
> +	 */
> +	if (ret < 0 && errno == EBADF)
> +		ret = lgetxattr(fname, attrname, valuebuf, valuelen);
>  	if (ret < 0) {
>  		if (errno == EOPNOTSUPP)
>  			return;
> @@ -425,6 +468,13 @@ writeattrs(
>  		fail(_("error allocating xattr name buffer"), errno);
> 
>  	ret = flistxattr(fd, namebuf, XATTR_LIST_MAX);
> +	/*
> +	 * In case of filedescriptors with O_PATH, flistxattr() will fail with
> +	 * EBADF.
> +	 * Let's try to fallback to llistxattr() using input path.
> +	 */
> +	if (ret < 0 && errno == EBADF)
> +		ret = llistxattr(fname, namebuf, XATTR_LIST_MAX);
>  	if (ret < 0) {
>  		if (errno == EOPNOTSUPP)
>  			goto out_namebuf;
> @@ -933,11 +983,27 @@ void
>  parse_proto(
>  	xfs_mount_t	*mp,
>  	struct fsxattr	*fsx,
> -	char		**pp,
> -	int		proto_slashes_are_spaces)
> +	struct proto_source	*protosource,
> +	int		proto_slashes_are_spaces,
> +	int		proto_preserve_atime)
>  {
>  	slashes_are_spaces = proto_slashes_are_spaces;
> -	parseproto(mp, NULL, fsx, pp, NULL);
> +	preserve_atime = proto_preserve_atime;
> +
> +	/*
> +	 * In case of a file input, we will use the prototype file logic else
> +	 * we will fallback to populate from dir.
> +	 */
> +	switch(protosource->type) {
> +	case PROTO_SRC_PROTOFILE:
> +		parseproto(mp, NULL, fsx, &protosource->data, NULL);
> +		break;
> +	case PROTO_SRC_DIR:
> +		populate_from_dir(mp, fsx, protosource->data);
> +		break;
> +	case PROTO_SRC_NONE:
> +		fail(_("invalid or unreadable source path"), ENOENT);
> +	}
>  }
> 
>  /* Create a sb-rooted metadata file. */
> @@ -1172,3 +1238,671 @@ filesize(
>  		return -1;
>  	return stb.st_size;
>  }
> +
> +/* Try to allow as many open directories as possible. */
> +static void
> +bump_max_fds(void)
> +{
> +	struct rlimit	rlim = {};
> +	int		ret;
> +
> +	ret = getrlimit(RLIMIT_NOFILE, &rlim);
> +	if (ret)
> +		return;
> +
> +	rlim.rlim_cur = rlim.rlim_max;
> +	ret = setrlimit(RLIMIT_NOFILE, &rlim);
> +	if (ret < 0)
> +		fprintf(stderr, _("%s: could not bump fd limit: [ %d - %s]\n"),
> +			progname, errno, strerror(errno));
> +}
> +
> +static void
> +writefsxattrs(
> +	struct xfs_inode	*ip,
> +	struct fsxattr	*fsxp)

		      ^ one more tab to line these up, please
> +{
> +	ip->i_projid = fsxp->fsx_projid;
> +	ip->i_extsize = fsxp->fsx_extsize;
> +	ip->i_diflags = xfs_flags2diflags(ip, fsxp->fsx_xflags);
> +	if (xfs_has_v3inodes(ip->i_mount)) {
> +		ip->i_diflags2 = xfs_flags2diflags2(ip, fsxp->fsx_xflags);
> +		ip->i_cowextsize = fsxp->fsx_cowextsize;
> +	}
> +}
> +
> +static void
> +writetimestamps(
> +	struct xfs_inode	*ip,
> +	struct stat	*statbuf)

		      ^ one more tab to line these up, please
> +{
> +	struct timespec64	ts;
> +
> +	/*
> +	 * Copy timestamps from source file to destination inode.
> +	 * Usually reproducible archives will delete or not register
> +	 * atime and ctime, for example:
> +	 *    https://www.gnu.org/software/tar/manual/html_section/Reproducibility.html
> +	 * hence we will only copy mtime, and let ctime/crtime be set to
> +	 * current time.
> +	 * atime will be copied over if atime is true.
> +	 */
> +	ts.tv_sec = statbuf->st_mtim.tv_sec;
> +	ts.tv_nsec = statbuf->st_mtim.tv_nsec;
> +	inode_set_mtime_to_ts(VFS_I(ip), ts);
> +
> +	/*
> +	 * In case of atime option, we will copy the atime  timestamp
> +	 * from source.
> +	 */
> +	if (preserve_atime) {
> +		ts.tv_sec = statbuf->st_atim.tv_sec;
> +		ts.tv_nsec = statbuf->st_atim.tv_nsec;
> +		inode_set_atime_to_ts(VFS_I(ip), ts);
> +	}
> +}
> +
> +struct hardlink {
> +	ino_t	src_ino;

	      ^ one more tab to line these up, please

> +	xfs_ino_t	dst_ino;
> +};
> +
> +struct hardlinks {
> +	size_t		count;
> +	size_t		size;
> +	struct hardlink	*entries;
> +};
> +
> +/* Growth strategy for hardlink tracking array */
> +/* Double size for small arrays */
> +#define HARDLINK_DEFAULT_GROWTH_FACTOR	2
> +/* Grow by 25% for large arrays */
> +#define HARDLINK_LARGE_GROWTH_FACTOR	0.25
> +/* Threshold to switch growth strategies */
> +#define HARDLINK_THRESHOLD		1024
> +/* Initial allocation size */
> +#define HARDLINK_TRACKER_INITIAL_SIZE	4096
> +
> +/*
> + * Keep track of source inodes that are from hardlinks so we can retrieve them
> + * when needed to setup in destination.
> + */
> +static struct hardlinks hardlink_tracker = { 0 };
> +
> +static void
> +init_hardlink_tracker(void)
> +{
> +	hardlink_tracker.size = HARDLINK_TRACKER_INITIAL_SIZE;
> +	hardlink_tracker.entries = calloc(
> +			hardlink_tracker.size,
> +			sizeof(struct hardlink));
> +	if (!hardlink_tracker.entries)
> +		fail(_("error allocating hardlinks tracking array"), errno);
> +}
> +
> +static void
> +cleanup_hardlink_tracker(void)
> +{
> +	free(hardlink_tracker.entries);
> +	hardlink_tracker.entries = NULL;
> +	hardlink_tracker.count = 0;
> +	hardlink_tracker.size = 0;
> +}
> +
> +static xfs_ino_t
> +get_hardlink_dst_inode(
> +	xfs_ino_t	i_ino)
> +{
> +	for (size_t i = 0; i < hardlink_tracker.count; i++) {
> +		if (hardlink_tracker.entries[i].src_ino == i_ino)
> +			return hardlink_tracker.entries[i].dst_ino;
> +	}
> +	return 0;
> +}
> +
> +static void
> +track_hardlink_inode(
> +	ino_t	src_ino,
> +	xfs_ino_t	dst_ino)
> +{
> +	if (hardlink_tracker.count >= hardlink_tracker.size) {
> +		/*
> +		 * double for smaller capacity.
> +		 * instead grow by 25% steps for larger capacities.
> +		 */
> +		const size_t old_size = hardlink_tracker.size;
> +		size_t new_size = old_size * HARDLINK_DEFAULT_GROWTH_FACTOR;
> +		if (old_size > HARDLINK_THRESHOLD)
> +			new_size = old_size + (old_size * HARDLINK_LARGE_GROWTH_FACTOR);
> +
> +		struct hardlink *resized_array = reallocarray(
> +			hardlink_tracker.entries,
> +			new_size,
> +			sizeof(struct hardlink));

Continuations should be indented twice (not once) so that they don't
line up with code that's in the body of an if statement:

> +		if (!resized_array)
> +			fail(_("error enlarging hardlinks tracking array"), errno);
> +
> +		memset(&resized_array[old_size], 0,
> +				(new_size - old_size) * sizeof(struct hardlink));
> +
> +		hardlink_tracker.entries = resized_array;
> +		hardlink_tracker.size = new_size;
> +	}
> +	hardlink_tracker.entries[hardlink_tracker.count].src_ino = src_ino;
> +	hardlink_tracker.entries[hardlink_tracker.count].dst_ino = dst_ino;
> +	hardlink_tracker.count++;
> +}
> +
> +/*
> + * This function will first check in our tracker if the input hardlink has
> + * already been stored, if not report false so create_inode() can continue
> + * handling the inode as regularly, and later save the source inode in our
> + * buffer for future consumption.
> + */
> +static bool
> +handle_hardlink(
> +	struct xfs_mount	*mp,
> +	struct xfs_inode	*pip,
> +	struct xfs_name	xname,
> +	struct stat	file_stat)
> +{
> +	int		error;

		      ^ one more tab to line these up, please

> +	xfs_ino_t		dst_ino;
> +	struct xfs_inode	*ip;
> +	struct xfs_trans	*tp;
> +	struct xfs_parent_args *ppargs = NULL;
> +
> +	tp = getres(mp, 0);
> +	ppargs = newpptr(mp);
> +	dst_ino = get_hardlink_dst_inode(file_stat.st_ino);
> +
> +	/*
> +	 * We didn't find the hardlink inode, this means it's the first time
> +	 * we see it, report error so create_inode() can continue handling the
> +	 * inode as a regular file type, and later save the source inode in our
> +	 * buffer for future consumption.
> +	 */
> +	if (dst_ino == 0)
> +		return false;
> +
> +	error = libxfs_iget(mp, NULL, dst_ino, 0, &ip);
> +	if (error)
> +		fail(_("failed to get inode"), error);
> +
> +	/*
> +	 * In case the inode was already in our tracker we need to setup the
> +	 * hardlink and skip file copy.
> +	 */
> +	libxfs_trans_ijoin(tp, pip, 0);
> +	libxfs_trans_ijoin(tp, ip, 0);
> +	newdirent(mp, tp, pip, &xname, ip, ppargs);
> +
> +	/*
> +	 * Increment the link count
> +	 */
> +	libxfs_bumplink(tp, ip);
> +
> +	libxfs_trans_log_inode(tp, ip, XFS_ILOG_CORE);
> +
> +	error = -libxfs_trans_commit(tp);
> +	if (error)
> +		fail(_("Error encountered creating file from prototype file"), error);
> +
> +	libxfs_parent_finish(mp, ppargs);
> +	libxfs_irele(ip);
> +
> +	return true;
> +}
> +
> +static void
> +create_directory_inode(
> +	struct xfs_mount	*mp,
> +	struct xfs_inode	*pip,
> +	struct fsxattr  	*fsxp,

                      ^^ space before tab

> +	int		mode,
> +	struct cred	creds,
> +	struct xfs_name	xname,
> +	int		flags,
> +	struct stat	file_stat,
> +	int 		fd,

           ^ space before tab

> +	char		*entryname,
> +	char		*path)
> +{
> +
> +	int		error;
		      ^ one more tab to line these up, please

> +	struct xfs_inode	*ip;
> +	struct xfs_trans	*tp;
> +	struct xfs_parent_args *ppargs = NULL;
> +
> +	tp = getres(mp, 0);
> +	ppargs = newpptr(mp);
> +
> +	error = creatproto(&tp, pip, mode, 0, &creds, fsxp, &ip);
> +	if (error)
> +		fail(_("Inode allocation failed"), error);
> +
> +	libxfs_trans_ijoin(tp, pip, 0);
> +
> +	newdirent(mp, tp, pip, &xname, ip, ppargs);
> +
> +	libxfs_bumplink(tp, pip);
> +	libxfs_trans_log_inode(tp, pip, XFS_ILOG_CORE);
> +	newdirectory(mp, tp, ip, pip);
> +
> +	/*
> +	 * Copy over timestamps.
> +	 */
> +	writetimestamps(ip, &file_stat);
> +
> +	libxfs_trans_log_inode(tp, ip, flags);
> +
> +	error = -libxfs_trans_commit(tp);
> +	if (error)
> +		fail(_("Directory inode allocation failed."), error);
> +
> +	libxfs_parent_finish(mp, ppargs);
> +	tp = NULL;
> +
> +	/*
> +	 * Copy over attributes.
> +	 */
> +	writeattrs(ip, entryname, fd);
> +	writefsxattrs(ip, fsxp);
> +	close(fd);
> +
> +	walk_dir(mp, ip, fsxp, path);
> +
> +	libxfs_irele(ip);
> +}
> +
> +static void
> +create_inode(

Might want to call this create_nondir_inode to distinguish it from
create_directory_inode.

> +	struct xfs_mount	*mp,
> +	struct xfs_inode	*pip,
> +	struct fsxattr  	*fsxp,

                      ^^ space before tab

> +	int		mode,
> +	struct cred	creds,
> +	struct xfs_name	xname,
> +	int		flags,
> +	struct stat	file_stat,
> +	xfs_dev_t	rdev,
> +	int 		fd,

           ^ space before tab

> +	char		*fname)

Hrmm, is @xname the filename we're creating in pip, and @fname is the
path to the original file so that we can copy the contents and various
other attributes?  If so, then maybe s/fname/src_fname/ to make this
clearer?

> +{
> +
> +	char		link_target[XFS_SYMLINK_MAXLEN];
> +	ssize_t		link_len;
> +	int		error;
		      ^ one more tab to line these up, please

> +	struct xfs_inode	*ip;
> +	struct xfs_trans	*tp;
> +	struct xfs_parent_args *ppargs = NULL;
> +
> +	/*
> +	 * If handle_hardlink() returns true it means the hardlink has been
> +	 * correctly found and set, so we don't need to do anything else.
> +	 */
> +	if (file_stat.st_nlink > 1 && handle_hardlink(mp, pip, xname, file_stat)) {
> +		close(fd);
> +		return;
> +	}
> +	/*
> +	 * If instead we have an error it means the hardlink was not registered,
> +	 * so we proceed to treat it like a regular file, and save it to our
> +	 * tracker later.
> +	 */
> +	tp = getres(mp, 0);
> +	/*
> +	 * In case of symlinks, we need to handle things a little differently.
> +	 * We need to read out our link target and act accordingly.
> +	 */
> +	if (xname.type == XFS_DIR3_FT_SYMLINK) {
> +		link_len = readlink(fname, link_target, XFS_SYMLINK_MAXLEN);
> +		if (link_len < 0)
> +			fail(_("could not resolve symlink"), errno);
> +		if (link_len >= PATH_MAX)
> +			fail(_("symlink target too long"), ENAMETOOLONG);
> +		tp = getres(mp, XFS_B_TO_FSB(mp, link_len));
> +	}
> +	ppargs = newpptr(mp);
> +
> +	error = creatproto(&tp, pip, mode, rdev, &creds, fsxp, &ip);
> +	if (error)
> +		fail(_("Inode allocation failed"), error);
> +
> +	/*
> +	 * In case of symlinks, we now write it down, for other file types
> +	 * this is handled later before cleanup.
> +	 */
> +	if (xname.type == XFS_DIR3_FT_SYMLINK)
> +		writesymlink(tp, ip, link_target, link_len);
> +
> +	libxfs_trans_ijoin(tp, pip, 0);
> +	newdirent(mp, tp, pip, &xname, ip, ppargs);
> +
> +	/*
> +	 * Copy over timestamps.
> +	 */
> +	writetimestamps(ip, &file_stat);
> +
> +	libxfs_trans_log_inode(tp, ip, flags);
> +
> +	error = -libxfs_trans_commit(tp);
> +	if (error)
> +		fail(_("Error encountered creating file from prototype file"), error);
> +
> +	libxfs_parent_finish(mp, ppargs);
> +
> +	/*
> +	 * Copy over file content, attributes, extended attributes and
> +	 * timestamps.
> +	 */
> +	if (fd >= 0) {
> +		/* We need to writefile only when not dealing with a symlink. */
> +		if (xname.type != XFS_DIR3_FT_SYMLINK)
> +			writefile(ip, fname, fd);
> +		writeattrs(ip, fname, fd);
> +		close(fd);
> +	}
> +	/*
> +	 * We do fsxattr also for file types where we don't have an fd,
> +	 * for example FIFOs.
> +	 */
> +	writefsxattrs(ip, fsxp);
> +
> +	/*
> +	 * If we're here it means this is the first time we're encountering an
> +	 * hardlink, so we need to store it.
> +	 */
> +	if (file_stat.st_nlink > 1)
> +		track_hardlink_inode(file_stat.st_ino, ip->i_ino);
> +
> +	libxfs_irele(ip);
> +}
> +
> +static void
> +handle_direntry(
> +	struct xfs_mount	*mp,
> +	struct xfs_inode	*pip,
> +	struct fsxattr		*fsxp,
> +	char			*path_buf,
> +	struct dirent		*entry)
> +{
> +	char		cur_path_buf[PATH_MAX];

Hrmm.  For each level of the source directory tree we allocate another
PATH_MAX buffer on the stack just to snprintf from @path_buf (which is
itself a stack buffer).

Even assuming the usual 8M thread stack, that means a directory
structure more than ~2000 levels deep will overflow the stack and crash
mkfs.

I really think you ought to consider allocating /one/ buffer at the
start, passing (path_buf, path_len) down the stack, and then snprinting
at the end of the buffer:

	size_t avail = PATH_MAX - path_len;
	size_t wrote = snprintf(path_buf + path_len, avail, "/%s", entry->d_name);

	if (wrote > avail)
		fail(path_buf, ENAMETOOLONG);

	...

	if (S_ISDIR(...)) {
		create_directory_inode(..., path_buf, path_len + strlen(entry->d_name), ...);

	...

	path_buf[path_len] = 0;

One buffer, much less memory usage.

> +	char		*fname = "";

                        ^ another tab to line things up with the
                          declarations

> +	int 		pathfd,fd = -1;
> +	int 		flags;
> +	int 		majdev;
> +	int 		mindev;
> +	int 		mode;
> +	int 		rdev = 0;

           ^ space before tab

> +	struct stat	file_stat;
> +	struct xfs_name	xname;
> +
> +	/* Ensure we're within the limits of PATH_MAX. */
> +	if (strlen(path_buf) + strlen(entry->d_name) >= PATH_MAX)
> +		fail(_("path name too long"), ENAMETOOLONG);
> +	/*
> +	 * Keep a reference to our current file path it will be useful
> +	 * afterwards for fds open with O_PATH where we will have to fallback
> +	 * to path based attribute handling.
> +	 */
> +	if (snprintf(cur_path_buf, PATH_MAX, "%s/%s",
> +		     path_buf, entry->d_name) < 0)
> +		fail(_("path encoding error"), -1);
> +
> +	pathfd = open(path_buf, O_NOFOLLOW | O_PATH);
> +	if (pathfd < 0){
> +		fprintf(stderr, _("%s: cannot open %s: %s\n"), progname,
> +			cur_path_buf, strerror(errno));
> +		exit(1);
> +	}
> +
> +	/*
> +	 * Symlinks and sockets will need to be opened with O_PATH to work, so
> +	 * we handle this special case.
> +	 */
> +	fd = openat(pathfd, entry->d_name, O_NOFOLLOW | O_PATH);
> +	if(fd < 0) {
> +		fprintf(stderr, _("%s: cannot open %s: %s\n"), progname,
> +			cur_path_buf, strerror(errno));
> +		exit(1);
> +	}
> +
> +	if (fstat(fd, &file_stat) < 0) {
> +		fprintf(stderr, _("%s: cannot stat '%s': %s (errno=%d)\n"),
> +				progname, cur_path_buf, strerror(errno), errno);
> +		exit(1);
> +	}
> +
> +	/*
> +	 * Regular files instead need to be reopened with broader flags so we
> +	 * check if that's the case and reopen those.
> +	 */
> +	if (!S_ISSOCK(file_stat.st_mode) &&
> +	    !S_ISLNK(file_stat.st_mode)  &&
> +	    !S_ISFIFO(file_stat.st_mode)) {
> +		close(fd);
> +		fd = openat(pathfd, entry->d_name,
> +			    O_NOFOLLOW | O_RDONLY | O_NOATIME);

Just out of curiosity, does O_NOATIME not work in the previous openat?

> +		if(fd < 0) {
> +			fprintf(stderr, _("%s: cannot open %s: %s\n"), progname,
> +				cur_path_buf, strerror(errno));
> +			exit(1);
> +		}
> +	}
> +
> +	struct cred creds = {
> +		.cr_uid = file_stat.st_uid,
> +		.cr_gid = file_stat.st_gid,
> +	};
> +
> +	xname.name = (unsigned char *)entry->d_name;
> +	xname.len = strlen(entry->d_name);
> +	xname.type = 0;
> +	mode = file_stat.st_mode;
> +	flags = XFS_ILOG_CORE;
> +
> +	switch (file_stat.st_mode & S_IFMT) {
> +	case S_IFDIR:
> +		xname.type = XFS_DIR3_FT_DIR;
> +		create_directory_inode(mp, pip, fsxp, mode, creds, xname, flags,
> +				       file_stat, fd, entry->d_name, cur_path_buf);
> +		return;
> +	case S_IFREG:
> +		xname.type = XFS_DIR3_FT_REG_FILE;
> +		fname = entry->d_name;
> +		break;
> +	case S_IFCHR:
> +		flags |= XFS_ILOG_DEV;
> +		xname.type = XFS_DIR3_FT_CHRDEV;
> +		majdev = major(file_stat.st_rdev);
> +		mindev = minor(file_stat.st_rdev);
> +		rdev = IRIX_MKDEV(majdev, mindev);
> +		fname = entry->d_name;
> +		break;
> +	case S_IFBLK:
> +		flags |= XFS_ILOG_DEV;
> +		xname.type = XFS_DIR3_FT_BLKDEV;
> +		majdev = major(file_stat.st_rdev);
> +		mindev = minor(file_stat.st_rdev);
> +		rdev = IRIX_MKDEV(majdev, mindev);
> +		fname = entry->d_name;
> +		break;
> +	case S_IFLNK:
> +		/*
> +		 * Being a symlink we opened the filedescriptor with O_PATH
> +		 * this will make flistxattr() and fgetxattr() fail wil EBADF,
> +		 * so we  will need to fallback to llistxattr() and lgetxattr(),
> +		 * this will need the full path to the original file, not just
> +		 * the entry name.
> +		 */
> +		xname.type = XFS_DIR3_FT_SYMLINK;
> +		fname = cur_path_buf;
> +		break;
> +	case S_IFIFO:
> +		/*
> +		 * Being a fifo we opened the filedescriptor with O_PATH
> +		 * this will make flistxattr() and fgetxattr() fail wil EBADF,

"fail with EBADF"...

> +		 * so we  will need to fallback to llistxattr() and lgetxattr(),
> +		 * this will need the full path to the original file, not just
> +		 * the entry name.
> +		 */
> +		xname.type = XFS_DIR3_FT_FIFO;
> +		fname = cur_path_buf;
> +		break;
> +	case S_IFSOCK:
> +		/*
> +		 * Being a socket we opened the filedescriptor with O_PATH
> +		 * this will make flistxattr() and fgetxattr() fail wil EBADF,
> +		 * so we  will need to fallback to llistxattr() and lgetxattr(),
> +		 * this will need the full path to the original file, not just
> +		 * the entry name.
> +		 */
> +		xname.type = XFS_DIR3_FT_SOCK;
> +		fname = cur_path_buf;
> +		break;
> +	default:
> +		break;
> +	}
> +
> +	create_inode(mp, pip, fsxp, mode, creds, xname, flags, file_stat,
> +		     rdev, fd, fname);
> +}
> +
> +/*
> + * Walk_dir will recursively list files and directories and populate the
> + * mountpoint *mp with them using handle_direntry().
> + */
> +static void
> +walk_dir(
> +	struct xfs_mount	*mp,
> +	struct xfs_inode	*pip,
> +	struct fsxattr		*fsxp,
> +	char			*path_buf)
> +{
> +	DIR 			*dir;

           ^ space before tab

> +	struct dirent		*entry;
> +
> +	/*
> +	 * Open input directory and iterate over all entries in it.
> +	 * when another directory is found, we will recursively call walk_dir.
> +	 */
> +	if ((dir = opendir(path_buf)) == NULL) {
> +		fprintf(stderr, _("%s: cannot open input dir: %s [%d - %s]\n"),
> +				progname, path_buf, errno, strerror(errno));
> +		exit(1);
> +	}
> +	while ((entry = readdir(dir)) != NULL) {
> +		if (strcmp(entry->d_name, ".") == 0 ||
> +		    strcmp(entry->d_name, "..") == 0)
> +			continue;
> +
> +		handle_direntry(mp, pip, fsxp, path_buf, entry);
> +	}
> +	closedir(dir);
> +}
> +
> +static void
> +populate_from_dir(
> +	struct xfs_mount	*mp,
> +	struct fsxattr		*fsxp,
> +	char			*cur_path)
> +{
> +	int 			error;
> +	int 			mode;
> +	int 			fd = -1;

           ^ space before tab

> +	char			path_buf[PATH_MAX];
> +	struct stat		file_stat;
> +	struct xfs_inode	*ip;
> +	struct xfs_trans	*tp;
> +
> +	/*
> +	 * Initialize path_buf cur_path, strip trailing slashes they're
> +	 * automatically added when walking the dir.
> +	 */
> +	if (strlen(cur_path) > 1 && cur_path[strlen(cur_path)-1] == '/')
> +		cur_path[strlen(cur_path)-1] = '\0';
> +	if (snprintf(path_buf, PATH_MAX, "%s", cur_path) >= PATH_MAX)
> +		fail(_("path name too long"), ENAMETOOLONG);
> +
> +	if (lstat(path_buf, &file_stat) < 0) {
> +		fprintf(stderr, _("%s: cannot stat '%s': %s (errno=%d)\n"),
> +			progname, path_buf, strerror(errno), errno);
> +		exit(1);
> +	}
> +	fd = open(path_buf, O_NOFOLLOW | O_RDONLY | O_NOATIME);
> +	if (fd < 0) {
> +		fprintf(stderr, _("%s: cannot open %s: %s\n"),
> +			progname, path_buf, strerror(errno));
> +		exit(1);
> +	}
> +
> +	/*
> +	 * We first ensure we have the root inode.
> +	 */
> +	struct cred creds = {
> +		.cr_uid = file_stat.st_uid,
> +		.cr_gid = file_stat.st_gid,
> +	};
> +	mode = file_stat.st_mode;
> +
> +	tp = getres(mp, 0);
> +
> +	error = creatproto(&tp, NULL, mode | S_IFDIR, 0, &creds, fsxp, &ip);
> +	if (error)
> +		fail(_("Inode allocation failed"), error);
> +
> +	mp->m_sb.sb_rootino = ip->i_ino;
> +	libxfs_log_sb(tp);
> +	newdirectory(mp, tp, ip, ip);
> +	libxfs_trans_log_inode(tp, ip, XFS_ILOG_CORE);
> +
> +	error = -libxfs_trans_commit(tp);
> +	if (error)
> +		fail(_("Inode allocation failed"), error);
> +
> +	libxfs_parent_finish(mp, NULL);
> +
> +	/*
> +	 * Copy over attributes.
> +	 */
> +	writeattrs(ip, path_buf, fd);

Nothing closes fd here; does it leak?

--D

> +	writefsxattrs(ip, fsxp);
> +
> +	/*
> +	 * RT initialization. Do this here to ensure that the RT inodes get
> +	 * placed after the root inode.
> +	 */
> +	error = create_metadir(mp);
> +	if (error)
> +		fail(_("Creation of the metadata directory inode failed"), error);
> +
> +	rtinit(mp);
> +
> +	/*
> +	 * By nature of walk_dir() we could be opening a great number of fds
> +	 * for deeply nested directory trees. try to bump max fds limit.
> +	 */
> +	bump_max_fds();
> +
> +	/*
> +	 * Initialize the hardlinks tracker.
> +	 */
> +	init_hardlink_tracker();
> +	/*
> +	 * Now that we have a root inode, let's walk the input dir and populate
> +	 * the partition.
> +	 */
> +	walk_dir(mp, ip, fsxp, path_buf);
> +
> +	/*
> +	 * Cleanup hardlinks tracker.
> +	 */
> +	cleanup_hardlink_tracker();
> +
> +	/*
> +	 * We free up our root inode only when we finished populating the root
> +	 * filesystem.
> +	 */
> +	libxfs_irele(ip);
> +}
> diff --git a/mkfs/proto.h b/mkfs/proto.h
> index be1ceb45..476f7851 100644
> --- a/mkfs/proto.h
> +++ b/mkfs/proto.h
> @@ -6,9 +6,21 @@
>  #ifndef MKFS_PROTO_H_
>  #define MKFS_PROTO_H_
> 
> -char *setup_proto(char *fname);
> -void parse_proto(struct xfs_mount *mp, struct fsxattr *fsx, char **pp,
> -		int proto_slashes_are_spaces);
> +enum proto_source_type {
> +	PROTO_SRC_NONE = 0,
> +	PROTO_SRC_PROTOFILE,
> +	PROTO_SRC_DIR
> +};
> +struct proto_source {
> +	enum	proto_source_type type;
> +	char	*data;
> +};
> +
> +struct proto_source setup_proto(char *fname);
> +void parse_proto(struct xfs_mount *mp, struct fsxattr *fsx,
> +		 struct proto_source *protosource,
> +		 int proto_slashes_are_spaces,
> +		 int proto_preserve_atime);
>  void res_failed(int err);
> 
>  #endif /* MKFS_PROTO_H_ */
> diff --git a/mkfs/xfs_mkfs.c b/mkfs/xfs_mkfs.c
> index 812241c4..885377f1 100644
> --- a/mkfs/xfs_mkfs.c
> +++ b/mkfs/xfs_mkfs.c
> @@ -123,6 +123,7 @@ enum {
> 
>  enum {
>  	P_FILE = 0,
> +	P_ATIME,
>  	P_SLASHES,
>  	P_MAX_OPTS,
>  };
> @@ -714,6 +715,7 @@ static struct opt_params popts = {
>  	.ini_section = "proto",
>  	.subopts = {
>  		[P_FILE] = "file",
> +		[P_ATIME] = "atime",
>  		[P_SLASHES] = "slashes_are_spaces",
>  		[P_MAX_OPTS] = NULL,
>  	},
> @@ -722,6 +724,12 @@ static struct opt_params popts = {
>  		  .conflicts = { { NULL, LAST_CONFLICT } },
>  		  .defaultval = SUBOPT_NEEDS_VAL,
>  		},
> +		{ .index = P_ATIME,
> +		  .conflicts = { { NULL, LAST_CONFLICT } },
> +		  .minval = 0,
> +		  .maxval = 1,
> +		  .defaultval = 1,
> +		},
>  		{ .index = P_SLASHES,
>  		  .conflicts = { { NULL, LAST_CONFLICT } },
>  		  .minval = 0,
> @@ -1079,6 +1087,7 @@ struct cli_params {
>  	int	lsunit;
>  	int	is_supported;
>  	int	proto_slashes_are_spaces;
> +	int	proto_atime;
>  	int	data_concurrency;
>  	int	log_concurrency;
>  	int	rtvol_concurrency;
> @@ -1206,6 +1215,7 @@ usage( void )
>  /* naming */		[-n size=num,version=2|ci,ftype=0|1,parent=0|1]]\n\
>  /* no-op info only */	[-N]\n\
>  /* prototype file */	[-p fname]\n\
> +/* populate from directory */	[-p dirname,atime=0|1]\n\
>  /* quiet */		[-q]\n\
>  /* realtime subvol */	[-r extsize=num,size=num,rtdev=xxx,rgcount=n,rgsize=n,\n\
>  			    concurrency=num,zoned=0|1,start=n,reserved=n]\n\
> @@ -2131,6 +2141,9 @@ proto_opts_parser(
>  	case P_SLASHES:
>  		cli->proto_slashes_are_spaces = getnum(value, opts, subopt);
>  		break;
> +	case P_ATIME:
> +		cli->proto_atime = getnum(value, opts, subopt);
> +		break;
>  	case P_FILE:
>  		fallthrough;
>  	default:
> @@ -5682,7 +5695,7 @@ main(
>  	int			discard = 1;
>  	int			force_overwrite = 0;
>  	int			quiet = 0;
> -	char			*protostring = NULL;
> +	struct proto_source	protosource;
>  	int			worst_freelist = 0;
> 
>  	struct libxfs_init	xi = {
> @@ -5832,8 +5845,6 @@ main(
>  	 */
>  	cfgfile_parse(&cli);
> 
> -	protostring = setup_proto(cli.protofile);
> -
>  	/*
>  	 * Extract as much of the valid config as we can from the CLI input
>  	 * before opening the libxfs devices.
> @@ -6010,7 +6021,11 @@ main(
>  	/*
>  	 * Allocate the root inode and anything else in the proto file.
>  	 */
> -	parse_proto(mp, &cli.fsx, &protostring, cli.proto_slashes_are_spaces);
> +	protosource = setup_proto(cli.protofile);
> +	parse_proto(mp, &cli.fsx,
> +			&protosource,
> +			cli.proto_slashes_are_spaces,
> +			cli.proto_atime);
> 
>  	/*
>  	 * Protect ourselves against possible stupidity
> --
> 2.50.0
> 

Re: [PATCH v11 1/1] proto: add ability to populate a filesystem from a directory

[Luca Di Maio]

Thanks Darrick for the review!
Sorry again for this indentation mess, I'm going to basically align all
function arguments and all the top-function declarations
I was a bit confused because elsewhere in the code is not like that so
it's a bit difficult to infere

Offtopic: maybe we could also introduce an editorconfig setup? so that
all various editors will be correctly set to see the tabs/spaces as
needed (https://editorconfig.org/)

Back on topic:

On Tue, Jul 29, 2025 at 02:43:22PM -0700, Darrick J. Wong wrote:
> > +will populate the root file system with the contents of the given directory.
>
> "...of the given directory tree."
>
> (It's a subtle hint that it recursively imports the whole tree, not
> one single directory)

Ack.

> > +Content, timestamps (atime, mtime), attributes and extended attributes
> > +are preserved for all file types.
> > +.TP
> > +.BI [file=] protofile
> > +If the optional
> > +.PD
> > +.I prototype
> > +argument is given, and points to a regular file,
> > +.B mkfs.xfs
> > +uses it as a prototype file and takes its directions from that file.
> >  The blocks and inodes specifiers in the
> >  .I protofile
> >  are provided for backwards compatibility, but are otherwise unused.
> > @@ -1136,8 +1145,16 @@ always terminated with the dollar (
> >  .B $
> >  ) token.
> >  .TP
> > +.BI atime= value
> > +If set to 1, when we're populating the root filesystem from a directory (
>
> Who is "we"?
>
> "If set to 1, mkfs will copy in access timestamps from the source
> files.
> Otherwise, access timestamps will be set to the current time."
>

Ack.

> > +	 */
> > +	if (S_ISDIR(statbuf.st_mode)) {
> > +		result.type = PROTO_SRC_DIR;
> > +		result.data = fname;
> > +		return result;
>
> Er... this leaks the fd that you opened above.
>

Ack.

> > +static void
> > +create_inode(
>
> Might want to call this create_nondir_inode to distinguish it from
> create_directory_inode.
>

Ack.

> > +	char		*fname)
>
> Hrmm, is @xname the filename we're creating in pip, and @fname is the
> path to the original file so that we can copy the contents and various
> other attributes?  If so, then maybe s/fname/src_fname/ to make this
> clearer?
>

Ack.

> > +static void
> > +handle_direntry(
> > +	struct xfs_mount	*mp,
> > +	struct xfs_inode	*pip,
> > +	struct fsxattr		*fsxp,
> > +	char			*path_buf,
> > +	struct dirent		*entry)
> > +{
> > +	char		cur_path_buf[PATH_MAX];
>
> Hrmm.  For each level of the source directory tree we allocate another
> PATH_MAX buffer on the stack just to snprintf from @path_buf (which is
> itself a stack buffer).
>
> Even assuming the usual 8M thread stack, that means a directory
> structure more than ~2000 levels deep will overflow the stack and crash
> mkfs.
>
> I really think you ought to consider allocating /one/ buffer at the
> start, passing (path_buf, path_len) down the stack, and then snprinting
> at the end of the buffer:
>
> 	size_t avail = PATH_MAX - path_len;
> 	size_t wrote = snprintf(path_buf + path_len, avail, "/%s", entry->d_name);
>
> 	if (wrote > avail)
> 		fail(path_buf, ENAMETOOLONG);
>
> 	...
>
> 	if (S_ISDIR(...)) {
> 		create_directory_inode(..., path_buf, path_len + strlen(entry->d_name), ...);
>
> 	...
>
> 	path_buf[path_len] = 0;
>
> One buffer, much less memory usage.

Right, It was like this before but I've Inadvertently unoptimized this
in V11 when switched to openat(), will fix in next path, sorry.

> > +	if (!S_ISSOCK(file_stat.st_mode) &&
> > +	    !S_ISLNK(file_stat.st_mode)  &&
> > +	    !S_ISFIFO(file_stat.st_mode)) {
> > +		close(fd);
> > +		fd = openat(pathfd, entry->d_name,
> > +			    O_NOFOLLOW | O_RDONLY | O_NOATIME);
>
> Just out of curiosity, does O_NOATIME not work in the previous openat?

Actually on my test setup (mainly using docker/podman to test), opening
with and without O_NOATIME when using O_PATH, does not change accesstime
checking with `stat`, but also it works if I add it.
As a precautionary measure (not sure if podman/docker is messing with
noatime) I'll add it, as it seems to work correctly.

> > +		 * this will make flistxattr() and fgetxattr() fail wil EBADF,
>
> "fail with EBADF"...
>

Ack.

> > +	/*
> > +	 * Copy over attributes.
> > +	 */
> > +	writeattrs(ip, path_buf, fd);
>
> Nothing closes fd here; does it leak?
>
> --D
>

Ack.

L.

Re: [PATCH v11 1/1] proto: add ability to populate a filesystem from a directory

[Darrick J. Wong]

On Wed, Jul 30, 2025 at 04:40:39PM +0200, Luca Di Maio wrote:
> Thanks Darrick for the review!
> Sorry again for this indentation mess, I'm going to basically align all
> function arguments and all the top-function declarations
> I was a bit confused because elsewhere in the code is not like that so
> it's a bit difficult to infere
> 
> Offtopic: maybe we could also introduce an editorconfig setup? so that
> all various editors will be correctly set to see the tabs/spaces as
> needed (https://editorconfig.org/)

Hrmm, that /would/ be useful.

> Back on topic:
> 
> On Tue, Jul 29, 2025 at 02:43:22PM -0700, Darrick J. Wong wrote:

<snipping>

> > > +	if (!S_ISSOCK(file_stat.st_mode) &&
> > > +	    !S_ISLNK(file_stat.st_mode)  &&
> > > +	    !S_ISFIFO(file_stat.st_mode)) {
> > > +		close(fd);
> > > +		fd = openat(pathfd, entry->d_name,
> > > +			    O_NOFOLLOW | O_RDONLY | O_NOATIME);
> >
> > Just out of curiosity, does O_NOATIME not work in the previous openat?

[narrator: it doesn't]

> Actually on my test setup (mainly using docker/podman to test), opening
> with and without O_NOATIME when using O_PATH, does not change accesstime
> checking with `stat`, but also it works if I add it.
> As a precautionary measure (not sure if podman/docker is messing with
> noatime) I'll add it, as it seems to work correctly.

On second thought I think you might leave the double opens because
O_NOATIME is only allowed if the current user owns source file, or has
CAP_FOWNER.  If you're running mkfs as an unprivileged user trying to
capture a rootfs (with uid 0 files) then O_NOATIME won't be allowed.

Maybe something along the lines of:

	/*
	 * Try to open the source file noatime to avoid a flood of
	 * writes to the source fs, but we can fall back to plain
	 * readonly mode if we don't have enough permission.
	 */
	fd = openat(pathfd, entry->d_name, O_NOFOLLOW | O_RDONLY | O_NOATIME);
	if (fd < 0)
		fd = openat(pathfd, entry->d_name, O_NOFOLLOW | O_RDONLY);
	if (fd < 0)
		/* whine and exit */

Just to see if you can open the source file without touching atime?

--D

> > > +		 * this will make flistxattr() and fgetxattr() fail wil EBADF,
> >
> > "fail with EBADF"...
> >
> 
> Ack.
> 
> > > +	/*
> > > +	 * Copy over attributes.
> > > +	 */
> > > +	writeattrs(ip, path_buf, fd);
> >
> > Nothing closes fd here; does it leak?
> >
> > --D
> >
> 
> Ack.
> 
> L.

Re: [PATCH v11 1/1] proto: add ability to populate a filesystem from a directory

[Luca Di Maio]

On Wed, Jul 30, 2025 at 08:04:09AM -0700, Darrick J. Wong wrote:
> On Wed, Jul 30, 2025 at 04:40:39PM +0200, Luca Di Maio wrote:
> > Thanks Darrick for the review!
> > Sorry again for this indentation mess, I'm going to basically align all
> > function arguments and all the top-function declarations
> > I was a bit confused because elsewhere in the code is not like that so
> > it's a bit difficult to infere
> >
> > Offtopic: maybe we could also introduce an editorconfig setup? so that
> > all various editors will be correctly set to see the tabs/spaces as
> > needed (https://editorconfig.org/)
>
> Hrmm, that /would/ be useful.
>

Nice, will try to put something together for a future patch

> > Back on topic:
> >
> > On Tue, Jul 29, 2025 at 02:43:22PM -0700, Darrick J. Wong wrote:
>
> <snipping>
>
> > > > +	if (!S_ISSOCK(file_stat.st_mode) &&
> > > > +	    !S_ISLNK(file_stat.st_mode)  &&
> > > > +	    !S_ISFIFO(file_stat.st_mode)) {
> > > > +		close(fd);
> > > > +		fd = openat(pathfd, entry->d_name,
> > > > +			    O_NOFOLLOW | O_RDONLY | O_NOATIME);
> > >
> > > Just out of curiosity, does O_NOATIME not work in the previous openat?
>
> [narrator: it doesn't]
>
> > Actually on my test setup (mainly using docker/podman to test), opening
> > with and without O_NOATIME when using O_PATH, does not change accesstime
> > checking with `stat`, but also it works if I add it.
> > As a precautionary measure (not sure if podman/docker is messing with
> > noatime) I'll add it, as it seems to work correctly.
>
> On second thought I think you might leave the double opens because
> O_NOATIME is only allowed if the current user owns source file, or has
> CAP_FOWNER.  If you're running mkfs as an unprivileged user trying to
> capture a rootfs (with uid 0 files) then O_NOATIME won't be allowed.
>
> Maybe something along the lines of:
>
> 	/*
> 	 * Try to open the source file noatime to avoid a flood of
> 	 * writes to the source fs, but we can fall back to plain
> 	 * readonly mode if we don't have enough permission.
> 	 */
> 	fd = openat(pathfd, entry->d_name, O_NOFOLLOW | O_RDONLY | O_NOATIME);
> 	if (fd < 0)
> 		fd = openat(pathfd, entry->d_name, O_NOFOLLOW | O_RDONLY);
> 	if (fd < 0)
> 		/* whine and exit */
>
> Just to see if you can open the source file without touching atime?
>
> --D

Right, will do like this, didn't think about rootless/unprivileged mkfs

L.

Re: [PATCH v11 1/1] proto: add ability to populate a filesystem from a directory

[Darrick J. Wong]

On Wed, Jul 30, 2025 at 05:20:42PM +0200, Luca Di Maio wrote:
> On Wed, Jul 30, 2025 at 08:04:09AM -0700, Darrick J. Wong wrote:
> > On Wed, Jul 30, 2025 at 04:40:39PM +0200, Luca Di Maio wrote:
> > > Thanks Darrick for the review!
> > > Sorry again for this indentation mess, I'm going to basically align all
> > > function arguments and all the top-function declarations
> > > I was a bit confused because elsewhere in the code is not like that so
> > > it's a bit difficult to infere
> > >
> > > Offtopic: maybe we could also introduce an editorconfig setup? so that
> > > all various editors will be correctly set to see the tabs/spaces as
> > > needed (https://editorconfig.org/)
> >
> > Hrmm, that /would/ be useful.
> >
> 
> Nice, will try to put something together for a future patch

Either that or a indent/clang-format wrapper?

The kernel has scripts/Lindent, maybe someone should try to write one
for xfs style if I don't get to it first.

<sigh>rustfmt<duck>

> > > Back on topic:
> > >
> > > On Tue, Jul 29, 2025 at 02:43:22PM -0700, Darrick J. Wong wrote:
> >
> > <snipping>
> >
> > > > > +	if (!S_ISSOCK(file_stat.st_mode) &&
> > > > > +	    !S_ISLNK(file_stat.st_mode)  &&
> > > > > +	    !S_ISFIFO(file_stat.st_mode)) {
> > > > > +		close(fd);
> > > > > +		fd = openat(pathfd, entry->d_name,
> > > > > +			    O_NOFOLLOW | O_RDONLY | O_NOATIME);
> > > >
> > > > Just out of curiosity, does O_NOATIME not work in the previous openat?
> >
> > [narrator: it doesn't]
> >
> > > Actually on my test setup (mainly using docker/podman to test), opening
> > > with and without O_NOATIME when using O_PATH, does not change accesstime
> > > checking with `stat`, but also it works if I add it.
> > > As a precautionary measure (not sure if podman/docker is messing with
> > > noatime) I'll add it, as it seems to work correctly.
> >
> > On second thought I think you might leave the double opens because
> > O_NOATIME is only allowed if the current user owns source file, or has
> > CAP_FOWNER.  If you're running mkfs as an unprivileged user trying to
> > capture a rootfs (with uid 0 files) then O_NOATIME won't be allowed.
> >
> > Maybe something along the lines of:
> >
> > 	/*
> > 	 * Try to open the source file noatime to avoid a flood of
> > 	 * writes to the source fs, but we can fall back to plain
> > 	 * readonly mode if we don't have enough permission.
> > 	 */
> > 	fd = openat(pathfd, entry->d_name, O_NOFOLLOW | O_RDONLY | O_NOATIME);
> > 	if (fd < 0)
> > 		fd = openat(pathfd, entry->d_name, O_NOFOLLOW | O_RDONLY);
> > 	if (fd < 0)
> > 		/* whine and exit */
> >
> > Just to see if you can open the source file without touching atime?
> >
> > --D
> 
> Right, will do like this, didn't think about rootless/unprivileged mkfs
> 
> L.
> 

Re: [PATCH v11 1/1] proto: add ability to populate a filesystem from a directory

[Luca Di Maio]

On Wed, Jul 30, 2025 at 08:26:52AM -0700, Darrick J. Wong wrote:
> On Wed, Jul 30, 2025 at 05:20:42PM +0200, Luca Di Maio wrote:
> > On Wed, Jul 30, 2025 at 08:04:09AM -0700, Darrick J. Wong wrote:
> > > On Wed, Jul 30, 2025 at 04:40:39PM +0200, Luca Di Maio wrote:
> > > > Thanks Darrick for the review!
> > > > Sorry again for this indentation mess, I'm going to basically align all
> > > > function arguments and all the top-function declarations
> > > > I was a bit confused because elsewhere in the code is not like that so
> > > > it's a bit difficult to infere
> > > >
> > > > Offtopic: maybe we could also introduce an editorconfig setup? so that
> > > > all various editors will be correctly set to see the tabs/spaces as
> > > > needed (https://editorconfig.org/)
> > >
> > > Hrmm, that /would/ be useful.
> > >
> >
> > Nice, will try to put something together for a future patch
>
> Either that or a indent/clang-format wrapper?

Yes that is what I was thinking, there should be the option to create
custom formatting rules, and stick with those, ideally with a pre-commit
hook

> The kernel has scripts/Lindent, maybe someone should try to write one
> for xfs style if I don't get to it first.

I'm unfamiliar with those, I'll try to check them in the next
days/weeks

L.

Re: [PATCH v11 1/1] proto: add ability to populate a filesystem from a directory

[Andrey Albershteyn]

On 2025-07-30 08:26:52, Darrick J. Wong wrote:
> On Wed, Jul 30, 2025 at 05:20:42PM +0200, Luca Di Maio wrote:
> > On Wed, Jul 30, 2025 at 08:04:09AM -0700, Darrick J. Wong wrote:
> > > On Wed, Jul 30, 2025 at 04:40:39PM +0200, Luca Di Maio wrote:
> > > > Thanks Darrick for the review!
> > > > Sorry again for this indentation mess, I'm going to basically align all
> > > > function arguments and all the top-function declarations
> > > > I was a bit confused because elsewhere in the code is not like that so
> > > > it's a bit difficult to infere
> > > >
> > > > Offtopic: maybe we could also introduce an editorconfig setup? so that
> > > > all various editors will be correctly set to see the tabs/spaces as
> > > > needed (https://editorconfig.org/)
> > >
> > > Hrmm, that /would/ be useful.
> > >
> > 
> > Nice, will try to put something together for a future patch
> 
> Either that or a indent/clang-format wrapper?

editorconfig and clang-format would be great, as far as I know
clang-format is pretty flexible in its style configuration

> 
> The kernel has scripts/Lindent, maybe someone should try to write one
> for xfs style if I don't get to it first.
> 
> <sigh>rustfmt<duck>

:(

-- 
- Andrey