[PATCH v3] btrfs: handle lack of space when cleaning up verity items

[Daniel Linjama <daniel@dev.linjama.com>]

When enable_verity() hits the qgroup limit, rollback_verity() needs its
own metadata reservation. When the qgroup limit or lack of space refuses
the rollback, the whole filesystem is forced read-only even though the
qgroup limit was for one subvolume only. Also orphan cleanup at the next
mount fails the same way, so the leftover items are never removed: with
-EDQUOT the subvolume stays unreachable, and with -ENOSPC on a full
filesystem the next read-write mount fails.

Start transactions with btrfs_start_transaction_fallback_global_rsv() in
btrfs_orphan_cleanup(), drop_verity_items() and rollback_verity(). Those
calls only delete items and free the space in the end, so they may use
the global reserve and skip the qgroup limit, which avoids -ENOSPC and
-EDQUOT.

Fixes: 146054090b08 ("btrfs: initial fsverity support")
Signed-off-by: Daniel Linjama <daniel@dev.linjama.com>
---
Changes since v2:
- Explain in the commit message why the global reserve fallback and
  skipping the qgroup limit are right here, as Qu Wenruo asked.
- A helper in verity.c for the two verity call sites carries the one
  comment, as Qu Wenruo suggested; the orphan cleanup call site only
  notes what it does.

v2: https://lore.kernel.org/linux-btrfs/20260915055109.311839-1-daniel@dev.linjama.com/
v1: https://lore.kernel.org/linux-btrfs/20260914062459.3889313-1-daniel@dev.linjama.com/

 fs/btrfs/inode.c  |  3 ++-
 fs/btrfs/verity.c | 18 ++++++++++++++++--
 2 files changed, 18 insertions(+), 3 deletions(-)

diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index 93ef3cec191e..84dc44af17d7 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -3877,7 +3877,8 @@ int btrfs_orphan_cleanup(struct btrfs_root *root)
 				if (ret)
 					goto out;
 			}
-			trans = btrfs_start_transaction(root, 1);
+			/* Only deletes the orphan */
+			trans = btrfs_start_transaction_fallback_global_rsv(root, 1);
 			if (IS_ERR(trans)) {
 				ret = PTR_ERR(trans);
 				goto out;
diff --git a/fs/btrfs/verity.c b/fs/btrfs/verity.c
index 4e0ab5842274..18b9d14f2e8c 100644
--- a/fs/btrfs/verity.c
+++ b/fs/btrfs/verity.c
@@ -93,6 +93,20 @@ static loff_t merkle_file_pos(const struct inode *inode)
 	return rounded;
 }
 
+/*
+ * Start a transaction for removing verity items or the verity orphan.
+ *
+ * Like unlink, this only deletes items and frees space in the end, so the
+ * reservation may come from the global reserve when the filesystem is full
+ * (-ENOSPC) and is not subject to the qgroup limit (-EDQUOT). Otherwise a
+ * failed enable could never be cleaned up in either situation.
+ */
+static struct btrfs_trans_handle *start_verity_cleanup_trans(struct btrfs_root *root,
+							      unsigned int num_items)
+{
+	return btrfs_start_transaction_fallback_global_rsv(root, num_items);
+}
+
 /*
  * Drop all the items for this inode with this key_type.
  *
@@ -120,7 +134,7 @@ static int drop_verity_items(struct btrfs_inode *inode, u8 key_type)
 
 	while (1) {
 		/* 1 for the item being dropped */
-		trans = btrfs_start_transaction(root, 1);
+		trans = start_verity_cleanup_trans(root, 1);
 		if (IS_ERR(trans))
 			return PTR_ERR(trans);
 
@@ -466,7 +480,7 @@ static int rollback_verity(struct btrfs_inode *inode)
 	 * 1 for updating the inode flag
 	 * 1 for deleting the orphan
 	 */
-	trans = btrfs_start_transaction(root, 2);
+	trans = start_verity_cleanup_trans(root, 2);
 	if (IS_ERR(trans)) {
 		ret = PTR_ERR(trans);
 		trans = NULL;

base-commit: 08df884136f1c1197bab2a27814404fd329d9aac
-- 
2.55.0