public inbox for linux-btrfs@vger.kernel.org
 help / color / mirror / Atom feed
From: fdmanana@kernel.org
To: linux-btrfs@vger.kernel.org
Subject: [PATCH 18/18] btrfs: send: bump the extent reference count limit for backref walking
Date: Tue,  1 Nov 2022 16:15:54 +0000	[thread overview]
Message-ID: <da7dff094eead7ccc0cb1cd186c767d48c8ae05d.1667315100.git.fdmanana@suse.com> (raw)
In-Reply-To: <cover.1667315100.git.fdmanana@suse.com>

From: Filipe Manana <fdmanana@suse.com>

After the previous patchset which is comprised of the following patches:

  01/17 btrfs: fix inode list leak during backref walking at resolve_indirect_refs()
  02/17 btrfs: fix inode list leak during backref walking at find_parent_nodes()
  03/17 btrfs: fix ulist leaks in error paths of qgroup self tests
  04/17 btrfs: remove pointless and double ulist frees in error paths of qgroup tests
  05/17 btrfs: send: avoid unnecessary path allocations when finding extent clone
  06/17 btrfs: send: update comment at find_extent_clone()
  07/17 btrfs: send: drop unnecessary backref context field initializations
  08/17 btrfs: send: avoid unnecessary backref lookups when finding clone source
  09/17 btrfs: send: optimize clone detection to increase extent sharing
  10/17 btrfs: use a single argument for extent offset in backref walking functions
  11/17 btrfs: use a structure to pass arguments to backref walking functions
  12/17 btrfs: reuse roots ulist on each leaf iteration for iterate_extent_inodes()
  13/17 btrfs: constify ulist parameter of ulist_next()
  14/17 btrfs: send: cache leaf to roots mapping during backref walking
  15/17 btrfs: send: skip unnecessary backref iterations
  16/17 btrfs: send: avoid double extent tree search when finding clone source
  17/17 btrfs: send: skip resolution of our own backref when finding clone source

we have now much better performance when doing backref walking in the send
code, so we can increase the current limit from 64 to 1024 references.
This limit is still a bit conservative because there are still edge cases
where backref walking will be too slow and spend a lot of cpu time, some IO
reading b+tree nodes/leaves and memory. The goal is to eventually get rid
of any limit, but for now bump it as it benefits users with extents shared
more than 64 times and up to 1024 times, allowing for more deduplication
at the destination without having to run a dedupe tool after a receive.

Signed-off-by: Filipe Manana <fdmanana@suse.com>
---
 fs/btrfs/send.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c
index 1bcbe386a24b..6950d3f9cbc1 100644
--- a/fs/btrfs/send.c
+++ b/fs/btrfs/send.c
@@ -39,7 +39,7 @@
  * avoid hitting limitations of the backreference walking code (taking a lot of
  * time and using too much memory for extents with large number of references).
  */
-#define SEND_MAX_EXTENT_REFS	64
+#define SEND_MAX_EXTENT_REFS	1024
 
 /*
  * A fs_path is a helper to dynamically build path names with unknown size.
-- 
2.35.1


  parent reply	other threads:[~2022-11-01 16:16 UTC|newest]

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-11-01 16:15 [PATCH 00/18] btrfs: make send scale and perform better with shared extents fdmanana
2022-11-01 16:15 ` [PATCH 01/18] btrfs: fix inode list leak during backref walking at resolve_indirect_refs() fdmanana
2022-11-01 16:15 ` [PATCH 02/18] btrfs: fix inode list leak during backref walking at find_parent_nodes() fdmanana
2022-11-01 16:15 ` [PATCH 03/18] btrfs: fix ulist leaks in error paths of qgroup self tests fdmanana
2022-11-01 16:15 ` [PATCH 04/18] btrfs: remove pointless and double ulist frees in error paths of qgroup tests fdmanana
2022-11-01 16:15 ` [PATCH 05/18] btrfs: send: avoid unnecessary path allocations when finding extent clone fdmanana
2022-11-01 16:15 ` [PATCH 06/18] btrfs: send: update comment at find_extent_clone() fdmanana
2022-11-01 16:15 ` [PATCH 07/18] btrfs: send: drop unnecessary backref context field initializations fdmanana
2022-11-01 16:15 ` [PATCH 08/18] btrfs: send: avoid unnecessary backref lookups when finding clone source fdmanana
2022-11-01 16:15 ` [PATCH 09/18] btrfs: send: optimize clone detection to increase extent sharing fdmanana
2022-11-01 16:15 ` [PATCH 10/18] btrfs: use a single argument for extent offset in backref walking functions fdmanana
2022-11-01 16:15 ` [PATCH 11/18] btrfs: use a structure to pass arguments to " fdmanana
2022-11-01 16:15 ` [PATCH 12/18] btrfs: reuse roots ulist on each leaf iteration for iterate_extent_inodes() fdmanana
2022-11-01 16:15 ` [PATCH 13/18] btrfs: constify ulist parameter of ulist_next() fdmanana
2022-11-01 16:15 ` [PATCH 14/18] btrfs: send: cache leaf to roots mapping during backref walking fdmanana
2022-11-01 16:15 ` [PATCH 15/18] btrfs: send: skip unnecessary backref iterations fdmanana
2022-11-01 16:15 ` [PATCH 16/18] btrfs: send: avoid double extent tree search when finding clone source fdmanana
2022-11-01 16:15 ` [PATCH 17/18] btrfs: send: skip resolution of our own backref " fdmanana
2022-11-01 16:15 ` fdmanana [this message]
2022-11-02 16:01 ` [PATCH 00/18] btrfs: make send scale and perform better with shared extents David Sterba

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=da7dff094eead7ccc0cb1cd186c767d48c8ae05d.1667315100.git.fdmanana@suse.com \
    --to=fdmanana@kernel.org \
    --cc=linux-btrfs@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox