[PATCH v3 0/3] Introduce per-profile available space array to avoid over-confident can_overcommit()

Qu Wenruo
2020-01-06T06:13:50Z

There are several bug reports of ENOSPC error in
btrfs_run_delalloc_range().

With some extra info from one reporter, it turns out that
can_overcommit() is using a wrong way to calculate allocatable metadata
space.

The most typical case would look like:
devid 1 unallocated: 1G
devid 2 unallocated: 10G
metadata profile: RAID1

In above case, we can at most allocate 1G chunk for metadata, due to
unbalanced disk free space.
But current can_overcommit() uses factor based calculation, which never
consider the disk free space balance.

To address this problem, here comes the per-profile available space
array, which gets updated every time a chunk get allocated/removed or a
device get grown or shrunk.

This provides a quick way for hotter place like can_overcommit() to grab
an estimation on how many bytes it can over-commit.

The per-profile available space calculation tries to keep the behavior
of chunk allocator, thus it can handle uneven disks pretty well.

Although per-profile is not clever enough to handle estimation when both
data and metadata chunks need to be considered, its virtual chunk
infrastructure is flex enough to handle such case.

So for statfs(), we also re-use virtual chunk allocator to handle
available data space, with metadata over-commit space considered.
This brings an unexpected advantage, now we can handle RAID5/6 pretty OK
in statfs().

The execution time of this per-profile calculation is a little below
20 us per 5 iterations in my test VM.
Although all such calculation will need to acquire chunk mutex, the
impact should be small enough.
For the full statfs execution time anaylse, please see the commit
message of the last patch.

Changelog:
v1:
- Fix a bug where we forgot to update per-profile array after allocating
a chunk.
To avoid ABBA deadlock, this introduce a small windows at the end
__btrfs_alloc_chunk(), it's not elegant but should be good enough
before we rework chunk and device list mutex.

- Make statfs() to use virtual chunk allocator to do better estimation
Now statfs() can report not only more accurate result, but can also
handle RAID5/6 better.

v2:
- Fix a deadlock caused by acquiring device_list_mutex under
__btrfs_alloc_chunk()
There is no need to acquire device_list_mutex when holding
chunk_mutex.
Fix it and remove the lockdep assert.

v3:
- Use proper chunk_mutex instead of device_list_mutex
Since they are protecting two different things, and we only care about
alloc_list, we should only use chunk_mutex.
With improved lock situation, it's easier to fold
calc_per_profile_available() calls into the first patch.

- Add performance benchmark for statfs() modification
As Facebook seems to run into some problems with statfs() calls, add
some basic ftrace results.

Qu Wenruo (3):
btrfs: Introduce per-profile available space facility
btrfs: space-info: Use per-profile available space in can_overcommit()
btrfs: statfs: Use virtual chunk allocation to calculation available
data space

fs/btrfs/space-info.c | 15 ++-
fs/btrfs/super.c | 190 +++++++++++++-----------------------
fs/btrfs/volumes.c | 218 ++++++++++++++++++++++++++++++++++++++----
fs/btrfs/volumes.h | 15 +++
4 files changed, 289 insertions(+), 149 deletions(-)

--
2.24.1