btrfs_extent_map memory consumption results in "Out of memory"

["Ospan, Abylay" <aospan@amazon.com>]

Greetings Btrfs development team!

I would like to express my gratitude for your outstanding work on Btrfs. However, I recently experienced an 'out of memory' issue as described below.

Steps to reproduce:

1. Run FIO test on a btrfs partition with random write on a 300GB file:

cat <<EOF >> rand.fio 
[global]
name=fio-rand-write
filename=fio-rand-write
rw=randwrite
bs=4K
direct=1
numjobs=16
time_based
runtime=90000

[file1]
size=300G
ioengine=libaio
iodepth=16
EOF

fio rand.fio

2. Monitor slab consumption with "slabtop -s -a"

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
25820620 23138538  89%    0.14K 922165       28   3688660K btrfs_extent_map

3. Observe oom-killer:
[49689.294138] ip invoked oom-killer: gfp_mask=0xc2cc0(GFP_KERNEL|__GFP_NOWARN|__GFP_COMP|__GFP_NOMEMALLOC), order=3, oom_score_adj=0
...
[49689.294425] Unreclaimable slab info:
[49689.294426] Name                      Used          Total	
[49689.329363] btrfs_extent_map     3207098KB    3375622KB
...

Memory usage by btrfs_extent_map gradually increases until it reaches a critical point, causing the system to run out of memory.

Test environment: Intel CPU, 8GB RAM (To expedite the reproduction of this issue, I also conducted tests within QEMU with a restricted amount of memory). 
Linux kernel tested: LTS 5.15.133, and mainline 6.6-rc5

Quick review of the 'fs/btrfs/extent_map.c' code reveals no built-in limitations on memory allocation for extents mapping.
Are there any known workarounds or alternative solutions to mitigate this issue?

Thank you!

--
Abylay Ospan