Btrfs lockups on ubuntu with bees

["Libor Klepáč" <libor.klepac@bcom.cz>]

Hi,
we are trying to use btrfs with compression and deduplication using
bees to host filesystem for nakivo repository.
Nakivo repository is in "incremental with full backups" format - ie.
one file per VM snapshot transferred from vmware, full backup every x
days, no internal deduplication. 
We have also disabled internal compression in nakivo repository and put
compression-force=zstd:13 on filesystem.

It's a VM on vmware 6.7.0 Update 3 (Build 17700523) on Dell R540.
It has 6vCPU, 16GB of ram.

Bees is run with this parameters
OPTIONS="--strip-paths --no-timestamps --verbose 5 --loadavg-target 5 
--thread-min 1"
DB_SIZE=$((8*1024*1024*1024)) # 8G in bytes



Until today it was running ubuntu provided kernel 5.11.0-40.44~20.04.2
(not sure about exact upstream version),
today we switched to 5.13.0-21.21~20.04.1 after first crash.

It was working ok for 7+days, all data were in (around 10TB), so i
started bees. 
It now locks the FS, bees runs on 100% CPU, i cannot enter directory
with btrfs

# btrfs filesystem usage /mnt/btrfs/repo02/
Overall:
    Device size:                  20.00TiB
    Device allocated:             10.88TiB
    Device unallocated:            9.12TiB
    Device missing:                  0.00B
    Used:                         10.87TiB
    Free (estimated):              9.13TiB      (min: 4.57TiB)
    Data ratio:                       1.00
    Metadata ratio:                   1.00
    Global reserve:              512.00MiB      (used: 0.00B)

Data,single: Size:10.85TiB, Used:10.83TiB (99.91%)
   /dev/sdd       10.85TiB

Metadata,single: Size:35.00GiB, Used:34.71GiB (99.17%)
   /dev/sdd       35.00GiB

System,DUP: Size:32.00MiB, Used:1.14MiB (3.56%)
   /dev/sdd       64.00MiB

Unallocated:
   /dev/sdd        9.12TiB

This happened yesterday on kernel 5.11
https://download.bcom.cz/btrfs/trace1.txt

This is today, on 5.13
https://download.bcom.cz/btrfs/trace2.txt

this is trace from sysrq later, when i noticed it happened again
https://download.bcom.cz/btrfs/trace3.txt


Any clue what can be done?
We would really like to use btrfs for this use case, because nakivo,
with this type of repository format, needs to be se to do full backup
every x days and does not do deduplication on its own.


With regards,
Libor