From: "Konstantin V. Gavrilenko" <k.gavrilenko@arhont.com>
To: linux-btrfs@vger.kernel.org
Subject: Btrfs + compression = slow performance and high cpu usage
Date: Fri, 28 Jul 2017 17:40:50 +0100 (BST) [thread overview]
Message-ID: <8446582.541.1501260065690.JavaMail.gkos@dynomob> (raw)
In-Reply-To: <33040946.535.1501254718807.JavaMail.gkos@dynomob>
Hello list,
I am stuck with a problem of btrfs slow performance when using compression.
when the compress-force=lzo mount flag is enabled, the performance drops to 30-40 mb/s and one of the btrfs processes utilises 100% cpu time.
mount options: btrfs relatime,discard,autodefrag,compress=lzo,compress-force,space_cache=v2,commit=10
The command I am testing the write throughput is
# pv -tpreb /dev/sdb | dd of=./testfile bs=1M oflag=direct
# top -d 1
top - 15:49:13 up 1:52, 2 users, load average: 5.28, 2.32, 1.39
Tasks: 320 total, 2 running, 318 sleeping, 0 stopped, 0 zombie
%Cpu0 : 0.0 us, 2.0 sy, 0.0 ni, 77.0 id, 21.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 0.0 us, 1.0 sy, 0.0 ni, 90.0 id, 9.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 : 0.0 us, 1.0 sy, 0.0 ni, 72.0 id, 27.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 0.0 us,100.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu4 : 0.0 us, 1.0 sy, 0.0 ni, 57.0 id, 42.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu5 : 0.0 us, 0.0 sy, 0.0 ni, 96.0 id, 4.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu6 : 0.0 us, 0.0 sy, 0.0 ni, 94.0 id, 6.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu7 : 0.0 us, 1.0 sy, 0.0 ni, 95.1 id, 3.9 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu8 : 1.0 us, 2.0 sy, 0.0 ni, 24.0 id, 73.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu9 : 0.0 us, 0.0 sy, 0.0 ni, 81.8 id, 18.2 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu10 : 1.0 us, 0.0 sy, 0.0 ni, 98.0 id, 1.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu11 : 0.0 us, 2.0 sy, 0.0 ni, 83.3 id, 14.7 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 32934136 total, 10137496 free, 602244 used, 22194396 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 30525664 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
37017 root 20 0 0 0 0 R 100.0 0.0 0:32.42 kworker/u49:8
36732 root 20 0 0 0 0 D 4.0 0.0 0:02.40 btrfs-transacti
40105 root 20 0 8388 3040 2000 D 4.0 0.0 0:02.88 dd
The keyworker process that causes the high cpu usage is most likely searching for the free space.
# echo l > /proc/sysrq-trigger
# dmest -T
Fri Jul 28 15:57:51 2017] CPU: 1 PID: 36430 Comm: kworker/u49:2 Not tainted 4.10.0-28-generic #32~16.04.2-Ubuntu
[Fri Jul 28 15:57:51 2017] Hardware name: Supermicro X8DTL/X8DTL, BIOS 2.1b 11/16/2012
[Fri Jul 28 15:57:51 2017] Workqueue: btrfs-delalloc btrfs_delalloc_helper [btrfs]
[Fri Jul 28 15:57:51 2017] task: ffff9ddce6206a40 task.stack: ffffaa9121f6c000
[Fri Jul 28 15:57:51 2017] RIP: 0010:rb_next+0x1e/0x40
[Fri Jul 28 15:57:51 2017] RSP: 0018:ffffaa9121f6fb40 EFLAGS: 00000282
[Fri Jul 28 15:57:51 2017] RAX: ffff9dddc34df1b0 RBX: 0000000000010000 RCX: 0000000000001000
[Fri Jul 28 15:57:51 2017] RDX: ffff9dddc34df708 RSI: ffff9ddccaf470a4 RDI: ffff9dddc34df2d0
[Fri Jul 28 15:57:51 2017] RBP: ffffaa9121f6fb40 R08: 0000000000000001 R09: 0000000000003000
[Fri Jul 28 15:57:51 2017] R10: 0000000000000000 R11: 0000000000020000 R12: ffff9ddccaf47080
[Fri Jul 28 15:57:51 2017] R13: 0000000000001000 R14: ffffaa9121f6fc50 R15: ffff9dddc34df2d0
[Fri Jul 28 15:57:51 2017] FS: 0000000000000000(0000) GS:ffff9ddcefa40000(0000) knlGS:0000000000000000
[Fri Jul 28 15:57:51 2017] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Fri Jul 28 15:57:51 2017] Call Trace:_space_for_alloc+0xde/0x270 [btrfs]
[Fri Jul 28 15:57:51 2017] btrfs_find_space_for_alloc+0xde/0x270 [btrfs]
[Fri Jul 28 15:57:51 2017] find_free_extent.isra.68+0x3c6/0x1040 [btrfs]s]
[Fri Jul 28 15:57:51 2017] btrfs_reserve_extent+0xab/0x210 [btrfs]btrfs]
[Fri Jul 28 15:57:51 2017] submit_compressed_extents+0x154/0x580 [btrfs]s]
[Fri Jul 28 15:57:51 2017] ? submit_compressed_extents+0x580/0x580 [btrfs]
[Fri Jul 28 15:57:51 2017] async_cow_submit+0x82/0x90 [btrfs]00 [btrfs]
[Fri Jul 28 15:57:51 2017] btrfs_scrubparity_helper+0x1fe/0x300 [btrfs]
[Fri Jul 28 15:57:51 2017] btrfs_delalloc_helper+0xe/0x10 [btrfs]
[Fri Jul 28 15:57:51 2017] process_one_work+0x16b/0x4a0a0
[Fri Jul 28 15:57:51 2017] worker_thread+0x4b/0x500+0x60/0x60
[Fri Jul 28 15:57:51 2017] kthread+0x109/0x1400x4a0/0x4a0
When the compression is turned off, I am able to get the maximum 500-600 mb/s write speed on this disk (raid array) with minimal cpu usage.
mount options: relatime,discard,autodefrag,space_cache=v2,commit=10
# iostat -m 1
avg-cpu: %user %nice %system %iowait %steal %idle
0.08 0.00 7.74 10.77 0.00 81.40
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 2376.00 0.00 594.01 0 594
I have tried deleting mounting with nospace_cache, clear_cache, and then rebuilding with space_cache=v2
but it doesn't make any difference. The same sluggish performance is experienced when I try to write over the NFS.
Any ideas why the compression makes such a big difference and causes a bottleneck?
# uname -a
Linux backup1 4.10.0-28-generic #32~16.04.2-Ubuntu SMP Thu Jul 20 10:19:48 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
# btrfs --version
btrfs-progs v4.8.1
# cat /etc/issue
Ubuntu 16.04.2 LTS \n \l
# btrfs fi show
Label: none uuid: f56bdc4a-239d-4268-81d8-01cdd7a3c1c9
Total devices 1 FS bytes used 9.32TiB
devid 2 size 21.83TiB used 9.33TiB path /dev/sda
# btrfs fi df /mnt/arh-backup1/
Data, single: total=9.28TiB, used=9.28TiB
System, single: total=32.00MiB, used=1.00MiB
Metadata, single: total=46.00GiB, used=44.20GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
# btrfs device usage /mnt/arh-backup1/
/dev/sda, ID: 2
Device size: 21.83TiB
Device slack: 0.00B
Data,single: 9.29TiB
Metadata,single: 46.00GiB
System,single: 32.00MiB
Unallocated: 12.49TiB
thanks in advance.
kos
next parent reply other threads:[~2017-07-28 16:47 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <33040946.535.1501254718807.JavaMail.gkos@dynomob>
2017-07-28 16:40 ` Konstantin V. Gavrilenko [this message]
2017-07-28 17:48 ` Btrfs + compression = slow performance and high cpu usage Roman Mamedov
2017-07-28 18:20 ` William Muriithi
2017-07-28 18:37 ` Hugo Mills
2017-07-28 18:08 ` Peter Grandi
2017-07-30 13:42 ` Konstantin V. Gavrilenko
2017-07-31 11:41 ` Peter Grandi
2017-07-31 12:33 ` Peter Grandi
2017-07-31 12:49 ` Peter Grandi
2017-08-01 9:58 ` Konstantin V. Gavrilenko
2017-08-01 10:53 ` Paul Jones
2017-08-01 13:14 ` Peter Grandi
2017-08-01 18:09 ` Konstantin V. Gavrilenko
2017-08-01 20:09 ` Peter Grandi
2017-08-01 23:54 ` Peter Grandi
2017-08-31 10:56 ` Konstantin V. Gavrilenko
2017-07-28 18:44 ` Peter Grandi
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=8446582.541.1501260065690.JavaMail.gkos@dynomob \
--to=k.gavrilenko@arhont.com \
--cc=linux-btrfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.