Enabling compression = lzo + defrag + balance grinded system to a halt :)

Enabling compression = lzo + defrag + balance grinded system to a halt :)

[Tomasz Kusmierz]

Folks ... it's me again :)

Just a preliminary word of warning - I did backup a data before doing
that, so I can rebuild array back ... just want to present this
problem to you because it's a interesting problem / use case issue /
bug. So please no bitching because I don't bitch either !

So, I got my self a spare server at work (some xeon with ECC ram) and
installed a rockstor on it. it's more or less a vanila centos with
tinny bit of python on top of it to do a GUI. Fair play to them, a
nice automation of terminal tasks ...

Anyway since I've installed a owncloud in docker there, that me and my
colleagues use as a form of backup / exchange of test data between us
(it's a text logs from vehicle CAN bus ... 50mb file usually with zip
compresses down to 2MB) I decided to go with LZO compression.

They have a lot of quirky ways of setting compression per subvolume (I
think it's per directory trick way) but I decided to go the old
fashion mount option.

Did set it, remounted & rebooted for sanity sake and did run:
btrfs defragment -r -v -clzo /mnt2/main_pool/

after a short time (20minutes) it did how ever NOT do anything with
disks - no IO, not physical operation of disks ... disk were just
spinning there and not doing anything.

So I decided to it even more old fashion
btrfs fi balance start /mnt2/main_pool/

Now this did make system virtually not usable ... CPU was stuck at
100%, all docker apps were not accessible BUT balance was chugging
along. I've got roughly 245 GB of occupied space in RAID1 on two 2TB
drives that I was using for this fun project and balance got up to 104
chunks out of 202 considered.

SInce I needed to access docker I performed a
btrfs fi balance cancel/mnt2/main_pool/
and let it finish gracefully. Did reboot afterwards to try to get to
the server and physically do something about it.

BUT, after a reboot system was still performing badly, high CPU
utilisation ... no disk activity for some reason :/ ... but I did
check and after a reboot a balance reappeared - I know that btrfs will
resume balance after reboot but I'm 105% sure that balance did cancel
before I rebooted.
Now, this balance is seems to be stuck for past 12 hours in:
2 out of about 4 chunks balanced (204 considered), 50% left

And what funnier, I did attempt to cancel it 2 hours ago and cancel
command is stuck in one terminal and in another terminal I've got
this:

[root@tevva-server ~]# btrfs balance status /mnt2/main_pool/
Balance on '/mnt2/main_pool/' is running, cancel requested
2 out of about 4 chunks balanced (204 considered),  50% left

So gents:
- before I drop a nuke on this FS and start over, anybody want to use
this as a guinea pig ?
- any way of telling what's going on ?
- any way of telling whenever defragment is working or not ? original
defragment command exited straight away :/



ps. Some data that I know people will ask for:


[root@tevva-server ~]# top
top - 13:56:55 up  1:23,  2 users,  load average: 10.62, 8.58, 8.28
Tasks: 432 total,   4 running, 428 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.1 us, 25.1 sy,  0.0 ni, 74.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 16372092 total, 10027296 free,  3868152 used,  2476644 buff/cache
KiB Swap: 15624188 total, 15624188 free,        0 used. 11973620 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 2476 root      20   0       0      0      0 R 100.0  0.0   4:46.46
kworker/u24:5
 5015 root      20   0       0      0      0 R 100.0  0.0  72:14.08
btrfs-transacti
29799 root      20   0       0      0      0 R 100.0  0.0  23:57.08
kworker/u24:2
    7 root      20   0       0      0      0 S   0.3  0.0   0:17.89 rcu_sched
    9 root      20   0       0      0      0 S   0.3  0.0   0:01.62 rcuos/0
 3187 systemd+  20   0 2440028 346268  14200 S   0.3  2.1   0:27.73 bundle
 4928 root      20   0  562168  49756  11296 S   0.3  0.3   0:17.58
data-collector
19951 root      20   0       0      0      0 S   0.3  0.0   0:00.27 kworker/0:0
20931 systemd+  20   0   18176   3092   2704 S   0.3  0.0   0:00.73
gitlab-unicorn-
25531 root      20   0  157988   4764   3732 R   0.3  0.0   0:00.04 top



[root@tevva-server ~]# btrfs fi df /mnt2/main_pool/
Data, RAID1: total=245.00GiB, used=244.16GiB
System, RAID1: total=32.00MiB, used=64.00KiB
Metadata, RAID1: total=2.00GiB, used=635.42MiB
GlobalReserve, single: total=224.00MiB, used=7.83MiB

[root@tevva-server ~]# btrfs fi show
Label: 'rockstor_tevva-server'  uuid: 1348a9ac-a247-432a-8307-84b5d03c9e62
        Total devices 1 FS bytes used 1.77GiB
        devid    1 size 96.40GiB used 7.06GiB path /dev/sdb3

Label: 'backup_pool'  uuid: c766d968-470c-451c-ab53-59b647c6eb43
        Total devices 3 FS bytes used 61.97GiB
        devid    1 size 1.82TiB used 42.00GiB path /dev/sdf
        devid    2 size 1.82TiB used 42.01GiB path /dev/sde
        devid    3 size 1.82TiB used 42.01GiB path /dev/sdc

Label: 'main_pool'  uuid: 98eff16e-10b2-4e84-a301-3d724b37b6fc
        Total devices 2 FS bytes used 244.79GiB
        devid    1 size 1.82TiB used 247.03GiB path /dev/sdd
        devid    2 size 1.82TiB used 247.03GiB path /dev/sda

[root@tevva-server ~]# uname -a
Linux tevva-server 4.6.0-1.el7.elrepo.x86_64 #1 SMP Mon May 16
10:54:52 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux

[root@tevva-server ~]# btrfs --version
btrfs-progs v4.6