Abysmal Performance

[Henning Rohlfs <x+btrfs@xehonk.de>]

[-- Attachment #1: Type: text/plain, Size: 3052 bytes --]

 Hello,

 I've migrated my system to btrfs (raid1) a few months ago. Since then 
 the performance has been pretty bad, but recently it's gotten 
 unbearable: a simple sync called while the system is idle can take 20 up 
 to 60 seconds. Creating or deleting files often has several seconds 
 latency, too.

 One curious - but maybe unrelated - observation is that even though I'm 
 using a raid1 btrfs setup, the hdds are often being written to 
 sequentially. One hard-drive sees some write activity and after it 
 subsides, the other drive sees some activity. (See attached 
 sequential-writes.txt.)

 - 64bit gentoo with vanilla 2.6.39 kernel
 - lzo compression enabled
 - 2x WD1000FYPS (1TB WD hdds)
 - Athlon x2 2.2GHz with 8GB RAM
 - space_cache was enabled, but it seemed to make the problem worse. 
 It's no longer in the mount options.

 Any help is appreciated. Thanks,
 Henning




 server ~ # sync; time sync
 real	0m28.869s
 user	0m0.000s
 sys	0m5.750s



 server ~ # uname -a
 Linux server 2.6.39 #3 SMP Sat May 28 17:25:31 CEST 2011 x86_64 AMD 
 Athlon(tm) 64 X2 Dual Core Processor 4200+ AuthenticAMD GNU/Linux



 server ~ # mount | grep btrfs
 /dev/sdb2 on / type btrfs (rw,noatime,compress=lzo,noacl)
 /dev/sda2 on /mnt/pool type btrfs (rw,noatime,subvolid=0,compress=lzo)
 /dev/sda2 on /usr/portage type btrfs 
 (rw,noatime,subvol=newportage,compress=lzo)
 /dev/sda2 on /home type btrfs (rw,noatime,subvol=home,compress=lzo)
 /dev/sda2 on /home/mythtv type btrfs 
 (rw,noatime,subvol=mythtv,compress=lzo)



 server ~ # btrfs fi show
 Label: none  uuid: 7676eb78-e411-4505-ac51-ccd12aa5a6b6
 	Total devices 2 FS bytes used 281.58GB
 	devid    1 size 931.28GB used 898.26GB path /dev/sda2
 	devid    3 size 931.27GB used 898.26GB path /dev/sdb2

 Btrfs v0.19-35-g1b444cd-dirty



 server ~ # btrfs fi df /
 Data, RAID1: total=875.00GB, used=279.30GB
 System, RAID1: total=8.00MB, used=140.00KB
 System: total=4.00MB, used=0.00
 Metadata, RAID1: total=23.25GB, used=2.28GB



 bonnie++

 Version  1.96       ------Sequential Output------ --Sequential Input- 
 --Random-
 Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
 --Seeks--
 Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  
 /sec %CP
 server          16G   147  90 76321  18 31787  16  1370  71 64812  14  
 27.0  66
 Latency             66485us    7581ms    4455ms   25011us     695ms     
 959ms
 Version  1.96       ------Sequential Create------ --------Random 
 Create--------
 server              -Create-- --Read--- -Delete-- -Create-- --Read--- 
 -Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
 /sec %CP
                  16   238  51 +++++ +++   219  51   284  52 +++++ +++   
 390  57
 Latency              1914ms     524us    3461ms    1141ms      39us    
 1308ms
 1.96,1.96,server,1,1308618030,16G,,147,90,76321,18,31787,16,1370,71,64812,14,27.0,66,16,,,,,238,51,+++++,+++,219,51,284,52,+++++,+++,390,57,66485us,7581ms,4455ms,25011us,695ms,959ms,1914ms,524us,3461ms,1141ms,39us,1308ms

[-- Attachment #2: sequential-writes.txt --]
[-- Type: text/plain, Size: 2015 bytes --]

server ~ # iostat -m 5

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.00    0.00   45.20    5.20    0.00   46.60

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda               0.00         0.00         0.00          0          0
sdb              15.20         0.06         0.00          0          0
md0               0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.30    0.00   37.46   42.36    0.00   15.88

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda              45.00         0.00         0.38          0          1
sdb             467.60         0.02         2.06          0         10
md0               0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.31    0.00   19.34   58.82    0.00   17.54

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda               8.80         0.00         0.04          0          0
sdb             649.80         0.02         2.67          0         13
md0               0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.20    0.00   63.24   31.97    0.00    1.60

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda             585.80         0.00         2.36          0         11
sdb              20.80         0.08         0.16          0          0
md0               0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.30    0.00   42.60   39.30    0.00   14.80

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda             514.20         0.00         2.29          0         11
sdb              59.20         0.10         0.17          0          0
md0               0.00         0.00         0.00          0          0