Expected Behavior

Expected Behavior

[Jonathan Tripathy]

Hi There,

On my WIndows DomU (Xen VM) which is running on a LV which is using 
bcache (against two SSD in MDRAID1 and a MD-RAID10 spindle array), I ran 
an IOMeter test for about 2 hours (with 30 workers and a io depth of 
256). This was a very heavy workload (Got an average iops of about 
6.5k). After I stopped the test, I then went back to fio on my Linux Xen 
Host (Dom0). The random write performance isn't as good as it was before 
I started the IOMeter test. It used to be about 25k and now showed about 
7k iops. I assumed that maybe this was due to the fact that bcache was 
writing out dirty data to the spindles so the SSD was busy.

However, this morning, after the spindles have calmed down, performance 
of fio is still not great (still about 7k).

Is there something wrong here? What is expected behavior?

Thanks

Re: Expected Behavior

[Jonathan Tripathy]

On 30/08/2012 08:15, Jonathan Tripathy wrote:
> Hi There,
>
> On my WIndows DomU (Xen VM) which is running on a LV which is using 
> bcache (against two SSD in MDRAID1 and a MD-RAID10 spindle array), I 
> ran an IOMeter test for about 2 hours (with 30 workers and a io depth 
> of 256). This was a very heavy workload (Got an average iops of about 
> 6.5k). After I stopped the test, I then went back to fio on my Linux 
> Xen Host (Dom0). The random write performance isn't as good as it was 
> before I started the IOMeter test. It used to be about 25k and now 
> showed about 7k iops. I assumed that maybe this was due to the fact 
> that bcache was writing out dirty data to the spindles so the SSD was 
> busy.
>
> However, this morning, after the spindles have calmed down, 
> performance of fio is still not great (still about 7k).
>
> Is there something wrong here? What is expected behavior?
>
> Thanks
>
BTW, I can confirm that this isn't an SSD issue, as I have a partition 
on the SSD that I kept seperate from bcache and I'm getting excellent 
(about 28k) iops performance there.

It's as if after the heavy workload I did with IOMeter, bcache has 
somehow throttled the writeback cache?

Any help is appreciated.

Re: Expected Behavior

[Jonathan Tripathy]

On 30/08/2012 08:21, Jonathan Tripathy wrote:
> On 30/08/2012 08:15, Jonathan Tripathy wrote:
>> Hi There,
>>
>> On my WIndows DomU (Xen VM) which is running on a LV which is using 
>> bcache (against two SSD in MDRAID1 and a MD-RAID10 spindle array), I 
>> ran an IOMeter test for about 2 hours (with 30 workers and a io depth 
>> of 256). This was a very heavy workload (Got an average iops of about 
>> 6.5k). After I stopped the test, I then went back to fio on my Linux 
>> Xen Host (Dom0). The random write performance isn't as good as it was 
>> before I started the IOMeter test. It used to be about 25k and now 
>> showed about 7k iops. I assumed that maybe this was due to the fact 
>> that bcache was writing out dirty data to the spindles so the SSD was 
>> busy.
>>
>> However, this morning, after the spindles have calmed down, 
>> performance of fio is still not great (still about 7k).
>>
>> Is there something wrong here? What is expected behavior?
>>
>> Thanks
>>
> BTW, I can confirm that this isn't an SSD issue, as I have a partition 
> on the SSD that I kept seperate from bcache and I'm getting excellent 
> (about 28k) iops performance there.
>
> It's as if after the heavy workload I did with IOMeter, bcache has 
> somehow throttled the writeback cache?
>
> Any help is appreciated.
>
>
Also, I'm not sure if this is related, but is there a memory leak 
somewhere in the bcache code? I haven't used this machine for anything 
else apart from running the above tests and here is my RAM usage:

free -m
              total       used       free     shared    buffers cached
Mem:          1155       1021        133          0 0          8
-/+ buffers/cache:       1013        142
Swap:          952         53        899

Any ideas? Please let me know if you need me to run any other commands.

Thanks

Re: Expected Behavior

[Jonathan Tripathy]

On 30/08/2012 08:26, Jonathan Tripathy wrote:
> On 30/08/2012 08:21, Jonathan Tripathy wrote:
>> On 30/08/2012 08:15, Jonathan Tripathy wrote:
>>> Hi There,
>>>
>>> On my WIndows DomU (Xen VM) which is running on a LV which is using 
>>> bcache (against two SSD in MDRAID1 and a MD-RAID10 spindle array), I 
>>> ran an IOMeter test for about 2 hours (with 30 workers and a io 
>>> depth of 256). This was a very heavy workload (Got an average iops 
>>> of about 6.5k). After I stopped the test, I then went back to fio on 
>>> my Linux Xen Host (Dom0). The random write performance isn't as good 
>>> as it was before I started the IOMeter test. It used to be about 25k 
>>> and now showed about 7k iops. I assumed that maybe this was due to 
>>> the fact that bcache was writing out dirty data to the spindles so 
>>> the SSD was busy.
>>>
>>> However, this morning, after the spindles have calmed down, 
>>> performance of fio is still not great (still about 7k).
>>>
>>> Is there something wrong here? What is expected behavior?
>>>
>>> Thanks
>>>
>> BTW, I can confirm that this isn't an SSD issue, as I have a 
>> partition on the SSD that I kept seperate from bcache and I'm getting 
>> excellent (about 28k) iops performance there.
>>
>> It's as if after the heavy workload I did with IOMeter, bcache has 
>> somehow throttled the writeback cache?
>>
>> Any help is appreciated.
>>
>>
> Also, I'm not sure if this is related, but is there a memory leak 
> somewhere in the bcache code? I haven't used this machine for anything 
> else apart from running the above tests and here is my RAM usage:
>
> free -m
>              total       used       free     shared    buffers cached
> Mem:          1155       1021        133          0 0          8
> -/+ buffers/cache:       1013        142
> Swap:          952         53        899
>
> Any ideas? Please let me know if you need me to run any other commands.
>
>
Here are some other outputs (meminfo and vmallocinfo) that you may find 
useful:

# cat /proc/meminfo
MemTotal:        1183420 kB
MemFree:          135760 kB
Buffers:            1020 kB
Cached:             8840 kB
SwapCached:         2824 kB
Active:              628 kB
Inactive:          13332 kB
Active(anon):        392 kB
Inactive(anon):     3664 kB
Active(file):        236 kB
Inactive(file):     9668 kB
Unevictable:          72 kB
Mlocked:              72 kB
SwapTotal:        975856 kB
SwapFree:         917124 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:          2248 kB
Mapped:             1940 kB
Shmem:                 0 kB
Slab:              47316 kB
SReclaimable:      13048 kB
SUnreclaim:        34268 kB
KernelStack:        1296 kB
PageTables:         2852 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     1567564 kB
Committed_AS:     224408 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      134624 kB
VmallocChunk:   34359595328 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:    17320640 kB
DirectMap2M:           0 kB


# cat /proc/vmallocinfo
0xffffc90000000000-0xffffc90002001000 33558528 
alloc_large_system_hash+0x14b/0x215 pages=8192 vmalloc vpages N0=8192
0xffffc90002001000-0xffffc90002012000   69632 
alloc_large_system_hash+0x14b/0x215 pages=16 vmalloc N0=16
0xffffc90002012000-0xffffc90003013000 16781312 
alloc_large_system_hash+0x14b/0x215 pages=4096 vmalloc vpages N0=4096
0xffffc90003013000-0xffffc9000301c000   36864 
alloc_large_system_hash+0x14b/0x215 pages=8 vmalloc N0=8
0xffffc9000301c000-0xffffc9000301f000   12288 
acpi_os_map_memory+0x98/0x119 phys=ddfa9000 ioremap
0xffffc90003020000-0xffffc9000302d000   53248 
acpi_os_map_memory+0x98/0x119 phys=ddf9e000 ioremap
0xffffc9000302e000-0xffffc90003030000    8192 
acpi_os_map_memory+0x98/0x119 phys=ddfbd000 ioremap
0xffffc90003030000-0xffffc90003032000    8192 
acpi_os_map_memory+0x98/0x119 phys=f7d05000 ioremap
0xffffc90003032000-0xffffc90003034000    8192 
acpi_os_map_memory+0x98/0x119 phys=ddfac000 ioremap
0xffffc90003034000-0xffffc90003036000    8192 
acpi_os_map_memory+0x98/0x119 phys=ddfbc000 ioremap
0xffffc90003036000-0xffffc90003038000    8192 
acpi_pre_map_gar+0xa9/0x1bc phys=dde34000 ioremap
0xffffc90003038000-0xffffc9000303b000   12288 
acpi_os_map_memory+0x98/0x119 phys=ddfaa000 ioremap
0xffffc9000303c000-0xffffc9000303e000    8192 
acpi_os_map_memory+0x98/0x119 phys=fed40000 ioremap
0xffffc9000303e000-0xffffc90003040000    8192 
acpi_os_map_memory+0x98/0x119 phys=fed1f000 ioremap
0xffffc90003040000-0xffffc90003061000  135168 
arch_gnttab_map_shared+0x58/0x70 ioremap
0xffffc90003061000-0xffffc90003064000   12288 
alloc_large_system_hash+0x14b/0x215 pages=2 vmalloc N0=2
0xffffc90003064000-0xffffc90003069000   20480 
alloc_large_system_hash+0x14b/0x215 pages=4 vmalloc N0=4
0xffffc9000306a000-0xffffc9000306c000    8192 
acpi_os_map_memory+0x98/0x119 phys=ddfbe000 ioremap
0xffffc9000306c000-0xffffc90003070000   16384 erst_init+0x196/0x2a5 
phys=dde34000 ioremap
0xffffc90003070000-0xffffc90003073000   12288 ghes_init+0x90/0x16f ioremap
0xffffc90003074000-0xffffc90003076000    8192 
acpi_pre_map_gar+0xa9/0x1bc phys=dde15000 ioremap
0xffffc90003076000-0xffffc90003078000    8192 
usb_hcd_pci_probe+0x228/0x3d0 phys=f7d04000 ioremap
0xffffc90003078000-0xffffc9000307a000    8192 pci_iomap+0x80/0xc0 
phys=f7d02000 ioremap
0xffffc9000307a000-0xffffc9000307c000    8192 
usb_hcd_pci_probe+0x228/0x3d0 phys=f7d03000 ioremap
0xffffc9000307c000-0xffffc9000307e000    8192 
pci_enable_msix+0x195/0x3d0 phys=f7c20000 ioremap
0xffffc9000307e000-0xffffc90003080000    8192 
pci_enable_msix+0x195/0x3d0 phys=f7b20000 ioremap
0xffffc90003080000-0xffffc90007081000 67112960 
pci_mmcfg_arch_init+0x30/0x84 phys=f8000000 ioremap
0xffffc90007081000-0xffffc90007482000 4198400 
alloc_large_system_hash+0x14b/0x215 pages=1024 vmalloc vpages N0=1024
0xffffc90007482000-0xffffc90007c83000 8392704 
alloc_large_system_hash+0x14b/0x215 pages=2048 vmalloc vpages N0=2048
0xffffc90007c83000-0xffffc90007d84000 1052672 
alloc_large_system_hash+0x14b/0x215 pages=256 vmalloc N0=256
0xffffc90007d84000-0xffffc90007e05000  528384 
alloc_large_system_hash+0x14b/0x215 pages=128 vmalloc N0=128
0xffffc90007e05000-0xffffc90007e86000  528384 
alloc_large_system_hash+0x14b/0x215 pages=128 vmalloc N0=128
0xffffc90007e86000-0xffffc90007e88000    8192 
pci_enable_msix+0x195/0x3d0 phys=f7a20000 ioremap
0xffffc90007e88000-0xffffc90007e8a000    8192 
pci_enable_msix+0x195/0x3d0 phys=f7920000 ioremap
0xffffc90007e8a000-0xffffc90007e8c000    8192 
swap_cgroup_swapon+0x60/0x170 pages=1 vmalloc N0=1
0xffffc90007e8c000-0xffffc90007e90000   16384 
e1000e_setup_tx_resources+0x34/0xc0 [e1000e] pages=3 vmalloc N0=3
0xffffc90007e90000-0xffffc90007e94000   16384 
e1000e_setup_rx_resources+0x2f/0x150 [e1000e] pages=3 vmalloc N0=3
0xffffc90007e94000-0xffffc90007e98000   16384 
e1000e_setup_tx_resources+0x34/0xc0 [e1000e] pages=3 vmalloc N0=3
0xffffc90007e98000-0xffffc90007e9c000   16384 
e1000e_setup_rx_resources+0x2f/0x150 [e1000e] pages=3 vmalloc N0=3
0xffffc90007eba000-0xffffc90007ebc000    8192 dm_vcalloc+0x2b/0x30 
pages=1 vmalloc N0=1
0xffffc90007ebc000-0xffffc90007ebe000    8192 dm_vcalloc+0x2b/0x30 
pages=1 vmalloc N0=1
0xffffc90007ec0000-0xffffc90007ee1000  135168 e1000_probe+0x23d/0xb64 
[e1000e] phys=f7c00000 ioremap
0xffffc90007ee3000-0xffffc90007ee7000   16384 
e1000e_setup_tx_resources+0x34/0xc0 [e1000e] pages=3 vmalloc N0=3
0xffffc90007ee7000-0xffffc90007eeb000   16384 
e1000e_setup_rx_resources+0x2f/0x150 [e1000e] pages=3 vmalloc N0=3
0xffffc90007f00000-0xffffc90007f21000  135168 e1000_probe+0x23d/0xb64 
[e1000e] phys=f7b00000 ioremap
0xffffc90007f40000-0xffffc90007f61000  135168 e1000_probe+0x23d/0xb64 
[e1000e] phys=f7a00000 ioremap
0xffffc90007f80000-0xffffc90007fa1000  135168 e1000_probe+0x23d/0xb64 
[e1000e] phys=f7900000 ioremap
0xffffc90007fa1000-0xffffc90007fde000  249856 sys_swapon+0x306/0xbe0 
pages=60 vmalloc N0=60
0xffffc90007fde000-0xffffc90007fe0000    8192 dm_vcalloc+0x2b/0x30 
pages=1 vmalloc N0=1
0xffffc9000803a000-0xffffc9000803c000    8192 dm_vcalloc+0x2b/0x30 
pages=1 vmalloc N0=1
0xffffc90008080000-0xffffc900080db000  372736 0xffffffffa0023046 
pages=90 vmalloc N0=90
0xffffc9000813a000-0xffffc900082c3000 1609728 register_cache+0x3d8/0x7e0 
pages=392 vmalloc N0=392
0xffffc90008766000-0xffffc9000876a000   16384 
xt_alloc_table_info+0xda/0x10e [x_tables] pages=3 vmalloc N0=3
0xffffc9000876a000-0xffffc9000876e000   16384 
xt_alloc_table_info+0xda/0x10e [x_tables] pages=3 vmalloc N0=3
0xffffc9000876e000-0xffffc90008772000   16384 
xt_alloc_table_info+0xda/0x10e [x_tables] pages=3 vmalloc N0=3
0xffffc90008772000-0xffffc90008776000   16384 
xt_alloc_table_info+0xda/0x10e [x_tables] pages=3 vmalloc N0=3
0xffffc90008776000-0xffffc9000877a000   16384 
xt_alloc_table_info+0xda/0x10e [x_tables] pages=3 vmalloc N0=3
0xffffc9000877a000-0xffffc9000877e000   16384 
xt_alloc_table_info+0xda/0x10e [x_tables] pages=3 vmalloc N0=3
0xffffc9000877e000-0xffffc90008782000   16384 
xt_alloc_table_info+0xda/0x10e [x_tables] pages=3 vmalloc N0=3
0xffffc90008782000-0xffffc90008786000   16384 
xt_alloc_table_info+0xda/0x10e [x_tables] pages=3 vmalloc N0=3
0xffffc900087a0000-0xffffc900087a2000    8192 do_replace+0xce/0x1e0 
[ebtables] pages=1 vmalloc N0=1
0xffffc900087a2000-0xffffc900087a4000    8192 do_replace+0xea/0x1e0 
[ebtables] pages=1 vmalloc N0=1
0xffffc900087a8000-0xffffc900087aa000    8192 
xenbus_map_ring_valloc+0x64/0x100 phys=3 ioremap
0xffffc900087aa000-0xffffc900087ac000    8192 
xenbus_map_ring_valloc+0x64/0x100 phys=2 ioremap
0xffffc900087ac000-0xffffc900087ae000    8192 
xenbus_map_ring_valloc+0x64/0x100 phys=e2 ioremap
0xffffc900087ae000-0xffffc900087b0000    8192 
xenbus_map_ring_valloc+0x64/0x100 phys=e7 ioremap
0xffffe8ffffc00000-0xffffe8ffffe00000 2097152 
pcpu_get_vm_areas+0x0/0x530 vmalloc
0xffffffffa0000000-0xffffffffa0005000   20480 
module_alloc_update_bounds+0x1d/0x80 pages=4 vmalloc N0=4
0xffffffffa0008000-0xffffffffa000d000   20480 
module_alloc_update_bounds+0x1d/0x80 pages=4 vmalloc N0=4
0xffffffffa0010000-0xffffffffa0016000   24576 
module_alloc_update_bounds+0x1d/0x80 pages=5 vmalloc N0=5
0xffffffffa0019000-0xffffffffa0023000   40960 
module_alloc_update_bounds+0x1d/0x80 pages=9 vmalloc N0=9
0xffffffffa0027000-0xffffffffa002c000   20480 
module_alloc_update_bounds+0x1d/0x80 pages=4 vmalloc N0=4
0xffffffffa002f000-0xffffffffa0046000   94208 
module_alloc_update_bounds+0x1d/0x80 pages=22 vmalloc N0=22
0xffffffffa0046000-0xffffffffa006d000  159744 
module_alloc_update_bounds+0x1d/0x80 pages=38 vmalloc N0=38
0xffffffffa0071000-0xffffffffa0076000   20480 
module_alloc_update_bounds+0x1d/0x80 pages=4 vmalloc N0=4
0xffffffffa0076000-0xffffffffa007b000   20480 
module_alloc_update_bounds+0x1d/0x80 pages=4 vmalloc N0=4
0xffffffffa007b000-0xffffffffa0087000   49152 
module_alloc_update_bounds+0x1d/0x80 pages=11 vmalloc N0=11
0xffffffffa0087000-0xffffffffa008d000   24576 
module_alloc_update_bounds+0x1d/0x80 pages=5 vmalloc N0=5
0xffffffffa008d000-0xffffffffa009a000   53248 
module_alloc_update_bounds+0x1d/0x80 pages=12 vmalloc N0=12
0xffffffffa009a000-0xffffffffa009f000   20480 
module_alloc_update_bounds+0x1d/0x80 pages=4 vmalloc N0=4
0xffffffffa009f000-0xffffffffa00a4000   20480 
module_alloc_update_bounds+0x1d/0x80 pages=4 vmalloc N0=4
0xffffffffa00a4000-0xffffffffa00aa000   24576 
module_alloc_update_bounds+0x1d/0x80 pages=5 vmalloc N0=5
0xffffffffa00ac000-0xffffffffa00b6000   40960 
module_alloc_update_bounds+0x1d/0x80 pages=9 vmalloc N0=9
0xffffffffa00b6000-0xffffffffa00d0000  106496 
module_alloc_update_bounds+0x1d/0x80 pages=25 vmalloc N0=25
0xffffffffa00d4000-0xffffffffa00da000   24576 
module_alloc_update_bounds+0x1d/0x80 pages=5 vmalloc N0=5
0xffffffffa00da000-0xffffffffa00e0000   24576 
module_alloc_update_bounds+0x1d/0x80 pages=5 vmalloc N0=5
0xffffffffa00e0000-0xffffffffa00e5000   20480 
module_alloc_update_bounds+0x1d/0x80 pages=4 vmalloc N0=4
0xffffffffa00ea000-0xffffffffa00ef000   20480 
module_alloc_update_bounds+0x1d/0x80 pages=4 vmalloc N0=4
0xffffffffa00ef000-0xffffffffa00f5000   24576 
module_alloc_update_bounds+0x1d/0x80 pages=5 vmalloc N0=5
0xffffffffa00f6000-0xffffffffa0103000   53248 
module_alloc_update_bounds+0x1d/0x80 pages=12 vmalloc N0=12
0xffffffffa0103000-0xffffffffa0108000   20480 
module_alloc_update_bounds+0x1d/0x80 pages=4 vmalloc N0=4
0xffffffffa0108000-0xffffffffa010d000   20480 
module_alloc_update_bounds+0x1d/0x80 pages=4 vmalloc N0=4
0xffffffffa010d000-0xffffffffa011c000   61440 
module_alloc_update_bounds+0x1d/0x80 pages=14 vmalloc N0=14
0xffffffffa011c000-0xffffffffa0121000   20480 
module_alloc_update_bounds+0x1d/0x80 pages=4 vmalloc N0=4
0xffffffffa0121000-0xffffffffa0127000   24576 
module_alloc_update_bounds+0x1d/0x80 pages=5 vmalloc N0=5
0xffffffffa0127000-0xffffffffa013a000   77824 
module_alloc_update_bounds+0x1d/0x80 pages=18 vmalloc N0=18
0xffffffffa013a000-0xffffffffa0144000   40960 
module_alloc_update_bounds+0x1d/0x80 pages=9 vmalloc N0=9
0xffffffffa0144000-0xffffffffa0154000   65536 
module_alloc_update_bounds+0x1d/0x80 pages=15 vmalloc N0=15
0xffffffffa0154000-0xffffffffa015a000   24576 
module_alloc_update_bounds+0x1d/0x80 pages=5 vmalloc N0=5
0xffffffffa015e000-0xffffffffa0163000   20480 
module_alloc_update_bounds+0x1d/0x80 pages=4 vmalloc N0=4
0xffffffffa0163000-0xffffffffa0169000   24576 
module_alloc_update_bounds+0x1d/0x80 pages=5 vmalloc N0=5
0xffffffffa0169000-0xffffffffa0172000   36864 
module_alloc_update_bounds+0x1d/0x80 pages=8 vmalloc N0=8
0xffffffffa0172000-0xffffffffa0177000   20480 
module_alloc_update_bounds+0x1d/0x80 pages=4 vmalloc N0=4
0xffffffffa0178000-0xffffffffa0180000   32768 
module_alloc_update_bounds+0x1d/0x80 pages=7 vmalloc N0=7
0xffffffffa0180000-0xffffffffa0188000   32768 
module_alloc_update_bounds+0x1d/0x80 pages=7 vmalloc N0=7
0xffffffffa018c000-0xffffffffa0191000   20480 
module_alloc_update_bounds+0x1d/0x80 pages=4 vmalloc N0=4
0xffffffffa0191000-0xffffffffa019a000   36864 
module_alloc_update_bounds+0x1d/0x80 pages=8 vmalloc N0=8
0xffffffffa019e000-0xffffffffa01a3000   20480 
module_alloc_update_bounds+0x1d/0x80 pages=4 vmalloc N0=4
0xffffffffa01a3000-0xffffffffa01a8000   20480 
module_alloc_update_bounds+0x1d/0x80 pages=4 vmalloc N0=4
0xffffffffa01a8000-0xffffffffa01bf000   94208 
module_alloc_update_bounds+0x1d/0x80 pages=22 vmalloc N0=22
0xffffffffa01bf000-0xffffffffa01c4000   20480 
module_alloc_update_bounds+0x1d/0x80 pages=4 vmalloc N0=4
0xffffffffa01c8000-0xffffffffa01cd000   20480 
module_alloc_update_bounds+0x1d/0x80 pages=4 vmalloc N0=4
0xffffffffa01cd000-0xffffffffa01d4000   28672 
module_alloc_update_bounds+0x1d/0x80 pages=6 vmalloc N0=6
0xffffffffa01d4000-0xffffffffa01db000   28672 
module_alloc_update_bounds+0x1d/0x80 pages=6 vmalloc N0=6
0xffffffffa01df000-0xffffffffa01e7000   32768 
module_alloc_update_bounds+0x1d/0x80 pages=7 vmalloc N0=7

Re: Expected Behavior

[Jonathan Tripathy]

On 30.08.2012 08:21, Jonathan Tripathy wrote:
> On 30/08/2012 08:15, Jonathan Tripathy wrote:
>> Hi There,
>>
>> On my WIndows DomU (Xen VM) which is running on a LV which is using 
>> bcache (against two SSD in MDRAID1 and a MD-RAID10 spindle array), I 
>> ran an IOMeter test for about 2 hours (with 30 workers and a io depth 
>> of 256). This was a very heavy workload (Got an average iops of about 
>> 6.5k). After I stopped the test, I then went back to fio on my Linux 
>> Xen Host (Dom0). The random write performance isn't as good as it was 
>> before I started the IOMeter test. It used to be about 25k and now 
>> showed about 7k iops. I assumed that maybe this was due to the fact 
>> that bcache was writing out dirty data to the spindles so the SSD was 
>> busy.
>>
>> However, this morning, after the spindles have calmed down, 
>> performance of fio is still not great (still about 7k).
>>
>> Is there something wrong here? What is expected behavior?
>>
>> Thanks
>>
> BTW, I can confirm that this isn't an SSD issue, as I have a
> partition on the SSD that I kept seperate from bcache and I'm getting
> excellent (about 28k) iops performance there.
>
> It's as if after the heavy workload I did with IOMeter, bcache has
> somehow throttled the writeback cache?
>
> Any help is appreciated.
>

I'd like to add that a reboot pretty much solves the issue. This leads 
me to believe that there is a bug in the bcache code that causes 
performance to drop the more it gets used.

Any ideas?

Thanks

Re: Expected Behavior

[Kent Overstreet]

On Thu, Aug 30, 2012 at 01:18:54PM +0100, Jonathan Tripathy wrote:
> 
> On 30.08.2012 08:21, Jonathan Tripathy wrote:
> >On 30/08/2012 08:15, Jonathan Tripathy wrote:
> >>Hi There,
> >>
> >>On my WIndows DomU (Xen VM) which is running on a LV which is
> >>using bcache (against two SSD in MDRAID1 and a MD-RAID10 spindle
> >>array), I ran an IOMeter test for about 2 hours (with 30 workers
> >>and a io depth of 256). This was a very heavy workload (Got an
> >>average iops of about 6.5k). After I stopped the test, I then
> >>went back to fio on my Linux Xen Host (Dom0). The random write
> >>performance isn't as good as it was before I started the IOMeter
> >>test. It used to be about 25k and now showed about 7k iops. I
> >>assumed that maybe this was due to the fact that bcache was
> >>writing out dirty data to the spindles so the SSD was busy.
> >>
> >>However, this morning, after the spindles have calmed down,
> >>performance of fio is still not great (still about 7k).
> >>
> >>Is there something wrong here? What is expected behavior?
> >>
> >>Thanks
> >>
> >BTW, I can confirm that this isn't an SSD issue, as I have a
> >partition on the SSD that I kept seperate from bcache and I'm getting
> >excellent (about 28k) iops performance there.
> >
> >It's as if after the heavy workload I did with IOMeter, bcache has
> >somehow throttled the writeback cache?
> >
> >Any help is appreciated.
> >
> 
> I'd like to add that a reboot pretty much solves the issue. This
> leads me to believe that there is a bug in the bcache code that
> causes performance to drop the more it gets used.
> 
> Any ideas?

Weird!

Yeah, that definitely sounds like a bug. I'm going to have to try and
reproduce it and go hunting. Can you think of anything that might help
with reproducing it?

Re: Expected Behavior

[Jonathan Tripathy]

On 30/08/2012 22:28, Kent Overstreet wrote:
> On Thu, Aug 30, 2012 at 01:18:54PM +0100, Jonathan Tripathy wrote:
>> On 30.08.2012 08:21, Jonathan Tripathy wrote:
>>> On 30/08/2012 08:15, Jonathan Tripathy wrote:
>>>> Hi There,
>>>>
>>>> On my WIndows DomU (Xen VM) which is running on a LV which is
>>>> using bcache (against two SSD in MDRAID1 and a MD-RAID10 spindle
>>>> array), I ran an IOMeter test for about 2 hours (with 30 workers
>>>> and a io depth of 256). This was a very heavy workload (Got an
>>>> average iops of about 6.5k). After I stopped the test, I then
>>>> went back to fio on my Linux Xen Host (Dom0). The random write
>>>> performance isn't as good as it was before I started the IOMeter
>>>> test. It used to be about 25k and now showed about 7k iops. I
>>>> assumed that maybe this was due to the fact that bcache was
>>>> writing out dirty data to the spindles so the SSD was busy.
>>>>
>>>> However, this morning, after the spindles have calmed down,
>>>> performance of fio is still not great (still about 7k).
>>>>
>>>> Is there something wrong here? What is expected behavior?
>>>>
>>>> Thanks
>>>>
>>> BTW, I can confirm that this isn't an SSD issue, as I have a
>>> partition on the SSD that I kept seperate from bcache and I'm getting
>>> excellent (about 28k) iops performance there.
>>>
>>> It's as if after the heavy workload I did with IOMeter, bcache has
>>> somehow throttled the writeback cache?
>>>
>>> Any help is appreciated.
>>>
>> I'd like to add that a reboot pretty much solves the issue. This
>> leads me to believe that there is a bug in the bcache code that
>> causes performance to drop the more it gets used.
>>
>> Any ideas?
> Weird!
>
> Yeah, that definitely sounds like a bug. I'm going to have to try and
> reproduce it and go hunting. Can you think of anything that might help
> with reproducing it?
> -
Hi Kent,

I'm going to try and reproduce it myself as well. I just used IOMeter in 
a Windows DomU with 30 workers, each having an io depth of 256. A *very* 
heavy workload indeed, but my point was to see if I could break 
something. Unless the issue is specific to windows causing problems 
(NTFS or whatever), I'm guessing running fio with 30 jobs and an iodepth 
of 256 would probably produce a similar load.

BTW, do you have access to a Xen node for testing?

Thanks

Re: Expected Behavior

[Kent Overstreet]

On Thu, Aug 30, 2012 at 3:59 PM, Jonathan Tripathy <jonnyt-Nf8S+5hNwl710XsdtD+oqA@public.gmane.org> wrote:
> I'm going to try and reproduce it myself as well. I just used IOMeter in a
> Windows DomU with 30 workers, each having an io depth of 256. A *very* heavy
> workload indeed, but my point was to see if I could break something. Unless
> the issue is specific to windows causing problems (NTFS or whatever), I'm
> guessing running fio with 30 jobs and an iodepth of 256 would probably
> produce a similar load.
>
> BTW, do you have access to a Xen node for testing?

No, I don't... and from what I remember, Xen was a _pain_ to set up...

RE: Expected Behavior

[James Harper]

> Hi Kent,
> 
> I'm going to try and reproduce it myself as well. I just used IOMeter in a
> Windows DomU with 30 workers, each having an io depth of 256. A *very*
> heavy workload indeed, but my point was to see if I could break something.
> Unless the issue is specific to windows causing problems (NTFS or whatever),
> I'm guessing running fio with 30 jobs and an iodepth of 256 would probably
> produce a similar load.
> 
> BTW, do you have access to a Xen node for testing?
> 

Does the problem resolve itself after you shut down the windows DomU? Or only when you reboot the whole Dom0?

James

RE: Expected Behavior

[Jonathan Tripathy]

On 31.08.2012 04:47, James Harper wrote:
>> Hi Kent,
>>
>> I'm going to try and reproduce it myself as well. I just used 
>> IOMeter in a
>> Windows DomU with 30 workers, each having an io depth of 256. A 
>> *very*
>> heavy workload indeed, but my point was to see if I could break 
>> something.
>> Unless the issue is specific to windows causing problems (NTFS or 
>> whatever),
>> I'm guessing running fio with 30 jobs and an iodepth of 256 would 
>> probably
>> produce a similar load.
>>
>> BTW, do you have access to a Xen node for testing?
>>
>
> Does the problem resolve itself after you shut down the windows DomU?
> Or only when you reboot the whole Dom0?
>

Hi There,

I managed to reproduce this again. I have to reboot the entire Dom0 
(the physical server) for it to work properly again.

James, are you able to reproduce this? Kent, are there any other 
tests/debug output you need from me?

Thanks

RE: Expected Behavior

[Jonathan Tripathy]

On 31.08.2012 13:36, Jonathan Tripathy wrote:
> On 31.08.2012 04:47, James Harper wrote:
>>> Hi Kent,
>>>
>>> I'm going to try and reproduce it myself as well. I just used 
>>> IOMeter in a
>>> Windows DomU with 30 workers, each having an io depth of 256. A 
>>> *very*
>>> heavy workload indeed, but my point was to see if I could break 
>>> something.
>>> Unless the issue is specific to windows causing problems (NTFS or 
>>> whatever),
>>> I'm guessing running fio with 30 jobs and an iodepth of 256 would 
>>> probably
>>> produce a similar load.
>>>
>>> BTW, do you have access to a Xen node for testing?
>>>
>>
>> Does the problem resolve itself after you shut down the windows 
>> DomU?
>> Or only when you reboot the whole Dom0?
>>
>
> Hi There,
>
> I managed to reproduce this again. I have to reboot the entire Dom0
> (the physical server) for it to work properly again.
>
> James, are you able to reproduce this? Kent, are there any other
> tests/debug output you need from me?
>

BTW, I was using IOMeter's 'default' Access Specification with the 
following modifications: 100% random, 66% read, 33% write, and a 2kB 
size. My bcache is formatted for 512bytes.

Re: Expected Behavior

[Jonathan Tripathy]

On 31/08/2012 13:41, Jonathan Tripathy wrote:
>
>
> On 31.08.2012 13:36, Jonathan Tripathy wrote:
>> On 31.08.2012 04:47, James Harper wrote:
>>>> Hi Kent,
>>>>
>>>> I'm going to try and reproduce it myself as well. I just used 
>>>> IOMeter in a
>>>> Windows DomU with 30 workers, each having an io depth of 256. A *very*
>>>> heavy workload indeed, but my point was to see if I could break 
>>>> something.
>>>> Unless the issue is specific to windows causing problems (NTFS or 
>>>> whatever),
>>>> I'm guessing running fio with 30 jobs and an iodepth of 256 would 
>>>> probably
>>>> produce a similar load.
>>>>
>>>> BTW, do you have access to a Xen node for testing?
>>>>
>>>
>>> Does the problem resolve itself after you shut down the windows DomU?
>>> Or only when you reboot the whole Dom0?
>>>
>>
>> Hi There,
>>
>> I managed to reproduce this again. I have to reboot the entire Dom0
>> (the physical server) for it to work properly again.
>>
>> James, are you able to reproduce this? Kent, are there any other
>> tests/debug output you need from me?
>>
>
> BTW, I was using IOMeter's 'default' Access Specification with the 
> following modifications: 100% random, 66% read, 33% write, and a 2kB 
> size. My bcache is formatted for 512bytes.
> -- 
>
Kent, is there any debug output of some sort I could switch on and help 
you figure out what's going on? If needs be, I can give you access to my 
setup here where you can run these tests yourself, if you're not keen 
installing Xen on your end :)

Thanks

Re: Expected Behavior

[Kent Overstreet]

On Sat, Sep 01, 2012 at 01:47:52PM +0100, Jonathan Tripathy wrote:
> On 31/08/2012 13:41, Jonathan Tripathy wrote:
> >
> >
> >On 31.08.2012 13:36, Jonathan Tripathy wrote:
> >>On 31.08.2012 04:47, James Harper wrote:
> >>>>Hi Kent,
> >>>>
> >>>>I'm going to try and reproduce it myself as well. I just
> >>>>used IOMeter in a
> >>>>Windows DomU with 30 workers, each having an io depth of 256. A *very*
> >>>>heavy workload indeed, but my point was to see if I could
> >>>>break something.
> >>>>Unless the issue is specific to windows causing problems
> >>>>(NTFS or whatever),
> >>>>I'm guessing running fio with 30 jobs and an iodepth of 256
> >>>>would probably
> >>>>produce a similar load.
> >>>>
> >>>>BTW, do you have access to a Xen node for testing?
> >>>>
> >>>
> >>>Does the problem resolve itself after you shut down the windows DomU?
> >>>Or only when you reboot the whole Dom0?
> >>>
> >>
> >>Hi There,
> >>
> >>I managed to reproduce this again. I have to reboot the entire Dom0
> >>(the physical server) for it to work properly again.
> >>
> >>James, are you able to reproduce this? Kent, are there any other
> >>tests/debug output you need from me?
> >>
> >
> >BTW, I was using IOMeter's 'default' Access Specification with the
> >following modifications: 100% random, 66% read, 33% write, and a
> >2kB size. My bcache is formatted for 512bytes.
> >-- 
> >
> Kent, is there any debug output of some sort I could switch on and
> help you figure out what's going on? If needs be, I can give you
> access to my setup here where you can run these tests yourself, if
> you're not keen installing Xen on your end :)

Shell access would probably be fastest, I suppose...

One thing that comes to mind is perhaps the load from background
writeback is slowing things down. Two things you can do:

set writeback_percent to 10, that'll enable a pd controller so it's not
going full blast

benchmark it with writeback_running set to 0 - that disables background
writeback completely.

Re: Expected Behavior

[Jonathan Tripathy]

On 03.09.2012 01:37, Kent Overstreet wrote:
> On Sat, Sep 01, 2012 at 01:47:52PM +0100, Jonathan Tripathy wrote:
>> On 31/08/2012 13:41, Jonathan Tripathy wrote:
>> >
>> >
>> >On 31.08.2012 13:36, Jonathan Tripathy wrote:
>> >>On 31.08.2012 04:47, James Harper wrote:
>> >>>>Hi Kent,
>> >>>>
>> >>>>I'm going to try and reproduce it myself as well. I just
>> >>>>used IOMeter in a
>> >>>>Windows DomU with 30 workers, each having an io depth of 256. A 
>> *very*
>> >>>>heavy workload indeed, but my point was to see if I could
>> >>>>break something.
>> >>>>Unless the issue is specific to windows causing problems
>> >>>>(NTFS or whatever),
>> >>>>I'm guessing running fio with 30 jobs and an iodepth of 256
>> >>>>would probably
>> >>>>produce a similar load.
>> >>>>
>> >>>>BTW, do you have access to a Xen node for testing?
>> >>>>
>> >>>
>> >>>Does the problem resolve itself after you shut down the windows 
>> DomU?
>> >>>Or only when you reboot the whole Dom0?
>> >>>
>> >>
>> >>Hi There,
>> >>
>> >>I managed to reproduce this again. I have to reboot the entire 
>> Dom0
>> >>(the physical server) for it to work properly again.
>> >>
>> >>James, are you able to reproduce this? Kent, are there any other
>> >>tests/debug output you need from me?
>> >>
>> >
>> >BTW, I was using IOMeter's 'default' Access Specification with the
>> >following modifications: 100% random, 66% read, 33% write, and a
>> >2kB size. My bcache is formatted for 512bytes.
>> >--
>> >
>> Kent, is there any debug output of some sort I could switch on and
>> help you figure out what's going on? If needs be, I can give you
>> access to my setup here where you can run these tests yourself, if
>> you're not keen installing Xen on your end :)
>
> Shell access would probably be fastest, I suppose...
>
> One thing that comes to mind is perhaps the load from background
> writeback is slowing things down. Two things you can do:
>
> set writeback_percent to 10, that'll enable a pd controller so it's 
> not
> going full blast
>

Hi Kent,

I will try the above configuration change and repeat the test. However, 
it's worth noting that I left an overnight gap between from when my 
iomater run finished and from when I started the fio test. This was to 
ensure that all data had been written out to backing storage. While I 
didn't check if the cache was clean or dirty in the morning, I can 
confirm that there was no disk activity according to the HDD lights on 
the server case.

Cheers

Re: Expected Behavior

[Kent Overstreet]

On Mon, Sep 03, 2012 at 09:30:26AM +0100, Jonathan Tripathy wrote:
> On 03.09.2012 01:37, Kent Overstreet wrote:
> >Shell access would probably be fastest, I suppose...
> >
> >One thing that comes to mind is perhaps the load from background
> >writeback is slowing things down. Two things you can do:
> >
> >set writeback_percent to 10, that'll enable a pd controller so
> >it's not
> >going full blast
> >
> 
> Hi Kent,
> 
> I will try the above configuration change and repeat the test.
> However, it's worth noting that I left an overnight gap between from
> when my iomater run finished and from when I started the fio test.
> This was to ensure that all data had been written out to backing
> storage. While I didn't check if the cache was clean or dirty in the
> morning, I can confirm that there was no disk activity according to
> the HDD lights on the server case.

Shoot. That probably rules that out.

Yeah, I'll probably just have to poke around at random stuff. Hopefully
it's just something burning cpu and profiling will make it obvious.