Strange behavior

Strange behavior

[CoRpO]

Hi,

I'm currently running the latest bcache (non-dev) on an ubuntu 12.04, 
and it seems to me that bcache doesn't cache anything

The setup is as follow:

- One drive for the system
- a 480 GB SSD caching device (/dev/sdb)
- a 10 drives Raid 5 (on a HP P410 controller) backing device 
(/dev/sdc)
- 36 GB ram

/dev/bcache0 mounted as /home

The server is a file server, with lots of file but few of them served 
frequently. So that's why I've put bcache on it, to have frequently 
asked files on the caching device (which are bigger than the server's 
ram)

I've filled the server with 16 TB of datas, and I see little to no 
usage of /dev/sdb

# iostat -x 30
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
            0.05    0.00    0.74   10.92    0.00   88.30

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s 
avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.40    0.00    0.83     0.00     5.07    
12.16     0.00    0.00    0.00    0.00   0.00   0.00
sdb               0.27     0.03    1.10    0.17   136.57     0.37   
216.21     0.00    0.95    1.09    0.00   0.53   0.07
sdc               0.00     0.00  185.53    0.03 23750.93     0.03   
255.98     3.14   16.90   16.90    0.00   4.40  81.59
bcache0           0.00     0.00  186.67    0.10 23896.00     0.32   
255.89     0.00   16.81   16.82    0.00   0.00   0.00

sequential_cutoff and sequential_merge are echoed 0 to disable them, 
because I want sequential IOs to be cached

I get strange numbers from the variables. Some seems to indicate that 
bcache is doing its work, others seems not.
I'm totally puzzled. Am I doing something wrong ? It bcache suited for 
what I want to do ? Is it a bug ?

I've even tried to recreate the caching device to get rid of the data 
that was on it during data was put on the backing device.
As noted in this mailing-list, I made some writes to the device to 
allow the btree to split/grow if needed, and seen no changes.

Here are the details :

# cat /sys/block/bcache0/bcache/cache/internal/active_journal_entries
3

# cat /sys/block/bcache0/bcache/cache/internal/bset_tree_stats
btree nodes:            1
written sets:           2
unwritten sets:         1
written key bytes:      156768
unwritten key bytes:    96
floats:                 1222
failed:                 63

# cat /sys/block/bcache0/bcache/cache/internal/btree_nodes
1
# cat /sys/block/bcache0/bcache/cache/internal/btree_used_percent
46
# cat /sys/block/bcache0/bcache/cache/average_key_size
384k
# cat /sys/block/bcache0/bcache/cache/btree_cache_size
0
# cat /sys/block/bcache0/bcache/cache/cache_available_percent
99
# cat /sys/block/bcache0/bcache/cache/root_usage_percent
47
# cat /sys/block/bcache0/bcache/cache/tree_depth
0
# cat /sys/block/bcache0/bcache/cache_mode
writethrough [writeback] writearound none
# cat /sys/block/bcache0/bcache/readahead
1.0M
# cat /sys/block/bcache0/bcache/stats_hour/bypassed
0
# cat /sys/block/bcache0/bcache/stats_hour/cache_hit_ratio
100
# cat /sys/block/bcache0/bcache/stats_hour/cache_hits
960779
# cat /sys/block/bcache0/bcache/stats_hour/cache_misses
0
# cat /sys/block/bcache0/bcache/stats_hour/cache_readaheads
0
# cat /sys/block/bcache0/bcache/cache/cache0/freelist_percent
0
# cat /sys/block/bcache0/bcache/cache/cache0/priority_stats
Unused:         89%
Metadata:       0%
Average:        4141
Sectors per Q:  3109760
Quantiles:      [4162 4189 4193 4197 4201 4205 4209 4213 4217 4221 4225 
4229 4233 4237 4241 4245 4249 4253 4257 4260 4264 4268 4272 4276 4280 
4284 4288 4292 4296 4300 4304]
# cat /sys/block/bcache0/bcache/cache/cache0/written
47.4G
# cat /sys/block/bcache0/bcache/cache/cache0/cache_replacement_policy
[lru] fifo random
# cat /sys/block/bcache0/bcache/cache/cache0/nbuckets
915661
# cat /sys/block/bcache0/bcache/cache/cache0/metadata_written
13.3M

I don't know what to try. At first I used bcache-3.2, and seeing this 
beavior made me switch to bcache and 3.8 kernel. Tried bcache-dev, too, 
but the server doesn't boot, kernel stuck in a softlock @[bcache:345] 
(no idea of the meaning)

What am I doing wrong ?
There's little to no activity on the caching device, I though there 
would be write operations to cache served files, but the only writes are 
when I put new files. The pattern is 99% read of big files (~ from 100 
MB to 10 GB)

Thank you

Re: Strange behavior

[CoRpO]

Le 2013-03-14 13:17, CoRpO a écrit :

> Hi,
>
> I'm currently running the latest bcache (non-dev) on an ubuntu 12.04,
> and it seems to me that bcache doesn't cache anything
>
> The setup is as follow:
>
> - One drive for the system
> - a 480 GB SSD caching device (/dev/sdb)
> - a 10 drives Raid 5 (on a HP P410 controller) backing device
> (/dev/sdc)
> - 36 GB ram
>
> /dev/bcache0 mounted as /home
>
> The server is a file server, with lots of file but few of them served
> frequently. So that's why I've put bcache on it, to have frequently
> asked files on the caching device (which are bigger than the server's
> ram)
>
> I've filled the server with 16 TB of datas, and I see little to no
> usage of /dev/sdb
>
> # iostat -x 30
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.05 0.00 0.74 10.92 0.00 88.30
>
> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
> avgrq-sz avgqu-sz await r_await w_await svctm %util
> sda 0.00 0.40 0.00 0.83 0.00 5.07
> 12.16 0.00 0.00 0.00 0.00 0.00 0.00
> sdb 0.27 0.03 1.10 0.17 136.57 0.37
> 216.21 0.00 0.95 1.09 0.00 0.53 0.07
> sdc 0.00 0.00 185.53 0.03 23750.93 0.03
> 255.98 3.14 16.90 16.90 0.00 4.40 81.59
> bcache0 0.00 0.00 186.67 0.10 23896.00 0.32
> 255.89 0.00 16.81 16.82 0.00 0.00 0.00
>
> sequential_cutoff and sequential_merge are echoed 0 to disable them,
> because I want sequential IOs to be cached
>
> I get strange numbers from the variables. Some seems to indicate that
> bcache is doing its work, others seems not.
> I'm totally puzzled. Am I doing something wrong ? It bcache suited 
> for
> what I want to do ? Is it a bug ?
>
> I've even tried to recreate the caching device to get rid of the data
> that was on it during data was put on the backing device.
> As noted in this mailing-list, I made some writes to the device to
> allow the btree to split/grow if needed, and seen no changes.
>
> Here are the details :
>
> # cat /sys/block/bcache0/bcache/cache/internal/active_journal_entries
> 3
>
> # cat /sys/block/bcache0/bcache/cache/internal/bset_tree_stats
> btree nodes: 1
> written sets: 2
> unwritten sets: 1
> written key bytes: 156768
> unwritten key bytes: 96
> floats: 1222
> failed: 63
>
> # cat /sys/block/bcache0/bcache/cache/internal/btree_nodes
> 1
> # cat /sys/block/bcache0/bcache/cache/internal/btree_used_percent
> 46
> # cat /sys/block/bcache0/bcache/cache/average_key_size
> 384k
> # cat /sys/block/bcache0/bcache/cache/btree_cache_size
> 0
> # cat /sys/block/bcache0/bcache/cache/cache_available_percent
> 99
> # cat /sys/block/bcache0/bcache/cache/root_usage_percent
> 47
> # cat /sys/block/bcache0/bcache/cache/tree_depth
> 0
> # cat /sys/block/bcache0/bcache/cache_mode
> writethrough [writeback] writearound none
> # cat /sys/block/bcache0/bcache/readahead
> 1.0M
> # cat /sys/block/bcache0/bcache/stats_hour/bypassed
> 0
> # cat /sys/block/bcache0/bcache/stats_hour/cache_hit_ratio
> 100
> # cat /sys/block/bcache0/bcache/stats_hour/cache_hits
> 960779
> # cat /sys/block/bcache0/bcache/stats_hour/cache_misses
> 0
> # cat /sys/block/bcache0/bcache/stats_hour/cache_readaheads
> 0
> # cat /sys/block/bcache0/bcache/cache/cache0/freelist_percent
> 0
> # cat /sys/block/bcache0/bcache/cache/cache0/priority_stats
> Unused: 89%
> Metadata: 0%
> Average: 4141
> Sectors per Q: 3109760
> Quantiles: [4162 4189 4193 4197 4201 4205 4209 4213 4217 4221 4225
> 4229 4233 4237 4241 4245 4249 4253 4257 4260 4264 4268 4272 4276 4280
> 4284 4288 4292 4296 4300 4304]
> # cat /sys/block/bcache0/bcache/cache/cache0/written
> 47.4G
> # cat /sys/block/bcache0/bcache/cache/cache0/cache_replacement_policy
> [lru] fifo random
> # cat /sys/block/bcache0/bcache/cache/cache0/nbuckets
> 915661
> # cat /sys/block/bcache0/bcache/cache/cache0/metadata_written
> 13.3M
>
> I don't know what to try. At first I used bcache-3.2, and seeing this
> beavior made me switch to bcache and 3.8 kernel. Tried bcache-dev, 
> too,
> but the server doesn't boot, kernel stuck in a softlock @[bcache:345]
> (no idea of the meaning)
>
> What am I doing wrong ?
> There's little to no activity on the caching device, I though there
> would be write operations to cache served files, but the only writes 
> are
> when I put new files. The pattern is 99% read of big files (~ from 
> 100
> MB to 10 GB)
>
> Thank you

In addition, here are the results of a 1 minutes blktrace

Total (sdb):
  Reads Queued:           3,       10KiB  Writes Queued:           8,    
    2KiB
  Read Dispatches:        3,       10KiB  Write Dispatches:        8,    
    2KiB
  Reads Requeued:         0               Writes Requeued:         0
  Reads Completed:        3,       10KiB  Writes Completed:        8,    
    2KiB
  Read Merges:            0,        0KiB  Write Merges:            0,    
    0KiB
  IO unplugs:             3               Timer unplugs:           0

Total (sdc):
  Reads Queued:       14674,     1878MiB  Writes Queued:           3,    
   10KiB
  Read Dispatches:    14674,     1878MiB  Write Dispatches:        3,    
   10KiB
  Reads Requeued:         0               Writes Requeued:         0
  Reads Completed:    14678,     1879MiB  Writes Completed:        3,    
   10KiB
  Read Merges:            0,        0KiB  Write Merges:            0,    
    0KiB
  IO unplugs:         14674               Timer unplugs:           0

Total (bcache0):
  Reads Queued:       14674,     1878MiB  Writes Queued:           3,    
    2KiB
  Read Dispatches:        0,        0KiB  Write Dispatches:        0,    
    0KiB
  Reads Requeued:         0               Writes Requeued:         0
  Reads Completed:        0,        0KiB  Writes Completed:        0,    
    0KiB
  Read Merges:            0,        0KiB  Write Merges:            0,    
    0KiB
  IO unplugs:             0               Timer unplugs:           0

Re: Strange behavior

[Kent Overstreet]

Can you try disabling congested_threshold? That's the one other option
that allows data to bypass the cache.

On Thu, Mar 14, 2013 at 7:32 AM, CoRpO <corpo-Zp/rcV4kTj0@public.gmane.org> wrote:
> Le 2013-03-14 13:17, CoRpO a écrit :
>
>
>> Hi,
>>
>> I'm currently running the latest bcache (non-dev) on an ubuntu 12.04,
>> and it seems to me that bcache doesn't cache anything
>>
>> The setup is as follow:
>>
>> - One drive for the system
>> - a 480 GB SSD caching device (/dev/sdb)
>> - a 10 drives Raid 5 (on a HP P410 controller) backing device
>> (/dev/sdc)
>> - 36 GB ram
>>
>> /dev/bcache0 mounted as /home
>>
>> The server is a file server, with lots of file but few of them served
>> frequently. So that's why I've put bcache on it, to have frequently
>> asked files on the caching device (which are bigger than the server's
>> ram)
>>
>> I've filled the server with 16 TB of datas, and I see little to no
>> usage of /dev/sdb
>>
>> # iostat -x 30
>> avg-cpu: %user %nice %system %iowait %steal %idle
>> 0.05 0.00 0.74 10.92 0.00 88.30
>>
>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
>> avgrq-sz avgqu-sz await r_await w_await svctm %util
>> sda 0.00 0.40 0.00 0.83 0.00 5.07
>> 12.16 0.00 0.00 0.00 0.00 0.00 0.00
>> sdb 0.27 0.03 1.10 0.17 136.57 0.37
>> 216.21 0.00 0.95 1.09 0.00 0.53 0.07
>> sdc 0.00 0.00 185.53 0.03 23750.93 0.03
>> 255.98 3.14 16.90 16.90 0.00 4.40 81.59
>> bcache0 0.00 0.00 186.67 0.10 23896.00 0.32
>> 255.89 0.00 16.81 16.82 0.00 0.00 0.00
>>
>> sequential_cutoff and sequential_merge are echoed 0 to disable them,
>> because I want sequential IOs to be cached
>>
>> I get strange numbers from the variables. Some seems to indicate that
>> bcache is doing its work, others seems not.
>> I'm totally puzzled. Am I doing something wrong ? It bcache suited for
>> what I want to do ? Is it a bug ?
>>
>> I've even tried to recreate the caching device to get rid of the data
>> that was on it during data was put on the backing device.
>> As noted in this mailing-list, I made some writes to the device to
>> allow the btree to split/grow if needed, and seen no changes.
>>
>> Here are the details :
>>
>> # cat /sys/block/bcache0/bcache/cache/internal/active_journal_entries
>> 3
>>
>> # cat /sys/block/bcache0/bcache/cache/internal/bset_tree_stats
>> btree nodes: 1
>> written sets: 2
>> unwritten sets: 1
>> written key bytes: 156768
>> unwritten key bytes: 96
>> floats: 1222
>> failed: 63
>>
>> # cat /sys/block/bcache0/bcache/cache/internal/btree_nodes
>> 1
>> # cat /sys/block/bcache0/bcache/cache/internal/btree_used_percent
>> 46
>> # cat /sys/block/bcache0/bcache/cache/average_key_size
>> 384k
>> # cat /sys/block/bcache0/bcache/cache/btree_cache_size
>> 0
>> # cat /sys/block/bcache0/bcache/cache/cache_available_percent
>> 99
>> # cat /sys/block/bcache0/bcache/cache/root_usage_percent
>> 47
>> # cat /sys/block/bcache0/bcache/cache/tree_depth
>> 0
>> # cat /sys/block/bcache0/bcache/cache_mode
>> writethrough [writeback] writearound none
>> # cat /sys/block/bcache0/bcache/readahead
>> 1.0M
>> # cat /sys/block/bcache0/bcache/stats_hour/bypassed
>> 0
>> # cat /sys/block/bcache0/bcache/stats_hour/cache_hit_ratio
>> 100
>> # cat /sys/block/bcache0/bcache/stats_hour/cache_hits
>> 960779
>> # cat /sys/block/bcache0/bcache/stats_hour/cache_misses
>> 0
>> # cat /sys/block/bcache0/bcache/stats_hour/cache_readaheads
>> 0
>> # cat /sys/block/bcache0/bcache/cache/cache0/freelist_percent
>> 0
>> # cat /sys/block/bcache0/bcache/cache/cache0/priority_stats
>> Unused: 89%
>> Metadata: 0%
>> Average: 4141
>> Sectors per Q: 3109760
>> Quantiles: [4162 4189 4193 4197 4201 4205 4209 4213 4217 4221 4225
>> 4229 4233 4237 4241 4245 4249 4253 4257 4260 4264 4268 4272 4276 4280
>> 4284 4288 4292 4296 4300 4304]
>> # cat /sys/block/bcache0/bcache/cache/cache0/written
>> 47.4G
>> # cat /sys/block/bcache0/bcache/cache/cache0/cache_replacement_policy
>> [lru] fifo random
>> # cat /sys/block/bcache0/bcache/cache/cache0/nbuckets
>> 915661
>> # cat /sys/block/bcache0/bcache/cache/cache0/metadata_written
>> 13.3M
>>
>> I don't know what to try. At first I used bcache-3.2, and seeing this
>> beavior made me switch to bcache and 3.8 kernel. Tried bcache-dev, too,
>> but the server doesn't boot, kernel stuck in a softlock @[bcache:345]
>> (no idea of the meaning)
>>
>> What am I doing wrong ?
>> There's little to no activity on the caching device, I though there
>> would be write operations to cache served files, but the only writes are
>> when I put new files. The pattern is 99% read of big files (~ from 100
>> MB to 10 GB)
>>
>> Thank you
>
>
> In addition, here are the results of a 1 minutes blktrace
>
> Total (sdb):
>  Reads Queued:           3,       10KiB  Writes Queued:           8,
> 2KiB
>  Read Dispatches:        3,       10KiB  Write Dispatches:        8,
> 2KiB
>  Reads Requeued:         0               Writes Requeued:         0
>  Reads Completed:        3,       10KiB  Writes Completed:        8,
> 2KiB
>  Read Merges:            0,        0KiB  Write Merges:            0,
> 0KiB
>  IO unplugs:             3               Timer unplugs:           0
>
> Total (sdc):
>  Reads Queued:       14674,     1878MiB  Writes Queued:           3,
> 10KiB
>  Read Dispatches:    14674,     1878MiB  Write Dispatches:        3,
> 10KiB
>  Reads Requeued:         0               Writes Requeued:         0
>  Reads Completed:    14678,     1879MiB  Writes Completed:        3,
> 10KiB
>  Read Merges:            0,        0KiB  Write Merges:            0,
> 0KiB
>  IO unplugs:         14674               Timer unplugs:           0
>
> Total (bcache0):
>  Reads Queued:       14674,     1878MiB  Writes Queued:           3,
> 2KiB
>  Read Dispatches:        0,        0KiB  Write Dispatches:        0,
> 0KiB
>  Reads Requeued:         0               Writes Requeued:         0
>  Reads Completed:        0,        0KiB  Writes Completed:        0,
> 0KiB
>  Read Merges:            0,        0KiB  Write Merges:            0,
> 0KiB
>  IO unplugs:             0               Timer unplugs:           0
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Strange behavior

[CoRpO]

I've forgotten to mention that I've also put 
congested_read_threshold_us
and congested_write_threshold_us at 0 before posting here

I don't understand what's happening

Le 2013-03-15 16:42, Kent Overstreet a écrit :

> Can you try disabling congested_threshold? That's the one other 
> option
> that allows data to bypass the cache.