Re: dm-crypt + btrfs preformance - long lockups during io

Re: dm-crypt + btrfs preformance - long lockups during io

[Anders Aagaard]

I should point out this as well:

# hdparm -tT /dev/sda

/dev/sda:
 Timing cached reads:   21408 MB in  1.99 seconds = 10762.26 MB/sec
 Timing buffered disk reads: 1156 MB in  3.00 seconds = 385.28 MB/sec

# hdparm -tT /dev/mapper/sda4_crypt

/dev/mapper/sda4_crypt:
 Timing cached reads:   21212 MB in  1.99 seconds = 10663.44 MB/sec
 Timing buffered disk reads: 1218 MB in  3.00 seconds = 405.66 MB/sec

So it doesn't seem to be an ssd performance issue.


On Sat, Apr 5, 2014 at 10:54 AM, Anders Aagaard <aagaande@gmail.com> wrote:
> Hi
>
> I just recently repartitioned my harddrive, and in the process switched from
> ext4+ecryptfs to dm-crypt and btrfs. I'm on ubuntu 14.04, using kernel
> 3.14.0-031400-generic. I'm using a intel ssd, which btrfs detects (ssd mode
> enabled according to dmesg).
>
> On IO access I'm occasionally experiencing huge hangs, I can do something as
> simple as installing a small package using dpkg and get what appears to be a
> 30 second complete hardlock, and then the system continuous fine. Nothing
> interesting shows up in dmesg, but I tried using latencytop to figure out
> what was going on.
>
> Target "chrome":
> Opening file - 6000ms
> fsync on file - 5500ms
> synchronous write - 4000ms
>
> Target "btrfs-endio-wri" (of wihch there are 3):
> wait_current_trans.isra.34 - 5800ms (for the other 2 it's 5800 and 3000ms).
>
> Target "kworker/u16:7)"
> wait.current.trans.isra34 - 1100ms
>
> Target "btrfs-transaction":
> sleep_on_page - 370ms
>
> Some info:
> # btrfs subvolume list /
> ID 257 gen 8282 top level 5 path @
> ID 258 gen 8282 top level 5 path @home
>
> # mount
> /dev/mapper/sda4_crypt on /home type btrfs (rw,noatime,subvol=@home)
> /dev/mapper/sda4_crypt on / type btrfs (rw,noatime,subvol=@)
>
> # cat /sys/block/sda/queue/scheduler
> noop [deadline] cfq
>
> Any ideas on what could be going on here, and what I can do to fix and or
> debug the problem?
>
> Best regards
> Anders



-- 
Weeks of coding can save you hours of planning.
- http://www.codebox.no
- http://code.google.com/p/aagaande/

Re: dm-crypt + btrfs preformance - long lockups during io

[Tobias Grosser]

On Sat, Apr 5, 2014 at 10:54 AM, Anders Aagaard <aagaande@xxxxxxxxx> wrote:
 > Hi
 >
 > I just recently repartitioned my harddrive, and in the process
switched from
 > ext4+ecryptfs to dm-crypt and btrfs. I'm on ubuntu 14.04, using kernel
 > 3.14.0-031400-generic. I'm using a intel ssd, which btrfs detects
(ssd mode
 > enabled according to dmesg).

I also saw and still see lockups on linux 3.8 with a similar 
configuration. btrfs on dm-crypt with SSD mode. I use slightly different 
mount options:

/dev/mapper/sda4_crypt on /home type btrfs 
(rw,noatime,flushoncommit,subvol=@home)
/dev/mapper/sda4_crypt on /store type btrfs (rw,subvol=@store)
/dev/mapper/sda4_crypt on /home/grosser type btrfs 
(rw,noatime,flushoncommit,subvol=@grosser)

(I previously also used compression. This is disabled at the moment, but 
compressed files are still around)

latencytop gives the following information:


firefox: 22909.8 ms latency, cause fsync() on a file

The corresponding backtrace is:

btrfs_log_inode_parent
[btrfs]
btrfs_log_dentry_safe
[btrfs]
btrfs_sync_file
[btrfs]
do_fsync
sys_fsync
system_call_fastpath

There are also 6 processes called

btrfs-endio-wri?? with 1050-1391 ms latency due to [sleep_on_page].

The corresponding backtraces are (3x):

sleep_on_page
wait_on_page_bit
read_extent_buffer_pages
[btrfs]
btree_read_extend_buffer_pages.constprop.119
[btrfs]
read_tree_block
[btrfs]
read_node_slot
[btrfs]
push_leaf_right
[btrfs]
split_leaf
[btrfs]
btrfs_search_slot
[btrfs]
btrfs_csum_file_blocks
[btrfs]
add_pending_csums.isra.42
[btrfs]
btrfs_finish_ordered_io
[btrfs]

1x:

sleep_on_page
wait_on_page_bit
read_extent_buffer_pages
[btrfs]
btree_read_extend_buffer_pages.constprop.119
[btrfs]
read_tree_block
[btrfs]
read_block_for_search.isra.51
[btrfs]
btrfs_search_slot
[btrfs]
btrfs_insert_empty_items
[btrfs]
run_clustered_refs
[btrfs]
btrfs_run_delayed_refs
[btrfs]
__btrfs_end_transaction
[btrfs]
btfs_end_transaction
[btrfs]

1x
sleep_on_page
wait_on_page_bit
read_extent_buffer_pages
[btrfs]
btree_read_extend_buffer_pages.constprop.119
[btrfs]
read_tree_block
[btrfs]
read_node_slot
[btrfs]
push_leaf_left
[btrfs]
split_leaf
[btrfs]
btrfs_search_slot
[btrfs]
btrfs_insert_empty_items
[btrfs]
btrfs_csum_file_blocks
[btrfs]
add_pending_csums.isra.42
[btrfs]

1x
sleep_on_page
wait_on_page_bit
read_extent_buffer_pages
[btrfs]
btree_read_extend_buffer_pages.constprop.119
[btrfs]
read_tree_block
[btrfs]
read_node_slot
[btrfs]
push_leaf_right
[btrfs]
split_leaf
[btrfs]
btrfs_search_slot
[btrfs]
btrfs_csum_file_blocks
[btrfs]
add_pending_csums.isra.42
[btrfs]
btrfs_finish_ordered_io
[btrfs]

At the moment of deadlock, I am just running normal desktop applications 
like thunderbird and firefox.

Cheers,
Tobias

Any idea where such long delays may come from.