dm-cache: clean cache resyncs to origin on every boot

dm-cache: clean cache resyncs to origin on every boot

[Bogdan Vasiliev]

Hello,

I have dm-cache configuraion with SSD in writeback mode. On every boot
an entire cache resyncs to an origin device, even if it was clean
before on shutdown. It takes time and full bandwidth of an origin
device.

Is there reason for this behavior or is it a bug? Why we do not trust
a cache metadata?

My configuration details, if necessary:
storage-cache_cdata:0 41943040 linear
storage-cache_cmeta: 0 1048576 linear
storage-data: 0 1953513472 cache 8 989/131072 128 173945/327680 610488
20046 574861 3817439 0 0 0 1 writeback 2 migration_threshold 2048 mq
10 random_threshold 4 sequential_threshold 512
discard_promote_adjustment 1 read_promote_adjustment 4
write_promote_adjustment 8 rw -
storage-data_corig: 0 1953513472 linear


- Bogdan

Re: dm-cache: clean cache resyncs to origin on every boot

[Joe Thornber]

On Tue, Oct 20, 2015 at 11:51:07AM +0500, Bogdan Vasiliev wrote:
> Hello,
> 
> I have dm-cache configuraion with SSD in writeback mode. On every boot
> an entire cache resyncs to an origin device, even if it was clean
> before on shutdown. It takes time and full bandwidth of an origin
> device.
> 
> Is there reason for this behavior or is it a bug? Why we do not trust
> a cache metadata?

If the cache is not taken down cleanly then dm-cache has to assume
every block is dirty.  Make sure your shutdown scripts are removing
the cache device.

> 
> My configuration details, if necessary:
> storage-cache_cdata:0 41943040 linear
> storage-cache_cmeta: 0 1048576 linear
> storage-data: 0 1953513472 cache 8 989/131072 128 173945/327680 610488
> 20046 574861 3817439 0 0 0 1 writeback 2 migration_threshold 2048 mq
> 10 random_threshold 4 sequential_threshold 512
> discard_promote_adjustment 1 read_promote_adjustment 4
> write_promote_adjustment 8 rw -
> storage-data_corig: 0 1953513472 linear
> 
> 
> - Bogdan
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: dm-cache: clean cache resyncs to origin on every boot

[sbp]

On 10/20/2015 04:33 PM, Joe Thornber wrote:
> On Tue, Oct 20, 2015 at 11:51:07AM +0500, Bogdan Vasiliev wrote:
>> Hello,
>>
>> I have dm-cache configuraion with SSD in writeback mode. On every boot
>> an entire cache resyncs to an origin device, even if it was clean
>> before on shutdown. It takes time and full bandwidth of an origin
>> device.
>>
>> Is there reason for this behavior or is it a bug? Why we do not trust
>> a cache metadata?
> If the cache is not taken down cleanly then dm-cache has to assume
> every block is dirty.  Make sure your shutdown scripts are removing
> the cache device.

Hi,

I have the same problem, but also in writethrough mode. In principle the 
behavior is the same in writethrough and writeback modes: Cache is fully 
written back to disk every time the machine boots and sets the cache up.

I've posted a similar report to this list beginning of this month, but 
got no answer. There is my exact startup and shutdown script included, 
also some additional information about the setup. 
https://www.redhat.com/archives/dm-devel/2015-October/msg00012.html

>
>> My configuration details, if necessary:
>> storage-cache_cdata:0 41943040 linear
>> storage-cache_cmeta: 0 1048576 linear
>> storage-data: 0 1953513472 cache 8 989/131072 128 173945/327680 610488
>> 20046 574861 3817439 0 0 0 1 writeback 2 migration_threshold 2048 mq
>> 10 random_threshold 4 sequential_threshold 512
>> discard_promote_adjustment 1 read_promote_adjustment 4
>> write_promote_adjustment 8 rw -
>> storage-data_corig: 0 1953513472 linear
>>
>>
>> - Bogdan
>>
>> --
>> dm-devel mailing list
>> dm-devel@redhat.com
>> https://www.redhat.com/mailman/listinfo/dm-devel
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: dm-cache: clean cache resyncs to origin on every boot

[Bogdan Vasiliev]

On 20 October 2015 at 19:33, Joe Thornber <thornber@redhat.com> wrote:
> On Tue, Oct 20, 2015 at 11:51:07AM +0500, Bogdan Vasiliev wrote:
>> Hello,
>>
>> I have dm-cache configuraion with SSD in writeback mode. On every boot
>> an entire cache resyncs to an origin device, even if it was clean
>> before on shutdown. It takes time and full bandwidth of an origin
>> device.
>>
>> Is there reason for this behavior or is it a bug? Why we do not trust
>> a cache metadata?
>
> If the cache is not taken down cleanly then dm-cache has to assume
> every block is dirty.  Make sure your shutdown scripts are removing
> the cache device.

How I do it:
# umount /dev/mapper/storage-data
# lvs  -o lv_name, cache_dirty_blocks
  LV   CacheDirtyBlocks
  data                0
# sync
# dmsetup remove_all
# vgchange -ay
  1 logical volume(s) in volume group "storage" now active
# lvs  -o lv_name, cache_dirty_blocks
  LV   CacheDirtyBlocks
  data           122657

Volume was clearly removed without dirty cahe blocks, re-initialized
and began full re-sync to the origin. Is this normal behavior?


>> My configuration details, if necessary:
>> storage-cache_cdata:0 41943040 linear
>> storage-cache_cmeta: 0 1048576 linear
>> storage-data: 0 1953513472 cache 8 989/131072 128 173945/327680 610488
>> 20046 574861 3817439 0 0 0 1 writeback 2 migration_threshold 2048 mq
>> 10 random_threshold 4 sequential_threshold 512
>> discard_promote_adjustment 1 read_promote_adjustment 4
>> write_promote_adjustment 8 rw -
>> storage-data_corig: 0 1953513472 linear

Re: dm-cache: clean cache resyncs to origin on every boot

[Joe Thornber]

On Tue, Oct 20, 2015 at 11:06:07PM +0500, Bogdan Vasiliev wrote:
> Volume was clearly removed without dirty cahe blocks, re-initialized
> and began full re-sync to the origin. Is this normal behavior?

No, there is indeed a bug, fixed with the patch below.

Thanks,

- Joe


commit 7e0a9739fd9e72dc51490bdff92796400b12a34f
Author: Joe Thornber <ejt@redhat.com>
Date:   Thu Oct 22 18:10:55 2015 +0100

    [dm-cache] The CLEAN_SHUTDOWN flag was not being set
    
    or rather, it was being set on the disk, then immediately cleared again.
    
    If the CLEAN_SHUTDOWN flag is not set when a cache is loaded then all cache
    blocks are marked as dirty and a full writeback occurs.

diff --git a/drivers/md/dm-cache-metadata.c b/drivers/md/dm-cache-metadata.c
index e956c44..2073d30 100644
--- a/drivers/md/dm-cache-metadata.c
+++ b/drivers/md/dm-cache-metadata.c
@@ -634,10 +634,11 @@ static int __commit_transaction(struct dm_cache_metadata *cmd,
 
        disk_super = dm_block_data(sblock);
 
+       disk_super->flags = cpu_to_le32(cmd->flags);
+
        if (mutator)
                update_flags(disk_super, mutator);
 
-       disk_super->flags = cpu_to_le32(cmd->flags);
        disk_super->mapping_root = cpu_to_le64(cmd->root);
        disk_super->hint_root = cpu_to_le64(cmd->hint_root);
        disk_super->discard_root = cpu_to_le64(cmd->discard_root);