Re: [PATCH v2] bcache: set max writeback rate when I/O request is idle

[Coly Li <colyli@suse.de>]

On 2018/7/26 1:44 PM, Stefan Priebe - Profihost AG wrote:
>> Am 25.07.2018 um 08:16 schrieb Stefan Priebe - Profihost AG <s.priebe@profihost.ag>:
>>
>>> Am 24.07.2018 um 18:36 schrieb Coly Li:
>>>> On 2018/7/24 4:33 PM, Stefan Priebe - Profihost AG wrote:
>>>>> Am 24.07.2018 um 10:28 schrieb Coly Li:
>>>>>> On 2018/7/24 3:16 PM, Stefan Priebe - Profihost AG wrote:
>>>>>>> Am 24.07.2018 um 06:03 schrieb Coly Li:
>>>>>>> Commit b1092c9af9ed ("bcache: allow quick writeback when backing idle")
>>>>>>> allows the writeback rate to be faster if there is no I/O request on a
>>>>>>> bcache device. It works well if there is only one bcache device attached
>>>>>>> to the cache set. If there are many bcache devices attached to a cache
>>>>>>> set, it may introduce performance regression because multiple faster
>>>>>>> writeback threads of the idle bcache devices will compete the btree level
>>>>>>> locks with the bcache device who have I/O requests coming.
>>>>>>>
>>>>>>> This patch fixes the above issue by only permitting fast writebac when
>>>>>>> all bcache devices attached on the cache set are idle. And if one of the
>>>>>>> bcache devices has new I/O request coming, minimized all writeback
>>>>>>> throughput immediately and let PI controller __update_writeback_rate()
>>>>>>> to decide the upcoming writeback rate for each bcache device.
>>>>>>>
>>>>>>> Also when all bcache devices are idle, limited wrieback rate to a small
>>>>>>> number is wast of thoughput, especially when backing devices are slower
>>>>>>> non-rotation devices (e.g. SATA SSD). This patch sets a max writeback
>>>>>>> rate for each backing device if the whole cache set is idle. A faster
>>>>>>> writeback rate in idle time means new I/Os may have more available space
>>>>>>> for dirty data, and people may observe a better write performance then.
>>>>>>>
>>>>>>> Please note bcache may change its cache mode in run time, and this patch
>>>>>>> still works if the cache mode is switched from writeback mode and there
>>>>>>> is still dirty data on cache.
>>>>>>>
>>>>>>> Fixes: Commit b1092c9af9ed ("bcache: allow quick writeback when backing idle")
>>>>>>> Cc: stable@vger.kernel.org #4.16+
>>>>>>> Signed-off-by: Coly Li <colyli@suse.de>
>>>>>>> Tested-by: Kai Krakow <kai@kaishome.de>
>>>>>>> Cc: Michael Lyle <mlyle@lyle.org>
>>>>>>> Cc: Stefan Priebe <s.priebe@profihost.ag>
>>>>>>
>>>>>> Hi Coly,
>>>>>>
>>>>>> i'm still experiencing the same bug as yesterday while rebooting every
>>>>>> two times i get a deadlock in cache_set_free.
>>>>>>
>>>>>> Sadly it's so late ion the shutdown process that netconsole is already
>>>>>> unloaded or at least i get no messages from while it happens. The traces
>>>>>> look the same like yesterday.
>>>>>
>>>>> Hi Stefan,
>>>>>
>>>>> Do you use the v2 patch on latest SLE15 kernel source?
>>>>
>>>> Yes - i use the kernel code from here:
>>>> https://github.com/openSUSE/kernel-source/commits/SLE15
>>>>
>>>>> Let me try to
>>>>> reproduce it on my machine. I assume when you reboot, the cache is still
>>>>> dirty and not clean up, am I right ?
>>>>
>>>> Yes with around 15GB of dirty data - ceph is running on top of it and
>>>> generated many random writes.
>>>>
>>>>> And which file system do you mount
>>>>> on top of the bcache device ?
>>>> XFS
>>>
>>> Hi Stefan,
>>>
>>> Thanks for the information. I managed to reproduce a similar deadlock on
>>> my small box. Let me see how to fix it and post a new version ASAP :-)
>>
>> Great!
> 
> Any news on this already?

I am testing the new version, and it also requires other patches I
posted earlier for 4.19 (which gets rid of bch_register_lock in
writeback thread).

Hope I can make it today :-)

Coly Li