Re: [PATCH v3] f2fs: introduce flush_policy sysfs entry

Re: [PATCH v3] f2fs: introduce flush_policy sysfs entry

[Christoph Hellwig]

On Thu, Aug 07, 2025 at 11:48:38AM +0800, Chao Yu wrote:
> This patch introduces a new sysfs entry /sys/fs/f2fs/<disk>/flush_policy
> in order to tune performance of f2fs data flush flow.
> 
> For example, checkpoint will use REQ_FUA to persist CP metadata, however,
> some kind device has bad performance on REQ_FUA command, result in that
> checkpoint being blocked for long time, w/ this sysfs entry, we can give
> an option to use REQ_PREFLUSH command instead of REQ_FUA during checkpoint,
> it can help to mitigate long latency of checkpoint.

That's and odd place to deal with this.  If that's a real issue it
should be a block layer tweak to disable FUA, potentially with a quirk
entry in the driver to disable it rather than having to touch a file
system sysfs attribute.

Re: [PATCH v3] f2fs: introduce flush_policy sysfs entry

[Bart Van Assche]

On 8/11/25 3:52 AM, Christoph Hellwig wrote:
> On Thu, Aug 07, 2025 at 11:48:38AM +0800, Chao Yu wrote:
>> This patch introduces a new sysfs entry /sys/fs/f2fs/<disk>/flush_policy
>> in order to tune performance of f2fs data flush flow.
>>
>> For example, checkpoint will use REQ_FUA to persist CP metadata, however,
>> some kind device has bad performance on REQ_FUA command, result in that
>> checkpoint being blocked for long time, w/ this sysfs entry, we can give
>> an option to use REQ_PREFLUSH command instead of REQ_FUA during checkpoint,
>> it can help to mitigate long latency of checkpoint.
> 
> That's and odd place to deal with this.  If that's a real issue it
> should be a block layer tweak to disable FUA, potentially with a quirk
> entry in the driver to disable it rather than having to touch a file
> system sysfs attribute.

Chao, two years ago Christoph already suggested to integrate this
functionality in the UFS driver. From
https://lore.kernel.org/linux-scsi/Y+NCDzvuLJYGwyhC@infradead.org/:
"Please add quirks for the actually affected devices, and do not
block fua for an entire transport."

See also the ufs_fixups[] array in drivers/ufs/core/ufshcd.c.

Bart.

Re: [PATCH v3] f2fs: introduce flush_policy sysfs entry

[Chao Yu]

On 8/11/25 18:52, Christoph Hellwig wrote:
> On Thu, Aug 07, 2025 at 11:48:38AM +0800, Chao Yu wrote:
>> This patch introduces a new sysfs entry /sys/fs/f2fs/<disk>/flush_policy
>> in order to tune performance of f2fs data flush flow.
>>
>> For example, checkpoint will use REQ_FUA to persist CP metadata, however,
>> some kind device has bad performance on REQ_FUA command, result in that
>> checkpoint being blocked for long time, w/ this sysfs entry, we can give
>> an option to use REQ_PREFLUSH command instead of REQ_FUA during checkpoint,
>> it can help to mitigate long latency of checkpoint.
> 
> That's and odd place to deal with this.  If that's a real issue it
> should be a block layer tweak to disable FUA, potentially with a quirk
> entry in the driver to disable it rather than having to touch a file
> system sysfs attribute.

Okay, it makes sense to control how FUA be handled inside block layer, so
let's drop this patch.

BTW, I suffered extremely long latency of checkpoint which may block every
update operations when testing generic/299 w/ mode=lfs mount option in qemu,
then I propose to use PREFLUSH instead of FUA to resolve this issue.

"F2FS-fs (vdc): checkpoint was blocked for 24495 ms"

I just realize that using cache=directsync option in qemu can avoid FUA hang
issue, anyway, let me test more w/ this option.

Thanks,

Re: [PATCH v3] f2fs: introduce flush_policy sysfs entry

[Chao Yu]

On 8/11/25 21:44, Bart Van Assche wrote:
> On 8/11/25 3:52 AM, Christoph Hellwig wrote:
>> On Thu, Aug 07, 2025 at 11:48:38AM +0800, Chao Yu wrote:
>>> This patch introduces a new sysfs entry /sys/fs/f2fs/<disk>/flush_policy
>>> in order to tune performance of f2fs data flush flow.
>>>
>>> For example, checkpoint will use REQ_FUA to persist CP metadata, however,
>>> some kind device has bad performance on REQ_FUA command, result in that
>>> checkpoint being blocked for long time, w/ this sysfs entry, we can give
>>> an option to use REQ_PREFLUSH command instead of REQ_FUA during checkpoint,
>>> it can help to mitigate long latency of checkpoint.
>>
>> That's and odd place to deal with this.  If that's a real issue it
>> should be a block layer tweak to disable FUA, potentially with a quirk
>> entry in the driver to disable it rather than having to touch a file
>> system sysfs attribute.
> 
> Chao, two years ago Christoph already suggested to integrate this
> functionality in the UFS driver. From
> https://lore.kernel.org/linux-scsi/Y+NCDzvuLJYGwyhC@infradead.org/:
> "Please add quirks for the actually affected devices, and do not
> block fua for an entire transport."
> 
> See also the ufs_fixups[] array in drivers/ufs/core/ufshcd.c.

Bart, thank you for letting me know the history and decision there. I had a
qemu option here to resolve my current issue, thanks.

Thanks,

> 
> Bart.

Re: [PATCH v3] f2fs: introduce flush_policy sysfs entry

[Christoph Hellwig]

On Tue, Aug 12, 2025 at 02:28:46PM +0800, Chao Yu wrote:
> BTW, I suffered extremely long latency of checkpoint which may block every
> update operations when testing generic/299 w/ mode=lfs mount option in qemu,
> then I propose to use PREFLUSH instead of FUA to resolve this issue.
> 
> "F2FS-fs (vdc): checkpoint was blocked for 24495 ms"
> 
> I just realize that using cache=directsync option in qemu can avoid FUA hang
> issue, anyway, let me test more w/ this option.

Well, for decent qemu performance you always want to use DIRECT I/O.
directsync is generally not a very good idea as it forces every write
to be synchronous and will give you very bad performance.

What did you use before?  At least for older qemu the default was
buffered I/O, which can lead to very expensive fua or flush calls.

Re: [PATCH v3] f2fs: introduce flush_policy sysfs entry

[Chao Yu]

On 8/12/25 15:32, Christoph Hellwig wrote:
> On Tue, Aug 12, 2025 at 02:28:46PM +0800, Chao Yu wrote:
>> BTW, I suffered extremely long latency of checkpoint which may block every
>> update operations when testing generic/299 w/ mode=lfs mount option in qemu,
>> then I propose to use PREFLUSH instead of FUA to resolve this issue.
>>
>> "F2FS-fs (vdc): checkpoint was blocked for 24495 ms"
>>
>> I just realize that using cache=directsync option in qemu can avoid FUA hang
>> issue, anyway, let me test more w/ this option.
> 
> Well, for decent qemu performance you always want to use DIRECT I/O.
> directsync is generally not a very good idea as it forces every write
> to be synchronous and will give you very bad performance.

Yeah, I think that may hurt the performance too, at least, I don't see
any obvious change for time cost of generic/299 testcases, but still I
need to run all my testcase to see what will happen. :)

generic/299 115s ...  113s

> 
> What did you use before?  At least for older qemu the default was
> buffered I/O, which can lead to very expensive fua or flush calls.

Previously, I didn't use any cache= option, as manual described, it
should equal to cache=wrteback.

Thanks,

Re: [PATCH v3] f2fs: introduce flush_policy sysfs entry

[Christoph Hellwig]

On Tue, Aug 12, 2025 at 03:53:54PM +0800, Chao Yu wrote:
> > What did you use before?  At least for older qemu the default was
> > buffered I/O, which can lead to very expensive fua or flush calls.
> 
> Previously, I didn't use any cache= option, as manual described, it
> should equal to cache=wrteback.

Modern qemu actually split the cache option.  You absolute want
cache.direct=on.  If you don't do simulated power fail testing by
killing qemu (or run real workloads for the matter, but who does that
:)) it might make sense to just ignore the flushes with cache.no-flush=on
as well, which is what I do for my test VMs on the laptop.

Re: [PATCH v3] f2fs: introduce flush_policy sysfs entry

[Chao Yu]

On 8/12/25 15:59, Christoph Hellwig wrote:
> On Tue, Aug 12, 2025 at 03:53:54PM +0800, Chao Yu wrote:
>>> What did you use before?  At least for older qemu the default was
>>> buffered I/O, which can lead to very expensive fua or flush calls.
>>
>> Previously, I didn't use any cache= option, as manual described, it
>> should equal to cache=wrteback.
> 
> Modern qemu actually split the cache option.  You absolute want
> cache.direct=on.  If you don't do simulated power fail testing by

Yes,

> killing qemu (or run real workloads for the matter, but who does that
> :)) it might make sense to just ignore the flushes with cache.no-flush=on

Yes, I don't care whether data can be persisted to host devices or not,
nor killing qemu for test, so cache.no-flush=on looks good to me as well.

> as well, which is what I do for my test VMs on the laptop.

Thanks for sharing this, it helps. :)

Thanks,