blog post on fio write zeroes performance

blog post on fio write zeroes performance

[Vincent Fu]

I am pleased to share a blog post about fio write zeroes performance:

https://github.com/vincentkfu/fio-blog/wiki/How-could-write-zeroes-performance-be-worse-at-higher-queue-depths

Read about the unexpected performance issue we encountered. Feedback is 
welcome.

Vincent

Re: blog post on fio write zeroes performance

[Adam Horshack]

Nice writeup Vincent. It's been a while since I've looked at the NVMe spec so perhaps I'm asking the wrong question but why is fio managing a data buffer for its handling of the Write Zero command? I would've assumed there's no data payload associated with that NVMe command, thus nothing that should even be subject to internal buffer scrambling inside fio.

Regards,

Adam


________________________________________
From: Vincent Fu <vincentfu@gmail.com>
Sent: Tuesday, November 5, 2024 2:41 PM
To: fio@vger.kernel.org
Cc: gost.dev@samsung.com
Subject: blog post on fio write zeroes performance

I am pleased to share a blog post about fio write zeroes performance:

https://github.com/vincentkfu/fio-blog/wiki/How-could-write-zeroes-performance-be-worse-at-higher-queue-depths

Read about the unexpected performance issue we encountered. Feedback is
welcome.

Vincent

Re: blog post on fio write zeroes performance

[Vincent Fu]

On 11/5/24 15:02, Adam Horshack wrote:
> Nice writeup Vincent. It's been a while since I've looked at the NVMe spec so perhaps I'm asking the wrong question but why is fio managing a data buffer for its handling of the Write Zero command? I would've assumed there's no data payload associated with that NVMe command, thus nothing that should even be subject to internal buffer scrambling inside fio.
> 
> Regards,
> 
> Adam
> 
> 
> ________________________________________
> From: Vincent Fu <vincentfu@gmail.com>
> Sent: Tuesday, November 5, 2024 2:41 PM
> To: fio@vger.kernel.org
> Cc: gost.dev@samsung.com
> Subject: blog post on fio write zeroes performance
> 
> I am pleased to share a blog post about fio write zeroes performance:
> 
> https://github.com/vincentkfu/fio-blog/wiki/How-could-write-zeroes-performance-be-worse-at-higher-queue-depths
> 
> Read about the unexpected performance issue we encountered. Feedback is
> welcome.
> 
> Vincent
> 
> 

When we added write zeroes support for the io_uring_cmd ioengine it was 
not on our mind that the default behavior for fio is to scramble 
buffers. Basically these patches just changed the opcode from a regular 
write to write zeroes.

You do have a good point that perhaps fio should be smart enough to 
avoid touching the buffers when doing write zeroes.

Thanks for the feedback!

Vincent

RE: blog post on fio write zeroes performance

[Terekhov, Mikhail]

Internal Use - Confidential
> From: Vincent Fu <vincentfu@gmail.com>
> Sent: Tuesday, November 5, 2024 3:37 PM
> To: Adam Horshack <horshack@live.com>; fio@vger.kernel.org
> Subject: Re: blog post on fio write zeroes performance
>
>
> When we added write zeroes support for the io_uring_cmd ioengine it was not on
> our mind that the default behavior for fio is to scramble buffers. Basically these
> patches just changed the opcode from a regular write to write zeroes.
>
> You do have a good point that perhaps fio should be smart enough to avoid
> touching the buffers when doing write zeroes.
>
> Thanks for the feedback!
>
> Vincent

Does it mean that in the regular case (not write zeroes) fio spends significant time scrambling buffers and IO performance is lower than it could be?

Regards,
Mikhail

Re: blog post on fio write zeroes performance

[Vincent Fu]

On 11/5/24 15:45, Terekhov, Mikhail wrote:
> 
> Internal Use - Confidential
>> From: Vincent Fu <vincentfu@gmail.com>
>> Sent: Tuesday, November 5, 2024 3:37 PM
>> To: Adam Horshack <horshack@live.com>; fio@vger.kernel.org
>> Subject: Re: blog post on fio write zeroes performance
>>
>>
>> When we added write zeroes support for the io_uring_cmd ioengine it was not on
>> our mind that the default behavior for fio is to scramble buffers. Basically these
>> patches just changed the opcode from a regular write to write zeroes.
>>
>> You do have a good point that perhaps fio should be smart enough to avoid
>> touching the buffers when doing write zeroes.
>>
>> Thanks for the feedback!
>>
>> Vincent
> 
> Does it mean that in the regular case (not write zeroes) fio spends significant time scrambling buffers and IO performance is lower than it could be?
> 
> Regards,
> Mikhail

For regular writes on contemporary devices the time that fio spends 
scrambling buffers is small compared to the device response time.

However, if you want the absolute highest write performance or are 
really pushing the limits with a device, it might be worth running the 
job with zero_buffers or scramble_buffers=0 to see what difference it makes.

Vincent

Re: blog post on fio write zeroes performance

[Sitsofe Wheeler]

On Tue, 5 Nov 2024 at 21:33, Vincent Fu <vincentfu@gmail.com> wrote:
>
> On 11/5/24 15:45, Terekhov, Mikhail wrote:
> >
> > Does it mean that in the regular case (not write zeroes) fio spends significant time scrambling buffers and IO performance is lower than it could be?
> >
> > Regards,
> > Mikhail
>
> For regular writes on contemporary devices the time that fio spends
> scrambling buffers is small compared to the device response time.
>
> However, if you want the absolute highest write performance or are
> really pushing the limits with a device, it might be worth running the
> job with zero_buffers or scramble_buffers=0 to see what difference it makes.

It's worth noting that even in the "not write zeros dedicated command"
case some devices (or things in the I/O path like filesystems etc.)
can detect or compress buffers of zeros and thus are able to achieve
(unrealistic) higher speeds that they cannot reach when "real" data is
used...


--
Sitsofe