fio output percentile latency seems upper bounded

fio output percentile latency seems upper bounded

[ylhsieh]

Hi, there

In fio output, sometimes I see tail percentile latency to be 17112760320, 
and it remains this value to 99.99%, why? Also, the mean latency is larger 
than this value? (intuitively, tail latency should be larger than mean 
latency...?) What's the reason and how to make it reasonable?

"clat_ns" : {
  ...
  "mean" : 41081227631.728256,
  ...
  "percentile" : {
    "1.000000" : 1669332992,
    "5.000000" : 10536091648,
    "10.000000" : 17112760320,
    "20.000000" : 17112760320,
    ...
    "99.950000" : 17112760320,
    "99.990000" : 17112760320
  }
},
"lat_ns" : {
  ...
  "mean" : 41087031313.177292,
  ...
}

Thank you
YL


CONFIDENTIALITY NOTE:

This e-mail and any attachments may contain confidential information 
and/or personal data, which is protected by applicable laws. Please be 
reminded that duplication, disclosure, distribution, or use of this e-mail 
(and/or its attachments) or any part thereof is prohibited. If you receive 
this e-mail in error, please notify us immediately and delete this mail as 
well as its attachment(s) from your system. In addition, please be 
informed that collection, processing, and/or use of personal data is 
prohibited unless expressly permitted by personal data protection laws. 
Thank you for your attention and cooperation.

Macronix International Co., Ltd.

=====================================================================

Re: fio output percentile latency seems upper bounded

[Vincent Fu]

On 4/16/24 04:55, ylhsieh@mxic.com.tw wrote:
> Hi, there
> 
> In fio output, sometimes I see tail percentile latency to be 17112760320,
> and it remains this value to 99.99%, why? Also, the mean latency is larger
> than this value? (intuitively, tail latency should be larger than mean
> latency...?) What's the reason and how to make it reasonable?
> 
> "clat_ns" : {
>    ...
>    "mean" : 41081227631.728256,
>    ...
>    "percentile" : {
>      "1.000000" : 1669332992,
>      "5.000000" : 10536091648,
>      "10.000000" : 17112760320,
>      "20.000000" : 17112760320,
>      ...
>      "99.950000" : 17112760320,
>      "99.990000" : 17112760320
>    }
> },
> "lat_ns" : {
>    ...
>    "mean" : 41087031313.177292,
>    ...
> }
> 
> Thank you
> YL
> 
> 
> CONFIDENTIALITY NOTE:
> 
> This e-mail and any attachments may contain confidential information
> and/or personal data, which is protected by applicable laws. Please be
> reminded that duplication, disclosure, distribution, or use of this e-mail
> (and/or its attachments) or any part thereof is prohibited. If you receive
> this e-mail in error, please notify us immediately and delete this mail as
> well as its attachment(s) from your system. In addition, please be
> informed that collection, processing, and/or use of personal data is
> prohibited unless expressly permitted by personal data protection laws.
> Thank you for your attention and cooperation.
> 
> Macronix International Co., Ltd.
> 
> =====================================================================
> 

Yes, there is an upper bound. See the discussion in stat.h:

https://github.com/axboe/fio/blob/master/stat.h

The intention is to calculate the min, max, and mean without loss of 
precision. But for the latency percentiles, each latency measurement is 
assigned to a bucket that includes a range of values. The midpoint of 
the bucket for the highest set of values is 17112760320 ns. Values 
exceeding this are not tracked accurately.

You could increase FIO_IO_U_PLAT_GROUP_NR if you wanted to track long 
latencies more accurately.

Vincent