Question about an fio profile which leads to a CPU lockup

Question about an fio profile which leads to a CPU lockup

[Haris Iqbal]

Hello,

I am running the following fio profile on an RDMA storage stack with a
raid5 in the back end, and I see a CPU lockup. (I can provide more
details about the setup if needed).

""
[global]
description=Emulation of Storage Server Access Pattern
bssplit=512/20:1k/16:2k/9:4k/12:8k/19:16k/10:32k/8:64k/4
fadvise_hint=0
rw=randrw
direct=1
random_distribution=zipf:1.2
time_based=1
runtime=60
ramp_time=1
ioengine=libaio
iodepth=128
iodepth_batch_submit=128
iodepth_batch_complete_min=1
iodepth_batch_complete_max=128
numjobs=1
group_reporting

[job1]
filename=/dev/rnbd0
""

First question I have is, is there wrong with the above profile?
Next, if I simply change the iodepth_batch_complete_min=0 in the above
profile, I see no hangup.

My next step is to reach out to the kernel raid group, but before that
I wanted to confirm that the fio profile leading to the hangup isn't
incorrect/illegal or something.

Regards
-Haris

Re: Question about an fio profile which leads to a CPU lockup

[Vincent Fu]

On 10/4/24 11:09, Haris Iqbal wrote:
> Hello,
> 
> I am running the following fio profile on an RDMA storage stack with a
> raid5 in the back end, and I see a CPU lockup. (I can provide more
> details about the setup if needed).
> 
> ""
> [global]
> description=Emulation of Storage Server Access Pattern
> bssplit=512/20:1k/16:2k/9:4k/12:8k/19:16k/10:32k/8:64k/4
> fadvise_hint=0
> rw=randrw
> direct=1
> random_distribution=zipf:1.2
> time_based=1
> runtime=60
> ramp_time=1
> ioengine=libaio
> iodepth=128
> iodepth_batch_submit=128
> iodepth_batch_complete_min=1
> iodepth_batch_complete_max=128
> numjobs=1
> group_reporting
> 
> [job1]
> filename=/dev/rnbd0
> ""
> 
> First question I have is, is there wrong with the above profile?
> Next, if I simply change the iodepth_batch_complete_min=0 in the above
> profile, I see no hangup.
> 
> My next step is to reach out to the kernel raid group, but before that
> I wanted to confirm that the fio profile leading to the hangup isn't
> incorrect/illegal or something.
> 
> Regards
> -Haris
> 

There is nothing obviously wrong with your job options. Try running it 
with --debug=io to see what is going on. There will be a lot of output 
since there will be multiple debug messages for each I/O.

Note that, from the documentation, if iodepth_batch_complete_min=0 then 
fio will always check for completed events before queuing more I/O.

Vincent