Re: [BUG] active zones exceeded error with max_open_zones

[Damien Le Moal <dlemoal@kernel.org>]

On 4/24/25 02:11, Sean Anderson wrote:
> Hi,
> 
> I'm getting an "active zones exceeded" error when running fio with
> --rw=randwrite mode:
> 
> # fio --bs=4k --rw=randwrite --norandommap --fsync=1 --number_ios=16384 --name=flushes --direct=1 --zonemode=zbd --max_open_zones=1978 --filename=/dev/my_zone_dev

--max_open_zones=1978 is an extremely large value that likely exceeds the drive
capabilities, which is what fio is telling you.
What are your drive maximum open and active zones limits ?

cat /sys/block/my_zone_dev/queue/max_open_zones
cat /sys/block/my_zone_dev/queue/max_active_zones

fio will use the min_not_zero of these 2 values as the maximum number of zones
that can be written simultaneously. Especially if your drive has an active zone
limit, you *cannot* write to more zones than that limit at the same time.
fio will default to max_open_zones=min_not_zero(drive max open, drive max
active) and for a random write workload, it will:
- pick zones randomly up to max_open_zones
- direct write IOs to a randomly chosen zone in the current set of open zones
and when an open zone becomes full, randomly pick another zone to replace it.

For your workload, if you want to measure the maximum "random" write performance
of your disk, simply do NOT specify --max_open_zones=. fio will pick the best
possible number for you.


> flushes: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
> fio-3.39
> Starting 1 process
> active zones exceeded error, dev my_zone_dev, sector 189520 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0
> fio: io_u error on file /dev/my_zone_dev: Value too large for defined data type: write offset=97034240, buflen=4096
> /dev/my_zone_dev: Exceeded max_active_zones limit. Check conditions of zones out of I/O ranges.
> fio: pid=2549, err=75/file:io_u.c:1976, func=io_u error, error=Value too large for defined data type
> 
> flushes: (groupid=0, jobs=1): err=75 (file:io_u.c:1976, func=io_u error, error=Value too large for defined data type): pid=2549: Wed Apr 23 17:01:03 2025
>    write: IOPS=262, BW=1050KiB/s (1075kB/s)(9092KiB/8661msec); 0 zone resets
>      clat (usec): min=983, max=20564, avg=3645.67, stdev=4347.94
>       lat (usec): min=984, max=20564, avg=3645.75, stdev=4347.94
>      clat percentiles (usec):
>       |  1.00th=[  996],  5.00th=[ 1012], 10.00th=[ 1029], 20.00th=[ 1418],
>       | 30.00th=[ 1434], 40.00th=[ 1434], 50.00th=[ 1450], 60.00th=[ 1450],
>       | 70.00th=[ 1467], 80.00th=[ 5669], 90.00th=[12256], 95.00th=[12780],
>       | 99.00th=[15008], 99.50th=[15533], 99.90th=[16712], 99.95th=[17171],
>       | 99.99th=[20579]
>     bw (  KiB/s): min=  500, max= 1205, per=100.00%, avg=1052.88, stdev=195.04, samples=17
>     iops        : min=  125, max=  301, avg=262.88, stdev=48.79, samples=17
>    lat (usec)   : 1000=1.76%
>    lat (msec)   : 2=74.05%, 4=1.10%, 10=4.75%, 20=18.25%, 50=0.04%
>    fsync/fdatasync/sync_file_range:
>      sync (usec): min=50, max=11641, avg=160.03, stdev=798.31
>      sync percentiles (usec):
>       |  1.00th=[   53],  5.00th=[   57], 10.00th=[   66], 20.00th=[   73],
>       | 30.00th=[   81], 40.00th=[   82], 50.00th=[   83], 60.00th=[   84],
>       | 70.00th=[   85], 80.00th=[   87], 90.00th=[  178], 95.00th=[  208],
>       | 99.00th=[  603], 99.50th=[ 1549], 99.90th=[11600], 99.95th=[11600],
>       | 99.99th=[11600]
>    cpu          : usr=0.00%, sys=49.31%, ctx=2823, majf=0, minf=181
>    IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
>       submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>       complete  : 0=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>       issued rwts: total=0,2274,0,2273 short=0,0,0,0 dropped=0,0,0,0
>       latency   : target=0, window=0, percentile=100.00%, depth=1
> 
> Run status group 0 (all jobs):
>    WRITE: bw=1050KiB/s (1075kB/s), 1050KiB/s-1050KiB/s (1075kB/s-1075kB/s), io=9092KiB (9310kB), run=8661-8661msec
> 
> Disk stats (read/write):
>    my_zone_dev: ios=170/4498, sectors=1336/17992, merge=0/0, ticks=0/118, in_queue=230, util=47.80%
> 
> The issue seems to be that fio writes to a bunch of zones but never
> finishes them because they're not full yet:
> 
> # blkzone report -c 16 /dev/my_block_dev
>    start: 0x000000000, len 0x000020, cap 0x00001f, wptr 0x000000 reset:0 non-seq:0, zcond: 1(em) [type: 2(SEQ_WRITE_REQUIRED)]
>    start: 0x000000020, len 0x000020, cap 0x00001f, wptr 0x000000 reset:0 non-seq:0, zcond: 1(em) [type: 2(SEQ_WRITE_REQUIRED)]
>    start: 0x000000040, len 0x000020, cap 0x00001f, wptr 0x000008 reset:0 non-seq:0, zcond: 4(cl) [type: 2(SEQ_WRITE_REQUIRED)]
>    start: 0x000000060, len 0x000020, cap 0x00001f, wptr 0x000000 reset:0 non-seq:0, zcond: 1(em) [type: 2(SEQ_WRITE_REQUIRED)]
>    start: 0x000000080, len 0x000020, cap 0x00001f, wptr 0x000000 reset:0 non-seq:0, zcond: 1(em) [type: 2(SEQ_WRITE_REQUIRED)]
>    start: 0x0000000a0, len 0x000020, cap 0x00001f, wptr 0x000000 reset:0 non-seq:0, zcond: 1(em) [type: 2(SEQ_WRITE_REQUIRED)]
>    start: 0x0000000c0, len 0x000020, cap 0x00001f, wptr 0x000000 reset:0 non-seq:0, zcond: 1(em) [type: 2(SEQ_WRITE_REQUIRED)]
>    start: 0x0000000e0, len 0x000020, cap 0x00001f, wptr 0x000000 reset:0 non-seq:0, zcond: 1(em) [type: 2(SEQ_WRITE_REQUIRED)]
>    start: 0x000000100, len 0x000020, cap 0x00001f, wptr 0x000008 reset:0 non-seq:0, zcond: 4(cl) [type: 2(SEQ_WRITE_REQUIRED)]
>    start: 0x000000120, len 0x000020, cap 0x00001f, wptr 0x000008 reset:0 non-seq:0, zcond: 4(cl) [type: 2(SEQ_WRITE_REQUIRED)]
>    start: 0x000000140, len 0x000020, cap 0x00001f, wptr 0x000008 reset:0 non-seq:0, zcond: 4(cl) [type: 2(SEQ_WRITE_REQUIRED)]
>    start: 0x000000160, len 0x000020, cap 0x00001f, wptr 0x000000 reset:0 non-seq:0, zcond: 1(em) [type: 2(SEQ_WRITE_REQUIRED)]
>    start: 0x000000180, len 0x000020, cap 0x00001f, wptr 0x000000 reset:0 non-seq:0, zcond: 1(em) [type: 2(SEQ_WRITE_REQUIRED)]
>    start: 0x0000001a0, len 0x000020, cap 0x00001f, wptr 0x000010 reset:0 non-seq:0, zcond: 4(cl) [type: 2(SEQ_WRITE_REQUIRED)]
>    start: 0x0000001c0, len 0x000020, cap 0x00001f, wptr 0x000000 reset:0 non-seq:0, zcond: 1(em) [type: 2(SEQ_WRITE_REQUIRED)]
>    start: 0x0000001e0, len 0x000020, cap 0x00001f, wptr 0x000000 reset:0 non-seq:0, zcond: 1(em) [type: 2(SEQ_WRITE_REQUIRED)]
> 
> This issue doesn't seem to occur with --rw=write because sequential
> writes fill up zones and they get finished automatically.
> 
> --Sean


-- 
Damien Le Moal
Western Digital Research