Re: [PATCH 0/3] Create large folios in iomap buffered write path

[Wang Yugui <wangyugui@e16-tech.com>]

Hi,

> On Sun, May 21, 2023 at 08:49:53AM +0800, Wang Yugui wrote:
> > Hi,
> > 
> > > Wang Yugui has a workload which would be improved by using large folios.
> > > Until now, we've only created large folios in the readahead path,
> > > but this workload writes without reading.  The decision of what size
> > > folio to create is based purely on the size of the write() call (unlike
> > > readahead where we keep history and can choose to create larger folios
> > > based on that history even if individual reads are small).
> > > 
> > > The third patch looks like it's an optional extra but is actually needed
> > > for the first two patches to work in the write path, otherwise it limits
> > > the length that iomap_get_folio() sees to PAGE_SIZE.
> > 
> > very good test result on 6.4.0-rc2. 
> > # just drop ';' in 'if (bytes > folio_size(folio) - offset);' of [PATCH 3/3].
> > 
> > fio -name write-bandwidth -rw=write -bs=1024Ki -size=32Gi -runtime=30 -iodepth 1
> > -ioengine sync -zero_buffers=1 -direct=0 -end_fsync=1 -numjobs=4
> > -directory=/mnt/test
> > fio WRITE: bw=7655MiB/s (8027MB/s).
> > 
> > Now it is the same as 5.15.y
> 
> Great!  How close is that to saturating the theoretical write bandwidth
> of your storage?

SSD:  PCIe3  SSD U.2 *4 (1.6T/3.2T/3.2T/3.2T, just 800G used) through a NVMe
adapater.
CPU: E5-2680 v2 @ 2.80GHz *2
memory: DDR3 * 3 channel

so theoretical write bandwidth maybe over 12GB/s,
but 8GB/s maybe the in fact bandwidth because of 
- write depth limit
- NVMe adapther limit
- CPU/memory limit

I also noticed a huge improvement for single thread dd.

# 6.4.0-rc2 with this patch
#dd-9.1 conv=fsync bs=1024K count=32K if=/dev/zero of=/mnt/test/dd.txt
34359738368 bytes (34 GB, 32 GiB) copied, 6.96108 s, 4.6 GiB/s

But it is about 2.2GiB/s in xfs/5.15.y.

Best Regards
Wang Yugui (wangyugui@e16-tech.com)
2023/05/21