Re: Creating large folios in iomap buffered write path

[Matthew Wilcox <willy@infradead.org>]

On Fri, May 19, 2023 at 10:55:29AM +0800, Wang Yugui wrote:
> Hi,
> 
> > On Thu, May 18, 2023 at 10:46:43PM +0100, Matthew Wilcox wrote:
> > > -struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos)
> > > +struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len)
> > >  {
> > >  	unsigned fgp = FGP_WRITEBEGIN | FGP_NOFS;
> > > +	struct folio *folio;
> > >  
> > >  	if (iter->flags & IOMAP_NOWAIT)
> > >  		fgp |= FGP_NOWAIT;
> > > +	fgp |= fgp_order(len);
> > >  
> > > -	return __filemap_get_folio(iter->inode->i_mapping, pos >> PAGE_SHIFT,
> > > +	folio = __filemap_get_folio(iter->inode->i_mapping, pos >> PAGE_SHIFT,
> > >  			fgp, mapping_gfp_mask(iter->inode->i_mapping));
> > > +	if (!IS_ERR(folio) && folio_test_large(folio))
> > > +		printk("index:%lu len:%zu order:%u\n", (unsigned long)(pos / PAGE_SIZE), len, folio_order(folio));
> > > +	return folio;
> > >  }
> > 
> > Forgot to take the debugging out.  This should read:
> > 
> > -struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos)
> > +struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len)
> >  {
> >  	unsigned fgp = FGP_WRITEBEGIN | FGP_NOFS;
> >  	if (iter->flags & IOMAP_NOWAIT)
> >  		fgp |= FGP_NOWAIT;
> > +	fgp |= fgp_order(len);
> >  
> >  	return __filemap_get_folio(iter->inode->i_mapping, pos >> PAGE_SHIFT,
> >  			fgp, mapping_gfp_mask(iter->inode->i_mapping));
> >  }
> 
> I test it (attachment file) on 6.4.0-rc2.
> fio -name write-bandwidth -rw=write -bs=1024Ki -size=32Gi -runtime=30 -iodepth 1 -ioengine sync -zero_buffers=1 -direct=0 -end_fsync=1 -numjobs=4 -directory=/mnt/test
> 
> fio  WRITE: bw=2430MiB/s.
> expected value: > 6000MiB/s
> so yet no fio write bandwidth improvement.

That's basically unchanged.  There's no per-page or per-block work being
done in start/end writeback, so if Dave's investigation is applicable
to your situation, I'd expect to see an improvement.

Maybe try the second version of the patch I sent with the debug in,
to confirm you really are seeing large folios being created (you might
want to use printk_ratelimit() instead of printk to ensure it doesn't
overwhelm your system)?  That fio command you were using ought to create
them, but there's always a chance it doesn't.

Perhaps you could use perf (the command Dave used) to see where the time
is being spent.