* [PATCH v2] erofs: relaxed temporary buffers allocation on readahead @ 2024-01-20 14:55 Chunhai Guo via Linux-erofs 2024-01-22 2:07 ` Gao Xiang 0 siblings, 1 reply; 20+ messages in thread From: Chunhai Guo via Linux-erofs @ 2024-01-20 14:55 UTC (permalink / raw) To: xiang; +Cc: Chunhai Guo, linux-erofs, huyue2 Even with inplace decompression, sometimes extra temporary buffers are still needed for decompression. In low-memory scenarios, it would be better to try to allocate with GFP_NOWAIT on readahead first. That can help reduce the time spent on page allocation under memory pressure. There is an average reduction of 21% in page allocation time under multi-app launch benchmark workload [1] on ARM64 Android devices running the 5.15 kernel with an 8-core CPU and 8GB of memory. [1] https://lore.kernel.org/r/20240109074143.4138783-1-guochunhai@vivo.com Suggested-by: Gao Xiang <xiang@kernel.org> Signed-off-by: Chunhai Guo <guochunhai@vivo.com> --- fs/erofs/compress.h | 5 ++--- fs/erofs/decompressor.c | 5 +++-- fs/erofs/decompressor_deflate.c | 19 +++++++++++++------ fs/erofs/decompressor_lzma.c | 17 ++++++++++++----- fs/erofs/zdata.c | 16 ++++++++++++---- 5 files changed, 42 insertions(+), 20 deletions(-) diff --git a/fs/erofs/compress.h b/fs/erofs/compress.h index 279933e007d2..7cc5841577b2 100644 --- a/fs/erofs/compress.h +++ b/fs/erofs/compress.h @@ -11,13 +11,12 @@ struct z_erofs_decompress_req { struct super_block *sb; struct page **in, **out; - unsigned short pageofs_in, pageofs_out; unsigned int inputsize, outputsize; - /* indicate the algorithm will be used for decompression */ - unsigned int alg; + unsigned int alg; /* the algorithm for decompression */ bool inplace_io, partial_decoding, fillgaps; + gfp_t gfp; /* allocation flags for extra temporary buffers */ }; struct z_erofs_decompressor { diff --git a/fs/erofs/decompressor.c b/fs/erofs/decompressor.c index 1d65b9f60a39..ef2b08ec9830 100644 --- a/fs/erofs/decompressor.c +++ b/fs/erofs/decompressor.c @@ -111,8 +111,9 @@ static int z_erofs_lz4_prepare_dstpages(struct z_erofs_lz4_decompress_ctx *ctx, victim = availables[--top]; get_page(victim); } else { - victim = erofs_allocpage(pagepool, - GFP_KERNEL | __GFP_NOFAIL); + victim = erofs_allocpage(pagepool, rq->gfp); + if (!victim) + return -ENOMEM; set_page_private(victim, Z_EROFS_SHORTLIVED_PAGE); } rq->out[i] = victim; diff --git a/fs/erofs/decompressor_deflate.c b/fs/erofs/decompressor_deflate.c index 4a64a9c91dd3..b98872058abe 100644 --- a/fs/erofs/decompressor_deflate.c +++ b/fs/erofs/decompressor_deflate.c @@ -95,7 +95,7 @@ int z_erofs_load_deflate_config(struct super_block *sb, } int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, - struct page **pagepool) + struct page **pgpl) { const unsigned int nrpages_out = PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT; @@ -158,8 +158,12 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, strm->z.avail_out = min_t(u32, outsz, PAGE_SIZE - pofs); outsz -= strm->z.avail_out; if (!rq->out[no]) { - rq->out[no] = erofs_allocpage(pagepool, - GFP_KERNEL | __GFP_NOFAIL); + rq->out[no] = erofs_allocpage(pgpl, rq->gfp); + if (!rq->out[no]) { + kout = NULL; + err = -ENOMEM; + break; + } set_page_private(rq->out[no], Z_EROFS_SHORTLIVED_PAGE); } @@ -211,8 +215,11 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, DBG_BUGON(erofs_page_is_managed(EROFS_SB(sb), rq->in[j])); - tmppage = erofs_allocpage(pagepool, - GFP_KERNEL | __GFP_NOFAIL); + tmppage = erofs_allocpage(pgpl, rq->gfp); + if (!tmppage) { + err = -ENOMEM; + goto failed; + } set_page_private(tmppage, Z_EROFS_SHORTLIVED_PAGE); copy_highpage(tmppage, rq->in[j]); rq->in[j] = tmppage; @@ -230,7 +237,7 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, break; } } - +failed: if (zlib_inflateEnd(&strm->z) != Z_OK && !err) err = -EIO; if (kout) diff --git a/fs/erofs/decompressor_lzma.c b/fs/erofs/decompressor_lzma.c index 2dd14f99c1dc..6ca357d83cfa 100644 --- a/fs/erofs/decompressor_lzma.c +++ b/fs/erofs/decompressor_lzma.c @@ -148,7 +148,7 @@ int z_erofs_load_lzma_config(struct super_block *sb, } int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, - struct page **pagepool) + struct page **pgpl) { const unsigned int nrpages_out = PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT; @@ -215,8 +215,11 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, PAGE_SIZE - pageofs); outlen -= strm->buf.out_size; if (!rq->out[no] && rq->fillgaps) { /* deduped */ - rq->out[no] = erofs_allocpage(pagepool, - GFP_KERNEL | __GFP_NOFAIL); + rq->out[no] = erofs_allocpage(pgpl, rq->gfp); + if (!rq->out[no]) { + err = -ENOMEM; + break; + } set_page_private(rq->out[no], Z_EROFS_SHORTLIVED_PAGE); } @@ -258,8 +261,11 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, DBG_BUGON(erofs_page_is_managed(EROFS_SB(rq->sb), rq->in[j])); - tmppage = erofs_allocpage(pagepool, - GFP_KERNEL | __GFP_NOFAIL); + tmppage = erofs_allocpage(pgpl, rq->gfp); + if (!tmppage) { + err = -ENOMEM; + goto failed; + } set_page_private(tmppage, Z_EROFS_SHORTLIVED_PAGE); copy_highpage(tmppage, rq->in[j]); rq->in[j] = tmppage; @@ -277,6 +283,7 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, break; } } +failed: if (no < nrpages_out && strm->buf.out) kunmap(rq->out[no]); if (ni < nrpages_in) diff --git a/fs/erofs/zdata.c b/fs/erofs/zdata.c index 692c0c39be63..a293de2a60ed 100644 --- a/fs/erofs/zdata.c +++ b/fs/erofs/zdata.c @@ -82,6 +82,9 @@ struct z_erofs_pcluster { /* L: indicate several pageofs_outs or not */ bool multibases; + /* L: whether extra buffer allocations are best-effort */ + bool besteffort; + /* A: compressed bvecs (can be cached or inplaced pages) */ struct z_erofs_bvec compressed_bvecs[]; }; @@ -964,7 +967,7 @@ static int z_erofs_read_fragment(struct super_block *sb, struct page *page, } static int z_erofs_do_read_page(struct z_erofs_decompress_frontend *fe, - struct page *page) + struct page *page, bool ra) { struct inode *const inode = fe->inode; struct erofs_map_blocks *const map = &fe->map; @@ -1014,6 +1017,7 @@ static int z_erofs_do_read_page(struct z_erofs_decompress_frontend *fe, err = z_erofs_pcluster_begin(fe); if (err) goto out; + fe->pcl->besteffort |= !ra; } /* @@ -1280,7 +1284,11 @@ static int z_erofs_decompress_pcluster(struct z_erofs_decompress_backend *be, .inplace_io = overlapped, .partial_decoding = pcl->partial, .fillgaps = pcl->multibases, + .gfp = pcl->besteffort ? + GFP_KERNEL | __GFP_NOFAIL : + GFP_NOWAIT | __GFP_NORETRY }, be->pagepool); + pcl->besteffort = false; /* must handle all compressed pages before actual file pages */ if (z_erofs_is_inline_pcluster(pcl)) { @@ -1785,7 +1793,7 @@ static void z_erofs_pcluster_readmore(struct z_erofs_decompress_frontend *f, if (PageUptodate(page)) unlock_page(page); else - (void)z_erofs_do_read_page(f, page); + (void)z_erofs_do_read_page(f, page, !!rac); put_page(page); } @@ -1806,7 +1814,7 @@ static int z_erofs_read_folio(struct file *file, struct folio *folio) f.headoffset = (erofs_off_t)folio->index << PAGE_SHIFT; z_erofs_pcluster_readmore(&f, NULL, true); - err = z_erofs_do_read_page(&f, &folio->page); + err = z_erofs_do_read_page(&f, &folio->page, false); z_erofs_pcluster_readmore(&f, NULL, false); z_erofs_pcluster_end(&f); @@ -1847,7 +1855,7 @@ static void z_erofs_readahead(struct readahead_control *rac) folio = head; head = folio_get_private(folio); - err = z_erofs_do_read_page(&f, &folio->page); + err = z_erofs_do_read_page(&f, &folio->page, true); if (err && err != -EINTR) erofs_err(inode->i_sb, "readahead error at folio %lu @ nid %llu", folio->index, EROFS_I(inode)->nid); -- 2.25.1 ^ permalink raw reply related [flat|nested] 20+ messages in thread
* Re: [PATCH v2] erofs: relaxed temporary buffers allocation on readahead 2024-01-20 14:55 [PATCH v2] erofs: relaxed temporary buffers allocation on readahead Chunhai Guo via Linux-erofs @ 2024-01-22 2:07 ` Gao Xiang 2024-01-22 3:49 ` Chunhai Guo via Linux-erofs 0 siblings, 1 reply; 20+ messages in thread From: Gao Xiang @ 2024-01-22 2:07 UTC (permalink / raw) To: Chunhai Guo, xiang; +Cc: linux-erofs, huyue2 On 2024/1/20 22:55, Chunhai Guo wrote: > Even with inplace decompression, sometimes extra temporary buffers are > still needed for decompression. In low-memory scenarios, it would be > better to try to allocate with GFP_NOWAIT on readahead first. That can > help reduce the time spent on page allocation under memory pressure. > > There is an average reduction of 21% in page allocation time under It would be better to add a table to show the absolute numbers too (like what you did in the global pool commit.) If it's possible, there is no need to send a update version for this, just reply the updated commit message and I will update the commit manually. Otherwise it looks good to me, Thanks, Gao Xiang > multi-app launch benchmark workload [1] on ARM64 Android devices running > the 5.15 kernel with an 8-core CPU and 8GB of memory. > > [1] https://lore.kernel.org/r/20240109074143.4138783-1-guochunhai@vivo.com > > Suggested-by: Gao Xiang <xiang@kernel.org> > Signed-off-by: Chunhai Guo <guochunhai@vivo.com> > --- > fs/erofs/compress.h | 5 ++--- > fs/erofs/decompressor.c | 5 +++-- > fs/erofs/decompressor_deflate.c | 19 +++++++++++++------ > fs/erofs/decompressor_lzma.c | 17 ++++++++++++----- > fs/erofs/zdata.c | 16 ++++++++++++---- > 5 files changed, 42 insertions(+), 20 deletions(-) > > diff --git a/fs/erofs/compress.h b/fs/erofs/compress.h > index 279933e007d2..7cc5841577b2 100644 > --- a/fs/erofs/compress.h > +++ b/fs/erofs/compress.h > @@ -11,13 +11,12 @@ > struct z_erofs_decompress_req { > struct super_block *sb; > struct page **in, **out; > - > unsigned short pageofs_in, pageofs_out; > unsigned int inputsize, outputsize; > > - /* indicate the algorithm will be used for decompression */ > - unsigned int alg; > + unsigned int alg; /* the algorithm for decompression */ > bool inplace_io, partial_decoding, fillgaps; > + gfp_t gfp; /* allocation flags for extra temporary buffers */ > }; > > struct z_erofs_decompressor { > diff --git a/fs/erofs/decompressor.c b/fs/erofs/decompressor.c > index 1d65b9f60a39..ef2b08ec9830 100644 > --- a/fs/erofs/decompressor.c > +++ b/fs/erofs/decompressor.c > @@ -111,8 +111,9 @@ static int z_erofs_lz4_prepare_dstpages(struct z_erofs_lz4_decompress_ctx *ctx, > victim = availables[--top]; > get_page(victim); > } else { > - victim = erofs_allocpage(pagepool, > - GFP_KERNEL | __GFP_NOFAIL); > + victim = erofs_allocpage(pagepool, rq->gfp); > + if (!victim) > + return -ENOMEM; > set_page_private(victim, Z_EROFS_SHORTLIVED_PAGE); > } > rq->out[i] = victim; > diff --git a/fs/erofs/decompressor_deflate.c b/fs/erofs/decompressor_deflate.c > index 4a64a9c91dd3..b98872058abe 100644 > --- a/fs/erofs/decompressor_deflate.c > +++ b/fs/erofs/decompressor_deflate.c > @@ -95,7 +95,7 @@ int z_erofs_load_deflate_config(struct super_block *sb, > } > > int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, > - struct page **pagepool) > + struct page **pgpl) > { > const unsigned int nrpages_out = > PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT; > @@ -158,8 +158,12 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, > strm->z.avail_out = min_t(u32, outsz, PAGE_SIZE - pofs); > outsz -= strm->z.avail_out; > if (!rq->out[no]) { > - rq->out[no] = erofs_allocpage(pagepool, > - GFP_KERNEL | __GFP_NOFAIL); > + rq->out[no] = erofs_allocpage(pgpl, rq->gfp); > + if (!rq->out[no]) { > + kout = NULL; > + err = -ENOMEM; > + break; > + } > set_page_private(rq->out[no], > Z_EROFS_SHORTLIVED_PAGE); > } > @@ -211,8 +215,11 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, > > DBG_BUGON(erofs_page_is_managed(EROFS_SB(sb), > rq->in[j])); > - tmppage = erofs_allocpage(pagepool, > - GFP_KERNEL | __GFP_NOFAIL); > + tmppage = erofs_allocpage(pgpl, rq->gfp); > + if (!tmppage) { > + err = -ENOMEM; > + goto failed; > + } > set_page_private(tmppage, Z_EROFS_SHORTLIVED_PAGE); > copy_highpage(tmppage, rq->in[j]); > rq->in[j] = tmppage; > @@ -230,7 +237,7 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, > break; > } > } > - > +failed: > if (zlib_inflateEnd(&strm->z) != Z_OK && !err) > err = -EIO; > if (kout) > diff --git a/fs/erofs/decompressor_lzma.c b/fs/erofs/decompressor_lzma.c > index 2dd14f99c1dc..6ca357d83cfa 100644 > --- a/fs/erofs/decompressor_lzma.c > +++ b/fs/erofs/decompressor_lzma.c > @@ -148,7 +148,7 @@ int z_erofs_load_lzma_config(struct super_block *sb, > } > > int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, > - struct page **pagepool) > + struct page **pgpl) > { > const unsigned int nrpages_out = > PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT; > @@ -215,8 +215,11 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, > PAGE_SIZE - pageofs); > outlen -= strm->buf.out_size; > if (!rq->out[no] && rq->fillgaps) { /* deduped */ > - rq->out[no] = erofs_allocpage(pagepool, > - GFP_KERNEL | __GFP_NOFAIL); > + rq->out[no] = erofs_allocpage(pgpl, rq->gfp); > + if (!rq->out[no]) { > + err = -ENOMEM; > + break; > + } > set_page_private(rq->out[no], > Z_EROFS_SHORTLIVED_PAGE); > } > @@ -258,8 +261,11 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, > > DBG_BUGON(erofs_page_is_managed(EROFS_SB(rq->sb), > rq->in[j])); > - tmppage = erofs_allocpage(pagepool, > - GFP_KERNEL | __GFP_NOFAIL); > + tmppage = erofs_allocpage(pgpl, rq->gfp); > + if (!tmppage) { > + err = -ENOMEM; > + goto failed; > + } > set_page_private(tmppage, Z_EROFS_SHORTLIVED_PAGE); > copy_highpage(tmppage, rq->in[j]); > rq->in[j] = tmppage; > @@ -277,6 +283,7 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, > break; > } > } > +failed: > if (no < nrpages_out && strm->buf.out) > kunmap(rq->out[no]); > if (ni < nrpages_in) > diff --git a/fs/erofs/zdata.c b/fs/erofs/zdata.c > index 692c0c39be63..a293de2a60ed 100644 > --- a/fs/erofs/zdata.c > +++ b/fs/erofs/zdata.c > @@ -82,6 +82,9 @@ struct z_erofs_pcluster { > /* L: indicate several pageofs_outs or not */ > bool multibases; > > + /* L: whether extra buffer allocations are best-effort */ > + bool besteffort; > + > /* A: compressed bvecs (can be cached or inplaced pages) */ > struct z_erofs_bvec compressed_bvecs[]; > }; > @@ -964,7 +967,7 @@ static int z_erofs_read_fragment(struct super_block *sb, struct page *page, > } > > static int z_erofs_do_read_page(struct z_erofs_decompress_frontend *fe, > - struct page *page) > + struct page *page, bool ra) > { > struct inode *const inode = fe->inode; > struct erofs_map_blocks *const map = &fe->map; > @@ -1014,6 +1017,7 @@ static int z_erofs_do_read_page(struct z_erofs_decompress_frontend *fe, > err = z_erofs_pcluster_begin(fe); > if (err) > goto out; > + fe->pcl->besteffort |= !ra; > } > > /* > @@ -1280,7 +1284,11 @@ static int z_erofs_decompress_pcluster(struct z_erofs_decompress_backend *be, > .inplace_io = overlapped, > .partial_decoding = pcl->partial, > .fillgaps = pcl->multibases, > + .gfp = pcl->besteffort ? > + GFP_KERNEL | __GFP_NOFAIL : > + GFP_NOWAIT | __GFP_NORETRY > }, be->pagepool); > + pcl->besteffort = false; > > /* must handle all compressed pages before actual file pages */ > if (z_erofs_is_inline_pcluster(pcl)) { > @@ -1785,7 +1793,7 @@ static void z_erofs_pcluster_readmore(struct z_erofs_decompress_frontend *f, > if (PageUptodate(page)) > unlock_page(page); > else > - (void)z_erofs_do_read_page(f, page); > + (void)z_erofs_do_read_page(f, page, !!rac); > put_page(page); > } > > @@ -1806,7 +1814,7 @@ static int z_erofs_read_folio(struct file *file, struct folio *folio) > f.headoffset = (erofs_off_t)folio->index << PAGE_SHIFT; > > z_erofs_pcluster_readmore(&f, NULL, true); > - err = z_erofs_do_read_page(&f, &folio->page); > + err = z_erofs_do_read_page(&f, &folio->page, false); > z_erofs_pcluster_readmore(&f, NULL, false); > z_erofs_pcluster_end(&f); > > @@ -1847,7 +1855,7 @@ static void z_erofs_readahead(struct readahead_control *rac) > folio = head; > head = folio_get_private(folio); > > - err = z_erofs_do_read_page(&f, &folio->page); > + err = z_erofs_do_read_page(&f, &folio->page, true); > if (err && err != -EINTR) > erofs_err(inode->i_sb, "readahead error at folio %lu @ nid %llu", > folio->index, EROFS_I(inode)->nid); ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v2] erofs: relaxed temporary buffers allocation on readahead 2024-01-22 2:07 ` Gao Xiang @ 2024-01-22 3:49 ` Chunhai Guo via Linux-erofs 2024-01-22 4:37 ` Gao Xiang 0 siblings, 1 reply; 20+ messages in thread From: Chunhai Guo via Linux-erofs @ 2024-01-22 3:49 UTC (permalink / raw) To: Gao Xiang, Chunhai Guo, xiang@kernel.org Cc: linux-erofs@lists.ozlabs.org, huyue2@coolpad.com On 2024/1/22 10:07, Gao Xiang wrote: > [你通常不会收到来自 hsiangkao@linux.alibaba.com 的电子邮件。请访问 https://aka.ms/LearnAboutSenderIdentification，以了解这一点为什么很重要] > > On 2024/1/20 22:55, Chunhai Guo wrote: >> Even with inplace decompression, sometimes extra temporary buffers are >> still needed for decompression. In low-memory scenarios, it would be >> better to try to allocate with GFP_NOWAIT on readahead first. That can >> help reduce the time spent on page allocation under memory pressure. >> >> There is an average reduction of 21% in page allocation time under > It would be better to add a table to show the absolute numbers too > (like what you did in the global pool commit.) If it's possible, there > is no need to send a update version for this, just reply the updated > commit message and I will update the commit manually. The table below shows detailed numbers. The reduction I mentioned before was not accurate enough. Please help correct the improvement from 21% to 20.21%. +--------------+----------------+---------------+---------+ | | w/o GFP_NOWAIT | w/ GFP_NOWAIT | diff | +--------------+----------------+---------------+---------+ | Average (ms) | 3364 | 2684 | -20.21% | +--------------+----------------+---------------+---------+ Thanks, > > Otherwise it looks good to me, > > Thanks, > Gao Xiang > >> multi-app launch benchmark workload [1] on ARM64 Android devices running >> the 5.15 kernel with an 8-core CPU and 8GB of memory. >> >> [1] https://lore.kernel.org/r/20240109074143.4138783-1-guochunhai@vivo.com >> >> Suggested-by: Gao Xiang <xiang@kernel.org> >> Signed-off-by: Chunhai Guo <guochunhai@vivo.com> >> --- >> fs/erofs/compress.h | 5 ++--- >> fs/erofs/decompressor.c | 5 +++-- >> fs/erofs/decompressor_deflate.c | 19 +++++++++++++------ >> fs/erofs/decompressor_lzma.c | 17 ++++++++++++----- >> fs/erofs/zdata.c | 16 ++++++++++++---- >> 5 files changed, 42 insertions(+), 20 deletions(-) >> >> diff --git a/fs/erofs/compress.h b/fs/erofs/compress.h >> index 279933e007d2..7cc5841577b2 100644 >> --- a/fs/erofs/compress.h >> +++ b/fs/erofs/compress.h >> @@ -11,13 +11,12 @@ >> struct z_erofs_decompress_req { >> struct super_block *sb; >> struct page **in, **out; >> - >> unsigned short pageofs_in, pageofs_out; >> unsigned int inputsize, outputsize; >> >> - /* indicate the algorithm will be used for decompression */ >> - unsigned int alg; >> + unsigned int alg; /* the algorithm for decompression */ >> bool inplace_io, partial_decoding, fillgaps; >> + gfp_t gfp; /* allocation flags for extra temporary buffers */ >> }; >> >> struct z_erofs_decompressor { >> diff --git a/fs/erofs/decompressor.c b/fs/erofs/decompressor.c >> index 1d65b9f60a39..ef2b08ec9830 100644 >> --- a/fs/erofs/decompressor.c >> +++ b/fs/erofs/decompressor.c >> @@ -111,8 +111,9 @@ static int z_erofs_lz4_prepare_dstpages(struct z_erofs_lz4_decompress_ctx *ctx, >> victim = availables[--top]; >> get_page(victim); >> } else { >> - victim = erofs_allocpage(pagepool, >> - GFP_KERNEL | __GFP_NOFAIL); >> + victim = erofs_allocpage(pagepool, rq->gfp); >> + if (!victim) >> + return -ENOMEM; >> set_page_private(victim, Z_EROFS_SHORTLIVED_PAGE); >> } >> rq->out[i] = victim; >> diff --git a/fs/erofs/decompressor_deflate.c b/fs/erofs/decompressor_deflate.c >> index 4a64a9c91dd3..b98872058abe 100644 >> --- a/fs/erofs/decompressor_deflate.c >> +++ b/fs/erofs/decompressor_deflate.c >> @@ -95,7 +95,7 @@ int z_erofs_load_deflate_config(struct super_block *sb, >> } >> >> int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, >> - struct page **pagepool) >> + struct page **pgpl) >> { >> const unsigned int nrpages_out = >> PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT; >> @@ -158,8 +158,12 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, >> strm->z.avail_out = min_t(u32, outsz, PAGE_SIZE - pofs); >> outsz -= strm->z.avail_out; >> if (!rq->out[no]) { >> - rq->out[no] = erofs_allocpage(pagepool, >> - GFP_KERNEL | __GFP_NOFAIL); >> + rq->out[no] = erofs_allocpage(pgpl, rq->gfp); >> + if (!rq->out[no]) { >> + kout = NULL; >> + err = -ENOMEM; >> + break; >> + } >> set_page_private(rq->out[no], >> Z_EROFS_SHORTLIVED_PAGE); >> } >> @@ -211,8 +215,11 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, >> >> DBG_BUGON(erofs_page_is_managed(EROFS_SB(sb), >> rq->in[j])); >> - tmppage = erofs_allocpage(pagepool, >> - GFP_KERNEL | __GFP_NOFAIL); >> + tmppage = erofs_allocpage(pgpl, rq->gfp); >> + if (!tmppage) { >> + err = -ENOMEM; >> + goto failed; >> + } >> set_page_private(tmppage, Z_EROFS_SHORTLIVED_PAGE); >> copy_highpage(tmppage, rq->in[j]); >> rq->in[j] = tmppage; >> @@ -230,7 +237,7 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, >> break; >> } >> } >> - >> +failed: >> if (zlib_inflateEnd(&strm->z) != Z_OK && !err) >> err = -EIO; >> if (kout) >> diff --git a/fs/erofs/decompressor_lzma.c b/fs/erofs/decompressor_lzma.c >> index 2dd14f99c1dc..6ca357d83cfa 100644 >> --- a/fs/erofs/decompressor_lzma.c >> +++ b/fs/erofs/decompressor_lzma.c >> @@ -148,7 +148,7 @@ int z_erofs_load_lzma_config(struct super_block *sb, >> } >> >> int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, >> - struct page **pagepool) >> + struct page **pgpl) >> { >> const unsigned int nrpages_out = >> PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT; >> @@ -215,8 +215,11 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, >> PAGE_SIZE - pageofs); >> outlen -= strm->buf.out_size; >> if (!rq->out[no] && rq->fillgaps) { /* deduped */ >> - rq->out[no] = erofs_allocpage(pagepool, >> - GFP_KERNEL | __GFP_NOFAIL); >> + rq->out[no] = erofs_allocpage(pgpl, rq->gfp); >> + if (!rq->out[no]) { >> + err = -ENOMEM; >> + break; >> + } >> set_page_private(rq->out[no], >> Z_EROFS_SHORTLIVED_PAGE); >> } >> @@ -258,8 +261,11 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, >> >> DBG_BUGON(erofs_page_is_managed(EROFS_SB(rq->sb), >> rq->in[j])); >> - tmppage = erofs_allocpage(pagepool, >> - GFP_KERNEL | __GFP_NOFAIL); >> + tmppage = erofs_allocpage(pgpl, rq->gfp); >> + if (!tmppage) { >> + err = -ENOMEM; >> + goto failed; >> + } >> set_page_private(tmppage, Z_EROFS_SHORTLIVED_PAGE); >> copy_highpage(tmppage, rq->in[j]); >> rq->in[j] = tmppage; >> @@ -277,6 +283,7 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, >> break; >> } >> } >> +failed: >> if (no < nrpages_out && strm->buf.out) >> kunmap(rq->out[no]); >> if (ni < nrpages_in) >> diff --git a/fs/erofs/zdata.c b/fs/erofs/zdata.c >> index 692c0c39be63..a293de2a60ed 100644 >> --- a/fs/erofs/zdata.c >> +++ b/fs/erofs/zdata.c >> @@ -82,6 +82,9 @@ struct z_erofs_pcluster { >> /* L: indicate several pageofs_outs or not */ >> bool multibases; >> >> + /* L: whether extra buffer allocations are best-effort */ >> + bool besteffort; >> + >> /* A: compressed bvecs (can be cached or inplaced pages) */ >> struct z_erofs_bvec compressed_bvecs[]; >> }; >> @@ -964,7 +967,7 @@ static int z_erofs_read_fragment(struct super_block *sb, struct page *page, >> } >> >> static int z_erofs_do_read_page(struct z_erofs_decompress_frontend *fe, >> - struct page *page) >> + struct page *page, bool ra) >> { >> struct inode *const inode = fe->inode; >> struct erofs_map_blocks *const map = &fe->map; >> @@ -1014,6 +1017,7 @@ static int z_erofs_do_read_page(struct z_erofs_decompress_frontend *fe, >> err = z_erofs_pcluster_begin(fe); >> if (err) >> goto out; >> + fe->pcl->besteffort |= !ra; >> } >> >> /* >> @@ -1280,7 +1284,11 @@ static int z_erofs_decompress_pcluster(struct z_erofs_decompress_backend *be, >> .inplace_io = overlapped, >> .partial_decoding = pcl->partial, >> .fillgaps = pcl->multibases, >> + .gfp = pcl->besteffort ? >> + GFP_KERNEL | __GFP_NOFAIL : >> + GFP_NOWAIT | __GFP_NORETRY >> }, be->pagepool); >> + pcl->besteffort = false; >> >> /* must handle all compressed pages before actual file pages */ >> if (z_erofs_is_inline_pcluster(pcl)) { >> @@ -1785,7 +1793,7 @@ static void z_erofs_pcluster_readmore(struct z_erofs_decompress_frontend *f, >> if (PageUptodate(page)) >> unlock_page(page); >> else >> - (void)z_erofs_do_read_page(f, page); >> + (void)z_erofs_do_read_page(f, page, !!rac); >> put_page(page); >> } >> >> @@ -1806,7 +1814,7 @@ static int z_erofs_read_folio(struct file *file, struct folio *folio) >> f.headoffset = (erofs_off_t)folio->index << PAGE_SHIFT; >> >> z_erofs_pcluster_readmore(&f, NULL, true); >> - err = z_erofs_do_read_page(&f, &folio->page); >> + err = z_erofs_do_read_page(&f, &folio->page, false); >> z_erofs_pcluster_readmore(&f, NULL, false); >> z_erofs_pcluster_end(&f); >> >> @@ -1847,7 +1855,7 @@ static void z_erofs_readahead(struct readahead_control *rac) >> folio = head; >> head = folio_get_private(folio); >> >> - err = z_erofs_do_read_page(&f, &folio->page); >> + err = z_erofs_do_read_page(&f, &folio->page, true); >> if (err && err != -EINTR) >> erofs_err(inode->i_sb, "readahead error at folio %lu @ nid %llu", >> folio->index, EROFS_I(inode)->nid); ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v2] erofs: relaxed temporary buffers allocation on readahead 2024-01-22 3:49 ` Chunhai Guo via Linux-erofs @ 2024-01-22 4:37 ` Gao Xiang 2024-01-22 7:42 ` Chunhai Guo via Linux-erofs 0 siblings, 1 reply; 20+ messages in thread From: Gao Xiang @ 2024-01-22 4:37 UTC (permalink / raw) To: Chunhai Guo, xiang@kernel.org Cc: linux-erofs@lists.ozlabs.org, huyue2@coolpad.com On 2024/1/22 11:49, Chunhai Guo wrote: > On 2024/1/22 10:07, Gao Xiang wrote: >> [你通常不会收到来自 hsiangkao@linux.alibaba.com 的电子邮件。请访问 https://aka.ms/LearnAboutSenderIdentification，以了解这一点为什么很重要] >> >> On 2024/1/20 22:55, Chunhai Guo wrote: >>> Even with inplace decompression, sometimes extra temporary buffers are >>> still needed for decompression. In low-memory scenarios, it would be >>> better to try to allocate with GFP_NOWAIT on readahead first. That can >>> help reduce the time spent on page allocation under memory pressure. >>> >>> There is an average reduction of 21% in page allocation time under >> It would be better to add a table to show the absolute numbers too >> (like what you did in the global pool commit.) If it's possible, there >> is no need to send a update version for this, just reply the updated >> commit message and I will update the commit manually. > > > The table below shows detailed numbers. The reduction I mentioned before > was not accurate enough. Please help correct the improvement from 21% to > 20.21%. > > > +--------------+----------------+---------------+---------+ > | | w/o GFP_NOWAIT | w/ GFP_NOWAIT | diff | > +--------------+----------------+---------------+---------+ > | Average (ms) | 3364 | 2684 | -20.21% | > +--------------+----------------+---------------+---------+ Did it test without the 16k sliding window change? https://lore.kernel.org/linux-erofs/69711d55-f7a2-420b-9ba8-fa2921f66a4c@vivo.com Could you benchmark these two optimizations together to show the extreme optimized case without a global pool? With a new table if possible? I will add this to the commit message too. Thanks, Gao Xiang > > Thanks, > > >> >> Otherwise it looks good to me, >> >> Thanks, >> Gao Xiang >> >>> multi-app launch benchmark workload [1] on ARM64 Android devices running >>> the 5.15 kernel with an 8-core CPU and 8GB of memory. >>> >>> [1] https://lore.kernel.org/r/20240109074143.4138783-1-guochunhai@vivo.com >>> >>> Suggested-by: Gao Xiang <xiang@kernel.org> >>> Signed-off-by: Chunhai Guo <guochunhai@vivo.com> >>> --- >>> fs/erofs/compress.h | 5 ++--- >>> fs/erofs/decompressor.c | 5 +++-- >>> fs/erofs/decompressor_deflate.c | 19 +++++++++++++------ >>> fs/erofs/decompressor_lzma.c | 17 ++++++++++++----- >>> fs/erofs/zdata.c | 16 ++++++++++++---- >>> 5 files changed, 42 insertions(+), 20 deletions(-) >>> >>> diff --git a/fs/erofs/compress.h b/fs/erofs/compress.h >>> index 279933e007d2..7cc5841577b2 100644 >>> --- a/fs/erofs/compress.h >>> +++ b/fs/erofs/compress.h >>> @@ -11,13 +11,12 @@ >>> struct z_erofs_decompress_req { >>> struct super_block *sb; >>> struct page **in, **out; >>> - >>> unsigned short pageofs_in, pageofs_out; >>> unsigned int inputsize, outputsize; >>> >>> - /* indicate the algorithm will be used for decompression */ >>> - unsigned int alg; >>> + unsigned int alg; /* the algorithm for decompression */ >>> bool inplace_io, partial_decoding, fillgaps; >>> + gfp_t gfp; /* allocation flags for extra temporary buffers */ >>> }; >>> >>> struct z_erofs_decompressor { >>> diff --git a/fs/erofs/decompressor.c b/fs/erofs/decompressor.c >>> index 1d65b9f60a39..ef2b08ec9830 100644 >>> --- a/fs/erofs/decompressor.c >>> +++ b/fs/erofs/decompressor.c >>> @@ -111,8 +111,9 @@ static int z_erofs_lz4_prepare_dstpages(struct z_erofs_lz4_decompress_ctx *ctx, >>> victim = availables[--top]; >>> get_page(victim); >>> } else { >>> - victim = erofs_allocpage(pagepool, >>> - GFP_KERNEL | __GFP_NOFAIL); >>> + victim = erofs_allocpage(pagepool, rq->gfp); >>> + if (!victim) >>> + return -ENOMEM; >>> set_page_private(victim, Z_EROFS_SHORTLIVED_PAGE); >>> } >>> rq->out[i] = victim; >>> diff --git a/fs/erofs/decompressor_deflate.c b/fs/erofs/decompressor_deflate.c >>> index 4a64a9c91dd3..b98872058abe 100644 >>> --- a/fs/erofs/decompressor_deflate.c >>> +++ b/fs/erofs/decompressor_deflate.c >>> @@ -95,7 +95,7 @@ int z_erofs_load_deflate_config(struct super_block *sb, >>> } >>> >>> int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, >>> - struct page **pagepool) >>> + struct page **pgpl) >>> { >>> const unsigned int nrpages_out = >>> PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT; >>> @@ -158,8 +158,12 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, >>> strm->z.avail_out = min_t(u32, outsz, PAGE_SIZE - pofs); >>> outsz -= strm->z.avail_out; >>> if (!rq->out[no]) { >>> - rq->out[no] = erofs_allocpage(pagepool, >>> - GFP_KERNEL | __GFP_NOFAIL); >>> + rq->out[no] = erofs_allocpage(pgpl, rq->gfp); >>> + if (!rq->out[no]) { >>> + kout = NULL; >>> + err = -ENOMEM; >>> + break; >>> + } >>> set_page_private(rq->out[no], >>> Z_EROFS_SHORTLIVED_PAGE); >>> } >>> @@ -211,8 +215,11 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, >>> >>> DBG_BUGON(erofs_page_is_managed(EROFS_SB(sb), >>> rq->in[j])); >>> - tmppage = erofs_allocpage(pagepool, >>> - GFP_KERNEL | __GFP_NOFAIL); >>> + tmppage = erofs_allocpage(pgpl, rq->gfp); >>> + if (!tmppage) { >>> + err = -ENOMEM; >>> + goto failed; >>> + } >>> set_page_private(tmppage, Z_EROFS_SHORTLIVED_PAGE); >>> copy_highpage(tmppage, rq->in[j]); >>> rq->in[j] = tmppage; >>> @@ -230,7 +237,7 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, >>> break; >>> } >>> } >>> - >>> +failed: >>> if (zlib_inflateEnd(&strm->z) != Z_OK && !err) >>> err = -EIO; >>> if (kout) >>> diff --git a/fs/erofs/decompressor_lzma.c b/fs/erofs/decompressor_lzma.c >>> index 2dd14f99c1dc..6ca357d83cfa 100644 >>> --- a/fs/erofs/decompressor_lzma.c >>> +++ b/fs/erofs/decompressor_lzma.c >>> @@ -148,7 +148,7 @@ int z_erofs_load_lzma_config(struct super_block *sb, >>> } >>> >>> int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, >>> - struct page **pagepool) >>> + struct page **pgpl) >>> { >>> const unsigned int nrpages_out = >>> PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT; >>> @@ -215,8 +215,11 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, >>> PAGE_SIZE - pageofs); >>> outlen -= strm->buf.out_size; >>> if (!rq->out[no] && rq->fillgaps) { /* deduped */ >>> - rq->out[no] = erofs_allocpage(pagepool, >>> - GFP_KERNEL | __GFP_NOFAIL); >>> + rq->out[no] = erofs_allocpage(pgpl, rq->gfp); >>> + if (!rq->out[no]) { >>> + err = -ENOMEM; >>> + break; >>> + } >>> set_page_private(rq->out[no], >>> Z_EROFS_SHORTLIVED_PAGE); >>> } >>> @@ -258,8 +261,11 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, >>> >>> DBG_BUGON(erofs_page_is_managed(EROFS_SB(rq->sb), >>> rq->in[j])); >>> - tmppage = erofs_allocpage(pagepool, >>> - GFP_KERNEL | __GFP_NOFAIL); >>> + tmppage = erofs_allocpage(pgpl, rq->gfp); >>> + if (!tmppage) { >>> + err = -ENOMEM; >>> + goto failed; >>> + } >>> set_page_private(tmppage, Z_EROFS_SHORTLIVED_PAGE); >>> copy_highpage(tmppage, rq->in[j]); >>> rq->in[j] = tmppage; >>> @@ -277,6 +283,7 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, >>> break; >>> } >>> } >>> +failed: >>> if (no < nrpages_out && strm->buf.out) >>> kunmap(rq->out[no]); >>> if (ni < nrpages_in) >>> diff --git a/fs/erofs/zdata.c b/fs/erofs/zdata.c >>> index 692c0c39be63..a293de2a60ed 100644 >>> --- a/fs/erofs/zdata.c >>> +++ b/fs/erofs/zdata.c >>> @@ -82,6 +82,9 @@ struct z_erofs_pcluster { >>> /* L: indicate several pageofs_outs or not */ >>> bool multibases; >>> >>> + /* L: whether extra buffer allocations are best-effort */ >>> + bool besteffort; >>> + >>> /* A: compressed bvecs (can be cached or inplaced pages) */ >>> struct z_erofs_bvec compressed_bvecs[]; >>> }; >>> @@ -964,7 +967,7 @@ static int z_erofs_read_fragment(struct super_block *sb, struct page *page, >>> } >>> >>> static int z_erofs_do_read_page(struct z_erofs_decompress_frontend *fe, >>> - struct page *page) >>> + struct page *page, bool ra) >>> { >>> struct inode *const inode = fe->inode; >>> struct erofs_map_blocks *const map = &fe->map; >>> @@ -1014,6 +1017,7 @@ static int z_erofs_do_read_page(struct z_erofs_decompress_frontend *fe, >>> err = z_erofs_pcluster_begin(fe); >>> if (err) >>> goto out; >>> + fe->pcl->besteffort |= !ra; >>> } >>> >>> /* >>> @@ -1280,7 +1284,11 @@ static int z_erofs_decompress_pcluster(struct z_erofs_decompress_backend *be, >>> .inplace_io = overlapped, >>> .partial_decoding = pcl->partial, >>> .fillgaps = pcl->multibases, >>> + .gfp = pcl->besteffort ? >>> + GFP_KERNEL | __GFP_NOFAIL : >>> + GFP_NOWAIT | __GFP_NORETRY >>> }, be->pagepool); >>> + pcl->besteffort = false; >>> >>> /* must handle all compressed pages before actual file pages */ >>> if (z_erofs_is_inline_pcluster(pcl)) { >>> @@ -1785,7 +1793,7 @@ static void z_erofs_pcluster_readmore(struct z_erofs_decompress_frontend *f, >>> if (PageUptodate(page)) >>> unlock_page(page); >>> else >>> - (void)z_erofs_do_read_page(f, page); >>> + (void)z_erofs_do_read_page(f, page, !!rac); >>> put_page(page); >>> } >>> >>> @@ -1806,7 +1814,7 @@ static int z_erofs_read_folio(struct file *file, struct folio *folio) >>> f.headoffset = (erofs_off_t)folio->index << PAGE_SHIFT; >>> >>> z_erofs_pcluster_readmore(&f, NULL, true); >>> - err = z_erofs_do_read_page(&f, &folio->page); >>> + err = z_erofs_do_read_page(&f, &folio->page, false); >>> z_erofs_pcluster_readmore(&f, NULL, false); >>> z_erofs_pcluster_end(&f); >>> >>> @@ -1847,7 +1855,7 @@ static void z_erofs_readahead(struct readahead_control *rac) >>> folio = head; >>> head = folio_get_private(folio); >>> >>> - err = z_erofs_do_read_page(&f, &folio->page); >>> + err = z_erofs_do_read_page(&f, &folio->page, true); >>> if (err && err != -EINTR) >>> erofs_err(inode->i_sb, "readahead error at folio %lu @ nid %llu", >>> folio->index, EROFS_I(inode)->nid); > > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v2] erofs: relaxed temporary buffers allocation on readahead 2024-01-22 4:37 ` Gao Xiang @ 2024-01-22 7:42 ` Chunhai Guo via Linux-erofs 2024-01-26 2:41 ` Chunhai Guo via Linux-erofs 0 siblings, 1 reply; 20+ messages in thread From: Chunhai Guo via Linux-erofs @ 2024-01-22 7:42 UTC (permalink / raw) To: Gao Xiang, Chunhai Guo, xiang@kernel.org Cc: linux-erofs@lists.ozlabs.org, huyue2@coolpad.com On 2024/1/22 12:37, Gao Xiang wrote: > [你通常不会收到来自 hsiangkao@linux.alibaba.com 的电子邮件。请访问 https://aka.ms/LearnAboutSenderIdentification，以了解这一点为什么很重要] > > On 2024/1/22 11:49, Chunhai Guo wrote: >> On 2024/1/22 10:07, Gao Xiang wrote: >>> [你通常不会收到来自 hsiangkao@linux.alibaba.com 的电子邮件。请访问 https://aka.ms/LearnAboutSenderIdentification，以了解这一点为什么很重要] >>> >>> On 2024/1/20 22:55, Chunhai Guo wrote: >>>> Even with inplace decompression, sometimes extra temporary buffers are >>>> still needed for decompression. In low-memory scenarios, it would be >>>> better to try to allocate with GFP_NOWAIT on readahead first. That can >>>> help reduce the time spent on page allocation under memory pressure. >>>> >>>> There is an average reduction of 21% in page allocation time under >>> It would be better to add a table to show the absolute numbers too >>> (like what you did in the global pool commit.) If it's possible, there >>> is no need to send a update version for this, just reply the updated >>> commit message and I will update the commit manually. >> >> The table below shows detailed numbers. The reduction I mentioned before >> was not accurate enough. Please help correct the improvement from 21% to >> 20.21%. >> >> >> +--------------+----------------+---------------+---------+ >> | | w/o GFP_NOWAIT | w/ GFP_NOWAIT | diff | >> +--------------+----------------+---------------+---------+ >> | Average (ms) | 3364 | 2684 | -20.21% | >> +--------------+----------------+---------------+---------+ > Did it test without the 16k sliding window change? > https://lore.kernel.org/linux-erofs/69711d55-f7a2-420b-9ba8-fa2921f66a4c@vivo.com The result is tested with 64k sliding window change. > Could you benchmark these two optimizations together to > show the extreme optimized case without a global pool? > With a new table if possible? I will add this to > the commit message too. OK. I will reply to this email when the benchmark is finished. > Thanks, > Gao Xiang > >> Thanks, >> >> >>> Otherwise it looks good to me, >>> >>> Thanks, >>> Gao Xiang >>> >>>> multi-app launch benchmark workload [1] on ARM64 Android devices running >>>> the 5.15 kernel with an 8-core CPU and 8GB of memory. >>>> >>>> [1] https://lore.kernel.org/r/20240109074143.4138783-1-guochunhai@vivo.com >>>> >>>> Suggested-by: Gao Xiang <xiang@kernel.org> >>>> Signed-off-by: Chunhai Guo <guochunhai@vivo.com> >>>> --- >>>> fs/erofs/compress.h | 5 ++--- >>>> fs/erofs/decompressor.c | 5 +++-- >>>> fs/erofs/decompressor_deflate.c | 19 +++++++++++++------ >>>> fs/erofs/decompressor_lzma.c | 17 ++++++++++++----- >>>> fs/erofs/zdata.c | 16 ++++++++++++---- >>>> 5 files changed, 42 insertions(+), 20 deletions(-) >>>> >>>> diff --git a/fs/erofs/compress.h b/fs/erofs/compress.h >>>> index 279933e007d2..7cc5841577b2 100644 >>>> --- a/fs/erofs/compress.h >>>> +++ b/fs/erofs/compress.h >>>> @@ -11,13 +11,12 @@ >>>> struct z_erofs_decompress_req { >>>> struct super_block *sb; >>>> struct page **in, **out; >>>> - >>>> unsigned short pageofs_in, pageofs_out; >>>> unsigned int inputsize, outputsize; >>>> >>>> - /* indicate the algorithm will be used for decompression */ >>>> - unsigned int alg; >>>> + unsigned int alg; /* the algorithm for decompression */ >>>> bool inplace_io, partial_decoding, fillgaps; >>>> + gfp_t gfp; /* allocation flags for extra temporary buffers */ >>>> }; >>>> >>>> struct z_erofs_decompressor { >>>> diff --git a/fs/erofs/decompressor.c b/fs/erofs/decompressor.c >>>> index 1d65b9f60a39..ef2b08ec9830 100644 >>>> --- a/fs/erofs/decompressor.c >>>> +++ b/fs/erofs/decompressor.c >>>> @@ -111,8 +111,9 @@ static int z_erofs_lz4_prepare_dstpages(struct z_erofs_lz4_decompress_ctx *ctx, >>>> victim = availables[--top]; >>>> get_page(victim); >>>> } else { >>>> - victim = erofs_allocpage(pagepool, >>>> - GFP_KERNEL | __GFP_NOFAIL); >>>> + victim = erofs_allocpage(pagepool, rq->gfp); >>>> + if (!victim) >>>> + return -ENOMEM; >>>> set_page_private(victim, Z_EROFS_SHORTLIVED_PAGE); >>>> } >>>> rq->out[i] = victim; >>>> diff --git a/fs/erofs/decompressor_deflate.c b/fs/erofs/decompressor_deflate.c >>>> index 4a64a9c91dd3..b98872058abe 100644 >>>> --- a/fs/erofs/decompressor_deflate.c >>>> +++ b/fs/erofs/decompressor_deflate.c >>>> @@ -95,7 +95,7 @@ int z_erofs_load_deflate_config(struct super_block *sb, >>>> } >>>> >>>> int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, >>>> - struct page **pagepool) >>>> + struct page **pgpl) >>>> { >>>> const unsigned int nrpages_out = >>>> PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT; >>>> @@ -158,8 +158,12 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, >>>> strm->z.avail_out = min_t(u32, outsz, PAGE_SIZE - pofs); >>>> outsz -= strm->z.avail_out; >>>> if (!rq->out[no]) { >>>> - rq->out[no] = erofs_allocpage(pagepool, >>>> - GFP_KERNEL | __GFP_NOFAIL); >>>> + rq->out[no] = erofs_allocpage(pgpl, rq->gfp); >>>> + if (!rq->out[no]) { >>>> + kout = NULL; >>>> + err = -ENOMEM; >>>> + break; >>>> + } >>>> set_page_private(rq->out[no], >>>> Z_EROFS_SHORTLIVED_PAGE); >>>> } >>>> @@ -211,8 +215,11 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, >>>> >>>> DBG_BUGON(erofs_page_is_managed(EROFS_SB(sb), >>>> rq->in[j])); >>>> - tmppage = erofs_allocpage(pagepool, >>>> - GFP_KERNEL | __GFP_NOFAIL); >>>> + tmppage = erofs_allocpage(pgpl, rq->gfp); >>>> + if (!tmppage) { >>>> + err = -ENOMEM; >>>> + goto failed; >>>> + } >>>> set_page_private(tmppage, Z_EROFS_SHORTLIVED_PAGE); >>>> copy_highpage(tmppage, rq->in[j]); >>>> rq->in[j] = tmppage; >>>> @@ -230,7 +237,7 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, >>>> break; >>>> } >>>> } >>>> - >>>> +failed: >>>> if (zlib_inflateEnd(&strm->z) != Z_OK && !err) >>>> err = -EIO; >>>> if (kout) >>>> diff --git a/fs/erofs/decompressor_lzma.c b/fs/erofs/decompressor_lzma.c >>>> index 2dd14f99c1dc..6ca357d83cfa 100644 >>>> --- a/fs/erofs/decompressor_lzma.c >>>> +++ b/fs/erofs/decompressor_lzma.c >>>> @@ -148,7 +148,7 @@ int z_erofs_load_lzma_config(struct super_block *sb, >>>> } >>>> >>>> int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, >>>> - struct page **pagepool) >>>> + struct page **pgpl) >>>> { >>>> const unsigned int nrpages_out = >>>> PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT; >>>> @@ -215,8 +215,11 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, >>>> PAGE_SIZE - pageofs); >>>> outlen -= strm->buf.out_size; >>>> if (!rq->out[no] && rq->fillgaps) { /* deduped */ >>>> - rq->out[no] = erofs_allocpage(pagepool, >>>> - GFP_KERNEL | __GFP_NOFAIL); >>>> + rq->out[no] = erofs_allocpage(pgpl, rq->gfp); >>>> + if (!rq->out[no]) { >>>> + err = -ENOMEM; >>>> + break; >>>> + } >>>> set_page_private(rq->out[no], >>>> Z_EROFS_SHORTLIVED_PAGE); >>>> } >>>> @@ -258,8 +261,11 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, >>>> >>>> DBG_BUGON(erofs_page_is_managed(EROFS_SB(rq->sb), >>>> rq->in[j])); >>>> - tmppage = erofs_allocpage(pagepool, >>>> - GFP_KERNEL | __GFP_NOFAIL); >>>> + tmppage = erofs_allocpage(pgpl, rq->gfp); >>>> + if (!tmppage) { >>>> + err = -ENOMEM; >>>> + goto failed; >>>> + } >>>> set_page_private(tmppage, Z_EROFS_SHORTLIVED_PAGE); >>>> copy_highpage(tmppage, rq->in[j]); >>>> rq->in[j] = tmppage; >>>> @@ -277,6 +283,7 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, >>>> break; >>>> } >>>> } >>>> +failed: >>>> if (no < nrpages_out && strm->buf.out) >>>> kunmap(rq->out[no]); >>>> if (ni < nrpages_in) >>>> diff --git a/fs/erofs/zdata.c b/fs/erofs/zdata.c >>>> index 692c0c39be63..a293de2a60ed 100644 >>>> --- a/fs/erofs/zdata.c >>>> +++ b/fs/erofs/zdata.c >>>> @@ -82,6 +82,9 @@ struct z_erofs_pcluster { >>>> /* L: indicate several pageofs_outs or not */ >>>> bool multibases; >>>> >>>> + /* L: whether extra buffer allocations are best-effort */ >>>> + bool besteffort; >>>> + >>>> /* A: compressed bvecs (can be cached or inplaced pages) */ >>>> struct z_erofs_bvec compressed_bvecs[]; >>>> }; >>>> @@ -964,7 +967,7 @@ static int z_erofs_read_fragment(struct super_block *sb, struct page *page, >>>> } >>>> >>>> static int z_erofs_do_read_page(struct z_erofs_decompress_frontend *fe, >>>> - struct page *page) >>>> + struct page *page, bool ra) >>>> { >>>> struct inode *const inode = fe->inode; >>>> struct erofs_map_blocks *const map = &fe->map; >>>> @@ -1014,6 +1017,7 @@ static int z_erofs_do_read_page(struct z_erofs_decompress_frontend *fe, >>>> err = z_erofs_pcluster_begin(fe); >>>> if (err) >>>> goto out; >>>> + fe->pcl->besteffort |= !ra; >>>> } >>>> >>>> /* >>>> @@ -1280,7 +1284,11 @@ static int z_erofs_decompress_pcluster(struct z_erofs_decompress_backend *be, >>>> .inplace_io = overlapped, >>>> .partial_decoding = pcl->partial, >>>> .fillgaps = pcl->multibases, >>>> + .gfp = pcl->besteffort ? >>>> + GFP_KERNEL | __GFP_NOFAIL : >>>> + GFP_NOWAIT | __GFP_NORETRY >>>> }, be->pagepool); >>>> + pcl->besteffort = false; >>>> >>>> /* must handle all compressed pages before actual file pages */ >>>> if (z_erofs_is_inline_pcluster(pcl)) { >>>> @@ -1785,7 +1793,7 @@ static void z_erofs_pcluster_readmore(struct z_erofs_decompress_frontend *f, >>>> if (PageUptodate(page)) >>>> unlock_page(page); >>>> else >>>> - (void)z_erofs_do_read_page(f, page); >>>> + (void)z_erofs_do_read_page(f, page, !!rac); >>>> put_page(page); >>>> } >>>> >>>> @@ -1806,7 +1814,7 @@ static int z_erofs_read_folio(struct file *file, struct folio *folio) >>>> f.headoffset = (erofs_off_t)folio->index << PAGE_SHIFT; >>>> >>>> z_erofs_pcluster_readmore(&f, NULL, true); >>>> - err = z_erofs_do_read_page(&f, &folio->page); >>>> + err = z_erofs_do_read_page(&f, &folio->page, false); >>>> z_erofs_pcluster_readmore(&f, NULL, false); >>>> z_erofs_pcluster_end(&f); >>>> >>>> @@ -1847,7 +1855,7 @@ static void z_erofs_readahead(struct readahead_control *rac) >>>> folio = head; >>>> head = folio_get_private(folio); >>>> >>>> - err = z_erofs_do_read_page(&f, &folio->page); >>>> + err = z_erofs_do_read_page(&f, &folio->page, true); >>>> if (err && err != -EINTR) >>>> erofs_err(inode->i_sb, "readahead error at folio %lu @ nid %llu", >>>> folio->index, EROFS_I(inode)->nid); >> ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v2] erofs: relaxed temporary buffers allocation on readahead 2024-01-22 7:42 ` Chunhai Guo via Linux-erofs @ 2024-01-26 2:41 ` Chunhai Guo via Linux-erofs 2024-01-26 2:47 ` Gao Xiang 0 siblings, 1 reply; 20+ messages in thread From: Chunhai Guo via Linux-erofs @ 2024-01-26 2:41 UTC (permalink / raw) To: Chunhai Guo, Gao Xiang, xiang@kernel.org Cc: linux-erofs@lists.ozlabs.org, huyue2@coolpad.com On 2024/1/22 15:42, Chunhai Guo wrote: > On 2024/1/22 12:37, Gao Xiang wrote: >> [你通常不会收到来自 hsiangkao@linux.alibaba.com 的电子邮件。请访问 https://aka.ms/LearnAboutSenderIdentification，以了解这一点为什么很重要] >> >> On 2024/1/22 11:49, Chunhai Guo wrote: >>> On 2024/1/22 10:07, Gao Xiang wrote: >>>> [你通常不会收到来自 hsiangkao@linux.alibaba.com 的电子邮件。请访问 https://aka.ms/LearnAboutSenderIdentification，以了解这一点为什么很重要] >>>> >>>> On 2024/1/20 22:55, Chunhai Guo wrote: >>>>> Even with inplace decompression, sometimes extra temporary buffers are >>>>> still needed for decompression. In low-memory scenarios, it would be >>>>> better to try to allocate with GFP_NOWAIT on readahead first. That can >>>>> help reduce the time spent on page allocation under memory pressure. >>>>> >>>>> There is an average reduction of 21% in page allocation time under >>>> It would be better to add a table to show the absolute numbers too >>>> (like what you did in the global pool commit.) If it's possible, there >>>> is no need to send a update version for this, just reply the updated >>>> commit message and I will update the commit manually. >>> The table below shows detailed numbers. The reduction I mentioned before >>> was not accurate enough. Please help correct the improvement from 21% to >>> 20.21%. >>> >>> >>> +--------------+----------------+---------------+---------+ >>> | | w/o GFP_NOWAIT | w/ GFP_NOWAIT | diff | >>> +--------------+----------------+---------------+---------+ >>> | Average (ms) | 3364 | 2684 | -20.21% | >>> +--------------+----------------+---------------+---------+ >> Did it test without the 16k sliding window change? >> https://lore.kernel.org/linux-erofs/69711d55-f7a2-420b-9ba8-fa2921f66a4c@vivo.com > The result is tested with 64k sliding window change. > >> Could you benchmark these two optimizations together to >> show the extreme optimized case without a global pool? >> With a new table if possible? I will add this to >> the commit message too. > > OK. I will reply to this email when the benchmark is finished. The benchmark has been completed and the table below shows that there is an average 52.14% reduction in page allocation time with these two optimizations. +--------------+----------------+---------------+---------+ | | 64k window | 16k window | | | | w/o GFP_NOWAIT | w/ GFP_NOWAIT | diff | +--------------+----------------+---------------+---------+ | Average (ms) | 3364 | 1610 | -52.14% | +--------------+----------------+---------------+---------+ Table below summarizes the results of these three benchmarks. +--------------+----------------+----------------+---------------+---------------+ | | 64k window | 16k window | 64k window | 16k window | | | w/o GFP_NOWAIT | w/o GFP_NOWAIT | w/ GFP_NOWAIT | w/ GFP_NOWAIT | +--------------+----------------+----------------+---------------+---------------+ | Average (ms) | 3364 | 2079 | 2684 | 1610 | +--------------+----------------+----------------+---------------+---------------+ | diff | | -38.19% | -20.81% | -52.14% | +--------------+----------------+----------------+---------------+---------------+ Thanks. >> Thanks, >> Gao Xiang >> >>> Thanks, >>> >>> >>>> Otherwise it looks good to me, >>>> >>>> Thanks, >>>> Gao Xiang >>>> >>>>> multi-app launch benchmark workload [1] on ARM64 Android devices running >>>>> the 5.15 kernel with an 8-core CPU and 8GB of memory. >>>>> >>>>> [1] https://lore.kernel.org/r/20240109074143.4138783-1-guochunhai@vivo.com >>>>> >>>>> Suggested-by: Gao Xiang <xiang@kernel.org> >>>>> Signed-off-by: Chunhai Guo <guochunhai@vivo.com> >>>>> --- >>>>> fs/erofs/compress.h | 5 ++--- >>>>> fs/erofs/decompressor.c | 5 +++-- >>>>> fs/erofs/decompressor_deflate.c | 19 +++++++++++++------ >>>>> fs/erofs/decompressor_lzma.c | 17 ++++++++++++----- >>>>> fs/erofs/zdata.c | 16 ++++++++++++---- >>>>> 5 files changed, 42 insertions(+), 20 deletions(-) >>>>> >>>>> diff --git a/fs/erofs/compress.h b/fs/erofs/compress.h >>>>> index 279933e007d2..7cc5841577b2 100644 >>>>> --- a/fs/erofs/compress.h >>>>> +++ b/fs/erofs/compress.h >>>>> @@ -11,13 +11,12 @@ >>>>> struct z_erofs_decompress_req { >>>>> struct super_block *sb; >>>>> struct page **in, **out; >>>>> - >>>>> unsigned short pageofs_in, pageofs_out; >>>>> unsigned int inputsize, outputsize; >>>>> >>>>> - /* indicate the algorithm will be used for decompression */ >>>>> - unsigned int alg; >>>>> + unsigned int alg; /* the algorithm for decompression */ >>>>> bool inplace_io, partial_decoding, fillgaps; >>>>> + gfp_t gfp; /* allocation flags for extra temporary buffers */ >>>>> }; >>>>> >>>>> struct z_erofs_decompressor { >>>>> diff --git a/fs/erofs/decompressor.c b/fs/erofs/decompressor.c >>>>> index 1d65b9f60a39..ef2b08ec9830 100644 >>>>> --- a/fs/erofs/decompressor.c >>>>> +++ b/fs/erofs/decompressor.c >>>>> @@ -111,8 +111,9 @@ static int z_erofs_lz4_prepare_dstpages(struct z_erofs_lz4_decompress_ctx *ctx, >>>>> victim = availables[--top]; >>>>> get_page(victim); >>>>> } else { >>>>> - victim = erofs_allocpage(pagepool, >>>>> - GFP_KERNEL | __GFP_NOFAIL); >>>>> + victim = erofs_allocpage(pagepool, rq->gfp); >>>>> + if (!victim) >>>>> + return -ENOMEM; >>>>> set_page_private(victim, Z_EROFS_SHORTLIVED_PAGE); >>>>> } >>>>> rq->out[i] = victim; >>>>> diff --git a/fs/erofs/decompressor_deflate.c b/fs/erofs/decompressor_deflate.c >>>>> index 4a64a9c91dd3..b98872058abe 100644 >>>>> --- a/fs/erofs/decompressor_deflate.c >>>>> +++ b/fs/erofs/decompressor_deflate.c >>>>> @@ -95,7 +95,7 @@ int z_erofs_load_deflate_config(struct super_block *sb, >>>>> } >>>>> >>>>> int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, >>>>> - struct page **pagepool) >>>>> + struct page **pgpl) >>>>> { >>>>> const unsigned int nrpages_out = >>>>> PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT; >>>>> @@ -158,8 +158,12 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, >>>>> strm->z.avail_out = min_t(u32, outsz, PAGE_SIZE - pofs); >>>>> outsz -= strm->z.avail_out; >>>>> if (!rq->out[no]) { >>>>> - rq->out[no] = erofs_allocpage(pagepool, >>>>> - GFP_KERNEL | __GFP_NOFAIL); >>>>> + rq->out[no] = erofs_allocpage(pgpl, rq->gfp); >>>>> + if (!rq->out[no]) { >>>>> + kout = NULL; >>>>> + err = -ENOMEM; >>>>> + break; >>>>> + } >>>>> set_page_private(rq->out[no], >>>>> Z_EROFS_SHORTLIVED_PAGE); >>>>> } >>>>> @@ -211,8 +215,11 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, >>>>> >>>>> DBG_BUGON(erofs_page_is_managed(EROFS_SB(sb), >>>>> rq->in[j])); >>>>> - tmppage = erofs_allocpage(pagepool, >>>>> - GFP_KERNEL | __GFP_NOFAIL); >>>>> + tmppage = erofs_allocpage(pgpl, rq->gfp); >>>>> + if (!tmppage) { >>>>> + err = -ENOMEM; >>>>> + goto failed; >>>>> + } >>>>> set_page_private(tmppage, Z_EROFS_SHORTLIVED_PAGE); >>>>> copy_highpage(tmppage, rq->in[j]); >>>>> rq->in[j] = tmppage; >>>>> @@ -230,7 +237,7 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, >>>>> break; >>>>> } >>>>> } >>>>> - >>>>> +failed: >>>>> if (zlib_inflateEnd(&strm->z) != Z_OK && !err) >>>>> err = -EIO; >>>>> if (kout) >>>>> diff --git a/fs/erofs/decompressor_lzma.c b/fs/erofs/decompressor_lzma.c >>>>> index 2dd14f99c1dc..6ca357d83cfa 100644 >>>>> --- a/fs/erofs/decompressor_lzma.c >>>>> +++ b/fs/erofs/decompressor_lzma.c >>>>> @@ -148,7 +148,7 @@ int z_erofs_load_lzma_config(struct super_block *sb, >>>>> } >>>>> >>>>> int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, >>>>> - struct page **pagepool) >>>>> + struct page **pgpl) >>>>> { >>>>> const unsigned int nrpages_out = >>>>> PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT; >>>>> @@ -215,8 +215,11 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, >>>>> PAGE_SIZE - pageofs); >>>>> outlen -= strm->buf.out_size; >>>>> if (!rq->out[no] && rq->fillgaps) { /* deduped */ >>>>> - rq->out[no] = erofs_allocpage(pagepool, >>>>> - GFP_KERNEL | __GFP_NOFAIL); >>>>> + rq->out[no] = erofs_allocpage(pgpl, rq->gfp); >>>>> + if (!rq->out[no]) { >>>>> + err = -ENOMEM; >>>>> + break; >>>>> + } >>>>> set_page_private(rq->out[no], >>>>> Z_EROFS_SHORTLIVED_PAGE); >>>>> } >>>>> @@ -258,8 +261,11 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, >>>>> >>>>> DBG_BUGON(erofs_page_is_managed(EROFS_SB(rq->sb), >>>>> rq->in[j])); >>>>> - tmppage = erofs_allocpage(pagepool, >>>>> - GFP_KERNEL | __GFP_NOFAIL); >>>>> + tmppage = erofs_allocpage(pgpl, rq->gfp); >>>>> + if (!tmppage) { >>>>> + err = -ENOMEM; >>>>> + goto failed; >>>>> + } >>>>> set_page_private(tmppage, Z_EROFS_SHORTLIVED_PAGE); >>>>> copy_highpage(tmppage, rq->in[j]); >>>>> rq->in[j] = tmppage; >>>>> @@ -277,6 +283,7 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, >>>>> break; >>>>> } >>>>> } >>>>> +failed: >>>>> if (no < nrpages_out && strm->buf.out) >>>>> kunmap(rq->out[no]); >>>>> if (ni < nrpages_in) >>>>> diff --git a/fs/erofs/zdata.c b/fs/erofs/zdata.c >>>>> index 692c0c39be63..a293de2a60ed 100644 >>>>> --- a/fs/erofs/zdata.c >>>>> +++ b/fs/erofs/zdata.c >>>>> @@ -82,6 +82,9 @@ struct z_erofs_pcluster { >>>>> /* L: indicate several pageofs_outs or not */ >>>>> bool multibases; >>>>> >>>>> + /* L: whether extra buffer allocations are best-effort */ >>>>> + bool besteffort; >>>>> + >>>>> /* A: compressed bvecs (can be cached or inplaced pages) */ >>>>> struct z_erofs_bvec compressed_bvecs[]; >>>>> }; >>>>> @@ -964,7 +967,7 @@ static int z_erofs_read_fragment(struct super_block *sb, struct page *page, >>>>> } >>>>> >>>>> static int z_erofs_do_read_page(struct z_erofs_decompress_frontend *fe, >>>>> - struct page *page) >>>>> + struct page *page, bool ra) >>>>> { >>>>> struct inode *const inode = fe->inode; >>>>> struct erofs_map_blocks *const map = &fe->map; >>>>> @@ -1014,6 +1017,7 @@ static int z_erofs_do_read_page(struct z_erofs_decompress_frontend *fe, >>>>> err = z_erofs_pcluster_begin(fe); >>>>> if (err) >>>>> goto out; >>>>> + fe->pcl->besteffort |= !ra; >>>>> } >>>>> >>>>> /* >>>>> @@ -1280,7 +1284,11 @@ static int z_erofs_decompress_pcluster(struct z_erofs_decompress_backend *be, >>>>> .inplace_io = overlapped, >>>>> .partial_decoding = pcl->partial, >>>>> .fillgaps = pcl->multibases, >>>>> + .gfp = pcl->besteffort ? >>>>> + GFP_KERNEL | __GFP_NOFAIL : >>>>> + GFP_NOWAIT | __GFP_NORETRY >>>>> }, be->pagepool); >>>>> + pcl->besteffort = false; >>>>> >>>>> /* must handle all compressed pages before actual file pages */ >>>>> if (z_erofs_is_inline_pcluster(pcl)) { >>>>> @@ -1785,7 +1793,7 @@ static void z_erofs_pcluster_readmore(struct z_erofs_decompress_frontend *f, >>>>> if (PageUptodate(page)) >>>>> unlock_page(page); >>>>> else >>>>> - (void)z_erofs_do_read_page(f, page); >>>>> + (void)z_erofs_do_read_page(f, page, !!rac); >>>>> put_page(page); >>>>> } >>>>> >>>>> @@ -1806,7 +1814,7 @@ static int z_erofs_read_folio(struct file *file, struct folio *folio) >>>>> f.headoffset = (erofs_off_t)folio->index << PAGE_SHIFT; >>>>> >>>>> z_erofs_pcluster_readmore(&f, NULL, true); >>>>> - err = z_erofs_do_read_page(&f, &folio->page); >>>>> + err = z_erofs_do_read_page(&f, &folio->page, false); >>>>> z_erofs_pcluster_readmore(&f, NULL, false); >>>>> z_erofs_pcluster_end(&f); >>>>> >>>>> @@ -1847,7 +1855,7 @@ static void z_erofs_readahead(struct readahead_control *rac) >>>>> folio = head; >>>>> head = folio_get_private(folio); >>>>> >>>>> - err = z_erofs_do_read_page(&f, &folio->page); >>>>> + err = z_erofs_do_read_page(&f, &folio->page, true); >>>>> if (err && err != -EINTR) >>>>> erofs_err(inode->i_sb, "readahead error at folio %lu @ nid %llu", >>>>> folio->index, EROFS_I(inode)->nid); ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v2] erofs: relaxed temporary buffers allocation on readahead 2024-01-26 2:41 ` Chunhai Guo via Linux-erofs @ 2024-01-26 2:47 ` Gao Xiang 2024-01-26 3:42 ` Chunhai Guo via Linux-erofs 0 siblings, 1 reply; 20+ messages in thread From: Gao Xiang @ 2024-01-26 2:47 UTC (permalink / raw) To: Chunhai Guo, xiang@kernel.org Cc: linux-erofs@lists.ozlabs.org, huyue2@coolpad.com On 2024/1/26 10:41, Chunhai Guo wrote: > On 2024/1/22 15:42, Chunhai Guo wrote: >> On 2024/1/22 12:37, Gao Xiang wrote: >>> [你通常不会收到来自 hsiangkao@linux.alibaba.com 的电子邮件。请访问 https://aka.ms/LearnAboutSenderIdentification，以了解这一点为什么很重要] >>> >>> On 2024/1/22 11:49, Chunhai Guo wrote: >>>> On 2024/1/22 10:07, Gao Xiang wrote: >>>>> [你通常不会收到来自 hsiangkao@linux.alibaba.com 的电子邮件。请访问 https://aka.ms/LearnAboutSenderIdentification，以了解这一点为什么很重要] >>>>> >>>>> On 2024/1/20 22:55, Chunhai Guo wrote: >>>>>> Even with inplace decompression, sometimes extra temporary buffers are >>>>>> still needed for decompression. In low-memory scenarios, it would be >>>>>> better to try to allocate with GFP_NOWAIT on readahead first. That can >>>>>> help reduce the time spent on page allocation under memory pressure. >>>>>> >>>>>> There is an average reduction of 21% in page allocation time under >>>>> It would be better to add a table to show the absolute numbers too >>>>> (like what you did in the global pool commit.) If it's possible, there >>>>> is no need to send a update version for this, just reply the updated >>>>> commit message and I will update the commit manually. >>>> The table below shows detailed numbers. The reduction I mentioned before >>>> was not accurate enough. Please help correct the improvement from 21% to >>>> 20.21%. >>>> >>>> >>>> +--------------+----------------+---------------+---------+ >>>> | | w/o GFP_NOWAIT | w/ GFP_NOWAIT | diff | >>>> +--------------+----------------+---------------+---------+ >>>> | Average (ms) | 3364 | 2684 | -20.21% | >>>> +--------------+----------------+---------------+---------+ >>> Did it test without the 16k sliding window change? >>> https://lore.kernel.org/linux-erofs/69711d55-f7a2-420b-9ba8-fa2921f66a4c@vivo.com >> The result is tested with 64k sliding window change. >> >>> Could you benchmark these two optimizations together to >>> show the extreme optimized case without a global pool? >>> With a new table if possible? I will add this to >>> the commit message too. >> >> OK. I will reply to this email when the benchmark is finished. > > The benchmark has been completed and the table below shows that there is > an average 52.14% reduction in page allocation time with these two > optimizations. > > +--------------+----------------+---------------+---------+ | | 64k > window | 16k window | | | | w/o GFP_NOWAIT | w/ GFP_NOWAIT | diff | > +--------------+----------------+---------------+---------+ | Average > (ms) | 3364 | 1610 | -52.14% | > +--------------+----------------+---------------+---------+ > > Table below summarizes the results of these three benchmarks. > > +--------------+----------------+----------------+---------------+---------------+ > | | 64k window | 16k window | 64k window | 16k > window | > | | w/o GFP_NOWAIT | w/o GFP_NOWAIT | w/ GFP_NOWAIT | w/ > GFP_NOWAIT | > +--------------+----------------+----------------+---------------+---------------+ > | Average (ms) | 3364 | 2079 | 2684 | > 1610 | > +--------------+----------------+----------------+---------------+---------------+ > | diff | | -38.19% | -20.81% | > -52.14% | > +--------------+----------------+----------------+---------------+---------------+ The tables shows in a mess, could you just list the numbers so I could refine this? Thanks, Gao Xiang ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v2] erofs: relaxed temporary buffers allocation on readahead 2024-01-26 2:47 ` Gao Xiang @ 2024-01-26 3:42 ` Chunhai Guo via Linux-erofs 2024-01-26 3:49 ` Gao Xiang 0 siblings, 1 reply; 20+ messages in thread From: Chunhai Guo via Linux-erofs @ 2024-01-26 3:42 UTC (permalink / raw) To: Gao Xiang, Chunhai Guo, xiang@kernel.org Cc: linux-erofs@lists.ozlabs.org, huyue2@coolpad.com On 2024/1/26 10:47, Gao Xiang wrote: > [你通常不会收到来自 hsiangkao@linux.alibaba.com 的电子邮件。请访问 https://aka.ms/LearnAboutSenderIdentification，以了解这一点为什么很重要] > > On 2024/1/26 10:41, Chunhai Guo wrote: >> On 2024/1/22 15:42, Chunhai Guo wrote: >>> On 2024/1/22 12:37, Gao Xiang wrote: >>>> [你通常不会收到来自 hsiangkao@linux.alibaba.com 的电子邮件。请访问 https://aka.ms/LearnAboutSenderIdentification，以了解这一点为什么很重要] >>>> >>>> On 2024/1/22 11:49, Chunhai Guo wrote: >>>>> On 2024/1/22 10:07, Gao Xiang wrote: >>>>>> [你通常不会收到来自 hsiangkao@linux.alibaba.com 的电子邮件。请访问 https://aka.ms/LearnAboutSenderIdentification，以了解这一点为什么很重要] >>>>>> >>>>>> On 2024/1/20 22:55, Chunhai Guo wrote: >>>>>>> Even with inplace decompression, sometimes extra temporary buffers are >>>>>>> still needed for decompression. In low-memory scenarios, it would be >>>>>>> better to try to allocate with GFP_NOWAIT on readahead first. That can >>>>>>> help reduce the time spent on page allocation under memory pressure. >>>>>>> >>>>>>> There is an average reduction of 21% in page allocation time under >>>>>> It would be better to add a table to show the absolute numbers too >>>>>> (like what you did in the global pool commit.) If it's possible, there >>>>>> is no need to send a update version for this, just reply the updated >>>>>> commit message and I will update the commit manually. >>>>> The table below shows detailed numbers. The reduction I mentioned before >>>>> was not accurate enough. Please help correct the improvement from 21% to >>>>> 20.21%. >>>>> >>>>> >>>>> +--------------+----------------+---------------+---------+ >>>>> | | w/o GFP_NOWAIT | w/ GFP_NOWAIT | diff | >>>>> +--------------+----------------+---------------+---------+ >>>>> | Average (ms) | 3364 | 2684 | -20.21% | >>>>> +--------------+----------------+---------------+---------+ >>>> Did it test without the 16k sliding window change? >>>> https://lore.kernel.org/linux-erofs/69711d55-f7a2-420b-9ba8-fa2921f66a4c@vivo.com >>> The result is tested with 64k sliding window change. >>> >>>> Could you benchmark these two optimizations together to >>>> show the extreme optimized case without a global pool? >>>> With a new table if possible? I will add this to >>>> the commit message too. >>> OK. I will reply to this email when the benchmark is finished. >> The benchmark has been completed and the table below shows that there is >> an average 52.14% reduction in page allocation time with these two >> optimizations. >> >> +--------------+----------------+---------------+---------+ | | 64k >> window | 16k window | | | | w/o GFP_NOWAIT | w/ GFP_NOWAIT | diff | >> +--------------+----------------+---------------+---------+ | Average >> (ms) | 3364 | 1610 | -52.14% | >> +--------------+----------------+---------------+---------+ >> >> Table below summarizes the results of these three benchmarks. >> >> +--------------+----------------+----------------+---------------+---------------+ >> | | 64k window | 16k window | 64k window | 16k >> window | >> | | w/o GFP_NOWAIT | w/o GFP_NOWAIT | w/ GFP_NOWAIT | w/ >> GFP_NOWAIT | >> +--------------+----------------+----------------+---------------+---------------+ >> | Average (ms) | 3364 | 2079 | 2684 | >> 1610 | >> +--------------+----------------+----------------+---------------+---------------+ >> | diff | | -38.19% | -20.81% | >> -52.14% | >> +--------------+----------------+----------------+---------------+---------------+ > > The tables shows in a mess, could you just list the > numbers so I could refine this? Sorry that there might be some issues with my email client. Here are the numerical results below. 64k window w/o GFP_NOWAIT : 3364 16k window w/o GFP_NOWAIT : 2079, diff: -38.19% 64k window w/ GFP_NOWAIT : 2684, diff: -20.81% 16k window w/ GFP_NOWAIT : 1610, diff: -52.14% Images size comparision: 64k: 9117044 KB 16k: 9113096 KB Thanks, > > Thanks, > Gao Xiang ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v2] erofs: relaxed temporary buffers allocation on readahead 2024-01-26 3:42 ` Chunhai Guo via Linux-erofs @ 2024-01-26 3:49 ` Gao Xiang 2024-01-26 3:56 ` 答复: " Chunhai Guo via Linux-erofs 0 siblings, 1 reply; 20+ messages in thread From: Gao Xiang @ 2024-01-26 3:49 UTC (permalink / raw) To: Chunhai Guo, xiang@kernel.org Cc: linux-erofs@lists.ozlabs.org, huyue2@coolpad.com On 2024/1/26 11:42, Chunhai Guo wrote: > On 2024/1/26 10:47, Gao Xiang wrote: >> [你通常不会收到来自 hsiangkao@linux.alibaba.com 的电子邮件。请访问 https://aka.ms/LearnAboutSenderIdentification，以了解这一点为什么很重要] >> >> On 2024/1/26 10:41, Chunhai Guo wrote: >>> On 2024/1/22 15:42, Chunhai Guo wrote: >>>> On 2024/1/22 12:37, Gao Xiang wrote: >>>>> [你通常不会收到来自 hsiangkao@linux.alibaba.com 的电子邮件。请访问 https://aka.ms/LearnAboutSenderIdentification，以了解这一点为什么很重要] >>>>> >>>>> On 2024/1/22 11:49, Chunhai Guo wrote: >>>>>> On 2024/1/22 10:07, Gao Xiang wrote: >>>>>>> [你通常不会收到来自 hsiangkao@linux.alibaba.com 的电子邮件。请访问 https://aka.ms/LearnAboutSenderIdentification，以了解这一点为什么很重要] >>>>>>> >>>>>>> On 2024/1/20 22:55, Chunhai Guo wrote: >>>>>>>> Even with inplace decompression, sometimes extra temporary buffers are >>>>>>>> still needed for decompression. In low-memory scenarios, it would be >>>>>>>> better to try to allocate with GFP_NOWAIT on readahead first. That can >>>>>>>> help reduce the time spent on page allocation under memory pressure. >>>>>>>> >>>>>>>> There is an average reduction of 21% in page allocation time under >>>>>>> It would be better to add a table to show the absolute numbers too >>>>>>> (like what you did in the global pool commit.) If it's possible, there >>>>>>> is no need to send a update version for this, just reply the updated >>>>>>> commit message and I will update the commit manually. >>>>>> The table below shows detailed numbers. The reduction I mentioned before >>>>>> was not accurate enough. Please help correct the improvement from 21% to >>>>>> 20.21%. >>>>>> >>>>>> >>>>>> +--------------+----------------+---------------+---------+ >>>>>> | | w/o GFP_NOWAIT | w/ GFP_NOWAIT | diff | >>>>>> +--------------+----------------+---------------+---------+ >>>>>> | Average (ms) | 3364 | 2684 | -20.21% | >>>>>> +--------------+----------------+---------------+---------+ >>>>> Did it test without the 16k sliding window change? >>>>> https://lore.kernel.org/linux-erofs/69711d55-f7a2-420b-9ba8-fa2921f66a4c@vivo.com >>>> The result is tested with 64k sliding window change. >>>> >>>>> Could you benchmark these two optimizations together to >>>>> show the extreme optimized case without a global pool? >>>>> With a new table if possible? I will add this to >>>>> the commit message too. >>>> OK. I will reply to this email when the benchmark is finished. >>> The benchmark has been completed and the table below shows that there is >>> an average 52.14% reduction in page allocation time with these two >>> optimizations. >>> >>> +--------------+----------------+---------------+---------+ | | 64k >>> window | 16k window | | | | w/o GFP_NOWAIT | w/ GFP_NOWAIT | diff | >>> +--------------+----------------+---------------+---------+ | Average >>> (ms) | 3364 | 1610 | -52.14% | >>> +--------------+----------------+---------------+---------+ >>> >>> Table below summarizes the results of these three benchmarks. >>> >>> +--------------+----------------+----------------+---------------+---------------+ >>> | | 64k window | 16k window | 64k window | 16k >>> window | >>> | | w/o GFP_NOWAIT | w/o GFP_NOWAIT | w/ GFP_NOWAIT | w/ >>> GFP_NOWAIT | >>> +--------------+----------------+----------------+---------------+---------------+ >>> | Average (ms) | 3364 | 2079 | 2684 | >>> 1610 | >>> +--------------+----------------+----------------+---------------+---------------+ >>> | diff | | -38.19% | -20.81% | >>> -52.14% | >>> +--------------+----------------+----------------+---------------+---------------+ >> >> The tables shows in a mess, could you just list the >> numbers so I could refine this? > > Sorry that there might be some issues with my email client. Here are the > numerical results below. > 64k window w/o GFP_NOWAIT : 3364 > 16k window w/o GFP_NOWAIT : 2079, diff: -38.19% > 64k window w/ GFP_NOWAIT : 2684, diff: -20.81% > 16k window w/ GFP_NOWAIT : 1610, diff: -52.14% > > Images size comparision: > 64k: 9117044 KB > 16k: 9113096 KB That is with 4k pcluster, yes? I guess the overall image size won't have great impacts, but it seems even getting smaller. :-) I think this optimization would be helpful to everyone without any extra memory reservation (which will be good too for much much low-ended devices), let me revise the commit for formal submission.. Thanks, Gao Xiang > > Thanks, > >> >> Thanks, >> Gao Xiang > > ^ permalink raw reply [flat|nested] 20+ messages in thread
* 答复: [PATCH v2] erofs: relaxed temporary buffers allocation on readahead 2024-01-26 3:49 ` Gao Xiang @ 2024-01-26 3:56 ` Chunhai Guo via Linux-erofs 2024-01-26 5:36 ` Gao Xiang 0 siblings, 1 reply; 20+ messages in thread From: Chunhai Guo via Linux-erofs @ 2024-01-26 3:56 UTC (permalink / raw) To: Gao Xiang, xiang@kernel.org Cc: linux-erofs@lists.ozlabs.org, huyue2@coolpad.com > -----邮件原件----- > 发件人: Gao Xiang <hsiangkao@linux.alibaba.com> > 发送时间: 2024年1月26日 11:50 > 收件人: Chunhai Guo <guochunhai@vivo.com>; xiang@kernel.org > 抄送: chao@kernel.org; huyue2@coolpad.com; jefflexu@linux.alibaba.com; > linux-erofs@lists.ozlabs.org > 主题: Re: [PATCH v2] erofs: relaxed temporary buffers allocation on readahead > > [你通常不会收到来自 hsiangkao@linux.alibaba.com 的电子邮件。请访问 > https://aka.ms/LearnAboutSenderIdentification，以了解这一点为什么很重要] > > On 2024/1/26 11:42, Chunhai Guo wrote: > > On 2024/1/26 10:47, Gao Xiang wrote: > >> [你通常不会收到来自 hsiangkao@linux.alibaba.com 的电子邮件。请访问 > >> https://aka.ms/LearnAboutSenderIdentification，以了解这一点为什么很重 > 要] > >> > >> On 2024/1/26 10:41, Chunhai Guo wrote: > >>> On 2024/1/22 15:42, Chunhai Guo wrote: > >>>> On 2024/1/22 12:37, Gao Xiang wrote: > >>>>> [你通常不会收到来自 hsiangkao@linux.alibaba.com 的电子邮件。请访 > 问 > >>>>> https://aka.ms/LearnAboutSenderIdentification，以了解这一点为什么 > 很重要] > >>>>> > >>>>> On 2024/1/22 11:49, Chunhai Guo wrote: > >>>>>> On 2024/1/22 10:07, Gao Xiang wrote: > >>>>>>> [你通常不会收到来自 hsiangkao@linux.alibaba.com 的电子邮件。请 > 访问 > >>>>>>> https://aka.ms/LearnAboutSenderIdentification，以了解这一点为什 > 么很重要] > >>>>>>> > >>>>>>> On 2024/1/20 22:55, Chunhai Guo wrote: > >>>>>>>> Even with inplace decompression, sometimes extra temporary > >>>>>>>> buffers are still needed for decompression. In low-memory > >>>>>>>> scenarios, it would be better to try to allocate with > >>>>>>>> GFP_NOWAIT on readahead first. That can help reduce the time spent > on page allocation under memory pressure. > >>>>>>>> > >>>>>>>> There is an average reduction of 21% in page allocation time > >>>>>>>> under > >>>>>>> It would be better to add a table to show the absolute numbers > >>>>>>> too (like what you did in the global pool commit.) If it's > >>>>>>> possible, there is no need to send a update version for this, > >>>>>>> just reply the updated commit message and I will update the commit > manually. > >>>>>> The table below shows detailed numbers. The reduction I mentioned > >>>>>> before was not accurate enough. Please help correct the > >>>>>> improvement from 21% to 20.21%. > >>>>>> > >>>>>> > >>>>>> +--------------+----------------+---------------+---------+ > >>>>>> | | w/o GFP_NOWAIT | w/ GFP_NOWAIT | diff | > >>>>>> +--------------+----------------+---------------+---------+ > >>>>>> | Average (ms) | 3364 | 2684 | -20.21% | > >>>>>> +--------------+----------------+---------------+---------+ > >>>>> Did it test without the 16k sliding window change? > >>>>> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2F > >>>>> lore.kernel.org%2Flinux-erofs%2F69711d55-f7a2-420b-9ba8-fa2921f66a > >>>>> > 4c%40vivo.com&data=05%7C02%7Cguochunhai%40vivo.com%7Ceadb2eb3d04 > 74 > >>>>> > b3b905708dc1e21d8a0%7C923e42dc48d54cbeb5821a797a6412ed%7C0%7C0% > 7C6 > >>>>> > 38418377978918986%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAi > LCJQ > >>>>> > IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=I > >>>>> QzcxbNhF8ZbG0zCnxAQTba6C3DU6tUC7bzZaISLYJE%3D&reserved=0 > >>>> The result is tested with 64k sliding window change. > >>>> > >>>>> Could you benchmark these two optimizations together to show the > >>>>> extreme optimized case without a global pool? > >>>>> With a new table if possible? I will add this to the commit > >>>>> message too. > >>>> OK. I will reply to this email when the benchmark is finished. > >>> The benchmark has been completed and the table below shows that > >>> there is an average 52.14% reduction in page allocation time with > >>> these two optimizations. > >>> > >>> +--------------+----------------+---------------+---------+ | | 64k > >>> window | 16k window | | | | w/o GFP_NOWAIT | w/ GFP_NOWAIT | diff | > >>> +--------------+----------------+---------------+---------+ | Averag > >>> +--------------+----------------+---------------+---------+ | e > >>> (ms) | 3364 | 1610 | -52.14% | > >>> +--------------+----------------+---------------+---------+ > >>> > >>> Table below summarizes the results of these three benchmarks. > >>> > >>> +--------------+----------------+----------------+---------------+---------------+ > >>> | | 64k window | 16k window | 64k window | 16k > >>> window | > >>> | | w/o GFP_NOWAIT | w/o GFP_NOWAIT | w/ GFP_NOWAIT | > >>> | w/ > >>> GFP_NOWAIT | > >>> +--------------+----------------+----------------+---------------+---------------+ > >>> | Average (ms) | 3364 | 2079 | 2684 | > >>> 1610 | > >>> +--------------+----------------+----------------+---------------+---------------+ > >>> | diff | | -38.19% | -20.81% | > >>> -52.14% | > >>> +--------------+----------------+----------------+---------------+---------------+ > >> > >> The tables shows in a mess, could you just list the numbers so I > >> could refine this? > > > > Sorry that there might be some issues with my email client. Here are > > the numerical results below. > > 64k window w/o GFP_NOWAIT : 3364 > > 16k window w/o GFP_NOWAIT : 2079, diff: -38.19% > > 64k window w/ GFP_NOWAIT : 2684, diff: -20.81% > > 16k window w/ GFP_NOWAIT : 1610, diff: -52.14% > > > > Images size comparision: > > 64k: 9117044 KB > > 16k: 9113096 KB > > That is with 4k pcluster, yes? I guess the overall image size won't have great > impacts, but it seems even getting smaller. :-) Yes, this is with 4k pcluster. Thanks, > > I think this optimization would be helpful to everyone without any extra memory > reservation (which will be good too for much much low-ended devices), let me > revise the commit for formal submission.. > > Thanks, > Gao Xiang > > > > > Thanks, > > > >> > >> Thanks, > >> Gao Xiang > > > > ^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v3] erofs: relaxed temporary buffers allocation on readahead 2024-01-26 3:56 ` 答复: " Chunhai Guo via Linux-erofs @ 2024-01-26 5:36 ` Gao Xiang 0 siblings, 0 replies; 20+ messages in thread From: Gao Xiang @ 2024-01-26 5:36 UTC (permalink / raw) To: linux-erofs; +Cc: LKML, Yue Hu, Gao Xiang, Chunhai Guo From: Chunhai Guo <guochunhai@vivo.com> Even with inplace decompression, sometimes very few temporary buffers are still needed for a single decompression shot (e.g. 16 pages for 64k sliding window or 4 pages for 16k sliding window). In low-memory scenarios, it would be better to try to allocate with GFP_NOWAIT on readahead first. That can help reduce the time spent on page allocation under durative memory pressure. Here are detailed performance numbers under multi-app launch benchmark workload [1] on ARM64 Android devices (8-core CPU and 8GB of memory) running a 5.15 LTS kernel with EROFS of 4k pclusters: +----------------+---------+---------+---------+ | LZ4 | vanilla | patched | diff | |----------------+---------+---------+---------| | Average (ms) | 3364 | 2684 | -20.21% | [64k sliding window] |----------------+---------+---------+---------| | Average (ms) | 2079 | 1610 | -22.56% | [16k sliding window] +----------------+---------+---------+---------+ The total size of system images for 4k pcluster is almost unchanged: (64k sliding window) 9,117,044 KB (16k sliding window) 9,113,096 KB Therefore, in addition to switch the sliding window from 64k to 16k, after applying this patch, it can eventually save 52.14% (3364 -> 1610) on average with no memory reservation. That is particularly useful for embedded devices with limited resources. [1] https://lore.kernel.org/r/20240109074143.4138783-1-guochunhai@vivo.com Suggested-by: Gao Xiang <xiang@kernel.org> Signed-off-by: Chunhai Guo <guochunhai@vivo.com> Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com> --- v2: https://lore.kernel.org/r/20240120145551.1941483-1-guochunhai@vivo.com change since v2: - update commit message according to test results. I plan to apply this version. fs/erofs/compress.h | 5 ++--- fs/erofs/decompressor.c | 5 +++-- fs/erofs/decompressor_deflate.c | 19 +++++++++++++------ fs/erofs/decompressor_lzma.c | 17 ++++++++++++----- fs/erofs/zdata.c | 16 ++++++++++++---- 5 files changed, 42 insertions(+), 20 deletions(-) diff --git a/fs/erofs/compress.h b/fs/erofs/compress.h index 279933e007d2..7cc5841577b2 100644 --- a/fs/erofs/compress.h +++ b/fs/erofs/compress.h @@ -11,13 +11,12 @@ struct z_erofs_decompress_req { struct super_block *sb; struct page **in, **out; - unsigned short pageofs_in, pageofs_out; unsigned int inputsize, outputsize; - /* indicate the algorithm will be used for decompression */ - unsigned int alg; + unsigned int alg; /* the algorithm for decompression */ bool inplace_io, partial_decoding, fillgaps; + gfp_t gfp; /* allocation flags for extra temporary buffers */ }; struct z_erofs_decompressor { diff --git a/fs/erofs/decompressor.c b/fs/erofs/decompressor.c index 072ef6a66823..d4cee95af14c 100644 --- a/fs/erofs/decompressor.c +++ b/fs/erofs/decompressor.c @@ -111,8 +111,9 @@ static int z_erofs_lz4_prepare_dstpages(struct z_erofs_lz4_decompress_ctx *ctx, victim = availables[--top]; get_page(victim); } else { - victim = erofs_allocpage(pagepool, - GFP_KERNEL | __GFP_NOFAIL); + victim = erofs_allocpage(pagepool, rq->gfp); + if (!victim) + return -ENOMEM; set_page_private(victim, Z_EROFS_SHORTLIVED_PAGE); } rq->out[i] = victim; diff --git a/fs/erofs/decompressor_deflate.c b/fs/erofs/decompressor_deflate.c index 4a64a9c91dd3..b98872058abe 100644 --- a/fs/erofs/decompressor_deflate.c +++ b/fs/erofs/decompressor_deflate.c @@ -95,7 +95,7 @@ int z_erofs_load_deflate_config(struct super_block *sb, } int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, - struct page **pagepool) + struct page **pgpl) { const unsigned int nrpages_out = PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT; @@ -158,8 +158,12 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, strm->z.avail_out = min_t(u32, outsz, PAGE_SIZE - pofs); outsz -= strm->z.avail_out; if (!rq->out[no]) { - rq->out[no] = erofs_allocpage(pagepool, - GFP_KERNEL | __GFP_NOFAIL); + rq->out[no] = erofs_allocpage(pgpl, rq->gfp); + if (!rq->out[no]) { + kout = NULL; + err = -ENOMEM; + break; + } set_page_private(rq->out[no], Z_EROFS_SHORTLIVED_PAGE); } @@ -211,8 +215,11 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, DBG_BUGON(erofs_page_is_managed(EROFS_SB(sb), rq->in[j])); - tmppage = erofs_allocpage(pagepool, - GFP_KERNEL | __GFP_NOFAIL); + tmppage = erofs_allocpage(pgpl, rq->gfp); + if (!tmppage) { + err = -ENOMEM; + goto failed; + } set_page_private(tmppage, Z_EROFS_SHORTLIVED_PAGE); copy_highpage(tmppage, rq->in[j]); rq->in[j] = tmppage; @@ -230,7 +237,7 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, break; } } - +failed: if (zlib_inflateEnd(&strm->z) != Z_OK && !err) err = -EIO; if (kout) diff --git a/fs/erofs/decompressor_lzma.c b/fs/erofs/decompressor_lzma.c index 2dd14f99c1dc..6ca357d83cfa 100644 --- a/fs/erofs/decompressor_lzma.c +++ b/fs/erofs/decompressor_lzma.c @@ -148,7 +148,7 @@ int z_erofs_load_lzma_config(struct super_block *sb, } int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, - struct page **pagepool) + struct page **pgpl) { const unsigned int nrpages_out = PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT; @@ -215,8 +215,11 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, PAGE_SIZE - pageofs); outlen -= strm->buf.out_size; if (!rq->out[no] && rq->fillgaps) { /* deduped */ - rq->out[no] = erofs_allocpage(pagepool, - GFP_KERNEL | __GFP_NOFAIL); + rq->out[no] = erofs_allocpage(pgpl, rq->gfp); + if (!rq->out[no]) { + err = -ENOMEM; + break; + } set_page_private(rq->out[no], Z_EROFS_SHORTLIVED_PAGE); } @@ -258,8 +261,11 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, DBG_BUGON(erofs_page_is_managed(EROFS_SB(rq->sb), rq->in[j])); - tmppage = erofs_allocpage(pagepool, - GFP_KERNEL | __GFP_NOFAIL); + tmppage = erofs_allocpage(pgpl, rq->gfp); + if (!tmppage) { + err = -ENOMEM; + goto failed; + } set_page_private(tmppage, Z_EROFS_SHORTLIVED_PAGE); copy_highpage(tmppage, rq->in[j]); rq->in[j] = tmppage; @@ -277,6 +283,7 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, break; } } +failed: if (no < nrpages_out && strm->buf.out) kunmap(rq->out[no]); if (ni < nrpages_in) diff --git a/fs/erofs/zdata.c b/fs/erofs/zdata.c index c1c77166b30f..1d0fdc145fd6 100644 --- a/fs/erofs/zdata.c +++ b/fs/erofs/zdata.c @@ -82,6 +82,9 @@ struct z_erofs_pcluster { /* L: indicate several pageofs_outs or not */ bool multibases; + /* L: whether extra buffer allocations are best-effort */ + bool besteffort; + /* A: compressed bvecs (can be cached or inplaced pages) */ struct z_erofs_bvec compressed_bvecs[]; }; @@ -960,7 +963,7 @@ static int z_erofs_read_fragment(struct super_block *sb, struct page *page, } static int z_erofs_do_read_page(struct z_erofs_decompress_frontend *fe, - struct page *page) + struct page *page, bool ra) { struct inode *const inode = fe->inode; struct erofs_map_blocks *const map = &fe->map; @@ -1010,6 +1013,7 @@ static int z_erofs_do_read_page(struct z_erofs_decompress_frontend *fe, err = z_erofs_pcluster_begin(fe); if (err) goto out; + fe->pcl->besteffort |= !ra; } /* @@ -1276,7 +1280,11 @@ static int z_erofs_decompress_pcluster(struct z_erofs_decompress_backend *be, .inplace_io = overlapped, .partial_decoding = pcl->partial, .fillgaps = pcl->multibases, + .gfp = pcl->besteffort ? + GFP_KERNEL | __GFP_NOFAIL : + GFP_NOWAIT | __GFP_NORETRY }, be->pagepool); + pcl->besteffort = false; /* must handle all compressed pages before actual file pages */ if (z_erofs_is_inline_pcluster(pcl)) { @@ -1787,7 +1795,7 @@ static void z_erofs_pcluster_readmore(struct z_erofs_decompress_frontend *f, if (PageUptodate(page)) unlock_page(page); else - (void)z_erofs_do_read_page(f, page); + (void)z_erofs_do_read_page(f, page, !!rac); put_page(page); } @@ -1808,7 +1816,7 @@ static int z_erofs_read_folio(struct file *file, struct folio *folio) f.headoffset = (erofs_off_t)folio->index << PAGE_SHIFT; z_erofs_pcluster_readmore(&f, NULL, true); - err = z_erofs_do_read_page(&f, &folio->page); + err = z_erofs_do_read_page(&f, &folio->page, false); z_erofs_pcluster_readmore(&f, NULL, false); z_erofs_pcluster_end(&f); @@ -1849,7 +1857,7 @@ static void z_erofs_readahead(struct readahead_control *rac) folio = head; head = folio_get_private(folio); - err = z_erofs_do_read_page(&f, &folio->page); + err = z_erofs_do_read_page(&f, &folio->page, true); if (err && err != -EINTR) erofs_err(inode->i_sb, "readahead error at folio %lu @ nid %llu", folio->index, EROFS_I(inode)->nid); -- 2.39.3 ^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v3] erofs: relaxed temporary buffers allocation on readahead @ 2024-01-26 5:36 ` Gao Xiang 0 siblings, 0 replies; 20+ messages in thread From: Gao Xiang @ 2024-01-26 5:36 UTC (permalink / raw) To: linux-erofs; +Cc: LKML, Chao Yu, Yue Hu, Chunhai Guo, Gao Xiang, Gao Xiang From: Chunhai Guo <guochunhai@vivo.com> Even with inplace decompression, sometimes very few temporary buffers are still needed for a single decompression shot (e.g. 16 pages for 64k sliding window or 4 pages for 16k sliding window). In low-memory scenarios, it would be better to try to allocate with GFP_NOWAIT on readahead first. That can help reduce the time spent on page allocation under durative memory pressure. Here are detailed performance numbers under multi-app launch benchmark workload [1] on ARM64 Android devices (8-core CPU and 8GB of memory) running a 5.15 LTS kernel with EROFS of 4k pclusters: +----------------+---------+---------+---------+ | LZ4 | vanilla | patched | diff | |----------------+---------+---------+---------| | Average (ms) | 3364 | 2684 | -20.21% | [64k sliding window] |----------------+---------+---------+---------| | Average (ms) | 2079 | 1610 | -22.56% | [16k sliding window] +----------------+---------+---------+---------+ The total size of system images for 4k pcluster is almost unchanged: (64k sliding window) 9,117,044 KB (16k sliding window) 9,113,096 KB Therefore, in addition to switch the sliding window from 64k to 16k, after applying this patch, it can eventually save 52.14% (3364 -> 1610) on average with no memory reservation. That is particularly useful for embedded devices with limited resources. [1] https://lore.kernel.org/r/20240109074143.4138783-1-guochunhai@vivo.com Suggested-by: Gao Xiang <xiang@kernel.org> Signed-off-by: Chunhai Guo <guochunhai@vivo.com> Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com> --- v2: https://lore.kernel.org/r/20240120145551.1941483-1-guochunhai@vivo.com change since v2: - update commit message according to test results. I plan to apply this version. fs/erofs/compress.h | 5 ++--- fs/erofs/decompressor.c | 5 +++-- fs/erofs/decompressor_deflate.c | 19 +++++++++++++------ fs/erofs/decompressor_lzma.c | 17 ++++++++++++----- fs/erofs/zdata.c | 16 ++++++++++++---- 5 files changed, 42 insertions(+), 20 deletions(-) diff --git a/fs/erofs/compress.h b/fs/erofs/compress.h index 279933e007d2..7cc5841577b2 100644 --- a/fs/erofs/compress.h +++ b/fs/erofs/compress.h @@ -11,13 +11,12 @@ struct z_erofs_decompress_req { struct super_block *sb; struct page **in, **out; - unsigned short pageofs_in, pageofs_out; unsigned int inputsize, outputsize; - /* indicate the algorithm will be used for decompression */ - unsigned int alg; + unsigned int alg; /* the algorithm for decompression */ bool inplace_io, partial_decoding, fillgaps; + gfp_t gfp; /* allocation flags for extra temporary buffers */ }; struct z_erofs_decompressor { diff --git a/fs/erofs/decompressor.c b/fs/erofs/decompressor.c index 072ef6a66823..d4cee95af14c 100644 --- a/fs/erofs/decompressor.c +++ b/fs/erofs/decompressor.c @@ -111,8 +111,9 @@ static int z_erofs_lz4_prepare_dstpages(struct z_erofs_lz4_decompress_ctx *ctx, victim = availables[--top]; get_page(victim); } else { - victim = erofs_allocpage(pagepool, - GFP_KERNEL | __GFP_NOFAIL); + victim = erofs_allocpage(pagepool, rq->gfp); + if (!victim) + return -ENOMEM; set_page_private(victim, Z_EROFS_SHORTLIVED_PAGE); } rq->out[i] = victim; diff --git a/fs/erofs/decompressor_deflate.c b/fs/erofs/decompressor_deflate.c index 4a64a9c91dd3..b98872058abe 100644 --- a/fs/erofs/decompressor_deflate.c +++ b/fs/erofs/decompressor_deflate.c @@ -95,7 +95,7 @@ int z_erofs_load_deflate_config(struct super_block *sb, } int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, - struct page **pagepool) + struct page **pgpl) { const unsigned int nrpages_out = PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT; @@ -158,8 +158,12 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, strm->z.avail_out = min_t(u32, outsz, PAGE_SIZE - pofs); outsz -= strm->z.avail_out; if (!rq->out[no]) { - rq->out[no] = erofs_allocpage(pagepool, - GFP_KERNEL | __GFP_NOFAIL); + rq->out[no] = erofs_allocpage(pgpl, rq->gfp); + if (!rq->out[no]) { + kout = NULL; + err = -ENOMEM; + break; + } set_page_private(rq->out[no], Z_EROFS_SHORTLIVED_PAGE); } @@ -211,8 +215,11 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, DBG_BUGON(erofs_page_is_managed(EROFS_SB(sb), rq->in[j])); - tmppage = erofs_allocpage(pagepool, - GFP_KERNEL | __GFP_NOFAIL); + tmppage = erofs_allocpage(pgpl, rq->gfp); + if (!tmppage) { + err = -ENOMEM; + goto failed; + } set_page_private(tmppage, Z_EROFS_SHORTLIVED_PAGE); copy_highpage(tmppage, rq->in[j]); rq->in[j] = tmppage; @@ -230,7 +237,7 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, break; } } - +failed: if (zlib_inflateEnd(&strm->z) != Z_OK && !err) err = -EIO; if (kout) diff --git a/fs/erofs/decompressor_lzma.c b/fs/erofs/decompressor_lzma.c index 2dd14f99c1dc..6ca357d83cfa 100644 --- a/fs/erofs/decompressor_lzma.c +++ b/fs/erofs/decompressor_lzma.c @@ -148,7 +148,7 @@ int z_erofs_load_lzma_config(struct super_block *sb, } int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, - struct page **pagepool) + struct page **pgpl) { const unsigned int nrpages_out = PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT; @@ -215,8 +215,11 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, PAGE_SIZE - pageofs); outlen -= strm->buf.out_size; if (!rq->out[no] && rq->fillgaps) { /* deduped */ - rq->out[no] = erofs_allocpage(pagepool, - GFP_KERNEL | __GFP_NOFAIL); + rq->out[no] = erofs_allocpage(pgpl, rq->gfp); + if (!rq->out[no]) { + err = -ENOMEM; + break; + } set_page_private(rq->out[no], Z_EROFS_SHORTLIVED_PAGE); } @@ -258,8 +261,11 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, DBG_BUGON(erofs_page_is_managed(EROFS_SB(rq->sb), rq->in[j])); - tmppage = erofs_allocpage(pagepool, - GFP_KERNEL | __GFP_NOFAIL); + tmppage = erofs_allocpage(pgpl, rq->gfp); + if (!tmppage) { + err = -ENOMEM; + goto failed; + } set_page_private(tmppage, Z_EROFS_SHORTLIVED_PAGE); copy_highpage(tmppage, rq->in[j]); rq->in[j] = tmppage; @@ -277,6 +283,7 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, break; } } +failed: if (no < nrpages_out && strm->buf.out) kunmap(rq->out[no]); if (ni < nrpages_in) diff --git a/fs/erofs/zdata.c b/fs/erofs/zdata.c index c1c77166b30f..1d0fdc145fd6 100644 --- a/fs/erofs/zdata.c +++ b/fs/erofs/zdata.c @@ -82,6 +82,9 @@ struct z_erofs_pcluster { /* L: indicate several pageofs_outs or not */ bool multibases; + /* L: whether extra buffer allocations are best-effort */ + bool besteffort; + /* A: compressed bvecs (can be cached or inplaced pages) */ struct z_erofs_bvec compressed_bvecs[]; }; @@ -960,7 +963,7 @@ static int z_erofs_read_fragment(struct super_block *sb, struct page *page, } static int z_erofs_do_read_page(struct z_erofs_decompress_frontend *fe, - struct page *page) + struct page *page, bool ra) { struct inode *const inode = fe->inode; struct erofs_map_blocks *const map = &fe->map; @@ -1010,6 +1013,7 @@ static int z_erofs_do_read_page(struct z_erofs_decompress_frontend *fe, err = z_erofs_pcluster_begin(fe); if (err) goto out; + fe->pcl->besteffort |= !ra; } /* @@ -1276,7 +1280,11 @@ static int z_erofs_decompress_pcluster(struct z_erofs_decompress_backend *be, .inplace_io = overlapped, .partial_decoding = pcl->partial, .fillgaps = pcl->multibases, + .gfp = pcl->besteffort ? + GFP_KERNEL | __GFP_NOFAIL : + GFP_NOWAIT | __GFP_NORETRY }, be->pagepool); + pcl->besteffort = false; /* must handle all compressed pages before actual file pages */ if (z_erofs_is_inline_pcluster(pcl)) { @@ -1787,7 +1795,7 @@ static void z_erofs_pcluster_readmore(struct z_erofs_decompress_frontend *f, if (PageUptodate(page)) unlock_page(page); else - (void)z_erofs_do_read_page(f, page); + (void)z_erofs_do_read_page(f, page, !!rac); put_page(page); } @@ -1808,7 +1816,7 @@ static int z_erofs_read_folio(struct file *file, struct folio *folio) f.headoffset = (erofs_off_t)folio->index << PAGE_SHIFT; z_erofs_pcluster_readmore(&f, NULL, true); - err = z_erofs_do_read_page(&f, &folio->page); + err = z_erofs_do_read_page(&f, &folio->page, false); z_erofs_pcluster_readmore(&f, NULL, false); z_erofs_pcluster_end(&f); @@ -1849,7 +1857,7 @@ static void z_erofs_readahead(struct readahead_control *rac) folio = head; head = folio_get_private(folio); - err = z_erofs_do_read_page(&f, &folio->page); + err = z_erofs_do_read_page(&f, &folio->page, true); if (err && err != -EINTR) erofs_err(inode->i_sb, "readahead error at folio %lu @ nid %llu", folio->index, EROFS_I(inode)->nid); -- 2.39.3 ^ permalink raw reply related [flat|nested] 20+ messages in thread
* Re: [PATCH v3] erofs: relaxed temporary buffers allocation on readahead 2024-01-26 5:36 ` Gao Xiang @ 2024-01-26 10:46 ` Yue Hu -1 siblings, 0 replies; 20+ messages in thread From: Yue Hu @ 2024-01-26 10:46 UTC (permalink / raw) To: Gao Xiang; +Cc: Chunhai Guo, Yue Hu, linux-erofs, LKML On Fri, 26 Jan 2024 13:36:16 +0800 Gao Xiang <hsiangkao@linux.alibaba.com> wrote: > From: Chunhai Guo <guochunhai@vivo.com> > > Even with inplace decompression, sometimes very few temporary buffers > are still needed for a single decompression shot (e.g. 16 pages for 64k > sliding window or 4 pages for 16k sliding window). In low-memory > scenarios, it would be better to try to allocate with GFP_NOWAIT on > readahead first. That can help reduce the time spent on page allocation > under durative memory pressure. > > Here are detailed performance numbers under multi-app launch benchmark > workload [1] on ARM64 Android devices (8-core CPU and 8GB of memory) > running a 5.15 LTS kernel with EROFS of 4k pclusters: > > +----------------+---------+---------+---------+ > | LZ4 | vanilla | patched | diff | > |----------------+---------+---------+---------| > | Average (ms) | 3364 | 2684 | -20.21% | [64k sliding window] > |----------------+---------+---------+---------| > | Average (ms) | 2079 | 1610 | -22.56% | [16k sliding window] > +----------------+---------+---------+---------+ > > The total size of system images for 4k pcluster is almost unchanged: > (64k sliding window) 9,117,044 KB > (16k sliding window) 9,113,096 KB > > Therefore, in addition to switch the sliding window from 64k to 16k, > after applying this patch, it can eventually save 52.14% (3364 -> 1610) > on average with no memory reservation. That is particularly useful for > embedded devices with limited resources. > > [1] https://lore.kernel.org/r/20240109074143.4138783-1-guochunhai@vivo.com > > Suggested-by: Gao Xiang <xiang@kernel.org> > Signed-off-by: Chunhai Guo <guochunhai@vivo.com> > Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com> > --- > v2: https://lore.kernel.org/r/20240120145551.1941483-1-guochunhai@vivo.com > change since v2: > - update commit message according to test results. > > I plan to apply this version. > > fs/erofs/compress.h | 5 ++--- > fs/erofs/decompressor.c | 5 +++-- > fs/erofs/decompressor_deflate.c | 19 +++++++++++++------ > fs/erofs/decompressor_lzma.c | 17 ++++++++++++----- > fs/erofs/zdata.c | 16 ++++++++++++---- > 5 files changed, 42 insertions(+), 20 deletions(-) > > diff --git a/fs/erofs/compress.h b/fs/erofs/compress.h > index 279933e007d2..7cc5841577b2 100644 > --- a/fs/erofs/compress.h > +++ b/fs/erofs/compress.h > @@ -11,13 +11,12 @@ > struct z_erofs_decompress_req { > struct super_block *sb; > struct page **in, **out; > - > unsigned short pageofs_in, pageofs_out; > unsigned int inputsize, outputsize; > > - /* indicate the algorithm will be used for decompression */ > - unsigned int alg; > + unsigned int alg; /* the algorithm for decompression */ > bool inplace_io, partial_decoding, fillgaps; > + gfp_t gfp; /* allocation flags for extra temporary buffers */ > }; > > struct z_erofs_decompressor { > diff --git a/fs/erofs/decompressor.c b/fs/erofs/decompressor.c > index 072ef6a66823..d4cee95af14c 100644 > --- a/fs/erofs/decompressor.c > +++ b/fs/erofs/decompressor.c > @@ -111,8 +111,9 @@ static int z_erofs_lz4_prepare_dstpages(struct z_erofs_lz4_decompress_ctx *ctx, > victim = availables[--top]; > get_page(victim); > } else { > - victim = erofs_allocpage(pagepool, > - GFP_KERNEL | __GFP_NOFAIL); > + victim = erofs_allocpage(pagepool, rq->gfp); > + if (!victim) > + return -ENOMEM; > set_page_private(victim, Z_EROFS_SHORTLIVED_PAGE); > } > rq->out[i] = victim; > diff --git a/fs/erofs/decompressor_deflate.c b/fs/erofs/decompressor_deflate.c > index 4a64a9c91dd3..b98872058abe 100644 > --- a/fs/erofs/decompressor_deflate.c > +++ b/fs/erofs/decompressor_deflate.c > @@ -95,7 +95,7 @@ int z_erofs_load_deflate_config(struct super_block *sb, > } > > int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, > - struct page **pagepool) > + struct page **pgpl) > { > const unsigned int nrpages_out = > PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT; > @@ -158,8 +158,12 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, > strm->z.avail_out = min_t(u32, outsz, PAGE_SIZE - pofs); > outsz -= strm->z.avail_out; > if (!rq->out[no]) { > - rq->out[no] = erofs_allocpage(pagepool, > - GFP_KERNEL | __GFP_NOFAIL); > + rq->out[no] = erofs_allocpage(pgpl, rq->gfp); > + if (!rq->out[no]) { > + kout = NULL; > + err = -ENOMEM; > + break; > + } > set_page_private(rq->out[no], > Z_EROFS_SHORTLIVED_PAGE); > } > @@ -211,8 +215,11 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, > > DBG_BUGON(erofs_page_is_managed(EROFS_SB(sb), > rq->in[j])); > - tmppage = erofs_allocpage(pagepool, > - GFP_KERNEL | __GFP_NOFAIL); > + tmppage = erofs_allocpage(pgpl, rq->gfp); > + if (!tmppage) { > + err = -ENOMEM; > + goto failed; > + } > set_page_private(tmppage, Z_EROFS_SHORTLIVED_PAGE); > copy_highpage(tmppage, rq->in[j]); > rq->in[j] = tmppage; > @@ -230,7 +237,7 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, > break; > } > } > - > +failed: > if (zlib_inflateEnd(&strm->z) != Z_OK && !err) > err = -EIO; > if (kout) > diff --git a/fs/erofs/decompressor_lzma.c b/fs/erofs/decompressor_lzma.c > index 2dd14f99c1dc..6ca357d83cfa 100644 > --- a/fs/erofs/decompressor_lzma.c > +++ b/fs/erofs/decompressor_lzma.c > @@ -148,7 +148,7 @@ int z_erofs_load_lzma_config(struct super_block *sb, > } > > int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, > - struct page **pagepool) > + struct page **pgpl) > { > const unsigned int nrpages_out = > PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT; > @@ -215,8 +215,11 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, > PAGE_SIZE - pageofs); > outlen -= strm->buf.out_size; > if (!rq->out[no] && rq->fillgaps) { /* deduped */ > - rq->out[no] = erofs_allocpage(pagepool, > - GFP_KERNEL | __GFP_NOFAIL); > + rq->out[no] = erofs_allocpage(pgpl, rq->gfp); > + if (!rq->out[no]) { > + err = -ENOMEM; > + break; > + } > set_page_private(rq->out[no], > Z_EROFS_SHORTLIVED_PAGE); > } > @@ -258,8 +261,11 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, > > DBG_BUGON(erofs_page_is_managed(EROFS_SB(rq->sb), > rq->in[j])); > - tmppage = erofs_allocpage(pagepool, > - GFP_KERNEL | __GFP_NOFAIL); > + tmppage = erofs_allocpage(pgpl, rq->gfp); > + if (!tmppage) { > + err = -ENOMEM; > + goto failed; > + } > set_page_private(tmppage, Z_EROFS_SHORTLIVED_PAGE); > copy_highpage(tmppage, rq->in[j]); > rq->in[j] = tmppage; > @@ -277,6 +283,7 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, > break; > } > } > +failed: > if (no < nrpages_out && strm->buf.out) > kunmap(rq->out[no]); > if (ni < nrpages_in) > diff --git a/fs/erofs/zdata.c b/fs/erofs/zdata.c > index c1c77166b30f..1d0fdc145fd6 100644 > --- a/fs/erofs/zdata.c > +++ b/fs/erofs/zdata.c > @@ -82,6 +82,9 @@ struct z_erofs_pcluster { > /* L: indicate several pageofs_outs or not */ > bool multibases; > > + /* L: whether extra buffer allocations are best-effort */ > + bool besteffort; > + > /* A: compressed bvecs (can be cached or inplaced pages) */ > struct z_erofs_bvec compressed_bvecs[]; > }; > @@ -960,7 +963,7 @@ static int z_erofs_read_fragment(struct super_block *sb, struct page *page, > } > > static int z_erofs_do_read_page(struct z_erofs_decompress_frontend *fe, > - struct page *page) > + struct page *page, bool ra) > { > struct inode *const inode = fe->inode; > struct erofs_map_blocks *const map = &fe->map; > @@ -1010,6 +1013,7 @@ static int z_erofs_do_read_page(struct z_erofs_decompress_frontend *fe, > err = z_erofs_pcluster_begin(fe); > if (err) > goto out; > + fe->pcl->besteffort |= !ra; > } > > /* > @@ -1276,7 +1280,11 @@ static int z_erofs_decompress_pcluster(struct z_erofs_decompress_backend *be, > .inplace_io = overlapped, > .partial_decoding = pcl->partial, > .fillgaps = pcl->multibases, > + .gfp = pcl->besteffort ? > + GFP_KERNEL | __GFP_NOFAIL : > + GFP_NOWAIT | __GFP_NORETRY > }, be->pagepool); > + pcl->besteffort = false; reposition it following `pcl->multibases = false`? > > /* must handle all compressed pages before actual file pages */ > if (z_erofs_is_inline_pcluster(pcl)) { > @@ -1787,7 +1795,7 @@ static void z_erofs_pcluster_readmore(struct z_erofs_decompress_frontend *f, > if (PageUptodate(page)) > unlock_page(page); > else > - (void)z_erofs_do_read_page(f, page); > + (void)z_erofs_do_read_page(f, page, !!rac); > put_page(page); > } > > @@ -1808,7 +1816,7 @@ static int z_erofs_read_folio(struct file *file, struct folio *folio) > f.headoffset = (erofs_off_t)folio->index << PAGE_SHIFT; > > z_erofs_pcluster_readmore(&f, NULL, true); > - err = z_erofs_do_read_page(&f, &folio->page); > + err = z_erofs_do_read_page(&f, &folio->page, false); > z_erofs_pcluster_readmore(&f, NULL, false); > z_erofs_pcluster_end(&f); > > @@ -1849,7 +1857,7 @@ static void z_erofs_readahead(struct readahead_control *rac) > folio = head; > head = folio_get_private(folio); > > - err = z_erofs_do_read_page(&f, &folio->page); > + err = z_erofs_do_read_page(&f, &folio->page, true); > if (err && err != -EINTR) > erofs_err(inode->i_sb, "readahead error at folio %lu @ nid %llu", > folio->index, EROFS_I(inode)->nid); ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v3] erofs: relaxed temporary buffers allocation on readahead @ 2024-01-26 10:46 ` Yue Hu 0 siblings, 0 replies; 20+ messages in thread From: Yue Hu @ 2024-01-26 10:46 UTC (permalink / raw) To: Gao Xiang; +Cc: linux-erofs, LKML, Yue Hu, Chunhai Guo On Fri, 26 Jan 2024 13:36:16 +0800 Gao Xiang <hsiangkao@linux.alibaba.com> wrote: > From: Chunhai Guo <guochunhai@vivo.com> > > Even with inplace decompression, sometimes very few temporary buffers > are still needed for a single decompression shot (e.g. 16 pages for 64k > sliding window or 4 pages for 16k sliding window). In low-memory > scenarios, it would be better to try to allocate with GFP_NOWAIT on > readahead first. That can help reduce the time spent on page allocation > under durative memory pressure. > > Here are detailed performance numbers under multi-app launch benchmark > workload [1] on ARM64 Android devices (8-core CPU and 8GB of memory) > running a 5.15 LTS kernel with EROFS of 4k pclusters: > > +----------------+---------+---------+---------+ > | LZ4 | vanilla | patched | diff | > |----------------+---------+---------+---------| > | Average (ms) | 3364 | 2684 | -20.21% | [64k sliding window] > |----------------+---------+---------+---------| > | Average (ms) | 2079 | 1610 | -22.56% | [16k sliding window] > +----------------+---------+---------+---------+ > > The total size of system images for 4k pcluster is almost unchanged: > (64k sliding window) 9,117,044 KB > (16k sliding window) 9,113,096 KB > > Therefore, in addition to switch the sliding window from 64k to 16k, > after applying this patch, it can eventually save 52.14% (3364 -> 1610) > on average with no memory reservation. That is particularly useful for > embedded devices with limited resources. > > [1] https://lore.kernel.org/r/20240109074143.4138783-1-guochunhai@vivo.com > > Suggested-by: Gao Xiang <xiang@kernel.org> > Signed-off-by: Chunhai Guo <guochunhai@vivo.com> > Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com> > --- > v2: https://lore.kernel.org/r/20240120145551.1941483-1-guochunhai@vivo.com > change since v2: > - update commit message according to test results. > > I plan to apply this version. > > fs/erofs/compress.h | 5 ++--- > fs/erofs/decompressor.c | 5 +++-- > fs/erofs/decompressor_deflate.c | 19 +++++++++++++------ > fs/erofs/decompressor_lzma.c | 17 ++++++++++++----- > fs/erofs/zdata.c | 16 ++++++++++++---- > 5 files changed, 42 insertions(+), 20 deletions(-) > > diff --git a/fs/erofs/compress.h b/fs/erofs/compress.h > index 279933e007d2..7cc5841577b2 100644 > --- a/fs/erofs/compress.h > +++ b/fs/erofs/compress.h > @@ -11,13 +11,12 @@ > struct z_erofs_decompress_req { > struct super_block *sb; > struct page **in, **out; > - > unsigned short pageofs_in, pageofs_out; > unsigned int inputsize, outputsize; > > - /* indicate the algorithm will be used for decompression */ > - unsigned int alg; > + unsigned int alg; /* the algorithm for decompression */ > bool inplace_io, partial_decoding, fillgaps; > + gfp_t gfp; /* allocation flags for extra temporary buffers */ > }; > > struct z_erofs_decompressor { > diff --git a/fs/erofs/decompressor.c b/fs/erofs/decompressor.c > index 072ef6a66823..d4cee95af14c 100644 > --- a/fs/erofs/decompressor.c > +++ b/fs/erofs/decompressor.c > @@ -111,8 +111,9 @@ static int z_erofs_lz4_prepare_dstpages(struct z_erofs_lz4_decompress_ctx *ctx, > victim = availables[--top]; > get_page(victim); > } else { > - victim = erofs_allocpage(pagepool, > - GFP_KERNEL | __GFP_NOFAIL); > + victim = erofs_allocpage(pagepool, rq->gfp); > + if (!victim) > + return -ENOMEM; > set_page_private(victim, Z_EROFS_SHORTLIVED_PAGE); > } > rq->out[i] = victim; > diff --git a/fs/erofs/decompressor_deflate.c b/fs/erofs/decompressor_deflate.c > index 4a64a9c91dd3..b98872058abe 100644 > --- a/fs/erofs/decompressor_deflate.c > +++ b/fs/erofs/decompressor_deflate.c > @@ -95,7 +95,7 @@ int z_erofs_load_deflate_config(struct super_block *sb, > } > > int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, > - struct page **pagepool) > + struct page **pgpl) > { > const unsigned int nrpages_out = > PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT; > @@ -158,8 +158,12 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, > strm->z.avail_out = min_t(u32, outsz, PAGE_SIZE - pofs); > outsz -= strm->z.avail_out; > if (!rq->out[no]) { > - rq->out[no] = erofs_allocpage(pagepool, > - GFP_KERNEL | __GFP_NOFAIL); > + rq->out[no] = erofs_allocpage(pgpl, rq->gfp); > + if (!rq->out[no]) { > + kout = NULL; > + err = -ENOMEM; > + break; > + } > set_page_private(rq->out[no], > Z_EROFS_SHORTLIVED_PAGE); > } > @@ -211,8 +215,11 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, > > DBG_BUGON(erofs_page_is_managed(EROFS_SB(sb), > rq->in[j])); > - tmppage = erofs_allocpage(pagepool, > - GFP_KERNEL | __GFP_NOFAIL); > + tmppage = erofs_allocpage(pgpl, rq->gfp); > + if (!tmppage) { > + err = -ENOMEM; > + goto failed; > + } > set_page_private(tmppage, Z_EROFS_SHORTLIVED_PAGE); > copy_highpage(tmppage, rq->in[j]); > rq->in[j] = tmppage; > @@ -230,7 +237,7 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, > break; > } > } > - > +failed: > if (zlib_inflateEnd(&strm->z) != Z_OK && !err) > err = -EIO; > if (kout) > diff --git a/fs/erofs/decompressor_lzma.c b/fs/erofs/decompressor_lzma.c > index 2dd14f99c1dc..6ca357d83cfa 100644 > --- a/fs/erofs/decompressor_lzma.c > +++ b/fs/erofs/decompressor_lzma.c > @@ -148,7 +148,7 @@ int z_erofs_load_lzma_config(struct super_block *sb, > } > > int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, > - struct page **pagepool) > + struct page **pgpl) > { > const unsigned int nrpages_out = > PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT; > @@ -215,8 +215,11 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, > PAGE_SIZE - pageofs); > outlen -= strm->buf.out_size; > if (!rq->out[no] && rq->fillgaps) { /* deduped */ > - rq->out[no] = erofs_allocpage(pagepool, > - GFP_KERNEL | __GFP_NOFAIL); > + rq->out[no] = erofs_allocpage(pgpl, rq->gfp); > + if (!rq->out[no]) { > + err = -ENOMEM; > + break; > + } > set_page_private(rq->out[no], > Z_EROFS_SHORTLIVED_PAGE); > } > @@ -258,8 +261,11 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, > > DBG_BUGON(erofs_page_is_managed(EROFS_SB(rq->sb), > rq->in[j])); > - tmppage = erofs_allocpage(pagepool, > - GFP_KERNEL | __GFP_NOFAIL); > + tmppage = erofs_allocpage(pgpl, rq->gfp); > + if (!tmppage) { > + err = -ENOMEM; > + goto failed; > + } > set_page_private(tmppage, Z_EROFS_SHORTLIVED_PAGE); > copy_highpage(tmppage, rq->in[j]); > rq->in[j] = tmppage; > @@ -277,6 +283,7 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, > break; > } > } > +failed: > if (no < nrpages_out && strm->buf.out) > kunmap(rq->out[no]); > if (ni < nrpages_in) > diff --git a/fs/erofs/zdata.c b/fs/erofs/zdata.c > index c1c77166b30f..1d0fdc145fd6 100644 > --- a/fs/erofs/zdata.c > +++ b/fs/erofs/zdata.c > @@ -82,6 +82,9 @@ struct z_erofs_pcluster { > /* L: indicate several pageofs_outs or not */ > bool multibases; > > + /* L: whether extra buffer allocations are best-effort */ > + bool besteffort; > + > /* A: compressed bvecs (can be cached or inplaced pages) */ > struct z_erofs_bvec compressed_bvecs[]; > }; > @@ -960,7 +963,7 @@ static int z_erofs_read_fragment(struct super_block *sb, struct page *page, > } > > static int z_erofs_do_read_page(struct z_erofs_decompress_frontend *fe, > - struct page *page) > + struct page *page, bool ra) > { > struct inode *const inode = fe->inode; > struct erofs_map_blocks *const map = &fe->map; > @@ -1010,6 +1013,7 @@ static int z_erofs_do_read_page(struct z_erofs_decompress_frontend *fe, > err = z_erofs_pcluster_begin(fe); > if (err) > goto out; > + fe->pcl->besteffort |= !ra; > } > > /* > @@ -1276,7 +1280,11 @@ static int z_erofs_decompress_pcluster(struct z_erofs_decompress_backend *be, > .inplace_io = overlapped, > .partial_decoding = pcl->partial, > .fillgaps = pcl->multibases, > + .gfp = pcl->besteffort ? > + GFP_KERNEL | __GFP_NOFAIL : > + GFP_NOWAIT | __GFP_NORETRY > }, be->pagepool); > + pcl->besteffort = false; reposition it following `pcl->multibases = false`? > > /* must handle all compressed pages before actual file pages */ > if (z_erofs_is_inline_pcluster(pcl)) { > @@ -1787,7 +1795,7 @@ static void z_erofs_pcluster_readmore(struct z_erofs_decompress_frontend *f, > if (PageUptodate(page)) > unlock_page(page); > else > - (void)z_erofs_do_read_page(f, page); > + (void)z_erofs_do_read_page(f, page, !!rac); > put_page(page); > } > > @@ -1808,7 +1816,7 @@ static int z_erofs_read_folio(struct file *file, struct folio *folio) > f.headoffset = (erofs_off_t)folio->index << PAGE_SHIFT; > > z_erofs_pcluster_readmore(&f, NULL, true); > - err = z_erofs_do_read_page(&f, &folio->page); > + err = z_erofs_do_read_page(&f, &folio->page, false); > z_erofs_pcluster_readmore(&f, NULL, false); > z_erofs_pcluster_end(&f); > > @@ -1849,7 +1857,7 @@ static void z_erofs_readahead(struct readahead_control *rac) > folio = head; > head = folio_get_private(folio); > > - err = z_erofs_do_read_page(&f, &folio->page); > + err = z_erofs_do_read_page(&f, &folio->page, true); > if (err && err != -EINTR) > erofs_err(inode->i_sb, "readahead error at folio %lu @ nid %llu", > folio->index, EROFS_I(inode)->nid); ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v3] erofs: relaxed temporary buffers allocation on readahead 2024-01-26 10:46 ` Yue Hu @ 2024-01-26 13:30 ` Gao Xiang -1 siblings, 0 replies; 20+ messages in thread From: Gao Xiang @ 2024-01-26 13:30 UTC (permalink / raw) To: Yue Hu; +Cc: Chunhai Guo, Yue Hu, linux-erofs, LKML Hi Yue, On 2024/1/26 18:46, Yue Hu wrote: > On Fri, 26 Jan 2024 13:36:16 +0800 > Gao Xiang <hsiangkao@linux.alibaba.com> wrote: > ... >> /* >> @@ -1276,7 +1280,11 @@ static int z_erofs_decompress_pcluster(struct z_erofs_decompress_backend *be, >> .inplace_io = overlapped, >> .partial_decoding = pcl->partial, >> .fillgaps = pcl->multibases, >> + .gfp = pcl->besteffort ? >> + GFP_KERNEL | __GFP_NOFAIL : >> + GFP_NOWAIT | __GFP_NORETRY >> }, be->pagepool); >> + pcl->besteffort = false; > > reposition it following `pcl->multibases = false`? Good idea! Let me update this. Thanks, Gao Xiang ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v3] erofs: relaxed temporary buffers allocation on readahead @ 2024-01-26 13:30 ` Gao Xiang 0 siblings, 0 replies; 20+ messages in thread From: Gao Xiang @ 2024-01-26 13:30 UTC (permalink / raw) To: Yue Hu; +Cc: linux-erofs, LKML, Yue Hu, Chunhai Guo Hi Yue, On 2024/1/26 18:46, Yue Hu wrote: > On Fri, 26 Jan 2024 13:36:16 +0800 > Gao Xiang <hsiangkao@linux.alibaba.com> wrote: > ... >> /* >> @@ -1276,7 +1280,11 @@ static int z_erofs_decompress_pcluster(struct z_erofs_decompress_backend *be, >> .inplace_io = overlapped, >> .partial_decoding = pcl->partial, >> .fillgaps = pcl->multibases, >> + .gfp = pcl->besteffort ? >> + GFP_KERNEL | __GFP_NOFAIL : >> + GFP_NOWAIT | __GFP_NORETRY >> }, be->pagepool); >> + pcl->besteffort = false; > > reposition it following `pcl->multibases = false`? Good idea! Let me update this. Thanks, Gao Xiang ^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v4] erofs: relaxed temporary buffers allocation on readahead 2024-01-26 10:46 ` Yue Hu @ 2024-01-26 14:01 ` Gao Xiang -1 siblings, 0 replies; 20+ messages in thread From: Gao Xiang @ 2024-01-26 14:01 UTC (permalink / raw) To: linux-erofs; +Cc: Gao Xiang, Chunhai Guo, LKML From: Chunhai Guo <guochunhai@vivo.com> Even with inplace decompression, sometimes very few temporary buffers may be still needed for a single decompression shot (e.g. 16 pages for 64k sliding window or 4 pages for 16k sliding window). In low-memory scenarios, it would be better to try to allocate with GFP_NOWAIT on readahead first. That can help reduce the time spent on page allocation under durative memory pressure. Here are detailed performance numbers under multi-app launch benchmark workload [1] on ARM64 Android devices (8-core CPU and 8GB of memory) running a 5.15 LTS kernel with EROFS of 4k pclusters: +----------------------------------------------+ | LZ4 | vanilla | patched | diff | |----------------+---------+---------+---------| | Average (ms) | 3364 | 2684 | -20.21% | [64k sliding window] |----------------+---------+---------+---------| | Average (ms) | 2079 | 1610 | -22.56% | [16k sliding window] +----------------------------------------------+ The total size of system images for 4k pclusters is almost unchanged: (64k sliding window) 9,117,044 KB (16k sliding window) 9,113,096 KB Therefore, in addition to switch the sliding window from 64k to 16k, after applying this patch, it can eventually save 52.14% (3364 -> 1610) on average with no memory reservation. That is particularly useful for embedded devices with limited resources. [1] https://lore.kernel.org/r/20240109074143.4138783-1-guochunhai@vivo.com Suggested-by: Gao Xiang <xiang@kernel.org> Signed-off-by: Chunhai Guo <guochunhai@vivo.com> Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com> --- change since v3: - reposition `pcl->besteffort = false;` suggested by Yue; fs/erofs/compress.h | 5 ++--- fs/erofs/decompressor.c | 5 +++-- fs/erofs/decompressor_deflate.c | 19 +++++++++++++------ fs/erofs/decompressor_lzma.c | 17 ++++++++++++----- fs/erofs/zdata.c | 16 ++++++++++++---- 5 files changed, 42 insertions(+), 20 deletions(-) diff --git a/fs/erofs/compress.h b/fs/erofs/compress.h index 279933e007d2..7cc5841577b2 100644 --- a/fs/erofs/compress.h +++ b/fs/erofs/compress.h @@ -11,13 +11,12 @@ struct z_erofs_decompress_req { struct super_block *sb; struct page **in, **out; - unsigned short pageofs_in, pageofs_out; unsigned int inputsize, outputsize; - /* indicate the algorithm will be used for decompression */ - unsigned int alg; + unsigned int alg; /* the algorithm for decompression */ bool inplace_io, partial_decoding, fillgaps; + gfp_t gfp; /* allocation flags for extra temporary buffers */ }; struct z_erofs_decompressor { diff --git a/fs/erofs/decompressor.c b/fs/erofs/decompressor.c index 072ef6a66823..d4cee95af14c 100644 --- a/fs/erofs/decompressor.c +++ b/fs/erofs/decompressor.c @@ -111,8 +111,9 @@ static int z_erofs_lz4_prepare_dstpages(struct z_erofs_lz4_decompress_ctx *ctx, victim = availables[--top]; get_page(victim); } else { - victim = erofs_allocpage(pagepool, - GFP_KERNEL | __GFP_NOFAIL); + victim = erofs_allocpage(pagepool, rq->gfp); + if (!victim) + return -ENOMEM; set_page_private(victim, Z_EROFS_SHORTLIVED_PAGE); } rq->out[i] = victim; diff --git a/fs/erofs/decompressor_deflate.c b/fs/erofs/decompressor_deflate.c index 4a64a9c91dd3..b98872058abe 100644 --- a/fs/erofs/decompressor_deflate.c +++ b/fs/erofs/decompressor_deflate.c @@ -95,7 +95,7 @@ int z_erofs_load_deflate_config(struct super_block *sb, } int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, - struct page **pagepool) + struct page **pgpl) { const unsigned int nrpages_out = PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT; @@ -158,8 +158,12 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, strm->z.avail_out = min_t(u32, outsz, PAGE_SIZE - pofs); outsz -= strm->z.avail_out; if (!rq->out[no]) { - rq->out[no] = erofs_allocpage(pagepool, - GFP_KERNEL | __GFP_NOFAIL); + rq->out[no] = erofs_allocpage(pgpl, rq->gfp); + if (!rq->out[no]) { + kout = NULL; + err = -ENOMEM; + break; + } set_page_private(rq->out[no], Z_EROFS_SHORTLIVED_PAGE); } @@ -211,8 +215,11 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, DBG_BUGON(erofs_page_is_managed(EROFS_SB(sb), rq->in[j])); - tmppage = erofs_allocpage(pagepool, - GFP_KERNEL | __GFP_NOFAIL); + tmppage = erofs_allocpage(pgpl, rq->gfp); + if (!tmppage) { + err = -ENOMEM; + goto failed; + } set_page_private(tmppage, Z_EROFS_SHORTLIVED_PAGE); copy_highpage(tmppage, rq->in[j]); rq->in[j] = tmppage; @@ -230,7 +237,7 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, break; } } - +failed: if (zlib_inflateEnd(&strm->z) != Z_OK && !err) err = -EIO; if (kout) diff --git a/fs/erofs/decompressor_lzma.c b/fs/erofs/decompressor_lzma.c index 2dd14f99c1dc..6ca357d83cfa 100644 --- a/fs/erofs/decompressor_lzma.c +++ b/fs/erofs/decompressor_lzma.c @@ -148,7 +148,7 @@ int z_erofs_load_lzma_config(struct super_block *sb, } int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, - struct page **pagepool) + struct page **pgpl) { const unsigned int nrpages_out = PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT; @@ -215,8 +215,11 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, PAGE_SIZE - pageofs); outlen -= strm->buf.out_size; if (!rq->out[no] && rq->fillgaps) { /* deduped */ - rq->out[no] = erofs_allocpage(pagepool, - GFP_KERNEL | __GFP_NOFAIL); + rq->out[no] = erofs_allocpage(pgpl, rq->gfp); + if (!rq->out[no]) { + err = -ENOMEM; + break; + } set_page_private(rq->out[no], Z_EROFS_SHORTLIVED_PAGE); } @@ -258,8 +261,11 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, DBG_BUGON(erofs_page_is_managed(EROFS_SB(rq->sb), rq->in[j])); - tmppage = erofs_allocpage(pagepool, - GFP_KERNEL | __GFP_NOFAIL); + tmppage = erofs_allocpage(pgpl, rq->gfp); + if (!tmppage) { + err = -ENOMEM; + goto failed; + } set_page_private(tmppage, Z_EROFS_SHORTLIVED_PAGE); copy_highpage(tmppage, rq->in[j]); rq->in[j] = tmppage; @@ -277,6 +283,7 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, break; } } +failed: if (no < nrpages_out && strm->buf.out) kunmap(rq->out[no]); if (ni < nrpages_in) diff --git a/fs/erofs/zdata.c b/fs/erofs/zdata.c index c1c77166b30f..ff0aa72b0db3 100644 --- a/fs/erofs/zdata.c +++ b/fs/erofs/zdata.c @@ -82,6 +82,9 @@ struct z_erofs_pcluster { /* L: indicate several pageofs_outs or not */ bool multibases; + /* L: whether extra buffer allocations are best-effort */ + bool besteffort; + /* A: compressed bvecs (can be cached or inplaced pages) */ struct z_erofs_bvec compressed_bvecs[]; }; @@ -960,7 +963,7 @@ static int z_erofs_read_fragment(struct super_block *sb, struct page *page, } static int z_erofs_do_read_page(struct z_erofs_decompress_frontend *fe, - struct page *page) + struct page *page, bool ra) { struct inode *const inode = fe->inode; struct erofs_map_blocks *const map = &fe->map; @@ -1010,6 +1013,7 @@ static int z_erofs_do_read_page(struct z_erofs_decompress_frontend *fe, err = z_erofs_pcluster_begin(fe); if (err) goto out; + fe->pcl->besteffort |= !ra; } /* @@ -1276,6 +1280,9 @@ static int z_erofs_decompress_pcluster(struct z_erofs_decompress_backend *be, .inplace_io = overlapped, .partial_decoding = pcl->partial, .fillgaps = pcl->multibases, + .gfp = pcl->besteffort ? + GFP_KERNEL | __GFP_NOFAIL : + GFP_NOWAIT | __GFP_NORETRY }, be->pagepool); /* must handle all compressed pages before actual file pages */ @@ -1318,6 +1325,7 @@ static int z_erofs_decompress_pcluster(struct z_erofs_decompress_backend *be, pcl->length = 0; pcl->partial = true; pcl->multibases = false; + pcl->besteffort = false; pcl->bvset.nextpage = NULL; pcl->vcnt = 0; @@ -1787,7 +1795,7 @@ static void z_erofs_pcluster_readmore(struct z_erofs_decompress_frontend *f, if (PageUptodate(page)) unlock_page(page); else - (void)z_erofs_do_read_page(f, page); + (void)z_erofs_do_read_page(f, page, !!rac); put_page(page); } @@ -1808,7 +1816,7 @@ static int z_erofs_read_folio(struct file *file, struct folio *folio) f.headoffset = (erofs_off_t)folio->index << PAGE_SHIFT; z_erofs_pcluster_readmore(&f, NULL, true); - err = z_erofs_do_read_page(&f, &folio->page); + err = z_erofs_do_read_page(&f, &folio->page, false); z_erofs_pcluster_readmore(&f, NULL, false); z_erofs_pcluster_end(&f); @@ -1849,7 +1857,7 @@ static void z_erofs_readahead(struct readahead_control *rac) folio = head; head = folio_get_private(folio); - err = z_erofs_do_read_page(&f, &folio->page); + err = z_erofs_do_read_page(&f, &folio->page, true); if (err && err != -EINTR) erofs_err(inode->i_sb, "readahead error at folio %lu @ nid %llu", folio->index, EROFS_I(inode)->nid); -- 2.39.3 ^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v4] erofs: relaxed temporary buffers allocation on readahead @ 2024-01-26 14:01 ` Gao Xiang 0 siblings, 0 replies; 20+ messages in thread From: Gao Xiang @ 2024-01-26 14:01 UTC (permalink / raw) To: linux-erofs; +Cc: LKML, Yue Hu, Chunhai Guo, Gao Xiang, Gao Xiang From: Chunhai Guo <guochunhai@vivo.com> Even with inplace decompression, sometimes very few temporary buffers may be still needed for a single decompression shot (e.g. 16 pages for 64k sliding window or 4 pages for 16k sliding window). In low-memory scenarios, it would be better to try to allocate with GFP_NOWAIT on readahead first. That can help reduce the time spent on page allocation under durative memory pressure. Here are detailed performance numbers under multi-app launch benchmark workload [1] on ARM64 Android devices (8-core CPU and 8GB of memory) running a 5.15 LTS kernel with EROFS of 4k pclusters: +----------------------------------------------+ | LZ4 | vanilla | patched | diff | |----------------+---------+---------+---------| | Average (ms) | 3364 | 2684 | -20.21% | [64k sliding window] |----------------+---------+---------+---------| | Average (ms) | 2079 | 1610 | -22.56% | [16k sliding window] +----------------------------------------------+ The total size of system images for 4k pclusters is almost unchanged: (64k sliding window) 9,117,044 KB (16k sliding window) 9,113,096 KB Therefore, in addition to switch the sliding window from 64k to 16k, after applying this patch, it can eventually save 52.14% (3364 -> 1610) on average with no memory reservation. That is particularly useful for embedded devices with limited resources. [1] https://lore.kernel.org/r/20240109074143.4138783-1-guochunhai@vivo.com Suggested-by: Gao Xiang <xiang@kernel.org> Signed-off-by: Chunhai Guo <guochunhai@vivo.com> Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com> --- change since v3: - reposition `pcl->besteffort = false;` suggested by Yue; fs/erofs/compress.h | 5 ++--- fs/erofs/decompressor.c | 5 +++-- fs/erofs/decompressor_deflate.c | 19 +++++++++++++------ fs/erofs/decompressor_lzma.c | 17 ++++++++++++----- fs/erofs/zdata.c | 16 ++++++++++++---- 5 files changed, 42 insertions(+), 20 deletions(-) diff --git a/fs/erofs/compress.h b/fs/erofs/compress.h index 279933e007d2..7cc5841577b2 100644 --- a/fs/erofs/compress.h +++ b/fs/erofs/compress.h @@ -11,13 +11,12 @@ struct z_erofs_decompress_req { struct super_block *sb; struct page **in, **out; - unsigned short pageofs_in, pageofs_out; unsigned int inputsize, outputsize; - /* indicate the algorithm will be used for decompression */ - unsigned int alg; + unsigned int alg; /* the algorithm for decompression */ bool inplace_io, partial_decoding, fillgaps; + gfp_t gfp; /* allocation flags for extra temporary buffers */ }; struct z_erofs_decompressor { diff --git a/fs/erofs/decompressor.c b/fs/erofs/decompressor.c index 072ef6a66823..d4cee95af14c 100644 --- a/fs/erofs/decompressor.c +++ b/fs/erofs/decompressor.c @@ -111,8 +111,9 @@ static int z_erofs_lz4_prepare_dstpages(struct z_erofs_lz4_decompress_ctx *ctx, victim = availables[--top]; get_page(victim); } else { - victim = erofs_allocpage(pagepool, - GFP_KERNEL | __GFP_NOFAIL); + victim = erofs_allocpage(pagepool, rq->gfp); + if (!victim) + return -ENOMEM; set_page_private(victim, Z_EROFS_SHORTLIVED_PAGE); } rq->out[i] = victim; diff --git a/fs/erofs/decompressor_deflate.c b/fs/erofs/decompressor_deflate.c index 4a64a9c91dd3..b98872058abe 100644 --- a/fs/erofs/decompressor_deflate.c +++ b/fs/erofs/decompressor_deflate.c @@ -95,7 +95,7 @@ int z_erofs_load_deflate_config(struct super_block *sb, } int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, - struct page **pagepool) + struct page **pgpl) { const unsigned int nrpages_out = PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT; @@ -158,8 +158,12 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, strm->z.avail_out = min_t(u32, outsz, PAGE_SIZE - pofs); outsz -= strm->z.avail_out; if (!rq->out[no]) { - rq->out[no] = erofs_allocpage(pagepool, - GFP_KERNEL | __GFP_NOFAIL); + rq->out[no] = erofs_allocpage(pgpl, rq->gfp); + if (!rq->out[no]) { + kout = NULL; + err = -ENOMEM; + break; + } set_page_private(rq->out[no], Z_EROFS_SHORTLIVED_PAGE); } @@ -211,8 +215,11 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, DBG_BUGON(erofs_page_is_managed(EROFS_SB(sb), rq->in[j])); - tmppage = erofs_allocpage(pagepool, - GFP_KERNEL | __GFP_NOFAIL); + tmppage = erofs_allocpage(pgpl, rq->gfp); + if (!tmppage) { + err = -ENOMEM; + goto failed; + } set_page_private(tmppage, Z_EROFS_SHORTLIVED_PAGE); copy_highpage(tmppage, rq->in[j]); rq->in[j] = tmppage; @@ -230,7 +237,7 @@ int z_erofs_deflate_decompress(struct z_erofs_decompress_req *rq, break; } } - +failed: if (zlib_inflateEnd(&strm->z) != Z_OK && !err) err = -EIO; if (kout) diff --git a/fs/erofs/decompressor_lzma.c b/fs/erofs/decompressor_lzma.c index 2dd14f99c1dc..6ca357d83cfa 100644 --- a/fs/erofs/decompressor_lzma.c +++ b/fs/erofs/decompressor_lzma.c @@ -148,7 +148,7 @@ int z_erofs_load_lzma_config(struct super_block *sb, } int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, - struct page **pagepool) + struct page **pgpl) { const unsigned int nrpages_out = PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT; @@ -215,8 +215,11 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, PAGE_SIZE - pageofs); outlen -= strm->buf.out_size; if (!rq->out[no] && rq->fillgaps) { /* deduped */ - rq->out[no] = erofs_allocpage(pagepool, - GFP_KERNEL | __GFP_NOFAIL); + rq->out[no] = erofs_allocpage(pgpl, rq->gfp); + if (!rq->out[no]) { + err = -ENOMEM; + break; + } set_page_private(rq->out[no], Z_EROFS_SHORTLIVED_PAGE); } @@ -258,8 +261,11 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, DBG_BUGON(erofs_page_is_managed(EROFS_SB(rq->sb), rq->in[j])); - tmppage = erofs_allocpage(pagepool, - GFP_KERNEL | __GFP_NOFAIL); + tmppage = erofs_allocpage(pgpl, rq->gfp); + if (!tmppage) { + err = -ENOMEM; + goto failed; + } set_page_private(tmppage, Z_EROFS_SHORTLIVED_PAGE); copy_highpage(tmppage, rq->in[j]); rq->in[j] = tmppage; @@ -277,6 +283,7 @@ int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq, break; } } +failed: if (no < nrpages_out && strm->buf.out) kunmap(rq->out[no]); if (ni < nrpages_in) diff --git a/fs/erofs/zdata.c b/fs/erofs/zdata.c index c1c77166b30f..ff0aa72b0db3 100644 --- a/fs/erofs/zdata.c +++ b/fs/erofs/zdata.c @@ -82,6 +82,9 @@ struct z_erofs_pcluster { /* L: indicate several pageofs_outs or not */ bool multibases; + /* L: whether extra buffer allocations are best-effort */ + bool besteffort; + /* A: compressed bvecs (can be cached or inplaced pages) */ struct z_erofs_bvec compressed_bvecs[]; }; @@ -960,7 +963,7 @@ static int z_erofs_read_fragment(struct super_block *sb, struct page *page, } static int z_erofs_do_read_page(struct z_erofs_decompress_frontend *fe, - struct page *page) + struct page *page, bool ra) { struct inode *const inode = fe->inode; struct erofs_map_blocks *const map = &fe->map; @@ -1010,6 +1013,7 @@ static int z_erofs_do_read_page(struct z_erofs_decompress_frontend *fe, err = z_erofs_pcluster_begin(fe); if (err) goto out; + fe->pcl->besteffort |= !ra; } /* @@ -1276,6 +1280,9 @@ static int z_erofs_decompress_pcluster(struct z_erofs_decompress_backend *be, .inplace_io = overlapped, .partial_decoding = pcl->partial, .fillgaps = pcl->multibases, + .gfp = pcl->besteffort ? + GFP_KERNEL | __GFP_NOFAIL : + GFP_NOWAIT | __GFP_NORETRY }, be->pagepool); /* must handle all compressed pages before actual file pages */ @@ -1318,6 +1325,7 @@ static int z_erofs_decompress_pcluster(struct z_erofs_decompress_backend *be, pcl->length = 0; pcl->partial = true; pcl->multibases = false; + pcl->besteffort = false; pcl->bvset.nextpage = NULL; pcl->vcnt = 0; @@ -1787,7 +1795,7 @@ static void z_erofs_pcluster_readmore(struct z_erofs_decompress_frontend *f, if (PageUptodate(page)) unlock_page(page); else - (void)z_erofs_do_read_page(f, page); + (void)z_erofs_do_read_page(f, page, !!rac); put_page(page); } @@ -1808,7 +1816,7 @@ static int z_erofs_read_folio(struct file *file, struct folio *folio) f.headoffset = (erofs_off_t)folio->index << PAGE_SHIFT; z_erofs_pcluster_readmore(&f, NULL, true); - err = z_erofs_do_read_page(&f, &folio->page); + err = z_erofs_do_read_page(&f, &folio->page, false); z_erofs_pcluster_readmore(&f, NULL, false); z_erofs_pcluster_end(&f); @@ -1849,7 +1857,7 @@ static void z_erofs_readahead(struct readahead_control *rac) folio = head; head = folio_get_private(folio); - err = z_erofs_do_read_page(&f, &folio->page); + err = z_erofs_do_read_page(&f, &folio->page, true); if (err && err != -EINTR) erofs_err(inode->i_sb, "readahead error at folio %lu @ nid %llu", folio->index, EROFS_I(inode)->nid); -- 2.39.3 ^ permalink raw reply related [flat|nested] 20+ messages in thread
* Re: [PATCH v4] erofs: relaxed temporary buffers allocation on readahead 2024-01-26 14:01 ` Gao Xiang @ 2024-01-27 1:49 ` Yue Hu -1 siblings, 0 replies; 20+ messages in thread From: Yue Hu @ 2024-01-27 1:49 UTC (permalink / raw) To: Gao Xiang; +Cc: Chunhai Guo, huyue2, linux-erofs, LKML On Fri, 26 Jan 2024 22:01:42 +0800 Gao Xiang <hsiangkao@linux.alibaba.com> wrote: > From: Chunhai Guo <guochunhai@vivo.com> > > Even with inplace decompression, sometimes very few temporary buffers > may be still needed for a single decompression shot (e.g. 16 pages for > 64k sliding window or 4 pages for 16k sliding window). In low-memory > scenarios, it would be better to try to allocate with GFP_NOWAIT on > readahead first. That can help reduce the time spent on page allocation > under durative memory pressure. > > Here are detailed performance numbers under multi-app launch benchmark > workload [1] on ARM64 Android devices (8-core CPU and 8GB of memory) > running a 5.15 LTS kernel with EROFS of 4k pclusters: > > +----------------------------------------------+ > | LZ4 | vanilla | patched | diff | > |----------------+---------+---------+---------| > | Average (ms) | 3364 | 2684 | -20.21% | [64k sliding window] > |----------------+---------+---------+---------| > | Average (ms) | 2079 | 1610 | -22.56% | [16k sliding window] > +----------------------------------------------+ > > The total size of system images for 4k pclusters is almost unchanged: > (64k sliding window) 9,117,044 KB > (16k sliding window) 9,113,096 KB > > Therefore, in addition to switch the sliding window from 64k to 16k, > after applying this patch, it can eventually save 52.14% (3364 -> 1610) > on average with no memory reservation. That is particularly useful for > embedded devices with limited resources. > > [1] https://lore.kernel.org/r/20240109074143.4138783-1-guochunhai@vivo.com > > Suggested-by: Gao Xiang <xiang@kernel.org> > Signed-off-by: Chunhai Guo <guochunhai@vivo.com> > Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com> Reviewed-by: Yue Hu <huyue2@coolpad.com> ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v4] erofs: relaxed temporary buffers allocation on readahead @ 2024-01-27 1:49 ` Yue Hu 0 siblings, 0 replies; 20+ messages in thread From: Yue Hu @ 2024-01-27 1:49 UTC (permalink / raw) To: Gao Xiang; +Cc: linux-erofs, LKML, Chunhai Guo, Gao Xiang, huyue2 On Fri, 26 Jan 2024 22:01:42 +0800 Gao Xiang <hsiangkao@linux.alibaba.com> wrote: > From: Chunhai Guo <guochunhai@vivo.com> > > Even with inplace decompression, sometimes very few temporary buffers > may be still needed for a single decompression shot (e.g. 16 pages for > 64k sliding window or 4 pages for 16k sliding window). In low-memory > scenarios, it would be better to try to allocate with GFP_NOWAIT on > readahead first. That can help reduce the time spent on page allocation > under durative memory pressure. > > Here are detailed performance numbers under multi-app launch benchmark > workload [1] on ARM64 Android devices (8-core CPU and 8GB of memory) > running a 5.15 LTS kernel with EROFS of 4k pclusters: > > +----------------------------------------------+ > | LZ4 | vanilla | patched | diff | > |----------------+---------+---------+---------| > | Average (ms) | 3364 | 2684 | -20.21% | [64k sliding window] > |----------------+---------+---------+---------| > | Average (ms) | 2079 | 1610 | -22.56% | [16k sliding window] > +----------------------------------------------+ > > The total size of system images for 4k pclusters is almost unchanged: > (64k sliding window) 9,117,044 KB > (16k sliding window) 9,113,096 KB > > Therefore, in addition to switch the sliding window from 64k to 16k, > after applying this patch, it can eventually save 52.14% (3364 -> 1610) > on average with no memory reservation. That is particularly useful for > embedded devices with limited resources. > > [1] https://lore.kernel.org/r/20240109074143.4138783-1-guochunhai@vivo.com > > Suggested-by: Gao Xiang <xiang@kernel.org> > Signed-off-by: Chunhai Guo <guochunhai@vivo.com> > Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com> Reviewed-by: Yue Hu <huyue2@coolpad.com> ^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2024-01-27 1:50 UTC | newest] Thread overview: 20+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2024-01-20 14:55 [PATCH v2] erofs: relaxed temporary buffers allocation on readahead Chunhai Guo via Linux-erofs 2024-01-22 2:07 ` Gao Xiang 2024-01-22 3:49 ` Chunhai Guo via Linux-erofs 2024-01-22 4:37 ` Gao Xiang 2024-01-22 7:42 ` Chunhai Guo via Linux-erofs 2024-01-26 2:41 ` Chunhai Guo via Linux-erofs 2024-01-26 2:47 ` Gao Xiang 2024-01-26 3:42 ` Chunhai Guo via Linux-erofs 2024-01-26 3:49 ` Gao Xiang 2024-01-26 3:56 ` 答复: " Chunhai Guo via Linux-erofs 2024-01-26 5:36 ` [PATCH v3] " Gao Xiang 2024-01-26 5:36 ` Gao Xiang 2024-01-26 10:46 ` Yue Hu 2024-01-26 10:46 ` Yue Hu 2024-01-26 13:30 ` Gao Xiang 2024-01-26 13:30 ` Gao Xiang 2024-01-26 14:01 ` [PATCH v4] " Gao Xiang 2024-01-26 14:01 ` Gao Xiang 2024-01-27 1:49 ` Yue Hu 2024-01-27 1:49 ` Yue Hu
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.