* [PATCH 1/2] erofs: allow readdir() to be interrupted @ 2025-07-10 7:36 Chao Yu 2025-07-10 7:36 ` [PATCH 2/2] erofs: support to readahead dirent blocks in erofs_readdir() Chao Yu 2025-07-10 7:41 ` [PATCH 1/2] erofs: allow readdir() to be interrupted Gao Xiang 0 siblings, 2 replies; 8+ messages in thread From: Chao Yu @ 2025-07-10 7:36 UTC (permalink / raw) To: xiang Cc: linux-erofs, linux-kernel, Yue Hu, Jeffle Xu, Sandeep Dhavale, Hongbo Li, Chao Yu In a quick slow device, readdir() may loop for long time in large directory, let's give a chance to allow it to be interrupted by userspace. Signed-off-by: Chao Yu <chao@kernel.org> --- fs/erofs/dir.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/fs/erofs/dir.c b/fs/erofs/dir.c index 2fae209d0274..cff61c5a172b 100644 --- a/fs/erofs/dir.c +++ b/fs/erofs/dir.c @@ -58,6 +58,13 @@ static int erofs_readdir(struct file *f, struct dir_context *ctx) struct erofs_dirent *de; unsigned int nameoff, maxsize; + /* allow readdir() to be interrupted */ + if (fatal_signal_pending(current)) { + err = -ERESTARTSYS; + break; + } + cond_resched(); + de = erofs_bread(&buf, dbstart, true); if (IS_ERR(de)) { erofs_err(sb, "failed to readdir of logical block %llu of nid %llu", -- 2.49.0 ^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH 2/2] erofs: support to readahead dirent blocks in erofs_readdir() 2025-07-10 7:36 [PATCH 1/2] erofs: allow readdir() to be interrupted Chao Yu @ 2025-07-10 7:36 ` Chao Yu 2025-07-10 7:46 ` Gao Xiang 2025-07-10 7:41 ` [PATCH 1/2] erofs: allow readdir() to be interrupted Gao Xiang 1 sibling, 1 reply; 8+ messages in thread From: Chao Yu @ 2025-07-10 7:36 UTC (permalink / raw) To: xiang Cc: linux-erofs, linux-kernel, Yue Hu, Jeffle Xu, Sandeep Dhavale, Hongbo Li, Chao Yu This patch supports to readahead more blocks in erofs_readdir(), it can enhance performance in large direcotry. readdir test in a large directory which contains 12000 sub-files. files_per_second Before: 926385.54 After: 2380435.562 Signed-off-by: Chao Yu <chao@kernel.org> --- fs/erofs/dir.c | 8 ++++++++ fs/erofs/internal.h | 3 +++ 2 files changed, 11 insertions(+) diff --git a/fs/erofs/dir.c b/fs/erofs/dir.c index cff61c5a172b..04113851fc0f 100644 --- a/fs/erofs/dir.c +++ b/fs/erofs/dir.c @@ -47,8 +47,10 @@ static int erofs_readdir(struct file *f, struct dir_context *ctx) struct inode *dir = file_inode(f); struct erofs_buf buf = __EROFS_BUF_INITIALIZER; struct super_block *sb = dir->i_sb; + struct file_ra_state *ra = &f->f_ra; unsigned long bsz = sb->s_blocksize; unsigned int ofs = erofs_blkoff(sb, ctx->pos); + unsigned long nr_pages = DIV_ROUND_UP_POW2(dir->i_size, PAGE_SIZE); int err = 0; bool initial = true; @@ -65,6 +67,12 @@ static int erofs_readdir(struct file *f, struct dir_context *ctx) } cond_resched(); + /* readahead blocks to enhance performance in large directory */ + if (nr_pages - dbstart > 1 && !ra_has_index(ra, dbstart)) + page_cache_sync_readahead(dir->i_mapping, ra, f, + dbstart, min(nr_pages - dbstart, + (pgoff_t)MAX_DIR_RA_PAGES)); + de = erofs_bread(&buf, dbstart, true); if (IS_ERR(de)) { erofs_err(sb, "failed to readdir of logical block %llu of nid %llu", diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h index a32c03a80c70..ef9d1ee8c688 100644 --- a/fs/erofs/internal.h +++ b/fs/erofs/internal.h @@ -238,6 +238,9 @@ EROFS_FEATURE_FUNCS(xattr_filter, compat, COMPAT_XATTR_FILTER) #define EROFS_I_BL_XATTR_BIT (BITS_PER_LONG - 1) #define EROFS_I_BL_Z_BIT (BITS_PER_LONG - 2) +/* maximum readahead pages of directory */ +#define MAX_DIR_RA_PAGES 4 + struct erofs_inode { erofs_nid_t nid; -- 2.49.0 ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH 2/2] erofs: support to readahead dirent blocks in erofs_readdir() 2025-07-10 7:36 ` [PATCH 2/2] erofs: support to readahead dirent blocks in erofs_readdir() Chao Yu @ 2025-07-10 7:46 ` Gao Xiang 2025-07-10 7:59 ` Chao Yu 0 siblings, 1 reply; 8+ messages in thread From: Gao Xiang @ 2025-07-10 7:46 UTC (permalink / raw) To: Chao Yu, xiang Cc: linux-erofs, linux-kernel, Yue Hu, Jeffle Xu, Sandeep Dhavale, Hongbo Li Hi Chao, On 2025/7/10 15:36, Chao Yu wrote: > This patch supports to readahead more blocks in erofs_readdir(), > it can enhance performance in large direcotry. > > readdir test in a large directory which contains 12000 sub-files. > > files_per_second > Before: 926385.54 > After: 2380435.562 > > Signed-off-by: Chao Yu <chao@kernel.org> > --- > fs/erofs/dir.c | 8 ++++++++ > fs/erofs/internal.h | 3 +++ > 2 files changed, 11 insertions(+) > > diff --git a/fs/erofs/dir.c b/fs/erofs/dir.c > index cff61c5a172b..04113851fc0f 100644 > --- a/fs/erofs/dir.c > +++ b/fs/erofs/dir.c > @@ -47,8 +47,10 @@ static int erofs_readdir(struct file *f, struct dir_context *ctx) > struct inode *dir = file_inode(f); > struct erofs_buf buf = __EROFS_BUF_INITIALIZER; > struct super_block *sb = dir->i_sb; > + struct file_ra_state *ra = &f->f_ra; > unsigned long bsz = sb->s_blocksize; > unsigned int ofs = erofs_blkoff(sb, ctx->pos); > + unsigned long nr_pages = DIV_ROUND_UP_POW2(dir->i_size, PAGE_SIZE); > int err = 0; > bool initial = true; > > @@ -65,6 +67,12 @@ static int erofs_readdir(struct file *f, struct dir_context *ctx) > } > cond_resched(); > > + /* readahead blocks to enhance performance in large directory */ > + if (nr_pages - dbstart > 1 && !ra_has_index(ra, dbstart)) > + page_cache_sync_readahead(dir->i_mapping, ra, f, > + dbstart, min(nr_pages - dbstart, > + (pgoff_t)MAX_DIR_RA_PAGES)); > + > de = erofs_bread(&buf, dbstart, true); > if (IS_ERR(de)) { > erofs_err(sb, "failed to readdir of logical block %llu of nid %llu", > diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h > index a32c03a80c70..ef9d1ee8c688 100644 > --- a/fs/erofs/internal.h > +++ b/fs/erofs/internal.h > @@ -238,6 +238,9 @@ EROFS_FEATURE_FUNCS(xattr_filter, compat, COMPAT_XATTR_FILTER) > #define EROFS_I_BL_XATTR_BIT (BITS_PER_LONG - 1) > #define EROFS_I_BL_Z_BIT (BITS_PER_LONG - 2) > > +/* maximum readahead pages of directory */ > +#define MAX_DIR_RA_PAGES 4 Could we set it as a per-sb sysfs configuration for users to config? Thanks, Gao Xiang ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 2/2] erofs: support to readahead dirent blocks in erofs_readdir() 2025-07-10 7:46 ` Gao Xiang @ 2025-07-10 7:59 ` Chao Yu 2025-07-10 8:04 ` Gao Xiang 0 siblings, 1 reply; 8+ messages in thread From: Chao Yu @ 2025-07-10 7:59 UTC (permalink / raw) To: Gao Xiang, xiang Cc: chao, linux-erofs, linux-kernel, Yue Hu, Jeffle Xu, Sandeep Dhavale, Hongbo Li On 7/10/25 15:46, Gao Xiang wrote: > Hi Chao, > > On 2025/7/10 15:36, Chao Yu wrote: >> This patch supports to readahead more blocks in erofs_readdir(), >> it can enhance performance in large direcotry. >> >> readdir test in a large directory which contains 12000 sub-files. >> >> files_per_second >> Before: 926385.54 >> After: 2380435.562 >> >> Signed-off-by: Chao Yu <chao@kernel.org> >> --- >> fs/erofs/dir.c | 8 ++++++++ >> fs/erofs/internal.h | 3 +++ >> 2 files changed, 11 insertions(+) >> >> diff --git a/fs/erofs/dir.c b/fs/erofs/dir.c >> index cff61c5a172b..04113851fc0f 100644 >> --- a/fs/erofs/dir.c >> +++ b/fs/erofs/dir.c >> @@ -47,8 +47,10 @@ static int erofs_readdir(struct file *f, struct dir_context *ctx) >> struct inode *dir = file_inode(f); >> struct erofs_buf buf = __EROFS_BUF_INITIALIZER; >> struct super_block *sb = dir->i_sb; >> + struct file_ra_state *ra = &f->f_ra; >> unsigned long bsz = sb->s_blocksize; >> unsigned int ofs = erofs_blkoff(sb, ctx->pos); >> + unsigned long nr_pages = DIV_ROUND_UP_POW2(dir->i_size, PAGE_SIZE); >> int err = 0; >> bool initial = true; >> @@ -65,6 +67,12 @@ static int erofs_readdir(struct file *f, struct dir_context *ctx) >> } >> cond_resched(); >> + /* readahead blocks to enhance performance in large directory */ >> + if (nr_pages - dbstart > 1 && !ra_has_index(ra, dbstart)) >> + page_cache_sync_readahead(dir->i_mapping, ra, f, >> + dbstart, min(nr_pages - dbstart, >> + (pgoff_t)MAX_DIR_RA_PAGES)); >> + >> de = erofs_bread(&buf, dbstart, true); >> if (IS_ERR(de)) { >> erofs_err(sb, "failed to readdir of logical block %llu of nid %llu", >> diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h >> index a32c03a80c70..ef9d1ee8c688 100644 >> --- a/fs/erofs/internal.h >> +++ b/fs/erofs/internal.h >> @@ -238,6 +238,9 @@ EROFS_FEATURE_FUNCS(xattr_filter, compat, COMPAT_XATTR_FILTER) >> #define EROFS_I_BL_XATTR_BIT (BITS_PER_LONG - 1) >> #define EROFS_I_BL_Z_BIT (BITS_PER_LONG - 2) >> +/* maximum readahead pages of directory */ >> +#define MAX_DIR_RA_PAGES 4 > > Could we set it as a per-sb sysfs configuration for users to config? Xiang, Oh, that will be better, how about introducing new sysfs in separated patch? Thanks, > > Thanks, > Gao Xiang ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 2/2] erofs: support to readahead dirent blocks in erofs_readdir() 2025-07-10 7:59 ` Chao Yu @ 2025-07-10 8:04 ` Gao Xiang 2025-07-10 13:15 ` Chao Yu 0 siblings, 1 reply; 8+ messages in thread From: Gao Xiang @ 2025-07-10 8:04 UTC (permalink / raw) To: Chao Yu, xiang Cc: linux-erofs, linux-kernel, Yue Hu, Jeffle Xu, Sandeep Dhavale, Hongbo Li On 2025/7/10 15:59, Chao Yu wrote: > On 7/10/25 15:46, Gao Xiang wrote: >> Hi Chao, >> >> On 2025/7/10 15:36, Chao Yu wrote: >>> This patch supports to readahead more blocks in erofs_readdir(), >>> it can enhance performance in large direcotry. >>> >>> readdir test in a large directory which contains 12000 sub-files. >>> >>> files_per_second >>> Before: 926385.54 >>> After: 2380435.562 >>> >>> Signed-off-by: Chao Yu <chao@kernel.org> >>> --- >>> fs/erofs/dir.c | 8 ++++++++ >>> fs/erofs/internal.h | 3 +++ >>> 2 files changed, 11 insertions(+) >>> >>> diff --git a/fs/erofs/dir.c b/fs/erofs/dir.c >>> index cff61c5a172b..04113851fc0f 100644 >>> --- a/fs/erofs/dir.c >>> +++ b/fs/erofs/dir.c >>> @@ -47,8 +47,10 @@ static int erofs_readdir(struct file *f, struct dir_context *ctx) >>> struct inode *dir = file_inode(f); >>> struct erofs_buf buf = __EROFS_BUF_INITIALIZER; >>> struct super_block *sb = dir->i_sb; >>> + struct file_ra_state *ra = &f->f_ra; >>> unsigned long bsz = sb->s_blocksize; >>> unsigned int ofs = erofs_blkoff(sb, ctx->pos); >>> + unsigned long nr_pages = DIV_ROUND_UP_POW2(dir->i_size, PAGE_SIZE); >>> int err = 0; >>> bool initial = true; >>> @@ -65,6 +67,12 @@ static int erofs_readdir(struct file *f, struct dir_context *ctx) >>> } >>> cond_resched(); >>> + /* readahead blocks to enhance performance in large directory */ >>> + if (nr_pages - dbstart > 1 && !ra_has_index(ra, dbstart)) >>> + page_cache_sync_readahead(dir->i_mapping, ra, f, >>> + dbstart, min(nr_pages - dbstart, >>> + (pgoff_t)MAX_DIR_RA_PAGES)); >>> + >>> de = erofs_bread(&buf, dbstart, true); >>> if (IS_ERR(de)) { >>> erofs_err(sb, "failed to readdir of logical block %llu of nid %llu", >>> diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h >>> index a32c03a80c70..ef9d1ee8c688 100644 >>> --- a/fs/erofs/internal.h >>> +++ b/fs/erofs/internal.h >>> @@ -238,6 +238,9 @@ EROFS_FEATURE_FUNCS(xattr_filter, compat, COMPAT_XATTR_FILTER) >>> #define EROFS_I_BL_XATTR_BIT (BITS_PER_LONG - 1) >>> #define EROFS_I_BL_Z_BIT (BITS_PER_LONG - 2) >>> +/* maximum readahead pages of directory */ >>> +#define MAX_DIR_RA_PAGES 4 >> >> Could we set it as a per-sb sysfs configuration for users to config? > > Xiang, > > Oh, that will be better, how about introducing new sysfs in separated patch? Hi Chao, Thanks for your interest but in that case it could cause some bisect issue anyway, could we just use one patch to add this optimization for slow devices? Thanks, Gao Xiang > > Thanks, > >> >> Thanks, >> Gao Xiang > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 2/2] erofs: support to readahead dirent blocks in erofs_readdir() 2025-07-10 8:04 ` Gao Xiang @ 2025-07-10 13:15 ` Chao Yu 0 siblings, 0 replies; 8+ messages in thread From: Chao Yu @ 2025-07-10 13:15 UTC (permalink / raw) To: Gao Xiang, xiang Cc: chao, linux-erofs, linux-kernel, Yue Hu, Jeffle Xu, Sandeep Dhavale, Hongbo Li On 7/10/25 16:04, Gao Xiang wrote: > > > On 2025/7/10 15:59, Chao Yu wrote: >> On 7/10/25 15:46, Gao Xiang wrote: >>> Hi Chao, >>> >>> On 2025/7/10 15:36, Chao Yu wrote: >>>> This patch supports to readahead more blocks in erofs_readdir(), >>>> it can enhance performance in large direcotry. >>>> >>>> readdir test in a large directory which contains 12000 sub-files. >>>> >>>> files_per_second >>>> Before: 926385.54 >>>> After: 2380435.562 >>>> >>>> Signed-off-by: Chao Yu <chao@kernel.org> >>>> --- >>>> fs/erofs/dir.c | 8 ++++++++ >>>> fs/erofs/internal.h | 3 +++ >>>> 2 files changed, 11 insertions(+) >>>> >>>> diff --git a/fs/erofs/dir.c b/fs/erofs/dir.c >>>> index cff61c5a172b..04113851fc0f 100644 >>>> --- a/fs/erofs/dir.c >>>> +++ b/fs/erofs/dir.c >>>> @@ -47,8 +47,10 @@ static int erofs_readdir(struct file *f, struct dir_context *ctx) >>>> struct inode *dir = file_inode(f); >>>> struct erofs_buf buf = __EROFS_BUF_INITIALIZER; >>>> struct super_block *sb = dir->i_sb; >>>> + struct file_ra_state *ra = &f->f_ra; >>>> unsigned long bsz = sb->s_blocksize; >>>> unsigned int ofs = erofs_blkoff(sb, ctx->pos); >>>> + unsigned long nr_pages = DIV_ROUND_UP_POW2(dir->i_size, PAGE_SIZE); >>>> int err = 0; >>>> bool initial = true; >>>> @@ -65,6 +67,12 @@ static int erofs_readdir(struct file *f, struct dir_context *ctx) >>>> } >>>> cond_resched(); >>>> + /* readahead blocks to enhance performance in large directory */ >>>> + if (nr_pages - dbstart > 1 && !ra_has_index(ra, dbstart)) >>>> + page_cache_sync_readahead(dir->i_mapping, ra, f, >>>> + dbstart, min(nr_pages - dbstart, >>>> + (pgoff_t)MAX_DIR_RA_PAGES)); >>>> + >>>> de = erofs_bread(&buf, dbstart, true); >>>> if (IS_ERR(de)) { >>>> erofs_err(sb, "failed to readdir of logical block %llu of nid %llu", >>>> diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h >>>> index a32c03a80c70..ef9d1ee8c688 100644 >>>> --- a/fs/erofs/internal.h >>>> +++ b/fs/erofs/internal.h >>>> @@ -238,6 +238,9 @@ EROFS_FEATURE_FUNCS(xattr_filter, compat, COMPAT_XATTR_FILTER) >>>> #define EROFS_I_BL_XATTR_BIT (BITS_PER_LONG - 1) >>>> #define EROFS_I_BL_Z_BIT (BITS_PER_LONG - 2) >>>> +/* maximum readahead pages of directory */ >>>> +#define MAX_DIR_RA_PAGES 4 >>> >>> Could we set it as a per-sb sysfs configuration for users to config? >> >> Xiang, >> >> Oh, that will be better, how about introducing new sysfs in separated patch? > > Hi Chao, > > Thanks for your interest but in that case it could cause some bisect > issue anyway, could we just use one patch to add this optimization > for slow devices? Xiang, Yeah, one patch includes all these looks fine to me, anyway, let me update v2. Thanks, > > Thanks, > Gao Xiang > >> >> Thanks, >> >>> >>> Thanks, >>> Gao Xiang >> > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 1/2] erofs: allow readdir() to be interrupted 2025-07-10 7:36 [PATCH 1/2] erofs: allow readdir() to be interrupted Chao Yu 2025-07-10 7:36 ` [PATCH 2/2] erofs: support to readahead dirent blocks in erofs_readdir() Chao Yu @ 2025-07-10 7:41 ` Gao Xiang 2025-07-10 7:57 ` Chao Yu 1 sibling, 1 reply; 8+ messages in thread From: Gao Xiang @ 2025-07-10 7:41 UTC (permalink / raw) To: Chao Yu, xiang Cc: linux-erofs, linux-kernel, Yue Hu, Jeffle Xu, Sandeep Dhavale, Hongbo Li On 2025/7/10 15:36, Chao Yu wrote: > In a quick slow device, readdir() may loop for long time in large > directory, let's give a chance to allow it to be interrupted by > userspace. > > Signed-off-by: Chao Yu <chao@kernel.org> Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com> > --- > fs/erofs/dir.c | 7 +++++++ > 1 file changed, 7 insertions(+) > > diff --git a/fs/erofs/dir.c b/fs/erofs/dir.c > index 2fae209d0274..cff61c5a172b 100644 > --- a/fs/erofs/dir.c > +++ b/fs/erofs/dir.c > @@ -58,6 +58,13 @@ static int erofs_readdir(struct file *f, struct dir_context *ctx) > struct erofs_dirent *de; > unsigned int nameoff, maxsize; > > + /* allow readdir() to be interrupted */ Hi Chao, It seems that comment is unnecessary since the following code is obvious, if you have no objection I will remove this comment when applying. Thanks, Gao Xiang ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 1/2] erofs: allow readdir() to be interrupted 2025-07-10 7:41 ` [PATCH 1/2] erofs: allow readdir() to be interrupted Gao Xiang @ 2025-07-10 7:57 ` Chao Yu 0 siblings, 0 replies; 8+ messages in thread From: Chao Yu @ 2025-07-10 7:57 UTC (permalink / raw) To: Gao Xiang, xiang Cc: chao, linux-erofs, linux-kernel, Yue Hu, Jeffle Xu, Sandeep Dhavale, Hongbo Li On 7/10/25 15:41, Gao Xiang wrote: > > > On 2025/7/10 15:36, Chao Yu wrote: >> In a quick slow device, readdir() may loop for long time in large >> directory, let's give a chance to allow it to be interrupted by >> userspace. >> >> Signed-off-by: Chao Yu <chao@kernel.org> > > Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com> > >> --- >> fs/erofs/dir.c | 7 +++++++ >> 1 file changed, 7 insertions(+) >> >> diff --git a/fs/erofs/dir.c b/fs/erofs/dir.c >> index 2fae209d0274..cff61c5a172b 100644 >> --- a/fs/erofs/dir.c >> +++ b/fs/erofs/dir.c >> @@ -58,6 +58,13 @@ static int erofs_readdir(struct file *f, struct dir_context *ctx) >> struct erofs_dirent *de; >> unsigned int nameoff, maxsize; >> + /* allow readdir() to be interrupted */ > > Hi Chao, > > It seems that comment is unnecessary since the following code > is obvious, if you have no objection I will remove this > comment when applying. Xiang, sure, no problem. :) Thanks, > > Thanks, > Gao Xiang ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2025-07-10 13:15 UTC | newest] Thread overview: 8+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2025-07-10 7:36 [PATCH 1/2] erofs: allow readdir() to be interrupted Chao Yu 2025-07-10 7:36 ` [PATCH 2/2] erofs: support to readahead dirent blocks in erofs_readdir() Chao Yu 2025-07-10 7:46 ` Gao Xiang 2025-07-10 7:59 ` Chao Yu 2025-07-10 8:04 ` Gao Xiang 2025-07-10 13:15 ` Chao Yu 2025-07-10 7:41 ` [PATCH 1/2] erofs: allow readdir() to be interrupted Gao Xiang 2025-07-10 7:57 ` Chao Yu
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.