[PATCH] fs: place f_ref to 3rd cache line in struct file to resolve false sharing

[PATCH] fs: place f_ref to 3rd cache line in struct file to resolve false sharing

[Pan Deng]

When running syscall pread in a high core count system, f_ref contends
with the reading of f_mode, f_op, f_mapping, f_inode, f_flags in the
same cache line.

This change places f_ref to the 3rd cache line where fields are not
updated as frequently as the 1st cache line, and the contention is
grealy reduced according to tests. In addition, the size of file
object is kept in 3 cache lines.

This change has been tested with rocksdb benchmark readwhilewriting case
in 1 socket 64 physical core 128 logical core baremetal machine, with
build config CONFIG_RANDSTRUCT_NONE=y
Command:
./db_bench --benchmarks="readwhilewriting" --threads $cnt --duration 60
The throughput(ops/s) is improved up to ~21%.
=====
thread		baseline	compare
16		 100%		 +1.3%
32		 100%		 +2.2%
64		 100%		 +7.2%
128		 100%		 +20.9%

It was also tested with UnixBench: syscall, fsbuffer, fstime,
fsdisk cases that has been used for file struct layout tuning, no
regression was observed.

Signed-off-by: Pan Deng <pan.deng@intel.com>
Reviewed-by: Tianyou Li <tianyou.li@intel.com>
Reviewed-by: Tim Chen <tim.c.chen@linux.intel.com>
Tested-by: Lipeng Zhu <lipeng.zhu@intel.com>
---
 include/linux/fs.h | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/include/linux/fs.h b/include/linux/fs.h
index 2c3b2f8a621f..63b402cc9219 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -1067,7 +1067,6 @@ static inline int ra_has_index(struct file_ra_state *ra, pgoff_t index)
 
 /**
  * struct file - Represents a file
- * @f_ref: reference count
  * @f_lock: Protects f_ep, f_flags. Must not be taken from IRQ context.
  * @f_mode: FMODE_* flags often used in hotpaths
  * @f_op: file operations
@@ -1090,9 +1089,9 @@ static inline int ra_has_index(struct file_ra_state *ra, pgoff_t index)
  * @f_llist: work queue entrypoint
  * @f_ra: file's readahead state
  * @f_freeptr: Pointer used by SLAB_TYPESAFE_BY_RCU file cache (don't touch.)
+ * @f_ref: reference count
  */
 struct file {
-	file_ref_t			f_ref;
 	spinlock_t			f_lock;
 	fmode_t				f_mode;
 	const struct file_operations	*f_op;
@@ -1102,6 +1101,7 @@ struct file {
 	unsigned int			f_flags;
 	unsigned int			f_iocb_flags;
 	const struct cred		*f_cred;
+	u8				padding[8];
 	/* --- cacheline 1 boundary (64 bytes) --- */
 	struct path			f_path;
 	union {
@@ -1127,6 +1127,7 @@ struct file {
 		struct file_ra_state	f_ra;
 		freeptr_t		f_freeptr;
 	};
+	file_ref_t			f_ref;
 	/* --- cacheline 3 boundary (192 bytes) --- */
 } __randomize_layout
   __attribute__((aligned(4)));	/* lest something weird decides that 2 is OK */
-- 
2.43.5

Re: [PATCH] fs: place f_ref to 3rd cache line in struct file to resolve false sharing

[Christian Brauner]

On Fri, 28 Feb 2025 10:00:59 +0800, Pan Deng wrote:
> When running syscall pread in a high core count system, f_ref contends
> with the reading of f_mode, f_op, f_mapping, f_inode, f_flags in the
> same cache line.
> 
> This change places f_ref to the 3rd cache line where fields are not
> updated as frequently as the 1st cache line, and the contention is
> grealy reduced according to tests. In addition, the size of file
> object is kept in 3 cache lines.
> 
> [...]

Applied to the vfs-6.15.misc branch of the vfs/vfs.git tree.
Patches in the vfs-6.15.misc branch should appear in linux-next soon.

Please report any outstanding bugs that were missed during review in a
new review to the original patch series allowing us to drop it.

It's encouraged to provide Acked-bys and Reviewed-bys even though the
patch has now been applied. If possible patch trailers will be updated.

Note that commit hashes shown below are subject to change due to rebase,
trailer updates or similar. If in doubt, please check the listed branch.

tree:   https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git
branch: vfs-6.15.misc

[1/1] fs: place f_ref to 3rd cache line in struct file to resolve false sharing
      https://git.kernel.org/vfs/vfs/c/6a9ebf00c3be

Re: [PATCH] fs: place f_ref to 3rd cache line in struct file to resolve false sharing

[Jan Kara]

On Fri 28-02-25 10:00:59, Pan Deng wrote:
> When running syscall pread in a high core count system, f_ref contends
> with the reading of f_mode, f_op, f_mapping, f_inode, f_flags in the
> same cache line.

Well, but you have to have mulithreaded process using the same struct file
for the IO, don't you? Otherwise f_ref is not touched...

> This change places f_ref to the 3rd cache line where fields are not
> updated as frequently as the 1st cache line, and the contention is
> grealy reduced according to tests. In addition, the size of file
> object is kept in 3 cache lines.
> 
> This change has been tested with rocksdb benchmark readwhilewriting case
> in 1 socket 64 physical core 128 logical core baremetal machine, with
> build config CONFIG_RANDSTRUCT_NONE=y
> Command:
> ./db_bench --benchmarks="readwhilewriting" --threads $cnt --duration 60
> The throughput(ops/s) is improved up to ~21%.
> =====
> thread		baseline	compare
> 16		 100%		 +1.3%
> 32		 100%		 +2.2%
> 64		 100%		 +7.2%
> 128		 100%		 +20.9%
> 
> It was also tested with UnixBench: syscall, fsbuffer, fstime,
> fsdisk cases that has been used for file struct layout tuning, no
> regression was observed.

So overall keeping the first cacheline read mostly with important stuff
makes sense to limit cache traffic. But:

>  struct file {
> -	file_ref_t			f_ref;
>  	spinlock_t			f_lock;
>  	fmode_t				f_mode;
>  	const struct file_operations	*f_op;
> @@ -1102,6 +1101,7 @@ struct file {
>  	unsigned int			f_flags;
>  	unsigned int			f_iocb_flags;
>  	const struct cred		*f_cred;
> +	u8				padding[8];
>  	/* --- cacheline 1 boundary (64 bytes) --- */
>  	struct path			f_path;
>  	union {
> @@ -1127,6 +1127,7 @@ struct file {
>  		struct file_ra_state	f_ra;
>  		freeptr_t		f_freeptr;
>  	};
> +	file_ref_t			f_ref;
>  	/* --- cacheline 3 boundary (192 bytes) --- */
>  } __randomize_layout

This keeps struct file within 3 cachelines but it actually grows it from
184 to 192 bytes (and yes, that changes how many file structs we can fit in
a slab). So instead of adding 8 bytes of padding, just pick some
read-mostly element and move it into the hole - f_owner looks like one
possible candidate.

Also did you test how moving f_ref to the second cache line instead of the
third one behaves?

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: [PATCH] fs: place f_ref to 3rd cache line in struct file to resolve false sharing

[Christian Brauner]

On Fri, Feb 28, 2025 at 08:51:27PM +0100, Jan Kara wrote:
> On Fri 28-02-25 10:00:59, Pan Deng wrote:
> > When running syscall pread in a high core count system, f_ref contends
> > with the reading of f_mode, f_op, f_mapping, f_inode, f_flags in the
> > same cache line.
> 
> Well, but you have to have mulithreaded process using the same struct file
> for the IO, don't you? Otherwise f_ref is not touched...

Yes, it's specifically designed to scale better under high contention.

> 
> > This change places f_ref to the 3rd cache line where fields are not
> > updated as frequently as the 1st cache line, and the contention is
> > grealy reduced according to tests. In addition, the size of file
> > object is kept in 3 cache lines.
> > 
> > This change has been tested with rocksdb benchmark readwhilewriting case
> > in 1 socket 64 physical core 128 logical core baremetal machine, with
> > build config CONFIG_RANDSTRUCT_NONE=y
> > Command:
> > ./db_bench --benchmarks="readwhilewriting" --threads $cnt --duration 60
> > The throughput(ops/s) is improved up to ~21%.
> > =====
> > thread		baseline	compare
> > 16		 100%		 +1.3%
> > 32		 100%		 +2.2%
> > 64		 100%		 +7.2%
> > 128		 100%		 +20.9%
> > 
> > It was also tested with UnixBench: syscall, fsbuffer, fstime,
> > fsdisk cases that has been used for file struct layout tuning, no
> > regression was observed.
> 
> So overall keeping the first cacheline read mostly with important stuff
> makes sense to limit cache traffic. But:
> 
> >  struct file {
> > -	file_ref_t			f_ref;
> >  	spinlock_t			f_lock;
> >  	fmode_t				f_mode;
> >  	const struct file_operations	*f_op;
> > @@ -1102,6 +1101,7 @@ struct file {
> >  	unsigned int			f_flags;
> >  	unsigned int			f_iocb_flags;
> >  	const struct cred		*f_cred;
> > +	u8				padding[8];
> >  	/* --- cacheline 1 boundary (64 bytes) --- */
> >  	struct path			f_path;
> >  	union {
> > @@ -1127,6 +1127,7 @@ struct file {
> >  		struct file_ra_state	f_ra;
> >  		freeptr_t		f_freeptr;
> >  	};
> > +	file_ref_t			f_ref;
> >  	/* --- cacheline 3 boundary (192 bytes) --- */
> >  } __randomize_layout
> 
> This keeps struct file within 3 cachelines but it actually grows it from
> 184 to 192 bytes (and yes, that changes how many file structs we can fit in
> a slab). So instead of adding 8 bytes of padding, just pick some
> read-mostly element and move it into the hole - f_owner looks like one
> possible candidate.

This is what I did. See vfs-6.15.misc! Thanks!

> 
> Also did you test how moving f_ref to the second cache line instead of the
> third one behaves?
> 
> 								Honza
> -- 
> Jan Kara <jack@suse.com>
> SUSE Labs, CR

Re: [PATCH] fs: place f_ref to 3rd cache line in struct file to resolve false sharing

[Jan Kara]

On Sat 01-03-25 11:07:12, Christian Brauner wrote:
> > > @@ -1127,6 +1127,7 @@ struct file {
> > >  		struct file_ra_state	f_ra;
> > >  		freeptr_t		f_freeptr;
> > >  	};
> > > +	file_ref_t			f_ref;
> > >  	/* --- cacheline 3 boundary (192 bytes) --- */
> > >  } __randomize_layout
> > 
> > This keeps struct file within 3 cachelines but it actually grows it from
> > 184 to 192 bytes (and yes, that changes how many file structs we can fit in
> > a slab). So instead of adding 8 bytes of padding, just pick some
> > read-mostly element and move it into the hole - f_owner looks like one
> > possible candidate.
> 
> This is what I did. See vfs-6.15.misc! Thanks!

Thanks. Looks good! BTW now I've realized that with struct file having 184
bytes, the cacheline alignment of the structure need not be the one
described by the struct definition due to the allocation starting in the
middle of the cacheline. It will introduce some noise in the results but I
guess overall this should still be a win.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

RE: [PATCH] fs: place f_ref to 3rd cache line in struct file to resolve false sharing

[Deng, Pan]

Thanks Honza and Brauner,
For the question moving f_ref to 2nd cache line, in rocksdb readwhilewriting test, f_ref contends with f_path, f_pos_lock and f_security; while in 3rd cache line, it contends with prev_pos of file_ra_state; the score of placing in 2nd cache line(exchange f_security) is observed ~10% worse than in 3rd cache line.
Moving f_owner to 1st cache line(in vfs-6.15.misc) keeps the same performance according to rocksdb test.

Best Regards
Pan

> -----Original Message-----
> From: Jan Kara <jack@suse.cz>
> Sent: Tuesday, March 4, 2025 6:36 PM
> To: Christian Brauner <brauner@kernel.org>
> Cc: Jan Kara <jack@suse.cz>; Deng, Pan <pan.deng@intel.com>;
> viro@zeniv.linux.org.uk; linux-fsdevel@vger.kernel.org; Li, Tianyou
> <tianyou.li@intel.com>; tim.c.chen@linux.intel.com; Zhu, Lipeng
> <lipeng.zhu@intel.com>
> Subject: Re: [PATCH] fs: place f_ref to 3rd cache line in struct file to resolve false
> sharing
> 
> On Sat 01-03-25 11:07:12, Christian Brauner wrote:
> > > > @@ -1127,6 +1127,7 @@ struct file {
> > > >  		struct file_ra_state	f_ra;
> > > >  		freeptr_t		f_freeptr;
> > > >  	};
> > > > +	file_ref_t			f_ref;
> > > >  	/* --- cacheline 3 boundary (192 bytes) --- */  }
> > > > __randomize_layout
> > >
> > > This keeps struct file within 3 cachelines but it actually grows it
> > > from
> > > 184 to 192 bytes (and yes, that changes how many file structs we can
> > > fit in a slab). So instead of adding 8 bytes of padding, just pick
> > > some read-mostly element and move it into the hole - f_owner looks
> > > like one possible candidate.
> >
> > This is what I did. See vfs-6.15.misc! Thanks!
> 
> Thanks. Looks good! BTW now I've realized that with struct file having 184
> bytes, the cacheline alignment of the structure need not be the one described
> by the struct definition due to the allocation starting in the middle of the
> cacheline. It will introduce some noise in the results but I guess overall this
> should still be a win.
> 
> 								Honza
> --
> Jan Kara <jack@suse.com>
> SUSE Labs, CR