All of lore.kernel.org
 help / color / mirror / Atom feed
* Re: [RFC PATCH 6/6] erofs: enable page cache sharing in fscache mode
@ 2023-01-19  1:06 kernel test robot
  0 siblings, 0 replies; 5+ messages in thread
From: kernel test robot @ 2023-01-19  1:06 UTC (permalink / raw)
  To: oe-kbuild; +Cc: lkp

:::::: 
:::::: Manual check reason: "gcc: unused function warning for stub function"
:::::: 

BCC: lkp@intel.com
CC: llvm@lists.linux.dev
CC: oe-kbuild-all@lists.linux.dev
In-Reply-To: <20230106125330.55529-7-jefflexu@linux.alibaba.com>
References: <20230106125330.55529-7-jefflexu@linux.alibaba.com>
TO: Jingbo Xu <jefflexu@linux.alibaba.com>

Hi Jingbo,

[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on xiang-erofs/dev-test]
[also build test WARNING on xiang-erofs/dev xiang-erofs/fixes linus/master v6.2-rc4 next-20230118]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Jingbo-Xu/erofs-remove-unused-device-mapping-in-the-meta-routine/20230106-205600
base:   https://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs.git dev-test
patch link:    https://lore.kernel.org/r/20230106125330.55529-7-jefflexu%40linux.alibaba.com
patch subject: [RFC PATCH 6/6] erofs: enable page cache sharing in fscache mode
:::::: branch date: 13 days ago
:::::: commit date: 13 days ago
config: x86_64-randconfig-a014 (https://download.01.org/0day-ci/archive/20230119/202301190850.EDJzy3Cz-lkp@intel.com/config)
compiler: clang version 14.0.6 (https://github.com/llvm/llvm-project f28c006a5895fc0e329fe15fead81e37457cb1d1)
reproduce (this is a W=1 build):
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # https://github.com/intel-lab-lkp/linux/commit/5c565bd6afacc699309e9fc20119d91501a5d3ae
        git remote add linux-review https://github.com/intel-lab-lkp/linux
        git fetch --no-tags linux-review Jingbo-Xu/erofs-remove-unused-device-mapping-in-the-meta-routine/20230106-205600
        git checkout 5c565bd6afacc699309e9fc20119d91501a5d3ae
        # save the config file
        mkdir build_dir && cp config build_dir/.config
        COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=x86_64 olddefconfig
        COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=x86_64 SHELL=/bin/bash fs/erofs/

If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <lkp@intel.com>

All warnings (new ones prefixed by >>):

>> fs/erofs/inode.c:244:13: warning: unused function 'erofs_can_share_page_cache' [-Wunused-function]
   static bool erofs_can_share_page_cache(struct inode *inode)
               ^
   1 warning generated.


vim +/erofs_can_share_page_cache +244 fs/erofs/inode.c

431339ba90423a0 drivers/staging/erofs/inode.c Gao Xiang 2018-07-26  243  
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06 @244  static bool erofs_can_share_page_cache(struct inode *inode)
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  245  {
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  246  	struct erofs_inode *vi = EROFS_I(inode);
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  247  
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  248  	/* enable page cache sharing only in share domain mode */
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  249  	if (!erofs_is_fscache_mode(inode->i_sb) ||
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  250  	    !EROFS_SB(inode->i_sb)->domain_id)
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  251  		return false;
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  252  
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  253  	if (vi->datalayout != EROFS_INODE_CHUNK_BASED)
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  254  		return false;
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  255  
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  256  	/* avoid crossing multi devicces/blobs */
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  257  	if (inode->i_size > 1UL << vi->chunkbits)
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  258  		return false;
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  259  
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  260  	/* avoid data leakage in mmap routine */
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  261  	if (EROFS_BLKSIZ % PAGE_SIZE)
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  262  		return false;
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  263  
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  264  	return true;
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  265  }
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  266  

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests

^ permalink raw reply	[flat|nested] 5+ messages in thread
* Re: [RFC PATCH 6/6] erofs: enable page cache sharing in fscache mode
@ 2023-01-18 23:01 kernel test robot
  0 siblings, 0 replies; 5+ messages in thread
From: kernel test robot @ 2023-01-18 23:01 UTC (permalink / raw)
  To: oe-kbuild; +Cc: lkp

:::::: 
:::::: Manual check reason: "gcc: unused function warning for stub function"
:::::: 

BCC: lkp@intel.com
CC: oe-kbuild-all@lists.linux.dev
In-Reply-To: <20230106125330.55529-7-jefflexu@linux.alibaba.com>
References: <20230106125330.55529-7-jefflexu@linux.alibaba.com>
TO: Jingbo Xu <jefflexu@linux.alibaba.com>

Hi Jingbo,

[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on xiang-erofs/dev-test]
[also build test WARNING on xiang-erofs/dev xiang-erofs/fixes linus/master v6.2-rc4 next-20230118]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Jingbo-Xu/erofs-remove-unused-device-mapping-in-the-meta-routine/20230106-205600
base:   https://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs.git dev-test
patch link:    https://lore.kernel.org/r/20230106125330.55529-7-jefflexu%40linux.alibaba.com
patch subject: [RFC PATCH 6/6] erofs: enable page cache sharing in fscache mode
:::::: branch date: 12 days ago
:::::: commit date: 12 days ago
config: i386-randconfig-a004-20220328 (https://download.01.org/0day-ci/archive/20230119/202301190654.hZ4dPPlb-lkp@intel.com/config)
compiler: gcc-11 (Debian 11.3.0-8) 11.3.0
reproduce (this is a W=1 build):
        # https://github.com/intel-lab-lkp/linux/commit/5c565bd6afacc699309e9fc20119d91501a5d3ae
        git remote add linux-review https://github.com/intel-lab-lkp/linux
        git fetch --no-tags linux-review Jingbo-Xu/erofs-remove-unused-device-mapping-in-the-meta-routine/20230106-205600
        git checkout 5c565bd6afacc699309e9fc20119d91501a5d3ae
        # save the config file
        mkdir build_dir && cp config build_dir/.config
        make W=1 O=build_dir ARCH=i386 olddefconfig
        make W=1 O=build_dir ARCH=i386 SHELL=/bin/bash fs/ lib//

If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <lkp@intel.com>

All warnings (new ones prefixed by >>):

>> fs/erofs/inode.c:244:13: warning: 'erofs_can_share_page_cache' defined but not used [-Wunused-function]
     244 | static bool erofs_can_share_page_cache(struct inode *inode)
         |             ^~~~~~~~~~~~~~~~~~~~~~~~~~


vim +/erofs_can_share_page_cache +244 fs/erofs/inode.c

431339ba90423a0 drivers/staging/erofs/inode.c Gao Xiang 2018-07-26  243  
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06 @244  static bool erofs_can_share_page_cache(struct inode *inode)
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  245  {
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  246  	struct erofs_inode *vi = EROFS_I(inode);
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  247  
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  248  	/* enable page cache sharing only in share domain mode */
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  249  	if (!erofs_is_fscache_mode(inode->i_sb) ||
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  250  	    !EROFS_SB(inode->i_sb)->domain_id)
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  251  		return false;
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  252  
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  253  	if (vi->datalayout != EROFS_INODE_CHUNK_BASED)
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  254  		return false;
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  255  
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  256  	/* avoid crossing multi devicces/blobs */
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  257  	if (inode->i_size > 1UL << vi->chunkbits)
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  258  		return false;
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  259  
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  260  	/* avoid data leakage in mmap routine */
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  261  	if (EROFS_BLKSIZ % PAGE_SIZE)
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  262  		return false;
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  263  
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  264  	return true;
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  265  }
5c565bd6afacc69 fs/erofs/inode.c              Jingbo Xu 2023-01-06  266  

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests

^ permalink raw reply	[flat|nested] 5+ messages in thread
* [RFC PATCH 0/6] erofs: support page cache sharing between EROFS images in fscache mode
@ 2023-01-06 12:53 Jingbo Xu
  2023-01-06 12:53   ` Jingbo Xu
  0 siblings, 1 reply; 5+ messages in thread
From: Jingbo Xu @ 2023-01-06 12:53 UTC (permalink / raw)
  To: xiang, chao, linux-erofs; +Cc: linux-fsdevel, huyue2, linux-kernel

[Background]
=============
Erofs already supports chunk deduplication across different images to
minimize disk usage since v6.1.  Furthermore, we can make inodes among
different images share page cache for these deduplicated chunks to
reduce the memory usage.  This shall be much usable in container
scenarios as deduplication is requisite for container images.


[Implementation]
================
This is achieved by managing page cache of deduplicated chunks in
blob's address space.  In this way, all inodes sharing the deduplicated
chunk will refer to and share the page cache in the blob's address
space.


[Restriction]
==============
The page cache sharing feature also supports .mmap().  The reverse
mapping requires that one vma can not be shared among inodes and can
be linked to only one inode.  As the vma will be finally linked to the
blob's address space when page cache sharing enabled, the restriction of
the reverse mapping actually requires that the mapped file area can not
be mapped to multiple blobs.  Thus page cache sharing can only be
enabled for those files mapped to one blob.

The chunk based data layout guarantees that a chunk will not cross the
device (blob) boundary.  Thus in chunk based data layout, those files
smaller than the chunk size shall be guaranteed to be mapped to one
blob.  As chunk size is tunable at a per-file basis, this restriction
can be relaxed at image building phase.  As long as we ensure that the
file can not be deduplicated, the file's chunk size can be set to a
reasonable value larger than the file size, so that the file contains
only one chunk, in which case page cache sharing feature can be enabled
on this file later.


[Effect]
========
The final optimization result of this feature depends on the following
factors:

1. The number of deduplicated (shared) chunks.  Images sharing most of
the layers (e.g. a base image and v1 image based on the base image) will
achieve better optimization.

2. As the restriction mentioned above, the number of files for which
page cache sharing can ben enabled among the files accessed.


I test the workload of starting up Tensorflow, which will access quite
many (~5K) files among the startup phase.  I get the base image of
Tensorflow from [1] and build a new image (e.g. v1 image) on top of this
base image.

Since the image got from [1] is in OCI format, I have to convert it to
erofs format with buildkit[2], with default chunk size of 1MB.

I run containers from these two images with containerd (base image first,
v2 image secondly).  The (page cache) memory usage of the rootfs
(container image) is shown as below:

			| page cache sharing	| page cache sharing
			| disabled		| enabled
------------------------|-----------------------|-------------------
First container       	|      			|
page cache usage (MB) 	| 150      		| 150
------------------------+-----------------------|-------------------
Second container      	|      			|
page cache usage (MB) 	| 150			| 7

It can be seen that most (~95%, 143MB/150MB) memory usage reduced under
this workload (when starting following containers sharing container image
layers).

The remained 7MB memory usage is consumed by directories, since page
cache sharing is enabled only for regular files in this RFC
implementation.


[1] docker.io/tensorflow/tensorflow:2.10.0
[2] https://github.com/moby/buildkit


Jingbo Xu (6):
  erofs: remove unused device mapping in the meta routine
  erofs: unify anonymous inodes for blob
  erofs: alloc anonymous file for blob in share domain mode
  erofs: implement .read_iter for page cache sharing
  erofs: implement .mmap for page cache sharing
  erofs: enable page cache sharing in fscache mode

 fs/erofs/fscache.c  | 231 ++++++++++++++++++++++++++++++++------------
 fs/erofs/inode.c    |  27 ++++++
 fs/erofs/internal.h |   8 +-
 fs/erofs/super.c    |   2 +
 4 files changed, 202 insertions(+), 66 deletions(-)

-- 
2.19.1.6.gb485710b


^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2023-01-19  8:57 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-01-19  1:06 [RFC PATCH 6/6] erofs: enable page cache sharing in fscache mode kernel test robot
  -- strict thread matches above, loose matches on Subject: below --
2023-01-18 23:01 kernel test robot
2023-01-06 12:53 [RFC PATCH 0/6] erofs: support page cache sharing between EROFS images " Jingbo Xu
2023-01-06 12:53 ` [RFC PATCH 6/6] erofs: enable page cache sharing " Jingbo Xu
2023-01-06 12:53   ` Jingbo Xu
2023-01-19  8:54   ` kernel test robot

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.